# Introduction to Digital agro

## Sensitivity Analysis of crop model

____

### Mikhail Gasanov

E-mail: Mikhail.Gasanov[a]skoltech.ru

tg:@misha_grol

## Clone utils and files from GitHub

```python
# !git clone https://github.com/mishagrol/Seminar_Sobol.git
# !cp -r ./Seminar_Sobol/* .
```

```python
%matplotlib inline
import sys, os
import pcse
import pandas as pd
import matplotlib
import yaml
matplotlib.style.use("ggplot")
import matplotlib.pyplot as plt
print("This notebook was built with:")
print("python version: %s " % sys.version)
print("PCSE version: %s" %  pcse.__version__)
```

    This notebook was built with:
    python version: 3.6.11 | packaged by conda-forge | (default, Aug  5 2020, 20:19:23)
    [GCC Clang 10.0.1 ]
    PCSE version: 5.4.2

```python
import warnings
warnings.filterwarnings("ignore")
```

# Sensitivity analysis of models
___

## WOFOST parameters

```python
df = pd.read_excel("./data/ScalarParametersOfWofost-Potential.xlsx")
display(df)
```

      Name
      Typical Value
      Category
      Description
      Unit

      0
      CFET
      1.000
      Water use
      correction factor for evapotranspiration
      -

      1
      CVL
      0.720
      Conversion of assimilates into biomass
      efficiency conversion of assimilates into leaf...
      kg / kg

      2
      CVO
      0.820
      Conversion of assimilates into biomass
      efficiency conversion of assimilates into stor...
      kg / kg

      3
      CVR
      0.720
      Conversion of assimilates into biomass
      efficiency conversion of assimilates into root...
      kg / kg

      4
      CVS
      0.690
      Conversion of assimilates into biomass
      efficiency conversion of assimilates into stem...
      kg / kg

      5
      DLC
      16.000
      Phenology
      critical day length for development (lower thr...
      hours

      6
      DLO
      8.000
      Phenology
      optimum day length for development
      hours

      7
      PERDL
      0.030
      Death rates
      max. relative death rate of leaves due to wate...
      -

      8
      Q10
      2.000
      Maintenance respiration
      increase of the respiration rate per 10 °C tem...
      -

      9
      RDI
      10.000
      Water use
      initial rooting depth
      cm

      10
      RDMCR
      120.000
      Water use
      crop-dependent maximum rooting depth
      cm

      11
      RGRLAI
      0.016
      Initial state
      max. relative increase in leaf area index
      day-1

      12
      RML
      0.030
      Maintenance respiration
      maintenance respiration rate coefficient of le...
      day-1

      13
      RMO
      0.003
      Maintenance respiration
      maintenance respiration rate coefficient of st...
      day-1

      14
      RMR
      0.015
      Maintenance respiration
      maintenance respiration rate coefficient of ro...
      day-1

      15
      RMS
      0.015
      Maintenance respiration
      maintenance respiration rate coefficient of st...
      day-1

      16
      SPA
      0.000
      Crop green area
      specific pod area
      ha / kg

      17
      SPAN
      37.000
      Crop green area
      life span of leaves growing at 35 Celsius
      day

      18
      TBASE
      3.000
      Crop green area
      lower threshold temperature for physiological ...
      ˚C

      19
      TBASEM
      3.000
      Crop green area
      lower threshold temperature for emergence
      ˚C

      20
      TDWI
      0.510
      Initial state
      initial total crop dry weight
      kg / ha

      21
      TEFFMX
      20.000
      Emergence
      max. effective temperature for emergence
      ˚C

      22
      TSUM1
      800.000
      Phenological development
      temperature sum from emergence to anthesis
      ˚C day

      23
      TSUM2
      1400.000
      Phenological development
      temperature sum from anthesis to maturity
      ˚C day

      24
      TSUMEM
      90.000
      Emergence
      temperature sum from sowing to emergence
      ˚C day

      25
      VERNBASE
      14.000
      Phenological development
      base vernalization requirement
      day

      26
      VERNDVS
      0.300
      Phenological development
      critical DVS for vernalization to switch off
      -

      27
      VERNSAT
      70.000
      Phenological development
      saturated vernalization requirement
      day

```python
# !pip install SALib
```

    Collecting SALib
      Downloading https://files.pythonhosted.org/packages/f7/33/cee4d64f7c40f33c08cf5ef5c9b1fb5e51f194b5deceefb5567112800b70/SALib-1.3.11.tar.gz (856kB)
         |████████████████████████████████| 860kB 3.3MB/s
    Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from SALib) (1.18.5)
    Requirement already satisfied: scipy in /usr/local/lib/python3.6/dist-packages (from SALib) (1.4.1)
    Requirement already satisfied: matplotlib in /usr/local/lib/python3.6/dist-packages (from SALib) (3.2.2)
    Requirement already satisfied: pandas in /usr/local/lib/python3.6/dist-packages (from SALib) (1.0.5)
    Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->SALib) (2.8.1)
    Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->SALib) (2.4.7)
    Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib->SALib) (0.10.0)
    Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->SALib) (1.2.0)
    Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas->SALib) (2018.9)
    Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.6/dist-packages (from python-dateutil>=2.1->matplotlib->SALib) (1.15.0)
    Building wheels for collected packages: SALib
      Building wheel for SALib (setup.py) ... done
      Created wheel for SALib: filename=SALib-1.3.11-py2.py3-none-any.whl size=729665 sha256=93e720543725a20b1f31d6b4dcd5681574a59bb5c49e364a65b935ee9a92a0b0
      Stored in directory: /root/.cache/pip/wheels/62/ed/f9/a0b98754ffb2191b98324b96cbbeb1bd5d9598b39ab996b429
    Successfully built SALib
    Installing collected packages: SALib
    Successfully installed SALib-1.3.11

## Sobol’ Sequences versus Random numbers and regular grid

```python
from SALib.sample import saltelli
from SALib.analyze import sobol
from SALib.test_functions import Ishigami
import numpy as np
```

__Docs [SALib](https://salib.readthedocs.io/en/latest/#)__

In this example, we will perform a Sobol’ sensitivity analysis of the _Ishigami_ function, shown below. The _Ishigami_ function is commonly used to test uncertainty and sensitivity analysis methods because it exhibits strong nonlinearity and nonmonotonicity.

$f(x)=\sin \left(x_{1}\right)+ \text{a}\, \operatorname{sin}^{2}\left(x_{2}\right)+ \text{b}\,  x_{3}^{4} \sin \left(x_{1}\right)$

```python
problem = {
  'num_vars': 3,
  'names': ['x1', 'x2', 'x3'],
  'bounds': [[-np.pi, np.pi]]*3
}
```

```python
# Generate samples
param_values = saltelli.sample(problem, 1000, calc_second_order=True)
param_values.shape
```

    (8000, 3)

Here, `param_values` is a NumPy matrix. If we run `param_values.shape`, we see that the matrix is **8000 by 3**. The Saltelli sampler generated 8000 samples. The Saltelli sampler generates $N∗(2D+2)$ samples, where in this example $N$ is 1000 (the argument we supplied) and $D$ is 3 (the number of model inputs). The keyword argument `calc_second_order=False` will exclude second-order indices, resulting in a smaller sample matrix with $N∗(D+2)$ rows instead.

```python
# Run model (example)
Y = Ishigami.evaluate(param_values)

# Perform analysis
Si = sobol.analyze(problem, Y, print_to_console=True)
# Returns a dictionary with keys 'S1', 'S1_conf', 'ST', and 'ST_conf'
# (first and total-order indices with bootstrap confidence intervals)
T_Si, first_Si, (idx, second_Si) = sobol.Si_to_pandas_dict(Si)
```

    Parameter S1 S1_conf ST ST_conf
    x1 0.307975 0.060295 0.560137 0.096384
    x2 0.447767 0.050511 0.438722 0.039817
    x3 -0.004255 0.057172 0.242845 0.024133

    Parameter_1 Parameter_2 S2 S2_conf
    x1 x2 0.012205 0.086730
    x1 x3 0.251526 0.085752
    x2 x3 -0.009954 0.065483

```python
Si_dict = dict(Si)
sensitivity_indices = ['S1', 'ST', 'S1_conf', 'ST_conf']
Si_df = pd.DataFrame()
for SI in sensitivity_indices:
    Si_df = Si_df.append(pd.Series(Si_dict[SI]), ignore_index=True)

Si_df = Si_df.T
Si_df.columns = sensitivity_indices
Si_df
```

      S1
      ST
      S1_conf
      ST_conf

      0
      0.307975
      0.560137
      0.060295
      0.096384

      1
      0.447767
      0.438722
      0.050511
      0.039817

      2
      -0.004255
      0.242845
      0.057172
      0.024133

```python
import seaborn as sns
```

```python
Si_df.bar
```

Consider the model output as
\begin{eqnarray*}
Y=f(X)=f\left(X_{1}, \ldots, X_{p}\right),
\end{eqnarray*}
where $f$ in our case part of agro-model simulator, $X$ are $p$ varied input parameters and $Y$ is the predicted output. Following the techniques by Sobol we represent the multi-variate random function $f$ using Hoeffding decomposition:
\begin{equation}
f(X_1,\dots,X_p) = f_0 + \sum_i^p f_i + \sum_i^p\sum_{j>i}^p f_{ij} + \dots + f_{1\dots p},
\end{equation}
where $f_0$ is a constant term, $f_i = f_i(X_i)$ denotes main effects, $f_{ij} = f_{ij}(X_i, X_j)$ and others describe higher-order interactions. These terms can be written as
\begin{equation*}
\begin{split}
f_0 &= E(Y),\\
f_i &= E_{X_{\sim i}}(Y | X_i) - E(Y),\\
f_{ij} &= E_{X_{\sim ij}}(Y | X_i, X_j) - f_i - f_j - f_0,\\
\dots
\end{split}
\end{equation*}
where $E$ is mathematical expectation and $X_{\sim i}$ denotes all parameters except $i^\text{th}$. Under the assumption that the input parameters are independent, total variance $V(Y)$ of the crop yield can be decomposed as follows:
\begin{equation*}
V(Y) = \sum_i^p V_i + \sum_i^p\sum_{j>i}^p V_{ij} + \dots + V_{12\dots p},
\end{equation*}
where partial variances are
\begin{equation*}
\begin{split}
V_i &= V[f_i(X_i)] = V_{X_i}\left[E_{X_{\sim i}}(Y | X_i)\right],\\
V_{ij} &= V[f_{ij}(X_i,X_j)] = V_{X_iX_j}\left[E_{X_{\sim ij}}(Y | X_i, X_j)\right] - V_i - V_j,\\
\dots
\end{split}
\end{equation*}

## Sobol index (first order, second order, total index)

This way, sensitivity indices (SI) can be introduced as
\begin{equation}
\Large
S_i = \frac{V_i}{V(Y)},~S_{ij} = \frac{V_{ij}}{V(Y)},~\dots
\end{equation}
In order to incorporate all of the interactions for a particular parameter, one can compute the total effect index:
\begin{equation}
S_{T_i} = \frac{E_{X_{\sim i}}\left[V_{X_i}(Y|X_{\sim i})\right]}{V(Y)} = 1 - \frac{V_{X_{\sim i}}\left[E_{X_i}(Y | X_{\sim i})\right]}{V(Y)}
\end{equation}

From this assumption we can conclude:
\begin{equation}
\Large
0 \leq S_i \leq S_{T_i} \leq 1
\end{equation}

More -
* [Wiki](https://en.wikipedia.org/wiki/Sobol_sequence)
* [Habr](https://habr.com/ru/post/440892/)
* Feature selection [Skoltech ML 2020](https://github.com/adasegroup/ML2020_lectures/blob/master/lecture9/Lecture_9_Model_Feature_Selection_Sensitivity.pdf)

# Sensitivity analysis of WOFOST model

## Model input info

```python
# Define location, crop and season
latitude = 52.0
longitude = 5.0
crop_name = 'sugarbeet'
variety_name = 'Sugarbeet_601'
campaign_start_date = '2006-01-01'
emergence_date = "2006-03-31"
harvest_date = "2006-10-20"
max_duration = 300
```

```python
# Here we define the agromanagement for sugar beet
agro_yaml = """
- {start}:
    CropCalendar:
        crop_name: {cname}
        variety_name: {vname}
        crop_start_date: {startdate}
        crop_start_type: emergence
        crop_end_date: {enddate}
        crop_end_type: harvest
        max_duration: {maxdur}
    TimedEvents: null
    StateEvents: null
""".format(cname=crop_name, vname=variety_name,
           start=campaign_start_date, startdate=emergence_date,
           enddate=harvest_date, maxdur=max_duration)
agro = yaml.safe_load(agro_yaml)
print(agro_yaml)
```

```python
# Weather data for Netherlands
wdp = NASAPowerWeatherDataProvider(latitude=latitude, longitude=longitude)

# Parameter sets for crop, soil and site
# Standard crop parameter library
cropd = YAMLCropDataProvider()
# We don't need soil for potential production, so we use dummy values
soild = DummySoilDataProvider()
# Some site parameters
sited = WOFOST71SiteDataProvider(WAV=50, CO2=360.)

# Retrieve all parameters in the form of a single object.
# In order to see all parameters for the selected crop already, we
# synchronise data provider cropd with the crop/variety:
firstkey = list(agro[0])[0]
cropcalendar = agro[0][firstkey]['CropCalendar']
cropd.set_active_crop(cropcalendar['crop_name'], cropcalendar['variety_name'])
params = ParameterProvider(cropdata=cropd, sitedata=sited, soildata=soild)
```

```python
# Define the target variable
target_variable = "TWSO"

# For each scalar parameter, determine a sensible interval
problem_yaml = """
    num_vars: 5
    names:
    - TSUM1
    - TSUM2
    - SPAN
    - Q10
    - TDWI
    bounds:
    - [500, 800]
    - [1200, 1600]
    - [28, 37]
    - [1.8, 2.2]
    - [0.4, 0.6]
"""
problem = yaml.safe_load(problem_yaml)

Model_parameters = {
    'num_vars':5,
    'names':['TSUM1','TSUM2','SPAN','Q10','TDWI'],
    'bounds':[[500, 800],
            [1200, 1600],
            [28, 37],
            [1.8, 2.2],
            [0.4, 0.6]]}
```

## Import SALib

```python
from SALib.sample import saltelli
from SALib.analyze import sobol
from SALib.test_functions import Ishigami
import numpy as np
import pandas as pd
```

## Parameters

```python
Model_parameters = {
    'num_vars':3,
    'names':['N_kg', 'P_kg', 'K_kg'],
    'bounds':[[30., 60.],
              [60., 90.],
              [100., 130.]]
}
```

## Generate input parameters

```python
param_values = saltelli.sample(Soil_parameters, 10)
```

$n = N \times (D \times 2 +2)$

```python
param_values.shape
```

    (120, 5)

## Loop for yield prediction

```python
from pcse.fileinput import YAMLAgroManagementReader
agromanagement_file = os.path.join(data_dir, 'agro', './sugarbeet_calendar.agro')
agromanagement = YAMLAgroManagementReader(agromanagement_file)
#print(agromanagement)
Soil_parameters = {
  'num_vars':5,
    'names':['SMV', 'SMFCF', 'SM0', 'CRAIRC', 'K0'],
    'bounds':[[0.7, 1.3],
              [0.1, 0.5],
              [0.2, 0.6],
             [0.04, 0.08],
             [22.5, 27.5]]}
param_values = saltelli.sample(Soil_parameters, N=10, calc_second_order=True)
```

Soil parameters in [PCSE model](https://pcse.readthedocs.io/en/stable/code.html?highlight=K0#pcse.soil.WaterbalanceFD)

```python
def sensitivity_soil(soil_parameters):
    SMV, SMFCF, SM0, CRAIRC, K0 = soil_parameters
    soildata['SMV'] = SMV
    soildata['SMFCF'] = SMFCF
    soildata['SM0'] = SM0
    soildata['CRAIRC'] = CRAIRC
    soildata['K0'] = K0
    parameters = ParameterProvider(cropdata=cropdata, soildata=soildata, sitedata=sitedata)
    wofsim = Wofost71_WLP_FD(parameters, wdp, agromanagement)
    wofsim.run_till_terminate()
    #df_results = pd.DataFrame(wofsim.get_output())
    #df_results = df_results.set_index("day")
    #df_results.tail()
    summary_output = wofsim.get_summary_output()
    yield_list.append(summary_output[0]['TWSO'])
```

```python
np_yield = np.array(yield_list)
Si = sobol.analyze(Soil_parameters, np_yield, print_to_console=False)
```

```python
Si_dict = dict(Si)
Si_df = pd.DataFrame()
Si_df = Si_df.append(pd.Series(Si_dict['S1']), ignore_index=True)
Si_df = Si_df.append(pd.Series(Si_dict['ST']), ignore_index=True)
Si_df = Si_df.append(pd.Series(Si_dict['S1_conf']), ignore_index=True)
Si_df = Si_df.append(pd.Series(Si_dict['ST_conf']), ignore_index=True)
Si_df = Si_df.T
Si_df.columns = ['Si', 'ST', 'Si_conf', 'ST_conf']
Si_df.rename(index={0:'SMV',1:'SMFCF', 2:'SM0', 3:'CRAIRC', 4:'K0'}, inplace=True)
Si_df
```

      Si
      ST
      Si_conf
      ST_conf

      SMV
      0.000000
      0.000000
      0.000000
      0.000000

      SMFCF
      -0.077417
      0.284878
      0.839456
      0.638290

      SM0
      0.267504
      0.381437
      0.693409
      0.688329

      CRAIRC
      0.000000
      0.000000
      0.000000
      0.000000

      K0
      0.000000
      0.000000
      0.000000
      0.000000

Is it ok?

```python
%%time
yield_list = []
param_values = saltelli.sample(Soil_parameters, 10, calc_second_order=True)
for step in range(len(param_values)):
    sensitivity_soil(param_values[step])
    print(param_values[step])
```

    [ 0.83183594  0.13867188  0.40742187  0.06707031 23.90136719]
    [ 1.24433594  0.13867188  0.40742187  0.06707031 23.90136719]
    [ 0.83183594  0.11835938  0.40742187  0.06707031 23.90136719]
    [ 0.83183594  0.13867188  0.55976563  0.06707031 23.90136719]
    [ 0.83183594  0.13867188  0.40742187  0.06003906 23.90136719]
    [ 0.83183594  0.13867188  0.40742187  0.06707031 22.84667969]
    [ 0.83183594  0.11835938  0.55976563  0.06003906 22.84667969]
    [ 1.24433594  0.13867188  0.55976563  0.06003906 22.84667969]
    [ 1.24433594  0.11835938  0.40742187  0.06003906 22.84667969]
    [ 1.24433594  0.11835938  0.55976563  0.06707031 22.84667969]
    [ 1.24433594  0.11835938  0.55976563  0.06003906 23.90136719]
    [ 1.24433594  0.11835938  0.55976563  0.06003906 22.84667969]
    [ 1.13183594  0.33867188  0.20742188  0.04707031 26.40136719]
    [ 0.94433594  0.33867188  0.20742188  0.04707031 26.40136719]
    [ 1.13183594  0.31835938  0.20742188  0.04707031 26.40136719]
    [ 1.13183594  0.33867188  0.35976562  0.04707031 26.40136719]
    [ 1.13183594  0.33867188  0.20742188  0.04003906 26.40136719]
    [ 1.13183594  0.33867188  0.20742188  0.04707031 25.34667969]
    [ 1.13183594  0.31835938  0.35976562  0.04003906 25.34667969]
    [ 0.94433594  0.33867188  0.35976562  0.04003906 25.34667969]
    [ 0.94433594  0.31835938  0.20742188  0.04003906 25.34667969]
    [ 0.94433594  0.31835938  0.35976562  0.04707031 25.34667969]
    [ 0.94433594  0.31835938  0.35976562  0.04003906 26.40136719]
    [ 0.94433594  0.31835938  0.35976562  0.04003906 25.34667969]
    [ 1.28183594  0.23867188  0.50742187  0.07707031 25.15136719]
    [ 0.79433594  0.23867188  0.50742187  0.07707031 25.15136719]
    [ 1.28183594  0.21835938  0.50742187  0.07707031 25.15136719]
    [ 1.28183594  0.23867188  0.25976562  0.07707031 25.15136719]
    [ 1.28183594  0.23867188  0.50742187  0.05003906 25.15136719]
    [ 1.28183594  0.23867188  0.50742187  0.07707031 26.59667969]
    [ 1.28183594  0.21835938  0.25976562  0.05003906 26.59667969]
    [ 0.79433594  0.23867188  0.25976562  0.05003906 26.59667969]
    [ 0.79433594  0.21835938  0.50742187  0.05003906 26.59667969]
    [ 0.79433594  0.21835938  0.25976562  0.07707031 26.59667969]
    [ 0.79433594  0.21835938  0.25976562  0.05003906 25.15136719]
    [ 0.79433594  0.21835938  0.25976562  0.05003906 26.59667969]
    [ 0.98183594  0.43867188  0.30742188  0.05707031 22.65136719]
    [ 1.09433594  0.43867188  0.30742188  0.05707031 22.65136719]
    [ 0.98183594  0.41835937  0.30742188  0.05707031 22.65136719]
    [ 0.98183594  0.43867188  0.45976563  0.05707031 22.65136719]
    [ 0.98183594  0.43867188  0.30742188  0.07003906 22.65136719]
    [ 0.98183594  0.43867188  0.30742188  0.05707031 24.09667969]
    [ 0.98183594  0.41835937  0.45976563  0.07003906 24.09667969]
    [ 1.09433594  0.43867188  0.45976563  0.07003906 24.09667969]
    [ 1.09433594  0.41835937  0.30742188  0.07003906 24.09667969]
    [ 1.09433594  0.41835937  0.45976563  0.05707031 24.09667969]
    [ 1.09433594  0.41835937  0.45976563  0.07003906 22.65136719]
    [ 1.09433594  0.41835937  0.45976563  0.07003906 24.09667969]
    [ 0.90683594  0.28867188  0.25742188  0.05207031 23.27636719]
    [ 1.16933594  0.28867188  0.25742188  0.05207031 23.27636719]
    [ 0.90683594  0.26835938  0.25742188  0.05207031 23.27636719]
    [ 0.90683594  0.28867188  0.20976563  0.05207031 23.27636719]
    [ 0.90683594  0.28867188  0.25742188  0.05503906 23.27636719]
    [ 0.90683594  0.28867188  0.25742188  0.05207031 25.97167969]
    [ 0.90683594  0.26835938  0.20976563  0.05503906 25.97167969]
    [ 1.16933594  0.28867188  0.20976563  0.05503906 25.97167969]
    [ 1.16933594  0.26835938  0.25742188  0.05503906 25.97167969]
    [ 1.16933594  0.26835938  0.20976563  0.05207031 25.97167969]
    [ 1.16933594  0.26835938  0.20976563  0.05503906 23.27636719]
    [ 1.16933594  0.26835938  0.20976563  0.05503906 25.97167969]
    [ 1.20683594  0.48867187  0.45742187  0.07207031 25.77636719]
    [ 0.86933594  0.48867187  0.45742187  0.07207031 25.77636719]
    [ 1.20683594  0.46835938  0.45742187  0.07207031 25.77636719]
    [ 1.20683594  0.48867187  0.40976563  0.07207031 25.77636719]
    [ 1.20683594  0.48867187  0.45742187  0.07503906 25.77636719]
    [ 1.20683594  0.48867187  0.45742187  0.07207031 23.47167969]
    [ 1.20683594  0.46835938  0.40976563  0.07503906 23.47167969]
    [ 0.86933594  0.48867187  0.40976563  0.07503906 23.47167969]
    [ 0.86933594  0.46835938  0.45742187  0.07503906 23.47167969]
    [ 0.86933594  0.46835938  0.40976563  0.07207031 23.47167969]
    [ 0.86933594  0.46835938  0.40976563  0.07503906 25.77636719]
    [ 0.86933594  0.46835938  0.40976563  0.07503906 23.47167969]
    [ 1.05683594  0.18867188  0.35742188  0.04207031 27.02636719]
    [ 0.71933594  0.18867188  0.35742188  0.04207031 27.02636719]
    [ 1.05683594  0.16835938  0.35742188  0.04207031 27.02636719]
    [ 1.05683594  0.18867188  0.50976562  0.04207031 27.02636719]
    [ 1.05683594  0.18867188  0.35742188  0.06503906 27.02636719]
    [ 1.05683594  0.18867188  0.35742188  0.04207031 24.72167969]
    [ 1.05683594  0.16835938  0.50976562  0.06503906 24.72167969]
    [ 0.71933594  0.18867188  0.50976562  0.06503906 24.72167969]
    [ 0.71933594  0.16835938  0.35742188  0.06503906 24.72167969]
    [ 0.71933594  0.16835938  0.50976562  0.04207031 24.72167969]
    [ 0.71933594  0.16835938  0.50976562  0.06503906 27.02636719]
    [ 0.71933594  0.16835938  0.50976562  0.06503906 24.72167969]
    [ 0.75683594  0.38867188  0.55742187  0.06207031 24.52636719]
    [ 1.01933594  0.38867188  0.55742187  0.06207031 24.52636719]
    [ 0.75683594  0.36835938  0.55742187  0.06207031 24.52636719]
    [ 0.75683594  0.38867188  0.30976562  0.06207031 24.52636719]
    [ 0.75683594  0.38867188  0.55742187  0.04503906 24.52636719]
    [ 0.75683594  0.38867188  0.55742187  0.06207031 27.22167969]
    [ 0.75683594  0.36835938  0.30976562  0.04503906 27.22167969]
    [ 1.01933594  0.38867188  0.30976562  0.04503906 27.22167969]
    [ 1.01933594  0.36835938  0.55742187  0.04503906 27.22167969]
    [ 1.01933594  0.36835938  0.30976562  0.06207031 27.22167969]
    [ 1.01933594  0.36835938  0.30976562  0.04503906 24.52636719]
    [ 1.01933594  0.36835938  0.30976562  0.04503906 27.22167969]
    [ 0.73808594  0.17617188  0.21992188  0.05832031 25.62011719]
    [ 0.85058594  0.17617188  0.21992188  0.05832031 25.62011719]
    [ 0.73808594  0.48085937  0.21992188  0.05832031 25.62011719]
    [ 0.73808594  0.17617188  0.52226562  0.05832031 25.62011719]
    [ 0.73808594  0.17617188  0.21992188  0.04128906 25.62011719]
    [ 0.73808594  0.17617188  0.21992188  0.05832031 24.56542969]
    [ 0.73808594  0.48085937  0.52226562  0.04128906 24.56542969]
    [ 0.85058594  0.17617188  0.52226562  0.04128906 24.56542969]
    [ 0.85058594  0.48085937  0.21992188  0.04128906 24.56542969]
    [ 0.85058594  0.48085937  0.52226562  0.05832031 24.56542969]
    [ 0.85058594  0.48085937  0.52226562  0.04128906 25.62011719]
    [ 0.85058594  0.48085937  0.52226562  0.04128906 24.56542969]
    [ 1.03808594  0.37617188  0.41992188  0.07832031 23.12011719]
    [ 1.15058594  0.37617188  0.41992188  0.07832031 23.12011719]
    [ 1.03808594  0.28085938  0.41992188  0.07832031 23.12011719]
    [ 1.03808594  0.37617188  0.32226562  0.07832031 23.12011719]
    [ 1.03808594  0.37617188  0.41992188  0.06128906 23.12011719]
    [ 1.03808594  0.37617188  0.41992188  0.07832031 27.06542969]
    [ 1.03808594  0.28085938  0.32226562  0.06128906 27.06542969]
    [ 1.15058594  0.37617188  0.32226562  0.06128906 27.06542969]
    [ 1.15058594  0.28085938  0.41992188  0.06128906 27.06542969]
    [ 1.15058594  0.28085938  0.32226562  0.07832031 27.06542969]
    [ 1.15058594  0.28085938  0.32226562  0.06128906 23.12011719]
    [ 1.15058594  0.28085938  0.32226562  0.06128906 27.06542969]
    CPU times: user 1min 21s, sys: 229 ms, total: 1min 21s
    Wall time: 1min 21s

\begin{equation}
\Large
0 \leq S_i \leq S_{T_i} \leq 1
\end{equation}

### For 5 years

## Visualizing simulation results

# Visualization for sensitivity analysis

Plots by [Water programming group](https://waterprogramming.wordpress.com/2019/08/27/a-python-implementation-of-grouped-radial-convergence-plots-to-visualize-sobol-sensitivity-analysis-results/)

How to repeat: [Repo of SampleVIS](https://github.com/charlesrouge/SampleVis)

```python
import numpy as np
from SALib.analyze import sobol
from SALib.sample import saltelli
from fishery import fish_game
import matplotlib.pyplot as plt
import itertools
import math
```

### Why number of samples is important?

```python
# Set up dictionary with system parameters
problem = {
  'num_vars': 6,
  'names': ['a', 'b', 'c','h',
            'K','m'],
  'bounds': [[ 0.002, 2],
             [0.005, 1],
             [0.2, 1],
             [0.001, 1],
             [100, 5000],
             [0.1, 1.5]]
}

# Array with n's to use
nsamples = np.arange(50, 500, 50)

# Arrays to store the index estimates
S1_estimates = np.zeros([problem['num_vars'],len(nsamples)])
ST_estimates = np.zeros([problem['num_vars'],len(nsamples)])

# Loop through all n values, create sample, evaluate model and estimate S1 & ST
for i in range(len(nsamples)):
    print('n= '+ str(nsamples[i]))
    # Generate samples
    sampleset = saltelli.sample(problem, nsamples[i],calc_second_order=False)
    # Run model for all samples
    output = [fish_game(*sampleset[j,:]) for j in range(len(sampleset))]
    # Perform analysis
    results = sobol.analyze(problem, np.asarray(output), calc_second_order=False,print_to_console=False)
    # Store estimates
    ST_estimates[:,i]=results['ST']
    S1_estimates[:,i]=results['S1']

np.save('ST_estimates.npy', ST_estimates)
np.save('S1_estimates.npy', S1_estimates)

S1_estimates = np.load('S1_estimates.npy')
ST_estimates = np.load('ST_estimates.npy')

# Generate figure showing evolution of indices
fig =  plt.figure(figsize=(18,9))
ax1 = fig.add_subplot(1,2,1)
handles = []
for j in range(problem['num_vars']):
    handles += ax1.plot(nsamples, S1_estimates[j,:], linewidth=5)
ax1.set_title('Evolution of S1 index estimates', fontsize=20)
ax1.set_ylabel('S1', fontsize=18)
ax1.set_xlabel('Number of samples (n)', fontsize=18)
ax1.tick_params(axis='both', which='major', labelsize=14)
ax2 = fig.add_subplot(1,2,2)
for j in range(problem['num_vars']):
    ax2.plot(nsamples, ST_estimates[j,:], linewidth=5)
ax2.set_title('Evolution of ST index estimates', fontsize=20)
ax2.set_ylabel('ST', fontsize=18)
ax2.tick_params(axis='both', which='major', labelsize=14)
ax2.set_xlabel('Number of samples (n)', fontsize=18)
fig.legend(handles, problem['names'], loc = 'right', fontsize=11)
plt.show()
#plt.savefig('indexevolution.png')

# Calculate parameter rankings
S1_ranks = np.zeros_like(S1_estimates)
ST_ranks = np.zeros_like(ST_estimates)
for i in range(len(nsamples)):
    orderS1 = np.argsort(S1_estimates[:,i])
    orderST = np.argsort(ST_estimates[:,i])
    S1_ranks[:,i] = orderS1.argsort()
    ST_ranks[:,i] = orderST.argsort()

# Generate figure showing evolution of ranks
fig =  plt.figure(figsize=(18,9))
ax1 = fig.add_subplot(1,2,1)
handles = []
for j in range(problem['num_vars']):
    handles += ax1.plot(nsamples, S1_ranks[j,:], linewidth=3)
ax1.set_title('Parameter ranking based on S1', fontsize=20)
ax1.set_ylabel('S1', fontsize=18)
ax1.set_xlabel('Number of samples (n)', fontsize=18)
ax1.set_yticklabels(np.arange(problem['num_vars']+1, 0, -1))
ax1.tick_params(axis='both', which='major', labelsize=14)
ax2 = fig.add_subplot(1,2,2)
for j in range(problem['num_vars']):
    ax2.plot(nsamples, ST_ranks[j,:], linewidth=3)
ax2.set_title('Parameter ranking based on ST', fontsize=20)
ax2.set_ylabel('ST', fontsize=18)
ax2.set_yticklabels(np.arange(problem['num_vars']+1, 0, -1))
ax2.tick_params(axis='both', which='major', labelsize=14)
ax2.set_xlabel('Number of samples (n)', fontsize=18)
fig.legend(handles, problem['names'], loc = 'right', fontsize=14)
#plt.show()
#plt.savefig('rankingevolution.png')
```

## Radial plot for SA

```python
import numpy as np
import itertools
import matplotlib.pyplot as plt
import seaborn as sns
import math
from numpy import genfromtxt
import matplotlib.patches as mpatches
import matplotlib.pyplot as plt
sns.set_style('whitegrid', {'axes_linewidth': 0, 'axes.edgecolor': 'white'})
```

## Plot function

```python
def is_significant(value, confidence_interval, threshold="conf"):
    if threshold == "conf":
        return value - abs(confidence_interval) > 0
    else:
        return value - abs(float(threshold)) > 0

def grouped_radial(SAresults, parameters, radSc=2.0, scaling=1, widthSc=0.5, STthick=1, varNameMult=1.3, colors=None, groups=None, gpNameMult=1.5, threshold="conf"):
    # Derived from https://github.com/calvinwhealton/SensitivityAnalysisPlots
    fig, ax = plt.subplots(1, 1)
    color_map = {}

    # initialize parameters and colors
    if groups is None:

        if colors is None:
            colors = ["k"]

        for i, parameter in enumerate(parameters):
            color_map[parameter] = colors[i % len(colors)]
    else:
        if colors is None:
            colors = sns.color_palette("deep", max(3, len(groups)))

        for i, key in enumerate(groups.keys()):
            #parameters.extend(groups[key])

            for parameter in groups[key]:
                color_map[parameter] = colors[i % len(colors)]

    n = len(parameters)
    angles = radSc*math.pi*np.arange(0, n)/n
    x = radSc*np.cos(angles)
    y = radSc*np.sin(angles)

    # plot second-order indices
    for i, j in itertools.combinations(range(n), 2):
        #key1 = parameters[i]
        #key2 = parameters[j]

        if is_significant(SAresults["S2"][i][j], SAresults["S2_conf"][i][j], threshold):
            angle = math.atan((y[j]-y[i])/(x[j]-x[i]))

            if y[j]-y[i] < 0:
                angle += math.pi

            line_hw = scaling*(max(0, SAresults["S2"][i][j])**widthSc)/2

            coords = np.empty((4, 2))
            coords[0, 0] = x[i] - line_hw*math.sin(angle)
            coords[1, 0] = x[i] + line_hw*math.sin(angle)
            coords[2, 0] = x[j] + line_hw*math.sin(angle)
            coords[3, 0] = x[j] - line_hw*math.sin(angle)
            coords[0, 1] = y[i] + line_hw*math.cos(angle)
            coords[1, 1] = y[i] - line_hw*math.cos(angle)
            coords[2, 1] = y[j] - line_hw*math.cos(angle)
            coords[3, 1] = y[j] + line_hw*math.cos(angle)

            ax.add_artist(plt.Polygon(coords, color="0.75"))

    # plot total order indices
    for i, key in enumerate(parameters):
        if is_significant(SAresults["ST"][i], SAresults["ST_conf"][i], threshold):
            ax.add_artist(plt.Circle((x[i], y[i]), scaling*(SAresults["ST"][i]**widthSc)/2, color='w'))
            ax.add_artist(plt.Circle((x[i], y[i]), scaling*(SAresults["ST"][i]**widthSc)/2, lw=STthick, color='0.4', fill=False))

    # plot first-order indices
    for i, key in enumerate(parameters):
        if is_significant(SAresults["S1"][i], SAresults["S1_conf"][i], threshold):
            ax.add_artist(plt.Circle((x[i], y[i]), scaling*(SAresults["S1"][i]**widthSc)/2, color='0.4'))

    # add labels
    for i, key in enumerate(parameters):
        ax.text(varNameMult*x[i], varNameMult*y[i], key, ha='center', va='center',
                rotation=angles[i]*360/(2*math.pi) - 90,
                color=color_map[key])

    if groups is not None:
        for i, group in enumerate(groups.keys()):
            print(group)
            group_angle = np.mean([angles[j] for j in range(n) if parameters[j] in groups[group]])

            ax.text(gpNameMult*radSc*math.cos(group_angle), gpNameMult*radSc*math.sin(group_angle), group, ha='center', va='center',
                rotation=group_angle*360/(2*math.pi) - 90,
                color=colors[i % len(colors)])

    ax.set_facecolor('white')
    ax.set_xticks([])
    ax.set_yticks([])
#     ax.
    plt.axis('equal')
    plt.axis([-2*radSc, 2*radSc, -2*radSc, 2*radSc])
    #plt.show()

    return fig
```

## Range of soil parameters

```python
problem = {
    'num_vars':6,
    'names':['SOC', 'Sand', 'Clay', 'pH', 'CN', 'BD'],
    'bounds':[[2.58, 6.20],
              [0.01, 0.30],
              [0.01, 0.30],
              [4.6, 6.9],
              [10.9, 12.4],
              [900, 1350]]
}
```

```python
#names for csv files
list_of_csv=['soybean-000-2015.csv', 'sugar-beet-2011.csv', 'sugar-beet-2017.csv',
'spring-barley-2012.csv', 'sugar-beet-2014.csv']
list_of_names=['soybean-000-2015', 'sugar-beat-2011', 'sugar-beat-2017',
'spring-barley-2012', 'sugar-beat-2014']
list_of_totals=['total_SI_'+x for x in list_of_names]
list_of_first=['fisrt_SI_'+x for x in list_of_names]
list_of_second=['second_SI_'+x for x in list_of_names]
list_of_SI=['SI_'+x for x in list_of_names]
```

```python
for j, i in enumerate(list_of_csv):
        all_data_csv = genfromtxt('./'+str(i), delimiter=',')
        output = all_data_csv[:,2]
        print(i)
        list_of_SI[j] = sobol.analyze(problem, output, calc_second_order=True, conf_level=0.95, print_to_console=False)
```

    soybean-000-2015.csv
    sugar-beet-2011.csv
    sugar-beet-2017.csv
    spring-barley-2012.csv
    sugar-beet-2014.csv

```python
groups={"Soil physics" : ["Sand", "Clay", "BD"],
        "Soil chemistry" : ["pH", "SOC", "CN"]}

fig = grouped_radial(list_of_SI[4], ['BD', 'Sand', 'Clay', 'pH', 'CN', 'SOC'], groups=groups, threshold=0.001)
red_patch = mpatches.Patch(color='red', label='The red data')
plt.title(list_of_names[4], loc='left')

plt.show()
```

## Homework

__[Tasks](https://skoltech-my.sharepoint.com/:w:/g/personal/mikhail_gasanov_skoltech_ru/EeTPQxbrzVdPqnSENKYyoTUBay1RDYgMMW3GO3qFT2ge5g?e=4hk45V)__

Usefull

__SA and UQ__

1) [Rhodium project](https://github.com/Project-Platypus/Rhodium.git)

2) [SALib](https://github.com/SALib/SALib)

__Model__

3) [PCSE](https://pcse.readthedocs.io/en/stable/index.html)

4) How to install PCSE at local machine
    `conda env create -f` [py3_pcse.yml](https://github.com/mishagrol/Seminar_Sobol/blob/master/py3_pcse.yml)

Аny questions  -

Telegram - `@misha_grol`

Part 1 – Crop Yield Prediction (PCSE, MONICA)

You can use the seminars’ colab:

“https://colab.research.google.com/drive/1j4AHD8KkTRThPuNsQzQFWYSJtptQ6bUA”

1) Assess the yield of one of the crops for the Moscow region over several years (potatoes, sugar beets for 2-3 years)

    Crop  - (https://github.com/mishagrol/Seminar_Sobol/tree/master/data/crop)

    Soil - (https://github.com/mishagrol/Seminar_Sobol/tree/master/data/soil)

    Weather  - NASAdataprovider in PCSE (https://pcse.readthedocs.io/en/stable/code.html?highlight=NASA#pcse.db.NASAPowerWeatherDataProvider)

    Agromanagement - (https://github.com/mishagrol/Seminar_Sobol/blob/master/data/agro/sugarbeet_calendar_Moscow_short.agro)

Part 2 – Sensitivity Analysis (SALib)

1) Perform sensitivity analysis of one of the model blocks (crop, soil, agromanagement *) with SALib. You can choose one of the methods that you consider necessary (Sobol, FAST, ...).

    Generate samples – In report provide the size of the resulting matrix and the sample size (N)

    Conduct parameter sensitivity analysis  - In report provide S1 and ST indices.

2) Generate plots (Hist, Radial convergence plot, etc.)

*3) Estimate the required number of simulations to obtain reliable values of the sensitivity indices. Try to estimate the sample size at the confidence interval of the sensitivity indices.

* Please note that working with discrete data can cause certain difficulties.

#Agro Hack https://agro-code.ru/

### bonus

__Morris method__

Generate a sample using the Method of Morris

Three variants of Morris' sampling for elementary effects is supported:

- Vanilla Morris
- Optimised trajectories when ``optimal_trajectories=True`` (using
    Campolongo's enhancements from 2007 and optionally Ruano's enhancement
    from 2012; ``local_optimization=True``)
- Groups with optimised trajectories when ``optimal_trajectories=True`` and
    the problem definition specifies groups (note that ``local_optimization``
    must be ``False``)

At present, optimised trajectories is implemented using either a brute-force
approach, which can be very slow, especially if you require more than four
trajectories, or a local method based which is much faster. Both methods now
implement working with groups of factors.

Note that the number of factors makes little difference,
but the ratio between number of optimal trajectories and the sample size
results in an exponentially increasing number of scores that must be
computed to find the optimal combination of trajectories.  We suggest going
no higher than 4 from a pool of 100 samples with the brute force approach.
With local_optimization = True (which is default),
it is possible to go higher than the previously suggested 4 from 100.

```python
import sys

from SALib.analyze import morris
from SALib.sample.morris import sample
from SALib.test_functions import Sobol_G
from SALib.util import read_param_file
from SALib.plotting.morris import horizontal_bar_plot, covariance_plot, \
    sample_histograms
import matplotlib.pyplot as plt

#sys.path.append('../..')

# Read the parameter range file and generate samples
#problem = read_param_file('/Users/mikhailgasanov/Documents/GIT/SALib/src/SALib/test_functions/params/Sobol_G.txt')
# or define manually without a parameter file:
problem = {
 'num_vars': 8,
 'names': ['x1', 'x2', 'x3', 'x4', 'x5', 'x6', 'x7', 'x8'],
 'groups': None,
 'bounds': [[0.0, 1.0],
            [0.0, 1.0],
            [0.0, 1.0],
            [0.0, 1.0],
            [0.0, 1.0],
            [0.0, 1.0],
            [0.0, 1.0],
            [0.0, 1.0]]
}
# Files with a 4th column for "group name" will be detected automatically, e.g.
# param_file = '../../src/SALib/test_functions/params/Ishigami_groups.txt'

# Generate samples
param_values = sample(problem, N=1000, num_levels=4,
                      optimal_trajectories=None)

# To use optimized trajectories (brute force method),
# give an integer value for optimal_trajectories

# Run the "model" -- this will happen offline for external models
Y = Sobol_G.evaluate(param_values)

# Perform the sensitivity analysis using the model output
# Specify which column of the output file to analyze (zero-indexed)
Si = morris.analyze(problem, param_values, Y, conf_level=0.95,
                    print_to_console=True,
                    num_levels=4, num_resamples=100)
# Returns a dictionary with keys 'mu', 'mu_star', 'sigma', and 'mu_star_conf'
# e.g. Si['mu_star'] contains the mu* value for each parameter, in the
# same order as the parameter file

fig, (ax1, ax2) = plt.subplots(1, 2)
horizontal_bar_plot(ax1, Si, {}, sortby='mu_star', unit=r"tCO$_2$/year")
covariance_plot(ax2, Si, {}, unit=r"tCO$_2$/year")

fig2 = plt.figure()
sample_histograms(fig2, param_values, problem, {'color': 'y'})
plt.show()
```

```python

```


# Latent Variable Models and Variational Bayes

### Preliminaries

- Goal
  - Introduction to latent variable models and variational inference by Free energy minimization
- Materials
  - Mandatory
    - These lecture notes
    - Ariel Caticha (2010), [Entropic Inference](https://arxiv.org/abs/1011.0723)
      - tutorial on entropic inference, which is a generalization to Bayes rule and provides a foundation for variational inference.
  - Optional
    - Bishop (2016), pp. 461-486 (sections 10.1, 10.2 and 10.3)
  - references
    - Blei et al. (2017), [Variational Inference: A Review for Statisticians](https://www.tandfonline.com/doi/full/10.1080/01621459.2017.1285773)
    - Lanczos (1961), [The variational principles of mechanics](https://www.amazon.com/Variational-Principles-Mechanics-Dover-Physics/dp/0486650677)

### Illustrative Example

- You're now asked to build a density model for a data set ([Old Faithful](https://en.wikipedia.org/wiki/Old_Faithful), Bishop pg. 681) that clearly is not distributed as a single Gaussian:

###  Unobserved Classes

Consider again a set of observed data $D=\{x_1,\dotsc,x_N\}$

- This time we suspect that there are _unobserved_ class labels that would help explain (or predict) the data, e.g.,
  - the observed data are the color of living things; the unobserved classes are animals and plants.
  - observed are wheel sizes; unobserved categories are trucks and personal cars.
  - observed is an audio signal; unobserved classes include speech, music, traffic noise, etc.

- Classification problems with unobserved classes are called **Clustering** problems. The learning algorithm needs to **discover the classes from the observed data**.

### The Gaussian Mixture Model

- The spread of the data in the illustrative example looks like it could be modeled by two Gaussians. Let's develop a model for this data set.

- Let $D=\{x_n\}$ be a set of observations. We associate a one-hot coded hidden class label $z_n$ with each observation:

$$\begin{equation*}
z_{nk} = \begin{cases} 1 & \text{if } x_n \in \mathcal{C}_k \text{ (the k-th class)}\\
                       0 & \text{otherwise} \end{cases}
\end{equation*}$$

- We consider the same model as we did in the [generative classification lesson](https://nbviewer.jupyter.org/github/bertdv/BMLIP/blob/master/lessons/notebooks/07-Generative-Classification.ipynb#GDA):
$$\begin{align*}
p(x_n | z_{nk}=1) &= \mathcal{N}\left( x_n | \mu_k, \Sigma_k\right)\\
p(z_{nk}=1) &= \pi_k
\end{align*}$$
which can be summarized with the selection variables $z_{nk}$ as
$$\begin{align*}
p(x_n,z_n) &= p(x_n | z_n) p(z_n) = \prod_{k=1}^K (\underbrace{\pi_k \cdot \mathcal{N}\left( x_n | \mu_k, \Sigma_k\right) }_{p(x_n,z_{nk}=1)})^{z_{nk}}
\end{align*}$$

- *Again*, this is the same model  as we defined for the generative classification model: A Gaussian Discriminant Analysis model but now with unobserved classes).

- This model (with unobserved class labels) is known as a **Gaussian Mixture Model** (GMM).

- The marginal distribution for an _observed_ data point $x_n$ is now given by

$$\begin{align*}{}
p(x_n) &= \sum_{z_n} p(x_n,z_n)  \\
  &= \sum_{k=1}^K \pi_k \cdot \mathcal{N}\left( x_n | \mu_k, \Sigma_k \right) \tag{B-9.12}
\end{align*}$$
  - (Eq. B-9.12 reveals the link to the name Gaussian *mixture model*).

###  GMM is a very flexible model

- GMMs are very popular models. They have decent computational properties and are **universal approximators of densities** (as long as there are enough Gaussians of course)

- (In the above figure, the Gaussian components are shown in red and the pdf of the mixture models in blue).

### Latent Variable Models

- The GMM contains both _observed_ variables $\{x_n\}$, (unobserved) _parameters_ $\theta= \{\pi_k,\mu_k, \Sigma_k\}$ _and_ unobserved (synonym: latent, hidden) variables $\{z_n\}$.

- From a Bayesian viewpoint, both latent variables $\{z_{nk}\}$ and parameters $\theta$ are just unobserved variables for which we can set a prior and compute a posterior by Bayes rule.

- Note that $z_{nk}$ has a subscript $n$, hence its value depends on the $n$-th observation (in constrast to parameters). These observation-dependent variables can be useful to encode structure in the model. Here (in the GMM), the latent variables $\{z_{nk}\}$ encode (unobserved) class membership.

- By adding model structure through (equations among) latent variables, we can build very complex models. Unfortunately, inference in latent variable models can also be much more complex than for fully observed models.

### Inference for GMM is Difficult

- We recall here the log-likelihood for the Gaussian Discriminative Analysis model (see [generative classification lesson)](https://nbviewer.jupyter.org/github/bertdv/BMLIP/blob/master/lessons/notebooks/07-Generative-Classification.ipynb)

$$
\log\, p(D|\theta) =  \sum_{n,k} y_{nk} \underbrace{ \log\mathcal{N}(x_n|\mu_k,\Sigma) }_{ \text{Gaussian} } + \underbrace{ \sum_{n,k} y_{nk} \log \pi_k }_{ \text{multinomial} } \,.
$$

- Since the class labels $y_{nk}$ were given, this expression decomposed into a set of simple updates for the Gaussian and multinomial distributions. For both distributions, we have conjugate priors, so inference is easily solved.

- However, for the Gaussian mixture model (same log-likelihood function with $z_{nk}$ replacing $y_{nk}$), the class labels $\{z_{nk}\}$ are _unobserved_ and need to estimated alongside with the parameters.

- There is no conjugate prior for the GMM likelihood function $p(\underbrace{D}_{\{x_n\}}\,|\,\underbrace{\{z_{nk}\},\{\mu_k,\Sigma_k,\pi_k\}}_{\text{latent variables}})$.

- In this lesson, we introduce an approximate Bayesian inference method known as **Variational Bayes** (VB) (also known as **Variational Inference**) that can be used for Bayesian inference in models with latent variables. Later in this lesson, we will use VB to do inference in the GMM.

- As a motivation for variational inference, we first discuss inference by the **Method of Maximum Relative Entropy**, [(Caticha, 2010)](https://arxiv.org/abs/1011.0723).

### Inference When Information is in the Form of Constraints

- In the [probability theory lesson](https://nbviewer.jupyter.org/github/bertdv/BMLIP/blob/master/lessons/notebooks/02-Probability-Theory-Review.ipynb#Bayes-rule), we recognized Bayes rule as the fundamental rule for learning from data.

- We will now generalize this notion and consider learning as a process that updates a prior into a posterior distribution whenever new information becomes available.

- In this context, *new information* is not necessarily a new observation, but could (for instance) also relate to *constraints* on the posterior distribution.

- For example, consider a model $p(x_n,\theta) = p(x_n|\theta)p(\theta)$. At this point, our beliefs about $\theta$ are represented by $p(\theta)$. We might be interested in obtaining rational posterior beliefs $q(\theta)$ in consideration of the following constraints:
  1. Observations: two new data points $x_1=5$ and $x_2=3$.
  2. Constraint: we only consider the Gamma distribution for $q(\theta) = \mathrm{Gam}(\theta|\alpha,\beta)$.
  3. Constraint: the first moment of the posterior is given by $\int \theta q(\theta) \mathrm{d}\theta = 3$.

- Note that this is not "just" applying Bayes rule, which would not be able to handle the constraints.

- Note also that observations _can_ be rephrased as constraints on the posterior, e.g., observation $x_1=5$ can be phrased as a constraint $q(x_1)=\delta(x_1-5)$.
  - $\Rightarrow$ Updating a prior to a posterior on the basis of constraints on the posterior is more general than updating based on observations alone.

- [Caticha (2010)](https://arxiv.org/abs/1011.0723) developed the **method of maximum (relative) entropy** for rational updating of priors to posteriors when faced with new information in the form of constraints.

### The Method of Maximum Relative Entropy

- Consider prior beliefs $p(z)$ about $z$. New information in the form of constraints is obtained and we are interested in the "best update" to a posterior $q(z)$.

- In order to define what "best update" means, we assume a ranking function $S[q,p]$ that generates a preference score for each candidate posterior $q$ for a given $p$. The best update from $p$ to $q$ is identified as $q^* = \arg\max_q S[q,p]$.

- Similarly to [Cox' method to deriving probability theory](https://en.wikipedia.org/wiki/Cox%27s_theorem), Caticha introduced the following mild criteria based on a rational principle (the **principle of minimal updating**, see [Caticha 2010](https://arxiv.org/abs/1011.0723)) that the ranking function needs to adhere to:
  1. *Locality*: local information has local effects.
  2. *Coordinate invariance*: the system of coordinates carries no information.
  3. *Independence*: When systems are known to be independent, it should not matter whether they are treated separately or jointly.

- These three criteria **uniquely identify the Relative Entropy** as the proper ranking function:
$$\begin{align*}
S[q,p] = - \int q(z) \log \frac{q(z)}{p(z)}
\end{align*}$$

- $\Rightarrow$ When information is supplied in the form of constaints on the posterior, we *should* select the posterior that maximizes the Relative Entropy. This is the **Method of Maximum (Relative) Entropy** (MRE).

### Some notes on the MRE method

- Note that the Relative Entropy is technically a *functional*, i.e., a function of function(s).

- **Bayes rule is (of course!) a special case of the method of maximum relative entropy**. [Proof at pg.6 of Caticha (2010)](https://arxiv.org/abs/1011.0723).

- Note the relation of the Maximum Relative Entropy method to Probability Theory, which describes how to _represent_ beliefs about events and how to _calculate_ beliefs about joint and conditional events. In contrast, the MRE method describes how to _update_ beliefs when new information becomes available. PT and the MRE method are both needed to describe the theory of optimal information processing.

- In principle, entropies can always be considered as a relative score against a reference distribution. For instance, the score $-\sum_{z_i} q(z_i) \log q(z_i)$ can be interpreted as a score against the uniform distribution, i.e., $-\sum_{z_i} q(z_i) \log q(z_i) \propto -\sum_{z_i} q(z_i) \log \frac{q(z_i)}{\mathrm{Uniform(z_i)}}$. Therefore, the "method of maximum relative entropy" is often simply known as the "method of maximum entropy".

- The negative relative entropy is known as the **Kullback-Leibler divergence**:
$$
\mathrm{KL}[q,p] \triangleq \sum_z q(z) \log \frac{q(z)}{p(z)} \tag{B-1.113}
$$

- The [Gibbs inequality](https://en.wikipedia.org/wiki/Gibbs%27_inequality) ([proof](https://en.wikipedia.org/wiki/Gibbs%27_inequality#Proof)), is a famous theorem in information theory that states that
$$\mathrm{KL}[q,p] \geq 0 $$
with equality only iff $p=q$.

  - The KL divergence can be interpreted as a "distance" between two probability distributions.
    - Note however that the KL divergence is an asymmetric "distance measure", i.e. in general $\mathrm{KL}[q,p] \neq \mathrm{KL}[p,q]$.

- In this class, we prefer to discuss inference in terms of minimizing KL divergence rather than maximizing relative entropy, but note that these two concepts are equivalent.

### Example KL divergence for Gaussians

source: By Mundhenk at English Wikipedia, CC BY-SA 3.0, Link

### The Free Energy Functional and Variational Bayes

- Let's get back to the issue of doing inference for models with latent variables (such as the GMM).

- Consider a generative model specified by $p(x,z)$ where $x$ and $z$ represent the observed and unobserved variables, respectively.

- Assume further that $x$ has been observed as $x=\hat{x}$ and we are interested in performing Bayesian inference, i.e., we want to compute the posterior $p(z|x=\hat{x})$ for the latent variables and the evidence $p(x=\hat{x})$.

- According to the Method of Maximum Relative Entropy, we want to find the posterior $q(x,z)$ that minimizes

$$
\mathrm{KL}[q,p] = \sum_{x,z} q(x,z) \log \frac{q(x,z)}{p(x,z)} \quad \text{subject to }x=\hat{x}
$$

- The constraint $x=\hat{x}$ fixes $p(x) = \delta(x-\hat{x})$ and $q(x) = \delta(x-\hat{x})$, so the KL-divergence evaluates to
$$\begin{align*}
\mathrm{KL}[q,p] &= \sum_{x,z} q(z|x)q(x) \log \frac{q(z|x) q(x)}{p(z|x) p(x)} \\
  &= \sum_{x,z} q(z|x)\delta(x-\hat{x}) \log \frac{q(z|x)\delta(x-\hat{x})}{p(z|x) \delta(x-\hat{x})} \\
  &= \sum_{z} q(z|x=\hat{x}) \log \frac{q(z|x=\hat{x})}{p(z|x=\hat{x}) }
\end{align*}$$

- Generally, we constrain the posterior $q(z|x=\hat{x})$ to be a parametric function $q(z;\mu(\hat{x}))$, where the parameters $\mu(\hat{x})$ are functions of the observed data. As a result, in the context of a generative model $p(x,z)$, with $x$ observed and $z$ unobserved, the method of MRE selects a posterior $q(z;\mu(\hat{x}))$ that minimizes
$$
\mathrm{KL}[q,p] = \sum_z q(z;\mu(\hat{x})) \log \frac{q(z;\mu(\hat{x}))}{p(z|x=\hat{x})}
$$

- (Continuing, we shortly write $q(z)$ for $q(z;\mu(\hat{x}))$ and $p(z|x)$ for $p(z|x=\hat{x})$.) If we scale the KL divergence by a constant (i.e., constant w.r.t. $z$) the ranking of candidate posteriors $q(z)$ is not affected. Hence, the posterior that we want also minimizes the so-called (variational) **Free Energy** (FE) functional
$$
\mathrm{F}[q] = \underbrace{\sum_z q(z) \log \frac{q(z)}{p(z|x)}}_{\text{KL divergence}\geq 0} - \underbrace{\log p(x)}_{\text{(Bayesian) log-evidence}} \tag{B-10.2}
$$
  - For brevity, we write $F[q]$ rather than $F[q,p]$ since in the current context we are only interested in minimizing $F$ with respect to $q$.
  - (As an aside), note that Bishop introduces in Eq. B-10.2 an _Evidence Lower BOund_ (in modern literature abbreviated as ELBO) $\mathcal{L}(z)$ that equals the _negative_ FE. We prefer to discuss variational inference in terms of a free energy (but it is the same story as he discusses with the ELBO).

- The log-evidence term for the FE does not depend on $q$. Hence, the global minimum $$q^* \triangleq \arg\min_q F[q]$$ is obtained for $q^*(z) = p(z|x)$, which is the Bayesian posterior.

- Furthermore, the minimal free energy $F^* \triangleq F[q^*] = -\log p(x)$ equals minus-log-evidence.

- $\Rightarrow$ It follows that Bayesian inference (computation of posterior and evidence) can be executed by FE minimization.

- Executing inference by minimizing the FE functional is called **Variational Bayes** (VB) or variational inference. Note that VB transforms an inference problem (that involves integration) to an optimization problem! Generally optimization problems are easier to solve than integration problems.

###  Approximate Inference by Free Energy Minimization

  - The FE as specified above depends on the posterior $p(z|x)$ and evidence $p(x)$, both of which are not accessible since they are the objectives of Bayesian inference. Fortunately, we can rewrite the FE in computable terms.

-  Making use of $p(x,z) = p(z|x)p(x) = p(x|z)p(z)$, the FE functional can be rewritten as

$$\begin{align*}
\mathrm{F}[q] &= \underbrace{\sum_z q(z) \log \frac{q(z)}{p(z|x)}}_{\text{KL divergence}\geq 0} - \underbrace{\log p(x)}_{\text{log-evidence}} \tag{DE}\\
&= \underbrace{-\sum_z q(z) \log p(x,z)}_{\text{energy}} - \underbrace{\sum_z q(z) \log \frac{1}{q(z)}}_{\text{entropy}} \tag{EE} \\
&= \underbrace{\sum_z q(z)\log\frac{q(z)}{p(z)}}_{\text{complexity}} - \underbrace{\sum_z q(z) \log p(x|z)}_{\text{accuracy}}  \tag{CA}
\end{align*}$$
   - These decompositions are very insightful (wil revisit them later) and we will label them respectively as _divergence-evidence_ (DE), _energy-entropy_ (EE), and _complexity-accuracy_ (CA) decompositions.

- The CA decomposition makes use of $q(z)$, the prior $p(z)$ and likelihood $p(x|z)$, all of which are selected by the engineer, so the FE can be evaluated with this decomposition!

- Often, we are unable to fully minimize the FE (to the global minimum), but rather get stuck in a local minimum.

- In that case, (by the DE composition and Gibbs inequality) the FE is an _upperbound_ on $-\log p(x)$, the (negative) log-evidence.

- Thus, FE minimization (variational Bayes) leads often to _approximate_ Bayesian inference, with approximate posterior
$$
\hat{q}(z) \approx \arg\min_q F[q] \\
$$
and model performance is assessed by the free energy $F[\hat{q}] \ge -\log p(x)$.

###  How to Solve  the FE Minimization task?

- Note that the FE is a _functional_ , i.e., the FE is a function ($F$) of a function ($q$). We are looking for a _function_ $q^*(z)$ that minimizes the FE.

- The mathematics of minimizing functionals is described by _variational calculus_ , see Bishop (2016), app.D and [Lanczos (1961)](https://www.amazon.com/Variational-Principles-Mechanics-Dover-Physics/dp/0486650677). (Optional reading).

- Generally speaking, it is not possible to solve the variational FE minimization problem without some additional constraints on $q(z)$ (i.e., in addition to the data constraints).

- Note that adding constraints to the functional form of $q(z)$ is fully compliant with the MRE method (and therefore also with the FE minimization framework).

- There are three important cases of adding functional constraints on $q(z)$ that sometimes makes inference possible:

##### 1. mean-field factorization
- We constrain the posterior to factorize into a set of _independent_ factors, i.e.,
$$
q(z) = \prod_{j=1}^m q_j(z_j)\,, \tag{B-10.5}
$$

#####  2. fixed-form parameterization
- We constrain the posterior to be part of a parameterized probability distribution, e.g., $$q(z) = \mathcal{N}\left( z | \mu, \Sigma \right)\,.$$
  - In this case, the functional minimization problem for $F[q]$ reduces to the minimization of a _function_ $F(\mu,\Sigma)$ w.r.t. its parameters.

##### 3. the Expectation-Maximization (EM) algorithm
- We place some constraints both on the prior and posterior for $z$ (to be dicussed below) that simplifies FE minimization to maximum-likelihood estimation.

- Next, we work out examples for the mean-field and EM constraints that makes FE minimization solvable.

### FE Minimization by Mean-field Factorization

- Given the mean-field constraint $q(z) = \prod_{j=1}^m q_j(z_j)$, it is possible to derive an expression for the optimal solutions $q_j^*(z_j)$, for $j=1,\ldots,m$, which is given by
$$\begin{align}
\log q_j^*(z_j) &\propto \mathrm{E}_{q_{-j}^*}\left[ \log p(x,z) \right] \tag{B-10.9}\\
  &=\underbrace{\sum_{z_{-j}} q_{-j}^*(z_{-j}) \underbrace{\log p(x,z)}_{\text{"field"}}}_{\text{"mean field"}} \notag
\end{align}$$
where we defined $q_{-j}^*(z_{-j}) \triangleq q_1^*(z_1)q_2^*(z_2)\cdots q_{j-1}^*(z_{j-1})q_{j+1}^*(z_{j+1})\cdots q_m^*(z_m)$.

  - **Proof** (from [Blei, 2017](https://www.tandfonline.com/doi/full/10.1080/01621459.2017.1285773)): We first rewrite the energy-entropy decomposition of the FE as a function of $q_j(z_j)$ only:
  $$ F[q_j] = \mathbb{E}_{q_{j}}\left[ \mathbb{E}_{q_{-j}}\left[ \log p(x,z_j,z_{-j})\right]\right] - \mathbb{E}_{q_j}\left[ \log q_j(z_j)\right] + \mathtt{const.}\,,$$
  where the constant holds all terms that do not depend on $z_j$. This expression can be written as
  $$ F[q_j] = \sum_{z_j} q_j(z_j) \log \frac{q_j(z_j)}{\exp\left( \mathbb{E}_{q_{-j}}\left[ \log p(x,z_j,z_{-j})\right]\right)}$$
  which is a KL-divergence that is minimized by Eq. B-10.9.

- This is not yet a full solution to the FE minimization task since the solution $q_j^*(z_j)$ depends on expectations that involve $q_{i\neq j}^*(z_{i \neq j})$, and each of the solutions $q_{i\neq j}^*(z_{i \neq j})$ depends on an expection that involves $q_j^*(z_j)$.

- In practice, we solve this chicken-and-egg problem by an iterative approach: we first initialize all $q_j(z_j)$ (for $j=1,\ldots,m$) to an appropriate initial distribution and then cycle through the factors in turn by solving eq.10.9 and update $q_{-j}^*(z_{-j})$ with the latest estimates. (See [Blei, 2017](https://www.tandfonline.com/doi/full/10.1080/01621459.2017.1285773), Algorithm 1, p864).

### Example: FEM for the Gaussian Mixture Model

##### model specification

- We consider a Gaussian Mixture Model, specified by
$$\begin{align*}
p(x,z|\theta) &= p(x|z,\mu,\Lambda)p(z|\pi) \\
  &= \prod_{n=1}^N \prod_{k=1}^K \pi_k^{z_{nk}}\cdot \mathcal{N}\left( x_n | \mu_k, \Lambda_k^{-1}\right)^{z_{nk}} \tag{B-10.37,38}
\end{align*}$$
with tuning parameters $\theta=\{\pi_k, \mu_k,\Lambda_k\}$.

- Let us introduce some priors for the parameters. We factorize the prior and choose conjugate distributions by
$$
p(\theta) = p(\pi,\mu,\Lambda) = p(\pi) p(\mu|\Lambda) p(\Lambda)
$$
with
$$\begin{align*}
p(\pi) &= \mathrm{Dir}(\pi|\alpha_0) = C(\alpha_0) \prod_k \pi_k^{\alpha_0-1} \tag{B-10.39}\\
p(\mu|\Lambda) &= \prod_k \mathcal{N}\left(\mu_k | m_0, \left( \beta_0 \Lambda_k\right)^{-1} \right) \tag{B-10.40}\\
p(\Lambda) &= \prod_k \mathcal{W}\left( \Lambda_k | W_0, \nu_0 \right) \tag{B-10.40}
\end{align*}$$
where $\mathcal{W}\left( \cdot \right)$ is a [Wishart distribution](https://en.wikipedia.org/wiki/Wishart_distribution) (i.e., a multi-dimensional Gamma distribution).

- The full generative model is now specified by
$$
p(x,z,\pi,\mu,\Lambda) = p(x|z,\mu,\Lambda) p(z|\pi) p(\pi) p(\mu|\Lambda) p(\Lambda) \tag{B-10.41}
$$
with hyperparameters $\{ \alpha_0, m_0, \beta_0, W_0, \nu_0\}$.

##### inference

- Assume that we have observed $D = \left\{x_1, x_2, \ldots, x_N\right\}$ and are interested to infer the posterior distribution for the tuning parameters:
$$
p(\theta|D) = p(\pi,\mu,\Lambda|D)
$$

- The (perfect) Bayesian solution is
$$
p(\theta|D) = \frac{p(x=D,\theta)}{p(x=D)} = \frac{\sum_z p(x=D,z,\pi,\mu,\Lambda)}{\sum_z \sum_{\pi} \iint p(x=D,z,\pi,\mu,\Lambda) \,\mathrm{d}\mu\mathrm{d}\Lambda}
$$
but this is intractable (See [Blei (2017), p861, eqs. 8 and 9](https://www.tandfonline.com/doi/full/10.1080/01621459.2017.1285773)).

- Alternatively, we can use **FE minimization with constraint**
$$\begin{equation}
q(\theta) = q(z) \cdot q(\pi,\mu,\Lambda) \tag{B-10.42}
\end{equation}$$
on the posterior. For the specified model, this leads to FE minimization wrt the hyperparameters, i.e., we need to minimize the function $F(\alpha_0, m_0, \beta_0, W_0, \nu_0)$.

- Bishop shows that the equations for the [optimal solutions (Eq. B-10.9)](#optimal-solutions) are analytically solvable for the GMM as specified above, leading to (for $k=1,\ldots,K$):
$$
\begin{align*}
\alpha_k &= \alpha_0 + N_k  \tag{B-10.58} \\
\beta_k &= \beta_0 + N_k  \tag{B-10.60} \\
m_k &= \frac{1}{\beta_k} \left( \beta_0 m_0 + N_k \bar{x}_k \right) \tag{B-10.61} \\
W_k^{-1} &= W_0^{-1} + N_k S_k + \frac{\beta_0 N_k}{\beta_0 + N_k}\left( \bar{x}_k - m_0\right) \left( \bar{x}_k - m_0\right)^T \tag{B-10.62} \\
\nu_k &= \nu_0 + N_k \tag{B-10.63}
\end{align*}
$$
where we used
$$
\begin{align*}
\log \rho_{nk} &= \mathbb{E}\left[ \log \pi_k\right] + \frac{1}{2}\mathbb{E}\left[ \log | \Lambda_k | \right] - \frac{D}{2} \log(2\pi) \\
 & \qquad - \frac{1}{2}\mathbb{E}\left[(x_k - \mu_k)^T \Lambda_k(x_k - \mu_k)  \right]  \tag{B-10.46} \\
r_{nk} &= \frac{\rho_{nk}}{\sum_{j=1}^K \rho_{nj}} \tag{B-10.49} \\
N_k &= \sum_{n=1}^N r_{nk} x_n \tag{B-10.51} \\
\bar{x}_k &= \frac{1}{N_k} \sum_{n=1}^N r_{nk} x_n \tag{B-10.52} \\
S_k &= \frac{1}{N_k} \sum_{n=1}^N r_{nk} \left( x_n - \bar{x}_k\right) \left( x_n - \bar{x}_k\right)^T \tag{B-10.53}
\end{align*}
$$

### Code Example FEM for GMM

- Below we exemplify training of a Gaussian Mixture Model on the Old Faithful data set by Free Energy Minimization, using the constraints as specified above, e.g., [(B-10.42)](#mf-constraint).

```julia
using Pkg;Pkg.activate("probprog/workspace/");Pkg.instantiate()
Pkg.add("PDMats")
Pkg.add("SpecialFunctions")
IJulia.clear_output();
```

```julia
using DataFrames, CSV, LinearAlgebra, PDMats, SpecialFunctions
include("scripts/gmm_plot.jl") # Holds plotting function
old_faithful = CSV.read("datasets/old_faithful.csv");
X = convert(Matrix{Float64}, [old_faithful[!,1] old_faithful[!,2]]');#data matrix
N = size(X, 2) #number of observations
K = 6

function sufficientStatistics(X,r,k::Int) #function to compute sufficient statistics
    N_k = sum(r[k,:])
    hat_x_k = sum([r[k,n]*X[:,n] for n in 1:N]) ./ N_k
    S_k = sum([r[k,n]*(X[:,n]-hat_x_k)*(X[:,n]-hat_x_k)' for n in 1:N]) ./ N_k
    return N_k, hat_x_k, S_k
end

function updateMeanPrecisionPi(m_0,β_0,W_0,ν_0,α_0,r) #variational maximisation function
    m = Array{Float64}(undef,2,K) #mean of the clusters
    β = Array{Float64}(undef,K) #precision scaling for Gausian distribution
    W = Array{Float64}(undef,2,2,K) #precision prior for Wishart distributions
    ν = Array{Float64}(undef,K) #degrees of freedom parameter for Wishart distribution
    α = Array{Float64}(undef,K) #Dirichlet distribution parameter
    for k=1:K
        sst = sufficientStatistics(X,r,k)
        α[k] = α_0[k] + sst[1]; β[k] = β_0[k] + sst[1]; ν[k] = ν_0[k] .+ sst[1]
        m[:,k] = (1/β[k])*(β_0[k].*m_0[:,k] + sst[1].*sst[2])
        W[:,:,k] = inv(inv(W_0[:,:,k])+sst[3]*sst[1] + ((β_0[k]*sst[1])/(β_0[k]+sst[1])).*(sst[2]-m_0[:,k])*(sst[2]-m_0[:,k])')
    end
    return m,β,W,ν,α
end

function updateR(Λ,m,α,ν,β) #variational expectation function
    r = Array{Float64}(undef,K,N) #responsibilities
    hat_π = Array{Float64}(undef,K)
    hat_Λ = Array{Float64}(undef,K)
    for k=1:K
        hat_Λ[k] = 1/2*(2*log(2)+logdet(Λ[:,:,k])+digamma(ν[k]/2)+digamma((ν[k]-1)/2))
        hat_π[k] = exp(digamma(α[k])-digamma(sum(α)))
        for n=1:N
           r[k,n] = hat_π[k]*exp(-hat_Λ[k]-1/β[k] - (ν[k]/2)*(X[:,n]-m[:,k])'*Λ[:,:,k]*(X[:,n]-m[:,k]))
        end
    end
    for n=1:N
        r[:,n] = r[:,n]./ sum(r[:,n]) #normalize to ensure r represents probabilities
    end
    return r
end

max_iter = 15
#store the inference results in these vectors
ν = fill!(Array{Float64}(undef,K,max_iter),3)
β = fill!(Array{Float64}(undef,K,max_iter),1.0)
α = fill!(Array{Float64}(undef,K,max_iter),0.01)
R = Array{Float64}(undef,K,N,max_iter)
M = Array{Float64}(undef,2,K,max_iter)
Λ = Array{Float64}(undef,2,2,K,max_iter)
clusters_vb = Array{Distribution}(undef,K,max_iter) #clusters to be plotted
#initialize prior distribution parameters
M[:,:,1] = [[3.0; 60.0];[4.0; 70.0];[2.0; 50.0];[2.0; 60.0];[3.0; 100.0];[4.0; 80.0]]
for k=1:K
    Λ[:,:,k,1] = [1.0 0;0 0.01]
    R[k,:,1] = 1/(K)*ones(N)
    clusters_vb[k,1] = MvNormal(M[:,k,1],PDMats.PDMat(convert(Matrix,Hermitian(inv(ν[1,1].*Λ[:,:,k,1])))))
end
#variational inference
for i=1:max_iter-1
    #variational expectation
    R[:,:,i+1] = updateR(Λ[:,:,:,i],M[:,:,i],α[:,i],ν[:,i],β[:,i])
    #variational minimisation
    M[:,:,i+1],β[:,i+1],Λ[:,:,:,i+1],ν[:,i+1],α[:,i+1] = updateMeanPrecisionPi(M[:,:,i],β[:,i],Λ[:,:,:,i],ν[:,i],α[:,i],R[:,:,i+1])
    for k=1:K
        clusters_vb[k,i+1] = MvNormal(M[:,k,i+1],PDMats.PDMat(convert(Matrix,Hermitian(inv(ν[k,i+1].*Λ[:,:,k,i+1])))))
    end
end

subplot(2,3,1); plotGMM(X, clusters_vb[:,1], R[:,:,1]); title("Initial situation")
for i=2:6
    subplot(2,3,i)
    plotGMM(X, clusters_vb[:,i*2], R[:,:,i*2]); title("After $(i*2) iterations")
end
PyPlot.tight_layout();
```

The generated figure looks much like Figure 10.6 in Bishop. The plots show results for Variational Bayesian
mixture of K = 6 Gaussians applied to the Old Faithful data set, in
which the ellipses denote the one
standard-deviation density contours
for each of the components, and the
color coding of the data points reflects the "soft" class label assignments. Components whose expected
mixing coefficient are numerically indistinguishable from zero are not
plotted. Note that this procedure learns the number of classes (two), learns the class label for each observation, and learns the mean and variance for the Gaussian data distributions.

### FE Minimization by the Expectation-Maximization (EM) Algorithm

- The EM algorithm is a special case of FE minimization that focusses on Maximum-Likelihood estimation for models with latent variables.
- Consider a model $$p(x,z,\theta)$$ with observations $x = \{x_n\}$, latent variables $z=\{z_n\}$ and parameters $\theta$.

- We can write the following FE functional for this model:
$$\begin{align*}
F[q] =  \sum_z \sum_\theta q(z) q(\theta) \log \frac{q(z) q(\theta)}{p(x,z,\theta)}
\end{align*}$$

- The EM algorithm makes the following simplifying assumptions:
  1. The prior for the parameters is uninformative (uniform). This implies that
  $$p(x,z,\theta) = p(x,z|\theta) p(\theta) \propto p(x,z|\theta)$$
  2. The posterior for the parameters is a delta function:
  $$q(\theta) = \delta(\theta - \hat{\theta})$$

- Basically, these two assumptions turn FE minimization into maximum likelihood estimation for the parameters $\theta$ and the FE simplifies to
$$\begin{align*}
F[q,\theta] =  \sum_z q(z) \log \frac{q(z)}{p(x,z|\theta)}
\end{align*}$$

- The EM algorithm minimizes this FE by iterating (iteration counter: $i$) over

\begin{equation*}
\boxed{
\begin{aligned}
\mathcal{L}^{(i)}(\theta) &= \sum_z \overbrace{p(z|x,\theta^{(i-1)})}^{q^{(i)}(z)}  \log p(x,z|\theta) \quad &&\text{the E-step} \\
\theta^{(i)} &= \arg\max_\theta \mathcal{L}^{(i)}(\theta) &&\text{the M-step}
\end{aligned}}
\end{equation*}

- These choices are optimal for the given FE functional. In order to see this, consider the two decompositions
$$\begin{align}
F[q,\theta] &= \underbrace{-\sum_z q(z) \log p(x,z|\theta)}_{\text{energy}} - \underbrace{\sum_z q(z) \log \frac{1}{q(z)}}_{\text{entropy}} \tag{EE}\\
  &= \underbrace{\sum_z q(z) \log \frac{q(z)}{p(z|x,\theta)}}_{\text{divergence}} - \underbrace{\log p(x|\theta)}_{\text{log-likelihood}}  \tag{DE}
\end{align}$$

- The DE decomposition shows that the FE is minimized for the choice $q(z) := p(z|x,\theta)$. Also, for this choice, the FE equals the (negative) log-evidence (, which is this case simplifies to the log-likelihood).

- The EE decomposition shows that the FE is minimized wrt $\theta$ by minimizing the energy term. The energy term is computed in the E-step and optimized in the M-step.
  - Note that in the EM literature, the energy term is often called the _expected complete-data log-likelihood_.)

- In order to execute the EM algorithm, it is assumed that we can analytically execute the E- and M-steps. For a large set of models (including models whose distributions belong to the exponential family of distributions), this is indeed the case and hence the large popularity of the EM algorithm.

- The EM algorihm imposes rather severe assumptions on the FE (basically approximating Bayesian inference by maximum likelihood estimation). Over the past few years, the rise of Probabilistic Programming languages has dramatically increased the range of models for which the parameters can by estimated autmatically by (approximate) Bayesian inference, so the popularity of EM is slowly waning. (More on this in the Probabilistic Programming lessons).

- Bishop (2006) works out EM for the GMM in section 9.2.

### CODE EXAMPLE: EM-algorithm for the GMM on the Old-Faithful data set

We'll perform clustering on the data set from the [illustrative example](#illustrative-example) by fitting a GMM consisting of two Gaussians using the EM algorithm.

```julia
using Pkg; Pkg.activate("probprog/workspace");Pkg.instantiate();
IJulia.clear_output();
```

```julia
using DataFrames, CSV, LinearAlgebra
include("scripts/gmm_plot.jl") # Holds plotting function
old_faithful = CSV.read("datasets/old_faithful.csv");

X =  Array(Matrix{Float64}(old_faithful)')
N = size(X, 2)

# Initialize the GMM. We assume 2 clusters.
clusters = [MvNormal([4.;60.], [.5 0;0 10^2]);
            MvNormal([2.;80.], [.5 0;0 10^2])];
π_hat = [0.5; 0.5]                    # Mixing weights
γ = fill!(Matrix{Float64}(undef,2,N), NaN)  # Responsibilities (row per cluster)

# Define functions for updating the parameters and responsibilities
function updateResponsibilities!(X, clusters, π_hat, γ)
    # Expectation step: update γ
    norm = [pdf(clusters[1], X) pdf(clusters[2], X)] * π_hat
    γ[1,:] = (π_hat[1] * pdf(clusters[1],X) ./ norm)'
    γ[2,:] = 1 .- γ[1,:]
end
function updateParameters!(X, clusters, π_hat, γ)
    # Maximization step: update π_hat and clusters using ML estimation
    m = sum(γ, dims=2)
    π_hat = m / N
    μ_hat = (X * γ') ./ m'
    for k=1:2
        Z = (X .- μ_hat[:,k])
        Σ_k = Symmetric(((Z .* (γ[k,:])') * Z') / m[k])
        clusters[k] = MvNormal(μ_hat[:,k], convert(Matrix, Σ_k))
    end
end

# Execute the algorithm: iteratively update parameters and responsibilities
subplot(2,3,1); plotGMM(X, clusters, γ); title("Initial situation")
updateResponsibilities!(X, clusters, π_hat, γ)
subplot(2,3,2); plotGMM(X, clusters, γ); title("After first E-step")
updateParameters!(X, clusters, π_hat, γ)
subplot(2,3,3); plotGMM(X, clusters, γ); title("After first M-step")
iter_counter = 1
for i=1:3
    for j=1:i+1
        updateResponsibilities!(X, clusters, π_hat, γ)
        updateParameters!(X, clusters, π_hat, γ)
        iter_counter += 1
    end
    subplot(2,3,3+i);
    plotGMM(X, clusters, γ);
    title("After $(iter_counter) iterations")
end
PyPlot.tight_layout()
```

Note that you can step through the interactive demo yourself by running [this script](https://github.com/bertdv/AIP-5SSB0/blob/master/lessons/notebooks/scripts/interactive_em_demo.jl) in julia. You can run a script in julia by
`julia> include("path/to/script-name.jl")`

###  Summary

- Latent variable models (LVM) contain a set of unobserved variables whose size grows with the number of observations.

- LVMs can model more complex phenomena than fully observed models, but inference in LVM models is usually not analytically solvable.

- The Free Energy (FE) functional transforms Bayesian inference computations (very large summations or integrals) to an optimization problem.

- Inference by minimizing FE, also known as variational inference, is fully consistent with the "Method of Maximum Relative Entropy", which is by design the rational way to update beliefs from priors to posteriors when new information becomes available. Thus, FE mimimization is a "correct" inference procedure that generalizes Bayes rule.

- In general, global FE minimization is an unsolved problem and finding good local minima is at the heart of current Bayesian technology research. Three simplifying constraints on the posterior $q(z)$ in the FE functional are currently popular in practical settings:
    - mean-field assumptions
    - assuming a parameterized PDF for $q$
    - EM algorithm

- These constraints often makes FE minimization implementable at the price of obtaining approximately Bayesian inference results.

- Also, while FE minimization with appropriate constraints may be implementable, it's definitely very complicated to derive the update equations manually. Hence there is currently a large research effort devoted to _automating_ variational inference (see also the Probabilistic Programming lessons later in this course).

```julia
open("../../styles/aipstyle.html") do f
    display("text/html", read(f, String))
end
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Nixie+One' rel='stylesheet' type='text/css'>

```julia

```


# $H_{\rm SSS}$, up to and including third post-Newtonian order

## This notebook constructs the spin-spin-spin coupling terms in the Hamiltonian up to and including third post-Newtonian order

**Notebook Status:**  Validated

**Validation Notes:** All expressions in this notebook were transcribed twice by hand on separate occasions, and expressions were corrected as needed to ensure consistency with published PN expressions. In addition, this tutorial notebook has been confirmed to be self-consistent with its corresponding NRPy+ module, as documented [below](#code_validation). **Additional validation tests may have been performed, but are as yet, undocumented.**

## Author: Zach Etienne

### This notebook exists as the following Python module:
1. [PN_Hamiltonian_SSS.py](../../edit/NRPyPN/PN_Hamiltonian_SSS.py)

### This notebook & corresponding Python module depend on the following NRPy+/NRPyPN Python modules:
1. [indexedexp.py](../../edit/indexedexp.py): [**documentation+tutorial**](../Tutorial-Indexed_Expressions.ipynb)
1. [NRPyPN_shortcuts.py](../../edit/NRPyPN/NRPyPN_shortcuts.py): [**documentation**](NRPyPN_shortcuts.ipynb)

# Table of Contents
$$\label{toc}$$

1. Part 1: [$H_{SSS,{\rm 3PN}}$](#threepn), as derived in [Levi and Steinhoff (2015)](https://arxiv.org/abs/1410.2601)
1. Part 2: [Validation against second transcription and corresponding Python module](#code_validation)
1. Part 3: [LaTeX PDF output](#latex_pdf_output): $\LaTeX$ PDF Output

# Part 1: $H_{\rm SSS, 3PN}$, as derived in [Levi and Steinhoff (2015)](https://arxiv.org/abs/1410.2601) \[Back to [top](#toc)\]
$$\label{twopn}$$

As described in the [nonspinning Hamiltonian notebook](PN-Hamiltonian-Nonspinning.ipynb), the basic physical system assumes two point particles of mass $m_1$ and $m_2$ with corresponding momentum vectors $\mathbf{P}_1$ and $\mathbf{P}_2$, and displacement vectors $\mathbf{X}_1$ and $\mathbf{X}_2$ with respect to the center of mass. Here we also consider the spin vectors of each point mass $\mathbf{S}_1$ and $\mathbf{S}_2$, respectively.

[Steinhoff, Hergt, and Schäfer (2008a)](https://arxiv.org/abs/0712.1716) and [Steinhoff, Hergt, and Schäfer (2008b)](https://arxiv.org/abs/0809.2200) adopt the additional notation

\begin{align}
\mathbf{r}_{12} &= (\mathbf{X}_1-\mathbf{X}_2)\\
r_{12} = r_{21} &= |\mathbf{r}_{12}|\\
\mathbf{n}_{12} &= \frac{\mathbf{r}_{12}}{r_{12}},
\end{align}
and when the numbers in subscripts are flipped, the particles are interchanged.

To reduce possibility of copying error, the equation for $H_{\rm SSS, 3PN}$ is taken directly from the arXiv LaTeX source code of Eq 3.12 in [Levi and Steinhoff (2015)](https://arxiv.org/abs/1410.2601), and only mildly formatted to (1) improve presentation in Jupyter notebooks and (2) to ensure some degree of consistency in notation across different terms in other Hamiltonian notebooks:

\begin{align}
H_{\text{SSS(BHB),\ 3PN}}&=\frac{1}{m_1^2r^4}\left\{\frac{3}{2}\left(\mathbf{S}_{1}^2\,
\mathbf{S}_{2}\cdot\mathbf{n}\times\mathbf{p}_{1}
+\mathbf{S}_{1}\cdot\mathbf{n}\,\mathbf{S}_{2}\cdot\mathbf{S}_{1}\times\mathbf{p}_{1}
-5\left(\mathbf{S}_{1}\cdot\mathbf{n}\right)^2\mathbf{S}_{2}\cdot\mathbf{n}\times\mathbf{p}_{1}\right.\right.\\
&\qquad\qquad\quad +\mathbf{n}\cdot\mathbf{S}_{1}\times\mathbf{S}_{2}\left(\mathbf{S}_{1}\cdot\mathbf{p}_{1}
-5\mathbf{S}_{1}\cdot\mathbf{n}\,\mathbf{p}_{1}\cdot\mathbf{n}\right)\\
&\qquad\quad -\frac{3m_1}{2m_2}\left[\mathbf{S}_{1}^2\,
\mathbf{S}_{2}\cdot\mathbf{n}\times\mathbf{p}_2+2\mathbf{S}_{1}\cdot\mathbf{n}\,
\mathbf{S}_{2}\cdot\mathbf{S}_{1}\times\mathbf{p}_{2}
-5\left(\mathbf{S}_{1}\cdot\mathbf{n}\right)^2\mathbf{S}_{2}\cdot\mathbf{n}\times\mathbf{p}_{2}\right)\bigg]\\
&\qquad\qquad \left.-\mathbf{S}_{1}\times\mathbf{n}\cdot\left(\mathbf{p}_{2}
-\frac{m_2}{4m_1}\mathbf{p}_{1}\right)\left(\mathbf{S}_{1}^2
-5\left(\mathbf{S}_{1}\cdot\mathbf{n}\right)^2\right)\right\}\\
&\quad+ 1\longleftrightarrow2.
\end{align}

```python
# Step 0: Add NRPy's directory to the path
# https://stackoverflow.com/questions/16780014/import-file-from-parent-directory
import os,sys                    # Standard Python modules for multiplatform OS-level functions
nrpy_dir_path = os.path.join("..")
if nrpy_dir_path not in sys.path:
    sys.path.append(nrpy_dir_path)
import indexedexp as ixp                   # NRPy+: Symbolic indexed expression (e.g., tensors, vectors, etc.) support
from NRPyPN_shortcuts import div,dot,cross # NRPyPN: shortcuts for e.g., vector operations

# Step 1: 3PN spin-spin-spin term, from Eq. 3.12 of
#        Levi and Steinhoff (2015):
#     https://arxiv.org/abs/1410.2601
def f_H_SSS_3PN(m1,m2, n12U,n21U, S1U,S2U, p1U,p2U, r12):
    def f_H_SSS_3PN_pt(m1,m2, nU, S1U,S2U, p1U,p2U, r):
        p2_minus_m2_over_4m1_p1  = ixp.zerorank1()
        for i in range(3):
            p2_minus_m2_over_4m1_p1[i]  = p2U[i] - m2/(4*m1)*p1U[i]
        H_SSS_3PN_pt = (+div(3,2)*(+dot(S1U,S1U)*dot(S2U,cross(nU,p1U))
                                   +dot(S1U,nU)*dot(S2U,cross(S1U,p1U))
                                   -5*dot(S1U,nU)**2*dot(S2U,cross(nU,p1U))
                                   +dot(nU,cross(S1U,S2U))*(+dot(S1U,p1U)
                                                            -5*dot(S1U,nU)*dot(p1U,nU)))
                        -3*m1/(2*m2)*(  +dot(S1U,S1U)  *dot(S2U,cross(nU,p2U))
                                      +2*dot(S1U,nU)   *dot(S2U,cross(S1U,p2U))
                                      -5*dot(S1U,nU)**2*dot(S2U,cross(nU,p2U)))
                        -dot(cross(S1U,nU),p2_minus_m2_over_4m1_p1)*(dot(S1U,S1U) - 5*dot(S1U,nU)**2))/(m1**2*r**4)
        return H_SSS_3PN_pt
    global H_SSS_3PN
    H_SSS_3PN = (+f_H_SSS_3PN_pt(m1,m2, n12U, S1U,S2U, p1U,p2U, r12)
                 +f_H_SSS_3PN_pt(m2,m1, n21U, S2U,S1U, p2U,p1U, r12))
```

```python
# Second version, for validation purposes only.
def f_H_SSS_3PNv2(m1,m2, n12U,n21U, S1U,S2U, p1U,p2U, q):
    def SHS2015_HSSS_3PNv2_pt(m1,m2, nU, S1U,S2U, p1U,p2U, q):
        p2_minus_m2_over_4m1_p1 = ixp.zerorank1()
        for i in range(3):
            p2_minus_m2_over_4m1_p1[i] = p2U[i] - m2/(4*m1)*p1U[i]
        H = ( div(3,2)*(+  dot(S1U,S1U)*dot(S2U,cross(nU,p1U))   # line 1
                        +  dot(S1U,nU)*dot(S2U,cross(S1U,p1U))   # line 1
                        -5*dot(S1U,nU)**2*dot(S2U,cross(nU,p1U)) # line 1
                        +dot(nU,cross(S1U,S2U))*(dot(S1U,p1U)-5*dot(S1U,nU)*dot(p1U,nU))) # line 2
                        -div(3,2)*m1/m2*(+  dot(S1U,S1U)*dot(S2U,cross(nU,p2U))     # line 3
                                         +2*dot(S1U,nU)*dot(S2U,cross(S1U,p2U))     # line 3
                                         -5*dot(S1U,nU)**2*dot(S2U,cross(nU,p2U))) # line 3
              -dot(cross(S1U,nU), p2_minus_m2_over_4m1_p1)*(dot(S1U,S1U) - 5*dot(S1U,nU)**2 ))/(m1**2*q**4)
        return H

    global H_SSS_3PNv2
    H_SSS_3PNv2 = (+SHS2015_HSSS_3PNv2_pt(m1,m2, n12U, S1U,S2U, p1U,p2U, q)
                   +SHS2015_HSSS_3PNv2_pt(m2,m1, n21U, S2U,S1U, p2U,p1U, q))
```

# Part 4: Validation against second transcription and corresponding Python module \[Back to [top](#toc)\]
$$\label{code_validation}$$

As a code validation check, we verify agreement between
* the SymPy expressions transcribed from the cited published work on two separate occasions, and
* the SymPy expressions generated in this notebook, and the corresponding Python module.

```python
import sympy as sp               # SymPy: The Python computer algebra package upon which NRPy+ depends
from NRPyPN_shortcuts import m1,m2, n12U,n21U, S1U,S2U, p1U,p2U, q # NRPyPN: import needed input variables

f_H_SSS_3PN(m1,m2, n12U,n21U, S1U,S2U, p1U,p2U, q)

def error(varname):
    print("ERROR: When comparing Python module & notebook, "+varname+" was found not to match.")
    sys.exit(1)

# Validation against second transcription of the expressions:
f_H_SSS_3PNv2(m1,m2, n12U,n21U, S1U,S2U, p1U,p2U, q)
if sp.simplify(H_SSS_3PN - H_SSS_3PNv2) != 0: error("H_SSS_3PNv2")

# Validation against corresponding Python module:
import PN_Hamiltonian_SSS as HSSS
HSSS.f_H_SSS_3PN(m1,m2, n12U,n21U, S1U,S2U, p1U,p2U, q)

if sp.simplify(H_SSS_3PN - HSSS.H_SSS_3PN) != 0: error("H_SSS_3PN")
print("ALL TESTS PASS")
```

    ALL TESTS PASS

# Part 5: Output this notebook to $\LaTeX$-formatted PDF file \[Back to [top](#toc)\]
$$\label{latex_pdf_output}$$

The following code cell converts this Jupyter notebook into a proper, clickable $\LaTeX$-formatted PDF file. After the cell is successfully run, the generated PDF may be found in the root NRPy+ tutorial directory, with filename
[PN-Hamiltonian-SSS.pdf](PN-Hamiltonian-SSS.pdf) (Note that clicking on this link may not work; you may need to open the PDF file through another means.)

```python
import os,sys                    # Standard Python modules for multiplatform OS-level functions
nrpy_dir_path = os.path.join("..")
if nrpy_dir_path not in sys.path:
    sys.path.append(nrpy_dir_path)
import cmdline_helper as cmd    # NRPy+: Multi-platform Python command-line interface
cmd.output_Jupyter_notebook_to_LaTeXed_PDF("PN-Hamiltonian-SSS",location_of_template_file=os.path.join(".."))
```

    Created PN-Hamiltonian-SSS.tex, and compiled LaTeX file to PDF file PN-
        Hamiltonian-SSS.pdf