```python
%pylab inline
import pandas as pd

import numpy as np
from __future__ import division
import itertools

import matplotlib.pyplot as plt
import seaborn as sns

import logging
logger = logging.getLogger()
```

    Populating the interactive namespace from numpy and matplotlib

9 Recommendation Systems
=============

two broad groups:

1. Content-based systems
   focus on the properities of items.

2. Collaborative filtering systems
   focus on the relationship between users and items.

### 9.1 A Model for Recommendation Systems

#### The Utility Matrix
record the preference given by users for certain items.

```python
# Example 9.1
M = pd.DataFrame(index=['A', 'B', 'C', 'D'], columns=['HP1', 'HP2', 'HP3', 'TW', 'SW1', 'SW2', 'SW3'])

M.loc['A', ['HP1', 'TW', 'SW1']] = [4, 5, 1]
M.iloc[1, 0:3] = [5, 5, 4]
M.iloc[2, 3:-1] = [2, 4, 5]
M.iloc[3, [1, -1]] = [3, 3]

M_9_1 = M
M_9_1
```

      HP1
      HP2
      HP3
      TW
      SW1
      SW2
      SW3

      A
      4
      NaN
      NaN
      5
      1
      NaN
      NaN

      B
      5
      5
      4
      NaN
      NaN
      NaN
      NaN

      C
      NaN
      NaN
      NaN
      2
      4
      5
      NaN

      D
      NaN
      3
      NaN
      NaN
      NaN
      NaN
      3

In practice, the matrix would be even **sparser**, with the typical user rating only a tiny fraction of all avalibale items.

the **goal** of a recommendation system is: to **predict the blanks** in the utility matrix.
+ slightly difference in many application:
  - predict every blank entry $<$ discover some potential entries in each row.
  - find all items with the highest expected ratings $<$ find a large subset of those.

#### The Long Tail
physical institutions | online institutions
---- | -----
provide only the most popular items | provide the entire range of items

the long tail force online institutions to recommend items to individual users:

1. It's no possible to present all avaliable items to the user.

2. Neither can we expect users to have heared of each of the items they might like.

#### Applications of Recommendation Systems
1. Product Recommendations

2. Movie Recommendations

3. News Articles

#### Populating the Utility Matrix
how to discovery the value users place on items:

1. We can ask users to rate items.
   cons: users are unwilling to do, and so samples are biased by very little fraction of peoples.

2. We can make inferences from users' behavior.
   eg: items purchased/viewed/rated.

### 9.2 Content-Based Recommendations

#### 9.2.1 Item Profiles
a record representing important characteristics of items.

##### Discovering Features
1. for Documents
   idea: find the identification of words that characterize the topic of a document.
   namely, we expect a sets of words to express the subjects or main ideas of the document.
   1. eliminate stop words.
   2. compute the TF.DIF score for each reamining word in the document.
   3. take as the features of a document the $n$ words with the highest TF.DIF scores.

   to measure the similarity of two documents, the distance measures we could use are:
   1. Jaccard distance
   2. cosine distance
      cosine distance of vectors is not affected by components in which both vectors have 0.

2. for Images
   invite users to tag the items.
   cons: users are unwilling to do $\implies$ there are not enough tags (bias).

##### generalize feature vector
1. feature is discrete. $\to$ boolean value.

2. feature is numerical. $\to$ normalization.

#### 9.2.5 User Profiles
create vectors with the same components of item profiles to describe the user's preferences.

It could be derived from utility matrix and item profiles.

1. normalizate untility matrix. ($[-1,1]$ for cosine distance).

2. value in user profiles = utility value * corresponding item vectors.

```python
# example 9.4

users_name = ['U', 'V']
items_name = ['F{}'.format(x) for x in range(4)]
features_name = ['Julia Roberts', 'others']

# utility matrix
M_uti = pd.DataFrame([
                        [3, 4, 5, 0],
                        [6, 2, 3, 5]
                     ],
                     index=users_name,
                     columns=items_name
                    )

M_uti
```

      F0
      F1
      F2
      F3

      U
      3
      4
      5
      0

      V
      6
      2
      3
      5

```python
# item profile
M_item = pd.DataFrame(index=items_name, columns=features_name)

M_item.loc[:, features_name[0]] = 1
M_item = M_item.fillna(value=0)

M_item
```

      Julia Roberts
      others

      F0
      1
      0

      F1
      1
      0

      F2
      1
      0

      F3
      1
      0

```python
M_uti.apply(lambda x: x - np.mean(x), axis=1)
```

      F0
      F1
      F2
      F3

      U
      0
      1
      2
      -3

      V
      2
      -2
      -1
      1

```python
M_user = M_uti.fillna(value=0).dot(M_item) / 4 #average = sum/len
M_user
```

      Julia Roberts
      others

      U
      3
      0

      V
      4
      0

#### 9.2.6 Recommending Items to Users Based on Content

1. to estimate:
   $$M_{utility}[user, item] = cosineDistant(M_{user}, M_{item})$$

   the more similar, the higher probility to recommend.

2. classification algorithms:
   Recommend or Not (machine learning):
   one decision per user $\to$ take too long time to construct.
   be used only for relatively small problem size.

```python
# exercise 9.2.1

raw_data = [
        [3.06, 2.68, 2.92],
        [500, 320, 640],
        [6, 4, 6]
    ]

M_item = pd.DataFrame(raw_data, index=['Processor Speed', 'Disk Size', 'Main-Memory Size'], columns=['A', 'B', 'C'])

# items: A, B, C; features: Processor Speed, Disk Size, ...
M_item
```

      A
      B
      C

      Processor Speed
      3.06
      2.68
      2.92

      Disk Size
      500.00
      320.00
      640.00

      Main-Memory Size
      6.00
      4.00
      6.00

```python
# exercise 9.2.1

# (d)
M_item.apply(lambda x: x / np.mean(x), axis=1)
```

      A
      B
      C

      Processor Speed
      1.060046
      0.928406
      1.011547

      Disk Size
      1.027397
      0.657534
      1.315068

      Main-Memory Size
      1.125000
      0.750000
      1.125000

```python
# exercise 9.2.2

# (a)
M_item.apply(lambda x: x - np.mean(x), axis=1)
```

      A
      B
      C

      Processor Speed
      0.173333
      -0.206667
      0.033333

      Disk Size
      13.333333
      -166.666667
      153.333333

      Main-Memory Size
      0.666667
      -1.333333
      0.666667

```python
# exercise 9.2.3

M_uti = pd.DataFrame([[4, 2, 5]], index=['user'], columns=['A', 'B', 'C'])
M_uti
```

      A
      B
      C

      user
      4
      2
      5

```python
# (a)
M_uti_nor = M_uti.apply(lambda x: x - np.mean(x), axis=1)
M_uti_nor
```

      A
      B
      C

      user
      0.333333
      -1.666667
      1.333333

```python
# (b)
M_user = M_item.dot(M_uti_nor.T) / 3
M_user
```

      user

      Processor Speed
      0.148889

      Disk Size
      162.222222

      Main-Memory Size
      1.111111

```python
logger.setLevel('WARN')

def create_user_profile(utility_matrix, item_profile):
    """Create user profile by combining utility matrix with item profile in 9.2.5 ."""

    assert np.array_equal(utility_matrix.columns, item_profile.columns), \
                      "utility matrix should keep same columns name with item profile."

    logger.info('utility_matrix: \n{}\n'.format(utility_matrix))
    M_uti_notnull = np.ones(utility_matrix.shape)
    M_uti_notnull[utility_matrix.isnull().values] = 0
    logger.info('utility_matrix_isnull: \n{}\n'.format(M_uti_notnull))

    logger.info('utility_matrix: \n{}\n'.format(item_profile))
    M_item_notnull = np.ones(item_profile.shape)
    M_item_notnull[item_profile.isnull().values] = 0
    logger.info('utility_matrix_isnull: \n{}\n'.format(M_item_notnull))

    utility_matrix = utility_matrix.fillna(value=0)
    item_profile = item_profile.fillna(value=0)
    M_user = item_profile.dot(utility_matrix.T).values / np.dot(M_item_notnull, M_uti_notnull.T)
    M_user[np.isinf(M_user)] = np.nan   # solve: divide zero
    logger.info('M_user: \n{}\n'.format(M_user))

    return pd.DataFrame(M_user, index=item_profile.index, columns=utility_matrix.index)

M_uti = pd.DataFrame([[4, 2, 5], [1, np.nan, 3]], index=['userA', 'userB'], columns=['A', 'B', 'C'])
M_uti_nor = M_uti.apply(lambda x: x - np.mean(x), axis=1)
print('utility matrix: \n{}\n'.format(M_uti_nor))
print('item profile: \n{}\n'.format(M_item))

create_user_profile(M_uti_nor, M_item)
```

    utility matrix:
                  A         B         C
    userA  0.333333 -1.666667  1.333333
    userB -1.000000       NaN  1.000000

    item profile:
                           A       B       C
    Processor Speed     3.06    2.68    2.92
    Disk Size         500.00  320.00  640.00
    Main-Memory Size    6.00    4.00    6.00

      userA
      userB

      Processor Speed
      0.148889
      -0.07

      Disk Size
      162.222222
      70.00

      Main-Memory Size
      1.111111
      0.00

### 9.3 Collaborative Filtering
identifying similar users and recommending what similar users like.

#### refine data
1. rounding the data
   eg: rates of 3, 4 and 5 are "1", otherwise "0".

2. normalizing rates
   subtracting from each rating the average rating of that user.

```python
# Fig 9.4
M_9_1
```

      HP1
      HP2
      HP3
      TW
      SW1
      SW2
      SW3

      A
      4
      NaN
      NaN
      5
      1
      NaN
      NaN

      B
      5
      5
      4
      NaN
      NaN
      NaN
      NaN

      C
      NaN
      NaN
      NaN
      2
      4
      5
      NaN

      D
      NaN
      3
      NaN
      NaN
      NaN
      NaN
      3

```python
# rounding the data
M_round = M_9_1.copy()
M_round[M_9_1 <= 2] = np.nan
M_round[M_9_1 > 2] = 1

M_round
```

      HP1
      HP2
      HP3
      TW
      SW1
      SW2
      SW3

      A
      1
      NaN
      NaN
      1
      NaN
      NaN
      NaN

      B
      1
      1
      1
      NaN
      NaN
      NaN
      NaN

      C
      NaN
      NaN
      NaN
      NaN
      1
      1
      NaN

      D
      NaN
      1
      NaN
      NaN
      NaN
      NaN
      1

```python
# normalizing ratings
M_norm = M_9_1.apply(lambda x: x - np.mean(x), axis=1)
M_norm
```

      HP1
      HP2
      HP3
      TW
      SW1
      SW2
      SW3

      A
      0.666667
      NaN
      NaN
      1.666667
      -2.333333
      NaN
      NaN

      B
      0.333333
      0.333333
      -0.666667
      NaN
      NaN
      NaN
      NaN

      C
      NaN
      NaN
      NaN
      -1.666667
      0.333333
      1.333333
      NaN

      D
      NaN
      0.000000
      NaN
      NaN
      NaN
      NaN
      0

#### 9.3.2 The Duality of Similarity
1. We can use information about users to recommend items, whereas even if we find pairs of similar items, it takes an **additional step** in order to recommend items to users.
   + find $n$ similar users $\to$ recommend item $I$ to user $U$.
     normalize the utility matrix first.
     \begin{align}
         M[U,I] &= Ave(M[U,:]) + Ave(M[0:n,I] - Ave(M[0:n,I])) \\
                 &\approx Ave(M[U,:]) + Std(M[0:n,I])
     \end{align}

   + find $m$ similar items $\to$ recommend item $I$ to user $U$.
     $$M[U,I] = Ave(M[U,0:m])$$

   + in order to recommend items to user $U$, we need to find all or most of entry in $M[U,:]$.
     **tradeoff**:
     1. user-item: find similar users, directly get all predict values of all potent items.
        item-item: find similar items, we need **calculate all items one by one (additional step)** to fill $M[U,:]$.

     2. item-item similarity often provides more **reliable** information due to the simplicity of items (genre).

   + **precompute** preferred items for each user.
     utility matrix evolves slowly $\implies$ compute it infrequently and assume that it remains fixed between recomputations.

2. Items tend to be classifiable in simple terms (eg: genre), whereas the individuals are complex.

#### 9.3.3 Clustering Users and Items
Hierachical approach is prefered:

1. leave many cluster unmerged at first.

2. cluster items, and average the corresponding value in utility matrix.

3. cluster users, and average as well.

4. repeat several times if we like.

Predict $M[U,I]:

1. $U \in C$, and $I \in D$.

2. predict:

   \begin{equation}
       M[U,I] = \begin{cases}
           M_{revised}[C,D] & \quad \text{if existed.}  \\
           \text{estimate using similar users/items} & \quad \text{otherwise}
       \end{cases}
   \end{equation}

```python
# Fig 9.8

raw_data = [
    [4, 5, np.nan, 5, 1, np.nan, 3, 2],
    [np.nan, 3, 4, 3, 1, 2, 1, np.nan],
    [2, np.nan, 1, 3, np.nan, 4, 5, 3]
]

import string
M_uti = pd.DataFrame(raw_data, index=list(string.uppercase[:3]), columns=list(string.lowercase[:8]))
M_uti
```

      a
      b
      c
      d
      e
      f
      g
      h

      A
      4
      5
      NaN
      5
      1
      NaN
      3
      2

      B
      NaN
      3
      4
      3
      1
      2
      1
      NaN

      C
      2
      NaN
      1
      3
      NaN
      4
      5
      3

```python
logger.setLevel('WARN')
# exercise 9.3.1
from scipy.spatial.distance import jaccard, cosine
from itertools import combinations

def calc_distance_among_matrix(M, func_dis):
    for c in list(combinations(M.index, 2)):
        logger.info('c: {}'.format(c))
        u, v = M.loc[c[0]], M.loc[c[1]]
        logger.info('\n u:{},\n v:{}\n'.format(u.values,v.values))
        print('{} {}: {}'.format(c, func_dis.__name__, func_dis(u,v)))

# (a)
calc_distance_among_matrix(M_uti.notnull(), jaccard)
```

    ('A', 'B') jaccard: 0.5
    ('A', 'C') jaccard: 0.5
    ('B', 'C') jaccard: 0.5

```python
# (b)
calc_distance_among_matrix(M_uti.fillna(value=0), cosine)
```

    ('A', 'B') cosine: 0.398959235991
    ('A', 'C') cosine: 0.385081306188
    ('B', 'C') cosine: 0.486129880223

```python
# (c)
M_tmp = M_uti.copy()
M_tmp[M_uti < 3] = 0
M_tmp[M_uti >= 3] = 1
calc_distance_among_matrix(M_tmp, jaccard)
```

    ('A', 'B') jaccard: 0.714285714286
    ('A', 'C') jaccard: 0.75
    ('B', 'C') jaccard: 0.875

```python
# (d)
calc_distance_among_matrix(M_tmp.fillna(value=0), cosine)
```

    ('A', 'B') cosine: 0.42264973081
    ('A', 'C') cosine: 0.5
    ('B', 'C') cosine: 0.711324865405

```python
# (e)
M_uti_nor = M_uti.apply(lambda x: x - np.mean(x), axis=1)
M_uti_nor
```

      a
      b
      c
      d
      e
      f
      g
      h

      A
      0.666667
      1.666667
      NaN
      1.666667
      -2.333333
      NaN
      -0.333333
      -1.333333

      B
      NaN
      0.666667
      1.666667
      0.666667
      -1.333333
      -0.333333
      -1.333333
      NaN

      C
      -1.000000
      NaN
      -2.000000
      0.000000
      NaN
      1.000000
      2.000000
      0.000000

```python
# (f)
calc_distance_among_matrix(M_uti_nor.fillna(value=0), cosine)
```

    ('A', 'B') cosine: 0.415693452532
    ('A', 'C') cosine: 1.11547005384
    ('B', 'C') cosine: 1.73957399695

```python
# exercise 9.3.2
#todo
```

### 9.4 Dimensionality Reduction

UV-decomposition: $$M = U \times V$$

measure: RMSE (root-mean-square error)

#### Building a Complete UV-Decomposition Algorithm
1. Preprocessing of the matrix $M$.
   **normalization**:
   1. subtract: average rating of user $i$, then average rating of item $j$.
   2. subtract: first item, then user.
   3. subtract: half of average of item and half of average of user.

2. Initializing $U$ and $V$.
   choice: gives the elements of $UV$ the average of the nonblank elements of $M$.
   $\implies$ the element of $U$ and $V$ should be $\sqrt{a/d}$,
   where $a$ is the average nonblank element of $M$, $d$ is the lengths of the short sides of $U$ and $V$.

   local minima contains global minima:
   1. vary the initial values of $U$ and $V$:
      perturb the value $\sqrt{a/d}$ randomly.
   2. vary the way we seek the optimum.

3. Performing the Optimization.
   different optimization path:
   choose a permutation of the elements and follow that order for every round.

   Gradient Descent $\to$ stochastic gradient descent.

4. Converging to a Minimum.
   track the amount of improvement in the RMSE obtained.

   stop condition:
   1. stop when that improvement in one round falls below a threshold.
   2. stop when the maximum improvement during a round is below a threshold.

##### Avoiding Overfitting
solutions:

1. optimized by only moving the value of a component a fraction of the way from its current value toward its optimized value.

2. Stop before the process has converged.

3. Take several different $UV$ decompositions, and average their predicts.

```python
# exercise 9.4.6

#todo
```

### 9.5 The NetFlix Challenge

some facts:

1. CineMatch was not a very good algorithm.

2. UV-decomposition alorithm given a 7\% improvement over CineMatch when couped with normalization and a few other tricks.

3. Combing different algorithms is a preferred strategy.

4. Genre and other information in IMDB was no useful.

5. Time of rating turned out to be useful: upward or downward slope with time.

todo: read the papers introduced in the chapter.


# **CS224W - Colab 3**

In Colab 2 we constructed GNN models by using PyTorch Geometric's built in GCN layer, `GCNConv`. In this Colab we will go a step deeper and implement the **GraphSAGE** ([Hamilton et al. (2017)](https://arxiv.org/abs/1706.02216)) layer directly. Then we will run our models on the CORA dataset, which is a standard citation network benchmark dataset.

**Note**: Make sure to **sequentially run all the cells in each section** so that the intermediate variables / packages will carry over to the next cell

Have fun and good luck on Colab 3 :)

# Device
We recommend using a GPU for this Colab.

Please click `Runtime` and then `Change runtime type`. Then set the `hardware accelerator` to **GPU**.

## Installation

```python
# Install torch geometric
import os
if 'IS_GRADESCOPE_ENV' not in os.environ:
    !pip install torch-scatter -f https://pytorch-geometric.com/whl/torch-1.10.0+cu111.html
    !pip install torch-sparse -f https://pytorch-geometric.com/whl/torch-1.10.0+cu111.html
    !pip install torch-geometric
    !pip install -q git+https://github.com/snap-stanford/deepsnap.git
```

    Looking in links: https://pytorch-geometric.com/whl/torch-1.10.0+cu111.html
    Requirement already satisfied: torch-scatter in /usr/local/lib/python3.7/dist-packages (2.0.9)
    Looking in links: https://pytorch-geometric.com/whl/torch-1.10.0+cu111.html
    Requirement already satisfied: torch-sparse in /usr/local/lib/python3.7/dist-packages (0.6.12)
    Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from torch-sparse) (1.4.1)
    Requirement already satisfied: numpy>=1.13.3 in /usr/local/lib/python3.7/dist-packages (from scipy->torch-sparse) (1.19.5)
    Requirement already satisfied: torch-geometric in /usr/local/lib/python3.7/dist-packages (2.0.3)
    Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from torch-geometric) (1.4.1)
    Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from torch-geometric) (1.1.5)
    Requirement already satisfied: PyYAML in /usr/local/lib/python3.7/dist-packages (from torch-geometric) (3.13)
    Requirement already satisfied: googledrivedownloader in /usr/local/lib/python3.7/dist-packages (from torch-geometric) (0.4)
    Requirement already satisfied: networkx in /usr/local/lib/python3.7/dist-packages (from torch-geometric) (2.6.3)
    Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from torch-geometric) (4.62.3)
    Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from torch-geometric) (2.23.0)
    Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from torch-geometric) (1.19.5)
    Requirement already satisfied: rdflib in /usr/local/lib/python3.7/dist-packages (from torch-geometric) (6.1.1)
    Requirement already satisfied: jinja2 in /usr/local/lib/python3.7/dist-packages (from torch-geometric) (2.11.3)
    Requirement already satisfied: pyparsing in /usr/local/lib/python3.7/dist-packages (from torch-geometric) (3.0.6)
    Requirement already satisfied: yacs in /usr/local/lib/python3.7/dist-packages (from torch-geometric) (0.1.8)
    Requirement already satisfied: scikit-learn in /usr/local/lib/python3.7/dist-packages (from torch-geometric) (1.0.2)
    Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.7/dist-packages (from jinja2->torch-geometric) (2.0.1)
    Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas->torch-geometric) (2018.9)
    Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.7/dist-packages (from pandas->torch-geometric) (2.8.2)
    Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.7.3->pandas->torch-geometric) (1.15.0)
    Requirement already satisfied: isodate in /usr/local/lib/python3.7/dist-packages (from rdflib->torch-geometric) (0.6.1)
    Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from rdflib->torch-geometric) (57.4.0)
    Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/dist-packages (from rdflib->torch-geometric) (4.10.0)
    Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->rdflib->torch-geometric) (3.7.0)
    Requirement already satisfied: typing-extensions>=3.6.4 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->rdflib->torch-geometric) (3.10.0.2)
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->torch-geometric) (2.10)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->torch-geometric) (2021.10.8)
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->torch-geometric) (3.0.4)
    Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->torch-geometric) (1.24.3)
    Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn->torch-geometric) (1.1.0)
    Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn->torch-geometric) (3.0.0)

```python
#import torch_geometric
#torch_geometric.__version__
```

# 1) GNN Layers

## Implementing Layer Modules

In Colab 2, we implemented a GCN model for node and graph classification tasks. However, for that notebook we took advantage of PyG's built in GCN module. For Colab 3, we provide a build upon a general Graph Neural Network Stack, into which we will be able to plugin our own module implementations: GraphSAGE and GAT.

We will then use our layer implemenations to complete node classification on the CORA dataset, a standard citation network benchmark. In this dataset, nodes correspond to documents and edges correspond to undirected citations. Each node or document in the graph is assigned a class label and features based on the documents binarized bag-of-words representation. Specifically, the Cora graph has 2708 nodes, 5429 edges, 7 prediction classes, and 1433 features per node.

## GNN Stack Module

Below is the implementation of a general GNN stack, where we can plugin any GNN layer, such as **GraphSage**, **GAT**, etc. This module is provided for you. Your implementations of the **GraphSage** and **GAT** (Colab 4) layers will function as components in the GNNStack Module.

```python
import torch
import torch_scatter
import torch.nn as nn
import torch.nn.functional as F

import torch_geometric.nn as pyg_nn
import torch_geometric.utils as pyg_utils

from torch import Tensor
from typing import Union, Tuple, Optional
from torch_geometric.typing import (OptPairTensor, Adj, Size, NoneType,
                                    OptTensor)

from torch.nn import Parameter, Linear
from torch_sparse import SparseTensor, set_diag
from torch_geometric.nn.conv import MessagePassing
from torch_geometric.utils import remove_self_loops, add_self_loops, softmax

class GNNStack(torch.nn.Module):

    def __init__(
        self,
        input_dim,
        hidden_dim,
        output_dim,
        args,
        emb=False
        ):

        # Parentizing
        super(GNNStack, self).__init__()

        # What type of GNN we are talking about
        conv_model = self.build_conv_model(args.model_type)

        # List of conv layers
        self.convs = nn.ModuleList()

        # Append input layer
        self.convs.append(conv_model(input_dim, hidden_dim))

        # Check if the number of layers is more than 1
        assert (args.num_layers >= 1), 'Number of layers is not >=1'

        # Go through all layers
        for l in range(args.num_layers-1):
            self.convs.append(conv_model(args.heads * hidden_dim, hidden_dim))

        # Post-message-passing
        self.post_mp = nn.Sequential(
            nn.Linear(args.heads * hidden_dim, hidden_dim),
            nn.Dropout(args.dropout),
            nn.Linear(hidden_dim, output_dim)
            )

        # Define dropout
        self.dropout = args.dropout

        # Store number of layers for future use into the methods
        self.num_layers = args.num_layers

        # Storing boolean about embeddings (True dont apply a classification
        # layer in the end)
        self.emb = emb

    def build_conv_model(self, model_type):

        if model_type == 'GraphSage':
            return GraphSage

        elif model_type == 'GAT':
            # When applying GAT with num heads > 1, you need to modify the
            # input and output dimension of the conv layers (self.convs),
            # to ensure that the input dim of the next layer is num heads
            # multiplied by the output dim of the previous layer.
            # HINT: In case you want to play with multiheads, you need to change the for-loop that builds up self.convs to be
            # self.convs.append(conv_model(hidden_dim * num_heads, hidden_dim)),
            # and also the first nn.Linear(hidden_dim * num_heads, hidden_dim) in post-message-passing.
            return GAT

    def forward(self, data):

        # Extract from batch
        x, edge_index, batch = data.x, data.edge_index, data.batch

        # Go through all layers
        for i in range(self.num_layers):
            x = self.convs[i](x, edge_index)
            x = F.relu(x)
            x = F.dropout(x, p=self.dropout, training=self.training)

        x = self.post_mp(x)

        if self.emb == True:
            return x

        return F.log_softmax(x, dim=1)

    def loss(self, pred, label):
        return F.nll_loss(pred, label)
```

## Creating Our Own Message Passing Layer

Now let's start implementing our own message passing layers! Working through this part will help us become acutely familiar with the behind the scenes work of implementing Pytorch Message Passing Layers, allowing us to build our own GNN models. To do so, we will work with and implement 3 critcal functions needed to define a PyG Message Passing Layer: `forward`, `message`, and `aggregate`.

Before diving head first into the coding details, let us quickly review the key components of the message passing process. To do so, we will focus on a single round of messsage passing with respect to a single central node $x$. Before message passing, $x$ is associated with a feature vector $x^{l-1}$, and the goal of message passing is to update this feature vector as $x^l$. To do so, we implement the following steps: 1) each neighboring node $v$ passes its current message $v^{l-1}$ across the edge $(x, v)$ - 2) for the node $x$, we aggregate all of the messages of the neighboring nodes (for example through a sum or mean) - and 3) we transform the aggregated information by for example applying linear and non-linear transformations. Altogether, the message passing process is applied such that every node $u$ in our graph updates its embedding by acting as the central node $x$ in step 1-3 described above.

Now, we extending this process to that of a single message passing layer, the job of a message passing layer is to update the current feature representation or embedding of each node in a graph by propagating and transforming information within the graph. Overall, the general paradigm of a message passing layers is: 1) pre-processing -> 2) **message passing** / propagation -> 3) post-processing.

The `forward` fuction that we will implement for our message passing layer captures this execution logic. Namely, the `forward` function handles the pre and post-processing of node features / embeddings, as well as initiates message passing by calling the `propagate` function.

The `propagate` function encapsulates the message passing process! It does so by calling three important functions: 1) `message`, 2) `aggregate`, and 3) `update`. Our implementation will vary slightly from this, as we will not explicitly implement `update`, but instead place the logic for updating node embeddings after message passing and within the `forward` function. To be more specific, after information is propagated (message passing), we can further transform the node embeddings outputed by `propagate`. Therefore, the output of `forward` is exactly the node embeddings after one GNN layer.

Lastly, before starting to implement our own layer, let us dig a bit deeper into each of the functions described above:

1.

```
def propagate(edge_index, x=(x_i, x_j), extra=(extra_i, extra_j), size=size):
```
Calling `propagate` initiates the message passing process. Looking at the function parameters, we highlight a couple of key parameters.

  - `edge_index` is passed to the forward function and captures the edge structure of the graph.
  - `x=(x_i, x_j)` represents the node features that will be used in message passing. In order to explain why we pass the tuple `(x_i, x_j)`, we first look at how our edges are represented. For every edge $(i, j) \in \mathcal{E}$, we can differentiate $i$ as the source or central node ($x_{central}$) and j as the neighboring node ($x_{neighbor}$).

    Taking the example of message passing above, for a central node $u$ we will aggregate and transform all of the messages associated with the nodes $v$ s.t. $(u, v) \in \mathcal{E}$ (i.e. $v \in \mathcal{N}_{u}$). Thus we see, the subscripts `_i` and `_j` allow us to specifcally differenciate features associated with central nodes (i.e. nodes  recieving message information) and neighboring nodes (i.e. nodes passing messages).

    This is definitely a somewhat confusing concept; however, one key thing to remember / wrap your head around is that depending on the perspective, a node $x$ acts as a central node or a neighboring node. In fact, in undirected graphs we store both edge directions (i.e. $(i, j)$ and $(j, i)$). From the central node perspective, `x_i`, x is collecting neighboring information to update its embedding. From a neighboring node perspective, `x_j`, x is passing its message information along the edge connecting it to a different central node.

  - `extra=(extra_i, extra_j)` represents additional information that we can associate with each node beyond its current feature embedding. In fact, we can include as many additional parameters of the form `param=(param_i, param_j)` as we would like. Again, we highlight that indexing with `_i` and `_j` allows us to differentiate central and neighboring nodes.

  The output of the `propagate` function is a matrix of node embeddings after the message passing process and has shape $[N, d]$.

2.
```
def message(x_j, ...):
```
The `message` function is called by propagate and constructs the messages from
neighboring nodes $j$ to central nodes $i$ for each edge $(i, j)$ in *edge_index*. This function can take any argument that was initially passed to `propagate`. Furthermore, we can again differentiate central nodes and neighboring nodes by appending `_i` or `_j` to the variable name, .e.g. `x_i` and `x_j`. Looking more specifically at the variables, we have:

  - `x_j` represents a matrix of feature embeddings for all neighboring nodes passing their messages along their respective edge (i.e. all nodes $j$ for edges $(i, j) \in \mathcal{E}$). Thus, its shape is $[|\mathcal{E}|, d]$!
  - In implementing GAT we will see how to access additional variables passed to propagate

  Critically, we see that the output of the `message` function is a matrix of neighboring node embeddings ready to be aggregated, having shape $[|\mathcal{E}|, d]$.

3.
```
def aggregate(self, inputs, index, dim_size = None):
```
Lastly, the `aggregate` function is used to aggregate the messages from neighboring nodes. Looking at the parameters we highlight:

  - `inputs` represents a matrix of the messages passed from neighboring nodes (i.e. the output of the `message` function).
  - `index` has the same shape as `inputs` and tells us the central node that corresponding to each of the rows / messages $j$ in the `inputs` matrix. Thus, `index` tells us which rows / messages to aggregate for each central node.

  The output of `aggregate` is of shape $[N, d]$.

For additional resources refer to the PyG documentation for implementing custom message passing layers: https://pytorch-geometric.readthedocs.io/en/latest/notes/create_gnn.html

## GraphSage Implementation

For our first GNN layer, we will implement the well known GraphSage ([Hamilton et al. (2017)](https://arxiv.org/abs/1706.02216)) layer!

For a given *central* node $v$ with current embedding $h_v^{l-1}$, the message passing update rule to tranform $h_v^{l-1} \rightarrow h_v^l$ is as follows:

\begin{equation}
h_v^{(l)} = W_l\cdot h_v^{(l-1)} + W_r \cdot AGG(\{h_u^{(l-1)}, \forall u \in N(v) \})
\end{equation}

where $W_1$ and $W_2$ are learanble weight matrices and the nodes $u$ are *neighboring* nodes. Additionally, we use mean aggregation for simplicity:

\begin{equation}
AGG(\{h_u^{(l-1)}, \forall u \in N(v) \}) = \frac{1}{|N(v)|} \sum_{u\in N(v)} h_u^{(l-1)}
\end{equation}

One thing to note is that we're adding a **skip connection** to our GraphSage implementation through the term $W_l\cdot h_v^{(l-1)}$.

Before implementing this update rule, we encourage you to think about how different parts of the formulas above correspond with the functions outlined earlier: 1) `forward`, 2) `message`, and 3) `aggregate`. As a hint, we are given what the aggregation function is (i.e. mean aggregation)! Now the question remains, what are the messages passed by each neighbor nodes and when do we call the `propagate` function?

Note: in this case the message function or messages are actually quite simple. Additionally, remember that the `propagate` function encapsulates the operations of / the outputs of the combined `message` and `aggregate` functions.

Lastly, $\ell$-2 normalization of the node embeddings is applied after each iteration.

For the following questions, DON'T refer to any existing implementations online.

```python
class GraphSage(MessagePassing):

    def __init__(
        self,
        in_channels,
        out_channels,
        normalize=True,
        bias=False,
        **kwargs
        ):

        super(GraphSage, self).__init__(**kwargs)

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.normalize = normalize

        self.lin_l = None
        self.lin_r = None

        ############################################################################
        # TODO: Your code here!
        # Define the layers needed for the message and update functions below.
        # self.lin_l is the linear transformation that you apply to embedding
        #            for central node.
        # self.lin_r is the linear transformation that you apply to aggregated
        #            message from neighbors.
        # Don't forget the bias!
        # Our implementation is ~2 lines, but don't worry if you deviate from this.

        self.lin_l = Linear(self.in_channels, self.out_channels, bias=bias)
        self.lin_r = Linear(self.in_channels, self.out_channels, bias=bias)

        ############################################################################

        self.reset_parameters()

    def reset_parameters(self):
        self.lin_l.reset_parameters()
        self.lin_r.reset_parameters()

    def forward(self, x, edge_index, size=None):
        """
        Implement message passing, as well as any post-processing (our update rule).

        Parameters
        ----------

        self : GraphSage object
            GraphSage.

        x : tensor of shape (num_nodes, in_channels)
            Data associated to each node.

        edge_index : tensor of shape (2, num_edges)
            Describe graph structure via edge list.

        size : tuple with shape
            Needed if edge_index is a matrix representing connections as
            (num_nodes, num_connected_nodes).

        See: https://github.com/pyg-team/pytorch_geometric/blob/master/torch_geometric/nn/conv/message_passing.py

        Returns
        -------

        out : tensor of shape (num_nodes, out_channels)
            Embedded data associated to each node.

        """
        out = None

        ############################################################################
        # TODO: Your code here!
        # Implement message passing, as well as any post-processing (our update rule).
        # 1. Call the propagate function to conduct the message passing.
        #    1.1 See the description of propagate above or the following link for more information:
        #        https://pytorch-geometric.readthedocs.io/en/latest/notes/create_gnn.html
        #    1.2 We will only use the representation for neighbor nodes (x_j), so by default
        #        we pass the same representation for central and neighbor nodes as x=(x, x).
        # 2. Update our node embedding with skip connection from the previous layer.
        # 3. If normalize is set, do L-2 normalization (defined in
        #    torch.nn.functional)
        #
        # Our implementation is ~5 lines, but don't worry if you deviate from this.

        ############################################################################

        # 1. Call the propagate function to conduct the message passing
        #       - propagate returns (num_nodes, output_channels)
        out = self.lin_r(self.propagate(edge_index, x=(x, x)))

        # 2. Update our node embedding with skip connection from the previous layer.
        out += self.lin_l(x)

        # 3. If normalize is set, do L-2 normalization (defined in torch.nn.functional)
        if self.normalize:
            out = F.normalize(out, p=2.0, dim=1, eps=1e-12, out=None)

        return out

    def message(self, x_j):
        """
        Messages from the neighbors.

        For GraphSage, the neighbors simply send their raw data.

        Parameters
        ----------

        x_j : tensor of shape (num_edges, in_channels)
            Data of the target node of each edge.

        Returns
        -------

        out : tensor of shape (num_edges, in_channels)
            Messages sent to the central node from the target node of each edge.

        """
        out = None

        ############################################################################
        # TODO: Your code here!
        # Implement your message function here.
        # Hint: Look at the formulation of the mean aggregation function, focusing on
        # what message each neighboring node passes.
        #
        # Our implementation is ~1 lines, but don't worry if you deviate from this.
        out = x_j

        ############################################################################

        return out

    def aggregate(self, inputs, index, dim_size=None):
        """
        Aggregates at the central node incoming messages from the neighbors.

        For GraphSage, the aggregation is an average over the data coming from
        the neighbors.

        Parameters
        ----------

        inputs : tensor of shape (E, in_channels)
            Messages sent to the central node from the target node of each edge.

        index : tensor of shape (num_edges)
            List o indexes of the (target) nodes receiving the message:
                        index = edge_list[1],
            since we are using the flow="source-to-target".

        dim_size : tuple with shape
            Tuple with shape of the aggregation.

        Returns
        -------

        out : tensor of shape (num_edges, out_channels)
            Aggregated messages at the central nodes.

        Example
        -------
          Imagine the following edge_index:

          1st row - [2, 3, 4, 5, 2, 4, 3]
          2nd row - [6, 7, 8, 2, 3, 9, 5]

          For central node 2, we need to aggregate msgs [6, 3]. This is what
          the scatter function does.
        """
        out = None

        # The axis along which to index number of nodes.
        node_dim = self.node_dim

        ############################################################################
        # TODO: Your code here!
        # Implement your aggregate function here.
        # See here as how to use torch_scatter.scatter:
        # https://pytorch-scatter.readthedocs.io/en/latest/functions/scatter.html#torch_scatter.scatter

        # Our implementation is ~1 lines, but don't worry if you deviate from this.
        out = torch_scatter.scatter(src=inputs, index=index, dim=node_dim, reduce="mean")

        ############################################################################

        return out

```

## Building Optimizers

This function has been implemented for you. **For grading purposes please use the default Adam optimizer**, but feel free to play with other types of optimizers on your own.

```python
import torch.optim as optim

def build_optimizer(args, params):

    # Get weight decay
    weight_decay = args.weight_decay

    # Filter parameters that need to compute gradients via PyTorch grad
    filter_fn = filter(lambda p : p.requires_grad, params)

    if args.opt == 'adam':
        optimizer = optim.Adam(filter_fn, lr=args.lr, weight_decay=weight_decay)

    elif args.opt == 'sgd':
        optimizer = optim.SGD(filter_fn, lr=args.lr, momentum=0.95, weight_decay=weight_decay)

    elif args.opt == 'rmsprop':
        optimizer = optim.RMSprop(filter_fn, lr=args.lr, weight_decay=weight_decay)

    elif args.opt == 'adagrad':
        optimizer = optim.Adagrad(filter_fn, lr=args.lr, weight_decay=weight_decay)

    if args.opt_scheduler == 'none':
        return None, optimizer

    elif args.opt_scheduler == 'step':
        scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=args.opt_decay_step, gamma=args.opt_decay_rate)

    elif args.opt_scheduler == 'cos':
        scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=args.opt_restart)

    return scheduler, optimizer
```

## Training and Testing

Here we provide you with the functions to train and test. **Please do not modify this part for grading purposes.**

```python
import time

import networkx as nx
import numpy as np
import torch
import torch.optim as optim
from tqdm import trange
import pandas as pd
import copy

from torch_geometric.datasets import TUDataset
from torch_geometric.datasets import Planetoid
from torch_geometric.data import DataLoader

import torch_geometric.nn as pyg_nn

import matplotlib.pyplot as plt

def train(dataset, args):

    print("Node task. Test set size:", np.sum(dataset[0]['test_mask'].numpy()))
    print(dataset[0])

    # Loading and batching data
    # PS: this batch stuff is doing nothing, the data set only has one graph!!
    test_loader = loader = DataLoader(dataset, batch_size=args.batch_size, shuffle=False)

    # Build model
    model = GNNStack(
        dataset.num_node_features,  # input dimension -- num_features
        args.hidden_dim,    # hidden dimension
        dataset.num_classes,    # output dimension -- num_classes
        args    # other arguments
        )

    # Get chosen schedulers and optmizers
    scheduler, opt = build_optimizer(args, model.parameters())

    # Train
    losses = []
    test_accs = []
    best_acc = 0
    best_model = None

    # Go through all training epocks
    for epoch in trange(args.epochs, desc="Training", unit="Epochs"):
        total_loss = 0
        model.train()

        for batch in loader:

            # Erase previous grads
            opt.zero_grad()

            # Get predicted value
            pred = model(batch)

            # Get true labels
            label = batch.y

            # Mask training
            pred = pred[batch.train_mask]
            label = label[batch.train_mask]

            # Compute loss
            loss = model.loss(pred, label)

            # Backpropagation
            loss.backward()

            # Update parameters
            opt.step()

            # Store total loss
            total_loss += loss.item() * batch.num_graphs

        # Average total loss
        total_loss /= len(loader.dataset)

        # Store loss over epochs
        losses.append(total_loss)

        # Every 10 epochs
        if epoch % 10 == 0:

            # Test the model: call test
            test_acc = test(test_loader, model)
            test_accs.append(test_acc)

          # Store best accuracy until now
            if test_acc > best_acc:
                best_acc = test_acc
                best_model = copy.deepcopy(model)

        else:

            # Repeat test value (?)
            test_accs.append(test_accs[-1])

    return test_accs, losses, best_model, best_acc, test_loader

def test(loader, test_model, is_validation=False, save_model_preds=False, model_type=None):
    test_model.eval()

    correct = 0

    # Note that Cora is only one graph!
    for data in loader:

        # Freeze the model
        with torch.no_grad():
            # max(dim=1) returns values, indices tuple; only need indices
            pred = test_model(data).max(dim=1)[1]
            label = data.y

        mask = data.val_mask if is_validation else data.test_mask

        # Node classification: only evaluate on nodes in test set
        pred = pred[mask]
        label = label[mask]

        if save_model_preds:
            print ("Saving Model Predictions for Model Type", model_type)

            data = {}
            data['pred'] = pred.view(-1).cpu().detach().numpy()
            data['label'] = label.view(-1).cpu().detach().numpy()

            df = pd.DataFrame(data=data)

            # Save locally as csv
            df.to_csv('CORA-Node-' + model_type + '.csv', sep=',', index=False)

        # Bizzarre, never have seen this .eq method
        correct += pred.eq(label).sum().item()

    # Number of validation or test samples
    total = 0
    for data in loader.dataset:
        total += torch.sum(data.val_mask if is_validation else data.test_mask).item()

    return correct / total

class objectview(object):
    def __init__(self, d):
        self.__dict__ = d

```

## Let's Start the Training!

We will be working on the CORA dataset on node-level classification.

This part is implemented for you. **For grading purposes, please do not modify the default parameters.** However, feel free to play with different configurations just for fun!

**Submit your best accuracy and loss on Gradescope.**

```python
if 'IS_GRADESCOPE_ENV' not in os.environ:
    for args in [
        {'model_type': 'GraphSage',
         'dataset': 'cora',
         'num_layers': 2,
         'heads': 1,
         'batch_size': 32,
         'hidden_dim': 32,
         'dropout': 0.5,
         'epochs': 500,
         'opt': 'adam',
         'opt_scheduler': 'none',
         'opt_restart': 0,
         'weight_decay': 5e-3,
         'lr': 0.01},
    ]:
        args = objectview(args)

        # Go through the different GNN models
        for model in ['GraphSage']:

            # Store evaluated model
            args.model_type = model

            # Match the dimension
            if model == 'GAT':
              args.heads = 2
            else:
              args.heads = 1

            # Get dataset
            if args.dataset == 'cora':
                dataset = Planetoid(root='/tmp/cora', name='Cora')
            else:
                raise NotImplementedError("Unknown dataset")

            # Train
            test_accs, losses, best_model, best_acc, test_loader = train(dataset, args)

            # Print best performance
            print("Maximum test set accuracy: {0}".format(max(test_accs)))
            print("Minimum loss: {0}".format(min(losses)))

            # Run test for our best model to save the predictions!
            test(test_loader, best_model, is_validation=False, save_model_preds=True, model_type=model)

            print()

            # Print the BEST :P
            plt.title(dataset.name)
            plt.plot(losses, label="training loss" + " - " + args.model_type)
            plt.plot(test_accs, label="test accuracy" + " - " + args.model_type)

        plt.legend()
        plt.show()

```

## Question 1.1: What is the maximum accuracy obtained on the test set for GraphSage? (10 points)

Running the cell above will show the results of your best model and save your best model's predictions to a file named *CORA-Node-GraphSage.csv*.

As we have seen before you can view this file by clicking on the *Folder* icon on the left side pannel. When you sumbit your assignment, you will have to download this file and attatch it to your submission.

```python
import networkx as nx
import matplotlib.pyplot as plt

from torch_geometric.utils.convert import to_networkx

# Passing from torch to nx
graph = to_networkx(dataset[0], to_undirected=~dataset[0].is_directed())

# Plotting graph with true label
plt.figure(figsize=(30,30))
nx.draw_networkx(graph, pos=nx.spring_layout(graph, seed=42), node_color=dataset[0].y, with_labels=False, cmap='Set2', node_size=20, width=2.0)
```

```python
# Plotting graph with predicted labels
plt.figure(figsize=(30,30))
nx.draw_networkx(graph, pos=nx.spring_layout(graph, seed=42), node_color=best_model(dataset[0]).max(dim=1)[1], with_labels=False, cmap='Set2', node_size=20, width=2.0)
```


# Problema de selección de portafolio con preferencias media-varianza

En la clase pasada hablamos acerca de:
- preferencias,
- funciones de utilidad,
- la actitud de los inversionistas de cara al riesgo,
- la aversión al riesgo, entre otros.

Todas ellas son piezas que necesitamos para responder la pregunta de ¿cómo un inversionista toma la decisión óptima de selección de portafolio?

En esta clase al fin estamos listos para ensamblar estos conceptos y escribir el problema de selección de portafolios.

En el camino aprenderemos acerca del concepto de **utilidad esperada**, que nos permite trabajar con incertidumbre en el modelado económico (una de las ideas más importantes en economía). Esta idea tiene más de 60 años, y básicamente dice que los individuos, cuando están de cara a incertidumbre, maximizan el valor esperado de su utilidad (solo cierto si somos homo economicus).

Además del concepto de utilidad esperada, aprenderemos acerca de **preferencias media-varianza**. Es decir, supondremos que los inversionistas toman decisiones basados en un tipo particular de preferencias.

Con lo anterior, estableceremos el problema de selección de portafolios.

**Objetivos:**

- ¿Qué es utilidad esperada?
- ¿Qué son las preferencias media-varianza?
- Funciones de utilidad media-varianza.
- Enunciado y solución del problema básico de selección de portafolio.

*Referencia:*
- Notas del curso "Portfolio Selection and Risk Management", Rice University, disponible en Coursera.
___

## 1. Utilidad esperada
- Básicamente combina las probabilidades de los resultados con cómo los inversionistas se sienten con dichos resultados.
- En otras palabras, la utilidad esperada multiplica la probabilidad de suceso de un evento con la utilidad que genera dicho evento.

Recordemos que las *funciones de utilidad* permiten a los inversionistas expresar cómo se sienten con los resultados, especialmente en los malos ratos.

Entonces la *utilidad esperada* es una herramienta que nos permite cuantificar cómo nos sentimos en nuestros malos momentos económicos, capturando el riesgo con la probabilidad de ocurrencia de dichos malos momentos.

Dado este marco de trabajo, cualquier decisión se puede escribir como la maximización de la utilidad esperada:
\begin{align}
\max_{\theta} & \quad E[U(W)], \\
\end{align}
mediante la escogencia de cierta variable $\theta$ (gastos, planes de ahorro, compra de activos, planes de producción, etc.).

Para nuestros propósitos, la variable de decisión serán los pesos o ponderaciones del portafolio.

Adicionalmente, en el contexto de la decisión de distribución de la riqueza entre activos, el problema de maximización tendrá comúnmente las siguientes restricciones:
- universo de inversión,
- posición en los activos dados.

**Ejemplo.**

Supongamos que un inversionista debe determinar la composición óptima de su portafolio, que contiene activos y bonos. Supongamos que son los únicos instrumentos disponibles.

Sean:
- $w_s$: peso o ponderación de activos en el portafolio,
- $w_b$: peso o ponderación de bonos en el portafolio,
- $r_s$: rendimiento de los activos, y
- $r_b$: rendimiento de los bonos.

De manera que podemos escribir el problema de selección de portafolios como la maximización de la utilidad esperade de nuestra riqueza futura, la cual dependerá de nuestros rendimientos:

\begin{align}
\max_{w_s,w_b} &\quad E[U(W)]\\
\text{s. a.}   &\quad W=W_0(1+w_sr_s+w_br_b)\\
               &\quad w_s+w_b=1\\
               &\quad w_s,w_b\geq0
\end{align}

Preguntas:
- ¿Qué significan las restricciones?
- Ya que tenemos planteado este problema básico, ¿qué haría falta para empezar a resolverlo?
___

## 2. Preferencias media-varianza

### 2.1. Utilidad media-varianza

Entonces, ¿qué funciones de utilidad deberíamos de usar en este problema de selección de portafolios?

- La respuesta es: **preferencias media-varianza**.
- Éstas serán representadas en términos de funciones de utilidad como: **utilidad media-varianza**.

Usamos la *utilidad media-varianza* en el problema de selección de portafolios dado que ésta decribe el "trade-off" entre riesgo y rendimiento que enfrentan los inversionistas. La *utilidad media-varianza* está dada por la siguiente expresión:

$$U=E[r_p]-\frac{1}{2}\gamma\sigma_p^2,$$

donde
- $E[r_p]$ es el rendimiento esperado del portafolio,
- $\sigma_p^2$ es la varianza del portafolio, y
- $\gamma$ es el coeficiente de aversión al riesgo.

#### Intuición acerca de la función de utilidad media-varianza:
- Sólo se preocupa por medias :) y varianzas :(.
- Incrementa con: rendimiento esperado del portafolio.
- Decrece con: varianza del portafolio.
- Malos tiempos: rendimientos son bajos y las volatilidades son altas.
- Conecta bastante bien con la teoría moderna de portafolios, la cual caracteriza los rendimientos con medias y varianzas únicamente.
- Criticada por su limitación: supone que los inversionistas sólo se preocupan por medias y varianzas.

### 2.2. Curvas de indiferencia

*¿Recuerdan las curvas de nivel que se ven en cálculo de varias variables?*
- Bien, acá nos servirán para representar la utilidad media-varianza gráficamente.
- En el contexto de utilidad media-varianza, las curvas de nivel se llaman **curvas de indiferencia**.

Dados ciertos niveles de utilidad $U_1>U_2>U_3$, las curvas de indiferencia relativas a estos niveles de utilidad, son los lugares geométricos en el espacio de rendimiento esperado vs. volatilidad representados por las siguientes expresiones

$$U_1=E[r_p]-\frac{1}{2}\gamma\sigma_p^2\Rightarrow E[r_p]=\frac{1}{2}\gamma\sigma_p^2+U_1,$$

$$U_2=E[r_p]-\frac{1}{2}\gamma\sigma_p^2\Rightarrow E[r_p]=\frac{1}{2}\gamma\sigma_p^2+U_2,$$

$$U_3=E[r_p]-\frac{1}{2}\gamma\sigma_p^2\Rightarrow E[r_p]=\frac{1}{2}\gamma\sigma_p^2+U_3.$$

**Gráficamente**

```python
# Importar numpy y pyplot
import numpy as np
import matplotlib.pyplot as plt
```

```python
# Coeficiente de aversión al riesgo (entre 1 y 10 comúnmente)
g = 5
# Niveles de utilidad
U1, U2, U3 = 0.05, 0.12, 0.19
# Vector de volatilidades (sugerido 1%-60%)
sp = np.linspace(0.01,  0.6)
# Curvas de indiferencia
Erp1 = U1 + 0.5 * g * sp**2
Erp2 = U2 + 0.5 * g * sp**2
Erp3 = U3 + 0.5 * g * sp**2
```

```python
# Gráfica
plt.figure(figsize=(6, 4))
plt.plot(sp, Erp1, label = f'$U_1$={U1}')
plt.plot(sp, Erp2, label = f'$U_2$={U2}')
plt.plot(sp, Erp3, label = f'$U_3$={U3}')
plt.grid()
plt.xlabel('Volatilidad $\sigma p$')
plt.ylabel('Rendimiento esperado sE[r p]$')
plt.legend(loc = 'best')
```

Bueno, ¿y porqué se llaman curvas de indiferencia?, ¿qué representa una cuva de indiferencia?

- Porque sobre una misma curva el nivel de utilidad es el mismo (es indiferente).
- Son todas las combinaciones de riesgo y rendimiento que producen un mismo nivel de utilidad.

Volviendo al problema de selección de portafolios, queremos la utilidad más alta.
- ¿Cuál de las anteriores curvas de indiferencia corresponde a la utilidad más alta?
- Intuitivamente, ¿porqué?
- Curvas de indiferencia para niveles de utilidad más altos, estarán...

Notamos además que las anteriores curvas de indiferencia son *paralelas* una con otra. Claro, las dibujamos con el mismo coeficiente de aversión al riesgo.

$$\frac{d E[r_p]}{d\sigma_p}=\frac{d }{d\sigma_p}\left[\frac{1}{2}\gamma\sigma_p^2+U\right]=\gamma\sigma_p.$$

¿Cómo cambian estas curvas para coeficientes de aversión al riesgo más altos?

```python
# Coeficientes de aversión al riesgo (entre 1 y 10 comúnmente)
g1, g2, g3 = 3, 5, 7
# Nivel de utilidad
U = 0.15
# Vector de volatilidades (sugerido 1%-60%)
sp = np.linspace(0.01, 0.6)
# Curvas de indiferencia
Erp1 = U + 0.5 * g1 * sp**2
Erp2 = U + 0.5 * g2 * sp**2
Erp3 = U + 0.5 * g3 * sp**2
```

```python
# Gráfica
plt.figure(figsize=(6, 4))
plt.plot(sp, Erp1, label = f'$\gamma_1$={g1}')
plt.plot(sp, Erp2, label = f'$\gamma_2$={g2}')
plt.plot(sp, Erp3, label = f'$\gamma_3$={g3}')
plt.grid()
plt.xlabel('Volatilidad $\sigma p$')
plt.ylabel('Rendimiento esperado sE[r p]$')
plt.legend(loc = 'best')

```

¿Cómo interpretamos las anteriores gráficas?, ¿qué pasa con las personas más aversas al riesgo?
- Se puede ver de dos maneras: para un mismo nivel de rendimiento esperado, una persona más aversa al riesgo soporta un nivel menor de riesgo; equivalentemente, para un mismo nivel de riesgo, una persona más aversa al riesgo requerirá un nivel de rendimiento esperado más alto.

Con todo lo anterior, el problema de selección de portafolios se puede plantear como *encontrar la curva de indeferencia más alta dado el conjunto de oportunidades de inversión y restricciones*.

## 3. Problema de selección de portafolios: una ilustración

Ahora ilustraremos el problema de selección de portafolios con algunos datos.
- Por ahora solo queremos ilustrar gráficamente cómo se resuelve este problema. Trabajar en la intuición.
- En las siguientes dos clases nos enfocaremos en cómo resolverlo analíticamente.

Acá tenemos el rendimiento medio anual y la volatilidad para dos instrumentos usando datos de EU: instrumentos de deuda (bonos) y acciones. Supondremos que el inversionista solo puede invertir en estas dos clases de instrumentos.

```python
# Importamos pandas
import pandas as pd
```

```python
# Datos
data = pd.DataFrame(index=['Stocks','Bonds', 'CorrSB'], columns=['Mean', 'Std'])
data['Mean'] = [0.119, 0.0591, 0.113]
data['Std'] = [0.1915, 0.0833, None]
data
```

      Mean
      Std

      Stocks
      0.1190
      0.1915

      Bonds
      0.0591
      0.0833

      CorrSB
      0.1130
      NaN

Entonces, ¿cuál es la distribución de riqueza óptima?, o más bien, ¿cuál es la composición óptima del portafolio para un inversionista dado su nivel de aversión al riesgo?

**Primero.** Recordamos que, para dos activos, podemos trazar la frontera de mínima varianza tomando todas las posibles combinaciones de los dos activos.

De nuevo, sean:
- $w_s=w$: peso o ponderación de activos en el portafolio,
- $w_b=1-w$: peso o ponderación de bonos en el portafolio,
- $r_s$: rendimiento de los activos, y
- $r_b$: rendimiento de los bonos.

Entonces

$$E[r_p]=wE[r_{s}]+(1-w)E[r_b]$$

$$\sigma_p^2=w^2\sigma_{s}^2+(1-w)^2\sigma_b^2+2w(1-w)\rho_{s,b}\sigma_s\sigma_b$$

```python
# Vector de w variando entre 0 y 1 con n pasos
w = np.linspace(0, 1, 100)
# Rendimientos esperados individuales
Es = data.loc['Stocks', 'Mean']
Eb = data.loc['Bonds', 'Mean']
# Volatilidades individuales
ss = data.loc['Stocks', 'Std']
sb = data.loc['Bonds', 'Std']
# Correlacion
rsb = data.loc['CorrSB', 'Mean']
```

```python
# Crear un DataFrame cuyas columnas sean rendimiento
# y volatilidad del portafolio para cada una de las w
# generadas
portafolios = pd.DataFrame(data={'w': w,
                                 'Media': w * Es + (1-w) * Eb,
                                 'Vol': ((w * ss)**2 + ((1-w) * sb)**2 + 2 * w *(1-w)* ss * sb * rsb)**0.5})
```

```python
# Gráfica
plt.figure(figsize=(6, 4))
plt.plot(portafolios['Vol'], portafolios['Media'], label = 'Portafolios')
plt.grid()
plt.xlabel('Volatilidad $\sigma p$')
plt.ylabel('Rendimiento esperado sE[r_p]$')
plt.legend(loc='best')
```

**Segundo.** Graficamos en la misma ventana, curvas de indiferencia.

```python
# Niveles de utilidad
U1, U2, U3 = 0.04, 0.06, 0.08
# Coeficiente de aversión al riesgo
g = 5
# Curvas de indiferencia
sp = np.linspace(0.08, 0.18)
Erp1 = U1 + 0.5 * g * sp**2
Erp2 = U2 + 0.5 * g * sp**2
Erp3 = U3 + 0.5 * g * sp**2
```

```python
# Gráfica
plt.figure(figsize=(6, 4))
plt.plot(portafolios['Vol'], portafolios['Media'], label = 'Portafolios')
plt.plot(sp, Erp1, label = f'$U_1$={U1}')
plt.plot(sp, Erp2, label = f'$U_2$={U2}')
plt.plot(sp, Erp3, label = f'$U_3$={U3}')
plt.grid()
plt.xlabel('Volatilidad $\sigma p$')
plt.ylabel('Rendimiento esperado sE[r p]$')
plt.legend(loc = 'best')
```

**Tercero.** La elección óptima está dada por la curva de indiferencia para el nivel de utilidad más alto que es tangente a la frontera media-varianza.
- Claramente, esta selección depende del coeficiente de aversión al riesgo.

```python
# Gráfica con zoom
# Gráfica
plt.figure(figsize=(6, 4))
plt.plot(portafolios['Vol'], portafolios['Media'], label = 'Portafolios')
plt.plot(sp, Erp1, label = f'$U_1$={U1}')
plt.plot(sp, Erp2, label = f'$U_2$={U2}')
plt.plot(sp, Erp3, label = f'$U_3$={U3}')
plt.grid()
plt.xlabel('Volatilidad $\sigma p$')
plt.ylabel('Rendimiento esperado sE[r p]$')
plt.legend(loc = 'best')
plt.axis([0.09, 0.11, 0.08, 0.09])
```

# Anuncios parroquiales

## 1. Quiz la siguiente clase.

Created with Jupyter by Esteban Jiménez Rodríguez.


**Note:**

You can download this demo as a Jupyter notebook [here](https://github.com/ejhigson/nestcheck/blob/master/docs/demos/quickstart_demo.ipynb) and run it interactively yourself. The PolyChord nested sampling run data it uses can be downloaded at [https://github.com/ejhigson/nestcheck_demo_data](https://github.com/ejhigson/nestcheck_demo_data) (this also contains runs using a few other likelihoods).

# Quickstart demo

This is a brief demonstration covering loading nested sampling run data, performing calculations, running error analysis and diagnostic tests, and making plots. For detailed explanations of the diagnostic tests and plots, see the papers which introduce them ([Higson et al. 2018](https://doi.org/10.1214/17-BA1075), [Higson et al. 2019](https://doi.org/10.1093/mnras/sty3090)).

More information about `nestcheck`'s code and and functionality can be found in the [documentation](http://nestcheck.readthedocs.io). For more examples of ``nestcheck``'s usage, including on a variety of likelihoods, see code used to make the results and diagrams in the diagnostic tests paper ([Higson et al. 2019](https://doi.org/10.1093/mnras/sty3090)) (available at [https://github.com/ejhigson/diagnostic](https://github.com/ejhigson/diagnostic)).

## Loading nested sampling runs

For this demo we will use some PolyChord nested sampling runs with a simple 2-dimensional Gaussian likelihood

\begin{equation}
\mathcal{L}(\theta) = \frac{1}{2\pi \sigma^2} \exp\left(\frac{{|\theta|}^2}{2 \sigma^2}\right),
\end{equation}

with $\sigma=0.5$ and a uniform prior on each parameter in $[-10, 10]$. For more information about the dictionary format and keys `nestcheck` uses to store nested sampling runs, see the [API documentation](http://nestcheck.readthedocs.io/en/latest/api.html).

For example, a PolyChord run can be loaded as follows:

```python
import nestcheck.data_processing

base_dir = 'polychord_chains'  # directory containing run (PolyChord's 'base_dir' setting)
file_root = 'gaussian_2d_100nlive_5nrepeats_1'  # output files' name root (PolyChord's 'file_root' setting)
run = nestcheck.data_processing.process_polychord_run(file_root, base_dir)
```

`nestcheck.data_processing` also has functions for loading nested sampling data from a variety of nested sampling software packages (including ``MultiNest``), and you can add your own method to load data from other sources.

Data from multiple runs can be loaded and processed together (with optional parallelisation):

```python
file_roots = ['gaussian_2d_100nlive_5nrepeats_' + str(i) for i in range(1, 11)]
run_list = nestcheck.data_processing.batch_process_data(
    file_roots, base_dir=base_dir, parallel=True,
    process_func=nestcheck.data_processing.process_polychord_run)
```

Failed to display Jupyter Widget of type HBox.

  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

## Evidence and parameter estimation calculations from runs

Nested sampling runs in the `nestcheck` format can be easily used to make posterior inferences (see `estimators.py` for more example functions)

```python
import nestcheck.estimators as e

print('The log evidence estimate using the first run is',
      e.logz(run_list[0]))
print('The estimateed the mean of the first parameter is',
      e.param_mean(run_list[0], param_ind=0))
```

    The log evidence estimate using the first run is -5.941675954408192
    The estimateed the mean of the first parameter is 0.003249535192363336

You can get a pandas DataFrame of results for a list of quantities and a list of runs as follows:

```python
import nestcheck.diagnostics_tables

estimator_list = [e.logz, e.param_mean, e.param_squared_mean, e.r_mean]
# Use nestcheck's stored LaTeX format estimator names
estimator_names = [e.get_latex_name(est) for est in estimator_list]
vals_df = nestcheck.diagnostics_tables.estimator_values_df(
    run_list, estimator_list, estimator_names=estimator_names)
vals_df
```

Failed to display Jupyter Widget of type HBox.

  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

      $\mathrm{log} \mathcal{Z}$
      $\overline{\theta_{\hat{1}}}$
      $\overline{\theta^2_{\hat{1}}}$
      $\overline{|\theta|}$

      run

      0
      -5.934184
      -0.000376
      0.242457
      0.625360

      1
      -5.941676
      0.003250
      0.238832
      0.594131

      2
      -5.997774
      -0.017127
      0.263610
      0.629681

      3
      -5.848036
      -0.026617
      0.279740
      0.654915

      4
      -5.761255
      -0.004223
      0.230528
      0.608089

      5
      -5.874927
      -0.012669
      0.224504
      0.598055

      6
      -5.784370
      0.011934
      0.249281
      0.599508

      7
      -5.880606
      0.019924
      0.253596
      0.616933

      8
      -6.297974
      0.031247
      0.264749
      0.663862

      9
      -5.863086
      -0.031087
      0.257504
      0.648641

## Bootstrap sampling error estimates

The sampling error on calculations from nested sampling runs can be estimated using the bootstrap approach method introduced in [Higson et al. (2018)](https://doi.org/10.1214/17-BA1075) Section 4 (see the paper for more details).

```python
import pandas as pd
import nestcheck.error_analysis

bs_error_df = pd.DataFrame(columns=estimator_names)
for i, run in enumerate(run_list[:2]):  # just use the first two runs as an example
    bs_error_df.loc[i] = nestcheck.error_analysis.run_std_bootstrap(run, estimator_list, n_simulate=100)
bs_error_df.index.name = 'run'
print('Run boostrap error estimates:')
bs_error_df
```

    Run boostrap error estimates:

      $\mathrm{log} \mathcal{Z}$
      $\overline{\theta_{\hat{1}}}$
      $\overline{\theta^2_{\hat{1}}}$
      $\overline{|\theta|}$

      run

      0
      0.193681
      0.026585
      0.016430
      0.022041

      1
      0.214706
      0.025536
      0.019318
      0.020765

## Diagrams of uncertainties on posterior distributions using bootstrap resamples

Bootstrap resamples of nested sampling runs can be used to plot numerical uncertainties on whole posterior distributions (rather than just scalar quantities) using `nestcheck`'s  `bs_param_dists` function.

```python
import nestcheck.plots
%matplotlib inline

fig = nestcheck.plots.bs_param_dists(run_list[:2])
```

Here the dashed dark red and dark blue lines mark the estimates of the mean of each parameter for the red and blue runs respectively. For a detailed explanation of this type of diagram and its uses, see [Higson et al. (2019)](https://doi.org/10.1093/mnras/sty3090) Section 4.1 and Figure 3.

## Diagrams of samples in log X

The `param_logx_diagram` function plots nested sampling diagrams of the type proposed in [Higson et al. (2019)](https://doi.org/10.1093/mnras/sty3090) Section 4.2 and shown in Figures 4 and 5.

```python
fig = nestcheck.plots.param_logx_diagram(run_list[0], logx_min=-15)
```

These diagrams illustrate the nested sampling algorithm's exponential compression of the prior by plotting samples in $\log X$, where $X(\mathcal{L}) \in [0, 1]$ is the fraction of the prior with likelihood greater than $\mathcal{L}$. The algorithm iterates towards higher likelihoods (towards lower $\log X$ values). The plots on the left are similar the distributions plotted in the previous cell, and the top right plot shows the relative posterior mass at each $\log X$ value. See [Higson et al. (2019)](https://doi.org/10.1093/mnras/sty3090) Section 4.2 for a detailed explanation and more examples.

## Calculating errors due to implementation-specific effects

Nested sampling software used for practical problems, such as MultiNest and PolyChord, uses numerical techniques to produce approximately uncorrelated samples within some iso-likelihood contour. However, for challenging problems - such as those with multimodal or degenerate posteriors - the software may fail to do this accurately, causing additional errors due to *implementation-specific effects* (see [Higson et al. 2019](https://doi.org/10.1093/mnras/sty3090) for more details).

The error due to implementation-specific effects can be estimated with `nestcheck` using the method described in Section 5 of [Higson et al. (2019)](https://doi.org/10.1093/mnras/sty3090). This involves computing the standard deviation of results from repeated calculations, and therefore requires multiple runs (the more runs used, the more precise the results).

```python
df = nestcheck.diagnostics_tables.run_list_error_summary(run_list, estimator_list, estimator_names, 100)
df
```

Failed to display Jupyter Widget of type HBox.

  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

Failed to display Jupyter Widget of type HBox.

  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

      $\mathrm{log} \mathcal{Z}$
      $\overline{\theta_{\hat{1}}}$
      $\overline{\theta^2_{\hat{1}}}$
      $\overline{|\theta|}$

      calculation type
      result type

      values mean
      value
      -5.918389
      -0.002574
      0.250480
      0.623917

      uncertainty
      0.047744
      0.006330
      0.005339
      0.007926

      values std
      value
      0.150980
      0.020019
      0.016884
      0.025065

      uncertainty
      0.035586
      0.004718
      0.003980
      0.005908

      bootstrap std mean
      value
      0.217638
      0.026168
      0.018131
      0.022129

      uncertainty
      0.010354
      0.000704
      0.000481
      0.000441

      implementation std
      value
      0.000000
      0.000000
      0.000000
      0.011771

      uncertainty
      0.055952
      0.009373
      0.010334
      0.014222

      implementation std frac
      value
      0.000000
      0.000000
      0.000000
      0.469616

      uncertainty
      6.940037
      2.363760
      1.141478
      1.763814

The 2-dimensional Gaussian likelihood is unimodal and easy for PolyChord to sample, so as expected we see that the standard deviation of the result values is close to the mean bootstrap standard deviation. Consequently the estimated errors due to implementation-specific effects are low.

## Tests for implementation specific effects using only 2 nested sampling runs

It is impossible to tell *a priori* if implementation-specific effects are present in a single nested sampling run without some additional knowledge of what the results should be. However diagnostic tests to determine if significant implementation-specific effects are present using only two nested sampling runs are proposed in [Higson et al. (2019)](https://doi.org/10.1093/mnras/sty3090).

1. The first test divides nested sampling runs into their constituent single live point runs ("threads") and assesses whether threads within each run are correlated with each other using the [Kolmogorov–Smirnov test](https://en.wikipedia.org/wiki/Kolmogorov-Smirnov_test). This yields a $p$-value, with $p \approx 0$ indicating implementation-specific effects are almost certainly present.
2. Secondly, the statistical distance between the uncertainty distributions (calculated from bootstrap replications) of quantities such as $\log X$ or parameter means for the two runs can be calculated. We use the Kolmogorov–Smirnov statistic as a distance measure; if this is close to 1 then there little or is no overlap between the distributions and additional errors from implementation-specific effects are likely present.

For a full explanation see the diagnostic tests paper ([Higson et al. 2019)](https://doi.org/10.1093/mnras/sty3090)).

These statistcs can be computed for pairs of runs using `nestcheck` as:

```python
# perform error analysis on two runs
error_vals_df = nestcheck.diagnostics_tables.run_list_error_values(
    run_list[:2], estimator_list, estimator_names, thread_pvalue=True, bs_stat_dist=True, n_simulate=100)
# select only rows containing pairwise tests to output
error_vals_df.loc[pd.IndexSlice[['thread ks pvalue', 'bootstrap ks distance'], :], :]
```

Failed to display Jupyter Widget of type HBox.

  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

Failed to display Jupyter Widget of type HBox.

  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

Failed to display Jupyter Widget of type HBox.

  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

      $\mathrm{log} \mathcal{Z}$
      $\overline{\theta_{\hat{1}}}$
      $\overline{\theta^2_{\hat{1}}}$
      $\overline{|\theta|}$

      calculation type
      run

      thread ks pvalue
      (1, 0)
      0.343886
      0.556017
      0.67662
      0.260553

      bootstrap ks distance
      (1, 0)
      0.190000
      0.290000
      0.58000
      0.570000

As expected, the $p$-values are not particularly low nor are the KS distances particularly high - indicating there are no significant implementation-specific effects for the simple 2-dimensional Gaussian.

If more than two runs are provided, the above function will calculate the diagnostics for each pairwise combination.