<p align="center">

</p>

## Subsurface Machine Learning with Tree-based Gradient Boosting Regressor Methods

### Tree-based Gradient Boosting for Subsurface Modeling in Python

#### Michael Pyrcz, Associate Professor, University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1) | [GeostatsPy](https://github.com/GeostatsGuy/GeostatsPy)

### PGE 383 Exercise: Simple Demonstration of Tree-based Gradient Boosting Regressors for Subsurface Modeling in Python

Here's a simple workflow, demonstration of tree-based gradient boosting for subsurface modeling workflows. This should help you get started with building subsurface models that data analytics and machine learning. Here's some basic details about tree-based gradient boosting.

#### Boosting Methods

Boosting applies mulitple week learners to build a stronger learner

* a weak learner is one that offers predictions just marginally better than random selection

I'll explain the method with words and then with equations.

* build a simple model with a high error rate, the model can be be quite inaccurate, but moves in the correct direction

* calculate the error from the model

* fit another model to the error

* calculate the error from this addition of the first and second model

* repeat until the desired accuracy is obtained or some other stopping criteria

The general workflow for predicting $Y$ from $X_1,\ldots,X_m$ is:

* build a week learner to predict $Y$ from $X_1,\ldots,X_m$, $\hat{F}_k(X)$ from the training data $x_{i,j}$.

* loop over number of desired estimators, $k = 1,\ldots,K$

    1. calculate the residuals at the the training data, $h_k(x_{i}) = y_i - \hat{F}_k(x_{i})$

    2. fit another week learner to predict $h_k$ from $X_1,\ldots,X_m$, $\hat{F}_k(X)$ from the training data $x_{i,j}$.

We have a hiearchy of simple $K$ models.

* each model builds on the previous to improve the accuracy

Our regression estimator is the summation over the $K$ simple models.

\begin{equation}
\hat{Y} =\sum_{k=1}^{K} F_k(X_1,\ldots,X_m)
\end{equation}

#### Gradient Boosting Methods

If you look at the previous method, it becomes clear that it could be mapped to a gradient descent problem

At each step, $k$, a model is being fit, then the error is calculated, $h_k(X_1,\ldots,X_m).

We can assign a loss function

\begin{equation}
L\left(y,F(X)\right) = \frac{\left(y - F(X)\right)^2}{2}
\end{equation}

So we want to minimize the $\ell2$ loos function:

\begin{equation}
J = \sum_{i=1}^{n} L\left(y_i, F_k(X) \right)
\end{equation}

by adjusting our model result over our training data $F(x_1), F(x_2),\ldots,F(x_n)$.

We can take the partial derivative of the error vs. our model.

\begin{equation}
\frac{\partial J}{\partial F(x_i)} = F(x_i) - y_i
\end{equation}

We can interpret the residuals as negative gradients.

\begin{equation}
y_i - F(x_i) = -1 \frac{\partial J}{\partial F(x_i)}
\end{equation}

So now we have  a gradient descent problem:

\begin{equation}
F_{k+1}(X_i) = F_k(X_i) + h(X_i)
\end{equation}

\begin{equation}
F_{k+1}(X_i) = F_k(X_i) + y_i - F_k(X_i)
\end{equation}

\begin{equation}
F_{k+1}(X_i) = F_k(X_i) - 1 \frac{\partial J}{\partial F_k(X_i)}
\end{equation}

Of the general form:

\begin{equation}
\phi_{k+1} = \phi_k - \rho \frac{\partial J}{\partial \phi_k}
\end{equation}

where $phi_k$ is the current state, $\rho$ is the learning rate, $J$ is the loss function, and $\phi_{k+1}$ is the next state of our estimator.

If we consider our residual at training data to be a gradient then we are performing gradient descent.

* fitting a series of models to negative gradients

By approaching the problem as a gradient decent problem we are able to apply a variety of loss functions

* $\ell2$ is our $\frac{\left(y - F(X)\right)^2}{2}$ is practical, but is not robust with outliers

\begin{equation}
- 1 \frac{\partial J}{\partial F_k(X_i)} = y_i - F_k(X_i)
\end{equation}

* $\ell1$ is our $|y - F(X)|$ is more robust with outliers

\begin{equation}
- 1 \frac{\partial J}{\partial F_k(X_i)} = sign(y_i - F_k(X_i))
\end{equation}

* there are others like Huber Loss

#### Classification

The response is a finite set of possible catergories.

* For each training data the truth is 100% probability in the observed category and 0% otherwise

* Estimate the probability of each category with the a decision tree

* Use a measure of difference between the true and estimated distributions as the loss function to minimize

#### Tree-based Boosting Methods

Machine learning method for supervised learning for classification and regression analysis. Here are some key aspects of random forest.

**Prediction**

* estimate a function $\hat{f}$ such that we predict a response feature $Y$ from a set of predictor features $X_1,\ldots,X_m$.

* the prediction is of the form $\hat{Y} = \hat{f}(X_1,\ldots,X_m)$

**Supervised Learning**

* the response feature label, $Y$, is available over the training and testing data

**Hiearchical, Binary Segmentation of the Feature Space**

The fundamental idea is to divide the predictor space, $𝑋_1,\ldots,X_m$, into $J$ mutually exclusive, exhaustive regions

* **mutually exclusive** – any combination of predictors only belongs to a single region, $R_j$

* **exhaustive** – all combinations of predictors belong a region, $R_j$, regions cover entire feature space (range of the variables being considered)

For every observation in a region, $R_j$, we use the same prediction, $\hat{Y}(R_j)$

For example predict production, $\hat{Y}$, from porosity, ${X_1}$

* given the data within a mD feature space, $X_1,\ldots,X_m$, find that boundary maximizes the gap between the two categories

* new cases are classified based on where they fall relative to this boundary

**Proceedure for Tree Construction**

The tree is constructed from the top down.  We begin with a sigle region that covers the entire feature space and then proceed with a sequence of splits.

* **Scan All Possible Splits** over all regions and over all features.

* **Greedy Optimization**  The method proceeds by finding the first segmentation (split) in any feature that minimizes the residual sum of squares of errors over all the training data $y_i$ over all of the regions $j = 1,\ldots,J$.

\begin{equation}
RSS = \sum^{J}_{j=1} \sum_{i \in R_j} (y_i - \hat{y}_{R_j})^2
\end{equation}

* **Stopping Criteria** is typically based on minimum number of training data in each region for a robust estimation and / or minimum reduction in RSS for the next split

**Interpretability**

Compared to decision trees, the ensemble methods have reduced interpretability.  One tool to improve model interpretability is feature importance.

We calculate variable importance through calculating the average of:

* residual sum of square reduction for all splits involving each predictor feature for regression

* the decrease in the Gini index for all splits involving each predictor feature for classification

Both are standardized to sum to 1.0 over the features.

##### Applications to Subsurface Modeling

We will predict unconventional well production from a single petrophysical and geomechanical predictor feature

##### Why Cover Ensemble Methods with Tree-based Gradient Boosting?

Here's some thoughts:

* build on from easy to understand decision trees
* demonstrate reduction in model variance through an ensemble approach
* random forest is quite powerful and is a top performing machine learning method in various types of problems

#### Workflow Goals

Learn the basics of ensemble tree methods in python to segment facies given petrophysical properties. This includes:

* Loading and visualizing sample data
* Trying out bagging tree and random forest
* Test and observe the model behavoir and prove concepts

#### Objective

In the PGE 383: Stochastic Subsurface Modeling class I want to provide hands-on experience with building subsurface modeling workflows. Python provides an excellent vehicle to accomplish this. I have coded a package called GeostatsPy with GSLIB: Geostatistical Library (Deutsch and Journel, 1998) functionality that provides basic building blocks for building subsurface modeling workflows.

The objective is to remove the hurdles of subsurface modeling workflow construction by providing building blocks and sufficient examples. This is not a coding class per se, but we need the ability to 'script' workflows working with numerical methods.

#### Getting Started

Here's the steps to get setup in Python with the GeostatsPy package:

1. Install Anaconda 3 on your machine (https://www.anaconda.com/download/).
2. From Anaconda Navigator (within Anaconda3 group), go to the environment tab, click on base (root) green arrow and open a terminal.
3. In the terminal type: pip install geostatspy.
4. Open Jupyter and in the top block get started by copy and pasting the code block below from this Jupyter Notebook to start using the geostatspy functionality.

You will need to copy the data file to your working directory.  They are available here:

* Tabular data - 12_sample_data.csv found [here](https://github.com/GeostatsGuy/GeoDataSets/blob/master/12_sample_data.csv).

There are exampled below with these functions. You can go here to see a list of the available functions, https://git.io/fh4eX, other example workflows and source code.

#### Import Required Packages

We will also need some standard packages. These should have been installed with Anaconda 3.

#### Load the required libraries

The following code loads the required libraries.

```python
import os                                                   # to set current working directory
import math                                                 # basic calculations like square root
from sklearn import tree                                    # tree program from scikit learn (package for machine learning)
from sklearn.tree import _tree                              # for accessing tree information
from sklearn import metrics                                 # measures to check our models
from sklearn.tree import export_graphviz                    # graphical visualization of trees
from sklearn.preprocessing import StandardScaler            # standardize variables to mean of 0.0 and variance of 1.0

from sklearn.model_selection import cross_val_score         # cross validation methods
from sklearn.tree import DecisionTreeRegressor              # decision tree method
from sklearn import model_selection
from sklearn.ensemble import GradientBoostingRegressor      # tree-based gradient boosting

import pandas as pd                                         # DataFrames and plotting
import pandas.plotting as pd_plot
import numpy as np                                          # arrays and matrix math
import matplotlib.pyplot as plt                             # plotting
from subprocess import check_call
import geostatspy.GSLIB as GSLIB                            # geostatistics and spatial methods
```

If you get a package import error, you may have to first install some of these packages. This can usually be accomplished by opening up a command window on Windows and then typing 'python -m pip install [package-name]'. More assistance is available with the respective package docs.

#### Declare functions

Let's define a couple of functions to streamline plotting correlation matrices and visualization and test our regression models.

#### Set the working directory

I always like to do this so I don't lose files and to simplify subsequent read and writes (avoid including the full address each time).

```python
os.chdir("c:/PGE383")                                       # set the working directory
```

#### Generate a Data Set

Let's make a simple synthetic dataset with a single predictor feature and response feature.

* we will build an additive model with a random component and a cyclic component

```python
nsample = 20                                                # number of data values
np.random.seed(63079)                                       # random number seed for a consistent model
wt_random = 0.3                                             # weights for the random and cyclic components (sum to 1.0)
wt_cycle = 0.7
x = np.linspace(0,100,20)                                   # make a set of values to evaluate our models
Y = (wt_random*np.random.rand(nsample) + wt_cycle*np.sin(x/10)+1.0)*0.5

indices = np.random.permutation(x.shape[0])                 # get the indicies for randomize the order of the values
training_idx, test_idx = indices[:int(nsample*4/5)], indices[int(nsample*4/5):] # extract 80% training and 20% testing
x_train, x_test = x[training_idx], x[test_idx]
Y_train, Y_test = Y[training_idx], Y[test_idx]
```

Let's visualize our synthetic dataset.

```python
plt.subplot(121)                                            # plot jackknife results for all cases
plt.scatter(x_train,Y_train,s=None, c='red', marker=None, cmap=None, norm=None, vmin=None, vmax=None, alpha=1.0, linewidths=0.3, verts=None, edgecolors="black", label = "Training")
plt.scatter(x_test,Y_test,s=None, c='blue', marker=None, cmap=None, norm=None, vmin=None, vmax=None, alpha=1.0, linewidths=0.3, verts=None, edgecolors="black", label = "Testing")

plt.title('Simple Dataset'); plt.xlabel('Predictor Feature'); plt.ylabel('Response Feature')
plt.xlim(0,100); plt.ylim(0,1.0); plt.legend(loc='lower right')

plt.subplots_adjust(left=0.0, bottom=0.0, right=4.5, top=0.8, wspace=0.2, hspace=0.2)
plt.show()
```

#### Build a Tree-based Boosting Model Sequentially

Let's build a tree based boosting model sequentially.

* Visualize the model fitting to the training data in original feature space

* Visualize the model fitting to the training data in the feature residual space

#### Tree-based Boosting Model in Original Feature Space

We will start in original feature space for $1,\dots,12$ trees.

```python
import warnings
warnings.filterwarnings('ignore')                           # suppress warnings

ntrees = 12                                                 # number of estimators / trees for our boosting model

params = {
    'loss': 'ls',                                           # L2 Norm - least squares
    'max_depth': 1,
    'learning_rate': 1,
    'criterion': 'mse'                                      # tree construction criteria is mean square error over training
}

num_trees = np.linspace(1,12,ntrees)                        # build a list of numbers of trees
boosting_models = []; score = []; pred = []                 # arrays for storage of models and model summaries
x_pred = np.linspace(0,100,100)

cmap = plt.get_cmap('gnuplot')
colors = [cmap(i) for i in np.linspace(0, 1, ntrees)]

index = 1
print('Complete of ' + str(len(num_trees)) + ': ', end =" ")
for num_tree in num_trees:                                  # loop over number of trees
    boosting_models.append(GradientBoostingRegressor(n_estimators=int(num_tree),**params))
    boosting_models[index-1].fit(X = x_train.reshape(-1, 1), y = Y_train)
    score.append(boosting_models[index-1].score(X = x_test.reshape(-1, 1), y = Y_test))
    Y_pred = boosting_models[index-1].predict(x_pred.reshape(-1, 1))
    plt.subplot(4,3,index)
    plt.scatter(x_train,Y_train,s=None, c='black', marker=None, cmap=None, norm=None, vmin=None, vmax=None, alpha=0.2, linewidths=0.3, verts=None, edgecolors="black", label = "Training")
    plt.plot(x_pred,Y_pred, c=colors[index-1], alpha=0.7, label = "Testing")
    plt.title('Boosting with ' + str(int(num_trees[index-1])) + ' Trees' ); plt.xlabel('Predictor Feature'); plt.ylabel('Response Feature')
    plt.xlim(0,100); plt.ylim(0,1.0);
    print(str(index)+ ', ', end =" ")
    index = index + 1

#plt.legend(loc='lower right')

plt.subplots_adjust(left=0.0, bottom=0.0, right=1.8, top=2.5, wspace=0.3, hspace=0.4)
plt.show()
```

Observe the additive nature of the boosting model.

* this is due to the fitting of residuals from the previous model

* we are fitting the model error from the previous models

We can actually visualize the model fits with the training data and model residuals.

#### Tree-based Boosting Model in Feature Residual Space

We will start in feature residual space for $1,\dots,12$ trees.

```python
import warnings
warnings.filterwarnings('ignore')                           # suppress warnings

ntrees = 12

params = {
    'loss': 'ls',                                           # L2 Norm - least squares
    'max_depth': 1,
    'learning_rate': 1,
    'criterion': 'mse'                                      # tree construction criteria is mean square error over training
}

num_trees = np.linspace(1,12,ntrees)                              # build a list of numbers of trees
boosting_models = []; score = []; pred = []                 # arrays for storage of models and model summaries
x_pred = np.linspace(0,100,100)

cmap = plt.get_cmap('gnuplot')
colors = [cmap(i) for i in np.linspace(0, 1, ntrees)]
Y_zero = np.zeros(100)

index = 1
print('Complete of ' + str(len(num_trees)) + ': ', end =" ")
for num_tree in num_trees:                                  # loop over number of trees
    boosting_models.append(GradientBoostingRegressor(n_estimators=int(num_tree),**params))
    boosting_models[index-1].fit(X = x_train.reshape(-1, 1), y = Y_train)
    Y_pred = boosting_models[index-1].predict(x_pred.reshape(-1, 1))
    if index == 1:
        Y_pred_prev = np.zeros(100)
    else:
        Y_pred_prev = boosting_models[index-2].predict(x_pred.reshape(-1, 1))
    Y_train_pred = boosting_models[index-1].predict(x_train.reshape(-1, 1))
    if index == 1:
        Y_train_pred_prev = np.zeros(x_train.shape[0])
    else:
        Y_train_pred_prev = boosting_models[index-2].predict(x_train.reshape(-1, 1))
    plt.subplot(4,3,index)
    plt.scatter(x_train,Y_train-Y_train_pred_prev,s=None, c='black', marker=None, cmap=None, norm=None, vmin=None, vmax=None, alpha=0.2, linewidths=0.3, verts=None, edgecolors="black", label = "Training")
    plt.plot(x_pred,Y_pred-Y_pred_prev, c=colors[index-1], alpha=0.7, label = "Testing")
    plt.title('Boosting with ' + str(int(num_trees[index-1])) + ' Trees' ); plt.xlabel('Predictor Feature'); plt.ylabel('Response Feature')
    if index == 1:
        plt.xlim(0,100); plt.ylim(0,1.0);
    else:
        plt.xlim(0,100); plt.ylim(-0.5,0.5);
        plt.plot(x_pred,Y_zero, c='black', alpha=0.7, linestyle='dashed', label = "Testing")
    print(str(index)+ ', ', end =" ")
    index = index + 1

plt.subplots_adjust(left=0.0, bottom=0.0, right=1.8, top=3.5, wspace=0.3, hspace=0.4)
plt.show()
```

Observations:

* See the initial model and then working with residuals.

* See the residual from $k$ fit by the $k+1$ model

* See the reduction in residual magnitude as the number boosting trees increases

#### Comments

This was a simple demonstration of tree-based boosting.

I have other demonstrations on the basics of working with DataFrames, ndarrays, univariate statistics, plotting data, declustering, data transformations, trend modeling and many other workflows available at https://github.com/GeostatsGuy/PythonNumericalDemos and https://github.com/GeostatsGuy/GeostatsPy.

I hope this was helpful,

*Michael*

#### The Author:

### Michael Pyrcz, Associate Professor, University of Texas at Austin
*Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions*

With over 17 years of experience in subsurface consulting, research and development, Michael has returned to academia driven by his passion for teaching and enthusiasm for enhancing engineers' and geoscientists' impact in subsurface resource development.

For more about Michael check out these links:

#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

#### Want to Work Together?

I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.

* Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I'd be happy to drop by and work with you!

* Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!

* I can be reached at mpyrcz@austin.utexas.edu.

I'm always happy to discuss,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin

#### More Resources Available at: [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

```python

```


```python
from unsys import *
import numpy as np
import sympy.physics.quantum as spq
```

```python
expr = spq.Ket(0) + spq.Ket('a')
expr2 = spq.Ket(1) + spq.Ket('a')
s1 = expr.free_symbols
s2 = expr2.free_symbols

print(len(s1.intersection(s2)))
```

    1

```python
qs = QuditSystem(2,2)

qs.postSelect(1,spq.Ket(1))
qs.draw()

```

```python
qs = QuditSystem(3,2)
qs.draw()

print(qs.getStatevectorIndices())
```

```python
import itertools as itertools
list1 = [1,2,3]
list2 = [4,5]

flat = []
all_combinations = [list(zip(each_permutation, list2)) for each_permutation in itertools.permutations(list1, len(list2))]
print(all_combinations)
for e in all_combinations:
    for ts in e:
        flat.append(ts)

print(set(flat))
```

    [[(1, 4), (2, 5)], [(1, 4), (3, 5)], [(2, 4), (1, 5)], [(2, 4), (3, 5)], [(3, 4), (1, 5)], [(3, 4), (2, 5)]]
    {(2, 4), (3, 4), (1, 5), (1, 4), (2, 5), (3, 5)}

```python

```


#  CS109B Data Science 2: Advanced Topics in Data Science

## Lecture 10 - Bayesian Analysis and Introduction to pyMC3

**Harvard University**

**Spring 2021**

**Instructors:** Pavlos Protopapas, Mark Glickman, and Chris Tanner

**Additional Instructor and content:** Eleni Angelaki Kaxiras

---

```python
## RUN THIS CELL TO PROPERLY HIGHLIGHT THE EXERCISES
import requests
from IPython.core.display import HTML
styles = requests.get("https://raw.githubusercontent.com/Harvard-IACS/2019-CS109B/master/content/styles/cs109.css").text
HTML(styles)
```

```python
import pymc3 as pm
# from pymc3 import summary
import arviz as az
from matplotlib import gridspec
# Ignore a common pymc3 warning that comes from library functions, not our code.
# Pymc3 may throw additional warnings, but other warnings should be manageable
# by following the instructions included within the warning messages.
import warnings
messages=[
    "Using `from_pymc3` without the model will be deprecated in a future release",
]

# or silence all warnings (not recommended)
# warnings.filterwarnings('ignore')

for m in messages:
    warnings.filterwarnings("ignore", message=m)
print(f"Using PyMC3 version: {pm.__version__}")
print(f"Using ArviZ version: {az.__version__}")
```

    WARNING (theano.configdefaults): g++ not available, if using conda: `conda install m2w64-toolchain`
    C:\Users\David\Anaconda3\envs\cs109a\lib\site-packages\theano\configdefaults.py:560: UserWarning: DeprecationWarning: there is no c++ compiler.This is deprecated and with Theano 0.11 a c++ compiler will be mandatory
      warnings.warn("DeprecationWarning: there is no c++ compiler."
    WARNING (theano.configdefaults): g++ not detected ! Theano will be unable to execute optimized C-implementations (for both CPU and GPU) and will default to Python implementations. Performance will be severely degraded. To remove this warning, set Theano flags cxx to an empty string.
    WARNING (theano.tensor.blas): Using NumPy C-API based implementation for BLAS functions.

    Using PyMC3 version: 3.9.3
    Using ArviZ version: 0.9.0

```python
import pymc3 as pm
```

```python
from scipy import optimize
```

```python
import numpy as np
import matplotlib.pyplot as plt
import scipy.stats as stats
import pandas as pd
import seaborn as sns
%matplotlib inline
```

```javascript
%%javascript
IPython.OutputArea.auto_scroll_threshold = 20000;
```

    <IPython.core.display.Javascript object>

```python
# pandas trick
pd.options.display.max_columns = 50  # or =None -> No Restrictions
pd.options.display.max_rows = 200    # or =None -> *Be careful with this*
pd.options.display.max_colwidth = 100
pd.options.display.precision = 3
```

## Learning Objectives

By the end of this lab, you should be able to:
* identify and describe some of the most important probability distributions.
* apply Bayes Rule in calculating probabilities (and would have had fun in the process).
* create probabilistic models in the PyMC3 library.

## Table of Contents
1. [Some common families of distributions (review)](#1)
2. [The Bayesian Way of Thinking](#2)
3. [Bayesian Regression with pyMC3](#3) and [Defining a model in pyMC3](#model)
4. [Bayesian Logistic Regression with pyMC3](#4)

[Appendix](#appe)

## 1. Some common families of distributions

Statistical distributions are characterized by one of more parameters, such as $\mu$ and $\sigma^2$ for a Gaussian distribution.

\begin{equation}
Y \sim  \mathcal{N}(\mu,\,\sigma^{2})
\end{equation}

### Discrete Distributions

The **probability mass function (pmf)** of a discrete random variable $Y$ is equal to the probability that our random variable will take a specific value $y$: $f_y=P(Y=y)$ for all $y$. The variable, most of the times, assumes integer values.  Plots for **pmf**s are better understood as stem plots since we have discrete values for the probabilities and not a curve. Probabilities should add up to 1 for proper distributions.

- **Bernoulli** for a binary outcome, success has probability $\theta$, and we only have $one$ trial:

\begin{equation}
P(Y=k) =  \theta^k(1-\theta)^{1-k}
\end{equation}

- **Binomial** for a binary outcome, success has probability $\theta$, $k$ successes in $n$ trials:

Our random variable is $Y$= number of successes in $n$ trials.

\begin{equation}
P(Y=k|n,\theta) =  {{n}\choose{k}} \cdot \theta^k(1-\theta)^{n-k} \quad k=0,1,2,...,n
\end{equation}

As a reminder ${{n}\choose{k}}$ is "from $n$ choose $k$":
\begin{equation}
{{n}\choose{k}} = \frac{n!}{k!(n-k)!}
\end{equation}

$EY=n\theta$, $VarY = np(1-p)$

**Note** : Binomial (1,$p$) = Bernouli ($p$)

Exercise: Run the code below (that plots the Binomial distribution using stats.binom.pmf) for various values of the probability for a success $\theta\in [0,1]$. Look at the ordinate values to verify that they add up to 1.

```python
# probabity of success
theta = 0.5
n = 5
k = np.arange(0, n+1)
print(k)
pmf = stats.binom.pmf(k, n, theta)
plt.style.use('seaborn-darkgrid')
plt.stem(k, pmf, label=r'n = {}, $\theta$ = {}'.format(n, theta))
plt.xlabel('Y', fontsize=14)
plt.ylabel('P(Y)', fontsize=14)
plt.legend()
plt.show()
```

- **Negative Binomial** for a binary outcome, success has probability $\theta$, we have $r$ successes in $x$ trials:

Our random variable is $X$= number of trials to get to $r$ successes.

\begin{equation}
P(X=x|r,\theta) =  {{x-1}\choose{r-1}} \cdot \theta^r(1-\theta)^{x-r} \quad x=r,r+1,...
\end{equation}

- **Poisson** (counts independent events and has a single parameter $\lambda$)

\begin{equation}
P\left( Y=y|\lambda \right) = \frac{{e^{ - \lambda } \lambda ^y }}{{y!}} \quad y = 0,1,2,...
\end{equation}

Exercise: Change the value of $\lambda$ in the Poisson pmf plot below and see how the plot changes. Remember that the y-axis in a discrete probability distribution shows the probability of the random variable having a specific value in the x-axis. We use stats.poisson.pmf(x, $\lambda$), where $\lambda$ is the parameter.

```python
plt.style.use('seaborn-darkgrid')
x = np.arange(0, 10)
lam = 4
pmf = stats.poisson.pmf(x, lam)
plt.stem(x, pmf, label='$\lambda$ = {}'.format(lam))
plt.xlabel('Y', fontsize=12)
plt.ylabel('P(Y)', fontsize=12)
plt.legend(loc=1)
plt.ylim=(-0.1)
plt.show()
```

- **Discrete Uniform** for a random variable $X\in(1,N)$:

\begin{equation}
P(X=x|N) = \frac{1}{N}, \quad x=1,2,...,N
\end{equation}

```python
# boring but useful as a prior
plt.style.use('seaborn-darkgrid')
N = 40
x = np.arange(0, N)
pmf = stats.randint.pmf(x, 0, N)
plt.plot(x, pmf, label='$N$ = {}'.format(N))
fontsize=14
plt.xlabel('X', fontsize=fontsize)
plt.ylabel(f'P(X|{N})', fontsize=fontsize)
plt.legend()
plt.show()
```

- **Categorical, or Multinulli** (random variables can take any of K possible categories, each having its own probability; this is a generalization of the Bernoulli distribution for a discrete variable with more than two possible outcomes, such as the roll of a die)

\begin{equation}
f(x_1,x_2,...,x_n) =  \frac{m}{x_1!\cdot x_2!\dotsb x_n!} \cdot p_1^{x_1}\cdot p_2^{x_2}\dotsb p_n^{x_n}
\end{equation}

### Continuous Distributions

The random variable has a **probability density function (pdf)** whose area under the curve equals to 1.
- **Uniform** (variable equally likely to be near each value in interval $(a,b)$)
\begin{equation}
f(x|a,b) = \frac{1}{b - a} \quad x\in [a,b] \quad \text{and 0 elsewhere}.
\end{equation}

Exercise: Change the value of $\mu$ in the Uniform PDF and see how the plot changes.

Remember that the y-axis in a continuous probability distribution does not shows the actual probability of the random variable having a specific value in the x-axis because that probability is zero!. Instead, to see the probability that the variable is within a small margin we look at the integral below the curve of the PDF.

The uniform is often used as a noninformative prior.

```
Uniform - numpy.random.uniform(a=0.0, b=1.0, size)
```

$\alpha$ and $\beta$ are our parameters. `size` is how many tries to perform.
The mean is $\mu = \frac{(a+b)}{2}$

```python
from scipy.stats import uniform
a = 0
b = 1
r = uniform.rvs(loc=a, scale=b-a, size=1000)
pdf = uniform.pdf(r,loc=a, scale=b-a)
plt.plot(r, pdf,'b-', lw=3, alpha=0.6, label='uniform pdf')
plt.hist(r, density=True, histtype='stepfilled', alpha=0.2)
plt.ylabel(r'pdf')
plt.xlabel(f'x')
plt.legend(loc='best', frameon=False)
plt.show()
```

- **Normal** (a.k.a. Gaussian)
\begin{equation}
X \sim  \mathcal{N}(\mu,\,\sigma^{2})
\end{equation}

    A Normal distribution can be parameterized either in terms of precision $\tau$ or variance $\sigma^{2}$. The link between the two is given by
\begin{equation}
\tau = \frac{1}{\sigma^{2}}
\end{equation}
 - Expected value (mean) $\mu$
 - Variance $\frac{1}{\tau}$ or $\sigma^{2}$
 - Parameters: `mu: float`, `sigma: float` or `tau: float`
 - Range of values (-$\infty$, $\infty$)

```python
plt.style.use('seaborn-darkgrid')
x = np.linspace(-5, 5, 1000)
mus = [0., 0., 0., -2.]
sigmas = [0.4, 1., 2., 0.4]
for mu, sigma in zip(mus, sigmas):
    pdf = stats.norm.pdf(x, mu, sigma)
    plt.plot(x, pdf, label=r'$\mu$ = '+ f'{mu},' + r'$\sigma$ = ' + f'{sigma}')
plt.xlabel('random variable', fontsize=12)
plt.ylabel('probability density', fontsize=12)
plt.legend(loc=1)
plt.show()
```

- **Beta** (where the variable ($\theta$) takes on values in the interval $(0,1)$, and is parametrized by two positive parameters, $\alpha$ and $\beta$ that control the shape of the distribution. Note that Beta is a good distribution to use for priors (beliefs) because its range is $[0,1]$ which is the natural range for a probability and because we can model a wide range of functions by changing the $\alpha$ and $\beta$ parameters. Its density (pdf) is:

\begin{equation}
\label{eq:beta}
P(\theta|a,b) = \frac{1}{B(\alpha, \beta)} {\theta}^{\alpha - 1} (1 - \theta)^{\beta - 1} \propto {\theta}^{\alpha - 1} (1 - \theta)^{\beta - 1}
\end{equation}

where the normalisation constant, $B$, is a beta function of $\alpha$ and $\beta$,

\begin{equation}
B(\alpha, \beta) = \int_{x=0}^1 x^{\alpha - 1} (1 - x)^{\beta - 1} dx.
\end{equation}
 - 'Nice', unimodal distribution
 - Range of values $[0, 1]$

$EX = \frac{a}{a+b}$,    $VarX = \frac{ab}{(a+b)^2(a+b+1)}$

Exercise: Try out various combinations of $a$ and $b$. We get an amazing set of shapes by tweaking the two parameters $a$ and $b$. Notice that for $a=b=1.$ we get the uniform distribution. As the values increase, we get a curve that looks more and more Gaussian.

```python
from scipy.stats import beta

fontsize = 15
alphas = [0.5] #, 0.5, 1., 3., 6.]
betas = [0.5] #, 1., 1., 3., 6.]
x = np.linspace(0, 1, 1000)
colors = ['red', 'green', 'blue', 'black', 'pink']

fig, ax = plt.subplots(figsize=(8, 5))

for a, b, colors in zip(alphas, betas, colors):
    plt.plot(x, beta.pdf(x,a,b), c=colors,
             label=f'a={a}, b={b}')

ax.set_ylim(0, 3)

ax.set_xlabel(r'$\theta$', fontsize=fontsize)
ax.set_ylabel(r'P ($\theta|\alpha,\beta)$', fontsize=fontsize)
ax.set_title('Beta Distribution', fontsize=fontsize*1.2)

ax.legend(loc='best')
fig.show();
```

 At home: Prove the formula mentioned in class which gives the probability density for a Beta distribution with parameters $2$ and $5$:

$p(\theta|2,5) = 30 \cdot \theta(1 - \theta)^4$

 #### Code Resources:
 - Statistical Distributions in numpy/scipy: [scipy.stats](https://docs.scipy.org/doc/scipy/reference/stats.html)

 [Top](#top)

## 2. The Bayesian way of Thinking

```
Here is my state of knowledge about the situation. Here is some data, I am now going to revise my state of knowledge.
```

### Bayes Rule

\begin{equation}
\label{eq:bayes}
P(A|\textbf{B}) = \frac{P(\textbf{B} |A) P(A) }{P(\textbf{B})}
\end{equation}

$P(A|\textbf{B})$ is the **posterior** distribution, probability(parameters| data).

$P(\textbf{B} |A)$ is the **likelihood** function, how probable is my data for different values of the parameters.

$P(A)$ is the marginal probability to observe the data, called the **prior**, this captures our belief about the data before observing it.

$P(\textbf{B})$ is the marginal distribution (sometimes called marginal likelihood). This serves a normalization purpose.

### Let's Make a Deal

The problem we are about to solve gained fame as part of a game show "Let's Make A Deal" hosted by Monty Hall, hence its name. It was first raised by Steve Selvin in American Statistician in 1975.

The game is as follows: there are 3 doors behind **one** of which are the keys to a new car. There is a goat behind each of the other two doors. Let's assume your goal is to get the car and not a goat.

You are asked to pick one door, and let's say you pick **Door1**. The host knows where the keys are. Of the two remaining closed doors, he will always open the door that has a goat behind it. He'll say "I will do you a favor and open **Door2**". So he opens Door2 inside which there is, of course, a goat. He now asks you, do you want to open the initial Door1 you chose or change to **Door3**? Generally, in this game, when you are presented with this choice should you swap the doors?

**Hint:**
- Start by defining the `events` of this probabilities game. One definition is:

    - $A_i$: car is behind door $i$

    - $B_i$ host opens door $i$

$i\in[1,2,3]$

- In more math terms, the question is: is the probability of **the price is behind Door 1** higher than the probability of **the price is behind Door2**, given that an event **has occured**?

Breakout Game: Solve the Monty Hall Paradox using Bayes Rule.

### Bayes rule revisited

We have data that we believe come from an underlying distribution of unknown parameters. If we find those parameters, we know everything about the process that generated this data and we can make inferences (create new data).

\begin{equation}
P(\theta|\textbf{D}) = \frac{P(\textbf{D} |\theta) P(\theta) }{P(\textbf{D})}
\end{equation}

#### But what is $\theta \;$?

$\theta$ is an unknown yet fixed set of parameters. In Bayesian inference we express our belief about what $\theta$ might be and instead of trying to guess $\theta$ exactly, we look for its **probability distribution**. What that means is that we are looking for the **parameters** of that distribution. For example, for a Poisson distribution our $\theta$ is only $\lambda$. In a normal distribution, our $\theta$ is often just $\mu$ and $\sigma$.

 [Top](#top)

## 3. Bayesian Regression with `pyMC3`

PyMC3 is a Python library for programming Bayesian analysis, and more specifically, data creation, model definition, model fitting, and posterior analysis. It uses the concept of a `model` which contains assigned parametric statistical distributions to unknown quantities in the model. Within models we define random variables and their distributions. A distribution requires at least a `name` argument, and other `parameters` that define it. You may also use the `logp()` method in the model to build the model log-likelihood function. We define and fit the model.

PyMC3 includes a comprehensive set of pre-defined statistical distributions that can be used as model building blocks. They are not meant to be used outside of a `model`, and you can invoke them by using the prefix `pm`, as in `pm.Normal`.

For more see: [PyMC3 Quickstart](https://docs.pymc.io/notebooks/api_quickstart.html)

 ### Distributions in `PyMC3`:
 - Statistical [Distributions in pyMC3](https://docs.pymc.io/api/distributions.html).

Information about PyMC3 functions including descriptions of distributions, sampling methods, and other functions, is available via the `help` command.

```python
#help(pm.Beta)
```

### Defining a Model in PyMC3

Our problem is the following: we want to perform multiple linear regression to predict an outcome variable $Y$ which depends on variables $\bf{x}_1$ and $\bf{x}_2$.

We will model $Y$ as normally distributed observations with an expected value $mu$ that is a linear function of the two predictor variables, $\bf{x}_1$ and $\bf{x}_2$.

\begin{equation}
Y \sim  \mathcal{N}(\mu,\,\sigma^{2})
\end{equation}

\begin{equation}
\mu = \beta_0 + \beta_1 \bf{x}_1 + \beta_2 x_2
\end{equation}

where $\sigma^2$ represents the measurement error (in this example, we will use $\sigma^2 = 10$)

We also choose the parameters to have normal distributions with those parameters set by us.

\begin{eqnarray}
\beta_i \sim  \mathcal{N}(0,\,10) \\
\sigma^2 \sim  |\mathcal{N}(0,\,10)|
\end{eqnarray}
We will artificially create the data to predict on. We will then see if our model predicts them correctly.

#### Let's create some synthetic data

```python
np.random.seed(123)

###################################
## Hidden true parameter values
sigma = 1
beta0 = 1
beta = [1, 2.5]
####################################

# Size of dataset
size = 100

# Predictor variable
x1 = np.linspace(0, 1., size)
x2 = np.linspace(0,2., size)

# Simulate outcome variable
Y = beta0 + beta[0]*x1 + beta[1]*x2 + np.random.randn(size)*sigma
```

```python
Y.shape, x1.shape, x2.shape
```

```python
from mpl_toolkits.mplot3d import Axes3D
fig = plt.figure()
fontsize=14
labelsize=8
title='Observed Data (created artificially by ' + r'$Y(x_1,x_2)$)'
ax = fig.add_subplot(111, projection='3d')

ax.scatter(x1, x2, Y)
ax.set_xlabel(r'$x_1$', fontsize=fontsize)
ax.set_ylabel(r'$x_2$', fontsize=fontsize)
ax.set_zlabel(r'$Y$', fontsize=fontsize)

ax.tick_params(labelsize=labelsize)

fig.suptitle(title, fontsize=fontsize)
fig.tight_layout(pad=.1, w_pad=10.1, h_pad=2.)
fig.subplots_adjust(); #top=0.5
plt.tight_layout
plt.show()
```

#### Building the model

**Step1:** Formulate the probability model for our data: $Y \sim  \mathcal{N}(\mu,\,\sigma^{2})$

```
Y_obs = pm.Normal('Y_obs', mu=mu, sd=sigma, observed=Y)
```

**Step2:** Choose a prior distribution for our unknown parameters.
```
 beta0 = Normal('beta0', mu=0, sd=10)
 # Note: betas is a vector of two variables, b1 and b2, (denoted by shape=2)
 # so, in array notation, our beta1 = betas[0], and beta2=betas[1]
 betas = Normal('betas', mu=0, sd=10, shape=2)
 sigma = HalfNormal('sigma', sd=1)

```
**Step3:** Construct the likelihood function.

**Step4:** Determine the posterior distribution, **this is our main goal**.

**Step5:** Summarize important features of the posterior and/or plot the parameters.

```python
with pm.Model() as my_linear_model:

    # Priors for unknown model parameters, specifically created stochastic random variables
    # with Normal prior distributions for the regression coefficients,
    # and a half-normal distribution for the standard deviation of the observations.
    # These are our parameters.

    # I need to give the prior some initial values for the parameters
    mu0 = 0
    sd0 = 10
    beta0 = pm.Normal('beta0', mu=mu0, sd=sd0)
    # Note: betas is a vector of two variables, b1 and b2, (denoted by shape=2)
    # so, in array notation, our beta1 = betas[0], and beta2=betas[1]
    mub = 0
    sdb = 10
    betas = pm.Normal('betas', mu=mub, sd=sdb, shape=2)
    sds = 1
    sigma = pm.HalfNormal('sigma', sd=sds)

    # mu is what is called a deterministic random variable, which implies that
    # its value is completely determined by its parents’ values
    # (betas and sigma in our case). There is no uncertainty in the
    # variable beyond that which is inherent in the parents’ values

    mu = beta0 + betas[0]*x1 + betas[1]*x2

    # Likelihood function = how probable is my observed data?
    # This is an observed variable; it is identical to a standard
    # stochastic variable, except that its observed argument,
    # which passes the data to the variable, indicates that the values for this
    # variable were observed, and should not be changed by any
    # fitting algorithm applied to the model.
    # The data can be passed in the form of a numpy.ndarray or pandas.DataFrame object.

    Y_obs = pm.Normal('Y_obs', mu=mu, sd=sigma, observed=Y)
```

```python
my_linear_model.basic_RVs
```

```python
my_linear_model.free_RVs
```

**Note**: If our problem was a classification for which we would use Logistic regression.

```python
## do not worry if this does not work, it's just a nice graph to have
## you need to install python-graphviz first
# conda install -c conda-forge python-graphviz
pm.model_to_graphviz(my_linear_model)
```

Now all we need to do is sample our model.

### ... to be continued

## Appendix A: Bayesian Logistic Regression with `pyMC3`

If the problem above was a classification that required a Logistic Regression, we would use the logistic function ( where $\beta_0$ is the intercept, and $\beta_i$ (i=1, 2, 3) determines the shape of the logistic function).

\begin{equation}
Pr(Y=1|X_1,X_2,X3) = {\frac{1}{1 + exp^{-(\beta_0 + \beta_1X_1 + \beta_2X_2 + \beta_3X_3)}}}
\end{equation}

Since both $\beta_0$ and the $\beta_i$s can be any possitive or negative number, we can model them as gaussian random variables.

\begin{eqnarray}
\beta_0 \sim  \mathcal{N}(\mu,\,\sigma^2) \\
\beta_i \sim  \mathcal{N}(\mu_i,\,\sigma_i^2)
\end{eqnarray}

In PyMC3 we can model those as:
```
pm.Normal('beta_0', mu=0, sigma=100)
```
(where $\mu$ and $\sigma^2$ can have some initial values that we assign them, e.g. 0 and 100)

The dererministic variable would be:
```
logitp = beta0 + beta_1 * X_1 + beta_2 * X_2 + beta_3 * X_3
```
To connect this variable (logit-p) with our observed data, we would use a Bernoulli as our likelihood.
```
our_likelihood = pm.Bernoulli('our_likelihood', logit_p=logitp, observed=our_data)
```

Notice that the main difference with Linear Regression is the use of a Bernoulli distribution instead of a Gaussian distribution, and the use of the logistic function instead of the identity function.

Breakout Room Exercise: Write the model above in code. Suppose that your training dataframe (df_train) has the following features:

**numerical**
- df_train['age']
- df_train['weight']

**categorical**
- df_train['hypertension']

```python
# A reminder of what the logistic function looks like.
# Change parameters a and b to see the shape of the curve change
b = 5.
x = np.linspace(-8, 8, 100)
plt.plot(x, 1 / (1 + np.exp(-b*x)))
plt.xlabel('y')
plt.ylabel('y=logistic(x)')
```

## Appendix B: Is this a fair coin?

### Is this a fair coin?

Let's say you visit the casino in **Monte Carlo**. You want to test your theory that casinos are dubious places where coins have been manipulated to have a larger probability for tails. So you will try to estimate how fair a coin is based on a certain amount of  flips.

You have no prior opinion on the coin's fairness (i.e. what $p$ might be), and begin flipping the coin. You get either Heads ($H$) or Tails ($T$) as our observed data and want to see if your posterior probabilities change as you obtain more data, that is, more coin flips. A nice way to visualize this is to plot the posterior probabilities as we observe more flips (data).

We will be using Bayes rule. $\textbf{D}$ is our data.

\begin{equation}
P(\theta|\textbf{D}) = \frac{P(\textbf{D} |\theta) P(\theta) }{P(\textbf{D})}
\end{equation}

We start with a non-informative prior, a Beta distribution with (a=b=1.)

\begin{equation}
P(\theta|\textbf{k=0}) = Beta(1., 1.)
\end{equation}

Then, as we get new data (say, we observe $k$ heads in $n$ tosses), we update our Beta with new a,b as follows:

\begin{equation}
P(\theta|\textbf{k}) = Beta(\alpha + \textbf{k}, \beta + (n - \textbf{k}))
\end{equation}

*(the proof is beyond our scope, if interested, see this [Wikipedia article](https://en.wikipedia.org/wiki/Conjugate_prior#Example))*

we can say that $\alpha$ and $\beta$ play the roles of a "prior number of heads" and "prior number of tails".

```python
# play with the priors - here we manually set them but we could be sampling from a separate Beta
trials = np.array([0, 1, 3, 5, 10, 15, 20, 100, 200, 300])
heads = np.array([0, 1, 2, 4, 8, 10, 10, 50, 180, 150])
x = np.linspace(0, 1, 100)

# for simplicity we set a,b=1
plt.figure(figsize=(10,8))
for k, N in enumerate(trials):
    sx = plt.subplot(len(trials)/2, 2, k+1)
    posterior = stats.beta.pdf(x, 1 + heads[k], 1 + trials[k] - heads[k])
    plt.plot(x, posterior, alpha = 0.5, label=f'{trials[k]} tosses\n {heads[k]} heads');
    plt.fill_between(x, 0, posterior, color="#348ABD", alpha=0.4)
    plt.legend(loc='upper left', fontsize=10)
    plt.legend()
    plt.autoscale(tight=True)

plt.suptitle("Posterior probabilities for coin flips", fontsize=15);
plt.tight_layout()
plt.subplots_adjust(top=0.88)
```

### References

- *Salvatier J, Wiecki TV, Fonnesbeck C. 2016. Probabilistic programming in Python using PyMC3. PeerJ Computer Science 2:e55* [(https://doi.org/10.7717/peerj-cs.55)](https://doi.org/10.7717/peerj-cs.55)
- [Distributions in PyMC3](https://docs.pymc.io/api/distributions.html)
- [More Details on Distributions](https://docs.pymc.io/developer_guide.html)

This linear regression example is from the original paper on PyMC3: *Salvatier J, Wiecki TV, Fonnesbeck C. 2016. Probabilistic programming in Python using PyMC3. PeerJ Computer Science 2:e55 https://doi.org/10.7717/peerj-cs.55*

#### Cool Reading

- How Bayesian Analysis and Lawrence D. Stone found the Wreckage of [Air France Flight AF 447](https://www.jstor.org/stable/pdf/43288452.pdf?refreqid=excelsior%3Ae972f3a7b38b5815e1570b93dad7d269).
- Search for the gold on the sunken [SS Central America](https://www.metsci.com/wp-content/uploads/2019/08/Search-for-the-SS-Central-America-Mathematical-Treasure-Hunting-Interfaces-22-1-Jan.-Feb.-1992.pdf).


```python
from __future__ import division, print_function
from operator import mul
from functools import reduce
import chempy
from chempy.chemistry import Species, Equilibrium
from chempy.equilibria import EqSystem, NumSysLog, NumSysLin
import numpy as np
import sympy as sp
sp.init_printing()
import matplotlib.pyplot as plt
%matplotlib inline
print((sp.__version__, chempy.__version__))
```

```python
substance_names = ['H+', 'OH-', 'NH4+', 'NH3', 'H2O']
subst = {n: Species.from_formula(n) for n in substance_names}
assert [subst[n].charge for n in substance_names] == [1, -1, 1, 0, 0]
init_conc = {'H+': 1e-7, 'OH-': 1e-7, 'NH4+': 1e-7, 'NH3': 1.0, 'H2O': 55.5}
x0 = [init_conc[k] for k in substance_names]
H2O_c = init_conc['H2O']
w_autop = Equilibrium({'H2O': 1}, {'H+': 1, 'OH-': 1}, 10**-14/H2O_c)
NH4p_pr = Equilibrium({'NH4+': 1}, {'H+': 1, 'NH3': 1}, 10**-9.26)
equilibria = w_autop, NH4p_pr
[(k, init_conc[k]) for k in substance_names]
```

```python
rs = EqSystem(equilibria, subst)
x, sol, sane = rs.root(init_conc)
x, sol['success'], sane
```

```python
logx, logsol, sane = rs.root(init_conc, x0=x, NumSys=(NumSysLog,))
logx, logsol['success'], sane
```

```python
x - logx
```

```python
ny = len(substance_names)
y = sp.symarray('y', ny)
i = sp.symarray('i', ny)
K = Kw, Ka = sp.symbols('K_w K_a')
w_autop.param = Kw
NH4p_pr.param = Ka
ss = sp.symarray('s', ny)
ms = sp.symarray('m', ny)
```

```python
numsys_log = NumSysLog(rs, backend=sp)
f = numsys_log.f(y, list(i)+list(K))
f
```

```python
numsys_lin = NumSysLin(rs, backend=sp)
numsys_lin.f(y, i)
```

```python
A, ks = rs.stoichs_constants(False, backend=sp)
[reduce(mul, [b**e for b, e in zip(y, row)]) for row in A]
```

```python
from pyneqsys.symbolic import SymbolicSys
subs = list(zip(i, x0)) + [(Kw, 10**-14), (Ka, 10**-9.26)]
numf = [_.subs(subs) for _ in f]
neqs = SymbolicSys(list(y), numf)
neqs.solve([0, 0, 0, 0, 0], solver='scipy')
```

```python
j = sp.Matrix(1, len(f), lambda _, q: f[q]).jacobian(y)
init_conc_j = {'H+': 1e-10, 'OH-': 1e-7, 'NH4+': 1e-7, 'NH3': 1.0, 'H2O': 55.5}
xj = rs.as_per_substance_array(init_conc_j)
jarr = np.array(j.subs(dict(zip(y, xj))).subs({Kw: 1e-14, Ka: 10**-9.26}).subs(
            dict(zip(i, xj))))
jarr = np.asarray(jarr, dtype=np.float64)
np.log10(np.linalg.cond(jarr))
```

```python
j.simplify()
j
```

```python
rs.composition_balance_vectors()
```

```python
numsys_rref_log = NumSysLog(rs, True, True, backend=sp)
numsys_rref_log.f(y, list(i)+list(K))
```

```python
np.set_printoptions(4, linewidth=120)
scaling = 1e8
for rxn in rs.rxns:
    rxn.param = rxn.param.subs({Kw: 1e-14, Ka: 10**-9.26})
```

```python
x, res, sane = rs.root(init_conc, rref_equil=True, rref_preserv=True)
x, res['success'], sane
```

```python
x, res, sane = rs.root(init_conc, x0=rs.as_per_substance_array(
        {'H+': 1e-11, 'OH-': 1e-3, 'NH4+': 1e-3, 'NH3': 1.0, 'H2O': 55.5}))
res['success'], sane
```

```python
x, res, sane = rs.root(init_conc, x0=rs.as_per_substance_array(
        {'H+': 1.7e-11, 'OH-': 3e-2, 'NH4+': 3e-2, 'NH3': 0.97, 'H2O': 55.5}))
res['success'], sane
```

```python
init_conc
```

```python
nc=60
Hp_0 = np.logspace(-3, 0, nc)
def plot_rref(**kwargs):
    fig = plt.figure(figsize=(16, 6))
    ax1 = plt.subplot(2, 2, 1, xscale='log', yscale='log')
    ff = Cout_1, ic1, success1 = rs.roots(init_conc, Hp_0, 'H+', plot_kwargs={'ax': ax1}, rref_equil=False, rref_preserv=False, **kwargs)
    ax2 = plt.subplot(2, 2, 2, xscale='log', yscale='log')
    ft = Cout_2, ic2, success2 = rs.roots(init_conc, Hp_0, 'H+', plot_kwargs={'ax': ax2}, rref_equil=False, rref_preserv=True, **kwargs)
    ax3 = plt.subplot(2, 2, 3, xscale='log', yscale='log')
    tf = Cout_3, ic3, success3 = rs.roots(init_conc, Hp_0, 'H+', plot_kwargs={'ax': ax3}, rref_equil=True, rref_preserv=False, **kwargs)
    ax4 = plt.subplot(2, 2, 4, xscale='log', yscale='log')
    tt = Cout_4, ic4, success4 = rs.roots(init_conc, Hp_0, 'H+', plot_kwargs={'ax': ax4}, rref_equil=True, rref_preserv=True, **kwargs)
    return ff, ft, tf, tt
```

```python
res_lin = plot_rref(method='lm')
[all(_[2]) for _ in res_lin]
```

```python
for col_id in range(len(substance_names)):
    for i in range(1, 4):
        plt.subplot(1, 3, i, xscale='log')
        plt.gca().set_yscale('symlog', linthreshy=1e-14)
        plt.plot(Hp_0, res_lin[0][0][:, col_id] - res_lin[i][0][:, col_id])
plt.tight_layout()
```

```python
rs.plot_errors(res_lin[0][0], init_conc, Hp_0, 'H+')
```

```python
init_conc, rs.ns
```

```python
res_log = plot_rref(NumSys=NumSysLog)
```

```python
rs.plot_errors(res_log[0][0], init_conc, Hp_0, 'H+')
```

```python
res_log_lin = plot_rref(NumSys=(NumSysLog, NumSysLin))
rs.plot_errors(res_log_lin[0][0], init_conc, Hp_0, 'H+')
```

```python
from chempy.equilibria import NumSysSquare
res_log_sq = plot_rref(NumSys=(NumSysLog, NumSysSquare))
rs.plot_errors(res_log_sq[0][0], init_conc, Hp_0, 'H+')
```

```python
res_sq = plot_rref(NumSys=(NumSysSquare,), method='lm')
rs.plot_errors(res_sq[0][0], init_conc, Hp_0, 'H+')
```

```python
x, res, sane = rs.root(x0, NumSys=NumSysLog, rref_equil=True, rref_preserv=True)
x, res['success'], sane
```

```python
x, res, sane = rs.root(x, NumSys=NumSysLin, rref_equil=True, rref_preserv=True)
x, res['success'], sane
```

```python
# x, res, sane = rs.root(x, NumSys=(NumSysLinTanh,), rref_equil=False, rref_preserv=False)
# x, res['success'], sane
```

```python
# res_tanh = plot_rref(NumSys=(NumSysLog, NumSysLinTanh,))
# rs.plot_errors(res_tanh[0][0], init_conc, np.logspace(-4, 0, nc), Hp)
```


# Automation's impact on the economic growth.

## Importing modules

```python
import numpy as np
import scipy as sp
from scipy import linalg
from scipy import optimize
from scipy import interpolate
import sympy as sm

%matplotlib inline
import matplotlib.pyplot as plt
from matplotlib import cm
from mpl_toolkits.mplot3d import Axes3D
import ipywidgets as widgets
from ipywidgets import interact, interactive, fixed, interact_manual
from IPython import display

sm.init_printing(use_unicode=True)
import time
```

## Introduction

This notebook considers a model that Klaus Prettner [Prettner, K. (2019)] use to analyze optimal growth with the presence of automation, thought as AI, robots 3D printers, and so forth.

The model can be viewed as an extension of the model of Robert Solow, where labor ($L_t$) and automation ($P_t$) is perfect substitutes.

**Framework:**

"The robots are coming - Do they take our jobs" - A topic has got more and more publicity the daily press now a days. The answer is difficult because it is ambiguous - New technology, can solve problems better than maintream labor, but the new technology also creates new opputuneties and jobs. In short,  it is clear that new technology has a large impact on some given industries.

*For example, in the car industry, robots perform many production steps in an autonomous way; 3D printers are capable of producing customized products with minimal labor input (Abeliansky et al., 2015); driverless cars and lorries could soon transport goods from location A to location B without any involvement of labor; and devices based on machine learning are already able to diagnose some forms of diseases, to translate texts from one language to another with an acceptable quality, and even to write simple newsflashes.
This implies that stock of automation capital, $P_t$, installed in the form of robots, 3D printers, driverless cars, and devices based on machine learning is close to being a perfect substitute for labor, $L_t$.*  - **Klaus Prettner**

That written, we will not go deeper into the discussion about the substitution between labor and automation, but simply analyse the specific model. We will investigate which savings rate for automation that is the optimal for the overall growth of the model.

**Model:**

Consider the following extension to the **Standard Solow-model** where:

1. $K(t)$ is capital
* $P(t)$ is the stock of automation capital
* $L(t)$ is labor (growing with a constant rate of $n$)
* $A(t)$ is technology (which we deliberately normalize to 1)
* $Y(t) = F[K(t),A(t),(L(t)+P(t))]$ is GDP

The production function is a Cobb-Douglas given by:
\\[ Y(t)=A(t)K(t)^{\alpha}(L(t)+P(t))^{1-\alpha}\\]

The economy is closed, and the savings rate, s, is given exogenously. This setup implies that investment equal savings, such that $I(t)=S(t)=sY(t)$. The investments can be made in terms of two different forms of capital: machines and automation, which implies that a share $s_m$ of savings are diverted to investment in terms of machines and a share $(1−s_m)$ are diverted to investment in terms of automation.

Putting it all together, are the accumulation equations for machines and automation given by respectively:
\\[\dot{K}(t) =s_mI(t)-\delta K(t) \quad \quad \dot{P}(t) =(1-s_m)I(t)-\delta P(t)\\]

This model setup gives us the following equations for the accumulation of capital and automation capital per worker:

\\[\dot{k}(t)=s_ms[1+p(t)]^{1-\alpha}k(t)^\alpha-\delta k(t)-nk(t) \\]
\\[\dot{p}(t)=(1-s_m)s[1+p(t)]^{1-\alpha}k(t)^{\alpha}-\delta p(t)-np(t) \\]

Above we have a problem, which can be turned into to a problem of two equations with two unknown parameters.
We divide the equations above by k(t) and p(t), respectively, and impose constant growth along the balanced growth path (BGP), which give us the equations below: (Steps in between, see Appendix):

\\[g = s_m s C^{1-\alpha}-\delta-n \\]
\\[g = (1-s_m) s [\frac{1}{C}]^\alpha-\delta-n \\]

In such a manner it can be shown that the economy converges to a situation, in which traditional capital per worker, automation capital per worker, and GDP per worker all grow at the same constant rate. The equation below is also calculated in section 1.3 using sympy.

\\[g = s\cdot s_m^\alpha(1-s_m)^{1-\alpha}-\delta-n \\]

In this model project (as written in the beginning), we would like to investigate the parameter $s_m$, to figure out what a share, channeled into traditional capital investment, that would maximize the economic growth.

## Analytical

We solve the problem of the two equation with two unknown parameter by using sympy:
First we equalize the RHS of the two equations to isolate C.

```python
#In order to use sympy, we define our parameters as sympy-symbols
alpha = sm.symbols('alpha')
delta = sm.symbols('delta')
s = sm.symbols('s')
s_m=sm.symbols('s_m')
n=sm.symbols('n')
C=sm.symbols('C')

#We make use of the equalize and solve-function
eq3 = sm.Eq(s_m * s * C **(1-alpha)-delta-n-(1-s_m) * s * (1/C)**alpha+delta+n)
Cstar = sm.solve(eq3,C)
Cstar
```

Afterwards we subsitute C into the equation for g:

```python
g = s_m * s * ((-s_m+1)/s_m) **(1-alpha)-delta-n
g1 = (1-s_m)*s*(1/(((-s_m+1)/s_m)))**alpha-delta-n
g
```

If we rearrange the expression above, it can easily be seen that it is equal to the expression for $g$ in the introduction.
The expression implies that the growth rate depend positively on the general savings rate plus the shares channeled into traditional and automation capital, respectively. Additionally, the growth rate depend negativly on the depreciation of capital,  and the growth of the population. For large values of $\delta$ and $n$, the growth rate would be negative, but for now we focus on the solutions for which the growth rate is postive. This is also the case for plausible parameter specifications by Prettner (see table 1) and why he use those.

Furthermore is it obvious that there is a trade-off problem in the expression for g, since the optimal investment rate in traditional and automation capital is in relation to each other.
This is exactly what we aim to figure out in this model project: What share of the parameter $s_m$ maximize the economic growth.

## Optimizing the parameter of the share invested in machines.

We start by are using sympy yet again, to find an analytical expression for the value of $s_m$ which maximize g.

```python
g = s*(s_m**alpha)*((1-s_m)**(1-alpha))-delta-n
gdiff=sm.diff(g,s_m)
gdiff
```

```python
#this expression is too muddy, so we simplify it using the simplify-function.
sm.simplify(gdiff)
```

```python
#Then we solve this equation, isolating s_m
sm.solve(gdiff,s_m)[0]
```

The result show that $s_m=\alpha$ would maximize the economic growth.
This means that, the economy would be best served with an investment in automation capital equal to 70 percent.

## Numerical optimization

Even though, this problem can be solved analytically, we will also show how to solve the problem numirically, since it is the core of this course. We already know, what the solution is, but we think that this procedure is relevant, because it provides an intuitive base for understanding how numerical optimization works.

```python
#We assign the parameters with the values given by Prettner (see table 1)

alpha = 0.3
delta = 0.04
s = 0.21
n = 0.009

# We turn our sympy-equation, into a python-function

f=lambda s_m: s*(s_m**(alpha-1))*((1-s_m)**(-alpha))*(alpha-s_m)
```

Before optimizing, we plot our function of balanced growth, to see how it looks like.

```python
# Data for plotting
s_m = np.arange(-1, 80, 0.01)
g = g = 0.21*(s_m**0.3)*((1-s_m)**(1-0.3))-0.04-0.009

fig, ax = plt.subplots()
ax.plot(s_m, g, label='Growth path')
plt.plot([0.3], [0.06500510506486332], marker='o', markersize=4, color="red", label='Optimum')
ax.set(xlabel='Saving rate for automation', ylabel='Growth in GDP per worker',
       title='Balanced growth')
ax.grid()
plt.legend(bbox_to_anchor=(0.5, -0.3), loc=8, ncol=4)

plt.show()
```

We optimize with bisection, since we think that it provides the users and readers with a fine understanding of how the optimal value of $s_m$ is found. We suplement this, with a widget that describes the steps of the bisection method. Overall, we supply our deriviative function with two values $a & b$, which we know will give the deriviative function a positive and a negative value. We thereby know that the optimum is between these to points (where $f=0$). Then we find the midpoint of these to values. If the midpoint is not equal to zero, we move our points of a & b, closer to each other. This progress repeated till the midpoint equals the root. If our function value of a & b multiplied is positive, our bisection will not work, beause our bounds are wrongly set.

```python
def bisection(f,a,b,max_iter=50,tol=1e-6,full_info=True):
    """ bisection

    Solve equation f(x) = 0 for a <= x <= b.

    Args:

        f (function): function
        a (float): left bound
        b (float): right bound
        tol (float): tolerance on solution

    Returns:

        m (float): root

    """

    # test inputs
    if f(a)*f(b) >= 0:
        print("bisection method fails.")
        return None

    # step 1: initialize
    _a = a
    _b = b
    a = np.zeros(max_iter)
    b = np.zeros(max_iter)
    m = np.zeros(max_iter)
    fm = np.zeros(max_iter)
    a[0] = _a
    b[0] = _b

    # step 2-4: main
    i = 0
    while i < max_iter:

        # step 2: midpoint and associated value
        m[i] = (a[i]+b[i])/2
        fm[i] = f(m[i])

        # step 3: determine sub-interval
        if abs(fm[i]) < tol:
            break
        elif f(a[i])*fm[i] < 0:
            a[i+1] = a[i]
            b[i+1] = m[i]
        elif f(b[i])*fm[i] < 0:
            a[i+1] = m[i]
            b[i+1] = b[i]
        else:
            print("bisection method fails.")
            return None

        i += 1

    if full_info == True:
        return m,i,a,b,fm
    else:
        return m[i],i
```

```python
#We assign the values of "a" and "b" - and we know it has to be between [0,1].
m,i,a,b,fm = bisection(f,0.2,0.9)

#Next, we print our value of our midtpoint, and the number of iterations.
#The number of iterations, is interesting becauses it shows how many loops the bisection has been through.
print(i, m[i])
```

    17 0.2999996185302734

So with 17 iterations, we find the optimal value of $s_m$ equal to 0.3, which is in lign with our value of $\alpha$ given by Prettner. To get a better grasp of how the bisection method works, we plot a widget, which illustrate how we find our optimal point.

```python
def plot_bisection(f,a,b,xmin=0.2,xmax=0.9,xn=100):

    # a. find root and return all information
    m,max_iter,a,b,fm = bisection(f,a,b,full_info=True)

    # b. compute function on grid
    xvec = np.linspace(xmin,xmax,xn)
    fxvec = f(xvec)

    # c. figure
    def _figure(i):

        # ii. figure
        fig = plt.figure(dpi=100)
        ax = fig.add_subplot(1,1,1)

        ax.plot(xvec,fxvec) # on grid
        ax.plot(m[i],fm[i],'o',color='black',label='current') # mid
        ax.plot([a[i],b[i]],[fm[i],fm[i]],'--',color='black',label='range') # range
        ax.axvline(a[i],ls='--',color='black')
        ax.axvline(b[i],ls='--',color='black')

        ax.legend(loc='lower right')
        ax.grid(True)
        ax.set_ylim([fxvec[-1],fxvec[0]])

    widgets.interact(_figure,
        i=widgets.IntSlider(description="iterations", min=0, max=max_iter, step=1, value=0)
    );

plot_bisection(f,0.2,0.9) #The command gives a figure which is interactive manually.
```

    interactive(children=(IntSlider(value=0, description='iterations', max=17), Output()), _dom_classes=('widget-i…

We also think it could be fun to make a figure, which moves automatically without scrolling on the interactive bar. We therefore rewrite the code, so it just plots data of one number of iterations.

```python
#transforming widget to plot - to be able to loop the different plots (with increasing iterations)
def plot_bi(f,a,b,i,xmin=0.2,xmax=0.9,xn=100):

    # a. find root and return all information
    m,max_iter,a,b,fm = bisection(f,a,b,full_info=True)

    # b. compute function on grid
    xvec = np.linspace(xmin,xmax,xn)
    fxvec = f(xvec)

    # c. figure
    def _figure(i):

        # ii. figure
        # fig = plt.figure(dpi=100)
        # ax = fig.add_subplot(1,1,1)
        fig, ax = plt.subplots(ncols=1, sharey=False)

        ax.plot(xvec,fxvec) # on grid
        ax.plot(m[i],fm[i],'o',color='black',label='current') # mid
        ax.plot([a[i],b[i]],[fm[i],fm[i]],'--',color='black',label='range') # range
        ax.axvline(a[i],ls='--',color='black')
        ax.axvline(b[i],ls='--',color='black')

        ax.legend(loc='lower right')
        ax.grid(True)
        ax.set_ylim([fxvec[-1],fxvec[0]])
        plt.show()
        return

    fig = _figure(i)

    return fig

plot_bi(f,0.2,0.9, 2) # The function makes it possible to choose which iterations to plot
```

Hereafter, we make a loop over the display of increasing numbers of iterations - and we make sure too clear the present output. In this way, it looks automatic.

```python
#loop set too 1 second
def loop():
    for i in range(0,9,1):
            display.display(plot_bi(f,a[i],b[i],i,xmin=0.2,xmax=0.9,xn=100))
            display.clear_output(wait=True);
            time.sleep(1)
loop()
```

We have now found the optimal parameter value of $s_m$, both by analytically(sympy) and via numerical optimization.

## Analysis: Using our results to plot the accumulation

We will now show how the optimal parameter value of $s_m$ express itself in the accumulation of capital and the respective growth rates.
For doing so, we use the following parameter values shown in table 1, which is taken from the paper of Klaus Prettner.
The table summarizes the parameter values we will use for our numerical analysis and provides a valuation of them.

\\[Table \ 1: Parameter \ values\\]

| Parameter | Value |                                                Comment |
|-----------|:-----:|-------------------------------------------------------:|
| s         |  0.21 |   Average gross investment rate (2000-2013) for the US |
| $s_m$       |  0.2, 0.3, 0.4  |                                        Arbitrary value |
| $\alpha$    |  1/3  | Jones (1995), Acemoglu (2009), Grossmann et al. (2013) |
| $\delta$    | 0.04  | Grossmann et al. (2013)                                |
| n         | 0.009 | Average rate (2000-2014) for the US (World Bank, 2015) |

 **Plotting the accumulation:**

```python
# Our model

# Allocate memory for time series
series_length=100

k = np.empty(series_length)
p = np.empty(series_length)
g_k = np.empty(series_length)
g_p = np.empty(series_length)

fig, axes = plt.subplots(2, 2, figsize=(12, 15))

# Trajectories with different s_m
alpha = 1/3
delta = 0.04
s = 0.21
#s = (0.21, 0.25) # Would generate the same figures as Klaus Prettner fig. 1. b
s_m = (0.2, 0.3, 0.4)
n = 0.009

for j in range(3):
    k[0] = 1
    p[0] = 1

    for t in range(series_length-1):
        k[t+1] = s_m[j] * s * (1 + p[t])**(1 - alpha) * k[t]**(alpha) + (1- delta) * k[t] - n * k[t]
        p[t+1] = (1-s_m[j]) * s * (1 + p[t])**(1 - alpha) * k[t]**(alpha) + (1- delta) * p[t] - n * p[t]
        #k[t+1] = s_m * s[j] * (1 + p[t])**(1 - alpha) * k[t]**(alpha) +(1- delta) * k[t] - n * k[t]       #Klaus Prettner-Copy
        #p[t+1] = (1-s_m) * s[j] * (1 + p[t])**(1 - alpha) * k[t]**(alpha) + (1- delta) * p[t] - n * p[t]  #Klaus Prettner-Copy
    axes[0,0].plot(k, '-', label=rf"$\alpha = {alpha:.3f},\; s_m = {s_m[j]},\; \delta={delta}$")
    axes[1,0].plot(p, '-', label=rf"$\alpha = {alpha:.3f},\; s_m = {s_m[j]},\; \delta={delta}$")

for j in range(3):

    for t in range(series_length-1):
        g_k[t+1] = s_m[j] * s * ((1+p[t])/(k[t]))**(1-alpha) - delta - n
        g_p[t+1] = (1-s_m[j]) * s * ((1+p[t])/p[t])**(1-alpha) *  (k[t]/p[t])**alpha - delta - n
        #g_k[t+1] = s_m * s[j] * ((1+p[t])/(k[t]))**(1-alpha) - delta - n                                  #Klaus Prettner-Copy
        #g_p[t+1] = (1-s_m) * s[j] * ((1+p[t])/p[t])**(1-alpha) *  (k[t]/p[t])**alpha - delta - n          #Klaus Prettner-Copy
    axes[0,1].plot(g_k, '-', label=rf"$\alpha = {alpha:.3f},\; s_m = {s_m[j]},\; \delta={delta}$")
    axes[1,1].plot(g_p, '-', label=rf"$\alpha = {alpha:.3f},\; s_m = {s_m[j]},\; \delta={delta}$")

axes[0,0].grid(lw=0.2)
axes[1,0].grid(lw=0.2)
axes[0,1].grid(lw=0.2)
axes[1,1].grid(lw=0.2)

#ajust limmit on the y and x axis.
axes[0,0].set_ylim(0, 600)
axes[0,1].set_xlim(1,series_length)
axes[1,1].set_xlim(1,series_length)
axes[0,1].set_ylim(0.0,0.3)
axes[1,1].set_ylim(0.0,0.3)

axes[0,0].set_xlabel('time')
axes[0,0].set_ylabel('capital')
axes[1,0].set_xlabel('time')
axes[1,0].set_ylabel('automation')
axes[0,1].set_xlabel('time')
axes[0,1].set_ylabel('growth of k')
axes[1,1].set_xlabel('time')
axes[1,1].set_ylabel('growth of k')

axes[0,0].legend(loc='upper left', frameon=True, fontsize=12)
axes[1,0].legend(loc='upper left', frameon=True, fontsize=12)
print(g_k[99])
print(g_p[99])
```

We observe, that the different values of $s_m$, gives a trade-off problem. Looking at the upper-graphs, we see that the function with $s_m=0.4$ will accumulate the biggest amount of capital. The reason is simple, since it is the function with the biggest investment-share in capital. We also observe that all three examples converge to a constant growth-rate.

Looking at the lower graphs, the opposite is the situation, since we observe automation capital - that is the case, when we see the growth rates, but not the accumultation. The biggest investmentshare in automation, does not give the largest accumulation. The reason is simple; automation requires investments in "normal" capital, otherwise it is useless. The growth in automation, will be affected negatively, if we end up in a situation, where there is more automation capital than normal capital (see the growth rate equation above).
Our result of $s_m=0.3$, can be viewed as the compromise (or share) between investments in automation and capital.

# Conclusion

In this project we have investigated the relationship between automation capital and "normal" capital. We have solved the model using ``sympy``. Furthermore we have made numrical optimization, using bisection, with the parameter values by Prettner. Finally we use our results to plot the capital accumulations of automation and normal capital, and analyse relationship between the two forms.

## Appendix

First we divide $\dot k(t)$ and $\dot p(t)$ by k(t) and p(t), respectively. $g_x$ denote the growth rate of k and p respectively.

(1) \\[ g_k = \frac {\dot k(t)}{k(t)} = s_ms\bigg[\frac{1+p(t)}{k(t)}\bigg]^{1-\alpha}-\delta-n\\]

(2) \\[      g_p=\frac{\dot p(t)}{p(t)} =(1-s_m)s\bigg[\frac{1+p(t)}{p(t)}\bigg]^{1-\alpha}\bigg[\frac{k(t)}{p(t)}\bigg]-\delta-n \\]

We then calculate the growth rate of its growth rate denoted by $g_{g_x}$:

\\[
    log(g_k+\delta+n)=log(s_m)+log(s)+(1-\alpha)log[1+p(t)]-(1-\alpha)log[k(t)] \\
    log(g_p+\delta+n)=log(1-s_m)+log(s)+(1-\alpha)log[1+p(t)]-(1-\alpha)log[p(t)]+\alpha log[k(t)]-\alpha log[p(t)] \\
\\]

\\[g_{({g_k}+\delta+n)}=\frac{\partial log(g_k)}{\partial t}=(1-\alpha)\frac{\dot p(t)}{1+p(t)}-(1-\alpha)g_k\\]

\\[g_{({g_p}+\delta+n)}=\frac{\partial log(g_p)}{\partial t}= (1-\alpha)\frac{\dot p(t)}{1+p(t)}-(1-\alpha)g_p+\alpha g_k-\alpha g_p\\]

We then impose constant growth along the balanced growth path, which does that we can equalize $g_{({g_k}+\delta+n)}$ and $g_{({g_p}+\delta+n)}$. By doing soo, it is *easily* seeing by reducing the equation, that $g_k=g_p$. This imply that the economy converges to a long-run growth rate with $g_p ≈ g_k ≡ g$

Note that, for large $p(t)$ and large $k(t)$, we have:
\\[ (\frac{1+p(t)}{p(t)} )^{1-\alpha} ≈ 1, \quad and \quad \frac{p(t)}{k(t)}≈\frac{1+p(t)}{k(t)}:=C\\]

With the following approximations is equation (1) and (2), respectively the growth rate of $k$ and $p$ rewritten as:

(1.a) \\[g = s_m s C^{1-\alpha}-\delta-n \\]
(2.a) \\[g = (1-s_m) s [\frac{1}{C}]^\alpha-\delta-n \\]

## Reference

Prettner, K. (2019). A NOTE ON THE IMPLICATIONS OF AUTOMATION FOR ECONOMIC GROWTH AND THE LABOR SHARE. Macroeconomic Dynamics, 23(3), 1294-1301. doi:10.1017/S1365100517000098

## Graphs of Klaus Prettner

To be sure that our results were correct, we have reproduced his graphs with our python code.

```python
#Extra code - Copy of Klaus Prettner's graphs turned into python code.

# Allocate memory for time series
series_length=100

k = np.empty(series_length)
p = np.empty(series_length)
y = np.empty(series_length)
g_k = np.empty(series_length)
g_p = np.empty(series_length)
g_y = np.empty(series_length)

fig, axes = plt.subplots(3, 2, figsize=(12, 15))

# Trajectories with different s_m
alpha = 1/3
delta = 0.04
s = (0.21, 0.25)
s_m = 0.7
#s_m = (0.3, 0.7, 0.9)
n = 0.009

for j in range(2):
    k[0] = 1
    p[0] = 1
    y[0] = 1

    for t in range(series_length-1):
        #k[t+1] = s_m[j] * s * (1 + p[t])**(1 - alpha) * k[t]**(alpha) - delta * k[t] - n * k[t]+k[t] #Plot til forskellige s_m-værdier
        k[t+1] = s_m * s[j] * (1 + p[t])**(1 - alpha) * k[t]**(alpha) +(1- delta) * k[t] - n * k[t]
        p[t+1] = (1-s_m) * s[j] * (1 + p[t])**(1 - alpha) * k[t]**(alpha) + (1- delta) * p[t] - n * p[t]
        y[t+1] = (1+p[t])**(1-alpha)*k[t]**alpha+y[t]
    axes[0,0].plot(k, '-', label=rf"$\alpha = {alpha:.3f},\; s = {s[j]},\; \delta={delta}$")
    axes[1,0].plot(p, '-', label=rf"$\alpha = {alpha:.3f},\; s = {s[j]},\; \delta={delta}$")
    axes[2,0].plot(y, '-', label=rf"$\alpha = {alpha:.3f},\; s = {s[j]},\; \delta={delta}$")

for j in range(2):
    g_k[0] = 1
    g_p[0] = 1
    g_y[0] = 1
    for t in range(series_length-1):
        g_k[t+1] = s_m * s[j] * ((1+p[t])/(k[t]))**(1-alpha) - delta - n
        g_p[t+1] = (1-s_m) * s[j] * ((1+p[t])/p[t])**(1-alpha) *  (k[t]/p[t])**alpha - delta - n
        g_y[t+1] = (1-alpha)*g_p[t]+alpha*g_k[t]
    axes[0,1].plot(g_k, '-', label=rf"$\alpha = {alpha:.3f},\; s = {s[j]},\; \delta={delta}$")
    axes[1,1].plot(g_p, '-', label=rf"$\alpha = {alpha:.3f},\; s = {s[j]},\; \delta={delta}$")
    axes[2,1].plot(g_y, '-', label=rf"$\alpha = {alpha:.3f},\; s = {s[j]},\; \delta={delta}$")

axes[0,0].grid(lw=0.2)
axes[1,0].grid(lw=0.2)
axes[0,1].grid(lw=0.2)
axes[1,1].grid(lw=0.2)

#ajust limmit on the y and x axis.
axes[0,0].set_ylim(0, 600)
axes[0,1].set_xlim(1,series_length)
axes[1,1].set_xlim(1,series_length)
axes[2,1].set_xlim(1,series_length)
axes[0,1].set_ylim(0.0,0.3)
axes[1,1].set_ylim(0.0,0.3)

axes[0,0].set_xlabel('time')
axes[0,0].set_ylabel('capital')
axes[1,0].set_xlabel('time')
axes[1,0].set_ylabel('automation')
axes[0,1].set_xlabel('time')
axes[0,1].set_ylabel('growth of k')
axes[1,1].set_xlabel('time')
axes[1,1].set_ylabel('growth of k')

axes[0,0].legend(loc='upper left', frameon=True, fontsize=12)
axes[1,0].legend(loc='upper left', frameon=True, fontsize=12)
print(g_k[99])
print(g_p[99])
```

```python

```


```
!pip install --upgrade seaborn
```

    Requirement already up-to-date: seaborn in /usr/local/lib/python3.6/dist-packages

```
import struct

import numpy as np
import matplotlib
import matplotlib.pyplot as plt
import seaborn as sns
from urllib import request
from pylab import rcParams
from scipy.special import expit

%matplotlib inline
%config InlineBackend.figure_format = 'retina'

flatui = ["#9b59b6", "#3498db", "#95a5a6", "#e74c3c", "#34495e", "#2ecc71"]

sns.set(style='whitegrid', palette='colorblind', font_scale=1.5)

rcParams['figure.figsize'] = 10, 8

RANDOM_SEED = 42

np.random.seed(RANDOM_SEED)
```

```
!mkdir data
```

    mkdir: cannot create directory ‘data’: File exists

```
filenames = [
"train-images-idx3-ubyte.gz",
"t10k-images-idx3-ubyte.gz",
"train-labels-idx1-ubyte.gz",
"t10k-labels-idx1-ubyte.gz"
]

data_dir = "data"

def download_mnist():
    base_url = "http://yann.lecun.com/exdb/mnist/"
    for f in filenames:
        print(f"Downloading {f}...")
        request.urlretrieve(base_url+f, f"{data_dir}/{f}")
    print("Download complete.")
```

```
download_mnist()
```

    Downloading train-images-idx3-ubyte.gz...
    Downloading t10k-images-idx3-ubyte.gz...
    Downloading train-labels-idx1-ubyte.gz...
    Downloading t10k-labels-idx1-ubyte.gz...
    Download complete.

```
!ls data/
```

    t10k-images-idx3-ubyte.gz  train-images-idx3-ubyte.gz
    t10k-labels-idx1-ubyte.gz  train-labels-idx1-ubyte.gz

```
def read_mnist(images_path, labels_path):
    """ Read the MNIST dataset from ubyte files """
    with open(labels_path, 'rb') as p:
        magic, n = struct.unpack('>II', p.read(8))
        labels = np.fromfile(p, dtype=np.uint8)
    with open(images_path, 'rb') as p:
        magic, num, rows, cols = struct.unpack(">IIII", p.read(16))
        images = np.fromfile(p, dtype=np.uint8).reshape(len(labels), 784)

    return images, labels
```

```
def shuffle_data(features, labels, random_seed=None):
    assert len(features) == len(labels)

    if random_seed:
        np.random.seed(random_seed)
    idx = np.random.permutation(len(features))
    return [a[idx] for a in [features, labels]]
```

# Background

## The Sigmoid function

The sigmoid function is used quite commonly in the realm of deep learning, at least it was until recently. It has distinct S shape and it is a differentiable real function for any real input value. Additionally, it has a positive derivative at each point. More importantly, we will use it as an activation function for the hidden layer of our model. Here's how it is defined:

$$\sigma (x) = \frac{1}{1+e^{-x}}$$

It’s first derivative (which we will use during the backpropagation step of our training algorithm) has the following formula:

$$\dfrac{d}{dx}\sigma(x) = \sigma(x)(1 - \sigma(x))$$

So, the derivative can be expressed using the original sigmoid function. Pretty cool, eh? Here is a more detailed derivation (thanks to [Mike Percy](https://math.stackexchange.com/a/1225116/)):

$$
\begin{align}
\dfrac{d}{dx} \sigma(x) &= \dfrac{d}{dx} \left[ \dfrac{1}{1 + e^{-x}} \right] \\
&= \dfrac{d}{dx} \left( 1 + \mathrm{e}^{-x} \right)^{-1} \\
&= -(1 + e^{-x})^{-2}(-e^{-x}) \\
&= \dfrac{e^{-x}}{\left(1 + e^{-x}\right)^2} \\
&= \dfrac{1}{1 + e^{-x}\ } \cdot \dfrac{e^{-x}}{1 + e^{-x}}  \\
&= \dfrac{1}{1 + e^{-x}\ } \cdot \dfrac{(1 + e^{-x}) - 1}{1 + e^{-x}}  \\
&= \dfrac{1}{1 + e^{-x}\ } \cdot \left( \dfrac{1 + e^{-x}}{1 + e^{-x}} - \dfrac{1}{1 + e^{-x}} \right) \\
&= \dfrac{1}{1 + e^{-x}\ } \cdot \left( 1 - \dfrac{1}{1 + e^{-x}} \right) \\
&= \sigma(x) \cdot (1 - \sigma(x))
\end{align}
$$

Proof of why $\dfrac{d}{dx}e^x = e^x$ can be found [here](https://www.khanacademy.org/math/in-in-grade-11-ncert/in-in-class11-differentiation/copy-of-sine-and-cosine-derivatives-ab/v/proof-of-derivative-of-ex). Don’t like formulas? Let’s look at a picture:

```
def sigmoid(z):
    # return 1.0 / (1.0 + np.exp(-z))
    return expit(z)

def sigmoid_prime(z):
    sg = sigmoid(z)
    return sg * (1 - sg)
```

```
x = np.linspace(-10., 10., num=100)
sig = sigmoid(x)
sig_prime = sigmoid_prime(x)

plt.plot(x, sig, label="sigmoid")
plt.plot(x, sig_prime, label="sigmoid prime")
plt.xlabel("x")
plt.ylabel("y")
plt.legend(prop={'size' : 16})
plt.show()
```

The derivative shows us the rate of change of a function. We can use it to determine the “slope” of that function. The highest rate of change for the sigmoid function is when $x=0$, as it is evident from the derivative graph (in green).

## Softmax

The softmax function can be easily differentiated, it is pure (output depends only on input) and the elements of the resulting vector sum to 1. Here it is:

$$\sigma(z)_j = \frac{e^{z_j}}{\sum_{K}^{k=1}e^{z_k}} \text{for}\,j = 1,...,k$$

In probability theory, the output of the softmax function is sometimes used as a representation of a categorical distribution. Let’s see an example result:

```
def softmax(z):
    return (np.exp(z.T) / np.sum(np.exp(z), axis=1)).T
```

```
softmax(np.array([[2, 4, 6, 8]]))
```

    array([[0.00214401, 0.0158422 , 0.11705891, 0.86495488]])

The output has most of its weight corresponding to the input 8. The softmax function highlights the largest value(s) and suppresses the smaller ones.

## Backpropagation

Backpropagation is the backbone of almost anything we do when using Neural Networks. The algorithm consists of 3 subtasks:

- Make a forward pass
- Calculate the error
- Make backward pass (backpropagation)

In the first step, backprop uses the data and the weights of the network to compute a prediction. Next, the error is computed based on the prediction and the provided labels. The final step propagates the error through the network, starting from the final layer. Thus, the weights get updated based on the error, little by little.

# Learning the XOR function

Let's try to create a Neural Network (NN) with 1 hidden layer that can properly predict values from the `XOR` function. Here is the truth table:

| A 	| B 	| Output 	|
|---	|---	|--------	|
| 0 	| 0 	| 0      	|
| 0 	| 1 	| 1      	|
| 1 	| 0 	| 1      	|
| 1 	| 1 	| 0      	|

Visually, we have:

## Data

We will use the data from the truth table above and convert it to numpy arrays:

```
X = np.array([[0,0], [0,1], [1,0], [1,1]])
y = np.array([ [0],   [1],   [1],   [0]])
```

## Setup

Let's define our Neural Net structure, learning rate and for how long we're going to train it:

```
epochs = 50000
input_size, hidden_size, output_size = 2, 3, 1
learning_rate = 0.1
```

Let's have a look at our Neural Net:

[Source](http://outlace.com/Gradient-Descent.html)

We have 2 weight matrices $\Theta_1$ and $\Theta_2$ holding the weight values between the input/hidden layer and hidden/output layer, respectively.

## Weight Initialization

We will initialize the weight of our NN using small random numbers:

```
Theta_hidden = np.random.uniform(size=(input_size, hidden_size))
Theta_output = np.random.uniform(size=(hidden_size, output_size))
```

### Untrained NN performance

Let's have a look at what our model can predict while it is still untrained:

```
# @author Denny Britz http://blog.dennybritz.com/
def plot_decision_boundary(pred_func, y):

  # Set min and max values and give it some padding
  x_min, x_max = X[:, 0].min() - .5, X[:, 0].max() + .5
  y_min, y_max = X[:, 1].min() - .5, X[:, 1].max() + .5
  h = 0.01
  # Generate a grid of points with distance h between them
  xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))
  # Predict the function value for the whole gid
  Z = pred_func(np.c_[xx.ravel(), yy.ravel()])
  Z = Z.reshape(xx.shape)
  # Plot the contour and training examples
  plt.contourf(xx, yy, Z, cmap=plt.cm.Spectral)
  plt.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.Spectral)
```

```
def predict_untrained(X):
  global Theta_hidden, Theta_output
  act_hidden = sigmoid(np.dot(X, Theta_hidden))
  return np.dot(act_hidden, Theta_output)

plot_decision_boundary(lambda x: predict_untrained(x), y.squeeze())
plt.xlabel("A")
plt.ylabel("B")
plt.title("Untrained Neural Network");
plt.show();
```

## Training

Finally, it is time to implement the **Backpropagation** algorithm:

### Loss function

First, let's define our loss function which will simply subtract the predicted label from the true one:

```
def criterion(y_hat, y):
  return y - y_hat
```

### Backpropagation

We can now train our model via the Backpropagation algorihm. It is time to use those derivates:

```
for epoch in range(epochs):

  # Forward pass (forward method in PyTorch modules)

  hidden_activation = sigmoid(np.dot(X, Theta_hidden))
  y_hat = np.dot(hidden_activation, Theta_output)

  # Calculate the error between the predictions and real values

  loss = criterion(y_hat, y)

  # Backward pass - use the error to nudge the weights of our NN so that it can make better predictions (layer by layer)

  delta_output = learning_rate * loss

  Theta_output += hidden_activation.T.dot(delta_output)

  delta_hidden = delta_output.dot(Theta_output.T) * sigmoid_prime(hidden_activation)

  Theta_hidden += X.T.dot(delta_hidden)

  if epoch % 5000 == 0:
    print(f'epoch {epoch}: error {np.mean(loss)}')
```

    epoch 0: error -0.44374003998462475
    epoch 5000: error 0.001466412518818322
    epoch 10000: error 0.001314247471488686
    epoch 15000: error 9.062613449728474e-05
    epoch 20000: error -0.00016164294643256083
    epoch 25000: error 0.00047382994733019546
    epoch 30000: error 9.477546220049149e-09
    epoch 35000: error 1.8540724511240114e-13
    epoch 40000: error 3.3029134982598407e-15
    epoch 45000: error 3.3029134982598407e-15

The backward pass works like this:

- Starting at the output neurons, we calculate their delta, $\delta$. Delta's represent the error for each node in the network. The delta of the output neuron is simply $\delta_1 = (h(x) - y)$ which is the output of the output neuron minus the real value $y$.

- To calculate the previous layer's delta, we backpropagate the deltas using the formula:

$$\delta_j^{l-1} = (\theta_2 . \delta_j^l) \odot (a^{l-1} \odot (1 - a^{l-1}))$$

where $.$ is dot product, $\odot$ is element-wise multiplication and $l$ is the number of layers

## Prediction

Our error seems to be decreasing! Let's try to use our trained model for prediction, doing just a forward step:

```
def predict(X):
  hidden_activation = sigmoid(np.dot(X, Theta_hidden))
  return np.dot(hidden_activation, Theta_output)
```

```
X_test = X[1] # [0, 1]

predict(X_test)
```

    array([1.])

Hmm, that seems correct!

```
plot_decision_boundary(lambda x: predict(x), y.squeeze())
plt.xlabel("A")
plt.ylabel("B")
plt.title("Trained Neural Network");
plt.show();
```


# A demonstration of SuSiE's motivations

This document explains with toy example illustration the unique type of inference SuSiE is interested in.

## The inference problem

We assume our audience are familiar or interested in large scale regression. Similar to eg LASSO, SuSiE is a method for variable selection in large scale regression. Yet the type of inference SuSiE attempts to accomplish is different from other large scale regression methods, and is worth motivating in this document for those not familiar with this particular type of inference.

The type of inference is motivated by the so-called "genetic fine-mapping" study. Consider fitting the regression model $$y = \sum_{j=1}^p x_j\beta_j + \epsilon \quad \epsilon \sim N(0, \sigma^2I_n)$$ where $x_1 = x_2, x_3 = x_4$ and
$\beta_1 \ne 0, \beta_4 \ne 0, \beta_{j \notin \{1,4\}} = 0$. Our goal is to make a statement that $$\beta_1 \ne 0 \text{ or } \beta_2 \ne 0, \text{ and } \beta_3 \ne 0 \text{ or } \beta_4 \ne 0,$$

that is, in SuSiE variable selection we want to capture the fact that variables $x_1$ and $x_2$ (likewise $x_3$ and $x_4$) are too similar to distinguish. This is an inference that to our knowledge few other large scale regression can make; at least not in an intuitive and interpretable fashion.

To illustrate here I set $p=1000$ and simulate a data-set:

```sos
set.seed(1)
n = 500
p = 1000
b = rep(0,p)
b[200] = 1
b[800] = 1
X = matrix(rnorm(n*p),nrow=n,ncol=p)
X[,200] = X[,400]
X[,600] = X[,800]
y = X %*% b + rnorm(n)
```

The "true" effects are:

```sos
pdf('truth.pdf', width =5, height = 5, pointsize=16)
plot(b, col="black", pch=16, main = 'True effect size')
pos = 1:length(b)
points(pos[b!=0],b[b!=0],col=2,pch=16)
dev.off()
```

png: 2

```sos
%preview truth.pdf -s png --dpi 100
```

## LASSO inference

LASSO will simply choose one of:

- $x_1, x_3$
- $x_1, x_4$
- $x_2, x_3$
- $x_2, x_4$

This is not what we want. For instance on the simulated data,

```sos
alpha = 1
y.fit = glmnet::glmnet(X,y,alpha = alpha,intercept = FALSE)
y.cv  = glmnet::cv.glmnet(X,y,alpha = alpha,intercept = FALSE,
                         lambda = y.fit$lambda)
bhat  = glmnet::predict.glmnet(y.fit,type ="coefficients",
                               s = y.cv$lambda.min)[-1,1]
```

```sos
pdf('lasso.pdf', width =5, height = 5, pointsize=16)
plot(bhat, col="black", pch=16, main = 'Lasso')
pos = 1:length(bhat)
points(pos[b!=0],bhat[b!=0],col=2,pch=16)
dev.off()
```

png: 2

```sos
%preview lasso.pdf -s png --dpi 100
```

LASSO randomly picked $x_1$ and $x_3$ even though the true non-zero effects are $\beta_1$ and $\beta_4$.

## Existing Bayesian methods for sparse regression

For sparse enough problems, most other Bayesian methods provide posterior probabilities on "models" by enumerating (CAVIAR), schochastic search (FINEMAP) or sampling (BIMBAM) from all possible combinations of variables. In our motivating example, the "true" posterior for models are
\begin{align}
p(\beta_1 \ne 0, \beta_2 = 0, \beta_3 \ne 0, \beta_4 = 0, \beta_{5:p} = 0) &= 0.25 \\
p(\beta_1 = 0, \beta_2 \ne 0, \beta_3 \ne 0, \beta_4 = 0, \beta_{5:p} = 0) &= 0.25 \\
p(\beta_1 \ne 0, \beta_2 = 0, \beta_3 = 0, \beta_4 \ne 0, \beta_{5:p} = 0) &= 0.25 \\
p(\beta_1 = 0, \beta_2 \ne 0, \beta_3 = 0, \beta_4 \ne 0, \beta_{5:p} = 0) &= 0.25.
\end{align}
Although compared to LASSO the model posterior contains the information to make statements about $\beta_1 \ne 0 \text{ or } \beta_2 \ne 0, \text{ and } \beta_3 \ne 0 \text{ or } \beta_4 \ne 0$, it is not explicitly provided. One has to post-process these posteriors in order to make the statement. Such pre-processing is non-trivial, see eg Stephens and Balding 2009 for an example in the context of genetic fine-mapping.

In addition, marginalized probabilities are often provided by these Bayesian methods, calcualted as
\begin{align}
p(\beta_1) &= p(\beta_1 \ne 0, \beta_2 = 0, \beta_3 \ne 0, \beta_4 = 0, \beta_{5:p} = 0) \\
& + p(\beta_1 \ne 0, \beta_2 = 0, \beta_3 = 0, \beta_4 \ne 0, \beta_{5:p} = 0)  \\
& = 0.5,
\end{align}
but this does not provide the type of inference we are interested in.

We illustrate Bayesian variable selection using `FINEMAP`. First we compute "summary statistics" that `FINEMAP` requires,

```sos
library(abind)
mm_regression = function(X, Y, Z=NULL) {
  if (!is.null(Z)) {
      Z = as.matrix(Z)
  }
  reg = lapply(seq_len(ncol(Y)), function (i) simplify2array(susieR:::univariate_regression(X, Y[,i], Z)))
  reg = do.call(abind, c(reg, list(along=0)))
  # return array: out[1,,] is betahat, out[2,,] is shat
  return(aperm(reg, c(3,2,1)))
}
sumstats = mm_regression(as.matrix(X), as.matrix(y))
dat = list(X=X,Y=as.matrix(y))
saveRDS(list(data=dat, sumstats=sumstats), '/tmp/Toy.with_sumstats.rds')
```

we then set up the path containing the `finemap` executable (here set to `~/GIT/github/mvarbvs/dsc/modules/linux`) and run a wrapper script provided in the `susieR` package under `inst/code` folder (the package source can be [downloaded here](https://github.com/stephenslab/susieR)).

```sos
export PATH=~/GIT/github/mvarbvs/dsc/modules/linux:$PATH
Rscript ~/GIT/susieR/inst/code/finemap.R input=\"/tmp/Toy.with_sumstats.rds\" output=\"N2finemapping.FINEMAP\" args=\"--n-causal-max\ 2\" 2> /dev/null
```

    |---------------------------------|
    | Welcome to FINEMAP v1.1         |
    |                                 |
    | (c) 2015 University of Helsinki |
    |                                 |
    | Help / Documentation:           |
    | - ./finemap --help              |
    | - www.christianbenner.com       |
    |                                 |
    | Contact:                        |
    | - christian.benner@helsinki.fi  |
    | - matti.pirinen@helsinki.fi     |
    |---------------------------------|

    --------
    SETTINGS
    --------
    - regions         : 1
    - n-causal-max    : 2
    - n-configs-top   : 50000
    - n-iterations    : 100000
    - n-convergence   : 1000
    - prob-tol        : 0.001
    - corr-config     : 0.95
    - prior-std       : 0.05
    - prior-k0        : 0

    ---------------
    RUNNING FINEMAP (1/1)
    ---------------
    - GWAS z-scores                   : /tmp/RtmpRHQEch/file7d275e5895af.finemap_condition_1.z
    - SNP correlations                : /tmp/RtmpRHQEch/file7d272073d9a6.ld
    - Causal SNP configurations       : /tmp/RtmpRHQEch/file7d275e5895af.finemap_condition_1.config
    - Single-SNP statistics           : /tmp/RtmpRHQEch/file7d275e5895af.finemap_condition_1.snp
    - Log file                        : /tmp/RtmpRHQEch/file7d275e5895af.finemap_condition_1.log

    - Reading input                   : done!
    - Number of SNPs in region        : 1000
    - Number of individuals in GWAS   : 500
    - Prior-Pr(# of causal SNPs is k) :
      (0 -> 0)
       1 -> 0.666667
       2 -> 0.333333
    - 6979 configurations evaluated (1.004/100%) : converged after 1004 iterations
    - log10(Evidence of at least 1 causal SNP) : 60.0667
    - Post-Pr(# of causal SNPs is k)  :
       0 -> 0
       1 -> 8.6199e-22
       2 -> 1
    - Writing output                  : done!
    - Run time                        : 0 hours, 0 minutes, 4 seconds

Posterior probabilities for "models" are:

```sos
finemap = readRDS("N2finemapping.FINEMAP.rds")[[1]]
head(finemap$set)
```

rank	config	config_prob	config_log10bf

1       	400,800 	0.25    	65.64032

2       	400,600 	0.25    	65.64032

3       	200,800 	0.25    	65.64032

4       	200,600 	0.25    	65.64032

5       	200     	0.00    	41.57626

6       	400     	0.00    	41.57626

`FINEMAP` correctly determined 4 "model configurations" as expected, each with probability 0.25. Simply marginalizing across all models we obtain posterior inclusion probability per feature,

```sos
snp = finemap$snp
pip = snp[order(as.numeric(snp$snp)),]$snp_prob
```

```sos
pdf('finemap.pdf', width =5, height = 5, pointsize=16)
susieR::susie_plot(pip, y='PIP', b=b, main = 'Bayesian sparse regression')
dev.off()
```

png: 2

```sos
%preview finemap.pdf -s png --dpi 100
```

From these marginalized posterior we can only make a statement that each of the 4 identified variables have 0.5 probability of being non-zero. That is, they all have equal contributions. It does not reflect the fact that the result comes from identical feature pairs $x_1, x_2$ and $x_3, x_4$.

Existing variational inference methods (`varbvs`) do not provide accurate calculation of marginal posterior probabilities, although like LASSO, existing variational methods are good for prediction purposes.

## SuSiE

SuSiE uses a variational inference algorithm which is not only computational convenient, but also directly obtains posterior statements of the desired form.

```sos
fitted = susieR::susie(X, y, L=5,
               estimate_residual_variance=TRUE,
               scaled_prior_variance=0.2,
               tol=1e-3, track_fit=TRUE, min_abs_corr=0.1)
```

```sos
pdf('susie.pdf', width =5, height = 5, pointsize=16)
susieR::susie_plot(fitted, y='PIP', b=b, max_cs=0.4, main = paste('SuSiE, ', length(fitted$sets$cs), 'CS identified'))
dev.off()
```

png: 2

```sos
%preview susie.pdf -s png --dpi 100
```

Here SuSiE identifies 2 sets of variables -- 2 confidence sets each containing one causal variable (blue and green circles). The marginal posterior is the same as from existing Bayesian methods (FINEMAP result), but with the two sets identified one can easily state that the causal variables are ($x_1$ or $x_2$), and ($x_3$ or $x_4$). Within each set, the contributions of variables are equal because there is no information to distinguish between them.

Notice that SuSiE does not directly provide posterior of model configuration, that is, inference on ($x_1$ and $x_3$), or ($x_1$ and $x_4$), or ($x_2$ and $x_3$), or ($x_2$ and $x_4$).

Effect size estimate by SuSiE,

```sos
bhat = coef(fitted)[-1]
pdf('susie_eff.pdf', width =5, height = 5, pointsize=16)
susieR::susie_plot(bhat, y='bhat', b=b, main = 'SuSiE, effect size estimate')
dev.off()
```

png: 2

```sos
%preview susie_eff.pdf -s png --dpi 100
```

```sos
convert -density 120 \( truth.pdf lasso.pdf +append \) \( finemap.pdf susie.pdf +append \) -append Motivating_Example.png
```