# Week 9 March 8-12: Resampling Techniques, Bootstrap and Blocking

**Morten Hjorth-Jensen  Email morten.hjorth-jensen@fys.uio.no**, Department of Physics and Center fo Computing in Science Education, University of Oslo, Oslo, Norway and Department of Physics and Astronomy and Facility for Rare Ion Beams, Michigan State University, East Lansing, Michigan, USA

Date: **Mar 11, 2021**

Copyright 1999-2021, Morten Hjorth-Jensen  Email morten.hjorth-jensen@fys.uio.no. Released under CC Attribution-NonCommercial 4.0 license

## Overview of week 9, March 8-12
**Topics.**

* Resampling Techniques and statistics: Bootstrap and Blocking

**Teaching Material, videos and written material.**

* Overview video on the [Bootstrap method](https://www.youtube.com/watch?v=O_Fj4q8lgmc&ab_channel=MarinStatsLectures-RProgramming%26Statistics)

* These Lecture notes

* [Marius Johnson's Master thesis on the Blocking Method](https://www.duo.uio.no/bitstream/handle/10852/68360/PhysRevE.98.043304.pdf?sequence=2&isAllowed=y)

## Why resampling methods ?
**Statistical analysis.**

    * Our simulations can be treated as *computer experiments*. This is particularly the case for Monte Carlo methods

    * The results can be analysed with the same statistical tools as we would use analysing experimental data.

    * As in all experiments, we are looking for expectation values and an estimate of how accurate they are, i.e., possible sources for errors.

## Statistical analysis
    * As in other experiments, many numerical  experiments have two classes of errors:

      * Statistical errors

      * Systematical errors

    * Statistical errors can be estimated using standard tools from statistics

    * Systematical errors are method specific and must be treated differently from case to case.

## Statistics, wrapping up from last week
Let us analyze the problem by splitting up the correlation term into
partial sums of the form:

$$
f_d = \frac{1}{n-d}\sum_{k=1}^{n-d}(x_k - \bar x_n)(x_{k+d} - \bar x_n)
$$

The correlation term of the error can now be rewritten in terms of
$f_d$

$$
\frac{2}{n}\sum_{k<l} (x_k - \bar x_n)(x_l - \bar x_n) =
2\sum_{d=1}^{n-1} f_d
$$

The value of $f_d$ reflects the correlation between measurements
separated by the distance $d$ in the sample samples.  Notice that for
$d=0$, $f$ is just the sample variance, $\mathrm{var}(x)$. If we divide $f_d$
by $\mathrm{var}(x)$, we arrive at the so called *autocorrelation function*

$$
\kappa_d = \frac{f_d}{\mathrm{var}(x)}
$$

which gives us a useful measure of pairwise correlations
starting always at $1$ for $d=0$.

## Statistics, final expression
The sample error can now be
written in terms of the autocorrelation function:

$$
\mathrm{err}_X^2 =
\frac{1}{n}\mathrm{var}(x)+\frac{2}{n}\cdot\mathrm{var}(x)\sum_{d=1}^{n-1}
\frac{f_d}{\mathrm{var}(x)}\nonumber
$$

$$
=
\left(1+2\sum_{d=1}^{n-1}\kappa_d\right)\frac{1}{n}\mathrm{var}(x)\nonumber
$$

$$
\begin{equation}
=\frac{\tau}{n}\cdot\mathrm{var}(x)
\label{_auto1} \tag{1}
\end{equation}
$$

and we see that $\mathrm{err}_X$ can be expressed in terms the
uncorrelated sample variance times a correction factor $\tau$ which
accounts for the correlation between measurements. We call this
correction factor the *autocorrelation time*:

$$
\begin{equation}
\tau = 1+2\sum_{d=1}^{n-1}\kappa_d
\label{eq:autocorrelation_time} \tag{2}
\end{equation}
$$

## Statistics, effective number of correlations
For a correlation free experiment, $\tau$
equals 1.

We can interpret a sequential
correlation as an effective reduction of the number of measurements by
a factor $\tau$. The effective number of measurements becomes:

$$
n_\mathrm{eff} = \frac{n}{\tau}
$$

To neglect the autocorrelation time $\tau$ will always cause our
simple uncorrelated estimate of $\mathrm{err}_X^2\approx \mathrm{var}(x)/n$ to
be less than the true sample error. The estimate of the error will be
too *good*. On the other hand, the calculation of the full
autocorrelation time poses an efficiency problem if the set of
measurements is very large.

## Can we understand this? Time Auto-correlation Function

The so-called time-displacement autocorrelation $\phi(t)$ for a quantity $\mathbf{M}$ is given by

$$
\phi(t) = \int dt' \left[\mathbf{M}(t')-\langle \mathbf{M} \rangle\right]\left[\mathbf{M}(t'+t)-\langle \mathbf{M} \rangle\right],
$$

which can be rewritten as

$$
\phi(t) = \int dt' \left[\mathbf{M}(t')\mathbf{M}(t'+t)-\langle \mathbf{M} \rangle^2\right],
$$

where $\langle \mathbf{M} \rangle$ is the average value and
$\mathbf{M}(t)$ its instantaneous value. We can discretize this function as follows, where we used our
set of computed values $\mathbf{M}(t)$ for a set of discretized times (our Monte Carlo cycles corresponding to moving all electrons?)

$$
\phi(t)  = \frac{1}{t_{\mathrm{max}}-t}\sum_{t'=0}^{t_{\mathrm{max}}-t}\mathbf{M}(t')\mathbf{M}(t'+t)
-\frac{1}{t_{\mathrm{max}}-t}\sum_{t'=0}^{t_{\mathrm{max}}-t}\mathbf{M}(t')\times
\frac{1}{t_{\mathrm{max}}-t}\sum_{t'=0}^{t_{\mathrm{max}}-t}\mathbf{M}(t'+t).
\label{eq:phitf} \tag{3}
$$

## Time Auto-correlation Function

One should be careful with times close to $t_{\mathrm{max}}$, the upper limit of the sums
becomes small and we end up integrating over a rather small time interval. This means that the statistical
error in $\phi(t)$ due to the random nature of the fluctuations in $\mathbf{M}(t)$ can become large.

One should therefore choose $t \ll t_{\mathrm{max}}$.

Note that the variable $\mathbf{M}$ can be any expectation values of interest.

The time-correlation function gives a measure of the correlation between the various values of the variable
at a time $t'$ and a time $t'+t$. If we multiply the values of $\mathbf{M}$ at these two different times,
we will get a positive contribution if they are fluctuating in the same direction, or a negative value
if they fluctuate in the opposite direction. If we then integrate over time, or use the discretized version of, the time correlation function $\phi(t)$ should take a non-zero value if the fluctuations are
correlated, else it should gradually go to zero. For times a long way apart
the different values of $\mathbf{M}$  are most likely
uncorrelated and $\phi(t)$ should be zero.

## Time Auto-correlation Function
We can derive the correlation time by observing that our Metropolis algorithm is based on a random
walk in the space of all  possible spin configurations.
Our probability
distribution function $\mathbf{\hat{w}}(t)$ after a given number of time steps $t$ could be written as

$$
\mathbf{\hat{w}}(t) = \mathbf{\hat{W}^t\hat{w}}(0),
$$

with $\mathbf{\hat{w}}(0)$ the distribution at $t=0$ and $\mathbf{\hat{W}}$ representing the
transition probability matrix.
We can always expand $\mathbf{\hat{w}}(0)$ in terms of the right eigenvectors of
$\mathbf{\hat{v}}$ of $\mathbf{\hat{W}}$ as

$$
\mathbf{\hat{w}}(0)  = \sum_i\alpha_i\mathbf{\hat{v}}_i,
$$

resulting in

$$
\mathbf{\hat{w}}(t) = \mathbf{\hat{W}}^t\mathbf{\hat{w}}(0)=\mathbf{\hat{W}}^t\sum_i\alpha_i\mathbf{\hat{v}}_i=
\sum_i\lambda_i^t\alpha_i\mathbf{\hat{v}}_i,
$$

with $\lambda_i$ the $i^{\mathrm{th}}$ eigenvalue corresponding to
the eigenvector $\mathbf{\hat{v}}_i$.

## Time Auto-correlation Function
If we assume that $\lambda_0$ is the largest eigenvector we see that in the limit $t\rightarrow \infty$,
$\mathbf{\hat{w}}(t)$ becomes proportional to the corresponding eigenvector
$\mathbf{\hat{v}}_0$. This is our steady state or final distribution.

We can relate this property to an observable like the mean energy.
With the probabilty $\mathbf{\hat{w}}(t)$ (which in our case is the squared trial wave function) we
can write the expectation values as

$$
\langle \mathbf{M}(t) \rangle  = \sum_{\mu} \mathbf{\hat{w}}(t)_{\mu}\mathbf{M}_{\mu},
$$

or as the scalar of a  vector product

$$
\langle \mathbf{M}(t) \rangle  = \mathbf{\hat{w}}(t)\mathbf{m},
$$

with $\mathbf{m}$ being the vector whose elements are the values of $\mathbf{M}_{\mu}$ in its
various microstates $\mu$.

## Time Auto-correlation Function

We rewrite this relation  as

$$
\langle \mathbf{M}(t) \rangle  = \mathbf{\hat{w}}(t)\mathbf{m}=\sum_i\lambda_i^t\alpha_i\mathbf{\hat{v}}_i\mathbf{m}_i.
$$

If we define $m_i=\mathbf{\hat{v}}_i\mathbf{m}_i$ as the expectation value of
$\mathbf{M}$ in the $i^{\mathrm{th}}$ eigenstate we can rewrite the last equation as

$$
\langle \mathbf{M}(t) \rangle  = \sum_i\lambda_i^t\alpha_im_i.
$$

Since we have that in the limit $t\rightarrow \infty$ the mean value is dominated by the
the largest eigenvalue $\lambda_0$, we can rewrite the last equation as

$$
\langle \mathbf{M}(t) \rangle  = \langle \mathbf{M}(\infty) \rangle+\sum_{i\ne 0}\lambda_i^t\alpha_im_i.
$$

We define the quantity

$$
\tau_i=-\frac{1}{log\lambda_i},
$$

and rewrite the last expectation value as

$$
\langle \mathbf{M}(t) \rangle  = \langle \mathbf{M}(\infty) \rangle+\sum_{i\ne 0}\alpha_im_ie^{-t/\tau_i}.
\label{eq:finalmeanm} \tag{4}
$$

## Time Auto-correlation Function

The quantities $\tau_i$ are the correlation times for the system. They control also the auto-correlation function
discussed above.  The longest correlation time is obviously given by the second largest
eigenvalue $\tau_1$, which normally defines the correlation time discussed above. For large times, this is the
only correlation time that survives. If higher eigenvalues of the transition matrix are well separated from
$\lambda_1$ and we simulate long enough,  $\tau_1$ may well define the correlation time.
In other cases we may not be able to extract a reliable result for $\tau_1$.
Coming back to the time correlation function $\phi(t)$ we can present a more general definition in terms
of the mean magnetizations $ \langle \mathbf{M}(t) \rangle$. Recalling that the mean value is equal
to $ \langle \mathbf{M}(\infty) \rangle$ we arrive at the expectation values

$$
\phi(t) =\langle \mathbf{M}(0)-\mathbf{M}(\infty)\rangle \langle \mathbf{M}(t)-\mathbf{M}(\infty)\rangle,
$$

resulting in

$$
\phi(t) =\sum_{i,j\ne 0}m_i\alpha_im_j\alpha_je^{-t/\tau_i},
$$

which is appropriate for all times.

## Correlation Time

If the correlation function decays exponentially

$$
\phi (t) \sim \exp{(-t/\tau)}
$$

then the exponential correlation time can be computed as the average

$$
\tau_{\mathrm{exp}}  =  -\langle  \frac{t}{log|\frac{\phi(t)}{\phi(0)}|} \rangle.
$$

If the decay is exponential, then

$$
\int_0^{\infty} dt \phi(t)  = \int_0^{\infty} dt \phi(0)\exp{(-t/\tau)}  = \tau \phi(0),
$$

which  suggests another measure of correlation

$$
\tau_{\mathrm{int}} = \sum_k \frac{\phi(k)}{\phi(0)},
$$

called the integrated correlation time.

## Resampling methods: Jackknife and Bootstrap

Two famous
resampling methods are the **independent bootstrap** and **the jackknife**.

The jackknife is a special case of the independent bootstrap. Still, the jackknife was made
popular prior to the independent bootstrap. And as the popularity of
the independent bootstrap soared, new variants, such as **the dependent bootstrap**.

The Jackknife and independent bootstrap work for
independent, identically distributed random variables.
If these conditions are not
satisfied, the methods will fail.  Yet, it should be said that if the data are
independent, identically distributed, and we only want to estimate the
variance of $\overline{X}$ (which often is the case), then there is no
need for bootstrapping.

## Resampling methods: Jackknife

The Jackknife works by making many replicas of the estimator $\widehat{\theta}$.
The jackknife is a resampling method, we explained that this happens by scrambling the data in some way. When using the jackknife, this is done by systematically leaving out one observation from the vector of observed values $\hat{x} = (x_1,x_2,\cdots,X_n)$.
Let $\hat{x}_i$ denote the vector

$$
\hat{x}_i = (x_1,x_2,\cdots,x_{i-1},x_{i+1},\cdots,x_n),
$$

which equals the vector $\hat{x}$ with the exception that observation
number $i$ is left out. Using this notation, define
$\widehat{\theta}_i$ to be the estimator
$\widehat{\theta}$ computed using $\vec{X}_i$.

## Resampling methods: Jackknife estimator

To get an estimate for the bias and
standard error of $\widehat{\theta}$, use the following
estimators for each component of $\widehat{\theta}$

$$
\widehat{\mathrm{Bias}}(\widehat \theta,\theta) = (n-1)\left( - \widehat{\theta} + \frac{1}{n}\sum_{i=1}^{n} \widehat \theta_i \right) \qquad \text{and} \qquad \widehat{\sigma}^2_{\widehat{\theta} } = \frac{n-1}{n}\sum_{i=1}^{n}( \widehat{\theta}_i - \frac{1}{n}\sum_{j=1}^{n}\widehat \theta_j )^2.
$$

## Jackknife code example

```python
from numpy import *
from numpy.random import randint, randn
from time import time

def jackknife(data, stat):
    n = len(data);t = zeros(n); inds = arange(n); t0 = time()
    ## 'jackknifing' by leaving out an observation for each i
    for i in range(n):
        t[i] = stat(delete(data,i) )

    # analysis
    print("Runtime: %g sec" % (time()-t0)); print("Jackknife Statistics :")
    print("original           bias      std. error")
    print("%8g %14g %15g" % (stat(data),(n-1)*mean(t)/n, (n*var(t))**.5))

    return t

# Returns mean of data samples
def stat(data):
    return mean(data)

mu, sigma = 100, 15
datapoints = 10000
x = mu + sigma*random.randn(datapoints)
# jackknife returns the data sample
t = jackknife(x, stat)
```

## Resampling methods: Bootstrap
Bootstrapping is a nonparametric approach to statistical inference
that substitutes computation for more traditional distributional
assumptions and asymptotic results. Bootstrapping offers a number of
advantages:
1. The bootstrap is quite general, although there are some cases in which it fails.

2. Because it does not require distributional assumptions (such as normally distributed errors), the bootstrap can provide more accurate inferences when the data are not well behaved or when the sample size is small.

3. It is possible to apply the bootstrap to statistics with sampling distributions that are difficult to derive, even asymptotically.

4. It is relatively simple to apply the bootstrap to complex data-collection plans (such as stratified and clustered samples).

## Resampling methods: Bootstrap background

Since $\widehat{\theta} = \widehat{\theta}(\hat{X})$ is a function of random variables,
$\widehat{\theta}$ itself must be a random variable. Thus it has
a pdf, call this function $p(\hat{t})$. The aim of the bootstrap is to
estimate $p(\hat{t})$ by the relative frequency of
$\widehat{\theta}$. You can think of this as using a histogram
in the place of $p(\hat{t})$. If the relative frequency closely
resembles $p(\vec{t})$, then using numerics, it is straight forward to
estimate all the interesting parameters of $p(\hat{t})$ using point
estimators.

## Resampling methods: More Bootstrap background

In the case that $\widehat{\theta}$ has
more than one component, and the components are independent, we use the
same estimator on each component separately.  If the probability
density function of $X_i$, $p(x)$, had been known, then it would have
been straight forward to do this by:
1. Drawing lots of numbers from $p(x)$, suppose we call one such set of numbers $(X_1^*, X_2^*, \cdots, X_n^*)$.

2. Then using these numbers, we could compute a replica of $\widehat{\theta}$ called $\widehat{\theta}^*$.

By repeated use of (1) and (2), many
estimates of $\widehat{\theta}$ could have been obtained. The
idea is to use the relative frequency of $\widehat{\theta}^*$
(think of a histogram) as an estimate of $p(\hat{t})$.

## Resampling methods: Bootstrap approach

But
unless there is enough information available about the process that
generated $X_1,X_2,\cdots,X_n$, $p(x)$ is in general
unknown. Therefore, [Efron in 1979](https://projecteuclid.org/euclid.aos/1176344552)  asked the
question: What if we replace $p(x)$ by the relative frequency
of the observation $X_i$; if we draw observations in accordance with
the relative frequency of the observations, will we obtain the same
result in some asymptotic sense? The answer is yes.

Instead of generating the histogram for the relative
frequency of the observation $X_i$, just draw the values
$(X_1^*,X_2^*,\cdots,X_n^*)$ with replacement from the vector
$\hat{X}$.

## Resampling methods: Bootstrap steps

The independent bootstrap works like this:

1. Draw with replacement $n$ numbers for the observed variables $\hat{x} = (x_1,x_2,\cdots,x_n)$.

2. Define a vector $\hat{x}^*$ containing the values which were drawn from $\hat{x}$.

3. Using the vector $\hat{x}^*$ compute $\widehat{\theta}^*$ by evaluating $\widehat \theta$ under the observations $\hat{x}^*$.

4. Repeat this process $k$ times.

When you are done, you can draw a histogram of the relative frequency of $\widehat \theta^*$. This is your estimate of the probability distribution $p(t)$. Using this probability distribution you can estimate any statistics thereof. In principle you never draw the histogram of the relative frequency of $\widehat{\theta}^*$. Instead you use the estimators corresponding to the statistic of interest. For example, if you are interested in estimating the variance of $\widehat \theta$, apply the etsimator $\widehat \sigma^2$ to the values $\widehat \theta ^*$.

## Code example for the Bootstrap method

The following code starts with a Gaussian distribution with mean value $\mu =100$ and variance $\sigma=15$. We use this to generate the data used in the bootstrap analysis. The bootstrap analysis returns a data set after a given number of bootstrap operations (as many as we have data points). This data set consists of estimated mean values for each bootstrap operation. The histogram generated by the bootstrap method shows that the distribution for these mean values is also a Gaussian, centered around the mean value $\mu=100$ but with standard deviation $\sigma/\sqrt{n}$, where $n$ is the number of bootstrap samples (in this case the same as the number of original data points). The value of the standard deviation is what we expect from the central limit theorem.

```python
%matplotlib inline

%matplotlib inline

from numpy import *
from numpy.random import randint, randn
from time import time
from scipy.stats import norm
import matplotlib.pyplot as plt

# Returns mean of bootstrap samples
def stat(data):
    return mean(data)

# Bootstrap algorithm
def bootstrap(data, statistic, R):
    t = zeros(R); n = len(data); inds = arange(n); t0 = time()

    # non-parametric bootstrap
    for i in range(R):
        t[i] = statistic(data[randint(0,n,n)])

    # analysis
    print("Runtime: %g sec" % (time()-t0)); print("Bootstrap Statistics :")
    print("original           bias      std. error")
    print("%8g %8g %14g %15g" % (statistic(data), std(data),\
                             mean(t), \
                             std(t)))
    return t

mu, sigma = 100, 15
datapoints = 100000
x = mu + sigma*random.randn(datapoints)
# bootstrap returns the data sample                                                                                t = bootstrap(x, stat, datapoints)
# the histogram of the bootstrapped  data
t = bootstrap(x, stat, datapoints)
# the histogram of the bootstrapped  data
n, binsboot, patches = plt.hist(t, bins=50, density='true',histtype='bar', color='red', alpha=0.75)

# add a 'best fit' line
y = norm.pdf( binsboot, mean(t), std(t))
lt = plt.plot(binsboot, y, 'r--', linewidth=1)
plt.xlabel('Smarts')
plt.ylabel('Probability')
plt.axis([99.5, 100.6, 0, 3.0])
plt.grid(True)

plt.show()
```

## Resampling methods: Blocking

The blocking method was made popular by [Flyvbjerg and Pedersen (1989)](https://aip.scitation.org/doi/10.1063/1.457480)
and has become one of the standard ways to estimate
$V(\widehat{\theta})$ for exactly one $\widehat{\theta}$, namely
$\widehat{\theta} = \overline{X}$.

Assume $n = 2^d$ for some integer $d>1$ and $X_1,X_2,\cdots, X_n$ is a stationary time series to begin with.
Moreover, assume that the time series is asymptotically uncorrelated. We switch to vector notation by arranging $X_1,X_2,\cdots,X_n$ in an $n$-tuple. Define:

$$
\begin{align*}
\hat{X} = (X_1,X_2,\cdots,X_n).
\end{align*}
$$

The strength of the blocking method is when the number of
observations, $n$ is large. For large $n$, the complexity of dependent
bootstrapping scales poorly, but the blocking method does not,
moreover, it becomes more accurate the larger $n$ is.

## Blocking Transformations
 We now define
blocking transformations. The idea is to take the mean of subsequent
pair of elements from $\vec{X}$ and form a new vector
$\vec{X}_1$. Continuing in the same way by taking the mean of
subsequent pairs of elements of $\vec{X}_1$ we obtain $\vec{X}_2$, and
so on.
Define $\vec{X}_i$ recursively by:

$$
(\vec{X}_0)_k \equiv (\vec{X})_k \nonumber
$$

$$
\begin{equation}
(\vec{X}_{i+1})_k \equiv \frac{1}{2}\Big( (\vec{X}_i)_{2k-1} +
(\vec{X}_i)_{2k} \Big) \qquad \text{for all} \qquad 1 \leq i \leq d-1
\label{_auto2} \tag{5}
\end{equation}
$$

The quantity $\vec{X}_k$ is
subject to $k$ **blocking transformations**.  We now have $d$ vectors
$\vec{X}_0, \vec{X}_1,\cdots,\vec X_{d-1}$ containing the subsequent
averages of observations. It turns out that if the components of
$\vec{X}$ is a stationary time series, then the components of
$\vec{X}_i$ is a stationary time series for all $0 \leq i \leq d-1$

We can then compute the autocovariance, the variance, sample mean, and
number of observations for each $i$.
Let $\gamma_i, \sigma_i^2,
\overline{X}_i$ denote the autocovariance, variance and average of the
elements of $\vec{X}_i$ and let $n_i$ be the number of elements of
$\vec{X}_i$. It follows by induction that $n_i = n/2^i$.

## Blocking Transformations

Using the
definition of the blocking transformation and the distributive
property of the covariance, it is clear that since $h =|i-j|$
we can define

$$
\gamma_{k+1}(h) = cov\left( ({X}_{k+1})_{i}, ({X}_{k+1})_{j} \right) \nonumber
$$

$$
=  \frac{1}{4}cov\left( ({X}_{k})_{2i-1} + ({X}_{k})_{2i}, ({X}_{k})_{2j-1} + ({X}_{k})_{2j} \right) \nonumber
$$

$$
\begin{equation}
=  \frac{1}{2}\gamma_{k}(2h) + \frac{1}{2}\gamma_k(2h+1) \hspace{0.1cm} \mathrm{h = 0}
\label{_auto3} \tag{6}
\end{equation}
$$

$$
\begin{equation}
=\frac{1}{4}\gamma_k(2h-1) + \frac{1}{2}\gamma_k(2h) + \frac{1}{4}\gamma_k(2h+1) \quad \mathrm{else}
\label{_auto4} \tag{7}
\end{equation}
$$

The quantity $\hat{X}$ is asymptotic uncorrelated by assumption, $\hat{X}_k$ is also asymptotic uncorrelated. Let's turn our attention to the variance of the sample mean $V(\overline{X})$.

## Blocking Transformations, getting there
We have

$$
\begin{equation}
V(\overline{X}_k) = \frac{\sigma_k^2}{n_k} + \underbrace{\frac{2}{n_k} \sum_{h=1}^{n_k-1}\left( 1 - \frac{h}{n_k} \right)\gamma_k(h)}_{\equiv e_k} = \frac{\sigma^2_k}{n_k} + e_k \quad \text{if} \quad \gamma_k(0) = \sigma_k^2.
\label{_auto5} \tag{8}
\end{equation}
$$

The term $e_k$ is called the **truncation error**:

$$
\begin{equation}
e_k = \frac{2}{n_k} \sum_{h=1}^{n_k-1}\left( 1 - \frac{h}{n_k} \right)\gamma_k(h).
\label{_auto6} \tag{9}
\end{equation}
$$

We can show that $V(\overline{X}_i) = V(\overline{X}_j)$ for all $0 \leq i \leq d-1$ and $0 \leq j \leq d-1$.

## Blocking Transformations, final expressions

We can then wrap up

$$
n_{j+1} \overline{X}_{j+1}  = \sum_{i=1}^{n_{j+1}} (\hat{X}_{j+1})_i =  \frac{1}{2}\sum_{i=1}^{n_{j}/2} (\hat{X}_{j})_{2i-1} + (\hat{X}_{j})_{2i} \nonumber
$$

$$
\begin{equation}
= \frac{1}{2}\left[ (\hat{X}_j)_1 + (\hat{X}_j)_2 + \cdots + (\hat{X}_j)_{n_j} \right] = \underbrace{\frac{n_j}{2}}_{=n_{j+1}} \overline{X}_j = n_{j+1}\overline{X}_j.
\label{_auto7} \tag{10}
\end{equation}
$$

By repeated use of this equation we get $V(\overline{X}_i) = V(\overline{X}_0) = V(\overline{X})$ for all $0 \leq i \leq d-1$. This has the consequence that

$$
\begin{equation}
V(\overline{X}) = \frac{\sigma_k^2}{n_k} + e_k \qquad \text{for all} \qquad 0 \leq k \leq d-1. \label{eq:convergence} \tag{11}
\end{equation}
$$

Flyvbjerg and Petersen demonstrated that the sequence
$\{e_k\}_{k=0}^{d-1}$ is decreasing, and conjecture that the term
$e_k$ can be made as small as we would like by making $k$ (and hence
$d$) sufficiently large. The sequence is decreasing (Master of Science thesis by Marius Jonsson, UiO 2018).
It means we can apply blocking transformations until
$e_k$ is sufficiently small, and then estimate $V(\overline{X})$ by
$\widehat{\sigma}^2_k/n_k$.

For an elegant solution and proof of the blocking method, see the recent article of [Marius Jonsson (former MSc student of the Computational Physics group)](https://journals.aps.org/pre/abstract/10.1103/PhysRevE.98.043304).


## Actividades
---

## 5.  [A partir de la versión con mini-batches, modificar el bucle de entrenamiento para que, sin modificar la iteración sobre el DataLoader, implemente descenso de gradiente estándar (*vanilla*) (actualización utilizando el gradiente sobre la función de costo acumulada, tras recorrer todo el dataset).](#actividadcinco)

## 7. [Usar una red preentrenada y aplica transfer learning para el dominio de  **(clasificar hormigas y abejas)**](#actividadsiete)

Primero voy a importar una librerias y crear las variables necesarias

---

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.datasets import load_iris
from sklearn import neighbors
import seaborn as sns
from sklearn.metrics import accuracy_score
#para que matplotlib tengo los gráficos de seaborn
from sklearn.model_selection import GridSearchCV
sns.set
from sklearn.metrics import plot_confusion_matrix
from sklearn.metrics import confusion_matrix
from sklearn import preprocessing
from sklearn.preprocessing import StandardScaler
import scikitplot as skplt
import xgboost as xgb
from scipy.stats import uniform, randint
from sklearn.metrics import auc, accuracy_score, confusion_matrix, mean_squared_error
from sklearn.model_selection import cross_val_score, GridSearchCV, KFold, RandomizedSearchCV, train_test_split
import pandas as pd
import seaborn as sn
import xgboost as xgb
import numpy as np
import torch
from torch import nn
import collections
from torchvision.datasets.utils import download_and_extract_archive
import os
import matplotlib
%matplotlib inline
import sympy as sp
from funciones_utiles import polynomial_features
from torch.optim import SGD
import matplotlib.pyplot as plt
from torch.utils import data
import warnings
from funciones_utiles import create_sinusoidal_data as create_data
warnings.simplefilter('ignore')

np.random.seed(12345)

```

## 5. A partir de la versión con mini-batches, modificar el bucle de entrenamiento para que, sin modificar la iteración sobre el DataLoader, implemente descenso de gradiente estándar (*vanilla*) (actualización utilizando el gradiente sobre la función de costo acumulada, tras recorrer todo el dataset).
---

A continuación defino el modelo básico, donde voy a comentar en el código las lineas que modifico.

```python
x_train, y_train, x_val, y_val = create_data()
P = 3

x_train_vec = polynomial_features(x_train, P)
y_train_vec = y_train.reshape(-1, 1) # aumentamos nuestro tensor de 1D a 2D pues así lo usará nuestro modelo

x_test_vec = polynomial_features(x_val, P)
y_test_vec = y_val.reshape(-1, 1)

print(x_train_vec.shape, x_test_vec.shape)
```

    (10, 3) (50, 3)

```python
class FeedforwardNet(nn.Module):
    def __init__(self, num_features, num_hidden, output):
        super(FeedforwardNet, self).__init__()
        # completa con tu código aquí
        self.fc1 = torch.nn.Linear(num_features, num_hidden, bias=True)
        self.ReLU = torch.nn.ReLU()
        self.fc2 = torch.nn.Linear(num_hidden, output, bias=True)

    def forward(self, x):
        # completa con tu código aquí
        out = self.fc1(x)
        out = self.ReLU(out)
        out = self.fc2(out)
        return out
```

```python
model = FeedforwardNet(num_features=P, num_hidden = 2, output=1)

# si tenemos GPU, llevamos el modelo ahí
if torch.cuda.is_available():
    device = torch.device('cuda:0')
else:
    device = torch.device('cpu')

model.to(device)

model.train()  # modelo nuevamente activado para entrenamiento

LEARNING_RATE = 0.01

WEIGHT_DECAY = 0.01  # L2 regularization coef.

criterion = nn.MSELoss(reduction='sum')

optimizer = SGD(model.parameters(), lr=LEARNING_RATE, weight_decay=WEIGHT_DECAY)

from torch.utils import data

BATCH_SIZE = 5
```

### ** En este siguiente bloque se puede visualizar la modificación **

ver comentarios en el código

```python
# ENTRENAMIENTO
EPOCHS = 10000
x_train_tensor = torch.from_numpy(np.float32(x_train_vec)).to(device)
y_train_tensor = torch.from_numpy(np.float32(y_train_vec)).to(device)
n_train = len(x_train_tensor)

x_test_tensor = torch.from_numpy(np.float32(x_test_vec)).to(device)
y_test_tensor = torch.from_numpy(np.float32(y_test_vec)).to(device)

train_data = data.TensorDataset(x_train_tensor, y_train_tensor)
train_loader = data.DataLoader(dataset=train_data, batch_size=BATCH_SIZE, shuffle=True, num_workers=0,)

val_data = data.TensorDataset(x_test_tensor, y_test_tensor)
val_loader = data.DataLoader(dataset=val_data, batch_size=BATCH_SIZE, shuffle=False, num_workers=0,)

print(len(train_data), len(train_loader))

train_loss, val_loss = [], []

for epoch in range(EPOCHS):

    train_loss_acc = 0.
    #####################
    # Aquí es donde hacemos los gradientes en cero, en lugar de adentro del bucle del "train_loader"
    # de esta forma, cuando ingrese al bucle, se actualizen los gradientes una vez que recorre
    # todo el bucle del train_loader, osea todas las muestras, como si fuera "vanilla"
    optimizer.zero_grad()
    #####################
    for x, y in train_loader:
        x.to(device, non_blocking=True)
        y.to(device, non_blocking=True)

        y_pred = model(x)
        loss = criterion(y_pred, y) / BATCH_SIZE
        loss.backward()

        train_loss_acc += loss.detach().item()

    train_loss.append(train_loss_acc / len(train_data))

    #####################
    # Aquí actualizamos los parámetros del modelo, una vez que recorrio todo el bucle del train loader
    # de esta forma conseguimos implementar el descenso de gradiente estándar (vanilla)
    optimizer.step()  # línea de código modificada, la baje con respecto al código original
    #####################

    with torch.no_grad():
        val_loss_acc = 0.
        for x, y in val_loader:
            x.to(device, non_blocking=True)
            y.to(device, non_blocking=True)

            y_pred = model(x)
            loss = criterion(y_pred, y)

            val_loss_acc += loss.item()
        val_loss.append(val_loss_acc / len(val_data))

    if epoch % (EPOCHS//10) == 0:
        print("epoch = {:02d}, train_loss = {:.3e}, val_loss = {:.3e}".format(epoch, train_loss_acc, val_loss_acc))
```

    10 2
    epoch = 00, train_loss = 1.414e+00, val_loss = 2.920e+01
    epoch = 1000, train_loss = 1.190e+00, val_loss = 2.451e+01
    epoch = 2000, train_loss = 1.190e+00, val_loss = 2.451e+01
    epoch = 3000, train_loss = 1.190e+00, val_loss = 2.451e+01
    epoch = 4000, train_loss = 1.190e+00, val_loss = 2.451e+01
    epoch = 5000, train_loss = 1.190e+00, val_loss = 2.451e+01
    epoch = 6000, train_loss = 1.190e+00, val_loss = 2.451e+01
    epoch = 7000, train_loss = 1.190e+00, val_loss = 2.451e+01
    epoch = 8000, train_loss = 1.190e+00, val_loss = 2.451e+01
    epoch = 9000, train_loss = 1.190e+00, val_loss = 2.451e+01

```python
plt.figure()
plt.plot(np.arange(EPOCHS), train_loss)
plt.plot(np.arange(EPOCHS), val_loss)
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'val'])
plt.grid()

model.eval()
y_pred = model(x_test_tensor)
y_pred = y_pred.cpu().detach().numpy().squeeze()
plt.figure()
plt.plot(x_val, y_val)
plt.plot(x_val, y_pred, color='orange')
plt.scatter(x_train, y_train, color='red')
plt.ylabel('$\sin(2 \pi x)$')
plt.xlabel('$x$')
plt.grid()
```

utlizar el dataset de abejas y hormigas, que se puede descargar de pytorch, para extrar los features, y despues usar algunos de los algortimos de SKLearn para clasificarlo.
---

#### Codigo base

El ejercició comienza a partir del Título KNN

```python

if not os.path.exists("./data/hymenoptera_data"):

    url = "https://download.pytorch.org/tutorial/hymenoptera_data.zip"
    root = './data/'

    download_and_extract_archive(url, root)
    print("Archivos descargados y extraidos con éxito")

else:
    print("La carpeta ya existe")
```

    Downloading https://download.pytorch.org/tutorial/hymenoptera_data.zip to ./data/hymenoptera_data.zip

    47287296it [00:10, 4575502.29it/s]

    Extracting ./data/hymenoptera_data.zip to ./data/
    Archivos descargados y extraidos con éxito

```python
import numpy as np
import matplotlib.pyplot as plt
import os

from torchvision.datasets import ImageFolder
from torchvision.utils import make_grid
from torchvision import transforms
from torch.utils.data import DataLoader
from funciones_utiles import imshow_tensor

path = './data/hymenoptera_data/'

batch_size = 4
num_workers = 2

# transformaciones que se aplicarán sobre cada imagen:
#   1. el lado más chico de la imagen a 256 pixels manteniendo relación de aspecto (PIL -> PIL)
#   2. crop de la región central de 224x224 pixels (PIL -> PIL)
#   1. transformar la imagen PIL a un tensor de pytorch (PIL -> Tensor)
#   2. aplicar la normalización (estandarización) con la que fueron entrenados los modelos en torchvision  (Tensor -> Tensor)
transform = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
])
```

```python
# dataset
data = ImageFolder(os.path.join(path, 'train'), transform)

# data loader
loader = DataLoader(data, batch_size=batch_size, shuffle=True, num_workers=num_workers)

image_batch, labels = next(iter(loader))
print(image_batch.size())
print(labels)
print([data.classes[lbl] for lbl in labels])
grid = make_grid(image_batch)
imshow_tensor(grid)
```

```python
import torch
from torch import nn
from torchvision import models
from tqdm import tqdm_notebook as progressbar

class FeatureExtractor(nn.Module):
    def __init__(self):
        super().__init__()
        # leer red pre-entrenada
        model = models.resnet18(pretrained=True)

        # congelamos los parámetros
        for p in model.parameters():
            p.requires_grad_(False)

        # armamos modelo secuencial que incluya todas las capas salvo la última (fc)
        layers = list(model.children())[:-1] # quitamos la última capa
        self.net = nn.Sequential(*layers)

    def forward(self, x):
        N = x.size(0)  # batch size
        x = self.net(x)
        return x.view(N, -1) # redimensionamos la salida para adaptarla a nuestro modelo
```

```python
def features_and_labels(set_):
    if set_ not in ('train', 'val'):
        raise ValueError('not a valid data set')

    transform = transforms.Compose([
        transforms.Resize(256),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ])

    data = ImageFolder(os.path.join(path, set_), transform)

    loader = DataLoader(data, batch_size=1, shuffle=False, num_workers=0)

    extractor = FeatureExtractor()
    extractor.eval()

    features, labels = [], []

    for img, lbl in progressbar(loader):
        feat = extractor(img)
        features.append(feat)
        labels.append(lbl)

    features = torch.cat(features, dim=0)
    labels = torch.cat(labels, dim=0)

    return features, labels

```

```python
train_features, train_labels = features_and_labels('train')
val_features, val_labels = features_and_labels('val')
print(train_features.size(), val_features.size())
```

      0%|          | 0/244 [00:00<?, ?it/s]

      0%|          | 0/153 [00:00<?, ?it/s]

    torch.Size([244, 512]) torch.Size([153, 512])

```python
x_train, y_train = train_features.cpu().numpy(), train_labels.cpu().numpy().reshape(-1, 1)
x_test, y_test = val_features.cpu().numpy(), val_labels.cpu().numpy().reshape(-1, 1)

# # para evitar recomputar los features cada vez, se pueden almacenar a disco
# np.save("resnet18.npy", {'x_train': x_train, 'y_train': y_train, 'x_test': x_test, 'y_test': y_test}, allow_pickle=True)
# mydata = np.load("resnet18.npy", allow_pickle=True)
# x_train = mydata.item().get('x_train')
# y_train = mydata.item().get('y_train')
# x_test = mydata.item().get('x_test')
# y_test = mydata.item().get('y_test')

print(x_train.shape, y_train.shape)
print(x_test.shape, y_test.shape)

# TODO:
# agrega tu código aquí para realizar una clasificación con modelos de sklearn
```

    (244, 512) (244, 1)
    (153, 512) (153, 1)

## KNN
---

El primer modelo que voy a aplicar va a ser KNN, ya que su funcionamiento es sencillo

#### Escalamos los datos

```python
sc_X = StandardScaler()
x_train = sc_X.fit_transform(x_train)
X_test = sc_X.transform(x_test)

```

#### Entrenamos el modelo

```python

clf_sk = neighbors.KNeighborsClassifier(n_neighbors=5)
clf_sk.fit(x_train,y_train)
y_hat = clf_sk.predict(x_test)

```

#### Métricas

```python
accuracy_escalado=accuracy_score(y_true=y_test, y_pred=y_hat)
print(f'Tasa de aciertos: {accuracy_escalado}')
```

    Tasa de aciertos: 0.8300653594771242

```python
from sklearn import metrics
print(metrics.classification_report(y_test, y_hat, target_names=["Abejas","Hormigas"]))
```

                  precision    recall  f1-score   support

          Abejas       0.96      0.66      0.78        70
        Hormigas       0.77      0.98      0.86        83

        accuracy                           0.83       153
       macro avg       0.86      0.82      0.82       153
    weighted avg       0.86      0.83      0.82       153

```python
plot_confusion_matrix(clf_sk, x_test, y_test,
                      display_labels=["Abejas","Hormigas"],
                      cmap=plt.cm.Blues,
                      normalize='true')
plt.show()
```

```python
y_hat_probas = clf_sk.predict_proba(x_test)
skplt.metrics.plot_roc(y_test, y_hat_probas, plot_micro=False, plot_macro=False)
plt.show()
```

## XGBoost
---

Qué es XGBoost ?

- En el aprendizaje automático, los modelos conjuntos obtienen mejores resultados que los modelos individuales con una alta probabilidad. Un modelo de conjunto combina diferentes modelos de aprendizaje automático en uno solo. El bosque aleatorio es un conjunto muy popular que toma la media de muchos árboles de decisión mediante el ensamblaje. Bagging es la abreviatura de "bootstrap aggregation", lo que significa que las muestras se eligen con reemplazo (bootstrapping), y se combinan (agregan) tomando su media.
El "boosting" es una buena alternativa al "bagging". En lugar de agregar predicciones, los reforzadores convierten a los aprendices débiles en aprendices fuertes centrándose en los puntos en los que los modelos individuales (normalmente árboles de decisión) se equivocan. En el Gradient Boosting, los modelos individuales se entrenan sobre los residuos, la diferencia entre la predicción y los resultados reales. En lugar de agregar árboles, los árboles con refuerzo de gradiente aprenden de los errores durante cada ronda de refuerzo.
XGBoost es la abreviatura de "eXtreme Gradient Boosting". El término "eXtreme" se refiere a las mejoras de velocidad, como la computación en paralelo y el conocimiento de la caché, que hacen que XGBoost sea aproximadamente 10 veces más rápido que el Gradient Boosting tradicional. Además, XGBoost incluye un algoritmo único de búsqueda de divisiones para optimizar los árboles, junto con una regularización incorporada que reduce el sobreajuste. En general, XGBoost es una versión más rápida y precisa del Gradient Boosting

XGBoost no requiere scalar los datos, ya que está basado en sus origenes en un árbol de decisión

Decidí implementar este algoritmo, para combinar el poder de las redes neuronales con uno de los algortimos más poderos que es XGBoost, y ver que resultados obtengo

[Fuente](https://towardsdatascience.com/getting-started-with-xgboost-in-scikit-learn-f69f5f470a97)

```python
x_train, y_train = train_features.cpu().numpy(), train_labels.cpu().numpy().reshape(-1, 1)
x_test, y_test = val_features.cpu().numpy(), val_labels.cpu().numpy().reshape(-1, 1)
```

```python
xgb_model = xgb.XGBClassifier(objective="binary:logistic",use_label_encoder=False )
xgb_model.fit(x_train, y_train)

y_pred = xgb_model.predict(x_test)

accuracy=accuracy_score(y_true=y_test, y_pred=y_pred)
print(f'Tasa de aciertos: {accuracy}')
```

    [20:39:11] WARNING: ..\src\learner.cc:1095: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.
    Tasa de aciertos: 0.9411764705882353

```python
plot_confusion_matrix(xgb_model, x_test, y_test,
                      display_labels=["Abejas","Hormigas"],
                      cmap=plt.cm.Blues,
                      normalize='true')
plt.show()
```

#### Búsqueda en grilla
---

Decidí aplicar una búsqueda en grilla para ver si podemos aumentar aún más la calidad de nuestro modelo

```python
x_train, y_train = train_features.cpu().numpy(), train_labels.cpu().numpy().reshape(-1, 1)
x_test, y_test = val_features.cpu().numpy(), val_labels.cpu().numpy().reshape(-1, 1)
```

```python
xgb_model = xgb.XGBClassifier( objective="binary:logistic", eval_metric='mlogloss')

params = {
    "colsample_bytree": uniform(0.7, 0.3),
    "gamma": uniform(0, 0.5),
    "learning_rate": uniform(0.03, 0.3), # default 0.1
    "max_depth": randint(2, 6), # default 3
    "n_estimators": randint(100, 150), # default 100
    "subsample": uniform(0.6, 0.4),
    'verbosity': [0],
    'n_jobs': [-1],
    'eval_metric':['mlogloss'],
    'use_label_encoder':[False]
}

search = RandomizedSearchCV(xgb_model, param_distributions=params,  n_iter=35, cv=3, verbose=1, n_jobs=1, return_train_score=True)

search.fit(x_train, y_train)

```

    Fitting 3 folds for each of 35 candidates, totalling 105 fits

    RandomizedSearchCV(cv=3,
                       estimator=XGBClassifier(base_score=None, booster=None,
                                               colsample_bylevel=None,
                                               colsample_bynode=None,
                                               colsample_bytree=None,
                                               eval_metric='mlogloss', gamma=None,
                                               gpu_id=None, importance_type='gain',
                                               interaction_constraints=None,
                                               learning_rate=None,
                                               max_delta_step=None, max_depth=None,
                                               min_child_weight=None, missing=nan,
                                               monotone_constraints...
                                            'learning_rate': <scipy.stats._distn_infrastructure.rv_frozen object at 0x000002A215EB9508>,
                                            'max_depth': <scipy.stats._distn_infrastructure.rv_frozen object at 0x000002A215EB98C8>,
                                            'n_estimators': <scipy.stats._distn_infrastructure.rv_frozen object at 0x000002A21DF92708>,
                                            'n_jobs': [-1],
                                            'subsample': <scipy.stats._distn_infrastructure.rv_frozen object at 0x000002A22409F088>,
                                            'use_label_encoder': [False],
                                            'verbosity': [0]},
                       return_train_score=True, verbose=1)

```python
search.best_params_
```

    {'colsample_bytree': 0.8233534396517617,
     'eval_metric': 'mlogloss',
     'gamma': 0.4413172225884209,
     'learning_rate': 0.2378124044384247,
     'max_depth': 2,
     'n_estimators': 105,
     'n_jobs': -1,
     'subsample': 0.6257760924741134,
     'use_label_encoder': False,
     'verbosity': 0}

Débido a que nuestra búsqueda es aleatoria, los resultados que se obtienen varian con cada ejecución, pero, al ejecutar varias pruebas, podremos ver que la tasa de precisión ronda entre los 92% a 95%, lo cual es un muy buen indicador de la calidad del XGBoost

```python
xgb_model = xgb.XGBClassifier(colsample_bytree=0.8233534396517617, eval_metric='mlogloss', gamma= 0.44131722258842093, learning_rate= 0.2378124044384247, max_depth= 2, n_estimators= 105, n_jobs= -1, subsample= 0.6257760924741134,
use_label_encoder= False, verbosity= 0)
xgb_model.fit(x_train, y_train)

y_pred = xgb_model.predict(x_test)

accuracy=accuracy_score(y_true=y_test, y_pred=y_pred)
print(f'Tasa de aciertos: {accuracy}')
```

    Tasa de aciertos: 0.9411764705882353

```python

columas=["Abejas","Hormigas"]
cm=confusion_matrix(y_test, y_pred)
df_cm = pd.DataFrame(cm, index = columas,
                  columns = columas)
plt.figure(figsize = (5,5))
sn.heatmap(df_cm, annot=True)
```

```python
print(metrics.classification_report(y_test, y_pred, target_names=["Abejas","Hormigas"]))
```

                  precision    recall  f1-score   support

          Abejas       0.93      0.94      0.94        70
        Hormigas       0.95      0.94      0.95        83

        accuracy                           0.94       153
       macro avg       0.94      0.94      0.94       153
    weighted avg       0.94      0.94      0.94       153

### Conclusión:
Podemos decir, que usar las redes neuronales para extraer los features de un dataset, para despues aplicarlos los "algoritmos clásicos" de machine learning, es una buena combinación, ya que conmuy poco esfuerzo, conseguimos excelentes resultados.

Si comparamos el KNN, que es un modelo sencillo, tuvimos una precisión del 80% (por tomar el accuracy como punto de comparación), mientras que con XGBoost obtuvimos un poco más del 90%.

Tambien de esta forma, desde mi punto de vista, ganamos mayor explicabilidad para nuestro modelo, sino, muchas veces las redes neuronales son vistas como cajas negras, donde entra algo, la computadora se encarga de realizar la extraccion de carácteristicas y nos devuelve un numero, y esto se vuelve más confuso mientras más compleja es la arquitectura de la red neuronal, cosa que no sucede con los algoritmos clásicos (obviamente al aumentar la dimensión es más dificil de explicar).


ScorecardODSC_7Aug
Created on Mon Aug  5 09:43:55 2019
@author: KAVITA DWIVEDI
"""

Edited: Paul Conway, Sep 2019

This shows the full machine learning pipeline for a realistic problem: financial credit scoring.

*See the acompanying slides for further explanation*

## Sec 1. Importing libraries

Here we import the necessary python packages

```
#Importing the Libraries

# Pandas for data manipulation
import pandas as pd
from pandas.core import algorithms

# Numpy for vectors, matrices and linear algebra
import numpy as np

# matplotlib and Seaborn for graphing
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns

# scikit utilities for preprocessing, model selection and measuring performance
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import train_test_split
from sklearn.metrics import (classification_report,
                             confusion_matrix,
                             accuracy_score,
                             roc_curve,
                             auc)

# ML models
from sklearn.svm import SVC
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier

# We'll use variane inflation factor to do some feature selection
from statsmodels.stats.outliers_influence import variance_inflation_factor

# Some special dictionary types
from collections import defaultdict, OrderedDict

from pathlib import Path  # for chekcing if we cloned the repository already

from scipy.stats import spearmanr
import re
import traceback
import string

from IPython.core.display import HTML

from pprint import pprint
```

```
# set random_state, to ensure results look the same every time
rs = 2
```

## Sec 2. Cloning The Github

```
if not Path("ScorecardODSC").is_dir():
  !git clone https://github.com/kavitablockyto/ScorecardODSC.git
```

##Section 3. Importing the Dataset and Sampling

```
# defining path of Collection_Home3.xlsx
path=("/content/ScorecardODSC/Collection_Home3.xlsx")

#Reading the Dataset

df = pd.read_excel(path)
```

```
data_sample = df.sample(frac=0.2, random_state=rs)
data_sample.head()
```

      CODE_GENDER
      NAME_INCOME_TYPE
      NAME_EDUCATION_TYPE
      NAME_FAMILY_STATUS
      DAYS_BIRTH
      DAYS_EMPLOYED
      DAYS_REGISTRATION
      DAYS_ID_PUBLISH
      FLAG_EMP_PHONE
      FLAG_WORK_PHONE
      FLAG_PHONE
      OCCUPATION_TYPE
      REG_CITY_NOT_LIVE_CITY
      REG_CITY_NOT_WORK_CITY
      ORGANIZATION_TYPE
      DAYS_LAST_PHONE_CHANGE
      TARGET

      177958
      F
      Working
      Higher education
      Married
      10512
      2147
      4746.0
      438
      1
      1
      0
      Managers
      0
      1
      Business Entity Type 3
      105
      0

      128842
      F
      Pensioner
      Secondary / secondary special
      Widow
      21497
      365243
      8186.0
      4042
      0
      0
      0
      NaN
      0
      0
      XNA
      2300
      0

      146276
      F
      Commercial associate
      Higher education
      Single / not married
      20031
      6874
      4092.0
      3589
      1
      0
      1
      Managers
      0
      0
      Other
      545
      0

      259956
      M
      Commercial associate
      Secondary / secondary special
      Married
      11835
      5047
      28.0
      4235
      1
      0
      0
      Drivers
      0
      0
      Business Entity Type 3
      768
      0

      42348
      F
      Pensioner
      Secondary / secondary special
      Married
      23096
      365243
      9705.0
      4078
      0
      0
      1
      NaN
      0
      0
      XNA
      582
      0

```
# Counting Target Variable numbers
df.TARGET.value_counts()
```

    0    255011
    1     23221
    Name: TARGET, dtype: int64

```

data_sample.TARGET.value_counts()
```

    0    50850
    1     4796
    Name: TARGET, dtype: int64

```
#Descriptive Stats
#Target Variable Count in data and sample

df.TARGET.value_counts()/len(df)
data_sample.TARGET.value_counts()/len(data_sample)
```

    0    0.913812
    1    0.086188
    Name: TARGET, dtype: float64

```
# Data Description
data_sample.describe()
```

      DAYS_BIRTH
      DAYS_EMPLOYED
      DAYS_REGISTRATION
      DAYS_ID_PUBLISH
      FLAG_EMP_PHONE
      FLAG_WORK_PHONE
      FLAG_PHONE
      REG_CITY_NOT_LIVE_CITY
      REG_CITY_NOT_WORK_CITY
      DAYS_LAST_PHONE_CHANGE
      TARGET

      count
      55646.000000
      55646.000000
      55646.000000
      55646.000000
      55646.000000
      55646.000000
      55646.000000
      55646.000000
      55646.000000
      55646.000000
      55646.000000

      mean
      16140.211893
      69265.894781
      5008.039482
      3011.903803
      0.815710
      0.203033
      0.282356
      0.078119
      0.230637
      972.483934
      0.086188

      std
      4336.041155
      140683.409755
      3518.077016
      1497.841880
      0.387724
      0.402261
      0.450150
      0.268361
      0.421244
      830.864961
      0.280644

      min
      7489.000000
      2.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000

      25%
      12533.000000
      956.000000
      2065.000000
      1770.250000
      1.000000
      0.000000
      0.000000
      0.000000
      0.000000
      274.000000
      0.000000

      50%
      15869.000000
      2266.000000
      4520.500000
      3271.000000
      1.000000
      0.000000
      0.000000
      0.000000
      0.000000
      779.000000
      0.000000

      75%
      19726.000000
      6001.750000
      7507.750000
      4308.000000
      1.000000
      0.000000
      1.000000
      0.000000
      0.000000
      1581.000000
      0.000000

      max
      25197.000000
      365243.000000
      24672.000000
      6258.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      4121.000000
      1.000000

##Sec 4. Data Preparation

```
#Correlation Plot
corr = data_sample.corr()

# Plot a heatmap fo the correlations using Seaborn
sns.heatmap(corr,
        xticklabels=corr.columns,
        yticklabels=corr.columns)
```

```
# Missing Data

# Function to calculate missing values by column# Funct
def missing_values_table(data_sample):
        # Total missing values
        mis_val = data_sample.isnull().sum()

        # Percentage of missing values
        mis_val_percent = 100 * data_sample.isnull().sum() / len(data_sample)

        # Make a table with the results
        mis_val_table = pd.concat([mis_val , mis_val_percent], axis=1)

        # Rename the columns
        mis_val_table_ren_columns = mis_val_table.rename(
        columns = {0 : 'Missing Values', 1 : '% of Total Values'})

        # Sort the table by percentage of missing descending
        mis_val_table_ren_columns = mis_val_table_ren_columns[
            mis_val_table_ren_columns.iloc[:, 1] != 0].sort_values(
        '% of Total Values', ascending=False).round(1)

        # Print some summary information
        print ("Your selected dataframe has " + str(data_sample.shape[1]) + " columns.\n"
            "There are " + str(mis_val_table_ren_columns.shape[0]) +
              " columns that have missing values.")

        # Return the dataframe with missing information
        return mis_val_table_ren_columns

# Missing values statistics
missing_values = missing_values_table(data_sample)
missing_values.head(20)
```

    Your selected dataframe has 17 columns.
    There are 1 columns that have missing values.

      Missing Values
      % of Total Values

      OCCUPATION_TYPE
      17541
      31.5

```
def transform_data(data):
  # Data transformation
  # Convert categorical values to numeric using label encoder
  # using defaultdict in this way allows us to fit_transform or inverse_transform
  # a categorical variable
  d = defaultdict(LabelEncoder)

  # Encoding the categorical variable
  fit = data.select_dtypes(include=['object']).fillna('NA').apply(lambda x: d[x.name].fit_transform(x))

  # Convert the categorical columns based on encoding
  for i in list(d.keys()):
      data[i] = d[i].transform(data[i].fillna('NA'))

      features = data[data.columns.difference(['TARGET'])]
  return features.fillna(0), data['TARGET']

features, labels = transform_data(data_sample)
```

Weight of evidence (WOE) and Information value (IV) are widely used in credit scoring to measure the separation of good vs bad customers.
$$
\begin{align}
WOE & = \ln \left( \frac{positive\%}{negative\%} \right)\\
IV & = \sum (positive\% - negative\%) \times WOE
\end{align}
$$

The IV value can be used to select variables quickly:

| Information Value | Predictive Power |
|---|---|
| < 0.02 | Useless for prediction |
| 0.02 - 0.1 | Weak predictor |
| 0.1 - 0.3 | Medium predictor |
| 0.3 - 0.5 | Strong predictor |
| > 0.5 | Possibly too good to be true |

From:
[https://medium.com/@sundarstyles89/weight-of-evidence-and-information-value-using-python-6f05072e83eb](https://medium.com/@sundarstyles89/weight-of-evidence-and-information-value-using-python-6f05072e83eb)

```
# Weight of evidence (WOE) and Information value (IV)
# Ref: https://github.com/Sundar0989/WOE-and-IV

# Maximum number of bins (categories) for numeric variable binning.
max_bin = 20
# Ensure binning produces at least 2 categories.
force_bin = 3

# define a monotonic numerical binning function for numeric values
def mono_bin(Y, X, n = max_bin):

    df1 = pd.DataFrame({"X": X, "Y": Y})
    justmiss = df1[['X', 'Y']][df1.X.isnull()]
    notmiss = df1[['X', 'Y']][df1.X.notnull()]
    r = 0
    while np.abs(r) < 1:
        try:
            d1 = pd.DataFrame({"X": notmiss.X, "Y": notmiss.Y,
                               "Bucket": pd.qcut(notmiss.X, n)})
            d2 = d1.groupby('Bucket', as_index=True)
            r, p = spearmanr(d2.mean().X, d2.mean().Y)
            n = n - 1
        except Exception as e:
            n = n - 1

    if len(d2) == 1:
        n = force_bin
        bins = algorithms.quantile(notmiss.X, np.linspace(0, 1, n))
        if len(np.unique(bins)) == 2:
            bins = np.insert(bins, 0, 1)
            bins[1] = bins[1]-(bins[1]/2)
        d1 = pd.DataFrame({"X": notmiss.X, "Y": notmiss.Y,
                           "Bucket": pd.cut(notmiss.X, np.unique(bins),
                                            include_lowest=True)})
        d2 = d1.groupby('Bucket', as_index=True)

    d3 = pd.DataFrame({}, index=[])
    d3["MIN_VALUE"] = d2.min().X
    d3["MAX_VALUE"] = d2.max().X
    d3["COUNT"] = d2.count().Y
    d3["EVENT"] = d2.sum().Y
    d3["NONEVENT"] = d2.count().Y - d2.sum().Y
    d3=d3.reset_index(drop=True)

    if len(justmiss.index) > 0:
        d4 = pd.DataFrame({'MIN_VALUE':np.nan}, index=[0])
        d4["MAX_VALUE"] = np.nan
        d4["COUNT"] = justmiss.count().Y
        d4["EVENT"] = justmiss.sum().Y
        d4["NONEVENT"] = justmiss.count().Y - justmiss.sum().Y
        d3 = d3.append(d4, ignore_index=True)

    d3["EVENT_RATE"] = d3.EVENT/d3.COUNT
    d3["NON_EVENT_RATE"] = d3.NONEVENT/d3.COUNT
    d3["DIST_EVENT"] = d3.EVENT/d3.sum().EVENT
    d3["DIST_NON_EVENT"] = d3.NONEVENT/d3.sum().NONEVENT
    d3["WOE"] = np.log(d3.DIST_EVENT/d3.DIST_NON_EVENT)
    d3["IV"] = (d3.DIST_EVENT-d3.DIST_NON_EVENT) * np.log(d3.DIST_EVENT/d3.DIST_NON_EVENT)
    d3["VAR_NAME"] = "VAR"
    d3 = d3[['VAR_NAME', 'MIN_VALUE', 'MAX_VALUE', 'COUNT', 'EVENT',
             'EVENT_RATE', 'NONEVENT', 'NON_EVENT_RATE', 'DIST_EVENT',
             'DIST_NON_EVENT', 'WOE', 'IV']]
    d3 = d3.replace([np.inf, -np.inf], 0)
    d3.IV = d3.IV.sum()

    return(d3)

# De# define a monotocharacterical binning funclues
def char_bin(Y, X):

    df1 = pd.DataFrame({"X": X, "Y": Y})
    justmiss = df1[['X', 'Y']][df1.X.isnull()]
    notmiss = df1[['X', 'Y']][df1.X.notnull()]
    df2 = notmiss.groupby('X', as_index=True)

    d3 = pd.DataFrame({}, index=[])
    d3["COUNT"] = df2.count().Y
    d3["MIN_VALUE"] = df2.sum().Y.index
    d3["MAX_VALUE"] = d3["MIN_VALUE"]
    d3["EVENT"] = df2.sum().Y
    d3["NONEVENT"] = df2.count().Y - df2.sum().Y

    if len(justmiss.index) > 0:
        d4 = pd.DataFrame({'MIN_VALUE':np.nan}, index=[0])
        d4["MAX_VALUE"] = np.nan
        d4["COUNT"] = justmiss.count().Y
        d4["EVENT"] = justmiss.sum().Y
        d4["NONEVENT"] = justmiss.count().Y - justmiss.sum().Y
        d3 = d3.append(d4, ignore_index=True)

    d3["EVENT_RATE"] = d3.EVENT/d3.COUNT
    d3["NON_EVENT_RATE"] = d3.NONEVENT/d3.COUNT
    d3["DIST_EVENT"] = d3.EVENT/d3.sum().EVENT
    d3["DIST_NON_EVENT"] = d3.NONEVENT/d3.sum().NONEVENT
    d3["WOE"] = np.log(d3.DIST_EVENT/d3.DIST_NON_EVENT)
    d3["IV"] = (d3.DIST_EVENT-d3.DIST_NON_EVENT)*np.log(d3.DIST_EVENT/d3.DIST_NON_EVENT)
    d3["VAR_NAME"] = "VAR"
    d3 = d3[['VAR_NAME', 'MIN_VALUE', 'MAX_VALUE', 'COUNT', 'EVENT', 'EVENT_RATE', 'NONEVENT', 'NON_EVENT_RATE', 'DIST_EVENT', 'DIST_NON_EVENT', 'WOE', 'IV']]
    d3 = d3.replace([np.inf, -np.inf], 0)
    d3.IV = d3.IV.sum()
    d3 = d3.reset_index(drop=True)

    return(d3)

def data_vars(df1, target):
    stack = traceback.extract_stack()
    filename, lineno, function_name, code = stack[-2]
    vars_name = re.compile(r'\((.*?)\).*$').search(code).groups()[0]
    final = (re.findall(r"[\w']+", vars_name))[-1]

    x = df1.dtypes.index
    count = -1

    for i in x:
        if i.upper() not in (final.upper()):
            if np.issubdtype(df1[i], np.number) and len(pd.Series.unique(df1[i])) > 2:
                conv = mono_bin(target, df1[i])
                conv["VAR_NAME"] = i
                count = count + 1
            else:
                conv = char_bin(target, df1[i])
                conv["VAR_NAME"] = i
                count = count + 1

            if count == 0:
                iv_df = conv
            else:
                iv_df = iv_df.append(conv, ignore_index=True)

    iv = pd.DataFrame({'IV':iv_df.groupby('VAR_NAME').IV.max()})
    iv = iv.reset_index()
    return(iv_df,iv)

final_iv, IV = data_vars(data_sample[data_sample.columns.difference(['TARGET'])], data_sample.TARGET)
IV = IV.rename(columns={'VAR_NAME':'index'})
IV.sort_values(['IV'], ascending=0)

hi_ivs = list(IV.loc[IV['IV'] >= 0.02].sort_values(['IV'],ascending=0)['index'])
```

##Sec 5.  Modeling using Support Vector Machine

```
# Split our data into training and test sets with the reduced feature set
train_features, test_features, train_labels, test_labels = train_test_split(features[hi_ivs], labels, test_size = 0.4, random_state=rs)

#Support Vector Machine Classsifier
clf = SVC(gamma='scale', random_state=rs)
clf.fit(train_features, train_labels)

train_pred = clf.predict(train_features)
test_pred = clf.predict(test_features)

clf.score(test_features, test_labels)
```

    0.9142369378678288

That accuracy looks really impressive ... until you look at how many times each label value was predicted by the model.

```
print(f"True label counts:\n{test_labels.value_counts()}\n\nPredicted label counts:\n{pd.Series(test_pred).value_counts()}")
```

    True label counts:
    0    20350
    1     1909
    Name: TARGET, dtype: int64

    Predicted label counts:
    0    22259
    dtype: int64

All the values predicted by our model are zero. Prediction accuracy can't give us a good sense of how well the model has worked because almost all of the data we used for training was negative. We need to dig deeper into our results by looking at some metrics related to how well each of the labels are picked relative to their true labels.

A **confusion matrix** shows counts of true negatives, true positives, false negatives and false positives.

```
def print_confusion_matrix(act_labels, pred_labels):
  cm = confusion_matrix(act_labels, pred_labels)
  sns.heatmap(cm, annot=True, fmt='d', annot_kws={"size": 18}, cmap=sns.color_palette("Blues", 100), cbar=None)
  plt.ylabel('True', fontsize=24)
  plt.xlabel('Predicted', fontsize=24)
  plt.tick_params(axis="x", labelsize=18)
  plt.tick_params(axis="y", labelsize=18)
  plt.plot()
```

```
print_confusion_matrix(test_labels, test_pred)
```

The *classification report* in scikit learn gives us:

**Precision** talks about how precise/accurate your model is out of those predicted positive. It represents the proportion of *positive results* that are correctly labeled as positive.

$$Precision = \frac{True\ Positives}{True\ Positives + False\ Positives}$$

**Recall** measures the proportion of *actual positives* that are correctly identified as such (e.g., the percentage of sick people who are correctly identified as having the condition).

$$Recall = \frac{True\ Positives}{True\ Positives + False\ Negatives}$$

**F1 Score** is a measure of accuracy using precision and recall. It takes the occurence of both false positives and false negatives into account.

$$F_1\ Score = 2 \times \frac{Precision \times Recall}{Precision + Recall}$$

```
print(classification_report(test_labels, test_pred))
```

                  precision    recall  f1-score   support

               0       0.91      1.00      0.96     20350
               1       0.00      0.00      0.00      1909

        accuracy                           0.91     22259
       macro avg       0.46      0.50      0.48     22259
    weighted avg       0.84      0.91      0.87     22259

    /usr/local/lib/python3.6/dist-packages/sklearn/metrics/classification.py:1437: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.
      'precision', 'predicted', average, warn_for)

##Sec 6. Modeling using RandomForest

The ramdom forest classifier can identify which features are more important for accurate classification. This can be used for feature selection.

```
# Variable Importance
clf = RandomForestClassifier(random_state=rs)

clf.fit(features, labels)

preds = clf.predict(features)

VI = pd.DataFrame(clf.feature_importances_, columns = ["RF"], index=features.columns)

VI = VI.reset_index()

print(f" Feature importance from random forest:\n{VI.sort_values(['RF'], ascending=0)}")

hi_vis = list(VI.loc[VI['RF'] >= 0.02].sort_values(['RF'],ascending=0)['index'])
```

    /usr/local/lib/python3.6/dist-packages/sklearn/ensemble/forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.
      "10 in version 0.20 to 100 in 0.22.", FutureWarning)

     Feature importance from random forest:
                         index        RF
    1               DAYS_BIRTH  0.162431
    3          DAYS_ID_PUBLISH  0.155856
    5        DAYS_REGISTRATION  0.154515
    4   DAYS_LAST_PHONE_CHANGE  0.135749
    2            DAYS_EMPLOYED  0.132665
    13       ORGANIZATION_TYPE  0.068926
    12         OCCUPATION_TYPE  0.057146
    10      NAME_FAMILY_STATUS  0.034265
    7               FLAG_PHONE  0.016602
    9      NAME_EDUCATION_TYPE  0.015661
    0              CODE_GENDER  0.014713
    8          FLAG_WORK_PHONE  0.014694
    11        NAME_INCOME_TYPE  0.014288
    15  REG_CITY_NOT_WORK_CITY  0.012741
    14  REG_CITY_NOT_LIVE_CITY  0.008769
    6           FLAG_EMP_PHONE  0.000980

```
# Split our data into training and test sets
train_features, test_features, train_labels, test_labels = train_test_split(features[hi_vis], labels, test_size = 0.2, random_state=rs)

# Refit the model with the reduced feature set
clf.fit(train_features, train_labels)

train_pred = clf.predict(train_features)
test_pred = clf.predict(test_features)

accuracy_train = accuracy_score(train_pred, train_labels)
accuracy_test = accuracy_score(test_pred, test_labels)
```

Once again we'll see that accuracy does not necessarily reflect how well the model is performing.

We'll look at the **area under the curve (AUC)** for the **receiver operating characteristic (ROC)** curve to get a better idea of true model performance.

```
def plot_roc_curve(fpr, tpr):
    # plt.rcParams["figure.figsize"] = (10, 8)
    plt.figure(figsize=(10,8))
    plt.plot(fpr, tpr, color='orange', label='ROC')
    plt.plot([0, 1], [0, 1], color='darkblue', linestyle='--')
    plt.xlabel('False Positive Rate')
    plt.ylabel('True Positive Rate')
    plt.title('Receiver Operating Characteristic (ROC) Curve')
    plt.legend()
    plt.show()
```

```
fpr_train, tpr_train, _ = roc_curve(np.array(train_labels), clf.predict_proba(train_features)[:,1])
auc_train = auc(fpr_train, tpr_train)
fpr_test, tpr_test, _ = roc_curve(np.array(test_labels), clf.predict_proba(test_features)[:,1])
auc_test = auc(fpr_test, tpr_test)
```

```
print(f"Training accuracy = {accuracy_train}\nTest accuracy = {accuracy_test}")
print(f"Training AUC = {auc_train}\nTest AUC = {auc_test}")
```

    Training accuracy = 0.9820289334171983
    Test accuracy = 0.9115902964959569
    Training AUC = 0.9996643803568557
    Test AUC = 0.5498794965471476

```
plot_roc_curve(fpr_test, tpr_test)
```

```
print_confusion_matrix(test_labels, test_pred)
print(classification_report(test_labels, test_pred))
```

##Sec 7. Modeling Using Gradient Boosting

**Multicollinearity** refers to predictors that are correlated with other predictors.  Multicollinearity occurs when your model includes multiple factors that are correlated not just to your response variable, but also to each other. In other words, it results when you have factors that are a bit redundant.

One way to measure multicollinearity is the **variance inflation factor (VIF)**, which assesses how much the variance of an estimated regression coefficient increases if your predictors are correlated.  If no factors are correlated, the VIFs will all be 1.

```
# Multicollinearity
def calculate_vif(features):
    vif = pd.DataFrame()
    vif["Features"] = features.columns
    vif["VIF"] = [variance_inflation_factor(features.values, i) for i in range(features.shape[1])]
    return(vif)

# Model
vif = calculate_vif(features)
print(vif.sort_values('VIF'))

print("\nFeatures with low multicollinearity:")
low_mc = list(vif.loc[vif['VIF'] <= 10].sort_values(['VIF'], ascending=0)['Features'])
pprint(low_mc)

# Split our data into training and test sets
train_features, test_features, train_labels, test_labels = train_test_split(features[low_mc], labels, test_size = 0.4, random_state=rs)
train_labels = train_labels.reset_index(drop=True)
test_labels = test_labels.reset_index(drop=True)
```

                      Features        VIF
    14  REG_CITY_NOT_LIVE_CITY   1.377137
    8          FLAG_WORK_PHONE   1.495648
    7               FLAG_PHONE   1.565310
    0              CODE_GENDER   1.614113
    15  REG_CITY_NOT_WORK_CITY   1.761182
    4   DAYS_LAST_PHONE_CHANGE   2.443368
    10      NAME_FAMILY_STATUS   2.996922
    11        NAME_INCOME_TYPE   3.347807
    5        DAYS_REGISTRATION   3.439765
    13       ORGANIZATION_TYPE   5.251824
    3          DAYS_ID_PUBLISH   5.584025
    12         OCCUPATION_TYPE   6.307613
    9      NAME_EDUCATION_TYPE   7.724784
    2            DAYS_EMPLOYED  15.217120
    1               DAYS_BIRTH  28.341692
    6           FLAG_EMP_PHONE  32.906376

    Features with low multicollinearity:
    ['NAME_EDUCATION_TYPE',
     'OCCUPATION_TYPE',
     'DAYS_ID_PUBLISH',
     'ORGANIZATION_TYPE',
     'DAYS_REGISTRATION',
     'NAME_INCOME_TYPE',
     'NAME_FAMILY_STATUS',
     'DAYS_LAST_PHONE_CHANGE',
     'REG_CITY_NOT_WORK_CITY',
     'CODE_GENDER',
     'FLAG_PHONE',
     'FLAG_WORK_PHONE',
     'REG_CITY_NOT_LIVE_CITY']

```
#GradientBoosting
# clf = GradientBoostingClassifier(learning_rate=.03, random_state=rs)
clf = GradientBoostingClassifier(learning_rate=.09, random_state=rs)
# clf = GradientBoostingClassifier(random_state=rs)

clf.fit(train_features, train_labels)

train_pred = clf.predict(train_features)
test_pred = clf.predict(test_features)
```

```
accuracy_train = accuracy_score(train_pred, train_labels)
accuracy_test = accuracy_score(test_pred, test_labels)

fpr_train, tpr_train, _ = roc_curve(np.array(train_labels), clf.predict_proba(train_features)[:, 1])
auc_train = auc(fpr_train, tpr_train)

fpr_test, tpr_test, _ = roc_curve(np.array(test_labels), clf.predict_proba(test_features)[:, 1])
auc_test = auc(fpr_test, tpr_test)
```

```
print(f"Training accuracy = {accuracy_train}\nTest accuracy = {accuracy_test}")
print(f"Training AUC = {auc_train}\nTest AUC = {auc_test}")
```

    Training accuracy = 0.9136490250696379
    Test accuracy = 0.9141920122197763
    Training AUC = 0.6878262249655038
    Test AUC = 0.642715817870349

```
plot_roc_curve(fpr_test, tpr_test)
```

```
print_confusion_matrix(test_labels, test_pred)
print(classification_report(test_labels, test_pred))
```

## Sec 8. Validation and Performance Measurement of Model

```
# Deciles and Kolmogarov Smirnoff Statistics

def plot_pandas_style(styler):
    html = '\n'.join([line.lstrip() for line in styler.render().split('\n')])
    return HTML(html)

def highlight_max(s, color='yellow'):
    '''
    highlight the maximum in a Series yellow.
    '''
    is_max = s == s.max()
    return ['background-color: {}'.format(color) if v else '' for v in is_max]

def decile_labels(agg1, label, color='skyblue'):
    agg_dummy = pd.DataFrame(OrderedDict((('TOTAL', 0), ('TARGET', 0), ('NONTARGET', 0), ('PCT_TGT', 0), ('CUM_TGT', 0), ('CUM_NONTGT', 0), ('DIST_TGT', 0), ('DIST_NONTGT', 0), ('SPREAD', 0))), index=[0])
    agg1 = agg1.append(agg_dummy).sort_index()
    agg1.index.name = label
    agg1 = agg1.style.apply(highlight_max, color = 'yellow', subset=['SPREAD'])
    agg1.bar(subset=['TARGET'], color='{}'.format(color))
    agg1.bar(subset=['TOTAL'], color='{}'.format(color))
    agg1.bar(subset=['PCT_TGT'], color='{}'.format(color))
    return(agg1)

def deciling(data, decile_by, target, nontarget):
    inputs = list(decile_by)
    inputs.extend((target, nontarget))
    decile = data[inputs]
    grouped = decile.groupby(decile_by)
    agg1 = pd.DataFrame({}, index=[])
    agg1['TOTAL'] = grouped.sum()[nontarget] + grouped.sum()[target]
    agg1['TARGET'] = grouped.sum()[target]
    agg1['NONTARGET'] = grouped.sum()[nontarget]
    agg1['PCT_TGT'] = grouped.mean()[target] * 100
    agg1['CUM_TGT'] = grouped.sum()[target].cumsum()
    agg1['CUM_NONTGT'] = grouped.sum()[nontarget].cumsum()
    agg1['DIST_TGT'] = agg1['CUM_TGT']/agg1['TARGET'].sum() * 100
    agg1['DIST_NONTGT'] = agg1['CUM_NONTGT']/agg1['NONTARGET'].sum() * 100
    agg1['SPREAD'] = (agg1['DIST_TGT'] - agg1['DIST_NONTGT'])
    agg1 = decile_labels(agg1, 'DECILE', color='skyblue')
    return(plot_pandas_style(agg1))

def scoring (features, clf, target):
  score=pd.DataFrame(clf.predict_proba(features)[:, 1], columns = ['SCORE'])
  score['DECILE'] = pd.qcut(score['SCORE'].rank(method = 'first'), 10, labels=range(10, 0, -1))
  score['DECILE'] = score['DECILE'].astype(float)
  score['TARGET'] = target
  score['NONTARGET'] = 1 - target
  return score
```

```
scores_train = scoring(train_features, clf, train_labels)
scores_test = scoring(test_features, clf, test_labels)
```

```
deciling(scores_train, ['DECILE'], 'TARGET', 'NONTARGET')
```

        	        TOTAL	        TARGET	        NONTARGET	        PCT_TGT	        CUM_TGT	        CUM_NONTGT	        DIST_TGT	        DIST_NONTGT	        SPREAD

        DECILE

0.0
0
0
0
0
0
0
0
0
0

1.0
3339
713
2626
21.3537
713
2626
24.6969
8.60984
16.0871

2.0
3339
476
2863
14.2558
1189
5489
41.1846
17.9967
23.1879

3.0
3338
371
2967
11.1144
1560
8456
54.0353
27.7246
26.3107

4.0
3339
319
3020
9.55376
1879
11476
65.0849
37.6262
27.4586

5.0
3338
242
3096
7.24985
2121
14572
73.4673
47.777
25.6902

6.0
3339
221
3118
6.61875
2342
17690
81.1223
58
23.1223

7.0
3339
192
3147
5.75022
2534
20837
87.7728
68.318
19.4547

8.0
3338
155
3183
4.6435
2689
24020
93.1417
78.7541
14.3876

9.0
3339
124
3215
3.71369
2813
27235
97.4368
89.2951
8.1417

10.0
3339
74
3265
2.21623
2887
30500
100
100
0

```
deciling(scores_test, ['DECILE'], 'TARGET', 'NONTARGET')
```

        	        TOTAL	        TARGET	        NONTARGET	        PCT_TGT	        CUM_TGT	        CUM_NONTGT	        DIST_TGT	        DIST_NONTGT	        SPREAD

        DECILE

0.0
0
0
0
0
0
0
0
0
0

1.0
2226
356
1870
15.9928
356
1870
18.6485
9.18919
9.45932

2.0
2226
331
1895
14.8697
687
3765
35.9874
18.5012
17.4862

3.0
2226
241
1985
10.8266
928
5750
48.6118
28.2555
20.3563

4.0
2226
201
2025
9.02965
1129
7775
59.1409
38.2064
20.9345

5.0
2225
188
2037
8.44944
1317
9812
68.989
48.2162
20.7728

6.0
2226
162
2064
7.27763
1479
11876
77.4751
58.3587
19.1164

7.0
2226
135
2091
6.06469
1614
13967
84.5469
68.6339
15.913

8.0
2226
106
2120
4.7619
1720
16087
90.0995
79.0516
11.0479

9.0
2226
102
2124
4.58221
1822
18211
95.4426
89.4889
5.9537

10.0
2226
87
2139
3.90836
1909
20350
100
100
0

```
 #Gains Chart2

def plots(agg1, target, type):
  plt.figure(1, figsize=(20, 5))
  plt.subplot(131)
  plt.plot(agg1['DECILE'], agg1['ACTUAL'], label='Actual')
  plt.plot(agg1['DECILE'], agg1['PRED'], label='Pred')
  plt.xticks(range(10, 110, 10))
  plt.legend(fontsize=15)
  plt.grid(True)
  plt.title('Actual vs Predicted', fontsize=20)
  plt.xlabel("Population %", fontsize=15)
  plt.ylabel(str(target) + " " + str(type) + " %", fontsize=15)

  plt.subplot(132)
  X = agg1['DECILE'].tolist()
  X.append(0)
  Y = agg1['DIST_TGT'].tolist()
  Y.append(0)
  plt.plot(sorted(X), sorted(Y))
  plt.plot([0, 100], [0, 100], 'r--')
  plt.xticks(range(0, 110, 10))
  plt.yticks(range(0, 110, 10))
  plt.grid(True)
  plt.title('Cumulative Gains Chart', fontsize=20)
  plt.xlabel("Population %", fontsize=15)
  plt.ylabel(str(target) + str(" DISTRIBUTION") + " %", fontsize=15)
  plt.annotate(round(agg1[agg1['DECILE'] == 30].DIST_TGT.item(), 2), xy=[30, 30],
          xytext=(25, agg1[agg1['DECILE'] == 30].DIST_TGT.item() + 5), fontsize = 13)
  plt.annotate(round(agg1[agg1['DECILE'] == 50].DIST_TGT.item(), 2), xy=[50, 50],
          xytext=(45, agg1[agg1['DECILE'] == 50].DIST_TGT.item() + 5), fontsize = 13)

  plt.subplot(133)
  plt.plot(agg1['DECILE'], agg1['LIFT'])
  plt.axhline(y=1, C='r', ls='--')
  plt.xticks(range(10, 110, 10))
  plt.grid(True)
  plt.title('Lift Chart', fontsize=20)
  plt.xlabel("Population %", fontsize=15)
  plt.ylabel("Lift", fontsize=15)

  plt.tight_layout()

def gains(data, decile_by, target, score):
  inputs = list(decile_by)
  inputs.extend((target, score))
  decile = data[inputs]
  grouped = decile.groupby(decile_by)
  agg1 = pd.DataFrame({}, index=[])
  agg1['ACTUAL'] = grouped.mean()[target]*100
  agg1['PRED'] = grouped.mean()[score]*100
  agg1['DIST_TGT'] = grouped.sum()[target].cumsum()/grouped.sum()[target].sum()*100
  agg1.index.name = 'DECILE'
  agg1 = agg1.reset_index()
  agg1['DECILE'] = agg1['DECILE']*10
  agg1['LIFT'] = agg1['DIST_TGT']/agg1['DECILE']
  plots(agg1, target, 'Distribution')

```

```
lift_train = pd.concat([train_features, scores_train], axis=1)
lift_test = pd.concat([test_features, scores_test], axis=1)

gains(lift_train, ['DECILE'], 'TARGET', 'SCORE')
```

```
gains(lift_test, ['DECILE'], 'TARGET', 'SCORE')
```


## GeostatsPy: Bootstrap for Subsurface Data Analytics in Python

### Michael Pyrcz, Associate Professor, University of Texas at Austin

#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

### PGE 383 Exercise: Bootstrap for Subsurface Data Analytics in Python

Here's a simple workflow, demonstration of bootstrap for subsurface modeling workflows. This should help you get started with building subsurface models that integrate uncertainty in the sample statistics.

#### Bootstrap

Uncertainty in the sample statistics
* one source of uncertainty is the paucity of data.
* do 200 or even less wells provide a precise (and accurate estimate) of the mean? standard deviation? skew? P13?

Would it be useful to know the uncertainty in these statistics due to limited sampling?
* what is the impact of uncertainty in the mean porosity e.g. 20%+/-2%?

**Bootstrap** is a method to assess the uncertainty in a sample statistic by repeated random sampling with replacement.

Assumptions
* sufficient, representative sampling, identical, idependent samples

Limitations
1. assumes the samples are representative
2. assumes stationarity
3. only accounts for uncertainty due to too few samples, e.g. no uncertainty due to changes away from data
4. does not account for boundary of area of interest
5. assumes the samples are independent
6. does not account for other local information sources

The Bootstrap Approach (Efron, 1982)

Statistical resampling procedure to calculate uncertainty in a calculated statistic from the data itself.
* Does this work?  Prove it to yourself, for uncertainty in the mean solution is standard error:

\begin{equation}
\sigma^2_\overline{x} = \frac{\sigma^2_s}{n}
\end{equation}

Extremely powerful - could calculate uncertainty in any statistic!  e.g. P13, skew etc.
* Would not be possible access general uncertainty in any statistic without bootstrap.
* Advanced forms account for spatial information and sampling strategy (game theory and Journel’s spatial bootstrap (1993).

Steps:

1. assemble a sample set, must be representative, reasonable to assume independence between samples

2. optional: build a cumulative distribution function (CDF)
    * may account for declustering weights, tail extrapolation
    * could use analogous data to support

3. For $\ell = 1, \ldots, L$ realizations, do the following:

    * For $i = \alpha, \ldots, n$ data, do the following:

        * Draw a random sample with replacement from the sample set or Monte Carlo simulate from the CDF (if available).

6. Calculate a realization of the sammary statistic of interest from the $n$ samples, e.g. $m^\ell$, $\sigma^2_{\ell}$. Return to 3 for another realization.

7. Compile and summarize the $L$ realizations of the statistic of interest.

This is a very powerful method.  Let's try it out.

#### Objective

In the PGE 383: Stochastic Subsurface Modeling class I want to provide hands-on experience with building subsurface modeling workflows. Python provides an excellent vehicle to accomplish this. I have coded a package called GeostatsPy with GSLIB: Geostatistical Library (Deutsch and Journel, 1998) functionality that provides basic building blocks for building subsurface modeling workflows.

The objective is to remove the hurdles of subsurface modeling workflow construction by providing building blocks and sufficient examples. This is not a coding class per se, but we need the ability to 'script' workflows working with numerical methods.

#### Getting Started

Here's the steps to get setup in Python with the GeostatsPy package:

1. Install Anaconda 3 on your machine (https://www.anaconda.com/download/).
2. From Anaconda Navigator (within Anaconda3 group), go to the environment tab, click on base (root) green arrow and open a terminal.
3. In the terminal type: pip install geostatspy.
4. Open Jupyter and in the top block get started by copy and pasting the code block below from this Jupyter Notebook to start using the geostatspy functionality.

You will need to copy the data file to your working directory.  They are available here:

* Tabular data - sample_data_biased.csv at https://git.io/fh0CW

There are exampled below with these functions. You can go here to see a list of the available functions, https://git.io/fh4eX, other example workflows and source code.

```python
import geostatspy.GSLIB as GSLIB          # GSLIB utilies, visualization and wrapper
import geostatspy.geostats as geostats    # GSLIB methods convert to Python
```

We will also need some standard packages. These should have been installed with Anaconda 3.

```python
import numpy as np                        # ndarrys for gridded data
import pandas as pd                       # DataFrames for tabular data
import os                                 # set working directory, run executables
import matplotlib.pyplot as plt           # for plotting
from scipy import stats                   # summary statistics
import math                               # trig etc.
import scipy.signal as signal             # kernel for moving window calculation
import random
```

#### Set the working directory

I always like to do this so I don't lose files and to simplify subsequent read and writes (avoid including the full address each time).

```python
os.chdir("c:/PGE383")                     # set the working directory
```

#### Loading Tabular Data

Here's the command to load our comma delimited data file in to a Pandas' DataFrame object.

```python
df = pd.read_csv('sample_data_biased.csv') # load our data table
```

Let's drop some samples so that we increase the variations in bootstrap samples for our demonstration below.

```python
df = df.sample(frac = 0.9)                  # extract 50 random samples to reduce the size of the dataset
print('Using ' + str(len(df)) + ' number of samples')
```

    Using 260 number of samples

Visualizing the DataFrame would be useful and we already learned about these methods in this demo (https://git.io/fNgRW).

We can preview the DataFrame by printing a slice or by utilizing the 'head' DataFrame member function (with a nice and clean format, see below). With the slice we could look at any subset of the data table and with the head command, add parameter 'n=13' to see the first 13 rows of the dataset.

```python
print(df.iloc[0:5,:])                   # display first 4 samples in the table as a preview
df.head(n=13)                           # we could also use this command for a table preview
```

           X    Y  Facies  Porosity       Perm
    71   800  249       1  0.177488  51.252076
    95   730  379       1  0.121767  18.610053
    206  400   99       1  0.123665   4.412897
    77   240  449       0  0.091766   1.327225
    189  510  179       1  0.106318   1.823808

      X
      Y
      Facies
      Porosity
      Perm

      71
      800
      249
      1
      0.177488
      51.252076

      95
      730
      379
      1
      0.121767
      18.610053

      206
      400
      99
      1
      0.123665
      4.412897

      77
      240
      449
      0
      0.091766
      1.327225

      189
      510
      179
      1
      0.106318
      1.823808

      102
      640
      49
      1
      0.109308
      7.097099

      103
      410
      899
      1
      0.214893
      1523.989064

      252
      200
      359
      0
      0.077625
      0.540486

      281
      350
      489
      1
      0.115234
      13.746590

      271
      20
      849
      1
      0.089798
      0.946485

      0
      100
      900
      1
      0.115359
      5.736104

      130
      330
      39
      1
      0.095795
      0.714528

      13
      300
      400
      1
      0.118448
      8.495566

#### Summary Statistics for Tabular Data

The table includes X and Y coordinates (meters), Facies 1 and 0 (1 is sandstone and 0 interbedded sand and mudstone), Porosity (fraction), and permeability as Perm (mDarcy).

There are a lot of efficient methods to calculate summary statistics from tabular data in DataFrames. The describe command provides count, mean, minimum, maximum, and quartiles all in a nice data table. We use transpose just to flip the table so that features are on the rows and the statistics are on the columns.

```python
df.describe().transpose()
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      X
      260.0
      483.884615
      256.070518
      0.000000
      300.000000
      450.000000
      672.500000
      990.000000

      Y
      260.0
      535.365385
      304.033377
      9.000000
      279.000000
      549.000000
      839.000000
      999.000000

      Facies
      260.0
      0.819231
      0.385569
      0.000000
      1.000000
      1.000000
      1.000000
      1.000000

      Porosity
      260.0
      0.135470
      0.037974
      0.065626
      0.106315
      0.127176
      0.154677
      0.228790

      Perm
      260.0
      218.886754
      580.642334
      0.093252
      3.636162
      15.034902
      73.947940
      5308.842566

#### Visualizing Tabular Data with Location Maps

It is natural to set the x and y coordinate and feature ranges manually. e.g. do you want your color bar to go from 0.05887 to 0.24230 exactly? Also, let's pick a color map for display. I heard that plasma is known to be friendly to the color blind as the color and intensity vary together (hope I got that right, it was an interesting Twitter conversation started by Matt Hall from Agile if I recall correctly). We will assume a study area of 0 to 1,000m in x and y and omit any data outside this area.

```python
xmin = 0.0; xmax = 1000.0               # range of x values
ymin = 0.0; ymax = 1000.0               # range of y values
pormin = 0.05; pormax = 0.25;           # range of porosity values
nx = 100; ny = 100; csize = 10.0
cmap = plt.cm.plasma                    # color map
```

Let's try out locmap. This is a reimplementation of GSLIB's locmap program that uses matplotlib. I hope you find it simpler than matplotlib, if you want to get more advanced and build custom plots lock at the source. If you improve it, send me the new code. Any help is appreciated. To see the parameters, just type the command name:

```python
GSLIB.locmap
```

    <function geostatspy.GSLIB.locmap(df, xcol, ycol, vcol, xmin, xmax, ymin, ymax, vmin, vmax, title, xlabel, ylabel, vlabel, cmap, fig_name)>

Now we can populate the plotting parameters and visualize the porosity data.

```python
plt.subplot(111)
GSLIB.locmap_st(df,'X','Y','Porosity',xmin,xmax,ymin,ymax,pormin,pormax,'Well Data - Porosity','X(m)','Y(m)','Porosity (fraction)',cmap)
plt.subplots_adjust(left=0.0, bottom=0.0, right=1.0, top=1.2, wspace=0.2, hspace=0.2)
plt.show()
```

#### A Very Simple Bootstrap Method in Python

If you are new to bootstrap and Python, here's the most simple code possible for bootstrap.

* specify the number of bootstrap realizations, $L$
* declare a list to store the bootstrap realizations of the statistic of interest
* loop over L bootstrap realizations
    * n MCS, random samples with replacement for a new realization of the data
    * calculate the realization of the statistic from the realization of the data
* summarize the resulting uncertainty model, histogram, summary statistics etc.

```python
import random                          # import random package
L = 1000                               # set the number of bootstrap realizations
por_avg_real = []                      # declare an empty list to store the bootstrap realizations of the statistic
for k in range(0,L):                   # loop over the L bootstrap realizations
    samples = random.choices(df['Porosity'].values, k=len(df)) # n Monte Carlo simulations
    por_avg_real.append(np.average(samples)) # calculate the statistic of interest from the new bootstrap dataset
plt.hist(por_avg_real,color = 'red',alpha = 0.2,edgecolor = 'black') # plot the distribution, could also calculate any summary statistics
plt.xlabel('Boostrap Realizations of Average Porosity'); plt.ylabel('Frequency'); plt.title('Uncertainty Distribution for Average Porosity')
```

Now we proceed with a more complicated, robust work by first quantifying the spatial bias in the samples and assigned data weights to mitigate this bias.

#### Declustering

Let's calculate some declustering weights. There is a demonstration on declustering here https://git.io/fhgJl if you need more information.

```python
wts, cell_sizes, dmeans = geostats.declus(df,'X','Y','Porosity',iminmax = 1, noff= 10, ncell=100,cmin=10,cmax=2000)
df['Wts'] = wts                            # add weights to the sample data DataFrame
df.head()                                  # preview to check the sample data DataFrame

def weighted_avg_and_std(values, weights): # function to calculate weighted mean and st. dev., from Eric O Lebigot, stack overflow,
    average = np.average(values, weights=weights)
    variance = np.average((values-average)**2, weights=weights)
    return (average, math.sqrt(variance))

sample_avg, sample_stdev = weighted_avg_and_std(df['Porosity'],df['Wts'])
print('Declustered mean = ' + str(round(sample_avg,3)) + ' and declustered standard deviation = ' + str(round(sample_stdev,3)))
```

    There are 260 data with:
       mean of      0.13417733651923078
       min and max  0.058547873 and 0.228790002
       standard dev 0.03831442273384987
    Declustered mean = 0.121 and declustered standard deviation = 0.032

##### A Couple of Bootstrap Realizations

We will attempt boostrap by-hand and manually loop over $L$ realizations and draw $n$ samples to calculate the summary statistics of interest, mean and variance. The choice function from the random package simplifies sampling with replacement from a set of samples with weights.

This command returns a ndarray with k samples with replacment from the 'Porosity' column of our DataFrame (df) accounting for the data weights in column 'Wts'.
```p
samples1 = random.choices(df['Porosity'].values, weights=df['Wts'].values, cum_weights=None, k=len(df))
```

It is instructive to look at a couple of these realizations from the original declustered data set.

```python
samples1 = random.choices(df['Porosity'].values, weights=df['Wts'].values, cum_weights=None, k=len(df))
samples2 = random.choices(df['Porosity'].values, weights=df['Wts'].values, cum_weights=None, k=len(df))

print('Bootstrap means, realization 1 = ' + str(np.average(samples1)) + ' and realization 2 = ' + str(np.average(samples2)))

plt.subplot(131)
GSLIB.hist_st(df['Porosity'],pormin,pormax,False,False,20,df['Wts'],'Porosity (fraction)','Declustered Porosity')

plt.subplot(132)
GSLIB.hist_st(samples1,pormin,pormax,False,False,20,None,'Bootstrap Sample - Realizaton 1','Bootstrap Porosity 1')

plt.subplot(133)
GSLIB.hist_st(samples2,pormin,pormax,False,False,20,None,'Bootstrap Sample - Realizaton 2','Bootstrap Porosity 2')

plt.subplots_adjust(left=0.0, bottom=0.0, right=3.0, top=1.2, wspace=0.2, hspace=0.2)
plt.show()
```

Note that the bootstrap distributions vary quite a bit from the original.

#### Summarizations Over Bootstrap Realizations

Let's make a loop to conduct $L$ resamples and calculate the average and standard deviation for each  ($m^\ell$, $\sigma^2_{\ell}$,  for $\ell = 0,\dots,L-1$). We then summarization over these $L$ realizations.

I did not find any built-in, concise functions to accomplish this, i.e. with a single line of code, so we are going to do it by hand.

To understand this code there are just a couple of Python concepts that you need to add to your Python arsenal.

1. declaring arrays - NumPy has a lot of great array (ndarray) functionality. There are build in functions to make a ndarray of any length (and dimension). This includes 'zeros', 'ones' and 'rand', so when we use this code:

```p
mean = np.zeros(L); stdev = np.zeros(L)
```

    we're making arrays of length $L$ pre-populated with zeros.

2. For Loops - when we are using the command below, we are instructing the computer to loop over all the indented code below the command for $l = 0,1,2,\ldots,L-1$ times. For each loop the $l$ variable increments, so we can use this to save each result to a different index in the arrays mean and stdev. Note, Python arrays index starting at 0 and stop at the length - 1.

```p
for l in range(0, L):
```

    we are running each bootstrap resampled realization, calculating the average and standard deviation and storing them in the arrays that we already declared.

```python
L = 1000                                 # set the number of realizations
mean = np.zeros(L); stdev = np.zeros(L)    # declare arrays to hold the realizations of the statistics
for l in range(0, L):                      # loop over realizations
    samples = random.choices(df['Porosity'].values, weights=df['Wts'].values, cum_weights=None, k=len(df))
    mean[l] = np.average(samples)
    stdev[l] = np.std(samples)

plt.subplot(121)
GSLIB.hist_st(mean,0.11,0.15,False,False,50,None,'Average Porosity (fraction)','Bootstrap Uncertainty in Porosity Average')

plt.subplot(122)
GSLIB.hist_st(stdev,0.015,0.045,False,False,50,None,'Standard Deviation Porosity (fraction)','Bootstrap Uncertainty in Porosity Standard Deviation')

plt.subplots_adjust(left=0.0, bottom=0.0, right=3.0, top=1.2, wspace=0.2, hspace=0.2)
plt.show()

print('Summary Statistics for Bootstrap Porosity Mean Realizations:')
print(stats.describe(mean))
print('P10 ' + str(round(np.percentile(mean,10),3)) + ', P50 ' + str(round(np.percentile(mean,50),3)) + ', P90 ' + str(round(np.percentile(mean,90),3)))

print('\nSummary Statistics for Bootstrap Porosity Standard Deviation Realizations:')
print(stats.describe(stdev))
print('P10 ' + str(round(np.percentile(stdev,10),3)) + ', P50 ' + str(round(np.percentile(stdev,50),3)) + ', P90 ' + str(round(np.percentile(stdev,90),3)))

```

#### Comments

This was a basic demonstration of bootstrap. Much more could be done, you could replace the statistics, average and standard deviation with any other statistics, for example P90, kurtosis, P13 etc. I have other demonstrations on the basics of working with DataFrames, ndarrays, univariate statistics, plotting data, declustering, data transformations, trend modeling and many other workflows available at https://github.com/GeostatsGuy/PythonNumericalDemos and https://github.com/GeostatsGuy/GeostatsPy.

I hope this was helpful,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin

#### More Resources Available at: [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)