# How to evaluate a monitoring system for adaptive policies: Criteria for signposts selection and their model-based evaluation

## Application to the Niger River

*Based on the omonymous article, accepted on [Climatic Change](https://link.springer.com/journal/10584), by Raso et al, 2018

Author: Luciano Raso, Delft University of Technology, Delft, The Netherlands. Section of Policy Analysis
email: l.raso@tudelft.nl

```python
import numpy as np
#%matplotlib notebook
from matplotlib import pyplot as plt
plt.style.use('ggplot')
import pandas as pd
#import scipy.stats as stats
from scipy.stats import gumbel_r
from IPython.display import Image
```

### Policy problem

The city of Niamey, capital of Niger, is located along the shore of the Niger River. The irregular flow of the river poses a risk of flood for the city. At the inception of the adaptive policy, in 1985, the levees at Niamey protect the city until a maximum discharge of **2000 $m^3/s$**. A discharge above this threshold causes expensive inundations.

```python
q_flood = 2000 # m3/s
```

```python
Image('../figures/Niger/Map_of_River_Niger_LR2.jpg',width=600)
```

The river authority, the ``Autorité du Bassin du Fleuve Niger'' (ABN), prepares an adaptive policy to possible future changes. The ABN wanst to make sure that the hazard level, i.e. the yearly probability of flood, does not exceed the level presently acceptable, $F^*$: the frequency of inundation considered acceptable is below one over twenty-five years, i.e. $F^* <1/25$ $yr^{-1}$.

```python
R_star = 25
F_star = 1/R_star
```

In the future, climate change may put at risk the capacity of levees to offer sufficient protection.
The adaptive policy envisages the construction of higher levees if flood hazard exceed this value. These new levees would raise the maximum discharge to **2700 $m^3/s$**. Raising the levees is an expensive action, therefore it will be taken only if justified by an increase in flood hazard.

```python
q_flood_action = 2700 #m^3/s
```

## Signposts

A signpost is a statistic, i.e. a rule that aggregates, transforms, or filters original data.

```python
class Signpost:
    def __init__(self,name,unit):
        self.name=name
        self.unit=unit
        self.ts=pd.DataFrame(None) #time series of signpost estimates
        self.statistic=[]
        self.trigger=[]
```

In the following analysis, we will compare two alternative monitoring systems based on a single signpost.

${S_E}$: Average of max yearly discharge at Niamey,
on a 15-year moving window.

```python
S_E = Signpost(name='$S_E$',unit='$m^3/s$')
S_E.window = 15
S_E.statistic = lambda data: pd.Series(data).rolling(S_E.window).mean() #average value on S_E.window past data
```

${S_{10}}$: 9/10 quantile of max yearly discharge at Niamey, on a 30-year moving window.

```python
S_10=Signpost(name='$S_{10}$',unit='$m^3/s$')
S_10.window=30
S_10.quantile=9/10
S_10.statistic=lambda data: pd.Series(data).rolling(S_10.window).quantile(S_10.quantile)# general function, 9th decile on a window of 30 years
```

Parameters that $S_E$ and $S_{10}$ track are:
$S_E$, Expected value of flood intensity at Niamey.
$S_{10}$, 10-year return period flood at Niamey.
$S_E$ has been selected because it is already used in flood protection policies, even if in a different context. $S_{10}$ has been selected for its closeness to the critical flood frequency. Comparing $S_{10}$ to $S_E$ we test whether a gain in relevance can compensate the loss in observability.

### Additional material

Reviewer #1 suggested to test a signpost designed to detect the 25-year return period directly.
Following his/her suggestion, we added the signpost below

$S_\max$: Maximum value of max yearly discharge, on a 25-year moving window.

```python
S_max=Signpost(name='$S_\max$',unit='$m^3/s$')
S_max.window=25
S_max.statistic=lambda data: pd.Series(data).rolling(S_10.window).max() #, max value on a window of 25 years
```

## Model

The Generalized Extreme Value (GEV) model, that relates single data-points to system characteristics (i.e. model parameters) fits well for the scope of demonstrating the use of a model for signposts assessment.

The GEV model [Coles, 2001] estimates the frequency of extreme events from maximum values over fixed periods. In flood frequency analysis, the period is generally a year, and the variable the max yearly discharge. A GEV model is defined by three parameters: location ($\mu$), scale ($\sigma$), and shape ($\xi$).

The critical assumptions are the location and scale parameters, $\mu$ and $\sigma$, which define the distribution. The shape parameter $\xi$ is particularly difficult to estimate from a short time series, requiring a much longer horizon than the period for which we have data, or some regional analysis [Panthou et al., 2012].
To reduce ambiguity in data interpretation, we will constrain the shape parameter assuming it to be zero. In this case the GEV reduces to a Gumbel distribution, a.k.a. GEV type III.

```python
model = gumbel_r   # define model
```

Equation below represents the Gumbel distribution.
$$ F(q)= e^{-e^{-\frac{q-\mu}{\sigma}}} $$

where $q$ is the yearly maximum discharge, $\mu$ and $\sigma$ the location and scale parameter. The Gumbel distribution is equivalent to a Generalized Extreme Value distribution for $\xi=0$, a.k.a. Type-I GEV.

Estimated on historical data until 1970, find $\mu_h=1150$ $m^3/s$  and $\sigma_h=170$ $m^3/s$

```python
mu_hist = mu_hist_85 = 1150 #m3/s
sigma_hist = sigma_hist_85 = 170 #m3/s
```

The GEV is a stochastic model whose model parameters are generally estimated from historical time series, assuming stationary conditions. Here, instead, we use the GEV in an exploratory mode, i.e. we explore the model response for a large set of critical assumptions.

Changed, over time, reaching $\mu_h=600$ $m^3/s$  and $\sigma_h=240$ $m^3/s$

```python
sigma_all = np.arange(0,400)
mu_all = np.arange(q_flood,0,-1)
```

Explore critical region for different values of $\mu$ and $\sigma$. Exploratory modeling, or stress test.

#### Non-stationary GEV

Instead of a quasi-stationary GEV, an explicit stationary GEV can be used. This has been suggested by *reviewer 3* of the article. We implement here a model of nonstationary Gumbel.
With this model, it is easy to test signposts both a-priori, using an exploratory approach, and a-posteriori, on observed discharge data. We leave to future readers any further testing of this model, or others, using the methods proposed in the manuscript.

Equation below represents the nonstationary Gumbel distribution implemented below.
$$ F(q_t)= e^{-e^{-\frac{q_t- (\mu_{0} + \mu_{1} \cdot t)}{\sigma}}} $$
We consider only the non-stationary of the location parameters as growing linearly with time

```python
def non_stationary_model(mu_0, sigma, mu_1 = 0, t = 0):
    """ Define a nostationary model for a Gumbel distribution,
    with linearly time-dependent location parameter, as in the following equation
    $$ F(q_t)= e^{-e^{-\frac{q_t- (\mu_{0} + \mu_{1} \cdot t)}{\sigma}}} $$

    """
    return(gumbel_r.freeze(loc = mu_0 + mu_1 * t, scale = sigma))
```

### Critical Region

The boundary of the critical region in $\mu$ and $\sigma$ is the set of $[\mu^*,\sigma^*]$, space in $R^2$, can be found by inverting the Equation of the Gumbel distribution, conditioning the flood frequency to be equal to the critical flood frequency, i.e. $F=F^*$, and the discharge equal to the flood threshold level, i.e. $q=q_{flood}$. Then one can find the relation at Equation
\begin{equation}\label{F_star}
\mu^*= q_{flood} \hspace{3mm} + \hspace{3mm} \log \Bigg(-\log \Big(1-{F^*}\Big)\Bigg) \cdot \sigma^*
\end{equation}
In the equation above, the relation [$\mu^*,\sigma^*$] is a straight line, in which $q_{flood}$ is the intercept for $\sigma=0$, and $\log(-\log\left(1- {F^*}\right))$ its slope.

```python
def critical_region(q_flood,F_star,hist=[],**kwargs):
    """"
    Create stress-test plot (critical region), for a given discharge threshold level of protection
    """

    mu_all= np.arange(q_flood,0,-1)
    slope= 1 / np.log(-np.log(1-F_star))
    sigma_critic = (mu_all - q_flood ) * slope

    # set plot boundaries
    mu_limit = [q_flood*0.4,q_flood*1.05]
    sigma_max = (q_flood*0.4 - q_flood ) * slope
    sigma_limit = [0,sigma_max*1.1]
    sigma_center = (q_flood*0.65 - q_flood ) * slope

    # create plot
    plt.plot(mu_all,sigma_critic,color='b',label='$F^*$ Frequency')
    plt.xlabel('Location $\mu$ $[m^3/s]$',fontsize=12)
    plt.ylabel('Scale $\sigma$ $[m^3/s]$',fontsize=12 )
    plt.title('Stress Test',fontsize=18)

    if hist is not []:
        plt.plot(hist[0],hist[1],marker='o')
    #ax.annotate('1985', xy=(mu_hist_85, sigma_hist), xytext=(mu_hist_85-200, sigma_hist),arrowprops=dict(facecolor='black', shrink=0.01))
        plt.text(hist[0]*1.02,hist[1]*0.95, 'Historical',fontsize=12)

    plt.text(mu_limit[1]*0.76, sigma_max*0.95, 'Failure Region ($F>F^*$)',fontsize=12,
            bbox={'facecolor':'white', 'alpha':0.5, 'pad':5})
    plt.text(mu_limit[0]*1.05, 50, 'Success Region ($F<F^*$)',fontsize=12,
            bbox={'facecolor':'white', 'alpha':0.5, 'pad':5})
    plt.text(q_flood*0.65, sigma_center, '$F^*$ Frequency ',color='b',fontsize=14)

    plt.xlim(mu_limit)
    plt.ylim(sigma_limit)

```

```python
# plot critical region
fig = plt.figure(figsize=(8,4))
critical_region(q_flood,F_star,hist=[mu_hist,sigma_hist])
#plt.savefig("../Figures/critical_region.pdf")
plt.show()
```

Figure above shows the ``critical region'', i.e. the policy success region for all possible values of critical assumptions. The blue line in the figure correspond to the boundary of policy success: if both $\mu$ and $\sigma$ fall below it, the policy is successful, otherwise there is a need for action. The frontier line corresponds to the critical flood frequency $F^*=1/25$ $yrs^{-1}$. This line can be derived analytically, as described in the procedure in [Raso et al].
The red point in the figure represents the location and scale parameters estimated from historical data until 1985.

### Signposts Analysis

The critical assumptions are the location and scale parameters, $\mu$ and $\sigma$, which define the distribution. We analyse here the signpost response when parameters change.

```python
#plot
def signpost_analysis_plot (signpost,par_name,par,S_mean,S_up,S_low):
    plt.plot(par,S_mean, label='average',color='b')
    plt.plot(par,S_up,ls='--',color='b',label='90% uncertainty band')
    plt.plot(par,S_low,ls='--',color='b')
    #unc=plt.fill_between(par,S_up, S_low,color='gray',label='uncertainty band')
    plt.xlabel(par_name+' $[m^3/s]$',fontsize='15')
    plt.ylabel(signpost.name +' ['+ signpost.unit +']',fontsize='15')
    plt.title(signpost.name + ' vs ' + par_name)
    plt.legend(loc='lower right')
    #plt.gca().set_ylim(bottom=0)
    #plt.gca().set_xlim(left=0)
    plt.grid(True)
```

#### $S_E$ variation to Model Parameters

```python
# Model exploratory
mu_expl=mu_all[::-1]
sigma_expl = sigma_all
model_expl_mu=model.freeze(loc=mu_expl,scale=sigma_hist)
model_expl_sigma=model.freeze(loc=mu_hist,scale=sigma_all)
```

```python

confidence = .90
conf = 1.64 # 1.15 = 75%  1.64: 90%

# S_E vs mu
S_E_mu_mean = model_expl_mu.mean()
S_E_mu_var = model_expl_mu.var()/S_E.window
S_E_mu_up=S_E_mu_mean + conf*np.sqrt(S_E_mu_var)
S_E_mu_low=S_E_mu_mean - conf*np.sqrt(S_E_mu_var)

#S_E vs sigma
S_E_sigma_mean=model_expl_sigma.mean()
S_E_sigma_var=model_expl_sigma.var()/S_E.window
S_E_sigma_up=S_E_sigma_mean + conf*np.sqrt(S_E_sigma_var)
S_E_sigma_low=S_E_sigma_mean - conf*np.sqrt(S_E_sigma_var)

# Signpost confidence
S_E.confidence_range_up=conf*np.sqrt(S_E_mu_var[sigma_hist-1])
S_E.confidence_range_low=conf*np.sqrt(S_E_mu_var[sigma_hist-1])
```

```python
f, (ax_mu, ax_sigma) = plt.subplots(1,2,sharey=True,figsize=(10,5))
plt.figure(1)
ax_mu=plt.subplot(121)
signpost_analysis_plot (S_E,'Location $\mu$',mu_expl,S_E_mu_mean,S_E_mu_up,S_E_mu_low)
plt.xlim(500)
plt.ylim((300,2500))
ax_sigma=plt.subplot(122)
plt.xlim([100,400])
ylim = ax_mu.get_ylim()
ax_sigma.set_ylim(ylim)
signpost_analysis_plot (S_E,'Scale $\sigma$',sigma_all,S_E_sigma_mean,S_E_sigma_up,S_E_sigma_low)
f.tight_layout()
plt.savefig("../Figures/Niger/S_E.pdf")

```

#### $S_{10}$ variation to Model Parameters
$S_{10}$ variance: Knowing the convergence of empirical distribution , i.e. $\hat{F}_n(t)  \rightarrow N \left(F(t), \frac{F(t)\cdot (1-F(t))}{n}\right)$, where $\hat{F}_n(t)$ is the empirical distribution, $F(t)$ the original one, $t$ the quantile and $n$ the sample size. $S_{10}$ is the distributed $F^{-1}\left(\hat{F}_n\left(\frac{9}{10}\right)\right)$

```python
sigma_F_hat = np.sqrt((S_10.quantile)*(1-(S_10.quantile))/S_10.window)
#+conf*sigma_F_hat+S_10.quantile #check
```

```python
#analytical
#S_10 vs mu
S_10_mu_mean = model_expl_mu.ppf(S_10.quantile)
S_10_mu_up=model_expl_mu.ppf(S_10.quantile+conf*sigma_F_hat)
S_10_mu_low=model_expl_mu.ppf(S_10.quantile-conf*sigma_F_hat)

#S_10 vs sigma
S_10_sigma_mean=model_expl_sigma.ppf(S_10.quantile)
S_10_sigma_up=model_expl_sigma.ppf(S_10.quantile+conf*sigma_F_hat)
S_10_sigma_low=model_expl_sigma.ppf(S_10.quantile-conf*sigma_F_hat)

# Signpost confidence
S_10.confidence_range_up=S_10_mu_up[sigma_hist-1]-S_10_mu_mean[sigma_hist-1]
S_10.confidence_range_low=-S_10_mu_low[sigma_hist-1]+S_10_mu_mean[sigma_hist-1]
```

    /Users/lraso/anaconda/lib/python3.5/site-packages/scipy/stats/_distn_infrastructure.py:1907: RuntimeWarning: invalid value encountered in multiply
      lower_bound = self.a * scale + loc
    /Users/lraso/anaconda/lib/python3.5/site-packages/scipy/stats/_distn_infrastructure.py:1908: RuntimeWarning: invalid value encountered in multiply
      upper_bound = self.b * scale + loc

```python
#plot
f,  (ax_muax_mu,  ax_sigmaax_sigma) = plt.subplots(1,2,sharey=True,figsize=(10,5))
ax_mu = plt.subplot(121)
signpost_analysis_plot (S_10,'Location $\mu$',mu_expl,S_10_mu_mean,S_10_mu_up,S_10_mu_low)
plt.xlim(500)
plt.ylim((500,2500))
ax_sigma = plt.subplot(122)
plt.xlim([100,400])
ylim = ax_mu.get_ylim()
ax_sigma.set_ylim(ylim)
signpost_analysis_plot (S_10,'Scale $\sigma$',sigma_all,S_10_sigma_mean,S_10_sigma_up,S_10_sigma_low)
f.tight_layout()
#plt.savefig("../Figures/S_10.pdf")

```

### Analysis for signpost $S_\max$

The convergence rule to estimate the quantiles distribution cannot be used for this signpost, the therefore we use here a montecarlo approach.

```python
sample_size = 15000

q_yearVSmu_random = model_expl_mu.rvs(size=(S_max.window,sample_size,len(mu_expl)))
q_yearVSsigma_random = model_expl_sigma.rvs(size=(S_max.window,sample_size,len(sigma_expl)))

#S_max vs sigma
#model_expl_sigma
# Signpost confidence
```

```python
S_maxVSmu_random = np.max(q_yearVSmu_random,axis=0)
S_maxVSsigma_random = np.max(q_yearVSsigma_random,axis=0)

```

```python
#S_max vs mu
S_max_mu_mean = np.mean(S_maxVSmu_random,axis=0)
S_max_mu_up = np.percentile(S_maxVSmu_random, (confidence+1)/2*100,axis=0)
S_max_mu_down = np.percentile(S_maxVSmu_random, (1-confidence)/2*100,axis=0 )

#S_max vs sigma
S_max_sigma_mean = np.mean(S_maxVSsigma_random,axis=0)
S_max_sigma_up = np.percentile(S_maxVSsigma_random, (confidence+1)/2*100,axis=0)
S_max_sigma_down = np.percentile(S_maxVSsigma_random, (1-confidence)/2*100,axis=0 )

# Signpost confidence
S_max.confidence_range_up=S_max_mu_up[mu_hist-1]-S_max_mu_mean[mu_hist-1]
S_max.confidence_range_low=-S_max_mu_down[mu_hist-1]+S_max_mu_mean[mu_hist-1]

```

```python
f, (ax_mu, ax_sigma) = plt.subplots(1,2,sharey=True,figsize=(10,5))
plt.figure(1)
ax_mu=plt.subplot(121)
signpost_analysis_plot (S_max,'Location $\mu$',mu_expl,S_max_mu_mean,S_max_mu_up,S_max_mu_down)
plt.xlim(500)
plt.ylim((800,3000))
ax_sigma=plt.subplot(122)
plt.xlim([100,400])
ylim = ax_mu.get_ylim()
ax_sigma.set_ylim(ylim)
signpost_analysis_plot (S_max,'Scale $\sigma$',sigma_all,S_max_sigma_mean,S_max_sigma_up,S_max_sigma_down)
f.tight_layout()
#plt.savefig("../Figures/Niger/S_max.pdf")

```

### Trigger Values Selection

A trigger is the signpost value at which adaptation is required. In the test case, policy requires adaptation if flood frequency $F$ exceeds the threshold level $F^*$, of one over 25 years. Flood frequency level corresponds to a [$\mu^*,\sigma^*$], that is a 2-dimensional space in model parameters.

Trigger points are selected by sampling three equidistant combinations of location and scale parameter from the parameters space, in proximity of the parameters historically observed. Then, the three sets of parameters are mapped over the expected signpost values, finding the signpost values that would be measured, on average, for each set of parameters.

```python
sigma_trigger = np.linspace(150.,270.,num=5)[1:-1] #three levels.
mu_trigger = q_flood + np.log (-np.log(1-F_star) ) * sigma_trigger
model_expl_trigger=model.freeze(loc=mu_trigger,scale=sigma_trigger)
```

```python
S_E.trigger = model_expl_trigger.mean()
S_10.trigger = model_expl_trigger.ppf(S_10.quantile)
S_max.trigger = model_expl_trigger.ppf(24/25)
```

## Signposts in action

```python
real_data=1
```

Loading observed data, or create artificial ones.

```python
if real_data:
    q_max_df = pd.read_csv('../data/Niger/rbflood_updated.csv',index_col='year')
    # TRY ELSE load max discharge data.
else: #artificial data
    half_data_length = 30
    mu_artificial = concatenate([half_data_length*[mu_hist] , mu_hist+(mu_present-mu_hist)*arange(0,half_data_length)/half_data_length])
    sigma_artificial = concatenate([half_data_length*[sigma_hist], sigma_hist+(sigma_present-sigma_hist)*arange(0,half_data_length)/half_data_length])
    model_artificial = model.freeze(loc=mu_artificial,scale=sigma_artificial)
    q_data=model_artificial.rvs()

    rng = pd.date_range('1/1/1950', periods=half_data_length*2, freq='A')
    q_max_ts = pd.DataFrame(q_data, index=rng)
```

```python
q_max_df_cut=q_max_df['red_flood_max'].loc[1950:2013]
```

```python
plt.figure(figsize=(10,4))
plt.plot(q_max_df_cut.loc[:2013])
plt.xlabel('year')
plt.ylabel('$[m^3/s]$')
plt.title('Max yearly discharge of red-flood at Niamey, Niger')
plt.show()

S_10.trigger[1:]
```

#### Calculate signposts estimate over time

```python
S_E.ts= S_E.statistic(q_max_df_cut)#.apply(S_E.statistic)
S_10.ts =S_10.statistic(q_max_df_cut)
S_max.ts = S_max.statistic(q_max_df_cut)
```

#### Signpost value: plot over time
Confidence range: band considering $\mu_{hist},\sigma_{hist}$, and confidence 90%

```python
def signpost_over_time(signpost,start_date,end_date):

    ts_period=signpost.ts.loc[start_date:end_date]
    plt.fill_between(ts_period.index,ts_period-signpost.confidence_range_low, ts_period+signpost.confidence_range_up,color='0.65',label='90% uncertainty band')

    sign=plt.plot(ts_period,label='Signpost value',linewidth=2,color='b')

    plt.title(signpost.name + ' over time')
    plt.ylabel(signpost.name+' ['+ signpost.unit+ ']',fontsize='15')
    plt.xlabel('Year',fontsize='15')
    pos_text=0.9*(plt.gcf().gca().get_xlim()[1])+0.1*(plt.gcf().gca().get_xlim()[0])

    trigger_line=plt.axhline(y=signpost.trigger[0],ls='--', color='k',label='Trigger values')
    for trigger_value in signpost.trigger[1:]:
        trigger_line=plt.axhline(y=trigger_value,ls='--', color='k')
        #plt.text(pos_text,trigger_value,str("{0:.0f}".format(trigger_value)))
    plt.gca().set_xlim(right=2015)
    plt.legend(loc='upper left')
    plt.grid(True)
    #plt.xlim((signpost.ts.index[0], 2015))
```

Signpost $S_E$ over time

```python
START_DATE = 1985
END_DATE = 2013
plt.figure(figsize=(10,5))
signpost_over_time(S_E,START_DATE,END_DATE)
plt.ylim([1000,1700])
#plt.savefig("../Figures/S_E_overtime.pdf")
plt.show()
```

Signpost $S_{10}$ over time

```python
plt.figure(figsize=(10,5))
signpost_over_time(S_10,START_DATE,END_DATE)
plt.ylim([1000,2400])
#plt.savefig("../Figures/Niger/S_10_overtime.pdf")
plt.show()
```

Signpost $S_{\max}$ over time

```python
plt.figure(figsize=(10,5))
signpost_over_time(S_max,START_DATE,END_DATE)
plt.ylim([1200,2800])
#plt.savefig("../Figures/Niger/S_max_overtime.pdf")
plt.show()
```

```python
f = plt.figure(figsize=(10,10))
plt.subplot(2, 1, 1)
signpost_over_time(S_E,START_DATE,END_DATE)
plt.ylim([900,1900])

plt.subplot(2, 1, 2)
signpost_over_time(S_max,START_DATE,END_DATE)
plt.ylim([1200,3000])
#plt.savefig("../Figures/Niger/S_overtime.pdf")

```

```python

```

```python

```

```python

```


```python
import matplotlib.pyplot as plt
import networkx as nx
import numpy as np
from scipy import stats
import scipy as sp

from ei_net import *
from utilities import show_values

%matplotlib inline
```

```python
##########################################
############ PLOTTING SETUP ##############
EI_cmap = "Greys"
where_to_save_pngs = "../figs/pngs/"
where_to_save_pdfs = "../figs/pdfs/"
save = True
plt.rc('axes', axisbelow=True)
plt.rc('axes', linewidth=2)
plt.rc('text', usetex=True)
hfont = {'fontname':'Helvetica'}
plt.rc('font',**{'family':'sans-serif','sans-serif':['Helvetica']})
##########################################
##########################################
```

# Uncertainty and causal emergence in complex networks

# Chapter 03: Determinism and Degeneracy

Determinism and degeneracy are two properties that exist in every causal structure, and they govern the degree of certainty about the relationships between causes and effects in a system based on the network connectivity. A visual explanation of these two quantities are shown this notebook. Determinism and degeneracy can be arrived at by decomposing the KL Divergence that defines $EI$, but since they are fundamental to the notion of causality, they can also be built up from first principles, as we do here.

Previously we defined an intervention as introducing random walkers on a node in the network, $do_{net}(v_i)$. The amount of uncertainty in the paths of those random walkers is how much information is lost from noise if the random walker is placed on a $v_i$. This is in turn a function of the strength of the output of a node, i.e., the certainty of transitions or dependencies (the determinism of those transition profiles). Therefore we define the determinism using the average entropy of $W^{out}_{i}$ and the degeneracy with the entropy of $W^{in}$ such that:

\begin{align}
    \text{determinism} &= \text{log}_2(N) - \dfrac{1}{N_{out}}\displaystyle \sum_i^{N_{out}} H[W^{out}_{i}] \tag{4}\\\
    \text{degeneracy} &= \text{log}_2(N) - H[W^{in}] \tag{5}
\end{align}

_______________________________

In a maximally deterministic system wherein all nodes have a single output and every entry in $W^{out}_{i}=1.00$, the determinism is just $\text{log}_2(N)$ because their $H[W^{out}_{i}]=0.00$. Conversely, the degeneracy is the amount of information lost via an overlap in input weights (e.g. if multiple nodes output to the same node). The degeneracy is $\text{log}_2(N) - H[W^{in}]$. In a perfectly non-degenerate system wherein all nodes have an equal input weight, the degeneracy is zero because $H[W^{in}] \approx H_{max}$.

These two quantities provide clear explanations for why canonical networks have the $EI$ they do. For example, as the size of an Erdös-Rényi random network increases, its value for degeneracy approaches zero, which means the the $EI$ of a random network is driven only by the determinism of the network. Similarly, in $d$-dimensional ring lattice networks, the degeneracy term is always zero. Therefore, the $EI$ of a ring lattice structure reduces to the determinism of that structure. Ring networks with an average degree $\langle k \rangle$ will have a higher $EI$ than ER networks with the same average degree because ring networks will have a higher determinism value. Interestingly, in the case of star networks, the degeneracy term governs the decay of the $EI$ such that hub-and-spoke-like structures quickly become uninformative in terms of cause and effect.

```python
col1 = np.array([0.06581203, 0.49116525, 0.47591044, 1.0])
col2 = np.array([0.68306918, 0.25012518, 0.37634705, 1.0])
col1 = '#5ec07f'
col2 = '#ee8f5f'
col1 = "#008cd0"
col2 = "#e49822"
```

```python
fig, ((ax00, ax01), (ax02, ax03), (ax04, ax05)) = plt.subplots(3, 2, figsize=(15,15))
nodenames = [r'\textbf{A}',r'\textbf{B}',r'\textbf{C}',
             r'\textbf{D}',r'\textbf{E}',r'\textbf{F}']
N = 6

plt.axes([0.325, 0.755, 0.125, 0.125])
G00 = nx.DiGraph()
G00.add_edges_from([(0,1), (1,2), (2,3), (3,4), (4,5), (5,0)])
plt.axis('off')
plt.xlim(-1.3, 1.3)
plt.ylim(-1.2, 1.2)
pos00 = nx.circular_layout(G00)
nx.draw_networkx_nodes(G00, pos00, node_color='white', node_size=500,
                       linewidths=3.0, alpha=0.98).set_edgecolor(col1)
nx.draw_networkx_nodes(G00, pos00, nodelist=[0],
                       node_color='#333333', linewidths=3.0,
                       node_size=700).set_edgecolor(col1)
nx.draw_networkx_edges(G00, pos00, edge_color="#333333",
                       width=2.5, alpha=0.98, arrowsize=14)
nx.draw_networkx_labels(G00, pos00,
                        labels=dict(zip([0,1,2,3,4,5], nodenames)))
nx.draw_networkx_labels(G00, pos00,
                        labels=dict(zip([0], [nodenames[0]])),
                        font_color='w',  font_size=18)
ax00.bar(G00.nodes(), W_out(G00)[0]+0.009, color=col1,
         linewidth=2.0, edgecolor='k', width=0.72)
ax00.set_ylim(0.0,1.02)
ax00.set_xticks([0,1,2,3,4,5])
ax00.set_xticklabels(['A','B','C','D','E','F'],
                     fontsize=20)
ax00.set_yticklabels(['0.0','0.2','0.4','0.6','0.8','1.0'],
                     fontsize=18)
ax00.set_ylabel(r'$W_{A}^{out}$', fontsize=30,
                rotation='horizontal', labelpad=35)
ax00.set_xlabel('high determinism', fontsize=24, labelpad=1)
ax00.grid(True, color='#999999', linestyle='-',
          linewidth=2.5, alpha=0.5)
ax00.set_axisbelow(True)

plt.axes([0.325, 0.485, 0.125, 0.125])
G01 = nx.DiGraph()
G01.add_edges_from([(0,1),(1,0),(0,2),(3,4),(2,5),(5,0),(5,3),(4,0)])
plt.axis('off')
plt.xlim(-1.3, 1.3)
plt.ylim(-1.2, 1.2)
pos01 = nx.circular_layout(G01)
nx.draw_networkx_nodes(G01, pos00, node_color='white', node_size=500,
                       linewidths=3.0, alpha=0.98).set_edgecolor(col1)
nx.draw_networkx_nodes(G01, pos00, nodelist=[5],
                       node_color='#333333', linewidths=3.0,
                       node_size=700).set_edgecolor(col1)
nx.draw_networkx_edges(G01, pos00, edge_color="#333333",
                       width=2.5, alpha=0.98, arrowsize=14)
nx.draw_networkx_labels(G01, pos00,
                        labels=dict(zip([0,1,2,3,4,5], nodenames)))
nx.draw_networkx_labels(G01, pos00,
                        labels=dict(zip([5], [nodenames[5]])),
                        font_color='w', font_size=18)
ax02.bar(G01.nodes(), W_out(G01)[5]+0.008, color=col1,
         linewidth=2.0, edgecolor='k', width=0.72)
ax02.set_ylim(0.0,1.02)
ax02.set_xticks([0,1,2,3,4,5])
ax02.set_xticklabels(['A','B','C','D','E','F'],
                     fontsize=20)
ax02.set_yticklabels(['0.0','0.2','0.4','0.6','0.8','1.0'],
                     fontsize=18)
ax02.set_ylabel(r'$W_{F}^{out}$', fontsize=30,
                rotation='horizontal', labelpad=35)
ax02.set_xlabel('med. determinism', fontsize=24, labelpad=1)
ax02.grid(True, color='#999999', linestyle='-',
          linewidth=2.5, alpha=0.5)
ax02.set_axisbelow(True)

plt.axes([0.325, 0.215, 0.125, 0.125])
G02 = nx.complete_graph(N)
G02 = nx.to_directed(G02)
plt.axis('off')
plt.xlim(-1.3, 1.3)
plt.ylim(-1.2, 1.2)
pos02 = nx.circular_layout(G02)
nx.draw_networkx_nodes(G02, pos00, node_color='white', node_size=500,
                       linewidths=3.0, alpha=0.98).set_edgecolor(col1)
nx.draw_networkx_nodes(G02, pos00, nodelist=[3],
                       node_color='#333333', linewidths=3.0,
                       node_size=700).set_edgecolor(col1)
nx.draw_networkx_edges(G02, pos00, edge_color="#333333",
                       width=2.5, alpha=0.98, arrowsize=14)
nx.draw_networkx_labels(G02, pos00,
                        labels=dict(zip([0,1,2,3,4,5], nodenames)))
nx.draw_networkx_labels(G02, pos00,
                        labels=dict(zip([3], [nodenames[3]])),
                        font_color='w', font_size=18)
ax04.bar(G02.nodes(), W_out(G02)[3]+0.008, color=col1,
         linewidth=2.0, edgecolor='k', width=0.72)
ax04.set_ylim(0.0,1.02)
ax04.set_xticks([0,1,2,3,4,5])
ax04.set_xticklabels(['A','B','C','D','E','F'], fontsize=20)
ax04.set_yticklabels(['0.0','0.2','0.4','0.6','0.8','1.0'], fontsize=18)
ax04.set_ylabel(r'$W_{D}^{out}$', fontsize=30,
                rotation='horizontal', labelpad=35)
ax04.set_xlabel('low determinism', fontsize=24, labelpad=1)
ax04.grid(True, color='#999999', linestyle='-',
          linewidth=2.5, alpha=0.5)
ax04.set_axisbelow(True)

plt.axes([0.77, 0.755, 0.125, 0.125])
G11 = nx.DiGraph()
G11.add_edges_from([(1,0),(2,0),(3,0),(4,0),(5,0)])
plt.axis('off')
plt.xlim(-1.3, 1.3)
plt.ylim(-1.2, 1.2)
pos11 = nx.circular_layout(G11)
nx.draw_networkx_nodes(G11, pos00, node_color='white', node_size=500,
                       linewidths=3.0, alpha=0.98).set_edgecolor(col2)
nx.draw_networkx_edges(G11, pos00, edge_color="#333333",
                       width=2.5, alpha=0.98, arrowsize=14)
nx.draw_networkx_labels(G11, pos00,
                        labels=dict(zip([0,1,2,3,4,5], nodenames)),
                        font_weight='bold')
ax01.bar(G11.nodes(), W_in(G11)+0.008, color=col2,
         linewidth=2.0, edgecolor='k', width=0.72)
ax01.set_ylim(0.0,1.02)
ax01.set_xticks([0,1,2,3,4,5])
ax01.set_xticklabels(['A','B','C','D','E','F'],
                     fontsize=20)
ax01.set_yticklabels(['0.0','0.2','0.4','0.6','0.8','1.0'],
                     fontsize=18)
ax01.set_ylabel(r'$\langle W_{i}^{out} \rangle$', fontsize=30,
                rotation='horizontal', labelpad=44)
ax01.set_xlabel('high degeneracy', fontsize=24, labelpad=1)
ax01.grid(True, color='#999999', linestyle='-',
          linewidth=2.5, alpha=0.5)
ax01.set_axisbelow(True)

plt.axes([0.77, 0.485, 0.125, 0.125])
G12 = nx.watts_strogatz_graph(6, 2, 0)
G12 = nx.DiGraph()
G12.add_edges_from([(1,0),(3,0),(4,0),(2,0),(1,4),
                    (3,2),(5,3),(0,1),(5,0),(0,5),(2,1)])
plt.axis('off')
plt.xlim(-1.3, 1.3)
plt.ylim(-1.2, 1.2)
pos12 = nx.circular_layout(G12)
nx.draw_networkx_nodes(G12, pos00, node_color='white', node_size=500,
                       linewidths=3.0, alpha=0.98).set_edgecolor(col2)
nx.draw_networkx_edges(G12, pos00, edge_color="#333333",
                       width=2.5, alpha=0.98, arrowsize=14)
nx.draw_networkx_labels(G12, pos00,
                        labels=dict(zip([0,1,2,3,4,5], nodenames)),
                        font_weight='bold')
ax03.bar(G12.nodes(), W_in(G12)+0.008, color=col2,
         linewidth=2.0, edgecolor='k', width=0.72)
ax03.set_ylim(0.0,1.02)
ax03.set_xticks([0,1,2,3,4,5])
ax03.set_xticklabels(['A','B','C','D','E','F'],
                     fontsize=20)
ax03.set_yticklabels(['0.0','0.2','0.4','0.6','0.8','1.0'],
                     fontsize=18)
ax03.set_ylabel(r'$\langle W_{i}^{out} \rangle$', fontsize=30,
                rotation='horizontal', labelpad=44)
ax03.set_xlabel('med. degeneracy', fontsize=24, labelpad=1)
ax03.grid(True, color='#999999', linestyle='-',
          linewidth=2.5, alpha=0.5)
ax03.set_axisbelow(True)

plt.axes([0.77, 0.215, 0.125, 0.125])
G13 = nx.DiGraph()
G13.add_edges_from([(0,1),(1,2),(2,3),(3,4),(4,5),(5,0),
                    (1,0),(2,1),(3,2),(4,3),(5,4),(0,5)])
plt.axis('off')
plt.xlim(-1.3, 1.3)
plt.ylim(-1.2, 1.2)
pos13 = nx.circular_layout(G13)
nx.draw_networkx_nodes(G13, pos00, node_color='white', node_size=500,
                       linewidths=3.0, alpha=0.98).set_edgecolor(col2)
nx.draw_networkx_edges(G13, pos00, edge_color="#333333",
                       width=2.5, alpha=0.98, arrowsize=14)
nx.draw_networkx_labels(G13, pos00,
                        labels=dict(zip([0,1,2,3,4,5], nodenames)),
                        font_weight='bold')
ax05.bar(G13.nodes(), W_in(G13)+0.008, color=col2,
         linewidth=2.0, edgecolor='k', width=0.72)
ax05.set_ylim(0.0,1.02)
ax05.set_xticks([0,1,2,3,4,5])
ax05.set_xticklabels(['A','B','C','D','E','F'],
                     fontsize=20)
ax05.set_yticklabels(['0.0','0.2','0.4','0.6','0.8','1.0'],
                     fontsize=18)
ax05.set_ylabel(r'$\langle W_{i}^{out} \rangle$', fontsize=30,
                rotation='horizontal', labelpad=44)
ax05.set_xlabel('low degeneracy', fontsize=24, labelpad=1)
ax05.grid(True, color='#999999', linestyle='-',
          linewidth=2.5, alpha=0.5)
ax05.set_axisbelow(True)

plt.subplots_adjust(left=None, bottom=None, right=None,
                    top=None, wspace=0.35, hspace=0.27)

if save:
    plt.savefig(
        where_to_save_pngs+\
        "EffectiveInformation_DetermDegen_WoutWin.png", bbox_inches='tight', dpi=425)
    plt.savefig(
        where_to_save_pdfs+\
        "EffectiveInformation_DetermDegen_WoutWin.pdf", bbox_inches='tight')

plt.show()
```

________________

```python
fig, ax = plt.subplots(figsize=(15,15))

plt.plot([-0.5, 10.5], [-0.5, 10.5], marker='',
         linestyle=':', linewidth=3.5,
         color='#333333', label='Y=X', alpha=0.7)

plt.plot([np.linspace(-0.2,1.2,num=10)[3]+0.0075,
          np.linspace(-0.2,1.2,num=10)[9]],
         [np.linspace(-0.2,1.2,num=10)[2]]*2,
         color='w',linewidth=5.3)
plt.plot([np.linspace(-0.2,1.2,num=10)[4]]*2,
         [np.linspace(-0.2,1.2,num=10)[1]+0.0075,
          np.linspace(-0.2,1.2,num=10)[3]-0.0075],
         color='w',linewidth=5.3)
plt.plot([np.linspace(-0.2,1.2,num=10)[5]]*2,
         [np.linspace(-0.2,1.2,num=10)[1]+0.0075,
          np.linspace(-0.2,1.2,num=10)[3]-0.0075],
         color='w',linewidth=5.3)
plt.plot([np.linspace(-0.2,1.2,num=10)[6]]*2,
         [np.linspace(-0.2,1.2,num=10)[1]+0.0075,
          np.linspace(-0.2,1.2,num=10)[3]-0.0075],
         color='w',linewidth=5.3)
plt.plot([np.linspace(-0.2,1.2,num=10)[7]]*2,
         [np.linspace(-0.2,1.2,num=10)[1]+0.0075,
          np.linspace(-0.2,1.2,num=10)[3]-0.0075],
         color='w',linewidth=5.3)
plt.plot([np.linspace(-0.2,1.2,num=10)[8]]*2,
         [np.linspace(-0.2,1.2,num=10)[1]+0.0075,
          np.linspace(-0.2,1.2,num=10)[3]-0.0075],
         color='w',linewidth=5.3)

ns = 175
cols = ["#c77739","#9765ca","#40a0e2","#c95980","#01bb96"]
cols = ["#927134","#9765ca","#40a0e2","#c95980","#01bb96"]
cols = ["#666666","#666666","#666666","#666666","#666666"]
cols = ["#b03e61","#aa592a","#588435","#6172cc","#814192"]
plt.axes([0.123, 0.645, 0.20, 0.20])
G10 = nx.watts_strogatz_graph(30, 2, 0.0)
plt.axis('off')
pos10 = nx.circular_layout(G10)
nx.draw_networkx_nodes(G10, pos10, node_color='white',
                       node_size=ns, linewidths=3.9,
                       alpha=0.98).set_edgecolor(cols[0])
nx.draw_networkx_edges(G10, pos10, edge_color="#333333",
                       width=2.95, alpha=0.98)
ax.text(-.072, 1.085, "Ring Lattice",
        ha='left', rotation=0, wrap=True, size=26)

plt.axes([0.123, 0.123, 0.20, 0.20])
G11 = nx.complete_graph(30)
plt.axis('off')
pos11 = nx.circular_layout(G11)
nx.draw_networkx_nodes(G11, pos11, node_color='white',
                       node_size=ns, linewidths=3.9,
                       alpha=0.98).set_edgecolor(cols[1])
nx.draw_networkx_edges(G11, pos11, edge_color="#333333",
                       width=0.45, alpha=0.98)
ax.text(-.128, 0.21, "Complete Network",
        ha='left', rotation=0, wrap=True, size=26)

plt.axes([0.123, 0.385, 0.22, 0.22])
G12 = nx.erdos_renyi_graph(40, .09)
plt.axis('off')
pos12 = nx.kamada_kawai_layout(G12)
nx.draw_networkx_nodes(G12, pos12, node_color='white',
                       node_size=ns, linewidths=3.9,
                       alpha=0.98).set_edgecolor(cols[2])
nx.draw_networkx_edges(G12, pos12, edge_color="#333333",
                       width=2.95, alpha=0.98)
ax.text(-.069, 0.67, "Erdős-Rényi",
        ha='left', rotation=0, wrap=True, size=26)

plt.axes([0.340, 0.668, 0.23, 0.215])
G13 = nx.barabasi_albert_graph(40, 1)
plt.axis('off')
pos13 = nx.kamada_kawai_layout(G13)
nx.draw_networkx_nodes(G13, pos13, node_color='white',
                       node_size=ns, linewidths=3.9,
                       alpha=0.98).set_edgecolor(cols[3])
nx.draw_networkx_edges(G13, pos13, edge_color="#333333",
                       width=2.95, alpha=0.98)
ax.text(0.29, 0.745, "Barabási-Albert",
        ha='left', rotation=0, wrap=True, size=26)

plt.axes([0.695, 0.675, 0.205, 0.205])
G14 = nx.star_graph(30)
plt.axis('off')
pos14 = nx.spring_layout(G14)
nx.draw_networkx_nodes(G14, pos14, node_color='white',
                       node_size=ns, linewidths=3.9,
                       alpha=0.98).set_edgecolor(cols[4])
nx.draw_networkx_edges(G14, pos14, edge_color="#333333",
                       width=2.95, alpha=0.98)
ax.text(0.86, 0.745, "Star Network",
        ha='left', rotation=0, wrap=True, size=26)

plt.axes([0.1, 0.1, 0.2, 0.2])
plt.axis('off')
string10 = r'$\rm EI = determinism - degeneracy$'
string21 = r'$\rm determinism$'
string22 = r'$= \log_2(N) - $'+\
           r'$\langle H(W_i^{out}) \rangle$'
string31 = r'$\rm degeneracy$'
string32 = r'$\hspace{0.6cm}= \log_2(N) - $'+\
           r'$H(\langle W_{i}^{out} \rangle)$'

ax.text(0.305, 0.1980, string10, ha='left',
        rotation=0, wrap=True, size=42)
ax.text(0.315, 0.1010, string21, ha='left',
        rotation=0, wrap=True, size=36)
ax.text(0.593, 0.0850, string22, ha='left',
        rotation=0, wrap=True, size=36)
ax.text(0.345, -.0070, string31, ha='left',
        rotation=0, wrap=True, size=36)
ax.text(0.593, -.0240, string32, ha='left',
        rotation=0, wrap=True, size=36)
ax.text(0.305, 0.1980, string10, ha='left',
        rotation=0, wrap=True, size=42)
ax.text(0.315, 0.1010, string21, ha='left',
        rotation=0, wrap=True, size=36)
ax.text(0.593, 0.0850, string22, ha='left',
        rotation=0, wrap=True, size=36)
ax.text(0.345, -.0070, string31, ha='left',
        rotation=0, wrap=True, size=36)
ax.text(0.593, -.0240, string32, ha='left',
        rotation=0, wrap=True, size=36)

ax.grid(True, linestyle='-', color='#999999', linewidth=3.0, alpha=0.4)
ax.set_xlabel("Degeneracy", size=37, labelpad=-40)
ax.set_ylabel("Determinism", size=37, labelpad=-40)
ax.set_xticklabels(
    labels=['','minimally\n degenerate','','','',
            '','','','maximally\n degenerate',''],size=24)
ax.set_xticks(ticks=np.linspace(-0.2,1.2,num=10))
ax.set_yticklabels(
    labels=['','','minimally  \n deterministic  ','','',
            '','','','   maximally\n   deterministic'],
            size=24, rotation='vertical')
ax.set_yticks(ticks=np.linspace(-0.2,1.2,num=10))
ax.set_xlim(-0.135,1.135)
ax.set_ylim(-0.135,1.135)

if save:
    plt.savefig(
        where_to_save_pngs+\
        "EffectiveInformation_DetermDegen_Canonical.png",
        bbox_inches='tight', dpi=425)
    plt.savefig(
        where_to_save_pdfs+\
        "EffectiveInformation_DetermDegen_Canonical.pdf",
        bbox_inches='tight')

plt.show()
```

## End of Chapter 03. In [Chapter 04](https://nbviewer.jupyter.org/github/jkbren/einet/blob/master/code/Chapter%2004%20-%20Effective%20Information%20in%20Real%20Networks.ipynb) we'll look at $EI$ in real networks.
_______________

code/Chapter 03 - Determinism and Degeneracy.ipynb


# Inferential Statistics Ia - Frequentism

## Learning objectives

Welcome to the first Frequentist inference mini-project! Over the course of working on this mini-project and the next frequentist mini-project, you'll learn the fundamental concepts associated with frequentist inference. The following list includes the topics you will become familiar with as you work through these two mini-projects:
* the _z_-statistic
* the _t_-statistic
* the difference and relationship between the two
* the Central Limit Theorem, including its assumptions and consequences
* how to estimate the population mean and standard deviation from a sample
* the concept of a sampling distribution of a test statistic, particularly for the mean
* how to combine these concepts to calculate a confidence interval

## Prerequisites

For working through this notebook, you are expected to have a very basic understanding of:
* what a random variable is
* what a probability density function (pdf) is
* what the cumulative density function is
* a high-level sense of what the Normal distribution

If these concepts are new to you, please take a few moments to Google these topics in order to get a sense of what they are and how you might use them.

While it's great if you have previous knowledge about sampling distributions, this assignment will introduce the concept and set you up to practice working using sampling distributions. This notebook was designed to bridge the gap between having a basic understanding of probability and random variables and being able to apply these concepts in Python. The second frequentist inference mini-project focuses on a real-world application of this type of inference to give you further practice using these concepts.

For this notebook, we will use data sampled from a known normal distribution. This allows us to compare our results with theoretical expectations.

## I An introduction to sampling from the Normal distribution

First, let's explore the ways we can generate the Normal distribution. While there's a fair amount of interest in [sklearn](https://scikit-learn.org/stable/) within the machine learning community, you're likely to have heard of [scipy](https://docs.scipy.org/doc/scipy-0.15.1/reference/index.html) if you're coming from the sciences. For this assignment, you'll use [scipy.stats](https://docs.scipy.org/doc/scipy-0.15.1/reference/tutorial/stats.html) to complete your work.

```python
from scipy.stats import norm
from scipy.stats import t
import numpy as np
import pandas as pd
from numpy.random import seed
import matplotlib.pyplot as plt
```

__Q:__ Call up the documentation for the `norm` function imported above. What is the second listed method?

```python
#Call documentation
#print(help(norm))
```

#### __A:__ Second Listed Method
1. norm_gen (First)
2. <b> scipy.stats._distn_infrastructure.rv_continuous (Answer)</b>

__Q:__ Use the method that generates random variates to draw five samples from the standard normal distribution.

__A:__

```python
seed(47)
# draw five samples here
sample_5 = np.random.random(5)
print('Sample:',sample_5)

# Say population size is 10,000
population_10000 = np.random.random(10000)
print('Population:', population_10000)

#Is sample mean equal to the population mean?
```

    Sample: [0.11348847 0.97448309 0.72873463 0.35146781 0.70760514]
    Population: [0.7996046  0.64556185 0.41459961 ... 0.34463857 0.19334307 0.39105519]

__Q:__ What is the mean of this sample? Is it exactly equal to the value you expected? Hint: the sample was drawn from the standard normal distribution.

__A:__ Sample Mean is: 0.575155828796599, As expected, sample mean is not the same as the population mean.

```python
# Calculate and print the mean here, hint: use np.mean()
sample_mean = np.mean(sample_5)
print('Sample Mean:', sample_mean)

population_mean = np.mean(population_10000)
print('population_mean:', population_mean)
```

    Sample Mean: 0.575155828796599
    population_mean: 0.4999494037262384

__Q:__ What is the standard deviation of these numbers? Calculate this manually here as $\sqrt{\frac{\sum_i(x_i - \bar{x})^2}{n}}$. Hint: np.sqrt() and np.sum() will be useful here and remember that numpy supports [broadcasting](https://docs.scipy.org/doc/numpy/user/basics.broadcasting.html).

__A:__ Sample Standard Deviation is 0.3045536451712141

```python
#Standard Deviation:
std_sample = np.sqrt(np.sum((sample_5-sample_mean)**2)/(len(sample_5)))
print('Sample Standard Deviation:', std_sample)
```

    Sample Standard Deviation: 0.3045536451712141

Here we have calculated the actual standard deviation of a small (size 5) data set. But in this case, this small data set is actually a sample from our larger (infinite) population. In this case, the population is infinite because we could keep drawing our normal random variates until our computers die. In general, the sample mean we calculate will not be equal to the population mean (as we saw above). A consequence of this is that the sum of squares of the deviations from the _population_ mean will be bigger than the sum of squares of the deviations from the _sample_ mean. In other words, the sum of squares of the deviations from the _sample_ mean is too small to give an unbiased estimate of the _population_ variance. An example of this effect is given [here](https://en.wikipedia.org/wiki/Bessel%27s_correction#Source_of_bias). Scaling our estimate of the variance by the factor $n/(n-1)$ gives an unbiased estimator of the population variance. This factor is known as [Bessel's correction](https://en.wikipedia.org/wiki/Bessel%27s_correction). The consequence of this is that the $n$ in the denominator is replaced by $n-1$.

__Q:__ If all we had to go on was our five samples, what would be our best estimate of the population standard deviation? Use Bessel's correction ($n-1$ in the denominator), thus $\sqrt{\frac{\sum_i(x_i - \bar{x})^2}{n-1}}$.

__A:__ Population standard deviation: 0.34050132669909267

```python
#Best estimate of the population standard deviation:
std_bessel = np.sqrt(np.sum((sample_5-sample_mean)**2)/(len(sample_5)-1))
print('Population standard deviation:', std_bessel)
```

    Population standard deviation: 0.34050132669909267

__Q:__ Now use numpy's std function to calculate the standard deviation of our random samples. Which of the above standard deviations did it return?

__A:__ Returned same as Sample Standard Deviation: 0.3045536451712141

```python
#Standard Deviation using Numpy function:
std_numpy= np.std(sample_5)
print('Standard Deviation Using Numpy Function:',std_numpy)
```

    Standard Deviation Using Numpy Function: 0.3045536451712141

__Q:__ Consult the documentation for np.std() to see how to apply the correction for estimating the population parameter and verify this produces the expected result.

__A:__ Yes, Standard Deviation of Population using Bessel Correction Function also resulted in the same outcome.

```python
#Call np.std documentation
#help(np.std)
```

```python
#Calculate Standard Deviation of Population using Bessel Correction function
std_bessel_function = np.std(sample_5, ddof=1) # ddof = Degree of Freedom which is equal to 1
print('Standard Deviation of Population using Function:', std_bessel_function)
```

    Standard Deviation of Population using Function: 0.34050132669909267

### Summary of section

In this section, you've been introduced to the scipy.stats package and used it to draw a small sample from the standard normal distribution. You've calculated the average (the mean) of this sample and seen that this is not exactly equal to the expected population parameter (which we know because we're generating the random variates from a specific, known distribution). You've been introduced to two ways of calculating the standard deviation; one uses $n$ in the denominator and the other uses $n-1$ (Bessel's correction). You've also seen which of these calculations np.std() performs by default and how to get it to generate the other.

You use $n$ as the denominator if you want to calculate the standard deviation of a sequence of numbers. You use $n-1$ if you are using this sequence of numbers to estimate the population parameter. This brings us to some terminology that can be a little confusing.

The population parameter is traditionally written as $\sigma$ and the sample statistic as $s$. Rather unhelpfully, $s$ is also called the sample standard deviation (using $n-1$) whereas the standard deviation of the sample uses $n$. That's right, we have the sample standard deviation and the standard deviation of the sample and they're not the same thing!

The sample standard deviation
\begin{equation}
s = \sqrt{\frac{\sum_i(x_i - \bar{x})^2}{n-1}} \approx \sigma,
\end{equation}
is our best (unbiased) estimate of the population parameter ($\sigma$).

If your data set _is_ your entire population, you simply want to calculate the population parameter, $\sigma$, via
\begin{equation}
\sigma = \sqrt{\frac{\sum_i(x_i - \bar{x})^2}{n}}
\end{equation}
as you have complete, full knowledge of your population. In other words, your sample _is_ your population. It's worth noting at this point if your sample is your population then you know absolutely everything about your population, there are no probabilities really to calculate and no inference to be done.

If, however, you have sampled _from_ your population, you only have partial knowledge of the state of your population and the standard deviation of your sample is not an unbiased estimate of the standard deviation of the population, in which case you seek to estimate that population parameter via the sample standard deviation, which uses the $n-1$ denominator.

You're now firmly in frequentist theory territory. Great work so far! Now let's dive deeper.

## II Sampling distributions

So far we've been dealing with the concept of taking a sample from a population to infer the population parameters. One statistic we calculated for a sample was the mean. As our samples will be expected to vary from one draw to another, so will our sample statistics. If we were to perform repeat draws of size $n$ and calculate the mean of each, we would expect to obtain a distribution of values. This is the sampling distribution of the mean. The Central Limit Theorem (CLT) tells us that such a distribution will approach a normal distribution as $n$ increases. For the sampling distribution of the mean, the standard deviation of this distribution is given by

\begin{equation}
\sigma_{mean} = \frac{\sigma}{\sqrt n}
\end{equation}

where $\sigma_{mean}$ is the standard deviation of the sampling distribution of the mean and $\sigma$ is the standard deviation of the population (the population parameter).

This is important because typically we are dealing with samples from populations and all we know about the population is what we see in the sample. From this sample, we want to make inferences about the population. We may do this, for example, by looking at the histogram of the values and by calculating the mean and standard deviation (as estimates of the population parameters), and so we are intrinsically interested in how these quantities vary across samples. In other words, now that we've taken one sample of size $n$ and made some claims about the general population, what if we were to take another sample of size $n$? Would we get the same result? Would we make the same claims about the general population? This brings us to a fundamental question: _when we make some inference about a population based on our sample, how confident can we be that we've got it 'right'?_

Let's give our normal distribution a little flavor. Also, for didactic purposes, the standard normal distribution, with its variance equal to its standard deviation of one, would not be a great illustration of a key point. Let us imagine we live in a town of 50000 people and we know the height of everyone in this town. We will have 50000 numbers that tell us everything about our population. We'll simulate these numbers now and put ourselves in one particular town, called 'town 47', where the population mean height is 172 cm and population standard deviation is 5 cm.

```python
seed(47)
pop_heights = norm.rvs(172, 5, size=50000)
```

```python
_ = plt.hist(pop_heights, bins=30)
_ = plt.xlabel('height (cm)')
_ = plt.ylabel('number of people')
_ = plt.title('Distribution of heights in entire town population')
_ = plt.axvline(172, color='r')
_ = plt.axvline(172+5, color='r', linestyle='--')
_ = plt.axvline(172-5, color='r', linestyle='--')
_ = plt.axvline(172+10, color='r', linestyle='-.')
_ = plt.axvline(172-10, color='r', linestyle='-.')
```

Now, 50000 people is rather a lot to chase after with a tape measure. If all you want to know is the average height of the townsfolk, then can you just go out and measure a sample to get a pretty good estimate of the average height?

```python
def townsfolk_sampler(n):
    return np.random.choice(pop_heights, n)
```

Let's say you go out one day and randomly sample 10 people to measure.

```python
seed(47)
daily_sample1 = townsfolk_sampler(10)
```

```python
_ = plt.hist(daily_sample1, bins=10)
_ = plt.xlabel('height (cm)')
_ = plt.ylabel('number of people')
_ = plt.title('Distribution of heights in sample size 10')
```

The sample distribution doesn't look much like what we know (but wouldn't know in real-life) the population distribution looks like. What do we get for the mean?

```python
np.mean(daily_sample1)
```

    173.47911444163503

And if we went out and repeated this experiment?

```python
daily_sample2 = townsfolk_sampler(10)
```

```python
np.mean(daily_sample2)
```

    172.21023667046654

__Q:__ Simulate performing this random trial every day for a year, calculating the mean of each daily sample of 10, and plot the resultant sampling distribution of the mean.

__A:__ Resulting Sampling Distribution of Mean Heights

```python
seed(47)
#Create empty column for daily sample:
daily_sample= []

#Create empty Column for the mean of daily sample:
mean = []

#Iterate daily over 365 days:
for i in range(365):
    daily_sample.append(townsfolk_sampler(10))
    mean.append(np.mean(daily_sample[i]))

#Plot histogram:
_ =  plt.hist(mean, bins = 20, histtype = 'step')
_ = plt.xlabel('height (cm)')
_ = plt.ylabel('number of people')
_ = plt.title('Distribution of mean heights over 365 days' '\n' 'Sample Size =10')
```

The above is the distribution of the means of samples of size 10 taken from our population. The Central Limit Theorem tells us the expected mean of this distribution will be equal to the population mean, and standard deviation will be $\sigma / \sqrt n$, which, in this case, should be approximately 1.58.

__Q:__ Verify the above results from the CLT.

__A:__ Standard Deviation of the Sampling Distribution of the mean is equal to 1.58

```python
#Central Theorem: Sampling distribution of the mean is equal to the population mean:
#Mean of the Sampling Distribution of the mean
print(np.mean(mean))

#Standard Deviation of the Sampling Distribution of the mean.
print(np.std(mean))

```

    171.8660049358649
    1.5756704135286475

Remember, in this instance, we knew our population parameters, that the average height really is 172 cm and the standard deviation is 5 cm, and we see some of our daily estimates of the population mean were as low as around 168 and some as high as 176.

__Q:__ Repeat the above year's worth of samples but for a sample size of 50 (perhaps you had a bigger budget for conducting surveys that year!) Would you expect your distribution of sample means to be wider (more variable) or narrower (more consistent)? Compare your resultant summary statistics to those predicted by the CLT.

__A:__

```python
seed(47)
# calculate daily means from the larger sample size here

#Create empty column for daily sample:
daily_mean_2 = []

#Create empty Column for the mean of daily sample:
mean_2 = []

##Iterate daily over 365 days:
for j in range(365):
    daily_mean_2.append(townsfolk_sampler(100)) #larger sampler size (100 instead of 10)
    mean_2.append(np.mean(daily_mean_2[j]))

#Central Theorem: Sampling distribution of the mean is equal to the population mean:
#Mean of the Sampling Distribution of the mean
print('Sampling Distribution Mean:',np.mean(mean_2))

#Standard Deviation of the Sampling Distribution of the mean.
print('Standard Deviation in Sampling Distribution Mean:',np.std(mean_2))

```

    Sampling Distribution Mean: 171.98525667397283
    Standard Deviation in Sampling Distribution Mean: 0.4797679855953269

```python
#Plot Histogram of Sampling Distribution Mean :
_ = plt.hist(mean_2, bins= 20, histtype= 'step', color = 'red')
_ = plt.xlabel('height (cm)')
_ = plt.ylabel('number of people')
_ = plt.title('Distribution of mean heights over 365 days' '\n' 'Sample Size =100')
_ = plt.axvline(np.mean(mean_2), label= 'Mean')
_ = plt.axvline(np.mean(mean_2)-np.std(mean_2), linestyle = '-.', color = 'black', label='Mean-Std')
_ = plt.axvline(np.mean(mean_2) + np.std(mean_2),  linestyle = '-.', color = 'black', label='Mean+Std')
_ = plt.legend(loc = 'best')
```

What we've seen so far, then, is that we can estimate population parameters from a sample from the population, and that samples have their own distributions. Furthermore, the larger the sample size, the narrower are those sampling distributions.

### III Normally testing times!

All of the above is well and good. We've been sampling from a population we know is normally distributed, we've come to understand when to use $n$ and when to use $n-1$ in the denominator to calculate the spread of a distribution, and we've  seen the Central Limit Theorem in action for a sampling distribution. All seems very well behaved in Frequentist land. But, well, why should we really care?

Remember, we rarely (if ever) actually know our population parameters but you still have to estimate them somehow. If we want to make inferences such as "is this observation unusual?" or "has my population mean changed?" then you need to have some idea of what the underlying distribution is so you can calculate relevant probabilities. In frequentist inference, you use the formulas above to deduce these population parameters. Take a moment in the next part of this assignment to refresh your understanding of how these probabilities work.

Recall some basic properties of the standard Normal distribution, such as about 68% of observations being within plus or minus 1 standard deviation of the mean.

__Q:__ Using this fact, calculate the probability of observing the value 1 or less in a single observation from the standard normal distribution. Hint: you may find it helpful to sketch the standard normal distribution (the familiar bell shape) and mark the number of standard deviations from the mean on the x-axis and shade the regions of the curve that contain certain percentages of the population.

__A:__ Probability Distribution for values of x when standard deviation deviation form the mean is +,-1 is equal to 65%(area under the curve) which should be approximated to 68% to account for the curve area.

```python
#Generate an array of equally spaced datapoints:
x= np.linspace(start= -3, stop =3, num = 1000)

#Transform datapoints using probability distribution function:
pdf_x= (norm.pdf(x))

#Plot Pdf_x:
_ = plt.plot(x, pdf_x)
_ = plt.xlabel('X-values')
_ = plt.ylabel('Probabilty of X-values')
_ = plt.title('Plot of a Probability Distribution Function')

#Mark STD lines:
#_ = plt.axvline(np.mean(x), color = 'black')
_ = plt.axvline(x = -1, color = 'red')
_ = plt.axvline(x = 1, color = 'red')

#Area under the curve:
traingle = (2 * (0.40-0.25)) / 2
rectangle = 0.25 *2

area =  traingle + rectangle + 0.03     #3% error rate due to curve area.
print('% Probability:' ,area * 100)
```

Calculating this probability involved calculating the area under the pdf from the value of 1 and below. To put it another way, we need to integrate the pdf. We could just add together the known areas of chunks (from -Inf to 0 and then 0 to $+\sigma$ in the example above. One way to do this is using look up tables (literally). Fortunately, scipy has this functionality built in with the cdf() function.

__Q:__ Use the cdf() function to answer the question above again and verify you get the same answer.

__A:__ Hence, CDF resulted in 68% probability for PDF when standard deviation is +,-1

```python
#Calculate CDF when std = -1 and std = +1:

cdf_minus_1 = norm.cdf(x = -1)

print(std_minus_1)

cdf_plus_1 = norm.cdf(x= 1)
print(std_plus_1)

#Probabiity when std = (+,-) 1 is 68%, which is the area under the curve
probability_std_1 = cdf_minus_1 - cdf_plus_1

print(probability_std_1)
```

    0.15865525393145707
    0.8413447460685429
    -0.6826894921370859

__Q:__ Using our knowledge of the population parameters for our townsfolk's heights, what is the probability of selecting one person at random and their height being 177 cm or less? Calculate this using both of the approaches given above.

__A:__ cdf_177 matches with cdf_plus_1 at 84%, which mean there is a 84% chance.

```python
#Height x = 177
cdf_177 = norm.cdf(x = 177, loc = 172, scale = 5)
print(cdf_177)

```

    0.8413447460685429

__Q:__ Turning this question around. Let's say we randomly pick one person and measure their height and find they are 2.00 m tall? How surprised should we be at this result, given what we know about the population distribution? In other words, how likely would it be to obtain a value at least as extreme as this? Express this as a probability.

__A:__ 92%

```python
#Height x 2cm taller = 179
cdf_179 = norm.cdf(x =179, loc = 172, scale = 5)
print(cdf_179)
```

    0.9192433407662289

We could calculate this probability by virtue of knowing the population parameters. We were then able to use the known properties of the relevant normal distribution to calculate the probability of observing a value at least as extreme as our test value. We have essentially just performed a z-test (albeit without having prespecified a threshold for our "level of surprise")!

We're about to come to a pinch, though here. We've said a couple of times that we rarely, if ever, know the true population parameters; we have to estimate them from our sample and we cannot even begin to estimate the standard deviation from a single observation. This is very true and usually we have sample sizes larger than one. This means we can calculate the mean of the sample as our best estimate of the population mean and the standard deviation as our best estimate of the population standard deviation. In other words, we are now coming to deal with the sampling distributions we mentioned above as we are generally concerned with the properties of the sample means we obtain.

Above, we highlighted one result from the CLT, whereby the sampling distribution (of the mean) becomes narrower and narrower with the square root of the sample size. We remind ourselves that another result from the CLT is that _even if the underlying population distribution is not normal, the sampling distribution will tend to become normal with sufficiently large sample size_. This is the key driver for us 'requiring' a certain sample size, for example you may frequently see a minimum sample size of 30 stated in many places. In reality this is simply a rule of thumb; if the underlying distribution is approximately normal then your sampling distribution will already be pretty normal, but if the underlying distribution is heavily skewed then you'd want to increase your sample size.

__Q:__ Let's now start from the position of knowing nothing about the heights of people in our town.
* Use our favorite random seed of 47, to randomly sample the heights of 50 townsfolk
* Estimate the population mean using np.mean
* Estimate the population standard deviation using np.std (remember which denominator to use!)
* Calculate the (95%) [margin of error](https://www.statisticshowto.datasciencecentral.com/probability-and-statistics/hypothesis-testing/margin-of-error/#WhatMofE) (use the exact critial z value to 2 decimal places - [look this up](https://www.statisticshowto.datasciencecentral.com/probability-and-statistics/find-critical-values/) or use norm.ppf())
* Calculate the 95% Confidence Interval of the mean
* Does this interval include the true population mean?

__A:__ Yes, confidence interval of 95% includes population mean.

```python
seed(47)
# take your sample now

#Random Sample of 50 townfolks:
sample_height=townsfolk_sampler(50) #sample size of 50
print(sample_height)
```

    [179.04949004 168.54189384 181.71974048 176.56433733 165.91968052
     170.65629646 179.37664509 166.40103988 177.15105034 169.41097044
     169.96353026 175.15960886 177.08203296 175.57499516 174.54290109
     172.47967099 168.61822612 174.73274884 170.88926244 178.27468993
     171.52874629 173.48970606 172.0752456  172.29707499 171.58202648
     169.84480352 178.67104246 174.28346436 174.29645483 167.42082214
     168.57939161 169.54392578 167.58621293 176.8657094  164.25755366
     177.38929575 171.97964488 164.94136054 177.90675289 175.2758845
     176.64336688 169.68013675 175.41851966 171.9408919  179.69843965
     169.93651872 169.3320981  172.17010876 171.18783138 171.14370134]

```python
#Mean:
mean_height = np.mean(sample_height)
print(mean_height)
```

    172.7815108576788

```python
#Standard Deviation:
std_height = np.std(sample_height,ddof = 1)
print(std_height)
```

    4.195424364433547

```python
# what is the margin of error when confidentce interval is 95%
#Margin of Error = z-Critical Value * Standard Deviation.
margin_error_z= norm.ppf(q = 0.975)*std_height/(np.sqrt(len(sample_height)))  #here q = 1- [(1-0.95)/2] = 0.975
print('Margin of Error:', margin_error_z) # So, with 95% confidence we can say that
```

    Margin of Error: 1.1628909342876539

```python
#Hence mean using 95% confidence interval:
print('Mean Confidence Interval Using Z:',[mean_height - margin_error_z, mean_height + margin_error_z]) # this range includes the population mean
```

    Mean Confidence Interval Using Z: [171.61861992339115, 173.94440179196644]

__Q:__ Above we calculated the confidence interval using the critical z value. What is the problem with this? What requirement, or requirements, are we (strictly) failing?

__A:__ Population parameters are estimated from samples. Hence it contains standard of error as we saw above.

__Q:__ Calculate the 95% confidence interval for the mean using the _t_ distribution. Is this wider or narrower than that based on the normal distribution above? If you're unsure, you may find this [resource](https://www.statisticshowto.datasciencecentral.com/probability-and-statistics/confidence-interval/) useful. For calculating the critical value, remember how you could calculate this for the normal distribution using norm.ppf().

__A:__ Mean Confidence Interval Using t-Distribution is slightly wider than the previous confidence interval.

```python
#Calculate margin of error using t-distrbution:
#Margin of Error = t-critical value * Standard Deviation

margin_error_t=t.ppf(0.975, df = 49) * std_height/(np.sqrt((len(sample_height)-1))) #here df = 49 is sample size with degree of freedom 1
print(margin_error_t)
```

    1.204431557276898

```python
print('Mean Confidence Interval Using t-Distribution:',[mean_height-margin_error_t, mean_height+margin_error_t])
```

    Mean Confidence Interval Using t-Distribution: [171.5770793004019, 173.9859424149557]

This is slightly wider than the previous confidence interval. This reflects the greater uncertainty given that we are estimating population parameters from a sample.

## Learning outcomes

Having completed this project notebook, you now have hands-on experience:
* sampling and calculating probabilities from a normal distribution
* the correct way to estimate the standard deviation of a population (the population parameter) from a sample
* what a sampling distribution is and how the Central Limit Theorem applies
* how to calculate critical values and confidence intervals


# Introduction

In this tutorial, we will learn how to implement different policies and temporal difference learning algorithms, as well as a hybrid (model building) algorithm, then compare their performance to the dynamic programming algorithms we implemented in the first tutorial.

# Notebook setup

## Instructions

- Import numpy, scipy and matplotlib
- Configure inline plots

```python
% matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from pylab import *
```

# Tutorial

## Learning algorithms and policies

__Learning algorithms__:

*Sarsa (on-policy)*

\begin{align}
Q(s_t,a_t) \leftarrow Q(s_t,a_t) + \alpha \big(r_{t+1} + \gamma Q(s_{t+1},a_{t+1}) - Q(s_t,a_t)\big)
\end{align}

with temporal discount rate $\gamma$ and learning rate $\alpha$.

*Q-learning (off-policy)*

\begin{align}
Q(s_t,a_t) \leftarrow Q(s_t,a_t) + \alpha \big(r_{t+1} + \gamma\max_\limits{a} Q(s_{t+1},a_{t+1}) - Q(s_t,a_t)\big)
\end{align}

__Policies__:

*Epsilon-greedy*

\begin{align}
P(a_t|s_t) = \epsilon \frac{1}{N_a}  + (1-\epsilon)1[a_t =\max_\limits{a}Q(a_t,s_t)]
\end{align}

*Softmax*

\begin{align}
P(a_t|s_t) = \frac{\exp(Q(a_t,s_t)/\tau)}{\Sigma_{i=1}^n \exp(Q(i)/\tau)}
\end{align}

Please familiarize yourself with the code below, as it will help your write your own code to solve the exercises.

```python
# Import definitions of the environments.
import RL_worlds as worlds

# Import helper functions for plotting.
from plot_util import *
```

```python
def init_state(params):
    """
    Initialize the state at the beginning of an episode.
    Args:
        params: a dictionary containing the default parameters.
    Returns:
        an integer corresponding to the initial state.
    """
    if params['environment'].name == 'windy_cliff_grid':
        return 0
    elif params['environment'].name == 'n_armed_bandit':
        return 0
    elif params['environment'].name == 'cheese_world':
        return 0
    elif params['environment'].name == 'cliff_world':
        return 0
    elif params['environment'].name == 'quentins_world':
        return 54

def update_state(state, action, params):
    """
    State transition based on world, action and current state.
    Args:
        state: integer corresponding to the current state.
        action: integer corresponding to the action taken.
        params: a dictionary containing the default parameters.
    Returns:
        an integer corresponding to the next state;
        an integer corresponding to the reward received.
    """
    next_state, reward = params['environment'].get_outcome(state, action)
    return next_state, reward

def call_policy(state, value, params):
    """
    Call a policy to choose actions, given current state and value function.
    Args:
        state: integer corresponding to the current state.
        value: a matrix indexed by state and action.
        params: a dictionary containing the default parameters.
    Returns:
        an integer corresponding action chosen according to the policy.
    """
    # multiple options for policy
    if params['policy'] == 'epsilon_greedy':
        return epsilon_greedy(state, value, params)
    elif params['policy'] == 'softmax':
        return softmax(state, value, params)
    else: # random policy (if policy not recognized, choose randomly)
        return randint(params['environment'].n_actions)

def update_value(prev_state, action, reward, state, value, params):
    """
    Update the value function.
    Args:
        prev_state: an integer corresponding to the previous state.
        action: an integer correspoding to action taken.
        reward: a float corresponding to the reward received.
        state: an integer corresponding to the current state;
          should be None if the episode ended.
        value: a matrix indexed by state and action.
        params: a dictionary containing the default parameters.
    Returns:
        the updated value function (matrix indexed by state and action).
    """
    if params['learning_rule'] == 'q_learning':
        # off policy learning
        return q_learning(prev_state, action, reward, state, value, params)
    elif params['learning_rule'] == 'sarsa':
        # on policy learning
        return sarsa(prev_state, action, reward, state, value, params)
    else:
        print('Learning rule not recognized')

def default_params(environment):
    """
    Define the default parameters.
    Args:
        environment: an object corresponding to the environment.
    Returns:
        a dictionary containing the default parameters, where the keys
            are strings (parameter names).
    """
    params = dict()
    params['environment'] = environment

    params['alpha'] = 0.1  # learning rate
    params['beta'] = 10  # inverse temperature
    params['policy'] = 'epsilon_greedy'
    params['epsilon'] = 0.05  # epsilon-greedy policy
    params['learning_rule'] = 'q_learning'
    params['epsilon_decay'] = 0.9

    if environment.name == 'windy_cliff_grid':
        params['gamma'] = 0.6  # temporal discount factor
    elif environment.name == 'n_armed_bandit':
        params['gamma'] = 0.9  # temporal discount factor
    elif environment.name == 'cliff_world':
        params['gamma'] = 1.0  # no discounting
    elif environment.name == 'cheese_world':
        params['gamma'] = 0.5  # temporal discount factor
    elif environment.name == 'quentins_world':
        params['gamma'] = 0.9  # temporal discount factor

    return params
```

## Exercise 1: Decision Policies

1. Write an epsilon-greedy policy function.

2. Write a softmax policy function.

Tip: both functions should take the current state, the value function and default parameters as input and return an action.

```python

```

## Exercise 2: Learning Algorithms

1. Write a Q-learning (off-policy) algorithm.
2. Modify your Q-learning algorithm to obtain a Sarsa (on-policy) algorithm.

Tip: both functions should take the previous state, action taken, reward received, value function, current state and default parameters and return the updated value function.

```python

```

## Exercise 3

1. Write code that allows you to select a world, a learning algorithm and a decision policy. Run 500 episodes (visits to the world) with learning across episodes. Make sure to set a maximum number of steps per episode (e.g. 1000). Use the functions provided in the plot_util module to:
    - Plot the value associated with each action at each state;
    - Plot the action corresponding to the maximum value at each state;
    - Plot the maximum value in each state;
    - Plot the total reward obtained in each episode.
2. Experiment with different values for the parameters:
    - Pick a range for the learning rate $\alpha$ and look at how the results change.
    - Pick a range for the inverse temperature $\beta$ (using a softmax policy) and look at how the results change.
    - Pick a range for $\epsilon$ (using an $\epsilon$-greedy policy) and look at how the results change.
    - Pick a range for the temporal discount factor $\gamma$ and look at how the results change.
3. Explore the cliff world with an $\epsilon$-greedy policy (try $\epsilon$=0.1) comparing the performance of Q-learning (off-policy) and Sarsa (on-policy). What differences do you notice? What do these differences tell us about on- and off-policy learning?
4. Compare your results using Q-learning and Sarsa with those obtained in the previous tutorial. What do you notice about the differences in performance between dynamic programming and TD learning algorithms? What are some of the advantages and disadvantages of each approach?

To make sure that your algorithms have been implemented correctly, compare your results to the ones shown below.

Cliff world using Q-learning and an $\epsilon$-greedy policy with $\epsilon$=0.1 and $\alpha$=0.3:

Quentin's world using Sarsa and a softmax policy with $\beta$=10 and $\alpha$=0.4:

```python

```

## Exercise 4: Dyna-Q

1. Implement the Dyna-Q algorithm for a deterministic environment.

Tip: the function should take the default parameters, an integer k for the number random updates, and a threshold for the stopping criterium as input and return a value function and deterministic model of the environment.

```python

```

## Exercise 5

1. Write code that allows you to test the performance of Dyna-Q for a selected world and a selected learning rule. Run 500 episodes (visits to the world) with learning across episodes. Make sure to set a maximum number of steps per episode (e.g. 1000). Use the functions provided in the plot_util module to:
    - Plot the value associated with each action at each state;
    - Plot the action corresponding to the maximum value at each state;
    - Plot the maximum value in each state;
    - Plot the total reward obtained in each episode.
2. Experiment with different values for the parameters:
    - Pick a range for the learning rate $\alpha$ and look at how the results change.
    - Pick a range for the temporal discount factor $\gamma$ and look at how the results change.
    - Pick a range for k (number of random updates performed in Dyna-Q) and look at how the results change.
3. Compare these results with those obtained for dynamic programming in the previous tutorial, as well as the TD algorithms from this tutorial.

To make sure that your algorithm has been implemented correctly, compare your results to the ones shown below.

Windy cliff grid using Dyna-Q and an $\epsilon$-greedy policy with $\epsilon$=0.05, $\alpha$=0.5 and $\gamma$=0.8:

```python

```

## Exercise 6
1. We will now compare Q-learning and Dyna-Q to see how each of these algorithms responds to a change in the structure of the environment. We will use a modified version of the windy cliff grid, windy_cliff_grid_2, which is the same as the original except for the location of the doors leading to the two windy rooms.

    - For each algorithm, run 500 episodes in the original world. Then, switch to the modified world and run 500 more episodes.
    - You can use an $\epsilon$-greedy policy with $\epsilon$=0.05 throughout.
    - After both sets of episodes, plot the value of each action at each state, the action of maximum value at each state, the maximum value at each state and the total reward obtained in each episode (plot the accumulated rewards only for the 500 episodes in the modified world).
    - What do you notice about the difference in each algorithm's performance once the environment has changed?

```python

```


## SAT terms on material boundary interfaces

```python
from sympy import *
from sympy import oo
```

v1/2  : velocity at + and - side.

p1/2  : pressure at + and - side.

Z1/2  : impedance at + and - side.

sat1/2: SAT coefficients at + and - side.

```python
A, v1, v2, p1, p2, Z1, Z2, sat1, sat2, p_c = symbols('A v1 v2 p1 p2 Z1 Z2 sat1 sat2 pc')
```

```python
init_printing(use_unicode=True)
```

```python
vhat = (p2 - p1 + Z1*v1 + Z2*v2)/(Z1 + Z2)
phat = (Z1*p2 + Z2*p1 + Z1*Z2*(v2 - v1))/(Z1 + Z2)
```

```python
vhat
```

```python
phat
```

```python
simplify(-sat1*(v1 - vhat))
```

```python
simplify(-sat2*(v2 - vhat))
```

```python
simplify(-sat1*(p1 - phat))
```

```python
simplify(-sat2*(p2 - phat))
```

### For a crack interface

```python
vhat1 = (Z1*v1-p1 + p_c)/Z1
vhat2 = (Z2*v2+p2 - p_c)/Z2
print(vhat1)
print(vhat2)
```

    (Z1*v1 - p1 + pc)/Z1
    (Z2*v2 + p2 - pc)/Z2

```python
simplify(-sat1*(v1 - vhat1)-sat2*(v2-vhat2))
```

```python
simplify(-sat1*(p1-p_c)-sat2*(p2-p_c))
```

```python
Q = A*(vhat2 - vhat1)
simplify(Q).expand().collect(p_c)
```

### For a crack bottom

```python
vhat1 = (Z1*v1-p1 + p_c)/Z1
vhat1
```

```python
simplify(-sat1*(v1 - vhat1))
```

```python
simplify(-sat1*(p1 - pc))
```

## Material interface considering moving surface and cross-section area jump

```python
A1, A2, v1, v2, p1, p2, rho1, c1, rho2, c2, Z1, Z2, eta, g = symbols('A1, A2, v1, v2, p1, p2, rho1, c1, rho2, c2, Z1, Z2, eta, g')
# hat variables.
v1hat, v2hat, p1hat, p2hat, sat1, sat2 = symbols('v1hat, v2hat, p1hat, p2hat, sat1, sat2')
```

### 1. a pure material interface with no jump in surface area, including the gravity term.

v1/2  : velocity at + and - side.

p1/2  : pressure at + and - side.

Z1/2  : impedance at + and - side.

eta   : moving surface location, positive in vertical up direction

```python
# continuity of pressure and velocity at the moving interface
# p_1(eta) = p_2(eta)
# v_1(eta) = v_2(eta)

# linearization the interface condition

# pressure:
# note that p1/2_z is dominated by the pressure of the background state.
# p1(eta) = p1(0)+ p1_z*eta = p1 - rho1*g*eta
# p2(eta) = p2(0)+ p2_z*eta = p2 - rho2*g*eta

# velocity:
# there's not velocity gradient in the background state, therefore v1/2_z*eta is a second order term
# v1(eta) = v1(0)+ v1_z*eta = v1
# v2(eta) = v2(0)+ v2_z*eta = v2

# Jump condition in terms of p1, and p2 (evaludated at the un-perturbed interface)
# p1hat - p2hat = (rho1-rho2)*g*eta
# v1hat - v2hat = 0

# At -/+ side of interface, we need to preserving the upgoing/downgoing characteristics
#  Z2*v2+p2 = Z2*v2hat+p2hat
#  Z1*v1-p1 = Z1*v1hat-p1hat

# Finally we have an kinematic equation relating moving surface height to interface velocity
# eta_t = vhat
```

```python
v1hat = v2hat = ((Z1*v1-p1) + (Z2*v2+p2) + (rho1-rho2)*g*eta)/(Z1+Z2)
p1hat = -(Z1*v1-p1 - Z1*v1hat)
p2hat = Z2*v2+p2 - Z2*v2hat
simplify(v1hat)
# if we set g=0, we should make sure the expressions recover to the previous simpler version.
```

```python
# sat terms for velocity and pressure.
simplify(-sat1*(v1 - v1hat))
```

```python
simplify(-sat2*(v2-v2hat))
```

```python
simplify(-sat1*(p1 - p1hat))
```

```python
simplify(-sat2*(p2-p2hat))
```

### 1. a pure jump in cross-section with no jump in material properties.

A1/2  : cross-section at + and - side.

v1/2  : velocity at + and - side.

p1/2  : pressure at + and - side.

Z1/2  : impedance at + and - side.

eta   : moving surface location, positive in vertical up direction

```python
# conservation of mass and momentum:
# (A*rho)_t + (A*rho*v)_z = 0
# (A*rho*v)_t + (A*p)_z = 0

# Jump conditions:
# A1*rho1*v1hat = A2*rho2*v2hat
# p1hat         = p2hat

# preserving the two characteristics:
#  Z2*v2+p2 = Z2*v2hat+p2hat
#  Z1*v1-p1 = Z1*v1hat-p1hat

# Z1*v1hat - Z2*v2hat = (Z1*v1+p1) - (Z2*v2+p2)
# Z1*v1hat - Z2*A1*rho1/(A2*rho2)*v1hat = (Z1*v1+p1)*A1 - (Z2*v2+p2)*A2
v1hat = ((Z1*v1+p1) - (Z2*v2+p2))/(Z1 - Z2*A1/A2*rho1/rho2)
v2hat = A1*rho1*v1hat/(A2*rho2)
p2hat = Z2*v2+p2-(Z2*v2hat)
p1hat = Z1*v1+p1-(Z1*v1hat)

simplify(v1hat)
simplify(v2hat)
simplify(p1hat)
simplify(p2hat)
```

```python
# sat terms for velocity and pressure.
simplify(-sat1*(v1 - v1hat)-sat2*(v2-v2hat))
simplify(-sat1*(p1 - p1hat)-sat2*(p2-p2hat))
```

### grid stretch Jacobian

```python
eta, r = symbols('eta r')
```

```python
z      = 0.5 + 0.5*tanh(r*(2*eta-1))/tanh(r)
J      = z.diff(eta)
J
```

    1.0*r*(-tanh(r*(2*eta - 1))**2 + 1)/tanh(r)

```python

```


```python
# Método para resolver las energías y eigenfunciones de un sistema cuántico numéricamente por Teoría de Pertubaciones
# Modelado Molecular 2
# By: José Manuel Casillas Martín     22-oct-2017
import numpy as np
from sympy import *
from sympy.physics.qho_1d import E_n, psi_n
from sympy.physics.hydrogen import E_nl, R_nl
from sympy import init_printing; init_printing(use_latex = 'mathjax')
from scipy import integrate
from scipy.constants import hbar, m_e, m_p, e
from mpmath import spherharm
from numpy import inf, array
import numpy as np
import matplotlib.pyplot as plt
import traitlets
from IPython.display import display
from ipywidgets import Layout, Box, Text, Dropdown, Label, IntRangeSlider, IntSlider, RadioButtons
```

Teoría de Perturbaciones

Consiste en resolver un sistema perturbado(se conoce la solución al no perturbado), y donde el interés es conocer la contribución de la parte perturbada $H'$ al nuevo sistema total.

$$ H = H^{0}+H'$$

La resolución adecuada del problema, depende en gran parte, de una correcta elección de $H'$.

```python
form_item_layout = Layout(display='flex',flex_flow='row',justify_content='space-between')

PType=Dropdown(options=['Particle in a one-dimensional box', 'Harmonic oscilator', 'Hydrogen atom (Helium correction)'])
Pert=Text()
Rang=IntRangeSlider(min=0, max=20, step=1, disabled=False, continuous_update=False, orientation='horizontal',\
                    readout=True, readout_format='d')
M=Text()
Correc=Dropdown(options=['1', '2'])
hbarra=Dropdown(options=[1, 1.0545718e-34])

form_items = [
    Box([Label(value='Problem'),PType], layout=form_item_layout),
    Box([Label(value='Perturbation'),Pert], layout=form_item_layout),
    Box([Label(value='Correction order'),Correc], layout=form_item_layout),
    Box([Label(value='n Range'),Rang], layout=form_item_layout),
    Box([Label(value='Mass'),M], layout=form_item_layout),
    Box([Label(value='Hbar'),hbarra], layout=form_item_layout),]

form = Box(form_items, layout=Layout(display='flex',flex_flow='column',border='solid 2px',align_items='stretch',width='40%'))
form
```

En esta caja interactiva llena los datos del problema que deseas resolver.

# Nota 1:
Es recomendable usar unidades atómicas de Hartree para eficientar los cálculos. 1 u.a. (energía)= 27.211eV.

# Nota 2:
Para la partícula en una caja unidimensional es recomendable que n sea mayor a 1.

## Nota 3:
Para la correción a la energía del átomo de Helio sólo es necesario seleccionar el problema, automáticamente se calcula la correción a primer orden y no se corrigen las funciones de onda.

```python
Problem=PType.value
form_item_layout = Layout(display='flex',flex_flow='row',justify_content='space-between')

L=Text()
W=Text()
atomic_number=RadioButtons(options=['1 (Show Hydrogen energies)','2 (Correct Helium first energy)'],disabled=False)

if Problem=='Particle in a one-dimensional box':
    form_items = [Box([Label(value='Large of box'),L], layout=form_item_layout)]

if Problem=='Harmonic oscilator':
    form_items = [Box([Label(value='Angular Fr'),W], layout=form_item_layout)]

if Problem=='Hydrogen atom (Helium correction)':
    form_items = [Box([Label(value='Atomic number'),atomic_number], layout=form_item_layout)]

form = Box(form_items, layout=Layout(display='flex',flex_flow='column',border='solid 2px',align_items='stretch',width='40%'))
form
```

```python
# Variables que se utilizarán
# x=variable de integracion, l=largo del pozo, m=masa del electrón, w=frecuencia angular
# n=número cuántico principal, Z=Número atómico, q=número cuántico angular(l)
var('x theta phi')
var('r1 r2', real=True)
var('l m hbar w n Z', positive=True, real=True)

# Perturbación
if Pert.value!='':
    H_p=sympify(Pert.value)
    h_p=eval(Pert.value)
else:
    H_p=0
    h_p=0

# Constantes
h=hbarra.value
a0=5.2917721067e-11
if M.value!='':
    mass=float(eval(M.value))
else:
    mass=1

# Energías y funciones que se desea corregir
n_inf=min(Rang.value)
n_sup=max(Rang.value)

if Problem=='Particle in a one-dimensional box':
    if L.value=='':
        large=1
    else:
        large=float(eval(L.value))
    omega=0
    # Energías del pozo de potencial infinito
    k=n*pi/l
    En=hbar**2*k**2/(2*m)

    # Funciones de onda del pozo de potencial infinito
    Psin=sqrt(2/l)*sin(n*pi*x/l)

    # Límites del pozo definido de 0 a l para sympy
    li_sympy=0
    ls_sympy=l
    # Mismo limites para scipy
    li_scipy=0
    ls_scipy=large

if Problem=='Harmonic oscilator':
    large=0
    if W.value=='':
        omega=1
    else:
        omega=float(eval(W.value))
    # Energías del oscilador armónico cuántico
    En=E_n(n,w)

    # Funciones de onda del oscilador armónico cuántico
    Psin=psi_n(n,x,m,w)

     # Límites del pozo definido de -oo a oo para sympy
    li_sympy=-oo
    ls_sympy=oo
    # Límites del pozo definido de -oo a oo para scipy
    li_scipy=-inf
    ls_scipy=inf

if Problem=='Hydrogen atom (Helium correction)':
    if atomic_number.value=='1 (Show Hydrogen energies)':
        z=1
    if atomic_number.value=='2 (Correct Helium first energy)':
        z=2
    large=0
    omega=0

    # Energías del átomo hidrogenoide
    En=z*E_nl(n,z)

    # Funciones de onda del átomo de hidrógeno
    # Número cuántico l=0
    q=0 # La variable l ya esta siendo utilizada para el largo de la caja por ello se sustituyo por q
    Psin=(R_nl(n,q,r1,z)*R_nl(n,q,r2,z))

     # Límites del átomo de hidrógeno de 0 a oo para sympy
    li_sympy=0
    ls_sympy=oo
    # Límites del átomo de hidrógeno de 0 a oo para scipy
    li_scipy=0
    ls_scipy=inf
```

Para sistemas no degenerados, la corrección a la energía a primer orden se calcula como

$$ E_{n}^{(1)} = \int\psi_{n}^{(0)*} H' \psi_{n}^{(0)}d\tau$$

** Tarea 1 : Programar esta ecuación si conoces $H^{0}$ y sus soluciones. **

```python
def correcion_1st_order_Energy(E_n,Psi_n,H_p,li,ls):
    E1_n=Integral(Psi_n*(H_p)*Psi_n,(x,li,ls)).doit()
    return(E_n+E1_n)
```

```python
# Correción de la energía a primer orden
E=[]
Eev=[]
Ec1=[]
if Problem=='Particle in a one-dimensional box' or Problem=='Harmonic oscilator':
    for i in range(n_inf,n_sup+1):
        E.append(En.subs({n:i}))
        Eev.append(E[i-n_inf].subs({m:mass, l:large, hbar:h}).evalf())
        Ec1.append(correcion_1st_order_Energy(En.subs({n:i}),Psin.subs({n:i}),H_p,li_sympy,ls_sympy))
if Problem=='Hydrogen atom (Helium correction)':
    for i in range(n_inf,n_sup+1):
        E.append(En.subs({n:i}))
        Eev.append(E[i-n_inf])
    if z==2:
        integral_1=Integral(Integral((16*z**6*r1*r2**2*exp(-2*z*(r1+r2))),(r2,0,r1)),(r1,0,oo)).doit()
        integral_2=Integral(Integral((16*z**6*r1**2*r2*exp(-2*z*(r1+r2))),(r2,r1,oo)),(r1,0,oo)).doit()
        integral_total=(integral_1+integral_2)
        Ec1.append(E[0]+integral_total)
```

Y la corrección a la función de onda, también a primer orden, se obtiene como:

 $$ \psi_{n}^{(1)} = \sum_{m\neq n} \frac{\langle\psi_{m}^{(0)} | H' | \psi_{n}^{(0)} \rangle}{E_{n}^{(0)} - E_{m}^{(0)}} \psi_{m}^{(0)}$$

**Tarea 2: Programar esta ecuación si conoces $H^{0}$ y sus soluciones. **

```python
# Correción de las funciones a primer orden
if Pert.value!='':
    if Problem=='Particle in a one-dimensional box' or Problem=='Harmonic oscilator':
        Psi_c=[]
        integrals=np.zeros((n_sup+1,n_sup+1))
        for i in range(n_inf,n_sup+1):
            a=0
            for j in range(n_inf,n_sup+1):
                if i!=j:
                    integ= lambda x: eval(str(Psin.subs({n:j})*(h_p)*Psin.subs({n:i}))).subs({m:mass,l:large,w:omega,hbar:h})
                    integrals[i,j]=integrate.quad(integ,li_scipy,ls_scipy)[0]
                    cte=integrals[i,j]/(En.subs({n:i,m:mass,l:large})-En.subs({n:j,m:mass,l:large})).evalf()
                    a=a+cte*Psin.subs({n:j})
            Psi_c.append(Psin.subs({n:i})+a)
```

**Tarea 3: Investigue las soluciones a segundo orden y también programe las soluciones. **

Y la corrección a la energía a segundo orden, se obtiene como:

 $$  E_{n}^{(2)} = \sum_{m\neq n} \frac{|\langle\psi_{m}^{(0)} | H' | \psi_{n}^{(0)} \rangle|^{2}}{E_{n}^{(0)} - E_{m}^{(0)}} $$

```python
# Correción a la energía a segundo orden
if Pert.value!='':
    if Problem=='Particle in a one-dimensional box' or Problem=='Harmonic oscilator':
        if Correc.value=='2':
            Ec2=[]
            for i in range(n_inf,n_sup+1):
                a=0
                for j in range(n_inf,n_sup+1):
                    if i!=j:
                        cte=((integrals[i,j])**2)/(En.subs({n:i,m:mass,l:large,hbar:h})-En.subs({n:j,m:mass,l:large,hbar:h})).evalf()
                        a=a+cte
                Ec2.append(Ec1[i-n_inf]+a)
```

**A continuación se muestran algunos de los resultados al problema resuelto**

Las energías sin perturbación son:

```python
if Problem=='Particle in a one-dimensional box' or Problem=='Harmonic oscilator':
    if Pert.value!='':
        print('%s'%eval(E))
    if Pert.value=='':
        print('No se ingreso ninguna perturbación')
if Problem=='Hydrogen atom (Helium correction)':
    if z==1:
        for i in range(n_inf,n_sup+1):
            print('La energía del estado %d es: %.3f' %i %(E[0]*27.211))
    print('Para el estado base del Helio la energía es: %.3f eV' %(E[0]*27.11))
```

    Para el estado base del Helio la energía es: -108.440 eV

La correción a primer orden de las energías son:

```python
if Problem=='Particle in a one-dimensional box' or Problem=='Harmonic oscilator':
    if Pert.value!='':
        print(Ec1)
    if Pert.value=='':
        print('No se ingreso ninguna perturbación')
if Problem=='Hydrogen atom (Helium correction)':
    print('Para el estado base del Helio la energía es: %.3f eV' %(Ec1[0]*27.11))
```

    Para el estado base del Helio la energía es: -74.552 eV

Si seleccionaste en los parámetros iniciales una correción a segundo orden entonces...

Las correciones a la energía a segundo orden son:

```python
if Problem=='Particle in a one-dimensional box' or Problem=='Harmonic oscilator':
    if Pert.value!='':
        if Correc.value=='2':
            print(Ec2)
    if Pert.value=='':
        print('No se ingreso ninguna perturbación')
if Problem=='Hydrogen atom (Helium correction)':
    print('Este programa no corrige las energías a segundo orden de un átomo hidrogenoide')
```

    Este programa no corrige las energías a segundo orden de un átomo hidrogenoide

Ahora vamos con la función de onda $(\psi)$

```python
form_item_layout = Layout(
    display='flex',
    flex_flow='row',
    justify_content='space-between')

Graph=IntSlider(min=n_inf, max=n_sup, step=1, disabled=False, continuous_update=False, orientation='horizontal',\
                readout=True, readout_format='d')

form_items = [
    Box([Label(value='What function do you want to see?'),
         Graph], layout=form_item_layout)]

form = Box(form_items, layout=Layout(
    display='flex',
    flex_flow='column',
    border='solid 2px',
    align_items='stretch',
    width='40%'))
form
```

La función de onda original es:

```python
Psin.subs({n:Graph.value})
```

$$\left(- 2 r_{1} + 2\right) \left(- 2 r_{2} + 2\right) e^{- r_{1}} e^{- r_{2}}$$

La correción a primer orden a la función de onda (utilizando todas las funciones en el rango seleccionado) es:

```python
if Problem=='Particle in a one-dimensional box' or Problem=='Harmonic oscilator':
    if Pert.value!='':
            Psi_c[Graph.value-n_inf]
    if Pert.value=='':
        print('No se ingreso ninguna perturbación')
if Problem=='Hydrogen atom (Helium correction)':
    print('Este programa no corrige las fucniones de un átomo hidrogenoide')
```

    Este programa no corrige las fucniones de un átomo hidrogenoide

Vamos a graficarlas para verlas mejor...

La función de onda original es:

```python
if Problem=='Particle in a one-dimensional box':
    plot(eval(str(Psin)).subs({n:Graph.value,m:mass,l:large,w:omega,hbar:h}),xlim=(li_scipy,ls_scipy),\
         title='$\psi_{%d}$'%Graph.value)
if Problem=='Harmonic oscilator':
    plot(eval(str(Psin)).subs({n:Graph.value,m:mass,l:large,w:omega,hbar:h}),xlim=(-10*h/(mass*omega),10*h/(mass*omega)),\
         title='$\psi_{%d}$'%Graph.value)
if Problem=='Hydrogen atom (Helium correction)':
    print('Densidad de probabilidad para un electrón')
    plot(eval(str((4*pi*x**2*R_nl(Graph.value,q,x,z)**2))),xlim=(0,10),ylim=(0,20/Graph.value), title='$\psi_{%ds}$'%Graph.value)
    print('Tome en cuenta que debido a la dificultad para seleccionar los límites de la gráfica se muestran bien los primeros\n\
3 estados. A partir de ahí visualizar la gráfica se complica.')
```

La corrección a la función de onda es:

```python
if Problem=='Particle in a one-dimensional box':
    if Pert.value!='':
        plot(eval(str(Psi_c[Graph.value-n_inf])).subs({n:Graph.value,m:mass,l:large,w:omega,hbar:h}),\
             xlim=(li_scipy,ls_scipy),title='$\psi_{%d}$'%Graph.value)
    if Pert.value=='':
        print('No se ingreso ninguna perturbación')
if Problem=='Harmonic oscilator':
    if Pert.value!='':
        plot(eval(str(Psi_c[Graph.value-n_inf])).subs({n:Graph.value,m:mass,l:large,w:omega,hbar:h}),\
             xlim=(-10*h/(mass*omega),10*h/(mass*omega)),title='$\psi_{%d}$' %Graph.value)
    if Pert.value=='':
        print('No se ingreso ninguna perturbación')
if Problem=='Hydrogen atom (Helium correction)':
    print('Este programa no corrige las fucniones de un átomo hidrogenoide')
```

    Este programa no corrige las fucniones de un átomo hidrogenoide

**Tarea 4.  Resolver el átomo de helio aplicando los programas anteriores.**

Para resolver el átomo de helio se utilizaron los conceptos, que sirvieron como base para las primeras tareas. Sin embargo, en el Apéndice 1 viene con mayor detalles las consideraciones tomadas para resolver el problema.

## Apéndice 1

Para el cálculo a las correciones del átomo de Helio se tomó en cuenta lo siguiente...

La función de onda del átomo de Helio puede ser representada como:

$$ \psi_{nlm} = \psi(r1)_{nlm} \psi(r2)_{nlm}$$

Donde, para el estado fundamental:

$$ \psi(r_{1}.r_{2})_{100} = \frac{Z^{3}}{\pi a_{0}^{3}} e^{\frac{-Z}{a_{0}}(r_{1}+r_{2})}$$

Y la perturbación sería el término de repulsión entre los dos electrones, es decir:

$$ H'= \frac{e^{2}}{r_{12}}=\frac{e^{2}}{|r_{1}-r_{2}|}$$

Finalmente la correción a primer orden de la energía sería:

$$ E^{1}= \langle\psi_{n}^{(0)} | H' | \psi_{n}^{(0)} \rangle =\frac{Z^{6}e^{2}}{\pi^{2} a_{0}^{6}} \int_{0}^{2\pi}\int_{0}^{2\pi}\int_{0}^{\pi}\int_{0}^{\pi}\int_{0}^{\infty}\int_{0}^{\infty} \frac{e^{\frac{-2Z}{a_{0}}(r_{1}+r_{2})}}{r_{12}} r_{1}^{2}r_{2}^{2}sen{\theta_{1}}sen{\theta_{2}} dr_{2} dr_{1} d\theta_{2} d\theta_{1} d\phi_{2} d\phi_{1}$$

Se utiliza una expansión del termino de repulsión con los armónicos esféricos y se integra la parte angular. Una vez hecho eso, la integral queda expresada de la siguiente manera:

$$ E^{1}= \frac{16Z^{6}e^{2}}{a_{0}^{6}} \left[\int_{0}^{\infty} r_{1}^{2} e^{\frac{-2Z}{a_{0}}r_{1}} \left(\int_{0}^{r_{1}} \frac{r_{2}^{2}}{r_{1}} e^{\frac{-2Z}{a_{0}}r_{2}} dr_{2}+\int_{r_{1}}^{\infty}r_{2} e^{\frac{-2Z}{a_{0}}r_{2}}dr_{2}\right) dr_{1} \right]$$

**Tarea 5: Método variacional-perturbativo. **

Este método nos permite estimar de forma precisa $E^{(2)}$ y correcciones perturbativas de la energía de órdenes más elevados para el estado fundamental del sistema, sin evaluar sumas infinitas. Ver ecuación 9.38 del libro.

```python

```

**Tarea 6. Revisar sección 9.7. **

Inicialmente a mano, y sengunda instancia favor de intentar programar sección del problema, i.e. integral de Coulomb  e integral de intercambio.

```python

```