# Frequentist Inference Case Study - Part B

## Learning objectives

Welcome to Part B of the Frequentist inference case study! The purpose of this case study is to help you apply the concepts associated with Frequentist inference in Python. In particular, you'll practice writing Python code to apply the following statistical concepts:
* the _z_-statistic
* the _t_-statistic
* the difference and relationship between the two
* the Central Limit Theorem, including its assumptions and consequences
* how to estimate the population mean and standard deviation from a sample
* the concept of a sampling distribution of a test statistic, particularly for the mean
* how to combine these concepts to calculate a confidence interval

In the previous notebook, we used only data from a known normal distribution. **You'll now tackle real data, rather than simulated data, and answer some relevant real-world business problems using the data.**

## Hospital medical charges

Imagine that a hospital has hired you as their data scientist. An administrator is working on the hospital's business operations plan and needs you to help them answer some business questions.

In this assignment notebook, you're going to use frequentist statistical inference on a data sample to answer the questions:
* has the hospital's revenue stream fallen below a key threshold?
* are patients with insurance really charged different amounts than those without?

Answering that last question with a frequentist approach makes some assumptions, and requires some knowledge, about the two groups.

We are going to use some data on medical charges obtained from [Kaggle](https://www.kaggle.com/easonlai/sample-insurance-claim-prediction-dataset).

For the purposes of this exercise, assume the observations are the result of random sampling from our single hospital. Recall that in the previous assignment, we introduced the Central Limit Theorem (CLT), and its consequence that the distributions of sample statistics approach a normal distribution as $n$ increases. The amazing thing about this is that it applies to the sampling distributions of statistics that have been calculated from even highly non-normal distributions of data! Recall, also, that hypothesis testing is very much based on making inferences about such sample statistics. You're going to rely heavily on the CLT to apply frequentist (parametric) tests to answer the questions in this notebook.

```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import matplotlib.ticker as mtick
from math import sqrt
from scipy.stats import norm
from scipy.stats import ttest_ind
from numpy.random import seed
file_location ='C:/Users/Greki/Frequentist_Inference/data/raw'
medical = pd.read_csv(file_location+'/insurance2.csv').sort_values(by='charges', ascending=True)
```

```python
medical.shape
```

    (1338, 8)

```python
medical.head()
```

      age
      sex
      bmi
      children
      smoker
      region
      charges
      insuranceclaim

      940
      18
      1
      23.21
      0
      0
      2
      1121.8739
      0

      808
      18
      1
      30.14
      0
      0
      2
      1131.5066
      1

      1244
      18
      1
      33.33
      0
      0
      2
      1135.9407
      1

      663
      18
      1
      33.66
      0
      0
      2
      1136.3994
      1

      22
      18
      1
      34.10
      0
      0
      2
      1137.0110
      1

__Q1:__ Plot the histogram of charges and calculate the mean and standard deviation. Comment on the appropriateness of these statistics for the data.

__A:__

```python
_=medical.hist('charges', bins=35)
_ = plt.xlabel('USD')
_ = plt.ylabel('number of patients')
_ = plt.title('Distribution of charges')
```

```python
mean= round(np.mean(medical.charges),2)
print("Mean: $" + str(mean))

std_dev =  round(np.std(medical.charges),2)
print("Standard Deviation: $" + str(std_dev))

min =  round(np.min(medical.charges),2)
print("Minimum: $" + str(min))

median =  round(np.median(medical.charges),2)
print("Median: $" + str(median))

max =  round(np.max(medical.charges),2)
print("Maximum: $" + str(max))

p25, p75 = np.percentile(medical.charges, [25, 75])
percen = np.percentile(medical.charges, [69])
print(percen)
print("25th Percentile: $"+str(round(p25,2)) + ", 75th Percentile: $"+str(round(p75,2)))
```

    Mean: $13270.42
    Standard Deviation: $12105.48
    Minimum: $1121.87
    Median: $9382.03
    Maximum: $63770.43
    [13422.456669]
    25th Percentile: $4740.29, 75th Percentile: $16639.91

Medical charges distribution is skewed heavily to the left side of the mean ($13270.42) with the median at $9382.03.  There is a long tail that tappers to the right.  The distribution is not remotely normal.

```python
#grapgh ecdf and mean, median,25th and 75th percentile to show if data is normal or not

_=sns.ecdfplot(data=medical.charges, label='ECDF' )
_=plt.axvline(mean, color='r', linestyle='--', label='Mean')
_=plt.axvline(median, color='g', linestyle='--', label='Median')
_=plt.axvline(p25, color='y', linestyle='--', label='25th Percentile')
_=plt.axvline(p75, color='b', linestyle='--', label='75th Percentile')
_=plt.title('ECDF of actual charges in patient population')
_= plt.xlabel('USD')
_=plt.legend()
```

__Q2:__ The administrator is concerned that the actual average charge has fallen below 12,000, threatening the hospital's operational model. On the assumption that these data represent a random sample of charges, how would you justify that these data allow you to answer that question? And what would be the most appropriate frequentist test, of the ones discussed so far, to apply?

__A:__ Although the mean exceedes 12,000 the sckewed nature of this distribution is a cause for concern. with a median of approx 9,000 (50th percentile) and the mean is 13,270 ( approx 69th percentile).  This means that the remaining 30 percent of the medical  have to be at a very high value to offset the values that are to the left of the mean.

```python
medical_charges=np.array(medical['charges'])
```

```python
def medrec_sampler(n):
    return np.random.choice(medical_charges, n)
```

```python
seed(47)
super_samples = [np.mean(medrec_sampler(50)) for i in range(300)]
medrec_super_samples= np.sort(super_samples)
mean_of_means= np.mean(medrec_super_samples)
STD= np.std(medrec_super_samples,ddof=1)
H_12000= norm.cdf(12000,mean_of_means,STD)
mean_x_max = norm.cdf(mean_of_means,mean_of_means,STD)
h_mean=norm.cdf(mean_of_means,mean_of_means,STD)
min_mean= np.min(medrec_super_samples)
max_mean=np.max(medrec_super_samples)
```

```python
print(min_mean)
print(max_mean)
print(mean_of_means)
print(STD)
```

    9036.8968028
    19267.910133799996
    13315.186266487599
    1662.349076113126

```python
_= plt.hist(medrec_super_samples, bins= 20, density = True )
_=dist = norm.pdf(medrec_super_samples, mean_of_means, STD)
_= plt.plot(medrec_super_samples,dist, color='y', label ='PDF')
_=plt.gca().yaxis.set_major_formatter(mtick.PercentFormatter(xmax=1.0))
_ = plt.xlabel('USD')
_ = plt.ylabel('% of invoices')
#_= plt.yaxis.set_major_formatter(mtick.PercentFormatter())
_ = plt.title('Distribution of the mean medical invoices in sample size 50')
_=plt.axvline(mean_of_means, color='r', linestyle='--', label='Mean')
_=plt.legend()
```

```python
#grapgh cdf and mean, median, and 12,000 cutoff

_=sns.ecdfplot(data=medrec_super_samples, label='CDF' )
_=plt.vlines(x=mean_of_means,ymin=0, ymax= mean_x_max,color='r', linestyle='--', label='Mean')
_=plt.hlines(y=h_mean,xmin=min_mean,xmax=mean_of_means, color='r', linestyle='--')
_=plt.vlines(x=12000,ymin=0, ymax= H_12000,color='g', linestyle='--', label='12,000 cutoff')
_=plt.hlines(y=H_12000,xmin=min_mean,xmax=12000, color='g', linestyle='--')
_= plt.xlabel('USD')
_=plt.legend()
_ = plt.title('Location of the mean of means medical invoices and $12,000 mean cutoff')
```

__Q3:__ Given the nature of the administrator's concern, what is the appropriate confidence interval in this case? A ***one-sided*** or ***two-sided*** interval? (Refresh your understanding of this concept on p. 399 of the *AoS*). Calculate the critical value and the relevant 95% confidence interval for the mean, and comment on whether the administrator should be concerned.

 __A:__  Given that the administrator is concerned with the mean being below $12,000, the answer would be to utilize a one-sided interval utilizing a Z score due to sample size of 200. critical value = 1 -0.95 = 0.05. Z Score = -1.65   lower confidence interval is calculated as ( Mean - Zscore(standard deviation/ sqrt of sample size)) = 13315 -1.65(1662/(300 ^ 1/2)) = 13315-158.33= 13156.67.

Given that the lower confidence interval is 13,156.67, then the Administrator should have 95% confidence that the mean will be greater 13,156.67 and $12,000 or less would be an exception.

The administrator then wants to know whether people with insurance really are charged a different amount to those without.

__Q4:__ State the null and alternative hypothesis here. Use the _t_-test for the difference between means, where the pooled standard deviation of the two groups is given by:
\begin{equation}
s_p = \sqrt{\frac{(n_0 - 1)s^2_0 + (n_1 - 1)s^2_1}{n_0 + n_1 - 2}}
\end{equation}

and the *t*-test statistic is then given by:

\begin{equation}
t = \frac{\bar{x}_0 - \bar{x}_1}{s_p \sqrt{1/n_0 + 1/n_1}}.
\end{equation}

(If you need some reminding of the general definition of ***t-statistic***, check out the definition on p. 404 of *AoS*).

What assumption about the variances of the two groups are we making here?

## Hypothesis

H0 = mean0 = mean1 = insured and uninsured are charged the same

Ha = mean0 ≠ mean1 = insured and uninsured are charged differently

## Insured Population

```python
def insur_sampler(data,n):
    return np.random.choice(data, n)
```

```python
#Calc for insured population
W_insur=(medical.query('insuranceclaim==1').charges)

#Histogram for insured population
_=W_insur.hist( bins=35)
_ = plt.xlabel('USD')
_ = plt.ylabel('number of patients')
_ = plt.title('Population Distribution of Charges with Insurance')

# Descriptive statistics for insured population
mean_W_insur= round(np.mean(W_insur),2)
print("Mean: $" + str(mean_W_insur))

std_dev_insur =  round(np.std(W_insur),2)
print("Standard Deviation: $" + str(std_dev_insur))

min =  round(np.min(W_insur),2)
print("Minimum: $" + str(min))

median =  round(np.median(W_insur),2)
print("Median: $" + str(median))

max =  round(np.max(W_insur),2)
print("Maximum: $" + str(max))

W_Insur_size = W_insur.size

print("Size: "+ str(W_Insur_size))
```

## Insured Sample

```python
#Statistics for insured sample
seed(47)
w_insur_samples = [insur_sampler(W_insur,1) for i in range(7)]

w_insur_sorted_samples= np.sort(np.array(w_insur_samples),axis=0)

mean_w_insur_samples= np.mean(w_insur_sorted_samples)
STD_w_insur_samples= np.std(w_insur_sorted_samples,ddof=1)
Var_w_insur_samples= STD_w_insur_samples**2
w_insur_count= (w_insur_sorted_samples.size)
print("Mean samples from with insurance: "+ str(mean_w_insur_samples))
print("STD samples from with insurance: "+ str(STD_w_insur_samples))
print("Variance samples from with insurance: "+ str(Var_w_insur_samples))
print("number of samples from with insurance: "+ str(w_insur_count))
```

    Mean samples from with insurance: 12479.178692857144
    STD samples from with insurance: 13895.209332881295
    Variance samples from with insurance: 193076842.40459144
    number of samples from with insurance: 7

```python
#histogram for insured sample
_= plt.hist(w_insur_sorted_samples, bins= 8, density = False )
_=dist_w_insur = norm.pdf(w_insur_sorted_samples, mean_w_insur_samples, STD_w_insur_samples)
#_= plt.plot(w_insur_sorted_samples,dist_w_insur, color='y', label ='PDF')
#_=plt.gca().yaxis.set_major_formatter(mtick.PercentFormatter(xmax=1.0))
_ = plt.xlabel('USD')
_ = plt.ylabel('Invoices')
_ = plt.title('Distribution of medical invoices with insurance in sample size' + str(w_insur_count) )
_=plt.axvline(mean_w_insur_samples, color='r', linestyle='--', label='Mean')
_=plt.legend()
```

## Uninsured Population

```python
#Calc for uninsured population
WO_insur=(medical.query('insuranceclaim==0').charges)

#Historam for uninsured population
_=WO_insur.hist( bins=35)
_ = plt.xlabel('USD')
_ = plt.ylabel('number of patients')
_ = plt.title('Population Distribution of charges without Insurance')

mean_WO_insur= round(np.mean(WO_insur),2)
print("Mean: $" + str(mean_WO_insur))

std_dev_WO_insur =  round(np.std(WO_insur),2)
print("Standard Deviation: $" + str(std_dev_WO_insur))

min =  round(np.min(WO_insur),2)
print("Minimum: $" + str(min))

median =  round(np.median(WO_insur),2)
print("Median: $" + str(median))

max =  round(np.max(WO_insur),2)
print("Maximum: $" + str(max))

WO_Insur_size = WO_insur.size

print("Size: "+ str(WO_Insur_size))
```

## Uninsured Sample

```python
#Statistics for uninsured sample
seed(47)
wo_insur_samples = [insur_sampler(WO_insur,1) for i in range(7)]

wo_insur_sorted_samples= np.sort(np.array(wo_insur_samples),axis=0)

mean_wo_insur_samples= np.mean(wo_insur_sorted_samples)
STD_wo_insur_samples= np.std(wo_insur_sorted_samples,ddof=1)
Var_wo_insur_samples= STD_wo_insur_samples**2
wo_insur_count= (wo_insur_sorted_samples.size)
print("Mean samples from without insurance: "+ str(mean_wo_insur_samples))
print("STD samples from without insurance: "+ str(STD_wo_insur_samples))
print("Var samples from without insurance: "+ str(Var_wo_insur_samples))
print("number of samples from without insurance: "+ str(wo_insur_count))
```

    Mean samples from without insurance: 4437.362942857143
    STD samples from without insurance: 2303.4757778349863
    Var samples from without insurance: 5306000.659072495
    number of samples from without insurance: 7

__A:__

```python
#Historam for uninsured sample
_= plt.hist(wo_insur_sorted_samples, bins= 8, density = False )
_=dist_wo_insur = norm.pdf(wo_insur_sorted_samples, mean_wo_insur_samples, STD_wo_insur_samples)
#_= plt.plot(wo_insur_sorted_samples,dist_wo_insur, color='y', label ='PDF')
#_=plt.gca().yaxis.set_major_formatter(mtick.PercentFormatter(xmax=1.0))
_ = plt.xlabel('USD')
_ = plt.ylabel('Invoices')
_ = plt.title('Distribution of medical invoices without insurance in sample size ' + str(wo_insur_count))
_=plt.axvline(mean_wo_insur_samples, color='r', linestyle='--', label='Mean')
_=plt.legend()
```

__Q5:__ Perform this hypothesis test both manually, using the above formulae, and then using the appropriate function from [scipy.stats](https://docs.scipy.org/doc/scipy/reference/stats.html#statistical-tests) (hint, you're looking for a function to perform a _t_-test on two independent samples). For the manual approach, calculate the value of the test statistic and then its probability (the p-value). Verify you get the same results from both.

__A:__

```python
# manual calc for ttest

sp=(((w_insur_count -1)*Var_w_insur_samples)+((wo_insur_count -1)*Var_wo_insur_samples))/(w_insur_count+wo_insur_count-2)

t_stat_manual=(mean_w_insur_samples - mean_wo_insur_samples)/sqrt(((sp/w_insur_count)+(sp/wo_insur_count)))

print("Manual T-Test statistic calculation: " + str(t_stat_manual))

print("Could not calculate statistic manually without t table that did not have a value for t 0f 1.51. Assumed df for t table should be derived from smaller n value.  both are 7 chose df of 6. value from table for two sided ttest is 0.184")
```

    Manual T-Test statistic calculation: 1.5106055764909945
    Could not calculate statistic manually without t table that did not have a value for t 0f 1.51. Assumed df for t table should be derived from smaller n value.  both are 7 chose df of 6. value from table for two sided ttest is 0.184

```python
ttest=ttest_ind(w_insur_sorted_samples, wo_insur_sorted_samples, axis=None, equal_var=False)
print(ttest)
```

    Ttest_indResult(statistic=1.5106055764909945, pvalue=0.17909593083942885)

Congratulations! Hopefully you got the exact same numerical results. This shows that you correctly calculated the numbers by hand. Secondly, you used the correct function and saw that it's much easier to use. All you need to do is pass your data to it.

__Q6:__ Conceptual question: look through the documentation for statistical test functions in scipy.stats. You'll see the above _t_-test for a sample, but can you see an equivalent one for performing a *z*-test from a sample? Comment on your answer.

__A:__ Ztest are used when population parameters are known.  T-test when population parameters are unknown

## Learning outcomes

##### Having completed this project notebook, you now have good hands-on experience:
* using the central limit theorem to help you apply frequentist techniques to answer questions that pertain to very non-normally distributed data from the real world
* performing inference using such data to answer business questions
* forming a hypothesis and framing the null and alternative hypotheses
* testing this using a _t_-test

notebooks/Frequentist Inference Case Study - Part B (2).ipynb


# Part 3: Softmax Regression

```
# Execute this code block to install dependencies when running on colab
try:
    import torch
except:
    from os.path import exists
    from wheel.pep425tags import get_abbr_impl, get_impl_ver, get_abi_tag
    platform = '{}{}-{}'.format(get_abbr_impl(), get_impl_ver(), get_abi_tag())
    cuda_output = !ldconfig -p|grep cudart.so|sed -e 's/.*\.\([0-9]*\)\.\([0-9]*\)$/cu\1\2/'
    accelerator = cuda_output[0] if exists('/dev/nvidia0') else 'cpu'

    !pip install -q http://download.pytorch.org/whl/{accelerator}/torch-1.0.0-{platform}-linux_x86_64.whl torchvision
```

In the second part of the lab we saw how to make a linear binary classifier using logisitic regression. In this  part of the lab we'll turn our attention to multi-class classification.

Softmax regression (or multinomial logistic regression) is a generalisation of logistic regression to the case where we want to handle multiple classes. In logistic regression we assumed that the labels were binary: $y_i\in \{0,1\}$. We used such a classifier to distinguish between two kinds of hand-written digits. Softmax regression allows us to handle $y_i \in \{1,\dots,K\}$ where $K$ is the number of classes.

Recall that in logistic regression, we had a training set $\{(\mathbf{x}_1,y_1),\dots,(\mathbf{x}_m,y_m)\}$ of $m$ labeled examples, where the input features are $\mathbf{x}_i \in \mathbb{R}^n$. In logistic regression, our hypothesis took the form:

\begin{align}
h_\theta(\mathbf{x}) &= \frac{1}{1 + \exp(-\mathbf{x}^\top\theta)} \equiv \sigma(\mathbf{x}^\top\theta)
\end{align}

and the model parameters $\theta$ were trained to minimise the cost function

\begin{align}
J(\theta) & = \sum_i y_i \log(\sigma(\mathbf{x}_i^\top\theta)) + (1-y_i) \log(1-\sigma(\mathbf{x}_i^\top\theta))
\end{align}

In the softmax regression setting, we are interested in multi-class classification, and so the label $y$
 can take on $K$ different values, rather than only two. Thus, in our training set $\{(\mathbf{x}_1,y_1),\dots,(\mathbf{x}_m,y_m)\}$, we now have that $y_i \in \{1,\dots,K\}$.

Given a test input $\mathbf{x}$, we want our hypothesis to estimate the probability that $P(y=k|\mathbf{x})$ for each value of $k=1,\dots,K$. That is to say, we want to estimate the probability of the class label taking on each of the $K$ different possible values. Thus, our hypothesis will output a $K$-dimensional vector (whose elements sum to 1) giving us our $K$ estimated probabilities. Concretely, our hypothesis $h_\theta(\mathbf{x})$ takes the form:

\begin{align}
h_\theta(\mathbf{x}) =
\begin{bmatrix}
P(y = 1 | \mathbf{x}; \theta) \\
P(y = 2 | \mathbf{x}; \theta) \\
\vdots \\
P(y = K | \mathbf{x}; \theta)
\end{bmatrix}
=
\frac{1}{ \sum_{j=1}^{K}{\exp(\theta^{(j)\top} \mathbf{x}) }}
\begin{bmatrix}
\exp(\theta^{(1)\top} \mathbf{x} ) \\
\exp(\theta^{(2)\top} \mathbf{x} ) \\
\vdots \\
\exp(\theta^{(K)\top} \mathbf{x} ) \\
\end{bmatrix}
\end{align}

Here $\theta^{(1)},\theta^{(2)},\dots,\theta^{(K)} \in \mathbb{R}^n$ are the parameters of our model. Notice that the term $\frac{1}{\sum_{j=1}^K exp(\theta^{(j)\top} \mathbf{x})}$ normalizes the distribution, so that it sums to one.

For convenience, we will also write $\theta$ to denote all the parameters of our model. When you implement softmax regression, it is usually convenient to represent $\theta$ as a $n$-by-$K$ matrix obtained by concatenating $\theta_{(1)},\theta^{(2)},\dots,\theta^{(K)}$ into columns, so that

\begin{align}
\theta = \left[\begin{array}{cccc}| & | & | & | \\
\theta^{(1)} & \theta^{(2)} & \cdots & \theta^{(K)} \\
| & | & | & |
\end{array}\right].
\end{align}

## Cost Function

We now describe the cost function that we’ll use for softmax regression. In the equation below, $1\{\cdot\}$
 is an "indicator function", such that $1\{\mathrm{a true statement}\}=1$, and $1\{\mathrm{a false statement}\}=0$. For example, $1\{2+2=4\}$ evaluates to $1$; whereas $1\{1+1=5\}$ evaluates to $0$. Our cost function will be:

\begin{align}
J(\theta) = - \left[ \sum_{i=1}^{m} \sum_{k=1}^{K}  1\left\{y_{i} = k\right\} \log \frac{\exp(\theta^{(k)\top} \mathbf{x}_i)}{\sum_{j=1}^K \exp(\theta^{(j)\top} \mathbf{x}_i)}\right]
\end{align}

Notice that this generalises the logistic regression cost function, which could also have been written:

\begin{align}
J(\theta) &= - \left[ \sum_{i=1}^m   (1-y^{(i)}) \log (1-h_\theta(\mathbf{x}_i)) + y^{(i)} \log h_\theta(\mathbf{x}_i) \right] \\
&= - \left[ \sum_{i=1}^{m} \sum_{k=0}^{1} 1\left\{y^{(i)} = k\right\} \log P(y^{(i)} = k | \mathbf{x}_i ; \theta) \right]
\end{align}

The softmax cost function is similar, except that we now sum over the $K$ different possible values of the class label. Note also that in softmax regression, we have that

\begin{equation}
P(y_i = k | \mathbf{x}_i ; \theta) = \frac{\exp(\theta^{(k)\top} \mathbf{x}_i)}{\sum_{j=1}^K \exp(\theta^{(j)\top} \mathbf{x}_i) }
\end{equation}

We cannot solve for the minimum of $J(\theta)$ analytically, and thus we'll resort to using gradient descent as before. Taking derivatives, one can show that the gradient is:

\begin{align}
\nabla_{\theta^{(k)}} J(\theta) = - \sum_{i=1}^{m}{ \left[ \mathbf{x}_i \left( 1\{ y_i = k\}  - P(y_i = k | \mathbf{x}_i; \theta) \right) \right]  }
\end{align}

Armed with this formula for the derivative, one can then use it directly with a gradient descent solver (or any other 1st-order gradient based optimiser).

__Use the code box below to complete the implementation of the functions that return the gradients of the softmax loss function, $\nabla_{\theta^{(k)}} J(\theta) \,\, \forall k$ and the loss function itself, $J(\theta)$:__

```
import torch
import torch.nn.functional as f

# we wouldn't normally do this, but for this lab we want to work in double precision
# as we'll need the numerical accuracy later on for doing checks on our gradients:
torch.set_default_dtype(torch.float64)

'''
Theta: f*K(20*10)
X: num_items*f(100*20)
y: num_items*1(100*1) value:0~9
out: f*K
'''
def softmax_regression_loss_grad(Theta, X, y):
    '''Implementation of the gradient of the softmax loss function.

    Theta is the matrix of parameters, with the parameters of the k-th class in the k-th column
    X contains the data vectors (one vector per row)
    y is a column vector of the targets
    '''
    # YOUR CODE HERE
    num_classes = Theta.size()[1]  #K
    features_dim = Theta.size()[0]  #f
    num_items = y.size()[0]
    y_label = torch.zeros(num_items,num_classes,dtype=torch.float64).scatter_(1,y,1)  #num_items * K
    prob = f.softmax(X @ Theta, dim=1) #num_items*K, dim=1 for column
    grad = - X.t() @ (y_label - prob)
    return grad

def exp_prob(Theta_k,Theta,x_i):
    sum=0
    for j in range(Theta.shape[1]):  #K
        sum+=torch.exp(x_i @ Theta[:,j])
    prob=torch.div(torch.exp(x_i @ Theta_k),sum)
    return prob

def softmax_regression_loss(Theta, X, y):
    '''Implementation of the softmax loss function.

    Theta is the matrix of parameters, with the parameters of the k-th class in the k-th column
    X contains the data vectors (one vector per row)
    y is a column vector of the targets
    '''
    # YOUR CODE HERE
    loss=0.0
    for i in range(X.shape[0]): #num_items
        x_i=X[i,:]  #the i-th row,1*f
        y_i=y[i]
        class_indicator=0
        for k in range(Theta.shape[1]): #K
            Theta_k = Theta[:,k] #the k-th column,f*1
            class_inidicator = int(y_i.numpy()==k)  #numpy
            prob=exp_prob(Theta_k,Theta,x_i)
            loss += float(class_inidicator) * torch.log(prob)  #float,* seperate
    return -loss  #-

def softmax_regression_loss2(Theta, X, y):
    num_classes = Theta.size()[1]  #K
    features_dim = Theta.size()[0]  #f
    num_items = y.size()[0]
    y_label=torch.zeros(num_items,num_classes,dtype=torch.float64).scatter_(1,y,1)  #num_items * K
    prob=-f.log_softmax(X @ Theta, dim=1)  #num_items * K
    loss=torch.sum(y_label*prob)
    return loss

```

__Use the following code block to confirm that your implementation is correct using gradient checking. If there are problems with your gradient or loss, go back and fix them!:__

```
 # from torch.autograd import gradcheck
from random import randrange

def grad_check(f, x, analytic_grad, num_checks=10, h=1e-3):
    sum_error = 0
    for i in range(num_checks):
        ix = tuple([randrange(m) for m in x.shape]) #randomly sample value to change
        print("ix:",ix)
        print("x[ix]:",x[ix])
        oldval = x[ix].item()
        x[ix] = oldval + h # increment by h
        fxph = f(x) # evaluate f(x + h)
        x[ix] = oldval - h # increment by h
        fxmh = f(x) # evaluate f(x - h)
        x[ix] = oldval # reset

        grad_numerical = (fxph - fxmh) / (2 * h)
        grad_analytic = analytic_grad[ix]
        rel_error = abs(grad_numerical - grad_analytic) / (abs(grad_numerical) + abs(grad_analytic) + 1e-8)
        sum_error += rel_error
        print('numerical: %f\tanalytic: %f\trelative error: %e' % (grad_numerical, grad_analytic, rel_error))
    return sum_error / num_checks

# Create some test data:
num_classes = 10   #K
features_dim = 20  #f
num_items = 100
Theta = torch.randn((features_dim, num_classes)) #features_dim * num_classes(f*K), filled with random numbers from a normal distribution
X = torch.randn((num_items,features_dim))  #num_items * features_dim(num_items*f)
y = torch.torch.randint(0, num_classes, (num_items, 1)) #0~num_classes-1, size:num_items*1

# compute the analytic gradient
grad = softmax_regression_loss_grad(Theta, X, y)

# run the gradient checker
grad_check(lambda th: softmax_regression_loss(th, X, y), Theta, grad)
```

    ix: (7, 3)
    x[ix]: tensor(0.0487)
    numerical: -3.040584	analytic: -3.040584	relative error: 8.887084e-09
    ix: (13, 4)
    x[ix]: tensor(-1.0416)
    numerical: -3.542292	analytic: -3.542292	relative error: 1.051560e-08
    ix: (1, 7)
    x[ix]: tensor(-2.0798)
    numerical: -6.047417	analytic: -6.047417	relative error: 6.418247e-09
    ix: (3, 4)
    x[ix]: tensor(1.8285)
    numerical: 1.309945	analytic: 1.309945	relative error: 7.116945e-08
    ix: (2, 5)
    x[ix]: tensor(1.9467)
    numerical: 14.097420	analytic: 14.097420	relative error: 3.242193e-09
    ix: (13, 0)
    x[ix]: tensor(0.9674)
    numerical: 9.549632	analytic: 9.549632	relative error: 4.199948e-09
    ix: (1, 7)
    x[ix]: tensor(-2.0798)
    numerical: -6.047417	analytic: -6.047417	relative error: 6.418247e-09
    ix: (12, 9)
    x[ix]: tensor(0.0319)
    numerical: 1.614008	analytic: 1.614008	relative error: 1.278455e-08
    ix: (4, 9)
    x[ix]: tensor(0.4010)
    numerical: 0.039318	analytic: 0.039318	relative error: 3.090074e-06
    ix: (6, 5)
    x[ix]: tensor(0.4945)
    numerical: 4.357503	analytic: 4.357503	relative error: 6.102861e-08

    tensor(3.2747e-07)

```
Theta = torch.Tensor([[1, 0], [0, 1]])
X = torch.Tensor([[1, 0], [0, 1]])
y = torch.LongTensor([[0], [1]])
loss=softmax_regression_loss(Theta, X, y)
print("loss:",loss)
assert torch.abs(softmax_regression_loss(Theta, X, y) - 0.6265) < 0.0001
grad = softmax_regression_loss_grad(Theta, X, y)
assert torch.torch.allclose(torch.abs(grad/0.2689), torch.ones_like(grad), atol=0.001)
#This function checks if all self and other satisfy the condition:∣self−other∣≤atol+rtol×∣other∣, rtol=1e-08

```

    loss: tensor(0.6265)

## Training Softmax regression with gradient descent on real data

We'll now try gradient descent with our softmax regression using the digits dataset. As before, when we looked at logistic regression, we load the data and create test and training sets. Note that this time we'll use all the classes:

```
from sklearn.datasets import load_digits

X, y = (torch.Tensor(z) for z in load_digits(10, True)) #convert to pytorch Tensors
X = torch.cat((X, torch.ones((X.shape[0], 1))), 1) # append a column of 1's to the X's
X /= 255
y = y.reshape(-1, 1) # reshape y into a column vector
y = y.type(torch.LongTensor)

# We're also going to break the data into a training set for computing the regression parameters
# and a test set to evaluate the predictive ability of those parameters
perm = torch.randperm(y.shape[0])
X_train = X[perm[0:260], :]  #261*f
y_train = y[perm[0:260]]
X_test = X[perm[260:], :]
y_test = y[perm[260:]]
```

We now define a simple gradient descent loop to train the model:

```
alpha = 0.1
theta_gd = torch.rand((X_train.shape[1], 10)) #Returns a tensor filled with random numbers from a uniform distribution on the interval [0, 1)
#num_items*10

for e in range(0, 1000):
    gr = softmax_regression_loss_grad(theta_gd, X_train, y_train)
    theta_gd -= alpha * gr
    if e%100 == 0:
        print("Training Loss: ", softmax_regression_loss(theta_gd, X_train, y_train))

# Compute the accuracy of the test set
proba = torch.softmax(X_test @ theta_gd, 1)
print(float((proba.argmax(1)-y_test[:,0]==0).sum()) / float(proba.shape[0]))  #argmax(1) returns the indices of the maximum values of prob
print()

```

    Training Loss:  tensor(592.3777)
    Training Loss:  tensor(270.9790)
    Training Loss:  tensor(174.5995)
    Training Loss:  tensor(132.5848)
    Training Loss:  tensor(109.0211)
    Training Loss:  tensor(93.7592)
    Training Loss:  tensor(82.9511)
    Training Loss:  tensor(74.8239)
    Training Loss:  tensor(68.4447)
    Training Loss:  tensor(63.2739)
    0.9232270657124269

Running the above, you should observe that the training loss decreases over time. The final accuracy on the test set is also printed and should be around 90% (it will depend on the particular training/test splits you generated as well as the initial parameters for the softmax).

# Overparameterisation in softmax regression

Softmax regression has an unusual property that it has a "redundant" set of parameters. To explain what this means, suppose we take each of our parameter vectors $\theta^{(j)}$, and subtract some fixed vector $\psi$ from it, so that every $\theta^{(j)}$ is now replaced with $\theta^{(j)}−\psi$ (for every $j=1,\dots,k$). Our hypothesis now estimates the class label probabilities as

\begin{align}
P(y^{(i)} = k | x^{(i)} ; \theta)
&= \frac{\exp((\theta^{(k)}-\psi)^\top x^{(i)})}{\sum_{j=1}^K \exp( (\theta^{(j)}-\psi)^\top x^{(i)})}  \\
&= \frac{\exp(\theta^{(k)\top} x^{(i)}) \exp(-\psi^\top x^{(i)})}{\sum_{j=1}^K \exp(\theta^{(j)\top} x^{(i)}) \exp(-\psi^\top x^{(i)})} \\
&= \frac{\exp(\theta^{(k)\top} x^{(i)})}{\sum_{j=1}^K \exp(\theta^{(j)\top} x^{(i)})}.
\end{align}

__In other words, subtracting $\psi$ from every $\theta^{(j)}$ does not affect our hypothesis’ predictions at all!__ This shows that softmax regression’s parameters are "redundant". More formally, we say that our softmax model is "overparameterised" meaning that for any hypothesis we might fit to the data, there are multiple parameter settings that give rise to exactly the same hypothesis function $h_\theta$ mapping from inputs $\mathbf{x}$ to the predictions.

Further, if the cost function $J(\theta)$ is minimized by some setting of the parameters $(\theta^{(1)},\theta^{(2)},\dots,\theta^{(k)})$, then it is also minimised by $\theta^{(1)}-\psi,\theta^{(2)}-\psi,\dots,\theta^{(k)}-\psi)$ for any value of $\psi$. Thus, the minimiser of $J(\theta)$ is not unique.

(Interestingly, $J(\theta)$ is still convex, and thus gradient descent will not run into local optima problems. The Hessian is however singular/non-invertible, which causes a straightforward implementation of Newton's method (a second-order optimiser) to run into numerical problems.)

Notice also that by setting $\psi=\theta^{(K)}$, one can always replace $\theta^{(K)}$ with $\theta^{(K)}-\psi=\mathbf{0}$ (the vector of all $0$’s), without affecting the hypothesis. Thus, one could "eliminate" the vector of parameters $\theta^{(K)}$ (or any other $\theta^{(k)}$, for any single value of $k$), without harming the representational power of our hypothesis. Indeed, rather than optimising over the $K \cdot n$ parameters $(\theta^{(1)},\theta^{(2)},\dots,\theta^{(k)})$  (where $\theta^{(k)} \in \mathbb{R}^n$, one can instead set $\theta^{(K)}=\mathbf{0}$  and optimize only with respect to the $(K-1) \cdot n$ remaining parameters.

__Use the following block to implement the softmax gradients for the case where the final column of the parameters theta is fixed to be zero:__

```
import torch
import torch.nn.functional as f

'''
Theta: f*K(20*10)
X: num_items*f(100*20)
y: num_items*1(100*1) value:0~9
out: f*K
'''
def softmax_regression_loss_grad_0(Theta, X, y):
    '''Implementation of the gradient of the softmax loss function, with the parameters of the
    last class fixed to be zero.

    Theta is the matrix of parameters, with the parameters of the k-th class in the k-th column;
            K-1 classes are included, and the parameters of the last class are implicitly zero.
    X contains the data vectors (one vector per row)
    y is a column vector of the targets
    '''

    # add the missing column of zeros:
    Theta = torch.cat((Theta, torch.zeros(Theta.shape[0],1)), 1)

    # YOUR CODE HERE
    num_classes = Theta.size()[1]  #(K+1)
    features_dim = Theta.size()[0]  #f
    num_items = y.size()[0]
    y_label = torch.zeros(num_items,num_classes,dtype=torch.float64).scatter_(1,y,1)  #num_items * (K+1)
    prob = f.softmax(X @ Theta, dim=1) #num_items*K, dim=1 for column
    grad = - X.t() @ (y_label - prob)

    # remove the last column from the gradients
    grad = grad[0:grad.shape[0], 0:grad.shape[1]-1]
    return grad
```

```
Theta = torch.Tensor([[1, 0], [0, 0]])
X = torch.Tensor([[1, 0], [0, 1]])
y = torch.LongTensor([[0], [1]])

grad = softmax_regression_loss_grad(Theta, X, y)
grad0 = softmax_regression_loss_grad_0(Theta[:,0:grad.shape[1]-1], X, y)
assert torch.torch.allclose(grad[:,0:grad.shape[1]-1], grad0)

```

Finally, we can run gradient descent with our reduced paramter gradient function, and confirm that the results are similar to before:

```
alpha = 0.1
theta_gd = torch.rand((X_train.shape[1], 9))

for e in range(0, 1000):
    gr = softmax_regression_loss_grad_0(theta_gd, X_train, y_train)
    theta_gd -= alpha * gr

theta_gd = torch.cat((theta_gd, torch.zeros(theta_gd.shape[0], 1)), 1)
proba = torch.softmax(X_test @ theta_gd, 1)
print(float((proba.argmax(1)-y_test[:,0]==0).sum()) / float(proba.shape[0]))
print()
```

    0.9108653220559532

```

```


# Single Particle Model (SPM)

## Model Equations

The SPM consists of two spherically symmetric diffusion equations: one within a representative negative particle ($\text{k}=\text{n}$) and one within a representative positive particle ($\text{k}=\text{p}$). In the centre of the particle the classical no-flux condition is imposed. Since the SPM assumes that all particles in an electrode behave in exactly the same way, the flux on the surface of a particle is simply the current $I$ divided by the thickness of the electrode $L_{\text{k}}$. The concentration of lithium in electrode $\text{k}$ is denoted $c_{\text{k}}$ and the current is denoted by $I$. All parameters in the model stated here are dimensionless and are given in terms of dimensional parameters at the end of this notebook. The model equations for the SPM are then:
\begin{align}
\mathcal{C}_{\text{k}} \frac{\partial c_{\text{s,k}}}{\partial t} &= -\frac{1}{r_{\text{k}}^2} \frac{\partial}{\partial r_{\text{k}}} \left(r_{\text{k}}^2 N_{\text{s,k}}\right), \\
N_{\text{s,k}} &= -D_{\text{s,k}}(c_{\text{s,k}}) \frac{\partial c_{\text{s,k}}}{\partial r_{\text{k}}}, \quad \text{k} \in \text{n, p}, \end{align}
$$
N_{\text{s,k}}\big|_{r_{\text{k}}=0} = 0, \quad \text{k} \in \text{n, p}, \quad \ \ - \frac{a_{\text{k}}\gamma_{\text{k}}}{\mathcal{C}_{\text{k}}} N_{\text{s,k}}\big|_{r_{\text{k}}=1} =
\begin{cases}
		  \frac{I}{L_{\text{n}}}, \quad &\text{k}=\text{n}, \\
		  -\frac{I}{L_{\text{p}}}, \quad &\text{k}=\text{p},
\end{cases} \\
c_{\text{s,k}}(r_{\text{k}},0) = c_{\text{s,k,0}}, \quad \text{k} \in \text{n, p},$$
where $D_{\text{s,k}}$ is the diffusion coefficient in the solid, $N_{\text{s,k}}$ denotes the flux of lithium ions in the solid particle within the region $\text{k}$, and $r_{\text{k}} \in[0,1]$ is the radial coordinate of the particle in electrode $\text{k}$.

### Voltage Expression
The terminal voltage is obtained from the expression:
$$
V = U_{\text{p}}(c_{\text{p}})\big|_{r_{\text{p}}=1} - U_{\text{n}}(c_{\text{n}})\big|_{r_{\text{n}}=1} -2\sinh^{-1}\left(\frac{I}{j_{\text{0,p}} L_{\text{p}}}\right) - 2\sinh^{-1}\left(\frac{I}{j_{\text{0,n}} L_{\text{n}}}\right)
$$
with the exchange current densities given by
$$j_{\text{0,k}} =  \frac{\gamma_{\text{k}}}{\mathcal{C}_{\text{r,k}}}(c_{\text{k}})^{1/2}(1-c_{\text{k}})^{1/2}  $$

## Example solving SPM using PyBaMM

Below we show how to solve the SPM model, using the default geometry, mesh, paramters, discretisation and solver provided with PyBaMM.

First we need to import `pybamm`, and then change our working directory to the root of the pybamm folder.

```python
import pybamm
import numpy as np
import os
import matplotlib.pyplot as plt
os.chdir(pybamm.__path__[0]+'/..')
```

We then get the SPM model equations:

```python
model = pybamm.lithium_ion.SPM()
```

The model object is a subtype of [`pybamm.BaseModel`](https://pybamm.readthedocs.io/en/latest/source/models/base_model.html), and contains all the equations that define this particular model. For example, the `rhs` dict contained in `model` has a dictionary mapping variables such as $c_n$ to the equation representing its rate of change with time (i.e. $\partial{c_n}/\partial{t}$). We can see this explicitly by visualising this entry in the `rhs` dict:

```python
variable = next(iter(model.rhs.keys()))
equation = next(iter(model.rhs.values()))
print('rhs equation for variable \'',variable,'\' is:')
path = 'examples/notebooks/models/'
equation.visualise(path+'spm1.png')
```

    rhs equation for variable ' X-averaged negative particle concentration ' is:

We need a geometry to define our model equations over. In pybamm this is represented by the [`pybamm.Geometry`](https://pybamm.readthedocs.io/en/latest/source/geometry/geometry.html) class. In this case we use the default geometry object defined by the model

```python
geometry = model.default_geometry
```

This geometry object defines a number of geometry domains, each with its own name, spatial variables and min/max limits (the latter are represented as equations similar to the rhs equation shown above). For instance, the SPM model has the following domains:

```python
print('SPM domains:')
for i, (k, v) in enumerate(geometry.items()):
    print(str(i+1)+'.',k,'with variables:')
    for var, rng in v['primary'].items():
        if 'min' in rng:
            print('  -(',rng['min'],') <=',var,'<= (',rng['max'],')')
        else:
            print(var, '=', rng['position'])
```

    SPM domains:
    1. negative electrode with variables:
      -( 0.0 ) <= x_n <= ( Negative electrode thickness [m] / Negative electrode thickness [m] + Separator thickness [m] + Positive electrode thickness [m] )
    2. separator with variables:
      -( Negative electrode thickness [m] / Negative electrode thickness [m] + Separator thickness [m] + Positive electrode thickness [m] ) <= x_s <= ( Negative electrode thickness [m] / Negative electrode thickness [m] + Separator thickness [m] + Positive electrode thickness [m] + Separator thickness [m] / Negative electrode thickness [m] + Separator thickness [m] + Positive electrode thickness [m] )
    3. positive electrode with variables:
      -( Negative electrode thickness [m] / Negative electrode thickness [m] + Separator thickness [m] + Positive electrode thickness [m] + Separator thickness [m] / Negative electrode thickness [m] + Separator thickness [m] + Positive electrode thickness [m] ) <= x_p <= ( 1.0 )
    4. current collector with variables:
    z = 1.0
    5. negative particle with variables:
      -( 0.0 ) <= r_n <= ( 1.0 )
    6. positive particle with variables:
      -( 0.0 ) <= r_p <= ( 1.0 )

Both the model equations and the geometry are defined by a set of parameters, such as $\gamma_p$ or $L_p$. We can substitute these symbolic parameters in the model with values by using the [`pybamm.ParameterValues`](https://pybamm.readthedocs.io/en/latest/source/parameters/parameter_values.html) class, which takes either a python dictionary or CSV file with the mapping between parameter names and values. Rather than create our own instance of `pybamm.ParameterValues`, we will use the default parameter set included in the model

```python
param = model.default_parameter_values
```

We can then apply this parameter set to the model and geometry

```python
param.process_model(model)
param.process_geometry(geometry)
```

The next step is to mesh the input geometry. We can do this using the [`pybamm.Mesh`](https://pybamm.readthedocs.io/en/latest/source/meshes/meshes.html) class. This class takes in the geometry of the problem, and also two dictionaries containing the type of mesh to use within each domain of the geometry (i.e. within the positive or negative electrode domains), and the number of mesh points.

The default mesh types and the default number of points to use in each variable for the SPM model are:

```python
for k, t in model.default_submesh_types.items():
    print(k,'is of type',t.__repr__())
for var, npts in model.default_var_pts.items():
    print(var,'has',npts,'mesh points')
```

    negative electrode is of type Generator for Uniform1DSubMesh
    separator is of type Generator for Uniform1DSubMesh
    positive electrode is of type Generator for Uniform1DSubMesh
    negative particle is of type Generator for Uniform1DSubMesh
    positive particle is of type Generator for Uniform1DSubMesh
    current collector is of type Generator for SubMesh0D
    x_n has 20 mesh points
    x_s has 20 mesh points
    x_p has 20 mesh points
    r_n has 10 mesh points
    r_p has 10 mesh points
    y has 10 mesh points
    z has 10 mesh points

With these defaults, we can then create our mesh of the given geometry:

```python
mesh = pybamm.Mesh(geometry, model.default_submesh_types, model.default_var_pts)
```

The next step is to discretise the model equations over this mesh. We do this using the [`pybamm.Discretisation`](https://pybamm.readthedocs.io/en/latest/source/discretisations/discretisation.html) class, which takes both the mesh we have already created, and a dictionary of spatial methods to use for each geometry domain. For the case of the SPM model, we use the following defaults for the spatial discretisation methods:

```python
for k, method in model.default_spatial_methods.items():
    print(k,'is discretised using',method.__name__,'method')
```

    macroscale is discretised using FiniteVolume method
    negative particle is discretised using FiniteVolume method
    positive particle is discretised using FiniteVolume method
    current collector is discretised using ZeroDimensionalMethod method

We then create the `pybamm.Discretisation` object, and use this to discretise the model equations

```python
disc = pybamm.Discretisation(mesh, model.default_spatial_methods)
disc.process_model(model)
```

    <pybamm.models.full_battery_models.lithium_ion.spm.SPM at 0x7f75886cf198>

After this stage, all the equations in `model` have been discretised into purely linear algebra expressions that are ready to be evaluated within a time-stepping loop of a given solver. For example, the rhs expression for $\partial{c_n}/\partial{t}$ that we visualised above is now represented by:

```python
model.concatenated_rhs.children[0].visualise(path+'spm2.png')
```

Now we are ready to run the time-stepping loop to solve the model. Once again we use the default ODE solver like so:

```python
# Solve the model at the given time points
solver = model.default_solver
n = 250
t_eval = np.linspace(0, 1, n)
print('Solving using',type(solver).__name__,'solver...')
solution = solver.solve(model, t_eval)
print('Finished.')
```

    Solving using ScipySolver solver...
    Finished.

Each model in pybamm has a list of relevent variables defined in the model, for use in visualising the model solution or for comparison with other models. The SPM model defines the following variables:

```python
print('SPM model variables:')
for v in model.variables.keys():
    print('\t-',v)
```

    SPM model variables:
    	- Total current density
    	- Total current density [A.m-2]
    	- Current [A]
    	- Time
    	- Time [s]
    	- Time [min]
    	- Time [h]
    	- Discharge capacity [A.h]
    	- x
    	- x [m]
    	- x_n
    	- x_n [m]
    	- x_s
    	- x_s [m]
    	- x_p
    	- x_p [m]
    	- Negative particle concentration
    	- Positive particle concentration
    	- Negative particle surface concentration
    	- Positive particle surface concentration
    	- Negative particle concentration [mol.m-3]
    	- Positive particle concentration [mol.m-3]
    	- Negative particle surface concentration [mol.m-3]
    	- Positive particle surface concentration [mol.m-3]
    	- r_n
    	- r_n [m]
    	- r_p
    	- r_p [m]
    	- Porosity
    	- Negative electrode porosity
    	- Separator porosity
    	- Positive electrode porosity
    	- X-averaged negative electrode porosity
    	- X-averaged separator porosity
    	- X-averaged positive electrode porosity
    	- Leading-order negative electrode porosity
    	- Leading-order separator porosity
    	- Leading-order positive electrode porosity
    	- Leading-order x-averaged negative electrode porosity
    	- Leading-order x-averaged separator porosity
    	- Leading-order x-averaged positive electrode porosity
    	- Porosity change
    	- Negative electrode porosity change
    	- Separator porosity change
    	- Positive electrode porosity change
    	- X-averaged porosity change
    	- X-averaged negative electrode porosity change
    	- X-averaged separator porosity change
    	- X-averaged positive electrode porosity change
    	- Leading-order x-averaged negative electrode porosity change
    	- Leading-order x-averaged separator porosity change
    	- Leading-order x-averaged positive electrode porosity change
    	- Volume-averaged velocity
    	- Volume-averaged velocity [m.s-1]
    	- Electrolyte pressure
    	- X-averaged negative particle concentration
    	- X-averaged negative particle concentration [mol.m-3]
    	- X-averaged negative particle surface concentration
    	- X-averaged negative particle surface concentration [mol.m-3]
    	- Negative electrode active volume fraction
    	- Negative electrode volume-averaged concentration
    	- Negative electrode volume-averaged concentration [mol.m-3]
    	- Negative electrode average extent of lithiation
    	- X-averaged positive particle concentration
    	- X-averaged positive particle concentration [mol.m-3]
    	- X-averaged positive particle surface concentration
    	- X-averaged positive particle surface concentration [mol.m-3]
    	- Positive electrode active volume fraction
    	- Positive electrode volume-averaged concentration
    	- Positive electrode volume-averaged concentration [mol.m-3]
    	- Positive electrode average extent of lithiation
    	- Electrolyte concentration
    	- Electrolyte concentration [mol.m-3]
    	- Electrolyte concentration [Molar]
    	- X-averaged electrolyte concentration
    	- X-averaged electrolyte concentration [mol.m-3]
    	- X-averaged electrolyte concentration [Molar]
    	- Negative electrolyte concentration
    	- Negative electrolyte concentration [mol.m-3]
    	- Negative electrolyte concentration [Molar]
    	- Separator electrolyte concentration
    	- Separator electrolyte concentration [mol.m-3]
    	- Separator electrolyte concentration [Molar]
    	- Positive electrolyte concentration
    	- Positive electrolyte concentration [mol.m-3]
    	- Positive electrolyte concentration [Molar]
    	- X-averaged negative electrolyte concentration
    	- X-averaged negative electrolyte concentration [mol.m-3]
    	- X-averaged separator electrolyte concentration
    	- X-averaged separator electrolyte concentration [mol.m-3]
    	- X-averaged positive electrolyte concentration
    	- X-averaged positive electrolyte concentration [mol.m-3]
    	- Electrolyte flux
    	- Electrolyte flux [mol.m-2.s-1]
    	- Negative current collector temperature
    	- Negative current collector temperature [K]
    	- X-averaged negative electrode temperature
    	- X-averaged negative electrode temperature [K]
    	- Negative electrode temperature
    	- Negative electrode temperature [K]
    	- X-averaged separator temperature
    	- X-averaged separator temperature [K]
    	- Separator temperature
    	- Separator temperature [K]
    	- X-averaged positive electrode temperature
    	- X-averaged positive electrode temperature [K]
    	- Positive electrode temperature
    	- Positive electrode temperature [K]
    	- Positive current collector temperature
    	- Positive current collector temperature [K]
    	- Cell temperature
    	- Cell temperature [K]
    	- X-averaged cell temperature
    	- X-averaged cell temperature [K]
    	- Volume-averaged cell temperature
    	- Volume-averaged cell temperature [K]
    	- Heat flux
    	- Heat flux [W.m-2]
    	- Negative current collector potential
    	- Negative current collector potential [V]
    	- Current collector current density
    	- Current collector current density [A.m-2]
    	- Leading-order current collector current density
    	- Negative electrode interfacial current density
    	- X-averaged negative electrode interfacial current density
    	- Negative electrode interfacial current density [A.m-2]
    	- X-averaged negative electrode interfacial current density [A.m-2]
    	- Negative electrode interfacial current density per volume [A.m-3]
    	- X-averaged negative electrode interfacial current density per volume [A.m-3]
    	- X-averaged negative electrode total interfacial current density
    	- X-averaged negative electrode total interfacial current density [A.m-2]
    	- X-averaged negative electrode total interfacial current density per volume [A.m-3]
    	- Negative electrode exchange current density
    	- X-averaged negative electrode exchange current density
    	- Negative electrode exchange current density [A.m-2]
    	- X-averaged negative electrode exchange current density [A.m-2]
    	- Negative electrode exchange current density per volume [A.m-3]
    	- X-averaged negative electrode exchange current density per volume [A.m-3]
    	- Negative electrode reaction overpotential
    	- X-averaged negative electrode reaction overpotential
    	- Negative electrode reaction overpotential [V]
    	- X-averaged negative electrode reaction overpotential [V]
    	- Negative electrode surface potential difference
    	- X-averaged negative electrode surface potential difference
    	- Negative electrode surface potential difference [V]
    	- X-averaged negative electrode surface potential difference [V]
    	- Negative electrode open circuit potential
    	- Negative electrode open circuit potential [V]
    	- X-averaged negative electrode open circuit potential
    	- X-averaged negative electrode open circuit potential [V]
    	- Negative electrode entropic change
    	- X-averaged negative electrode entropic change
    	- Positive electrode interfacial current density
    	- X-averaged positive electrode interfacial current density
    	- Positive electrode interfacial current density [A.m-2]
    	- X-averaged positive electrode interfacial current density [A.m-2]
    	- Positive electrode interfacial current density per volume [A.m-3]
    	- X-averaged positive electrode interfacial current density per volume [A.m-3]
    	- X-averaged positive electrode total interfacial current density
    	- X-averaged positive electrode total interfacial current density [A.m-2]
    	- X-averaged positive electrode total interfacial current density per volume [A.m-3]
    	- Positive electrode exchange current density
    	- X-averaged positive electrode exchange current density
    	- Positive electrode exchange current density [A.m-2]
    	- X-averaged positive electrode exchange current density [A.m-2]
    	- Positive electrode exchange current density per volume [A.m-3]
    	- X-averaged positive electrode exchange current density per volume [A.m-3]
    	- Positive electrode reaction overpotential
    	- X-averaged positive electrode reaction overpotential
    	- Positive electrode reaction overpotential [V]
    	- X-averaged positive electrode reaction overpotential [V]
    	- Positive electrode surface potential difference
    	- X-averaged positive electrode surface potential difference
    	- Positive electrode surface potential difference [V]
    	- X-averaged positive electrode surface potential difference [V]
    	- Positive electrode open circuit potential
    	- Positive electrode open circuit potential [V]
    	- X-averaged positive electrode open circuit potential
    	- X-averaged positive electrode open circuit potential [V]
    	- Positive electrode entropic change
    	- X-averaged positive electrode entropic change
    	- Interfacial current density
    	- Interfacial current density [A.m-2]
    	- Interfacial current density per volume [A.m-3]
    	- Exchange current density
    	- Exchange current density [A.m-2]
    	- Exchange current density per volume [A.m-3]
    	- Negative particle flux
    	- X-averaged negative particle flux
    	- Positive particle flux
    	- X-averaged positive particle flux
    	- Negative electrode potential
    	- Negative electrode potential [V]
    	- X-averaged negative electrode potential
    	- X-averaged negative electrode potential [V]
    	- Negative electrode ohmic losses
    	- Negative electrode ohmic losses [V]
    	- X-averaged negative electrode ohmic losses
    	- X-averaged negative electrode ohmic losses [V]
    	- Gradient of negative electrode potential
    	- Negative electrode current density
    	- Negative electrode current density [A.m-2]
    	- Negative electrolyte potential
    	- Negative electrolyte potential [V]
    	- Separator electrolyte potential
    	- Separator electrolyte potential [V]
    	- Positive electrolyte potential
    	- Positive electrolyte potential [V]
    	- Electrolyte potential
    	- Electrolyte potential [V]
    	- X-averaged electrolyte potential
    	- X-averaged electrolyte potential [V]
    	- X-averaged negative electrolyte potential
    	- X-averaged negative electrolyte potential [V]
    	- X-averaged separator electrolyte potential
    	- X-averaged separator electrolyte potential [V]
    	- X-averaged positive electrolyte potential
    	- X-averaged positive electrolyte potential [V]
    	- X-averaged electrolyte overpotential
    	- X-averaged electrolyte overpotential [V]
    	- Gradient of negative electrolyte potential
    	- Gradient of separator electrolyte potential
    	- Gradient of positive electrolyte potential
    	- Gradient of electrolyte potential
    	- Electrolyte current density
    	- Electrolyte current density [A.m-2]
    	- Negative electrolyte current density
    	- Negative electrolyte current density [V]
    	- Positive electrolyte current density
    	- Positive electrolyte current density [V]
    	- X-averaged concentration overpotential
    	- X-averaged electrolyte ohmic losses
    	- X-averaged concentration overpotential [V]
    	- X-averaged electrolyte ohmic losses [V]
    	- Positive electrode potential
    	- Positive electrode potential [V]
    	- X-averaged positive electrode potential
    	- X-averaged positive electrode potential [V]
    	- Positive electrode ohmic losses
    	- Positive electrode ohmic losses [V]
    	- X-averaged positive electrode ohmic losses
    	- X-averaged positive electrode ohmic losses [V]
    	- Gradient of positive electrode potential
    	- Positive electrode current density
    	- Positive electrode current density [A.m-2]
    	- Electrode current density
    	- Positive current collector potential
    	- Local current collector potential difference
    	- Local current collector potential difference [V]
    	- Ohmic heating
    	- Ohmic heating [A.V.m-3]
    	- Irreversible electrochemical heating
    	- Irreversible electrochemical heating [A.V.m-3]
    	- Reversible heating
    	- Reversible heating [A.V.m-3]
    	- Total heating
    	- Total heating [A.V.m-3]
    	- X-averaged total heating
    	- X-averaged total heating [A.V.m-3]
    	- Volume-averaged total heating
    	- Volume-averaged total heating [A.V.m-3]
    	- Positive current collector potential [V]
    	- Local potential difference
    	- Local potential difference [V]
    	- X-averaged open circuit voltage
    	- Measured open circuit voltage
    	- X-averaged open circuit voltage [V]
    	- Measured open circuit voltage [V]
    	- X-averaged reaction overpotential
    	- X-averaged reaction overpotential [V]
    	- X-averaged solid phase ohmic losses
    	- X-averaged solid phase ohmic losses [V]
    	- Terminal voltage
    	- Terminal voltage [V]
    	- X-averaged battery open circuit voltage [V]
    	- Measured battery open circuit voltage [V]
    	- X-averaged battery reaction overpotential [V]
    	- X-averaged battery solid phase ohmic losses [V]
    	- X-averaged battery electrolyte ohmic losses [V]
    	- X-averaged battery concentration overpotential [V]
    	- Battery voltage [V]

To help visualise the results, pybamm provides the `pybamm.ProcessedVariable` class, which takes the output of a solver and a variable, and allows the user to evaluate the value of that variable at any given time or $x$ value. For example, we can create a `pybamm.ProcessedVariable` using the SPM voltage variable, and plot the voltage versus time, along with the surface particle concentration.

```python
voltage = pybamm.ProcessedVariable(model.variables['Terminal voltage [V]'], solution.t, solution.y, mesh=mesh)#
c_s_n_surf = pybamm.ProcessedVariable(
    model.variables['Negative particle surface concentration'], solution.t, solution.y, mesh=mesh
)
c_s_p_surf = pybamm.ProcessedVariable(
    model.variables['Positive particle surface concentration'], solution.t, solution.y, mesh=mesh
)
```

One we have these variables in hand, we can begin generating plots using a library such as Matplotlib. Below we plot the terminal voltage and surface particle concentrations versus time

```python
t = np.linspace(0,1,250)
f, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize=(13,4))

ax1.plot(solution.t, voltage(solution.t))
ax1.set_xlabel(r'$t$')
ax1.set_ylabel('Terminal voltage')

ax2.plot(solution.t, c_s_n_surf(t=solution.t, x=0))
ax2.set_xlabel(r'$t$')
ax2.set_ylabel('Negative particle surface concentration')

ax3.plot(solution.t, c_s_p_surf(t=solution.t, x=1))
ax3.set_xlabel(r'$t$')
ax3.set_ylabel('Positive particle surface concentration')

plt.tight_layout()
plt.show()
```

Some of the output variables are defined over space as well as time. Once option to visualise these variables is to use the `interact` slider widget. Below we plot the negative/positive particle concentration over $r$, using a slider to change the current time point

```python
c_s_n = pybamm.ProcessedVariable(model.variables['Negative particle concentration'], solution.t, solution.y, mesh=mesh)
c_s_p = pybamm.ProcessedVariable(model.variables['Positive particle concentration'], solution.t, solution.y, mesh=mesh)
r = np.linspace(0,1,100)

def plot_concentrations(t):
    f, (ax1, ax2) = plt.subplots(1, 2 ,figsize=(10,5))
    plot_c_n, = ax1.plot(r, c_s_n_surf(x=r,t=t))
    plot_c_p, = ax2.plot(r, c_s_p_surf(x=r,t=t))
    ax1.set_ylabel('Negative particle concentration')
    ax2.set_ylabel('Positive particle concentration')
    ax1.set_xlabel(r'$r_n$')
    ax2.set_xlabel(r'$r_p$')
    ax1.set_ylim(0, 1)
    ax2.set_ylim(0, 1)
    plt.show()

import ipywidgets as widgets
widgets.interact(plot_concentrations, t=widgets.FloatSlider(min=0,max=0.2,step=0.01,value=0));

```

    interactive(children=(FloatSlider(value=0.0, description='t', max=0.2, step=0.01), Output()), _dom_classes=('w…

The QuickPlot class can be used to plot the common set of useful outputs which should give you a good initial overview of the model. The method Quickplot.plot(t) is simply a function like plot_concentrations(t) above. We can therefore either use it statically for a particularl t or employ the slider widget.

```python
quick_plot = pybamm.QuickPlot(model, mesh, solution)
widgets.interact(quick_plot.plot, t=widgets.FloatSlider(min=0,max=quick_plot.max_t,step=0.05,value=0));
```

    interactive(children=(FloatSlider(value=0.0, description='t', max=0.6863442760242967, step=0.05), Output()), _…

## Dimensionless Parameters

In the table below, we provide the dimensionless parameters in the SPM in terms of the dimensional parameters in LCO.csv. We use a superscript * to indicate dimensional quanitities.

| Parameter                 | Expression                              |Interpretation                             |
|:--------------------------|:----------------------------------------|:------------------------------------------|
| $L_{\text{k}}$            | $L_{\text{k}}^*/L^*$                    | Ratio of region thickness to cell thickness|
|$\mathcal{C}_{\text{k}}$   | $\tau_{\text{k}}^*/\tau_{\text{d}}^*$   | Ratio of solid diffusion and discharge timescales |
|$\mathcal{C}_{\text{r,k}}$ |$\tau_{\text{r,k}}^*/\tau_{\text{d}}^*$  |Ratio of reaction and discharge timescales|
|$a_{\text{k}}$             |$a_{\text{k}}^* R_{\text{k}}^*$          | Product of particle radius and surface area density|
|$\gamma_{\text{k}}$        |$c_{\text{k,max}}^*/c_{\text{n,max}}^*$  |Ratio of maximum lithium concentrations in solid|

```python

```


Solution to: [Day 6: The Central Limit Theorem III](https://www.hackerrank.com/challenges/s10-the-central-limit-theorem-3/problem)

Table of Contents

- Table of Contents
- Notes
- Solution
    - Imports
    - Inputs
    - Z Score
    - Format
    - Main

```javascript
%%javascript
$.getScript('https://kmahelona.github.io/ipython_notebook_goodies/ipython_notebook_toc.js')
```

    <IPython.core.display.Javascript object>

# Notes
This solution uses the following z-score formula:
\begin{equation}
\large
z = \frac
{x - \mu}
{\sigma}
\end{equation}

Because the problem gives us the z-score and asks us to find the bounds, we can solve the formula for x:

\begin{equation}
\large
x = z * \sigma + \mu
\end{equation}

We can assume:
- mean_sample = mean_population, and
- sd_sample = sd_population / sqrt(n)

*NOTE*: When extrapolating sample statistics to population statistics, the following are derived from the Central Limit Theorem:
1. mean_sample = mean_population
2. standard error (population) = sd_sample / sqrt(n)
    As the sample increases, means cluster more closely around the mean

# Solution

## Imports

```python
from typing import Tuple
```

## Inputs

```python
def get_input() -> Tuple[int, int, int, float, float]:
	"""Returns input for Day 6: Central Limit Theorem III.

	Returns:
		Tuple[int, int, int, float, float]: num_students, mean, sd, distribution percent, z-score
	"""
	return 100, 500, 80, 0.95, 1.96
```

## Z Score

```python
def calc_std_err(sd: int, n: int) -> float:
	"""Returns standard error of population

	Args:
		sd (int): Standard deviation
		n (int): Sample

	Returns:
		float: Standard error
	"""
	return sd / (n ** (1/2))
```

```python
def calc_val_from_zscore(z_score: float, mean: int, sd: int) -> float:
	"""Returns value associated with the z-score from sample mean & sd.

	Args:
		z_score (float): z-score
		mean (int): sample mean
		sd (int): sample sd

	Returns:
		float: value assocaited with z-score given sample mean & sd
	"""
	return z_score * sd + mean
```

## Format

```python
def print_to_scale(num: int) -> None:
	"""Prints number to 2 decimal point scale

	Args:
		num (int): Number to print
	"""
	print(f"{num :.2f}")
```

## Main

```python
def main():
	num_students, mean, sd, dist_percent, z_score = get_input()
	std_err = calc_std_err(sd, num_students)

	## Bottom bound of dist_percent
	a = calc_val_from_zscore(-z_score, mean, std_err)
	## Top bound of dist_percent
	b = calc_val_from_zscore(z_score, mean, std_err)

	## Print
	print_to_scale(a)
	print_to_scale(b)
```

```python
if __name__ == "__main__":
	main()
```

    484.32
    515.68


# Frequentist Inference Case Study - Part A

## 1. Learning objectives

Welcome to part A of the Frequentist inference case study! The purpose of this case study is to help you apply the concepts associated with Frequentist inference in Python. Frequentist inference is the process of deriving conclusions about an underlying distribution via the observation of data. In particular, you'll practice writing Python code to apply the following statistical concepts:
* the _z_-statistic
* the _t_-statistic
* the difference and relationship between the two
* the Central Limit Theorem, including its assumptions and consequences
* how to estimate the population mean and standard deviation from a sample
* the concept of a sampling distribution of a test statistic, particularly for the mean
* how to combine these concepts to calculate a confidence interval

## Prerequisites

To be able to complete this notebook, you are expected to have a basic understanding of:
* what a random variable is (p.400 of Professor Spiegelhalter's *The Art of Statistics, hereinafter AoS*)
* what a population, and a population distribution, are (p. 397 of *AoS*)
* a high-level sense of what the normal distribution is (p. 394 of *AoS*)
* what the t-statistic is (p. 275 of *AoS*)

Happily, these should all be concepts with which you are reasonably familiar after having read ten chapters of Professor Spiegelhalter's book, *The Art of Statistics*.

We'll try to relate the concepts in this case study back to page numbers in *The Art of Statistics* so that you can focus on the Python aspects of this case study. The second part (part B) of this case study will involve another, more real-world application of these tools.

For this notebook, we will use data sampled from a known normal distribution. This allows us to compare our results with theoretical expectations.

## 2. An introduction to sampling from the normal distribution

First, let's explore the ways we can generate the normal distribution. While there's a fair amount of interest in [sklearn](https://scikit-learn.org/stable/) within the machine learning community, you're likely to have heard of [scipy](https://docs.scipy.org/doc/scipy-0.15.1/reference/index.html) if you're coming from the sciences. For this assignment, you'll use [scipy.stats](https://docs.scipy.org/doc/scipy-0.15.1/reference/tutorial/stats.html) to complete your work.

This assignment will require some digging around and getting your hands dirty (your learning is maximized that way)! You should have the research skills and the tenacity to do these tasks independently, but if you struggle, reach out to your immediate community and your mentor for help.

```python
import math
from scipy.stats import norm
from scipy.stats import t
import numpy as np
import pandas as pd
from numpy.random import seed
import matplotlib.pyplot as plt
```

__Q1:__ Call up the documentation for the `norm` function imported above. (Hint: that documentation is [here](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.norm.html)). What is the second listed method?

**Probability density function** - pdf(x, loc=0, scale=1)

__Q2:__ Use the method that generates random variates to draw five samples from the standard normal distribution.

__A:__See below

```python
seed(47)
# draw five samples here

# Initialize random numbers: random_numbers
random_numbers = np.empty(5)

for i in range(5):
    random_numbers[i] = np.random.normal()
```

__Q3:__ What is the mean of this sample? Is it exactly equal to the value you expected? Hint: the sample was drawn from the standard normal distribution. If you want a reminder of the properties of this distribution, check out p. 85 of *AoS*.

See below__A:__See below

```python
# Calculate and print the mean here, hint: use np.mean()
print("The mean of this sample is: {0:f}".format(np.mean(random_numbers)))
```

    The mean of this sample is: 0.193556

__Q4:__ What is the standard deviation of these numbers? Calculate this manually here as $\sqrt{\frac{\sum_i(x_i - \bar{x})^2}{n}}$ (This is just the definition of **standard deviation** given by Professor Spiegelhalter on p.403 of *AoS*). Hint: np.sqrt() and np.sum() will be useful here and remember that numPy supports [broadcasting](https://docs.scipy.org/doc/numpy/user/basics.broadcasting.html).

__A:__ Using random.normal()

```python
# Array of differences to mean: differences
differences = random_numbers - np.mean(random_numbers)

# Square the differences: diff_sq
diff_sq = differences*differences

# Compute the mean square difference: variance_explicit
variance_explicit = np.mean(diff_sq)

# Print the standard deviation as sqrt of variance_explicit
print("The sample standard deviation of these numbers is: {0:f}".format(np.sqrt(variance_explicit)))
```

    The sample standard deviation of these numbers is: 0.960620

Here we have calculated the actual standard deviation of a small data set (of size 5). But in this case, this small data set is actually a sample from our larger (infinite) population. In this case, the population is infinite because we could keep drawing our normal random variates until our computers die!

In general, the sample mean we calculate will not be equal to the population mean (as we saw above). A consequence of this is that the sum of squares of the deviations from the _population_ mean will be bigger than the sum of squares of the deviations from the _sample_ mean. In other words, the sum of squares of the deviations from the _sample_ mean is too small to give an unbiased estimate of the _population_ variance. An example of this effect is given [here](https://en.wikipedia.org/wiki/Bessel%27s_correction#Source_of_bias). Scaling our estimate of the variance by the factor $n/(n-1)$ gives an unbiased estimator of the population variance. This factor is known as [Bessel's correction](https://en.wikipedia.org/wiki/Bessel%27s_correction). The consequence of this is that the $n$ in the denominator is replaced by $n-1$.

You can see Bessel's correction reflected in Professor Spiegelhalter's definition of **variance** on p. 405 of *AoS*.

__Q5:__ If all we had to go on was our five samples, what would be our best estimate of the population standard deviation? Use Bessel's correction ($n-1$ in the denominator), thus $\sqrt{\frac{\sum_i(x_i - \bar{x})^2}{n-1}}$.

__A:__

```python
def pop_variance(data):
    n = len(data)
    mean = sum(data) / n
    return sum((x - mean) ** 2 for x in data) / (n - 1)

def pop_stdev(data):
    pop_var = pop_variance(data)
    pop_std_dev = math.sqrt(pop_var)
    return pop_std_dev
```

__Q6:__ Now use numpy's std function to calculate the standard deviation of our random samples. Which of the above standard deviations did it return?

__A:__ See below

```python
print("The population standard deviation of these numbers is: {0:f}".format(pop_stdev(random_numbers)))
```

    The population standard deviation of these numbers is: 1.074005

**Observation** :The sum of squares of the distance from samples to the population mean (**1.074**) will always be bigger than the sum of squares of the distance to the sample mean (**0.96**)

__Q7:__ Consult the documentation for np.std() to see how to apply the correction for estimating the population parameter and verify this produces the expected result.

__A:__See below

```python
# if N-1 is the denominator, divisor N-ddof is used instead, so ddof = 1 i.e. Degree of freedom for estimating population
# parameter is supposed to be 1

print("The population standard deviation of these numbers as calculated using np.std() is: {0:f}".format(np.std(random_numbers, ddof = 1)))
```

    The population standard deviation of these numbers as calculated using np.std() is: 1.074005

**Conclusion**: Both calcualtions of standard deviation estimation for the population match

### Summary of section

In this section, you've been introduced to the scipy.stats package and used it to draw a small sample from the standard normal distribution. You've calculated the average (the mean) of this sample and seen that this is not exactly equal to the expected population parameter (which we know because we're generating the random variates from a specific, known distribution). You've been introduced to two ways of calculating the standard deviation; one uses $n$ in the denominator and the other uses $n-1$ (Bessel's correction). You've also seen which of these calculations np.std() performs by default and how to get it to generate the other.

You use $n$ as the denominator if you want to calculate the standard deviation of a sequence of numbers. You use $n-1$ if you are using this sequence of numbers to estimate the population parameter. This brings us to some terminology that can be a little confusing.

The population parameter is traditionally written as $\sigma$ and the sample statistic as $s$. Rather unhelpfully, $s$ is also called the sample standard deviation (using $n-1$) whereas the standard deviation of the sample uses $n$. That's right, we have the sample standard deviation and the standard deviation of the sample and they're not the same thing!

The sample standard deviation
\begin{equation}
s = \sqrt{\frac{\sum_i(x_i - \bar{x})^2}{n-1}} \approx \sigma,
\end{equation}
is our best (unbiased) estimate of the population parameter ($\sigma$).

If your dataset _is_ your entire population, you simply want to calculate the population parameter, $\sigma$, via
\begin{equation}
\sigma = \sqrt{\frac{\sum_i(x_i - \bar{x})^2}{n}}
\end{equation}
as you have complete, full knowledge of your population. In other words, your sample _is_ your population. It's worth noting that we're dealing with what Professor Spiegehalter describes on p. 92 of *AoS* as a **metaphorical population**: we have all the data, and we act as if the data-point is taken from a population at random. We can think of this population as an imaginary space of possibilities.

If, however, you have sampled _from_ your population, you only have partial knowledge of the state of your population. In this case, the standard deviation of your sample is not an unbiased estimate of the standard deviation of the population, in which case you seek to estimate that population parameter via the sample standard deviation, which uses the $n-1$ denominator.

Great work so far! Now let's dive deeper.

## 3. Sampling distributions

So far we've been dealing with the concept of taking a sample from a population to infer the population parameters. One statistic we calculated for a sample was the mean. As our samples will be expected to vary from one draw to another, so will our sample statistics. If we were to perform repeat draws of size $n$ and calculate the mean of each, we would expect to obtain a distribution of values. This is the sampling distribution of the mean. **The Central Limit Theorem (CLT)** tells us that such a distribution will approach a normal distribution as $n$ increases (the intuitions behind the CLT are covered in full on p. 236 of *AoS*). For the sampling distribution of the mean, the standard deviation of this distribution is given by

\begin{equation}
\sigma_{mean} = \frac{\sigma}{\sqrt n}
\end{equation}

where $\sigma_{mean}$ is the standard deviation of the sampling distribution of the mean and $\sigma$ is the standard deviation of the population (the population parameter).

This is important because typically we are dealing with samples from populations and all we know about the population is what we see in the sample. From this sample, we want to make inferences about the population. We may do this, for example, by looking at the histogram of the values and by calculating the mean and standard deviation (as estimates of the population parameters), and so we are intrinsically interested in how these quantities vary across samples.

In other words, now that we've taken one sample of size $n$ and made some claims about the general population, what if we were to take another sample of size $n$? Would we get the same result? Would we make the same claims about the general population? This brings us to a fundamental question: _when we make some inference about a population based on our sample, how confident can we be that we've got it 'right'?_

We need to think about **estimates and confidence intervals**: those concepts covered in Chapter 7, p. 189, of *AoS*.

Now, the standard normal distribution (with its variance equal to its standard deviation of one) would not be a great illustration of a key point. Instead, let's imagine we live in a town of 50,000 people and we know the height of everyone in this town. We will have 50,000 numbers that tell us everything about our population. We'll simulate these numbers now and put ourselves in one particular town, called 'town 47', where the population mean height is 172 cm and population standard deviation is 5 cm.

```python
seed(47)
pop_heights = norm.rvs(172, 5, size=50000)
```

```python
_ = plt.hist(pop_heights, bins=30)
_ = plt.xlabel('height (cm)')
_ = plt.ylabel('number of people')
_ = plt.title('Distribution of heights in entire town population')
_ = plt.axvline(172, color='r')
_ = plt.axvline(172+5, color='r', linestyle='--')
_ = plt.axvline(172-5, color='r', linestyle='--')
_ = plt.axvline(172+10, color='r', linestyle='-.')
_ = plt.axvline(172-10, color='r', linestyle='-.')
```

Now, 50,000 people is rather a lot to chase after with a tape measure. If all you want to know is the average height of the townsfolk, then can you just go out and measure a sample to get a pretty good estimate of the average height?

```python
def townsfolk_sampler(n):
    return np.random.choice(pop_heights, n)
```

Let's say you go out one day and randomly sample 10 people to measure.

```python
seed(47)
daily_sample1 = townsfolk_sampler(10)
```

```python
_ = plt.hist(daily_sample1, bins=10)
_ = plt.xlabel('height (cm)')
_ = plt.ylabel('number of people')
_ = plt.title('Distribution of heights in sample size 10')
```

The sample distribution doesn't resemble what we take the population distribution to be. What do we get for the mean?

```python
np.mean(daily_sample1)
```

    173.47911444163503

And if we went out and repeated this experiment?

```python
daily_sample2 = townsfolk_sampler(10)
```

```python
np.mean(daily_sample2)
```

    173.7317666636263

__Q8:__ Simulate performing this random trial every day for a year, calculating the mean of each daily sample of 10, and plot the resultant sampling distribution of the mean.

__A:__See below

```python
n = 365
avg = []

for i in range(1,n):
    a = np.random.choice(townsfolk_sampler(10))
    avg.append(np.mean(a))
```

```python
plt.hist(avg[0:], 20)
plt.axvline(pop_heights.mean(), color = 'C1')
```

The above is the distribution of the means of samples of size 10 taken from our population. The Central Limit Theorem tells us the expected mean of this distribution will be equal to the population mean, and standard deviation will be $\sigma / \sqrt n$, which, in this case, should be approximately 1.58.

__Q9:__ Verify the above results from the CLT.

__A:__Verify the above histogram results with the population mean and standard deviation of the population (computed below)

```python
print("The population standard deviation of these numbers is: {0:f}".format(np.mean(pop_heights)))
```

    The population standard deviation of these numbers is: 172.019260

```python
print("The population standard deviation of these numbers is: {0:f}".format(pop_stdev(pop_heights)/np.sqrt(10)))
```

    The population standard deviation of these numbers is: 1.578258

Remember, in this instance, we knew our population parameters, that the average height really is 172 cm and the standard deviation is 5 cm, and we see some of our daily estimates of the population mean were as low as around 168 and some as high as 176.

__Q10:__ Repeat the above year's worth of samples but for a sample size of 50 (perhaps you had a bigger budget for conducting surveys that year)! Would you expect your distribution of sample means to be wider (more variable) or narrower (more consistent)? Compare your resultant summary statistics to those predicted by the CLT.

__A:__ AS you can see below, as the sample size increases, the sample mean tends to get closer to the population mean

```python
seed(47)
# calculate daily means from the larger sample size here
daily_sample50 = townsfolk_sampler(50)

_ = plt.hist(daily_sample1, bins=10)
_ = plt.xlabel('height (cm)')
_ = plt.ylabel('number of people')
_ = plt.title('Distribution of heights in sample size 50')
```

```python
np.mean(daily_sample50)
```

    172.7815108576788

```python
daily_sample200 = townsfolk_sampler(200)
np.mean(daily_sample200)
```

    172.5599929095019

What we've seen so far, then, is that we can estimate population parameters from a sample from the population, and that samples have their own distributions. Furthermore, the larger the sample size, the narrower are those sampling distributions.

### Normally testing time!

All of the above is well and good. We've been sampling from a population we know is normally distributed, we've come to understand when to use $n$ and when to use $n-1$ in the denominator to calculate the spread of a distribution, and we've  seen the Central Limit Theorem in action for a sampling distribution. All seems very well behaved in Frequentist land. But, well, why should we really care?

Remember, we rarely (if ever) actually know our population parameters but we still have to estimate them somehow. If we want to make inferences to conclusions like "this observation is unusual" or "my population mean has changed" then we need to have some idea of what the underlying distribution is so we can calculate relevant probabilities. In frequentist inference, we use the formulae above to deduce these population parameters. Take a moment in the next part of this assignment to refresh your understanding of how these probabilities work.

Recall some basic properties of the standard normal distribution, such as that about 68% of observations are within plus or minus 1 standard deviation of the mean. Check out the precise definition of a normal distribution on p. 394 of *AoS*.

__Q11:__ Using this fact, calculate the probability of observing the value 1 or less in a single observation from the standard normal distribution. Hint: you may find it helpful to sketch the standard normal distribution (the familiar bell shape) and mark the number of standard deviations from the mean on the x-axis and shade the regions of the curve that contain certain percentages of the population.

__A:__See below

Calculating this probability involved calculating the area under the curve from the value of 1 and below. To put it in mathematical terms, we need to *integrate* the probability density function. We could just add together the known areas of chunks (from -Inf to 0 and then 0 to $+\sigma$ in the example above). One way to do this is to look up tables (literally). Fortunately, scipy has this functionality built in with the cdf() function.

```python
def ecdf(data):
    """Compute ECDF for a one-dimensional array of measurements."""
    # Number of data points: n
    n = len(data)

    # x-data for the ECDF: x
    x = np.sort(data)

    # y-data for the ECDF: y
    y = np.arange(1, n+1) / n

    return x, y
```

```python
mu = np.mean(pop_heights)
sigma = np.std(pop_heights)

samples_std1 = np.random.normal(mu, 1*sigma, size = n)
samples_std2 = np.random.normal(mu, 2*sigma, size = n)
samples_std3 = np.random.normal(mu, 3*sigma, size = n)

# Generate CDFs
x_std1, y_std1 = ecdf(samples_std1)
x_std2, y_std2 = ecdf(samples_std2)
x_std3, y_std3 = ecdf(samples_std3)

# Plot CDFs
_ = plt.plot(x_std1, y_std1, marker = '.', color='r', linestyle = '-')
_ = plt.plot(x_std2, y_std2, marker = '.', color='b', linestyle = '-')
_ = plt.plot(x_std3, y_std3, marker = '.', color='g', linestyle = '-')
_ = plt.axhline(norm.cdf(1, loc=0,scale=1), color='r', linewidth=0.5)

# Make a legend and show the plot
_ = plt.legend(('std = 1', 'std = 2', 'std = 3'), loc='lower right')
plt.show()
```

__Q12:__ Use the cdf() function to answer the question above again and verify you get the same answer.

__A:__

??

__Q13:__ Using our knowledge of the population parameters for our townsfolks' heights, what is the probability of selecting one person at random and their height being 177 cm or less? Calculate this using both of the approaches given above.

__A:__

```python
mu = np.mean(pop_heights)
sigma = np.std(pop_heights)
norm.cdf(177, mu, sigma)
```

    0.8408546025640735

__Q14:__ Turning this question around — suppose we randomly pick one person and measure their height and find they are 2.00 m tall. How surprised should we be at this result, given what we know about the population distribution? In other words, how likely would it be to obtain a value at least as extreme as this? Express this as a probability.

__A:__

```python
1 - norm.cdf(200, mu, sigma)
```

    1.0327726340797483e-08

What we've just done is calculate the ***p-value*** of the observation of someone 2.00m tall (review *p*-values if you need to on p. 399 of *AoS*). We could calculate this probability by virtue of knowing the population parameters. We were then able to use the known properties of the relevant normal distribution to calculate the probability of observing a value at least as extreme as our test value.

We're about to come to a pinch, though. We've said a couple of times that we rarely, if ever, know the true population parameters; we have to estimate them from our sample and we cannot even begin to estimate the standard deviation from a single observation.

This is very true and usually we have sample sizes larger than one. This means we can calculate the mean of the sample as our best estimate of the population mean and the standard deviation as our best estimate of the population standard deviation.

In other words, we are now coming to deal with the sampling distributions we mentioned above as we are generally concerned with the properties of the sample means we obtain.

Above, we highlighted one result from the CLT, whereby the sampling distribution (of the mean) becomes narrower and narrower with the square root of the sample size. We remind ourselves that another result from the CLT is that _even if the underlying population distribution is not normal, the sampling distribution will tend to become normal with sufficiently large sample size_. (**Check out p. 199 of AoS if you need to revise this**). This is the key driver for us 'requiring' a certain sample size, for example you may frequently see a minimum sample size of 30 stated in many places. In reality this is simply a rule of thumb; if the underlying distribution is approximately normal then your sampling distribution will already be pretty normal, but if the underlying distribution is heavily skewed then you'd want to increase your sample size.

__Q15:__ Let's now start from the position of knowing nothing about the heights of people in our town.
* Use the random seed of 47, to randomly sample the heights of 50 townsfolk
* Estimate the population mean using np.mean
* Estimate the population standard deviation using np.std (remember which denominator to use!)
* Calculate the (95%) [margin of error](https://www.statisticshowto.datasciencecentral.com/probability-and-statistics/hypothesis-testing/margin-of-error/#WhatMofE) (use the exact critial z value to 2 decimal places - [look this up](https://www.statisticshowto.datasciencecentral.com/probability-and-statistics/find-critical-values/) or use norm.ppf()) Recall that the ***margin of error*** is mentioned on p. 189 of the *AoS* and discussed in depth in that chapter).
* Calculate the 95% Confidence Interval of the mean (***confidence intervals*** are defined on p. 385 of *AoS*)
* Does this interval include the true population mean?

__A:__

```python
seed(47)
# take your sample now
sample_50 = townsfolk_sampler(50)
```

```python
# Calculate the mean
pop_mean = np.mean(sample_50)
pop_mean
print("The population mean is: {0:f}".format(pop_mean))
```

    The population mean is: 172.781511

```python
pop_sigma = np.std(sample_50, ddof = 1)
pop_sigma
print("The population standard deviation of these numbers as calculated using np.std() is: {0:f}".format(pop_sigma))
```

    The population standard deviation of these numbers as calculated using np.std() is: 4.195424

```python
z_score = norm.ppf((1 + 0.95) / 2)
z_score
```

    1.959963984540054

```python
# Calculate the standard error
std_error = pop_sigma / np.sqrt(len(sample_50))
std_error
```

    0.5933226036092445

```python
# Margin of Error = z-score * standard error
moe = z_score * std_error
moe
```

    1.1628909342876539

```python
# Calculate 95% confidence interval for the mean
[pop_mean - moe, pop_mean + moe]
```

    [171.61861992339115, 173.94440179196644]

__Q16:__ Above, we calculated the confidence interval using the critical z value. What is the problem with this? What requirement, or requirements, are we (strictly) failing?

__A:__
If we want to calculate confidence interval for a sample size that is smaller or equal to 30, we use the t-distribution, but only if we feel it is appropriate to assume that the population distribution itself is normal (or close to normal). In above case for sample size 50, we know it is close to normal, so we can use t distribution for calculating Confidence interval

**Reason**: Small samples are less precise and hence confidence interval range is larger, in which case, t-distribution provides a better estimation of Confidence interval by providing a broader margin of error whereas z-score calculation underestimates the error and tends to give incorrect results

__Q17:__ Calculate the 95% confidence interval for the mean using the _t_ distribution. Is this wider or narrower than that based on the normal distribution above? If you're unsure, you may find this [resource](https://www.statisticshowto.datasciencecentral.com/probability-and-statistics/confidence-interval/) useful. For calculating the critical value, remember how you could calculate this for the normal distribution using norm.ppf().

__A:__see below

```python
#Reference: https://stackoverflow.com/questions/15033511/compute-a-confidence-interval-from-sample-data

import statsmodels.stats.api as sms

a = 1.0 * np.array(sample_50)
sms.DescrStatsW(a).tconfint_mean()
```

    (171.589184447403, 173.9738372679546)

This is slightly wider than the previous confidence interval. This reflects the greater uncertainty given that we are estimating population parameters from a sample.

## 4. Learning outcomes

Having completed this project notebook, you now have hands-on experience:
* sampling and calculating probabilities from a normal distribution
* identifying the correct way to estimate the standard deviation of a population (the population parameter) from a sample
* with sampling distribution and now know how the Central Limit Theorem applies
* with how to calculate critical values and confidence intervals

Frequentist Case Study/Frequentist Inference Case Study - Part A.ipynb