# Confidence interval approximations for the AUROC

The area under the receiver operating curve (AUROC) is one of the most commonly used performance metrics for binary classification. Visually, the AUROC is the integral between the sensitivity and false positive rate curves across all thresholds for a binary classifier. The AUROC can also be shown to be equivalent to an instance of the [Mann-Whitney-U test](https://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U_test) (MNU), a non-parametric rank-based statistic. This post addresses two challenges when doing statistical testing for the AUROC: i) how to speed up the calculation of the AUROC, and ii) which inference procedure to use to obtain the best possible coverage. The AUROC's relationship to the MNU will be shown to be important for both speed ups in calculation and resampling approaches for the bootstrap.

## (1) Methods for calculating the AUROC

In the binary classification paradigm a model produces a score associated with the probability that an observation belongs to class 1 (as opposed to class 0). The AUROC of any model is a probabilistic term: $P(s^1 > s^0)$, where $s^k$ is the distribution of scores from the model for class $k$. In practice the AUROC is never known because the distribution of data is unknown! However, an unbiased estimate of the AUROC (a.k.a the empirical AUROC) can be calculated through one of several approaches.

The first method is to draw the [ROC curve](https://en.wikipedia.org/wiki/Receiver_operating_characteristic) by measuring the sensitivity/specificity across all thresholds, and then using the [trapezoidal rule](https://en.wikipedia.org/wiki/Trapezoidal_rule) for calculating the integral. This approach is computationally inefficient and should only be done for visualization purposes. A second method to obtain the empirical AUROC is to simply calculate the percentage of times the positive class score exceeds the negative class score:

$$
\begin{align}
AUC &= \frac{1}{n_1n_0} \sum_{i: y_i=1} \sum_{j: y_j=0} I(s_i > s_j) + 0.5\cdot I(s_i = s_j) \label{eq:auc_pair}
\end{align}
$$

Where $y_i$ is the binary label for the $i^{th}$ observation and $n_k$ is the number of instances for class $k$. If we assume that the positive class is some fraction of the observation in the population: $P(y=1) = c$, then on average, calculating the AUROC via \eqref{eq:auc_pair} requires $c(1-c)n^2$ operations which means $O(AUC)=n^2$. For larger sample sizes this quadratic complexity will lead to long run times. One method to bound the computational complexity of \eqref{eq:auc_pair} is to randomly sample, with replacement, $m$ samples from each class the data to get a stochastic approximation of the AUC.

$$
\begin{align}
\tilde{AUC} &= \frac{1}{m} \sum_{i} P(\tilde{s_i}^1 > \tilde{s_i}^0) \label{eq:auc_rand}
\end{align}
$$

Where $\tilde{s_i}^k$ is a random instance from the scores of class $k$. The stochastic AUROC approach has the nice computational advantage that it is $O(m)$. As with other stochastic methods, \eqref{eq:auc_rand} requires knowledge of the sampling variation of the statistic and seeding, which tends to discourage its use in practice. This post will encourage the use of the rank order of the data to calculate the empirical AUROC.

$$
\begin{align}
rAUC &= \frac{1}{n_1n_0} \sum_{i: y_i=1} r_i - \frac{n_1(n_1 +1)}{2} \label{eq:auc_rank}
\end{align}
$$

Where $r_i$ is the sample rank of the data. Since ranking a vector is $O(n\log n)$, the computational complexity of \eqref{eq:auc_rank} is linearithmic, which will mean significant speed ups over \eqref{eq:auc_pair}.

## (2) Run-time comparisons

The code block below shows the run-times for the different approaches to calculate the AUROC from section (1) across different sample sizes ($n$) with different positive class proportions ($n_1/n$). The stochastic approach using $m = 5 n$. It is easy to generate data from two distributions so that the population AUROC can be known in advance. For example, if $s^1$ and $s^0$ come from the normal distribution:

$$
\begin{align*}
s_i^0 \sim N(0,1)&, \hspace{2mm} s_i^1 \sim N(\mu,1), \hspace{2mm} \mu \geq 0, \\
P(s_i^1 > s_i^0) &= \Phi\big(\mu / \sqrt{2}\big).
\end{align*}
$$

Alternatively one could use two exponential distributions:

$$
\begin{align*}
s_i^0 \sim Exp(1)&, \hspace{2mm} s_i^1 \sim Exp(\lambda^{-1}), \hspace{2mm} \lambda \geq 1, \\
P(s_i^1 > s_i^0) &= \frac{\lambda}{1+\lambda}.
\end{align*}
$$

It is easy to see that scale parameter of the normal or exponential distribution can determined *a priori* to match some pre-specific AUROC target.

$$
\begin{align*}
\mu^* &= \sqrt{2} \cdot \Phi^{-1}(AUC) \\
\lambda^* &= \frac{AUC}{1-AUC}
\end{align*}
$$

The simulations in this post will use the normal distribution for simplicity, although using the exponential distribution will change the results of the analysis. The reason is that the variance of the AUROC will be identical regardless of the distribution that generated it, as long as those two distributions have the same AUROC, of course.

```python
"""
DEFINE HELPER FUNCTIONS NEEDED THROUGHOUT POST
"""

import os
import numpy as np
import pandas as pd
import plotnine
from plotnine import *
from scipy import stats
from scipy.interpolate import UnivariateSpline
from timeit import timeit
from sklearn.metrics import roc_curve, auc

def rvec(x):
    return np.atleast_2d(x)

def cvec(x):
    return rvec(x).T

def auc_pair(y, s):
    s1, s0 = s[y == 1], s[y == 0]
    n1, n0 = len(s1), len(s0)
    count = 0
    for i in range(n1):
        count += np.sum(s1[i] > s0)
        count += 0.5*np.sum(s1[i] == s0)
    return count/(n1*n0)

def auc_rand(y, s, m):
    s1 = np.random.choice(s[y == 1], m, replace=True)
    s0 = np.random.choice(s[y == 0], m, replace=True)
    return np.mean(s1 > s0)

def auc_rank(y, s):
    n1 = sum(y)
    n0 = len(y) - n1
    den = n0 * n1
    num = sum(stats.rankdata(s)[y == 1]) - n1*(n1+1)/2
    return num / den

def dgp_auc(n, p, param, dist='normal'):
    n1 = np.random.binomial(n,p)
    n0 = n - n1
    if dist == 'normal':
        s0 = np.random.randn(n0)
        s1 = np.random.randn(n1) + param
    if dist == 'exp':
        s0 = np.random.exponential(1,n0)
        s1 = np.random.exponential(param,n1)
    s = np.concatenate((s0, s1))
    y = np.concatenate((np.repeat(0, n0), np.repeat(1, n1)))
    return y, s
```

```python
target_auc = 0.75
mu_75 = np.sqrt(2) * stats.norm.ppf(target_auc)
lam_75 = target_auc / (1 - target_auc)
n, p = 500, 0.5

np.random.seed(2)
y_exp, s_exp = dgp_auc(n, p, lam_75, 'exp')
y_norm, s_norm = dgp_auc(n, p, mu_75, 'normal')
fpr_exp, tpr_exp, _ = roc_curve(y_exp, s_exp)
fpr_norm, tpr_norm, _ = roc_curve(y_norm, s_norm)
df = pd.concat([pd.DataFrame({'fpr':fpr_exp,'tpr':tpr_exp,'tt':'Exponential'}),
                pd.DataFrame({'fpr':fpr_norm,'tpr':tpr_norm, 'tt':'Normal'})])
tmp_txt = df.groupby('tt')[['fpr','tpr']].mean().reset_index().assign(fpr=[0.15,0.15],tpr=[0.85,0.95])
tmp_txt = tmp_txt.assign(lbl=['AUC: %0.3f' % auc_rank(y_exp, s_exp),
                    'AUC: %0.3f' % auc_rank(y_norm, s_norm)])

plotnine.options.figure_size = (4, 3)
gg_roc = (ggplot(df,aes(x='fpr',y='tpr',color='tt')) + theme_bw() +
          geom_step() + labs(x='FPR',y='TPR') +
          scale_color_discrete(name='Distrubition') +
          geom_abline(slope=1,intercept=0,linetype='--') +
          geom_text(aes(label='lbl'),size=10,data=tmp_txt))
gg_roc # ggtitle('ROC curve by distribution')
```

```python
# Get run-times for different sizes of n
p_seq = [0.1, 0.3, 0.5]
n_seq = np.arange(25, 500, 25)
nrun = 1000
c = 5

if 'df_rt.csv' in os.listdir():
    df_rt = pd.read_csv('df_rt.csv')
else:
    np.random.seed(nrun)
    holder = []
    for p in p_seq:
        print(p)
        for n in n_seq:
            cont = True
            m = c * n
            while cont:
                y, s = dgp_auc(n, p, 0, dist='normal')
                cont = sum(y) == 0
            ti_rand = timeit('auc_rand(y, s, m)',number=nrun,globals=globals())
            ti_rank = timeit('auc_rank(y, s)',number=nrun,globals=globals())
            ti_pair = timeit('auc_pair(y, s)',number=nrun,globals=globals())
            tmp = pd.DataFrame({'rand':ti_rand, 'rank':ti_rank, 'pair':ti_pair, 'p':p, 'n':n},index=[0])
            holder.append(tmp)
    df_rt = pd.concat(holder).melt(['p','n'],None,'method')
    df_rt.to_csv('df_rt.csv',index=False)

plotnine.options.figure_size = (7, 3.0)
gg_ti = (ggplot(df_rt,aes(x='n',y='value',color='method')) + theme_bw() +
         facet_wrap('~p',labeller=label_both) + geom_line() +
         scale_color_discrete(name='Method',labels=['Pairwise','Stochastic','Rank']) +
         labs(y='Seconds (1000 runs)', x='n'))
gg_ti  # ggtitle('AUROC run-time') +
```

Figure 1 provides an example of two ROC curves coming from a Normal and Exponential distribution. Though the empirical AUROCs between the two curves is virtually identical, their respective sensitivity/specificity trade-offs are different. The Exponential distribution tends to have a more favourable sensitivity for high thresholds because of the right skew of the data. This figure is a reminder of some of the inherent limitations with using the AUROC as an evaluation measure. Although to repeat, the distribution of the AUROC statistic between these, or other, distributions would be the same.

The significant runtime performance gains from using the ranking approach in \eqref{eq:auc_rank} is shown in Figure 2. The pairwise method from \eqref{eq:auc_pair} is many orders of magnitude slower once the sample size is more than a few dozen observations. The stochastic method's run time is shown to be slightly better than the ranking method. This is to be expected given that \eqref{eq:auc_rand} is linear in $n$. However, using the stochastic approach requires picking a permutation size that leads to sufficiently tight bounds around the point estimate. The simulations below show the variation around the estimate by the number of draws.

```python
# Get the quality of the stochastic approximation
nsim = 100
n_seq = [100, 500, 1000]
c_seq = np.arange(1,11,1).astype(int)

if 'df_se.csv' in os.listdir():
    df_se = pd.read_csv('df_se.csv')
else:
    np.random.seed(nsim)
    holder = []
    for n in n_seq:
        holder_n = []
        for ii in range(nsim):
            y, s = dgp_auc(n, p, 0, dist='normal')
            gt_auc = auc_pair(y, s)
            sim_mat = np.array([[auc_rand(y, s, n*c) for c in c_seq] for x in range(nsim)])
            dat_err = np.std(gt_auc - sim_mat,axis=0)
            holder_n.append(dat_err)
        tmp = pd.DataFrame(np.array(holder_n)).melt(None,None,'c','se').assign(n=n)
        holder.append(tmp)
    df_se = pd.concat(holder).reset_index(None, True)
    df_se.c = df_se.c.map(dict(zip(list(range(len(c_seq))),c_seq)))
    df_se.to_csv('df_se.csv',index=False)
df_se = df_se.assign(sn=lambda x: pd.Categorical(x.n.astype(str),[str(z) for z in n_seq]))

plotnine.options.figure_size = (4, 3)
gg_se = (ggplot(df_se, aes(x='c',y='se',color='sn')) +
         theme_bw() + labs(y='Standard error',x='Number of draws * n') +
         geom_jitter(height=0,width=0.1,size=0.5,alpha=0.5) +
         scale_color_discrete(name='n') +
         scale_x_continuous(breaks=list(c_seq)))
gg_se  # ggtitle('Variation around point estimate from randomization method')
```

Figure 3 shows that the number of samples needed to get a small standard error to the ±1% is 4000 draws. In other words, if the actual empirical AUROC was 71%, we would expect 95% of the realizations to be around the 69-73% range. To get to the ±0.5% requires 10K draws. This shows that unless the user is happy to tolerate an error range of more than a percentage point, hundred of thousands of draws will likely be needed.

## (3) Inference approaches

After reviewing the different approaches for calculating the point estimate of the empirical AUROC, attention can now be turned to doing inference on this term. Knowing that a classifier has an AUROC on 78% on a test set provides little information if there is no quantification of the uncertainty around this range. In this section, we'll discuss three different approaches for generating confidence intervals ([CIs](https://en.wikipedia.org/wiki/Confidence_interval)) which are the most common method of uncertainty quantification in frequentist statistics. A two-sided CI at the $1-\alpha$% level is a random variable that has the following property: $P([l, u] \in AUC ) \geq 1-\alpha$. In other words, the probability that the true AUROC is contained within this upper and lower bound, $l$ and $u$ (which are random variables), is at least $1-\alpha$%, meaning the true statistic of interest (the AUROC) fails to be *covered* by this interval at most $\alpha$% of the time. An exact CI will cover the true statistic of interest exactly $1-\alpha$% of the time, given the test maximum power.

The approaches below are by no means exhaustive. Readers are encouraged to review other [methods](https://arxiv.org/pdf/1804.05882.pdf) for other ideas.

### Approach #1: Asymptotic U

As was previously mentioned, the AUROC is equivalent to an MNU test. The asymptotic properties of this statistic have been known for [more than 70 years](https://projecteuclid.org/euclid.aoms/1177730491). Under the null hypothesis assumption that $P(s_i^1 > s_i^0) = 0.5$, the asymptotic properties of the U statistic for ranks can be shown to be:

$$
\begin{align*}
z &= \frac{U - \mu_U}{\sigma_U} \sim N(0,1) \\
\mu_U &= \frac{n_0n_1}{2} \\
\sigma^2_U &= \frac{n_0n_1(n_0+n_1+1)}{12} \\
U &= n_1n_0 \cdot \max \{ AUC, (1-AUC) \} \\
\end{align*}
$$

Note that additional corrections that need to be applied in the case of data which has ties, but I will not cover this issue here. There are two clear weaknesses to this approach. First, it appeals to the asymptotic normality of the $U$ statistic, which may be a poor approximation when $n$ is small. Second, this formula only makes sense for testing a null hypothesis of $AUC_0=0.5$. Notice that the constant in the denominator of the variance, 12, is the same as the constant in the variance of a [uniform distribution](https://en.wikipedia.org/wiki/Continuous_uniform_distribution). This is not a coincidence as the distribution of rank order statistics is uniform when the data come from the same distribution. To estimate this constant for $AUC\neq 0.5$, Monte Carlo simulations will be needed. Specifically we want to find the right constant $c(AUC)$ for the variance of the AUROC:

$$
\begin{align*}
\sigma^2_U(AUC) &= \frac{n_0n_1(n_0+n_1+1)}{c(AUC)}
\end{align*}
$$

Even though it is somewhat computationally intensive to calculate these normalizing constants, their estimates hold true regardless of the sample of the sample sizes, as in $c(AUC;n_0,n_1)=c(AUC;n_0';n_1')$ for all $n_k, n_k' \in \mathbb{R}^+$. The code below estimates $c()$ and uses a spline to interpolate for values of the AUROC between the realized draws.

```python
# PRECOMPUTE THE VARIANCE CONSTANT...
if 'dat_var.csv' in os.listdir():
    dat_var = pd.read_csv('dat_var.csv')
else:
    np.random.seed(1)
    nsim = 10000
    n1, n0 = 500, 500
    den = n1 * n0
    auc_seq = np.arange(0.5, 1, 0.01)
    holder = np.zeros(len(auc_seq))
    for i, auc in enumerate(auc_seq):
        print(i)
        mu = np.sqrt(2) * stats.norm.ppf(auc)
        Eta = np.r_[np.random.randn(n1, nsim)+mu, np.random.randn(n0,nsim)]
        Y = np.r_[np.zeros([n1,nsim],dtype=int)+1, np.zeros([n0,nsim],dtype=int)]
        R1 = stats.rankdata(Eta,axis=0)[:n1]
        Amat = (R1.sum(0) - n1*(n1+1)/2) / den
        holder[i] = (n0+n1+1) / Amat.var() / den
    dat_var = pd.DataFrame({'auc':auc_seq, 'c':holder})
    dat_var = pd.concat([dat_var.iloc[1:].assign(auc=lambda x: 1-x.auc), dat_var]).sort_values('auc').reset_index(None, True)
    dat_var.to_csv('dat_var.csv', index=False)
# Calculate the spline
spl = UnivariateSpline(x=dat_var.auc, y=dat_var.c)
dat_spline = pd.DataFrame({'auc':dat_var.auc, 'spline':spl(dat_var.auc)})
plotnine.options.figure_size=(4,3)
gg_c = (ggplot(dat_var,aes(x='auc',y='np.log(c)')) + theme_bw() +
        geom_point()+labs(y='log c(AUC)',x='AUROC') +
        geom_line(aes(x='auc',y='np.log(spline)'), data=dat_spline,color='red') +
       ggtitle('Red line is spline (k=3)'))
gg_c
```

Figure 4 shows that the constant term is growing quite rapidly. The stochastic estimate of the constant at AUROC=0.5 of 11.9 is close to the true population value of 12.

### Approach #2: Newcombe's Wald Method

A second approach is to use a (relatively) new approach from [Newcombe (2006)](https://onlinelibrary.wiley.com/doi/10.1002/sim.2324). Unlike the asymptotic approach above, Newcombe's method automatically calculates the different level of the variance for different values of the AUROC.

$$
\begin{align*}
\sigma^2_{AUC} &= \frac{AUC(1-AUC)}{(n_1-1)(n_0-1)} \cdot \Bigg[ 2n - 1 - \frac{3n-3}{(2-AUC)(1+AUC)} \Bigg]
\end{align*}
$$

Assuming $n_1 = c\cdot n$ then $O(\sigma^2_{AUC})=\frac{AUC(1-AUC)}{n}$, which is very similar to the variance of the binomial proportion (see [here](https://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval)).

### Approach #3: Bootstrapping ranks

The final inference approach is that of [bootstrap](https://en.wikipedia.org/wiki/Bootstrapping_(statistics)), which generates new copies of the statistic by resampling the data. Though the ability to get additional randomness by resampling rows of the data seems a little mysterious, if not dubious, it has a solid mathematical foundation. The bootstrap is equivalent to drawing from the empirical CDF (eCDF) of a random variable. Since the eCDF is known to be a [consistent](https://en.wikipedia.org/wiki/Glivenko%E2%80%93Cantelli_theorem) estimate of the true CDF, the error of the bootstrap will naturally decrease as $n$ grows. The bootstrap has the attractive property that it is fully non-parametric and works from a broad class of statistics. Note that there is no one way to do the "bootstrap" for inference, and I compare three common approaches: i) quantile, ii) classic, iii) studentized. For a review of other approaches, see [here](http://users.stat.umn.edu/~helwig/notes/bootci-Notes.pdf).

$$
\begin{align*}
\tilde{AUC}^{(k)} &= \frac{1}{n_1n_0} \sum_{i: y_i=1} \tilde{r}_i^{(k)} - \frac{n_1(n_1 +1)}{2} \\
\sigma^2_{BS} &= \frac{1}{K-1}\sum_{k=1}^K (\tilde{AUC}^{(k)} - \bar{\tilde{AUC}})
\end{align*}
$$

The $k^{th}$ bootstrap (out of $K$ total bootstraps), is generated by sampling, with replacement, the ranks of the positive score classes, and the bootstrap AUROC is calculated using the same formula from \eqref{eq:auc_rank}. Bootstrapping the ranks has the incredibly attractive property that the relative runtime is going to scale with the total number of bootstraps ($K$). If we had to recalculate the ranks for every bootstrap sample, then this would require an additional sorting call. The formulas for the three bootstrapping approaches are shown below for a $1-\alpha$% symmetric CI.

$$
\begin{align*}
\text{Quantile}& \\
[l, u] &= \big[\tilde{AUC}^{(k)}_{\lfloor\alpha/2\cdot K\rfloor}, \tilde{AUC}^{(k)}_{\lceil(1-\alpha/2)\cdot K\rceil} \big] \\
\\
\text{SE}& \\
[l, u] &= \big[AUC + \sigma_{BS}\cdot z_{\alpha/2}, AUC - \sigma_{BS}\cdot z_{\alpha/2}\big] \\
\\
\text{Studentized}& \\
[l, u] &= \big[AUC + \sigma_{BS}\cdot z_{\alpha/2}^*, AUC - \sigma_{BS}\cdot z_{1-\alpha/2}^*\big] \\
z_\alpha^* &= \Bigg[ \frac{\tilde{AUC}^{(k)} - AUC}{\sigma^{(k)}_{BS}} \Bigg]_{\lfloor\alpha\cdot K\rfloor}
\end{align*}
$$

The quantile approach simply takes the empirical $\alpha/2$ and $1-\alpha/2$ quantiles of the AUROC from its bootstrapped distribution. Though the quantile approach is easily for suited to skewed bootstrapped distributions (i.e. the CIs are not symmetric), it is known to be biased for sample sizes. The classic bootstrap, simply uses the bootstrapped AUROCs to estimate its empirical variance, and then use the standard normal approximation to generate CIs. The Studentized approach combines the estimate of the variance from the SE/classic approach but also takes into account the possibility for a skewed distribution. For each bootstrap sample, an additional $K$ (or some large number) samples are drawn, so that each bootstrapped sample has an estimate of its variance. These studentized, or normalized, scores are then used in place of the quantile from the normal distribution.

## (4) Simulations

Now we are ready to test the bootstrapping methods against their analytic counterparts. The simulations below will use a 10% positive class balance, along with a range of different sample sizes. Symmetric CIs will be calculated for the 80%, 90%, and 95% level. A total of 1500 simulations are run. An 80% symmetric CI that is exact should a coverage of 80%, meaning that the true AUROC is contained within the CI 80% of the time. A CI that has a coverage below its nominal level will have a type-1 error rate that is greater than expected, whilst a CI that has coverage above its nominal level will have less power (i.e. a higher type-II error). In other words, the closer a CI is to its nominal level, the better.

```python
"""
HELPER FUNCTION TO RETURN +- INTERVALS
A: array of AUCs
se: array of SEs
cv: critical values (can be array: will be treated as 1xk)
"""
def ret_lbub(A, se, cv, method):
    ub = cvec(A)+cvec(se)*rvec(cv)
    lb = cvec(A)-cvec(se)*rvec(cv)
    df_ub = pd.DataFrame(ub,columns=cn_cv).assign(bound='upper')
    df_lb = pd.DataFrame(lb,columns=cn_cv).assign(bound='lower')
    df = pd.concat([df_ub, df_lb]).assign(tt=method)
    return df

nsim = 1500
prop = 0.1
n_bs = 1000
n_student = 250
n_seq = [50, 100, 250, 1000]#[]
auc_seq = [0.5, 0.7, 0.9 ]  #"true" AUROC between the distributions
pvals = (1-np.array([0.8, 0.9, 0.95]))/2
crit_vals = np.abs(stats.norm.ppf(pvals))
cn_cv = ['p'+str(i+1) for i in range(len(pvals))]

np.random.seed(1)

if 'res.csv' in os.listdir():
    res = pd.read_csv('res.csv')
else:
    holder = []
    for n in n_seq:
        for auc in auc_seq:
            print('n: %i, AUROC: %0.2f' % (n, auc))
            n1 = int(np.round(n * prop))
            n0 = n - n1
            den = n1*n0
            mu = np.sqrt(2) * stats.norm.ppf(auc)
            Eta = np.r_[np.random.randn(n1, nsim)+mu, np.random.randn(n0,nsim)]
            Y = np.r_[np.zeros([n1,nsim],dtype=int)+1, np.zeros([n0,nsim],dtype=int)]
            # Calculate the AUCs across the columns
            R1 = stats.rankdata(Eta,axis=0)[:n1]
            Amat = (R1.sum(0) - n1*(n1+1)/2) / den
            # --- Approach 1: Asymptotic U --- #
            sd_u = np.sqrt((n0+n1+1)/spl(Amat)/den)
            df_asym = ret_lbub(Amat, sd_u, crit_vals, 'asymptotic')
            # --- Approach 2: Newcombe's wald
            sd_newcombe = np.sqrt(Amat*(1-Amat)/((n1-1)*(n0-1))*(2*n-1-((3*n-3)/((2-Amat)*(1+Amat)))))
            df_newcombe = ret_lbub(Amat, sd_newcombe, crit_vals, 'newcombe')
            # --- Approach 3: Bootstrap the ranks --- #
            R1_bs = pd.DataFrame(R1).sample(frac=n_bs,replace=True).values.reshape([n_bs]+list(R1.shape))
            auc_bs = (R1_bs.sum(1) - n1*(n1+1)/2) / den
            sd_bs = auc_bs.std(0,ddof=1)
            # - (i) Standard error method - #
            df_bs_se = ret_lbub(Amat, sd_bs, crit_vals, 'bootstrap_se')
            # - (ii) Quantile method - #
            df_lb_bs = pd.DataFrame(np.quantile(auc_bs,pvals,axis=0).T,columns=cn_cv).assign(bound='lower')
            df_ub_bs = pd.DataFrame(np.quantile(auc_bs,1-pvals,axis=0).T,columns=cn_cv).assign(bound='upper')
            df_bs_q = pd.concat([df_ub_bs, df_lb_bs]).assign(tt='bootstrap_q')
            # - (iii) Studentized - #
            se_bs_s = np.zeros(auc_bs.shape)
            for j in range(n_bs):
                R1_bs_s = pd.DataFrame(R1_bs[j]).sample(frac=n_student,replace=True).values.reshape([n_student]+list(R1.shape))
                auc_bs_s = (R1_bs_s.sum(1) - n1*(n1+1)/2) / den
                se_bs_s[j] = auc_bs_s.std(0,ddof=1)
            # Get the t-score dist
            t_bs = (auc_bs - rvec(Amat))/se_bs_s
            df_lb_t = pd.DataFrame(cvec(Amat) - cvec(sd_bs)*np.quantile(t_bs,1-pvals,axis=0).T,columns=cn_cv).assign(bound='lower')
            df_ub_t = pd.DataFrame(cvec(Amat) - cvec(sd_bs)*np.quantile(t_bs,pvals,axis=0).T,columns=cn_cv).assign(bound='upper')
            df_t = pd.concat([df_ub_t, df_lb_t]).assign(tt='bootstrap_s')
            # Combine
            tmp_sim = pd.concat([df_asym, df_newcombe, df_bs_se, df_bs_q, df_t]).assign(auc=auc, n=n)
            holder.append(tmp_sim)
    # Merge and save
    res = pd.concat(holder)
    res = res.rename_axis('idx').reset_index().melt(['idx','bound','tt','auc','n'],cn_cv,'tpr')
    res = res.pivot_table('value',['idx','tt','auc','n','tpr'],'bound').reset_index()
    res.tpr = res.tpr.map(dict(zip(cn_cv, 1-2*pvals)))
    res = res.assign(is_covered=lambda x: (x.lower <= x.auc) & (x.upper >= x.auc))
    res.to_csv('res.csv',index=False)
res_cov = res.groupby(['tt','auc','n','tpr']).is_covered.mean().reset_index()
res_cov = res_cov.assign(sn = lambda x: pd.Categorical(x.n, x.n.unique()))
lvls_approach = ['asymptotic','newcombe','bootstrap_q','bootstrap_se','bootstrap_s']
lbls_approach = ['Asymptotic', 'Newcombe', 'BS (Quantile)', 'BS (Classic)', 'BS (Studentized)']
res_cov = res_cov.assign(tt = lambda x: pd.Categorical(x.tt, lvls_approach).map(dict(zip(lvls_approach, lbls_approach))))
res_cov.rename(columns={'tpr':'CoverageTarget', 'auc':'AUROC'}, inplace=True)

tmp = pd.DataFrame({'CoverageTarget':1-2*pvals, 'ybar':1-2*pvals})
plotnine.options.figure_size = (6.5, 5)
gg_cov = (ggplot(res_cov, aes(x='tt', y='is_covered',color='sn')) +
         theme_bw() + geom_point() +
         facet_grid('AUROC~CoverageTarget',labeller=label_both) +
         theme(axis_text_x=element_text(angle=90), axis_title_x=element_blank()) +
         labs(y='Coverage') +
         geom_hline(aes(yintercept='ybar'),data=tmp) +
         scale_color_discrete(name='Sample size'))
gg_cov
```

Figure 5 shows the coverage results for the different approaches across different conditions. Newcombe's method is consistently the worst performer, with the CIs being much too conservative. The estimated standard errors (SEs) are at least 40% larger than the asymptotic ones (code not shown), leading to a CI with significantly reduced power. The asymptotic approach and quantile/classic bootstrap have SEs which are too small when the sample size is limited, leading under-coverage and an inflated type-I error rate. For sample sizes of at least 1000, the asymptotic intervals are quite accurate. The studentized bootstrap is by far the most accurate approach, especially for small sample sizes, and tends to be conservative (over-coverage). Overall the studentized bootstrap is the clear winner. However, it is also the most computationally costly, which means for large samples the asymptotic estimates may be better.

## (5) Ranking bootstraps?

Readers may be curious whether ranking the bootstraps, rather than bootstrapping the ranks, may lead to better inference. Section (3) has already noted the obvious computational gains from bootstrapping the ranks. Despite my initial impression that ranking the bootstraps would lead to more variation because of the additional variation in the negative class, this turned out not to be the case due to the creation of ties in the scores which reduces the variation in the final AUROC estimate. The simulation block shows that the SE of the bootstrapped ranks is higher than the ranked bootstraps in terms of the AUROC statistic. Since the bootstrap approach did not have a problem of over-coverage, the smaller SEs will lead to higher type-I error rates, especially for small sample sizes. In this case, the statistical advantages of bootstrapping the ranks also coincidence with a computational benefit.

```python
if 'df_bs.csv' in os.listdir():
    df_bs = pd.read_csv('df_bs.csv')
else:
    seed = 1
    np.random.seed(seed)
    n_bs, nsim = 1000, 1500
    n1, n0, mu = 25, 75, 1
    s = np.concatenate((np.random.randn(n1, nsim)+mu, np.random.randn(n0,nsim)))
    y = np.concatenate((np.repeat(1,n1),np.repeat(0,n0)))
    r = stats.rankdata(s,axis=0)[:n1]
    s1, s0 = s[:n1], s[n1:]
    r_bs = pd.DataFrame(r).sample(frac=n_bs,replace=True,random_state=seed).values.reshape([n_bs]+list(r.shape))
    s_bs1 = pd.DataFrame(s1).sample(frac=n_bs,replace=True,random_state=seed).values.reshape([n_bs]+list(s1.shape))
    s_bs0 = pd.DataFrame(s0).sample(frac=n_bs,replace=True,random_state=seed).values.reshape([n_bs]+list(s0.shape))
    s_bs = np.concatenate((s_bs1, s_bs0),axis=1)
    r_s_bs = stats.rankdata(s_bs,axis=1)[:,:n1,:]
    auc_bs = (r_bs.sum(1) - n1*(n1+1)/2)/(n1*n0)
    auc_s_bs = (r_s_bs.sum(1) - n1*(n1+1)/2)/(n1*n0)
    se_bs = auc_bs.std(0)
    se_s_bs = auc_s_bs.std(0)
    df_bs = pd.DataFrame({'bs_r':se_bs, 'r_bs':se_s_bs})
    df_bs.to_csv('df_bs.csv', index=False)
print('Mean AUROC for bootstrapping ranks: %0.3f, and ranking bootstraps: %0.3f' %
      (np.mean(df_bs.bs_r),np.mean(df_bs.r_bs)))
```

    Mean AUROC for bootstrapping ranks: 0.064, and ranking bootstraps: 0.054

```python

```


###### Content under Creative Commons Attribution license CC-BY 4.0, code under MIT license © 2015 L.A. Barba, C.D. Cooper, G.F. Forsyth.  Based on [CFD Python](https://github.com/barbagroup/CFDPython), © 2013 L.A. Barba, also under CC-BY license.

# Relax and hold steady

This is **Module 5** of the open course [**"Practical Numerical Methods with Python"**](https://openedx.seas.gwu.edu/courses/course-v1:MAE+MAE6286+2017/about), titled *"Relax and hold steady: elliptic problems"*.
If you've come this far in the [#numericalmooc](https://twitter.com/hashtag/numericalmooc) ride, it's time to stop worrying about **time** and relax.

So far, you've learned to solve problems dominated by convection—where solutions have a directional bias and can form shocks—in [Module 3](https://nbviewer.jupyter.org/github/numerical-mooc/numerical-mooc/tree/master/lessons/03_wave/): *"Riding the Wave."* In [Module 4](https://nbviewer.jupyter.org/github/numerical-mooc/numerical-mooc/tree/master/lessons/04_spreadout/) (*"Spreading Out"*), we explored diffusion-dominated problems—where solutions spread in all directions. But what about situations where solutions are steady?

Many problems in physics have no time dependence, yet are rich with physical meaning: the gravitational field produced by a massive object, the electrostatic potential of a charge distribution, the displacement of a stretched membrane and the steady flow of fluid through a porous medium ... all these can be modeled by **Poisson's equation**:

$$
\begin{equation}
\nabla^2 u = f
\end{equation}
$$

where the unknown $u$ and the known $f$ are functions of space, in a domain $\Omega$. To find the solution, we require boundary conditions. These could be  Dirichlet boundary conditions, specifying the value of the solution on the boundary,

$$
\begin{equation}
u = b_1 \text{ on } \partial\Omega,
\end{equation}
$$

or Neumann boundary conditions, specifying the normal derivative of the solution on the boundary,

$$
\begin{equation}
\frac{\partial u}{\partial n} = b_2 \text{ on } \partial\Omega.
\end{equation}
$$

A boundary-value problem consists of finding $u$, given the above information. Numerically, we can do this using *relaxation methods*, which start with an initial guess for $u$ and then iterate towards the solution. Let's find out how!

## Laplace's equation

The particular case of $f=0$ (homogeneous case) results in Laplace's equation:

$$
\begin{equation}
\nabla^2 u = 0
\end{equation}
$$

For example, the equation for steady, two-dimensional heat conduction is:

$$
\begin{equation}
\frac{\partial ^2 T}{\partial x^2} + \frac{\partial ^2 T}{\partial y^2} = 0
\end{equation}
$$

This is similar to the model we studied in [lesson 3](https://nbviewer.jupyter.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/04_spreadout/04_03_Heat_Equation_2D_Explicit.ipynb) of **Module 4**, but without the time derivative: i.e., for a temperature $T$ that has reached steady state. The Laplace equation models the equilibrium state of a system under the supplied boundary conditions.

The study of solutions to Laplace's equation is called *potential theory*, and the solutions themselves are often potential fields. Let's use $p$ from now on to represent our generic dependent variable, and write Laplace's equation again (in two dimensions):

$$
\begin{equation}
\frac{\partial ^2 p}{\partial x^2} + \frac{\partial ^2 p}{\partial y^2} = 0
\end{equation}
$$

Like in the diffusion equation of the previous course module, we discretize the second-order derivatives with *central differences*. You should be able to write down a second-order central-difference formula by heart now! On a two-dimensional Cartesian grid, it gives:

$$
\begin{equation}
\frac{p_{i+1, j}  - 2p_{i,j}  + p_{i-1,j} }{\Delta x^2} + \frac{p_{i,j+1}  - 2p_{i,j}  + p_{i, j-1} }{\Delta y^2} = 0
\end{equation}
$$

When $\Delta x = \Delta y$, we end up with the following equation:

$$
\begin{equation}
p_{i+1, j}   + p_{i-1,j} + p_{i,j+1}  + p_{i, j-1}- 4 p_{i,j} = 0
\end{equation}
$$

This tells us that the Laplacian differential operator at grid point $(i,j)$ can be evaluated discretely using the value of $p$ at that point (with a factor $-4$) and the four neighboring points to the left and right, above and below grid point $(i,j)$.

The stencil of the discrete Laplacian operator is shown in Figure 1. It is typically called the *five-point stencil*, for obvious reasons.

#### Figure 1: Laplace five-point stencil.

The discrete equation above is valid for every interior point in the domain. If we write the equations for *all* interior points, we have a linear system of algebraic equations. We *could* solve the linear system directly (e.g., with Gaussian elimination), but we can be more clever than that!

Notice that the coefficient matrix of such a linear system has mostly zeroes. For a uniform spatial grid, the matrix is *block diagonal*: it has diagonal blocks that are tridiagonal with $-4$ on the main diagonal and $1$ on two off-center diagonals, and two more diagonals with $1$. All of the other elements are zero. Iterative methods are particularly suited for a system with this structure, and save us from storing all those zeroes.

We will start with an initial guess for the solution, $p_{i,j}^{0}$, and use the discrete Laplacian to get an update, $p_{i,j}^{1}$, then continue on computing $p_{i,j}^{k}$ until we're happy.  Note that $k$ is _not_ a time index here, but an index corresponding to the number of iterations we perform in the *relaxation scheme*.

At each iteration, we compute updated values $p_{i,j}^{k+1}$ in a (hopefully) clever way so that they converge to a set of values satisfying Laplace's equation. The system will reach equilibrium only as the number of iterations tends to $\infty$, but we can approximate the equilibrium state by iterating until the change between one iteration and the next is *very* small.

The most intuitive method of iterative solution is known as the [**Jacobi method**](https://en.wikipedia.org/wiki/Jacobi_method), in which the values at the grid points are replaced by the corresponding weighted averages:

$$
\begin{equation}
p^{k+1}_{i,j} = \frac{1}{4} \left(p^{k}_{i,j-1} + p^k_{i,j+1} + p^{k}_{i-1,j} + p^k_{i+1,j} \right)
\end{equation}
$$

This method does indeed converge to the solution of Laplace's equation. Thank you Professor Jacobi!

##### Challenge task

Grab a piece of paper and write out the coefficient matrix for a discretization with 7 grid points in the $x$ direction (5 interior points) and 5 points in the $y$ direction (3 interior). The system should have 15 unknowns, and the coefficient matrix three diagonal blocks. Assume prescribed Dirichlet boundary conditions on all sides (not necessarily zero).

### Boundary conditions and relaxation

Suppose we want to model steady-state heat transfer on (say) a computer chip with one side insulated (zero Neumann BC), two sides held at a fixed temperature (Dirichlet condition) and one side touching a component that has a sinusoidal distribution of temperature.
We would need to solve Laplace's equation with boundary conditions like

$$
\begin{equation}
  \begin{gathered}
p=0 \text{ at } x=0\\
\frac{\partial p}{\partial x} = 0 \text{ at } x = L_x\\
p = 0 \text{ at }y = 0 \\
p = \sin \left(  \frac{\frac{3}{2}\pi x}{L_x} \right) \text{ at } y = L_y
  \end{gathered}
\end{equation}
$$

We'll take $L_x=1$ and $L_y=1$ for the sizes of the domain in the $x$ and $y$ directions.

One of the defining features of elliptic PDEs is that they are "driven" by the boundary conditions.  In the iterative solution of Laplace's equation, boundary conditions are set and **the solution relaxes** from an initial guess to join the boundaries together smoothly, given those conditions.  Our initial guess will be $p=0$ everywhere. Now, let's relax!

First, we import our usual smattering of libraries (plus a few new ones!)

```python
import numpy
from matplotlib import pyplot
%matplotlib inline
```

```python
# Set the font family and size to use for Matplotlib figures.
pyplot.rcParams['font.family'] = 'serif'
pyplot.rcParams['font.size'] = 16
```

To visualize 2D data, we can use [`pyplot.imshow()`](http://matplotlib.org/api/pyplot_api.html#matplotlib.pyplot.imshow), like we've done before, but a 3D plot can sometimes show a more intuitive view the solution. Or it's just prettier!

Be sure to enjoy the many examples of 3D plots in the `mplot3d` section of the [Matplotlib Gallery](http://matplotlib.org/gallery.html#mplot3d).

We'll import the `mplot3d` module to create 3D plots and also grab the `cm` package, which provides different colormaps for visualizing plots.

```python
from mpl_toolkits import mplot3d
from matplotlib import cm
```

Let's define a function for setting up our plotting environment, to avoid repeating this set-up over and over again. It will save us some typing.

```python
def plot_3d(x, y, p, label='$z$', elev=30.0, azim=45.0):
    """
    Creates a Matplotlib figure with a 3D surface plot
    of the scalar field p.

    Parameters
    ----------
    x : numpy.ndarray
        Gridline locations in the x direction as a 1D array of floats.
    y : numpy.ndarray
        Gridline locations in the y direction as a 1D array of floats.
    p : numpy.ndarray
        Scalar field to plot as a 2D array of floats.
    label : string, optional
        Axis label to use in the third direction;
        default: 'z'.
    elev : float, optional
        Elevation angle in the z plane;
        default: 30.0.
    azim : float, optional
        Azimuth angle in the x,y plane;
        default: 45.0.
    """
    fig = pyplot.figure(figsize=(8.0, 6.0))
    ax = mplot3d.Axes3D(fig)
    ax.set_xlabel('$x$')
    ax.set_ylabel('$y$')
    ax.set_zlabel(label)
    X, Y = numpy.meshgrid(x, y)
    ax.plot_surface(X, Y, p, cmap=cm.viridis)
    ax.set_xlim(x[0], x[-1])
    ax.set_ylim(y[0], y[-1])
    ax.view_init(elev=elev, azim=azim)
```

##### Note
This plotting function uses *Viridis*, a new (and _awesome_) colormap available in Matplotlib versions 1.5 and greater.  If you see an error when you try to plot using `cm.viridis`, just update Matplotlib using `conda` or `pip`.

### Analytical solution

The Laplace equation with the boundary conditions listed above has an analytical solution, given by

$$
\begin{equation}
p(x,y) = \frac{\sinh \left( \frac{\frac{3}{2} \pi y}{L_y}\right)}{\sinh \left(  \frac{\frac{3}{2} \pi L_y}{L_x}\right)} \sin \left( \frac{\frac{3}{2} \pi x}{L_x} \right)
\end{equation}
$$

where $L_x$ and $L_y$ are the length of the domain in the $x$ and $y$ directions, respectively.

We previously used `numpy.meshgrid` to plot our 2D solutions to the heat equation in Module 4.  Here, we'll use it again as a plotting aid. Always useful, `linspace` creates 1-row arrays of equally spaced numbers: it helps for defining $x$ and $y$ axes in line plots, but now we want the analytical solution plotted for every point in our domain.  To do this, we'll use in the analytical solution the 2D arrays generated by `numpy.meshgrid`.

```python
def laplace_solution(x, y, Lx, Ly):
    """
    Computes and returns the analytical solution of the Laplace equation
    on a given two-dimensional Cartesian grid.

    Parameters
    ----------
    x : numpy.ndarray
        The gridline locations in the x direction
        as a 1D array of floats.
    y : numpy.ndarray
        The gridline locations in the y direction
        as a 1D array of floats.
    Lx : float
        Length of the domain in the x direction.
    Ly : float
        Length of the domain in the y direction.

    Returns
    -------
    p : numpy.ndarray
        The analytical solution as a 2D array of floats.
    """
    X, Y = numpy.meshgrid(x, y)
    p = (numpy.sinh(1.5 * numpy.pi * Y / Ly) /
         numpy.sinh(1.5 * numpy.pi * Ly / Lx) *
         numpy.sin(1.5 * numpy.pi * X / Lx))
    return p
```

Ok, let's try out the analytical solution and use it to test the `plot_3D` function we wrote above.

```python
# Set parameters.
Lx = 1.0  # domain length in the x direction
Ly = 1.0  # domain length in the y direction
nx = 41  # number of points in the x direction
ny = 41  # number of points in the y direction

# Create the gridline locations.
x = numpy.linspace(0.0, Lx, num=nx)
y = numpy.linspace(0.0, Ly, num=ny)

# Compute the analytical solution.
p_exact = laplace_solution(x, y, Lx, Ly)

# Plot the analytical solution.
plot_3d(x, y, p_exact)
```

It worked!  This is what the solution *should* look like when we're 'done' relaxing. (And isn't viridis a cool colormap?)

### How long do we iterate?

We noted above that there is no time dependence in the Laplace equation.  So it doesn't make a lot of sense to use a `for` loop with `nt` iterations, like we've done before.

Instead, we can use a `while` loop that continues to iteratively apply the relaxation scheme until the difference between two successive iterations is small enough.

But how small is small enough?  That's a good question.  We'll try to work that out as we go along.

To compare two successive potential fields ($\mathbf{p}^k$ and $\mathbf{p}^{k+1}$), a good option is to use the [L2 norm](http://en.wikipedia.org/wiki/Norm_%28mathematics%29#Euclidean_norm) of the difference.  It's defined as

$$
\begin{equation}
    \parallel \mathbf{p}^{k+1} - \mathbf{p}^k \parallel_{L_2} = \sqrt{\sum_{i, j} \left| p_{i, j}^{k+1} - p_{i, j}^k \right|^2}
\end{equation}
$$

But there's one flaw with this formula.  We are summing the difference between successive iterations at each point on the grid. So what happens when the grid grows? (For example, if we're refining the grid, for whatever reason.) There will be more grid points to compare and so more contributions to the sum. The norm will be a larger number just because of the grid size!

That doesn't seem right.  We'll fix it by normalizing the norm, dividing the above formula by the norm of the potential field at iteration $k$.

For two successive iterations, the relative L2 norm is then calculated as

$$
\begin{equation}
    \frac{\parallel \mathbf{p}^{k+1} - \mathbf{p}^k \parallel_{L_2}}{\parallel \mathbf{p}^k \parallel_{L_2}} = \frac{\sqrt{\sum_{i, j} \left| p_{i, j}^{k+1} - p_{i, j}^k \right|^2}}{\sqrt{\sum_{i, j} \left| p_{i, j}^k \right|^2}}
\end{equation}
$$

For this purpose, we define the `l2_norm` function:

```python
def l2_norm(p, p_ref):
    """
    Computes and returns the relative L2-norm of the difference
    between a solution p and a reference solution p_ref.

    Parameters
    ----------
    p : numpy.ndarray
        The solution as an array of floats.
    p_ref : numpy.ndarray
        The reference solution as an array of floats.

    Returns
    -------
    diff : float
        The relative L2-norm of the difference.
    """
    l2_diff = (numpy.sqrt(numpy.sum((p - p_ref)**2)) /
               numpy.sqrt(numpy.sum(p_ref**2)))
    return l2_diff
```

Now, let's define a function that will apply Jacobi's method for Laplace's equation.  Three of the boundaries are Dirichlet boundaries and so we can simply leave them alone. Only the Neumann boundary needs to be explicitly calculated at each iteration.

```python
def laplace_2d_jacobi(p0, maxiter=20000, rtol=1e-6):
    """
    Solves the 2D Laplace equation using Jacobi relaxation method.

    The function assumes Dirichlet condition with value zero
    at all boundaries except at the right boundary where it uses
    a zero-gradient Neumann condition.
    The exit criterion of the solver is based on the relative L2-norm
    of the solution difference between two consecutive iterations.

    Parameters
    ----------
    p0 : numpy.ndarray
        The initial solution as a 2D array of floats.
    maxiter : integer, optional
        Maximum number of iterations to perform;
        default: 20000.
    rtol : float, optional
        Relative tolerance for convergence;
        default: 1e-6.

    Returns
    -------
    p : numpy.ndarray
        The solution after relaxation as a 2D array of floats.
    ite : integer
        The number of iterations performed.
    diff : float
        The final relative L2-norm of the difference.
    """
    p = p0.copy()
    diff = rtol + 1.0  # initial difference
    ite = 0  # iteration index
    while diff > rtol and ite < maxiter:
        pn = p.copy()
        # Update the solution at interior points.
        p[1:-1, 1:-1] = 0.25 * (p[1:-1, :-2] + p[1:-1, 2:] +
                                p[:-2, 1:-1] + p[2:, 1:-1])
        # Apply Neumann condition (zero-gradient)
        # at the right boundary.
        p[1:-1, -1] = p[1:-1, -2]
        # Compute the residual as the L2-norm of the difference.
        diff = l2_norm(p, pn)
        ite += 1
    return p, ite, diff
```

##### Rows and columns, and index order

Recall that in the [2D explicit heat equation](http://nbviewer.jupyter.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/04_spreadout/04_03_Heat_Equation_2D_Explicit.ipynb) we stored data with the $y$ coordinates corresponding to the rows of the array and $x$ coordinates on the columns (this is just a code design decision!). We did that so that a plot of the 2D-array values would have the natural ordering, corresponding to the physical domain ($y$ coordinate in the vertical).

We'll follow the same convention here (even though we'll be plotting in 3D, so there's no real reason), just to be consistent. Thus, $p_{i,j}$ will be stored in array format as `p[j,i]`. Don't be confused by this.

### Let's relax!

The initial values of the potential field are zero everywhere (initial guess), except at the top boundary:

$$
p = \sin \left(  \frac{\frac{3}{2}\pi x}{L_x} \right) \text{ at } y=L_y
$$

To initialize the domain, `numpy.zeros` will handle everything except that one Dirichlet condition. Let's do it!

```python
# Set the initial conditions.
p0 = numpy.zeros((ny, nx))
p0[-1, :] = numpy.sin(1.5 * numpy.pi * x / Lx)
```

Now let's visualize the initial conditions using the `plot_3D` function, just to check we've got it right.

```python
# Plot the initial conditions.
plot_3d(x, y, p0)
```

The `p` array is equal to zero everywhere, except along the boundary $y = 1$.  Hopefully you can see how the relaxed solution and this initial condition are related.

Now, run the iterative solver with a target L2-norm difference between successive iterations of $10^{-8}$.

```python
# Compute the solution using Jacobi relaxation method.
p, ites, diff = laplace_2d_jacobi(p0, rtol=1e-8)
print('Jacobi relaxation: {} iterations '.format(ites) +
      'to reach a relative difference of {}'.format(diff))
```

    Jacobi relaxation: 4473 iterations to reach a relative difference of 9.989253685041417e-09

Let's make a gorgeous plot of the final field using the newly minted `plot_3d` function.

```python
# Plot the numerical solution.
plot_3d(x, y, p)
```

Awesome!  That looks pretty good.  But we'll need more than a simple visual check, though. The "eyeball metric" is very forgiving!

## Convergence analysis

### Convergence, Take 1

We want to make sure that our Jacobi function is working properly.  Since we have an analytical solution, what better way than to do a grid-convergence analysis?  We will run our solver for several grid sizes and look at how fast the L2 norm of the difference between consecutive iterations decreases.

Now run Jacobi's method on the Laplace equation using four different grids, with the same exit criterion of $10^{-8}$ each time. Then, we look at the error versus the grid size in a log-log plot. What do we get?

```python
# List of the grid sizes to investigate.
nx_values = [11, 21, 41, 81]

# Create an empty list to record the error on each grid.
errors = []

# Compute the solution and error for each grid size.
for nx in nx_values:
    ny = nx  # same number of points in all directions.
    # Create the gridline locations.
    x = numpy.linspace(0.0, Lx, num=nx)
    y = numpy.linspace(0.0, Ly, num=ny)
    # Set the initial conditions.
    p0 = numpy.zeros((ny, nx))
    p0[-1, :] = numpy.sin(1.5 * numpy.pi * x / Lx)
    # Relax the solution.
    # We do not return the number of iterations or
    # the final relative L2-norm of the difference.
    p, _, _ = laplace_2d_jacobi(p0, rtol=1e-8)
    # Compute the analytical solution.
    p_exact = laplace_solution(x, y, Lx, Ly)
    # Compute and record the relative L2-norm of the error.
    errors.append(l2_norm(p, p_exact))
```

```python
# Plot the error versus the grid-spacing size.
pyplot.figure(figsize=(6.0, 6.0))
pyplot.xlabel(r'$\Delta x$')
pyplot.ylabel('Relative $L_2$-norm\nof the error')
pyplot.grid()
dx_values = Lx / (numpy.array(nx_values) - 1)
pyplot.loglog(dx_values, errors,
              color='black', linestyle='--', linewidth=2, marker='o')
pyplot.axis('equal');
```

Hmm. That doesn't look like 2nd-order convergence, but we're using second-order finite differences.  *What's going on?*  The culprit is the boundary conditions. Dirichlet conditions are order-agnostic (a  set value is a set value), but the scheme we used for the Neumann boundary condition is 1st-order.

Remember when we said that the boundaries drive the problem?  One boundary that's 1st-order completely tanked our spatial convergence.  Let's fix it!

### 2nd-order Neumann BCs

Up to this point, we have used the first-order approximation of a derivative to satisfy Neumann B.C.'s. For a boundary located at $x=0$ this reads,

$$
\begin{equation}
\frac{p^{k+1}_{1,j} - p^{k+1}_{0,j}}{\Delta x} = 0
\end{equation}
$$

which, solving for $p^{k+1}_{0,j}$ gives us

$$
\begin{equation}
p^{k+1}_{0,j} = p^{k+1}_{1,j}
\end{equation}
$$

Using that Neumann condition will limit us to 1st-order convergence.  Instead, we can start with a 2nd-order approximation (the central-difference approximation):

$$
\begin{equation}
\frac{p^{k+1}_{1,j} - p^{k+1}_{-1,j}}{2 \Delta x} = 0
\end{equation}
$$

That seems problematic, since there is no grid point $p^{k}_{-1,j}$.  But no matter … let's carry on. According to the 2nd-order approximation,

$$
\begin{equation}
p^{k+1}_{-1,j} = p^{k+1}_{1,j}
\end{equation}
$$

Recall the finite-difference Jacobi equation with $i=0$:

$$
\begin{equation}
p^{k+1}_{0,j} = \frac{1}{4} \left(p^{k}_{0,j-1} + p^k_{0,j+1} + p^{k}_{-1,j} + p^k_{1,j} \right)
\end{equation}
$$

Notice that the equation relies on the troublesome (nonexistent) point $p^k_{-1,j}$, but according to the equality just above, we have a value we can substitute, namely $p^k_{1,j}$. Ah! We've completed the 2nd-order Neumann condition:

$$
\begin{equation}
p^{k+1}_{0,j} = \frac{1}{4} \left(p^{k}_{0,j-1} + p^k_{0,j+1} + 2p^{k}_{1,j} \right)
\end{equation}
$$

That's a bit more complicated than the first-order version, but it's relatively straightforward to code.

##### Note

Do not confuse $p^{k+1}_{-1,j}$ with `p[-1]`:
`p[-1]` is a piece of Python code used to refer to the last element of a list or array named `p`.  $p^{k+1}_{-1,j}$ is a 'ghost' point that describes a position that lies outside the actual domain.

### Convergence, Take 2

We can copy the previous Jacobi function and replace only the line implementing the Neumann boundary condition.

##### Careful!
Remember that our problem has the Neumann boundary located at $x = L$ and not $x = 0$ as we assumed in the derivation above.

```python
def laplace_2d_jacobi_neumann(p0, maxiter=20000, rtol=1e-6):
    """
    Solves the 2D Laplace equation using Jacobi relaxation method.

    The function assumes Dirichlet condition with value zero
    at all boundaries except at the right boundary where it uses
    a zero-gradient second-order Neumann condition.
    The exit criterion of the solver is based on the relative L2-norm
    of the solution difference between two consecutive iterations.

    Parameters
    ----------
    p0 : numpy.ndarray
        The initial solution as a 2D array of floats.
    maxiter : integer, optional
        Maximum number of iterations to perform;
        default: 20000.
    rtol : float, optional
        Relative tolerance for convergence;
        default: 1e-6.

    Returns
    -------
    p : numpy.ndarray
        The solution after relaxation as a 2D array of floats.
    ite : integer
        The number of iterations performed.
    diff : float
        The final relative L2-norm of the difference.
    """
    p = p0.copy()
    diff = rtol + 1.0  # intial difference
    ite = 0  # iteration index
    while diff > rtol and ite < maxiter:
        pn = p.copy()
        # Update the solution at interior points.
        p[1:-1, 1:-1] = 0.25 * (p[1:-1, :-2] + p[1:-1, 2:] +
                                p[:-2, 1:-1] + p[2:, 1:-1])
        # Apply 2nd-order Neumann condition (zero-gradient)
        # at the right boundary.
        p[1:-1, -1] = 0.25 * (2.0 * pn[1:-1, -2] +
                              pn[2:, -1] + pn[:-2, -1])
        # Compute the residual as the L2-norm of the difference.
        diff = l2_norm(p, pn)
        ite += 1
    return p, ite, diff
```

Again, this is the exact same code as before, but now we're running the Jacobi solver with a 2nd-order Neumann boundary condition. Let's do a grid-refinement analysis, and plot the error versus the grid spacing.

```python
# List of the grid sizes to investigate.
nx_values = [11, 21, 41, 81]

# Create an empty list to record the error on each grid.
errors = []

# Compute the solution and error for each grid size.
for nx in nx_values:
    ny = nx  # same number of points in all directions.
    # Create the gridline locations.
    x = numpy.linspace(0.0, Lx, num=nx)
    y = numpy.linspace(0.0, Ly, num=ny)
    # Set the initial conditions.
    p0 = numpy.zeros((ny, nx))
    p0[-1, :] = numpy.sin(1.5 * numpy.pi * x / Lx)
    # Relax the solution.
    # We do not return the number of iterations or
    # the final relative L2-norm of the difference.
    p, _, _ = laplace_2d_jacobi_neumann(p0, rtol=1e-8)
    # Compute the analytical solution.
    p_exact = laplace_solution(x, y, Lx, Ly)
    # Compute and record the relative L2-norm of the error.
    errors.append(l2_norm(p, p_exact))
```

```python
# Plot the error versus the grid-spacing size.
pyplot.figure(figsize=(6.0, 6.0))
pyplot.xlabel(r'$\Delta x$')
pyplot.ylabel('Relative $L_2$-norm\nof the error')
pyplot.grid()
dx_values = Lx / (numpy.array(nx_values) - 1)
pyplot.loglog(dx_values, errors,
              color='black', linestyle='--', linewidth=2, marker='o')
pyplot.axis('equal');
```

Nice!  That's much better.  It might not be *exactly* 2nd-order, but it's awfully close. (What is ["close enough"](http://ianhawke.github.io/blog/close-enough.html) in regards to observed convergence rates is a thorny question.)

Now, notice from this plot that the error on the finest grid is around $0.0002$. Given this, perhaps we didn't need to continue iterating until a target difference between two solutions of $10^{-8}$. The spatial accuracy of the finite difference approximation is much worse than that! But we didn't know it ahead of time, did we? That's the "catch 22" of iterative solution of systems arising from discretization of PDEs.

## Final word

The Jacobi method is the simplest relaxation scheme to explain and to apply. It is also the *worst* iterative solver! In practice, it is seldom used on its own as a solver, although it is useful as a smoother with multi-grid methods. As we will see in the [third lesson](https://nbviewer.jupyter.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/05_relax/05_03_Iterate.This.ipynb) of this module, there are much better iterative methods! But first, let's play with [Poisson's equation](https://nbviewer.jupyter.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/05_relax/05_02_2D.Poisson.Equation.ipynb).

---
###### The cell below loads the style of the notebook

```python
from IPython.core.display import HTML
css_file = '../../styles/numericalmoocstyle.css'
HTML(open(css_file, 'r').read())
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Nixie+One' rel='stylesheet' type='text/css'>
<link href='https://fonts.googleapis.com/css?family=Source+Code+Pro' rel='stylesheet' type='text/css'>
<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: rgb(245,245,245);
}

div.cell { /* set cell width */
    width: 750px;
}

div #notebook { /* centre the content */
    background: #fff; /* white background for content */
    width: 1000px;
    margin: auto;
    padding-left: 0em;
}

#notebook li { /* More space between bullet points */
    margin-top:0.8em;
}

/* draw border around running cells */
div.cell.border-box-sizing.code_cell.running {
    border: 1px solid #111;
}

/* Put a solid color box around each cell and its output, visually linking them*/
div.cell.code_cell {
    background-color: rgb(256,256,256);
    border-radius: 0px;
    padding: 0.5em;
    margin-left:1em;
    margin-top: 1em;
}

div.text_cell_render{
    font-family: 'Alegreya Sans' sans-serif;
    line-height: 140%;
    font-size: 125%;
    font-weight: 400;
    width:600px;
    margin-left:auto;
    margin-right:auto;
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Nixie One', serif;
    font-style:regular;
    font-weight: 400;
    font-size: 45pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}

.text_cell_render h2 {
    font-family: 'Nixie One', serif;
    font-weight: 400;
    font-size: 30pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.1em;
    margin-top: 0.3em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Nixie One', serif;
    margin-top:16px;
    font-size: 22pt;
    font-weight: 600;
    margin-bottom: 3px;
    font-style: regular;
    color: rgb(102,102,0);
}

.text_cell_render h4 {    /*Use this for captions*/
    font-family: 'Nixie One', serif;
    font-size: 14pt;
    text-align: center;
    margin-top: 0em;
    margin-bottom: 2em;
    font-style: regular;
}

.text_cell_render h5 {  /*Use this for small titles*/
    font-family: 'Nixie One', sans-serif;
    font-weight: 400;
    font-size: 16pt;
    color: rgb(163,0,0);
    font-style: italic;
    margin-bottom: .1em;
    margin-top: 0.8em;
    display: block;
}

.text_cell_render h6 { /*use this for copyright note*/
    font-family: 'PT Mono', sans-serif;
    font-weight: 300;
    font-size: 9pt;
    line-height: 100%;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

.CodeMirror{
    font-family: "Source Code Pro";
    font-size: 90%;
}

.alert-box {
    padding:10px 10px 10px 36px;
    margin:5px;
}

.success {
    color:#666600;
    background:rgb(240,242,229);
}
</style>


# Nonlinear Equations and their Roots
## CH EN 2450 - Numerical Methods
**Prof. Tony Saad (www.tsaad.net)
Department of Chemical Engineering
University of Utah**

The purpose of root finding methods for nonlinear functions is to find the roots - or values of the independent variable, e.g. x - that make the nonlinear function equal to zero. The general form of a nonlinear root finding proposition is given as:

find $x$ such that
\begin{equation}
f(x) = 0
\end{equation}

Alternatively, a problem may be presented as:

find $x$ such that
\begin{equation}
f(x) = a
\end{equation}

then, one redefines this into what is called "residual" form
\begin{equation}
r(x) \equiv f(x) - a
\end{equation}
and reformulates the problem to find $x$ such that
\begin{equation}
r(x) = 0
\end{equation}

In all examples below, we will explore the roots of the following function
\begin{equation}
\ln(x) + \cos(x)e^{-0.1x} = 2
\end{equation}
or, in residual form
\begin{equation}
r(x) \equiv \ln(x) + \cos(x)e^{-0.1x} - 2 = 0
\end{equation}
This function has three roots, 5.309, 8.045, and 10.02

```python
import numpy as np
%matplotlib inline
%config InlineBackend.figure_format = 'svg'
import matplotlib.pyplot as plt
from scipy.optimize import fsolve
```

```python
res = lambda x: np.log(x) + np.cos(x)*np.exp(-0.1*x)-2.0
x = np.linspace(4,15,200)
plt.grid()
plt.axhline(y=0,color='k')
plt.plot(x,res(x))
```

    [<matplotlib.lines.Line2D at 0x101282f9e8>]

# Root Finding Methods

There are two classes of methods to find roots of equations:
1. Closed domain methods,
2. Open domain methods

Closed domain methods work by bracketing the root while open domain methods can start with an arbitrary initial guess.

# Close Domain Methods

## Bisection Method
Perhaps the most popular and intuitive root finding method, the Bisection method works by first bracketing a root and then successively improving the bracket by taking the midway point. An algorithm looks like the following
1. Choose values $a$ and $b$ such that the root, $x_0$ is $a \leq x_0 \leq b$
2. Calculate $c = \frac{a+b}{2}$ as the midway point between $a$ and $b$
3. Check which side the root is: if $f(a)\times f(c) < 0$ then $b = c$ else $a=c$
4. Check for convergence and repeat as necessary

Below is an example implementation of the bisection method

```python
def bisect(f,a,b,tol, maxiter):
    err = tol + 100
    niter = 0
    print('{:<12} {:<12} {:<12} {:<12}'.format('Iteration','a','b','error'))
    while err > tol and niter < maxiter:
        niter +=1
        c = (a + b)/2.0
        fc = f(c)
        fa = f(a)
        if (fa * fc < 0.0):
            b = c
        else:
            a = c
        err = abs(fc)
        print('{:<12} {:<12} {:<12} {:<12}'.format(niter, round(a,6), round(b,6), round(err,10)))
    print('Iterations=',niter,' Root=',c)
    return c
```

```python
bisect(res,4,5.5,1e-5,10)
```

    Iteration    a            b            error
    1            4.75         5.5          0.418471165
    2            5.125        5.5          0.1256704432
    3            5.125        5.3125       0.0020525841
    4            5.21875      5.3125       0.0599408749
    5            5.265625     5.3125       0.0284566567
    6            5.289062     5.3125       0.0130777303
    7            5.300781     5.3125       0.0054812004
    8            5.306641     5.3125       0.0017064285
    9            5.306641     5.30957      0.0001750523
    10           5.308105     5.30957      0.0007651951
    Iterations= 10  Root= 5.30810546875

    5.30810546875

## Method of False Position (Regula-Falsi)
The Methods of False Position or Regula-Falsi takes into consideration how close a guess might be to a root. It requires two initial guesses that bracket the root but instead of cutting the bracket in half, the Falsi method connects the two guesses via a straight line and then finds the point at which this straight line intersects the x-axis and uses that as a new guess.

Here's the algorithm for the Regula-Falsi method:
1. Choose values $a$ and $b$ such that the root, $x_0$ is $a \leq x_0 \leq b$
2. Calculate the slope of the line connecting $a$ and $b$: $m = \frac{f(b) - f(a)}{b-a}$
3. Find the point at which this line intersects the x-axis: $c = b - \frac{f(b)}{m}$
3. Check which side the root is: if $f(a)\times f(c) < 0$ then $b = c$ else $a=c$
4. Check for convergence and repeat as necessary

```python
def falsi(f,a,b,tol):
    niter = 0
    err = tol + 100
    print('{:<12} {:<12} {:<12} {:<12}'.format('Iteration','a','b','error'))

    while err > tol:
        fa = f(a)
        fb = f(b)
        m = (fb - fa)/(b - a)
        c = b - fb/m
        fc = f(c)
        err = abs(fc)
        if fc * fa < 0.0:
            b = c
        else:
            a = c
        err = abs(fc)
        print('{:<12} {:<12} {:<12} {:<12}'.format(niter, round(a,6), round(b,6), round(err,10)))
        niter += 1
    print('Iterations:', niter, 'Root=',c)
    return c
```

```python
falsi(res,4,5.5,1e-5)
```

    Iteration    a            b            error
    0            4            5.353774     0.0280786749
    1            4            5.318575     0.0059333086
    2            4            5.311179     0.0012065276
    3            4            5.309677     0.0002433628
    4            4            5.309374     4.90066e-05
    5            4            5.309313     9.8653e-06
    Iterations: 6 Root= 5.30931285069788

    5.30931285069788

# Open Domain Methods
Closed domain methods require two initial guesses that bracket a root and are guaranteed to converge to the root, but in general are not practical when the root's location is unknown. Open domain methods relax this requirement and do not need initial guesses that bracket a root.

## The Secant Method
The secant method is identical to the regula-falsi method but does not require the initial guesses to bracket a root. Here's an algorithm for the secant method:
1. Choose values $a$ and $b$ that do not necessarily bracket a root
2. Calculate the slope of the line connecting $a$ and $b$: $m = \frac{f(b) - f(a)}{b-a}$
3. Find the point at which this line intersects the x-axis: $c = b - \frac{f(b)}{m}$
3. Set $a = b$ and $b = c$
4. Check for convergence and repeat as necessary

```python
def secant(f,a,b,tol):
    niter = 0
    err = tol + 100
    print('{:<12} {:<12} {:<12} {:<12}'.format('Iteration','a','b','error'))

    while err > tol:
        fa = f(a)
        fb = f(b)
        m = (fb - fa)/(b - a)
        c = b - fb/m
        fc = f(c)
        err = abs(fc)
        a = b
        b = c
        print('{:<12} {:<12} {:<12} {:<12}'.format(niter, round(a,6), round(b,6), round(err,10)))
        niter += 1
    print('Iterations:', niter, 'Root=',c)
    return c
```

```python
secant(res,7,7.5,1e-5)
```

    Iteration    a            b            error
    0            7.5          8.130598     0.0254859413
    1            8.130598     8.051866     0.0019750738
    2            8.051866     8.045252     4.76383e-05
    3            8.045252     8.045407     7.92e-08
    Iterations: 4 Root= 8.04540742136555

    8.04540742136555

## Newton's Method
Newton's method is one of the most popular open domain nonlinear solvers. It is based on a two-term approximation of the Taylor series of a function $f(x)$. Given an initial guess $x_0$, Newton's method is implemented in the following steps:
1. Choose $x_0$ as an initial guess
2. Compute $f(x^0)$ and $f'(x^0)$
3. Compute $x_1 = x_0 - \frac{f(x_0)}{f'(x_0)}
3. Set $x_0 = x_1$
4. Check for convergence and repeat as necessary

```python
def newton(f,df,x0,tol):
    niter = 0
    err = tol + 100
    while err > tol and niter < 100:
        x1 = x0 - f(x0)/df(x0)
        x0 = x1
        err = abs(f(x0))
        niter += 1
    print('Iterations:', niter, 'Root=',x1)
    return x1
```

In many cases, the derivative is not known and one must use a finite difference approximation to the derivative.

```python
def newton2(f,x0,tol):
    niter = 0
    err = tol + 100
    while err > tol and niter < 100:
        delta = 1e-4 * x0 + 1e-12
        df = (f(x0 + delta) - f(x0))/delta
        x1 = x0 - f(x0)/df
        x0 = x1
        err = abs(f(x0))
        niter += 1
    print('Iterations:', niter, 'Root=',x1)
    return x1
```

```python
newton2(res,1,1e-5)
```

    Iterations: 7 Root= 5.309297481151985

    5.309297481151985

```python
import urllib
import requests
from IPython.core.display import HTML
def css_styling():
    styles = requests.get("https://raw.githubusercontent.com/saadtony/NumericalMethods/master/styles/custom.css")
    return HTML(styles.text)
css_styling()

```

CSS style adapted from https://github.com/barbagroup/CFDPython. Copyright (c) Barba group
<link href='http://fonts.googleapis.com/css?family=Merriweather' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Bitter' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Oxygen' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Lora' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Fenix' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>


#  Arbitrary Periodic Forcing (Fourier Series)

## Sympy

SymPy is a Python package for symbolic computing. It can do many symbolic operations, for instance, integration, differentiation, linear algebra, etc. See http://sympy.org for more details of the features and the documentation. Today we will cover how to do integrals using SymPy and use it to find the Fourier series that represents a sawtooth function.

```python
import sympy as sm
```

The function `init_printing()` enables LaTeX based rendering in the Jupyter notebook of all SymPy objects.

```python
sm.init_printing()
```

Symbols can be created by using the `symbols()` function.

```python
x, y, z = sm.symbols('x, y, z')
```

```python
x, y, z
```

The `integrate()` function allows you to do symbolic indefinite or definite integrals. Note that the constants of integration are not included in indefinite integrals.

```python
sm.integrate(x * y, x)
```

The `Integral` class creates and unevaluated integral, where as the `integrate()` function automatically evaluates the integral.

```python
expr = sm.Integral(x * y, x)
expr
```

To evaluate the unevaluated form you call the `.doit()` method. Note that all unevaluated SymPy objects have this method.

```python
expr.doit()
```

This shows how to create an unevaluated definite integral, store it in a variable, and then evaluate it.

```python
expr = sm.Integral(x * y, (x, 0, 5))
expr
```

```python
expr.doit()
```

# Fourier Coefficients for the Sawtooth function

Now let's compute the Fourier coefficients for a saw tooth function. The function that describes the saw tooth is:

$$
F(t) =
\begin{cases}
      A \left( \frac{4t}{T} - 1 \right) & 0 \leq t \leq T/2 \\
      A \left( 3 - \frac{4t}{t} \right) & T/2 \leq t \leq T
\end{cases}
$$

where:

- $A$ is the amplitude of the saw tooth
- $T$ is the period of the saw tooth
- $\omega_T$ is the frequency of the saw tooth, i.e. $\omega_T = \frac{2\pi}{T}$
- $t$ is time

This is a piecewise function with two parts from $t=0$ to $t=T$.

```python
A, T, wT, t = sm.symbols('A, T, omega_T, t', real=True, positive=True)
A, T, wT, t
```

The first Fourier coefficient $a_0$ describes the average value of the periodic function. and is:

$$a_0 = \frac{2}{T} \int_0^T F(t) dt$$

This integral will have to be done in two parts:

$$a_0 = a_{01} + a_{02} = \frac{2}{T} \int_0^{T/2} F(t) dt + \frac{2}{T} \int_{T/2}^T F(t) dt$$

These two integrals are evaluated below. Note that $a_0$ evaluates to zero. This is because the average of our function is 0.

```python
ao_1 = 2 / T * sm.Integral(A * (4 * t / T - 1), (t, 0, T / 2))
ao_1
```

```python
ao_1.doit()
```

```python
ao_2 = 2 / T * sm.Integral(A * (3 - 4 * t / T), (t, T / 2, T))
ao_2
```

```python
ao_2.doit()
```

But SymPy can also handle piecewise directly. The following shows how to define a piecewise function.

```python
F_1 = A * (4 * t / T - 1)
F_2 = A * (3 - 4 * t / T)
```

```python
F = sm.Piecewise((F_1, t<=T/2),
                 (F_2, T/2<=t))
F
```

The integral can be taken of the entire piecewise function in one call.

```python
sm.integrate(F, (t, 0, T))
```

Now the Fourier coefficients $a_n$ and $b_n$ can be computed.

$$
a_n = \frac{2}{T}\int_0^T F(t) \cos n\omega_Tt dt \\
b_n = \frac{2}{T}\int_0^T F(t) \sin n\omega_Tt dt
$$

```python
n = sm.symbols('n', real=True, positive=True)
```

For $a_n$:

```python
an = 2 / T * sm.Integral(F * sm.cos(n * wT * t), (t, 0, T))
an
```

```python
an.doit()
```

This can be simplified:

```python
an = an.doit().simplify()
an
```

Now substitute the $2\pi/T$ for $\omega_T$.

```python
an = an.subs({wT: 2 * sm.pi / T})
an
```

Let's see how this function varies with increasing $n$. We will use a loop but the SymPy expressions will not automatically display because they are inside a loop. So we need to use SymPy's `latex()` function and the IPython display tools. SymPy's `latex()` function  transforms the SymPy expression into a string of matching LaTeX commands.

```python
sm.latex(an, mode='inline')
```

    '$\\frac{A}{2 \\pi^{2} n^{2}} \\left(- 2 \\pi n \\sin{\\left (2 \\pi n \\right )} + 8 \\cos{\\left (\\pi n \\right )} - 4 \\cos{\\left (2 \\pi n \\right )} - 4\\right)$'

The `display()` and `LaTeX()` functions then turn the LaTeX string in to a displayed version.

```python
from IPython.display import display, Latex
```

Now we can see how $a_n$ varies with $n=1,2,\ldots$.

```python
for n_i in range(1, 6):
    ans = an.subs({n: n_i})
    display(Latex('$a_{} = $'.format(n_i) + sm.latex(ans, mode='inline')))
```

$a_1 = $$- \frac{8 A}{\pi^{2}}$

$a_2 = $$0$

$a_3 = $$- \frac{8 A}{9 \pi^{2}}$

$a_4 = $$0$

$a_5 = $$- \frac{8 A}{25 \pi^{2}}$

For even $n$ values the coefficient is zero and for even values it variers with the inverse of $n^2$. More precisely:

$$
a_n =
\begin{cases}
0 & \textrm{if }n\textrm{ is even} \\
-\frac{8A}{n^2\pi^2} & \textrm{if }n\textrm{ is odd}
\end{cases}
$$

SymPy can actually reduce this further if your set the assumption that $n$ is an integer.

```python
n = sm.symbols('n', real=True, positive=True, integer=True)
an = 2 / T * sm.Integral(F * sm.cos(n * wT * t), (t, 0, T))
an = an.doit().simplify()
an.subs({wT: 2 * sm.pi / T})
```

The odd and even versions can be computed by setting the respective assumptions.

```python
n = sm.symbols('n', real=True, positive=True, integer=True, odd=True)
an = 2 / T * sm.Integral(F * sm.cos(n * wT * t), (t, 0, T))
an = an.doit().simplify()
an.subs({wT: 2 * sm.pi / T})
```

Note that $b_n$ is always zero:

```python
bn = 2 / T * sm.Integral(F * sm.sin(n * wT * t), (t, 0, T))
bn
```

```python
bn.doit().simplify().subs({wT: 2 * sm.pi / T})
```

# Random questions questions from class

The source code for any function is shown with the double question mark.

```python
sm.Integral??
```

The location of the file the function comes from can be found by accessing the `__file__` attribute.

```python
sm.__file__
```

    '/home/moorepants/miniconda3/lib/python3.5/site-packages/sympy/__init__.py'

SymPy can handle unit for you too.

```python
from sympy.physics.units import meter, kilogram, second
```

```python
a = 10 * second
a
```

```python
b = 5 * meter
```

```python
c = 6 * kilogram
```

```python
a, b, c
```

```python
expr = c * b / a**2
expr
```

```python
from sympy.physics.units import newton
```

```python
5 * newton
```

# Numerical evalution of the Fourier Series

Now the Fourier coefficients can be used to plot the approximation of the saw tooth forcing function.

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib notebook
```

The following function plots the actual sawtooth function. It does it all in one line by cleverly using the absolute value and the modulo functions.

```python
def sawtooth(A, T, t):
    return (4 * A / T) * (T / 2 - np.abs(t % T - T / 2) ) - A
```

```python
A = 1
T = 2
t = np.linspace(0, 5, num=500)
```

```python
plt.figure()

plt.plot(t, sawtooth(A, T, t));
```

    <IPython.core.display.Javascript object>

# Exercise

Write a function that computes the Fourier approximation of the sawtooth function for a given value of $n$, i.e. using a finite number of terms. Then plot it for $n=2, 4, 6, 8, 10$ on top of the actual sawtooth function. How many terms of the infinite series are needed to get a good sawtooth?

```python
def sawtooth_approximation(n, A, T, t):
    # type code here
    return
```

# Exercise

Now, using the Fourier series you've developed compute the response of a linear system to the sawtooth function and plot the steady state trajectory of $x$ give:

$$m\ddot{x} + c\dot{x} + kx = F(t)$$

Use:

- $m$: 100 kg
- $\omega_n$: 10 rad/s
- $\zeta$: 0.01
- $T=\frac{2\pi}{5 rad/s}$
- $A=1000N$

Investigate the difference in steady state response for different approximations of $F(t)$, i.e. various $n$, and report your findings in a markdown cell.

```python

```


# K Means Algorithm

K-Means is a popular algorithm for clustering in data science. Using non-parametrical techniques is great to define categories from a set of data. But, How it really works? We have a lot of packages showing the immediate way of using it and few information on the web about it.

In this notebook we are going to program it manually to understand what is going on and then you can implement it using other languages. I based the methodology on Christopher Bishop's book: Pattern Recognition and Machine Learning (2009). You can check the bibliography at the end.

## Initialize the enviroment for the algorithm

To test our algorithm, we are going to use the iris dataset to see if we can classify all the iris species. The repository is available from sklearn. Just call the package and database.

```python
from sklearn import datasets
iris = datasets.load_iris()
```

Then, we are just to transform it into a Data Frame with pandas to do all easier.

```python
import pandas as pd
iris=pd.DataFrame(iris.data)
iris.head()
```

      0
      1
      2
      3

      0
      5.1
      3.5
      1.4
      0.2

      1
      4.9
      3.0
      1.4
      0.2

      2
      4.7
      3.2
      1.3
      0.2

      3
      4.6
      3.1
      1.5
      0.2

      4
      5.0
      3.6
      1.4
      0.2

Is necessary for K-Means to select the number $k$ of clusters to work. For this particular case we want to set up 3 classes

```python
K=3
```

So, will define some initial random values for this $\mu_k$ clusters. Let's see first the range of each dimension to find the limits for each $\mu_k$.

```python
import numpy as np #To calculate min and max
```

```python
zero_min=np.min(iris.loc[:,0])
zero_max=np.max(iris.loc[:,0])

one_min=np.min(iris.loc[:,1])
one_max=np.max(iris.loc[:,1])

two_min=np.min(iris.loc[:,2])
two_max=np.max(iris.loc[:,2])

three_min=np.min(iris.loc[:,3])
three_max=np.max(iris.loc[:,3])
```

We are going to work with a transformation of each value, just having the integer from the multiplication by 10. This is to create the random vector easier using numpy.

```python
print(int(zero_min*10),int(zero_max*10))
```

    43 79

```python
zero_min=int(zero_min*10)
zero_max=int(zero_max*10)

one_min=int(one_min*10)
one_max=int(one_max*10)

two_min=int(two_min*10)
two_max=int(two_max*10)

three_min=int(three_min*10)
three_max=int(three_max*10)
```

We have now all limits for each dimension in the dataset. Then, we can establish the initial values for $\mu_k$, we are going to insert all this values into a new vector called mu

```python
np.random.seed(1500) #Define first a random seed to have the same results as me
```

```python
mu=[]
for i in range(0,K):
    zeros=np.random.randint(zero_min,zero_max)
    ones=np.random.randint(one_min,one_max)
    twos=np.random.randint(two_min,two_max)
    threes=np.random.randint(three_min,three_max)

    mu.append([zeros,ones,twos,threes])
```

Then, we are going to transform centroinds into a numpy array and divide by 10 to have the $\mu_k$ to have the right values for the dataset

```python
mu=np.array(mu)/10
mu
```

    array([[6.2, 3.1, 1.1, 1.5],
           [4.5, 2.9, 2.5, 2. ],
           [5.9, 3. , 2.7, 0.8]])

Will be necessary to create a matrix to add $k$ columns. We will call them as $r_{n,k}$. Where $n$ is one single dataset observation and $k$ corresponds to each cluster defined. Consider $r_{n,k}\in\{0,1\}$

This should be look as:

```python
r=np.zeros(iris.shape[0]*K).reshape(iris.shape[0],K)
pd.DataFrame(r).head()
```

      0
      1
      2

      0
      0.0
      0.0
      0.0

      1
      0.0
      0.0
      0.0

      2
      0.0
      0.0
      0.0

      3
      0.0
      0.0
      0.0

      4
      0.0
      0.0
      0.0

## How the algorithm works?

Now, the complex step, or maybe not. We are going to define a function $J$, this will be called as distortion measure and is given by:

\begin{equation}
J=\sum_{n=1}^{N}\sum_{k=1}^{K}r_{n,k}||x_{n}-\mu_{k}
||^2
\end{equation}

Where $N$ is the total number of observations in the dataset, $K$ is the total clusters defined, $x_n$ is each observation  and we know what $\mu_k $ and $r_{n,k}$ means. As we can see, all the dataset will be used to find the centroid vector given by $\mu_k $. There is other way to estimate the best centroids using the Robbins-Monro procedure, applying an on-line stochastic algorithm, but this will be in other moment.

I. Our first objective is to minimize $J$ respect to $r_{n,k}$, keeping $\mu_k$ fixed. And considering the next rule:

\begin{equation}
  r_{n,k}=\begin{cases}
    1, & \text{if $k=argmin_j||x_{n}-\mu_{j}||^2$}.\\
    0, & \text{otherwise}.
  \end{cases}
\end{equation}

II. Then, we are going to minimize again $J$, but respect to $\mu_k$, keeping $r_{n,k}$ fixed. Consider the next formula:

\begin{equation}
\mu_k=\frac{\sum_n r_{n,k} x_n}{\sum_n r_{n,k}}
\end{equation}

We will do the above steps until convergence.

```python
plot_counter=0
J_plot=[]
for i in range(1,10): #For this case we only want as maximun 10 iterations
    J=0 #Initializing the distortion measure

    #This cicle will be to create all values for r_{n,k} using the database
    for n in range(0,iris.shape[0]):
        x=np.array(iris.iloc[n,0:4])
        mins=[]

        for k in range(0,K):
            distance=np.linalg.norm(x-mu[k])# Using the Euclidean distance with numpy
            mins.append(distance**2) #Just add all distances into the new vector mins, this vector only keep them

        k_r=np.argmin(np.array(mins))#Here we apply the rule shown in I.
        r[n,k_r]=1 #For the otherwise case, at the end we initialize again the cicle to have a vector of zeros.

    #The next lines will calculate the distortion measure and keep the value of J for plot.
    for n in range(0,iris.shape[0]):
        for k in range(0,K):
            distance=np.linalg.norm(x-mu[k])
            J=r[n,k]*(distance**2)+J
    J_plot.append(J)

    #Now, we are going to update the centroids with all the mu using II. This is why is a K-Means algorithm
    for k in range(0,K):
        count_1,count_2=0,0
        for n in range(0,iris.shape[0]):
            x=np.array(iris.iloc[n,0:4])
            count_1=r[n,k]*x+count_1
            count_2=r[n,k]+count_2
        mu[k]=count_1/count_2

    print(J)
    #This simple rule is just to break the cicle if the previous J and the actual J don't change considerably
    if i>1 and abs(J-J_prev)<0.5:
        break
    J_prev=J
    plot_counter=1+plot_counter
    #As I said before, this reinitialize the r vector with zeros
    r=np.zeros(iris.shape[0]*K).reshape(iris.shape[0],K)
```

    1100.879999999999
    1077.8460571113642
    870.987494853379
    869.126836320042
    923.8112637226989
    925.4275481881676
    925.7231759999969

Now we are going to call matplotlib to se how the J is updated and the distortion measure decreases

```python
import matplotlib.pyplot as plt
```

```python
plt.plot(J_plot,linestyle='-', marker='o',color='green')
plt.ylabel('J: Distortion measure')
plt.xlabel('Iterations')
plt.show()
```

As we saw, the convergence of J is assured, but is more common to converge to a local minimum than a global one for J. This was studied by MacQueen (1967). The final vector for all centroids or mu, will be:

```python
mu
```

    array([[5.006     , 3.428     , 1.462     , 0.246     ],
           [5.82962963, 2.73148148, 4.31481481, 1.39259259],
           [6.76956522, 3.03695652, 5.6       , 2.00869565]])

This how looks the dataset if we only graph the features 0 and 1

```python
plt.scatter(iris[0],iris[1],marker='.')
plt.show()
```

If we add the centroids $\mu$ to the scatterplot we will see where are the main zones of each K category

```python
mu=pd.DataFrame(mu)
plt.scatter(iris[0],iris[1],marker='.')
plt.scatter(mu[0],mu[1],s=100,marker='x',color='red')
plt.show()
```

Now we want to know which point belongs to each of the $K=3$ categories. We need to calculate the distance of each point to each centroid $\mu$ and identify the closest. This information already exist in the $r_{n,k}$ vector after the fit. If there are a one new observation just calculate the distances of the new $x_i$ with $\mu_k$ for $k=1,2,3$ in this case.

```python
r_new=[]
for row in r:
    r_new.append(np.argmax(row)+1)
iris[4]=r_new
```

After creating a new vector just to set the correct labels we just plot each category. The "x" mark show the centroid and the coloured area next to it is the category.

```python
plt.scatter(iris[0],iris[1],c=iris[4],marker='.')
plt.scatter(mu[0],mu[1],s=100,marker='x',color='red')
plt.show()
```

It seems that the algorithm don't classify the closest points as part of a $\mu_k$ category. Remember that we are just plotting in 2D, the complete dataset belongs to 4D and we are using the euclidean distance. Unfortunately we just can see 3D as much.

```python
from mpl_toolkits.mplot3d import Axes3D

fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')

x1 =iris[0]
y1 =iris[1]
z1 =iris[2]

x2=mu[0]
y2=mu[1]
z2=mu[2]

ax.scatter(x1, y1, z1, c=iris[4], marker='.')
ax.scatter(x2, y2, z2, c='red', s=200, marker='x')

ax.set_xlabel('feature 0')
ax.set_ylabel('feature 1')
ax.set_zlabel('feature 2')

plt.show()
```

## Bibliography

Bishop C.M (2009). Pattern recognition and machine learning. Springer Science & Business Media, pp. 424-426

MacQueen, J (1967). Some methods for classification and analysis of multivariate observations. In L.M LeCam and J. Neyman (Eds.), Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, Volume I, pp. 281-297. University of California Press


```python
# HIDDEN
from datascience import *
from prob140 import *
import numpy as np
import matplotlib.pyplot as plt
plt.style.use('fivethirtyeight')
%matplotlib inline
import math
from scipy import stats
from sympy import *
init_printing()
```

### Independence ###
Random variables $X$ and $Y$ are *independent* if
$$
P(X \in A, Y \in B) = P(X \in A)P(Y \in B)
$$
for all intervals $A$ and $B$.

Let $X$ have density $f_X$, let $Y$ have density $f_Y$, and suppose $X$ and $Y$ are independent. Then if $f$ is the joint density of $X$ and $Y$,

\begin{align*}
f(x, y)dxdy &\sim P(X \in dx, Y \in dy) \\
&= P(X \in dx)P(Y \in dy) ~~~~~ \text{(independence)} \\
&= f_X(x)dx f_Y(y)dy \\
&= f_X(x)f_Y(y)dxdy
\end{align*}

Thus if $X$ and $Y$ are independent then their joint density is given by

$$
f(x, y) = f_X(x)f_Y(y)
$$

This is the *product rule for densities*: the joint density of two independent random variables is the product of their densities.

### Two Independent Normal Random Variables ###
For example, suppose $X$ and $Y$ are i.i.d. standard normal random variables. Then their joint density is given by

$$
f(x, y) = \frac{1}{\sqrt{2\pi}} e^{-\frac{1}{2}x^2} \cdot \frac{1}{\sqrt{2\pi}} e^{-\frac{1}{2}y^2}, ~~~~ -\infty < x, y < \infty
$$

Equivalently,
$$
f(x, y) = \frac{1}{2\pi} e^{-\frac{1}{2}(x^2 + y^2)}, ~~~~ -\infty < x, y < \infty
$$

Here is a graph of the joint density surface.

```python
def indep_standard_normals(x,y):
    return 1/(2*math.pi) * np.exp(-0.5*(x**2 + y**2))

Plot_3d((-4, 4), (-4, 4), indep_standard_normals, rstride=4, cstride=4)
```

Notice the circular symmetry of the surface. This is because the formula for the joint density involves the pair $(x, y)$ through the expression $x^2 + y^2$.

Notice also that $P(X = Y) = 0$, as the probability is the volume over a line. This is true of all pairs of independent random variables with a joint density: $P(X = Y) = 0$. So for example $P(X > Y) = P(X \ge Y)$. You don't have to worry about whether or not to the inequality should be strict.

### The Larger of Two Independent Exponential Random Variables ###
Let $X$ and $Y$ be independent random variables. Suppose $X$ has the exponential $(\lambda)$ distribution $Y$ has the exponential $(\mu)$ distribution. The goal of this example is to find $P(Y > X)$.

By the product rule, the joint density of $X$ and $Y$ is given by

$$
f(x, y) ~ = ~ \lambda e^{\lambda x} \mu e^{-\mu y}, ~~~~ x > 0, ~ y > 0
$$

The graph below shows the joint density surface in the case $\lambda = 0.5$ and $\mu = 0.25$, so that $E(X) = 2$ and $E(Y) = 4$.

```python
def independent_exp(x, y):
    return 0.5 * 0.25 * np.e**(-0.5*x - 0.25*y)

Plot_3d((0, 10), (0, 10), independent_exp)
```

To find $P(Y > X)$ we must integrate the joint density over the upper triangle of the first quadrant, a portion of which is shown below.

```python
# HIDDEN
xx = np.arange(0, 10.1, 0.1)
yy = 10*np.ones(len(xx))
plt.fill_between(xx, xx, yy, alpha=0.3)
plt.axes().set_aspect('equal')
plt.xlabel('$x$')
plt.ylabel('$y$', rotation=0)
plt.title('$Y > X$ (portion of infinite region)');
```

The probability is therefore
$$
P(Y > X) ~ = ~ \int_0^\infty \int_x^\infty \lambda e^{-\lambda x} \mu e^{-\mu y} dy dx
$$

Let's do this in `SymPy` and then take another look at the calculation. Notice that `SymPy` doesn't want to write the joint density with negative exponents.

```python
declare('x', 'y', 'lamda', 'mu', positive=True)
```

```python
f_X = lamda * exp(-lamda * x)

f_Y = mu * exp(-mu * y)

jt_density = f_X * f_Y
jt_density
```

```python
p_Y_greater_than_X = Integral(jt_density, (y, x, oo), (x, 0, oo))
p_Y_greater_than_X
```

```python
p_Y_greater_than_X.doit()
```

Easy algebra, or the `rearrange_1` algebraic simplification method, provides an alterative form of the answer:

```python
rearrange_1(p_Y_greater_than_X.doit())
```

Thus
$$
P(Y > X) ~ = ~ \frac{\lambda}{\lambda + \mu}
$$
Analogously,
$$
P(X > Y) ~ = ~ \frac{\mu}{\lambda + \mu}
$$

Notice that the two chances are proportional to the parameters. This is consistent with intuition if you think of $X$ and $Y$ as two lifetimes. If $\lambda$ is large, the corresponding lifetime $X$ is likely to be short, and therefore $Y$ is likely to be larger than $X$ as the formula implies.

If $\lambda = \mu$ then $P(Y > X) = 1/2$ which you can see by symmetry since $P(X = Y) = 0$.

It's instructive to do the integral for calculating $P(Y > X)$ without resorting to `SymPy`. The calculation shows how you can use probability theory to do calculus.

\begin{align*}
P(Y > X) &= \int_0^\infty \int_x^\infty \lambda e^{-\lambda x} \mu e^{-\mu y} dy dx \\ \\
&= \int_0^\infty \lambda e^{-\lambda x} \big{(} \int_x^\infty \mu e^{-\mu y} dy\big{)} dx \\ \\
&= \int_0^\infty \lambda e^{-\lambda x} e^{-\mu x} dx ~~~~~~ \text{(survival function of } Y\text{)} \\ \\
&= \frac{\lambda}{\lambda + \mu} \int_0^\infty (\lambda + \mu) e^{-(\lambda + \mu)x} dx \\ \\
&= \frac{\lambda}{\lambda + \mu} ~~~~~~~ \text{(total integral of exponential }
(\lambda + \mu) \text{ density is 1)}
\end{align*}

```python

```