# Fitting a model to data with MCMC #

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import plot_helper as plot_helper
import pandas as pd
import emcee # 2.2.1
import corner
import progressbar
import scipy.optimize as op
```

This is a very short introduction to using MCMC for fitting a model to data; see references below for much more detailed examples.

### The ground truth

Let us suppose we are interested in some physical process which relates the quantities $x$ and $y$ as

\begin{equation}
y=y_{max}\frac{x}{x+K},
\end{equation}
with true parameter values $y_{max}=1$ and $K=2$.

```python
def model(x,ymax,K):
    return ymax*x/(x+K)
ymax=1
K=2
x=np.linspace(0,10,50)
y=model(x,ymax,K)
plot_helper.plot1(x,y,title='Ground truth')
```

Suppose we make some observations to measure $y_{max}$ and $K$.

```python
N=9
xobs=(np.random.rand(N))*10
yerrtrue=0.03*np.random.randn(N) # normally distributed errors
yobs=model(xobs,ymax,K)+yerrtrue
yerr=yerrtrue*1 # Our estimated error is not necessarily equal to the true error
plot_helper.plot2(x,y,xobs,yobs,yerr,title='Ground truth+observations')
```

We would like to estimate the posterior probability distribution for $y_{max}$ and $K$, given these observations. In other words, we want $P(model|data)$, the probability of our model parameters given the data. Bayes' theorem gives an expression for this quantity:

\begin{equation}
P(model|data)=\frac{P(data|model)P(model)}{P(data)}
\end{equation}

Let's unpack this equation.

### The prior

$P(model)$ is the prior; it is a description of the uncertainty we palce on the parameters in our model. For instance, let us assume that our parameters are initially normally distributed:

\begin{align}
y_{max}&=\mathcal{N}(1,0.2) \\
K&=\mathcal{N}(2,0.2)
\end{align}

so that our model becomes

\begin{equation}
\hat{y}=\mathcal{N}(1,0.2)\frac{x}{x+\mathcal{N}(2,0.2)}.
\end{equation}

The prior probability of our model given parameters $y_{max}$ and $K$ is

\begin{equation}
P(model)=\mathcal{N}(y_{max}-1,0.2)\mathcal{N}(\mu-2,0.2).
\end{equation}

Typically we express these probablities in terms of log-probabilities so that the terms become additive:

\begin{equation}
\ln P(model)=\ln\mathcal{N}(y_{max}-1,0.2)+\ln\mathcal{N}(\mu-2,0.2).
\end{equation}

```python
def prior(x,mu,sigma):
    return 1/np.sqrt(2*np.pi*sigma**2)*np.exp(-(x-mu)**2/(2*sigma**2))
mu1=1
mu2=2
sigma=0.2

xp=np.linspace(0,3,100)
y1=prior(xp,mu1,sigma)
y2=prior(xp,mu2,sigma)
plot_helper.plot3(xp,y1,xp,y2,title='Prior')
```

 ### The likelihood

$P(data|model)$ is known as the likelihood. It's a measure of how likely it is that our model generates the observed data. In order to calculate this term we need a measure of how far our model predictions are from the actual observed data; typically we assume that deviations are due to normally-distributed noise, in which case our likelihood takes the simple form of squared residuals for each of the data points $y_n$ with error $s_n$:

\begin{equation}
P(data|model)=\prod_n\frac{1}{2\pi s_n^2}\exp\left(-\frac{(y_n-\hat{y}_n)^2}{2s_n^2}\right)
\end{equation}

The negative log-likelihood is therefore

\begin{equation}
\ln P(data|model)=-\frac{1}{2}\sum_n \left(\frac{(y_n-\hat{y}_n)^2}{s_n^2}+\ln (2 \pi s_n^2) \right)
\end{equation}

### MCMC

What we want to do is determine the posterior probability distribution $\Pi(model|data)$. From this distribution we can determine probabilities as well as the expectation values of any quantity of interest by integrating. In other words, we would like to generate the probability landscape of likely model parameters, given our observations. In order to do this we must sample the landscape by varying the parameters. MCMC allows us to do this, without having to calculate the third term $P(data)$ in the Bayes formula which is nontrivial. The simplest MCMC algorithm is that of Metropolis:

#### The Metropolis algorithm

1) First we start at an initial point for the parameters $y_{max,0}$, $K_0$. We compute the probabilities

\begin{equation}
P(data|y_{max,0},K_0)P(y_{max,0},K_0).
\end{equation}

2) Then we move to a new location $y_{max,1}$, $K_1$. This new location is called the proposal, and it's generated by randomly moving to a new point with probability given by a normal distribution centered around the current location, and a fixed variance (the proposal width).

3) We calculate the new probabilities

\begin{equation}
P(data|y_{max,1},K_1)P(y_{max,1},K_1).
\end{equation}

4) We then calculate the acceptance ratio:

\begin{equation}
\alpha=\frac{P(data|y_{max,1},K_1)P(y_{max,1},K_1)}{P(data|y_{max,0},K_0)P(y_{max,0},K_0)}.
\end{equation}

If $\alpha$ is greater than 1, i.e. the probability at the new point is higher, we accept the new point and move there. If $\alpha$ is smaller than 1, then we accept the move with a probability equal to $\alpha$.

```python
def normalprior(param,mu,sigma):
    return np.log( 1.0 / (np.sqrt(2*np.pi)*sigma) ) - 0.5*(param - mu)**2/sigma**2

def like(pos,x,y,yerr):
    ymax=pos[0]
    K=pos[1]
    model=ymax*x/(x+K)
    inv_sigma2=1.0/(yerr**2)
    return -0.5*(np.sum((y-model)**2*inv_sigma2-np.log(inv_sigma2)))

def prior(pos):
    ymax=pos[0]
    K=pos[1]

    mu1=1
    sigma1=0.5
    log_Prymax=normalprior(ymax,mu1,sigma1)

    mu2=2
    sigma2=0.5
    log_PrK=normalprior(K,mu2,sigma2)

    return log_Prymax+log_PrK

def norm(pos,width):
    return pos+width*np.random.randn(2)

def metropolis(pos,MC,steps,width):
    for i in range(steps):
        proposal=norm(pos,width)
        newloglike=like(proposal,xobs,yobs,yerr)+prior(proposal)
        oldloglike=like(pos,xobs,yobs,yerr)+prior(pos)
        if newloglike>=oldloglike: # If new probability is higher then accept
            pos=proposal
        else:
            a=np.exp(newloglike-oldloglike)
            if np.random.rand()<a: # If old probability is higher than only accept with probability a.
                pos=proposal
            else:
                pos=pos

        MC[i]=pos
    return MC

steps=5000
width=0.1
MC=np.zeros(steps*2).reshape(steps,2)
pos=np.array([1,2])
MC=metropolis(pos,MC,steps,width)

plt.plot(MC[:,0],MC[:,1],'-')
plt.show()
```

Our Markov chain samples positions in parameter space, spending proportionately more time in regions of high probability mass. While the Metropolis algorithm is intuitive and instructive it is not the most efficient MCMC algorithm, so for the next part we will apply a more efficient ensemble sampler.

#### A more efficient algorithm: Goodman and Weare affine-invariant ensemble samplers

```python
def lnlike(theta,x,y,yerr):
    ymax,K=theta
    model=ymax*x/(x+K)
    inv_sigma2=1.0/(yerr**2)
    return -0.5*(np.sum((y-model)**2*inv_sigma2-np.log(inv_sigma2)))

def lnprior(theta):
    ymax,K=theta

    if not (0<ymax and 0<K) :
        return -np.inf # Hard-cutoff for positive value constraint

    mu1=1
    sigma1=0.5
    log_Prymax=np.log( 1.0 / (np.sqrt(2*np.pi)*sigma1) ) - 0.5*(ymax - mu1)**2/sigma1**2

    mu2=2
    sigma2=0.5
    log_PrK=np.log( 1.0 / (np.sqrt(2*np.pi)*sigma2) ) - 0.5*(K - mu2)**2/sigma2**2

    return log_Prymax+log_PrK

def lnprob(theta, x, y, yerr):
    lp = lnprior(theta)
    if not np.isfinite(lp):
        return -np.inf
    return lp + lnlike(theta, x, y, yerr)
```

```python
ndim,nwalkers,threads,iterations,tburn=2,20,8,1000,200
labels=["$y_{max}$","$K$"]
parametertruths=[1,2]
pos=[np.array([
    1*(1+0.05*np.random.randn()),
    1*(1+0.05*np.random.randn())]) for i in range(nwalkers)]
sampler=emcee.EnsembleSampler(nwalkers,ndim,lnprob,a=2,args=(xobs,yobs,yerr),threads=threads)
```

```python
### Start MCMC
iterations=iterations
bar=progressbar.ProgressBar(max_value=iterations)
for i, result in enumerate(sampler.sample(pos, iterations=iterations)):
    bar.update(i)
### Finish MCMC

samples=sampler.chain[:,:,:].reshape((-1,ndim)) # shape = (nsteps, ndim)

df=pd.DataFrame(samples)
df.to_csv(path_or_buf='samplesout_.csv',sep=',')
df1=pd.read_csv('samplesout_.csv',delimiter=',')
data=np.zeros(df1.shape[0]*(df1.shape[1]-1)).reshape(df1.shape[0],(df1.shape[1]-1))
for i in range(0,int(df1.shape[1]-1)):
    data[:,i]=np.array(df1.iloc[:,i+1]) # Put dataframe into array. Dataframe has no. columns = no. parameters.
data2=np.zeros((df1.shape[0]-tburn*nwalkers)*(df1.shape[1]-1)).reshape((df1.shape[0]-(tburn*nwalkers)),(df1.shape[1]-1))
for i in range(0,int(df1.shape[1]-1)):
    for j in range(1,nwalkers+1):
        data2[(iterations-tburn)*(j-1):(iterations-tburn)*(j),i]=np.array(df1.iloc[iterations*j-iterations+tburn:iterations*j,i+1])
samplesnoburn=data2

#plot_helper.plottraces(samples,labels,parametertruths,nwalkers,iterations,1)
fig=corner.corner(samplesnoburn, labels=labels,truths=parametertruths,quantiles=[0.16, 0.5, 0.84],show_titles=True, title_fmt='.2e', title_kwargs={"fontsize": 10},verbose=False)
fig.savefig("triangle.pdf")
```

```python
plot_helper.plot4(xp,y1,xp,y2,samplesnoburn,title='Posterior')
```

```python
plot_helper.plot5(x,y,xobs,yobs,yerr,samplesnoburn,xlabel='x',ylabel='y',legend=False,title=False)
```

**References**

* MacKay 2003 http://www.inference.org.uk/itprnn/book.html - the bible for MCMC and inferential methods in general

* Goodman and Weare 2010 https://projecteuclid.org/euclid.camcos/1513731992 - original paper describing affine-invariant ensemble sampling

* emcee http://dfm.io/emcee/current/user/line/ - Python implementation of the Goodman and Weare algorithm

* Fitting a model to data https://arxiv.org/abs/1008.4686 - excellent tutorial on how to 'properly' fit your data

* Hamiltonian Monte Carlo https://arxiv.org/abs/1701.02434 - a more efficient MCMC algorithm, as implemented in Stan (http://mc-stan.org)

* Another nice online tutorial http://twiecki.github.io/blog/2015/11/10/mcmc-sampling/

# Fit to tellurium ODE model

```python
import tellurium as te
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import plot_helper as plot_helper
import pandas as pd
import emcee
import corner
import progressbar
```

Here is a more sophisticated example similar to what you might encounter in the lab. Suppose we have a dynamical system

```python
def MM_genmodel():
    ''' Michaelis-Menten enzyme model '''

    rr = te.loada('''
        J1: E+S->ES ; k1*E*S-k2*ES ;
        J2: ES->E+S+P ; k3*ES ;

        k1=0;
        k2=0;
        k3=0;

        ''')

    return(rr)

def simulatemodel(rr,tmax,nsteps,paramdict):

    for j in rr.model.getGlobalParameterIds():
        rr[j]=paramdict[j] # set parameters

    for j in rr.model.getFloatingSpeciesIds():
        rr[j]=paramdict[j] # set concentrations

    out=rr.simulate(0,tmax,points=nsteps)

    return(out,rr)

tmax=20
nsteps=51

keys=['k1','k2','k3','E','S','ES','P']
params=[1,1,1,1,10,0,0]
paramdict=dict(zip(keys,params))

# Generate model
rr=MM_genmodel()

# Simulate model
out,_=simulatemodel(rr,tmax,nsteps,paramdict)
rr.plot()
```

Let's do a titration experiment and MCMC to extract kinetic parameters for this enzyme.

```python
np.random.seed(42)

def titration_expt(titration,k1,k2,k3,tmax,nsteps,rr):

    Parr=np.zeros((nsteps,len(titration)))
    for j in range(len(titration)):
        keys=['k1','k2','k3','E','S','ES','P']
        params=[k1,k2,k3,1,titration[j],0,0]
        paramdict=dict(zip(keys,params))
        out,_=simulatemodel(rr,tmax,nsteps,paramdict)
        Parr[:,j]=out[:,4]
    return Parr

rr=MM_genmodel()
tmax=20
nsteps=51
Parr=titration_expt([0,5,10,15,20],1,10,1,tmax,nsteps,rr)
Parr+=0.2*np.random.randn(Parr.shape[0],Parr.shape[1])*Parr+0.0001*np.random.randn(Parr.shape[0],Parr.shape[1]) # Add noise
plt.plot(Parr,'o') ; plt.show()
```

```python
# Define MCMC functions

def normalprior(param,mu,sigma):
    return np.log( 1.0 / (np.sqrt(2*np.pi)*sigma) ) - 0.5*(param - mu)**2/sigma**2

def lnlike(theta,inputs):
    k1,k2,k3=theta

    # DATA
    y=inputs['y']
    yerr=inputs['yerr']

    # MODEL INPUTS
    tmax=inputs['tmax']
    nsteps=inputs['nsteps']
    titration=inputs['titration']
    rr=inputs['model']

    ymodel=titration_expt(titration,k1,k2,k3,tmax,nsteps,rr)

    inv_sigma2=1.0/(yerr**2)
    return -0.5*(np.sum((y-ymodel)**2*inv_sigma2-np.log(inv_sigma2)))

def lnprior(theta):
    k1,k2,k3=theta

    if not (0<k1 and 0<k2 and 0<k3) :
        return -np.inf # Hard-cutoff for positive value constraint

    log_PRs=[normalprior(k1,5,10),
            normalprior(k2,10,10),
            normalprior(k3,1,0.01)]

    return np.sum(log_PRs)

def lnprob(theta,inputs):
    lp = lnprior(theta)
    if not np.isfinite(lp):
        return -np.inf
    return lp + lnlike(theta,inputs)

def gelman_rubin(chain):
    ''' Gelman-Rubin diagnostic for one walker across all parameters. This value should tend to 1. '''
    ssq=np.var(chain,axis=1,ddof=1)
    W=np.mean(ssq,axis=0)
    Tb=np.mean(chain,axis=1)
    Tbb=np.mean(Tb,axis=0)
    m=chain.shape[0]*1.0
    n=chain.shape[1]*1.0
    B=n/(m-1)*np.sum((Tbb-Tb)**2,axis=0)
    varT=(n-1)/n*W+1/n*B
    Rhat=np.sqrt(varT/W)
    return Rhat
```

```python
# Load data
yobs=Parr
yerr=Parr*0.2

# Generate model
rr=MM_genmodel()

inputkeys=['tmax','nsteps','titration','model','y','yerr']
inputvalues=[20,51,[0,5,10,15,20],rr,yobs,yerr]
inputs=dict(zip(inputkeys,inputvalues))
np.random.seed(42)
```

```python
# MLE

pos=[
5,  # k1
10, # k2
1 # k3
]

nll= lambda *args: -lnlike(*args)
result=op.minimize(nll,pos,method='BFGS', args=(inputs))
paramstrue = result["x"]

k1_MLE=paramstrue[0]
k2_MLE=paramstrue[1]
k3_MLE=paramstrue[2]

print(k1_MLE,k2_MLE,k3_MLE)

tmax=20
nsteps=51
titration=[0,5,10,15,20]
ymodel=titration_expt(titration,k1_MLE,k2_MLE,k3_MLE,tmax,nsteps,rr)
plt.plot(yobs,'o')
plt.plot(ymodel,'k-',alpha=1) ; plt.show()

```

```python
# Run MCMC

ndim,nwalkers,threads,iterations,tburn=3,50,1,3000,1000
labels=["$k_1$","$k_2$","$k_3$"]
parametertruths=[1,10,1]
pos=[np.array([
    k1_MLE*(1+0.05*np.random.randn()),
    k2_MLE*(1+0.05*np.random.randn()),
    k3_MLE*(1+0.05*np.random.randn())]) for i in range(nwalkers)]

sampler=emcee.EnsembleSampler(nwalkers,ndim,lnprob,a=2,args=([inputs]),threads=threads)

### Start MCMC
iterations=iterations
bar=progressbar.ProgressBar(max_value=iterations)
for i, result in enumerate(sampler.sample(pos, iterations=iterations)):
    bar.update(i)
### Finish MCMC

samples=sampler.chain[:,:,:].reshape((-1,ndim)) # shape = (nsteps, ndim)
samplesnoburn=sampler.chain[:,tburn:,:].reshape((-1,ndim)) # shape = (nsteps, ndim)

df=pd.DataFrame(samples)
df.to_csv(path_or_buf='samplesout_MM.csv',sep=',')

plot_helper.plottraces(samples,labels,parametertruths,nwalkers,iterations,1)
fig=corner.corner(samplesnoburn, labels=labels,truths=parametertruths,quantiles=[0.16, 0.5, 0.84],show_titles=True, title_fmt='.2e', title_kwargs={"fontsize": 10},verbose=False)
fig.savefig("triangle_MM.pdf")

### Gelman-Rubin diagnostic
# NOT RELIABLE ESTIMATE FOR EMCEE AS WALKERS NOT INDEPENDENT!
plt.close("all")
figure_options={'figsize':(8.27,5.83)} #figure size in inches. A4=11.7x8.3.
font_options={'size':'12','family':'sans-serif','sans-serif':'Arial'}
plt.rc('figure', **figure_options)
plt.rc('font', **font_options)

chain=sampler.chain[:,tburn:,:] # shape = nwalkers, iterations-tburn, ndim
print('Mean acceptance fraction', np.mean(sampler.acceptance_fraction))
print('GR diagnostic for one walker', gelman_rubin(chain)[0]) # Change index to get a different walker

chain_length=chain.shape[1]
step_sampling=np.arange(int(0.2*chain_length),chain_length,50)
rhat=np.array([gelman_rubin(chain[:,:steps,:])[0] for steps in step_sampling])
plt.plot(step_sampling,rhat); ax=plt.gca(); ax.axhline(y=1.1,color='k'); ax.set_title('GR diagnostic');
plt.show()

```

```python
# Autocorrelation time analysis. 'c' should be as large as possible (default is 5)
tau = np.mean([emcee.autocorr.integrated_time(walker,c=1) for walker in sampler.chain[:,:,:]], axis=0)
print('Tau', tau)
```

    Tau [ 41.18420271  40.2389607   29.97377401]

```python
for k1,k2,k3 in samplesnoburn[np.random.randint(len(samplesnoburn), size=10)]:
    tmax=20
    nsteps=51
    titration=[0,5,10,15,20]
    ymodel=titration_expt(titration,k1,k2,k3,tmax,nsteps,rr)
    plt.plot(ymodel,'k-',alpha=0.1)
plt.plot(yobs,'o'); plt.show()
```

```python

```

.ipynb_checkpoints/Fitting a model to data with MCMC-checkpoint.ipynb


#Imports

```python
import numpy as np
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from IPython import display
plt.style.use('seaborn-white')
```

# Read and process data.

Download the file from this URL: https://drive.google.com/file/d/1UWWIi-sz9g0x3LFvkIZjvK1r2ZaCqgGS/view?usp=sharing

```python
data = open('text.txt', 'r').read()
```

Process data and calculate indices

```python
chars = list(set(data))
data_size, X_size = len(data), len(chars)
print("Corona Virus article has %d characters, %d unique characters" %(data_size, X_size))
char_to_idx = {ch:i for i,ch in enumerate(chars)}
idx_to_char = {i:ch for i,ch in enumerate(chars)}
```

    Corona Virus article has 10223 characters, 75 unique characters

# Constants and Hyperparameters

```python
Hidden_Layer_size = 100 #size of the hidden layer
Time_steps = 40 # Number of time steps (length of the sequence) used for training
learning_rate = 1e-1 # Learning Rate
weight_sd = 0.1 #Standard deviation of weights for initialization
z_size = Hidden_Layer_size + X_size #Size of concatenation(H, X) vector
```

# Activation Functions and Derivatives

```python
def sigmoid(x): # sigmoid function
  return 1/(1 + np.exp(-x)) # write your code here

def dsigmoid(y): # derivative of sigmoid function
  return y*(1-y) # write your code here

def tanh(x): # tanh function
  return np.tanh(x) # write your code here

def dtanh(y): # derivative of tanh
  return (1-y**2) # write your code here
```

# Quiz Question 1

What is the value of sigmoid(0) calculated from  your code? (Answer up to 1 decimal point, e.g. 4.2 and NOT 4.29999999, no rounding off).

# Quiz Question 2

What is the value of dsigmoid(sigmoid(0)) calculated from your code?? (Answer up to 2 decimal point, e.g. 4.29 and NOT 4.29999999, no rounding off).

# Quiz Question 3

What is the value of tanh(dsigmoid(sigmoid(0))) calculated from your code?? (Answer up to 5 decimal point, e.g. 4.29999 and NOT 4.29999999, no rounding off).

# Quiz Question 4

What is the value of dtanh(tanh(dsigmoid(sigmoid(0)))) calculated from your code?? (Answer up to 5 decimal point, e.g. 4.29999 and NOT 4.29999999, no rounding off).

```python
print(f'Ans 1: {sigmoid(0)}')
print(f'Ans 2: {dsigmoid(sigmoid(0))}')
print(f'Ans 3: {tanh(dsigmoid(sigmoid(0)))}')
print(f'Ans 4: {dtanh(tanh(dsigmoid(sigmoid(0))))}')
```

    Ans 1: 0.5
    Ans 2: 0.25
    Ans 3: 0.24491866240370913
    Ans 4: 0.940014848806378

# Parameters

```python
class Param:
    def __init__(self, name, value):
      self.name = name
      self.v = value # parameter value
      self.d = np.zeros_like(value) # derivative
      self.m = np.zeros_like(value) # momentum for Adagrad
```

We use random weights with normal distribution (0, weight_sd) for  tanh  activation function and (0.5, weight_sd) for  `sigmoid`  activation function.

Biases are initialized to zeros.

# LSTM
You are making this network, please note f, i, c and o (also "v") in the image below:

Please note that we are concatenating the old_hidden_vector and new_input.

# Quiz Question 4

In the class definition below, what should be size_a, size_b, and size_c? ONLY use the variables defined above.

size_a = Hidden_Layer_size

size_b = z_size

size_c = X_size

```python
size_a = Hidden_Layer_size # write your code here
size_b = z_size # write your code here
size_c = X_size # write your code here

class Parameters:
    def __init__(self):
        self.W_f = Param('W_f', np.random.randn(size_a, size_b) * weight_sd + 0.5)
        self.b_f = Param('b_f', np.zeros((size_a, 1)))

        self.W_i = Param('W_i', np.random.randn(size_a, size_b) * weight_sd + 0.5)
        self.b_i = Param('b_i', np.zeros((size_a, 1)))

        self.W_C = Param('W_C', np.random.randn(size_a, size_b) * weight_sd)
        self.b_C = Param('b_C', np.zeros((size_a, 1)))

        self.W_o = Param('W_o', np.random.randn(size_a, size_b) * weight_sd + 0.5)
        self.b_o = Param('b_o', np.zeros((size_a, 1)))

        #For final layer to predict the next character
        self.W_v = Param('W_v', np.random.randn(X_size, size_a) * weight_sd)
        self.b_v = Param('b_v', np.zeros((size_c, 1)))

    def all(self):
        return [self.W_f, self.W_i, self.W_C, self.W_o, self.W_v,
               self.b_f, self.b_i, self.b_C, self.b_o, self.b_v]

parameters = Parameters()
```

Look at these operations which we'll be writing:

**Concatenation of h and x:**

$z\:=\:\left[h_{t-1},\:x\right]$

$f_t=\sigma\left(W_f\cdot z\:+\:b_f\:\right)$

$i_i=\sigma\left(W_i\cdot z\:+\:b_i\right)$

$\overline{C_t}=\tanh\left(W_C\cdot z\:+\:b_C\right)$

$C_t=f_t\ast C_{t-1}+i_t\ast \overline{C}_t$

$o_t=\sigma\left(W_o\cdot z\:+\:b_o\right)$

$h_t=o_t\ast\tanh\left(C_t\right)$

**Logits:**

$v_t=W_v\cdot h_t+b_v$

**Softmax:**

$\hat{y}=softmax\left(v_t\right)$

```python
def forward(x, h_prev, C_prev, p = parameters):
    assert x.shape == (X_size, 1)
    assert h_prev.shape == (Hidden_Layer_size, 1)
    assert C_prev.shape == (Hidden_Layer_size, 1)

    z = np.row_stack((h_prev, x))
    f = sigmoid(np.dot(p.W_f.v, z) + p.b_f.v) # write your code here
    i = sigmoid(np.dot(p.W_i.v, z) + p.b_i.v) # write your code here
    C_bar = tanh(np.dot(p.W_C.v, z) + p.b_C.v) # write your code here

    C = f * C_prev + i * C_bar # write your code here
    o = sigmoid(np.dot(p.W_o.v, z) + p.b_o.v) # write your code here
    h = o * tanh(C) # write your code here

    v = np.dot(p.W_v.v, h) + p.b_v.v # write your code here
    y = np.exp(v) / np.sum(np.exp(v)) #softmax

    return z, f, i, C_bar, C, o, h, v, y
```

You must finish the function above before you can attempt the questions below.

# Quiz Question 5

What is the output of 'print(len(forward(np.zeros((X_size, 1)), np.zeros((Hidden_Layer_size, 1)), np.zeros((Hidden_Layer_size, 1)), parameters)))'?

```python
print(len(forward(np.zeros((X_size, 1)), np.zeros((Hidden_Layer_size, 1)), np.zeros((Hidden_Layer_size, 1)), parameters)))
```

    9

# Quiz Question 6.

Assuming you have fixed the forward function, run this command:
z, f, i, C_bar, C, o, h, v, y = forward(np.zeros((X_size, 1)), np.zeros((Hidden_Layer_size, 1)), np.zeros((Hidden_Layer_size, 1)))

Now, find these values:

1.   print(z.shape)
2.   print(np.sum(z))
3.   print(np.sum(f))

Copy and paste exact values you get in the logs into the quiz.

```python
z, f, i, C_bar, C, o, h, v, y = forward(np.zeros((X_size, 1)), np.zeros((Hidden_Layer_size, 1)), np.zeros((Hidden_Layer_size, 1)))
```

```python
print(z.shape)
print(np.sum(z))
print(np.sum(f))
```

    (85, 1)
    0.0
    5.0

# Backpropagation

Here we are defining the backpropagation. It's too complicated, here is the whole code. (Please note that this would work only if your earlier code is perfect).

```python
def backward(target, dh_next, dC_next, C_prev,
             z, f, i, C_bar, C, o, h, v, y,
             p = parameters):

    assert z.shape == (X_size + Hidden_Layer_size, 1)
    assert v.shape == (X_size, 1)
    assert y.shape == (X_size, 1)

    for param in [dh_next, dC_next, C_prev, f, i, C_bar, C, o, h]:
        assert param.shape == (Hidden_Layer_size, 1)

    dv = np.copy(y)
    dv[target] -= 1

    p.W_v.d += np.dot(dv, h.T)
    p.b_v.d += dv

    dh = np.dot(p.W_v.v.T, dv)
    dh += dh_next
    do = dh * tanh(C)
    do = dsigmoid(o) * do
    p.W_o.d += np.dot(do, z.T)
    p.b_o.d += do

    dC = np.copy(dC_next)
    dC += dh * o * dtanh(tanh(C))
    dC_bar = dC * i
    dC_bar = dtanh(C_bar) * dC_bar
    p.W_C.d += np.dot(dC_bar, z.T)
    p.b_C.d += dC_bar

    di = dC * C_bar
    di = dsigmoid(i) * di
    p.W_i.d += np.dot(di, z.T)
    p.b_i.d += di

    df = dC * C_prev
    df = dsigmoid(f) * df
    p.W_f.d += np.dot(df, z.T)
    p.b_f.d += df

    dz = (np.dot(p.W_f.v.T, df)
         + np.dot(p.W_i.v.T, di)
         + np.dot(p.W_C.v.T, dC_bar)
         + np.dot(p.W_o.v.T, do))
    dh_prev = dz[:Hidden_Layer_size, :]
    dC_prev = f * dC

    return dh_prev, dC_prev
```

# Forward and Backward Combined Pass

Let's first clear the gradients before each backward pass

```python
def clear_gradients(params = parameters):
    for p in params.all():
        p.d.fill(0)
```

Clip gradients to mitigate exploding gradients

```python
def clip_gradients(params = parameters):
    for p in params.all():
        np.clip(p.d, -1, 1, out=p.d)
```

Calculate and store the values in forward pass. Accumulate gradients in backward pass and clip gradients to avoid exploding gradients.

input, target are list of integers, with character indexes.
h_prev is the array of initial h at  h−1  (size H x 1)
C_prev is the array of initial C at  C−1  (size H x 1)
Returns loss, final  hT  and  CT

```python
def forward_backward(inputs, targets, h_prev, C_prev):
    global paramters

    # To store the values for each time step
    x_s, z_s, f_s, i_s,  = {}, {}, {}, {}
    C_bar_s, C_s, o_s, h_s = {}, {}, {}, {}
    v_s, y_s =  {}, {}

    # Values at t - 1
    h_s[-1] = np.copy(h_prev)
    C_s[-1] = np.copy(C_prev)

    loss = 0
    # Loop through time steps
    assert len(inputs) == Time_steps
    for t in range(len(inputs)):
        x_s[t] = np.zeros((X_size, 1))
        x_s[t][inputs[t]] = 1 # Input character

        (z_s[t], f_s[t], i_s[t],
        C_bar_s[t], C_s[t], o_s[t], h_s[t],
        v_s[t], y_s[t]) = \
            forward(x_s[t], h_s[t - 1], C_s[t - 1]) # Forward pass

        loss += -np.log(y_s[t][targets[t], 0]) # Loss for at t

    clear_gradients()

    dh_next = np.zeros_like(h_s[0]) #dh from the next character
    dC_next = np.zeros_like(C_s[0]) #dh from the next character

    for t in reversed(range(len(inputs))):
        # Backward pass
        dh_next, dC_next = \
            backward(target = targets[t], dh_next = dh_next,
                     dC_next = dC_next, C_prev = C_s[t-1],
                     z = z_s[t], f = f_s[t], i = i_s[t], C_bar = C_bar_s[t],
                     C = C_s[t], o = o_s[t], h = h_s[t], v = v_s[t],
                     y = y_s[t])

    clip_gradients()

    return loss, h_s[len(inputs) - 1], C_s[len(inputs) - 1]
```

# Sample the next character

```python
def sample(h_prev, C_prev, first_char_idx, sentence_length):
    x = np.zeros((X_size, 1))
    x[first_char_idx] = 1

    h = h_prev
    C = C_prev

    indexes = []

    for t in range(sentence_length):
        _, _, _, _, C, _, h, _, p = forward(x, h, C)
        idx = np.random.choice(range(X_size), p=p.ravel())
        x = np.zeros((X_size, 1))
        x[idx] = 1
        indexes.append(idx)

    return indexes
```

# Training (Adagrad)

Update the graph and display a sample output

```python
def update_status(inputs, h_prev, C_prev):
    #initialized later
    global plot_iter, plot_loss
    global smooth_loss

    # Get predictions for 200 letters with current model

    sample_idx = sample(h_prev, C_prev, inputs[0], 200)
    txt = ''.join(idx_to_char[idx] for idx in sample_idx)

    # Clear and plot
    plt.plot(plot_iter, plot_loss)
    display.clear_output(wait=True)
    plt.show()

    #Print prediction and loss
    print("----\n %s \n----" % (txt, ))
    print("iter %d, loss %f" % (iteration, smooth_loss))
```

# Update Parameters

\begin{align}
\theta_i &= \theta_i - \eta\frac{d\theta_i}{\sum dw_{\tau}^2} \\
d\theta_i &= \frac{\partial L}{\partial \theta_i}
\end{align}

```python
def update_paramters(params = parameters):
    for p in params.all():
        p.m += p.d * p.d # Calculate sum of gradients
        #print(learning_rate * dparam)
        p.v += -(learning_rate * p.d / np.sqrt(p.m + 1e-8))
```

To delay the keyboard interrupt to prevent the training from stopping in the middle of an iteration

```python
# Exponential average of loss
# Initialize to a error of a random model
smooth_loss = -np.log(1.0 / X_size) * Time_steps

iteration, pointer = 0, 0

# For the graph
plot_iter = np.zeros((0))
plot_loss = np.zeros((0))
```

# Training Loop

```python
iter = 1000
while iter > 0:
  # Reset
  if pointer + Time_steps >= len(data) or iteration == 0:
      g_h_prev = np.zeros((Hidden_Layer_size, 1))
      g_C_prev = np.zeros((Hidden_Layer_size, 1))
      pointer = 0

  inputs = ([char_to_idx[ch]
              for ch in data[pointer: pointer + Time_steps]])
  targets = ([char_to_idx[ch]
              for ch in data[pointer + 1: pointer + Time_steps + 1]])

  loss, g_h_prev, g_C_prev = \
      forward_backward(inputs, targets, g_h_prev, g_C_prev)
  smooth_loss = smooth_loss * 0.999 + loss * 0.001

  # Print every hundred steps
  if iteration % 100 == 0:
      update_status(inputs, g_h_prev, g_C_prev)

  update_paramters()

  plot_iter = np.append(plot_iter, [iteration])
  plot_loss = np.append(plot_loss, [loss])

  pointer += Time_steps
  iteration += 1
  iter = iter -1
```

# Quiz Question 7.

Run the above code for 50000 iterations making sure that you have 100 hidden layers and time_steps is 40. What is the loss value you're seeing?

```python
iter = 50000
while iter > 0:
  # Reset
  if pointer + Time_steps >= len(data) or iteration == 0:
      g_h_prev = np.zeros((Hidden_Layer_size, 1))
      g_C_prev = np.zeros((Hidden_Layer_size, 1))
      pointer = 0

  inputs = ([char_to_idx[ch]
              for ch in data[pointer: pointer + Time_steps]])
  targets = ([char_to_idx[ch]
              for ch in data[pointer + 1: pointer + Time_steps + 1]])

  loss, g_h_prev, g_C_prev = \
      forward_backward(inputs, targets, g_h_prev, g_C_prev)
  smooth_loss = smooth_loss * 0.999 + loss * 0.001

  # Print every hundred steps
  if iteration % 100 == 0:
      update_status(inputs, g_h_prev, g_C_prev)

  update_paramters()

  plot_iter = np.append(plot_iter, [iteration])
  plot_loss = np.append(plot_loss, [loss])

  pointer += Time_steps
  iteration += 1
  iter = iter -1
```

```python
z, f, i, C_bar, C, o, h, v, y = forward(np.zeros((X_size, 1)), np.zeros((Hidden_Layer_size, 1)), np.zeros((Hidden_Layer_size, 1)))
print(z.shape)
print(np.sum(z))
print(np.sum(f))
```

    (175, 1)
    0.0
    45.316084800205815


## Equilibrium of a system of weights connected by strings/springs

In this notebook we show how to solve the following problem: Find the equlibrium of a system of masses connected by a system of strings, with some masses being assigned fixed coordinates (attached to the wall, say). See the next picture.

Suppose we have $n$ masses with weights $w_1,\ldots,w_n$, and the length of the string between $i$ and $j$ is $\ell_{ij}$ for some set $L$ of pairs of indices $(i,j)$ (we assume $\ell_{ij}$ is not defined if there is no connection). The strings themselves have no mass. We also have a set $F$ of indices such that the $i$-th point is fixed to have coordinates $f_i$ if $i\in F$. The equilibrium of the system is a configuration which minimizes potential energy. With this setup we can write our problem as:

\begin{equation}
\begin{array}{ll}
minimize & g\cdot \sum_i w_ix_i^{(2)} \\
s.t.     & \|x_i-x_j\|\leq \ell_{ij},\ ij\in L \\
         & x_i = f_i,\ i\in F
\end{array}
\end{equation}

where $x\in (\mathbf{R}^n)^2$, $x_i^{(2)}$ denotes the second (vertical) coordinate of $x_i$ and $g$ is the gravitational constant.

Here is a sample problem description.

```python
w = [0.0, 1.1, 2.2, 0.0, 2.1, 2.2, 0.2]
l = {(0,1): 1.0, (1,2): 1.0, (2,3): 1.0, (1,4): 1.0, (4,5): 0.3, (5,2): 1.0, (5,6): 0.5, (1,3): 8.0}
f = {0: (0.0,1.0), 3: (2.0,1.0)}
g = 9.81
```

Now we can formulate the problem using Mosek Fusion:

```python
from mosek.fusion import *

# w - masses of points
# l - lengths of strings
# f - coordinates of fixed points
# g - gravitational constant
def stringModel(w, l, f, g):
    n, m = len(w), len(l)
    starts = [ lKey[0] for lKey in l.keys() ]
    ends = [ lKey[1] for lKey in l.keys() ]

    M = Model("strings")

    # Coordinates of points
    x = M.variable("x", [n, 2])

    # A is the signed incidence matrix of points and strings
    A = Matrix.sparse(m, n, list(range(m))+list(range(m)), starts+ends, [1.0]*m+[-1.0]*m)

    # ||x_i-x_j|| <= l_{i,j}
    c = M.constraint("c", Expr.hstack(Expr.constTerm(list(l.values())), Expr.mul(A, x)),
        Domain.inQCone() )

    # x_i = f_i for fixed points
    for i in f:
        M.constraint(x.slice([i,0], [i+1,2]), Domain.equalsTo(list(f[i])).withShape([1,2]))

    # sum (g w_i x_i_2)
    M.objective(ObjectiveSense.Minimize,
        Expr.mul(g, Expr.dot(w, x.slice([0,1], [n,2]))))

    # Solve
    M.solve()
    if M.getProblemStatus(SolutionType.Interior) == ProblemStatus.PrimalAndDualFeasible:
        return x.level().reshape([n,2]), c.dual().reshape([m,3])
    else:
        return None, None

```

Here is a quick description of how we use vectorization to deal with all the conic constraints in one go. The matrix $A$ is the incidence matrix between the masses and the strings, with coefficients $+1, -1$ for the two endpoints of each string. It is chosen so that the product $Ax$ has rows of the form

$$
(x_i^{(1)} - x_j^{(1)}, x_i^{(2)} - x_j^{(2)})
$$

for all pairs $i,j$ for which $\ell_{ij}$ is bounded. Stacking the values of $\ell$ in the left column produces a matrix with each row of the form

$$
(\ell_{ij}, x_i^{(1)} - x_j^{(1)}, x_i^{(2)} - x_j^{(2)})
$$

and a conic constraint is imposed on all the rows, as required.

The objective and linear constraints show examples of slicing the variable $x$.

The function returns the coordinates of the masses and the values of the dual conic variables. A zero dual value indicates that a particular string is hanging loose, and a nonzero value means it is fully stretched.

All we need now is to define a display function and we can look at some plots.

```python
%matplotlib inline
# x - coordinates of the points
# c - dual values of string length constraints
# d - pairs of points to connect
def display(x, c, d):
    import matplotlib.pyplot as plt
    fig, ax = plt.subplots()
    # Plot points
    ax.scatter(x[:,0], x[:,1], color="r")
    # Plot fully stretched strings (nonzero dual value) as solid lines, else dotted lines
    for i in range(len(c)):
        col = "b" if c[i][0] > 1e-4 else "b--"
        ax.plot([x[d[i][0]][0], x[d[i][1]][0]], [x[d[i][0]][1], x[d[i][1]][1]], col)
    ax.axis("equal")
    plt.show()
```

```python
x,c = stringModel(w, l, f, g)

if x is not None:
    display(x, c, list(l.keys()))
```

How about we find a discrete approximation to the [catenary](#https://en.wikipedia.org/wiki/Catenary):

```python
n = 1000
w = [1.0]*n
l = {(i,i+1): 1.0/n for i in range(n-1)}
f = {0: (0.0,1.0), n-1: (0.7,1.0)}
g = 9.81

x,c = stringModel(w, l, f, g)
if x is not None:
    display(x, c, list(l.keys()))
```

We can also have more suspension points and more complicated shapes:

```python
n = 20
w = [1.0]*n
l = {(i,i+1): 0.09 for i in range(n-1)}
l.update({(5,14): 0.3})
f = {0: (0.0,1.0), 13: (0.5,0.9), 17: (0.7,1.1)}
g = 9.81

x,c = stringModel(w, l, f, g)
if x is not None:
    display(x, c, list(l.keys()))
```

## Duality and feasibility

The dual problem is as follows:

\begin{equation}
\begin{array}{ll}
maximize & -\sum_{ij\in L}\ell_{ij}y_{ij} - \sum_{i\in F}f_i\circ z_i\\
s.t.     & y_{ij}\geq \|v_{ij}\|,\ ij\in L \\
         & \sum_{j~:~ij\in L} v_{ij}\mathrm{sgn}_{ij} + \left(\begin{array}{c}0\\ gw_i\end{array}\right) +z_i = 0, \ i=1,\ldots,n
\end{array}
\end{equation}

where $\mathrm{sgn}_{ij}=+1$ if $i>j$ and $-1$ otherwise and $\circ$ is the dot product. The variables are $(y_{ij},v_{ij})\in \mathbf{R}\times\mathbf{R}^2$ for $ij\in L$ and $z_i\in\mathbf{R}^2$ for $i\in F$ (we assume $z_i=0$ for $i\not\in F$).

Obviously (!) the linear constraints describe the equilibrium of forces at every mass. The ingredients are: the vectors of forces applied through adjacent strings ($v_{ij}$), gravity, and the attaching force holding a fixed point in its position. By proper use of vectorization this is much easier to express in Fusion than it looks:

```python
def dualStringModel(w, l, f, g):
    n, m = len(w), len(l)
    starts = [ lKey[0] for lKey in l.keys() ]
    ends = [ lKey[1] for lKey in l.keys() ]

    M = Model("dual strings")

    x = M.variable(Domain.inQCone(m,3))       #(y,v)
    y = x.slice([0,0],[m,1])
    v = x.slice([0,1],[m,3])
    z = M.variable([n,2])

    # z_i = 0 if i is not fixed
    for i in range(n):
        if i not in f:
            M.constraint(z.slice([i,0], [i+1,2]), Domain.equalsTo(0.0))

    B = Matrix.sparse(m, n, list(range(m))+list(range(m)), starts+ends, [1.0]*m+[-1.0]*m).transpose()
    w2 = Matrix.sparse(n, 2, range(n), [1]*n, [-wT*g for wT in w])

    # sum(v_ij *sgn(ij)) + z_i = -(0, gw_i) for all vertices i
    M.constraint(Expr.add( Expr.mul(B, v), z ), Domain.equalsTo(w2))

    # Objective -l*y -fM*z
    fM = Matrix.sparse(n, 2, list(f.keys())+list(f.keys()), [0]*len(f)+[1]*len(f),
                       [pt[0] for pt in f.values()] + [pt[1] for pt in f.values()])

    M.objective(ObjectiveSense.Maximize, Expr.neg(Expr.add(Expr.dot(list(l.values()), y),Expr.dot(fM, z))))
    M.solve()
```

Let us quickly discuss the possible situations regarding feasibility:

* The system has an equilibrium --- the problem is **primal feasible** and **dual feasible**.
* The strings are too short and it is impossible to stretch the required distance between fixed points --- the problem is **primal infeasible**.
* The system has a component that is not connected to any fixed point, hence some masses can keep falling down indefinitely, causing the problem **primal unbounded**. Clearly the forces within such component cannot be balanced, so the problem is **dual infeasible**.

## Springs

We can extend this to consider infinitely strechable springs instead of fixed-length strings connecting the masses. The next model appears in [Applications of SOCP](#http://stanford.edu/~boyd/papers/pdf/socp.pdf) by Lobo, Boyd, Vandenberghe, Lebret. We will now interpret $\ell_{ij}$ as the base length of the spring and assume that the elastic potential energy stored in the spring at length $x$ is

$$
E_{ij}=\left\{\begin{array}{ll}0 & x\leq \ell_{ij}\\ \frac{k}{2}(x-\ell_{ij})^2 & x>\ell_{ij}\end{array}\right.
$$

That leads us to consider the following second order cone program minimizing the total potential energy:

\begin{equation}
\begin{array}{ll}
minimize & g\cdot \sum_i w_ix_i^{(2)} + \frac{k}{2}\sum_{ij\in L} t_{ij}^2 \\
s.t.     & \|x_i-x_j\|\leq \ell_{ij}+t_{ij},\ ij\in L \\
         & 0\leq t_{ij},\ ij\in L \\
         & x_i = f_i,\ i\in F
\end{array}
\end{equation}

If $t$ denotes the vector of $t_{ij}$ then using a rotated quadratic cone for $(1,T,t)$:

$$
2\cdot 1\cdot T\geq \|t\|^2
$$

will place a bound on $\frac12\sum t_{ij}^2$. We now have a simple extension of the first model.

```python
# w - masses of points
# l - lengths of strings
# f - coordinates of fixed points
# g - gravitational constant
# k - stiffness coefficient
def elasticModel(w, l, f, g, k):
    n, m = len(w), len(l)
    starts = [ lKey[0] for lKey in l.keys() ]
    ends = [ lKey[1] for lKey in l.keys() ]

    M = Model("strings")
    x = M.variable("x", [n, 2])                 # Coordinates
    t = M.variable(m, Domain.greaterThan(0.0))  # Streching

    T = M.variable(1)                           # Upper bound
    M.constraint(Expr.vstack(T, Expr.constTerm(1.0), t), Domain.inRotatedQCone())

    # A is the signed incidence matrix of points and strings
    A = Matrix.sparse(m, n, list(range(m))+list(range(m)), starts+ends, [1.0]*m+[-1.0]*m)

    # ||x_i-x_j|| <= l_{i,j} + t_{i,j}
    c = M.constraint("c", Expr.hstack(Expr.add(t, Expr.constTerm(list(l.values()))), Expr.mul(A, x)),
        Domain.inQCone() )

    # x_i = f_i for fixed points
    for i in f:
        M.constraint(x.slice([i,0], [i+1,2]), Domain.equalsTo(list(f[i])).withShape([1,2]))

    # sum (g w_i x_i_2) + k*T
    M.objective(ObjectiveSense.Minimize,
        Expr.add(Expr.mul(k,T), Expr.mul(g, Expr.dot(w, x.slice([0,1], [n,2])))))

    # Solve
    M.solve()
    if M.getProblemStatus(SolutionType.Interior) == ProblemStatus.PrimalAndDualFeasible:
        return x.level().reshape([n,2]), c.dual().reshape([m,3])
    else:
        return None, None
```

```python
n = 20
w = [1.0]*n
l = {(i,i+1): 0.09 for i in range(n-1)}
l.update({(5,14): 0.3})
f = {0: (0.0,1.0), 13: (0.5,0.9), 17: (0.7,1.1)}
g = 9.81
k = 800

x, c = elasticModel(w, l, f, g, k)
if x is not None:
    display(x, c, list(l.keys()))
```

This work is licensed under a Creative Commons Attribution 4.0 International License. The **MOSEK** logo and name are trademarks of Mosek ApS. The code is provided as-is. Compatibility with future release of **MOSEK** or the `Fusion API` are not guaranteed. For more information contact our [support](mailto:support@mosek.com).


# 15 PDEs: Solution with Time Stepping (Students)

## Heat Equation
The **heat equation** can be derived from Fourier's law and energy conservation (see the [lecture notes on the heat equation (PDF)](15_PDEs_LectureNotes_HeatEquation.pdf))

$$
\frac{\partial T(\mathbf{x}, t)}{\partial t} = \frac{K}{C\rho} \nabla^2 T(\mathbf{x}, t),
$$

## Problem: insulated metal bar (1D heat equation)
A metal bar of length $L$ is insulated along it lengths and held at 0ºC at its ends. Initially, the whole bar is at 100ºC. Calculate $T(x, t)$ for $t>0$.

### Analytic solution
Solve by separation of variables and power series: The general solution that obeys the boundary conditions $T(0, t) = T(L, t) = 0$ is

$$
T(x, t) = \sum_{n=1}^{+\infty} A_n \sin(k_n x)\, \exp\left(-\frac{k_n^2 K t}{C\rho}\right), \quad k_n = \frac{n\pi}{L}
$$

The specific solution that satisfies $T(x, t=0) = T_0 = 100^\circ\text{C}$ leads to $A_n = 4 T_0/n\pi$ for $n$ odd:

$$
T(x, t) = \sum_{n=1,3,5,\dots}^{+\infty} \frac{4 T_0}{n \pi} \sin(k_n x)\, \exp\left(-\frac{k_n^2 K t}{C\rho}\right)
$$

```python
import numpy as np
import matplotlib.pyplot as plt
plt.style.use('ggplot')
```

```python
%matplotlib inline
```

Analytical solution:

```python
def T_bar(x, t, T0, L, K=237, C=900, rho=2700, nmax=1000):
    T = np.zeros_like(x)
    eta = K / (C*rho)
    for n in range(1, nmax, 2):
        kn = n*np.pi/L
        T += 4*T0/(np.pi * n) * np.sin(kn*x) * np.exp(-kn*kn * eta * t)
    return T
```

```python
T0 = 100.
L = 1.0
X = np.linspace(0, L, 100)
for t in np.linspace(0, 3000, 50):
    plt.plot(X, T_bar(X, t, T0, L))
plt.xlabel(r"$x$ (m)")
plt.ylabel(r"$T$ ($^\circ$C)");
```

### Numerical solution: Leap frog
Discretize (finite difference):

For the time domain we only have the initial values so we use a simple forward difference for the time derivative:

$$
\frac{\partial T(x,t)}{\partial t} \approx \frac{T(x, t+\Delta t) - T(x, t)}{\Delta t}
$$

For the spatial derivative we have initially all values so we can use the more accurate central difference approximation:

$$
\frac{\partial^2 T(x, t)}{\partial x^2} \approx \frac{T(x+\Delta x, t) + T(x-\Delta x, t) - 2 T(x, t)}{\Delta x^2}
$$

Thus, the heat equation can be written as the finite difference equation

$$
\frac{T(x, t+\Delta t) - T(x, t)}{\Delta t} = \frac{K}{C\rho} \frac{T(x+\Delta x, t) + T(x-\Delta x, t) - 2 T(x, t)}{\Delta x^2}
$$

which can be reordered so that the RHS contains only known terms and the LHS future terms. Index $i$ is the spatial index, and $j$ the time index: $x = x_0 + i \Delta x$, $t = t_0 + j \Delta t$.

$$
T_{i, j+1} = (1 - 2\eta) T_{i,j} + \eta(T_{i+1,j} + T_{i-1, j}), \quad \eta := \frac{K \Delta t}{C \rho \Delta x^2}
$$

Thus we can step forward in time ("leap frog"), using only known values.

### Activity: Solve the 1D heat equation numerically for an iron bar
* $K = 237$ W/mK
* $C = 900$ J/K
* $\rho = 2700$ kg/m3
* $L = 1$ m
* $T_0 = 373$ K and $T_b = 273$ K
* $T(x, 0) = T_0$ and $T(0, t) = T(L, t) = T_b$

Implement the Leapfrog time-stepping algorithm and visualize the results.

```python
import numpy as np

import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
```

For 3D rotatable output:

```python
%matplotlib widget
```

For HTML/nbviewer output, use inline:

```python
%matplotlib inline
```

Numerical solution

```python
L_rod = 1.    # m
t_max = 3000. # s

Dx = 0.02   # m
Dt = 2    # s

Nx = int(L_rod // Dx)
Nt = int(t_max // Dt)

Kappa = 237 # W/(m K)
CHeat = 900 # J/K
rho = 2700  # kg/m^3

T0 = 373 # K
Tb = 273 # K

raise NotImplementedError
# eta =

step = 20  # plot solution every n steps

print("Nx = {0}, Nt = {1}".format(Nx, Nt))
print("eta = {0}".format(eta))

T = np.zeros(Nx)
T_new = np.zeros_like(T)
T_plot = np.zeros((Nt//step + 1, Nx))

raise NotImplementedError
# initial conditions
# ...

# boundary conditions
# ...

t_index = 0
T_plot[t_index, :] = T
for jt in range(1, Nt):

    raise NotImplementedError

    if jt % step == 0 or jt == Nt-1:
        t_index += 1
        # save the new solution for later plotting
        # T_plot[t_index, :] =
        print("Iteration {0:5d}".format(jt), end="\r")
else:
    print("Completed {0:5d} iterations: t={1} s".format(jt, jt*Dt))
```

#### Visualization
Visualize (you can use the code as is).

Note how we are making the plot use proper units by mutiplying with `Dt * step` and `Dx`.

```python
X, Y = np.meshgrid(range(T_plot.shape[0]), range(T_plot.shape[1]))
Z = T_plot[X, Y]
fig = plt.figure()
ax = fig.add_subplot(111, projection="3d")
ax.plot_wireframe(X*Dt*step, Y*Dx, Z)
ax.set_xlabel(r"time $t$ (s)")
ax.set_ylabel(r"position $x$ (m)")
ax.set_zlabel(r"temperature $T$ (K)")
fig.tight_layout()
```

## Stability of the solution

### Empirical investigation of the stability
Investigate the solution for different values of `Dt` and `Dx`. Can you discern patters for stable/unstable solutions?

Report `Dt`, `Dx`, and `eta`
* for 3 stable solutions
* for 3 unstable solutions

Wrap your heat diffusion solver in a function so that it becomes easier to run it:

```python
def calculate_T(L_rod=1, t_max=3000, Dx=0.02, Dt=2, T0=373, Tb=273,
               step=20):
    Nx = int(L_rod // Dx)
    Nt = int(t_max // Dt)

    Kappa = 237 # W/(m K)
    CHeat = 900 # J/K
    rho = 2700  # kg/m^3

    raise NotImplementedError

    return T_plot

def plot_T(T_plot, Dx, Dt, step):
    X, Y = np.meshgrid(range(T_plot.shape[0]), range(T_plot.shape[1]))
    Z = T_plot[X, Y]
    fig = plt.figure()
    ax = fig.add_subplot(111, projection="3d")
    ax.plot_wireframe(X*Dt*step, Y*Dx, Z)
    ax.set_xlabel(r"time $t$ (s)")
    ax.set_ylabel(r"position $x$ (m)")
    ax.set_zlabel(r"temperature $T$ (K)")
    fig.tight_layout()
    return ax
```

```python
T_plot = calculate_T(Dx=0.02, Dt=2, step=20)
plot_T(T_plot, 0.02, 2, 20)
```

For which values of $\Delta t$ and $\Delta x$ does the solution become unstable?

### Von Neumann stability analysis

If the difference equation solution diverges then we *know* that we have a bad approximation to the original PDE.

Von Neumann stability analysis starts from the assumption that *eigenmodes* of the difference equation can be written as

$$
T_{m,j} = \xi(k)^j e^{ikm\Delta x}, \quad t=j\Delta t,\ x=m\Delta x
$$

with the unknown wave vectors $k=2\pi/\lambda$ and unknown complex functions $\xi(k)$.

Solutions of the difference equation can be written as linear superpositions of these basis functions. But they are only stable if the eigenmodes are stable, i.e., will not grow in time (with $j$). This is the case when
$$
|\xi(k)| < 1
$$
for all $k$.

Insert the eigenmodes into the finite difference equation

$$
T_{m, j+1} = (1 - 2\eta) T_{m,j} + \eta(T_{m+1,j} + T_{m-1, j})
$$

to obtain

\begin{align}
\xi(k)^{j+1} e^{ikm\Delta x} &= (1 - 2\eta) \xi(k)^{j} e^{ikm\Delta x}
    + \eta(\xi(k)^{j} e^{ik(m+1)\Delta x} + \xi(k)^{j} e^{ik(m-1)\Delta x})\\
\xi(k) &= (1 - 2\eta) + \eta(e^{ik\Delta x} + e^{-ik\Delta x})\\
\xi(k) &= 1 - 2\eta + 2\eta \cos k\Delta x\\
\xi(k) &= 1 + 2\eta\big(\cos k\Delta x - 1\big)
\end{align}

For $|\xi(k)| < 1$ (and all possible $k$):

\begin{align}
|\xi(k)| < 1 \quad &\Leftrightarrow \quad \xi^2(k) < 1\\
(1 + 2y)^2 = 1 + 4y + 4y^2 &< 1 \quad \text{with}\ \  y = \eta(\cos k\Delta x - 1)\\
y(1 + y) &< 0 \quad \Leftrightarrow \quad -1 < y < 0\\
\eta(\cos k\Delta x - 1) &\leq 0 \quad \forall k \quad (\eta > 0, -1 \leq \cos x \leq 1)\\
\eta(\cos k\Delta x - 1) &> -1\\
\eta &< \frac{1}{1 - \cos k\Delta x}\\
\eta &= \frac{K \Delta t}{C \rho \Delta x^2} < \frac{1}{2}
\end{align}

Thus, solutions are only stable for $\eta < 1/2$. In particular, decreasing $\Delta t$ will always improve stability, But decreasing $\Delta x$ requires an quadratic *increase* in $\Delta t$!

Note
* Perform von Neumann stability analysis when possible (depends on PDE and the specific discretization).
* Test different combinations of $\Delta t$ and $\Delta x$.
* Not guarantee that decreasing both will lead to more stable solutions!

```python

```


```python
from sympy import *
init_printing()
#from IPython.display import display
```

```python
th,x,y = symbols('theta,x,y')

display(th)
```

```python
Rth = Matrix([[cos(th),-sin(th)],[sin(th), cos(th)]])

simplify(Rth*Rth.T)
```

```python
X = Matrix([x,y])   # <--- this is a vector

Rth*X
```

```python
simplify((X.T*Rth.T)*(Rth*X))
```

```python
simplify((X.T*Rth.T)*(Rth*X))
```

```python

```


# Analytical problem
Defining a problem with an explicit mathematical representation is straightforward.

As an example, consider the following multiobjective optimization problem:

\begin{equation}
\begin{aligned}
& \underset{\mathbf x}{\text{min}}
& & x_1^2 - x_2; x_2^2 - 3x_1 \\
& \text{s.t.} & &  x_1 + x_2 \leq 10 \\
& & &  \mathbf{x} \; \in S, \\
\end{aligned}
\end{equation}

where the feasible region is

\begin{equation}
x_i \in \left[-5, 5\right] \; \forall i \;\in \left[1,2\right].
\end{equation}

Begin by importing the necessary classes:

```python
from desdeo_problem.Problem import ScalarMOProblem
from desdeo_problem.Objective import _ScalarObjective
from desdeo_problem.Variable import Variable
from desdeo_problem.Constraint import ScalarConstraint
```

Define the variables:

```python
# Args: name, starting value, lower bound, upper bound
x1 = Variable("x_1", 0, -0.5, 0.5)
x2 = Variable("x_2", 0, -0.5, 0.5)
```

Define the objectives, notice the argument of the callable objective function, it is assumed to be array-like.

```python
# Args: name, callable
obj1 = _ScalarObjective("f_1", lambda x: x[:,0]**2 - x[:,1])
obj2 = _ScalarObjective("f_2", lambda x: x[:,1]**2 - 3*x[:,0])
```

Define the constraints. Constraint may depend on objective function as well (second argument to the lambda, notice the underscore). In that case, the objectives should not be defined inline, like above, but as their own function definitions. The constraint should be defined so, that when evaluated, it should return a positive value, if the constraint is adhered to, and a negative, if the constraint is breached.

```python
# Args: name, n of variables, n of objectives, callable
cons1 = ScalarConstraint("c_1", 2, 2, lambda x, _: 10 - (x[:,0] + x[:,1]))
```

Finally, put it all together and create the problem.

```python
# Args: list of objevtives, variables and constraints
problem = ScalarMOProblem([obj1, obj2]
                         ,[x1, x2]
                         ,[cons1])
```

Now, the problem is fully specified and can be evaluated and played around with.

```python
import numpy as np

print("N of objectives:", problem.n_of_objectives)
print("N of variables:", problem.n_of_variables)
print("N of constraints:", problem.n_of_constraints)

res1 = problem.evaluate(np.array([2, 4]))
res2 = problem.evaluate(np.array([6, 6]))
res3 = problem.evaluate(np.array([[6, 3], [4,3], [7,4]]))

print("Single feasible decision variables:", res1.objectives, "with constraint values", res1.constraints)
print("Single non-feasible decision variables:", res2.objectives, "with constraint values", res2.constraints)
print("Multiple decision variables:", res3.objectives, "with constraint values", res3.constraints)
```

    N of objectives: 2
    N of variables: 2
    N of constraints: 1
    Single feasible decision variables: [[ 0. 10.]] with constraint values [[4.]]
    Single non-feasible decision variables: [[30. 18.]] with constraint values [[-2.]]
    Multiple decision variables: [[33. -9.]
     [13. -3.]
     [45. -5.]] with constraint values [[ 1.]
     [ 3.]
     [-1.]]


---

# Item X

Let
$$
H(x) = \begin{cases}
    1 & \text{if} -0.5 \leq x \leq 0.5
    \\ 0 & \text{otherwise}
\end{cases} \, .
$$

Compute $F(x) = \int_{-\infty}^{\infty} \tilde{H}_a(y) \tilde{H}_b(x-y) \, dx$ where $\tilde{H}_a(x) = \frac{1}{a} H\left( \frac{x}{a} \right)$.

---

We can work with the definition:
\begin{align}
\int_{-\infty}^{\infty} \tilde{H}_a(y) \tilde{H}_b(x-y) \, dx
&= \frac{1}{a} \int_{-\infty}^{\infty} H\left(\frac{y}{a}\right) \tilde{H}_b(x-y) \, dx
\\ &= \frac{1}{a} \int_{-a/2}^{a/2} \tilde{H}_b(x-y) \, dx
\\ &= \frac{1}{ab} \int_{-a/2}^{a/2} H\left(\frac{x-y}{b}\right) \, dx
\\ &= \frac{1}{ab} \int_{-a/2}^{a/2} H\left(\frac{y-x}{b}\right) \, dx
\\ &= \frac{1}{ab} \max\left( R-L, 0 \right)
\end{align}

where
$$
L = \max\left(-\frac{a}{2}, x-\frac{b}{2} \right)
$$
$$
R = \min\left(\frac{a}{2}, x+\frac{b}{2} \right)
$$

```python
def convolution(a,b):
    return lambda x: (a*b)**-1 * max(min(a/2.0,x+b/2.0)-max(-a/2.0,x-b/2.0),0)
```

```python
conv = convolution(4,3)
conv(-1.5)
```

    0.16666666666666666

```python

```


# Mathematical model of the curtain

## Model parameters
- $a$: The height of the curtain, $\in [3,7]$m
- $b$: The width of the curtain, $\in [4,8]$m
- $c$: The thickness of the curtain, $\in [0.5, 2]$mm
- $\rho = 1300$ kg/(m^3): Density of PVC
- $r_r$: The radius of the core of the roll
- $m_r$: The mass of the core, assumed to be a thin-walled tube.
- $m_{w}$: Mass of the weights in the bottom of the curtain which keep it stretched

## Signals
- $y$: Output signal. The height of the opening. $a \in [0, a)$ and $a-y$ is the amount of hanging curtain.
- $\theta$: The angle of the roller axis. This has a static relationship with $y$ derived below.
- $u$: Input signal. The torque acting on the roller axis.

## Kinematic relationships
### The radius of the roll as a function of the opening $y$
We assume a circular shape of the cross-section. The area of this is $A=\pi r^2$, which must equal the sum of the area of the core $A_c = \pi r_r^2$ and the area of the rolled up curtain $A_c = cy$. So,
$$r = \sqrt{r_r^2 + \frac{c}{\pi}y}, \quad \text{and}$$
$$ y = \frac{\pi}{c}(r^2 - r_r^2).$$
### Relationship between $y$ and $\theta$
Start with the relationship between the velocities
$$ r(y)\frac{d\theta}{dt} = \frac{dy}{dt},$$
which gives
$$ \frac{dy}{r(y)} = d\theta$$
$$ \frac{1}{\sqrt{r_r^2 + \frac{c}{\pi}y}} dy = d\theta.$$
Integrate both sides to obtain
$$ \frac{2\pi}{c}\sqrt{r_r^2 + \frac{c}{\pi} y} = \theta + K. $$
We need an initial condition between $y$ and $\theta$ to determine the constant $K$.
Let $y=0$ imply $\theta = 0$, hence
$$ K = \frac{2\pi}{c}r_r.$$
To find $y$ as a function of $\theta$:
$$ r_r^2 + \frac{c}{\pi}y = (\frac{c}{2\pi} \theta + r_r)^2$$
$$ r_r^2 + \frac{c}{\pi}y = (\frac{c}{2\pi})^2 \theta^2  + \frac{cr_r}{\pi}\theta + r_r^2$$
$$ \frac{c}{\pi}y = (\frac{c}{2\pi})^2 \theta^2  + \frac{cr_r}{\pi}\theta$$
$$ y = \frac{c}{4\pi}\theta^2 + r_r \theta = \theta(\frac{c}{4\pi}\theta + r_r).$$

## Inertial properties
### Mass of the hanging curtain and rolled-up curtain
The hanging curtain has mass
$$m_c = m_w + \rho b c (a-y),$$
and the rolled up part
$$m_{rc} = \rho b c y.$$
### Moment of inertia of the rolled-up and hanging curtain
The moment of inertia of the core is
$$I_r = m_rr_r^2, $$
the moment of inertia of the rolled-up portion of the curtain is
$$I_{rc}(y) = \frac{1}{2} m_{rc}(r^2 - r_r^2) = \frac{\rho b c^2}{\pi}y^2, $$ and the hanging mass, considering it to be a point mass at distance $r(y)$ from the axis of the roller is
$$I_c(y) = m_c r(y)^2 = (m_w + \rho b c (a-y))(r_r^2 + \frac{c}{\pi}y). $$
The complete moment of inertia with respect to the axis of the roller is
$$I(y) = I_r + I_{cr}(y) + I_c(y).$$

## Friction
We consider two types of friction. A friction in the bearings of the roller, and a friction between the curtain and the rails it travels in. The friction moment in the roller bearings is
$$ F_{r} = f_r \frac{d \theta}{dt}.$$
The friction in the curtain will be proportional to both the velocity $\frac{dy}{dt}$ and the length of hanging curtain $a-y$:
$$F_c = f_c (a-y) \frac{d y}{dt}.$$

## ODE
$$I(y) \ddot{\theta} = -m_c(y)r(y)g - f_r\dot{\theta}-f_c(a-y)\dot{y} + u.$$

```python
import sympy as sy
sy.init_printing()
```

```python
a,b,c,rr,rho,m_r,m_w = sy.symbols('a b c r_r rho m_r m_w')
y,u,theta,r = sy.symbols('y u theta r')
```

```python
At = sy.pi*r**2
Ar = sy.pi*rr**2
Ac = y*c
eq1 = At - Ar -Ac
eq1
```

```python
sy.solve(eq1, y)
```

```python
sy.integrate(1/sy.sqrt(rr**2 + c/sy.pi*y), y)
```

```python

```


# Supplementary Practice Problems

These are similar to programming problems you may encounter in the mid-terms. They are not graded but we will review them in lab sessions.

**1**. (10 points) The logistic map is defined by the following simple function

$$
f(x) = rx(1-x)
$$

For $x_0 = 0.1$ and $r = 4.0$, store the first 10 values of the iterated logistic map $x_{i+1} = rx_i(1-x_i)$ in a list. The first value in the list should be $x_0$.

```python
def logistic(x, r):
    """Blah"""
    return r*x*(1-x)
```

```python
import numpy as np
```

```python
n = 10
xs = np.zeros(n)
xs[0] = 0.1
r = 4.0

for i in range(1, n):
    xs[i] = logistic(xs[i-1], r)
xs
```

    array([ 0.1       ,  0.36      ,  0.9216    ,  0.28901376,  0.82193923,
            0.58542054,  0.97081333,  0.11333925,  0.40197385,  0.9615635 ])

**2**. (10 points) Write a function to find the greatest common divisor (GCD) of 2 numbers using Euclid's algorithm.:

\begin{align}
\gcd(a,0) &= a \\
\gcd(a, b) &= \gcd(b, a \mod b)
\end{align}

Find the GCD of 5797 and 190978.

Now write a function to find the GCD given a collection of numbers.

Find the GCD of (24, 48, 60, 120, 8).

```python
def gcd(a, b):
    """Blah"""
    if b == 0:
        return a
    else:
        return gcd(b, a % b)
```

```python
gcd( 5797, 190978)
```

    17

```python
from functools import reduce
```

```python
def gcds(xs):
    """Blah"""
    return reduce(lambda a, b: gcd(a, b), xs)
```

```python
xs = (24, 48, 60, 120, 8)
gcds(xs)
```

    4

**3**. (10 points) Find the least squares linear solution to the following data

```
y = [1,2,3,4]
x1 = [1,2,3,4]
x2 = [2,3,4,5]
```

That is, find the "best" intercept and slope for the variables `x1` and `x2`.

```python
from scipy.linalg import lstsq
```

```python
y = [1,2,3,4]
x1 = [1,2,3,4]
x2 = [2,3,4,5]

X = np.c_[np.ones(len(y)), x1, x2]
lstsq(X, y)
```

    (array([-0.33333333,  0.66666667,  0.33333333]),
     array([], dtype=float64),
     2,
     array([  9.34413269e+00,   8.28965828e-01,   1.18039779e-16]))

**4**. (10 points) Read the `mtcars` data frame from R to a `pandas` DataFrame. Find the mean `wt` and `mpg` for all cars grouped by the number of `gear`s.

```python
%load_ext rpy2.ipython
```

```python
mtcars = %R mtcars
```

```python
mtcars.groupby('gear')[['wt', 'mpg']].mean()
```

      wt
      mpg

      gear

      3.0
      3.892600
      16.106667

      4.0
      2.616667
      24.533333

      5.0
      2.632600
      21.380000

**5**. (10 points) Read the `iris` data frame from R to a `pandas` DataFrame. Make a `seaborn` plot showing a linear regression of `Petal.Length` (y) against `Sepal.Length` (x). Make a separate regression line for each `Species`.

```python
iris = %R iris
```

```python
%matplotlib inline
```

```python
import seaborn as sns
```

```python
sns.lmplot(x='Petal.Length', y='Sepal.Length', hue='Species', data=iris)
pass
```

**6**. (10 points) Write a function that can flatten a nested list of arbitrary depth. Check that

```python
flatten([1,[2,3],[4,[5,[6,7],8],9],10,[11,12]])
```

returns

```python
[1,2,3,4,5,6,7,8,9,10,11,12]
```

For simplicity, assume that the only data structure you will encounter is a list. You can check if an item is a list by using

```python
isinstance(item, list)
```

```python
def flatten(xs, acc=[]):
    """Blah"""
    if not isinstance(xs, list):
        acc.append(xs)
    else:
        for x in xs:
            flatten(x)
    return acc
```

```python
flatten([1,[2,3],[4,[5,[6,7],8],9],10,[11,12]])
```

    [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]

**7**. (10 points) Create the following table

```python
array([[  1,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0],
       [  1,   1,   0,   0,   0,   0,   0,   0,   0,   0,   0],
       [  1,   2,   1,   0,   0,   0,   0,   0,   0,   0,   0],
       [  1,   3,   3,   1,   0,   0,   0,   0,   0,   0,   0],
       [  1,   4,   6,   4,   1,   0,   0,   0,   0,   0,   0],
       [  1,   5,  10,  10,   5,   1,   0,   0,   0,   0,   0],
       [  1,   6,  15,  20,  15,   6,   1,   0,   0,   0,   0],
       [  1,   7,  21,  35,  35,  21,   7,   1,   0,   0,   0],
       [  1,   8,  28,  56,  70,  56,  28,   8,   1,   0,   0],
       [  1,   9,  36,  84, 126, 126,  84,  36,   9,   1,   0],
       [  1,  10,  45, 120, 210, 252, 210, 120,  45,  10,   1]])
```

Start with the first row

```
[  1,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0]
```

and build the subsequent rows using a simple rule that only depends on the previous row.

```python
xs = []

r = [ 0, 1,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0]
for i in range(10):
    s = [0]
    for x, y in zip(r[:], r[1:]):
        s.append(x + y)
    xs.append(s[1:])
    r = s
```

```python
np.array(xs)
```

    array([[  1,   1,   0,   0,   0,   0,   0,   0,   0,   0,   0],
           [  1,   2,   1,   0,   0,   0,   0,   0,   0,   0,   0],
           [  1,   3,   3,   1,   0,   0,   0,   0,   0,   0,   0],
           [  1,   4,   6,   4,   1,   0,   0,   0,   0,   0,   0],
           [  1,   5,  10,  10,   5,   1,   0,   0,   0,   0,   0],
           [  1,   6,  15,  20,  15,   6,   1,   0,   0,   0,   0],
           [  1,   7,  21,  35,  35,  21,   7,   1,   0,   0,   0],
           [  1,   8,  28,  56,  70,  56,  28,   8,   1,   0,   0],
           [  1,   9,  36,  84, 126, 126,  84,  36,   9,   1,   0],
           [  1,  10,  45, 120, 210, 252, 210, 120,  45,  10,   1]])

**8**. (10 points) Read the following data sets into DataFrames.

- url1 = "https://raw.github.com/vincentarelbundock/Rdatasets/master/csv/DAAG/hills.csv"
- url2 = "https://raw.github.com/vincentarelbundock/Rdatasets/master/csv/DAAG/hills2000.csv"

Create a new DataFraem only containing the names present in both DataFrames. Drop the `timef` column and have a single column for `dist` , `climb` and `time` that shows the average value of the two DataFrames. The final DtataFrame will thus have 4 columns (name, dist, climb, time).

```python
import pandas as pd
```

```python
url1 = "https://raw.github.com/vincentarelbundock/Rdatasets/master/csv/DAAG/hills.csv"
url2 = "https://raw.github.com/vincentarelbundock/Rdatasets/master/csv/DAAG/hills2000.csv"
```

```python
df1 = pd.read_csv(url1)
df2 = pd.read_csv(url2)
```

```python
df1.head(3)
```

      Unnamed: 0
      dist
      climb
      time

      0
      Greenmantle
      2.5
      650
      16.083

      1
      Carnethy
      6.0
      2500
      48.350

      2
      Craig Dunain
      6.0
      900
      33.650

```python
df2.head(3)
```

      Unnamed: 0
      dist
      climb
      time
      timef

      0
      Tiso Carnethy
      6.0
      2500
      0.782222
      0.919167

      1
      Criffel
      7.0
      1800
      0.793333
      1.003333

      2
      Chapelgill
      1.5
      1400
      0.314444
      0.376667

```python
df = pd.merge(df1, df2, on='Unnamed: 0')
```

```python
df.head(3)
```

      Unnamed: 0
      dist_x
      climb_x
      time_x
      dist_y
      climb_y
      time_y
      timef

      0
      Craig Dunain
      6.0
      900
      33.650
      6.0
      900
      0.546111
      0.625833

      1
      Ben Lomond
      8.0
      3070
      62.267
      9.0
      3192
      1.037778
      1.199167

      2
      Goatfell
      8.0
      2866
      73.217
      8.0
      2866
      1.227778
      1.526389

```python
df = df.drop('timef', axis=1)
df.head(3)
```

      Unnamed: 0
      dist_x
      climb_x
      time_x
      dist_y
      climb_y
      time_y

      0
      Craig Dunain
      6.0
      900
      33.650
      6.0
      900
      0.546111

      1
      Ben Lomond
      8.0
      3070
      62.267
      9.0
      3192
      1.037778

      2
      Goatfell
      8.0
      2866
      73.217
      8.0
      2866
      1.227778

```python
df.dtypes
```

    Unnamed: 0     object
    dist_x        float64
    climb_x         int64
    time_x        float64
    dist_y        float64
    climb_y         int64
    time_y        float64
    dtype: object

```python
df['dist'] = df.loc[:, ['dist_x', 'dist_y']].mean(skipna=True, axis=1)
df['climb'] = df.loc[:, ['climb_x', 'climb_y']].mean(skipna=True, axis=1)
df['time'] = df.loc[:, ['time_x', 'time_y']].mean(skipna=True, axis=1)
df = df.drop(['dist_x', 'dist_y', 'climb_x', 'climb_y', 'time_x', 'time_y'], axis=1)
df.columns = ['name', 'dist', 'climb', 'time']
```

```python
df.head(3)
```

      name
      dist
      climb
      time

      0
      Craig Dunain
      6.0
      900.0
      17.098056

      1
      Ben Lomond
      8.5
      3131.0
      31.652389

      2
      Goatfell
      8.0
      2866.0
      37.222389

```python

```


# Linear Regression

## Model

given a dataset $\left \{ (x^{(1)},y^{(1)}),...,(x^{(n)},y^{(n)}) \right \} $, where $x^{(i)} \in \mathbb{R}^{d},\ y^{(i)} \in \mathbb{R}$.

linear regression is trying to model $y^{(i)}$ from $x^{(i)}$ by a linear model:

$$h_{\theta}(x^{(i)}) = \theta_{0} + \theta_{1}x_{1}^{(i)} + \theta_{2}x_{2}^{(i)} + ... + \theta_{d}x_{d}^{(i)}\approx y^{(i)}$$

for the sake of simplicity, we set $x_{0}^{(i)}=1,\ x^{(i)} = (x_{0}^{(i)},...,x_{d}^{(i)}),\ \theta=(\theta_{0},...,\theta_{d})$, then:

$$h_{\theta}(x^{(i)}) = \sum_{j=0}^{d}\theta_{j}x_{j}^{(i)} = \theta^{T}x^{(i)}$$

## Loss Function

given $\hat{y}=(h_{\theta}^{(1)},...,h_{\theta}^{(n)}),\ y=(y^{(1)},...,y^{(n)})$.

we want to approximate $y$ by $\hat{y}$, or equivalent to say, we want to minimize the distance between $\hat{y}$ and $y$.

using euclidean distance, we derive the loss function for linear regression:

$$J(\theta) = \frac{1}{2}\sum_{i=1}^{n}(h_{\theta}x^{(i)} - y^{(i)})^2$$

## Gradient Descent

after defining model and loss function, our goal now is to find the model that minimize the loss function:

$$\hat{\theta} = \underset{\theta}{argmin}\ J(\theta)$$

to minimize the loss function, we can set

$$\nabla J(\theta)=0$$

but to find the analytic solution of this equation is usually impossible.

alternatively, we can init $\theta$ randomly, then iteratively move $\theta$ towards the direction that makes $J(\theta)$ smaller.

remmenber the oposite direction of gradient is the fastest direction that makes functions smaller, we derive gradient descent:

$$\theta := \theta - \alpha\nabla{J(\theta )}$$

$\alpha > 0$ is called the learning rate

appendix: proof of opposite gradient as fastest descent

for all $l\in \mathbb{R}^{d}, \left \| l \right \| =1$

$$\lim_{x \to 0} \frac{J(\theta + x l) - J(\theta ) }{x}=l \cdot \nabla{J(\theta )} >= -\left \| l \right \|\left \|J(\theta)  \right \| = -\left \|  J(\theta )\right \|   $$

equality obtained only if $l$ is in the opposite direction of $\nabla{J(\theta )}$

```python
import numpy as np

X = 2 * np.random.rand(100, 1)
y = 4 + 3 * X + np.random.randn(100, 1)
X_new = np.array([[0], [2]])
```

```python
"""LinearRegression actually uses SVD to get pseudoinverse, based on scipy.linalg.lstsq()"""
from sklearn.linear_model import LinearRegression

lin_reg = LinearRegression()
lin_reg.fit(X, y)
lin_reg.intercept_, lin_reg.coef_
```

    (array([4.0134073]), array([[3.00194802]]))

```python
y_predict = lin_reg.predict(X_new)
y_predict
```

    array([[ 4.0134073 ],
           [10.01730333]])

```python
import matplotlib.pyplot as plt

plt.plot(X_new, y_predict, "r-", linewidth=2, label="Predictions")
plt.plot(X, y, "b.")
plt.xlabel("$x_1$", fontsize=18)
plt.ylabel("$y$", rotation=0, fontsize=18)
plt.legend(loc="upper left", fontsize=14)
plt.axis([0, 2, 0, 15])
plt.show()
```

### stochastic gradient descent

in practice, to update $\theta$ by gradient descent(concretely batch gradient descent), at each step, we need to calculate all sample's gradient, too slow.

to fix this problem, we can only use one sample's gradient at a time, choose that sample randomly:

$$\theta := \theta - \alpha{l(h_{\theta}(x^{(i)}), y^{(i)})}$$

```python
from sklearn.linear_model import SGDRegressor

sgd_reg = SGDRegressor(max_iter=1000, tol=1e-3, penalty=None, eta0=0.1)
sgd_reg.fit(X, y.ravel())

sgd_reg.intercept_, sgd_reg.coef_
```

    (array([4.02841216]), array([3.02324524]))

### mini-batch gradient descent

sgd too random, batch gradient descent too slow, we can use some(not one, not all) samples at a time:

$$\theta := \theta - \alpha\sum_{i \in batch}{l(h_{\theta}(x^{(i)}), y^{(i)})}$$

## update rule of linear regression

for linear regression, we have:

$$
\begin{equation}
\begin{split}
\frac{\partial }{\partial \theta_{j}}J(\theta ) &=  \frac{\partial }{\partial \theta_{j}}\frac{1}{2}\sum_{i=1}^{n}(h_{\theta }(x^{(i)}) - y^{(i)})^2  \\
&=\sum_{i=1}^{n}(h_{\theta }(x^{(i)}) - y^{(i)})\cdot{}\frac{\partial }{\partial \theta_{j}}(h_{\theta }(x^{(i)}) - y^{(i)})\\
& =\sum_{i=1}^{n}(h_{\theta }(x^{(i)}) - y^{(i)})x_{j}^{(i)}
\end{split}
\end{equation}
$$

so we have the update rule for linear regression:

$$\theta_{j}: =\theta_{j} - \alpha\sum_{i=1}^{n} (h_{\theta }(x^{(i)}) - y^{(i)})x_{j}^{(i)} $$

combine all dimensions, we have:

$$\theta: =\theta - \alpha\sum_{i=1}^{n} (h_{\theta }(x^{(i)}) - y^{(i)})\cdot x^{(i)} $$

### matrix form

define $X = [(x^{(1)})^{T},...,(x^{(n)})^{T}]$, then we can write $J(\theta)$ in matrix form:

$$
\begin{equation}
\begin{split}
J(\theta) &= \frac{1}{2}\sum_{i=1}^{n}(h_{\theta}x^{(i)} - y^{(i)})^2 \\
&= \frac{1}{2}\sum_{i=1}^{n}(\theta^{T}x^{(i)} - y^{(i)})^2 \\
&= \frac{1}{2}(X\theta - y)^{T}(X\theta - y)
\end{split}
\end{equation}
$$

we then have:

$$
\begin{equation}
\begin{split}
\nabla{J(\theta )} &= \nabla\frac{1}{2}(X\theta - y)^{T}(X\theta - y) \\
&= \frac{1}{2}\nabla(\theta^{T}X^{T}X\theta - y^{T}(X\theta) - (X\theta)^{T}y) \\
&= \frac{1}{2}\nabla(\theta^{T}X^{T}X\theta - 2(X^{T}y)^{T}\theta) \\
&= \frac{1}{2}(2X^{T}X\theta - 2(X^{T}y)) \\
&= X^{T}X\theta - X^{T}y
\end{split}
\end{equation}
$$

here we use

1.$a^{T}b=b^{T}a$, obvious.

2.$\nabla{a^{T}x}=a$, obvious.

3.$\nabla{x^{T}Ax} = (A + A^{T})x$, proof:

$$
\begin{equation}
\begin{split}
\frac{\partial}{\partial x_{i}}x^{T}Ax &=
\frac{\partial}{\partial x_{i}}{\sum_{j=1}^{n}}{\sum_{k=1}^{n}}a_{jk}x_{j}x_{k} \\
&= \frac{\partial}{\partial x_{i}}(\sum_{j\ne{i}}a_{ji}x_{j}x_{i} + \sum_{k\ne{i}}a_{ik}x_{i}x_{k} + a_{ii}x_{i}^{2}) \\
&= \sum_{j\ne{i}}a_{ji}x_{j} + \sum_{k\ne{i}}a_{ik}x_{k} + 2a_{ii}x_{ii} \\
&= \sum_{j=1}^{n}a_{ji}x_{j} + \sum_{k=1}^{n}a_{ik}x_{k} \\
&= (A^{T}x)_{i} + (Ax)_{i}
\end{split}
\end{equation}
$$

finally, we get the matrix form of the update rule:

$$\theta: =\theta - \alpha X^{T}(X\theta-\mathbf{y} ) $$

## analytic solution

from above, we have:

$$\nabla{J(\theta )} = X^{T}X\theta - X^{T}y$$

so the equation of zero gradient change to:

$$X^{T}X\theta - X^{T}y = 0$$

if $X^{T}X$ is invertible:

$$\theta = (X^{T}X)^{-1}X^{T}y$$

if $X^{T}X$ is not invertible, the equation also have solution. proof:

on one hand $X\theta = 0 \Rightarrow X^{T}X\theta=0$

on the other hand $X^{T}X\theta=0 \Rightarrow \theta^{T}X^{T}X\theta=0 \Rightarrow (X\theta)^{T}X\theta=0 \Rightarrow X\theta=0$

so $X\theta=0 \Leftrightarrow X^{T}X\theta=0$, that is to say $null(X^{T}X)=null(X)$

we easily derive from the above that $rank(X^{T}X) = rank(X) = rank(X^{T})$

but $range(X^{T}X) \subseteq  range(X^{T})$, so we must have $range(X^{T}X)=range(X^{T})$

```python
X_b = np.c_[np.ones((100, 1)), X]
theta_best = np.linalg.inv(X_b.T.dot(X_b)).dot(X_b.T).dot(y)
theta_best
```

    array([[4.0134073 ],
           [3.00194802]])

## geometric interpretation of the linear regression

consider the linear space $S = span\left \{  columns\ of\ X \right \}$, linear combination of $S$ could be write as $X\theta$, then:

$X\theta$ is the projection of $y$ on $S \Leftrightarrow$ $X\theta - y$ orthogonal with $S \Leftrightarrow$ orthogonal with $columns\ of\ X \Leftrightarrow X^{T}(X\theta - y)=0$

so linear regression could be interpret as finding the projection of $y$ on $S$.

## probabilistic interpretation

assume targets and inputs are related via:

$$y^{(i)} = \theta^{T}x^{(i)} + \epsilon^{(i)}$$

where $\epsilon^{(i)}$ is the error term, assume that $\epsilon^{(i)}$ are distributed IID according to Gaussian with mean 0 and variance $\sigma^{2}$, i.e the density of $\epsilon^{(i)}$ is given by:

$$p(\epsilon^{(i)}) = \frac{1}{\sqrt{2\pi}\sigma}exp\left (-\frac{(\epsilon^{(i)})^{2}}{2\sigma^{2}}\right )$$

this implies:

$$p(y^{(i)}|x^{(i)}; \theta) = \frac{1}{\sqrt{2\pi}\sigma}exp\left ( -\frac{(y^{(i)} - \theta^{T}x^{(i)})^{2}}{2\sigma^{2}}\right)$$

we should denote that $\theta$ is not a random variable.

given $X$ and $\theta$, what is the probability of $y$? we call it the likelihood function:

$$L(\theta) = L(\theta; X,y) = p(y|X; \theta)$$

for the above assumptions:

$$
\begin{equation}
\begin{split}
L(\theta) &= \prod_{i=1}^{n}p(y^{(i)}|x^{(i)}; \theta) \\
&= \prod_{i=1}^{n}\frac{1}{\sqrt{2\pi}\sigma}exp\left ( -\frac{(y^{(i)} - \theta^{T}x^{(i)})^{2}}{2\sigma^{2}}\right)
\end{split}
\end{equation}
$$

we should choose $\theta$ so as to make the data as high probability as possible, i.e we should choose $\theta$ to maxmize $L(\theta)$, this is called maximum likelihood. one step further:

$$maximize\ L(\theta) \Leftrightarrow maxmize\ log(L(\theta))$$

so we maximize the log likelihood, this is simpler.

$$
\begin{equation}
\begin{split}
l(\theta) &= log(L(\theta)) \\
&= log\prod_{i=1}^{n}\frac{1}{\sqrt{2\pi}\sigma}exp\left ( -\frac{(y^{(i)} - \theta^{T}x^{(i)})^{2}}{2\sigma^{2}}\right) \\
&= \sum_{i=1}^{n}log\frac{1}{\sqrt{2\pi}\sigma}exp\left ( -\frac{(y^{(i)} - \theta^{T}x^{(i)})^{2}}{2\sigma^{2}}\right) \\
&= n\ log\frac{1}{\sqrt{2\pi}\sigma} - \frac{1}{2\sigma^{2}}\sum_{i=1}^{n}(y^{(i)} - \theta^{T}x^{(i)})^{2}
\end{split}
\end{equation}
$$

hence, maximizing $l(\theta)$ gives the same answer as minimizing

$$\frac{1}{2}\sum_{i=1}^{n}(y^{(i)} - \theta^{T}x^{(i)})^{2} = J(\theta)$$

so linear regression $\Leftrightarrow $ maximum likelihood given Gaussian error

## regularization

to lower variance $\Rightarrow $ to limit model's complexity $\Rightarrow $ to prevent the absolute value of parameters to be too large $\Rightarrow $ we add punishment term concerning the absolute value of parameters on $J(\theta)$

choose $\lambda{\left \| \theta   \right \|}_{2}^{2}$ as the punishment term:

$$J(\theta) := J(\theta) + \lambda{\left \| \theta   \right \|}_{2}^{2}$$

$\lambda$ is the regularization hyperparameter, linear regression with $l_{2}$ loss is ridge regression.

otherwise, if we choose $\lambda{\left \| \theta   \right \|}_{1}$ as the punishment term:

$$J(\theta) := J(\theta) + \lambda{\left \| \theta   \right \|}_{1}$$

that is called lasso regression.

## probabilistic interpretation of regularization

as before, we assume:

$$p(y^{(i)}|x^{(i)}; \theta) = \frac{1}{\sqrt{2\pi}\sigma}exp\left ( -\frac{(y^{(i)} - \theta^{T}x^{(i)})^{2}}{2\sigma^{2}}\right)$$

in bayes perspective, we can also assume $\theta \sim N(0, \sigma_{0})$, that is:

$$p(\theta) = \frac{1}{\sqrt{2\pi}\sigma_{0}}exp\left ( -\frac{\left \| \theta   \right \|_{2}^{2}}{2\sigma_{0}^{2}}\right)$$

by bayes rule:

$$P(\theta|y)=\frac{P(y|\theta)P(\theta)}{P(y)}$$

thus we have maximum a posteriori estimation(MAP):

$$
\begin{equation}
\begin{split}
\hat{\theta} =&\underset{\theta}{argmax}P(\theta|y)\\
=&\underset{\theta}{argmax}P(y|\theta)P(\theta) \\
=&\underset{\theta}{argmax}\ logP(y|\theta)P(\theta) \\
=&\underset{\theta}{argmax}\ log\ exp\left ( \sum_{i=1}^{n}-\frac{(y^{(i)} - \theta^{T}x^{(i)})^{2}}{2\sigma^{2}}\right)exp\left ( -\frac{\left \| \theta   \right \|_{2}^{2}}{2\sigma_{0}^{2}}\right)\\
=&\underset{\theta}{argmin}\left(\sum_{i=1}^{n}\frac{(y^{(i)} - \theta^{T}x^{(i)})^{2}}{2\sigma^{2}} + \frac{\left \| \theta   \right \|_{2}^{2}}{2\sigma_{0}^{2}}\right)\\
=&\underset{\theta}{argmin}\left(\sum_{i=1}^{n}(y^{(i)} - \theta^{T}x^{(i)})^{2} + \frac{\sigma^{2}}{\sigma_{0}^{2}}\left \| \theta   \right \|_{2}^{2}\right)
\end{split}
\end{equation}
$$

that is exactly minimize:

$$J(\theta) = \frac{1}{2}\sum_{i=1}^{n}(y^{(i)} - \theta^{T}x^{(i)})^2 + \lambda{\left \| \theta   \right \|}^{2}$$

with $\lambda=\frac{\sigma^{2}}{2\sigma_{0}^{2}}$

in one word:

1. linear regression $\Leftrightarrow $ maximum likelihood estimation(MLE) where noise is guassian.
2. ridge regression $\Leftrightarrow $ maximum a posteriori estimation(MAP) where noise and prior are guassian.

```python
from sklearn.linear_model import Ridge

ridge_reg = Ridge(alpha=1, solver="cholesky", random_state=42)
ridge_reg.fit(X, y)
ridge_reg.predict([[1.5]])
```

    array([[8.46777839]])

```python
"""set SGDRegressor penalty=l2"""
sgd_reg = SGDRegressor(penalty="l2", max_iter=1000, tol=1e-3, random_state=42)
sgd_reg.fit(X, y.ravel())
sgd_reg.predict([[1.5]])
```

    array([8.63160742])

## Lasso

```python
"""use Lasso or set SGDRegressor penalty=l1"""
from sklearn.linear_model import Lasso

lasso_reg = Lasso(alpha=0.1)
lasso_reg.fit(X, y)
lasso_reg.predict([[1.5]])
```

    array([8.34961266])

```python
"""combination of ridge and lasso"""
from sklearn.linear_model import ElasticNet

elastic_net = ElasticNet(alpha=0.1, l1_ratio=0.5)
elastic_net.fit(X, y)
elastic_net.predict([[1.5]])
```

    array([8.2272819])

## Polynomial Regression

```python
m = 100
X = 6 * np.random.rand(m, 1) - 3
y = 0.5 * X**2 + X + 2 + np.random.randn(m, 1)
```

```python
plt.plot(X, y, "b.")
plt.xlabel("$x_1$", fontsize=18)
plt.ylabel("$y$", rotation=0, fontsize=18)
plt.axis([-3, 3, 0, 10])
plt.show()
```

```python
from sklearn.preprocessing import PolynomialFeatures

poly_features = PolynomialFeatures(degree=2, include_bias=False)
X_poly = poly_features.fit_transform(X)

lin_reg = LinearRegression()
lin_reg.fit(X_poly, y)
lin_reg.intercept_, lin_reg.coef_
```

    (array([2.07532595]), array([[0.86616713, 0.50615805]]))

```python
X_new=np.linspace(-3, 3, 100).reshape(100, 1)
X_new_poly = poly_features.transform(X_new)
y_new = lin_reg.predict(X_new_poly)
```

```python
plt.plot(X, y, "b.")
plt.plot(X_new, y_new, "r-", linewidth=2, label="Predictions")
plt.xlabel("$x_1$", fontsize=18)
plt.ylabel("$y$", rotation=0, fontsize=18)
plt.legend(loc="upper left", fontsize=14)
plt.axis([-3, 3, 0, 10])
plt.show()
```

```python
"""using pipeline"""
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

def polynomial_regression(degree, include_bias=True):
    return Pipeline([
        ("poly_features", PolynomialFeatures(degree=degree, include_bias=False)),
        ("std_scaler", StandardScaler()),
        ("lin_reg", LinearRegression()),
    ])
```

```python
poly_reg = polynomial_regression(degree=2)
poly_reg.fit(X, y)

X_new = np.linspace(-3, 3, 100).reshape(100, 1)
y_new = poly_reg.predict(X_new)

plt.plot(X, y, "b.")
plt.plot(X_new, y_new, "r-", linewidth=2, label="Predictions")
plt.xlabel("$x_1$", fontsize=18)
plt.ylabel("$y$", rotation=0, fontsize=18)
plt.legend(loc="upper left", fontsize=14)
plt.axis([-3, 3, 0, 10])
plt.show()
```

```python

```


# kernel density estimation (KDE)
given a sample from a population (i.e., given data), kernel density estimation is a non-parametric technique to estimate the density of the underlying probability distribution that generated the data.

the technique is [non-parametric](https://en.wikipedia.org/wiki/Nonparametric_statistics) because it does not assume that the distribution takes the shape of any particular (parameterized) probability distribution.

let $f(x)$ be the [probability density function](https://en.wikipedia.org/wiki/Probability_density_function) of a continuous random variable $X$. then, when we sample from this distribution, the probability that $X$ falls in an infinitesimally small interval of width $dx$ centered at $x$ is $f(x)dx$. i.e.
\begin{equation}
    \text{Pr}(x-dx/2<X<x+dx/2)=f(x)dx
\end{equation}

the probability that $x$ falls in an interval $(a, b)$ is the sum of the density in that interval:
\begin{equation}
    \text{Pr}(a<X<b)=\int_a^bf(x)dx
\end{equation}

In typical applications, $f(x)$ is unknown, and all we have is a random sample from the distribution $f(x)$.

Say we have a random sample $x_1, x_2, ..., x_n$ drawn from the probability density $f(x)$, and we want to estimate $f(x_0)$, where $x_0$ is (probably) not in our sample. Kernel density estimation is a principled way to do this. Since we can do this for all $x_0$, we are using the data to estimate the data generating probability distribution.

we are still in the domain of *statistical inference* because we are aiming to make conclusions about the distribution in the population from a random sample.

[here](https://mathisonian.github.io/kde/) is an interactive description of KDE.

```julia
using CSV
using DataFrames
using PyPlot
using ScikitLearn # machine learning package
using Statistics
using Random
using LaTeXStrings # for L"$x$" to work instead of needing to do "\$x\$"
using Printf

# (optional)change settings for all plots at once, e.g. font size
rcParams = PyPlot.PyDict(PyPlot.matplotlib."rcParams")
rcParams["font.size"] = 16
```

    16

## the distribution of masses of apples on an orchard

we own an apple orchard in Oregon. we want to know the distribution of the masses of apples so we can (a) estimate our revenues and then (b) make sound investments in our orchard without taking on too much risk. it is prohibitively expensive to measure the mass of every apple, so we instead take a random sample of the apples in our orchard and record their masses. the data is in `apples.csv`.

let $M$ be the mass of an apple on our orchard, a random variable. the idea is that some probability density $f(m)$ governs $M$; that is, when we sample an apple and measure its mass, we are drawing a number from the distribution $f(m)$. we do not know $f(m)$; all we have is a sample of apple masses, $m_1, m_2,...,m_{20}$, in other words, 20 draws from the distribution $f(m)$. we will use this sample to estimate $f(m)$.

```julia
df = CSV.read("apples.csv")
first(df, 5)
```

	mass

	Float64

5 rows × 1 columns

1	108.205

2	92.1145

3	107.959

4	58.8771

5	100.104

```julia
println("# apples sampled: ", nrow(df))
@printf("apple masses range from %.2f g to %.2f g",
    minimum(df[:, :mass]), maximum(df[:, :mass]))
```

    # apples sampled: 20
    apple masses range from 58.88 g to 129.10 g

## histograms are ineffective for estimating probability densities

in a histogram, we partition the set of possible values of the random variable $M$ into discrete bins (usually of equal width) and count the number of samples (in this case, apples) that fall in each bin. we can normalize the height of the bins such that the sum of bin heights times bin widths is one (passing `normed=true` to `hist()` in matplotlib). then, the histogram serves as a (very poor) estimate of the data generating distribution $f(m)$. this estimate is poor because:

* the density estimate is discontinuous (not smooth) at some points, flat within the bins, and exactly zero at many points. the locations of the discontinuities are an artifact of where we choose the bins to be (centers and widths), rather than the underlying probability distribution
* the density estimate depends on where the bins start and end-- especially the drop-offs to exactly zero density
* qualitative features in the data can be hidden depending on shifts in the bins, even if the bins are the same width (we illustrate below)
* as we add dimensions, the number of bins grows exponentially; thus, most bins will be empty in a higher dimensional histogram unless we have a very large number of samples. empty bins => zero density estimate in that bin

to illustrate, make a histogram by partitioning the apple mass interval $[0, 130]$ into 8 equal-width bins. then, shift the bins to the right anywhere from 0 to 50 g. the histogram can reveal/hide bimodality in the distribution of apple masses depending on the offset!

```julia
# first choice of bins
bins = range(0.0, stop=130.0, length=8)

# offsets to add to the bins to shift them to the right
offsets = range(0.0, stop=50.0, length=5)

for offset in offsets
    figure()
    # normed=true important so the histogram is a density estimate.
    hist(df[:, :mass], bins=bins .+ offset, normed=true)
    xlabel(L"mass of apple [g], $m$")
    ylabel(L"density [1/g], $\hat{f}(m)$") # decorate with hat b/c it's our estimate
    scatter(df[:, :mass], [-0.001 for i = 1:nrow(df)], marker="+")
    xlim([0, 170.0])
end
```

## kernel function
a kernel function $K(m, m_i)$ is a weighting function that assigns a weight to $m_i$ based on its distance from $m$. such kernel functions are typically indexed by a parameter $\lambda$ that dictates the width of the neighborhood around $m$ where the kernel with $m_i$ is "sizeable" (or, for some kernels, nonzero). the kernel assigns a weight to $m_i$ that decays as its distance from $m$ increases. the kernel is our way of smearing out the data points $m_i$ such that they contribute some to our estimate of the density at $m$. the further a data point $m_i$ is from $m$, the less that data point contributes.

kernel properties:
* symmetry: $K_\lambda(m, m_i)=K_\lambda(m_i, m)$
* non-negative: $K_\lambda(m, m_i) \geq 0$
* like a probability density function: $\int_{-\infty}^\infty K_\lambda(m, m_i) dm = 1$
* decays as $m_i$ gets far from $m$: $\lim_{m_i \rightarrow \pm \infty} K_\lambda(m, m_i)=0$

**the Gaussian kernel**
\begin{equation}
K_\lambda(m,m_i):=\frac{1}{(2\pi\lambda^2)^{d/2}}\exp{\left(-\dfrac{||m-m_i||^2}{2\lambda^2}\right)}
\end{equation}
here, the bandwidth $\lambda$ of the kernel function is the variance of the Gaussian. $d$ is the dimension of the (possibly,) vectors $m$ and $m_i$.

**the Epanechnikov kernel**
\begin{equation}
K_\lambda(m,m_i):=\frac{3}{4\lambda}\bigl(1-\bigl[||m-m_i||/\lambda\bigr]^2\bigr)\mathbf{1}\bigl(||m_0-m_i||^2 < \lambda^2\bigr)
\end{equation} where $\mathbf{1}$ is the indicator function. the Epanechnikov kernel only has finite support, in contrast the Gaussian kernel.

let's write a function `K` that can compute both of these kernels, then visualize them.

```julia
function K(m::Float64, mᵢ::Float64, λ::Float64, wut_kernel::String)
    # the distance between m and mᵢ, normalized by bandwidth, scaled
    u = ((m - mᵢ) / λ) ^ 2
    if wut_kernel == "gaussian"
        return exp(-u / 2) / sqrt(2 * π * λ ^ 2)
    elseif wut_kernel == "epanechnikov"
        if u < 1.0
            return 3 / 4 * (1 - u) / λ
        else
            return 0.0
        end
    else # not implemented
        return NaN
    end
end

m = 0.0
mᵢs = range(-5.0, stop=5.0, length=300)
λs = [0.5, 1.0, 1.5]

fig, axs = subplots(2, 1, figsize=(10, 8),
                    sharex=true, sharey=true, tight_layout=true)

for λ in λs
    for (i, kernel_type) in enumerate(["gaussian", "epanechnikov"])
        k = [K(m, mᵢ, λ, kernel_type) for mᵢ in mᵢs]
        axs[i].plot(mᵢs, k, label="λ=$λ")
        axs[i].set_title(kernel_type)
        axs[i].set_ylabel(L"$K(m, m_i)$")
    end
end
xlabel(L"$m-m_i$")
legend()
```

## kernel density estimation

The kernel density estimation of the probability density $f(m)$ based on the sample data $m_1, m_2, ..., m_n$ is:
\begin{equation}
    \hat{f}_\lambda(m) = \frac{1}{n}\displaystyle \sum_{i=1}^n K_\lambda(m, m_i)
\end{equation}

To estimate the density at $m$, we add up the kernel of $m$ with all other data points. If $m$ is close to many data points ("close" determined by the kernel and, mostly, its bandwidth), the sum will "pile up" a lot of density at $m$ from the nearby "smeared out" data points and yield a high density. if $m$ is far from most data points, then the kernel of $m$ with the data points $m_i$ will be small or zero and little density will "pile up" at $m$.

A well-state description of KDE [here](http://compdiag.molgen.mpg.de/docs/talk_05_01_04_stefanie.pdf). To estimate the probability density:
* the center of the kernel is placed right over each data point
* the influence of each data point is spread out over its neighborhood (determined by the bandwidth of the kernel)
* the contribution from each data point to the density is summed to obtain the overall density estimate

Let's write our own KDE function that takes in:
* `m`: the mass at which we aim to estimate the density
* `m_sample`: the array of masses we sampled
* `λ`: the kernel bandwidth
* `wut_kernel`: which kernel we should use ("gaussian" or "epanechnikov")

and returns the kernel density estimate at `m`.

```julia
function corys_kde(m::Float64, m_sample::Array{Float64}, λ::Float64, wut_kernel::String)
    density_estimate = 0.0
    for i = 1:length(m_sample)
        density_estimate += K(m, m_sample[i], λ, wut_kernel)
    end
    return density_estimate / length(m_sample)
end

m = 100.0 # estimate the density here
λ = 10.9 # bandwidth
corys_kde(m, df[:, :mass], λ, "gaussian")
```

    0.020605031170185327

finally, let's use our function to estimate the data generating distribution $f(m)$ from our sample. we'll use both kernels, Gaussian and Epanechnikov, so we can compare.

```julia
# define apple masses at which we want to estimate the density (purely for visualization)
masses = range(10.0, stop=140.0, length=250)

# do the KDE
fig, axs = subplots(2, 1, figsize=(10, 8),
                    sharex=true, sharey=true, tight_layout=true)

for (i, wut_kernel) in enumerate(["gaussian", "epanechnikov"])
    # do KDE with this kernel
    ρ = [corys_kde(mass, df[:, :mass], λ, wut_kernel) for mass in masses]
    # plot estimated densities vs. masses
    axs[i].plot(masses, ρ)
    axs[i].set_ylabel(L"density of apples [1/g], $\hat{f}(m)$")
    axs[i].set_title(wut_kernel)
end
xlabel(L"mass of apple [g], $m$")
```

## using the estimated probability density
aside from visualizing the estimated data generating distribution, we can use it to answer questions, such as:

what is the probability that a randomly drawn apple from the orchard has a mass of less than 80 g?

\begin{equation}
\text{Pr}(M<80)=\int_0^{80} f(m)dx
\end{equation}

We can do this integration via Monte Carlo integration!
\begin{equation}
\int_0^{80} f(m)dx =80\frac{1}{80} \int_0^{80} f(m)dm = 80 \langle f(m) \rangle_{(0,80)}
\end{equation}

```julia
nb_samples = 1000 # number of Monte Carlo samples for the integration

avg_f = 0.0 # will be ⟨f⟩
# collect random samples of f(m) in m [0, 80]
for s = 1:nb_samples
    # sample a mass from 0 to 80
    mass = rand() * 80 # uniform number in [0, 80]
    avg_f += corys_kde(mass, df[:, :mass], λ, "gaussian")
end
avg_f / nb_samples * 80
```

    0.19362273913500289

## how to choose the bandwidth?
* bandwidth too large => oversmoothed density (data points smeared out so much that you lose the features of the underlying distribution)
* bandwidth too small => undersmoothed density (you see features in your density estimate that are not truly present in the underlying distribution)

$\lambda=1.0$: jagged features in the distribution that are [probably] not there

$\lambda=15.0$: smooth out data to the extent that you don't see bimodality

$\lambda=8.0$: just right, we see bimodality

there are several methods for choosing the bandwidths... we will do this in class later when we do cross-validation.

```julia
λs = range(1.0, stop=15.0, length=3)
wut_kernel = "gaussian"

for λ in λs
    # do KDE with this kernel
    ρ = [corys_kde(mass, df[:, :mass], λ, wut_kernel) for mass in masses]
    # plot estimated densities vs. masses
    plot(masses, ρ, label="λ=$λ")
end
xlabel("mass of apple [g]")
title("Gaussian KDE")
ylabel("density of apples [1/g]")
legend()
```

Silverman's rule of thumb (for one-dimensional data) for the bandwidth [[source](http://www.buch-kromann.dk/tine/nonpar/Nonparametric_Density_Estimation_multidim.pdf)]:

\begin{equation}
    \lambda = \biggl(\frac{4\hat{\sigma}^5}{3n} \biggr)^{1/5}
\end{equation}
where $n$ is the number of data points, $\hat{\sigma}$ is the standard deviation of the data.

```julia
λ_silverman = (4 * std(df[:, :mass]) ^ 5 / (3 * nrow(df))) ^ (1 / 5)
```

    10.911747572936896

## scikit-learn for KDE

the package `scikitlearn`, which we call from Python, allows us to do kernel density estimation with a variety of kernels. see the documentation [here](https://scikit-learn.org/stable/modules/density.html)

here is how we import scikit-learn functions into Julia.

```julia
@sk_import neighbors.kde : KernelDensity
```

    PyObject <class 'sklearn.neighbors.kde.KernelDensity'>

now, let's do kernel density estimation in scikitlearn with a Gaussian kernel and the same bandwidth `λ` as above. first, construct a `KernelDensity` object.

```julia
kde = KernelDensity(kernel="gaussian", bandwidth=λ)
```

    PyObject KernelDensity(algorithm='auto', atol=0, bandwidth=10.9, breadth_first=True,
                  kernel='gaussian', leaf_size=40, metric='euclidean',
                  metric_params=None, rtol=0)

next, we need to put our data in a format that scikitlearn needs. although our samples are scalars, we need to represent them as 1D vectors for scikitlearn. scikitlearn's `KernelDensity` object was designed with high-dimensional data in mind. so to make our sampled masses an array of 1D arrays for scikitlearn:

```julia
sample_masses = [[row[:mass]] for row in eachrow(df)]
```

    20-element Array{Array{Float64,1},1}:
     [108.20535675585407]
     [92.11445827312127]
     [107.95894817321351]
     [58.87711063761283]
     [100.10360270241122]
     [67.44315510523454]
     [92.14622115463766]
     [129.096902994266]
     [96.72133629711364]
     [66.9553570753678]
     [99.59985147205123]
     [128.11149783031908]
     [97.20868238556788]
     [81.27085357175854]
     [98.01357496629956]
     [113.1726581261136]
     [91.82359854827018]
     [95.53643088149084]
     [114.20489522388259]
     [114.88416683301007]

next, we pass scikitlearn the sampled data, via the "fit" function associated with the `KernelDensity` object.

```julia
kde.fit(sample_masses)
```

    PyObject KernelDensity(algorithm='auto', atol=0, bandwidth=10.9, breadth_first=True,
                  kernel='gaussian', leaf_size=40, metric='euclidean',
                  metric_params=None, rtol=0)

finally, we can score new samples, i.e. estimate the density at a different location in feature space. to estimate the density at 100 g, i.e. to estimate $f(100)$, we again need to represent our sample as an array of arrays (this could take multiple samples as input.

we also need to exponentiate the result because scikitlearn returns the *log* of the density.

**warning:** scikitlearn's `KernelDensity.score_samples()` returns the `log` of the density.

```julia
ρ_at_100g = exp.(kde.score_samples([[100.0]]))
```

    1-element Array{Float64,1}:
     0.020605031170185327

compare this to our KDE function; we get the same thing! so we succesfully reproduced scikitlearn's KDE module.

```julia
corys_kde(100.0, df[:, :mass], λ, "gaussian")
```

    0.020605031170185327

to be sure, let's plot the kernel density estimate of $f(m)$ from scikitlearn using the Gaussian and Epanechnikov kernels and ensure they are the same as ours. Indeed!

```julia
# do the KDE
fig, axs = subplots(2, 1, figsize=(10, 8),
                    sharex=true, sharey=true, tight_layout=true)

for (i, wut_kernel) in enumerate(["gaussian", "epanechnikov"])
    #
    # use scikit-learn for KDE
    #
    # construct KernelDenstiy object. tell it kernl and bandwidth to use
    kde = KernelDensity(kernel=wut_kernel, bandwidth=λ)
    # write sample as an array of arrays (to accommodate format of higher dimensional data)
    sample_masses = [[row[:mass]] for row in eachrow(df)]
    # pass the KernelDensity object the data, "fit"
    kde.fit(sample_masses)
    # estimate the denstiy at our specified `masses`.
    # this returns the log of the density so need to exponentiate
    ρ = exp.(kde.score_samples([[mass] for mass in masses]))

    # plot estimated densities vs. masses
    axs[i].plot(masses, ρ)
    axs[i].set_ylabel(L"density of apples [1/g], $\hat{f}(m)$")
    axs[i].set_title(wut_kernel)
end
xlabel(L"mass of apple [g], $m$")
```

## KDE in multiple dimensions

e.g. theft from vehicles in Vancouver, Canada. the raw data consist of geo-coordinates of theft from vehicles.

## Old Faithful (2D KDE)

>      Waiting time between eruptions and the duration of the eruption
     for the Old Faithful geyser in Yellowstone National Park, Wyoming,
     USA. A data frame with 272 observations on 2 variables.

[source](https://www.stat.cmu.edu/~larry/all-of-statistics/=data/faithful.dat)

```julia
df = CSV.read("faithful.dat", datarow=27, header=26, delim=" ", ignorerepeated=true)
df = df[:, [:eruption, :waiting]]
first(df, 6)
```

	eruption	waiting

	Float64	Int64

6 rows × 2 columns

1	3.6	79

2	1.8	54

3	3.333	74

4	2.283	62

5	4.533	85

6	2.883	55

```julia
figure()
scatter(df[:, :eruption], df[:, :waiting], marker="x")
xlabel("eruption time [min]")
ylabel("waiting time [min]")
title("old faithful data")
```

2D KDE in Scikitlearn is painful because it only allows us to supply a scalar value for $\lambda$, whereas the scale for the eruption time and waiting time is different. we would have to scale the data so one can use a radially symmetric Gaussian kernel as scikitlearn offers.

instead, we can use [Seaborn](https://seaborn.pydata.org/), a Python package built on top of matplotlib.

Seaborn automatically chooses the bandwidth using Scott's rule.

```julia
using PyCall
seaborn = pyimport("seaborn")
```

    PyObject <module 'seaborn' from '/usr/local/lib/python3.6/dist-packages/seaborn/__init__.py'>

```julia
seaborn.kdeplot(df[:, :eruption], df[:, :waiting], shade=true, bw="scott")
scatter(df[:, :eruption], df[:, :waiting], marker="x")
xlabel("eruption time [min]")
ylabel("waiting time [min]")
```

check out the [Seaborn Gallery](https://seaborn.pydata.org/examples/index.html). We can make a plot that shows the kernel density estimation in feature space as well as a kernel density estimation of the marginal distributions on the side.

`seaborn.jointplot` returns an object that has `ax_joint` as an attribute. This is a matplotlib axis on which we can draw.

```julia
p = seaborn.jointplot(df[:, :eruption], df[:, :waiting], kind="kde")
p.ax_joint.set_xlabel("eruption time [min]")
p.ax_joint.set_ylabel("waiting time [min]")
p.ax_joint.scatter(df[:, :eruption], df[:, :waiting], marker="o", s=2)
```

## seaborn 1D KDE

```julia
df = CSV.read("apples.csv")

seaborn.kdeplot(df[:, :mass], bw="silverman", shade=true)
seaborn.rugplot(df[:, :mass])
xlabel(L"mass of apple [g], $m$")
ylabel(L"density [1/g], $\hat{f}(m)$")
```