# Consistent Bayes: ODE Examples
---

Copyright 2018 Michael Pilosov

### Import Libraries

```python
# Mathematics and Plotting
from HelperFuns import * # pyplot wrapper functions useful for visualizations, numpy, scipy, etc.
%matplotlib inline
# %matplotlib notebook
# plt.xkcd(scale=1, length=1000, randomness=0)
from cbayes import sample, solve, distributions
# Interactivity
from ipywidgets import *
import matplotlib.colors as colors

# import numpy as np
from scipy.integrate import odeint
```

```python
plt.rcParams.update({'font.size': 14})
plt.rcParams['figure.figsize'] = 20, 10
```

```python
problem = 'ed'
```

# Example Summary

---

The second example is defined by the model for a harmonic oscillator given by the initial value problem

\begin{equation}
	\begin{cases}
		\frac{d^2u}{dt^2} &= -\lambda_2^2 u, \ t>0, \\
		u(0) &= \lambda_1.
	\end{cases}
\end{equation}

Choosing a particular parameter $\lambda=(\lambda_1,\lambda_2)\in\Lambda\subset[-1,1]\times\mathbb{R}^+$ corresponds to fixing an initial condition, $\lambda_1\in[-1,1]$, and frequency of oscillation, $\lambda_2$, where we again assume that $\lambda_2>0$.

The solution to the above is
\begin{equation}
	u(t) = \cos(\lambda_2 t + \arccos(\lambda_1)).
\end{equation}

---
## Define Functions

```python
def MSE_generator(model, obs_data, sigma=None):   # this generates a sum of squared residuals.
    def QoI_fun(inputs): # that conforms to our desired model input
        M = len(obs_data)
        predictions = model(inputs)
        assert predictions.shape[1] == M
        residuals = predictions - obs_data
        if sigma is not None:
            QoI = (1./M)*np.sum( (residuals/sigma)**2, axis=1 )   # MSE
        else:
            QoI = (1./M)*np.sum( (residuals/obs_data)**2, axis=1 )  # MRSE
        return QoI
    return QoI_fun

if problem == 'ed':
    # The model of the exponential decay
    def makemodel(t):
        def model(lam = np.array([[0.5, 0.5]]) ):
            QoI = lam[:,0].reshape(-1,1)*np.exp(np.outer(lam[:,1], -t))
            if QoI.shape[0] == 1:
                return QoI.ravel() # this allows support for simpler 1D plotting.
            else:
                return QoI
        return model
elif problem == 'ho':
    # The model of the harmonic oscillator
    def makemodel(t):
        def model(lam = np.array([[0, 1]]) ):
    #         return np.cos(lam[:,1]*t + np.arccos(lam[:,0]) )
            QoI = np.cos(np.outer(lam[:,1],t) + np.arccos(lam[:,0]).reshape(-1,1) )
            if QoI.shape[0] == 1:
                return QoI.ravel() # this allows support for simpler 1D plotting.
            else:
                return QoI
        return model

```

```python
##### FIXED PARAMETERS - DEFINE YOUR EXPERIMENT #####
num_observations = 50 # how many observations are you able to make?
start_time = 1
end_time = 5
####

t = np.linspace(start_time, end_time, num_observations)
model = makemodel(t)
u = model()
```

## Get a sense of the variation in the system you are studying

```python
n = 50
lam = np.random.random(size=(n,2))
if problem == 'ho':
    lam[:,0] = 0.5*lam[:,0] - 0.25
    lam[:,1] = lam[:,1] + 0.5
elif problem == 'ed':
    pass # keep as is.

sd_test_n = 0.025
# sd_test_u = 0.1
# sd_test_c = 0.1
mintime = 0
maxtime = 5
tt = np.linspace(mintime, maxtime, 1000)
m = makemodel(tt)
ui = m(lam)

um = m()
plt.cla()
show_obs = False
# Plotting the default IR results from the epidemic_IR function.
for i in range(n):
    d = ui[i,:]
    if i == n-1:
        plt.plot(tt, d, c='k', alpha=5/n, label='Potential Signals')
    else:
        plt.plot(tt, d, c='k', alpha=5/n) # I

# plt.scatter(t, u + np.random.randn(num_observations)*sd_test_n, s=50, c='k', label='Normal', alpha=0.5)
# plt.scatter(t, u + (np.random.rand(num_observations)-0.5)*sd_test_u,c='b', s=5, label='Uniform')
# plt.scatter(t, u + (np.random.standard_cauchy(num_observations))*sd_test_c,c='g', s=50, marker='x',label='Cauchy')
plt.plot(tt, um, c='k', lw=3, ls='-', label='True Signal')
if show_obs:
    plt.scatter(t, u + np.random.randn(num_observations)*sd_test_n, marker='x', s=50, c='k', label='{} Observations'.format(num_observations), alpha=1)

plt.legend(fontsize=18,loc='upper left')
plt.ylabel('Magnitude', fontsize=18)
plt.xlabel('Time (s)', fontsize=18)
# plt.ylim([-.25,1.25])
plt.xlim([0,5])
ftype = 'png'
if show_obs:
    summary_save_str = "saveimgs/HO/HO_%d_r-%d_obs-%de-3_sd-t_%d_%d_st%d_%d.%s"%(n, num_observations, 1000*sd_test_n, mintime, maxtime, start_time, end_time, ftype)
else:
    summary_save_str = "saveimgs/HO/HO_signal-%d_t_%d_%d_st%d_%d.%s"%(n, mintime, maxtime, start_time, end_time, ftype)

print("saving " + summary_save_str)
plt.savefig(summary_save_str)
plt.show()
```

---

# Define Prior Measure $P_\Lambda$

```python
input_dim = 2 # Specify input space dimension (n) -- this is a handle that the widgets depend on.
num_samples = int(1E4) # number of input samples (N)
s_set = sample.sample_set(size=(num_samples, input_dim))

if problem == 'ed':
    s_set.set_dist('beta', kwds={'a': 1, 'b': 1, 'loc': 0, 'scale': 1}, dim=0) # initial condition
    s_set.set_dist('beta', kwds={'a': 1, 'b': 1, 'loc': 0, 'scale': 1}, dim=1) # frequency
elif problem == 'ho':
    s_set.set_dist('beta', kwds={'a': 1, 'b': 1, 'loc': -0.25, 'scale': 0.5}, dim=0) # initial condition
    s_set.set_dist('beta', kwds={'a': 1, 'b': 1, 'loc': .5, 'scale': 1}, dim=1) # frequency

s_set.generate_samples()
lam = s_set.samples # create a pointer for ease of reference later with plotting.
```

### Visualize Prior

```python
# widgets.interactive(pltdata, data = fixed(lam), inds = fixed(None),
#                     N = widgets.IntSlider(value=500, min = 100, max=5000, step=100, continuous_update=False),
#                     eta_r = fixed(None), space=fixed(0.05), svd=widgets.Checkbox(value=False), color=widgets.Text(value="orange", continuous_update=False),
#                     view_dim_1 = widgets.IntSlider(value=0, min=0, max=input_dim-1, step=1, continuous_update=False),
#                     view_dim_2 = widgets.IntSlider(value=input_dim-1, min=0, max=input_dim-1, step=1, continuous_update=False))

```

---
# Compute Data Space $O(\Lambda) = \mathcal{D}$

Format: `(n_dims, n_samples)`

```python
u_true = model() # t as a subscript stands for "truth"
sd_true = sd_test_n
sd_guess = sd_true

# sd_true = 0.05
# sd_guess = 0.025

obs_data = u_true + np.random.randn(int(num_observations))*sd_true
# obs_data = u_true + np.random.standard_cauchy(int(num_observations))*sd_true
# obs_data = u_true + (np.random.rand(int(num_observations))-0.5)*sd_true

PtO_fun = MSE_generator(model, obs_data, sd_guess)
```

```python
p_set = sample.map_samples_and_create_problem(s_set, PtO_fun)
D = p_set.output.samples

# This is how we handle trying to infer the dimension based on what the map put out.
# You can delete this once you are certain your model is correctly defined.
try:
    output_dim = D.shape[1] # if your function was coded correctly, you should have an (n, d) data space.
except IndexError:
    print(Warning("Warning: Your map might be returning the wrong dimensional data."))
    try:
       output_dim = D.shape[0]
    except IndexError:
        print(Warning("Warning: Guessing it's 1-dimensional."))
        output_dim = 1
print('dimensions :  lambda = '+str(lam.shape)+'   D = '+str(D.shape) )
```

# Compute Push-Forward of the Prior $P_{O(\Lambda)}$
_ ... i.e. Characterize the Data Space_

```python
# Interactive Marginal Visualization
# p_set.compute_pushforward_dist(mirror=False, kwds={'kernel': 'gaussian', 'bandwidth': .25})
# p_set.compute_pushforward_dist(mirror=False, kwds={'kernel': 'gaussian', 'bandwidth': 5})
p_set.compute_pushforward_dist()
pf_dist = p_set.pushforward_dist
```

```python
# widgets.interactive(pltdata, data = fixed(pf_dist), inds = fixed(None),
#         N = widgets.IntSlider(value=10000, min = 100, max=10000, step=100, continuous_update=False),
#         eta_r = fixed(None), space=fixed(0.05), svd=fixed(False), color=widgets.Text(value="brown", continuous_update=False),
#         view_dim_1 = widgets.IntSlider(value=0, min=0, max=output_dim-1, step=1, continuous_update=False),
#         view_dim_2 = widgets.IntSlider(value=output_dim-1, min=0, max=output_dim-1, step=1, continuous_update=False))
```

# Define Observed Probability Measure $P_\mathcal{D}$

```python
p_set.set_observed_dist('gamma', {'a':num_observations/2, 'scale':2/num_observations}, dim=0)
#
# p_set.set_observed_dist('chi2', {'df':num_observations}, dim=0)
# p_set.set_observed_dist('normal', {'loc': 0, 'scale': 0.05}, dim=0)
# p_set.set_observed_dist('uniform', {'loc': 0, 'scale': 0.0025}, dim=0)

obs_dist = p_set.observed_dist # this is define a pointer for ease of reference.

# widgets.interactive(pltdata, data = fixed(obs_dist), inds = fixed(None),
#         N = widgets.IntSlider(value=500, min = 100, max=5000, step=100, continuous_update=False),
#         eta_r = fixed(None), space=fixed(0.05), svd=fixed(False), color=widgets.Text(value="wine", continuous_update=False),
#         view_dim_1 = widgets.IntSlider(value=0, min=0, max=output_dim-1, step=1, continuous_update=False),
#         view_dim_2 = widgets.IntSlider(value=output_dim-1, min=0, max=output_dim-1, step=1, continuous_update=False))
xx = np.linspace(-1,10,500).reshape(-1,1)
obs_dist = p_set.observed_dist # this is define a pointer for ease of reference.
zz = pf_dist.pdf(xx)
yy = obs_dist.pdf(xx)
```

```python
plt.plot(xx, zz, c='blue',label='Pushforward of Prior')
plt.plot(xx, yy,c='orange',label='Observed Density')
# plt.ylim([0,0.05])
plt.legend()
plt.show()
```

---

At this point we have performed the computations we need to. We have evaluated the input points through our map and performed a KDE on them. It would be useful at this point to save this object and/or its evaluation at every point in the data space for later re-use. Doing so here would be an appropriate place.

---

# Accept/Reject Sampling of Posterior

Since we have already used the samples in our prior to compute the pushforward density, we can re-use these with an accept/reject algorithm to get a set of samples generated from the posterior according to the solution of the stochastic inverse problem as outlined in the Consistent Bayes formulation.

```python
p_set.set_ratio()
eta_r = p_set.ratio
s_set.seed = 11
solve.problem(p_set)
```

```python
accept_inds = p_set.accept_inds
lam_accept = p_set.input.samples[accept_inds,:]
if problem == 'ho':
    lam_true = np.array([0, 1])
elif problem == 'ed':
    lam_true = np.array([0.5, 0.5])
num_accept = len(accept_inds)
print('Number accepted: %d = %2.2f%%'%(num_accept, 100*np.float(num_accept)/num_samples))
```

## Visualize Posterior Density
### (Visualize Accept/Reject Samples)

```python
# widgets.interactive(pltdata, data = fixed(lam), inds = fixed(accept_inds),
#         N = widgets.IntSlider(value=num_accept/2+1, min = 2, max=num_accept, step=1, continuous_update=False),
#         eta_r = fixed(None), space=fixed(0.05), svd=widgets.Checkbox(value=False), color=widgets.Text(value="orange", continuous_update=False),
#         view_dim_1 = widgets.IntSlider(value=0, min=0, max=input_dim-1, step=1, continuous_update=False),
#         view_dim_2 = widgets.IntSlider(value=input_dim-1, min=0, max=input_dim-1, step=1, continuous_update=False))

# You will visualize the accepted samples in a subset of size N of the input samples.
# This is mostly for faster plotting, but also so you can see the progression of accepted sampling in the algorithm.
plt.figure()
plt.title("Accept/Reject Results\n", fontsize=18)
plt.scatter(lam_accept[:,0], lam_accept[:,1], c='black', s=25)
plt.scatter(lam_true[0],lam_true[1], c='yellow', edgecolor='black', s=200, label='True Parameter')
# plt.scatter(lam[:,0], lam[:,1], c='gray', s=10)

plt.scatter(lam_accept[:,0].mean(), lam_accept[:,1].mean(),  c='red', edgecolor='black', s=100, label='Mean Estimate from\n{} Accepted Samples'.format(num_accept))
plt.legend(loc='upper center', fontsize=18)
if problem == 'ho':
    plt.xlim([-.25, .25])
    plt.ylim([0.5, 1.5])
if problem == 'ed':
    plt.xlim([0, 1])
    plt.ylim([0, 1])
plt.show()
```

# Alternative Visualization Approach - High Fidelity Mesh Plot (expensive)

Now that we have constructed the posterior distribution, we can evaluate it at whatever set of samples we want, so we plot it directly, rather than use Accept/Reject. This is done at the cost of a lot of model runs.

```python
nbins = 200
# mins = np.min(p_set.input.samples,axis=0)
# maxs = np.max(p_set.input.samples,axis=0)

if problem == 'ho':
    mins = np.array([-0.25, 0.5])
    maxs = np.array([0.5, 1.5])
elif problem == 'ed':
    mins = np.array([0.25, 0.25])
    maxs = np.array([0.75, 0.75])

X = ngrid(nbins, mins, maxs)
post_eval = p_set.evaluate_posterior(X)
# post_eval = p_set.compute_ratio(PtO_fun(X))*p_set.input.dist.pdf(X)
```

```python
# Make the plot
xi = X[:,0].reshape(nbins, nbins)
yi = X[:,1].reshape(nbins, nbins)
zi = post_eval.reshape(nbins, nbins)
plt.pcolormesh(xi, yi, zi)
plt.scatter(lam_true[0],lam_true[1], c='white', edgecolor='black', s=300, label='True Parameter')
plt.scatter(lam_accept[:,0].mean(), lam_accept[:,1].mean(),  c='red', edgecolor='black', s=200, label='Mean Accepted Sample')
# plt.scatter(X[inds_new,0].mean(), X[inds_new,1].mean(),  c='yellow', edgecolor='black', s=100, label='Mean Estimate') # can run after the cell below is executed
plt.legend(loc='upper right', fontsize=16)
if problem == 'ho':
    plt.xlim([-.25, 0.25])
    plt.ylim([0.5, 1.5])
if problem == 'ed':
    plt.xlim([0.25, 0.75])
    plt.ylim([0.25, 0.75])
plt.show()
# ?plt.pcolormesh
```

## Perform accept/reject with the samples we just used for the high-fidelity plot.
(because why not?)

```python
plot_new = True
XX = ngrid(nbins, mins, maxs)
# Perform accept/reject on that grid you see plotted above.
if plot_new:
    Y = PtO_fun(XX)
    inds_new = solve.perform_accept_reject(XX, p_set.compute_ratio(Y))
    print('Num accepted:', len(inds_new))
    new_accepted_in = XX[inds_new,:]
    new_accepted_out = Y[inds_new].reshape(-1,1)
    new_num_accept = len(inds_new)
```

```python
if plot_new:
    # Make the plot
    xi = XX[:,0].reshape(nbins, nbins)
    yi = XX[:,1].reshape(nbins, nbins)
    post_eval2 = p_set.evaluate_posterior(XX)
    zi = post_eval2.reshape(nbins, nbins)
    plt.pcolormesh(xi, yi, zi)
    plt.scatter(lam_true[0],lam_true[1], c='white', edgecolor='black', s=300, label='True Parameter')
    plt.scatter(lam_accept[:,0].mean(), lam_accept[:,1].mean(),  c='red', edgecolor='black', s=200, label='Mean Accepted Sample')
    plt.scatter(XX[inds_new,0].mean(), XX[inds_new,1].mean(),  c='yellow', edgecolor='black', s=100, label='Mean Estimate') # can run after the cell below is executed
    plt.legend(loc='lower right', fontsize=16)
    plt.show()
    print('Abs Error in Parameter and MAP:', [*np.abs( lam_true - np.mean(X[inds_new,:],axis=0) )] )
    print('Param. Test Stats for Gamma Dist:', [*np.abs( p_set.observed_dist.mean()-np.mean(Y[inds_new],axis=0))/p_set.observed_dist.mean() , *np.abs( p_set.observed_dist.std()-np.std(Y[inds_new],axis=0))/p_set.observed_dist.std()])

```

---
# Now what?

Well, we can...

## _Visualize the Quality of our SIP Solution by Comparing it to the Observed_
_We compare the push-forward of the posterior using accepted samples against the observed density_
_(SIP = Stochastic Inverse Problem)_
### Observed:

```python
xxi = np.linspace(0,4,100)
plt.plot(xxi, p_set.observed_dist.pdf(xxi), label='Observed', ls=':')
dy = distributions.skde(data=p_set.output.samples[accept_inds,:], **{'kernel': 'gaussian', 'bandwidth': 0.25})
plt.plot(xxi, dy.pdf(xxi), label='Pushforward of Posterior - {} AS Originally'.format(len(accept_inds)));
if plot_new:
    dyy = distributions.skde(data=new_accepted_out, **{'kernel': 'gaussian', 'bandwidth': 0.25})
    plt.plot(xxi, dyy.pdf(xxi), label='Pushforward of Posterior - {} AS from Refined Eyeball Run'.format(len(inds_new)))

plt.legend(loc='best')
plt.xlabel('Response')
plt.ylabel('Relative Likelihood')
plt.title('Pushforward of Posterior')
plt.show()
```

```python
show_prior = False # show the potential signals?
u_acc = m(lam_accept)

plt.scatter(t, obs_data, marker='x', c='k', s=100, label='Observed Data')

dd = np.mean(u_acc, axis=0)
plt.plot(tt, dd, c='k', alpha=1, lw=3, label='Mean Predicted Signal')

for i in range(u_acc.shape[0]):
    d = u_acc[i,:]
    if i==1:
        plt.plot(tt, d, c='b', alpha=0.5, lw=1, label='Accepted Samples')
    else:
        plt.plot(tt, d, c='b', alpha=0.025, lw=1)

if show_prior:
    for i in range(n):
        d = ui[i,:]
        if i == n-1:
            plt.plot(tt, d, c='k', alpha=5/n, label='Potential Signals')
        else:
            plt.plot(tt, d, c='k', alpha=5/n) # I

plt.ylabel('Height', fontsize=18)
plt.xlabel('Time (s)', fontsize=18)
plt.title('Recovered Signal based on Accepted Samples', fontsize=28)
plt.legend(fontsize=18,loc='upper left')
plt.xlim([0,5])
plt.savefig('recovered{}.png'.format(problem.upper()))
plt.legend()
plt.show()
```

```python
# widgets.interactive(pltdata, data = fixed(obs_dist), inds = fixed(None),
#         N = widgets.IntSlider(value=500, min = 100, max=5000, step=100, continuous_update=False),
#         eta_r = fixed(None), space=fixed(0.05), svd=fixed(False), color=widgets.Text(value="wine", continuous_update=False),
#         view_dim_1 = widgets.IntSlider(value=0, min=0, max=output_dim-1, step=1, continuous_update=False),
#         view_dim_2 = widgets.IntSlider(value=output_dim-1, min=0, max=output_dim-1, step=1, continuous_update=False))
```

## Pushforward of Posterior:

```python
# widgets.interactive(pltdata, data = fixed(D), inds = fixed(accept_inds),
#         N = widgets.IntSlider(value=num_accept/2, min = 2, max=num_accept-1, step=1, continuous_update=False),
#         eta_r = fixed(None), space=fixed(0.05), svd=fixed(False), color=widgets.Text(value="eggplant", continuous_update=False),
#         view_dim_1 = widgets.IntSlider(value=0, min=0, max=output_dim-1, step=1, continuous_update=False),
#         view_dim_2 = widgets.IntSlider(value=output_dim-1, min=0, max=output_dim-1, step=1, continuous_update=False))

```

## Verify our parametric test statistics.
Let's see if the pushforward of the posterior results in a sample mean and standard deviation that are close to the observed.

```python
# [np.mean(D[accept_inds,:]), np.std(D[accept_inds,:])]
```

```python
# [p_set.observed_dist.mean(), p_set.observed_dist.std()]
```

Now go back to the [Define Functions](#Define-Functions) and change the distribution on the prior or choose another example problem.

```python

```

```python

```


```python
# Imports
import torch
cuda = torch.cuda.is_available()
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
import sys
from sklearn.decomposition import PCA
from sklearn.manifold import TSNE
sys.path.append("../../semi-supervised")
```

# Auxiliary Deep Generative Model

The Auxiliary Deep Generative Model [[Maaløe, 2016]](https://arxiv.org/abs/1602.05473) posits a model that with an auxiliary latent variable $a$ that infers the variables $z$ and $y$. This helps in terms of semi-supervised learning by delegating causality to their respective variables. This model was state-of-the-art in semi-supervised until 2017, and is still very powerful with an MNIST accuracy of *99.4%* using just 10 labelled examples per class.

```python
from models import AuxiliaryDeepGenerativeModel

y_dim = 10
z_dim = 32
a_dim = 32
h_dim = [256, 128]

model = AuxiliaryDeepGenerativeModel([784, y_dim, z_dim, a_dim, h_dim])
model
```

    AuxiliaryDeepGenerativeModel(
      (encoder): Encoder(
        (hidden): ModuleList(
          (0): Linear(in_features=826, out_features=256, bias=True)
          (1): Linear(in_features=256, out_features=128, bias=True)
        )
        (sample): GaussianSample(
          (mu): Linear(in_features=128, out_features=32, bias=True)
          (log_var): Linear(in_features=128, out_features=32, bias=True)
        )
      )
      (decoder): Decoder(
        (hidden): ModuleList(
          (0): Linear(in_features=42, out_features=128, bias=True)
          (1): Linear(in_features=128, out_features=256, bias=True)
        )
        (reconstruction): Linear(in_features=256, out_features=784, bias=True)
        (output_activation): Sigmoid()
      )
      (classifier): Classifier(
        (dense): Linear(in_features=816, out_features=256, bias=True)
        (logits): Linear(in_features=256, out_features=10, bias=True)
      )
      (aux_encoder): Encoder(
        (hidden): ModuleList(
          (0): Linear(in_features=784, out_features=256, bias=True)
          (1): Linear(in_features=256, out_features=128, bias=True)
        )
        (sample): GaussianSample(
          (mu): Linear(in_features=128, out_features=32, bias=True)
          (log_var): Linear(in_features=128, out_features=32, bias=True)
        )
      )
      (aux_decoder): Encoder(
        (hidden): ModuleList(
          (0): Linear(in_features=826, out_features=128, bias=True)
          (1): Linear(in_features=128, out_features=256, bias=True)
        )
        (sample): GaussianSample(
          (mu): Linear(in_features=256, out_features=32, bias=True)
          (log_var): Linear(in_features=256, out_features=32, bias=True)
        )
      )
    )

## Training

The lower bound we derived in the notebook for the **deep generative model** is similar to the one for the ADGM. Here, we also need to integrate over a continuous auxiliary variable $a$.

For labelled data, the lower bound is given by.
\begin{align}
\log p(x,y) &= \log \int \int p(x, y, a, z) \ dz \ da\\
&\geq \mathbb{E}_{q(a,z|x,y)} \bigg [\log \frac{p(x,y,a,z)}{q(a,z|x,y)} \bigg ] = - \mathcal{L}(x,y)
\end{align}

Again when no label information is available we sum out all of the labels.

\begin{align}
\log p(x) &= \log \int \sum_{y} \int p(x, y, a, z) \ dz \ da\\
&\geq \mathbb{E}_{q(a,y,z|x)} \bigg [\log \frac{p(x,y,a,z)}{q(a,y,z |x)} \bigg ] = - \mathcal{U}(x)
\end{align}

Where we decompose the q-distribution into its constituent parts. $q(a, y, z|x) = q(z|a,y,x)q(y|a,x)q(a|x)$, which is also what can be seen in the figure.

The distribution over $a$ is similar to $z$ in the sense that it is also a diagonal Gaussian distribution. However by introducing the auxiliary variable we allow for $z$ to become arbitrarily complex - something we can also see when using normalizing flows.

```python
from datautils import get_mnist

# Only use 10 labelled examples per class
# The rest of the data is unlabelled.
labelled, unlabelled, validation = get_mnist(location="./", batch_size=64, labels_per_class=10)
alpha = 0.1 * (len(unlabelled) + len(labelled)) / len(labelled)

def binary_cross_entropy(r, x):
    return -torch.sum(x * torch.log(r + 1e-8) + (1 - x) * torch.log(1 - r + 1e-8), dim=-1)

optimizer = torch.optim.Adam(model.parameters(), lr=3e-4, betas=(0.9, 0.999))
```

    /home/nightmare/miniconda3/envs/intel/lib/python3.6/site-packages/torchvision/datasets/mnist.py:43: UserWarning: train_labels has been renamed targets
      warnings.warn("train_labels has been renamed targets")
    /home/nightmare/miniconda3/envs/intel/lib/python3.6/site-packages/torchvision/datasets/mnist.py:48: UserWarning: test_labels has been renamed targets
      warnings.warn("test_labels has been renamed targets")

```python
from itertools import cycle
from inference import SVI, DeterministicWarmup

# We will need to use warm-up in order to achieve good performance.
# Over 200 calls to SVI we change the autoencoder from
# deterministic to stochastic.
beta = DeterministicWarmup(n=200)

if cuda: model = model.cuda()
elbo = SVI(model, likelihood=binary_cross_entropy, beta=beta)
```

The library is conventially packed with the `SVI` method that does all of the work of calculating the lower bound for both labelled and unlabelled data depending on whether the label is given. It also manages to perform the enumeration of all the labels.

Remember that the labels have to be in a *one-hot encoded* format in order to work with SVI.

```python
from torch.autograd import Variable

for epoch in range(10):
    model.train()
    total_loss, accuracy = (0, 0)
    for (x, y), (u, _) in zip(cycle(labelled), unlabelled):
        # Wrap in variables
        x, y, u = Variable(x), Variable(y), Variable(u)
        if cuda:
            # They need to be on the same device and be synchronized.
            x, y = x.cuda(device=0), y.cuda(device=0)
            u = u.cuda(device=0)

        L = -elbo(x, y)
#         print(L)
        U = -elbo(u)

        # Add auxiliary classification loss q(y|x)
        logits = model.classify(x)

        # Regular cross entropy
        classication_loss = torch.sum(y * torch.log(logits + 1e-8), dim=1).mean()

        J_alpha = L - alpha * classication_loss + U

        J_alpha.backward()
        optimizer.step()
        optimizer.zero_grad()

        total_loss += J_alpha.item()
        accuracy += torch.mean((torch.max(logits, 1)[1].data == torch.max(y, 1)[1].data).float())
    print("Acc:", accuracy / len(unlabelled))
    if epoch % 1 == 0:
        model.eval()
        m = len(unlabelled)
        print("Epoch: {}".format(epoch))
        print("[Train]\t\t J_a: {:.2f}, accuracy: {:.2f}".format(total_loss / m, accuracy / m))

        total_loss, accuracy = (0, 0)
        for x, y in validation:
            x, y = Variable(x), Variable(y)

            if cuda:
                x, y = x.cuda(device=0), y.cuda(device=0)

            L = -elbo(x, y)
            U = -elbo(x)

            logits = model.classify(x)
            classication_loss = -torch.sum(y * torch.log(logits + 1e-8), dim=1).mean()

            J_alpha = L + alpha * classication_loss + U

            total_loss += J_alpha.data[0]

            _, pred_idx = torch.max(logits, 1)
            _, lab_idx = torch.max(y, 1)
            accuracy += torch.mean((torch.max(logits, 1)[1].data == torch.max(y, 1)[1].data).float())

        m = len(validation)
        print("[Validation]\t J_a: {:.2f}, accuracy: {:.2f}".format(total_loss / m, accuracy / m))
```

## Conditional generation

When the model is done training you can generate samples conditionally given some normal distributed noise $z$ and a label $y$.

*The model below has only trained for 10 iterations, so the perfomance is not representative*.

```python
from utils import onehot
model.eval()

z = Variable(torch.randn(16, 32))

# Generate a batch of 5s
y = Variable(onehot(10)(5).repeat(16, 1))

x_mu = model.sample(z, y)
```

```python
f, axarr = plt.subplots(1, 16, figsize=(18, 12))

samples = x_mu.data.view(-1, 28, 28).numpy()

for i, ax in enumerate(axarr.flat):
    ax.imshow(samples[i])
    ax.axis("off")
```

examples/notebooks/Auxiliary Deep Generative Model.ipynb


# Neuromatch Academy: Week 1, Day 3, Tutorial 4
# Model Fitting: Multiple linear regression and polynomial regression

**Content creators**: Pierre-Étienne Fiquet, Anqi Wu, Alex Hyafil with help from Byron Galbraith, Ella Batty

**Content reviewers**: Lina Teichmann, Saeed Salehi, Patrick Mineault, Michael Waskom

---
#Tutorial Objectives

This is Tutorial 4 of a series on fitting models to data. We start with simple linear regression, using least squares optimization (Tutorial 1) and Maximum Likelihood Estimation (Tutorial 2). We will use bootstrapping to build confidence intervals around the inferred linear model parameters (Tutorial 3). We'll finish our exploration of regression models by generalizing to multiple linear regression and polynomial regression (Tutorial 4). We end by learning how to choose between these various models. We discuss the bias-variance trade-off (Tutorial 5) and Cross Validation for model selection (Tutorial 6).

In this tutorial, we will generalize the regression model to incorporate multiple features.
- Learn how to structure inputs for regression using the 'Design Matrix'
- Generalize the MSE for multiple features using the ordinary least squares estimator
- Visualize data and model fit in multiple dimensions
- Fit polynomial regression models of different complexity
- Plot and evaluate the polynomial regression fits

```python
#@title Video 1: Multiple Linear Regression and Polynomial Regression
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="d4nfTki6Ejc", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
```

    Video available at https://youtube.com/watch?v=d4nfTki6Ejc

---
# Setup

```python
import numpy as np
import matplotlib.pyplot as plt
```

```python
#@title Figure Settings
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")
```

```python
#@title Helper Functions

def plot_fitted_polynomials(x, y, theta_hat):
  """ Plot polynomials of different orders

  Args:
    x (ndarray): input vector of shape (n_samples)
    y (ndarray): vector of measurements of shape (n_samples)
    theta_hat (dict): polynomial regression weights for different orders
  """

  x_grid = np.linspace(x.min() - .5, x.max() + .5)

  plt.figure()

  for order in range(0, max_order + 1):
    X_design = make_design_matrix(x_grid, order)
    plt.plot(x_grid, X_design @ theta_hat[order]);

  plt.ylabel('y')
  plt.xlabel('x')
  plt.plot(x, y, 'C0.');
  plt.legend([f'order {o}' for o in range(max_order + 1)], loc=1)
  plt.title('polynomial fits')
  plt.show()

```

```python
# Central Limit Theorem
n_iter = int(1e4)
n_sample = int(1e3)
bino_p = 0.5
sum_array = np.zeros(n_iter)

for idx in range(n_iter):
  sum_array[idx] = np.random.randint(0, high=2, size=n_sample).sum()

h = plt.hist(sum_array, bins=30)
```

---
# Section 1: Multiple Linear Regression

Now that we have considered the univariate case and how to produce confidence intervals for our estimator, we turn to the general linear regression case, where we can have more than one regressor, or feature, in our input.

Recall that our original univariate linear model was given as

\begin{align}
y = \theta x + \epsilon
\end{align}

where $\theta$ is the slope and $\epsilon$ some noise. We can easily extend this to the multivariate scenario by adding another parameter for each additional feature

\begin{align}
y = \theta_0 + \theta_1 x_1 + \theta_1 x_2 + ... +\theta_d x_d + \epsilon
\end{align}

where $\theta_0$ is the intercept and $d$ is the number of features (it is also the dimensionality of our input).

We can condense this succinctly using vector notation for a single data point

\begin{align}
y_i = \boldsymbol{\theta}^{\top}\mathbf{x}_i + \epsilon
\end{align}

and fully in matrix form

\begin{align}
\mathbf{y} = \mathbf{X}\boldsymbol{\theta} + \mathbf{\epsilon}
\end{align}

where $\mathbf{y}$ is a vector of measurements, $\mathbf{X}$ is a matrix containing the feature values (columns) for each input sample (rows), and $\boldsymbol{\theta}$ is our parameter vector.

This matrix $\mathbf{X}$ is often referred to as the "[design matrix](https://en.wikipedia.org/wiki/Design_matrix)".

For this tutorial we will focus on the two-dimensional case ($d=2$), which allows us to easily visualize our results. As an example, think of a situation where a scientist records the spiking response of a retinal ganglion cell to patterns of light signals that vary in contrast and in orientation. Then contrast and orientation values can be used as features / regressors to predict the cells response.

In this case our model can be writen as:

\begin{align}
y = \theta_0 + \theta_1 x_1 + \theta_2 x_2 + \epsilon
\end{align}

or in matrix form where

\begin{align}
\mathbf{X} =
\begin{bmatrix}
1 & x_{1,1} & x_{1,2} \\
1 & x_{2,1} & x_{2,2} \\
\vdots & \vdots & \vdots \\
1 & x_{n,1} & x_{n,2}
\end{bmatrix},
\boldsymbol{\theta} =
\begin{bmatrix}
\theta_0 \\
\theta_1 \\
\theta_2 \\
\end{bmatrix}
\end{align}

For our actual exploration dataset we shall set $\boldsymbol{\theta}=[0, -2, -3]$ and draw $N=40$ noisy samples from $x \in [-2,2)$. Note that setting the value of $\theta_0 = 0$ effectively ignores the offset term.

```python
#@title
#@markdown Execute this cell to simulate some data

# Set random seed for reproducibility
np.random.seed(1234)

# Set parameters
theta = [0, -2, -3]
n_samples = 40

# Draw x and calculate y
n_regressors = len(theta)
x0 = np.ones((n_samples, 1))
x1 = np.random.uniform(-2, 2, (n_samples, 1))
x2 = np.random.uniform(-2, 2, (n_samples, 1))
X = np.hstack((x0, x1, x2))
noise = np.random.randn(n_samples)
y = X @ theta + noise

ax = plt.subplot(projection='3d')
ax.plot(X[:,1], X[:,2], y, '.')

ax.set(
    xlabel='$x_1$',
    ylabel='$x_2$',
    zlabel='y'
)
plt.tight_layout()

```

Now that we have our dataset, we want to find an optimal vector of paramters $\boldsymbol{\hat\theta}$. Recall our analytic solution to minimizing MSE for a single regressor:

\begin{align}
\hat\theta = \frac{\sum_{i=1}^N x_i y_i}{\sum_{i=1}^N x_i^2}.
\end{align}

The same holds true for the multiple regressor case, only now expressed in matrix form

\begin{align}
\boldsymbol{\hat\theta} = (\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{X}^\top\mathbf{y}.
\end{align}

This is called the [ordinary least squares](https://en.wikipedia.org/wiki/Ordinary_least_squares) (OLS) estimator.

### Exercise 1: Ordinary Least Squares Estimator

In this exercise you will implement the OLS approach to estimating $\boldsymbol{\hat\theta}$ from the design matrix $\mathbf{X}$ and measurement vector $\mathbf{y}$. You can use the `@` symbol for matrix multiplication, `.T` for transpose, and `np.linalg.inv` for matrix inversion.

```python
def ordinary_least_squares(X, y):
  """Ordinary least squares estimator for linear regression.

  Args:
    x (ndarray): design matrix of shape (n_samples, n_regressors)
    y (ndarray): vector of measurements of shape (n_samples)

  Returns:
    ndarray: estimated parameter values of shape (n_regressors)
  """

  # Compute theta_hat using OLS
  theta_hat = np.linalg.inv(X.T @ X) @ X.T @ y

  return theta_hat

# Uncomment below to test your function
theta_hat = ordinary_least_squares(X, y)
print(theta_hat)
```

    [ 0.13861386 -2.09395731 -3.16370742]

After filling in this function, you should see that $\hat{\theta}$ = [ 0.13861386, -2.09395731, -3.16370742]

Now that we have our $\mathbf{\hat\theta}$, we can obtain $\mathbf{\hat y}$ and thus our mean squared error.

```python
# Compute predicted data
theta_hat = ordinary_least_squares(X, y)
y_hat = X @ theta_hat

# Compute MSE
print(f"MSE = {np.mean((y - y_hat)**2):.2f}")
```

    MSE = 0.91

Finally, the following code will plot a geometric visualization of the data points (blue) and fitted plane. You can see that the residuals (green bars) are orthogonal to the plane.

Extra: how would you check that the residuals are orthogonal to the fitted plane? (this is sometimes called the orthogonality principle, see [least squares notes](https://www.cns.nyu.edu/~eero/NOTES/leastSquares.pdf))

```python
#@title
#@markdown Execute this cell to visualize data and predicted plane

theta_hat = ordinary_least_squares(X, y)
xx, yy = np.mgrid[-2:2:50j, -2:2:50j]
y_hat_grid = np.array([xx.flatten(), yy.flatten()]).T @ theta_hat[1:]
y_hat_grid = y_hat_grid.reshape((50, 50))

ax = plt.subplot(projection='3d')
ax.plot(X[:, 1], X[:, 2], y, '.')
ax.plot_surface(xx, yy, y_hat_grid, linewidth=0, alpha=0.5, color='C1',
                cmap=plt.get_cmap('coolwarm'))

for i in range(len(X)):
  ax.plot((X[i, 1], X[i, 1]),
          (X[i, 2], X[i, 2]),
          (y[i], y_hat[i]),
          'g-', alpha=.5)

ax.set(
    xlabel='$x_1$',
    ylabel='$x_2$',
    zlabel='y'
)
plt.tight_layout()

```

---
# Section 2:  Polynomial Regression

So far today, you learned how to predict outputs from inputs by fitting a linear regression model. We can now model all sort of relationships, including in neuroscience!

One potential problem with this approach is the simplicity of the model. Linear regression, as the name implies, can only capture a linear relationship between the inputs and outputs. Put another way, the predicted outputs are only a weighted sum of the inputs. What if there are more complicated computations happening? Luckily, many more complex models exist (and you will encounter many more over the next 3 weeks). One model that is still very simple to fit and understand, but captures more complex relationships, is **polynomial regression**, an extension of linear regression.

Since polynomial regression is an extension of linear regression, everything you learned so far will come in handy now! The goal is the same: we want to predict the dependent variable $y_{n}$ given the input values $x_{n}$. The key change is the type of relationship between inputs and outputs that the model can capture.

Linear regression models predict the outputs as a weighted sum of the inputs:

$$y_{n}= \theta_0 + \theta x_{n} + \epsilon_{n}$$

With polynomial regression, we model the outputs as a polynomial equation based on the inputs. For example, we can model the outputs as:

$$y_{n}= \theta_0 + \theta_1 x_{n} + \theta_2 x_{n}^2 + \theta_3 x_{n}^3 + \epsilon_{n}$$

We can change how complex a polynomial is fit by changing the order of the polynomial. The order of a polynomial refers to the highest power in the polynomial. The equation above is a third order polynomial because the highest value x is raised to is 3. We could add another term ($+ \theta_4 x_{n}^4$) to model an order 4 polynomial and so on.

First, we will simulate some data to practice fitting polynomial regression models. We will generate random inputs $x$ and then compute y according to $y = x^2 - x - 2 $, with some extra noise both in the input and the output to make the model fitting exercise closer to a real life situation.

```python
#@title
#@markdown Execute this cell to simulate some data

# setting a fixed seed to our random number generator ensures we will always
# get the same psuedorandom number sequence
np.random.seed(121)
n_samples = 30
x = np.random.uniform(-2, 2.5, n_samples)  # inputs uniformly sampled from [-2, 2.5)
y =  x**2 - x - 2   # computing the outputs

output_noise = 1/8 * np.random.randn(n_samples)
y += output_noise  # adding some output noise

input_noise = 1/2 * np.random.randn(n_samples)
x += input_noise  # adding some input noise

fig, ax = plt.subplots()
ax.scatter(x, y)  # produces a scatter plot
ax.set(xlabel='x', ylabel='y');
```

## Section 2.1: Design matrix for polynomial regression

Now we have the basic idea of polynomial regression and some noisy data, let's begin! The key difference between fitting a linear regression model and a polynomial regression model lies in how we structure the input variables.

For linear regression, we used $X = x$ as the input data. To add a constant bias (a y-intercept in a 2-D plot), we use $X = \big[ \boldsymbol 1, x \big]$, where $\boldsymbol 1$ is a column of ones.  When fitting, we learn a weight for each column of this matrix. So we learn a weight that multiples with column 1 - in this case that column is all ones so we gain the bias parameter ($+ \theta_0$). We also learn a weight for every column, or every feature of x, as we learned in Section 1.

This matrix $X$ that we use for our inputs is known as a **design matrix**. We want to create our design matrix so we learn weights for $x^2, x^3,$ etc. Thus, we want to build our design matrix $X$ for polynomial regression of order $k$ as:

$$X = \big[ \boldsymbol 1 , x^1, x^2 , \ldots , x^k \big],$$

where $\boldsymbol{1}$ is the vector the same length as $x$ consisting of of all ones, and $x^p$ is the vector or matrix $x$ with all elements raised to the power $p$. Note that $\boldsymbol{1} = x^0$ and $x^1 = x$

### Exercise 2: Structure design matrix

Create a function (`make_design_matrix`) that structures the design matrix given the input data and the order of the polynomial you wish to fit. We will print part of this design matrix for our data and order 5.

```python
def make_design_matrix(x, order):
  """Create the design matrix of inputs for use in polynomial regression

  Args:
    x (ndarray): input vector of shape (n_samples)
    order (scalar): polynomial regression order

  Returns:
    ndarray: design matrix for polynomial regression of shape (samples, order+1)
  """

  design_matrix = np.ones((x.shape[0], order + 1))

  # Loop through rest of degrees and stack columns (hint: np.hstack)
  for degree in range(1, order + 1):
      design_matrix[:, degree] = (x ** degree)

  return design_matrix

# Uncomment to test your function
order = 5
X_design = make_design_matrix(x, order)
print(X_design[0:2, 0:2])

```

    [[ 1.         -1.51194917]
     [ 1.         -0.35259945]]

```python
# to_remove solution
def make_design_matrix(x, order):
  """Create the design matrix of inputs for use in polynomial regression

  Args:
    x (ndarray): input vector of shape (samples,)
    order (scalar): polynomial regression order

  Returns:
    ndarray: design matrix for polynomial regression of shape (samples, order+1)
  """

  # Broadcast to shape (n x 1) so dimensions work
  if x.ndim == 1:
    x = x[:, None]

  #if x has more than one feature, we don't want multiple columns of ones so we assign
  # x^0 here
  design_matrix = np.ones((x.shape[0], 1))

  # Loop through rest of degrees and stack columns
  for degree in range(1, order + 1):
      design_matrix = np.hstack((design_matrix, x**degree))

  return design_matrix

order = 5
X_design = make_design_matrix(x, order)
print(X_design[0:2, 0:2])
```

    [[ 1.         -1.51194917]
     [ 1.         -0.35259945]]

You should see that the printed section of this design matrix is `[[ 1.         -1.51194917]
 [ 1.         -0.35259945]]`

## Section 2.2: Fitting polynomial regression models

Now that we have the inputs structured correctly in our design matrix, fitting a polynomial regression is the same as fitting a linear regression model! All of the polynomial structure we need to learn is contained in how the inputs are structured in the design matrix. We can use the same least squares solution we computed in previous exercises.

### Exercise 3: Fitting polynomial regression models with different orders

Here, we will fit polynomial regression models to find the regression coefficients ($\theta_0, \theta_1, \theta_2,$ ...) by solving the least squares problem. Create a function `solve_poly_reg` that loops over different order polynomials (up to `max_order`), fits that model, and saves out the weights for each. You may invoke the `ordinary_least_squares` function.

We will then qualitatively inspect the quality of our fits for each order by plotting the fitted polynomials on top of the data. In order to see smooth curves, we evaluate the fitted polynomials on a grid of $x$ values (ranging between the largest and smallest of the inputs present in the dataset).

```python
def solve_poly_reg(x, y, max_order):
  """Fit a polynomial regression model for each order 0 through max_order.

  Args:
    x (ndarray): input vector of shape (n_samples)
    y (ndarray): vector of measurements of shape (n_samples)
    max_order (scalar): max order for polynomial fits

  Returns:
    dict: fitted weights for each polynomial model (dict key is order)
  """

  # Create a dictionary with polynomial order as keys, and np array of theta_hat
  # (weights) as the values
  theta_hats = {}

  # Loop over polynomial orders from 0 through max_order
  for order in range(max_order + 1):

    # Create design matrix
    X_design = make_design_matrix(x, order)

    # Fit polynomial model
    this_theta = ordinary_least_squares(X_design, y)

    theta_hats[order] = this_theta

  return theta_hats

# Uncomment to test your function
max_order = 5
theta_hats = solve_poly_reg(x, y, max_order)
plot_fitted_polynomials(x, y, theta_hats)

```

## Section 2.4: Evaluating fit quality

As with linear regression, we can compute mean squared error (MSE) to get a sense of how well the model fits the data.

We compute MSE as:

$$ MSE = \frac 1 N ||y - \hat y||^2 = \sum_{i=1}^N (y_i - \hat y_i)^2 $$

where the predicted values for each model are given by $ \hat y = X \hat \theta$.

*Which model (i.e. which polynomial order) do you think will have the best MSE?*

### Exercise 4: Compute MSE and compare models

We will compare the MSE for different polynomial orders with a bar plot.

```python
mse = np.zeros((max_order + 1))
for order in range(0, max_order + 1):

  X_design = make_design_matrix(x, order)

  ## Uncomment below and fill in with your code

  # Get prediction for the polynomial regression model of this order
  y_hat = X_design @ ordinary_least_squares(X_design, y)

  # Compute the residuals
  residuals = y - y_hat

  # Compute the MSE
  mse[order] = np.mean(residuals ** 2)

fig, ax = plt.subplots()

# Uncomment once above exercise is complete
ax.bar(range(max_order + 1), mse);
ax.set(title='Comparing Polynomial Fits', xlabel='Polynomial order', ylabel='MSE')
```

```python
# to_remove solution
mse = np.zeros((max_order + 1))
for order in range(0, max_order + 1):

  X_design = make_design_matrix(x, order)

  # Get prediction for the polynomial regression model of this order
  y_hat = X_design @ theta_hats[order]

  # Compute the residuals
  residuals = y - y_hat

  # Compute the MSE
  mse[order] = np.mean(residuals ** 2)

with plt.xkcd():
  fig, ax = plt.subplots()

  ax.bar(range(max_order + 1), mse);

  ax.set(title='Comparing Polynomial Fits', xlabel='Polynomial order', ylabel='MSE')
```

---
# Summary

* Linear regression generalizes naturally to multiple dimensions
* Linear algebra affords us the mathematical tools to reason and solve such problems beyond the two dimensional case

* To change from a linear regression model to a polynomial regression model, we only have to change how the input data is structured

* We can choose the complexity of the model by changing the order of the polynomial model fit

* Higher order polynomial models tend to have lower MSE on the data they're fit with

**Note**: In practice, multidimensional least squares problems can be solved very efficiently (thanks to numerical routines such as LAPACK).

**Suggested readings**

[Introduction to Applied Linear Algebra – Vectors, Matrices, and Least Squares](http://vmls-book.stanford.edu/)
Stephen Boyd and Lieven Vandenberghe


# PHY321: Two-body problems, gravitational forces, scattering and begin Lagrangian formalism

**[Morten Hjorth-Jensen](http://mhjgit.github.io/info/doc/web/)**, Department of Physics and Astronomy and Facility for Rare Ion Beams (FRIB), Michigan State University, USA and Department of Physics, University of Oslo, Norway

Date: **Mar 29, 2021**

Copyright 1999-2021, [Morten Hjorth-Jensen](http://mhjgit.github.io/info/doc/web/). Released under CC Attribution-NonCommercial 4.0 license

## Aims and Overarching Motivation

### Monday

1. Physical interpretation of various orbit types and summary gravitational forces, examples on whiteboard and handwritten notes

2. Start discussion two-body scattering

**Reading suggestion**: Taylor chapter 8 and sections 14.1-14.2 and Lecture notes

### Wednesday

1. Two-body scattering

**Reading suggestion**: Taylor and sections 14.3-14.6

### Friday

1. Lagrangian formalism

**Reading suggestion**: Taylor Sections 6.1-6.2

### Code example with gravitional force

The code example here is meant to illustrate how we can make a plot of the final orbit. We solve the equations in polar coordinates (the example here uses the minimum of the potential as initial value) and then we transform back to cartesian coordinates and plot $x$ versus $y$. We see that we get a perfect circle when we place ourselves at the minimum of the potential energy, as expected.

```python
%matplotlib inline

# Common imports
import numpy as np
import pandas as pd
from math import *
import matplotlib.pyplot as plt
# Simple Gravitational Force   -alpha/r

DeltaT = 0.01
#set up arrays
tfinal = 8.0
n = ceil(tfinal/DeltaT)
# set up arrays for t, v and r
t = np.zeros(n)
v = np.zeros(n)
r = np.zeros(n)
phi = np.zeros(n)
x = np.zeros(n)
y = np.zeros(n)
# Constants of the model, setting all variables to one for simplicity
alpha = 1.0
AngMom = 1.0  #  The angular momentum
m = 1.0  # scale mass to one
c1 = AngMom*AngMom/(m*m)
c2 = AngMom*AngMom/m
rmin = (AngMom*AngMom/m/alpha)
# Initial conditions, place yourself at the potential min
r0 = rmin
v0 = 0.0  # starts at rest
r[0] = r0
v[0] = v0
phi[0] = 0.0
# Start integrating using the Velocity-Verlet  method
for i in range(n-1):
    # Set up acceleration
    a = -alpha/(r[i]**2)+c1/(r[i]**3)
    # update velocity, time and position using the Velocity-Verlet method
    r[i+1] = r[i] + DeltaT*v[i]+0.5*(DeltaT**2)*a
    anew = -alpha/(r[i+1]**2)+c1/(r[i+1]**3)
    v[i+1] = v[i] + 0.5*DeltaT*(a+anew)
    t[i+1] = t[i] + DeltaT
    phi[i+1] = t[i+1]*c2/(r0**2)
# Find cartesian coordinates for easy plot
x = r*np.cos(phi)
y = r*np.sin(phi)
fig, ax = plt.subplots(3,1)
ax[0].set_xlabel('time')
ax[0].set_ylabel('radius')
ax[0].plot(t,r)
ax[1].set_xlabel('time')
ax[1].set_ylabel('Angle $\cos{\phi}$')
ax[1].plot(t,np.cos(phi))
ax[2].set_ylabel('y')
ax[2].set_xlabel('x')
ax[2].plot(x,y)

#save_fig("Phasespace")
plt.show()
```

## Changing initial conditions

Try to change the initial value for $r$ and see what kind of orbits you get.
In order to test different energies, it can be useful to look at the plot of the effective potential discussed above.

However, for orbits different from a circle the above code would need modifications in order to allow us to display say an ellipse. For the latter, it is much easier to run our code in cartesian coordinates, as done here. In this code we test also energy conservation and see that it is conserved to numerical precision. The code here is a simple extension of the code we developed for homework 4.

```python
%matplotlib inline

# Common imports
import numpy as np
import pandas as pd
from math import *
import matplotlib.pyplot as plt

DeltaT = 0.01
#set up arrays
tfinal = 10.0
n = ceil(tfinal/DeltaT)
# set up arrays
t = np.zeros(n)
v = np.zeros((n,2))
r = np.zeros((n,2))
E = np.zeros(n)
# Constants of the model
m = 1.0   # mass, you can change these
alpha = 1.0
# Initial conditions as compact 2-dimensional arrays
x0 = 0.5; y0= 0.
r0 = np.array([x0,y0])
v0 = np.array([0.0,1.0])
r[0] = r0
v[0] = v0
rabs = sqrt(sum(r[0]*r[0]))
E[0] = 0.5*m*(v[0,0]**2+v[0,1]**2)-alpha/rabs
# Start integrating using the Velocity-Verlet  method
for i in range(n-1):
    # Set up the acceleration
    rabs = sqrt(sum(r[i]*r[i]))
    a =  -alpha*r[i]/(rabs**3)
    # update velocity, time and position using the Velocity-Verlet method
    r[i+1] = r[i] + DeltaT*v[i]+0.5*(DeltaT**2)*a
    rabs = sqrt(sum(r[i+1]*r[i+1]))
    anew = -alpha*r[i+1]/(rabs**3)
    v[i+1] = v[i] + 0.5*DeltaT*(a+anew)
    E[i+1] = 0.5*m*(v[i+1,0]**2+v[i+1,1]**2)-alpha/rabs
    t[i+1] = t[i] + DeltaT
# Plot position as function of time
fig, ax = plt.subplots(3,1)
ax[0].set_ylabel('y')
ax[0].set_xlabel('x')
ax[0].plot(r[:,0],r[:,1])
ax[1].set_xlabel('time')
ax[1].set_ylabel('y position')
ax[1].plot(t,r[:,0])
ax[2].set_xlabel('time')
ax[2].set_ylabel('y position')
ax[2].plot(t,r[:,1])

fig.tight_layout()
#save_fig("2DimGravity")
plt.show()
print(E)
```

## Scattering and Cross Sections

Scattering experiments don't measure entire trajectories. For elastic
collisions, they measure the distribution of final scattering angles
at best. Most experiments use targets thin enough so that the number
of scatterings is typically zero or one. The cross section, $\sigma$,
describes the cross-sectional area for particles to scatter with an
individual target atom or nucleus. Cross section measurements form the
basis for MANY fields of physics. BThe cross section, and the
differential cross section, encapsulates everything measurable for a
collision where all that is measured is the final state, e.g. the
outgoing particle had momentum $\boldsymbol{p}_f$. y studying cross sections,
one can infer information about the potential interaction between the
two particles. Inferring, or constraining, the potential from the
cross section is a classic {\it inverse} problem. Collisions are
either elastic or inelastic. Elastic collisions are those for which
the two bodies are in the same internal state before and after the
collision. If the collision excites one of the participants into a
higher state, or transforms the particles into different species, or
creates additional particles, the collision is inelastic. Here, we
consider only elastic collisions.

## Scattering: Coulomb forces

For Coulomb forces, the cross section is infinite because the range of
the Coulomb force is infinite, but for interactions such as the strong
interaction in nuclear or particle physics, there is no long-range
force and cross-sections are finite. Even for Coulomb forces, the part
of the cross section that corresponds to a specific scattering angle,
$d\sigma/d\Omega$, which is a function of the scattering angle
$\theta_s$ is still finite.

If a particle travels through a thin target, the chance the particle
scatters is $P_{\rm scatt}=\sigma dN/dA$, where $dN/dA$ is the number
of scattering centers per area the particle encounters. If the density
of the target is $\rho$ particles per volume, and if the thickness of
the target is $t$, the areal density (number of target scatterers per
area) is $dN/dA=\rho t$. Because one wishes to quantify the collisions
independently of the target, experimentalists measure scattering
probabilities, then divide by the areal density to obtain
cross-sections,

$$
\begin{eqnarray}
\sigma=\frac{P_{\rm scatt}}{dN/dA}.
\end{eqnarray}
$$

## Scattering, more details

Instead of merely stating that a particle collided, one can measure
the probability the particle scattered by a given angle. The
scattering angle $\theta_s$ is defined so that at zero the particle is
unscattered and at $\theta_s=\pi$ the particle is scattered directly
backward. Scattering angles are often described in the center-of-mass
frame, but that is a detail we will neglect for this first discussion,
where we will consider the scattering of particles moving classically
under the influence of fixed potentials $U(\boldsymbol{r})$. Because the
distribution of scattering angles can be measured, one expresses the
differential cross section,

$$
\begin{equation}
\frac{d^2\sigma}{d\cos\theta_s~d\phi}.
\label{_auto1} \tag{1}
\end{equation}
$$

Usually, the literature expresses differential cross sections as

$$
\begin{equation}
d\sigma/d\Omega=\frac{d\sigma}{d\cos\theta d\phi}=\frac{1}{2\pi}\frac{d\sigma}{d\cos\theta},
\label{_auto2} \tag{2}
\end{equation}
$$

where the last equivalency is true when the scattering does not depend
on the azimuthal angle $\phi$, as is the case for spherically
symmetric potentials.

The differential solid angle $d\Omega$ can be thought of as the area
subtended by a measurement, $dA_d$, divided by $r^2$, where $r$ is the
distance to the detector,

$$
\begin{eqnarray}
dA_d=r^2 d\Omega.
\end{eqnarray}
$$

With this definition $d\sigma/d\Omega$ is independent of the distance
from which one places the detector, or the size of the detector (as
long as it is small).

## Differential scattering cross sections

Differential scattering cross sections are calculated by assuming a
random distribution of impact parameters $b$. These represent the
distance in the $xy$ plane for particles moving in the $z$ direction
relative to the scattering center. An impact parameter $b=0$ refers to
being aimed directly at the target's center. The impact parameter
describes the transverse distance from the $z=0$ axis for the
trajectory when it is still far away from the scattering center and
has not yet passed it. The differential cross section can be expressed
in terms of the impact parameter,

$$
\begin{equation}
d\sigma=2\pi bdb,
\label{_auto3} \tag{3}
\end{equation}
$$

which is the area of a thin ring of radius $b$ and thickness $db$. In
classical physics, one can calculate the trajectory given the incoming
kinetic energy $E$ and the impact parameter if one knows the mass and
potential.

## More on Differential  Cross Sections

From the trajectory, one then finds the scattering angle
$\theta_s(b)$. The differential cross section is then

$$
\begin{equation}
\frac{d\sigma}{d\Omega}=\frac{1}{2\pi}\frac{d\sigma}{d\cos\theta_s}=b\frac{db}{d\cos\theta_s}=\frac{b}{(d/db)\cos\theta_s(b)}.
\label{_auto4} \tag{4}
\end{equation}
$$

Typically, one would calculate $\cos\theta_s$ and $(d/db)\cos\theta_s$
as functions of $b$. This is sufficient to plot the differential cross
section as a function of $\theta_s$.

The total cross section is

$$
\begin{equation}
\sigma_{\rm tot}=\int d\Omega\frac{d\sigma}{d\Omega}=2\pi\int d\cos\theta_s~\frac{d\sigma}{d\Omega}.
\label{_auto5} \tag{5}
\end{equation}
$$

Even if the total cross section is infinite, e.g. Coulomb forces, one
can still have a finite differential cross section as we will see
later on.

## Rutherford Scattering

This refers to the calculation of $d\sigma/d\Omega$ due to an inverse
square force, $F_{12}=\pm\alpha/r^2$ for repulsive/attractive
interaction. Rutherford compared the scattering of $\alpha$ particles
($^4$He nuclei) off of a nucleus and found the scattering angle at
which the formula began to fail. This corresponded to the impact
parameter for which the trajectories would strike the nucleus. This
provided the first measure of the size of the atomic nucleus. At the
time, the distribution of the positive charge (the protons) was
considered to be just as spread out amongst the atomic volume as the
electrons. After Rutherford's experiment, it was clear that the radius
of the nucleus tended to be roughly 4 orders of magnitude smaller than
that of the atom, which is less than the size of a football relative
to Spartan Stadium.

## Rutherford Scattering, more details

In order to calculate differential cross section, we must find how the
impact parameter is related to the scattering angle. This requires
analysis of the trajectory. We consider our previous expression for
the trajectory where we derived the elliptic form for the trajectory,
For that case we considered an attractive
force with the particle's energy being negative, i.e. it was
bound. However, the same form will work for positive energy, and
repulsive forces can be considered by simple flipping the sign of
$\alpha$. For positive energies, the trajectories will be hyperbolas,
rather than ellipses, with the asymptotes of the trajectories
representing the directions of the incoming and outgoing
tracks.

## Rutherford Scattering, final trajectories

We have

$$
\begin{equation}\label{eq:ruthtraj} \tag{6}
r=\frac{1}{\frac{m\alpha}{L^2}+A\cos\theta}.
\end{equation}
$$

Once $A$ is large enough, which will happen when the energy is
positive, the denominator will become negative for a range of
$\theta$. This is because the scattered particle will never reach
certain angles. The asymptotic angles $\theta'$ are those for which
the denominator goes to zero,

$$
\begin{equation}
\cos\theta'=-\frac{m\alpha}{AL^2}.
\label{_auto6} \tag{7}
\end{equation}
$$

## Rutherford Scattering, Closest Approach

The trajectory's point of closest approach is at $\theta=0$ and the
two angles $\theta'$, which have this value of $\cos\theta'$, are the
angles of the incoming and outgoing particles. From
Fig (**to come**), one can see that the scattering angle
$\theta_s$ is given by,

$$
\begin{eqnarray}
\label{eq:sthetover2} \tag{8}
2\theta'-\pi&=&\theta_s,~~~\theta'=\frac{\pi}{2}+\frac{\theta_s}{2},\\
\nonumber
\sin(\theta_s/2)&=&-\cos\theta'\\
\nonumber
&=&\frac{m\alpha}{AL^2}.
\end{eqnarray}
$$

Now that we have $\theta_s$ in terms of $m,\alpha,L$ and $A$, we wish
to re-express $L$ and $A$ in terms of the impact parameter $b$ and the
energy $E$. This will set us up to calculate the differential cross
section, which requires knowing $db/d\theta_s$. It is easy to write
the angular momentum as

$$
\begin{equation}
L^2=p_0^2b^2=2mEb^2.
\label{_auto7} \tag{9}
\end{equation}
$$

## Rutherford Scattering, getting there

Finding $A$ is more complicated. To accomplish this we realize that
the point of closest approach occurs at $\theta=0$, so from
Eq. ([6](#eq:ruthtraj))

$$
\begin{eqnarray}
\label{eq:rminofA} \tag{10}
\frac{1}{r_{\rm min}}&=&\frac{m\alpha}{L^2}+A,\\
\nonumber
A&=&\frac{1}{r_{\rm min}}-\frac{m\alpha}{L^2}.
\end{eqnarray}
$$

Next, $r_{\rm min}$ can be found in terms of the energy because at the
point of closest approach the kinetic energy is due purely to the
motion perpendicular to $\hat{r}$ and

$$
\begin{equation}
E=-\frac{\alpha}{r_{\rm min}}+\frac{L^2}{2mr_{\rm min}^2}.
\label{_auto8} \tag{11}
\end{equation}
$$

## Rutherford Scattering, More Manipulations

One can solve the quadratic equation for $1/r_{\rm min}$,

$$
\begin{equation}
\frac{1}{r_{\rm min}}=\frac{m\alpha}{L^2}+\sqrt{(m\alpha/L^2)^2+2mE/L^2}.
\label{_auto9} \tag{12}
\end{equation}
$$

We can plug the expression for $r_{\rm min}$ into the expression for $A$, Eq. ([10](#eq:rminofA)),

$$
\begin{equation}
A=\sqrt{(m\alpha/L^2)^2+2mE/L^2}=\sqrt{(\alpha^2/(4E^2b^4)+1/b^2}
\label{_auto10} \tag{13}
\end{equation}
$$

## Rutherford Scattering, final expression

Finally, we insert the expression for $A$ into that for the scattering angle, Eq. ([8](#eq:sthetover2)),

$$
\begin{eqnarray}
\label{eq:scattangle} \tag{14}
\sin(\theta_s/2)&=&\frac{m\alpha}{AL^2}\\
\nonumber
&=&\frac{a}{\sqrt{a^2+b^2}}, ~~a\equiv \frac{\alpha}{2E}
\end{eqnarray}
$$

## Rutherford Scattering, the Differential Cross Section

The differential cross section can now be found by differentiating the
expression for $\theta_s$ with $b$,

$$
\begin{eqnarray}
\label{eq:rutherford} \tag{15}
\frac{1}{2}\cos(\theta_s/2)d\theta_s&=&\frac{ab~db}{(a^2+b^2)^{3/2}}=\frac{bdb}{a^2}\sin^3(\theta_s/2),\\
\nonumber
d\sigma&=&2\pi bdb=\frac{\pi a^2}{\sin^3(\theta_s/2)}\cos(\theta_s/2)d\theta_s\\
\nonumber
&=&\frac{\pi a^2}{2\sin^4(\theta_s/2)}\sin\theta_s d\theta_s\\
\nonumber
\frac{d\sigma}{d\cos\theta_s}&=&\frac{\pi a^2}{2\sin^4(\theta_s/2)},\\
\nonumber
\frac{d\sigma}{d\Omega}&=&\frac{a^2}{4\sin^4(\theta_s/2)}.
\end{eqnarray}
$$

where $a= \alpha/2E$. This the Rutherford formula for the differential
cross section. It diverges as $\theta_s\rightarrow 0$ because
scatterings with arbitrarily large impact parameters still scatter to
arbitrarily small scattering angles. The expression for
$d\sigma/d\Omega$ is the same whether the interaction is positive or
negative.

## Rutherford Scattering, Example

Consider a particle of mass $m$ and charge $z$ with kinetic energy $E$
(Let it be the center-of-mass energy) incident on a heavy nucleus of
mass $M$ and charge $Z$ and radius $R$. We want to find the angle at which the
Rutherford scattering formula breaks down.

Let $\alpha=Zze^2/(4\pi\epsilon_0)$. The scattering angle in Eq. ([14](#eq:scattangle)) is

$$
\sin(\theta_s/2)=\frac{a}{\sqrt{a^2+b^2}}, ~~a\equiv \frac{\alpha}{2E}.
$$

The impact parameter $b$ for which the point of closest approach
equals $R$ can be found by using angular momentum conservation,

$$
\begin{eqnarray*}
p_0b&=&b\sqrt{2mE}=Rp_f=R\sqrt{2m(E-\alpha/R)},\\
b&=&R\frac{\sqrt{2m(E-\alpha/R)}}{\sqrt{2mE}}\\
&=&R\sqrt{1-\frac{\alpha}{ER}}.
\end{eqnarray*}
$$

## Rutherford Scattering, Example, wrapping up

Putting these together

$$
\theta_s=2\sin^{-1}\left\{
\frac{a}{\sqrt{a^2+R^2(1-\alpha/(RE))}}
\right\},~~~a=\frac{\alpha}{2E}.
$$

It was from this departure of the experimentally measured
$d\sigma/d\Omega$ from the Rutherford formula that allowed Rutherford
to infer the radius of the gold nucleus, $R$.

## Variational Calculus

The calculus of variations involves
problems where the quantity to be minimized or maximized is an integral.

The usual minimization problem one faces involves taking a function
${\cal L}(x)$, then finding the single value $x$ for which ${\cal L}$
is either a maximum or minimum. In multivariate calculus one also
learns to solve problems where you minimize for multiple variables,
${\cal L}(x_1,x_2,\cdots x_n)$, and finding the points $(x_1\cdots
y_n)$ in an $n$-dimensional space that maximize or minimize the
function. Here, we consider what seems to be a much more ambitious
problem. Imagine you have a function ${\cal L}(x(t),\dot{x}(t),t)$,
and you wish to find the extrema for an infinite number of values of
$x$, i.e. $x$ at each point $t$. The function ${\cal L}$ will not only
depend on $x$ at each point $t$, but also on the slope at each point,
plus an additional dependence on $t$. Note we are NOT finding an
optimum value of $t$, we are finding the set of optimum values of $x$
at each point $t$, or equivalently, finding the function $x(t)$.

## Variational Calculus, introducing the action

One treats the function $x(t)$ as being unknown while minimizing the action

$$
S=\int_{t_1}^{t_2}dt~{\cal L}(x(t),\dot{x}(t),t).
$$

Thus, we are minimizing $S$ with respect to an infinite number of
values of $x(t_i)$ at points $t_i$. As an additional criteria, we will
assume that $x(t_1)$ and $x(t_2)$ are fixed, and that that we will
only consider variations of $x$ between the boundaries. The dependence
on the derivative, $\dot{x}=dx/dt$, is crucial because otherwise the
solution would involve simply finding the one value of $x$ that
minimized ${\cal L}$, and $x(t)$ would equal a constant if there were no
explicit $t$ dependence. Furthermore, $x$ wouldn't need to be
continuous at the boundary.

## Variational Calculus, general Action

In the general case we have an integral of the type

$$
S[q]= \int_{t_1}^{t_2} {\cal L}(q(t),\dot{q}(t),t)dt,
$$

where $S$ is the quantity which is sought minimized or maximized.  The
problem is that although ${\cal L}$  is a function of the general variables
$q(t),\dot{q}(t),t$ (note our change of variables), the exact dependence of $q$ on $t$ is not known.
This means again that even though the integral has fixed limits $t_1$
and $t_2$, the path of integration is not known. In our case the unknown
quantities are the positions and general velocities of a given number
of objects and we wish to choose an integration path which makes the
functional $S[q]$ stationary. This means that we want to find minima,
or maxima or saddle points. In physics we search normally for minima.
Our task is therefore to find the minimum of $S[q]$ so that its
variation $\delta S$ is zero subject to specific constraints.  The
constraints can be treated via the technique of Lagrangian multipliers
as we will see below.

## Variational Calculus, Optimal Path

We assume the existence of an optimum path, that is a path for which
$S[q]$ is stationary. There are infinitely many such paths.  The
difference between two paths $\delta q$ is called the variation of
$q$.

We call the variation $\eta(t)$ and it is scaled by a factor $\alpha$.
The function $\eta(t)$ is arbitrary except for

$$
\eta(t_1)=\eta(t_2)=0,
$$

and we assume that we can model the change in $q$ as

$$
q(t,\alpha) = q(t)+\alpha\eta(t),
$$

and

$$
\delta q = q(t,\alpha) -q(t,0)=\alpha\eta(t).
$$

## Variational Calculus, Condition for an Extreme Value

We choose $q(t,\alpha=0)$ as the unkonwn path  that will minimize $S$.  The value
$q(t,\alpha\ne 0)$  describes a neighbouring path.

We have

$$
S[q(\alpha)]= \int_{t_1}^{t_2} {\cal L}(q(t,\alpha),\dot{q}(t,\alpha),t)dt.
$$

The condition for an extreme of

$$
S[q(\alpha)]= \int_{t_1}^{t_2} {\cal L}(q(t,\alpha),\dot{q}(t,\alpha),t)dt,
$$

is

$$
\left[\frac{\partial  S[q(\alpha)]}{\partial t}\right]_{\alpha=0} =0.
$$

## Variational Calculus. $\alpha$ Dependence

The $\alpha$ dependence is contained in $q(t,\alpha)$ and $\dot{q}(t,\alpha)$ meaning that

$$
\left[\frac{\partial  E[q(\alpha)]}{\partial \alpha}\right]=\int_{t_1}^{t_2} \left( \frac{\partial {\cal l}}{\partial q}\frac{\partial q}{\partial \alpha}+\frac{\partial {\cal L}}{\partial \dot{q}}\frac{\partial \dot{q}}{\partial \alpha}\right)dt.
$$

We have defined

$$
\frac{\partial q(x,\alpha)}{\partial \alpha}=\eta(x)
$$

and thereby

$$
\frac{\partial \dot{q}(t,\alpha)}{\partial \alpha}=\frac{d(\eta(t))}{dt}.
$$

## INtegrating by Parts

Using

$$
\frac{\partial q(t,\alpha)}{\partial \alpha}=\eta(t),
$$

and

$$
\frac{\partial \dot{q}(t,\alpha)}{\partial \alpha}=\frac{d(\eta(t))}{dt},
$$

in the integral gives

$$
\left[\frac{\partial  S[q(\alpha)]}{\partial \alpha}\right]=\int_{t_1}^{t_2} \left( \frac{\partial {\cal L}}{\partial q}\eta(t)+\frac{\partial {\cal L}}{\partial \dot{q}}\frac{d(\eta(t))}{dt}\right)dt.
$$

Integrating the second term by parts

$$
\int_{t_1}^{t_2} \frac{\partial {\cal L}}{\partial \dot{q}}\frac{d(\eta(t))}{dt}dt =\eta(t)\frac{\partial {\cal L}}{\partial \dot{q}}|_{t_1}^{t_2}-
\int_a^b \eta(t)\frac{d}{dx}\frac{\partial {\cal L}}{\partial \dot{q}}dt,
$$

and since the first term dissappears due to $\eta(a)=\eta(b)=0$, we obtain

$$
\left[\frac{\partial  S[q(\alpha)]}{\partial \alpha}\right]=\int_{t_1}^{t_2} \left( \frac{\partial {\cal L}}{\partial q}-\frac{d}{dx}\frac{\partial {\cal L}}{\partial \dot{q}}
\right)\eta(t)dt=0.
$$

## Euler-Lagrange Equations

The latter can be written as

$$
\left[\frac{\partial  S[q(\alpha)]}{\partial \alpha}\right]_{\alpha=0}=\int_{t_1}^{t_2} \left( \frac{\partial {\cal L}}{\partial q}-\frac{d}{\
dx}\frac{\partial {\cal L}}{\partial \dot{q}}\right)\delta q(t)dt=\delta S = 0.
$$

The condition for a stationary value is thus a partial differential equation

$$
\frac{\partial {\cal L}}{\partial q}-\frac{d}{dx}\frac{\partial {\cal L}}{\partial \dot{q}}=0,
$$

known as the **Euler-Lagrange** equation.


# PAC-LEARNING AND VC-DIMENSION

---

In this chapter we will consider two important aspect of the statistical learning theory. We will focus our attention on the problem of inductively learning an unknown target function given only training examples and a space of candidate hypothesis. We already know that overfitting can occurs since the training error is a bad estimate of the generalization error and also when the learner doesn't see enough samples.Our goal is to answer question such as:

*  Sample complexity : how many examples are needed for a learner to converge to a successful hypothesis?

*  Computational complexity : how much computational effort is needed for a learner to converge to a successful hypothesis?

*  Mistake bound : how many training examples will the learner misclassify before converging to a successful hypothesis?

## Probably learning and approximately correct hypothesis

---

For simplicity we will restrict our discussion to the case of learning boolean-valued concepts from noise-free data. Despite this restriction most of the results can be generalized to other cases. Let's consider the set of all possible instances as $\textbf X$ and the set of target concepts that our learner will try to learn as $\textbf C$. In the case of boolean-valued function we have that each concept $c$ corresponds to $c: \textbf X \rightarrow \{0, 1\}$. Let $\mathcal D$ the training data drawn from $\textbf X$ according to some stationary distribution $\mathcal P$. The learner L will try to find an hypothesis $h$, among all the hypothesis contained in the hypothesis space $\mathcal H$, that is a good approximation of the concept $c$.

$$
\begin{align}
h^* = arg \min_{h \in \mathcal H} error_{train}(h)
\end{align}
$$

We define the error of an hypothesis, as the probability that such hypothesis will misclassify an instance.

$$
\begin{align}
error_{\mathcal D}(h) = \Pr_{x \in \mathcal D}[h(x) \neq c(x)]
\end{align}
$$

This is the definition of the train error since $x$ is defined from the data set. The true error is the one in which the input can be drawn from the true, unknown probability.

$$
\begin{align}
error_{true}(h) = \Pr_{x \in ~ P}[h(x) \neq c(x)]
\end{align}
$$

This, the true error, is the one that we expect to see when we will use the learner L onto unseen samples. The most of the analysis on the complexity of learning centers around the question *"how probable is that the the training error for $h$ gives a misleading estimate of the true error?"*. We say that when the training error is higher than the true error the model is overfitting.

Should be noted that the error strictly depends on the probability.

In our analysis we will not require that the learner L output a zero error hypothesis, we will just require that the error of the learner is bounded by some constant $\varepsilon$, that can be made arbitrarily small. Moreover, we will not require that the learner succeed for every training set, we will require that its probability of failure is bounded by a constant $\delta$, that can be made arbitrarily small.

> We want a learner L that probably learn a hypothesis that is approximately correct.

### Version space

We have now understood that one the most critical aspect of a machine learning problem in the sample size, indeed, in most practical settings the factor that limits the success of a learner is the limited number of samples. This problem is usually called *sample complexity*.

We define a learner **consistent** if it outputs hypotheses that perfectly fit the training data.

$$ Consistent(h, \mathcal D) = \forall \langle x, c(x) \rangle \in \mathcal D, h(x) = c(x) $$

We now define the **version space** as the set of all hypotheses $h \in \mathcal H$ that correctly classify the training examples

$$ VS_{\mathcal H, \mathcal D} = \{h \in \mathcal H | Consistent(h, \mathcal D) \} $$

Every consistent learner outputs a hypothesis belonging to the version space since, by definition, the version space contains every consistent hypothesis in $\mathcal H$. Now the question, can we bound the $error_{true}$ of a consistent learner?

Theorem

*If the hypothesis space $\mathcal H$ is __finite__ and $\mathcal D$ is a sequence of $N \geq 1$ independent random examples of some target concept c, then for any $0 \leq \epsilon \leq 1$, the probability that $VS_{\mathcal H, \mathcal D}$ contains an hypothesis error greater than $\epsilon$ is less than $|\mathcal H|e^{- \epsilon N}$:*

$$
\begin{align}
\Pr(\exists h \in \mathcal H : L_{train}(h) = 0 \land L_{true} \geq \epsilon ) \leq |\mathcal H|e^{- \epsilon N}
\end{align}
$$

With the theorem above we are bounding the *probability* that a bad event occur. We have found an upper bound that relates the error on the training set of consistent learner to the true error. This bound depends on the number of samples, on the allowed error and on the size of the hypothesis space. We have explicitly found the parameters that govern the generalization capability of a model.

Proof

$$
\begin{align}
\Pr((L_{train}(h_1) = 0 \land L_{true}(h_1) \geq \epsilon) \lor ... \lor (L_{train}(h_{|\mathcal H|}) = 0 \land L_{true}(h_{|\mathcal H|}) \geq \epsilon
\end{align}
$$

$$
\begin{aligned}
&\leq \sum_{h \in \mathcal H} \Pr(L_{train}(h) = 0 \land L_{true}(h) \geq \epsilon) \quad &\textrm{(union bound)}\\
&\leq \sum_{h \in \mathcal H} \Pr(L_{train}(h) = 0 | L_{true}(h) \geq \epsilon) \quad &\textrm{(bound using Bayes' rule)} \\
 &\leq \sum_{h \in \mathcal H} (\ - \epsilon)^N \quad &\textrm{(bound on individual $h_is$)} \\
 &\leq |\mathcal H|(1 - \epsilon)^N \quad &\textrm{($k \leq |\mathcal H|$)} \\
 &\leq |\mathcal H|e^{-\epsilon N} \quad &\textrm{($1 - \epsilon \leq e^{-\epsilon}$ for $0 \leq \epsilon \leq 1$)}
\end{aligned}
$$

We can now use this result to determine the number of training samples required to reduce the probability of failure below some desired level $\delta$:

$$
\begin{align}
|\mathcal H|e^{- \epsilon N} < \delta
\end{align}
$$

Fixing, the accuracy $\epsilon$ and the confidence $\delta$  we can found the required number of samples:

$$
\begin{align}
N \geq \frac{1}{\epsilon} \Big(\ln |\mathcal H| + \ln \Big( \frac{1}{\delta}\Big)\Big)
\end{align}
$$

The expression above provides a general bound on the number of training points sufficient for any consistent learner to successfully learn any target concept in the hypothesis space, for any desired level of confidence and accuracy. In a similar manner, given the number of samples and a level of confidence, we can calculate the accuracy:

$$
\begin{align}
\epsilon \geq \frac{1}{N} \Big(\ln |\mathcal H| + \ln \Big( \frac{1}{\delta}\Big)\Big)
\end{align}
$$

Example: Learning conjunctions of boolean literals
A boolean literal is any boolean variable, e.g. "$Flat$", or its negation "$\neg Flat$". Conjunction of boolean literals includes concepts as "$Flat \land \neg Earth$". Now consider the hypothesis space $\mathcal H$ defined by conjunctions of literals based on $M$ boolean variables. The size of $\mathcal H$ is $3^M$, where the number $3$ is due to the possibility to include the variable as a literal in the hypothesis, include its negation as a literal and its absence. The question is, how many examples are sufficient to ensure with probability at least $(1 - \delta)$ that every $h$ in $VS_{\mathcal H, \mathcal D}$ satisfy $L_{true} \leq \epsilon$?

$$
\begin{align}
N \geq \frac{1}{\epsilon} \Big(\ln 3^M + \ln \Big(\frac{1}{\delta}\Big) \Big)
\end{align}
$$

For example, if the boolean literals are $10$ and we want a confidence of $95\%$ with an error less than $0.1$, the number of samples should be at lest $140$.

Consider now a class $C$ of possible target concepts and a learner $L$ using hypothesis space $\mathcal H$, we will say that the concept class $C$ is PAC-learnable by $L$ using $\mathcal H$ if, for any target concept $c$ in $C$, $L$ will output a hypothesis $h$ with $error_{\mathcal D}(h) < \epsilon$ with probability $(1-\delta)$, after observing a reasonable number of training examples and performing a reasonable amount of computation.

Definition

$C$ is **PAC-learnable** if there exists an algorithm $L$ such that for every $f \in C$, for any distribution $\mathcal P$, for any $\epsilon$ such that $0 \leq \epsilon \leq 1/2$, and $\delta$ such that $0 \leq \delta \leq 1$, algorithm $L$, with probability at least $1 - \delta$, outputs a concept $h$ such that $L_{true}(h) \leq \epsilon$ using a n umber of samples that is polynomial of $1/\epsilon, 1/\delta$.

Definition

$C$ is **efficiently PAC-learnable** by $L$ using $\mathcal H$ iff for all $c \in C$, distributions $\mathcal P$ over $\textbf X$, $\epsilon$ such that $0 \leq \epsilon \leq 1/2$, and $\delta$ such that $0 \leq \delta \leq 1$, algorithm $L$, with probability at least $1 - \delta$, outputs a concept $h \in \mathcal H$ such that $L_{true}(h) \leq \epsilon$, in a time that is **polynomial** in $1/\epsilon, 1/\delta, M$ and $size(c)$.

### Agnostic learning

In all the formulation just presented, the root assumption is that, in $\mathcal H$ exists a concepts $c$ such that the training error is zero. Obviously, this assumption cannot always be satisfied. A learner that makes no assumption that the target concept is in the hypothesis space, and that simply finds the hypothesis with minimum error on the training set, is called **agnostic** learner. Agnostic learner will always output an hypothesis $h$ such that $error_{\mathcal D}(h) > 0$, so, can we bound $error_{true}(h)$ given $error_{\mathcal D}(h)$? Explicitly, we want that:

$$
\begin{align}
error_{true}(h) \leq error_{\mathcal D}(h) + \epsilon
\end{align}
$$

We can use the **Hoeffding bound** which allows to characterize the deviation between the true probability of some event and its probability observed frequency over $N$ independent trials. For any $N$ i.i.d. coin flips $X_1, ..., X_N$, where $X_i \in \{0, 1\}$ and $0 \leq \epsilon \leq 1$, we define the empirical mean as $\bar{X}$, obtaining the following bound:

$$
\begin{align}
\Pr(\mathbb E[\bar{X}] - \bar{X} > \epsilon) < e^{-2N\epsilon^2}
\end{align}
$$

Theorem

*If the hypothesis space $\mathcal H$ is finite and the dataset $\mathcal D$ is a sequence of i.i.d. examples of some concept $c$, than for any $0 \leq \epsilon \leq 1$, then for any learned hypothesis $h$ in $\mathcal H$*:

$$
\begin{align}
\Pr(L_{true}(h) - L_{train}(h) > \epsilon) \leq |\mathcal H|e^{-2N\epsilon^2}
\end{align}
$$

We can bound the true error as:

$$
\begin{align}
L_{true}(h) \leq L_{train}(h) + \sqrt{\frac{\ln|\mathcal H| + \ln(1/\delta)}{2N}}
\end{align}
$$

We found again the bias-variance decomposition since the first term on the right-hand-side of the equation is the bias and the second one is the variance. So:

* For large $|\mathcal H|$

    * we have low bias, assuming we can find a good hypothesis

    * we have high variance, because the bound is loser

* For small $|\mathcal H|$

    * we have high bias

    * we have lowee variance

Given $\delta$ and $\epsilon$, how large should be $N$?

$$
\begin{align}
N \geq \frac{1}{2\epsilon^2} \Big(\ln |\mathcal H| + \ln \Big( \frac{1}{\delta}\Big)\Big)
\end{align}
$$

This bound is nothing but the generalization of the previous equation when the learner picks the best hypothesis but, the best hypothesis, may have nonzero training error.

## Sample complexity for infinite hypothesis spaces

---

In the above section we were able to bound the sample complexity in the case in which the cardinality of the hypothesis space was finite. But, what about continuous hypothesis spaces?
Using the same equation as before we will obtain an infinite bound and so an infinite variance. Actually no, the bound previously provided is just a pessimistic bound.

Example: Learning axes aligned rectangle

We want to learn an unknown target axes-aligned rectangle $R$. We randomly drawn samples with a label that indicates whether or not the point is contained in $R$. Consider the hypothesis corresponding to the tightest rectangle $R'$ around positive samples. The error region is the difference between $R$ and $R'$. This region can be seen as the union of four rectangular regions.

In each of these regions we want an error less than $\epsilon/4$. When $N$ samples are drawn, a bad event for a single rectangular region is when the probability of all the $N$ samples of being outside this region, which is at most $(1 - \epsilon/4)^N$. The same holds  for the other three regions. The same holds also for the other three regions and by union bound we obtain $4(1-\epsilon/4)^N$. If we want that the probability of a bad event is less than $\delta$ we simply impose:

$$
\begin{align}
(1 - \epsilon/4)^N \leq \delta
\end{align}
$$

By exploiting the inequality $(1-x) \leq e^{-x}$, we get:

$$
\begin{align}
N \geq (4/\epsilon) \ln (4/\delta)
\end{align}
$$

It is important in the example above the number of points that can be correctly classified. Can we get a bound error as a function of the number of points that can be completely labeled? Here we will consider another measure of the complexity of $\mathcal H$, called the **Vapnik-Chervonenkis dimension** which allows to state bounds on sample complexity that use the $VC(\mathcal H)$ instead of $|\mathcal H|$.

### Shattering a set of instances

The VC dimension measures the complexity of the hypothesis space not by the number of distinct hypotheses, but by the number of distinct instances from $\mathcal D$ that can be completely discriminated using $\mathcal H$.

Definition (Dichotomy)

A **dichotomy** of a set $S$ is a partition of $S$ into two disjoint subsets

Definition (Shattering)

A set of instances $S$ is **shattered** by hypothesis space $\mathcal H$ iff for every dichotomy of $S$ there exists some hypothesis $h \in \mathcal H$ consistent with this dichotomy

It is important, when we use the VC dimension, to specify the definition of the dichotomy and of the shuttering!

Let's now consider, as an example, the set $X$ of instances corresponding to points on the $x, y$ plane.

Let $\mathcal H$ be the set of all linear decision surfaces in the plane, what is the $VC$ dimension of this hypothesis space? Any two distinct points in the plane can be shattered by $\mathcal H$, because we can find four linear surfaces that the plane include neither, either, or both points. In the case of three points we will be able to find $2^3$ linear surfaces that shatter them. Of course three co-linear points cannot be shattered..so, what is the $VC$ dimension in this case? It will be at least three. The definition of the $VC$ dimension requires that, if we can find a configuration of instances (points) of size $d$ for which, we can correctly shatter any possible dichotomy of these instances, than the $VC$ dimension is at least $d$. To upper bound the $VC$ dimension we just have to show that $VC(\mathcal H) < d$

Definition

The **Vapnik-Chervonenkis dimension**, $VC(\mathcal H)$, of hypothesis space $\mathcal H$ defined over instance space $X$ is the size of the largest finite subset of $X$ shattered by $\mathcal H$. If arbitrarily large finite sets of $X$ can be shattered, then $VC(\mathcal H) \equiv \infty$

How many points can a linear boundary classify exactly in $M-$dimension? The solution is $M+1$. The rule of thumb is that the number of parameters in the model often matches the maximum number of points. However, in general, these two numbers can be completely different. There are problems where the number of parameters is infinite but the $VC$ dimension is finite and others in which with just $1$ parameter we have a $VC$ dimension which is infinite.

Some examples of $VC$ dimension:

* Linear classifier: $VC(\mathcal H) = M+1$, for $M$ features plus the constant term

* Neural networks: $VC(\mathcal H) = \textrm{number of parameters}$

* 1-Nearest neighbor: $VC(\mathcal H) =\infty$

* SVM with Gaussian kernel: $VC(\mathcal H) = \infty$

How many randomly drawn examples suffice to guarantee error of at most $\epsilon$ with probability at least $(1-\delta)$?

$$
\begin{align}
N \geq \frac{1}{\epsilon} \Big(4 \log_2\Big(\frac{2}{\delta}\Big) + 8 VC(\mathcal H) \log_2\Big(\frac{13}{\epsilon}\Big)\Big)
\end{align}
$$

We can now express the PAC bound using $VC$ dimension as:

$$
\begin{align}
L_{true}(h) \leq L_{train}(h) + \sqrt{\frac{VC(\mathcal H) \Big(\ln \frac{2N}{VC(\mathcal H)} + 1 \Big) + \ln \frac{4}{\delta}}{N} }
\end{align}
$$

Also in this case we can recognize the bias-variance decomposition and we have stated in statistical terms the intuition behind the dependence of the error on the training size. In **structural risk minimization** we choose the hypothesis space to minimize the above bound on expected true error.

Theorem

*The $VC$ dimension of a hypothesis space $|\mathcal H| < \infty$ is bounded from above:*

$$
\begin{align}
VC(\mathcal H) \leq \log_2(|\mathcal H|)
\end{align}
$$

Proof

*If $VC(\mathcal H) = d$ then there exist at least $2^d$ functions in $\mathcal H$, since there are at least $2^d$ possible labelings: $|\mathcal H| \geq 2^d$*

Theorem

*Concept class $C$ with $VC(C) = \infty$ is not PAC-learneble.*

References:

1. *Mithcell T.M., "Machine Learning", chapters: 7.1, 7.2, 7.3, 7.4*

2. *Restelli M., Course slides*