# Tutorial 3a: Parameter estimation by optimziation

(c) 2016 Justin Bois. This work is licensed under a [Creative Commons Attribution License CC-BY 4.0](https://creativecommons.org/licenses/by/4.0/). All code contained herein is licensed under an [MIT license](https://opensource.org/licenses/MIT).

*This tutorial was generated from an Jupyter notebook.  You can download the notebook [here](t3a_parameter_estimation_by_optimization.ipynb).*

```python
import numpy as np
import pandas as pd
import scipy.optimize
import statsmodels.tools.numdiff as smnd

# Import pyplot for plotting
import matplotlib.pyplot as plt

# Some pretty Seaborn settings
import seaborn as sns
rc={'lines.linewidth': 2, 'axes.labelsize': 14, 'axes.titlesize': 14}
sns.set(rc=rc)

# Make Matplotlib plots appear inline
%matplotlib inline

# We'll use Bokeh a bit
import bokeh.io
import bokeh.charts
bokeh.io.output_notebook()
```

    Loading BokehJS ...

In this tutorial, we will extend our introduction to parameter estimation from lecture toward computing regressions on data.  We will start with a little theory to set up the problem, and then proceed to performing nonlinear regressions on real data.

## Introduction to Bayesian regression
Recall the problem of estimating a single parameter from repeated measurements from [lecture](../lecture_notes/l02_parameter_estimation.pdf).  We had a likelihood of

\begin{align}
P(D \mid \mu, \sigma, I) = \prod_{i\in d} \frac{1}{\sqrt{2\pi\sigma^2}}\,
\exp\left[-\frac{(x_i - \mu)^2}{2\sigma^2}\right].
\end{align}

In other words, we assumed our measurements were Gaussian distributed with some mean $\mu$ and a standard deviation $\sigma$.  We can think about this in terms of mathematical and statistical models.  Specifying the likelihood (and prior) is a *statistical model*, where we describe how the data are distributed.  The *mathematical model* is kind of trivial, so we didn't really pause to discuss it before.  Our mathematical model is that physical process underlying the data produces the same value $\mu$ of the data.  That is, the equation of our mathematical model is $x = \mu$.

The analysis we did is not limited to this special choice of mathematical.  For example, let's say we perform a measurement, getting result $y$ for a given value of $x$.  In this case, our data set would consist of $(x,y)$ pairs; $D = \{x_i, y_i\}$.  Say we derive a mathematical model such that $y = ax + b$.  If this is the case, we have a likelihood of

\begin{align}
P(D \mid a, b, \sigma, I) = \prod_{i\in D} \frac{1}{\sqrt{2\pi\sigma^2}}\,
\exp\left[-\frac{(y_i - a x_i - b)^2}{2\sigma^2}\right],
\end{align}

where $\sigma$ is the unknown variance describing how the data deviate from the model.  This likelihood again assumes that the data vary off of the "ideal" given by the (linear) mathematical model in a Gaussian manner.

More generally, we could put any function in there.  Our mathematical model could be $y = f(x, \mathbf{a})$, where $\mathbf{a}$ is a set of parameters.  Then,

\begin{align}
P(D \mid \mathbf{a}, \sigma, I) = \prod_{i\in D} \frac{1}{\sqrt{2\pi\sigma^2}}\,
\exp\left[-\frac{(y_i - f(x,\mathbf{a}))^2}{2\sigma^2}\right].
\end{align}

After specifying the prior, we have the posterior and are in principle done with our parameter estimation.  However, we do often want to summarize the posterior, and actually computing it for a general $f(x, \mathbf{a})$ can be intractable.  In this tutorial, we will work with a real data set and demonstrate some of the ways we describe the posterior without actually computing it fully.

## The data set

The data set we will use for this analysis comes from [Good, et al., Cytoplasmic volume modulates spindle size during embryogenesis
 *Science*, **342**, 856-860, 2013](../protected/papers/good_2013.pdf).  You can download the data set [here](../data/good_invitro_droplet_data.csv).  In this work, Matt Good and coworkers developed a microfluidic device where they could create droplets of cytoplasm extracted from *Xenopus* eggs and embryos (see image from the paper, below, with 20 µm scale bars).  A remarkable property about *Xenopus* extract is that mitotic spindles spontaneously form; the extracted cytoplasm has all the ingredients to form them.  This makes it an excellent model system for studying spindles.  With their device, Good and his colleagues were able to study how the size of the cell affects the dimensions of the mitotic spindle; a simple, yet beautiful, question.  The experiment is conceptually simple; they made the droplets and then measured their dimensions and the dimensions of the spindles using microscope images.  In this tutorial, we will analyze their data relating droplet size and spindle size.

## Theoretical models for spindle size

Since we will be fitting the data with a theoretical model, I briefly summarize the two competing models for spindle size proposed by Good, et al. in their paper.

### Model a: Set spindle size

As a first model, we propose that the size of a mitotic spindle is inherent to the spindle itself.  This means that the size of the spindle is independent of the size of the droplet or cell in which it resides.  This would be the case, for example, if construction of the spindle involves length-sensing molecules, such as depolymerizing motor proteins (see, e.g., [this paper](../protected/papers/varga_2009.pdf) by Varga, et al.).  We define that set length as $\theta$.

This is the same problem as before: estimating the value of a parameter by repeated measurements.  The mathematical model in this case is $l = \theta$, where $l$ denotes the length of the spindle.  Each datum is then

\begin{align}
l_i = \theta + e_i,
\end{align}

where $e_i$ is the noise component of the $i$th datum.  The noise is assumed to come from a Gaussian distribution with variance $\sigma^2$.  The value of $\sigma$ is the same for all data.  In this case, the likelihood of a set of measurements $D = \{l_1, l_2,\ldots\}$ is

\begin{align}
P(D~|~\theta, \sigma, M_a, I) = \prod_{i\in D} \frac{1}{\sqrt{2\pi\sigma^2}}\,
\exp\left[-\frac{(l_i - \theta)^2}{2\sigma^2}\right],
\end{align}

where $M_a$ denotes that Model a is true.

### Model b: Spindle size is dependent on total tubulin concentration

This model hinges on the following principles:
1. The total amount of tubulin in the droplet or cell is conserved.
2. The total length of polymerized microtubules is a function of the total tubulin concentration after assembly of the spindle.  This results from the balances of microtubule polymerization rate with catastrophe frequencies.
3. The density of tubulin in the spindle is independent of droplet or cell volume.

From these principles, we'll proceed to derive a mathematical model to describe how spindle size should depend on droplet size.

Assumption 1 (conservation of tubulin) implies

\begin{align}
T_0 V_0 = T_1(V_0 - V_\mathrm{s}) + T_\mathrm{s}V_\mathrm{s},
\end{align}

where $V_0$ is the volume of the droplet or cell, $V_\mathrm{s}$ is the volume of the spindle, $T_0$ is the total tubulin concentration (polymerized or not), $T_1$ is the tubulin concentration in the cytoplasm after the the spindle has formed, and $T_\mathrm{s}$ is the concentration of tubulin in the spindle.  If we assume the spindle does not take up much of the total volume of the droplet or cell ($V_0 \gg V_\mathrm{s}$, which is the case as we will see when we look at the data), we have

\begin{align}
T_1 \approx T_0 - \frac{V_\mathrm{s}}{V_0}\,T_\mathrm{s}.
\end{align}

The amount of tubulin in the spindle can we written in terms of the total length of polymerized microtubules, $L_\mathrm{MT}$ as

\begin{align}
T_s V_\mathrm{s} = \alpha L_\mathrm{MT},
\end{align}

where $\alpha$ is the tubulin concentration per unit microtubule length.  (We will see that it is unimportant, but from the known geometry of microtubules, $\alpha \approx 2.7$ nmol/µm.)

We formalize assumption 2 into a mathematical expression.  Microtubule length should grow with increasing $T_1$.  There should also be a minimal threshold $T_\mathrm{min}$ where polymerization stops.  We therefore approximate the total microtubule length as a linear function,

\begin{align}
L_\mathrm{MT} \approx \left\{\begin{array}{ccl}
0 & &T_1 \le T_\mathrm{min} \\
\beta(T_1 - T_\mathrm{min}) & & T_1 > T_\mathrm{min}.
\end{array}\right.
\end{align}

Because spindles form in *Xenopus* extract, $T_0 > T_\mathrm{min}$, so there exists a $T_1$ with $T_\mathrm{min} < T_1 < T_0$, so going forward, we are assured that $T_1 > T_\mathrm{min}$.  Thus, we have

\begin{align}
V_\mathrm{s} \approx \alpha\beta\,\frac{T_1 - T_\mathrm{min}}{T_\mathrm{s}}.
\end{align}

With insertion of our expression for $T_1$, this becomes

\begin{align}
V_{\mathrm{s}} \approx \alpha \beta\left(\frac{T_0 - T_\mathrm{min}}{T_\mathrm{s}} - \frac{V_\mathrm{s}}{V_0}\right).
\end{align}

Solving for $V_\mathrm{s}$, we have

\begin{align}
V_\mathrm{s} \approx \frac{\alpha\beta}{1 + \alpha\beta/V_0}\,\frac{T_0 - T_\mathrm{min}}{T_\mathrm{s}}
=\frac{V_0}{1 + V_0/\alpha\beta}\,\frac{T_0 - T_\mathrm{min}}{T_\mathrm{s}}.
\end{align}

We approximate the shape of the spindle as a prolate spheroid with major axis length $l$ and minor axis length $w$, giving

\begin{align}
V_\mathrm{s} = \frac{\pi}{6}\,l w^2 = \frac{\pi}{6}\,k^2 l^3,
\end{align}

where $k \equiv w/l$ is the aspect ratio of the spindle.  We can now write an expression for the spindle length as

\begin{align}
l \approx \left(\frac{6}{\pi k^2}\,
\frac{T_0 - T_\mathrm{min}}{T_\mathrm{s}}\,
\frac{V_0}{1+V_0/\alpha\beta}\right)^{\frac{1}{3}}.
\end{align}

For small droplets, with $V_0\ll \alpha \beta$, this becomes

\begin{align}
l(d) \approx \left(\frac{6}{\pi k^2}\,
\frac{T_0 - T_\mathrm{min}}{T_\mathrm{s}}\,
V_0\right)^{\frac{1}{3}}
= \left(\frac{T_0 - T_\mathrm{min}}{k^2T_\mathrm{s}}\right)^{\frac{1}{3}}\,d,
\end{align}

where $d$ is the diameter of the spherical droplet or cell. So, we expect the spindle size to increase linearly with the droplet diameter for small droplets.

For large $V_0$, the spindle size becomes independent of droplet size;

\begin{align}
l \approx \left(\frac{6 \alpha \beta}{\pi k^2}\,
\frac{T_0 - T_\mathrm{min}}{T_\mathrm{s}}\right)^{\frac{1}{3}}.
\end{align}

We can define two parameters to describe the data,

\begin{align}
\gamma &= \left(\frac{T_0-T_\mathrm{min}}{k^2T_\mathrm{s}}\right)^\frac{1}{3} \\
\theta &= \gamma\left(\frac{6\alpha\beta}{\pi}\right)^{\frac{1}{3}}.
\end{align}

We assume that $\gamma$ and $\theta$ are the same for all data.  We can rewrite the general model expression in terms of these parameters as

\begin{align}
l(d) \approx \frac{\gamma d}{\left(1+(\gamma d/\theta)^3\right)^{\frac{1}{3}}}.
\end{align}

For small and large droplets, respectively, we have

\begin{align}
l(d) &\approx \gamma d ~\text{ for } \gamma d/\theta \ll 1, \\[1em]
l &\approx \theta ~~~\text{ for } \gamma d/\theta \gg 1.
\end{align}

Note that the expression for the linear regime gives bounds for $\gamma$.  Obviously, $\gamma > 0$.  Because $l \le d$, lest the spindle not fit in the droplet, we also have $\gamma \le 1$.  The parameter $\theta$ is independent of the system geometry, so it only has the physical lower bound of $\theta > 0$.

To get an idea of what we might expect from the shapes of curves from these two models, let's make some plots.  This is easiest if we choose $\theta$ as the units for both the $x$- and $y$-axes.

```python
# Generate dimensionless d range to plot
d = np.linspace(0.0, 10.0, 200)

# Values of gamma to use in the plot
gamma = np.array([0.1, 0.3, 0.7, 1.0])

# Make the plots
with sns.color_palette('Blues_r'):
    legend_labels = []
    for i in range(len(gamma)-1, -1, -1):
        plt.plot(d, gamma[i] * d / (1 + (gamma[i] * d)**3)**(1.0 / 3.0), '-')
        legend_labels.append(r'$\gamma = {0:g}$'.format(gamma[i]))

# Make plot pretty
plt.margins(y=0.02)
plt.xlabel(r'$d/\theta$')
plt.ylabel(r'$l/\theta$')
plt.legend(legend_labels, loc='lower right');
```

We can also look at how the different limits compare to the theoretical curves.  We will do this for $\gamma = 0.3$, plotting the limits as dashed lines.

```python
# Plot the full theoretical curve for gamma = 0.3
gamma = 0.3
plt.plot(d, gamma * (d**3 / (1 + (gamma * d)**3))**(1.0 / 3.0), 'k-')

# Plot plateau region (horizontal line at y = 1)
plt.plot([d[0], d[-1]], [1.0, 1.0], 'k--')

# Plot linear region
plt.plot([d[0], d[-1]], gamma * np.array([d[0], d[-1]]), 'k--')

# Make plot pretty
plt.ylim((-0.02, 1.02))
plt.xlabel(r'$d/\theta$')
plt.ylabel(r'$l/\theta$');
```

We can now write the likelihood under this model.

\begin{align}
P(D \mid \theta, \gamma, M_b, I) = \prod_{i\in D} \frac{1}{\sqrt{2\pi\sigma^2}}\,
\exp\left[-\frac{(l_i - l(d_i;\theta, \gamma))^2}{2\sigma^2}\right],
\end{align}

where

\begin{align}
l(d; \theta, \gamma) = \frac{\gamma d}{\left(1+(\gamma d/\theta)^3\right)^{\frac{1}{3}}}.
\end{align}

### A comment on the model parameters

We went through some algebraic manipulations to get our mathematical model in a form with two parameters.  We want to try to identify *independent* parameters in your mathematical before doing regression analysis.  In a trivial example, imagine someone proposed the following model to use in a regression on $(x, y)$ data:

\begin{align}
y = ax + bx + c.
\end{align}

Obviously, it would be silly to have both $a$ and $b$ as regression parameters, and we should instead define a new parameter $d = a + b$ and use that as a regression parameter.  In the case of spindle length, we had parameters $T_0$, $T_\mathrm{min}$, $T_\mathrm{s}$, $k$, $\alpha$, and $\beta$, but, as we saw, we can only resolve two parameters, $\gamma$ and $\theta$.  Furthermore, if we happen to be in the linear regime, $\theta$ does not enter the expressions, so we obviously cannot resolve it.  Similarly, we can only determine $\theta$ if we are in the plateau regime.

Finally, we note that if the parameters are such that we are in the plateau regime, we cannot discern Model a from Model b.  This tells us that as experimenters, we need to study a range of droplet sizes so we can cover both regimes.

## Loading the data

Matt Good sent me beautifully organized and formatted (tidy!) data files (you should have  something similar available for all of your publications!).  We will load in the data used to generate Fig. 1C of the paper.

```python
# Load data into DataFrame
df = pd.read_csv('good_invitro_droplet_data.csv', comment='#')

# Check it out
df.head()
```

      Droplet Diameter (um)
      Droplet Volume (uL)
      Spindle Length (um)
      Spindle Width (um)
      Spindle Area (um2)

      0
      27.1
      0.000010
      28.9
      10.8
      155.8

      1
      28.2
      0.000012
      22.7
      7.2
      81.5

      2
      29.4
      0.000013
      26.2
      10.5
      138.3

      3
      31.0
      0.000016
      19.2
      9.4
      90.5

      4
      31.0
      0.000016
      28.4
      12.1
      172.4

Conveniently, the data are tidy, and we're ready to go!

## Checking the model 2 assumptions

As an aside, it is often useful to check the assumptions associated with the model we chose.  Remember, we assumed that the volume of the spindle was much less than the volume of the droplet ($V_\mathrm{s} / V_0 \ll 1$) and that the aspect ratio of the spindle, $k$, is the same for all spindles.

### Is $V_\mathrm{s} / V_0 \ll 1$?

Let's do a quick verification that the droplet volume is indeed much larger than the spindle volume.  Remember, the spindle volume for a prolate spheroid of length $l$ and width $w$ is $V_\mathrm{s} = \pi l w^2 / 6$.  Also remember that 1 µL = $10^{-9}$ µm$^3$.

```python
# Compute spindle volume
spindle_volume = np.pi * df['Spindle Length (um)'] \
                        * df['Spindle Width (um)']**2 / 6.0 * 1e-9

# Compute the ratio V_s / V_0
vol_ratio = spindle_volume / df['Droplet Volume (uL)']

# Compute ECDF of the results
ecdf = lambda x: (np.sort(x), np.arange(1, len(x)+1) / len(x))
x, y = ecdf(vol_ratio)

# Plot the ECDF
plt.plot(x, y, marker='.', linestyle='none')
plt.margins(0.02)
plt.xlabel(r'$V_\mathrm{s} / V_0$')
plt.ylabel('ECDF');
```

We see that for the vast majority of spindles that were measured, $V_\mathrm{s} / V_0$ is small.

### Do all spindles have the same aspect ratio $k$?

In setting up our model, we assumed that all spindles had the same aspect ratio.  We can check this assumption because we have the data to do so available to us.

```python
# Compute the aspect ratio
k = df['Spindle Width (um)'] / df['Spindle Length (um)']

# Plot an ECDF of aspect ratio
x, y= ecdf(k)
plt.plot(x, y, marker='.', linestyle='none')
plt.margins(0.02)
plt.xlabel(r'$k$')
plt.ylabel('ECDF');
```

The mean aspect ratio is about 0.4, and we see spindle lengths about $\pm 25\%$ of that.  This could be significant variation, but we will leave it to the homework to test that.  For the purposes of this tutorial, we will assume that $k$ is constant.

## Exploratory data analysis

As we have mentioned before in class, it is often useful to explore the data by plotting it ahead of more formal analysis.  So, to start, we will plot the data generating a plot similar to Fig. 1C in the paper.  We plot spindle length vs. droplet diameter.

```python
# Use alpha when you have lots of points; helps see overlap
plt.plot(df['Droplet Diameter (um)'], df['Spindle Length (um)'], marker='.',
         linestyle='none', markersize=10, alpha=0.25)
plt.ylim(0.0, 50.0) # Be sure to include origin
plt.xlabel('droplet diameter (µm)')
plt.ylabel('spindle length (µm)');
```

Of course, for quick, exploratory data analysis, Bokeh is even better.

```python
# Use alpha when you have lots of points; helps see overlap
p = bokeh.charts.Scatter(df, x='Droplet Diameter (um)', y='Spindle Length (um)',
                         height=350, width=600, color='dodgerblue')
bokeh.io.show(p)
```

<Bokeh Notebook handle for In[5]>

## Parameter estimation under Model a

In Model a, we assume that spindle length is independent of the droplet diameter and that each measurement has the same Gaussian distribution for its error.  We have already worked out how to perform the estimate for the spindle length, $\theta$, in lecture.  The posterior is given by the Student-t distribution.

\begin{align}
P(\theta \mid D, M_a, I) = \frac{\Gamma\left(\frac{n}{2}\right)}{\sqrt{\pi}\,\Gamma\left(\frac{n-1}{2}\right)}\,\frac{1}{r}\left(1 + \frac{\left(\theta - \bar{l}\right)^2}{r^2}\right)^{-\frac{n}{2}},
\end{align}

where $n = |D|$, and

\begin{align}
\bar{l} &= \frac{1}{n}\sum_{i\in D} l_i,\\[1em]
\text{and } r^2 &= \frac{1}{n}\sum_{i\in D} (l_i - \bar{l})^2.
\end{align}

Clearly, the most probable value of $\theta$ is $\bar{l}$.  To compute its error bar, we will employ a trick we will use more generally in regression.  We will approximate the posterior distribution as Gaussian at its maximum and then report an error bar based on the variance of that Gaussian.  To make the Gaussian approximation, we expand the logarithm of the posterior in a Taylor series to second order.

\begin{align}
\ln P(\theta \mid D, M_a, I) = \text{constant} - \frac{n}{2}\,\ln\left(1 + \frac{(\theta - \bar{l})^2}{r^2}\right)
\approx \text{constant} - \frac{n(\theta - \bar{l})^2}{r^2}.
\end{align}

Exponentiating this expression, we get that

\begin{align}
P(\theta \mid D, M_a, I) \approx \frac{1}{\sqrt{2\pi r^2 / n}}\,
\exp\left[-\frac{(\theta - \bar{l})^2}{2 r^2 / n}\right].
\end{align}

Thus, $\theta$ is Gaussian distributed with a variance of $r^2/n$.  We could then report an error bar, which contains approximately 68% of the probability, as

\begin{align}
\theta = \bar{l} \pm\left.r\middle/\sqrt{n}\right..
\end{align}

This error bar of $\left.r\middle/\sqrt{n}\right.$ is often called the **standard error of the mean**, as it represents the error in an estimate of a mean.  Now that we know how to do this, we can directly compute our estimate using Numpy functions

```python
# Estimate spindle length for Model a
theta = np.mean(df['Spindle Length (um)'])
r = np.std(df['Spindle Length (um)'])
n = len(df['Spindle Length (um)'])

# Print results
print("""
Model a results (≈68% of total posterior probability)
-----------------------------------------------------
θ = {0:.1f} ± {1:.1f} µm
""".format(theta, r / np.sqrt(n)))
```

    Model a results (≈68% of total posterior probability)
    -----------------------------------------------------
    θ = 32.9 ± 0.2 µm

Note that we have a very good estimate of the mean; the error bar is small compared to the value.  This is often the case when there is a large number of measurements.  We know the mean very well; but error of the mean does not directly tell us about spread in the data.

Let's plot the model prediction with the data.

```python
# Use alpha when you have lots of points; helps see overlap
plt.plot(df['Droplet Diameter (um)'], df['Spindle Length (um)'], marker='.',
         linestyle='none', markersize=10, alpha=0.25)

# Plot the result
plt.plot([0, 250], [theta, theta], color=sns.color_palette()[2])
plt.xlabel('droplet diameter (µm)')
plt.ylabel('spindle length (µm)');
```

## Parameter estimation for Model b (nonlinear regression)
Nonlinear regression is just a fancy name for a certain kind of parameter estimation.  We will do the parameter estimation as we laid out at the beginning.  We already computed the likelihood.  We have three parameters, the two which came from the mathematical model, $\theta$ and $\gamma$, plus the unknown variance in the data, given by $\sigma$.  We assume uniform priors for $\theta$ and $\gamma$ and a Jeffreys prior for $\sigma$.  The posterior is then

\begin{align}
P(\theta, \gamma, \sigma \mid D, I) \propto\frac{1}{\sigma^{n+1}}
\exp\left[-\frac{1}{2\sigma^2}\sum_{i\in D}(l_i - l(d_i;\theta, \gamma))^2\right],
\end{align}

where

\begin{align}
l(d; \theta, \gamma) = \frac{\gamma d}{\left(1+(\gamma d/\theta)^3\right)^{\frac{1}{3}}}.
\end{align}

We can marginalize the posterior over $\sigma$ exactly as before, giving a Student-t distribution.

\begin{align}
P(\theta, \gamma \mid D, I) \propto \left(\sum_{i\in D}(l_i - l(d_i;\theta, \gamma))^2\right)^{-\frac{n}{2}} .
\end{align}

Note that this marginalization is independent of the choice of the functional form of $l(d;\theta, \gamma)$, and is therefore valid for any regression where the errors are Gaussian distributed.  With the posterior in hand, we can plot it.  We don't really care about the normalization, since we just want a plot and can always numerically normalize it.

```python
tt
```

    array([[ 37.        ,  37.03030303,  37.06060606, ...,  39.93939394,
             39.96969697,  40.        ],
           [ 37.        ,  37.03030303,  37.06060606, ...,  39.93939394,
             39.96969697,  40.        ],
           [ 37.        ,  37.03030303,  37.06060606, ...,  39.93939394,
             39.96969697,  40.        ],
           ...,
           [ 37.        ,  37.03030303,  37.06060606, ...,  39.93939394,
             39.96969697,  40.        ],
           [ 37.        ,  37.03030303,  37.06060606, ...,  39.93939394,
             39.96969697,  40.        ],
           [ 37.        ,  37.03030303,  37.06060606, ...,  39.93939394,
             39.96969697,  40.        ]])

```python
def spindle_length(p, d):
    """
    Theoretical model for spindle length
    """
    theta, gamma = p
    return gamma * d / np.cbrt(1 + (gamma * d / theta)**3)

def log_post(p, d, ell):
    """
    Compute log of posterior for single set of parameters.
    p[0] = theta
    p[1] = gamma
    """
    # Unpack parameters
    theta, gamma = p

    # Theoretical spindle length
    ell_theor = spindle_length(p, d)

    return -len(d) / 2 * np.log(np.sum((ell - ell_theor)**2))

# Parameter values to plot
th = np.linspace(37, 40, 100)
gamma = np.linspace(0.8, 0.95, 100)

# Make a grid
tt, gg = np.meshgrid(th, gamma)

# Compute log posterior
log_posterior = np.empty_like(tt)
for j in range(len(th)):
    for i in range(len(gamma)):
        log_posterior[i, j] = log_post(np.array([tt[i,j], gg[i,j]]),
                                       df['Droplet Diameter (um)'],
                                       df['Spindle Length (um)'])

# Get things to scale better
log_posterior -= log_posterior.max()

# Plot the results
plt.contourf(tt, gg, np.exp(log_posterior), cmap=plt.cm.Blues, alpha=0.7)
plt.xlabel(r'$\theta$ (µm)')
plt.ylabel(r'$\gamma$');
```

Again, in principle, we are finished with parameter estimation for this model.  The posterior tells us all we could ask for.  In this case, we see that the most probable value of $\theta$ is about 38.25 µm, and the most probable $\gamma$ is about 0.86.  They are also negatively correlated; a larger $\theta$ results in a smaller $\gamma$.

### Gaussian approximation of the posterior

However, we often want to report summaries of the posterior, so we will take the same approach as in Model a and approximate it as a Gaussian.  We will do the same procedure: expand the logarithm of the posterior in a Taylor series to give a two-dimensional Gaussian.  In general, for a set of parameters $\mathbf{a}$, where $\mathbf{a}^*$ is the most probable values, the approximation to give a Gaussian is

\begin{align}
\ln P(\mathbf{a}\mid D, I) \approx \text{constant} + \frac{1}{2}\left(\mathbf{a} - \mathbf{a}^*\right)^T \cdot \mathsf{H} \cdot \left(\mathbf{a} - \mathbf{a}^*\right),
\end{align}

where $\mathsf{H}$ is the symmetric **Hessian matrix** of second derivatives,

\begin{align}
H_{ij} = \left.\frac{\partial^2\,\ln P(\mathbf{a}\mid D, I)}{\partial a_i\,\partial a_j}\right|_{\mathbf{a} = \mathbf{a}^*}.\\[0.1em]
\phantom{blah}
\end{align}

Because we are evaluating the second derivatives at the maximal posterior probability, the Hessian is positive definite and therefore invertible.  Its negative inverse is the **covariance matrix**, $\sigma^2$.

So, our task is two-fold.  First, find the parameter values that give the maximal posterior probability, the so-called **maximal a posteriori estimate**, or MAP.  Second, compute the Hessian of the log posterior at the MAP and invert it to get your covariance matrix.  Note that the covariance matrix is **not** the inverse of every element of the Hessian; it is the inverse of the Hessian matrix.  The error bars for each parameter are given by the diagonal elements of the covariance matrix.

### Finding the MAP
Finding the MAP amounts to an **optimization problem** in which we find the arguments (parameters) for which a function (the posterior distribution) is maximal. We already did this analytically for Model a. For this model, though, an analytical calculation is very difficult. If we look at the form of the posterior, we see that the posterior is maximal when the sum

\begin{align}
\sum_{i \in D} (l_i - l(d_i;\theta,\gamma))^2
\end{align}

is minimal.  So, we need to find the values of $\theta$ and $\gamma$ that minimize the sum of the square of the residuals, where a **residual** is the difference from the prediction of the mathematical model and the measured value.

This is very common in curve fitting applications, though not all (can you think of exceptions?).  As such, highly optimized algorithms exist for numerically solving this optimization problem.  In particular, the **Levenberg-Marquardt** algorithm effectively solves the problem of minimizing the sum of the squares of a function.  In our case, the function is the residual.  This is implemented in the `scipy.optimize.leastsq()` function, which we will use to find the MAP.  As usual, you should read the [docs](http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.leastsq.html).

We start be defining a residual function, as the function requires.  The first argument must be a Numpy array of the parameter values, and remaining arguments can be passed in after that.

```python
def resid(p, d, ell):
    """
    Residuals for spindle length model.
    """
    return ell - spindle_length(p, d)
```

Note that this function returns all residuals (not their squares) as a NumPy array.  Next, we have to provide an initial guess for parameter values.  The optimization routine only finds a *local* maximum and is not in general guaranteed to converge.  Therefore, the initial guess can be very important.  Looking at the data, and remembering that the spindle length is approximately $\theta$ for large droplets, we guess $\theta = 40$.  For $\gamma$, we make a guess based on the initial slope of the data, probably about 30/50 = 0.6.  We store these in a NumPy array.

```python
# Initial guess
p0 = np.array([40, 0.6])
```

We are now ready to use `scipy.optimize.leastsq()` to compute the MAP.  We use the `args` kwarg to pass in the other arguments to the `resid()` function.  In our case, these arguments are the data points.  The `leastsq()` function returns multiple values, but the first, the optimal parameter values (the MAP), is all we are interested in.  We therefore use a dummy variable, `_`, for the other values that are returned.

Note that when we pass slices of `DataFrame`s as arguments into functions that will be called over and over again, it is worthwhile to convert them to Numpy arrays using the `values` attribute. While `DataFrame`s are very powerful for organizing and selecting which data you want, they are less efficient to use in computation than Numpy arrays.

```python
# Extra arguments as a tuple
args = (df['Droplet Diameter (um)'].values, df['Spindle Length (um)'].values)

# Compute the MAP
popt, _ = scipy.optimize.leastsq(resid, p0, args=args)

# Extract the values
theta, gamma = popt

# Print results
print("""
Model 2b, most probable parameters
----------------------------------
θ = {0:.1f} µm
γ = {1:.2f}
""".format(theta, gamma))
```

    Model 2b, most probable parameters
    ----------------------------------
    θ = 38.2 µm
    γ = 0.86

Just because we are excited and can't wait, we can plot our curve along with the data to qualitatively assess the fit.

```python
# Values of droplet diameter to plot
d_plot = np.linspace(0, 250, 200)

# Theoretical curve
spindle_theor = spindle_length(popt, d_plot)

# Plot results
plt.plot(df['Droplet Diameter (um)'], df['Spindle Length (um)'], marker='.',
         linestyle='none', markersize=10, alpha=0.25)

# Plot the result
plt.plot(d_plot, spindle_theor, color=sns.color_palette()[2])
plt.xlabel('droplet diameter (µm)')
plt.ylabel('spindle length (µm)');
```

We see that if Model b is in fact correct, most of the droplet data lie in the nonlinear part of the curve.

```python
smnd.approx_hess?
```

### Computing the covariance matrix
To compute the covariance matrix, we need to compute the Hessian of the log of the posterior at the MAP.  We will do this numerically using the `statsmodels.tools.numdiff` module, which we imported as `smnd`. We use the function `smnd.approx_hess()` to compute the Hessian at the optimal parameter values.

```python
hes = smnd.approx_hess(popt, log_post, args=args)
```

Now that we have the Hessian, we take its negative inverse to get the covariance matrix.

```python
# Compute the covariance matrix
cov = -np.linalg.inv(hes)

# Look at it
cov
```

    array([[ 0.15357222, -0.00538284],
           [-0.00538284,  0.0003034 ]])

The diagonal terms give the approximate variance in the  regression parameters.  The off-diagonal terms give the covariance, which describes how the parameters relate to each other.  Nonzero covariance indicates that they are not completely independent.

We can now report our error bars with the measured covariance.

```python
# Report results
print("""
Results for Model b (≈ 68% of total probability)
------------------------------------------------
θ = {0:.1f} ± {1:.1f} µm
γ = {2:.2f} ± {3:.2f}
""".format(theta, np.sqrt(cov[0,0]), gamma, np.sqrt(cov[1,1])))
```

    Results for Model b (≈ 68% of total probability)
    ------------------------------------------------
    θ = 38.2 ± 0.4 µm
    γ = 0.86 ± 0.02

## Conclusions
You now know how to do nonlinear regression!  Formally, it is just computing the posterior probability distribution, as usual.  To summarize it, we approximated it as Gaussian and computed the MAP and covariance matrix.

It is important to note that for models with a large number of parameters, plotting the posterior distribution is intractable.  Fortunately, the summary (MAP and covariance matrix) can be computed numerically by optimization without having to plot a multidimensional posterior.  This fact, and the speed of the optimization MAP/covariance method, make it attractive.

```python

```


```python
# This code block is for automatic testing purposes, please ignore.
try:
    import openfermionprojectq
except:
    import os
    os.chdir('..')
```

## Simulating a variational quantum eigensolver using OpenFermion-ProjectQ
We now demonstrate how one can use both openfermion and ProjectQ to run a simple VQE example using a Unitary Coupled Cluster ansatz. It demonstrates a simple way to evaluate the energy, optimize the energy with respect to the ansatz and build the corresponding compiled quantum circuit. It utilizes OpenFermion to prepare the Hamiltonians as well as initial parameters and ProjectQ to build and simulate the circuit.

```python
import os

from numpy import array, concatenate, zeros
from numpy.random import randn
from scipy.optimize import minimize

from openfermion.config import *
from openfermionprojectq import *

from openfermion.hamiltonians import MolecularData
from openfermion.transforms import jordan_wigner
from openfermion.utils import uccsd_singlet_paramsize

from projectq.ops import X, All, Measure
from projectq.backends import CommandPrinter, CircuitDrawer
```

Here we load $\textrm{H}_2$ from a precomputed molecule file found in the test data directory, and initialize the ProjectQ circuit compiler to a standard setting that uses a first-order Trotter decomposition to break up the exponentials of non-commuting operators.

```python
# Load the molecule.
filename = os.path.join(DATA_DIRECTORY, 'H2_sto-3g_singlet_0.7414')
molecule = MolecularData(filename=filename)

# Use a Jordan-Wigner encoding, and compress to remove 0 imaginary components
qubit_hamiltonian = jordan_wigner(molecule.get_molecular_hamiltonian())
qubit_hamiltonian.compress()
compiler_engine = uccsd_trotter_engine()
```

The Variational Quantum Eigensolver (or VQE), works by parameterizing a wavefunction $| \Psi(\theta) \rangle$ through some quantum circuit, and minimzing the energy with respect to that angle, which is defined by

\begin{align}
E(\theta) = \langle \Psi(\theta)| H | \Psi(\theta) \rangle
\end{align}

To perform the VQE loop with a simple molecule, it helps to wrap the evaluation of the energy into a simple objective function that takes the parameters of the circuit and returns the energy.  Here we define that function using ProjectQ to handle the qubits and the simulation.

```python
def energy_objective(packed_amplitudes):
    """Evaluate the energy of a UCCSD singlet wavefunction with packed_amplitudes
    Args:
        packed_amplitudes(ndarray): Compact array that stores the unique
            amplitudes for a UCCSD singlet wavefunction.

    Returns:
        energy(float): Energy corresponding to the given amplitudes
    """
    os.environ["KMP_DUPLICATE_LIB_OK"]="TRUE"

    # Set Jordan-Wigner initial state with correct number of electrons
    wavefunction = compiler_engine.allocate_qureg(molecule.n_qubits)
    for i in range(molecule.n_electrons):
        X | wavefunction[i]

    # Build the circuit and act it on the wavefunction
    evolution_operator = uccsd_singlet_evolution(packed_amplitudes,
                                                 molecule.n_qubits,
                                                 molecule.n_electrons)
    evolution_operator | wavefunction
    compiler_engine.flush()

    # Evaluate the energy and reset wavefunction
    energy = compiler_engine.backend.get_expectation_value(qubit_hamiltonian, wavefunction)
    All(Measure) | wavefunction
    compiler_engine.flush()
    return energy
```

While we could plug this objective function into any optimizer, SciPy offers a convenient framework within the Python ecosystem.  We'll choose as starting amplitudes the classical CCSD values that can be loaded from the molecule if desired.  The optimal energy is found and compared to the exact values to verify that our simulation was successful.

```python
n_amplitudes = uccsd_singlet_paramsize(molecule.n_qubits, molecule.n_electrons)
initial_amplitudes = [0, 0.05677]
initial_energy = energy_objective(initial_amplitudes)

# Run VQE Optimization to find new CCSD parameters
opt_result = minimize(energy_objective, initial_amplitudes,
                      method="CG", options={'disp':True})

opt_energy, opt_amplitudes = opt_result.fun, opt_result.x
print("\nOptimal UCCSD Singlet Energy: {}".format(opt_energy))
print("Optimal UCCSD Singlet Amplitudes: {}".format(opt_amplitudes))
print("Classical CCSD Energy: {} Hartrees".format(molecule.ccsd_energy))
print("Exact FCI Energy: {} Hartrees".format(molecule.fci_energy))
print("Initial Energy of UCCSD with CCSD amplitudes: {} Hartrees".format(initial_energy))
```

    Optimization terminated successfully.
             Current function value: -1.137270
             Iterations: 1
             Function evaluations: 12
             Gradient evaluations: 3

    Optimal UCCSD Singlet Energy: -1.13727017463
    Optimal UCCSD Singlet Amplitudes: [ -1.03662149e-08   5.65340580e-02]
    Classical CCSD Energy: -1.13727017465 Hartrees
    Exact FCI Energy: -1.13727017463 Hartrees
    Initial Energy of UCCSD with CCSD amplitudes: -1.13726981456 Hartrees

As we can see, the optimization terminates extremely quickly because the classical coupled cluster amplitudes were (for this molecule) already optimal. We can now use ProjectQ to compile this simulation circuit to a set of two-body quanutm gates.

```python
compiler_engine = uccsd_trotter_engine(CommandPrinter())
wavefunction = compiler_engine.allocate_qureg(molecule.n_qubits)
for i in range(molecule.n_electrons):
    X | wavefunction[i]

# Build the circuit and act it on the wavefunction
evolution_operator = uccsd_singlet_evolution(opt_amplitudes,
                                             molecule.n_qubits,
                                             molecule.n_electrons)
evolution_operator | wavefunction
compiler_engine.flush()
```

    Allocate | Qureg[0]
    Allocate | Qureg[1]
    Allocate | Qureg[2]
    Allocate | Qureg[3]
    X | Qureg[0]
    X | Qureg[1]
    Rx(1.57079632679) | Qureg[1]
    H | Qureg[3]
    CX | ( Qureg[1], Qureg[2] )
    CX | ( Qureg[2], Qureg[3] )
    Rz(12.566370604) | Qureg[3]
    CX | ( Qureg[2], Qureg[3] )
    CX | ( Qureg[1], Qureg[2] )
    H | Qureg[3]
    Rx(10.9955742876) | Qureg[1]
    Rx(1.57079632679) | Qureg[0]
    H | Qureg[1]
    Rx(1.57079632679) | Qureg[2]
    Rx(1.57079632679) | Qureg[3]
    CX | ( Qureg[0], Qureg[1] )
    CX | ( Qureg[1], Qureg[2] )
    CX | ( Qureg[2], Qureg[3] )
    Rz(12.5381035853) | Qureg[3]
    CX | ( Qureg[2], Qureg[3] )
    CX | ( Qureg[1], Qureg[2] )
    CX | ( Qureg[0], Qureg[1] )
    Rx(10.9955742876) | Qureg[3]
    Rx(10.9955742876) | Qureg[2]
    H | Qureg[1]
    Rx(10.9955742876) | Qureg[0]
    H | Qureg[0]
    H | Qureg[1]
    Rx(1.57079632679) | Qureg[2]
    H | Qureg[3]
    CX | ( Qureg[0], Qureg[1] )
    CX | ( Qureg[1], Qureg[2] )
    CX | ( Qureg[2], Qureg[3] )
    Rz(12.5381035853) | Qureg[3]
    CX | ( Qureg[2], Qureg[3] )
    CX | ( Qureg[1], Qureg[2] )
    CX | ( Qureg[0], Qureg[1] )
    H | Qureg[3]
    Rx(10.9955742876) | Qureg[2]
    H | Qureg[1]
    H | Qureg[0]
    H | Qureg[1]
    Rx(1.57079632679) | Qureg[3]
    CX | ( Qureg[1], Qureg[2] )
    CX | ( Qureg[2], Qureg[3] )
    Rz(1.03662148687e-08) | Qureg[3]
    CX | ( Qureg[2], Qureg[3] )
    CX | ( Qureg[1], Qureg[2] )
    Rx(10.9955742876) | Qureg[3]
    H | Qureg[1]
    H | Qureg[0]
    Rx(1.57079632679) | Qureg[1]
    H | Qureg[2]
    H | Qureg[3]
    CX | ( Qureg[0], Qureg[1] )
    CX | ( Qureg[1], Qureg[2] )
    CX | ( Qureg[2], Qureg[3] )
    Rz(0.0282670290194) | Qureg[3]
    CX | ( Qureg[2], Qureg[3] )
    CX | ( Qureg[1], Qureg[2] )
    CX | ( Qureg[0], Qureg[1] )
    H | Qureg[3]
    H | Qureg[2]
    Rx(10.9955742876) | Qureg[1]
    H | Qureg[0]
    H | Qureg[0]
    H | Qureg[1]
    H | Qureg[2]
    Rx(1.57079632679) | Qureg[3]
    CX | ( Qureg[0], Qureg[1] )
    CX | ( Qureg[1], Qureg[2] )
    CX | ( Qureg[2], Qureg[3] )
    Rz(12.5381035853) | Qureg[3]
    CX | ( Qureg[2], Qureg[3] )
    CX | ( Qureg[1], Qureg[2] )
    CX | ( Qureg[0], Qureg[1] )
    Rx(10.9955742876) | Qureg[3]
    H | Qureg[2]
    H | Qureg[1]
    H | Qureg[0]
    Rx(1.57079632679) | Qureg[0]
    H | Qureg[1]
    H | Qureg[2]
    H | Qureg[3]
    CX | ( Qureg[0], Qureg[1] )
    CX | ( Qureg[1], Qureg[2] )
    CX | ( Qureg[2], Qureg[3] )
    Rz(0.0282670290194) | Qureg[3]
    CX | ( Qureg[2], Qureg[3] )
    CX | ( Qureg[1], Qureg[2] )
    CX | ( Qureg[0], Qureg[1] )
    H | Qureg[3]
    H | Qureg[2]
    H | Qureg[1]
    Rx(10.9955742876) | Qureg[0]
    Rx(1.57079632679) | Qureg[0]
    H | Qureg[2]
    CX | ( Qureg[0], Qureg[1] )
    CX | ( Qureg[1], Qureg[2] )
    Rz(12.566370604) | Qureg[2]
    CX | ( Qureg[1], Qureg[2] )
    CX | ( Qureg[0], Qureg[1] )
    H | Qureg[2]
    Rx(10.9955742876) | Qureg[0]
    H | Qureg[0]
    Rx(1.57079632679) | Qureg[2]
    CX | ( Qureg[0], Qureg[1] )
    CX | ( Qureg[1], Qureg[2] )
    Rz(1.03662148687e-08) | Qureg[2]
    CX | ( Qureg[1], Qureg[2] )
    CX | ( Qureg[0], Qureg[1] )
    Rx(10.9955742876) | Qureg[2]
    H | Qureg[0]
    H | Qureg[0]
    Rx(1.57079632679) | Qureg[1]
    Rx(1.57079632679) | Qureg[2]
    Rx(1.57079632679) | Qureg[3]
    CX | ( Qureg[0], Qureg[1] )
    CX | ( Qureg[1], Qureg[2] )
    CX | ( Qureg[2], Qureg[3] )
    Rz(12.5381035853) | Qureg[3]
    CX | ( Qureg[2], Qureg[3] )
    CX | ( Qureg[1], Qureg[2] )
    CX | ( Qureg[0], Qureg[1] )
    Rx(10.9955742876) | Qureg[3]
    Rx(10.9955742876) | Qureg[2]
    Rx(10.9955742876) | Qureg[1]
    H | Qureg[0]
    Rx(1.57079632679) | Qureg[0]
    Rx(1.57079632679) | Qureg[1]
    Rx(1.57079632679) | Qureg[2]
    H | Qureg[3]
    CX | ( Qureg[0], Qureg[1] )
    CX | ( Qureg[1], Qureg[2] )
    CX | ( Qureg[2], Qureg[3] )
    Rz(0.0282670290194) | Qureg[3]
    CX | ( Qureg[2], Qureg[3] )
    CX | ( Qureg[1], Qureg[2] )
    CX | ( Qureg[0], Qureg[1] )
    H | Qureg[3]
    Rx(10.9955742876) | Qureg[2]
    Rx(10.9955742876) | Qureg[1]
    Rx(10.9955742876) | Qureg[0]
    Rx(1.57079632679) | Qureg[0]
    Rx(1.57079632679) | Qureg[1]
    H | Qureg[2]
    Rx(1.57079632679) | Qureg[3]
    CX | ( Qureg[0], Qureg[1] )
    CX | ( Qureg[1], Qureg[2] )
    CX | ( Qureg[2], Qureg[3] )
    Rz(0.0282670290194) | Qureg[3]
    CX | ( Qureg[2], Qureg[3] )
    CX | ( Qureg[1], Qureg[2] )
    CX | ( Qureg[0], Qureg[1] )
    Rx(10.9955742876) | Qureg[3]
    H | Qureg[2]
    Rx(10.9955742876) | Qureg[1]
    Rx(10.9955742876) | Qureg[0]


# A Comparison of Basis Expansions in Regression

Beginners in machine learning often write off regression methods after learning of more exotic algorithms like Boosting, Random Forests, and Support Vector machines; why bother with a *linear* method when powerful non-parametric methods are readily avalable?

It's easy to point out that the *linear* in linear regression is not meant to convey that the resulting model predictions are linear in the raw *feaure*, just the estimated parameters!  The modeler can certainly capture non-linearities in thier regression, they only need to transform the raw predictors!

A common response is that it is error prone and  annoying work to explore data by hand and somehow divine correct transformations of predictors: other methods do it automatically.

Ususally the only truly flexible method beginners learn to capture non-linearities in regression is polynomial regression, which is a real shame, as it is about the worst performing method available.

The purpose of this post is to spread awareness of better options for capturing non-lineararities in regression models, we would like to advocate more widespread adoption of linear and cubic splines.

### Acknowledgements

The idea for this post is based on my answer to [hxd1011](https://stats.stackexchange.com/users/113777/hxd1011)s [question regarding grouping vs. splines](https://stats.stackexchange.com/questions/230750/when-should-we-discretize-bin-continuous-independent-variables-features-and-when) at CrossValidated.

### Software

I've taken the oppurtunity to write a small [python module](https://github.com/madrury/basis-expansions) that is useful for using the basis expansions in this post here.  It conforms to the sklearn transfrmation interface, so can be used in pipelines and other high level processes in sklearn.

## Basis Expansions in Regression

To capture non-linearities in regression models, we need to transform some or all of the predictors.  To avoid having to treat every predictor as a special case needing investigation, we would like some way of applying a very general *family* of transformations to our predictors, which is flexible enough to adapt (when the model is fit) to a wide variety of shapes.

This takes the general form of a *basis expansion*.  Basis here is used in the linear algebraic sense: a linearly independent set of objects.  In this case our objects are *functions*:

$$ B = f_1, f_2, \ldots, f_k $$

and we create new sets of features by applying every function in our basis to the given feature:

$$ f_1(x), \ f_2(x), \ \ldots, \ f_k(x) $$

### Polynomial Expansion

The most commonly, and often *only*, example taught in introductory modeling courses or textbooks is **polynomial regression**.

In polynomial regression we choose as our basis a set of polynomial terms of increasing degree:

$$ f_1(x) = x, \ f_2(x) = x^2, \ \ldots, \ f_r(x) = x^r $$

This allows us to fit *polynomial* curves to features:

Unfortunately, polynomial regression has a fair number of issues.  The most often observed is a very high varaince (sensitivity to data), especially near the boundaries of the data:

Above we have a fixed data set, and we have fit and plotted polynomial regressions of various degrees.  The most striking feature is how badly the higher degree polynomials fit the data near the edges.  The variance explodes!  This is especially problamatic in high dimensional situations where almost *all* of the data is near the boundaries, due to the curse of dimensionality.

Another way to look at this is to plot residuals for each data point $x$ over many samples from the same population, as we vary the degree of the polynomial we fit to the data:

Here we see the same pattern from earlier, the instability in our fits starts at the edges of the data, and moves inward as we increase the degree.

Another final way to observe this effect is to estimate the average testing error of polynomial regressions fit repretedly to the same population as the degree is changed:

The polynomial regression eventually drasticly overfits, even to this simple one dimensional data set.

There are other issues with polynomial regression; for example, it is inherently non-local, changing the value of $y$ at one point in the training set can affect the fit of the polynomial at data points very far away, resulting in tight coupling across the space of our data (often referred to as *rigidity*).  You can get a feel for this by playing around with the interactive [scatterplot somoothers](http://madrury.github.io/smoothers/) app hoseted on this site.

The methods we will lay out in the rest of this post will go some way to alleviate these issues with polynomial regression, and serve as superior solutions to the same underlying problems.

### Binning Expansion

Probably the first thing that occurs to most modelers when reflecting on other ways to capture non-linear effects in regression is to **bin** the predictor varaible:

In binned regression we simply cut the range of the predictor varaible into equally sized intervals (though we could use a more sophisticated rule, like cutting into intervals at percentiles of the marginal distribution of the predictor).  Membership in any interval is used to create a set of indicator variables, which are then regressed upon.  In the one predictor case, this results in our regression predicting the mean value of $y$ in each bin.

Binning has its obvious [conceptual issues](https://stats.stackexchange.com/questions/68834/what-is-the-benefit-of-breaking-up-a-continuous-predictor-variable).  Most prominently, we expect most phenomina we study to vary continuously with inputs.  Binned regression does not create continuous functions of the predictor, so in most cases we would expect there to be some unavoidable bias within each bin.  In the simple case where the true relationship is monotonic in an interval, we expect to be underpredicting the truth on the left hand side of each bin, and on the right hand side we are expect to overpredict.

Even so, binning is popular.  It is easy to discover, implement, and explain, and often does a good enough job of capturing the non-linear behaviour of the predictor response relationship.  There are better options though, we will see later that other methods are both more cenceptually appealing, and do a better job of capturing the predictive power in the data with less estiamted parameters.

### Piecewise Linear Splines

As a first step towards a general non-parameteric *continuous* basis expansion, we would like to fit a **piecewise linear funtion** to our data.  This turns out the be rather easy to do using translations of the template function $f(x) = max(0, x)$

Taking linear combinations of these functions, we can create many piecewise linear shapes

If we choose a fixed set of points where we allow the slope of the linear segment to change, then we can use these as basis functions in our regression:

$$ \begin{align}
   f_0(x) &= x \\
   f_1(x) &= max(0, x - k_0)\\
   f_2(x) &= max(0, x - k_1)\\
   &\vdots \\
   f_r(x) &= max(0, x - k_r)
\end{align}$$

The result will be to fit a continuous piecewise linear function to our data:

The values $\{k_1, k_2, \ldots, k_r\}$, which are the $x$ coordinates at which the slope of the segments may change, are called **knots**.

Note that we include the basis function $f_0(x) = x$ which allows the spline to assume some non-zero slope *before* the first knot.  If we had not included this basis function, we would have been forced to use a zero slope in the interval $(\infty, k_1)$.

The estimated parameters for the basis function elements have a simple interpretation, they represent the *change in slope* when crossing from the left hand to the right hand side fo a knot.

Fitting piecewise linear curves instead of polynomials prevents the explosion of varaince when estimating a large number of parameters:

We see that as the number of knots increases, the linear spline *does* begin to overfit, but much more slowly than the polynomial regression with the sampe number of parameters.

The varaince does not tend to accumulate in any one area (as it did in the polynomial case, always accumulating near the boundaries of the data).  Instead the pockets of high varaiance are dependent on the specific data we happen to be fitting to

### Natural Cubic Splines

Our final example of a basis expansion will be **natural cubic splines**.  Cubic splines are motivated by the philosophy that most phenomina we would like to study vary as a *smooth* function of thier inputs.  While the linear splines changed abruptly at the knots, we would like some way to fit curves to our data where the dependence is less violent.

A **natural cubic spline** is a curve that is continuous, has countinuous first derivatives (the tangent line makes no abrupt changes), continuous second derivatives (the tangent lines rate of rotation makes no abrupt changes); and it equal to a cubic polynomial except a points where it is allowed to high higher order discontinuities, these points are the **knots**.  Additionally, we require that the curve be *linear* beyond the knots, i.e. to the left of the first knot, and right of the final knot.

We will skip writing down the exact form of the basis functions for natural cubic spline, the interested reader can consult [The Elements of Statistical Learning](http://web.stanford.edu/~hastie/ElemStatLearn/) or the [source code](https://github.com/madrury/basis-expansions/blob/master/basis_expansions.py#L205) used in this post.

All thought they seem more complex, it is important to realize that the continuity constraints on the shape of the spline are strong.  Whlie a estimating a piecewise linear spline with $r$ knots uses $r+1$ parameters (ignoring the intercept), estiamting a cubic spline takes only $r$ parameters.  Indeed, in the above picture, the spline with only one knot is a *line*; this is because the slopes of the left hand and right hand line must match.

The linear constraints near the edge of the data are intended to prevent the spline for overfitting near the boundaries of the data, as polynomial regression tends to do:

## Experiments

We describe below four experiments comparing the bahaviour of the basis expansions above.  In each experiment we fit a regression to a dataset using one of the above expansions.  The degrees of freedom (number of estimated parameters) was varied, and the preformance of the fit was measured on a held out dataset[^1].  This procedure what then repeated many times[^2] for each expansion and degree of freedom, and the average and varaince of the hold out error was computed.

Our four experiments each fit to datasets sampled from a different true signal:
  - A line plus random gaussian noise.
  - A sin curve plus random gaussian noise.
  - A weird continuous curve of mostly random design, plus random gaussian noise.
  - A sin curve with random discontinuities added, plus random gaussian noise.

### Fitting to a Noisy Line

As a first esperiment, let's fit to linear truth.  This is the simplest possible case for our regression methods.

Let's look at how the training and testing error varies as we increase the number of fit parameters for each of our basis expansions

The polynomial expansion eventually overfits **drastically**, while the other tree transformations only overfit gradually.  None of the models have any bias, so the minimum expected testing error occurs for the minimum number of fit parameters.

The binning, piecewise linear, and piecewise cubic transformations do about equally well in terms of average testing error throughout, and seem to have about the same varaition in testing error (i.e. the bands around the testing error curves area ll approximately the same width).

We can get a clearer comparison by plotting only the testing errors all on the same axis:

Between two and fifteen fit parameters, all the methods but polynomial regression perform about eqaully well, and overfit the training data at the same rate.  So in this simple case, if we are only going for test performance, it doesnt so much matter what we choose.

To examine the varaince of the methods, we can plot the width of the varaince bands all on the same axis:

In this example the variance of the testing error stays the same for all basis exapansions throughout the entire range of complexity, with the obvious exception of polynomial regression, where the varaince explodes along with the expectation.

### Fitting to a Noisy Sinusoid

Our next experiment will fit the various basis expansions to a sinusoidal signal:

The training / testing error plots are created in the same way as for the linear signal:

The most noticable difference in this sitation is that at a low number of estimated parameters, all the models are badly biased: while the fits to the linear function started at their lowest testing and training errors, for the sinuousid each basis expansion regression starts with high error, and decreases quickly as more parameters enter the model and the bias decreases.

The rates of overfitting tell the same story as before.  The polynomial regression eventaully overfits drastically, while the others overfit slowly.

Plotting the hold out error curves all on the same plot highlights another interesting feature to the story:

The binned regression fits to the data much more slowly than polynomial and the two splines.  While the splines achive thier lowest average testing error at around five estimated parameters, it takes around eighteen bins to minimize the average testing error.  Furthermore, the minimum testing error for the polynomial and spline regressions is *lower* than for the bins (and again, this minimum is achieved with *less* estimated parameters).

A similar story holds for the testing error varainces: the binning transformation has slightly higher varaince than the two spline methods.

### Fitting to a Noisy Weird Curve

Our next example is a weird function constructed by hand to have an unusual shape

```python
def weird_signal(x):
    return (x*x*x*(x-1)
            + 2*(1/(1 + np.exp(-0.5*(x - 0.5))))
            - 3.5*(x > 0.2)*(x < 0.5)*(x - 0.2)*(x - 0.5)
            - 0.95)
```

The graph of this function has exponential, quadratic, and linear qualities in different segments of its domain.

We follow the usual experiment at this point, here are the training and testing curves for the various basis expansion methods we are investigating.

The story here captures the same general themes as in the previous example:

  - The quadratic basis expansion eventually drastically overfits.
  - The other basis expansions overfit slowly.
  - For very a very small number of estiamted parameters, each of the basis expansions is biased.
  - The varaince in test error is generally stable as more parameters are added to the model, *except* for the polynomial regression, which explodes.

Comparing all the test errors on the same plot again highlights similar points to the sin curve

The bias of the binned regression decreases more slowly than the spline methods, and its minimum hold out error is both larger than the splines, and achieved at more estiamted parameters.  The linear and cubic spline generally achieve a similar testing error for the same number of estimated parameters.

The varainces for the non-polynomial models are stable, with a hint that the binned regression has a larger overall varaince across a range of model complexities.

### Fitting to a Noisy Broken Sinusoid

For our final example, we try a discontinuous function.

```python
cutpoints = sorted(np.random.uniform(size=6))
def broken_sin_signal(x):
    return  (np.sin(2*np.pi*x)
             - (cutpoints[0] <= x)*(x <= cutpoints[2])
             - (cutpoints[1] <= x)*(x <= cutpoints[2])
             - 2*(cutpoints[3] <= x)*(x <= cutpoints[4]))
```

To create this function, we have taken the sin curve from before, and created discontinuities by shifting intervals of the graph up or down.  The result is a broken sin curve

Let's repreat our experiments and see if the discontinuities change the patterns we have been observing among the different basis expansions

The major difference in this example is the difficulty in decreasing the bias of the fit.  Since the models we are fitting are all continuous, it is *impossible* for any of them to be unbiased.  The best we can do is trap the discontinuities between two ever closer knots (in the case of splines), or inside a small bin.  Indeed, even at 29 estiamted parameters, the test error is still decreasing (slowly), indicating the our models can still do a better job decreasing the bias of thier fits before becoming overhelmed with varaince.

Otherwise, there are no striking differneces to the patterns we observe when comparing the various basis expansions.

Comapring the testing errors of the various methods, this time the binned expansions does noticably worse than the other methods throughout the entire range of complexities.  Otherwise, the two spline models are mostly equivelent in performance.

Finally, the variance of the binned model is also significantly worse than the other methods, while the two spline methodologies are equivelent.

## Conclusions

In this post we have compared four different approches to capturing general non-linear behaviour in regression.

### Binning

Binning, which conceptualy simple, was seen to suffer from a few issues in comparison to the other methods

  - With a small number of estiamted parameters, the binned regression suffered from a higher bias than its competitors.  It was seen across multiple experiments to achive its minimal hold out error at a larger number of estiamted parameters than the other methods, and furthermore, often the minimal error achieved by the binning method was *larger* than the minimum achieved by the other methods.
  - The binned regression's hold out error often had a higher varaince than the other method's.  This means that, even if it performed just as well as another method *on average*, any *individual* binned regression is less trustrowthy than if using another method.

Additionally, the binned regression method has the disadvantage of producing *discontinuous* functions, while we expect most processes we encounter in nature or business to vary continuously in thier inputs.  This is philosophically unnapealling, and also accounts for some of the bias seen when comparing the binning regressions to the other basis expansions.

### Polynomial Regression

The major problem with polynomial regression is its instability once a moderate number of estimated parameters is reached.  Polynomial regressions as *highly* sensitive to the noise in our data, and in every example were seen to eventuall overfit violently; this feature was not seen in any of the other methods we surveyed.

The polynomial regression also does not accumulate varaince uniformly throughout the support of the data.  We saw that polynomial fits become highly unstable near the boundaries of the available data.  This can be a sweious issue in higher dimensional situations (though we did not explore this), where the curse of dimensionality tells us that the *majority* of our data will lie close to the boundary.

### Linear and Cubic Splines

These methods performed well across the range of our experiments.

  - They overfit slowly, especially compared to polynomial regression.
  - They generally achive a lower hold out error than the binned regressions, and achive thier minimum hold out error at a lower number of estimated parameters.
  - Their hold out error is generally the lowest out of the methods we surveyed, making each *individual* regression more trustworthy.

The two spline models are a clear winner over the binned and polynomial regressions.  This is a real shame, because they are often not taught in introductory courses in predictive modeling, even though they (at least the piecewise linear spline) are just as easy to implement as polynomial or binned regression.

We hope we have convinced you to use these methods in your own modeling work. The author sees no real reason to fit polynomial or binned expansions in any predictive or inferential model.


## Cálculo I Para Ingeniería en Sistemas

## Proyecto Final

### Estudiantes:

 - Alvaro Alfaro Santiesteban
 - Francela Carranza Pacheco
 - Chien Ting Lin

### Profesor:
Jose Antonio Otarola Cambronero
### II Cuatrimestre, 2018

# Derivadas de orden superior

## Justificación

La presente investigación se enfocará en estudiar la representación gráfica de las derivadas de orden superior, esto debido a que muchas veces la representación gráfica de un elemento matemático nos ayuda a entender mejor el problea que estamos resolviendo. Así, el presente trabajo permitiría mostrar los cambios que presentan las diferentes gráficas de funciones al resolverse sus respectivas derivadas, y profundizar los conocimientos teóricos sobre sobre caálculo diferencial, además de ofrecer una mirada mas enfocada al mundo fisico y no a la abstracción matemática.

# Objetivos

### Objetivo General:

Facilitar el entendimiento del tema de derivadas de orden superior, pudiendo ver la recta de la función junto con sus derivadas.

Para este proyecto hemos de utilizar el lenguaje de programación Python y las siguientes librerías:
 - NumPy: Para la generación de N cantidad números equidistantes en un rango específico
 - Plotly: Para crear las visualizaciones interactivas
 - SymPy: Esta librería nos permite resolver ecuaciones de una manera sencilla

### Objetivos específicos:

1. Mostrar las fórmulas y los ejemplos de las derivadas de orden superior para comprender la materia que hemos visto durante el curso
2. Definir de una manera mas gráfica el concepto de la derivada de orden superior

# Marco Teórico

Los orígenes del Cálculo estuvieron motivados por el deseo de resolver diversos problemas vinculados al movimiento de los cuerpos, simplificando, podemos destacar que dentro de los problemas principales esta el determinar la tangente a una curva en un punto (el problema de las tangentes).
El concepto de derivada es el que permite resolver satisfactoriamente dicho problema. La derivada, no se formuló sino hasta el siglo XVII. Fue el descubrimiento efectuado por Sir Isaac Newton (1642 - 1727) lo que inició el magnífico desarrollo del Cálculo diferencial

### Concepto de derivada

Para entender los resultados del Cálculo diferencial es necesario, antes que nada, comprender la idea básica del mismo: el concepto de derivada. La derivada de una función puede interpretarse geométricamente como la pendiente de una curva, y físicamente como una razón “instantánea” de cambio.

La definición matemática de una derivada es la siguiente:

$\displaystyle{\lim_{h \rightarrow{0}}{\frac{f'(x+h)-f'(x)}{h}}}$

### Derivadas de orden superior

Para una función cualquiera $f$, al calcular su derivada, obtenemos una nueva función $f'$ a la que se le puede calcular su derivada

La función $(f')'$ se suele escrbir $f''$ y recibe el nombre de segunda derivada de $f$. Si $f''$ existe, se dice que, $f$ es dos veces derivable.

De manera similar podemos definir $(f'')'$ = $f'''$ o $f^3$

La notación usual es:
$f' = f^1, f'' = f^2, f''' = f^3, ..., f^{k+1} = (f^k)'$

Las distintas funciones $f^k$ para $k ≥ 2$ son llamadas derivadas de orden superior.

Por ejemplo podemos decir que dada la función $f(x)=2x^4-3x+3$ sus derivadas son:

$f'(x)=8x^3-3$
$f''(x)=24x^2$
$f'''(x)=48x$

Y su representacion gráfica sería la siguiente:

```python
plot(2*x**4-3*x+3, derivatives=3, xlower_limit=-2, xupper_limit=2)
```

### Utilizando Python para graficar las derivadas de orden superior
Motivados por el hecho de que la mayoría de los cursos de matemática se centran en la parte abstracta de como resolver los problemas y que las herramientas para generación de gráficos que existen actualmente son muchas veces muy complejas de utilizar, nos dimos a la tarea de crear un programa en el que fuera mas sencillo crear visualizaciones. Debido a que nuestra investigación esta centrada en las derivadas de orden superior el programa está diseñado para grafica las mismas

Se detalla el código a continuación:

```python
import plotly
import numpy as np
from sympy import *
import plotly.graph_objs as go

x = Symbol('x')

def plot(function, *, derivatives=1, xlower_limit=-10, xupper_limit=10, ylower_limit=0, yupper_limit=0, points=1000):
    """Función para graficar derivadas de orden superior dada una función f(x).

    Keyword arguments:
    ------------------
    function -- Cadena de texto que contiene f(x)
    derivatives -- Cantidad de derivadas a graficar (default 1)
    xlower_limit -- Límite inferior del eje X (default -10)
    xupper_limit -- Límite superior del eje X (default 10)
    ylower_limit -- Límite inferior del eje Y (default 0)
    yupper_limit -- Límite superior del eje y (default 0)
    points -- Cantidad de puntos a generar en el eje X (default 1000)
    """

    #Generar valores equidistantes para el eje X
    x_values = np.linspace(xlower_limit, xupper_limit, points)

    #Definir límites de visualización para el eje Y
    if ylower_limit == 0 and yupper_limit == 0:
        yaxis_dict = dict(autorange = True)
    else:
        yaxis_dict = dict(range = [ylower_limit, yupper_limit])

    #Definir el diseño para la gráfica
    layout_dict = dict(
        layout = go.Layout(
            title="Funcion y derivadas de orden superior",
            yaxis=yaxis_dict
        ))

    #Ciclo que calcula las derivadas y las agrega a la variable data
    data = dict(data = list())
    for n in range(derivatives + 1):
        f = lambdify(x, function, 'numpy')
        fx = f(x_values) if type(f(x_values)) is not int else np.full(points, f(x_values))

        trace = go.Scatter(
            x=x_values,
            y=fx,
            name=f'''f{"'"*n}(x)={function}'''
            )

        data['data'].append(trace)
        #Calcular la siguiente derivada
        function = function.diff(x)

    #Mostrar el gráfico
    plotly.offline.iplot({**data, **layout_dict})
```

    /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
      return f(*args, **kwds)
    /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
      return f(*args, **kwds)

Como puede notarse en el código anterior se pueden definir tanto los límites para ambos ejes ($X$ y $Y$) como la cantidad de puntos a generar en el eje $X$ y la cantidad de derivadas que queremos graficar. A continuación un ejemplo en el que vamos a utilizar todos los parámetros:

 - Graficar 5 derivadas
 - Con un eje $X$ entre los valores -$5$ y $5$
 - Va a contener 1500 puntos equidistantes en el eje $X$
 - Un eje $Y$ entre los valores $-250$ y $250$
 - La función será: $4x^4+2x^3-12\sin(x)^2$

```python
plot(4*x**4+2*x**3-12*sin(x)**2, derivatives=5, xlower_limit=-5, xupper_limit=5, ylower_limit=-250, yupper_limit=250, points=1500)
```

# Conclusión
Como resultado de la investigación matemática presentada, se pretende demostrar que es más fácil comprender las teorias matemáticas cuando se tiene una herramienta para la generación de gráficas ya que permite poner ua representación del mundo real a un problema abstracto

## Bibliografía:
 - Beltrán Beltrán J.C. Cálculo de derivadas de orden superior. Recuperado de: http://www.calculo.jcbmat.com/id444.htm
 - Stewart, James (2002). Calculus (5ª edición). Brooks Cole. ISBN 978-0-534-39339-7
 - Plotly User Guide in Python. Recuperado de: https://plot.ly/python/user-guide/
 - numpy.linspace. NumPy v1.14 Manual. Recuperado de: https://docs.scipy.org/doc/numpy-1.14.0/reference/generated/numpy.linspace.html
 - SymPy documentation. Recuperado de: http://docs.sympy.org/latest/index.html
 - Derivadas de orden superior. Recuperado de: https://www.aprendematematicas.org.mx/unit/derivadas-orden-superior/
 - Cálculo diferencial e integral. Javier Pérez. Recuperado de: https://www.uam.es/personal_pdi/ciencias/fchamizo/asignaturas/calc1inf1011/apjperez/calculo_cap06.pdf


# Advanced usage

The standard functionality of the `sensemakr`, demonstrated in the previous section, will suﬃce for most users, most of the time. When needed, more ﬂexibility can be obtained by employing the sensitivity functions of the package directly, such as: (i) functions for sensitivity plots (`ovb_contour_plot`, `ovb_extreme_plot`); (ii) functions for computing bias-adjusted estimates and t-values (`adjusted_estimate`, `adjusted_t`); (iii) functions for computing the robustness value and partial R2 directly (`robustness_value`, `partial_r2`); and, (iv) functions for bounding the strength of unobserved confounders (`ovb_bounds`), among others. These functions are in the modules `ovb_plots`, `bias_functions`, `sensitivity_statistics`, and `ovb_bounds` but these are also automatically imported when import the package.

In this notebook we demonstrate how to use these functions in two examples.

## The risks of informal benchmarking

Informal "benchmarking" procedures have been widely suggested in the sensitivity analysis literature as a means to aid interpretation. It intends to describe how an unobserved confounder $Z$ "not unlike" some observed covariate $X_j$ would alter the results of a study. [Cinelli and Hazlett (2020)](https://doi.org/10.1111/rssb.12348) show why these proposals may lead users to erroneous conclusions, and offer formal bounds on the bias that could be produced by unobserved confounding "as strong" as certain observed covariates.

Here we use `sensemakr` to replicate the example in Section 6.1 of [Cinelli and Hazlett (2020)](https://doi.org/10.1111/rssb.12348). This example provides a useful tutorial on how users can construct their own sensitivity contour plots with customized bounds, beyond what is offered by default on the package.

### Simulating the data

We begin by simulating the data generating process used in our example. Consider a treatment variable $D$, an outcome variable $Y$, one observed confounder $X$, and one *unobserved* confounder $Z$. Again, all disturbance variables $U$ are standardized mutually independent gaussians, and note that, in reality, the treatment $D$ has *no causal effect* on the outcome $Y$.

\begin{align}
Z &= U_{z}\\
X &= U_{x}\\
D &= X + Z + U_d\\
Y &= X + Z + U_y
\end{align}

Also note that, in this model: (i) the unobserved confounder $Z$ is independent of $X$; and, (ii) the unobserved confounder $Z$ is *exactly like* $X$ in terms of its strength of association with the treatment and the outcome. The code below creates a sample of size 100 of this data generating process. We make sure to create residuals that are standardized and orthogonal so that all properties that we describe here will hold exactly even in this finite sample.

```python
# imports statsmodels, numpy, and pands
import statsmodels.formula.api as smf
import numpy as np
import pandas as pd

# defines function to simulate orthogonal residuals
def resid_maker(n,df):
    N = np.random.normal(0,1,n)
    form = 'N~' +'+'.join(df.columns)
    df['N'] = N
    model = smf.ols(formula=form,data=df).fit()
    e = model.resid
    e = (e-np.mean(e))/np.std(e)
    return(e)

# simulates data
n = 100
X = np.random.normal(0,1,n)
X = (X - np.mean(X))/np.std(X)
Z = resid_maker(n,pd.DataFrame({'x':X}))
D = X + Z + resid_maker(n,pd.DataFrame({'x':X,'z':Z}))
Y = X + Z + resid_maker(n,pd.DataFrame({'x':X,'z':Z,'d':D}))
```

### Fitting the model

In this example, the investigator does not observe the confounder $Z$. Therefore, she is forced to fit the restricted linear model $Y \sim D + X$, resulting in the following estimated values

```python
# creates data.frame with the data
df = pd.DataFrame({'Y':Y,'X':X,'Z':Z,'D':D})

# runs OLS
model_ydx = smf.ols(formula='Y~D+X',data=df).fit()

# prints OLS summary
model_ydx.summary()
```

OLS Regression Results

  Dep. Variable:	            Y	          R-squared:         	    0.500

  Model:	                   OLS	         Adj. R-squared:    	    0.490

  Method:	             Least Squares	    F-statistic:       	    48.50

  Date:	             Wed, 26 Jan 2022	   Prob (F-statistic):	 2.51e-15

  Time:	                 21:12:01	       Log-Likelihood:    	  -162.17

  No. Observations:	         100	        AIC:               	    330.3

  Df Residuals:	              97	        BIC:               	    338.1

  Df Model:	                   2

  Covariance Type:	      nonrobust

      	         coef	     std err	      t	      P>|t|	  [0.025	    0.975]

  Intercept	          0	     0.124	         0	  1.000	    -0.247	     0.247

  D	             0.5000	     0.088	     5.686	  0.000	     0.325	     0.675

  X	             0.5000	     0.152	     3.283	  0.001	     0.198	     0.802

  Omnibus:	        3.229	   Durbin-Watson:     	    1.722

  Prob(Omnibus):	  0.199	   Jarque-Bera (JB):  	    2.614

  Skew:	           0.290	   Prob(JB):          	    0.271

  Kurtosis:	       3.539	   Cond. No.          	     2.41

Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

Note we obtain a large and statistically significant coefficient estimate of the effect of $X$ on $Y$ ($0.5$). However, we know that the variable $Z$ is not observed, and there is the fear that this estimated effect is in fact due to the bias caused by $Z$. On the other hand, let us suppose the investigator correctly knows that: (i) $Z$ and $X$ have the same strength of association with $D$ and $Y$; and, (ii) $Z$ is independent of $X$. Can we leverage this information to understand how much bias a confounder $Z$ "not unlike" $X$ could cause?

### Informal benchmarks

Computing the bias due to the omission of $Z$ requires two sensitivity parameters: its partial $R^2$ with the treatment $D$ and its partial $R^2$ with the outcome $Y$. How could we go about computing the bias that a confounder $Z$ "not unlike $X$" would cause?

Intuitively, it seems that we could take as reference the observed partial $R^2$ of $X$ with $D$ and $Y$, and use those as the plausible values for the sensitivity parameters. So let us now compute those observed partial $R^2$ using the `partial_r2()`. For the partial $R^2$ of $X$ with the treatment, we also need to fit a treatment regression $D \sim X$ first.

```python
# import sensemakr
import sensemakr as smkr

# fits treatment regression
model_dx = smf.ols(formula = 'D~X', data = df).fit()

# computes observed partial R2 of X with D and Y
r2dx   = smkr.partial_r2(model_dx,  covariates='X')
r2yx_d = smkr.partial_r2(model_ydx, covariates='X')
```

Once both partial $R^2$ are computed, we can determine the implied adjusted estimate due to an unobserved confounder $Z$ using the `adjusted_estimate()`.

```python
# computes adjusted estimate
informal_adjusted_estimate = smkr.adjusted_estimate(model     = model_ydx,
                                                    treatment = "D",
                                                    r2dz_x    = r2dx,
                                                    r2yz_dx   = r2yx_d)
```

We can now plot the sensitivity contours with `ovb_contour_plot()` and add our informal benchmark with `add_bound_to_contour()`.

```python
# draws sensitivity contours
smkr.ovb_contour_plot(model = model_ydx, treatment = "D", lim = 0.6)

# adds informal benchmark
smkr.add_bound_to_contour(r2dz_x      = r2dx,
                          r2yz_dx     = r2yx_d,
                          bound_value = informal_adjusted_estimate,
                          bound_label = 'Informal benchmark')
```

As we can see, the results of the informal benchmark are different from what we expected. The informal benchmark point is still far away from zero, and this would lead an investigator to incorrectly conclude that an unobserved confounder $Z$ "not unlike $X$" is not sufficient to explain away the observed effect. Moreover, this incorrect conclusion occurs despite *correctly* assuming that: (i) $Z$ and $X$ have the same strength of association with $D$ and $Y$; and, (ii) $Z$ is independent of $X$. See Section 6.1 of [Cinelli and Hazlett (2020)](https://doi.org/10.1111/rssb.12348) for details of why this happens.

### Formal bounds

We now show how to compute formal bounds using the function `ovb_bounds()`.

```python
# computes formal bounds
formal_bound = smkr.ovb_bounds(model = model_ydx,
                               treatment = 'D',
                               benchmark_covariates = "X",
                               kd = 1,
                               ky = 1)
```

In this function we specify the linear model being used (`model.ydx`), the treatment of interest ($D$), the observed variable used for benchmarking ($X$), and how stronger $Z$ is in explaining treatment (`kd`) and outcome (`ky`) variation, as compared to the benchmark variable $X$. We can now plot the proper bound against the informal benchmark.

```python
# contour plot contrasting informal vs formal bounds

smkr.ovb_contour_plot(model = model_ydx, treatment = "D",lim = 0.6)

smkr.add_bound_to_contour(r2dz_x  = r2dx,
                          r2yz_dx = r2yx_d,
                          bound_value = informal_adjusted_estimate,
                          bound_label = 'Informal benchmark')

smkr.add_bound_to_contour(bounds      = formal_bound,
                          bound_label = 'Proper bound')
```

Note that, using the formal bounds, the researcher now reaches the correct conclusion that, an unobserved confounder $Z$ similar to $X$ is strong enough to explain away all the observed association.

## Manually providing input data

We now demonstrate how to replicate the sensitivity analysis of the Darfur example (see Quickstart) without access to the microdata, using only usual statistics found in the regression tables. All results of the Darfur example, except the formal benchmark bounds as we explaim below, can be recoverded by simply providing: (i) the point estimate of `directlyharmed` ($0.097$); (ii) its estimated standard error ($0.023$); and, (ii) the degrees of freedom of the regression ($783$). This type of "manual" analysis can be useful when reading or reviewing papers that did not perform sensitivity.

```python
# computes the robustness value
smkr.robustness_value(t_statistic = 0.097/0.023, dof = 783)
```

    0    0.139787
    dtype: float64

```python
# computes the partial R2
smkr.partial_r2(t_statistic = 0.097/0.023, dof = 783)
```

    0.022211153497507175

```python
# computes what the adjusted estimate would have been
# after adjusting for confounders with strength r2dz_x = .1, r2yz_dx = 0.15
smkr.adjusted_estimate(estimate = 0.097, se = 0.023, dof = 783, r2dz_x = .1, r2yz_dx = 0.15)
```

    0.013912997406333186

```python
# computes what the adjusted t-value would have been
# after adjusting for confounders with strength r2dz_x = .1, r2yz_dx = 0.15
smkr.adjusted_t(estimate = 0.097, se = 0.023, dof = 783, r2dz_x = .1, r2yz_dx = 0.15)
```

    0.6220526656946492

```python
# contour plot of the point estimate
smkr.ovb_contour_plot(estimate = 0.097, se = 0.023, dof = 783)
```

```python
# contour plot of the t-value
smkr.ovb_contour_plot(estimate = 0.097, se = 0.023, dof = 783, sensitivity_of="t-value")
```

```python
# extreme plot
smkr.ovb_extreme_plot(estimate = 0.097, se = 0.023, dof = 783)
```

Finally, we show how users can compute bounds on the strength of confounding using only summary statistics, if the paper also provides a treatment regression table, i.e., a regression of the treatment on the observed covariates. Such regressions are sometimes shown in published works as part of efforts to describe the “determinants” of the treatment, or as “balance tests” in which the investigator assesses whether observed covariates predict treatment assigment. For the Darfur example, the regression coefficient and standard errors (in parenthesis) of `female` for the outcome and treatment equation are -0.097 (0.036) and -0.232 (0.024). With these, we can now compute the formal bounds as shown below.

```python
# computes partial R2 based on t-value of female
r2yxj_dx = smkr.partial_r2(t_statistic = -0.232/0.024, dof = 783)
r2dxj_x  = smkr.partial_r2(t_statistic = -0.097/0.036, dof = 783)

# computes bounds on the strength of Z if it were 1, 2 or 3 times as strong as female
bounds   = smkr.ovb_partial_r2_bound(r2dxj_x = r2dxj_x,
                                     r2yxj_dx = r2yxj_dx,
                                     kd = [1,2,3],
                                     ky = [1,2,3],
                                     benchmark_covariates="female")

bounds
```

      bound_label
      r2dz_x
      r2yz_dx

      0
      1x female
      0.009272
      0.121575

      1
      2x female
      0.018544
      0.243193

      2
      3x female
      0.027816
      0.364854

```python
# computes adjusted estimate based on bounds
bound_values = smkr.adjusted_estimate(estimate = 0.0973, se = 0.0232, dof = 783,
                                      r2dz_x= bounds['r2dz_x'], r2yz_dx=bounds['r2yz_dx'])

# draws contours
smkr.ovb_contour_plot(estimate = 0.097, se = 0.023, dof = 783)

# adds bounds
smkr.add_bound_to_contour(bounds = bounds, bound_value = bound_values)
```

## References

Cinelli, C. Hazlett, C. (2020) "Making Sense of Sensitivity: Extending Omitted Variable Bias". Journal of the Royal Statistical Society, Series B (Statistical Methodology). ( [link](https://doi.org/10.1111/rssb.12348) )


Trusted Notebook" width="500 px" align="left">

# _*Qiskit Aqua: Generating Random Variates*_

The latest version of this notebook is available on https://github.com/Qiskit/qiskit-tutorials.

***
### Contributors
Albert Akhriev[1], Jakub Marecek[1]

### Affliation
- [1]IBMQ

## Introduction

While classical computers use only pseudo-random routines, quantum computers
can generate true random variates.
For example, the measurement of a quantum superposition is intrinsically random,
as suggested by Born's rule.
Consequently, some of the
best random-number generators are based on such quantum-mechanical effects. (See the
Further, with a logarithmic amount of random bits, quantum computers can produce
linearly many more bits, which is known as
randomness expansion protocols.

In practical applications, one wishes to use random variates of well-known
distributions, rather than random bits.
In this notebook, we illustrate ways of generating random variates of several popular
distributions on IBM Q.

## Random Bits and the Bernoulli distribution

It is clear that there are many options for generating random bits (i.e., Bernoulli-distributed scalars, taking values either 0 or 1). Starting from a simple circuit such as a Hadamard gate followed by measurement, one can progress to vectors of Bernoulli-distributed elements. By addition of such random variates, we could get binomial distributions. By multiplication we could get geometric distributions, although perhaps leading to a circuit depth that may be impratical at the moment, though.

Let us start by importing the basic modules and creating a quantum circuit for generating random bits:

```python
import matplotlib.pyplot as plt
%matplotlib inline
import numpy as np
import sys, math, time
import warnings
warnings.filterwarnings("ignore", category=DeprecationWarning)

from qiskit import BasicAer
from qiskit import QuantumCircuit, QuantumRegister, ClassicalRegister, execute

# In this example we use 'qasm_simulator' backend.
glo_backend = BasicAer.get_backend("qasm_simulator")
```

In the next step we create a quantum circuit, which will be used for generation:

```python
# Number of qubits utilised simultaneously.
glo_num_qubits = 5

def create_circuit(num_target_qubits: int) -> QuantumCircuit:
    """
    Creates and returns quantum circuit for random variate generation.
    :param num_target_qubits: number of qubits to be used.
    :return: quantum curcuit.
    """
    assert isinstance(num_target_qubits, int) and num_target_qubits > 0
    q = QuantumRegister(num_target_qubits)
    c = ClassicalRegister(num_target_qubits)
    circuit = QuantumCircuit(q, c)
    circuit.h(q)
    circuit.barrier()
    circuit.measure(q, c)
    return circuit

# Create and plot generating quantum circuit.
circuit = create_circuit(glo_num_qubits)
#print(circuit)
circuit.draw(output='mpl')
```

## Uniformly-distributed scalars and vectors

It is clear that there are many options for approximating uniformly-distributed scalars by the choice of an integer from a finite range uniformly at random, e.g., by a binary-code construction from the Bernoulli-distributed vectors. In the following snippet, we generate random bits, which we then convert using the binary-code construction, up to the machine precision of a classical computer.

```python
def uniform_rand_float64(circuit: QuantumCircuit, num_target_qubits: int,
                         size: int, vmin: float, vmax: float) -> np.ndarray:
    """
    Generates a vector of random float64 values in the range [vmin, vmax].
    :param circuit: quantum circuit for random variate generation.
    :param num_target_qubits: number of qubits to be used.
    :param size: length of the vector.
    :param vmin: lower bound.
    :param vmax: upper bound.
    :return: vector of random values.
    """
    assert sys.maxsize == np.iinfo(np.int64).max    # sizeof(int) == 64 bits
    assert isinstance(size, int) and size > 0
    assert isinstance(vmin, float) and isinstance(vmax, float) and vmin <= vmax
    nbits = 7 * 8                                   # nbits > mantissa of float64
    bit_str_len = (nbits * size + num_target_qubits - 1) // num_target_qubits
    job = execute(circuit, glo_backend, shots=bit_str_len, memory=True)
    bit_str = ''.join(job.result().get_memory())
    scale = float(vmax - vmin) / float(2**nbits - 1)
    return np.array([vmin + scale * float(int(bit_str[i:i+nbits], 2))
                     for i in range(0, nbits * size, nbits)], dtype=np.float64)

def uniform_rand_int64(circuit: QuantumCircuit, num_target_qubits: int,
                       size: int, vmin: int, vmax: int) -> np.ndarray:
    """
    Generates a vector of random int64 values in the range [vmin, vmax].
    :param circuit: quantum circuit for random variate generation.
    :param num_target_qubits: number of qubits to be used.
    :param size: length of the vector.
    :param vmin: lower bound.
    :param vmax: upper bound.
    :return: vector of random values.
    """
    assert sys.maxsize == np.iinfo(np.int64).max        # sizeof(int) == 64 bits
    assert isinstance(size, int) and size > 0
    assert isinstance(vmin, int) and isinstance(vmax, int) and vmin <= vmax
    assert abs(vmin) <= 2**52 and abs(vmax) <= 2**52    # 52 == mantissa of float64
    return np.rint(uniform_rand_float64(circuit, num_target_qubits,
                                        size, float(vmin), float(vmax))).astype(np.int64)
```

### Uniform distribution over floating point numbers.
In this example we draw a random vector of floating-point values uniformly distributed within some arbitrary selected interval:

```python
# Draw a sample from uniform distribution.
start_time = time.time()
sample = uniform_rand_float64(circuit, glo_num_qubits, size=54321, vmin=-7.67, vmax=19.52)
sampling_time = time.time() - start_time

# Print out some details.
print("Uniform distribution over floating point numbers:")
print("  sample type:", type(sample), ", element type:", sample.dtype, ", shape:", sample.shape)
print("  sample min: {:.4f}, max: {:.4f}".format(np.amin(sample), np.amax(sample)))
print("  sampling time: {:.2f} secs".format(sampling_time))

# Plotting the distribution.
plt.hist(sample.ravel(),
         bins=min(int(np.ceil(np.sqrt(sample.size))), 100),
         density=True, facecolor='b', alpha=0.75)
plt.xlabel("value", size=12)
plt.ylabel("probability", size=12)
plt.title("Uniform distribution over float64 numbers in [{:.2f} ... {:.2f}]".format(
            np.amin(sample), np.amax(sample)), size=12)
plt.grid(True)
# plt.savefig("uniform_distrib_float.png", bbox_inches="tight")
plt.show()
```

### Uniform distribution over integers.
Our next example is similar to the previous one, but here we generate a random vector of integers:

```python
# Draw a sample from uniform distribution.
start_time = time.time()
sample = uniform_rand_int64(circuit, glo_num_qubits, size=54321, vmin=37, vmax=841)
sampling_time = time.time() - start_time

# Print out some details.
print("Uniform distribution over bounded integer numbers:")
print("  sample type:", type(sample), ", element type:", sample.dtype, ", shape:", sample.shape)
print("  sample min: {:d}, max: {:d}".format(np.amin(sample), np.amax(sample)))
print("  sampling time: {:.2f} secs".format(sampling_time))

# Plotting the distribution.
plt.hist(sample.ravel(),
         bins=min(int(np.ceil(np.sqrt(sample.size))), 100),
         density=True, facecolor='g', alpha=0.75)
plt.xlabel("value", size=12)
plt.ylabel("probability", size=12)
plt.title("Uniform distribution over int64 numbers in [{:d} ... {:d}]".format(
            np.amin(sample), np.amax(sample)), size=12)
plt.grid(True)
# plt.savefig("uniform_distrib_int.png", bbox_inches="tight")
plt.show()
```

## Normal distribution

To generate random variates with a standard normal distribution using two independent
samples $u_1, u_2$ of the uniform distribution on the unit interval [0, 1], one can
consider the Box-Muller transform to obtain a 2-vector:

\begin{align}
\begin{bmatrix}
%R\cos(\Theta )=
{\sqrt {-2\ln u_{1}}}\cos(2\pi u_{2}) \\
% R\sin(\Theta )=
{\sqrt {-2\ln u_{1}}}\sin(2\pi u_{2})
\end{bmatrix},
\end{align}

wherein we have two independent samples of the standard normal distribution.
In IBM Q, this is implemented as follows:

```python
def normal_rand_float64(circuit: QuantumCircuit, num_target_qubits: int,
                        size: int, mu: float, sigma: float) -> np.ndarray:
    """
    Draws a sample vector from the normal distribution given the mean and standard
    deviation, using the Box-Muller method.
    """
    TINY = np.sqrt(np.finfo(np.float64).tiny)
    assert isinstance(size, int) and size > 0
    rand_vec = np.zeros((size,), dtype=np.float64)

    # Generate array of uniformly distributed samples, factor 1.5 longer that
    # actually needed.
    n = (3 * size) // 2
    x = np.reshape(uniform_rand_float64(circuit, num_target_qubits,
                                        2*n, 0.0, 1.0), (-1, 2))

    x1 = 0.0                # first sample in a pair
    c = 0                   # counter
    for d in range(size):
        r2 = 2.0
        while r2 >= 1.0 or r2 < TINY:
            # Regenerate array of uniformly distributed samples upon shortage.
            if c >= n:
                c = 0
                n = max(size // 10, 1)
                x = np.reshape(uniform_rand_float64(circuit, num_target_qubits,
                                                    2*n, 0.0, 1.0), (-1, 2))

            x1 = 2.0 * x[c, 0] - 1.0        # first sample in a pair
            x2 = 2.0 * x[c, 1] - 1.0        # second sample in a pair
            r2 = x1 * x1 + x2 * x2
            c += 1

        f = np.sqrt(np.abs(-2.0 * np.log(r2) / r2))
        rand_vec[d] = f * x1

    return (rand_vec * sigma + mu)
```

The following example demonstrates how to draw a random vector of normally distributed variates:

```python
# Mean and standard deviation.
mu = 2.4
sigma = 5.1

# Draw a sample from the normal distribution.
start_time = time.time()
sample = normal_rand_float64(circuit, glo_num_qubits, size=4321, mu=mu, sigma=sigma)
sampling_time = time.time() - start_time

# Print out some details.
print("Normal distribution (mu={:.3f}, sigma={:.3f}):".format(mu, sigma))
print("  sample type:", type(sample), ", element type:", sample.dtype, ", shape:", sample.shape)
print("  sample min: {:.4f}, max: {:.4f}".format(np.amin(sample), np.amax(sample)))
print("  sampling time: {:.2f} secs".format(sampling_time))

# Plotting the distribution.
x = np.linspace(mu - 4.0 * sigma, mu + 4.0 * sigma, 1000)
analyt = np.exp(-0.5 * ((x - mu) / sigma)**2) / (sigma * math.sqrt(2.0 * math.pi))
plt.hist(sample.ravel(),
         bins=min(int(np.ceil(np.sqrt(sample.size))), 100),
         density=True, facecolor='r', alpha=0.75)
plt.plot(x, analyt, '-b', lw=1)
plt.xlabel("value", size=12)
plt.ylabel("probability", size=12)
plt.title("Normal distribution: empirical vs analytic", size=12)
plt.grid(True)
# plt.savefig("normal_distrib.png", bbox_inches="tight")
plt.show()
```

There is a substantial amount of further work needed to either certify the quality of the source of random numbers (cf. NIST SP 800-90B, Recommendation for the Entropy Sources Used for Random Bit Generation) or to use random variates within quantum algorithms (cf. uncertainty_models within Qiskit Aqua).

```python

```