### University of Washington: Machine Learning and Statistics

# Lecture 1:  Regression (linear, errors on variables, outliers)

Andrew Connolly and Stephen Portillo

##### Resources for this notebook include:
- [Textbook](https://press.princeton.edu/books/hardcover/9780691198309/statistics-data-mining-and-machine-learning-in-astronomy) Chapter 8.
- [astroML website](https://www.astroml.org/index.html)

This notebook is developed based on material from A. Connolly, Z. Ivezic, M. Juric, S. Portillo, G. Richards, B. Sipocz, J. VanderPlas, D. Hogg, and many others.

The notebook and assoociated material are available from [github](https://github.com/uw-astro/astr-598a-win22).

Make sure you are using the latest version of Jupyterlab (>3.0)
> pip install jupyterlab --upgrade

### Installing the latest (v1.0) of astroML from git

> pip install --pre -U astroml

## This notebook includes:

[Ordinary least squares method](#ordinaryLSQ)

[Total least squares method](#totalLSQ)

[Dealing witth Outliers](#outliers)

### Fitting a Line using a Maximum Likelihood Estimator

Assume the scatter in our measurements (the residuals) is generated by a gaussian process. I.e.:

>$ y_i = m x_i + b + r_i $

where $r_i$ is drawn from $N(0, \sigma)$. Here, $\sigma$ is the error the measurement induces.

Let us compute the likelihood. First, we ask ourselves what is the probability $p(y_i|x_i, M(a, b), \sigma)$ that a particular point $y_i$ would be measured (M is our model). It is just the normal distribution:

>$ p(y_i|x_i, M(m, b), \sigma) = N(y_i - M(x)|\sigma) = \frac{1}{\sqrt{2 \pi \sigma^2}} \exp \left( - \frac{(y_i - M(x_i))^2}{2 \sigma^2} \right) $.

We can write down the $\ln L$

>$ \ln L(m, b) = constant - \frac{1}{2 \sigma^2} \sum_{i=1}^N (y_i - M(x_i))^2 $

This is the expression that we now minimize with respect to $m$ and $b$ to find ML estimators for those parameters. This is equivalent to minimizing the sum of the squares or a _least-squares method_

## Ordinary least squares method
[Go to top](#toc)

### NOTE: We  suppress warnings for the packages (this is not recommended)

```python
def warn(*args, **kwargs):
    pass
import warnings
warnings.warn = warn
```

```python
import numpy as np
from matplotlib import pyplot as plt
from matplotlib.patches import Ellipse
import matplotlib
matplotlib.rc('text', usetex=False)
import seaborn as sns
from scipy import optimize

from astroML.linear_model import TLS_logL
from astroML.plotting import setup_text_plots
from astroML.plotting.mcmc import convert_to_stdev
setup_text_plots(fontsize=8, usetex=True)
# random seed
np.random.seed(42)
```

    WARNING (theano.tensor.blas): Using NumPy C-API based implementation for BLAS functions.

```python
# We'll use the data from Table 1 of Hogg et al. 2010
from astroML.datasets import fetch_hogg2010test
data = fetch_hogg2010test()
data = data[5:]  # no outliers (the first 5 points are outliers, discussed later)
x = data['x']
y = data['y']
sigma_x = data['sigma_x']
sigma_y = data['sigma_y']
rho_xy = data['rho_xy']
y_obs = y
```

```python
# Plot the data with y error bars
plt.errorbar(x, y, yerr=sigma_y, fmt=".k", capsize=0)
plt.xlabel('x', fontsize=18)
plt.ylabel('y', fontsize=18)
plt.xlim(0, 300)
plt.ylim(100, 600)
```

### We have data $y(x)$ and we want to fit this model (i.e. we want to obtain m and b):

$$\mathbf{y} = m \, \mathbf{x} + b$$

For this problem the maximum likelihood and full posterior probability distribution (under infinitely broad priors) for the slope and intercept of the line are known analytically. The analytic result for the posterior probability distribution is a 2-d Gaussian with mean

$$\mathbf{w} = \left(\begin{array}{c}
    m \\ b
\end{array}\right) = (\mathbf{A}^\mathrm{T}\,C^{-1}\mathbf{A})^{-1} \, \mathbf{A}^\mathrm{T}\,C^{-1}\,\mathbf{y}$$

and covariance matrix

$$\mathbf{V} = (\mathbf{A}^\mathrm{T}\,C^{-1}\mathbf{A})^{-1}$$

where

$$\mathbf{y} = \left(\begin{array}{c}
y_1 \\ y_2 \\ \vdots \\ y_N
\end{array}\right) \quad , \quad \mathbf{A} = \left(\begin{array}{cc}
x_1 & 1 \\ x_2 & 1 \\ \vdots & \vdots \\ x_N & 1
\end{array}\right) \quad,\, \mathrm{and} \quad
\mathbf{C} = \left(\begin{array}{cccc}
\sigma_1^2 & 0 & \cdots & 0 \\
0 & \sigma_2^2 & \cdots & 0 \\
&&\ddots& \\
0 & 0 & \cdots & \sigma_N^2
\end{array}\right)$$

Sometimes we call $A$ the design matrix. There are various functions in Python (and astroML/scikit-learn) for computing this but let's do it explicitly and step by step. With numpy, it only takes a few lines of code  - here it is:

```python
A = np.vander(x, 2)  # Take a look at the documentation to see what this function does!
                     # https://numpy.org/doc/stable/reference/generated/numpy.vander.html
ATA = np.dot(A.T, A / sigma_y[:, None]**2)
w = np.linalg.solve(ATA, np.dot(A.T, y / sigma_y**2))
V = np.linalg.inv(ATA)
```

Let's take a look and see what this prediction looks like.

To do this, we'll sample 99 slopes and intercepts from this 2-D Gaussian and overplot them on the data.

```python
plt.errorbar(x, y, yerr=sigma_y, fmt=".k", capsize=0)
xGrid = np.linspace(40, 240)
# note that we are drawing 99 lines here, with m and b randomly sampled from w and V
for m, b in np.random.multivariate_normal(w, V, size=99):
    plt.plot(xGrid, m*xGrid + b, "g", alpha=0.02)
plt.xlim(0, 250)
plt.ylim(100, 600)
plt.xlabel('x', fontsize=16)
plt.ylabel('y', fontsize=16)
```

```python
# let's visualize the covariance between m and b
a = np.random.multivariate_normal(w, V, size=2400)
plt.scatter(a[:,0], a[:,1], alpha=0.1)
plt.xlabel('m', fontsize=16)
plt.ylabel('b', fontsize=16)
```

### Another approach: the probabilistic model

In order to perform posterior inference on a model and dataset, we need a function that computes the value of the posterior probability given a proposed setting of the parameters of the model. For reasons that will become clear below, we actually only need to return a value that is *proportional* to the probability.

The posterior probability for parameters $\mathbf{w} = (m,\,b)$ conditioned on a dataset $\mathbf{y}$ is given by

$$p(\mathbf{w} \,|\, \mathbf{y}) = \frac{p(\mathbf{y} \,|\, \mathbf{w}) \, p(\mathbf{w})}{p(\mathbf{y})}$$

where $p(\mathbf{y} \,|\, \mathbf{w})$ is the *likelihood* and $p(\mathbf{w})$ is the *prior*. For this example, we're modeling the likelihood by assuming that the datapoints are independent with known Gaussian uncertainties $\sigma_n$. This specifies a likelihood function:

$$p(\mathbf{y} \,|\, \mathbf{w}) = \prod_{n=1}^N \frac{1}{\sqrt{2\,\pi\,\sigma_n^2}} \,
\exp \left(-\frac{[y_n - f_\mathbf{w}(x_n)]^2}{2\,\sigma_n^2}\right)$$

where $f_\mathbf{w}(x) = m\,x + b$ is the linear model.

For numerical reasons, we will acutally want to compute the logarithm of the likelihood. In this case, this becomes:

$$\ln p(\mathbf{y} \,|\, \mathbf{w}) = -\frac{1}{2}\sum_{n=1}^N \frac{[y_n - f_\mathbf{w}(x_n)]^2}{\sigma_n^2} + \mathrm{constant} \quad.$$

By maxmizing $p(\mathbf{y} \,|\, \mathbf{w})$, we obtain posterior probability distributions for $m$ and $b$.

### Using astroML (or scikit-learn) to estimate the MLE for the parameters

We use a standardized form for the regression
- Define the regression model (clf = LinearRegression())
- Fit the model (clf.fit)
- Predict the values given the model (clf.predict)

```python
from astroML.linear_model import LinearRegression
clf = LinearRegression()
clf.fit(x[:, None], y, sigma_y)
y_fit = clf.predict(x[:, None])

plt.errorbar(x, y, yerr=sigma_y, fmt=".k", capsize=0)
plt.plot(x, y_fit, "g", alpha=1)
plt.plot(xGrid, w[0]*xGrid + w[1], "r", alpha=0.5)
plt.xlim(0, 250)
plt.ylim(100, 600)
plt.xlabel('x', fontsize=16)
plt.ylabel('y', fontsize=16)
```

##  Total Least Squares regression
[Go to top](#toc)

What do we do if we have errors on the dependent and independent axes (or if the x axes errors are larger)?

### Exercise - using the approaches given above estimate the slope and intercept when the errors in x dominate

```python
# read data and select training and cross validation sample
cal_x,cal_dx,cal_y,cal_dy = np.loadtxt("data/X_Y.clean.txt", skiprows=1, unpack=True)
cal_dxy=np.zeros(len(x))

# define classifiers
n_constraints = [2,]

# set plots
fig = plt.figure(figsize=(10, 20))
ax = fig.add_subplot(211)
plt.errorbar(cal_x, cal_y, yerr=cal_dy, xerr=cal_dx, fmt=".k", capsize=0)
ax.scatter(cal_x, cal_y,c='black')
ax.set_xlim(cal_x.min()/1.05,cal_x.max()*1.05)
ax.set_ylim(cal_y.min()*1.05,cal_y.max()/1.05)
ax.set_xlabel("$\log \Sigma_{H_2}\ (M_\odot\ pc^{-2})$")
ax.set_ylabel("$\log \Sigma_{SFR}\ (M_\odot\ yr^{-1}\ kpc^{-2})$")
```

### LSQ with uncertainties in both the dependent and independent axes

In almost all real-world applications, the assumption that one
variable (the independent variable) is essentially free
from any uncertainty is not valid. Both the dependent and independent
variables will have measurement uncertainties (e.g. Tully-Fisher relations).

The impact of errors on the ``independent'' variables is a bias in the
derived regression coefficients. This is straightforward to show if
we consider a linear model with a dependent and independent
variable, $y^*$ and $x^*$. We can write the objective function as
before,

\begin{equation}
y^*_i=\theta_0 + \theta_1x^*_{i}.
\end{equation}

Now let us assume that we observe
$y$ and $x$, which are noisy
representations of $y^*$ and $x^*$, i.e.,
\begin{eqnarray}
x_i&=&x^*_i + \delta_i,\\
y_i &=& y^* + \epsilon_i,
\end{eqnarray}
with $\delta$ and $\epsilon$ centered normal distributions.

Solving for $y$ we get
\begin{equation}
      y=  \theta_0 + \theta_1 (x_i - \delta_i) +\epsilon_i.
\end{equation}

The uncertainty in $x$ is now part of the regression equation and
scales with the regression coefficients (biasing the regression
coefficient).  This problem is known in the statistics literature
as *total least squares* and belongs to the class of ``errors-in-variables''
problems.

For a detailed discussion of the solution to this problem, which is essentially
maximum likelihood estimation, please see
Chapter 8 in the reference book. Two other recommended references are
Hogg et al. (2010, astro-ph/1008.4686) and Kelly et al. (2011, astro-ph/1112.1745).

How can we account for the measurement uncertainties in both the
independent and dependent variables? Assuming they are Gaussian

> $ \Sigma_i = \left[
\begin{array}{cc}
\sigma_{x_i}^2 & \sigma_{xy_i} \\
\sigma_{xy_i} & \sigma_{y_i}^2
\end{array}
\right]
$

If we go back to the start of the lecture and write the equation for a line in terms of its normal vector

> $ {\bf n} = \left [
\begin{array}{c}
-\sin \alpha\\
\cos \alpha\\
\end{array}
\right ] $

with $\theta_1 = \arctan(\alpha)$ and $\alpha$ is the angle
between the line and the $x$-axis.

The covariance matrix projects onto
this space as
>$ S_i^2 = {\bf n}^T \Sigma_i {\bf n} $

and the  distance between a point and the line is

>$\Delta_i = {\bf n}^T
  z_i - \theta_0\ \cos \alpha, $

where $z_i$ represents the data point
>$(x_i,y_i)$.

The log-likelihood is then
>$ {\rm lnL} = - \sum_i \frac{\Delta_i^2}{2 S_i^2}$

and we can maximize the liklihood as a brute-force search or through MCMC

 THINKING OF A PROJECT: IT WOULD BE A REALLY INTERESTING PROBLEM TO EXTEND AND GENERALIZE THIS - IT WOULD GET A LOT OF CITATIONS!

```python
# Define some convenience functions

# translate between typical slope-intercept representation,
# and the normal vector representation
def get_m_b(beta):
    b = np.dot(beta, beta) / beta[1]
    m = -beta[0] / beta[1]
    return m, b

def get_beta(m, b):
    denom = (1 + m * m)
    return np.array([-b * m / denom, b / denom])

# compute the ellipse principal axes and rotation from covariance
def get_principal(sigma_x, sigma_y, rho_xy):
    sigma_xy2 = rho_xy * sigma_x * sigma_y

    alpha = 0.5 * np.arctan2(2 * sigma_xy2,
                             (sigma_x ** 2 - sigma_y ** 2))
    tmp1 = 0.5 * (sigma_x ** 2 + sigma_y ** 2)
    tmp2 = np.sqrt(0.25 * (sigma_x ** 2 - sigma_y ** 2) ** 2 + sigma_xy2 ** 2)

    return np.sqrt(tmp1 + tmp2), np.sqrt(tmp1 - tmp2), alpha

# plot ellipses
def plot_ellipses(x, y, sigma_x, sigma_y, rho_xy, factor=2, ax=None):
    if ax is None:
        ax = plt.gca()

    sigma1, sigma2, alpha = get_principal(sigma_x, sigma_y, rho_xy)

    for i in range(len(x)):
        ax.add_patch(Ellipse((x[i], y[i]),
                             factor * sigma1[i], factor * sigma2[i],
                             alpha[i] * 180. / np.pi,
                             fc='none', ec='k'))

# Find best-fit parameters
def get_best_fit(x, y, sigma_x, sigma_y, rho_xy):
    X = np.vstack((x, y)).T
    dX = np.zeros((len(x), 2, 2))
    dX[:, 0, 0] = sigma_x ** 2
    dX[:, 1, 1] = sigma_y ** 2
    dX[:, 0, 1] = dX[:, 1, 0] = rho_xy * sigma_x * sigma_y

    # note: TLS_logL was imported from astroML.linear_model
    min_func = lambda beta: -TLS_logL(beta, X, dX)
    # this is optimization, not MCMC
    return optimize.fmin(min_func, x0=[-1, 1])

# plot results
def plot_best_fit(x, y, sigma_x, sigma_y, rho_xy, beta_fit, mLSQ, bLSQ):
    fig = plt.figure(figsize=(8, 5))
    fig.subplots_adjust(left=0.1, right=0.95, wspace=0.25, bottom=0.15, top=0.9)
    ax = fig.add_subplot(121)
    ax.scatter(x, y, c='k', s=9)
    plot_ellipses(x, y, sigma_x, sigma_y, rho_xy, ax=ax)

    # plot the best-fit line
    m_fit, b_fit = get_m_b(beta_fit)
    x_fit = np.linspace(0, 300, 10)
    ax.plot(x_fit, m_fit * x_fit + b_fit, '-k')
    ax.plot(x_fit, mLSQ * x_fit + bLSQ, '--', c='red')

    ax.set_xlim(40, 250)
    ax.set_ylim(100, 600)
    ax.set_xlabel('$x$')
    ax.set_ylabel('$y$')

    # plot the likelihood contour in m, b
    ax = fig.add_subplot(122)
    m = np.linspace(1.7, 2.8, 100)
    b = np.linspace(-60, 110, 100)
    logL = np.zeros((len(m), len(b)))

    X = np.vstack((x, y)).T
    dX = np.zeros((len(x), 2, 2))
    dX[:, 0, 0] = sigma_x ** 2
    dX[:, 1, 1] = sigma_y ** 2
    dX[:, 0, 1] = dX[:, 1, 0] = rho_xy * sigma_x * sigma_y
    for i in range(len(m)):
        for j in range(len(b)):
            logL[i, j] = TLS_logL(get_beta(m[i], b[j]), X, dX)

    ax.contour(m, b, convert_to_stdev(logL.T),
           levels=(0.683, 0.955, 0.997), colors='k')
    ax.plot([-1000, 1000], [bLSQ, bLSQ], ':k', lw=1, c='red')
    ax.plot([mLSQ, mLSQ], [-1000, 1000], ':k', lw=1, c='red')

    ax.set_xlabel('slope')
    ax.set_ylabel('intercept')
    ax.set_xlim(1.7, 2.8)
    ax.set_ylim(-60, 110)

    plt.show()
```

```python
# for comparison, let's get the standard LSQ solution - no errors
mux = np.mean(x)
muy = np.mean(y)
mLSQ = np.sum(x*y-mux*muy)/np.sum((x-mux)**2)
bLSQ = muy - mLSQ*mux
print('mLSQ=', mLSQ)
print('bLSQ=', bLSQ)
```

    mLSQ= 2.191027996426704
    bLSQ= 32.00396939102313

```python
## let's do only errors in y - this is standard LSQ
# Find best-fit parameters
err_x = 0*sigma_x
err_y = sigma_y
rho = 0*rho_xy
best_fit1 = get_best_fit(x, y, err_x, err_y, rho)
m_fit1, b_fit1 = get_m_b(best_fit1)
print('m=', m_fit1)
print('b=', b_fit1)
# plot best fit
plot_best_fit(x, y, err_x, err_y, rho, best_fit1, mLSQ, bLSQ)
```

```python
## now only errors in x; note that we could switch the axes and use standard LSQ
# Find best-fit parameters
err_x = sigma_x
err_y = 0*sigma_y
rho = rho_xy
best_fit2 = get_best_fit(x, y, err_x, err_y, rho)
m_fit2, b_fit2 = get_m_b(best_fit2)
print('m=', m_fit2)
print('b=', b_fit2)
# plot best fit
plot_best_fit(x, y, err_x, err_y, rho, best_fit2, mLSQ, bLSQ)
```

```python
## errors in x and y, but without covariance
# Find best-fit parameters
err_x = sigma_x
err_y = sigma_y
rho = 0*rho_xy
best_fit3 = get_best_fit(x, y, err_x, err_y, rho)
m_fit3, b_fit3 = get_m_b(best_fit3)
print('m=', m_fit3)
print('b=', b_fit3)
# plot best fit
plot_best_fit(x, y, err_x, err_y, rho, best_fit3, mLSQ, bLSQ)
```

```python
## errors in x and y with covariance
# Find best-fit parameters
err_x = sigma_x
err_y = sigma_y
rho = rho_xy
best_fit4 = get_best_fit(x, y, err_x, err_y, rho)
m_fit4, b_fit4 = get_m_b(best_fit4)
print('m=', m_fit4)
print('b=', b_fit4)
# plot best fit
plot_best_fit(x, y, err_x, err_y, rho, best_fit4, mLSQ, bLSQ)
```

```python
# compare all 4 versions and LSQ
print('m=', m_fit1, m_fit2, m_fit3, m_fit4, mLSQ)
```

    m= 2.299303276704902 2.596107117541477 2.3849344880407286 2.248785375229947 2.191027996426704

### Using astroML

## Data sets used in the examples below

Use simulation data from [Kelly 2007](https://iopscience.iop.org/article/10.1086/519947/pdf).
This simulator, called `simulation_kelly`  is available from `astroML.datasets`.

The function returns the $\xi_i$, $\eta_i$, $x_i$, $y_i$, $\epsilon_{x,i}$, $\epsilon_{y,i}$ and
the input regression coefficients $\alpha$ and $\beta$ and intrinsic scatter $\epsilon$.

A total of ``size`` values generated, measurement errors are scaled by parameters ``scalex`` and
 ``scaley`` following section 7.1 in [Kelly 2007](https://iopscience.iop.org/article/10.1086/519947/pdf).

```python
from astroML.datasets import simulation_kelly

ksi, eta, xi, yi, xi_error, yi_error, alpha_in, beta_in = simulation_kelly(size=100, scalex=0.2, scaley=0.2,
                                                                           alpha=2, beta=1, epsilon=(0, 0.75))

ksi_0 = np.arange(np.min(xi[0]) - 0.5, np.max(xi[0]) + 0.5)
eta_0 = alpha_in + ksi_0 * beta_in

figure = plt.figure(figsize=(10, 8))
figure.subplots_adjust(left=0.1, right=0.95,
                       bottom=0.1, top=0.95,
                       hspace=0.1, wspace=0.15)
ax = figure.add_subplot(111)
ax.scatter(xi[0], yi, alpha=0.5)
ax.errorbar(xi[0], yi, xerr=xi_error[0], yerr=yi_error, alpha=0.3, ls='')
ax.set_xlabel(r'$\xi$')
ax.set_ylabel(r'$\eta$')
ax.plot(ksi_0, eta_0, color='orange')
```

### Linear regression with uncertainties in both dependent and independent axes

The class ``LinearRegressionwithErrors`` can be used to take into account measurement errors in
both the dependent and independent variables.
The implementation relies on the [PyMC3](https://docs.pymc.io/) and [Theano](http://deeplearning.net/software/theano/)
packages.

Note: The first initialization of the fitter is expected to take a couple of minutes, as ``Theano``
performs some code compilation for the underlying model. Sampling for consecutive runs is expected
to start up significantly faster.

```python
from astroML.linear_model import LinearRegressionwithErrors
from astroML.plotting import  plot_regressions, plot_regression_from_trace
```

```python
linreg_xy_err = LinearRegressionwithErrors()
linreg_xy_err.fit(xi, yi, yi_error, xi_error)
```

```python
plot_regressions(ksi, eta, xi[0], yi, xi_error[0], yi_error, add_regression_lines=True, alpha_in=alpha_in, beta_in=beta_in)
plot_regression_from_trace(linreg_xy_err, (xi, yi, xi_error, yi_error), ax=plt.gca(), chains=20)
```

## Multivariate regression
For multivariate data (where we fit a hyperplane rather than a straight line) we simply extend the description
of the regression function to multiple dimensions. The formalism used in the previous example becomes:

$$ \eta_i = \alpha + \beta^T \xi_i + \epsilon_i $$

where both $\beta^T$ and $\xi_i$ are now N-element vectors.

### Generate a dataset:

We use the same function as above to generate 100 datapoints in 2 dimensions. Note that the size of the ``beta``
parameter needs to match the dimensions.

```python
ksi2, eta2, xi2, yi2, xi_error2, yi_error2, alpha_in2, beta_in2 = simulation_kelly(size=100, scalex=0.2, scaley=0.2,
                                                                                alpha=2, beta=[0.5, 1],
                                                                                multidim=2)
```

The previously used ``LinearRegressionwithErrors`` class can be used with multidimensional data, thus the fitting is done the exact same way as before:

```python
linreg_xy_err2 = LinearRegressionwithErrors()
linreg_xy_err2.fit(xi2, yi2, yi_error2, xi_error2)
```

There are several ways to explore the fits, in the following we show a few ways to plot this dataset. As in this example the fitted hyperplane was 2D, we can use a 3D plot to show both the fit and the underlying regession we used to generate the data from. In this 3D plot, the blue plane is the true regression, while the red plane is our fit, that takes into account the errors on the data points.

Other plotting libraries can also be used to e.g. create pairplots of the parameters (e.g. Arviz' ``plot_pair`` function, or Seaborn's ``jointplot``).

```python
x0 = np.linspace(np.min(xi2[0])-0.5, np.max(xi2[0])+0.5, 50)
x1 = np.linspace(np.min(xi2[1])-0.5, np.max(xi2[1])+0.5, 50)
x0, x1 = np.meshgrid(x0, x1)

y0 = alpha_in + x0 * beta_in2[0] + x1 * beta_in2[1]

y_fitted = linreg_xy_err2.coef_[0] + x0 * linreg_xy_err2.coef_[1] + x1 * linreg_xy_err2.coef_[2]

import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.set_xlabel('xi2[0] ')
ax.set_ylabel('xi2[1] ')
ax.set_zlabel('yi2 ')
ax.scatter(xi2[0], xi2[1], yi2, s=20)
ax.plot_surface(x0, x1, y0, alpha=0.2, facecolor='blue', label='True regression')
ax.plot_surface(x0, x1, y_fitted, alpha=0.2, facecolor='red', label='Fitted regression')
```

### Sampler statistics and traceplots

The PyMC3 trace is available in the ``.trace`` attribute of the class instances (e.g. ``linreg_xy_err2.trace`` in the previous example), after we performed the fit. This can be then used for checking for convergence, and generating statistics for the samples. We would refer to use the tools provided by PyMC3, e.g. the ``traceplot()`` that takes the trace object as its input. Note that in the multidimensional case, there will be multiple ``ksi`` and ``slope`` traces in aggreement with the dimensionality of the input ``xi`` data, in the ``traceplot`` they are plotted with different colours.

```python
import pymc3 as pm
matplotlib.rc('text', usetex=False)

pm.traceplot(linreg_xy_err2.trace)
```

### CONCLUSION

Beware of your measurement uncertainties, especially if both variables have them!

##  Dealing with outliers
[Go to top](#toc)

The $L_2$ ($\sum_{i=1}^N (y_i - M(x_i))^2$) norm is sensitive to outliers (i.e. it squares the residuals). A number of approaches exist for correcting for outliers. These include "sigma-clipping", using interquartile ranges, taking the median of solutions of subsets of the data, and least trimmed squares (which searchs for the subset of points that minimizes $\sum_i^K (y_i - \theta_ix_i)^2$).

We can also change the **loss function** or **likelihood** to reduce the weight of outliers. An example of this is known as the _Huber loss function_

> $
\sum_{i=1}^N e(y_i|y),
$

where

>$
e(t) = \left\{
\begin{array}{ll}
\frac{1}{2} t^2 & \mbox{if} \; |t| \leq c, \\
c|t| - \frac{1}{2} c^2 & \mbox{if} \; |t| \geq c,
\end{array}
\right )
$

this is continuous and differentiable and transitions to an $L_1$ norm ($\sum_{i=1}^N |y_i - M(x_i)|$) for large excursions

```python
%matplotlib inline
import numpy as np
from matplotlib import pyplot as plt

from scipy import optimize

#------------------------------------------------------------
# Get data: this includes outliers
data = fetch_hogg2010test()
x = data['x']
y = data['y']
dy = data['sigma_y']

fig = plt.figure(figsize=(6, 6))
ax = fig.add_subplot(111)
ax.errorbar(x[4:], y[4:], dy[4:], fmt='.k', lw=1, ecolor='gray')
ax.errorbar(x[:4], y[:4], dy[:4], fmt='.k', lw=1, ecolor='red')
ax.set_xlim(0, 350)
ax.set_ylim(100, 700)
ax.set_xlabel('$x$')
ax.set_ylabel('$y$')
```

```python
# Define the standard squared-loss function
def squared_loss(m, b, x, y, dy):
    y_fit = m * x + b
    return np.sum(((y - y_fit) / dy) ** 2, -1)

# Define the log-likelihood via the Huber loss function
def huber_loss(m, b, x, y, dy, c=2):
    y_fit = m * x + b
    t = abs((y - y_fit) / dy)
    flag = t > c
    return np.sum((~flag) * (0.5 * t ** 2) - (flag) * c * (0.5 * c - t), -1)

f_squared = lambda beta: squared_loss(beta[0], beta[1], x=x[4:], y=y[4:], dy=dy[4:])

f_squared_outlier = lambda beta: squared_loss(beta[0], beta[1], x=x, y=y, dy=dy)

f_huber = lambda beta: huber_loss(beta[0], beta[1], x=x, y=y, dy=dy, c=1)

#------------------------------------------------------------
# compute the maximum likelihood using the huber loss
beta0 = (2, 30)
beta_squared = optimize.fmin(f_squared, beta0)
beta_squared_outlier = optimize.fmin(f_squared_outlier, beta0)

beta_huber = optimize.fmin(f_huber, beta0)

#------------------------------------------------------------
# Plot the results
fig = plt.figure(figsize=(6, 6))
ax = fig.add_subplot(111)

x_fit = np.linspace(0, 350, 10)
ax.plot(x_fit, beta_squared[0] * x_fit + beta_squared[1], '--k',
        label="squared loss:\n $y=%.2fx + %.1f$" % tuple(beta_squared))
ax.plot(x_fit, beta_squared_outlier[0] * x_fit + beta_squared_outlier[1], '--k', color='red',
        label="squared loss with outliers:\n $y=%.2fx + %.1f$" % tuple(beta_squared_outlier))
ax.plot(x_fit, beta_huber[0] * x_fit + beta_huber[1], '-k', color='blue',
        label="huber loss:\n $y=%.2fx + %.1f$" % tuple(beta_huber))
ax.legend(loc=4, prop=dict(size=14))

ax.errorbar(x[4:], y[4:], dy[4:], fmt='.k', lw=1, ecolor='gray')
ax.errorbar(x[:4], y[:4], dy[:4], fmt='.k', lw=1, ecolor='red')

ax.set_xlim(0, 350)
ax.set_ylim(100, 700)
ax.set_xlabel('$x$')
ax.set_ylabel('$y$')
```

###  From a Bayesian Perspective

We can assume the data are drawn from two Gaussians error distribution (one for the function and the other for the outliers)

$\begin{eqnarray}
&  p(\{y_i\}|\{x_i\}, \{\sigma_i\}, \theta_0, \theta_1, \mu_b, V_b, p_b)
  \propto \nonumber\\
&  \prod_{i=1}^{N} \bigg[
    \frac{1-p_b}{\sqrt{2\pi\sigma_i^2}}
      \exp\left(-\frac{(y_i - \theta_1 x_i - \theta_0)^2}
               {2 \sigma_i^2}\right)
    + \frac{p_b}{\sqrt{2\pi(V_b + \sigma_i^2)}}
    \exp\left(-\frac{(y_i - \mu_b)^2}{2(V_b + \sigma_i^2)}\right)
    \bigg].
\end{eqnarray}
$

$V_b$ is the variance of the outlier distribution. If we use MCMC we can marginalize over the nuisance  parameters $p_b$, $V_b$, $\mu_b$. We could also calculate the probability that a point is drawn from the outlier or "model" Gaussian.

```python
import numpy as np

import pymc3 as pm

from matplotlib import pyplot as plt
from theano import shared as tshared
import theano.tensor as tt

from astroML.datasets import fetch_hogg2010test
from astroML.plotting.mcmc import convert_to_stdev

# ----------------------------------------------------------------------
# This function adjusts matplotlib settings for a uniform feel in the textbook.
# Note that with usetex=True, fonts are rendered with LaTeX.  This may
# result in an error if LaTeX is not installed on your system.  In that case,
# you can set usetex to False.
if "setup_text_plots" not in globals():
    from astroML.plotting import setup_text_plots
setup_text_plots(fontsize=8, usetex=True)

np.random.seed(0)

# ------------------------------------------------------------
# Get data: this includes outliers. We need to convert them to Theano variables
data = fetch_hogg2010test()
xi = tshared(data['x'])
yi = tshared(data['y'])
dyi = tshared(data['sigma_y'])
size = len(data)

# ----------------------------------------------------------------------
# Define basic linear model

def model(xi, theta, intercept):
    slope = np.tan(theta)
    return slope * xi + intercept

# ----------------------------------------------------------------------
# First model: no outlier correction
with pm.Model():
    # set priors on model gradient and y-intercept
    inter = pm.Uniform('inter', -1000, 1000)
    theta = pm.Uniform('theta', -np.pi / 2, np.pi / 2)

    y = pm.Normal('y', mu=model(xi, theta, inter), sd=dyi, observed=yi)

    trace0 = pm.sample(draws=5000, tune=1000)

# ----------------------------------------------------------------------
# Second model: nuisance variables correcting for outliers
# This is the mixture model given in equation 17 in Hogg et al
def mixture_likelihood(yi, xi):
    """Equation 17 of Hogg 2010"""

    sigmab = tt.exp(log_sigmab)
    mu = model(xi, theta, inter)

    Vi = dyi ** 2
    Vb = sigmab ** 2

    root2pi = np.sqrt(2 * np.pi)

    L_in = (1. / root2pi / dyi * np.exp(-0.5 * (yi - mu) ** 2 / Vi))

    L_out = (1. / root2pi / np.sqrt(Vi + Vb)
             * np.exp(-0.5 * (yi - Yb) ** 2 / (Vi + Vb)))

    return tt.sum(tt.log((1 - Pb) * L_in + Pb * L_out))

with pm.Model():
    # uniform prior on Pb, the fraction of bad points
    Pb = pm.Uniform('Pb', 0, 1.0, testval=0.1)

    # uniform prior on Yb, the centroid of the outlier distribution
    Yb = pm.Uniform('Yb', -10000, 10000, testval=0)

    # uniform prior on log(sigmab), the spread of the outlier distribution
    log_sigmab = pm.Uniform('log_sigmab', -10, 10, testval=5)

    inter = pm.Uniform('inter', -200, 400)
    theta = pm.Uniform('theta', -np.pi / 2, np.pi / 2, testval=np.pi / 4)

    y_mixture = pm.DensityDist('mixturenormal', logp=mixture_likelihood,
                               observed={'yi': yi, 'xi': xi})

    trace1 = pm.sample(draws=5000, tune=1000)

# ----------------------------------------------------------------------
# Third model: marginalizes over the probability that each point is an outlier.
# define priors on beta = (slope, intercept)

def outlier_likelihood(yi, xi):
    """likelihood for full outlier posterior"""

    sigmab = tt.exp(log_sigmab)
    mu = model(xi, theta, inter)

    Vi = dyi ** 2
    Vb = sigmab ** 2

    logL_in = -0.5 * tt.sum(qi * (np.log(2 * np.pi * Vi)
                                  + (yi - mu) ** 2 / Vi))

    logL_out = -0.5 * tt.sum((1 - qi) * (np.log(2 * np.pi * (Vi + Vb))
                                         + (yi - Yb) ** 2 / (Vi + Vb)))

    return logL_out + logL_in

with pm.Model():
    # uniform prior on Pb, the fraction of bad points
    Pb = pm.Uniform('Pb', 0, 1.0, testval=0.1)

    # uniform prior on Yb, the centroid of the outlier distribution
    Yb = pm.Uniform('Yb', -10000, 10000, testval=0)

    # uniform prior on log(sigmab), the spread of the outlier distribution
    log_sigmab = pm.Uniform('log_sigmab', -10, 10, testval=5)

    inter = pm.Uniform('inter', -1000, 1000)
    theta = pm.Uniform('theta', -np.pi / 2, np.pi / 2)

    # qi is bernoulli distributed
    qi = pm.Bernoulli('qi', p=1 - Pb, shape=size)

    y_outlier = pm.DensityDist('outliernormal', logp=outlier_likelihood,
                               observed={'yi': yi, 'xi': xi})

    trace2 = pm.sample(draws=5000, tune=1000)

# ------------------------------------------------------------
# plot the data
fig = plt.figure(figsize=(5, 5))
fig.subplots_adjust(left=0.1, right=0.95, wspace=0.25,
                    bottom=0.1, top=0.95, hspace=0.2)

# first axes: plot the data
ax1 = fig.add_subplot(221)
ax1.errorbar(data['x'], data['y'], data['sigma_y'], fmt='.k', ecolor='gray', lw=1)
ax1.set_xlabel('$x$')
ax1.set_ylabel('$y$')

#------------------------------------------------------------
# Go through models; compute and plot likelihoods
linestyles = [':', '--', '-']
labels = ['no outlier correction\n(dotted fit)',
          'mixture model\n(dashed fit)',
          'outlier rejection\n(solid fit)']

x = np.linspace(0, 350, 10)

bins = [(np.linspace(140, 300, 51), np.linspace(0.6, 1.6, 51)),
        (np.linspace(-40, 120, 51), np.linspace(1.8, 2.8, 51)),
        (np.linspace(-40, 120, 51), np.linspace(1.8, 2.8, 51))]

for i, trace in enumerate([trace0, trace1, trace2]):
    H2D, bins1, bins2 = np.histogram2d(np.tan(trace['theta']),
                                       trace['inter'], bins=50)
    w = np.where(H2D == H2D.max())

    # choose the maximum posterior slope and intercept
    slope_best = bins1[w[0][0]]
    intercept_best = bins2[w[1][0]]

    # plot the best-fit line
    ax1.plot(x, intercept_best + slope_best * x, linestyles[i], c='k')

    # For the model which identifies bad points,
    # plot circles around points identified as outliers.
    if i == 2:
        Pi = trace['qi'].mean(0)
        outlier_x = data['x'][Pi < 0.32]
        outlier_y = data['y'][Pi < 0.32]
        ax1.scatter(outlier_x, outlier_y, lw=1, s=400, alpha=0.5,
                    facecolors='none', edgecolors='red')

    # plot the likelihood contours
    ax = plt.subplot(222 + i)

    H, xbins, ybins = np.histogram2d(trace['inter'],
                                     np.tan(trace['theta']), bins=bins[i])
    H[H == 0] = 1E-16
    Nsigma = convert_to_stdev(np.log(H))

    ax.contour(0.5 * (xbins[1:] + xbins[:-1]),
               0.5 * (ybins[1:] + ybins[:-1]),
               Nsigma.T, levels=[0.683, 0.955], colors='black')

    ax.set_xlabel('intercept')
    ax.set_ylabel('slope')
    ax.grid(color='gray')
    ax.xaxis.set_major_locator(plt.MultipleLocator(40))
    ax.yaxis.set_major_locator(plt.MultipleLocator(0.2))

    ax.text(0.96, 0.96, labels[i], ha='right', va='top',
            bbox=dict(fc='w', ec='none', alpha=0.5),
            transform=ax.transAxes)
    ax.set_xlim(bins[i][0][0], bins[i][0][-1])
    ax.set_ylim(bins[i][1][0], bins[i][1][-1])

ax1.set_xlim(0, 350)
ax1.set_ylim(100, 700)

plt.show()
```

### Exercise - using the approaches given above estimate the slope and intercept when the errors are in both axes


## 2-3 量子フーリエ変換

この節では、量子アルゴリズムの中でも最も重要なアルゴリズムの一つである量子フーリエ変換について学ぶ。
量子フーリエ変換はその名の通りフーリエ変換を行う量子アルゴリズムであり、様々な量子アルゴリズムのサブルーチンとしても用いられることが多い。
(参照：Nielsen-Chuang 5.1 `The quantum Fourier transform`)

※なお、最後のコラムでも多少述べるが、回路が少し複雑である・入力状態を用意することが難しいといった理由から、いわゆるNISQデバイスでの量子フーリエ変換の実行は難しいと考えられている。

### 定義
まず、$2^n$成分の配列 $\{x_j\}$ に対して$(j=1,\cdots,2^n)$、その[離散フーリエ変換](https://ja.wikipedia.org/wiki/離散フーリエ変換)である配列$\{ y_k \}$を

$$
y_k = \frac{1}{\sqrt{2^n}} \sum_{j=1}^{2^n} x_j e^{i\frac{2\pi kj}{2^n}}   \tag{1}
$$

で定義する$(k=1, \cdots 2^n)$。配列 $\{x_j\}$ は$\sum_{j=1}^{2^n} |x_j|^2 = 1$ と規格化されているものとする。

量子フーリエ変換アルゴリズムは、入力の量子状態

$$
|x\rangle := \sum_{j=1}^{2^n} x_j |j\rangle
$$

を、

$$
|y \rangle := \sum_{k=1}^{2^n} y_k |k\rangle  \tag{2}
$$

となるように変換する量子アルゴリズムである。ここで、$|i \rangle$は、整数$i$の二進数での表示$i_1 \cdots i_n$ ($i_m = 0,1$)に対応する量子状態$|i_1 \cdots i_n \rangle$の略記である。（例えば、$|2 \rangle = |0\cdots0 10 \rangle, |7 \rangle = |0\cdots0111 \rangle$となる)

ここで、式(1)を(2)に代入してみると、

$$
 |y \rangle = \frac{1}{\sqrt{2^n}} \sum_{k=1}^{2^n} \sum_{j=1}^{2^n} x_j e^{i\frac{2\pi kj}{2^n}} |k\rangle
             = \sum_{j=1}^{2^n} x_j \left( \frac{1}{\sqrt{2^n}} \sum_{k=1}^{2^n}  e^{i\frac{2\pi kj}{2^n}} |k\rangle \right)
$$

となる。よって、量子フーリエ変換では、

$$
 |j\rangle \to \frac{1}{\sqrt{2^n}} \sum_{k=1}^{2^n}  e^{i\frac{2\pi kj}{2^n}} |k\rangle
$$

を行う量子回路(変換)$U$を見つければ良いことになる。（余裕のある読者は、これがユニタリ変換であることを実際に計算して確かめてみよう）
この式はさらに式変形できて（やや複雑なので最後の結果だけ見てもよい）

$$
\begin{eqnarray}
  \sum_{k=1}^{2^n} e^{i\frac{2\pi kj}{2^n}} |k\rangle
 &=& \sum_{k_1=0}^1 \cdots \sum_{k_n=0}^1  e^{i\frac{2\pi (k_1 2^{n-1} + \cdots k_n 2^0 )\cdot j}{2^n}} |k_1 \cdots k_n\rangle \:\:\:\: \text{(kの和を2進数表示で書き直した)} \\
 &=& \sum_{k_1=0}^1 \cdots \sum_{k_n=0}^1  e^{i 2\pi j (k_1 2^{-1} + \cdots k_n 2^{-n})} |k_1 \cdots k_n\rangle \\
 &=& \left( \sum_{k_1=0}^1 e^{i 2\pi j k_1 2^{-1}} |k_1 \rangle \right) \otimes \cdots  \otimes \left( \sum_{k_n=0}^1 e^{i 2\pi j k_n 2^{-n}} |k_n \rangle \right) \:\:\:\: \text{("因数分解"をして、全体をテンソル積で書き直した)} \\
 &=& \left( |0\rangle +  e^{i 2\pi 0.j_n} |1 \rangle \right) \otimes \left( |0\rangle +  e^{i 2\pi 0.j_{n-1}j_n} |1 \rangle \right) \otimes \cdots \otimes \left( |0\rangle +  e^{i 2\pi 0.j_1j_2\cdots j_n} |1 \rangle \right) \:\:\:\: \text{(カッコの中の和を計算した)}
\end{eqnarray}
$$

となる。ここで、

$$
0.j_l\cdots j_n = \frac{j_l}{2} + \frac{j_{l-1}}{2^2} + \cdots + \frac{j_n}{2^{n-l+1}}
$$

は二進小数であり、$e^{i 2\pi j/2^{-l} } = e^{i 2\pi j_1 \cdots j_l . j_{l-1}\cdots j_n }
= e^{i 2\pi 0. j_{l-1}\cdots j_n }$となることを用いた。（$e^{i2\pi}=1$なので、整数部分は関係ない）

まとめると、量子フーリエ変換では、

$$
 |j\rangle = |j_1 \cdots j_n \rangle \to \frac{ \left( |0\rangle +  e^{i 2\pi 0.j_n} |1 \rangle \right) \otimes \left( |0\rangle +  e^{i 2\pi 0.j_{n-1}j_n} |1 \rangle \right) \otimes \cdots \otimes \left( |0\rangle +  e^{i 2\pi 0.j_1j_2\cdots j_n} |1 \rangle \right)  }{\sqrt{2^n}}
 \tag{*}
$$

という変換ができればよい。

### 回路の構成
それでは、量子フーリエ変換を実行する回路を実際にどのように構成するかを見ていこう。
そのために、次のアダマールゲート$H$についての等式（計算すると合っていることが分かる）

$$
 |m \rangle = \frac{|0\rangle + e^{i 2\pi 0.m}|1\rangle }{\sqrt{2}}  \:\:\: (m=0,1)
$$

と、角度 $2\pi/2^l$ の一般位相ゲート

$$
 R_l =
 \begin{pmatrix} 1 & 0\\ 0 & e^{i \frac{2\pi}{2^l} } \end{pmatrix}
$$

を多用する。

1. まず、状態$\left( |0\rangle +  e^{i 2\pi 0.j_1j_2\cdots j_n} |1\rangle \right)$の部分をつくる。1番目の量子ビット$|j_1\rangle$にアダマールゲートをかけると

    $$
 |j_1 \cdots j_n \rangle \to \frac{1}{\sqrt{2}} \left( |0\rangle + e^{i2\pi 0.j_1} |1\rangle \right) |j_2 \cdots j_n \rangle
$$

    となるが、ここで、2番目のビット$|j_2\rangle$を制御ビットとする一般位相ゲート$R_2$を1番目の量子ビットにかけると、$j_2=0$の時は何もせず、$j_2=1$の時のみ1番目の量子ビットの$|1\rangle$部分に位相 $2\pi/2^2 = 0.01$（二進小数）がつくから、

    $$
     \frac{1}{\sqrt{2}} \left( |0\rangle + e^{i2\pi 0.j_1} |1\rangle \right) |j_2 \cdots j_n \rangle
 \to \frac{1}{\sqrt{2}} \left( |0\rangle + e^{i2\pi 0.j_1j_2} |1\rangle \right) |j_2 \cdots j_n \rangle
$$

    となる。以下、$l$番目の量子ビット$|j_l\rangle$を制御ビットとする一般位相ゲート$R_l$をかければ（$l=3,\cdots n$）、最終的に

    $$
     \frac{1}{\sqrt{2}} \left( |0\rangle + e^{i2\pi 0.j_1\cdots j_n} \right) |j_2 \cdots j_n \rangle
$$

    が得られる。

2. 次に、状態$\left( |0\rangle + e^{i2\pi 0.j_{n-1} j_n} |1\rangle\right)$の部分をつくる。先ほどと同様に、2番目のビット$|j_2\rangle$にアダマールゲートをかければ

    $$
     \frac{1}{\sqrt{2}} \left( |0\rangle + e^{i2\pi 0.j_1\cdots j_n} \right) \frac{1}{\sqrt{2}} \left( |0\rangle + e^{i2\pi 0.j_2} \right) |j_3 \cdots j_n \rangle
$$

    ができる。再び、3番目の量子ビットを制御ビット$|j_3\rangle$とする位相ゲート$R_2$をかければ

    $$
     \frac{1}{\sqrt{2}} \left( |0\rangle + e^{i2\pi 0.j_1\cdots j_n} \right) \frac{1}{\sqrt{2}} \left( |0\rangle + e^{i2\pi 0.j_2j_3} \right) |j_3 \cdots j_n \rangle
$$

    となり、これを繰り返して

    $$
     \frac{1}{\sqrt{2}} \left( |0\rangle + e^{i2\pi 0.j_1\cdots j_n} \right) \frac{1}{\sqrt{2}} \left( |0\rangle + e^{i2\pi 0.j_2\cdots j_n} \right) |j_3 \cdots j_n \rangle
$$

    を得る。

3. 1,2と同様の手順で、$l$番目の量子ビット$|j_l\rangle$にアダマールゲート・制御位相ゲート$R_l$をかけていく($l=3,\cdots,n$)。すると最終的に

    $$
 |j_1 \cdots j_n \rangle \to \left( \frac{|0\rangle +  e^{i 2\pi 0.j_1\cdots j_n} |1 \rangle}{\sqrt{2}} \right) \otimes \cdots \otimes
 \left( \frac{|0\rangle +  e^{i 2\pi 0.j_{n-1}j_n} |1 \rangle}{\sqrt{2}} \right)  \otimes
 \left( \frac{|0\rangle +  e^{i 2\pi 0.j_n} |1 \rangle}{\sqrt{2}} \right)
$$

    が得られるので、最後にビットの順番をSWAPゲートで反転させてあげれば、量子フーリエ変換を実行する回路が構成できたことになる（式($*$)とはビットの順番が逆になっていることに注意）。
    SWAPを除いた部分を回路図で書くと以下のようである。

### SymPyを用いた実装
量子フーリエ変換への理解を深めるために、SymPyを用いて$n=3$の場合の回路を実装してみよう。

```python
from sympy import *
from sympy.physics.quantum import *
from sympy.physics.quantum.qubit import Qubit,QubitBra
init_printing() # ベクトルや行列を綺麗に表示するため
from sympy.physics.quantum.gate import X,Y,Z,H,S,T,CNOT,SWAP,CPHASE,CGateS
```

```python
# Google Colaboratory上でのみ実行してください
from IPython.display import HTML
def setup_mathjax():
    display(HTML('''

    '''))
get_ipython().events.register('pre_run_cell', setup_mathjax)
```

まず、フーリエ変換される入力$|x\rangle$として、

$$
 |x\rangle =  \sum_{j=1}^8 \frac{1}{\sqrt{8}} |i\rangle
$$

という全ての状態の重ね合わせ状態を考える（$x_1 = \cdots = x_8 = 1/\sqrt{8}$）。

```python
input = 1/sqrt(8) *( Qubit("000")+Qubit("001")+Qubit("010")+Qubit("011")+Qubit("100")+Qubit("101")+Qubit("110")+Qubit("111"))
input
```

この状態に対応する配列をnumpyでフーリエ変換すると

```python
import numpy as np
input_np_array = 1/np.sqrt(8)*np.ones(8)
print( input_np_array ) ## 入力
print( np.fft.ifft(input_np_array) * np.sqrt(8) ) ## 出力.  ここでのフーリエ変換の定義とnumpyのifftの定義を合わせるため、sqrt(2^3)をかける
```

    [0.35355339 0.35355339 0.35355339 0.35355339 0.35355339 0.35355339
     0.35355339 0.35355339]
    [1.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j]

となり、フーリエ変換すると $y_1=1,y_2=\cdots=y_8=0$ という簡単な配列になることが分かる。これを量子フーリエ変換で確かめてみよう。

まず、$R_1, R_2, R_3$ゲートはそれぞれ$Z, S, T$ゲートに等しいことに注意する（$e^{i\pi}=-1, e^{i\pi/2}=i$）。

```python
represent(Z(0),nqubits=1), represent(S(0),nqubits=1), represent(T(0),nqubits=1)
```

$\displaystyle \left( \left[\begin{matrix}1 & 0\\0 & -1\end{matrix}\right], \  \left[\begin{matrix}1 & 0\\0 & i\end{matrix}\right], \  \left[\begin{matrix}1 & 0\\0 & e^{\frac{i \pi}{4}}\end{matrix}\right]\right)$

量子フーリエ変換(Quantum Fourier TransformなのでQFTと略す)を実行回路を構成していく。
最初に、1番目（SymPyは右から0,1,2とビットを数えるので、SymPyでは2番目）の量子ビットにアダマール演算子をかけ、2番目・3番目のビットを制御ビットとする$R_2, R_3$ゲートをかける。

```python
QFT_gate = H(2)
QFT_gate = CGateS(1, S(2))  * QFT_gate
QFT_gate = CGateS(0, T(2))  * QFT_gate
```

2番目（SymPyでは1番目）の量子ビットにもアダマールゲートと制御$R_2$演算を施す。

```python
QFT_gate = H(1)  * QFT_gate
QFT_gate = CGateS(0, S(1))  * QFT_gate
```

3番目（SymPyでは0番目）の量子ビットにはアダマールゲートのみをかければ良い。

```python
QFT_gate = H(0)  * QFT_gate
```

最後に、ビットの順番を合わせるためにSWAPゲートをかける。

```python
QFT_gate = SWAP(0, 2)  * QFT_gate
```

これで$n=3$の時の量子フーリエ変換の回路を構成できた。回路自体はやや複雑である。

```python
QFT_gate
```

入力ベクトル$|x\rangle$ にこの回路を作用させると、以下のようになり、正しくフーリエ変換された状態が出力されていることが分かる。（$y_1=1,y_2=\cdots=y_8=0$）

```python
simplify( qapply( QFT_gate * input) )
```

読者は是非、入力を様々に変えてこの回路を実行し、フーリエ変換が正しく行われていることを確認してみてほしい。

---
### コラム：計算量について
「量子コンピュータは計算を高速に行える」とは、どういうことだろうか。本節で学んだ量子フーリエ変換を例にとって考えてみる。

量子フーリエ変換を行うために必要なゲート操作の回数は、1番目の量子ビットに$n$回、2番目の量子ビットに$n-1$回、...、$n$番目の量子ビットに1回で合計$n(n-1)/2$回、そして最後のSWAP操作が約$n/2$回であるから、全て合わせると$\mathcal{O}(n^2)$回である（$\mathcal{O}$記法について詳しく知りたい人は、下記セクションを参照）。
一方、古典コンピュータでフーリエ変換を行う[高速フーリエ変換](https://ja.wikipedia.org/wiki/高速フーリエ変換)は、同じ計算を行うのに$\mathcal{O}(n2^n)$の計算量を必要とする。この意味で、量子フーリエ変換は、古典コンピュータで行う高速フーリエ変換に比べて「高速」と言える。
これは一見喜ばしいことに見えるが、落とし穴がある。フーリエ変換した結果$\{y_k\}$は量子フーリエ変換後の状態$|y\rangle$の確率振幅として埋め込まれているが、この振幅を素直に読み出そうとすると、結局は**指数関数的な回数の観測を繰り返さなくてはならない**。さらに、そもそも入力$|x\rangle$を用意する方法も簡単ではない（素直にやると、やはり指数関数的な時間がかかってしまう）。
このように、量子コンピュータや量子アルゴリズムを「実用」するのは簡単ではなく、さまざまな工夫・技術発展がまだまだ求められている。

一体どのような問題で量子コンピュータが高速だと思われているのか、理論的にはどのように扱われているのかなど、詳しく学びたい方はこのQmediaの記事[「量子計算機が古典計算機より優れている」とはどういうことか](https://www.qmedia.jp/computational-complexity-and-quantum-computer/)(竹嵜智之)を参照されたい。

#### オーダー記法$\mathcal{O}$についての註
そもそも、アルゴリズムの性能はどのように定量評価できるのだろうか。ここでは、アルゴリズムの実行に必要な資源、主に時間をその基準として考える。とくに問題のサイズを$n$としたとき、計算ステップ数（時間）や消費メモリなど、必要な計算資源が$n$の関数としてどう振る舞うかを考える。（問題のサイズとは、例えばソートするデータの件数、あるいは素因数分解したい数の二進数表現の桁数などである。）

例えば、問題のサイズ$n$に対し、アルゴリズムの要求する計算資源が次の$f(n)$で与えられるとする。

$$ f(n) = 2n^2 + 5n + 8 $$

$n$が十分大きいとき（例えば$n=10^{10}$）、$2n^2$に比べて$5n$や$6$は十分に小さい。したがって、このアルゴリズムの評価という観点では$5n+8$という因子は重要ではない。また、$n^2$の係数が$2$であるという情報も、$n$が十分大きいときの振る舞いには影響を与えない。こうして、計算時間$f(n)$の一番**「強い」**項の情報が重要であると考えることができる。このような考え方を漸近的評価といい、計算量のオーダー記法では次の式で表す。

$$f(n) = \mathcal{O}(n^2)$$

一般に$f(n) = \mathcal{O}(g(n))$とは、ある正の数$n_0, c$が存在して、任意の$n > n_0$に対して

$$|f(n)| \leq c |g(n)|$$

が成り立つことである。上の例では、$n_0=7, c=3$とすればこの定義の通りである（グラフを描画してみよ）。練習として、$f(n) = 6n^3 +5n$のオーダー記法$f(n) = \mathcal{O}(n^3)$を与える$n_0, c$の組を考えてみよ。

アルゴリズムの性能評価では、その入力のサイズを$n$としたときに必要な計算資源を$n$の関数として表す。特にオーダー記法による漸近評価は、入力のサイズが大きくなったときの振る舞いを把握するときに便利である。そして、こうした漸近評価に基づいた計算量理論というものを用いて、様々なアルゴリズムの分類が行われている。詳細は上記のQmedia記事を参照されたい。


```python
# Header starts here.
from sympy.physics.units import *
from sympy import *

# Rounding:
import decimal
from decimal import Decimal as DX
from copy import deepcopy
def iso_round(obj, pv, rounding=decimal.ROUND_HALF_EVEN):
    import sympy
    """
    Rounding acc. to DIN EN ISO 80000-1:2013-08
    place value = Rundestellenwert
    """
    assert pv in set([
        # place value   #  round to:
        1,              #  1
        0.1,            #  1st digit after decimal
        0.01,           #  2nd
        0.001,          #  3rd
        0.0001,         #  4th
        0.00001,        #  5th
        0.000001,       #  6th
        0.0000001,      #  7th
        0.00000001,     #  8th
        0.000000001,    #  9th
        0.0000000001,   # 10th
        ])
    objc = deepcopy(obj)
    try:
        tmp = DX(str(float(objc)))
        objc = tmp.quantize(DX(str(pv)), rounding=rounding)
    except:
        for i in range(len(objc)):
            tmp = DX(str(float(objc[i])))
            objc[i] = tmp.quantize(DX(str(pv)), rounding=rounding)
    return objc

# LateX:
kwargs = {}
kwargs["mat_str"] = "bmatrix"
kwargs["mat_delim"] = ""

# kwargs["symbol_names"] = {FB: "F^{\mathsf B}", }

# Units:
(k, M, G ) = ( 10**3, 10**6, 10**9 )
(mm, cm) = ( m/1000, m/100 )
Newton = kg*m/s**2
Pa     = Newton/m**2
MPa    = M*Pa
GPa    = G*Pa
kN     = k*Newton
deg    = pi/180

half = S(1)/2

# Header ends here.
#
a = var("a", positive=True)
w1 = var("omega1")

# Equations (1) to (4):
# Shortcut:
xi = sqrt(15)/2
# Unknowns:
vB, vC = var("vB, vC")
w2, w3 = var("omega2, omega3")

eq1 = Eq(   vB       ,   w1*a        )
eq2 = Eq(   vC       ,   w3*2*a      )
eq3 = Eq( - vC * xi/2, - w2*xi*a     )
eq4 = Eq( - vC / 4   , - vB + w2/2*a )

unks = [vB, vC, w2, w3]
eqns = [eq1, eq2, eq3, eq4]

print("\nSolution:")
sol = solve(eqns, unks)
pprint(sol)

# Equations (5) to (10):
# Shortcuts
w2, w3 = sol[w2], sol[w3]
xi2, xi3 = w2*w2, w3*w3

# Unknowns:
aBx, aBy = var("aBx, aBy")
aCx, aCy = var("aCx, aCy")
w2p, w3p = var("alpha2, alpha3")

eq5  = Eq( aBx, -w1*w1*a                   )
eq6  = Eq( aBy, 0                          )
eq7  = Eq( aCx,     - w3p*xi*a + xi3*a/2   )
eq8  = Eq( aCy,     - w3p*a/2  - xi3*xi*a  )
eq9  = Eq( aCx, aBx - w2p*xi*a - xi2*a/2   )
eq10 = Eq( aCy, aBy + w2p*a/2  - xi2*xi*a  )

unks = [aBx, aBy, aCx, aCy, w2p, w3p]
eqns = [eq5, eq6, eq7, eq8, eq9, eq10]

pprint("\nSolution:")
sol = solve(eqns, unks)
pprint(sol)

# Solution:
# {ω₂: ω₁, ω₃: ω₁, vB: a⋅ω₁, vC: 2⋅a⋅ω₁}
#
# Solution:
# ⎧                               2                    2                2               2⎫
# ⎪          2               -a⋅ω₁         -17⋅√15⋅a⋅ω₁        -2⋅√15⋅ω₁        2⋅√15⋅ω₁ ⎪
# ⎨aBx: -a⋅ω₁ , aBy: 0, aCx: ───────, aCy: ──────────────, α₂: ───────────, α₃: ─────────⎬
# ⎪                             2                30                 15              15   ⎪
# ⎩                                                                                ⎭

```


```python
import numpy as np
import pandas as pd
from sympy import sieve

# prep
cities = pd.read_csv('../data/raw/cities.csv', index_col=['CityId'])
pnums = list(sieve.primerange(0, cities.shape[0]))

# function
def score_it(path):
    path_df = cities.reindex(path).reset_index()
    path_df['step'] = np.sqrt((path_df.X - path_df.X.shift())**2 + (path_df.Y - path_df.Y.shift())**2)
    path_df['step_adj'] = np.where((path_df.index) % 10 != 0, path_df.step, path_df.step +
                            path_df.step*0.1*(~path_df.CityId.shift().isin(pnums)))
    return path_df.step_adj.sum()

# usage: path is array_like
sub = pd.read_csv('../data/raw/sample_submission.csv')
print(score_it(sub.Path.values))
```

    446884407.521


# METOD algorithm - Minimum of several quadratic forms

# 1. Import libraries

The following libraries are required to run the METOD Algorithm.

```python
import numpy as np
from numpy import linalg as LA
import pandas as pd

import metod_alg as mt
from metod_alg import objective_functions as mt_obj
```

# 2. Define function and gradient

Minimum of several quadratic forms:
 \begin{equation}
 \label{eq:model}
f(x_n^{(k)})=\min_{1\le p \le P} \frac{1}{2} (x_n^{(k)}-x_{0p})^T A_p^T \Sigma_p A_p (x_n^{(k)}-x_{0p}),
\end{equation}
where $x_n^{(k)}$ is the $n$-th point after $k$ iterations of anti-gradient descent, $P$ is the number of minima; $A_p$ is a random rotation
matrix of size $d\times d$; $\Sigma_p$ is a diagonal positive
definite matrix of size $d\times d$ with smallest and largest
eigenvalues $\lambda_{min}$ and $\lambda_{max}$ respectively;
$x_{0p} \in \mathfrak{X}=[0,1]^d$ and $p=1,...,P$.

Note : Anti-gradient descent iterations are terminated at the smallest $k=K_n$ such that $\nabla f(x_n^{(k)}) < \delta$, where $\delta$ is some small positive constant.

```python
f = mt_obj.several_quad_function
g = mt_obj.several_quad_gradient
```

# 3. Defining parameters

The following parameters are required to derive $A_p$ ($p=1,...,P$) ; $\Sigma_p$ ($p=1,...,P$) and $x_{0p}$ ($p=1,...,P$).

•d: dimension

•P: number of minima

•lambda_1: smallest eigenvalue of $\Sigma_p$ ($p=1,...,P$)

•lambda_2: largest eigenvalue of $\Sigma_p$ ($p=1,...,P$)

In order to replicate results, we will control the pseudo-random number generator seed, so that the same random objective function and random starting points $x_n^{(0)}$ $(n=1,...,1000)$ will be generated each time the code is run. The random seed number will be set to 90.

```python
d = 50
P = 5
lambda_1 = 1
lambda_2 = 10
seed = 90
```

```python
np.random.seed(seed)
store_x0, matrix_combined = mt_obj.function_parameters_several_quad(P, d, lambda_1, lambda_2)
```

Where,

•store_x0: $x_{0p}$ ($p=1,...,P$)

•matrix_combined: $A_p^T \Sigma_p A_p$ ($p=1,...,P$)

```python
args = P, store_x0, matrix_combined
```

# 4. Run METOD Algorithm

```python
(discovered_minimizers,
 number_minimizers,
 func_vals_of_minimizers,
 excessive_no_descents,
 starting_points,
 grad_evals) = mt.metod(f, g, args, d)
```

# 5. Results of the METOD Algorithm

Total number of minimizers found:

```python
number_minimizers
```

Positions of minimizers:

```python
discovered_minimizers
```

Function values of minimizers:

```python
func_vals_of_minimizers
```

Total number of excessive descents:

```python
excessive_no_descents
```

Number of gradient evaluations for each starting point

```python
grad_evals
```

# 6. Save results to csv file (optional)

The below csv files will be saved to the same folder which contains the METOD Algorithm - Minimum of several quadratic forms notebook.

Rows in discovered_minimizers_d_%s_p_%s_quad.csv represent discovered minimizers. The total number of rows will be the same as the value for number_minimizers.

```python
np.savetxt('discovered_minimizers_d_%s_p_%s_quad.csv' % (d, P), discovered_minimizers, delimiter=",")
```

Each row in func_vals_discovered_minimizers_d_%s_p_%s_quad.csv represents the function value of each discovered minimizer. The total number of rows will be the same as the value for number_minimizers.

```python
np.savetxt('quad_func_vals_discovered_minimizers_d_%s_p_%s_quad.csv' % (d, P), func_vals_of_minimizers, delimiter=",")
```

summary_table_d_%s_p_%s_quad.csv will contain the total number of minimizers discovered and the total number of excessive descents.

```python
summary_table = pd.DataFrame({
"Total number of unique minimizers": [number_minimizers],
"Extra descents": [excessive_no_descents]})
summary_table.to_csv('summary_table_d_%s_p_%s_quad.csv' % (d, P))
```

# 7. Test results (optional)

This test can only be used for the minimum of several quadratic forms function.

To check each discovered minimizer is unique, we do the following:

For each minimizer $x_l^{(K_l)}$ ($l=1,...,L$)

\begin{equation}
p_l = {\rm argmin}_{1\le p \le P} \frac{1}{2} (x_l^{(K_l)}-x_{0p})^T A_p^T \Sigma_p A_p (x_l^{(K_l)}-x_{0p})
\end{equation}

We must have $p_l$ is different for all $x_l^{(K_l)}$ ($l=1,...,L$), in order for minimizers to be unique.

We also test that $\|x_l^{(K_l)} - x_{0p_l}\| \text{  is small}$.

```python
def calc_pos(point, d,  p, store_x0, matrix_combined):
    """Position (p_l) of the minimum of the function"""
    function_values = np.zeros((p))
    for i in range(p):
        function_values[i] = 0.5 * np.transpose(point - store_x0[i]) @ matrix_combined[i] @ (point - store_x0[i])
    position_minimum = np.argmin(function_values)
    return position_minimum
```

```python
"""Store values from calc_pos function"""
norms_with_minimizer = np.zeros((number_minimizers))
pos_list = np.zeros((number_minimizers))
for j in range(number_minimizers):
    pos = calc_pos(discovered_minimizers[j], d, *args)
    pos_list[j] = pos
    norms_with_minimizer[j] = LA.norm(discovered_minimizers[j]- store_x0[pos])
```

${\max}_{1\le l \le L}  \|x_l^{(K_l)}-x_{0p_l}\|$ should be small

```python
np.max(norms_with_minimizer)
```

Ensure that the number of unique minimizers is $L$

```python
np.unique(pos_list).shape[0] == number_minimizers
```

```python

```

Examples/Notebooks/METOD Algorithm - Minimum of several quadratic forms.ipynb


# Phase Shifts and Virtual Z gates

```python
import numpy as np
from pulser import Pulse, Sequence, Register
from pulser.devices import MockDevice
```

## Introduction

Under the right circumstances, phase shifts can be a great way of implementing so-called *virtual-Z gates*. Let's see how these arise and how we can use them to our advantage.
Consider an arbitrary 2x2 unitary matrix (up to a global phase) and a possible decompostion in terms of rotations around the $x$ and $z$ axes ($R_X$ and $R_Z$) of the Bloch sphere:

$$U(\gamma, \theta, \phi) =  R_Z(\gamma)R_X(\theta)R_Z(\phi)$$

Our goal is to be able to apply this transformation on our qubit states through our pulses. A pulse that is on-resonance with its target transition (i.e. has detuning $\delta = 0$) can be described as an arbitrary rotation around an axis contained in the Bloch sphere's equator, $\hat{n}= (\cos \phi, -\sin \phi, 0)$. From the general form of such a rotation, $R_{\hat{n}}(\theta) =     \exp\left(-i \frac{\theta}{2}  \hat{n}\cdot \vec{\sigma}\right)$, we arrive at:

$$
\begin{align}
 R_{\hat{n}(\phi)}(\theta) &= \exp\left(-i \frac{\theta}{2} \left[\cos(\phi)\sigma_x -\sin(\phi)\sigma_y\right]\right)\\
          &= e^{i\frac{\phi}{2}\sigma_z}e^{-i\frac{\theta}{2}\sigma_x}e^{-i\frac{\phi}{2}\sigma_z}\\
          &=  R_Z(-\phi)R_X(\theta)R_Z(\phi)
\end{align}
$$

Here, we have two free parameters: the angle of rotation $\theta$, which is determined by the integral of the amplitude waveform, and $\phi$, the pulse's phase. Thus, we can see that a pulse is a particular case of the arbitrary single-qubit gate $U$, where $\gamma = -\phi$, i.e.:

$$ R_{\hat{n}(\phi)}(\theta) = U(-\phi, \theta, \phi) $$

Thus, to reach the desired arbitrary single-qubit, we need an extra $R_Z$ gate, such that:

$$ U(\gamma, \theta, \phi) = R_Z(\gamma + \phi) R_Z(-\phi)R_X(\theta)R_Z(\phi) = R_Z(\gamma + \phi) U(-\phi, \theta, \phi)$$

Now, how do we implement such a gate? In fact, to physically change the phase of the qubit's state in this reference frame with a single pulse, we would have to be able to apply a detuned pulse of zero amplitude, which is no pulse at all! Instead, what we can do is change the frame of rotation such that the phase gate is applied *virtually*.

To understand how this can be done, we first have to realise that this last phase gate is irrelevant if it is the last one applied to the qubit before it is measured - a change of the phase between the $\sigma_z$ eigenstates will produce no change in the measurement outcome, since that occurs in the same basis. But what if it is not the last gate that we apply on this qubit? In that case, we can describe the situation as a new arbitrary gate being applied after the existent one, i.e.

$$ U(\alpha, \beta, \nu)~~U(\gamma, \theta, \phi) = R_Z(\alpha + \nu) R_Z(-\nu)R_X(\beta)R_Z(\nu) ~~ R_Z(k)~
U(-\phi, \theta, \phi) $$

where we define the *carry*, $k=\gamma + \phi$, as the phase of the unrealized phase gate. Now, we can restructure the previous expression such that:

$$
\begin{align}
& R_Z(\alpha + \nu) R_Z(-\nu)R_X(\beta)R_Z(\nu)~~ R_Z(k) ~U(-\phi, \theta, \phi) = \\
&= R_Z(\alpha + \nu + k)~~ R_Z(-\nu - k)R_X(\beta)R_Z(\nu + k) ~~U(-\phi, \theta, \phi) \\
&= R_Z(k') ~~U(-\nu - k, \beta, \nu + k)~~U(-\phi, \theta, \phi), ~~~k' = \alpha + \nu + k
\end{align}
$$

As shown, the previously existent phase gate of angle $k$ can be realized as a **shift** on the phase of the second pulse, $\nu \rightarrow \nu + k$. In this way, we go back to a situation where we have a phase gate at the end (with an updated carry $k'$). We can repeat this process until the moment we measure the qubit, at which point, as we've seen, the remaining phase gate is redundant.

This is the **virtual-Z gate**: the ability to perform phase gates through the adjustment of a qubit's phase reference.

## Phase shifts in Pulser

As shown above, implementing virtual-Z gates requires tracking the *carry* and adapting the phase of each provided pulse accordingly. Although this could certainly be done externally, Pulser offers a convenient way to automatically update a qubit's phase reference (i.e. the *carry*) through **phase shifts**.

A phase shift in Pulser is defined by three things: the *value* of the shift, the *target* qubits and the *basis*. A phase shift of *value* $\phi$ corresponds to a change in the *target*'s phase reference from $k \rightarrow k + \phi$.

It is important to realise that, unlike all other aspects so far, phase shifts are associated to the a qubit's transition, not a channel. Therefore, in principle, all qubit's can keep different phase references, as a result of being subjected to different phase shifts throughout a sequence.

### Programming a Hadamard gate

To exemplify the need for phase shifts, let's try to encode an Hadamard gate using only resonant pulses.
In our decomposition of a unitary matrix, the Hadamard is (you can check this for yourself):

$$ H = U\left(\frac{\pi}{2},\frac{\pi}{2}, \frac{\pi}{2}\right) = R_Z(\pi)~~U\left(-\frac{\pi}{2},\frac{\pi}{2}, \frac{\pi}{2}\right) $$

meaning that we have to apply a $\frac{\pi}{2}$-pulse with phase $\phi=\frac{\pi}{2}$, followed by a phase shift of $\pi$.

But first, let's create a simple setup.

```python
reg = Register({'q0': (0, 0)})
device = MockDevice
seq = Sequence(reg, device)
seq.available_channels
```

    {'rydberg_global': Rydberg.Global(Max Absolute Detuning: 1000 rad/µs, Max Amplitude: 200 rad/µs, Basis: 'ground-rydberg'),
     'rydberg_local': Rydberg.Local(Max Absolute Detuning: 1000 rad/µs, Max Amplitude: 200 rad/µs, Target time: 0 ns, Max targets: 2000, Basis: 'ground-rydberg'),
     'raman_global': Raman.Global(Max Absolute Detuning: 1000 rad/µs, Max Amplitude: 200 rad/µs, Basis: 'digital'),
     'raman_local': Raman.Local(Max Absolute Detuning: 1000 rad/µs, Max Amplitude: 200 rad/µs, Target time: 0 ns, Max targets: 2000, Basis: 'digital'),
     'mw_global': Microwave.Global(Max Absolute Detuning: 1000 rad/µs, Max Amplitude: 200 rad/µs, Basis: 'XY')}

```python
seq.declare_channel('ch0', 'raman_local', initial_target = 'q0')
```

```python
# Defining the waveform for a pi/2 pulse
from pulser.waveforms import BlackmanWaveform
pi2_wf = BlackmanWaveform(1000, np.pi/2)   # Duration: 1us, Area: pi/2
pi2_wf.draw()
```

```python
# 2. Create the pi/2 pulse
pi_2 = Pulse.ConstantDetuning(pi2_wf, detuning=0, phase=np.pi/2)
pi_2.draw()
```

```python
#3. Applying the H gate

seq.add(pi_2, 'ch0')   # The first pi/2-pulse
# Now the phase shift of pi on 'q0', for the 'digital' basis, which is usually where phase shifts are useful
seq.phase_shift(np.pi, 'q0', basis='digital')

seq.draw(draw_phase_shifts=True)
```

This produced the desired effect: we have a $\frac{\pi}{2}$ pulse with $\phi=\frac{\pi}{2}$ followed by a phase shift of $\pi$. However, the need to specify the target qubit and basis is mildly incovenient. Moreover, it would be even better if the entire hadamard could be in the same pulse object. Fortunately, there's a way.

The `Pulse` object has an optional argument called `post_phase_shift`, with which the user can order a phase shift to be applied immediatly after the physical pulse. In this case, the target and basis are **implicitly defined** to be the channel's current target and basis.

Here's how we could define the Hadamard in a single pulse:

```python
h = Pulse.ConstantDetuning(pi2_wf, detuning=0, phase=np.pi/2, post_phase_shift=np.pi)

seq.add(h, 'ch0')
seq.draw(draw_phase_shifts=True)
```

Notice how the two pulse shapes are naturally identical, and are both followed by the adequate $\pi$ phase shift. However, we expect to see an adjustment on the phase of the second pulse, simply because it is following a phase shift. To inspect the phase of each pulse, we can print the sequence:

```python
print(seq)
```

    Channel: ch0
    t: 0 | Initial targets: q0 | Phase Reference: 0.0
    t: 0->1000 | Pulse(Amp=Blackman(Area: 1.57), Detuning=0 rad/µs, Phase=1.57) | Targets: q0
    t: 1000->2000 | Pulse(Amp=Blackman(Area: 1.57), Detuning=0 rad/µs, Phase=4.71) | Targets: q0

Here, we see that the phase of the second pulse has the appropriate adjustment of $\pi$. What happens if we apply the Hadamard yet again?

```python
seq.add(h, 'ch0')
print(seq)
```

    Channel: ch0
    t: 0 | Initial targets: q0 | Phase Reference: 0.0
    t: 0->1000 | Pulse(Amp=Blackman(Area: 1.57), Detuning=0 rad/µs, Phase=1.57) | Targets: q0
    t: 1000->2000 | Pulse(Amp=Blackman(Area: 1.57), Detuning=0 rad/µs, Phase=4.71) | Targets: q0
    t: 2000->3000 | Pulse(Amp=Blackman(Area: 1.57), Detuning=0 rad/µs, Phase=1.57) | Targets: q0

That's right, the phase of the third pulse is back to $\frac{\pi}{2}$ because, at the moment the third Hadamard was constructed, there had been two $\pi$ phase shifts, which means the reference is back to $\phi=0$.

### Phase shifts with multiple channels and different targets

Now, let's construct a more complex example where there are multiple channels and multiple qubits.

```python
reg = Register({'q0': (0, 0), 'q1': (5, 5)})
device = MockDevice
seq = Sequence(reg, device)
seq.declare_channel('raman', 'raman_local', initial_target = 'q0')
seq.declare_channel('ryd1', 'rydberg_local', initial_target = 'q0')
seq.declare_channel('ryd2', 'rydberg_local', initial_target = 'q0')
seq.declared_channels
```

    {'raman': Raman.Local(Max Absolute Detuning: 1000 rad/µs, Max Amplitude: 200 rad/µs, Target time: 0 ns, Max targets: 2000, Basis: 'digital'),
     'ryd1': Rydberg.Local(Max Absolute Detuning: 1000 rad/µs, Max Amplitude: 200 rad/µs, Target time: 0 ns, Max targets: 2000, Basis: 'ground-rydberg'),
     'ryd2': Rydberg.Local(Max Absolute Detuning: 1000 rad/µs, Max Amplitude: 200 rad/µs, Target time: 0 ns, Max targets: 2000, Basis: 'ground-rydberg')}

We see that we have two qubits and three channels, all `Local`, with `raman` acting on the `digital` basis and the other two on the `ground-rydberg` basis. Let's use the Hadamard from before and add it to channels `raman` and `ryd1`, which are both targeting `q0` on different bases:

```python
seq.add(h, 'raman')
seq.add(h, 'ryd1')
seq.draw(draw_phase_shifts=True)
```

We can see that the pulse in channel `ryd1` waited for the pulse in `raman` to be finished (because they are acting on the same target). We also noticed that the phase shift in `raman` does not appear in the other channels, the reason for that being the fact that they act on **different bases**. We can check this by printing the sequence:

```python
print(seq)
```

    Channel: raman
    t: 0 | Initial targets: q0 | Phase Reference: 0.0
    t: 0->1000 | Pulse(Amp=Blackman(Area: 1.57), Detuning=0 rad/µs, Phase=1.57) | Targets: q0

    Channel: ryd1
    t: 0 | Initial targets: q0 | Phase Reference: 0.0
    t: 0->1000 | Delay
    t: 1000->2000 | Pulse(Amp=Blackman(Area: 1.57), Detuning=0 rad/µs, Phase=1.57) | Targets: q0

    Channel: ryd2
    t: 0 | Initial targets: q0 | Phase Reference: 0.0

Here, we confirm that the phase of the pulse in `ryd1` is $\frac{\pi}{2}$, which indicates a phase reference of $\phi=0$ as expected. What about when the phase shift targets the same qubit, the same basis, but the pulses are on different channels? In that case, we expect that the channel is irrelevant, and we can already see evidence of that in how the phase shift at the end of the Hadamard on `ryd1` also appears in `ryd2`. We can confirm this by adding another pulse to `ryd2` (e.g. the `pi_2` pulse we defined before) and then priting the sequence:

```python
seq.add(pi_2, 'ryd2')
print(seq)
```

    Channel: raman
    t: 0 | Initial targets: q0 | Phase Reference: 0.0
    t: 0->1000 | Pulse(Amp=Blackman(Area: 1.57), Detuning=0 rad/µs, Phase=1.57) | Targets: q0

    Channel: ryd1
    t: 0 | Initial targets: q0 | Phase Reference: 0.0
    t: 0->1000 | Delay
    t: 1000->2000 | Pulse(Amp=Blackman(Area: 1.57), Detuning=0 rad/µs, Phase=1.57) | Targets: q0

    Channel: ryd2
    t: 0 | Initial targets: q0 | Phase Reference: 0.0
    t: 0->2000 | Delay
    t: 2000->3000 | Pulse(Amp=Blackman(Area: 1.57), Detuning=0 rad/µs, Phase=4.71) | Targets: q0

Notice how the `pi_2` pulse has a phase of $\frac{3\pi}{2}$: it's phase of $\frac{\pi}{2}$ plus the shift of $\pi$ accrued by the Hadamard in channel `ryd1`.

Let's now shift our attention torwards what happens when the basis and the channel stay the same, but the target qubit changes. By now, you can already predict that qubit `q1` has a phase reference of $\phi=0$ on both basis, since all phase shifts so far were always targeting `q0`. We can see this if we change the target on some channels and apply pulses again:

```python
seq.target('q1', 'raman')
seq.add(h, 'raman')
seq.target('q1', 'ryd1')
seq.add(h, 'ryd1')
print(seq)
seq.draw(draw_phase_shifts=True)
```

We can see how the second pulse on the `raman` and `ryd1` channels are exactly identical to their first one, except for the target. Had the target not changed, they would have their phase shifted by $\pi$, just like we saw earlier. We also see how, this time, the phase shift in `ryd1` does not appear in `ryd2` like before because they have **different targets**. Notice what happens if we also make `ryd2` target `q1`:

```python
seq.target('q1', 'ryd2')
seq.draw(draw_phase_shifts=True)
```

As you can see, channel `ryd2` starts off with a phase of $\phi = \pi$, which it picked up from the phase shift induced by the Hadamard on `ryd1`. On the other hand, the Hadamard in the `raman` channel did not affect the phase of `ryd2`(we would have $\phi=0$ in that case) because, again, it acts on a different basis.

tutorials/composition/Phase Shifts and Virtual Z gates.ipynb


```python
from scipy import stats
from scipy import integrate
import numpy as np
%matplotlib inline
import matplotlib.pyplot as plt
from matplotlib import rc,rcParams
rc('text', usetex=True)
rcParams['figure.figsize'] = (9, 5.5)
rcParams['ytick.labelsize'],rcParams['xtick.labelsize'] = 17.,17.
rcParams['axes.labelsize']=19.
rcParams['legend.fontsize']=17.
rcParams['axes.titlesize']=20.
import seaborn as sb
sb.despine()
sb.set_style('white', {'xes.linewidth': 0.5, 'axes.edgecolor':'black'})
sb.despine(left=True)
```

    <Figure size 648x396 with 0 Axes>

```python

```

```python
stats.norm.p
```

```python
Nmax=1000
NN=np.arange(1,Nmax,1)
def uprior(N): return np.ones(N.shape)/Nmax
def nprior(N,N0,sigma): return np.exp(-(N-N0)**2/(2*sigma**2))/np.sqrt(2.*np.pi *sigma**2 )
def plprior(N,a): return N**a
def lkh(N,D): return np.where(N<D,0,1/N)
```

```python
def post(likehood,prior): return likehood*prior/np.sum(likehood*prior)
```

```python
plprior(NN,-2.).sum()
```

    1.6439335666815602

```python
plt.plot(NN,plprior(NN,-2.),'--')
plt.plot(NN,lkh(NN,60))
plt.plot(post(lkh(NN,60),plprior(NN,-2.)))
plt.xlim(0,200)
print('Mean of the Posteriror = {:.2f}'.format(np.average(NN, weights=posterior)))
```

```python
tanks=np.array([60])#,30,90,380])
prior=nprior(NN,100,50)
#prior=uprior(NN)
prior=plprior(NN,-2.)/np.sum(plprior(NN,-2.))
fig, ax = plt.subplots(figsize=(20, 10))
ax.plot(NN,prior,'-.',label='prior')
for tank in tanks:
    posterior=post(lkh(NN,tank),prior)
    prior=posterior
    ax.plot(NN,lkh(NN,tank),'--',label=tank)
ax.plot(NN,posterior,label='posterior')
plt.legend()
ax.axvline(np.average(NN, weights=posterior))
print('Mean of the Posterior = {:.2f}'.format(np.average(NN, weights=posterior)))
plt.ylim(0,0.1)
```

```python
np.average(NN, weights=posterior)
```

    515.396902239978

### The locomotive problem (analytical)
The prior distribution is:
\begin{align}
P(N)=\frac{1}{N_\text{max}} && (N=1,\cdots,N_\text{max})
\end{align}

where $N$ is the number of locomotives the railroad owns, and Nmax is the upper bound of the possible numbers.

If you observed a locomotive with the number $D$, the likelihood is:
$$
P(D∣N)=
\begin{cases}
\frac{1}{N} & (N\ge D) \\
0 & (N<D)
\end{cases}
$$
Then the posterior distribution becomes:
$$
P(N∣D) \propto P(D∣N)P(N)=
\begin{cases}
\frac{1}{NN_\text{max}} & (N\ge D)\\
0 & (N<D)
\end{cases}
$$

The normalization constant is:
$$
Z=\int \frac{1}{N_\text{max}} \frac{1}{N} =
$$

Hence,

P(N∣D)=P(D∣N)P(N)Z∼{1logNmaxD×1N0(N≥D)(N<D)
Especially, when Nmax=1000,D=60,N=60,

P(N∣D)∼1log100060×160≒0.0059
This matches with the result of Figure 3-1.

As another case, if you observed three locomotives with the number D1,D2,D3(D1≤D2≤D3), the likelihood is:

P(D1,D2,D3∣N)={1N30(D3≤N)(D3>N)
Then the posterior distribution becomes:

P(N∣D1,D2,D3)∝P(D1,D2,D3∣N)P(N)={1N3Nmax0(D3≤N)(D3>N)
The normalization constant is:

Z=∑NmaxN=D31N3Nmax∼1Nmax∫NmaxD31x3dx
　=12Nmax(1D23−1N2max)=12NmaxN2max−D23N2maxD23
Hence,

P(N∣D1,D2,D3)=P(D1,D2,D3∣N)P(N)Z∼{2N2maxD23N2max−D23×1N30(N≥D3)(N<D3)
Now, the mean of the posterior is:

∑NmaxN=D3(N×2N2maxD23N2max−D231N3)∼2N2maxD23N2max−D23∫NmaxD31x2dx
　=2N2maxD23N2max−D23(1D3−1Nmax)=2NmaxD3Nmax+D3
Especially, when Nmax=500,1000,2000 and D3=90,

Nmax=500:2×500×90500+90≒152.5
Nmax=1000:2×1000×901000+90≒165.1
Nmax=2000:2×2000×902000+90≒172.2
This matches with the result in the book.

From this calculation, you can see that the posterior depends only on the largest number you observed.


# The Fourier Transform

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Communications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Definition

The [Fourier transform](https://en.wikipedia.org/wiki/Fourier_transform), as commonly used in electrical engineering, is defined as

\begin{equation}
X(j \omega) = \int_{-\infty}^{\infty} x(t) \, e^{-j \omega t} \; dt
\end{equation}

where $X(j \omega) = \mathcal{F} \{ x(t) \}$ denotes the Fourier transform of $x(t)$. It is also termed as *spectrum* of the signal $x(t)$. The argument $j \omega$ copes for the fact that the kernel $e^{-j \omega t}$ of above [integral transform](https://en.wikipedia.org/wiki/Integral_transform) is a [harmonic exponential signal](../continuous_signals/standard_signals.ipynb#Complex-Exponential-Signal) $e^{-s t}$ with $s= j \omega$. Other common definitions of the Fourier transform may be found in the literature. They differ with respect to the sign of the exponential function and normalization factors. Sometimes the Fourier transform is evaluated in terms of the frequency $f$ instead of the angular frequency $\omega = 2 \pi f$. In both cases, the properties, theorems and transforms may differ from the ones derived in the following.

The inverse Fourier transform $x(t) = \mathcal{F}^{-1} \{ X(j \omega) \}$ is defined as

\begin{equation}
x(t) = \frac{1}{2 \pi} \int_{-\infty}^{\infty} X(j \omega) \, e^{j \omega t} \; d\omega
\end{equation}

A sufficient but not necessary condition for the existence of the Fourier transform may be derived from the absolute integrability of above Fourier integral

\begin{equation}
\int_{-\infty}^{\infty} | x(t) \, e^{-j \omega t}| \; dt \leq \int_{-\infty}^{\infty} | x(t) | \; dt < \infty
\end{equation}

where the triangle inequality and $|e^{-j \omega t}|=1$ have been used to find an upper limit of the first integral. Hence, the signal $x(t)$ has to be an [absolutely integrable function](https://en.wikipedia.org/wiki/Absolutely_integrable_function).

### Relation to the Laplace Transform

The [Laplace transform](../laplace_transform/definition.ipynb) $X(s) = \mathcal{L} \{ x(t) \}$ of a signal $x(t)$ can be related to its Fourier transform $X(j \omega) = \mathcal{F} \{ x(t) \}$ by inspecting the kernels of both transforms. The Laplace transform has the complex exponential function $e^{- s t}$ with $s \in \mathbb{C}$ as kernel. The Fourier transform, as introduced above, has the harmonic exponential function $e^{- j \omega t}$ with $\omega \in \mathbb{R}$ as kernel. Both can be related to each other by considering that $s = \sigma + j \omega$. Hence, if the ROC includes the imaginary axis $s = j \omega$ of the $s$-plane, the Fourier transform of a signal $x(t)$ can be derived from its Laplace transform by

\begin{equation}
\mathcal{F} \{ x(t) \} = \mathcal{L} \{ x(t) \} \bigr\rvert_{s = j \omega}
\end{equation}

If the ROC does not include the imaginary axis, the Fourier transform cannot be derived from the Laplace transform of $x(t)$. If the ROC includes the imaginary axis $s = j \omega$ and the integration path of the inverse Laplace transform is chosen along the imaginary axis $\sigma = 0$, the inverse Laplace transform becomes identical to the inverse Fourier transform.

A benefit of the Laplace over the Fourier transform is that it exists for a wider class of signals. The one-sided Laplace transform also allows to conveniently consider initial values of differential equations.

**Example - Fourier transform of causal exponential signal**

The Fourier transform $X(j \omega) = \mathcal{F} \{ x(t) \}$ of the causal exponential signal

\begin{equation}
x(t) = \epsilon(t) \cdot e^{- \alpha t}
\end{equation}

with $\alpha \in \mathbb{R}^+$ is derived from the Laplace transform of the [causal complex exponential signal](#Transformation-of-the-Causal-Complex-Exponential-Signal). Using the substitution $s_0 = \alpha$ yields

\begin{equation}
X(s) = \frac{1}{s + \alpha}
\end{equation}

with the ROC $\Re \{ s \} > - \alpha$. Hence for $0 < \alpha < \infty$, the ROC includes the imaginary axis. In this case the signal comprises an exponential signal whose level decays over time for $t>0$. The Fourier transform can be derived from the Laplace transform as

\begin{equation}
X(j \omega) = \frac{1}{j \omega + \alpha}
\end{equation}

### Interpretation of the Spectrum

It can be concluded from the inverse Fourier transformation

\begin{equation}
x(t) = \frac{1}{2 \pi} \int_{-\infty}^{\infty} X(j \omega) \, e^{j \omega t} \; d\omega
\end{equation}

that a signal $x(t)$ for which the Fourier transform $X(j \omega)$ exists can be represented as a superposition of weighted harmonic exponential signals $X(j \omega) \cdot e^{j \omega t}$. The angular frequency of these contributions can be related to the frequency by $f = \frac{\omega}{2 \pi}$.

The Fourier transform is in general complex valued, $X(j \omega) \in \mathbb{C}$. It is commonly represented by its real and imaginary part $X(j \omega) = \Re \{ X(j \omega) \} + j \cdot \Im \{ X(j \omega) \}$ or by its magnitude and phase $X(j \omega) = |X(j \omega)| \cdot e^{j \varphi(j \omega)}$. The magnitude spectrum $|X(\omega)|$ provides insights into the composition of an unknown signal in terms of the level of its harmonic contributions, the phase spectrum $\varphi(j \omega)$ into their phase relations. The Fourier transformation is therefore a valuable tool in the analysis of signals.

A general spectrum $X(j \omega)$ will have non-negative contributions for negative angular frequencies $\omega < 0$. The concept of [negative frequencies](https://en.wikipedia.org/wiki/Negative_frequency) has no physical meaning. However, in the context of the Fourier transformation with its complex kernel $e^{j \omega t}$ it is for instance required to express real-valued signals.

### Transformation of the Dirac Impulse

The transform $\mathcal{F} \{ \delta(t) \}$ of the [Dirac impulse](../continuous_signals/standard_signals.ipynb#Dirac-Impulse) is derived by introducing $\delta(t)$ into the definition of the Fourier transform and exploiting the sifting property of the Dirac delta function

\begin{equation}
\mathcal{F} \{ \delta(t) \} = \int_{-\infty}^{\infty} \delta(t) \, e^{-j \omega t} dt = 1
\end{equation}

The transform of the Dirac impulse is equal to one. Hence, all angular frequencies $\omega$ are present with equal weight. This is an important property in the theory of signals and systems, since the Dirac impulse is used to characterize linear time-invariant (LTI) systems by their [impulse response](../systems_time_domain/impulse_response.ipynb) $h(t) = \mathcal{H} \{ \delta(t) \}$.

**Example**

The Fourier transform of the Dirac impulse can be derived in `SymPy` by direct evaluation of above integral

```python
import sympy as sym
sym.init_printing()

t, w = sym.symbols('t omega', real=True)

X = sym.integrate(sym.DiracDelta(t)*sym.exp(-sym.I*w*t), (t, -sym.oo, sym.oo))
X
```

`SymPy` provides a [Fourier transform](http://docs.sympy.org/latest/modules/integrals/integrals.html?highlight=fourier#sympy.integrals.transforms.fourier_transform). However, it is defined in terms of the frequency $f$ instead of the angular frequency $\omega$. Therefore a custom function is defined which matches the definition used here

```python
def fourier_transform(x):
    return sym.integrals.transforms._fourier_transform(x, t, w, 1, -1, 'Fourier')
```

Applying this function to the Dirac impulse yields the same result as the direct evaluation of the integral above

```python
X = fourier_transform(sym.DiracDelta(t))
X
```

**Exercise**

* Derive the Fourier transform of a shifted Dirac impulse $\delta(t - \tau)$ either by manual evaluation of the integral or by modification of above example.

### Transformation of the Rectangular Signal

The Fourier transform $\mathcal{F} \{ \text{rect}(t) \}$ of the [rectangular signal](../continuous_signals/standard_signals.ipynb#Rectangular-Signal) is derived by explicit evaluation of the definition

\begin{equation}
\mathcal{F} \{ \text{rect}(t) \} = \int_{-\infty}^{\infty} \text{rect}(t) \, e^{-j \omega t} \; dt =
\int_{- \frac{1}{2}}^{\frac{1}{2}} e^{-j \omega t} \; dt = \frac{\sin(\frac{\omega}{2})}{\frac{\omega}{2}}
\end{equation}

The result is commonly abbreviated in terms of the [cardinal sine or sinc function](https://en.wikipedia.org/wiki/Sinc_function)

\begin{equation}
\text{sinc}(x) = \begin{cases}
\frac{sin(x)}{x} & \text{for } x \neq 0 \\
1 & \text{for } x = 0
\end{cases}
\end{equation}

Note, that the value at $x = 0$ has been derived by applying [L'Hôpital's rule](https://en.wikipedia.org/wiki/L'H%C3%B4pital's_rule) to $\frac{sin(x)}{x}$. The sinc function is an important function in (digital) signal processing and communications engineering. It shows the following properties

1. Its maximum value is one at $x=0$

2. Its amplitude decays with $\frac{1}{|x|}$

3. Its zeros are located at $x_0 = n \cdot \pi$ for $n \in \mathcal{Z} \setminus \{0\}$

4. $\int_{-\infty}^{\infty} \text{sinc}(t) \; dt = \pi$

Using the sinc function, the Fourier transform of $\text{rect}(t)$ reads

\begin{equation}
\mathcal{F} \{ \text{rect}(t) \} = \text{sinc}\left( \frac{\omega}{2} \right)
\end{equation}

The [Laplace transform of the rectangular signal](../laplace_transform/theorems.ipynb#Transformation-of-the-rectangular-signal) is given as

\begin{equation}
\mathcal{L} \{ \text{rect}(t) \} = \frac{\sinh \left( \frac{s}{2} \right) }{\frac{s}{2}}
\end{equation}

with the $\text{ROC} \{ \text{rect}(t) \} = \mathbb{C}$. The Laplace transform of the rectangular signal can be related to its Fourier transform by using $\sinh (s) = -j \sin(j s)$ with the substitution $s = j \omega$

\begin{equation}
\frac{\sinh \left( \frac{s}{2} \right) }{\frac{s}{2}} \Bigg\vert_{s= j \omega} = \frac{\sin \left( \frac{\omega}{2} \right) }{\frac{\omega}{2}}
\end{equation}

**Example**

The Fourier transform of the rectangular signal is computed in the following. First the rectangular signal is defined

```python
class rect(sym.Function):

    @classmethod
    def eval(cls, arg):
        return sym.Heaviside(arg + sym.S.Half) - sym.Heaviside(arg - sym.S.Half)
```

Now the Fourier transform is computed

```python
X = fourier_transform(rect(t))
X
```

```python
sym.plot(X, (w, -30, 30), xlabel=r'$\omega$', ylabel=r'sinc($\omega / 2$)');
```

**Exercise**

* Derive the Fourier transform of a scaled rectangular signal $\text{rect}(a t)$ with $a \in \mathbb{R}$ by manual evaluation of the Fourier integral or by modification of above example.
* What happens to the sinc function if you increase/decrease the width of the rectangular signal?

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Continuous- and Discrete-Time Signals and Systems - Theory and Computational Examples*.


```python
from IPython.display import HTML, Latex
import numpy as np
import pandas as pd
```

# Simulating the Apportionment of Congressional Seats

## Demo: What if D.C. or Puerto Rico Were States?

This demo recalculates Congressional apportionments based on the new [2020 Census decennial population count](https://www.census.gov/data/tables/2020/dec/2020-apportionment-data.html), supposing that the District of Columbia and/or Puerto Rico were also states

## Loading the Data

The [GitHub repo for this demo](https://github.com/TimeMagazineLabs/CongressionalApportionment) has data files from Census.gov for the 2010 and 2020 decennial Census counts as well as the official apportionments so that we can check our work. Here's what the data looks like:

```python
data_2020 = pd.read_csv('https://raw.githubusercontent.com/TimeMagazineLabs/CongressionalApportionment/main/data/apportionment_2020.csv',dtype={'State': 'string', 'Abbr': 'string', 'Reps': 'Int64'})
data_2020['Per_Rep'] = np.int64(data_2020['Population'] / data_2020['Reps'])
data_2020.head()
```

      State
      Abbr
      Year
      Population
      Reps
      Per_Rep

      0
      Alabama
      AL
      2020
      5030053
      7
      718579

      1
      Alaska
      AK
      2020
      736081
      1
      736081

      2
      Arizona
      AZ
      2020
      7158923
      9
      795435

      3
      Arkansas
      AR
      2020
      3013756
      4
      753439

      4
      California
      CA
      2020
      39576757
      52
      761091

## How Apportionment Works: The Equal Proportions Method

Since 1940, Congress [has used the "Equal Proportions Method"](https://www.census.gov/topics/public-sector/congressional-apportionment/about/computing.html) to apportion the 435 House seats to each state.

First, per the Constitution, every state gets one seat. For the remaining 385, each state's "priority" is measured by a simple formula that divides its population, as of the most recent decennial Census, by the square root of the product of the number of seats it currently has and that number plus 1 (the [geometric mean](https://mathworld.wolfram.com/GeometricMean.html)):
\begin{equation}
priority = \frac{P}{\sqrt{n*(n+1)}}
\end{equation}
For each seat, one at a time, the state with the highest priority is awarded that seat. Then its priority is recalculated and it moves further back in the line. Let's write a Python function to calculate the priority of a state, representated as a row in the table (aka "DataFrame") above. (We'll be adding a `RepsCalculated` column to compare to the official tally.)

```python
def EqualProportionsMethod(st):
  reps = st["RepsCalculated"]
  priority = st["Population"] / np.sqrt(reps * (reps + 1))
  return priority
```

## Recalculating the apportionments
Before we get fancy, let's recalculate the apportionment with the usual 50 states and make sure it matches the official tally. We'll make a copy of the data frame for each trial run and add columns for the calculated reps and the priority from the Equal Proportions Method algorithm, using the Pandas `.apply` method for DataFrames. Remember, each state starts with 1 seat, so we'll initialize `RepsCalculated` to 1 and start the apportionment with 385 seats remaining.

Since the data includes Puerto Rico and D.C., neither of which are eligible for reps, we'll need to remove them.

```python
data_2020_states_only = data_2020[~data_2020['State'].isin(["District of Columbia", "Puerto Rico"])]
data_2020_states_only.reset_index(drop=True)
print(data_2020.shape[0], data_2020_states_only.shape[0])
```

    52 50

```python
test_2020 = data_2020_states_only.copy()
test_2020['RepsCalculated'] = 1
test_2020['Priority'] = test_2020.apply(EqualProportionsMethod, 1)
test_2020.head()
```

      State
      Abbr
      Year
      Population
      Reps
      Per_Rep
      RepsCalculated
      Priority

      0
      Alabama
      AL
      2020
      5030053
      7
      718579
      1
      3.556785e+06

      1
      Alaska
      AK
      2020
      736081
      1
      736081
      1
      5.204879e+05

      2
      Arizona
      AZ
      2020
      7158923
      9
      795435
      1
      5.062123e+06

      3
      Arkansas
      AR
      2020
      3013756
      4
      753439
      1
      2.131047e+06

      4
      California
      CA
      2020
      39576757
      52
      761091
      1
      2.798499e+07

As expected, California has the highest priority, which we can determine with the `.idxmax()` method, which returns the index of the row with the highest value of a specified column

```python
test_2020.loc[test_2020['Priority'].idxmax(),'State']
```

    'California'

Great, now we just need a function to find the state with the highest priority, add a seat to that state, and recalculate it's priority. (It would be wasteful to recompute each state's priority after each assignment since the priority doesn't change unless a seat is added.) While we're at it, let's make a list called `ORDER` to see the order in which states get a representative. This should be initialized once per complete trial

```python
def addNextSeat(df, ORDER=[]):
  indexNext = df['Priority'].idxmax() # The index of the row with the highest priority
  # Add a seat to the state in that row
  df.loc[indexNext,'RepsCalculated'] += 1
  # Recompute the priority for this state
  df.loc[indexNext,'Priority'] = EqualProportionsMethod(df.loc[indexNext])
  # Add the state to the ORDER list
  ORDER.append(df.loc[indexNext,'Abbr'])
```

Let's see the function works. Updating DataFrames inside functions can be tricky.

```python
ORDER_TEST = []
addNextSeat(test_2020, ORDER_TEST)
print(ORDER_TEST)
test_2020.head()
```

    ['CA']

      State
      Abbr
      Year
      Population
      Reps
      Per_Rep
      RepsCalculated
      Priority

      0
      Alabama
      AL
      2020
      5030053
      7
      718579
      1
      3.556785e+06

      1
      Alaska
      AK
      2020
      736081
      1
      736081
      1
      5.204879e+05

      2
      Arizona
      AZ
      2020
      7158923
      9
      795435
      1
      5.062123e+06

      3
      Arkansas
      AR
      2020
      3013756
      4
      753439
      1
      2.131047e+06

      4
      California
      CA
      2020
      39576757
      52
      761091
      2
      1.615714e+07

Great! California got a seat, and there are now 384 seats left. Let's give the remaining ones a go.

```python
SEATS_LEFT = 384
while SEATS_LEFT > 0:
  addNextSeat(test_2020, ORDER_TEST)
  SEATS_LEFT -= 1

test_2020.head()
```

      State
      Abbr
      Year
      Population
      Reps
      Per_Rep
      RepsCalculated
      Priority

      0
      Alabama
      AL
      2020
      5030053
      7
      718579
      7
      672169.105833

      1
      Alaska
      AK
      2020
      736081
      1
      736081
      1
      520487.866603

      2
      Arizona
      AZ
      2020
      7158923
      9
      795435
      9
      754616.742459

      3
      Arkansas
      AR
      2020
      3013756
      4
      753439
      4
      673896.328360

      4
      California
      CA
      2020
      39576757
      52
      761091
      52
      753877.180693

```python
print(ORDER_TEST[0:10])
print(ORDER_TEST[-10:])
print("Last state to get a seat is", ORDER_TEST[-1])
```

    ['CA', 'TX', 'CA', 'FL', 'NY', 'TX', 'CA', 'PA', 'IL', 'CA']
    ['TX', 'IL', 'RI', 'AL', 'NC', 'OR', 'CO', 'CA', 'MT', 'MN']
    Last state to get a seat is MN

Looks good! We can double-check them all easily by taking the absolute value of the difference between our calculations from the official apportionment:

```python
test_2020['Error'] = np.abs(test_2020['Reps'] - test_2020['RepsCalculated'])
print("ERROR:", sum(test_2020['Error']))
test_2020.head()
```

    ERROR: 0

      State
      Abbr
      Year
      Population
      Reps
      Per_Rep
      RepsCalculated
      Priority
      Error

      0
      Alabama
      AL
      2020
      5030053
      7
      718579
      7
      672169.105833
      0

      1
      Alaska
      AK
      2020
      736081
      1
      736081
      1
      520487.866603
      0

      2
      Arizona
      AZ
      2020
      7158923
      9
      795435
      9
      754616.742459
      0

      3
      Arkansas
      AR
      2020
      3013756
      4
      753439
      4
      673896.328360
      0

      4
      California
      CA
      2020
      39576757
      52
      761091
      52
      753877.180693
      0

Let's wrap this whole process in a function for easy manipulation:

```python
def calculateApportionment(popData=data_2020, ignoreStates=['District of Columbia', 'Puerto Rico'], TOTAL_SEATS=435):
  ORDER_SEATS = []

  data_filtered = popData[~popData['State'].isin(ignoreStates)]
  data_filtered.reset_index(drop=True)

  test_apportionment = data_filtered.copy()
  test_apportionment['RepsCalculated'] = 1
  test_apportionment["Priority"] = test_apportionment.apply(EqualProportionsMethod, 1)

  SEATS_LEFT = TOTAL_SEATS - test_apportionment.shape[0] # This is the number of rows, which will account for any hypothetical states we add
  while SEATS_LEFT > 0:
    addNextSeat(test_apportionment, ORDER_SEATS)
    SEATS_LEFT -= 1

  return test_apportionment, ORDER_SEATS
```

## Now to Have Some Fun!

Let's start messing around with statehood and the total number of seats

```python
demo_2020_with_dc, _ = calculateApportionment(ignoreStates=['Puerto Rico'])
demo_2020_with_dc.head()
```

      State
      Abbr
      Year
      Population
      Reps
      Per_Rep
      RepsCalculated
      Priority

      0
      Alabama
      AL
      2020
      5030053
      7
      718579
      7
      672169.105833

      1
      Alaska
      AK
      2020
      736081
      1
      736081
      1
      520487.866603

      2
      Arizona
      AZ
      2020
      7158923
      9
      795435
      9
      754616.742459

      3
      Arkansas
      AR
      2020
      3013756
      4
      753439
      4
      673896.328360

      4
      California
      CA
      2020
      39576757
      52
      761091
      52
      753877.180693

## What's the difference?

Let's write a function to location which states lost seat(s) if DC was a state

```python
def compareToReality(df):
  df['Difference'] = df['RepsCalculated'] - df['Reps']
  df['Per_Rep_Computed'] = df['Population'] / df['RepsCalculated']

  changes = df[df['Difference'] != 0]
  for index, row in df.iterrows():
    if (row['Difference'] < 0):
      print('%s LOST %s seat(s)' % (row['State'], -row['Difference']))
    elif (row['Difference'] > 0):
      print('%s GAINED %s seat(s)' % (row['State'], row['Difference']))

```

Adding DC, which gets one seat, takes it's spot from Minnesota, which makes sense since Minnesota otherwise gets the last seat in the current apportionment

```python
compareToReality(demo_2020_with_dc)
```

    District of Columbia GAINED 1 seat(s)
    Minnesota LOST 1 seat(s)

What if we add a seat for DC?

```python
order_436 = []
demo_2020_with_dc_436, order_436 = calculateApportionment(ignoreStates=['Puerto Rico'], TOTAL_SEATS=436)
compareToReality(demo_2020_with_dc_436)
print(order_436[-1])
```

    District of Columbia GAINED 1 seat(s)
    MN

So adding D.C. _plus_ an extra seat for D.C. wouldn't alter any other apportionments, as we'd expect since D.C. would be the third smallest state:

```python
demo_2020_with_dc_436.sort_values("Population").head(10)[["State", "Population", "Year", "Reps"]]
```

      State
      Population
      Year
      Reps

      51
      Wyoming
      577719
      2020
      1

      46
      Vermont
      643503
      2020
      1

      8
      District of Columbia
      689545
      2020
      0

      1
      Alaska
      736081
      2020
      1

      34
      North Dakota
      779702
      2020
      1

      42
      South Dakota
      887770
      2020
      1

      7
      Delaware
      990837
      2020
      1

      26
      Montana
      1085407
      2020
      2

      40
      Rhode Island
      1098163
      2020
      2

      19
      Maine
      1363582
      2020
      2

And what if we add Puerto Rico but not DC?

```python
demo_2020_with_pr, _ = calculateApportionment(ignoreStates=['District of Columbia'])
compareToReality(demo_2020_with_pr)
```

    California LOST 1 seat(s)
    Colorado LOST 1 seat(s)
    Minnesota LOST 1 seat(s)
    Montana LOST 1 seat(s)
    Puerto Rico GAINED 4 seat(s)

Likewise, adding 4 seats for PR doesn't distrupt the apportionment otherwise

```python
demo_2020_with_pr_439, _ = calculateApportionment(ignoreStates=['District of Columbia'], TOTAL_SEATS=439)
compareToReality(demo_2020_with_pr_439)
```

    Puerto Rico GAINED 4 seat(s)

Here's DC and PR, without expanding the House

```python
demo_2020_with_dc_and_pr, _ = calculateApportionment(ignoreStates=[])
compareToReality(demo_2020_with_dc_and_pr)
```

    California LOST 1 seat(s)
    Colorado LOST 1 seat(s)
    District of Columbia GAINED 1 seat(s)
    Minnesota LOST 1 seat(s)
    Montana LOST 1 seat(s)
    Oregon LOST 1 seat(s)
    Puerto Rico GAINED 4 seat(s)

And making room at the table

```python
demo_2020_with_dc_and_pr_440, _ = calculateApportionment(ignoreStates=[], TOTAL_SEATS=440)
compareToReality(demo_2020_with_dc_and_pr_440)
```

    District of Columbia GAINED 1 seat(s)
    Puerto Rico GAINED 4 seat(s)

One last thing we can check: It's been reported that New York [very narrowly missed out on a seat by a different in 89 people. Would it have gotten the 436th seat?

```python
demo_2020_436, ORDER_436 = calculateApportionment(TOTAL_SEATS=436)
print(ORDER_436[-1])
```

    NY

Add if we give it an extra 90 people?

```python
data_2020_ny_plus90 = data_2020_states_only.copy()
nyIdx = data_2020_ny_plus90.loc[data_2020_ny_plus90['Abbr'] == 'NY'].index[0]
data_2020_ny_plus90.loc[nyIdx, 'Population'] += 90

test_NY90, _ = calculateApportionment(popData = data_2020_ny_plus90)
compareToReality(test_NY90)
```

    Minnesota LOST 1 seat(s)
    New York GAINED 1 seat(s)

It's true! Now let's move on to the next Notebook to try different seat values in more depth