# Testing Convergence and $\chi^2$ in Redundant Calibration

by Aaron Parsons and Josh Dillon

Redundant calibration boils down to solving a system of equations of the form:
\begin{equation}
V_{ij} = g_i g_j^* y_b
\end{equation}
where $V_{ij}$ is a measured visibility, $g_i,g_j$ are antenna gains ('$^*$' denotes complex conjugation), and $y_b$ is a ``model visibility" shared by all visibilities that are instantaneously redundant.

In order to solve this system of equations in practice, we iteratively linearize them, solve the linearized form, and then re-linearize around the improved solution.  Approaches to linearization discussed in Liu et al. (2010) include using the logarithm (we call the solver based on this method *logcal*) and first-order Taylor expansion, e.g.:
\begin{equation}
V_{ij} - g_i g_j^* y_b = (\Delta g_i) g_j^* y_b + (\Delta g_j^*) g_i y_b + (\Delta y_b) g_i g_j^*
\end{equation}

Once the system of equations has been linearized, in can generically be written as
\begin{equation}
Ax+n = y,
\end{equation}
which has the generic solution
\begin{equation}
\hat x = (A^\dagger N^{-1}A)^{-1} A^\dagger N^{-1} y,
\end{equation}
where $N=nn^\dagger$ is the noise covariance matrix.  The explicit construction and inversion of the matrix $A^\dagger N^{-1} A$ is computationally expensive, but can be worthwhile if the result can be used repeatedly.  This is the case for *logcal*, which can use the same matrix inversion to solve measurements made at different times and frequencies.  This makes it a relatively fast way to optimize a system of equations. However, *logcal* suffers from the deficiency that noise is not Gaussian under application of the logarithm, and so the results of *logcal* are biased.

Since Taylor expansion generates coefficients in $A$ that are generally unique to each instance of a system of equations ($A$ is different across times and frequency), solution by matrix inversion is typically very expensive.  The *LinProductSolver* method implements this approach.  It has attractive properties regarding convergence and lack of bias, but is too computationally expensive to be used in any but the most restricted cases.

## How the Omnical Algorithm Works

The *omnical* algorithm is a specialization of a linear solver for redundancy-based calibration (see, e.g., Liu et al. 2010) that avoids explicitly constructing and inverting the $A^\dagger N^{-1} A$ matrix for computational efficiency.  It was devised for the work presented in Zheng et al. (2014).

The *omnical* algorithm iteratively updates the estimate of the solution (the $g$'s and $y$'s) based on the current estimated values.  Namely, the updated value for $g_i$ is constructed as
\begin{equation}
g_i^\prime = g_i \frac{\sum_j{w_{ij}\frac{V_{ij}}{g_i g_j^* y_b}}}{\sum_j{w_{ij}}},
\end{equation}
for chosen weights $w_{ij}$ assigned to each measured visibility.  Similarly,
\begin{equation}
y_b^\prime = y_b \frac{\sum_{ij}{w_{ij}\frac{V_{ij}}{g_i g_j^* y_b}}}{\sum_j{w_{ij}}}.
\end{equation}
In practice, these iterations are dampened to prevent oscillation or divergence which can result from over-pursuing a solution.  Typically, each iteration is constructed as a weighted average of the current solution and estimated next step using a weighting factor $\epsilon$ with values in the range 0.1--0.5:
\begin{align}
g_{i,{\rm next}} &= (1-\epsilon)g_i + \epsilon g_i^\prime\\
y_{b,{\rm next}} &= (1-\epsilon)y_b + \epsilon y_b^\prime\\
\end{align}

The reason this expression works can be seen by assuming $V_{ij} = g_{i,{\rm true}} g_{j,{\rm true}}^* y_{b,{\rm true}} + n_{ij}$.  For simplicity, let's assume $w_{ij}=1$, although this is just for illustrative purposes.  In this case, the equation above becomes:
\begin{align}
g_i^\prime &= \frac{g_i}{N} \sum_j{\frac{g_{i,{\rm true}} g_{j,{\rm true}} y_{b,{\rm true}} + n_{ij}}{g_i g_j^* y_b}}\\
&= \frac{1}{N} g_{i,{\rm true}} \sum_j{\frac{g_{j,{\rm true}} y_{b,{\rm true}} + n_{ij}}{g_j^* y_b}}\\
&\approx g_{i,{\rm true}} \left(1+\frac{1}{N}\sum_j{\frac{n_{ij}}{g_j^* y_b}}\right),
\end{align}
where $N$ is the number of equations involving $i$, and the final step approximates that $g_{j,{\rm true}}/g_j\approx 1$ and $y_{b,{\rm true}}/y_b\approx 1$.
This algorithm extrapolates to $g_j$ (just conjugate $V_{ij}$ to put $g_j$ out front) and $y_b$ in the same manner.

In choosing a weight for each $V_{ij}$, an optimal choice for maximum sensitivity would be to assign $w_{ij}\propto {\rm SNR}^2$, or
\begin{equation}
w_{ij}=\frac{|g_{i,{\rm true}} g_{j,{\rm true}}^* y_{b,{\rm true}}|^2}{\langle|n_{ij}|^2\rangle}.
\end{equation}
However, we do not know the true solutions yet.  Our best option is to use our current estimate of the solution to choose the weighting, and then update the weights as we improve our estimate of the solution.

The implementation of *omnical* that currently exists in *hera_cal.redcal* makes the approximation that
\begin{equation}
w_{ij}\approx |g_i g_j^* y_b|^2 w_{ij,{\rm user}},
\end{equation}
where $g_i$, $g_j$, and $y_b$ are the current best estimates for the true parameter values, and $w_{ij,{\rm user}}$ is a user-supplied weighting for each measurement that ideally should be proportional to $|n_{ij}|^{-2}$.  These weights are used to update estimates of $g_i$, $g_j$, and $y_b$ for a series of minor-cycle iterations, and then the weights themselves are updated in a major-cycle iteration that is more computationally expensive.

# Application of Redundant Calibration Solvers

```python
%matplotlib inline
from __future__ import print_function
import hera_cal.redcal as om
import hera_cal.omni
import omnical.calib
import numpy as np
import time
import uvtools
import pylab as plt
from scipy.stats import kde, chi2
```

```python
def build_linear_array(nants, sep=14.7):
    antpos = {i: np.array([sep * i, 0, 0]) for i in range(nants)}
    return antpos

def build_hex_array(hexNum, sep=14.7):
    antpos, i = {}, 0
    for row in range(hexNum - 1, -(hexNum), -1):
        for col in range(2 * hexNum - abs(row) - 1):
            xPos = ((-(2 * hexNum - abs(row)) + 2) / 2.0 + col) * sep
            yPos = row * sep * 3**.5 / 2
            antpos[i] = np.array([xPos, yPos, 0])
            i += 1
    return antpos
```

```python
np.random.seed(0)
SHAPE = (60,1024)
NANTS = 18
NOISE = 1e-3

antpos = build_linear_array(NANTS)
reds = om.get_reds(antpos, pols=['xx'], pol_mode='1pol')
info = om.RedundantCalibrator(reds)

gains, true_vis, d = om.sim_red_data(reds, shape=SHAPE, gain_scatter=.01)
d = {key: value.astype(np.complex64) for key,value in d.items()}
d_nos = {key: value + NOISE * om.noise(value.shape) for key,value in d.items()}
d_nos = {key: value.astype(np.complex64) for key,value in d_nos.items()}
w = dict([(k, np.float32(1.)) for k in d.keys()])
sol0 = dict([(k, np.ones_like(v)) for k, v in gains.items()])
sol0.update(info.compute_ubls(d, sol0))
sol0 = {k:v.astype(np.complex64) for k,v in sol0.items()}
```

```python
def gen_chisq(data, sol, noise):
    d_mdl = {}
    for bls in reds:
        if len(bls) <= 1: continue
        ubl = sol[bls[0]]
        for bl in bls:
            d_mdl[(bl[0],bl[1],'xx')] = \
                sol[(bl[0],'x')] * sol[(bl[1],'x')].conj() * ubl
    res = [data[k] - v for k,v in d_mdl.items()]
    chisq = np.sum(np.abs(res)**2 / noise**2, axis=0) / (len(res) - len(sol))
    return chisq
```

```python
def plot_gain_residuals(sol, true_gains, lim, nbins=20):
    sol_degen = info.remove_degen(antpos, sol, degen_sol=true_gains)
    keys = [k for k in sol.keys() if len(k) == 2]
    for k in keys:
        res = sol_degen[k] - true_gains[k]
        res = res.flatten()
        res = res[np.where(np.isfinite(res))]

        ax = plt.subplot(121)
        plt.grid()
        _ = plt.plot(res.real, res.imag, '.', alpha=.2)
        res = np.array([res.real, res.imag])
        x,y = res
        k = kde.gaussian_kde(res)
        xi, yi = np.mgrid[x.min():x.max():nbins*1j, y.min():y.max():nbins*1j]
        zi = k(np.vstack([xi.flatten(), yi.flatten()]))
        plt.xlabel('$\Delta$g, real')
        plt.ylabel('$\Delta$g, imag')
        plt.xlim(-lim,lim); plt.ylim(-lim,lim)
        ax.grid(b=True, which='major', color='k', linestyle='-')

        ax2 = plt.subplot(122, sharey=ax)
        plt.grid()
        _ = plt.contour(xi, yi, zi.reshape(xi.shape), alpha=.5)
        plt.xlabel('$\Delta$g, real')
        plt.xlim(-lim,lim)
        plt.setp(ax2.get_yticklabels(), visible=False)
        ax2.grid(b=True, which='major', color='k', linestyle='-')
```

## LOGCAL

```python
# LOGCAL
t0 = time.time()
sol_logcal = info.logcal(d, wgts=w)
print('LOGCAL: %4.1f s' % (time.time() - t0),
      sol_logcal.values()[0].dtype)
t0 = time.time()
sol_nos_logcal = info.logcal(d_nos, wgts=w)
print('LOGCAL: %4.1f s' % (time.time() - t0),
     sol_nos_logcal.values()[0].dtype)
```

    LOGCAL:  2.0 s complex64
    LOGCAL:  1.9 s complex64

```python
plt.figure(figsize=(8,4))
plot_gain_residuals(sol_nos_logcal, gains, 0.02)
```

This figure, which will be reproduced for each solver implementation below, illustrates the distribution of residual errors in the complex plane for estimated parameter values over 61,440 repeated solutions of the same system of equations with independent noise draws.  Color in the left panel indicates antenna index.  The right panel shows density contours for each antenna's gain parameters, with color indicating higher value contours.

By eye, *logcal*'s output appears roughly unbiased (centered around 0), but has a non-Gaussian distribution.

## OMNICAL (linsolve based)

```python
# OMNICAL
t0 = time.time()
meta_omnical, sol_omnical = info.omnical(d, sol0,
        gain=.5, maxiter=500, check_after=30, check_every=6)
print('OMNICAL: %4.1f s' % (time.time() - t0),
     sol_omnical.values()[0].dtype)
t0 = time.time()
meta_nos_omnical, sol_nos_omnical = info.omnical(d_nos, sol0,
        gain=.5, maxiter=500, check_after=30, check_every=6)
print('OMNICAL: %4.1f s' % (time.time() - t0),
     sol_nos_omnical.values()[0].dtype)
```

    OMNICAL: 17.4 s complex64
    OMNICAL: 11.2 s complex64

```python
plt.figure()
plot_gain_residuals(sol_nos_omnical, gains, 0.001)
```

The figure for *linsolve*-based *omnical* shows a tighter and more Gaussian distribution of residuals relative to *logcal* above.  The contours in the right panel overlap for all antennas and are centered at zero, indicating a roughly unbiased and consistent error distribution for all antennas.

## LinProductSolver

*LinProductSolver* is the full matrix-inversion form of the Taylor-series linearization solver.  It is quite slow, so much so that to have this converge in a reasonable amount of time, I actually initialize it with a much better guess than all other solvers.  It is initialized with the converged solution from *logcal* above.

```python
# LinProductSolver (only solving 1/60th of the equations)
# XXX lincal doesn't converge for dtype complex64 because of np.linalg.lstsq shenanigans
#dtype, conv_crit, mode = np.complex64, 1e-7, 'pinv'
dtype, conv_crit, mode = np.complex128, 1e-10, 'lsqr'
t0 = time.time()
_d = {k:v[:1].astype(dtype) for k,v in d.items()}
_sol = {k:v[:1].astype(dtype) for k,v in sol_logcal.items()}
meta_linsolve, sol_linsolve = \
    info.lincal(_d, _sol, maxiter=10, conv_crit=conv_crit, mode=mode)
print('LINPRODUCTSOLVER: %4.1f s' % (time.time() - t0),
      sol_linsolve.values()[0].dtype)
t0 = time.time()
_d_nos = {k:v[:1].astype(dtype) for k,v in d_nos.items()}
_sol_nos = {k:v[:1].astype(dtype) for k,v in sol_nos_logcal.items()}
meta_nos_linsolve, sol_nos_linsolve = \
    info.lincal(_d_nos, _sol_nos, maxiter=10, conv_crit=conv_crit, mode=mode)
print('LINPRODUCTSOLVER: %4.1f s' % (time.time() - t0),
      sol_nos_linsolve.values()[0].dtype)
```

    LINPRODUCTSOLVER:  3.9 s complex128
    LINPRODUCTSOLVER:  7.7 s complex128

```python
plt.figure()
_gains = {k:v[:1] for k,v in gains.items()}
plot_gain_residuals(sol_nos_linsolve, _gains, .001)
```

The figure above for *LinProductSolver*'s full *linsolve*-based matrix inversion solution looks nearly identical to the *omnical* version above.

## OMNICAL (legacy)

The legacy implementation of *omnical*, written in C and wrapped into Python, has several known shortcomings, including an apparent bias in the solutions discovered by Wenyang Li.  Nonetheless, is has been our reference implementation up to this point, so it is included here both as a performance benchmark and as a comparison for whether final solutions are biased.

```python
# OMNICAL legacy
hcreds = om.get_reds(antpos, pols=['xx'], pol_mode='1pol')
pols = ['x']
antpos_ideal = np.array(antpos.values())
xs, ys, zs = antpos_ideal.T
layout = np.arange(len(xs))
_antpos = -np.ones((NANTS * len(pols), 3))
for ant, x, y in zip(layout.flatten(), xs.flatten(), ys.flatten()):
    for z, pol in enumerate(pols):
        z = 2**z  # exp ensures diff xpols aren't redundant w/ each other
        i = hera_cal.omni.Antpol(ant, pol, NANTS)
        _antpos[int(i), 0], _antpos[int(i), 1], _antpos[int(i), 2] = x, y, z
_reds = hera_cal.omni.compute_reds(NANTS, pols, _antpos[:NANTS], tol=.01)
_info = hera_cal.omni.RedundantInfo(NANTS)
_info.init_from_reds(_reds, antpos_ideal)
_d, _d_nos = {}, {}
for key in d.keys():
    if not _d.has_key(key[:2]):
        _d[key[:2]] = {}
        _d_nos[key[:2]] = {}
    _d[key[:2]][key[-1]] = d[key]
    _d_nos[key[:2]][key[-1]] = d_nos[key]

t0 = time.time()
meta_legacy, g, v = omnical.calib.lincal(_d, _info, maxiter=150)
print('OMNICAL: %4.1f s' % (time.time() - t0),
     g.values()[0].values()[0].dtype)

t0 = time.time()
meta_nos_legacy, g_nos, v_nos = omnical.calib.lincal(_d_nos, _info, maxiter=150)
print('OMNICAL: %4.1f s' % (time.time() - t0),
     g_nos.values()[0].values()[0].dtype)
```

    OMNICAL: 110.5 s complex64
    OMNICAL: 44.7 s complex64

```python
def legacy2sol(g,v):
    sol = {}
    for i in g['x'].keys():
        sol[(i,'x')] = g['x'][i]
    for bls in _reds:
        i,j = int(bls[0][0]), int(bls[0][1])
        sol[(i,j,'xx')] = v['xx'][(i,j)]
    return sol

sol_legacy = legacy2sol(g,v)
sol_nos_legacy = legacy2sol(g_nos,v_nos)
```

```python
plt.figure()
plot_gain_residuals(sol_nos_legacy, gains, 0.001)
```

The residuals plotted for the legacy implementation of *omnical* look roughly consistent with *LinProductSolver* and the *linsolve*-based *omnical* version, but close examination of the right panel shows that the individual solutions for each antenna are not individually centered at zero. This is our first indication of a possible bias in the *legacy* implementation of *omnical*.

# Comparing Results

## Computing Time

$\begin{array}{lrr}
  & {\rm Noiseless} & {\rm Noisy} \\
  \hline
  {\rm logcal} & 1.9s & 1.9s \\
  {\rm omnical~(linsolve)} & 17.6s & 11.4s \\
  {\rm omnical~(legacy~C)} &  108.1s &  44.6s \\
  {\rm LinProductSolver}^\dagger & \sim250s & \sim601s \\
\end{array}$

$~$

Running from the same starting point with identical data, we have the following results.  We include the run-time of *LinProductSolver*, but mark it with $\dagger$ to indicate that it was given a better starting position than the other solvers and used double precision.  The increase in run-time in the noisy case for *LinProductSolver* is likely a result of the starting position being much worse from *logcal*, rather than being a result of the solution algorithm performing worse.

Note that the *linsolve*-based implementation of *omnical* runs faster than the C implementation because of slight algorithmic differences, including reuse of computed values between different parameter estimators and the introduction of major/minor-cycle computing cycles which allows the reuse of computed values between solver iterations.

Convergence is defined differently between the two *omnical* implementations.  For the legacy C version it was defined by the fractional change in the $\chi^2$ between iterations.  For the *linsolve* version, it is defined by the fractional change in the estimated parameter values between iterations.  In practice, it seems that the *linsolve* version more reliably converges within the maximum number of iterations.  Also, because of performance differences, the *linsolve* version can be allowed to run for more iterations, which also tends to ensure better convergence.

## Checking Convergence in $\chi^2$

For the systems of equations with noise, we can check the distribution of $\chi^2$ for the solved values.  For this idealized calibration case, convergence should result in a $\chi^2$ distribution that closely resembles the ideal analytic case.

```python
chisq_logcal = gen_chisq(d_nos, sol_nos_logcal, NOISE)
chisq_omnical = gen_chisq(d_nos, sol_nos_omnical, NOISE)
chisq_linsolve = gen_chisq(d_nos, sol_nos_linsolve, NOISE)
chisq_legacy = gen_chisq(d_nos, sol_nos_legacy, NOISE)

plt.figure()
for label in ['logcal','omnical','linsolve','legacy']:
    chisq = eval('chisq_' + label)
    hist,bins = np.histogram(chisq, bins=100, range=(0,5))
    _ = plt.semilogy(0.5*(bins[1:] + bins[:-1]), hist, label=label)
sol = sol_omnical
df = 2*(len(d) - len(sol))
x = 0.5*(bins[1:]+bins[:-1])
plt.plot(x, 7e5*chi2.pdf(x*df, df), label=r'$\chi^2$ PDF')
plt.grid()
plt.legend()
plt.ylim(1e0,3e4)
plt.xlim(.5,2.5)
_ = plt.xlabel('Chi-Square')
_ = plt.ylabel('Bin Count')
```

As we can see in the figure above, *logcal* deviates significantly from the ideal $\chi^2$ distribution.  This is expected.  The other algorithms all achieve similar $\chi^2$ distributions.  These distributions closely resemble the ideal $\chi^2$ distribution, but have a slight excess of values at higher $\chi^2$.

## Checking for Bias in Solutions

Even though an individual solution to a system of equations may be consistent with the true answer to within the noise, there may be a low-level bias that becomes visible when we average many repeated solutions together.  Here we compute $|\langle g_i-g_{i,{\rm true}}\rangle|$, the magnitude of the averaged residual deviation from the truth, and compare that to $\sqrt{\langle|g_i-g_{i,{\rm true}}|^2\rangle}/\sqrt{N}$, where $N$ is the number of measurements averaged over.  Ideally, the averaged residual deviation should integrate down in accordance with Gaussian statistics, with the result that the two values should be comparable.  If the averaged residual deviation is in excess of the expected level of fluctuations predicted by Gaussian statistics, that would be an indication of bias.

```python
plt.figure()
for i,label in enumerate(['logcal','omnical','linsolve','legacy']):
    sol = eval('sol_nos_'+label)
    sol_degen = info.remove_degen(antpos, sol, degen_sol=gains)
    keys = [k for k in sol.keys() if len(k) == 2]
    res = [sol_degen[k] - gains[k] for k in keys]
    ants = [k[0] for k in keys]
    avg = [np.abs(np.average(r)) for r in res]
    err = [np.sqrt(np.average(np.abs(r)**2) / r.size) for r in res]
    color = 'rgbk'[i]
    _ = plt.semilogy(ants, avg, color+'.', label=label+' <$x-x_0$>')
    _ = plt.semilogy(ants, err, color+'x', label=label+' $\sigma$')
plt.ylabel('Residual Deviation')
plt.xlabel('Antenna Index')
plt.legend()
plt.grid()
```

In the figure above, we see that the expected level for residuals obeying Gaussian statistics (indicated by 'X' for each antenna) are higher for the *logcal* case, but are consistent for all the remaining cases.  This indicates the the *logcal* residuals are larger to begin with.  The actual averaged residuals (indicated by '.' for each antenna) are roughly consistent with the predicted noise level for the *logcal*, *linsolve*=*LinProductSolver*, and *omnical* cases, but not for the *legacy* C implementation of the *omnical* algorithm.  Note that the *omnical* and *linsolve* residuals match nearly exactly.

This seems to reproduce Wenyang Li's result that the legacy implementation of *omnical* is indeed biased in its final results, even though each solution is consistent with the truth to within the noise.

## Checking Noise-free Convergence

```python
for i,label in enumerate(['logcal','omnical','linsolve','legacy']):
    sol = eval('sol_'+label)
    sol_degen = info.remove_degen(antpos, sol, degen_sol=gains)
    keys = [k for k in sol.keys() if len(k) == 2]
    res = [sol_degen[k] - gains[k] for k in keys]
    print(label,sol.values()[0].dtype, np.sqrt(np.average(np.abs(res)**2)))
```

    logcal complex64 1.51592919317e-07
    omnical complex64 1.87193822464e-07
    linsolve complex128 8.19520014588e-09
    legacy complex64 3.67838346406e-07

It seems that all the solvers can converge on the true solution in the noise-free case with better than 4e-7 precision.  The *LinProductSolver* based method does better, but remember that it was given double precision data.

## Checking Repeatability from Different Starting Points

Here we just check that the omnical solver we intend to use going forward can converge to the same solution from different starting points.

```python
sols = {0:sol_nos_omnical}
for i in range(1,4):
    # Build an alternate starting point for solving
    sol1 = {k:(v + NOISE * om.noise(v.shape)).astype(np.complex64)
            for k,v in sol0.items()}
    _, sols[i] = info.omnical(d_nos, sol1,
            gain=.5, maxiter=500, check_after=30, check_every=6)
```

```python
plt.figure()
for i in range(4):
    sol = sols[i]
    if i == 0:
        label1, label2 = '<$x-x_0$>', '$\sigma$'
    else:
        label1, label2 = '', ''
    sol_degen = info.remove_degen(antpos, sol, degen_sol=gains)
    keys = [k for k in sol.keys() if len(k) == 2]
    res = [sol_degen[k] - gains[k] for k in keys]
    ants = [k[0] for k in keys]
    avg = [np.abs(np.average(r)) for r in res]
    err = [np.sqrt(np.average(np.abs(r)**2) / r.size) for r in res]
    color = 'rgbk'[i]
    _ = plt.semilogy(np.array(ants)+.1*i, avg, color+'.', label=label1)
    _ = plt.semilogy(np.array(ants)+.1*i, err, color+'x', label=label2)
plt.ylabel('Residual Deviation')
plt.xlabel('Antenna Index')
plt.legend()
plt.grid()
```

So repeatability looks very good.


## MSW Triangle

Paremeters to be used

1. mixing angle $\theta$
2. mass difference $\Delta m^2$
3. energy of neutrinos $E$
4. matter profile $n_e(x)$

Survival probability is given by

\begin{equation}
P_{\nu_e\to\nu_e} =  \frac{1}{2} + \left( \frac{1}{2} - P_f \right) \cos 2\theta \cos 2\theta_m,
\end{equation}

where

\begin{equation}
P_f = e^{-\frac{\pi}{2}\gamma},
\end{equation}

and

\begin{equation}
\gamma = \frac{\Delta m^2 \sin^2 2\theta}{2E \cos 2\theta \left\lvert \frac{1}{n_e} \frac{dn_e}{dr} \right\rvert } = \frac{\omega \sin^2 2\theta}{\cos 2\theta \left\lvert \frac{1}{n_e} \frac{dn_e}{dr} \right\rvert }  = \frac{\omega \sin^2 2\theta}{\cos 2\theta \left\lvert \frac{1}{\hat\lambda} \frac{d\hat\lambda}{dr} \right\rvert }
\end{equation}

in which

\begin{equation}
\cos 2\theta_m = \frac{\cos 2\theta - \hat\lambda}{\sqrt{\hat\lambda^2 + 1 - 2\hat\lambda \cos 2\theta}},
\end{equation}

What we would like to produce is the survival probability of electron neutrinos as a heat map with two parameters mass difference $\Delta m^2$ and $\sin^22\theta$ at each distance $x$.

The formula we have above is only valid far away from the resonance where the Landau-Zener transition happens.

The gamma for our sun is

\begin{equation}
\gamma_S = 3.36\times 10^{14} \omega \frac{\sin^22\theta}{\cos2\theta} = 2.554\times 10^3 \frac{\sin^22\theta}{\cos2\theta} \frac{1}{\hat\lambda}
\end{equation}

```python
%matplotlib inline
```

```python
import numpy as np
import sys
sys.path.insert(0,'../../module')
import neuosc as ns
import sympy as sp
import matplotlib.pyplot as plt
```

```python
figureSize = (13,9)
```

```python
reload(ns)
```

    <module 'neuosc' from '../../module/neuosc.py'>

```python
neu_e = 1000000  #eV
msw_eff = ns.MSW(neu_e)
```

```python
msw_eff.adiabatic_p(0.1,0)
```

    0.98026524850072128

```python
size = [200,100]

sinma = np.logspace(-4, 0, num=size[0])
lam = np.logspace(6, -1, size[1])
```

```python
su_prob = [[0 for x in range(size[0])] for x in range(size[1])]

md = 0

for matter in lam:
    agl = 0
    for sin22theta in sinma:
        angle = np.arcsin( np.sqrt(sin22theta) )/2
        su_prob[md][agl] = msw_eff.solar_landau_zener_p(angle, matter)#adiabatic_p(angle, matter)
        agl = agl + 1
    md = md + 1

```

```python
sinma, lam = np.meshgrid(np.log10(sinma), np.log10(lam) )
plt.figure(figsize=figureSize)
plt.pcolormesh(sinma, lam, np.array(su_prob) )
plt.colorbar() #need a colorbar to show the intensity scale
plt.show() #boom
```

```python

```


```python
from statsmodels.tsa.seasonal import seasonal_decompose
import matplotlib.pyplot as plt
import scipy.stats as st
import seaborn as sns
import pandas as pd
from scipy.stats import norm, uniform, expon, t, probplot
import scipy.stats as st
from scipy.integrate import quad
from sympy.solvers import solve
from sympy import Symbol
import numpy as np
from pandas import Series, DataFrame
from statsmodels.graphics.tsaplots import plot_acf
from statsmodels.tsa.stattools import acf
from pandas.plotting import lag_plot
```

```python
Tesla = pd.read_csv("Tesla.csv", sep="\t",header=0)
Tesla["Date"] = pd.DatetimeIndex(Tesla["Date"])
Tesla.set_index("Date", inplace=True)
Tesla["log_volume"] = np.log(Tesla["Volume"])
Tesla["log_return"] = Tesla["log_volume"] - Tesla["log_volume"].shift(1)
Tesla["log_return"].plot()
```

```python
Tesla = pd.read_csv("Tesla.csv", sep="\t",header=0)
Tesla["Date"] = pd.DatetimeIndex(Tesla["Date"])
Tesla.set_index("Date", inplace=True)
Tesla["log_volume"] = np.log(Tesla["Volume"])
Tesla["log_return"] = Tesla["log_volume"] - Tesla["log_volume"].shift(1)
fig, (ax1, ax2) = plt.subplots(ncols=2)
Tesla["log_return"].plot(ax=ax1)
plot_acf(DataFrame(Tesla["log_return"]).dropna(), lags=120, ax=ax2)
```

```python

```


# Random Signals

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Cumulative Distribution Functions

A random process can be characterized by the statistical properties of its amplitude values. [Cumulative distribution functions](https://en.wikipedia.org/wiki/Cumulative_distribution_function) (CDFs) are one possibility to do so.

### Univariate Cumulative Distribution Function

The univariate CDF $P_x(\theta, k)$ of a continuous-amplitude real-valued random signal $x[k]$ is defined as

\begin{equation}
P_x(\theta, k) := \Pr \{ x[k] \leq \theta\}
\end{equation}

where $\Pr \{ \cdot \}$ denotes the probability that the given condition holds. The univariate CDF quantifies the probability that for the entire ensemble and for a fixed time index $k$ the amplitude $x[k]$ is smaller or equal to $\theta$. The term '*univariate*' reflects the fact that only one random process is considered.

The CDF shows the following properties which can be concluded directly from its definition

\begin{equation}
\lim_{\theta \to -\infty} P_x(\theta, k) = 0
\end{equation}

and

\begin{equation}
\lim_{\theta \to \infty} P_x(\theta, k) = 1
\end{equation}

The former property results from the fact that all amplitude values $x[k]$ are larger than $- \infty$, the latter from the fact that all amplitude values lie within $- \infty$ and $\infty$. The univariate CDF $P_x(\theta, k)$ is furthermore a non-decreasing function

\begin{equation}
P_x(\theta_1, k) \leq  P_x(\theta_2, k) \quad \text{for } \theta_1 \leq \theta_2
\end{equation}

The probability that $\theta_1 < x[k] \leq \theta_2$ is given as

\begin{equation}
\Pr \{\theta_1 < x[k] \leq \theta_2\} = P_x(\theta_2, k) - P_x(\theta_1, k)
\end{equation}

Hence, the probability that a continuous-amplitude random signal takes a specific value $x[k]=\theta$ is zero when calculated by means of the CDF. This motivates the definition of probability density functions introduced later.

### Bivariate Cumulative Distribution Function

The statistical dependencies between two signals are frequently of interest in statistical signal processing. The bivariate or joint CDF $P_{xy}(\theta_x, \theta_y, k_x, k_y)$ of two continuous-amplitude real-valued random signals $x[k]$ and $y[k]$ is defined as

\begin{equation}
P_{xy}(\theta_x, \theta_y, k_x, k_y) := \Pr \{ x[k_x] \leq \theta_x  \wedge y[k_y] \leq \theta_y  \}
\end{equation}

The joint CDF quantifies the probability for the entire ensemble of sample functions that for a fixed $k_x$ the amplitude value $x[k_x]$ is smaller or equal to $\theta_x$ and that for a fixed $k_y$ the amplitude value $y[k_y]$ is smaller or equal to $\theta_y$. The term '*bivariate*' reflects the fact that two random processes are considered. The following properties can be concluded from its definition

\begin{align}
\lim_{\theta_x \to -\infty} P_{xy}(\theta_x, \theta_y, k_x, k_y) &= 0 \\
\lim_{\theta_y \to -\infty} P_{xy}(\theta_x, \theta_y, k_x, k_y) &= 0
\end{align}

and

\begin{equation}
\lim_{\substack{\theta_x \to \infty \\ \theta_y \to \infty}} P_{xy}(\theta_x, \theta_y, k_x, k_y) = 1
\end{equation}

The bivariate CDF can also be used to characterize the statistical properties of one random signal $x[k]$ at two different time-instants $k_x$ and $k_y$ by setting $y[k] = x[k]$

\begin{equation}
P_{xx}(\theta_1, \theta_2, k_1, k_2) := \Pr \{ x[k_1] \leq \theta_1  \wedge y[k_2] \leq \theta_2  \}
\end{equation}

The definition of the bivariate CDF can be extended straightforward to the case of more than two random variables. The resulting CDF is termed as multivariate CDF.

## Probability Density Functions

[Probability density functions](https://en.wikipedia.org/wiki/Probability_density_function) (PDFs) describe the probability for one or multiple random signals to take on a specific value. Again the univariate case is discussed first.

### Univariate Probability Density Function

The univariate PDF $p_x(\theta, k)$ of a continuous-amplitude real-valued random signal $x[k]$ is defined as the derivative of the univariate CDF

\begin{equation}
p_x(\theta, k) = \frac{\partial}{\partial \theta} P_x(\theta, k)
\end{equation}

This can be seen as the differential equivalent of the limit case $\lim_{\theta_2 \to \theta_1} \Pr \{\theta_1 < x[k] \leq \theta_2\}$. As a consequence of above definition, the CDF can be computed from the PDF by integration

\begin{equation}
P_x(\theta, k) = \int\limits_{-\infty}^{\theta} p_x(\alpha, k) \, \mathrm{d}\alpha
\end{equation}

Due to the properties of the CDF and its definition, the PDF shows the following properties

\begin{equation}
p_x(\theta, k) \geq 0
\end{equation}

and

\begin{equation}
\int\limits_{-\infty}^{\infty} p_x(\theta, k) \, \mathrm{d}\theta = \lim_{\theta \to \infty} P_x(\theta, k) = 1
\end{equation}

The univariate PDF has only positive values and the area below the PDF is equal to one. The latter property may be used for normalization.

#### Example - Estimate of an univariate PDF by the histogram

In the process of calculating a [histogram](https://en.wikipedia.org/wiki/Histogram), the entire range of amplitude values of a random signal is split into a series of intervals (bins). For a given random signal the number of samples is counted which fall into one of these intervals. This is repeated for all intervals. The counts are finally normalized with respect to the total number of samples. This process constitutes a numerical estimation of the PDF of a random process.

In the following example the histogram of an ensemble of random signals is computed for each time index $k$. The CDF is computed by taking the cumulative sum over the histogram bins. This constitutes a numerical approximation of above integral

\begin{equation}
\int\limits_{-\infty}^{\theta} p_x(\alpha, k) \, \mathrm{d}\alpha \approx \sum_{i=0}^{N} p_x(\theta_i, k) \, \Delta\theta_i
\end{equation}

where $p_x(\theta_i, k)$ denotes the $i$-th bin of the PDF and $\Delta\theta_i$ its width.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

K = 32  # number of temporal samples
N = 10000  # number of sample functions
bins = 100  # number of bins for the histogram

# draw sample functions from a random process
np.random.seed(2)
x = np.random.normal(size=(N, K))
x += np.tile(np.cos(2*np.pi/K*np.arange(K)), [N, 1])

# compute the histogram
px = np.zeros((bins, K))
for k in range(K):
    px[:, k], edges = np.histogram(x[:, k], bins=bins, range=(-4,4), density=True)

# compute the CDF
Px = np.cumsum(px, axis=0) * 8/bins

# plot the PDF
plt.figure(figsize=(10,6))
plt.pcolor(np.arange(K), edges, px)
plt.title(r'Estimated PDF $\hat{p}_x(\theta, k)$')
plt.xlabel(r'$k$')
plt.ylabel(r'$\theta$')
plt.colorbar()
plt.autoscale(tight=True)

# plot the CDF
plt.figure(figsize=(10,6))
plt.pcolor(np.arange(K), edges, Px, vmin=0, vmax=1)
plt.title(r'Estimated CDF $\hat{P}_x(\theta, k)$')
plt.xlabel(r'$k$')
plt.ylabel(r'$\theta$')
plt.colorbar()
plt.autoscale(tight=True)
```

**Exercise**

* Change the number of sample functions `N` or/and the number of `bins` and rerun the examples. What changes? Why?

Solution: In numerical simulations of random processes only a finite number of sample functions and temporal samples can be considered. This holds also for the number of intervals (bins) used for the histogram. As a result, numerical approximations of the CDF/PDF will be subject to statistical uncertainties that typically will become smaller if the number of sample functions `N` is increased.

### Bivariate Probability Density Function

The bivariate or joint PDF $p_{xy}(\theta_x, \theta_y, k_x, k_y)$ of two continuous-amplitude real-valued random signals $x[k]$ and $y[k]$ is defined as

\begin{equation}
p_{xy}(\theta_x, \theta_y, k_x, k_y) := \frac{\partial^2}{\partial \theta_x \partial \theta_y} P_{xy}(\theta_x, \theta_y, k_x, k_y)
\end{equation}

This constitutes essentially the generalization of the univariate PDF. The bivariate PDF quantifies the joint probability that $x[k]$ takes the value $\theta_x$ and that $y[k]$ takes the value $\theta_y$ for the entire ensemble of sample functions. Analogous to the univariate case, the bivariate CDF is given by integration

\begin{equation}
P_{xy}(\theta_x, \theta_y, k_x, k_y) = \int\limits_{-\infty}^{\theta_x} \int\limits_{-\infty}^{\theta_y} p_{xy}(\alpha, \beta, k_x, k_y) \, \mathrm{d}\alpha \mathrm{d}\beta
\end{equation}

Due to the properties of the bivariate CDF and its definition the bivariate PDF shows the following properties

\begin{equation}
p_{xy}(\theta_x, \theta_y, k_x, k_y) \geq 0
\end{equation}

and

\begin{equation}
\int_{-\infty}^{\infty} \int_{-\infty}^{\infty} p_{xy}(\theta_x, \theta_y, k_x, k_y) \, \mathrm{d}\theta_x \mathrm{d}\theta_y = 1
\end{equation}

For the special case of one signal only, the bivariate PDF

\begin{equation}
p_{xx}(\theta_1, \theta_2, k_1, k_2) := \frac{\partial^2}{\partial \theta_1 \partial \theta_2} P_{xx}(\theta_1, \theta_2, k_1, k_2)
\end{equation}

describes the probability that the random signal $x[k]$ takes the values $\theta_1$ at time instance $k_1$ and $\theta_2$ at time instance $k_2$. Hence, $p_{xx}(\theta_1, \theta_2, k_1, k_2)$ provides insights into the temporal dependencies of the amplitudes of the random signal $x[k]$.

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples, 2016*.


# Error Propagation using Random Gaussian Numbers

Example calculation of propagating uncertainties, both when adding and multiplying number, and also in the general case. The propagation can be done both analytically (using the error propagation formula) and also using simulation.

The example is based on FIRST doing the error propagation **analytically**, and then verifying it by running a so-called Monte-Carlo (MC) program, which uses random numbers for propagating errors.

## References:
- Barlow: page 48-61
- Bevington: page 36-48

## Author(s), contact(s), and dates:
- Author: Troels C. Petersen (NBI)
- Email:  petersen@nbi.dk
- Date:   25th of November 2021

***

DO THE FOLLOWING ANALYTICAL EXERCISE FIRST!!!

1. A class of students estimate by eye, that the length of the table in Auditorium A is $L = (3.5\pm 0.4)$m, and that the width is $W = (0.8\pm 0.2)$m.

   Assuming that there is no correlation between these two measurements, calculate ANALYTICALLY what the Perimeter (P), area (A), and diagonal (D) length is including (propagated) uncertainties. Repeat the calculation, given that the correlation between length and width is $\rho(L,W) = 0.5$ - not an unreasonable number, given that they are estimated by the same (uncertain) scale.

NOTE: This is a complete standard problem, that you will be asked to solve again and again in the course. For this reason, make sure that you understand how to do it, and become good at doing it reasonably fast.

```python
# Defining the parameters:
mu1   =  3.5
sig1  =  0.4
mu2   =  0.8
sig2  =  0.2
rho12 =  0.5           # Correlation parameter!
```

```python
if not (-1.0 <= rho12 <= 1.0):
    raise ValueError(f"Correlation factor not in interval [-1,1], as it is {rho12:6.2f}")
```

### Note on analytic solutions:

Python includes symbolic algebra in the package *SymPy*, which is both simple and powerful (and in Python). In addition, printing with Latex can also be included (see below), which (in combination) is very nice.

Below is a SymPy and Latex example with the hope that it will wet your appetite.

```python
from IPython.core.display import Latex

def lprint(*args,**kwargs):
    """Pretty print arguments as LaTeX using IPython display system

    Parameters
    ----------
    args : tuple
        What to print (in LaTeX math mode)
    kwargs : dict
        optional keywords to pass to `display`
    """
    display(Latex('$$'+' '.join(args)+'$$'),**kwargs)

from IPython.core.display import Latex

def lprint(*args,**kwargs):
    """Pretty print arguments as LaTeX using IPython display system

    Parameters
    ----------
    args : tuple
        What to print (in LaTeX math mode)
    kwargs : dict
        optional keywords to pass to `display`
    """
    display(Latex('$$'+' '.join(args)+'$$'),**kwargs)

def myDiff(formula):
    return sqrt((formula.diff(L) * dL)**2 + (formula.diff(W) * dW)**2)

def myDiffWithCorr(formula, name = "", printNow = False):
    dd = sqrt((formula.diff(L) * dL)**2 + (formula.diff(W) * dW)**2 + 2*(formula.diff(L)*formula.diff(W)*(sigCorr**2)))
    if(printNow):
        lprint(latex(Eq(symbols('sigma_'+name), dd)))
    fd = lambdify((L,dL,W,dW,sigCorr),dd)
    return dd, fd

def diff_and_print(formula, name = ""):
    # Calculate uncertainty and print original relation/formula and the uncertainty
    dd = myDiff(formula)
    lprint(latex(Eq(symbols(name),formula)))
    lprint(latex(Eq(symbols('sigma_'+name), dd)))

def lambdifyFormula(formula, *args, name = ""):
    # Turn expression into numerical functions
    f = lambdify((L,W),formula)
    d = myDiff(formula)
    fd = lambdify((L,dL,W,dW),d)
    return f, fd
```

```python
# Import SymPy:
from sympy import *

# Define variables:
L,W,P,A,D = symbols("L, W, P, A, D")
dL,dW,dP,dA,dD = symbols("sigma_L, sigma_W, sigma_P, sigma_A, sigma_D")

# Define relations:
# Perimeter:
P = 2*L + 2*W
# Area:
A = L*W
# Diagonal
D = sqrt(L**2 + W**2)

# Try writing a simple function to not repeat yourself! (See cell above)
diff_and_print(P,"P")
diff_and_print(A,"A")
diff_and_print(D,"D")

dP = myDiff(P)
dA = myDiff(A)
dD = myDiff(D)

# Turn expressions into numerical functions
fP, fdP = lambdifyFormula(P,"P")
fA, fdA = lambdifyFormula(A,"A")
fD, fdD = lambdifyFormula(D,"D")

# Define values and their errors
vL, vdL = mu1,sig1
vW, vdW = mu2,sig2

# Numerically evaluate expressions and print
vP = fP(vL,vW)
vdP = fdP(vL,vdL,vW,vdW)
vA = fA(vL,vW)
vdA = fdA(vL,vdL,vW,vdW)
vD = fD(vL,vW)
vdD = fdD(vL,vdL,vW,vdW)

lprint(fr'P = ({vP:.1f} \pm {vdP:.1f})\,\mathrm{{m}}')
lprint(fr'A = ({vA:.1f} \pm {vdA:.1f})\,\mathrm{{m}}')
lprint(fr'D = ({vD:.1f} \pm {vdD:.1f})\,\mathrm{{m}}')

#Adding correlations (and also derivation, printing and lambdifying)
sigCorr = symbols("sigma_LW")
rho = symbols("rho_LW")

dP, fdP = myDiffWithCorr(P, "P", True)
dA, fdA = myDiffWithCorr(A, "A", True)
dD, fdD = myDiffWithCorr(D, "D", True)

sCorr = sqrt(rho*dL*dW)
fSC = lambdify((rho,dL,dW),sCorr)

vSigmaCorr = fSC(rho12,vdL,vdW)

# Numerically evaluate expressions and print
vdP = fdP(vL,vdL,vW,vdW,vSigmaCorr)
vdA = fdA(vL,vdL,vW,vdW,vSigmaCorr)
vdD = fdD(vL,vdL,vW,vdW,vSigmaCorr)

lprint(fr'P = ({vP:.1f} \pm {vdP:.1f})\,\mathrm{{m}}')
lprint(fr'A = ({vA:.1f} \pm {vdA:.1f})\,\mathrm{{m}}')
lprint(fr'D = ({vD:.1f} \pm {vdD:.1f})\,\mathrm{{m}}')
```

$$P = 2 L + 2 W$$

$$\sigma_{P} = \sqrt{4 \sigma_{L}^{2} + 4 \sigma_{W}^{2}}$$

$$A = L W$$

$$\sigma_{A} = \sqrt{L^{2} \sigma_{W}^{2} + W^{2} \sigma_{L}^{2}}$$

$$D = \sqrt{L^{2} + W^{2}}$$

$$\sigma_{D} = \sqrt{\frac{L^{2} \sigma_{L}^{2}}{L^{2} + W^{2}} + \frac{W^{2} \sigma_{W}^{2}}{L^{2} + W^{2}}}$$

$$P = (8.6 \pm 0.9)\,\mathrm{m}$$

$$A = (2.8 \pm 0.8)\,\mathrm{m}$$

$$D = (3.6 \pm 0.4)\,\mathrm{m}$$

$$\sigma_{P} = \sqrt{4 \sigma_{L}^{2} + 8 \sigma_{LW}^{2} + 4 \sigma_{W}^{2}}$$

$$\sigma_{A} = \sqrt{L^{2} \sigma_{W}^{2} + 2 L W \sigma_{LW}^{2} + W^{2} \sigma_{L}^{2}}$$

$$\sigma_{D} = \sqrt{\frac{L^{2} \sigma_{L}^{2}}{L^{2} + W^{2}} + \frac{2 L W \sigma_{LW}^{2}}{L^{2} + W^{2}} + \frac{W^{2} \sigma_{W}^{2}}{L^{2} + W^{2}}}$$

$$P = (8.6 \pm 1.1)\,\mathrm{m}$$

$$A = (2.8 \pm 0.9)\,\mathrm{m}$$

$$D = (3.6 \pm 0.4)\,\mathrm{m}$$

```python
import numpy as np                                     # Matlab like syntax for linear algebra and functions
import matplotlib.pyplot as plt                        # Plots and figures like you know them from Matlab
import seaborn as sns                                  # Make the plots nicer to look at
from iminuit import Minuit                             # The actual fitting tool, better than scipy's
import sys                                             # Modules to see files and folders in directories
```

```python
sys.path.append('../../../External_Functions')
from ExternalFunctions import Chi2Regression
from ExternalFunctions import nice_string_output, add_text_to_ax # useful functions to print fit results on figure
```

## Error propagation - Simulation

Now we want to try to see, if we can solve the above error propagation problem using simulation. The method is relatively straight forward: You simply take "realistic" values of the input parameters x (here Length (x1) and Width (x2)), calculate the resulting value y (here Perimeter, Area, and Diagonal), and do this many times. The resulting distribution of y should be centered around the value y(x1,x2), and the standard deviation should reflect the uncertainty in y from the uncertainties in the input variables.

This is a much more clumsy way of calculating the uncertainty, but comes with the advantage, that if the resulting uncertainty is not Gaussian, then one can actually see this (i.e. it avoids the assumptions used in the usual error propagation formula).

```python
# First we set the parameters of the program:
N_exp = 10000           # Number of "experiments" (i.e. drawing from random distributions)
save_plots = False
r = np.random
r.seed(42)
```

Define parameters for two random numbers (Gaussianly distributed):

```python
mu1   =  3.5
sig1  =  0.4
mu2   =  0.8
sig2  =  0.2
rho12 =  0.5           # Correlation parameter!
```

```python
if not (-1.0 <= rho12 <= 1.0):
    raise ValueError(f"Correlation factor not in interval [-1,1], as it is {rho12:6.2f}")
```

Now we calculate numbers that allows the transform from uncorrelated variables `u` and `v` to correlated random numbers `x1` and `x2` below (see Barlow page 42-44 for method):

```python
# Define the parameters needed for the transformation:
theta = 0.5 * np.arctan( 2.0 * rho12 * sig1 * sig2 / ( np.square(sig1) - np.square(sig2) ) )
sigu = np.sqrt( np.abs( (((sig1*np.cos(theta))**2) - (sig2*np.sin(theta))**2 ) / ( (np.cos(theta))**2 - np.sin(theta)**2) ) )
sigv = np.sqrt( np.abs( (((sig2*np.cos(theta))**2) - (sig1*np.sin(theta))**2 ) / ( (np.cos(theta))**2 - np.sin(theta)**2) ) )

# Produce random numbers with the (possible) correlation:
u = r.normal(0.0, sigu, N_exp)
v = r.normal(0.0, sigv, N_exp)
x1_all = mu1 + np.cos(theta)*u - np.sin(theta)*v
x2_all = mu2 + np.sin(theta)*u + np.cos(theta)*v
x12_all = np.array([x1_all, x2_all])

y_all = x1_all - 2*x2_all         # Silly formula - you have to put this in yourself!
```

Note that the above is nothing more than a matrix multiplication written out! Also note that the absolute value is taken before the square root to avoid `np.sqrt(x)` with `x<0`.

***
Plot both input distribution and resulting 2D-histogram on screen:

```python
fig, ax = plt.subplots(figsize=(14, 8))
counts, xedges, yedges, im = ax.hist2d(x1_all, x2_all, bins=[120, 80], range=[[0.0, 6.0], [-1.0, 3.0]], cmin=1)
ax.plot([0.0, 6.0], [0.0, 0.0], "--k")   # NOTE: This draws a line from [x1, x2], [y1, y2] with dashed line ("--") and in black ("k")
fig.colorbar(im) # ticks=[-1, 0, 1]

ax.set(title='Histogram of lengths (x) and widths (y)',
       xlabel='x',
       ylabel='y',
       aspect='equal', # NOTE: This forces the x- and y-axis to have the SAME scale!!!
      )

d = {'Entries': len(x12_all),
     'Mean x': x1_all.mean(),
     'Mean y': x2_all.mean(),
     'Std x': x1_all.std(ddof=1),
     'Std y': x2_all.std(ddof=1),
    }

text = nice_string_output(d, extra_spacing=2, decimals=3)
add_text_to_ax(0.02, 0.97, text, ax, fontsize=15);

fig.tight_layout()
fig

if save_plots :
    fig.savefig("Dist_2Dgauss.pdf")
```

Now we bin `y_all` and fit it with a Gaussian distribution:

```python
def gaussian(x, N, mu, sigma):
    return N * binwidth / (sigma*np.sqrt(2*np.pi)) * np.exp(-0.5* (x-mu)**2/sigma**2)
```

```python
nbins = 100
xmin, xmax = 0.0, 4.0
binwidth = (xmax-xmin)/nbins

fig2, ax2 = plt.subplots(figsize=(16, 6));
counts, bin_edges, _ = ax2.hist(y_all, nbins, range=(xmin, xmax), histtype='step', linewidth=2)
bin_centers = (bin_edges[1:] + bin_edges[:-1])/2
s_counts = np.sqrt(counts)
```

This is the distribution of "whatever you put into it" (initially x1-2*x2), which shows what output you get and what uncertainty to expect (given by the width - think about this!). We can thus get the result by simply recording the mean and width (SD):

```python
mean = y_all.mean()
std = y_all.std(ddof=1)
print(f"  Mean = {mean:5.3f},    Std = {std:5.3f}")
```

      Mean = 1.895,    Std = 0.402

However, we are in principle not even sure, if this distribution is Gaussian, so in order to check this, we draw a Gaussian on top using the above mean and standard deviation values.

```python
xaxis = np.linspace(xmin, xmax, 1000)
yaxis = gaussian(xaxis, N_exp, mean, std)
ax2.plot(xaxis, yaxis, linewidth=2)
fig2.tight_layout()
fig2
```

```python
if save_plots:
    fig2.savefig("Dist_ErrorProp.pdf")
```

---

# Questions:

0. First solve the problem of obtaining the Perimeter, Area & Diagonal with uncertainty ANALYTICALLY.

**Example answer 0**:

See cell 4. No need to differentiate by hand, SymPy can do it easily for you.

---

1. Now look at the program, and assure yourself that you understand what is going on. Put in the correct expression for y in terms of x1=L and x2=W in order to calculate the perimeter, area, and diagonal length, and run the program. Does the output correspond well with the results you expected from your analytical calculations to begin with?

**Example answer 1**:
Yes, both the mean value and the uncertainty are in a very good agreement to the results from analytical calculations. In the area calculation, there is a slight discrepancy, simply because multiplying two Gaussians does not give (exactly) a Gaussian... only when adding!
Also, one little thing to note is, that the width can actually come out negatively in rare cases (i.e. beyond 4 sigma low). When producing a simulation of cases, one should of course be aware of such things!

---

2. Imagine that you wanted to know the central value and uncertainty of y1 and y2, given the
   same above PDFs for `x1`=$L$ and `x2`=$W$:

     `y1 = log(square(x1*tan(x2))+sqrt((x1-x2)/(cos(x2)+1.0+x1)))`

     `y2 = 1.1+sin(20*x1)`

   Get the central value of y, and see if you can quickly differentiate this with
   respect to `x1` and `x2`, and thus predict what uncertainty to expect for y using
   the error propagation formula. It is (for once) OK to give up on the first expression :-)
   Next, try to estimate the central value and uncertainty using random numbers
   like above - do you trust this result more? And are the distributions Gaussian?

**Example answer 2**:
The first expression "y1" looks horrible, and it is! This is an obvious case where simulation is probably a better way forward, and at least a good cross check.
The second expression "y2" is simple, but made to make the error propagation formula break down! The function varies wildly, and as a result, the error propagation formula assumption is broken (badly). The resulting error distribution is also very far from Gaussian, as can be seen below.

For code solution example, see cell 19 below.

### Advanced questions:

3. Try to generate `x1` and `x2` with non-linear correlation, which yields zero linear correlation,
   and see that despite not having any linear correlation, the result on perimeter, area, and diagonal
   length is still affected.

```python
y1,y2,x1,x2 = symbols('y_1,y_2,x_1,x_2')
dy1,dy2,dx1,dx2 = symbols('sigma_y_1,sigma_y_2,sigma_x_1,sigma_x_2')

y1 = log(sqrt(x1+tan(x2)))+sqrt((x1-x2))/(cos(x2)+1+x1)
lprint(latex(Eq(symbols('y_1'),y1)))

y2 = 1.1 + sin(20*x1)
lprint(latex(Eq(symbols('y_2'),y2)))

dy1 = sqrt((y1.diff(x1) * dx1)**2 + (y1.diff(x2) * dx2)**2)
lprint(latex(Eq(symbols('sigma_y_1'),dy1)))

dy2 = sqrt((y2.diff(x1) * dx1)**2 + (y2.diff(x2) * dx2)**2)
lprint(latex(Eq(symbols('sigma_y_2'),dy2)))

fy1 = lambdify((x1,x2),y1)
fy2 = lambdify((x1,x2),y2)

fdy1 = lambdify((x1,dx1,x2,dx2),dy1)
fdy2 = lambdify((x1,dx1,x2,dx2),dy2)

vx1,vdx1 = mu1,sig1
vx2,vdx2 = mu2,sig2

vy1 = fy1(vx1,vx2)
vdy1 = fdy1(vx1,vdx1,vx2,vdx2)
vy2 = fy2(vx1,vx2)
vdy2 = fdy2(vx1,vdx1,vx2,vdx2)

lprint(fr'y1 = ({vy1:.1f} \pm {vdy1:.1f})\,\mathrm{{m}}')
lprint(fr'y2 = ({vy2:.1f} \pm {vdy2:.1f})\,\mathrm{{m}}')

u = r.normal(0.0, sigu, N_exp)
v = r.normal(0.0, sigv, N_exp)
x1_all = mu1 + np.cos(theta)*u - np.sin(theta)*v
x2_all = mu2 + np.sin(theta)*u + np.cos(theta)*v
x12_all = np.array([x1_all, x2_all])

y_all = 1.1 + np.sin(20*x1_all)

xmin, xmax = 0.0, 3.0

fig3, ax3 = plt.subplots(figsize=(10, 6));
counts, bin_edges, _ = ax3.hist(y_all, 100, range=(xmin, xmax), histtype='step');
bin_centers = (bin_edges[1:] + bin_edges[:-1])/2
s_counts = np.sqrt(counts)

print(f"  Mean = {y_all.mean():5.3f},    RMS = {y_all.std(ddof=1):5.3f}")
```


# Chapter 2 - Math Tools II (Linear Operators)

At the end of the last chapter, we showed that the state of a quantum system can be represented with a complex unit vector. Seeing as how a quantum computer is a machine that manipulates the quantum states of its qubits, we need to characterize how the state vector can evolve. Under the framework of quantum mechanics, _Linear Operators_ are used to describe the time evolution and measurement of quantum systems.

In this chapter, we will describe what linear operators are and show how they relate to matrices. Numpy will also be used for Python implementations.

## 2.1 Linear Operators
Operators are functions that map vectors to vectors.
$$ A : V \rightarrow W,\quad\text{ where V and W are vector spaces.}$$
If the operator $A$ maps the vector $|v\rangle$ to $|w\rangle$, then we write it as
$$ A|v\rangle = |w\rangle.$$

A linear operator is an operator that satisfies the conditions of lineararity, i.e.
1. $$A \left(|v\rangle + |w\rangle\right) = \left(A|v\rangle\right) + \left(A|w\rangle\right)$$
2. $$A \left(c|v\rangle\right) = c \left(A|v\rangle\right)$$

Since every vector $|v\rangle\in V$ can be written as a linear combination of a set of basis vectors,
$$ |v\rangle = \sum_{j} \alpha_j |v_j\rangle,$$
knowing which vectors each basis vector is mapped to under the action of a linear operator allows us to calculate which vector any input vector is mapped to:
$$ A|v\rangle = A \left(\sum_{j} \alpha_j |v_j\rangle\right) = \sum_j \alpha_j A|v_j\rangle. $$

```python
#Importing necessary modules: numpy can be used to describe matrices in two ways: np.array and np.matrix
#Both were used in Chapter 1, but from now on, we will carry on with np.array
#We'll also import a sympy function so that we can visualize matrices in a clearer way. This does not affect any calculation

import numpy as np
from sympy import Matrix, init_printing

#This function uses SymPy to convert a numpy array into a clear matrix image

def view(mat):
    display(Matrix(mat))
```

```python
#Importing necessary modules: numpy can be used to describe matrices in two ways: np.array and np.matrix
#Both were used in Chapter 1, but from now on, we will carry on with np.array
#We'll also import a sympy function so that we can visualize matrices in a clearer way. This does not affect any calculation

import numpy as np
from sympy import Matrix, init_printing

#This function uses SymPy to convert a numpy array into a clear matrix image

def view(mat):
    display(Matrix(mat))#Defining a matrix in numpy, as an array

A = np.array([[1,2,3],[4,5,6],[7,8,9]])

print('Matrix A:')
print(A)

#Multiplying two matrices:

B = np.array([[3,2,1],[6,5,4],[9,8,7]])

print('AB: ')

#Printing it

print(A@B)

print('BA:')

#Viewing it... Looks better

view(B@A)

```

    Matrix A:
    [[1 2 3]
     [4 5 6]
     [7 8 9]]
    AB:
    [[ 42  36  30]
     [ 96  81  66]
     [150 126 102]]
    BA:

$\displaystyle \left[\begin{matrix}18 & 24 & 30\\54 & 69 & 84\\90 & 114 & 138\end{matrix}\right]$

```python
#Let's say we have a vector |u> ε R^3, where |u> = (5,2,1). Let's say we have a vector |v> = 3|u> + (2,2,2)
#In other words, every component of |u>, u_i, is now mapped to 3u_i + 2
import numpy as np

u = np.array([5,2,1])
v = 3*u+2

v

#How do we generally represent this operation, though?
```

    array([17,  8,  5])

## 2.2 Representing Linear Operators with Matrices

A matrix is a two-dimensional array of numbers, and in chapter 1 we saw how to represent vectors as either column or row matrices — matrices with exactly one column or row respectively. A matrix can also be used to numerically represent a linear operator.

Let $V$ be an $n$-dimensional vector space with a basis set $\left\{|v_k\rangle : 1\leq k \leq n \right\}$, $W$ be an $m$-dimensional vector space with a basis set $\left\{|w_j\rangle : 1\leq j \leq m \right\}$ and $A : V \rightarrow W$ be a linear operator such that
$$ A|v_k\rangle = \sum_{j=1}^m a_{jk} |w_j\rangle. $$

Then an $m\times n$ matrix (a matrix with $m$ rows and $n$ columns) can be used to represent $A$:
$$ A \equiv \begin{pmatrix}
a_{11} & a_{12} & \cdots & a_{1n} \\
a_{21} & a_{22} & \cdots & a_{2n} \\
\vdots & \vdots & \ddots & \vdots \\
a_{m1} & a_{m2} & \cdots & a_{mn}
\end{pmatrix} $$

The columns of this matrix are the column vectors $A|v_k\rangle$.

From here onwards, we will assume that our matrix representations use orthonormal basis sets, and for any operator $A: V\rightarrow V$, we use the same basis set for input and output bases.

### 2.2.1 Matrix-Vector Multiplication
A matrix is a numerical representation of an operator, and matrix-vector multiplication is a numerical
method to calculate the action of an operator on a vector.

First, use the same basis of $V$ to represent $|v\rangle$ as a column matrix:
$$|v\rangle = \sum_{k=1}^n \alpha_k |v_k\rangle \equiv \begin{pmatrix} \alpha_1 \\ \alpha_2 \\ \vdots \\ \alpha_n \end{pmatrix}.$$

Now since $A|v\rangle = \sum_{k=1}^n \alpha_k |v_k\rangle$, we can calculate the outcome of applying the operator:

$$
\begin{align}
\begin{pmatrix}
a_{11} & a_{12} & \cdots & a_{1n} \\
a_{21} & a_{22} & \cdots & a_{2n} \\
\vdots & \vdots & \ddots & \vdots \\
a_{m1} & a_{m2} & \cdots & a_{mn}
\end{pmatrix}
\begin{pmatrix} \alpha_1 \\ \alpha_2 \\ \vdots \\ \alpha_n \end{pmatrix}
&=
\alpha_1 \begin{pmatrix} a_{11} \\ a_{21} \\ \vdots \\ a_{m1} \end{pmatrix} +
\alpha_2 \begin{pmatrix} a_{12} \\ a_{22} \\ \vdots \\ a_{m2} \end{pmatrix} +
\cdots +
\alpha_n \begin{pmatrix} a_{1n} \\ a_{2n} \\ \vdots \\ a_{mn} \end{pmatrix} \\
&=
\begin{pmatrix}
\sum_{k=1}^n \alpha_k a_{1k} \\
\sum_{k=1}^n \alpha_k a_{2k} \\
\vdots \\
\sum_{k=1}^n \alpha_k a_{mk}
\end{pmatrix}
\end{align}
$$

### 2.2.2 Composition of Operators and Matrix-Matrix Multiplication
Let $A: V\rightarrow W$ and $B: W \rightarrow X$ be linear operators acting on the vector spaces $V, W$ and $X$. The linear operator $BA : V \rightarrow X$ is defined as the composition of $A$ and $B$:
$$ BA|v\rangle = B(A |v\rangle)$$

Since this composition is also a linear operator, it can also be numerically represented with a matrix. Given basis sets $\{|v_i\rangle\}, \{|w_j\rangle\}$ and $\{|x_k\rangle\}$ for $V,W$ and $X$ respectively, and the matrices $A$ and $B$ written according to those basis sets:

$$
A =
\begin{pmatrix}
| & | &  & | \\
A|v_1\rangle & A|v_2\rangle & \cdots & A|v_n\rangle \\
| & | &  & | \end{pmatrix}
=
\begin{pmatrix}
a_{11} & a_{12} & \cdots & a_{1n} \\
a_{21} & a_{22} & \cdots & a_{2n} \\
\vdots & \vdots & \ddots & \vdots \\
a_{m1} & a_{m2} & \cdots & a_{mn} \end{pmatrix}
$$

$$
B =
\begin{pmatrix}
| & | &  & | \\
B|w_1\rangle & B|w_2\rangle & \cdots & B|w_m\rangle \\
| & | &  & | \end{pmatrix}
=
\begin{pmatrix}
b_{11} & b_{12} & \cdots & b_{1m} \\
b_{21} & b_{22} & \cdots & b_{2m} \\
\vdots & \vdots & \ddots & \vdots \\
b_{o1} & b_{o2} & \cdots & b_{om} \end{pmatrix}
$$
$$
\begin{align}
BA &=
\begin{pmatrix}
| & | &  & | \\
BA|v_1\rangle & BA|v_2\rangle & \cdots & BA|v_n\rangle \\
| & | &  & | \end{pmatrix} \\
&=
\begin{pmatrix}
b_{11} & b_{12} & \cdots & b_{1m} \\
b_{21} & b_{22} & \cdots & b_{2m} \\
\vdots & \vdots & \ddots & \vdots \\
b_{o1} & b_{o2} & \cdots & b_{om} \end{pmatrix}
\begin{pmatrix}
a_{11} & a_{12} & \cdots & a_{1n} \\
a_{21} & a_{22} & \cdots & a_{2n} \\
\vdots & \vdots & \ddots & \vdots \\
a_{m1} & a_{m2} & \cdots & a_{mn} \end{pmatrix} \\
&=
\begin{pmatrix}
c_{11} & c_{12} & \cdots & c_{1n} \\
c_{21} & c_{22} & \cdots & c_{2n} \\
\vdots & \vdots & \ddots & \vdots \\
c_{o1} & c_{o2} & \cdots & c_{on}
\end{pmatrix}, \quad\text{where } c_{ij} = \sum_{k=1}^m b_{ik} a_{kj}
\end{align}
$$

```python
#Defining a matrix in numpy, as an array

A = np.array([[1,2,3],[4,5,6],[7,8,9]])

print('Matrix A:')
print(A)

#Multiplying two matrices:

B = np.array([[3,2,1],[6,5,4],[9,8,7]])

print('AB: ')

#Printing it

print(A@B)

print('BA:')

#Viewing it... Looks better. Doesn't affect any calculation

view(B@A)

```

    Matrix A:
    [[1 2 3]
     [4 5 6]
     [7 8 9]]
    AB:
    [[ 42  36  30]
     [ 96  81  66]
     [150 126 102]]
    BA:

$\displaystyle \left[\begin{matrix}18 & 24 & 30\\54 & 69 & 84\\90 & 114 & 138\end{matrix}\right]$

```python
#Representing a Linear Operator as Matrices

#Application to Quantum Computing

#X|0> = |1>, where, according to section 1.10, |0> = (1,0) and |1> = (0,1)

zero = np.array([1,0])
one = np.array([0,1])

X = np.array([[0,1],[1,0]])
print('X = ')
view(X)

print('')

#Checking to see if X|0>
XZero = X@zero
print('Is X|0> = |1>:')
#Prints true if all elements in both matrices are equal
print(XZero.all() == one.all())

print('')

print('X is a very important operator in quantum computing because it essentially acts as a quantum NOT gate. The implication of this tranformation is taking a qubit which is always 0 and making it always 1 and vice versa (X|1>=|0>).')
```

    X =

$\displaystyle \left[\begin{matrix}0 & 1\\1 & 0\end{matrix}\right]$

    Is X|0> = |1>:
    True

    X is a very important operator in quantum computing because it essentially acts as a quantum NOT gate. The implication of this tranformation is taking a qubit which is always 0 and making it always 1 and vice versa (X|1>=|0>).

## 2.3 Adjoint / Hermitian Conjugate
Let $A : V\rightarrow V$ be a linear operator, then there exists a linear operator $A^\dagger : V \rightarrow V$ defined such that for any vectors $|v\rangle, |w\rangle \in V$ the inner product
$$ (|v\rangle, A|w\rangle) = (A^\dagger|v\rangle, |w\rangle)$$

$A^\dagger$ is called the _adjoint_ or _Hermitian conjugate_ of $A$.

Although it may not seem obvious, given a matrix representation of $A$, we can compute the matrix representation of $A^\dagger$ in the same basis by taking the transpose of $A$ and replacing every element with its complex conjugate:

$$
A \equiv \begin{pmatrix}
a_{11} & a_{12} & \cdots & a_{1n} \\
a_{21} & a_{22} & \cdots & a_{2n} \\
\vdots & \vdots & \ddots & \vdots \\
a_{n1} & a_{n2} & \cdots & a_{nn}
\end{pmatrix}
\implies
A^\dagger \equiv \begin{pmatrix}
a_{11}^* & a_{21}^* & \cdots & a_{n1}^* \\
a_{12}^* & a_{22}^* & \cdots & a_{n2}^* \\
\vdots & \vdots & \ddots & \vdots \\
a_{1n}^* & a_{2n}^* & \cdots & a_{nn}^*
\end{pmatrix}
$$

```python
# Calculating the adjoint of a matrix
A = np.array([ [0+1j, 1+2j], [3+0j, 0-1j] ])
print('A = ')
view(A)

def adjoint(A):
    return A.conj().transpose()

A_dagger = adjoint(A)
print('Adjoint of A = ')
view(A_dagger)
```

    A =

$\displaystyle \left[\begin{matrix}1.0 i & 1.0 + 2.0 i\\3.0 & - 1.0 i\end{matrix}\right]$

    Adjoint of A =

$\displaystyle \left[\begin{matrix}- 1.0 i & 3.0\\1.0 - 2.0 i & 1.0 i\end{matrix}\right]$

## 2.4 Eigenvectors and Eigenvalues
For linear operators that map to the same vector space, $A: V\rightarrow V$, a vector $|v\rangle\neq \mathbf{0}$ that is
mapped to a scaled version of itself
$$ A|v\rangle = \lambda|v\rangle$$
is called an _eigenvector_ of $A$, and the scale factor $\lambda$ is called the _eigenvalue_ of $|v\rangle$.

For example, the matrix
$$ X = \begin{pmatrix} 0 & 1 \\ 1 & 0 \end{pmatrix}$$
has eigenvectors $|+\rangle = \frac{1}{\sqrt2} \binom{1}{1}$ and $|-\rangle = \frac{1}{\sqrt2}\binom{1}{-1}$ with eigenvalues +1 and -1 respectively.

The set of all eigenvectors of an operator, sharing the same eigenvalue $\lambda$, forms a vector space which is a subspace of $V$, we call this the $\lambda$-eigenspace of the operator.

### 2.4.1 Normal Operators
If a set of eigenvectors forms a basis of $V$, we call it an eigenbasis. In the context of quantum computing, we will only consider operators that have at least one orthonormal eigenbasis that spans $V$. We call these operators _Normal operators_. A normal operator $N$ can be written in the form
$$ N = \sum_{k} \lambda_k |v_k\rangle \langle v_k| $$
where $\{|v_k\rangle\}$ forms an orthonormal eigenbasis of $V$, and $\lambda_k$ is the eigenvalue of $|v_k\rangle$. This is called the _spectral decomposition_ of $N$.

The spectral decomposition of $X$ is
$$ X = (+1)|+\rangle \langle +| + (-1)|-\rangle \langle -| $$

### 2.4.2 Trace
The trace of an operator is the sum of all of its eigenvalues. It can be calculated by adding all of the diagonal elements of any matrix representation of the operator.

### 2.4.3 Determinant
The determinant of an operator is the product of all of its eigenvalues. [Click here](https://en.wikipedia.org/wiki/Determinant) to see how to numerically calculate the determinant of a matrix representation of an operator.

Given this definition, you may now understand why solving for $\lambda$ in the equation
$$ \det(A - \lambda I) = 0$$
allows us to calculate the eigenvalues of a matrix.

```python
# Verifying the eigenvalues of X
X = np.array([ [0, 1], [1, 0] ])
print('X = ')
view(X)

eigenvalues = np.linalg.eigvals(X)
print(f'X has {len(eigenvalues)} eigenvalues:', eigenvalues)

# Verifying the spectral decomposition of X

# The eigenvectors
plus  = np.array([ [np.sqrt(0.5)], [np.sqrt(0.5)] ])
minus = np.array([ [np.sqrt(0.5)], [-np.sqrt(0.5)] ])

# |+><+|
plus_plus = plus @ adjoint(plus)
print('|+><+| = ')
view(plus_plus)
# |-><-|
minus_minus = minus @ adjoint(minus)
print('|-><-| = ')
view(minus_minus)

# the spectral decomposition
sd = plus_plus - minus_minus
print('|+><+| - |-><-| = ')
view(sd)
```

    X =

$\displaystyle \left[\begin{matrix}0 & 1\\1 & 0\end{matrix}\right]$

    X has 2 eigenvalues: [ 1. -1.]
    |+><+| =

$\displaystyle \left[\begin{matrix}0.5 & 0.5\\0.5 & 0.5\end{matrix}\right]$

    |-><-| =

$\displaystyle \left[\begin{matrix}0.5 & -0.5\\-0.5 & 0.5\end{matrix}\right]$

    |+><+| - |-><-| =

$\displaystyle \left[\begin{matrix}0.0 & 1.0\\1.0 & 0.0\end{matrix}\right]$

## 2.5 Types of Linear Operators
The following classifications of linear operators will be used all the time in quantum computing:
### 2.5.1 Identity Operator
The identity operator maps all vectors onto themselves.
$$ I |v\rangle = |v\rangle, \forall |v\rangle \in V $$
The matrix representation of the identity operator is a square matrix with 1s in all of the diagonal elements and 0s everywhere else in any choice of basis. So in a 2-dimensional space,
$$ I_2 = \begin{pmatrix} 1 & 0 \\ 0 & 1 \end{pmatrix}. $$

Every vector in $V$ is an eigenvector of the identity, with eigenvalue 1.
### Unitary Operators
A unitary operator $U$ is one whose adjoint is its own inverse:
$$ U U^\dagger = I = U^\dagger U. $$

The eigenvalues of a unitary operator all have length 1 (hence the name unitary).

The columns of a unitary matrix are all mutually orthogonal to eachother, i.e. a unitary operator maps orthogonal bases to orthogonal bases.

Unitary operators preserve inner products, i.e
$$ (|v\rangle, |w\rangle) = (U|v\rangle, U|w\rangle) $$
This can be easily shown using bra-ket notation:
$$ \begin{align}
(U|v\rangle, U|w\rangle) &= (U|v\rangle)^\dagger U|w\rangle \\
&= \langle v | U^\dagger U |w\rangle \\
&= \langle v | I |w\rangle \\
&= \langle v|w \rangle \\
&= (|v\rangle, |w\rangle)
\end{align} $$

Unitary operators are used to describe the time evolution of quantum systems.
### Hermitian Operators
A Hermitian operator $H$ is one that is equal to its adjoint:
$$ H = H^\dagger. $$

The eigenvalues of a Hermitian operator are all real numbers, moreover the matrix representation of a Hermitian operator must have real numbers on the diagonal.

Eigenvectors of a Hermitian operator with different eigenvalues are orthogonal.

Hermitian operators are used to describe measurements of quantum systems.

```python
# Unitary operators
#Defining U and U†

#U = 0.5*np.array([[1+1j, 1-1j],[1-1j, 1+1j]])
U = np.sqrt(0.5)*np.array([ [1, 1], [0+1j, 0-1j] ])
UDagger = adjoint(U)

print('U = ')
view(U)
print('U† = ')
view(UDagger)

#|v> and |psi> = U|v>
v = np.array([1,2])
w = np.array([3,4])

# <v|w>
v_w = adjoint(v)@w
# <v| UDagger U |w>
v_U_w = adjoint(v)@UDagger@U@w

#UU† (Supposed to be I_2)

print('UU† = ')
view(U@UDagger)

print('This +/- 4.2*10^(-17)i is 0, so UU† = I')

print('')

print('Is <v|w> = <v| U† U |w>?')
print('Yes' if np.abs(v_w - v_U_w) < 1e-5 else 'No')

print('\nDo the eigenvalues have unit length?')
eigvals = np.linalg.eigvals(U)
for x in eigvals:
    print('Yes' if np.abs(np.abs(x) - 1.0 < 1e-5) else 'No')
```

    U =

$\displaystyle \left[\begin{matrix}0.707106781186548 & 0.707106781186548\\0.707106781186548 i & - 0.707106781186548 i\end{matrix}\right]$

    U† =

$\displaystyle \left[\begin{matrix}0.707106781186548 & - 0.707106781186548 i\\0.707106781186548 & 0.707106781186548 i\end{matrix}\right]$

    UU† =

$\displaystyle \left[\begin{matrix}1.0 & - 4.26642158858964 \cdot 10^{-17} i\\4.26642158858964 \cdot 10^{-17} i & 1.0\end{matrix}\right]$

    This +/- 4.2*10^(-17)i is 0, so UU† = I

    Is <v|w> = <v| U† U |w>?
    Yes

    Do the eigenvalues have unit length?
    Yes
    Yes

```python
# Hermitian Operator
Y = np.array([ [0, 0+1j], [0-1j, 0] ])
print('Y = ')
view(Y)
print('Y† = ')
view(adjoint(Y))

print('Are the eigenvalues real?')
for x in np.linalg.eigvals(Y):
    print('Yes' if np.abs(x.imag) < 1e-5 else 'No')
```

    Y =

$\displaystyle \left[\begin{matrix}0 & 1.0 i\\- 1.0 i & 0\end{matrix}\right]$

    Y† =

$\displaystyle \left[\begin{matrix}0 & 1.0 i\\- 1.0 i & 0\end{matrix}\right]$

    Are the eigenvalues real?
    Yes
    Yes

## 2.6 Tensor Product

A _tensor product_ is a way to "combine" two vector spaces into a larger vector space.
$$\otimes : V \times W \rightarrow V\otimes W$$

### 2.6.1 Kronecker Product
Within the context of quantum computing, the _Kronecker product_ is be used to numerically evaluate the tensor product of vectors and/or matrices.

For a matrix
$$ A = \begin{pmatrix}
a_{11} & a_{12} & \cdots a_{1n} \\
a_{21} & a_{22} & \cdots a_{2n} \\
\vdots & \vdots & \ddots \vdots \\
a_{m1} & a_{m2} & \cdots a_{mn} \\
\end{pmatrix} $$
and any matrix $B$, the Kronecker product can be evaluated as
$$
A\otimes B = \begin{pmatrix}
a_{11} B & a_{12} B & \cdots a_{1n} B \\
a_{21} B & a_{22} B & \cdots a_{2n} B \\
\vdots & \vdots & \ddots \vdots \\
a_{m1} B & a_{m2} B & \cdots a_{mn} B \\
\end{pmatrix} $$
Where each $a_{jk} B$ is a submatrix that is just $B$ scaled by $a_{jk}$.

The outer product $|v\rangle \langle w|$ is implicitly the tensor product $|v\rangle\otimes\langle w|$,

We often drop the $\otimes$ symbol between two kets or two bras,
$$ |v\rangle\otimes|w\rangle \equiv |v\rangle|w\rangle \equiv |vw\rangle $$

### 2.6.2 Properties of the Tensor Product
* $$ (\alpha |v\rangle) \otimes |w\rangle = \alpha (|v\rangle|w\rangle) = |v\rangle \otimes (\alpha |w\rangle) $$
* $$(|v_1\rangle + |v_2\rangle)\otimes |w\rangle = \alpha|v_1\rangle|w\rangle + |v_2\rangle|w\rangle $$
* $$|v\rangle \otimes( |w_1\rangle + |w_2\rangle) = |v\rangle|w_1\rangle + |v\rangle|w_2\rangle $$

```python
v = np.array([1,2,3]).transpose()
w = np.array([4,5]).transpose()

print('v = ')
view(v)
print('w = ')
view(w)

print('Tensor product |v>⊗|w> = ')
view(np.kron(v,w))

print('Outer product |v><w| = ')
view(np.outer(v,w))

X = np.array([[0,1],[1,0]])
Z = np.array([[1,0],[0,-1]])

print('X =')
view(X)
print('Z = ')
view(Z)

print('X⊗Z = ')
view(np.kron(X,Z))

print('Z⊗X = ')
view(np.kron(Z,X))

#Excercise. Create the equivelent to np.kron() for ket vectors (i.e two column matrices)
```

    v =

$\displaystyle \left[\begin{matrix}1\\2\\3\end{matrix}\right]$

    w =

$\displaystyle \left[\begin{matrix}4\\5\end{matrix}\right]$

    Tensor product |v>⊗|w> =

$\displaystyle \left[\begin{matrix}4\\5\\8\\10\\12\\15\end{matrix}\right]$

    Outer product |v><w| =

$\displaystyle \left[\begin{matrix}4 & 5\\8 & 10\\12 & 15\end{matrix}\right]$

    X =

$\displaystyle \left[\begin{matrix}0 & 1\\1 & 0\end{matrix}\right]$

    Z =

$\displaystyle \left[\begin{matrix}1 & 0\\0 & -1\end{matrix}\right]$

    X⊗Z =

$\displaystyle \left[\begin{matrix}0 & 0 & 1 & 0\\0 & 0 & 0 & -1\\1 & 0 & 0 & 0\\0 & -1 & 0 & 0\end{matrix}\right]$

    Z⊗X =

$\displaystyle \left[\begin{matrix}0 & 1 & 0 & 0\\1 & 0 & 0 & 0\\0 & 0 & 0 & -1\\0 & 0 & -1 & 0\end{matrix}\right]$

Pre-requisites/Chapter 2 - Math Tools II (Linear Operators).ipynb


# Artificial Intelligence in Finance

## Normative Finance

Dr Yves J Hilpisch | The AI Machine

http://aimachine.io | http://twitter.com/dyjh

## Uncertainty and Risk

```python
import numpy as np
```

```python
S0 = 10
B0 = 10
```

```python
S1 = np.array((20, 5))
B1 = np.array((11, 11))
```

```python
M0 = np.array((S0, B0))
M0
```

```python
M1 = np.array((S1, B1)).T
M1
```

```python
K = 14.5
```

```python
C1 = np.maximum(S1 - K, 0)
C1
```

```python
phi = np.linalg.solve(M1, C1)
phi
```

```python
np.allclose(C1, np.dot(M1, phi))
```

```python
C0 = np.dot(M0, phi)
C0
```

## Expected Utility Theory

```python
def u(x):
    return np.sqrt(x)
```

```python
phi_A = np.array((0.75, 0.25))
phi_D = np.array((0.25, 0.75))
```

```python
np.dot(M0, phi_A) == np.dot(M0, phi_D)
```

```python
A1 = np.dot(M1, phi_A)
A1
```

```python
D1 = np.dot(M1, phi_D)
D1
```

```python
P = np.array((0.5, 0.5))
```

```python
def EUT(x):
    return np.dot(P, u(x))
```

```python
EUT(A1)
```

```python
EUT(D1)
```

```python
from scipy.optimize import minimize
```

```python
w = 10
```

```python
cons = {'type': 'eq', 'fun': lambda phi: np.dot(M0, phi) - w}
```

```python
def EUT_(phi):
    x = np.dot(M1, phi)
    return EUT(x)
```

```python
opt = minimize(lambda phi: -EUT_(phi),
               x0=phi_A,
               constraints=cons)
```

```python
opt
```

```python
EUT_(opt['x'])
```

```python
np.dot(M0, opt['x'])
```

## Mean-Variance Portfolio Theory

```python
rS = S1 / S0 - 1
rS
```

```python
rB = B1 / B0 - 1
rB
```

```python
def mu(rX):
    return np.dot(P, rX)
```

```python
mu(rS)
```

```python
mu(rB)
```

```python
rM = M1 / M0 - 1
rM
```

```python
mu(rM)
```

```python
def var(rX):
    return ((rX - mu(rX)) ** 2).mean()
```

```python
var(rS)
```

```python
var(rB)
```

```python
def sigma(rX):
    return np.sqrt(var(rX))
```

```python
sigma(rS)
```

```python
sigma(rB)
```

```python
np.cov(rM.T, aweights=P, ddof=0)
```

```python
phi = np.array((0.5, 0.5))
```

```python
def mu_phi(phi):
    return np.dot(phi, mu(rM))
```

```python
mu_phi(phi)
```

```python
def var_phi(phi):
    cv = np.cov(rM.T, aweights=P, ddof=0)
    return np.dot(phi, np.dot(cv, phi))
```

```python
var_phi(phi)
```

```python
def sigma_phi(phi):
    return var_phi(phi) ** 0.5
```

```python
sigma_phi(phi)
```

```python
from pylab import plt, mpl
plt.style.use('seaborn')
mpl.rcParams['savefig.dpi'] = 300
mpl.rcParams['font.family'] = 'serif'
```

```python
phi_mcs = np.random.random((2, 200))
```

```python
phi_mcs = (phi_mcs / phi_mcs.sum(axis=0)).T
```

```python
mcs = np.array([(sigma_phi(phi), mu_phi(phi))
                for phi in phi_mcs])
```

```python
plt.figure(figsize=(10, 6))
plt.plot(mcs[:, 0], mcs[:, 1], 'ro')
plt.xlabel('expected volatility')
plt.ylabel('expected return');
```

```python
P = np.ones(3) / 3
P
```

```python
S1 = np.array((20, 10, 5))
```

```python
T0 = 10
T1 = np.array((1, 12, 13))
```

```python
M0 = np.array((S0, T0))
M0
```

```python
M1 = np.array((S1, T1)).T
M1
```

```python
rM = M1 / M0 - 1
rM
```

```python
mcs = np.array([(sigma_phi(phi), mu_phi(phi))
                for phi in phi_mcs])
```

```python
plt.figure(figsize=(10, 6))
plt.plot(mcs[:, 0], mcs[:, 1], 'ro')
plt.xlabel('expected volatility')
plt.ylabel('expected return');
```

```python
cons = {'type': 'eq', 'fun': lambda phi: np.sum(phi) - 1}
```

```python
bnds = ((0, 1), (0, 1))
```

```python
min_var = minimize(sigma_phi, (0.5, 0.5),
                   constraints=cons, bounds=bnds)
```

```python
min_var
```

```python
def sharpe(phi):
    return mu_phi(phi) / sigma_phi(phi)
```

```python
max_sharpe = minimize(lambda phi: -sharpe(phi), (0.5, 0.5),
               constraints=cons, bounds=bnds)
```

```python
max_sharpe
```

```python
plt.figure(figsize=(10, 6))
plt.plot(mcs[:, 0], mcs[:, 1], 'ro', ms=5)
plt.plot(sigma_phi(min_var['x']), mu_phi(min_var['x']),
         '^', ms=12.5, label='minimum volatility')
plt.plot(sigma_phi(max_sharpe['x']), mu_phi(max_sharpe['x']),
         'v', ms=12.5, label='maximum Sharpe ratio')
plt.xlabel('expected volatility')
plt.ylabel('expected return')
plt.legend();
```

```python
cons = [{'type': 'eq', 'fun': lambda phi: np.sum(phi) - 1},
       {'type': 'eq', 'fun': lambda phi: mu_phi(phi) - target}]
```

```python
bnds = ((0, 1), (0, 1))
```

```python
targets = np.linspace(mu_phi(min_var['x']), 0.16)
```

```python
frontier = []
for target in targets:
    phi_eff = minimize(sigma_phi, (0.5, 0.5),
                       constraints=cons, bounds=bnds)['x']
    frontier.append((sigma_phi(phi_eff), mu_phi(phi_eff)))
frontier = np.array(frontier)
```

```python
plt.figure(figsize=(10, 6))
plt.plot(frontier[:, 0], frontier[:, 1], 'mo', ms=5,
         label='efficient frontier')
plt.plot(sigma_phi(min_var['x']), mu_phi(min_var['x']),
         '^', ms=12.5, label='minimum volatility')
plt.plot(sigma_phi(max_sharpe['x']), mu_phi(max_sharpe['x']),
         'v', ms=12.5, label='maximum Sharpe ratio')
plt.xlabel('expected volatility')
plt.ylabel('expected return')
plt.legend();
```

## Capital Asset Pricing Model

```python
plt.figure(figsize=(10, 6))
plt.plot((0, 0.3), (0.01, 0.22), label='capital market line')
plt.plot(0, 0.01, 'o', ms=9, label='risk-less asset')
plt.plot(0.2, 0.15, '^', ms=9, label='market portfolio')
plt.annotate('$(0, \\bar{r})$', (0, 0.01), (-0.01, 0.02))
plt.annotate('$(\sigma_M, \mu_M)$', (0.2, 0.15), (0.19, 0.16))
plt.xlabel('expected volatility')
plt.ylabel('expected return')
plt.legend();
```

```python
phi_M = np.array((0.8, 0.2))
```

```python
mu_M = mu_phi(phi_M)
mu_M
```

```python
sigma_M = sigma_phi(phi_M)
sigma_M
```

```python
r = 0.0025
```

```python
plt.figure(figsize=(10, 6))
plt.plot(frontier[:, 0], frontier[:, 1], 'm.', ms=5,
         label='efficient frontier')
plt.plot(0, r, 'o', ms=9, label='risk-less asset')
plt.plot(sigma_M, mu_M, '^', ms=9, label='market portfolio')
plt.plot((0, 0.6), (r, r + ((mu_M - r) / sigma_M) * 0.6),
         'r', label='capital market line', lw=2.0)
plt.annotate('$(0, \\bar{r})$', (0, r), (-0.015, r + 0.01))
plt.annotate('$(\sigma_M, \mu_M)$', (sigma_M, mu_M),
             (sigma_M - 0.025, mu_M + 0.01))
plt.xlabel('expected volatility')
plt.ylabel('expected return')
plt.legend();
```

```python
def U(p):
    mu, sigma = p
    return mu - 1 / 2 * (sigma ** 2 + mu ** 2)
```

```python
cons = {'type': 'eq',
        'fun': lambda p: p[0] - (r + (mu_M - r) / sigma_M * p[1])}
```

```python
opt = minimize(lambda p: -U(p), (0.1, 0.3), constraints=cons)
```

```python
opt
```

```python
from sympy import *
init_printing(use_unicode=False, use_latex=False)
```

```python
mu, sigma, b, v = symbols('mu sigma b v')
```

```python
sol = solve('mu - b / 2 * (sigma ** 2 + mu ** 2) - v', mu)
```

```python
sol
```

```python
u1 = sol[0].subs({'b': 1, 'v': 0.1})
u1
```

```python
u2 = sol[0].subs({'b': 1, 'v': 0.125})
u2
```

```python
f1 = lambdify(sigma, u1)
f2 = lambdify(sigma, u2)
```

```python
sigma_ = np.linspace(0.0, 0.5)
u1_ = f1(sigma_)
u2_ = f2(sigma_)
```

```python
plt.figure(figsize=(10, 6))
plt.plot(sigma_, u1_, label='$v=0.1$')
plt.plot(sigma_, u2_, '--', label='$v=0.125$')
plt.xlabel('expected volatility')
plt.ylabel('expected return')
plt.legend();
```

```python
u = sol[0].subs({'b': 1, 'v': -opt['fun']})
u
```

```python
f = lambdify(sigma, u)
```

```python
u_ = f(sigma_)
```

```python
plt.figure(figsize=(10, 6))
plt.plot(0, r, 'o', ms=9, label='risk-less asset')
plt.plot(sigma_M, mu_M, '^', ms=9, label='market portfolio')
plt.plot(opt['x'][1], opt['x'][0], 'v', ms=9, label='optimal portfolio')
plt.plot((0, 0.5), (r, r + (mu_M - r) / sigma_M * 0.5),
         label='capital market line', lw=2.0)
plt.plot(sigma_, u_, '--', label='$v={}$'.format(-round(opt['fun'], 3)))
plt.xlabel('expected volatility')
plt.ylabel('expected return')
plt.legend();
```

## Arbitrage Pricing Theory

```python
M1
```

```python
M0
```

```python
V1 = np.array((12, 15, 7))
```

```python
reg = np.linalg.lstsq(M1, V1, rcond=-1)[0]
reg
```

```python
np.dot(M1, reg)
```

```python
np.dot(M1, reg) - V1
```

```python
V0 = np.dot(M0, reg)
V0
```

```python
U0 = 10
U1 = np.array((12, 5, 11))
```

```python
M0_ = np.array((S0, T0, U0))
```

```python
M1_ = np.concatenate((M1.T, np.array([U1,]))).T
```

```python
M1_
```

```python
np.linalg.matrix_rank(M1_)
```

```python
reg = np.linalg.lstsq(M1_, V1, rcond=-1)[0]
reg
```

```python
np.allclose(np.dot(M1_, reg), V1)
```

```python
V0_ = np.dot(M0_, reg)
V0_
```

http://tpq.io | @dyjh | ai@tpq.io


```R
library(foreach)
```

Create data from the following experiment:

- The treatment has a higher effect on people older than 31.
- Optional:People are more likely to be available for treatment when they are under 31.

```R
make_experiment <- function(n=100, treatment_prob){
    response <- integer()
    group <- integer()
    tau <- numeric()

    X <- runif(n = n, 21,40)

    if(length(treatment_prob)==1){
        if(treatment_prob=="age"){
              treatment_prob = ifelse(X>31, 0.3, 0.7)
        }else{
              treatment_prob <- rep(treatment_prob,n)
    }}

    for(i in 1:n){
        group[i] <- rbinom(n = 1, size = 1, prob = treatment_prob[i])
        tau[i] <- ifelse(X[i] > 31, rnorm(1, mean = 0.3, sd = 0.01), rnorm(1, mean = 0.1, sd = 0.01))
        if(group[i] == 1){
            response[i] <- rbinom(n = 1, size = 1, prob = 0.3 + tau[i])
        }else{
            response[i] <- rbinom(n = 1, size = 1, prob = 0.3)
        }
        }

    data <- data.frame(response, group, X, treatment_prob, tau)

    return(data)
}
```

```R
data <- make_experiment(2000, 0.5)

sum(data$group==1)
sum(data$group==0)

mean(data$response[data$group==1])
mean(data$response[data$group==0])
```

980

1020

0.506122448979592

0.314705882352941

```R
summary(data)
```

        response          group            X         treatment_prob
     Min.   :0.0000   Min.   :0.00   Min.   :21.01   Min.   :0.5
     1st Qu.:0.0000   1st Qu.:0.00   1st Qu.:25.63   1st Qu.:0.5
     Median :0.0000   Median :0.00   Median :30.40   Median :0.5
     Mean   :0.4085   Mean   :0.49   Mean   :30.48   Mean   :0.5
     3rd Qu.:1.0000   3rd Qu.:1.00   3rd Qu.:35.39   3rd Qu.:0.5
     Max.   :1.0000   Max.   :1.00   Max.   :40.00   Max.   :0.5
          tau
     Min.   :0.07255
     1st Qu.:0.09907
     Median :0.11618
     Mean   :0.19391
     3rd Qu.:0.29973
     Max.   :0.32915

Intuition: We look at how often we get a response in the group with treatment and compare this number to how often we get a response without treatment.

```R
sum(data$response[data$group==1])
sum(data$response[data$group==0])
sum(data$response*data$group) - sum(data$response*(1-data$group))

```

496

321

175

Or as a difference of ratios

```R
sum(data$response[data$group==1])/1000
sum(data$response[data$group==0])/1000
sum(data$response*data$group)/1000 - sum(data$response*(1-data$group))/1000

```

0.496

0.321

0.175

## Average treatment effect under equiprobable sampling

The treatment effect is defined as
$$ ATE = \mathbf{E}[y(Treatment) - y(Control)] \\
       = \mathbf{E}[y(Treatment)] - \mathbf{E}[y(Control)] $$

The fundamental problem of causal inference is that we cannot observe the same observation $i$ both with and without treatment. But we can collect data to estimate the the expected response with and without treatment.

$$ \hat{ATE} = \hat{\mathbf{E}}[y(Treatment)] - \hat{\mathbf{E}}[y(Control)] $$

If individuals are exchangeable, i.e. there are no structural differenes between the invidiuals in the treatment and controls groups, then a good estimator for the expectation of a random variable is its mean

$$ \hat{ATE} = \frac{1}{N_{Treatment}} \sum_{i\in {Treatment}}\{ {y_i 1_{Group=Treatment}} \}
       - \frac{1}{N_{Control}}   \sum_{i\in {Control}}\{ {y_i 1_{Group=Control}}   \}$$

```R
simple_ATE <- function(response, group){
    mean(response[group==1]) - mean(response[group==0])
}
```

```R
simple_ATE(data$response, data$group)
```

0.191416566626651

## Inverse propensity weighting

Mostly harmless econometrics (p.60, equation 3.3.11)

\begin{align}
 \mathbf{E} &= \mathbf{E}\left[\frac{Y_i D_i}{p(X_i)} - \frac{Y_i(1-D_i}{1-p(X_i)} \right] \\
  &=\mathbf{E}\left[ \frac{(D_i - p(X_i))Y_i}{p(X_i)(1-p(X_i)}  \right]
\end{align}

```R
propensity_corrected_ATE <- function(response, group, propensity_score){
    mean(
        ((group - propensity_score) * response) /
        (propensity_score * (1-propensity_score))
    )
}
```

Assume our experiment assigns treatment randomly, but with a low probability.

```R
treatment_ratio <- rep(0.05, 1000)
```

```R
exp <- foreach(i=1:200, .combine = rbind) %do%{

data <- make_experiment(1000, treatment_ratio)

cbind("simple"=simple_ATE(data$response, data$group),
      "corrected"=propensity_corrected_ATE(data$response, data$group,
                                           propensity_score = treatment_ratio)
      )
}
```

```R
boxplot(exp)
abline(h=0.2, col="red")
```

## Inverse propensity weighting with individual treatment probability

Now let's assume that people over 31 are less likely to receive treatment.

```R
data <- make_experiment(1000, treatment_prob="age")
summary(data[data$X>31,])
summary(data[data$X<31,])
```

        response          group              X         treatment_prob
     Min.   :0.0000   Min.   :0.0000   Min.   :31.13   Min.   :0.3
     1st Qu.:0.0000   1st Qu.:0.0000   1st Qu.:33.18   1st Qu.:0.3
     Median :0.0000   Median :0.0000   Median :35.44   Median :0.3
     Mean   :0.3718   Mean   :0.2821   Mean   :35.52   Mean   :0.3
     3rd Qu.:1.0000   3rd Qu.:1.0000   3rd Qu.:37.95   3rd Qu.:0.3
     Max.   :1.0000   Max.   :1.0000   Max.   :39.99   Max.   :0.3
          tau
     Min.   :0.2629
     1st Qu.:0.2932
     Median :0.2990
     Mean   :0.2991
     3rd Qu.:0.3053
     Max.   :0.3307

        response          group              X         treatment_prob
     Min.   :0.0000   Min.   :0.0000   Min.   :21.02   Min.   :0.7
     1st Qu.:0.0000   1st Qu.:0.0000   1st Qu.:23.69   1st Qu.:0.7
     Median :0.0000   Median :1.0000   Median :26.15   Median :0.7
     Mean   :0.3402   Mean   :0.6992   Mean   :26.14   Mean   :0.7
     3rd Qu.:1.0000   3rd Qu.:1.0000   3rd Qu.:28.61   3rd Qu.:0.7
     Max.   :1.0000   Max.   :1.0000   Max.   :30.99   Max.   :0.7
          tau
     Min.   :0.07219
     1st Qu.:0.09340
     Median :0.10018
     Mean   :0.10008
     3rd Qu.:0.10655
     Max.   :0.12682

```R
exp <- foreach(i=1:200, .combine = rbind) %do%{

     data <- make_experiment(2000, treatment_prob="age")

     out <- cbind("simple"=simple_ATE(data$response, data$group),
                  "corrected"=propensity_corrected_ATE(data$response, data$group,
                                           propensity_score = data$treatment_prob)
      )
}
```

Not correcting for the indvidual (as in "age group specific") chance to receive treatment AKA propensity score will return a biased treatment effect estimate. The treated group contains more young people, on which the treatment has a lower effect.

```R
boxplot(exp)
abline(h=0.2, col="red")
```

```R

```


```python
import numpy as np
from sklearn.manifold import SpectralEmbedding
from sklearn.datasets import make_swiss_roll, make_s_curve

import matplotlib.pyplot as plt
import mpl_toolkits.mplot3d.axes3d as p3
plt.style.use('ggplot')

%matplotlib inline
```

This notebook will walk through the steps of Laplacian Eigenmaps (LE) algorithm. It will be a step-by-step walkthrough of the algorithm and towards the notebook, I will work on some common known speed-up attempts.

```python
seed = 123
rng = np.random.seed(123)
n_samples = 1500
noise = 0.1
random_state = seed

data, color = make_swiss_roll(n_samples=n_samples, noise=noise, random_state=random_state)
data, color = make_s_curve(n_samples=n_samples, noise=noise, random_state=random_state)
```

\begin{align}
    g &= \int_a^b f(x)dx \label{eq1} \\
    a &= b + c \label{eq2}
\end{align}

See (\ref{eq1})

```python
fig = plt.figure()

ax = p3.Axes3D(fig)

ax.scatter(data[:,0], data[:, 1], data[:,2], c=color, cmap=plt.cm.Spectral)
ax.set_title("Original Data")
plt.show()
```

### Laplacian Eigenmaps (Sklearn)

```python
%%time

n_components = 2
affinity = 'nearest_neighbors'
n_neighbors = 10
n_jobs = -1
random_state = 123
eigen_solver = 'arpack'

# initialize le model
le_model = SpectralEmbedding(
    n_components=n_components,
    affinity=affinity,
    n_neighbors=n_neighbors,
    n_jobs=n_jobs,
    random_state=random_state,
    eigen_solver=eigen_solver
)

# fit and transform data
embedding = le_model.fit_transform(data);
```

    CPU times: user 175 ms, sys: 15.9 ms, total: 191 ms
    Wall time: 214 ms

```python
fig, ax = plt.subplots(figsize=(10, 6))

ax.spy(le_model.affinity_matrix_, markersize=1.0)
plt.show()
```

```python
print(embedding.shape)
```

    (1500, 2)

```python
fig, ax = plt.subplots()

ax.scatter(embedding[:, 0], embedding[:, 1], c=color)
ax.set_title('Projected Data')
plt.show()
```

## Adjacency Matrix Construction

### Nearest Neighbours Search

```python
# some baseline parameters
n_neighbors = 10
algorithm = 'brute'
metric = 'euclidean'
p=2
n_jobs = -1

# initialize nn model
nn_model = NearestNeighbors(
    n_neighbors=n_neighbors,
    metric=metric,
    algorithm=algorithm,
    p=p,
    n_jobs=n_jobs
)

# fit nn model to data
nn_model.fit(data);

# grab distances and indices
dists, indices = nn_model.kneighbors(
    data,
    n_neighbors=n_neighbors,
    return_distance=True
)
```

### Weighted Distances

```python
# Heat kernel
def heat_kernel(distances, length_scale=None):
    if length_scale is None:
        length_scale = 1.0
#         length_scale = np.sqrt(distances.shape[1] / 2.0)
#         length_scale = 1.0 / distances.shape[1]
#     return np.exp(- length_scale * distances**2)
    return np.exp(- distances**2 / length_scale)
```

```python
# transform distances with heat kernel
dists = heat_kernel(dists)
```

### Construct Graph

```python
# Construct sparse KNN Graph
n_samples = data.shape[0]

indptr = np.arange(0, n_samples * n_neighbors + 1, n_neighbors)

adjacency_matrix = csr_matrix((dists.ravel(), indices.ravel(), indptr), shape=(n_samples, n_samples))

# ensure that its symmetrix
adjacency_matrix = 0.5 * (adjacency_matrix + adjacency_matrix.T)
```

#### Peak at Adjacency Matrix

```python

fig, ax = plt.subplots()

ax.spy(adjacency_matrix, markersize=1.0)
ax.set_title('Adjacency Matrix', pad=15.0)

plt.show()
```

## Laplacian Matrix

Some notes about some different Laplacian matrices:

**Unnormalized Graph Laplacian**

$$L=D-W$$

**Symmetric Normalized Graph Laplacian**

$$L_{Sym}=D^{-\frac{1}{2}}LD^{-\frac{1}{2}}$$

**Random Walk Normalized Laplacian**

$$L_{rw}=D^{-1}L$$

**Random Walk Transition Matrix**

$$L_{rwt}=D^{-1}W$$

### Laplacian and Degree Matrix

```python
def graph_laplacian(adjacency_matrix, graph_type='normalized', return_diag=True):

    n_samples = adjacency_matrix.shape[0]

    # Get degree vector
    degree = np.array(adjacency_matrix.sum(axis=1)).squeeze()

    # Create Sparse matrix for degree
    degree_mat = spdiags(
        degree, diags=0, m=n_samples, n=n_samples
    )

    if graph_type in ['unnormalized']:
        laplacian = degree_mat - adjacency_matrix
    elif graph_type in ['normalized']:
        laplacian = degree_mat - adjacency_matrix
        norm = 1 / np.sqrt(degree_mat)
        laplacian = norm @ adjacency_matrix @ norm

    # Set all values

    # set diagonal elements of array to negative

    return laplacian, degree_mat
```

```python
%%time
laplacian, degree = graph_laplacian(adjacency_matrix)
```

```python
# print(degree)
```

```python

```

```python
print(degree.shape)
print(laplacian.diagonal().shape)
```

    (1500,)
    (1500,)

```python
laplacian, degree = graph_laplacian(adjacency_matrix, normed=True, return_diag=True)
```

```python
print(laplacian.shape, degree.shape)
```

    (1500, 1500) (1500,)

```python
fig, ax = plt.subplots(nrows=1, ncols=2)

ax[0].spy(laplacian, markersize=1.0)
ax[0].set_title('Laplacian Matrix', pad=15.0)

ax[1].spy(degree, markersize=1.0)
ax[1].set_title('Degree Matrix', pad=15.0)

plt.show()
```

## EigenValue Decomposition

|        Algorithm       | Laplacian Equation      | Generalized Eigenvalue | Standard Eigenvalue |
|:----------------------:|-------------------------|:----------------------:|---------------------|
| Unnormalized Laplacian |        $$L=D-W$$        |          $$$$          |                     |
| Normalized Laplacian   | $$L=D^{-1/2}LD^{-1/2}$$ |          2.75          |                     |
| Random Walk            |                         |          2.03          |                     |
| ReNormalized           |                         |          0.64          |                     |
| Geometric              |                         |                        |                     |

### Generalized Eigenvalue Solver

```python
%%time

# Transform equation
laplacian_sol = -1 * laplacian

n_components = 2
solver = 'LM'      # Smallest to Largest
sigma = 1.0
eigen_tol = 0.0
v0 = np.random.uniform(-1, 1, laplacian.shape[0])

eigenvalues, eigenvectors = eigsh(
    laplacian_sol,
    k=n_components,
    which=solver,
    sigma=sigma,
    tol=eigen_tol,
    v0=v0
)

# Transform eigenvectors
embedding = eigenvectors.T[n_components::-1]

```

    CPU times: user 141 ms, sys: 3.27 ms, total: 144 ms
    Wall time: 83.6 ms

```python
print(eigenvalues.shape, eigenvectors.shape)
```

    (2,) (1500, 2)

```python
fig, ax = plt.subplots()

ax.scatter(embedding[0, :], embedding[1,:], c=color)
ax.set_title('Projected Data')
plt.show()
```

```python

```