# Model-Based Multi-Compartment Constrained Spherical Deconvolution

Constrained Spherical Deconvolution (CSD) *(Tournier et al. 2007)* is one of the most proven ways to estimate Fiber Orientation Distributions (FODs) that describe the orientation of the white matter tissue. In short, CSD's *single-shell* formulation states that any composition of oriented white matter (dispersion/crossings) can be described as the spherical convolution of a *positive* probability density on the sphere and a convolution kernel $K$ that describes one parallel axon bundle. With some abuse of notation:

$$
\begin{align}
\begin{aligned}
 E_{\textrm{CSD}}= \overbrace{\operatorname{FOD}(\textbf{c})}^{\textrm{Fiber Distribution}}\,*_{\mathbb{S}^2}\,\overbrace{K(\cdot)}^{\textrm{Convolution Kernel}}\quad\textrm{subject to}\quad \operatorname{FOD}(\textbf{c}) > 0.
\end{aligned}
\end{align}$$

Here, the FOD is described in terms of a truncated even spherical harmonics series $FOD=\sum_{l=0}^{lmax}\sum_{m=-l}^l\textbf{c}_{lm}Y_{lm}$, with $l$ and $m$ describing the order and moment of the spherical harmonic *(Descoteaux et al. 2006)*. Furthermore, $*_{\mathbb{S}^2}$ describes a spherical convolution on the $\mathbb{S}^2$ sphere, and $K$ describes the *rotational* harmonics describing a single axon bundle. The kernel $K$ is typically estimated from the data *(Tournier et al. 2007, Tax et al. 2014)*, and CSD is therefore considered a *model-free* approach.

The multi-shell implementation of CSD is straight-forward in that the spherical convolution is just performed per acquisition shell.

Estimating multi-shell CSD with multiple convolution kernels has also successfully been explored in Multi-Tissue CSD (MT-CSD) *(Jeurissen et al. 2014*).

As with `MultiCompartmentModel` and `MultiCompartmentSphericalMeanModel`, Dmipy allows the user to create a Multi-Compartment CSD (MC-CSD) model using `MultiCompartmentSphericalHarmonicsModel`. If all parameters of the convolution kernel models are fixed, it is possible to simultaneously fit the the FOD and volume fractions of **one** anisotropic model (e.g. a Stick, Zeppelin or possibly distributed Cylinder) and $N$ isotropic compartments (Gaussian Ball or Spheres).

$$
\begin{align}
\begin{aligned}
 E_{\textrm{MC-CSD}}= vf_1 \times \overbrace{\operatorname{FOD}(\textbf{c})}^{\textrm{Fiber Distribution}}\,*_{\mathbb{S}^2}\,\overbrace{K_{\boldsymbol{\mu}}(\cdot)}^{\textrm{Anisotropic Kernel}}+\sum_{i=2}^{N+1}vf_i\times\overbrace{K_{iso}(\cdot)}^{\textrm{Isotropic Kernel}}\quad\textrm{subject to}\quad \operatorname{FOD}(\textbf{c}) > 0, \sum_{i=1}^Nvf_i=1.
\end{aligned}
\end{align}$$

NOTE: Dmipy's MC-CSD is NOT the same as Jeurissen et al.'s MT-CSD because at this point we are not including T2 differences between the compartment models. We will explore this in a later example.

## Multi-Shell Single-Compartment CSD

As our first example we will reproduce the standard CSD formulation, using only a single Zeppelin kernel describing a white matter bundle. Notice that in this model representation we must define the maximum spherical harmonics order at which to truncate the FOD.

```python
from dmipy.core.modeling_framework import MultiCompartmentSphericalHarmonicsModel
from dmipy.signal_models import gaussian_models
zeppelin = gaussian_models.G2Zeppelin()
csd_mod = MultiCompartmentSphericalHarmonicsModel([zeppelin], sh_order=8)
```

### Response Function Estimation

Before fitting the FOD, we must first fix the parameters of the Zeppelin convolution kernel. First, we load our example slice of HCP data.

```python
from dmipy.data import saved_data
scheme_hcp, data_hcp = saved_data.wu_minn_hcp_coronal_slice()
```

    This data slice originates from Subject 100307 of the Human Connectome Project, WU-Minn Consortium (Principal Investigators: David Van Essen and Kamil Ugurbil; 1U54MH091657) funded by the 16 NIH Institutes and Centers that support the NIH Blueprint for Neuroscience Research; and by the McDonnell Center for Systems Neuroscience at Washington University.

```python
import matplotlib.pyplot as plt
import matplotlib.patches as patches
%matplotlib inline

fig, ax = plt.subplots(1)
ax.imshow(data_hcp[:, 0, :, 0].T, origin=True)
rect = patches.Rectangle((70,70),20,20,linewidth=1,edgecolor='r',facecolor='none')
ax.add_patch(rect)
ax.set_axis_off()
ax.set_title('HCP coronal slice B0 with ROI');
```

We gratefully make use of Dipy's automatic response function estimation. We convert Dmipy's acquisition scheme to a Dipy gradient table using `gtab_dmipy2dipy`.

```python
from dmipy.core.acquisition_scheme import gtab_dmipy2dipy
from dipy.reconst.csdeconv import auto_response
gtab = gtab_dmipy2dipy(scheme_hcp)
response, ratio = auto_response(gtab, data_hcp, roi_radius=10, fa_thr=0.7)
lambdas = response[0]
lambdas
```

    array([0.00130588, 0.00023963, 0.00023963])

Notice that Dipy's response provides the Zeppelin's eigenvalues in mm$^2$/s, so we must first put them in SI units.

```python
lambda_par = lambdas[0] * 1e-6
lambda_perp = lambdas[1] * 1e-6
```

We then fix the parameters of the kernel.

```python
csd_mod.set_fixed_parameter('G2Zeppelin_1_lambda_par', lambda_par)
csd_mod.set_fixed_parameter('G2Zeppelin_1_lambda_perp', lambda_perp)
```

### Fitting CSD to HCP data and visualizing features

We are now ready to fit the CSD model as per usual. Using the default `solver='csd'` option will let the algorithm decide which optimizer to use - if volume fractions need to be estimated it will use `csd_cvxpy`, and otherwise the basic CSD `csd_tournier07` optimizer proposed by *(Tournier et al. 2007)*.

Using the `csd` option will also automatically select whether to enforce that the FOD integrates to one over the sphere (i.e. it is a proper distribution). If a non-voxel-varying convolution kernel is given, it will set `unity_constraint=False`, otherwise it will be set to `True`.

```python
csd_fit = csd_mod.fit(
    acquisition_scheme=scheme_hcp,
    data=data_hcp,
    mask=data_hcp[..., 0]>0,
    solver='csd')
```

    Parallel processing turned off for tournier07 optimizer because it does not improve fitting speed.
    Setup Tournier07 FOD optimizer in 1.45701694489 seconds
    Fitting of 8181 voxels complete in 16.890802145 seconds.
    Average of 0.00206463783706 seconds per voxel.

The `csd_fit` instance now constains the spherical harmonics of the FOD for every fitted voxel. For the used spherical harmonics order of $8$ that means we have estimated a total of 45 coefficients:

```python
csd_fit.fitted_parameters_vector.shape
```

    (145, 1, 145, 45)

We have can estimate 2 metric that quantify FOD anisotropy:
- The anisotropy index (AI) following *(Jespersen et al. 2007)*, ranging from 1 for completely anistropic, to zero being isotropic.
- The norm of the laplacian of the spherical harmonics of the FOD, explored in e.g. *(Descoteaux et al. 2006)*, which is zero for isotropic FODs and increasingly positive for more anistropic FODs.

It is relevant to not that AI is dependent on the value of the spherical mean of the FOD, but the Laplacian-norm is not.

```python
ai = csd_fit.anisotropy_index()
lb = csd_fit.norm_of_laplacian_fod()

fig, axs = plt.subplots(1, 2, figsize=[10, 5])
axs = axs.ravel()

for counter, (name, values) in enumerate(zip(['Anisotropy Index', 'Norm of FOD Laplacian'], [ai, lb])):
    cf = axs[counter].imshow(values.squeeze().T, origin=True, interpolation='nearest')
    axs[counter].set_title(name)
    axs[counter].set_axis_off()
    fig.colorbar(cf, ax=axs[counter], shrink=0.8)
```

As you can see, the AI and laplacian-norm anisotropy metrics have different contrasts:
- AI shows well the constrast between white and grey matter, but not too much between different white matter areas.
- The laplacian-norm emphasizes anisotropic/coherent white matter areas, but does not show as clear the grey/white matter areas.

### Visualizing Fiber Orientation Distributions

As before, we can again visualize the FOD estimated using CSD.

```python
from dipy.data import get_sphere
from dipy.viz.actor import slicer
from dipy.viz import fvtk
import numpy as np
import matplotlib.image as mpimg

sphere = get_sphere(name='symmetric724')
fods = csd_fit.fod(sphere.vertices)

affine = np.eye(4)
affine[0,3] = -10
affine[1,3] = -10
ai_im = slicer(ai[70:90,0, 70:90, None],
    interpolation='nearest', affine=affine, opacity=0.7)

ren = fvtk.ren()
fod_spheres = fvtk.sphere_funcs(fods[70:90,:, 70:90], sphere, scale=1., norm=False)
fod_spheres.RotateX(90)
fod_spheres.RotateZ(180)
fod_spheres.RotateY(180)
fvtk.add(ren, fod_spheres)
fvtk.add(ren, ai_im)
fvtk.record(ren=ren, size=[700, 700])

img = mpimg.imread('dipy.png')

plt.figure(figsize=[10, 10])
plt.imshow(img[100:-97, 100:-85])
plt.title('CSD FODs with Anistropy Index background', fontsize=20)
plt.axis('off');
```

Notice that in white matter the FODs are nicely shaped and we have a high AI, but in non-white matter they are very noisy and prominent. This is because the response function that we provided does not resemble the data in these areas, and we therefore get a non-sensical result, which is especially detrimental in partial-volumed voxels at the edge of the ventricles. In the following Multi-Compartment CSD example we will address this.

## Multi-Shell Multi-Compartment CSD

Dmipy allows to estimate multiple convolution kernels at the same time, i.e., multi-compartment CSD. The only limitation is that we only allow for one *anisotropic* kernel, like the Zeppelin before, and any number of isotropic compartments, like a Gaussian Ball or Sphere, as long as all the kernel parameters are fixed.

For this example, let us insert both a Zeppelin and a Ball into the MultiCompartmentSphericalHarmonicsModel to fit the white matter and CSF, respectively.

```python
ball = gaussian_models.G1Ball()
mod_mc_csd = MultiCompartmentSphericalHarmonicsModel(
    [zeppelin, ball], sh_order=8)
mod_mc_csd.parameter_names
```

    ['G2Zeppelin_1_lambda_perp',
     'G1Ball_1_lambda_iso',
     'G2Zeppelin_1_lambda_par',
     'partial_volume_0',
     'partial_volume_1',
     'sh_coeff']

Notice that we must now fix both the Zeppelin's and the Ball's diffusivities. For now, let us fix the Ball's diffusivity to free-water and keep the Zeppelin's diffusivities as before.

```python
mod_mc_csd.set_fixed_parameter('G2Zeppelin_1_lambda_par', lambda_par)
mod_mc_csd.set_fixed_parameter('G2Zeppelin_1_lambda_perp', lambda_perp)
mod_mc_csd.set_fixed_parameter('G1Ball_1_lambda_iso', 3e-9)
```

Since we use multiple kernels, the algorithm now chooses the 'csd_cvxpy' optimizer to solve the CSD problem. It can estimate the volume fractions of one anisotropic kernel and several isotropic kernels at the same time, but it is also significantly slower than the simpler 'csd_tournier07' optimizer.

Estimating volume fractions in this framework will automatically set `unity_constraint=True` since the volume fractions of white matter and CSD should add up to one.

```python
fit_mc_csd = mod_mc_csd.fit(
    acquisition_scheme=scheme_hcp,
    data=data_hcp,
    mask=data_hcp[..., 0]>0,
    solver='csd')
```

    Setup CVXPY FOD optimizer in 0.00613498687744 seconds
    Using parallel processing with 8 workers.
    Fitting of 8181 voxels complete in 126.349025011 seconds.
    Average of 0.0154442030328 seconds per voxel.

This time, we can also visualize the estimated volume fractions of the different kernel models.

```python
fitted_parameters = fit_mc_csd.fitted_parameters

fig, axs = plt.subplots(1, 2, figsize=[10, 5])
axs = axs.ravel()

counter = 0
for name, values in fitted_parameters.items():
    if values.squeeze().ndim != 2:
        continue
    cf = axs[counter].imshow(values.squeeze().T, origin=True, interpolation='nearest')
    axs[counter].set_title(name)
    axs[counter].set_axis_off()
    fig.colorbar(cf, ax=axs[counter], shrink=0.8)
    counter += 1
```

Here, partial_volume_0 corresponds to the Zeppelin representing white matter. partial_volume_1 corresponds to the Ball representing CSF. Notice it is not perfect and the Ball fraction is also significant in white matter . This is because 1) we did not include TE differences for white matter and CSF as *(Jeurissen et al. 2014)*, and 2) because we did not estimate the diffusivity of the CSF from the data. Still when we visualize the FODs we can see that a simple approach like this already helped.

```python
fods = fit_mc_csd.fod(sphere.vertices)
vf0_im = slicer(fitted_parameters['partial_volume_0'][70:90,0, 70:90, None],
    interpolation='nearest', affine=affine, opacity=0.7)

ren = fvtk.ren()
fod_spheres = fvtk.sphere_funcs(fods[70:90,:, 70:90], sphere, scale=1., norm=False)
fod_spheres.RotateX(90)
fod_spheres.RotateZ(180)
fod_spheres.RotateY(180)
fvtk.add(ren, fod_spheres)
fvtk.add(ren, vf0_im)
fvtk.record(ren=ren, size=[700, 700])

img = mpimg.imread('dipy.png')

plt.figure(figsize=[10, 10])
plt.imshow(img[100:-97, 100:-85])
plt.title('MC-CSD FODs with WM volume fraction background', fontsize=20)
plt.axis('off');
```

Despite our very simplistic implementation, notice how the FODs at the edge of ventricles are now less noisy and properly aligned, and the CSF FODs are nearly invisible because they are scaled with the volume fraction.

In this example we illustrated Dmipy's model-based multi-compartment CSD implementation. In a next example we will illustrate a proper model-free implementation of Multi-Tissue CSD implementation by *(Jeurissen et al. 2014)*.

## References
- Tournier, J-Donald, Fernando Calamante, and Alan Connelly. "Robust determination of the fibre orientation distribution in diffusion MRI: non-negativity constrained super-resolved spherical deconvolution." Neuroimage 35.4 (2007): 1459-1472.
- Descoteaux, Maxime, et al. "Regularized, fast, and robust analytical Q‐ball imaging." Magnetic resonance in medicine 58.3 (2007): 497-510.
- Tax, Chantal MW, et al. "Recursive calibration of the fiber response function for spherical deconvolution of diffusion MRI data." Neuroimage 86 (2014): 67-80.
- Jespersen, Sune N., et al. "Modeling dendrite density from magnetic resonance diffusion measurements." Neuroimage 34.4 (2007): 1473-1486.
- Jeurissen, Ben, et al. "Multi-tissue constrained spherical deconvolution for improved analysis of multi-shell diffusion MRI data." NeuroImage 103 (2014): 411-426.


## Variance Estimation

In statistics we know that the mean and variance of a population $Y$ are defined to be:

\begin{equation}
\left\{
\begin{aligned}
    \text{Mean}(Y) &= \mu = \frac{1}{N} \sum_{i=1}^{N} Y_i \\
    \text{Var}(Y) &= \sigma^2 = \frac{1}{N} \sum_{i=1}^{N} (Y_i - \mu)^2 \\
\end{aligned}
\right.
\end{equation}

where $N$ is the size of the population.

Given the population $Y$, we can draw a sample $X$ and compute statistics for $X$:

\begin{equation}
\left\{
\begin{aligned}
    \text{Mean}(X) &= \bar{X} = \frac{1}{n} \sum_{j=1}^{n} X_j \\
    \text{Var}(X) &= s^2 = \frac{1}{n - 1} \sum_{j=1}^{n} (X_j - \bar{X})^2 \\
\end{aligned}
\right.
\end{equation}

where lowercase $n$ is the size of the sample, typically a much smaller number than $N$. One detail that is often not clearly explained in introductory statistics is why we should divide by $n - 1$ instead of $n$ in the calculation for the sample variance.

## Why divide by n - 1?

It turns out that we should divide by $n - 1$ because dividing by $n$ would give us a **biased estimator** of the population variance. Let's look at a concrete example before diving into the math for why. Let's say we have a population of 100,000 data points. These can represent, for instance, a movie rating for each of 100,000 people.

```python
%matplotlib inline
import matplotlib.pyplot as plt
from IPython.core.pylabtools import figsize
figsize(15, 5)
```

```python
import pandas as pd
import numpy as np
```

```python
np.random.seed(42)
```

```python
N = 100000  # size of population
```

```python
population = pd.Series(np.random.randint(1, 11, N))
```

We can easily calculate the **population mean** and **population variance**:

```python
population.mean()
```

    5.5155599999999998

```python
((population - population.mean()) ** 2).sum() / N
```

    8.2592178864000001

Note that we are dividing by $N$ in the variance calculation, also that in `numpy` or `pandas` this is the same as simply using the method `var` with `ddof=0`

```python
population.var(ddof=0)
```

    8.2592178864000001

where `ddof=0` means to divide by $N$, and `ddof=1` means to divide by $N - 1$.

## Simulation

As usual in statistics, the population parameters are often unknown. But we can estimate them by drawing samples from the population. Here we are drawing a random sample of size $30$. As of version `0.16.1`, `pandas` has a convenient `Series.sample()` function for this:

```python
samples = {}
n = 30  # size of each sample
num_samples = 500  # we are drawing 500 samples, each with size n
for i in range(num_samples):
    samples[i] = population.sample(n).reset_index(drop=True)
```

```python
samples = pd.DataFrame(samples)
samples.T.tail()
```

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      ...
      20
      21
      22
      23
      24
      25
      26
      27
      28
      29

      495
      4
      6
      3
      7
      9
      1
      3
      6
      8
      7
      ...
      2
      9
      4
      5
      3
      1
      3
      5
      9
      6

      496
      8
      3
      5
      4
      7
      6
      10
      4
      1
      5
      ...
      5
      8
      1
      5
      3
      10
      2
      5
      9
      2

      497
      8
      4
      7
      5
      3
      10
      5
      7
      9
      2
      ...
      6
      9
      10
      3
      10
      3
      2
      6
      5
      8

      498
      3
      3
      6
      6
      1
      4
      6
      8
      10
      8
      ...
      8
      1
      3
      8
      3
      6
      4
      9
      7
      3

      499
      10
      5
      1
      3
      10
      7
      3
      6
      8
      9
      ...
      7
      7
      7
      4
      9
      6
      7
      8
      9
      2

5 rows × 30 columns

As we expect, if we average all the sample means we can see that the it is a good estimate for the true population mean:

```python
df = pd.DataFrame({'estimated mean': pd.expanding_mean(samples.mean()),
                   'actual population mean': pd.Series(population.mean(), index=samples.columns)})
df.plot(ylim=(4.5, 6.5))
```

Now let's compare the results we would get by using the **biased estimator** (dividing by $n$) and the **unbiased estimator** (dividing by $n-1$)

```python
df = pd.DataFrame({'biased var estimate (divide by n)': pd.expanding_mean(samples.var(ddof=0)),
                   'unbiased var estimate (divide by n - 1)': pd.expanding_mean(samples.var(ddof=1)),
                   'actual population var': pd.Series(population.var(ddof=0), index=samples.columns)})
df.plot(ylim=(6.5, 10.5))
```

We can see that the biased estimator (dividing by $n$) is clearly not estimating the true population variance as accurately as the unbiased estimator (dividing by $n-1$).

## Mathematical Proof

To prove that dividing by $n - 1$ is an unbiased estimator, we need to show that expected value of the estimaor is indeed $\sigma^2$:
\begin{equation}
    E(s^2) = E\left(\frac{1}{n - 1} \sum_{j=1}^{n} (X_j - \bar{X})^2\right) = \sigma^2
\end{equation}

First we'll need to recall a few basic properties of expectation and variance:

\begin{equation}
\left\{
\begin{aligned}
    & E(Z_1 + Z_2) = E(Z_1) + E(Z_2), \text{ for any } Z_1, Z_2 \\
    & \text{Var}(a Z) = a^2 \text{Var}(Z), \text{ for any } Z \\
    & \text{Var}(Z_1 + Z_2) = \text{Var}(Z_1) + \text{Var}(Z_2), \text{ if } Z_1 \text{ and } Z_2 \text{ are independent} \\
\end{aligned}
\right.
\end{equation}

Also, the following is a useful form for variance:
\begin{equation}
    \text{Var}(Z) = E((Z - E(Z))^2) = E(Z^2 - 2ZE(Z) + E(Z)^2) = E(Z^2) - E(Z)^2
\end{equation}

This is equivalent to
\begin{equation}
    E(Z^2) = \text{Var}(Z) + E(Z)^2
\end{equation}

Using the above properties we can now simplify the expression for $E(s^2)$:

\begin{aligned}
    E(s^2) = E\left(\frac{1}{n - 1} \sum_{j=1}^{n} (X_j - \bar{X})^2\right) = & \frac{1}{n - 1} E \left( \sum_{j=1}^{n} (X_j^2 - 2X_j\bar{X} + \bar{X}^2) \right) \\
    = & \ \frac{1}{n - 1} E \left( \sum_{j=1}^{n} X_j^2 - 2n\bar{X}^2 + n\bar{X}^2 \right) \\
    = & \ \frac{1}{n - 1} E \left( \sum_{j=1}^{n} X_j^2 - n\bar{X}^2 \right) \\
    = & \ \frac{1}{n - 1} \left[ E \left( \sum_{j=1}^{n} X_j^2 \right) - E \left( n\bar{X}^2 \right) \right] \\
    = & \ \frac{1}{n - 1} \left[ \sum_{j=1}^{n} E \left( X_j^2 \right) - n E \left( \bar{X}^2 \right) \right] \\
\end{aligned}

Now notice that the first term can be simplied as:

\begin{aligned}
    \sum_{j=1}^{n} E \left( X_j^2 \right) = & \sum_{j=1}^{n} \left( Var(X_j) + E(X_j)^2 \right) \\
    = & \sum_{j=1}^{n} \left( \sigma^2 + \mu ^2 \right) \\
    = & \ n \sigma^2 + n \mu ^2 \\
\end{aligned}

Using the same trick, the second term becomes:

\begin{aligned}
    E(\bar{X}^2) = & \ Var(\bar{X}) + E(\bar{X})^2 \\
    = & Var(\frac{1}{n} \sum_{j=1}^{n} X_j) + \mu ^2 \\
    = & \frac{1}{n^2} Var(\sum_{j=1}^{n} X_j) + \mu ^2 \\
    = & \frac{1}{n^2} \sum_{j=1}^{n} Var(X_j) + \mu ^2, \text{ because all } X_j\text{'s are independent} \\
    = & \frac{1}{n^2} n\sigma^2 + \mu ^2 \\
    = & \frac{1}{n} \sigma^2 + \mu ^2 \\
\end{aligned}

Plugging the two terms back we finally get:

\begin{aligned}
    E(s^2) = & \ \frac{1}{n-1} \left[ \sum_{j=1}^{n} E \left( X_j^2 \right) - n E \left(\bar{X}^2 \right) \right] \\
    = & \ \frac{1}{n-1} \left[n \sigma^2 + n \mu ^2 - n \left( \frac{1}{n} \sigma^2 + \mu ^2 \right) \right] \\
    = & \ \frac{1}{n-1} \left[n \sigma^2 + n \mu ^2 - \sigma^2 - n \mu ^2 \right] \\
    = & \ \sigma^2 \\
\end{aligned}

Dividing by $n-1$ gives us an unbiased estimate for the population variance indeed!

## Source of Bias

One intuitive way to think about why the bias exists is to notice that we generally don't actually know the true population mean $\mu$, and therefore the sample variance is being computed using the *estimated* mean $\bar{X}$. However the quadratic form $\sum_{j=1}^{n} (X_j - a)^2$ is actually minimized by $a = \bar{X}$, which means that whatever the true population mean $\mu$ is, we will always have

\begin{equation}
    \sum_{j=1}^{n} (X_j - \mu)^2 \geq \sum_{j=1}^{n} (X_j - \bar{X})^2
\end{equation}

Therefore we are underestimating the true variance because we don't know the true mean.

In fact, we can see that we are underestimating by exactly $\sigma^2$ on average:

\begin{aligned}
    E\left(\sum_{j=1}^{n} (X_j - \mu)^2\right) = & \ E \left(\sum_{j=1}^{n} (X_j - \bar{X} + \bar{X} - \mu)^2\right) \\
    = & \ E \left(\sum_{j=1}^{n} (X_j - \bar{X})^2 + \sum_{j=1}^{n} 2(X_j - \bar{X})(\bar{X} - \mu) + \sum_{j=1}^{n} (\bar{X} - \mu)^2 \right) \\
    = & \ E \left(\sum_{j=1}^{n} (X_j - \bar{X})^2 + \sum_{j=1}^{n} (\bar{X} - \mu)^2 \right) \\
    = & \ E \left(\sum_{j=1}^{n} (X_j - \bar{X})^2 \right) + E \left(\sum_{j=1}^{n} (\bar{X} - \mu)^2 \right) \\
    = & \ E \left(\sum_{j=1}^{n} (X_j - \bar{X})^2 \right) + \sum_{j=1}^{n} E \left((\bar{X} - \mu)^2 \right) \\
    = & \ E \left(\sum_{j=1}^{n} (X_j - \bar{X})^2 \right) + \sum_{j=1}^{n} \left( \text{Var} (\bar{X} - \mu) + E (\bar{X} - \mu)^2 \right) \\
    = & \ E \left(\sum_{j=1}^{n} (X_j - \bar{X})^2 \right) + \sum_{j=1}^{n} \left( \text{Var} (\bar{X}) + E (\bar{X} - \mu)^2 \right) \\
    = & \ E \left(\sum_{j=1}^{n} (X_j - \bar{X})^2 \right) + \sum_{j=1}^{n} \text{Var} (\bar{X}) \\
    = & \ E \left(\sum_{j=1}^{n} (X_j - \bar{X})^2 \right) + n \text{Var} (\bar{X}) \\
    = & \ E \left(\sum_{j=1}^{n} (X_j - \bar{X})^2 \right) + n \text{Var} (\frac{1}{n} \sum_{j=1}^{n} X_j) \\
    = & \ E \left(\sum_{j=1}^{n} (X_j - \bar{X})^2 \right) + n \frac{1}{n^2} \sum_{j=1}^{n} \text{Var} (X_j) \\
    = & \ E \left(\sum_{j=1}^{n} (X_j - \bar{X})^2 \right) + \sigma^2 \\
\end{aligned}

Combined with the result we have from the proof in the previous section, we can see that if we somehow *magically knew* the true mean $\mu$, dividing by $n$ would be unbiased:

\begin{aligned}
    E\left(\frac{1}{n} \sum_{j=1}^{n} (X_j - \mu)^2\right) = & \ \frac{1}{n} E \left(\sum_{j=1}^{n} (X_j - \mu)^2\right) \\
    = & \ \frac{1}{n} \left[ E \left(\sum_{j=1}^{n} (X_j - \bar{X})^2\right) + \sigma^2 \right] \\
        = & \ \frac{1}{n} \left[ (n - 1) \sigma^2 + \sigma^2 \right] \\
    = & \ \sigma^2 \\
\end{aligned}

However since we don't know the true mean and are using the estimated mean $\bar{X}$ instead, we'd need to divide by $n - 1$ to correct for the bias. This is also known as [Bessel's correction](http://en.wikipedia.org/wiki/Bessel%27s_correction#Source_of_bias).


```python
import matplotlib.pyplot as plt
import scipy.stats as st
import seaborn as sns
import pandas as pd
from scipy.stats import norm, uniform, expon
from scipy.integrate import quad
from sympy.solvers import solve
from sympy import Symbol
import numpy as np
from pandas import Series
```

```python
print(expon.cdf(x=4, scale=1/3))
```

    0.9999938557876467

```python
1-norm.cdf(x=130, loc=100, scale=15)
```

    0.02275013194817921

```python
norm.ppf(q=[0.025,0.975], loc=100, scale=15)
```

    array([ 70.60054023, 129.39945977])

```python
norm.cdf(x=115, loc=100, scale=15)-norm.cdf(x=85, loc=100, scale=15)
```

    0.6826894921370859

```python
f = lambda x: x * (15-x/4)
ans, _ = quad(f, 0,60)
print(ans)
```

    9000.0

```python
x = Symbol('x')
solve(x**2/9000*(15/2),x)
```

    [0.0]

```python
expon.cdf(x = 2/0.04, scale=1/0.04)
```

    0.8646647167633873

```python
f = lambda x: x * np.e**(-(1/1000)*x)
ans, _ = quad(f)
print(ans)
```

```python
1/(1/1000)
```

    1000.0

```python
1/(1/1000)
```

    1000.0

```python
1-(expon.cdf(x = 200, scale=1/0.04))*(expon.cdf(x = 200, scale=1/0.1000))
```

    0.0003354646883646506

```python
Series(norm.rvs(loc=32, scale=6, size=100000)).plot(kind="hist", bins=100)

```

```python
norm.cdf(x=40, loc=32, scale=6)
```

    0.9087887802741321

# Aufgabe 3.4 kann erst später gelöst werden

```python
norm.cdf(x=27, loc=32, scale=6)
```

    0.20232838096364308

```python
1-norm.cdf(x=0.9, loc=0, scale=0.45)
```

    0.02275013194817921

```python
norm.cdf(x=0.9, loc=1.8, scale=0.45)
```

    0.022750131948179195

```python
a,b = norm.cdf(x=[0.2485,0.2515], loc=0.2508, scale=0.0015)
```

```python
b-a
```

    0.6170339363078196

```python
norm.cdf(x=0.2515, loc=0.2508, scale=0.0005) - norm.cdf(x=0.2485, loc=0.2508, scale=0.0005)
```

    0.91924122831152

```python
norm.cdf(x=0.2515, loc=0.2500, scale=0.0005) - norm.cdf(x=0.2485, loc=0.2500, scale=0.0005)
```

    0.9973002039367398

```python
norm.ppf(q=0.9)
```

    1.2815515655446004

```python
norm.cdf(x=2)
```

    0.9772498680518208

```python
norm.cdf(x=-2)
```

    0.022750131948179195

```python
norm.cdf(x=2)+norm.cdf(x=-2)
```

    1.0

```python
from scipy.integrate import quad
f = lambda x: 1/10 - x/200
ans, _ = quad(f, 0, 5)
print(ans)
```

    0.43750000000000006

```python
print(5/10 - 25/400)
```

    0.4375

```python

```


```python
import numpy as np
import math
import sympy
import sympy as sp
from sympy import latex, Eq, IndexedBase, symbols, Idx, Indexed, Sum, S, diff, Array, tensorproduct, Matrix
from sympy.functions.special.tensor_functions import KroneckerDelta
import matplotlib.pyplot as plt
from sympy.vector import Vector, CoordSys3D, AxisOrienter, BodyOrienter, Del, curl, divergence, gradient, is_conservative, is_solenoidal, scalar_potential, Point, scalar_potential_difference
from sympy.physics.vector import ReferenceFrame
from sympy.vector import express
import matplotlib as mpl
from mpl_toolkits.mplot3d import Axes3D
from IPython.display import display, display_latex
```

```python
a, b, c, d, e, f, g, h, i, j, k, l, m, n, o, p, q, r, s, t, u, v, w, x, y, z = sympy.symbols('a:z')
a_1, a_2, a_3, b_1, b_2, b_3, c_1, c_2, c_3, d_1, d_2, d_3 = symbols('a_1 a_2 a_3 b_1 b_2 b_3 c_1 c_2 c_3 d_1 d_2 d_3')
alpha, beta, gamma, theta, phi, rho, omega = sympy.symbols('alpha beta gamma theta phi rho omega')
Alpha, Beta, Gamma, Theta, Phi, Rho, Omega = sympy.symbols('Alpha Beta Gamma Theta Phi Rho Omega')
```

## Setup

```python
sympy.init_printing()
np.set_printoptions(precision=3)
```

### Symbols

Cross product: ^
Dot product: &
Outer product: |

```python
a, b, c, d, e, f = symbols('a:f')
u1, u2, u3, v1, v2, v3 = symbols('u_1 u_2 u_3 v_1 v_2 v_3')
E = ReferenceFrame('E')
u = u1 * E.x + u2 * E.y + u3 * E.z
v = v1 * E.x + v2 * E.y + v3 * E.z
u
```

#### Dot Product

```python
u & v
```

#### Cross Product

```python
u ^ v
```

### Normalize

```python
u.normalize()
```

### Magnitude

```python
u.magnitude()
```

```python
N = CoordSys3D('N')
```

```python
v = 2 * N.i + N.j
v
```

```python
Vector.zero
```

```python
N.origin
```

```python
P = N.origin.locate_new('P', a*N.i + b*N.j + c*N.k)
Q = P.locate_new('Q', -b*N.j)
P
```

```python
P.position_wrt(Q)
```

```python
Q.express_coordinates(N)
```

#### Dyadics

```python
dyad = N.i.outer(N.k)
dyad
```

```python
A = CoordSys3D('A')
B = A.orient_new_axis('B', theta, A.k)
B
```

$\displaystyle CoordSys3D\left(B, \left( \left[\begin{matrix}\cos{\left(\theta \right)} & \sin{\left(\theta \right)} & 0\\- \sin{\left(\theta \right)} & \cos{\left(\theta \right)} & 0\\0 & 0 & 1\end{matrix}\right], \  \mathbf{\hat{0}}\right), CoordSys3D\left(A, \left( \left[\begin{matrix}1 & 0 & 0\\0 & 1 & 0\\0 & 0 & 1\end{matrix}\right], \  \mathbf{\hat{0}}\right)\right)\right)$

```python
axis_orienter = AxisOrienter(theta, A.k)
B = A.orient_new('B', axis_orienter)
B
```

$\displaystyle CoordSys3D\left(B, \left( \left[\begin{matrix}\cos{\left(\theta \right)} & \sin{\left(\theta \right)} & 0\\- \sin{\left(\theta \right)} & \cos{\left(\theta \right)} & 0\\0 & 0 & 1\end{matrix}\right], \  \mathbf{\hat{0}}\right), CoordSys3D\left(A, \left( \left[\begin{matrix}1 & 0 & 0\\0 & 1 & 0\\0 & 0 & 1\end{matrix}\right], \  \mathbf{\hat{0}}\right)\right)\right)$

#### Chain Rotations

```python
A = CoordSys3D('A')
body_orienter = BodyOrienter(a, b, c, 'XYZ')
B = A.orient_new('B', (axis_orienter, body_orienter)).subs(theta, 0)
B
```

$\displaystyle CoordSys3D\left(B, \left( \left[\begin{matrix}\cos{\left(b \right)} \cos{\left(c \right)} & \sin{\left(a \right)} \sin{\left(b \right)} \cos{\left(c \right)} + \sin{\left(c \right)} \cos{\left(a \right)} & \sin{\left(a \right)} \sin{\left(c \right)} - \sin{\left(b \right)} \cos{\left(a \right)} \cos{\left(c \right)}\\- \sin{\left(c \right)} \cos{\left(b \right)} & - \sin{\left(a \right)} \sin{\left(b \right)} \sin{\left(c \right)} + \cos{\left(a \right)} \cos{\left(c \right)} & \sin{\left(a \right)} \cos{\left(c \right)} + \sin{\left(b \right)} \sin{\left(c \right)} \cos{\left(a \right)}\\\sin{\left(b \right)} & - \sin{\left(a \right)} \cos{\left(b \right)} & \cos{\left(a \right)} \cos{\left(b \right)}\end{matrix}\right], \  \mathbf{\hat{0}}\right), CoordSys3D\left(A, \left( \left[\begin{matrix}1 & 0 & 0\\0 & 1 & 0\\0 & 0 & 1\end{matrix}\right], \  \mathbf{\hat{0}}\right)\right)\right)$

#### Use Multiple Orienters

```python
C = A.orient_new('C', (axis_orienter, body_orienter))
C
```

$\displaystyle CoordSys3D\left(C, \left( \left[\begin{matrix}- \sin{\left(c \right)} \sin{\left(\theta \right)} \cos{\left(b \right)} + \cos{\left(b \right)} \cos{\left(c \right)} \cos{\left(\theta \right)} & \left(- \sin{\left(a \right)} \sin{\left(b \right)} \sin{\left(c \right)} + \cos{\left(a \right)} \cos{\left(c \right)}\right) \sin{\left(\theta \right)} + \left(\sin{\left(a \right)} \sin{\left(b \right)} \cos{\left(c \right)} + \sin{\left(c \right)} \cos{\left(a \right)}\right) \cos{\left(\theta \right)} & \left(\sin{\left(a \right)} \sin{\left(c \right)} - \sin{\left(b \right)} \cos{\left(a \right)} \cos{\left(c \right)}\right) \cos{\left(\theta \right)} + \left(\sin{\left(a \right)} \cos{\left(c \right)} + \sin{\left(b \right)} \sin{\left(c \right)} \cos{\left(a \right)}\right) \sin{\left(\theta \right)}\\- \sin{\left(c \right)} \cos{\left(b \right)} \cos{\left(\theta \right)} - \sin{\left(\theta \right)} \cos{\left(b \right)} \cos{\left(c \right)} & \left(- \sin{\left(a \right)} \sin{\left(b \right)} \sin{\left(c \right)} + \cos{\left(a \right)} \cos{\left(c \right)}\right) \cos{\left(\theta \right)} - \left(\sin{\left(a \right)} \sin{\left(b \right)} \cos{\left(c \right)} + \sin{\left(c \right)} \cos{\left(a \right)}\right) \sin{\left(\theta \right)} & - \left(\sin{\left(a \right)} \sin{\left(c \right)} - \sin{\left(b \right)} \cos{\left(a \right)} \cos{\left(c \right)}\right) \sin{\left(\theta \right)} + \left(\sin{\left(a \right)} \cos{\left(c \right)} + \sin{\left(b \right)} \sin{\left(c \right)} \cos{\left(a \right)}\right) \cos{\left(\theta \right)}\\\sin{\left(b \right)} & - \sin{\left(a \right)} \cos{\left(b \right)} & \cos{\left(a \right)} \cos{\left(b \right)}\end{matrix}\right], \  \mathbf{\hat{0}}\right), CoordSys3D\left(A, \left( \left[\begin{matrix}1 & 0 & 0\\0 & 1 & 0\\0 & 0 & 1\end{matrix}\right], \  \mathbf{\hat{0}}\right)\right)\right)$

#### Get Rotation Matrix W.R.T

```python
B.rotation_matrix(A)
```

$\displaystyle \left[\begin{matrix}- \sin{\left(c \right)} \sin{\left(\theta \right)} \cos{\left(b \right)} + \cos{\left(b \right)} \cos{\left(c \right)} \cos{\left(\theta \right)} & \left(- \sin{\left(a \right)} \sin{\left(b \right)} \sin{\left(c \right)} + \cos{\left(a \right)} \cos{\left(c \right)}\right) \sin{\left(\theta \right)} + \left(\sin{\left(a \right)} \sin{\left(b \right)} \cos{\left(c \right)} + \sin{\left(c \right)} \cos{\left(a \right)}\right) \cos{\left(\theta \right)} & \left(\sin{\left(a \right)} \sin{\left(c \right)} - \sin{\left(b \right)} \cos{\left(a \right)} \cos{\left(c \right)}\right) \cos{\left(\theta \right)} + \left(\sin{\left(a \right)} \cos{\left(c \right)} + \sin{\left(b \right)} \sin{\left(c \right)} \cos{\left(a \right)}\right) \sin{\left(\theta \right)}\\- \sin{\left(c \right)} \cos{\left(b \right)} \cos{\left(\theta \right)} - \sin{\left(\theta \right)} \cos{\left(b \right)} \cos{\left(c \right)} & \left(- \sin{\left(a \right)} \sin{\left(b \right)} \sin{\left(c \right)} + \cos{\left(a \right)} \cos{\left(c \right)}\right) \cos{\left(\theta \right)} - \left(\sin{\left(a \right)} \sin{\left(b \right)} \cos{\left(c \right)} + \sin{\left(c \right)} \cos{\left(a \right)}\right) \sin{\left(\theta \right)} & - \left(\sin{\left(a \right)} \sin{\left(c \right)} - \sin{\left(b \right)} \cos{\left(a \right)} \cos{\left(c \right)}\right) \sin{\left(\theta \right)} + \left(\sin{\left(a \right)} \cos{\left(c \right)} + \sin{\left(b \right)} \sin{\left(c \right)} \cos{\left(a \right)}\right) \cos{\left(\theta \right)}\\\sin{\left(b \right)} & - \sin{\left(a \right)} \cos{\left(b \right)} & \cos{\left(a \right)} \cos{\left(b \right)}\end{matrix}\right]$

#### Rotate and Translate

```python
C = A.orient_new_axis('C', a, A.k, location=2*A.j)
C.position_wrt(A)
```

#### Express : Vector instances can be expressed in user defined systems using express.

```python
M = N.orient_new_axis('M', a, N.k)
v1 = N.i + N.j + N.k
express(v1, M)
```

```python
v = N.i + N.j + N.k
v.to_matrix(M)
```

$\displaystyle \left[\begin{matrix}\sin{\left(a \right)} + \cos{\left(a \right)}\\- \sin{\left(a \right)} + \cos{\left(a \right)}\\1\end{matrix}\right]$

### Scalar and Vector Fields and Base Scalar

potential field = $2x^2y$

```python
R = CoordSys3D('R')
electric_potential = 2*R.x**2 * R.y
electric_potential
```

```python
diff(electric_potential, R.x)
```

```python
v = R.x**2*R.i + 2*R.x*R.z*R.k
v
```

#### Gradient

```python
C = CoordSys3D('C')
delop = Del()
gradient_field = delop(C.x*C.y*C.z)
gradient_field
```

```python
gradient_field.doit()
```

```python
gradient(C.x*C.y*C.z)
```

```python
delop.gradient(C.x*C.y*C.z).doit()
```

#### Curl

##### Method 1

```python
delop = Del()
delop.cross(C.x*C.y*C.z*C.i)
```

```python
delop.cross(C.x*C.y*C.z*C.i).doit()
```

##### Method 2

```python
curl(C.x*C.y*C.z*C.i)
```

#### Divergence

```python
divergence(C.x*C.y*C.z*(C.i + C.j + C.k))
```

```python
delop.dot(C.x*C.y*C.z*(C.i + C.j + C.k)).doit()
```

#### Directional Derivative

```python
from sympy.vector import directional_derivative
directional_derivative(C.x*C.y*C.z, 3*C.i + 4*C.j + C.k)
```

#### Conservative Field
* Curl = 0
* Irrotational
* Conservation of Energy
* Gradient of some scalar field

```python
R = CoordSys3D('R')
field = R.y*R.z*R.i + R.x*R.z*R.j + R.x*R.y*R.k
is_conservative(field)
```

    True

```python
curl(field)
```

#### Solenoidal Field
* Div = 0

```python
R = CoordSys3D('R')
field = R.y*R.z*R.i + R.x*R.z*R.j + R.x*R.y*R.k
is_solenoidal(field)
```

    True

```python
divergence(field)
```

#### Scalar Potential Functions

```python
R = CoordSys3D('R')
conservative_field = 4*R.x*R.y*R.z*R.i + 2*R.x**2*R.z*R.j + 2*R.x**2*R.y*R.k
scalar_potential(conservative_field, R)
```

```python
R = CoordSys3D('R')
P = R.origin.locate_new('P', 1*R.i + 2*R.j + 3*R.k)
vectfield = 4*R.x*R.y*R.i + 2*R.x**2*R.j
scalar_potential_difference(vectfield, R, R.origin, P)
```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

## CHAPTER 1

```python
#E = ReferenceFrame('E')
#E = CoordSys3D('E')
```

```python
E = CoordSys3D('E')
A = a_1*E.i + a_2*E.j + a_3 * E.k
B = b_1*E.i + b_2*E.j + b_3 * E.k
```

```python
A
```

```python
B
```

```python
A & B
```

### Dot/Inner Product Using Vector Lengths

$A.B \equiv \frac{1}{4} [(A + B)^2 - (A - B)^2]$

```python
1/4*((A + B).magnitude()**2 - (A - B).magnitude()**2).simplify()
```

### Metric Tensor

$g(A, B) \equiv A.B$

### Tensor Product $\otimes$
* Every second rank tensor can be expressed as a sum of tensor products

### Contraction

* $contraction(A \otimes B) \equiv A.B$
* Reduces the rank of a tensor by 2
*  If $T = A \otimes B + C \otimes D + ..., contraction(T) = A.B + C.D + ...$
* If T is a rank 3 tensor $A \otimes B \otimes C + E \otimes F \otimes G + . . . $, $1 \& 3contraction(A \otimes B \otimes C + E \otimes F \otimes G + . . .) \equiv (A . C)B + (E . G)F + . . . .$

### Exercise 1.1

Knowledge:
* $\frac{dp}{dt} = F$ = m$a$
* Lorentz force $F$ = q$(E + v \times B)$
* Lorentz force $F$ = q$E$ (Assumming no magnetic field)
* E = m$v^2$

```python
m, q, t, v, B, E, F = sympy.symbols('m q t v B E F')
```

```python
Energy = m * v**2
```

```python
rate = Energy / t
rate
```

```python
rate = rate.subs(m*v/t, F)
rate
```

```python
rate = rate.subs(F, q*E)
rate
```

### Exercise 1.2

```python

```

```python

```

```python

```

```python

```

### 1.5 Component Representation of Tensor Algebra

```python
p, R = sympy.symbols('\overline{p} R')
```

```python
a = sympy.Matrix([1, 2, 3])
```

```python

```

```python

```

### Exercise 1.6

```python
E = CoordSys3D('E')
E_ = E.orient_new_axis('\overline{E}', phi, E.k)
# A = E.i + E.j + E.k # To create an actual vector object that supports to_matrix([CoordSys3D]) to get the transformation matrix
#A.to_matrix(E)
#E_.position_wrt(E)
#E_.origin.express_coordinates(E)
#type(E_.origin)

T = E_.rotation_matrix(E) # Immutable dense matrix
T
```

$\displaystyle \left[\begin{matrix}\cos{\left(\phi \right)} & \sin{\left(\phi \right)} & 0\\- \sin{\left(\phi \right)} & \cos{\left(\phi \right)} & 0\\0 & 0 & 1\end{matrix}\right]$

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```


# Logistic Regression

```python
from IPython.display import Image
Image('../../../python_for_probability_statistics_and_machine_learning.jpg')
```

The Bernoulli distribution we studied earlier answers the question of which of
two outcomes ($Y \in \lbrace 0,1 \rbrace$) would be selected with probability,
$p$.

$$
\mathbb{P}(Y) = p^Y (1-p)^{ 1-Y }
$$

 We also know how to solve the corresponding likelihood function for
the maximum likelihood estimate of $p$ given observations of the output,
$\lbrace Y_i \rbrace_{i=1}^n$. However, now we want to include other factors in
our estimate of $p$. For example, suppose we observe not just the outcomes, but
a corresponding continuous variable, $x$. That is, the observed data is now
$\lbrace (x_i,Y_i) \rbrace_{i=1}^n$  How can we incorporate $x$ into our
estimation of $p$?

The most straightforward idea is to model $p= a x + b$ where $a,b$ are
parameters of a fitted line. However, because $p$ is a probability with value
bounded between zero and one, we need to wrap this estimate in another function
that can map the entire real line into the $[0,1]$ interval. The logistic
(a.k.a. sigmoid) function has this property,

$$
\theta(s) = \frac{e^s}{1+e^s}
$$

 Thus, the new parameterized estimate for $p$ is the following,

$$
\begin{equation}
\hat{p} = \theta(a x+b)= \frac{e^{a x + b}}{1+e^{a x + b}}
\label{eq:prob} \tag{1}
\end{equation}
$$

 This is usually expressed using the *logit* function,

$$
\texttt{logit}(t)= \log \frac{t}{1-t}
$$

 as,

$$
\texttt{logit}(p) = b + a x
$$

 More continuous variables can be accommodated easily as

$$
\texttt{logit}(p) = b + \sum_k a_k x_k
$$

 This can be further extended beyond the binary case to multiple
target labels. The maximum likelihood estimate of this uses
numerical optimization methods that are implemented in Scikit-learn.

Let's construct some data to see how this works. In the following, we assign
class labels to a set of randomly scattered points in the two-dimensional
plane,

```python
%matplotlib inline

import numpy as np
from matplotlib.pylab import subplots
v = 0.9
@np.vectorize
def gen_y(x):
    if x<5: return np.random.choice([0,1],p=[v,1-v])
    else:   return np.random.choice([0,1],p=[1-v,v])

xi = np.sort(np.random.rand(500)*10)
yi = gen_y(xi)
```

**Programming Tip.**

The `np.vectorize` decorator used in the code above makes it easy to avoid
looping in code that uses Numpy arrays by embedding the looping semantics
inside of the so-decorated function. Note, however, that this does not
necessarily accelerate the wrapped function. It's mainly for convenience.

[Figure](#fig:logreg_001) shows a scatter plot of the data we constructed in
the above code, $\lbrace (x_i,Y_i) \rbrace$. As constructed, it is more
likely that large values of $x$ correspond to $Y=1$. On the other hand, values
of $x \in [4,6]$ of either category are heavily overlapped. This means that $x$
is not a particularly strong indicator of $Y$ in this region.
[Figure](#fig:logreg_002) shows the fitted logistic regression curve against the
same
data. The points along the curve are the probabilities that each point lies in
either of the two categories. For large values of $x$ the curve is near one,
meaning that the probability that the associated $Y$ value is equal to one. On
the other extreme, small values of $x$ mean that this probability is close to
zero.  Because there are only two possible categories, this means that the
probability of $Y=0$ is thereby higher. The region in the middle corresponding
to the middle probabilities reflect the ambiguity between the two catagories
because of the overlap in the data for this region. Thus, logistic regression
cannot make a strong case for one category here.
The following code fits the logistic regression model,

```python
fig,ax=subplots()
_=ax.plot(xi,yi,'o',color='gray',alpha=.3)
_=ax.axis(ymax=1.1,ymin=-0.1)
_=ax.set_xlabel(r'$X$',fontsize=22)
_=ax.set_ylabel(r'$Y$',fontsize=22)
```

```python
from sklearn.linear_model import LogisticRegression
lr = LogisticRegression()
lr.fit(np.c_[xi],yi)
```

    LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,
              intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,
              penalty='l2', random_state=None, solver='liblinear', tol=0.0001,
              verbose=0, warm_start=False)

```python
fig,ax=subplots()
xii=np.linspace(0,10,20)
_=ax.plot(xii,lr.predict_proba(np.c_[xii])[:,1],'k-',lw=3)
_=ax.plot(xi,yi,'o',color='gray',alpha=.3)
_=ax.axis(ymax=1.1,ymin=-0.1)
_=ax.set_xlabel(r'$x$',fontsize=20)
_=ax.set_ylabel(r'$\mathbb{P}(Y)$',fontsize=20)
```

This scatterplot shows the binary $Y$ variables and the corresponding $x$
data for each category.

This shows the fitted logistic regression on the data shown in
[Figure](#fig:logreg_001). The points along the curve are the probabilities that
each point lies in either of the two categories.

For a deeper understanding of logistic regression, we need to alter our
notation slightly and once again use our projection methods. More generally we
can rewrite Equation [eq:prob](#eq:prob) as the following,

$$
\begin{equation}
p(\mathbf{x}) = \frac{1}{1+\exp(-\boldsymbol{\beta}^T \mathbf{x})}
\label{eq:probbeta} \tag{2}
\end{equation}
$$

 where $\boldsymbol{\beta}, \mathbf{x}\in \mathbb{R}^n$.  From our
prior work on projection we know that the signed perpendicular distance between
$\mathbf{x}$ and the linear boundary described by $\boldsymbol{\beta}$ is
$\boldsymbol{\beta}^T \mathbf{x}/\Vert\boldsymbol{\beta}\Vert$.  This means
that the probability that is assigned to any point in $\mathbb{R}^n$ is a
function of how close that point is to the linear boundary described by the
following equation,

$$
\boldsymbol{\beta}^T \mathbf{x} = 0
$$

 But there is something subtle hiding here. Note that
for any $\alpha\in\mathbb{R}$,

$$
\alpha\boldsymbol{\beta}^T \mathbf{x} = 0
$$

 describes the *same* hyperplane. This means that we can multiply
$\boldsymbol{\beta}$ by an arbitrary scalar and still get the same geometry.
However, because of $\exp(-\alpha\boldsymbol{\beta}^T \mathbf{x})$ in Equation
[eq:probbeta](#eq:probbeta), this scaling determines the intensity of the
probability
attributed to $\mathbf{x}$. This is illustrated in [Figure](#fig:logreg_003).
The panel on the left shows two categories (squares/circles) split by the
dotted line that is determined by $\boldsymbol{\beta}^T\mathbf{x}=0$. The
background colors shows the probabilities assigned to points in the plane.  The
right panel shows that by scaling with $\alpha$, we can increase the
probabilities of class membership for the given points, given the exact same
geometry. The points near the boundary have lower probabilities because they
could easily be on the opposite side.  However, by scaling by $\alpha$, we can
raise those probabilities to any desired level at the cost of driving the
points further from the boundary closer to one. Why is this a problem? By
driving the probabilities arbitrarily using $\alpha$, we can overemphasize the
training set at the cost of out-of-sample data. That is, we may wind up
insisting on emphatic class membership of yet unseen points that are close to
the boundary that otherwise would have more equivocal probabilities (say, near
$1/2$).  Once again, this is another manifestation of bias/variance trade-off.

Scaling can arbitrarily increase the probabilities of points near the
decision boundary.

Regularization is a method that controls this effect by penalizing the size of
$\beta$ as part of its solution. Algorithmically, logistic regression works by
iteratively solving a sequence of weighted least squares problems. Regression
adds a $\Vert\boldsymbol{\beta}\Vert/C$ term to the least squares error. To see
this in action, let's create some data from a logistic regression and see if we
can recover it using Scikit-learn. Let's start with a scatter of points in the
two-dimensional  plane,

```python
x0,x1=np.random.rand(2,20)*6-3
X = np.c_[x0,x1,x1*0+1] # stack as columns
```

Note that `X` has a third column of all ones. This is a
trick to allow the corresponding line to be offset from the origin
in the two-dimensional plane. Next, we create a linear boundary
and assign the class probabilities according to proximity to the
boundary.

```python
beta = np.array([1,-1,1]) # last coordinate for affine offset
prd = X.dot(beta)
probs = 1/(1+np.exp(-prd/np.linalg.norm(beta)))
c = (prd>0) # boolean array class labels
```

This establishes the training data.  The next block
creates the logistic regression object and fits the data.

```python
lr = LogisticRegression()
_=lr.fit(X[:,:-1],c)
```

Note that we have to omit the third dimension because of
how Scikit-learn internally breaks down the components of the
boundary. The resulting code extracts the corresponding
$\boldsymbol{\beta}$ from the `LogisticRegression` object.

```python
betah = np.r_[lr.coef_.flat,lr.intercept_]
```

**Programming Tip.**

The Numpy `np.r_`  object provides a quick way to stack Numpy
arrays horizontally instead of using `np.hstack`.

 The resulting boundary is shown in the left panel in
[Figure](#fig:logreg_004). The crosses and triangles represent the two classes
we
created above, along with the separating gray line.  The logistic regression
fit produces the dotted black line. The dark circle is the point that logistic
regression categorizes incorrectly. The regularization parameter is $C=1$ by
default. Next, we can change the strength of the regularization parameter as in
the following,

```python
lr = LogisticRegression(C=1000)
```

and the re-fit the data to produce the right panel in
[Figure](#fig:logreg_004). By increasing the regularization
parameter, we essentially nudged the fitting algorithm to
*believe* the data more than the general model. That is, by doing
this we accepted more variance in exchange for better bias.

The left panel shows the resulting boundary (dashed line) with $C=1$ as the
regularization parameter. The right panel is for $C=1000$. The gray line is the
boundary used to assign the class membership for the synthetic data. The dark
circle is the point that logistic regression categorizes incorrectly.

## Generalized Linear Models

Logistic regression is one example of a wider class of generalized linear
models that embed non-linear transformations in the fitting process. Let's back
up and break down logistic regression into smaller parts. As usual, we want to
estimate the conditional expectation $\mathbb{E}(Y\vert X=\mathbf{x})$.  For
plain linear regression, we have the following approximation,

$$
\mathbb{E}(Y\vert X=\mathbf{x})\approx\boldsymbol{\beta}^T\mathbf{x}
$$

 For notation sake, we call $r(x):=\mathbb{E}(Y\vert X=\mathbf{x})$
the response. For logistic regression, because $Y\in\left\{0,1\right\}$, we
have $\mathbb{E}(Y\vert X=\mathbf{x})=\mathbb{P}(Y\vert X=\mathbf{x})$ and the
transformation makes $r(\mathbf{x})$ linear.

$$
\begin{align*}
\eta(\mathbf{x}) &= \boldsymbol{\beta}^T\mathbf{x}     \\\
        &= \log \frac{r(\mathbf{x})}{1-r(\mathbf{x})}  \\\
        &= g(r(\mathbf{x}))
\end{align*}
$$

 where $g$ is defined as the logistic *link* function.
The $\eta(x)$ function is the linear predictor. Now that we have
transformed the original data space using the logistic function to
create the setting for the linear predictor, why don't we just do
the same thing for the $Y_i$ data? That is, for plain linear
regression, we usually take data, $\left\{X_i,Y_i\right\}$ and
then use it to fit an approximation to $\mathbb{E}(Y\vert X=x)$.
If we are transforming the conditional expectation using the
logarithm, which we are approximating using $Y_i$, then why don't
we correspondingly transform the binary $Y_i$ data? The answer is
that if we did so then we would get the logarithm of zero (i.e.,
infinity) or one (i.e., zero), which is not workable. The
alternative is to use a linear Taylor approximation, like we did
earlier with the delta method, to expand the $g$ function around
$r(x)$, as in the following,

$$
\begin{align*}
g(Y) &\approx \log\frac{r(x)}{1-r(x)} + \frac{Y-r(x)}{r(x)-r(x)^2} \\\
     &= \eta(x)+ \frac{Y-r(x)}{r(x)-r(x)^2}
\end{align*}
$$

 The interesting part is the $Y-r(x)$ term, because this is where the
class label data enters the problem. The expectation $\mathbb{E}(Y-r(x)\vert
X)=0$ so we can think of this differential as additive noise that dithers
$\eta(x)$. The variance of $g(Y)$ is the following,

$$
\begin{align*}
\mathbb{V}(g(Y)\vert X)&= \mathbb{V}(\eta(x)\vert X)+\frac{1}{(r(x)(1-r(x)))^2}
\mathbb{V}(Y-r(x)\vert X) \\\
                       &=\frac{1}{(r(x)(1-r(x)))^2} \mathbb{V}(Y-r(x)\vert X)
\end{align*}
$$

 Note that $\mathbb{V}(Y\vert X)=r(x)(1-r(x))$ because $Y$ is
a binary variable. Ultimately, this boils down to the following,

$$
\mathbb{V}(g(Y)\vert X)=\frac{1}{r(x)(1-r(x))}
$$

 Note that the variance is a function of $x$, which means it is
*heteroskedastic*, meaning that the iterative minimum-variance-finding
algorithm that computes $\boldsymbol{\beta}$ downplays $x$ where $r(x)\approx
0$ and $r(x)\approx 1$ because the peak of the variance occurs where
$r(x)\approx 0.5$, which are those equivocal points are close to the boundary.

For generalized linear models, the above sequence is the same and consists of
three primary ingredients: the linear predictor ($\eta(x)$), the link function
($g(x)$), and the *dispersion scale function*, $V_{ds}$ such that
$\mathbb{V}(Y\vert X)=\sigma^2 V_{ds}(r(x))$. For logistic regression, we have
$V_{ds}(r(x))=r(x)(1-r(x))$ and $\sigma^2=1$. Note that absolute knowledge of
$\sigma^2$ is not important because the iterative algorithm needs only a
relative proportional scale. To sum up, the iterative algorithm takes a linear
prediction for $\eta(x_i)$, computes the transformed responses, $g(Y_i)$,
calculates the weights $w_i=\left[(g^\prime(r(x_i))V_{ds}(r(x_i)))
\right]^{-1}$, and then does a weighted linear regression of $g(y_i)$ onto
$x_i$ with the weights $w_i$ to compute the next $\boldsymbol{\beta}$.
More details can be found in the following [[fox2015applied]](#fox2015applied),
[[lindsey1997applying]](#lindsey1997applying),
[[campbell2009generalized]](#campbell2009generalized).


# 勾配

- ある関数の偏微分
- その関数の変化を表す
- ロス関数の値を減らす方向が分かる

```python
import numpy as np
```

```python
# 数値微分
def numerical_diff(f, x):
    h = 0.0000000001
    e = (f(x+h) - f(x-h)) / (2*h) # 中心差分
    return e
```

2つの変数を持つ関数

\begin{equation}
  f(x_1, x_2) = x_1^2 + x_2^2
\end{equation}

x1 = 2.0, x2 = 3.0のとき、偏微分を求める

$$
\frac{\partial f}{\partial x_1}
$$

```python
def func1(x1):
    x2 = 3.0
    e = pow(x1, 2) + pow(x2, 2)
    return e
```

```python
numerical_diff(func1, 2.0)
```

    4.000000330961484

# 勾配法

- 勾配方向への移動を繰り返すことで、関数の値を減らす（または増やす方向）
- ディープラーニングでは勾配降下法（ロス関数の値を減らすため）

\begin{equation}
  x_n = x_n - \eta\frac{\partial f}{\partial x_n}
\end{equation}

```python
def gradient(f, x):
    h = 0.0000000001
    g = np.zeros_like(x)

    for i in range(x.size):
        # 偏微分 : 変数一つずつを対象に微分
        tmp = x[i]
        x[i] = tmp + h
        f1 = f(x)
        x[i] = tmp - h
        f2 = f(x)
        x[i] = tmp # 戻す
        g[i] = (f1-f2) / (2*h)

    return g
```

```python
def gradient_descent(f, x, le, step):
    for i in range(step):
        g = gradient(f, x)
        x -= le * g
    return x
```

```python
def func2(x):
    y = 2.0 * pow(x[0], 2.0) + 2.0 * pow(x[1], 2.0)
    return y
```

テスト

\begin{equation}
  f(x_1, x_2) = 2x_1^2 + 2x_2^2
\end{equation}

の最小値を求める

解 : (0, 0)

### 学習率について

```python
# 学習率 0.1(まあまあ適切)
x = np.array([5.0, -5.0])
gradient_descent(func2, x, 0.1, 100)
```

    array([  3.26660446e-22,  -3.26660446e-22])

```python
# 学習率 1.0（大きい）
x = np.array([5.0, -5.0])
gradient_descent(func2, x, 1.0, 100)
```

    array([-1244739.50700518,  1244739.50700518])

```python
# 学習率 0.01（小さい）
x = np.array([5.0, -5.0])
gradient_descent(func2, x, 0.01, 100)
```

    array([ 0.08435156, -0.08435156])

### ステップ数について

```python
# ステップ数 10
x = np.array([5.0, -5.0])
gradient_descent(func2, x, 0.1, 10)
```

    array([ 0.03023296, -0.03023296])

```python
# ステップ数 1000
x = np.array([5.0, -5.0])
gradient_descent(func2, x, 0.1, 1000)
```

    array([  6.17694113e-27,  -6.17694113e-27])

```python
# 学習率 0.01 ステップ数 1000
x = np.array([5.0, -5.0])
gradient_descent(func2, x, 0.01, 1000)
```

    array([  9.33690313e-18,  -9.33690313e-18])

# 確立的勾配降下法

ランダムに選んだデータから勾配を求める

# モーメンタムを使用

- momentum
- 確立的勾配降下法では、ランダムにデータを選ぶため、すべてのデータから求めた勾配の方向とは、一致しない
- 勾配の方向を足しあげることで、ばらつきが減らせる

# Momentum SGD

- v : 更新量
- γ : 係数
- g : 勾配

\begin{equation}
  v_step = \gamma v_{step-1} + (1 - \gamma)g
\end{equation}

```python

```


# Bouncing ball with the Euler integrator
As an introduction to solving ODEs we are solving the problem of the bouncing ball with the simplest of all integrators: the forward Euler scheme.

We have to solve the *second order ODE* (Newton's equations of motion with constant acceleration)
$$
\frac{d^2 y}{dt^2} = -g.
$$

The Euler scheme for any *first order ODE*

$$
\frac{dy}{dt} = f(y, t)
$$

is
$$
y(t + h) = y(t) + h f(y(t), t).
$$

In order to solve the original 2nd order equation of motion we make use of the fact that one $n$-th order ODE can be written as $n$ coupled first order ODEs, namely

\begin{align}
\frac{dx}{dt} &= v\\
\frac{dv}{dt} &= -g.
\end{align}

Solve each of the first order ODEs with Euler:

\begin{align}
y(t + h) &= y(t) + h v(t)\\
v(t + h) &= v(t) - h g.
\end{align}

```python
import numpy as np
```

## Free fall

Start with free fall

```python
g = -9.81
y = 0.0
v = 0.0

t = 0
dt = 0.01

data = []

while t < 10:
    y = y + v*dt
    v = v + g*dt
    data.append([t, y, v])
    t += dt

data = np.array(data)
```

```python
data.shape
```

    (1001, 3)

```python
data = data.transpose()
data.shape  # t, y, v
```

    (3, 1001)

```python
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
plt.plot(data[0], data[1])
plt.xlabel("time (s)")
plt.ylabel("position (m)")
```

## Bouncing
Add a floor at $y = -5$.

What happens at the floor? – The velocity changes (elastic collision).

```python
g = -9.81
y = 0.0
v = 0.0

t = 0
dt = 0.01

y_floor = -5

data = []

while t < 10:
    y += v*dt
    if y > y_floor:
        v += g*dt
    else:
        v = -v   # bounce off floor
    data.append([t, y, v])
    t += dt

data = np.array(data).transpose()
```

```python
plt.plot(data[0], data[1])
plt.xlabel("time (s)")
plt.ylabel("position (m)")
```

```python

```


# Characterization of Systems in the Time Domain

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Communications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Analysis of a Passive Electrical Network

[Electrical networks](https://en.wikipedia.org/wiki/Electrical_network) composed of linear passive elements, like resistors, capacitors and inductors can be described by linear ordinary differential equations (ODEs) with constant coefficients. Hence, in view of the theory of signals and systems they can be interpreted as linear time-invariant (LTI) systems. The different ways to characterize the properties of an LTI system introduced before are illustrated at the example of a second-order analog [low-pass filter](https://en.wikipedia.org/wiki/Low-pass_filter).

It is assumed that no energy is stored in the capacitor and inductor for $t<0$. It is furthermore assumed that $x(t) = 0$ for $t<0$. Hence $y(t) = 0$ and $\frac{d y(t)}{dt} = 0$ for $t<0$. For illustration, the normalized values $L = 0.5$, $R = 1$, $C = 0.4$ are used for the elements of the electrical network.

### Differential Equation

The differential equation describing the input/output relation of the electrical network is derived by applying [Kirchhoff's circuit laws](https://en.wikipedia.org/wiki/Kirchhoff's_circuit_laws) to the network. This results in the following ODE

\begin{equation}
C L \frac{d^2 y(t)}{dt^2} + C R \frac{d y(t)}{dt} + y(t) = x(t)
\end{equation}

This ODE is defined in `SymPy`

```python
import sympy as sym
sym.init_printing()

t, L, R, C = sym.symbols('t L R C', real=True)
x = sym.Function('x')(t)
y = sym.Function('y')(t)

ode = sym.Eq(L*C*y.diff(t, 2) + R*C*y.diff(t) + y, x)
ode
```

The normalized values of the network elements are stored in a dictionary for later substitution

```python
RLC = {R: 1, L: sym.Rational('.5'), C: sym.Rational('.4')}
RLC
```

### Impulse Response

The passive electrical network and the ODE describing its input/output relation can be interpreted as an LTI system. Hence, the system can be characterized by its [impulse response](impulse_response.ipynb) $h(t)$ which is defined as the output of the system for a Dirac Delta impulse $x(t) = \delta(t)$ at the input. For the given system, the impulse response is calculated by explicit solution of the ODE

```python
solution_h = sym.dsolve(
    ode.subs(x, sym.DiracDelta(t)).subs(y, sym.Function('h')(t)))
solution_h
```

The integration constants $C_1$ and $C_2$ have to be determined from the initial conditions $y(t) = 0$ and $\frac{d y(t)}{dt} = 0$ for $t<0$.

```python
integration_constants = sym.solve((solution_h.rhs.limit(
    t, 0, '-'), solution_h.rhs.diff(t).limit(t, 0, '-')), ['C1', 'C2'])
integration_constants
```

Substitution of the values for the integration constants $C_1$ and $C_2$ into the result from above yields the impulse response of the low-pass

```python
h = solution_h.subs(integration_constants)
h
```

The impulse response is plotted for the values of $R$, $L$ and $C$ given above

```python
sym.plot(h.rhs.subs(RLC), (t, -1, 10), ylabel=r'h(t)');
```

### Step Response

The [step response](step_response.ipynb) is derived by integrating over the impulse response $h(t)$. For ease of illustration this is performed for the specific values of the elements given above

```python
tau = sym.symbols('tau', real=True)

he = sym.integrate(h.rhs.subs(RLC).subs(t, tau), (tau, 0, t))
he
```

Let's plot the step response

```python
sym.plot(he, (t, -1, 10), ylabel=r'$h_\epsilon(t)$');
```

### Transfer Function

For an exponential input signal $x(t) = e^{s t}$, the [transfer function](eigenfunctions.ipynb#Transfer-Function) $H(s)$ represents the complex weight of the exponential output signal $y(t) = H(s) \cdot e^{s t}$. The transfer function is derived by introducing $x(t)$ and $y(t)$ into the ODE and solving for $H(s)$

```python
s = sym.symbols('s')
H = sym.Function('H')(s)

H, = sym.solve(ode.subs(x, sym.exp(s*t)).subs(y, H*sym.exp(s*t)).doit(), H)
H
```

The transfer characteristic of an LTI system for harmonic exponential signals $e^{j \omega t} = \cos(\omega t) + j \sin(\omega t)$ is of special interest for the analysis of electrical circuits. It can be derived from $H(s)$ by substituting the complex frequency $s$ by $s = j \omega$. The resulting transfer function $H(j \omega)$ provides the attenuation/amplification and phase the system adds to an harmonic input signal.

```python
w = sym.symbols('omega', real=True)

Hjw = H.subs(s, sym.I * w)
Hjw
```

The magnitude of the transfer function $|H(j \omega)|$ is plotted for illustration for the specific values of the elements given above

```python
sym.plot(abs(Hjw.subs(RLC)), (w, -10, 10),
         ylabel=r'$|H(j \omega)|$', xlabel=r'$\omega$');
```

It now becomes obvious, that the low frequencies pass through the system and that high frequencies are attenuated. This motivates the term 'low-pass' for such systems.

As alternative to the solution of the ODE, the transfer function $H(s)$ is [computed from the impulse response](eigenfunctions.ipynb#Link-between-Transfer-Function-and-Impulse-Response) and plotted for the specific values of the elements given above

```python
H2 = sym.integrate(h.rhs.subs(RLC)*sym.exp(-s*t), (t, 0, sym.oo), conds='none')
H2.simplify()
```

```python
sym.plot(abs(H2.subs(s, sym.I*w)), (w, -10, 10),
         ylabel=r'$|H(j \omega)|$', xlabel=r'$\omega$');
```

The phase of the transfer function $\varphi(j \omega) = \arg \{ H(j \omega) \}$ provides insight into the phase added to an harmonic signal when passing through the system. It is computed and plotted for the specific values of the elements given above

```python
phi = sym.arg(Hjw)
sym.plot(phi.subs(RLC), (w, -10, 10),
         ylabel=r'$\varphi(j \omega)$', xlabel=r'$\omega$');
```

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Continuous- and Discrete-Time Signals and Systems - Theory and Computational Examples*.


```python
%matplotlib inline
```

```python
import matplotlib.pyplot as plt
from numba import jit
from sympy import integrate, oo, var
from sympy.physics.hydrogen import R_nl
from numerov.cy.core import radial_wf as radial_wf_cy
from numerov.core import radial_wf as radial_wf_py
```

numba.jit can provide significant speed improvements (faster than cython for `radial_wf` and comparable for `radial_integral`).

```python
radial_wf_jit = jit(radial_wf_py)
```

```python
step = 0.0001
n = 10
l = 5
```

```python
offset = 0.002

fig, ax = plt.subplots()

# python
r_py, y_py = radial_wf_py(n, l, step=step)
ax.plot(r_py, y_py + 3*offset, label="py")

# jit
r_jit, y_jit = radial_wf_jit(n, l, step=step)
ax.plot(r_jit, y_jit + 2*offset, label="jit")

# cython
r_cy, y_cy = radial_wf_cy(n, l, step=step)
ax.plot(r_cy, y_cy + offset, label="cy")

# sympy
y_sympy = [R_nl(n, l, r).evalf() for r in r_cy]
ax.plot(r_cy, y_sympy, label="sympy")

ax.legend(loc=0)
plt.show()
```

```python
%timeit radial_wf_py(n, l, step=step)
```

    66.4 ms ± 359 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

```python
%timeit radial_wf_jit(n, l, step=step)
```

    4.28 ms ± 23 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

```python
%timeit radial_wf_cy(n, l, step=step)
```

    15.3 ms ± 1.93 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)

```python
from numerov.cy.core import radial_integral as radial_integral_cy
from numerov.core import radial_integral as radial_integral_py
radial_integral_jit = jit(radial_integral_py)
```

```python
n1, l1 = 14, 1
n2, l2 = 13, 2
```

```python
# python
radial_integral_py(n1, l1, n2 ,l2, step=step)
```

    47.82964249999974

```python
%timeit radial_integral_py(n1, l1, n2 ,l2, step=step)
```

    115 ms ± 7.1 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

```python
# numba.jit
radial_integral_jit(n1, l1, n2 ,l2, step=step)
```

    47.829642499999736

```python
%timeit radial_integral_jit(n1, l1, n2 ,l2, step=step)
```

    3.48 ms ± 10.4 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

```python
# cython
radial_integral_cy(n1, l1, n2 ,l2, step=step)
```

    47.82964250000002

```python
%timeit radial_integral_cy(n1, l1, n2 ,l2, step=step)
```

    3.76 ms ± 7.75 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

```python
# sympy
var("r")
integrate(R_nl(n1, l1, r) * r**3 * R_nl(n2, l2, r), (r, 0, oo)).evalf()
```

$\displaystyle 47.8295909083453$

```python
%timeit integrate(R_nl(n1, l1, r) * r**3 * R_nl(n2, l2, r), (r, 0, oo)).evalf()
```

    158 ms ± 3.22 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

```python

```