# Dimensional Reduction
G. Richards (2016, 2018), based on materials from Ivezic, Connolly, Leighly, and VanderPlas

**Before class starts, please try to do the following:**

> find . -name “sdss_corrected_spectra.py” -print

> ./anaconda/lib/python2.7/site-packages/astroML/datasets/sdss_corrected_spectra.py

> emacs -nw ./anaconda/lib/python2.7/site-packages/astroML/datasets/sdss_corrected_spectra.py

> #DATA_URL = 'http://www.astro.washington.edu/users/vanderplas/spec4000.npz'

> DATA_URL = 'http://staff.washington.edu/jakevdp/spec4000.npz'

Just in case that doesn't work, I've put "spec4000.npz" in PHYS_T480_F18/data.  Copy this to your "astroML_data" directory.

## Curse of Dimensionality

You want to buy a car.  Right now--you don't want to wait.  But you are picky and have certain things that you would like it to have.  Each of those things has a probability between 0 and 1 of being on the the car dealer's lot.  You want a red car which has a probability of being on the lot of $p_{\rm red}$; you want good gas mileage, $p_{\rm gas}$; you want leather seats, $p_{\rm leather}$; and you want a sunroof, $p_{\rm sunroof}$.  The probability that the dealer has a car on the lot that meets all of those requirements is
$$p_{\rm red} \, p_{\rm gas} \, p_{\rm leather} \, p_{\rm sunroof},$$
or $p^n$ where $n$ is the number of features (assuming equal probability for each).

If the probability of each of these is 50%, then the probability of you driving off with your car of choice is only $0.5*0.5*0.5*0.5 = 0.0625$.  Not very good.  Imagine if you also wanted other things.  This is the [Curse of Dimensionality](https://en.wikipedia.org/wiki/Curse_of_dimensionality).

Let's illustrate the curse of dimensionality with two figures from [here.](https://medium.freecodecamp.org/the-curse-of-dimensionality-how-we-can-save-big-data-from-itself-d9fa0f872335)

In the first example we are trying to find which box hold some treasure, which gets harder and harder with more dimensions, despite there just being 5 boxes in each dimension:

In the next example we inscribe a circle in a square.  The area outside of the circle grows larger and larger as the number of dimensions increase:

Mathematically we can describe this as: the more dimensions that your data span, the more points needed to uniformly sample the space.

For $D$ dimensions with coordinates $[-1,1]$, the fraction of points in a unit hypersphere (with radius $r$, as illustrated above) is
$$f_D = \frac{V_D(r)}{(2r)^D} = \frac{\pi^{D/2}}{D2^{D-1}\Gamma(D/2)}$$
which goes to $0$ as $D$ goes to infinity!  Actually, as you can see from the plot below, it is effectively 0 much earlier than that!

```python
# Execute this cell
# from Andy Connolly
%matplotlib inline
import numpy as np
import scipy.special as sp
from matplotlib import pyplot as plt

def unitVolume(dimension, radius=1.):
    return 2*(radius**dimension *np.pi**(dimension/2.))/(dimension*sp.gamma(dimension/2.))

dim = np.linspace(1,100)

#------------------------------------------------------------
# Plot the results
fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(dim,unitVolume(dim)/2.**dim)

ax.set_yscale('log')
ax.set_xlabel('$Dimension$')
ax.set_ylabel('$Volume$')

plt.show()
```

Note that this works in the opposite direction too: let's say you want to find "rare" objects in 10 dimensions, where we'll define rare as <1% of the population.  Then you'll need to accept objects from 63% of the distribution in all 10 dimensions!  So are those really "rare" or are they just a particular 1% of the population?

```python
import numpy as np
p = 10**(np.log10(0.01)/10.0)
print(p)
```

    0.6309573444801932

N.B.  Dimensionality isn't just measuring $D$ parameters for $N$ objects.  It could be a spectrum with $D$ values or an image with $D$ pixels, etc.  In the book the examples used just happen to be spectra of galaxies from the SDSS project.  But we can insert the data of our choice instead.

For example: the SDSS comprises a sample of 357 million sources:
- each source has 448 measured attributes
- selecting just 30 (e.g., magnitude, size..) and normalizing the data range $-1$ to $1$

yields a probability of having one of the 357 million sources reside within a unit hypersphere of 1 in 1.4$\times 10^5$.

## Principal Component Analysis (PCA)

In [Principal Component Analysis (PCA)](https://en.wikipedia.org/wiki/Principal_component_analysis) we seek to take a data set like the one shown below and apply a transform to the data such that the new axes are aligned with the maximal variance of the data.  As can be seen in the Figure, this is basically just the same as doing regression by minimizing the square of the perpendicular distances to the new axes.  Note that we haven't made any changes to the data, we have just defined new axes.

```python
# Execute this cell
# Ivezic, Figure 7.2
# Author: Jake VanderPlas
# License: BSD
#   The figure produced by this code is published in the textbook
#   "Statistics, Data Mining, and Machine Learning in Astronomy" (2013)
#   For more information, see http://astroML.github.com
#   To report a bug or issue, use the following forum:
#    https://groups.google.com/forum/#!forum/astroml-general
%matplotlib inline
import numpy as np
from matplotlib import pyplot as plt
from matplotlib.patches import Ellipse

#------------------------------------------------------------
# Set parameters and draw the random sample
np.random.seed(42)
r = 0.9

sigma1 = 0.25
sigma2 = 0.08
rotation = np.pi / 6
s = np.sin(rotation)
c = np.cos(rotation)

X = np.random.normal(0, [sigma1, sigma2], size=(100, 2)).T
R = np.array([[c, -s],[s, c]])
X = np.dot(R, X) #Same data, now rotated by R matrix.

#------------------------------------------------------------
# Plot the diagram
fig = plt.figure(figsize=(5, 5), facecolor='w')
ax = plt.axes((0, 0, 1, 1), xticks=[], yticks=[], frameon=False)

# draw axes
ax.annotate(r'$x$', (-r, 0), (r, 0),
            ha='center', va='center',
            arrowprops=dict(arrowstyle='<->', color='k', lw=1))
ax.annotate(r'$y$', (0, -r), (0, r),
            ha='center', va='center',
            arrowprops=dict(arrowstyle='<->', color='k', lw=1))

# draw rotated axes
ax.annotate(r'$x^\prime$', (-r * c, -r * s), (r * c, r * s),
            ha='center', va='center',
            arrowprops=dict(color='k', arrowstyle='<->', lw=1))
ax.annotate(r'$y^\prime$', (r * s, -r * c), (-r * s, r * c),
            ha='center', va='center',
            arrowprops=dict(color='k', arrowstyle='<->', lw=1))

# scatter points
ax.scatter(X[0], X[1], s=25, lw=0, c='k', zorder=2)

# draw lines
vnorm = np.array([s, -c])
for v in (X.T):
    d = np.dot(v, vnorm)
    v1 = v - d * vnorm
    ax.plot([v[0], v1[0]], [v[1], v1[1]], '-k')

# draw ellipses
for sigma in (1, 2, 3):
    ax.add_patch(Ellipse((0, 0), 2 * sigma * sigma1, 2 * sigma * sigma2,
                         rotation * 180. / np.pi,
                         ec='k', fc='gray', alpha=0.2, zorder=1))

ax.set_xlim(-1, 1)
ax.set_ylim(-1, 1)

plt.show()
```

Note that the points are correlated along a particular direction which doesn't align with the initial choice of axes.  So, we should rotate our axes to align with this correlation.

We'll choose the rotation to maximize the ability to discriminate between the data points:
*   the first axis, or **principal component**, is direction of maximal variance
*   the second principal component is orthogonal to the first component and maximizes the residual variance
*   ...

PCA is a dimensional reduction process because we can generally account for nearly "all" of the variance in the data set with fewer than the original $K$ dimensions.  See more below.

We start with a data set $\{x_i\}$ which consists of $N$ objects for which we measure $K$ features.  We start by subtracting the mean for each feature in $\{x_i\}$ and write $X$ as a $N\times K$ matrix.

The covariance of this matrix is
$$C_X=\frac{1}{N-1}X^TX.$$

There are off-diagonal terms if there are correlations between the measurements (e.g., maybe two of the features are temperature dependent and the measurements were taken at the same time).

If $R$ is a projection of the data that is aligned with the maximal variance, then we have $Y= X R$ with covariance

$$ C_{Y} = R^T X^T X R = R^T C_X R.$$

$r_1$ is the first principal component of $R$, which can be derived using Langrange multipliers with the following cost function:

$$ \phi(r_1,\lambda_1) = r_1^TC_X r_1 - \lambda_1(r_1^Tr_1-1). $$

If we take derivative of $\phi(r_1,\lambda)$ with respect to $r_1$ and set it to 0, then we have

$$ C_Xr_1 - \lambda_1 r_1 = 0. $$

$\lambda_1$ (the largest eigenvalue of the matrix) is the root of the equation $\det(C_X -
\lambda_1 {\bf I})=0$ for which the eigenvalue is

$$ \lambda_1 =  r_1^T C_X r_1.$$

The columns of the full matrix, $R$ are the eigenvectors (known here as principal components).

The diagonal values of $C_Y$ are the variance contained within each component.

We aren't going to go through the linear algebra more than that here.  But it would be a good group project for someone.  See the end of 7.3.1 starting at the bottom on page 294 or go through [Karen Leighly's PCA lecture notes](http://seminar.ouml.org/lectures/principal-components-analysis/) if you want to walk through the math in more detail.

### Preparing data for PCA

   * Subtract the mean of each dimension (to "center" the data)
   * Divide by the variance in each dimension (to "whiten" the data)
   * (For spectra and images) normalize each row to yield an integral of unity.

Below is a typical call to the PCA algorithm.  Note that this is somewhat backwards.  We are starting with `X` and then we are making it higher dimensional--to create a mock high-$D$ data set.  Then we are applying PCA as a dimensionality reduction technique.

```python
#Example call from 7.3.2
import numpy as np
from sklearn.decomposition import PCA

X = np.random.normal(size=(100,3)) # 100 points in 3D
R = np.random.random((3,10)) # projection matrix
X = np.dot(X,R) # X is now 10-dim, with 3 intrinsic dims
pca = PCA(n_components=4) # n_components can be optionally set
pca.fit(X)
comp = pca.transform(X) # compute the subspace projection of X, 4 eigenvalues for each of the 100 samples
mean = pca.mean_ # length 10 mean of the data
components = pca.components_ # 4x10 matrix of components, multiply each by respective "comp" to reconstruct

#Reconstruction of object1
#Xreconstruct[0] = mean + [components][comp[0]]
```

To illustrate what is happening here is a PCA reconstruction of handwritten "3s" from [Hastie et al.](https://web.stanford.edu/~hastie/ElemStatLearn/) :

[Scikit-Learn's decomposition module](http://scikit-learn.org/stable/modules/classes.html#module-sklearn.decomposition) has a number of [PCA type implementations](http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html#sklearn.decomposition.PCA).
Let's work through an example using spectra of galaxies take during the Sloan Digital Sky Survey.  In this sample there are 4000 spectra with flux measurements in 1000 bins.  15 example spectra are shown below and our example will use half of the spectra chosen at random.

```python
%matplotlib inline
# Example from Andy Connolly
# See Ivezic, Figure 7.4
import numpy as np
from matplotlib import pyplot as plt

from sklearn.decomposition import PCA
from sklearn.decomposition import RandomizedPCA

from astroML.datasets import sdss_corrected_spectra
from astroML.decorators import pickle_results

#------------------------------------------------------------
# Download data
data = sdss_corrected_spectra.fetch_sdss_corrected_spectra()
spectra = sdss_corrected_spectra.reconstruct_spectra(data)
wavelengths = sdss_corrected_spectra.compute_wavelengths(data)
print(len(spectra), len(wavelengths))

#----------------------------------------------------------------------
# Compute PCA
np.random.seed(500)
nrows = 2000 # We'll just look at 2000 random spectra
n_components = 5 # Do the fit with 5 components, which is the mean plus 4
ind = np.random.randint(spectra.shape[0], size=nrows)

spec_mean = spectra[ind].mean(0) # Compute the mean spectrum, which is the first component
# spec_mean = spectra[:50].mean(0)

# use Randomized PCA for speed
#pca = RandomizedPCA(n_components - 1)
pca = PCA(n_components - 1,svd_solver='randomized')
pca.fit(spectra[ind])
pca_comp = np.vstack([spec_mean,pca.components_]) #Add the mean to the components
evals = pca.explained_variance_ratio_
print(evals)
```

    downloading PCA-processed SDSS spectra from http://staff.washington.edu/jakevdp/spec4000.npz to /home/pranphy/astroML_data
    Downloading http://staff.washington.edu/jakevdp/spec4000.npz
    [=========================================]  27.15Mb / 27.15Mb
    4000 1000
    [0.889316   0.06058301 0.02481432 0.01012148]

Now let's plot the components.  See also Ivezic, Figure 7.4.  The left hand panels are just the first 5 spectra for comparison with the first 5 PCA components, which are shown on the right.  They are ordered by the size of their eigenvalues.

```python
#Make plots
fig = plt.figure(figsize=(10, 8))
fig.subplots_adjust(left=0.05, right=0.95, wspace=0.05,
                    bottom=0.1, top=0.95, hspace=0.05)

titles = 'PCA components'

for j in range(n_components):

    # plot the components
    ax = fig.add_subplot(n_components, 2, 2*j+2)
    ax.yaxis.set_major_formatter(plt.NullFormatter())
    ax.xaxis.set_major_locator(plt.MultipleLocator(1000))
    if j < n_components - 1:
        ax.xaxis.set_major_formatter(plt.NullFormatter())
    else:
        ax.set_xlabel('wavelength (Angstroms)')
    ax.plot(wavelengths, pca_comp[j], '-k', lw=1)

    # plot zero line
    xlim = [3000, 7999]
    ax.plot(xlim, [0, 0], '-', c='gray', lw=1)
    ax.set_xlim(xlim)

    # adjust y limits
    ylim = plt.ylim()
    dy = 0.05 * (ylim[1] - ylim[0])
    ax.set_ylim(ylim[0] - dy, ylim[1] + 4 * dy)

    # plot the first j spectra
    ax2 = fig.add_subplot(n_components, 2, 2*j+1)
    ax2.yaxis.set_major_formatter(plt.NullFormatter())
    ax2.xaxis.set_major_locator(plt.MultipleLocator(1000))
    if j < n_components - 1:
        ax2.xaxis.set_major_formatter(plt.NullFormatter())
    else:
        ax2.set_xlabel('wavelength (Angstroms)')
    ax2.plot(wavelengths, spectra[j], '-k', lw=1)

    # plot zero line
    ax2.plot(xlim, [0, 0], '-', c='gray', lw=1)
    ax2.set_xlim(xlim)

    if j == 0:
        ax.set_title(titles, fontsize='medium')
    if j == 0:
        label = 'mean'
    else:
        label = 'component %i' % j

    # adjust y limits
    ylim = plt.ylim()
    dy = 0.05 * (ylim[1] - ylim[0])
    ax2.set_ylim(ylim[0] - dy, ylim[1] + 4 * dy)

    ax.text(0.02, 0.95, label, transform=ax.transAxes,
            ha='left', va='top', bbox=dict(ec='w', fc='w'),
            fontsize='small')

plt.show()
```

Now let's make "scree" plots.  These plots tell us how much of the variance is explained as a function of the each eigenvector.  Our plot won't look much like Ivezic, Figure 7.5, so I've shown it below to explain where "scree" comes from.

```python
# Execute this cell
import numpy as np
from matplotlib import pyplot as plt

#----------------------------------------------------------------------
# Plot the results
fig = plt.figure(figsize=(10, 5))

ax = fig.add_subplot(121)
ax.plot(np.arange(n_components-1), evals)
ax.set_xlabel("eigenvalue number")
ax.set_ylabel("eigenvalue ")

ax = fig.add_subplot(122)
ax.plot(np.arange(n_components-1), evals.cumsum())
ax.set_xlabel("eigenvalue number")
ax.set_ylabel("cumulative eigenvalue")

plt.show()
```

How much of the variance is explained by the first two components?  How about all of the components?

```python
print("The first component explains {:.3f} of the variance in the data.".format(pca.explained_variance_ratio_[0]))
print("The second component explains {:.3f} of the variance in the data.".format(pca.explained_variance_ratio_[0]))
print("All components explain {:.3f} of the variance in the data.".format(sum(pca.explained_variance_ratio_)))
```

    The first component explains 0.889 of the variance in the data.
    The second component explains 0.889 of the variance in the data.
    All components explain 0.996 of the variance in the data.

This is why PCA enables dimensionality reduction.

How many components would we need to explain 99.5% of the variance?

```python
for num_feats in np.arange(1,20, dtype = int):
    pca = PCA(n_components=num_feats)
    pca.fit(spectra[ind])
    if (sum(pca.explained_variance_ratio_[:num_feats])>0.995):
        break
print("{:d} features are needed to explain 99.5% of the variance".format(num_feats))
```

    8 features are needed vo explain 99.5% of the variance

Note that we would need 1000 components to encode *all* of the variance.

## Interpreting the PCA

- The output eigenvectors are ordered by their associated eigenvalues
- The eigenvalues reflect the variance within each eigenvector
- The sum of the eigenvalues is total variance of the system
- Projection of each spectrum onto the first few eigenspectra is a compression of the data

Once we have the eigenvectors, we can try to reconstruct an observed spectrum, ${x}(k)$, in the eigenvector basis, ${e}_i(k)$, as

$$ \begin{equation}
  {x}_i(k) = {\mu}(k) + \sum_j^R \theta_{ij} {e}_j(k).
\end{equation}
$$

That would give a full (perfect) reconstruction of the data since it uses all of the eigenvectors.  But if we truncate (i.e., $r<R$), then we will have reduced the dimensionality while still reconstructing the data with relatively little loss of information.

For example, we started with 4000x1000 floating point numbers.  If we can explain nearly all of the variance with 8 eigenvectors, then we have reduced the problem to 4000x8+8x1000 floating point numbers!

Execute the next cell to see how the reconstruction improves by adding more components.

```python
# Execute this cell
import numpy as np
from matplotlib import pyplot as plt

from sklearn.decomposition import PCA

from astroML.datasets import sdss_corrected_spectra
from astroML.decorators import pickle_results

#------------------------------------------------------------
# Download data
data = sdss_corrected_spectra.fetch_sdss_corrected_spectra()
spectra = sdss_corrected_spectra.reconstruct_spectra(data)
wavelengths = sdss_corrected_spectra.compute_wavelengths(data)

#------------------------------------------------------------
# Compute PCA components

# Eigenvalues can be computed using PCA as in the commented code below:

#from sklearn.decomposition import PCA
#pca = PCA()
#pca.fit(spectra)
#evals = pca.explained_variance_ratio_
#evals_cs = evals.cumsum()

#  because the spectra have been reconstructed from masked values, this
#  is not exactly correct in this case: we'll use the values computed
#  in the file compute_sdss_pca.py
evals = data['evals'] ** 2
evals_cs = evals.cumsum()
evals_cs /= evals_cs[-1]
evecs = data['evecs']
spec_mean = spectra.mean(0)

#------------------------------------------------------------
# Find the coefficients of a particular spectrum
spec = spectra[1]
coeff = np.dot(evecs, spec - spec_mean)

#------------------------------------------------------------
# Plot the sequence of reconstructions
fig = plt.figure(figsize=(8, 8))
fig.subplots_adjust(hspace=0)

for i, n in enumerate([0, 4, 8, 20]):
    ax = fig.add_subplot(411 + i)
    ax.plot(wavelengths, spec, '-', c='gray')
    ax.plot(wavelengths, spec_mean + np.dot(coeff[:n], evecs[:n]), '-k')

    if i < 3:
        ax.xaxis.set_major_formatter(plt.NullFormatter())

    ax.set_ylim(-2, 21)
    ax.set_ylabel('flux')

    if n == 0:
        text = "mean"
    elif n == 1:
        text = "mean + 1 component\n"
        text += r"$(\sigma^2_{tot} = %.2f)$" % evals_cs[n - 1]
    else:
        text = "mean + %i components\n" % n
        text += r"$(\sigma^2_{tot} = %.2f)$" % evals_cs[n - 1]

    ax.text(0.01, 0.95, text, ha='left', va='top', transform=ax.transAxes)

fig.axes[-1].set_xlabel(r'${\rm wavelength\ (\AA)}$')
plt.show()
```

### Caveats I

PCA is a linear process, whereas the variations in the data may not be.  So it may not always be appropriate to use and/or may require a relatively large number of components to fully describe any non-linearity.

Note also that PCA can be very impractical for large data sets which exceed the memory per core as the computational requirement goes as $\mathscr{O}(D^3$) and the memory requirement goes as $\mathscr{O}(2D^2)$.

### Missing Data
We have assumed so far that there is no missing data (e.g., bad pixels in the spectrum, etc.).  But often the data set is incomplete.  Since PCA encodes the flux correlation with wavelength (or whatever parameters are in your data set), we can actually use it to determine missing values.

An example is shown below.  Here, black are the observed spectra.  Gray are the regions where we have no data.  Blue is the PCA reconstruction, including the regions where there are no data.  Awesome, isn't it?

```python
# Execute this cell
%matplotlib inline
import numpy as np
from matplotlib import pyplot as plt
from matplotlib import ticker

from astroML.datasets import fetch_sdss_corrected_spectra
from astroML.datasets import sdss_corrected_spectra

#------------------------------------------------------------
# Get spectra and eigenvectors used to reconstruct them
data = fetch_sdss_corrected_spectra()
spec = sdss_corrected_spectra.reconstruct_spectra(data)
lam = sdss_corrected_spectra.compute_wavelengths(data)
evecs = data['evecs']
mu = data['mu']
norms = data['norms']
mask = data['mask']

#------------------------------------------------------------
# plot the results
i_plot = ((lam > 5750) & (lam < 6350))
lam = lam[i_plot]

specnums = [20, 8, 9]
subplots = [311, 312, 313]

fig = plt.figure(figsize=(8, 10))
fig.subplots_adjust(hspace=0)

for subplot, i in zip(subplots, specnums):
    ax = fig.add_subplot(subplot)

    # compute eigen-coefficients
    spec_i_centered = spec[i] / norms[i] - mu
    coeffs = np.dot(spec_i_centered, evecs.T)

    # blank out masked regions
    spec_i = spec[i]
    mask_i = mask[i]
    spec_i[mask_i] = np.nan

    # plot the raw masked spectrum
    ax.plot(lam, spec_i[i_plot], '-', color='k', lw=2,
            label='True spectrum')

    # plot two levels of reconstruction
    for nev in [10]:
        if nev == 0:
            label = 'mean'
        else:
            label = 'N EV=%i' % nev
        spec_i_recons = norms[i] * (mu + np.dot(coeffs[:nev], evecs[:nev]))
        ax.plot(lam, spec_i_recons[i_plot], label=label)

    # plot shaded background in masked region
    ylim = ax.get_ylim()
    mask_shade = ylim[0] + mask[i][i_plot].astype(float) * ylim[1]
    plt.fill(np.concatenate([lam[:1], lam, lam[-1:]]),
             np.concatenate([[ylim[0]], mask_shade, [ylim[0]]]),
             lw=0, fc='k', alpha=0.2)

    ax.set_xlim(lam[0], lam[-1])
    ax.set_ylim(ylim)
    ax.yaxis.set_major_formatter(ticker.NullFormatter())

    if subplot == 311:
        ax.legend(loc=1, prop=dict(size=14))

    ax.set_xlabel('$\lambda\ (\AA)$')
    ax.set_ylabel('normalized flux')

plt.show()
```

The example that we have been using above is "spectral" PCA.  Some examples from the literature include:
- [Francis et al. 1992](http://articles.adsabs.harvard.edu/cgi-bin/nph-iarticle_query?1992ApJ...398..476F&data_type=PDF_HIGH&whole_paper=YES&type=PRINTER&filetype=.pdf)
- [Connolly et al. 1995](http://articles.adsabs.harvard.edu/cgi-bin/nph-iarticle_query?1995AJ....110.1071C&data_type=PDF_HIGH&whole_paper=YES&type=PRINTER&filetype=.pdf)
- [Yip et al. 2004](http://iopscience.iop.org/article/10.1086/425626/meta;jsessionid=31BB5F11B85D2BF4180834DC71BA0B85.c3.iopscience.cld.iop.org)

One can also do PCA on features that aren't ordered (as they were for the spectra).  E.g., if you have $D$ different parameters measured for your objects.  The classic example in astronomy is
[Boroson & Green 1992](http://articles.adsabs.harvard.edu/cgi-bin/nph-iarticle_query?1992ApJS...80..109B&data_type=PDF_HIGH&whole_paper=YES&type=PRINTER&filetype=.pdf).

### Caveats II

One of the things that I don't like about PCA is that the eigenvectors are defined relative to the mean.  So they can be positive or negative and they often don't look anything like the original data itself.  Whereas it is often the case that you might expect that the components would look like, well, the physical components.  For example, quasars are fundamentally galaxies.  So, part of their flux comes from the galaxy that they live in.  But PCA doesn't return any component that looks like a typical galaxy.

## Non-negative Matrix Factorization (NMF)

This is where [Non-negative Matrix Factorizaiton (NMF)](https://en.wikipedia.org/wiki/Non-negative_matrix_factorization) comes in.  Here we are treating the data as a linear sum of positive-definite components.

NMF assumes  any data matrix can be factored into two matrices, $W$ and $Y$, with
$$\begin{equation}
X=W Y,
\end{equation}
$$
where both $W$ and $Y$ are nonnegative.

So, $WY$ is an approximation of $X$. Minimizing the reconstruction error $|| (X - W Y)^2 ||$,
nonnegative bases can be derived through an iterative process.

Note, however, that the iterative process does not guarantee nonlocal minima (like $K$-means and EM), but using
random initialization and cross-validation can be used to find the global minimum.

An example from the literature is [Allen et al. 2008](http://arxiv.org/abs/0810.4231)

In Scikit-Learn the [NMF implementation](http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.NMF.html) looks like:

```python
# Execute this cell
import numpy as np
from sklearn.decomposition import NMF
X = np.random.random((100,10)) # 100 points in 10-D
nmf = NMF(n_components=3)
nmf.fit(X)
proj = nmf.transform(X) # project to 3 dimension
comp = nmf.components_ # 3x10 array of components
err = nmf.reconstruction_err_ # how well 3 components capture the data
```

An example (and comparison to PCA) is given below.

```python
# Execute the next 2 cells
# Example from Figure 7.4
# Author: Jake VanderPlas
# License: BSD
%matplotlib inline

import numpy as np
from matplotlib import pyplot as plt

from sklearn.decomposition import NMF
from sklearn.decomposition import RandomizedPCA
from sklearn.decomposition import PCA

from astroML.datasets import sdss_corrected_spectra
from astroML.decorators import pickle_results

#------------------------------------------------------------
# Download data
data = sdss_corrected_spectra.fetch_sdss_corrected_spectra()
spectra = sdss_corrected_spectra.reconstruct_spectra(data)
wavelengths = sdss_corrected_spectra.compute_wavelengths(data)
```

```python
#----------------------------------------------------------------------
# Compute PCA, and NMF components
def compute_PCA_NMF(n_components=5):
    spec_mean = spectra.mean(0)

    # PCA: use randomized PCA for speed
    #pca = RandomizedPCA(n_components - 1)
    pca = PCA(n_components - 1,svd_solver='randomized')
    pca.fit(spectra)
    pca_comp = np.vstack([spec_mean, pca.components_])

    # NMF requires all elements of the input to be greater than zero
    spectra[spectra < 0] = 0
    nmf = NMF(n_components)
    nmf.fit(spectra)
    nmf_comp = nmf.components_

    return pca_comp, nmf_comp

n_components = 5
decompositions = compute_PCA_NMF(n_components)

#----------------------------------------------------------------------
# Plot the results
fig = plt.figure(figsize=(10, 10))
fig.subplots_adjust(left=0.05, right=0.95, wspace=0.05,
                    bottom=0.1, top=0.95, hspace=0.05)

titles = ['PCA components', 'NMF components']

for i, comp in enumerate(decompositions):
    for j in range(n_components):
        ax = fig.add_subplot(n_components, 3, 3 * j + 1 + i)

        ax.yaxis.set_major_formatter(plt.NullFormatter())
        ax.xaxis.set_major_locator(plt.MultipleLocator(1000))
        if j < n_components - 1:
            ax.xaxis.set_major_formatter(plt.NullFormatter())
        else:
            ax.set_xlabel('wavelength (Angstroms)')

        ax.plot(wavelengths, comp[j], '-k', lw=1)

        # plot zero line
        xlim = [3000, 7999]
        ax.plot(xlim, [0, 0], '-', c='gray', lw=1)
        ax.set_xlim(xlim)

        if j == 0:
            ax.set_title(titles[i])

        if titles[i].startswith('PCA') or titles[i].startswith('ICA'):
            if j == 0:
                label = 'mean'
            else:
                label = 'component %i' % j
        else:
            label = 'component %i' % (j + 1)

        ax.text(0.03, 0.94, label, transform=ax.transAxes,
                ha='left', va='top')

        for l in ax.get_xticklines() + ax.get_yticklines():
            l.set_markersize(2)

        # adjust y limits
        ylim = plt.ylim()
        dy = 0.05 * (ylim[1] - ylim[0])
        ax.set_ylim(ylim[0] - dy, ylim[1] + 4 * dy)

plt.show()
```

## Independent Component Analysis (ICA)

For data where the components are statistically independent (or nearly so) [Independent Component Analysis (ICA)](https://en.wikipedia.org/wiki/Independent_component_analysis) has become a popular method for separating mixed components.  The classical example is the so-called "cocktail party" problem.  This is illustrated in the following figure from Hastie, Tibshirani, and Friedman (Figure 14.27 on page 497 in my copy, so they have clearly added some stuff!).  Think of the "source signals" as two voices at a party.  You are trying to concentrate on just one voice.  What you hear is something like the "measured signals" pattern.  You could run the data through PCA and that would do an excellent job of reconstructing the signal with reduced dimensionality, but it wouldn't actually isolate the different physical components (bottom-left panel).  ICA on the other hand can (bottom-right panel).

.](../images/HastieFigure14_37.png)

[Hastie et al.](https://web.stanford.edu/~hastie/ElemStatLearn/): "ICA applied to multivariate data looks for a sequence of orthogonal projections such that the projected data look as far from Gaussian as possible. With pre-whitened data, this amounts to looking for
components that are as independent as possible."

In short you want to find components that are maximally non-Gaussian since the sum of 2 random variables will be more Gaussian than either of the components (remember the Central Limit Theorem).  Hastie et al. illustrate this as follows:

ICA is a good choice for a complex system with relatively indepent components.  For example a galaxy is roughly a linear combination of cool stars and hot stars, and a quasar is just a galaxy with others component from an accretion disk and emission line regions.  Ideally we want "eigenvectors" that are aligned with those physical traits/regions as opposed to mathematical constructs.

The basic call to the [FastICA algoirthm](http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.FastICA.html) in Scikit-Learn looks like:

```python
# Execute this cell
import numpy as np
from sklearn.decomposition import FastICA
X = np.random.normal(size=(100,2)) # 100 objects in 2D
R = np.random.random((2,5)) # mixing matrix
X = np.dot(X,R) # Simulation of a 5D data space
ica = FastICA(2) # Now reproject to 2-D
ica.fit(X)
proj = ica.transform(X) # 100x2 projection of the data
comp = ica.components_ # 2x5 matrix of independent components
## sources = ica.sources_ # 100x2 matrix of sources
```

Execute the next 2 cells to produce a plot showing the ICA components.

```python
%matplotlib inline
#Example from Andy Connolly
import numpy as np
from matplotlib import pyplot as plt

from sklearn.decomposition import FastICA

from astroML.datasets import sdss_corrected_spectra
from astroML.decorators import pickle_results

#------------------------------------------------------------
# Download data
data = sdss_corrected_spectra.fetch_sdss_corrected_spectra()
spectra = sdss_corrected_spectra.reconstruct_spectra(data)
wavelengths = sdss_corrected_spectra.compute_wavelengths(data)

#----------------------------------------------------------------------
# Compute PCA
np.random.seed(500)
nrows = 500
n_components = 5
ind = np.random.randint(spectra.shape[0], size=nrows)

spec_mean = spectra[ind].mean(0)
# spec_mean = spectra[:50].mean(0)

ica = FastICA(n_components - 1)
ica.fit(spectra[ind])
ica_comp = np.vstack([spec_mean,ica.components_]) #Add the mean to the components
```

```python
#Make plots
fig = plt.figure(figsize=(10, 8))
fig.subplots_adjust(left=0.05, right=0.95, wspace=0.05,
                    bottom=0.1, top=0.95, hspace=0.05)

titles = 'ICA components'

for j in range(n_components):

    # plot the components
    ax = fig.add_subplot(n_components, 2, 2*j+2)
    ax.yaxis.set_major_formatter(plt.NullFormatter())
    ax.xaxis.set_major_locator(plt.MultipleLocator(1000))
    if j < n_components - 1:
        ax.xaxis.set_major_formatter(plt.NullFormatter())
    else:
        ax.set_xlabel(r'wavelength ${\rm (\AA)}$')
    ax.plot(wavelengths, ica_comp[j], '-k', lw=1)

    # plot zero line
    xlim = [3000, 7999]
    ax.plot(xlim, [0, 0], '-', c='gray', lw=1)
    ax.set_xlim(xlim)

    # adjust y limits
    ylim = plt.ylim()
    dy = 0.05 * (ylim[1] - ylim[0])
    ax.set_ylim(ylim[0] - dy, ylim[1] + 4 * dy)

    # plot the first j spectra
    ax2 = fig.add_subplot(n_components, 2, 2*j+1)
    ax2.yaxis.set_major_formatter(plt.NullFormatter())
    ax2.xaxis.set_major_locator(plt.MultipleLocator(1000))
    if j < n_components - 1:
        ax2.xaxis.set_major_formatter(plt.NullFormatter())
    else:
        ax2.set_xlabel(r'wavelength ${\rm (\AA)}$')
    ax2.plot(wavelengths, spectra[j], '-k', lw=1)

    # plot zero line
    ax2.plot(xlim, [0, 0], '-', c='gray', lw=1)
    ax2.set_xlim(xlim)

    if j == 0:
        ax.set_title(titles, fontsize='medium')

    if j == 0:
        label = 'mean'
    else:
        label = 'component %i' % j

    # adjust y limits
    ylim = plt.ylim()
    dy = 0.05 * (ylim[1] - ylim[0])
    ax2.set_ylim(ylim[0] - dy, ylim[1] + 4 * dy)

    ax.text(0.02, 0.95, label, transform=ax.transAxes,
            ha='left', va='top', bbox=dict(ec='w', fc='w'),
            fontsize='small')

plt.show()
```

As with PCA and NMF, we can similarly do a reconstruction:

```python
# Execute this cell
#------------------------------------------------------------
# Find the coefficients of a particular spectrum
spec = spectra[1]
evecs = data['evecs']
coeff = np.dot(evecs, spec - spec_mean)

#------------------------------------------------------------
# Plot the sequence of reconstructions
fig = plt.figure(figsize=(8, 8))
fig.subplots_adjust(hspace=0)

for i, n in enumerate([0, 2, 4, 8]):
    ax = fig.add_subplot(411 + i)
    ax.plot(wavelengths, spec, '-', c='gray')
    ax.plot(wavelengths, spec_mean + np.dot(coeff[:n], evecs[:n]), '-k')

    if i < 3:
        ax.xaxis.set_major_formatter(plt.NullFormatter())

    ax.set_ylim(-2, 21)
    ax.set_ylabel('flux')

    if n == 0:
        text = "mean"
    elif n == 1:
        text = "mean + 1 component\n"
        #text += r"$(\sigma^2_{tot} = %.2f)$" % evals_cs[n - 1]
    else:
        text = "mean + %i components\n" % n
        #text += r"$(\sigma^2_{tot} = %.2f)$" % evals_cs[n - 1]

    ax.text(0.01, 0.95, text, ha='left', va='top', transform=ax.transAxes)

fig.axes[-1].set_xlabel(r'${\rm wavelength\ (\AA)}$')
plt.show()
```

Ivezic, Figure 7.4 compares the components found by the PCA, ICA, and NMF algorithms.  Their differences and similarities are quite interesting.

If you think that I was pulling your leg about the cocktail problem, try it yourself!
Load the code instead of running it and see what effect changing some things has.

```python
# %load ../code/plot_ica_blind_source_separation.py
"""
=====================================
Blind source separation using FastICA
=====================================

An example of estimating sources from noisy data.

:ref:`ICA` is used to estimate sources given noisy measurements.
Imagine 3 instruments playing simultaneously and 3 microphones
recording the mixed signals. ICA is used to recover the sources
ie. what is played by each instrument. Importantly, PCA fails
at recovering our `instruments` since the related signals reflect
non-Gaussian processes.

"""
print(__doc__)

import numpy as np
import matplotlib.pyplot as plt
from scipy import signal

from sklearn.decomposition import FastICA, PCA

###############################################################################
# Generate sample data
np.random.seed(0)
n_samples = 2000
time = np.linspace(0, 8, n_samples)

s1 = np.sin(2 * time)  # Signal 1 : sinusoidal signal
s2 = np.sign(np.sin(3 * time))  # Signal 2 : square signal
s3 = signal.sawtooth(2 * np.pi * time)  # Signal 3: saw tooth signal

#
s3[time>4] = 0
S = np.c_[s1, s2, s3]
S += 0.2 * np.random.normal(size=S.shape)  # Add noise

S /= S.std(axis=0)  # Standardize data
# Mix data
A = np.array([[1, 1, 1], [0.5, 2, 1.0], [1.5, 1.0, 2.0]])  # Mixing matrix
X = np.dot(S, A.T)  # Generate observations

# Compute ICA
ica = FastICA(n_components=3)
S_ = ica.fit_transform(X)  # Reconstruct signals
A_ = ica.mixing_  # Get estimated mixing matrix

# We can `prove` that the ICA model applies by reverting the unmixing.
assert np.allclose(X, np.dot(S_, A_.T) + ica.mean_)

# For comparison, compute PCA
pca = PCA(n_components=3)
H = pca.fit_transform(X)  # Reconstruct signals based on orthogonal components

###############################################################################
# Plot results

plt.figure(figsize=(8,8))

models = [X, S, S_, H]
names = ['Observations (mixed signal)',
         'True Sources',
         'ICA recovered signals',
         'PCA recovered signals']
colors = ['red', 'steelblue', 'orange']

for ii, (model, name) in enumerate(zip(models, names), 1):
    plt.subplot(4, 1, ii)
    plt.title(name)
    for sig, color in zip(model.T, colors):
        plt.plot(sig, color=color)

plt.subplots_adjust(0.09, 0.04, 0.94, 0.94, 0.26, 0.46)
plt.show()

```

Let's revisit the digits sample and see what PCA, NMF, and ICA do for it.

```python
## Execute this cell to load the digits sample
%matplotlib inline
import numpy as np
from sklearn.datasets import load_digits
from matplotlib import pyplot as plt
digits = load_digits()
grid_data = np.reshape(digits.data[0], (8,8)) #reshape to 8x8
plt.imshow(grid_data, interpolation = "nearest", cmap = "bone_r")
print(grid_data)
X = digits.data
y = digits.target
```

Do the PCA transform, projecting to 2 dimensions and plot the results.

```python
# PCA
from sklearn.decomposition import PCA
pca = PCA(n_components = 2)
pca.fit_transform(X)
X_reduced = pca.transform(X)

plt.scatter(X_reduced[:,0], X_reduced[:,1], c=y, cmap="nipy_spectral", edgecolor="None")
plt.colorbar()
```

Similarly for NMF and ICA

```python

from sklearn.decomposition import NMF
nmf = NMF(n_components=2)
nmf.fit(X)
X_reduced = nmf.transform(X)

plt.scatter(X_reduced[:,0], X_reduced[:,1], c=y, cmap="nipy_spectral", edgecolor="None")
plt.colorbar()
```

```python
# ICA
from sklearn.decomposition import FastICA
ica = FastICA(n_components=2)
ica.fit(X)
X_reduced = ica.transform(X)

plt.scatter(X_reduced[:,0],X_reduced[:,1], c=y, cmap="nipy_spectral", edgecolor="None")
plt.colorbar()
```

Take a second to think about what ICA is doing.  What if you had digits from digital clocks instead of handwritten?

I wasn't going to introduce [Neural Networks](https://en.wikipedia.org/wiki/Artificial_neural_network) yet, but it is worth noting that Scikit-Learn's [`Bernoulli Restricted Boltzman Machine (RBM)`](http://scikit-learn.org/stable/modules/generated/sklearn.neural_network.BernoulliRBM.html) is discussed in the [(unsupervised) neural network](http://scikit-learn.org/stable/modules/neural_networks_unsupervised.html) part of the User's Guide and is relevant here as the data input must be either binary or values between 0 and 1, which is the case that we have here.

We could think about doing dimensional reduction of the digits data set in another way.  There are 64 pixels in each of our images.  Presumably all of them aren't equally useful.  Let's figure out exactly which pixels are the most relevant.  We'll use Scikit-Learn's [`RandomForestRegressor`](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html).  We won't get to regression until next week, but you don't need to understand the algorithm to do this, just look at the inputs and outputs.  Which pixels are the most important?  As a bonus see if you can plot digit images with those pixels highlighted.

```python
from sklearn.ensemble import RandomForestRegressor
RFreg = RandomForestRegressor()# Complete or leave blank as you see fit
RFreg.fit(X,y)# Do Fitting
importances = RFreg.feature_importances_# Determine "importances"
pixelorder = np.argsort(importances)[::-1] #Rank importances (highest to lowest)
print(pixelorder)

plt.figure()
plt.imshow(np.reshape(importances,(8,8)),interpolation="nearest")
plt.show()
```

```python

```


```python
%matplotlib inline
import sys
sys.path.append('../../')
from ef.config.visualizer import Visualizer3d
from ef.config.efconf import EfConf
from ef.config.components import *
```

Dynamics of a single particle in free space is the simplest kind of simulation. It allows to check basic functionality of the program, and, besides, it is a good way to become familiar with how the program works.

In absence of external forces, a body moves in straight line with constant velocity

\begin{align}
& \textbf{r}(t) = \textbf{r}_0 + \textbf{v}_0 t
\\
& \textbf{v}(t) = \textbf{v}_0
\end{align}

To get some sense of scales (see below), suppose the particle is an electron that has just passed an 1 keV accelerating potential difference. It's mass `m` and charge `q` are `q = 4.8e-10 [cgs], m = 9.1e-28 [g]`. Since it's energy is nonrelativistic, it's possible to calculate it's speed simply as $v = \sqrt{ 2 E / m } = 1.808e+09 ~ [cm/s]$. To cover a `10 [cm]` distance with such speed, it will take the electron `t = 5.530e-09 [s] ~ 6 [ns]`.

```python
from math import *

m = 9.8e-28
q = 4.8e-10
print( "q = {:.3e} [cgs]".format( q ) )
print( "m = {:.3e} [g]".format( m ) )

ev_to_cgs = 1.60218e-12
E = 1000 * ev_to_cgs
v = sqrt( 2 * E / m )
z = 10
t = z / v
print( "E = {:.3e} [eV] = {:.3e} [erg]".format( E / ev_to_cgs, E ) )
print( "v = {:.3e} [cm/s]; p = {:.3e} [g * cm/s]".format( v, v * m ) )
print( "z = {:.3e} [cm]".format( z ) )
print( "t = {:.3e} [s]".format( t ) )
```

To perform a simulation, it's necessary to prepare a config file.
First, it's necessary to set a total time of the simulation and a time step.
Let's use the estimates above as guiding values and set the total time to `6e-9 [s]`.
We use `1000` time steps, so that `time_step_size = 6.0e-12 [s]`.
Besides, it's necessary to decide at which time steps the state of the whole simulation should be saved to the disk.
To save each step, `time_save_step` is set equal to `time_step_size`: `time_save_step = 6.0e-12`.

```python
#tmp_conf.visualize_all( v )

single_particle_in_free_space = EfConf()
vis = Visualizer3d()

single_particle_in_free_space.time_grid = TimeGrid(
    total = 6.0e-9,
    step = 6.0e-12,
    save_step = 6.0e-12
)

print( single_particle_in_free_space.export_to_string() )
```

Next we need to define size of a computational volume.
Domain size is set to 15 [cm] along the z-axis and 5 [cm] along the x- and y-axes.
The parameters for PIC-mesh are also defined in this section.
Since we are going to use noninteracting particle model instead of PIC, they are not used and can be set arbitrary.

```python
single_particle_in_free_space.spatial_mesh = SpatialMesh(
    size = ( 5.0, 5.0, 15.0 ),
    step = ( 0.5, 0.5, 1.5 )
)

print( single_particle_in_free_space.export_to_string() )
```

Next, a particle source.
We need a single particle at startup ( `initial_number_of_particles = 1` ), approximately 1 [mm] away from
coordinate axis origin at the bottom-left-near corner of the domain (`box_x_left = 0.10` etc).
Charge and mass are set to those of the electron.
Momentum corresponding to 1 keV energy is `1.772e-18 [g * cm / s]`.
Such value is set for momentum along the z-axis.
Momenta along the x- and y-axes could be set to zero; instead they are chosen 3 times smaller than the z-axis momentum
(domain size along the x and y is 3 times smaller than along the z).
To prevent any variation in momentum, the `temperature` is set to 0.
Notice a source is given a descriptive: `emit_single_particle` in this case.

```python
# todo: add method to add single source

single_particle_in_free_space.sources = [
    ParticleSource(
        name = "emit_single_particle",
        initial_particles = 1,
        particles_to_generate_each_step = 0,
        #shape = Box( origin = (0.1, 0.1, 0.1), size = ( 0.01, 0.01, 0.01 ) ),
        shape = Box( origin = (0.1, 0.1, 0.1), size = ( -0.01, 0.01, 0.01 ) ), # hack left > right error
        momentum = ( 0.6e-18, 0.6e-18, 1.77e-18 ),
        temperature = 0.0,
        charge = 4.8e-10,
        mass = 9.8e-28
    )
]

print( single_particle_in_free_space.export_to_string() )
```

Next section is boundary conditions.
In a free space, potentials on each boundary should be equal.
It is possible to simply set them to zero.

```python
single_particle_in_free_space.boundary_conditions = BoundaryConditions( potential = 0 )

print( single_particle_in_free_space.export_to_string() )
```

Particle interaction model allows to choose between noninteracting particles and particles interacting by
particle-in-cell method. Since there is only one particle, there is no need to use PIC, and noninteracting mode is enough. Moreover, PIC will lead to wrong results (because particle creates electric field that acts back on the particle).

```python
single_particle_in_free_space.particle_interaction_model = ParticleInteractionModel(
    model = "noninteracting"
)

print( single_particle_in_free_space.export_to_string() )
```

The last step is to specify pattern for output file names.
They will be of the form `single_particle_free_space_0001000.h5`, where `0001000` is a time step number.

```python
single_particle_in_free_space.output_file = OutputFile(
    prefix = "single_particle_free_space_",
    suffix = ".h5"
)

print( single_particle_in_free_space.export_to_string() )
```

To start the simulation, the config should be provided as an argument to the `ef.out`.

```python
from ef.util.runner import EfRunner

runner = EfRunner( conf = single_particle_in_free_space, ef_command="python3 ../../main.py" )
runner.run()
```

After the simulation finishes, a bunch of `*.h5` files will emerge in the directory.
To open them and glance over the results, an [Hdfview](https://support.hdfgroup.org/products/java/hdfview/) is a convenient tool.
In this simulation, two places are of interest: `current time` and particle position
and momentum at this time moment:

<p align="center">

</p>

Each output `h5`-file corresponds to a different time step.
If we were able to extract time, position and momentum of the particle from each file, we could have obtained numerical trajectory, which could be compared to the analytical one.

Data extraction and plotting are accomplished by the accompanying [`plot.py`](https://github.com/epicf/ef/blob/master/examples/single_particle_in_free_space/plot.py) python script.
It's `main` function does exactly what is described above: extracts numerical trajectory from the output
files, evaluates analytical trajectory at the extracted time points and plots both numerical and analytical
trajectories on the same axes.

```python
def main():
    num = extract_num_trajectory_from_out_files()
    an = eval_an_trajectory_at_num_time_points( num )
    plot_trajectories( num , an )
```

It uses [h5py](http://www.h5py.org/) library to read from h5 files, [numpy](http://www.numpy.org/) to simplify working with arrays and [matplotlib](https://matplotlib.org/) to perform plotting of the trajectory.

```python
import os, glob
import h5py
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
```

Let's examine the function to obtain the numerical trajectory:

```python
def extract_num_trajectory_from_out_files():
    out_files = find_necessary_out_files()                           # (*1)

    num_trajectory = []
    for f in out_files:
        num_trajectory.append( extract_time_pos_mom( f ) )           # (*2)

    num_trajectory = remove_empty_and_sort_by_time( num_trajectory ) # (*3)
    num_trajectory = np.array( num_trajectory,                       # (*4)
                               dtype=[('t','float'),
                                      ('x','float'), ('y','float'), ('z','float'),
                                      ('px','float'), ('py','float'),('pz','float')])
    return( num_trajectory )
```

(\*1): To extract data, first we find all *.h5 files in the current directory.
(\*2): Then the script iterates over them and extracts a relevant data from each one.
(\*3): In the end of this process, `num_trajectory` array contains positions and velocities of particle at different
time steps. In is convenient to sort it over time values.
(\*4): After the previous step the numerical trajectory is in a format of a list of tuples:
`[ (t, x, y, z, px, py, pz), ..... ]` . It is convenient to convert it into one of the `np.array` types to simplify further manipulations with it.

For analytical trajectory we need the initial position and momentum.
They can be extracted from the first h5-file:

```python
def get_mass_and_initial_pos_and_mom():
    initial_out_file = "single_particle_free_space_0000000.h5"
    h5 = h5py.File( initial_out_file, mode="r")
    m = h5["/Particle_sources/emit_single_particle"].attrs["mass"][0]
    x0 = h5["/Particle_sources/emit_single_particle/position_x"][0]
    y0 = h5["/Particle_sources/emit_single_particle/position_y"][0]
    z0 = h5["/Particle_sources/emit_single_particle/position_z"][0]
    px0 = h5["/Particle_sources/emit_single_particle/momentum_x"][0]
    py0 = h5["/Particle_sources/emit_single_particle/momentum_y"][0]
    pz0 = h5["/Particle_sources/emit_single_particle/momentum_z"][0]
    h5.close()
    return( m, x0, y0, z0, px0, py0, pz0 )
```

With this function, the analytical trajectory can be computed at the same
time points as the numerical one:

```python
def eval_an_trajectory_at_num_time_points( num_trajectory ):
    global particle_mass
    particle_mass, x0, y0, z0, px0, py0, pz0 =  get_mass_and_initial_pos_and_mom()
    #
    an_trajectory = np.empty_like( num_trajectory )
    for i, t in enumerate( num_trajectory['t'] ):
        x, y, z = coords( particle_mass, t, x0, y0, z0, px0, py0, pz0 )
        px, py, pz = momenta( t, px0, py0, pz0 )
        an_trajectory[i] = ( t, x, y ,z, px, py, pz )
    #
    return( an_trajectory )
```

Finally, both numerical and analytical trajectories are plotted in 3d and side views.
Kinetic energies are also compared.

```python
def plot_trajectories( num , an ):
    plot_3d( num, an )
    plot_2d( num, an )
    plot_kin_en( num , an )
```

This results in the following figures:

    _insert figs_

It can be seen, that energy is conserved during the simulation:

_insert fig_

```python

```


# Detector simulation
In this notebook we load a track dataset generated by `edep-sim` and we calculate the ADC counts corresponding to each pixel. The result is exported to a HDF5 file.

```python
# This is need so you can import larndsim without doing python setup.py install
import os,sys,inspect
currentdir = os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
parentdir = os.path.dirname(currentdir)
sys.path.insert(0,parentdir)
```

```python
from math import ceil
from time import time

import matplotlib.pyplot as plt
from matplotlib import cm, colors
import mpl_toolkits.mplot3d.art3d as art3d

import numpy as np
import cupy as cp
import fire
import h5py

from numba import cuda
from numba.cuda.random import create_xoroshiro128p_states

import matplotlib as mpl
mpl.rcParams['figure.dpi'] = 100

from tqdm.notebook import tqdm
```

```python
from larndsim import consts
import importlib
importlib.reload(consts)
consts.load_detector_properties("../larndsim/detector_properties/module0.yaml","../larndsim/pixel_layouts/multi_tile_layout-2.2.16.yaml")

from larndsim import quenching, drifting, pixels_from_track, fee, detsim
```

### Dataset import
First of all we load the `edep-sim` output. For this sample we need to invert $z$ and $y$ axes.

```python
unique_pix = cp.empty((0,2))
pixels_signals = cp.zeros((len(unique_pix), len(consts.time_ticks)*3))
```

    /global/homes/t/tairaeli/.conda/envs/myenv/lib/python3.7/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.
      and should_run_async(code)

```python
with h5py.File('module0_corsika.h5', 'r') as f:
    tracks = np.array(f['segments'])

x_start = np.copy(tracks['x_start'] )
x_end = np.copy(tracks['x_end'])
x = np.copy(tracks['x'])

tracks['x_start'] = np.copy(tracks['z_start'])
tracks['x_end'] = np.copy(tracks['z_end'])
tracks['x'] = np.copy(tracks['z'])

tracks['z_start'] = x_start
tracks['z_end'] = x_end
tracks['z'] = x

selected_tracks = tracks[:100]
```

### Quenching and drifting
We calculate the number of electrons after recombination (`quenching` module) and the position and number of electrons after drifting (`drifting` module).

```python
TPB = 256
BPG = ceil(selected_tracks.shape[0] / TPB)
import importlib
importlib.reload(drifting)
quenching.quench[BPG,TPB](selected_tracks, consts.birks)
drifting.drift[BPG,TPB](selected_tracks)
```

We find the pixels intersected by the projection of the tracks on the anode plane using the Bresenham's algorithm. We also take into account the neighboring pixels, due to the transverse diffusion of the charges.

```python
# Here we build a map between tracks and event IDs
unique_eventIDs = cp.unique(selected_tracks['eventID'])
event_id_map = cp.searchsorted(unique_eventIDs,cp.asarray(selected_tracks['eventID']))

longest_pix = ceil(max(selected_tracks["dx"])/consts.pixel_pitch)
max_radius = ceil(max(selected_tracks["tran_diff"])*5/consts.pixel_pitch)

MAX_PIXELS = int((longest_pix*4+6)*max_radius*1.5)
MAX_ACTIVE_PIXELS = int(longest_pix*1.5)

active_pixels = cp.full((selected_tracks.shape[0], MAX_ACTIVE_PIXELS, 2), -1, dtype=np.int32)
neighboring_pixels = cp.full((selected_tracks.shape[0], MAX_PIXELS, 2), -1, dtype=np.int32)
n_pixels_list = cp.zeros(shape=(selected_tracks.shape[0]))
TPB = 128
BPG = ceil(selected_tracks.shape[0] / TPB)
pixels_from_track.get_pixels[BPG,TPB](selected_tracks,
                                      active_pixels,
                                      neighboring_pixels,
                                      n_pixels_list,
                                      max_radius+1)
```

    /global/homes/t/tairaeli/.conda/envs/myenv/lib/python3.7/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.
      and should_run_async(code)

```python
def cupy_unique_axis0(array):
    # axis is still not supported for cupy.unique, this
    # is a workaround
    if len(array.shape) != 2:
        raise ValueError("Input array must be 2D.")
    sortarr     = array[cp.lexsort(array.T[::-1])]
    mask        = cp.empty(array.shape[0], dtype=cp.bool_)
    mask[0]     = True
    mask[1:]    = cp.any(sortarr[1:] != sortarr[:-1], axis=1)
    return sortarr[mask]

shapes = neighboring_pixels.shape
joined = neighboring_pixels.reshape(shapes[0]*shapes[1],2)
this_unique_pix = cupy_unique_axis0(joined)
this_unique_pix = this_unique_pix[(this_unique_pix[:,0] != -1) & (this_unique_pix[:,1] != -1),:]
```

    /global/homes/t/tairaeli/.conda/envs/myenv/lib/python3.7/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.
      and should_run_async(code)

```python
unique_pix = cp.concatenate((unique_pix, this_unique_pix),axis=0)
```

### Charge distribution calculation
Here we calculate the current induced by each track on the pixels, taking into account longitudinal and transverse diffusion. The track segment is parametrized as:
\begin{align}
x'(r') &=x_s + \frac{\Delta x}{\Delta r}r'\\
y'(r') &=y_s + \frac{\Delta y}{\Delta r}r'\\
z'(r') &=z_s + \frac{\Delta z}{\Delta r}r',
\end{align}
where $\Delta r$ is the segment length. Here we assume $z$ as the drift direction.
The diffused charge distribution is calculated with the following integral:
\begin{equation}
\rho(x,y,z) = \frac{Q}{\sqrt{(2\pi)^3}\sigma_x\sigma_y\sigma_z\Delta r}\exp\left[-\frac{(x-x_s)^2}{2\sigma_x^2}-\frac{(y-y_s)^2}{2\sigma_y^2}-\frac{(z-z_s)^2}{2\sigma_z^2}\right]\int^{r'=\Delta r}_{r'=0}dr'\exp[-(ar'^2+br')],
\end{equation}
where
\begin{align}
a &= \left[\left(\frac{\Delta x}{\Delta r}\right)^2\frac{1}{2\sigma_x^2} + \left(\frac{\Delta y}{\Delta r}\right)^2\frac{1}{2\sigma_y^2} + \left(\frac{\Delta z}{\Delta r}\right)^2\frac{1}{2\sigma_z^2} \right]\\
b &= -\left[\frac{(x-x_s)}{\sigma_x^2}\frac{\Delta x}{\Delta r}+
\frac{(y-y_s)}{\sigma_y^2}\frac{\Delta y}{\Delta r}+
\frac{(z-z_s)}{\sigma_z^2}\frac{\Delta z}{\Delta r}\right].
\end{align}

The simmetry of the transverse diffusion along the track allows to take a slice on the $xy$ plane and solve the integral once at a fixed $z$ coordinate (e.g. at $z_{m} = (z_s+z_e)/2$) and re-use it at other $z$ coordinates away from the endpoints (where $\rho(x,y,z)$ varies along $z$ so must be calculated at each $z$).

```python
max_length = cp.array([0])
track_starts = cp.empty(selected_tracks.shape[0])
threadsperblock = 128
blockspergrid = ceil(selected_tracks.shape[0] / threadsperblock)
detsim.time_intervals[blockspergrid,threadsperblock](track_starts, max_length,  event_id_map, selected_tracks)
```

```python
signals = cp.zeros((selected_tracks.shape[0],
                    neighboring_pixels.shape[1],
                    cp.asnumpy(max_length)[0]), dtype=np.float32)
threadsperblock = (1,1,64)
importlib.reload(detsim)
blockspergrid_x = ceil(signals.shape[0] / threadsperblock[0])
blockspergrid_y = ceil(signals.shape[1] / threadsperblock[1])
blockspergrid_z = ceil(signals.shape[2] / threadsperblock[2])
blockspergrid = (blockspergrid_x, blockspergrid_y, blockspergrid_z)
detsim.tracks_current[blockspergrid,threadsperblock](signals,
                                                     neighboring_pixels,
                                                     selected_tracks)
```

    /global/homes/t/tairaeli/.conda/envs/myenv/lib/python3.7/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.
      and should_run_async(code)

```python
pixel_index_map = cp.full((selected_tracks.shape[0], neighboring_pixels.shape[1]), -1)
try:
    compare = neighboring_pixels[..., np.newaxis, :] == unique_pix
    indices = cp.where(cp.logical_and(compare[..., 0], compare[..., 1]))
except cp.cuda.memory.OutOfMemoryError:
    print("out of memory")
pixel_index_map[indices[0], indices[1]] = indices[2]
```

    /global/homes/t/tairaeli/.conda/envs/myenv/lib/python3.7/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.
      and should_run_async(code)

```python
blockspergrid_x = ceil(signals.shape[0] / threadsperblock[0])
blockspergrid_y = ceil(signals.shape[1] / threadsperblock[1])
blockspergrid_z = ceil(signals.shape[2] / threadsperblock[2])
blockspergrid = (blockspergrid_x, blockspergrid_y, blockspergrid_z)
this_pixels_signals = cp.zeros((len(this_unique_pix), len(consts.time_ticks)*3))
pixels_signals = cp.concatenate((pixels_signals, this_pixels_signals), axis=0)
detsim.sum_pixel_signals[blockspergrid,threadsperblock](pixels_signals,
                                                        signals,
                                                        track_starts,
                                                        pixel_index_map)
currents = np.sum(pixels_signals,axis=1)*consts.t_sampling/consts.e_charge
```

```python
track = selected_tracks[0]
pixel_plane = track['pixel_plane']
z_anode = consts.tpc_borders[pixel_plane][2][0]
z = (track['z_start']+track['z_end'])/2
drift_distance = abs(z - z_anode)
drift_start = abs(min(track["z_start"],track["z_end"]) - z_anode)
drift_end = abs(max(track["z_start"],track["z_end"]) - z_anode)
drift_time = drift_distance / consts.vdrift
drift_start_time = drift_start / consts.vdrift
drift_end_time = drift_end / consts.vdrift
```

    /global/homes/t/tairaeli/.conda/envs/myenv/lib/python3.7/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.
      and should_run_async(code)

### 3D event display

```python
cmap = cm.Spectral_r
norm = colors.Normalize(vmin=0, vmax=256)
m = cm.ScalarMappable(norm=norm, cmap=cmap)

cmap = cm.viridis
norm_curr = colors.LogNorm(vmin=min(currents[currents>0]), vmax=max(currents))
m_curr = cm.ScalarMappable(norm=norm_curr, cmap=cmap)
fig = plt.figure(figsize=(7,7))
ax = fig.add_subplot(111, projection='3d')

for it,t in enumerate(selected_tracks):
    if it == 0:
        ax.plot((t["x_start"], t["x_end"]),
            (t["z_start"], t["z_end"]),
            (t["y_start"], t["y_end"]),
            c='r',
            lw=1,
            alpha=1,
            zorder=10,
            label='Geant4 detector segment')
    else:
        ax.plot((t["x_start"], t["x_end"]),
                (t["z_start"], t["z_end"]),
                (t["y_start"], t["y_end"]),
                c='r',
                lw=1,
                alpha=1,
                zorder=9999)

    ax.plot((t["x_start"], t["x_end"]),
            (consts.tpc_borders[t['pixel_plane']][2][0], consts.tpc_borders[t['pixel_plane']][2][0]),
            (t["y_start"], t["y_end"]),
            c='r',
            lw=1,
            ls=':',
            alpha=1,
            zorder=9999)

for ip, p in enumerate(unique_pix):
    x_rect, y_rect = detsim.get_pixel_coordinates(p.get())
    pixel_plane = int(p[0] // consts.n_pixels[0])
    row = int(pixel_plane // 4)
    column = int(pixel_plane % 4)
    if currents[ip] > 0:
        rect = plt.Rectangle((x_rect, y_rect),
                             consts.pixel_pitch, consts.pixel_pitch,
                             linewidth=0.1, fc=m_curr.to_rgba(currents[ip].get()),
                             edgecolor='gray')
        ax.add_patch(rect)
        art3d.pathpatch_2d_to_3d(rect, z=consts.tpc_borders[pixel_plane][2][0], zdir="y")

anode1 = plt.Rectangle((consts.tpc_borders[0][0][0], consts.tpc_borders[0][1][0]),
                        consts.tpc_borders[0][0][1]-consts.tpc_borders[0][0][0],
                        consts.tpc_borders[0][1][1]-consts.tpc_borders[0][1][0],
                       linewidth=1, fc='none',
                       edgecolor='gray', label=('Pixel' if ip == 5 else ''))
ax.add_patch(anode1)
art3d.pathpatch_2d_to_3d(anode1, z=consts.tpc_borders[0][2][0], zdir="y")

anode2 = plt.Rectangle((consts.tpc_borders[0][0][0], consts.tpc_borders[0][1][0]),
                        consts.tpc_borders[0][0][1]-consts.tpc_borders[0][0][0],
                        consts.tpc_borders[0][1][1]-consts.tpc_borders[0][1][0],
                       linewidth=1, fc='none',
                       edgecolor='gray', label=('Pixel' if ip == 5 else ''))
ax.add_patch(anode2)
art3d.pathpatch_2d_to_3d(anode2, z=consts.tpc_borders[1][2][0], zdir="y")

cathode = plt.Rectangle((consts.tpc_borders[0][0][0], consts.tpc_borders[0][1][0]),
                        consts.tpc_borders[0][0][1]-consts.tpc_borders[0][0][0],
                        consts.tpc_borders[0][1][1]-consts.tpc_borders[0][1][0],
                        linewidth=1, fc='gray', alpha=0.5,
                        edgecolor='gray', label=('Pixel' if ip == 5 else ''))
ax.add_patch(cathode)
art3d.pathpatch_2d_to_3d(cathode, z=0, zdir="y")

ax.plot((consts.tpc_borders[0][0][0],consts.tpc_borders[0][0][0]),(consts.tpc_borders[0][2][0],consts.tpc_borders[1][2][0]),
        (consts.tpc_borders[0][1][0],consts.tpc_borders[0][1][0]), lw=1,color='gray')

ax.plot((consts.tpc_borders[0][0][0],consts.tpc_borders[0][0][0]),(consts.tpc_borders[0][2][0],consts.tpc_borders[1][2][0]),
        (consts.tpc_borders[0][1][1],consts.tpc_borders[0][1][1]), lw=1,color='gray')

ax.plot((consts.tpc_borders[0][0][1],consts.tpc_borders[0][0][1]),(consts.tpc_borders[0][2][0],consts.tpc_borders[1][2][0]),
        (consts.tpc_borders[0][1][0],consts.tpc_borders[0][1][0]), lw=1,color='gray')

ax.plot((consts.tpc_borders[0][0][1],consts.tpc_borders[0][0][1]),(consts.tpc_borders[0][2][0],consts.tpc_borders[1][2][0]),
        (consts.tpc_borders[0][1][1],consts.tpc_borders[0][1][1]), lw=1,color='gray')

# ax.set_ylim(consts.module_borders[pixel_plane][2][0],50)
ax.set_xlim(consts.tpc_borders[0][0][0],consts.tpc_borders[1][0][1])
ax.set_ylim(consts.tpc_borders[0][2][0],consts.tpc_borders[1][2][0])
ax.set_zlim(consts.tpc_borders[0][1][0],consts.tpc_borders[1][1][1])

ax.set_box_aspect((2,2,4))
ax.grid(False)
ax.xaxis.set_major_locator(plt.MaxNLocator(3))
ax.yaxis.set_major_locator(plt.MaxNLocator(3))
# ax.set_axis_off()
ax.w_xaxis.set_pane_color((1.0, 1.0, 1.0, 1.0))
ax.w_yaxis.set_pane_color((1.0, 1.0, 1.0, 1.0))
ax.w_zaxis.set_pane_color((1.0, 1.0, 1.0, 1.0))

ax.view_init(azim=20)
# def rotate(angle):
#     ax.view_init(azim=angle)

# from matplotlib import animation

# rot_animation = animation.FuncAnimation(fig, rotate, frames=np.arange(0,362,2),interval=50)

ax.set_ylabel("z [cm]")
ax.set_xlabel("x [cm]")
ax.set_zlabel("y [cm]")
_ = plt.colorbar(m_curr,fraction=0.035, pad=0.05,label='Induced current integral [# electrons]')

```

```python
# rot_animation.save('animation.gif', writer='imagemagick', fps=10, bitrate=10, dpi=100)
```

    /global/homes/t/tairaeli/.conda/envs/myenv/lib/python3.7/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.
      and should_run_async(code)

### Electronics response and digitization
Here we simulate the electronics response (the self-triggering cycle) and the signal digitization.

```python
time_ticks = cp.linspace(0, len(unique_eventIDs)*consts.time_interval[1]*2, pixels_signals.shape[1]+1)
integral_list = cp.zeros((pixels_signals.shape[0], fee.MAX_ADC_VALUES))
adc_ticks_list = cp.zeros((pixels_signals.shape[0], fee.MAX_ADC_VALUES))
TPB = 128
BPG = ceil(pixels_signals.shape[0] / TPB)

rng_states = create_xoroshiro128p_states(TPB * BPG, seed=0)
fee.get_adc_values[BPG,TPB](pixels_signals,
                            time_ticks,
                            integral_list,
                            adc_ticks_list,
                            0,
                            rng_states)
adc_list = fee.digitize(integral_list)
```

### 2D event display with induced current and ADC counts

```python
fig,ax = plt.subplots(1,2,figsize=(6,3.8))
importlib.reload(detsim)
for ip, p in enumerate(unique_pix):
    x_rect, y_rect = detsim.get_pixel_coordinates(p.get())
    pixel_plane = int(p[0] // consts.n_pixels[0])
    c = currents[ip].get()
    if c >= 1:
        rect = plt.Rectangle((x_rect, y_rect),
                             consts.pixel_pitch, consts.pixel_pitch,
                             linewidth=0.2, fc=m_curr.to_rgba(c),
                             edgecolor='grey')
        ax[0].add_patch(rect)

    a = adc_list[ip][adc_list[ip]>fee.digitize(0)]
    if len(a):
        rect = plt.Rectangle((x_rect, y_rect),
                             consts.pixel_pitch, consts.pixel_pitch,
                             linewidth=0.2, fc=m.to_rgba(np.sum(a.get())),
                             edgecolor='grey')
        ax[1].add_patch(rect)

for it,t in enumerate(selected_tracks):
    ax[0].plot((t["x_start"], t["x_end"]),
            (t["y_start"], t["y_end"]),
            c='r',
            lw=1.25,
            ls=':',
            alpha=1,
            zorder=10)
    ax[1].plot((t["x_start"], t["x_end"]),
            (t["y_start"], t["y_end"]),
            c='r',
            lw=0,
            ls=':',
            alpha=1,
            zorder=10)
    ax[0].scatter((t["x_start"], t["x_end"]),
                  (t["y_start"], t["y_end"]),
                  c='r', s=1, zorder=99999)
#     ax[1].scatter((t["x_start"], t["x_end"]),
#                   (t["y_start"], t["y_end"]),
#                   c='r', s=1, zorder=99999)

ax[0].set_aspect("equal")
ax[1].set_aspect("equal")
ax[0].set_xlabel("x [cm]")
ax[1].set_xlabel("x [cm]")
ax[0].set_ylabel("y [cm]")
from mpl_toolkits.axes_grid1.axes_divider import make_axes_locatable

divider0 = make_axes_locatable(ax[1])
cax0 = divider0.append_axes("right", size="7%", pad=0.07)
fig.colorbar(m, ax=ax[1], cax=cax0, label='ADC counts sum')

divider1 = make_axes_locatable(ax[0])
cax1 = divider1.append_axes("right", size="7%", pad=0.07)
fig.colorbar(m_curr, ax=ax[0], cax=cax1, label='Induced current integral [# electrons]')

plt.subplots_adjust(hspace=0.5)
fig.savefig("currentadc.pdf")
```

### Export result
As a last step we backtrack the ADC counts to the Geant4 tracks and we export the result in a HDF5 file.

```python
MAX_TRACKS_PER_PIXEL = 5

# Mapping between unique pixel array and track array index
track_pixel_map = cp.full((unique_pix.shape[0], MAX_TRACKS_PER_PIXEL), -1)
TPB = 32
BPG = ceil(unique_pix.shape[0] / TPB)
detsim.get_track_pixel_map[BPG, TPB](track_pixel_map, unique_pix, neighboring_pixels)

# Here we backtrack the ADC counts to the Geant4 tracks
TPB = 128
BPG = ceil(adc_list.shape[0] / TPB)
backtracked_id = cp.full((adc_list.shape[0], adc_list.shape[1], MAX_TRACKS_PER_PIXEL), -1)

detsim.backtrack_adcs[BPG,TPB](selected_tracks,
                               adc_list,
                               adc_ticks_list,
                               track_pixel_map,
                               event_id_map,
                               unique_eventIDs,
                               backtracked_id,
                               0)
```

    /global/homes/t/tairaeli/.conda/envs/myenv/lib/python3.7/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.
      and should_run_async(code)

```python
!rm test.h5
pc = fee.export_to_hdf5(adc_list.get(),
                        adc_ticks_list.get(),
                        unique_pix.get(),
                        backtracked_id.get(),
                        "test.h5")
```

    rm: cannot remove 'test.h5': No such file or directory

    /global/homes/t/tairaeli/.conda/envs/myenv/lib/python3.7/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.
      and should_run_async(code)
    Writing to HDF5...: 100%|██████████| 2170/2170 [00:00<00:00, 49468.93it/s]

```python

```


```javascript
%%javascript
$('#appmode-leave').hide();
$('#copy-binder-link').hide();
$('#visit-repo-link').hide();
```

```python
import ipywidgets as ipw
import json
import random
import time
import pandas as pd
import os
import webbrowser
import math
from IPython.display import display, Markdown

# set kinetic parameters
with open("rate_parameters.json") as infile:
    jsdata = json.load(infile)

params = jsdata["equi2"]
```

Copyright **Jacob Martin and Paolo Raiteri**, January 2021

## Equilibrium problem \#2
The water auto-protolysis
\begin{equation}
\mathrm{H_2O \rightleftharpoons H^+ + OH^-}
\end{equation}
is one of the best studied reactions and it is known that neutral pH means different concentrations of [H$^+$] and [OH$^-$] at different temperature.
Use the slide bar below to obtain a data set for the water self-dissociation constant (the data have some noise) and use the van't Hoff equation to extract the enthalpy and entropy of the reaction.

As you probably noticed, the fit was far from perfect.
This was due to the fact that the van't Hoff equation is just a first order approximations of the true temperature dependence of most equilibrium constants.

Try to use a second order expansion of that equation to fit your data:
\begin{equation}
\ln K_{eq} = a + \dfrac{b}{T} + \dfrac{c}{T^2}
\end{equation}

Using the differential van't Hoff equation
\begin{equation}
\frac{\mathrm{d} \ln K_{eq}}{\mathrm{d}\ 1/T} = - \dfrac{\Delta H}{R}
\end{equation}

it's easy to demonstrate that the temperature dependent enthalpy of the reaction is

\begin{equation}
\Delta H(T) = -R \bigg(b+\frac{2c}{T}\bigg)
\end{equation}

and therefore the temperature dependent extropy of the reaction is

\begin{equation}
\Delta S(T) = R \bigg(a-\frac{c}{T^2}\bigg)
\end{equation}

What are the enthalpy and entropy of the reaction at 353 K?

Would a third order polynomial function fit the data better?

### Tips:
- Because the generalised van't Hoff equation cannot be linearised, you have to perform  non-linear regression in your spreadsheet

### Instructions:

- Use the slide bar below to select temperature at which you perform the virtual experiment,
- Click `Perform measurement` to run the virtual experiment and obtain the result of the experiment,
- Click `Download CSV` to export the complete data set for all the experiments as a CSV file.

```python
# define path to results.csv file
respath = os.path.join(os.getcwd(),"results.csv")

# delete existing result file and setup rng
if os.path.exists(respath):
    os.remove(respath)

#random.seed(params["error"].get("seed", 0))
t = int( time.time() * 1000.0 )
random.seed( ((t & 0xff000000) >> 24) +
             ((t & 0x00ff0000) >>  8) +
             ((t & 0x0000ff00) <<  8) +
             ((t & 0x000000ff) << 24)   )

class system:
    def __init__(self, vol=0, conc=0, press=0):
        self.vol = vol
        self.conc = conc
        self.press = press

class data:
    def __init__(self, start=-1, error=0, label='none', units='pure', value=0,
                minval=-1, maxval=3):
        self.start = start
        self.minval = minval
        self.maxval = maxval
        self.error = error
        self.label = label
        self.units = units
        self.value = value

# Experiment setup (+ hidden paramters)
system = system()
def initialiseExperiment():
    global n
    global system
    global columns_list
    global scatter

    scatter = 0.01

    n = []
    columns_list = []

    n.append(len(args)) # number of input adjustable parameters
    n.append(len(result)) # number of results for the experiment

    for i in range(0, n[0]):
        columns_list.append(f"{args[i].label} [{args[i].units}]")
    for i in range(0, n[1]):
        columns_list.append(f"{result[i].label} [{result[i].units}]")

    # Random initial concentration
    system.conc = random.random()

```

```python
# Adjustable input parameters
def initialiseVariables():
    global logScale
    logScale = False
    global args
    args = []
    args.append(
        data(
            label = "Temperature",
            minval = 273,
            maxval = 373,
            start = 298.,
            units = "K",
            value = 0.
        )
    )

# Results
def initialiseResults():
    global result
    result = []
    result.append(
        data(
            label = "pK$_w$",
            start = 0.,
            error = random.random() / 100.,
            units = "pure"
        )
    )

def measure():
    T = args[0].value.value
    lnK = params["a"] + params["b"] / T + params["c"] / T /T
    res = lnK
    return res

initialiseVariables()

```

```python
out_P = ipw.Output()
out_L = ipw.Output()

with out_L:
    display(Markdown("[Download CSV](results.csv)"))

def calc(btn):
    out_P.clear_output()

    # Measurement result
    result[0].value = measure()

    # Random error
    result[0].error = result[0].value * scatter * (0.5 - random.random()) * 2

    # Output result
    out_R[0].value = f"{result[0].value + result[0].error:.3e}"

    # Read previous lines
    res = pd.read_csv(respath)

    var_list = []
    for i in range(0, n[0]):
        var_list.append(args[i].value.value)
    for i in range(0, n[1]):
        var_list.append(result[i].value + result[i].error)

    # Append result
    res.loc[len(res)] = var_list
    res.to_csv(respath, index=False)
    with out_P:
        display(res.tail(50))

def reset(btn):
    if os.path.exists(respath):
        os.remove(respath)

    initialiseResults()
    initialiseExperiment()

    res = pd.DataFrame(columns=columns_list)
    res.to_csv(respath, index=False)
    with out_P:
        out_P.clear_output()
        display(res.tail(50))

# interactive buttons ---
btn_reset = ipw.Button(description="Restart Laboratory", layout=ipw.Layout(width="150px"))
btn_reset.on_click(reset)

btn_calc = ipw.Button(description="Perform measurement", layout=ipw.Layout(width="150px"))
btn_calc.on_click(calc)
# ---

reset(btn_reset)

rows = []
for i in range(0, n[0]):
    if logScale:
        args[i].value = ipw.FloatLogSlider(value=args[i].start, min=args[i].minval, max=args[i].maxval)
    else:
        args[i].value = ipw.FloatSlider(value=args[i].start, min=args[i].minval, max=args[i].maxval)

    rows.append(ipw.HBox([ipw.Label(value=f"{args[i].label} [{args[i].units}]:",
                                    layout=ipw.Layout(width="250px")),
                          args[i].value]))

out_R = []
for i in range(0, n[1]):
    out_R.append(ipw.Label(value=""))
    rows.append(ipw.HBox([ipw.Label(value=f"Measured {result[i].label} [{result[i].units}]:",
                                    layout=ipw.Layout(width="250px")),
                            out_R[i]]))

rows.append(ipw.HBox([btn_reset, btn_calc, out_L]))
rows.append(ipw.HBox([out_P]))

ipw.VBox(rows)

```

```python

```

```python

```