# Introduction to zfit

In this notebook, we will have a walk through the main components of zfit and their features. Especially the extensive model building part will be discussed separately.
zfit consists of 5 mostly independent parts. Other libraries can rely on this parts to do plotting or statistical inference, such as hepstats does. Therefore we will discuss two libraries in this tutorial: zfit to build models, data and a loss, minimize it and get a fit result and hepstats, to use the loss we built here and do inference.

## Data

This component in general plays a minor role in zfit: it is mostly to provide a unified interface for data.

Preprocessing is therefore not part of zfit and should be done beforehand. Python offers many great possibilities to do so (e.g. Pandas).

zfit `Data` can load data from various sources, most notably from Numpy, Pandas DataFrame, TensorFlow Tensor and ROOT (using uproot). It is also possible, for convenience, to convert it directly `to_pandas`. The constructors are named `from_numpy`, `from_root` etc.

```python
import zfit
from zfit import z
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
```

A `Data` needs not only the data itself but also the observables: the human readable string identifiers of the axes (corresponding to "columns" of a Pandas DataFrame). It is convenient to define the `Space` not only with the observable but also with a limit: this can directly be re-used as the normalization range in the PDF.

First, let's define our observables

```python
obs = zfit.Space('obs1', (-5, 10))
```

This `Space` has limits. Next to the effect of handling the observables, we can also play with the limits: multiple `Spaces` can be added to provide disconnected ranges. More importantly, `Space` offers functionality:
- limit1d: return the lower and upper limit in the 1 dimensional case (raises an error otherwise)
- rect_limits: return the n dimensional limits
- area(): calculate the area (e.g. distance between upper and lower)
- inside(): return a boolean Tensor corresponding to whether the value is _inside_ the `Space`
- filter(): filter the input values to only return the one inside

```python
size_normal = 10000
data_normal_np = np.random.normal(size=size_normal, scale=2)

data_normal = zfit.Data.from_numpy(obs=obs, array=data_normal_np)
```

The main functionality is
- nevents: attribute that returns the number of events in the object
- data_range: a `Space` that defines the limits of the data; if outside, the data will be cut
- n_obs: defines the number of dimensions in the dataset
- with_obs: returns a subset of the dataset with only the given obs
- weights: event based weights

Furthermore, `value` returns a Tensor with shape `(nevents, n_obs)`.

To retrieve values, in general `z.unstack_x(data)` should be used; this returns a single Tensor with shape (nevents) or a list of tensors if `n_obs` is larger then 1.

```python
print(f"We have {data_normal.nevents} events in our dataset with the minimum of {np.min(data_normal.unstack_x())}")  # remember! The obs cut out some of the data
```

    We have 9950 events in our dataset with the minimum of -4.979805501079585

```python
data_normal.n_obs
```

    1

## Model

Building models is by far the largest part of zfit. We will therefore cover an essential part, the possibility to build custom models, in an extra chapter. Let's start out with the idea that you define your parameters and your observable space; the latter is the expected input data.

There are two types of models in zfit:
- functions, which are rather simple and "underdeveloped"; their usage is often not required.
- PDF that are function which are normalized (over a specified range); this is the main model and is what we gonna use throughout the tutorials.

A PDF is defined by

\begin{align}
\mathrm{PDF}_{f(x)}(x; \theta) = \frac{f(x; \theta)}{\int_{a}^{b} f(x; \theta)}
\end{align}

where a and b define the normalization range (`norm_range`), over which (by inserting into the above definition) the integral of the PDF is unity.

zfit has a modular approach to things and this is also true for models. While the normalization itself (e.g. what are parameters, what is normalized data) will already be pre-defined in the model, models are composed of functions that are transparently called inside. For example, a Gaussian would usually be implemented by writing a Python function `def gauss(x, mu, sigma)`, which does not care about the normalization and then be wrapped in a PDF, where the normalization and what is a parameter is defined.

In principle, we can go far by using simply functions (e.g. [TensorFlowAnalysis/AmpliTF](https://github.com/apoluekt/AmpliTF) by Anton Poluektov uses this approach quite successfully for Amplitude Analysis), but this design has limitations for a more general fitting library such as zfit (or even [TensorWaves](https://github.com/ComPWA/tensorwaves), being built on top of AmpliTF).
The main thing is to keep track of the different ordering of the data and parameters, especially the dependencies.

Let's create a simple Gaussian PDF. We already defined the `Space` for the data before, now we only need the parameters. This are a different object than a `Space`.

### Parameter
A `Parameter` (there are different kinds actually, more on that later) takes the following arguments as input:
`Parameter(human readable name, initial value[, lower limit, upper limit])` where the limits are recommended but not mandatory. Furthermore, `step_size` can be given (which is useful to be around the given uncertainty, e.g. for large yields or small values it can help a lot to set this). Also, a `floating` argument is supported, indicating whether the parameter is allowed to float in the fit or not (just omitting the limits does _not_ make a parameter constant).

Parameters have a unique name. This is served as the identifier for e.g. fit results. However, a parameter _cannot_ be retrieved by its string identifier (its name) but the object itself should be used. In places where a parameter maps to something, the object itself is needed, not its name.

```python
mu = zfit.Parameter('mu', 1, -3, 3, step_size=0.2)
sigma_num = zfit.Parameter('sigma42', 1, 0.1, 10, floating=False)
```

These attributes can be changed:

```python
print(f"sigma is float: {sigma_num.floating}")
sigma_num.floating = True
print(f"sigma is float: {sigma_num.floating}")
```

    sigma is float: False
    sigma is float: True

*PITFALL NOTEBOOKS: since the parameters have a unique name, a second parameter with the same name cannot be created; the behavior is undefined and therefore it raises an error.
While this does not pose a problem in a normal Python script, it does in a Jupyter-like notebook, since it is an often practice to "rerun" a cell as an attempt to "reset" things. Bear in mind that this does not make sense, from a logic point of view. The parameter already exists. Best practice: write a small wrapper, do not rerun the parameter creation cell or simply rerun the notebook (restart kernel & run all). For further details, have a look at the discussion and arguments [here](https://github.com/zfit/zfit/issues/186)*

Now we have everything to create a Gaussian PDF:

```python
gauss = zfit.pdf.Gauss(obs=obs, mu=mu, sigma=sigma_num)
```

Since this holds all the parameters and the observables are well defined, we can retrieve them

```python
gauss.n_obs  # dimensions
```

    1

```python
gauss.obs
```

    ('obs1',)

```python
gauss.space
```

    <zfit Space obs=('obs1',), axes=(0,), limits=(array([[-5.]]), array([[10.]]))>

```python
gauss.norm_range
```

    <zfit Space obs=('obs1',), axes=(0,), limits=(array([[-5.]]), array([[10.]]))>

As we've seen, the `obs` we defined is the `space` of Gauss: this acts as the default limits whenever needed (e.g. for sampling). `gauss` also has a `norm_range`, which equals by default as well to the `obs` given, however, we can explicitly change that with `set_norm_range`.

We can also access the parameters of the PDF in two ways, depending on our intention:
either by _name_ (the parameterization name, e.g. `mu` and `sigma`, as defined in the `Gauss`), which is useful if we are interested in the parameter that _describes_ the shape

```python
gauss.params
```

    OrderedDict([('mu', <zfit.Parameter 'mu' floating=True value=1>),
                 ('sigma', <zfit.Parameter 'sigma42' floating=True value=1>)])

or to retrieve all the parameters that the PDF depends on. As this now may sounds trivial, we will see later that models can depend on other models (e.g. sums) and parameters on other parameters. There is one function that automatically retrieves _all_ dependencies, `get_params`. It takes three arguments to filter:
- floating: whether to filter only floating parameters, only non-floating or don't discriminate
- is_yield: if it is a yield, or not a yield, or both
- extract_independent: whether to recursively collect all parameters. This, and the explanation for why independent, can be found later on in the `Simultaneous` tutorial.

Usually, the default is exactly what we want if we look for _all free parameters that this PDF depends on_.

```python
gauss.get_params()
```

    OrderedSet([<zfit.Parameter 'mu' floating=True value=1>, <zfit.Parameter 'sigma42' floating=True value=1>])

The difference will also be clear if we e.g. use the same parameter twice:

```python
gauss_only_mu = zfit.pdf.Gauss(obs=obs, mu=mu, sigma=mu)
print(f"params={gauss_only_mu.params}")
print(f"get_params={gauss_only_mu.get_params()}")
```

    params=OrderedDict([('mu', <zfit.Parameter 'mu' floating=True value=1>), ('sigma', <zfit.Parameter 'mu' floating=True value=1>)])
    get_params=OrderedSet([<zfit.Parameter 'mu' floating=True value=1>])

## Functionality

PDFs provide a few useful methods. The main features of a zfit PDF are:

- `pdf`: the normalized value of the PDF. It takes an argument `norm_range` that can be set to `False`, in which case we retrieve the unnormalized value
- `integrate`: given a certain range, the PDF is integrated. As `pdf`, it takes a `norm_range` argument that integrates over the unnormalized `pdf` if set to `False`
- `sample`: samples from the pdf and returns a `Data` object

```python
integral = gauss.integrate(limits=(-1, 3))  # corresponds to 2 sigma integral
integral
```

    <tf.Tensor: shape=(1,), dtype=float64, numpy=array([0.95449974])>

### Tensors

As we see, many zfit functions return Tensors. This is however no magical thing! If we're outside of models, than we can always safely convert them to a numpy array by calling `zfit.run(...)` on it (or any structure containing potentially multiple Tensors). However, this may not even be required often! They can be added just like numpy arrays and interact well with Python and Numpy:

```python
np.sqrt(integral)
```

    array([0.97698502])

They also have shapes, dtypes, can be slices etc. So do not convert them except you need it. More on this can be seen in the talk later on about zfit and TensorFlow 2.0.

```python
sample = gauss.sample(n=1000)  # default space taken as limits
sample
```

    <zfit.core.data.SampleData at 0x7f1790089d00>

```python
sample.unstack_x()[:10]
```

    <tf.Tensor: shape=(10,), dtype=float64, numpy=
    array([-0.09952247,  1.49016828,  2.12148953,  0.39491123,  1.08061772,
            0.49297195,  0.57305784,  1.98737622,  1.46084697,  0.30197322])>

```python
sample.n_obs
```

    1

```python
sample.obs
```

    ('obs1',)

We see that sample returns also a zfit `Data` object with the same space as it was sampled in. This can directly be used e.g.

```python
probs = gauss.pdf(sample)
probs[:10]
```

    <tf.Tensor: shape=(10,), dtype=float64, numpy=
    array([0.21796662, 0.35378319, 0.21271375, 0.33220443, 0.39764798,
           0.35082167, 0.36419045, 0.24502515, 0.35875036, 0.31268492])>

**NOTE**: In case you want to do this repeatedly (e.g. for toy studies), there is a way more efficient way (see later on)

## Plotting

so far, we have a dataset and a PDF. Before we go for fitting, we can make a plot. This functionality is not _directly_ provided in zfit (but can be added to [zfit-physics](https://github.com/zfit/zfit-physics)). It is however simple enough to do it:

```python
def plot_model(model, data, scale=1, plot_data=True):  # we will use scale later on

    nbins = 50

    lower, upper = data.data_range.limit1d
    x = tf.linspace(lower, upper, num=1000)  # np.linspace also works
    y = model.pdf(x) * size_normal / nbins * data.data_range.area()
    y *= scale
    plt.plot(x, y)
    data_plot = zfit.run(z.unstack_x(data))  # we could also use the `to_pandas` method
    if plot_data:
        plt.hist(data_plot, bins=nbins)
```

```python
plot_model(gauss, data_normal)
```

We can of course do better (and will see that later on, continuously improve the plots), but this is quite simple and gives us the full power of matplotlib.

### Different models

zfit offers a selection of predefined models (and extends with models from zfit-physics that contain physics specific models such as ARGUS shaped models).

```python
print(zfit.pdf.__all__)
```

    ['BasePDF', 'BaseFunctor', 'Exponential', 'CrystalBall', 'DoubleCB', 'Gauss', 'Uniform', 'TruncatedGauss', 'WrapDistribution', 'Cauchy', 'Chebyshev', 'Legendre', 'Chebyshev2', 'Hermite', 'Laguerre', 'RecursivePolynomial', 'ProductPDF', 'SumPDF', 'GaussianKDE1DimV1', 'ZPDF', 'SimplePDF', 'SimpleFunctorPDF']

To create a more realistic model, we can build some components for a mass fit with a
- signal component: CrystalBall
- combinatorial background: Exponential
- partial reconstructed background on the left: Kernel Density Estimation

```python
mass_obs = zfit.Space('mass', (0, 1000))
```

```python
# Signal component

mu_sig = zfit.Parameter('mu_sig', 400, 100, 600)
sigma_sig = zfit.Parameter('sigma_sig', 50, 1, 100)
alpha_sig = zfit.Parameter('alpha_sig', 300, 100, 400)
n_sig = zfit.Parameter('n sig', 4, 0.1, 30)
signal = zfit.pdf.CrystalBall(obs=mass_obs, mu=mu_sig, sigma=sigma_sig, alpha=alpha_sig, n=n_sig)
```

```python
# combinatorial background

lam = zfit.Parameter('lambda', -0.01, -0.05, -0.001)
comb_bkg = zfit.pdf.Exponential(lam, obs=mass_obs)
```

```python
part_reco_data = np.random.normal(loc=200, scale=150, size=700)
part_reco_data = zfit.Data.from_numpy(obs=mass_obs, array=part_reco_data)  # we don't need to do this but now we're sure it's inside the limits

part_reco = zfit.pdf.GaussianKDE1DimV1(obs=mass_obs, data=part_reco_data, bandwidth='adaptive')
```

## Composing models

We can also compose multiple models together. Here we'll stick to one dimensional models, the extension to multiple dimensions is explained in the "custom models tutorial".

Here we will use a `SumPDF`. This takes pdfs and fractions. If we provide n pdfs and:
- n - 1 fracs: the nth fraction will be 1 - sum(fracs)
- n fracs: no normalization attempt is done by `SumPDf`. If the fracs are not implicitly normalized, this can lead to bad fitting
  behavior if there is a degree of freedom too much

```python
sig_frac = zfit.Parameter('sig_frac', 0.3, 0, 1)
comb_bkg_frac = zfit.Parameter('comb_bkg_frac', 0.25, 0, 1)
model = zfit.pdf.SumPDF([signal, comb_bkg, part_reco], [sig_frac, comb_bkg_frac])
```

In order to have a corresponding data sample, we can just create one. Since we want to fit to this dataset later on, we will create it with slightly different values. Therefore, we can use the ability of a parameter to be set temporarily to a certain value with

```python
print(f"before: {sig_frac}")
with sig_frac.set_value(0.25):
    print(f"new value: {sig_frac}")
print(f"after 'with': {sig_frac}")
```

    before: <zfit.Parameter 'sig_frac' floating=True value=0.3>
    new value: <zfit.Parameter 'sig_frac' floating=True value=0.25>
    after 'with': <zfit.Parameter 'sig_frac' floating=True value=0.3>

While this is useful, it does not fully scale up. We can use the `zfit.param.set_values` helper therefore.
(_Sidenote: instead of a list of values, we can also use a `FitResult`, the given parameters then take the value from the result_)

```python
with zfit.param.set_values([mu_sig, sigma_sig, sig_frac, comb_bkg_frac, lam], [370, 34, 0.18, 0.15, -0.006]):
    data = model.sample(n=10000)
```

```python
plot_model(model, data);
```

Plotting the components is not difficult now: we can either just plot the pdfs separately (as we still can access them) or in a generalized manner by accessing the `pdfs` attribute:

```python
def plot_comp_model(model, data):
    for mod, frac in zip(model.pdfs, model.params.values()):
        plot_model(mod, data, scale=frac, plot_data=False)
    plot_model(model, data)
```

```python
plot_comp_model(model, data)
```

Now we can add legends etc. Btw, did you notice that actually, the `frac` params are zfit `Parameters`? But we just used them as if they were Python scalars and it works.

```python
print(model.params)
```

    OrderedDict([('frac_0', <zfit.Parameter 'sig_frac' floating=True value=0.3>), ('frac_1', <zfit.Parameter 'comb_bkg_frac' floating=True value=0.25>), ('frac_2', <zfit.ComposedParameter 'Composed_autoparam_2' params=OrderedDict([('param_0', <zfit.Parameter 'sig_frac' floating=True value=0.3>), ('param_1', <zfit.Parameter 'comb_bkg_frac' floating=True value=0.25>)]) value=0.45>)])

### Extended PDFs

So far, we have only looked at normalized PDFs that do contain information about the shape but not about the _absolute_ scale. We can make a PDF extended by adding a yield to it.

The behavior of the new, extended PDF does **NOT change**, any methods we called before will act the same. Only exception, some may require an argument _less_ now. All the methods we used so far will return the same values. What changes is that the flag `model.is_extended` now returns `True`. Furthermore, we have now a few more methods that we can use which would have raised an error before:
- `get_yield`: return the yield parameter (notice that the yield is _not_ added to the shape parameters `params`)
- `ext_{pdf,integrate}`: these methods return the same as the versions used before, however, multiplied by the yield
- `sample` is still the same, but does not _require_ the argument `n` anymore. By default, this will now equal to a _poissonian sampled_ n around the yield.

The `SumPDF` now does not strictly need `fracs` anymore: if _all_ input PDFs are extended, the sum will be as well and use the (normalized) yields as fracs

The preferred way to create an extended PDf is to use `PDF.create_extended(yield)`. However, since this relies on copying the PDF (which may does not work for different reasons), there is also a `set_yield(yield)` method that sets the yield in-place. This won't lead to ambiguities, as everything is supposed to work the same.

```python
yield_model = zfit.Parameter('yield_model', 10000, 0, 20000, step_size=10)
model_ext = model.create_extended(yield_model)
```

alternatively, we can create the models as extended and sum them up

```python
sig_yield = zfit.Parameter('sig_yield', 2000, 0, 10000, step_size=1)
sig_ext = signal.create_extended(sig_yield)

comb_bkg_yield = zfit.Parameter('comb_bkg_yield', 6000, 0, 10000, step_size=1)
comb_bkg_ext = comb_bkg.create_extended(comb_bkg_yield)

part_reco_yield = zfit.Parameter('part_reco_yield', 2000, 0, 10000, step_size=1)
part_reco.set_yield(part_reco_yield)  # unfortunately, `create_extended` does not work here. But no problem, it won't change anyting.
part_reco_ext = part_reco
```

```python
model_ext_sum = zfit.pdf.SumPDF([sig_ext, comb_bkg_ext, part_reco_ext])
```

# Loss

A loss combines the model and the data, for example to build a likelihood. Furthermore, it can contain constraints, additions to the likelihood. Currently, if the `Data` has weights, these are automatically taken into account.

```python
nll_gauss = zfit.loss.UnbinnedNLL(gauss, data_normal)
```

The loss has several attributes to be transparent to higher level libraries. We can calculate the value of it using `value`.

```python
nll_gauss.value()
```

    <tf.Tensor: shape=(), dtype=float64, numpy=32625.929833399066>

Notice that due to graph building, this will take significantly longer on the first run. Rerun the cell above and it will be way faster.

Furthermore, the loss also provides a possibility to calculate the gradients or, often used, the value and the gradients.

We can access the data and models (and possible constraints)

```python
nll_gauss.model
```

    [<zfit.Gauss  params=[mu, sigma42] dtype=float64>0]

```python
nll_gauss.data
```

    [<zfit.core.data.Data at 0x7f17900ed820>]

```python
nll_gauss.constraints
```

    []

Similar to the models, we can also get the parameters via `get_params`.

```python
nll_gauss.get_params()
```

    OrderedSet([<zfit.Parameter 'mu' floating=True value=1>, <zfit.Parameter 'sigma42' floating=True value=1>])

### Extended loss

More interestingly, we can now build a loss for our composite sum model using the sampled data. Since we created an extended model, we can now also create an extended likelihood, taking into account a Poisson term to match the yield to the number of events.

```python
nll = zfit.loss.ExtendedUnbinnedNLL(model_ext_sum, data)
```

```python
nll.get_params()
```

    OrderedSet([<zfit.Parameter 'sig_yield' floating=True value=2000>, <zfit.Parameter 'comb_bkg_yield' floating=True value=6000>, <zfit.Parameter 'part_reco_yield' floating=True value=2000>, <zfit.Parameter 'alpha_sig' floating=True value=300>, <zfit.Parameter 'mu_sig' floating=True value=400>, <zfit.Parameter 'n sig' floating=True value=4>, <zfit.Parameter 'sigma_sig' floating=True value=50>, <zfit.Parameter 'lambda' floating=True value=-0.01>])

# Minimization

While a loss is interesting, we usually want to minimize it. Therefore we can use the minimizers in zfit, most notably `Minuit`, a wrapper around the [iminuit minimizer](https://github.com/scikit-hep/iminuit).

The philosophy is to create a minimizer instance that is mostly _stateless_, e.g. does not remember the position (there are considerations to make it possible to have a state, in case you feel interested, [contact us](https://github.com/zfit/zfit#contact))

Given that iminuit provides us with a very reliable and stable minimizer, it is usually recommended to use this. Others are implemented as well and could easily be wrapped, however, the convergence is usually not as stable.

Minuit has a few options:
- `tolerance`: the Estimated Distance to Minimum (EDM) criteria for convergence (default 1e-3)
- `verbosity`: between 0 and 10, 5 is normal, 7 is verbose, 10 is maximum
- `use_minuit_grad`: if True, uses the Minuit numerical gradient instead of the TensorFlow gradient. This is usually more stable for smaller fits; furthermore the TensorFlow gradient _can_ (experience based) sometimes be wrong.

```python
minimizer = zfit.minimize.Minuit(use_minuit_grad=True)
```

For the minimization, we can call `minimize`, which takes a
- loss as we created above
- optionally: the parameters to minimize

By default, `minimize` uses all the free floating parameters (obtained with `get_params`). We can also explicitly specify which ones to use by giving them (or better, objects that depend on them) to `minimize`; note however that non-floating parameters, even if given explicitly to `minimize` won 't be minimized.

## Pre-fit parts of the PDF

Before we want to fit the whole PDF however, it can be useful to pre-fit it. A way can be to fix the combinatorial background by fitting the exponential to the right tail.

Therefore we create a new data object with an additional cut and furthermore, set the normalization range of the background pdf to the range we are interested in.

```python
values = z.unstack_x(data)
obs_right_tail = zfit.Space('mass', (700, 1000))
data_tail = zfit.Data.from_tensor(obs=obs_right_tail, tensor=values)
with comb_bkg.set_norm_range(obs_right_tail):
    nll_tail = zfit.loss.UnbinnedNLL(comb_bkg, data_tail)
    minimizer.minimize(nll_tail)
```

    ------------------------------------------------------------------
    | FCN = 328                     |      Ncalls=19 (19 total)      |
    | EDM = 9.18e-10 (Goal: 0.001)  |            up = 0.5            |
    ------------------------------------------------------------------
    |  Valid Min.   | Valid Param.  | Above EDM | Reached call limit |
    ------------------------------------------------------------------
    |     True      |     True      |   False   |       False        |
    ------------------------------------------------------------------
    | Hesse failed  |   Has cov.    | Accurate  | Pos. def. | Forced |
    ------------------------------------------------------------------
    |     False     |     True      |   True    |   True    | False  |
    ------------------------------------------------------------------

Since we now fit the lambda parameter of the exponential, we can fix it.

```python
lam.floating = False
lam
```

    <zfit.Parameter 'lambda' floating=False value=-0.008587>

```python
result = minimizer.minimize(nll)
```

    ------------------------------------------------------------------
    | FCN = -1.93e+04               |     Ncalls=185 (185 total)     |
    | EDM = 6.35e-06 (Goal: 0.001)  |            up = 0.5            |
    ------------------------------------------------------------------
    |  Valid Min.   | Valid Param.  | Above EDM | Reached call limit |
    ------------------------------------------------------------------
    |     True      |     True      |   False   |       False        |
    ------------------------------------------------------------------
    | Hesse failed  |   Has cov.    | Accurate  | Pos. def. | Forced |
    ------------------------------------------------------------------
    |     False     |     True      |   True    |   True    | False  |
    ------------------------------------------------------------------

```python
plot_comp_model(model_ext_sum, data)
```

# Fit result

The result of every minimization is stored in a `FitResult`. This is the last stage of the zfit workflow and serves as the interface to other libraries. Its main purpose is to store the values of the fit, to reference to the objects that have been used and to perform (simple) uncertainty estimation.

```python
print(result)
```

    FitResult of
    <ExtendedUnbinnedNLL model=[<zfit.SumPDF  params=[Composed_autoparam_5, Composed_autoparam_6, Composed_autoparam_7] dtype=float64>0] data=[<zfit.core.data.SampleData object at 0x7f176002fdc0>] constraints=[]>
    with
    <Minuit strategy=PushbackStrategy tolerance=0.001>

    ╒═════════╤═════════════╤══════════════════╤═════════╤═════════════╕
    │ valid   │ converged   │ param at limit   │ edm     │ min value   │
    ╞═════════╪═════════════╪══════════════════╪═════════╪═════════════╡
    │ True    │ True        │ False            │ 6.4e-06 │ -1.93e+04   │
    ╘═════════╧═════════════╧══════════════════╧═════════╧═════════════╛

    Parameters
    name               value    at limit
    ---------------  -------  ----------
    sig_yield           1804       False
    comb_bkg_yield      1095       False
    part_reco_yield     7101       False
    alpha_sig            300       False
    mu_sig             370.8       False
    n sig                  4       False
    sigma_sig          33.87       False

This gives an overview over the whole result. Often we're mostly interested in the parameters and their values, which we can access with a `params` attribute.

```python
print(result.params)
```

    name               value    at limit
    ---------------  -------  ----------
    sig_yield           1804       False
    comb_bkg_yield      1095       False
    part_reco_yield     7101       False
    alpha_sig            300       False
    mu_sig             370.8       False
    n sig                  4       False
    sigma_sig          33.87       False

This is a `dict` which stores any knowledge about the parameters and can be accessed by the parameter (object) itself:

```python
result.params[mu_sig]
```

    {'value': 370.7878667059073}

'value' is the value at the minimum. To obtain other information about the minimization process, `result` contains more attributes:
- fmin: the function minimum
- edm: estimated distance to minimum
- info: contains a lot of information, especially the original information returned by a specific minimizer
- converged: if the fit converged

```python
result.fmin
```

    -19300.779346305906

## Estimating uncertainties

The `FitResult` has mainly two methods to estimate the uncertainty:
- a profile likelihood method (like MINOS)
- Hessian approximation of the likelihood (like HESSE)

When using `Minuit`, this uses (currently) it's own implementation. However, zfit has its own implementation, which are likely to become the standard and can be invoked by changing the method name.

Hesse is also [on the way to implement](https://github.com/zfit/zfit/pull/244) the [corrections for weights](https://inspirehep.net/literature/1762842).

We can explicitly specify which parameters to calculate, by default it does for all.

```python
result.hesse()
```

    OrderedDict([(<zfit.Parameter 'sig_yield' floating=True value=1804>,
                  {'error': 70.15087267737408}),
                 (<zfit.Parameter 'comb_bkg_yield' floating=True value=1095>,
                  {'error': 70.35114369167526}),
                 (<zfit.Parameter 'part_reco_yield' floating=True value=7101>,
                  {'error': 131.80924912505543}),
                 (<zfit.Parameter 'alpha_sig' floating=True value=300>,
                  {'error': 141.4213562373095}),
                 (<zfit.Parameter 'mu_sig' floating=True value=370.8>,
                  {'error': 1.3661545484142485}),
                 (<zfit.Parameter 'n sig' floating=True value=4>,
                  {'error': 10.069756698215553}),
                 (<zfit.Parameter 'sigma_sig' floating=True value=33.87>,
                  {'error': 1.2650183734125646})])

```python
# result.hesse(method='hesse_np')
```

We get the result directly returned. This is also added to `result.params` for each parameter and is nicely displayed with an added column

```python
print(result.params)
```

    name               value    minuit_hesse    at limit
    ---------------  -------  --------------  ----------
    sig_yield           1804     +/-      70       False
    comb_bkg_yield      1095     +/-      70       False
    part_reco_yield     7101     +/- 1.3e+02       False
    alpha_sig            300     +/- 1.4e+02       False
    mu_sig             370.8     +/-     1.4       False
    n sig                  4     +/-      10       False
    sigma_sig          33.87     +/-     1.3       False

```python
errors, new_result = result.errors(params=[sig_yield, part_reco_yield, mu_sig])  # just using three for speed reasons
```

    /home/jonas/Documents/physics/software/zfit_project/zfit_repo/zfit/minimizers/fitresult.py:360: FutureWarning: 'minuit_minos' will be changed as the default errors method to a custom implementationwith the same functionality. If you want to make sure that 'minuit_minos' will be used in the future, add it explicitly as in `errors(method='minuit_minos')`
      warnings.warn("'minuit_minos' will be changed as the default errors method to a custom implementation"

```python
# errors, new_result = result.errors(params=[yield_model, sig_frac, mu_sig], method='zfit_error')
```

```python
print(errors)
```

    OrderedDict([(<zfit.Parameter 'sig_yield' floating=True value=1804>, MError(name='sig_yield', is_valid=True, lower=-69.66325485797651, upper=70.75759128186598, lower_valid=True, upper_valid=True, at_lower_limit=False, at_upper_limit=False, at_lower_max_fcn=False, at_upper_max_fcn=False, lower_new_min=False, upper_new_min=False, nfcn=138, min=1803.8532804234746)), (<zfit.Parameter 'part_reco_yield' floating=True value=7101>, MError(name='part_reco_yield', is_valid=True, lower=-131.88637854089905, upper=132.34447403753458, lower_valid=True, upper_valid=True, at_lower_limit=False, at_upper_limit=False, at_lower_max_fcn=False, at_upper_max_fcn=False, lower_new_min=False, upper_new_min=False, nfcn=60, min=7101.093509366213)), (<zfit.Parameter 'mu_sig' floating=True value=370.8>, MError(name='mu_sig', is_valid=True, lower=-1.36717243612375, upper=1.356060293846917, lower_valid=True, upper_valid=True, at_lower_limit=False, at_upper_limit=False, at_lower_max_fcn=False, at_upper_max_fcn=False, lower_new_min=False, upper_new_min=False, nfcn=106, min=370.7878667059073))])

```python
print(result.params)
```

    name               value    minuit_hesse         minuit_minos    at limit
    ---------------  -------  --------------  -------------------  ----------
    sig_yield           1804     +/-      70  -     70   +     71       False
    comb_bkg_yield      1095     +/-      70                            False
    part_reco_yield     7101     +/- 1.3e+02  -1.3e+02   +1.3e+02       False
    alpha_sig            300     +/- 1.4e+02                            False
    mu_sig             370.8     +/-     1.4  -    1.4   +    1.4       False
    n sig                  4     +/-      10                            False
    sigma_sig          33.87     +/-     1.3                            False

#### What is 'new_result'?

When profiling a likelihood, such as done in the algorithm used in `errors`, a new minimum can be found. If this is the case, this new minimum will be returned, otherwise `new_result` is `None`. Furthermore, the current `result` would be rendered invalid by setting the flag `valid` to `False`. _Note_: this behavior only applies to the zfit internal error estimator.

### A simple profile

There is no default function (yet) for simple profiling plot. However, again, we're in Python and it's simple enough to do that for a parameter. Let's do it for `sig_yield`

```python
x = np.linspace(1600, 2000, num=50)
y = []
sig_yield.floating = False
for val in x:
    sig_yield.set_value(val)
    y.append(nll.value())

sig_yield.floating = True
zfit.param.set_values(nll.get_params(), result)
```

    <zfit.util.temporary.TemporarilySet at 0x7f16cc8d7550>

```python
plt.plot(x, y)
```

We can also access the covariance matrix of the parameters

```python
result.covariance()
```

    array([[ 4.92114494e+03,  1.14332473e+03, -4.33133015e+03,
             0.00000000e+00, -1.99905281e+01,  0.00000000e+00,
             4.04511667e+01],
           [ 1.14332473e+03,  4.94928342e+03, -5.67290390e+03,
             0.00000000e+00, -6.88067541e+00,  0.00000000e+00,
             1.48550756e+01],
           [-4.33133015e+03, -5.67290390e+03,  1.73736782e+04,
             0.00000000e+00,  2.77291911e+01,  0.00000000e+00,
            -5.58205907e+01],
           [ 0.00000000e+00,  0.00000000e+00,  0.00000000e+00,
             2.00000000e+04,  0.00000000e+00,  0.00000000e+00,
             0.00000000e+00],
           [-1.99905281e+01, -6.88067541e+00,  2.77291911e+01,
             0.00000000e+00,  1.86637825e+00,  0.00000000e+00,
            -3.46142640e-01],
           [ 0.00000000e+00,  0.00000000e+00,  0.00000000e+00,
             0.00000000e+00,  0.00000000e+00,  1.01400000e+02,
             0.00000000e+00],
           [ 4.04511667e+01,  1.48550756e+01, -5.58205907e+01,
             0.00000000e+00, -3.46142640e-01,  0.00000000e+00,
             1.60027149e+00]])

# End of zfit

This is where zfit finishes and other libraries take over.

# Beginning of hepstats

`hepstats` is a library containing statistical tools and utilities for high energy physics. In particular you do statistical inferences using the models and likelhoods function constructed in `zfit`.

Short example: let's compute for instance a confidence interval at 68 % confidence level on the mean of the gaussian defined above.

```python
from hepstats.hypotests.parameters import POIarray
from hepstats.hypotests.calculators import AsymptoticCalculator
from hepstats.hypotests import ConfidenceInterval
```

```python
calculator = AsymptoticCalculator(input=result, minimizer=minimizer)
```

```python
value = result.params[mu_sig]["value"]
error = result.params[mu_sig]["minuit_hesse"]["error"]

mean_scan = POIarray(mu_sig, np.linspace(value - 1.5*error, value + 1.5*error, 10))
```

```python
ci = ConfidenceInterval(calculator, mean_scan)
```

```python
ci.interval()
```

    Confidence interval on mu_sig:
    	369.42424650773955 < mu_sig < 372.1404588905455 at 68.0% C.L.

    {'observed': 370.7878667059073,
     'upper': 372.1404588905455,
     'lower': 369.42424650773955}

```python
from utils import one_minus_cl_plot

ax = one_minus_cl_plot(ci)
ax.set_xlabel("mean")
```

There will be more of `hepstats` later.


# Week 6 - Sequential Models
## 1. Recurrent Neural Networks (RNNs)

### 1.1 RNN as a sequential model

The neural networks in previous tutorials are **feedforward neural networks**, which take one input (e.g. information about one credit card transaction) and transform it onto one output (e.g. fraudulent or not). Here, the model processes each instance independently.

In sequential data however, instances of data are not independent. Sequential data is data where order matters, including text streams, audio clips, video clips, time series like stock prices. Predicting the next value in the sequence often requires information about many past values in the sequence, not just the latest value.

**Recurrent neural networks** are a type of neural networks suitable for sequential data. Unike feedforward neural networks, RNN keeps a memory of past inputs. The newest input (latest time step in the sequence) is combined with this memory to generate an output that continues the sequence.

(E.g. to predict the closing price of a stock tomorrow, the RNN takes today's closing price, as well as memory of closing prices in the past n number of days.)

```python
import pandas_datareader.data as web
import pandas as pd
import matplotlib.pyplot as plt
import datetime as dt
```

```python
# Use pandas datareader to obtain stock data from Yahoo Finance
data = web.DataReader('AAPL', 'yahoo', start='2021-01-01', end='2021-11-23')

data.head()
```

      High
      Low
      Open
      Close
      Volume
      Adj Close

      Date

      2021-01-04
      133.610001
      126.760002
      133.520004
      129.410004
      143301900.0
      128.617111

      2021-01-05
      131.740005
      128.429993
      128.889999
      131.009995
      97664900.0
      130.207291

      2021-01-06
      131.050003
      126.379997
      127.720001
      126.599998
      155088000.0
      125.824326

      2021-01-07
      131.630005
      127.860001
      128.360001
      130.919998
      109578200.0
      130.117859

      2021-01-08
      132.630005
      130.229996
      132.429993
      132.050003
      105158200.0
      131.240921

```python
# visualizing the daily closing price
close=data['Close']

fig, ax = plt.subplots(figsize=(8,5))
ax.plot(close.index,close,label="AAPL")
ax.set_xlabel("Date")
ax.set_ylabel("Closing Price ($)")
fig.suptitle('Daily closing price', fontsize=15)
ax.legend()
print(close)
```

### 1.2 RNN structure and layers

#### How is a recurrent neuron different from the usual neuron?

1. **The recurrent neuron as a hidden state / cell state** ($h_t$): the internal memory of the recurrent neuron. It is initialized as a zero matrix, and updated after each time step to include the information inputted at that time step.

2. **The recurrent neuron's output is a function of input and memory ($y_t = f(X_t,  h_{t-1})$), instead of just a function of input($y_t = f(X_t)$)**

#### Two layers of an RNN:
1. **Recurrent layer**: updating the hidden state

$h_t = f(h_{t-1} \times W_{hh} + X_t \times W_xh + B_1)$

$h_t$: hidden state at time step $t$. $h_0$ is initialized as a matrix of zeros.
$W_{hh}$: weight matrix determining how the previous hidden state affects the current hidden state
$W_{xh}$: weight matrix determining how the current input ($X$) affects the current hidden state
$B_1$: bias of the recurrent layer
$f()$: activation function of the recurrent layer

2. **Feedforward layer**: using the updated hidden state to produce an output

$\hat{y}_t = g(h_t \times W_{hy} + B_2)$

$\hat{y}_t$: output at time step $t$
$W_{hy}$: weight matrix determining how current hidden state affects output
$B_1$: bias of the feedforward layer
$g()$: activation function of the feedforward layer

Note: once trained, the same weight matrices will be used across all time steps.

#### Activation functions
Just like neural networks in previous tutorials, RNNs need activation functions to introduce non-linearity and enable learning of more complex patterns.

Common activation functions for RNN:
- sigmoid
- ReLU
- tanh

### 1.3 Back propagation through time
- Step 1: Values of an input sequence are passed through the model in order. At each time step, the model generates an output and calculates its loss.

    $$L_t = f_l(\hat{y}, y)$$

    $L_t$: loss of the output at time $t$

    $f_l$: loss function

    $\hat{y}$: prediction output

    $y$: actual label

- Step 2: The losses of all the time steps are summed into a total loss for the input sequence.

    $$L = \sum_{t} L_t $$

    $L$: total loss

- Step 3: Error is propagated back through each time step by taking the derivative of total loss with respect to each trainable parameter

    $$\frac{\partial L}{\partial \theta} = \sum_{t} \frac{\partial L_t}{\partial \theta}
    $$

    Where $\theta$ is any trainable parameter.

    The equation above represents the derivative of the total loss as a sum of the derivatives of losses at each time step. These time step losses are each calculated with:
    \begin{equation}
    \frac{\partial L_t}{\partial \theta} = \sum_{k=1}^{t} \frac{\partial L_t}{\partial y_t} \frac{\partial y_t}{\partial h_t}  \frac{\partial h_t}{\partial h_k} \frac{\partial h_k}{\partial \theta}
    \end{equation}

    Thus, the derivative of the total loss is:

    \begin{equation}
    \frac{\partial L}{\partial \theta} = \sum_{t} \sum_{k=1}^{t} \frac{\partial L_t}{\partial y_t} \frac{\partial y_t}{\partial h_t}  \frac{\partial h_t}{\partial h_k} \frac{\partial h_k}{\partial \theta}
    \end{equation}

    This derivative w.r.t each weight is multiplied by a learning rate to adjust the weights, and learning is achieved.

### 1.4 RNN implementation from scratch

So far, we have defined the structure of simple recurrent neural network; however, how does this translate into code?

In the following section, we will have a look into implementation from scratch for the prediction of simple sine function.

For starters, let's create a sample dataset by dividing the sine function into sequences of 50 units. For the model itself, we will try to predict the next value for the sequence.

```python
import math
import numpy as np
import matplotlib.pyplot as plt

#Preparing our dataset
sine =  np.array([math.sin(x) for x in np.arange(200)])

#Creating our train dataset
X = []
Y = []

#Defining the lenght of our sequence and number of samples
seq_len = 50
num_records = len(sine) - seq_len

for i in range(num_records - 50):
    X.append(sine[i:i+seq_len])
    Y.append(sine[i+seq_len])

X = np.array(X)
X = np.expand_dims(X, axis=2)

Y = np.array(Y)
Y = np.expand_dims(Y, axis=1)
```

To later test our model's performance, let's also create a validation dataset (we will use to after the training).

```python
#Creating our validation dataset
X_val = []
Y_val = []

for i in range(num_records - 50, num_records):
    X_val.append(sine[i:i+seq_len])
    Y_val.append(sine[i+seq_len])

X_val = np.array(X_val)
X_val = np.expand_dims(X_val, axis=2)

Y_val = np.array(Y_val)
Y_val = np.expand_dims(Y_val, axis=1)
```

Now, let's start defining our simple RNN model. As it has been mentioned earlier, our model will produce a single output after processing input through a hidden state of 100 units.

In the following cell, we also need to initialize the starting weights of our network:
- U - weights between the input and hidden layers
- V - weights between the hidden and output layers
- W - weights for shared weights in the RNN hidden layer

```python
#Creating RNN
learning_rate = 0.0001
num_epoch = 25
T = 50                   # length of sequence
hidden_dim = 100
output_dim = 1

bptt_truncate = 5
min_clip_value = -10
max_clip_value = 10

#Initializing weights
U = np.random.uniform(0, 1, (hidden_dim, T))
W = np.random.uniform(0, 1, (hidden_dim, hidden_dim))
V = np.random.uniform(0, 1, (output_dim, hidden_dim))
```

Finally, let's also define our activation function used in the hidden layer (sigmoid):

```python
def sigmoid(x):
    return 1 / (1 + np.exp(-x))
```

#### Training our model

As it has been mentioned in the previous sections, the training of RNN model can be divided into the following steps:
- Calculating loss on train dataset
- Forward pass
- Backward pass
- Updating weights

```python
for epoch in range(num_epoch):
    # defining loss
    loss = 0.0

    # generating initial prediction
    for i in range(Y.shape[0]):

        #Retrieving input and output for sequence
        x, y = X[i], Y[i]

        #Initializing value of previous hidden layer
        prev_s = np.zeros((hidden_dim, 1))
        for t in range(T):

            #Forward pass for every sequence
            new_input = np.zeros(x.shape)
            new_input[t] = x[t]

            mulu = np.dot(U, new_input)
            mulw = np.dot(W, prev_s)
            add = mulw + mulu

            #Passing through activation function
            s = sigmoid(add)
            mulv = np.dot(V, s)

            #Defining previous state
            prev_s = s

        #Calculating error
        loss_per_seq = (y - mulv)**2 / 2
        loss += loss_per_seq

    loss = loss / float(y.shape[0])

    print('Epoch: ', epoch + 1, ', Loss: ', loss)

    for i in range(Y.shape[0]):
        x, y = X[i], Y[i]

        layers = []
        prev_s = np.zeros((hidden_dim, 1))
        dU = np.zeros(U.shape)
        dV = np.zeros(V.shape)
        dW = np.zeros(W.shape)

        dU_t = np.zeros(U.shape)
        dV_t = np.zeros(V.shape)
        dW_t = np.zeros(W.shape)

        dU_i = np.zeros(U.shape)
        dW_i = np.zeros(W.shape)

        # Forward pass
        for t in range(T):
            new_input = np.zeros(x.shape)
            new_input[t] = x[t]

            #Multiplying the input witht the weights
            mulu = np.dot(U, new_input)
            mulw = np.dot(W, prev_s)

            #Adding to capture information from the previous timestep
            add = mulw + mulu

            #Passing through activation function
            s = sigmoid(add)
            mulv = np.dot(V, s)

            #Saving current and previous states
            layers.append({'s':s, 'prev_s':prev_s})
            prev_s = s

        dmulv = (mulv - y)

        # Backpropagation
        for t in range(T):
            dV_t = np.dot(dmulv, np.transpose(layers[t]['s']))
            dsv = np.dot(np.transpose(V), dmulv)

            ds = dsv
            dadd = add * (1 - add) * ds

            dmulw = dadd * np.ones_like(mulw)

            dprev_s = np.dot(np.transpose(W), dmulw)

            for i in range(t-1, max(-1, t-bptt_truncate-1), -1):
                ds = dsv + dprev_s
                dadd = add * (1 - add) * ds

                dmulw = dadd * np.ones_like(mulw)
                dmulu = dadd * np.ones_like(mulu)

                dW_i = np.dot(W, layers[t]['prev_s'])
                dprev_s = np.dot(np.transpose(W), dmulw)

                new_input = np.zeros(x.shape)
                new_input[t] = x[t]
                dU_i = np.dot(U, new_input)
                dx = np.dot(np.transpose(U), dmulu)

                dU_t += dU_i
                dW_t += dW_i

            dV += dV_t
            dU += dU_t
            dW += dW_t

            # Ensuring that the gradients' values fall within our predefined range
            # between min_clip_value and max_clip_value

            if dU.max() > max_clip_value:
                dU[dU > max_clip_value] = max_clip_value
            if dV.max() > max_clip_value:
                dV[dV > max_clip_value] = max_clip_value
            if dW.max() > max_clip_value:
                dW[dW > max_clip_value] = max_clip_value

            if dU.min() < min_clip_value:
                dU[dU < min_clip_value] = min_clip_value
            if dV.min() < min_clip_value:
                dV[dV < min_clip_value] = min_clip_value
            if dW.min() < min_clip_value:
                dW[dW < min_clip_value] = min_clip_value

        # Updating our weights
        U -= learning_rate * dU
        V -= learning_rate * dV
        W -= learning_rate * dW
```

    Epoch:  1 , Loss:  [[99818.7089637]]
    Epoch:  2 , Loss:  [[60143.48517752]]
    Epoch:  3 , Loss:  [[30468.26139105]]
    Epoch:  4 , Loss:  [[10793.03559883]]
    Epoch:  5 , Loss:  [[1116.70193538]]
    Epoch:  6 , Loss:  [[34.77460136]]
    Epoch:  7 , Loss:  [[45.98724941]]
    Epoch:  8 , Loss:  [[44.384122]]
    Epoch:  9 , Loss:  [[44.59125663]]
    Epoch:  10 , Loss:  [[44.96802207]]
    Epoch:  11 , Loss:  [[44.20991682]]
    Epoch:  12 , Loss:  [[44.40532668]]
    Epoch:  13 , Loss:  [[44.36853105]]
    Epoch:  14 , Loss:  [[44.79813355]]
    Epoch:  15 , Loss:  [[44.9557158]]
    Epoch:  16 , Loss:  [[44.52688458]]
    Epoch:  17 , Loss:  [[43.92585762]]
    Epoch:  18 , Loss:  [[45.15783118]]
    Epoch:  19 , Loss:  [[44.10425958]]
    Epoch:  20 , Loss:  [[46.47860668]]
    Epoch:  21 , Loss:  [[44.619954]]
    Epoch:  22 , Loss:  [[44.33465024]]
    Epoch:  23 , Loss:  [[44.64385089]]
    Epoch:  24 , Loss:  [[44.57802768]]
    Epoch:  25 , Loss:  [[44.86906721]]

Finally, after training, we can test our model with previously defined validation dataset.

```python
preds = []
for i in range(Y_val.shape[0]):
    x, y = X_val[i], Y_val[i]
    prev_s = np.zeros((hidden_dim, 1))

    for t in range(T):
        mulu = np.dot(U, x)
        mulw = np.dot(W, prev_s)
        add = mulw + mulu
        s = sigmoid(add)
        mulv = np.dot(V, s)
        prev_s = s

    preds.append(mulv)

preds = np.array(preds)

plt.plot(preds[:, 0, 0], 'b')
plt.plot(Y_val[:, 0], 'r')
plt.show()
```

### 1.5 RNN implementation in Tensorflow

```python
import tensorflow as tf
from keras.models import Sequential
from keras.layers import Dense, SimpleRNN
from tensorflow.keras.optimizers import SGD, Adam

from sklearn.preprocessing import MinMaxScaler
from sklearn.metrics import mean_squared_error
import numpy as np
```

    2021-11-24 16:12:28.881246: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
    2021-11-24 16:12:28.881263: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.

```python
#check missing value
data.isna().sum()
```

    High         0
    Low          0
    Open         0
    Close        0
    Volume       0
    Adj Close    0
    dtype: int64

```python
#scale data
scaler = MinMaxScaler()
scaled_data = scaler.fit_transform(close.values.reshape(-1,1))
scaled_data.shape
```

    (226, 1)

```python
#define functions to split data
def input_output_split(dataset, y_start, y_end, history):
  X = []
  y = []
  y_start += history
  for i in range(y_start, y_end):
    indices = range(i-history, i)
    X.append(np.reshape(dataset[indices], (history, 1)))
    y.append(dataset[i])
  return np.array(X), np.array(y)

def test_train_split(dataset, train_ratio, history):
  training_size = int(dataset.size * train_ratio)
  X_train, y_train = input_output_split(dataset, 0, training_size, history)
  X_test, y_test = input_output_split(dataset, training_size, len(dataset), history)
  return X_train, y_train, X_test, y_test

```

RNN expects input dataset to be 3D-shaped (instances, timesteps, features)

```python
#split data
X_train, y_train, X_test, y_test = test_train_split(scaled_data, 0.8, 10)

X_train.shape, y_train.shape
```

    ((170, 10, 1), (170, 1))

```python
#create model
def create_RNN(hidden_units, dense_units, input_shape, activation):
    rmodel = Sequential()

    # recurrent layer:
    rmodel.add(SimpleRNN(hidden_units, input_shape=input_shape,
                        activation=activation[0]))
    # feedfoward layer:
    rmodel.add(Dense(units=dense_units, activation=activation[1]))

    rmodel.compile(optimizer=Adam(learning_rate=1e-3), loss=tf.keras.losses.MeanSquaredError(), metrics=[tf.keras.metrics.MeanSquaredError()])

    return rmodel

RNN_model = create_RNN(8, 1, (10,1), activation=['sigmoid', 'relu'])

RNN_model.summary()
```

    Model: "sequential"
    _________________________________________________________________
     Layer (type)                Output Shape              Param #
    =================================================================
     simple_rnn (SimpleRNN)      (None, 8)                 80

     dense (Dense)               (None, 1)                 9

    =================================================================
    Total params: 89
    Trainable params: 89
    Non-trainable params: 0
    _________________________________________________________________

    2021-11-24 16:12:34.478962: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
    2021-11-24 16:12:34.478981: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
    2021-11-24 16:12:34.478995: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (Zephyr): /proc/driver/nvidia/version does not exist
    2021-11-24 16:12:34.479293: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
    To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

```python
#train model
RNN_model.fit(X_train, y_train, epochs=20, batch_size=1, verbose=2)
```

    Epoch 1/20
    170/170 - 1s - loss: 0.2237 - mean_squared_error: 0.2237 - 554ms/epoch - 3ms/step
    Epoch 2/20
    170/170 - 0s - loss: 0.2237 - mean_squared_error: 0.2237 - 165ms/epoch - 972us/step
    Epoch 3/20
    170/170 - 0s - loss: 0.2237 - mean_squared_error: 0.2237 - 172ms/epoch - 1ms/step
    Epoch 4/20
    170/170 - 0s - loss: 0.2237 - mean_squared_error: 0.2237 - 144ms/epoch - 845us/step
    Epoch 5/20
    170/170 - 0s - loss: 0.2237 - mean_squared_error: 0.2237 - 154ms/epoch - 907us/step
    Epoch 6/20
    170/170 - 0s - loss: 0.2237 - mean_squared_error: 0.2237 - 155ms/epoch - 910us/step
    Epoch 7/20
    170/170 - 0s - loss: 0.2237 - mean_squared_error: 0.2237 - 162ms/epoch - 950us/step
    Epoch 8/20
    170/170 - 0s - loss: 0.2237 - mean_squared_error: 0.2237 - 156ms/epoch - 917us/step
    Epoch 9/20
    170/170 - 0s - loss: 0.2237 - mean_squared_error: 0.2237 - 157ms/epoch - 921us/step
    Epoch 10/20
    170/170 - 0s - loss: 0.2237 - mean_squared_error: 0.2237 - 157ms/epoch - 926us/step
    Epoch 11/20
    170/170 - 0s - loss: 0.2237 - mean_squared_error: 0.2237 - 153ms/epoch - 899us/step
    Epoch 12/20
    170/170 - 0s - loss: 0.2237 - mean_squared_error: 0.2237 - 155ms/epoch - 912us/step
    Epoch 13/20
    170/170 - 0s - loss: 0.2237 - mean_squared_error: 0.2237 - 155ms/epoch - 914us/step
    Epoch 14/20
    170/170 - 0s - loss: 0.2237 - mean_squared_error: 0.2237 - 158ms/epoch - 927us/step
    Epoch 15/20
    170/170 - 0s - loss: 0.2237 - mean_squared_error: 0.2237 - 154ms/epoch - 908us/step
    Epoch 16/20
    170/170 - 0s - loss: 0.2237 - mean_squared_error: 0.2237 - 150ms/epoch - 884us/step
    Epoch 17/20
    170/170 - 0s - loss: 0.2237 - mean_squared_error: 0.2237 - 159ms/epoch - 937us/step
    Epoch 18/20
    170/170 - 0s - loss: 0.2237 - mean_squared_error: 0.2237 - 156ms/epoch - 915us/step
    Epoch 19/20
    170/170 - 0s - loss: 0.2237 - mean_squared_error: 0.2237 - 153ms/epoch - 898us/step
    Epoch 20/20
    170/170 - 0s - loss: 0.2237 - mean_squared_error: 0.2237 - 151ms/epoch - 885us/step

    <keras.callbacks.History at 0x7fea0868bb50>

```python
#predict using test set
y_pred = RNN_model.predict(X_test)
mse = mean_squared_error(y_test, y_pred)
print('MSE: %.3f' % (mse))

y_pred = scaler.inverse_transform(y_pred).flatten()
print(y_pred)
```

    MSE: 0.544
    [116.36 116.36 116.36 116.36 116.36 116.36 116.36 116.36 116.36 116.36
     116.36 116.36 116.36 116.36 116.36 116.36 116.36 116.36 116.36 116.36
     116.36 116.36 116.36 116.36 116.36 116.36 116.36 116.36 116.36 116.36
     116.36 116.36 116.36 116.36 116.36 116.36]

```python
#visualize the predictions
y_pred = np.concatenate(([None]*(len(data)-len(y_pred)),y_pred))
result = pd.DataFrame({'predicted':y_pred, 'target':close}, index=data.index)
result.plot().set_ylabel("AAPL Closing Price ($)")
```

Reference:

https://www.ibm.com/cloud/learn/recurrent-neural-networks

http://introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L2.pdf

https://www.youtube.com/watch?v=qjrad0V0uJE

https://towardsdatascience.com/sequence-models-and-recurrent-neural-networks-rnns-62cadeb4f1e1

https://towardsdatascience.com/under-the-hood-of-neural-networks-part-2-recurrent-af091247ba78

https://medium.com/deep-math-machine-learning-ai/chapter-10-deepnlp-recurrent-neural-networks-with-math-c4a6846a50a2

https://machinelearningmastery.com/understanding-simple-recurrent-neural-networks-in-keras/

https://www.youtube.com/watch?v=p5pDBGoJmR0


# Tutorial 3: Dimensionality Reduction & Reconstruction
**Week 1, Day 5: Dimensionality Reduction**

**By Neuromatch Academy**

__Content creators:__ Alex Cayco Gajic, John Murray

__Content reviewers:__ Roozbeh Farhoudi, Matt Krause, Spiros Chavlis, Richard Gao, Michael Waskom

---
# Tutorial Objectives

In this notebook we'll learn to apply PCA for dimensionality reduction, using a classic dataset that is often used to benchmark machine learning algorithms: MNIST. We'll also learn how to use PCA for reconstruction and denoising.

Overview:
- Perform PCA on MNIST
- Calculate the variance explained
- Reconstruct data with different numbers of PCs
- (Bonus) Examine denoising using PCA

You can learn more about MNIST dataset [here](https://en.wikipedia.org/wiki/MNIST_database).

```python
# @title Video 1: PCA for dimensionality reduction
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="oO0bbInoO_0", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
```

---
# Setup
Run these cells to get the tutorial started.

```python
# Imports
import numpy as np
import matplotlib.pyplot as plt
```

```python
# @title Figure Settings
import ipywidgets as widgets   # interactive display
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")
```

```python
# @title Helper Functions

def plot_variance_explained(variance_explained):
  """
  Plots eigenvalues.

  Args:
    variance_explained (numpy array of floats) : Vector of variance explained
                                                 for each PC

  Returns:
    Nothing.

  """

  plt.figure()
  plt.plot(np.arange(1, len(variance_explained) + 1), variance_explained,
           '--k')
  plt.xlabel('Number of components')
  plt.ylabel('Variance explained')
  plt.show()

def plot_MNIST_reconstruction(X, X_reconstructed):
  """
  Plots 9 images in the MNIST dataset side-by-side with the reconstructed
  images.

  Args:
    X (numpy array of floats)               : Data matrix each column
                                              corresponds to a different
                                              random variable
    X_reconstructed (numpy array of floats) : Data matrix each column
                                              corresponds to a different
                                              random variable

  Returns:
    Nothing.
  """

  plt.figure()
  ax = plt.subplot(121)
  k = 0
  for k1 in range(3):
    for k2 in range(3):
      k = k + 1
      plt.imshow(np.reshape(X[k, :], (28, 28)),
                 extent=[(k1 + 1) * 28, k1 * 28, (k2 + 1) * 28, k2 * 28],
                 vmin=0, vmax=255)
  plt.xlim((3 * 28, 0))
  plt.ylim((3 * 28, 0))
  plt.tick_params(axis='both', which='both', bottom=False, top=False,
                  labelbottom=False)
  ax.set_xticks([])
  ax.set_yticks([])
  plt.title('Data')
  plt.clim([0, 250])
  ax = plt.subplot(122)
  k = 0
  for k1 in range(3):
    for k2 in range(3):
      k = k + 1
      plt.imshow(np.reshape(np.real(X_reconstructed[k, :]), (28, 28)),
                 extent=[(k1 + 1) * 28, k1 * 28, (k2 + 1) * 28, k2 * 28],
                 vmin=0, vmax=255)
  plt.xlim((3 * 28, 0))
  plt.ylim((3 * 28, 0))
  plt.tick_params(axis='both', which='both', bottom=False, top=False,
                  labelbottom=False)
  ax.set_xticks([])
  ax.set_yticks([])
  plt.clim([0, 250])
  plt.title('Reconstructed')
  plt.tight_layout()

def plot_MNIST_sample(X):
  """
  Plots 9 images in the MNIST dataset.

  Args:
     X (numpy array of floats) : Data matrix each column corresponds to a
                                 different random variable

  Returns:
    Nothing.

  """

  fig, ax = plt.subplots()
  k = 0
  for k1 in range(3):
    for k2 in range(3):
      k = k + 1
      plt.imshow(np.reshape(X[k, :], (28, 28)),
                 extent=[(k1 + 1) * 28, k1 * 28, (k2+1) * 28, k2 * 28],
                 vmin=0, vmax=255)
  plt.xlim((3 * 28, 0))
  plt.ylim((3 * 28, 0))
  plt.tick_params(axis='both', which='both', bottom=False, top=False,
                  labelbottom=False)
  plt.clim([0, 250])
  ax.set_xticks([])
  ax.set_yticks([])
  plt.show()

def plot_MNIST_weights(weights):
  """
  Visualize PCA basis vector weights for MNIST. Red = positive weights,
  blue = negative weights, white = zero weight.

  Args:
     weights (numpy array of floats) : PCA basis vector

  Returns:
     Nothing.
  """

  fig, ax = plt.subplots()
  cmap = plt.cm.get_cmap('seismic')
  plt.imshow(np.real(np.reshape(weights, (28, 28))), cmap=cmap)
  plt.tick_params(axis='both', which='both', bottom=False, top=False,
                  labelbottom=False)
  plt.clim(-.15, .15)
  plt.colorbar(ticks=[-.15, -.1, -.05, 0, .05, .1, .15])
  ax.set_xticks([])
  ax.set_yticks([])
  plt.show()

def add_noise(X, frac_noisy_pixels):
  """
  Randomly corrupts a fraction of the pixels by setting them to random values.

  Args:
     X (numpy array of floats)  : Data matrix
     frac_noisy_pixels (scalar) : Fraction of noisy pixels

  Returns:
     (numpy array of floats)    : Data matrix + noise

  """

  X_noisy = np.reshape(X, (X.shape[0] * X.shape[1]))
  N_noise_ixs = int(X_noisy.shape[0] * frac_noisy_pixels)
  noise_ixs = np.random.choice(X_noisy.shape[0], size=N_noise_ixs,
                               replace=False)
  X_noisy[noise_ixs] = np.random.uniform(0, 255, noise_ixs.shape)
  X_noisy = np.reshape(X_noisy, (X.shape[0], X.shape[1]))

  return X_noisy

def change_of_basis(X, W):
  """
  Projects data onto a new basis.

  Args:
    X (numpy array of floats) : Data matrix each column corresponding to a
                                different random variable
    W (numpy array of floats) : new orthonormal basis columns correspond to
                                basis vectors

  Returns:
    (numpy array of floats)   : Data matrix expressed in new basis
  """

  Y = np.matmul(X, W)

  return Y

def get_sample_cov_matrix(X):
  """
  Returns the sample covariance matrix of data X.

  Args:
    X (numpy array of floats) : Data matrix each column corresponds to a
                                different random variable

  Returns:
    (numpy array of floats)   : Covariance matrix
"""

  X = X - np.mean(X, 0)
  cov_matrix = 1 / X.shape[0] * np.matmul(X.T, X)
  return cov_matrix

def sort_evals_descending(evals, evectors):
  """
  Sorts eigenvalues and eigenvectors in decreasing order. Also aligns first two
  eigenvectors to be in first two quadrants (if 2D).

  Args:
    evals (numpy array of floats)    :   Vector of eigenvalues
    evectors (numpy array of floats) :   Corresponding matrix of eigenvectors
                                         each column corresponds to a different
                                         eigenvalue

  Returns:
    (numpy array of floats)          : Vector of eigenvalues after sorting
    (numpy array of floats)          : Matrix of eigenvectors after sorting
  """

  index = np.flip(np.argsort(evals))
  evals = evals[index]
  evectors = evectors[:, index]
  if evals.shape[0] == 2:
    if np.arccos(np.matmul(evectors[:, 0],
                           1 / np.sqrt(2) * np.array([1, 1]))) > np.pi / 2:
      evectors[:, 0] = -evectors[:, 0]
    if np.arccos(np.matmul(evectors[:, 1],
                           1 / np.sqrt(2)*np.array([-1, 1]))) > np.pi / 2:
      evectors[:, 1] = -evectors[:, 1]

  return evals, evectors

def pca(X):
  """
  Performs PCA on multivariate data. Eigenvalues are sorted in decreasing order

  Args:
     X (numpy array of floats) :   Data matrix each column corresponds to a
                                   different random variable

  Returns:
    (numpy array of floats)    : Data projected onto the new basis
    (numpy array of floats)    : Vector of eigenvalues
    (numpy array of floats)    : Corresponding matrix of eigenvectors

  """

  X = X - np.mean(X, 0)
  cov_matrix = get_sample_cov_matrix(X)
  evals, evectors = np.linalg.eigh(cov_matrix)
  evals, evectors = sort_evals_descending(evals, evectors)
  score = change_of_basis(X, evectors)

  return score, evectors, evals

def plot_eigenvalues(evals, limit=True):
  """
  Plots eigenvalues.

  Args:
     (numpy array of floats) : Vector of eigenvalues

  Returns:
    Nothing.

  """

  plt.figure()
  plt.plot(np.arange(1, len(evals) + 1), evals, 'o-k')
  plt.xlabel('Component')
  plt.ylabel('Eigenvalue')
  plt.title('Scree plot')
  if limit:
    plt.show()
```

---
# Section 1: Perform PCA on MNIST

The MNIST dataset consists of a 70,000 images of individual handwritten digits. Each image is a 28x28 pixel grayscale image. For convenience, each 28x28 pixel image is often unravelled into a single 784 (=28*28) element vector, so that the whole dataset is represented as a 70,000 x 784 matrix. Each row represents a different image, and each column represents a different pixel.

Enter the following cell to load the MNIST dataset and plot the first nine images.

```python
from sklearn.datasets import fetch_openml
mnist = fetch_openml(name='mnist_784', as_frame = False)
X = mnist.data
plot_MNIST_sample(X)
```

The MNIST dataset has an extrinsic dimensionality of 784, much higher than the 2-dimensional examples used in the previous tutorials! To make sense of this data, we'll use dimensionality reduction. But first, we need to determine the intrinsic dimensionality $K$ of the data. One way to do this is to look for an "elbow" in the scree plot, to determine which eigenvalues are signficant.

## Exercise 1: Scree plot of MNIST

In this exercise you will examine the scree plot in the MNIST dataset.

**Steps:**
- Perform PCA on the dataset and examine the scree plot.
- When do the eigenvalues appear (by eye) to reach zero? (**Hint:** use `plt.xlim` to zoom into a section of the plot).

```python
help(pca)
help(plot_eigenvalues)
```

```python
#################################################
## TO DO for students: perform PCA and plot the eigenvalues
#################################################

# perform PCA
# score, evectors, evals = ...
# plot the eigenvalues
# plot_eigenvalues(evals, limit=False)
# plt.xlim(...)  # limit x-axis up to 100 for zooming
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D5_DimensionalityReduction/solutions/W1D5_Tutorial3_Solution_a876e927.py)

*Example output:*

---
# Section 2: Calculate the variance explained

The scree plot suggests that most of the eigenvalues are near zero, with fewer than 100 having large values. Another common way to determine the intrinsic dimensionality is by considering the variance explained. This can be examined with a cumulative plot of the fraction of the total variance explained by the top $K$ components, i.e.,

\begin{equation}
\text{var explained} = \frac{\sum_{i=1}^K \lambda_i}{\sum_{i=1}^N \lambda_i}
\end{equation}

The intrinsic dimensionality is often quantified by the $K$ necessary to explain a large proportion of the total variance of the data (often a defined threshold, e.g., 90%).

## Exercise 2: Plot the explained variance

In this exercise you will plot the explained variance.

**Steps:**
- Fill in the function below to calculate the fraction variance explained as a function of the number of principal componenets. **Hint:** use `np.cumsum`.
- Plot the variance explained using `plot_variance_explained`.

**Questions:**
- How many principal components are required to explain 90% of the variance?
- How does the intrinsic dimensionality of this dataset compare to its extrinsic dimensionality?

```python
help(plot_variance_explained)
```

```python
def get_variance_explained(evals):
  """
  Calculates variance explained from the eigenvalues.

  Args:
    evals (numpy array of floats) : Vector of eigenvalues

  Returns:
    (numpy array of floats)       : Vector of variance explained

  """

  #################################################
  ## TO DO for students: calculate the explained variance using the equation
  ## from Section 2.
  # Comment once you've filled in the function
  raise NotImplementedError("Student excercise: calculate explaine variance!")
  #################################################

  # cumulatively sum the eigenvalues
  csum = ...
  # normalize by the sum of eigenvalues
  variance_explained = ...

  return variance_explained

#################################################
## TO DO for students: call the function and plot the variance explained
#################################################

# calculate the variance explained
variance_explained = ...

# Uncomment to plot the variance explained
# plot_variance_explained(variance_explained)
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D5_DimensionalityReduction/solutions/W1D5_Tutorial3_Solution_0f5f51b9.py)

*Example output:*

---
# Section 3: Reconstruct data with different numbers of PCs

```python
# @title Video 2: Data Reconstruction
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="ZCUhW26AdBQ", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
```

Now we have seen that the top 100 or so principal components of the data can explain most of the variance. We can use this fact to perform *dimensionality reduction*, i.e., by storing the data using only 100 components rather than the samples of all 784 pixels. Remarkably, we will be able to reconstruct much of the structure of the data using only the top 100 components. To see this, recall that to perform PCA we projected the data $\bf X$ onto the eigenvectors of the covariance matrix:
\begin{equation}
\bf S = X W
\end{equation}
Since $\bf W$ is an orthogonal matrix, ${\bf W}^{-1} = {\bf W}^T$. So by multiplying by ${\bf W}^T$ on each side we can rewrite this equation as
\begin{equation}
{\bf X = S W}^T.
\end{equation}
This now gives us a way to reconstruct the data matrix from the scores and loadings. To reconstruct the data from a low-dimensional approximation, we just have to truncate these matrices.  Let's call ${\bf S}_{1:K}$ and ${\bf W}_{1:K}$ as keeping only the first $K$ columns of this matrix. Then our reconstruction is:
\begin{equation}
{\bf \hat X = S}_{1:K} ({\bf W}_{1:K})^T.
\end{equation}

## Exercise 3: Data reconstruction

Fill in the function below to reconstruct the data using different numbers of principal components.

**Steps:**

* Fill in the following function to reconstruct the data based on the weights and scores. Don't forget to add the mean!
* Make sure your function works by reconstructing the data with all $K=784$ components. The two images should look identical.

```python
help(plot_MNIST_reconstruction)
```

```python
def reconstruct_data(score, evectors, X_mean, K):
  """
  Reconstruct the data based on the top K components.

  Args:
    score (numpy array of floats)    : Score matrix
    evectors (numpy array of floats) : Matrix of eigenvectors
    X_mean (numpy array of floats)   : Vector corresponding to data mean
    K (scalar)                       : Number of components to include

  Returns:
    (numpy array of floats)          : Matrix of reconstructed data

  """

  #################################################
  ## TO DO for students: Reconstruct the original data in X_reconstructed
  # Comment once you've filled in the function
  raise NotImplementedError("Student excercise: reconstructing data function!")
  #################################################

  # Reconstruct the data from the score and eigenvectors
  # Don't forget to add the mean!!
  X_reconstructed =  ...

  return X_reconstructed

K = 784

#################################################
## TO DO for students: Calculate the mean and call the function, then plot
## the original and the recostructed data
#################################################

# Reconstruct the data based on all components
X_mean = ...
X_reconstructed = ...

# Plot the data and reconstruction
# plot_MNIST_reconstruction(X, X_reconstructed)
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D5_DimensionalityReduction/solutions/W1D5_Tutorial3_Solution_e3395916.py)

*Example output:*

## Interactive Demo: Reconstruct the data matrix using different numbers of PCs

Now run the code below and experiment with the slider to reconstruct the data matrix using different numbers of principal components.

**Steps**
* How many principal components are necessary to reconstruct the numbers (by eye)? How does this relate to the intrinsic dimensionality of the data?
* Do you see any information in the data with only a single principal component?

```python
# @title

# @markdown Make sure you execute this cell to enable the widget!

def refresh(K=100):
  X_reconstructed = reconstruct_data(score, evectors, X_mean, K)
  plot_MNIST_reconstruction(X, X_reconstructed)
  plt.title('Reconstructed, K={}'.format(K))

_ = widgets.interact(refresh, K=(1, 784, 10))
```

## Exercise 4: Visualization of the weights

Next, let's take a closer look at the first principal component by visualizing its corresponding weights.

**Steps:**

* Enter `plot_MNIST_weights` to visualize the weights of the first basis vector.
* What structure do you see? Which pixels have a strong positive weighting? Which have a strong negative weighting? What kinds of images would this basis vector differentiate?
* Try visualizing the second and third basis vectors. Do you see any structure? What about the 100th basis vector? 500th? 700th?

```python
help(plot_MNIST_weights)
```

```python
#################################################
## TO DO for students: plot the weights calling the plot_MNIST_weights function
#################################################

# Plot the weights of the first principal component
# plot_MNIST_weights(...)
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D5_DimensionalityReduction/solutions/W1D5_Tutorial3_Solution_f358e413.py)

*Example output:*

---
# Summary
* In this tutorial, we learned how to use PCA for dimensionality reduction by selecting the top principal components. This can be useful as the intrinsic dimensionality ($K$) is often less than the extrinsic dimensionality ($N$) in neural data. $K$ can be inferred by choosing the number of eigenvalues necessary to capture some fraction of the variance.
* We also learned how to reconstruct an approximation of the original data using the top $K$ principal components. In fact, an alternate formulation of PCA is to find the $K$ dimensional space that minimizes the reconstruction error.
* Noise tends to inflate the apparent intrinsic dimensionality, however the higher components reflect noise rather than new structure in the data. PCA can be used for denoising data by removing noisy higher components.
* In MNIST, the weights corresponding to the first principal component appear to discriminate between a 0 and 1. We will discuss the implications of this for data visualization in the following tutorial.

---
# Bonus: Examine denoising using PCA

In this lecture, we saw that PCA finds an optimal low-dimensional basis to minimize the reconstruction error. Because of this property, PCA can be useful for denoising corrupted samples of the data.

## Exercise 5: Add noise to the data
In this exercise you will add salt-and-pepper noise to the original data and see how that affects the eigenvalues.

**Steps:**
- Use the function `add_noise` to add noise to 20% of the pixels.
- Then, perform PCA and plot the variance explained. How many principal components are required to explain 90% of the variance? How does this compare to the original data?

```python
help(add_noise)
```

```python
###################################################################
# Insert your code here to:
# Add noise to the data
# Plot noise-corrupted data
# Perform PCA on the noisy data
# Calculate and plot the variance explained
###################################################################
np.random.seed(2020)  # set random seed
X_noisy = ...
# score_noisy, evectors_noisy, evals_noisy = ...
# variance_explained_noisy = ...
# plot_MNIST_sample(X_noisy)
# plot_variance_explained(variance_explained_noisy)
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D5_DimensionalityReduction/solutions/W1D5_Tutorial3_Solution_d4a41b8c.py)

*Example output:*

## Exercise 6: Denoising

Next, use PCA to perform denoising by projecting the noise-corrupted data onto the basis vectors found from the original dataset. By taking the top K components of this projection, we can reduce noise in dimensions orthogonal to the K-dimensional latent space.

**Steps:**
- Subtract the mean of the noise-corrupted data.
- Project the data onto the basis found with the original dataset (`evectors`, not `evectors_noisy`) and take the top $K$ components.
- Reconstruct the data as normal, using the top 50 components.
- Play around with the amount of noise and K to build intuition.

```python
###################################################################
# Insert your code here to:
# Subtract the mean of the noise-corrupted data
# Project onto the original basis vectors evectors
# Reconstruct the data using the top 50 components
# Plot the result
###################################################################

X_noisy_mean = ...
projX_noisy = ...
X_reconstructed = ...
# plot_MNIST_reconstruction(X_noisy, X_reconstructed)
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D5_DimensionalityReduction/solutions/W1D5_Tutorial3_Solution_e3ee8262.py)

*Example output:*


```python
%matplotlib inline
```

Learning Hybrid Frontend Syntax Through Example
===============================================
**Author:** `Nathan Inkawhich <https://github.com/inkawhich>`_

This document is meant to highlight the syntax of the Hybrid Frontend
through a non-code intensive example. The Hybrid Frontend is one of the
new shiny features of Pytorch 1.0 and provides an avenue for developers
to transition their models from **eager** to **graph** mode. PyTorch
users are very familiar with eager mode as it provides the ease-of-use
and flexibility that we all enjoy as researchers. Caffe2 users are more
aquainted with graph mode which has the benefits of speed, optimization
opportunities, and functionality in C++ runtime environments. The hybrid
frontend bridges the gap between the the two modes by allowing
researchers to develop and refine their models in eager mode (i.e.
PyTorch), then gradually transition the proven model to graph mode for
production, when speed and resouce consumption become critical.

Hybrid Frontend Information
---------------------------

The process for transitioning a model to graph mode is as follows.
First, the developer constructs, trains, and tests the model in eager
mode. Then they incrementally **trace** and **script** each
function/module of the model with the Just-In-Time (JIT) compiler, at
each step verifying that the output is correct. Finally, when each of
the components of the top-level model have been traced and scripted, the
model itself is traced. At which point the model has been transitioned
to graph mode, and has a complete python-free representation. With this
representation, the model runtime can take advantage of high-performance
Caffe2 operators and graph based optimizations.

Before we continue, it is important to understand the idea of tracing
and scripting, and why they are separate. The goal of **trace** and
**script** is the same, and that is to create a graph representation of
the operations taking place in a given function. The discrepency comes
from the flexibility of eager mode that allows for **data-dependent
control flows** within the model architecture. When a function does NOT
have a data-dependent control flow, it may be *traced* with
``torch.jit.trace``. However, when the function *has* a data-dependent
control flow it must be *scripted* with ``torch.jit.script``. We will
leave the details of the interworkings of the hybrid frontend for
another document, but the code example below will show the syntax of how
to trace and script different pure python functions and torch Modules.
Hopefully, you will find that using the hybrid frontend is non-intrusive
as it mostly involves adding decorators to the existing function and
class definitions.

Motivating Example
------------------

In this example we will implement a strange math function that may be
logically broken up into four parts that do, and do not contain
data-dependent control flows. The purpose here is to show a non-code
intensive example where the use of the JIT is highlighted. This example
is a stand-in representation of a useful model, whose implementation has
been divided into various pure python functions and modules.

The function we seek to implement, $Y(x)$, is defined for
$x \epsilon \mathbb{N}$ as

\begin{align}z(x) = \Biggl \lfloor \frac{\sqrt{\prod_{i=1}^{|2 x|}i}}{5} \Biggr \rfloor\end{align}

\begin{align}Y(x) = \begin{cases}
      \frac{z(x)}{2}  &  \text{if } z(x)\%2 == 0, \\
      z(x)             &  \text{otherwise}
    \end{cases}\end{align}

\begin{align}\begin{array}{| r  | r |} \hline
    x &1 &2 &3 &4 &5 &6 &7 \\ \hline
    Y(x) &0 &0 &-5 &20 &190 &-4377 &-59051 \\ \hline
    \end{array}\end{align}

As mentioned, the computation is split into four parts. Part one is the
simple tensor calculation of $|2x|$, which can be traced. Part two
is the iterative product calculation that represents a data dependent
control flow to be scripted (the number of loop iteration depends on the
input at runtime). Part three is a trace-able
$\lfloor \sqrt{a/5} \rfloor$ calculation. Finally, part 4 handles
the output cases depending on the value of $z(x)$ and must be
scripted due to the data dependency. Now, let's see how this looks in
code.

Part 1 - Tracing a pure python function
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

We can implement part one as a pure python function as below. Notice, to
trace this function we call ``torch.jit.trace`` and pass in the function
to be traced. Since the trace requires a dummy input of the expected
runtime type and shape, we also include the ``torch.rand`` to generate a
single valued torch tensor.

```python
import torch

def fn(x):
    return torch.abs(2*x)

# This is how you define a traced function
# Pass in both the function to be traced and an example input to ``torch.jit.trace``
traced_fn = torch.jit.trace(fn, torch.rand(()))
```

Part 2 - Scripting a pure python function
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

We can also implement part 2 as a pure python function where we
iteratively compute the product. Since the number of iterations depends
on the value of the input, we have a data dependent control flow, so the
function must be scripted. We can script python functions simply with
the ``@torch.jit.script`` decorator.

```python
# This is how you define a script function
# Apply this decorator directly to the function
@torch.jit.script
def script_fn(x):
    z = torch.ones([1], dtype=torch.int64)
    for i in range(int(x)):
        z = z * (i + 1)
    return z
```

Part 3 - Tracing a nn.Module
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Next, we will implement part 3 of the computation within the forward
function of a ``torch.nn.Module``. This module may be traced, but rather
than adding a decorator here, we will handle the tracing where the
Module is constructed. Thus, the class definition is not changed at all.

```python
# This is a normal module that can be traced.
class TracedModule(torch.nn.Module):
    def forward(self, x):
        x = x.type(torch.float32)
        return torch.floor(torch.sqrt(x) / 5.)
```

Part 4 - Scripting a nn.Module
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In the final part of the computation we have a ``torch.nn.Module`` that
must be scripted. To accomodate this, we inherit from
``torch.jit.ScriptModule`` and add the ``@torch.jit.script_method``
decorator to the forward function.

```python
# This is how you define a scripted module.
# The module should inherit from ScriptModule and the forward should have the
#   script_method decorator applied to it.
class ScriptModule(torch.jit.ScriptModule):
    @torch.jit.script_method
    def forward(self, x):
        r = -x
        if int(torch.fmod(x, 2.0)) == 0.0:
            r = x / 2.0
        return r
```

Top-Level Module
~~~~~~~~~~~~~~~~

Now we will put together the pieces of the computation via a top level
module called ``Net``. In the constructor, we will instantiate the
``TracedModule`` and ``ScriptModule`` as attributes. This must be done
because we ultimately want to trace/script the top level module, and
having the traced/scripted modules as attributes allows the Net to
inherit the required submodules' parameters. Notice, this is where we
actually trace the ``TracedModule`` by calling ``torch.jit.trace()`` and
providing the necessary dummy input. Also notice that the
``ScriptModule`` is constructed as normal because we handled the
scripting in the class definition.

Here we can also print the graphs created for each individual part of
the computation. The printed graphs allows us to see how the JIT
ultimately interpreted the functions as graph computations.

Finally, we define the ``forward`` function for the Net module where we
run the input data ``x`` through the four parts of the computation.
There is no strange syntax here and we call the traced and scripted
modules and functions as expected.

```python
# This is a demonstration net that calls all of the different types of
# methods and functions
class Net(torch.nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        # Modules must be attributes on the Module because if you want to trace
        # or script this Module, we must be able to inherit the submodules'
        # params.
        self.traced_module = torch.jit.trace(TracedModule(), torch.rand(()))
        self.script_module = ScriptModule()

        print('traced_fn graph', traced_fn.graph)
        print('script_fn graph', script_fn.graph)
        print('TracedModule graph', self.traced_module.__getattr__('forward').graph)
        print('ScriptModule graph', self.script_module.__getattr__('forward').graph)

    def forward(self, x):
        # Call a traced function
        x = traced_fn(x)

        # Call a script function
        x = script_fn(x)

        # Call a traced submodule
        x = self.traced_module(x)

        # Call a scripted submodule
        x = self.script_module(x)

        return x
```

Running the Model
~~~~~~~~~~~~~~~~~

All that's left to do is construct the Net and compute the output
through the forward function. Here, we use $x=5$ as the test input
value and expect $Y(x)=190.$ Also, check out the graphs that were
printed during the construction of the Net.

```python
# Instantiate this net and run it
n = Net()
print(n(torch.tensor([5]))) # 190.
```

Tracing the Top-Level Model
~~~~~~~~~~~~~~~~~~~~~~~~~~~

The last part of the example is to trace the top-level module, ``Net``.
As mentioned previously, since the traced/scripted modules are
attributes of Net, we are able to trace ``Net`` as it inherits the
parameters of the traced/scripted submodules. Note, the syntax for
tracing Net is identical to the syntax for tracing ``TracedModule``.
Also, check out the graph that is created.

```python
n_traced = torch.jit.trace(n, torch.tensor([5]))
print(n_traced(torch.tensor([5])))
print('n_traced graph', n_traced.graph)
```

Hopefully, this document can serve as an introduction to the hybrid
frontend as well as a syntax reference guide for more experienced users.
Also, there are a few things to keep in mind when using the hybrid
frontend. There is a constraint that traced/scripted methods must be
written in a restricted subset of python, as features like generators,
defs, and Python data structures are not supported. As a workaround, the
scripting model *is* designed to work with both traced and non-traced
code which means you can call non-traced code from traced functions.
However, such a model may not be exported to ONNX.