# Combining measurements

When we do a fit, we can have additional knowledge about a parameter from other measurements. This can be taken into account either through a simultaneous fit or by adding a constraint (subsidiary measurement).

## Adding a constraint

If we know a parameters value from a different measurement and want to constraint this using its uncertainty, a Gaussian constraint can be added to the likelihood as

\begin{align}
\mathrm{constr_i} = \mathrm{Gauss}(\mu_{measured}; \theta_i, \sigma_{measured})
\end{align}

In general, additional terms can be added to the likelihood arbitrarily in zfit, be it to incorporate other shaped measurements or to add penalty terms to confine a fit within boundaries.

```python
import hepunits as u
import matplotlib.pyplot as plt
import mplhep
import numpy as np
import particle.literals as lp
import tensorflow as tf
import zfit

plt.rcParams['figure.figsize'] = (8,6)
```

```python
mu_true =  lp.B_plus.mass * u.MeV
sigma_true = 50 * u.MeV

# number of signal and background
n_sig_rare = 120
n_bkg_rare = 700
```

```python
# create some data
signal_np = np.random.normal(loc=mu_true, scale=sigma_true, size=n_sig_rare)
bkg_np_raw = np.random.exponential(size=20000, scale=700)
bkg_np = bkg_np_raw[bkg_np_raw<1000][:n_bkg_rare] + 5000  # just cutting right, but zfit could also cut
```

```python
# Firstly, the observable and its range is defined
obs = zfit.Space('Bmass', (5000, 6000))  # for whole range
```

```python
# load data into zfit
data = zfit.Data.from_numpy(obs=obs, array=np.concatenate([signal_np, bkg_np], axis=0))
```

```python
# Parameters are specified:  (name (unique), initial, lower, upper) whereas lower, upper are optional
mu = zfit.Parameter('mu', 5279, 5100, 5400)
sigma = zfit.Parameter('sigma', 20, 1, 200)
signal = zfit.pdf.Gauss(mu=mu, sigma=sigma, obs=obs)

lam = zfit.Parameter('lambda', -0.002, -0.1, -0.00001, step_size=0.001)  # floating, also without limits
comb_bkg = zfit.pdf.Exponential(lam, obs=obs)

sig_yield = zfit.Parameter('sig_yield', n_sig_rare + 30,
                                step_size=3)  # step size: default is small, use appropriate
bkg_yield = zfit.Parameter('bkg_yield', n_bkg_rare - 40, step_size=1)
# Create extended PDFs
extended_sig = signal.create_extended(sig_yield)
extended_bkg = comb_bkg.create_extended(bkg_yield)

# The final model is the combination of the signal and backgrond PDF
model = zfit.pdf.SumPDF([extended_bkg, extended_sig])
```

```python
constraint = zfit.constraint.GaussianConstraint(mu, observation=5275 * u.MeV, uncertainty=15 * u.MeV)
```

```python
nll = zfit.loss.ExtendedUnbinnedNLL(model, data, constraints=constraint)
minimizer = zfit.minimize.Minuit(use_minuit_grad=True)
result = minimizer.minimize(nll)
result.hesse();
```

```python
print(result.params)
```

## Simultaneous fits

Sometimes, we don't want to fit a single channel but multiple with the same likelihood and having shared parameters between them. In this example, we will fit the decay simultaneously to its resonant control channel.

A simultaneous likelihood is the product of different likelihoods defined by
\begin{align}
\mathcal{L}_{f(x)}(\theta | {data_0, data_1, ..., data_n}) &= \prod_{i=1}^{n} \mathcal{L}(\theta_i, data_i)
\end{align}

and becomes in the NLL a sum

\begin{align}
\mathrm{NLL}_{f(x)}(\theta | {data_0, data_1, ..., data_n}) &= \sum_{i=1}^{n} \mathrm{NLL}(\theta_i, data_i)
\end{align}

```python
n_sig_reso = 40000
n_bkg_reso = 3000
```

```python
# create some data
signal_np_reso = np.random.normal(loc=mu_true, scale=sigma_true * 0.7, size=n_sig_reso)
bkg_np_raw_reso = np.random.exponential(size=20000, scale=900)
bkg_np_reso = bkg_np_raw_reso[bkg_np_raw_reso<1000][:n_bkg_reso] + 5000

# load data into zfit
obs_reso = zfit.Space('Bmass_reso', (5000, 6000))
data_reso = zfit.Data.from_numpy(obs=obs_reso, array=np.concatenate([signal_np_reso, bkg_np_reso], axis=0))
```

### Sharing and composing parameters

As an example, the same signal shape will be used with the identical `mu` yet a scaled `sigma`. This means that the `sigma` of our control mode corresponds to the `sigma` of our signal times a scaling parameter. Therefore, the `scaled_sigma` is a function of two other parameters, `sigma` and `sigma_scaling` and _cannot_ change it's value independently. There are two fundamentally distinct types to represent this behavior in zfit, independent (`Parameter`) and dependent parameters (`ComplexParameter`, `ComposedParameter`,...)

#### Independent parameter

An independent parameter has, as a distinctive method, a `set_value` that _changes_ the value of the parameter. In a fit, or in general, these are _the only object_ that can directly change their value and therefore do not depend on other objects while most other objects depend on Parameters.
As a consequence, this parameters can have limits (which effectively restrict the possible values a `Parameter` can be assigned to) and have a `step_size`, a hint to any minimization algorithm about the order of magnitude that a change in the parameter will have on the loss.

Another attribute is a `floating` flag: if set to `False`, the parameter won't be floating in the fit, whether explicitly given or implicitly inferred from the dependencies.

#### Dependent parameter

These are single-valued functions effectively that depend on other objects, usually other parameters. Therefore, a dependent parameter does not have a `set_value` function and also does not posses limits. The latter is preferred to be set with the `Parameter` it depends on, however, if a hard limit is required, this can always be enforced in the definition of a `ComposedParameter`.

The most notable parameter is the `ComposedParameter`, which returns an arbitrary function of its input arguments, the latter which can be specified with the `params` argument.

While this parameters *cannot* change there value explicitly and therefore won't be used by a minimizer, the zfit minimizers automatically extract the independent parameters that a dependent parameter depends on (if this is given as an argument.)

As a consequence, these parameters also miss a `step_size` attribute. Furthermore, `floating` can't be used, neither set nor retrieved; it is rather advised to check directly with its dependencies.

#### Sharing parameters

Since in zfit, every parameter object is unique, also defined by its name, it is straightforward to know when a parameter is shared in the loss and when it is not: if the same object is used in two places, it is shared. This can be arbitrarily mixed.

```python
# Firstly, we create a free scaling parameter
sigma_scaling = zfit.Parameter('sigma_scaling', 0.9, 0.1, 10, step_size=0.1)

def sigma_scaled_fn(sigma, sigma_scaling):
    return sigma * sigma_scaling  # this can be an arbitrary function

sigma_scaled = zfit.ComposedParameter('sigma scaled',  # name
                                      sigma_scaled_fn,  # function
                                      params=[sigma, sigma_scaling]  # the objects used inside the function
                                      )
```

```python
signal_reso = zfit.pdf.Gauss(mu=mu,  # the same as for the rare mode
                             sigma=sigma_scaled,
                             obs=obs_reso
                            )

lambda_reso = zfit.Parameter('lambda_reso', -0.002, -0.01, 0.0001)  # floating
comb_bkg_reso_pdf = zfit.pdf.Exponential(lambda_reso, obs=obs_reso)

reso_sig_yield = zfit.Parameter('reso_sig_yield', n_sig_reso - 100, 0, n_sig_reso * 3,
                                step_size=1)  # step size: default is small, use appropriate
reso_bkg_yield = zfit.Parameter('reso_bkg_yield', n_bkg_reso + 70, 0, 2e5, step_size=1)

# Create the extended models
extended_sig_reso = signal_reso.create_extended(reso_sig_yield)
extended_bkg_reso = comb_bkg_reso_pdf.create_extended(reso_bkg_yield)
model_reso = zfit.pdf.SumPDF([extended_bkg_reso, extended_sig_reso])
```

To implement the simultaneous fit, there are two ways to achieve this in zfit. As an important distinction to other frameworks, zfit translates the above equation
\begin{align}
\mathrm{NLL}_{f(x)}(\theta | {data_0, data_1, ..., data_n}) &= \sum_{i=1}^{n} \mathrm{NLL}(\theta_i, data_i)
\end{align}

directly into code.

We can build two losses and add them directly or give a list of models and data, which build a loss each one and add up.

```python
nll_rare = zfit.loss.ExtendedUnbinnedNLL(model, data)
nll_reso = zfit.loss.ExtendedUnbinnedNLL(model_reso, data_reso)
nll_simultaneous = nll_rare + nll_reso
```

```python
result_simultaneous = minimizer.minimize(nll_simultaneous)
```

```python
result_simultaneous.hesse()
```

```python
print(result_simultaneous.params)
```

### Plotting a simultaneous loss

Since the definition of a simultaneous fit is as above, it is simple to plot each component separately: either my using the attributes of the loss to access the models and plot in a general fashion or directly reuse the model and data from before; we created them manually before.

```python
# Sets the values of the parameters to the result of the simultaneous fit
# in case they were modified.
zfit.param.set_values(nll_simultaneous.get_params(), result_simultaneous)

def plot_fit_projection(model, data, nbins=30, ax=None):
    # The function will be reused.
    if ax is None:
        ax = plt.gca()

    lower, upper = data.data_range.limit1d

    # Creates and histogram of the data and plots it with mplhep.
    counts, bin_edges = np.histogram(data.unstack_x(), bins=nbins)
    mplhep.histplot(counts, bins=bin_edges, histtype="errorbar", yerr=True,
                    label="Data", ax=ax, color="black")

    binwidth = np.diff(bin_edges)[0]
    x = tf.linspace(lower, upper, num=1000)  # or np.linspace

    # Line plots of the total pdf and the sub-pdfs.
    y = model.ext_pdf(x) * binwidth
    ax.plot(x, y, label="total", color="royalblue")
    for m, l, c in zip(model.get_models(), ["background", "signal"], ["forestgreen", "crimson"]):
        ym = m.ext_pdf(x) * binwidth
        ax.plot(x, ym, label=l, color=c)

    ax.set_title(data.data_range.obs[0])
    ax.set_xlim(lower, upper)
    ax.legend(fontsize=15)

    return ax

fig, axs = plt.subplots(1, 2, figsize=(16, 6))

for mod, dat, ax, nb in zip(nll_simultaneous.model, nll_simultaneous.data, axs, [30, 60]):
    plot_fit_projection(mod, dat, nbins=nb, ax=ax)
```

## Discovery test

We observed an excess of our signal:

```python
print(result_simultaneous.params[sig_yield])
```

Now we would like to compute the significance of this observation or in other words the probabilty that this observation is the result of the statistical fluctuation. To do so we have to perform an hypothesis test where the null and alternative hypotheses are defined as:

* $H_{0}$, the null or background only hypothesis, i.e. $N_{sig} = 0$;
* $H_{1}$, the alternative hypothesis, i.e $N_{sig} = \hat{N}_{sig}$, where $\hat{N}_{sig}$ is the fitted value of $N_{sig}$ printed above.

In `hepstats` to formulate a hypothesis you have to use the `POI` (Parameter Of Interest) class.

```python
from hepstats.hypotests.parameters import POI

# the null hypothesis
sig_yield_poi = POI(sig_yield, 0)
```

What the `POI` class does is to take as input a `zfit.Parameter` instance and a value corresponding to a given hypothesis. You can notice that we didn't define here the alternative hypothesis as in the discovery test the value of POI for alternate is set to the best fit value.

The test statistic used is the profile likelihood ratio and defined as:

\begin{equation}
q_{0} = \left\{
    \begin{array}{ll}
        -2 \ln \frac{\mathcal{L}(N_{sig}=0, \; \hat{\hat{\theta}})}{\mathcal{L}(N_{sig}=\hat{N}_{sig}, \; \hat{\theta})} & \mbox{if } \; \hat{N}_{sig} \geq 0 \\
        0 & \mbox{if } \; \hat{N}_{sig} < 0
    \end{array}
\right.
\end{equation}

where $\hat{\theta}$ are the best fitted values of the nuisances parameters (i.e. background yield, exponential slope...), while $\hat{\hat{\theta}}$ are the fitted values of the nuisances when ${N}_{sig} = 0$.

From the test statistic distribution a p-value can computed as

\begin{equation}
p_{0} = \int_{q_{0}^{obs}}^{\infty} f(q_{0} |H_{0}) dq_{0}
\end{equation}

where $q_{0}^{obs}$ is the value of the test statistic evaluated on observed data.

The construction of the test statistic and the computation of the p-value is done in a `Calculator` object in `hepstats`. In this example we will use in this example the `AsymptoticCalculator` calculator which assumes that $q_{0}$ follows a $\chi^2(ndof=1)$ which simplifies the p-value computation to

\begin{equation}
p_{0} = 1 - \Phi\bigg({\sqrt{q_{0}^{obs}}}\bigg).
\end{equation}

The calculator objects takes as input the likelihood function and a minimizer to profile the likelihood.

```python
from hepstats.hypotests.calculators import AsymptoticCalculator

# construction of the calculator instance
calculator = AsymptoticCalculator(input=nll_simultaneous, minimizer=minimizer)
calculator.bestfit = result_simultaneous

# equivalent to above
calculator = AsymptoticCalculator(input=result_simultaneous, minimizer=minimizer)
```

There is another calculator in `hepstats` called `FrequentistCalculator` which constructs the test statistic distribution $f(q_{0} |H_{0})$ with pseudo-experiments (toys), but it takes more time.

The `Discovery` class is a high-level class that takes as input a calculator and a `POI` instance representing the null hypothesis, it basically asks the calculator to compute the p-value and also computes the signifance as

\begin{equation}
Z = \Phi^{-1}(1 - p_0).
\end{equation}

```python
from hepstats.hypotests import Discovery

discovery = Discovery(calculator=calculator, poinull=sig_yield_poi)
discovery.result()
```

So we get a significance of about $7\sigma$ which is well above the $5 \sigma$ threshold for discoveries 😃.

## Upper limit calculation

Let's try to compute the discovery significance with a lower number of generated signal events.

```python
# Sets the values of the parameters to the result of the simultaneous fit
zfit.param.set_values(nll_simultaneous.get_params(), result_simultaneous)
sigma_scaling.floating=False

# Creates a sampler that will draw events from the model
sampler = model.create_sampler()

# Creates new simultaneous loss
nll_simultaneous_low_sig = zfit.loss.ExtendedUnbinnedNLL(model, sampler) + nll_reso
```

```python
# Samples with sig_yield = 10. Since the model is extended the number of
# signal generated is drawn from a poisson distribution with lambda = 10.
sampler.resample({sig_yield: 10})
```

```python
calculator_low_sig = AsymptoticCalculator(input=nll_simultaneous_low_sig, minimizer=minimizer)

discovery_low_sig = Discovery(calculator=calculator_low_sig, poinull=sig_yield_poi)
discovery_low_sig.result()
print(f"\n {calculator_low_sig.bestfit.params} \n")
```

We might consider computing an upper limit on the signal yield instead. The test statistic for an upper limit calculation is

\begin{equation}
q_{N_{sig}} = \left\{
    \begin{array}{ll}
        -2 \ln \frac{\mathcal{L}(N_{sig}, \; \hat{\hat{\theta}})}{\mathcal{L}(N_{sig}=\hat{N}_{sig}, \; \hat{\theta})} & \mbox{if } \; \hat{N}_{sig} \leq N_{sig} \\
        0 & \mbox{if } \; \hat{N}_{sig} > N_{sig}.
    \end{array}
\right.
\end{equation}

and the p-value is

\begin{equation}
p_{N_{sig}} = \int_{q_{N_{sig}}^{obs}}^{\infty} f(q_{N_{sig}} |N_{sig}) dq_{N_{sig}}.
\end{equation}

The upper limit on $N_{sig}$, $N_{sig, \uparrow}$ is found for $p_{N_{sig, \uparrow}} = 1 - \alpha$, $\alpha$ being the confidence level (typically $95 \%$). The upper limit is found by interpolation of the p-values as a function of $N_{sig}$, which is done the `UpperLimit` class. We have to give the range of values of $N_{sig}$ to scan using the `POIarray` class which as the `POI` class takes as input the parameter but takes several values to evaluate the parameter instead of one.

```python
from hepstats.hypotests import UpperLimit
from hepstats.hypotests.parameters import POIarray

# Background only hypothesis.
bkg_only = POI(sig_yield, 0)
# Range of Nsig values to scan.
sig_yield_scan = POIarray(sig_yield, np.linspace(0, 70, 10))

ul = UpperLimit(calculator=calculator_low_sig, poinull=sig_yield_scan, poialt=bkg_only)
ul.upperlimit(alpha=0.05);
```

```python
from utils import plotlimit

plotlimit(ul, CLs=False)
```

## Splot

This is now an demonstration of the **sPlot** algorithm, described in [Pivk:2004ty](https://arxiv.org/pdf/physics/0402083.pdf).

If a data sample is populated by different sources of events, like signal and background, **sPlot** is able to unfold the contributions of the different sources for a given variable.

Let's construct a dataset with two variables, the invariant mass and lifetime, for the resonant signal defined above and the combinatorial background.

```python
# Signal distributions.
nsig_sw = 20000
np_sig_m_sw = signal_reso.sample(nsig_sw).numpy().reshape(-1,)
np_sig_t_sw = np.random.exponential(size=nsig_sw, scale=1)

# Background distributions.
nbkg_sw = 150000
np_bkg_m_sw = comb_bkg_reso_pdf.sample(nbkg_sw).numpy().reshape(-1,)
np_bkg_t_sw = np.random.normal(size=nbkg_sw, loc=2.0, scale=2.5)

# Lifetime cut.
t_cut = np_bkg_t_sw > 0
np_bkg_t_sw = np_bkg_t_sw[t_cut]
np_bkg_m_sw = np_bkg_m_sw[t_cut]

# Mass distribution
np_m_sw = np.concatenate([np_sig_m_sw, np_bkg_m_sw])

# Lifetime distribution
np_t_sw = np.concatenate([np_sig_t_sw, np_bkg_t_sw])

# Plots the mass and lifetime distribution.
fig, axs = plt.subplots(1, 2, figsize=(16, 6))
axs[0].hist([np_bkg_m_sw, np_sig_m_sw], bins=50, stacked=True, label=("background", "signal"), alpha=.7)
axs[0].set_xlabel("m")
axs[0].legend(fontsize=15)
axs[1].hist([np_bkg_t_sw, np_sig_t_sw], bins=50, stacked=True, label=("background", "signal"), alpha=.7)
axs[1].set_xlabel("t")
axs[1].legend(fontsize=15);
```

In this particular example we want to unfold the signal lifetime distribution. To do so **sPlot** needs a discriminant variable to determine the yields of the various sources using an extended maximum likelihood fit.

```python
# Builds the loss.
data_sw = zfit.Data.from_numpy(obs=obs_reso, array=np_m_sw)
nll_sw = zfit.loss.ExtendedUnbinnedNLL(model_reso, data_sw)

# This parameter was useful in the simultaneous fit but not anymore so we fix it.
sigma_scaling.floating=False

# Minimizes the loss.
result_sw = minimizer.minimize(nll_sw)
print(result_sw.params)
```

```python
# Visualization of the result.
zfit.param.set_values(nll_sw.get_params(), result_sw)
plot_fit_projection(model_reso, data_sw, nbins=100)
```

**sPlot** will use the fitted yield for each sources to derive the so-called **sWeights** for each data point:

\begin{equation}
W_{n}(x) = \frac{\sum_{j=1}^{N_S} V_{nj} f_j(x)}{\sum_{k=1}^{N_S} N_{k}f_k(x)}
\end{equation}

with

\begin{equation}
V_{nj}^{-1} = \sum_{e=1}^{N} \frac{f_n(x_e) f_j(x_e)}{(\sum_{k=1}^{N_S} N_{k}f_k(x))^2}
\end{equation}

where ${N_S}$ is the number of sources in the data sample, here 2. The index $n$ represents the source, for instance $0$ is the signal and $1$ is the background, then $f_0$ and $N_0$ are the pdf and yield for the signal.

In `hepstats` the **sWeights** are computed with the `compute_sweights` function which takes as arguments the fitted extended model and the discrimant data (on which the fit was performed).

```python
from hepstats.splot import compute_sweights

weights = compute_sweights(model_reso, data_sw)

print(weights)
```

```python
print("Sum of signal sWeights: ", np.sum(weights[reso_sig_yield]))
print("Sum of background sWeights: ", np.sum(weights[reso_bkg_yield]))
```

Now we can apply the signal **sWeights** on the lifetime distribution and retrieve its signal components.

```python
fig, axs = plt.subplots(1, 2, figsize=(16, 6))
nbins = 40

sorter = np_m_sw.argsort()

axs[0].plot(np_m_sw[sorter], weights[reso_sig_yield][sorter], label="$w_\\mathrm{sig}$")
axs[0].plot(np_m_sw[sorter], weights[reso_bkg_yield][sorter], label="$w_\\mathrm{bkg}$")
axs[0].plot(np_m_sw[sorter], weights[reso_sig_yield][sorter] + weights[reso_bkg_yield][sorter],
            "-k", label="$w_\\mathrm{sig} + w_\\mathrm{bkg}$")
axs[0].axhline(0, color="0.5")
axs[0].legend(fontsize=15)
axs[0].set_xlim(5000, 5600)

axs[1].hist(np_t_sw, bins=nbins, range=(0, 6), weights=weights[reso_sig_yield], label="weighted histogram", alpha=.5)
axs[1].hist(np_sig_t_sw, bins=nbins, range=(0, 6), histtype="step", label="true histogram", lw=1.5)
axs[1].set_xlabel("t")
axs[1].legend(fontsize=15);
```

Be careful the **sPlot** technique works only on variables that are uncorrelated with the discriminant variable.

```python
print(f"Correlation between m and t: {np.corrcoef(np_m_sw, np_t_sw)[0, 1]}")
```

Let's apply to signal **sWeights** on the mass distribution to see how bad the results of **sPlot** is when applied on a variable that is correlated with the discrimant variable.

```python
plt.hist(np_m_sw, bins=100, range=(5000, 6000), weights=weights[reso_sig_yield]);
```

```python

```


# One-dimensional advection equation

We want to solve the following PDE:

\begin{equation}
    \frac{\partial \phi}{\partial t} + u \frac{\partial \phi}{\partial x} = 0
\end{equation}

The independen variables (i.e, $x$ and $t$) are used as input values for the NN, and the solution (i.e. $\phi$) is the output. In order to find the solution, at each step the NN outputs are derived w.r.t the inputs. Then, a loss function that matches the PDE is built and the weights are updated accordingly. If the loss function goes to zero, we can assume that our NN is indeed the solution to our PDE. We will try to find a general solution for different values of $u$, so it will be set also as an input.

```
# autoreload nangs
%reload_ext autoreload
%autoreload 2
%matplotlib inline
```

```
# imports
import math
import numpy as np
import matplotlib.pyplot as plt

import torch
cuda = True
device = "cuda" if torch.cuda.is_available() and cuda else "cpu"
```

## Define PDE

First we define our PDE and set the values for training.

```
# imports
from nangs.pde import PDE
from nangs.bocos import PeriodicBoco, DirichletBoco
from nangs.solutions import MLP

# define our PDE
class MyPDE(PDE):
    def __init__(self, inputs=None, outputs=None):
        super().__init__(inputs, outputs)
    # the loss of our NN is the PDE !
    def computePDELoss(self, grads, inputs, outputs, params):
        dpdt, dpdx = grads['p']['t'], grads['p']['x']
        u = inputs['u']
        return [dpdt + u*dpdx]

# instanciate pde with keys for inputs/outputs
pde = MyPDE(inputs=['x', 't', 'u'], outputs=['p'])

# define input values for training
x = np.linspace(0,1,20)
t = np.linspace(0,1,30)
u = np.linspace(0,1,10)
pde.setValues({'x': x, 't': t[:-1], 'u': u})

# periodic b.c for the space dimension
boco = PeriodicBoco('boco', {'x': x[:1], 't': t[:-1], 'u': u}, {'x': x[-1:], 't': t[:-1], 'u': u})
pde.addBoco(boco)

# initial condition (dirichlet for temporal dimension)
p00, p0 = np.sin(2.*math.pi*x), np.array([])
for i in u:
    p0 = np.concatenate((p0,p00)) # one for each value of 'u', keeping the order (x, t, u)
boco = DirichletBoco('initial_condition', {'x': x, 't': t[:1], 'u': u}, {'p': p0})
pde.addBoco(boco)

# define input values for validation
x_v = np.linspace(0,1,25)
t_v = np.linspace(0,1,15)
u_v = np.linspace(0,1,5)
pde.setValues({'x': x_v, 't': t_v[:-1], 'u': u_v}, train=False)
```

Now we define a topology for our solution and set the training parameters. Then we can find a solution for our PDE.

```
# define solution topology
mlp = MLP(pde.n_inputs, pde.n_outputs, 5, 2048)
optimizer = torch.optim.Adam(mlp.parameters(), lr=3e-4)
pde.compile(mlp, optimizer)
```

```
# find the solution
hist = pde.solve()
```

Epoch 1/30 Losses 0.16774
 PDE  [ 0.02709 ]
 boco 0.03435
 initial_condition 0.10630
 Val [ 0.00943 ]
Epoch 2/30 Losses 0.03078
 PDE  [ 0.00818 ]
 boco 0.01269
 initial_condition 0.00991
 Val [ 0.00743 ]
Epoch 3/30 Losses 0.02054
 PDE  [ 0.00712 ]
 boco 0.00877
 initial_condition 0.00465
 Val [ 0.00916 ]
Epoch 4/30 Losses 0.01845
 PDE  [ 0.00665 ]
 boco 0.00757
 initial_condition 0.00423
 Val [ 0.00574 ]
Epoch 5/30 Losses 0.02056
 PDE  [ 0.00628 ]
 boco 0.00885
 initial_condition 0.00543
 Val [ 0.00436 ]
Epoch 6/30 Losses 0.01482
 PDE  [ 0.00593 ]
 boco 0.00590
 initial_condition 0.00298
 Val [ 0.00449 ]
Epoch 7/30 Losses 0.01756
 PDE  [ 0.00600 ]
 boco 0.00659
 initial_condition 0.00498
 Val [ 0.00625 ]
Epoch 8/30 Losses 0.01163
 PDE  [ 0.00602 ]
 boco 0.00337
 initial_condition 0.00223
 Val [ 0.00561 ]
Epoch 9/30 Losses 0.01628
 PDE  [ 0.00619 ]
 boco 0.00651
 initial_condition 0.00358
 Val [ 0.00575 ]
Epoch 10/30 Losses 0.01855
 PDE  [ 0.00673 ]
 boco 0.00747
 initial_condition 0.00435
 Val [ 0.00574 ]
Epoch 11/30 Losses 0.01386
 PDE  [ 0.00682 ]
 boco 0.00453
 initial_condition 0.00251
 Val [ 0.00557 ]
Epoch 12/30 Losses 0.01559
 PDE  [ 0.00657 ]
 boco 0.00562
 initial_condition 0.00340
 Val [ 0.00934 ]
Epoch 13/30 Losses 0.01459
 PDE  [ 0.00721 ]
 boco 0.00464
 initial_condition 0.00275
 Val [ 0.00656 ]
Epoch 14/30 Losses 0.01102
 PDE  [ 0.00600 ]
 boco 0.00315
 initial_condition 0.00188
 Val [ 0.00774 ]
Epoch 15/30 Losses 0.01472
 PDE  [ 0.00603 ]
 boco 0.00536
 initial_condition 0.00333
 Val [ 0.00678 ]
Epoch 16/30 Losses 0.01430
 PDE  [ 0.00663 ]
 boco 0.00477
 initial_condition 0.00290
 Val [ 0.00680 ]
Epoch 17/30 Losses 0.01375
 PDE  [ 0.00608 ]
 boco 0.00498
 initial_condition 0.00269
 Val [ 0.00632 ]
Epoch 18/30 Losses 0.01422
 PDE  [ 0.00769 ]
 boco 0.00389
 initial_condition 0.00263
 Val [ 0.00917 ]
Epoch 19/30 Losses 0.01551
 PDE  [ 0.00681 ]
 boco 0.00558
 initial_condition 0.00311
 Val [ 0.00712 ]
Epoch 20/30 Losses 0.01247
 PDE  [ 0.00699 ]
 boco 0.00324
 initial_condition 0.00223
 Val [ 0.00739 ]
Epoch 21/30 Losses 0.01001
 PDE  [ 0.00679 ]
 boco 0.00173
 initial_condition 0.00149
 Val [ 0.00584 ]
Epoch 22/30 Losses 0.01696
 PDE  [ 0.00725 ]
 boco 0.00609
 initial_condition 0.00362
 Val [ 0.00786 ]
Epoch 23/30 Losses 0.01185
 PDE  [ 0.00722 ]
 boco 0.00267
 initial_condition 0.00196
 Val [ 0.00670 ]
Epoch 24/30 Losses 0.01341
 PDE  [ 0.00690 ]
 boco 0.00394
 initial_condition 0.00257
 Val [ 0.00774 ]
Epoch 25/30 Losses 0.01379
 PDE  [ 0.00760 ]
 boco 0.00360
 initial_condition 0.00258
 Val [ 0.00673 ]
Epoch 26/30 Losses 0.01136
 PDE  [ 0.00689 ]
 boco 0.00261
 initial_condition 0.00186
 Val [ 0.00617 ]
Epoch 27/30 Losses 0.01130
 PDE  [ 0.00732 ]
 boco 0.00217
 initial_condition 0.00181
 Val [ 0.00675 ]
Epoch 28/30 Losses 0.01118
 PDE  [ 0.00722 ]
 boco 0.00221
 initial_condition 0.00175
 Val [ 0.00744 ]
Epoch 29/30 Losses 0.01406
 PDE  [ 0.00688 ]
 boco 0.00455
 initial_condition 0.00264
 Val [ 0.00639 ]
Epoch 30/30 Losses 0.01065
 PDE  [ 0.00726 ]
 boco 0.00182
 initial_condition 0.00157
 Val [ 0.00759 ]

    /opt/conda/lib/python3.7/site-packages/numpy/core/fromnumeric.py:3257: RuntimeWarning: Mean of empty slice.
      out=out, **kwargs)
    /opt/conda/lib/python3.7/site-packages/numpy/core/_methods.py:161: RuntimeWarning: invalid value encountered in double_scalars
      ret = ret.dtype.type(ret / rcount)

```
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15,5))
ax1.plot(hist['train_loss'], label="train_loss")
ax1.plot(hist['val_loss'], label="val_loss")
ax1.grid(True)
ax1.legend()
ax1.set_yscale("log")
for boco in pde.bocos:
    ax2.plot(hist['bocos'][boco.name], label=boco.name)
ax2.legend()
ax2.grid(True)
ax2.set_yscale("log")
plt.show()
```

Finally, we can evaluate our solution. Since we trained it for also the parameter *u* we can obtain a solution for any value of *u* without having to perform additional computations.

```
# evaluate the solution
x = np.linspace(0,1,50)
t = np.linspace(0,1,50)
u = np.linspace(0,1,5)
p, p0, l2 = [], [], []
for _t in t:
    _p, _p0, _l2 = [], [], []
    for _u in u:
        _p0.append(np.sin(2.*math.pi*(x-_u*_t)))
        pde.evaluate({'x': x, 't': np.array([_t]), 'u': np.array([_u])}, device)
        _p.append(pde.outputs['p'])
        _l2.append(np.mean((pde.outputs['p']-np.sin(2.*math.pi*(x-_u*_t)))**2))
    p.append(_p)
    p0.append(_p0)
    l2.append(_l2)

from matplotlib import animation, rc
rc('animation', html='html5')

def plot(x, p, p0, t, l2, u):
    ax.clear()
    #tit = ax.set_title(f"t = {t:.2f}, l2 = {l2:.5f}", fontsize=14)
    tit = ax.set_title(f"t = {t:.2f}", fontsize=14)
    for i, _u in enumerate(u):
        ax.plot(x, p0[i], label=f"Exact (u = {_u})")
        ax.plot(x, p[i], ".k", label=f"NN (u = {_u}, l2 = {l2[i]:.5f})")
    ax.set_xlabel("x", fontsize=14)
    ax.set_ylabel("p", fontsize=14, rotation=np.pi/2)
    ax.legend(loc="upper right")
    ax.grid(True)
    ax.set_xlim([0, 1])
    ax.set_ylim([-1.2, 1.2])
    return [tit]

def get_anim(fig, ax, x, p, p0, t, l2, u):
    def anim(i):
        return plot(x, p[i], p0[i], t[i], l2[i], u)
    return anim

fig = plt.figure(figsize=(10,5))
ax = fig.add_subplot(111, autoscale_on=False)
animate = get_anim(fig, ax, x, p, p0, t, l2, u)
anim = animation.FuncAnimation(fig, animate, frames=len(t), interval=100, blit=True)
```

```
anim
```

```

```

```

```


## Day 3: Cells in Silicon

Welcome to Day 3! Today, we start with our discussion of Hodgkin Huxley Neurons and how we can simulate them in Python using Tensorflow and Numerical Integration.

The electric potential measured across the membranes of excitable cells, such as neurons or heart cells, can undergo transient changes when perturbed by external inputs. When the inputs to a neuron are sufficiently large these transient changes can regeneratively build up into a large deviation from the resting state known as an action potential. Action potentials propagate undiminished along the axon and perturb post-synaptic neurons. The Hodgkin-Huxley model is a system of differential equations that describe the generation an action potential and its propagation along the axon. We provide only a brief overview of the Hodgkin-Huxley model. A number of classic references (Dayan 2005, Johnston 1995) and the original papers by Hodgkin and Huxley (Huxley 1952) chronicle the history and the details of the model. An excellent set of MOOCS and the accompanying textbooks (Gerstner 2014, Dayan 2005) give an accessible introduction to the topic

### What is the Hodgkin Huxley Neuron Model?

The cell membrane, a 5nm thick lipid bilayer, separates the inside from the outside of the neuron. The membrane is largely impermeable to charged ions present on either side. The concentration of $\text{Na}^{+}$ ions outside the cell is greater than its concentration inside, while $\text{K}^{+}$ ions are relatively abundant inside compared to the outside. In addition to these there are chloride ($\text{Cl}^{-}$), calcium ($\text{Ca}^{2+}$) and magnesium ions ($\text{Mg}^{+}$) that populate the cellular milieu. The differences in ionic abundances across the membrane cause a net accumulation of positive ions on one side of the membrane and negative ions on the other, and thus a potential difference across the membrane. Embedded on the membrane are ion channels that are highly selective to the ion species it lets across. In the squid axon, Hodgkin and Huxley found that there were only two types of ion channels ($\text{Na}^{+}$ and $\text{K}^{+}$), in addition to a non-specific leak channel. The Hodgkin-Huxley model of neurons can be understood with the help of an equivalent electrical circuit given below. The cell membrane acts as a capacitor. The total injected current ($I$) can be written as the sum of the capacitive current $I_{C}$, ionic currents $I_{Na}$ and $I_{K}$ and the leak current $I_L$.

\begin{equation}
I = I_{C}(t) + I_{Na}(t) + I_{K}(t)
\end{equation}
where,
\begin{eqnarray}
C_m = 1 \mu F/cm^2 \\
I_{Na} = g_{Na}(u-E_{Na})\\
I_{K} = g_{k}(u-E_K)\\
I_{L} = g_{L}(u-E_L)
\end{eqnarray}
The equation describing the membrane potential can thus be written as follows,

\begin{eqnarray}
C_m\frac{dV}{dt}=−I_{Na}(t)−I_{K}(t)−I_{L}(t)+I(t)
\end{eqnarray}

Hodgkin and Huxley discovered that the $Na$ and the $K$ channels do not act as Ohmic conductances, but are modulated by the potential across the membrane.
Changes in potential had a nonlinear effect on flow of ionic currents. Based in their experimental results they obtained a system of differential equations that described the temporal evolution of the membrane potential in terms of changes in ionic currents (chiefly $\text{Na}^{+}$ and $\text{K}^{+}$).

\begin{eqnarray}
I = g_{Na}m^3h(u−E_{Na}) \\
I_K = g_Kn^4(u−E_K)\\
I_L = g_L(u−E_L)
\end{eqnarray}

where $E_{Na}=50\ mV$, $E_K = -95\ mV$ and $E_L=-55\ mV$ are the reversal potentials; $g_{Na} = 100\ \mu S/cm^2$, $g_K = 10\ \mu S/cm^2$ and $g_L = 0.15\ \mu S/cm^2$ are the channel conductances; and m,h, and n are gating variables that follow the dynamics given by:

\begin{eqnarray}
\frac{dm}{dt} = - \frac{1}{\tau_m}(m-m_0)\\
\frac{dh}{dt} = - \frac{1}{\tau_h}(h-h_0)\\
\frac{dn}{dt} = - \frac{1}{\tau_n}(n-n_0)
\end{eqnarray}

where $\tau_m$, $\tau_h$ and $\tau_n$ are empirically determined voltage dependent time constants and $m_0$, $h_0$ and $n_0$ are voltage dependent asymptotic gating values.

On day 2, we had created a RK4 based numerical integrator. Recall this implementation:

```python
import numpy as np
import matplotlib.pyplot as plt
import tensorflow as tf
## OR for Tensorflow 2.0 and above ##
import tensorflow.compat.v1 as tf
tf.disable_v2_behavior()

%matplotlib inline

def tf_check_type(t, y0): # Ensure Input is Correct
    if not (y0.dtype.is_floating and t.dtype.is_floating):
        raise TypeError('Error in Datatype')

class _Tf_Integrator():

    def integrate(self, func, y0, t):
        time_delta_grid = t[1:] - t[:-1]

        def scan_func(y, t_dt):
            t, dt = t_dt
            dy = self._step_func(func,t,dt,y) # Make code more modular.
            return y + dy

        y = tf.scan(scan_func, (t[:-1], time_delta_grid),y0)
        return tf.concat([[y0], y], axis=0)

    def _step_func(self, func, t, dt, y):
        k1 = func(y, t)
        half_step = t + dt / 2
        dt_cast = tf.cast(dt, y.dtype) # Failsafe

        k2 = func(y + dt_cast * k1 / 2, half_step)
        k3 = func(y + dt_cast * k2 / 2, half_step)
        k4 = func(y + dt_cast * k3, t + dt)
        return tf.add_n([k1, 2 * k2, 2 * k3, k4]) * (dt_cast / 6)

def odeint(func, y0, t):
    t = tf.convert_to_tensor(t, name='t')
    y0 = tf.convert_to_tensor(y0, name='y0')
    tf_check_type(y0,t)
    return _Tf_Integrator().integrate(func,y0,t)
```

#### Implementing the Hodgkin-Huxley neuron model

The variables of the Hodgkin Huxley neuron model that are updated at each integration time step are, the membrane potential, $V$, the sodium activation gating variable, $m$, the sodium inactivation gating variable, $h$, and the potassium channel gating variable, $n$. The dynamics are given by Equations above. In the following code, we define the parameters associated with the conductances, including the formulae for $\tau_{m}$, $\tau_{h}$, $\tau_{n}$ and the voltage dependent steady state values of the gating variables.

##### Step 1: Defining Parameters of the Neuron

```python
C_m = 1      # Membrane Capacitance

g_K = 10
E_K = -95

g_Na = 100
E_Na = 50

g_L = 0.15
E_L = -55
```

##### Step 2: Defining functions that calculate $\tau_m$, $\tau_h$, $\tau_n$, $m_0$, $h_0$, $n_0$

Note: Always use Tensorflow functions for all mathematical operations.

For our Hodgkin Huxley Model, we will determine the values of $\tau_m$, $\tau_h$, $\tau_n$, $m_0$, $h_0$, $n_0$  by the following equations:

```python
def K_prop(V):
    T = 22
    phi = 3.0**((T-36.0)/10)
    V_ = V-(-50)

    alpha_n = 0.02*(15.0 - V_)/(tf.exp((15.0 - V_)/5.0) - 1.0)
    beta_n = 0.5*tf.exp((10.0 - V_)/40.0)

    t_n = 1.0/((alpha_n+beta_n)*phi)
    n_0 = alpha_n/(alpha_n+beta_n)

    return n_0, t_n

def Na_prop(V):
    T = 22
    phi = 3.0**((T-36)/10)
    V_ = V-(-50)

    alpha_m = 0.32*(13.0 - V_)/(tf.exp((13.0 - V_)/4.0) - 1.0)
    beta_m = 0.28*(V_ - 40.0)/(tf.exp((V_ - 40.0)/5.0) - 1.0)

    alpha_h = 0.128*tf.exp((17.0 - V_)/18.0)
    beta_h = 4.0/(tf.exp((40.0 - V_)/5.0) + 1.0)

    t_m = 1.0/((alpha_m+beta_m)*phi)
    t_h = 1.0/((alpha_h+beta_h)*phi)

    m_0 = alpha_m/(alpha_m+beta_m)
    h_0 = alpha_h/(alpha_h+beta_h)

    return m_0, t_m, h_0, t_h
```

##### Step 3: Defining function that calculate Neuronal currents

```python
def I_K(V, n):
    return g_K  * n**4 * (V - E_K)

def I_Na(V, m, h):
    return g_Na * m**3 * h * (V - E_Na)

def I_L(V):
    return g_L * (V - E_L)
```

##### Step 4: Define the function dX/dt where X is the State Vector

```python
def dXdt(X, t):
    V = X[0:1]
    m = X[1:2]
    h = X[2:3]
    n = X[3:4]

    dVdt = (5 - I_Na(V, m, h) - I_K(V, n) - I_L(V)) / C_m
    # Here the current injection I_injected = 5 uA

    m0,tm,h0,th = Na_prop(V)
    n0,tn = K_prop(V)

    dmdt = - (1.0/tm)*(m-m0)
    dhdt = - (1.0/th)*(h-h0)
    dndt = - (1.0/tn)*(n-n0)

    out = tf.concat([dVdt,dmdt,dhdt,dndt],0)
    return out
```

##### Step 5: Define Initial Condition and Integrate

```python
y0 = tf.constant([-71,0,0,0], dtype=tf.float64)

epsilon = 0.01
t = np.arange(0,200,epsilon)

state = odeint(dXdt,y0,t)

with tf.Session() as sess:
    state = sess.run(state)
```

##### Step 6: Plot Output

```python
plt.plot(t,state.T[0,:])

plt.xlabel("Time (in ms)")
plt.ylabel("Voltage (in mV)")

plt.show()
```

#### Simulating Multiple Independent HH Neurons at the Same Time

Here we illustrate some simple steps that can be used to simulate populations of neurons efficiently. Key to setting up the equations is to order it in a manner that utilizes TensorFlow's algorithms that distribute vector, matrix and tensor computations over multiple cores. Consider a system of 20 independent HH neurons with different input currents that characterise the firing rates.

##### Methods of Parallelization
TensorFlow has built-in functions that speed up Tensor computations using available multi-cores, and GPU/TPU setups. There are two major parts of the code where such a speed-up can be effected

1. **RK4 iterations** Our implementation of the integrator utilizes Tensors as inputs.
2. **Functional Evaluations:** The form of the equations that describe the neuronal dynamics,  are common across neurons. Only the parameters for differ across neurons. This can be used to `vectorize' the equations.

Say $\vec{X}=[V,m,n,h]$ is the state vector of a single neuron and its dynamics are defined using parameters $C_m,g_K,...E_L$ equations of the form:

\begin{eqnarray}\frac{d\vec{X}}{dt} = [f_1(\vec{X},C_m,g_K,...E_L),f_2(\vec{X},C_m,g_K,...E_L)...f_m(\vec{X},C_m,g_K,...E_L)]\end{eqnarray}

We can convert these equations to a form in which all evaluations are done as vector calculations and NOT scalar calculations. Despite the parameters being different, the functional forms of the equations are similar for the same state variable of different neurons. Thus, the trick is to reorganize $\mathbf{X}$ as $\mathbf{X'}=[(V_1,V_2,...V_n),(m_1,m_2,...m_n),(h_1,h_2,...h_n),(n_1,n_2,...n_n)]=[\vec{V},\vec{m},\vec{h},\vec{n}]$. And the parameters as $[\vec{C_m},\vec{g_K}] = [C_{m_{1}}\dots C_{m_{n}},g_{K_{1}}\dots g_{K_{n}}]$ and so on.

The advantage of this re-ordering is that the differential equation of the form,
\begin{eqnarray}\frac{dV_i}{dt}=f(V_i,m_i,h_i,n_i,C_{m_i},g_{K_i}...)\end{eqnarray}

is now easily parallelizable using a vector computation of the form,

\begin{eqnarray}\frac{d\vec{V}}{dt}=f(\vec{V},\vec{m},\vec{h},\vec{n},\vec{C_m},\vec{g_K}...)\end{eqnarray}

The equations can now be written in the form,
\begin{eqnarray}\frac{d\mathbf{X'}}{dt}= \Big[\frac{d\vec{V}}{dt},\frac{d\vec{m}}{dt},\frac{d\vec{h}}{dt},\frac{d\vec{n}}{dt}\Big]\end{eqnarray}

```python
n_n = 20 # number of simultaneous neurons to simulate

# parameters will now become n_n-vectors

C_m = [1.0]*n_n
g_K = [10.0]*n_n
E_K = [-95.0]*n_n

g_Na = [100]*n_n
E_Na = [50]*n_n

g_L = [0.15]*n_n
E_L = [-55.0]*n_n

def K_prop(V):
    T = 22
    phi = 3.0**((T-36.0)/10)
    V_ = V-(-50)

    alpha_n = 0.02*(15.0 - V_)/(tf.exp((15.0 - V_)/5.0) - 1.0)
    beta_n = 0.5*tf.exp((10.0 - V_)/40.0)

    t_n = 1.0/((alpha_n+beta_n)*phi)
    n_0 = alpha_n/(alpha_n+beta_n)

    return n_0, t_n

def Na_prop(V):
    T = 22
    phi = 3.0**((T-36)/10)
    V_ = V-(-50)

    alpha_m = 0.32*(13.0 - V_)/(tf.exp((13.0 - V_)/4.0) - 1.0)
    beta_m = 0.28*(V_ - 40.0)/(tf.exp((V_ - 40.0)/5.0) - 1.0)

    alpha_h = 0.128*tf.exp((17.0 - V_)/18.0)
    beta_h = 4.0/(tf.exp((40.0 - V_)/5.0) + 1.0)

    t_m = 1.0/((alpha_m+beta_m)*phi)
    t_h = 1.0/((alpha_h+beta_h)*phi)

    m_0 = alpha_m/(alpha_m+beta_m)
    h_0 = alpha_h/(alpha_h+beta_h)

    return m_0, t_m, h_0, t_h

def I_K(V, n):
    return g_K  * n**4 * (V - E_K)

def I_Na(V, m, h):
    return g_Na * m**3 * h * (V - E_Na)

def I_L(V):
    return g_L * (V - E_L)

def dXdt(X, t):
    V = X[:1*n_n]       # First n_n values are Membrane Voltage
    m = X[1*n_n:2*n_n]  # Next n_n values are Sodium Activation Gating Variables
    h = X[2*n_n:3*n_n]  # Next n_n values are Sodium Inactivation Gating Variables
    n = X[3*n_n:]       # Last n_n values are Potassium Gating Variables

    dVdt = (np.linspace(0,10,n_n) - I_Na(V, m, h) - I_K(V, n) -I_L(V)) / C_m
    # Input current is linearly varied between 0 and 10

    m0,tm,h0,th = Na_prop(V)
    n0,tn = K_prop(V)

    dmdt = - (1.0/tm)*(m-m0)
    dhdt = - (1.0/th)*(h-h0)
    dndt = - (1.0/tn)*(n-n0)

    out = tf.concat([dVdt,dmdt,dhdt,dndt],0)
    return out

y0 = tf.constant([-71]*n_n+[0,0,0]*n_n, dtype=tf.float64)

epsilon = 0.01
t = np.arange(0,200,epsilon)

state = odeint(dXdt,y0,t)

with tf.Session() as sess:
    state = sess.run(state)
```

```python
plt.figure(figsize=(12,17))
for i in range(20):
    plt.subplot(10,2,i+1)
    plt.plot(t,state[:,i])
    plt.title("Injected Current = {:0.1f}".format(i/2))
    plt.ylim([-90,60])
    plt.xlabel("Time (in ms)")
    plt.ylabel("Voltage (in mV)")

plt.tight_layout()
plt.show()
```

#### Quantifying the Firing Rates against Input Current

The firing frequency as a function of the input is shown in the figure below. The code to generate the firing rate is below.

```python
plt.plot(np.linspace(0,10,20),np.bitwise_and(state[:-1,:20]<0,state[1:,:20]>0).sum(axis=0)/0.2,"o")
plt.xlabel("Injected Current(mA)")
plt.ylabel("Firing Rate (Hz)")
plt.show()
```

# References

(Dayan and Abbott, 2005) Peter Dayan and Larry F. Abbott, ``Theoretical Neuroscience - Computational and Mathematical Modeling of Neural Systems``,  2005.

(Johnston and Wu, 1995) D. Johnston and S. M.S. Wu, ``Foundations of cellular neurophysiology``,  1995.

(Huxley and Hodgkin, 1952) Huxley A. L. and Hodgkin A. F., ``Quantitative description of nerve current``, Journal of Physiology, vol. , number , pp. ,  1952.

(MOOC) , ``Neuronal dynamics``,  .  [online](https://www.edx.org/course/neuronal-dynamics)

(MOOC) , ``Computational Neuroscience``,  .  [online](https://www.coursera.org/learn/computational-neuroscience)

(Gerstner, Kistler et al., 2014) Wulfram Gerstner, Werner M. Kistler, Richard Naud et al., ``Neuronal dynamics: From single neurons to networks and models of cognition``,  2014.

Tutorial/Day 3 Cells in Silicon/Day 3.ipynb


```python
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

from scipy.integrate import odeint
```

```python
from sklearn.metrics import r2_score, mean_absolute_error
from sklearn.metrics import mean_squared_error, median_absolute_error
```

```python
def mean_absolute_percentage_error(y_true, y_pred):
    return np.mean(np.abs((y_true - y_pred) / y_true)) * 100
```

```python
def symmetric_mape(y_true, y_pred, eps = 1e-8):
    summ = ((np.abs(y_true) + np.abs(y_pred)) + eps)
    return np.mean(np.abs(y_pred - y_true) / summ) * 100
```

```python
def print_scores(y_test, y_pred):
    print(f"R2 score: {r2_score(y_test, y_pred)}")
    print(f"MSE score: {mean_squared_error(y_test, y_pred)}")
    print(f"MAE score: {mean_absolute_error(y_test, y_pred)}")
    print(f"Median AE score: {median_absolute_error(y_test, y_pred)}")
    print(f"MAPE score: {mean_absolute_percentage_error(y_test, y_pred)}")
    print(f"SMAPE score: {symmetric_mape(y_test, y_pred)}")
```

# 1. Discovering governing equations using the SINDy algorithm

Paper: https://www.pnas.org/content/113/15/3932

SINDy provides an algorithm to discover governing dynamical equations given a time series **x(t)**. The three steps in this algorithm are:
 - Time derivative: Perhaps the trickiest part, especially for noisy time series although one can use total variation regularized derivatives for such cases as in [here](https://github.com/stur86/tvregdiff)
 - Basis: Some non-linear basis of the original time series that could be polynomial, trigonometric (see the paper above)
 - Regularized linear regression: Apply LASSO and/or sequential thresholding least squares.

I have personally found that SINDy is not so good with series that are too noisy and/or have trends. Both these problems could be addressed using pre-processing.

SINDy can also be used for **forecasting**, which is trivial once one discovers the dynamical equations.

## Create synthetic dataset

Had trouble finding a good function.

**REMEMBER**: change `dx1dt`, `dx2dt` also if you change the ranfun below.

```python
def fun(y, t):
    x1, x2 = y
    dxdt = [-x1**3 - x2, x1 - x2**3]
    return dxdt

t_full = np.linspace(0, 15, 1501)

# The following two functions work great, with alpha = 0.0001, they are being correctly classified.
#y0 = [0.1, 0.05] # LINEAR: small x1, x2 => Linear coupled oscillator
y0 = [0.5, 0.5] # NON-LINEAR: Getting close to triggering non-linear effects
```

**Solve** the ODE:

$$
\begin{align}
\frac{dx_1}{dt} &= -x_1^3 - x_2 \\
\frac{dx_2}{dt} &= x_1 - x_2^3
\end{align}
$$

```python
sol = odeint(fun, y0, t_full)
sol_new = sol[:1001]
```

#### Plot the solutions

```python
t_discover = t_full[:1001]

plt.plot(t_discover, sol_new[:, 0], 'b', label='x1(t)')
plt.plot(t_discover, sol_new[:, 1], 'g', label='x2(t)')
plt.legend(loc='best')
plt.xlabel('t')
plt.grid()
plt.savefig('time_series.png', bbox_inches='tight')
```

## Step 1: Compute the time derivative.

```python
x1 = sol_new[:,0]
x2 = sol_new[:,1]
```

**Actual** time derivatives

```python
dx1dt = -x1**3 - x2
dx2dt =  x1    - x2**3
```

**Numerically** computed derivatives from data

```python

dt = t_discover[1] - t_discover[0]
dx1dt_data = np.gradient(x1, dt)
dx2dt_data = np.gradient(x2, dt)
# or using TVRegDiff
# dx1dt = TVRegDiff(x1, 10, 0.1, dx=dt, plotflag=0)
# See https://github.com/stur86/tvregdiff
# for information on different parameters
```

```python
print_scores(dx1dt, dx1dt_data)
```

    R2 score: 0.9999999965191242
    MSE score: 2.8269156140981004e-10
    MAE score: 4.206507155221408e-06
    Median AE score: 3.6181352647546294e-06
    MAPE score: 0.004477342875542224
    SMAPE score: 0.0022391799382288345

```python
print_scores(dx2dt, dx2dt_data)
```

    R2 score: 0.9999996026514059
    MSE score: 2.195307457531667e-08
    MAE score: 9.892101759080555e-06
    Median AE score: 4.016199533329878e-06
    MAPE score: 0.009102822948966595
    SMAPE score: 0.004535741601342113

```python
plt.plot(dx1dt[-50:], label='Actual', alpha=0.4)
plt.plot(dx1dt_data[-50:], ':', lw=2, label='from_data')

plt.xlabel('Time', fontsize=14)
plt.ylabel(r'$dx_1/dt$', fontsize=14)
plt.legend()
```

```python
plt.plot(dx2dt[-50:], label='Actual', alpha=0.4)
plt.plot(dx2dt_data[-50:], ':', lw=2, label='from_data')

plt.xlabel('Time', fontsize=14)
plt.ylabel(r'$dx_2/dt$', fontsize=14)
plt.legend()
```

Since the `np.gradient` derivatives are nearly identical to the original, I will be using the original derivatives.

```python
X = np.zeros((sol_new.shape[0], sol_new.shape[1]))

X[:, 0] = x1 # dx1/dt
X[:, 1] = x2 # dx2/dt
```

## Step 2: Construct a polynomial (or trig) basis to do the regression

```python
from sklearn.preprocessing import PolynomialFeatures

dum_data = pd.DataFrame(x1, columns=['x1'])
dum_data['x2'] = x2

deg = 3

p = PolynomialFeatures(degree=deg,include_bias=True).fit(dum_data)
xpoly = p.fit_transform(dum_data)
newdf = pd.DataFrame(xpoly, columns = p.get_feature_names(dum_data.columns))

print("Feature names:", list(newdf))#newdf.columns.values.tolist())
print("Feature array shape:", newdf.shape)
```

    Feature names: ['1', 'x1', 'x2', 'x1^2', 'x1 x2', 'x2^2', 'x1^3', 'x1^2 x2', 'x1 x2^2', 'x2^3']
    Feature array shape: (1001, 10)

## Step 3: LASSO

(or Ridge/OLS with sequential thresholding: that is first do the fit with Ridge/OLS. Then discard any features below a certain value. Do the fit again...)

low alpha -> OLS

high alpha -> most features zero

**alpha** is a hyperparameter

```python
from sklearn.linear_model import Lasso
mod = Lasso(alpha=0.0001)
mod
```

    Lasso(alpha=0.0001, copy_X=True, fit_intercept=True, max_iter=1000,
          normalize=False, positive=False, precompute=False, random_state=None,
          selection='cyclic', tol=0.0001, warm_start=False)

### Prepare training data

```python
newdf_train, newdf_test = newdf[:800], newdf[800:]
dx1dt_train, dx1dt_test = dx1dt[:800], dx1dt[800:]
dx2dt_train, dx2dt_test = dx2dt[:800], dx2dt[800:]
```

### dx1/dt

```python
mod.fit(newdf_train, dx1dt_train)
print(mod.coef_) # should give the 3rd (x2) + 4th (x1^2) argument non-zero
print(mod.intercept_)
mod.score(newdf_test, dx1dt_test)
```

    [ 0.         -0.05576026 -0.99852418  0.         -0.         -0.01819165
     -0.50202469 -0.          0.         -0.        ]
    0.0033302444886464805

    0.9943287924302203

```python
fit_dx1 = pd.DataFrame(columns=newdf.columns)
fit_dx1.loc[0] = mod.coef_
fit_dx1.abs().sort_values(by=0, axis=1, ascending=False)
```

      x2
      x1^3
      x1
      x2^2
      1
      x1^2
      x1 x2
      x1^2 x2
      x1 x2^2
      x2^3

      0
      0.998524
      0.502025
      0.05576
      0.018192
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

```python
ypred_x1 = mod.predict(newdf_test)
print_scores(dx1dt_test, ypred_x1)
```

    R2 score: 0.9943287924302203
    MSE score: 8.443005835967606e-05
    MAE score: 0.009127412469564916
    Median AE score: 0.00942351242559783
    MAPE score: 59.78370313040233
    SMAPE score: 9.034670809972106

### Identified ODE:

$dx_1/dt \sim -x_2 - x_1^3$

with minor contributions from other terms that could be gotten rid of by applying a threshold

```python
# Drop features with absolute values less than 0.1
dx1_thr = fit_dx1[fit_dx1.columns[fit_dx1.abs().max() > 0.1]]
dx1_thr
```

      x2
      x1^3

      0
      -0.998524
      -0.502025

```python
# PLOT results
t_test  = np.linspace(8, 10, 201)

#plt.plot(t, sol_new[:, 1], label='Actual', alpha = 0.4)
#plt.plot(ypred, label='Actual')
plt.plot(t_test, ypred_x1, 'r--', lw = 2, label='Prediction')
plt.plot(t_discover, dx1dt, 'g', label='Actual', alpha=0.4)
plt.axvline(x = 8, color='k', linestyle='--')
plt.xlabel('Time', fontsize=14)
plt.ylabel(r'$dx_1/dt$', fontsize=14)
plt.grid()
plt.legend()
plt.savefig('dx1dt_fit.png', bbox_inches='tight')
```

### dx2/dt

```python
mod.fit(newdf_train, dx2dt_train)
print(mod.coef_) # should give the 2nd (x1) + last (x2^3) argument non-zero
print(mod.intercept_)
mod.score(newdf_test, dx2dt_test)
```

    [ 0.          0.99491832 -0.02198579 -0.         -0.         -0.04791986
      0.          0.         -0.         -0.82689872]
    0.0022126622685510883

    0.9965028656692734

```python
fit_dx2 = pd.DataFrame(columns=newdf.columns)
fit_dx2.loc[0] = mod.coef_
fit_dx2.abs().sort_values(by=0, axis=1, ascending=False)
```

      x1
      x2^3
      x2^2
      x2
      1
      x1^2
      x1 x2
      x1^3
      x1^2 x2
      x1 x2^2

      0
      0.994918
      0.826899
      0.04792
      0.021986
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

```python
ypred_x2 = mod.predict(newdf_test)
print_scores(dx2dt_test, ypred_x2)
```

    R2 score: 0.9965028656692734
    MSE score: 1.3128657613492761e-05
    MAE score: 0.003284761866157685
    Median AE score: 0.0036951175861815594
    MAPE score: 1.962931213051049
    SMAPE score: 0.9952953131151921

### Identified ODE:

$dx_2/dt \sim x_1 - x_2^3$

with minor contributions from other terms that could be gotten rid of by applying a threshold

```python
# Drop features with absolute values less than 0.1
dx2_thr = fit_dx2[fit_dx2.columns[fit_dx2.abs().max() > 0.1]]
dx2_thr
```

      x1
      x2^3

      0
      0.994918
      -0.826899

```python
# PLOT results
t_test  = np.linspace(8, 10, 201)

#plt.plot(t, sol_new[:, 0], label='Actual', alpha = 0.4)
#plt.plot(ypred, label='Actual')
plt.plot(t_test, ypred_x2, 'r--', lw = 2, label='Prediction')
plt.plot(t_discover, dx2dt, 'g', label='Actual', alpha = 0.4)
plt.axvline(x = 8, color='k', linestyle='--')
plt.xlabel('Time', fontsize=14)
plt.ylabel(r'$dx_2/dt$', fontsize=14)
plt.grid()
plt.legend()
plt.savefig('dx2dt_fit.png', bbox_inches='tight')
```

```python

```

```python

```

# 2. Forecasting using SINDy

Now that we have discovered the differential equations, we can use the ODE solver to forecast the future.

```python
# Manually entering values for coefficients but this can be automated
def fun_forecast(y, t):
    x1, x2 = y
    dxdt = [-0.5*(x1**3) - 0.9985*x2,
            0.995*x1     - 0.8269*x2**3]
    return dxdt

t_forecast = np.linspace(10, 15, 500)

y0 = [x1[-1], x2[-1]]

sol_forecast = odeint(fun_forecast, y0, t_forecast)
```

```python
plt.plot(t_forecast, sol_forecast[:, 0], 'b', label='x1(t)')
plt.plot(t_forecast, sol_forecast[:, 1], 'g', label='x2(t)')
plt.legend(loc='best')
plt.xlabel('t')
plt.grid()
```

```python
plt.plot(t_forecast, sol_forecast[:, 0], 'b--', label='Forecast')
plt.plot(t_full, sol[:, 0], 'g', label='Actual', alpha=0.4)
plt.axvline(x = 10, color='k', linestyle='--')
plt.legend(loc='best')
plt.xlabel('Time', fontsize=14)
plt.ylabel(r'$x_1$', fontsize=14)
plt.grid()
```

```python
plt.plot(t_forecast, sol_forecast[:, 1], 'b--', label='Forecast')
plt.plot(t_full, sol[:, 1], 'g', label='Actual', alpha=0.4)
plt.axvline(x = 10, color='k', linestyle='--')
plt.legend(loc='best')
plt.xlabel('Time', fontsize=14)
plt.ylabel(r'$x_2$', fontsize=14)
plt.grid()
```

```python
y_pred1, y_pred2 = sol_forecast[:, 0], sol_forecast[:, 1]
y_test1, y_test2 = sol[1001:, 0],      sol[1001:, 1]
```

```python
for (y_test, y_pred) in zip([y_test1, y_test2], [y_pred1, y_pred2]):
    print(f"R2 score: {r2_score(y_test, y_pred)}")
    print(f"MSE score: {mean_squared_error(y_test, y_pred)}")
    print(f"MAE score: {mean_absolute_error(y_test, y_pred)}")
    print(f"Median AE score: {median_absolute_error(y_test, y_pred)}")
    print(f"MAPE score: {mean_absolute_percentage_error(y_test, y_pred)}")
    print(f"SMAPE score: {symmetric_mape(y_test, y_pred)}")
```

    R2 score: 0.9986014512322046
    MSE score: 2.9194295629861532e-05
    MAE score: 0.004498499556886438
    Median AE score: 0.004095611629683124
    MAPE score: 8.356195293001417
    SMAPE score: 3.427959986309787
    R2 score: 0.9988787480251635
    MSE score: 2.8082044451838518e-05
    MAE score: 0.004387715903581068
    Median AE score: 0.0032235369101030545
    MAPE score: 8.17055503299457
    SMAPE score: 3.0643658219157226

```python

```


# Numerical Methods

# Lecture 6: Numerical Linear Algebra II

## Contents

## I.   What are Matrices?
## II.  Introduction to LU Decomposition
## III. LU Decomposition Doolittle Algorithm
## IV. LU Decomposition Implementation
## V.  Partial Pivoting

```python
import numpy as np
import scipy.linalg as sl
```

## Learning objectives:

* More on direct methods: LU decomposition
* Doolittle's algorithm
* Properties of lower-triangular matrices
* Partial pivoting

## I.   What are Matrices?

Now that you have gained a bit of understanding about Matrix, it is important to think, for once, what exactly are matrices, determinantsm singularity and solvability of systems. And also, the various operations that Python has for Matrix.

First things first, a bad thing with highschool mathematics is that it always makes matrices appear as something unrelated to the other fields of mathematics. But as you might have seen in the previous lectures, matrices can be used to solve systems of linear equations. Matrices have another bunch of useful uses in mathematics, as you might recall from Remote Sensing and GIS for image manipulations etc. You should therefore think of matrices as another tool we use in Mathematics, and a tool in mathematics that has been passed to the hands of other scientists who applied to all kinds of important scientific magic.

Second, you should think of Matrices of something like a storage device, and stores values inside. Take our system of linear equations. Think about a poor chap, who before, the age of computers, had to solve everything by the power of hand and mind. Imagine given a system of linear equations for you to solve, and well, to solve this system of linear equations, the poor chap has to write $x,y,z$ again and again and again, just like all of you might have done in highschool. It's rather inconvenient, bothersome, wastes lots of paper and ink. What you need to solve any system of these linear equations, i.e. find the value of the unknowns, are the coefficients and the result of each system of linear equation on the RHS, and therefore, they thought, is there a way to write the coefficients, variables, and the results seprately, so that you don't have to repeatedly write stuff again and again?

Well here comes matrices！Matrix $\pmb{A}$ stores the coefficients, $\pmb{x}$ stores the variables and $\pmb{b}$ stores the results of the linear equations!

$$\pmb{Ax =b }$$

You have already seent the capabilities of Gaussian elimination in solving equations. Now, when there are only 2 equations with 2 unknowns, Gaussian elimination is probably not the best method for solvng it. Rather, some of you should be able to solve these but inspection alone. But what happens when you have dozens of variables and dozens of unknowns, can you still solve it simply through inspection? Maybe some of you can, but I know that I cannot. The capabilities of matrices really shine when you have large complicated problems and it becomes difficult to keep track.

Imagine a system with dozens of equation and dozens of unknown, and let's say that there exist a unique solution to our problem. Now, think about how times you would have to write $x,y,z$, if you well, did not have matrices, in order to solve the equation at hand. Basically, when solving these large equations, matrices make your life easier by decreasing the amount of things that you need to write.

Moreover, think about your computer, which is neither omniscent and omnipotent, and sometimes confused at the simplest of things. While your Python today can easily deal with symbols through your Sympy libary, and can even solve system of linear equations quite easily, think about the first computers that were invented, with, well, very limited processing power and memory. For example, one of the earliest computers, ENIAC, whose first program was to study the feasibility of thermonuclear weapons. ENIAC had to do a lot of mathematical calculations, while having very limited memory to play with, thus requiring some way of organizing the information used for these mathematical calculations in such a way that it would take less memory. So, MATRIX.

Even today, with Moore's Law having continued for so many years, we are still in a continous arms race between computational power and the problems we are seeking to solve. While the amount of memory you could equip a computer with has increased significantly since ENIAC, so has the power hungriness of the problems we seek to solve. Just look at the list of supercomputers in the world, would the government waste so much of the taxpayer's money on computational power they do not need? Thus, while we can increase memory, we should also try to make our methodology more memory efficient, so that they could solve really complex problem while armed with very limited memory. Making the problems we have into Matrices is a great way for saving lots of memory and computational power!

Last lecture we introduced the calculation for determinants, but how do determinants determine if an equation is solvable or not?

Let's start with the determinant's definition. From wikipedia, In linear algebra, the determinant is a scalar value that can be computed from the elements of a square matrix and encodes certain properties of the linear transformation described by the matrix. One of the most important properties that the determinant encodes is the solvability, meaning that finding the determinant can help us determine if the equaiton is solvable or not.

Then, let's discuss solvability. When we try to solve an equation, we need information about the unknowns. If we are given enough information, the equation can be solved. If we are given too much information, or too little information, thus onverconstraining or underconstraining the problem, then the equation cannot be solved. You would encounter this especially when attempting to solve ODEs and PDEs in real world settings. Knowing how computationally expensive is to solve large scale problem, it is very important to know if the problem can be solved before beginning to solve the problem. Some of the most important topics in mathematics are about proving the solvability of certain forms or types of problems.

Ok, so let's take a system of 2 linear equations, with 2 unknowns. If we make a graph, there could be 3 situations. The lines of the linear equation could intersect, the lines of the linear equations could not intersect,i.e. parallel or the two lines could simply be actually the same line in disguise, i.e. the line is still parallel, since a line is always parallel to itself, giving us infinitely many solutions. What we seek is the situation where the two lines intersect, and gives us a point, i.e. a unique solution, or the lines are not parallel.

We could write our equation in general form as

$$
ax + by = e \\
cx + dy = f
$$

If we wrote our equation in matrix form it would be

$$
\begin{pmatrix}
a & b  \\
c & d \\
\end{pmatrix}
\begin{pmatrix}
x  \\
y \\
\end{pmatrix}
=
\begin{pmatrix}
e  \\
f \\
\end{pmatrix}
$$

We know the determinant for the matrix form above is

$$
\pmb{det} = ad - bc
$$

Let's write the equation instead in slope - intercept form

$$
y = - \frac{a}{b} x + \frac{e}{b} \\
y = - \frac{c}{d} x + \frac{f}{d}
$$

We know that there is unique solution when the lines are not parallel, and there is not a unique solution if the lines are not parallel. A line is parallel if the slope are equal, and not parallel if the slopes are not equal. We find the slopes as

$$
m_1 = - \frac{a}{b}\\
m_2 = - \frac{c}{d}
$$

Let's consider the case when the lines are parallel, i.e. slopes are equal and let's manipulate it a little

$$
- \frac{a}{b} = - \frac{c}{d} \\
ad - bc = 0
$$

Let's consider the case when the lines are not parallel, i.e. slopes are not equal and let's manipulate it a little

$$
- \frac{a}{b} \neq - \frac{c}{d} \\
ad - bc \neq 0
$$

Hmmm, the $ad - bc$ looks oddly familiar, what could it be?

Speaking about methodology improvement, we should also try to improve the algorithms we used. For example, is there some better algorithm than Gaussian elimination for solving linear equations?

```python

```

## Introduction to LU Decomposition

Last week we developed and implemented the Gaussian elimination method to solve the linear matrix system ($A\pmb{x}=\pmb{b}$).

This week we will consider a closely related solution method: *LU decomposition* or *LU factorisation*.

Both are examples of *direct* solution methods - in the next lecture we will consider the alternate approach to solve linear systems, namely *iterative* or *indirect* methods.

## LU decomposition - motivation

Last week we implemented Gaussian elimination to solve the matrix system with *one* RHS vector $\pmb{b}$.

We often have to deal with problems where we have multiple RHS vectors, all with the same matrix $A$.

We could call the same code (e.g. our 'upper_triangle' and 'back_substitution' codes from last lecture) multiple times to solve all of these corresponding linear systems, but note that as the elimination algorithm is actually performing operations based upon (the same) $A$ each time, we would actually be wasting time repeating exactly the same operations - this is therefore clearly not an efficient solution to this problem.

We could easily generalise our Gaussian elimination/back substitution algorithms to include multiple RHS column vectors in the augmented system, perform the same sequence of row operations (but now only once) to transform the matrix to upper-triangular form, and then perform back substitution on each of the transformed RHS vectors from the augmented system - cf. the use of Gaussian elimination to compute the inverse to a matrix by placing the identity on the right of the augmented system.

However, it is often the case that each RHS vector depends on the solutions to the matrix systems obtained from some or all of the earlier RHS vectors, and so this generalisation would not work in this case. For example, our discrete system could be of the form

$$A\pmb{x}^{n+1} = \pmb{b}(\pmb{x}^n, \ldots)$$

where $\pmb{x}^{n+1}$ is the numerical solution of a (ordinary or partial) differential equation at time level $n+1$, the RHS is a function of the the solution at the previous time level $n$ (and possibly other things such as focing terms, represented by the $\ldots$ above. $A$ is then a discretisation of the differential equation written in the form that it maps the old solution to the new one. Time stepping this problem invovles solving the same linear system multiple times with different RHSs.

[NB. this is true if our model is *linear*, in the nonlinear case $A$ would be updated evey time step].

Ok, so Gaussian elimination is nice, but way too inefficient, and basically, maybe we could do better?

## LU decomposition - theory

To deal with this situation efficiently we *decompose* or *factorise* the matrix $A$ in such a way that it is cheap to compute a new solution vector $\pmb{x}$ for any given RHS vector $\pmb{b}$.  This decomposition involves a lower- and an upper-triangular matrix, hence the name LU decomposition. These matrices essentially *encode* the steps conducted in Gaussian elimination, so we don't have to explicilty conduct all of the operations again and again.

Basically, we assume (well, there is actual rigorous mathematical proof that this can almost always be done) that our matrix **A** can be expressed as the matrix multiplication result of 2 other matrices, one in the upper triangle form, represented by the letter **U** and the other one in the lower triangle form, represented by the letter **L**. Since **U** is a upper triangle matrix, all entries below the diagonal are $0$, and for **L**, which is a lower triangle matrix, all entries above the diagonal are $0$.

Mathematically, let's assume that we have already found/constructed a lower-triangular matrix ($L$ - where all entries above the diagonal are zero) and an upper-triangular matrix ($U$ - where all entries below the diagonal are zero) such that we can write

$$ A = LU $$

To prove this, let's say that

In this case the matrix system we need to solve for $\pmb{x}$ becomes

$$ A\pmb{x} = \pmb{b} \iff (LU)\pmb{x} = L(U\pmb{x}) = \pmb{b} $$

Notice that the matrix-vector product $U\pmb{x}$ is itself a vector, let's call it $\pmb{c}$ for the time-being (i.e.
$\pmb{c}=U\pmb{x}$). Basically $U\pmb{x}$ is a column like matrix. To illustrate

Let's take a 3 by 3 matrix for example for $U$ and $x$ will correspndingly store 3 variables.

We can generalize a 3 by 3 $U$ as below, with $a,b,c,d,e,f$ being arbitrary constants, remebering that $U$ is a upper triangle matrix, so all values below the diagonal are zero

$$
\begin{equation*}
U =
\begin{pmatrix}
a & b & c \\
0 & d & e \\
0 & 0 & f
\end{pmatrix}
\end{equation*}
$$

We also have our matrix $x$ where we store the variable (Yeah, I know it's hard getting used to $x$ as a matrix to store variables. If you are annoyed by the confusion, you could always replace the name $x$ with the name "Matrix for storing variables")

$$
\begin{equation*}
\pmb{x}_{a.k.aThe Matrix for Storing the Variables} =
\begin{pmatrix}
x \\
y \\
z
\end{pmatrix}
\end{equation*}
$$

Then, if we do $Ux$, we get

$$
\begin{equation*}
\pmb{Ux} =
\begin{pmatrix}
a & b & c \\
0 & d & e \\
0 & 0 & f
\end{pmatrix}
\begin{pmatrix}
x \\
y  \\
z
\end{pmatrix}
=
\begin{pmatrix}
ax + by + cz \\
0x + dy + ez  \\
0z + 0y + fz
\end{pmatrix}
\end{equation*}
$$

We understand that $\pmb{Ux}$ is a column matrix, a.k.a a vector matrix, a.k.a a vector, which has 3 rows and 1 column.

The above system then reads

$$ L\pmb{c} = \pmb{b} $$

where $L$ is a matrix and $\pmb{c}$ is an unknown.

Note: The $L$ and $U$ here are not same as the one in the previous lecture in Gaussian elimination.

Previous Lecture
$$
\pmb{Upper Triangle Form_{Gaussian} x = B}
$$

This Lecture
$$
\pmb{LU x = B}
$$

Note: The $L$ and $U$ here are not same as the one in the previous lecture in Gaussian elimination.

As $L$ is in lower-triangular form we can use forward substitution (generalising the back subsitution algorithm/code we developed last week) to very easily find $\pmb{c}$ in relatively few operations (we don't need to call the entire Gaussian elimination algorithm).

Once we know $\pmb{c}$ we then solve the second linear system

$$ U\pmb{x} = \pmb{c} $$

where now we can use the fact that $U$ is upper-triangular to use our back substitution algorithm again very efficiently to give the solution $\pmb{x}$ we require.

So for a given $\pmb{b}$ we can find the corresponding $\pmb{x}$ very efficiently, we can therefore do this repeatedly as each new $\pmb{b}$ is given to us.

Our challenge is therefore to find the matrices $L$ and $U$ allowing us to perform the decomposition $A=LU$.

1. We assume that we have already found $\pmb{L}$ and $\pmb{U}$. Our objective is to find $\pmb{x}$, i.e. we aim to find the unknowns stored in $\pmb{x}$ which we will find by finding $\pmb{x}$ We begin with the matrix form

$$
\pmb{Ax =b}
$$

2. Since we know that $\pmb{A = LU}$, we can write

$$
\pmb{LUx = b}
$$

3. We know that $\pmb{Ux}$ is a column vector, and we can set a column vector $\pmb{c}$ and make it $\pmb{c =Ux}$, thus we write

$$
\pmb{Lc = b}
$$

4. Since we know $\pmb{L}$, we will see how we find $\pmb{L}$ and $\pmb{U}$ later, we know that $\pmb{L}$ is applied to column matrix and results in another column matrix, thus we can use back substitution to very easily find $\pmb{c}$

5. Once we have found $\pmb{c}$, since we have $\pmb{U}$, and we know that $\pmb{x}$ is also a vector, then, using backward substitution again, we can easily find $\pmb{x}$

## LU Decomposition Doolittle Algorithm

Recall the comment above on the $L$ and $U$ matrices encoding the steps taken in Gaussian elimination.  Let's see how this works through the development of the so-called *Doolittle algorithm*.

Let's consider an example matrix:

$$
  A=\begin{bmatrix}
    {\color{black}5} & {\color{black}7} & {\color{black}5} & {\color{black}9}\\
{\color{black}5} & {\color{black}14} & {\color{black}7} & {\color{black}10}\\
{\color{black}20} & {\color{black}77} & {\color{black}41} & {\color{black}48}\\
{\color{black}25} & {\color{black}91} & {\color{black}55} & {\color{black}67}\\
    \end{bmatrix}
$$

the first step of Gaussian elimination is to set the
sub-diagonal elements in the first column to zero by subtracting multiples of
the first row from each of the subsequent rows.

For this example, using the symbolic notiation from last week
this requires the row operations

\begin{align}
Eq. (2) &\leftarrow Eq. (2) - 1\times Eq. (1)\\
Eq. (3) &\leftarrow Eq. (3) - 4\times Eq. (1)\\
Eq. (4) &\leftarrow Eq. (4) - 5\times Eq. (1)\\
\end{align}

or mathematically, and for each element of the matrix (remembering that we are adding rows together - while one of
the entries of a row will end up being zero, this also has the consequence of updating the rest of the values in that row, hence the iteration over $j$ below):

\begin{align}
A_{2j} &\leftarrow A_{2j} - \frac{A_{21}}{A_{11}} A_{1j} = A_{2j} - \frac{5}{5} \times A_{1j}, \quad j=1,2,3,4\\
A_{3j} &\leftarrow A_{3j} - \frac{A_{31}}{A_{11}} A_{1j} = A_{3j} - \frac{20}{5} \times A_{1j}, \quad j=1,2,3,4\\
A_{4j} &\leftarrow A_{4j} - \frac{A_{41}}{A_{11}} A_{1j} = A_{4j} - \frac{25}{5} \times A_{1j}, \quad j=1,2,3,4\\
\end{align}

Notice that we can also write these exact operations on elements in terms of multiplication by a carefully chosen lower-triangular matrix where the non-zero's below the diagonal are restricted to a single column, e.g. for the example above

$$
  \begin{bmatrix}
    {\color{black}1} & {\color{black}0} & {\color{black}0} & {\color{black}0}\\
    {\color{Orange}{-1}} & {\color{black}1} & {\color{black}0} & {\color{black}0}\\
    {\color{Orange}{-4}} & {\color{black}0} & {\color{black}1} & {\color{black}0}\\
    {\color{Orange}{-5}} & {\color{black}0} & {\color{black}0} & {\color{black}1}\\
  \end{bmatrix}\qquad\times\qquad\begin{bmatrix}
    {\color{black}5} & {\color{black}7} & {\color{black}5} & {\color{black}9}\\
    {\color{black}5} & {\color{black}14} & {\color{black}7} & {\color{black}10}\\
    {\color{black}20} & {\color{black}77} & {\color{black}41} & {\color{black}48}\\
    {\color{black}25} & {\color{black}91} & {\color{black}55} & {\color{black}67}\\
  \end{bmatrix}\qquad=\qquad\begin{bmatrix}
    {\color{black}5} & {\color{black}7} & {\color{black}5} & {\color{black}9}\\
    {\color{blue}{0}} & {\color{blue}{7}} & {\color{blue}{2}} & {\color{blue}{1}}\\
    {\color{blue}{0}} & {\color{blue}{49}} & {\color{blue}{21}} & {\color{blue}{12}}\\
    {\color{blue}{0}} & {\color{blue}{56}} & {\color{blue}{30}} & {\color{blue}{22}}\\
  \end{bmatrix}
$$

The lower-triangular matrix (let's call this one $L_0$) is thus encoding the first step in Gaussian elimination.

The next step involves taking the second row of the updated matrix as the new pivot (we will ignore partial pivoting for simplicity), and subtracting multiples of this row from those below in order to set the zeros below the diagonal in the second column to zero.

Note that thios step can be achieved here with the multiplication by the following lower-triangular matrix (call this one $L_1$)

\begin{equation*}
  \begin{bmatrix}
    {\color{black}1} & {\color{black}0} & {\color{black}0} & {\color{black}0}\\
    {\color{black}0} & {\color{black}1} & {\color{black}0} & {\color{black}0}\\
    {\color{black}0} & {\color{Orange}{-7}} & {\color{black}1} & {\color{black}0}\\
    {\color{black}0} & {\color{Orange}{-8}} & {\color{black}0} & {\color{black}1}\\
  \end{bmatrix}\qquad\times\qquad\begin{bmatrix}
    {\color{black}5} & {\color{black}7} & {\color{black}5} & {\color{black}9}\\
    {\color{black}0} & {\color{black}7} & {\color{black}2} & {\color{black}1}\\
    {\color{black}0} & {\color{black}49} & {\color{black}21} & {\color{black}12}\\
    {\color{black}0} & {\color{black}56} & {\color{black}30} & {\color{black}22}\\
  \end{bmatrix}\qquad=\qquad\begin{bmatrix}
    {\color{black}5} & {\color{black}7} & {\color{black}5} & {\color{black}9}\\
    {\color{black}0} & {\color{black}7} & {\color{black}2} & {\color{black}1}\\
    {\color{black}0} & {\color{blue}{0}} & {\color{blue}{7}} & {\color{blue}{5}}\\
    {\color{black}0} & {\color{blue}{0}} & {\color{blue}{14}} & {\color{blue}{14}}\\
  \end{bmatrix}
\end{equation*}

and finally for this example to get rid the of the leading 14 on the final row:

\begin{equation*}
  \begin{bmatrix}
    {\color{black}1} & {\color{black}0} & {\color{black}0} & {\color{black}0}\\
    {\color{black}0} & {\color{black}1} & {\color{black}0} & {\color{black}0}\\
    {\color{black}0} & {\color{black}0} & {\color{black}1} & {\color{black}0}\\
    {\color{black}0} & {\color{black}0} & {\color{Orange}{-2}} & {\color{black}{1}}\\
  \end{bmatrix}\qquad\times\qquad\begin{bmatrix}
    {\color{black}5} & {\color{black}7} & {\color{black}5} & {\color{black}9}\\
    {\color{black}0} & {\color{black}7} & {\color{black}2} & {\color{black}1}\\
    {\color{black}0} & {\color{black}0} & {\color{black}7} & {\color{black}5}\\
    {\color{black}0} & {\color{black}0} & {\color{black}14} & {\color{black}14}\\
  \end{bmatrix}\qquad=\qquad\begin{bmatrix}
    {\color{black}5} & {\color{black}7} & {\color{black}5} & {\color{black}9}\\
    {\color{black}0} & {\color{black}7} & {\color{black}2} & {\color{black}1}\\
    {\color{black}0} & {\color{black}0} & {\color{black}7} & {\color{black}5}\\
    {\color{black}0} & {\color{black}0} & {\color{blue}{0}} & {\color{blue}{4}}\\
  \end{bmatrix}
\end{equation*}

where this lower triangular matrix we will call $L_2$, and the RHS matrix is now in upper-triangular form as we expect from Gaussian elimination (call this $U$).

In summary, the above operations (starting from $A$, first multipling pn the left by $L_0$, the multiplying the result on the left by $L_1$ and so on to arrive at an upper trianglualr matrix $U$) can be written as

$$ L_2(L_1(L_0A)) = U $$

 Note that these special lower-triangular matrices $L_0, \ldots$ are all examples of what is known as *atomic* lower-triangular matrices: a special form of unitriangular matrix - the diagonals are unity, where the off-diagonal entries are all zero apart from in a single column.

Notice that for these special, simple matrices, their inverse is simply the original with the sign of those off-diagnonals changed:

$$
\left[
  \begin{array}{rrrrrrrrr}
    1      & 0      & \cdots      &        &              &        &         &   & 0 \\
    0      & 1      & 0           & \cdots &              &        &         &   & 0 \\
    0      & \ddots & \ddots      & \ddots &              &        &         &   &  \vdots \\
    \vdots & \ddots & \ddots      & \ddots &              &        &         &   &  \\
           &        &             &   0    &   1          &  0     &         & &  &  \\
           &        &             &   0    &   l_{i+1,i}  &  1     &  \ddots &   &  &  \\
           &        &             &   0    &   l_{i+2,i}  &  0     &  \ddots &   & &  \\
           &        &             & \vdots &   \vdots     & \vdots &  \ddots &   & 0 &  \\
    0      & \cdots &             & 0      &   l_{n,i}    & 0      &  \cdots & 0 & 1 &  \\
\end{array}
\right]^{-1}
=
\left[
  \begin{array}{rrrrrrrrr}
    1      & 0      & \cdots      &        &              &        &         &   & 0 \\
    0      & 1      & 0           & \cdots &              &        &         &   & 0 \\
    0      & \ddots & \ddots      & \ddots &              &        &         &   &  \vdots \\
    \vdots & \ddots & \ddots      & \ddots &              &        &         &   &  \\
           &        &             &   0    &   1          &  0     &         & &  &  \\
           &        &             &   0    &   -l_{i+1,i}  &  1     &  \ddots &   &  &  \\
           &        &             &   0    &   -l_{i+2,i}  &  0     &  \ddots &   & &  \\
           &        &             & \vdots &   \vdots     & \vdots &  \ddots &   & 0 &  \\
    0      & \cdots &             & 0      &   -l_{n,i}    & 0      &  \cdots & 0 & 1 &  \\
\end{array}
\right]
$$

### Exercise 6.1: lower-triangular matrices

Convince yourselves of the following facts:

* The above statement on what the inverse of the special $L$ matrices is - check on a small example.

* The multiplication of arbitrary lower-triangular square matrices is also lower-triangular.

* $L_2(L_1(L_0A)) = U \implies A = L_0^{-1}(L_1^{-1}(L_2^{-1}U))$

* and hence that $A=LU$ where $U$ is the upper-triangular matrix found at the end of Guassian elimination, and where $L$ is the
following  matrix
$$ L = L_0^{-1}L_1^{-1}L_2^{-1} $$

* Finally, compute this product of these lower-triangular matrices to show that
$$L =
  \begin{bmatrix}
    {\color{black}1} & {\color{black}0} & {\color{black}0} & {\color{black}0}\\
    {\color{black}{1}} & {\color{black}1} & {\color{black}0} & {\color{black}0}\\
    {\color{black}{4}} & {\color{black}7} & {\color{black}1} & {\color{black}0}\\
    {\color{black}{5}} & {\color{black}8} & {\color{black}2} & {\color{black}1}\\
  \end{bmatrix}
$$
i.e. that the multiplication of these individual atomic matrices (importantly in this order) simply merges the entries from the non-zero columns of each atomic matrix, and hence is both lower-triangular, as well as trivial to compute.

 So now, by this slide, we have found $\pmb{L}$, Hooray!

## LU Decomposition Implementation

So we can build an LU code easily from our Gaussian elimination code from last lecture which already works out and performs these tasks.

The final $U$ matrix we need here is as was already constructed through Gaussian elimination, the entries of $L$ we need are simply the ${A_{ik}}/{A_{kk}}$ multipliers we computed as part of the elimination, but threw away previously.

For a given pivot row $k$, for each of these multipliers (for every row below the pivot), as we compute them we know that we are going to transform the augmented matrix in order to achieve a new zero below the diagonal - we can store each multiplier in this position before moving on to the following row, we implicitly know that the diagonals of $L$ will be unity and so don't need to store these (and noting that we don't actually have a space for them anyway!). We then move on to the following pivot row, replacing the zeros in the corresponding column we are zero'ing, but again using the now spare space to store the multipliers.

For example, for the case above

$$ A =
  \begin{bmatrix}
    {\color{black}5} & {\color{black}7} & {\color{black}5} & {\color{black}9}\\
    {\color{black}5} & {\color{black}14} & {\color{black}7} & {\color{black}10}\\
    {\color{black}20} & {\color{black}77} & {\color{black}41} & {\color{black}48}\\
    {\color{black}25} & {\color{black}91} & {\color{black}55} & {\color{black}67}\\
  \end{bmatrix}\quad\rightarrow\quad
  \begin{bmatrix}
    {\color{black}5} & {\color{black}7} & {\color{black}5} & {\color{black}9}\\
    {\color{blue}{1}} & {\color{black}{14 - 1\times7}} & {\color{black}{7-1\times5}} & {\color{black}{10-1\times9}}\\
    {\color{blue}{4}} & {\color{black}{77 - 4\times7}} & {\color{black}{41-4\times5}} & {\color{black}{48-4\times9}}\\
    {\color{blue}{5}} & {\color{black}{91 - 5\times7}} & {\color{black}{55-5\times5}} & {\color{black}{67-5\times9}}\\
  \end{bmatrix}\quad\rightarrow\quad
  \begin{bmatrix}
    {\color{black}5} & {\color{black}7} & {\color{black}5} & {\color{black}9}\\
    {\color{blue}{1}} & {\color{black}{7}} & {\color{black}{2}} & {\color{black}{1}}\\
    {\color{blue}{4}} & {\color{black}{49}} & {\color{black}{21}} & {\color{black}{12}}\\
    {\color{blue}{5}} & {\color{black}{56}} & {\color{black}{30}} & {\color{black}{22}}\\
  \end{bmatrix}\quad\rightarrow\quad
  \begin{bmatrix}
    {\color{black}5} & {\color{black}7} & {\color{black}5} & {\color{black}9}\\
    {\color{blue}{1}} & {\color{black}{7}} & {\color{black}{2}} & {\color{black}{1}}\\
    {\color{blue}{4}} & {\color{blue}{7}} & {\color{black}{21-2\times7}} & {\color{black}{12-1\times7}}\\
    {\color{blue}{5}} & {\color{blue}{8}} & {\color{black}{30-2\times8}} & {\color{black}{22-1\times8}}\\
  \end{bmatrix}\quad\rightarrow\quad
  \begin{bmatrix}
    {\color{black}5} & {\color{black}7} & {\color{black}5} & {\color{black}9}\\
    {\color{blue}1} & {\color{black}7} & {\color{black}2} & {\color{black}1}\\
    {\color{blue}4} & {\color{blue}{7}} & {\color{black}{7}} & {\color{black}{5}}\\
    {\color{blue}5} & {\color{blue}{8}} & {\color{black}{14}} & {\color{black}{14}}\\
  \end{bmatrix}\quad\rightarrow\quad
  \begin{bmatrix}
    {\color{black}5} & {\color{black}7} & {\color{black}5} & {\color{black}9}\\
    {\color{blue}1} & {\color{black}7} & {\color{black}2} & {\color{black}1}\\
    {\color{blue}4} & {\color{blue}{7}} & {\color{black}{7}} & {\color{black}{5}}\\
    {\color{blue}5} & {\color{blue}{8}} & {\color{blue}{2}} & {\color{black}{14-2\times5}}\\
  \end{bmatrix}\quad\rightarrow\quad
  \begin{bmatrix}
    {\color{black}5} & {\color{black}7} & {\color{black}5} & {\color{black}9}\\
    {\color{blue}1} & {\color{black}7} & {\color{black}2} & {\color{black}1}\\
    {\color{blue}4} & {\color{blue}7} & {\color{black}7} & {\color{black}5}\\
    {\color{blue}5} & {\color{blue}8} & {\color{blue}{2}} & {\color{black}{4}}\\
  \end{bmatrix}
  = [\color{blue}L\backslash U]
$$

We find that

$$
U =
  \begin{bmatrix}
    {\color{black}5} & {\color{black}7} & {\color{black}5} & {\color{black}9}\\
    {\color{black}0} & {\color{black}7} & {\color{black}2} & {\color{black}1}\\
    {\color{black}0} & {\color{black}0} & {\color{black}7} & {\color{black}5}\\
    {\color{black}0} & {\color{black}0} & {\color{black}0} & {\color{black}{4}}\\
  \end{bmatrix}
$$

### Exercise 6.2: LU decomposition

Starting from your Gaussian elimination code produce a new code to compute the LU decomposition of a matrix. First, store $L$ and $U$ in two different matrices; you could then try a version where you store the entries of both $L$ and $U$ in $A$ as described above.

 So now, by this point, we have found $\pmb{U}$, Hooray!

 After finding both $\pmb{L}$ and $\pmb{U}$, we could use our aforementioned method to solve equations!

##  V.  Partial Pivoting

At the end of last week we commented that a problem could occur in a numeical implementation of our algorithm if we had a situation where the $A_{kk}$ we divide through by in the Gaussian elimination and/or back substitution algorithms might be zero (or close to zero).

Using Gaussian elimination as an example, let's again consider the algorithm mid-way working on an arbitrary matrix system, i.e. assume that the first $k$ rows have already been transformed into upper-triangular form, while the equations/rows below are not yet in this form:

$$
\left[
  \begin{array}{rrrrrrr|r}
    A_{11} & A_{12} & A_{13} & \cdots & A_{1k}  & \cdots & A_{1n} & b_1 \\
    0      & A_{22} & A_{23} & \cdots & A_{2k} & \cdots & A_{2n} & b_2 \\
    0      & 0      & A_{33} & \cdots & A_{3k}  & \cdots & A_{3n} & b_3 \\
    \vdots & \vdots & \vdots & \ddots & \vdots  & \ddots & \vdots & \vdots \\
\hdashline
    0      & 0      & 0      & \cdots & A_{kk}  & \cdots & A_{kn} & b_k \\
    \vdots & \vdots & \vdots & \ddots & \vdots  & \ddots & \vdots & \vdots \\
    0      & 0      & 0      & \cdots & A_{nk}  & \cdots & A_{nn} & b_n \\
\end{array}
\right]
$$

Note we have drawn the horizontal dashed line one row higher, as we are not going to blindly asssume that it is wise to take the current row $k$ as the pivot row, and $A_{kk}$ as the so-called pivot element.

*Partial pivoting* selects the best pivot (row or element) as the one where the $A_{ik}$ (for $i\ge k$) value is largest (relative to the other values of components in its own row $i$), and then swaps this row with the current $k$ row.

To generalise our codes above we would simply need to search for this row, and perform the row swap operation.

Python's `scipy.linalg` library has its own implementation of the LU decomposition, that uses partial pivoting.

Ok, so let's illustrate partial pivoting with an example

Suppose that we have the matrix $\pmb{A}$ from before

$$
A =
  \begin{bmatrix}
    5 & 7 & 5 &  9 \\
    5 & 14 & 7 & 10 \\
    20 & 77 &41 & 48 \\
    25 & 91 & 55 & 67 \\
  \end{bmatrix}
$$

And then we do some timeskipping and skip to partway through

$$
A =
  \begin{bmatrix}
    5 & 7 & 5 &  9 \\
    0 & 7 & 2 & 1 \\
    0 & 49 &21 & 12 \\
    0 & 56 & 30 & 22 \\
  \end{bmatrix}
$$

Now, magic happens for the sake of illustrating the Partial Pivoting
Magic magic poof poof and magically $7$ becomes $0$

$$
A =
  \begin{bmatrix}
    5 & 7 & 5 &  9 \\
    0 & 0 & 2 & 1 \\
    0 & 49 &21 & 12 \\
    0 & 56 & 30 & 22 \\
  \end{bmatrix}
$$

Suppose that the matrix storing our unknowns is

$$
\pmb{x} =
\begin{bmatrix}
  x \\
  y \\
  z \\
  letter\;\;after\;\;z
\end{bmatrix}
$$

And the matrix storing the results of our equations, after all of the Gaussian elimination previously that have modified the matrix $\pmb{b}$

$$
\pmb{b} =
\begin{bmatrix}
  b_1 \\
  b_2 \\
  b_3 \\
  b_4
\end{bmatrix}
$$

Writing this in equation this would be

$$
5x + 7y + 5z + 9(letter\;\;after\;\;z) = b_1 \\
0x + 0y + 2z + 1(letter\;\;after\;\;z) = b_2 \\
0x + 49y + 21z + 12(letter\;\;after\;\;z) = b_3  \\
0x + 56y + 30z + 22(letter\;\;after\;\;z) = b_4
$$

Now, we are going to use equation 2 and equation 3

$$
A =
  \begin{bmatrix}
    5 & 7 & 5 &  9 \\
    0 & 0 & 2 & 1 \\
    0 & 49 &21 & 12 \\
    0 & 56 & 30 & 22 \\
  \end{bmatrix}
$$

The coefficient of the second unknown in the second equation is 0. The coefficient of the second unknown in the third equation is 49. Thus, I will divide the coefficient of the 3rd equation with the coefficient of the 2nd equation, and then I would multiply the coefficient of the 3rd equation by the quotient. As such, I will divide $49/0$, and then, yup I will get an error since you cannot divide by $0$. It does not neccesarily have to be $0$ to give errors, even if it is very close to $0$, it will still give an error.
Thus what can I do?

Well, I need two equations. I cannot use the equation 2 because there is a $0$. Then, I will use equation 3 and equation 4. Therefore, let's swap the position of equation 4 and equation 2.

$$
A =
  \begin{bmatrix}
    5 & 7 & 5 &  9 \\
    0 & 56 & 30 & 22 \\
    0 & 49 &21 & 12 \\
    0 & 0 & 2 & 1 \\
  \end{bmatrix}
$$

Remember: when you swap rows in matrix $\pmb{A}$, you must also swap rows in the matrix $\pmb{b}$. Matrix $\pmb{x}$ does not have to swapped since the order of the variables has not changed.

The matrix storing our unknowns remains

$$
\pmb{x} =
\begin{bmatrix}
  x \\
  y \\
  z \\
  letter\;\;after\;\;z
\end{bmatrix}
$$

And the matrix storing the results of our equations

$$
\pmb{b} =
\begin{bmatrix}
  b_1 \\
  b_4 \\
  b_3 \\
  b_2
\end{bmatrix}
$$

Writing this in equation this would be

$$
5x + 7y + 5z + 9(letter\;\;after\;\;z) = b_1 \\
0x + 56y + 30z + 22(letter\;\;after\;\;z) = b_4 \\
0x + 49y + 21z + 12(letter\;\;after\;\;z) = b_3  \\
0x + 0y + 2z + 1(letter\;\;after\;\;z) = b_2
$$

Inspeacing this equation, we see that the equation is essentially the same as it was before the swap, so we can say that our action to swap it is a valid action. I have rewritten the equation before the swap as reference

$$
5x + 7y + 5z + 9(letter\;\;after\;\;z) = b_1 \\
0x + 0y + 2z + 1(letter\;\;after\;\;z) = b_2 \\
0x + 49y + 21z + 12(letter\;\;after\;\;z) = b_3  \\
0x + 56y + 30z + 22(letter\;\;after\;\;z) = b_4
$$

Now let's continue, after we finished swapping, and we will focus only on getting to the upper triangle form, and only on matrix $\pmb{A}$. The matrix $\pmb{b}$ will also change accordingly as detailed in the previous lecture, but let's focus on matrix $\pmb{A}$

$$
A =
  \begin{bmatrix}
    5 & 7 & 5 &  9 \\
    0 & 56 & 30 & 22 \\
    0 & 56 & 24 & 96/7 \\
    0 & 0 & 2 & 1 \\
  \end{bmatrix}
$$

$$
A =
  \begin{bmatrix}
    5 & 7 & 5 &  9 \\
    0 & 56 & 30 & 22 \\
    0 & 0 & -6 & 96/7 -22 \\
    0 & 0 & 2 & 1 \\
  \end{bmatrix}
$$

We will not encounter the problem with dividing by zero anymore, at least for this matrix !

Note : this matrix was nice and only required one swap. But if your matrix has lots of $0$ or very small numbers, then maybe you might need more swaps. You might also want to nondimensionalize that matrix! That is, of course assuming that the equations contain enough information to actually allow you to solve the equation. You could use the determinant, or the Gauss-Jordan in the previous lecture to check if there is actually a solution. You should always check the solvability of an equation before solving it, since trying to solve an equation which cannot be solved is wasted effort.  You may spend days debugging your code and scratching your head as to what went wrong, when in fact, the equations you are solving may not even be able to give a solution, i.e. ill conditioned

Note: You could use conditionals to control the swapping or the skipping of rows. Although the example here used $0$, it could be a very small number too, and not neccesary $0$ that could cause the problem, so maybe it would be nice to compare the conditional to a certain threshold to decided whether to swap or not.

```python
A=np.array([[ 5., 7.,   5.,  9.],
               [ 5., 14.,  7., 10.],
               [20., 77., 41., 48.],
               [25., 91. ,55., 67.]])
print(A)
```

    [[ 5.  7.  5.  9.]
     [ 5. 14.  7. 10.]
     [20. 77. 41. 48.]
     [25. 91. 55. 67.]]

```python
P,L,U=sl.lu(A)

# P here is a 'permutation matrix' that performs swaps based upon partial pivoting
print(P)
```

    [[0. 1. 0. 0.]
     [0. 0. 0. 1.]
     [0. 0. 1. 0.]
     [1. 0. 0. 0.]]

```python
# the lower-triangular matrix
print(L)
```

    [[ 1.          0.          0.          0.        ]
     [ 0.2         1.          0.          0.        ]
     [ 0.8        -0.375       1.          0.        ]
     [ 0.2         0.375       0.33333333  1.        ]]

```python
# the upper-triangular matrix
print(U)
```

    [[ 25.          91.          55.          67.        ]
     [  0.         -11.2         -6.          -4.4       ]
     [  0.           0.          -5.25        -7.25      ]
     [  0.           0.           0.           0.66666667]]

```python
# double check that P*L*U does indeed equal A
print(P @ L @ U)
```

    [[ 5.  7.  5.  9.]
     [ 5. 14.  7. 10.]
     [20. 77. 41. 48.]
     [25. 91. 55. 67.]]

Looking at the form of $P$ above, we can re-order the rows in advance and consider the LU decomposition of the matrix where $P=I$, as below. As we haven't bothered implementing pivoting ourselves, check that your LU implementation recreates the $A$, $L$ and $U$ below.

```python
A=np.array([[25. ,91. ,55. ,67.],
               [ 5.,  7.,  5.,  9.],
               [20., 77., 41., 48.],
               [ 5., 14., 7.,  10.]])
print(A)
```

    [[25. 91. 55. 67.]
     [ 5.  7.  5.  9.]
     [20. 77. 41. 48.]
     [ 5. 14.  7. 10.]]

```python
P,L,U=sl.lu(A)
# P now should be the identity as pivoting no longer actually actions any row swaps with this A
print(P)
```

    [[1. 0. 0. 0.]
     [0. 1. 0. 0.]
     [0. 0. 1. 0.]
     [0. 0. 0. 1.]]

```python
print(L)
```

    [[ 1.          0.          0.          0.        ]
     [ 0.2         1.          0.          0.        ]
     [ 0.8        -0.375       1.          0.        ]
     [ 0.2         0.375       0.33333333  1.        ]]

```python
print(U)
```

    [[ 25.          91.          55.          67.        ]
     [  0.         -11.2         -6.          -4.4       ]
     [  0.           0.          -5.25        -7.25      ]
     [  0.           0.           0.           0.66666667]]

```python
print(P @ L @ U)
```

    [[25. 91. 55. 67.]
     [ 5.  7.  5.  9.]
     [20. 77. 41. 48.]
     [ 5. 14.  7. 10.]]

### Exercise 6.3: Partial pivoting

Implement partial pivoting.

```python

```