 Tutorial 2

 Physics Informed Neural Networks Part 3

 PINN Navier Stokes Example

# Overview

This notebook is based on two papers: *[Physics-Informed Neural Networks:  A Deep LearningFramework for Solving Forward and Inverse ProblemsInvolving Nonlinear Partial Differential Equations](https://www.sciencedirect.com/science/article/pii/S0021999118307125)* and *[Hidden Physics Models:  Machine Learning of NonlinearPartial Differential Equations](https://www.sciencedirect.com/science/article/pii/S0021999117309014)* with the help of  Fergus Shone and Michael Macraild.

These tutorials will go through solving Partial Differential Equations using Physics Informed Neuaral Networks focusing on the Burgers Equation and a more complex example using the Navier Stokes Equation

**This introduction section is replicated in all PINN tutorial notebooks (please skip if you've already been through)**

If you have not already then in your repositoy directory please run the following code. Via the terminal (mac or linux) or gitbash (windows)

```bash
git submodule init
git submodule update --init --recursive
```
**If this does not work please clone the [PINNs](https://github.com/maziarraissi/PINNs) repository into your Physics_Informed_Neural_Networks folder**

Physics Informed Neural Networks

For a typical Neural Network using algorithims like gradient descent to look for a hypothesis, data is the only guide, however if the data is noisy or sparse and we already have governing physical models we can use the knowledge we already know to optamize and inform the algoithms. This can be done via [feature enginnering]() or by adding a physicall inconsistency term to the loss function.

## The very basics

If you know nothing about neural networks there is a [toy neural network python code example](https://github.com/cemac/LIFD_ENV_ML_NOTEBOOKS/tree/main/ToyNeuralNetwork) included in the [LIFD ENV ML Notebooks Repository]( https://github.com/cemac/LIFD_ENV_ML_NOTEBOOKS). Creating a 2 layer neural network to illustrate the fundamentals of how Neural Networks work and the equivlent code using the python machine learning library [tensorflow](https://keras.io/).

## Recommended reading

The in-depth theory behind neural networks will not be covered here as this tutorial is focusing on application of machine learning methods. If you wish to learn more here are some great starting points.

* [All you need to know on Neural networks](https://towardsdatascience.com/nns-aynk-c34efe37f15a)
* [Introduction to Neural Networks](https://victorzhou.com/blog/intro-to-neural-networks/)
* [Physics Guided Neural Networks](https://towardsdatascience.com/physics-guided-neural-networks-pgnns-8fe9dbad9414)
* [Maziar Rassi's Physics informed GitHub web Page](https://maziarraissi.github.io/PINNs/)

 Machine Learning Theory

## Physics informed Neural Networks

Neural networks work by using lots of data to calculate weights and biases from data alone to minimise the loss function enabling them to act as universal fuction approximators. However these loose their robustness when data is limited. However by using know physical laws or empirical validated relationships the solutions from neural networks can be sufficiently constrianed by disregardins no realistic solutions.

A Physics Informed Nueral Network considers a parameterized and nonlinear partial differential equation in the genral form;

\begin{align}
u_t + \mathcal{N}[u;  \lambda] = 0,   x \in \Omega, t \in [0,T],\\
\end{align}

where $\mathcal{u(t,x)}$ denores the hidden solution, $\mathcal{N}$ is a nonlinear differential operator acting on $u$, $\mathcal{\lambda}$ and $\Omega$ is a subset of $\mathbb{R}^D$ (the perscribed data). This set up an encapuslate a wide range of problems such as diffusion processes, conservation laws,  advection-diffusion-reaction  systems,  and  kinetic  equations and conservation laws.

Here we will go though this for the 1D Heat equation and Navier stokes equations

 Python

## Tensorflow

There are many machine learning python libraries available, [TensorFlow](https://www.tensorflow.org/) a is one such library. If you have GPUs on the machine you are using TensorFlow will automatically use them and run the code even faster!

## Further Reading

* [Running Jupyter Notebooks](https://jupyter.readthedocs.io/en/latest/running.html#running)
* [Tensorflow optimizers](https://www.tutorialspoint.com/tensorflow/tensorflow_optimizers.htm)

 Requirements

These notebooks should run with the following requirements satisfied

 Python Packages:

* Python 3
* tensorflow > 2
* numpy
* matplotlib
* scipy

 Data Requirements

This notebook referes to some data included in the git hub repositroy

**Contents:**

1. [1D Heat Equation Non ML Example](PINNs_1DHeatEquations_nonML.ipynb)
2. [1D Heat Equation PINN Example](PINNs_1DEquationExample.ipynb)
3. **[Navier-Stokes PINNs discovery of PDE’s](PINNs_Navier_Stokes_example.ipynb)**
4. [Navier-Stokes PINNs Hidden Fluid Mechanics](PINNs_NavierStokes_HFM.ipynb)

Load in all required modules (includig some auxillary code) and turn off warnings.

```python
# For readability: disable warnings
import warnings
warnings.filterwarnings('ignore')
```

```python
import sys
sys.path.insert(0, 'PINNs/Utilities/')
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
import scipy.io
from scipy.interpolate import griddata
import time
from itertools import product, combinations
from mpl_toolkits.mplot3d import Axes3D
from mpl_toolkits.mplot3d.art3d import Poly3DCollection
from mpl_toolkits.axes_grid1 import make_axes_locatable
import matplotlib.gridspec as gridspec
from time import time
import scipy.sparse as sp
import scipy.sparse.linalg as la
```

 Navier-Stokes inverse data driven discovery of PDE’s

Navier-Stokes equations describe the physics of many phenomena of scientific and engineering interest. They may be used to model the weather, ocean currents, water flow in a pipe and air flow around a wing. The Navier-Stokes equations in their full and simplified forms help with the design of aircraft and cars, the study of blood flow, the design of power stations, the analysis of the dispersion of pollutants, and many other applications. Let us consider the Navier-Stokes equations in two dimensions (2D) given explicitly by

\begin{equation}
u_t + \lambda_1 (u u_x + v u_y) = -p_x + \lambda_2(u_{xx} + u_{yy}),\\
v_t + \lambda_1 (u v_x + v v_y) = -p_y + \lambda_2(v_{xx} + v_{yy}),
\end{equation}

where $u(t, x, y)$ denotes the $x$-component of the velocity field, $v(t, x, y)$ the $y$-component, and $p(t, x, y)$ the pressure. Here, $\lambda = (\lambda_1, \lambda_2)$ are the unknown parameters. Solutions to the Navier-Stokes equations are searched in the set of divergence-free functions; i.e.,

\begin{equation}
u_x + v_y = 0.
\end{equation}

This extra equation is the continuity equation for incompressible fluids that describes the conservation of mass of the fluid. We make the assumption that

\begin{equation}
u = \psi_y,\ \ \ v = -\psi_x,
\end{equation}

for some latent function $\psi(t,x,y)$. Under this assumption, the continuity equation will be automatically satisfied. Given noisy measurements

\begin{equation}
\{t^i, x^i, y^i, u^i, v^i\}_{i=1}^{N}
\end{equation}

of the velocity field, we are interested in learning the parameters $\lambda$ as well as the pressure $p(t,x,y)$. We define $f(t,x,y)$ and $g(t,x,y)$ to be given by

\begin{equation}
\begin{array}{c}
f := u_t + \lambda_1 (u u_x + v u_y) + p_x - \lambda_2(u_{xx} + u_{yy}),\\
g := v_t + \lambda_1 (u v_x + v v_y) + p_y - \lambda_2(v_{xx} + v_{yy}),
\end{array}
\end{equation}

and proceed by jointly approximating

\begin{equation}
\begin{bmatrix}
\psi(t,x,y) & p(t,x,y)
\end{bmatrix}
\end{equation}

using a single neural network with two outputs. This prior assumption results into a [physics informed neural network](https://arxiv.org/abs/1711.10566)

\begin{equation}
\begin{bmatrix}
f(t,x,y) & g(t,x,y)
\end{bmatrix}.
\end{equation}

The parameters $\lambda$ of the Navier-Stokes operator as well as the parameters of the neural networks

\begin{equation}
\begin{bmatrix}
\psi(t,x,y) & p(t,x,y)
\end{bmatrix}
\end{equation}
and

\begin{equation}
\begin{bmatrix}
f(t,x,y) & g(t,x,y)
\end{bmatrix}
\end{equation}

can be trained by minimizing the mean squared error loss$

\begin{equation}
\begin{array}{rl}
MSE :=& \frac{1}{N}\sum_{i=1}^{N} \left(|u(t^i,x^i,y^i) - u^i|^2 + |v(t^i,x^i,y^i) - v^i|^2\right) \\
    +& \frac{1}{N}\sum_{i=1}^{N} \left(|f(t^i,x^i,y^i)|^2 + |g(t^i,x^i,y^i)|^2\right).
\end{array}
\end{equation}

```python
def xavier_init( size):
    in_dim = size[0]
    out_dim = size[1]
    xavier_stddev = np.sqrt(2/(in_dim + out_dim))
    return tf.Variable(tf.random.truncated_normal([in_dim, out_dim], stddev=xavier_stddev), dtype=tf.float32)

```

```python
def initialize_NN( layers):
    weights = []
    biases = []
    num_layers = len(layers)
    for l in range(0,num_layers-1):
        W = xavier_init(size=[layers[l], layers[l+1]])
        b = tf.Variable(tf.zeros([1,layers[l+1]], dtype=tf.float32), dtype=tf.float32)
        weights.append(W)
        biases.append(b)
    return weights, biases

```

# Initalise the neural network

`init` is called passing in the training data `x_train`, `y_train`, `t_train`, `u_train` and  `v_train` with information about the neural network layers

# Extract vars

`init` reformats some of the data and outputs model features that we need to pass into the training function `train`

# Advanced

Once you have run through the notebook once you may wish to alter the optamizer used in the `init()` function to see the large effect optamizer choice may have.

We've highlighted in the comments a number of possible optamizers to use from the [tf.compat.v1.train](https://www.tensorflow.org/api_docs/python/tf/compat/v1/train) module.
*This method was chosen to limit tensorflow version modifications required from the original source code*

You can learn more about different optamizers [here](https://towardsdatascience.com/optimizers-for-training-neural-network-59450d71caf6)

# init

```python
def init(x, y, t, u, v, layers):
        # This line of code is required to prevent some tensorflow errors arrising from the
        # inclusion of some tensorflw v 1 code
        tf.compat.v1.disable_eager_execution()
        X = np.concatenate([x, y, t], 1)
        # lb and ub denote lower and upper bounds on the inputs to the network
        # these bounds are used to normalise the network variables
        lb = X.min(0)
        ub = X.max(0)

        X = X

        x = X[:,0:1]
        y = X[:,1:2]
        t = X[:,2:3]

        u = u
        v = v

        layers = layers

        # Initialize NN
        weights, biases = initialize_NN(layers)

        # Initialize parameters
        lambda_1 = tf.Variable([0.0], dtype=tf.float32)
        lambda_2 = tf.Variable([0.0], dtype=tf.float32)

        # tf placeholders and graph
        ## This converts the data into a Tensorflow format
        sess = tf.compat.v1.Session(config=tf.compat.v1.ConfigProto(allow_soft_placement=True,
                                                     log_device_placement=True))

        x_tf = tf.compat.v1.placeholder(tf.float32, shape=[None, x.shape[1]])
        y_tf = tf.compat.v1.placeholder(tf.float32, shape=[None, y.shape[1]])
        t_tf = tf.compat.v1.placeholder(tf.float32, shape=[None, t.shape[1]])

        u_tf = tf.compat.v1.placeholder(tf.float32, shape=[None, u.shape[1]])
        v_tf = tf.compat.v1.placeholder(tf.float32, shape=[None, v.shape[1]])
        u_pred, v_pred, p_pred, f_u_pred, f_v_pred = net_NS(x_tf, y_tf, t_tf,lambda_1, lambda_2, weights, biases,lb, ub)

        loss = tf.reduce_sum(tf.square(u_tf - u_pred)) + \
            tf.reduce_sum(tf.square(v_tf - v_pred)) + \
            tf.reduce_sum(tf.square(f_u_pred)) + \
            tf.reduce_sum(tf.square(f_v_pred))

        ##############################################################################################
        #                                                                                #
        ## the optimizer is something that can be tuned to different requirements                    #
        ## we have not investigated using different optimizers, the orignal code uses L-BFGS-B which #
        ## is not tensorflow 2 compatible                                                            #
        #                                                                                #
        #  SELECT OPTAMIZER BY UNCOMMENTING OUT one of the below lines AND RERUNNING CODE            #
        #  You can alsoe edit the learning rate to see the effect of that                            #
        #                                                                                #
        ##############################################################################################

        learning_rate = 0.001
        optimizer = tf.compat.v1.train.MomentumOptimizer(learning_rate, 0.9)
        # optimizer = tf.compat.v1.train.AdagradOptimizer(learning_rate) # 8 %
        # optimizer = tf.compat.v1.train.ProximalGradientDescentOptimizer(learning_rate)
        # optimizer = tf.compat.v1.train.GradientDescentOptimizer(learning_rate)
        # optimizer = tf.compat.v1.train.AdadeltaOptimizer(learning_rate) # yeilds poor results
        # ptimizer = tf.compat.v1.train.FtrlOptimizer(learning_rate)
        # optimizer = tf.compat.v1.train.AdamOptimizer(learning_rate)

        # LEAVE THESE OPIMISERS ALONE
        optimizer_Adam = tf.compat.v1.train.AdamOptimizer()
        train_op_Adam = optimizer_Adam.minimize(loss)

        init = tf.compat.v1.global_variables_initializer()
        sess.run(init)
        xvars=[X,lb,ub,x,y,t,u,v]
        NNvars=[layers, weights, biases, lambda_1, lambda_2]
        tfvars=[sess, x_tf,y_tf, t_tf ,u_tf,v_tf]
        preds=[u_pred,v_pred, p_pred, f_u_pred, f_v_pred]
        optvars=[loss, optimizer,optimizer_Adam,train_op_Adam]

        return xvars,NNvars,tfvars,preds,optvars
```

`neural_net()` constructs the network Y where X is a matrix containing the input and output coordinates, i.e. x,t,u and X is normalised so that all values lie between -1 and 1, this improves training

`net_NS()` is where the PDE is encoded:

```python
def neural_net( X, weights, biases,lb, ub):

    num_layers = len(weights) + 1

    H = 2.0*(X - lb)/(ub - lb) - 1.0
    for l in range(0,num_layers-2):
        W = weights[l]
        b = biases[l]
        H = tf.tanh(tf.add(tf.matmul(H, W), b))
    W = weights[-1]
    b = biases[-1]
    Y = tf.add(tf.matmul(H, W), b)
    return Y

```

```python
def net_NS( x, y, t,lambda_1, lambda_2, weights, biases,lb,ub):

    psi_and_p = neural_net(tf.concat([x,y,t], 1), weights, biases, lb,ub)
    psi = psi_and_p[:,0:1]
    p = psi_and_p[:,1:2]

    u = tf.gradients(psi, y)[0]
    v = -tf.gradients(psi, x)[0]

    u_t = tf.gradients(u, t)[0]
    u_x = tf.gradients(u, x)[0]
    u_y = tf.gradients(u, y)[0]
    u_xx = tf.gradients(u_x, x)[0]
    u_yy = tf.gradients(u_y, y)[0]

    v_t = tf.gradients(v, t)[0]
    v_x = tf.gradients(v, x)[0]
    v_y = tf.gradients(v, y)[0]
    v_xx = tf.gradients(v_x, x)[0]
    v_yy = tf.gradients(v_y, y)[0]

    p_x = tf.gradients(p, x)[0]
    p_y = tf.gradients(p, y)[0]

    f_u = u_t + lambda_1*(u*u_x + v*u_y) + p_x - lambda_2*(u_xx + u_yy)
    f_v = v_t + lambda_1*(u*v_x + v*v_y) + p_y - lambda_2*(v_xx + v_yy)

    return u, v, p, f_u, f_v

```

# Load data and set input parameters

Once you have run through the notebook once you may wish to alter any the following

- number of data training points `N_train`
- number of layers in the network `layers`
- number of neurons per layer `layers`

to see the impact on the results

```python
N_train = 5000
# structure of network:
# 8 fully connected layers with 20 nodes per layer
layers = [3, 20, 20, 20, 20, 20, 20, 20, 20, 2]
```

```python
# Load Data
data = scipy.io.loadmat('PINNs/main/Data/cylinder_nektar_wake.mat')

U_star = data['U_star'] # N x 2 x T
P_star = data['p_star'] # N x T
t_star = data['t'] # T x 1
X_star = data['X_star'] # N x 2

N = X_star.shape[0]
T = t_star.shape[0]

# Rearrange Data
XX = np.tile(X_star[:,0:1], (1,T)) # N x T
YY = np.tile(X_star[:,1:2], (1,T)) # N x T
TT = np.tile(t_star, (1,N)).T # N x T

UU = U_star[:,0,:] # N x T
VV = U_star[:,1,:] # N x T
PP = P_star # N x T

x = XX.flatten()[:,None] # NT x 1
y = YY.flatten()[:,None] # NT x 1
t = TT.flatten()[:,None] # NT x 1

u = UU.flatten()[:,None] # NT x 1
v = VV.flatten()[:,None] # NT x 1
p = PP.flatten()[:,None] # NT x 1

######################################################################
######################## Noiseles Data ###############################
######################################################################
# Training Data
## We downsample the boundary data to leave N_train randomly distributed points
## This makes the training more difficult -
## if we used all the points then there is not much for the network to do!
idx = np.random.choice(N*T, N_train, replace=False)
x_train = x[idx,:]
y_train = y[idx,:]
t_train = t[idx,:]
u_train = u[idx,:]
v_train = v[idx,:]
```

If this fails you may need to restarted the notebook with a flag:
```bash

jupyter notebook --NotebookApp.iopub_data_rate_limit=1.0e10

```

# Initalise the nerual network

`init` is called passing in the training data `x_train`,  `y_train`, `u_train` and `v_train` with information about the neural network layers. The bound information `lb` `ub` is included in the `init()` function

# Extract vars

`init` reformats some of the data and outputs model features that we need to pass into the training function `train`

```python
xvars, NNvars, tfvars, preds, optvars = init(x_train, y_train, t_train, u_train, v_train, layers)
X, lb, ub, x, y, t, u, v = xvars
layers, weights, biases, lambda_1, lambda_2 = NNvars
sess, x_tf,y_tf, t_tf ,u_tf,v_tf = tfvars
u_pred,v_pred, p_pred, f_u_pred, f_v_pred = preds
loss, optimizer, optimizer_Adam, train_op_Adam = optvars
```

```python
def train(sess, nIter,x_tf, y_tf, t_tf,u_tf, v_tf,x, y, t,u, v, loss, train_op_Adam, optimizer, path):
    # Add ops to save and restore all the variables.
    # path = "modelckpts/NS/"
    netSaveDir = path
    saver = tf.compat.v1.train.Saver()
    tf_dict = {x_tf: x, y_tf: y, t_tf: t,
             u_tf: u, v_tf: v}

    start_time = time()
    for it in range(nIter):
        sess.run(train_op_Adam, tf_dict)

        # Print
        if it % 50 == 0:
            elapsed = time() - start_time
            loss_value = sess.run(loss, tf_dict)
            lambda_1_value = sess.run(lambda_1)
            lambda_2_value = sess.run(lambda_2)
            print('It: %d, Loss: %.3e, l1: %.3f, l2: %.5f, Time: %.2f' %
                (it, loss_value, lambda_1_value, lambda_2_value, elapsed))
            start_time = time()

        if it % 2000 == 0:
            save_path = saver.save(sess, netSaveDir + 'model_at_iter%s.ckpt'%(it))
            print('Model saved in path: %s' % save_path)

    print('Model saved in path: %s' % save_path)
    return optimizer.minimize(loss)

```

```python
def predict( x_star, y_star, t_star, u_pred, v_pred, p_pred):

    tf_dict = {x_tf: x_star, y_tf: y_star, t_tf: t_star}

    u_star = sess.run(u_pred, tf_dict)
    v_star = sess.run(v_pred, tf_dict)
    p_star = sess.run(p_pred, tf_dict)

    return u_star, v_star, p_star

def plot_solution(X_star, u_star, index):

    lb = X_star.min(0)
    ub = X_star.max(0)
    nn = 200
    x = np.linspace(lb[0], ub[0], nn)
    y = np.linspace(lb[1], ub[1], nn)
    X, Y = np.meshgrid(x,y)

    U_star = griddata(X_star, u_star.flatten(), (X, Y), method='cubic')

    plt.figure(index)
    plt.pcolor(X,Y,U_star, cmap = 'jet')
    plt.colorbar()

def axisEqual3D(ax):
    extents = np.array([getattr(ax, 'get_{}lim'.format(dim))() for dim in 'xyz'])
    sz = extents[:,1] - extents[:,0]
    centers = np.mean(extents, axis=1)
    maxsize = max(abs(sz))
    r = maxsize/4
    for ctr, dim in zip(centers, 'xyz'):
        getattr(ax, 'set_{}lim'.format(dim))(ctr - r, ctr + r)

```

```python
# set random seeds
np.random.seed(1234)
tf.random.set_seed(1234)

```

**Training might take a long time depending on value of Train_iterations**

If you set Train_iterations too low the end results will be garbage. 20000 was used to achieve excellent results in the original papers but this value is too high to run on a laptop.

* If you are using a machine with GPUs please set `Train_iterations=20000` to achieve the best results
* If you are using a well spec'ed laptop/computer and can leave this setting `Train_iterations=10000` should suffice (may take a while)
* If you are using a low spec'ed laptop/computer or cannont leave the code running `Train_interations=5000` is the reccomended values (high errors will remain)

```python
# Training
Train_iterations = 20000
```

# Loading Pre trained model option

If the training time is too slow you can skip the following line and load in a pretrained model instead set `loadweights = True` in the next cell. You can play around with different number of iterations to see the effects e.g. setting `saver.restore(sess, netSaveDir + 'model_at_iter15000.ckpt')`

```python
train(sess, Train_iterations,x_tf, y_tf, t_tf, u_tf, v_tf,x, y, t,u_train, v_train, loss, train_op_Adam, optimizer_Adam, "modelckpts/NS/")
```

```python
loadweights = False
if loadweights:
    print("loading pre trained model")
    netSaveDir = "modelckpts/NS/"
    saver = tf.compat.v1.train.Saver()
    saver.restore(sess, netSaveDir + 'model_at_iter19000.ckpt')
```

# Use trained model to predict from data sample

`predict` will predict `u`, `v` and `p` using the trained model

The `predict` function has an option `load=False` set by default. Alter this to `load=True` if you wish to load the previously trained model

```python
# Test Data
snap = np.array([100])
x_star = X_star[:,0:1]
y_star = X_star[:,1:2]
t_star = TT[:,snap]

u_star = U_star[:,0,snap]
v_star = U_star[:,1,snap]
p_star = P_star[:,snap]

# Prediction
u_pred,v_pred, p_pred, f_u_pred, f_v_pred = preds
#
# EDIT BELOW TO LOAD TRAINED MODEL --> load=True
#
u_pred, v_pred, p_pred = predict(x_star, y_star, t_star, u_pred, v_pred, p_pred)
lambda_1_value = sess.run(lambda_1)
lambda_2_value = sess.run(lambda_2)
```

# Calculate Errors

if you have set the number of training iterations large enough the errors should be small.

```python
# Error
error_u = np.linalg.norm(u_star-u_pred,2)/np.linalg.norm(u_star,2)
error_v = np.linalg.norm(v_star-v_pred,2)/np.linalg.norm(v_star,2)
error_p = np.linalg.norm(p_star-p_pred,2)/np.linalg.norm(p_star,2)

error_lambda_1 = np.abs(lambda_1_value - 1.0)*100
error_lambda_2 = np.abs(lambda_2_value - 0.01)/0.01 * 100
```

```python
print('Error u: %e' % (error_u))
print('Error v: %e' % (error_v))
print('Error p: %e' % (error_p))
print('Error l1: %.5f%%' % (error_lambda_1))
print('Error l2: %.5f%%' % (error_lambda_2))
```

```python
# Predict for plotting
lb = X_star.min(0)
ub = X_star.max(0)
nn = 200
x = np.linspace(lb[0], ub[0], nn)
y = np.linspace(lb[1], ub[1], nn)
X, Y = np.meshgrid(x,y)

UU_star = griddata(X_star, u_pred.flatten(), (X, Y), method='cubic')
VV_star = griddata(X_star, v_pred.flatten(), (X, Y), method='cubic')
PP_star = griddata(X_star, p_pred.flatten(), (X, Y), method='cubic')
P_exact = griddata(X_star, p_star.flatten(), (X, Y), method='cubic')
```

# Using Noisy Data

We're now going to repeat the previous steps but include some noise in our data to see the effect of that on our results

```python
######################################################################
########################### Noisy Data ###############################
######################################################################
noise = 0.01
u_train = u_train + noise*np.std(u_train)*np.random.randn(u_train.shape[0], u_train.shape[1])
v_train = v_train + noise*np.std(v_train)*np.random.randn(v_train.shape[0], v_train.shape[1])

```

```python
xvars, NNvars, tfvars, preds, optvars = init(x_train, y_train, t_train, u_train, v_train, layers)
X, lb, ub, x, y, t, u, v = xvars
layers, weights, biases, lambda_1, lambda_2 = NNvars
sess, x_tf, y_tf, t_tf ,u_tf,v_tf = tfvars
u_pred, v_pred, p_pred, f_u_pred, f_v_pred = preds
loss, optimizer, optimizer_Adam, train_op_Adam = optvars
```

**Training might take a while depending on value of Train_iterations**

If you set Train_iterations too low the end results will be garbage. 20000 was used to achieve excellent results.

* If you are using a machine with [GPUs](https://towardsdatascience.com/what-is-a-gpu-and-do-you-need-one-in-deep-learning-718b9597aa0d) please set `Train_iterations` to 20000 and this will run in a few mins
* If you are using a well spec'ed laptop/computer then setting `Train_iterations=10000` but it will take a little while
* If you are using a low spec'ed laptop/computer or cannont leave the code running `Train_iterations=5000` is the reccomended value (this solution may not be accurate)

# Loading Pre trained model option

If the training time is too slow you can skip the following line and load in a pretrained model instead set `loadweights = True` in the next cell. You can play around with different number of iterations to see the effects e.g. setting `saver.restore(sess, netSaveDir + 'model_at_iter15000.ckpt')`

```python
# Training
train(sess, 20000, x_tf, y_tf, t_tf, u_tf, v_tf, x, y, t, u_train, v_train, loss, train_op_Adam, optimizer_Adam,"modelckpts/NSn/")
```

```python
loadweights = False
if loadweights:
    print("loading pre trained model")
    netSaveDir = "modelckpts/NSn/"
    saver = tf.compat.v1.train.Saver()
    saver.restore(sess, netSaveDir + 'model_at_iter18000.ckpt')
```

```python
lambda_1_value_noisy = sess.run(lambda_1)
lambda_2_value_noisy = sess.run(lambda_2)

error_lambda_1_noisy = np.abs(lambda_1_value_noisy - 1.0)*100
error_lambda_2_noisy = np.abs(lambda_2_value_noisy - 0.01)/0.01 * 100

print('Error l1: %.5f%%' % (error_lambda_1_noisy))
print('Error l2: %.5f%%' % (error_lambda_2_noisy))
```

```python
######################################################################
############################# Plotting ###############################
######################################################################
# Load Data
data_vort = scipy.io.loadmat('PINNs/main/Data/cylinder_nektar_t0_vorticity.mat')

x_vort = data_vort['x']
y_vort = data_vort['y']
w_vort = data_vort['w']
modes = np.asscalar(data_vort['modes'])
nel = np.asscalar(data_vort['nel'])

xx_vort = np.reshape(x_vort, (modes+1,modes+1,nel), order = 'F')
yy_vort = np.reshape(y_vort, (modes+1,modes+1,nel), order = 'F')
ww_vort = np.reshape(w_vort, (modes+1,modes+1,nel), order = 'F')

box_lb = np.array([1.0, -2.0])
box_ub = np.array([8.0, 2.0])

fig, ax = plt.subplots()
ax.axis('off')
plt.figure(figsize=(16, 8))
####### Row 0: Vorticity ##################
gs0 = gridspec.GridSpec(1, 2)
gs0.update(top=1-0.06, bottom=1-2/4 + 0.12, left=0.0, right=1.0, wspace=0)
ax = plt.subplot(gs0[:, :])

for i in range(0, nel):
    h = ax.pcolormesh(xx_vort[:,:,i], yy_vort[:,:,i], ww_vort[:,:,i], cmap='seismic',shading='gouraud',  vmin=-3, vmax=3)
divider = make_axes_locatable(ax)
cax = divider.append_axes("right", size="5%", pad=0.05)
fig.colorbar(h, cax=cax)

ax.plot([box_lb[0],box_lb[0]],[box_lb[1],box_ub[1]],'k',linewidth = 1)
ax.plot([box_ub[0],box_ub[0]],[box_lb[1],box_ub[1]],'k',linewidth = 1)
ax.plot([box_lb[0],box_ub[0]],[box_lb[1],box_lb[1]],'k',linewidth = 1)
ax.plot([box_lb[0],box_ub[0]],[box_ub[1],box_ub[1]],'k',linewidth = 1)

ax.set_aspect('equal', 'box')
ax.set_xlabel('$x$')
ax.set_ylabel('$y$')
ax.set_title('Vorticity', fontsize = 10)
plt.tight_layout()
```

```python
####### Row 1: Training data ##################
########      u(t,x,y)     ###################
plt.figure(figsize=(20, 8))
gs1 = gridspec.GridSpec(1, 2)
gs1.update(top=1-2/4, bottom=0.0, left=0.01, right=0.99, wspace=0)
ax = plt.subplot(gs1[:, 0],  projection='3d')
ax.axis('off')

r1 = [x_star.min(), x_star.max()]
r2 = [data['t'].min(), data['t'].max()]
r3 = [y_star.min(), y_star.max()]

for s, e in combinations(np.array(list(product(r1,r2,r3))), 2):
    if np.sum(np.abs(s-e)) == r1[1]-r1[0] or np.sum(np.abs(s-e)) == r2[1]-r2[0] or np.sum(np.abs(s-e)) == r3[1]-r3[0]:
        ax.plot3D(*zip(s,e), color="k", linewidth = 0.5)

ax.scatter(x_train, t_train, y_train, s = 0.1)
# Predict for plotting
lb = X_star.min(0)
ub = X_star.max(0)
nn = 200
x = np.linspace(lb[0], ub[0], nn)
y = np.linspace(lb[1], ub[1], nn)
X, Y = np.meshgrid(x,y)
ax.contourf(X,UU_star,Y, zdir = 'y', offset = t_star.mean(), cmap='rainbow', alpha = 0.8)

ax.text(x_star.mean(), data['t'].min() - 1, y_star.min() - 1, '$x$')
ax.text(x_star.max()+1, data['t'].mean(), y_star.min() - 1, '$t$')
ax.text(x_star.min()-1, data['t'].min() - 0.5, y_star.mean(), '$y$')
ax.text(x_star.min()-3, data['t'].mean(), y_star.max() + 1, '$u(t,x,y)$')
ax.set_xlim3d(r1)
ax.set_ylim3d(r2)
ax.set_zlim3d(r3)
axisEqual3D(ax)

########      v(t,x,y)     ###################
ax = plt.subplot(gs1[:, 1],  projection='3d')
ax.axis('off')

r1 = [x_star.min(), x_star.max()]
r2 = [data['t'].min(), data['t'].max()]
r3 = [y_star.min(), y_star.max()]

for s, e in combinations(np.array(list(product(r1,r2,r3))), 2):
    if np.sum(np.abs(s-e)) == r1[1]-r1[0] or np.sum(np.abs(s-e)) == r2[1]-r2[0] or np.sum(np.abs(s-e)) == r3[1]-r3[0]:
        ax.plot3D(*zip(s,e), color="k", linewidth = 0.5)

ax.scatter(x_train, t_train, y_train, s = 0.1)
ax.contourf(X,VV_star,Y, zdir = 'y', offset = t_star.mean(), cmap='rainbow', alpha = 0.8)

ax.text(x_star.mean(), data['t'].min() - 1, y_star.min() - 1, '$x$')
ax.text(x_star.max()+1, data['t'].mean(), y_star.min() - 1, '$t$')
ax.text(x_star.min()-1, data['t'].min() - 0.5, y_star.mean(), '$y$')
ax.text(x_star.min()-3, data['t'].mean(), y_star.max() + 1, '$v(t,x,y)$')
ax.set_xlim3d(r1)
ax.set_ylim3d(r2)
ax.set_zlim3d(r3)
axisEqual3D(ax)

```

```python
fig, ax = plt.subplots()
plt.figure(figsize=(20, 8))
ax.axis('off')

######## Row 2: Pressure #######################
########      Predicted p(t,x,y)     ###########
gs2 = gridspec.GridSpec(1, 2)
gs2.update(top=1, bottom=1-1/2, left=0.1, right=0.9, wspace=0.5)
ax = plt.subplot(gs2[:, 0])
h = ax.imshow(PP_star, interpolation='nearest', cmap='rainbow',
            extent=[x_star.min(), x_star.max(), y_star.min(), y_star.max()],
            origin='lower', aspect='auto')
divider = make_axes_locatable(ax)
cax = divider.append_axes("right", size="5%", pad=0.05)

fig.colorbar(h, cax=cax)
ax.set_xlabel('$x$')
ax.set_ylabel('$y$')
ax.set_aspect('equal', 'box')
ax.set_title('Predicted pressure', fontsize = 10)

########     Exact p(t,x,y)     ###########
ax = plt.subplot(gs2[:, 1])
h = ax.imshow(P_exact, interpolation='nearest', cmap='rainbow',
            extent=[x_star.min(), x_star.max(), y_star.min(), y_star.max()],
            origin='lower', aspect='auto')
divider = make_axes_locatable(ax)
cax = divider.append_axes("right", size="5%", pad=0.05)

fig.colorbar(h, cax=cax)
ax.set_xlabel('$x$')
ax.set_ylabel('$y$')
ax.set_aspect('equal', 'box')
ax.set_title('Exact pressure', fontsize = 12)
```

Predicted versus exact instantaneous pressure field at a representative time instant. By definition, the pressure can be recovered up to a constant, hence justifying the different magnitude between the two plots. This remarkable qualitative agreement highlights the ability of physics-informed neural networks to identify the entire pressure field, despite the fact that no data on the pressure are used during model training.

**NB** train must be set to approx 10000 to achieve the desired results.

```python
######## Row 3: Table #######################
gs3 = gridspec.GridSpec(1, 2)
gs3.update(top=1-1/2, bottom=0.0, left=0.0, right=1.0, wspace=0)
ax = plt.subplot(gs3[:, :])
ax.axis('off')
plt.rc('text', usetex=False)
s=''
s = s + "Correct PDE \n "
s = s + "$u_t + (u u_x + v u_y) = -p_x + 0.01 (u_{xx} + u_{yy})$"
s = s + "\n"
s = s + "$v_t + (u v_x + v v_y) = -p_y + 0.01 (v_{xx} + v_{yy})$"
s = s + "\n \n \n"
s = s + r'Identified PDE (clean data) '
s = s + "\n"
s = s + '$u_t + %.3f (u u_x + v u_y) = -p_x + %.5f (u_{xx} + u_{yy})$' % (lambda_1_value, lambda_2_value)
s = s + "\n"
s = s + '$v_t + %.3f (u v_x + v v_y) = -p_y + %.5f (v_{xx} + v_{yy})$' % (lambda_1_value, lambda_2_value)
s = s + "\n\n \n"

s = s + r'Identified PDE (1% noise) & '
s = s + "\n"
s = s + '$u_t + %.3f (u u_x + v u_y) = -p_x + %.5f (u_{xx} + u_{yy})$' % (lambda_1_value_noisy, lambda_2_value_noisy)
s = s + "\n"
s = s + '$v_t + %.3f (u v_x + v v_y) = -p_y + %.5f (v_{xx} + v_{yy})$' % (lambda_1_value_noisy, lambda_2_value_noisy)
s = s + "\n"

plt.rc('font',family='serif')

plt.rc('font',size=16)
ax.text(0,0,s,fontsize=18)

# savefig('./figures/NavierStokes_prediction')
```

if you have not been able to run enough training iterations the figures produced running 10000 iterations can be found:

* [Solution with network trained over 10000 iterations](figures/PINNS_NS_10000_PDE.png)
* [Figure comparing predicted vs exact with network trained over 10000 iterations](figures/PINNS_NS_10000_predict_vs_exact.png)

**Further Work**

Congratulations, you have now trained your another physics-informed neural network!

This network contains a number of hyper-parameters that could be tuned to give better results. Various hyper-parameters include:
- number of data training points `N_train`
- number of `layers` in the network
- number of neurons per layer
- optimisation

It is also possible to use different sampling techniques for training data points. We randomly select $N_u$ data points, but alternative methods could be choosing only boundary points or choosing more points near the $t=0$ boundary.

return [here](#init) to alter optimization method used

## Next steps

Now we've demonstrated using PINNs for more complex equations we can take a breif look at Hidden Fluid Mechanics (*this final notebook is beyond the scope of these tutorials but provided to give a breif example of the methodology*)

[Navier-Stokes PINNs Hidden Fluid Mechanics](PINNs_NavierStokes_HFM.ipynb)

```python

```


```python
import numpy as np

from scipy.integrate import simps
#my things
from FermatPrincipleCartesian import *
from Geometry import *
from Symbolic import *
from sympy import Matrix
from RealData import PrepareData
from ForwardEquation import *

def LMSolContinous(dataDict,mu = 0.5):
    '''
    ``rays`` origin and dir are in ENU frame.
    data is d = dtec = int_i ne ds - int_i0 ne ds.
        neFunc = f(beta)
        g(beta) = int_i f(beta) + rho_i ds - int_i0 f(beta) + rho_i0 ds
        minimize (dobs - d)Cdinv(dobs - d) + mu (log(neFunc)  - log(neprior))Cminv(log(neFunc) - log(neprior))

        Solve in continuous basis.
    Steps:
    1. propagate rays
    2. dd = d - g
    3. wdd = Cdinv.dd
    4. S = G^t.Cdinv.G + mu*lambda^t.Cminv.lambda
    5. T = Sinv
    6. dm = T.G^t.wdd
    '''
    #first fit just iri layers and global offsets
    Nsol = 0
    print("Constructing the model with {0} solitons".format(Nsol))
    model = ForwardModel(dataDict['numAntennas'],dataDict['numDirections'],dataDict['numTimes'],
                 pathlength=2000,filename='ww-background',numThreads=1,
                 numSolitons = Nsol,radioArray = None)
    #a priori
    params = model.getForwardKernelParams()
    g = model.doForward(dataDict['rays'],N=100,load=False)
    dd = dataDict['dtec'] - g
    Cd = np.eye(np.size(params))*np.var(g)*1.2
    Cdinv = np.linalg.pinv(Cd)
    wdd = Cdinv.dot(dd)
    rays = model.calcRays(dataDict['rays'],load=True)
    plotWavefront(lambda x,y,z : model.generateSolitonModel()(x,y,z,0),rays,*getSolitonCube(model))
    g = model.doForward(dataDict['rays'],N=100,load=True)
    dd = dataDict['dtec'] - g
    print("Computing observation covariance.")
    Cd = np.eye(np.size(params))*np.var(g)*1.2
    Cdinv = np.linalg.pinv(Cd)
    J = self.doJkernel(inRays,N=100,load=True)
    S = J.transpose().dot(Cdinv).dot(J)
    T = np.linalg.pinv(S)

    wdd = J.transpose().dot(Cdinv).dot(dd)
    dbeta = T.dot(wdd)
    params += dbeta
    model.setModelParams(params)
    #monte carlo L.Cminv.L
    #neFunc = model.solitonModelSymbolic
    #paramDict = self.getModelParamDict()
    #L = []
    #for param i paramDict.keys():
    #    L.append(neFunc.diff(param))

def testForwardProblem():
    sol = SolitonModel(8)

    neFunc = sol.generateSolitonModel()

    theta = np.linspace(-np.pi/8.,np.pi/8.,2)
    #phi = np.linspace(0,2*np.pi,6)
    rays = []
    origin = ac.ITRS(sol.enu.location).cartesian.xyz.to(au.km).value
    for t in theta:
        for p in theta:
            direction = ac.SkyCoord(np.sin(t),
                                    np.sin(p),
                                    1.,frame=sol.enu).transform_to('itrs').cartesian.xyz.value
            rays.append(Ray(origin,direction))
    forwardProblem = ForwardProblem(sol)
    times = np.zeros(len(rays))
    d = forwardProblem.doForward(rays,times,N=1000)
    print(d)
    #plotWavefront(f.nFunc.subs({'t':0}),rays,*getSolitonCube(sol))
    #plotFuncCube(f.nFunc.subs({'t':0}), *getSolitonCube(sol),rays=rays)

if __name__ == '__main__':
    np.random.seed(1234)
    #testForwardProblem()
    dataDict = PrepareData(infoFile='SB120-129/WendysBootes.npz',
                           dataFolder='SB120-129/',
                           timeStart = 0, timeEnd = 0,
                           arrayFile='arrays/lofar.hba.antenna.cfg',load=True)
    LMSolContinous(dataDict,mu = 0.5)
    #LMSolContinous(**dataDict)
```

```python
import pylab as plt
plt.hist(dataDict['dtec'])
plt.show()
```

```python

```


```python
import sys
sys.path.append('..')
import os
import torch
import numpy as np
import matplotlib.pyplot as plt
from sympy import simplify_logic
import time
from sklearn.metrics import accuracy_score
import pandas as pd
from sklearn.tree import DecisionTreeClassifier, plot_tree
from sklearn.tree import _tree, export_text
from torchvision import datasets, transforms
from torch.utils.data import TensorDataset, DataLoader
from torch.utils.data.sampler import SubsetRandomSampler
import torch.nn.functional as F
from sklearn.preprocessing import OneHotEncoder, MinMaxScaler
from sklearn.model_selection import StratifiedKFold
from sklearn.model_selection import train_test_split

from lens.utils.base import validate_network, set_seed, tree_to_formula
from lens.utils.relu_nn import get_reduced_model, prune_features
from lens.utils.psi_nn import prune_equal_fanin
from lens import logic

results_dir = 'results/celldiff'
if not os.path.isdir(results_dir):
    os.makedirs(results_dir)

device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
n_rep = 10
tot_epochs = 15001
prune_epochs = 5000
seed = 42
```

```python
gene_expression_matrix = pd.read_csv('./data/celldiff/data_matrix.csv', index_col=0)
clustering_labels = pd.read_csv('./data/celldiff/cluster_labels.csv', index_col=0)
biomarkers = pd.read_csv('./data/celldiff/markers.csv', index_col=0)

task_id = 2
labels = (clustering_labels.values==task_id)

scaler = MinMaxScaler((0, 1))
scaler.fit(gene_expression_matrix.values)
data_normalized = scaler.transform(gene_expression_matrix.values)

x = torch.FloatTensor(data_normalized)
y = torch.FloatTensor(labels).to(torch.long).squeeze()
print(x.shape)
print(y.shape)
```

    torch.Size([1422, 1845])
    torch.Size([1422])

```python
concepts = list(gene_expression_matrix.columns)
concepts[:10]
```

    ['alas2',
     'si:xx-by187g17.1',
     'si:ch211-5k11.6',
     'tspan36',
     'ba1l',
     'mmp9',
     'il21r.1',
     'plekhf1',
     'slc3a2a',
     'si:dkey-222f2.7']

```python
n_splits = 10
skf = StratifiedKFold(n_splits=n_splits, shuffle=True, random_state=0)
```

```python
def train_nn(x_train, y_train, need_pruning, seed, device, relu=False, verbose=False):
    set_seed(seed)
    x_train = x_train.to(device)
    y_train = y_train.to(device)
    layers = [
        torch.nn.Linear(x_train.size(1), 200),
        torch.nn.ReLU() if relu else torch.nn.LeakyReLU(),
        torch.nn.Linear(200, 80),
        torch.nn.ReLU() if relu else torch.nn.LeakyReLU(),
        torch.nn.Linear(80, 30),
        torch.nn.ReLU() if relu else torch.nn.LeakyReLU(),
        torch.nn.Linear(30, 2),
        torch.nn.Softmax(dim=1),
    ]
    model = torch.nn.Sequential(*layers).to(device)

    optimizer = torch.optim.AdamW(model.parameters(), lr=0.00001)
    loss_form = torch.nn.CrossEntropyLoss()
    model.train()
    for epoch in range(tot_epochs):
        # forward pass
        optimizer.zero_grad()
        y_pred = model(x_train)
        # Compute Loss
        loss = loss_form(y_pred, y_train)

        for module in model.children():
            if isinstance(module, torch.nn.Linear):
                loss += 0.000001 * torch.norm(module.weight, 1)
                loss += 0.000001 * torch.norm(module.bias, 1)
                break

        # backward pass
        loss.backward()
        optimizer.step()

        if epoch > prune_epochs and need_pruning and epoch % 1000 == 1:
            prune_features(model, n_classes=1, device=device)
            need_pruning = True

        # compute accuracy
        if epoch % 500 == 0 and verbose:
            y_pred_d = torch.argmax(y_pred, dim=1)
            accuracy = y_pred_d.eq(y_train).sum().item() / y_train.size(0)
            print(f'\t Epoch {epoch}: train accuracy: {accuracy:.4f}')

    return model
```

```python
def train_psi_nn(x_train, y_train, need_pruning, seed, device, verbose=False):
    set_seed(seed)
    x_train = x_train.to(device)
    y_train = y_train.to(device).to(torch.float)
    layers = [
        torch.nn.Linear(x_train.size(1), 10),
        torch.nn.Sigmoid(),
        torch.nn.Linear(10, 4),
        torch.nn.Sigmoid(),
        torch.nn.Linear(4, 1),
        torch.nn.Sigmoid(),
    ]
    model = torch.nn.Sequential(*layers).to(device)

    optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
    loss_form = torch.nn.BCELoss()
    model.train()
    for epoch in range(tot_epochs):
        # forward pass
        optimizer.zero_grad()
        y_pred = model(x_train).squeeze()
        # Compute Loss
        loss = loss_form(y_pred, y_train)

        for module in model.children():
            if isinstance(module, torch.nn.Linear):
                loss += 0.000001 * torch.norm(module.weight, 1)

        # backward pass
        loss.backward()
        optimizer.step()

        if epoch > prune_epochs and need_pruning:
            model = prune_equal_fanin(model, 2, validate=True, device=device)
            need_pruning = False

        # compute accuracy
        if epoch % 500 == 0 and verbose:
            y_pred_d = y_pred > 0.5
            accuracy = y_pred_d.eq(y_train).sum().item() / y_train.size(0)
            print(f'\t Epoch {epoch}: train accuracy: {accuracy:.4f}')

    return model
```

```python
def c_to_y(method, need_pruning, relu, verbose=False):
    methods = []
    splits = []
    explanations = []
    explanations_inv = []
    model_accuracies = []
    explanation_accuracies = []
    explanation_accuracies_inv = []
    elapsed_times = []
    elapsed_times_inv = []
    for split, (trainval_index, test_index) in enumerate(skf.split(x.cpu().detach().numpy(), y.cpu().detach().numpy())):
        print(f'Split [{split+1}/{n_splits}]')
        x_trainval, x_test = torch.FloatTensor(x[trainval_index]), torch.FloatTensor(x[test_index])
        y_trainval, y_test = torch.LongTensor(y[trainval_index]), torch.LongTensor(y[test_index])
        x_train, x_val, y_train, y_val = train_test_split(x_trainval, y_trainval, test_size=0.3, random_state=42)

        explanation, explanation_inv = '', ''
        explanation_accuracy, explanation_accuracy_inv = 0, 0

        if method == 'tree':
            classifier = DecisionTreeClassifier(random_state=seed)
            classifier.fit(x_trainval.cpu().detach().numpy(), y_trainval.cpu().detach().numpy())
            y_preds = classifier.predict(x_test.cpu().detach().numpy())
            model_accuracy = accuracy_score(y_test.cpu().detach().numpy(), y_preds)

            target_class = 1
            start = time.time()
            explanation = tree_to_formula(classifier, concepts, target_class)
            elapsed_time = time.time() - start
            explanation_accuracy = model_accuracy

            target_class_inv = 0
            start = time.time()
            explanation_inv = tree_to_formula(classifier, concepts, target_class_inv)
            elapsed_time_inv = time.time() - start
            explanation_accuracy_inv = model_accuracy

        else:
            if method == 'psi':
                # positive class
                target_class = 1
                model = train_psi_nn(x_trainval, y_trainval.eq(target_class), need_pruning, seed, device, verbose)
                y_preds = model(x_test.to(device)).cpu().detach().numpy()
                model_accuracy = accuracy_score(y_test.cpu().detach().numpy(), y_preds>0.5)

            else:
                model = train_nn(x_trainval, y_trainval, need_pruning, seed, device, relu, verbose)
                y_preds = model(x_test.to(device)).cpu().detach().numpy()
                model_accuracy = accuracy_score(y_test.cpu().detach().numpy(), y_preds.argmax(axis=1))

            # positive class
            target_class = 1
            start = time.time()
            if method == 'psi':
                global_explanation = logic.generate_fol_explanations(model, device)[0]
            else:
                global_explanation, _, _ = logic.relu_nn.combine_local_explanations(model,
                                                                                x_val.to(device),
                                                                                y_val.to(device),
                                                                                topk_explanations=1,
                                                                                target_class=target_class,
                                                                                method=method, device=device)
            elapsed_time = time.time() - start

            if global_explanation:
                explanation_accuracy, _ = logic.base.test_explanation(global_explanation, target_class, x_test, y_test)
                explanation = logic.base.replace_names(global_explanation, concepts)

#             # negative class
#             target_class_inv = 0
#             if method == 'psi':
#                 model = train_psi_nn(x_trainval, y_trainval.eq(target_class_inv), need_pruning, seed, device, verbose)

#             start = time.time()
#             if method == 'psi':
#                 global_explanation_inv = logic.generate_fol_explanations(model, device)[0]
#             else:
#                 global_explanation_inv, _, _ = logic.relu_nn.combine_local_explanations(model,
#                                                                                x_val.to(device),
#                                                                                y_val.to(device),
#                                                                                topk_explanations=2,
#                                                                                target_class=target_class_inv,
#                                                                                method=method, device=device)
#             elapsed_time_inv = time.time() - start
#             if global_explanation_inv:
#                 explanation_accuracy_inv, _ = logic.base.test_explanation(global_explanation_inv,
#                                                                           target_class_inv, x_test, y_test)
#                 explanation_inv = logic.base.replace_names(global_explanation_inv, concepts)

        if verbose:
            print(f'\t Model\'s accuracy: {model_accuracy:.4f}')
            print(f'\t Class {target_class} - Global explanation: "{explanation}" - Accuracy: {explanation_accuracy:.4f}')
            print(f'\t Elapsed time {elapsed_time}')
#             print(f'\t Class {target_class_inv} - Global explanation: "{explanation_inv}" - Accuracy: {explanation_accuracy_inv:.4f}')
#             print(f'\t Elapsed time {elapsed_time_inv}')

        methods.append(method)
        splits.append(split)
        explanations.append(explanation)
#         explanations_inv.append(explanation_inv)
        model_accuracies.append(model_accuracy)
        explanation_accuracies.append(explanation_accuracy)
#         explanation_accuracies_inv.append(explanation_accuracy_inv)
        elapsed_times.append(elapsed_time)
#         elapsed_times_inv.append(elapsed_time_inv)

    results = pd.DataFrame({
        'method': methods,
        'split': splits,
        'explanation': explanations,
#         'explanation_inv': explanations_inv,
        'model_accuracy': model_accuracies,
        'explanation_accuracy': explanation_accuracies,
#         'explanation_accuracy_inv': explanation_accuracies_inv,
        'elapsed_time': elapsed_times,
#         'elapsed_time_inv': elapsed_times_inv,
    })
    results.to_csv(os.path.join(results_dir, f'results_{method}.csv'))

    return results
```

# General pruning

```python
method = 'pruning'
need_pruning = True
relu = False
results_pruning = c_to_y(method, need_pruning, relu)
results_pruning
```

    Split [1/10]
    Split [2/10]
    Split [3/10]
    Split [4/10]
    Split [5/10]
    Split [6/10]
    Split [7/10]
    Split [8/10]
    Split [9/10]
    Split [10/10]

      method
      split
      explanation
      model_accuracy
      explanation_accuracy
      elapsed_time

      0
      pruning
      0
      dpysl5b & ap5z1 & pcxa & nr2c2 & ~si:ch211-103...
      0.930070
      85.314685
      1.163631

      1
      pruning
      1
      FO704779.1 & pcxa & ~CD37 & ~si:ch211-103n10.5...
      0.909091
      83.916084
      1.183507

      2
      pruning
      2
      dpysl5b & pcxa & nr2c2 & ~si:ch211-103n10.5 & ...
      0.950704
      89.436620
      0.559986

      3
      pruning
      3
      si:dkey-183p4.10 & slc43a2b & si:ch211-132g1.1...
      0.943662
      83.098592
      3.528456

      4
      pruning
      4
      dpysl5b & ap5z1 & pcxa & nr2c2 & ~CD37 & ~mafb...
      0.971831
      85.211268
      1.184820

      5
      pruning
      5
      gyg1b & dpysl5b & yrk & asap2b & riok3 & hsd3b...
      0.922535
      83.098592
      14.036478

      6
      pruning
      6
      dpysl5b & ap5z1 & ppip5k1b & pcxa & ~CD37 & ~s...
      0.936620
      83.802817
      3.152489

      7
      pruning
      7
      anxa3b & Unnamed: 247 & si:ch211-132g1.1 & zgc...
      0.950704
      83.802817
      15.447285

      8
      pruning
      8
      si:dkeyp-118b1.2 & si:ch211-132g1.1 & plekha6 ...
      0.936620
      83.802817
      9.054447

      9
      pruning
      9
      anxa3b & FYB & si:ch211-132g1.1 & zgc:154055 &...
      0.929577
      83.802817
      15.162649

# Weights

```python
method = 'weights'
need_pruning = False
relu = True
results_weights = c_to_y(method, need_pruning, relu)
results_weights
```

    Split [1/10]
    Split [2/10]
    Split [3/10]
    Split [4/10]
    Split [5/10]
    Split [6/10]
    Split [7/10]
    Split [8/10]
    Split [9/10]
    Split [10/10]

      method
      split
      explanation
      model_accuracy
      explanation_accuracy
      elapsed_time

      0
      weights
      0
      ap5z1 & pcxa & nr2c2 & ~si:ch211-137j23.8 & ~s...
      0.944056
      85.314685
      1.326543

      1
      weights
      1
      pcxa & nr2c2 & ~si:ch211-103n10.5 & ~im:715144...
      0.944056
      90.209790
      0.605859

      2
      weights
      2
      pcxa & nr2c2 & ~si:ch211-103n10.5 & ~tmem206 &...
      0.971831
      88.732394
      0.620259

      3
      weights
      3
      si:ch211-132g1.1 & pcxa & nr2c2 & ~CD37 & ~si:...
      0.971831
      90.140845
      0.794775

      4
      weights
      4
      pcxa & nr2c2 & ~si:ch211-103n10.5 & ~im:715144...
      0.985915
      89.436620
      0.740607

      5
      weights
      5
      si:ch211-132g1.1 & ppip5k1b & pcxa & nr2c2 & ~...
      0.936620
      84.507042
      1.093919

      6
      weights
      6
      pcxa & ~si:ch211-103n10.5 & ~im:7151449 & ~smi...
      0.978873
      92.957746
      0.712162

      7
      weights
      7
      pcxa & nr2c2 & ~si:ch211-103n10.5 & ~tmem206 &...
      0.964789
      87.323944
      1.033122

      8
      weights
      8
      si:ch211-132g1.1 & pcxa & nr2c2 & ~si:ch211-10...
      0.943662
      89.436620
      0.812649

      9
      weights
      9
      pcxa & nr2c2 & ~si:ch211-103n10.5 & ~im:715144...
      0.957746
      85.211268
      0.544006

# Psi network

```python
method = 'psi'
need_pruning = True
relu = False
results_psi = c_to_y(method, need_pruning, relu, verbose=False)
results_psi
```

# Decision tree

```python
method = 'tree'
need_pruning = False
relu = False
results_tree = c_to_y(method, need_pruning, relu)
results_tree
```

# Summary

```python
cols = ['model_accuracy', 'explanation_accuracy', 'explanation_accuracy_inv', 'elapsed_time', 'elapsed_time_inv']
mean_cols = [f'{c}_mean' for c in cols]
sem_cols = [f'{c}_sem' for c in cols]

# pruning
df_mean = results_pruning[cols].mean()
df_sem = results_pruning[cols].sem()
df_mean.columns = mean_cols
df_sem.columns = sem_cols
summary_pruning = pd.concat([df_mean, df_sem])
summary_pruning.name = 'pruning'

# # lime
# df_mean = results_lime[cols].mean()
# df_sem = results_lime[cols].sem()
# df_mean.columns = mean_cols
# df_sem.columns = sem_cols
# summary_lime = pd.concat([df_mean, df_sem])
# summary_lime.name = 'lime'

# weights
df_mean = results_weights[cols].mean()
df_sem = results_weights[cols].sem()
df_mean.columns = mean_cols
df_sem.columns = sem_cols
summary_weights = pd.concat([df_mean, df_sem])
summary_weights.name = 'weights'

# psi
df_mean = results_psi[cols].mean()
df_sem = results_psi[cols].sem()
df_mean.columns = mean_cols
df_sem.columns = sem_cols
summary_psi = pd.concat([df_mean, df_sem])
summary_psi.name = 'psi'

# tree
df_mean = results_tree[cols].mean()
df_sem = results_tree[cols].sem()
df_mean.columns = mean_cols
df_sem.columns = sem_cols
summary_tree = pd.concat([df_mean, df_sem])
summary_tree.name = 'tree'

summary = pd.concat([summary_pruning,
#                      summary_lime,
                     summary_weights,
                     summary_psi,
                     summary_tree], axis=1).T
summary.columns = mean_cols + sem_cols
summary
```

```python
summary.to_csv(os.path.join(results_dir, 'summary.csv'))
```


# Quantum pipeline using t|ket> backend

```python
import numpy as np

def read_data(filename):
    labels, sentences = [], []
    with open(filename) as f:
        for line in f:
            labels.append([1, 0] if line[0] == '1' else [0, 1])
            sentences.append(line[1:].strip())
    return np.array(labels), sentences

train_labels, train_data = read_data('datasets/mc_train_data.txt')
dev_labels, dev_data = read_data('datasets/mc_dev_data.txt')
test_labels, test_data = read_data('datasets/mc_test_data.txt')
```

### Create diagrams

```python
from lambeq.ccg2discocat import DepCCGParser

reader = DepCCGParser(possible_root_cats=['S[dcl]'])

train_diagrams = reader.sentences2diagrams(train_data)
dev_diagrams = reader.sentences2diagrams(dev_data)
test_diagrams = reader.sentences2diagrams(test_data)
```

### Create circuits

```python
from lambeq.circuit import IQPAnsatz
from lambeq.core.types import AtomicType

ansatz = IQPAnsatz({AtomicType.NOUN: 1, AtomicType.SENTENCE: 1},
                   n_layers=1, n_single_qubit_params=3)

train_circuits = [ansatz(diagram) for diagram in train_diagrams]
dev_circuits =  [ansatz(diagram) for diagram in dev_diagrams]
test_circuits = [ansatz(diagram) for diagram in test_diagrams]

train_circuits[0].draw(figsize=(8, 8))
```

### Parameterise

```python
from sympy import default_sort_key

all_circuits = train_circuits + dev_circuits + test_circuits

# sort the symbols since they are returned as a set
parameters = sorted(
    {s for circ in all_circuits for s in circ.free_symbols},
    key=default_sort_key)
```

```python
from discopy.quantum import Id, Measure
from discopy.tensor import Tensor
from pytket.extensions.qiskit import AerBackend

backend_config = {
    'backend': AerBackend(),
    'compilation': AerBackend().default_compilation_pass(2)
}

def normalise(predictions):
    # apply smoothing to predictions
    predictions = np.abs(predictions + 1e-9)
    return predictions / predictions.sum()

def make_pred_fn(circuits):
    def predict(params):
        measured_circuits = [c >> Id().tensor(*[Measure()] * len(c.cod)) for c in circuits]
        return np.array(
            [normalise(np.abs(c.lambdify(*parameters)(*params).eval(**backend_config).array))
             for c in measured_circuits])
    return predict

train_pred_fn = make_pred_fn(train_circuits)
dev_pred_fn = make_pred_fn(dev_circuits)
test_pred_fn = make_pred_fn(test_circuits)
```

### Train

```python
from noisyopt import minimizeSPSA
import numpy

def make_cost_fn(pred_fn, labels):
    def cost_fn(params, **kwargs):
        predictions = pred_fn(params)

        cost = -np.sum(labels * np.log2(predictions)) / len(labels)  # binary cross-entropy loss
        costs.append(cost)

        acc = np.sum(np.round(predictions) == labels) / len(labels) / 2  # half due to double-counting
        accuracies.append(acc)

        return cost

    costs, accuracies = [], []
    return cost_fn, costs, accuracies

numpy.random.seed(0)
rng = numpy.random.default_rng(0)
x0 = np.array(rng.random(len(parameters)))

train_cost_fn, train_costs, train_accs = make_cost_fn(train_pred_fn, train_labels)
dev_cost_fn, dev_costs, dev_accs = make_cost_fn(dev_pred_fn, dev_labels)

result = minimizeSPSA(train_cost_fn, x0=x0, a=0.13, c=0.1, niter=400, callback=dev_cost_fn)
```

### Show results

```python
import matplotlib.pyplot as plt
plt.rcParams['figure.figsize'] = (10, 6)

fig, ((ax_tl, ax_tr), (ax_bl, ax_br)) = plt.subplots(2, 2, sharex=True, sharey='row')
ax_tl.set_title('Training set')
ax_tr.set_title('Development set')
ax_bl.set_xlabel('Iterations')
ax_br.set_xlabel('Iterations')
ax_bl.set_ylabel('Accuracy')
ax_tl.set_ylabel('Loss')

colours = iter(plt.rcParams['axes.prop_cycle'].by_key()['color'])
ax_tl.plot(train_costs[1::2], color=next(colours))  # training evaluates twice per iteration
ax_bl.plot(train_accs[1::2], color=next(colours))   # so take every other entry
ax_tr.plot(dev_costs, color=next(colours))
ax_br.plot(dev_accs, color=next(colours))

# print test accuracy
test_cost_fn, _, test_accs = make_cost_fn(test_pred_fn, test_labels)
test_cost_fn(result.x)
print('Test accuracy:', test_accs[0])
```


+ This notebook is part of lecture 3 *Multiplication and inverse matrices* in the OCW MIT course 18.06 [1]
+ Created by me, Dr Juan H Klopper
    + Head of Acute Care Surgery
    + Groote Schuur Hospital
    + University Cape Town
    + Email me with your thoughts, comments, suggestions and corrections

Linear Algebra OCW MIT18.06 IPython notebook [2] study notes by Dr Juan H Klopper is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

+ [1] OCW MIT 18.06
+ [2] Fernando Pérez, Brian E. Granger, IPython: A System for Interactive Scientific Computing, Computing in Science and Engineering, vol. 9, no. 3, pp. 21-29, May/June 2007, doi:10.1109/MCSE.2007.53. URL: http://ipython.org

```python
from IPython.core.display import HTML, Image
css_file = 'style.css'
HTML(open(css_file, 'r').read())
```

```python
from sympy import init_printing, Matrix, symbols, eye, Rational
from warnings import filterwarnings
```

```python
init_printing(use_latex = 'mathjax')
filterwarnings('ignore')
```

# Matrix multiplication, inverse and transpose

## Multiplying matrices

### Method 1

+ Consider multiply matrices A and B to result in C
+ We have already seen that the column size of the first must equal the row size of the second, *n*A **must equal** *m*B
$$ {m}_{A}\times{n}_{A}\cdot{m}_{B}\times{n}_{B} \\ = {m}_{A}\cdot{n}_{B} $$
+ C will then be of size *m*A × *n*B
+ Every position *c*ij, with *i* as the row position and *j* as the column position is calculated by taking the dot product (i.e. each element times it's corresponding element, all added), *c*ij = (row *i* in A ⋅ column *j* of B)
+ Here we calculate the row 2, column 1 position in C by the dot product of row 2 in A by column 1 in B
$$ { \begin{bmatrix} \cdots  & \cdots  & \cdots  \\ 3 & 2 & -1 \\ \cdots  & \cdots  & \cdots  \\ \cdots  & \cdots  & \cdots  \end{bmatrix} }_{ 4\times 3 }{ \begin{bmatrix} 1 & \vdots  \\ 2 & \vdots  \\ 1 & \vdots  \end{bmatrix} }_{ 3\times 2 }={ \begin{bmatrix} { c }_{ 11 } & { c }_{ 12 } \\ \left( 3\times 1 \right) +\left( 2\times 2 \right) +\left( -1\times 1 \right)  & { c }_{ 22 } \\ { c }_{ 31 } & { c }_{ 32 } \\ { c }_{ 41 } & { c }_{ 42 } \end{bmatrix} }_{ 4\times 2 } $$
+ Notice how this multiplication is only possible because the row size of A equals the column size of B
$$ { \begin{bmatrix} \cdots  & \cdots  & \cdots  \\ { a }_{ 21 } & { a }_{ 22 } & { a }_{ 23 } \\ \cdots  & \cdots  & \cdots  \\ \cdots  & \cdots  & \cdots  \end{bmatrix} }_{ 4\times 3 }{ \begin{bmatrix} { b }_{ 11 } & \vdots  \\ { b }_{ 21 } & \vdots  \\ { b }_{ 31 } & \vdots  \end{bmatrix} }_{ 3\times 2 }={ \begin{bmatrix} { c }_{ 11 } & { c }_{ 12 } \\ \left( { a }_{ 21 }{ b }_{ 11 } \right) +\left( { a }_{ 22 }{ b }_{ 21 } \right) +\left( { a }_{ 23 }{ b }_{ 31 } \right)  & { c }_{ 22 } \\ { c }_{ 31 } & { c }_{ 32 } \\ { c }_{ 41 } & { c }_{ 42 } \end{bmatrix} }_{ 4\times 2 }\\ =\sum _{ k=1 }^{ n }{ { a }_{ 2k }{ b }_{ k1 } }  $$

### Method 2

+ In this method we note that each column in C is the result of the matrix A times the corresponding column in B
+ This is akin to a matrix multiplied by a vector A**x**=**b**
+ We see B as made up of vector columns
+ The columns of C are thus combinations of columns of A
    + The numbers in the corresponding columns in B is this combination

### Method 3

+ Here every row in A produces the same numbered row in C by multiplying it with the matrix B
+ The rows of C are linear combinations of B

### Method 4

+ In method 1 we looked at rowA × colB producing a single number in C
+ What if we did column × row?
+ The size of column of A is *m*A × 1 and a row of B is of size 1 × *n*B
+ This results in C of size *m*A × *n*B
+ Let's look at a simple example using python (with sympy)

```python
A = Matrix([[2], [3], [4]])
B = Matrix([[1, 6]])
A, B
```

```python
C = A * B
C
```

+ Notice how the columns of C are linear combinations of the values in the columns of A
+ The rows of C are multiples of the rows of B
+ So in method 4, C is the sum of *the columns of A* × *the rows of B*
$$ \begin{bmatrix} { a }_{ 11 } & { a }_{ 12 } \\ { a }_{ 21 } & { a }_{ 22 } \\ { a }_{ 31 } & { a }_{ 32 } \end{bmatrix}\begin{bmatrix} { b }_{ 11 } & { b }_{ 12 } \\ b_{ 21 } & { b }_{ 22 } \end{bmatrix}=\begin{bmatrix} { a }_{ 11 } \\ { a }_{ 21 } \\ { a }_{ 31 } \end{bmatrix}\begin{bmatrix} { b }_{ 11 } & { b }_{ 12 } \end{bmatrix}+\begin{bmatrix} { a }_{ 12 } \\ { a }_{ 22 } \\ { a }_{ 32 } \end{bmatrix}\begin{bmatrix} { b }_{ 21 } & { b }_{ 22 } \end{bmatrix} $$

### Block multiplication

+ Combining the above we can do the following
    + Both A and B can be broken into block of sizes that allow for multiplication
    + Here is an example of two square matrices
$$ \begin{bmatrix} { A }_{ 1 } & { A }_{ 2 } \\ { A }_{ 3 } & { A }_{ 4 } \end{bmatrix}\begin{bmatrix} { B }_{ 1 } & { B }_{ 2 } \\ { B }_{ 3 } & { B }_{ 4 } \end{bmatrix}=\begin{bmatrix} { A }_{ 1 }{ B }_{ 1 }+{ A }_{ 2 }{ B }_{ 3 } & { A }_{ 1 }{ B }_{ 2 }+{ A }_{ 2 }{ B }_{ 4 } \\ { A }_{ 3 }{ B }_{ 1 }+{ A }_{ 4 }{ B }_{ 3 } & { A }_{ 3 }{ B }_{ 2 }+{ A }_{ 4 }{ B }_{ 4 } \end{bmatrix} $$

## Inverses

+ **If** the inverse of a matrix A exists then A-1=I, the identity matrix
+ Above is a left inverse, but what about a right inverse, AA-1?
    + This is also equal to the identity for invertible square inverses
+ Invertible matrices are also called *non-singular* matrices

+ Non-invertible matrices are also called *singular* matrices
+ An example would look like this
$$ \begin{bmatrix}1&3\\2&6\end{bmatrix} $$
+ Note how the elements on row two are just two times the elements in row 1 (A linear combination)
+ The same go for the columns, the first being a linear combination of the second, multiplying each element by 3
+ More profoundly, note that you could find a column vector **x** such that A**x**=**0**
$$ \begin{bmatrix}1&3\\2&6\end{bmatrix}\begin{bmatrix}3\\-1\end{bmatrix}=\begin{bmatrix}0\\0\end{bmatrix} $$
+ This says 3 times column 1 in A plus -1 times column 2 gives *nothing*

+ Let construct as example
$$ \begin{bmatrix} 1 & 3 \\ 2 & 7 \end{bmatrix}\begin{bmatrix} a & c \\ b & d \end{bmatrix}=\begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} $$
+ In essence we have to solve two systems
    + A × column *j* of A-1 = column *j* of I
    + This is the Gauss-Jordan idea of solving two systems at once
    $$ \begin{bmatrix} 1 & 3 \\ 2 & 7 \end{bmatrix}\begin{bmatrix} a \\ b \end{bmatrix}=\begin{bmatrix} 1 \\ 0 \end{bmatrix}\\ \begin{bmatrix} 1 & 3 \\ 2 & 7 \end{bmatrix}\begin{bmatrix} c \\ d \end{bmatrix}=\begin{bmatrix} 0 \\ 1 \end{bmatrix} $$
    + This gives us the two columns of A-1
    + We now create the augmented matrix
    $$ \begin{bmatrix} 1 & 3 & 1 & 0 \\ 2 & 7 & 0 & 1 \end{bmatrix} $$
    + Now we use elementary row operations to reduced row-echelon form (leading 1's in the pivot positions, with 0's below and above each)
    $$ \begin{bmatrix} 1 & 3 & 1 & 0 \\ 2 & 7 & 0 & 1 \end{bmatrix}\rightarrow \begin{bmatrix} 1 & 3 & 1 & 0 \\ 0 & 1 & -2 & 1 \end{bmatrix}\rightarrow \begin{bmatrix} 1 & 0 & 7 & -3 \\ 0 & 1 & -2 & 1 \end{bmatrix} $$
    + We now read off the two columns of A-1
    $$ \begin{bmatrix}7&-3\\-2&1\end{bmatrix} $$
+ To do all of the elimination, we created a lot of elimination (elementary) matrices
+ If we combine all of them into E we have E[AI]=[IA-1], because EA=I tells us E=A-1

## Example problems

### Example problem 1

+ Find the conditions on *a* and *b* that makes the matrix A invertible and find A-1
$$ A=\begin{bmatrix} a & b & b \\ a & a & b \\ a & a & a \end{bmatrix} $$

#### Solution

+ A matrix is singular (non-invertible) if we have a row or column of zeros, so *a* ≠ 0
+ We can also not have similar columns, so *a* ≠ b
+ Using Gauss-Jordan elimination we will have the following
$$ \begin{bmatrix} a & b & b & 1 & 0 & 0 \\ a & a & b & 0 & 1 & 0 \\ a & a & a & 0 & 0 & 1 \end{bmatrix}\rightarrow \begin{bmatrix} a & b & b & 1 & 0 & 0 \\ 0 & a-b & 0 & -1 & 1 & 0 \\ 0 & a-b & a-b & -1 & 0 & 1 \end{bmatrix}\rightarrow \begin{bmatrix} a & b & b & 1 & 0 & 0 \\ 0 & a-b & 0 & -1 & 1 & 0 \\ 0 & 0 & a-b & 0 & -1 & 1 \end{bmatrix}\\ \rightarrow \begin{bmatrix} a & b & b & 1 & 0 & 0 \\ 0 & \frac { a-b }{ a-b }  & 0 & \frac { -1 }{ a-b }  & \frac { 1 }{ a-b }  & 0 \\ 0 & 0 & \frac { a-b }{ a-b }  & 0 & \frac { -1 }{ a-b }  & \frac { 1 }{ a-b }  \end{bmatrix}\rightarrow \begin{bmatrix} a & b & b & 1 & 0 & 0 \\ 0 & 1 & 0 & \frac { -1 }{ a-b }  & \frac { 1 }{ a-b }  & 0 \\ 0 & 0 & 1 & 0 & \frac { -1 }{ a-b }  & \frac { 1 }{ a-b }  \end{bmatrix}\\ \rightarrow \begin{bmatrix} a & b & 0 & 1 & \frac { 1 }{ a-b } \left( b \right)  & -\frac { 1 }{ a-b } \left( b \right)  \\ 0 & 1 & 0 & \frac { -1 }{ a-b }  & \frac { 1 }{ a-b }  & 0 \\ 0 & 0 & 1 & 0 & \frac { -1 }{ a-b }  & \frac { 1 }{ a-b }  \end{bmatrix}\rightarrow \begin{bmatrix} a & 0 & 0 & 1+\frac { b }{ a-b }  & 0 & -\frac { 1 }{ a-b } \left( b \right)  \\ 0 & 1 & 0 & \frac { -1 }{ a-b }  & \frac { 1 }{ a-b }  & 0 \\ 0 & 0 & 1 & 0 & \frac { -1 }{ a-b }  & \frac { 1 }{ a-b }  \end{bmatrix}\\ \rightarrow \begin{bmatrix} 1 & 0 & 0 & \frac { 1 }{ a-b }  & 0 & -\frac { 1 }{ a\left( a-b \right)  } \left( b \right)  \\ 0 & 1 & 0 & \frac { -1 }{ a-b }  & \frac { 1 }{ a-b }  & 0 \\ 0 & 0 & 1 & 0 & \frac { -1 }{ a-b }  & \frac { 1 }{ a-b }  \end{bmatrix}\\ { A }^{ -1 }=\frac { 1 }{ a-b } \begin{bmatrix} 1 & 0 & \frac { -b }{ a }  \\ -1 & 1 & 0 \\ 0 & -1 & 1 \end{bmatrix} $$

+ Additionally then we note that for the inverse of A to exist *a* - *b* ≠ 0, which is the same as *a* ≠ *b* and again a ≠ 0

```python

```