Text provided under a Creative Commons Attribution license, CC-BY.  All code is made available under the FSF-approved BSD-3 license.  (c) Lorena A. Barba, Gilbert F. Forsyth 2017. Thanks to NSF for support via CAREER award #1149784.
[@LorenaABarba](https://twitter.com/LorenaABarba)

[@ggruszczynski](https://github.com/ggruszczynski)

12 steps to Navier–Stokes
======
***

You should have completed Steps [1](./01_Step_1.ipynb) and [2](./02_Step_2.ipynb) before continuing. This Jupyter notebook continues the presentation of the **12 steps to Navier–Stokes**, the practical module taught in the interactive CFD class of [Prof. Lorena Barba](http://lorenabarba.com).

Step 3: Diffusion Equation in 1-D
-----
***

The one-dimensional diffusion equation is:

$$\frac{\partial u}{\partial t}= \nu \frac{\partial^2 u}{\partial x^2}$$

The first thing you should notice is that —unlike the previous two simple equations we have studied— this equation has a second-order derivative. We first need to learn what to do with it!

### Discretizing $\frac{\partial ^2 u}{\partial x^2}$

The second-order derivative can be represented geometrically as the line tangent to the curve given by the first derivative.  We will discretize the second-order derivative with a Central Difference scheme: a combination of Forward Difference and Backward Difference of the first derivative.  Consider the Taylor expansion of $u_{i+1}$ and $u_{i-1}$ around $u_i$:

$u_{i+1} = u_i + \Delta x \frac{\partial u}{\partial x}\bigg|_i + \frac{\Delta x^2}{2} \frac{\partial ^2 u}{\partial x^2}\bigg|_i + \frac{\Delta x^3}{3!} \frac{\partial ^3 u}{\partial x^3}\bigg|_i + O(\Delta x^4)$

$u_{i-1} = u_i - \Delta x \frac{\partial u}{\partial x}\bigg|_i + \frac{\Delta x^2}{2} \frac{\partial ^2 u}{\partial x^2}\bigg|_i - \frac{\Delta x^3}{3!} \frac{\partial ^3 u}{\partial x^3}\bigg|_i + O(\Delta x^4)$

If we add these two expansions, you can see that the odd-numbered derivative terms will cancel each other out.  If we neglect any terms of $O(\Delta x^4)$ or higher (and really, those are very small), then we can rearrange the sum of these two expansions to solve for our second-derivative.

$u_{i+1} + u_{i-1} = 2u_i+\Delta x^2 \frac{\partial ^2 u}{\partial x^2}\bigg|_i + O(\Delta x^4)$

Then rearrange to solve for $\frac{\partial ^2 u}{\partial x^2}\bigg|_i$ and the result is:

$$\frac{\partial ^2 u}{\partial x^2}=\frac{u_{i+1}-2u_{i}+u_{i-1}}{\Delta x^2} + O(\Delta x^4)$$

### Back to Step 3

We can now write the discretized version of the diffusion equation in 1D:

$$\frac{u_{i}^{n+1}-u_{i}^{n}}{\Delta t}=\nu\frac{u_{i+1}^{n}-2u_{i}^{n}+u_{i-1}^{n}}{\Delta x^2}$$

As before, we notice that once we have an initial condition, the only unknown is $u_{i}^{n+1}$, so we re-arrange the equation solving for our unknown:

$$u_{i}^{n+1}=u_{i}^{n}+\frac{\nu\Delta t}{\Delta x^2}(u_{i+1}^{n}-2u_{i}^{n}+u_{i-1}^{n})$$

The above discrete equation allows us to write a program to advance a solution in time. But we need an initial condition. Let's continue using our favorite: the hat function. So, at $t=0$, $u=2$ in the interval $0.5\le x\le 1$ and $u=1$ everywhere else. We are ready to number-crunch!

```python
import numpy                     #loading our favorite library
from matplotlib import pyplot    #and the useful plotting library
%matplotlib inline

nx = 128
domain_length = 64
dx = domain_length / (nx-1)
xspace = numpy.linspace(0, domain_length, nx)

nt = 100  #the number of timesteps we want to calculate
nu = 5   #the value of viscosity
sigma = .2 #sigma is a parameter, we'll learn more about it later
dt = sigma * dx**2 / nu #dt is defined using sigma ... more later!

u_IC = numpy.ones(nx)      #numpy function ones()
u_IC[int((nx-1)/4):int(nx/2 + 1)] = 2 #setting u = 2 between 0.5 and 1 as per our I.C.s

```

```python

def calc_diffusion(IC,nx,nt,nu,dt):
    u = IC.copy()
    un = IC.copy() #our placeholder array, un, to advance the solution in time

    for n in range(nt):  #iterate through time
        un = u.copy() ##copy the existing values of u into un
        for i in range(1, nx - 1):
            # u[i] = ...
    return u

u_FD = calc_diffusion(u_IC,nx,nt,nu,dt)
pyplot.plot(xspace, u_FD)

```

# Advection-Diffusion of a Gaussian Hill

In case of an isotropic diffusion, the analytical solution describing evolution of a Gaussian Hill can be expressed as

$$
C(\boldsymbol{x}, t)=\frac{\left(2\pi\sigma_{0}^{2}\right)^{D/2} }{\left(2\pi(\sigma_{0}^{2} + 2 k t)\right)^{D/2}}
C_0 \exp \left(-\frac{\left(\boldsymbol{x}-\boldsymbol{x}_{0}-\boldsymbol{u} t\right)^{2}}{2\left(\sigma_{0}^{2}+ 2 k t\right)}\right)
$$

where:

* $C_0$ - initial concentration,
* $D$ - number of dimensions,
* $t$ - time,
* $k$ - conductivity,
* $\boldsymbol{u}$ - velocity of advection
* $\sigma_{0}$ the initial variance of the distribution.

## Task

1) Implement the `GaussianHillAnal` class.
It shall have a method `get_concentration_ND(self, X, t)`, which will return the concentration at given time and space.

2) Benchmark the FD code against analytical solution.

```python
from sympy.matrices import Matrix
import numpy as np
import sympy as sp

class GaussianHillAnal:
    def __init__(self, C0, X0, Sigma2_0, k, U, D):
        """
        :param C0: initial concentration
        :param X0: initial position of the hill's centre = Matrix([x0, y0])
        :param U:  velocity = Matrix([ux, uy])
        :param Sigma2_0: initial width of the Gaussian Hill
        :param k: conductivity
        :param dimenions: number of dimensions
        """
        self.C0 = C0
        self.X0 = X0
        self.U = U
        self.Sigma2_0 = Sigma2_0
        self.k = k
        self.dim = D

    def get_concentration_ND(self, X, t):
        # TODO
        return C

```

```python
conductivity = nu

nt = 200  #the number of timesteps we want to calculate
nu = 5   #the value of viscosity
sigma = .2 #sigma is a parameter, we'll learn more about it later
dt = sigma * dx**2 / conductivity #dt is defined using sigma ... more later!

time_0    = dt*nt/2     # initial contidion for FD
time_spot = dt*nt       # time to be simulated (by FD and analytically)

X0 = Matrix([domain_length/2.]) # center of the hill
C0 = 1.                 # concentration
variance = 30           # initial variance
reference_level = 1

T_0 = np.zeros(nx)
T_anal = np.zeros(nx)

gha = GaussianHillAnal(C0, X0, variance, conductivity, Matrix([0]), D=1)

for i in range(nx):
    T_0[i] = reference_level + gha.get_concentration_ND(Matrix([xspace[i]]), time_0)
    T_anal[i] = reference_level + gha.get_concentration_ND(Matrix([xspace[i]]), time_spot)

T_FD = calc_diffusion(T_0,nx,nt,nu,dt)

import matplotlib.pyplot as plt
plt.rcParams.update({'font.size': 16})
figure, axis = plt.subplots(1, 1, figsize=(8, 6))
plt.subplots_adjust(hspace=1)
axis.set_title('Diffusion of a Gaussian Hill')
axis.plot(xspace, T_0, label=r'$T_{0}$')
axis.plot(xspace, T_anal, label=r'$T_{anal}$')
axis.plot(xspace, T_FD, label=r'$T_{FD}$', marker='x', linestyle="", markevery=1)
axis.set_xlabel('x')
axis.set_ylabel('Concentration')
axis.legend(loc="upper right")

```

## Questions:

* How do you find the FD solution compared to analytical one? Experiment with different dx, dt.

* How would you asses that your mesh is fine enought in a real CFD simulation (without analytical solution)?

# Balance of Enthalpy

The simplified balance of Enthalpy $H=\rho c_p T$ is known as:
\begin{align}
\int \frac{\partial }{\partial t} (\rho c_p T ) dV +  \oint  ( \rho c_p T \boldsymbol{u} ) \cdot \boldsymbol{n} dS &= \oint \boldsymbol{n} \cdot \boldsymbol{q} dS + \int \dot{q} dV \\
\boldsymbol{q} &= k \nabla T
\end{align}
The heat flux $ \boldsymbol{q} $ is related to the thermal conductivity coefficient $k [ W/mK] $.

Applying the GGO theorem, we obtain:

$$
\frac{\partial }{\partial t} (\rho c_p T ) + \nabla \cdot (\boldsymbol{u} \rho c_p T ) = \nabla \cdot (k \nabla T)  + \dot{q}
$$

## Conservative and non-conservative scheme

Consider a 1D, unsteady heat transfer equation with variable conduction coefficient.
For simplicity, assume that the heat capacity $ \rho c_p = 1 $.

The equation be expressed in two mathematically equivalent forms:
$$
\frac{\partial u}{\partial t}=\frac{\partial}{\partial x}\left(k(x) \frac{\partial u}{\partial x}\right) \\
$$
or
$$
\frac{\partial u}{\partial t}=\frac{\partial u}{\partial x} \frac{\partial k}{\partial x}+k(x) \frac{\partial^{2} u}{\partial x^{2}}
$$

Both of these continous forms can be discretized.

First form leads to the *conservative scheme*

$$
\frac{u_{i}^{n+1}-u_{i}^{n}}{\Delta t}=\frac{1}{\Delta x^{2}}\left(k_{i+\frac{1}{2}}\left(u_{i+1}^{n}-u_{i}^{n}\right)-k_{i-\frac{1}{2}}\left(u_{i}^{n}-u_{i-1}^{n}\right)\right)
$$

while the second form results in a *non-conservative scheme*

$$
\frac{u_{i}^{n+1}-u_{i}^{n}}{\Delta t}=\frac{1}{\Delta x^{2}}\left(\frac{\left(k_{i+1}-k_{i-1}\right)}{2} \frac{\left(u_{i+1}^{n}-u_{i-1}^{n}\right)}{2}+k_{i}\left(u_{i+1}^{n}+u_{i-1}^{n}-2 u_{i}^{n}\right)\right)
$$

## Task

* Implement both the *conservative* and the *non-conservative* schemes.

* Impose the following BC:

```
u[int(nx/4)] = 10.
u[-int(nx/4)] = 1.

u[0] = 5.
u[-1] = 5.
```

* Run the simulation for different $k$ and compare results. Which physical quantity may become *not conserved* ?

```python
# variable conductivity
k = 1.*numpy.ones(nx)                   # be aware that dt = sigma * dx**2 / nu
k[:int(nx/2)] = 0.5                     # try with 0.1

u_IC = 5.*numpy.ones(nx)                 #numpy function ones()
u_IC[int((nx-1)/4):int(nx/2 + 1)] = 10   #setting u = 2 between 0.5 and 1 as per our I.C.s
dt = sigma * dx**2 / max(k)              #dt is defined using sigma

nt = 10000  # the number of timesteps we want to calculate

def calc_diffusion_variable_conductivity(IC,nx,nt,k,dt):
    u = IC.copy()
    un = IC.copy() #our placeholder array, un, to advance the solution in time

    for n in range(nt):  #iterate through time
        # dont forget about the BC...

        un = u.copy() ##copy the existing values of u into un
        for i in range(1, nx - 1):
            # this is obviously bad: du/dt = k (d^2)u/(dx)^2
            # u[i] = un[i] + k[i] * dt / dx**2 * (un[i+1] - 2 * un[i] + un[i-1])

            # non conservative scheme
            # u[i] = ...

            # conservative scheme
            # kf = ... # forward  stencil for k_(i+1/2)
            # kb = ... # backward stencil for k_(i-1/2)
            # u[i] = ...

    return u

u_FD = calc_diffusion_variable_conductivity(u_IC,nx,nt,k,dt)
pyplot.plot(xspace, u_FD)
```

## Learn More

For a careful walk-through of the discretization of the diffusion equation with finite differences (and all steps from 1 to 4), watch **Video Lesson 4** by Prof. Barba on YouTube.

```python
from IPython.display import YouTubeVideo
YouTubeVideo('y2WaK7_iMRI')
```

```python
from IPython.core.display import HTML
def css_styling():
    styles = open("../styles/custom.css", "r").read()
    return HTML(styles)
css_styling()
```

> (The cell above executes the style for this notebook.)


# A short & practical introduction to Tensor Flow!

Part 6

Example of a single-layer one-directional long short-term memory network (LSTM) trained with
[connectionist temporal classification](http://www.cs.toronto.edu/~graves/icml_2006.pdf) to predict character sequences from nFeatures x nFrames
arrays of Mel-Frequency Cepstral Coefficients.  This is test code to run on the
8-item data set in the "sample_data" directory, for those without access to TIMIT.

Author: [Jon Rein](https://github.com/jonrein/tensorflow_CTC_example)

Adapted by: Pablo M. Olmos (olmos@tsc.uc3m.es)

Date: March 2017

```python
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import tensorflow as tf
import zipfile
from tensorflow.python.ops import ctc_ops as ctc
import numpy as np
from utils import load_batched_data
```

```python
# Lets check what version of tensorflow we have installed. The provided scripts should run with tf 1.0 and above

print(tf.__version__)
```

```python
batchSize = 4 # Batch of sequences

## Change according to the folder where you saved the dataset provided
INPUT_PATH = 'XX/MCC_sample_data_phoneme_recog/mfcc' #directory of MFCC nFeatures x nFrames 2-D array .npy files
TARGET_PATH = 'XX/MCC_sample_data_phoneme_recog/char_y/' #directory of nCharacters 1-D array .npy files

####Load data
print('Loading data')
batchedData, maxTimeSteps, totalN = load_batched_data(INPUT_PATH, TARGET_PATH, batchSize)
```

```python
####Learning Parameters
learningRate = 0.001
momentum = 0.9
nEpochs = 200

####Network Parameters
nFeatures = 26 #12 MFCC coefficients + energy, and derivatives
nHidden = 128
nClasses = 28#27 characters, plus the "blank" for CTC
```

## Creating the computation graph

We will create a LSTM layer with 128 memory cells. On top of this, we use a fully connected soft-max layer.

We use [CTC classification](http://www.cs.toronto.edu/~graves/icml_2006.pdf) to compute the loss function that we can optimize by gradient descend. This function is already provided in the TF [contributions library](        https://www.tensorflow.org/versions/r0.10/api_docs/python/nn/conectionist_temporal_classification__ctc_)

### LSTMs

Recall the fundamental model

Also, the un-regularized cost function is

\begin{align}
J(\boldsymbol{\theta})=\frac{1}{N}\sum_{n=1}^N\sum_{t=1}^{T_n}d(\boldsymbol{y}_t^{(n)},\text{softmax}(\boldsymbol{W}_h \boldsymbol{h}_t^{(n)}+\mathbf{b}))
\end{align}
where $d(\cdot,\cdot)$ is the cross-entropy loss function.

### Bi-directional LSTMs

\begin{align}
J(\boldsymbol{\theta})=\frac{1}{N}\sum_{n=1}^N\sum_{t=1}^{T_n}d(\boldsymbol{y}_t^{(n)},\text{softmax}(\boldsymbol{W}_h \boldsymbol{h}_t^{(n)}+\boldsymbol{W}_h \boldsymbol{z}_t^{(n)}+\mathbf{b})),
\end{align}
where $\boldsymbol{z}_t^{(n)}$ is the output at time t of the backward LSTM NN.

```python
####Define graph
print('Defining graph')
graph = tf.Graph()
with graph.as_default():

    #### We start by encoding the forward LSTM with nHidden cells

    #i(t) parameters
    # Input gate: input, previous output, and bias.
    ix = tf.Variable(tf.truncated_normal([nFeatures, nHidden], -0.05, 0.05))   ##W^ix
    im = tf.Variable(tf.truncated_normal([nHidden, nHidden], -0.05, 0.05)) ## W^ih
    ib = tf.Variable(tf.zeros([1, nHidden])) ##b_i

    #f(t) parameters
    # Forget gate: input, previous output, and bias.
    fx = tf.Variable(tf.truncated_normal([nFeatures, nHidden], -0.05, 0.05)) ##W^fx
    fm = tf.Variable(tf.truncated_normal([nHidden, nHidden], -0.05, 0.05)) ##W^fh
    fb = tf.Variable(tf.zeros([1, nHidden])) ##b_f

    #g(t) parameters
    # Memory cell: input, state and bias.
    cx = tf.Variable(tf.truncated_normal([nFeatures, nHidden], -0.05, 0.05)) ##W^gx
    cm = tf.Variable(tf.truncated_normal([nHidden, nHidden], -0.05, 0.05)) ##W^gh
    cb = tf.Variable(tf.zeros([1, nHidden]))  ##b_g

    #o(t) parameters
    # Output gate: input, previous output, and bias.
    ox = tf.Variable(tf.truncated_normal([nFeatures, nHidden], -0.05, 0.05))  ##W^ox
    om = tf.Variable(tf.truncated_normal([nHidden, nHidden], -0.05, 0.05))  ##W^oh
    ob = tf.Variable(tf.zeros([1, nHidden])) ##b_o

    # Variable saving state across unrollings.
    saved_output = tf.Variable(tf.zeros([batchSize, nHidden]), trainable=False) #h(t)
    saved_state = tf.Variable(tf.zeros([batchSize, nHidden]), trainable=False) #s(t)

    ####  ####  ####  ####  ####  ####  ####  ####  ####  ####  ####  ####  ####  ####  ####  ####  ####

    # Definition of the cell computation.
    def lstm_cell(i, o, state):
        """Create a LSTM cell. See e.g.: http://arxiv.org/pdf/1402.1128v1.pdf
        Note that in this formulation, we omit the various connections between the
        previous state and the gates."""

        input_gate = tf.sigmoid(tf.matmul(i, ix) + tf.matmul(o, im) + ib)
        forget_gate = tf.sigmoid(tf.matmul(i, fx) + tf.matmul(o, fm) + fb)
        update = tf.matmul(i, cx) + tf.matmul(o, cm) + cb
        state = forget_gate * state + input_gate * tf.tanh(update)    #tf.tanh(update) is g(t)
        output_gate = tf.sigmoid(tf.matmul(i, ox) + tf.matmul(o, om) + ob)
        return output_gate * tf.tanh(state), state      #h(t) is output_gate * tf.tanh(state)

    #### Now the backward LSTM with nHidden cells

    #i(t) parameters
    # Input gate: input, previous output, and bias.
    b_ix = tf.Variable(tf.truncated_normal([nFeatures, nHidden], -0.05, 0.05))   ##W^ix
    b_im = tf.Variable(tf.truncated_normal([nHidden, nHidden], -0.05, 0.05)) ## W^ih
    b_ib = tf.Variable(tf.zeros([1, nHidden])) ##b_i

    #f(t) parameters
    # Forget gate: input, previous output, and bias.
    b_fx = tf.Variable(tf.truncated_normal([nFeatures, nHidden], -0.05, 0.05)) ##W^fx
    b_fm = tf.Variable(tf.truncated_normal([nHidden, nHidden], -0.05, 0.05)) ##W^fh
    b_fb = tf.Variable(tf.zeros([1, nHidden])) ##b_f

    #g(t) parameters
    # Memory cell: input, state and bias.
    b_cx = tf.Variable(tf.truncated_normal([nFeatures, nHidden], -0.05, 0.05)) ##W^gx
    b_cm = tf.Variable(tf.truncated_normal([nHidden, nHidden], -0.05, 0.05)) ##W^gh
    b_cb = tf.Variable(tf.zeros([1, nHidden]))  ##b_g

    #o(t) parameters
    # Output gate: input, previous output, and bias.
    b_ox = tf.Variable(tf.truncated_normal([nFeatures, nHidden], -0.05, 0.05))  ##W^ox
    b_om = tf.Variable(tf.truncated_normal([nHidden, nHidden], -0.05, 0.05))  ##W^oh
    b_ob = tf.Variable(tf.zeros([1, nHidden])) ##b_o

    # Variable saving state across unrollings.
    b_saved_output = tf.Variable(tf.zeros([batchSize, nHidden]), trainable=False) #h(t)
    b_saved_state = tf.Variable(tf.zeros([batchSize, nHidden]), trainable=False) #s(t)

    # Definition of the backward_cell computation.
    def lstm_cell_back(i, o, state):
        """Create a LSTM cell. See e.g.: http://arxiv.org/pdf/1402.1128v1.pdf
        Note that in this formulation, we omit the various connections between the
        previous state and the gates."""

        input_gate = tf.sigmoid(tf.matmul(i, b_ix) + tf.matmul(o, b_im) + b_ib)
        forget_gate = tf.sigmoid(tf.matmul(i, b_fx) + tf.matmul(o, b_fm) + b_fb)
        update = tf.matmul(i, b_cx) + tf.matmul(o, b_cm) + b_cb
        state = forget_gate * state + input_gate * tf.tanh(update)    #tf.tanh(update) is g(t)
        output_gate = tf.sigmoid(tf.matmul(i, b_ox) + tf.matmul(o, b_om) + b_ob)
        return output_gate * tf.tanh(state), state      #h(t) is output_gate * tf.tanh(state)

    ####  ####  ####  ####  ####  ####  ####  ####  ####  ####  ####  ####  ####  ####  ####  ####  ####

    # Classifier weights and biases (over h(t) and b_h(t) to labels)
    w = tf.Variable(tf.truncated_normal([nHidden, nClasses], -0.05, 0.05))
    b = tf.Variable(tf.zeros([nClasses]))
    b_w = tf.Variable(tf.truncated_normal([nHidden, nClasses], -0.05, 0.05))

    ####  ####  ####  ####  ####  ####  ####  ####  ####  ####  ####  ####  ####  ####  ####  ####

    # Now we define the placeholders for the input data

    train_data = list()
    for _ in range(maxTimeSteps):
        train_data.append(tf.placeholder(tf.float32, shape=[batchSize,nFeatures]))

    targetIxs = tf.placeholder(tf.int64)
    targetVals = tf.placeholder(tf.int32)
    targetShape = tf.placeholder(tf.int64)
    targetY = tf.SparseTensor(targetIxs, targetVals, targetShape)
    seqLengths = tf.placeholder(tf.int32, shape=(batchSize))

    # Given the input, we indicate how to compute the hidden states

    # Unrolled forward LSTM loop.
    outputs = list()
    output = saved_output
    state = saved_state
    for i in train_data:
        output, state = lstm_cell(i, output, state)
        outputs.append(output)

    with tf.control_dependencies([saved_output.assign(output),saved_state.assign(state)]):

        # Unrolled backward LSTM loop. Initilialized with the last state of the forward loop!!
        # With the control dependencies command, we make sure we do not run the backward unrolling
        # until the forward one is finished.

        b_outputs = list()
        b_output = output
        b_state = state
        for i in reversed(train_data):
            b_output, b_state = lstm_cell_back(i, b_output, b_state)
            b_outputs.append(b_output)

        b_outputs=b_outputs[::-1]

        with tf.control_dependencies([b_saved_output.assign(b_output),b_saved_state.assign(b_state)]):

            logits = tf.reshape(tf.matmul(tf.concat(axis=0,values=outputs),w)+
                                tf.matmul(tf.concat(axis=0,values=b_outputs),b_w)+b,[-1,batchSize,nClasses])

            #https://www.tensorflow.org/versions/r0.050/api_docs/python/nn/conectionist_temporal_classification__ctc_
            loss = tf.reduce_mean(ctc.ctc_loss(inputs=logits, labels=targetY,  sequence_length=seqLengths))

                ####Optimizing
    optimizer = tf.train.MomentumOptimizer(learningRate, momentum).minimize(loss)

    ####Evaluating
    logitsMaxTest = tf.slice(tf.argmax(logits, 2), [0, 0], [seqLengths[0], 1])
    predictions = tf.to_int32(ctc.ctc_beam_search_decoder(logits, seqLengths)[0][0])
    errorRate = tf.reduce_sum(tf.edit_distance(predictions, targetY, normalize=False)) / tf.to_float(tf.size(targetY.values))
```

```python
####Run session
with tf.Session(graph=graph) as session:
    print('Initializing')
    tf.global_variables_initializer().run()

    for epoch in range(nEpochs):
        print('Epoch', epoch+1, '...')
        batchErrors = np.zeros(len(batchedData))
        batchRandIxs = np.random.permutation(len(batchedData)) #randomize batch order
        for batch, batchOrigI in enumerate(batchRandIxs):
            batchInputs, batchTargetSparse, batchSeqLengths = batchedData[batchOrigI]
            batchTargetIxs, batchTargetVals, batchTargetShape = batchTargetSparse
            feedDict = {targetIxs: batchTargetIxs, targetVals: batchTargetVals,
                        targetShape: batchTargetShape, seqLengths: batchSeqLengths}
            for i in range(maxTimeSteps):
                feedDict[train_data[i]] = batchInputs[i,:,:]

            _, l, er, lmt,logits_out = session.run([optimizer, loss, errorRate, logitsMaxTest,logits], feed_dict=feedDict)
            print(np.unique(lmt)) #print unique argmax values of first sample in batch; should be blank for a while, then spit out target values
            if (batch % 1) == 0:
                print('Minibatch', batch, '/', batchOrigI, 'loss:', l)
                print('Minibatch', batch, '/', batchOrigI, 'error rate:', er)
            batchErrors[batch] = er*len(batchSeqLengths)
        epochErrorRate = batchErrors.sum() / totalN
        print('Epoch', epoch+1, 'error rate:', epochErrorRate)
```

```python
# Lets visualize the prediction for the first training sequence
np.argmax(logits_out[:,0,:],axis=1)
```

```python
import matplotlib.pyplot as plt
```

```python
%matplotlib notebook
plt.plot(logits_out[:,0,27],'k-',label='blank')
plt.plot(logits_out[:,0,10],'b-',label='Index 10')
plt.plot(logits_out[:,0,9],'r-',label='Index 9')
plt.plot(logits_out[:,0,26],'g-',label='Index 12')
plt.legend(loc=4)
plt.xlabel('Time Steps')
plt.ylabel('Output Logits')
```

More about the spykes that tend to apear at the RNN output when using CTC

[Supervised Sequence Labelling](https://www.cs.toronto.edu/~graves/preprint.pdf)

```python

```


```python
import modern_robotics as mr
import sympy as sp
from sympy import*
from sympy.physics.mechanics import dynamicsymbols, mechanics_printing
mechanics_printing()
```

```python
def exp3(omega, theta):
    omega = skew(omega)
    R = sp.eye(3) + sp.sin(theta) * omega + (1 - sp.cos(theta)) * omega * omega
    return R

def skew(v):
    return Matrix([[0, -v[2], v[1]],
                    [v[2], 0, -v[0]],
                    [-v[1], v[0], 0]])

def exp6(twist, theta):
    omega = skew(twist[:3])
    v = Matrix(twist[3:])
    T = eye(4)
    T[:3,:3] = exp3(twist[:3], theta)
    T[:3,3] = (eye(3) * theta + (1 - cos(theta)) * omega +
              (theta-sin(theta)) * omega * omega) * v
    return T

def Ad(T):
    AdT = sp.zeros(6)
    R = sp.Matrix(T[:3, :3])
    AdT[:3, :3] = R
    AdT[3:, 3:] = R
    AdT[3:, :3] = skew(T[:3, 3]) * R
    return AdT

def rotX(alfa_im1):
    Rx = sp.eye(4)
    Rx[1,1] =    sp.cos(alfa_im1)
    Rx[1,2] =   -sp.sin(alfa_im1)
    Rx[2,1] =    sp.sin(alfa_im1)
    Rx[2,2] =    sp.cos(alfa_im1)
    return Rx

def rotZ(alfa_im1):
    Rz = sp.eye(4)
    Rz[0,0] =    sp.cos(alfa_im1)
    Rz[0,1] =   -sp.sin(alfa_im1)
    Rz[1,0] =    sp.sin(alfa_im1)
    Rz[1,1] =    sp.cos(alfa_im1)
    return Rz

def transX(a_im1):
    trA = sp.eye(4)
    trA[0,3] =  a_im1
    return trA

def transZ(d_i):
    trA = sp.eye(4)
    trA[2,3] =  d_i
    return trA

```

```python
# Oppg 1
print("Task 1:")

th1, th2, th3, th4, th5, th6 = dynamicsymbols('theta_1, theta_2, theta_3, theta_4, theta_5, theta_6')

config = sp.Matrix([[0,0,0,th1],[sp.pi/2,0,0,th2],[0,455,0,th3 + sp.pi/2],[sp.pi/2, 35, 420, th4],[sp.pi/2,0,0,th5],[sp.pi/2, 0, -80,th6]])
config
```

    Task 1:

$\displaystyle \left[\begin{matrix}0 & 0 & 0 & \theta_{1}\\\frac{\pi}{2} & 0 & 0 & \theta_{2}\\0 & 455 & 0 & \theta_{3} + \frac{\pi}{2}\\\frac{\pi}{2} & 35 & 420 & \theta_{4}\\\frac{\pi}{2} & 0 & 0 & \theta_{5}\\\frac{\pi}{2} & 0 & -80 & \theta_{6}\end{matrix}\right]$

```python
# oppg 2
Mi = sp.Matrix([[sp.eye(4)]*6])

for i in range(6):
    if i == 1:
        Mi[:,4*i:4*(i+1)] = rotX(config[i,0]) * transX(config[i,1]) * transZ(config[i,2]) * rotZ(-sp.pi/2) # We compansate for the rotation of -pi/2 done when finding the D-H parameters
    else:
        Mi[:,4*i:4*(i+1)] = rotX(config[i,0]) * transX(config[i,1]) * transZ(config[i,2])

M = sp.eye(4)
for n in range(5,-1,-1):
    M = Mi[:,4*n:4*(n+1)] * M

M

```

$\displaystyle \left[\begin{matrix}0 & 0 & 1 & -500\\0 & 1 & 0 & 0\\-1 & 0 & 0 & -490\\0 & 0 & 0 & 1\end{matrix}\right]$

```python
# oppg 3
Ai = sp.Matrix([[0,-1,0,0],[1,0,0,0],[0,0,0,0],[0,0,0,0]]) # This is a given matrix due to revolute joints

S_sp = sp.zeros(6)
for i in range(6):
    dot_sum = sp.eye(4)
    for n in range(i,-1,-1):
        dot_sum = Mi[:,4*n:4*(n+1)] * dot_sum
    S_skew = dot_sum * Ai * sp.Inverse(dot_sum)
    S_sp[0,i] = S_skew[2,1]
    S_sp[1,i] = S_skew[0,2]
    S_sp[2,i] = S_skew[1,0]
    S_sp[3,i] = S_skew[0,3]
    S_sp[4,i] = S_skew[1,3]
    S_sp[5,i] = S_skew[2,3]

S_sp
```

$\displaystyle \left[\begin{matrix}0 & 0 & 0 & -1 & 0 & 1\\0 & -1 & -1 & 0 & 1 & 0\\1 & 0 & 0 & 0 & 0 & 0\\0 & 0 & -455 & 0 & 490 & 0\\0 & 0 & 0 & 490 & 0 & -490\\0 & 0 & 0 & 0 & -420 & 0\end{matrix}\right]$

```python
# oppg 4, find body frame screw axis, S_bp

M_inv = mr.TransInv(M)          #Finding inverse of M
Ad_M_inv = mr.Adjoint(M_inv)    #Computing [Ad_M^-1]

#Using B_i = [Ad_M^-1]S_i
S_bp = sp.zeros(6,6)
for i in range(6):
     S_bp[:, i] = Ad_M_inv * S_sp[:, i]

S_bp
```

$\displaystyle \left[\begin{matrix}-1 & 0 & 0 & 0 & 0 & 0\\0 & -1 & -1 & 0 & 1 & 0\\0 & 0 & 0 & -1 & 0 & 1\\0 & 500 & 500 & 0 & -80 & 0\\-500 & 0 & 0 & 0 & 0 & 0\\0 & 490 & 35 & 0 & 0 & 0\end{matrix}\right]$

## Oppgave 5

```python
import open3d as o3d
import numpy as np

def Ry_sym(theta):
    ct = sp.cos(theta)
    st = sp.sin(theta)
    R = sp.Matrix([[ct, 0.0, st], [0.0, 1.0, 0.0], [-st, 0, ct]])
    return R

class Robot_object:
    Robot_objects = []
    def __init__(self, num_joints, length_links):
        self.num_joints = num_joints
        self.length_links = length_links
        self.Joints = []
        self.Links = []
        self.make_robot_objects()

    def set_colour(self, colour):
        self.colour = colour

    def make_robot_objects(self):
        for i in range(self.num_joints):
            self.Joints.append(Joint())
        for i in range(len(self.length_links)):
            self.Links.append(Link(self.length_links[i]))

    def transform(self, T_list):
        for i, J in enumerate(self.Joints):
            J.transform(T_list[i])
        for i, L in enumerate(self.Links):
            L.transform(T_list[i])

    def draw_robot(self):
        o3d.visualization.draw_geometries(self.Robot_objects)

class Joint(Robot_object):
    colour = [0,1,0]
    def __init__(self):
        self.joint = o3d.geometry.TriangleMesh.create_cylinder(radius=0.1, height=0.3)
        self.coord = o3d.geometry.TriangleMesh.create_coordinate_frame(size=0.5)
        self.update_mesh_list()
        self.set_colour(self.colour)

    def update_mesh_list(self):
        self.Robot_objects.append(self.joint)
        self.Robot_objects.append(self.coord)

    def set_colour(self, colour):
        self.joint.paint_uniform_color(colour)

    def transform(self,T):
        self.joint = self.joint.transform(T)
        self.coord = self.coord.transform(T)

class Link(Robot_object):
    colour = [1,0,1]
    def __init__(self, lenght):
        self.lenght = lenght
        self.link = o3d.geometry.TriangleMesh.create_cylinder(radius=0.01, height=self.lenght).rotate(Ry_sym(np.pi/2)).translate(np.array([self.lenght/2,0,0]))
        self.update_mesh_list()
        self.set_colour(self.colour)

    def update_mesh_list(self):
        self.Robot_objects.append(self.link)

    def set_colour(self, colour):
        self.link.paint_uniform_color(colour)

    def transform(self,T):
        self.link = self.link.transform(T)

f1 = o3d.geometry.TriangleMesh.create_coordinate_frame(size=0.5)

def make_transform_list(Mi, S_list, thetas):
    T_list = [] # List with T01,T02,T03...
    T = np.eye(4)
    for i in range(len(thetas)):
        T = T @ mr.MatrixExp6(mr.VecTose3(S_list[:, i] * thetas[i])) @ M
    T01 = mr.MatrixExp6(mr.VecTose3(S1*th1)) @ M1
```

```python
#"Old" M:

M1=sp.Matrix([[0, 1, 0, 0],
             [1, 0, 0, 0],
             [0, 0, -1, 200],
             [0, 0, 0, 1]])

M2=sp.Matrix([[0, 1, 0, 25],
             [0, 0, 1, 0],
             [1, 0, 0, 400],
             [0, 0, 0, 1]])

M3=sp.Matrix([[1, 0, 0, 25],
             [0, 0, 1, 0],
             [0, -1, 0, 855],
             [0, 0, 0, 1]])

M4=sp.Matrix([[0, 0, -1, 25+420],
             [0, 1, 0, 0],
             [1, 0, 0, 400+455+35],
             [0, 0, 0, 1]])
M5=sp.Matrix([[1, 0, 0, 25+420],
             [0, 0, 1, 0],
             [0, -1, 0, 400+455+35],
             [0, 0, 0, 1]])
M6=sp.Matrix([[0, 0, -1, 525],
             [0, 1, 0, 0],
             [1, 0, 0, 400+455+35],
             [0, 0, 0, 1]])
Mlist = np.array([M1,M2,M3,M4,M5,M6], dtype=float)

om = sp.zeros(3,6)
om1 = om[:, 0] = M1[:3, 2]
om2 = om[:, 1] = M2[:3, 2]
om3 = om[:, 2] = M3[:3, 2]
om4 = om[:, 3] = M4[:3, 2]
om5 = om[:, 4] = M5[:3, 2]
om6 = om[:, 5] = M6[:3, 2]
q = sp.zeros(3,6)
q1 = q[:,0] = M1[:3, 3]
q2 = q[:,1] = M2[:3, 3]
q3 = q[:,2] = M3[:3, 3]
q4 = q[:,3] = M4[:3, 3]
q5 = q[:,4] = M5[:3, 3]
q6 = q[:,5] = M6[:3, 3]

Slist = Slist_maker(om,q)
Slist
```

```python
# oppg 6

M_poe = Matrix([[1,0,0,490],
                [0,1,0,0],
                [0,0,1,-420],
                [0,0,0,1]])

T_s = (exp6(S_sp[:,0], theta[0]) * exp6(S_sp[:,1], theta[1]) * exp6(S_sp[:,2], theta[2]) * exp6(S_sp[:,3], theta[3]) * exp6(S_sp[:,4], theta[4]) * exp6(S_sp[:,5], theta[5])) * M

T_b = M * (exp6(S_bp[:,0], theta[0]) * exp6(S_bp[:,1], theta[1]) * exp6(S_bp[:,2], theta[2]) * exp6(S_bp[:,3], theta[3]) * exp6(S_bp[:,4], theta[4]) * exp6(S_bp[:,5], theta[5]))

T_s.simplify()
T_b.simplify()

```

$\displaystyle \left( \left[\begin{matrix}1 & 0 & 0 & 490\\0 & 1 & 0 & 0\\0 & 0 & 1 & -420\\0 & 0 & 0 & 1\end{matrix}\right], \  \left[\begin{matrix}\left(\left(\sin{\left(\theta_{1} \right)} \sin{\left(\theta_{4} \right)} + \cos{\left(\theta_{2} + \theta_{3} \right)} \cos{\left(\theta_{1} \right)} \cos{\left(\theta_{4} \right)}\right) \cos{\left(\theta_{5} \right)} + \sin{\left(\theta_{2} + \theta_{3} \right)} \sin{\left(\theta_{5} \right)} \cos{\left(\theta_{1} \right)}\right) \cos{\left(\theta_{6} \right)} - \left(\sin{\left(\theta_{1} \right)} \cos{\left(\theta_{4} \right)} - \sin{\left(\theta_{4} \right)} \cos{\left(\theta_{2} + \theta_{3} \right)} \cos{\left(\theta_{1} \right)}\right) \sin{\left(\theta_{6} \right)} & - \left(\left(\sin{\left(\theta_{1} \right)} \sin{\left(\theta_{4} \right)} + \cos{\left(\theta_{2} + \theta_{3} \right)} \cos{\left(\theta_{1} \right)} \cos{\left(\theta_{4} \right)}\right) \cos{\left(\theta_{5} \right)} + \sin{\left(\theta_{2} + \theta_{3} \right)} \sin{\left(\theta_{5} \right)} \cos{\left(\theta_{1} \right)}\right) \sin{\left(\theta_{6} \right)} - \left(\sin{\left(\theta_{1} \right)} \cos{\left(\theta_{4} \right)} - \sin{\left(\theta_{4} \right)} \cos{\left(\theta_{2} + \theta_{3} \right)} \cos{\left(\theta_{1} \right)}\right) \cos{\left(\theta_{6} \right)} & \left(\sin{\left(\theta_{1} \right)} \sin{\left(\theta_{4} \right)} + \cos{\left(\theta_{2} + \theta_{3} \right)} \cos{\left(\theta_{1} \right)} \cos{\left(\theta_{4} \right)}\right) \sin{\left(\theta_{5} \right)} - \sin{\left(\theta_{2} + \theta_{3} \right)} \cos{\left(\theta_{1} \right)} \cos{\left(\theta_{5} \right)} & 35 \left(12 \sin{\left(\theta_{2} + \theta_{3} \right)} + \cos{\left(\theta_{2} + \theta_{3} \right)} + 13 \cos{\left(\theta_{2} \right)}\right) \cos{\left(\theta_{1} \right)}\\\left(\left(\sin{\left(\theta_{1} \right)} \cos{\left(\theta_{2} + \theta_{3} \right)} \cos{\left(\theta_{4} \right)} - \sin{\left(\theta_{4} \right)} \cos{\left(\theta_{1} \right)}\right) \cos{\left(\theta_{5} \right)} + \sin{\left(\theta_{2} + \theta_{3} \right)} \sin{\left(\theta_{1} \right)} \sin{\left(\theta_{5} \right)}\right) \cos{\left(\theta_{6} \right)} + \left(\sin{\left(\theta_{1} \right)} \sin{\left(\theta_{4} \right)} \cos{\left(\theta_{2} + \theta_{3} \right)} + \cos{\left(\theta_{1} \right)} \cos{\left(\theta_{4} \right)}\right) \sin{\left(\theta_{6} \right)} & \left(\left(- \sin{\left(\theta_{1} \right)} \cos{\left(\theta_{2} + \theta_{3} \right)} \cos{\left(\theta_{4} \right)} + \sin{\left(\theta_{4} \right)} \cos{\left(\theta_{1} \right)}\right) \cos{\left(\theta_{5} \right)} - \sin{\left(\theta_{2} + \theta_{3} \right)} \sin{\left(\theta_{1} \right)} \sin{\left(\theta_{5} \right)}\right) \sin{\left(\theta_{6} \right)} + \left(\sin{\left(\theta_{1} \right)} \sin{\left(\theta_{4} \right)} \cos{\left(\theta_{2} + \theta_{3} \right)} + \cos{\left(\theta_{1} \right)} \cos{\left(\theta_{4} \right)}\right) \cos{\left(\theta_{6} \right)} & \left(\sin{\left(\theta_{1} \right)} \cos{\left(\theta_{2} + \theta_{3} \right)} \cos{\left(\theta_{4} \right)} - \sin{\left(\theta_{4} \right)} \cos{\left(\theta_{1} \right)}\right) \sin{\left(\theta_{5} \right)} - \sin{\left(\theta_{2} + \theta_{3} \right)} \sin{\left(\theta_{1} \right)} \cos{\left(\theta_{5} \right)} & 35 \left(12 \sin{\left(\theta_{2} + \theta_{3} \right)} + \cos{\left(\theta_{2} + \theta_{3} \right)} + 13 \cos{\left(\theta_{2} \right)}\right) \sin{\left(\theta_{1} \right)}\\\left(\sin{\left(\theta_{2} + \theta_{3} \right)} \cos{\left(\theta_{4} \right)} \cos{\left(\theta_{5} \right)} - \sin{\left(\theta_{5} \right)} \cos{\left(\theta_{2} + \theta_{3} \right)}\right) \cos{\left(\theta_{6} \right)} + \sin{\left(\theta_{2} + \theta_{3} \right)} \sin{\left(\theta_{4} \right)} \sin{\left(\theta_{6} \right)} & - \left(\sin{\left(\theta_{2} + \theta_{3} \right)} \cos{\left(\theta_{4} \right)} \cos{\left(\theta_{5} \right)} - \sin{\left(\theta_{5} \right)} \cos{\left(\theta_{2} + \theta_{3} \right)}\right) \sin{\left(\theta_{6} \right)} + \sin{\left(\theta_{2} + \theta_{3} \right)} \sin{\left(\theta_{4} \right)} \cos{\left(\theta_{6} \right)} & \sin{\left(\theta_{2} + \theta_{3} \right)} \sin{\left(\theta_{5} \right)} \cos{\left(\theta_{4} \right)} + \cos{\left(\theta_{2} + \theta_{3} \right)} \cos{\left(\theta_{5} \right)} & 35 \sin{\left(\theta_{2} + \theta_{3} \right)} + 455 \sin{\left(\theta_{2} \right)} - 420 \cos{\left(\theta_{2} + \theta_{3} \right)}\\0 & 0 & 0 & 1\end{matrix}\right], \  \left[\begin{matrix}\left(\left(\sin{\left(\theta_{1} \right)} \sin{\left(\theta_{4} \right)} + \cos{\left(\theta_{2} + \theta_{3} \right)} \cos{\left(\theta_{1} \right)} \cos{\left(\theta_{4} \right)}\right) \cos{\left(\theta_{5} \right)} + \sin{\left(\theta_{2} + \theta_{3} \right)} \sin{\left(\theta_{5} \right)} \cos{\left(\theta_{1} \right)}\right) \cos{\left(\theta_{6} \right)} - \left(\sin{\left(\theta_{1} \right)} \cos{\left(\theta_{4} \right)} - \sin{\left(\theta_{4} \right)} \cos{\left(\theta_{2} + \theta_{3} \right)} \cos{\left(\theta_{1} \right)}\right) \sin{\left(\theta_{6} \right)} & - \left(\left(\sin{\left(\theta_{1} \right)} \sin{\left(\theta_{4} \right)} + \cos{\left(\theta_{2} + \theta_{3} \right)} \cos{\left(\theta_{1} \right)} \cos{\left(\theta_{4} \right)}\right) \cos{\left(\theta_{5} \right)} + \sin{\left(\theta_{2} + \theta_{3} \right)} \sin{\left(\theta_{5} \right)} \cos{\left(\theta_{1} \right)}\right) \sin{\left(\theta_{6} \right)} - \left(\sin{\left(\theta_{1} \right)} \cos{\left(\theta_{4} \right)} - \sin{\left(\theta_{4} \right)} \cos{\left(\theta_{2} + \theta_{3} \right)} \cos{\left(\theta_{1} \right)}\right) \cos{\left(\theta_{6} \right)} & \left(\sin{\left(\theta_{1} \right)} \sin{\left(\theta_{4} \right)} + \cos{\left(\theta_{2} + \theta_{3} \right)} \cos{\left(\theta_{1} \right)} \cos{\left(\theta_{4} \right)}\right) \sin{\left(\theta_{5} \right)} - \sin{\left(\theta_{2} + \theta_{3} \right)} \cos{\left(\theta_{1} \right)} \cos{\left(\theta_{5} \right)} & 35 \left(12 \sin{\left(\theta_{2} + \theta_{3} \right)} + \cos{\left(\theta_{2} + \theta_{3} \right)} + 13 \cos{\left(\theta_{2} \right)}\right) \cos{\left(\theta_{1} \right)}\\\left(\left(\sin{\left(\theta_{1} \right)} \cos{\left(\theta_{2} + \theta_{3} \right)} \cos{\left(\theta_{4} \right)} - \sin{\left(\theta_{4} \right)} \cos{\left(\theta_{1} \right)}\right) \cos{\left(\theta_{5} \right)} + \sin{\left(\theta_{2} + \theta_{3} \right)} \sin{\left(\theta_{1} \right)} \sin{\left(\theta_{5} \right)}\right) \cos{\left(\theta_{6} \right)} + \left(\sin{\left(\theta_{1} \right)} \sin{\left(\theta_{4} \right)} \cos{\left(\theta_{2} + \theta_{3} \right)} + \cos{\left(\theta_{1} \right)} \cos{\left(\theta_{4} \right)}\right) \sin{\left(\theta_{6} \right)} & \left(\left(- \sin{\left(\theta_{1} \right)} \cos{\left(\theta_{2} + \theta_{3} \right)} \cos{\left(\theta_{4} \right)} + \sin{\left(\theta_{4} \right)} \cos{\left(\theta_{1} \right)}\right) \cos{\left(\theta_{5} \right)} - \sin{\left(\theta_{2} + \theta_{3} \right)} \sin{\left(\theta_{1} \right)} \sin{\left(\theta_{5} \right)}\right) \sin{\left(\theta_{6} \right)} + \left(\sin{\left(\theta_{1} \right)} \sin{\left(\theta_{4} \right)} \cos{\left(\theta_{2} + \theta_{3} \right)} + \cos{\left(\theta_{1} \right)} \cos{\left(\theta_{4} \right)}\right) \cos{\left(\theta_{6} \right)} & \left(\sin{\left(\theta_{1} \right)} \cos{\left(\theta_{2} + \theta_{3} \right)} \cos{\left(\theta_{4} \right)} - \sin{\left(\theta_{4} \right)} \cos{\left(\theta_{1} \right)}\right) \sin{\left(\theta_{5} \right)} - \sin{\left(\theta_{2} + \theta_{3} \right)} \sin{\left(\theta_{1} \right)} \cos{\left(\theta_{5} \right)} & 35 \left(12 \sin{\left(\theta_{2} + \theta_{3} \right)} + \cos{\left(\theta_{2} + \theta_{3} \right)} + 13 \cos{\left(\theta_{2} \right)}\right) \sin{\left(\theta_{1} \right)}\\\left(\sin{\left(\theta_{2} + \theta_{3} \right)} \cos{\left(\theta_{4} \right)} \cos{\left(\theta_{5} \right)} - \sin{\left(\theta_{5} \right)} \cos{\left(\theta_{2} + \theta_{3} \right)}\right) \cos{\left(\theta_{6} \right)} + \sin{\left(\theta_{2} + \theta_{3} \right)} \sin{\left(\theta_{4} \right)} \sin{\left(\theta_{6} \right)} & - \left(\sin{\left(\theta_{2} + \theta_{3} \right)} \cos{\left(\theta_{4} \right)} \cos{\left(\theta_{5} \right)} - \sin{\left(\theta_{5} \right)} \cos{\left(\theta_{2} + \theta_{3} \right)}\right) \sin{\left(\theta_{6} \right)} + \sin{\left(\theta_{2} + \theta_{3} \right)} \sin{\left(\theta_{4} \right)} \cos{\left(\theta_{6} \right)} & \sin{\left(\theta_{2} + \theta_{3} \right)} \sin{\left(\theta_{5} \right)} \cos{\left(\theta_{4} \right)} + \cos{\left(\theta_{2} + \theta_{3} \right)} \cos{\left(\theta_{5} \right)} & 35 \sin{\left(\theta_{2} + \theta_{3} \right)} + 455 \sin{\left(\theta_{2} \right)} - 420 \cos{\left(\theta_{2} + \theta_{3} \right)}\\0 & 0 & 0 & 1\end{matrix}\right]\right)$

```python
T_s, T_b
```

$\displaystyle \left( \left[\begin{matrix}\left(\left(\sin{\left(\theta_{1} \right)} \sin{\left(\theta_{4} \right)} + \cos{\left(\theta_{2} + \theta_{3} \right)} \cos{\left(\theta_{1} \right)} \cos{\left(\theta_{4} \right)}\right) \cos{\left(\theta_{5} \right)} + \sin{\left(\theta_{2} + \theta_{3} \right)} \sin{\left(\theta_{5} \right)} \cos{\left(\theta_{1} \right)}\right) \cos{\left(\theta_{6} \right)} - \left(\sin{\left(\theta_{1} \right)} \cos{\left(\theta_{4} \right)} - \sin{\left(\theta_{4} \right)} \cos{\left(\theta_{2} + \theta_{3} \right)} \cos{\left(\theta_{1} \right)}\right) \sin{\left(\theta_{6} \right)} & - \left(\left(\sin{\left(\theta_{1} \right)} \sin{\left(\theta_{4} \right)} + \cos{\left(\theta_{2} + \theta_{3} \right)} \cos{\left(\theta_{1} \right)} \cos{\left(\theta_{4} \right)}\right) \cos{\left(\theta_{5} \right)} + \sin{\left(\theta_{2} + \theta_{3} \right)} \sin{\left(\theta_{5} \right)} \cos{\left(\theta_{1} \right)}\right) \sin{\left(\theta_{6} \right)} - \left(\sin{\left(\theta_{1} \right)} \cos{\left(\theta_{4} \right)} - \sin{\left(\theta_{4} \right)} \cos{\left(\theta_{2} + \theta_{3} \right)} \cos{\left(\theta_{1} \right)}\right) \cos{\left(\theta_{6} \right)} & \left(\sin{\left(\theta_{1} \right)} \sin{\left(\theta_{4} \right)} + \cos{\left(\theta_{2} + \theta_{3} \right)} \cos{\left(\theta_{1} \right)} \cos{\left(\theta_{4} \right)}\right) \sin{\left(\theta_{5} \right)} - \sin{\left(\theta_{2} + \theta_{3} \right)} \cos{\left(\theta_{1} \right)} \cos{\left(\theta_{5} \right)} & 35 \left(12 \sin{\left(\theta_{2} + \theta_{3} \right)} + \cos{\left(\theta_{2} + \theta_{3} \right)} + 13 \cos{\left(\theta_{2} \right)}\right) \cos{\left(\theta_{1} \right)}\\\left(\left(\sin{\left(\theta_{1} \right)} \cos{\left(\theta_{2} + \theta_{3} \right)} \cos{\left(\theta_{4} \right)} - \sin{\left(\theta_{4} \right)} \cos{\left(\theta_{1} \right)}\right) \cos{\left(\theta_{5} \right)} + \sin{\left(\theta_{2} + \theta_{3} \right)} \sin{\left(\theta_{1} \right)} \sin{\left(\theta_{5} \right)}\right) \cos{\left(\theta_{6} \right)} + \left(\sin{\left(\theta_{1} \right)} \sin{\left(\theta_{4} \right)} \cos{\left(\theta_{2} + \theta_{3} \right)} + \cos{\left(\theta_{1} \right)} \cos{\left(\theta_{4} \right)}\right) \sin{\left(\theta_{6} \right)} & \left(\left(- \sin{\left(\theta_{1} \right)} \cos{\left(\theta_{2} + \theta_{3} \right)} \cos{\left(\theta_{4} \right)} + \sin{\left(\theta_{4} \right)} \cos{\left(\theta_{1} \right)}\right) \cos{\left(\theta_{5} \right)} - \sin{\left(\theta_{2} + \theta_{3} \right)} \sin{\left(\theta_{1} \right)} \sin{\left(\theta_{5} \right)}\right) \sin{\left(\theta_{6} \right)} + \left(\sin{\left(\theta_{1} \right)} \sin{\left(\theta_{4} \right)} \cos{\left(\theta_{2} + \theta_{3} \right)} + \cos{\left(\theta_{1} \right)} \cos{\left(\theta_{4} \right)}\right) \cos{\left(\theta_{6} \right)} & \left(\sin{\left(\theta_{1} \right)} \cos{\left(\theta_{2} + \theta_{3} \right)} \cos{\left(\theta_{4} \right)} - \sin{\left(\theta_{4} \right)} \cos{\left(\theta_{1} \right)}\right) \sin{\left(\theta_{5} \right)} - \sin{\left(\theta_{2} + \theta_{3} \right)} \sin{\left(\theta_{1} \right)} \cos{\left(\theta_{5} \right)} & 35 \left(12 \sin{\left(\theta_{2} + \theta_{3} \right)} + \cos{\left(\theta_{2} + \theta_{3} \right)} + 13 \cos{\left(\theta_{2} \right)}\right) \sin{\left(\theta_{1} \right)}\\\left(\sin{\left(\theta_{2} + \theta_{3} \right)} \cos{\left(\theta_{4} \right)} \cos{\left(\theta_{5} \right)} - \sin{\left(\theta_{5} \right)} \cos{\left(\theta_{2} + \theta_{3} \right)}\right) \cos{\left(\theta_{6} \right)} + \sin{\left(\theta_{2} + \theta_{3} \right)} \sin{\left(\theta_{4} \right)} \sin{\left(\theta_{6} \right)} & - \left(\sin{\left(\theta_{2} + \theta_{3} \right)} \cos{\left(\theta_{4} \right)} \cos{\left(\theta_{5} \right)} - \sin{\left(\theta_{5} \right)} \cos{\left(\theta_{2} + \theta_{3} \right)}\right) \sin{\left(\theta_{6} \right)} + \sin{\left(\theta_{2} + \theta_{3} \right)} \sin{\left(\theta_{4} \right)} \cos{\left(\theta_{6} \right)} & \sin{\left(\theta_{2} + \theta_{3} \right)} \sin{\left(\theta_{5} \right)} \cos{\left(\theta_{4} \right)} + \cos{\left(\theta_{2} + \theta_{3} \right)} \cos{\left(\theta_{5} \right)} & 35 \sin{\left(\theta_{2} + \theta_{3} \right)} + 455 \sin{\left(\theta_{2} \right)} - 420 \cos{\left(\theta_{2} + \theta_{3} \right)}\\0 & 0 & 0 & 1\end{matrix}\right], \  \left[\begin{matrix}\left(\left(\sin{\left(\theta_{1} \right)} \sin{\left(\theta_{4} \right)} + \cos{\left(\theta_{2} + \theta_{3} \right)} \cos{\left(\theta_{1} \right)} \cos{\left(\theta_{4} \right)}\right) \cos{\left(\theta_{5} \right)} + \sin{\left(\theta_{2} + \theta_{3} \right)} \sin{\left(\theta_{5} \right)} \cos{\left(\theta_{1} \right)}\right) \cos{\left(\theta_{6} \right)} - \left(\sin{\left(\theta_{1} \right)} \cos{\left(\theta_{4} \right)} - \sin{\left(\theta_{4} \right)} \cos{\left(\theta_{2} + \theta_{3} \right)} \cos{\left(\theta_{1} \right)}\right) \sin{\left(\theta_{6} \right)} & - \left(\left(\sin{\left(\theta_{1} \right)} \sin{\left(\theta_{4} \right)} + \cos{\left(\theta_{2} + \theta_{3} \right)} \cos{\left(\theta_{1} \right)} \cos{\left(\theta_{4} \right)}\right) \cos{\left(\theta_{5} \right)} + \sin{\left(\theta_{2} + \theta_{3} \right)} \sin{\left(\theta_{5} \right)} \cos{\left(\theta_{1} \right)}\right) \sin{\left(\theta_{6} \right)} - \left(\sin{\left(\theta_{1} \right)} \cos{\left(\theta_{4} \right)} - \sin{\left(\theta_{4} \right)} \cos{\left(\theta_{2} + \theta_{3} \right)} \cos{\left(\theta_{1} \right)}\right) \cos{\left(\theta_{6} \right)} & \left(\sin{\left(\theta_{1} \right)} \sin{\left(\theta_{4} \right)} + \cos{\left(\theta_{2} + \theta_{3} \right)} \cos{\left(\theta_{1} \right)} \cos{\left(\theta_{4} \right)}\right) \sin{\left(\theta_{5} \right)} - \sin{\left(\theta_{2} + \theta_{3} \right)} \cos{\left(\theta_{1} \right)} \cos{\left(\theta_{5} \right)} & 35 \left(12 \sin{\left(\theta_{2} + \theta_{3} \right)} + \cos{\left(\theta_{2} + \theta_{3} \right)} + 13 \cos{\left(\theta_{2} \right)}\right) \cos{\left(\theta_{1} \right)}\\\left(\left(\sin{\left(\theta_{1} \right)} \cos{\left(\theta_{2} + \theta_{3} \right)} \cos{\left(\theta_{4} \right)} - \sin{\left(\theta_{4} \right)} \cos{\left(\theta_{1} \right)}\right) \cos{\left(\theta_{5} \right)} + \sin{\left(\theta_{2} + \theta_{3} \right)} \sin{\left(\theta_{1} \right)} \sin{\left(\theta_{5} \right)}\right) \cos{\left(\theta_{6} \right)} + \left(\sin{\left(\theta_{1} \right)} \sin{\left(\theta_{4} \right)} \cos{\left(\theta_{2} + \theta_{3} \right)} + \cos{\left(\theta_{1} \right)} \cos{\left(\theta_{4} \right)}\right) \sin{\left(\theta_{6} \right)} & \left(\left(- \sin{\left(\theta_{1} \right)} \cos{\left(\theta_{2} + \theta_{3} \right)} \cos{\left(\theta_{4} \right)} + \sin{\left(\theta_{4} \right)} \cos{\left(\theta_{1} \right)}\right) \cos{\left(\theta_{5} \right)} - \sin{\left(\theta_{2} + \theta_{3} \right)} \sin{\left(\theta_{1} \right)} \sin{\left(\theta_{5} \right)}\right) \sin{\left(\theta_{6} \right)} + \left(\sin{\left(\theta_{1} \right)} \sin{\left(\theta_{4} \right)} \cos{\left(\theta_{2} + \theta_{3} \right)} + \cos{\left(\theta_{1} \right)} \cos{\left(\theta_{4} \right)}\right) \cos{\left(\theta_{6} \right)} & \left(\sin{\left(\theta_{1} \right)} \cos{\left(\theta_{2} + \theta_{3} \right)} \cos{\left(\theta_{4} \right)} - \sin{\left(\theta_{4} \right)} \cos{\left(\theta_{1} \right)}\right) \sin{\left(\theta_{5} \right)} - \sin{\left(\theta_{2} + \theta_{3} \right)} \sin{\left(\theta_{1} \right)} \cos{\left(\theta_{5} \right)} & 35 \left(12 \sin{\left(\theta_{2} + \theta_{3} \right)} + \cos{\left(\theta_{2} + \theta_{3} \right)} + 13 \cos{\left(\theta_{2} \right)}\right) \sin{\left(\theta_{1} \right)}\\\left(\sin{\left(\theta_{2} + \theta_{3} \right)} \cos{\left(\theta_{4} \right)} \cos{\left(\theta_{5} \right)} - \sin{\left(\theta_{5} \right)} \cos{\left(\theta_{2} + \theta_{3} \right)}\right) \cos{\left(\theta_{6} \right)} + \sin{\left(\theta_{2} + \theta_{3} \right)} \sin{\left(\theta_{4} \right)} \sin{\left(\theta_{6} \right)} & - \left(\sin{\left(\theta_{2} + \theta_{3} \right)} \cos{\left(\theta_{4} \right)} \cos{\left(\theta_{5} \right)} - \sin{\left(\theta_{5} \right)} \cos{\left(\theta_{2} + \theta_{3} \right)}\right) \sin{\left(\theta_{6} \right)} + \sin{\left(\theta_{2} + \theta_{3} \right)} \sin{\left(\theta_{4} \right)} \cos{\left(\theta_{6} \right)} & \sin{\left(\theta_{2} + \theta_{3} \right)} \sin{\left(\theta_{5} \right)} \cos{\left(\theta_{4} \right)} + \cos{\left(\theta_{2} + \theta_{3} \right)} \cos{\left(\theta_{5} \right)} & 35 \sin{\left(\theta_{2} + \theta_{3} \right)} + 455 \sin{\left(\theta_{2} \right)} - 420 \cos{\left(\theta_{2} + \theta_{3} \right)}\\0 & 0 & 0 & 1\end{matrix}\right]\right)$

```python

```


Aim: compute QCRB for different lossless systems with input-output transfer functions of the form

$$
T_{uy}(\omega) = \prod_{j=0}^n \frac{\omega + \Delta_j + i\gamma_j}{\omega + \Delta_j - i\gamma_j},
$$

where $\gamma_j \in \mathbb{R}$ is the bandwidth for each mode and $\Delta_j \in \mathbb{R}$ is the detuning. Then calculate QCRB for each internal mode,

$$
\sigma_{xx}^\text{QCRB}(\omega) = \frac{\hbar^2}{4 \bar{S}_{FF}(\omega)} = \frac{\hbar^2}{4 |T_{uF}(\omega)|^2},
$$

assuming measurement shot noise at the input and where $T_{uF}$ is the transfer function from the input $\hat{u}$ to an internal degree of freedom $\hat{F}$ which is coupled to a classical signal $x(t)$ via $\hat{H}_\text{int} = - \hat{F} x$.

So for each $\hat{F}$ want to optimise with respect to $\gamma_j, \Delta_j, \forall j$, then choose the $\hat{F}$ that maximises the QCRB.

```python
%load_ext autoreload
%autoreload
from simba import transfer_function_to_graph, tf2rss, adiabatically_eliminate
from sympy import symbols, simplify, Matrix, sqrt,\
    conjugate, lambdify, I, pi, fraction, solve, Eq, limit
import matplotlib.pyplot as plt
```

## $n = 1$ Detuned Resonator

Note: due to how the transformation to a realisable state space is performed, we have to treat the negative bandwidth and positive bandwidth separately. Need to fix at some point (else will need to perform the transformation $2^n$ times for an $n$ degree-of-freedom system for all possible combinations)

```python
s = symbols('s')
gamma_f = symbols('gamma_f', real=True, positive=True)
Delta = symbols('Delta', real=True)
tf = (s + I * Delta + gamma_f) / (s + I * Delta - gamma_f)

split_network = tf2rss(tf).to_slh().split()
gamma, = split_network.aux_coupling_constants
```

```python
split_network.state_vector
```

$\displaystyle \left[\begin{matrix}a\\\overline{a}\\a'\\\overline{a'}\\ain\\\overline{ain}\\aout\\\overline{aout}\end{matrix}\right]$

```python
tfm = split_network.tfm
input_output = tfm.open_loop('l', 'aout').simplify()
adiabatically_eliminate(input_output, gamma)
```

$\displaystyle \frac{- i \Delta + \gamma_{f} + s}{i \Delta + \gamma_{f} - s}$

```python
Omega = symbols('Omega')
tf_a = tfm.open_loop('ain', 'a').simplify().subs(s, I * Omega)
tf_a
```

$\displaystyle \frac{\sqrt{2} \gamma \sqrt{\gamma_{f}}}{\gamma \gamma_{f} + \left(i \Delta - i \Omega\right) \left(- i \Omega + \gamma\right)}$

```python
tf = adiabatically_eliminate(tf_a, gamma)
tf
```

$\displaystyle \frac{\sqrt{2} \sqrt{\gamma_{f}}}{i \Delta - i \Omega + \gamma_{f}}$

```python
numer, denom = fraction(tf)
solns = solve(denom, Omega)[0]
solns
```

$\displaystyle \Delta - i \gamma_{f}$

We want to evaluate

$$
\int_{-\infty}^{\infty} d\Omega \frac{2 \gamma_f}{(\Omega - \Delta - i \gamma_f)(\Omega - \Delta + i \gamma_f)}
$$

Since the modulus of the integrand $\to 0$ as $\Omega \to \infty$ then we can use Jordan's lemma and this integral is equivalent to,

$$
\oint_C dz \frac{2 \gamma_f}{(z - \Delta - i \gamma_f)(z - \Delta + i \gamma_f)}
$$

where $C$ is an anti-clockwise contour containing the entire upper-half plane.

The function has one simple pole in the upper-half plane: $z = \Delta + i \gamma_f$. The residue of the integrand here is $-i$.

So the integral is

$$
\int_{-\infty}^{\infty} d\Omega \frac{2 \gamma_f}{(\Omega - \Delta - i \gamma_f)(\Omega - \Delta + i \gamma_f)} = 2 \pi
$$

and so it is independent of detuning and bandwidth (Mizuno limit)

## $n = 2$ Coupled Cavity

```python
    gamma_f, omega_s = symbols('gamma_f omega_s', real=True, positive=True)
tf = (s**2 + s * gamma_f + omega_s**2) / (s**2 - s * gamma_f + omega_s**2)
split_network = tf2rss(tf).to_slh().split()
tfm = split_network.tfm
gamma_1, _ = split_network.aux_coupling_constants
```

```python
split_network.state_vector.T
```

$\displaystyle \left[\begin{array}{cccccccccccccccc}a_{1} & \overline{a_{1}} & a_{2} & \overline{a_{2}} & a'_{1} & \overline{a'_{1}} & a'_{2} & \overline{a'_{2}} & ain_{1} & \overline{ain_{1}} & aout_{1} & \overline{aout_{1}} & ain_{2} & \overline{ain_{2}} & aout_{2} & \overline{aout_{2}}\end{array}\right]$

```python
tf = tfm.open_loop('ain_1', 'a_1').simplify()
tf_1 = adiabatically_eliminate(tf, gamma_1).simplify().subs(s, I * Omega)
tf = tfm.open_loop('ain_1', 'a_2').simplify()
tf_2 = adiabatically_eliminate(tf, gamma_1).simplify().subs(s, I * Omega)
```

```python
tf_1
```

$\displaystyle - \frac{\sqrt{2} i \Omega \sqrt{\gamma_{f}}}{- \Omega^{2} - i \Omega \gamma_{f} + \omega_{s}^{2}}$

```python
tf_2
```

$\displaystyle \frac{\sqrt{2} \sqrt{\gamma_{f}} \omega_{s}}{- \Omega^{2} - i \Omega \gamma_{f} + \omega_{s}^{2}}$

```python
numer, denom = fraction(tf_1)
solns = solve(denom, Omega)
Matrix(solns)
```

$\displaystyle \left[\begin{matrix}- \frac{i \gamma_{f}}{2} - \frac{\sqrt{- \gamma_{f}^{2} + 4 \omega_{s}^{2}}}{2}\\- \frac{i \gamma_{f}}{2} + \frac{\sqrt{- \gamma_{f}^{2} + 4 \omega_{s}^{2}}}{2}\end{matrix}\right]$

```python
abs(tf_1)**2
```

$\displaystyle 2 \gamma_{f} \left|{\frac{\Omega}{\Omega^{2} + i \Omega \gamma_{f} - \omega_{s}^{2}}}\right|^{2}$

Note that it goes to zero as $\Omega \to \infty$

For now we assume that $C^2 \equiv -\gamma_f^2 + 4 \omega_s^2 > 0$.

We want to evaluate

$$
\int_{\infty}^{\infty} 2 \gamma_f \frac{d\Omega \Omega^2}{(\Omega - i \gamma_f/2 - C/2)(\Omega - i \gamma_f/2 + C/2)(\Omega + i \gamma_f/2 - C/2)(\Omega + i \gamma_f/2 + C/2)}
$$

Follow same method as before. We have two simple poles in upper half plane: $z = i \gamma_f / 2 + C/2$, $z = i \gamma_f / 2 - C/2$.

The residues are $-i/2 \left(\frac{i \gamma_f}{C} + 1\right)$ and $i/2 \left(\frac{i \gamma_f}{C} - 1\right)$

So the integral is equal to $2 \pi$ and the Mizuno limit is not surpassed.

Now will consider $-C^2 \equiv -\gamma_f^2 + 4 \omega_s^2 < 0$ so the integral is instead,

$$
\int_{\infty}^{\infty} 2 \gamma_f \frac{d\Omega \Omega^2}{(\Omega - i \gamma_f/2 - i C /2)(\Omega - i \gamma_f/2 + i C /2)(\Omega + i \gamma_f/2 - i C /2)(\Omega + i \gamma_f/2 + i C /2)}
$$

Without loss of generality we will choose the positive root so that $C > 0$, so that $C /2 = \sqrt{\gamma_f^2 / 4 - \omega_s^2} < \gamma_f / 2$. So we have two simple poles in the upper-half plane: $z = i \gamma_f / 2 + i C / 2$ and $z = i \gamma_f / 2 - i C / 2$.

The residues are $-i/2 (\gamma_f / C + 1)$ and $i/2 (\gamma_f / C- 1)$ and so, again, the integral is equal to $2 \pi$.

```python
abs(tf_2)**2
```

$\displaystyle 2 \gamma_{f} \omega_{s}^{2} \left|{\frac{1}{\Omega^{2} + i \Omega \gamma_{f} - \omega_{s}^{2}}}\right|^{2}$

Same process for other degree of freedom:

$$
\int_{\infty}^{\infty} 2 \gamma_f \omega_s^2 \frac{d\Omega}{(\Omega - i \gamma_f/2 - C/2)(\Omega - i \gamma_f/2 + C/2)(\Omega + i \gamma_f/2 - C/2)(\Omega + i \gamma_f/2 + C/2)}
$$

Same simple poles $z = i \gamma_f / 2 + C/2$, $z = i \gamma_f / 2 - C/2$. Residues are

$$
-i 2 \omega_s^2 \frac{1}{C(i \gamma_f +C)}
$$

$$i 2 \omega_s^2 \frac{1}{C(i \gamma_f - C)}$$

The sum is $$-i 2 \omega_s^2 (\frac{1}{C(i \gamma_f +C)} - \frac{1}{C(i \gamma_f - C)})$$

$$-2 i \omega_s^2 \frac{(i \gamma_f - C) - (i \gamma_f + C)}{C(i \gamma_f + C)(i \gamma_f - C)} = -2 i \omega_s^2 \frac{-2}{-\gamma_f^2 - C^2} = -2 i \omega_s^2 \frac{2}{4\omega_s^2} = -i.$$

So the integral is equal to $2 \pi$ as expected.

I will assume it is the same for the case where $\gamma_f > 4 \omega_s^2$.

## $n = 2$ Active Coupled Cavity

```python
# parameterise with lambda = g**2 - omega_s**2 > 0
lmbda = symbols('lambda', real=True, positive=True)
tf = (s**2 + s * gamma_f - lmbda) / (s**2 - s * gamma_f - lmbda)
split_network = tf2rss(tf).to_slh().split()
tfm = split_network.tfm
```

```python
tf = tfm.open_loop('ain_1', 'a_1').simplify()
tf_1 = adiabatically_eliminate(tf, gamma_1).simplify().subs(s, I * Omega)
tf = tfm.open_loop('conjugate(ain_1)', 'a_2').simplify()
tf_2 = adiabatically_eliminate(tf, gamma_1).simplify().subs(s, I * Omega)
```

```python
abs(tf_1)**2
```

$\displaystyle 2 \gamma_{f} \left|{\frac{\Omega}{\Omega^{2} + i \Omega \gamma_{f} + \lambda}}\right|^{2}$

```python
numer, denom = fraction(tf_1)
solns = solve(denom, Omega)
Matrix(solns)
```

$\displaystyle \left[\begin{matrix}\frac{i \left(- \gamma_{f} + \sqrt{\gamma_{f}^{2} + 4 \lambda}\right)}{2}\\- \frac{i \left(\gamma_{f} + \sqrt{\gamma_{f}^{2} + 4 \lambda}\right)}{2}\end{matrix}\right]$

Note that square root is always positive here, so the root is always pure imaginary. Let us calculate the residue of the 2 simple poles that lie in the upper-half plane.

```python
integrand = (2*gamma_f*abs(Omega)**2)/ \
    ((Omega + solns[0])*(Omega + solns[0].conjugate())*
     (Omega + solns[1])*(Omega + solns[1].conjugate()))

def get_residue(expr, c):
    return (expr * (Omega - c)).subs(Omega, c).simplify()

first_residue = get_residue(integrand, solns[0])
second_residue = get_residue(integrand, solns[1])

2 * pi * I * (first_residue + second_residue)
```

$\displaystyle 2 \pi$

No improvement over Mizuno limit when coupling signal into first cavity

Let's look at the transfer function from $a_\text{in}^\dagger$ to $a_2$

```python
abs(tf_2)**2
```

$\displaystyle 2 \gamma_{f} \lambda \left|{\frac{1}{\Omega^{2} + i \Omega \gamma_{f} + \lambda}}\right|^{2}$

Roots are the same again.

```python
integrand = (2*gamma_f*lmbda)/ \
    ((Omega + solns[0])*(Omega + solns[0].conjugate())*
     (Omega + solns[1])*(Omega + solns[1].conjugate()))

first_residue = get_residue(integrand, solns[0])
second_residue = get_residue(integrand, solns[1])

2 * pi * I * (first_residue + second_residue).simplify()
```

$\displaystyle 2 \pi$


# Tutorial 4

## Mean Squared Displacement (MSD)

Molecules in liquds, gases and solids do not stay in the same place and move constantly. Think about a drop of dye in a glass of water, as time passes the dye distributes throughout the water. This process is called diffusion and is common throughout nature and an incredibly relevant property for materials scientists who work on things like batteries.

Using the dye as an example, the motion of a dye molecule is not simple. As it moves it is jostled by collisions with other molecules, preventing it from moving in a straight path. If the path is examined in close detail, it will be seen to be a good approximation to a random walk. In mathmatics a random walk is a series of steps, each taken in a random direction. This was analysed by Albert Einstein in a study of Brownian motion and he showed that the mean square of the distance travelled by a particle following a random walk is proportional to the time elapsed.

\begin{align}
\Big \langle r_{i}^{2} \big \rangle & = 6 D_t + C
\end{align}

where

\begin{align}
\Big \langle r_{i}^{2} \big \rangle = \frac{1}{3} \Big< | r_{i}(t) - r_{i}(0) |^2 \Big>.
\end{align}

where $\Big \langle r^2 \big \rangle$ is the mean squared distance, t is time, $D_t$ is the diffusion rate and C is a constant. If $\Big \langle r_{i}^{2} \big \rangle$ is plotted as a function of time, the gradient of the curve obtained is equal to 6 times the self-diffusion coefficient of particle i.
The state of the matter effects the shape of the MSD plot, solids, where little to no diffusion is occuring, has a flat MSD profile. In a liquid however, the particles diffusion randomly and the gradient of the curve is proportional to the diffusion coefficient.

## What is the mean squared displacement

Going back to the example of the dye in water, lets assume for the sake of simplicity that we are in one dimension. Each step can either be forwards or backwards and we cannot predict which. From a given starting position, what distance is our dye molecule likely to travel after 1000 steps? This can be determined simply by adding together the steps, taking into account the fact that steps backwards subtract from the total, while steps forward add to the total. Since both forward and backward steps are equally probable, we come to the surprising conclusion that the probable distance travelled sums up to zero.

By adding the square of the distance we will always be adding positive numbers to our total which now increases linearly with time. Based upon equation 1 it should now be clear that a plot of $\Big \langle r_{i}^{2} \big \rangle$ vs time with produce a line, the gradient of which is equal to 6D. Giving us direct access to the diffusion coefficient of the system.

```python
from polypy import read as rd
from polypy.msd import MSD
from polypy.msd import RegionalMSD
from polypy import analysis
from polypy import utils as ut
from polypy import plotting
import numpy as np
import matplotlib.pyplot as plt
```

This example will use a short (50,000 steps), pre-prepared trajectory of bulk $CaF_2$. In reality we probably want a considerably longer simulation (~10,000,000 steps). Such simulations generate huge files (5GB) and the analysis would take too long for this tutorial.

The first step is to read the history file to generate the data. The `HISTORY` class expects two things, the filename of the history file and a list of atoms to read. It will return a `polypy.read.Trajectory` object, which stores the the atom labels (`Trajectory.atom_labels`), datatype (`Trajectory.data_type`), cartesian coordinates (`Trajectory.cartesian_coordinates`), fractiona coordinates (`Trajectory.fractional_coordinates`), reciprocal lattice vectors (`Trajectory.reciprocal_lv`), lattice vectors (`Trajectory.lv`) cell lengths (`Trajectory.cell_lengths`), total atoms in the file (`Trajectory.atoms_in_history`), timesteps (`Trajectory.timesteps`), total atoms per timestep (`Trajectory.total_atoms`).

```python
history_caf2 = rd.History("../example_data/HISTORY_CaF2", ["F"])
```

Once the data has been read into the code the MSD calculation can be performed using the `MSD` class. The code will return a `polypy.MSD.MSDContainer` object, which contains the MSD information.

```python
f_msd = MSD(history_caf2.trajectory, sweeps=2)

output = f_msd.msd()
```

```python
ax = plotting.msd_plot(output)

plt.show()
```

MSD calculations require a large number of statistics to be considered representative. A full msd will use every single frame of the trajectory as a starting point and effectively do a seperate msd from each starting point, these are then averaged to give the final result.  An MSD is technically an ensemble average over all sweeps and number of particles.
The sweeps paramter is used to control the number of frames that are used as starting points in the calculation. For simulations with lots of diffusion events, a smaller number will be sufficient whereas simulations with a small number of diffusion events will require a larger number.

```python
f_msd = MSD(history_caf2.trajectory, sweeps=10)

output = f_msd.msd()

ax = plotting.msd_plot(output)
plt.show()
```

```python
print("Three Dimensional Diffusion Coefficient", output.xyz_diffusion_coefficient())
print("One Dimensional Diffusion Coefficient in X", output.x_diffusion_coefficient())
print("One Dimensional Diffusion Coefficient in Y", output.y_diffusion_coefficient())
print("One Dimensional Diffusion Coefficient in Z", output.z_diffusion_coefficient())
```

    Three Dimensional Diffusion Coefficient 1.6077008453783734
    One Dimensional Diffusion Coefficient in X 1.6043984537894274
    One Dimensional Diffusion Coefficient in Y 1.6855680306977678
    One Dimensional Diffusion Coefficient in Z 1.5331360516479267

Note:
An MSD is supposed to be linear only after a ballistic regime and it usually lacks statistics for longer times. Thus the linear fit to extract the slope and thus the diffusion coefficient should be done on a portion of the MSD only.
This can be accomplished using the `exclude_initial` and `exclude_final` parameters

```python
print("Three Dimensional Diffusion Coefficient", output.xyz_diffusion_coefficient(exclude_initial=50,
                                                                                exclude_final=50))
print("One Dimensional Diffusion Coefficient in X", output.x_diffusion_coefficient(exclude_initial=50,
                                                                                exclude_final=50))
print("One Dimensional Diffusion Coefficient in Y", output.y_diffusion_coefficient(exclude_initial=50,
                                                                                exclude_final=50))
print("One Dimensional Diffusion Coefficient in Z", output.z_diffusion_coefficient(exclude_initial=50,
                                                                                exclude_final=50))
```

    Three Dimensional Diffusion Coefficient 1.5912662736409342
    One Dimensional Diffusion Coefficient in X 1.5862517497696607
    One Dimensional Diffusion Coefficient in Y 1.6753802400942055
    One Dimensional Diffusion Coefficient in Z 1.5121668310589353

## Arrhenius

It is then possible to take diffusion coefficients, calculated over a large temperature range and, using the Arrhenius equation calculate the activation energy for diffusion. Common sense and chemical intuition suggest that the higher the temperature, the faster a given chemical reaction will proceed. Quantitatively this relationship between the rate a reaction proceeds and its temperature is determined by the Arrhenius Equation. At higher temperatures, the probability that two molecules will collide is higher. This higher collision rate results in a higher kinetic energy, which has an effect on the activation energy of the reaction. The activation energy is the amount of energy required to ensure that a reaction happens.

\begin{align}
k = A * e^{(-Ea / RT)}
\end{align}

where k is the rate coefficient, A is a constant, Ea is the activation energy, R is the universal gas constant, and T is the temperature (in kelvin).

## Ionic Conductivity

Usefully, as we have the diffusion coefficient, the number of particles (charge carriers) and the ability to calculate the volume, we can convert this data into the ionic conductivity and then the resistance.

\begin{align}
\sigma & = \frac{D C_F e^2}{k_B T}
\end{align}

where $\sigma$ is the ionic conductivity, D is the diffusion coefficient, $C_F$ is the concentration of charge carriers, which in this case if F ions, $e^2$ is the charge of the diffusing species, $k_B$ is the Boltzmann constant and T is the temperature.

The resitance can then be calculated according to

\begin{align}
\Omega & = \frac{1}{\sigma}
\end{align}

So the first step is to calculate the volume, the system voume module will do this from the given data.

```python
volume, step = analysis.system_volume(history_caf2.trajectory)
average_volume = np.mean(volume[:50])
```

The number of charge carriers is just the total number of atoms.

```python
sigma = analysis.conductivity(history_caf2.trajectory.total_atoms,
                        average_volume,
                        output.xyz_diffusion_coefficient(),
                        1500, 1)
```

```python
print("Ionic Conductivity :", sigma)
```

    Ionic Conductivity : 0.0008752006872146488

```python
print("Resistivity :", (1 / sigma))
```

    Resistivity : 1142.5950808865662

## Simulation Length

It is important to consider the lenght of your simulation (Number of steps). The above examples use a short trajectory but it is at a sufficient temperature that there are enough diffusion events to get a good MSD plot. The following example is of a very short simulation, you will hopefully note that the MSD plot is clearly not converged.

```python
history_short = rd.History("../example_data/HISTORY_short", atom_list=["F"])
```

```python
f_msd_short = MSD(history_short.trajectory, sweeps=2)

output = f_msd_short.msd()
```

```python
ax = plotting.msd_plot(output)
plt.show()
```

```python
print("Three Dimensional Diffusion Coefficient", output.xyz_diffusion_coefficient())
print("One Dimensional Diffusion Coefficient in X", output.x_diffusion_coefficient())
print("One Dimensional Diffusion Coefficient in Y", output.y_diffusion_coefficient())
print("One Dimensional Diffusion Coefficient in Z", output.z_diffusion_coefficient())
```

    Three Dimensional Diffusion Coefficient 1.5862125967428773
    One Dimensional Diffusion Coefficient in X 1.572862308748596
    One Dimensional Diffusion Coefficient in Y 1.6321092976984717
    One Dimensional Diffusion Coefficient in Z 1.553666183781564

Amusingly, this actually does not seem to have a huge effect on the diffusion coefficient compared to the longer simulation. However these trajectories are from a CaF$_2$ simulation at 1500 K and there are thus a large number of diffusion events in the short time frame.

## State of Matter

It is possible to identify the phase of matter from the MSD plot.

    Figure 1. The anticipated MSD form for each state of matter.

The fluorine diffusion discussed already clearly shows that the fluorine sub lattice has melted and the diffusion is liquid like. Whereas, carrying out the same analysis on the calcium sub lattice shows that while the fluorine sub lattice has melted, the Calcium sub lattice is still behaving like a solid.

```python
f_msd = MSD(history_caf2.trajectory, sweeps=2)

output = f_msd.msd()

ax = plotting.msd_plot(output)
plt.show()
```

## Regional MSD Calculations

Often in solid state chemistry simulations involve defects, both structural e.g. grain boundaries, dislocations and surface, and chemical e.g. point defects. It is important to try and isolate the contributions of these defects to the overall properties. Regarding diffusion, it could be imagined that a certain region within a structure will have different properties compared with the stoichiometric bulk, e.g. a grain boundary vs the grains, or the surface vs the bulk. `polypy` has the capability to isolate trajectories that pass within certain regions of a structure and thus calculate a diffusion coefficient for those regions.

In this example we will calculate the diffusion coefficient in a box between -5.0 and 5.0 in the dimension of the first lattice vector.

```python
f_msd = RegionalMSD(history_caf2.trajectory, -5, 5, dimension="x")
output = f_msd.analyse_trajectory()
```

```python
ax = plotting.msd_plot(output)

plt.show()
```

```python
print("Three Dimensional Diffusion Coefficient", output.xyz_diffusion_coefficient())
print("One Dimensional Diffusion Coefficient in X", output.x_diffusion_coefficient())
print("One Dimensional Diffusion Coefficient in Y", output.y_diffusion_coefficient())
print("One Dimensional Diffusion Coefficient in Z", output.z_diffusion_coefficient())
```

    Three Dimensional Diffusion Coefficient 1.5967417785595026
    One Dimensional Diffusion Coefficient in X 1.611906338211519
    One Dimensional Diffusion Coefficient in Y 1.670973618673711
    One Dimensional Diffusion Coefficient in Z 1.5073453787932771

#### DLMONTE

```python
archive = rd.Archive("../example_data/ARCHIVE_LLZO", atom_list=["O"])
```

```python
f_msd = MSD(archive.trajectory, sweeps=2)
```


 INF285 - Computación Científica

 Least Squares

  [S]cientific [C]omputing [T]eam

 Version: 1.34

## Table of Contents
* [Introduction](#intro)
* [QR Factorization](#qr)
    * [Do we really have _truly_ unitary vectors?](#DoWeReallyHaveUnitaryVectors)
    * [Examples of QR: Classic vs Modified Gram-Schmidt](#ex)
* [Overdetermined Linear System of Equations](#over)
    * [Initial Example](#initialExample)
* [Examples of models](#sm)
* [Acknowledgements](#acknowledgements)

```python
import numpy as np
import matplotlib.pyplot as plt
import scipy.linalg as spla
%matplotlib inline
# https://scikit-learn.org/stable/modules/classes.html#module-sklearn.datasets
from sklearn import datasets
import ipywidgets as widgets
import matplotlib as mpl
mpl.rcParams['font.size'] = 14
mpl.rcParams['axes.labelsize'] = 20
mpl.rcParams['xtick.labelsize'] = 14
mpl.rcParams['ytick.labelsize'] = 14
M=8
from ipywidgets import interact
from colorama import Fore, Back, Style
# https://pypi.org/project/colorama/
# Fore: BLACK, RED, GREEN, YELLOW, BLUE, MAGENTA, CYAN, WHITE, RESET.
# Back: BLACK, RED, GREEN, YELLOW, BLUE, MAGENTA, CYAN, WHITE, RESET.
# Style: DIM, NORMAL, BRIGHT, RESET_ALL
textBold = lambda x: Style.BRIGHT+x+Style.RESET_ALL
textBoldH = lambda x: Style.BRIGHT+Back.YELLOW+x+Style.RESET_ALL
```

## Introduction
[Back to TOC](#toc)

We have learned about square linear system of equations.
However, how can we solve a non-square linear system of equations? i.e. when we have more equations than unknowns! Well, the traditional step will be to find a least squares approximation!

# QR Factorization

## Gram-Schmidt Orthogonalization
[Back to TOC](#toc)

This algorithm orthonormalize a set of input vectors, returning an **orthonormal set** that spans the same vector space.
We will only consider now that the input set of vectors are **linearly independent**.

Let $A=[\mathbf{a}_1\, ...., \mathbf{a}_n]$ a matrix with linearly independent column vectors $\in\mathbb{R}^m$ and $n \le m$.

We know the following for the orthogonal set:

\begin{align*}
    \mathbf{q}_i^T\,\mathbf{q}_i & = \|\mathbf{q}_i\|_2^2= 1\\
    \mathbf{q}_i^T\,\mathbf{q}_j & = 0, \, \text{ for } i\neq j
\end{align*}

Then the **classic** Gram-Schmidt orthonormalization finds the $\mathbf{q}_i$ and $r_{ij}$ from the following set of equations and considering the previous constraints:
\begin{align*}
    \mathbf{a}_1 &= r_{11}\,\mathbf{q}_1\\
    r_{11} &= \|\mathbf{a}_1\|_2\\
    \mathbf{q}_1 &= \dfrac{\mathbf{a}_1}{r_{11}}\\
    \mathbf{a}_2 &= r_{12}\,\mathbf{q}_1+r_{22}\,\mathbf{q}_2\\
    r_{12} &= \mathbf{q}_1^T\,\mathbf{a}_2\\
    r_{22} &= \|\mathbf{a}_2-r_{12}\,\mathbf{q}_1\|_2\\
    \mathbf{q}_2 &= \dfrac{\mathbf{a}_2-r_{12}\,\mathbf{q}_1}{r_{22}}\\
    \vdots &=  \vdots\\
    \mathbf{a}_k &= \sum_{i=1}^k r_{i,k}\,\mathbf{q}_i\\
    r_{i,k} &= \mathbf{q}_i^T\,\mathbf{a}_k, \, \text{ for } i<j\\
    r_{k,k} &= \left\|\mathbf{a}_k-\sum_{i=1}^{k-1} r_{i,k}\,\mathbf{q}_i\right\|_2\\
    \mathbf{q}_k &= \dfrac{\mathbf{a}_k-\sum_{i=1}^{k-1} r_{i,k}\,\mathbf{q}_i}{r_{k,k}}\\
    \vdots &=  \vdots\\
    \mathbf{a}_n &= \sum_{i=1}^n r_{in}\,\mathbf{q}_i\\
    r_{in} &= \mathbf{q}_i^T\,\mathbf{a}_n, \, \text{ for } i<n\\
    r_{nn} &= \left\|\mathbf{a}_n-\sum_{i=1}^{n-1} r_{in}\,\mathbf{q}_i\right\|_2\\
    \mathbf{q}_n &= \dfrac{\mathbf{a}_n-\sum_{i=1}^{n-1} r_{in}\,\mathbf{q}_i}{r_{nn}}
\end{align*}

Thus, we obtain the QR decomposition as follows:

\begin{equation}
                 \mathbf{A}_{m\times n} = \widehat{Q}_{m\times n}\widehat{R}_{n\times n},
\end{equation}
where $\widehat{Q}$ is a matrix of vectors $\mathbf{q}_{k}$, and $\widehat{R}$ is an upper-triangular matrix, with the coefficients $r_{i,k}$:

This is known as the **Reduced QR factorization**.

**[IMPORTANT]**
- What is then a **full** QR decomposition?
- What is then a **modified** Gram-Schmidt orthonormalization?

```python
def QR(A, type_factorization = 'reduced', type_gram_schmidt='classic'):
    A.astype('float')
    m,n = A.shape # m: number of rows, n: number of columns.
    if type_factorization == 'reduced':
        Q = np.zeros((m,n))
        R = np.zeros((n,n))
    elif type_factorization == 'full':
        Q = np.zeros((m,m))
        R = np.zeros((m,n))
    for k in range(n):
        y = A[:,k]
        for i in range(k):
            if type_gram_schmidt == 'classic':
                R[i,k] = np.dot(Q[:,i],A[:,k])
            elif type_gram_schmidt == 'modified':
                R[i,k] = np.dot(Q[:,i],y)
            y=y-R[i,k]*Q[:,i]
        R[k,k] = np.linalg.norm(y)
        Q[:,k] = y/np.linalg.norm(R[k,k])
    # The following lines must be completed by you!
    #if type_factorization == 'full':
        # (1) We need to add 0's to the R matrix so it is of the same shape as the matrix A,
        # fortunately this was already done!
        # (2) We need to add orthogonal vectors to Q so it is square,
        # how do we do this?
    return Q,R
```

 ### First example:

```python
A = np.array([[1,-4],[2,3],[2,2]])
Q_hat, R_hat = QR(A, type_factorization ='reduced', type_gram_schmidt='classic')
print(textBold('A:\n'),A)
print(textBold('Q_hat @ R_hat:\n'), Q_hat @ R_hat)
print(textBold('Q_hat:\n'), Q_hat)
print(textBold('R_hat:\n'), R_hat)
print(textBold('Q_hat.T @ Q_hat:\n'), Q_hat.T @ Q_hat)
```

    A:
     [[ 1 -4]
     [ 2  3]
     [ 2  2]]
    Q_hat @ R_hat:
     [[ 1. -4.]
     [ 2.  3.]
     [ 2.  2.]]
    Q_hat:
     [[ 0.33333333 -0.93333333]
     [ 0.66666667  0.33333333]
     [ 0.66666667  0.13333333]]
    R_hat:
     [[3. 2.]
     [0. 5.]]
    Q_hat.T @ Q_hat:
     [[1.00000000e+00 2.28212511e-17]
     [2.28212511e-17 1.00000000e+00]]

## Do we really have _truly_ unitary vectors?

[Back to TOC](#toc)

This is an example that shows that unitary vectors are not truly unitary in _double precision_!
For instance, in this case we define the vector $\mathbf{v}=[1,\delta]$, and we then make it unitary, which should result in $\mathbf{w}=\dfrac{\mathbf{v}}{\|\mathbf{v}\|}=\left[\dfrac{1}{\sqrt{1+\delta^2}},\dfrac{\delta}{\sqrt{1+\delta^2}}\right]$.
This is indeed true for _not too small_ values of $\delta$, however if $\delta^2<\epsilon_{\text{mach}}$, then the resulting vector $\mathbf{w}$ will be equal to $\mathbf{v}$.

```python
def show_unitary_vector(n=0):
    # Small value 'delta'.
    delta = np.power(10.,n)

    print(textBold('delta=10^n= \t\t:'),delta)

    v=np.array([1,delta])
    print(textBold('Vector v\t\t:'),v)

    w=v/np.linalg.norm(v)

    print(textBold('Vector w=v/||v||\t:'),w)

    print(textBold('Vector v-w=\t\t:'),v-w)

    #print('Thus, they are the same for small values of d!!\nThis means that w is not truly unitary!')
    print(textBoldH('Are they different for all values of $\delta=10^{n}$?'))

interact(show_unitary_vector,n=(-20,20,1))
```

    interactive(children=(IntSlider(value=0, description='n', max=20, min=-20), Output()), _dom_classes=('widget-i…

    <function __main__.show_unitary_vector(n=0)>

## Examples of QR: Classic vs Modified Gram-Schmidt

[Back to TOC](#toc)

```python
# Fore: BLACK, RED, GREEN, YELLOW, BLUE, MAGENTA, CYAN, WHITE, RESET.
# Back: BLACK, RED, GREEN, YELLOW, BLUE, MAGENTA, CYAN, WHITE, RESET.
# Style: DIM, NORMAL, BRIGHT, RESET_ALL
textQRClassic = lambda x: Style.BRIGHT+Fore.RED+x+Style.RESET_ALL
textQRModified = lambda x: Style.BRIGHT+Fore.BLUE+x+Style.RESET_ALL
textAnswer = lambda x: Back.BLACK+Fore.CYAN+x+Style.RESET_ALL
```

```python
# Example 4.16 from textbook.
d = 1e-10
A = np.array([[1,1,1],[d,0,0],[0,d,0],[0,0,d]])
print(textBold('A: \n'), A)

Qc,Rc = QR(A, type_gram_schmidt = 'classic')
Qm,Rm = QR(A, type_gram_schmidt = 'modified')

################################
print(textBoldH('\n What are the Q\'s?'))
print(textQRClassic('Qc: \n'), Qc)
print(textQRModified('Qm: \n'), Qm)

################################
print(textBoldH('\n Are truly orthogonal the q_i\'s?'))
print(textQRClassic('Qc.T*Qc: \n'),np.dot(Qc.T,Qc))
print(textQRClassic('||Qc.T*Qc-I_3||: '),np.linalg.norm(np.dot(Qc.T,Qc)-np.eye(3))) # Warning: We are just using the transpose since the matrices are real!
print(textAnswer('Not really, since q2^T*q3 \\neq 0.'))

print(textQRModified('\n Qm.T*Qm: \n'),np.dot(Qm.T,Qm))
print(np.linalg.norm(np.dot(Qm.T,Qm)-np.eye(3)))
print(textQRModified('||Qm.T*Qm-I_3||: '),np.linalg.norm(np.dot(Qm.T,Qm)-np.eye(3)))
print(textAnswer('This looks much better!'))

################################
print(textBoldH('\n Do we recover A with each algorithm?'))
print(textQRClassic('A-Qc*Rc: \n'),A-np.dot(Qc,Rc))
print(textQRClassic('||A-Qc*Rc||: '),np.linalg.norm(A-np.dot(Qc,Rc)))
print(textAnswer('Yes!'))

print(textQRModified('\n A-Qm*Rm: \n'),A-np.dot(Qm,Rm))
print(textQRModified('||A-Qm*Rm||: '),np.linalg.norm(A-np.dot(Qm,Rm)))
print(textAnswer('Yes!'))

```

    A:
     [[1.e+00 1.e+00 1.e+00]
     [1.e-10 0.e+00 0.e+00]
     [0.e+00 1.e-10 0.e+00]
     [0.e+00 0.e+00 1.e-10]]

     What are the Q's?
    Qc:
     [[ 1.00000000e+00  0.00000000e+00  0.00000000e+00]
     [ 1.00000000e-10 -7.07106781e-01 -7.07106781e-01]
     [ 0.00000000e+00  7.07106781e-01  0.00000000e+00]
     [ 0.00000000e+00  0.00000000e+00  7.07106781e-01]]
    Qm:
     [[ 1.00000000e+00  0.00000000e+00  0.00000000e+00]
     [ 1.00000000e-10 -7.07106781e-01 -4.08248290e-01]
     [ 0.00000000e+00  7.07106781e-01 -4.08248290e-01]
     [ 0.00000000e+00  0.00000000e+00  8.16496581e-01]]

     Are truly orthogonal the q_i's?
    Qc.T*Qc:
     [[ 1.00000000e+00 -7.07106781e-11 -7.07106781e-11]
     [-7.07106781e-11  1.00000000e+00  5.00000000e-01]
     [-7.07106781e-11  5.00000000e-01  1.00000000e+00]]
    ||Qc.T*Qc-I_3||:  0.7071067811865477
    Not really, since q2^T*q3 \neq 0.

     Qm.T*Qm:
     [[ 1.00000000e+00 -7.07106781e-11 -4.08248290e-11]
     [-7.07106781e-11  1.00000000e+00 -1.66533454e-16]
     [-4.08248290e-11 -1.66533454e-16  1.00000000e+00]]
    1.1547005383859233e-10
    ||Qm.T*Qm-I_3||:  1.1547005383859233e-10
    This looks much better!

     Do we recover A with each algorithm?
    A-Qc*Rc:
     [[0.00000000e+00 0.00000000e+00 0.00000000e+00]
     [0.00000000e+00 0.00000000e+00 3.62324963e-27]
     [0.00000000e+00 0.00000000e+00 0.00000000e+00]
     [0.00000000e+00 0.00000000e+00 0.00000000e+00]]
    ||A-Qc*Rc||:  3.62324963348974e-27
    Yes!

     A-Qm*Rm:
     [[0.00000000e+00 0.00000000e+00 0.00000000e+00]
     [0.00000000e+00 0.00000000e+00 1.77879510e-27]
     [0.00000000e+00 0.00000000e+00 3.12420006e-27]
     [0.00000000e+00 0.00000000e+00 0.00000000e+00]]
    ||A-Qm*Rm||:  3.5950991686535524e-27
    Yes!

# Overdetermined Linear Systems of Equations
[Back to TOC](#toc)

There is cases where the number of equations is greater than variables. Many times, those systems don't have an exact solution (inconsistent system). Then, in this case we needs an approximation closest to the data. Based in orthogonality, the shortest distance from a point to plane. The orthogonal distance represents the error which would be minimum.

\begin{equation}
                 \mathbf{b} - A\,\overline{\mathbf{x}} = \mathbf{r}\\
                 \mathbf{b} - A\,\overline{\mathbf{x}} \perp \{A\,\mathbf{x}\, |\, \mathbf{x} \in \mathbb{R}^m\}
\end{equation}

The idea is that $\mathbf{r}$ would be closest to zero. We need to apply orthogonality to find the vector that satisfied this condition.

\begin{equation}
                 (A\,\mathbf{x})^*\,(\mathbf{b}-A\,\overline{\mathbf{x}})=0, \hspace{1cm} \text{for all } \mathbf{x} \in \mathbb{R}^n\\
                 \mathbf{x}^*\, A^*\,(\mathbf{b}-A\,\overline{\mathbf{x}})=0, \hspace{1cm} \text{for all } \mathbf{x} \in \mathbb{R}^n\\
                 A^*\,(\mathbf{b}-A\,\overline{\mathbf{x}})=\mathbf{0} \\
                 A^*\,A\,\overline{\mathbf{x}}= A^*\,\mathbf{b} \\
\end{equation}

This last equation gives us a new square $n\times n$ matrix, which let us resolve the equation system.
This linear system of equations is known as the **Normal Equations**.

### Initial example:
[Back to TOC](#toc)

In this example we will approximate $m$ points considering a linear relationship.
This means that we will have the data points $(x_i,y_i)$ for $i\in\{1,2,\dots,m\}$ and consider the relationshop $y=a_0+a_1\,b$.
The error that will be added follows a normal distribution.

```python
# Number of points to be used
m = 10
# Relationship considered
fv = np.vectorize(lambda x, a0, a1: a0+a1*x)
# Coefficients considered
a0, a1 = 1, 4

# Error:
np.random.seed(0)
# Standard deviation for the error
sigma = 5e-1
# Error to be added
e = np.random.normal(0,sigma,m)

# Generating data points
x = np.linspace(0,1,m)
y = fv(x,a0,a1)+e
# Challenging question: What if all the xi are the same?
# This means that considering the following definition for x, for instance:
# x = np.linspace(0,1,m)*0+0.5

# Plotting the data
plt.figure(figsize=(10,10))
plt.plot(x,y,'b.',markersize=20)
plt.grid(True)
plt.xlabel(r'$x$')
plt.ylabel(r'$y$')
plt.show()
```

```python
# Build the data matrix
A = np.ones((m,2))
A[:,1] = x
# Setting up the right hand side
b = y

# Building and solving the normal equations
# A^T A x_bar = A^T b
x_bar = np.linalg.solve(A.T @ A, A.T @ b)
# Showing solution found, please compare with a0 and a1 defined previously.
# Are the the same? Are they close? What does it happen with them if we increase "m" and decrease "sigma"?
print(x_bar)
```

    [1.75181763 3.23438791]

```python
# Showing the comparison between the "original function" and the "least-squared reconstructed approximation".
# We added in red a "sample" of possible functions.
# Notice that the colors used follow the description included in the classnotes.
# This means to consider the following analogy:
# blue: data points, this correspond to the right-hand-side vector "b".
# red: this correspond to the sub-space generated by Ax, i.e. the span of the columns of A.
# violet: This correspond to the least-square solution found.
plt.figure(figsize=(10,10))
for i in range(100):
    plt.plot(x,fv(x,x_bar[0]+np.random.normal(0,1),x_bar[1]+np.random.normal(0,1)),'r-',linewidth=1,alpha=0.2)
plt.plot(x,y,'b.',markersize=20)
plt.plot(x,fv(x,x_bar[0],x_bar[1]),'r.',markersize=30)
plt.plot(x,fv(x,a0,a1),'k-',linewidth=8,alpha=0.8)
plt.plot(x,fv(x,x_bar[0],x_bar[1]),'--',color='darkviolet',linewidth=4)
plt.grid(True)
plt.xlabel(r'$x$')
plt.ylabel(r'$y$')
plt.show()
```

```python
# Computing the quadratic error ||r||_2^2 of the least-square solution
print('Quadratic error of LS solution: ',np.linalg.norm(b-A @ x_bar)**2)
# Computing the quadratic error ||r||_2^2 of the "original" parameters
x_no_bar = np.array([a0,a1])
print('Quadratic error of original parameters: ',np.linalg.norm(b-A @ x_no_bar)**2)

# Verifying if the columns of A are orthogonal to the "residual" vector rLS=b-A x_bar.
print('Inner product of rLS and each column of A: ',A.T @ (b-A @ x_bar))

# Computing the ortogonality condition against the "original" a0 y a1, i.e. rOR=b-A x_no_bar
print('Inner product of rOR and each column of A: ',A.T @ (b-A @ x_no_bar))
```

    Quadratic error of LS solution:  1.7410887838975284
    Quadratic error of original parameters:  3.6998010025823156
    Inner product of rLS and each column of A:  [-2.22044605e-16 -1.72084569e-15]
    Inner product of rOR and each column of A:  [3.69011585 1.06526784]

# Examples of models
[Back to TOC](#toc)

```python
def least_squares(A,b):
    Q,R = QR(A,type_gram_schmidt='modified')
    return spla.solve_triangular(R,np.dot(Q.T,b))

def solve_model(M):
    A=M['A']
    b=M['b']
    M['x_bar']=least_squares(A,b)
    return M

def create_model(data, type_model='linear'):
    if type_model == 'linear': # f(x)=a0+a1*x
        A = np.ones((data.shape[0],2))
        A[:,1] = data[:,0]
        b = data[:,1]
    if type_model == 'parabollic': # f(x)=a0+a1*x+a_2*x^2
        A = np.ones((data.shape[0],3))
        A[:,1] = data[:,0]
        A[:,2] = data[:,0]**2
        b = data[:,1]
    if type_model == 'exponential': #f(x)=a0 \exp(a1*x) = \exp(\log(a0)+a1*x) -> log(f(x))=log(a0)+a1*x = A0+a1+x (it is linear now!)
        A = np.ones((data.shape[0],2))
        A[:,1] = data[:,0]
        b = np.log(data[:,1])
    M = {'A':A,
         'b':b,
         'type_model':type_model}
    M=solve_model(M)
    return M

def evaluate_model(M,x):
    x_bar=M['x_bar']
    if M['type_model'] == 'linear':
        return x_bar[0] + x_bar[1]*x
    if M['type_model'] == 'parabollic':
        return x_bar[0] + x_bar[1]*x + x_bar[2]*x**2
    if M['type_model'] == 'exponential':
        return np.exp(x_bar[0]+x_bar[1]*x)
```

```python
def generate_data(type_of_data='linear'):
    n=40
    np.random.seed(0)
    x = np.linspace(0,10,n)
    y = np.random.rand(n)
    x = np.concatenate((x,x,y),axis=0)
    n = 3*n
    if type_of_data=='linear':
        y = x+0.1*np.random.normal(0,1,n)+1.5
    elif type_of_data=='parabollic':
        y = 4*x**2+0.1*x*np.random.normal(0,1,n)+1.5
    elif type_of_data=='exponential':
        y = np.exp(x+0.1*np.random.normal(0,1,n)+1.5)
    elif type_of_data=='sinusoidal':
        y = np.sin(2*np.pi*x/10)+0.1*np.random.normal(0,1,n)+1.5
    elif type_of_data=='random':
        y = 0.1*np.random.normal(0,1,n)+1.5
    elif type_of_data=='boston house-prices':
        x,y=datasets.load_boston(return_X_y=True)
        x=x[:,5]
    elif type_of_data=='diabetes':
        x,y=datasets.load_diabetes(return_X_y=True)
        x=x[:,2]
    data = np.stack((x, y)).T
    return data
```

```python
def looking_at_data(type_of_data='diabetes'):
    data=generate_data(type_of_data)
    Ml = create_model(data, type_model='linear')
    Mp = create_model(data, type_model='parabollic')
    Me = create_model(data, type_model='exponential')
    xx=np.linspace(np.min(data[:,0])-0.1,np.max(data[:,0])+0.1,1000)
    yyl=evaluate_model(Ml,xx)
    yyp=evaluate_model(Mp,xx)
    yye=evaluate_model(Me,xx)

    error_l=data[:,1]-evaluate_model(Ml,data[:,0])
    error_p=data[:,1]-evaluate_model(Mp,data[:,0])
    error_e=data[:,1]-evaluate_model(Me,data[:,0])

    plt.figure(figsize=(2*M,M))

    plt.subplot(1, 2, 1)
    plt.plot(xx,yyl,'k-',linewidth=5,label='linear model')
    plt.plot(xx,yyp,'y-',linewidth=20,label='parabollic model',alpha=0.4)
    plt.plot(xx,yye,'g-',linewidth=5,label='exponential model')
    plt.plot(data[:,0],data[:,1],'.b',markersize=20,label='original data',alpha=0.3)
    plt.grid(True)
    plt.xlabel(r'$x$')
    plt.ylabel(r'$y$')
    plt.legend(loc='best')
    #plt.ylim(0,10)

    plt.subplot(1, 2, 2)
    plt.title('What does this histogram tell us?')
    three_errors=np.vstack((error_l, error_p, error_e)).T
    plt.hist(three_errors, bins=20,
             label=['linear','parabollic','exponential'],
             color=['k','y','g'], alpha=0.5)
    plt.legend(loc='best')
    plt.grid(True)

    plt.show()
widgets.interact(looking_at_data,type_of_data=['linear','parabollic','exponential','sinusoidal','random','boston house-prices','diabetes'])
```

    interactive(children=(Dropdown(description='type_of_data', index=6, options=('linear', 'parabollic', 'exponent…

    <function __main__.looking_at_data(type_of_data='diabetes')>

## References
[Back to TOC](#toc)

### Numpy Least Squares
http://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.lstsq.html

### Numpy QR Factorization
http://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.qr.html

### Scikit Learn Datasets
https://scikit-learn.org/stable/modules/classes.html#module-sklearn.datasets

### Colorama
https://pypi.org/project/colorama/

# Acknowledgements
[Back to TOC](#toc)
* _Material created by professor Claudio Torres_ (`ctorres@inf.utfsm.cl`) _and assistans: Laura Bermeo, Alvaro Salinas, Axel Símonsen and Martín Villanueva. DI UTFSM. April 2016._
* _Material updated by professor Claudio Torres_ (`ctorres@inf.utfsm.cl`) DI UTFSM. June 2017.
* _Material updated by professor Claudio Torres_ (`ctorres@inf.utfsm.cl`) DI UTFSM. July 2019.
* _Material updated by professor Claudio Torres_ (`ctorres@inf.utfsm.cl`) DI UTFSM. August 2019.
* _Update July 2020 - v1.27 - C.Torres_ : Fixing formatting issues.
* _Update June 2021 - v1.28 - C.Torres_ : Re-ordering notebook and adding TOC link.
* _Update June 2021 - v1.29 - C.Torres_ : Adding "initial example" in the section "Overdetermined Linear Systems of Equations".
* _Update June 2021 - v1.30 - C.Torres_ : Adding colors to the numerical example of QR factorization.
* _Update June 2021 - v1.31 - C.Torres_ : Adding more colors and an example to show that unitary vectors in double precision may not be truly unitary! And answers in the QR example.
* _Update June 2021 - v1.32 - C.Torres_ : Update of QR implementation, using 'm' and 'n', and including sub-index 'k' to match the 'LaTex' explanation and classnotes.
* _Update November 2021 - v1.33 - C.Torres_ : Updating text, adding a widget for unitary vectors.
* _Update November 2021 - v1.34 - C.Torres_ : Updating TOC.

```python

```