# Demo: Classification of XOR

Here we present a concrete example showing how to use the subroutines of QClassify. We consider binary classification of points on 2D plane using a 2-qubit quantum circuit. In particular the data points are distributed in an XOR-like fashion:

Here we encode a 2-dimensional vector $(\theta_1, \theta_2)$ into a 2-qubit state. For simplicity we set up the encoding such that the preprocessing step is trivial, namely each element of the data vector directly becomes a rotation angle on the circuit. A non-trivial classical preprocessing step may be found in for example [this paper](https://arxiv.org/abs/1806.08321). The circuit encoding is then $E(\boldsymbol\theta)=R_x(\theta_1)\otimes R_x(\theta_2)$. We then use a parametrized circuit $U({\bf w})$ to perform further classification.

The complete variational circuit is the following:

The encoding circuit serves as a feature map which maps a data point $(\theta_1,\theta_2)$ into a 2-qubit state. The second part is a processing circuit, which is a variational circuit with two parameters $w_1$ and $w_2$. The postprocessing steps consist of collecting the measurement statistics of the top qubit in the $Z$ basis and gleaning the probability $p$ of measuring $|1\rangle$ in the top qubit. The objective of training is to find a parameter setting ${\bf w}$ which minimizes the cross entropy between the circuit outputs on the training data and the training labels:

$$
\begin{equation}
\sum_i-\ell_i\log(p_i)-(1-\ell_i)\log(1-p_i)
\end{equation}
$$

where the sum is over the training set, $\ell_i\in\{0,1\}$ is the training label of the $i$-th data point and $p_i$ is the probability that the top qubit measures $|1\rangle$ for the $i$-th training point.

Now we describe step by step how to this up in QClassify. Before we start, note that the aim of this framework is to maximize customization. Therefore for a specific classification task the user can specify her own subroutines that suit the particular problem setting that is being considered.

First, we need to specify how classical data vectors are to be encoded in a quantum state. This is done in two parts:

1. Preprocess the classical data vector into circuit parameters. This is specified in `preprocessing.py`. For the purpose of this example we use the identity function `id_func` which simply returns the input.

2. For a given set of parameters $\bf w$, build an $n$-qubit circuit $E({\bf w})$ which prepares the state $E({\bf w})|0^n\rangle$. This is specified in `encoding_circ.py`. For the specific example considered here we have built the `x_product` function to realize the encoding circuit $E(\boldsymbol\theta)=R_x(\theta_1)\otimes\cdots\otimes R_x(\theta_n)$.

Having built these two functions, we specify the settings for encoding circuit in a dictionary as below.

```python
import qclassify

from qclassify.preprocessing import *
from qclassify.encoding_circ import *

qencoder_options={
                'preprocessing':id_func,        # see preprocessing.py
                'encoding_circ':x_product,      # see encoding_circ.py
        }
```

We then move on to describe the processing of the encoded quantum state for classification. This is also a two-part effort:

1. Variational circuit $U({\bf w})$ for transforming the encoded state into a final state $|\boldsymbol\theta,{\bf w}\rangle$. This is specified in `proc_circ.py`. For the particular variational circuit used in this example we have built the function `layer_xz`. There is a specific template that a function should follow. The template is described in the doc string at the beginning of `proc_circ.py`.
2. Extracting information from the output state $|\boldsymbol\theta,{\bf w}\rangle$. This is specified in `postprocessing.py`. Here we have the opportunity to specify the measurements that are made to the output state, as well as any classical postprocessing functions that turn the measurement statistics into a class label. For this particular we use `measure_top` for specifying measurement on the top qubit and `prob_one` for computing the probability of measuring $|1\rangle$ as the classifier output.

Having constructed these functions, we specify the settings for processing of the encoded state in a dictionary below. Here under `'postprocessing'` tab the `'quantum'` entry is for measurements and `'classical'` is for classical postprocessing of measurement statistics.

```python
from qclassify.proc_circ import *
from qclassify.postprocessing import *

qproc_options={
                'proc_circ':layer_xz,   # see proc_circ.py
                'postprocessing':{
                        'quantum':measure_top, # see postprocessing.py
                        'classical':prob_one, # see postprocessing.py
                }
        }
```

With all the components of the classifier specified we can now create a `QClassify` object which encapsulates all the information in the components.

```python
from qclassify.qclassifier import QClassifier

qubits_chosen = [0, 1] # indices of the qubits that the circuit will act on
qclassifier_options = {
                'encoder_options':qencoder_options,
                'proc_options':qproc_options,
        }
qc = QClassifier(qubits_chosen, qclassifier_options)
```

So far the `QClassifier` contains only abstract information about the classifier. In order to obtain a concrete circuit, let's plug in concrete assignments of the input vector $\boldsymbol\theta$ and parameters ${\bf w}$.

```python
input_vec = [1, 1] # input vector
params = [3.0672044712460114,\
          3.3311348339721203] # parameters for the circuit

qc_prog = qc.circuit(input_vec, params) # assign the input vector and parameters into the circuit
print(qc_prog)
```

    RX(1) 0
    RX(1) 1
    CZ 0 1
    RX(3.0672044712460114) 0
    RX(3.3311348339721203) 1
    DECLARE ro BIT[1]
    MEASURE 0 ro[0]

The `circuit` method of the `QClassifier` object not only returns a concrete circuit with input vectors and parameters assignments, but also stores the circuit information internally. This allows one to execute the quantum classifier and return a particular classification output.

```python
prob = qc.execute()
print(prob)
```

    0.7858

In addition if we would like to show the decision boundary of the classifier for a particular pair of features in the input vector we can use the `plot_decision_boundary` function.

```python
from math import pi

input_vec = [0,0]   # Input vector. Here [0,0] is used only as a placeholder since both features are chosen to be
                    # swept over.
features_chosen = [0,1] # Indices of features that are chosen. Here there are only two features so naturally we let it
                    # be [0,1].
filename = 'qclassifier_db.pdf' # name of the plot to be saved

plot_db_options = {
                'nmesh':10,      # number of grid points
                'xmin':-pi,     # min. limit on the x axis
                'xmax':pi,      # max. limit on the x axis
                'ymin':-pi/2,   # min. limit on the y axis
                'ymax':3*pi/2   # max. limit on the y axis
        }

qc.plot_decision_boundary(input_vec, features_chosen, filename, plot_db_options)
```

From the above decision boundary plot we can see that apparently the classifier is not doing a great job in separating the data points. Therefore we need to tune the parameters to improve it. The standard training objective function is the cross entropy (see `crossentropy` in `training.py`). The training data needs to be provided as a Python list of tuples `(features, label)` where `features` is a list of floating point numbers and `label` is either 0 or 1.

```python
from qclassify.xor_example import *
from qclassify.training import *

train_options={
                'training_data':XOR_TRAINING_DATA,      # Example. See xor_example.py
                'objective_func':crossentropy,  # See training.py
                'training_method':'nelder-mead', # Optimization algorithm
                'init_params':[3.0672044712460114, 3.3311348339721203], # initial guess for parameters
                'maxiter':10, # maximum number of iterations
                'xatol':1e-3, # tolerance in solution
                'fatol':1e-3, # tolerance in function value
                'verbose':True # Print intermediate values
        }

qc.train()
```

    Iter   Obj
       1   0.018
       2   0.018
       3   0.019
       4   0.014
       5   0.014
       6   0.013
       7   0.013
       8   0.013
       9   0.014
      10   0.013
      11   0.013
      12   0.014
      13   0.013
      14   0.014
      15   0.013
      16   0.013
      17   0.014
      18   0.013
      19   0.014

The `min_loss_history` attribute of `QClassifier` class stores the value of the objective function each time it is called during the optimization. This may be used for further processing (for example plotting the training curve).

```python
print(qc.min_loss_history)
```

    [0.01785220234453517, 0.01848265965455959, 0.01860949110296039, 0.013617471648894883, 0.013810264525585902, 0.013235283510689566, 0.01337114296678497, 0.01328582257290195, 0.01385394241480525, 0.012893824496953976, 0.01346527546182597, 0.013501980887877024, 0.012937323886782048, 0.013930548172307659, 0.01339673874455748, 0.01258386751929006, 0.013696623567530642, 0.012857984171322261, 0.013550070439161298]

After training we again plot the decision boundary. It seems that the classifier is able to perform much better on the training data.

```python
qc.plot_decision_boundary([0,0], [0,1], 'qclassifier_db_train.pdf', plot_db_options)
```

We may also test the classifier on previously unseen data points to see how well it has performed. For this specific example we use `gen_xor` for generating random data points for testing. But in general this can come from a different source, as long as the format of the data set follows the same structure as described previously for training data: A list of tuples `(feature, label)` where `feature` is a list of floats and `label` is a discrete value in $\{0,1\}$. The `test` method in the class `QClassifier` allows one to test the performance of the classifier on a given test data set.

```python
from qclassify.xor_example import *
from qclassify.training import *

test_data = gen_xor(100, pi/8.) # generate more XOR-like testing data. Here we use 100 data points. The second value
                                # specifies how spread out the testing data are. See xor_example.py for details.
test_options = {
                'objective_func': crossentropy, # Objective function used for evaluating the classifier performance
        }
print(qc.test(test_data), test_options) # print the objective function value over the testing data
```

    0.01589438299249235 {'objective_func': <function crossentropy at 0x11df2f730>}

## Thoughts and suggestions?

Of course this repo is by no means complete. There are many improvements and additions that can be considered. If you have more thoughts and suggestions please feel free to contact Yudong Cao at [yudong@zapatacomputing.com](yudong@zapatacomputing.com).


# Non-Associative Commutative Algebras for Dynamical Systems

We will consider quadratic dynamical systems, and go to general quasipolynomial differential equations of the form $$\dot{x_i} = x_i\sum_{j=1}^mA_{ij}\prod_{k=1}^nx_k^{B_{jk}}$$

In one dimension we have $$\dot{x}=ax^2+bx+c$$ or in the notation above $$\dot{x}=x(ax+b+c/x)=x\sum_{j=1}^3A_{j}x^{B_j}.$$  With $A_1=a$, $A_2=b$, $A_3=c$ and $B_1=1$, $B_2=0$, $B_3=-1$.

```python
import numpy as np
import sympy as sp
```

```python
a1,a2,a3=sp.Symbol("a1"),sp.Symbol("a2"),sp.Symbol("a3")
b1,b2,b3=sp.Symbol("b1"),sp.Symbol("b2"),sp.Symbol("b3")
U1,U2,U3=sp.Symbol("U1"),sp.Symbol("U2"),sp.Symbol("U3")
A=np.matrix([a1,a2,a3])
B=np.matrix([1,0,-1])
U=np.matrix([U1,U2,U3])
```

```python
M=np.outer(B,A)
```

```python
print M
```

    [[a1 a2 a3]
     [0 0 0]
     [-a1 -a2 -a3]]

```python
U0=np.array([7.,1.,1./7])
```

```python
np.multiply(np.matrix(M),np.matrix(U0).transpose())
```

    matrix([[7.0*a1, 7.0*a2, 7.0*a3],
            [0, 0, 0],
            [-0.142857142857143*a1, -0.142857142857143*a2, -0.142857142857143*a3]], dtype=object)

```python
np.matrix(U0)*np.matrix(M)*np.matrix(U0).transpose()
```

    matrix([[48.0*a1 + 6.85714285714286*a2 + 0.979591836734694*a3]], dtype=object)

```python
U*M*U.transpose()
```

    matrix([[U1*(U1*a1 - U3*a1) + U2*(U1*a2 - U3*a2) + U3*(U1*a3 - U3*a3)]], dtype=object)

```python

```


# Solving Linear Systems: Iterative Methods

This notebook by Xiaozhou Li is licensed under a Creative Commons Attribution 4.0 International License.
All code examples are also licensed under the [MIT license](http://opensource.org/licenses/MIT).

## General Form
For solving the linear system
$$
    Ax = b,
$$
with the exact solution $x^{*}$.  The general form based on the fixed point interation:
\begin{equation}
     \begin{split}
          x^{(0)} & = \text{initial guess} \\
          x^{(k+1)} & = g(x^{(k)}) \quad k = 0,1,2,\ldots,
     \end{split}
\end{equation}
where
$$
    g(x) = x - C(Ax - b).
$$
Difficult: find a matrix $C$ such that
$$
    \lim\limits_{k\rightarrow\infty}x^{(k)} = x^{*}
$$
and the algorithm needs to be converge fast and economy.

**Example 1**
\begin{equation*}
    A = \left[\begin{array}{ccc} 9& -1 & -1 \\ -1 & 10 & -1 \\ -1 & -1& 15\end{array}\right],\quad b = \left[\begin{array}{c} 7 \\ 8 \\ 13\end{array}\right],
\end{equation*}
has the exact solution $x^{*} = {[1, 1, 1]}^T$

```python
import numpy as np
%matplotlib inline
%config InlineBackend.figure_format = 'retina'
import matplotlib.pyplot as plt
from ipywidgets import interact, interactive, fixed, interact_manual
import ipywidgets as widgets
from IPython.display import clear_output, display
```

```python
def IterC(A, b, C, x0, x_star, iters):
    x = np.copy(x0)
    print ('Iteration No.     Numerical Solution     Max norm error ')
    print (0, x, np.linalg.norm(x_star-x, np.inf))
    for i in range(iters):
        x = x + np.dot(C, b - np.dot(A,x))
        print (i+1, x, np.linalg.norm(x_star-x,np.inf))
```

```python
A = np.array([[9., -1., -1.],[-1.,10.,-1.],[-1.,-1.,15.]])
b = np.array([7.,8.,13.])
```

**Naive Choice**

Choosing $C = I$, then
$$g(x) = x - (Ax - b),$$
and the fixed-point iteration
$$x^{(k+1)}  = (I - A)x^{(k)} + b \quad k = 0,1,2,\ldots. $$
Let the intial guess $x_0 = [0, 0, 0]^T$.

```python
C = np.eye(3)
x0 = np.zeros(3)
x_star = np.array([1.,1.,1.])

w = interactive(IterC, A=fixed(A), b=fixed(b), C=fixed(C), x0=fixed(x0), x_star=fixed(x_star), iters=widgets.IntSlider(min=0,max=20,value=0))
display(w)
```

    interactive(children=(IntSlider(value=0, description='iters', max=20), Output()), _dom_classes=('widget-intera…

**Best Choice (theoretically)**

Choosing $C = A^{-1}$, then
$$g(x) = x - A^{-1}(Ax - b),$$
and the fixed-point iteration
$$x^{(k+1)}  = A^{-1}b \quad k = 0,1,2,\ldots. $$

* It equals to solve $Ax = b$ directly.
* However, it gives a hint that $C$ should be close to $A^{-1}$

**First Approach**

Let $D$ denote the main diagonal of $A$, $L$ denote the lower triangle of $A$ (entries below the main diagonal), and $U$ denote the upper triangle (entries above the main diagonal). Then $A = L + D + U$

Choosing $C = \text{diag}(A)^{-1} = D^{-1}$, then
$$g(x) = x - D^{-1}(Ax - b),$$
and the fixed-point iteration
$$Dx^{(k+1)}  = (L + U)x^{(k)} + b \quad k = 0,1,2,\ldots. $$

```python
C = np.diag(1./np.diag(A))
x0 = np.zeros(np.size(b))
#x0 = np.array([0,1.,0])
x_star = np.array([1.,1.,1.])
#IterC(A, b, C, x0, x_star, 10)
```

```python
w = interactive(IterC, A=fixed(A), b=fixed(b), C=fixed(C), x0=fixed(x0), x_star=fixed(x_star), iters=widgets.IntSlider(min=0,max=20,value=0))
display(w)
```

    interactive(children=(IntSlider(value=0, description='iters', max=20), Output()), _dom_classes=('widget-intera…

## Jacobi Method
### Matrix Form:
$$
    x^{(k+1)} = x^{(k)} - D^{-1}(Ax^{(k)} - b)
$$
or
$$
    Dx^{(k+1)} = b - (L+U)x^{(k)}
$$
### Algorithm
$$
    x^{(k+1)}_i = \frac{b_i - \sum\limits_{j  i}a_{ij}x^{(k)}_j}{a_{ii}}
$$

```python
def Jacobi(A, b, x0, x_star, iters):
    x_old = np.copy(x0)
    x_new = np.zeros(np.size(x0))
    print (0, x_old, np.linalg.norm(x_star-x_old,np.inf))
    for k in range(iters):
        for i in range(np.size(x0)):
            x_new[i] = (b[i] - np.dot(A[i,:i],x_old[:i]) - np.dot(A[i,i+1:],x_old[i+1:]))/A[i,i]
        print (k+1, x_new, np.linalg.norm(x_star-x_new,np.inf))
        x_old = np.copy(x_new)
```

```python
w = interactive(Jacobi, A=fixed(A), b=fixed(b), x0=fixed(x0), x_star=fixed(x_star), iters=widgets.IntSlider(min=0,max=20,value=0))
display(w)
```

    interactive(children=(IntSlider(value=0, description='iters', max=20), Output()), _dom_classes=('widget-intera…

**Second Approach**

Let $D$ denote the main diagonal of $A$, $L$ denote the lower triangle of $A$ (entries below the main diagonal), and $U$ denote the upper triangle (entries above the main diagonal). Then $A = L + D + U$

Choosing $C = (L + D)^{-1}$, then
$$g(x) = x - (L + D)^{-1}(Ax - b),$$
and the fixed-point iteration
$$(L + D)x^{(k+1)}  = Ux^{(k)} + b \quad k = 0,1,2,\ldots. $$

```python
def GS(A, b, x0, x_star, iters):
    x = np.copy(x0)
    print (0, x, np.linalg.norm(x_star-x,np.inf))
    for k in range(iters):
        for i in range(np.size(x0)):
            x[i] = (b[i] - np.dot(A[i,:i],x[:i]) - np.dot(A[i,i+1:],x[i+1:]))/A[i,i]
        print (k+1, x, np.linalg.norm(x_star-x,np.inf))
```

```python
w = interactive(GS, A=fixed(A), b=fixed(b), x0=fixed(x0), x_star=fixed(x_star), iters=widgets.IntSlider(min=0,max=20,value=0))
display(w)
```

    interactive(children=(IntSlider(value=0, description='iters', max=20), Output()), _dom_classes=('widget-intera…

## Gauss-Seidel Method
### Algorithm
$$
    x^{(k+1)}_i = \frac{b_i - \sum\limits_{j  i}a_{ij}x^{(k)}_j}{a_{ii}}
$$
### Matrix Form:
$$
    x^{(k+1)} = x^{(k)} - (L+D)^{-1}(Ax^{(k)} - b)
$$
or
$$
    (L+D)x^{(k+1)} = b - Ux^{(k)}
$$

**Example 2**
\begin{equation*}
    A = \left[\begin{array}{ccc} 3& 1 & -1 \\ 2 & 4 & 1 \\ -1 & 2& 5\end{array}\right],\quad b = \left[\begin{array}{c} 4 \\ 1 \\ 1\end{array}\right],
\end{equation*}
has the exact solution $x^{*} = {[2, -1, 1]}^T$

```python
A = np.array([[3, 1, -1],[2,4,1],[-1,2,5]])
b = np.array([4,1,1])

x0 = np.zeros(np.size(b))
x_star = np.array([2.,-1.,1.])
```

```python
w = interactive(GS, A=fixed(A), b=fixed(b), x0=fixed(x0), x_star=fixed(x_star), iters=widgets.IntSlider(min=0,max=40,value=0))
display(w)
```

    interactive(children=(IntSlider(value=0, description='iters', max=40), Output()), _dom_classes=('widget-intera…

**Example 3**
\begin{equation*}
    A = \left[\begin{array}{ccc} 1& 2 & -2 \\ 1 & 1 & 1 \\ 2 & 2& 1\end{array}\right],\quad b = \left[\begin{array}{c} 7 \\ 8 \\ 13\end{array}\right],
\end{equation*}
has the exact solution $x^{*} = {[-3, 8, 3]}^T$

```python
A = np.array([[1, 2, -2],[1,1,1],[2,2,1]])
b = np.array([7,8,13])

#x0 = np.zeros(np.size(b))
x0 = np.array([-1,1,1])
x_star = np.array([-3.,8.,3.])

w = interactive(GS, A=fixed(A), b=fixed(b), x0=fixed(x0), x_star=fixed(x_star), iters=widgets.IntSlider(min=0,max=20,value=0))
display(w)
```

    interactive(children=(IntSlider(value=0, description='iters', max=20), Output()), _dom_classes=('widget-intera…

```python
B = np.eye(3) - np.dot(np.diag(1./np.diag(A)),A)
print(B)
print (np.linalg.eig(B))
```

    [[ 0. -2.  2.]
     [-1.  0. -1.]
     [-2. -2.  0.]]
    (array([-6.16610918e-06+1.06799587e-05j, -6.16610918e-06-1.06799587e-05j,
            1.23322184e-05+0.00000000e+00j]), array([[-0.57734849-6.16605800e-06j, -0.57734849+6.16605800e-06j,
             0.57735383+0.00000000e+00j],
           [ 0.57735027+3.08300999e-06j,  0.57735027-3.08300999e-06j,
            -0.57735027+0.00000000e+00j],
           [ 0.57735205+0.00000000e+00j,  0.57735205-0.00000000e+00j,
            -0.57734671+0.00000000e+00j]]))

**Example 4**
\begin{equation*}
    A = \left[\begin{array}{cc} 1& 2 \\ 3 & 1 \end{array}\right],\quad b = \left[\begin{array}{c} 5 \\ 5\end{array}\right],
\end{equation*}
has the exact solution $x^{*} = {[1, 2]}^T$

or
\begin{equation*}
    A = \left[\begin{array}{cc} 3& 1 \\ 1 & 2 \end{array}\right],\quad b = \left[\begin{array}{c} 5 \\ 5\end{array}\right],
\end{equation*}

```python
#A = np.array([[1, 2],[3,1]])
A = np.array([[3, 1],[1,2]])
b = np.array([5,5])

#x0 = np.zeros(np.size(b))
x0 = np.array([0,0])
x_star = np.array([1.,2.,])

w = interactive(GS, A=fixed(A), b=fixed(b), x0=fixed(x0), x_star=fixed(x_star), iters=widgets.IntSlider(min=0,max=20,value=0))
display(w)
```

    interactive(children=(IntSlider(value=0, description='iters', max=20), Output()), _dom_classes=('widget-intera…

**Example 5**
Are Jacobi iteration and Gauss-Seidel iteration convergent for the following equations?
\begin{equation*}
    A_1 = \left[\begin{array}{ccc} 3& 0 & 4 \\ 7 & 4 & 2 \\ -1 & 1 & 2\end{array}\right],\quad A_2 = \left[\begin{array}{ccc} -3& 3 & -6 \\ -4 & 7 & -8 \\ 5 & 7 & -9\end{array}\right],
\end{equation*}

* Consider the **spectral radius** of the iterative matrix
* $B_J = -D^{-1}(L+U)$ and $B_{GS} = -(L+D)^{-1}U$

```python
def Is_Jacobi_Gauss(A):
    L = np.tril(A,-1)
    U = np.triu(A,1)
    D = np.diag(np.diag(A))

    B_J = np.dot(np.diag(1./np.diag(A)), L+U)
    B_GS = np.dot(np.linalg.inv(L+D),U)

    rho_J = np.linalg.norm(np.linalg.eigvals(B_J), np.inf)
    rho_GS = np.linalg.norm(np.linalg.eigvals(B_GS), np.inf)

    print ("Spectral Radius")
    print ("Jacobi: ", rho_J)
    print ("Gauss Sediel: ", rho_GS)

A1 = np.array([[3, 0, 4],[7, 4, 2], [-1,1,2]])
A2 = np.array([[-3, 3, -6], [-4, 7, -8], [5, 7, -9]])
Is_Jacobi_Gauss(A2)
```

    Spectral Radius
    Jacobi:  0.8133091054692768
    Gauss Sediel:  1.1111111111111105

## Successive Over-Relaxation (SOR)
### Algorithm
$$
    x^{(k+1)}_i = x^{(k)} + \omega \frac{b_i - \sum\limits_{j < i}a_{ij}x^{(k+1)}_j - \sum\limits_{j \geq i}a_{ij}x^{(k)}_j}{a_{ii}}
$$
### Matrix Form:
$$
    x^{(k+1)} = x^{(k)} - \omega(\omega L+D)^{-1}(Ax^{(k)} - b)
$$
or
$$
    (\omega L+D)x^{(k+1)} = ((1-\omega)D - \omega U)x^{(k)} + \omega b
$$

```python
def SOR(A, b, x0, x_star, omega, iters):
    x = np.copy(x0)
    print (0, x, np.linalg.norm(x_star-x,np.inf))
    for k in range(iters):
        for i in range(np.size(x0)):
            x[i] = x[i] + omega*(b[i] - np.dot(A[i,:i],x[:i]) - np.dot(A[i,i:],x[i:]))/A[i,i]
        print (k+1, x, np.linalg.norm(x_star-x,np.inf))
```

```python
def SOR2(A, b, x0, x_star, omega, iters):
    x = np.copy(x0)
    for k in range(iters):
        for i in range(np.size(x0)):
            x[i] = x[i] + omega*(b[i] - np.dot(A[i,:i],x[:i]) - np.dot(A[i,i:],x[i:]))/A[i,i]
    return (np.linalg.norm(x_star-x,np.inf))
```

```python
def SOR3(A, b, x0, x_star, omega, iters):
    x = np.copy(x0)
    print (0, np.linalg.norm(x_star-x,np.inf))
    for k in range(iters):
        for i in range(np.size(x0)):
            x[i] = x[i] + omega*(b[i] - np.dot(A[i,:i],x[:i]) - np.dot(A[i,i:],x[i:]))/A[i,i]
        print (k+1, np.linalg.norm(x_star-x,np.inf))
```

```python
A = np.array([[9., -1., -1.],[-1.,10.,-1.],[-1.,-1.,15.]])
b = np.array([7.,8.,13.])
```

```python
x0 = np.array([0.,0.,0.])
x_star = np.array([1.,1.,1.])
omega = 1.01

w = interactive(SOR, A=fixed(A), b=fixed(b), x0=fixed(x0), x_star=fixed(x_star), omega=fixed(omega), iters=widgets.IntSlider(min=0,max=20,value=0))
display(w)
```

    interactive(children=(IntSlider(value=0, description='iters', max=20), Output()), _dom_classes=('widget-intera…

```python
w = interactive(GS, A=fixed(A), b=fixed(b), x0=fixed(x0), x_star=fixed(x_star), iters=widgets.IntSlider(min=0,max=20,value=0))
display(w)
```

    interactive(children=(IntSlider(value=0, description='iters', max=20), Output()), _dom_classes=('widget-intera…

**Example 6**
\begin{equation*}
    A = \left[\begin{array}{ccc} 2& -1 & 0 \\ -1 & 2 & -1 \\ 0 & -1& 2\end{array}\right],\quad b = \left[\begin{array}{c} 1 \\ 0 \\ 1.8\end{array}\right],
\end{equation*}
has the exact solution $x^{*} = {[1.2, 1.4, 1.6]}^T$

```python
A = np.array([[2, -1, 0],[-1, 2, -1], [0, -1, 2]])
b = np.array([1., 0, 1.8])

x0 = np.array([1.,1.,1.])
x_star = np.array([1.2,1.4,1.6])
omega = 1.2

w = interactive(SOR, A=fixed(A), b=fixed(b), x0=fixed(x0), x_star=fixed(x_star), omega=fixed(omega), iters=widgets.IntSlider(min=0,max=20,value=0))
display(w)
```

    interactive(children=(IntSlider(value=0, description='iters', max=20), Output()), _dom_classes=('widget-intera…

```python
w = interactive(GS, A=fixed(A), b=fixed(b), x0=fixed(x0), x_star=fixed(x_star), iters=widgets.IntSlider(min=0,max=20,value=0))
display(w)
```

    interactive(children=(IntSlider(value=0, description='iters', max=20), Output()), _dom_classes=('widget-intera…

```python
num = 21
omega = np.linspace(0.8, 1.8, num)
err1 = np.zeros(num)
for i in range(num):
    err1[i] = SOR2(A, b, x0, x_star, omega[i], 10)
print (err1)
plt.plot(omega, np.log10(err1), 'o')
```

**Example 7**
\begin{equation*}
    A = \left[\begin{array}{cccc} -4& 1 & 1 & 1 \\ 1 & -4 & 1 & 1 \\ 1 & 1& -4 &1 \\ 1 & 1 &1 & -4\end{array}\right],\quad b = \left[\begin{array}{c} 1 \\ 1 \\ 1 \\ 1\end{array}\right],
\end{equation*}
has the exact solution $x^{*} = {[-1, -1, -1, -1]}^T$

```python
A = np.array([[-4, 1, 1, 1],[1, -4, 1, 1], [1, 1, -4, 1], [1, 1, 1, -4]])
b = np.array([1, 1, 1, 1])

x0 = np.zeros(np.size(b))
x_star = np.array([-1,-1,-1,-1])
omega = 1.25

w = interactive(SOR, A=fixed(A), b=fixed(b), x0=fixed(x0), x_star=fixed(x_star), omega=fixed(omega), iters=widgets.IntSlider(min=0,max=20,value=0))
display(w)
```

    interactive(children=(IntSlider(value=0, description='iters', max=20), Output()), _dom_classes=('widget-intera…

```python
w = interactive(GS, A=fixed(A), b=fixed(b), x0=fixed(x0), x_star=fixed(x_star), iters=widgets.IntSlider(min=0,max=100,value=0))
display(w)
```

    interactive(children=(IntSlider(value=0, description='iters'), Output()), _dom_classes=('widget-interact',))

```python
num = 21
omega = np.linspace(0.8, 1.8, num)
err1 = np.zeros(num)
for i in range(num):
    err1[i] = SOR2(A, b, x0, x_star, omega[i], 10)
print (err1)
plt.plot(omega, np.log10(err1), 'o')
```

**Example 8**
\begin{equation*}
    A=\begin{pmatrix}{3} & {-1} & {0} & 0 & 0 & \frac{1}{2}  \\ {-1} & {3} & {-1} & {0} & \frac{1}{2} & 0\\ {0} & {-1} & {3} & {-1} & {0} & 0 \\ 0& {0} & {-1} & {3} & {-1} & {0} \\ {0} & \frac{1}{2} & {0} & {-1} & {3} & {-1} \\ \frac{1}{2} & {0} & 0 & 0 & {-1} & {3}\end{pmatrix},\,\,b=\begin{pmatrix}\frac{5}{2} \\ \frac{3}{2} \\ 1 \\ 1 \\ \frac{3}{2}    \\ \frac{5}{2}  \end{pmatrix}
\end{equation*}
has the exact solution $x^{*} = {[1, 1, 1, 1, 1, 1]}^T$

```python
n0 = 6
A = 3*np.eye(n0) - np.diag(np.ones(n0-1),-1) - np.diag(np.ones(n0-1),+1)
for i in range(n0):
    if (abs(n0-1 - 2*i) > 1):
        A[i, n0-1-i] = - 1/2
print (A)
x_star = np.ones(n0)
b = np.dot(A, x_star)

x0 = np.zeros(np.size(b))
omega = 1.25

w = interactive(SOR, A=fixed(A), b=fixed(b), x0=fixed(x0), x_star=fixed(x_star), omega=fixed(omega), iters=widgets.IntSlider(min=0,max=20,value=0))
display(w)
```

    [[ 3.  -1.   0.   0.   0.  -0.5]
     [-1.   3.  -1.   0.  -0.5  0. ]
     [ 0.  -1.   3.  -1.   0.   0. ]
     [ 0.   0.  -1.   3.  -1.   0. ]
     [ 0.  -0.5  0.  -1.   3.  -1. ]
     [-0.5  0.   0.   0.  -1.   3. ]]

    interactive(children=(IntSlider(value=0, description='iters', max=20), Output()), _dom_classes=('widget-intera…

```python
num = 21
omega = np.linspace(0.8, 1.8, num)
err1 = np.zeros(num)
for i in range(num):
    err1[i] = SOR2(A, b, x0, x_star, omega[i], 10)
print (err1)
plt.plot(omega, np.log10(err1), 'o')
```

```python
w = interactive(Jacobi, A=fixed(A), b=fixed(b), x0=fixed(x0), x_star=fixed(x_star), iters=widgets.IntSlider(min=0,max=100,value=0))
display(w)
```

    interactive(children=(IntSlider(value=0, description='iters'), Output()), _dom_classes=('widget-interact',))

## Sparse Matrix Computations
A coefficient matrix is called sparse if many of the matrix entries are known to be zero. Often, of the $n^2$ eligible entries in a sparse matrix, only $\mathcal{O}(n)$ of them are nonzero. A full matrix is the opposite, where few entries may be assumed to be zero.

**Example 9**
Consider the $n$-equation version of
\begin{equation*}
    A=\begin{pmatrix}{3} & {-1} & {0} & 0 & 0 & \frac{1}{2}  \\ {-1} & {3} & {-1} & {0} & \frac{1}{2} & 0\\ {0} & {-1} & {3} & {-1} & {0} & 0 \\ 0& {0} & {-1} & {3} & {-1} & {0} \\ {0} & \frac{1}{2} & {0} & {-1} & {3} & {-1} \\ \frac{1}{2} & {0} & 0 & 0 & {-1} & {3}\end{pmatrix},
\end{equation*}
has the exact solution $x^{*} = {[1, 1,\ldots, 1]}^T$ and $b = A x^{*}$

* First, let us have a look about the matrix $A$

```python
n0 = 10000
A = 3*np.eye(n0) - np.diag(np.ones(n0-1),-1) - np.diag(np.ones(n0-1),+1)
for i in range(n0):
    if (abs(n0-1 - 2*i) > 1):
        A[i, n0-1-i] = - 1/2
#plt.spy(A)
#plt.show()
```

* How about the $PA = LU$ for the above matrix $A$?
* Are the $L$ and $U$ matrices still sparse?

```python
import scipy.linalg
#P, L, U = scipy.linalg.lu(A)

#plt.spy(L)
#plt.show()
```

Gaussian elimination applied to a sparse matrix usually causes **fill-in**, where the coefficient matrix changes from sparse to full due to the necessary row operations. For this reason, the efficiency of Gaussian elimination and its $PA = LU$ implementation become questionable for sparse matrices, leaving iterative methods as a feasible alternative.

* Let us solve it with SOR method

```python
x_star = np.ones(n0)
b = np.dot(A, x_star)

x0 = np.zeros(np.size(b))
omega = 1.25
w = interactive(SOR3, A=fixed(A), b=fixed(b), x0=fixed(x0), x_star=fixed(x_star), omega=fixed(omega), iters=widgets.IntSlider(min=0,max=200,value=0, step=10))
display(w)
```

    interactive(children=(IntSlider(value=0, description='iters', max=200, step=10), Output()), _dom_classes=('wid…

## Application for Solving Laplace's Equation

### Laplace's equation
Consider the Laplace's equation given as
$$
    \nabla^2 u = 0,\quad\quad (x,y) \in D,
$$
where $\nabla^2 = \frac{\partial^2}{\partial x^2} + \frac{\partial^2}{\partial y^2}$, and the boundary conditions are given as

### Finite Difference Approximation
Here, we use a rectangular grid $(x_i,y_j)$, where
$$
    x_i = i\Delta x, \,\,\text{for }\, i = 0,1,\ldots,N+1;\quad y_j = j\Delta y,\,\,\text{for }\, j = 0,1,\ldots,M+1.
$$
Five-points scheme:
$$
    -\lambda^2 u_{i+1,j} + 2(1+\lambda^2)u_{i,j} - \lambda^2u_{i-1,j} - u_{i,j+1} - u_{i,j-1} = 0,\quad\text{for}\,\, i = 1,\ldots,N,\,\, j = 1,\ldots,M，
$$
where $\lambda = \frac{\Delta y}{\Delta x}$. The boundary conditions are
- $x = 0: u_{0,j} = g_L(y_j), \quad\text{for }\, j = 1,\ldots,M$,
- $x = a: u_{N+1,j} = g_R(y_j), \quad\text{for }\, j = 1,\ldots,M$,
- $y = 0: u_{i,0} = g_B(x_i), \quad\text{for }\, i = 1,\ldots,N$,
- $y = b: u_{i,M+1} = g_T(x_i), \quad\text{for }\, i = 1,\ldots,N$.

```python
def generate_TD(N, dx, dy):
    T = np.zeros([N,N])
    a = - (dy/dx)**2
    b = 2*(1 - a)
    for i in range(N):
        T[i,i] += b
        if (i < N-1):
            T[i,i+1] += a
        if (i > 0):
            T[i,i-1] += a
    D = -np.identity(N)
    return T, D

def assemble_matrix_A(dx, dy, N, M):
    T, D = generate_TD(N, dx, dy)
    A = np.zeros([N*M, N*M])
    for j in range(M):
        A[j*N:(j+1)*N,j*N:(j+1)*N] += T
        if (j < M-1):
            A[j*N:(j+1)*N,(j+1)*N:(j+2)*N] += D
        if (j > 0):
            A[j*N:(j+1)*N,(j-1)*N:j*N] += D
    return A
```

```python
N = 4
M = 4
dx = 1./(N+1)
dy = 1./(M+1)
T, D = generate_TD(N, dx, dy)
#print (T)
A = assemble_matrix_A(dx, dy, N, M)
#print (A)
plt.spy(A)
plt.show()
```

```python
# Set boundary conditions
def gL(y):
    return 0.

def gR(y):
    return 0.

def gB(x):
    return 0.

def gT(x):
    return 1.
    #return x*(1-x)*(4./5-x)*np.exp(6*x)

def assemble_vector_b(x, y, dx, dy, N, M, gL, gR, gB, gT):
    b = np.zeros(N*M)
    # Left BCs
    for j in range(M):
        b[(j-1)*N] += (dy/dx)**2*gL(y[j+1])

    # Right BCs
    # b +=

    # Bottom BCs
    # b +=

    # Top BCs:
    for i in range(N):
        b[(M-1)*N+i] += gT(x[i+1])
    return b
```

```python
from mpl_toolkits import mplot3d
from mpl_toolkits.mplot3d import axes3d
```

```python
def Laplace_solver(a, b, N, M, gL, gR, gB, gT):
    dx = b/(M+1)
    dy = a/(N+1)
    x = np.linspace(0, a, N+2)
    y = np.linspace(0, b, M+2)

    A = assemble_matrix_A(dx, dy, N, M)
    b = assemble_vector_b(x, y, dx, dy, N, M, gL, gR, gB, gT)

    v = np.linalg.solve(A,b)

    # add boundary points + plotting
    u = np.zeros([(N+2),(M+2)])
    #u[1:(N+1),1:(M+1)] = np.reshape(v, (N, M))
    # Top BCs
    for i in range(N+2):
        u[i,M+1] = gT(x[i])
    u = np.transpose(u)
    u[1:(M+1),1:(N+1)] = np.reshape(v, (M, N))

    X, Y = np.meshgrid(x, y)
    #Z = np.sin(2*np.pi*X)*np.sin(2*np.pi*Y)

    fig = plt.figure()
    #ax = plt.axes(projection='3d')
    ax = fig.add_subplot(1, 1, 1, projection='3d')

    ax.plot_surface(X, Y, u, rstride=1, cstride=1,
                    cmap='viridis', edgecolor='none')
    ax.set_title('surface')
    plt.show()

Laplace_solver(1, 1, 40, 40, gL, gR, gB, gT)
```

```python
def Jacobi_tol(A, b, x0, tol):
    x_old = np.copy(x0)
    x_new = np.zeros(np.size(x0))
    for i in range(np.size(x0)):
        x_new[i] = (b[i] - np.dot(A[i,:i],x_old[:i]) - np.dot(A[i,i+1:],x_old[i+1:]))/A[i,i]
    iters = 1
    while ((np.linalg.norm(x_new-x_old,np.inf)) > tol):
        x_old = np.copy(x_new)
        for i in range(np.size(x0)):
            x_new[i] = (b[i] - np.dot(A[i,:i],x_old[:i]) - np.dot(A[i,i+1:],x_old[i+1:]))/A[i,i]
        iters += 1
    return x_new, iters
```

```python
def GS_tol(A, b, x0, tol):
    x_old = np.copy(x0)
    x = np.copy(x0)
    for i in range(np.size(x0)):
        x[i] = (b[i] - np.dot(A[i,:i],x[:i]) - np.dot(A[i,i+1:],x[i+1:]))/A[i,i]
    iters = 1
    while ((np.linalg.norm(x-x_old,np.inf)) > tol):
        x_old = np.copy(x)
        for i in range(np.size(x0)):
            x[i] = (b[i] - np.dot(A[i,:i],x[:i]) - np.dot(A[i,i+1:],x[i+1:]))/A[i,i]
        iters += 1
    return x, iters
```

```python
def SOR_tol(A, b, x0, omega, tol):
    x_old = np.copy(x0)
    x = np.copy(x0)
    for i in range(np.size(x0)):
            x[i] = x[i] + omega*(b[i] - np.dot(A[i,:i],x[:i]) - np.dot(A[i,i:],x[i:]))/A[i,i]
    iters = 1
    while ((np.linalg.norm(x-x_old,np.inf)) > tol):
        x_old = np.copy(x)
        for i in range(np.size(x0)):
            x[i] = x[i] + omega*(b[i] - np.dot(A[i,:i],x[:i]) - np.dot(A[i,i:],x[i:]))/A[i,i]
        iters += 1
    return x, iters
```

```python
def CG_tol(A, b, x0, x_star, tol):
    r_new = b - np.dot(A, x0)
    r_old = np.copy(np.size(x0))
    d_old = np.zeros(np.size(x0))
    x = np.copy(x0)
    iters = 0
    while ((np.linalg.norm(x-x_star,np.inf)) > tol):
        if (iters == 0):
            d_new = np.copy(r_new)
        else:
            beta = np.dot(r_new,r_new)/np.dot(r_old,r_old)
            d_new = r_new + beta*d_old
        Ad = np.dot(A, d_new)
        alpha = np.dot(r_new,r_new)/np.dot(d_new,Ad)
        x += alpha*d_new
        d_old = d_new
        r_old = r_new
        r_new = r_old - alpha*Ad
        iters += 1
    return x, iters
```

```python
def Iterative_solver(a, b, N, M, gL, gR, gB, gT, tol):
    dx = b/(M+1)
    dy = a/(N+1)
    x = np.linspace(0, a, N+2)
    y = np.linspace(0, b, M+2)

    A = assemble_matrix_A(dx, dy, N, M)
    b = assemble_vector_b(x, y, dx, dy, N, M, gL, gR, gB, gT)

    v = np.linalg.solve(A,b)
    #tol = 1.e-8
    v0 = np.zeros(np.size(b))
    #v_J, iters = Jacobi_tol(A, b, v0, tol)
    #print ("Jacobi Method:  %4d    %7.2e" %(iters, np.linalg.norm(v - v_J, np.inf)))

    #v_GS, iters = GS_tol(A, b, v0, tol)
    #print ("Gauss Seidel :  %4d    %7.2e" %(iters, np.linalg.norm(v - v_GS, np.inf)))

    omega = 2./(1 + np.sin(np.pi*dx))
    print ("omega = ", omega)
    v_SOR, iters = SOR_tol(A, b, v0, omega, tol)
    print ("SOR Method   :  %4d    %7.2e" %(iters, np.linalg.norm(v - v_SOR, np.inf)))

    v_CG, iters = CG_tol(A, b, v0, v, tol)
    print ("CG Method    :  %4d    %7.2e" %(iters, np.linalg.norm(v - v_CG, np.inf)))
```

```python
Iterative_solver(1, 1, 80, 80, gL, gR, gB, gT, 1.e-4)
```

    omega =  1.9253440689301244
    SOR Method   :   166    1.64e-04
    CG Method    :   148    9.80e-05

```python

```


# Probabilistic Principal Component Analysis versus Factor Analysis

Consider $\mathbf{z}_i \sim \mathcal{N}(0,I)$. Let

\begin{equation}
\mathbf{x}_i = W\mathbf{z}_i,~\text{where}~
W = \begin{pmatrix}
1 & 0 \\
1 & 0.001 \\
0 & 10
\end{pmatrix}.
\end{equation}

Let us generate 1,000 observations of such data.

```python
%matplotlib inline

import matplotlib.pyplot as plt
import numpy as np
from scipy import stats

np.random.seed(2016)

N = 1000
Z = stats.norm.rvs(loc=0, scale=1, size=2*N)
Z = Z.reshape((-1,2))
W = np.array([[1,0],[1,0.001],[0,10]])
X = np.dot(Z, W.T)
X[:10,:]
```

    array([[  0.29485409,   0.29536213,   5.08037685],
           [  0.7047403 ,   0.70556643,   8.26130625],
           [  2.13545214,   2.1348843 ,  -5.67838211],
           [  0.73523438,   0.73426439,  -9.69995451],
           [  0.69423613,   0.69560746,  13.71332032],
           [ -1.50154155,  -1.50107505,   4.66498411],
           [  1.48276751,   1.48449822,  17.30708419],
           [ -0.80440668,  -0.80654125, -21.3457542 ],
           [ -0.49926355,  -0.49931611,  -0.52567443],
           [ -0.40167833,  -0.40209348,  -4.15155475]])

Let's apply probabilistic principal component analysis, now, with $L = 1$. Here, it's assumed that

\begin{equation}
\mathbf{x}_i \sim \mathcal{N}\left(Wz_i,\sigma^2I\right).
\end{equation}

We can estimate the parameters with Equations 12.58 and 12.59.

```python
L = 1
U,D,V = np.linalg.svd(X)
V = V.T
Z = np.dot(U[:,:L], np.diag(D[:L]))
W = V[:,:L]
print(W)
print(np.dot(Z, W.T)) # reconstruction
print(X) # actual
```

    [[ 0.00109895]
     [ 0.00119895]
     [ 0.99999868]]
    [[  5.58380432e-03   6.09190848e-03   5.08104156e+00]
     [  9.08050899e-03   9.90679947e-03   8.26290481e+00]
     [ -6.23484264e-03  -6.80218871e-03  -5.67346073e+00]
     ...,
     [ -1.88548960e-03  -2.05706171e-03  -1.71572112e+00]
     [  1.38976272e-02   1.51622564e-02   1.26462922e+01]
     [  1.60906484e-03   1.75548338e-03   1.46418550e+00]]
    [[  0.29485409   0.29536213   5.08037685]
     [  0.7047403    0.70556643   8.26130625]
     [  2.13545214   2.1348843   -5.67838211]
     ...,
     [ -0.75720341  -0.75737481  -1.71398548]
     [  1.69950387   1.70076811  12.6424189 ]
     [  0.21678486   0.21693123   1.46369104]]

Thus, we see that this points along $x_3$, which is the direction of highest variance. Notice that when look at $Z$, the reconstructed $X$, the we accurately predict $x_3$, whereas are estimates for $x_1$ and $x_2$ are way off.

Now, let's try factor analysis. This time, we have

\begin{equation}
\mathbf{x}_i \sim \mathcal{N}\left(Wz_i + \boldsymbol\mu,\Psi\right),
\end{equation}

where $\Psi$ is diagonal. We can estimate $W$ and $\Psi$ with the EM algorithm described in 12.1.5.

```python
## initialize
W = stats.uniform.rvs(size=L*X.shape[1])
W = W.reshape((-1,L))
W = np.hstack((W, np.zeros(shape=(X.shape[1],1))))
Psi = np.diag(np.diag(np.dot(X.T, X)))/len(X)
m = np.empty(shape=(N, L)) # latent variable

def update_W(X, B, C):
    numerator = np.zeros(shape=(X.shape[1], B.shape[1]))
    denominator = np.zeros_like(C[0])
    for i in range(len(X)):
        numerator += np.outer(X[i], B[i])
        denominator += C[i]
    denominator
    return np.dot(numerator, np.linalg.inv(denominator))

def update_Psi(X, W, B):
    Psi = np.zeros(shape=(X.shape[1], X.shape[1]))
    for i in range(len(X)):
        Psi += np.outer(X[i] - np.dot(W, B[i]), X[i])
    return np.diag(np.diag(Psi))/len(X)

for k in range(100): # 50 iterations should be enough
    # E step
    Sigma = np.linalg.inv(np.eye(L) + np.dot(np.dot(W[:,:L].T, np.linalg.inv(Psi)), W[:,:L]))
    C = np.empty(shape=(N, L + 1, L + 1))
    for i in range(N):
        m[i] = np.dot(Sigma, np.dot(np.dot(W[:,:L].T, np.linalg.inv(Psi)), X[i] - W[:,L]))
        C[i][:L,:L] = Sigma + np.outer(m[i], m[i])
        C[i][L,:L] = m[i]
        C[i][:L,L] = m[i]
        C[i][L,L] = 1
    B = np.hstack((m, np.ones((N,1))))
    # M step
    W = update_W(X, B, C)
    Psi = update_Psi(X, W, B)
print(W[:,:L]) # principal directions
print(W[:,L]) # mu
print(Psi)
print(m)
```

    [[ 0.84388586]
     [ 0.84389397]
     [ 0.08117066]]
    [-0.03664242 -0.03669304 -0.50622523]
    [[  5.15256797e-07   0.00000000e+00   0.00000000e+00]
     [  0.00000000e+00   5.14875191e-07   0.00000000e+00]
     [  0.00000000e+00   0.00000000e+00   1.03013188e+02]]

Now, we see that the direction of this vector is approximately $(1,1,0)$, which is the direction of max correlation. Recall that

\begin{align}
x_{i1} &= z_{i1} \\
x_{i2} &= z_{i1} + 0.001z_{i2} \\
x_{i3} &= 10z_{i2},
\end{align}

so $x_{i1}$ and $x_{i2}$ are nearly identical, so they are very correlated. $x_{i3}$ can almost be treated as an indepedent normal variable.

The main difference between the two methods is $\Psi$: for factor analysis, we have that $\Psi$ is merely diagonal, whereas, PPCA requires that $\Psi = \sigma^2I$.

In this case, nearly all the variance is found in $x_{i3}$ since the coefficient is much larger. Thus, that direction explains the variance, so it's no surprise that our weight vector found in PPCA points in that direction. In this way, it's almost like when we project onto this direction, we're just observing a single-dimensional normal distribution. From another perspective, if we were going to try to reconstruct $\mathbf{x}_i$ with one dimension, we'd simply take the sample standard deviation of the third dimension since that has the largest variance and multiply it by a standard normal.

On the other hand, that single-dimensional normal distribution can be seen as independent from $x_{i1}$ and $x_{i2}$. Thus, we can explain everything that happens along that dimension with $\Psi$ by setting the variance for the third component high. In this way, we have maximized the likelihood for that dimension, so all that is left is to explain relationship between $x_{i1}$ and $x_{i2}$. They happen to be nearly identical, so the weight vector approximately points in the direction $(1,1,0)$.


```python
# default_exp datasets
%load_ext autoreload
%autoreload 2
```

    The autoreload extension is already loaded. To reload it, use:
      %reload_ext autoreload

```python
#export
import numpy as np
from diffusion_curvature.core import plot_3d
from nbdev.showdoc import *
```

In this notebook, we'll build various toy datasets and calculate their sectional curvatures.

This will use the Python symbolic computation library, sympy. Note: this library is not required to use the `diffusion_curvature` package. We merely employ it to calculate the curvature and appropriate expressions for rejection sampling.

```python
import sympy as sym
```

```python
theta = sym.Symbol('theta')
phi = sym.Symbol('phi')
R = sym.Symbol("R")
r = sym.Symbol("r")
```

```python
f1 = (R + r*sym.cos(theta))*sym.cos(phi)
```

```python
sym.diff(f1,theta)
```

$\displaystyle - r \sin{\left(\theta \right)} \cos{\left(\phi \right)}$

```python
f = sym.Matrix([(R + r*sym.cos(theta))*sym.cos(phi), (R + r*sym.cos(theta))*sym.sin(phi), r*sym.sin(theta)])
```

```python
sym.diff(f, theta)
```

$\displaystyle \left[\begin{matrix}- r \sin{\left(\theta \right)} \cos{\left(\phi \right)}\\- r \sin{\left(\phi \right)} \sin{\left(\theta \right)}\\r \cos{\left(\theta \right)}\end{matrix}\right]$

```python
(sym.diff(f, theta).T  * sym.diff(f, theta))[0]
```

$\displaystyle r^{2} \sin^{2}{\left(\phi \right)} \sin^{2}{\left(\theta \right)} + r^{2} \sin^{2}{\left(\theta \right)} \cos^{2}{\left(\phi \right)} + r^{2} \cos^{2}{\left(\theta \right)}$

# Rejection Sampling
How to sample uniformaly from a manifold? This is surprisingly difficult. You want to sample *uniformly in the manifold space*, but this is decidedly not Equivalent to sampling uniformly in the input space. The solution is a powerful technique called rejection sampling, which uses an analytic form of the manifold to "reject" points until only ones which approximate a uniform sampling *on* the manifold are allowed through.

The formula is
$$
\sqrt{Det(f'.T f')}
$$

```python
def rejection_sample_formula(f, variables):
    G = sym.Matrix.zeros(2,2)
    for i, x1 in enumerate(variables):
        for j, x2 in enumerate(variables):
            G[i,j] = (sym.diff(f, x1).T  * sym.diff(f, x2))[0]
    return sym.sqrt(G.det().simplify()).simplify()
```

```python
t = rejection_sample_formula(f,[theta, phi])
t
```

$\displaystyle \sqrt{r^{2} \left(R + r \cos{\left(\theta \right)}\right)^{2}}$

# Torus

The curvature of the torus is given by
$$ S(\theta) = \frac{8 \cos{\theta}}{5 + \cos{\theta}} $$

```python
#export
def rejection_sample_for_torus(n, r, R):
    # Rejection sampling torus method [Sampling from a torus (Revolutions)](https://blog.revolutionanalytics.com/2014/02/sampling-from-a-torus.html)
    xvec = np.random.random(n) * 2 * np.pi
    yvec = np.random.random(n) * (1/np.pi)
    fx = (1 + (r/R)*np.cos(xvec)) / (2*np.pi)
    return xvec[yvec < fx]

def torus(n=2000, c=2, a=1, noise=None, seed=None, use_guide_points = False):
    """
    Sample `n` data points on a torus. Modified from [tadasets.shapes — TaDAsets 0.1.0 documentation](https://tadasets.scikit-tda.org/en/latest/_modules/tadasets/shapes.html#torus)
    Uses rejection sampling.

    In addition to the randomly generated points, a few constant points have been added.
    The 0th point is on the outer rim, in a region of high positive curvature. The 1st point is in the inside, in a region of negative curvature, and the 2nd point is on the top, where the curvature should be closer to zero.

    Parameters
    -----------
    n : int
        Number of data points in shape.
    c : float
        Distance from center to center of tube.
    a : float
        Radius of tube.
    ambient : int, default=None
        Embed the torus into a space with ambient dimension equal to `ambient`. The torus is randomly rotated in this high dimensional space.
    seed : int, default=None
        Seed for random state.
    """

    assert a <= c, "That's not a torus"

    np.random.seed(seed)
    theta = rejection_sample_for_torus(n-2, a, c)
    phi = np.random.random((len(theta))) * 2.0 * np.pi

    data = np.zeros((len(theta), 3))
    data[:, 0] = (c + a * np.cos(theta)) * np.cos(phi)
    data[:, 1] = (c + a * np.cos(theta)) * np.sin(phi)
    data[:, 2] = a * np.sin(theta)

    if use_guide_points:
        data = np.vstack([[[0,-c-a,0],[0,c-a,0],[0,c,a]],data])

    if noise:
        data += noise * np.random.randn(*data.shape)

    # compute curvature of sampled torus
    ks = 8*np.cos(theta)/(5 + np.cos(theta))

    return data, ks
```

```python
show_doc(torus)
```

torus[source]

> torus(**`n`**=*`2000`*, **`c`**=*`2`*, **`a`**=*`1`*, **`noise`**=*`None`*, **`seed`**=*`None`*, **`use_guide_points`**=*`False`*)

Sample `n` data points on a torus. Modified from [tadasets.shapes — TaDAsets 0.1.0 documentation](https://tadasets.scikit-tda.org/en/latest/_modules/tadasets/shapes.html#torus)
Uses rejection sampling.

In addition to the randomly generated points, a few constant points have been added.
The 0th point is on the outer rim, in a region of high positive curvature. The 1st point is in the inside, in a region of negative curvature, and the 2nd point is on the top, where the curvature should be closer to zero.

Parameters
-----------
n : int
    Number of data points in shape.
c : float
    Distance from center to center of tube.
a : float
    Radius of tube.
ambient : int, default=None
    Embed the torus into a space with ambient dimension equal to `ambient`. The torus is randomly rotated in this high dimensional space.
seed : int, default=None
    Seed for random state.

Visualize with the curvature

```python
X,ks = torus(n=5000)
plot_3d(X, ks, title="Torus with scalar curvature")
```

# One-Sheet Hyperboloid

First, let's determine the rejection sampling formula

```python
a = sym.Symbol('a')
b = sym.Symbol('b')
theta = sym.Symbol("theta")
u = sym.Symbol('u')
f = sym.Matrix(
    [a*sym.cos(theta)*sym.sqrt(u**2+1),b*sym.sin(theta)*sym.sqrt(u**2+1),u]
)

```

```python
variables = [theta, u]
rej = rejection_sample_formula(f, variables)
rej
```

$\displaystyle \sqrt{a^{2} b^{2} u^{2} + a^{2} u^{2} \sin^{2}{\left(\theta \right)} + a^{2} \sin^{2}{\left(\theta \right)} - b^{2} u^{2} \sin^{2}{\left(\theta \right)} + b^{2} u^{2} - b^{2} \sin^{2}{\left(\theta \right)} + b^{2}}$

```python
#export
def rejection_sample_for_hyperboloid(n,a,b,c,u_limit):
    theta = np.random.random(n)*2*np.pi
    u = (np.random.random(n)*2 - 1)*u_limit
    fx = np.sqrt(a**2 * b**2 * u**2 + a**2 * u**2 * np.sin(theta)**2 + a**2 * np.sin(theta)**2  - b**2 * u**2 * np.sin(theta)**2 + b**2 * u**2 - b**2 * np.sin(theta)**2 + b**2)
    yvec = np.random.random(n) * (1/np.max(fx))
    return theta[yvec < fx], u[yvec < fx]

def hyperboloid(n=2000,a=2,b=2,c=1, u_limit = 2, seed=None):
    """Sample roughly n points on a hyperboloid, using rejection sampling.

    Parameters
    ----------
    n : int, optional
        number of points, by default 2000
    a : int, optional
        hyperboloid param1, by default 2
    b : int, optional
        hyperboloid param2, by default 2
    c : int, optional
        stretchiness in z, by default 1
    u_limit : int, optional
        Constrain the free parameter u to [-l,l], by default 2
    seed : int, optional
        For repeatability, seed the randomness, by default None

    Returns
    -------
    The sampled points, and the curvatures of each point
    """

    np.random.seed(seed)
    theta, u = rejection_sample_for_hyperboloid(n,a,b,c,u_limit)
    data = np.zeros((len(theta), 3))
    data[:, 0] = a*np.cos(theta)*np.sqrt(u**2 + 1)
    data[:, 1] = b*np.sin(theta)*np.sqrt(u**2 + 1)
    data[:, 2] = c*u

    # compute curvature of sampled hyperboloid
    ks = -(2/(5*data[:,2]**2 + 1)**2)

    return data, ks
```

```python
show_doc
```

    <function nbdev.showdoc.show_doc(elt, doc_string: bool = True, name=None, title_level=None, disp=True, default_cls_level=2, show_all_docments=False, verbose=False)>

```python
X, ks = hyperboloid(2000)
plot_3d(X,ks,colorbar=True,use_plotly=False)
```

# Ellipsoid

```python
a = sym.Symbol('a')
b = sym.Symbol('b')
c = sym.Symbol('c')
theta = sym.Symbol("theta")
phi = sym.Symbol("phi")
u = sym.Symbol('u')
f = sym.Matrix(
    [a*sym.cos(theta)*sym.sin(phi),b*sym.sin(theta)*sym.sin(phi),c*sym.cos(phi)]
)
```

```python
variables = [theta, phi]
rej = rejection_sample_formula(f, variables)
rej
```

$\displaystyle \sqrt{- a^{2} b^{2} \sin^{4}{\left(\phi \right)} + a^{2} b^{2} \sin^{2}{\left(\phi \right)} + a^{2} c^{2} \sin^{4}{\left(\phi \right)} \sin^{2}{\left(\theta \right)} - b^{2} c^{2} \sin^{4}{\left(\phi \right)} \sin^{2}{\left(\theta \right)} + b^{2} c^{2} \sin^{4}{\left(\phi \right)}}$

```python
#export
def rejection_sample_for_ellipsoid(n,a,b,c):
    theta = np.random.random(n)*2*np.pi
    phi = np.random.random(n)*2*np.pi
    fx = np.sqrt(-a**2 * b**2 * np.sin(phi)**4 + a**2 * b**2 * np.sin(phi)**2 + a**2 * c**2 * np.sin(phi)**4 * np.sin(theta)**2 - b**2 * c**2 * np.sin(phi)**4 * np.sin(theta)**2 + b**2 * c**2 * np.sin(phi)**4)
    yvec = np.random.random(n) * (1/np.max(fx))
    return theta[yvec < fx], phi[yvec < fx]

def ellipsoid(n=2000,a=3,b=2,c=1, seed=None):
    """Sample roughly n points on an ellipsoid, using rejection sampling.

    Parameters
    ----------
    n : int, optional
        number of points, by default 2000
    a : int, optional
        ellipsoid param1, by default 3
    b : int, optional
        ellipsoid param2, by default 2
    c : int, optional
        stretchiness in z, by default 1
    seed : int, optional
        For repeatability, seed the randomness, by default None

    Returns
    -------
    The sampled points, and the curvatures of each point
    """

    np.random.seed(seed)
    theta, phi = rejection_sample_for_ellipsoid(n,a,b,c)
    data = np.zeros((len(theta), 3))
    data[:, 0] = a*np.cos(theta)* np.sin(phi)
    data[:, 1] = b*np.sin(theta)*np.sin(phi)
    data[:, 2] = c*np.cos(phi)

    # compute curvature of sampled torus (gaussian curvature)
    ks = 2* (a**2 * b**2 * c**2) / (a**2 * b**2 * np.cos(phi)**2 + c**2 * (b**2 * np.cos(theta)**2 + a**2 * np.sin(theta)**2)*np.sin(phi)**2)**2

    return data, ks
```

```python
X, ks = ellipsoid(n=5000)
plot_3d(X,ks,colorbar=True)
```

```python

```

# Hypersphere

# Random Cube

```python

def random_cube(n):
    """Return a random cube

    Parameters
    ----------
    n : _type_
        _description_

    Returns
    -------
    _type_
        _description_
    """
    data = np.random.rand(n,3)
    return data
```

# Saddle Regions
Generate hyperbolic regions as test cases of Laziness curvature.

```python
a = sym.Symbol('a')
b = sym.Symbol('b')
x = sym.Symbol("x")
y = sym.Symbol("y")
f = sym.Matrix(
    [x,y,a*x**2 + b*y**2]
)
```

```python
variables = [x, y]
rej = rejection_sample_formula(f, variables)
rej
```

$\displaystyle \sqrt{4 a^{2} x^{2} + 4 b^{2} y^{2} + 1}$

```python
#export
def rejection_sample_for_saddle(n,a,b):
    x = np.random.random(n)*2 - 1 # random values in -1, 1
    y = np.random.random(n)*2 - 1
    fx = np.sqrt(4*a**2*x**2 + 4*b**2*y**2 + 1)
    yvec = np.random.random(n) * (1/np.max(fx))
    return x[yvec < fx], y[yvec < fx]

def paraboloid(n=2000,a=1,b=-1, seed=None, use_guide_points = False):
    """Sample roughly n points on a saddle, using rejection sampling for even density coverage
    Defined by $ax^2 + by^2$.

    Parameters
    ----------
    n : int, optional
        number of points, by default 2000
    a : int, optional
        ellipsoid param1, by default 1
    b : int, optional
        ellipsoid param2, by default -1
    seed : int, optional
        For repeatability, seed the randomness, by default None

    Returns
    -------
    The sampled points, and the curvatures of each point
    """
    if use_guide_points:
        n = n - 1
    np.random.seed(seed)
    x, y = rejection_sample_for_saddle(n,a,b)
    if use_guide_points:
        x = np.concatenate([[0],x])
        y = np.concatenate([[0],y])
    data = np.zeros((len(x), 3))
    data[:, 0] = x
    data[:, 1] = y
    data[:, 2] = a*x**2 + b*y**2
    # compute curvature of sampled saddle region
    # TODO: Compute gaussian curvature
    # TODO: Currently assuming that b is negative (hyperbolic paraboloid)
    ap = np.sqrt(1/a)
    bp = b/np.abs(b) * np.sqrt(1/np.abs(b))
    ks = -(4*a**6 * b**6)/(a**4*b**4 + 4*b**4*x**2+4*a**4*y**2)**2

    return data, ks
```

```python
X, ks = paraboloid(n=10000, a = 1, b = -1,use_guide_points=True)
```

```python
plot_3d(X,ks)
```

```python
x = np.zeros(10)
```

```python
np.concatenate([[0],x])
```

    array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])

```python

```

11_dataset tests avg transition probs.ipynb


# Disambiguating Advection and KdV

Any single soliton travelling wave solution of the KdV equation (1) also solves the advection equation with $c$ depending on the speed of translation, which for KdV depends on the amplitude.  However, if two waves have different amplitudes and speeds, they may both solve the KdV equation but will not solve the same advection equation.

\begin{align}
u_t &+ 6uu_x + u_{xxx} = 0 \hspace{2cm} (1)\\
u_t &+ cu_x = 0\hspace{3.5cm} (2)
\end{align}

Here we show that looking at a single travelling wave solution, we recover the advection equation (at least for the two amplitudes shown), but when we look at both we get the KdV equation.  Observing that waves with different amplitudes travel at different rates indicates the need for nonlinearity in the evolution equation.

```python
%pylab inline
pylab.rcParams['figure.figsize'] = (12, 8)
import numpy as np
from mpl_toolkits.mplot3d import Axes3D
from PDE_FIND import *
```

    Populating the interactive namespace from numpy and matplotlib

# Generate a few solutions to each equation

For any $c$, the function below is a very simple closed form solution to the KdV and advection equations.  However, it always solves the KdV equation regardless of the choice of $c$ while solutions with different values of $c$ will each solve an advection equation but they won't be the same one.

```python
# Soliton solution to KdV/advection equations
def soliton(x,t,c,a):
    return c/2*np.cosh(np.sqrt(c)/2*(x-c*t-a))**-2
```

```python
c1 = 5.0
c2 = 1.0
n = 256
m = 50
x = linspace(-10, 12, n)
dt = 0.025
dx = x[1]-x[0]
t = linspace(dt,m*dt,m)

U1 = np.zeros((n,m))
U2 = np.zeros((n,m))

for i in range(n):
    for j in range(m):
        U1[i,j] = soliton(x[i],t[j],c1,-3)
        U2[i,j] = soliton(x[i],t[j],c2,-1)
```

```python
X, T = np.meshgrid(x, t)
fig1 = figure()
ax = fig1.gca(projection='3d')
surf = ax.plot_surface(X, T, U1.T, rstride=1, cstride=1, cmap=cm.coolwarm,
    linewidth=0, antialiased=False)
```

```python
X, T = np.meshgrid(x, t)
fig1 = figure()
ax = fig1.gca(projection='3d')
surf = ax.plot_surface(X, T, U2.T, rstride=1, cstride=1, cmap=cm.coolwarm,
    linewidth=0, antialiased=False)
```

## Look at the solutions individually

For each of the travelling waves, we get an advection equation.

```python
# First look at the soliton with speed = 5
Ut1, R1, rhs_des1 = build_linear_system(U1, dt, dx, D=3, P=2, time_diff = 'FD', space_diff = 'FD')
lam = 10**-5
d_tol = 5

w = TrainSTRidge(R1,Ut1,lam,d_tol)
print "PDE derived using STRidge with L0 penalty"
print_pde(w, rhs_des1)
```

    PDE derived using STRidge with L0 penalty
    u_t = (-4.981273 -0.000000i)u_{x}

```python
# Now with speed 1
Ut2, R2, _ = build_linear_system(U2, dt, dx, D=3, P=2, time_diff = 'FD', space_diff = 'FD')

w = TrainSTRidge(R2,Ut2,lam,d_tol)
print "PDE derived using STRidge with L0 penalty"
print_pde(w, rhs_des1)
```

    PDE derived using STRidge with L0 penalty
    u_t = (-1.000820 -0.000000i)u_{x}

## Both solutions $\rightarrow$ KdV

Taking the two linear systems and simply stacking them so that we're looking for a PDE that accuractely reflects the dynamics of each yields the KdV equation.

```python
# Now look for a solution for both linear systems -> kdv
R = np.vstack([R1, R2])
Ut = np.vstack([Ut1,Ut2])

w = TrainSTRidge(R,Ut,lam,d_tol)
print "PDE derived using STRidge with L0 penalty"
print_pde(w, rhs_des1)
```

    PDE derived using STRidge with L0 penalty
    u_t = (-6.073982 +0.000000i)uu_{x}
        + (-1.039532 +0.000000i)u_{xxx}

```python

```


```julia
using Plots            # To generate figures
using JuMP
using Cbc              # Solver used
using JLD2             # File input and output
```

## Exercise 11.2 - Capacitated vehicle routing problem (CVRP)

### Background

Consider a centralised depot, from which deliveries are supposed to be made from. The delivers have to be made to a number of clients, and each client has a specific demand. Some assumptions we will consider:

* The deliveries have to be made by vehicles that are of limited capacity;
* Multiple routes are created, and a single vehicle is assignmed to each route;
* We assume that the number of vehicles is not a limitation.

Our objective is to define **optimal routes** such that the total distance travelled is minimised.

### Problem structure and input data

Let us define the elements that form our problem. You will notice that we use a graph-based notation, referring to elements such as arcs and nodes.

#### Structural elements
* $n$ is the total number of clients
* $N$ is the *set* of clients, with $N = \{2, \dots, n+1\}$
* $V$ is the set of *nodes*, representing a depot (node 1) and the clients (nodes $i \in N$). Thus $V = \{1\} \cup N$.
* $A$ is a set of *arcs*, with $A = \{(i,j) \in V \times V : i \neq j\}$

#### Parameters (input data structure)
* $C_{i,j}$ - cost of travelling via arc $(i,j) \in A$ (equals distance between $i$ and $j$);
* $Q$ - vehicle capacity in units;
* $D_i$ - amount that has to be delivered to customer $i \in N$, in units;

```julia
# Problem structural elements
struct Instance
   n     # Total of nodels
   N     # Set of client nodes
   V     # Set of all nodes, including depot
   A     # Set of arcs
   loc_x # x-coordinates of all points
   loc_y # y-coordinates of all points
   Q     # vehicle capacity
   D     # demand at the node (to be delivered)
   C     # arc trasversal cost
   bigM  # A suitable big M value to be used in all big M constraints
end
```

```julia
f = jldopen("ex11_ins.jld2")
ins = nothing
try
    ins = f["ins"]
finally
    close(f)
end;
```

The cell below shows a plot of the problem, indicting demands and time windows.

```julia
## Plotting nodes

scatter(ins.loc_x, ins.loc_y,
    legend = false,
    size = (1400,700),
    xaxis = ("x", (-50.0,1100.0)),
    yaxis = ("y", (-50.0,700.0)),
)

for i in ins.N
    annotate!(ins.loc_x[i], ins.loc_y[i] +15, ("$i", 7))

    ## These make the plot somewhat hard to read
    annotate!(ins.loc_x[i], ins.loc_y[i] - 17, ("D[$(i)] = $(ins.D[i])", 7))
end

scatter!((ins.loc_x[1], ins.loc_y[1]), color=:orange)
```

### Item (a)

If we allow multiple departures and arrivals for the depot node and each client node is visited exactly once, we end up with a
MIP formulation:

\begin{align}
    \min & \sum_{(i,j) \in A} C_{i,j}x_{i,j} \\
    \text{s.t.: } & \sum_{j \in V, j \neq i} x_{i,j} = 1, \ \forall i \in N  \\
    & \sum_{i \in V, i \neq j} x_{i,j} = 1, \ \forall j \in N  \\
    & u_i - D_j \leq u_j + M*( 1 - x_{i,j}) , \ \forall i \in V, j \in N \\
    & u_i - D_j \geq u_j - M*( 1 - x_{i,j}) , \ \forall i \in V, j \in N \\
    & 0 \leq u_i \leq Q, \ \forall i \in V  \\
    & u_1 = Q \\
    & x_{i,j} \in \{0,1\}, \ \forall i \in V, \forall j \in V
\end{align}

We use variables $x_{i,j}$ for modelling whether the arc $(i,j)$ is "active" in the solution. If $x_{i,j}=1$, the arc $(i,j)$ is active, meaning that there is a vehicle going from node $i$ to node $j$. The objective function $\sum_{(i,j) \in A} C_{i,j}x_{i,j}$ represents the total cost of moving along the active arcs in the solution. The next two constraints $\sum_{j \in V, j \neq i} x_{i,j} = 1, \ \forall i \in N$ and $\sum_{i \in V, i \neq j} x_{i,j} = 1, \ \forall j \in N$ state that each node in $N$ (the set of *client* nodes) must be visited exactly once. A visit is defined as a vehicle entering and leaving that node. So far, this should be familiar, these same ideas would be found in a TSP formulation.

The differences between the capacitated vehicle routing problem (CVRP) and travelling salesperson problems (TSP) are
1. We have multiple vehicles available instead of a single salesperson
2. The vehicles have a limited capacity

To model the limited capacity, we introduce variables $u_i$, representing the remaining cargo in a vehicle when leaving a node $i$. We assume that all vehicles leave the depot full of cargo ($U_1 = Q$). We then want a constraint saying that if a vehicle goes from node $i \in V$ to node $j \in N$, the amount of cargo in the vehicle must decrease by $D_j$. For this, we use two big-M constraints $u_i - D_j \leq u_j + M*( 1 - x_{i,j}) , \ \forall i \in V, j \in N$ and $u_i - D_j \geq u_j - M*( 1 - x_{i,j}) , \ \forall i \in V, j \in N$. When combined, these result in something like:
$$ x_{i,j}=1 \Rightarrow u_i - D_j = u_j$$
$$ x_{i,j}=0 \Rightarrow -\infty \le u_i - D_j - u_j \le \infty$$

The second line can also be written as $ x_{i,j}=0 \Rightarrow u_i - D_j - u_j \in \mathbb{R}$, meaning that there is no relationship between $u_i$ and $u_j$.
#'
### Item (b)
Complete the function `create_VRP_model()` below to implement the CVRP model.

```julia
function create_VRP_model(ins; max_time = 300)

    n = ins.n          # Number of client nodes
    N = ins.N          # Set of client nodes
    V = ins.V          # Set of all nodes, including depot
    A = ins.A          # Set of arcs
    loc_x = ins.loc_x  # x-coordinates of all points
    loc_y = ins.loc_y  # y-coordinates of all points
    Q = ins.Q          # Vehicle capacity
    D = ins.D          # Demand at the node (to be delivered)
    C = ins.C          # Arc traversal cost
    bigM = ins.bigM    # A large enough big M value to be used in all big M constraints

    model = Model(Cbc.Optimizer)                # Declaring the model object.
    set_optimizer_attribute(model, "seconds", max_time)

    # Declare decision variables
    @variable(model, x[i in V, j in V], Bin)
    @variable(model, 0 <= u[i in V] <= Q)
    @constraint(model, u[1]==Q)

    # Objective function
    @objective(model, Min, sum(C[i,j] * x[i,j] for (i,j) in A))

    # Constraints
    @constraint(model, c1[i in N], sum(x[i,j] for j in V if j != i) == 1)
    @constraint(model, [j in N], sum(x[i,j] for i in V if i != j) == 1)
    @constraint(model, [i in V, j in N], u[i] - D[j] <= u[j] + bigM * (1 - x[i,j]))
    @constraint(model, [i in V, j in N], u[i] - D[j] >= u[j] - bigM * (1 - x[i,j]))

    return model
end
```

```julia
## Create the basic model and optimise it
basic_model = create_VRP_model(ins; max_time = 300)
optimize!(basic_model)
```

The cell below shows a plot of the solution.

```julia
## Retrieving active arcs.
active_arcs = [(i,j) for (i,j) in ins.A if value.(basic_model[:x])[i,j] >= 1-(1E-9)]

scatter(ins.loc_x, ins.loc_y,
    legend = false,
    size = (1100,700),
    xaxis = ("x", (-10.0,1100.0)),
    yaxis = ("y", (-10.0,700.0)),
)

for i in ins.N
    annotate!(ins.loc_x[i], ins.loc_y[i] +30, ("D[$i]=$(ins.D[i])", 7))
    annotate!(ins.loc_x[i], ins.loc_y[i] +15, ("u[$i]=$(round(value.(basic_model[:u])[i], digits=1))", 7))
end

## Plotting the arcs
for (i,j) in active_arcs
    quiver!([ins.loc_x[i]], [ins.loc_y[i]], quiver=([ins.loc_x[j] - ins.loc_x[i]], [ins.loc_y[j] - ins.loc_y[i]]),
        color = 1
    )
end

scatter!((ins.loc_x[1], ins.loc_y[1]), color=:orange)
```

## Exercise 11.1 - Preprocessing and primal heuristics
#'
### b) RINS and RENS for UFL-W

From exercise 9: Let $N = \{1,\dots,n\}$ be a set of potential facilities and $M = \{1,\dots,m\}$ a set of clients. Let $y_j = 1$ if facility $j$ is opened, and $y_j = 0$ otherwise. Moreover, let $x_{ij}$ be the fraction of client $i$'s demand satisfied from facility $j$. The UFL can be formulated as the mixed-integer problem (MIP):

$$\begin{align}
 \text{(UFL-W)} : \quad &\min_{x,y} \sum_{j\in N} f_jy_j + \sum_{i\in M}\sum_{j\in N} c_{ij}x_{ij}\\
       &\text{s.t.} \\
       &\quad \sum_{j\in N}x_{ij} = 1, &\forall i \in M,\\
       &\quad \sum_{i\in M}x_{ij} \leq my_j, &\forall  j \in N,\\
       &\quad x_{ij} \geq 0, &\forall i \in M, \forall j \in N,\\
       &\quad y_j \in \{0,1\}, &\forall j\in N,
\end{align}$$

where $f_j$ is the cost of opening facility $j$, and $c_{ij}$ is the cost of satisfying client $i$'s demand from facility $j$. Consider an instance (different from exercise 9) of the UFL with opening costs $f=(21,16,30,24,11)$ and client costs

$$\begin{align*}
 (c_{ij}) = \left(
	\begin{array}{ccccc}
		6  & 9  & 3 & 4  & 12 \\
		1  & 2  & 4 & 9  & 2  \\
		15 & 2  & 6 & 3  & 18 \\
		9  & 23 & 4 & 8  & 1  \\
		7  & 11 & 2 & 5  & 14 \\
		4  & 3  & 10& 11 & 3
	\end{array}
 \right)
\end{align*}$$

```julia
f = [21 16 30 24 11] # Facility opening costs
c = [6 9 3 4 12; 1 2 4 9 2; 15 2 6 3 18; 9 23 4 8 1; 7 11 2 5 14; 4 3 10 11 3] # Cost of satisfying demand
(m, n) = size(c)
M = 1:m # Set of clients
N = 1:n;# Set of facilities
```

#### Obtain MIP solution for comparison

```julia
ufl_w = Model(Cbc.Optimizer)

@variable(ufl_w, x[M,N] >= 0) # Fraction of demand (client i) satisfied by facility j
@variable(ufl_w, y[N], Bin)   # Facility location

# Minimize total cost
@objective(ufl_w, Min, sum(f[j]*y[j] for j in N) + sum(c[i,j]*x[i,j] for i in M, j in N))

# For each client, the demand must be fulfilled
@constraint(ufl_w, demand[i in M], sum(x[i,j] for j in N) == 1)
# A big-M style constraint stating that facility j can't send out anything if y[j]==0
@constraint(ufl_w, supply[j in N], sum(x[i,j] for i in M) <= m*y[j])

optimize!(ufl_w)
```

```julia
println("UFL-W MILP:")
println("Optimal value $(objective_value(ufl_w))")
println("with y = $(value.(y).data)")
```

#### Obtain LP relaxation solution for heuristics

```julia
ufl_w_rel = Model(Cbc.Optimizer)

@variable(ufl_w_rel, x[M,N] >= 0) # Fraction of demand (client i) satisfied by facility j
@variable(ufl_w_rel, 0<=y[N]<=1)  # Facility location

# Minimize total cost
@objective(ufl_w_rel, Min, sum(f[j]*y[j] for j in N) + sum(c[i,j]*x[i,j] for i in M, j in N))

# For each client, the demand must be fulfilled
@constraint(ufl_w_rel, demand[i in M], sum(x[i,j] for j in N) == 1)
# A big-M style constraint stating that facility j can't send out anything if y[j]==0
@constraint(ufl_w_rel, supply[j in N], sum(x[i,j] for i in M) <= m*y[j])

optimize!(ufl_w_rel)
```

```julia
println("UFL-W LP:")
println("Optimal value $(objective_value(ufl_w_rel))")
println("with y = $(value.(y).data)")
```

#### RINS
We are given a primal feasible solution $\bar{y}=(1,0,0,0,0)$ for starting RINS. The solution for this UFL-W LP relaxation is $y^{LP} = (0, 1/6, 1/3, 0, 1/2)$.

```julia
y_bar = [1,0,0,0,0]
y_lp = [0, 1/6, 1/3, 0, 1/2]

ufl_rins = Model(Cbc.Optimizer)

@variable(ufl_rins, x[M,N] >= 0) # Fraction of demand (client i) satisfied by facility j
@variable(ufl_rins, y[N], Bin)   # Facility location

# Minimize total cost
@objective(ufl_rins, Min, sum(f[j]*y[j] for j in N) + sum(c[i,j]*x[i,j] for i in M, j in N))

# For each client, the demand must be fulfilled
@constraint(ufl_rins, demand[i in M], sum(x[i,j] for j in N) == 1)
# A big-M style constraint stating that facility j can't send out anything if y[j]==0
@constraint(ufl_rins, supply[j in N], sum(x[i,j] for i in M) <= m*y[j])

# Fixing variables that have the same value in our primal feasible solution and the LP relaxation
# TODO: add your code here

optimize!(ufl_rins)
```

```julia
println("UFL-W RINS solution:")
println("Optimal value $(objective_value(ufl_rins))")
println("with y = $(value.(y).data)")
```

#### RENS
RINS gave us a pretty good solution, let's see what happens with RENS.

```julia
y_lp = [0, 1/6, 1/3, 0, 1/2]

ufl_rens = Model(Cbc.Optimizer)

@variable(ufl_rens, x[M,N] >= 0) # Fraction of demand (client i) satisfied by facility j
@variable(ufl_rens, y[N], Bin)   # Facility location

# Minimize total cost
@objective(ufl_rens, Min, sum(f[j]*y[j] for j in N) + sum(c[i,j]*x[i,j] for i in M, j in N))

# For each client, the demand must be fulfilled
@constraint(ufl_rens, demand[i in M], sum(x[i,j] for j in N) == 1)
# A big-M style constraint stating that facility j can't send out anything if y[j]==0
@constraint(ufl_rens, supply[j in N], sum(x[i,j] for i in M) <= m*y[j])

# Fixing variables that have an integer solution in the LP relaxation
# TODO: add your code here

optimize!(ufl_rens)
```

```julia
println("UFL-W RENS solution:")
println("Optimal value $(objective_value(ufl_rens))")
println("with y = $(value.(y).data)")
```

In this case, RINS and RENS gave the same solution, but RINS depends on the initial feasible solution provided.


```python
from sympy import *
init_printing()
import numpy as np
%matplotlib widget
import matplotlib as mpl
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from scipy.integrate import solve_ivp
```

```python
Y, X = np.mgrid[-7:7:500j, -7:7:500j]

U = np.sin(X + Y)
V = X - Y
plt.streamplot(X, Y, U, V)
```

    Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …

    <matplotlib.streamplot.StreamplotSet at 0x7f8f6dd053d0>

```python
fig = plt.figure()
ax = fig.add_subplot(1, 1, 1, projection='3d')

def lorenz(t, X):
    # Unpack `X` into 3 coordinates
    x, y, z = X
    # Formulas for `\dot x`, `\dot y`, `\dot z`
    dx = 10 * (y - x)
    dy = x * (28 - z) - y
    dz = x * y - (8 / 3) * z
    return (dx, dy, dz)

for pt in [(0.01, 0, 0), (0, 0, 1)]:
    r = solve_ivp(lorenz, (0, 200), pt, dense_output=True)
    T = np.linspace(50, 80, 10000)
    Y = r.sol(T)
    ax.plot(Y[0], Y[1], Y[2])
```

    Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …

```python
fig = plt.figure()
ax = fig.add_subplot(1, 1, 1, projection='3d')

def pendulum2(t, X):
    # Unpack `X` into 4 coordinates
    x, y, z, u = X
    # Formulas for `\dot x`, `\dot y`, `\dot z`, `\dot u`
    dx = y
    dy = -x
    dz = u
    du = -z
    return (dx, dy, dz, du)

for pt in [(0.5, 0, 0, 0.5), (0, 0.5, 1, 0)]:
    r = solve_ivp(pendulum2, (0, 200), pt, dense_output=True)
    T = np.linspace(50, 80, 10000)
    Y = r.sol(T)
    ax.plot(Y[0], Y[1], Y[2])
```

    Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …

```python
fig = plt.figure()
ax = fig.add_subplot(1, 1, 1)

for pt in [(0.5, 0, 0, 0.5), (0, 0.5, 1, 0)]:
    r = solve_ivp(pendulum2, (0, 200), pt, dense_output=True)
    T = np.linspace(50, 80, 10000)
    Y = r.sol(T)
    ax.plot(Y[0], Y[2])
```

    Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …


# Gaussian Process Distribution of Relaxation Times

## In this tutorial we will reproduce Figure 7 of the article https://doi.org/10.1016/j.electacta.2019.135316

GP-DRT is our newly developed approach that can be used to obtain both the mean and covariance of the DRT from the EIS data by assuming that the DRT is a Gaussian process (GP). The GP-DRP can  predict the DRT and the imaginary part of the impedance at frequencies that were not previously measured.

To obtain the DRT from the impedance we take that $\gamma(\xi)$ is a GP where $f$ is the frequency and $\xi=\log f$. Under the DRT model and considering that GPs are closed linear transformations, it follows that $Z^{\rm DRT}_{\rm im}\left(\xi\right)$ is also a GP.

More precisely we can write

$$\begin{pmatrix}
\gamma(\xi) \\
Z^{\rm DRT}_{\rm im}\left(\xi\right)
\end{pmatrix}\sim \mathcal{GP}\left(\mathbf 0, \begin{pmatrix}
k(\xi, \xi^\prime) & \mathcal L^{\rm im}_{\xi^\prime} \left(k(\xi, \xi^\prime)\right)\\
\mathcal L^{\rm im}_{\xi} k(\xi, \xi^\prime) & \mathcal L^{\rm im}_{\xi^\prime}\left(\mathcal L^{\rm im}_{\xi} \left(k(\xi, \xi^\prime)\right)\right)
\end{pmatrix}\right)$$

where

$$\mathcal L^{\rm im}_\xi \left(\cdot\right) = -\displaystyle \int_{-\infty}^\infty \frac{2\pi \displaystyle e^{\xi-\hat \xi}}{1+\left(2\pi \displaystyle e^{\xi-\hat \xi}\right)^2}  \left(\cdot\right) d \hat \xi$$

is a linear functional. The latter functional, transforms the DRT to the imaginary part of the impedance.

Assuming we have $N$ observations, we can set $\left(\mathbf Z^{\rm exp}_{\rm im}\right)_n = Z^{\rm exp}_{\rm im}(\xi_n)$ with $\xi_n =\log f_n$ and $n =1, 2, \ldots N $. The corresponding multivariate Gaussian random variable can be written as

$$\begin{pmatrix}
\boldsymbol{\gamma} \\
\mathbf Z^{\rm exp}_{\rm im}
\end{pmatrix}\sim \mathcal{N}\left(\mathbf 0, \begin{pmatrix}
\mathbf K & \mathcal L_{\rm im} \mathbf K\\
\mathcal L_{\rm im}^\sharp \mathbf K & \mathcal L^2_{\rm im} \mathbf K + \sigma_n^2 \mathbf I
\end{pmatrix}\right)$$

where

$$\begin{align}
(\mathbf K)_{nm} &= k(\xi_n, \xi_m)\\
(\mathcal L_{\rm im} \mathbf K)_{nm} &= \left. \mathcal L^{\rm im}_{\xi^\prime} \left(k(\xi, \xi^\prime)\right) \right |_{\xi_n, \xi_m}\\
(\mathcal L_{\rm im}^\sharp \mathbf K)_{nm} &= \left.\mathcal L^{\rm im}_{\xi} \left(k(\xi, \xi^\prime)\right) \right|_{\xi_n, \xi_m}\\
(\mathcal L^2_{\rm im} \mathbf K)_{nm} &= \left.\mathcal L^{\rm im}_{\xi^\prime}\left(\mathcal L^{\rm im}_{\xi} \left(k(\xi, \xi^\prime)\right)\right) \right|_{\xi_n, \xi_m}
\end{align}$$

and $\mathcal L_{\rm im} \mathbf K^\top = \mathcal L_{\rm im}^\sharp \mathbf K$.

To obtain the DRT from impedance, the distribution of $\mathbf{\gamma}$ conditioned on $\mathbf Z^{\rm exp}_{\rm im}$ can be written as

$$\boldsymbol{\gamma}|\mathbf Z^{\rm exp}_{\rm im}\sim \mathcal N\left( \mathbf \mu_{\gamma|Z^{\rm exp}_{\rm im}}, \mathbf\Sigma_{\gamma| Z^{\rm exp}_{\rm im}}\right)$$

with

$$\begin{align}
\mathbf \mu_{\gamma|Z^{\rm exp}_{\rm im}} &= \mathcal L_{\rm im} \mathbf K \left(\mathcal L^2_{\rm im} \mathbf K + \sigma_n^2 \mathbf I \right)^{-1} \mathbf Z^{\rm exp}_{\rm im} \\
\mathbf \Sigma_{\gamma| Z^{\rm exp}_{\rm im}} &= \mathbf K- \mathcal L_{\rm im} \mathbf K \left(\mathcal L^2_{\rm im} \mathbf K + \sigma_n^2 \mathbf I \right)^{-1}\mathcal L_{\rm im} \mathbf K^\top
\end{align}$$

The above formulas depend on 1) the kernel, $k(\xi, \xi^\prime)$; 2) the noise level, $\sigma_n$; and 3) the experimental data, $\mathbf Z^{\rm exp}_{\rm im}$ (at the log-frequencies $\mathbf \xi$).

```python
import numpy as np
import matplotlib.pyplot as plt
from math import sin, cos, pi
import GP_DRT
from scipy.optimize import minimize
%matplotlib inline
```

## 1) Define parameters of the ZARC circuit which will be used for the synthetic experiment generation

The impedance of a ZARC can be written as
$$
Z^{\rm exact}(f) = R_\infty + \displaystyle \frac{1}{\displaystyle \frac{1}{R_{\rm ct}}+C \left(i 2\pi f\right)^\phi}
$$

where $\displaystyle C = \frac{\tau_0^\phi}{R_{\rm ct}}$.

The analytical DRT can be computed analytically as

$$
\gamma(\log \tau) =  \displaystyle \frac{\displaystyle R_{\rm ct}}{\displaystyle 2\pi} \displaystyle \frac{\displaystyle \sin\left((1-\phi)\pi\right)}{\displaystyle \cosh(\phi \log(\tau/\tau_0))-\cos(\pi(1-\phi))}
$$

```python
# define the frequency range
N_freqs = 81
freq_vec = np.logspace(-4., 4., num=N_freqs, endpoint=True)
xi_vec = np.log(freq_vec)
tau  = 1/freq_vec

# define the frequency range used for prediction
# note: we could have used other values
freq_vec_star = np.logspace(-4., 4., num=81, endpoint=True)
xi_vec_star = np.log(freq_vec_star)

# parameters for ZARC model, the impedance and analytical DRT are calculated as the above equations
R_inf = 10
R_ct = 50
phi = 0.8
tau_0 = 1.

C = tau_0**phi/R_ct
Z_exact = R_inf+1./(1./R_ct+C*(1j*2.*pi*freq_vec)**phi)
gamma_fct = (R_ct)/(2.*pi)*sin((1.-phi)*pi)/(np.cosh(phi*np.log(tau/tau_0))-cos((1.-phi)*pi))

# we will use a finer mesh for plotting the results
freq_vec_plot  = np.logspace(-4., 4., num=10*(N_freqs-1), endpoint=True)
tau_plot  = 1/freq_vec_plot
gamma_fct_plot = (R_ct)/(2.*pi)*sin((1.-phi)*pi)/(np.cosh(phi*np.log(tau_plot/tau_0))-cos((1.-phi)*pi)) # for plotting only

# we will add noise to the impedance computed analytically
rng = np.random.seed(214975)
sigma_n_exp = 1.
Z_exp = Z_exact + sigma_n_exp*(np.random.normal(0, 1, N_freqs)+1j*np.random.normal(0, 1, N_freqs))
```

## 2) Show the synthetic impedance in the Nyquist plot - this is similar to Figure 7 (a)

```python
# Nyquist plot of the impedance
plt.plot(np.real(Z_exact), -np.imag(Z_exact), linewidth=4, color="black", label="exact")
plt.plot(np.real(Z_exp), -np.imag(Z_exp), "o", markersize=10, color="red", label="synth exp")
plt.plot(np.real(Z_exp[20:60:10]), -np.imag(Z_exp[20:60:10]), 's', markersize=10, color="black")

plt.rc('text', usetex=True)
plt.rc('font', family='serif', size=15)
plt.rc('xtick', labelsize=15)
plt.rc('ytick', labelsize=15)
plt.legend(frameon=False, fontsize = 15)
plt.axis('scaled')

plt.xticks(range(10, 70, 10))
plt.yticks(range(0, 60, 10))
plt.gca().set_aspect('equal', adjustable='box')
plt.xlabel(r'$Z_{\rm re}/\Omega$', fontsize = 20)
plt.ylabel(r'$-Z_{\rm im}/\Omega$', fontsize = 20)
# label the frequency points
plt.annotate(r'$10^{-2}$', xy=(np.real(Z_exp[20]), -np.imag(Z_exp[20])),
             xytext=(np.real(Z_exp[20])-2, 10-np.imag(Z_exp[20])),
             arrowprops=dict(arrowstyle="-",connectionstyle="arc"))
plt.annotate(r'$10^{-1}$', xy=(np.real(Z_exp[30]), -np.imag(Z_exp[30])),
             xytext=(np.real(Z_exp[30])-2, 6-np.imag(Z_exp[30])),
             arrowprops=dict(arrowstyle="-",connectionstyle="arc"))
plt.annotate(r'$1$', xy=(np.real(Z_exp[40]), -np.imag(Z_exp[40])),
             xytext=(np.real(Z_exp[40]), 10-np.imag(Z_exp[40])),
             arrowprops=dict(arrowstyle="-",connectionstyle="arc"))
plt.annotate(r'$10$', xy=(np.real(Z_exp[50]), -np.imag(Z_exp[50])),
             xytext=(np.real(Z_exp[50])-1, 10-np.imag(Z_exp[50])),
             arrowprops=dict(arrowstyle="-",connectionstyle="arc"))
plt.show()
```

## 3) Obtain the optimal hyperparameters of the GP-DRT model by minimizing the negative marginal log likelihood (NMLL)

We constrain the kernel to be a squared exponential, _i.e._

$$
k(\xi, \xi^\prime) = \sigma_f^2 \exp\left(-\frac{1}{2 \ell^2}\left(\xi-\xi^\prime\right)^2 \right)
$$

and modify its two parameters, $\sigma_f$ and $\ell$ as well as the noise level $\sigma_n$. Therefore, the vector of hyperparameters of the GP-DRT is assumed to be $\boldsymbol \theta = \begin{pmatrix} \sigma_n, \sigma_f, \ell  \end{pmatrix}^\top$.

Following the same derivation from the article we can write that

$$
\log p(\mathbf Z^{\rm exp}_{\rm im}|\boldsymbol \xi, \boldsymbol \theta)= - \frac{1}{2} {\mathbf Z^{\rm exp}_{\rm im}}^\top \left(\mathcal L^2_{\rm im} \mathbf K +\sigma_n^2\mathbf I \right)^{-1} \mathbf Z^{\rm exp}_{\rm im} -\frac{1}{2} \log \left| \mathcal L^2_{\rm im} \mathbf K+\sigma_n^2\mathbf I  \right| - \frac{N}{2} \log 2\pi
$$

We will call $L(\boldsymbol \theta)$ the negative (and shifted) MLL (NMLL):
$$
L(\boldsymbol \theta) = - \log p(\mathbf Z^{\rm exp}_{\rm im}|\boldsymbol \xi, \boldsymbol \theta) - \frac{N}{2} \log 2\pi
$$

the experimental evidence is maximized for

$$
\boldsymbol \theta = \arg \min_{\boldsymbol \theta^\prime}L(\boldsymbol \theta^\prime)
$$

The above minimization problem is solved using the `optimize` method provided by `scipy` package

```python
# initialize the parameter for global 3D optimization to maximize the marginal log-likelihood as shown in eq (31)
sigma_n = sigma_n_exp
sigma_f = 5.
ell = 1.

theta_0 = np.array([sigma_n, sigma_f, ell])
seq_theta = np.copy(theta_0)
def print_results(theta):
    global seq_theta
    seq_theta = np.vstack((seq_theta, theta))
    print('{0:.7f}  {1:.7f}  {2:.7f}'.format(theta[0], theta[1], theta[2]))

GP_DRT.NMLL_fct(theta_0, Z_exp, xi_vec)
GP_DRT.grad_NMLL_fct(theta_0, Z_exp, xi_vec)
print('sigma_n,   sigma_f,   ell')

# minimize the NMLL L(\theta) w.r.t sigma_n, sigma_f, ell using the Newton-CG method as implemented in scipy
res = minimize(GP_DRT.NMLL_fct, theta_0, args=(Z_exp, xi_vec), method='Newton-CG', \
               jac=GP_DRT.grad_NMLL_fct,  callback=print_results, options={'disp': True})

# collect the optimized parameters
sigma_n, sigma_f, ell = res.x
```

    sigma_n,   sigma_f,   ell
    0.8903599  5.0014151  1.0120875
    0.8136354  5.0035337  1.0291912
    0.8291863  5.0357867  1.2588675
    0.8303934  5.0832369  1.2117785
    0.8304436  5.2060775  1.2283659
    0.8305175  5.3874489  1.2524148
    0.8305141  5.4069050  1.2546655
    0.8305265  5.4070743  1.2546849
    0.8305263  5.4070861  1.2546867
    Optimization terminated successfully.
             Current function value: 53.657989
             Iterations: 9
             Function evaluations: 11
             Gradient evaluations: 89
             Hessian evaluations: 0

## 4) Core of the GP-DRT

### 4a) Compute matrices
Once we have identified the optimized parameters we can compute $\mathbf K$, $\mathcal L_{\rm im} \mathbf K$, and $\mathcal L^2_{\rm im} \mathbf K$, which are given in equation `(18)` in the article

```python
K = GP_DRT.matrix_K(xi_vec, xi_vec, sigma_f, ell)
L_im_K = GP_DRT.matrix_L_im_K(xi_vec, xi_vec, sigma_f, ell)
L2_im_K = GP_DRT.matrix_L2_im_K(xi_vec, xi_vec, sigma_f, ell)
Sigma = (sigma_n**2)*np.eye(N_freqs)
```

### 4b) Factorize the matrices and solve the linear equations
We are computing
$$
\boldsymbol{\gamma}|\mathbf Z^{\rm exp}_{\rm im}\sim \mathcal N\left( \boldsymbol \mu_{\gamma|Z^{\rm exp}_{\rm im}}, \boldsymbol \Sigma_{\gamma| Z^{\rm exp}_{\rm im}}\right)
$$

using
$$
\begin{align}
\boldsymbol \mu_{\gamma|Z^{\rm exp}_{\rm im}} &= \mathcal L_{\rm im} \mathbf K\left(\mathcal L^2_{\rm im} \mathbf K+\sigma_n^2\mathbf I\right)^{-1}\mathbf Z^{\rm exp}_{\rm im} \\
\boldsymbol \Sigma_{\gamma| Z^{\rm exp}_{\rm im}} &= \mathbf K-\mathcal L_{\rm im} \mathbf K\left(\mathcal L^2_{\rm im} \mathbf K+\sigma_n^2\mathbf I\right)^{-1}\mathcal L_{\rm im} \mathbf K^\top
\end{align}
$$

The key ingredient is to do Cholesky factorization of $\mathcal L^2_{\rm im} \mathbf K+\sigma_n^2\mathbf I$

```python
# the matrix $\mathcal L^2_{\rm im} \mathbf K + \sigma_n^2 \mathbf I$ whose inverse is needed
K_im_full = L2_im_K + Sigma

# Cholesky factorization, L is a lower-triangular matrix
L = np.linalg.cholesky(K_im_full)

# solve for alpha
alpha = np.linalg.solve(L, Z_exp.imag)
alpha = np.linalg.solve(L.T, alpha)

# estimate the gamma of eq (21a), the minus sign, which is not included in L_im_K, refers to eq (65)
gamma_fct_est = -np.dot(L_im_K.T, alpha)

# covariance matrix
inv_L = np.linalg.inv(L)
inv_K_im_full = np.dot(inv_L.T, inv_L)
np.diag(np.dot(inv_K_im_full, K_im_full))

# estimate the sigma of gamma for eq (21b)
cov_gamma_fct_est = K - np.dot(L_im_K.T, np.dot(inv_K_im_full, L_im_K))
sigma_gamma_fct_est = np.sqrt(np.diag(cov_gamma_fct_est))
```

### 4c) Plot the obtained DRT against the analytical DRT

```python
# plot the DRT and its confidence region
plt.semilogx(freq_vec_plot, gamma_fct_plot, linewidth=4, color="black", label="exact")
plt.semilogx(freq_vec, gamma_fct_est, linewidth=4, color="red", label="GP-DRT")
plt.fill_between(freq_vec, gamma_fct_est-3*sigma_gamma_fct_est, gamma_fct_est+3*sigma_gamma_fct_est, color="0.4", alpha=0.3)
plt.rc('text', usetex=True)
plt.rc('font', family='serif', size=15)
plt.rc('xtick', labelsize=15)
plt.rc('ytick', labelsize=15)
plt.axis([1E-4,1E4,-5,25])
plt.legend(frameon=False, fontsize = 15)
plt.xlabel(r'$f/{\rm Hz}$', fontsize = 20)
plt.ylabel(r'$\gamma/\Omega$', fontsize = 20)
plt.show()
```

### 4d) Predict the $\gamma$ and the imaginary part of the GP-DRT impedance

This part is explained in Section `2.3.3` of the main article

```python
# initialize the imaginary part of impedance vector
Z_im_vec_star = np.empty_like(xi_vec_star)
Sigma_Z_im_vec_star = np.empty_like(xi_vec_star)

gamma_vec_star = np.empty_like(xi_vec_star)
Sigma_gamma_vec_star = np.empty_like(xi_vec_star)

# calculate the imaginary part of impedance at each $\xi$ point for the plot
for index, val in enumerate(xi_vec_star):
    xi_star = np.array([val])

    # compute matrices shown in eq (18), k_star corresponds to a new point
    k_star = GP_DRT.matrix_K(xi_vec, xi_star, sigma_f, ell)
    L_im_k_star = GP_DRT.matrix_L_im_K(xi_vec, xi_star, sigma_f, ell)
    L2_im_k_star = GP_DRT.matrix_L2_im_K(xi_vec, xi_star, sigma_f, ell)
    k_star_star = GP_DRT.matrix_K(xi_star, xi_star, sigma_f, ell)
    L_im_k_star_star = GP_DRT.matrix_L_im_K(xi_star, xi_star, sigma_f, ell)
    L2_im_k_star_star = GP_DRT.matrix_L2_im_K(xi_star, xi_star, sigma_f, ell)

    # compute Z_im_star mean and standard deviation using eq (26)
    Z_im_vec_star[index] = np.dot(L2_im_k_star.T, np.dot(inv_K_im_full, Z_exp.imag))
    Sigma_Z_im_vec_star[index] = L2_im_k_star_star-np.dot(L2_im_k_star.T, np.dot(inv_K_im_full, L2_im_k_star))

    # compute Z_im_star mean and standard deviation
    gamma_vec_star[index] = -np.dot(L_im_k_star.T, np.dot(inv_K_im_full, Z_exp.imag))
    Sigma_gamma_vec_star[index] = k_star_star-np.dot(L_im_k_star.T, np.dot(inv_K_im_full, L_im_k_star))
```

### 4e) Plot the imaginary part of the GP-DRT impedance together with the exact one and the synthetic experiment

```python
plt.semilogx(freq_vec_star, -np.imag(Z_exact), ":", linewidth=4, color="blue", label="exact")
plt.semilogx(freq_vec, -Z_exp.imag, "o", markersize=10, color="black", label="synth exp")
plt.semilogx(freq_vec_star, -Z_im_vec_star, linewidth=4, color="red", label="GP-DRT")
plt.fill_between(freq_vec_star, -Z_im_vec_star-3*np.sqrt(abs(Sigma_Z_im_vec_star)), -Z_im_vec_star+3*np.sqrt(abs(Sigma_Z_im_vec_star)), alpha=0.3)
plt.rc('text', usetex=True)
plt.rc('font', family='serif', size=15)
plt.rc('xtick', labelsize=15)
plt.rc('ytick', labelsize=15)
plt.axis([1E-4,1E4,-5,25])
plt.legend(frameon=False, fontsize = 15)
plt.xlabel(r'$f/{\rm Hz}$', fontsize = 20)
plt.ylabel(r'$-Z_{\rm im}/\Omega$', fontsize = 20)
plt.show()
```

```python

```