<h1 style='font-size:4rem;color:orange;'>Math 267 Project #4

# Dynamical Systems

In simplest terms a Dynamical System is a point moving in space. We call this space a phase space
and the path for the point a trajectory or orbit. The point could represent an actual particle that is moving in physical space or it could represent the instantaneous state of our Dynamical System. The trajectory thus illustrates how the system evolves with time. The evolution follows a vector field. The trajectories could be fixed points, spirals, closed loops that represent periodic behavior or the trajectory could be a chaotic orbit. Both the predator-prey system and the mass-spring system are examples of Dynamical Systems. For the predator-prey system a point the phase space (the xy plane) represents the current population of the foxes and rabbits. For the mass spring system a point represents the current position and velocity of the mass. Thus for this case the instantaneous energy in the system is a function of the point. More on this later..

---

# The Pendulum

The equation of motion for a driven damped pendulum with mass 1 and arm length 1 is:

$$ \theta''(t) + b\; \theta'(t) + g \sin(\theta) = A \sin(\omega_f  t)$$

where $\theta$ is the angular displacement for the pendulum. The forcing term is a sinusoidal driving force with an amplitude of A and a period of $2 \pi / \omega_f$. We derive this equation in class. Please refer to your text for further details. It is assumed that the pendulum is attached to a wall such that it is free to rotate through 360 degrees, thus $\omega + 2\pi$ and $\omega$ correspond to the same pendulum position. We can reduce this second order equation to a system by introducing the variable $\omega = \theta’$ , where $\omega$ represents the angular velocity for the pendulum motion. For convenience let us assume that units are chosen so that g = 1. The system is written as:

\begin{align}
\theta'(t) &=  \; \omega(t) \\
\omega'(t) &= -b \omega(t) + A \sin(\omega_ft)
\end{align}

## 1. The unforced and undamped Pendulum.

For the case where there is both no damping or external forcing, the equation for the pendulum
becomes:

$$\theta'(t) + \sin(\theta) = 0$$

This equation while simple in appearance cannot be solve analytically. For small $\theta$ we could use the small angle approximation, namely $\sin(\theta) ≈\theta$, and the equation becomes linear. Otherwise numerical methods are required to solve the system. We can however get an idea of what the solutions should look like. With no damping or means to dissipate the energy, the energy E will be constant along all solutions of the system. The total energy in the system, the potential and kinetic energy, is given by the energy function:

$$E(\theta,\omega) = \frac{1}{2} \omega^2 + 1 - \cos(\theta)$$.

Each trajectory in the phase plane is confined to one or another of the level curves for the energy function. So by plotting the contour map for the energy function we get an idea of what the trajectories should look like. A plot of the contour map for the energy function is shown in the figure below. If we add directions to the level curves we have a phase portrait for the pendulum system. Note that only a unique trajectory can pass through each point in the phase plane. Why?

To determine the direction for these trajectories we simply look at the vector field for the system. Writing the unforced, undamped equation as a system gives

\begin{align}
\theta'(t) &=  \; f(\theta, \omega) =\omega \\
\omega'(t) &= \;g(\theta,\omega)= -\sin(\omega).
\end{align}

So our vector field is:

$$\mathbf{F}(θ,ω) = (f(θ,ω),g(θ,ω)) = (ω,-sin(θ)).$$

Evaluating the vector field at the point (0,1) gives $F(0,1) = (1,0)$ and the trajectory is thus pointing to the right at this point meaning that $\theta$ is increasing. Repeating this for other points in the phase plane allows us to attach direction arrows to the contour curves. The phase portrait for the undamped pendulum is shown in the figure below.

The solution curves encircling the origin where $-π < θ(t) < π$ correspond to the pendulum oscillating about the downward rest position. The solution curve at the top of the phase portrait corresponds to the pendulum spinning forever in the clockwise direction. The opposite is true for the trajectory at the bottom of the phase portrait.

---

## 2. The unforced and damped Pendulum

For the case where there is damping but no external forcing, the equations for the pendulum becomes:

\begin{align}
\theta'(t) &=  \; f(\theta, \omega) =\omega \\
\omega'(t) &= \;g(\theta,\omega)= -\sin(\theta)-b\omega.
\end{align}

The equilibrium points for this plane autonomous system are determined by setting:

$$\mathbf{F}(θ, ω) = 0 \;or f (θ, ω) = g(θ, ω) = 0$$

Solving gives the points (0, 0) and (0, π).

These points correspond to the pendulum being positioned either in the downward rest position or straight up (θ=π). To determine the stability of these points we need to linearize the system by computing the Jacobian matrix. See 10.3 in the text for further information on the Jacobian. The Jacobian matrix A for the system is defined as a matrix of partial derivatives.

\begin{align} A=
        \begin{pmatrix}
        \frac{\partial}{\partial \theta}f(\theta,\omega)
         & \frac{\partial}{\partial \omega}f(\theta,\omega)\\
        \frac{\partial}{\partial \theta}g(\theta,\omega)
         & \frac{\partial}{\partial \omega}g(\theta,\omega)
         \end{pmatrix}
        \end{align}

For the undriven, damped pendulum system the Jacobian is:

\begin{align} A=
        \begin{pmatrix}
        0 & 1\\
        -\cos(\theta)
         & -b
         \end{pmatrix}
        \end{align}

Now to determine the stability of an equilibrium point, the Jacobian is evaluated at that point and the eigenvalues are tested. For example, for the equilibrium point (0,0) (the rest position) the linearized system is:

\begin{align}
        \begin{pmatrix}
        \theta'(t) \\
        \omega'(t)
        \end{pmatrix}=
        \begin{pmatrix}
        0 & 1 \\
        -1 & -b
        \end{pmatrix}\cdot
        \begin{pmatrix}
        \theta(t) \\
        \omega(t)
        \end{pmatrix}
\end{align}

With a damping constant b = 0, the eigenvalues are pure imaginary and the origin is a “center” thus the trajectories will be closed loops with the origin as their center. For b>0 both eigenvalues will have negative real parts and the point (0,0) will be an attractor. Which is what we expect.

---

## 3. The forced and damped Pendulum

For the case where there is damping and external forcing, the equations for the pendulum becomes:

\begin{align}
\theta'(t) &=\omega \\
\omega'(t) &= \; -\sin(\theta)-b\omega + A \sin(\omega_f t).
\end{align}

This is where things get interesting and we will see that for certain parameter values the pendulum will exhibit chaotic behavior. Sensitive dependence on initial conditions is a hallmark of chaos. The plot below shows the response of a pendulum to the following initial conditions. For s(t) the initial angular displacement θ(0) = 0 and for u(t) the initial displacement is θ(0) = 0.001. We notice the curves diverge considerably after about 40 seconds. This is sensitive dependence on initial conditions.

---

### Run the cell below to import necessary libraries.

```python
# import libraries

import numpy as np
from numpy import sin,pi
import matplotlib.pyplot as plt
from IPython.display import Image

# uncomment the line below if you are running a macbook
#%config InlineBackend.figure_format ='retina'
```

## Exericse 1.a

For this exercise you will approximate the solution of an unforced, undamped defined by the system:

\begin{align}
\theta'(t) &=  \;  \omega \\
\omega'(t) &= \; -\sin(\theta).
\end{align}

You are to create two plots of $\theta$ versus t. For the first plot solve the sytem with initial condition:

$$
\begin{align}
\theta(0) &=  \;0.1 \\
\omega(0) &= \; 0
\end{align}
$$

For the second plot solve the system with the initial conditions:

$$
\begin{align}
\theta(0) &=  \;1 \\
\omega(0) &= \; 0
\end{align}
$$

Use the improved Euler method( Δt=h = 0.1) and a time interval of 12 seconds.

## Complete the code cell below to solve the sytstem.

```python
# define the slope functions

def f(theta,omega,t=None):
    return omega

def g(theta,omega,t=None):
    return -np.sin(theta)

h = 0.1  # set delta t
t = np.arange(0,12,h)

# Initialize arrays to store the results.

theta = np.zeros_like(t)
omega = np.zeros_like(t)

# Set initial conditions:

theta[0] = 0
omega[0] = 0.1

# implement the Improved Euler's method

# Your code goes here, look at project #2
```

### Execute the cell below to graph your results.
You should see a  sinusoid as the pendulum is set into infinite oscillation (no damping).

```python
plt.figure(figsize=(8,5))
plt.plot(t,theta)
plt.grid()
plt.xlabel('t')
plt.ylabel(r'$\theta$');
plt.plot(2*np.pi,0,'o')
plt.title(r"Exercise #1a. $\theta(0)=0\;,\omega(0)=0.1$");
```

### Execute the cell below to graph the trajectory in the phase plane.
You should see a  small circle as the pendulum is set into infinite oscillation (no damping).

```python
plt.figure(figsize=(8,5))
plt.plot(theta,omega,linewidth=2)
plt.xlim(-2*pi,2*pi)
plt.ylim(-4,4)
plt.grid()
plt.xlabel(r'$\theta$-displacement')
plt.ylabel(r'$\omega$');
plt.xticks([-2*pi,-pi,0,pi,2*pi],['-2$\pi$','$-\pi$',0,'$\pi$','$2\pi$'])
plt.title(r"Trajectory for pendulum-Exercise #1a. $\theta(0)=0\;,\omega(0)=0.1$");
```

 ## Exercise 1.b

 Repeat the steps above for $\theta(0)=0 \;\text{and}\; \omega(0) = 1.0$ Copy and paste cells above and make the necessary modifications. Make a plot of $\theta$ versus t. Also plot the trajectory in the phase plane. Make the appropriate change to the title of your plot.

```python

```

```python

```

## Exericse 2.

Use the improved Euler’s method to approximate and graph the solution of the above system with the initial conditions θ(0) = 0 and ω(0) =2.5. Create a θ versus t and a phase plot. Your $\theta$ versus t plot will not be a sinusoid. Again plot your results for the interval [0,12].

```python

```

```python

```

## Exercise 3.

Use the improved Euler’s method to approximate and graph the solution of the above system with a damping constant of b = 0.5. Use the initial conditions θ(0) = 1 and ω(0) =0. Create a θ versus t plot
and a phase plot θ versus ω. Give each plot the following title: "Exercise #3 damped pendulum b=0.5".

```python
# Your code here.
```

```python

```

```python

```

## Exercise 4

In the following exercises we will approximate and graph the solutions for the forced,damped pendulum system with different damping constants. For each exercise we use  A = 1.5, ωf = 2/3, and initial condition (θ(0) ,ω(0)) = (0,0).

## Exercise 4.a

Use the improved Euler method to solve the system with b = 0.8, h= Δt=0.1 and time interval of \[0,90]. Make a $\theta$ versus t plot and a phase plot. Give your graphs appropriate titles.

Note: When creating the phase plot we want to omit the transient from the plot and only view the steady state response.  This can be accomplished by plotting just the last third of our data values as follows:

```python

plt.plot(theta[600:900],omega[600:900],linewidth=1)

```

Also note a line width of 1 is recommended for the phase plot.

```python
# define the slope functions

b=0.8

def f(theta,omega,t):
    return omega

def g(theta,omega,t):
    return -np.sin(theta) -b*omega + 1.5*np.sin(2/3*t)

# Your code starts here....

```

### Execute the cell below to graph your results.

```python
plt.figure(figsize=(8,5))
plt.plot(t,theta)
plt.grid()
plt.xlabel('t')
plt.ylabel(r'$\theta$');
plt.title(r"Exercise #4a. b =0.8");
```

### Execute the cell below to see the trajectory in the phase plane for this problem.

```python
plt.figure(figsize=(10,7))
plt.plot(theta[600:900],omega[600:900],linewidth=1)
plt.xlim(-2*pi,2*pi)
plt.ylim(-4,4)
plt.grid()
plt.xlabel(r'$\theta$-displacement')
plt.ylabel(r'$\omega$');
plt.xticks([-2*pi,-pi,0,pi,2*pi],['-2$\pi$','$-\pi$',0,'$\pi$','$2\pi$'])
plt.title(r"Exercise #4a. b =0.8");
```

## Exercise 4.b

Change b = 0.735 and repeat the steps above. Use h = 0.1 and a time interval of \[0,90].

```python
# Place your code here to solve the system in this cell.
```

```python
# Place your code here to make a theta versus t plot in this cell.
```

### Execute the cell below to see the trajectory in the phase plane for this problem.

```python
plt.figure(figsize=(10,7))
plt.plot(theta[600:900],omega[600:900],linewidth=1)
plt.xlim(-2*pi,2*pi)
plt.ylim(-4,4)
plt.grid()
plt.xlabel(r'$\theta$-displacement')
plt.ylabel(r'$\omega$');
plt.xticks([-2*pi,-pi,0,pi,2*pi],['-2$\pi$','$-\pi$',0,'$\pi$','$2\pi$'])
plt.title(r"Exercise #4b. b =0.735");
```

## Exercise 5.

Write the necessary code to  recreate the plot that demonstrates sensive dependence on intial condtions. See \#3 above The plot labeled s(t) has an initial value of 0.0 and u(t) has intial value of 0.001. The sinusoid input has an amplitude of 1 and omega = 2/3. The damping for the spring is 0.05. Plot the output for the range (0,80).

For this exercise you will be solving the system twice.  Just cut and past code above make the necessary modificatons and in the last line of the cell enter the Python
command.

```python
s=theta;

```
Do the same for a second cell and again end with the command:

```python
u=theta;

```

```python
# Your code here.
```

```python
# Your code here.
```

### Execute the cell below to see a plot of your results

```python
plt.figure(figsize=(8,5))
plt.plot(t,s,label="$s(t)$")
plt.plot(t,u,label="$u(t)$")
plt.grid()
plt.xlabel('t')
plt.legend()
plt.title(r"Exercise #5");
```

## Answer the questions below.  You can edit the text cell to enter your answers.
1. For exercises 1.a and 1.b compute the period of oscillation using the small angle approximation.  Compare this to the value obtained from the graphs and explain the results.

2. Explain the results of exercise 2, that is, explain the shape of the graphs.

3. Look at the phase plots for exercise \#4. Outputs for both b = 0.8 and b = 0.735 are periodic.  How does the period of the output (b=0.735) compare to the output for b=0.8.

To see your input along with the output add the line to the cell that creates the theta versus t plot.

```python
plt.plot(t,1.5*sin(2/3*t))

```

4. In exercise 5. we see that a very small change in initial conditions produce a dramatic change in output.  Will the same phenomenon occur with the spring-mass system?  In other words can our spring-mass system exhibit sensitive dependence on initial conditions?


# Determining the Exponent of Oscillatory Matrices

## Introduction
A matrix $A\in\R^{n\times n}$ is called **oscillatory** if it is invertible and TN (I-TN) and there exists a postive integer $q$ such that $A^q$ is TP. The least positive integer $r=r(A)$ such that $A^r$ is TP is called the *exponent* of the oscillatory matrix $A$. Determining the exponent of an oscillatory matrix is an important task.

Oscillatory matrices admit Successive Elementary Bidiagonal (SEB) factorizations that are associated with planar networks. To show that $A^r$ is TP, for $A$ oscillatory, it is enough to show that all corner minors of $A^r$ are positive. Equivalently, it is enough to show that for each corner minor, at least one family of vertex-disjoint paths, connecting the minor's source and destination indexes, exists in the planar network associated with the SEB factorization of $A^r$.

Here, we implemented few tools that can be used to determine the exponent of an oscillatory matrix based on the families of vertex-disjoing paths. For more details on oscillatory matrices, SEB factorization and associated planar networks, please refer to the paper: [On the Exponent of Several Classes of Oscillatory Matrices](https://arxiv.org/abs/1910.10709).

## SEB Factorization and Planar Networks
Let $L_i(q)\in\R^{n\times n}$, $q\ge 0$, denote a matrix with $1$ along the diagonal, $q$ in entry $(i,i-1)$ and $0$ elsewhere, and let $U_i(q):=(L_i(q))^T$. $q$ is called the parameter of $L_i(q)$. Note that $L_i(0)=I$ for all $i$. Then an oscillatory matrix $A$ can be written as $A=LDU$, with (we omit the parameter values) $L=[L_n\cdots L_2]\cdots [L_n]$, $U=[U_n]\cdots [U_2\cdots U_n]$, and $D$ diagonal with positive diagonal entries. This is called the *SEB factorization* of the oscillatory matrix $A$.
For example, for $n=5$, an oscillatory matrix $A\in\R^{5\times 5}$ can be written as $A=LDU$ with
$$
\begin{equation}
L:=[L_5L_4L_3L_2][L_5L_4L_3][L_5L_4][L_5],
\label{eq:L}
\tag{1}
\end{equation}
$$
and
$$
\begin{equation}
U:=[U_5][U_4U_5][U_3U_4U_5][U_2U_3U_4U_5].
\label{eq:U}
\tag{2}
\end{equation}
$$

Associated with the SEB factorization is a *planar network*. Each $L_i(q)$ [$U_i(q)$] is described by a diagram consisting of a diagonal directed edge (with weight $q$) from index $i$ to index $i-1$ [from index $i-1$ to index $i$], and $n$ horizontal directed edges (with weight of $1$). The diagram corresponding to the matrix $D$ consists of $n$ horizontal directed edges with weights $d_{ii}$. The planar network associated with an SEB factorization is established by concatenating, in order, the diagrams corresponding to all $L_i$, $D$, and all the $U_i$ matrices.
The following planar network is associated with the SEB factorization given in $\eqref{eq:L}$ and $\eqref{eq:U}$:

Given an oscillatory matrix $A=LDU$, we implement the computation of the family of vertex-disjoint paths corresponding to all lower-left [upper-right] corner minors of $L$ [$U$]. The paths corresponding to $U$ are determined by computing the paths of all lower-left corner minors of $U^T$, and then converting them to the paths of all upper-right corner minors of $U$.
Specifically, for each lower-left corner minor, we compute the minimum number of copies of $A$ such that a family of vertex-disjoint paths can be realized. In addition, we display the corresponding family for each corner minor, for both $L$ and $U$.

We use a **triangle graph** that is derived from the planar network associated with the left-hand side of the SEB factorization of I-TN matrices to deduce the families of paths.

## Triangle Graph
A triangle graph consists of $n-1$ columns of cells ordered left to right. The first (left-most) column represents the product $[L_n\cdots L_2]$, the second column represents the product $[L_{n-1}\cdots L_2]$, and so on, where the last column represents $[L_n]$. In each column, the top cell represents the existence/absence of $L_n$ (in the product $[L_n\cdots L_2]$), that is, it represents the boolean value $\ell_n>0$, where $\ell_n$ is the parameter of $L_n$. The second-top cell represents the existence/absence of $L_{n-1}$, and so on, where the bottom cell represents the existence/absence of $L_{2}$. Similarly for the other columns. Thus, the first column contains $n-1$ cells, the second $n-2$ cells and so on, where the last column contains a single cell.

We say that a triangle graph represents the *signature* of the left-hand side SEB factorization, that is, it represents $\ell_i>0$ rather than the value $\ell_i$ for all $i$. Thus, an oscillatory matrix $A=LDU$, is associated with two triangle graphs: one corresponding to $L$ and one to $U^T$.

The graph also contains indexes on the left and in the bottom. These represent the source and destination indexes, respectively, of a lower-left corner minor.
The indexes of a lower-left corner minor are denoted by $s|d$, where $s$ [$d$] denotes the source [destination] indexes. For example, the lower-left corner minor indexes for $n=5$ (excluding the determinant) are:
1. $5|1$. In the context of paths along the triangle graph, this corresponds to a family of one path from the source index 5 to the destination index 1.
2. $4,5|1,2$. This corresponds to a family of two vertex-disjoint paths: a path from 4 to 1, and a path from 5 to 2.
3. $3,4,5|1,2,3$. This corresponds to a family of three vertex-disjoint paths: a path from 3 to 1, from 4 to 2, and from 5 to 3.
4. $2,3,4,5|1,2,3,4$. This corresponds to a family of four vertex-disjoint paths: a path from 2 to 1, a path from 3 to 2, a path from 4 to 3, and a path from 5 to 4.

Below is an example of a triangle graph for $n=5$. Here, "O" indicates the existence
of the corresponding $L_i$ matrix, and "X" the absence.
The first column in the triangle graph corresponds to $L_5L_4L_3L_2$, the second to $L_5L_4L_3$,
the third to $L_5L_4$ and the last column to $L_5$. Recall that in each column in the graph, the top sign (i.e. "O" or "X")
corresponds to $L_n$ in $[L_nL_{n-1}....L_i]$.

The following triangle graph corresponds to $L=[L_5L_3L_2][L_5L_3][L_4][L_5]$:

***
                       5 O
                       4 X O
                       3 O X X
                       2 O O O O
                         1 2 3 4

***

This triangular graph corresponds to the left-hand side of the following planar network:

## Vertex-Disjoint Paths
A path in the triangle graph is defined by a sequence of 2D points along the graph. We
start at a source index on the left, and move **either down or diagonally** along the graph.
Moving diagonally in the triangle graph corresponds to a horizontal path in the planar network.
Moving down in the graph corresponds to moving down along an edge in the planar network.

The flow along the cells in the triangle diagram is done as follows:
* At a "O" cell, we move down if we have yet to reach the destination index, otherwise we move diagonally.
* At a "X" cell, we move diagonally.

We stop once we reach the destination index.

The following figure displays the triangle graph given above with the possible flow from each node. That is, both diagonal and down-pointing edges emerge from a "O" cell, whereas only a diagonal edge emerges from a "X" cell.

A path in the triangle graph is represented by a list of 2D points (which are lists by themself
of two numbers: the y-axis value and the x-axis value).
For example, the path that starts in the source index 5 (i.e. in the 2D point [5,1]) in the
graph above and ends in the destination index 3 (i.e. in the 2D point [1,3]) is:

 [[5,1],[4,1],[3,2],[2,3],[1,3]].

Note that in this case, this is the minimal destination index that the
source index 5 ([5,1]) can reach in the graph. This path corresponds to the following steps:

1. Starting from [5,1] corresponds to starting in the cell that corresponds to the index 5 (the top cell of the first column). This cell contains an "O" sign.
2. Moving down to the "X" sign below it (i.e. to [4,1]) corresponds to using the diagonal edge in $L_5$ in the product $[L_5L_3L_2]$  (in the planar network).
3. Moving diagonally to the "X" sign in the second column (i.e. to [3,2]) corresponds to using an horizontal edge (in the planar network).
4. Moving diagonally to the "X" sign in the third column (i.e. to [2,3]) corresponds to using an horizontal edge (in the planar network).
5. Moving down to [1,3] corresponds to using the diagonal edge in $L_4$ in $[L_4]$. At that point we reached the destination index.

Given a 2D point in the triangle graph $[q_y, q_x]$ the corresponding index in the planar network
is $q_y+q_x-1$. Thus, the path above corresponds to the following path in the planar network:

  [5, 4, 4, 4, 3].

In cases where a destination can not be reached (for example,
in the example above, the source [5,1] (i.e. index 5) can not reach the destination [1,1] (i.e. index 1)) an additional
"copy" of the triangle graph is used. This is repeated until a path is realized.

Consider the left corner minor indexes $5|1$ and the triangle diagram above. Since a path starting from the source index 5 can only terminate in the minimal destination index 3, which is larger than the required destination index 1, we say that in a single copy of the graph, index 5 can reach index 3. We then use another copy of the triangle graph, however this time we start from the source index 3. In this copy, the path is $[[3,1],[2,1],[1,1]]$, that is in the second copy of the graph index 3 can reach index 1. Thus, we require two copies of the graph for index 5 to reach index 1, and the corresponding path is represented as a list of two lists, the first is the path in the first copy and the second is the path in the second copy:

$$[ [[5,1],[4,1],[3,2],[2,3],[1,3]],  [[3,1],[2,1],[1,1]] ].$$

For a general lower-left corner minor indexes $n-x+1,\cdots, n|1,\dots,n$, ($x\in\{1,\dots,n-1\}$), we start by trying to realize a path from the smallest index, i.e. a path from $n-x+1$ to $1$. We then move to realizing a path from $n-x+2$ to $2$, etc. For a given source index, once a path is generated in a single copy of the triangle diagram (which doesn't necessarily terminates in the required destination index), the cells above the path's cells are marked by "X". This is done to insure that all paths are vertex-disjoint paths. However, when using a new copy of the triangle diagram, we revert back to the original diagram without the added "X" marks.

In the context an oscillatory matrix $A=LDU$, we use the triangle graph to compute the families of
vertex-disjoint paths corresponding to all lower-left corner minors of $L$ and $U^T$, and thus deduce
the exponent of $A$.

## Imports

```python
import numpy as np

import TpTnOsc.utils as tpf
import TpTnOsc.osc_exp as oscp
```

## An example from the description section above
This is the example used above in the description section for $n=5$.

```python
%matplotlib inline
import matplotlib.pyplot as plt
from IPython.display import display
''' Define the input which is a list of L_i with positive parameters.
Here, L_4 is missing from [L_5L_4L_3L_2], L_4 is missing from
[L_5L_4L_3], L_5 is missing from [L_5L_4] and L_5 is not missing from [L_5]
'''
mat_info = [[5,3,2], [5,3], [4], [5]]
# -----------------------------------------------------------------------------
# create a triangle graph object
G_L = oscp.Trig_G(info=mat_info)
# display the triangle graph
print("Triangle graph corresponding to mat_info:")
G_L.show_me()
# can also draw the graph
print("A draw of the triangle graph:")
plt1, ax = plt.subplots(figsize=(8,8))
G_L.draw_me(ax, nd_offset=[0.2, 0.3])
ax.margins(.1, .1)
display(plt1)
plt.close()
# check if it corresponds to a rhs of the SEB of an I-TN matrix
print(f"\nGraph correponds to rhs of I-TN matrix: {G_L.valid_ITN}")
# create a lower-left corner minor object based on the triangle graph object
LC = oscp.LLC_families(G_L, name='L')
# run the paths computation
LC.run(verbose=False)
# display paths and number of copies results
print('\nSource->Destination results (verbose):')
LC.display_results()
print('\nSource->Destination results (compressed):')
LC.display_results(compress=True)
```

## Another example of using Trig_G and LLC_families classes
Here, the input is a list of available $L_i$ matrices in each group in the left-hand side of the SEB factorization.

```python
''' Define the input which is a list of L_i with positive parameters.
Here, L_4 and L_3 are missing from [L_5L_4L_3L_2], L5 and L_4 are missing from
[L_5L_4L_3], L_5 is missing from [L_5L_4] and L_5 is not missing from [L_5]
'''
mat_info = [[5,2], [3], [4], [5]]
# -----------------------------------------------------------------------------

# create a triangle graph object
G_L = oscp.Trig_G(info=mat_info)
# display the triangle graph
print("Triangle graph corresponding to mat_info:")
G_L.show_me()
# check if it corresponds to a rhs of the SEB of an I-TN matrix
print(f"\nGraph correponds to rhs of I-TN matrix: {G_L.valid_ITN}")
# create a lower-left corner minor object based on the triangle graph object
LC = oscp.LLC_families(G_L, name='Q')
# run the paths computation
LC.run(verbose=False)
# display paths and number of copies results
print('\nSource->Destination results:')
LC.display_results()
# can also display the reverse paths and number of copies results
print('\nDestination->Source results:')
LC.display_results(reverse=True)
# can also display raw results
#LC.display_raw_results()
print(f"\nMinimum number of copies to realize all lower-left corner minors = {LC.r}.")

```

    Triangle graph corresponding to mat_info:
     5 O
     4 X X
     3 X X X
     2 O O O O
       1 2 3 4

    Graph correponds to rhs of I-TN matrix: True

    Source->Destination results:
    Q:
    	5|1 (3 copies):
    		[5->4->4->4->3][3->3->2][2->1]
    	4,5|1,2 (3 copies):
    		[4->4->4->3][3->3->2][2->1]
    		[5->5->5->5->4][4->4->4->3][3->3->2]
    	3,4,5|1,2,3 (2 copies):
    		[3->3->2][2->1]
    		[4->4->4->3][3->3->2]
    		[5->5->5->5->4][4->4->4->3]
    	2,3,4,5|1,2,3,4 (1 copies):
    		[2->1]
    		[3->3->2]
    		[4->4->4->3]
    		[5->5->5->5->4]
    Minimum number of copies to realize all families = 3.

    Destination->Source results:
    Q:
    	1|5 (3 copies):
    		[1->2][2->3->3][3->4->4->4->5]
    	1,2|4,5 (3 copies):
    		[2->3->3][3->4->4->4][4->5->5->5->5]
    		[1->2][2->3->3][3->4->4->4]
    	1,2,3|3,4,5 (2 copies):
    		[3->4->4->4][4->5->5->5->5]
    		[2->3->3][3->4->4->4]
    		[1->2][2->3->3]
    	1,2,3,4|2,3,4,5 (1 copies):
    		[4->5->5->5->5]
    		[3->4->4->4]
    		[2->3->3]
    		[1->2]
    Minimum number of copies to realize all families = 3.

    Minimum number of copies to realize all lower-left corner minors = 3.

## An example of using OSC_exp class
Here, the input is an oscillatory matrix.

```python
# define an oscillatory matrix
A = np.array([[3, 1, 0, 0], [1, 4, 1, 0.1], [0.1, 1, 5, 3], [0, 0, 2, 7]])
# -----------------------------------------------------------------------------
print(f"A =\n{A}")
# check that this is indeed an oscillatory matrix
print(f"\nMatrix A is OSC: {tpf.is_osc_from_factorization(A)}")
try:
    osc_cls = oscp.OSC_exp(A) # create an OSC_exp object
except ValueError as ve:
    print(f"{ve}. Aborting...")
else:
    osc_cls.run()  # run it
    osc_cls.display_results()  # display its results
    # or can show paths corresponding to both lower-right and upper-right corner minors
    print('\nShowing paths for both lower-left and upper-right corner minors for both L and U^T:')
    osc_cls.display_results(show_also_reverse=True)
finally:
    for k in osc_cls.G.keys():
        print(f'Triangle graph of {k}:')
        plt1, ax = plt.subplots(figsize=(8,8))
        osc_cls.G[k].draw_me(ax, nd_offset=[0.15, 0.2])
        ax.margins(.1, .1)
        display(plt1)
        plt.close()
    print('Done.')

```

## Another example
Here, for $A=LDU$, we set $U=[U_2\cdots U_n]$ and $L = [L_n\cdots L_2][L_n\cdots L_3]\cdots[L_n]$,
that is, $L$ consists of all $L_i$ matrices in the right-hand side of the SEB factorization, and
$U$ consists of only the last group in the right-hand factorization. It is known that
in this case, $r(A)=n-1$.

```python
n = 8  # set n value
# --------------------------------------------------------------------------------------------
k = tpf.EB_factorization_n2k(n)
valsl = np.ones(k)
valsd = np.ones(n)
valsu = np.r_[np.ones(n-1), np.zeros(k-(n-1))]
A = tpf.compute_matrix_from_EB_factorization(valsl, valsd, valsu)
print(f"A =\n{A}")
# check that this is indeed an oscillatory matrix
print(f"\nMatrix A is OSC: {tpf.is_osc_from_factorization(A)}")
try:
    #osc_cls = OSC_exp(A) # create an OSC_exp object
    osc_cls = oscp.OSC_exp.from_vals_ldu(valsl, valsd, valsu) # can also create from the SEB factorization
except ValueError as ve:
    print(f"{ve}. Aborting...")
else:
    osc_cls.run()  # run it
    osc_cls.display_results(compress=True)  # display its results
finally:
    for k in osc_cls.G.keys():
        print(f'Triangle graph of {k}:')
        plt1, ax = plt.subplots(figsize=(10,10))
        osc_cls.G[k].draw_me(ax, nd_offset=[0.3, 0.4])
        ax.margins(.1, .1)
        display(plt1)
        plt.close()
    print('Done.')

```


# Regret Optimization

Regret optimization is a technique to optimize a portfolio given that one is uncertain about future prospects. Since one must eventually settle for a single portfolio out of all the possible scenarios, the best portfolio in this case is determine by the one that will have the *least* regret. This will be elaborated further

In general, this is a 2-stage optimization. The algorithm works as such,

1. Generate several possible scenarios in the future
    * For each scenario, give it a discrete probability of occurrence
    * These probabilities must sum to 1 across all scenarios!
2. In the first stage, derive the optimal weights for the portfolio for each scenario
    * If there are 5 scenarios, there will be 5 sets of these weights
    * Thus a model with 5 scenarios and 3 assets will yield a 5 x 3 matrix
3. In the second stage, solve for the minimal regret portfolio, these time using the previous sets optimal weights
    * You will receive the final set of of weights here (a 1 x 3 vector)

## Regret

Regret is defined as the cost of choosing one portfolio (which is optimal for a scenario) when another different scenario occurs instead. Functionally, the simplest mathematical formuation is

$$
    D(R(w_s) - R(w_o))
$$

where $D$ is a distance function, $R$ is the profit function, $w_s$ is the optimal weights for scenario $s$ and $w_o$ is the "optimal" weights that was chosen. Thus to solve for the minimal regret function, the exact problem formulation is as listed below

$$
    \underset{w}{\min} \sum_s^S p_s D(R(w_s) - R(w))
$$

where $p_s$ is the probability of scenario $s$ occurring.

### Distance function

The distance function could be anything that makes sense. Some common examples include a linear function, absolute function or quadratic function. Different functions will penalise regret differently and lead to different outcomes. For example, if a portfolio that does not have a wide swings in terms of objectives is desired, a quadratic function will work better than a linear function.

$$
    \begin{align}
    \text{Assuming} \quad R(w) &\geq 1 \quad \forall w \in W \\
    [R(w_s) - R(w_o)]^2 &\geq R(w_s) - R(w_o)
    \end{align}
$$

### Linear Approximations

Suppose we have convex objective and constraints functions for the first stage, we alter the second stage optimization a little. The outcome will be similar but it will give another nice interpretation of the results which will be explained later. Ideally these functions should be **strictly linear**. However, in practice, the differences are usually negligible.

Suppose we want to maximize the returns of the portfolio subject to some CVaR constraints. For simplicity, the returns function will be $R$ and CVaR constraint functions will be $C$. Thus our first stage optimization will be

For every single scenario $s$ in $S$
$$
    \underset{w_s}{\max} R(w_s) \\
    \text{subject to} \\
    C_s(w_s) \geq 0 \\
    \sum_i w_{s, i} = 1 \\
    0 \leq w \leq 1 \quad \forall w \in w_s
$$

From this we would get

$$
    W =
    \begin{bmatrix}
        w_{1, 1} & w_{1, 2} & \dots  & w_{1, n} \\
        \vdots   & \vdots   & \ddots & \vdots   \\
        w_{s, 1} & w_{s, 2} & \dots  & w_{s, n}
    \end{bmatrix}
$$

where $s$ is the number of scenarios adn $n$ is the number of assets. We would then tweak our second (Regret) optimization to

$$
    \underset{a}{\min} \sum_s^S p_s D(R(w_s) - R(W \cdot a)) \\
    \text{subject to} \\
    \sum_s^S a_s = 1
$$

The solution of the problem, $a$, will represent the proportion of importance that is taken from each scenario. Suppose there are 3 scenarios - **X**, **Y**, **Z** and that the final proportion derived is `[0.2, 0.3, 0.5]`. This means that **20%** of the weights are taken from **X**, **30%** from **Y** and **50%** from **Z**. In essence, it weights the importance of each scenario for the final outcome.

To get the final weights, simply do a dot product of $W \cdot a$.

## Example

We will run through the Regret Optimization using both the `RegretOptimizer` and `PortfolioRegretOptimizer` classes. The `PortfolioRegretOptimizer` is a helper class that has several common built-in problems within itself. Underneath the hood, it uses the `RegretOptimizer` for the same operations. The `RegretOptimizer` is the more flexible tool that is useful for modelling more exotic scenarios.

```python
import numpy as np
from muarch.calibrate import calibrate_data

from allopy import OptData, RegretOptimizer
from allopy.datasets import load_monte_carlo
```

```python
# Generate different scenarios
num_assets = 7
num_scenarios = 4
scenario_probability = [0.57, 0.1, 0.14, 0.19]

main_adjustments = np.array([
    [ 0.0061,  0.0601,  0.0466,  0.0051, -0.0066, -0.0013, -0.0026],
    [ 0.0642,  0.0537,  0.0818,  0.0713,  0.0177,  0.0099,  0.0116],
    [-0.0219, -0.0381, -0.0059,  0.0242, -0.0153,  0.0164, -0.001 ],
    [-0.0333, -0.0617, -0.0405, -0.0251,  0.0084,  0.0054, -0.0035]
])

cvar_adjustments = np.array([
    [-0.0436,  0.0586, -0.0081,  0.0051, -0.0078,  0.0135, -0.0081],
    [ 0.0662,  0.079 ,  0.0896,  0.0501, -0.0265, -0.0572,  0.0025],
    [-0.1192, -0.1701, -0.1143,  0.0736, -0.0831,  0.0134, -0.0047],
    [-0.1728, -0.257 , -0.1933, -0.1432,  0.0342,  0.0079,  0.003 ]
])

def make_scenarios(adjustments, truncate=False):
    scenarios = []
    for adj in adjustments:
        data = OptData(load_monte_carlo()[..., :num_assets], 'quarterly')

        if truncate:  # cut for CVaR
            data = data.cut_by_horizon(3)

        scenarios.append(data.calibrate_data(adj))

    return scenarios

main_scenarios = make_scenarios(main_adjustments)
cvar_scenarios = make_scenarios(cvar_adjustments, True)
```

```python
# objective and constraint functions

def make_max_returns_obj_fun(cube: OptData):
    def obj_fun(w):
        return 1e2 * cube.expected_return(w, True)

    return obj_fun

def make_cvar_constraint_fun(cube: OptData, limit: float):
    def cvar_fun(w):
        return 1e3 * (limit - cube.cvar(w, True, 5.0))

    return cvar_fun

# limits and bounds
lb = [0, 0, 0.13, 0.11, 0, 0.05, 0.04]
ub = [1, 0.18, 0.13, 0.11, 1, 0.05, 0.04]

cvar_limit = [-0.34, -0.253, -0.501, -0.562]
```

```python
# optimization model formulation and execution
opt = RegretOptimizer(num_assets, num_scenarios, scenario_probability, sum_to_1=True)
opt.set_bounds(lb, ub)

obj_funcs = []
constraint_funcs = []
for m, c, limit in zip(main_scenarios, cvar_scenarios, cvar_limit):
    obj_funcs.append(make_max_returns_obj_fun(m))
    constraint_funcs.append(make_cvar_constraint_fun(c, limit))

opt.set_max_objective(obj_funcs)
opt.add_inequality_constraint(constraint_funcs)

final_weights = opt.optimize()
```

You can get the summary of the results. The first table show the optimal weight for each scenario. The second shows the proportion of each scenario and is only available when the approx option is set to `True`. The final table shows the *final* optimal weights.

```python
opt.summary()
```

     	     Scenario_1	 Scenario_2	 Scenario_3	 Scenario_4

  Asset_1	   0.2499	     0.3495	     0.1003	     0.0000

  Asset_2	   0.1800	     0.0946	     0.0000	     0.0000

  Asset_3	   0.1300	     0.1300	     0.1300	     0.1300

  Asset_4	   0.1100	     0.1100	     0.1100	     0.1100

  Asset_5	   0.2401	     0.2259	     0.5697	     0.6700

  Asset_6	   0.0500	     0.0500	     0.0500	     0.0500

  Asset_7	   0.0400	     0.0400	     0.0400	     0.0400

  	   Scenario	  Proportion (%)

  0	 Scenario_1	     68.5900

  1	 Scenario_2	     0.0000

  2	 Scenario_3	     0.0000

  3	 Scenario_4	     47.5900

     	     Weight

  Asset_1	 0.1714

  Asset_2	 0.1235

  Asset_3	 0.1510

  Asset_4	 0.1278

  Asset_5	 0.4835

  Asset_6	 0.0581

  Asset_7	 0.0465

The `PortfolioRegretOptimizer` contains a number of common optimization regimes with respect to regret optimization. We can apply the same optimization we did with the `maximize_returns()` method.

```python
from allopy import PortfolioRegretOptimizer

opt = PortfolioRegretOptimizer(main_scenarios,
                               cvar_scenarios,
                               scenario_probability,
                               rebalance=True,
                               sum_to_1=True,
                               time_unit="quarterly")

opt.set_bounds(lb, ub)
opt.maximize_returns(max_cvar=cvar_limit)
opt.summary()
```

     	     Scenario_1	 Scenario_2	 Scenario_3	 Scenario_4

  Asset_1	   0.2499	     0.3494	     0.1003	     0.0000

  Asset_2	   0.1800	     0.0947	     0.0000	     0.0000

  Asset_3	   0.1300	     0.1300	     0.1300	     0.1300

  Asset_4	   0.1100	     0.1100	     0.1100	     0.1100

  Asset_5	   0.2401	     0.2259	     0.5697	     0.6700

  Asset_6	   0.0500	     0.0500	     0.0500	     0.0500

  Asset_7	   0.0400	     0.0400	     0.0400	     0.0400

  	   Scenario	  Proportion (%)

  0	 Scenario_1	     68.5900

  1	 Scenario_2	     0.0000

  2	 Scenario_3	     0.0000

  3	 Scenario_4	     47.5900

     	     Weight

  Asset_1	 0.1714

  Asset_2	 0.1235

  Asset_3	 0.1510

  Asset_4	 0.1278

  Asset_5	 0.4835

  Asset_6	 0.0581

  Asset_7	 0.0465


```python
from sympy import *
```

```python
x1, x2, x3, x4, x5 = symbols('x1 x2 x3 x4 x5')
```

```python
Phi1 = x1 | x2 | x3 | x4 | x5
```

```python
x6, x7, x8, x9, x10 = symbols('x6 x7 x8 x9 x10')
```

```python
Phi2 = x6 | x7 | x8 | x9 | x10
```

```python
Psi = (Phi1 & Phi2) | (Not(Phi1) & Not(Phi2))
```

```python
to_cnf(Psi, simplify=True)
```

    (x1 | x2 | x3 | x4 | x5 | ~x10) & (x1 | x2 | x3 | x4 | x5 | ~x6) & (x1 | x2 | x3 | x4 | x5 | ~x7) & (x1 | x2 | x3 | x4 | x5 | ~x8) & (x1 | x2 | x3 | x4 | x5 | ~x9) & (x10 | x6 | x7 | x8 | x9 | ~x1) & (x10 | x6 | x7 | x8 | x9 | ~x2) & (x10 | x6 | x7 | x8 | x9 | ~x3) & (x10 | x6 | x7 | x8 | x9 | ~x4) & (x10 | x6 | x7 | x8 | x9 | ~x5)

```python
Psi = ( Not(Phi1) | Phi2) & (Not(Phi2) | Phi1)
```

```python
to_cnf(Psi, simplify=True)
```

    (x1 | x2 | x3 | x4 | x5 | ~x10) & (x1 | x2 | x3 | x4 | x5 | ~x6) & (x1 | x2 | x3 | x4 | x5 | ~x7) & (x1 | x2 | x3 | x4 | x5 | ~x8) & (x1 | x2 | x3 | x4 | x5 | ~x9) & (x10 | x6 | x7 | x8 | x9 | ~x1) & (x10 | x6 | x7 | x8 | x9 | ~x2) & (x10 | x6 | x7 | x8 | x9 | ~x3) & (x10 | x6 | x7 | x8 | x9 | ~x4) & (x10 | x6 | x7 | x8 | x9 | ~x5)

```python
Psi = Not(Phi1) | Phi2
```

```python
to_cnf(Psi, simplify=True)
```

    (x10 | x6 | x7 | x8 | x9 | ~x1) & (x10 | x6 | x7 | x8 | x9 | ~x2) & (x10 | x6 | x7 | x8 | x9 | ~x3) & (x10 | x6 | x7 | x8 | x9 | ~x4) & (x10 | x6 | x7 | x8 | x9 | ~x5)

```python
Psi = Phi1 >> Phi2
```

```python
to_cnf(Psi, simplify=True)
```

    (x10 | x6 | x7 | x8 | x9 | ~x1) & (x10 | x6 | x7 | x8 | x9 | ~x2) & (x10 | x6 | x7 | x8 | x9 | ~x3) & (x10 | x6 | x7 | x8 | x9 | ~x4) & (x10 | x6 | x7 | x8 | x9 | ~x5)

```python
y1, y2, y3 = symbols('y1 y2 y3')
```

```python
sign1 = y1 >> Not(y2 & y3)
```

```python
to_cnf(sign1,simplify=True)
```

    ~y1 | ~y2 | ~y3

```python
sign2 = y1 >> (y2 | y3)
```

```python
to_cnf(sign2,simplify=True)
```

    y2 | y3 | ~y1

```python
sign3 = y1 >> ((y2 >> y3)&(y3 >> y2))
```

```python
to_cnf(sign3,simplify=True)
```

    (y2 | ~y1 | ~y3) & (y3 | ~y1 | ~y2)

```python
extension1 = y1 >> (y2 >> y3)
```

```python
to_cnf(extension1,simplify=True)
```

    y3 | ~y1 | ~y2

```python

```


# Attention mechanism from scratch

## Recap

* Before the attention mechanism appeared, if we want to solve a machine translation problem (or any seq-seq problem), we use RNN to compute some representation of a sequence using the encoder and then predict sequence by different RNN using obtained representations.  **Bottleneck problem!**

**Soltution:** to reduce bottleneck problem, we could somehow connect predicted token not only with the previous token but with all elements from the encoder.

### Key concepts of attention

Attention is based on three core objects **query, key, value**.

Attention principles are very naturals and could be interpreted as:

1. Query - what you want to find (decoder)
2. Key - representation of data (encoder)
3. Value - the importance of the object in data (encoder)

The query is what you want to find in a date, key - represents data for query search, and value is information stored in an object. A natural example could be illustrated in the machine translation task.

In this picture, queries from one language have bigger attention for words with the same meaning but in a different language.

## Lets go into details!

### Similarity coefficient
First of all, we need to compute the similarity coefficient between each question and key.

\begin{align}
 cᵢⱼ = f(qᵢ, kⱼ)
\end{align}
where $c$ is similairty coefficient between $i$-query with $j$ key.

Function $f$ could be an arbitrary function that returns the real number and describes two elements' similarities.

### Attention weights
For each $ q_i $, we compute similarity coefficient with whole keys and obtain vector from $ R^{n} $, where n - number of keys.
\begin{align}
 C_i  = [c_{i 1}, c_{i 2} ... c_{i n}]
\end{align}

One of the problems is that elements in $ C_i $ could be on a different scale. So, we want to map the sum into $(0, 1)$ and interpret elements as a probability. To do it, we use the softmax function:

\begin{align}
a_{ij} = \frac{\text{exp}(c_{i j})}{\sum_n \text{exp}(a_{in})}
\end{align}

\begin{align}
A_i  = softmax(C_i) = softmax([c_{i 1}, c_{i 2} ... c_{i n}])
\end{align}

### Attention matrix
Finally we get an attemtion matrix, where each row is softmaxed similarity coefficent of one query with all keys.

\begin{align}
A_i  = [
    A_1,\\
    A_2,\\
    ...\\
    A_{n - 1}\\
    A_n]
\end{align}

### Attention output

The output of the attention block is a weighted sum of values.
\begin{align}
 O_k = \sum^n_{i = 1} a_{k i} v_i = A_k V
\end{align}

## Lets code!

```python
import torch
import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from tqdm.notebook import tqdm
```

### Similarity coefficient
We should create a function that takes two vectors with the same shape and return a real value. The most straightforward measure is the dot product.

### Dot product attention
The core idea is to use a scaled dot product as a similarity coefficient.

\begin{align}
c  =  \frac{\langle \mathbf{q}, \mathbf{k} \rangle}{\sqrt{d}}
\end{align}
The function takes two vectors with the same dimension $q, k \in R^d$, and compute dot product and device by dimension size. Dividing helps reduce std difference for vectors with different size and make the function more general.

```python
torch.manual_seed(0)

d = 256
x = torch.rand(d)
y = torch.rand(d)

def scalar_similarity_coefficient(x,y, d):
    ## TODO write scaled dot product
    return torch.dot(x,y) / np.sqrt(d)

similarity_score = scalar_similarity_coefficient(x,y, d)
assert(
    torch.allclose(similarity_score,
                   torch.tensor(3.854549))
)
```

But there are many different functions we can use as a score function:

1.   Multilayer Perceptron Attention
\begin{align}
c = \mathbf{v}^T \text{tanh}(\mathbf{W}_k \mathbf{k} + \mathbf{W}_q \mathbf{q}).
\end{align}
2.   Neural network
\begin{align}
c = \text{NN}(q \mathbin\Vert k)
\end{align} where $q \mathbin\Vert k$ - concatenating the key and value in the feature dimension

To compute coefficient for multiple queries and keys efficient, we need to rewrite all in **vectorized form**.

We assume  $q \in R^{b, t, d}$ and $ k \in R^{b, k, d} $ where $b$ - batch size, $t$ and $k$ - number of queries and keys, $d$ - the shape of the data.

PyTorch framework has the function for multiplication elements in a batch manner.

`torch.bnn(tensor1, tensor2)`
, where tensor1 is a $ (b \times n \times m) $ and tensor2 is a $ (b \times m \times p) $ . This function produce matrix multiplication for each object in a batch independent and output has shape $(b \times  n \; x \times p)$

```python
## imagine that we have 10 queries and 8 keys and need to construce matrix of cofficent.

torch.manual_seed(0)
d = 64
t,k = 10, 8
batch_size = 3
query = torch.rand(10, d)
query = query.repeat(batch_size, 1, 1)

key = torch.rand(8, d)
key = key.repeat(batch_size, 1, 1)

def compute_coefficient_matrix(query, key, d):
    ## TODO write vectorized scaled dot product
    return query @ key.transpose(-1,-2) / np.sqrt(d)

coefficient_matrix = compute_coefficient_matrix(query, key, d)

second_query = query[0, 2]
third_key = key[0, 3]

assert torch.allclose(coefficient_matrix[0, 2,3], scalar_similarity_coefficient(second_query, third_key, d))
assert torch.allclose(coefficient_matrix[0], coefficient_matrix[1])
assert torch.Size([batch_size, t,k]) == coefficient_matrix.shape
```

```python
def slow_non_vector_compute_coefficient_matrix(query, key, d):
    b_s, q_l, _ = query.shape
    _, k_l, _ = key.shape
    out = torch.zeros(b_s, q_l, k_l)

    for b in range(b_s):
        for q in range(q_l):
            for k in range(k_l):
                current_query = query[b, q]
                current_key = key[b, k]
                out[b, q, k] = scalar_similarity_coefficient(current_query, current_key, d)
    return out

```

```python
d = 64
t,k = 32, 10
batch_size = 128

query = torch.rand(batch_size, t, d)
key = torch.rand(batch_size, k, d)
```

```python
%%time
slow_attention_results = slow_non_vector_compute_coefficient_matrix(query, key, d)
```

```python
%%time
fast_attention_result = compute_coefficient_matrix(query, key, d)
```

**Using vectorization significant reduce attention computation time.**

```python
assert torch.allclose(slow_attention_results, fast_attention_result)
```

After getting the coefficient matrix, we need to transform it into probabilities by applying the softmax function. Each row contains a coefficient between one query and all keys, which should be a probability distribution.

```python
## imagine that we have 10 queries and 8 keys and need to construce matrix of cofficent.

torch.manual_seed(2)
d = 64
t,k = 10, 8
batch_size = 3
query = torch.rand(batch_size, 10, d)
key = torch.rand(batch_size, 8, d)

def compute_attention_matrix(query, key, d):
    coefficient_matrix = compute_coefficient_matrix(query, key, d)
    ## TODO apply softmax
    attention_matrix = torch.softmax(coefficient_matrix, dim=-1)
    return attention_matrix

A = compute_attention_matrix(query, key, d)

assert np.allclose((A.sum(axis=-1) - 1).sum().numpy(), 0, atol=1e-6)
assert torch.all(A >= 0.0)
```

We could visualize the attention matrix where several rows correspond to the query and column number to the key, and the value shows the attention coefficient between them.

```python
plt.figure(figsize=(10,10))
plt.title('Attention matrix')
plt.imshow(A[0])
plt.colorbar()
```

In this image attention, distribution should be almost uniform as we randomly initialize all elements, and the attention coefficient should be similar for all elements.

The last step is to use attention weights to compute the weighted sum of values. Sum each value with coefficient taken from the attention matrix.

\begin{align}
 O_k = \sum^n_{i = 1} a_{k i} v_i = A_k V
\end{align}
 where k - is a number of attention output

```python
## imagine that we have 10 queries and 8 keys and need to construce matrix of cofficent.

torch.manual_seed(6)
d = 64
t,k = 10, 8
batch_size = 3
query = torch.rand(batch_size, t, d)
key = torch.rand(batch_size, k, d)
value = torch.rand(batch_size, k, d)

def attention(query, key, value, d):
    coefficient_matrix = compute_coefficient_matrix(query, key, d)
    attention_matrix = torch.softmax(coefficient_matrix, dim=-1)
    out = attention_matrix @ value
    return attention_matrix, out

A, out = attention(query, key, value, d)
assert out.shape == query.shape
assert torch.allclose(out.sum(), torch.tensor(973.154541015625))
```

### Attention layer
To help attention find interesting patterns in data, we use linear projections to create queries, keys and values. A model can train such a projection matrix and describe data in the best way to solve the problem.

\begin{align}
q = w_q x_q \\
k = w_k x_k \\
v = w_v x_v \\
\end{align}

The following steps are the same with attention.

```python
class AttentionLayer(torch.nn.Module):
    def __init__(self, d):
        super().__init__()
        self.q_layer = torch.nn.Linear(d, d)
        self.k_layer = torch.nn.Linear(d, d, bias=False)
        self.v_layer = torch.nn.Linear(d, d)
        self.d = d

    def forward(self, query, key, value):
        self.q_proj = self.q_layer(query)
        self.k_proj = self.k_layer(key)
        self.v_proj = self.v_layer(value)
        self.A, out = attention(self.q_proj, self.k_proj, self.v_proj, self.d)
        return out

d = 32
t,k = 10, 8
batch_size = 3
query = torch.rand(batch_size, t, d)
key = torch.rand(batch_size, k, d)
value = torch.rand(batch_size, k, d)

att_layer_out = AttentionLayer(32)(query, key, value)
assert att_layer_out.shape == query.shape
assert not torch.allclose(attention(query, key, value, d)[1], att_layer_out)
```

### Self-attention
Attention mechanism does not suggest any restriction for input data, and a large number of modern architectures use attention when all $x_q, x_k, x_v$ the same input data. In this case query, key, the value produced by the same source but after application of trainable projection matrix represent data from a different point of view.

\begin{align}
 x_q = x_k = x_v = x
\end{align}

\begin{align}
q = w_q x \\
k = w_k x \\
v = w_v x \\
\end{align}

```python
d = 32
t = 10
batch_size = 3
x = torch.rand(batch_size, t, d)

att_layer_out = AttentionLayer(32)(x, x, x)
```

# Let's train models!

Let's consider toy examples to see how the attention mechanism work.
Our dataset is a sequence where each object represents a real number and flag. Our task is to find the sum of elements where the flag equals 1.

```python
def generate_dataset(seq_len, sample_count, target_elements_per_seq=3):
    marker = torch.zeros(sample_count, seq_len)
    for i in range(sample_count):
        pos = np.random.permutation(np.arange(seq_len))[:target_elements_per_seq]
        marker[i][pos] = 1.
    scaler = 10
    number = torch.rand(sample_count, seq_len) * scaler - scaler / 2
    y = (number * marker).sum(axis=1)
    x = torch.stack((number, marker), dim=-1)
    return x,y

```

```python
## generating data
## [
##    (x_1, flag_1), (x_2, flag_2) ... (x_n, flag_n)
## ]
x,y = generate_dataset(20, 10000)
```

```python
## split data into train and test
x_train, x_test, y_train, y_test = train_test_split(x,y )
```

```python
## constructing train and validation
train_dataset = torch.utils.data.TensorDataset(x_train, y_train)
test_dataset = torch.utils.data.TensorDataset(x_test, y_test)

batch_size = 32

train_dataloader = torch.utils.data.DataLoader(train_dataset, batch_size=batch_size, num_workers=2)
test_dataloader = torch.utils.data.DataLoader(test_dataset, batch_size=1024, num_workers=2)
```

To construct keys and values, we use input data and then apply a projection matrix. We utilise one trainable query to filter our data (such construction equals a row of attention matrix). Then we compute attention between all keys and one query with attention layer and use the obtained coefficient to make a weighted sum of values.

```python
class SummationAttentionModel(torch.nn.Module):
    def __init__(self, attention_layer, d):
        super().__init__()
        ## trainiable query
        self.syntetic_query = torch.nn.Parameter(torch.rand(1, 1, d))
        ## init layer for attention operation
        self.attention_layer = attention_layer
        ## we have one q in R^(d) -> output R^(d)
        ## linear layer map it in single output
        self.output_layer = torch.nn.Linear(d, 1)

    def forward(self, x):
        ## make attention between syntetic_query and x
        out = self.attention_layer(self.syntetic_query, x, x)
        ## out shape [n, 1, d] -> remove one dimension
        out = out[:, 0, :]
        ##  [n, d] -> [n, 1] for solving regression task
        prediction = self.output_layer(out)[:, 0]
        return prediction
```

```python
d = 2
## init model for our dataset with dimension size == 2
attention_layer = AttentionLayer(d)
model = SummationAttentionModel(attention_layer, d)
```

```python
## model use MSE loss with adam optimizer with default parameters
optimizer = torch.optim.Adam(model.parameters())
criterion = torch.nn.MSELoss()
```

```python
model.train()
losses_by_epoch = []
## train model during 100 epochs
for epoch in tqdm(range(50)):
    epoch_loss = []
    for batch in train_dataloader:
        x,y = batch
        ## take input data and target
        pred = model(x)
        ## compute loss function and calculate gradients
        loss = criterion(y, pred)
        loss.backward()

        ## weights update
        optimizer.step()
        optimizer.zero_grad()

        ## store loss for future plots
        epoch_loss.append(loss.detach().item())
    losses_by_epoch.append(np.mean(epoch_loss))

```

Training loss during epoch should decrease

```python
plt.figure(figsize=(10,5))
plt.title('Train loss')
plt.yscale('log')
plt.plot(losses_by_epoch);
```

Attention visualization

```python
model.eval()
batch = next(iter(test_dataloader))
x,y = batch
pred = model(x)

## save attention matrix for validation examples
valid_attention = model.attention_layer.A.detach().numpy()[:, 0]

k = model.attention_layer.k_proj.detach().numpy()
v = model.attention_layer.v_proj.detach().numpy()
q = model.attention_layer.q_proj.detach().numpy()
```

Queries for target elements should be a closer query with respect to the zero point to make the scalar product bigger.

```python
idx = 5
idx_of_target_examples = np.where(x[idx][:, 1] == 1)[0]

plt.figure(figsize=(10,7))
plt.scatter(k[idx, :, 0], k[idx, :, 1], label='Keys of non-target elements')
plt.scatter(k[idx, idx_of_target_examples, 0], k[idx, idx_of_target_examples, 1], label='Keys of target elements')

plt.scatter(q[0, :, 0], q[0, :, 1], label='Trainiable query')
plt.grid()
plt.legend()
```

The attention of query is close to zero for non-marked elements in a sequence, but considerable attention equals 1/3 because of softmax normalization.

```python
plt.figure(figsize=(10,5))
plt.title('Row of Attention matrix')
plt.plot(valid_attention[idx])
```

For different examples, the situation is the same. There are three peaks for each sequence that corresponds to the marked objects.

```python
## attention of differernt examples
idxes = [0, 1,10 ,256, 777,228, 412]

plt.figure(figsize=(10,8))
plt.title('Attention for several sequences')
plt.yticks(range(len(idxes)), labels=idxes)
plt.ylabel('Sequences number')
plt.xlabel('Object number')
plt.imshow(valid_attention[idxes])
plt.colorbar()
```

All plots summarize our understanding of the attention mechanism in training queries and keys to pay more attention to relevant objects and reduce the influence of other objects.

```python
## todo visualize values for marked and non marked objets
```

### Masked attention

There are many datasets where sequence length is different in real life, but we still need batch processing to make a model work fast and efficient. For this purpose, we use padding for a sequence.

Our task is to create a model that pushes the model to ignore padding elements and make attention for them equal to zero.

Your task is:

1.   Pad keys and values using
`torch.nn.utils.rnn.pad_sequence(sequences, batch_first=False, padding_value=0.0)`. **Use parameter batch_first=True, to make output compatible with our attention function**
2.   Create a mask for keys and values, which help you make zeros in the attention matrix. To change the number in the attention matrix, you can use
`Tensor.masked_fill_(mask, value). This function takes Tensor and boolean masks with the same shape and then fills tensor elements with value parameters where the mask is True.

```python
def masked_attention(query, key, value, mask, d):
    coefficient_matrix = compute_coefficient_matrix(query, key, d)
    ## todo make an masked attention
    return attention_matrix, out

d = 64
t = 4
batch_size = 1
query = torch.rand(batch_size, t, d)
key = [
    torch.rand(4, d),
    torch.rand(1, d),
    torch.rand(6, d),
    torch.rand(3, d),
]
value = [
    torch.rand(4, d),
    torch.rand(1, d),
    torch.rand(6, d),
    torch.rand(3, d),
]
```

```python
## print key and values, and find zeros at the end of short sequences
## plot attention matix, to see zeros coefficents
```

## Evaluate real model!

Based on [NLP example](https://teddykoker.com/2020/02/nlp-from-scratch-annotated-attention/)

Our task is to translate a sentence from German to English. We use the model where the encoder is the LSTM model, and the attention layer is applied as the decoder part. LSTM model generates hidden states and then uses the last hidden state of the predicted token as a query to find attention coefficients. After that model, make a weighted sum of hidden states of LSTM based attention coefficient and compute the next predicted token.

```python
import torch
import torch.nn as nn
import torch.nn.functional as F
import random
from tqdm.notebook import tqdm
import numpy as np

%matplotlib inline
import matplotlib.pyplot as plt
import matplotlib.ticker as ticker
```

```python
class Decoder(nn.Module):
    """
    Decode output from hidden state and context
    """
    def __init__(self, output_dim, embed_dim, hidden_dim, num_layers, dropout):
        super(Decoder, self).__init__()
        self.output_dim = output_dim
        self.embed = nn.Embedding(output_dim, embed_dim)
        # stacking LSTM
        self.lstm = nn.LSTM(embed_dim, hidden_dim, n_layers, dropout=dropout)
        self.attention = Attention(hidden_dim) # we'll get to later
        self.wc = nn.Linear(hidden_dim * 2, hidden_dim)
        self.ws = nn.Linear(hidden_dim, output_dim)
        self.dropout = nn.Dropout(dropout)

    def forward(self, trg, hidden, encoder_out):
        trg = trg.unsqueeze(0)
        embed = self.dropout(self.embed(trg))
        decoder_out, hidden = self.lstm(embed, hidden)

        # we'll go over how these are computed later
        atten, context = self.attention(decoder_out, encoder_out)

        # "We employ a simple concatenation layer to combine the
        # information from both vectors:"
        atten_hidden = self.wc(torch.cat((decoder_out, context), dim=2)).tanh()

        # "The attentional vector ~h_t is then fed through the softmax layer
        # to produce the predictive distribution:"
        out = self.ws(atten_hidden.squeeze(0))
        # softmax will be included in loss function

        return out, hidden, atten
```

```python
class Attention(nn.Module):
    """
    Compute alignment vector and context vector from hidden states
    """
    def __init__(self, hidden_dim, score_fn="general"):
        super(Attention, self).__init__()
        self.score_fn = score_fn
        if score_fn == "general":
            self.w = nn.Linear(hidden_dim, hidden_dim)
        if score_fn == "concat":
            self.w = nn.Linear(hidden_dim * 2, hidden_dim)
            self.v = torch.nn.Parameter(torch.FloatTensor(hidden_dim))

    def score(self, decoder_out, encoder_outs):
        if self.score_fn == "dot":
            return torch.sum(decoder_out * encoder_outs, dim=2)
        if self.score_fn == "general":
            return torch.sum(decoder_out * self.w(encoder_outs), dim=2)
        if self.score_fn == "concat":
            decoder_outs = decoder_out.repeat(encoder_outs.shape[0], 1, 1)
            cat = torch.cat((decoder_outs, encoder_outs))
            return torch.sum(self.v * self.w(cat), dim=2)

    def forward(self, decoder_out, encoder_outs):
        score = self.score(decoder_out, encoder_outs)
        a = F.softmax(score, dim=0)

        # "Given the alignment vector as weights, the context vector
        # c_t is computed as the weighted average over all the source
        # hidden states:"
        context = torch.bmm(
            a.transpose(1, 0).unsqueeze(1),
            encoder_outs.transpose(1, 0)
        ).transpose(1, 0)
        return a, context
```

```python
class Encoder(nn.Module):
    """
    Stacked LSTM encoder
    """
    def __init__(self, input_dim, embed_dim, hidden_dim,
                 n_layers, dropout):
        super(Encoder, self).__init__()
        self.embed = nn.Embedding(input_dim, embed_dim)
        self.lstm = nn.LSTM(embed_dim, hidden_dim, n_layers, dropout=dropout)
        self.dropout = nn.Dropout(dropout)

    def forward(self, src):
        embed = self.dropout(self.embed(src))
        out, hidden = self.lstm(embed)
        return self.dropout(out), hidden

class Model(nn.Module):
    """
    Sequence to Sequence model with attention
    """
    def __init__(self, encoder, decoder):
        super(Model, self).__init__()
        self.encoder = encoder
        self.decoder = decoder

    def forward(self, src, trg, teacher_force_ratio = 0.5):
        outs = torch.zeros(
            trg.shape[0], trg.shape[1], self.decoder.output_dim
        ).to(src.device)
        encoder_out, hidden = self.encoder(src)

        x = trg[0]
        for t in range(1, trg.shape[0]):
            outs[t], hidden, _ = self.decoder(x, hidden, encoder_out)
            x = trg[t] if random.random() < teacher_force_ratio else outs[t].argmax(1)

        return outs
```

We choose a relatively small dataset with 30 thousand English-German sentence pairs.

```python
! pip install torchtext
```

```python
import torchtext
```

```python
## download dataset
!python -m spacy download en
!python -m spacy download de
```

Common library torchtext includes a lot of powerful tools for text preprocessing and often appears in different NLP tasks. For our purpose, first of all, we initialize a ready tokenizer. Then put it in the special class Field ("analogue" of torch's Dataset) with other instructions for converting text to Tensor.

```python
from torchtext.legacy.datasets import Multi30k
from torchtext.legacy.data import Field, BucketIterator
import spacy

# tokenizers
spacy_de = spacy.load('de')
spacy_en = spacy.load('en')
tokenize_de = lambda text: [tok.text for tok in spacy_de.tokenizer(text)]
tokenize_en = lambda text: [tok.text for tok in spacy_en.tokenizer(text)]

# fields
SRC = Field(tokenize=tokenize_de, init_token='<sos>',
            eos_token="<eos>", lower=True)
TRG = Field(tokenize=tokenize_de, init_token='<sos>',
            eos_token="<eos>", lower=True)

# data
train_data, valid_data, test_data = Multi30k.splits(('.de', '.en'), (SRC, TRG))
SRC.build_vocab(train_data, min_freq=2)
TRG.build_vocab(train_data, min_freq=2)
```

As we use a relatively small set of data, we can initialize a pretty simple model. The training time from scratch is ~30 minutes.

```python
input_dim = len(SRC.vocab)
output_dim = len(TRG.vocab)
embed_dim = 256
hidden_dim = 512
n_layers = 2
dropout = 0.2
batch_size = 128
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')

def init_weights(model):
    for param in model.parameters():
        nn.init.uniform_(param.data, -0.1, 0.1)

train_loader, valid_loader, test_loader = BucketIterator.splits(
    (train_data, valid_data, test_data),
    batch_size=batch_size,
    device=device
)

encoder = Encoder(input_dim, embed_dim, hidden_dim, n_layers, dropout)

decoder = Decoder(output_dim, embed_dim, hidden_dim, n_layers, dropout)
model = Model(encoder, decoder).to(device)
model.apply(init_weights);
```

```python
def step(model, data, criterion, train=False, optimizer=None):
    model.train() if train else model.eval()
    total_loss = 0
    for batch in tqdm(data, leave=False):
        if train: optimizer.zero_grad()
        pred = model(batch.src, batch.trg)
        loss = criterion(pred.view(-1, pred.size(2)), batch.trg.view(-1))
        if train:
            loss.backward()
            torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
            optimizer.step()
        total_loss += loss.item()
    return total_loss / len(data)
```

```python
# optimizer = torch.optim.Adam(model.parameters())
# criterion = nn.CrossEntropyLoss(ignore_index=TRG.vocab.stoi[TRG.pad_token])
# num_epochs = 10

# best_loss = float('inf')
# train_loss, valid_loss = np.zeros((2, num_epochs))
# for e in range(num_epochs):
#     train_loss[e] = step(model, train_loader, criterion, train=True, optimizer=optimizer)
#     valid_loss[e] = step(model, valid_loader, criterion)
#     #print(f"epoch: {e} train_loss: {train_loss[e]:.2f} valid_loss: {valid_loss[e]:.2f}")
#     if valid_loss[e] < best_loss:
#         torch.save(model.state_dict(), 'model.pt')
#         best_loss = valid_loss[e]

# plt.figure(figsize=(5, 3), dpi=300)
# plt.plot(train_loss[:10], label="Train")
# plt.plot(valid_loss[:10], label="Valid.")
# plt.legend(); plt.ylabel("Cross-Entropy Loss"); plt.xlabel("Epoch")
# plt.show()
```

Of course, we can't wait this time, so we upload pre-trained weights. The interested listener can try to train the model by himself (uncomment above):)

```python
criterion = nn.CrossEntropyLoss(ignore_index=TRG.vocab.stoi[TRG.pad_token])
model.load_state_dict(torch.load("model.pt"))
test_loss = step(model, test_loader, criterion)
print(f"Test loss: {test_loss:.2f}")
print(f"Test perplexity: {np.exp(test_loss):.2f}")
```

**Let's translate!**

```python
@torch.no_grad()
def translate(sentence, model, device, max_len=50):
    model.eval()
    src = SRC.process([sentence]).to(device)
    trg = torch.ones(1, dtype=torch.int64).to(device) * TRG.vocab.stoi[TRG.init_token]
    trgs, attention = [], []
    encoder_out, hidden = model.encoder(src)

    for t in range(max_len):
        trg, hidden, atten = model.decoder(trg, hidden, encoder_out)
        trg = trg.argmax(1)
        trgs.append(trg)
        attention.append(atten.T)
        if trg == TRG.vocab.stoi[TRG.eos_token]: break

    trg = [TRG.vocab.itos[i] for i in trgs]
    src = [SRC.vocab.itos[i] for i in src]
    attention = torch.cat(attention).cpu().numpy()[:-1, 1:]
    return src, trg, attention
```

```python
example = 6
src, trg = test_data[example].src, test_data[example].trg
print(f"Source: {' '.join(src)}")
print(f"Target: {' '.join(trg)}")
```

```python
src, pred, attention = translate(src, model, device)
print(f"Prediction: {' '.join(pred[:-1])}")
```

```python
def plot_attention(src, trg, attention):
    fig = plt.figure(figsize=(5, 5), dpi=150)
    ax = fig.add_subplot(111)
    ax.matshow(attention)
    ax.xaxis.set_major_locator(ticker.MultipleLocator(1))
    ax.yaxis.set_major_locator(ticker.MultipleLocator(1))
    ax.set_xticklabels([''] + src, rotation=60)
    ax.set_yticklabels([''] + trg)

plot_attention(src, pred, attention)
```

```python
src = 'Mann im schwarzen Anzug geht zur Bushaltestelle'.lower().split(' ')
```

```python
src, pred, attention = translate(src, model, device)
print(f"Prediction: {' '.join(pred[:-1])}")
print(src)
```

```python
plot_attention(src, pred, attention)
```

## Summary

*   Query, key helps the model find a relevant object in a sequence + interpretable.
*   Computing attention matrix in a vectorize form significant improve the speed.
*   For batch processing with sequences with different lengths, masked attention should be used.
*   Attention mechanism help model directly use objects on a sequence and improve quality of RNN models.
*   There are several functions used as the score function.


# 12.850 - Assignment 3  |  Bryce Corlett

## Exploring Convergence Properties

This assignment was motivated by examining the different convergence properties of Jacobi, Gauss-Seidel, and SOR iteration schemes as applied to a toy 1-D elliptic problem:
\begin{align}
    \frac{\partial}{\partial x} \kappa \frac{\partial}{\partial x} u = \nu u
\end{align}

The problem is similar to those observed in the previous two assignments, but with the additional forced advection term ($\nu u$), which appears in the $C_{k,k}$ term in the discretized equation:
\begin{align}
    u_{k+1}\underbrace{\left[\frac{K_{k}}{\Delta z^{c}_{k+1} \Delta z^{f}_{k}}\right]}_{C_{k,k+1}} - u_{k}\underbrace{\left[\frac{K_{k}}{\Delta z^{c}_{k+1} \Delta z^{f}_{k}} + \frac{K_{k-1}}{\Delta z^{c}_{k} \Delta z^{f}_{k}} - \nu \right]}_{C_{k,k}} + u_{k-1}\underbrace{\left[\frac{K_{k-1}}{\Delta z^{c}_{k} \Delta z^{f}_{k}}\right]}_{C_{k,k-1}} = 0
\end{align}

When solving, the equation can be written as $[A]x = b$, where $[A]$ is the matrix of discrete values ($C_{k,k-1}$,$C_{k,k}$, and $C_{k,k+1}$), $x$ is a vector of the values you are solving for, and $b$ is a vector of the boundary conditions. The boundary conditions will affect the values within $[A]$, as the top and bottom cells are either forced with known values in the cells beneath (Dirichlet), or one known boundary value and a forced boundary flux (Dirichlet for the known boundary, and Neumann for the known flux boundary). In the first case, the known values move the respective portion of the $C_{k,k}$ term to the boundary conditions vector; in the second case, the *flux* is forced within the boundary conditions vector.

Once the discrete matrix $[A]$ is formed, the iteration processes break down the procedure in different manners according to whether they are wholly dependent on past iterations (Jacobi), use current-iteration estimates where available (Gauss-Seidel), or use a combination of current-iteration and past-iteration estimates (relaxed by some factor) to reach the solution more quickly (Succesive Over-Relaxation, or SOR).

To further grasp the differences, and what the following code is doing, we write the original equation as:
\begin{align}
    Ax = b
\end{align}
where the matrix $A$ can be broken into three discrete matrices: the lower triangle ($\triangleright$), the diagonal ($\diagdown$), and the upper triangle ($\triangleleft$), where $[A] = [\triangleright] + [\diagdown] + [\triangleleft]$.

Using this notation, we can write the iterative methods as $B x_{n+1} = b - C x_{n}$, where $B$ and $C$ are formed from the original matrix $A$. Thus:
\begin{align}
    Jacobi: & B = [\diagdown] & C = [\triangleright] + [\triangleleft] \\
    Gauss-Seidel: & B = [\triangleright] + [\diagdown] & C = [\triangleleft] \\
    SOR: & B = \omega[\triangleright] + [\diagdown] & C = (1-\omega)[\diagdown] + \omega[\triangleleft] \\
\end{align}

The convergence of a system can be quantified as its spectral radius ($\rho$), where $\rho = max(~|\lambda_{i}|~)$, and $\lambda_{i}$ are the eigenvalues of $\left[ B^{-1}C \right]$; thus, how quickly a system converges is intimately related to the iterative method and the type of boundary conditions.

Now, we get to the code.

```python
#Import toolboxes
from scipy import sparse #Allows me to create sparse matrices (i.e. not store all of the zeros in the 'A' matrix)
from scipy.sparse import linalg as linal
from numpy import * #To make matrices and do matrix manipulation
import matplotlib.pyplot as plt

%matplotlib inline

```

Here I create the code that will perform a specified iteration scheme, with the option of specifying a value for $\omega$.

```python
def space_iterate(method,A,b,resid,**optional):
    '''Uses sparse matrices'''

    if ('w' in optional):
        #print 'w found, it is ', optional['w']
        w = optional['w']
    else:
        #print 'no w found, assumed to be 1.17 if needed.'
        w = 1.17
        #w = 1.02

    n = max(A.get_shape())
    t = 0;
    Q = b[:,0]
    Rold = 100 #initialize value
    Rnew = 1
    while (absolute(Rnew - Rold)/float((absolute([Rold]))))*100.0 > resid:
        t = t+1
        Rold = Rnew
        Q = append(Q,Q[:,0].dot(0.),axis=1)

        if method == 'jacobi': #Jacobi iteration scheme

            B = sparse.tril(A,0) - sparse.tril(A,-1) #only the diagonal
            C = sparse.triu(A,1) + sparse.tril(A,-1) #only off-diagonal

            Q[:,t] = linal.inv(B).dot(b - C.dot(Q[:,t-1]))

        elif method == 'gaussseidel': #Gauss-Seidel iteration scheme

            B = sparse.tril(A,0)
            C = sparse.triu(A,1)

            Q[:,t] = linal.inv(B).dot(b - C.dot(Q[:,t-1]))

        elif method == 'sor': #SOR method
            #Q[i,t] = w*Q[i,t] + (1-w)*Q[i,t-1]

            B = sparse.tril(A,-1)*(float(w)) + (sparse.tril(A,0)-sparse.tril(A,-1))
            C = ( (sparse.tril(A,0)-sparse.tril(A,-1)).dot(float(w-1))) + (sparse.triu(A,1).dot(float(w)) )

            #B = - sparse.tril(A,-1)*(float(w)) + (sparse.tril(A,0)-sparse.tril(A,-1))
            #C = ( (sparse.tril(A,0)-sparse.tril(A,-1)).dot(float(1-w))) + (sparse.triu(A,1).dot(float(w)) )

            Q[:,t] = linal.inv(B).dot((float(w)*b) - C.dot(Q[:,t-1]))

        else:
            print('Improper Option - program closing.')
            return

        Rnew = mean(Q[:,t])
        #print (absolute(Rnew - Rold)/float((absolute(Rold))))*100.0

    B = B.tocsc() #convert sparse matrices to csc format
    C = C.tocsc()

    print('Iterations = '+str(t)+'; Spectral Radius = '+str(absolute(linalg.eigvals(linal.inv(B).dot(C).todense())).max()))
    #(absolute(linal.eigs(linal.inv(B.tocsc()).dot(C.tocsc()),k=ndim(b)-1,return_eigenvectors=False)).max())

    I = t
    S = absolute(linalg.eigvals(linal.inv(B).dot(C).todense())).max()

    #print('The spectral radius of the problem is '+str(absolute(linal.eigs(linal.inv(B).dot(C),k=ndim(b)-1,return_eigenvectors=False)).max()) )
    return[Q,I,S]
```

The following code will initialize the $[A]$ and $[b]$ matrices according to the inputs, and depending on which code I call, as they use Neumann + Dirichlet and Dirichlet-only boundary conditions, respectively.

```python
def neumann_stable(n,v,u0,Z_f,Z_c,K):
    #Create Neumann + Dirichlet boundary conditions, yielding matrices A + b
    A=zeros((3,n)) # For solving for 'n+1' solution
    for item in range(1,n+1): #Start from bed and work to surface
        #j-1
        if item>1:
            A[0,item-2]=+(K[item-1]/((Z_f[item]-Z_f[item-1])*(Z_c[item]-Z_c[item-1])) )
        #j
        A[1,item-1]=-( (K[item-1]/((Z_f[item]-Z_f[item-1])*(Z_c[item]-Z_c[item-1]))) + (K[item]/((Z_f[item]-Z_f[item-1])*(Z_c[item+1]-Z_c[item]))) + v)
        if item == n: #Sets free-slip boundary condition at the surface
            A[1,item-1]=-( (K[item-1]/((Z_f[item]-Z_f[item-1])*(Z_c[item]-Z_c[item-1]))) + v)
        #j+1
        if item != n:
            A[2,item]=+(K[item]/((Z_f[item]-Z_f[item-1])*(Z_c[item+1]-Z_c[item])) )

    A = sparse.spdiags(A,array([-1,0,1]),n,n)

    # Construct Boundary Condition Matrix
    b=zeros(size(A,1))
    b[0]=b[0] + (u0* (K[item-1]/((Z_f[item]-Z_f[item-1])*(Z_c[item]-Z_c[item-1]))) ) #Because u0 is zero, this line does nothing.

    # Define + Apply guess + boundary conditions
    b=matrix(b).T
    return[A,b]

def dirichlet(n,v,u0,u1,Z_f,Z_c,K):
    #Create Dirichlet boundary conditions, yielding matrices A + b
    A=zeros((3,n)) # For solving for 'n+1' solution
    for item in range(1,n+1): #Start from bed and work to surface
        #j-1
        if item>1:
            A[0,item-2]=+(K[item-1]/((Z_f[item]-Z_f[item-1])*(Z_c[item]-Z_c[item-1])) )
        #j
        A[1,item-1]=-( (K[item-1]/((Z_f[item]-Z_f[item-1])*(Z_c[item]-Z_c[item-1]))) + (K[item]/((Z_f[item]-Z_f[item-1])*(Z_c[item+1]-Z_c[item]))) + v )
        #j+1
        if item != n:
            A[2,item]=+(K[item]/((Z_f[item]-Z_f[item-1])*(Z_c[item+1]-Z_c[item])) )

    A = sparse.spdiags(A,array([-1,0,1]),n,n)

    # Construct Boundary Condition Matrix
    b=zeros(size(A,1))
    b[0]=b[0] + (u0* (K[1-1]/((Z_f[1]-Z_f[1-1])*(Z_c[1]-Z_c[1-1]))) )
    b[n-1]=b[n-1] + (u1* (K[n-1]/((Z_f[n]-Z_f[n-1])*(Z_c[n]-Z_c[n-1]))) )

    # Define + Apply guess + boundary conditions
    b=matrix(b).T
    return [A,b]
```

```python
#Initialize a comparison
n=20 #n must be greater than 6 for my SOR code to work - the issue lies with calculating eigenvalues.
K=1
v=0.3
u0=0.
u1=1.

[A,b]=dirichlet(n,v,u0,1,matrix(arange(0,n+2)).T,matrix(arange(0,n+2)).T,K*ones([n+1,1]));
[Q,I,S]=space_iterate('sor',A,b,1,w=1.25);
```

    Iterations = 10; Spectral Radius = 0.539349306957

# 1. Sensitivity of the System to Boundary Conditions

Note that the spectral radius changes as the boundary conditions change, but not when the values change. The spectral radius is not influenced by boundary condition values per say, but it is influenced by the type of boundary conditions present within the system. This is because the values are not introduced to the matrices $[B]$ or $[C]$, but the boundary conditions will influence the values on the diagonals of the matrix $[A]$, which will either end up in matrices $[B]$ or $[C]$ depending on the iteration scheme.

See below for comparison values:

```python
#Initialize a comparison
n=20 #n must be greater than 6 for my SOR code to work - the issue lies with calculating eigenvalues.
K=0.3
v=0.5
u0=0.
u1=1.

#[A,b]=dirichlet(n,v,u0,u1,matrix(arange(0,n+2)).T,matrix(arange(0,n+2)).T,K*ones([n+1,1]))
#[A,b]=neumann_stable(n,v,u1,matrix(arange(0,n+2)).T,matrix(arange(0,n+2)).T,K*ones([n+1,1]))
```

### Variations in boundary condition values:

```python
print 'Variations under changing boundary values, using Dirichlet boundary conditions:'
print '\n Conditions with u_surface = 1: '
[A,b]=dirichlet(n,v,u0,1,matrix(arange(0,n+2)).T,matrix(arange(0,n+2)).T,K*ones([n+1,1]));
[Q,I,S]=space_iterate('sor',A,b,1);

print '\n\n Conditions with u_surface = 2: '
[A,b]=dirichlet(n,v,u0,2,matrix(arange(0,n+2)).T,matrix(arange(0,n+2)).T,K*ones([n+1,1]));
[Q,I,S]=space_iterate('sor',A,b,1);

print '\n\n Conditions with u_surface = 3: '
[A,b]=dirichlet(n,v,u0,3,matrix(arange(0,n+2)).T,matrix(arange(0,n+2)).T,K*ones([n+1,1]));
[Q,I,S]=space_iterate('sor',A,b,1);
```

    Variations under changing boundary values, using Dirichlet boundary conditions:

     Conditions with u_surface = 1:
    Iterations = 5; Spectral Radius = 0.17

     Conditions with u_surface = 2:
    Iterations = 5; Spectral Radius = 0.17

     Conditions with u_surface = 3:
    Iterations = 5; Spectral Radius = 0.17

### Variations in the boundary condition itself:

```python
print 'Variations under changing boundary conditions | Dirichlet v Neumann + Dirichlet:'
print '\n\n Conditions with Neumann + Dirichlet: '
[A,b]=neumann_stable(n,v,1,matrix(arange(0,n+2)).T,matrix(arange(0,n+2)).T,K*ones([n+1,1]));
[Q,I,S]=space_iterate('sor',A,b,1);

print '\n\n Conditions with Dirichlet: '
[A,b]=dirichlet(n,v,1,0,matrix(arange(0,n+2)).T,matrix(arange(0,n+2)).T,K*ones([n+1,1]));
[Q,I,S]=space_iterate('sor',A,b,1);
```

    Variations under changing boundary conditions | Dirichlet v Neumann + Dirichlet:

     Conditions with Neumann + Dirichlet:
    Iterations = 6; Spectral Radius = 0.5

     Conditions with Dirichlet:
    Iterations = 6; Spectral Radius = 0.5

# 2. Sensitivity of the solution to $\nu$

The convergence of the system *is* sensitive to the value of $\nu$, as $\nu$ will change the values on the diagonal of the matrix $[A]$, changing the determinate of $[B]$ in the process of calculating its inverse, and thus the spectral radius ($\rho$).

```python
[A,b]=dirichlet(n,0.5,1,0,matrix(arange(0,n+2)).T,matrix(arange(0,n+2)).T,K*ones([n+1,1]))
[Q,I,S]=space_iterate('jacobi',A,b,1)
[Q,I,S]=space_iterate('gaussseidel',A,b,1)
[Q,I,S]=space_iterate('sor',A,b,1)

```

    /Users/bcorlett/anaconda/lib/python2.7/site-packages/IPython/kernel/__main__.py:16: RuntimeWarning: divide by zero encountered in double_scalars
    /Users/bcorlett/anaconda/lib/python2.7/site-packages/scipy/sparse/linalg/dsolve/linsolve.py:90: SparseEfficiencyWarning: spsolve requires A be CSC or CSR matrix format
      SparseEfficiencyWarning)

    Iterations = 14; Spectral Radius = 0.79106466098
    Iterations = 7; Spectral Radius = 0.625783297852
    Iterations = 3; Spectral Radius = 0.452811254436

    /Users/bcorlett/anaconda/lib/python2.7/site-packages/scipy/sparse/linalg/dsolve/linsolve.py:157: SparseEfficiencyWarning: spsolve requires sparse b be in CSC or CSR matrix format
      SparseEfficiencyWarning)

```python
nu=[1.0e3,1.0e2,1.0e1,9.0,8.0,7.0,6.0,5.0,4.0,3.0,2.0,1.0e0,.9,.8,.7,.6,.5,.4,.3,.2,1.0e-1,1.0e-2,1.0e-3]

#fig, axs = plt.subplots(1,2)
#,ax=axs[1]
plt.figure(figsize=(20,5))
plt.subplot(121)
for R in range(0,size(nu)): #Varies the value of nu
    [A,b]=dirichlet(n,nu[R],1,0,matrix(arange(0,n+2)).T,matrix(arange(0,n+2)).T,K*ones([n+1,1]))

    [Q,Is,Ss]=space_iterate('sor',A,b,1)
    [Q,Ig,Sg]=space_iterate('gaussseidel',A,b,1)
    [Q,Ij,Sj]=space_iterate('jacobi',A,b,1)

    plt.semilogx(nu[R],Ss,'ok')
    plt.semilogx(nu[R],Sg,'sk')
    plt.semilogx(nu[R],Sj,'+k')

    if int == 1.0:
        plt.legend(['SOR','Gauss-Seidel','Jacobi'])
        #plt.title('Surface Velocity')
        plt.ylabel(r'$\rho$', fontsize=20)
        plt.xlabel(r'$\nu$', fontsize=20)

plt.subplot(122)
for R in range(0,size(nu)): #Varies the value of nu
    [A,b]=dirichlet(n,nu[R],1,0,matrix(arange(0,n+2)).T,matrix(arange(0,n+2)).T,K*ones([n+1,1]))

    [Q,Is,Ss]=space_iterate('sor',A,b,1)
    [Q,Ig,Sg]=space_iterate('gaussseidel',A,b,1)
    [Q,Ij,Sj]=space_iterate('jacobi',A,b,1)

    plt.semilogx(nu[R],Is,'ok')
    plt.semilogx(nu[R],Ig,'sk')
    plt.semilogx(nu[R],Ij,'+k')

    if R == 1.0:
        plt.legend(['SOR','Gauss-Seidel','Jacobi'])
        #plt.title('Surface Velocity')
        plt.ylabel(r'$Iterations$', fontsize=20)
        plt.xlabel(r'$\nu$', fontsize=20)

```

### Reviewing the effect of $\nu$

After plotting these values, it became apparent that further investigation of the SOR factor was needed. After testing the matrices with varying values for this factor, it appears that the factor can be chosen specifically for given conditions to maximize the code's efficiency. The downside of this is that the SOR method will become inaccurate if the SOR factor is *not* tailored to matrix $[A]$, i.e. adjusted relative to the magnitude of the terms $C_{k,k}$, as seen in the flat-lining of the SOR spectral radius for given values of $\nu$, or effectively values of $\frac{\nu}{C_{k,k}}$.

#### Questioning the effectiveness of the SOR code

As the SOR code appears to yield highly-consistent values for the number of iterations required, I am unsure as to whether the issue is with my SOR code, or if the issue is with the SOR procedure. I am not sure if there is a way to pre-calibrate the SOR factor to work with the given conditions, but I hope to look into this in the next week or so.

# 3. Convergence of the system on N

The system takes longer *time* to converge for matrices with larger N's, as the computations involve matrices of size N x N; however, it appears that the system requires the same number of iterations in spite of the spectral radius increasing (which should indicate that the procedure takes more iterations to reach the specified tolerance). This may be an artifact of my SOR code; further investigation is needed, which will take place this weekend and next week to iron out any bugs that I might be able to find in the SOR portion of my code. Fortunately, the Gauss-Seidel and Jacobi schemes appear to be working fine.

```python
K=1.
v=1.
u0=0.
u1=1.

N=(arange(0,20)+1)*10

#fig, axs = plt.subplots(1,2)
#,ax=axs[1]
plt.figure(figsize=(20,5))
plt.subplot(121)

for R in range(0,size(N)): #Varies the value of n
    [A,b]=dirichlet(N[R],v,1,0,matrix(arange(0,N[R]+2)).T,matrix(arange(0,N[R]+2)).T,K*ones([N[R]+1,1]))

    [Q,Is,Ss]=space_iterate('sor',A,b,1)
    [Q,Ig,Sg]=space_iterate('gaussseidel',A,b,1)
    [Q,Ij,Sj]=space_iterate('jacobi',A,b,1)

    plt.plot(N[R],Ss,'ok')
    plt.plot(N[R],Sg,'sk')
    plt.plot(N[R],Sj,'+k')

    if R == 1.0:
        plt.legend(['SOR','Gauss-Seidel','Jacobi'])
        #plt.title('Surface Velocity')
        plt.ylabel(r'$\rho$', fontsize=20)
        plt.xlabel(r'$n$', fontsize=20)

plt.subplot(122)
for R in range(0,size(N)): #Varies the value of n
    [A,b]=dirichlet(N[R],v,1,0,matrix(arange(0,N[R]+2)).T,matrix(arange(0,N[R]+2)).T,K*ones([N[R]+1,1]))

    [Q,Is,Ss]=space_iterate('sor',A,b,1)
    [Q,Ig,Sg]=space_iterate('gaussseidel',A,b,1)
    [Q,Ij,Sj]=space_iterate('jacobi',A,b,1)

    plt.plot(N[R],Is,'ok')
    plt.plot(N[R],Ig,'sk')
    plt.plot(N[R],Ij,'+k')

    if R == 1.0:
        plt.legend(['SOR','Gauss-Seidel','Jacobi'])
        #plt.title('Surface Velocity')
        plt.ylabel(r'$Iterations$', fontsize=20)
        plt.xlabel(r'$n$', fontsize=20)
```

# 4. Estimate Iterations

The number of iterations required to reach a given error threshold ($\delta$) can be estimated by referring back to the base of our definition of the spectral radius ($\rho$). From $\rho = max(~|\lambda_{i}|~)$, where $\lambda_{i}$ are the eigenvalues of $\left[ B^{-1}C \right]$, from
\begin{align}
    B x_{n+1} = b - C x_{n}
\end{align}
Taking $e_{n} = x_{n+1}-x_{n}$, we can simplify the above equation to
\begin{align}
    e_{n} = \left[ B^{-1} C \right] e_{n-1}
\end{align}
Keeping in mind that $e_{n-1} = \left[ B^{-1} C \right] e_{n-2}$, we can further simplify to
\begin{align}
    e_{n} = \left[ B^{-1} C \right]^{n} e_{0}
\end{align}
where the final difference is a function of the initial difference times the basis of $\rho$ to some power. As we are requiring that $e_{n}$, or the difference between iterations, be smaller than some threshold ($\delta$), we can restate the problem as
\begin{align}
    e_{n} = \left[ B^{-1} C \right]^{n} e_{0} < \delta
\end{align}
where $n$ is the number of iterations required to reach our error threshold, and all values other than $n$ are matrices. Solving for $n$, we reach the following relation:
\begin{align}
    e_{0}^{-1} \delta > \left[ B^{-1}C \right]^{n} \\
    \frac{\log( e_{0}^{-1} \delta )}{\log( B^{-1}C )} < n \\
    \\
    \log( e_{0}^{-1} \delta ) - \log( B^{-1}C ) < n
\end{align}

In effect, we can estimate the number of iterations required to reach the threshold by calculating the difference between our initial conditions and our first estimate ($e_{0}$), knowing $\delta$, and calculating $\left[ B^{-1}C \right]$ from our given system and boundary conditions. The number of iterations required to reach a given tolerance can be calculated, but depends on the specific conditions of the problem. In general, however, the smaller $\left[ B^{-1}C \right]$ becomes (and thus $\rho$), the faster the scheme will converge to the specified tolerance.

For a specified problem, the number of iterations required to reach a final result will also change depending on the method of iteration, as each method utilizes a different form of matrices $B$ and $C$, as shown previously.