# Species eradications and introductions

## A scaling of the LV food web

The Lotka-Volterra food web has dynamics

\begin{equation}
\frac{dn_{t;i}}{dt} = n_{t;i}
\left(r_i + \sum_j a_{ij} n_{t;j}\right)
\end{equation}

with a steady state

\begin{equation}
\mathbf{n} = A^{-1} (-\mathbf{r})
\end{equation}

Let $N$ be the diagonal matrix formed from $\mathbf{n}$. Then the Jacobian

\begin{equation}
J = \left[ \left. \frac{\partial n_{t;i}'}{\partial
n_{t;j}}\right|_{\mathbf{n}}\right] = NA
\end{equation}

Interested in scaling a system such that $\mathbf{\hat{n}} = 1$, so choose

\begin{equation}
\hat{n}_{t;i} = \frac{n_{t;i}}{n_i}
\end{equation}

so the dynamics

\begin{equation}
\hat{n}_{t;i}'
= \hat{n}_{t;i} \left(
    r_i + \sum_j a_{ij} \hat{n}_{t;i} n_j
\right)
\end{equation}

and the Jacobian

\begin{equation}
M = \left[ \left.\frac{\partial \hat{n}_{t;i}'}{\partial
\hat{n}_{t;j}}\right|_{\mathbf{\hat{n}} = [1]} \right] =
AN = N^{-1}JN
\end{equation}

At the steady state

\begin{equation}
r_{i} = -\sum_{j} a_{ij} n_j = -\sum_j m_{ij}
\end{equation}

The sensitivity matrix of the scaled system is

\begin{equation}
M^{-1} = N^{-1} J^{-1} N
\end{equation}

## Special properties of the scaled system

1. $M$ is a similarity transform of $J$, therefore the eigenvalues (but not eigenvectors) are equal.

2. $\mathbf{n}$ is strictly positive and $N$ is a diagonal matrix, therefore the signs (but not the magnitudes) of the elements of $J^{-1}$ are preserved.

$M$ can be further scaled so that its elements $m_{ij} = (0,1)$, scaling the eigenvalues and the elements of the sensitivity matrix.

$M$ therefore represents a suite of systems conforming the the transformations above.

### Example of scaling and special properties

Consider an LV food web with the following structure

```
import networkx as nx
import numpy as np
import matplotlib.pyplot as plt

%matplotlib inline

# Define positive and negative interactions
positive_edges_list = [(4,2), (4,3), (2,3), (2,1), (3,1), (3,0)]
negative_edges_list = [(0,0), (1,1), (2,2), (3,3), (4,4),
                       (2,4), (3,4), (3,2), (1,2), (1,3), (0,3)]

# Create digraph
G = nx.DiGraph()
G.add_edges_from(negative_edges_list, color='red', weight=-1)
G.add_edges_from(positive_edges_list, color='green', weight=1)
node_labels = {node:str(node) for node in G.nodes()}

# Draw graph
pos=nx.spring_layout(G)
nx.draw_networkx_nodes(G, pos)
nx.draw_networkx_edges(G, pos,
                        edgelist=positive_edges_list, edge_color='g')
nx.draw_networkx_edges(G, pos,
                        edgelist=negative_edges_list, edge_color='r')
nx.draw_networkx_labels(G,pos,node_labels)
```

We can create a scaled Jacobian $M$

```
sz = G.order()

# Interaction-sign matrix
Mq = np.zeros(shape=(sz,sz))
Mq[tuple(zip(*positive_edges_list))] = 1
# i.e. Mq[((4, 4, 2, 2, 3, 3), (2, 3, 3, 1, 1, 0))]
Mq[tuple(zip(*negative_edges_list))] = -1

# Assign random uniform values to M
M = np.multiply(np.random.random_sample((sz, sz)), Mq)
eigM = np.linalg.eigvals(M)

print 'M = \n', M, '\n'
print 'eigenvalues of M: \n', eigM
```

    M =
    [[-0.58152272  0.          0.         -0.14127391  0.        ]
     [ 0.         -0.80860701 -0.57168714 -0.59986014  0.        ]
     [ 0.          0.5646323  -0.55274168  0.50760127 -0.59904304]
     [ 0.16858447  0.95959136 -0.85644885 -0.06868719 -0.24935855]
     [ 0.          0.          0.03009555  0.53615731 -0.07643393]]

    eigenvalues of M:
    [-0.49151243+1.23541285j -0.49151243-1.23541285j  0.18055314+0.j
     -0.61868079+0.j         -0.66684001+0.j        ]

This corresponds to a whole family of possible systems. For example, we could define them by their steady state values:

```
n1 = np.random.random_sample(sz)
n2 = np.random.random_sample(sz)
print n1, '\n'
print n2, '\n'
```

    [ 0.49459123  0.42365168  0.70069605  0.5194794   0.60422636]

    [ 0.83482282  0.83135622  0.88581108  0.33283174  0.1965191 ]

```
N1 = np.diag(n1)
N2 = np.diag(n2)
A1 = np.dot(M, np.linalg.inv(N1))
A2 = np.dot(M, np.linalg.inv(N2))

print A1, '\n'
print A2, '\n'
```

    [[-1.17576432  0.          0.         -0.27195286  0.        ]
     [ 0.         -1.90865998 -0.81588463 -1.15473326  0.        ]
     [ 0.          1.33277485 -0.78884658  0.97713455 -0.99142157]
     [ 0.34085617  2.26504792 -1.22228296 -0.13222313 -0.41269063]
     [ 0.          0.          0.04295093  1.03210506 -0.12649883]]

    [[-0.6965822   0.          0.         -0.42446044  0.        ]
     [ 0.         -0.97263602 -0.64538269 -1.80229244  0.        ]
     [ 0.          0.67917012 -0.623995    1.52509872 -3.04826878]
     [ 0.20194042  1.15424812 -0.96685272 -0.20637211 -1.26887693]
     [ 0.          0.          0.03397513  1.61089596 -0.38893892]]

They have equal eigenvalues at the steady state

```
J1 = np.dot(N1,A1)
J2 = np.dot(N2,A2)

eigM = np.linalg.eigvals(M)
eigJ1 = np.linalg.eigvals(J1)
eigJ2 = np.linalg.eigvals(J2)

print eigM, '\n'
print eigJ1, '\n'
print eigJ2, '\n'
```

    [-0.49151243+1.23541285j -0.49151243-1.23541285j  0.18055314+0.j
     -0.61868079+0.j         -0.66684001+0.j        ]

    [-0.49151243+1.23541285j -0.49151243-1.23541285j  0.18055314+0.j
     -0.66684001+0.j         -0.61868079+0.j        ]

    [-0.49151243+1.23541285j -0.49151243-1.23541285j -0.61868079+0.j
     -0.66684001+0.j          0.18055314+0.j        ]

The sign structure of the sensitivity matrix is also shared

```
print np.linalg.inv(-M), '\n'
print np.linalg.inv(-J1), '\n'
print np.linalg.inv(-J2), '\n'
```

    [[ 1.74245556  0.01785382 -0.10828187  0.0787587   0.59170512]
     [-0.16038225  0.69565713  0.16539543 -0.55322963  0.50859035]
     [ 0.32546384  0.84236854 -0.70162248  1.12266937  1.83629001]
     [-0.09398404 -0.07349128  0.4457183  -0.3241927  -2.43562293]
     [-0.53111518 -0.1838365   2.85029731 -1.83205217 -3.27881743]]

    [[ 1.74245556  0.0208434  -0.07643152  0.07498538  0.48434193]
     [-0.13737852  0.69565713  0.10000064 -0.45117605  0.35659675]
     [ 0.46109033  1.39323018 -0.70162248  1.5143045   2.12946878]
     [-0.09871338 -0.09011461  0.33044496 -0.3241927  -2.09400983]
     [-0.6488465  -0.26219384  2.45787708 -2.13092995 -3.27881743]]

    [[ 1.74245556  0.01792826 -0.10204905  0.19754593  2.51359249]
     [-0.15971627  0.69565713  0.15522782 -1.38187209  2.1515453 ]
     [ 0.34534212  0.89754471 -0.70162248  2.987915    8.27708869]
     [-0.03747007 -0.02942208  0.16747273 -0.3241927  -4.12505766]
     [-0.12502566 -0.04345596  0.63234462 -1.08172749 -3.27881743]]

The systems in the family also share their $\mathbf{r}$, so whether this family of systems make sense or not depends upon what kinds of assumptions we're willing to make about the intrinsic growth rates. Perhaps for some top predators we might like to require that $r_i \leq 0$

```
r1 = -np.dot(A1,n1)
r2 = -np.dot(A2,n2)

print r1
print r2
```

    [ 0.72279663  1.98015428  0.07955115  0.04631876 -0.48981893]
    [ 0.72279663  1.98015428  0.07955115  0.04631876 -0.48981893]

Here is another scaled system, this time assuming that some scaling factor has been applied to the $M$ (for example, to keep its values between zero and 1). The sign-structure of the eigenvalues, the sensitivity matrix, and the intrinsic growth rates $\mathbf{r}$ is preserved. Given that our concerns are usually about the signs of these attributes, we can say that our $M$s search the entire space of possible outcomes in this regard.

```
alpha = 42
Mf = alpha*M # Just apply some scaling
n3 = np.random.random_sample(sz)
N3 = np.diag(n3)
A3 = np.dot(Mf, np.linalg.inv(N3))
r3 = -np.dot(A3,n3)
J3 = np.dot(N3,A3)

print 'eiganvalues: \n', eigJ1
print np.linalg.eigvals(J3)/alpha, '\n'
print 'r vector: \n', r1
print r3/alpha
```

    eiganvalues:
    [-0.49151243+1.23541285j -0.49151243-1.23541285j  0.18055314+0.j
     -0.66684001+0.j         -0.61868079+0.j        ]
    [-0.49151243+1.23541285j -0.49151243-1.23541285j  0.18055314+0.j
     -0.61868079+0.j         -0.66684001+0.j        ]

    r vector:
    [ 0.72279663  1.98015428  0.07955115  0.04631876 -0.48981893]
    [ 0.72279663  1.98015428  0.07955115  0.04631876 -0.48981893]

## Species deletion

### Effect on steady state

The steady state of the full system $-\mathbf{r} = A \mathbf{n}$
may also be expressed as

\begin{equation}
-\mathbf{r}_{i\neq k} = A_{i \neq k,j \neq k}
\mathbf{n}_{i\neq k} +
A_{i \neq k,k} n_k
\end{equation}

where e.g. $A_{i \neq k,j \neq k}$ denotes the original $A$ matrix with its $k^{\text{th}}$ rows and columns removed.

A vector of the steady-state population size for all but the $k^{\text{th}}$ species can then be found

\begin{equation}
\mathbf{n}_{i\neq k} = (A_{i \neq k,j\neq k})^{-1}
\left(
    - \mathbf{r}_{i\neq k} -
    n_k A_{i \neq k, k}
\right)
\end{equation}

We are interested in removal of the $k^{\text{th}}$ species. The steady state of the subsystem formed by removing species
$k$ is

\begin{equation}
\mathbf{s} =
(A_{i \neq k,j \neq k})^{-1}
\left(
-\mathbf{r}_{i\neq k} \right)
\end{equation}

We wish to know
if the removal of species $k$ caused any other species to go
extinct. Species extinction occurs when

\begin{equation}
\begin{split}
s_i &< 0 \\
\delta_i = \frac{s_i - n_i}{n_i} &< -1
\end{split}
\end{equation}

Taking the difference between the original full system and the system that results after removal of species $k$

\begin{equation}
\begin{split}
\mathbf{s} - \mathbf{n}_{i \neq k}
& = A^{-1}_{i \neq k,j \neq k} \left(
    n_k A_{i\neq k, k} \right) \\
& = A^{-1}_{i \neq k,j \neq k}
    M_{i\neq k,k}
\end{split}
\end{equation}

Because $A^{-1} = N M^{-1}$ and because $N$ is a diagonal matrix then $A^{-1}_{i\neq k, j \neq k} =
N_{i\neq k, j \neq k}(M_{i\neq k, j \neq k})^{-1}$. Therefore

\begin{equation}
\mathbf{s} - \mathbf{n}_{i \neq k}
= N_{i\neq k, j \neq k}(M_{i\neq k, j \neq k})^{-1}
    M_{i\neq k,k}
\end{equation}

and so the effect of the removal of species $k$ on the steady state of the remaining species in the system can be found for all systems conforming to $M$ from the elements of $M$ alone

\begin{equation}
\delta_i
= \left[ (M_{i\neq k, j \neq k})^{-1}
    M_{i\neq k,k} \right]_i
\end{equation}

#### Example of effect on steady state

First we'll verify the re-expression of the steady state

```
k = 2 # Our species to remove

# Removal of kth row and column from system 1 from above
A_notk = np.delete(A1, (k), axis=0)
A_notk_k = A_notk[:,k]
A_notk = np.delete(A_notk, (k), axis=1)
r_notk = np.delete(r1, (k), axis=0)

print 'A = \n', A1, '\n'
print 'A_notk = \n', A_notk, '\n'

# Recalculate n
n_notk = np.linalg.solve(A_notk,-r_notk - n1[k]*A_notk_k)

print 'n = \n', n1, '\n'
print 'n_notk = \n', n_notk, '\n'
```

    A =
    [[-1.17576432  0.          0.         -0.27195286  0.        ]
     [ 0.         -1.90865998 -0.81588463 -1.15473326  0.        ]
     [ 0.          1.33277485 -0.78884658  0.97713455 -0.99142157]
     [ 0.34085617  2.26504792 -1.22228296 -0.13222313 -0.41269063]
     [ 0.          0.          0.04295093  1.03210506 -0.12649883]]

    A_notk =
    [[-1.17576432  0.         -0.27195286  0.        ]
     [ 0.         -1.90865998 -1.15473326  0.        ]
     [ 0.34085617  2.26504792 -0.13222313 -0.41269063]
     [ 0.          0.          1.03210506 -0.12649883]]

    n =
    [ 0.49459123  0.42365168  0.70069605  0.5194794   0.60422636]

    n_notk =
    [ 0.49459123  0.42365168  0.5194794   0.60422636]

We can find the new steady state after removal of species $k$.

```
s = np.linalg.solve(A_notk, -r_notk)

print 's = \n', s, '\n'
```

    s =
    [ 0.41826063  0.52352028  0.84948803  3.05885802]

The scaled changes in the steady-state $\delta$ can be found directly

```
delta = (s - n_notk)/n_notk

print 'delta = \n', delta, '\n'
```

    delta =
    [-0.15433067  0.2357328   0.63526799  4.06243725]

and we see here that the species that have gone extinct have a $\delta_i < -1$.

These scaled changes in the steady-state population size can be found directly from our scaled $M$

```
Mf_notk = np.delete(Mf, (k), axis=0)
Mf_notk_k = Mf_notk[:,k]
Mf_notk = np.delete(Mf_notk, (k), axis=1)

delta_from_M = np.linalg.solve(Mf_notk, Mf_notk_k)

print 'delta calculated from M = \n', delta_from_M, '\n'
```

    delta calculated from M =
    [-0.15433067  0.2357328   0.63526799  4.06243725]

$M$ can also be used to find the effect of species deletion for the whole family of systems. The scaled changes $\delta$ are the same for the second system above that scaled to $M$.

```
# Removal of kth row and column from system 2 above, to compare
A_notk = np.delete(A2, (k), axis=0)
A_notk = np.delete(A_notk, (k), axis=1)
s = np.linalg.solve(A_notk, -r_notk)
n_notk = np.delete(n2, (k), axis=0)
delta2 = (s - n_notk)/n_notk

print 'delta from another system corresponding to M = \n', delta2, '\n'

```

    delta from another system corresponding to M =
    [-0.15433067  0.2357328   0.63526799  4.06243725]

### Effect on stability

The scaled changes relate $\mathbf{s}$ and $\mathbf{n}$ by

\begin{equation}
s_i = n_i (\delta_i + 1)
\end{equation}

In the original system

\begin{equation}
J_{ij} = n_i a_{ij}
\end{equation}

and in the subsystem

\begin{equation}
\begin{split}
J_{s,ij} &= s_i a_{ij} \\
&= J_{ij} (\delta_i + 1)
\end{split}
\end{equation}
so
\begin{equation}
J_s = \text{diag}(\boldsymbol{\delta}+1) J
\end{equation}

In the scaled system
\begin{equation}
m_{ij} = a_{ij} n_j
\end{equation}

and in the scaled subsystem

\begin{equation}
b_{ij} = m_{ij} (\delta_i + 1)
\end{equation}

so

\begin{equation}
B = M \text{diag}(\boldsymbol{\delta}+1)
\end{equation}

So
\begin{equation}
\begin{split}
B &= N^{-1} J N \text{diag}(\boldsymbol{\delta}+1) \\
 &= N^{-1} (\text{diag}(\boldsymbol{\delta}+1))^{-1} J_s
 N \text{diag}(\boldsymbol{\delta}+1) \\
 &= N^{-1} (\text{diag}(\boldsymbol{\delta}+1))^{-1} J_s
 \text{diag}(\boldsymbol{\delta}+1) N
\end{split}
\end{equation}
which is a similarity transform and therefore
\begin{equation}
\text{eig}(B) = \text{eig}(J_s)
\end{equation}

Also if $M$ is further scaled by $\alpha$, then $B$ and the eigenvalues are scaled by $\alpha$.

#### Example of effect on stability

```
J_s = np.dot(np.diag(s), A_notk)
print 'eig(Js) = \n', np.linalg.eigvals(J_s), '\n'

B = np.dot(Mf_notk, np.diag(delta2+1))

print 'eig(B) = \n', np.linalg.eigvals(B)/alpha, '\n'
```

    eig(Js) =
    [-0.39291970+1.46001967j -0.39291970-1.46001967j -0.71629101+0.j
     -0.48813165+0.j        ]

    eig(B) =
    [-0.39291970+1.46001967j -0.39291970-1.46001967j -0.71629101+0.j
     -0.48813165+0.j        ]

# Species addition

## Effect on steady state

We wish to know if the addition of species $k$ caused any other species to go extinct. Extinction of a resident species occurs when
\begin{equation}
\varepsilon_i = \frac{n_{i \neq k} - s_i}{s_i} < -1
\end{equation}
As before,
\begin{equation}
\mathbf{n}_{i\neq k} = (A_{i \neq k,j\neq k})^{-1}
\left( - \mathbf{r}_{i\neq k} - n_k A_{i \neq k, k}
\right)
\end{equation}
\begin{equation}
\mathbf{s} =
(A_{i \neq k,j \neq k})^{-1}
\left(
-\mathbf{r}_{i\neq k} \right)
\end{equation}
so
\begin{equation}
\begin{split}
\mathbf{n}_{i \neq k} - \mathbf{s}
 & = -(A_{i \neq k,j\neq k})^{-1} (n_k A_{i \neq k, k}) \\
 & = -(A_{i \neq k,j\neq k})^{-1} M_{i\neq k,k} \\
\end{split}
\end{equation}

Let the Jacobian of the subsystem be *** FIX THIS
$B = A_{i \neq k,j\neq k} S$ where $S$ is the diagonal matrix formed of
$\mathbf{s}$. Then, because $S$ is diagonal,
$(A_{i \neq k,j\neq k})^{-1} = S B^{-1}$. So

\begin{equation}
\mathbf{n}_{i \neq k} -
\mathbf{s}
= SB^{-1} M_{i\neq k,k}
\end{equation}
so
\begin{equation}
\varepsilon_i = [- B^{-1} M_{i\neq k,k}]_i
\end{equation}

## Invasion criteria for species $k$

At the new steady state
\begin{equation}
r_k  = -a_{kk} n_k - \sum_{j \neq k} a_{kj} n_j
\end{equation}

The new species introduced at low numbers can invade if
\begin{equation}
\begin{split}
\gamma_k
 & = r_k + \sum_{i \neq k} a_{ki} s_i > 0 \\
\gamma_k
 & = -m_{kk} - \sum_{j \neq k} m_{kj}
     \left( \frac{\varepsilon_j}{\varepsilon_j + 1} \right) > 0\\
\end{split}
\end{equation}

## Filling in the remainder of the new $M$

The old scaled Jacobian

\begin{equation}
b_{ij} = a_{ij} s_j
\end{equation}

At the new steady state, the scaled Jacobian

\begin{equation}
\begin{split}
m_{ij}
 & = a_{ij} n_j \\
 & = a_{ij} s_j(\varepsilon_j + 1) \\
 & = b_{ij} (\varepsilon_j + 1) \\
\end{split}
\end{equation}
so
\begin{equation}
M_{i \neq k, j \neq k} = B \; \text{diag}(\varepsilon + 1)
\end{equation}

## Example of successful species addition

We'll take one of the scaled Jacobians above as our starting subystem

```
B = M # Turn previous system into our new subsystem
```

We'll add one species $k$ to the final row and column, such that it feeds upon species 2 and 4, and is fed upon by species 3.

```
k = sz
Mq_notk_k = np.array([0, 0, +1, -1, +1])
```

First, we'll find values for the new final column of $M$ that will not lead to the extinction of any of the resident species

```
# Find an invasion that will not cause the extinction of
# resident species

mineps = -2

while mineps <= -1:

    M_notk_k = np.random.random_sample(sz) * Mq_notk_k
    eps = np.dot(-np.linalg.inv(B), M_notk_k)
    mineps = min(eps)

print 'eps = \n', eps
```

    eps =
    [-0.03633213  0.40438499 -0.72889433  0.14955316  1.18732908]

The new species' ability to invade the system is determined by the final row of the new $M$. We search for values that will satisfy the invasion criteria $\gamma_k > 0$

```
# Find an invasion that is successful

Mq_k_notk = -Mq_notk_k

gamma = -1

while gamma <= 0:
    m_kk = - np.random.random_sample()
    M_k_notk = np.random.random_sample(sz) * Mq_k_notk

    gamma = - m_kk - sum(M_k_notk * (eps/(1+eps)))

print 'gamma = ', gamma

```

    gamma =  0.842156209921

We can now construct the complete $M$, with which we may wish to test for stability or further invasions / removals.

```
# Construct the new M
M = np.dot(B, np.diag(eps+1))

M = np.concatenate((M, np.array([M_notk_k]).T), axis=1)
M_k_all = np.concatenate( (np.array([M_k_notk]), np.array([[m_kk]])), axis=1)
M = np.concatenate( (M, M_k_all), axis=0)

print 'M = \n', M, '\n'
print 'Stability test: eig(M) = \n', np.linalg.eigvals(M), '\n'
```

    M =
    [[-0.56039476  0.          0.         -0.16240187  0.          0.        ]
     [ 0.         -1.13559554 -0.15498763 -0.68957112  0.          0.        ]
     [ 0.          0.79296113 -0.1498514   0.58351464 -1.31030426  0.00412874]
     [ 0.16245944  1.34763569 -0.23218814 -0.07895958 -0.54542922 -0.69983696]
     [ 0.          0.          0.00815907  0.61634134 -0.16718615  0.03250467]
     [ 0.          0.         -0.05252907  0.85987774 -0.72924792 -0.69940172]]

    Stability test: eig(M) =
    [-0.52642204+1.43510357j -0.52642204-1.43510357j  0.18233554+0.j
     -0.49341171+0.j         -0.56296611+0.j         -0.86450279+0.j        ]

To show that this is consistent with the original system, we can choose one of the original unscaled systems in the same way as in the previous section. First we can look at $r$ (the value of $r_k$ might be a constraint on our invader)

```
# Find an original system to compare it to

r1 = -np.sum(B, axis=1)
r2 = -np.sum(M, axis=1)

print 'r from B \n', r1, '\n'
print 'r from M \n', r2, '\n'
```

    r from B
    [ 0.72279663  1.98015428  0.07955115  0.04631876 -0.48981893]

    r from M
    [ 0.72279663  1.98015428  0.07955115  0.04631876 -0.48981893  0.62130097]

Here is one of the possible unscaled systems, derived from the subsystem, a $s$ and $A$ pairing

```
s = np.random.random_sample(sz)
S = np.diag(s)

A_notk = np.dot(B, np.linalg.inv(S))

print 's = ', s, '\n'
print 'Old A = \n', A_notk
```

    s =  [ 0.77594037  0.80386546  0.94726495  0.54612625  0.10132065]

    Old A =
    [[-0.74944254  0.          0.         -0.25868361  0.        ]
     [ 0.         -1.00589844 -0.60351345 -1.09839097  0.        ]
     [ 0.          0.70239653 -0.58351328  0.92945774 -5.91234908]
     [ 0.21726472  1.19372135 -0.90412808 -0.12577163 -2.46108328]
     [ 0.          0.          0.03177099  0.9817461  -0.75437661]]

We'll use this information and the change in the steady state to get our new $n$ and from that the extended $A$

```
# Choose a_kk such that n_k positive, find new A
n_k = np.random.random_sample()
n = np.append(s*(eps+1), n_k)
N = np.diag(n)
A = np.dot(M, np.linalg.inv(N))

print 'New A = \n', A, '\n'
print 'Stability test: eig(J) = \n', np.linalg.eigvals(np.dot(N,A)),
```

    New A =
    [[-0.74944254  0.          0.         -0.25868361  0.          0.        ]
     [ 0.         -1.00589844 -0.60351345 -1.09839097  0.          0.        ]
     [ 0.          0.70239653 -0.58351328  0.92945774 -5.91234908  0.04953504]
     [ 0.21726472  1.19372135 -0.90412808 -0.12577163 -2.46108328 -8.39636731]
     [ 0.          0.          0.03177099  0.9817461  -0.75437661  0.38997823]
     [ 0.          0.         -0.20454535  1.36966575 -3.29050922 -8.39114545]]

    Stability test: eig(J) =
    [-0.52642204+1.43510357j -0.52642204-1.43510357j  0.18233554+0.j
     -0.49341171+0.j         -0.56296611+0.j         -0.86450279+0.j        ]

```

```


# Transformada de Fourier de alguns sinais

Neste notebook avaliaremos a transformada de Fourier de alguns sinais. A TF é definida por

\begin{equation}
X(\mathrm{j} \omega)=\int^{\infty}_{-\infty}x(t)e^{-\mathrm{j}\omega t}\mathrm{d}t
\tag{1}
\end{equation}

E a TF inversa por:

\begin{equation}
x(t)=\frac{1}{2 \pi}\int^{\infty}_{-\infty}X(\mathrm{j} \omega)e^{\mathrm{j}\omega t}\mathrm{d}\omega
\tag{2}
\end{equation}

```python
# importar as bibliotecas necessárias
import numpy as np # arrays
import matplotlib.pyplot as plt # plots
plt.rcParams.update({'font.size': 14})
```

### Exemplo 1

$x(t) = \delta(t)$

neste caso

$X(\mathrm{j} \omega)=1$

```python
freq = np.linspace(-50, 50, 1000)
w = 2*np.pi*freq

Xjw = np.ones(len(w))

plt.figure(figsize=(15, 5))
plt.subplot(1,2,1)
plt.plot(freq, np.abs(Xjw), '-b', linewidth = 2)
plt.grid(linestyle = '--', which='both')
plt.xlabel('Frequência [Hz]')
plt.ylabel(r'$|X(\mathrm{j}\omega)|$ [-]')
plt.xticks(np.arange(freq[0], freq[-1]+10, 10))
plt.xlim((freq[0], freq[-1]))

plt.subplot(1,2,2)
plt.plot(freq, np.angle(Xjw), '-b', linewidth = 2)
plt.grid(linestyle = '--', which='both')
plt.xlabel('Frequência [Hz]')
plt.ylabel(r'$\angle X(\mathrm{j}\omega)$ [rad]')
plt.xticks(np.arange(freq[0], freq[-1]+10, 10))
plt.xlim((freq[0], freq[-1]))
plt.tight_layout()
plt.show()
```

### Exemplo 2

$x(t) = \mathrm{e}^{-at}u(t)$, $a>0$

neste caso

$X(\mathrm{j} \omega)=\frac{1}{a+\mathrm{j}\omega}$

```python
time = np.linspace(-1, 1, 1000)
xt = np.zeros(len(time))
a = 2
xt[time>=0] = np.exp(-a*time[time>=0])

# Time domain
plt.figure(figsize=(20, 2.5))
plt.plot(time, xt, '-b', linewidth = 2)
plt.grid(linestyle = '--', which='both')
plt.xlabel('Tempo [s]')
plt.ylabel(r'$x(t)$ [-]')
plt.xticks(np.arange(time[0], time[-1]+0.1, 0.1))
plt.xlim((time[0], time[-1]))

# Freq domain
freq = np.linspace(-5, 5, 10000)
w = 2*np.pi*freq

Xjw = 1/(a+1j*w)

plt.figure(figsize=(15, 5))
plt.subplot(1,2,1)
plt.plot(freq, np.abs(Xjw)/np.amax(np.abs(Xjw)), '-b', linewidth = 2)
plt.grid(linestyle = '--', which='both')
plt.xlabel('Frequência [Hz]')
plt.ylabel(r'$|X(\mathrm{j}\omega)|$ [-]')
plt.xticks(np.arange(freq[0], freq[-1]+1, 1))
plt.xlim((freq[0], freq[-1]))
plt.ylim((0, 1.2))

plt.subplot(1,2,2)
plt.plot(freq, np.angle(Xjw), '-b', linewidth = 2)
plt.grid(linestyle = '--', which='both')
plt.xlabel('Frequência [Hz]')
plt.ylabel(r'$\angle X(\mathrm{j}\omega)$ [rad]')
plt.xticks(np.arange(freq[0], freq[-1]+1, 1))
plt.xlim((freq[0], freq[-1]))
plt.tight_layout()
plt.show()
```

```python
time = np.linspace(-5, 5, 1000)
xt = np.ones(len(time))
Tp = 5
xt[time<=-Tp/2] = 0
xt[time>=Tp/2] = 0

# Time domain
plt.figure(figsize=(20, 2.5))
plt.plot(time, xt, '-b', linewidth = 2)
plt.grid(linestyle = '--', which='both')
plt.xlabel('Tempo [s]')
plt.ylabel(r'$x(t)$ [-]')
plt.xticks(np.arange(time[0], time[-1]+1, 1))
plt.xlim((time[0], time[-1]))
```

```python
time = np.linspace(-5, 5, 1000)
xt = np.cos(2*np.pi*time)
Tp = 1
xt[time<=-Tp] = 0
xt[time>=Tp] = 0

# Time domain
plt.figure(figsize=(20, 2.5))
plt.plot(time, xt, '-b', linewidth = 2)
plt.grid(linestyle = '--', which='both')
plt.xlabel('Tempo [s]')
plt.ylabel(r'$x(t)$ [-]')
plt.xticks(np.arange(time[0], time[-1]+1, 1))
plt.xlim((time[0], time[-1]))
```

```python

```

Aula 24 - TF de alguns sinais/.ipynb_checkpoints/TF alguns espectros-checkpoint.ipynb


# Finite difference operator notation

$$
\begin{equation}
u'(t_n) \approx
\lbrack D_tu\rbrack^n = \frac{u^{n+\frac{1}{2}} - u^{n-\frac{1}{2}}}{\Delta t} \label{Dop:fd1:center} \tag{1}
\end{equation}
$$

$$
\begin{equation}
u'(t_n) \approx
\lbrack D_{2t}u\rbrack^n = \frac{u^{n+1} - u^{n-1}}{2\Delta t}
\label{Dop:fd1:center2} \tag{2}
\end{equation}
$$

$$
\begin{equation}
u'(t_n) =
\lbrack D_t^-u\rbrack^n = \frac{u^{n} - u^{n-1}}{\Delta t}
\label{Dop:fd1:bw} \tag{3}
\end{equation}
$$

$$
\begin{equation}
u'(t_n) \approx
\lbrack D_t^+u\rbrack^n = \frac{u^{n+1} - u^{n}}{\Delta t}
\label{Dop:fd1:fw} \tag{4}
\end{equation}
$$

$$
\begin{equation}
u'(t_{n+\theta}) =
\lbrack \bar D_tu\rbrack^{n+\theta} = \frac{u^{n+1} - u^{n}}{\Delta t}
\label{Dop:fd1:theta} \tag{5}
\end{equation}
$$

$$
\begin{equation}
u'(t_n) \approx
\lbrack D_t^{2-}u\rbrack^n = \frac{3u^{n} - 4u^{n-1} + u^{n-2}}{2\Delta t}
\label{Dop:fd1:bw2} \tag{6}
\end{equation}
$$

$$
\begin{equation}
u''(t_n) \approx
\lbrack D_tD_t u\rbrack^n = \frac{u^{n+1} - 2u^{n} + u^{n-1}}{\Delta t^2}
\label{Dop:fd2:center} \tag{7}
\end{equation}
$$

$$
\begin{equation}
u(t_{n+\frac{1}{2}}) \approx \lbrack \overline{u}^{t}\rbrack^{n+\frac{1}{2}}
= \frac{1}{2}(u^{n+1} + u^n)
\label{Dop:avg:arith} \tag{8}
\end{equation}
$$

$$
\begin{equation}
u(t_{n+\frac{1}{2}})^2 \approx \lbrack \overline{u^2}^{t,g}\rbrack^{n+\frac{1}{2}}
= u^{n+1}u^n
\label{Dop:avg:geom} \tag{9}
\end{equation}
$$

$$
\begin{equation}
u(t_{n+\frac{1}{2}}) \approx \lbrack \overline{u}^{t,h}\rbrack^{n+\frac{1}{2}}
= \frac{2}{\frac{1}{u^{n+1}} + \frac{1}{u^n}}
\label{Dop:avg:harm} \tag{10}
\end{equation}
$$

$$
\begin{equation}
u(t_{n+\theta}) \approx \lbrack \overline{u}^{t,\theta}\rbrack^{n+\theta}
= \theta u^{n+1} + (1-\theta)u^n ,
\label{Dop:avg:theta} \tag{11}
\end{equation}
$$

$$
\begin{equation}
\qquad t_{n+\theta}=\theta t_{n+1} + (1-\theta)t_{n-1}
\label{_auto1} \tag{12}
\end{equation}
$$

Some may wonder why $\theta$ is absent on the right-hand side
of ([5](#Dop:fd1:theta)). The fraction is an approximation to the
derivative at the point $t_{n+\theta}=\theta t_{n+1} + (1-\theta) t_{n}$.

# Truncation errors of finite difference approximations

$$
u'(t_n) =
[D_tu]^n + R^n = \frac{u^{n+\frac{1}{2}} - u^{n-\frac{1}{2}}}{\Delta t} +R^n\nonumber,
$$

$$
\begin{equation}
R^n = -\frac{1}{24}u'''(t_n)\Delta t^2 + {\cal O}(\Delta t^4)
\label{form:trunc:fd1:center} \tag{13}
\end{equation}
$$

$$
u'(t_n) =
[D_{2t}u]^n +R^n = \frac{u^{n+1} - u^{n-1}}{2\Delta t} +
R^n\nonumber,
$$

$$
\begin{equation}
R^n = -\frac{1}{6}u'''(t_n)\Delta t^2 + {\cal O}(\Delta t^4)
\label{form:trunc:fd1:center2} \tag{14}
\end{equation}
$$

$$
u'(t_n) =
[D_t^-u]^n +R^n = \frac{u^{n} - u^{n-1}}{\Delta t}
+R^n\nonumber,
$$

$$
\begin{equation}
R^n = -\frac{1}{2}u''(t_n)\Delta t + {\cal O}(\Delta t^2)
\label{form:trunc:fd1:bw} \tag{15}
\end{equation}
$$

$$
u'(t_n) =
[D_t^+u]^n +R^n = \frac{u^{n+1} - u^{n}}{\Delta t}
+R^n\nonumber,
$$

$$
\begin{equation}
R^n = \frac{1}{2}u''(t_n)\Delta t + {\cal O}(\Delta t^2)
\label{form:trunc:fd1:fw} \tag{16}
\end{equation}
$$

$$
u'(t_{n+\theta}) =
[\bar D_tu]^{n+\theta} +R^{n+\theta} = \frac{u^{n+1} - u^{n}}{\Delta t}
+R^{n+\theta}\nonumber,
$$

$$
R^{n+\theta} = -\frac{1}{2}(1-2\theta)u''(t_{n+\theta})\Delta t +
\frac{1}{6}((1 - \theta)^3 - \theta^3)u'''(t_{n+\theta})\Delta t^2 +
\nonumber
$$

$$
\begin{equation}
\quad {\cal O}(\Delta t^3)
\label{form:trunc:fd1:theta} \tag{17}
\end{equation}
$$

$$
u'(t_n) =
[D_t^{2-}u]^n +R^n = \frac{3u^{n} - 4u^{n-1} + u^{n-2}}{2\Delta t}
+R^n\nonumber,
$$

$$
\begin{equation}
R^n = \frac{1}{3}u'''(t_n)\Delta t^2 + {\cal O}(\Delta t^3)
\label{form:trunc:fd1:bw2} \tag{18}
\end{equation}
$$

$$
u''(t_n) =
[D_tD_t u]^n +R^n = \frac{u^{n+1} - 2u^{n} + u^{n-1}}{\Delta t^2}
+R^n\nonumber,
$$

$$
\begin{equation}
R^n = -\frac{1}{12}u''''(t_n)\Delta t^2 + {\cal O}(\Delta t^4)
\label{form:trunc:fd2:center} \tag{19}
\end{equation}
$$

$$
u(t_{n+\theta}) = [\overline{u}^{t,\theta}]^{n+\theta} +R^{n+\theta}
= \theta u^{n+1} + (1-\theta)u^n +R^{n+\theta},\nonumber
$$

$$
\begin{equation}
R^{n+\theta} = -\frac{1}{2}u''(t_{n+\theta})\Delta t^2\theta (1-\theta) +
{\cal O}(\Delta t^3)
\thinspace .
\label{form:trunc:avg:theta} \tag{20}
\end{equation}
$$

# Finite differences of exponential functions

### Complex exponentials

Let $u^n = \exp{(i\omega n\Delta t)} = e^{i\omega t_n}$.

$$
\begin{equation}
[D_tD_t u]^n = u^n \frac{2}{\Delta t}(\cos \omega\Delta t - 1) =
-\frac{4}{\Delta t}\sin^2\left(\frac{\omega\Delta t}{2}\right),
\label{form:exp:fd2:center} \tag{21}
\end{equation}
$$

$$
\begin{equation}
[D_t^+ u]^n = u^n\frac{1}{\Delta t}(\exp{(i\omega\Delta t)} - 1),
\label{form:exp:fd1:fw} \tag{22}
\end{equation}
$$

$$
\begin{equation}
[D_t^- u]^n = u^n\frac{1}{\Delta t}(1 - \exp{(-i\omega\Delta t)}),
\label{form:exp:fd1:bw} \tag{23}
\end{equation}
$$

$$
\begin{equation}
[D_t u]^n = u^n\frac{2}{\Delta t}i\sin{\left(\frac{\omega\Delta t}{2}\right)},
\label{form:exp:fd1:center} \tag{24}
\end{equation}
$$

$$
\begin{equation}
[D_{2t} u]^n = u^n\frac{1}{\Delta t}i\sin{(\omega\Delta t)}
\label{form:exp:fd1c:center} \tag{25}
\thinspace .
\end{equation}
$$

### Real exponentials

Let $u^n = \exp{(\omega n\Delta t)} = e^{\omega t_n}$.

$$
\begin{equation}
[D_tD_t u]^n = u^n \frac{2}{\Delta t}(\cos \omega\Delta t - 1) =
-\frac{4}{\Delta t}\sin^2\left(\frac{\omega\Delta t}{2}\right),
\label{form:rexp:fd2:center} \tag{26}
\end{equation}
$$

$$
\begin{equation}
[D_t^+ u]^n = u^n\frac{1}{\Delta t}(\exp{(i\omega\Delta t)} - 1),
\label{form:rexp:fd1:fw} \tag{27}
\end{equation}
$$

$$
\begin{equation}
[D_t^- u]^n = u^n\frac{1}{\Delta t}(1 - \exp{(-i\omega\Delta t)}),
\label{form:rexp:fd1:bw} \tag{28}
\end{equation}
$$

$$
\begin{equation}
[D_t u]^n = u^n\frac{2}{\Delta t}i\sin{\left(\frac{\omega\Delta t}{2}\right)},
\label{form:rexp:fd1:center} \tag{29}
\end{equation}
$$

$$
\begin{equation}
[D_{2t} u]^n = u^n\frac{1}{\Delta t}i\sin{(\omega\Delta t)}
\label{form:rexp:fd1c:center} \tag{30}
\thinspace .
\end{equation}
$$

# Finite differences of $t^n$

The following results are useful when checking if a polynomial term in a
solution fulfills the discrete equation for the numerical method.

$$
\begin{equation}
\lbrack D_t^+ t\rbrack^n = 1,
\label{form:Dop:tn:fw} \tag{31}
\end{equation}
$$

$$
\begin{equation}
\lbrack D_t^- t\rbrack^n = 1,
\label{form:Dop:tn:bw} \tag{32}
\end{equation}
$$

$$
\begin{equation}
\lbrack D_t t\rbrack^n = 1,
\label{form:Dop:tn:cn} \tag{33}
\end{equation}
$$

$$
\begin{equation}
\lbrack D_{2t} t\rbrack^n = 1,
\label{form:Dop:tn:2cn} \tag{34}
\end{equation}
$$

$$
\begin{equation}
\lbrack D_{t}D_t t\rbrack^n = 0
\label{form:Dop2:tn:cn} \tag{35}
\thinspace .
\end{equation}
$$

The next formulas concern the action of difference operators on a $t^2$ term.

$$
\begin{equation}
\lbrack D_t^+ t^2\rbrack^n = (2n+1)\Delta t,
\label{form:Dop:tn2:fw} \tag{36}
\end{equation}
$$

$$
\begin{equation}
\lbrack D_t^- t^2\rbrack^n = (2n-1)\Delta t,
\label{form:Dop:tn2:bw} \tag{37}
\end{equation}
$$

$$
\begin{equation}
\lbrack D_t t^2\rbrack^n = 2n\Delta t,
\label{form:Dop:tn2:cn} \tag{38}
\end{equation}
$$

$$
\begin{equation}
\lbrack D_{2t} t^2\rbrack^n = 2n\Delta t,
\label{form:Dop:tn2:2cn} \tag{39}
\end{equation}
$$

$$
\begin{equation}
\lbrack D_{t}D_t t^2\rbrack^n = 2,
\label{form:Dop2:tn2:cn} \tag{40}
\end{equation}
$$

Finally, we present formulas for a $t^3$ term:

$$
\begin{equation}
\lbrack D_t^+ t^3\rbrack^n = 3(n\Delta t)^2 + 3n\Delta t^2 + \Delta t^2,
\label{form:Dop:tn3:fw} \tag{41}
\end{equation}
$$

$$
\begin{equation}
\lbrack D_t^- t^3\rbrack^n = 3(n\Delta t)^2 - 3n\Delta t^2 + \Delta t^2,
\label{form:Dop:tn3:bw} \tag{42}
\end{equation}
$$

$$
\begin{equation}
\lbrack D_t t^3\rbrack^n = 3(n\Delta t)^2 + \frac{1}{4}\Delta t^2,
\label{form:Dop:tn3:cn} \tag{43}
\end{equation}
$$

$$
\begin{equation}
\lbrack D_{2t} t^3\rbrack^n = 3(n\Delta t)^2 + \Delta t^2,
\label{form:Dop:tn3:2cn} \tag{44}
\end{equation}
$$

$$
\begin{equation}
\lbrack D_{t}D_t t^3\rbrack^n = 6n\Delta t,
\label{form:Dop2:tn3:cn} \tag{45}
\end{equation}
$$

## Software

Application of finite difference operators to polynomials and exponential
functions, resulting in the formulas above, can easily be computed by
some `sympy` code (from the file [`lib.py`](${src_formulas}/lib.py)):

```python
from sympy import *
t, dt, n, w = symbols('t dt n w', real=True)

# Finite difference operators

def D_t_forward(u):
    return (u(t + dt) - u(t))/dt

def D_t_backward(u):
    return (u(t) - u(t-dt))/dt

def D_t_centered(u):
    return (u(t + dt/2) - u(t-dt/2))/dt

def D_2t_centered(u):
    return (u(t + dt) - u(t-dt))/(2*dt)

def D_t_D_t(u):
    return (u(t + dt) - 2*u(t) + u(t-dt))/(dt**2)

op_list = [D_t_forward, D_t_backward,
           D_t_centered, D_2t_centered, D_t_D_t]

def ft1(t):
    return t

def ft2(t):
    return t**2

def ft3(t):
    return t**3

def f_expiwt(t):
    return exp(I*w*t)

def f_expwt(t):
    return exp(w*t)

func_list = [ft1, ft2, ft3, f_expiwt, f_expwt]
```

To see the results, one can now make a simple loop over the different
types of functions and the various operators associated with them:

```python
for func in func_list:
    for op in op_list:
        f = func
        e = op(f)
        e = simplify(expand(e))
        print(e)
        if func in [f_expiwt, f_expwt]:
            e = e/f(t)
        e = e.subs(t, n*dt)
        print(expand(e))
        print(factor(simplify(expand(e))))
```

    1
    1
    1
    1
    1
    1
    1
    1
    1
    1
    1
    1
    0
    0
    0
    dt + 2*t
    2*dt*n + dt
    dt*(2*n + 1)
    -dt + 2*t
    2*dt*n - dt
    dt*(2*n - 1)
    2*t
    2*dt*n
    2*dt*n
    2*t
    2*dt*n
    2*dt*n
    2
    2
    2
    dt**2 + 3*dt*t + 3*t**2
    3*dt**2*n**2 + 3*dt**2*n + dt**2
    dt**2*(3*n**2 + 3*n + 1)
    dt**2 - 3*dt*t + 3*t**2
    3*dt**2*n**2 - 3*dt**2*n + dt**2
    dt**2*(3*n**2 - 3*n + 1)
    dt**2/4 + 3*t**2
    3*dt**2*n**2 + dt**2/4
    dt**2*(12*n**2 + 1)/4
    dt**2 + 3*t**2
    3*dt**2*n**2 + dt**2
    dt**2*(3*n**2 + 1)
    6*t
    6*dt*n
    6*dt*n
    (-exp(I*t*w) + exp(I*w*(dt + t)))/dt
    exp(I*dt*w)/dt - 1/dt
    (exp(I*dt*w) - 1)/dt
    (-exp(I*t*w) + exp(I*w*(dt + t)))*exp(-I*dt*w)/dt
    1/dt - exp(-I*dt*w)/dt
    (exp(I*dt*w) - 1)*exp(-I*dt*w)/dt
    (-exp(I*t*w) + exp(I*w*(dt + t)))*exp(-I*dt*w/2)/dt
    exp(I*dt*w/2)/dt - exp(-I*dt*w/2)/dt
    2*I*sin(dt*w/2)/dt
    (-exp(I*t*w) + exp(I*w*(2*dt + t)))*exp(-I*dt*w)/(2*dt)
    exp(I*dt*w)/(2*dt) - exp(-I*dt*w)/(2*dt)
    I*sin(dt*w)/dt
    (exp(2*I*dt*w) - 2*exp(I*dt*w) + 1)*exp(I*w*(-dt + t))/dt**2
    exp(I*dt*w)/dt**2 - 2/dt**2 + exp(-I*dt*w)/dt**2
    (exp(I*dt*w) - 1)**2*exp(-I*dt*w)/dt**2
    (-exp(t*w) + exp(w*(dt + t)))/dt
    exp(dt*w)/dt - 1/dt
    (exp(dt*w) - 1)/dt
    (-exp(t*w) + exp(w*(dt + t)))*exp(-dt*w)/dt
    1/dt - exp(-dt*w)/dt
    (exp(dt*w) - 1)*exp(-dt*w)/dt
    (-exp(t*w) + exp(w*(dt + t)))*exp(-dt*w/2)/dt
    exp(dt*w/2)/dt - exp(-dt*w/2)/dt
    2*sinh(dt*w/2)/dt
    (-exp(t*w) + exp(w*(2*dt + t)))*exp(-dt*w)/(2*dt)
    exp(dt*w)/(2*dt) - exp(-dt*w)/(2*dt)
    sinh(dt*w)/dt
    (exp(2*dt*w) - 2*exp(dt*w) + 1)*exp(w*(-dt + t))/dt**2
    exp(dt*w)/dt**2 - 2/dt**2 + exp(-dt*w)/dt**2
    (exp(dt*w) - 1)**2*exp(-dt*w)/dt**2


# DynaFu ICP Math
## Differentiating and Linearising Rt matrices

In dynafu, the warp function looks like the following for each node $i$:

$
\begin{equation*}
f_i(x_i, V_g) = T_{x_i} * V_g = R(x_i) * V_g + t(x_i)
\end{equation*}
$

where ${x_i}$ are the transformation parameters for node $i$ and the rotation is performed around the corresponding node (and not a global reference)

For linearising a transform around the parameters $\mathbf{x}$, we need to find the derivative

$
\begin{equation*}
\displaystyle
\frac{\partial f_i(\mathbf{x} \circ \epsilon,   V_g)}{\partial \epsilon} |_{\epsilon = 0}
\end{equation*}
$

We calculate this as follows:

$
\begin{equation*}
f_i(\mathbf{x} \circ \epsilon, V_g) = f_i(\epsilon, V) = T_{inc} * V
\end{equation*}
$ where $V = f_i(\mathbf{x}, V_g)$ and $T_{inc}$ is the infinitesimal transform with parameters $\epsilon$

According to Lie algebra, each Rt matrix can be represented as $A = e^\xi$ where $\xi$ are the transform parameters. Therefore,

$
\begin{equation*}
f_i(\mathbf{x}, V_g) = e^\xi V
\end{equation*}
$

Therefore,

$
\begin{equation*}
\displaystyle
\frac{\partial f_i(\mathbf{x} \circ \xi,   V_g)}{\partial \xi} |_{\xi = 0} =
\frac{\partial e^\xi V}{\partial \xi} |_{\xi=0} =
\begin{pmatrix} -[V]_{\times} & I_{3x3} \end{pmatrix}_{3 \times 6}
\end{equation*}
$

Let us denote $\begin{pmatrix} -[V]_{\times} & I_{3x3} \end{pmatrix}$ as $G(V)$ from now on.

This result is mentioned in [this manifold optimisation tutorial](http://ingmec.ual.es/~jlblanco/papers/jlblanco2010geometry3D_techrep.pdf) (equation 10.23).

With this result, we can now linearise our transformation around $\mathbf{x}$:

$
\begin{equation*}
f_i(x_i, V_g) = G(V) * \epsilon + V
\end{equation*}
$

I suppose the following is an equivalent excerpt from the dynafu paper (Section about efficient optimisation) that mentions this way of calculating derivatives:
> We formulate compositional updates $\hat x$ through the exponential map with a per-node twist $ξ_i ∈ se(3)$, requiring 6 variables per node transform, and perform linearisation  around $ξ_i=  0$.

As a side note, the derivative $\large \frac{\partial e^\xi}{\partial \xi}|_{\xi=0}$ is called the tangent (esentially the derivative) to the SE(3) manifold (the space in which Rt matrix $T_{inc}$ exists) at identity ($\xi = 0$)

## Estimating Warp Field Parameters
The total energy to be minimised is

$
E = E_{data} + \lambda E_{reg}
$

#### Data term rearrangement
$
\displaystyle
E_{data} = \sum_{u \in \Omega} \rho_{Tukey}( (T_u N_g)^T ((T_u V_g) - V_c))
$

The quadcopter paper tells us that the following expression has the same minimiser, so we can use this instead:

$
\displaystyle
E_{data} = \sum_{u \in \Omega} w_{Tukey}(r_u) \cdot (r_u)^2
$

where $w_{Tukey}(x) = \rho'(x)/x$ which behaves like a constant term and $r_u = N_g^T (V_g - T_u^{-1}\cdot V_c)$

#### Regularisation term rearrangement
$
\begin{equation}
\displaystyle
E_{reg} = \sum_{i = 0}^n \sum_{j \in \varepsilon(i)} \alpha_{ij} \rho_{Huber} (T_{i}V_g^j - T_{j}V_g^j)
\end{equation}
$

This needs to be changed to the form of weighted least squares to be useful. So incorporate the same rearrangement as the data term and sum over edges instead:

$
\begin{equation}
\displaystyle
E_{reg} = \sum_{e \in E} w_{Huber}(r_e) (r_e)^2
\end{equation}
$

Here $E$ is the set of the directed edges in the regularisation graph between all nodes from current level and the next coarser level. And $w_{Huber}(x) = \alpha_x \rho'(x)/x$

#### Obtaining normal equation

Therefore to solve an iteration, we equate the derivative with 0

$
\begin{equation*}
\large
\frac{\partial E_{data}}{\partial \xi} + \lambda \frac{\partial E_{reg}}{\partial \xi} = 0
\end{equation*}
$

which gives us

$
\begin{equation*}
J_d^T W_d(r_d + J_d\mathbf{\hat x}) + \lambda J_r^T W_r (r_r + J_r\mathbf{\hat x}) = 0
\end{equation*}
$

$
(J_d^T W_d J_d + \lambda J_r^T W_r J_r)\mathbf{\hat x} = -(J_d^T W_d r_d + \lambda J_r^T W_r r_r)
$

Here $W_d$ and $W_r$ are the weight matrices as described in quadcopter paper. However for $W_r, \alpha$ is also incorporated in this matrix

### Calculating Data Term Jacobian ($J_d$)

We estimate the inverse of Rt matrices instead of the Rt matrices themselves. So firstly we have to write $T_u^{-1} V_g$ in terms of inverse matrices. However, I realised that

$$
\begin{equation*}
T_u^{-1} V_g \ne \sum_{k \in N(V_u)} \frac{w_k T_k^{-1}V_g}{w_k}
\end{equation*}
$$

Unfortunately, I could not find a representation of $T_u^{-1} V_g$ in terms of $T_k^{-1}V_g$ and got stuck here. Below is an approach without estimating the inverse Rt matrices, I think we can use that instead as the math is now fixed.

### Alternative calculation for $J_d$
The residual $r_u$ in the data term is

$$
r_u = (T_u N_g)^T (T_u V_g - V_c)
$$

Let $a, b$ be column vectors such that $a = T_u N_g$ and $b = (T_u V_g - V_c)$. Now we can state the residual as

$$
r_u = a^Tb
$$

Each entry in $J_d$ for node paramter $x_j$ associated with node $j$ is:

$$
(J_d)_{uj} = \frac{\partial r_u}{\partial x_j} = \frac{\partial (a^Tb)}{\partial x_j}
$$

**Please note that numerator layout is assumed in all the derivatives**

Applying chain rule for multiple variables, we get

$$
\begin{equation}\begin{aligned}
\frac{\partial (a^Tb)}{\partial x_j} & =
\frac{\partial (a^Tb)}{\partial a} \cdot \frac{\partial a}{\partial x_j} +
\frac{\partial (a^Tb)}{\partial b} \cdot \frac{\partial b}{\partial x_j} \\
& =
\frac{\partial (a^Tb)}{\partial a} \cdot \frac{\partial a}{\partial x_j} +
\frac{\partial (b^Ta)}{\partial b} \cdot \frac{\partial b}{\partial x_j} && \text{Since  $a^Tb = b^Ta$} \\
& =
b^T \cdot \frac{\partial a}{\partial x_j} +
a^T \cdot \frac{\partial b}{\partial x_j} && \text{Since $\frac{\partial x^TA}{\partial x} = A^T$}
\end{aligned}\end{equation}\tag{1}\label{1}
$$

The identity $\frac{\partial x^TA}{\partial x} = A^T$ is mentioned in [this wikipedia page](https://en.wikipedia.org/wiki/Matrix_calculus#Vector-by-vector_identities). Now we calculate $\frac{\partial a}{\partial x_j}$ and $\frac{\partial b}{\partial x_j}$ as follows:

$$
\begin{equation}\begin{aligned}
\frac{\partial a}{\partial x_j} & = \frac{\partial (T_u N_g)}{\partial x_j} \\
& = \sum_{k \in N(V_u)} \frac{w_k \frac{\partial T_k N_g}{\partial x_j}}{w_k} \\
& =
\begin{cases}
 \frac{w_j \frac{\partial T_j N_g}{\partial x_j}}{\sum_{k \in N(V_u)} w_k} && \text{if $j \in N(V_u)$} \\
 0 && \text{otherwise}
\end{cases} \\
& =
\begin{cases}
 \frac{w_j \begin{pmatrix}-[T_j N_g]_\times & I_{3\times3}\end{pmatrix}}{\sum_{k \in N(V_u)} w_k} && \text{if $j \in N(V_u)$} \\
 0 && \text{otherwise}
\end{cases}
\end{aligned}\end{equation}\tag{2}\label{2}
$$

$$
\begin{equation}\begin{aligned}
\frac{\partial b}{\partial x_j} & = \frac{\partial (T_uV_g - V_c)}{\partial x_j} \\
& = \frac{\partial T_uV_g}{\partial x_j}\\
& =
\begin{cases}
 \frac{w_j \begin{pmatrix}-[T_j V_g]_\times & I_{3\times3}\end{pmatrix}}{\sum_{k \in N(V_u)} w_k} && \text{if $j \in N(V_u)$} \\
 0 && \text{otherwise}
\end{cases} && \text{Calculated similarly to ($\ref{2}$)}
\end{aligned}\end{equation}\tag{3}\label{3}
$$

Substituting equations $(\ref{2})$, $(\ref{3})$ as well as the values of $a^T$ and $b^T$ in $(\ref{1})$, we obtain the required result:

$$
(J_d)_{uj} =
\begin{cases}
\frac{w_j}{\sum_{k \in N(V_u)} w_k}
\left(
  (T_u V_g - V_c)^T
  \begin{pmatrix}-[T_j N_g] & I_{3\times3}\end{pmatrix} +
  (T_u N_g)^T \begin{pmatrix}-[T_j V_g] & I_{3\times3}\end{pmatrix}
\right) & \text{if} j \in N(V_u) \\
0 & \text{otherwise}
\end{cases}
$$

We can simplify this expression further:

$$
\begin{equation*}
\left(
  (T_u V_g - V_c)^T \begin{pmatrix}-[T_j N_g] & I_{3\times3}\end{pmatrix} +
  (T_u N_g)^T \begin{pmatrix}-[T_j V_g] & I_{3\times3}\end{pmatrix}
\right) = \\
\left [
(T_u V_g - V_c)^T \left ( -[T_j N_g]_\otimes \right ) \mid (T_u V_g - V_c)^T
 \right ]
+
\left [
(T_u N_g)^T \left ( -[T_j V_g]_\otimes \right ) \mid (T_u N_g)^T
 \right ] = \\
\left [
(T_u V_g - V_c)^T \left ( -[T_j N_g]_\otimes \right ) +
(T_u N_g)^T \left ( -[T_j V_g]_\otimes \right )
    \mid (T_u V_g + T_u N_g - V_c )^T
\right ] = \\
\left [
(T_u V_g - V_c)^T  ( [T_j N_g]_\otimes )^T +
(T_u N_g)^T ( [T_j V_g]_\otimes )^T
    \mid (T_u V_g + T_u N_g - V_c )^T
\right ] = \\
\begin{bmatrix}
( [T_j N_g]_\otimes )(T_u V_g - V_c) +
( [T_j V_g]_\otimes )(T_u N_g)
\\
T_u V_g + T_u N_g - V_c
\end{bmatrix}^T = \\
\begin{bmatrix}
( T_j N_g) \times (T_u V_g - V_c) +
( T_j V_g) \times (T_u N_g)
\\
T_u V_g + T_u N_g - V_c
\end{bmatrix}^T = \\
\begin{bmatrix}
( T_j N_g) \times (T_u V_g - V_c) +
( T_j V_g) \times (T_u N_g)
\\
T_u V_g + T_u N_g - V_c
\end{bmatrix}^T
\end{equation*}
$$

So, we get the final expression as:
$$
\begin{equation*}
(J_d)_{uj} =
\begin{cases}
\frac{w_j}{\sum_{k \in N(V_u)} w_k}
\begin{bmatrix}
( T_j N_g) \times (T_u V_g - V_c) +
( T_j V_g) \times (T_u N_g)
\\
T_u V_g + T_u N_g - V_c
\end{bmatrix}^T
&
\text{if} j \in N(V_u) \\
0 & \text{otherwise}
\end{cases}
\end{equation*}
$$

Now that we have expression for the Jacobian, we can form the normal equation corresponding to the data term of the ICP

$$
\begin{equation*}
\\
\Omega \text{ pixels, N nodes}
\\
J_d^T W_d J_d \mathbf{\hat x} = -J_d^T W_d r_d \\
\underbrace{J_d^T}_{6N \times\Omega} \underbrace {W_d}_{\Omega \times \Omega}
\underbrace{J_d}_{\Omega \times 6N} \underbrace{\mathbf{\hat x}}_{6N \times 1} =
-J_d^T W_d \underbrace{r_d}_{\Omega \times 1} \\
\underbrace{\mathbf{A}_d}_{6N \times 6N} \mathbf{\hat x} = \underbrace{\mathbf{b}_d}_{6N \times 1} \\
\\
\text {for each block (i, j) which are 6}\times\text{6:}
\\
(\mathbf{A}_d)_{ij} = \sum_{\Omega} \underbrace{w_d(\Omega)}_{\text{robust for pix}}
\left(\frac{\partial E_d}{\partial x_i}\right)^T_\Omega \left(\frac{\partial E_d}{\partial x_j}\right)_\Omega \\
\\
\text {for each block j which are 6}\times\text{1:}
\\
(\mathbf{b}_d)_{j} = - \sum_{\Omega} \underbrace{w_d(\Omega)}_{\text{robust for pix}} r_d(\Omega)
\left(\frac{\partial E_d}{\partial x_j}\right)^T_\Omega \\
(\mathbf{b}_d)_{j} = - \sum_{\Omega} \underbrace{w_d(\Omega)}_{\text{robust for pix}} ((T_u N_g)^T (T_u V_g - V_c))_{\Omega}
\left(
    \frac{w_j \text{ or 0} }{\sum_{k \in N(V_u)} w_k}
\begin{bmatrix}
( T_j N_g) \times (T_u V_g - V_c) +
( T_j V_g) \times (T_u N_g)
\\
T_u V_g + T_u N_g - V_c
\end{bmatrix}
    \right)_\Omega \\
(\mathbf{A}_d)_{ij} = \sum_{\Omega} \underbrace{w_d(\Omega)}_{\text{robust for pix}}
\left(
\frac{(w_i\text{ or 0})(w_j\text{ or 0})}{(\sum_{k \in N(V_u)} w_k)^2}
    \underbrace{(A^T_{i} A_{j})}_{6 \times 6}
\right)_{\Omega}
\\ \\
A_{i} =
\begin{bmatrix}
( T_i N_g) \times (T_u V_g - V_c) +
( T_i V_g) \times (T_u N_g)
\\
T_u V_g + T_u N_g - V_c
\end{bmatrix}
\end{equation*}
$$

### Calculating Regularisation Term Jacobian ($J_r$)

Each row in $J_r$ corresponds to derivative of summand for each edge $e$ in the regularisation graph $\epsilon$ and column $k$ corresponds to node $k$ with respect to which the derivative is calculated.

$$
\begin{equation*}
\displaystyle
(J_r)_{ek} =
\sum_{e_{ij} \in \epsilon} \frac{\partial ( T_iV_g^j - T_jV_g^j)}{\partial x_k}
=
\sum_{e_{ij} \in \epsilon} \begin{cases}
\begin{pmatrix} -[T_iV_g^j] & I_{3x3} \end{pmatrix} & \text {if   }  i = k \\
-\begin{pmatrix} -[T_jV_g^j] & I_{3x3} \end{pmatrix} & \text {if  }  j = k \\
0 & \text {otherwise}
\end{cases}
\end{equation*}
$$

Using this Jacobian we can set up a normal equation corresponding to the regularisation term similarly to the data term as mentioned in the previous section


# $ \log(1+x) $ のテーラー展開

WolframAlphaで `taylor series log(1+x)` の方が簡単と思われる

```
from sympy import *
from sympy.abc import x
init_printing()
```

```
series(log(1+x), x)
```


# Realization of Non-Recursive Filters

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Fast Convolution

The straightforward convolution of two finite-length signals $x[k]$ and $h[k]$ is a numerically complex task. This has led to the development of various techniques with considerably lower complexity. The basic concept of the *fast convolution* is to exploit the correspondence between the convolution and the scalar multiplication in the frequency domain.

### Convolution of Finite-Length Signals

The convolution of a causal signal $x_L[k]$ of length $L$ with a causal impulse response $h_N[k]$ of length $N$ is given as

\begin{equation}
y[k] = x_L[k] * h_N[k] = \sum_{\kappa = 0}^{L-1} x_L[\kappa] \; h_N[k - \kappa] = \sum_{\kappa = 0}^{N-1} h_N[\kappa] \; x_L[k - \kappa]
\end{equation}

where $x_L[k] = 0$ for $k<0 \wedge k \geq L$ and $h_N[k] = 0$ for $k<0 \wedge k \geq N$. The resulting signal $y[k]$ is of finite length $M = N+L-1$. The computation of $y[k]$ for $k=0,1, \dots, M-1$ requires $M \cdot N$ multiplications and $M \cdot (N-1)$ additions. The computational complexity of the convolution is consequently [in the order of](https://en.wikipedia.org/wiki/Big_O_notation) $\mathcal{O}(M \cdot N)$. Discrete-time Fourier transformation (DTFT) of above relation yields

\begin{equation}
Y(e^{j \Omega}) = X_L(e^{j \Omega}) \cdot H_N(e^{j \Omega})
\end{equation}

Discarding the effort of transformation, the computationally complex convolution is replaced by a scalar multiplication with respect to the frequency $\Omega$. However, $\Omega$ is a continuous frequency variable which limits the numerical evaluation of this scalar multiplication. In practice, the DTFT is replaced by the discrete Fourier transformation (DFT). Two aspects have to be considered before a straightforward application of the DFT

1. The DFTs $X_L[\mu]$ and $H_N[\mu]$ are of length $L$ and $N$ respectively and cannot be multiplied straightforwardly

2. For $N = L$, the multiplication of the two spectra $X_L[\mu]$ and $H_L[\mu]$ would result in the [periodic/circular convolution](https://en.wikipedia.org/wiki/Circular_convolution) $x_L[k] \circledast h_L[k]$ due to the periodicity of the DFT. Since we aim at realizing the linear convolution $x_L[k] * h_N[k]$ with the DFT, special care has to be taken to avoid cyclic effects.

### Linear Convolution by Periodic Convolution

The periodic convolution of the two signals $x_L[k]$ and $h_N[k]$ is defined as

\begin{equation}
x_L[k] \circledast h_N[k] = \sum_{\kappa=0}^{M-1} \tilde{x}_M[k - \kappa] \; \tilde{h}_M[\kappa]
\end{equation}

where the periodic continuations $\tilde{x}_M[k]$ of $x_L[k]$ and $\tilde{h}_M[k]$ of $h_N[k]$ with period $M$ are given as

\begin{align}
\tilde{x}_M[k] &= \sum_{m = -\infty}^{\infty} x_L[m \cdot M + k] \\
\tilde{h}_M[k] &= \sum_{m = -\infty}^{\infty} h_N[m \cdot M + k]
\end{align}

The result of the circular convolution has a periodicity of $M$.

To compute the linear convolution by the periodic convolution one has to take care that the result of the linear convolution fits into one period of the periodic convolution. Hence, the periodicity has to be chosen as $M \geq N+L-1$. This can be achieved by zero-padding of $x_L[k]$ and $h_N[k]$ to a total length of $M$

\begin{align}
x_M[k] &= \begin{cases}
x_L[k] & \mathrm{for} \; k=0, 1, \dots, L-1 \\
0 & \mathrm{for} \; k=L, L+1, \dots, M-1
\end{cases}
\\
h_M[k] &= \begin{cases}
h_N[k] & \mathrm{for} \; k=0, 1, \dots, N-1 \\
0 & \mathrm{for} \; k=N, N+1, \dots, M-1
\end{cases}
\end{align}

This results in the desired equality of linear and periodic convolution

\begin{equation}
x_L[k] * h_N[k] = x_M[k] \circledast h_M[k]
\end{equation}

for $k = 0,1,\dots, M-1$ with $M = N+L-1$.

#### Example - Linear by periodic convolution

The following example computes the linear, periodic and linear by periodic convolution of a rectangular signal $x[k] = \text{rect}_L[k]$ of length $L$ with a triangular signal $h[k] = \Lambda_N[k]$ of length $N$.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import scipy.signal as sig

L = 32  # length of signal x[k]
N = 16  # length of signal h[k]
M = 16  # periodicity of periodic convolution

def periodic_summation(x, N):
    "Zero-padding to length N or periodic summation with period N."
    M = len(x)
    rows = int(np.ceil(M/N))

    if (M < int(N*rows)):
        x = np.pad(x, (0, int(N*rows-M)), 'constant')

    x = np.reshape(x, (rows, N))

    return np.sum(x, axis=0)

def periodic_convolve(x, y, P):
    "Periodic convolution of two signals x and y with period P."
    x = periodic_summation(x, P)
    h = periodic_summation(y, P)

    return np.array([np.dot(np.roll(x[::-1], k+1), h) for k in range(P)], float)

# generate signals
x = np.ones(L)
h = sig.triang(N)

# linear convolution
y1 = np.convolve(x, h, 'full')
# periodic convolution
y2 = periodic_convolve(x, h, M)
# linear convolution via periodic convolution
xp = np.append(x, np.zeros(N-1))
hp = np.append(h, np.zeros(L-1))
y3 = periodic_convolve(xp, hp, L+N-1)

# plot results
def plot_signal(x):
    plt.figure(figsize = (10, 3))
    plt.stem(x)
    plt.xlabel(r'$k$')
    plt.ylabel(r'$y[k]$')
    plt.axis([0, N+L, 0, 1.1*x.max()])

plot_signal(x)
plt.title('Signal $x[k]$')
plot_signal(y1)
plt.title('Linear convolution')
plot_signal(y2)
plt.title('Periodic convolution with period M = %d' %M)
plot_signal(y3)
plt.title('Linear convolution by periodic convolution');
```

**Exercise**

* Change the lengths `L`, `N` and `M` and check how the results for the different convolutions change

### The Fast Convolution

Using the above derived equality of the linear and periodic convolution one can express the linear convolution $y[k] = x_L[k] * h_N[k]$ by the DFT as

\begin{equation}
y[k] = \text{IDFT}_M \{ \; \text{DFT}_M\{ x_M[k] \} \cdot \text{DFT}_M\{ h_M[k] \} \; \}
\end{equation}

This operation requires three DFTs of length $M$ and $M$ complex multiplications. On first sight this does not seem to be an improvement, since one DFT/IDFT requires $M^2$ complex multiplications and $M \cdot (M-1)$ complex additions. The overall numerical complexity is hence in the order of $\mathcal{O}(M^2)$. The DFT can be realized efficiently by the [fast Fourier transformation](https://en.wikipedia.org/wiki/Fast_Fourier_transform) (FFT), which lowers the computational complexity to $\mathcal{O}(M \log_2 M)$. The resulting algorithm is known as *fast convolution* due to its computational efficiency.

The fast convolution algorithm is composed of the following steps

1. Zero-padding of the two input signals $x_L[k]$ and $h_N[k]$ to at least a total length of $M \geq N+L-1$

2. Computation of the DFTs $X[\mu]$ and $H[\mu]$ using a FFT of length $M$

3. Multiplication of the spectra $Y[\mu] = X[\mu] \cdot H[\mu]$

4. Inverse DFT of $Y[\mu]$ using an inverse FFT of length $M$

The overall complexity depends on the particular implementation of the FFT. Many FFTs are most efficient for lengths which are a power of two. It therefore can make sense, in terms of computational complexity, to choose $M$ as a power of two instead of the shortest possible length $N+L-1$. For real valued signals $x[k] \in \mathbb{R}$ and $h[k] \in \mathbb{R}$ the computational complexity can be reduced significantly by using a real valued FFT.

#### Example - Fast convolution

The implementation of the fast convolution algorithm is straightforward. Most implementations of the FFT include zero-padding to a given length $M$, e.g in `numpy` by `numpy.fft.fft(x, M)`. In the following example an implementation of the fast convolution is shown. For illustration the convolution of a rectangular signal $x[k] = \text{rect}_L[k]$ of length $L$ with a triangular signal $h[k] = \Lambda_N[k]$ of length $N$ is considered.

```python
L = 16  # length of signal x[k]
N = 16  # length of signal h[k]
M = N+L-1

# generate signals
x = np.ones(L)
h = sig.triang(N)

# linear convolution
y1 = np.convolve(x, h, 'full')
# fast convolution
y2 = np.fft.ifft(np.fft.fft(x, M) * np.fft.fft(h, M))

plt.figure(figsize=(10, 6))
plt.subplot(211)
plt.stem(y1)
plt.xlabel(r'$k$')
plt.ylabel(r'$y[k] = x_L[k] * h_N[k]$')
plt.title('Result of linear convolution')

plt.subplot(212)
plt.stem(y1)
plt.xlabel(r'$k$')
plt.ylabel(r'$y[k] = x_L[k] * h_N[k]$')
plt.title('Result of fast convolution')
plt.tight_layout()
```

#### Example - Numerical complexity

It was already argued that the numerical complexity of the fast convolution is considerably lower due to the usage of the FFT. The gain with respect to the convolution is evaluated in the following. In order to measure the execution times for both algorithms the `timeit` module is used. The algorithms are evaluated for the convolution of two random signals $x_L[k]$ and $h_N[k]$ of length $L=N=2^n$ for $n=0, 1, \dots, 16$.

```python
import timeit

n = np.arange(17)  # lengths = 2**n to evaluate
reps = 50  # number of repetitions for timeit

gain = np.zeros(len(n))
for N in n:
    length = 2**N
    # setup environment for timeit
    tsetup = 'import numpy as np; from numpy.fft import rfft, irfft; \
            x=np.random.randn(%d); h=np.random.randn(%d)' % (length, length)
    # direct convolution
    tc = timeit.timeit('np.convolve(x, x, mode="full")', setup=tsetup, number=reps)
    # fast convolution
    tf = timeit.timeit('irfft(rfft(x, %d) * rfft(h, %d))' % (2*length, 2*length), setup=tsetup, number=reps)
    # speedup by using the fast convolution
    gain[N] = tc/tf

# show the results
plt.figure(figsize = (15, 10))
plt.barh(n, gain, log=True)
plt.plot([1, 1], [-1, n[-1]+1], 'r-')
plt.yticks(n, 2**n)
plt.xlabel('Gain of fast convolution')
plt.ylabel('Length of signals')
plt.title('Comparison between direct/fast convolution')
plt.grid()
```

**Exercise**

* When is the fast convolution more efficient/faster than a direct convolution?
* Why is it slower below a given signal length?
* Is the trend of the gain as expected by the numerical complexity of the FFT?

Solution: The gain in execution time of a fast convolution over a direct implementation of the the convolution for different signal lengths depends heavily on the particular implementation and hardware used. The fast convolution in this example is faster for two signals having a length equal or larger than 1024 samples. Discarding the outliers and short lengths, the overall trend in the gain is approximately logarithmic as predicted above.

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples, 2016-2018*.


# Energy dispatch problem

To meet the energy demand, power plants run day and night across the country to produce electricy from a variety of sources such as fossil fuels and renewable energy. On the short-time scale, the best operating levels for electric power plants are derived every 15 minutes by solving the so-called *Optimal Power Flow (OPF)* model. The OPF model is an optimization problem with the objective of minimizing the total energy dispatching cost, while ensuring that the generation meets the total energy demand. Furthermore, the model takes into account many constraints, among which operational and physical constraints.

```python
!curl -s https://raw.githubusercontent.com/jckantor/MO-book/main/tools/_mobook.py -o mobook.py
import mobook
mobook.setup_pyomo()
mobook.setup_glpk()
mobook.svg()
```

## Background: Power networks and power flow physics

We model the nation-wide transmission power network as a directed graph $G=(V, E)$, where $V$ represents the set of nodes (e.g., cities, industrial districts, power plants) and $E$ denotes the set of directed edges (e.g., physical transmission lines).

Each node $i \in V$ has a power injection $p_i$ and demand $d_i$. The set of nodes are separated into *generator* and *load* nodes. The set of generators $\mathcal{G} \subseteq V$ corresponds to the nodes $i \in V$ for which $p_i \geq 0$ and $d_i = 0$. Each generator $i \in \mathcal{G}$ has a minimum $p_i^{\min}$ and maximum $p_i^{\max}$ production capacity. The set of load nodes $\mathcal{D} \subseteq V$ corresponds to the nodes for which $p_i = 0$ and $d_i \geq 0$. The load nodes thus correspond to the places where electricity is being consumed, e.g., cities and industrial districts. We say that supply and demand is *matched* if $\sum_{i \in V} p_i - d_i = 0$. Since we cannot store electricity in large volumes, supply must meet demand at all times, hence adjusting to it every 15 minutes by solving the OPF.

Each edge $(i, j) \in E$ carries a *power flow* $f_{ij} \in R$ and has a capacity $f_{ij}^\max \geq 0$, i.e., the maximum power flow that it may carry. Note that our choice to model a *directed* graph is to make the modeling of the network easier. In particular, a directed edge $(i,j)$ may carry a 'negative' flow $f_{ij} < 0$, which implies that there is flow going from $j$ to $i$ where $f_{ji} = -f_{ij}$. The capacity does not depend on the direction of the flow, implying that the flow capacity constraints are given by $|f_{ij}| = |f_{ji}| \leq f_{ij}^\max$.

One crucial difference of power flow problems compared to typical network flow problems is that the power flows cannot be controlled directly. Instead, as you might recall from high-school physics, the power flows are determined by the laws of electricity, which we will now present as the *power flow equations*. Ignore for a moment the flow capacity constraints. Let $\theta_{i} \in \mathbb{R}$ denote the *phase angle* of node $i$. For each edge $(i,j)$, let $b_{ij} > 0$ denote the *line susceptance*. Assuming that supply and demand is matched, i.e., $\sum_{i=1}^{n} p_i - d_i = 0$, the power flows $\mathbf{f} \in \mathbb{R}^{m}$ and phase angles $\mathbf{\theta} \in \mathbb{R}^{n}$ are obtained by solving the following linear system of equations:

$$\begin{align}
p_i - d_i &= \sum_{j: (i, j) \in E} f_{ij} - \sum_{j: (j, i) \in E} f_{ji}, & \forall \, i \in V,\\
  f_{ij} &=  b_{ij}(\theta_i - \theta_j), & \forall \, (i, j) \in E.
\end{align}$$

The first set of constraints ensures flow conservation and the second set of constrations captures the flow dependency on susceptances and angle differences. The DC power flow equations admit a unique power flow solution $\mathbf{f}$ given matched power injections $\mathbf{p}$ and demand $\mathbf{d}$.

For a given matched supply and demand vector $\mathbf{p}$ and $\mathbf{d}$, we can compute the power flows on the network by solving the linear equations as described above. There are exactly $|V|$ and $|E|$ equations for the $\theta_i$ variables and $f_{ij}$ variables, meaning that this system of equations admit a solution.

## Optimal Power Flow
We assumed above that the power injections $\mathbf{p}$ were given. However, in practice, the power injections need to be determined for each generator in the power network, where some types of generators may be cheaper than others. Moreover, we need to take into account operational constraints, such as the maximum flow and generator limits.

On the short-time scale, the power injections are calculated for each generator by solving the so-called *Optimal Power Flow (OPF)* problem. The goal of the OPF problem is to determine a solution $(\mathbf{p}, \mathbf{f}, \mathbf{\theta})$ with minimal costs such that:
- Supply meets demand
- Line capacity constraints are met
- Generator capacity constraints are met

Let $c_i > 0$ be the cost associated with the production of a unit energy by generator $i$. Then, the OPF problem can be formulated as

$$\begin{align}
\begin{array}{llll}
\max        & \sum_{i \in V} c_i p_i \\
\mbox{s.t.} & \sum_{j: (i, j) \in E} f_{ij} - \sum_{j: (j, i) \in E} f_{ji} = p_i - d_i & \forall \, i \in V,\\
& f_{ij} =  b_{ij}(\theta_i - \theta_j), & \forall \, (i, j) \in E, \\
            & |f_{ij}| \leq  f_{ij}^{\max}    & \forall (i, j) \in E,\\
            & p_{i}^{\min } \leq p_{i} \leq p_{i}^{\max }         & \forall i \in V, \\
            & p_i \in \mathbb{R}_{\geq 0}        & \forall i \in V,                \\
            & \theta_i \in \mathbb{R} & \forall i \in V, \\
            & f_{ij} \in \mathbb{R}                 & \forall (i, j) \in E                \\
\end{array}
\end{align}$$

For simplicity, you may assume that all load nodes do not produce energy, i.e., $p_i = p_i^{\min} = p_i^\max = 0$ for all $i \in \mathcal{D}$. You may therefore model $p_i$ as decision variables for all nodes (both generator and load nodes). Similarly, you may assume that all generator nodes have no demand, i.e., $d_i = 0$ for all $i \in \mathcal{G}$.

To summarize, the decision variables in the OPF problem are:
- $p_i$ power injections
- $\theta_i$ phase angles
- $f_{ij}$ power flows

All the other quantities are instance dependent parameters.

## Data

You will solve the OPF problem on the IEEE-118 power network, which is a standard test network consisting of 118 nodes and 179 edges. In the following, we will load the data into the notebook and provide a description of the data. Make sure to run the setup code below. We will first describe the data which are initially provided as `pandas.DataFrames`. We will then provide a new network data structure to easily access the data for the implementation of the optimization model.

### Setup code

```python
# Load packages
import pyomo.environ as pyo
from IPython.display import Markdown, HTML
import numpy as np
import numpy.random as rnd
import pandas as pd
import matplotlib.pyplot as plt
from ast import literal_eval as make_tuple
import networkx as nx
import time

# Load GLPK solver
solver = pyo.SolverFactory('glpk')
```

### Data import

```python
# Download the data
nodes_df = pd.read_csv('https://gist.githubusercontent.com/leonlan/0fce00d4ea330fd11d49f35c71ba5d8c/raw/146d180aad2ef67d451cd8745ccca98a6d2db68a/final_nodes.csv', index_col=0)
edges_df = pd.read_csv('https://gist.githubusercontent.com/leonlan/0fce00d4ea330fd11d49f35c71ba5d8c/raw/5ce01cafcdf659e7f073fda31df731a1550683ef/final_edges.csv', index_col=0)

# Read all instances
nodes = nodes_df.groupby("instance").apply(lambda data: data.set_index("node_id").T.to_dict())
edges = edges_df.set_index(edges_df["edge_id"].apply(make_tuple)).T.to_dict()
I = [{"nodes": n, "edges": edges} for n in nodes]

# Initialize a network for demonstration purposes
network = I[0]
example_nodes = nodes_df[nodes_df.instance == 0]
example_edges = nodes_df[nodes_df.instance == 0]
```

### Network data

```python
def visualize_network(network, edge_flows=None, ax=None):
    """Visualize a network instance, highlighting the generators in orange and the load buses in green."""
    plt.figure(figsize=[12, 10])
    g = nx.DiGraph(network["edges"].keys())
    pos = nx.layout.kamada_kawai_layout(g, weight=None)

    color_mapping = {'solar': '#ffcb36',
     'wind': 'white',
     'hydro': '#a5efff',
     'coal': '#686868',
     'gas': '#00ab4e',
     np.nan: '#b6b6b6'}

    vertex2color = {i: color_mapping[data['energy_type']] for i, data in network['nodes'].items()}
    v2c_list = [vertex2color[i] for i in g.nodes] # Order based on networkx

    nodes = nx.draw_networkx_nodes(g, pos, node_size=250, node_color=v2c_list, linewidths=2,)
    edges = nx.draw_networkx_edges(g, pos, width=2, edge_color='#595959',)

    # Gives node colors
    ax = plt.gca()
    ax.collections[0].set_edgecolor("#595959")
    ax.set_axis_off()
```

```python
visualize_network(network)
```

The IEEE-118 network has 18 generators, of which:
- 2 hydro plants (cyan)
- 4 coal plants (dark gray)
- 4 solarfarms (yellow)
- 2 windmills (white)
- 6 gas plants (green)

Moreover, the network has 100 load nodes (gray). Each node has the following parameters:
- `node_id`
- `d`: demand
- `p_min`: minimum production capacity
- `p_max`: maximum production capacity
- `c_var`: variable production costs
- `is_generator`: boolean value to indicate whether the node is a generator or not
- `energy_type`: the type of energy used for electricity production

All generator nodes can filtered by the `is_generator` parameter. All generators have a zero demand $d_i=0$. For the renewable energy sources (i.e., hydro, solar and wind) there are no variable costs `c_var`. For solar and wind, the production is fixed, i.e., `p_min = p_max`, meaning that all available solar and wind energy must be produced.

```python
 example_nodes[example_nodes.is_generator]
```

      node_id
      d
      p_min
      p_max
      c_var
      is_generator
      energy_type
      instance

      9
      9
      0.0
      0.000000
      400.000000
      0.000000
      True
      hydro
      0

      11
      11
      0.0
      0.000000
      200.000000
      0.000000
      True
      hydro
      0

      24
      24
      0.0
      0.000000
      397.800000
      28.948321
      True
      coal
      0

      25
      25
      0.0
      0.000000
      873.000000
      22.220980
      True
      coal
      0

      30
      30
      0.0
      0.000000
      612.000000
      25.993982
      True
      coal
      0

      45
      45
      0.0
      0.000000
      720.000000
      24.202306
      True
      coal
      0

      48
      48
      0.0
      0.000000
      0.000000
      0.000000
      True
      solar
      0

      53
      53
      0.0
      0.000000
      0.000000
      0.000000
      True
      solar
      0

      58
      58
      0.0
      0.000000
      0.000000
      0.000000
      True
      solar
      0

      60
      60
      0.0
      0.000000
      0.000000
      0.000000
      True
      solar
      0

      64
      64
      0.0
      38.484861
      38.484861
      0.000000
      True
      wind
      0

      65
      65
      0.0
      180.602933
      180.602933
      0.000000
      True
      wind
      0

      79
      79
      0.0
      0.000000
      916.200000
      50.330000
      True
      gas
      0

      86
      86
      0.0
      0.000000
      360.000000
      50.330000
      True
      gas
      0

      88
      88
      0.0
      0.000000
      1146.600000
      50.330000
      True
      gas
      0

      99
      99
      0.0
      0.000000
      1175.400000
      50.330000
      True
      gas
      0

      102
      102
      0.0
      0.000000
      194.400000
      50.330000
      True
      gas
      0

      110
      110
      0.0
      0.000000
      142.200000
      50.330000
      True
      gas
      0

For the load nodes, the only important parameter is the demand $d_i \geq 0$. All other parameters are either zero, `False`, or `NaN`.

```python
 example_nodes[~example_nodes.is_generator]
```

      node_id
      d
      p_min
      p_max
      c_var
      is_generator
      energy_type
      instance

      0
      0
      28.186145
      0.0
      0.0
      0.0
      False
      NaN
      0

      1
      1
      10.921628
      0.0
      0.0
      0.0
      False
      NaN
      0

      2
      2
      22.884795
      0.0
      0.0
      0.0
      False
      NaN
      0

      3
      3
      21.961682
      0.0
      0.0
      0.0
      False
      NaN
      0

      4
      4
      0.000000
      0.0
      0.0
      0.0
      False
      NaN
      0

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      113
      113
      4.396000
      0.0
      0.0
      0.0
      False
      NaN
      0

      114
      114
      11.864042
      0.0
      0.0
      0.0
      False
      NaN
      0

      115
      115
      108.311328
      0.0
      0.0
      0.0
      False
      NaN
      0

      116
      116
      10.703998
      0.0
      0.0
      0.0
      False
      NaN
      0

      117
      117
      18.242759
      0.0
      0.0
      0.0
      False
      NaN
      0

100 rows × 8 columns

### Edges

The IEEE-118 network has 179 edges. Each edge has the following parameters:
- `edge_id`: the edge indices
- `b`: the line susceptance, and
- `f_max`: the maximum edge capacity.

```python
edges_df
```

      edge_id
      b
      f_max

      0
      (0, 1)
      10.0100
      271

      1
      (0, 2)
      23.5849
      271

      2
      (3, 4)
      125.3133
      316

      3
      (2, 4)
      9.2593
      315

      4
      (4, 5)
      18.5185
      316

      ...
      ...
      ...
      ...

      174
      (64, 65)
      28.9059
      1427

      175
      (67, 68)
      28.9059
      1427

      176
      (80, 79)
      28.9059
      1427

      177
      (86, 85)
      4.8216
      253

      178
      (67, 115)
      246.9136
      12992

179 rows × 3 columns

### Network data structure
When implementing optimization problems, it is often not the easiest way to access data directly through `pandas.DataFrames`. We have therefore provided a new type of data structure that will help to more easily implement an optimization problem. This data structure is also found in Exercise 5 & 6 from Computer Lab 1, which is as follows:
```
network = {"nodes": {i: {"attr1":  1, ...},
                       ...
                      },
             "edges": {(i, j): {"attr1":  4, "attr2": 1, ...},
                      ...
                      }
            }
```
In other words, each network consists of a dictionary, that contains a dictionary for nodes and edges, which again holds a dictionary for each node and edge. It might a be little bit confusing at first, but this data structure allows to easily access the parameters that you need for a specific constraint. For instance,

```python
network['nodes'][1]['d']
```

    10.921627803299574

gives the demand of node `1`. To filter on generator nodes, you can use the following:

```python
[i for i, data in network['nodes'].items() if data['is_generator']]
```

    [9, 11, 24, 25, 30, 45, 48, 53, 58, 60, 64, 65, 79, 86, 88, 99, 102, 110]

### Instances
Since the OPF problem is solved every 15 minutes in practice, you will be given `24 * 4 = 96` network instances that need to be solved, hence solving an entire day's worth of OPF problems. The first instance thus corresponds to the power generation within time window `[00:00, 00:15]`, the second instance corresponds to `[00:15, 00:30]`, and so on. The data takes into account a realistic demand and (renewable energy) generation pattern. We assume that the decisions in each time window are independent of the previous and subsequent time windows, so every 15 minutes a new OPF instance is solved independently.

The network instances are stored in the variable `I` as a list and can be accessed using indexing, i.e., `I[0]` gives the first network instance and `I[95]` gives the 96th network instance.

---

# Solving OPF
Observe that the stated OPF problem contains absolute decision values. We first rewrite the OPF problem into a linear optimization problem.

$$\begin{align}
\begin{array}{llll}
\min        & \sum_{i \in V} c_i p_i \\
\mbox{s.t.} & \sum_{j: (i, j) \in E} f^+_{ij} - f^-_{ij} - \sum_{j: (j, i) \in E} f^+_{ji} - f_{ji}^- = p_i - d_i & \forall \, i \in V,\\
& f^+_{ij} - f^-_{ij} =  b_{ij}(\theta_i - \theta_j), & \forall \, (i, j) \in E, \\
            & f^+_{ij} + f^-_{ij} \leq  f_{ij}^{\max}    & \forall (i, j) \in E,\\
            & p_{i}^{\min } \leq p_{i} \leq p_{i}^{\max }         & \forall i \in V, \\
            & p_i \in \mathbb{R}_{\geq 0}        & \forall i \in V,                \\
            & \theta_i \in \mathbb{R} & \forall i \in V, \\
            & f_{ij}^+, f_{ij}^- \in \mathbb{R}                 & \forall (i, j) \in E                \\
\end{array}
\end{align}$$

We then implement the model using `pyomo` and solve it for all instances `I[0]` to `I[95]`, reporting the average objective value across all the 96 instances.

```python
def OPF1(network):
    """
    Input:
    - network: a dictionary containing:
      - a dictionary of nodes with a dictionary of attributes
      - a dictionary of edges with a dictionary of attributes

    Output:
    - power_generation: a dictionary containing the power generation for each node
    - power_flows: a dictionary containing the power flow for each edge
    """

    # Define a model
    model = pyo.ConcreteModel("Q1")

    # Declare decision variables
    model.p = pyo.Var(network["nodes"], domain=pyo.NonNegativeReals)
    model.theta = pyo.Var(network["nodes"], domain=pyo.Reals)
    model.fabs = pyo.Var(network["edges"], domain=pyo.NonNegativeReals)
    model.fp = pyo.Var(network["edges"], domain=pyo.NonNegativeReals)
    model.fm = pyo.Var(network["edges"], domain=pyo.NonNegativeReals)

    # Declare objective value
    @model.Objective(sense=pyo.maximize)
    def objective(model):
        return sum(data["c_var"] * model.p[i] for i, data in network["nodes"].items() if data["is_generator"])

    @model.Expression(network["nodes"])
    def outgoing_flow(m, i):
        return sum(m.fp[i, j] - m.fm[i, j] for j in network["nodes"] if (i, j) in network["edges"])

    @model.Expression(network["nodes"])
    def incoming_flow(m, i):
        return sum(m.fp[j, i] - m.fm[j, i] for j in network["nodes"] if (j, i) in network["edges"])

    @model.Constraint(network["nodes"])
    def flow_conservation(m, i):
        return m.outgoing_flow[i] - m.incoming_flow[i] == m.p[i] - network["nodes"][i]["d"]

    @model.Constraint(network["edges"])
    def susceptance(m, i, j):
        return m.fp[i, j] - m.fm[i, j] == network["edges"][(i, j)]["b"] * (m.theta[i] - m.theta[j])

    @model.Constraint(network["edges"])
    def abs_flow(m, *e):
        return m.fabs[e] == m.fp[e] + m.fm[e]

    @model.Constraint(network["edges"])
    def flows_upper_bound(m, *e):
        return m.fabs[e] <= network["edges"][e]["f_max"]

    @model.Constraint(network["nodes"])
    def generation_upper_bound(m, i):
        return m.p[i] <= network["nodes"][i]["p_max"]

    @model.Constraint(network["nodes"])
    def generation_lower_bound(m, i):
        return network["nodes"][i]["p_min"] <= m.p[i]

    # Solve
    result = solver.solve(model)

    return model.objective()
```

```python
print(f"The average objective value over all instances is: {np.mean([OPF1(instance) for instance in I])}")
```

    The average objective value over all instances is: 102898.75786181616

# Strict fossil fuel policy pt.1
Gas and coal plants emit CO2, while renewable energy sources are carbon neutral. For this reason, the Dutch government has decided to constrain the number of active fossil-fuel-based power plants for the generation of electricity. More specifically, a maximum of 2 gas plants and 1 coal plant may be *active* during a single OPF instance. Any plant that is set *inactive* for a specific instance cannot produce any electricity.

We first write down the new model. To this end, we introduce new decision variables $x_i, i\in V$ to the model, which indicate whether a generator $i$ is active or inactive.

$$\begin{align}
\begin{array}{llll}
\min        & \sum_{i \in V} c_i p_i \\
\mbox{s.t.} & \sum_{j: (i, j) \in E} f^+_{ij} - f^-_{ij} - \sum_{j: (j, i) \in E} f^+_{ji} - f_{ji}^- = p_i - d_i & \forall \, i \in V,\\
& f^+_{ij} - f^-_{ij} =  b_{ij}(\theta_i - \theta_j), & \forall \, (i, j) \in E, \\
& f^{abs}_{ij} = f^+_{ij} + f^-_{ij}, & \forall \, (i, j) \in E, \\
            & f_{ij}^{abs} \leq  f_{ij}^{\max}    & \forall (i, j) \in E,\\
            & p_{i}^{\min } x_i \leq p_{i} \leq p_{i}^{\max } x_i        & \forall i \in V, \\
            & \sum_{i \in \mathcal{G}_{\text{gas}}} x_i \leq 2 & \\
            & \sum_{i \in \mathcal{G}_{\text{coal}}} x_i \leq 1 &  \\
            & p_i \in \mathbb{R}_{\geq 0}        & \forall i \in V,                \\
            & \theta_i \in \mathbb{R} & \forall i \in V, \\
            & f_{ij} \in \mathbb{R}                 & \forall (i, j) \in E                \\
            & x_i  \in \{0, 1\} & \forall i \in V\\
\end{array}
\end{align}$$

We then implement the new model using `pyomo` and solve it for all instances `I[0]` to `I[95]`, reporting the average objective value across the instances.

```python
def OPF2(network):
    """
    Input:
    - network: a dictionary containing:
      - a dictionary of nodes with a dictionary of attributes
      - a dictionary of edges with a dictionary of attributes

    Output:
    - power_generation: a dictionary containing the power generation for each node
    - power_flows: a dictionary containing the power flow for each edge
    """

    # Define a model
    model = pyo.ConcreteModel("Q2")

    # Declare decision variables
    model.p = pyo.Var(network["nodes"], domain=pyo.NonNegativeReals)
    model.theta = pyo.Var(network["nodes"], domain=pyo.Reals)
    model.fabs = pyo.Var(network["edges"], domain=pyo.NonNegativeReals)
    model.fp = pyo.Var(network["edges"], domain=pyo.NonNegativeReals)
    model.fm = pyo.Var(network["edges"], domain=pyo.NonNegativeReals)
    model.x = pyo.Var(network["nodes"], domain=pyo.Binary)

    # Declare objective value
    @model.Objective(sense=pyo.minimize)
    def objective(model):
        return sum(data["c_var"] * model.p[i] for i, data in network["nodes"].items() if data["is_generator"])

    # Declare constraints
    @model.Expression(network["nodes"])
    def outgoing_flow(m, i):
        return sum(m.fp[i, j] - m.fm[i, j] for j in network["nodes"] if (i, j) in network["edges"])

    @model.Expression(network["nodes"])
    def incoming_flow(m, i):
        return sum(m.fp[j, i] - m.fm[j, i] for j in network["nodes"] if (j, i) in network["edges"])

    @model.Constraint(network["nodes"])
    def flow_conservation(m, i):
        return m.outgoing_flow[i] - m.incoming_flow[i] == m.p[i] - network["nodes"][i]["d"]

    @model.Constraint(network["edges"])
    def susceptance(m, i, j):
        return m.fp[i, j] - m.fm[i, j] == network["edges"][(i, j)]["b"] * (m.theta[i] - m.theta[j])

    @model.Constraint(network["edges"])
    def abs_flows(m, *e):
        return m.fabs[e] == m.fp[e] + m.fm[e]

    @model.Constraint(network["edges"])
    def flows_upper_bound(m, *e):
        return m.fabs[e] <= network["edges"][e]["f_max"]

    @model.Constraint(network["nodes"])
    def generation_upperound(m, i):
        return m.p[i] <= network["nodes"][i]["p_max"] * m.x[i]

    @model.Constraint(network["nodes"])
    def generation_lower_bound(m, i):
        return network["nodes"][i]["p_min"] * m.x[i] <= m.p[i]

    @model.Constraint()
    def max_gas_plants(model):
        return sum(model.x[i] for i, data in network["nodes"].items() if data['energy_type'] == 'gas') <= 2

    @model.Constraint()
    def max_coal_plants(model):
        return sum(model.x[i] for i, data in network["nodes"].items() if data['energy_type'] == 'coal') <= 1

    # Solve
    result = solver.solve(model)

    return model.objective()
```

```python
print(f"The average objective value over all instances is: {np.mean([OPF2(instance) for instance in I])}")
```

    The average objective value over all instances is: 53120.80867986209

# Strict fossil fuel policy pt.2
The restriction on the number of gas and coal plants may pose a threat to the availability of electricity production when renewable energy sources fail to deliver. For this reason, the grid operators have decided to slightly change the constraint that was introduced for OPF2. If the total production of energy from renewable energy sources (i.e., solar, wind and hydro) is above 1000, then the number of gas and coal plants is restricted to 2 and 1, respectively. Otherwise, the restriction on the number of gas and coal plants is lifted. These constraints can be modeled as *either-or* constraints.

We first write down the new model, using big-$M$ constraints to model the either-or constraint.

$$\begin{align}
\begin{array}{llll}
\min        & \sum_{i \in V} c_i p_i \\
\mbox{s.t.} & \sum_{j: (i, j) \in E} f^+_{ij} - f^-_{ij} - \sum_{j: (j, i) \in E} f^+_{ji} - f_{ji}^- = p_i - d_i & \forall \, i \in V,\\
& f^+_{ij} - f^-_{ij} =  b_{ij}(\theta_i - \theta_j), & \forall \, (i, j) \in E, \\
& f^{abs}_{ij} = f^+_{ij} + f^-_{ij}, & \forall \, (i, j) \in E, \\
            & f_{ij}^{abs} \leq  f_{ij}^{\max}    & \forall (i, j) \in E,\\
            & p_{i}^{\min } x_i \leq p_{i} \leq p_{i}^{\max } x_i        & \forall i \in V, \\
            & \sum_{i \in \mathcal{G}_{\text{gas}}} x_i \leq 2 + (1-y)M_0 & \\
            & \sum_{i \in \mathcal{G}_{\text{coal}}} x_i \leq 1 + (1-y)M_1 &  \\
            & \sum_{i \in \mathcal{G}_\text{solar}} p_i + \sum_{i \in \mathcal{G}_\text{wind}} p_i + \sum_{i \in \mathcal{G}_\text{hydro}} p_i \leq 1000 + yM_2&\\
            & p_i \in \mathbb{R}_{\geq 0}        & \forall i \in V,                \\
            & \theta_i \in \mathbb{R} & \forall i \in V, \\
            & f_{ij} \in \mathbb{R}                 & \forall (i, j) \in E                \\
            & x_i  \in \{0, 1\} & \forall i \in V\\
            & y \in \{0, 1\} &\\
\end{array}
\end{align}$$

We now implement the new model using `pyomo` and solve it for all instances `I[0]` to `I[95]`, reporting the average objective value across the instances.

```python
def OPF3(network):
    """
    Input:
    - network: a dictionary containing:
      - a dictionary of nodes with a dictionary of attributes
      - a dictionary of edges with a dictionary of attributes

    Output:
    - power_generation: a dictionary containing the power generation for each node
    - power_flows: a dictionary containing the power flow for each edge
    """

    # Define a model
    model = pyo.ConcreteModel("Q3")

    # Declare decision variables
    model.p = pyo.Var(network["nodes"], domain=pyo.NonNegativeReals)
    model.theta = pyo.Var(network["nodes"], domain=pyo.Reals)
    model.fabs = pyo.Var(network["edges"], domain=pyo.NonNegativeReals)
    model.fp = pyo.Var(network["edges"], domain=pyo.NonNegativeReals)
    model.fm = pyo.Var(network["edges"], domain=pyo.NonNegativeReals)
    model.x = pyo.Var(network["nodes"], domain=pyo.Binary)
    model.y = pyo.Var(domain=pyo.Binary)

    # Big-Ms
    max_total_renewable_production = sum(data['p_max'] for i, data in network["nodes"].items() if data['energy_type'] in ['solar', 'wind', 'hydro'])
    M = [4, 3, max_total_renewable_production - 1000]

    # Declare objective value
    @model.Objective(sense=pyo.minimize)
    def objective(m):
        return sum(data["c_var"] * m.p[i] for i, data in network["nodes"].items() if data["is_generator"])

    # Declare constraints
    @model.Expression(network["nodes"])
    def outgoing_flow(m, i):
        return sum(m.fp[i, j] - m.fm[i, j] for j in network["nodes"] if (i, j) in network["edges"])

    @model.Expression(network["nodes"])
    def incoming_flow(m, i):
        return sum(m.fp[j, i] - m.fm[j, i] for j in network["nodes"] if (j, i) in network["edges"])

    @model.Constraint(network["nodes"])
    def flow_conservation(m, i):
        return m.outgoing_flow[i] - m.incoming_flow[i] == m.p[i] - network["nodes"][i]["d"]

    @model.Constraint(network["edges"])
    def susceptance(m, i, j):
        return m.fp[i, j] - m.fm[i, j] == network["edges"][(i, j)]["b"] * (m.theta[i] - m.theta[j])

    @model.Constraint(network["edges"])
    def abs_flows(m, *e):
        return m.fabs[e] == m.fp[e] + m.fm[e]

    @model.Constraint(network["edges"])
    def flows_upper_bound(m, *e):
        return m.fabs[e] <= network["edges"][e]["f_max"]

    @model.Constraint(network["nodes"])
    def generation_upper_bound(m, i):
        return m.p[i] <= network["nodes"][i]["p_max"] * m.x[i]

    @model.Constraint(network["nodes"])
    def generation_lower_bound(m, i):
        return network["nodes"][i]["p_min"] * m.x[i] <= m.p[i]

    @model.Constraint()
    def max_gas_plants(m):
        return sum(m.x[i] for i, data in network["nodes"].items() if data['energy_type'] == 'gas') <= 2 + (1 - m.y) * M[0]

    @model.Constraint()
    def max_coal_plants(m):
        return sum(m.x[i] for i, data in network["nodes"].items() if data['energy_type'] == 'coal') <= 1 + (1 - m.y) * M[1]

    @model.Constraint()
    def renewable_energy_production(m):
        return sum(m.p[i] for i, data in network["nodes"].items() if data['energy_type'] in ['solar', 'wind', 'hydro']) <= 1000 + m.y * M[2]

    # Solve
    result = solver.solve(model)

    return  model.objective()

```

```python
print(f"The average objective value over all instances is: {np.mean([OPF3(instance) for instance in I])}")
```

    The average objective value over all instances is: 41608.72683549779

# Comparing the three models
For all three implemented models, plot the objective values for all instances. Explain the differences between the objective values in view of the different feasible regions.

```python
objs = [[OPF1(instance) for instance in I],
        [OPF2(instance) for instance in I],
        [OPF3(instance) for instance in I]]
```

```python
plt.plot(objs[0], color='blue', label='OPF1')
plt.plot(objs[1], color='green', label='OPF2')
plt.plot(objs[2], color='orange', label='OPF3')
plt.title('Optimal objective values over all instances and models')
plt.xlabel('Instance number')
plt.ylabel('Optimal objective value')
plt.legend()
plt.show()
```

The goal of the OPF problem is to minimize the total costs of dispatching energy. In what follows, we refer to the earlier described models as OPF1, OPF2 and OPF3, respectively.

OPF1 is the original OPF formulation, whereas OPF2 and OPF3 are restricted versions of OPF1. This means that the feasible region of OPF1 is at least as large as OPF2 and OPF3, where we may assume that $x_i = 1$ for all the generators $i$.

If we let $F_1, F_2, F_3$ denote the feasible region of OPF1, OPF2 and OPF3, then we observe that
which explains that the optimal objective value of OPF1 always remains below the optimal objectives values of OPF2 and OPF3.

The most important observation to make is that based on the problem descriptions, we would expect OPF2 to take on the objective values of OPF1 or OPF3, depending on which of the either-or-constraints are activated.
- OPF3 uses expensive generators, and possibly not all the renewable energy
- OPF2 does only uses 1000 renewable energy power, because then it may keep using all the gas and coal generators.
- OPF1 uses all renewable energy at all times, because it has the flexibility to use all generators in order to mitigate operational restrictions due to line flows.

```python

```