# Linear Algebra

## Dot Products

A dot product is defined as

$ a \cdot b = \sum_{i}^{n} a_{i}b_{i} = a_{1}b_{1} + a_{2}b_{2} + a_{3}b_{3} + \dots + a_{n}b_{n}$

The geometric definition of a dot product is

$ a \cdot b = $\|\|b\|\|\|\|a\|\|

### What does a dot product conceptually mean?

A dot product is a representation of the similarity between two components, because it is calculated based upon shared elements.

The actual value of a dot product reflects the direction of change:

* **Zero**: we don't have any growth in the original direction
* **Positive** number: we have some growth in the original direction
* **Negative** number: we have negative (reverse) growth in the original direction

```python
A = [0,2]
B = [0,1]

def dot_product(x,y):
    return sum(a*b for a,b in zip(x,y))

dot_product(A,B)
# What will the dot product of A and B be?
```

    -0.4480736161291701

```python
A = [1,2]
B = [2,4]
# What will the dot product of A and B be?
```

```python
from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer()
data_corpus = ["John likes to watch movies. Mary likes movies too.",
"John also likes to watch football games. Mary does not like football much."]
X = vectorizer.fit_transform(data_corpus)
print(vectorizer.get_feature_names())
```

    ['also', 'does', 'football', 'games', 'john', 'like', 'likes', 'mary', 'movies', 'much', 'not', 'to', 'too', 'watch']

# Bag of Words Models

You can use **`sklearn.feature_extraction.text.CountVectorizer`** to easily convert your corpus into a bag of words matrix:

```python
from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer()
data_corpus = ["John likes to watch movies. Mary likes movies too.",
"John also likes to watch football games. Mary does not like football much."]
X = vectorizer.fit_transform(data_corpus)
```
Note that the output `X` here is not your traditional Numpy matrix! Calling **`type(X)`** here will yield **`<class 'scipy.sparse.csr.csr_matrix'>`**, which is a **CSR ([compressed sparse row format matrix](https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.sparse.csr_matrix.html))**. To convert it into an actual matrix, call the `toarray()` method:

```python
X.toarray()
```
Your output will be

```
array([[0, 0, 0, 0, 1, 0, 2, 1, 2, 0, 0, 1, 1, 1],
       [1, 1, 2, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 1]], dtype=int64)
```
Notice that using **`X.shape`** $\rightarrow$ `(2,14)`, indicating a total vocabulary size $V$ of 14. To get what word each of the 14 columns corresponds to, use **`vectorizer.get_feature_names()`**:
```
['also', 'does', 'football', 'games', 'john', 'like', 'likes', 'mary', 'movies', 'much', 'not', 'to', 'too', 'watch']
```

Notice, however, that as the vocabulary size $V$ increases, the percent of the matrix taken up by zero values increases:

```python
corpus = [
    "Some analysts think demand could drop this year because a large number of homeowners take on remodeling projectsafter buying a new property. With fewer homes selling, home values easing, and mortgage rates rising, they predict home renovations could fall to their lowest levels in three years.",

          "Most home improvement stocks are expected to report fourth-quarter earnings next month.",

         "The conversation boils down to how much leverage management can get out of its wide-ranging efforts to re-energize operations, branding, digital capabilities, and the menu–and, for investors, how much to pay for that.",

    "RMD’s software acquisitions, efficiency, and mix overcame pricing and its gross margin improved by 90 bps Y/Y while its operating margin (including amortization) improved by 80 bps Y/Y. Since RMD expects the slower international flow generator growth to continue for the next few quarters, we have lowered our organic growth estimates to the mid-single digits."
]

X = vectorizer.fit_transform(corpus).toarray()
```

```python
corpus = [
        "Some analysts think demand could drop this year because a large number of homeowners take on remodeling projectsafter buying a new property. With fewer homes selling, home values easing, and mortgage rates rising, they predict home renovations could fall to their lowest levels in three years.",

          "Most home improvement stocks are expected to report fourth-quarter earnings next month.",

         "The conversation boils down to how much leverage management can get out of its wide-ranging efforts to re-energize operations, branding, digital capabilities, and the menu–and, for investors, how much to pay for that.",

    "RMD’s software acquisitions, efficiency, and mix overcame pricing and its gross margin improved by 90 bps Y/Y while its operating margin (including amortization) improved by 80 bps Y/Y. Since RMD expects the slower international flow generator growth to continue for the next few quarters, we have lowered our organic growth estimates to the mid-single digits. "
]

X = vectorizer.fit_transform(corpus).toarray()
import numpy as np
from sys import getsizeof

zeroes = np.where(X.flatten() == 0)[0].size
percent_sparse = zeroes / X.size
print(f"The bag of words feature space is {round(percent_sparse * 100,2)}% sparse. \n\
That's approximately {round(getsizeof(X) * percent_sparse,2)} bytes of wasted memory. This is why sklearn uses CSR (compressed sparse rows) instead of normal matrices!")
```

    The bag of words feature space is 72.63% sparse.
    That's approximately 2777.34 bytes of wasted memory. This is why sklearn uses CSR (compressed sparse rows) instead of normal matrices!

# Distance Measures

## Euclidean Distance

Euclidean distances can range from 0 (completely identically) to $\infty$ (extremely dissimilar). **Magnitude** plays an extremely important role:

```python
from math import sqrt

def euclidean_distance_1(x,y):
    distance = sum((a-b)**2 for a, b in zip(x, y))
    return sqrt(distance)
```

There's typically an easier way to write this function that takes advantage of Numpy's vectorization capabilities:

```python
import numpy as np
def euclidean_distance_2(x,y):
    x = np.array(x)
    y = np.array(y)
    return np.linalg.norm(x-y)
```

# Similarity Measures

Similarity measures will always range between -1 and 1. A similarity of -1 means the two objects are complete opposites, while a similarity of 1 indicates the objects are identical.

## Pearson Correlation Coefficient
* We use **ρ** when the correlation is being measured from the population, and **r** when it is being generated from a sample.
* An r value of 1 represents a **perfect linear** relationship, and a value of -1 represents a perfect inverse linear relationship.

The equation for Pearson's correlation coefficient is
$$
ρ_{Χ_Υ} = \frac{cov(X,Y)}{σ_Xσ_Y}
$$

### Intuition Behind Pearson Correlation Coefficient

#### When $ρ_{Χ_Υ} = 1$ or  $ρ_{Χ_Υ} = -1$

This requires **$cov(X,Y) = σ_Xσ_Y$** or **$-1 * cov(X,Y) = σ_Xσ_Y$** (in the case of $ρ = -1$) . This corresponds with all the data points lying perfectly on the same line.

## Cosine Similarity

The cosine similarity of two vectors (each vector will usually represent one document) is a measure that calculates $ cos(\theta)$, where $\theta$ is the angle between the two vectors.

Therefore, if the vectors are **orthogonal** to each other (90 degrees), $cos(90) = 0$. If the vectors are in exactly the same direction, $\theta = 0$ and $cos(0) = 1$.

Cosine similiarity **does not care about the magnitude of the vector, only the direction** in which it points. This can help normalize when comparing across documents that are different in terms of word count.

### Shift Invariance

* The Pearson correlation coefficient between X and Y does not change with you transform $X \rightarrow a + bX$ and $Y \rightarrow c + dY$, assuming $a$, $b$, $c$, and $d$ are constants and $b$ and $d$ are positive.
* Cosine similarity does, however, change when transformed in this way.

Exercise (20 minutes):

>In Python, find the **cosine similarity** and the **Pearson correlation coefficient** of the two following sentences, assuming a **one-hot encoded binary bag of words** model. You may use a library to create the BoW feature space, but do not use libraries other than `numpy` or `scipy` to compute Pearson and cosine similarity:

>`A = "John likes to watch movies. Mary likes movies too"`

>`B = "John also likes to watch football games, but he likes to watch movies on occasion as well"`

# Use the Example Below to Create Your Own Cosine Similarity Function

#### 1. Create a list of all the **vocabulary $V$**

Using **`sklearn`**'s **`CountVectorizer`**:
```python
from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer()
data_corpus = ["John likes to watch movies. Mary likes movies too",
"John also likes to watch football games, but he likes to watch movies on occasion as well"]
X = vectorizer.fit_transform(data_corpus)
V = vectorizer.get_feature_names()
```

##### Native Implementation:
```python
def get_vocabulary(sentences):
    vocabulary = {} # create an empty set - question: Why not a list?
    for sentence in sentences:
         # this is a very crude form of "tokenization", would not actually use in production
        for word in sentence.split(" "):
            if word not in vocabulary:
                vocabulary.add(word)
    return vocabulary
```

#### 2. Create your Bag of Words model
```python
X = X.toarray()
print(X)
```
Your console output:
```python
[[0 0 0 1 2 1 2 1 1 1]
 [1 1 1 1 1 0 0 1 0 1]]
```

#### 3. Define your cosine similarity functions

```python
from scipy.spatial.distance import cosine # we are importing this library to check that our own cosine similarity func works
from numpy import dot # to calculate dot product
from numpy.linalg import norm # to calculate the norm

def cosine_similarity(A, B):
    numerator = dot(A, B)
    denominator = norm(A) * norm(B)
    return numerator / denominator

def cosine_distance(A,B):
    return 1 - cosine_similarity

A = [0,2,3,4,1,2]
B = [1,3,4,0,0,2]

# check that your native implementation and 3rd party library function produce the same values
assert round(cosine_similarity(A,B),4) ==  round(cosine(A,B),4)
```

#### 4. Get the two documents from the BoW feature space and calculate cosine similarity

```python
cosine_similarity(X[0], X[1])
```
>0.5241424183609592

```python
from scipy.spatial.distance import cosine
from numpy import dot
import numpy as np
from numpy.linalg import norm

def cosine_similarity(A, B):
    numerator = dot(A, B)
    denominator = norm(A) * norm(B)
    return numerator / denominator # remember, you take 1 - the distance to get the distance

def cosine_distance(A,B):
    return 1 - cosine_similarity

A = [0,2,3,4,1,2]
B = [1,3,4,0,0,2]

# check that your native implementation and 3rd party library function produce the same values
assert round(cosine_similarity(A,B),4) ==  round(1 - cosine(A,B),4)

# check for shift invariance
cosine(np.array(A), B)
```

    0.31115327980633567

```python
from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer()

# take two very similar sentences, should have high similarity
# edit these sentences to become less similar, and the similarity score should decrease
data_corpus = ["John likes to watch movies. Mary likes movies too.",
"John also likes to watch football games"]

X = vectorizer.fit_transform(data_corpus)
X = X.toarray()
print(vectorizer.get_feature_names())
cosine_similarity(X[0], X[1])
```

    ['also', 'football', 'games', 'john', 'likes', 'mary', 'movies', 'to', 'too', 'watch']

    0.5241424183609592

# Pointwise Mutual Information (We'll Cover Next Week)

Pointwise mutual information measures the ratio between the **joint probability of two events happening** with the probabilities of the two events happening, assuming they are independent. It can be defined with the following equation:

$$
\begin{equation}
MI_{i,j} = log(\frac{P(i,j)}{P(i)P(j)})
\end{equation}
$$

Remember that when two events are independent, $P(i,j) = P(i)P(j)$.

```python
from nltk.collocations import BigramCollocationFinder, BigramAssocMeasures
from nltk.stem import WordNetLemmatizer
from nltk import word_tokenize
lemmatizer = WordNetLemmatizer()
from nltk.corpus import stopwords
stopwords = set(stopwords.words('english') + [".",",",":", "''", "'s", "'", "``"])
```

```python
documents = []
articles = [
    "bbcsport/football/001.txt",
    "bbcsport/football/002.txt",
    "bbcsport/football/003.txt"
]

for article in articles:
    article = open(article)
    for line in article.readlines():
        line = line.replace("\n", "")
        if len(line) > 0:

            line = [lemmatizer.lemmatize(token) for token in word_tokenize(line)]
            for word in line:
                if word in stopwords:
                    line.remove(word)
            documents.append(line)
```

```python
collocation_finder = BigramCollocationFinder.from_documents(documents)
measures = BigramAssocMeasures()

collocation_finder.nbest(measures.raw_freq, 10)
```

    [('Manchester', 'United'),
     ('Van', 'Nistelrooy'),
     ('``', 'I'),
     ('said', '``'),
     ('``', 'He'),
     ('.', "''"),
     ('23', 'minute'),
     ('Alex', 'Ferguson'),
     ('But', 'wa'),
     ('City', 'Sunday')]

```python

```

week2/Linear Algebra, Distance and Similarity (Completed).ipynb


## Decay Law and Other Functions
\begin{equation}
 A      (t)=A_{0}e^{-\lambda t}
\end{equation}

```python
import numpy as np
import matplotlib.pyplot as plt

Ao=50 # Initial Activity of radioactive substance in Curie
#Half Life of Ir192= 74 Years

t = np.linspace(0,74,10)
A = Ao*np.exp(-0.693*t/74)

#xerr = np.random.random_sample(10)
yerr = np.random.random_sample(10)

fig, ax = plt.subplots()

ax.errorbar(t, A,

            yerr=yerr,
            fmt='-o',capsize=5)

ax.set_xlabel('Time (Year)')
ax.set_ylabel('Activity (Curie)')
ax.set_title(' Decay Scheme')

plt.show()

```

```python
import numpy as np
import matplotlib.pyplot as plt
Ao=50 # Initial Activity of radioactive substance in Curie
#Half Life of Ir192= 74 Years
t = np.linspace(0,74,20)
A = Ao*np.exp(-0.693*t/74)
xerr = 0.3 #year
yerr = 5 # Curie

fig, ax = plt.subplots()

ax.errorbar(t, A,
            xerr=xerr,
            yerr=yerr,
            fmt='-o', color='blue',
             ecolor='black', elinewidth=2, capsize=2)

ax.set_xlabel('Time (Year)')
ax.set_ylabel('Activity (Curie)')
ax.set_title(' Decay Scheme')

plt.show()
```

```python
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import scipy.stats as st
df = pd.read_csv('D.csv')
df

```

      Source
      HalfL_days
      Energy_MeV
      Range_Steel (mm)

      0
      Co60
      1934.5
      1.250
      200

      1
      Se75
      120.0
      0.970
      28

      2
      Cs137
      10950.0
      0.660
      100

      3
      Ir192
      74.0
      0.355
      70

      4
      Th170
      47450.0
      0.072
      12

      5
      Yb169
      30.0
      0.053
      12

```python
df.describe().T
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      HalfL_days
      6.0
      10093.083333
      18782.683936
      30.000
      85.50000
      1027.2500
      8696.1250
      47450.00

      Energy_MeV
      6.0
      0.560000
      0.488143
      0.053
      0.14275
      0.5075
      0.8925
      1.25

      Range_Steel (mm)
      6.0
      70.333333
      72.513907
      12.000
      16.00000
      49.0000
      92.5000
      200.00

```python
plt.figure(figsize = [8,6])
plt.plot(df.HalfL_days,df['Energy_MeV'])
plt.xlabel('Half Life (days)')
plt.ylabel('Energy(MeV)')
plt.title('Energy Versus Half Life')
plt.show()
```

```python
plt.figure(figsize = [8,6])
plt.plot(df.Energy_MeV,df['Range_Steel (mm) '])
plt.xlabel('Energy (MeV)')
plt.ylabel('Thickness of Steel (mm)')
plt.title('Energy Versus Thickness')
plt.show()

```

## Plot Four functions plots in a single call to plot.

```python
import numpy as np
import matplotlib.pyplot as plt

# evenly sampled time at 200ms intervals
x = np.arange(1., 5., 0.4)

# red dashes, blue squares, green triangles and reverse dashes with opposite triangle
plt.plot(x, x, 'r--', x, x**2, 'bs', x, x**3, 'g^', x,1/x,'v--')

plt.show()
```

```python
import matplotlib.pyplot as plt
plt.style.use('classic')
%matplotlib inline
import numpy as np

x = np.linspace(0, 10, 1000)

fig, ax = plt.subplots()
ax.plot(x, np.sin(x), '-b', label='Sine')
ax.plot(x, np.cos(x), '--r', label='Cosine')
plt.ylabel('trigronometic function')
plt.xlabel('x(radian)')

ax.axis('equal')
leg = ax.legend();

```


This notebook is part of https://github.com/AudioSceneDescriptionFormat/splines, see also https://splines.readthedocs.io/.

[back to overview](natural.ipynb)

# Uniform Natural Splines

For deriving natural splines,
we first look at the *uniform* case,
which means that the parameter interval in each segment
is chosen to be 1.

The more general case with arbitrary parameter intervals
is derived in a separate
[notebook about non-uniform natural splines](natural-non-uniform.ipynb).

```python
import sympy as sp
sp.init_printing(order='grevlex')
```

We import a helper class for named SymPy expressions
from [utility.py](utility.py):

```python
from utility import NamedExpression
```

```python
t = sp.symbols('t')
```

In the beginning, we are looking at two neighboring segments:
The fourth segment, from $\boldsymbol{x}_3$ to $\boldsymbol{x}_4$,
defined by the polynomial $\boldsymbol{p}_3$,
and the fifth segment, from $\boldsymbol{x}_4$ to $\boldsymbol{x}_5$,
defined by the polynomial $\boldsymbol{p}_5$.
In both cases, we use $0 \le t \le 1$.

```python
a3, a4, b3, b4, c3, c4, d3, d4 = sp.symbols('a:dbm3:5')
```

We are using the [monomial basis](polynomials.ipynb)
to define the two polynomials ...

```python
b_monomial = sp.Matrix([t**3, t**2, t, 1]).T
```

```python
p3 = NamedExpression('pbm3', d3 * t**3 + c3 * t**2 + b3 * t + a3)
p4 = NamedExpression('pbm4', d4 * t**3 + c4 * t**2 + b4 * t + a4)
display(p3, p4)
```

... and we calculate their first derivatives:

```python
pd3 = p3.diff(t)
pd4 = p4.diff(t)
display(pd3, pd4)
```

From this,
we obtain 8 equations
containing the 8 yet unknown coefficients.

```python
equations = [
    p3.evaluated_at(t, 0).with_name('xbm3'),
    p3.evaluated_at(t, 1).with_name('xbm4'),
    p4.evaluated_at(t, 0).with_name('xbm4'),
    p4.evaluated_at(t, 1).with_name('xbm5'),
    pd3.evaluated_at(t, 0).with_name('xbmdot3'),
    pd3.evaluated_at(t, 1).with_name('xbmdot4'),
    pd4.evaluated_at(t, 0).with_name('xbmdot4'),
    pd4.evaluated_at(t, 1).with_name('xbmdot5'),
]
display(*equations)
```

We can solve the system of equations
to get an expression for each coefficient:

```python
coefficients = sp.solve(equations, [a3, a4, b3, b4, c3, c4, d3, d4])
for c, e in coefficients.items():
    display(NamedExpression(c, e))
```

So far, this is the same as we have done in
[the notebook about uniform Hermite splines](hermite-uniform.ipynb).
In fact, the above constants are the same as in $M_H$!

An additional constraint for natural splines
is that the second derivatives are continuous,
so let's calculate those derivatives ...

```python
pdd3 = pd3.diff(t)
pdd4 = pd4.diff(t)
display(pdd3, pdd4)
```

... and set them to be equal at the segment border:

```python
sp.Eq(pdd3.expr.subs(t, 1), pdd4.expr.subs(t, 0))
```

Inserting the equations from above
leads to this equation:

```python
_.subs(coefficients).simplify()
```

We can generalize this expression by renaming index $4$ to $i$:

\begin{equation*}
\dot{\boldsymbol{x}}_{i-1}
+
4 \dot{\boldsymbol{x}}_{i}
+
\dot{\boldsymbol{x}}_{i+1}
=
3 (\boldsymbol{x}_{i+1} - \boldsymbol{x}_{i-1})
\end{equation*}

This can be used for each segment
-- except for the very first and last one --
yielding a matrix with $N$ columns and $N-2$ rows:

\begin{equation*}
\left[
\begin{matrix}
1 & 4 & 1 && \cdots & 0 \\
& 1 & 4 & 1 && \vdots \\
&& \ddots & \ddots && \\
\vdots && 1 & 4 & 1 & \\
0 & \cdots && 1 & 4 & 1
\end{matrix}
\right]
\left[
\begin{matrix}
\dot{\boldsymbol{x}}_0\\
\dot{\boldsymbol{x}}_1\\
\vdots\\
\dot{\boldsymbol{x}}_{N-2}\\
\dot{\boldsymbol{x}}_{N-1}
\end{matrix}
\right]
=
\left[
\begin{matrix}
3 (\boldsymbol{x}_2 - \boldsymbol{x}_0)\\
3 (\boldsymbol{x}_3 - \boldsymbol{x}_1)\\
\vdots\\
3 (\boldsymbol{x}_{N-2} - \boldsymbol{x}_{N-4})\\
3 (\boldsymbol{x}_{N-1} - \boldsymbol{x}_{N-3})
\end{matrix}
\right]
\end{equation*}

## End Conditions

We need a first and last row for this matrix
to be able to fully define a natural spline.
The following subsections show a selection of a few end conditions
which can be used to obtain the missing rows of the matrix.
End conditions (except "closed") can be mixed,
e.g. "clamped" at the beginning and "natural" at the end.
The Python class [splines.Natural](../python-module/splines.rst#splines.Natural)
uses "natural" end conditions by default.

### Natural

Natural end conditions are commonly used for natural splines,
which is probably why they are named that way.

There is a
[separate notebook about "natural" end conditions](end-conditions-natural.ipynb),
from which we can get the uniform case
by setting $\Delta_i = 1$:

\begin{align*}
2 \dot{\boldsymbol{x}}_0
+
\dot{\boldsymbol{x}}_1
&=
3 (\boldsymbol{x}_1 - \boldsymbol{x}_0)
\\
\dot{\boldsymbol{x}}_{N-2}
+
2 \dot{\boldsymbol{x}}_{N-1}
&=
3 (\boldsymbol{x}_{N-1} - \boldsymbol{x}_{N-2})
\end{align*}

Adding this to the matrix from above leads to a full $N \times N$ matrix:

\begin{equation*}
\left[
\begin{matrix}
2 & 1 &&& \cdots & 0\\
1 & 4 & 1 &&& \vdots \\
& 1 & 4 & 1 && \\
&& \ddots & \ddots && \\
&& 1 & 4 & 1 & \\
\vdots &&& 1 & 4 & 1\\
0 & \cdots &&& 1 & 2
\end{matrix}
\right]
\left[
\begin{matrix}
\dot{\boldsymbol{x}}_0\\
\dot{\boldsymbol{x}}_1\\
\vdots\\
\dot{\boldsymbol{x}}_{N-2}\\
\dot{\boldsymbol{x}}_{N-1}
\end{matrix}
\right]
=
\left[
\begin{matrix}
3 (\boldsymbol{x}_1 - \boldsymbol{x}_0)\\
3 (\boldsymbol{x}_2 - \boldsymbol{x}_0)\\
3 (\boldsymbol{x}_3 - \boldsymbol{x}_1)\\
\vdots\\
3 (\boldsymbol{x}_{N-2} - \boldsymbol{x}_{N-4})\\
3 (\boldsymbol{x}_{N-1} - \boldsymbol{x}_{N-3})\\
3 (\boldsymbol{x}_{N-1} - \boldsymbol{x}_{N-2})
\end{matrix}
\right]
\end{equation*}

### Clamped

We can simply provide arbitrarily chosen values for the end tangents.
This is called *clamped* end conditions.

\begin{align*}
\dot{\boldsymbol{x}}_0 &= D_\text{begin}\\
\dot{\boldsymbol{x}}_{N-1} &= D_\text{end}
\end{align*}

This leads to a very simple first and last line:

\begin{equation*}
\left[
\begin{matrix}
1 &&&& \cdots & 0\\
1 & 4 & 1 &&& \vdots \\
& 1 & 4 & 1 && \\
&& \ddots & \ddots && \\
&& 1 & 4 & 1 & \\
\vdots &&& 1 & 4 & 1\\
0 & \cdots &&&& 1
\end{matrix}
\right]
\left[
\begin{matrix}
\dot{\boldsymbol{x}}_0\\
\dot{\boldsymbol{x}}_1\\
\vdots\\
\dot{\boldsymbol{x}}_{N-2}\\
\dot{\boldsymbol{x}}_{N-1}
\end{matrix}
\right]
=
\left[
\begin{matrix}
D_\text{begin}\\
3 (\boldsymbol{x}_2 - \boldsymbol{x}_0)\\
3 (\boldsymbol{x}_3 - \boldsymbol{x}_1)\\
\vdots\\
3 (\boldsymbol{x}_{N-2} - \boldsymbol{x}_{N-4})\\
3 (\boldsymbol{x}_{N-1} - \boldsymbol{x}_{N-3})\\
D_\text{end}
\end{matrix}
\right]
\end{equation*}

### Closed

We can close the spline by connecting $\boldsymbol{x}_{N-1}$
with $\boldsymbol{x}_0$.
This can be realized by cyclically extending the matrix
in both directions:

\begin{equation*}
\left[
\begin{matrix}
4 & 1 && \cdots & 0 & 1\\
1 & 4 & 1 && 0 & 0 \\
& 1 & 4 & 1 && \vdots \\
&& \ddots & \ddots && \\
\vdots && 1 & 4 & 1 & \\
0 & 0 && 1 & 4 & 1\\
1 & 0 & \cdots && 1 & 4
\end{matrix}
\right]
\left[
\begin{matrix}
\dot{\boldsymbol{x}}_0\\
\dot{\boldsymbol{x}}_1\\
\vdots\\
\dot{\boldsymbol{x}}_{N-2}\\
\dot{\boldsymbol{x}}_{N-1}
\end{matrix}
\right]
=
\left[
\begin{matrix}
3 (\boldsymbol{x}_1 - \boldsymbol{x}_{N-1})\\
3 (\boldsymbol{x}_2 - \boldsymbol{x}_0)\\
3 (\boldsymbol{x}_3 - \boldsymbol{x}_1)\\
\vdots\\
3 (\boldsymbol{x}_{N-2} - \boldsymbol{x}_{N-4})\\
3 (\boldsymbol{x}_{N-1} - \boldsymbol{x}_{N-3})\\
3 (\boldsymbol{x}_{0} - \boldsymbol{x}_{N-2})
\end{matrix}
\right]
\end{equation*}

## Solving the System of Equations

The matrices above are *tridiagonal* and can therefore
be solved efficiently with a
[tridiagonal matrix algorithm](https://en.wikipedia.org/wiki/Tridiagonal_matrix_algorithm).
The class [splines.Natural](../python-module/splines.rst#splines.Natural),
however,
is not very concerned about efficiency and simply uses NumPy's
[linalg.solve()](https://numpy.org/doc/stable/reference/generated/numpy.linalg.solve.html)
function to solve the system of equations.


Table of Contents

Chapter 2 - Elements of Matrix Theory

2.1.2 The Jordan Normal Form

Example 2.5 Revisiting the wireless sensor network example

NumPy/ SciPy approach

SymPy approach

2.1.3 Semi-convergence and convergence for discrete-time linear systems

Definition 2.6 (Spectrum and spectral radius of a matrix)

2.2.1 The spectral radius for row-stochastic matrices

Theorem 2.8 (Geršgorin Disks Theorem)

2.3.3 Applications to matrix powers and averaging systems

Theorem 2.13 (Powers of non-negative matrices with a simple and strictly dominant eigenvalue)

Example 2.14 Wireless sensor network

Exercises 2.18

Exercises 2.19

```python
%matplotlib widget

# Import packages
import numpy as np
import matplotlib.pyplot as plt
import matplotlib as mpl
import networkx as nx
import scipy.linalg as spla
from sympy import Matrix

# For interactive graphs
import ipywidgets as widgets

# Import self defined functions
import lib  # General library

# Settings
custom_figsize= (6, 4) # Might need to change this value to fit the figures to your screen
custom_figsize_square = (5, 5)
```

# Chapter 2 - Elements of Matrix Theory
These Jupyter Notebook scripts contain some examples, visualization and supplements accompanying the book "Lectures on Network Systems" by Francesco Bullo http://motion.me.ucsb.edu/book-lns/. These scripts are published with the MIT license. **Make sure to run the first cell above to import all necessary packages and functions and adapt settings in case.** In this script it is necessary to execute cell by cell chronologically due to reocurring examples (excepts for e.g. the Exercises in the end). (Tip: Use the shortcut Shift+Enter to execute each cell). Most of the functions are kept in separate files to keep this script neat.

## 2.1.2 The Jordan Normal Form
### Example 2.5 Revisiting the wireless sensor network example
The following cells are showing the computation of the Jordan Normal Form $J$, the invertible transformation matrix $T$ and some of its dependencies.

```python
# Defining the A matrix again
A = np.array([[1/2, 1/2, 0., 0.],
              [1/4, 1/4, 1/4, 1/4],
              [0., 1/3, 1/3, 1/3],
              [0., 1/3, 1/3, 1/3]
])
```

There is the possibility to calculate the Jordan Normal Form directly with the package SymPy https://docs.sympy.org/latest/index.html. However, we are determining the Jordan Normal Form via determining the generalized eigenvectors (read more for literature recommendations about generalized eigenvectors in the book) with the SciPy package first to discuss some possibilities and problems with non symbolic toolboxes.

### NumPy/ SciPy approach

From the documentation of scipy.linalg.eig: *'Solve an ordinary or generalized eigenvalue problem of a square matrix.'*

```python
# Right eigenvectors
lambdas, eigv = spla.eig(A)

# Left eigenvectors
lambdas2, eigw = spla.eig(A.T)
```

Due to numerical instabilities, the zero values are not reflected and it can be seen, how the expected eigenvalue of 1 is not precise. The zeros can be fixed with:

```python
def correct_close_to_zero(M, tol=1e-12):
    M.real[abs(M.real) < tol] = 0.0
    if M.imag.any():
        M.imag[abs(M.imag) < tol] = 0.0
    return M

eigv_cor = correct_close_to_zero(eigv)
eigw_cor = correct_close_to_zero(eigw)
lambdas_cor = correct_close_to_zero(lambdas)
lambdas2_cor = correct_close_to_zero(lambdas2)

print("Right eigenvectors:")
lib.matprint(eigv_cor)
print("\n")
print("Left eigenvectors:")
lib.matprint(eigw_cor)
print("\n")
print("Eigenvalues (right):")
lib.matprint(lambdas_cor)
print("\n")
print("Eigenvalues (left) for matching later:")
lib.matprint(lambdas2_cor)
```

There are two options now for $T^{-1}$: Taking the inverse of the right eigenvectors (which contains again numerical instabilities) or building it from the left eigenvectors, what would include some sorting to match the eigenvalue order from the  right eigenvector (often it is the case, that they are already aligned since calling scipy.linalg.eig twice on a matrix with the same eigenvalues).

```python
T = eigv_cor.copy()*-1  # Rescale the eigenvectors to match eigenvalues later
# Sorting if necessary, remember to use transpose, since in T^-1 the rows represent the left eigenvectors.
Tinv = eigw_cor.T.copy()
```

Now we can simply compute J, when compared, is fairly close to the solution in the book, however, due to numerical intabilities not precise. Further on, the order of the eigenvalues might be different than the on from the book.

```python
J = correct_close_to_zero(Tinv@A@T)
print("Jordan Normal Form via SciPy/Numpy:")
lib.matprint(J)
```

### SymPy approach

Now we use a symbolic toolbox package SymPy from python as a blackbox. Note, that also here the order of the eigenvalues might be different!

```python
Asym = Matrix(A) # Sympy Matrix toolbox object
Tsym, Jsym = Asym.jordan_form()
```

Here we can compare them with our previous results:

```python
print("Jordan Normal Form SymPy:")
lib.matprint(np.array(Jsym).astype(np.float64))
print("Jordan Normal Form SciPy:")
lib.matprint(J)
```

## 2.1.3 Semi-convergence and convergence for discrete-time linear systems

### Definition 2.6 (Spectrum and spectral radius of a matrix)
We display the spectrum of the previous A matrix with the spectrum radius for visualization purpose. Additionally, we also show how the spectrum of a randomly generated matrix.

```python
fig, ax213 = plt.subplots(figsize=custom_figsize_square)
lib.plot_spectrum(A, ax213);
```

```python
n_M1=8

# A unifornmly distributed, positive, row stochastic matrix vs not row stochastic
M1 = np.random.uniform(0, 1,(n_M1,n_M1))
M1 = M1 / M1.sum(axis=1, keepdims=1) # Row-stochastic
M2 = M1 - 0.05 # Not row-stochastic

fig, (ax2131, ax2132) = plt.subplots(1,2, figsize=(custom_figsize_square[0]*2, custom_figsize_square[1]))
lib.plot_spectrum(M1, ax2131);
lib.plot_spectrum(M2, ax2132);
```

## 2.2.1 The spectral radius for row-stochastic matrices
### Theorem 2.8 (Geršgorin Disks Theorem)

Similar to before, the Geršgorin Disks are now visualized for a row-stochastic matrix and another matrix.

```python
fig, (ax2211, ax2212) = plt.subplots(1,2, figsize=(custom_figsize_square[0]*2, custom_figsize_square[1]))
lib.plot_gersgorin_disks(M1, ax2211)
lib.plot_gersgorin_disks(M2, ax2212)
```

## 2.3.3 Applications to matrix powers and averaging systems

### Theorem 2.13 (Powers of non-negative matrices with a simple and strictly dominant eigenvalue)

Here is an example for Theorem 2.13, which shows, how the powers of primitive, row-stochastic matrices converges to rank 1. This is also done for the wireless sensor network example.

#### Example 2.14 Wireless sensor network
In the book it is shown, that the wireless sensor network matrix is primitive. Here, the eigenvectors and eigenvalues are printed again and compared for the semi convergence result $\lim_{k \to \infty} A^k = \mathbb{1}_n w^T$ to demonstrate Theorem 2.13 for a row-stochastic matrix.

```python
print("Left eigenvectors of A:")
lib.matprint(eigw_cor)
print("\n")

print("Eigenvalues (left) of A:")
lib.matprint(lambdas2_cor)
print("\n")

print("Normalizing dominant eigenvector:")
dom_eigv = eigw_cor[:, 0] / sum(eigw_cor[:, 0])
lib.matprint(dom_eigv)
print("\n")

print("Convergence result of A:")
lib.matprint(np.linalg.matrix_power(A, 50))
print("\n")

print("equals 1n*w^T")
lib.matprint(np.ones((4,1))@dom_eigv[:, None].T)
```

Below is a randomly generated example to show, that primitive, row-stochastic matrices always converge to rank 1. *Note: The code is not robust for semisimple eigenvalue of 1*

```python
# Creating a new random primitive (positiv), row stochastic matrix here
n_M11=5
M11 = np.random.uniform(0, 1,(n_M11,n_M11))
M11 = M11 / M11.sum(axis=1, keepdims=1) # Row-stochastic
print("Random primitive row-stochastic matrix M:")
lib.matprint(M11)
print("\n")

print("Left eigenvectors of M:")
l_M, m_eigv = spla.eig(M11.T)
m_eigv = correct_close_to_zero(m_eigv)
l_M = correct_close_to_zero(l_M)
lib.matprint(m_eigv)
print("\n")

print("Eigenvalues (left) of M:")
lib.matprint(l_M)
print("\n")

# Here we check the position with numerical unprecision of the eigenvalue 1
print("Normalizing dominant eigenvector:")
idx_dom = np.where(abs(l_M - 1) < 0.005)[0][0]
dom_eigv_M = m_eigv[:, 0] / sum(m_eigv[:, 0])
lib.matprint(dom_eigv_M)
print("\n")

print("Convergence result of M:")
lib.matprint(np.linalg.matrix_power(M11, 500))
print("\n")

print("equals 1n*w^T")
lib.matprint(np.ones((n_M11,1))@dom_eigv_M[:, None].T)
```

## Exercises 2.18

This section is similar to exercise 1.4, however, here we actually visualize the graph and its node values. Additionally, we show that the values converge to the initial values multiplied b the dominant left eigenvector as presented in Theorem 2.13 for row stochastic matrices.

First, we define the graphs and their adjacency Matrix A and simulate the results. Then, for each graph a cell can be executed for the interactive visualization. A plot of the states is available in the Jupyter Notebook script for Chapter 1.

```python
# Define x_0
xinitial = np.array([1., -1., 1., -1., 1.])

# Defining the 3 different systems.
# Complete graph
A_complete = np.ones((5,5)) / 5

# Cycle graph
A_cycle = np.array([
    [1/3, 1/3, 0, 0, 1/3],
    [1/3, 1/3, 1/3, 0, 0],
    [0, 1/3, 1/3, 1/3, 0],
    [0, 0, 1/3, 1/3, 1/3],
    [1/3, 0, 0, 1/3,  1/3] ]  )

# Star topology. center = node 1
A_star = np.array([
    [1/5, 1/5, 1/5, 1/5, 1/5],
    [1/2, 1/2, 0, 0, 0],
    [1/2, 0, 1/2, 0, 0],
    [1/2, 0, 0, 1/2, 0],
    [1/2, 0, 0, 0, 1/2]   ])

# Defining simulation time
ts = 15

# Defining graphs for plotting later
n = 5
G_star = nx.star_graph(n-1)
pos_star = {0:[0.5,0.8], 1:[0.2,0.6],2:[.4,.2],3:[.6,.2],4:[.8,.6]}
G_cycle = nx.cycle_graph(n)
pos_cycle = {0:[0.5,0.8], 1:[0.35,0.6],2:[.4,.3],3:[.6,.3],4:[.65,.6]}
G_complete = nx.complete_graph(n)
pos_complete = pos_cycle.copy()

# Simulating and saving each network
states_complete = lib.simulate_network(A_complete,xinitial, ts)
states_star = lib.simulate_network(A_star,xinitial, ts)
states_cycle = lib.simulate_network(A_cycle,xinitial, ts)
```

**Complete graph**

Showing complete graph interactive simulation and Theorem 2.13

```python
fig, ax2181 = plt.subplots(figsize=custom_figsize)

# If this cell is executed twice we are making sure in the following, that the previous widget instances are all closed
try:
    [c.close() for c in widget2181.children]  # Note: close_all() does also affect plot, thus list compr.
except NameError:  # Only want to except not defined variable error
    pass

widget2181 = lib.interactive_network_plot(G_complete, states_complete, pos_complete, ts, fig, ax2181)

display(widget2181)

# Verifying the results
eigval, eigvec = np.linalg.eig(A_complete.transpose())
idx_dom = np.argmax(eigval)
dom_eigvec = eigvec[0:5,idx_dom]/eigvec[0:5,idx_dom].sum()
print("Showing Theorem 2.13 for the complete graph")
print("Dominant eigenvector: \n", dom_eigvec)
print("Final values : \n", xinitial@dom_eigvec*np.ones(5))
```

**Star graph**

Showing star graph interactive simulation and Theorem 2.13

```python
fig, ax2182 = plt.subplots(figsize=custom_figsize)

# If this cell is executed twice we are making sure in the following, that the previous widget instances are all closed
try:
    [c.close() for c in widget2182.children]  # Note: close_all() does also affect plot, thus list compr.
except NameError:  # Only want to except not defined variable error
    pass

widget2182 = lib.interactive_network_plot(G_star, states_star, pos_star, ts, fig, ax2182)

display(widget2182)

# Verifying the results
eigval, eigvec = np.linalg.eig(A_star.transpose() )
idx_dom = np.argmax(eigval)
dom_eigvec = eigvec[0:5,idx_dom]/eigvec[0:5,idx_dom].sum()
print("Showing Theorem 2.13 for the star graph")
print("Dominant eigenvector: \n", dom_eigvec)
print("Final values : \n", xinitial@dom_eigvec*np.ones(5))
```

**Cycle graph**

Showing cycle graph interactive simulation and Theorem 2.13

```python
fig, ax2183 = plt.subplots(figsize=custom_figsize)

# If this cell is executed twice we are making sure in the following, that the previous widget instances are all closed
try:
    [c.close() for c in widget2183.children]  # Note: close_all() does also affect plot, thus list compr.
except NameError:  # Only want to except not defined variable error
    pass

widget2183 = lib.interactive_network_plot(G_cycle, states_cycle, pos_cycle, ts, fig, ax2183)

display(widget2183)

# Verifying the results
eigval, eigvec = np.linalg.eig(A_cycle.transpose())
idx_dom = np.argmax(eigval)
dom_eigvec = eigvec[0:5,idx_dom]/eigvec[0:5,idx_dom].sum()
print("Showing Theorem 2.13 for the cycle graph")
print("Dominant eigenvector: \n", dom_eigvec)
print("Final values : \n", xinitial@dom_eigvec*np.ones(5))
```

## Exercises 2.19

This exercise is about $n$ robots moving on the line trying to gather at a common location (i.e., reach rendezvous), where each robot heads for the centroid of its neighbors. The visualization of the algorithm deals with the discrete part of the task, where on can explore values of the sampling period $T$ for the Euler discretization.

```python
# Setup - change these parameters if wanted
n_robots = 8
# Number of timesteps and sampling period T (If T is too small, need very high n_dt)
n_dt = 25
T = 0.3  # Play around with this value, something interesting is happening around (2(n_robots-1)/n_robots)
#T = 2*(n_robots-1)/n_robots

# Set up initial position matrix and further saving variables
current_positions = 2*np.random.random((n_robots,1))-1
new_position = current_positions.copy()
all_positions = np.zeros((n_dt, n_robots, 1))
all_positions[0] = current_positions.copy()

for tt in range(1, n_dt):
    for index, own_pos in np.ndenumerate(current_positions):
        new_position[index] = own_pos + T*(1/(n_robots-1)*(np.sum(current_positions)-own_pos) - own_pos)
    all_positions[tt] = new_position.copy()
    current_positions = new_position.copy()
```

```python
fig, ax219 = plt.subplots(figsize=custom_figsize)
# Set colors of robots for tracking
all_colors = np.random.rand(n_robots,3)

def plot_robot_pos(ax, pos):
    # Set xlim, ylim and aspect ratio
    ax219.set_xlim(-1.5, 1.5)
    ax219.set_ylim(-0.5, 0.5)
    ax219.set_aspect('equal')
    # Add horizontal line
    ax219.axhline(y=0.0, color='k', linestyle='-')
    for i in range(0, pos.shape[0]):
        # Add a robot as circle
        bug = mpl.patches.Circle((pos[i], 0), radius=0.06, ec='black', color=all_colors[i])
        ax.add_patch(bug)

def interactive_robots(timestep):
    ax219.clear()
    plot_robot_pos(ax219, all_positions[timestep['new'], :]) # Take the new value received from the slider dict
    return None

# Plot initial configuration
plot_robot_pos(ax219, all_positions[0, :])

# Widget
# If this cell is executed twice we are making sure in the following, that the previous widget instances are all closed
try:
    [c.close() for c in widget219.children]  # Note: close_all() does also affect plot, thus list compr.
except NameError:  # Only want to except not defined variable error
    pass

widget219 = lib.create_widgets_play_slider(fnc=interactive_robots, minv=0, maxv=n_dt-1, step=1, play_speed=500)

display(widget219)
```

We can even compute the convergence solution in advance. Please refer and solve the given Excersice first to try to understand the following calculations.

```python
A_robot = 1/(n_robots-1) * np.ones((n_robots, n_robots)) - n_robots/(n_robots-1)*np.identity(n_robots)
eigval, eigvec = np.linalg.eig(A_robot)
idx = np.argmin(abs(eigval))
z_eigvec = eigvec[:,idx]/np.sqrt(np.sum(eigvec[:,idx]**2))

final_values = z_eigvec[None, :] @ all_positions[0] @ z_eigvec[None, :]
print("Final values :", final_values)
```

```python

```