# Linear Regression
Linear Regression is one of the very basic supervised learning. It is usually used to predict continuous values. A simple example will be used to explain it.

A linear function will be used in linear regression. Usually it looks like this

\begin{gather}
h_{\theta}(x) = \theta_0 + \theta_1 x_1 + \theta_2 x_2 + .... + \theta_n x_n
\end{gather}

where $x$ are features, specificly $x_0 = 1$, $\theta$s are weights.

With matrix operation, it can be further simplified as:

\begin{gather}
h_{\theta}(x) =
\begin{bmatrix}
\theta_0 & \theta_1 & ... & \theta_n
\end{bmatrix}
\begin{bmatrix}
1\\
x_1\\
... \\
x_n
\end{bmatrix}
\end{gather}

We then define a **cost function** to measure how close the $h_{\theta}(x)$'s to the corresponding $y^{(i)}$:

\begin{gather}
J(\theta) = \frac{1}{2}\sum_{i=1}^m{(h_\theta(x^{(i)})} - y^{(i)})^2
\end{gather}

where $m$ is the size of the training data, $(y^{(i)}, x^{(i)})$ denotes the $i_{th}$ training example.

## LMS algorithm
The goal is to minimize $J(\theta)$. TO do so, let's first use a search algorithm which starts with initial guess for $\theta$, and keep changing $\theta$ to make $J(\theta)$ smaller, until we converge to a value of $\theta$ that minimize $J(\theta)$. One very popular algorithm is **gradient descent**, which starts with some initial $\theta$, and repeatedly performs the updates:
\begin{gather}
\theta_j := \theta_j - \alpha\frac{\partial{J(\theta)}}{\partial\theta_j}
\end{gather}

$\alpha$ is the **training rate**, which will desice the step size in the direction of steepest decrease of $J$. $j = 0, 1 ..., n$

Inserting $J(\theta)$ into the derivative, we have:

\begin{align}
\frac{\partial{J(\theta)}}{\partial\theta_j} &= \frac{\partial}{\partial{\theta_j}}\frac{1}{2}(h_{\theta}(x) - y)^2 \\
& = 2*\frac{1}{2}(h_{\theta}(x) - y)\frac{\partial}{\partial\theta_j}(h_{\theta}(x)-y)\\
& = (h_{\theta}(x)-y)\frac{\partial}{\partial\theta_j}(\sum\limits_{ i=0}^n (\theta_ix_i - y))\\
& = (h_\theta(x) - y)x_j
\end{align}

For a single training example, the update will be:

\begin{gather}
\theta_j := \theta_j + \alpha(y^{(i)}-h_\theta(x^{(i)}))x_j^{(i)}
\end{gather}

This is **LMS** (least mean squares). There are two ways to modify this method for a training set of more than one example. The first is as following:

>>*Repeat until convergence {*
>>>$\theta_j :=  \theta_j + \alpha\sum_{i=0}^m (y^{(i)} - h_{\theta}(x^{(i)}))x_j^{(i)}$    (for every $j$)

>>*}*

This method looks at every training example in the taining data set on every step, and is called **batch gradient descent**.

Now, let's implement batch gradient descent.

Given a data set, we first load the data to check the information.

```
import numpy as np
import pandas as pd
import seaborn as sns

```

```
df_tr = pd.read_csv("salary_data.csv")
```

```
df_tr.info()
df_tr.head()
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 30 entries, 0 to 29
    Data columns (total 2 columns):
     #   Column           Non-Null Count  Dtype
    ---  ------           --------------  -----
     0   YearsExperience  30 non-null     float64
     1   Salary           30 non-null     int64
    dtypes: float64(1), int64(1)
    memory usage: 608.0 bytes

      YearsExperience
      Salary

      0
      1.1
      39343

      1
      1.3
      46205

      2
      1.5
      37731

      3
      2.0
      43525

      4
      2.2
      39891

Good, the training data set has no missing values. Next, we plot the data and check the distribution.

```
sns.scatterplot(data=df_tr, x="YearsExperience", y="Salary")
```

From the plot, we see that YearsExperience and Salary have positive linear relation. For this section linear reression, we construct a linear function as follows:

\begin{gather}
h_{\theta}(x) = \theta_0 + \theta_1 x_1
\end{gather}

where $x_0 = 1$, $x_1$ is YearsExperience, $\theta$ are weights. With matrix form:

\begin{gather}
h_{\theta}(x) =
\begin{bmatrix}
\theta_0 & \theta_1
\end{bmatrix}
\begin{bmatrix}
1\\
x_1
\end{bmatrix}
\end{gather}

The following code is the implementation.

```
def batch_gradient_descent(x, y, *args, **kwargs):
    """
    Algorithm for simple linear regression.
    Input: x training data, column vector or matrix, each column for one feature
           y training data, ground truth, column vector
           alpha learning rate optional
           tol tolorance optional
    Output: theta estimated parameters
            h estimations
    """

    alpha = 10**-3
    tol = 10**-5

    if len(kwargs) > 2:
      raise Exception("Too many inputs.")
    elif len(kwargs) > 0:
      if "alpha" in kwargs.keys():
        alpha = kwargs["alpha"]
      if "tol" in kwargs.keys():
        tol = kwargs["tol"]

    num_feature = np.shape(x)[1]
    num_example = np.shape(x)[0]
    theta = np.ones((num_feature + 1, 1)) * 0.2
    new_col = np.ones(num_example)
    x = np.insert(x, 0, new_col, axis=1)
    # self.tol = self.tol * num_example

    h = np.dot(x, theta)  # our hypothesis
    num_loop = 0
    pj = 0

    while num_loop <= 10000:
        tmp = alpha * sum((y - h) * x)
        theta = theta + tmp.reshape((num_feature + 1, 1))
        h = np.dot(x, theta)
        j = 0.5 * sum((h - y) ** 2)
        if abs(j - pj) <= tol:
            return theta, h
        pj = j
        num_loop = num_loop + 1
    print("gradient descent does not convergence after maximum interation.")
    return theta, h

```

```
x = df_tr["YearsExperience"].values
y = df_tr['Salary'].values
n = np.size(y)
x = x.reshape((n, 1))
y = y.reshape((n, 1))
theta1, h1 = batch_gradient_descent(x, y)
```

Plot of our estimation over the training data, we get a stright line which go through the data. The esimation is reasonable.

```
sns.scatterplot(data=df_tr, x="YearsExperience", y="Salary")
sns.lineplot(data=df_tr, x="YearsExperience", y=h1.reshape(np.size(h1)), color="red")
```

An alternative to batch gradient descent is **stochastic gradient descent**. Instead of looking at every example in each update step, this method update the parameters with every single training example. It starts the estimation right away:
>> *Loop {*
>>>for i = 1 to m, {
>>>>$\theta_j := \theta_j + \alpha(y^{(i)} - h_\theta(x^{(i)}))x_j^{(i)}$

>>>}

>>*}*

Therefore, stochastic gradient descent is much faster tha batch gradient descent, especially when the size of traing set is large. (Note however $\theta$ may never converge to the minimum, and it will keep oscillating around the minimum of $J(\theta)$. In practice, the values near the minimum will be reasonally good approximations to the true minimum.) In practice, when the training data size is large, stochastic gradient descent is often preferred than batch gradient descent.

The implementation is as follows:

```
def stochastic_gradient_descent(x, y, *args, **kwargs):
    """
    Algorithm for simple linear regression.
    Input: x training data, column vector or matrix, each column for one feature
           y training data, ground truth, column vector
           alpha learning rate optional
           tol tolorance optional
    Output: theta estimated parameters
            h estimations
    """

    alpha = 10**-3
    tol = 10**-5

    if len(kwargs) > 2:
      raise Exception("Too many inputs.")
    elif len(kwargs) > 0:
      if "alpha" in kwargs.keys():
        alpha = kwargs["alpha"]
      if "tol" in kwargs.keys():
        tol = kwargs["tol"]

    num_feature = np.shape(x)[1]
    num_example = np.shape(x)[0]
    theta = np.ones((num_feature + 1, 1)) * 0.2
    new_col = np.ones(num_example)
    x = np.insert(x, 0, new_col, axis=1)
    # self.tol = self.tol * num_example

    h = np.dot(x, theta)  # our hypothesis
    num_loop = 0
    pj = 0

    while num_loop <= 10000:
      for i in range(num_example):
        h = np.dot(x[i, :], theta)
        j = 0.5 * ((h - y[i]) ** 2)
        if abs(j - pj) <= tol:
          h_all = np.dot(x, theta)
          return theta, h_all
        else:
          tmp = alpha * ((y[i] - h) * x[i, :])
          theta = theta + tmp.reshape((num_feature + 1, 1))
        pj = j
      num_loop = num_loop + 1
    print("gradient descent does not convergence after maximum interation.")
    h_all = np.dot(x, theta)
    return theta, h_all

```

```
 x = df_tr["YearsExperience"].values
 y = df_tr['Salary'].values
 n = np.size(y)
 x = x.reshape((n, 1))
 y = y.reshape((n, 1))
theta2, h2 = stochastic_gradient_descent(x, y)
```

    gradient descent does not convergence after maximum interation.

```
sns.scatterplot(data=df_tr, x="YearsExperience", y="Salary")
sns.lineplot(data=df_tr, x="YearsExperience", y=h2.reshape(np.size(h2)), color="red")
```

As told above, stochastic gradient descent may not converge, but the result is resonably good. The stop condition of stochastic gradient descent usually use early stopping in validation.

## The normal equations
This method will minimize $J$ by explicitly taking its derivatives with respect to th $\theta$s, and setting them to zeros.

With matrix operations, we can have **normal equations**:

\begin{align}
\nabla_\theta J(\theta) &= \nabla_\theta\frac{1}{2}(X\theta-\vec y)^T(X\theta-\vec y)\\
&= X^TX\theta - X^T \vec y
\end{align}

Set it to zero, further have:
\begin{gather}
 X^TX\theta = X^T \vec y
 \end{gather}

 Thus, the value $\theta$ is given in closed form by the equation
 \begin{gather}
 \theta = (X^TX)^{-1}X^T\vec y
 \end{gather}

 (Details see [here](https://github.com/DakaiZhou/machine-learnng-notes-andrew-ng/blob/master/cs229-notes-all/cs229-notes1.pdf).)

```
def normal_euqation(x, y):
  '''
  Input: x features in traning data
         y ground truth in training data

  Output: theta estimated parameter
          h final extimations
  '''
  num_example = np.shape(x)[0]
  new_col = np.ones(num_example)
  x = np.insert(x, 0, new_col, axis=1)
  theta = np.dot(np.dot(np.linalg.inv(np.dot(x.transpose(), x)), x.transpose()), y)
  h = np.dot(x, theta)
  return theta, h
```

```
# x = df_tr["YearsExperience"].values
# y = df_tr['Salary'].values
# n = np.size(y)
# x = x.reshape((n, 1))
# y = y.reshape((n, 1))
theta3, h3 = normal_euqation(x, y)
```

```
sns.scatterplot(data=df_tr, x="YearsExperience", y="Salary")
sns.lineplot(data=df_tr, x="YearsExperience", y=h3.reshape(np.size(h3)), color="red")
```

```
theta3 - theta1
```

    array([[ 0.01550162],
           [-0.00230038]])

The estimated parameters by normal equations are very close to the other two methods.

## Locally weighted linear regression

The choice of features is important. Less features or too much features may cause underfitting and overfitting. **Locally weighted linear regression** is one of the algorithms that makes the choice of features less critical.

The locally weighted linear regression algorithm does the following:

1.   Fit $\theta$ to minimize $\sum_iw^{(i)}(y^{(i)}-\theta^Tx^{(i)})^2.$
2.   Output $\theta^Tx$.

When $w^{(i)}$ is large, $\theta$ will try hard to make $y^{(i)}-\theta^Tx^{(i)}$ small. When $w^{(i)}$ is small, $y^{(i)}-\theta^Tx^{(i)}$ will be pretty much ignored in the fitting process.

A fairly standard choice for the weights is:

\begin{gather}
w^{(i)} = \exp(-\frac{(x^{(i)} - x)^2}{2\tau^2})

\end{gather}

It is like a bell shape function, the $x^{(i)}$ close to $x$ will have larger weight. Hence, $\theta$ is choosing giving a much higher weight when the training examples close to query point x. The parameter $\tau$ controls how quickly the weight of a training example falls oﬀ with distance of its $x^{(i)}$ from the query point $x$, it is named **bandwidth**.

Unlike unweighted linear regression, locally weighted linear regression needs to keep the training set to make prediction. It is called **non-parametric **.  The term “non-parametric” (roughly) refers to the fact that the amount of stuﬀ we need to keep in order to represent the hypothesis $h$ grows linearly with the size of the training set.

The normal equation for locally weighted linear regression is:
\begin{gather}
\nabla_\theta J(\theta) = \nabla_\theta\frac{1}{2} (W(X\theta-\vec y))^T(X\theta-\vec y) := 0\\
\theta = (X^TWX)^{-1}(X^TW\vec y)
\end{gather}
where $W$ is
\begin{gather}
\begin{bmatrix}
w^0 & 0 & 0 & ... & 0\\
0 & w^1 & 0 & ... & 0\\
. & . & . & ... &   .\\
. & . & . & ... &   .\\
. & . & . & ... &   .\\
0 & 0 & 0 & ... & w^m
\end{bmatrix}
\end{gather}

The implementation is as follows:

```
def locally_weighted_linear_regression(x, y, i, tau):
  '''
  Input: x features in traning data
         y ground truth in training data
         i index of the query point
         tau bandwidth

  Output: theta estimated parameter
          h final extimations
          w weights
  '''

  num_example = np.shape(x)[0]
  new_col = np.ones(num_example)
  x = np.insert(x, 0, new_col, axis=1)
  w = np.exp(-(x[i, 1] - x[:, 1])**2 / (2 * tau**2))
  W = np.diag(w)
  theta = np.dot(np.linalg.inv(np.dot(np.dot(x.transpose(), W), x)), np.dot(np.dot(x.transpose(), W), y))
  h = np.dot(x, theta)
  return theta, h, w

```

```
theta4, h4, w = locally_weighted_linear_regression(x, y, 9, 0.3)
```

```
w
```

    array([4.89586526e-017, 1.26641655e-014, 2.10040929e-012, 1.06453714e-007,
           3.72665317e-006, 2.85655008e-002, 6.57285286e-002, 2.49352209e-001,
           2.49352209e-001, 1.00000000e+000, 8.00737403e-001, 6.06530660e-001,
           6.06530660e-001, 4.11112291e-001, 2.85655008e-002, 3.35462628e-004,
           1.86644691e-005, 6.65836147e-007, 2.10040929e-012, 1.72412094e-013,
           6.50878852e-024, 1.28423137e-028, 2.74878501e-043, 1.38634329e-049,
           4.80089224e-061, 1.68241585e-068, 6.84205918e-082, 1.02866199e-084,
           7.95674389e-106, 2.72002624e-112])

```
sns.scatterplot(data=df_tr, x="YearsExperience", y="Salary")
sns.scatterplot(data=df_tr, x=df_tr.loc[w > 0.01]["YearsExperience"], y=df_tr.loc[w>0.01]["Salary"], color="green")
sns.lineplot(data=df_tr, x="YearsExperience", y=h4.reshape(np.size(h4)), color="red")
```

In the above locally weighted linear regression plot, the line fits with the green points. The middle green point has the largest weight, and the weights decrease proportional to the distance away from the middle point. Only a segment of this line(which fits the green points) will be counted in the final result. The final fit will be a combination of segments from multiple lines.


```
import numpy as np
from scipy.integrate import odeint
import matplotlib.pyplot as plt
from ipywidgets import interactive,IntSlider,FloatSlider,IntText,FloatText
from IPython.display import Javascript
def resize_colab_cell():
  display(Javascript('google.colab.output.setIframeHeight(0, true, {maxHeight: 5000})'))
get_ipython().events.register('pre_run_cell', resize_colab_cell)
```

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

We denote with $X$ and $Y$ the number of preys and predators, respectively. We now make the following assumptions:
* The prey population *Y* have an unlimited food supply at all times.
*  In the absence of predators, the prey population $X$ would grow proportionally to its size with a constant of proportionality α>0 which Volterra called coefficient of auto-increase.
* In the absence of preys, the predator population $Y$ would decline proportionally to its size and the constant of proportionality is β>0.
* When both predator and prey are present, a decline in the prey population and a growth in the predator population will occur, each at a rate proportional to the frequency of encounters between individuals of the two species. These two constants of proportionality are indicate by δ>0 and γ>0.

We can translate these words into a system of two coupled ODEs, (remember $X=$ preys while $Y$= predators)
    \begin{cases}
      \frac{dX}{dt}=\alpha X- \delta XY\\
      \frac{dY}{dt}=-\beta Y + \gamma XY
    \end{cases}

The stationary points of this model can be found by rewriting the above equations as follows
    \begin{cases}
      \frac{dX}{dt}=(\alpha - \delta Y)X\\
      \frac{dY}{dt}=-(\beta - \gamma X)Y
    \end{cases}
  and imposing the two equations to be equal to zero simultaneously
  \begin{cases}
      X(\alpha - \delta Y)=0\\
      Y(-\beta  +\gamma X)=0
 \end{cases}

ln this model there are two stationary points given by:

$P_{1}=(0,0)$
$P_{2}=(\beta/\gamma,\alpha/\delta)$

In order to determine the nature of the stationary point we perform a linear analysis of our system be defining the *Jacobian* matrix $J$ as
\begin{equation}
J =
\begin{bmatrix}
  \frac{\partial F(X,Y)}{\partial X} &
    \frac{\partial F(X,Y)}{\partial Y}\\[1ex]
  \frac{\partial G(X,Y)}{\partial X} &
    \frac{\partial G(X,Y)}{\partial Y}
\end{bmatrix}
\end{equation}
where we have defined
\begin{equation}
    F(X,Y)= \alpha X- \delta XY,\quad\text{and}\quad  G(X,Y)=-\beta Y + \gamma XY.
\end{equation}

The Jacobian matrix becomes
\begin{equation}
J(X,Y)=
\begin{bmatrix}
  \alpha-\delta Y&
    -\delta X\\[1ex]
  \gamma Y& -\beta +\gamma X
\end{bmatrix},
\end{equation}

\begin{equation}
J(0,0)=
\begin{bmatrix}
  \alpha&
    0\\[1ex]
  0& -\beta
\end{bmatrix}.
\end{equation}

\begin{equation}
    \lambda_1=\alpha, \qquad \text{and}\qquad \lambda_2=-\beta.   \end{equation}

 The eigenvectors associated with the two eigenvalues are given by solving the following system of equations
\begin{equation}
\begin{bmatrix}
  \alpha &
    0\\[1ex]
  0& -\beta
\end{bmatrix}
\begin{bmatrix}
  x\\[1ex]
  y\end{bmatrix}=\begin{bmatrix}
  \lambda x\\[1ex]
  \lambda y\end{bmatrix},
\end{equation}
which means
\begin{equation}
    \begin{cases}
      \alpha x = \lambda x\\
      -\beta y = \lambda y
    \end{cases}
    \qquad
    \begin{cases}
     (\alpha -\lambda) x = 0\\
     (-\beta -\lambda)y = 0
    \end{cases}
\end{equation}

Therefore, if $\lambda=\alpha$
\begin{equation}
    \begin{cases}
      0=0\\
      (-\beta-\alpha)y =0
    \end{cases}
    \qquad
    \begin{cases}
      x \quad \text{any value}\\
      y =0
    \end{cases}
\end{equation}
and if $\lambda=-\beta$
\begin{equation}
    \begin{cases}
      (\alpha + \beta)y =0 \\
      0=0
    \end{cases}
    \qquad
    \begin{cases}
      x=0\\
      y \quad \text{any value}
    \end{cases}
\end{equation}
In conclusion, the eigenvectors associated with the two eigenvalues are given by
\begin{equation}
    \textbf{e}_1=\left( {\begin{array}{c}
   1\\
   0\\
  \end{array} } \right)
\qquad \text{and} \qquad
    \textbf{e}_2=\left( {\begin{array}{c}
   0\\
   1\\
  \end{array} } \right).
\end{equation}

Knowledge of eigenvalues and eigenvectors allows us to write down the general solution for our system
\begin{equation}
\left( {\begin{array}{c}
   X(t) \\
   Y(t) \\
  \end{array} } \right) =C_1e^{-\lambda_1 t}\mathbf{e_1} +C_2 e^{\lambda_2 t}\mathbf{e_2}=C_1e^{-\alpha t}\left( {\begin{array}{c}
   0 \\
   1 \\
  \end{array} } \right) +C_2 e^{\beta t}\left( {\begin{array}{c}
   1 \\
   0 \\
  \end{array} } \right).
\end{equation}
We can see that if $t\to+\infty$, the second exponential diverges while the first one converges to zero.

```
def lotkaVolterra(N,t,a,d,b,g):
	dN0=a*N[0]-d*N[0]*N[1]
	dN1=g*N[0]*N[1]-b*N[1]
	return np.array([dN0,dN1])

```

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

```
a=2.0
d=0.06
b=1.0
g=0.05
T=np.linspace(1,40,10000)
```

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

```
fig, ax = plt.subplots()
ax.plot(0,0,"ko")
ax.plot(b/g,a/d,"bo")
ax.set_xlabel('X')
ax.set_ylabel('Y')
plt.show()
```

```
fig, ax = plt.subplots()
ax.plot(0,0,"ko")
ax.plot(b/g,a/d,"bo")

ax.set_xlim(-1,70)
ax.set_ylim(-1,70)
my_xlim=ax.get_xlim()[1]
my_ylim=ax.get_ylim()[1]
dx=my_xlim/10
dy=my_ylim/10
[ax.arrow(0,6+i*dy,0,-dy,width=.3,head_width=1.5) for i in range(10)]
[ax.arrow(.1+i*dx,0,dx,0,width=.3,head_width=1.5) for i in range(10)]

ax.set_xlabel('X')
ax.set_ylabel('Y')
ax.grid()
plt.show()
```

\begin{cases}
      \frac{dX}{dt}=(\alpha - \delta Y)X\\
      \frac{dY}{dt}=-(\beta + \gamma X)Y
    \end{cases}

$\frac{dX}{dt}>0$ if $Y<\frac{\alpha}{\delta}$

and $\frac{dY}{dt}>0$ if $X<\frac{\beta}{\gamma}$

We now focus on the second stationary point $P_2=(\alpha/\delta , \beta/\gamma)$ and we evaluate the Jacobian
\begin{equation}
J(\beta /\gamma,\alpha /\delta)=
\begin{bmatrix}
  \alpha-\delta\cdot \frac{\alpha}{\delta}&
    -\delta \cdot \frac{\beta}{\gamma}\\[1ex]
  \gamma \cdot \frac{\alpha}{\delta}& -\beta +\gamma \cdot\frac{\beta}{\gamma}
\end{bmatrix}=\begin{bmatrix}
  0& -\delta \cdot \frac{\beta}{\gamma}
    \\[1ex]
  \gamma\cdot \frac{\alpha}{\delta}& 0
\end{bmatrix}.
\end{equation}
The eigenvalues are obtained by evaluating the determinant of the following matrix
\begin{equation}
    M=\begin{bmatrix}
  -\lambda& -\delta \cdot \frac{\beta}{\gamma}
    \\[1ex]
  \gamma\cdot \frac{\alpha}{\delta}& -\lambda
\end{bmatrix},
\end{equation}
which is given by
\begin{equation}
    \text{det}M = \lambda^2 + \gamma\frac{\alpha}{\delta}\cdot \delta\frac{\beta}{\gamma}=\lambda^2 +\alpha\beta,
\end{equation}
whose solutions are given by
\begin{equation}
    \lambda_1=+i\sqrt{\alpha\beta}\qquad \text{and} \qquad \lambda_2=-i\sqrt{\alpha\beta}.
\end{equation}
The eigenvalues are complex and the real part is zero. We conclude that the point is a *center*.

```
fig, ax = plt.subplots()
ax.plot(0,0,"ko")
ax.plot(b/g,a/d,"bo")

ax.set_xlim(-1,70)
ax.set_ylim(-1,70)
my_xlim=ax.get_xlim()[1]
my_ylim=ax.get_ylim()[1]
dx=my_xlim/10
dy=my_ylim/10
[ax.arrow(0,6+i*dy,0,-dy,width=.3,head_width=1.5) for i in range(10)]
[ax.arrow(.1+i*dx,0,dx,0,width=.3,head_width=1.5) for i in range(10)]

plt.vlines(b/g,0,70,color='b',linestyle='dashed')
plt.hlines(a/d,0,70,color='b',linestyle='dashed')
plt.text(b/g,20,r'$\frac{\beta}{\gamma}$',fontsize=20)
plt.text(2,4+a/d,r'$\frac{\alpha}{\delta}$',fontsize=20)
ax.set_aspect('equal')
plt.xlabel('$preys\quad (X)$')
plt.ylabel('$predators\quad (Y)$');
```

\begin{cases}
      \frac{dX}{dt}=(\alpha - \delta Y)X\\
      \frac{dY}{dt}=-(\beta - \gamma X)Y
    \end{cases}

$\frac{dX}{dt}>0$  if $Y\lt\frac{\alpha}{\delta}$

$\frac{dY}{dt}>0$ if $X\gt\frac{\beta}{\gamma}$

```
circ=lambda theta,r,hx,hy: [hx+r*np.cos(theta),hy+r*np.sin(theta)]
fig, ax = plt.subplots()
ax.plot(0,0,"ko")
ax.plot(b/g,a/d,"bo")
print(b/g,a/d)
ax.set_xlim(-1,70)
ax.set_ylim(-1,70)
my_xlim=ax.get_xlim()[1]
my_ylim=ax.get_ylim()[1]
dx=my_xlim/10
dy=my_ylim/10
[ax.arrow(0,6+i*dy,0,-dy,width=.3,head_width=1.5) for i in range(10)]
[ax.arrow(.1+i*dx,0,dx,0,width=.3,head_width=1.5) for i in range(10)]

plt.vlines(b/g,0,70,color='b',linestyle='dashed')
plt.hlines(a/d,0,70,color='b',linestyle='dashed')
plt.text(b/g,20,r'$\frac{\beta}{\gamma}$',fontsize=20)
plt.text(60,3+a/d,r'$\frac{\alpha}{\delta}$',fontsize=20)
plt.arrow(*circ(0.75*np.pi,15,b/g,a/d),0,-5,width=.3,head_width=1.5)
plt.arrow(*circ(-0.75*np.pi,15,b/g,a/d),0,-5,width=.3,head_width=1.5)
plt.arrow(*circ(0.25*np.pi,15,b/g,a/d),0,5,width=.3,head_width=1.5)
plt.arrow(*circ(-0.25*np.pi,15,b/g,a/d),0,5,width=.3,head_width=1.5)
#
plt.arrow(*circ(0.75*np.pi,15,b/g,a/d),-5,0,width=.3,head_width=1.5)
plt.arrow(*circ(-0.75*np.pi,15,b/g,a/d),5,0,width=.3,head_width=1.5)
plt.arrow(*circ(0.25*np.pi,15,b/g,a/d),-5,0,width=.3,head_width=1.5)
plt.arrow(*circ(-0.25*np.pi,15,b/g,a/d),5,0,width=.3,head_width=1.5)
ax.set_aspect('equal')
plt.xlabel('$preys\quad (X)$')
plt.ylabel('$predators\quad (Y)$')
```

```
N=np.array([50,1])
T=np.linspace(0,16,1000)
Nt=odeint(lotkaVolterra,N,T,args=(a,d,b,g))
```

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

```
ax=plt.subplot()
plt.plot(Nt[:,0],Nt[:,1])
plt.plot(Nt[0,0],Nt[0,1],'ro',markerSize=10)
plt.vlines(b/g,0,300,color='b',linestyle='dashed')
plt.hlines(a/d,0,300,color='b',linestyle='dashed')
ax.set_aspect('equal')
plt.xlabel('$preys\quad (X)$')
plt.ylabel('$predators\quad (Y)$')
```

```
N_ini=np.array([[16,26.666667],[44,73.3333333],[72,120],[100,166.66666667],[140,233.33333333]])
Nts=[]
for No in N_ini:
  Nt=odeint(lotkaVolterra,No,T,args=(a,d,b,g))
  Nts.append(Nt)

```

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

```
fig, ax = plt.subplots()
for Nt in Nts:
  ax.plot(Nt[:,0],Nt[:,1])
  ax.plot(Nt[0,0],Nt[0,1],'ro',markerSize=10)

ax.vlines(b/g,0,340,color='b',linestyle='dashed')
ax.hlines(a/d,0,340,color='b',linestyle='dashed')

ax.set_aspect('equal')
plt.xlabel('$preys\quad (X)$')
plt.ylabel('$predators\quad (Y)$')
```

```
ymax=100
xmax=100
n_points=25
x=np.linspace(0,xmax,n_points)
y=np.linspace(0,ymax,n_points)
X,Y=np.meshgrid(x,y)
dX,dY=lotkaVolterra([X,Y],0,a,d,b,g)

```

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

```
hypotenusa =lambda x,y : np.sqrt(x**2+y**2)
```

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

```
h=hypotenusa(dX,dY)
h[h==0]=1
dX=dX/h
dY=dY/h
fig, ax = plt.subplots()
Q = ax.quiver(X,Y,dX, dY, h, pivot='mid', cmap=plt.cm.jet)
plt.vlines(g/b,0,100,color='b',linestyle='dashed')
plt.hlines(a/d,0,100,color='b',linestyle='dashed')

ax.set_aspect('equal')
plt.xlabel('$preys\quad (X)$');
plt.ylabel('$predators\quad (Y)$');

```

```
N_ini=np.column_stack((X.diagonal()[4:20:4],Y.diagonal()[4:20:4]))
Nts=[]
for No in N_ini:
  Nt=odeint(lotkaVolterra,No,T,args=(a,d,b,g))
  Nts.append(Nt)
fig, ax = plt.subplots()
for Nt in Nts:
  ax.plot(Nt[:,0],Nt[:,1],color='r',alpha=0.8)

Q = ax.quiver(X,Y,dX, dY, h, pivot='mid', cmap=plt.cm.jet)

ax.set_aspect('equal')
plt.xlabel('$preys\quad (X)$')
plt.ylabel('$predators\quad (Y)$');
```

```
N=np.array([50,1])
Nt=odeint(lotkaVolterra,N,T,args=(a,d,b,g))
plt.plot(T,Nt[:,0])
plt.plot(T,Nt[:,1])
plt.xlabel('time')
plt.ylabel('N')
plt.legend(['Prey','Predator'])

```

**Species average number**

\begin{equation}
    \frac{dX}{dt}=(\alpha -\delta Y(t))X,
\end{equation}

\begin{equation}
    \frac{1}{X}\frac{dX}{dt}=\alpha -\delta Y(t),
\end{equation}

\begin{equation}
    \int_{X(0)}^{X(T)}\frac{dX}{X}=\int_0^T(\alpha -\delta Y(t))dt.
\end{equation}
The left-hand side can be immediately integrated and it gives
\begin{equation}
    \log X(T) - \log X(0)=0,
\end{equation}
which gives zero because the system is periodic and therefore the number of predator at time $t=0$ is equal to the number of predators at time $t=T$.
The integral on the right-hand side gives
\begin{equation}
    \int_0^T(\alpha -\delta Y(t))dt=\alpha T - \delta\int_0^TY(t)dt,
\end{equation}
which cannot be evaluated exactly since we do not know the expression of $Y(t)$. By combining these two results we have
\begin{equation}
    0=\alpha T -\delta\int_0^TY(t)dt,
\end{equation}
which can be written as
\begin{equation}
    \frac{1}{T}\int_0^TY(t)dt=\frac{\alpha}{\delta}.
\end{equation}
The left-hand side is the average number of preys over a period of length $T$. If we consider the second equation, by doing the same calculations we obtain
\begin{equation}
    \frac{1}{T}\int_0^TX(t)dt=\frac{\beta}{\gamma}.
\end{equation}

```
def update(N0X,N0Y,T,alpha,beta,delta,gamma):
    N0=np.array([N0X,N0Y])
    T=np.linspace(0,T,1000)
    sol=odeint(foo,N0,T,args=(alpha,delta,beta,gamma))
    fig,axes=plt.subplots(nrows=1,ncols=2,figsize=(25,10))
    [axes[0].plot(T,sol[:,i]) for i in (0,1)]
    axes[0].grid()
    axes[0].set_title(r"$\overline{X}\quad (prey) = %.3f\quad\overline{Y}\quad(predator) = %.3f$"%(beta/gamma,alpha/delta))
    axes[0].set_xlabel("time",fontsize=20)
    axes[0].set_ylabel("species",fontsize=20)
    axes[0].legend(['prey','predator'],fontsize=20)
    axes[1].plot(sol[:,0],sol[:,1])
    axes[1].set_xlabel("predator",fontsize=20)
    axes[1].set_ylabel("prey",fontsize=20)
    axes[1].grid()
    plt.show()
timeSlider=FloatText(value=7.72,min=1,max=100,step=1,description="T")
N0XSlider=FloatSlider(value=1, min=0.,max=10,step=1,description="N0X")
N0YSlider=FloatSlider(value=1, min=0.,max=10,step=1,description="N0Y")
alphaSlider=FloatSlider(value=2.5, min=0.1,max=10,step=0.1,readout_format='.3f',description=r'<MATH>α</MATH>')
betaSlider=FloatSlider(value=1, min=0.01,max=10,step=0.1,readout_format='.3f',description=r'<MATH>β</MATH>')
deltaSlider=FloatSlider(value=0.07, min=0.01,max=0.1,step=0.01,readout_format='.3f',description=r'<MATH>δ</MATH>')
gammaSlider=FloatSlider(value=0.14, min=0.,max=0.3,step=0.01,description=r'<MATH>γ</MATH>')

foo=lotkaVolterra
interactive_plot = interactive(update, N0X=N0XSlider,N0Y=N0YSlider, T=timeSlider,
                               alpha=alphaSlider, beta=betaSlider,gamma=gammaSlider,delta=deltaSlider)
output = interactive_plot.children[-1]
output.layout.height = '650px'
interactive_plot
```

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    interactive(children=(FloatSlider(value=1.0, description='N0X', max=10.0, step=1.0), FloatSlider(value=1.0, de…

  \begin{cases}
      \frac{dX}{dt}=(\alpha-\rho) X- \delta XY\\
      \frac{dY}{dt}=-(\beta+\psi) Y + \gamma XY
    \end{cases}

```
def lotkaVolterra2(N,t,a,d,b,g,r,p):
	dN0=(a-r)*N[0]-d*N[0]*N[1]
	dN1=g*N[0]*N[1]-(b+p)*N[1]
	return np.array([dN0,dN1])
```

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

```
def update2(N0X,N0Y,T,alpha,beta,delta,gamma,rho,psi):
    N0=np.array([N0X,N0Y])
    T=np.linspace(0,T,1000)
    sol=odeint(foo,N0,T,args=(alpha,delta,beta,gamma,rho,psi))
    fig,axes=plt.subplots(nrows=1,ncols=2,figsize=(25,10))
    [axes[0].plot(T,sol[:,i]) for i in (0,1)]
    axes[0].grid()
    axes[0].set_title(r"$\overline{X}\quad (prey) = %.3f\quad\overline{Y}\quad(predator) = %.3f$"%((beta+psi)/gamma,(alpha-rho)/delta))
    axes[0].set_xlabel("time",fontsize=20)
    axes[0].set_ylabel("species",fontsize=20)
    axes[0].legend(['prey','predator'],fontsize=20)
    axes[1].plot(sol[:,0],sol[:,1])
    axes[1].set_xlabel("predator",fontsize=20)
    axes[1].set_ylabel("prey",fontsize=20)
    axes[1].grid()
    plt.show()
timeSlider=FloatText(value=7.72,min=1,max=100,step=1,description="T")
N0XSlider=FloatSlider(value=1, min=0.,max=10,step=1,description="N0X")
N0YSlider=FloatSlider(value=1, min=0.,max=10,step=1,description="N0Y")
alphaSlider=FloatSlider(value=2.5, min=0.1,max=10,step=0.1,readout_format='.3f',description=r'<MATH>α</MATH>')
betaSlider=FloatSlider(value=1, min=0.01,max=10,step=0.1,readout_format='.3f',description=r'<MATH>β</MATH>')
deltaSlider=FloatSlider(value=0.07, min=0.01,max=0.1,step=0.01,readout_format='.3f',description=r'<MATH>δ</MATH>')
gammaSlider=FloatSlider(value=0.14, min=0.,max=0.3,step=0.01,description=r'<MATH>γ</MATH>')
rhoSlider=FloatSlider(value=0., min=0.,max=10,step=0.01,description=r'<MATH>ρ</MATH>')
psiSlider=FloatSlider(value=0., min=0.,max=10,step=0.01,description=r'<MATH>ψ</MATH>')
foo=lotkaVolterra2
interactive_plot = interactive(update2, N0X=N0XSlider,N0Y=N0YSlider, T=timeSlider,
                               alpha=alphaSlider, beta=betaSlider,gamma=gammaSlider,delta=deltaSlider,rho=rhoSlider,psi=psiSlider)
output = interactive_plot.children[-1]
output.layout.height = '650px'
interactive_plot
```

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    interactive(children=(FloatSlider(value=1.0, description='N0X', max=10.0, step=1.0), FloatSlider(value=1.0, de…

# Predator Prey with logistic growth

$
\frac{dX}{dt}=\alpha X(1-\frac{X}{\omega})-\beta XY\\
$

$
\frac{dY}{dt}=\gamma XY - \delta Y\\
$

```
def lotkaVolterraMod(N,t,a,w,b,d,g):
	dN0=a*N[0]*(1-N[0]/w)-b*N[0]*N[1]
	dN1=g*N[0]*N[1]-d*N[1]
	return np.array([dN0,dN1])

```

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

```
a=2
w=4
d=1
b=2
g=1
No=np.array([3,1])
T=np.linspace(1,100,10000)
print(d<g*w,a*(g*w-d)/(b*g*w),d/g)

```

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    True 0.75 1.0

```
Nt=odeint(lotkaVolterraMod,No,T,args=(a,w,b,d,g))
Nt2=odeint(lotkaVolterraMod,No,T,args=(a-0.5,w,b,d,g))
```

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

```
fig, ax = plt.subplots()
ax.plot(Nt[:,0],Nt[:,1],color='r',alpha=0.8)
#ax.plot(Nt2[:,0],Nt2[:,1],color='b',alpha=0.8)
X=np.linspace(np.min(Nt[:,0]),np.max(Nt[:,0]))
plt.plot(X,a*(w - X)/(b*w))
plt.vlines(d/g,0,np.max(Nt[:,1]))
ax.set_xlabel('preys');
ax.set_ylabel('predators');
ax.grid()

```

```
print(Nt[-1])
fig, ax = plt.subplots()
ax.plot(T,Nt[:,0],color='r')
ax.plot(T,Nt[:,1],color='k')
ax.set_xlabel('time');
ax.set_ylabel('N');
ax.grid()
```

```
ymax=6
xmax=3
n_points=25
x=np.linspace(0,xmax,n_points)
y=np.linspace(0,ymax,n_points)
X,Y=np.meshgrid(x,y)
dX,dY=lotkaVolterraMod([X,Y],0,a,w,d,b,g)

```

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

```

h=hypotenusa(dX,dY)
h[h==0]=1
dX=dX/h
dY=dY/h
fig, ax = plt.subplots()
Q = ax.quiver(X,Y,dX, dY, h, pivot='mid', cmap=plt.cm.jet)
ax.set_xlabel('preys');
ax.set_ylabel('predators');
```

```
Nts=[]
N_ini=np.array([[3,1],[3,2],[3,3],[3,4]])
for No in N_ini:
  Nt=odeint(lotkaVolterraMod,No,T,args=(a,w,d,b,g))
  Nts.append(Nt)
```

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

```
fig, ax = plt.subplots()
for Nt in Nts:
  ax.plot(Nt[:,0],Nt[:,1],color='r',alpha=0.8)

Q = ax.quiver(X,Y,dX, dY, h, pivot='mid', cmap=plt.cm.jet)
ax.set_xlabel('preys');
ax.set_ylabel('predators');
```

```
a=2
w=4
d=1
b=2
g=1
No=np.array([3,1])
T=np.linspace(1,100,10000)
print(d<g*w,a*(g*w-d)/(b*g*w),d/g)

```

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    True 0.75 1.0

```
def update3(N0X,N0Y,T,alpha,omega,beta,delta,gamma):
    N0=np.array([N0X,N0Y])
    T=np.linspace(0,T,1000)
    sol=odeint(foo,N0,T,args=(alpha,omega,delta,beta,gamma))
    fig,axes=plt.subplots(nrows=1,ncols=2,figsize=(25,10))
    SSy=(alpha/delta)*(1-beta/(omega*gamma))
    SSx=beta/gamma
    axes[0].hlines(SSx,0,T[-1],color='k',lw=2,ls=':')
    axes[0].hlines(SSy,0,T[-1],color='r',lw=2,ls=':')

    [axes[0].plot(T,sol[:,i]) for i in (0,1)]
    axes[0].set_xlabel("time",fontsize=20)
    axes[0].set_ylabel("species",fontsize=20)
    axes[0].legend([r'prey',r'predator',r'$SS_{prey}$',r'$SS_{predator}$'],fontsize=20)
    axes[0].grid()

    axes[1].plot(sol[:,0],sol[:,1])
    xlim=axes[1].get_xlim()
    ylim=axes[1].get_ylim()
    axes[1].vlines(SSx,ylim[0],ylim[1],color='k',lw=2,ls=':',label=r'$SS_{prey}$')
    axes[1].hlines(SSy,xlim[0],xlim[1],color='r',lw=2,ls=':',label=r'$SS_{predator}$')
    axes[1].legend()
    axes[1].set_xlabel("predator",fontsize=20)
    axes[1].set_ylabel("prey",fontsize=20)
    axes[1].grid()

    plt.show()

timeSlider=FloatText(value=20,min=1,step=1,description="T")
N0XSlider=FloatSlider(value=1, min=0.,max=10,step=1,description="N0X")
N0YSlider=FloatSlider(value=1, min=0.,max=10,step=1,description="N0Y")
alphaSlider=FloatSlider(value=2, min=0.,max=4,step=0.1,readout_format='.3f',description=r'<MATH>α</MATH>')
omegaSlider=FloatSlider(value=4, min=0.,max=10,step=0.1,readout_format='.3f',description=r'<MATH>ω</MATH>')

betaSlider=FloatSlider(value=1, min=0.01,max=4,step=0.1,readout_format='.3f',description=r'<MATH>β</MATH>')
deltaSlider=FloatSlider(value=1, min=0.01,max=2,step=0.01,readout_format='.3f',description=r'<MATH>δ</MATH>')
gammaSlider=FloatSlider(value=1, min=0.,max=2,step=0.01,description=r'<MATH>γ</MATH>')

#N0XSlider=FloatText(value=1, min=0.,max=10,step=1,description="N0X")
#N0YSlider=FloatText(value=1, min=0.,max=10,step=1,description="N0Y")
##alphaSlider=FloatText(value=2, min=0.,max=4,step=0.1,readout_format='.3f',description=r'<MATH>α</MATH>')
#omegaSlider=FloatText(value=4, min=0.,max=10,step=0.1,readout_format='.3f',description=r'<MATH>ω</MATH>')

#betaSlider=FloatText(value=1, min=0.01,max=4,step=0.1,readout_format='.3f',description=r'<MATH>β</MATH>')
#deltaSlider=FloatText(value=1, min=0.01,max=2,step=0.01,readout_format='.3f',description=r'<MATH>δ</MATH>')
#gammaSlider=FloatText(value=1, min=0.,max=2,step=0.01,description=r'<MATH>γ</MATH>')

foo=lotkaVolterraMod
interactive_plot = interactive(update3, N0X=N0XSlider,N0Y=N0YSlider, T=timeSlider,
                               alpha=alphaSlider,omega=omegaSlider, beta=betaSlider,gamma=gammaSlider,delta=deltaSlider)
output = interactive_plot.children[-1]
output.layout.height = '650px'

interactive_plot
```


# Gaussian Process Regression

A Gaussian Process (GP) is a Gaussian distribution over a family of functions.  GP regression fits a GP to a given set of training data.  The use of Gaussians allow the modeling of similarity (through correlation) between data points.  For example, given two points $(x_{1},f_{1})$, $(x_{2},f_{2})$, if $x_{1}$ and $x_{2}$ are close to one another, then we would expect $f_{1}$ and $f_{2}$ to also be close to one another.  This implies a notion of smootheness of the functions in the GP which is captured by the Radial Basis Function (RBF) kernel.  This kernel is used to construct the covaraince matrix of the GP prior and posterior distributions.

sources: https://github.com/jwangjie/Gaussian-Processes-Regression-Tutorial, https://youtu.be/4vGiHC35j9s

```python
import numpy as np
import numpy.linalg as LA
import matplotlib.pyplot as plt
```

## Radial Basis Function Kernel

The radial basis function kernel is given by:

\begin{equation}
\kappa (x_{i},x_{j}) = \sigma_{f}^{2} \text{ exp} \Big( -\frac{1}{2l}(x_{i}-x_{j})^{T}(x_{i}-x_{j})\Big)
\end{equation}

where $\sigma_{f}$ and $l$ are hyperparameters.

```python
#Define the RBF kernel
#from Nando De Freitas youtube video
def kernel(a, b, l = 1.0, sigma = 1.0):
    sqdist = np.sum(a**2,1).reshape(-1,1) + np.sum(b**2,1) - 2*np.dot(a, b.T)
    return sigma * np.exp(-0.5/l * sqdist)
```

To draw samples from the prior distribution, note that:

\begin{equation}
x_{prior} \sim  \mu + L \text{ } \mathcal{N}(0,I)
\end{equation}

where $LL^{T} = K$, the Cholesky decomposition of $K$ (the kernel of the training data with itself)

```python
#Plot the kernalized prior distribution
n = 50
X_test = np.linspace(-5,5,n).reshape(-1,1)
K_ = kernel(X_test,X_test)

#draw samples
num_samples = 10
L = LA.cholesky(K_ + 1e-6 * np.eye(n))
#note the prior has zero mean
f_prior = np.dot(L, np.random.normal(size=(n,num_samples)))

plt.plot(X_test,f_prior)
plt.title(str(num_samples) + " samples from the GP prior")
plt.show()
```

```python
# This is the true unknown function we are trying to approximate
f = lambda x: np.sin(0.9*x).flatten()
#f = lambda x: (0.25*(x**2)).flatten()
x = np.arange(-5, 5, 0.1)

plt.plot(x, f(x))
plt.axis([-5, 5, -3, 3])
plt.show()
```

The posterior distribution is the conditional probability of a multivariate gaussian formed from the kernelized training data and the kernelized testing data.  Let $(x,f)$ denote the training data and $(f_{*},x_{*})$ denote the testing data.  Define the multivariate gaussian:

$$
\begin{pmatrix}
    f \\
    f_{*}
\end{pmatrix} \sim \begin{pmatrix}
    \begin{pmatrix}
    \mu \\
    \mu_{*}
    \end{pmatrix} ,
    \begin{pmatrix}
        K & K_{*} \\
        K_{*}^{T} & K_{**}
    \end{pmatrix}
\end{pmatrix}
$$

where $K = \kappa(x,x)$, $K_{*} = \kappa(x,x_{*})$, and $K_{**} = \kappa(x_{*},x_{*})$.  From the mean and covariance matrices of the multivariate gaussian, the mean and variance of the conditional gaussian can be obtained from:

$$
    p(f_{*}\vert x_{*}, x, f) = p(f_{*} \vert \mu_{*}, \Sigma_{*})\\
    \mu_{*} = \mu(x_{*}) + K_{*}^{T}K^{-1}(f - \mu(x))\\
    \Sigma_{*} = K_{**} - K_{*}^{T}K^{-1}K_{*}
$$

```python
#Compute the posterior (conditional distribution)

#Noisy function observations (training data)
N = 10     #number of observations
s = 0.00005   #variance of noise
X = np.random.uniform(-5,5,size=(N,1))
y = f(X) + s*np.random.rand(N)

#compute L (square root of covariance matrix)
K = kernel(X, X, l = 1.0, sigma = 1.0)
L = LA.cholesky(K + s*np.eye(N))

#compute testing set to make predictions
n = 200     #number of points to predict
Xtest = np.linspace(-5,5,n).reshape(-1,1)

#compute mean of test set
#note trick to avoid taking matrix inverse
#k_star = kernel(X,Xtest)
#v = l\k_star
Lk = LA.solve(L, kernel(X, Xtest))
#alpha = LA.solve(L,y)
mu_star = np.dot(Lk.T, LA.solve(L,y))

#compute variance of test set
K_star_star = kernel(Xtest,Xtest)
s_ = np.diag(K_star_star) - np.sum(Lk**2,axis=0)
s_star = np.sqrt(s_)

#Plot posterior statistics
plt.figure(1)
plt.clf()
plt.plot(X, y, 'k+', ms=18, label='observation')
plt.plot(Xtest, f(Xtest), 'b-', label='true')
plt.gca().fill_between(Xtest.flat, mu_star-2*s_, mu_star+2*s_, color="#dddddd")
plt.plot(Xtest, mu_star, 'r--', lw=2, label='prediction')
plt.title('Mean predictions plus 2 st.deviations')
plt.legend()
plt.show()
```

```python
#Draw samples from the posterior distribution
L = LA.cholesky(K_star_star + 1e-6 * np.eye(n) - np.dot(Lk.T, Lk))
f_post = mu_star.reshape(-1,1) + np.dot(L, np.random.normal(size=(n,40)))

#plot
plt.figure(2)
plt.clf()
plt.figure(figsize=(18,9))
plt.plot(X, y, 'k+', markersize=20, markeredgewidth=3)
plt.plot(Xtest, f(Xtest), 'b-', label='true', linewidth=3)
plt.plot(Xtest, mu_star, 'r--', linewidth=3)
plt.plot(Xtest, f_post, linewidth=0.8)
plt.title('40 samples from the GP posterior, mean prediction function and observation points')
plt.show()
```


# Finding a Boolean propositional formula from a truth table

```
from sympy import *
init_printing()
```

```
var('x y z')
```

```
P = x & (y | ~z)
P
```

```
P.subs({x: True, y: False, z: True})
```

```
minterms = [[1, 0, 1], [1, 0, 0], [0, 0, 0]]
dontcare = [[1, 1, 1], [1, 1, 0]]
```

```
Q = SOPform(['x', 'y', 'z'], minterms, dontcare)
Q
```

```
Q.subs({x: True, y: False, z: False}), Q.subs(
    {x: False, y: True, z: True})
```


### Analysis of deterministc approximation to the mitochondrial graph

Functionality of this notebook is similar to the accompanying python [script](./src/odes.py).
Please feel free to adjust any of them to your needs.

Time-dependent behaviour of the mitochondria network can be represented as a system of differential-algebraic equations for the number of nodes $x_i$ of degrees $i$ = 1 to 3:
\begin{array}
\dot{x_1} = -a_2x_1x_2 + (3/2)bx_3 - a_1x_1(x_1-1) + 2bx_2 \\
\dot{x_2} = a_2x_1x_2 - (3/2)bx_3 \\
x_1 + 2x_2 +3x_3 = 2h
\end{array}
where $a_1$, $a_2$ and $b$ are ratec onstants for bulk fusion, side fusion and fission respectively.

```python
# uncomment to plot into separate windows
#%matplotlib

import math

import numpy as np
import scipy.integrate

from src.odes import \
    eqs, \
    is_stable, \
    node_numbers_equil, \
    num_deg2, \
    plot_node_numbers, \
    plot_phase_equil, \
    plot_stability, \
    plot_time_evol
```

```python
# Initialize the parameters:

def a(m):
     """Initialize fusion rate constants.
     """
     init = -np.floor(2 * m / 3)
     return np.logspace(init, init + m - 1, num=m, base=2)

m1, m2 = 57, 57          # grid dimensions
b = 1                    # fission rate constant
a1, a2 = a(m1), a(m2)    # fusion rate constant
c1, c2 = a1/b, a2/b      # reduced rates

h = [10000, 30000]       # total number of edges in the graph
```

#### The Steady state

Find the equilibrium solution as $t\rightarrow+\infty$
using reduced parameters $c_1 = a_1/b$, $c_2 = a_2/b$:
\begin{align}
& 0 = c_1c_2x_1^3 + c_1(1 - c_2)x_1^2 + (1 - c_1)x_1 - 2h \\
& x_2 = c_1x_1(x_1 - 1)/2 \\
& x_3 = 2c_2x_1x_2/3
\end{align}

and plot it.

```python
x = [[[node_numbers_equil(cc1, cc2, hh)
       for cc1 in c1]
      for cc2 in c2]
     for hh in h]
```

```python
for xx, hh in zip(x, h):
    plot_node_numbers(c1, c2, hh, xx, figsize=(15, 5))
```

Plot the solution in phase coordinates:

```python
plot_phase_equil(x, c1, h, figsize=(10,10))
```

Examine the solution.
The equilibrium is asymptotically stable if real parts of all eigenvalues of the Jacobian are strictly negative.

```python
st = [is_stable(xx, b, a1, a2, hh) for xx, hh in zip(x, h)]
```

Plot the stability map indicating stable and unstable solutions with blue and red markers respectively:

```python
for s, hh in zip(st, h):
    plot_stability(b, c1, c2, hh, s)
```

#### Ttransient bechavior

Slove the ODEs directly for specific parameters and plot the results:

```python
ht = h[0]                                   # graph total mass (in edges)
bt = b                                      # fission rate constant
a1t = a1[20]                                # end-to-end fusion rate constant
a2t = a2[30]                                # end-to-side fusion rate constant
tspan = [0., 20.]                           # time interval
tsol = np.linspace(tspan[0], tspan[1], 100) # time points for plotting
# initial values:
x1_0 = np.linspace(0, ht, 10)
x3_0 = (ht - x1_0) / 2

x123 = []
for x10, x30 in zip(x1_0, x3_0):
    # new scipy ivp solver: requires scipy >= 1.4:
    sol = scipy.integrate.solve_ivp(
        eqs,
        t_span=tspan,
        y0=[x10, x30],
        args=(b, a1t, a2t, ht),
        dense_output=True
    )
    x13 = sol.sol(tsol)
    x123.append([x13[0,:], num_deg2(x13, ht), x13[1,:]])
```

```python
plot_time_evol(b, a1t, a2t, ht, x123, tsol, figsize=(16, 5))
```

```python

```


```python
import sympy as sm
```

```python
sm.init_printing()
```

```python
theta, alpha = sm.symbols('theta, alpha')
```

```python
B_C_A = sm.Matrix([[sm.cos(theta), sm.sin(theta), 0],
                   [-sm.sin(theta), sm.cos(theta), 0],
                   [0, 0, 1]])
B_C_A
```

```python
C_C_B = sm.Matrix([[1, 0, 0],
                   [0, sm.cos(alpha), sm.sin(alpha)],
                   [0, -sm.sin(alpha), sm.cos(alpha)]])
C_C_B
```

```python
C_C_A = C_C_B * B_C_A
C_C_A
```

# SymPy Mechanics Orientation

```python
import sympy.physics.mechanics as me
```

```python
A = me.ReferenceFrame('A')
B = me.ReferenceFrame('B')
C = me.ReferenceFrame('C')
```

```python
B.orient_axis(A, theta, A.z)
```

```python
A.x
```

```python
A.y
```

```python
A.z
```

```python
B.dcm(A)
```

```python
C.orient_axis(B, alpha, B.x)
```

```python
C.dcm(B)
```

```python
C.dcm(A)
```

# Euler Z-X-Z

```python
A = me.ReferenceFrame('A')
C = me.ReferenceFrame('C')
```

```python
psi, theta, phi = sm.symbols('psi, theta, varphi')
```

```python
C.orient_body_fixed(A, (psi, theta, phi), 'ZXZ')
```

```python
C.dcm(A)
```

```python

```


```python
from epipack import SymbolicEpiModel, EpiModel, StochasticEpiModel
import sympy as sy
import numpy as np
%matplotlib widget
```

```python
%matplotlib widget
```

```python
S, I, R, eta, rho, omega = sy.symbols("S I R eta rho omega")
```

```python
SIRS = SymbolicEpiModel([S,I,R])

SIRS.set_processes([
    #### transmission process ####
    # S + I (eta)-> I + I
    (S, I, eta, I, I),

    #### transition processes ####
    # I (rho)-> R
    # R (omega)-> S
    (I, rho, R),
    (R, omega, S),

])
```

    <epipack.symbolic_epi_models.SymbolicEpiModel at 0x15c684880>

```python
SIRS.ODEs()
```

    [Eq(Derivative(S, t), -I*S*eta + R*omega),
     Eq(Derivative(I, t), I*(S*eta - rho)),
     Eq(Derivative(R, t), I*rho - R*omega)]

```python
SIRS.ODEs_jupyter()
```

$\displaystyle \frac{d}{d t} S = - I S \eta + R \omega$

$\displaystyle \frac{d}{d t} I = I \left(S \eta - \rho\right)$

$\displaystyle \frac{d}{d t} R = I \rho - R \omega$

```python
SIRS.find_fixed_points()
```

$\displaystyle \left\{\left( S, \  0, \  0\right), \left( \frac{\rho}{\eta}, \  \frac{R \omega}{\rho}, \  R\right)\right\}$

```python
SIRS.jacobian()
```

$\displaystyle \left[\begin{matrix}- I \eta & - S \eta & \omega\\I \eta & S \eta - \rho & 0\\0 & \rho & - \omega\end{matrix}\right]$

```python
SIRS.get_eigenvalues_at_disease_free_state()
```

    {eta - rho: 1, -omega: 1, 0: 1}

```python
S, I, R = list("SIR")
R0 = 2.5
rho = recovery_rate = 1 # let's say 1/days
eta = infection_rate = R0 * recovery_rate
omega = 1/14 # in units of 1/days

SIRS = EpiModel([S,I,R])

SIRS.set_processes([
    #### transmission process ####
    # S + I (eta)-> I + I
    (S, I, eta, I, I),

    #### transition processes ####
    # I (rho)-> R
    # R (omega)-> S
    (I, rho, R),
    (R, omega, S),

])

SIRS.set_initial_conditions({S:1-0.01, I:0.01})
```

    <epipack.numeric_epi_models.EpiModel at 0x106990f40>

```python
t = np.linspace(0,40,1000)
result = SIRS.integrate(t)
```

```python
import matplotlib.pyplot as pl
pl.figure()
for compartment, timeseries in result.items():
    pl.plot(t, timeseries, label=compartment)
pl.xlabel(r"time $t\times\rho$")
pl.ylabel("frequency")
pl.ylim([0,1])
pl.legend()
pl.gcf().tight_layout()
```

    Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …

```python
import networkx as nx

k0 = 50
eta = R0 * rho / k0
N = int(1e4)
edges = [ (e[0], e[1], 1.0) for e in nx.fast_gnp_random_graph(N,k0/(N-1)).edges() ]

SIRS = StochasticEpiModel([S,I,R],N,edge_weight_tuples=edges)

SIRS.set_link_transmission_processes([
    #### transmission process ####
    # I + S (eta)-> I + I
    (I, S, eta, I, I),
])

SIRS.set_node_transition_processes([
    #### transition processes ####
    # I (rho)-> R
    # R (omega)-> S
    (I, rho, R),
    (R, omega, S),

])

SIRS.set_random_initial_conditions({S:N-int(1e-2*N), I:int(1e-2*N)})
```

    <epipack.stochastic_epi_models.StochasticEpiModel at 0x15d253be0>

```python
t_s, result_s = SIRS.simulate(40)
```

```python
pl.figure()

for compartment, timeseries in result.items():
    #pl.plot(t, timeseries, label=compartment)
    pl.plot(t_s, result_s[compartment]/N, label=compartment+" (sim.)")
pl.xlabel(r"time $t\times\rho$")
pl.ylabel("frequency")
pl.ylim([0,1])
pl.legend()
pl.gcf().tight_layout()
pl.show()
```

    Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …

```python

```


# Portfolio Optimization
> This model is an example of the classic [Markowitz portfolio selection optimization model](https://en.wikipedia.org/wiki/Markowitz_model). We want to find the fraction of the portfolio to invest among a set of stocks that balances risk and return. It is a Quadratic Programming (QP) model with vector and matrix data for returns and risk, respectively.

- toc: true
- badges: true
- hide_binder_badge: true
- comments: true
- categories: [Investment strategy, Portfolio Optimization, Financial Modelling, Markowitz Portfolio Selection, Risk & Return, Quadratic Programming, Mathematical OPtimization, Gurobi, Python]
- image: images/when-did-you-become-an-expert-in-markowitz-portfolio-optimization-last-night.png

## Problem Statement
The expected return and standard deviation of stock 1 are 4% and 3%, respectively. For stock 2, the expected return and standard deviation are 2% and 2%, respectively. For stock 3, the expected return and standard deviation are 1% and 1.5%, respectively. Suppose the correlation between stocks 1 and 2 is 0.2, between stocks
1 and 3 0.4, and between stocks 2 and 3 0.1.

## Model Formulation
### Sets & Indices
$i ∈ \text{stock} = \{1,2,3\}$: set of stocks
### Parameters
$\mu_i ∈ \mathbb{R}^+$:  expected return of stock $i$

$\sigma_i ∈ \mathbb{R}^+$:  standard deviation of stock $i$

$b ∈ \mathbb{R}^+$:  the minimum required expected return for the portfolio.

### Decision Variables
- $x_i$: n-element vector where each element represents the fraction of the porfolio to invest in each stock

### Objective Function
\begin{equation}
\text{Minimize} \sum_{i=1}^{n} \sum_{j=1}^{n} \mu_i \mu_j \sigma_{ij}  \tag{0}
\end{equation}

### Constraints
- **minimum expected return**

\begin{equation}
\sum_{i=1}^{n} \mu_i x_i \geq b
\tag{1}
\end{equation}

- **total investment**

\begin{equation}
\sum_{i=1}^{n} x_i = 1
\tag{2}
\end{equation}

- **not short-selling**

\begin{equation}
x_i \geq 0 \quad \forall i \in \text{stock}
\tag{3}
\end{equation}

## 1) Minimum return
Find the portfolio with the lowest risk, among those with an expected return of at least 2%

```python
%pip install gurobipy
```

    Collecting gurobipy
      Downloading gurobipy-9.5.1-cp37-cp37m-manylinux2014_x86_64.whl (11.5 MB)
         |████████████████████████████████| 11.5 MB 4.4 MB/s
    Installing collected packages: gurobipy
    Successfully installed gurobipy-9.5.1

```python
import gurobipy as gu
```

```python
# Let us start with defining the vector and matrices that contain the data.

n = 3 #number of stocks
ExpectedReturn = [0.04, 0.02, 0.01] #vector of expected returns with length n
Covariance = [[0.03, 0.00012, 0.00018], [0.00012, 0.02, 0.00003], [0.00018, 0.00003, 0.015]]
#covariance matrix with dimension nxn
MinReturn = 0.02
```

```python
# First, we create the model by function gu.Model(), where “gu” specifies that function Model() is from Gurobi
m = gu.Model("Portfolio Optimization 1")
```

    Restricted license - for non-production use only - expires 2023-10-25

```python
# Let us add the decision variables. Note that "lb" indicates the lower bound of the decision variable
StockWeight = m.addVars(n, name = 'StockWeight', lb = 0.0, vtype=gu.GRB.CONTINUOUS)
```

```python
# Let us add the objective function. Note that “for i in range(n)” indicates the loop from 0 to n-1.
m.setObjective(sum(sum(StockWeight[i]*StockWeight[j]*Covariance[i][j] for i in range(n)) for j in range(n)), gu.GRB.MINIMIZE)
```

```python
# Let us add the constaints.
m.addConstr(sum(StockWeight[i] for i in range(n)) == 1, name = 'SumofWeights')
m.addConstr(sum(StockWeight[i]*ExpectedReturn[i] for i in range(n)) >= MinReturn, name = 'ConstraintonMinReturn')
```

    <gurobi.Constr *Awaiting Model Update*>

```python
m.optimize()
```

    Gurobi Optimizer version 9.5.1 build v9.5.1rc2 (linux64)
    Thread count: 1 physical cores, 2 logical processors, using up to 2 threads
    Optimize a model with 2 rows, 3 columns and 6 nonzeros
    Model fingerprint: 0x390fa4c2
    Model has 6 quadratic objective terms
    Coefficient statistics:
      Matrix range     [1e-02, 1e+00]
      Objective range  [0e+00, 0e+00]
      QObjective range [1e-04, 6e-02]
      Bounds range     [0e+00, 0e+00]
      RHS range        [2e-02, 1e+00]
    Presolve time: 0.02s
    Presolved: 2 rows, 3 columns, 6 nonzeros
    Presolved model has 6 quadratic objective terms
    Ordering time: 0.00s

    Barrier statistics:
     Free vars  : 2
     AA' NZ     : 6.000e+00
     Factor NZ  : 1.000e+01
     Factor Ops : 3.000e+01 (less than 1 second per iteration)
     Threads    : 1

                      Objective                Residual
    Iter       Primal          Dual         Primal    Dual     Compl     Time
       0   3.20152377e+05 -3.20152377e+05  2.50e+03 2.47e-07  1.00e+06     0s
       1   1.15177644e+04 -1.17907196e+04  1.97e+02 1.95e-08  8.42e+04     0s
       2   4.34195814e-02 -3.58271785e+02  6.80e-01 6.72e-11  3.82e+02     0s
       3   6.98591118e-03 -1.98336951e+02  6.80e-07 6.94e-17  4.96e+01     0s
       4   6.98585719e-03 -1.92714187e-01  4.66e-12 0.00e+00  4.99e-02     0s
       5   6.95633171e-03  5.66424294e-03  1.48e-14 0.00e+00  3.23e-04     0s
       6   6.75813420e-03  6.68576945e-03  6.94e-17 6.94e-18  1.81e-05     0s
       7   6.73329099e-03  6.72497295e-03  3.41e-15 2.74e-18  2.08e-06     0s
       8   6.72958274e-03  6.72842709e-03  1.83e-14 1.25e-18  2.89e-07     0s
       9   6.72896822e-03  6.72882136e-03  5.30e-14 6.94e-18  3.67e-08     0s
      10   6.72886123e-03  6.72885109e-03  6.70e-14 2.74e-19  2.54e-09     0s
      11   6.72885183e-03  6.72885180e-03  1.17e-13 6.94e-18  6.72e-12     0s

    Barrier solved model in 11 iterations and 0.05 seconds (0.00 work units)
    Optimal objective 6.72885183e-03

```python
# Let us have a look at the solution.
var = m.getVars()
for i in range(n):
  print(var[i].varName, var[i].x)
```

    StockWeight[0] 0.221851545261853
    StockWeight[1] 0.3344459277157383
    StockWeight[2] 0.4437025270221763

```python
print('Objective function', m.objVal)
```

    Objective function 0.00672885182871444

## 2) What if we cannot invest more than half of our money in stocks 1 and 2?
To answer part 2, we need to add a new constraint involving stocks’ 1 and 2 decision variables.

### Parameters
$a ∈ \mathbb{R}^+$:  the maximum required investment for the stock 1 and 2.

### Constraints
- **Maximum investment**

\begin{equation}
x_1 + x_2 = a
\tag{1}
\end{equation}

```python
# Let us start with defining the vector and matrices that contain the data.
MaxInvest = 0.5
```

```python
# Let us add the constaints.
m.addConstr(StockWeight[0] + StockWeight[1] == MaxInvest, name = 'MaxInvest')
```

    <gurobi.Constr *Awaiting Model Update*>

```python
m.optimize()
```

    Gurobi Optimizer version 9.5.1 build v9.5.1rc2 (linux64)
    Thread count: 1 physical cores, 2 logical processors, using up to 2 threads
    Optimize a model with 3 rows, 3 columns and 8 nonzeros
    Model fingerprint: 0xb98dfd3d
    Model has 6 quadratic objective terms
    Coefficient statistics:
      Matrix range     [1e-02, 1e+00]
      Objective range  [0e+00, 0e+00]
      QObjective range [1e-04, 6e-02]
      Bounds range     [0e+00, 0e+00]
      RHS range        [2e-02, 1e+00]
    Presolve time: 0.02s
    Presolved: 3 rows, 3 columns, 8 nonzeros
    Presolved model has 6 quadratic objective terms
    Ordering time: 0.00s

    Barrier statistics:
     Free vars  : 2
     AA' NZ     : 1.000e+01
     Factor NZ  : 1.500e+01
     Factor Ops : 5.500e+01 (less than 1 second per iteration)
     Threads    : 1

                      Objective                Residual
    Iter       Primal          Dual         Primal    Dual     Compl     Time
       0   3.20072058e+05 -3.20072058e+05  2.50e+03 6.01e-02  1.00e+06     0s
       1   4.83954583e+03 -5.08987963e+03  2.54e+02 6.11e-03  1.05e+05     0s
       2   3.32693448e-02 -3.05887397e+02  5.42e-01 1.30e-05  3.01e+02     0s
       3   6.97658803e-03 -1.15906961e+02  5.42e-07 1.30e-11  2.90e+01     0s
       4   6.97658263e-03 -1.09052744e-01  5.43e-13 1.35e-14  2.90e-02     0s
       5   6.97397122e-03  6.75207038e-03  1.11e-15 1.04e-16  5.55e-05     0s
       6   6.94299555e-03  6.94200376e-03  1.11e-16 6.94e-18  2.48e-07     0s
       7   6.94250050e-03  6.94249950e-03  8.33e-16 6.94e-18  2.48e-10     0s

    Barrier solved model in 7 iterations and 0.05 seconds (0.00 work units)
    Optimal objective 6.94250050e-03

```python
# Let us have a look at the solution.
var = m.getVars()
for i in range(n):
  print(var[i].varName, var[i].x)
```

    StockWeight[0] 0.25000009613332785
    StockWeight[1] 0.24999990386667328
    StockWeight[2] 0.5000000000000022

```python
print('Objective function', m.objVal)
```

    Objective function 0.006942500495087108


# Playing with coupled oscillators - v3

This notebook builds on previous notebooks such as Orbit_games.ipynb to implement the solution of linear coupled differential equations using `solve_ivp` as well as with more analytic methods.  The formalism and notation is based on Taylor, Chapter 11: *Coupled Oscillators and Normal Modes*.

The general form of the equations to solve is compactly expressed in matrix form:

$\begin{align}
  \mathbf{M} \mathbf{\ddot q} = - \mathbf{K} \mathbf{q}
\end{align}$

where $\mathbf{M}$ and $\mathbf{K}$ are $N \times N$ symmetric matrices and $\mathbf{q}$ is a vector of generalized coordinates (this is $\mathbf{x}$ in the beginning discussion in Taylor).

We can solve this directly as differential equations or solve for normal modes.  For the former, we map the problem onto linear differential equations and the vectors $\mathbf{y}$ and $d\mathbf{y}/dt$, which are each of length $2N$.  Here is the $N=2$ mapping:

$\begin{align}
  \mathbf{y} = \left( \begin{array}{c}
                         q_1 \\
                         q_2 \\
                         \dot q_1 \\
                         \dot q_2
                      \end{array}
                       \right)
 \qquad\qquad
\frac{d\mathbf{y}}{dt} = \left( \begin{array}{c}
                         \dot q_1 \\
                         \dot q_2 \\
                         \ddot q_1 \\
                         \ddot q_2
                      \end{array}
                       \right)
\end{align}$

Then for the $\ddot q$ parts we use the matrix solution of our equation:

$\begin{align}
  \mathbf{\ddot q} = - \mathbf{M}^{-1} \mathbf{K} \mathbf{q}
\end{align}$

where $\mathbf{M}^{-1}$ means the matrix inverse.  Check out the implementation below!

To solve for normal modes, we solve the generalized eigenvalue problem

$\begin{align}
  (\mathbf{K} - \omega^2 \mathbf{M}) \mathbf{a} = 0
\end{align}$

using functions from `scipy.linalg` to find the eigenvalues and eigenvectors.  We use these below together with the numerical solutions to make plots of the normal modes.

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.integrate import solve_ivp
from scipy import linalg as LA
```

```python
# Change the common font size
font_size = 14
plt.rcParams.update({'font.size': font_size})
```

```python
class CoupledOscillators:
    """
    Solve the equations for linear coupled oscillators in matrix formulation.
    """

    def __init__(self, M_matrix, K_matrix, b_matrix=np.zeros(1)):
        self.M_matrix = M_matrix
        self.K_matrix = K_matrix
        self.q_len = len(self.M_matrix)
        if b_matrix.any():
            self.b_matrix = b_matrix
        else:
            self.b_matrix = np.zeros((self.q_len))
        self.full_matrix = np.zeros((self.q_len, self.q_len))

    def dy_dt(self, t, y):
        """
        This function returns the right-hand side of the diffeq:
        [dq_vec/dt d^2q_vec/dt^2]

        Parameters
        ----------
        t : float
            time
        y : float
            vector with y[:q_len] = q_vec, y[q_len:] = dqvec/dt

        """
        dy_dt_vec = np.zeros(2*self.q_len)   # Initialize to twice q_vec size
        # Set the upper half [:self.q_len] of dy_dt to the lower half of y
        dy_dt_vec[:self.q_len] = y[self.q_len:]
        # Set the lower half [self.q_len:] of dy_dt to the vector that
        #  results from M^{-1} K y where @ is matrix multiplication
        dy_dt_vec[self.q_len:] = -1.* LA.inv(self.M_matrix) @ \
                    ( self.K_matrix @ y[:self.q_len] + \
                      self.b_matrix @ y[self.q_len:] )
        return dy_dt_vec

    def solve_ode(self, t_pts, q_vec_0, q_vec_dot_0,
                  method='RK45',
                  abserr=1.0e-10, relerr=1.0e-10):
        """
        Solve the ODE given initial conditions.
        Use solve_ivp with the option of specifying the method.
        Specify smaller abserr and relerr to get more precision.
        """
        y = np.concatenate((q_vec_0, q_vec_dot_0))
        solution = solve_ivp(self.dy_dt, (t_pts[0], t_pts[-1]),
                             y, t_eval=t_pts, method=method,
                             atol=abserr, rtol=relerr)
        q_vec, q_vec_dot = np.array_split(solution.y, 2)
        return q_vec, q_vec_dot

    def find_eigenmodes(self):
        """
        Find the normal modes.
        """
        eig_vals, eig_vecs = LA.eigh(K_matrix, M_matrix)
        self.frequencies = np.sqrt(eig_vals)
        return eig_vals, eig_vecs

    def plot_eigenmodes(self, t_pts, output_file=None):
        """
        plot the normal modes separately
        """
        from itertools import cycle
        colors = ['blue', 'green', 'red', 'purple']

        eig_vals, eig_vecs = self.find_eigenmodes()
        for i in np.arange(self.q_len):
            q_vec_0 = eig_vecs[:,i]   # this is the i'th eigenvector
            q_vec_dot_0 = np.zeros(self.q_len)  # start with all qdots = 0
            q_vec, q_vec_dot = self.solve_ode(t_pts, q_vec_0, q_vec_dot_0)

            fig, axes = plt.subplots(self.q_len, 1,
                                     figsize=(10, 2.*self.q_len))
            color_list = cycle(colors)
            for j in np.arange(self.q_len):
               axes[j].plot(t_pts, q_vec[j], color=next(color_list))
               axes[j].set_xlabel(r'$t$')
               axes[j].set_ylabel(fr'$q_{j+1:d}$')
               axes[j].axvline(2.*np.pi/np.sqrt(eig_vals[i]),
                               color='black', linestyle=':')
            overall_title = fr'Normal mode {i+1:d}, ' + \
                            fr'$\omega = {self.frequencies[i]:.2f}$'
            fig.suptitle(overall_title, va='center')
            fig.tight_layout()
            if output_file:
                 output_file_name = output_file.replace('.', f'_{i:d}.', 1)
                 fig.savefig(output_file_name, dpi=200, bbox_inches='tight')

```

```python
def start_stop_indices(t_pts, plot_start, plot_stop):
    start_index = (np.fabs(t_pts-plot_start)).argmin()  # index in t_pts array
    stop_index = (np.fabs(t_pts-plot_stop)).argmin()  # index in t_pts array
    return start_index, stop_index
```

## Section 11.2

```python
m = 1.
M_matrix = np.array([ [m, 0.], [0., m] ])
print('M matrix: \n', M_matrix, '\n')

k = 20.
k2 = 1.
K_matrix = np.array([ [k + k2, -k2,], [-k2, k2]])
print('K matrix: \n', K_matrix, '\n')

# b = 0
# b_matrix = np.array([ [b, 0], [0, b]])
# print('b matrix: \n', b_matrix, '\n')

co1 = CoupledOscillators(M_matrix, K_matrix)

eig_vals, eig_vecs = co1.find_eigenmodes() # LA.eigh(K_matrix, M_matrix)

print(f'omega^2 = {eig_vals}')
print(f'frequencies = {co1.frequencies}')
```

    M matrix:
     [[1. 0.]
     [0. 1.]]

    K matrix:
     [[21. -1.]
     [-1.  1.]]

    omega^2 = [ 0.95012438 21.04987562]
    frequencies = [0.97474324 4.58801434]

```python
print(eig_vecs[:,0])
print(eig_vecs[:,1])
```

    [-0.0498137  -0.99875853]
    [-0.99875853  0.0498137 ]

```python
t_start = 0.
t_end = 10.
delta_t = 0.001

t_pts = np.arange(t_start, t_end+delta_t, delta_t)
```

```python
co1.plot_eigenmodes(t_pts, 'Section11p2.png')
```

### Figure 11.6

This has $A_1 = 1$, $A_2 = 0.7$, $\delta_1 = 0$ and $\delta_2 = \pi/2$.

```python
t_start = 0.
t_end = 50.
delta_t = 0.001

t_pts = np.arange(t_start, t_end+delta_t, delta_t)
```

```python
colors = ['blue', 'green', 'red', 'purple']

# changing A_i -> -A_i to match convention in Figure 11.6
A_1 = -1.
delta_1 = 0.
A_2 = -0.7
delta_2 = 0.

omega_1 = co1.frequencies[0]
omega_2 = co1.frequencies[1]

q_vec_0 = A_1 * eig_vecs[:,0] * np.cos(-delta_1) + \
          A_2 * eig_vecs[:,1] * np.cos(-delta_2)
q_vec_dot_0 = omega_1 * A_1 * eig_vecs[:,0] * np.sin(delta_1) + \
              omega_2 * A_2 * eig_vecs[:,1] * np.sin(delta_2)
q_vec_11p6, q_vec_dot_11p6 = co1.solve_ode(t_pts, q_vec_0, q_vec_dot_0)

fig, axes = plt.subplots(co1.q_len, 1, figsize=(10, 2.*co1.q_len))
for j in np.arange(co1.q_len):
    axes[j].plot(t_pts, q_vec_11p6[j], color=colors[j])
    axes[j].set_xlabel(r'$t$')
    axes[j].set_ylabel(fr'$q_{j+1:d}$')
    axes[j].set_xlim(t_pts[0], t_pts[-1])
    axes[j].axhline(0., color='black', alpha=0.3)
overall_title = 'General solution: ' + fr'$A_1 = {A_1:.2f}$, ' + \
                                       fr'$\delta_1 = {delta_1:.2f}$, ' + \
                                       fr'$A_2 = {A_2:.2f}$, ' + \
                                       fr'$\delta_2 = {delta_2:.2f}$'
fig.suptitle(overall_title, va='center')
fig.tight_layout()

print( q_vec_0, q_vec_dot_0 )  # debugging check
```

## Problem 11.5

Almost the same as section 11.2, but mass m2 is not attached by a spring to the wall.

```python
m = 1.
M_matrix = np.array([ [m, 0.], [0., m] ])
print('M matrix: \n', M_matrix, '\n')

k = 1.
K_matrix = np.array([ [2.*k, -k], [-k, k]])
print('K matrix: \n', K_matrix, '\n')

# b = 0
# b_matrix = np.array([ [b, 0], [0, b]])
# print('b matrix: \n', b_matrix, '\n')

co1 = CoupledOscillators(M_matrix, K_matrix)

eig_vals, eig_vecs = co1.find_eigenmodes() # LA.eigh(K_matrix, M_matrix)

print(f'omega^2 = {eig_vals}')
print(f'frequencies = {co1.frequencies}')
```

    M matrix:
     [[1. 0.]
     [0. 1.]]

    K matrix:
     [[ 2. -1.]
     [-1.  1.]]

    omega^2 = [0.38196601 2.61803399]
    frequencies = [0.61803399 1.61803399]

```python
print(eig_vecs[:,0])
print(eig_vecs[:,1])
```

    [-0.52573111 -0.85065081]
    [-0.85065081  0.52573111]

```python
t_start = 0.
t_end = 10.
delta_t = 0.001

t_pts = np.arange(t_start, t_end+delta_t, delta_t)
co1.plot_eigenmodes(t_pts)
```

## Problem 11.7

Writing the general solution to the cart problem in Section 11.2 in a different form and plot two solutions.

```python
m = 1.
M_matrix = np.array([ [m, 0.], [0., m] ])
print('M matrix: \n', M_matrix, '\n')

k = 1.
K_matrix = np.array([ [2.*k, -k], [-k, 2.*k]])
print('K matrix: \n', K_matrix, '\n')

# b = 0
# b_matrix = np.array([ [b, 0], [0, b]])
# print('b matrix: \n', b_matrix, '\n')

co1 = CoupledOscillators(M_matrix, K_matrix)

eig_vals, eig_vecs = co1.find_eigenmodes() # LA.eigh(K_matrix, M_matrix)

print(f'omega^2 = {eig_vals}')
print(f'frequencies = {co1.frequencies}')
```

    M matrix:
     [[1. 0.]
     [0. 1.]]

    K matrix:
     [[ 2. -1.]
     [-1.  2.]]

    omega^2 = [1. 3.]
    frequencies = [1.         1.73205081]

```python
t_start = 0.
t_end = 30.
delta_t = 0.001

t_pts = np.arange(t_start, t_end+delta_t, delta_t)
```

```python
colors = ['blue', 'green', 'red', 'purple']

# changing A -> -A to match convention in Taylor
A = -1.
B_1 = A
B_2 = 0.
C_1 = 0.
C_2 = 0.

omega_1 = co1.frequencies[0]
omega_2 = co1.frequencies[1]

q_vec_0 = B_1 * eig_vecs[:,0] + \
          B_2 * eig_vecs[:,1]
q_vec_dot_0 = omega_1 * C_1 * eig_vecs[:,0] + \
              omega_2 * C_2 * eig_vecs[:,1]
q_vec_11p7a, q_vec_dot_11p7a = co1.solve_ode(t_pts, q_vec_0, q_vec_dot_0)

fig, axes = plt.subplots(co1.q_len, 1, figsize=(10, 2.*co1.q_len))
for j in np.arange(co1.q_len):
    axes[j].plot(t_pts, q_vec_11p7a[j], color=colors[j])
    axes[j].set_xlabel(r'$t$')
    axes[j].set_ylabel(fr'$q_{j+1:d}$')
    axes[j].set_xlim(t_pts[0], t_pts[-1])
    axes[j].axhline(0., color='black', alpha=0.3)
overall_title = 'Specific solution 1: ' + fr'$B_1 = {B_1:.2f}$, ' + \
                                          fr'$C_1 = {C_1:.2f}$, ' + \
                                          fr'$B_2 = {B_2:.2f}$, ' + \
                                          fr'$C_2 = {C_2:.2f}$'
fig.suptitle(overall_title, va='center')
fig.tight_layout()

print( q_vec_0, q_vec_dot_0 )  # debugging check
```

```python
colors = ['blue', 'green', 'red', 'purple']

# changing A -> -A to match convention in Taylor
A = -1.
B_1 = A / 2.
B_2 = A / 2.
C_1 = 0.
C_2 = 0.

omega_1 = co1.frequencies[0]
omega_2 = co1.frequencies[1]

q_vec_0 = B_1 * eig_vecs[:,0] + \
          B_2 * eig_vecs[:,1]
q_vec_dot_0 = omega_1 * C_1 * eig_vecs[:,0] + \
              omega_2 * C_2 * eig_vecs[:,1]
q_vec_11p7a, q_vec_dot_11p7a = co1.solve_ode(t_pts, q_vec_0, q_vec_dot_0)

fig, axes = plt.subplots(co1.q_len, 1, figsize=(10, 2.*co1.q_len))
for j in np.arange(co1.q_len):
    axes[j].plot(t_pts, q_vec_11p7a[j], color=colors[j])
    axes[j].set_xlabel(r'$t$')
    axes[j].set_ylabel(fr'$q_{j+1:d}$')
    axes[j].set_xlim(t_pts[0], t_pts[-1])
    axes[j].axhline(0., color='black', alpha=0.3)
overall_title = 'Specific solution 1: ' + fr'$B_1 = {B_1:.2f}$, ' + \
                                          fr'$C_1 = {C_1:.2f}$, ' + \
                                          fr'$B_2 = {B_2:.2f}$, ' + \
                                          fr'$C_2 = {C_2:.2f}$'
fig.suptitle(overall_title, va='center')
fig.tight_layout()

print( q_vec_0, q_vec_dot_0 )  # debugging check
```

## Double pendulum with small oscillations

```python
m = 1.
L = 1.
M_matrix = m * L**2 * np.array([ [2., 1.], [1., 1.] ])
print('M matrix: \n', M_matrix, '\n')

k = 1.
omega_0 = 1.
K_matrix = m * L**2 * np.array([ [2.*omega_0**2, 0.], [0., omega_0**2]])
print('K matrix: \n', K_matrix, '\n')

# b = 0
# b_matrix = np.array([ [b, 0], [0, b]])
# print('b matrix: \n', b_matrix, '\n')

double_pendulum = CoupledOscillators(M_matrix, K_matrix)

eig_vals, eig_vecs = double_pendulum.find_eigenmodes()

print(f'omega^2 = {eig_vals}')
print(f'frequencies = {double_pendulum.frequencies}')
print(f'{2. + np.sqrt(2.):.4f}  {2. - np.sqrt(2.):.4f}')
```

    M matrix:
     [[2. 1.]
     [1. 1.]]

    K matrix:
     [[2. 0.]
     [0. 1.]]

    omega^2 = [0.58578644 3.41421356]
    frequencies = [0.76536686 1.84775907]
    3.4142  0.5858

```python
print(eig_vecs[:,0])
print(eig_vecs[:,1])
```

    [-0.38268343 -0.5411961 ]
    [-0.92387953  1.30656296]

```python
t_start = 0.
t_end = 10.
delta_t = 0.001

t_pts = np.arange(t_start, t_end+delta_t, delta_t)
double_pendulum.plot_eigenmodes(t_pts)
```

```python

```


```python
from sympy import init_session
init_session()
```

    IPython console for SymPy 1.6 (Python 3.7.3-64-bit) (ground types: python)

    These commands were executed:
    >>> from __future__ import division
    >>> from sympy import *
    >>> x, y, z, t = symbols('x y z t')
    >>> k, m, n = symbols('k m n', integer=True)
    >>> f, g, h = symbols('f g h', cls=Function)
    >>> init_printing()

    Documentation can be found at https://docs.sympy.org/1.6/

```python
# this is for rac[4,2]
a, b, d, e, k, l0 = symbols('a b d e k l0', real=True, positive=True)
```

```python
A, B, D, E = a**2, b**2, d**2, e**2
TA = 2*A
A2B = A*A + B

f1 = k**2 + TA*k + A2B
f2 = 1 + D*k
den = A2B + k*(TA + D*(A2B)) + E*k**2
f= l0 * f1 * f2 /den
f.factor()
```

```python
diff(f,l0)
```

```python
da=diff(f,a)
da.factor()
```

```python
db=diff(f,b)
db.factor()
```

```python
dd=diff(f,d)
dd.factor()
```

```python
de=diff(f,e)
de.factor()
```

```python

```