# Solving Differential Equations of Systems of Interactions

Differential equations are central in the mathematical treatment of the dynamics of biological systems. Now that we know how to write the set of differential equations that govern a certain system of interacting species (using differential form of Mass Action Law and the Mass Conservation), we will proceed now to integrate them and obtain an analytical solution and extract relevant information.

##  1. Irreversible First Order Reaction

 We will start with the simplest example of one species  $X_1$ transforming into another $X_2$ with a kinetic constant $k_1$. The scheme of interactions is simply:

$$ X_1 \overset{k_1}{\longrightarrow} X_2  \tag{1} $$

with initial concentrations $X_1(0)$ and $X_2(0)$ and stoichometric matrices

$$
A=\begin{bmatrix}
 1& 0\end{bmatrix} ;
B=\begin{bmatrix}
0 & 1
\end{bmatrix} ; (B-A)^T= \begin{bmatrix}
 - 1\\ 1 \end{bmatrix} \tag{2}
$$
 in this particular case

$$
K=\begin{pmatrix}
 k\\
\end{pmatrix}
$$

and

$$X^A=\begin{pmatrix}
X_1^1\cdot X_2^0
\end{pmatrix}
$$

so

$$
\begin{align}
 \begin{bmatrix}
\frac{\mathrm{d} X_1}{\mathrm{d} t}\\ \frac{\mathrm{d} X_2}{\mathrm{d} t} \end{bmatrix}= \begin{bmatrix}
 - 1\\ 1 \end{bmatrix} \begin{bmatrix}
k \end{bmatrix}\begin{pmatrix}
X_1^1\cdot X_2^0
\end{pmatrix} = \begin{bmatrix}
 - 1\\ 1 \end{bmatrix} \begin{pmatrix}
k \cdot X_1
\end{pmatrix} = \begin{bmatrix}
 - k \cdot X_1\\ k \cdot X_1 \end{bmatrix}
\end{align}
$$

The set of ODEs that define the system is

$$
\begin{align*}
   \frac{\mathrm{d} X_1}{\mathrm{d} t} &=  - k_1 X_1 \tag{3}\\
   \frac{\mathrm{d} X_2}{\mathrm{d} t} &=   k_1 X_1  \tag{4}\\
\end{align*}
$$

It is called a first order reaction because the rate is proportional to the first power of the concentration of one reactant. Taking eq. 2, we reorganize variables and integrate.

$$\begin{align*}
   \frac{\mathrm{d} X_1}{X_1} &=  - k_1 \mathrm{d}t \tag{5}\\
   \int  \frac{\mathrm{d} X_1}{X_1} &=  \int  - k_1 \mathrm{d}t \tag{6}\\
    \log(X_1(t)) &=  - k_1 t + C  \tag{7}\\
    X_1 (t)&=  e^{- k_1 t + C}  \tag{8}\\
    X_1 (t)&=  e^{- k_1 t}e^{C}  \tag{9}\\
\end{align*}$$

To calculate the integration constant, we use the intial condition at $t=0$.

$$\begin{align*}
    X_1 (0)&=  e^{- k_1 0} e^{C}  \tag{10}\\
    X_1 (0)&=   e^{C}  \tag{11}\\
\end{align*}$$
so the final differential equation for $X_1$ takes the form:

$$ X_1 (t)= X_1 (0) e^{- k_1 t}  \tag{12}$$

Let's use now the conservation of mass to obtain the equation for the reactant $X_2$. Based on this law, if $C \cdot (B-A)^T =0$, then $\sum_{i=1}^{r} C_i \cdot X_i = cte $, so for the case of two variables and one single irreversible reaction

$$ \begin{align*}
C_1 \cdot X_1 + C_2 \cdot X_2 &= cte \tag{13}\\
\end{align*}$$

so
$$ \begin{align*}
C \cdot (B-A)^T &=0\\
\begin{pmatrix}C_1 & C_2\end{pmatrix} \begin{bmatrix}
 - 1\\ 1 \end{bmatrix}&=0\\
 -C_1 + C_2 &= 0\\
 C_1 &= C_2
\end{align*}$$
Therefore,

$$ \begin{align*}
C_1 \cdot X_1(t) + C_1 \cdot X_2(t) &= cte\\
\end{align*}$$

for any value of $C_1$, so taking $C_1=1$, we have

$$X_1(t)+X_2(t)=cte $$.

This constant value has to be true also at $t=0$,

$$X_1(t)+X_2(t)= X_1(0)+X_2(0) = cte $$.

using this conservation of mass, we can also obtain the analytical solution for $X_2$:

$$ \begin{align*}
X_2 (t) &= X_1 (0) + X_2 (0) - X_1 (t) \\
X_2 (t) &= X_1 (0) + X_2 (0) - X_1 (0) e^{- k_1 t}  \\
X_2 (t) &=  X_2 (0) + X_1 (0) (1-e^{- k_1 t})  \tag{14}
\end{align*}$$

### Exercise:
Use the solution for Irreversible First Order Reaction, write a computer program that consists on a function that takes as input the time of the reaction, a vector of initial concentrations of the species $X_i$ and the reaction rate $k$ and plots the time evolution of the variables $X_i(t)$.

```julia
using Plots
gr()
```

    Plots.GRBackend()

```julia
t=collect(0:0.1:10)
X₀ = [0.1,0.0]
k_1 = 1.;
```

```julia
plot(t,t->X₀[1]*exp(-k_1*t),label="\\ X_1",seriestype=:line)
plot!(t,t->X₀[2]+X₀[1]*(1-exp(-k_1*t)),label="\\ X_2",seriestype=:line)
title!("First order reaction")
xaxis!("Time")
yaxis!("Concentration")
```

## 2. Reversible First Order Reaction

Another simple example that can be solved analytically is the reversible first order reaction :

$ X_1 \overset{k_1}{\underset{k_2}{\longleftrightarrow}} X_2  \tag{15} $

$$
A=\begin{bmatrix}
 1 & 0\\
 0 & 1 \end{bmatrix} ;
B=\begin{bmatrix}
0 & 1\\
 1 & 0
\end{bmatrix} ; (B-A)^T= \begin{bmatrix}
-1 & 1\\
 1 & -1   \end{bmatrix}
$$

 in this particular case

$$
K=\begin{pmatrix}
 k_1 & 0\\
 0 & k_2
\end{pmatrix}
$$

$$X^A=\begin{pmatrix}
X_1^1\cdot X_2^0\\
X_1^0\cdot X_2^1
\end{pmatrix} = \begin{pmatrix}
X_1\\
 X_2
\end{pmatrix}
$$

so

$$
\begin{align}
 \begin{bmatrix}
\frac{\mathrm{d} X_1}{\mathrm{d} t}\\ \frac{\mathrm{d} X_2}{\mathrm{d} t} \end{bmatrix}= \begin{bmatrix}
-1 & 1\\
 1 & -1   \end{bmatrix} \begin{pmatrix}
 k_1 & 0\\
 0 & k_2
\end{pmatrix}\begin{pmatrix}
X_1\\
 X_2
\end{pmatrix}  =  \begin{bmatrix}
-1 & 1\\
 1 & -1   \end{bmatrix}\begin{pmatrix}
k_1 \cdot X_1\\
  k_2 \cdot X_2
\end{pmatrix} =   \begin{bmatrix}
- k_1 \cdot X_1 +  k_2 \cdot X_2\\
 k_1 \cdot X_1 -k_2 \cdot X_2   \end{bmatrix}
\end{align}
$$

The differential equations of the system can be written using the differential form of Mass Action Law as:

$$\begin{align}
\frac{dX_1}{dt} = -k_1 \cdot X_1 +k_2 \cdot X_2 \tag{17}\\
\frac{dX_2}{dt} = k_1 \cdot X_1 - k_2 \cdot X_2 \tag{18}\\
\end{align}$$

The mass balance law is equivalent to the previous case (only $X_1$ and $X_2$ as reactants). We start by using mass balance to de-couple the two equations. For the first one we have:

$$\begin{align}
\frac{dX_1}{dt} = k_2\left( X_1(0)+X_2(0)-X_1\right)-k_1X_1 \tag{18}\\
\end{align}$$

which, when we rearrange terms becomes:
$$\frac{dX_1}{dt}+\left(k_2+k_1\right)X_1=k_2\left( X_1(0)+X_2(0)\right) \tag{19}$$

To solve this diferential equation, we need to calculate the integration factor, $e^{ \int p(x)dx }$, that allows us to put the first term of the equation as one diferentiation. In this case $p(x)=k_2+k1 $, so $e^{ \int (k_2+k1)dt }=e^{(k_2+k1)t}$. This way, when we multiply both terms in eq. 18 by the integrating factor, we obtain:

$$\frac{dX_1}{dt} e^{(k_2+k1)t} +\left(k_2+k_1\right) e^{(k_2+k1)t} X_1 = \left( X_1(0)+X_2(0)\right) k_2 e^{(k_2+k1)t} \tag{20}$$

The first term now is the derivative of $X_1 \cdot  e^{(k_2+k1)t}$ so we can reqwrite:

$$\frac{d(X_1 e^{(k_2+k1)t})}{dt} = \left( X_1(0)+X_2(0)\right) k_2 e^{(k_2+k1)t} \tag{21}$$

we move the `dt` to the right side we integrate

$$ \int d(X_1 e^{(k_2+k1)t}) = \left( X_1(0)+X_2(0)\right) k_2 \int e^{(k_2+k1)t} dt \tag{22}$$

solving

$$ X_1 e^{(k_2+k1)t} = \frac{\left( X_1(0)+X_2(0)\right) k_2}{k_2+k1}  e^{(k_2+k1)t} + C \tag{23}$$

rearranging terms becomes

$$ X_1 (t) = \frac{\left( X_1(0)+X_2(0)\right) k_2}{k_2+k1} + C \cdot e^{-(k_2+k1)t} \tag{24}$$

to calculate the integration constant we use initial value of $X_1$

$$
\begin{align}
X_1 (0) &= \frac{\left( X_1(0)+X_2(0)\right) k_2}{k_2+k1} + C \cdot e^{-(k_2+k1)0} \tag{25}\\
X_1 (0) &=\frac{\left( X_1(0)+X_2(0)\right) k_2}{k_2+k1} + C \tag{26}\\
C&= X_1 (0) - \frac{\left( X_1(0)+X_2(0)\right) k_2}{k_2+k1} \tag{27}\\
C&= \frac{X_1 (0)(k_2 + k 1) - X_1(0) k_2 - X_2(0) k_2 }{k_2+k1}\tag{28} \\
C&= \frac{X_1 (0) k 1 - X_2(0) k_2 }{k_2+k1}\tag{29} \\
\end{align}$$

Therefore the full solutions for $X_1$ is:
$$\label{eq:aeq} X_1(t)=\frac{k_2\left ( X_1(0)+X_2(0) \right )}{k_1+k_2}+\frac{k_1 X_1(0)-k_2X_2(0) }{k_1+k_2}e^{-\left (k_1+k_2)\right )t} \tag{30}$$

In this case, the conservation of mass is exactly the same as the previous case of irreversible reaction, so:
$$X_1(t)+X_2(t)=X_1(0)+X_2(0) \tag{16} $$

and, using the conservation of mass, the solution for $X_2$ is
$$
\begin{align}
X_2(t)&=X_1(0)+X_2(0)-\frac{k_2\left ( X_1(0)+X_2(0) \right )}{k_1+k_2}-\frac{k_1 X_1(0)-k_2X_2(0) }{k_1+k_2}e^{-\left (k_1+k_2)\right )t} \tag{31}\\
X_2(t)&=\frac{(k_1+k_2)(X_1(0)+X_2(0))} {k_1+k_2}-\frac{k_2\left ( X_1(0)+X_2(0) \right )}{k_1+k_2}-\frac{k_1 X_1(0)-k_2X_2(0) }{k_1+k_2}e^{-\left (k_1+k_2)\right )t} \tag{32}\\
X_2(t)&=\frac{k_1 (X_1(0)+X_2(0))} {k_1+k_2}+ \frac{k_2 (X_1(0)+X_2(0))}{k_1+k_2}-\frac{k_2\left ( X_1(0)+X_2(0) \right )}{k_1+k_2}-\frac{k_1 X_1(0)-k_2X_2(0) }{k_1+k_2}e^{-\left (k_1+k_2)\right )t} \tag{33}
\end{align}$$

so the final set of equations is:
$$
\begin{align}
X_1(t)&=\frac{k_2\left ( X_1(0)+X_2(0) \right )}{k_1+k_2}+\frac{k_1 X_1(0)-k_2X_2(0) }{k_1+k_2}e^{-\left (k_1+k_2)\right )t} \tag{34}\\
X_2(t)&=\frac{k_1 (X_1(0)+X_2(0))} {k_1+k_2}-\frac{k_1 X_1(0)-k_2X_2(0) }{k_1+k_2}e^{-\left (k_1+k_2)\right )t} \tag{35}
\end{align}$$

where the equilibrium concentrations for $X_1$ and $X_2$ can be calculated using limits:

$$
\begin{align}X_1(t->inf)\frac{k_2\left ( X_1(0)+X_2(0) \right )}{k_1+k_2} \tag{36}\\
X_2(t->inf)\frac{k_1\left ( X_1(0)+X_2(0) \right )}{k_1+k_2} \tag{37}
\end{align}$$

### Exercise:
Use the solution for Reversible First Order Reaction, write a computer program that consists on a function that takes as input the time of the reaction, a vector of initial concentrations of the species $X_i$ and the reaction rate $k$ and plots the time evolution of the variables $X_i(t)$.

```julia
k_1 = 1
k_2 = 0.5
X₀ = [0.5,0.2] #Concentraciones iniciales
```

    2-element Array{Float64,1}:
     0.5
     0.2

```julia
plot(t,t->((k_2*(X₀[1]+X₀[2]))/(k_1+k_2)) + ((k_1*X₀[1]-k_2*X₀[2])/(k_1+k_2))*exp(-t*(k_1+k_2)),label="\\ X_1",seriestype=:line)
plot!(t,t->X₀[1] + X₀[2] - ((k_2*(X₀[1]+X₀[2]))/(k_1+k_2)) - ((k_1*X₀[1]-k_2*X₀[2])/(k_1+k_2))*exp(-t*(k_1+k_2)),label="\\ X_2",seriestype=:line)
title!("Reversible first order")
xaxis!("Time")
yaxis!("Concentration")
```

## 3. Sequence of First Order Reactions
The next case that we can solve is the squence of first order reactions, such as:

$$X_1\overset{k_1}{\longrightarrow}X_2\overset{k_2}{\longrightarrow}X_3 \tag{38}$$

$$
A=\begin{bmatrix}
 1 & 0 & 0\\
 0 & 1 & 0\end{bmatrix} ;
B=\begin{bmatrix}
0 & 1 & 0\\
0 & 0 & 1
\end{bmatrix} ; (B-A)^T= \begin{bmatrix}
-1 & 0\\
 1 & -1 \\
 0 & 1 \end{bmatrix}
$$

 in this particular case

$$
K=\begin{pmatrix}
 k_1 & 0\\
 0 & k_2
\end{pmatrix}
$$

$$X^A=\begin{pmatrix}
X_1^1 \cdot X_2^0  \cdot X_3^0 \\
X_1^0\cdot X_2^1 \cdot X_3^0
\end{pmatrix} = \begin{pmatrix}
X_1\\
 X_2
\end{pmatrix}
$$

so, using the general formulation of mass action,

$$
\begin{align}
 \begin{bmatrix}
\frac{\mathrm{d} X_1}{\mathrm{d} t}\\ \frac{\mathrm{d} X_2}{\mathrm{d} t}\\ \frac{\mathrm{d} X_3}{\mathrm{d} t} \end{bmatrix}=(B-A)^T= \begin{bmatrix}
-1 & 0\\
 1 & -1 \\
 0 & 1 \end{bmatrix} \begin{pmatrix}
 k_1 & 0\\
 0 & k_2
\end{pmatrix}\begin{pmatrix}
X_1\\
 X_2
\end{pmatrix}  =  (B-A)^T= \begin{bmatrix}
-1 & 0\\
 1 & -1 \\
 0 & 1 \end{bmatrix}\begin{pmatrix}
k_1 \cdot X_1\\
  k_2 \cdot X_2
\end{pmatrix} =   \begin{bmatrix}
- k_1 \cdot X_1 \\
k_1 \cdot X_1- k_2 \cdot X_2 \\
 k_2 \cdot X_2  \end{bmatrix}
\end{align}
$$

we will use the law of conservation of mass:

$$ \begin{align*}
C_1 \cdot X_1 + C_2 \cdot X_2 + C_3 \cdot X_3& = cte \\
\end{align*}$$

so
$$ \begin{align*}
C \cdot (B-A)^T &=0\\
\begin{pmatrix}C_1 & C_2 & C_3\end{pmatrix} \begin{bmatrix}
-1 & 0\\
 1 & -1 \\
 0 & 1 \end{bmatrix}&=0\\
 -C_1 + C_2 = 0 &\implies  C_1 = C_2\\
 -C_2 + C_3 = 0 &\implies  C_2 = C_3
\end{align*}$$
Therefore,

$$ \begin{align*}
C_1 \cdot X_1(t) + C_1 \cdot X_2(t) + C_1 \cdot X_3(t)&= cte\\
\end{align*}$$

for any value of $C_1$, so taking $C_1=1$, we have

$$X_1(t)+X_2(t)+X_3(t)=cte $$.

This constant value has to be true also at $t=0$,

$$X_1(t)+X_2(t)+X_3(t)= X_1(0)+X_2(0)+X_3(0) = cte $$.

We will assume, for simplicity, that initially we only have $X_1$ in the system, so the conservation of mass is $$X_1(t)+X_2(t)+X_3(t)=X_1(0)$$

The set of ODEs is:

$$\begin{align}
\frac{dX_1}{dt} &= -k_1 \cdot X_1 \tag{40}\\
\frac{dX_2}{dt} &= k_1 \cdot X_1  - k_2 \cdot X_2 \tag{41}\\
\frac{dX_3}{dt} &= k_2 \cdot X_2 \tag{42}\\
\end{align}$$

From the first case where we solved the first order reaction, we obtain the solution to the first equation, so we rewrite Eq. 15 using this result. We obtain:

$$ \frac{dX_2}{dt} = k_1 \cdot X_1 (0) \cdot e^{- k_1 t}  - k_2 \cdot X_2 \tag{43}$$

Next, this equation allows us to separate variables, as

$$ \frac{dX_2}{dt} +  k_2 \cdot X_2 = k_1 \cdot X_1 (0) \cdot e^{- k_1 t}  \tag{44}$$

Again, we need to calculate the integrating factor, $e^{ \int p(x)dx }$, which in this case is $e^{ \int k_2dt }=e^{k_2t}$. We then multiply both terms in eq. 18 by the integrating factor.

$$ \frac{dX_2}{dt}e^{k_2t} +  k_2 \cdot X_2 e^{k_2t}= k_1 \cdot X_1 (0) \cdot e^{- k_1 t}e^{k_2t}=  k_1 \cdot X_1 e^{(k_2-k_1)t}\tag{45}$$

In the next step, we need to recognize that the left-hand side of the equation is the derivative of the product of the dependent variable times the integrating factor:

$$\frac{d}{dt}\left( X_2e^{k_2t}\right)=k_1 X_1 (0) e^{(k_2-k_1)t} \tag{46}$$

We integrate

$$ \int d \left( X_2e^{k_2t}\right)=\int X_1 (0) e^{(k_2-k_1)t}dt \tag{47}$$

$$ X_2e^{k_2t}=\frac{1}{k_2-k_1} k_1 X_1 (0) e^{(k_2-k_1)t}+c \tag{48}$$

$$X_2=\frac{1}{k_2-k_1} k_1 X_1 (0) \frac{e^{(k_2-k_1)t}}{e^{k_2t}}+\frac{c}{e^{k_2t}} \tag{49}$$

$$X_2=\frac{k_1}{k_2-k_1} X_1 (0) e^{-k_1t}+ce^{-k_2t} \tag{50}$$

We have an arbitrary constant because this is a first order differential equation. Let’s calculate $c$  using the initial condition $X_2(0)=0$:

$$0=\frac{k_1}{k_2-k_1} X_1 (0) e^{-k_1 0}+ce^{-k_2 0}=\frac{k_1}{k_2-k_1} X_1 (0) + c \tag{51}$$

$$c=-\frac{k_1}{k_2-k_1} X_1 (0) \tag{52}$$

so finally we obtain the solution for $X_2$:

$$X_2(t)=\frac{k_1}{k_2-k_1}  X_1 (0) e^{-k_1t}-\frac{k_1}{k_2-k_1}  X_1 (0) e^{-k_2t} \tag{53}$$

which reorganizing terms becomes simply:

$$X_2(t)=\frac{k_1 X_1 (0)}{k_2-k_1}  \left(e^{-k_1t}-e^{-k_2t}\right) \tag{54}$$

Now that we have $X_1(t)$ and $X_2(t)$ we can get the expression for $X_3(t)$ by using simply the conservation of mass in Eq. 15

$$X_3(t)=X_1(0)-X_1(t)-X_2(t) \tag{55}$$

plugging the two solutions we obtain

$$X_3(t)=X_1(0)-X_1(0)e^{-k_1t}-\frac{k_1}{k_2-k_1} X_1(0)\left(e^{-k_1t}-e^{-k_2t}\right) \tag{56}$$

that rearranging terms becomes:

$$X_3(t)=X_1(0)\left[1-e^{-k_1t}-\frac{k_1}{k_2-k_1}\left(e^{-k_1t}-e^{-k_2t}\right)\right] \tag{57}$$

```julia
k_1 = 1.
k_2 = 0.5
X₀ = [0.1,0.0,0.0]
```

    3-element Array{Float64,1}:
     0.1
     0.0
     0.0

```julia
plot(t,t->X₀[1]*exp(-k_1*t),label="\\ X_1",seriestype=:line)
plot!(t,t->((k_1*X₀[1])/(k_2-k_1))*(exp(-k_1*t)-exp(-k_2*t)),label="\\ X_2",seriestype=:line)
plot!(t,t->X₀[1] - X₀[1]*exp(-k_1*t) - ((k_1*X₀[1])/(k_2-k_1))*(exp(-k_1*t)-exp(-k_2*t)),label="\\ X_3",seriestype=:line)
title!("Sequential First Order Reaction")
xaxis!("Time")
yaxis!("Concentration")
```

We can calculate the maximum of $X_2$ by differnetiating Eq. 53
$$\begin{align}
\frac{dX_2}{dt}=0 &\rightarrow   \frac{k_1 X_1 (0)}{k_2-k_1}  \left(-k_1 \cdot e^{-k_1t}+k_2 \cdot e^{-k_2t}\right) = 0 \tag{58}\\
k_1 \cdot e^{-k_1t}&=k_2 \cdot e^{-k_2t} \tag{59}\\
\frac{k_1}{k_2} \cdot e^{-k_1t}&= e^{-k_2t} \tag{60}\\
\frac{k_1}{k_2} &= \frac{e^{-k_2t}}{e^{-k_1t}}=e^{-k_2t}\cdot e^{k_1t}=e^{(k_1-k_2)t} \tag{61}
\end{align}$$

applying logs to both sides of the equation
$$(k_1-k_2)t=\log\frac{k_1}{k_2} \rightarrow t= \frac{1}{k_1-k_2}\log\frac{k_1}{k_2} \tag{62}$$

```julia
t_max=(1/(k_1-k_2))*log(k_1/k_2)
vline!([t_max],label="Max \\ X_2")
```

## 4. Second Order Reaction:

Consider the second order reaction $$ X_1 + X_2 \overset{k}{\longrightarrow} X_3  \tag{63} $$. This corresponds to a typical example of formation of a dimer complex.

$$\begin{align}
\frac{dX_1}{dt} &= -k \cdot X_1 \cdot X_2\tag{64}\\
\frac{dX_2}{dt} &= -k \cdot X_1 \cdot X_2 \tag{65}\\
\frac{dX_3}{dt} &= k \cdot X_1 \cdot X_2 \tag{66}\\
\end{align}$$

to solve this set of ODEs, we will have to consider two separate situations $[X_1](0) \neq [X_2](0)$ and $[X_1](0) = [X_2](0)$.

### 4.1 Situation $[X_1](0) \neq [X_2](0)$

We will define `x` as the concentration of each species that have reacted at time `t`. Since both change at the same rate, we can  rewrite the expression of rate law as:

$$\begin{align}
\frac{dX_1}{dt} &= -k \cdot (X_1(0)-x) \cdot (X_2(0)-x)\tag{67}\\
\frac{dX_2}{dt} &= -k \cdot (X_1(0)-x) \cdot (X_2(0)-x) \tag{68}\\
\frac{dX_3}{dt} &= k \cdot (X_1(0)-x) \cdot (X_2(0)-x) \tag{69}
\end{align}$$

for the derivatives we can also write:

$$\begin{align}
\frac{dX_1}{dt} &=  \frac{dX_1(0)}{dt} -  \frac{dx}{dt} = -\frac{dx}{dt} \tag{70}\\
\frac{dX_2}{dt} &= \frac{dX_2(0)}{dt} -  \frac{dx}{dt} = -\frac{dx}{dt} \tag{71}
\end{align}$$

therefore, we rewrite the full set of equations in terms of `x` and the initial conditions:
$$\begin{align}
\frac{dX_1}{dt} & = -\frac{dx}{dt} = -k \cdot (X_1(0)-x) \cdot (X_2(0)-x)\tag{72}\\
\frac{dX_2}{dt} & = -\frac{dx}{dt} = -k \cdot (X_1(0)-x) \cdot (X_2(0)-x) \tag{73}\\
\end{align}$$

we can rearrange the equations as we often do:
$$\begin{align}
 \frac{dx}{(X_1(0)-x) \cdot (X_2(0)-x)} = k dt \tag{74}\\
\end{align}$$

We now integrate between  t=0(when  x=0 t= t, the time of interest.

$$\begin{align}
 \int\frac{dx}{(X_1(0)-x) \cdot (X_2(0)-x)} = k \int  dt \tag{75}\\
\end{align}$$
we separate the left part into two integrals by using the scheme:

$$\begin{align}
\frac{1}{(a-x)(b-x)}  &= \frac{A}{a-x} + \frac{B}{b-x} \tag{76}
\end{align}$$

we need to find the values for `A` and `B`, which in this case can be obtained by solving:

$$\begin{align}
A (b-x) + B (a-x) &= 1 \tag{77}\\
Ab - Ax + Ba - Bx &= 1 \tag{78}\\
(bA + Ba) - x (A + B) + &= 1 \tag{79}
\end{align}$$

this is true if $A + B=0$ and $bA + Ba=1$ so, we substitute  $B=-A$ in the second equation and obtain

$$\begin{align}
bA - Aa &= 1 \tag{80}\\
A &= \frac{1}{b-a} \tag{81}\\
B &= -\frac{1}{b-a} \tag{82}
\end{align}$$

so the equation becomes:

$$\begin{align}
\frac{1}{(a-x)(b-x)}  &= \frac{1}{b-a}(\frac{1}{a-x} + \frac{1}{b-x}) \tag{83}
\end{align}$$

applying this solution to the left side of the equation above

$$\begin{align}
 \frac{dx}{(X_1(0)-x) \cdot (X_2(0)-x)} = \frac{1}{X_2(0)-X_1(0)}(\frac{dx}{X_1(0)-x} - \frac{dx}{X_2(0)-x})\tag{84}\\
\end{align}$$

so the equation becomes:

$$\begin{align}
 \frac{1}{X_2(0)-X_1(0)}(\int\frac{dx}{X_1(0)-x} - \int \frac{dx}{X_2(0)-x}) = k_1 \int  dt \tag{85}\\
\end{align}$$

solving

$$\begin{align}
\frac{1}{X_2(0)-X_1(0)}\left(-\ln(X_1(0) - x) + \ln(X_2(0) - x)\right) = k_1 \, t + C \tag{86}
\end{align}$$

using the definition of `x`

$$
\dfrac{1}{X_2(0)-X_1(0)}\left(-\ln X_1 + \ln X_2 \right)=  k_1 \, t + C \tag{87}
$$

using the properties of logs

$$
\dfrac{1}{ } \ln\dfrac{X_2}{X_1}=  k_1 \, t + C \tag{88}
$$

and rearranging terms

$$
\ln\dfrac{X_2}{X_1}= (k_1 \, t + C) (X_2(0)-X_1(0))\tag{89}
$$

to determinte C, we use the initial condition

$$\begin{align}
\ln\dfrac{X_2(0)}{X_1(0)}= (k_1 \, 0 + C) (X_2(0)-X_1(0))\tag{90}\\
C = \dfrac{1}{X_2(0)-X_1(0)} \ln\dfrac{X_2(0)}{X_1(0)}   \tag{91}
\end{align}$$

so the final solution is
$$
\ln\dfrac{X_2 }{X_1}= (k \, t + \dfrac{1}{(X_2(0)-X_1(0))} \ln\dfrac{X_2(0)}{X_1(0)}) \, ( X_2(0)-X_1(0)) =(k \, t (X_2(0)- X_1(0)) + \ln\dfrac{X_2(0)}{X_1(0)})\tag{92}
$$
 rearranging terms

 $$
\dfrac{X_2 }{X_1 }= e^{(k \, t ( X_2(0)-X_1(0)) + \ln\dfrac{X_2(0)}{X_1(0)})}=e^{(k \, t ( X_2(0)-X_1(0))}  e^{\ln\dfrac{X_2(0)}{X_1(0)})}\tag{93}
$$

In conclusion, in this simple second order reaction, we cannot calculate analitically $X_1$ and $X_2$, just a relation between them

$$
\dfrac{X_2}{X_1}= \dfrac{X_2(0)}{X_1(0)} e^{k \, t \, (X_2(0)- X_1(0))} \tag{94}
$$

### 4.2 Situation $[X_1](0) = [X_2](0)$

Since $X_1$ and $X_2$  react 1 to 1, allways the concetration of $X_1=X_2 $, therefore, we can use the same solution as in the previous second order reaction.

$$\text{rate} = k[X_1][X_2] = k[X_1][X_1] = k[X_1]^2 \tag{95}$$

The equations are simply:
$$
\begin{align}
\dfrac{d[X_1]}{dt} = -  k_1\,[X_1]^2 \tag{96}\\
\dfrac{d[X_2]}{dt} =  -  k_1\,[X_1]^2 \tag{97}\\
\dfrac{d[X_3]}{dt} =  k_1\,[X_1]^2 \tag{98}
\end{align}$$

The conservation of mass is now $ X_1 (t)+X_2 (t) + 2 X_3 (t) = 2 X_1 (t) + 2 X_3 (t) = 2 X_1 (0) + 2 X_3 (0) $. Taking eq. 2, we reorganize variables and integrate.

$$\begin{align*}
   \frac{\mathrm{d} X_1}{X_1^2} &=  - k_1 \mathrm{d}t \tag{99}\\
   \int  \frac{\mathrm{d} X_1}{X_1^2} &=  \int  - k_1 \mathrm{d}t \tag{100}\\
     \frac{-1}{X_1} &=  - k_1 t + C \tag{101}\\
     X_1(t) &=\frac{1}{k_1   t - C}  \tag{102}\\
\end{align*}$$

To calculate the integration constant, we use the intial condition at $t=0$.

$$\begin{align*}
     X_1(0) &=\frac{1}{k_1   0 - C} = - \frac{1}{C} \tag{103}
\end{align*}$$

so the final equation of is
$$\begin{align}
X_1(t) &=\frac{1}{k_1  t + \frac{1}{X_1(0)} } \tag{104}
\end{align}$$
so, multiplying numerator and denominator of the right side of the equation by $X_1(0)$ we obtain
$$\begin{align}
X_1(t) &=\frac{X_1(0)}{(X_1(0))( k_1  t  + \frac{1}{X_1(0)}) }= \frac{X_1(0)}{k_1 X_1(0)t + 1}\tag{105}
\end{align}$$

taking the conservation of mass we have:

$$\begin{align}
X_3(t) &= X_3 (0) + 2 X_1 (0) -  \frac{X_1(0)}{k_1 X_1(0)t + 1}=X_3 (0) +  X_1 (0) (1- \frac{1}{k_1 X_1(0)t + 1})\tag{106}
\end{align}$$

```julia
k_1 = 1.
X₀ = [0.1,0.1,0.]
plot(t,t->X₀[1]/((k_1*t*X₀[1])+1),label=" X_1 = X_2")
plot!(t,t->X₀[3]+X₀[1]*(1-1/((k_1*t*X₀[1])+1)),label="\\ X_3 ")
title!("Case  X_1  =  X_2 ")
xaxis!("Time")
yaxis!("Concentration")
```

## 5. Second order reaction: Dimerization

A special case of second order reaction is when conversion into product occur in pairs. We derive the solution for dimers, where two identical molecules interact to form a complex to then be converted to another molecule. The scheme of interaction is the following:

$$ 2X_1 \overset{k_1}{\longrightarrow} 2X_2  \tag{107} $$

with initial conditions $X_1(0)$ and $X_2(0)$, and conservation of mass:

$$X_1(t)+X_2(t)=X_1(0)+X_2(0)  \tag{108}$$

$$
\begin{align*}
   \frac{\mathrm{d} X_1}{\mathrm{d} t} &=  - 2 k_1 X_1^2 \tag{109}\\
   \frac{\mathrm{d} X_2}{\mathrm{d} t} &=   2 k_1 X_1^2  \tag{110}\\
\end{align*}
$$

It is called a second order reaction because the rate is proportional to the second power of the concentration of reactant. Taking eq. 65, we reorganize variables and integrate.
$$\begin{align*}
   \frac{\mathrm{d} X_1}{X_1^2} &=  - 2k_1 \mathrm{d}t \tag{111}\\
   \int  \frac{\mathrm{d} X_1}{X_1^2} &=  \int  - 2k_1 \mathrm{d}t \tag{112}\\
     \frac{-1}{X_1} &=  - 2k_1 t + C \tag{113}\\
     X_1(t) &=\frac{1}{2k_1   t - C}  \tag{114}\\
\end{align*}$$

To calculate the integration constant, we use the intial condition at $t=0$.

$$\begin{align*}
     X_1(0) &=\frac{1}{2k_1   0 - C} = - \frac{1}{C} \tag{115}
\end{align*}$$

so the final equation of is
$$\begin{align}
X_1(t) &=\frac{1}{2k_1  t + \frac{1}{X_1(0)} } \tag{116}
\end{align}$$
so, multiplying numerator and denominator of the right side of the equation by $X_1(0)$ we obtain
$$\begin{align}
X_1(t) &=\frac{X_1(0)}{(X_1(0))(2 k_1  t  + \frac{1}{X_1(0)}) }= \frac{X_1(0)}{2k_1 X_1(0)t + 1}\tag{117}
\end{align}$$
so using the mass conservation we obtain for $X_2$:
$$\begin{align}
X_2(t) &= X_1(0)+X_2(0) - \frac{X_1(0)}{2k_1 X_1(0)t + 1}\tag{118}\\
X_2(t) &= X_2(0) + X_1(0) (1 - \frac{1}{2k_1 X_1(0)t + 1})\tag{119}\\
X_2(t) &= X_2(0) + X_1(0) \frac{2k_1 X_1(0)t}{2k_1 X_1(0)t + 1}\tag{120}\\
X_2(t) &= X_2(0) +  \frac{2k_1 X_1(0)^2 t}{2k_1 X_1(0)t + 1}\tag{121}\\
\end{align}$$

Now we can compare the conversions of monomers versus dimers.

```julia
X₀ = [0.1,0.0]
k_1 = 1.
P1=plot(t,t->X₀[1]*exp(-k_1*t),label="monomer reaction \\  X_1",seriestype=:line)
P1=plot!(t,t->X₀[2]+X₀[1]*(1-exp(-k_1*t)),label="monomer reaction \\ X_2",seriestype=:line)
title!("Monomer reaction")
P2=plot(t,t->X₀[1]/((2*k_1*t*X₀[1])+1),label="dimer reaction \\ X_1 ")
P2=plot!(t,t->X₀[2]+(2*k_1*t*X₀[1]^2)/((2*k_1*t*X₀[1])+1),label="dimer reaction \\ X_2 ")
title!("Dimer reaction")

plot(P1,P2,layout=(1,2),legend=true)

xaxis!("Time")
yaxis!("Concentration")
```

## Open systems

Despite being developed in the context of closed chemical systems, the mass action law is also applicable to open systems, i.e., systems where tehre is mop consevarion of mass. This is very relevanty for biological systems, since they are often an open system that ecxchanges mass with the environment. A first example of an open system is when a given reactant is degraded or flows away, for instance:
$$   X_1  \overset{k_1}{\longrightarrow} 0 \tag{121} $$

In this case, the system is described by a single first order differential equation that is equivalent to eq. 3 in this notebook.

 $$\frac{\mathrm{d} X_1}{\mathrm{d} t} =  - k_1 X_1 \tag{122}$$ with a solution

 $$ X_1 (t)= X_1 (0) e^{- k_1 t}  \tag{123}$$

```julia
X₀ = [0.1,0.0]
k_1 = 1.
plot(t,t->X₀[1]*exp(-k_1*t),label="\\ X_1")
title!("Degradation first order")
xaxis!("Time")
yaxis!("Concentration")
```

Another simple example is the generation of a given reactant in the system, such as:
$$   0  \overset{k_1}{\longrightarrow} X_1 \tag{124} $$

For this particular scheme, the corresponding differential equation is simply:

 $$\frac{\mathrm{d} X_1}{\mathrm{d} t} =  k_1 \tag{125}$$ wich when we integrate we obtain the following solution:

$$\begin{align}
\int \mathrm{d} X_1 &= \int k_1  \mathrm{d} t  \tag{126}\\
 X_1(t) &=  k_1  \cdot t + C \tag{122}
\end{align} $$

again, to calculate the constant $C$ we use the initial condition for $t=0$, so $ X_1(0)=C$. Therefore:
$$ X_1(t) =  k_1  \cdot t + X_1(0) \tag{127}$$
which shows that the amount of $X_1$ grows linearly with $t$.

```julia
plot(t,t->X₀[1]+k_1*t,label="\\ X_1")
title!("Production zero order")
xaxis!("Time")
yaxis!("Concentration")
```

Next we will this solution with two reactions that look similar in terms of the reaction kinetics and equilibrium, but they are quite different in terms of their dynamics.

$$   X_1  \overset{k_1}{\longrightarrow} 2 X_1 \tag{128} $$

with this one

$$   2X_1  \overset{k_1}{\longrightarrow} 3 X_1 \tag{129} $$

in both reactions, the reaction produces a new molecule of $X_1$ but when we write the differential equations we have:

 $$\frac{\mathrm{d} X_1}{\mathrm{d} t} =  k_1 \cdot X_1 \tag{130}$$

 for the first reaction, while the second one produces:

 $$\frac{\mathrm{d} X_1}{\mathrm{d} t} =  - 2 k_1 \cdot X_1^2 + 3 k_1 \cdot X_1^2 = k_1 \cdot X_1^2 \tag{131}$$

 We know the solution when we integrate the first one:

  $$ X_1 (t)= X_1 (0) \cdot e^{k_1 t}  \tag{132}$$

withe the second one is :

$$\begin{align}
\int \mathrm{d} X_1^{-2} &=  \int k_1 \cdot \mathrm{d}t \tag{133} \\
-X_1^{-1} &=  k_1  t + C \tag{12}\\
X_1(t) &=\frac{-1}{k_1  t + C}  \tag{134}
\end{align}$$

using the intial condition of $X_1(t=0)$ we obtain the value of the integration constant.
$$\begin{align}
X_1(0) &=\frac{-1}{k_1  0 + C} = \frac{-1}{C}  \tag{135}\\
C &= \frac{-1}{X_1(0)}  \tag{131}\\
\end{align}$$

so the final equation of is
$$\begin{align}
X_1(t) &=\frac{-1}{k_1  t - \frac{1}{X_1(0)} } \tag{136}
\end{align}$$
so, multiplying numerator and denominator of the right side of the equation by $-X_1(0)$ we obtain
$$\begin{align}
X_1(t) &=\frac{(-1)\cdot(-X_1(0))}{(-X_1(0))(k_1  t  - \frac{1}{X_1(0)}) }= \frac{X_1(0)}{(1-X_1(0)k_1 t)}\tag{137}
\end{align}$$

```julia
X₀ = [0.1,0.0]
k_1 = 0.1
plot(t,t->X₀[1]+k_1*t,label="\\ zero order reaction \\ X_1 ")
plot!(t,t->X₀[1]*exp(k_1*t),label=" first order reaction \\ X_1 ")
plot!(t,t->X₀[1]/(1-k_1*t*X₀[1]),label=" Second order reaction \\ X_1 ")
title!("Production  first order")
xaxis!("Time")
yaxis!("Concentration")
```

We can see that, despite being similar in terms of the production of molecules of $X_1$ per reaction, the dynamics of zero first and second order reactions are completely different, even when using identical parameter values.


# Statistics

## 1.1 Sales for the past week was the following amounts: [3505, 2400, 3027, 2798, 3700, 3250, 2689]. Without using library functions, what is the mean, variance, and standard deviation of of sales from last week? (for extra bonus points, write your own function that can calculate these two values for any sized list)

```
import pandas as pd
import numpy as np

sales = [3505, 2400, 3027, 2798, 3700, 3250, 2689]

def mean1(numbs):
  return sum(numbs)/len(numbs)

def variance(numbs):
  mean = mean1(numbs)
  return sum([(mean-x)**2 for x in numbs])/len(numbs)

def std(numbs):
  return variance(numbs)**.5

SalesMean = mean(sales)
SalesVariance = variance(sales)
SalesSTDEV = std(sales)

print("Mean of Sales: ", SalesMean)
print("Variance of Sales: ", SalesVariance)
print("Standard Deviation of Sales: ", SalesSTDEV)
```

    Mean of Sales:  3052.714285714286
    Variance of Sales:  183761.06122448976
    Standard Deviation of Sales:  428.67360686714756

## 1.2 Find the covariance between last week's sales numbers and the number of customers that entered the store last week: [127, 80, 105, 92, 120, 115, 93] (you may use librray functions for calculating the covariance since we didn't specifically talk about its formula)

```
import math

sales = [3505, 2400, 3027, 2798, 3700, 3250, 2689]
customers = [127, 80, 105, 92, 120, 115, 93]

df = pd.DataFrame({'LWsales': sales, 'LWcustomers':customers})

print(np.cov(df['LWsales'], df['LWcustomers']))
covariance = df.cov()['LWsales']['LWcustomers']

print(covariance)
```

    [[214387.9047619    7604.35714286]
     [  7604.35714286    290.95238095]]
    7604.357142857142

## 1.3 Find the standard deviation of customers who entered the store last week. Then, use the standard deviations of both sales and customers to standardize the covariance to find the correlation coefficient that summarizes the relationship between sales and customers. (You may use library functions to check your work.)

```
last_weekstd = std(customers)
print(last_weekstd)

co_co = covariance / (df['LWsales'].std() * df['LWcustomers'].std())
print(co_co)
```

    15.792015549069118
    0.9628339778148909

## 1.4 Use pandas to import a cleaned version of the titanic dataset from the following link: [Titanic Dataset](https://raw.githubusercontent.com/Geoyi/Cleaning-Titanic-Data/master/titanic_clean.csv)

## Calculate the variance-covariance matrix and correlation matrix for the titanic dataset's numeric columns. (you can encode some of the categorical variables and include them as a stretch goal if you finish early)

```
df = pd.read_csv('https://raw.githubusercontent.com/Geoyi/Cleaning-Titanic-Data/master/titanic_clean.csv')
df.head(3)
```

      Unnamed: 0
      pclass
      survived
      name
      sex
      age
      sibsp
      parch
      ticket
      fare
      cabin
      embarked
      boat
      body
      home.dest
      has_cabin_number

      0
      1
      1.0
      1.0
      Allen, Miss. Elisabeth Walton
      female
      29.0000
      0.0
      0.0
      24160
      211.3375
      B5
      S
      2
      NaN
      St Louis, MO
      1

      1
      2
      1.0
      1.0
      Allison, Master. Hudson Trevor
      male
      0.9167
      1.0
      2.0
      113781
      151.5500
      C22 C26
      S
      11
      NaN
      Montreal, PQ / Chesterville, ON
      1

      2
      3
      1.0
      0.0
      Allison, Miss. Helen Loraine
      female
      2.0000
      1.0
      2.0
      113781
      151.5500
      C22 C26
      S
      NaN
      NaN
      Montreal, PQ / Chesterville, ON
      1

```
df.cov()
```

      Unnamed: 0
      pclass
      survived
      age
      sibsp
      parch
      fare
      body
      has_cabin_number

      Unnamed: 0
      143117.500000
      284.357034
      -53.967125
      -1442.939812
      25.828746
      1.172783
      -9410.735123
      591.579132
      -95.438885

      pclass
      284.357034
      0.701969
      -0.127248
      -3.954605
      0.053090
      0.013287
      -24.227788
      -2.876653
      -0.249992

      survived
      -53.967125
      -0.127248
      0.236250
      -0.314343
      -0.014088
      0.034776
      6.146023
      0.000000
      0.061406

      age
      -1442.939812
      -3.954605
      -0.314343
      165.850021
      -2.559806
      -1.459378
      114.416613
      81.622922
      1.463138

      sibsp
      25.828746
      0.053090
      -0.014088
      -2.559806
      1.085052
      0.336833
      8.641768
      -8.708471
      -0.003946

      parch
      1.172783
      0.013287
      0.034776
      -1.459378
      0.336833
      0.749195
      9.928031
      4.237190
      0.013316

      fare
      -9410.735123
      -24.227788
      6.146023
      114.416613
      8.641768
      9.928031
      2678.959738
      -179.164684
      10.976961

      body
      591.579132
      -2.876653
      0.000000
      81.622922
      -8.708471
      4.237190
      -179.164684
      9544.688567
      3.625689

      has_cabin_number
      -95.438885
      -0.249992
      0.061406
      1.463138
      -0.003946
      0.013316
      10.976961
      3.625689
      0.174613

```
df.corr()
```

      Unnamed: 0
      pclass
      survived
      age
      sibsp
      parch
      fare
      body
      has_cabin_number

      Unnamed: 0
      1.000000
      0.897822
      -0.293717
      -0.296172
      0.065594
      0.003584
      -0.481215
      0.015558
      -0.603727

      pclass
      0.897822
      1.000000
      -0.312469
      -0.366370
      0.060832
      0.018322
      -0.558629
      -0.034642
      -0.713857

      survived
      -0.293717
      -0.312469
      1.000000
      -0.050199
      -0.027825
      0.082660
      0.244265
      NaN
      0.302250

      age
      -0.296172
      -0.366370
      -0.050199
      1.000000
      -0.190747
      -0.130872
      0.171892
      0.059059
      0.271887

      sibsp
      0.065594
      0.060832
      -0.027825
      -0.190747
      1.000000
      0.373587
      0.160238
      -0.099961
      -0.009064

      parch
      0.003584
      0.018322
      0.082660
      -0.130872
      0.373587
      1.000000
      0.221539
      0.051099
      0.036806

      fare
      -0.481215
      -0.558629
      0.244265
      0.171892
      0.160238
      0.221539
      1.000000
      -0.043110
      0.507253

      body
      0.015558
      -0.034642
      NaN
      0.059059
      -0.099961
      0.051099
      -0.043110
      1.000000
      0.083796

      has_cabin_number
      -0.603727
      -0.713857
      0.302250
      0.271887
      -0.009064
      0.036806
      0.507253
      0.083796
      1.000000

# Orthogonality

## 2.1 Plot two vectors that are orthogonal to each other. What is a synonym for orthogonal?

```
import matplotlib.pyplot as plt
# A synonym for orthogonal is perpendicular

v1 = np.array([10,0])
v2 = np.array([0,10])

plt.arrow(0,0, *v1, head_width=1, head_length=1, color='purple')
plt.arrow(0,0, *v2, head_width=1, head_length=1, color='blue')
plt.xlim(-1, 15)
plt.ylim(-1, 15)
plt.show()
```

## 2.2 Are the following vectors orthogonal? Why or why not?

\begin{align}
a = \begin{bmatrix} -5 \\ 3 \\ 7 \end{bmatrix}
\qquad
b = \begin{bmatrix} 6 \\ -8 \\ 2 \end{bmatrix}
\end{align}

```
#We can calculate the dot product to find out:

a = np.array([-5,3,7]).T
b = np.array([6,-8,2]).T

np.dot(a, b)

# Not orthogonal bc the dot product is -40, not zero+
```

    -40

## 2.3 Compute the following values: What do these quantities have in common?

## What is $||c||^2$?

## What is $c \cdot c$?

## What is $c^{T}c$?

\begin{align}
c = \begin{bmatrix} 2 & -15 & 6 & 20 \end{bmatrix}
\end{align}

```
c = np.array([2, -15, 6, 20])

one = (np.linalg.norm(c) ** 2)

two = (np.dot(c, c))

three = (np.matmul(c.T, c))
print (one, two, three)

#All the same number
```

    665.0 665 665

# Unit Vectors

## 3.1 Using Latex, write the following vectors as a linear combination of scalars and unit vectors:

\begin{align}
d = \begin{bmatrix} 7 \\ 12 \end{bmatrix}
\qquad
e = \begin{bmatrix} 2 \\ 11 \\ -8  \end{bmatrix}
\end{align}

Your text here

## 3.2 Turn vector $f$ into a unit vector:

\begin{align}
f = \begin{bmatrix} 4 & 12 & 11 & 9 & 2 \end{bmatrix}
\end{align}

```
f = [4, 12, 11, 9, 2]

unitvec = f/np.linalg.norm(f)
print(unitvec)
```

    [0.20908335 0.62725005 0.57497921 0.47043754 0.10454167]

# Linear Independence / Dependence

## 4.1 Plot two vectors that are linearly dependent and two vectors that are linearly independent (bonus points if done in $\mathbb{R}^3$).

```

```

# Span

## 5.1 What is the span of the following vectors?

\begin{align}
g = \begin{bmatrix} 1 & 2 \end{bmatrix}
\qquad
h = \begin{bmatrix} 4 & 8 \end{bmatrix}
\end{align}

```
g = np.array([1, 2])
h = np.array([4, 8])

span = np.linalg.matrix_rank(g, h)
print(span)
```

    1

## 5.2 What is the span of $\{l, m, n\}$?

\begin{align}
l = \begin{bmatrix} 1 & 2 & 3 \end{bmatrix}
\qquad
m = \begin{bmatrix} -1 & 0 & 7 \end{bmatrix}
\qquad
n = \begin{bmatrix} 4 & 8  & 2\end{bmatrix}
\end{align}

```
l = np.array([1,2,3])
m = np.array([-1,0,7])
n = np.array([4, 8, 2])

span2 = np.linalg.matrix_rank(l, m, n)
print(span2)
#that didnt work.....

np.linalg.matrix_rank(np.array([[1, 2, 3], [-1, 0, 7], [4, 8, 2]]))

```

    1

    3

# Basis

## 6.1 Graph two vectors that form a basis for $\mathbb{R}^2$

```
# Could use my orthogonal vectors from 2.1 but I'll make a new one

v1 = np.array([20,0])
v2 = np.array([0,20])

plt.arrow(0,0, *v1, head_width=1, head_length=1, color='purple')
plt.arrow(0,0, *v2, head_width=1, head_length=1, color='pink')
plt.xlim(-5, 25)
plt.ylim(-5, 25)
plt.show()
```

## 6.2 What does it mean to form a basis?

Vectors in a space that are linearly independent and span the whole vector space

# Rank

## 7.1 What is the Rank of P?

\begin{align}
P = \begin{bmatrix}
1 & 2 & 3 \\
 -1 & 0 & 7 \\
4 & 8  & 2
\end{bmatrix}
\end{align}

```
# Exactly the same as 5.2 but this comined it to one big matrix. Can solve it the same way
RoP = np.linalg.matrix_rank(np.array([[1, 2, 3], [-1, 0, 7], [4, 8, 2]]))
RoP
```

    3

## 7.2 What does the rank of a matrix tell us?

It can tell us how many independent columns and rows are in a given matrix and the different ways the matrix may be legally transformed. And other things too.

# Linear Projections

## 8.1 Line $L$ is formed by all of the vectors that can be created by scaling vector $v$
\begin{align}
v = \begin{bmatrix} 1 & 3 \end{bmatrix}
\end{align}

\begin{align}
w = \begin{bmatrix} -1 & 2 \end{bmatrix}
\end{align}

## find $proj_{L}(w)$

## graph your projected vector to check your work (make sure your axis are square/even)

```
v = np.array([1, 3])
w = np.array([-1, 2])

sca_v = np.dot(w, v)/(np.dot(v, v))
proj= sca_v * v
proj
```

    array([0.5, 1.5])

```
#Checking

plt.arrow(0, 0, *v, head_width=.3, head_length=.3, color='purple')
plt.arrow(0, 0, *w, head_width=.3, head_length=.3, color='pink')
plt.arrow(0, 0, *proj, head_width=.3, head_length=.3, color='blue')
plt.xlim(-2, 5)
plt.ylim(-1, 5)
plt.show()
```

# Stretch Goal

## For vectors that begin at the origin, the coordinates of where the vector ends can be interpreted as regular data points. (See 3Blue1Brown videos about Spans, Basis, etc.)

## Write a function that can calculate the linear projection of each point (x,y) (vector) onto the line y=x. run the function and plot the original points in blue and the new projected points on the line y=x in red.

## For extra points plot the orthogonal vectors as a dashed line from the original blue points to the projected red points.

```
import pandas as pd
import matplotlib.pyplot as plt

# Creating a dataframe for you to work with -Feel free to not use the dataframe if you don't want to.
x_values = [1, 4, 7, 3, 9, 4, 5 ]
y_values = [4, 2, 5, 0, 8, 2, 8]

data = {"x": x_values, "y": y_values}

df = pd.DataFrame(data)

df.head()

plt.scatter(df.x, df.y)
plt.show()
```

```

```


##  Taylor problem 5.2

last revised: 14-Jan-2019 by Dick Furnstahl [furnstahl.1@osu.edu]

The potential energy for two atoms in a molecule can sometimes be approximated by the Morse function

$\begin{align}
  U(r) = A\left[ \left(e^{(R-r)/S} -1\right)^2 -1 \right]
\end{align}$

for $0 < r < \infty$ with $A$, $R$, and $S$ all positive constants and $S \ll R$.

The goals are to find the equilibrium position $r_0$ and then the frequency of small oscillations.

Take the derivative and solve for the equilibrium $r_0$:

$\begin{align}
 \frac{dU}{dr} = 2A\left(e^{(R-r)/S} -1\right)\times e^{(R-r)/S} \times \frac{-1}{S} = 0
 \quad\Longrightarrow\quad
 r_0 = R
  \quad\Longrightarrow\quad
 U(r_0) = -A
\end{align}$

Expand around $r = r_0$ up to $(r - r_0)^2$ to find the small angle approximation equation and frequency:

$\begin{align}
U_{sa}(r) = A + \frac{A}{S^2} (r - r_0)^2
 \quad\Longrightarrow\quad
 k = \frac{2 A}{S^2}
 \quad\Longrightarrow\quad
 \omega = \sqrt{k/m} = \sqrt{\frac{2 A}{ m S^2}}
\end{align}$

## Make a plot to check $r_0$

```python
%matplotlib notebook
# Use notebook rather than inline so plot remains active between
#  cells and so we can zoom in.
```

```python
import numpy as np

import matplotlib.pyplot as plt
```

```python
def U_pot(r, S=0.1, big_R=1, A=1):
    """Potential from Taylor problem 5.13."""
    return A * ( (np.exp((big_R - r)/S) - 1.)**2 - 1. )

def U_pot_min(S=0.1, big_R=1, A=1):
    """Returns the value of r at the minimum of U(r) and the value of
       U(r) at that point."""
    return big_R,  -A
```

```python
def U_pot_sa(r, S=0.1, big_R=1, A=1):
    """Plot the potential in the small angle approximation"""
    r0 = big_R
    return  -A + A * (r - r0)**2 / S**2

```

```python
r_pts = np.arange(0.01, 6., .001)
U_pot_1 = U_pot(r_pts, S=0.2)
U_pot_2 = U_pot(r_pts, S=0.5)
#U_pot_3 = U_pot(r_pts, S=0.3)

fig = plt.figure(figsize=(4,4))
ax = fig.add_subplot(1,1,1)
ax.plot(r_pts, U_pot_1, label=r'$S=0.2$', color='blue')
ax.plot(r_pts, U_pot_2, label=r'$S=0.5$', color='red')
#ax.plot(r_pts, U_pot_3, label=r'$S=0.3$', color='green')

ax.set_xlim(0.,2.5)
ax.set_ylim(-2.,5.)

x0, y0 = U_pot_min(S=0.2)
ax.scatter(x0, y0, color='blue')
x0, y0 = U_pot_min(S=0.5)
ax.scatter(x0, y0, color='red')
#x0, y0 = U_pot_min(S=0.3)
#ax.scatter(x0, y0, color='green')

ax.set_xlabel('r')
ax.set_ylabel('U(r)')

ax.legend()
fig.tight_layout()
```

Now put in small angle potentials (note that this changes the figure above, which is still the active figure (because we are using `%matplotlib notebook`).  Use the controls to zoom in to verify that the small-angle approximated $U(r)$ really does agree for $r$ close enough to the minimum.

```python
# "sa" here stands for "small-angle".
U_pot_sa_1 = U_pot_sa(r_pts, S=0.2)
U_pot_sa_2 = U_pot_sa(r_pts, S=0.5)
#U_pot_sa_3 = U_pot_sa(r_pts, S=0.3)
ax.plot(r_pts, U_pot_sa_1, color='blue', ls='--')
ax.plot(r_pts, U_pot_sa_2, color='red', ls='--')
#ax.plot(r_pts, U_pot_sa_3, color='green', ls='--')

```

Can we solve for exact and small angle oscillations numerically?

```python

```


# 2021-10-18 Iterative methods

## Last time

* Sparse direct solvers
  * matrix orderings
  * impact on formulation
  * cost scaling
* Discussion with Ioana Fleming

## Today

* Why iterative solvers
* Stationary iterative methods
* Preconditioning

```julia
using Plots
using LinearAlgebra
using SparseArrays

default(linewidth=4)

function advdiff_matrix(n; kappa=1, wind=[0, 0])
    "Advection-diffusion with Dirichlet boundary conditions eliminated"
    h = 2 / (n + 1)
    rows = Vector{Int64}()
    cols = Vector{Int64}()
    vals = Vector{Float64}()
    idx((i, j),) = (i-1)*n + j
    in_domain((i, j),) = 1 <= i <= n && 1 <= j <= n
    stencil_advect = [-wind[1], -wind[2], 0, wind[1], wind[2]] / h
    stencil_diffuse = [-1, -1, 4, -1, -1] * kappa / h^2
    stencil = stencil_advect + stencil_diffuse
    for i in 1:n
        for j in 1:n
            neighbors = [(i-1, j), (i, j-1), (i, j), (i+1, j), (i, j+1)]
            mask = in_domain.(neighbors)
            append!(rows, idx.(repeat([(i,j)], 5))[mask])
            append!(cols, idx.(neighbors)[mask])
            append!(vals, stencil[mask])
        end
    end
    sparse(rows, cols, vals)
end

function my_spy(A)
    cmax = norm(vec(A), Inf)
    s = max(1, ceil(120 / size(A, 1)))
    spy(A, marker=(:square, s), c=:diverging_rainbow_bgymr_45_85_c67_n256, clims=(-cmax, cmax))
end
```

    my_spy (generic function with 1 method)

# Why iterative solvers over direct solvers?

* Less reliable, more leaky abstraction
* More sensitive to problem formulation
* Slower for small problems
* Several different strategies, each with tuning knobs
* Accuracy tolerances needed

## $O(N)$ solvers available for many important problems
* High-order discretization *can* be okay

# Gradient descent

Suppose $A$ is a symmetric positive definite matrix and consider the scalar functional

$$f(u) = \frac 1 2 u^T A u - b^T u . $$

Then the gradient is

$$\nabla_u f = A u - b .$$

```julia
x = LinRange(-4, 4, 40)
A = [1 0; 0 3]
b = [1, 1]
f(u) = .5 * u' * A * u - b' * u
contour(x, x, (u1, u2) -> f([u1, u2]), aspect_ratio=:equal)
```

# Aside: Derivative of a dot product

Let $f(\boldsymbol x) = \boldsymbol y^T \boldsymbol x = \sum_i y_i x_i$ and compute the derivative

$$ \frac{\partial f}{\partial \boldsymbol x} = \begin{bmatrix} y_0 & y_1 & \dotsb \end{bmatrix} = \boldsymbol y^T . $$

Note that $\boldsymbol y^T \boldsymbol x = \boldsymbol x^T \boldsymbol y$ and we have the product rule,

$$ \frac{\partial \lVert \boldsymbol x \rVert^2}{\partial \boldsymbol x} = \frac{\partial \boldsymbol x^T \boldsymbol x}{\partial \boldsymbol x} = 2 \boldsymbol x^T . $$

Also,
$$ \frac{\partial \lVert \boldsymbol x - \boldsymbol y \rVert^2}{\partial \boldsymbol x} = \frac{\partial (\boldsymbol x - \boldsymbol y)^T (\boldsymbol x - \boldsymbol y)}{\partial \boldsymbol x} = 2 (\boldsymbol x - \boldsymbol y)^T .$$

# Aside: Variational notation

It's convenient to express derivatives in terms of how they act on an infinitessimal perturbation. So we might write

$$ \delta f = \frac{\partial f}{\partial x} \delta x .$$

(It's common to use $\delta x$ or $dx$ for these infinitesimals.) This makes inner products look like a normal product rule

$$ \delta(\mathbf x^T \mathbf y) = (\delta \mathbf x)^T \mathbf y + \mathbf x^T (\delta \mathbf y). $$

A powerful example of variational notation is differentiating a matrix inverse

$$ 0 = \delta I = \delta(A^{-1} A) = (\delta A^{-1}) A + A^{-1} (\delta A) $$
and thus
$$ \delta A^{-1} = - A^{-1} (\delta A) A^{-1} $$

# Try gradient descent

$$ u_{k+1} = u_k - \omega \nabla_u f $$

```julia
function grad_descent(loss, grad, u0; omega=1e-3, tol=1e-5)
    """Minimize loss(c) via gradient descent with initial guess u0
    using learning rate gamma.  Declares convergence when gradient
    is less than tol or after 500 steps.
    """
    u = copy(u0)
    uhist = [copy(u)]
    lhist = [loss(u)]
    for it in 1:500
        g = grad(u)
        u -= omega * g
        push!(uhist, copy(u))
        push!(lhist, loss(u))
        if norm(g) < tol
            break
        end
    end
    (u, hcat(uhist...), lhist)
end
```

    grad_descent (generic function with 1 method)

# Visualize gradient descent

```julia
A = [1 1; 1 4]
loss(u) = .5 * u' * A * u
grad(u) = A * u
u, uhist, lhist = grad_descent(loss, grad, [.9, .9],
    omega=.48)
plot(lhist, yscale=:log10)
```

```julia
plot(uhist[1, :], uhist[2, :], marker=:circle)
x = LinRange(-1, 1, 30)
contour!(x, x, (x,y) -> loss([x, y]))
```

# Richardson iteration

The simplest iterative method is [Richardson's method](https://en.wikipedia.org/wiki/Modified_Richardson_iteration), which solves $A u = b$ by the iteration
$$ u_{k+1} = u_k + \omega (b - A u_k) $$
where $\omega > 0$ is a damping parameter and $u_0$ is an initial guess (possibly the zero vector).

* Algebraically equivalent to gradient descent when $A$ is SPD
* Non-symmetric matrices are harder to visualize

# Richardson convergence

$$ u_{k+1} = u_k + \omega (b - A u_k) $$

If $b = A u_*$, this iteration is equivalent to
\begin{align}
u_{k+1} - u_* &= (u_k - u_*) - \omega A (u_k - u_*) \\
&= (I - \omega A) (u_k - u_*) .
\end{align}
It is convenient for convergence analysis to identify the "error" $e_k = u_k - u_*$, in which this becomes
$$ e_{k+1} = (I - \omega A) e_k $$
or
$$ e_k = (I - \omega A)^k e_0 $$
in terms of the initial error.  Evidently powers of the *iteration matrix* $I - \omega A$ tell the whole story.

Suppose that the eigendecomposition
$$ X \Lambda X^{-1} = I - \omega A $$
exists.  Then
$$ (I - \omega A)^k = (X \Lambda X^{-1})^k = X \Lambda^k X^{-1} $$
and the convergence (or divergence) rate depends only on the largest magnitude eigenvalue.
This analysis isn't great for two reasons:

1. Not all matrices are diagonalizable.
2. The matrix $X$ may be very ill-conditioned.

# Aside: Schur decomposition

We can repair these weaknesses by using the [Schur decomposition](https://en.wikipedia.org/wiki/Schur_decomposition)
$$ Q R Q^h = I - \omega A $$
where $R$ is right-triangular and $Q$ is unitary (i.e., orthogonal if real-valued; $Q^h$ is the Hermitian conjugate of $Q$).
The Schur decomposition always exists and $Q$ has a condition number of 1.

* Where are the eigenvalues in $R$?

Evidently we must find $\omega$ to minimize the maximum eigenvalue of $I - \omega A$.  We can do this if $A$ is well conditioned, but not in general.

# Ill-conditioning

## Question: What is the condition number of the Laplacian on 100 evenly spaced points?

* How does it scale under grid refinement?

```julia
A = advdiff_matrix(10)
cond(Matrix(A))
```

    48.37415007870825

```julia
omega = .01
ev = eigvals(Matrix(I - omega * A))
scatter(real.(ev), imag.(ev))
```

# Monic polynomials small on the spectrum

Equivalently to finding $\omega$ such that $\lVert I - \omega A \rVert$ is minimized, we may seek a monic polynomial $p(z) = 1 - \omega z$ that minimizes

$$ \max_{\lambda \in \sigma(A)} \lvert p(\lambda) \rvert . $$

This concept can be extended to higher degree polynomials, which is essentially what Krylov methods do (discovering the polynomial adaptively, weighted by the right hand side).

```julia
ev = eigvals(Matrix(A))
scatter(real.(ev), zero.(ev))
plot!(x -> 1 - omega * x)
```

# Preconditioning

Preconditioning is the act of creating an "affordable" operation "$P^{-1}$" such that $P^{-1} A$ (or $A P^{-1}$) is is well-conditoned or otherwise has a "nice" spectrum.  We then solve the system

$$ P^{-1} A x = P^{-1} b \quad \text{or}\quad A P^{-1} \underbrace{(P x)}_y = b $$

in which case the convergence rate depends on the spectrum of the iteration matrix
$$ I - \omega P^{-1} A . $$

* The preconditioner must be applied on each iteration.
* It is *not* merely about finding a good initial guess.

There are two complementary techniques necessary for efficient iterative methods:

* "accelerators" or Krylov methods, which use orthogonality to adaptively converge faster than Richardson
* preconditioners that improve the spectrum of the preconditioned operator

Although there is ongoing research in Krylov methods and they are immensely useful, I would say preconditioning is 90% of the game for practical applications, particularly as a research area.


# Laboratório 3: Bactéria

### Referente ao capítulo 7

Nesse laboratório, queremos controlar o crescimento de uma população de bactérias através do uso de um químico, que acelera o crescimento dessa população, mas também cria um produto secundário com efeito negativo, como descrito abaixo na equação diferencial. Seja $u(t)$ a quantidade de químico adicionado. Nesse meio, queremos maximizar a quantidade de bactérias no tempo final, com a menor quantidade de químico possível. Consideramos o intervalo de tempo $[0,1]$

## O problema

$$
\max_u Cx(1) - \int_0^1 u(t)^2 dt \\
\text{sujeito a  }x'(t) = rx(t) + Au(t)x(t) - Bu(t)^2e^{-x(t)}, x(0) = x_0
$$
tal que $A,B,C \geq 0$.

$A$ é o parâmetro da força do químico em crescer a população, enquanto $B$ mede a força do efeito negativo. Note que quanto maior a população de bactérias, exponencialmente menor será o efeito negativo do químico.

É fácil notar que $f$ e $g$ são continuamente diferenciáveis e côncavas em $x$ e $u$. Vamos verificar as condições necessárias.

## Condições Necessárias

### Hamiltoniano

$$
H = - u(t)^2 + \lambda(t)[rx(t) + Au(t)x(t) - Bu(t)^2e^{-x(t)}]
$$

### Condição de otimalidade

$$
0 = H_u = -2u + A\lambda x - 2B\lambda u e^{-x} \implies u^*(t) = \frac{A\lambda x}{2 + 2B\lambda e^{-x}}
$$

### Equação adjunta

$$
\lambda '(t) = - H_x = - \lambda(t)\left(r + Au(t) + Bu(t)^2e^{-x(t)}\right)
$$

### Condição de transversalidade

$$
\lambda(1) = \phi '(x) = C
$$

Temos que provar que $\lambda(t) \ge 0$. Observe que, utilizando matemática de EDOs,

$$\lambda(t) = \lambda(0)\exp\left\{-\int \left(r + Au(s) + Bu(s)^2e^{-x(s)}\right) ds\right\}$$

Se $\lambda(0) < 0$, então $\lambda(1) = C < 0$, o que é um absurdo, dado que $C \ge 0$. Portanto $\lambda(t) \ge 0$.

### Importanto as bibliotecas

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.integrate import solve_ivp
import sympy as sp

import sys
sys.path.insert(0, '../pyscripts/')

from optimal_control_class import OptimalControl
```

Mais uma vez usaremos a biblioteca de matemática simbólica para fazer as derivadas

```python
x_sp, u_sp, lambda_sp, r_sp, A_sp, B_sp = sp.symbols('x u lambda r A B')
H = - u_sp**2 + lambda_sp*(r_sp*x_sp + A_sp*u_sp*x_sp - B_sp*u_sp**2*sp.exp(-x_sp))
H
```

$\displaystyle \lambda \left(A u x - B u^{2} e^{- x} + r x\right) - u^{2}$

```python
print('H_x = {}'.format(sp.diff(H,x_sp)))
print('H_u = {}'.format(sp.diff(H,u_sp)))
print('H_lambda = {}'.format(sp.diff(H,lambda_sp)))
```

    H_x = lambda*(A*u + B*u**2*exp(-x) + r)
    H_u = lambda*(A*x - 2*B*u*exp(-x)) - 2*u
    H_lambda = A*u*x - B*u**2*exp(-x) + r*x

Podemos resolver a equação $H_u = 0$, mas esse passo é importante conferir manualmente também.

```python
eq = sp.Eq(sp.diff(H,u_sp), 0)
sp.solve(eq,u_sp)
```

    [A*lambda*x*exp(x)/(2*(B*lambda + exp(x)))]

Primeiro precisamos definir as equações importantes das condições necessárias. É importante escrever no formato descrito nesse notebook. `par` é um dicionário com os parâmetros específicos do modelo. Agora vamos precisar usar o tempo payoff.

```python
parameters = {'r': None, 'A': None, 'B': None, 'C': None}

diff_state = lambda t, x, u, par: par['A']*u*x - par['B']*u**2*np.exp(-x) + par['r']*x
diff_lambda = lambda t, x, u, l, par: - l*(par['B']*u**2*np.exp(-x) + par['r'] + par['A']*u)
update_u = lambda t, x, lambda_, par: par['A']*lambda_*x*np.exp(x)/(2*(par['B']*lambda_ + np.exp(x)))
diff_payoff = lambda x, par: par['C']
```

## Aplicando a classe ao exemplo

Vamos fazer algumas exeperimentações. Sinta-se livre para variar os parâmetros.

```python
problem = OptimalControl(diff_state, diff_lambda, update_u, diff_phi = diff_payoff)
```

```python
x0 = 1
T = 1
parameters['A'] = 1
parameters['B'] = 12
parameters['C'] = 1
parameters['r'] = 1
```

```python
t,x,u,lambda_ = problem.solve(x0, T, parameters)
ax = problem.plotting(t,x,u,lambda_)
```

Vemos que a concentração do químico se dá no final do intervalo. Quando $x$ cresce, o termo exponencial decresce e, portanto, o produto secundário perde efeito. O que faz com que o aumento do químico cause um efeito mais positivo no nível de bactérias, e portanto a derivada de $x$ cresce mais no decorrer.

### Variando a condição inicial

Observe que pequenas variações no valor inicial geram valores mais altos da quantidade de bactérias. Quanto mais próximo de $1.15$, mais problemas numéricos começam a aparecer.

```python
x0_values = [0.9, 1.0, 1.1, 1.1495]
x_values = []
for x0 in x0_values:
    _,x,_,_ = problem.solve(x0, T, parameters)
    x_values.append(x)
```

```python
fig = plt.figure(figsize = (10,5))
plt.xlabel("Tempo")
plt.ylabel("Bacterias")
plt.title("Quantidade populacional de bactérias")
for i, x0 in enumerate(x0_values):
    plt.plot(t, x_values[i],label = r'$x_0$ = {}'.format(x0))
plt.legend()
plt.grid(alpha = 0.5)
```

### Variando o efeito do químico

O aumento do parâmetro $A$ deve aumentar o efeito positivo do químico para o crescimento da bactéria. Isso deve gerar um aumento no uso do químico para aumentar $x(1)$.

```python
x0 = 1
A_values = [0, 0.4, 1.1]
u_values = []
for A in A_values:
    parameters['A'] = A
    _,_,u,_ = problem.solve(x0, T, parameters)
    u_values.append(u)

fig = plt.figure(figsize = (10,5))
plt.xlabel("Tempo")
plt.ylabel("Químico")
plt.title("Quantidade de químico utilizada")
for i, A in enumerate(A_values):
    plt.plot(t, u_values[i],label = r'$A$ = {}'.format(A))
plt.legend()
plt.grid(alpha = 0.5)
```

## Experimentação

Descomente a célula a seguir e varie os parâmetros para ver seus efeitos:

1. Aumentar $r$ para ver o crescimento das bactérias. O que isso gera na quantidade de químico?
2. Variar $B$ deve causar que tipo de efeito?

```python
#x0 = 1
#parameters['r'] = 1
#parameters['A'] = 1
#parameters['B'] = 12
#parameters['C'] = 1
#
#t,x,u,lambda_ = problem.solve(x0, T, parameters)
#roblem.plotting(t,x,u,lambda_)
```

### Este é o final do notebook


# Gaussian Process (GP) smoothing

This example deals with the case when we want to **smooth** the observed data points $(x_i, y_i)$ of some 1-dimensional function $y=f(x)$, by finding the new values $(x_i, y'_i)$ such that the new data is more "smooth" (see more on the definition of smoothness through allocation of variance in the model description below) when moving along the $x$ axis.

It is important to note that we are **not** dealing with the problem of interpolating the function $y=f(x)$ at the unknown values of $x$. Such problem would be called "regression" not "smoothing", and will be considered in other examples.

If we assume the functional dependency between $x$ and $y$ is **linear** then, by making the independence and normality assumptions about the noise, we can infer a straight line that approximates the dependency between the variables, i.e. perform a linear regression. We can also fit more complex functional dependencies (like quadratic, cubic, etc), if we know the functional form of the dependency in advance.

However, the **functional form** of $y=f(x)$ is **not always known in advance**, and it might be hard to choose which one to fit, given the data. For example, you wouldn't necessarily know which function to use, given the following observed data. Assume you haven't seen the formula that generated it:

```python
%pylab inline
figsize(12, 6);
```

    Populating the interactive namespace from numpy and matplotlib

```python
import numpy as np
import scipy.stats as stats

x = np.linspace(0, 50, 100)
y = (np.exp(1.0 + np.power(x, 0.5) - np.exp(x/15.0)) +
     np.random.normal(scale=1.0, size=x.shape))

plot(x, y);
xlabel("x");
ylabel("y");
title("Observed Data");
```

### Let's try a linear regression first

As humans, we see that there is a non-linear dependency with some noise, and we would like to capture that dependency. If we perform a linear regression, we see that the "smoothed" data is less than satisfactory:

```python
plot(x, y);
xlabel("x");
ylabel("y");

lin = stats.linregress(x, y)
plot(x, lin.intercept + lin.slope * x);
title("Linear Smoothing");
```

### Linear regression model recap

The linear regression assumes there is a linear dependency between the input $x$ and output $y$, sprinkled with some noise around it so that for each observed data point we have:

$$ y_i = a + b\, x_i + \epsilon_i $$

where the observation errors at each data point satisfy:

$$ \epsilon_i \sim N(0, \sigma^2) $$

with the same $\sigma$, and the errors are independent:

$$ cov(\epsilon_i, \epsilon_j) = 0 \: \text{ for } i \neq j $$

The parameters of this model are $a$, $b$, and $\sigma$. It turns out that, under these assumptions, the maximum likelihood estimates of $a$ and $b$ don't depend on $\sigma$. Then $\sigma$ can be estimated separately, after finding the most likely values for $a$ and $b$.

### Gaussian Process smoothing model

This model allows departure from the linear dependency by assuming that the dependency between $x$ and $y$ is a Brownian motion over the domain of $x$. This doesn't go as far as assuming a particular functional dependency between the variables. Instead, by **controlling the standard deviation of the unobserved Brownian motion** we can achieve different levels of smoothness of the recovered functional dependency at the original data points.

The particular model we are going to discuss assumes that the observed data points are **evenly spaced** across the domain of $x$, and therefore can be indexed by $i=1,\dots,N$ without the loss of generality. The model is described as follows:

\begin{equation}
\begin{aligned}
z_i & \sim \mathcal{N}(z_{i-1} + \mu, (1 - \alpha)\cdot\sigma^2) \: \text{ for } i=2,\dots,N \\
z_1 & \sim ImproperFlat(-\infty,\infty) \\
y_i & \sim \mathcal{N}(z_i, \alpha\cdot\sigma^2)
\end{aligned}
\end{equation}

where $z$ is the hidden Brownian motion, $y$ is the observed data, and the total variance $\sigma^2$ of each ovservation is split between the hidden Brownian motion and the noise in proportions of $1 - \alpha$ and $\alpha$ respectively, with parameter $0 < \alpha < 1$ specifying the degree of smoothing.

When we estimate the maximum likelihood values of the hidden process $z_i$ at each of the data points, $i=1,\dots,N$, these values provide an approximation of the functional dependency $y=f(x)$ as $\mathrm{E}\,[f(x_i)] = z_i$ at the original data points $x_i$ only. Therefore, again, the method is called smoothing and not regression.

### Let's describe the above GP-smoothing model in PyMC3

```python
import pymc3 as pm

from pymc3.distributions.timeseries import GaussianRandomWalk
from scipy import optimize
from theano import shared
```

Let's create a model with a shared parameter for specifying different levels of smoothing. We use very wide priors for the "mu" and "tau" parameters of the hidden Brownian motion, which you can adjust according to your application.

```python
LARGE_NUMBER = 1e5

model = pm.Model()
with model:
    smoothing_param = shared(0.9)
    mu = pm.Normal("mu", sigma=LARGE_NUMBER)
    tau = pm.Exponential("tau", 1.0/LARGE_NUMBER)
    z = GaussianRandomWalk("z",
                           mu=mu,
                           tau=tau / (1.0 - smoothing_param),
                           shape=y.shape)
    obs = pm.Normal("obs",
                    mu=z,
                    tau=tau / smoothing_param,
                    observed=y)
```

Let's also make a helper function for inferring the most likely values of $z$:

```python
def infer_z(smoothing):
    with model:
        smoothing_param.set_value(smoothing)
        res = pm.find_MAP(vars=[z], fmin=optimize.fmin_l_bfgs_b)
        return res['z']
```

Please note that in this example, we are only looking at the MAP estimate of the unobserved variables. We are not really interested in inferring the posterior distributions. Instead, we have a control parameter $\alpha$ which lets us allocate the variance between the hidden Brownian motion and the noise. Other goals and/or different models may require sampling to obtain the posterior distributions, but for our goal a MAP estimate will suffice.

### Exploring different levels of smoothing

Let's try to allocate 50% variance to the noise, and see if the result matches our expectations.

```python
smoothing = 0.5
z_val = infer_z(smoothing)

plot(x, y);
plot(x, z_val);
title("Smoothing={}".format(smoothing));
```

It appears that the variance is split evenly between the noise and the hidden process, as expected.

Let's try gradually increasing the smoothness parameter to see if we can obtain smoother data:

```python
smoothing = 0.9
z_val = infer_z(smoothing)

plot(x, y);
plot(x, z_val);
title("Smoothing={}".format(smoothing));
```

### Smoothing "to the limits"

By increading the smoothing parameter, we can gradually make the inferred values of the hidden Brownian motion approach the average value of the data. This is because as we increase the smoothing parameter, we allow less and less of the variance to be allocated to the Brownian motion, so eventually it aproaches the process which almost doesn't change over the domain of $x$:

```python
fig, axes = subplots(2, 2)

for ax, smoothing in zip(axes.ravel(), [0.95, 0.99, 0.999, 0.9999]):

    z_val = infer_z(smoothing)

    ax.plot(x, y)
    ax.plot(x, z_val)
    ax.set_title('Smoothing={:05.4f}'.format(smoothing))
```

```python
%load_ext watermark
%watermark -n -u -v -iv -w
```

    pymc3            3.9.0
    re               2.2.1
    numpy            1.18.5
    matplotlib       3.2.1
    matplotlib.pylab 1.18.5
    logging          0.5.1.2
    last updated: Fri Jun 12 2020

    CPython 3.7.7
    IPython 7.15.0
    watermark 2.0.2

This example originally contributed by: Andrey Kuzmenko, http://github.com/akuz


ENGR 1330 Computational Thinking with Data Science

Copyright © 2021 Theodore G. Cleveland and Farhang Forghanparast

Last GitHub Commit Date:

# Appendix: Newton's Method
- Background
- Analytical Derivatives
- Numerical Derivatives
- Examples

## Numerical Methods - Single Variable Newtons Method
The application of fundamental principles of modeling and mechanics often leads to an
algebraic or transcendental equation that cannot be easily solved and represented in a closed
form. In these cases a numerical method is required to obtain an estimate of the root or
roots of the expression.

Newton's method is an iterative technique that can produce good estimates of solutions to
such equations. The method is employed by rewriting the equation in the form $f(x) = 0$,
then successively manipulating guesses for $x$ until the function evaluates to a value close
enough to zero for the modeler to accept.

The figure above is a graph of some function whose intercept with the *x-axis* is unknown. The goal
of Newton's method is to find this intersection (root) from a realistic first guess. Suppose
the first guess is $x_1$, shown on the figure as the right-most specific value of $x$. The value
of the function at this location is $f(x1)$. Because $x_1$ is supposed to be a root the difference
from the value zero represents an error in the estimate. Newton's method simply provides a
recipe for corrections to this error.

Provided $x_1$ is not near a minimum or maximum (slope of the function is not zero) then a
better estimate of the root can be obtained by extending a tangent line from $x_1, f(x_1)$ to
the x-axis. The intersection of this line with the axis represents a better estimate of the
root.  This new estimate is `x2`. A formula for $x_2$ can be derived from the geometry of the triangle
$x_2,f(x_1),x_1$. Recall from calculus that the tangent to a function at a particular point is
the first derivative of the function. Therefore, from the geometry of the triangle and the
definition of tangent we can write,

$\begin{equation}
tan(\theta)=\frac{df}{dx}\Biggr\vert_{x_1} = \frac{f(x_1)}{x_1 - x_2}
\end{equation}$

Solving the equation for $x_2$ results in a formula that expresses $x_2$ in terms of the first guess
plus a correction term.

$\begin{equation}
x_2=x_1 - \frac{f(x_1)}{\frac{df}{dx}\vert_{x_1}}
\end{equation}$

The second term on the right hand side is the correction term to the estimate on the right
hand side. Once $x_2$ is calculated we can repeat the formula substituting $x_2$ for $x_1$ and $x_3$ for
$x_2$ in the formula. Repeated application usually leads to one of three outcomes:

1. a root;
2. divergence to +/- $\inf$ ; or
3. cycling.

These three outcomes are discussed below in various subsections along with some remedies.
The generalized formula is

$\begin{equation}
x_{k+1}=x_{k} - \frac{  f(x_{k})  }{   \frac{df}{dx}\rvert_{x_k} }
\label{eqn:NewtonFormula}
\end{equation}$

If the derivative is evaluated using analytical derivatives the method is called Newton's
method, if approximations to the derivative are used, it is called a quasi-Newton method.

## Newton's Method --- Using analytical derivatives
This subsection is an example in Python of implementing Newton's method with analytical derivatives.

The recipe itself is:
1) Write the function in proper form, and code it into a computer.
2) Write the derivative in proper form and code it into a computer.
3) Make an initial guess of the solution (0 and 1 are always convenient guesses).
4) Evaluate the function, evaluate the derivative, calculate their ratio.
5) Subtract the ratio from the current guess and save the result as the update.
6) Test for stopping:
     i) Did the update stay the same value? Yes, then stop, probably have a solution.
    ii) Is the function nearly zero?  Yes, then stop we probably have a solution.
    iii) Have we tried too many updates? Yes, then stop the process is probably cycling, stop.
7) If stopping is indicated proceed to next step, otherwise proceed back to step 4.
8) Stopping indicated, report last update as the result (or report failure to find solution), and related information about the status of the numerical method.

The following example illustrates these step as well as an ipython implementation of Newton's method.

Suppose we wish to find a root (value of `x`) that satisfies:

$\begin{equation}
f(x) = e^x - 10 cos(x) -100
\end{equation}$

Then we will need to code it into a script.   Here is a code fragment that will generate the prototype function

```python
#  import built in function for e^x, cosine
from math import exp, cos, sin
# Define the function
def func(x):
    func = exp(x) - 10*cos(x) - 100  #using the name as the temp var
    return func
```

Notice in the code fragment we import three built-in functions from the Python math package, specifically $\exp()$, $\sin()$, and $\cos ()$.
The next step is to code the derivative.   In this case the derivative is

$\begin{equation}
\frac{df}{dx}\vert{(x)} = e^x + 10 \sin(x)
\end{equation}$

and the prototype function is coded as

```python
def dfdx(x):
    dfdx = exp(x) + 10*sin(x)
    return dfdx
```

Next we will need script to read in an initial guess, and ask us how many trials we will use to try to find a solution, as well as how close to zero we should be before we declare victory.

```{note}
The script below is a rudimentary interactive interface to read in data.  The examples use assigned inputs, but can be easily replaced with a script like the one below
```

```
# Now for the Newton Method Implementation
# Get initial guess, use a simple error trap
yes=0
while yes == 0:
    xnow = input("Enter an initial guess for Newton method \n")
    try:
        xnow = float(xnow)
        yes =1
    except:
        print ("Value should be numeric, try again \n")
# Get number trials, use a simple error trap
yes=0
while yes == 0:
    HowMany = input("Enter iteration maximum \n")
    try:
        HowMany = int(HowMany)
        yes =1
    except:
        print ("Value should be numeric, try again \n")
# Get stopping criterion
yes=0
while yes == 0:
    HowSmall = input("Enter a solution tolerance (e.g. 1e-06) \n")
    try:
        HowSmall= float(HowSmall)
        yes =1
    except:
        print ("Value should be numeric, try again \n")
```

```python
xnow = 1.0
HowMany=40
HowSmall=1e-06
```

The use of `HowSmall` is called a zero tolerance.
We will use the same numerical value for two tolerance tests.
Also notice how we are using error traps to force numeric input.
Probably overkill for this example, but we already wrote the code in an earlier essay, so might as well reuse the code.
Professional codes do a lot of error checking before launching into the actual processing - especially if the processing part is time consuming, its worth the time to check for obvious errors before running for a few hours then at some point failing because of an input value error that was predictable.

Now back to the tolerance tests.
The first test is to determine if the update has changed or not.
If it has not, we may not have a correct answer, but there is no point continuing because the update is unlikely to move further.
The test is something like

$\begin{equation}
\text{IF}~\lvert x_{k+1} - x_{k} \rvert < \text{Tol.~ THEN Exit and Report Results}
\end{equation}$

The second test is if the function value is close to zero.
The structure of the test is similar, just an different argument.   The second test is something like

$\begin{equation}
\text{IF}~\lvert f(x_{k+1}) \rvert < \text{Tol.~ THEN Exit and Report Results}
\end{equation}$

One can see from the nature of the two tests that a programmer might want to make the tolerance values different.
This modification is left as a reader exercise.

Checking for maximum iterations is relatively easy, we just include code that checks for normal exit the loop.

Here is code fragment that implements the method, makes the various tests, and reports results.

```python
# now we begin the process
count = 0
for i in range(0,HowMany,1):
    xnew = xnow - func(xnow)/dfdx(xnow)
# stopping criteria -- update not changing
    if abs(xnew - xnow) < HowSmall:
        print ("Update not changing \n")
        print("Function value =",func(xnew))
        print(" Root value    =",xnew)
        break
    else:
        xnow = xnew
        count = count +1
        continue
# stopping criteria -- function close to zero
    if abs( func(xnew) ) < HowSmall:
        print ("Function value close to zero \n")
        print("Function value =",func(xnew))
        print(" Root value    =",xnew)
        break
    else:
        xnow = xnew
        count = count +1
        continue
# next step, then have either broken from the loop or iteration counted out
if count == HowMany:
    print(" Iteration Limit Reached ")
    print("Function value =",func(xnew))
    print(" Root value    =",xnew)
print("End of NewtonMethod.py ")
```

    Update not changing

    Function value = 1.4210854715202004e-14
     Root value    = 4.593209147284144
    End of NewtonMethod.py

Now we simply connect the three fragments, and we would have a working Python script that implements Newton's method for the example equation.  The example is specific to the particular function provided, but the programmer could move the two functions `func` and `dfdx` into a user specified module, and then load that module in the program to make it even more generic.   The next section will use such an approach to illustrate the ability to build a generalized Newton method and only have to program the function itself

### Newton's Method --- Using finite-differences to estimate derivatives
A practical difficulty in using Newton's method is determining the value of the derivative in cases where differentiation is difficult.
In these cases we can replace the derivative by a finite difference equation and then proceed as in Newton's method.

Recall from calculus that the derivative was defined as the limit of the difference quotient:

$\begin{equation}
\frac{df}{dx}\vert_{x} = \lim_{\Delta x \rightarrow 0}\frac{f(x + \Delta x) - f(x) }{\Delta x}
\end{equation}$

A good approximation to the derivative should be possible by using this formula with a small, but non-zero value for $\Delta x$.

$\begin{equation}
\frac{df}{dx}\vert_{x} \approx \frac{f(x + \Delta x) - f(x) }{\Delta x}
\end{equation}$

When one replaces the derivative with the difference formula the root finding method the resulting update formula is

$\begin{equation}
x_{k+1}=x_k - \frac{f(x_k) \Delta x}{f(x_k + \Delta x)-f(x_k)}
\end{equation}$

This root-finding method is called a quasi-Newton method.

Here is the code fragment that we change by commenting out the analytical derivative and replacing it with a first-order finite difference approximation of the derivative.  The numerical value $1e-06$ is called the step size ($\Delta x$)  and should be an input value (rather than built-in to the code as shown here) like the tolerance test values, and be passed to the function as another argument.

```python
# reset the notebook
%reset -f
```

```python
#  import built in function for e^x, cosine
from math import exp, cos, sin
# Define the function
def func(x):
    func = exp(x) - 10*cos(x) - 100  #using the name as the temp var
    return func
def dfdx(x):
#    dfdx = exp(x) + 10*sin(x)
    dfdx = (func(x + 1e-06) - func(x) )/ (1e-06)
    return (dfdx)
```

```python
xnow = 2.0
HowMany=33
HowSmall=1e-09
```

```python
# now we begin the process
count = 0
for i in range(0,HowMany,1):
    xnew = xnow - func(xnow)/dfdx(xnow)
# stopping criteria -- update not changing
    if abs(xnew - xnow) < HowSmall:
        print ("Update not changing \n")
        print("Function value =",func(xnew))
        print(" Root value    =",xnew)
        break
    else:
        xnow = xnew
        count = count +1
        continue
# stopping criteria -- function close to zero
    if abs( func(xnew) ) < HowSmall:
        print ("Function value close to zero \n")
        print("Function value =",func(xnew))
        print(" Root value    =",xnew)
        break
    else:
        xnow = xnew
        count = count +1
        continue
# next step, then have either broken from the loop or iteration counted out
if count == HowMany:
    print(" Iteration Limit Reached ")
    print("Function value =",func(xnew))
    print(" Root value    =",xnew)
print("End of NewtonMethod.py ")
```

    Update not changing

    Function value = 1.4210854715202004e-14
     Root value    = 4.593209147284144
    End of NewtonMethod.py

Pretty much the same result, but now we don't have to determine the analytical derivative.

## Packages

## References

## Laboratory XX

## Exercise Set XX


# Continuous Signals

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Communications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Elementary Operations

Operations like superposition, temporal shifting and scaling are used to construct signals with a more complex structure than the previously introduced [standard signals](standard_signals.ipynb). A set of elementary operations are introduced that are frequently used in signal processing.

### Superposition

The weighted superposition $x(t)$ of two signals $x_\text{A}(t)$ and $x_\text{B}(t)$ is given as

\begin{equation}
x(t) = A \cdot x_\text{A}(t) + B \cdot x_\text{B}(t)
\end{equation}

with the complex weights $A, B \in \mathbb{C}$.

**Example**

The following example illustrates the superposition of two harmonic signals $x_\text{A}(t) = A \cdot \cos(\omega_\text{A} t)$ and  $x_\text{B}(t) = B \cdot \cos(\omega_\text{B} t)$ with weights $A$, $B$ and angular frequencies $\omega_\text{A}$ and $\omega_\text{B}$.

```python
import sympy as sym
sym.init_printing()

t = sym.symbols('t', real=True)

A = .3
omA = 3
B = .5
omB = 5

x = A*sym.cos(omA*t) + B*sym.cos(omB*t)

sym.plot(x, (t, -5, 5), ylim=[-1.2, 1.2], ylabel=r'$x(t)$');
```

### Temporal Shift

The temporal shift of a signal $x(t)$ by the time $\tau$ is a frequently applied operation in signal processing. For instance, to model the propagation of signals from an actuator to a sensor.

The temporally shifted signal $x(t)$ is defined as

\begin{equation}
y(t) = x(t-\tau)
\end{equation}
with $\tau \in \mathbb{R}$. The signal $x(t)$ is

* shifted to the right (*delayed*) for $\tau > 0$
* shifted to the left (*leading*) for $\tau < 0$

**Example**

In order to illustrate the temporal shifting of signals, the construction of a staircase signal by a superposition of shifted [rectangular signals](standard_signals.ipynb#Rectangular-Signal) is considered

\begin{equation}
x(t) = \text{rect}\left(t - \frac{1}{2} \right) + \frac{2}{3} \cdot \text{rect}\left(t-\frac{3}{2} \right) + \frac{1}{3} \cdot \text{rect} \left(t-\frac{5}{2} \right)
\end{equation}

```python
rect = sym.Heaviside(t + 1/2) - sym.Heaviside(t - 1/2)
x = rect.subs(t, t-1/2) + 2/3*rect.subs(t, t-3/2) + 1/3*rect.subs(t, t-5/2)

sym.plot(x, (t, -1, 5), ylim=[-0.2, 1.2], ylabel='$x(t)$');
```

**Exercise**

* Add another step to the beginning of the staircase signal by modifying above example.

### Temporal Scaling

The temporal scaling of a signal $x(t)$ is defined as

\begin{equation}
y(t) = x(a \cdot t)
\end{equation}

with $a \in \mathbb{R}$. The signal $x(t)$ is

* stretched for $0 < a < 1$
* compressed for $a > 1$
* time-reversed and scaled for $a < 0$

An application of temporal scaling in signal processing is the adaption of the time scale for standard signals and the modeling of the [Doppler effect](https://en.wikipedia.org/wiki/Doppler_effect).

**Example**

The following example illustrates the temporal scaling of the staircase signal $y(t) = x(a \cdot t)$ introduced in the previous example. The original $x(t)$ is plotted in gray, the scaled signal $y(t)$ in blue. Here stretching is realized, such that $y(t)$ is twice as long as $x(t)$.

```python
a = 1/2
y = x.subs(t, a*t)

px = sym.plot(x, (t, -3, 7), ylim=[-0.2, 1.2],
              ylabel=r'$x(t)$', show=False, line_color='gray')
py = sym.plot(y, (t, -3, 7),
              ylim=[-0.2, 1.2], ylabel=r'$y(t)$', show=False)
py.extend(px)
py.show()
```

**Exercise**

* Modify above example such that the signal is compressed.
* Modify above example such that the signal is scaled and time reversed. What scaling factors `a` lead to  stretching/compression in this context?

### Temporal Flipping

The temporal flipping of a signal $x(t)$ is defined as

\begin{equation}
y(t) = x(\tau - t)
\end{equation}

for $\tau \in \mathbb{R}$. As $x(\tau - t) = x(- (t - \tau))$ the flipping operation can also be represented as a time-reversal of the signal $x(t)$ followed by a shift of $\tau$ of the reversed signal. For $\tau = 0$ this results in only a time-reversal of the signal.

The temporal flipping operation can be interpreted geometrically as a mirroring of the signal $x(t)$ at the vertical axis $t=\frac{\tau}{2}$.

**Example**

The following example illustrates the temporal flipping $y(t) = x(\tau - t)$ of the staircase signal $x(t)$ introduced before.

```python
tau = -1
y = x.subs(t, tau - t)

px = sym.plot(x, (t, -5, 5), ylim=[-0.2, 1.2],
              ylabel=r'$x(t)$', show=False, line_color='gray')
py = sym.plot(y, (t, -5, 5), ylim=[-0.2, 1.2], ylabel=r'$y(t)$', show=False)
py.extend(px)
py.show()
```

**Excercise**

* For what value $\tau$ does the flipped signal $y(t)$ start at $t=0$?
* Realize the temporal flipping by splitting it into two consecutive operations: (i) time-reversal and (ii) temporal shift.

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Continuous- and Discrete-Time Signals and Systems - Theory and Computational Examples*.