# Mass Action Law

## 1. Introduction

Let's assume a system of interacting species (chemicals, cells, genes, proteins, organisms...). The ```Law of Mass Action``` states that the change in the amount of a given species is proportional to the product of the amount of the interacting substances.  Lets define some species `A`, `B`, `C` and `D` that interact between them, so the amount of each one is allowed to change over time until the reaction comes to an end.

Some possible schemes of interaction are, for instance:
$$
\begin{align*}
A + B  &\overset{k_1}{\longrightarrow} C  \tag{1} \\
2A + B  &\overset{k_2}{\longrightarrow} C  \tag{2} \\
C  &\overset{k_3}{\longrightarrow} A + B   \tag{3} \\
AB + C  &\overset{k_4}{\longrightarrow} AC + B   \tag{4} \\
AB + CD  &\overset{k_5}{\longrightarrow} AC + BD   \tag{5} \\
\end{align*}$$

The speed of a given reaction (i.e., the change in the amount of a given reactant) is proportional to the amount of collisions (the amount of times that two given reactants crash into each other in the solution), and this rate of collisions is proportional to the amount of reactants in the solution. For instance, for reaction 1

```julia
using Plots
p1 =scatter(fill(rand(10),1), fill(rand(10),1), fill(rand(10),1), m=(10, 0.8, :blues),framestyle=:box, title="few collisions",label="A")
p1 =scatter!(fill(rand(10),1), fill(rand(10),1), fill(rand(10),1), m=(10, 0.8, :reds),framestyle=:box, title="few collisions",label="B")
p2 =scatter(fill(rand(50),1), fill(rand(50),1), fill(rand(50),1), m=(10, 0.8, :blues),framestyle=:box, title="Many collisions",label="A")
p2 =scatter!(fill(rand(50),1), fill(rand(50),1), fill(rand(50),1), m=(10, 0.8, :reds),framestyle=:box, title="Many collisions",label="B")
plot(p1, p2)
```

Therefore, the speed of each reaction above can be written as:

$$
\begin{align*}
speed~of~reaction~1~&\propto A \cdot B \tag{6} \\
speed~of~reaction~2~ &\propto A \cdot A \cdot B \tag{7} \\
speed~of~reaction~3 ~ &\propto C \tag{7} \\
speed ~of ~reaction~4 ~&\propto AB \cdot B    \tag{8} \\
speed ~of ~reaction~5 ~&\propto AB \cdot CD  \tag{9} \\
\end{align*}$$

The proportionality constant is the value of the reaction rates $k_i$, which is a measure of how many of the collisions are effecive, so:

$$
\begin{align*}
speed~of~reaction~1~&= k_1 \cdot A \cdot B \tag{10} \\
speed~of~reaction~2~ &= k_2 \cdot A \cdot A \cdot B \tag{11} \\
speed~of~reaction~3 ~ &= k_3 \cdot C \tag{12} \\
speed ~of ~reaction~4 ~&= k_4 \cdot AB \cdot B    \tag{13} \\
speed ~of ~reaction~5 ~&= k_5 \cdot AB \cdot CD  \tag{14}
\end{align*}$$

The equilibrium is reached when the speed is zero, i.e., when the reaction is finished because one of the reactants has been fully consumed. This statement is correct in irreversible reactions. On the contrary, in reversible reactions, the equilibrium is a more interesting concept. For instance, let's asumme a very simple equilibrium between two forms of a chemical:

$$
\begin{align*}
2 NO_2   &\overset{k_1}{\underset{k_2}{\longleftrightarrow}} N_{2}O_4 \tag{15} \\
\end{align*}
$$

This system is actually composed of two different reactions,

$$
\begin{align*}
2 NO_2   &\overset{k_1}{\longrightarrow}  N_{2}O_4 \tag{16} \\
N_{2}O_4  &\overset{k_2}{\longrightarrow} 2 NO_2 \tag{17}
\end{align*}
$$

and the speed of each reactant is

$$
\begin{align*}
speed~of~reaction~16~&= k_1 \cdot NO_2 \cdot NO_2 = k_1 \cdot NO_{2}^{2} \tag{18} \\
speed~of~reaction~17~ &= k_2 \cdot N_{2}O_4 \tag{19}
\end{align*}$$

At room temperature, it is impossible to have pure $NO_2$ or  $N_{2}O_4$, so the equilibrium cannot be defined as the point when no reactions are taking place, or the point where one of the rectats has been fully consumed. In these conditions, the equilibrium is better defined as the point in time when there is no change in the amount of the reactans. In other words, the equilibirum occurs when the speed of the two reactions is equal.
$$
\begin{align*}
k_1 \cdot NO_{2}^{2} = k_2 \cdot N_{2}O_4  \tag{20}
\end{align*}$$

Thererefore, in conditions of equilibrum in reversible reactions, the ratio $\frac{[NO_2]^2}{[N_{2}O_4]}$ is constant and proportional to the ratios between the reaction rates. Interestingly, this ratio is allways the same independently on the amount of reactats that you added in initially in the system. This ratio depends on the temperature, since the reactions rates depend on the temperature (the number of collisions depens on the temperature). Such a law of nature is called the Law of Mass Action or mass action law. Although the Law of Mass Action was initially derived in the context of chemical reactions, it can be used also to describe the equilibirum of any system of interacting entities. We can define an equilibrium constant $K_{eq}$ as the ratio between the final concentrations of the two reactants $\frac{[NO_2]^2}{[N_{2}O_4]}$. The value of the equilibrium constant is then
$$
\begin{align*}
K_{eq}=\frac{k_2}{k_1} \tag{21}
\end{align*}$$

```julia
using Plots
NO2= 0.1*LinRange(0,1,100)
K_eq=0.3
plot(NO2,K_eq*NO2.^2)
title!("Equilibrium concentrations")
xlabel!("NO_2 [Mol]")
ylabel!("N_2O_4 [Mol]")
```

## 2.General Formulation of the Mass Action Law
We introduce the mass action law by using a general chemical reaction equation in which species `A` and `B` react to give species `C` and `D`

$$
\begin{align*}
aA + bB  &\overset{k_1}{\underset{k_2}{\longleftrightarrow}} cC + dD \tag{22} \\
\end{align*}$$

where `a`, `b`, `c`, `d` are the stoichiometric coefficients for a balanced interaction. At any instant in time during the course of the reaction, we can define a relation between the values of each concentration:

$$
Q= \frac{[C]^c[D]^d}{[A]^a[B]^b} \tag{23}
$$
where `Q` is defined as the reaction quotient, and measures the relative amounts of the interacting species present during a reaction at a particular point in time. As the reaction takes place and the system moves towards equilibrium, the value of `Q` gradually approaches to the equilibrium constant $K_{eq}$. The Mass Action Law states that if the system is at its equilibrium, then the following ratio is a constant:

$$
K_{eq}= \frac{[C]^c[D]^d}{[A]^a[B]^b} \tag{24}
$$

The values of `[A]`, `[B]`, `[C]` and `[D]` represent the amount of each species at equilibirum. If this amount is given as a concentration, lets say in moles/volume, `M`, the units of the equilibrium constant $K_{eq}$ are:

$$
[K_{eq}]=M^{c+d-a-b} \tag{25}
$$

For instance, lets consider the following reaction

$$NaCO_3 + CaCl_2 \overset{k_1}{\underset{k_2}{\longleftrightarrow}} CaCO_3 + 2 \cdot NaCl \tag{26}\\  $$
The equilibrium constant is

$$
K_{eq}= \frac{[CaCO_3][NaCl]^2}{[NaCO_3][CaCl_2]} \tag{27}\\
$$

and the units are

$$
[K_{eq}]=M^{1+2-1-1}=M  \tag{28}\\
$$

If we can measure the final concentration at equilibrium of each species, for instance:

```julia
NaCO_3=2;
CaCl_2=0.5;
CaCO_3=2;
NaCl=1.2;
```

Then the equilibrium constant can be calculated as:

```julia
K_eq=(CaCO_3*NaCl^2)/(NaCO_3*CaCl_2)
```

    2.88

In conclusion, the ``Law of Mass Action`` says that the ratio between the concentrations of the reactants at equilibrium depends only on the ratio between the kinetic constants, and it is independent on the initial concentration of the reactants. To illustrate that, we will solve the reaction and plot the value of ``Q(t)``

```julia
using DifferentialEquations
using ParameterizedFunctions
```

```julia
simpleODE3! = @ode_def abetterway begin
  da = -k1 * a * b + k2 * c * d^2
  db = -k1 * a * b + k2 * c * d^2
  dc = k1 * a * b - k2 * c * d^2
  dd = 2 * k1 * a * b - 2 * k2 * c * d^2
    end k1 k2
```

    (::abetterway{var"#19#23",var"#20#24",var"#21#25",Nothing,Nothing,var"#22#26",Expr,Expr}) (generic function with 2 methods)

```julia
k1=2.3e0;  # units 1/(Ms)
k2=2.5e0;  # units 1/(M M s)

p = (k1,k2)

a₀=0.02; # units (M)
b₀=0.01; # units (M)
c₀=0; # units (M)
d₀=0; # units (M)

u₀=[a₀,b₀,c₀,d₀];
```

```julia
tspan = (0.0,300.0)
prob = ODEProblem(simpleODE3!,u₀,tspan,p)
sol = solve(prob)
plot(sol,title= "Solution of differential equations",xlabel="Time [s]",ylabel="Concentration [M]")
```

```julia
plot(sol.t,(sol[3,:].*sol[4,:].^2)./(sol[1,:].*sol[2,:]),label="Q",title= "Plot of the quotient coefficient",xlabel="Time [s]",ylabel="Q [M]")
```

Now we solve the system again but with different initial conditions. We see that, although the solution is very different, the value of $K_eq$ (the value of $Q$ at equilibrium) is maintained.

```julia
k1=2.3e0;  # units 1/(Ms)
k2=2.5e0;  # units 1/(M M s)

p = (k1,k2)

a₀=0.05; # units (M)
b₀=0.01; # units (M)
c₀=0; # units (M)
d₀=0; # units (M)

u₀=[a₀,b₀,c₀,d₀];

tspan = (0.0,300.0)
prob = ODEProblem(simpleODE3!,u₀,tspan,p)
sol = solve(prob)
plot(sol,title= "Solution of the differential equations",xlabel="Time [s]",ylabel="Concentration [M]")
```

```julia
plot(sol.t,(sol[3,:].*sol[4,:].^2)./(sol[1,:].*sol[2,:]),label="Q",title= "Plot of the quotient coefficient",xlabel="Time [s]",ylabel="Q [M]")
```

## 3. Matrix Formulation for systems of interactions

Sometimes, it is convenient to write down the Mass Action Law as a vector-matrix formulation. This will be useful when dealing with large numbers of species and or reactions. Consider the familiar reaction:

$$ 2H_2 + O_2 \overset{k}{\longrightarrow}  2 H_2O \tag{29}$$

Eq. 12 can be rewritten in a more general form as

 $$ 2 X_1 + X_2 \overset{k}{\longrightarrow}  2 X_3 \tag{30}$$

where $X_1$, $X_2$, and $X_3$ are the species $H_2$, $O_2$, and $H_2O$,

The first step to write the system as a matrix form, we place all reactants at both sides of the reaction.

  $$ 2 X_1 + X_2  + 0 X_3 \overset{k}{\longrightarrow}  0 X_1 +  0 X_2  + 2 X_3 \tag{31}$$

The values in front of each number are the stoichimetric coefficients, which state the number of molecules that participate in each reaction. We can write these stoichiometric coefficiens as

$$ A_1 X_1 + A_2 X_2  + A_3 X_3 \overset{k}{\longrightarrow}  B_1 X_1 +  B_2 X_2  + B_3 X_3 \tag{32}$$

$A_1$ = 2, $A_2$ = 1, $A_3$ = 0, $B_1$ = 0, $B_2$ = 0, and $B_3$ = 2.

In compact form, the reaction can be written as:

$$
\sum_{j=1}^{3} A_j X_j \overset{k}{\longrightarrow}  \sum_{j=1}^{3} B_j X_j \tag{33}
$$

or in vector form:$ A X \overset{k}{\longrightarrow} B X  \tag{34}$

where `X` is the state vector (i.e, the vector with all reactants), and `A` and `B` are the matrices of stoichiometric coefficients

$$\begin{align*}
X&=[X_1,X_2,X_3]^T=[H_2,O_2,H_2O]^T \tag{35}\\
A&=[A_1,A_2,A_3]=[2,1,0] \tag{36}\\
B&=[B_1,B_2,B_3]=[0,0,2] \tag{37}\\
\end{align*}$$

To learn how to write reversible reactions as a matrix form, let's practice with the following reaction:

$$NaCO_3 + CaCl_2 \overset{k_1}{\underset{k_2}{\longleftrightarrow}} CaCO_3 + 2 \cdot NaCl \tag{38}\\  $$

We will decouple the reversible reaction into two irreversible steps:

$$\begin{align*}
NaCO_3 + CaCl_2 \overset{k_1}{\longrightarrow} CaCO_3 + 2 \cdot NaCl \tag{39}\\
CaCO_3 + 2 \cdot NaCl \overset{k_2}{\longrightarrow} NaCO_3 + CaCl_2 \tag{40}
\end{align*}$$

Using the same matrix notation as in Eq. 13:

$$\begin{align*}
X_1 + X_2 \overset{k_1}{\longrightarrow} X_3 + 2 \cdot X_4 \tag{41}\\
X_3 + 2 \cdot X_4 \overset{k_2}{\longrightarrow} X_1 + X_2 \tag{42}
\end{align*}$$

where $X_1 = Na_2CO_3$, $X_2  = Ca_2Cl$,$X_3  = CaCo_3$, $X_4 = NaCl$. Now we put all reactants at both sides of the reaction.

$$\begin{align*}
1 \cdot X_1 + 1 \cdot X_2  + 0 \cdot X_3  + 0 \cdot X_4 & \overset{k_1}  \longrightarrow  0 \cdot X_1+ 0 \cdot X_2 + 1 \cdot X_3 + 2\cdot  X_4 \tag{43}\\
 0 \cdot X_1+ 0 \cdot X_2 +1 \cdot X_3 + 2\cdot X_4 &\overset{k_2} \longrightarrow 1 \cdot X_1 + 1\cdot  X_2 + 0 \cdot X_3 + 0 \cdot X_4 \tag{44}
\end{align*}$$

in a more general form:
$$\begin{align*}
A_{11} X_1 + A_{12} X_2  + A_{13} + A_{14} X_4 &\overset{k_1}  \longrightarrow  B_{11}X_1+  B_{12} X_2 +  B_{13} X_3 + B_{14} X_4 \tag{45}\\
A_{21} X_1+ A_{22} X_2 +A_{23} X_3 + A_{24} X_4 & \overset{k_2} \longrightarrow B_{21}  X_1 + B_{22}  X_2 + B_{23} X_3 + B_{24} X_4 \tag{46}
\end{align*}$$

The numbers $A_{ij}$ are the stoichiometric coefficientes of the reactants, $B_{ij}$ are the stoichiometric coefficientes of the products.  Using the notation of Eq 16, these values are now the components of matrices `A` and `B`:

$$
A=\begin{bmatrix}
 1& 1 & 0 & 0  \\
 0 & 0 &1   &2
\end{bmatrix} ;
B=\begin{bmatrix}
0 & 0 &1   &2 \tag{47}\\
1 & 1 &0   &0
\end{bmatrix}
\\
$$

The numbers of rows $i= 1...r$, corresponds to the numbert of reactions `r` involved. The number of columns $j= 1...s$ corresponds to the number of species involved `s`. In conclusion, when we want to write the mass action law for `r` reactions, `k` is now also a vector with components:

$$k=[k_1,k_2,...k_r]^T \tag{48}$$

altogether we have

$$
\sum_{j=1}^{s} A_{ij} X_j \overset{k}{\longrightarrow}  \sum_{j=1}^{s} B_{ij} X_j,\,\,\,\, \,\,\,\,\,\,\,\, \,\,\,\,  i=1...r\tag{49}
$$

### Exercise:
White a computer program that consists on a function that takes as input the matrices A and B and returns the value $(B-A)'$

## 4. Order of reaction and units of k

The Order of a Reaction refers to the power dependence of the rate on the concentration of each reactant. In brief, the order of the reaction indicates the correlation of its velocity to the amount of reactants.
- For a ``zero-order reaction``, the rate does not depend on the concentration of any species.
- For a ``first-order reaction``, the rate is dependent on the concentration of a single species.
- For a ``second-order reaction``, the rate is dependent on the square of the concentration of a single reactant, or two reactants.

In Mass-Action kinetics, we can define the reaction order of a given reaction $i =1…r$ as:

$$
\sum_{j=1}^{s} A_{ij} \tag{50}
$$

i.e., the reaction order corresponds to the sum of the stoichiometric coefficients of the species appearing in the reactant of the `i` reaction.
For example, in the reaction 20, using the matrix A defined in Eq 27,

$$
A=\begin{bmatrix}
 1& 1 & 0 & 0  \\  \tag{51}
 0 & 0 &1  &2
\end{bmatrix} ;
$$

The forward reaction `i=1` has an order or 2, the backward reaction `i=2` has an order of 3. The units of the kinetic constant `k_i` of each reaction `i` can be written in terms of the order of the reaction:

$$[k_i]=\frac{M^{1-\sum_{j=1}^{s} A_{ij}}}{s} \tag{52}$$

```julia
A=([1,1,0,0],[2,1,0,0])
println("The order of the 1st reaction is $(sum(A[1]))")
println("The order of the 2nd reaction is $(sum(A[2]))")

println("The Units of k1 are s^-1 . k^$(1-(sum(A[1])))")
println("The Units of k2 are s^-1 . k^$(1-(sum(A[2])))")
```

    The order of the 1st reaction is 2
    The order of the 2nd reaction is 3
    The Units of k1 are s^-1 . k^-1
    The Units of k2 are s^-1 . k^-2

### Exercise:
White a computer program that consists on a function that takes as input the matrices A and B for a given interaction scheme and returns: the value $(B-A)'$, the order of each reaction and the units of the rate constants.

## 5. Differential Form of Mass Action

Now that we now how to obtain the stoichiometric matrices and the order of any given reaction, the next step is to write the differential equations.  We will use now the same formation for general interaction of eq.5 but in vector notation:

$$\begin{align*}
X_1 + X_2 \overset{k}{\longrightarrow} X_3  \tag{53}\\
\end{align*}$$
The stoichometrix matrix is:

```julia
A=([1,1,0])
println("The order of the 1st reaction is $(sum(A))")
println("The Units of k are s^-1 . k^$(1-(sum(A)))")
```

    The order of the 1st reaction is 2
    The Units of k are s^-1 . k^-1

We will write down the differential equations that account for the dynamics of each species.
$$
\begin{align}
\frac{\mathrm{d} X_1}{\mathrm{d} t}  &=&  \tag{54}\\
\frac{\mathrm{d} X_2}{\mathrm{d} t} &=& \tag{55}\\
\frac{\mathrm{d} X_3}{\mathrm{d} t} &=&  \tag{56}
\end{align}
$$

The Law of Mas Action states that the rate of change of any reactant is proportional to the concetrations of the reactants involved. The velocity depends on the affinity of the reaction and the concentrations of each of the reactants (what is before the arrow). The affinity of the reactions is defined by the rate constant `k` of the reaction. The `k` is the proportionallyty constant that correlates the speed fo the reaction with the concentrations of the species involved. In this case is $X_1$ and $X_2$:

$$
\begin{align}
\frac{\mathrm{d} X_1}{\mathrm{d} t}  &= - k X_1 X_2 \tag{57}\\
\frac{\mathrm{d} X_2}{\mathrm{d} t} &=- k X_1 X_2 \tag{58}\\
\frac{\mathrm{d} X_3}{\mathrm{d} t} &=  k X_1 X_2\tag{59}
\end{align}
$$

Lets see if the units are correct:

$$
\begin{align}
\frac{[M]}{[s]}  &= - \frac{1}{[s][M]}[M] [M] \tag{60}\\
\frac{[M]}{[s]}  &= - \frac{1}{[s][M]}[M] [M] \tag{61}\\
\frac{[M]}{[s]}  &=  \frac{1}{[s][M]}[M] [M] \tag{62}\\
\end{align}
$$

For the case where more than one molecule of reactant is involved, we will use the reaction of formation of water.

```julia
A=[2 1 0]
B=[0 0 2]
stoichiometric_matrix= (B-A)'
println("The order of the reaction is $(sum(A))")
println("The units of k are s^-1 . k^$(1-(sum(A)))")
println("The stoichiometric matrix is $stoichiometric_matrix ")
```

    The order of the reaction is 3
    The units of k are s^-1 . k^-2
    The stoichiometric matrix is [-2; -1; 2]

we can decompose the equation as
 $$  X_1 +  X_1 + X_2 \overset{k}{\longrightarrow}  X_3 + X_3 \tag{63}$$
We will write down the differential equations that account for the dynamics of each species. The speed of each reaction depends on the concentration of the species involved (what is before the arrow). In this case is two units of $X_1$ and one unit of $X_2$. In addition, since the reaction consumes 2 units of $X_1$ for each unit of $X_2$, the rate of consumption of $X_1$ is twice the rate of consumption $X_2$. Another more automaticic way of doing it is to use teh matrix of stochoimetric coefficients $(B-A)'$. Therefore:
$$
\begin{align}
\frac{\mathrm{d} X_1}{\mathrm{d} t}  &= - 2k X_1^2 X_2 \tag{64}\\
\frac{\mathrm{d} X_2}{\mathrm{d} t} &=- k X_1^2 X_2 \tag{65}\\
\frac{\mathrm{d} X_3}{\mathrm{d} t} &=2  k X_1^2 X_2 \tag{66}
\end{align}
$$
Lets check if units are correct:
$$
\begin{align}
\frac{[M]}{[s]}  &=& - \frac{1}{[s][M]^2} [M]^2 [M] \tag{67}\\
\frac{[M]}{[s]}  &=& - \frac{1}{[s][M]^2} [M]^2 [M] \tag{68}\\
\frac{[M]}{[s]}  &=& \frac{1}{[s][M]^2} [M]^2 [M] \tag{69}
\end{align}
$$

## 6. General formulation of differential form of Mass action

When we have to write the differenetial equations of more complex systems of interactions, such as Eq 21, we can use a general method based omn the following equation:
$$
\begin{align}
\frac{\mathrm{d} X}{\mathrm{d} t}&= (B-A)^T \cdot K \cdot X^A \tag{70}\\
\end{align}
$$
where  `A` and `B` are the matrices with the stoichiometric coefficients, `X` is the state vector $X=[X_1,X_2,X_3]^T$, and `K` is a matrix in the form:

$$
K=\begin{pmatrix}
 k_1 & 0   & 0  & \dots & 0 \\
 0 &  k_2  & 0  & \dots & 0\\
 0 & 0  & k_3 &\dots  &0 \\
 \vdots &  \vdots   &  \vdots  &  \vdots  &  \vdots  \tag{71}\\
 0 & 0  & 0  &  \dots & k_r
\end{pmatrix}
$$

The calculation of $X^A$ is perfomed as:

$$X^A=\begin{pmatrix}
X_1^{A_{11}} \cdot X_2^{A_{12}} \cdot X_3^{A_{13}} \cdot  ... \cdot  X_s^{A_{1s}} \\
X_1^{A_{21}} \cdot X_2^{A_{22}} \cdot X_3^{A_{23}} \cdot  ... \cdot  X_s^{A_{2s}} \\
 \vdots  \,\,\,\,\,\,\,\,\,\,\,  \vdots  \,\,\,\,\,\,\,\,\,\,\,\,    \vdots \,\,\,\,\,\,\,\,\,   \vdots  \,\,\,\,\,\,\,\,   \vdots  \\
X_1^{A_{r1}} \cdot X_2^{A_{r2}} \cdot X_3^{A_{r3}} \cdot  ... \cdot  X_s^{A_{rs}} \\
\end{pmatrix} =\begin{pmatrix}
 \prod_{i=1}^{s} X_{i}^{A_{1i}}\\
\prod_{i=1}^{s} X_{i}^{A_{2i}}\\
   \vdots  \tag{72} \\
\prod_{i=1}^{s} X_{i}^{A_{ri}}\\
\end{pmatrix}
$$

We will use as example the reaction in eq 21. This example has two reactions $r=2$ and four species $s=4$, with the stoichiometric matrices `A` and `B` listed in Eq. 30

```julia
A=[1 1 0 0; 0 0 1 2]
B=[0 0 1 2; 1 1 0 0]
stoichiometric_matrix= (B-A)'
println("The stoichiometric matrix is $stoichiometric_matrix ")
```

    The stoichiometric matrix is [-1 1; -1 1; 1 -1; 2 -2]

in our case

$$
K=\begin{pmatrix}
 k_1 & 0   \tag{73}\\
 0 &  k_2  \\
\end{pmatrix}
$$

and

$$X^A=\begin{pmatrix}
X_1\cdot X_2  \tag{74}\\
X_3\cdot X_4^2
\end{pmatrix}
$$

therefore

$$
\begin{align}
 \begin{bmatrix}
\frac{\mathrm{d} X_1}{\mathrm{d} t}\\ \frac{\mathrm{d} X_2}{\mathrm{d} t} \\ \frac{\mathrm{d} X_3}{\mathrm{d} t} \\ \frac{\mathrm{d} X_4}{\mathrm{d} t}
\end{bmatrix}&= \begin{bmatrix}
 -1   &  1 \\
    -1  &   1\\
     1  &  -1\\
     2  &  -2
\end{bmatrix}\begin{bmatrix}
k_1 & 0\\
 0& k_2
\end{bmatrix}\begin{bmatrix}
X_1 \cdot X_2\\
X_3 \cdot X_4^2
\end{bmatrix} \tag{75}
\end{align}
$$

so calculating

$$
\begin{align}
\begin{bmatrix}
\frac{\mathrm{d} X_1}{\mathrm{d} t}\\ \frac{\mathrm{d} X_2}{\mathrm{d} t} \\ \frac{\mathrm{d} X_3}{\mathrm{d} t} \\ \frac{\mathrm{d} X_4}{\mathrm{d} t}
\end{bmatrix}&= \begin{bmatrix}
 -1   &  1 \\
    -1  &   1\\
     1  &  -1\\
     2  &  -2
\end{bmatrix}\begin{bmatrix}
k_1 \cdot  X_1 \cdot X_2\\
k_2 \cdot X_3 \cdot X_4^2
\end{bmatrix} \tag{76}
\end{align}
$$

for the first equation

$$
\begin{align}
\begin{bmatrix}
\frac{\mathrm{d} X_1}{\mathrm{d} t}\\ \frac{\mathrm{d} X_2}{\mathrm{d} t} \\ \frac{\mathrm{d} X_3}{\mathrm{d} t} \\ \frac{\mathrm{d} X_4}{\mathrm{d} t}
\end{bmatrix}&= \begin{bmatrix}
 \mathbf{-1}   &  1 \\
    -1  &   1\\
     1  &  -1\\
     2  &  -2
\end{bmatrix}\begin{bmatrix}
\mathbf{k_1 \cdot  X_1 \cdot X_2}\\
k_2 \cdot X_3 \cdot X_4^2
\end{bmatrix} \tag{77}
\end{align}
$$

so, it is basically the Stoichiometric matrices multiplied by the vector of speeds of the reactions involved. This way, the final set of equations is:

$$\begin{align}
            \frac{ \mathrm{d}X_1 }{\mathrm{d}t} &= - k_1 \cdot X_1 \cdot X_2 + k_2 \cdot X_3 \cdot X_4^2  \tag{78}\\
            \frac{ \mathrm{d}X_2 }{\mathrm{d}t} &= - k_1 \cdot X_1 \cdot X_2 + k_2 \cdot X_3 \cdot X_4^2   \tag{79}  \\
             \frac{ \mathrm{d} X_3 }{\mathrm{d}t} &=  k_1 \cdot X_1 \cdot X_2 - k_2 \cdot X_3 \cdot X_4^2  \tag{80}  \\
              \frac{ \mathrm{d} X_4 }{\mathrm{d}t} &= 2 k_1 \cdot X_1  \cdot X_2 - 2 k_2 \cdot X_3 \cdot X_4^2   \tag{81}  \\
\end{align}
$$

## 7. Mass Conservation Law

The law of conservation of mass or principle of mass conservation states that for any system closed to all transfers of matter and energy, the mass of the system must remain constant over time, as the system's mass cannot change, so quantity can neither be added nor be removed. Therefore, the quantity of mass is conserved over time. For example, in chemical reactions, the mass of the chemical components before the reaction is equal to the mass of the components after the reaction. Thus, during any chemical reaction and low-energy thermodynamic processes in an isolated system, the total mass of the reactants, or starting materials, must be equal to the mass of the products.

We can use this law to find relationships between species of a given reaction, in such a way that we can reduce the number of variables of the systems. The mathematical formulation of thsi law is the following:

Suppose we have a solution of the system of equations defined by the stoichiometric matrix in the form $C \cdot (B-A)^T =0$ such that $C$ is a vector with number of rows $j= 1...s$ equal to the number of species involved `s`, then the following is true:

$$
\begin{align}
 C_1 \cdot X_1 + C_2 \cdot X_2 + ... + C_s \cdot X_s = cte \tag{82}
\end{align}
$$

this equation is a mass conservation law of the reaction.

$$
\begin{align}
 C_1 \cdot X_1 + C_2 \cdot X_2 + ... + C_s \cdot X_s = cte \tag{83}
\end{align}
$$

this is true because, if we take the derivative of the last equation:
$$
\begin{align}
\frac{\mathrm{d} [C_1 \cdot X_1 + C_2 \cdot X_2 + ... + C_s \cdot X_s ]}{\mathrm{d} t}&= \frac{\mathrm{d} [C_1] \cdot X_1}{\mathrm{d} t} + \frac{\mathrm{d} [C_2] \cdot X_2}{\mathrm{d} t} +...+ \frac{\mathrm{d} [C_s] \cdot X_s}{\mathrm{d} t}  \tag{84}
\end{align}
$$
since $C$ is a vector of constant values we can write:
$$
\begin{align}
[C_1] \frac{\mathrm{d}   X_1}{\mathrm{d} t} + [C_2] \frac{\mathrm{d} X_2}{\mathrm{d} t} +...+ [C_s] \frac{\mathrm{d}  X_s}{\mathrm{d} t} = \begin{bmatrix}
C_1 & C_2 & ... & C_s \end{bmatrix} \begin{bmatrix}
\frac{\mathrm{d} X_1}{\mathrm{d} t}\\ \frac{\mathrm{d} X_2}{\mathrm{d} t} \\ \frac{\mathrm{d} X_3}{\mathrm{d} t} \\ \vdots  \\ \frac{\mathrm{d} X_s}{\mathrm{d} t} \tag{85}
\end{bmatrix}
\end{align}
$$
Now we know that, based on the general equation of the differentiatl form of mass action

$$
\begin{align}
 \begin{bmatrix}
C_1 & C_2 & ... & C_s \end{bmatrix} \begin{bmatrix}
\frac{\mathrm{d} X_1}{\mathrm{d} t}\\ \frac{\mathrm{d} X_2}{\mathrm{d} t} \\ \frac{\mathrm{d} X_3}{\mathrm{d} t} \\ \vdots  \\ \frac{\mathrm{d} X_s}{\mathrm{d} t} \end{bmatrix} =  \begin{bmatrix}
C_1 & C_2 & ... & C_s \end{bmatrix}  (B-A)^T \cdot K \cdot X^A \tag{86}
\end{align}
$$

we also know that  $C \cdot (B-A)^T =0$ by definition (vector C was a solution of the system of equations), so

$$
\begin{align}
 \begin{bmatrix}
C_1 & C_2 & ... & C_s \end{bmatrix} \begin{bmatrix}
\frac{\mathrm{d} X_1}{\mathrm{d} t}\\ \frac{\mathrm{d} X_2}{\mathrm{d} t} \\ \frac{\mathrm{d} X_3}{\mathrm{d} t} \\ \vdots  \\ \frac{\mathrm{d} X_s}{\mathrm{d} t} \end{bmatrix} =  0 \cdot K \cdot X^A \tag{87}  = 0
\end{align}
$$

therefore
$$
\begin{align}
 C_1 \cdot X_1 + C_2 \cdot X_2 + ... + C_s \cdot X_s = cte \tag{88}
\end{align}
$$

## 8. Example of Mass Conservation Law

Lets now apply the theory to find conservation laws in the reaction of the example.

$$NaCO_3 + CaCl_2 \overset{k_1}{\underset{k_2}{\longleftrightarrow}} CaCO_3 + 2 \cdot NaCl \tag{89}\\  $$

we have to find  $C \cdot (B-A)^T =0$ with $C$ being a vector with $j= 1...s$, being $s=4$.

$$
\begin{bmatrix}
 C_1   &  C_2  &  C_3  &  C_4 \end{bmatrix}\begin{bmatrix}
 -1   &  1 \\
    -1  &   1\\
     1  &  -1\\
     2  &  -2
\end{bmatrix}=0 \tag{90}
$$

So we have to solve this system of equations:

$$\begin{cases} -C_1 - C_2 + C_3 + 2 \cdot C_4 = 0 \\ C_1 + C_2 - C_3 - 2 \cdot C_4 = 0 \end{cases} \tag{91}$$

since the two reactions correspond to a reversible system, the second equation is simply the first one multiplied by $-1$. Therefore the solytion of the system is simply
$$
C_3 + 2 \cdot C_4 - C_2 = C_1 \tag{92}
$$

so, we obtain one equation and three unknown variables. If we give values for three of the constants ($C_{1 \dots 3}=1$) and calculate the other one, we obtain:

$$
\begin{align}
1 + 2 \cdot C_4 - 1 &= 1 \tag{93}\\
C_4&=\frac{1}{2}\tag{94}
\end{align}
$$

the following the equation for the mass conservation, we obtain:

$$
\begin{align}
 X_1 + X_2 + X_3 +\frac{X_4}{2}= cte \tag{95}
\end{align}
$$

Let's test if this is correct by solving numerically the system (we will learn later how to do this).

```julia
k1=2.3e0;  # units 1/(Ms)
k2=2.5e0;  # units 1/(M M s)

p = (k1,k2)

a₀=0.02; # units (M)
b₀=0.01; # units (M)
c₀=0; # units (M)
d₀=0; # units (M)

u₀=[a₀,b₀,c₀,d₀];
```

```julia
tspan = (0.0,300.0)
prob = ODEProblem(simpleODE3!,u₀,tspan,p)
sol = solve(prob)
plot(sol,title= "Solution of differential equations",xlabel="Time [s]",ylabel="Concentration [M]")
plot!(sol.t,sol[1,:].+sol[2,:].+sol[3,:].+0.5.*sol[4,:],label="conservation")
```

## 9. Conclusions

The law of mass action allows us to calculate the dynamcis of a set of interacting entitites of a system. It is the most widely used tool to study biologivcal systems from a quantitative predictive perspective. When possible, the mass conservation allows us to reduce the complexity of the system by reducing the number of independent variables, by providing us with dependencies between the variables of the system. This tool allows us to reduce the number of equations to solve.
This search for mass conservation laws becomes more useful once whe have systems of more that one reversible reactions.

## 10. Practical Exercices

-  Find the order of the reactions, the units of the kinetic rate constants, the mass conservation, and write the differential equations that govern the dynamcis of the following system:

$$ X_1  \overset{k_1}{\underset{k_2}{\longleftrightarrow}} X_2 \tag{96} $$

-  Find the order of the reactions, the units of the kinetic rate constants, the mass conservation, and write the differential equations that govern the dynamcis of the following system:
$$X_1\overset{k_1}{\longrightarrow}X_2\overset{k_2}{\longrightarrow}X_3 \tag{97}$$

-  Find the order of the reactions, the units of the kinetic rate constants, the mass conservation, and write the differential equations that govern the dynamcis of the following system:
$$
\begin{align}
 X_1 &\overset{k_1}{\longrightarrow} 2X_1  \tag{98} \\
 X_1 + X_2 &\overset{k_2}{\longrightarrow} 2X_2 \tag{99}\\
 X_2 &\overset{k_3}{\longrightarrow} 0 \tag{100}\\
 \end{align}
 $$


# "Cool Softplus Function Properties"
> "The y-intercept of the tangent of a softplus function is the entropy of a Bernoulli distribution with probability equal to its slope"

- toc: true
- branch: master
- badges: false
- comments: false
- categories: [probability]
- hide: false
- search_exclude: false
- image: images/blog_posts/cool_softplus.png
- redirect_from: blog/cool_softplus

## The Special Case, the Bernoulli Distribution

Suppose $\eta$ is the log odds, $\log\left(\frac{p}{1-p}\right)$ of some event that occurs with probability p, i.e., of a Bernoulli random variable with parameter $p$.

The local linearization of the softplus function, $s(\eta) = \log(1+\exp\eta)$ at any given value for $\eta$ has the following interesting properties:
- its slope is the corresponding probability $p$.
- its y-intercept is the entropy of the Bernoulli distribution that has probability $p$.

The first is true because $s^{\prime}(\eta) = \frac{1}{1 + \exp(-\eta)}$, the logistic sigmoid (inverse-log-odds) function.

The second can be found be found by expressing the entropy in terms of the log-odds:
\begin{align}
-p\log p - (1-p)\log (1-p)
&= -p (\log p - \log (1-p)) - \log (1-p) \\
&= -p (\log \frac{p}{1-p}) - \log (1-p) \\
&= s(\eta) - s^{\prime}(\eta)\cdot \eta \quad .
\end{align}

## Plots

The following plots show the entropy, via the y-intercept of the linearization of the softplus function, of 5 Bernoulli distributions.

```python
import matplotlib.pyplot as plt
import numpy as np

def softplus(eta):
    return np.log(1 + np.exp(eta))

def sigmoid(eta):
    return 1 / (1 + np.exp(-eta))

xs = np.linspace(-3, 3, 50)
etas = [-3, -2, 0.0, 2, 3]

fig, axes = plt.subplots(1, len(etas), figsize=(15, 4))
for plot_i, (eta, ax) in enumerate(zip(etas, axes)):
    ax.plot(xs, softplus(xs), label='softplus', c='C0')
    ax.plot(xs, sigmoid(eta) * xs + softplus(eta) - sigmoid(eta) * eta, label='linearization', c='C1')
    ax.axvline(x=0.0, c='k', alpha=0.3)
    ax.axhline(y=0.0, c='k', alpha=0.3)
    ax.axvline(x=0.0, ymin=0.3333, ymax=(softplus(eta) - sigmoid(eta) * eta + 1)/3, c='r')
    ax.scatter([0], [softplus(eta) - sigmoid(eta) * eta], c='r', s=80, label='entropy')
    ax.axhline(y=0.0, xmin=0.5, xmax=(eta + 3)/6, c='g')
    ax.scatter([eta], [0], c='g', s=80, label='log odds')
    ax.axvline(x=1.0, ymin=(softplus(eta) - sigmoid(eta) * eta + 1)/3, ymax=(softplus(eta + 1) - sigmoid(eta + 1) * (eta + 1) + 1)/3, c='r')
    ax.scatter([0], [softplus(eta) - sigmoid(eta) * eta], c='r', s=80, label='entropy')
    ax.set_xlim((-3, 3))
    ax.set_ylim((-1, 2))
    ax.set_xlabel('Log odds')
    if plot_i == 0:
        ax.legend()
```

## The General Case, Exponential Families

More generally, consider an [exponential family](https://en.wikipedia.org/wiki/Exponential_family). This is a parameterized distribution of the form:

$$ p(x; \eta) = \exp\left( \eta\cdot T(x) + k(x) - A(\eta) \right) \quad . $$

Here, $\eta$ is a vector of parameters, $T(x)$ is a vector of *sufficient statistics* for the distribution, $k(x)$ is called the carrier measure, and $A(\eta)$ is a normalizer that makes the distribution integrate to 1.

For some popular distributions the carrier measure is zero. This is true of the Gaussian distribution and the Bernoulli distribution. Then we have

$$ p(x; \eta) = \exp\left( \eta\cdot T(x) - A(\eta) \right) \quad , $$

and the entropy of the distribution is given by

\begin{align}
H(p) &= A(\eta) \int p(x; \eta) \text{dx} - \eta \int T(x) p(x; \eta) \text{dx} \\
&= A(\eta) - \eta \cdot \mathbb{E}_p\left[ T(x) \right] \quad .
\end{align}

One useful property of exponential families is that the gradient of the log normalizer $A$ is equal to the expected value of the sufficient statistics: $\mathbb{E}_p\left[ T(x) \right] = \nabla A(\eta)$, so the entropy is equal to:

\begin{align}
H(p) = A(\eta) - \eta \cdot \nabla A(\eta) \quad .
\end{align}

This means that if we locally linearize, at some value of the parameters $\eta$, the log normalizing function $A$ of an exponential family with carrier measure zero, the resulting hyperplane has slope equal to the expected value (with respect to the distribution) of the sufficient statistics, and has a y-intercept equal to the entropy of the distribution.


# Final project
The Allen–Cahn equation (after John W. Cahn and Sam Allen) is a reaction–diffusion equation of mathematical physics which describes the process of phase separation in multi-component alloy systems, including order-disorder transitions.

The equation describes the time evolution of a scalar-valued state variable $\eta$  on a domain $\Omega=[0,1]$  during a time interval $[0,T]$, and is given (in one dimension) by:

$$
\frac{\partial \eta}{\partial t} - \varepsilon^2 \eta'' + f'(\eta) = 0, \qquad \eta'(0, t) = \eta'(1, t) = 0,\qquad\eta(x,0) = \eta_0(x)
$$

where $f$ is a double-well potential, $\eta_0$ is the initial condition, and $\varepsilon$ is the characteristic width of the phase transition.

This equation is the L2 gradient flow of the Ginzburg–Landau free energy functional, and it is closely related to the Cahn–Hilliard equation.

A typical example of double well potential is given by the following function

$$
f(\eta) = \eta^2(\eta-1)^2
$$

which has two minima in $0$ and $1$ (the two wells, where its value is zero), one local maximum in $0.5$, and it is always greater or equal than zero.

The two minima above behave like "attractors" for the phase $\eta$. Think of a solid-liquid phase transition (say water+ice) occupying the region $[0,1]$. When $\eta = 0$, then the material is liquid, while when $\eta = 1$ the material is solid (or viceversa).

Any other value for $\eta$ is *unstable*, and the equation will pull that region towards either $0$ or $1$.

Discretisation of this problem can be done by finite difference in time. For example, a fully explicity discretisation in time would lead to the following algorithm.

We split the interval $[0,T]$ in `n_steps` intervals, of dimension `dt = T/n_steps`. Given the solution at time `t[k] = k*dt`, it i possible to compute the next solution at time `t[k+1]` as

$$
\eta_{k+1} = \eta_{k} + \Delta t \varepsilon^2 \eta_k'' - \Delta t f'(\eta_k)
$$

Such a solution will not be stable. A possible remedy that improves the stability of the problem, is to treat the linear term $\Delta t \varepsilon^2 \eta_k''$ implicitly, and keep the term $-f'(\eta_k)$ explicit, that is:

$$
\eta_{k+1} - \Delta t \varepsilon^2 \eta_k'' = \eta_{k} - \Delta t f'(\eta_k)
$$

Grouping together the terms on the right hand side, this problem is identical to the one we solved in the python notebook number 9, with the exception of the constant $\Delta t \varepsilon^2$ in front the stiffness matrix.

In particular, given a set of basis functions $v_i$, representing $\eta = \eta^j v_j$ (sum is implied), we can solve the problem using finite elements by computing

$$
\big((v_i, v_j) + \Delta t \varepsilon^2  (v_i', v_j')\big) \eta^j_{k+1} = \big((v_i, v_j) \eta^j_{k} - \Delta t (v_i, f'(\eta_k)\big)
$$
where a sum is implied over $j$ on both the left hand side and the right hand side. Let us remark that while writing this last version of the equation we moved from a forward Euler scheme to a backward Euler scheme for the second spatial derivative term: that is, we used $\eta^j_{k+1}$ instead of $\eta^j_{k}$.

This results in a linear system

$$
A x = b
$$

where

$$
A_{ij} = M_{ij}+ \Delta t \varepsilon^2 K_{ij} = \big((v_i, v_j) + \Delta t \varepsilon^2  (v_i', v_j')\big)
$$

and

$$
b_i = M_{ij} \big(\eta_k^j - \Delta t f'(\eta_k^j)\big)
$$

where we simplified the integration on the right hand side, by computing the integral of the interpolation of $f'(\eta)$.

```python
%reset -f
%pylab inline
pylab.rcParams['figure.figsize'] = (9.0, 6.0)
import sympy as sym
import scipy
from scipy.interpolate import *
from scipy.integrate import *
```

    Populating the interactive namespace from numpy and matplotlib

```python
def f(eta):
    return eta**2*(eta-1)**2
```

## Step 1

Write a finite element solver, to solve one step of the problem above, given the solution at the previous time step, using the same techniques used in notebook number 9.

In particular:

1. Write a function that takes in input a vector representing $\eta$, an returns a vector containing $f'(\eta)$. Call this function `F`.

* Derivative of given, $f(\eta) = \eta^2(\eta-1)^2$, double well potential function is:
$$ f'(\eta) = 2 \eta (\eta-1)^2 + \eta^2 2 (\eta-1) \\
= 4 \eta^3 -6 \eta^2 +2 \eta$$

```python
# Step 1.1 : Fill in with the derivative of the double well potential function

def F(eta):
    '''Derivative of f(eta)'''
    return 4*eta**3 - 6*eta**2 + 2*eta
```

```python
# plot of the derivative of double well potential function
N = 1025
x = linspace(0,1, N)
plt.title("$F(\Omega)$")
plt.xlabel("Domain")
plt.ylabel("$f'(\eta)$")
_ = plot(x,F(x))
```

2. Write a function that takes in input a vector of support points of dimension `ndofs` and the degree `degree` of the polynomial basis, and returns a list of basis functions (piecewise polynomial objects of type `PPoly`) of dimension `ndofs`, representing the interpolatory spline basis of degree `degree`

```python
# Step 1.2

def compute_basis_functions(support_points, degree):
    # Insert here what was in notebook 9
    ''' Compute the piecewise polynomial objects (taken from notebook 09)'''

    basis = []
    dbasis = []

    for i in range(len(support_points)):
        c = support_points*0
        c[i] = 1
        bi = PPoly.from_spline(splrep(support_points,c,k=degree))

        basis.append(bi)
        dbasis.append(bi.derivative(1))

    # after you have computed the basis, return it
    # return basis,
    # it is convinient to return the dbasis to use in solve_allen_cahan for computing matrices M,K
    return basis, dbasis
```

```python
# plot the basis functions

ndofs = 16
degree = 2

support_points = linspace(0,1, ndofs)

B = zeros((N,ndofs))

basis_func , _ = compute_basis_functions(support_points,degree)

for i in range(len(support_points)):
    B[:,i] = basis_func[i](x)

fig, axs = plt.subplots(2,figsize = (10,8))

# basis functions (degree 2)
_ = axs[0].plot(x,B)
# piecewise polynomial interpolation of our function
_ = axs[1].plot(x, f(x))
_ = axs[1].plot(support_points, f(support_points), 'ro')
_ = axs[1].plot(x, B.dot(f(support_points)))
```

3. Write a function that, given a piecewise polynomial object of type `PPoly` and a number `n_gauss_quadrature_points`, computes the vector of global_quadrature_points and global_quadrature_weights, that contains replicas of a Gauss quadrature formula with `n_gauss_quadrature_points` on each of the intervals defined by `unique(PPoly.x)`

```python
# Step 1.3

def compute_global_quadrature(basis, n_gauss_quadrature_points):
    '''Create a Gauss quadrature formula with n_gauss_quadrature_points, extract the intervals from basis (i.e., unique(basis.x)), and
    create len(x)-1 shifted and scaled Gauss quadrature formulas that can be used to integrate on each interval. Put all of these
    together, and return the result'''

    intervals = unique(basis[0].x) # Make sure every interval border is taken only once
    qp, w = numpy.polynomial.legendre.leggauss(n_gauss_quadrature_points+1) # and make sure we can integrate exactly the product of two basis functions
    qp = (qp+1)/2 # rescaling
    w /= 2

    # Now replicate these points and weights in all the intervals
    h = diff(intervals)
    global_quadrature = array([intervals[i]+h[i]*qp for i in range(len(h))]).reshape((-1,))
    global_weights = array([w*h[i] for i in range(len(h))]).reshape((-1,))

    # return global_quadrature, global_weights
    return global_quadrature, global_weights
```

4. Write a function that, given the basis and the quadrature points and weights, returns the two matrices $M$ and $K$

```python
# Step 1.4

def compute_system_matrices(basis, dbasis, global_quadrature, global_weights):
    '''Compute the matrices M_ij = (v_i, v_j) and K_ij = (v_i', v_j') and return them'''
    Bq = array([basis[i](global_quadrature) for i in range(len(basis))]).T
    dBq = array([dbasis[i](global_quadrature) for i in range(len(basis))]).T

    M = einsum('qi, q, qj', Bq, global_weights, Bq)
    K = einsum('qi, q, qj', dBq, global_weights, dBq)

    # return M, K
    return M, K
```

## Step 2

Solve the Allen-Cahan equation on the interval $[0,1]$, from time $t=0$ and time $t=1$, given a time step `dt`, a number of degrees of freedom `ndofs`, and a polynomial degree `k`.

1. Write a function that takes the initial value of $\eta_0$ as a function, eps, dt, ndofs, and degree, and returns a matrix of dimension `(int(T/dt), ndofs)`  containing all the coefficients $\eta_k^i$ representing the solution, and the set of basis functions used to compute the solution

```python
# Step 2.1

def solve_allen_cahan(eta_0_function, eps, dt, ndofs, degree):
    # put together all the above functions, loop over time, and produce the result matrix eta, containing the solution at all points
    ''' Forward Euler solution'''

    support_points = linspace(0,1, ndofs)
    basis, dbasis = compute_basis_functions(support_points,degree)
    Q, W = compute_global_quadrature(basis,degree+1)
    M, K = compute_system_matrices(basis, dbasis, Q, W)

    A = M + (dt*eps**2) * K
    steps = int(1/dt)
    time_interval = [i*dt for i in range(steps+1)]
    eta = zeros((len(time_interval),ndofs))             # the matrix that will store the results
    eta[0,:] = eta_k = eta_0_function(support_points)           # initial values
    for t in range(1,len(time_interval)):
        # b_i = M_{ij} \big(\eta_k^j - \Delta t f'(\eta_k^j)\big)
        b = M.dot((eta_k - dt*F(eta_k)))
        eta_k = linalg.solve(A,b)
        eta[t,:] = eta_k

    return eta, basis
```

2. Write a function that takes all the solutions `eta`, the basis functions, a stride number `s`, and a resolution `res`, and plots on a single plot the solutions $\eta_0$, $\eta_s$, $\eta_{2s}$, computed on `res` equispaced points between zero and one

```python
# Step 2.2

def plot_solution(eta, basis, stride, resolution):
    x = linspace(0,1,resolution)
    B = zeros((resolution, len(basis)))
    for i in range(len(basis)):
        B[:,i] = basis[i](x)

    n_t = shape(eta)[0]
    t = ['t ='+str(round((i/(n_t-1)),2)) for i in range(n_t)]

    for eta, label in zip(eta[::-stride], t[::-stride]):
        plot(x,eta.dot(B.T), label = label)

    _ = legend()
    _ = title("Solution")
    _ = xlabel('$x$')
    _ = ylabel('$y$')
```

```python
# initial function
def eta_0(x):
    return sin(2*pi*x)+1

eta, basis = solve_allen_cahan(eta_0,eps = 0.01, dt=0.025, ndofs=32, degree=1)
plot_solution(eta,basis,stride=10,resolution=1025)
```

## Step 3

Solve the problem for all combinations of

with $\eta_0 = \sin(2 \pi x)+1$.

Plot the final solution at $t=1$ in all cases. What do you observe? What happens when you increase ndofs and keep dt constant?

*Clarifying the plot structures:*

Belove you can find the plots for all combinations. Totally, there are 40 subplots and each subplot has three lines which represent the degree. When you move to the right, $ndofs$ increases and when you move to the belove $dt$ decreases.

```python
eps = [0.01, 0.001]

ndofs = [16, 32, 64, 128]

degree = [1, 2, 3]

dt = [0.25, 0.125, 0.0625, 0.03125, 0.015625]
```

```python
def plot_once(eta_0, eps, ndofs, degree:list, ax, dt, resolution):
    for j in degree:
        eta, basis = solve_allen_cahan(eta_0,eps = eps, dt=dt, ndofs=ndofs, degree=j)
        x = linspace(0,1,resolution)
        B = zeros((resolution, len(basis)))
        for k in range(len(basis)):
            B[:,k] = basis[k](x)

        ax.plot(x,eta[-1].dot(B.T),label = "degree = "+str(j))
        ax.legend(loc='lower left')
```

```python
# plotting for eps = 0.01
fig,ax=plt.subplots(nrows=len(dt),ncols=len(ndofs),figsize=(20,20))
for col,n in enumerate(ndofs):
    for row,d in enumerate(dt):
        ax[row,col].set_title("$ndofs$ = "+str(n)+" $dt$ = "+str(d))
        plot_once(eta_0, eps = eps[0], ndofs=n, degree = degree, dt = d,ax = ax[row,col],resolution=1025)
```

```python
# plotting for eps = 0.001
fig,ax=plt.subplots(nrows=len(dt),ncols=len(ndofs),figsize=(20,20))
for col,n in enumerate(ndofs):
    for row,d in enumerate(dt):
        ax[row,col].set_title("$ndofs$ = "+str(n)+" $dt$ = "+str(d))
        plot_once(eta_0, eps = eps[1], ndofs=n, degree = degree, dt = d,ax = ax[row,col],resolution=1025)
```

The solution converges when time step is small enough which is consistent with the solution method because Forward Euler is conditionally stable. For example, we can observe abnormal behaviours when $dt$=0.25. Increasing $ndofs$ and $degree$ makes the function smoother.

## Step 4 (Optional)

Instead of solving the problem explicitly, solve it implicitly, by using backward euler method also for the non linear term. This requires the solution of a Nonlinear problem at every step. Use scipy and numpy methods to solve the non linear iteration.

By using the backward euler method we have the following equations:
$$
\frac{\eta_{k+1} - \eta_{k}}{\Delta t} = \varepsilon^2 \eta''_{k+1} - f'(\eta_{k+1}) \\
\eta_{k+1} - \eta_{k} - \Delta t \varepsilon^2 \eta''_{k+1} + \Delta t f'(\eta_{k+1}) = 0
$$

By using the given basis functions, grouping the similar terms and taking all the variables to left hand side we have the following equation:

$$
\big((v_i, v_j) + \Delta t \varepsilon^2  (v_i', v_j')\big) \eta^j_{k+1} - \big((v_i, v_j) \eta^j_{k} - \Delta t (v_i, f'(\eta_{k+1})\big) = 0
$$

By using the same matrix notation $A_{ij} = M_{ij}+ \Delta t \varepsilon^2 K_{ij} = \big((v_i, v_j) + \Delta t \varepsilon^2  (v_i', v_j')\big)$ we have:

$$
A_{ij} \eta^j_{k+1} - M_{ij} \eta^j_{k} + \Delta t M_{ij} f'(\eta_{k+1}) = 0
$$

So for each step we have to find $\eta^j_{k+1}$ which makes above equation 0. To do this, `scipy` method [scipy.optimize.fsolve](https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.fsolve.html) is used which `returns the roots of the (non-linear) equations defined by func(x) = 0 given a starting estimate.`

```python
def solve_backward(eta_0_function, eps, dt, ndofs, degree):
    ''' Backward Euler solution'''

    support_points = linspace(0,1, ndofs)
    basis, dbasis = compute_basis_functions(support_points,degree)
    Q, W = compute_global_quadrature(basis,degree+1)
    M, K = compute_system_matrices(basis, dbasis, Q, W)
    A = M + (dt*eps**2) * K

    steps = int(1/dt)
    time_interval = [i*dt for i in range(steps+1)]

    eta = zeros((len(time_interval),ndofs))
    eta_k = eta_0_function(support_points)
    eta[0,:] = eta_k

    for t in range(1,len(time_interval)):
        eta_k = scipy.optimize.fsolve(lambda x: A.dot(x) - M.dot(eta_k) + dt*M.dot(F(x)),eta_k)
        eta[t,:] = eta_k

    return eta, basis
```

```python
plt.figure(figsize=(14,6))

plt.subplot(1, 2, 1)
eta, basis = solve_allen_cahan(eta_0,eps = 0.01, dt=0.25, ndofs=32, degree=1)
_ = plot_solution(eta,basis,stride=2,resolution=1025)
_ = title("Forward Euler Solution")

plt.subplot(1, 2, 2)
eta, basis = solve_backward(eta_0,eps = 0.01, dt=0.25, ndofs=32, degree=1)
_ = plot_solution(eta,basis,stride=2,resolution=1025)
_ = title("Backward Euler Solution")
```

Unlike forward euler method, backward euler method is unconditionally stable, including the large time steps. As we can observe from above plot, forward euler method shows abnormal behaviour for `dt = 0.25` whereas backward euler method is stable for any time step value.


# 10 Ordinary differential equations – Integrators
More about integrators to solve ODEs.

* Euler's Rule (and it's shortcomings)
* Runge-Kutta methods

## Euler's Rule

From the last lecture: Given the $n$-dimensional vectors from the ODE standard form

$$
\frac{d\mathbf{y}}{dt} = \mathbf{f}(t, \mathbf{y})
$$
the **Euler rule** amounts to

\begin{align}
\mathbf{f}(t, \mathbf{y}) = \frac{d\mathbf{y}(t)}{dt} &\approx \frac{\mathbf{y}(t_{n+1}) - \mathbf{y}(t_n)}{h}\\
\mathbf{y}_{n+1} &\approx \mathbf{y}_n + h \mathbf{f}(t_n, \mathbf{y}_n) \quad \text{with} \quad \mathbf{y}_n := \mathbf{y}(t_n)
\end{align}

Euler's rule cannot obtain high precision without very small $h$ but this leads to higher round-off error accumulation.

### Euler's Rule applied to the 1D harmonic oscillator (theory)

\begin{alignat}{3}
f^{(0)}(t, \mathbf{y}) &= y^{(1)},
    &\quad y^{(0)}_0 &= x_0,\\
f^{(1)}(t, \mathbf{y}) &=   - \frac{k}{m} y^{(0)},
    &\quad y^{(1)}_0 &= v_0.
\end{alignat}

with $k=1$; $x_0 = 0$ and $v_0 = +1$.

First step from $t=0$ to $t=h$:

\begin{align}
y^{(0)}_1 &= y^{(0)}_0 + h y^{(1)}_0 = x_0 + v_0 h\\
y^{(1)}_1 &= y^{(1)}_0 + h \left(-\frac{k}{m}y^{(0)}_0\right) = v_0 + \frac{-k x_0}{m} h
\end{align}

Compare to the equations of movement under a constant force (*ballistic motion*):

$$
x = x_0 + v_0 h + \frac{1}{2} a h^2, \quad v = v_0 + a h
$$

In Euler, the position does not even contain the $h^2$ term, i.e., the acceleration only contributes *indirectly* via the velocity of the *next* step. (Remember: Euler's error is $\mathcal{O}(h^2)$!)

### Euler's Rule: Application to 3 oscillators (numerical)

Open [integrators.py](https://github.com/ASU-CompMethodsPhysics-PHY494/PHY494-resources/blob/master/10_ODEs/integrators.py) to have a look at it: it contains code to integrate Newton's equations of motions for different forces and with different integrators (based on the work done in the last lecture).

We will complete `integrators.py` during the class together (when we need them), namely functions:
* `kinetic_energy()`
* `rk2()`
* `rk4()`

```python
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline
matplotlib.style.use('ggplot')

try:
    # if we have the solutions then use it
    import integrators_solution as integrators
except:
    # else ... sigh, we still have work to do
    import integrators
```

#### Integrate Newton's EOMs for different potentials with Euler
Use the `integrate_newton()` function (Euler is the default):

```python
t_max = 20
t_harm, y_harm = integrators.integrate_newton(t_max=t_max)
t_anharm, y_anharm = integrators.integrate_newton(t_max=t_max,
                                                  force=integrators.F_anharmonic)
t_power, y_power = integrators.integrate_newton(t_max=t_max,
                                                force=integrators.F_power)
```

```python
plt.plot(t_harm, y_harm[:, 0], label=r"harmonic $\frac{k}{2} x^2$")
plt.plot(t_anharm, y_anharm[:, 0], label=r"anharmonic $\frac{k}{2} x^2(1 - \frac{2}{3} \alpha x)$")
plt.plot(t_power, y_power[:, 0], label=r"power $\frac{k}{6} x^6$")
plt.legend(loc="best")
```

#### Phase-space portrait
Plot position against momentum ($x, p$):

```python
m = 1
x = y_harm[:, 0]
p = m * y_harm[:, 1]
ax = plt.subplot(1,1,1)
ax.plot(x, p)
ax.set_aspect(1)
```

```python
t, y = integrators.integrate_newton(t_max=40,
                                    force=integrators.F_harmonic,
                                    h=0.01)
t2, y2 = integrators.integrate_newton(t_max=40, h=0.001,
                                      force=integrators.F_harmonic)
```

```python
ax = plt.subplot(1,1,1)
ax.plot(y[:, 0], m * y[:, 1], 'b-', label=r"$h=0.01$")
ax.plot(y2[:, 0], m * y2[:, 1], 'r--', label="$h=0.001$")
ax.set_xlabel("position $x$")
ax.set_ylabel("momentum $p$")
ax.set_aspect(1)
ax.legend(loc="best")
```

#### Energy
Energy is conserved: stringent test for the integrator!

$$
E = T_\text{kin} + U\\
\frac{dE}{dt} = 0
$$
with $T_\text{kin} = \frac{1}{2} m v^2$.

Calculate and plot the three energy terms:

```python
KE = 0.5 * m * y_harm[:, 1]**2
PE = integrators.U_harmonic(y_harm[:, 0])
energy = KE + PE
plt.plot(t_harm, KE, 'r-', t_harm, PE, 'b-', t_harm, energy, 'k--')
```

Harmonic oscillator with larger timestep $h=0.01$

```python
m = 1
t, y = integrators.integrate_newton(t_max=500, force=integrators.F_harmonic, h=0.01, mass=m)
x, v = y.T
KE = 1/2 * m * v**2
PE = integrators.U_harmonic(x)
energy = KE + PE
```

```python
ax = plt.subplot(1,1,1)
ax.plot(t, KE, 'r-', t, PE, 'b-', t, energy, 'k--')
ax.set_xlim(0, 20)
ax.set_ylim(0, 0.7)
ax.set_xlabel("time")
ax.set_ylabel("energy")
ax.set_title("Euler: harmonic oscillator")
```

Note how energy conservation becomes worse when $h$ is increased.

For longer time scales, the energy just keeps on increasing — clearly, we are not solving the equations of motions in a physically meaningful way.

```python
ax.set_ylim(0, 80)
ax.set_xlim(0,500)
ax.figure
```

##### Quantify energy drift
Measure the relative departure from energy conservation

$$
\epsilon_E(t) = \log_{10}\left|\frac{E(t) - E(t=0)}{E(t=0)}\right|
$$

(By using the decadic logarithm, $-\epsilon_E$ is a direct measure of the precision.)

###### Harmonic oscillator
Analyze energy drift for the last harmonic oscillator example ($h=0.01$):

```python
energy_precision = np.log10(np.abs(energy/energy[0] - 1))
plt.plot(t, energy_precision)
plt.ylabel("log(relative error in $E$)")
plt.xlabel("time");
```

As we already know, with such a large time step, energy conservation is very bad with the *Euler* integrator: The relative (!) error quickly becomes 100% (after $t=200$).

### Code re-use: energy analysis as a function: Application to power law oscillator
The module `integrators` contains the function `integrators.analyze_energies(t, y, U)` to automate the task of analysing the energy. You *must* provide the appropriate potential energy function `U`.

##### Problem: Make `analyze_energies()` work and analyze the power law oscillator
The code is *incomplete*.
1. Open `integrators.py` in your editor.
2. Complete the `kinetic_energy()` function (remember: $T_\text{kin} = \frac{1}{2} m v^2$)

Then `reload()` the module:

```python
# when working on the integrator module you need to reload() whenever you make changes
from importlib import reload   # for Python 3
reload(integrators);
```

and try it out for power law oscillator with a large time step $h = 0.01$. Can you find a value of $h$ that will give you energy precision to 4th decimal over 40 time units?

###### Energy stability analysis of the 6-th power oscillator
$x^6$ potential with $h=0.01$:

```python
t, y = integrators.integrate_newton(t_max=40, force=integrators.F_power,
                                    h=0.01)
```

```python
integrators.analyze_energies(t, y, integrators.U_power)
```

We can also plot the total energy along the trajectory, $E(x)$, in relationship to the potential energy $U(x)$:

```python
x, v = y.T
E = integrators.kinetic_energy(v, m=1) + integrators.U_power(x)
```

```python
X = np.linspace(-1.6, 1.6, 100)
plt.plot(x, E, label=r"$E=T+V$")
plt.plot(X, integrators.U_power(X), label=r"$U(x)$")
plt.xlabel(r"position $x$")
plt.ylabel(r"energy")
plt.legend();
```

This looks terrible: the particle _should_ remain at a constant energy but instead its energy increases and it moves up in the potential.

Trying 100 times smaller $h$ (takes 100 times longer!!)

```python
t, y = integrators.integrate_newton(t_max=40, force=integrators.F_power,
                                    h=0.0001)
integrators.analyze_energies(t, y, integrators.U_power)
```

*Reducing $h$ further does not seem to help and takes order of magnitude longer to run*. **We need a better integrator!**

## Runge-Kutta methods
Explicit RK methods propagate with the general form

$$
y_{n+1} = y_n + h \sum_{i=1}^s b_i k_i, \quad t_{n+1} = t_n + h
$$

where $s$ is the number of *stages* and the coefficients $a_{ij}$, $b_i$ and $c_i$ define the method.

\begin{align}
k_1 &= f(t_n, y_n)\\
k_2 &= f(t_n + c_2 h, y_n + h(a_{21} k_1))\\
k_3 &= f(t_n + c_3 h, y_n + h(a_{31} k_1 + a_{32} k_2))\\
 \vdots& \\
k_s &= f(t_n + c_s h, y_n + h(a_{s1} k_1 + a_{s2} k_2 + \cdots + a_{s,s-1} k_{s-1})).
\end{align}

Basic idea: improve estimate for the slope by using weighted averages of slopes over the interval.

### RK2
`rk2` is simpler to derive and understand than `rk4` (which we will use more often).

Start with the exact solution

$$
\frac{dy}{dt} = f(t, y) \quad\text{and}\quad y(t) = \int f(t, y)dt\\
y_{n+1} = y_n + \int_{t_n}^{t_{n+1}} f(t, y)dt
$$

Expand $f(t, y)$ around *midpoint* $t_{n+1/2} = t + \frac{h}{2}$ of integration interval

\begin{gather}
f(t, y) \approx f(t_{n+1/2}, y_{n+1/2}) + (t - t_{n+1/2})\,  f^{(1)}(t_{n+1/2}) +
   \mathcal{O}(h^2)
\end{gather}

Integrate the *expansion* (Taylor series around midpoint):
\begin{align}
\int_{t_n}^{t_{n+1}} f(t, y) \, dt &\approx h\, f(t_{n+1/2}, y_{n+1/2}) + \mathcal{O}(h^3)\\
y_{n+1} &= y_n + h\, f(t_{n+1/2}, y_{n+1/2}) + \mathcal{O}(h^3)
\end{align}

I.e., use **slope at midpoint**! (The $\mathcal{O}(h^2)$ term in the integral is zero because $[(h/2)^2 - (h/2)^2] = 0$.)

But we don't have $y_{n+1/2}$ so we use Euler's rule to estimate it to $\mathcal{O}(h^2)$:

$$
y_{n+1/2} \approx y_n + \frac{1}{2} h\, f(t_n, y_n)
$$

and insert into $y_{n+1} = y_n + h\, f(t_{n+1/2}, y_{n+1/2})$ (so it is still $\mathcal{O}(h^3)$ accurate).

Altogether: **RK2** algorithm

$$
\mathbf{y}_{n+1} = \mathbf{y}_n + h \mathbf{k}_2, \quad t_{n+1} = t_n + h
$$

and

\begin{align}
\mathbf{k}_1 &= \mathbf{f}(t_n, \mathbf{y}_n)\\
\mathbf{k}_2 &= \mathbf{f}(t_n + \frac{1}{2} h, \mathbf{y}_n + \frac{1}{2} h \mathbf{k}_1)
\end{align}

with $\mathcal{O}(h^3)$ accuracy.

#### RK2 applied to the 1D harmonic oscillator (theory)

\begin{alignat}{3}
f^{(0)}(t, \mathbf{y}) &= y^{(1)},
    &\quad y^{(0)}_0 &= x_0,\\
f^{(1)}(t, \mathbf{y}) &=   - \frac{k}{m} y^{(0)},
    &\quad y^{(1)}_0 &= v_0.
\end{alignat}

with $k=1$; $x_0 = 0$ and $v_0 = +1$.

First step from $t=0$ to $t=h$:

\begin{align}
  y^{(0)}_{1} &=
    y^{(0)} _{0} + hf^{(0)} \Big(\frac{h}{2}, y^{(0)}_{0} + hk_{1} \Big)\\
    &= x_{0} + h\left[v_{0} + \frac{h}{2} \left(-\frac{k}{m} x_{0}\right)  \right]\\
    &= x_0 + v_0 h + \frac{1}{2} \left(-\frac{k}{m} x_{0}\right) h^2\\
  y^{(1)}_{1} &= y^{(1)}_{0} +
                h f^{(1)}\left(\frac{h}{2},
                     \mathbf{y}_{0} + \frac{h}{2} \mathbf{f}(0, \mathbf{y}_{0})\right) \\
             &= v_{0} + \frac{h}{m} \left[-k\left(x_{0} - \frac{h}{2} k x_{0}\right)\right]
\end{align}

Note that `rk2` contains a $h^2$ term in the position update.

#### RK2 applied in practice
Let's implement `rk2`

\begin{align}
\mathbf{y}_{n+1} &= \mathbf{y}_n + h \mathbf{k}_2\\
\mathbf{k}_1 &= \mathbf{f}(t_n, \mathbf{y}_n)\\
\mathbf{k}_2 &= \mathbf{f}(t_n + \frac{1}{2} h, \mathbf{y}_n + \frac{1}{2} h \mathbf{k}_1)
\end{align}

in our `integrators.py`.

```python
from importlib import reload
reload(integrators);
```

##### RK2 for the power potential

```python
t, y = integrators.integrate_newton(t_max=40, force=integrators.F_power,
                                    integrator=integrators.rk2, h=0.01)
```

###### phase-space portrait

```python
plt.plot(y[:, 0], m*y[:, 1])
```

###### energy drift

```python
integrators.analyze_energies(t, y, integrators.U_power)
```

### RK4
General purpose ODE solver.

$$
y_{n+1} = y_n + \frac{h}{6}\Big(k_1 + 2 k_2 + 2 k_3 + k_4\Big), \quad t_{n+1} = t_n + h
$$

and

\begin{align}
k_1 &= f(t_n, y_n)\\
k_2 &= f(t_n + \frac{1}{2} h, y_n + \frac{1}{2} h k_1)\\
k_3 &= f(t_n + \frac{1}{2} h, y_n + \frac{1}{2} h k_2)\\
k_4 &= f(t_n + h, y_n + h k_3)
\end{align}

`rk4` can obtain $\mathcal{O}(h^4)$ precision.

#### RK4 applied in practice
Implement `rk4` in `integrator.py`:

\begin{align}
y_{n+1} &= y_n + \frac{h}{6}\Big(k_1 + 2 k_2 + 2 k_3 + k_4\Big)\\
k_1 &= f(t_n, y_n)\\
k_2 &= f(t_n + \frac{1}{2} h, y_n + \frac{1}{2} h k_1)\\
k_3 &= f(t_n + \frac{1}{2} h, y_n + \frac{1}{2} h k_2)\\
k_4 &= f(t_n + h, y_n + h k_3)
\end{align}

```python
reload(integrators)
```

    <module 'integrators_solution' from '/Volumes/ASU/oliver/Documents/Teaching/ASU/CompPhys_PHY494/2021/PHY494-resources/10_ODEs/integrators_solution.py'>

and test the 6th power potential again. What level of energy precision can you obtain over 40 time units?

##### RK4 for the 6th power potential
Note that we can easily run `rk4` with a $h=0.01$!

```python
t, y = integrators.integrate_newton(t_max=40, force=integrators.F_power,
                                    integrator=integrators.rk4, h=0.01)
```

###### phase-space protrait

```python
plt.plot(y[:, 0], m*y[:, 1])
```

###### energy drift

```python
integrators.analyze_energies(t, y, integrators.U_power)
```

You should be able to get 8 - 11 decimals precision from `rk4` with a smaller $h$.

##### High precision RK4 for the 6th power potential
Try a 10 times smaller step $h=0.001$

```python
t, y = integrators.integrate_newton(t_max=40, force=integrators.F_power,
                                    integrator=integrators.rk4, h=0.001)
integrators.analyze_energies(t, y, integrators.U_power)
```

This looks like stable integration with an error of not worse than $10^{-12}$: *Looking very good!* (Especially when you compare it to *Euler* with $h=0.001$!)

The error spikes when the potential energy is dominant, which happens near the classical turning points where (in this case) the potential is the steepest and the forces are highest.

```python
x, v = y.T
E = integrators.kinetic_energy(v, m=1) + integrators.U_power(x)
```

```python
X = np.linspace(-1.3, 1.3, 100)
plt.plot(x, E, label=r"$E=T+V$")
plt.plot(X, integrators.U_power(X), label=r"$U(x)$")
plt.xlabel(r"position $x$")
plt.ylabel(r"energy")
plt.legend()
```

```python
X = np.linspace(-1.3, 1.3, 100)
plt.plot(x, E, label=r"$E=T+V$")
plt.plot(X, integrators.U_power(X), label=r"$U(x)$")
plt.xlabel(r"position $x$")
plt.ylabel(r"energy")
plt.legend()
eps = 2e-13
plt.ylim(0.5 - eps, 0.5 + eps)
```

```python

```


 Total Generalised Variation Inpainting

### In this demo, we solve the following minimisation problem:

$$\begin{equation}
\underset{u}{\operatorname{argmin}} \|\mathcal{M}u-g\|_{1} + \mathrm{TGV}_{\alpha, \beta}(u)
\tag{1}
\end{equation}$$

where,

* $g$ is a corrupted image with missing pixels, defined by the inpainting domain $\mathcal{D}\setminus\Omega$.
* $\mathcal{M}$ is a diagonal operator with ones in the diagonal elements corresponding to pixels in $\Omega\setminus\mathcal{D}$ and zeros in $\mathcal{D}$.
* $\mathrm{TGV}_{\alpha, \beta}$ stands for the **Total Generalised Variation** introduced in [Bredies_et_al](https://epubs.siam.org/doi/abs/10.1137/090769521?mobileUi=0) and is defined as

$$\begin{equation}
\mathrm{TGV}_{\alpha, \beta}(u) = \min_{w} \alpha \|D u - w \|_{2,1} + \beta\|\mathcal{E}w\|_{2,1},
\end{equation}
$$
where,

* $\alpha$ and $\beta$ are regularising parameters and
* $\mathcal{E}w = \frac{1}{2}(D w + D w^{T})$ denotes the symmetrized gradient operator.

```python
# Import libraries
from cil.framework import ImageGeometry
from cil.utilities import dataexample, noise
from cil.optimisation.operators import MaskOperator, BlockOperator, SymmetrisedGradientOperator, \
                                GradientOperator, ZeroOperator, IdentityOperator, ChannelwiseOperator
from cil.optimisation.functions import ZeroFunction, L1Norm, MixedL21Norm, BlockFunction
from cil.optimisation.algorithms import PDHG

from mpl_toolkits.axes_grid1 import AxesGrid
import matplotlib.pyplot as plt
import numpy as np

from PIL import Image
from PIL import ImageFont
from PIL import ImageDraw
```

We first load the _RAINBOW_ image from the dataexample class and create a text inpainted domain using the **Pillow** library.

```python
# Load Rainbow data
data = dataexample.RAINBOW.get(size=(256,256))
data.reorder(['horizontal_y', 'horizontal_x','channel'])
ig = data.geometry
im = data.array
```

```python
# Create inpainted image using Pillow
data_PIL = Image.fromarray(np.uint8(im*255)).convert('RGB')
text = "\n               This is a double rainbow.\n               Remove the text using CIL. \n"*5
draw = ImageDraw.Draw(data_PIL)
font = ImageFont.truetype('DejaVuSerif.ttf', 12)
draw.text((0, 0), text, (0, 0, 0), font=font)

tmp_np = np.array(data_PIL)
data_np = tmp_np/tmp_np.max()
data_inpainted= ig.allocate()
data_inpainted.fill(data_np)

plt.figure(figsize=(10,10))
plt.imshow(data_inpainted.as_array())
plt.colorbar()
plt.show()
```

Then, we create a mask array, based on the missing text and applied it to the Red, Green and Blue channels of the coloured image using the **ChannelwiseOperator**. Finally, salt and pepper noise is added to create the `noisy_data`.

```python
# Create mask from corrupted image and apply MaskOperator channelwise
mask = (data_inpainted - data).abs()
mask_ind = mask.abs()==0

mask2D = mask_ind[:,:,0]
mask = ig.subset(channel=0).allocate(True,dtype=np.bool)
mask.fill(mask2D)

# Define ChannelwiseOperator
MO = ChannelwiseOperator(MaskOperator(mask), 3, dimension = 'append')

# Add salt and pepper noise
noisy_data = noise.saltnpepper(data_inpainted, amount=0.01, seed = 10)
noisy_data = MO.direct(noisy_data)

f, ax = plt.subplots(1,2, figsize=(15,10))
ax[0].imshow(mask.as_array())
ax[0].title.set_text("Mask")

ax[1].imshow(noisy_data.as_array())
ax[1].title.set_text("Corrupted data")
```

In order to solve [(1)](#tgv_reg), we use the  **Primal-Dual Hybrid Gradient (PDHG)** algorithm, introduced in [ChambollePock](https://link.springer.com/article/10.1007/s10851-010-0251-1). We setup and run the PDHG algorithm using the Total Generalised variation regularisation. We need to write the minimisation problem:

$$\begin{equation}
    (u^{*},w^{*}) =\underset{u, w}{\operatorname{argmin}} \|\mathcal{M}u -b\|_{1} + \alpha \|D u - w \|_{2,1} + \beta\|\mathcal{E}w\|_{2,1}
\end{equation}$$

in the following general form $$\underset{x\in \mathbb{X}}{\operatorname{argmin}} f(Kx) + g(x).$$

Let $x = (u, w)\in \mathbb{X}$ and

* define an operator $K:\mathbb{X}\rightarrow\mathbb{Y}$ as$\\[10pt]$

    $$\begin{equation}
    K =
    \begin{bmatrix}
    \mathcal{M} & \mathcal{O}\\
    D & -\mathcal{I}\\
    \mathcal{O} & \mathcal{E}
    \end{bmatrix} \quad\Rightarrow\quad
    Kx =
    K \begin{bmatrix}
    u\\
    w
    \end{bmatrix}=
    \begin{bmatrix}
    \mathcal{M}u\\
    Du - w\\
    \mathcal{E}w
    \end{bmatrix} =
    \begin{bmatrix}
    y_{1}\\
    y_{2}\\
    y_{3}
    \end{bmatrix} = y\in \mathbb{Y},
    \label{def_K}
    \end{equation}$$

* define a function $f:\mathbb{Y}\rightarrow\mathbb{R}$ as$\\[10pt]$

    $$\begin{equation}
    \begin{aligned}
    & f(y)  := f(y_{1}, y_{2}, y_{3}) = f_{1}(y_{1}) +  f_{2}(y_{2})  +  f_{3}(y_{3}), \mbox{ where},\\[10pt]
    & f_{1}(y_{1}) :=  \| y_{1} - b\|_1,\, f_{2}(y_{2}) :=  \alpha \|y_{2}\|_{2,1},\, f_{3}(y_{3}) := \beta\|y_{3}\|_{2,1},
    \end{aligned}
    \label{def_f}
    \end{equation}\\[10pt]$$

* and define a function $g(x) = g(u,w) = O(u)\equiv 0 $, i.e., the zero function.

```python
# Setup PDHG for TGV regularisation
alpha = 0.7
beta = 0.2

# Define BlockFunction f
f1 = L1Norm(b=noisy_data)
f2 = alpha * MixedL21Norm()
f3 = beta * MixedL21Norm()
f = BlockFunction(f1, f2, f3)

# Define function g
g = ZeroFunction()

# Define BlockOperator K
K11 = MO
K21 = GradientOperator(ig)
K32 = SymmetrisedGradientOperator(K21.range)
K12 = ZeroOperator(K32.domain, ig)
K22 = IdentityOperator(K21.range)
K31 = ZeroOperator(ig, K32.range)
K = BlockOperator(K11, K12, K21, -K22, K31, K32, shape=(3,2) )

# Compute operator Norm
normK = K.norm()
sigma = 1.
tau = 1./(sigma*normK**2)

# Setup and run the PDHG algorithm
pdhg = PDHG(f=f,g=g,operator=K,
            max_iteration = 500, sigma=sigma, tau=tau,
            update_objective_interval = 100)
pdhg.run(verbose = 2)
```

```python
show_images = [data, noisy_data, pdhg.solution.get_item(0),
               (pdhg.solution.get_item(0)-data).abs()+0.1]
title_image = ["Ground Truth", "Corrupted Data", "TGV", "Absolute difference"]

fig = plt.figure(figsize=(20, 20))
grid = AxesGrid(fig, 111,
                nrows_ncols=(2, 2),
                axes_pad=0.45)

k = 0
for ax in grid:

    im = ax.imshow(show_images[k].as_array())
    ax.set_title(title_image[k],fontsize=25)
    k+=1

plt.show()
```


```python
import numpy as np
import matplotlib.pyplot as plt

from sympy.solvers import solve
from sympy import Symbol
```

```python
x = Symbol('x')

sols = solve(8.99 * x - 6.56 * x - 1312.13, x)
sols
```

    [539.971193415638]

```python
xs = np.linspace(0, 1000, 1000)
plt.plot(xs, 2.43 * xs - 1312.13)
plt.axhline(0, c='k')

plt.show()
```

```python
sols1 = solve(400 * x - 3936.13, x)
sols1
```

    [9.84032500000000]

```python
xs = np.linspace(0, 10, 1000)
plt.plot(xs, 400 * xs - 3936.13)
plt.axhline(0, c='k')

plt.show()
```

```python

```


# Solving Statics Problem using SymPy

## Question

## Solution

### Installing [SymPy](https://pypi.org/project/sympy/) using [pip](https://pypi.org/project/pip/)

~~~bash
python3 -m pip install sympy
~~~

### Importing necessary modules from SymPy

* [printing](https://docs.sympy.org/latest/modules/printing.html) - Useful for printing complex equations
* [plotting](https://docs.sympy.org/latest/modules/plotting.html) - To print necessary plots in part (b) of question

```python
from sympy.interactive import printing
printing.init_printing(use_latex=True)
from sympy.plotting import plot
from sympy import *
import sympy as sp
```

### Inserting equations and displaying the result

```python
eq1 = sp.Function('eq1')
eq2 = sp.Function('eq2')
eq3 = sp.Function('eq3')
eq4 = sp.Function('eq4')

Ax, Ay, A, T, lc, l, d, w = sp.symbols('A_x A_y A T l_c l d w')

#defining the equations
eq1 = Eq(Ax-T*(d/lc),0)
eq2 = Eq(Ay-(T*(sqrt(lc**2 - d**2))/lc)-w, 0)
eq3 = Eq((T*(sqrt(lc**2 - d**2)/lc)*d)-w*l, 0)

print("Given equations are")
display(eq1)
display(eq2)
display(eq3)
```

### Solving the given equations using ``solve``

Solving the given three equations gives the three forces **$ T $**, **$ {A_x} $** and **$ A_y $** in terms of *$ d, l_c\ , l $ and $ w $*

```python
#----------------------------------------PART (a)----------------------------------------#

#solving the above three equations in terms of constants l, l_c, w and d
print("The forces in terms of given constants are")
solution = solve([eq1, eq2, eq3], [Ax, Ay, T])

#to display result in proper format
solution_dict = {sympify(key): sympify(solution[key], locals={'A_x': Ax, 'A_y' : Ay, 'T' : T}) for key in solution}
for key in solution_dict:
    display(Eq(key, dict_parser(key, solution_dict)))
```

### Subsituting given values in above equations

Substiting *$ l = 10\,ft\ , \, l_c=5.5\,ft \ $ and $ w = 200 \, lbs $* in given equations and displaying the results

```python
#----------------------------------------PART (b)----------------------------------------#

#equations after subsituting given values of
#l = 10 ft
#l_c = 5.5 ft
#w = 200 lbs
eq1_s = eq1.subs([(l, 10), (lc, 5.5), (w, 200)])
eq2_s = eq2.subs([(l, 10), (lc, 5.5), (w, 200)])
eq3_s = eq3.subs([(l, 10), (lc, 5.5), (w, 200)])

print("\n\nEquations after subsituting given values are")
display(eq1_s)
display(eq2_s)
display(eq3_s)
```

### Solving above equations in terms of $ d $

On solving above three equations in terms of $ d $ gives **$ T $**, **$ {A_x} $** and **$ A_y $** as function of a single variable $ d $

```python
#solving the above three equations in terms of d
print("\nThe quantities in terms of d are")
solution_s = solve([eq1_s, eq2_s, eq3_s], [Ax, Ay, T])

#printing the solved equations
solution_s_dict = {sympify(key): sympify(solution_s[key], locals={'A_x': Ax, 'A_y' : Ay, 'T' : T}) for key in solution}
for key in solution_s_dict:
    display(Eq(key, dict_parser(key, solution_s_dict)))
```

### Obtaining equation for force $ A $

The resultant of force components **$ A_x $** and **$ A_y $** is obtained as force **$ A $**

By subsituting values of **$ A_x $** and **$ A_y $** in equation **$ A = \sqrt {A_x^2 +A_y^2} $** , force **$ A $** is obtained as a function of $ d $

```python
#equation for A (resultant of Ax and Ay)
eq4 = Eq(A, sqrt(Ax**2 + Ay**2))
display(eq4)

#getting equations of Ax and Ay after subsitution
Ax_s = dict_parser(Ax, solution_s_dict)
Ay_s = dict_parser(Ay, solution_s_dict)

#equation of A(d) after subsituting the equation of Ax and Ay
A_s = sqrt(Ax_s**2 + Ay_s**2)
display(Eq(A, A_s))
```

### Plotting $ A(d) $ and $ T(d) $ in the domain $ d \in [0.5,5]$

```python
#plotting A(d) as a function of d
print("\nThe the plot of A(d) is")
plot(A_s, (d, 0.5, 5), xlabel="d", ylabel="A(d)")

#getting the equation of T(d) after subsitution of known values
T_s = dict_parser(T, solution_s_dict)
print("\nThe the plot of T(d) is")
plot(T_s, (d, 0.5, 5), xlabel="d", ylabel="T(d)")
```

### Finding minima of $ T(d) $

For maxima or minima the first derivative of function must be equal to **ZERO**

Hence, $T'(d) = 0$

On solving above equation only those values of $ d $ are to be selected which lie within the given domain $ d \in [0.5,5]$

For a minima to occur at above point required condition is that second derivative of $T(d)$ must be greater than zero (_positive_)

Hence, for minima, $T''(d) > 0$

```python
#----------------------------------------PART (c)----------------------------------------#

#evaluating the derivative of T(d) with respect to d
dTdd = T_s.diff(d)

#equating the derivative of T(d) to zero and solving for d to give the values of minima
eq5 = Eq(dTdd, 0)
extremum_values_d_T = solve(eq5, d)
#evaluating second derivative of T(d) with respect to T
d2Tdd2 = dTdd.diff(d)

for d_min in extremum_values_d_T:
    d2Tdd2_d_min = d2Tdd2.subs(d, d_min)

    #condition for minima T''(d_min) > 0
    if d2Tdd2_d_min > 0:
        if 0.5 <= d_min <= 5:
            #saving the minimum value of d within the range in d_min_T variable
            d_min_T = d_min
            print("\n\nThe minima of T(d) occurs at %0.3f" % d_min_T)

#evaluating the minimun value of T(d)
min_T = T_s.subs(d, d_min_T)
print("The minimum value of T(d) at d = %.3f" % d_min_T, " is %.3f" % min_T)
```

    The minima of T(d) occurs at 3.889
    The minimum value of T(d) at d = 3.889  is 727.273

### Finding minima of $ A(d) $

Similiar to previous part, for maxima or minima the first derivative of function must be equal to **ZERO**

Hence, $A'(d) = 0$

On solving above equation only those values of $ d $ are to be selected which lie within the given domain $ d \in [0.5,5]$

For a minima to occur at above point required condition is that second derivative of $A(d)$ must be greater than zero (_positive_)

Hence, for minima, $A''(d) > 0$

```python
#----------------------------------------PART (d)----------------------------------------#

#evaluating the derivative of A(d) with respect to d
dAdd = A_s.diff(d)

#equating the derivative of A(d) to zero and solving for d to give the values of minima
eq6 = Eq(dAdd, 0)
extremum_values_d_A = solve(eq6, d)
#evaluating second derivative of T(d) with respect to T
d2Add2 = dAdd.diff(d)

for d_min in extremum_values_d_A:
    #considering only real values of d_min
    check_if_real = d_min.is_real
    if check_if_real == 1:
        d2Add2_d_min = d2Add2.subs(d, d_min)
        #condition for minima T''(d_min) > 0
        if d2Add2_d_min > 0:
            #considering values only within the desired range
            if 0.5 <= d_min <= 5:
            #saving the minimum value of d within the range in d_min_T variable
                d_min_A = d_min
                print("\n\nThe minima of A(d) occurs at %0.3f" % d_min_A)

#evaluating the minimun value of T(d)
min_A = A_s.subs(d, d_min_A)
print("The minimum value of A(d) at d = %.3f" % d_min_A, " is %.3f" % min_A)
```

    The minima of A(d) occurs at 4.051
    The minimum value of A(d) at d = 4.051  is 877.643

### Dictionary parser function

A function to parse dictionary obtained after subsituting values in the equations.

This function makes assigning values to newer variables easier. Also, the function allows the equations to be displayed with and $ = $ 'equal to' sign *(displaying a dictionary as it is uses a $ : $ colon to associate variables with the function)*

```python
#function to print the results
def dict_parser(expr, dict):
    while True:
        new_expr = expr.subs(dict)
        if new_expr == expr:
            return expr
        else:
            expr = new_expr
```


# Unconstrained Scalar Optimization

Introductory calculus courses introduce the minimization (or maximization) of a function of a single variable. Given a function $f(x)$, find values $x^*$ such that $f(x^*) \leq f(x)$ (or $f(x^*) \geq f(x)$) for all $x$ in an interval containing $x^*$. Such points are called local optima.

If the derivative exists at all points in a given interval, then the local optima are found by solving for values $x^*$ that satisfy

\begin{align}
f'(x^*) = 0
\end{align}

Let's see how this is put to work in the context of process engineering.

## Application: Maximizing Production of a Reaction Intermediate

A desired product $B$ is produced as intermediate in a series reaction

\begin{align}
A \overset{k_A}{\longrightarrow} B \overset{k_B}{\longrightarrow} C
\end{align}

where $A$ is a raw material and $C$ is a undesired by-product. The reaction operates at temperature where the rate constants are $k_A = 0.5\ \mbox{min}^{-1}$ and $k_A = 0.1\ \mbox{min}^{-1}$. The raw material is available as a solution with concenration $C_{A,f} = 2.0\ \mbox{moles/liter}$.

A 100 liter tank is avialable to run the reaction. Below we will answer the following questions:

1. If the goal is obtain the maximum possible concentration of $B$, and the tank is operated as a continuous stirred tank reactor, what should be the flowrate?

2. What is the production rate of $B$ at maximum concentration?

## Mathematical Model for a Continuous Stirred Tank Reactor

The reaction dynamics for an isothermal continuous stirred tank reactor with a volume $V = 40$ liters and feed concentration $C_{A,f}$ are modeled as

\begin{align}
V\frac{dC_A}{dt} & = q(C_{A,f} - C_A) - V k_A C_A \\
V\frac{dC_B}{dt} & = - q C_B + V k_A C_A - V k_B C_B
\end{align}

At steady-state the material balances become

\begin{align}
0 & = q(C_{A,f} - \bar{C}_A) - V k_A \bar{C}_A \\
0 & = - q \bar{C}_B + V k_A \bar{C}_A - V k_B \bar{C}_B
\end{align}

which can be solved for $C_A$

\begin{align}
\bar{C}_A & = \frac{qC_{A,f}}{q + Vk_A} \\
\end{align}

and then for $C_B$

\begin{align}
\bar{C}_B & = \frac{q V k_A C_{A,f}}{(q + V k_A)(q + Vk_B)}
\end{align}

The numerator is first-order in flowrate $q$, and the denominator is quadratic.  This is consistent with an intermediate value of $q$ corresponding to a maximum concentration $\bar{C}_B$.

The next cell plots $\bar{C}_B$ as a function of flowrate $q$.

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np

V = 40     # liters
kA = 0.5   # 1/min
kB = 0.1   # l/min
CAf = 2.0  # moles/liter

def cstr(q):
    return q*V*kA*CAf/(q + V*kB)/(q + V*kA)

q = np.linspace(0,30,200)
plt.plot(q, cstr(q))
plt.xlabel('flowrate q / liters per minute')
plt.ylabel('concentration C_B / moles per liter')
plt.title('Outlet concentration for a CSTR')
plt.grid()
```

We see that, for the parameters given, there is an optimal flowrate somewhere between 5 and 10 liters per minute.

## Analytical Solution using Calculus

As it happens, this problem has an interesting analytical solution that can be found by hand, and which can be used to check the accuracy of numerical solutions. Setting the first derivative of $\bar{C}_B$ to zero,

\begin{align}
\left.\frac{d\bar{C}_B}{dq}\right|_{q^*} = \frac{V k_A C_{A,f}}{(q^* + V k_A)(q^* + Vk_B)} - \frac{q^* V k_A C_{A,f}}{(q^* + V k_A)^2(q^* + Vk_B)} - \frac{q^* V k_A C_{A,f}}{(q^* + V k_A)(q^* + Vk_B)^2} = 0
\end{align}

Clearing out the non-negative common factors yields

\begin{align}
1 - \frac{q^*}{(q^* + V k_A)} - \frac{q^*}{(q^* + Vk_B)} = 0
\end{align}

and multiplying by the non-negative denominators produces

\begin{align}
{q^*}^2 + q^*V(k_A + k_B) + V^2k_Ak_B - q^*(q^* + Vk_B) - q^*(q^* + Vk_A) = 0
\end{align}

Expanding these expressions followed by arithmetic cancelations gives the final result

\begin{align}
q^*  =  V\sqrt{k_Ak_B}
\end{align}

which shows the optimal dilution rate, $\frac{q^*}{V}$, is equal the geomtric mean of the rate constants.

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np

V = 40     # liters
kA = 0.5   # 1/min
kB = 0.1   # l/min
CAf = 2.0  # moles/liter

qmax = V*np.sqrt(kA*kB)
CBmax = cstr(qmax)
print('Flowrate at maximum CB = ', qmax, 'liters per minute.')
print('Maximum CB =', CBmax, 'moles per liter.')
print('Productivity = ', qmax*CBmax, 'moles per minute.')
```

    Flowrate at maximum CB =  8.94427190999916 liters per minute.
    Maximum CB = 0.9549150281252629 moles per liter.
    Productivity =  8.541019662496845 moles per minute.

## Numerical Solution with Pyomo

This problem can also be solved using Pyomo to create a model instance, then

```python
from pyomo.environ import *

V = 40     # liters
kA = 0.5   # 1/min
kB = 0.1   # l/min
CAf = 2.0  # moles/liter

# create a model instance
m = ConcreteModel()

# create the decision variable
m.q = Var(domain=NonNegativeReals)

# create the objective
m.CBmax = Objective(expr=m.q*V*kA*CAf/(m.q + V*kB)/(m.q + V*kA), sense=maximize)

# solve using the nonlinear solver ipopt
SolverFactory('ipopt').solve(m)

# print solution
print('Flowrate at maximum CB = ', m.q(), 'liters per minute.')
print('Maximum CB =', m.CBmax(), 'moles per liter.')
print('Productivity = ', m.q()*m.CBmax(), 'moles per minute.')
```

    Flowrate at maximum CB =  8.944271964904415 liters per minute.
    Maximum CB = 0.9549150281252627 moles per liter.
    Productivity =  8.541019714926698 moles per minute.

One advantage of using Pyomo for solving problems like these is that you can reduce the amount of algebra needed to prepare the problem for numerical solution. This not only minimizes your work, but also reduces possible sources of error in your solution.

In this example, the steady-state equations are

\begin{align}
0 & = q(C_{A,f} - \bar{C}_A) - V k_A \bar{C}_A \\
0 & = - q \bar{C}_B + V k_A \bar{C}_A - V k_B \bar{C}_B
\end{align}

with unknowns $C_B$ and $C_A$. The modeling strategy is to introduce variables for the flowrate $q$ and these unknowns, and introduce the steady state equations as constraints.

```python
from pyomo.environ import *

V = 40     # liters
kA = 0.5   # 1/min
kB = 0.1   # l/min
CAf = 2.0  # moles/liter

# create a model instance
m = ConcreteModel()

# create the decision variable
m.q  = Var(domain=NonNegativeReals)
m.CA = Var(domain=NonNegativeReals)
m.CB = Var(domain=NonNegativeReals)

# equations as constraints
m.eqn = ConstraintList()
m.eqn.add(0 == m.q*(CAf - m.CA) - V*kA*m.CA)
m.eqn.add(0 == -m.q*m.CB + V*kA*m.CA - V*kB*m.CB)

# create the objective
m.CBmax = Objective(expr=m.CB, sense=maximize)

# solve using the nonlinear solver ipopt
SolverFactory('ipopt').solve(m)

# print solution
print('Flowrate at maximum CB = ', m.q(), 'liters per minute.')
print('Maximum CB =', m.CBmax(), 'moles per liter.')
print('Productivity = ', m.q()*m.CBmax(), 'moles per minute.')
```

    Flowrate at maximum CB =  8.944272002876573 liters per minute.
    Maximum CB = 0.9549150281377385 moles per liter.
    Productivity =  8.541019751298471 moles per minute.

```python

```


# Logistic Regression

## Task 1
Recall the data processing routines from the last lab course. The following excercises build on top of the extracted feature representations, but instead of the prebuilt classifier, we want to implement logistic regression by hand. To this end, make sure, that the variables `train`, `test`, `train_data_features` and `test_data_features` are loaded to your IPython shell.

```python
import pandas as pd
from bs4 import BeautifulSoup
import re
from nltk.corpus import stopwords
from sklearn.feature_extraction.text import CountVectorizer

#%% download nltk stopwords
# import nltk
# ntlk.download('stopwords')

# load stopwords
stops = set(stopwords.words('english'))

# function for preprocessing the data
def review_prepro(data, remove_stopwords=False):
    # remove HTML tags
    review_text = BeautifulSoup(data, 'lxml').get_text()
    # remove non-letters and numbers
    letters_only = re.sub( '[^a-zA-Z]',
                          ' ',
                          review_text )
    # make all characters lower case and split the documents into single words
    words = letters_only.lower().split()

    if remove_stopwords:
        # remove stop words
        meaningful_words = [ w for w in words if not w in stops ]
        # return concatenated single string
        return ' '.join(meaningful_words)
    else:
        # or don't and concatenate to single string
        return ' '.join(words)

# load data as pandas dataframe
train = pd.read_csv('labeledTrainData.tsv',
                    header=0,
                    delimiter="\t",
                    quoting=3 )

test = pd.read_csv('labeledTestData.tsv',
                   header=0,
                   delimiter="\t",
                   quoting=3 )

# preprocess train and test data
num_reviews = train['review'].size

clean_train_reviews = []
for i in range(num_reviews):
  #  if (i+1)%1000 == 0:
    #    print('Review {} of {}\n'.format(i+1, num_reviews))
    clean_train_reviews.append( review_prepro(train['review'][i], remove_stopwords=True) )

num_test_reviews = test['review'].size

clean_test_reviews = []
for i in range(num_test_reviews):
    #if (i+1)%1000 == 0:
     #   print('Review {} of {}\n'.format(i+1, num_test_reviews))
    clean_test_reviews.append( review_prepro(test['review'][i], remove_stopwords=True) )

#%% create BoW
# Documentation:
# http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html
vectorizer = CountVectorizer(analyzer = 'word',
                             tokenizer = None,
                             preprocessor = None,
                             stop_words = stops,
                             max_features = 5000)

# fit the vectorizer and return transformed reviews
vectorizer.fit(clean_train_reviews)
train_data_features = vectorizer.transform(clean_train_reviews)
clean_train_reviews=None
# convert to numpy array
train_data_features = train_data_features.toarray()

# create BoW representation of test data
test_data_features = vectorizer.transform(clean_test_reviews)
clean_test_reviews=None
test_data_features = test_data_features.toarray()
```

a) Write a PYTHON function `logistic_gradient` that expects a training set matrix `X_train`, a ground truth label vector `y_train` and a current weight vector `w` as its input and returns the gradient `g` of the negative log-likelihood function of the logistic regression. Refer to the lecture notes for the mathematical definition.

```python
import numpy as np

def sigmoid(x):
    # https://timvieira.github.io/blog/post/2014/02/11/exp-normalize-trick/
    z = np.exp(-np.abs(x))
    return np.where(x>=0.0,1.0/(1.0+z),z/(1.0+z))

def logistic_gradient(X_train,y_train,w,reg=0.0):
    # Rows are variables, Columns are samples
    g=np.dot(X_train,sigmoid(np.dot(X_train.T, w))-y_train)+reg*w
    return g
```

b )Write a PYTHON function `logistic_hessian` that expects a training set matrix `X_train` and a current weight vector `w` as its input and returns the Hessian `H` of the negative log-likelihood function of the logistic regression. Refer to the lecture notes for the mathematical definition.

```python
def logistic_hessian(X_train,w, reg=0.0):
    S=sigmoid(np.dot(X_train.T,w))
    diagB=S*(1.0-S)
    XS=X_train*np.expand_dims(diagB,axis=0)
    H=np.dot(XS, X_train.T)+reg*np.eye(w.shape[0])
    return H
```

c) Write a PYTHON function `find_w` that expects a training set matrix `X_train`, a ground truth label vector `y_train`, a fixed step size `h` and a maximum iteration number `max_it` that determines the optimal logistic regression weight vector `w_star` by performing gradient descent via calling `logistic_gradient` in each iteration. Make sure to include the affine offset $w_0$ in your model.

```python
def find_w(X_train, y_train, max_it, reg=0.0,w_init=None):
    X_offs=np.ones((X_train.shape[0]+1,X_train.shape[1]))
    X_offs[1:,:]=X_train
    w=np.zeros((X_offs.shape[0]))
    if w_init is not None:
        w=w_init
    cost=np.inf
    for i in range(max_it):
        delta_w=np.linalg.lstsq(logistic_hessian(X_offs, w,reg=reg),logistic_gradient(X_offs, y_train, w,reg=reg))[0]
        w-=np.array(delta_w)
        S=sigmoid(np.dot(w.T,X_offs))
        oldcost=cost
        cost=-(np.dot(np.log(np.where(S<np.finfo('float32').eps,np.finfo('float32').eps,S)),y_train)
              +np.dot(np.log(np.where((1.0-S)<np.finfo('float32').eps,np.finfo('float32').eps,1.0-S)),1.0-y_train)
              -reg*np.sum(w**2))
        if cost>=oldcost:
            break
        w_star=w
        print('Iteration:', i+1, ' Current cost:', cost)
    return w_star
```

d) Write a function `classify_log` that expects a weight vector `w` and a test set matrix `X_test` and classifies the samples in `X_test` via logistic regression, returning a label vector `y_test`. Test your implementation on `train_data_features` and `test_data_features` with one iteration and with 10 iterations. What do you observe?

```python
from sklearn.metrics import roc_auc_score as AUC

def classify_log(w, X_test):
    w0=w[0]
    y_test=sigmoid(w0+np.dot(X_test.T,w[1:]))
    return y_test

#Testing implementation
y_train=train['sentiment'].values
y_test=test['sentiment'].values
train=None
test=None
w=find_w(train_data_features.T,y_train,1,reg=0)
y_pred=classify_log(w,test_data_features.T)
auc = AUC( y_test, y_pred )
print('AUC score after 1 iteration:',auc)
w=find_w(train_data_features.T,y_train,10,reg=0)
y_pred=classify_log(w,test_data_features.T)
auc = AUC( y_test, y_pred )
print('AUC score after 10 iteration:',auc)

```

    Iteration: 1  Current cost: 6165.48143183
    AUC score after 1 iteration: 0.906356178126
    Iteration: 1  Current cost: 6165.48143183
    Iteration: 2  Current cost: 3961.1406912
    Iteration: 3  Current cost: 2634.09296401
    Iteration: 4  Current cost: 1738.31616292
    Iteration: 5  Current cost: 1054.12358097
    Iteration: 6  Current cost: 514.98272427
    Iteration: 7  Current cost: 196.676386247
    Iteration: 8  Current cost: 71.7497222914
    Iteration: 9  Current cost: 26.299656854
    Iteration: 10  Current cost: 9.67265985876
    AUC score after 10 iteration: 0.875910041024

e) Logistic regression is prone to *overfitting*. To prevent this, regularizing parameters can be used. Adjust your implementation in such a way that instead of minimizing $L(\mathbf{w})$, it minimizes the term
    	\begin{equation}
    	L(\mathbf{w})+\alpha\|\mathbf{w}\|^2,
    	\end{equation}
    	where $\alpha$ is a non-negative regularization parameter. Test your implementation with $\alpha=1$ with one iteration and with 10 iterations.

```python
w=find_w(train_data_features.T,y_train,1,reg=1)
y_pred=classify_log(w,test_data_features.T)
auc = AUC( y_test, y_pred )
print('AUC score after 1 iteration:',auc)
w=find_w(train_data_features.T,y_train,10,reg=1)
y_pred=classify_log(w,test_data_features.T)
auc = AUC( y_test, y_pred )
print('AUC score after 10 iteration:',auc)
```

    Iteration: 1  Current cost: 6304.09530912
    AUC score after 1 iteration: 0.910314236004
    Iteration: 1  Current cost: 6304.09530912
    Iteration: 2  Current cost: 4330.85401281
    Iteration: 3  Current cost: 3437.94965462
    Iteration: 4  Current cost: 3205.4784196
    AUC score after 10 iteration: 0.92835827158


# Excercises Electric Machinery Fundamentals
## Chapter 6

## Animation: Determining the rotor-slip with  a compass

```python
from sympy import init_session
init_session()
%matplotlib notebook
```

    IPython console for SymPy 1.8 (Python 3.8.10-64-bit) (ground types: python)

    These commands were executed:
    >>> from __future__ import division
    >>> from sympy import *
    >>> x, y, z, t = symbols('x y z t')
    >>> k, m, n = symbols('k m n', integer=True)
    >>> f, g, h = symbols('f g h', cls=Function)
    >>> init_printing()

    Documentation can be found at https://docs.sympy.org/1.8/

    Warning: Cannot change to a different GUI toolkit: notebook. Using qt instead.

Look at a superposition of two sinusoidal signals:

```python
f=sin(x)*sin(y)
f
```

This product can be rewritten as a sum using trigonometric equalities. SymPy has a special function for those:

```python
from sympy.simplify.fu import *
g=TR8(f)     # TR8 is a trigonometric expression function from Fu paper
Eq(f, g)
```

Now let us take an example with two different angular frequencies:

```python
s = 0.03        # slip
fs = 50         # stator frequency in Hz
fr = (1-s)*fs   # rotor frequency in Hz
fr
```

```python
alpha=2*pi*fs*t
beta=2*pi*fr*t
```

Now plot the product of both frequencies:

```python
# Create the plot of the stator rotation frequency in blue:
p1=plot(sin(alpha), (t, 0, 1), show=False, line_color='b', adaptive=False, nb_of_points=5000)
# Create the plot of the rotor rotation frequency in green:
p2=plot(0.5*sin(beta), (t, 0, 1), show=False, line_color='g', adaptive=False, nb_of_points=5000)
# Create the plot of the combined flux in red:
p3=plot(0.5*f.subs([(x, alpha), (y, beta)]), (t, 0, 1),
            show=False, line_color='r', adaptive=False, nb_of_points=5000)
# Make the second and third one a part of the first one.
p1.extend(p2)
p1.extend(p3)
# Display the modified plot object.
p1.show()
```


# Resolución de sistemas de ecuaciones lineales

Juan Pablo Echeagaray González

Data Science Club Academy

27 de septiembre del 2021

## Librerías básicas

Siempre vale la pena tener estas 2 librerías a la mano, numpy suele ser mucho más eficaz que las funciones nativas de python en cuanto a operaciones matemáticas se refiere, y matplotlib es básicamente el estándar para realizar visualizaciones en este lenguaje.

```python
import numpy as np
import sympy as sm
```

Para más información, consulta el cuaderno de OneNote de la clase.

[Presiona aquí](https://tecmx-my.sharepoint.com/:o:/g/personal/a00830646_itesm_mx/EjsM5FC-YCxHnkM61ZZqfdwBub6PRS6cHAuM-EmeLETCEg?e=lLrH5s) para ver el cuaderno de OneNote.

## Ejemplos

### Ejemplo 1

```python
A = np.array([[1, 1, 1], [1, -1, 2], [2, 0, 3]])
b = np.array([[3], [2], [1]])
np.linalg.solve(A, b)
```

```python
augmented = np.concatenate((A, b), axis=1)
augmented = sm.Matrix(augmented)
augmented.rref()[0]
```

$\displaystyle \left[\begin{matrix}1 & 0 & \frac{3}{2} & 0\\0 & 1 & - \frac{1}{2} & 0\\0 & 0 & 0 & 1\end{matrix}\right]$

No te asustes por el error de la sección pasada, numpy ha funcionado como debería, y nos ha avisado que le fue imposible calcular el inverso de la matriz A, así que la función ```np.linalg.solve(A,b)``` nos dio un error.

El siguiente paso que podemos tomar es calcular su forma **RREF**, *row reduced echelon form*, y de ahí hacer el análisis pertinente. En este caso se tiene una incosistencia, ya que no es posible que al multiplicar un conjunto de variables por 0 y sumarlas se obtenga un 1.

### Ejemplo 2

```python
A = np.array([[1, 1, 1], [1, -1, 2], [0, 1, 1]])
b = np.array([[3], [2], [2]])
x = np.linalg.solve(A, b)
x
```

    array([[1.],
           [1.],
           [1.]])

```python
augmented = np.concatenate((A, b), axis=1)
augmented = sm.Matrix(augmented)
augmented.rref()[0]
```

$\displaystyle \left[\begin{matrix}1 & 0 & 0 & 1\\0 & 1 & 0 & 1\\0 & 0 & 1 & 1\end{matrix}\right]$

Aquí tenemos un ejemplo bonito, la función ```np.linalg.solve(A,b)``` pudo encontrar un vector que satisfaciera la ecuación matricial del sistema. No es necesario que hagamos otra operación, pero de cualquier manera quisiera que veas cómo se ve la matriz en su forma RREF.

Puedes notar que dentro de esa matriz se encuentra la mismísima matriz identidad I3 , y a su derecha se encuentra el vector solución x.

### Ejemplo 3

```python
A = np.array([[1, 1, 1], [1, -1, 2], [2, 0, 3]])
b = np.array([[3], [2], [5]])
x = np.linalg.solve(A, b)
x
```

```python
augmented = np.concatenate((A, b), axis=1)
augmented = sm.Matrix(augmented)
augmented.rref()[0]
```

$\displaystyle \left[\begin{matrix}1 & 0 & \frac{3}{2} & \frac{5}{2}\\0 & 1 & - \frac{1}{2} & \frac{1}{2}\\0 & 0 & 0 & 0\end{matrix}\right]$

Este es por mucho el ejemplo más interesante de los 3. La función ```np.linalg.solve(A,b)``` no pudo encontrar un vector que satisfaciera la ecuación matricial del sistema, y la forma **RREF** de la matriz tiene una fila entera de ceros. Esto no significa que la matriz no tenga soluciones; no, al contrario, tiene una cantidad infinita de ellas!

Ve el cuaderno de la clase para que identifiques la forma que tiene la solución del sistema.