# Interpolación polinomial
## Vandermonde, Lagrange, Diferencias divididas de Newton
### SCT 2020

```python
import numpy as np
import matplotlib.pyplot as plt
```

### Implementación de métodos de interpolación

```python
def vandermonde(x_i, y_i):
    n = x_i.shape[-1]
    A = np.vander(x_i, increasing=True)
    a = np.linalg.solve(A, y_i)
    V = lambda x: np.dot(np.array([x ** i for i in range(n)]), a)
    return np.vectorize(V)
```

```python
def lagrange(x_i, y_i):
    n = x_i.shape[-1]
    L = lambda x: np.dot(y_i, np.array([np.prod(x - np.delete(x_i, k))
                                        / np.prod(x_i[k] - np.delete(x_i, k)) for k in range(n)]))
    return np.vectorize(L)
```

```python
def barycentric(x_i, y_i):
    n = x_i.shape[-1]
    w = 1 / np.array([np.prod(x_i[i] - np.delete(x_i, i)) for i in range(n)])
    b1 = lambda x: y_i[np.where(np.in1d(x_i, x))]
    b2 = lambda x: np.dot(y_i, w /(x - x_i))/ np.dot(w, 1 / (x - x_i))
    B = lambda x: b1(x) if x in x_i else b2(x)
    return np.vectorize(B)
```

```python
def newtonDD(x_i, y_i):
    n = x_i.shape[-1]
    pyramid = np.zeros((n, n)) # Create a square matrix to hold pyramid
    pyramid[:,0] = y_i # first column is y
    for j in range(1,n):
        for i in range(n-j):
            # create pyramid by updating other columns
            pyramid[i][j] = (pyramid[i+1][j-1] - pyramid[i][j-1]) / (x_i[i+j] - x_i[i])
    a = pyramid[0] # f[ ... ] coefficients
    N = lambda x: a[0] + np.dot(a[1:], np.array([np.prod(x - x_i[:i]) for i in range(1, n)]))
    return np.vectorize(N)
```

```python
f1 = lambda x: 1 / (1 + 25 * x ** 2) # Runge Function
f2 = lambda x: np.piecewise(x, [np.abs(x) <= 1e-14], [1])
```

### Gráfico de funciones

```python
xx = np.linspace(-2, 2, 1001)
plt.figure(figsize=(12, 6))
plt.plot(xx, f1(xx), label=r"$f_1(x)$")
plt.plot(xx, f2(xx), label=r"$f_2(x)$")
plt.xlabel(r"$x$")
plt.ylabel(r"$y$")
plt.legend()
plt.grid(True)
plt.show()
```

Interpolación utilizando puntos de $f_1(x)$.

```python
f = f1
```

```python
N_i = 13
x_a, x_b = -1, 1
x_i = np.linspace(x_a, x_b, N_i)
y_i = f(x_i)
```

```python
Pv = vandermonde(x_i, y_i)
```

```python
Pl = lagrange(x_i, y_i)
```

```python
Pb = barycentric(x_i, y_i)
```

```python
Pn = newtonDD(x_i, y_i)
```

```python
N_e = 500
x_e = np.linspace(x_a, x_b, N_e)
y_e = f(x_e)
```

### Gráfico de función y el resultado de interpolación

```python
plt.figure(figsize=(14, 8))
plt.plot(x_e, y_e, 'b-', label=r"$f(x)$")
plt.plot(x_i, y_i, 'ro', label="Puntos para interpolar")
plt.plot(x_e, Pv(x_e), 'r--', label=r'$P_V(x)$')
plt.plot(x_e, Pl(x_e), 'b--', label=r'$P_L(x)$')
plt.plot(x_e, Pb(x_e), 'g--', label=r'$P_B(x)$')
plt.plot(x_e, Pn(x_e), 'k--', label=r'$P_N(x)$')
plt.grid(True)
plt.legend()
plt.xlabel(r"$x$")
plt.ylabel(r"$y$")
plt.show()
```

Notamos como el interpolador efectivamente pasa por los puntos de interpolación, pero aparece el **Fenómeno de Runge**.

# Nodos de Chebyshev

Recordar que los puntos de Chebyshev se definen como:
\begin{equation}
    x_i = \cos\left(\frac{(2i - 1)\pi}{2n}\right), \quad i=1, \dots, n
\end{equation}

```python
def chebyshevNodes(n):
    i = np.arange(1, n+1)
    t = (2*i - 1) * np.pi / (2 * n)
    return np.cos(t)
```

```python
N_c = 13
```

```python
xc_i = chebyshevNodes(N_c)
```

```python
yc_i = f(xc_i)
```

### Puntos de Chebyshev

Visualización de algunos puntos...

```python
t = np.linspace(0, np.pi, 100)
plt.figure(figsize=(12, 6))
plt.plot(np.cos(t), np.sin(t))
plt.plot(xc_i, np.zeros(N_c), 'bo')
plt.plot(xc_i, xc_i * np.tan((2 * np.arange(1, N_c+1) - 1) * np.pi / (2 * N_c)), 'ro')
plt.axhline(y=0, color='k')
plt.axis('scaled')
plt.grid(True)
plt.xlabel(r"$x$")
plt.show()
```

### Obtención de polinomios utilizando los puntos de Chebyshev

```python
Pv_c = vandermonde(xc_i, yc_i)
```

```python
Pl_c = lagrange(xc_i, yc_i)
```

```python
Pb_c = barycentric(xc_i, yc_i)
```

```python
Pn_c = newtonDD(xc_i, yc_i)
```

```python
xc_e = chebyshevNodes(500)
```

### Gráfico utilizando los puntos de Chebysev

```python
plt.figure(figsize=(12, 6))
plt.plot(xc_e, f(xc_e), 'b-', label=r"$f(x)$")
plt.plot(xc_i, yc_i, 'ro', label="Puntos para interpolar")
plt.plot(xc_e, Pv_c(xc_e), 'r--', label=r'$P_V(x)$')
plt.plot(xc_e, Pl_c(xc_e), 'b--', label=r'$P_L(x)$')
plt.plot(xc_e, Pb_c(xc_e), 'g--', label=r'$P_B(x)$')
plt.plot(xc_e, Pn_c(xc_e), 'k--', label=r'$P_N(x)$')
plt.grid(True)
plt.legend()
plt.xlabel(r"$x$")
plt.ylabel(r"$y$")
plt.show()
```

### Comparación interpoladores

Comparamos un error relativo entre el interpolador utilizando puntos equiespaciados y los de Chebyshev.

```python
error = lambda y, p: np.linalg.norm(y - p, np.inf) / np.linalg.norm(y, np.inf)
```

```python
print("Error puntos equiespaciados:", error(f(x_e), Pb(x_e))) # Equispaced points interpolation comparison
print("Error puntos Chebyshev:", error(f(x_e), Pb_c(x_e))) # Chebyshev points interpolation comparison
```

    Error puntos equiespaciados: 3.663235759944772
    Error puntos Chebyshev: 0.06919634037617944

# Conclusión

En este ejemplo/ejercicio se puede ver como afecta la calidad de la interpolación con el uso de puntos de Chebyshev en comparación a puntos equiespaciados.

# Referencias
* https://medium.com/@sddkal/newtons-divided-difference-method-for-polynomial-interpolation-4bc094ba90d7


# Perceptrons as Logical Operators

## AND Perceptron

#### What are the weights and bias for the AND perceptron?

Set the weights (weight1, weight2) and bias (bias) to values that will correctly determine the AND operation as shown above.

More than one set of values will work!

```python
import pandas as pd

# TODO: Set weight1, weight2, and bias
weight1 = 0.8
weight2 = 0.8
bias = -1

# DON'T CHANGE ANYTHING BELOW
# Inputs and outputs
test_inputs = [(0, 0), (0, 1), (1, 0), (1, 1)]
correct_outputs = [False, False, False, True]
outputs = []

# Generate and check output
for test_input, correct_output in zip(test_inputs, correct_outputs):
    linear_combination = weight1 * test_input[0] + weight2 * test_input[1] + bias
    output = int(linear_combination >= 0)
    is_correct_string = 'Yes' if output == correct_output else 'No'
    outputs.append([test_input[0], test_input[1], linear_combination, output, is_correct_string])

# Print output
num_wrong = len([output[4] for output in outputs if output[4] == 'No'])
output_frame = pd.DataFrame(outputs, columns=['Input 1', '  Input 2', '  Linear Combination', '  Activation Output', '  Is Correct'])
if not num_wrong:
    print('Nice!  You got it all correct.\n')
else:
    print('You got {} wrong.  Keep trying!\n'.format(num_wrong))
print(output_frame.to_string(index=False))

```

    Nice!  You got it all correct.

    Input 1    Input 2    Linear Combination    Activation Output   Is Correct
          0          0                  -1.0                    0          Yes
          0          1                  -0.2                    0          Yes
          1          0                  -0.2                    0          Yes
          1          1                   0.6                    1          Yes

## OR Perceptron

The OR perceptron is very similar to an AND perceptron. In the image below, the OR perceptron has the same line as the AND perceptron, except the line is shifted down. What can you do to the weights and/or bias to achieve this? Use the following AND perceptron to create an OR Perceptron.

The two ways to go from an AND perceptron to an OR perceptron

- Increase the weights
- Decrease the magnitude of the bias

```python
import pandas as pd

# TODO: Set weight1, weight2, and bias
weight1 = 1.1
weight2 = 1.1
bias = -1

#weight1 = 0.8
#weight2 = 0.8
#bias = -0.5

# DON'T CHANGE ANYTHING BELOW
# Inputs and outputs
test_inputs = [(0, 0), (0, 1), (1, 0), (1, 1)]
correct_outputs = [False, True, True, True]
outputs = []

# Generate and check output
for test_input, correct_output in zip(test_inputs, correct_outputs):
    linear_combination = weight1 * test_input[0] + weight2 * test_input[1] + bias
    output = int(linear_combination >= 0)
    is_correct_string = 'Yes' if output == correct_output else 'No'
    outputs.append([test_input[0], test_input[1], linear_combination, output, is_correct_string])

# Print output
num_wrong = len([output[4] for output in outputs if output[4] == 'No'])
output_frame = pd.DataFrame(outputs, columns=['Input 1', '  Input 2', '  Linear Combination', '  Activation Output', '  Is Correct'])
if not num_wrong:
    print('Nice!  You got it all correct.\n')
else:
    print('You got {} wrong.  Keep trying!\n'.format(num_wrong))
print(output_frame.to_string(index=False))

```

    You got 1 wrong.  Keep trying!

    Input 1    Input 2    Linear Combination    Activation Output   Is Correct
          0          0                   0.0                    1           No
          0          1                   1.1                    1          Yes
          1          0                   1.1                    1          Yes
          1          1                   2.2                    1          Yes

## NOT Perceptron

Unlike the other perceptrons we looked at, the NOT operation only cares about one input. The operation returns a 0 if the input is 1 and a 1 if it's a 0. The other inputs to the perceptron are ignored.

In this quiz, you'll set the weights (weight1, weight2) and bias bias to the values that calculate the NOT operation on the second input and ignores the first input.

```python
import pandas as pd

# TODO: Set weight1, weight2, and bias
weight1 = 0.5
weight2 = -0.7
bias = 0.0

# DON'T CHANGE ANYTHING BELOW
# Inputs and outputs
test_inputs = [(0, 0), (0, 1), (1, 0), (1, 1)]
correct_outputs = [True, False, True, False]
outputs = []

# Generate and check output
for test_input, correct_output in zip(test_inputs, correct_outputs):
    linear_combination = weight1 * test_input[0] + weight2 * test_input[1] + bias
    output = int(linear_combination >= 0)
    is_correct_string = 'Yes' if output == correct_output else 'No'
    outputs.append([test_input[0], test_input[1], linear_combination, output, is_correct_string])

# Print output
num_wrong = len([output[4] for output in outputs if output[4] == 'No'])
output_frame = pd.DataFrame(outputs, columns=['Input 1', '  Input 2', '  Linear Combination', '  Activation Output', '  Is Correct'])
if not num_wrong:
    print('Nice!  You got it all correct.\n')
else:
    print('You got {} wrong.  Keep trying!\n'.format(num_wrong))
print(output_frame.to_string(index=False))
```

    Nice!  You got it all correct.

    Input 1    Input 2    Linear Combination    Activation Output   Is Correct
          0          0                   0.0                    1          Yes
          0          1                  -0.7                    0          Yes
          1          0                   0.5                    1          Yes
          1          1                  -0.2                    0          Yes

## XOR Perceptron

#### Build an XOR Multi-Layer Perceptron
Now, let's build a multi-layer perceptron from the AND, NOT, and OR perceptrons to create XOR logic!

The neural network below contains 3 perceptrons, A, B, and C. The last one (AND) has been given for you. The input to the neural network is from the first node. The output comes out of the last node.

The multi-layer perceptron below calculates XOR. Each perceptron is a logic operation of AND, OR, and NOT. However, the perceptrons A, B, and C don't indicate their operation. In the following quiz, set the correct operations for the perceptrons to calculate XOR.

#### Solution

## Perceptron Algorithm

Implement the perceptron algorithm to separate the following data (given in the file data.csv).

Recall that the perceptron step works as follows. For a point with coordinates (p,q)(p,q), label yy, and prediction given by the equation

\begin{align}
\hat{y} = step(w_1x_1 + w_2x_2 + b)
\end{align}

- If the point is correctly classified, do nothing.
- If the point is classified positive, but it has a negative label, subtract αp,αq, and α from w_1, w_2,w_1,w_2, and bb respectively.
- If the point is classified negative, but it has a positive label, add αp,αq, and α to w_1, w_2,w_1,w_2, and bb respectively.

graph the solution that the perceptron algorithm gives you. It'll actually draw a set of dotted lines, that show how the algorithm approaches to the best solution, given by the black solid line.

```python
import numpy as np
# Setting the random seed, feel free to change it and see different solutions.
np.random.seed(42)

def stepFunction(t):
    if t >= 0:
        return 1
    return 0

def prediction(X, W, b):
    return stepFunction((np.matmul(X,W)+b)[0])

# TODO: Fill in the code below to implement the perceptron trick.
# The function should receive as inputs the data X, the labels y,
# the weights W (as an array), and the bias b,
# update the weights and bias W, b, according to the perceptron algorithm,
# and return W and b.
def perceptronStep(X, y, W, b, learn_rate = 0.01):
    # Fill in code
    for i in range(len(X)):
        y_hat = prediction(X[i],W,b)
        if y[i]-y_hat == 1:
            W[0] += X[i][0]*learn_rate
            W[1] += X[i][1]*learn_rate
            b += learn_rate
        elif y[i]-y_hat == -1:
            W[0] -= X[i][0]*learn_rate
            W[1] -= X[i][1]*learn_rate
            b -= learn_rate
    return W, b

# This function runs the perceptron algorithm repeatedly on the dataset,
# and returns a few of the boundary lines obtained in the iterations,
# for plotting purposes.
# Feel free to play with the learning rate and the num_epochs,
# and see your results plotted below.
def trainPerceptronAlgorithm(X, y, learn_rate = 0.01, num_epochs = 25):
    x_min, x_max = min(X.T[0]), max(X.T[0])
    y_min, y_max = min(X.T[1]), max(X.T[1])
    W = np.array(np.random.rand(2,1))
    b = np.random.rand(1)[0] + x_max
    # These are the solution lines that get plotted below.
    boundary_lines = []
    for i in range(num_epochs):
        # In each epoch, we apply the perceptron step.
        W, b = perceptronStep(X, y, W, b, learn_rate)
        boundary_lines.append((-W[0]/W[1], -b/W[1]))
    return boundary_lines

```


```python
#format the book
from __future__ import division, print_function
%matplotlib inline
import sys
sys.path.insert(0, '..')
import book_format
book_format.set_style()
```

<style>
.output_wrapper, .output {
    height:auto !important;
    max-height:100000px;
}
.output_scroll {
    box-shadow:none !important;
    webkit-box-shadow:none !important;
}
</style>

# Computing and plotting PDFs of discrete data

So let's investigate how to compute and plot probability distributions.

First, let's make some data according to a normal distribution. We use `numpy.random.normal` for this. The parameters are not well named. `loc` is the mean of the distribution, and `scale` is the standard deviation. We can call this function to create an arbitrary number of data points that are distributed according to that mean and std.

```python
import numpy as np
import numpy.random as random

mean = 3
std = 2

data = random.normal(loc=mean, scale=std, size=50000)
print(len(data))
print(data.mean())
print(data.std())
```

    50000
    2.9928466729147547
    1.9934066626288809

As you can see from the print statements we got 5000 points that have a mean very close to 3, and a standard deviation close to 2.

We can plot this Gaussian by using `scipy.stats.norm` to create a frozen function that we will then use to compute the pdf (probability distribution function) of the Gaussian.

```python
%matplotlib inline
import matplotlib.pyplot as plt
import scipy.stats as stats

def plot_normal(xs, mean, std, **kwargs):
    # method 1
    #norm = stats.norm(mean, std)
    #plt.plot(xs, norm.pdf(xs), **kwargs)

    # alt method (my way)
    #syntax: pdf(x, loc=0, scale=1)
    plt.plot(xs, stats.norm.pdf(xs, loc=mean, scale=std))

xs = np.linspace(-5, 15, num=200) #Returns `num` evenly spaced samples, calculated over the interval [start, stop].
plot_normal(xs, mean, std, color='k')
```

But we really want to plot the PDF of the discrete data, not the idealized function.

There are a couple of ways of doing that. First, we can take advantage of `matplotlib`'s `hist` method, which computes a histogram of a collection of data. Normally `hist` computes the number of points that fall in a bin, like so:

```python
plt.hist(data, bins=200) # fit data of size=50000 into bins of size = 200
plt.show()
```

that is not very useful to us - we want the PDF, not bin counts. Fortunately `hist` includes a `density` parameter which will plot the PDF for us.

```python
plt.hist(data, bins=200, density=True)
plt.show()
```

I may not want bars, so I can specify the `histtype` as 'step' to get a line.

```python
plt.hist(data, bins=200, density=True, histtype='step', lw=2)
plt.show()
```

To be sure it is working, let's also plot the idealized Gaussian in black.

```python
plt.hist(data, bins=200, density=True, histtype='step', lw=2)
norm = stats.norm(mean, std)
plt.plot(xs, norm.pdf(xs), color='k', lw=2)
plt.show()
```

There is another way to get the approximate distribution of a set of data. There is a technique called *kernel density estimate* that uses a kernel to estimate the probability distribution of a set of data. SciPy implements it with the function `gaussian_kde`. **Do not be mislead by the name - Gaussian refers to the type of kernel used in the computation. This works for any distribution, not just Gaussians. In this section we have a Gaussian distribution, but soon we will not, and this same function will work.**

```python
# popo_notes: Important
# prepare distribution with mean = 3, std=2 (not gauusian necessarily)
kde = stats.gaussian_kde(data) #kde is now our distribution
# sample E(x) from this distribution
xs = np.linspace(-5, 15, num=200)
plt.plot(xs, kde(xs))
plt.show()
```

## Monte Carlo Simulations

We (well I) want to do this sort of thing because I want to use monte carlo simulations to compute distributions. It is easy to compute Gaussians when they pass through linear functions, but difficult to impossible to compute them analytically when passed through nonlinear functions. **Techniques like particle filtering handle this by taking a large sample of points, passing them through a nonlinear function, and then computing statistics on the transformed points. Let's do that.**

> popo_notes

### 'Lets compute statistics on points transformed using linear and non-linear functions

---

We will start with the linear function $f(x) = 2x + 12$ just to prove to ourselves that the code is working. I will alter the mean and std of the data we are working with to help ensure the numbers that are output are unique It is easy to be fooled, for example, if the formula multipies x by 2, the mean is 2, and the std is 2. If the output of something is 4, is that due to the multication factor, the mean, the std, or a bug? It's hard to tell.

```python
def f(x):
    return 2*x + 12

mean = 1.
std = 1.4
data = random.normal(loc=mean, scale=std, size=50000)

d_t = f(data) # transform data through f(x)

plt.hist(data, bins=200, density=True, histtype='step', lw=2)
plt.hist(d_t, bins=200, density=True, histtype='step', lw=2)

plt.ylim(0, .35)
plt.show()
print('mean = {:.2f}'.format(d_t.mean()))
print('std  = {:.2f}'.format(d_t.std()))
```

This is what we expected. The input is the Gaussian $\mathcal{N}(\mu=1, \sigma=1.4)$, and the function is $f(x) = 2x+12$. Therefore we expect the mean to be shifted to $f(\mu) = 2*1+12=14$. We can see from the plot and the print statement that this is what happened.

Before I go on, can you explain what happened to the standard deviation? You may have thought that the new $\sigma$ should be passed through $f(x)$ like so $2(1.4) + 12=14.81$. But that is not correct - **the standard deviation is only affected by the multiplicative factor, not the shift.** If you think about that for a moment you will see it makes sense. We multiply our samples by 2, so they are twice as spread out as before. Standard deviation is a measure of how spread out things are, so it should also double. It doesn't matter if we then shift that distribution 12 places, or 12 million for that matter - the spread is still twice the input data.

## Nonlinear Functions

Now that we believe in our code, lets try it with nonlinear functions.

```python
def f2(x):
    return (np.cos((1.5*x + 2.1))) * np.sin(0.3*x) - 1.6*x

d_t = f2(data)
plt.subplot(121)
plt.title('f(x)')
plt.hist(d_t, bins=200, density=True, histtype='step', lw=2)

plt.subplot(122)
kde = stats.gaussian_kde(d_t)
xs = np.linspace(-10, 10, 200)
plt.plot(xs, kde(xs), 'k') # kde doesn't approximate distribution, so it prints non-gaussian distribution here
plot_normal(xs, d_t.mean(), d_t.std(), color='g', lw=3)# plot approximated gaussian
plt.show()
print('mean = {:.2f}'.format(d_t.mean()))
print('std  = {:.2f}'.format(d_t.std()))
```

Here I passed the data through the nonlinear function $f(x) = \cos(1.5x+2.1)\sin(\frac{x}{3}) - 1.6x$. That function is quite close to linear, but we can see how much it alters the pdf of the sampled data.

There is a lot of computation going on behind the scenes to transform 50,000 points and then compute their PDF. The Extended Kalman Filter (EKF) gets around this by linearizing the function at the mean and then passing the Gaussian through the linear equation. We saw above how easy it is to pass a Gaussian through a linear function. So lets try that.

We can linearize this by taking the derivative of the function at x. We can use sympy to get the derivative.

```python
#get the derivative of f(x) (=dfx)
import sympy
x = sympy.symbols('x')
f = sympy.cos(1.5*x+2.1) * sympy.sin(x/3) - 1.6*x
dfx = sympy.diff(f, x)
dfx
```

$\displaystyle - 1.5 \sin{\left(\frac{x}{3} \right)} \sin{\left(1.5 x + 2.1 \right)} + \frac{\cos{\left(\frac{x}{3} \right)} \cos{\left(1.5 x + 2.1 \right)}}{3} - 1.6$

We can now compute the slope of the function by evaluating the derivative at the mean.

```python
m = dfx.subs(x, mean)
m
```

$\displaystyle -1.66528051815545$

> popo_notes:

**There are 2 different and independent things: data (with its mean and std) and some function. When this data (x) is passed through a function, it gets transformed into f(x). The mean is also transformed as f(mean) but the std is transformed as `function_slope` * std.**

---

The equation of a line is $y=mx+b$, so the new standard deviation should be $~1.67$ times the input std. We can compute the new mean by passing it through the original function because the linearized function is just the slope of f(x) evaluated at the mean. The slope is a tangent that touches the function at $x$, so both will return the same result. So, let's plot this and compare it to the results from the monte carlo simulation.

```python
plt.hist(d_t, bins=200, density=True, histtype='step', lw=2)
plot_normal(xs, f2(mean), abs(float(m)*std), color='k', lw=3, label='EKF')
plot_normal(xs, d_t.mean(), d_t.std(), color='r', lw=3, label='MC')
plt.legend()
plt.show()
```

We can see from this that the estimate from the EKF (in red) is not exact, but it is not a bad approximation either.


## Classical Mechanics - Week 7

### Last Week:
- Simulated planetary motion
- Saw the limitations of Euler's Method
- Gained experience with the Velocity Verlet Method

### This Week:
- Introduce the SymPy package
- Visualize Potential Energy surfaces
- Explore packages in Python

# Why use packages, libraries, and functions in coding?
Another great question!

**Simply put:** We could hard code every little algorithm into our program and retype them every time we need them, OR we could call upon the functions from packages and libraries to do these tedious calculations and reduce the possibility of error.

We have done this with numpy.linalg.norm() to calculate the magnitude of vectors.

We will be introducing a new package call SymPy, a very useful [symbolic mathematics](https://en.wikipedia.org/wiki/Computer_algebra) library.

```python
# Let's import packages, as usual
import numpy as np
import matplotlib.pyplot as plt
import sympy as sym
sym.init_printing(use_unicode=True)
```

Let's analyze a simple projectile motion again, but this time using SymPy.

Assume we have the following equation to express our trajectory in the $x-y$ coordinates:

$y = y_0 - (\beta -x)^2$, where $y_0$ and $\beta$ are constants.

```python
# First we must declare and define our variables. Examine the syntax in this cell then run it. Notice that ordering matters
x, y0, beta = sym.symbols('x, y_0, beta')
```

```python
# Next we will define our function
y = y0 - (beta - x)**2
y # This line outputs a visualization of our equation y(x) below
```

```python
# Now we will set our constants, but leave x alone so we can plot as a function of x
y1 = sym.simplify(y.subs({y0:10, beta:1}))
y1 # This line outputs a visualization of our equation y1(x) below
```

```python
# Run this cell. What happens as you play around with the constants?
sym.plot(y1,(x,0,5), title='Height vs Distance',xlabel='x',ylabel='y')
```

## Q1.) How would you compare plotting with sympy versus what we have done so far? Which method do you prefer?

✅ Double click this cell, erase its content, and put your answer to the above question here.

################ Possible Answer #################

Sympy Gives a more direct way of creating graphs.
Numpy is better for numerical analysis.

################ Possible Answer #################

#### Using what we just learned, please set up the following equation using sympy where $U(\vec{r})$ is our potential energy:

$U(\vec{r}) = -e^{-(x^2+y^2)}$

```python
U, x, y = sym.symbols('U, x, y') ## Set up our variables here. What should be on the left-hand and right-hand side?
U = -sym.exp(-(x**2+y**2)) ## What should go in the exp? Notice that using SymPy we need to use the SymPy function for exp
U
```

### We have two ways in which we can graph this:

Either perform the substitution $r^2 = x^2+y^2$ in order to plot in a 2D space ($U(r)$) or we could plot in a 3D space keeping $x$ and $y$ in our equation (U(x,y)).

## Q2.) What do you think are the benefits/draw-backs of these two methods of analyzing our equation? Which do you prefer?

✅ Double click this cell, erase its content, and put your answer to the above question here.

####### Possible Answer #######

The 2D method keeps things simple and allows us to see how potential changes as the magnitude of our distance changes. A drawback is that we don't get to see how the individual x and y components affect our potential.

The 3D method lets us see how potential changes as a result of both x and y, allowing us to see a more in-depth analysis of the potential.  For instance, the rotational symmetry in the xy plane is apparent in the 3D plot. However, the graph is a bit more complicated since visualizing 3 dimensions on a 2D surface (a sheet of paper or a computer screen) is sometimes difficult.

I prefer the air.

####### Possible Answer #######

#### Now let's graph the potential
For now we will use sympy to perform both a 2D and 3D plot. Let's do the 2D version first using what we just learned.

```python
r = sym.symbols('r') # Creating our variables
U2D = -sym.exp(-r**2) # Finish the equation using our replacement for x^2 + y^2
sym.plot(U2D,title='Potential vs Distance',xlabel='Distance (r)',ylabel='Potential (U)')
```

## Q3.) What can you learn from this 2D graph?

✅ Double click this cell, erase its content, and put your answer to the above question here.

####### Possible Answer #######

The Potential Energy has a minimum at $r=0$.  As we get closer to the origin the potential energy becomes more negative with $U(0) = -1$.  The width of the well is of order 2.

####### Possible Answer #######

The cell below imports a function from the sympy package that allows us to graph in 3D. Using the "plot3d" call, make a 3D plot of our originally initalized equation.

For the 3D plot, try setting the x and y plot limits as close as you can to the origin while having a clear picture of what is happening at the origin. For example x and y could range from (-4,4)

```python
# The below import will allow us to plot in 3D with sympy
# Define "U" as your potential
from sympy.plotting import plot3d
```

```python
# Once you have your potential function set up, execute this cell to obtain a graph
# Play around with the x and y scale to get a better view of the potential curve
plot3d(U,(x,-2,2),(y,-2,2))
```

## Q4.) What can you learn from this 3D graph? (Feel free to make the graph's x and y-range smaller to observe the differences)

✅ Double click this cell, erase its content, and put your answer to the above question here.

####### Possible Answer #######

It shows similar information as the 2D plot, but now we can see the azimuthal symmetry in the xy plane.

####### Possible Answer #######

##### Let's get some more in-depth coding experience:
Try to graph this last potential energy function using SymPy or with Numpy (there is a 3d plotting example back in Week 2's notebook).

$U(r) = 3.2\cdot e^{-(0.5x^2+0.25y^3)}$

```python
## Set up and graph the potential here
U = 3.2*sym.exp(-(0.5*x**2+0.25*y**2))
plot3d(U,(x,-4,4),(y,-4,4))
```

## Q5.) How would you describe the new potential?

✅ Double click this cell, erase its content, and put your answer to the above question here.

####### Possible Answer #######

Now the Potential Energy has a maximum at $r=0$, with a value of $U(0)=3.2$.  It is no longer azimuthally symmetric in the xy plane, although it is difficult to see this in the 3D plot.

####### Possible Answer #######

### Try this:
Center the new potential at (1,1) instead of (0,0).  (That is, move the peak of the graph from (0,0) to (1,1).)

```python
## Plot the adjustment here
U = 3.2*sym.exp(-(0.5*(x-1)**2+0.25*(y-1)**2))
plot3d(U,(x,-4,4),(y,-4,4))
```

## Q6.) How did you move the peak of the graph?

✅ Double click this cell, erase its content, and put your answer to the above question here.

####### Possible Answer #######

Replace $x$ with $x-1$ and $y$ with $y-1$ in the potential.

####### Possible Answer #######

# Notebook Wrap-up.
Run the cell below and copy-paste your answers into their corresponding cells.

```python
from IPython.display import HTML
HTML(
"""

"""
)
```

# Congratulations! Another week, another Notebook.

As we can see, there are many tools we can use to model and analyze different problems in Physics on top of Numerical methods. Libraries and packages are such tools that have been developed by scientists to work on different topics, each package specific to a different application. But this is just food for thought. Although we use some basic package functions, we won't be using advanced scientific packages to do simulations and calculations in this class.

```python

```


## Worked example: Symmetry reduction

```python
import numpy as np
import sympy as sp
import scipy.linalg as la
```

### Some routines for creating rotation matrices and transpositions

```python
def get_rotation_matrix(phi, axis):
    cp = np.cos(phi)
    sp = np.sin(phi)
    r1, r2, r3 = axis
    Rm = np.array(
        [
            [
                r1 ** 2 * (1 - cp) + cp,
                r1 * r2 * (1 - cp) - r3 * sp,
                r1 * r3 * (1 - cp) + r2 * sp,
            ],
            [
                r1 * r2 * (1 - cp) + r3 * sp,
                r2 ** 2 * (1 - cp) + cp,
                r2 * r3 * (1 - cp) - r1 * sp,
            ],
            [
                r3 * r1 * (1 - cp) - r2 * sp,
                r2 * r3 * (1 - cp) + r1 * sp,
                r3 ** 2 * (1 - cp) + cp,
            ],
        ]
    )
    # Clean spurious terms
    for ii, jj in np.ndindex(Rm.shape):
        if abs(Rm[ii, jj]) < 1e-10:
            Rm[ii, jj] = 0
    sp = np.sin(phi)
    return Rm

def get_transposition(nd, verbose=0):
    # Make a basis:
    basis_nd = []
    for ii in range(nd):
        for jj in range(nd):
            emp = np.zeros([nd, nd], dtype="int8")
            bb = emp
            bb[ii, jj] = 1
            basis_nd.append(bb)
    #
    # Build transpose and return:
    transp = np.array([bb.T.flatten() for bb in basis_nd])

    return transp

# pretty print matrix
def pprint(M, maxl=80):
    M = np.atleast_2d(M)
    dim1, dim2 = M.shape
    lengths = [len(str(el)) for el in M.flatten()]
    maxl = min(maxl, max(lengths))
    for ii in range(dim1):
        print(
            (f" | " + " ,  ".join([f"{str(el)[:maxl]:{maxl}s}" for el in M[ii]]) + " |")
        )
```

### Start: Define a matrix $A$
* Define a general 3x3 matrix $A$ and its 9x1 vectorized representation
$$ \boldsymbol{a} = {\textbf{vec}} (A)$$

* $A$ and $\boldsymbol a$ can, e.g., represent a physical tensor (conductivity, stress, ...)

```python
ndim = 3
sym = sp.Symbol
A = sp.ones(ndim, ndim)
for ii in range(ndim):
    for jj in range(ndim):
        A[ndim * ii + jj] = sym(f"a_{ii}{jj}")
A = np.array(A)
a = A.flatten()
print("Matrix A:")
pprint(A)
print("vec(A):")
pprint(a)
```

    Matrix A:
     | a_00 ,  a_01 ,  a_02 |
     | a_10 ,  a_11 ,  a_12 |
     | a_20 ,  a_21 ,  a_22 |
    vec(A):
     | a_00 ,  a_01 ,  a_02 ,  a_10 ,  a_11 ,  a_12 ,  a_20 ,  a_21 ,  a_22 |

### Case 1: Cubic system
* Define 3 rotation matrices $M_i$ that implement a 90° rotation about $x$, $y$, and $z$ axis
* Construct the rotation matrices in the flattened representatin by Roth's Relationship, i.e.
$$ M_\text{flat} = M \otimes M $$
* Add transposition (not necessary in cubic case)

```python
r1 = get_rotation_matrix(np.pi / 2, [1, 0, 0])
r2 = get_rotation_matrix(np.pi / 2, [0, 1, 0])
r3 = get_rotation_matrix(np.pi / 2, [0, 0, 1])
pprint(r1)
```

     | 1.0  ,  0.0  ,  0.0  |
     | 0.0  ,  0.0  ,  -1.0 |
     | 0.0  ,  1.0  ,  0.0  |

#### Construct big matrices implementing rotations (+transposition) of the vectorized tensor

```python
R1 = np.kron(r1, r1)
R2 = np.kron(r2, r2)
R3 = np.kron(r3, r3)
T  = get_transposition(ndim)
```

#### Now sum up the matrices we want to invariant under, i.e.
$$ \sum_i (\mathbf 1 - M_i)~a = 0$$

```python
id = np.eye(len(a))
inv = 4*id - R1 - R2 - R3 - T
```

#### Consctruct nullspace by SVD
__[scipy-cookbook.readthedocs.io/items/RankNullspace.html](http://scipy-cookbook.readthedocs.io/items/RankNullspace.html)__

```python
u, s, vh = la.svd(inv, lapack_driver="gesdd")
rank = (s > 1e-12).sum()

print(f"Initial Dimension:                              {len(a)}")
print(f"Rank of Nullspace (= No. irred. elements):      {len(a) - rank}")
```

    Initial Dimension:                              9
    Rank of Nullspace (= No. irred. elements):      1

#### Construct matrices translating between full and reduced representation

* $S$ reconstructs the full representation $a$ from a given reduced $\tilde a$. One can think of $\tilde a$ as being the components of $a$ in the basis given by the vectors in $S$:
$$ a = S \, \tilde{a}$$

* $S^+$ (pseudo-inverse, often just transpose) projects a given $a$ onto the irreducible components $\tilde a$
$$\tilde{a} = S^+ a$$

```python
S = vh[rank:].T
Sp = S.T
```

#### Build projectors
* $P = S^\vphantom{+} S^+$

* $\boldsymbol{1}_\text{irred} = S^+ S^\phantom{+}$

```python
P = S @ Sp
id_irred = Sp @ S
print("Projector onto invariant space")
pprint(P, 4)
print("Identity within invariant space")
pprint(id_irred)
```

    Projector onto invariant space
     | 0.33 ,  0.0  ,  1.77 ,  -6.9 ,  0.33 ,  1.01 ,  1.29 ,  -5.2 ,  0.33 |
     | 0.0  ,  0.0  ,  0.0  ,  0.0  ,  0.0  ,  0.0  ,  0.0  ,  0.0  ,  0.0  |
     | 1.77 ,  0.0  ,  9.49 ,  -3.6 ,  1.77 ,  5.44 ,  6.91 ,  -2.8 ,  1.77 |
     | -6.9 ,  0.0  ,  -3.6 ,  1.43 ,  -6.9 ,  -2.1 ,  -2.6 ,  1.09 ,  -6.9 |
     | 0.33 ,  0.0  ,  1.77 ,  -6.9 ,  0.33 ,  1.01 ,  1.29 ,  -5.2 ,  0.33 |
     | 1.01 ,  0.0  ,  5.44 ,  -2.1 ,  1.01 ,  3.11 ,  3.96 ,  -1.6 ,  1.01 |
     | 1.29 ,  0.0  ,  6.91 ,  -2.6 ,  1.29 ,  3.96 ,  5.04 ,  -2.0 ,  1.29 |
     | -5.2 ,  0.0  ,  -2.8 ,  1.09 ,  -5.2 ,  -1.6 ,  -2.0 ,  8.28 ,  -5.2 |
     | 0.33 ,  0.0  ,  1.77 ,  -6.9 ,  0.33 ,  1.01 ,  1.29 ,  -5.2 ,  0.33 |
    Identity within invariant space
     | 1.0000000000000002 |

#### Symmetrize the tensor with the projector obtained
$$ \text{sym} (a) = P a = S^\vphantom{+} S^+ a = S \, \tilde a $$

```python
aT = np.dot(Sp, a)
aS = np.dot(S, aT) # = np.dot(P, a)
```

#### How does the matrix $A$ now look like?
$$A = \text{unvec} \left( \text{sym} (a) \right)$$

```python
As = aS.reshape(3,3)

print('1/3*')
pprint(3*As)
```

    1/3*
     | 1.0*a_00 + 5.33729362479519e-33*a_02 - 2.07749100017982e-35*a_10 + 1.0*a_11 + 3. ,  0                                                                                ,  5.33729362479519e-33*a_00 + 2.84867032372794e-65*a_02 - 1.10881794708291e-67*a_1 |
     | -2.07749100017982e-35*a_00 - 1.10881794708291e-67*a_02 + 4.31596885582815e-70*a_ ,  1.0*a_00 + 5.33729362479519e-33*a_02 - 2.07749100017982e-35*a_10 + 1.0*a_11 + 3. ,  3.05816280424746e-34*a_00 + 1.63223128386957e-66*a_02 - 6.35330570290877e-69*a_1 |
     | 3.8891592043194e-34*a_00 + 2.07575846270274e-66*a_02 - 8.07969324524005e-69*a_10 ,  -1.57643859172956e-33*a_00 - 8.41391564551927e-66*a_02 + 3.2750369866543e-68*a_1 ,  1.0*a_00 + 5.33729362479519e-33*a_02 - 2.07749100017982e-35*a_10 + 1.0*a_11 + 3. |

$= \frac{1}{3} \text{Tr A}$

## How about hexagonal?
Start with defining three-fold rotations in $x,y$ plane, i.e., about $z$ axis

```python
r1 = get_rotation_matrix(np.pi / 6, [0, 0, 1])
r2 = get_rotation_matrix(np.pi / 3, [0, 0, 1])
```

Construct the matrices in the flatten represenation as before

```python
R1 = np.kron(r1, r1)
R2 = np.kron(r2, r2)
T = get_transposition(ndim)
```

```python
id = np.eye(len(a))
inv = 3*id - R1 - R2 - T
```

```python
# Consctruct nullspace
u, s, vh = la.svd(inv, lapack_driver="gesdd")
rank = (s > 1e-12).sum()

print(f"Dimension:          {len(a)}")
print(f"Rank of Nullspace:  {len(a) - rank}")
```

    Dimension:          9
    Rank of Nullspace:  2

```python
# Nullspace:
S = vh[rank:].T
# clean S
for ii, jj in np.ndindex(S.shape):
    if abs(S[ii, jj]) < 1e-10:
        S[ii, jj] = 0
Sp = S.T
```

How do the projectors look like?

```python
print(S@Sp)
#print(Sp)
#print(S@Sp)
print(Sp@S)
```

    [[0.5 0.  0.  0.  0.5 0.  0.  0.  0. ]
     [0.  0.  0.  0.  0.  0.  0.  0.  0. ]
     [0.  0.  0.  0.  0.  0.  0.  0.  0. ]
     [0.  0.  0.  0.  0.  0.  0.  0.  0. ]
     [0.5 0.  0.  0.  0.5 0.  0.  0.  0. ]
     [0.  0.  0.  0.  0.  0.  0.  0.  0. ]
     [0.  0.  0.  0.  0.  0.  0.  0.  0. ]
     [0.  0.  0.  0.  0.  0.  0.  0.  0. ]
     [0.  0.  0.  0.  0.  0.  0.  0.  1. ]]
    [[1. 0.]
     [0. 1.]]

```python
# Projector
P = S@Sp
```

```python
# Symmetrize a
aS = np.dot(P, a)
```

```python
# Restore shape
As = aS.reshape(3,3)
```

```python
pprint(As)
```

     | 0.5*a_00 + 0.5*a_11 ,  0                   ,  0                   |
     | 0                   ,  0.5*a_00 + 0.5*a_11 ,  0                   |
     | 0                   ,  0                   ,  1.0*a_22            |

```python

```


## Enzyme-substrate kinetics

We will now solve the differential equations that result from a generic enzimatic reaction. The scheme of the reaction is as follows:
$$ s + e \overset{k_1}{\underset{k_2}{\longleftrightarrow}}  c \overset{k_3}{\longrightarrow} p + e \tag{1}$$

where $s$ represents the substrate, $e$ represents the enzyme, $c$ represents the complex, and $p$ the product of the reaction. The reaction can be written as three irreversible reactions:
$$\begin{align*}
s + e \rightarrow  c \tag{2}\\
c \rightarrow s + e \tag{3}\\
c \rightarrow p + e  \tag{4}
 \end{align*}$$

with kinetic constants `k1`, `k2` and `k3`, respectively. If we rename variables as $X_1=s,X_2=c, X_3=e, X_4=p$ to use the state vector approach.

$$\begin{align*}
X_1 + X_3 \rightarrow  X_2 \tag{2}\\
X_2 \rightarrow X_1 + X_3 \tag{3}\\
X_2 \rightarrow X_3 +  X_4 \tag{4}
 \end{align*}$$

The kinetics of these chemical equations is described by the __Mass action Law__

$$
A=\begin{bmatrix}
 1 & 0 & 1 & 0\\
 0 & 1 & 0 & 0 \\
 0 & 1 & 0 & 0\end{bmatrix} ;
B=\begin{bmatrix}
 0 & 1 & 0 & 0\\
 1 & 0 & 1 & 0 \\
 0 & 0 & 1 & 1\end{bmatrix} ; (B-A)^T= \begin{bmatrix}
-1 & 1 & 0\\
 1 & -1  & -1\\
 -1 & 1 & 1 \\
 0 & 0 & 1\end{bmatrix}
$$

 in this particular case

$$
K=\begin{pmatrix}
 k_1 & 0 & 0 \\
 0 & k_2 & 0 \\
 0 & 0 & k_3
\end{pmatrix}
$$

$$X^A=\begin{pmatrix}
X_1^1\cdot X_2^0 \cdot X_3^1 \cdot X_4^0 \\
X_1^0\cdot X_2^1 \cdot X_3^0 \cdot X_4^0 \\
X_1^0\cdot X_2^1 \cdot X_3^0 \cdot X_4^0 \\
\end{pmatrix} = \begin{pmatrix}
X_1 \cdot X_3\\
 X_2\\
  X_2
\end{pmatrix}
$$

so

$$
\begin{align}
 \begin{bmatrix}
\frac{\mathrm{d} X_1}{\mathrm{d} t}\\ \frac{\mathrm{d} X_2}{\mathrm{d} t} \\ \frac{\mathrm{d} X_3}{\mathrm{d} t} \\ \frac{\mathrm{d} X_4}{\mathrm{d} t}\end{bmatrix}= \begin{bmatrix}
-1 & 1 & 0\\
 1 & -1  & -1\\
 -1 & 1 & 1 \\
 0 & 0 & 1\end{bmatrix}\begin{pmatrix}
 k_1 & 0 & 0 \\
 0 & k_2 & 0 \\
 0 & 0 & k_3
\end{pmatrix} \begin{pmatrix}
X_1 \cdot X_3\\
 X_2\\
  X_2
\end{pmatrix}
\end{align}
$$
that when we operate the matrices becomes

$$
\begin{align}
 \begin{bmatrix}
\frac{\mathrm{d} X_1}{\mathrm{d} t}\\ \frac{\mathrm{d} X_2}{\mathrm{d} t} \\ \frac{\mathrm{d} X_3}{\mathrm{d} t} \\ \frac{\mathrm{d} X_4}{\mathrm{d} t}\end{bmatrix}= \begin{bmatrix}
-1 & 1 & 0\\
 1 & -1  & -1\\
 -1 & 1 & 1 \\
 0 & 0 & 1\end{bmatrix} \begin{pmatrix}
k_1 \cdot X_1 \cdot X_3\\
k_2 \cdot X_2\\
k_3 \cdot  X_2
\end{pmatrix}
\end{align}
$$
and operating further,

$$
\begin{align}
 \begin{bmatrix}
\frac{\mathrm{d} X_1}{\mathrm{d} t}\\ \frac{\mathrm{d} X_2}{\mathrm{d} t} \\ \frac{\mathrm{d} X_3}{\mathrm{d} t} \\ \frac{\mathrm{d} X_4}{\mathrm{d} t}\end{bmatrix}= \begin{bmatrix}
-k_1 \cdot X_1 \cdot X_3 + k_2 \cdot X_2\\
 k_1 \cdot X_1 \cdot X_3 - k_2 \cdot X_2  - k_3 \cdot  X_2\\
-k_1 \cdot X_1 \cdot X_3 + k_2 \cdot X_2 + k_3 \cdot  X_2\\
 k_3 \cdot  X_2\end{bmatrix}
\end{align}
$$

which rearranging terms provides the following set of coupled differential equations

$$
\begin{align*}
  \dot{X_1}&= k_2 \cdot X_2 - k_1 X_3 \cdot X_1 \tag{8}\\
  \dot{X_2}&= k_1 \cdot X_3 \cdot X_1 - (k_2+k_3) \cdot X_2 \tag{9}\\
  \dot{X_3}&= (k_2+k_3) \cdot X_2 - k_1 \cdot X_3(t) \cdot X_1 \tag{10}\\
  \dot{X_4}&= k_3 \cdot X_2 \tag{11}
\end{align*}
$$

To solve it numericaly, we define the ODE problem as follows:

```julia
function simpleODEEnzyme!(du,u,p,t)
    k1,k2,k3 = p
    du[1] = k2*u[2] - k1*u[1]*u[3]
    du[2] = k1*u[1]*u[3] - (k2 + k3) * u[2]
    du[3] = -k1*u[1]*u[3]+(k2 + k3) * u[2]
    du[4] = k3 * u[2]
end
```

    simpleODEEnzyme! (generic function with 1 method)

```julia
#import Pkg; Pkg.add("ParameterizedFunctions")
#import Pkg; Pkg.add("DifferentialEquations")
import Pkg; Pkg.add("Plots")
```

      Updating registry at `~/.julia/registries/General`
      Updating git-repo `https://github.com/JuliaRegistries/General.git`
     Resolving package versions...
      Updating `~/.julia/environments/v1.3/Project.toml`
     [no changes]
      Updating `~/.julia/environments/v1.3/Manifest.toml`
     [no changes]

```julia
using DifferentialEquations
using ParameterizedFunctions
```

```julia
tspan = (0.0,100)
k1=1e3
k2=0.1
k3=0.05
e0=0.002
s0=0.002

u0=[s0,0,e0,0]
p=[k1,k2,k3];
```

```julia
problem = ODEProblem(simpleODEEnzyme!,u0,tspan,p)
```

    ODEProblem with uType Array{Float64,1} and tType Float64. In-place: true
    timespan: (0.0, 100.0)
    u0: [0.002, 0.0, 0.002, 0.0]

```julia
using Plots; gr()
```

    Plots.GRBackend()

```julia
sol = solve(problem)
plot(sol)
title!("Enzyme kinetics using ODE and vector matrix notation")
xlabel!("Time [s]")
ylabel!("Concentration [M]")
#png("Michaelis_menten.png")
```

We can also use the simpler DSL direct notation, this is useful because we can use directly the differential equations with the original names of the variables:

$$\begin{align*}
  \dot{s}  &= k_2 \cdot c - k_1 \cdot e \cdot s \tag{5}\\
  \dot{c} &= k_1 \cdot e\cdot s - (k_2+k_3) \cdot c \tag{6}\\
  \dot{e} &= (k_2+k_3) \cdot c -k_1 \cdot e \cdot s \tag{7}\\
  \dot{p} &= k_3 \cdot c
  \end{align*}$$

(in Julia, we cannot use $e$ for the enzyme, so we use $en$)

```julia
enzyme_kinetics! = @ode_def ab begin
  ds = -k1*en*s+k2*c
  dc = k1*en*s-(k2+k3)*c
  den = -k1*en*s+(k2+k3)*c
  dp = k3*c
    end k1 k2 k3
```

    (::ab{var"#3#7",var"#4#8",var"#5#9",Nothing,Nothing,var"#6#10",Expr,Expr}) (generic function with 2 methods)

```julia
prob = ODEProblem(enzyme_kinetics!,u0,tspan,p)
sol = solve(prob)
plot(sol)
title!("Enzyme-substrate kinetics using DSL notation")
xlabel!("Time [s]")
ylabel!("Concentration [M]")
```

The next step is to take advantage of the __Mass Conservation Law__ to reduce the number of independent variables (i.e, the number of differential equations to solve).

$$ \begin{align*}
C_1 \cdot X_1 + C_2 \cdot X_2 + C_3 \cdot X_3 + C_4 \cdot X_4 & = cte \\
\end{align*}$$

so
$$ \begin{align*}
C \cdot (B-A)^T &=0\\
\begin{pmatrix}C_1 & C_2 & C_3 & C_4 \end{pmatrix} \begin{bmatrix}
-1 & 1 & 0\\
 1 & -1  & -1\\
 -1 & 1 & 1 \\
 0 & 0 & 1\end{bmatrix} &=0
 \end{align*}$$

the solution of the system of equations is:
 $$ \begin{align*}
 -C_1 + C_2 - C_3= 0 &\implies  C_2 = C_3 + C_1\\
 -C_2 + C_3 + C_4= 0 &\implies  C_2 = C_3 + C_4
\end{align*}$$

Therefore, from the two equations above, $C_1 = C_4$, and the conservation is:

$$ \begin{align*}
C_1 \cdot X_1(t) + (C_3 + C_1) \cdot X_2(t) + C_3 \cdot X_3(t) + C_1 \cdot X_4(t) &= cte\\
C_1 \cdot X_1(t) + C_3 \cdot X_2(t) + C_1 \cdot X_2(t)  + C_3 \cdot X_3(t) + C_1 \cdot X_4(t) &= cte\\
\end{align*}$$

which is valid for any value of $C_1$ and $C_3$, so, separating these two dependencies

$$ \begin{align*}
C_1 \cdot X_1(t) + C_1 \cdot X_2(t)  + C_1 \cdot X_4(t) &= cte\\
\end{align*}$$

which makes sense, since the $s$, $c$ and $p$ are three forms of the same protein, and their sum has to be constant,a nd equal to the initial value of substrate used. Using the value $C_1=1$ we obtain:

$$ \begin{align*}
X_1(t) + X_2(t) + X_4(t) &= cte = X_1(0) + X_2(0) + X_4(0) =  X_1(0) \\
\end{align*}$$

is our first conservation law (typically we start a biochemical reaction adding only $e$ and $s$). For the second, we use the parameters that depend on the other constant $C_2$

$$ \begin{align*}
C_3 \cdot X_2(t) + C_3 \cdot X_3(t) &= cte
\end{align*}$$

which also makes sense, because the amoun of enzyme is not consumed and is only in teh form of the reactans $e$ and $c$. So, giving the value $C_3=1$, we obtain the conservation law,
$$ \begin{align*}
X_2(t) + X_3(t) & = cte &= X_2(0) + X_3(0) = X_3 (0)
\end{align*}$$

which is true because the enzyme is a catalyst that facilitates the reaction but does not react itself, and if we assume that there is no complex $c(0)=0$ before the reacton starts, we can assume ($e(0)+c(0)=e_0$). This conservation law allows us to reduce the four differential equations into the following three coupled ordinary differential equations:

$$\begin{align*}
  \dot{s}  &= k_2 \cdot c - k_1 \cdot (e_0-c) \cdot s \tag{12}\\
  \dot{c} &= k_1 \cdot (e_0-c)\cdot s - (k_2+k_3) \cdot c \tag{13}\\
  \dot{p} &= k_3 \cdot c \tag{14}
  \end{align*}$$

which rearranging terms become:

$$\begin{align*}
  \dot{s}  &= (k_2+k_1 \cdot s) \cdot c -k_1 \cdot e_0 \cdot s \tag{15}\\
  \dot{c} &= k_1\cdot e_0\cdot s-(k_3+k_2+k_1 \cdot s) \cdot c \tag{16}\\
  \dot{p} &= k_3 \cdot c \tag{17}
  \end{align*}$$

```julia
enzyme_kinetics2! = @ode_def ab begin
  ds = (k2+s*k1)*c-k1*e0*s
  dc = k1*e0*s-(k2+k3+k1*s)*c
  dp = k3*c
    end k1 k2 k3 e0
```

    (::ab{var"#67#71",var"#68#72",var"#69#73",Nothing,Nothing,var"#70#74",Expr,Expr}) (generic function with 2 methods)

```julia
tspan = (0.0,100.0)
u0=[s0,0,0]
p=[k1,k2,k3,e0];

prob = ODEProblem(enzyme_kinetics2!,u0,tspan,p)
```

    ODEProblem with uType Array{Float64,1} and tType Float64. In-place: true
    timespan: (0.0, 100.0)
    u0: [0.002, 0.0, 0.0]

```julia
sol_ = solve(prob)
plot(sol_)
e=[p[4]-u[2] for (u,t) in tuples(sol_)]
plot!(sol_.t,e,
        label="e",
        linealpha = 1,
        linewidth = 3,
        linecolor = :purple)
title!("Enzyme-substrate kinetics simplified using mass action")
xlabel!("Time [s]")
ylabel!("Concentration [M]")
```

## Quasi steady state approximation:

The other conservation law could be used to reduce teh system to just two equations, but instead, we will use another approiximation that allows us to go further and reduce the system to a single differential equation. Thsi aproximation si swidely used in all biochemistry books and has been famously proposed by Miachelis and Menten. In briefm it assumes that the formation and degradation of the intermediate complex is often very fast and reaches equilibrium early in the reaction.

$$k_1\cdot x_1(0) ≈ k_2 >> k_3 \tag{18}$$

The substrate-enzyme binding occurs at much faster time scales than the turnover into product (often the case in biologically relevant biochemical reactions)

Under these circumstances one expects that after an initial short transient period there will be a balance between the formation of the enzyme-substrate complex and the breaking apart of complex (either to enzyme and substrate, or to enzyme and product)

$$ \frac{d c}{dt}= \frac{d e}{dt}=0 \tag{19}$$

In these conditions, we can calculate the equilibrium for $c$ from eq. 2.16 as

$$c_{eq}=\frac{k_1\cdot e_0\cdot s}{k_3+k_2+k_1 \cdot s} \tag{20}$$

This is called the __pseudo-steady state aproximation__ for biochemical systems

```julia
c_eq=[(p[1] * p[4] * u[1])/(p[3]+p[2]+p[1] * u[1]) for (u,t) in tuples(sol)]
plot!(sol.t,c_eq,
        label="c_eq",
        linealpha = 0.5,
        linewidth = 3,
        linestyle= :dash,
        linecolor = :red)
```

As we can see, the equilibrium solution $c_{eq}$ is quite close to the concentration of $c$, in this conditions we can simplify all equations as:

$$\begin{align*}
  \dot{s}  &= (k_2+k_1 \cdot s) \cdot \frac{k_1\cdot e_0\cdot s}{k_3+k_2+k_1 \cdot s} -k_1 \cdot e_0 \cdot s \tag{21}\\
  \dot{p}  &= k_3 \frac{k_1\cdot e_0\cdot s}{k_3+k_2+k_1 \cdot s} \tag{22}
\end{align*}$$

Eq. 2.20 can be rewritten in a more simplified form, after some some algebraic manipulation:

$$\begin{align*}
  \dot{s}  &= (k_2+k_1 \cdot s) \cdot \frac{k_1\cdot e_0\cdot s}{k_3+k_2+k_1 \cdot s} -k_1 \cdot e_0 \cdot s \tag{23}\\
   \dot{s}  &= k_1 \cdot e_0 \cdot s  \left [ \frac{k_2+k_1 \cdot s}{k_3+k_2+k_1 \cdot s} -1 \right ]  \tag{24}\\
\dot{s}  &= k_1 \cdot e_0 \cdot s  \left [ \frac{k_2+k_1 \cdot s - k_3-k_2-k_1 \cdot s}{k_3+k_2+k_1 \cdot s} \right ]  \tag{25}\\
\dot{s}  &=   \left [ \frac{- k_3\cdot k_1 \cdot e_0  \cdot s}{k_3+k_2+k_1 \cdot s} \right ]  \tag{26}\\
\end{align*}$$

So the final set of equations takes the form:

$$\begin{align*}
  \dot{s}  &=- \frac{k_1 \cdot k_3 \cdot e_0 \cdot s}{k_3+k_2+k_1 \cdot s} \tag{27}\\
  \dot{p}  &=  \frac{ k_3 \cdot k_1\cdot e_0\cdot s}{k_3+k_2+k_1 \cdot s} \tag{28}
\end{align*}$$

```julia
enzyme_kinetics10! = @ode_def ab begin
     ds = - (k3 * e0 * s)/((k3+k2)/k1 +  s)
     dp = (k3 * e0 * s)/((k3+k2)/k1 +  s)
    end k1 k2 k3 e0

```

    (::ab{var"#79#83",var"#80#84",var"#81#85",Nothing,Nothing,var"#82#86",Expr,Expr}) (generic function with 2 methods)

```julia
tspan = (0.0,100.0)

k1=1e3
k2=0.1
k3=0.05
e0=0.002
s0=0.002

u0=[s0,0.00001]
p=[k1,k2,k3,e0];
prob10 = ODEProblem(enzyme_kinetics10!,u0,tspan,p)

```

    ODEProblem with uType Array{Float64,1} and tType Float64. In-place: true
    timespan: (0.0, 100.0)
    u0: [0.002, 1.0e-5]

```julia
sol10 = solve(prob10)
plot(sol10)
```

Now, the conservation of mass $s+p=s_0$ allows us to reduce everything to a single ODE:
$$
  \dot{p}  =  \frac{ k_3 \cdot k_1\cdot e_0\cdot (s_0-p)}{k_3+k_2+k_1 \cdot (s_0-p)} \tag{29}
$$

Or, if the divide every term by $k_1$
$$
  \dot{p}  =  \frac{ k_3 \cdot e_0\cdot (s_0-p)}{\frac{k_3+k_2}{k_1}+(s_0-p)} \tag{30}
$$

```julia
enzyme_kinetics3! = @ode_def ab begin
  dp = (k3 * e0 * (s0 - p))/((k3+k2)/k1 +  (s0 - p))
    end k1 k2 k3 e0 s0
tspan = (0.0,100.0)

u0=[0.00001]
p=[k1,k2,k3,e0,s0];
prob2 = ODEProblem(enzyme_kinetics3!,u0,tspan,p)
sol2 = solve(prob2)
plot(sol2)
```

We can compare directly this steady state aproximation wit the mass action solution for the product

```julia
pp=[u[3] for (u,t) in tuples(sol_)]
plot!(sol_.t,pp,
        label="p full mass action",
        linealpha = 1,
        linewidth = 3,
        linecolor = :purple)
```

The steady state solution aproximates quite well to the rate of the reaction at the starting point of the reaction, but not at the intermediate time points. This, of course, will depend on the reaction parameters and the initial conditions. We can set up a program to test when the steady state aproximation is correct.

## Michaelis-Menten equation:

The previous equation

$$
  \dot{p}  =  \frac{ k_3 \cdot e_0\cdot (s_0-p)}{\frac{k_3+k_2}{k_1}+(s_0-p)} \tag{31}
$$

leads to the traditional Michaelis-Menten equation, which predicts the initial turnover rate of the enzymatic reaction $V_0$ as a function of initial substrate concentration $s_0$. So at the initial state of the reaction ($p=0$):

$$
  V_0  =  \frac{ k_3 \cdot e_0\cdot s_0}{\frac{k_3+k_2}{k_1}+s_0}=\frac{ v_{max} \cdot s_0}{K_M+s_0} \tag{32}
$$

where the constant $K_M = \frac{k_3+k_2}{k_1}$ is called the Michaelis-Menten constant and $v_{max}= k_3 \cdot e_0$ is the maximum turn-over rate.
The $K_M$  reflects the affinity of the reaction. Strong affinity means small $K_M$. At $s_0=K_M$ the turn-over rate is half maximal, i.e., $\dot{p}=\frac{v_{max}}{2}$

### Michaelis-Menten curve

The famous Michaelis-Menten plot is a culve that shows the dependence of the velocity of the reaction in terms of production of product (i.e., $\dot p$) on teh concetration of substrate. It showns three regimes:
- __linear__: there is a lot of enzyme available to bind the substrate, so the velocity of teh reaction increases linearly with the concetration of subtrate, as it ocurs in noncatalized reactions.
- __Constrained__: the intermediate regime
- __Saturated__: there is a lot of substrate available, and the enzyme is the limitant factor for the catalisis, so the speed of teh reactioon does not increase if the increase further the concetration of substrate.

```julia
s_vector= s0*LinRange(0,1,100)
plot(s_vector,k3 * e0 * s_vector ./(s_vector.+(k3+k2)/k1),label="Michaelis Menten",)
println("K_m = ",(k3+k2)/k1)
title!("Michaelis-Menten Plot")
xlabel!("concentration of substrate")
ylabel!("speed of the reaction")
```

    K_m = 0.00015000000000000001

### How good is the Michaelis-Menten Approach

We can compare this analytical equation with the result of the numerical simulation of the full ODE system. To do that we solve the numerical system, and compare the rates of the reactions predicted by the MAss action and the Michaelis-Menten approach.

```julia
"k1=1e2
k2=1e0
k3=0.01
e0=1e-2
s0=1e-2"

s_vector= s0*LinRange(0,1,100)*5
p=[k1,k2,k3,e0]
plot(s_vector,k3 * e0 * s_vector ./(s_vector.+(k3+k2)/k1),label="Michaelis Menten",)
title!("Michaelis-Menten Plot")
xlabel!("Concentration of substrate")
ylabel!("Speed of the reaction")

tspan = (0.0,10)
#p=[1e3,0.1,0.05,0.001]
speed_vector = similar(s_vector)
for i in 1:100
u0=[s_vector[i],0,0]
    prob = ODEProblem(enzyme_kinetics2!,u0,tspan,p)
    sol_ = solve(prob)
    # we shoudl compute the speed at the maximum formation of Complex.
    speed=[k3*u[2] for (u,t) in tuples(sol_)]
 speed_vector[i]=maximum(speed)
end
plot!(s_vector,speed_vector,label="rate full mass action",)
title!("Michaelis-Menten vs. Mass action")
xlabel!("Concentration of substrate")
ylabel!("Speed of the reaction")
```

This analysis shows that the Michaelis-Menten approach is valid when the amount of $s$ is higher that amount of $e$

## Kinase-Phosphatase systems

If we want to compute the equilibrium concentration for the product `p`, we have to make the last eq equal to `0` :,
$$
\dot{p}=\frac{k_3 \cdot e_0 (s_0-p)}{K_M + (s_0-p)} =0  \tag{33}
$$

so we simply obtain:

 $$\begin{align*}
k_3 \cdot e_0 (s_0-p_{eq})=0\tag{34}\\
p_{eq}=s_0\tag{35}
 \end{align*}$$

which means that all substrate is converted into product eventually. This means that, once all substrate is consumed the reaction is finished, and there is no way to reverse it. Of course, living systems do not work like that, i.e., there is allways a balance in the form of a dynamical equilibrium between reactions, (such as in reversible chemical reactions). This equilibrium, in enzyme-catalized reactions, is achieved by combining pairs of enzymes that perform opposite tasks, i.e., that catalize opposite reactions.

Quite often, one enzyme catalizes the formation from `s` to `p`, there is another enzyme that catalizes the transformation of `p` back to `s`.

$$\begin{align*}
s + e1 \leftrightarrow  e1s \rightarrow p + e1 \tag{36}\\
p + e2 \leftrightarrow  e2p \rightarrow s + e2  \tag{37}
\end{align*}$$

whith kinetic rate constants $k_1$, $k_2$ and $k_3$, for the first reaction, and $k_4$, $k_5$ and $k_6$, for the second reaction. If we write the sistem as irreversible reactions, we obatin 6 reactions:

$$\begin{align*}
s + e1 \rightarrow  e1s \tag{38}\\
e1s \rightarrow s + e1 \tag{39}\\
e1s \rightarrow p + e1 \tag{40}\\
p + e2 \rightarrow e2p \tag{41}\\
e2p \rightarrow p + e2 \tag{42}\\
e2p \rightarrow s + e2 \tag{43}
 \end{align*}$$

Lets compute the matrices form the Mass action law. If we rename variables as $X_1=s, X_2=e1s, X_3=e1, X_4=p, X_5=e2p, X_6=e2$ to use the state vector approach.

$$\begin{align*}
X_1 + X_3 \rightarrow  X_2 \\
X_2 \rightarrow X_1 + X_3 \\
X_2 \rightarrow X_3 +  X_4 \\
X_4 + X_6 \rightarrow X_5\\
X_5 \rightarrow X_4 + X_6 \\
X_5 \rightarrow X_1 + X_6
 \end{align*}$$

The kinetics of these chemical equations is described by the __Mass action Law__

$$
A=\begin{bmatrix}
 1 & 0 & 1 & 0 & 0 & 0 \\
 0 & 1 & 0 & 0 & 0 & 0\\
 0 & 1 & 0 & 0 & 0 & 0 \\
 0 & 0 & 0 & 1 & 0 & 1\\
  0 & 0 & 0 & 0 & 1 & 0\\
   0 & 0 & 0 & 0 & 1 & 0\end{bmatrix} ;
B=\begin{bmatrix}
 0 & 1 & 0 & 0 & 0 & 0 \\
 1 & 0 & 1 & 0 & 0 & 0 \\
 0 & 0 & 1 & 1 & 0 & 0 \\
 0 & 0 & 0 & 0 & 1 & 0 \\
 0 & 0 & 0 & 1 & 0 & 1 \\
 1 & 0 & 0 & 0 & 0 & 1
 \end{bmatrix} ;
 (B-A)^T= \begin{bmatrix}
 -1 &  1 &  0 &  0 &  0 &  1\\
  1 & -1 & -1 &  0 &  0 &  0\\
 -1 &  1 &  1 &  0 &  0 &  0 \\
  0 &  0 &  1 & -1 &  1 &  0 \\
  0 &  0 &  0 &  1 & -1 & -1 \\
  0 &  0 &  0 & -1 &  1 &  1\end{bmatrix}
$$
 in this particular case

$$
K=\begin{pmatrix}
 k_1 & 0 & 0 & 0 & 0 & 0\\
 0 & k_2 & 0 & 0 & 0  & 0\\
 0 & 0 & k_3 & 0 & 0 & 0\\
 0 & 0 & 0 & k_4 & 0  &0\\
 0 & 0 & 0 & 0 & k_5 & 0 \\
  0 & 0 & 0 & 0 & 0 & k_6
\end{pmatrix}
$$

$$X^A=\begin{pmatrix}
X_1^1\cdot X_2^0 \cdot X_3^1 \cdot X_4^0 \cdot X_5^0 \cdot X_6^0\\
X_1^0\cdot X_2^1 \cdot X_3^0 \cdot X_4^0 \cdot X_5^0 \cdot X_6^0\\
X_1^0\cdot X_2^1 \cdot X_3^0 \cdot X_4^0 \cdot X_5^0 \cdot X_6^0\\
X_1^0\cdot X_2^0 \cdot X_3^0 \cdot X_4^1 \cdot X_5^0 \cdot X_6^1\\
X_1^0\cdot X_2^0 \cdot X_3^0 \cdot X_4^0 \cdot X_5^1 \cdot X_6^0\\
X_1^0\cdot X_2^0 \cdot X_3^0 \cdot X_4^0 \cdot X_5^1 \cdot X_6^0
\end{pmatrix} = \begin{pmatrix}
X_1 \cdot X_3\\
 X_2\\
  X_2\\
  X_4 \cdot X_6\\
  X_5\\
  X_5
\end{pmatrix}
$$
so

$$
\begin{align}
 \begin{bmatrix}
\frac{\mathrm{d} X_1}{\mathrm{d} t}\\ \frac{\mathrm{d} X_2}{\mathrm{d} t} \\ \frac{\mathrm{d} X_3}{\mathrm{d} t} \\ \frac{\mathrm{d} X_4}{\mathrm{d} t} \\ \frac{\mathrm{d} X_5}{\mathrm{d} t} \\ \frac{\mathrm{d} X_6}{\mathrm{d} t}\end{bmatrix}= \begin{bmatrix}
 -1 &  1 &  0 &  0 &  0 &  1\\
  1 & -1 & -1 &  0 &  0 &  0\\
 -1 &  1 &  1 &  0 &  0 &  0 \\
  0 &  0 &  1 & -1 &  1 &  0 \\
  0 &  0 &  0 &  1 & -1 & -1 \\
  0 &  0 &  0 & -1 &  1 &  1\end{bmatrix}\begin{pmatrix}
 k_1 & 0 & 0 & 0 & 0 & 0\\
 0 & k_2 & 0 & 0 & 0  & 0\\
 0 & 0 & k_3 & 0 & 0 & 0\\
 0 & 0 & 0 & k_4 & 0  &0\\
 0 & 0 & 0 & 0 & k_5 & 0 \\
  0 & 0 & 0 & 0 & 0 & k_6
\end{pmatrix} \begin{pmatrix}
X_1 \cdot X_3\\
 X_2\\
  X_2\\
  X_4 \cdot X_6\\
  X_5\\
  X_5
\end{pmatrix}
\end{align}
$$
that when we operate the matrices becomes

$$
\begin{align}
 \begin{bmatrix}
\frac{\mathrm{d} X_1}{\mathrm{d} t}\\ \frac{\mathrm{d} X_2}{\mathrm{d} t} \\ \frac{\mathrm{d} X_3}{\mathrm{d} t} \\ \frac{\mathrm{d} X_4}{\mathrm{d} t} \\ \frac{\mathrm{d} X_5}{\mathrm{d} t} \\ \frac{\mathrm{d} X_6}{\mathrm{d} t}\end{bmatrix}= \begin{bmatrix}
 -1 &  1 &  0 &  0 &  0 &  1\\
  1 & -1 & -1 &  0 &  0 &  0\\
 -1 &  1 &  1 &  0 &  0 &  0 \\
  0 &  0 &  1 & -1 &  1 &  0 \\
  0 &  0 &  0 &  1 & -1 & -1 \\
  0 &  0 &  0 & -1 &  1 &  1\end{bmatrix} \begin{pmatrix}
k_1 \cdot X_1 \cdot X_3\\
 k_2 \cdot X_2\\
  k_3 \cdot X_2\\
  k_4 \cdot X_4 \cdot X_6\\
  k_5 \cdot X_5\\
  k_6 \cdot X_5
\end{pmatrix}
\end{align}
$$

and operating further,

$$
\begin{align}
 \begin{bmatrix}
\frac{\mathrm{d} X_1}{\mathrm{d} t}\\ \frac{\mathrm{d} X_2}{\mathrm{d} t} \\ \frac{\mathrm{d} X_3}{\mathrm{d} t} \\ \frac{\mathrm{d} X_4}{\mathrm{d} t} \\ \frac{\mathrm{d} X_5}{\mathrm{d} t} \\ \frac{\mathrm{d} X_6}{\mathrm{d} t}\end{bmatrix}=  \begin{bmatrix}
-k_1 \cdot X_1 \cdot X_3 + k_2 \cdot X_2 + k_6 \cdot X_5\\
 k_1 \cdot X_1 \cdot X_3 - k_2 \cdot X_2  - k_3 \cdot  X_2\\
-k_1 \cdot X_1 \cdot X_3 + k_2 \cdot X_2 + k_3 \cdot  X_2\\
k_3 \cdot X_2 -  k_4 \cdot X_4 \cdot X_6 + k_5 \cdot X_5\\
  k_4 \cdot X_4 \cdot X_6 - k_5 \cdot X_5 - k_6 \cdot X_5\\
-  k_4 \cdot X_4 \cdot X_6 + k_5 \cdot X_5 + k_6 \cdot X_5
\end{bmatrix}
\end{align}
$$

 which,operating further and substituting for the original names of the variables, it gives us a set of six differential equations:

 $$\begin{align*}
  \dot{s}   &= -k_1 \cdot s \cdot e1 + k_2 \cdot e1s + k_6 \cdot e2p \\
  \dot{e1s} &= k_1 \cdot e1 \cdot s - (k_2 + k_3) \cdot e1s \\
  \dot{e1}  &= -k_1 \cdot s \cdot e1 + (k_2 + k_3) \cdot e1s \\
  \dot{p}   &= -k_4 \cdot p \cdot e2  + k_5 \cdot e2p + k_3 \cdot e1s \\
   \dot{e2p} &= k_4 \cdot e2 \cdot p - (k_5 + k_6) \cdot e2p \\
  \dot{e2}  &= -k_4 \cdot p \cdot e2 + (k_5 + k_6) \cdot e2p
  \end{align*}$$

With these equations we define an ODE vector function using the DSL notation:

```julia
enzyme_kinetics4! = @ode_def ab begin
  ds   = -k1 * s * e1 + k2 * e1s + k6 * e2p
  de1  = -k1 * s * e1 + (k2 + k3) * e1s
  de2  = -k4 * p * e2 + (k5 + k6) * e2p
  de1s =  k1 * e1 * s - (k2 + k3) * e1s
  de2p =  k4 * e2 * p - (k5 + k6) * e2p
  dp   = -k4 * p* e2  + k5 * e2p + k3 * e1s
    end k1 k2 k3 k4 k5 k6
```

    (::ab{var"#43#47",var"#44#48",var"#45#49",Nothing,Nothing,var"#46#50",Expr,Expr}) (generic function with 2 methods)

```julia
tspan = (0.0,100)
k1=0.5e3
k2=0.15
k3=0.03
k4=1e3
k5=0.1
k6=0.01

e10=0.002
e20=0.002
s0=0.002
p0=0.001

u0=[s0,e10,e20,0,0,p0]
p=[k1,k2,k3,k4,k5,k6];
```

```julia
prob4 = ODEProblem(enzyme_kinetics4!,u0,tspan,p)
```

    ODEProblem with uType Array{Float64,1} and tType Float64. In-place: true
    timespan: (0.0, 100.0)
    u0: [0.002, 0.002, 0.002, 0.0, 0.0, 0.001]

```julia
sol4 = solve(prob4)
plot(sol4)
title!("kinase-phosphatase")
xlabel!("Time [s]")
ylabel!("Concentration [M]")
#png("Michaelis_menten.png")
```

We can take advantage of the conservation of mass (as in systems with one enzyme) to go from six to four differential equations.

$$ \begin{align*}
C_1 \cdot X_1 + C_2 \cdot X_2 + C_3 \cdot X_3 + C_4 \cdot X_4 + C_5 \cdot X_5 + C_6 \cdot X_6 & = cte \\
\end{align*}$$

so
$$ \begin{align*}
C \cdot (B-A)^T &=0\\
\begin{pmatrix}C_1 & C_2 & C_3 & C_4 & C_5 & C_6\end{pmatrix} \begin{bmatrix}
 -1 &  1 &  0 &  0 &  0 &  1\\
  1 & -1 & -1 &  0 &  0 &  0\\
 -1 &  1 &  1 &  0 &  0 &  0 \\
  0 &  0 &  1 & -1 &  1 &  0 \\
  0 &  0 &  0 &  1 & -1 & -1 \\
  0 &  0 &  0 & -1 &  1 &  1\end{bmatrix}  &=0
 \end{align*}$$

 the solution of the system of equations is:
 $$ \begin{align*}
 -C_1 + C_2 - C_3= 0 &\implies  C_2 = C_3 + C_1\\
 -C_2 + C_3 + C_4= 0 &\implies  C_2 = C_3 + C_4\\
 - C_4 + C_5 - C_6 = 0 &\implies  C_5=C_4 + C_6\\
 C_1 - C_5 + C_6 =0 &\implies  C_5= C_1 + C_6
\end{align*}$$

Therefore, from the set of equations above, we obatin $C_1 = C_4$ (as in the case of one enzyem and one substrate. Therefore, the conservation of mass is:

$$ \begin{align*}
C_1 \cdot X_1(t) + (C_3 + C_1) \cdot X_2(t) + C_3 \cdot X_3(t) + C_1 \cdot X_4(t) +  (C_6 + C_1) \cdot X_5(t) + C_6 \cdot X_6(t)&= cte\\
C_1 \cdot X_1(t) + C_3 \cdot X_2(t) + C_1 \cdot X_2(t)  + C_3 \cdot X_3(t) + C_1 \cdot X_4(t) + C_6 \cdot X_5(t) + C_1 \cdot X_5(t) + C_6 \cdot X_6(t)&= cte\\
\end{align*}$$
which is valid for any value of $C_1$, $C_3$ and $C_6$, so, separating these dependencies

$$ \begin{align*}
C_1 \cdot X_1(t) + C_1 \cdot X_2(t) + C_1 \cdot X_4(t) + C_1 \cdot X_5 (t)&= cte\\
\end{align*}$$

which makes sense, since the $s$, $e1s$, $e2p$ and $p$ are four forms of the same protein, and their sum has to be constant, and equal to the initial value of substrate used. Using the value $C_1=1$ we obtain:

$$ \begin{align*}
X_1(t) + X_2(t) + X_4(t) + X_5 (t) &= cte = X_1(0) + X_2(0) + X_4(0) + X_5(0)=  X_1(0) + X_4(0)\\
\end{align*}$$

is our first conservation law (typically we start a biochemical reaction adding only the two substrates $s$ and $p$ and the two enzymes $e1$ and $e2$). For the second, we use the parameters that depend on the other constant $C_2$

$$ \begin{align*}
C_3 \cdot X_2(t) + C_3 \cdot X_3(t) &= cte
\end{align*}$$

This is the same conservation law that we obtained for the simple single enzyme-substrate system studiend previously. So, giving the value $C_3=1$, we obtain the conservation law,

$$ \begin{align*}
X_2(t) + X_3(t) & = cte &= X_2(0) + X_3(0) = X_3 (0)
\end{align*}$$

For the other constant $C_6$:
$$ \begin{align*}
C_6 \cdot X_5(t) + C_6 \cdot X_6(t) &= cte
\end{align*}$$

which is true because the enzyme $e2$ is also a catalyst that facilitates the reaction but does not react itself, and if we assume that there is no complex $e2p(0)=0$ before the reacton starts, we can assume ($e2(0)+e2p(0)=e2_0$).

This conservation law together with the previous one, allows us to reduce the system of 6 differential equations to just four  coupled ordinary differential equations:

 $$\begin{align*}
  \dot{s}   &= -k_1 \cdot s \cdot e1_0 + (k_2 + k_1 \cdot s) \cdot e1s + k_6 \cdot e2p \tag{52}\\
  \dot{e1s} &= k_1 \cdot e1_0 \cdot s - (k_2 + k_3 + k_1 \cdot s) \cdot e1s \tag{53}\\
  \dot{e2p} &= k_4 \cdot e2_0 \cdot p - (k_5 + k_6 + k_4 \cdot p) \cdot e2p \tag{54}\\
  \dot{p}   &= -k_4 \cdot p \cdot e2_0  + (k_5 + k_4 \cdot p) \cdot e2p + k_3 \cdot e1s \tag{55}
  \end{align*}$$

```julia
enzyme_kinetics5! = @ode_def ab begin
  ds   = -k1 * s * e10 + (k2 + k1 * s) * e1s + k6 * e2p
  de1s =  k1 * e10 * s - (k2 + k3 + k1 * s) * e1s
  de2p =  k4 * e20 * p - (k5 + k6 + k4 * p) * e2p
  dp   = -k4 * p * e20  + (k5 + k4 * p) * e2p + k3 * e1s
    end k1 k2 k3 k4 k5 k6 e10 e20
```

    (::ab{var"#51#55",var"#52#56",var"#53#57",Nothing,Nothing,var"#54#58",Expr,Expr}) (generic function with 2 methods)

```julia
u0=[s0,0,0,p0]
p=[k1,k2,k3,k4,k5,k6,e10,e20];
prob5 = ODEProblem(enzyme_kinetics5!,u0,tspan,p)
sol5 = solve(prob5)
plot(sol5,ylims = (0,s0+p0))
title!("kinase-phosphatase")
xlabel!("Time [s]")
ylabel!("Concentration [M]")
#png("Michaelis_menten.png")
```

The concentration of `e1` and `e2` can be also plotted by simply using the mass conservation:;

```julia
plot(sol5.t,[e10.-sol5[2,:],e20.-sol5[3,:]],label = ["e1" "e2"],ylims = (0,s0+p0))
```

## Quasi steady state approximation:

If we consider the pseudo-steady state approximation typical for biochemical systems (i.e., dynamic of intermediate complexes very fast compared with the other reactions),

$$
\begin{align*}
k_1 \cdot s(0) ≈ k_2 >> k_3 \tag{56}\\
k_4 \cdot p(0) ≈ k_5 >> k_6 \tag{57}
 \end{align*}$$

if this is true, we wan simplify the system by using the following rules:

$$ \frac{d e1s}{dt}= \frac{d e1}{dt}=\frac{d e2p}{dt}=\frac{d e2}{dt}=0 \tag{58}$$

so, the equilibrium concentrations for `e1s`and `e2p` are simply:

 $$\begin{align*}
  e1s &= \frac{k_1 \cdot e1_0 \cdot s}{k_2 + k_3 + k_1 \cdot s} \tag{59}\\
  e2p &= \frac{k_4 \cdot e2_0 \cdot p }{k_5 + k_6 + k_4 \cdot p} \tag{60}
  \end{align*}$$

```julia
e1s_eq=(p[1] .* p[7] .* sol5[1,:])./(p[2].+p[3].+p[1] .* sol5[1,:])
e2sp_eq=(p[4] .* p[8] .* sol5[4,:])./(p[5].+p[6].+p[4] .* sol5[4,:])

plot(sol5,label=["s" "e1s" "e2p" "p"])
title!("kinase-phosphatase")
xlabel!("Time [s]")
ylabel!("Concentration [M]")
#png("Michaelis_menten.png")

plot!(sol5.t,e1s_eq,
        ylims = (0,s0+p0),
        label="e1s_eq",
        linealpha = 0.5,
        linewidth = 3,
        linestyle= :dash,
        linecolor = :red)

plot!(sol5.t,e2sp_eq,
        label="e1s_eq",
        linealpha = 0.5,
        linewidth = 3,
        linestyle= :dash,
        linecolor = :green)

```

As we can see, the two equilibrium solutions are again quite close to the concentration of `e1s` and `e2p`, in this conditions we can simplify all equations as:

$$\begin{align*}
  \dot{s}   &= -k_1 \cdot s \cdot e1_0 + (k_2 + k_1 \cdot s) \cdot \frac{k_1 \cdot e1_0 \cdot s}{k_2 + k_3 + k_1 \cdot s} +  \frac{k_6 \cdot k_4 \cdot e2_0 \cdot p }{k_5 + k_6 + k_4 \cdot p} \tag{61}\\
  \dot{p}   &= -k_4 \cdot p \cdot e2_0  + (k_5 + k_4 \cdot p) \cdot \frac{k_4 \cdot e2_0 \cdot p }{k_5 + k_6 + k_4 \cdot p} +  \frac{ k_3 \cdot k_1 \cdot e1_0 \cdot s}{k_2 + k_3 + k_1 \cdot s} \tag{62}
\end{align*}$$

which, after some manipulation become

$$\begin{align*}
  \dot{s}   &= k_1 \cdot e1_0 \cdot  s \left [  \frac{k_2 + k_1 \cdot s}{k_2 + k_3 + k_1 \cdot s} -1\right ]  +  \frac{k_6 \cdot k_4 \cdot e2_0 \cdot p }{k_5 + k_6 + k_4 \cdot p} \tag{63}\\
  \dot{p}   &= k_4 \cdot e2_0 \cdot p   \left [  \frac{k_5 + k_4 \cdot p }{k_5 + k_6 + k_4 \cdot p} -1 \right ] +  \frac{ k_3 \cdot k_1 \cdot e1_0 \cdot s}{k_2 + k_3 + k_1 \cdot s} \tag{64}
\end{align*}$$

$$\begin{align*}
  \dot{s}   &= k_1 \cdot e1_0 \cdot  s \left [  \frac{k_2 + k_1 \cdot s- k_2 -  k_3 - k_1 \cdot s}{k_2 + k_3 + k_1 \cdot s} \right ]  +  \frac{k_6 \cdot k_4 \cdot e2_0 \cdot p }{k_5 + k_6 + k_4 \cdot p} \tag{65}\\
  \dot{p}   &= k_4 \cdot e2_0 \cdot p   \left [  \frac{k_5 + k_4 \cdot p -k_5 - k_6-  k_4 \cdot p}{k_5 + k_6 + k_4 \cdot p}  \right ] +  \frac{ k_3 \cdot k_1 \cdot e1_0 \cdot s}{k_2 + k_3 + k_1 \cdot s} \tag{66}
\end{align*}$$

$$\begin{align*}
  \dot{s}   &=  \frac{k_6 \cdot k_4 \cdot e2_0 \cdot p }{k_5 + k_6 + k_4 \cdot p}  - \frac{ k_3 \cdot k_1 \cdot e1_0 \cdot  s }{k_2 + k_3 + k_1 \cdot s}  \tag{67}  \\
  \dot{p}   &=   \frac{ k_3 \cdot k_1 \cdot e1_0 \cdot s}{k_2 + k_3 + k_1 \cdot s} -   \frac{k_6 \cdot k_4 \cdot e2_0 \cdot p}{k_5 + k_6 + k_4 \cdot p}   \tag{68}
\end{align*}$$

Finally, using the definition of the Michaelis-Menten constant for each enzyme-substrate pair, $K_{1M} = \frac{k_3+k_2}{k_1}$, $K_{2M} = \frac{k_5+k_6}{k_4}$, the previous equations can be simplified as:

$$\begin{align*}
  \dot{s}   &=   \frac{k_6 \cdot e2_0 \cdot p}{K_{2M} + p}   -  \frac{ k_3  \cdot e1_0 \cdot s}{K_{1M}+  s}    \tag{69}\\
  \dot{p}   &=   \frac{ k_3  \cdot e1_0 \cdot s}{K_{1M}+  s} -   \frac{k_6 \cdot e2_0 \cdot p}{K_{2M} + p}   \tag{70}
\end{align*}$$

```julia
enzyme_kinetics6! = @ode_def ab begin
  ds   =  (k6 *  e20 * p)/(K2M + p) - (k3 * e10 * s)/(K1M + s)
  dp   =  (k3 * e10 * s)/(K1M + s) - (k6 *  e20 * p)/(K2M + p)
    end k3 k6 e10 e20 K1M K2M
```

    (::ab{var"#59#63",var"#60#64",var"#61#65",Nothing,Nothing,var"#62#66",Expr,Expr}) (generic function with 2 methods)

```julia
u0=[s0,p0]
K1M=(k3+k2)/k1
K2M=(k5+k6)/k4
p=[k3,k6,e10,e20,K1M,K2M];
prob6 = ODEProblem(enzyme_kinetics6!,u0,tspan,p)
sol6 = solve(prob6)
plot(sol6,label=["s" "p"],ylims = (0,s0+p0))
title!("kinase-phosphatase full simplified")
xlabel!("Time [s]")
ylabel!("Concentration [M]")
#png("Michaelis_menten.png")
```

We can see that, comparing the last set of curves curves, that the dynamics predicted using the `pseudo-steady state condition` is far from the expected behavior of the dual enzyme-substrate. The approximation is much better when we change the chemical parameters of the reaction, and/or the initial concentration of enzyme.

```julia
k1=0.5e2
k2=0.015
k3=0.5
k4=1e2
k5=0.1
k6=0.01

e10=0.0002
e20=0.0002
s0=0.002
p0=0.001
tspan = (0.0,500)
u0=[s0,p0]
K1M=(k3+k2)/k1
K2M=(k5+k6)/k4
p=[k3,k6,e10,e20,K1M,K2M];
prob6 = ODEProblem(enzyme_kinetics6!,u0,tspan,p)
sol6 = solve(prob6)
plot(sol6,label=["s" "p"],ylims = (0,s0+p0))
title!("kinase-phosphatase full simplified")
xlabel!("Time [s]")
ylabel!("Concentration [M]")
#png("Michaelis_menten.png")
```

which if we compare it directly with the full systems without the pseudo-steady state aproximation:

```julia
u0=[s0,0,0,p0]

p=[k1,k2,k3,k4,k5,k6,e10,e20];
prob5 = ODEProblem(enzyme_kinetics5!,u0,tspan,p)
sol5 = solve(prob5)
plot(sol5,label=["s" "e1s" "e2p" "p"],ylims = (0,s0+p0))
title!("kinase-phosphatase")
xlabel!("Time [s]")
ylabel!("Concentration [M]")
#png("Michaelis_menten.png")
```

## Further simplification:

Now, with the system of just two equations, and in conditions of small concentration fo the intermediate complexes $e1s$ and $e2p$, we can use the other conservation law $s(t)+p(t)+e1s(t)+e2p(t)=cte$ to obtain a analyitical solution of the system. If we assume that the concetrations of $e1s(t)$ and $e2p(t)$ are small compared to the concetrations of $e1(t)$ and $e2(t)$, we can write  $s(t)+p(t)+e1s(t)+e2p(t)=s(0)+p(0)= $. Subsitituting this into the previous equations, we obtain:

$$\begin{align*}
  \dot{p}   &=   \frac{ k_3  \cdot e1_0 \cdot (s_0 + p_0 - p )}{K_{1M}+ s_0 + p_0 - p} -   \frac{k_6 \cdot e2_0 \cdot p}{K_{2M} + p} \tag{71}
\end{align*}$$

Now, we have only one equation, so we can calculate the equilibrium as
$$\begin{align*}
  \dot{p}   & =  0 \rightarrow  \frac{ k_3  \cdot e1_0 \cdot (s_0 + p_0 - p )}{K_{1M}+ s_0 + p_0 - p} =   \frac{k_6 \cdot e2_0 \cdot p}{K_{2M} + p} \tag{72}
\end{align*}$$

which is a second order equation with solution:

$$\frac{k_3  \cdot e1_0 \cdot (s_0 + p_0 - p )}{k_6 \cdot e2_0 \cdot p}= \frac{K_{1M}+ s_0 + p_0 - p}{K_{2M} + p }\tag{73}
$$

we can rename variables as:

$$W=\frac{k_3  \cdot e1_0 }{k_6 \cdot e2_0}\tag{74}$$
so

$$W\frac{(s_0 + p_0 - p )}{p}= \frac{K_{1M}+ s_0 + p_0 - p}{K_{2M} + p }\tag{75}
$$

$$W(s_0 + p_0 - p )(K_{2M} + p)= K_{1M} \cdot p+ s_0 \cdot p + p_0 \cdot p - p^2\tag{76}
$$

$$W(s_0 + p_0 - p )(K_{2M} + p)= (K_{1M} +p_0 +s_0) p - p^2\tag{77}
$$

$$W (s_0 \cdot K_{2M} + p_0 \cdot K_{2M} - p \cdot K_{2M} + s_0 \cdot p + p_0 \cdot p - p^2)= (K_{1M} +p_0 +s_0) p - p^2\tag{78}
$$

$$W (s_0 \cdot K_{2M} + p_0 \cdot K_{2M}  + (s_0 + p_0 -\cdot K_{2M}) p - p^2)= (K_{1M} +p_0 +s_0) p - p^2\tag{79}
$$

$$  W \cdot s_0 \cdot K_{2M} + W \cdot p_0 \cdot K_{2M}  + (s_0 + p_0 -\cdot K_{2M}) W \cdot p - W \cdot p^2= (K_{1M} +p_0 +s_0) p - p^2\tag{80}
$$

$$  W \cdot s_0 \cdot K_{2M} + W \cdot p_0 \cdot K_{2M}  + (s_0 + p_0 -\cdot K_{2M}) W \cdot p + p^2(1-W)= (K_{1M} +p_0 +s_0) p \tag{81}
$$

$$  W \cdot s_0 \cdot K_{2M} + W \cdot p_0 \cdot K_{2M}  + (s_0 W+ p_0 W- W\cdot K_{2M} -K_{1M} -p_0 -s_0)  \cdot p + p^2(1-W)= 0 \tag{82}
$$

$$  W \cdot s_0 \cdot K_{2M} + W \cdot p_0 \cdot K_{2M}  + (s_0 (W-1) + p_0 (W-1) - W \cdot K_{2M} -K_{1M})  \cdot p + p^2(1-W)= 0 \tag{83}
$$

 thsi is a second order equation $A \cdot  P^2 + B \cdot p +C=0$ with

$$A=1-W \tag{84}$$

$$B=s_0 (W-1) + p_0 (W-1) - W \cdot K_{2M} -K_{1M} \tag{85}$$

$$C=W \cdot s_0 \cdot K_{2M} + W \cdot p_0 \cdot K_{2M} \tag{86}$$

```julia
function quadratic(a, b, c)
          discr = b^2 - 4*a*c
          discr >= 0 ?   ( (-b + sqrt(discr))/(2a), (-b - sqrt(discr))/(2a) ) : error("Only complex roots")
        end
```

    quadratic (generic function with 1 method)

```julia
W=k3*e10/k6/e20
A=1-W
B=s0*(W-1)+p0*(W-1)-W*K2M-K1M
C=W*s0*K2M+W*p0*K2M
```

    0.00016500000000000003

```julia
roots=quadratic(A,B,C)
roots[2]
```

    0.002849202777618357

```julia
plot(sol6,label=["s" "p"],ylims = (0,s0+p0))
title!("kinase-phosphatase full simplified steady state")
xlabel!("Time [s]")
ylabel!("Concentration [M]")
hline!([roots[2]],label=" \\ P_{ss}")
```

## Conclusion

Enzyme kinetics is one of the most basics sistems where Mass Action combined with mathematical modeling allows us to extract relevant information of the dynamics of the interactions that shape the responses of biological systems


# Linear Discriminant Functions
Linear Discriminant Functions distinguish (in the simple case) two different classes from each other (assuming they are linear seperable). For the twodimensional case (N = 2), we want to find a line $d(x) = w_1x_1+w_2x_2+w_3 = 0$ which outputs a value $>0$ for the first class and a value $<0$ for the second class. If the value is 0 (and thus the point lies on the decision line), we reject the decision*.

The goal is to learn the weights ($w_1$ and $w_2$) as well as the bias to discriminate the two classes. There are many algorithms to achieve this. In the lecture by Prof. Jiang two different approaches were shown: The perceptron algorithm and the pseudoinverse algorithm.

## Perceptron Algorithm
The perceptron algorithm minimizes the number of misclassified training examples by using the gradient descend approach. This is done by using the model of a perceptron which has (in the twodimensional case) two inputs and a bias. The output of the perceptron is the sum of the inputs combined with some weights which need to be learned. For learning we use our initial solution and compute for every training example the output of the neuron. Every misclassified sample causes a slight change to the weights until no misclassification happens.

$$
\begin{align}
J(w) &= \sum_{y \in \text{misclassified examples}}(-wy^t)\\
\nabla J(w) &= \sum_{y \in \text{misclassified examples}}(-y)\\
w_{k+1} &= w_k - \sum_{y \in \text{misclassified examples}}(-y)\\
w_{k+1} &= w_k + \sum_{y \in \text{misclassified examples}}(y)
\end{align}
$$

#### Example
Consider the following data (I already added the bias)

| x | y | bias | class  | $d(w)$ shall be... |
|---|---|------|--------|--------------------|
| 0 | 0 | 1    |  $C_1$ | $>0$               |
| 0 | 1 | 1    | $C_1$  | $>0$               |
| 1 | 0 | 1    | $C_2$  | $<0$               |
| 1 | 1 | 1    | $C_2$  | $<0$               |

For simplicity reasons we multiply the values of the second class * -1 so we get

| x  | y  | bias | class  | $d(w)$ shall be... |
|----|----|------|--------|--------------------|
| 0  | 0  | 1    |  $C_1$ | $>0$               |
| 0  | 1  | 1    | $C_1$  | $>0$               |
| -1 | 0  | -1   | $C_2$  | $>0$               |
| -1 | -1 | -1   | $C_2$  | $>0$               |

Let's imagine we initialize with $w_0 = (0,0,0)$. Now we can find out if an example is correctly classified or not, by multiplying the sample with the weight.

| x  | y  | bias | class  | $d(w)$ shall be... | $d(w_0)$ is be... |
|----|----|------|--------|--------------------|-----------------|
| 0  | 0  | 1    |  $C_1$ | $>0$               | $\leq 0$        |
| 0  | 1  | 1    | $C_1$  | $>0$               | $\leq 0$        |
| -1 | 0  | -1   | $C_2$  | $>0$               | $\leq 0$        |
| -1 | -1 | -1   | $C_2$  | $>0$               | $\leq 0$        |

As we can see, all four values are misclassified. Thus we can sum up all x's and y's and biases and we get as a $\delta_0 = (-2,0,0)$. Now we calculate $w_1 = w_0 + p * \delta_0 = (0,0,0) + 1*(-2,0,0) = (-2,0,0)$, assuming a learning rate $p$ of $1$.

For the second iteration we use the new weight and get...

| x  | y  | bias | class  | $d(w)$ shall be... | $d(w_1)$ is be... |
|----|----|------|--------|--------------------|-------------------|
| 0  | 0  | 1    |  $C_1$ | $>0$               | $\leq 0$          |
| 0  | 1  | 1    | $C_1$  | $>0$               | $\leq 0$          |
| -1 | 0  | -1   | $C_2$  | $>0$               | $> 0$             |
| -1 | -1 | -1   | $C_2$  | $>0$               | $> 0$             |

Now there are only two misclassified examples (the first and the second one). If we now add up the two misclassified examples, we get $\delta_1=(0,0,1)+(0,1,1)=(0,1,2)$. Secondly, we get $w_2 = w_1 + p * \delta_1 = (-2,0,0) + 1 * (0,1,2) = (-2,1,2)$. The algorithm continues until it converges to $(-4,1,2)$.

## Pseudoinverse Algorithm
The pseudoinverse algorithm works by introducing a vector $b$, so that
$$
wX = b
$$
with $b$ having positive values. Thus, we replace the requirement to be greater than zero with concrete values.

$X$ is a matrix with the trainingsdata, i.e.

$$
X=
  \begin{pmatrix}
    0 & 0 & -1 & -1 \\
    0 & 1 & 0 & -1 \\
    1 & 1 & -1 & -1
  \end{pmatrix}
$$

In this case the first column corresponds to the first example, the second to the second example etc. The third and the forth example are again multiplied with -1 since they belong to class 2 (just as before).

To calculate the corresponding $w$ we need to have the inverse of $X$, however there is no exact solution, so instead we calculate.

$$
w = bX^t(XX^t)^{-1}
$$

The challenge here is to determine a suitable b. You might need mulitple trials or another optimization strategy for b.

# Code-Part
### Perceptron Algorithm

```python
# First we import the required packages

import numpy as np

import matplotlib.pyplot as plt
from matplotlib import animation, rc

from IPython.display import HTML

from numpy.linalg import inv # For pseudoinverse algorithm
```

Now we set the data and the learning rate.

```python
# Two classes
c1 = np.array([[0,0],[0,1]])
c2 = np.array([[1,0],[1,1]])
learning_rate = 1

# Assertion check, some kind of input validation
assert c1[0].shape == c2[0].shape
```

Now we ...
* initialize the weights vector w with (0,0,0)
* add the bias terms to the samples
* multiply the samples of class 2 with -1 as explained above.

```python
# Initialize w (i.e. with zeros)
w = np.zeros(c1[0].shape[0] + 1) #+1 for bias

# Add bias-term for classes
_c1 = np.append(c1, np.repeat(1,c1[0].shape[0]).reshape((c1[0].shape[0], 1)), axis=1)
_c2 = np.append(c2, np.repeat(1,c1[0].shape[0]).reshape((c1[0].shape[0], 1)), axis=1)

# Multiply _c2*-1 so we can check if > 0 more conveniently
_c2 = _c2*-1
```

Now this is the main loop. The main functionality is described above, not more magic is happening here. We save each weight in weights, so we can visualize the progress of the linear discriminant function later.

```python
# Main loop

weights = [w] # We save the weights, so we can display the line fitting later

while True:

    # Test which samples are wrong classified
    wrong_classified = []

    for sample in _c1:
        # Calculate the function value (i.e. w0 + w1*x1 + w2*x2 + ...)
        # It is correctly classified if the value is > 0, thus wrongly if it is <= 0
        if sum(sample*w) <= 0:
            wrong_classified.append(sample)

    for sample in _c2:
        # Same for _c2
        if sum(sample*w) <= 0:
            wrong_classified.append(sample)

    # Break if no samples were wrong classified
    if len(wrong_classified) == 0:
        break

    # Otherwise calculate delta
    delta = np.sum(wrong_classified, axis=0)

    # Add delta to weight
    w = w + learning_rate*delta
    weights.append(w)
```

This part is just for creating the visualization of the data and the discriminant function. It creates a fancy animation. This works by using the weights which have been previously saved.

```python
fig, ax = plt.subplots()
ax.plot(c1[:,0], c1[:,1], 'ro')
ax.plot(c2[:,0], c2[:,1], 'bo')
line, = ax.plot([], [])

ax.set_xlim(( -2, 2)) # Set x axis here
ax.set_ylim((-2, 2)) # Set y axis here

def init():
    return (line,)

def animate(i):
    x = np.linspace(*ax.get_xlim(), 1000)
    weight = weights[i]
    #w0x1 + w1x2 + w2 = 0
    # w1x2 = -w2 - w0x1
    # x2 = (-w2 -w0*x1) / w1
    if weight[1] == 0: # This would be a divide by zero.
        y = [0] * len(x)
    else:
        y = (-weight[0]*x-weight[2])/weight[1]
    line.set_data(x, y)
    return (line,)

anim = animation.FuncAnimation(fig, animate, init_func=init,
                               frames=len(weights), interval=1000,
                               blit=True)
HTML(anim.to_jshtml()) # Print the animation
```

<link rel="stylesheet"
href="https://maxcdn.bootstrapcdn.com/font-awesome/4.4.0/
css/font-awesome.min.css">

    <input id="_anim_slider2b16e4dfcbd3421daf4aa278a03bd82b" type="range" style="width:350px"
           name="points" min="0" max="1" step="1" value="0"
           onchange="anim2b16e4dfcbd3421daf4aa278a03bd82b.set_frame(parseInt(this.value));"></input>

    <button onclick="anim2b16e4dfcbd3421daf4aa278a03bd82b.slower()"></button>
    <button onclick="anim2b16e4dfcbd3421daf4aa278a03bd82b.first_frame()">
        </button>
    <button onclick="anim2b16e4dfcbd3421daf4aa278a03bd82b.previous_frame()">
        </button>
    <button onclick="anim2b16e4dfcbd3421daf4aa278a03bd82b.reverse_animation()">
        </button>
    <button onclick="anim2b16e4dfcbd3421daf4aa278a03bd82b.pause_animation()">
        </button>
    <button onclick="anim2b16e4dfcbd3421daf4aa278a03bd82b.play_animation()">
        </button>
    <button onclick="anim2b16e4dfcbd3421daf4aa278a03bd82b.next_frame()">
        </button>
    <button onclick="anim2b16e4dfcbd3421daf4aa278a03bd82b.last_frame()">
        </button>
    <button onclick="anim2b16e4dfcbd3421daf4aa278a03bd82b.faster()"></button>
  <form action="#n" name="_anim_loop_select2b16e4dfcbd3421daf4aa278a03bd82b" class="anim_control">
    <input type="radio" name="state"
           value="once" > Once </input>
    <input type="radio" name="state"
           value="loop" checked> Loop </input>
    <input type="radio" name="state"
           value="reflect" > Reflect </input>
  </form>

### Pseudoinverse Part
This is the part where we calculate the pseudoinverse. However, we do not optimize b. This is part of other algorithms, i.e. Ho-Kashyap algorithm. Instead, we set $b$ to a vector of ones.

```python
## Define X

#(_c1.shape[0] + _c2.shape[0]) = Number of training examples
X = np.array([_c1,_c2]).reshape((_c1.shape[0] + _c2.shape[0]),3).T
b = np.ones((1,(_c1.shape[0] + _c2.shape[0]))) # Just fill b with ones (one 1 per training example)

# Calculate the w with the pseudoinverse
# We need to use the dotproduct, so we cannot write X*X.T, but i.e.
# np.dot(X,X.T)
w = np.dot(b, np.dot(X.T, inv(np.dot(X,X.T))))

# For simplicity reasons we flatten the matrix to a vector
w = w.flatten()
```

Here we just visualize again

```python
plt.clf()
fig, ax = plt.subplots()
ax.plot(c1[:,0], c1[:,1], 'ro')
ax.plot(c2[:,0], c2[:,1], 'bo')
line, = ax.plot([], [])

ax.set_xlim(( -2, 2)) # Set x axis here
ax.set_ylim((-2, 2)) # Set y axis here

x = np.linspace(*ax.get_xlim(), 1000)
y = (-w[0]*x-w[2])/w[1]
line.set_data(x, y)
plt.show()
```

#### Literature
* Haykin, Simon S., et al. Neural networks and learning machines. Vol. 3. Upper Saddle River, NJ, USA:: Pearson, 2009.
* Duda, Richard O., Peter E. Hart, and David G. Stork. Pattern classification. John Wiley & Sons, 2012.


# Introduction to Matrices
In general terms, a matrix is an array of numbers that are arranged into rows and columns.

## Matrices and Matrix Notation
A matrix arranges numbers into rows and columns, like this:

\begin{equation}A = \begin{bmatrix}
  1 & 2 & 3 \\
  4 & 5 & 6
 \end{bmatrix}
\end{equation}

Note that matrices are generally named as a capital letter. We refer to the *elements* of the matrix using the lower case equivalent with a subscript row and column indicator, like this:

\begin{equation}A = \begin{bmatrix}
  a_{1,1} & a_{1,2} & a_{1,3} \\
  a_{2,1} & a_{2,2} & a_{2,3}
 \end{bmatrix}
\end{equation}

In Python, you can define a matrix as a 2-dimensional *numpy.**array***, like this:

```python
import numpy as np

A = np.array([[1,2,3],
              [4,5,6]])
print (A)
```

    [[1 2 3]
     [4 5 6]]

You can also use the *numpy.**matrix*** type, which is a specialist subclass of ***array***:

```python
import numpy as np

M = np.matrix([[1,2,3],
               [4,5,6]])
print (M)
```

    [[1 2 3]
     [4 5 6]]

There are some differences in behavior between ***array*** and ***matrix*** types - particularly with regards to multiplication (which we'll explore later). You can use either, but most experienced Python programmers who need to work with both vectors and matrices tend to prefer the ***array*** type for consistency.

## Matrix Operations
Matrices support common arithmetic operations.

### Adding Matrices
To add two matrices of the same size together, just add the corresponding elements in each matrix:

\begin{equation}\begin{bmatrix}1 & 2 & 3 \\4 & 5 & 6\end{bmatrix}+ \begin{bmatrix}6 & 5 & 4 \\3 & 2 & 1\end{bmatrix} = \begin{bmatrix}7 & 7 & 7 \\7 & 7 & 7\end{bmatrix}\end{equation}

In this example, we're adding two matrices (let's call them ***A*** and ***B***). Each matrix has two rows of three columns (so we describe them as 2x3 matrices). Adding these will create a new matrix of the same dimensions with the values a1,1 + b1,1, a1,2 + b1,2, a1,3 + b1,3,a2,1 + b2,1, a2,2 + b2,2, and a2,3 + b2,3. In this instance, each pair of corresponding elements(1 and 6, 2, and 5, 3 and 4, etc.) adds up to 7.

Let's try that with Python:

```python
import numpy as np

A = np.array([[1,2,3],
              [4,5,6]])
B = np.array([[6,5,4],
              [3,2,1]])
print(A + B)
```

    [[7 7 7]
     [7 7 7]]

### Subtracting Matrices
Matrix subtraction works similarly to matrix addition:

\begin{equation}\begin{bmatrix}1 & 2 & 3 \\4 & 5 & 6\end{bmatrix}- \begin{bmatrix}6 & 5 & 4 \\3 & 2 & 1\end{bmatrix} = \begin{bmatrix}-5 & -3 & -1 \\1 & 3 & 5\end{bmatrix}\end{equation}

Here's the Python code to do this:

```python
import numpy as np

A = np.array([[1,2,3],
              [4,5,6]])
B = np.array([[6,5,4],
              [3,2,1]])
print (A - B)
```

    [[-5 -3 -1]
     [ 1  3  5]]

#### Conformability
In the previous examples, we were able to add and subtract the matrices, because the *operands* (the matrices we are operating on) are ***conformable*** for the specific operation (in this case, addition or subtraction). To be conformable for addition and subtraction, the operands must have the same number of rows and columns. There are different conformability requirements for other operations, such as multiplication; which we'll explore later.

### Negative Matrices
The nagative of a matrix, is just a matrix with the sign of each element reversed:

\begin{equation}C = \begin{bmatrix}-5 & -3 & -1 \\1 & 3 & 5\end{bmatrix}\end{equation}

\begin{equation}-C = \begin{bmatrix}5 & 3 & 1 \\-1 & -3 & -5\end{bmatrix}\end{equation}

Let's see that with Python:

```python
import numpy as np

C = np.array([[-5,-3,-1],
              [1,3,5]])
print (C)
print (-C)
```

    [[-5 -3 -1]
     [ 1  3  5]]
    [[ 5  3  1]
     [-1 -3 -5]]

### Matrix Transposition
You can *transpose* a matrix, that is switch the orientation of its rows and columns. You indicate this with a superscript **T**, like this:

\begin{equation}\begin{bmatrix}1 & 2 & 3 \\4 & 5 & 6\end{bmatrix}^{T} = \begin{bmatrix}1 & 4\\2 & 5\\3 & 6 \end{bmatrix}\end{equation}

In Python, both *numpy.**array*** and *numpy.**matrix*** have a **T** function:

```python
import numpy as np

A = np.array([[1,2,3],
              [4,5,6]])
print(A.T)
# or use
print('\n', A.transpose())
```

    [[1 4]
     [2 5]
     [3 6]]

     [[1 4]
     [2 5]
     [3 6]]

```python

```


```python
# Geometric Properties of Hollow Rectangle
# E.Durham     5-Jul-2019
```

```python
from sympy import *
# from sympy import symbols
```

```python
# define symbols
b, d, t = symbols('b d t')
```

```python
A = 2*t*(d+b-2*t) # Area
I_x = (b*d**3 - (b-2*t) * (d-2*t)**3) / 12 # Second Moment of Inertia for Major Axis
I_y = (d*b**3 - (d-2*t) * (b-2*t)**3) / 12 # Second Moment of Inertia for Minor Axis
S_x = (b*d**3 - (b-2*t) * (d-2*t)**3) / (6*d) # Elastic Section Modulus for Major Axis
S_y = (d*b**3 - (d-2*t) * (b-2*t)**3) / (6*b) # Elastic Section Modulus for Minor Axis
r_x = sqrt((b*d**3 - (b-2*t) * (d-2*t)**3) / (24*t * (d+b-2*t))) # Radius of Gyration for Major Axis
r_y = sqrt((d*b**3 - (d-2*t) * (b-2*t)**3) / (24*t * (b+d-2*t))) # Radius of Gyration for Minor Axis
Z_x = b*t*(d-t) + 2*t * ((d/2)-t)**2 # Plastic Section Modulus for Major Axis
Z_y = d*t*(b-t) + 2*t * ((b/2)-t)**2 # Plastic Section Modulus for Minor Axis
J = (2*t*(d-t)**2 * (b-t)**2) / (d+b-2*t) # St. Venant's Torsional Constant
```

```python
# Specify Depth, d, Width, b, AND Wall Thickness, t in identical units
# Example:
# if case values are: d = 3.00, b = 2.00 and t = 0.250
# then use case_values = [(d, 3.00), (b, 2.00), (t, 0.250)]
# Enter actual case values below per above
case_values = [(d, 3.00), (b, 2.00), (t, 0.250)]
```

```python
print('Geometric Properties:')
print('Given:')
print('Depth, ', case_values[0], 'unit')
print('Width, ', case_values[1], 'unit')
print('Wall, ',  case_values[2], 'unit')
print('A =', A.subs(case_values).evalf(3), 'unit**2')
print('I_x =', I_x.subs(case_values).evalf(3), 'unit**4')
print('S_x =', S_x.subs(case_values).evalf(3), 'unit**3')
print('Z_x =', Z_x.subs(case_values).evalf(3), 'unit**3')
print('r_x =', r_x.subs(case_values).evalf(3), 'unit')
print('I_y =', I_y.subs(case_values).evalf(3), 'unit**4')
print('S_y =', S_y.subs(case_values).evalf(3), 'unit**3')
print('Z_y =', Z_y.subs(case_values).evalf(3), 'unit**3')
print('r_y =', r_y.subs(case_values).evalf(3), 'unit')
print('J =', J.subs(case_values).evalf(3), 'unit**4')
```

    Geometric Properties:
    Given:
    Depth,  (d, 3.0) unit
    Width,  (b, 2.0) unit
    Wall,  (t, 0.25) unit
    A = 2.25 unit**2
    I_x = 2.55 unit**4
    S_x = 1.70 unit**3
    Z_x = 2.16 unit**3
    r_x = 1.06 unit
    I_y = 1.30 unit**4
    S_y = 1.30 unit**3
    Z_y = 1.59 unit**3
    r_y = 0.759 unit
    J = 2.57 unit**4

```python
# Display formulas in proper math formatting
init_printing()
print('Area , A ='); A

```

```python
print('Second Moment of Inertia, I ='); I_x
```

```python
print('Elastic Section Modulus, S ='); S_x
```

```python
print('Plastic Section Modulus, Z ='); Z_x
```

```python
print('Radius of Gyration, r ='); r_x
```

```python
print("St. Venant's Torsional Constant, J ="); J
```

## Test Values and Expected Results
### RT 3 x x 1/4 from Aluminum Design Manual, 2015 page V-39
d = 3
b = 2
t = 0.25
A = 2.25
I_x = 2.55
S_x = 1.70
Z_x = 2.16
r_x = 1.06
I_y = 1.30
S_y = 1.30
Z_y = 1.59
r_y = 0.759
J = 2.57

## Glossary of Torsional Terms [1]
### HSS Shear Constant
The shear constant, $C_{RT}$, is used for calculating the maximum shear stress due to an applied shear force.
For hollow structural section, the maximum shear stress in the cross section is given by:
$\tau_{max} = \frac{V Q}{2 t I}$
where $V$ is the applied shear force, $Q$ is the statical moment of the portion of the section lying outside the neutral axis taken about the neutral axis, $I$ is the moment of inertia, and $t$ is the wall thickness.
The shear constant is expressed as the ratio of the applied shear force to the maximum shear stress [3]:
$C_{RT} = \frac{V}{\tau_{max}} = \frac{2 t I}{Q}$
### HSS Torsional Constant
The torsional constant, $C$, is used for calculating the shear stress due to an applied torqu. It is expressed as the ratio of the applied torque, $T$, to the shear stress in the cross section, $\tau$:
$C = \frac{T}{\tau}$

### St. Venant Torsional Constant
The St. Venant torsional constant, $J$, measures the resistance of a structural member to *pure* or *uniform* torsion. It is used in calculating the buckling moment resistance of laterally unsupported beams and torsional-flexural buckling of compression members in accordance with CSA S16.

For open cross section, the general formula is given by Galambos (1968):
$J = \sum(\frac{b't^3}{3})$
where $b'$ are the plate lengths between points of intersection on their axes, and $t$ are the plate thicknesses. Summation includes all component plates. It is noted that the tabulated values in the Handbook of Steel Construction are based on net plate lengths instead of lengths between intersection points, a mostly conservative approach.

The expressions for $J$ given herein do not take into account the flange-to-web fillets. Formulas which account for this effect are given by El Darwish and Johnston (1965).

For thin-walled closed sections, the general formula is given by Salmon and Johnson (1980):
$J = \frac{4 A_o^2}{\int_s ds/t}$

where $A_o$ is the enclosed area by the walls, $t$ is the wall thickness, $ds$ is a length element along the perimeter. Integration is performed over the entire perimeter $S$.

### Warping Torsional Constant
The warping torsional constant, $C_w$, measures the resistance of a structural member to *nonuniform* or *warping* torsion. It is used in calculating the buckling moment resistance of laterally unsupported beams and torsional-flexural buckling of compression members in accordance with CSA Standard S16.

For open section, a general calculation method is given by Galambos (1968). For section in which all component plates meet at a single poitn, such as angles and T-sections, a calculation method is given by Bleich (1952). For hollow structural sections (HSS), warping deformations are small, and the warping torsional constant is generally taken as zero.

### References for Torsional Constants
[1] CISC, 2002. Torsional Section Properties of Steel Shapes.
Canadian Institute of Steel Construction, 2002

[2] Seaburg, P.A. and Carter, C.J. 1997. Torsional Analysis of Structural Steel Members.
American Institute of Steel Construction, Chicago, Ill.

[3] Stelco. 1981. Hollow Structural Sections - Sizes and Section Properties, 6th Edition.
Stelco Inc., Hamilton, Ont.

[4] CISC 2016. Handbook of Steel Construction, 11th Edition, page 7-84
www.cisc-icca.ca

### Revision History
0.1 - 2019-07-08 - E.Durham - added graphic image
0.0 - 2019-07-05 - E.Durham - Created initial notebook