# Symbolic Mathematics in Python

There are times when you need to solve a difficult problem symbollically or analytically. If you have ever used Wolfram Alpha, then you have already done this. Sympy is a python library that allows you to do symbolic mathematics in python.

```python
import sympy as sym
```

## 1. Introduction

### Example 1.1

If you try to write the follwing in python by itself, you will get an error telling you x is undefined:
$$x-x$$

```python
x-x
```

(The error above is on purpose). Variables in python need to be defined before you can say something specific about them

```python
x=102
x-x
```

    0

If you are trying to show that $x-x=0$ is true for any $x$, the above answer would not be valid. Instead you can use a symbolic expression to show that it is true

**First we define the variable as a symmbolic expression**

```python
x = sym.symbols('x')
```

**Now we can use the variable in a symbolic expression**

```python
x-x
```

    0

### Example 1.2

Sympy can be used to perform algebreic operations (among other things). Consider the following expression: $$(3a-4b)^3$$

We can use symppy to expand the expression algebraically.

**First we need to define the variables as symbolic expressions**

```python
import sympy as sym
a,b = sym.symbols('a,b')
```

**Side note** Notice that the left hand side of the epression has two variables being defined. Python can define more than one variable at a time:

```python
x1,y1 =10,20
print(x1)
print(y1)
```

    10
    20

**Back to the expression** We can define an expression using the variables $a$ and $b$.

```python
expr = (3*a-4*b)**3
print(expr)
```

    (3*a - 4*b)**3

We can also make it look nicer in our notebook. This doesn't affect the math, but it makes our notebook more readable.

```python
sym.init_printing()
```

```python
import sympy as sym

a, b = sym.symbols("a, b")
expr = (3*a-4*b)**3
sym.init_printing(expr)
print(expr)
```

    (3*a - 4*b)**3

**Now we expand the function algebreically**

```python
expr.expand()
```

Sympy can also factor the equation

```python
sym.factor(26*a**3-108*a**2*b+144*a*b**2-64*b**3)
```

If you want to copy and paste a result, you print the result.

```python
print(sym.factor(26*a**3-108*a**2*b+144*a*b**2-64*b**3))
```

    2*(a - 2*b)*(13*a**2 - 28*a*b + 16*b**2)

You can also chain together functions

```python
expr.expand().factor()
```

### Exercise 1.1

Show that the following two expressions are true.
$$(2w-3z)(2w+3z)=4w^2-9z^2$$
$$(2w-3z)^2\ne4w^2-9z^2$$

```python
import sympy as sym
w,z = sym.symbols('w,z')
expr1 = (2*w-3*z)*(2*w+3*z)
expr2 = (2*w-3*z)**2

expr1.expand()
expr2.expand()

print(expr1.expand())
print(expr2.expand())
```

    4*w**2 - 9*z**2
    4*w**2 - 12*w*z + 9*z**2

## 2. Solving Equations

Sympy can be used to symbolilically solve equations. As before, you need to define which variables are symbols

### Example 2.1

Use sympy to solve the following equation
$$ax^3+bx^2+cx+d=0$$

```python
# Define the variables
a,b,c,d,x = sym.symbols('a,b,c,d,x')
```

```python
# Define the expression
expr=a*x**3+b*x**2+c*x+d
expr
```

We can use the `solvset` function to solve this equation

```python
solutions=sym.solveset(expr,x)
```

```python
print(solutions)
```

    {-(-3*c/a + b**2/a**2)/(3*(sqrt(-4*(-3*c/a + b**2/a**2)**3 + (27*d/a - 9*b*c/a**2 + 2*b**3/a**3)**2)/2 + 27*d/(2*a) - 9*b*c/(2*a**2) + b**3/a**3)**(1/3)) - (sqrt(-4*(-3*c/a + b**2/a**2)**3 + (27*d/a - 9*b*c/a**2 + 2*b**3/a**3)**2)/2 + 27*d/(2*a) - 9*b*c/(2*a**2) + b**3/a**3)**(1/3)/3 - b/(3*a), -(-3*c/a + b**2/a**2)/(3*(-1/2 - sqrt(3)*I/2)*(sqrt(-4*(-3*c/a + b**2/a**2)**3 + (27*d/a - 9*b*c/a**2 + 2*b**3/a**3)**2)/2 + 27*d/(2*a) - 9*b*c/(2*a**2) + b**3/a**3)**(1/3)) - (-1/2 - sqrt(3)*I/2)*(sqrt(-4*(-3*c/a + b**2/a**2)**3 + (27*d/a - 9*b*c/a**2 + 2*b**3/a**3)**2)/2 + 27*d/(2*a) - 9*b*c/(2*a**2) + b**3/a**3)**(1/3)/3 - b/(3*a), -(-3*c/a + b**2/a**2)/(3*(-1/2 + sqrt(3)*I/2)*(sqrt(-4*(-3*c/a + b**2/a**2)**3 + (27*d/a - 9*b*c/a**2 + 2*b**3/a**3)**2)/2 + 27*d/(2*a) - 9*b*c/(2*a**2) + b**3/a**3)**(1/3)) - (-1/2 + sqrt(3)*I/2)*(sqrt(-4*(-3*c/a + b**2/a**2)**3 + (27*d/a - 9*b*c/a**2 + 2*b**3/a**3)**2)/2 + 27*d/(2*a) - 9*b*c/(2*a**2) + b**3/a**3)**(1/3)/3 - b/(3*a)}

```python
solutions
```

What if I need help. You can do this with any python function. `function?`

```python
# Run this command to see a help box
sym.solveset?
```

### Exercise 2.1

Use the `solveset` function to solve the following chemical problem.

Phosgene gas, $\text{COCl}_2$, dissociates at high temperatures according to the following equilibrium:

$$ \text{COCl}_2 \rightleftharpoons  \text{CO} + \text{Cl}_2 $$

At $\text{400 C}$, the equilibrium constant $K_c=8.05$.

If you start with a  $\text{0.250 M}$ phosgene sample at $\text{400 C}$, determine the concentrations of all species at equilibrium.

```python
# Your code here
import sympy as sym
x = sym.symbols('x')
expr = ((x**2)/(.25-x)) - 8.05

solutions=sym.solveset(expr,x)
print(solutions)

solutions
```

Why did you pick your answer?
#Because concentration must be positive#

## 3. Calculus

We can use also Sympy to differentiate and integrate. Let us experiment with differentiating the following expression:

$$x ^ 2 - \cos(x)$$

```python
sym.diff(x ** 2 - sym.cos(x), x)
```

Similarly we can integrate:

```python
sym.integrate(x ** 2 - sym.cos(x), x)
```

We can also carry out definite integrals:

```python
sym.integrate(x ** 2 - sym.cos(x), (x, 0, 5))
```

### Exercise 3.1

Use Sympy to calculate the following:

1. $\frac{d(x ^2 + xy - \ln(y))}{dy}$
1. $\int_0^5 e^{2x}\;dx$

```python
# Exercise 3.1-1
import sympy as sym
x,y = sym.symbols('x,y')

sym.diff(x**2+x*y-sym.ln(y))/sym.diff(y)

```

```python
# Exercise 3.1-2

```

### Exercise 3.2
Solve the following definate integral
$$\int\limits_{ - \infty }^\infty  {\frac{1}{{\sigma \sqrt {2\pi } }}{e^{ - \frac{1}{2}{{\left( {\frac{{x - \mu }}{\sigma }} \right)}^2}}}}$$

Hint, the sympy symbol for infinity is `oo`

```python
# Your code here
```

```python

```

Lookup Gaussian functions: https://en.wikipedia.org/wiki/Gaussian_function
Does your answer maake sense?

## 4. Plotting with Sympy

Finally Sympy can be used to plot functions. Note that this makes use of [matplotlib](http://matplotlib.org/).

Let us plot $x^2$:

```python
expr = x ** 2
p = sym.plot(expr)
```

### Exercise 4.1 Plot the following function:

1. $y=x + cos(x)$
1. ${\frac{1}{{ \sqrt {2\pi } }}{e^{ - \frac{x^2}{2}}}}$

```python
# Your code here
```

# Lecture

## L1. Hydrogen Atom

Sympy has built in modules for the eigenfunctions of the hydrogen atom.

```python
import sympy.physics.hydrogen
import numpy as np
```

You can caluclate the eigenvalues ($E$) in Hartrees

`sym.physics.hydrogen.E_nl(n,Z)`

```python
sym.physics.hydrogen.E_nl(1,1)
```

We can use a loop to print out many energies

```python
for n in range(1,5):
    print(sym.physics.hydrogen.E_nl(n,1))
```

    -1/2
    -1/8
    -1/18
    -1/32

We can plot the hydrogen radial wavefunction (1s orbital)

```python
r=sympy.symbols('r')
sympy.physics.hydrogen.R_nl(1, 0, r, 1)
```

```python
sym.plot(sympy.physics.hydrogen.R_nl(1, 0, r, 1),(r,0,10.50))
```

And the probablity distribution function

```python
sympy.symbols('r')
prob_1s=sympy.physics.hydrogen.R_nl(1, 0, r, 1)*sympy.physics.hydrogen.R_nl(1, 0, r, 1)
prob_1s
```

```python
sym.plot(prob_1s,(r,0,10))
```

Plot a 2s orbital

```python
sympy.symbols('r')
prob_2s=sympy.physics.hydrogen.R_nl(2, 0, r, 1)*sympy.physics.hydrogen.R_nl(2, 0, r, 1)
prob_2s
```

```python
sym.plot(prob_2s,(r,0,10))
```

We can change the range to see the node better.

```python
sym.plot(prob_2s,(r,1,8))
```

Notice the node!

### Exercise L1.1

Plot the radial distriubution function for a 2p, 3s, 4s, and 3d orbital.

```python
# Your code here
import sympy.physics.hydrogen
import numpy as np

r=sympy.symbols('r')
prob_2p=sympy.physics.hydrogen.R_nl(2, 1, r, 1)

sym.plot(prob_2p),(r,0,10.50)
```

```python
import sympy.physics.hydrogen
import numpy as np

r=sympy.symbols('r')
sympy.physics.hydrogen.R_nl(3, 0, r, 1)

sym.plot(sympy.physics.hydrogen.R_nl(3, 0, r, 1),(r,0,10.50))
```

```python
import sympy.physics.hydrogen
import numpy as np

r=sympy.symbols('r')
sympy.physics.hydrogen.R_nl(4, 0, r, 1)

sym.plot(sympy.physics.hydrogen.R_nl(4, 0, r, 1),(r,0,10.50))
```

```python
import sympy.physics.hydrogen
import numpy as np

r=sympy.symbols('r')
sympy.physics.hydrogen.R_nl(3, 2, r, 1)

sym.plot(sympy.physics.hydrogen.R_nl(3, 2, r, 1),(r,0,10.50))
```

```python

```


# Übung

Nach DIN 1946-6 sind Türen mit einer Überströmöffnung zu versehen. Das ist erforderlich, damit die Luft auch bei geschlossenen Türen von den Zulufträumen zu den Ablufträumen strömen kann.

Die Überströmöffnung wird meist durch Kürzung des Türspalts um einen bestimmten Betrag sichergestellt. Nach DIN 1946-6 ist die folgende Tabelle einzuhalten:

Luftmenge
Tür mit Dichtung | $\frac{m^3}{h}$ | 10 | 20 | 30 | 40 | 50 | 60 | 70 | 80 | 90 | 100
------------------------------|-----------------|----|----|----|----|----|----|----|----|----|----
Überströmfläche               | $cm^2$          | 25 | 50 | 75 | 100|125 |150 |175 |200 | 225| 250
Kürzungsmaß                   | $mm$            |  3 |  6 |  8 | 11 | 14 | 17 | 20 | 22 | 25 | 28

- Berechnen Sie die lichten Breiten der Tür
- berechnen Sie die Strömungsgeschwindigkeiten $v$ der Luft im Türspalt,
- stellen Sie die Überströmfläche und das Kürzungsmaß jeweils in einem Diagramm dar, wobei die $x$-Achse der Volumenstrom in $\frac{m^3}{h}$ ist.

```python
# pandas ist die Bibliothek für DataFrames (Tabellen)
import pandas as pd

# numpy ist die Bibliothek für numerisches rechnen (num py)
import numpy as np

# matplotlib wird zum Plotten benötigt
import matplotlib.pyplot as plt, matplotlib.ticker as tk
%matplotlib inline
%config InlineBackend.figure_format='retina'
```

**Die Ausgangstabelle als DataFrame**

```python
df = pd.DataFrame(
    {
        'dV_tuer': range(10,101,10),
        'A_tuer': range(25,251,25),
        'k_tuer': [3,6,8,11,14,17,20,22,25,28]
    }
)
df
```

      dV_tuer
      A_tuer
      k_tuer

      0
      10
      25
      3

      1
      20
      50
      6

      2
      30
      75
      8

      3
      40
      100
      11

      4
      50
      125
      14

      5
      60
      150
      17

      6
      70
      175
      20

      7
      80
      200
      22

      8
      90
      225
      25

      9
      100
      250
      28

- Lichte Breiten der Tür:

  Die Lichten Breiten werden nicht in den Dataframe `df` integriert, weil schon bekannt ist, dass diese Größen
  nicht sinnvoll sind:

  Die Formel zur Berechnung ergibt sich aus der Formel für den Flächeninhalt:

  \begin{align}
    A_\text{tuer} &= k_\text{tuer}\cdot b_\text{tuer} \\[2ex]
    b_\text{tuer} &= \dfrac{A_\text{tuer}}{k_\text{tuer}}
                  \quad = \quad
                  \dfrac{A_\text{tuer}\cdot\dfrac{100\,mm^2}{cm^2}}{k_\text{tuer}}
  \end{align}

```python
df_b_tuer = pd.DataFrame(
    {'A_tuer': df.A_tuer,
     'k_tuer': df.k_tuer,
     'b_tuer': df.A_tuer*100/df.k_tuer
    }
)
df_b_tuer
```

      A_tuer
      k_tuer
      b_tuer

      0
      25
      3
      833.333333

      1
      50
      6
      833.333333

      2
      75
      8
      937.500000

      3
      100
      11
      909.090909

      4
      125
      14
      892.857143

      5
      150
      17
      882.352941

      6
      175
      20
      875.000000

      7
      200
      22
      909.090909

      8
      225
      25
      900.000000

      9
      250
      28
      892.857143

Realistischer ist es, von der Standardbreite einer Innentür `b = 900 # mm` ausgehend, die Kürzungsmaße der Tür
zu berechnen (`df.k_ber`) und mit den Werten zu vergleichen, die in der Tabelle angegeben sind. So wird deutlich, dass die Kürzungsmaße auf ganze $mm$ gerundete Werte sind.

- Berechnung des Kürzungsmaßes und der Strömungsgeschwindigkeit im Luftspalt:

```python
b = 900 # Breite einer Standardinnentür in mm

df['k_ber'] = df.A_tuer*100/b
df
```

      dV_tuer
      A_tuer
      k_tuer
      k_ber

      0
      10
      25
      3
      2.777778

      1
      20
      50
      6
      5.555556

      2
      30
      75
      8
      8.333333

      3
      40
      100
      11
      11.111111

      4
      50
      125
      14
      13.888889

      5
      60
      150
      17
      16.666667

      6
      70
      175
      20
      19.444444

      7
      80
      200
      22
      22.222222

      8
      90
      225
      25
      25.000000

      9
      100
      250
      28
      27.777778

Die Strömungsgeschwindigkeit ergibt sich aus dem Zusammenhang zwischen Volumenstrom, Strömungsquerschnitt und Strömungsgeschwindigkeit:

\begin{align}
  \dot V &= A\,v \\
  v &= \dfrac{\dot V}{A}
    \quad = \quad
    \dfrac{\dot V\cdot\dfrac{1\,h}{3600\,s}}{A\cdot\dfrac{1\,m^2}{10^4\,cm^2}}
\end{align}

```python
df['v_tuer'] = df.dV_tuer/3600/(df.A_tuer/1e4)

df
```

      dV_tuer
      A_tuer
      k_tuer
      k_ber
      v_tuer

      0
      10
      25
      3
      2.777778
      1.111111

      1
      20
      50
      6
      5.555556
      1.111111

      2
      30
      75
      8
      8.333333
      1.111111

      3
      40
      100
      11
      11.111111
      1.111111

      4
      50
      125
      14
      13.888889
      1.111111

      5
      60
      150
      17
      16.666667
      1.111111

      6
      70
      175
      20
      19.444444
      1.111111

      7
      80
      200
      22
      22.222222
      1.111111

      8
      90
      225
      25
      25.000000
      1.111111

      9
      100
      250
      28
      27.777778
      1.111111

Die Strömungsgeschwindigkeit ist eine Konstante und beträgt etwa $1.1\,\frac{m}{s}$

- Diagramm mit Überströmfläche und Kürzungsmaß:

```python
ax = df.set_index('dV_tuer')[['A_tuer','k_ber']]\
    .plot(subplots=True,
          color=['b','r'],
          grid=True,
          rot=90,
    )
ax[0].set(ylabel=r'Fläche in $\left[cm^2\right]$')
ax[1].scatter(df.dV_tuer,df.k_tuer,c='r',label='k_tuer')
ax[1].legend(loc='best')
ax[1].set(
    xlabel=r'Volumenstrom in $\left[\dfrac{m^3}{h}\right]$',
    ylabel=r'Kürzungsmaß in $\left[mm\right]$'
)

plt.show() # verhindert ungewollte Textausgaben
```


```python
import numpy as np
```

```python
# old base vectors in old frame
e1 = np.array([1., 0.])
e2 = np.array([0., 1.])

# new base vectors in old frame
E1 = np.array([0., 1.])
E2 = np.array([-1., 0.])

# direction cosine matrix
DCM = np.stack([E1, E2])

# the transpose of DCM transforms the base vectors as column arrays
assert np.all(np.isclose(DCM.T @ e1, E1, rtol=1e-05, atol=1e-08))
assert np.all(np.isclose(DCM.T @ e2, E2, rtol=1e-05, atol=1e-08))

# the DCM transforms the base vectors as row arrays
assert np.all(np.isclose(e1 @ DCM, E1, rtol=1e-05, atol=1e-08))
assert np.all(np.isclose(e2 @ DCM, E2, rtol=1e-05, atol=1e-08))

# transform the complete frame at once
assert np.all(np.isclose(DCM @ np.stack([e1, e2]), np.stack([E1, E2]),
                         rtol=1e-05, atol=1e-08))
```

```python
DCM
```

    array([[ 0.,  1.],
           [-1.,  0.]])

```python
from sympy.physics.vector import ReferenceFrame
```

```python
source = ReferenceFrame('source')
target = source.orientnew('target', 'Body', [0, 0, 90*np.pi/180],  'XYZ')
DCM = np.array(target.dcm(source).evalf()).astype(float)[:2, :2]
DCM[np.abs(DCM) < 1e-12] = 0.
```

```python
DCM
```

    array([[ 0.,  1.],
           [-1.,  0.]])


# More Reading Material

* [The Math of Machine Learning](https://gwthomas.github.io/docs/math4ml.pdf)

# Taking Symbolic Derivatives with Python

```python
from sympy import *
from IPython.display import display
from sympy.printing.mathml import mathml
from IPython.display import display, Math, Latex

x, y, z = symbols('x y z')
init_printing(use_unicode=True)
```

```python
def mprint(e):
    display(Math(latex(e)))
```

```python
mprint(x**2)
```

$\displaystyle x^{2}$

```python
mprint(diff(2*x**3))
```

$\displaystyle 6 x^{2}$

```python
expr = (x**3 + x**2 - x - 1)/(x**2 + 2*x + 1)
display(Math(latex(expr)))
expr = simplify(expr)
print(type(expr))
print(latex(expr))
display(Math(latex(expr)))
```

$\displaystyle \frac{x^{3} + x^{2} - x - 1}{x^{2} + 2 x + 1}$

    <class 'sympy.core.add.Add'>
    x - 1

$\displaystyle x - 1$

```python
from IPython.display import display, Math, Latex
display(Math('\\frac{1_a^x}{2}'))
```

$\displaystyle \frac{1_a^x}{2}$

```python
print(expr.subs(x,5))
```

    4

```python
eql = Eq(3*x+5,10)
```

```python
z = solveset(eql,x)
display(Math(latex(z)))
```

$\displaystyle \left\{\frac{5}{3}\right\}$

```python
from sympy import *
x, y, z = symbols('x y z')
init_printing(use_unicode=True)
expr = diff(sin(x)/x**2, x)
mprint(expr)
```

$\displaystyle \frac{\cos{\left(x \right)}}{x^{2}} - \frac{2 \sin{\left(x \right)}}{x^{3}}$

```python
expr_i = integrate(expr,x)
mprint(expr_i)
```

$\displaystyle \frac{\sin{\left(x \right)}}{x^{2}}$

# Keras Customization: Loss and Activation Functions

Your functions must be defined with TensorFlow graph commands.  The derivative will be taken automatically. (assuming all components of your function are differentiable)

# TensorFlow for Calculation

```python
import tensorflow as tf

tf.multiply(tf.constant(2.0),tf.constant(5.0)).numpy()
```

    10.0

```python
import numpy as np
tf.multiply(np.array([2,4]),np.array([2,4]))
```

    <tf.Tensor: id=10, shape=(2,), dtype=int64, numpy=array([ 4, 16])>

```python
tf.multiply(2,4).numpy()
```

    8

```python
tf.divide(2,4)
```

```python
tf.pow(2.0,4).numpy()
```

    16.0

```python
x = 5.0
y = tf.divide(1.0,tf.add(1,tf.exp(tf.negative(x))))
y
```

    <tf.Tensor: id=34, shape=(), dtype=float32, numpy=0.9933072>

# Calculus with TensorFlow

How do we take derivatives?

* [Symbolic differentiation](http://tutorial.math.lamar.edu/pdf/common_derivatives_integrals.pdf)
* [Numerical differentiation](https://en.wikipedia.org/wiki/Finite_difference) (the method of finite differences)
* [Automatic differentiation](https://en.wikipedia.org/wiki/Automatic_differentiation)

Take the derivative of $f(x) = x^2$.

Symbolic derivative $f'(x) = rx^{r-1}$

$f(4) = 4^2 = 16$

$f'(4) = 2 \cdot 4 = 8$

This can be done in TensorFlow:

```python
x = tf.constant(4.0)

with tf.GradientTape() as t:
  t.watch(x)
  z = tf.multiply(x, x)

# Derivative of z with respect to the original input tensor x
dz_dx = t.gradient(z, x)
print(dz_dx)
```

    tf.Tensor(8.0, shape=(), dtype=float32)

Lets express the [Logistic function](https://en.wikipedia.org/wiki/Logistic_function) in TensorFlow. This is also called the Sigmoid Activation function in neural network literature.

$f(x) = \frac{1}{1 + e^{-x}}$

Written in TensorFlow:

```python
x = tf.constant([5.0])
with tf.GradientTape() as t:
    t.watch(x)
    y = tf.divide(1.0,tf.add(1,tf.exp(tf.negative(x))))

print(y)
dy_dx = t.gradient(y, x)
print(dy_dx)
```

    tf.Tensor([0.9933072], shape=(1,), dtype=float32)
    tf.Tensor([0.00664806], shape=(1,), dtype=float32)

Lets check the regular function.

```python
import math

1/(1+math.exp(-5))
```

And lets check the derivative:

$f'(x) = \frac{e^x}{(e^x + 1)^2}$

```python
math.exp(-5)/(math.exp(-5)+1)**2
```

```python
x = tf.ones((2, 2))
y = tf.reduce_sum(x)
z = tf.multiply(y, y)
y.numpy()
```

How to take second (and beyond) derivatives:

```python
x = tf.constant(3.0)
with tf.GradientTape() as g:
  g.watch(x)
  with tf.GradientTape() as gg:
    gg.watch(x)
    y = x * x
  dy_dx = gg.gradient(y, x)     # Will compute to 6.0
d2y_dx2 = g.gradient(dy_dx, x)  # Will compute to 2.0
```

# Custom Loss (Objective) Function

$ \operatorname{RMSE}=\sqrt{\frac{\sum_{t=1}^T (\hat y_t - y_t)^2}{T}} $

```python
def mean_pred(y_true, y_pred):
    return tf.sqrt(tf.divide(tf.reduce_sum(tf.pow(tf.subtract(y_true, y_pred),2.0)),tf.cast(tf.size(y_true), tf.float32)))
```

```python
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Activation
import pandas as pd
import io
import os
import requests
import numpy as np
from sklearn import metrics

df = pd.read_csv(
    "https://data.heatonresearch.com/data/t81-558/auto-mpg.csv",
    na_values=['NA', '?'])

cars = df['name']

# Handle missing value
df['horsepower'] = df['horsepower'].fillna(df['horsepower'].median())

# Pandas to Numpy
x = df[['cylinders', 'displacement', 'horsepower', 'weight',
       'acceleration', 'year', 'origin']].values
y = df['mpg'].values # regression

# Build the neural network
model = Sequential()
model.add(Dense(25, input_dim=x.shape[1], activation='relu')) # Hidden 1
model.add(Dense(10, activation='relu')) # Hidden 2
model.add(Dense(1)) # Output
model.compile(loss=mean_pred, optimizer='adam')
model.fit(x,y,verbose=2,epochs=100)
```

    WARNING: Logging before flag parsing goes to stderr.
    W0930 19:48:43.874238 140735522108288 deprecation.py:323] From /Users/jheaton/miniconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/ops/math_grad.py:1205: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
    Instructions for updating:
    Use tf.where in 2.0, which has the same broadcast rule as np.where

    Train on 398 samples
    Epoch 1/100
    398/398 - 0s - loss: 11.9127
    Epoch 2/100
    398/398 - 0s - loss: 9.8074
    Epoch 3/100
    398/398 - 0s - loss: 8.4966
    Epoch 4/100
    398/398 - 0s - loss: 7.8323
    Epoch 5/100
    398/398 - 0s - loss: 7.3702
    Epoch 6/100
    398/398 - 0s - loss: 7.6900
    Epoch 7/100
    398/398 - 0s - loss: 7.1135
    Epoch 8/100
    398/398 - 0s - loss: 6.9701
    Epoch 9/100
    398/398 - 0s - loss: 6.6870
    Epoch 10/100
    398/398 - 0s - loss: 7.2692
    Epoch 11/100
    398/398 - 0s - loss: 6.2498
    Epoch 12/100
    398/398 - 0s - loss: 7.7551
    Epoch 13/100
    398/398 - 0s - loss: 7.5446
    Epoch 14/100
    398/398 - 0s - loss: 7.5419
    Epoch 15/100
    398/398 - 0s - loss: 6.2173
    Epoch 16/100
    398/398 - 0s - loss: 5.4177
    Epoch 17/100
    398/398 - 0s - loss: 5.7693
    Epoch 18/100
    398/398 - 0s - loss: 5.4172
    Epoch 19/100
    398/398 - 0s - loss: 5.1264
    Epoch 20/100
    398/398 - 0s - loss: 5.1853
    Epoch 21/100
    398/398 - 0s - loss: 4.8951
    Epoch 22/100
    398/398 - 0s - loss: 4.8044
    Epoch 23/100
    398/398 - 0s - loss: 5.6221
    Epoch 24/100
    398/398 - 0s - loss: 5.1763
    Epoch 25/100
    398/398 - 0s - loss: 4.6223
    Epoch 26/100
    398/398 - 0s - loss: 4.6465
    Epoch 27/100
    398/398 - 0s - loss: 4.5917
    Epoch 28/100
    398/398 - 0s - loss: 4.6002
    Epoch 29/100
    398/398 - 0s - loss: 5.7661
    Epoch 30/100
    398/398 - 0s - loss: 4.7825
    Epoch 31/100
    398/398 - 0s - loss: 4.5775
    Epoch 32/100
    398/398 - 0s - loss: 4.5815
    Epoch 33/100
    398/398 - 0s - loss: 5.1788
    Epoch 34/100
    398/398 - 0s - loss: 5.9627
    Epoch 35/100
    398/398 - 0s - loss: 4.5032
    Epoch 36/100
    398/398 - 0s - loss: 4.0651
    Epoch 37/100
    398/398 - 0s - loss: 4.0356
    Epoch 38/100
    398/398 - 0s - loss: 4.0107
    Epoch 39/100
    398/398 - 0s - loss: 4.0198
    Epoch 40/100
    398/398 - 0s - loss: 4.2375
    Epoch 41/100
    398/398 - 0s - loss: 3.9991
    Epoch 42/100
    398/398 - 0s - loss: 3.9840
    Epoch 43/100
    398/398 - 0s - loss: 3.9681
    Epoch 44/100
    398/398 - 0s - loss: 3.9333
    Epoch 45/100
    398/398 - 0s - loss: 3.8129
    Epoch 46/100
    398/398 - 0s - loss: 4.0683
    Epoch 47/100
    398/398 - 0s - loss: 3.9216
    Epoch 48/100
    398/398 - 0s - loss: 4.5209
    Epoch 49/100
    398/398 - 0s - loss: 4.1421
    Epoch 50/100
    398/398 - 0s - loss: 3.8114
    Epoch 51/100
    398/398 - 0s - loss: 3.6369
    Epoch 52/100
    398/398 - 0s - loss: 3.7025
    Epoch 53/100
    398/398 - 0s - loss: 3.8485
    Epoch 54/100
    398/398 - 0s - loss: 4.4703
    Epoch 55/100
    398/398 - 0s - loss: 4.1166
    Epoch 56/100
    398/398 - 0s - loss: 3.8752
    Epoch 57/100
    398/398 - 0s - loss: 3.9621
    Epoch 58/100
    398/398 - 0s - loss: 3.6541
    Epoch 59/100
    398/398 - 0s - loss: 3.6920
    Epoch 60/100
    398/398 - 0s - loss: 4.1113
    Epoch 61/100
    398/398 - 0s - loss: 4.0092
    Epoch 62/100
    398/398 - 0s - loss: 4.3133
    Epoch 63/100
    398/398 - 0s - loss: 4.0480
    Epoch 64/100
    398/398 - 0s - loss: 4.3408
    Epoch 65/100
    398/398 - 0s - loss: 4.5607
    Epoch 66/100
    398/398 - 0s - loss: 4.1189
    Epoch 67/100
    398/398 - 0s - loss: 3.6281
    Epoch 68/100
    398/398 - 0s - loss: 3.9074
    Epoch 69/100
    398/398 - 0s - loss: 3.6370
    Epoch 70/100
    398/398 - 0s - loss: 3.7702
    Epoch 71/100
    398/398 - 0s - loss: 3.8023
    Epoch 72/100
    398/398 - 0s - loss: 4.1845
    Epoch 73/100
    398/398 - 0s - loss: 3.9606
    Epoch 74/100
    398/398 - 0s - loss: 3.8198
    Epoch 75/100
    398/398 - 0s - loss: 3.5847
    Epoch 76/100
    398/398 - 0s - loss: 3.6095
    Epoch 77/100
    398/398 - 0s - loss: 3.5945
    Epoch 78/100
    398/398 - 0s - loss: 4.8085
    Epoch 79/100
    398/398 - 0s - loss: 3.6670
    Epoch 80/100
    398/398 - 0s - loss: 3.7169
    Epoch 81/100
    398/398 - 0s - loss: 3.7726
    Epoch 82/100
    398/398 - 0s - loss: 3.8390
    Epoch 83/100
    398/398 - 0s - loss: 4.6407
    Epoch 84/100
    398/398 - 0s - loss: 4.0283
    Epoch 85/100
    398/398 - 0s - loss: 4.4923
    Epoch 86/100
    398/398 - 0s - loss: 3.7569
    Epoch 87/100
    398/398 - 0s - loss: 3.5686
    Epoch 88/100
    398/398 - 0s - loss: 3.5874
    Epoch 89/100
    398/398 - 0s - loss: 3.8769
    Epoch 90/100
    398/398 - 0s - loss: 4.2424
    Epoch 91/100
    398/398 - 0s - loss: 3.6022
    Epoch 92/100
    398/398 - 0s - loss: 3.6107
    Epoch 93/100
    398/398 - 0s - loss: 3.6487
    Epoch 94/100
    398/398 - 0s - loss: 3.5627
    Epoch 95/100
    398/398 - 0s - loss: 3.7136
    Epoch 96/100
    398/398 - 0s - loss: 3.7639
    Epoch 97/100
    398/398 - 0s - loss: 3.9539
    Epoch 98/100
    398/398 - 0s - loss: 4.1260
    Epoch 99/100
    398/398 - 0s - loss: 3.9041
    Epoch 100/100
    398/398 - 0s - loss: 3.6862

    <tensorflow.python.keras.callbacks.History at 0x1a3bab35c0>

# Custom Activation (Transfer) Functions

```python
import tensorflow as tf
def elliot_sym(x):
    return tf.divide(x,tf.add(1.0,tf.abs(x)))
```

```python
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Activation
import pandas as pd
import io
import os
import requests
import numpy as np
from sklearn import metrics

df = pd.read_csv(
    "https://data.heatonresearch.com/data/t81-558/auto-mpg.csv",
    na_values=['NA', '?'])

cars = df['name']

# Handle missing value
df['horsepower'] = df['horsepower'].fillna(df['horsepower'].median())

# Pandas to Numpy
x = df[['cylinders', 'displacement', 'horsepower', 'weight',
       'acceleration', 'year', 'origin']].values
y = df['mpg'].values # regression

# Build the neural network
sgd = tf.keras.optimizers.SGD(lr=1e-10, decay=1e-6, momentum=0.9, nesterov=True)
model = Sequential()
model.add(Dense(25, input_dim=x.shape[1], activation=elliot_sym)) # Hidden 1
model.add(Dense(10, activation=elliot_sym)) # Hidden 2
model.add(Dense(1)) # Output
model.compile(loss='mean_squared_error', optimizer='adam')
model.fit(x,y,verbose=2,epochs=400)
```

    Train on 398 samples
    Epoch 1/400
    398/398 - 0s - loss: 580.7997
    Epoch 2/400
    398/398 - 0s - loss: 559.8733
    Epoch 3/400
    398/398 - 0s - loss: 542.7060
    Epoch 4/400
    398/398 - 0s - loss: 528.3312
    Epoch 5/400
    398/398 - 0s - loss: 517.3115
    Epoch 6/400
    398/398 - 0s - loss: 507.8317
    Epoch 7/400
    398/398 - 0s - loss: 497.6923
    Epoch 8/400
    398/398 - 0s - loss: 488.9701
    Epoch 9/400
    398/398 - 0s - loss: 481.3325
    Epoch 10/400
    398/398 - 0s - loss: 473.9950
    Epoch 11/400
    398/398 - 0s - loss: 466.9197
    Epoch 12/400
    398/398 - 0s - loss: 459.9756
    Epoch 13/400
    398/398 - 0s - loss: 453.2480
    Epoch 14/400
    398/398 - 0s - loss: 446.6219
    Epoch 15/400
    398/398 - 0s - loss: 440.1158
    Epoch 16/400
    398/398 - 0s - loss: 433.7691
    Epoch 17/400
    398/398 - 0s - loss: 427.5683
    Epoch 18/400
    398/398 - 0s - loss: 421.3823
    Epoch 19/400
    398/398 - 0s - loss: 415.3867
    Epoch 20/400
    398/398 - 0s - loss: 409.4597
    Epoch 21/400
    398/398 - 0s - loss: 403.6996
    Epoch 22/400
    398/398 - 0s - loss: 397.9255
    Epoch 23/400
    398/398 - 0s - loss: 392.3375
    Epoch 24/400
    398/398 - 0s - loss: 386.7781
    Epoch 25/400
    398/398 - 0s - loss: 381.3748
    Epoch 26/400
    398/398 - 0s - loss: 375.9911
    Epoch 27/400
    398/398 - 0s - loss: 370.7443
    Epoch 28/400
    398/398 - 0s - loss: 365.5234
    Epoch 29/400
    398/398 - 0s - loss: 360.4638
    Epoch 30/400
    398/398 - 0s - loss: 355.4051
    Epoch 31/400
    398/398 - 0s - loss: 350.4503
    Epoch 32/400
    398/398 - 0s - loss: 345.5780
    Epoch 33/400
    398/398 - 0s - loss: 340.7539
    Epoch 34/400
    398/398 - 0s - loss: 335.9997
    Epoch 35/400
    398/398 - 0s - loss: 331.2908
    Epoch 36/400
    398/398 - 0s - loss: 326.6801
    Epoch 37/400
    398/398 - 0s - loss: 322.1323
    Epoch 38/400
    398/398 - 0s - loss: 317.6861
    Epoch 39/400
    398/398 - 0s - loss: 313.3140
    Epoch 40/400
    398/398 - 0s - loss: 308.9250
    Epoch 41/400
    398/398 - 0s - loss: 304.6521
    Epoch 42/400
    398/398 - 0s - loss: 300.4291
    Epoch 43/400
    398/398 - 0s - loss: 296.2750
    Epoch 44/400
    398/398 - 0s - loss: 292.1553
    Epoch 45/400
    398/398 - 0s - loss: 288.1540
    Epoch 46/400
    398/398 - 0s - loss: 284.1031
    Epoch 47/400
    398/398 - 0s - loss: 280.1751
    Epoch 48/400
    398/398 - 0s - loss: 276.2384
    Epoch 49/400
    398/398 - 0s - loss: 272.4429
    Epoch 50/400
    398/398 - 0s - loss: 268.7119
    Epoch 51/400
    398/398 - 0s - loss: 264.9631
    Epoch 52/400
    398/398 - 0s - loss: 261.2771
    Epoch 53/400
    398/398 - 0s - loss: 257.6984
    Epoch 54/400
    398/398 - 0s - loss: 254.1226
    Epoch 55/400
    398/398 - 0s - loss: 250.6449
    Epoch 56/400
    398/398 - 0s - loss: 247.1738
    Epoch 57/400
    398/398 - 0s - loss: 243.7975
    Epoch 58/400
    398/398 - 0s - loss: 240.4601
    Epoch 59/400
    398/398 - 0s - loss: 237.1640
    Epoch 60/400
    398/398 - 0s - loss: 233.8297
    Epoch 61/400
    398/398 - 0s - loss: 230.6481
    Epoch 62/400
    398/398 - 0s - loss: 227.4332
    Epoch 63/400
    398/398 - 0s - loss: 224.3163
    Epoch 64/400
    398/398 - 0s - loss: 221.2258
    Epoch 65/400
    398/398 - 0s - loss: 218.1860
    Epoch 66/400
    398/398 - 0s - loss: 215.2228
    Epoch 67/400
    398/398 - 0s - loss: 212.3084
    Epoch 68/400
    398/398 - 0s - loss: 209.3812
    Epoch 69/400
    398/398 - 0s - loss: 206.5258
    Epoch 70/400
    398/398 - 0s - loss: 203.6957
    Epoch 71/400
    398/398 - 0s - loss: 200.9445
    Epoch 72/400
    398/398 - 0s - loss: 198.2295
    Epoch 73/400
    398/398 - 0s - loss: 195.5232
    Epoch 74/400
    398/398 - 0s - loss: 192.8750
    Epoch 75/400
    398/398 - 0s - loss: 190.2599
    Epoch 76/400
    398/398 - 0s - loss: 187.6881
    Epoch 77/400
    398/398 - 0s - loss: 185.1539
    Epoch 78/400
    398/398 - 0s - loss: 182.6923
    Epoch 79/400
    398/398 - 0s - loss: 180.2004
    Epoch 80/400
    398/398 - 0s - loss: 177.8139
    Epoch 81/400
    398/398 - 0s - loss: 175.4295
    Epoch 82/400
    398/398 - 0s - loss: 173.1007
    Epoch 83/400
    398/398 - 0s - loss: 170.8023
    Epoch 84/400
    398/398 - 0s - loss: 168.5209
    Epoch 85/400
    398/398 - 0s - loss: 166.3050
    Epoch 86/400
    398/398 - 0s - loss: 164.1033
    Epoch 87/400
    398/398 - 0s - loss: 161.9304
    Epoch 88/400
    398/398 - 0s - loss: 159.7803
    Epoch 89/400
    398/398 - 0s - loss: 157.7115
    Epoch 90/400
    398/398 - 0s - loss: 155.6226
    Epoch 91/400
    398/398 - 0s - loss: 153.5967
    Epoch 92/400
    398/398 - 0s - loss: 151.6195
    Epoch 93/400
    398/398 - 0s - loss: 149.6920
    Epoch 94/400
    398/398 - 0s - loss: 147.7518
    Epoch 95/400
    398/398 - 0s - loss: 145.8823
    Epoch 96/400
    398/398 - 0s - loss: 144.0274
    Epoch 97/400
    398/398 - 0s - loss: 142.1929
    Epoch 98/400
    398/398 - 0s - loss: 140.4112
    Epoch 99/400
    398/398 - 0s - loss: 138.6662
    Epoch 100/400
    398/398 - 0s - loss: 136.9243
    Epoch 101/400
    398/398 - 0s - loss: 135.2350
    Epoch 102/400
    398/398 - 0s - loss: 133.6141
    Epoch 103/400
    398/398 - 0s - loss: 131.9906
    Epoch 104/400
    398/398 - 0s - loss: 130.3974
    Epoch 105/400
    398/398 - 0s - loss: 128.8241
    Epoch 106/400
    398/398 - 0s - loss: 127.2509
    Epoch 107/400
    398/398 - 0s - loss: 125.7629
    Epoch 108/400
    398/398 - 0s - loss: 124.2734
    Epoch 109/400
    398/398 - 0s - loss: 122.8298
    Epoch 110/400
    398/398 - 0s - loss: 121.4015
    Epoch 111/400
    398/398 - 0s - loss: 119.9862
    Epoch 112/400
    398/398 - 0s - loss: 118.5762
    Epoch 113/400
    398/398 - 0s - loss: 117.2532
    Epoch 114/400
    398/398 - 0s - loss: 115.8546
    Epoch 115/400
    398/398 - 0s - loss: 114.5412
    Epoch 116/400
    398/398 - 0s - loss: 113.3085
    Epoch 117/400
    398/398 - 0s - loss: 111.9642
    Epoch 118/400
    398/398 - 0s - loss: 110.7543
    Epoch 119/400
    398/398 - 0s - loss: 109.5120
    Epoch 120/400
    398/398 - 0s - loss: 108.3341
    Epoch 121/400
    398/398 - 0s - loss: 107.1654
    Epoch 122/400
    398/398 - 0s - loss: 105.9951
    Epoch 123/400
    398/398 - 0s - loss: 104.8567
    Epoch 124/400
    398/398 - 0s - loss: 103.7335
    Epoch 125/400
    398/398 - 0s - loss: 102.6758
    Epoch 126/400
    398/398 - 0s - loss: 101.6068
    Epoch 127/400
    398/398 - 0s - loss: 100.5657
    Epoch 128/400
    398/398 - 0s - loss: 99.5396
    Epoch 129/400
    398/398 - 0s - loss: 98.5638
    Epoch 130/400
    398/398 - 0s - loss: 97.5750
    Epoch 131/400
    398/398 - 0s - loss: 96.6493
    Epoch 132/400
    398/398 - 0s - loss: 95.7024
    Epoch 133/400
    398/398 - 0s - loss: 94.8202
    Epoch 134/400
    398/398 - 0s - loss: 93.8833
    Epoch 135/400
    398/398 - 0s - loss: 93.0028
    Epoch 136/400
    398/398 - 0s - loss: 92.1690
    Epoch 137/400
    398/398 - 0s - loss: 91.3089
    Epoch 138/400
    398/398 - 0s - loss: 90.4499
    Epoch 139/400
    398/398 - 0s - loss: 89.6939
    Epoch 140/400
    398/398 - 0s - loss: 88.8778
    Epoch 141/400
    398/398 - 0s - loss: 88.1154
    Epoch 142/400
    398/398 - 0s - loss: 87.3610
    Epoch 143/400
    398/398 - 0s - loss: 86.6266
    Epoch 144/400
    398/398 - 0s - loss: 85.8834
    Epoch 145/400
    398/398 - 0s - loss: 85.1884
    Epoch 146/400
    398/398 - 0s - loss: 84.4776
    Epoch 147/400
    398/398 - 0s - loss: 83.7920
    Epoch 148/400
    398/398 - 0s - loss: 83.1500
    Epoch 149/400
    398/398 - 0s - loss: 82.4968
    Epoch 150/400
    398/398 - 0s - loss: 81.8893
    Epoch 151/400
    398/398 - 0s - loss: 81.2401
    Epoch 152/400
    398/398 - 0s - loss: 80.6445
    Epoch 153/400
    398/398 - 0s - loss: 80.0572
    Epoch 154/400
    398/398 - 0s - loss: 79.4956
    Epoch 155/400
    398/398 - 0s - loss: 78.9192
    Epoch 156/400
    398/398 - 0s - loss: 78.3741
    Epoch 157/400
    398/398 - 0s - loss: 77.8235
    Epoch 158/400
    398/398 - 0s - loss: 77.3335
    Epoch 159/400
    398/398 - 0s - loss: 76.7990
    Epoch 160/400
    398/398 - 0s - loss: 76.3360
    Epoch 161/400
    398/398 - 0s - loss: 75.8388
    Epoch 162/400
    398/398 - 0s - loss: 75.3893
    Epoch 163/400
    398/398 - 0s - loss: 74.9382
    Epoch 164/400
    398/398 - 0s - loss: 74.4963
    Epoch 165/400
    398/398 - 0s - loss: 74.0519
    Epoch 166/400
    398/398 - 0s - loss: 73.6186
    Epoch 167/400
    398/398 - 0s - loss: 73.2164
    Epoch 168/400
    398/398 - 0s - loss: 72.8161
    Epoch 169/400
    398/398 - 0s - loss: 72.4290
    Epoch 170/400
    398/398 - 0s - loss: 72.0344
    Epoch 171/400
    398/398 - 0s - loss: 71.6713
    Epoch 172/400
    398/398 - 0s - loss: 71.3272
    Epoch 173/400
    398/398 - 0s - loss: 70.9791
    Epoch 174/400
    398/398 - 0s - loss: 70.6605
    Epoch 175/400
    398/398 - 0s - loss: 70.3308
    Epoch 176/400
    398/398 - 0s - loss: 70.0010
    Epoch 177/400
    398/398 - 0s - loss: 69.7020
    Epoch 178/400
    398/398 - 0s - loss: 69.3984
    Epoch 179/400
    398/398 - 0s - loss: 69.0993
    Epoch 180/400
    398/398 - 0s - loss: 68.8194
    Epoch 181/400
    398/398 - 0s - loss: 68.5440
    Epoch 182/400
    398/398 - 0s - loss: 68.2829
    Epoch 183/400
    398/398 - 0s - loss: 68.0232
    Epoch 184/400
    398/398 - 0s - loss: 67.7583
    Epoch 185/400
    398/398 - 0s - loss: 67.5099
    Epoch 186/400
    398/398 - 0s - loss: 67.2846
    Epoch 187/400
    398/398 - 0s - loss: 67.0426
    Epoch 188/400
    398/398 - 0s - loss: 66.8092
    Epoch 189/400
    398/398 - 0s - loss: 66.5920
    Epoch 190/400
    398/398 - 0s - loss: 66.3845
    Epoch 191/400
    398/398 - 0s - loss: 66.1705
    Epoch 192/400
    398/398 - 0s - loss: 65.9765
    Epoch 193/400
    398/398 - 0s - loss: 65.7886
    Epoch 194/400
    398/398 - 0s - loss: 65.5992
    Epoch 195/400
    398/398 - 0s - loss: 65.4229
    Epoch 196/400
    398/398 - 0s - loss: 65.2662
    Epoch 197/400
    398/398 - 0s - loss: 65.1019
    Epoch 198/400
    398/398 - 0s - loss: 64.9430
    Epoch 199/400
    398/398 - 0s - loss: 64.7934
    Epoch 200/400
    398/398 - 0s - loss: 64.6446
    Epoch 201/400
    398/398 - 0s - loss: 64.5025
    Epoch 202/400
    398/398 - 0s - loss: 64.3684
    Epoch 203/400
    398/398 - 0s - loss: 64.2242
    Epoch 204/400
    398/398 - 0s - loss: 64.1108
    Epoch 205/400
    398/398 - 0s - loss: 63.9893
    Epoch 206/400
    398/398 - 0s - loss: 63.8638
    Epoch 207/400
    398/398 - 0s - loss: 63.7647
    Epoch 208/400
    398/398 - 0s - loss: 63.6336
    Epoch 209/400
    398/398 - 0s - loss: 63.5306
    Epoch 210/400
    398/398 - 0s - loss: 63.4151
    Epoch 211/400
    398/398 - 0s - loss: 63.3242
    Epoch 212/400
    398/398 - 0s - loss: 63.2086
    Epoch 213/400
    398/398 - 0s - loss: 63.1222
    Epoch 214/400
    398/398 - 0s - loss: 63.0260
    Epoch 215/400
    398/398 - 0s - loss: 62.9294
    Epoch 216/400
    398/398 - 0s - loss: 62.8411
    Epoch 217/400
    398/398 - 0s - loss: 62.7635
    Epoch 218/400
    398/398 - 0s - loss: 62.6778
    Epoch 219/400
    398/398 - 0s - loss: 62.6034
    Epoch 220/400
    398/398 - 0s - loss: 62.5289
    Epoch 221/400
    398/398 - 0s - loss: 62.4619
    Epoch 222/400
    398/398 - 0s - loss: 62.3901
    Epoch 223/400
    398/398 - 0s - loss: 62.3387
    Epoch 224/400
    398/398 - 0s - loss: 62.2619
    Epoch 225/400
    398/398 - 0s - loss: 62.2037
    Epoch 226/400
    398/398 - 0s - loss: 62.1516
    Epoch 227/400
    398/398 - 0s - loss: 62.1013
    Epoch 228/400
    398/398 - 0s - loss: 62.0436
    Epoch 229/400
    398/398 - 0s - loss: 61.9967
    Epoch 230/400
    398/398 - 0s - loss: 61.9455
    Epoch 231/400
    398/398 - 0s - loss: 61.8937
    Epoch 232/400
    398/398 - 0s - loss: 61.8499
    Epoch 233/400
    398/398 - 0s - loss: 61.8029
    Epoch 234/400
    398/398 - 0s - loss: 61.7635
    Epoch 235/400
    398/398 - 0s - loss: 61.7247
    Epoch 236/400
    398/398 - 0s - loss: 61.6847
    Epoch 237/400
    398/398 - 0s - loss: 61.6517
    Epoch 238/400
    398/398 - 0s - loss: 61.6056
    Epoch 239/400
    398/398 - 0s - loss: 61.5796
    Epoch 240/400
    398/398 - 0s - loss: 61.5413
    Epoch 241/400
    398/398 - 0s - loss: 61.5116
    Epoch 242/400
    398/398 - 0s - loss: 61.4861
    Epoch 243/400
    398/398 - 0s - loss: 61.4562
    Epoch 244/400
    398/398 - 0s - loss: 61.4317
    Epoch 245/400
    398/398 - 0s - loss: 61.4075
    Epoch 246/400
    398/398 - 0s - loss: 61.3811
    Epoch 247/400
    398/398 - 0s - loss: 61.3659
    Epoch 248/400
    398/398 - 0s - loss: 61.3367
    Epoch 249/400
    398/398 - 0s - loss: 61.3204
    Epoch 250/400
    398/398 - 0s - loss: 61.2958
    Epoch 251/400
    398/398 - 0s - loss: 61.2899
    Epoch 252/400
    398/398 - 0s - loss: 61.2630
    Epoch 253/400
    398/398 - 0s - loss: 61.2419
    Epoch 254/400
    398/398 - 0s - loss: 61.2319
    Epoch 255/400
    398/398 - 0s - loss: 61.2128
    Epoch 256/400
    398/398 - 0s - loss: 61.1957
    Epoch 257/400
    398/398 - 0s - loss: 61.1809
    Epoch 258/400
    398/398 - 0s - loss: 61.1678
    Epoch 259/400
    398/398 - 0s - loss: 61.1538
    Epoch 260/400
    398/398 - 0s - loss: 61.1401
    Epoch 261/400
    398/398 - 0s - loss: 61.1282
    Epoch 262/400
    398/398 - 0s - loss: 61.1227
    Epoch 263/400
    398/398 - 0s - loss: 61.1088
    Epoch 264/400
    398/398 - 0s - loss: 61.1048
    Epoch 265/400
    398/398 - 0s - loss: 61.0889
    Epoch 266/400
    398/398 - 0s - loss: 61.0808
    Epoch 267/400
    398/398 - 0s - loss: 61.0710
    Epoch 268/400
    398/398 - 0s - loss: 61.0624
    Epoch 269/400
    398/398 - 0s - loss: 61.0548
    Epoch 270/400
    398/398 - 0s - loss: 61.0484
    Epoch 271/400
    398/398 - 0s - loss: 61.0411
    Epoch 272/400
    398/398 - 0s - loss: 61.0391
    Epoch 273/400
    398/398 - 0s - loss: 61.0247
    Epoch 274/400
    398/398 - 0s - loss: 61.0232
    Epoch 275/400
    398/398 - 0s - loss: 61.0141
    Epoch 276/400
    398/398 - 0s - loss: 61.0081
    Epoch 277/400
    398/398 - 0s - loss: 61.0057
    Epoch 278/400
    398/398 - 0s - loss: 60.9989
    Epoch 279/400
    398/398 - 0s - loss: 60.9980
    Epoch 280/400
    398/398 - 0s - loss: 60.9915
    Epoch 281/400
    398/398 - 0s - loss: 60.9883
    Epoch 282/400
    398/398 - 0s - loss: 60.9884
    Epoch 283/400
    398/398 - 0s - loss: 60.9839
    Epoch 284/400
    398/398 - 0s - loss: 60.9796
    Epoch 285/400
    398/398 - 0s - loss: 60.9842
    Epoch 286/400
    398/398 - 0s - loss: 60.9736
    Epoch 287/400
    398/398 - 0s - loss: 60.9723
    Epoch 288/400
    398/398 - 0s - loss: 60.9698
    Epoch 289/400
    398/398 - 0s - loss: 60.9737
    Epoch 290/400
    398/398 - 0s - loss: 60.9675
    Epoch 291/400
    398/398 - 0s - loss: 60.9665
    Epoch 292/400
    398/398 - 0s - loss: 60.9650
    Epoch 293/400
    398/398 - 0s - loss: 60.9605
    Epoch 294/400
    398/398 - 0s - loss: 60.9592
    Epoch 295/400
    398/398 - 0s - loss: 60.9574
    Epoch 296/400
    398/398 - 0s - loss: 60.9582
    Epoch 297/400
    398/398 - 0s - loss: 60.9577
    Epoch 298/400
    398/398 - 0s - loss: 60.9542
    Epoch 299/400
    398/398 - 0s - loss: 60.9532
    Epoch 300/400
    398/398 - 0s - loss: 60.9527
    Epoch 301/400
    398/398 - 0s - loss: 60.9523
    Epoch 302/400
    398/398 - 0s - loss: 60.9518
    Epoch 303/400
    398/398 - 0s - loss: 60.9530
    Epoch 304/400
    398/398 - 0s - loss: 60.9513
    Epoch 305/400
    398/398 - 0s - loss: 60.9501
    Epoch 306/400
    398/398 - 0s - loss: 60.9496
    Epoch 307/400
    398/398 - 0s - loss: 60.9486
    Epoch 308/400
    398/398 - 0s - loss: 60.9499
    Epoch 309/400
    398/398 - 0s - loss: 60.9466
    Epoch 310/400
    398/398 - 0s - loss: 60.9471
    Epoch 311/400
    398/398 - 0s - loss: 60.9499
    Epoch 312/400
    398/398 - 0s - loss: 60.9487
    Epoch 313/400
    398/398 - 0s - loss: 60.9479
    Epoch 314/400
    398/398 - 0s - loss: 60.9457
    Epoch 315/400
    398/398 - 0s - loss: 60.9463
    Epoch 316/400
    398/398 - 0s - loss: 60.9440
    Epoch 317/400
    398/398 - 0s - loss: 60.9446
    Epoch 318/400
    398/398 - 0s - loss: 60.9450
    Epoch 319/400
    398/398 - 0s - loss: 60.9438
    Epoch 320/400
    398/398 - 0s - loss: 60.9426
    Epoch 321/400
    398/398 - 0s - loss: 60.9438
    Epoch 322/400
    398/398 - 0s - loss: 60.9468
    Epoch 323/400
    398/398 - 0s - loss: 60.9459
    Epoch 324/400
    398/398 - 0s - loss: 60.9435
    Epoch 325/400
    398/398 - 0s - loss: 60.9439
    Epoch 326/400
    398/398 - 0s - loss: 60.9423
    Epoch 327/400
    398/398 - 0s - loss: 60.9450
    Epoch 328/400
    398/398 - 0s - loss: 60.9433
    Epoch 329/400
    398/398 - 0s - loss: 60.9434
    Epoch 330/400
    398/398 - 0s - loss: 60.9441
    Epoch 331/400
    398/398 - 0s - loss: 60.9427
    Epoch 332/400
    398/398 - 0s - loss: 60.9422
    Epoch 333/400
    398/398 - 0s - loss: 60.9450
    Epoch 334/400
    398/398 - 0s - loss: 60.9426
    Epoch 335/400
    398/398 - 0s - loss: 60.9433
    Epoch 336/400
    398/398 - 0s - loss: 60.9474
    Epoch 337/400
    398/398 - 0s - loss: 60.9415
    Epoch 338/400
    398/398 - 0s - loss: 60.9458
    Epoch 339/400
    398/398 - 0s - loss: 60.9450
    Epoch 340/400
    398/398 - 0s - loss: 60.9451
    Epoch 341/400
    398/398 - 0s - loss: 60.9411
    Epoch 342/400
    398/398 - 0s - loss: 60.9473
    Epoch 343/400
    398/398 - 0s - loss: 60.9409
    Epoch 344/400
    398/398 - 0s - loss: 60.9432
    Epoch 345/400
    398/398 - 0s - loss: 60.9449
    Epoch 346/400
    398/398 - 0s - loss: 60.9420
    Epoch 347/400
    398/398 - 0s - loss: 60.9420
    Epoch 348/400
    398/398 - 0s - loss: 60.9405
    Epoch 349/400
    398/398 - 0s - loss: 60.9420
    Epoch 350/400
    398/398 - 0s - loss: 60.9415
    Epoch 351/400
    398/398 - 0s - loss: 60.9431
    Epoch 352/400
    398/398 - 0s - loss: 60.9438
    Epoch 353/400
    398/398 - 0s - loss: 60.9430
    Epoch 354/400
    398/398 - 0s - loss: 60.9424
    Epoch 355/400
    398/398 - 0s - loss: 60.9424
    Epoch 356/400
    398/398 - 0s - loss: 60.9448
    Epoch 357/400
    398/398 - 0s - loss: 60.9424
    Epoch 358/400
    398/398 - 0s - loss: 60.9416
    Epoch 359/400
    398/398 - 0s - loss: 60.9412
    Epoch 360/400
    398/398 - 0s - loss: 60.9422
    Epoch 361/400
    398/398 - 0s - loss: 60.9414
    Epoch 362/400
    398/398 - 0s - loss: 60.9414
    Epoch 363/400
    398/398 - 0s - loss: 60.9426
    Epoch 364/400
    398/398 - 0s - loss: 60.9410
    Epoch 365/400
    398/398 - 0s - loss: 60.9412
    Epoch 366/400
    398/398 - 0s - loss: 60.9411
    Epoch 367/400
    398/398 - 0s - loss: 60.9427
    Epoch 368/400
    398/398 - 0s - loss: 60.9409
    Epoch 369/400
    398/398 - 0s - loss: 60.9447
    Epoch 370/400
    398/398 - 0s - loss: 60.9394
    Epoch 371/400
    398/398 - 0s - loss: 60.9409
    Epoch 372/400
    398/398 - 0s - loss: 60.9426
    Epoch 373/400
    398/398 - 0s - loss: 60.9440
    Epoch 374/400
    398/398 - 0s - loss: 60.9432
    Epoch 375/400
    398/398 - 0s - loss: 60.9467
    Epoch 376/400
    398/398 - 0s - loss: 60.9386
    Epoch 377/400
    398/398 - 0s - loss: 60.9432
    Epoch 378/400
    398/398 - 0s - loss: 60.9426
    Epoch 379/400
    398/398 - 0s - loss: 60.9412
    Epoch 380/400
    398/398 - 0s - loss: 60.9415
    Epoch 381/400
    398/398 - 0s - loss: 60.9409
    Epoch 382/400
    398/398 - 0s - loss: 60.9409
    Epoch 383/400
    398/398 - 0s - loss: 60.9425
    Epoch 384/400
    398/398 - 0s - loss: 60.9407
    Epoch 385/400
    398/398 - 0s - loss: 60.9410
    Epoch 386/400
    398/398 - 0s - loss: 60.9468
    Epoch 387/400
    398/398 - 0s - loss: 60.9434
    Epoch 388/400
    398/398 - 0s - loss: 60.9440
    Epoch 389/400
    398/398 - 0s - loss: 60.9433
    Epoch 390/400
    398/398 - 0s - loss: 60.9418
    Epoch 391/400
    398/398 - 0s - loss: 60.9458
    Epoch 392/400
    398/398 - 0s - loss: 60.9443
    Epoch 393/400
    398/398 - 0s - loss: 60.9424
    Epoch 394/400
    398/398 - 0s - loss: 60.9415
    Epoch 395/400
    398/398 - 0s - loss: 60.9417
    Epoch 396/400
    398/398 - 0s - loss: 60.9427
    Epoch 397/400
    398/398 - 0s - loss: 60.9402
    Epoch 398/400
    398/398 - 0s - loss: 60.9490
    Epoch 399/400
    398/398 - 0s - loss: 60.9405
    Epoch 400/400
    398/398 - 0s - loss: 60.9441

    <tensorflow.python.keras.callbacks.History at 0x1a3c488438>

```python

```


# Exercise session nº 5
---
# Furrow Constriction in Animal Cell Cytokinesis

__*Sacha Ichbiah, 21/02/22, ENS Paris*__

This subject is extracted from :
> Hervé Turlier et al., *Furrow Constriction in Animal Cell Cytokinesis*, Biophysical Journal, 2014. \
> https://doi.org/10.1016/j.bpj.2013.11.014

Cytokinesis is the process of physical cleavage at the end of cell division; it proceeds by ingression of an actomyosin furrow at the equator of the cell. Its failure leads to multinucleated cells and is a possible cause of tumorigenesis. Despite its ubiquity in developmental biology, its precise description and understanding have challenged biologists and physicists.

In this paper, the authors propose a model based on a minimal geometry and scaling arguments that gives a physical interpretation of the process appearing during cytokinesis. It notably demonstrates that because of the cytoplasm incompressibility, the cytokinesis leads to a competition between the furrow line tension and the cell poles' surface tension. This competition sets a threshold for cytokinesis completion, and explains cytokinesis dynamics.

During these session, we will derive the equations of this scaling model of furrow constriction, and we will integrate these equations to study constriction dynamics. We will show that it allows to have a cytokinesis duration independant of cell size, which has been observed in C-Elegans.

---

## I - The Scaling Model

The geometry of the dividing cell is described by the apposition of two spherical caps, parametrized by an angle $\theta$ as shown on the left sketch.

The volume of a spherical cap (in blue) is : $\mathcal{V}_{sc}(r,h) = \dfrac{\pi}{3} h^2 (3r - h)$, and its area : $\mathcal{A}_{sc}(r,h)=2\pi r h $ (right sketch).

#### **Question 1 :**
> Noting that the cytoplasmic is an incompressible fluid, establish that $R_0 = R F(\theta)$

**Correction** :

Noting that $h = R(1-\text{cos}(\theta))$, we have :

$\begin{align}
\frac{V}{2} & = \frac{4}{3} \pi R^3 - V_{sc}(R) \newline
&= \frac{\pi R^3}{3}(4 - (1-\text{cos}(\theta))^2 (3R - R(1-\text{cos}(\theta))) \newline
&= \frac{\pi R^3}{3}(2 + 3\text{cos}(\theta) - \text{cos}^3(\theta)) \newline
&= \frac{2\pi R^3}{3}(1 + \frac{3}{2}\text{cos}(\theta) - \frac{1}{2}\text{cos}^3(\theta)) \newline
\end{align}
$

The incompressibility gives us $V = V_0 = \frac{4}{3} \pi R_0^3$, thus $R_0 = R.F(\theta)$ with $F(\theta)=1 + \frac{3}{2}\text{cos}(\theta) - \frac{1}{2}\text{cos}^3(\theta)$

We define a dimensionless parameter $\kappa$ to express the competition between the mean contractile surface tension at the furrow and the tension at the cell poles: $\kappa = \dfrac{\gamma}{2R_0N^a_0}$.

The polar contractility tends to reduce the surface $A_p = 2\pi R^2 (1+\text{cos}(\theta))$ of each cell poles, whereas the line tension tends to reduce the contractile ring circumference $r_f$. These effects are captured by a simple mechanical energy $\mathcal{E} = 2\pi r_f \gamma + 2 A_p N^a_0$.

#### **Question 2 :**
> Rescale the energy $\mathcal{E}$ by an energy $\mathcal{E}_0 = 4 \pi R_0^2 N^a_0$ to make it only depend on $\theta$ and $\kappa$.

**Correction** :

As $r_f = R\text{sin}(\theta)$, we have :

$\begin{align}
\mathcal{E} &= 2\pi \gamma \frac{R_0}{F(\theta)^{1/3}} \text{sin}(\theta) + 4 \pi \frac{R_0^2}{F(\theta)^{2/3}} (1+\text{cos}(\theta)) N^a_0 \newline
\Rightarrow \frac{\mathcal{E}}{4 \pi R_0^2 N^a_0} &= \frac{\gamma}{2 R_0 N^a_0} \frac{\text{sin}(\theta)}{F(\theta)^{1/3}} + \frac{1+\text{cos}(\theta)}{F(\theta)^{2/3}} \newline
\Rightarrow \frac{\mathcal{E}}{\mathcal{E}_0} &= \kappa \frac{ \text{sin}(\theta)}{F(\theta)^{1/3}} + \frac{1+\text{cos}(\theta)}{F(\theta)^{2/3}}
\end{align}
$

## II - Mechanical Equilibrium

The local minimum of the energy gives the equilibrium configuration of the cell.

To find this minimum, we will use a library doing symbolic calculus in Python called sympy. This will allow us to compute the derivatives effortlessly.

### Symbolic Computation with Sympy *(In french "Calcul Formel")*

We will use sympy, a library that allows to do symbolic computation. Analytical results are always the best, but sometimes the equations does not lead to beautiful simplifications. If we are interested in the numerical result of the equations, we can use sympy to work on the analytical expression directly, obtain derivatives, etc.. before evaluating them on real values. There are three main functions that we will use in sympy, that we will present briefly. If interested, the best symbolic calculus tools are present in Wolfram Mathematica, which is under license.

#### a) Defining expression with symbols and trigonometric functions, and obtain derivatives :

```python
#!pip install sympy

import sympy
from sympy import symbols, diff,lambdify, simplify
from sympy import cos, sin
import numpy as np
from scipy.optimize import minimize
from scipy.optimize import fsolve,root
import matplotlib.pyplot as plt
from tqdm import tqdm
```

```python
#Symbols are unknown variables, that we aim to replace with real values in the end. We define them with the function symbols :
a,b,c = symbols("a b c")

#We can then define a symbolic expression, containing or not trigonometric functions (among many other possibilities !)
E = a**2 + a*b + cos(c)

#And obtain its derivatives with respect to any variables, eg a :
First_derivative = E.diff(a)
Second_derivative = First_derivative.diff(a)
First_derivative, Second_derivative
```

    (2*a + b, 2)

#### b) Substituting variables and evaluating symbolic expressions :

```python
# We can replace symbols with real variables with the method subs :
print("c = pi gives :",E.subs([(c,np.pi)]))
print("Subs method : ",E.subs([(a,2),(b,1),(c,0)]) )

#We can also transform a symbolic expression into a lambda function
#This is a faster process than subs if we need to evaluate the function on many points :
f = lambdify((a,b,c),E,"numpy")
print("Lambify method : ",f(2,1,0))

#We can combine both to replace certain variables before creating a lambda function with the remaining variables :
g = lambdify(a,E.subs([(b,1),(c,0)]))
print("Subs and lambdify combined :",g(2))

#Short benchmarking
from time import time
values_evaluated = np.linspace(0,np.pi,1000)
t1 = time()
g = lambdify(a,E.subs([(b,2),(c,0.2)]))
g(values_evaluated)
t2 = time()
for value in values_evaluated :
    E.subs([(a,value),(b,2),(c,0.2)])
t3 = time()

print("Time with lambdify :",round((t2-t1),4))
print("Time with subs :",round((t3-t2),4))
```

    c = pi gives : a**2 + a*b - 1.0
    Subs method :  7
    Lambify method :  7.0
    Subs and lambdify combined : 7
    Time with lambdify : 0.0019
    Time with subs : 1.0361

### The equilibrium configuration during cytokinesis

Let's go back to the initial problem. Our goal is to study the properties of the normalized energy $\overline{\mathcal{E}} = \mathcal{E}/\mathcal{E}_0$.

```python
x, k = symbols("x k")
F = 1 +1.5*cos(x) - 0.5*(cos(x))**3
energy = k * sin(x)/(F**(1/3)) + (1+ cos(x))/(F**(2/3))

#We see that there is no simplification easily given by sympy :
print(simplify(energy),'\n')

#We can replace the values of k and x :
print(energy.subs([(x,np.pi/3),(k,.5)]))

energy
```

    (0.873580464736299*k*(-0.333333333333333*cos(x)**3 + cos(x) + 0.666666666666667)**0.666666666666667*sin(x) + 0.763142828368888*(cos(x) + 1)*(-0.333333333333333*cos(x)**3 + cos(x) + 0.666666666666667)**0.333333333333333)*(-0.333333333333333*cos(x)**3 + cos(x) + 0.666666666666667)**(-1.0)

    1.42197524663604

$\displaystyle \frac{0.873580464736299 k \sin{\left(x \right)}}{\left(- 0.333333333333333 \cos^{3}{\left(x \right)} + \cos{\left(x \right)} + 0.666666666666667\right)^{0.333333333333333}} + \frac{0.763142828368888 \left(\cos{\left(x \right)} + 1\right)}{\left(- 0.333333333333333 \cos^{3}{\left(x \right)} + \cos{\left(x \right)} + 0.666666666666667\right)^{0.666666666666667}}$

Now that we have implemented our energy in sympy, we can automatically obtain the derivatives with the diff engine. We see that the analytical formulas are quite long and obtaining by hand the derivatives would be both painful and prone to errors.

#### **Question 3 :**
> Obtain the expression of the first and second derivatives of the energy $\overline{\mathcal{E}}$ with respect to theta (i.e x) with the diff function :

**Correction** :

```python
first_derivative = diff(energy,x)
second_derivative = diff(first_derivative,x)
```

#### **Question 4 :**
> Plot the energy profile $\overline{\mathcal{E}}(\theta), \theta \in [0,\dfrac{\pi}{2}]$ for $\kappa \in \{0.0,0.1,0.2,0.3,0.4,0.5\}$. What do you observe ?

**Correction** :

We see that at a moment, for a value $\kappa_c \in [0.4,0.5]$, the local minimum disappears. It means that there is no value $\theta >0$ where the cell is in a stable configuration. It gives a threshold from incomplete constriction (stable regime) towards full division (unstable regime).

```python
plt.figure(figsize = (15,25))
vals_theta = np.linspace(0,np.pi/2,10000)
for k_val in np.linspace(0,0.5,6):
    e = lambdify(x, energy.subs(k,k_val), "numpy")
    plt.plot(vals_theta,e(vals_theta),label = k_val.round(2))
plt.legend()
```

#### **Question 5 :**
> Starting with $\theta = \pi/4$, find the angle giving the local energy minimum for each k in values_k. Plot the equilibrium angle $\theta_{min}$, and the value of the derivatives of the energy $\left. \dfrac{\partial \mathcal{E}}{\partial \theta}\right\rvert_{\theta_{min}}$, $\left. \dfrac{\partial^2 \mathcal{E}}{\partial \theta^2}\right\rvert_{\theta_{min}}$, $\left (\left. \dfrac{\partial^2 \mathcal{E}}{\partial \theta^2}\right\rvert_{\theta_{min}}\right)^2$ at this angle for each k.

**Correction** :

```python
e = lambdify((x,k), energy, "numpy")
d = lambdify((x,k), first_derivative, "numpy")
s = lambdify((x,k), second_derivative, "numpy")
npoints = 10001
B = []
Derivatives = []
Second_Derivatives =[]
Solutions = []
values_k = np.linspace(0, 1, npoints)
eps = 1e-3

for j,k_val in enumerate(tqdm(values_k)) :
    #print(j)

    f = lambda x : e(x,k_val)
    g = lambda x : d(x,k_val)
    h = lambda x : s(x,k_val)
    sols = minimize(fun = f,x0=(np.pi/4),method = "SLSQP",bounds=[(0,np.pi/2)])

    assert (sols.success)

    min_theta = sols['x']
    if g(min_theta)<eps :
        B.append(k_val)
    Derivatives.append(g(min_theta))
    Second_Derivatives.append(h(min_theta))
    Solutions.append(min_theta)

Derivatives=np.array(Derivatives)
Second_Derivatives=np.array(Second_Derivatives)
Solutions=np.array(Solutions)

fig,ax = plt.subplots(1,4,figsize = (21,3))
ax[0].plot(values_k, Solutions*180/np.pi)
ax[0].set_xlabel("k")
ax[0].set_ylabel("minimum found")
ax[1].plot(values_k, Derivatives)
ax[1].set_xlabel("k")
ax[1].set_ylabel("derivative value at minimum")
ax[2].plot(values_k, Second_Derivatives)
ax[2].set_xlabel("k")
ax[2].set_ylabel("second derivative value at minimum")
ax[3].plot(values_k, Second_Derivatives**2)
ax[3].set_xlabel("k")
ax[3].set_ylabel("square of second derivative value at minimum")
```

#### **Question 6 :**
> Estimate the value $k_c$ where this local minimum disappears, and its associated angle $\theta_c$. Compute the values of the first two derivatives $\left. \dfrac{\partial \mathcal{E}}{\partial \theta}\right\rvert_{\theta_c} $ and  $\left. \dfrac{\partial^2 \mathcal{E}}{\partial \theta^2}\right\rvert_{\theta_c} $.

**Correction**

```python
k_crit=values_k[np.argmin(Second_Derivatives**2)]
g =  lambda x : d(x,k_crit)
h =  lambda x : s(x,k_crit)
theta_c = fsolve(g,np.pi/4)

print("Critical point :",(k_crit,theta_c[0]*180/np.pi))
print("First derivative :",g(theta_c),"Second derivative :", h(theta_c))
```

    Critical point : (0.4033, 43.47264310305924)
    First derivative : [5.55111512e-17] Second derivative : [-0.04922366]

In this rescaled energy, the disappearance of the local minimum $\theta > 0$ when k is increased above some threshold defines a critical point $(\kappa_c, \theta_c)$. This critical point, where the final cell jumps from an incomplete constriction to a full division, is given by :

$0 = \left. \dfrac{\partial \mathcal{E}}{\partial \theta}\right\rvert_{\theta_c} $ and  $0 = \left. \dfrac{\partial^2 \mathcal{E}}{\partial \theta^2}\right\rvert_{\theta_c} $

This is a first order phase transition.

#### **Question 7 :**
> Plot the equilibrium angles and draw the shape of the cells for $\kappa \in \{0,0.2,0.4,0.6\}$

**Correction**

```python
fig,ax = plt.subplots(1,5,figsize =(25,5))
colors = ['tab:blue','tab:orange','tab:green','tab:red']
R0 = 1
for j,idx_val in enumerate([0,2000,4000,6000]) :

    theta_sol = Solutions[idx_val]
    k_value = values_k[idx_val]
    R = R0/((F.subs(x,theta_sol[0]))**(1/3))
    e = lambdify((x), energy.subs(k,k_value), "numpy")

    theta_values_k = np.linspace(theta_sol, 2*np.pi-theta_sol,100)
    circle_x = R*np.cos(theta_values_k)
    circle_y = R*np.sin(theta_values_k)

    ax[0].plot(vals_theta*180/np.pi,e(vals_theta))
    ax[0].scatter(theta_sol*180/np.pi,e(theta_sol),s = 180)
    ax[0].set_ylabel("Energy")
    ax[0].set_xlabel("Angle value")
    ax[0].set_xlim(-5,95)

    ax[j+1].plot(circle_x-R*np.cos(theta_sol),circle_y,color=colors[j],linewidth=5)
    ax[j+1].plot(R*np.cos(theta_sol)-circle_x,circle_y,color=colors[j],linewidth=5)
    ax[j+1].set_title("Equilibrium angle value :" + str((theta_sol[0]*180/np.pi).round(2)))
    ax[j+1].set_xlim(-2,2)
    ax[j+1].set_ylim(-2,2)
    ax[j+1].set_aspect('equal')#, adjustable='box')
```

## III - Dynamics

We now want to study the furrow constriction dynamics, i.e the temporal evolution of $\dfrac{r_f}{R_0}$. We will establish these dynamics by expressing the derivative of this quantity with respect to $\theta$. As before, we will use the symbolic computation library sympy to evaluate numerical quantities.

To establish the dynamic equation, we note that the power of active effects is exactly dissipated by viscous cell deformations. The viscous dissipation is made of two contributions, the stretching of the poles and the constriction of the ring, which we estimate in scaling. The volume of acto-myosin in the poles is $V_p = 2A_p e_p$ and in the ring $V_f = 2\pi r_f w e_f$, where $w$ and $e_f$ are the width and thickness of the contractile ring. (We remind that the surface of each cell pole writes : $A_p = 2\pi R^2 (1+\text{cos}(\theta))$). The value $e_p \approx e_0$ and the ring thickness $e_f$ reach a steady-state value that depends on turnover. This yields the viscous dissipated power :

$\begin{align}
P_d &= \dfrac{1}{2} \eta \left[ V_p \left(\dfrac{1}{R} \dfrac{dr_f}{dt} \right)^2 + V_f \left( \dfrac{1}{r_f} \dfrac{dr_f}{dt}\right)^2  \right] \newline
P_d &= \dfrac{1}{2} \eta \left[ e_f 4 \pi R^2 (1+\text{cos}\theta) \dfrac{1}{R^2}  \left(\dfrac{dr_f}{dt} \right)^2 + 2\pi w e_f r_f \dfrac{1}{r_f^2} \left(  \dfrac{dr_f}{dt}\right)^2  \right]  \newline
&\approx \dfrac{1}{2} \eta \left[ 4 e_0 \pi(1+\text{cos}\theta)  \left(\dfrac{dr_f}{dt} \right)^2 + 2\pi w e_f\dfrac{1}{r_f} \left( \dfrac{dr_f}{dt}\right)^2  \right]  \newline
&= \left(\dfrac{dr_f}{dt} \right)^2 \dfrac{1}{2} \eta \left[ 4 \pi e_0 (1+\text{cos}\theta)   + \dfrac{4\pi}{2} w e_f\dfrac{F(\theta)^{1/3}}{R_0 \sin \theta}  \right]  \newline
&= \left(\dfrac{dr_f}{dt} \right)^2 4 \pi e_0 \eta \left[ (1+\text{cos}\theta)   + \dfrac{1}{2 R_0} w \dfrac{e_f}{e_0} \dfrac{F(\theta)^{1/3}}{\sin \theta}  \right]  \newline
&= \left(\dfrac{dr_f}{dt} \right)^2 4 \pi e_0 \eta \left[ (1+\text{cos}\theta)   + \lambda \dfrac{F(\theta)^{1/3}}{\sin \theta}  \right]  \newline
\end{align}
$

The balance of mechanical and dissipated powers yields :

$\dfrac{d \mathcal{E}}{dt} + P_d = 0$

Besides :

$
\dfrac{1}{\mathcal{E}_0} \dfrac{d\mathcal{E}}{dt} = \dfrac{\partial \mathcal{E}/\mathcal{E}_0}{\partial \theta} \dfrac{\partial \theta}{\partial r_f} \dfrac{d r_f}{d_t} = \dfrac{\partial \mathcal{E}/\mathcal{E}_0}{\partial \theta} \left(\dfrac{\partial r_f}{\partial \theta}\right)^{-1} \dfrac{d r_f}{d_t}
$

And, with $T_a=\frac{\eta e_0}{N^a_0}$ :

$\dfrac{1}{\mathcal{E}_0} P_d = \left(\dfrac{dr_f}{dt} \right)^2 \dfrac{4\pi e_0 \eta}{4 \pi R_0^2 N^a_0} \left[ (1+\text{cos}\theta)   + \dfrac{\lambda}{2} \dfrac{F(\theta)^{1/3}}{\sin \theta}  \right] = \left(\dfrac{dr_f}{dt} \right)^2 \dfrac{T_a}{ R_0^2} \left[ (1+\text{cos}\theta)   + \dfrac{\lambda}{2} \dfrac{F(\theta)^{1/3}}{\sin \theta}  \right] $

We have thus :

$
\begin{align}
\dfrac{\partial \mathcal{E}/\mathcal{E}_0}{\partial \theta} \left(\dfrac{\partial r_f}{\partial \theta}\right)^{-1} \dfrac{d r_f}{d_t}  &= - \left(\dfrac{dr_f}{dt} \right)^2 \dfrac{T_a}{ R_0^2} \left[ (1+\text{cos}\theta)   + \dfrac{\lambda}{2} \dfrac{F(\theta)^{1/3}}{\sin \theta}  \right]
\newline
\dfrac{\partial \mathcal{E}/\mathcal{E}_0}{\partial \theta} \left(\dfrac{\partial r_f/R_0}{\partial \theta}\right)^{-1} &= - \dfrac{dr_f}{dt} \dfrac{T_a}{ R_0} \left[ (1+\text{cos}\theta)   + \dfrac{\lambda}{2} \dfrac{F(\theta)^{1/3}}{\sin \theta}  \right] \newline
\dfrac{dr_f}{dt} \dfrac{T_a}{ R_0} &= - \dfrac{\partial \mathcal{E}/\mathcal{E}_0}{\partial \theta} \left(\dfrac{\partial r_f/R_0}{\partial \theta}\right)^{-1} \left[ (1+\text{cos}\theta)   + \dfrac{\lambda}{2} \dfrac{F(\theta)^{1/3}}{\sin \theta}  \right]^{-1} = -\mathcal{H}(\theta,\kappa,\lambda)
\end{align}
$

We will compute numerical the values of this function $\mathcal{H}$ to obtain the evolution of the furrow radius $r_f$

#### **Question 8 :**
> From the last equation, express the angle temporal variation $\dot \theta$.

**Correction** :

We have $\dfrac{r_f}{R_0} = \sin(\theta) F(\theta)^{-1/3}$, thus $\dfrac{d r_f/R_0}{dt} = \dot \theta \left(\dfrac{\partial r_f/R_0}{\partial \theta}\right) = \dot \theta \dfrac{\partial \left(\sin (\theta) F(\theta)^{-1/3}\right)}{\partial \theta} = \dot \theta \left(\sin (\theta) F(\theta)^{-1/3}\right)'$

Eventually :

$T_a \dot \theta
=- \mathcal{H}(\theta, \kappa, \lambda)\left(\dfrac{\partial r_f/R_0}{\partial \theta}\right)^{-1}
= - \dfrac{\partial \mathcal{E}/\mathcal{E}_0}{\partial \theta} \left(\dfrac{\partial r_f/R_0}{\partial \theta}\right)^{-2} \left[ (1+\text{cos}\theta)   + \dfrac{\lambda}{2} \dfrac{F(\theta)^{1/3}}{\sin \theta}  \right]^{-1} $

#### **Question 9 :**
> Compute numerically $\dot \theta$ with sympy, and integrate the evolution of $\theta$, $r_f$ in time with a forward-euler-scheme for $t \in [0,15]$, starting with $\theta(0)=\pi/2$, with $\lambda = 1$ and $\kappa \in \{0.1,0.25,0.4,0.5,0.75,1\}$. Check that it is compatible with the previous results obtained from the static analysis.

**Correction** :

At each timestep, we have :

$
\begin{align}
\theta(t+dt) &= \theta(t) + \dot \theta (t) dt \newline
r_f(t+dt) &= sin(\theta(t+dt)) F(\theta(t+dt))^{-1/3} R_0
\end{align}
$

```python
x, k, l = symbols("x k l")
F = 1 +1.5*cos(x) - 0.5*(cos(x))**3
energy = k * sin(x)/(F**(1/3)) + (1+ cos(x))/(F**(2/3))

r_f = sin(x)*(F**(-1/3))

dr_f = diff(r_f,x)
first_derivative = diff(energy,x)

H = ((1+cos(x)) + l*(F**(1/3))/sin(x))**(-1)*first_derivative/dr_f

dtheta = - H/dr_f
dtheta = dtheta.subs(l,0.1)
```

```python
theta0 = np.pi/2
R0 = 1

for k_value in [0.1,0.25,0.4,0.5,0.75,1]:

    expr_dtheta = dtheta.subs(k,k_value)
    func_dtheta = lambdify(x,expr_dtheta,"numpy")
    func_rf = lambdify(x,r_f,"numpy")

    npoints = 1000
    timepoints = np.linspace(0,15,npoints)
    Thetas = np.zeros(npoints)
    Rf = np.zeros(npoints)
    dt = timepoints[1]-timepoints[0]

    for j,t in enumerate(timepoints) :

        if j == 0 :
            Thetas[0] = theta0
            Rf[0]  = func_rf(Thetas[0])*R0
            continue

        Thetas[j] = Thetas[j-1] + dt*func_dtheta(Thetas[j-1])
        Rf[j]  = func_rf(Thetas[j])*R0

    plt.plot(timepoints,Rf,label = k_value)

plt.title("Constriction completion or failure")
plt.xlabel("time t/Ta")
plt.ylabel("furrow radius r_f/R_0")
plt.legend()
```

#### **Question 10 :**
> Determine the cytokinesis duration with $R_0 \in \{0.5,1,2,4\}$. Show that in case of cytokinesis completion $\lambda = 1, \kappa = 0.75$, the initial cell radius R0 has no impact on the cytokinesis time.

**Correction** :

```python
theta0 = np.pi/2
k_value = 0.75

for R0 in [0.5,1,2,4]:

    expr_dtheta = dtheta.subs(k,k_value)
    func_dtheta = lambdify(x,expr_dtheta,"numpy")
    func_rf = lambdify(x,r_f,"numpy")

    npoints = 1000
    timepoints = np.linspace(0,6,npoints)
    Thetas = np.zeros(npoints)
    Rf = np.zeros(npoints)
    dt = timepoints[1]-timepoints[0]

    for j,t in enumerate(timepoints) :

        if j == 0 :
            Thetas[0] = theta0
            Rf[0]  = func_rf(Thetas[0])*R0
            continue

        Thetas[j] = Thetas[j-1] + dt*func_dtheta(Thetas[j-1])
        Rf[j]  = func_rf(Thetas[j])*R0

    plt.plot(timepoints,Rf,label ="R0: " +str(R0))

plt.title("Cytokinesis time is independent of the cell initial radius")
plt.xlabel("time t/Ta")
plt.ylabel("furrow radius r_f/R_0")
plt.legend()
```

## Conclusion

Using both numerical and scaling analyses, the authors rationalized the physical mechanisms governing ring constriction in the cytokinesis of animal cells. A gradient of surface contractility from the poles to the equator is sufficient to drive cytokinesis and to reproduce its main features: formation of the contractile ring, cortical flow toward the equator, and furrow constriction.
This model provides quantitative criteria for cytokinesis completion and propose a general framework for interpreting and characterizing constriction failure. The authors furthermore demonstrate the physical mechanism leading to the independence of cytokinesis duration on the initial cell size in embryos.

The thin shell model described in the paper studied can be generalized in 3D. The numerical integration of the equations describing cortex dynamics is done via finite-element methods.
The results gives a biology accurate (and beautiful !) view of the cortex actin flows and of the cytokinesis dynamics. For more information, see :
> H. Borja Da Rocha et al., *A viscous active shell theory of the cell cortex*, 2021 \
https://arxiv.org/abs/2110.12089


# Laplace approxmation & free-action

$\DeclareMathOperator{\vec}{vec}$

### § Free-action

Free-action is the time-integral of free-energy

$$
\begin{eqnarray*}
    \overline F &=& \overline U - \overline H\\
    \overline U &=& \int dt \langle L(\psi)\rangle_{q_\psi} = \int dt \langle \ln p(y, \psi)\rangle_{q_\psi}\\
    \overline H &=& \int dt \langle q(\psi)\rangle_{q_\psi}
    \end{eqnarray*}
$$

### § Mean-field assumption

Now let our parameters factorise into states, parameters, and hyperparameters, $ \psi \to \{u(t), \theta, \lambda\}$,

and let the approximate densities over these variables follow mean-field assumption $q(\psi) = q(u, t)q(\theta)q(\lambda)$.

$\theta$ parameterises the first moment of the states, and is independent of $\lambda$, which parameterises the second moment.

Write

$$
\begin{eqnarray*}
    \overline U &=& \int dt \langle L(u, t, \theta, \lambda)\rangle_{q_uq_\theta q_\lambda}\\
    \overline H &=& \int dt \langle q(u, t)\rangle +
        \langle q(\theta)\rangle +
        \langle q(\lambda)\rangle
    \end{eqnarray*}
$$

### § Laplace approximation

Let

$$
\begin{eqnarray*}
    q(u, t) &=& N(\mu_u(t), \Sigma_u(t))\\
    q(\theta) &=& N(\mu_\theta, \Sigma_\theta)\\
    q(\lambda) &=& N(\mu_\lambda, \Sigma_\lambda)
    \end{eqnarray*}
$$

Naturally,

$$
\overline H = \frac 1 2\int dt \ln|\Sigma_u| +
    \frac 1 2 \ln|\Sigma_\theta| + \frac 1 2 \ln|\Sigma_\lambda| +
    \frac 1 2 \left( ND_u + D_\theta + D_\lambda\right)\ln 2\pi e
$$

For the internal action, $\overline U$, find its second-order truncation around its mode $\mu = \left[\mu_u, \mu_\theta, \mu_\lambda\right]^T$ (ignoring bilinear terms).

$$
\DeclareMathOperator{\tr}{tr}
\begin{eqnarray*}
    \overline U &=&
        \int L(\mu, t) \\
        &&+
        \left\langle
        \frac 1 2\left[
            (u - \mu_u)^T L^{(uu)} (u - \mu_u) \right.\right.\\
            && +\left.\left.
            (\theta - \mu_\theta)^T L^{(\theta\theta)} (\theta - \mu_\theta) \right.\right.\\
            && +\left.\left.
            (\lambda - \mu_\lambda)^T L^{(\lambda\lambda)} (\lambda - \mu_\lambda)
            \right]
        \right\rangle_{q_u q_\theta q_\lambda}dt\\
    &=& \int L(\mu, t) + \tr\left(\Sigma_u L^{(uu)}\right) +
        \tr\left(\Sigma_\theta L^{(\theta\theta)}\right) +
        \tr\left(\Sigma_\lambda L^{(\lambda\lambda)}\right)dt
    \end{eqnarray*}
$$

#### Finding conditional precision

Solve for $\partial\overline F/\partial{\Sigma_u(t)}=0$

$$
\begin{eqnarray*}
    &&\frac 1 2 \int dt L^{(uu)} - \frac 1 2 \int dt\Sigma^{-1}_u = 0\\
    \Rightarrow && \Sigma^{-1}_u(t) = - L^{(uu)}(\mu, t)
    \end{eqnarray*}
$$

Similarly,

$$
\begin{eqnarray*}
    \Sigma^{-1}_\theta &=& -\int dt L^{(\theta\theta)}(\mu, t)\\
    \Sigma^{-1}_\lambda &=& -\int dt L^{(\lambda\lambda)}(\mu, t)
    \end{eqnarray*}
$$

Note that

$$
\begin{eqnarray*}
    L(u, t, \theta, \lambda) &=&
        L(u, t|\theta, \lambda) + L(\theta) + L(\lambda)\\
    L^{(uu)} &=&
        L^{(uu)}(u, t| \theta, \lambda)\\
    L^{(\theta\theta)} &=&
        L^{(\theta\theta)}(u, t|\theta, \lambda) +
        L^{(\theta\theta)}(\theta)\\
    L^{(\lambda\lambda)} &=&
        L^{(\lambda\lambda)}(u, t|\theta, \lambda) +
        L^{(\lambda\lambda)}(\lambda)
    \end{eqnarray*}
$$

### § Variational action under Laplace approximation

With the following notation, one can write down the variational action, which is the internal action expected under their resepctive Markov Blanket.

$$
\begin{align}
    L_u &= L(u, t, \mu_\theta, \mu_\lambda)\\
    L_\theta &= L(\mu_u, t, \theta, \mu_\lambda)\\
    L_\lambda &= L(\mu_u, t, \mu_\theta, \lambda)
    \end{align}
$$

$$
\begin{align}
    {V}_u =
    V(u, t) &= \langle L(u, t, \theta, \lambda)\rangle_{q_\theta q_\lambda}\\
    &= L_u +
        \frac 1 2
        \left(
        \tr[\Sigma_\theta L_\theta^{(\theta\theta)}] +
        \tr[\Sigma_\lambda L_\lambda^{(\lambda\lambda)}]
        \right)\\
    \overline{V}_\theta =
    \overline V(\theta) &= \int dt \langle L(u, t, \theta, \lambda)\rangle_{q_u q_\lambda}\\
    &= \int L_\theta +
        \frac 1 2
        \left(
        \tr[\Sigma_u L_u^{(uu)}] +
        \tr[\Sigma_\lambda L_\lambda^{(\lambda\lambda)}]
        \right)dt\\
    \overline{V}_\lambda =
    \overline V(\lambda) &= \int dt \langle L(u, t, \theta, \lambda)\rangle_{q_u q_\theta}\\
    &= \int L_\lambda +
        \frac 1 2
        \left(
        \tr[\Sigma_u L_u^{(uu)}] +
        \tr[\Sigma_\theta L_\theta^{(\theta\theta)}]
        \right)dt\\
    \end{align}
$$

And the following differentials on variational actions will become useful later.

Note that the notation, $A\!\!:$, stands for matrix vectorisation, e.g., $L\!\!:_\theta^{(\theta\theta)}$ is a vectorisation of $L_\theta^{(\theta\theta)}$

$$
\left(
\begin{array}{ccc}
    L_\theta^{(\theta_{:1}\theta_{:1})} &
    L_\theta^{(\theta_{:1}\theta_{:2})} &
    \\
    L_\theta^{(\theta_{:2}\theta_{:1})} &
    L_\theta^{(\theta_{:2}\theta_{:2})} &
    \\
    &  &
    \ddots
    \end{array}
    \right)
$$

which becomes

$$\{
L_u^{(\theta_{:1}\theta_{:1})},
L_u^{(\theta_{:2}\theta_{:1})},
\cdots,
L_u^{(\theta_{:1}\theta_{:2})},
L_u^{(\theta_{:2}\theta_{:2})},
\cdots
\}^T$$

This is to be contrasted with, say, $L\!\!:_\theta^{(\theta\theta)(u)} and L\!\!:_\theta^{(\theta\theta)(uu)}$, which read

$$
\begin{align}
    &\left(
    \begin{array}{c}
        L\!\!:_\theta^{(\theta\theta)(u_{:1})}\\
        L\!\!:_\theta^{(\theta\theta)(u_{:2})}\\
        \vdots
        \end{array}
        \right), \;\;\;\text{ and}\\
    &\left(
    \begin{array}{ccc}
        L\!\!:_\theta^{(\theta\theta)(u_{:1}u_{:1})} &
        L\!\!:_\theta^{(\theta\theta)(u_{:1}u_{:2})} &
        \\
        L\!\!:_\theta^{(\theta\theta)(u_{:2}u_{:1})} &
        L\!\!:_\theta^{(\theta\theta)(u_{:2}u_{:2})} &
        \\
        & \ddots &
        \end{array}
        \right),
    \end{align}
$$

respectively.

Adopting these notations, the differentials of variational actions are

$$
\begin{eqnarray*}
    V_u^{(u)} &= L_u^{(u)}& +\frac 1 2
        \left(I\otimes\Sigma^T\!\!\!:_\theta\right)^T
        L\!\!:_\theta^{(\theta\theta)(u)} +
        \frac 1 2
        \left(I\otimes\Sigma^T\!\!\!:_\lambda\right)^T
        L\!\!:_\lambda^{(\lambda\lambda)(u)}\\
    V_u^{(uu)} &= L_u^{(uu)}& +\frac 1 2
        \left(I\otimes\Sigma^T\!\!\!:_\theta\right)^T
        L\!\!:_\theta^{(\theta\theta)(uu)} +
        \frac 1 2
        \left(I\otimes\Sigma^T\!\!\!:_\lambda\right)^T
        L\!\!:_\lambda^{(\lambda\lambda)(uu)}
    \end{eqnarray*}
$$

$$
\begin{eqnarray*}
    \overline V_\theta^{(\theta)} &= \int L_\theta^{(\theta)}& +
        \frac 1 2 \left(
        I\otimes \Sigma^T\!\!\!:_u\right)^T
        L\!\!:_u^{(uu)(\theta)} +
        \frac 1 2 \left(
        I\otimes \Sigma^T\!\!\!:_\lambda\right)^T
        L\!\!:_\lambda^{(\lambda\lambda)(\theta)}dt\\
    \overline V_\theta^{(\theta\theta)} &= \int L_\theta^{(\theta\theta)}& +
        \frac 1 2 \left(
        I\otimes \Sigma^T\!\!\!:_u\right)^T
        L\!\!:_u^{(uu)(\theta\theta)} +
        \frac 1 2 \left(
        I\otimes \Sigma^T\!\!\!:_\lambda\right)^T
        L\!\!:_\lambda^{(\lambda\lambda)(\theta\theta)}dt
    \end{eqnarray*}
$$

$$
\begin{eqnarray*}
    \overline V_\lambda^{(\lambda)} &= \int L_\lambda^{(\lambda)}& +
        \frac 1 2 \left(
        I\otimes \Sigma^T\!\!\!:_u\right)^T
        L\!\!:_u^{(uu)(\lambda)} +
        \frac 1 2 \left(
        I\otimes \Sigma^T\!\!\!:_\theta\right)^T
        L\!\!:_\theta^{(\theta\theta)(\lambda)}dt\\
    \overline V_\lambda^{(\lambda\lambda)} &= \int L_\lambda^{(\lambda\lambda)}& +
        \frac 1 2 \left(
        I\otimes \Sigma^T\!\!\!:_u\right)^T
        L\!\!:_u^{(uu)(\lambda\lambda)} +
        \frac 1 2 \left(
        I\otimes \Sigma^T\!\!\!:_\theta\right)^T
        L\!\!:_\theta^{(\theta\theta)(\lambda\lambda)}dt
    \end{eqnarray*}
$$

### § Optimisation: embedding and mode following

Suppose the time-dependent state, $u$, subsumes its motion up to arbitrary high order, one may unpack this and write $\tilde u = (u, u', u'', \dots)^T$.

Let this generalised state move along the gradient of variational energy/action, hoping to catch up the motion one level above when the gradient vanishes:

$$
\begin{align}
    \dot{\tilde u} &= V_u^{(u)} + \mathcal D\tilde u
    \end{align}
$$

This way, when $V_u^{(u)} = 0$ (this happens at the mode where $\tilde u = \tilde\mu$), one has

$$
\begin{align}
    \dot u &= u'\\
    \dot u' &= u''\\
    \dot u'' &= u'''\\
    &\vdots
    \end{align}
$$

Thus, motion of the modes becomes modes of the motion. Here, $\mathcal D$ is a differential operator, or simply a delay matrix.

Let us find the linearisation of this state motion around its mode, $\tilde\mu$, which follows that

$$
V_u^{(u)} = V(\tilde\mu)_u^{(u)} + V(\tilde\mu_u)^{(uu)}(\tilde u - \tilde\mu) = V(\tilde\mu_u)^{(uu)}(\tilde u - \tilde\mu)
$$

And have $\tilde\varepsilon = \tilde u - \tilde\mu$, so that $\dot{\tilde\varepsilon} =
\dot{\tilde u} - \dot{\tilde\mu} = \dot{\tilde u} - \mathcal D\tilde\mu$.

With substitution, write

$$
\begin{align}
    \dot{\tilde\varepsilon} &=
        V_u^{(uu)}\tilde\varepsilon + \mathcal D\tilde u - \mathcal D\tilde\mu\\
    &= \left(V_u^{(uu)} +\mathcal D\right)\tilde\varepsilon
    \end{align}
$$

and note $\mathcal J = \left( V_u^{(uu)} + \mathcal D\right) = \partial\dot{\tilde u}/\partial\tilde u$.

#### Finding conditional expectation (updating scheme)

The updating scheme is again derived from Ozaki's local linearisation:

$$
\begin{align}
    \Delta \tilde u &= \left(
        \exp(\mathcal J) - I\right)
        \mathcal J^{-1}
        \dot{\tilde u}\\
    &= \left(
        \exp(\mathcal J) - I\right)
        \mathcal J^{-1}
        \left(V_u^{(u)} + \mathcal D\tilde u\right)
    \end{align}
$$

For parameters and hyperparameters, this reduces to

$$
\begin{align}
    \Delta \theta &= {\overline V_\theta^{(\theta\theta)}}^{-1}
        \overline V_\theta^{(\theta)}\\
    \Delta \lambda &= {\overline V_\lambda^{(\lambda\lambda)}}^{-1}
        \overline V_\lambda^{(\lambda)}
    \end{align}
$$

### § Example 1: Dynamic causal model

$$
\begin{align}
    y &= g(x, v; \theta) + z,\;\;\; z\sim N(0, \Pi(\lambda)^{-1}_z)\\
    \dot x &= f(x, v; \theta) + w,\;\;\; w\sim N(0, \Pi(\lambda)^{-1}_w)
    \end{align}
$$

$$
\begin{align}
    \ln p(y, u, t, \theta, \lambda) &= L(y|x, v, \theta, \lambda) + L(x|v, \theta,\lambda) + L(\theta) + L(\lambda)
    \end{align}
$$

where $u=(v, x)^T$ and let $p(v)$ be uninformative for now. And let the parameter, $\theta$, and hyperparameter, $\lambda$, be independent and take the following form

$$
\begin{align}
    p(\theta) &= N(\theta|\eta_\theta, P^{-1}_\theta)\\
    p(\lambda) &= N(\lambda|\eta_\lambda, P^{-1}_\lambda)
    \end{align}
$$

which altogether lend the generative density to an analytical form

$$
\begin{align}
    L(t) &= -
        \frac 1 2 \varepsilon_v^T\Pi_z\varepsilon_v -
        \frac 1 2 \varepsilon_x^T\Pi_w\varepsilon_x -
        \frac 1 2 \varepsilon_\theta^T P_\theta \varepsilon_\theta -
        \frac 1 2 \varepsilon_\lambda^T P_\lambda \varepsilon_\lambda -
        \frac 1 2 \ln|\Pi_z| -
        \frac 1 2 \ln|\Pi_w| -
        \frac 1 2 \ln|P_\theta| -
        \frac 1 2 \ln|P_\lambda|
    \end{align}
$$

where

$$
\begin{align}
    \varepsilon_v &= y - g(x, v)\\
    \varepsilon_x &= \dot x - f(x, v)\\
    \varepsilon_\theta &= \theta - \eta_\theta\\
    \varepsilon_\lambda &= \lambda - \eta_\lambda
    \end{align}
$$

If one lumps the time-dependent terms together:

$$
\begin{align}
    \varepsilon_u &= \left( \varepsilon_v, \varepsilon_x \right)^T\\
    \Pi &= \left(\begin{array}{cc}
        \Pi_z & \\
        & \Pi_w
        \end{array}\right) =
        \left(\begin{array}{cc}
        C_z^{-1} &\\
        & C_w^{-1}
        \end{array}
        \right) = C^{-1}
    \end{align}
$$

one writes

$$
\begin{align}
    L(t) &= -
        \frac 1 2 \varepsilon_u^T\Pi\varepsilon_u -
        \frac 1 2 \varepsilon_\theta^T P_\theta \varepsilon_\theta -
        \frac 1 2 \varepsilon_\lambda^T P_\lambda \varepsilon_\lambda -
        \frac 1 2 \ln|\Pi| -
        \frac 1 2 \ln|P_\theta| -
        \frac 1 2 \ln|P_\lambda|
    \end{align}
$$

And let this expression prescribe generalised motion over its states, write

$$
\begin{align}
    L(t) &= -
        \frac 1 2 \tilde{\varepsilon_u}^T \tilde\Pi \tilde{\varepsilon_u} -
        \frac 1 2 \varepsilon_\theta^T P_\theta \varepsilon_\theta -
        \frac 1 2 \varepsilon_\lambda^T P_\lambda \varepsilon_\lambda -
        \frac 1 2 \ln|\tilde\Pi| -
        \frac 1 2 \ln|P_\theta| -
        \frac 1 2 \ln|P_\lambda|
    \end{align}
$$

#### Conditional precision $\pmb\Lambda_u$

For state $u$, the conditional precision, $\Lambda_u=-L(t)^{(uu)}$, is

$$
\tilde{\varepsilon_u^{(u)}}^T \tilde\Pi \tilde{\varepsilon_u^{(u)}}
$$

where

$$
\begin{align}
    \tilde{\varepsilon_u^{(u)}} &= \left(
        \begin{array}{cc}
            \tilde{\varepsilon_v^{(v)}} &
            \tilde{\varepsilon_v^{(x)}} \\
            \tilde{\varepsilon_x^{(v)}} &
            \tilde{\varepsilon_x^{(x)}}
            \end{array}
            \right)\\
    \tilde{\varepsilon_v^{(v)}} &= \left(
        \begin{array}{c}
            \varepsilon_v^{(\tilde v)}\\
            \varepsilon_{v'}^{(\tilde v)}\\
            \varepsilon_{v''}^{(\tilde v)}\\
            \vdots
            \end{array}
            \right) = \left(
        \begin{array}{cccc}
            \varepsilon_v^{(v)} &
            \varepsilon_v^{(v')} &
            \varepsilon_v^{(v'')} &
            \\
            \varepsilon_{v'}^{(v)} &
            \varepsilon_{v'}^{(v')} &
            \varepsilon_{v'}^{(v'')} &
            \cdots
            \\
            \varepsilon_{v''}^{(v)} &
            \varepsilon_{v''}^{(v')} &
            \varepsilon_{v''}^{(v'')} &
            \\
             & \vdots & & \ddots
            \end{array}
            \right) =
        -I \otimes g^{(v)}\\
    \tilde{\varepsilon_x^{(x)}} &= \left[\mathcal D\tilde x - \tilde f\right]^{(\tilde x)} =
        \mathcal D - I \otimes f^{(x)}
    \end{align}
$$

See [A.4 generalised motion](?).

Thus,

$$
\tilde{\varepsilon_u^{(u)}} = -\left(
    \begin{array}{cc}
        I\otimes g^{(v)} &
        I\otimes g^{(x)} \\
        I\otimes f^{(v)} &
        I\otimes f^{(x)} - \mathcal D
        \end{array}
        \right)
$$

#### Conditional precision $\pmb\Lambda_\theta$

Conditional precision over parameter, $\Lambda_\theta = -L(t)^{(\theta\theta)}$ is

$$
\tilde{\varepsilon_u^{(\theta)}}^T \tilde\Pi \tilde{\varepsilon_u^{(\theta)}} + P_\theta
$$

Let $\theta = (\theta_{:1}, \theta_{:2}, \dots, \theta_{:k}, \dots, \theta_{:K})^T$.

$$
\begin{align}
    \tilde{\varepsilon_u^{(\theta_{:k})}} &= \left(
        \begin{array}{c}
            \tilde y - \tilde g\\
            \mathcal D\tilde x - \tilde f
            \end{array}
            \right)^{(\theta_{:k})} = -\left(
        \begin{array}{c}
            \tilde{g^{(\theta_{:k})}}\\
            \tilde{f^{(\theta_{:k})}}
            \end{array}
            \right)\\
    &= \left(
        \begin{array}{cc}
            I\otimes g^{(v\theta_{:k})} &
            I\otimes g^{(x\theta_{:k})} \\
            I\otimes f^{(v\theta_{:k})} &
            I\otimes f^{(x\theta_{:k})}
            \end{array}
            \right) \left(
        \begin{array}{c}
            \tilde v\\
            \tilde x
            \end{array}
            \right)
    \end{align}
$$

#### Conditional precision $\pmb\Lambda_\lambda$

Conditional precision over hyperparameter, $\Lambda_\lambda = -L(t)^{(\lambda\lambda)}$, is, assuming $\lambda_i, \lambda_j\in\lambda$

$$
\begin{align}
    &\frac{\partial^2}{\partial\lambda_j\partial\lambda_i}\left(
        -\frac 1 2 \tilde{\varepsilon_u}^T\tilde\Pi\tilde{\varepsilon_u} -
        \frac 1 2 \ln|\Pi|\right)\\
    =& -\frac 1 2
        \frac{\partial}{\partial\lambda_j}\left[
        \tilde{\varepsilon_u}^T \frac{\partial}{\partial\lambda_i}\tilde\Pi \tilde{\varepsilon_u} +
        \frac{\partial}{\partial\lambda_i}\ln|\tilde\Pi|
        \right]\\
    =& -\frac 1 2
        \tilde\Pi^{(\lambda_i)}
        \tr\left[
        \tilde C \tilde\Pi^{(\lambda_j)} \tilde C \tilde\Pi^{(\lambda_i)}
        \right]
    \end{align}
$$

where

$$
\begin{align}
    & \tilde{\varepsilon_u}^T \frac{\partial}{\partial\lambda_i}\tilde\Pi \tilde{\varepsilon_u}\\
    &= \vec(\tilde{\varepsilon_u}\tilde{\varepsilon_u}^T)^T
        \vec(\tilde{\Pi}^{(\lambda_i)})\\
    &= \tr\left[
        \tilde{\varepsilon_u} \tilde{\varepsilon_u}^T \tilde{\Pi}^{(\lambda_i)}
        \right]
    \end{align}
$$

$$
\begin{align}
    \frac{\partial}{\partial\lambda_i}\ln|\tilde\Pi| &=
        \frac{\partial\tilde\Pi}{\partial\lambda_i}
        \frac{\partial|\tilde\Pi|}{\partial\tilde\Pi}
        \frac{\partial}{\partial|\tilde\Pi|}\ln|\tilde\Pi|\\
    &= \tilde\Pi^{(\lambda_i)}
        |\tilde\Pi|\tr\left[\tilde C \tilde\Pi^{(\lambda_i)}\right]
        \frac{1}{|\tilde\Pi|}\\
    &= \tilde\Pi^{(\lambda_i)}
        \tr\left[\tilde C \tilde\Pi^{(\lambda_i)}\right]
    \end{align}
$$

\begin{align}
    \frac{\partial^2}{\partial\lambda_j\partial\lambda_i}\ln|\tilde\Pi| &=
        \frac{\partial}{\partial\lambda_j}
        \tilde\Pi^{(\lambda_i)}
        \tr\left[\tilde C \tilde\Pi^{(\lambda_i)}\right]\\
    &=\tilde\Pi^{(\lambda_i\lambda_j)}
        \tr\left[
        \tilde C \tilde\Pi^{(\lambda_i)}
        \right] +
        \tilde\Pi^{(\lambda_i)}
        \tr\left[
        \tilde C \tilde\Pi^{(\lambda_j)} \tilde C \tilde\Pi^{(\lambda_i)} +
        \tilde C \tilde\Pi^{(\lambda_i\lambda_j)}
        \right]\\
    &= \tilde\Pi^{(\lambda_i)}
        \tr\left[
        \tilde C \tilde\Pi^{(\lambda_j)} \tilde C \tilde\Pi^{(\lambda_i)}
        \right]\;\;\;\;(
        \text{if $\partial^2 \tilde\Pi/\partial\lambda\partial\lambda = 0$})
    \end{align}

See [Differentials of determinant](http://www.ee.ic.ac.uk/hp/staff/dmb/matrix/calculus.html#deriv_det)
, [Differentials of inverses and trace](http://www.ee.ic.ac.uk/hp/staff/dmb/matrix/calculus.html#deriv_inv) from Matrix Reference Manual.

#### Conditional expectation over state (updating scheme)

Before calling upon Ozaki's scheme, one recalls that the observation, which affects the variational energy as well, has to be considered:

$$
\left(\begin{array}{c}
    \dot{\tilde y}\\
    \dot{\tilde u}
    \end{array}\right) =
\left(\begin{array}{c}
    \mathcal D\tilde y\\
    V_u^{(u)} + \mathcal D \tilde u
    \end{array}\right) \implies
\mathcal J =
\left(\begin{array}{cc}
    \partial\dot{\tilde y}/\partial\tilde y &
    \partial\dot{\tilde y}/\partial\tilde u \\
    \partial\dot{\tilde u}/\partial\tilde y &
    \partial\dot{\tilde u}/\partial\tilde u
    \end{array}\right) =
\left(\begin{array}{cc}
    \mathcal D & 0\\
    V_u^{(uy)} &
    V_u^{(uu)} + \mathcal D
    \end{array}\right)
$$

Thus,

$$
\left(\begin{array}{c}
    \Delta\tilde y\\
    \Delta\tilde u
    \end{array}
    \right) =
\left(\exp(\mathcal J) - I\right)
\mathcal J^{-1}
\left(\begin{array}{c}
    \mathcal D\tilde y\\
    V_u^{(u)} + \mathcal D
    \end{array}
    \right)
$$

*****

$$
\begin{align}
    \dot u &= g(u)\\
    \ddot u &= \dot u g^{(u)}\\
    \dddot u &= \frac{\partial}{\partial t} \ddot u\\
    &= \dot u \frac{\partial}{\partial u} (\dot u g^{(u)})\\
    &= \dot u \frac{\partial}{\partial u} (g(u) g^{(u)})\\
    &= \underbrace{\dot u g^{(u)}}_{=\ddot u} g^{(u)} +
        \underbrace{\dot u g(u)g^{(uu)}}_{\text{ignored under local linearisation}}
    \end{align}
$$

Laplace approximation for dynamic model.ipynb