```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.cm as cm
import numpy.random as rng
np.set_printoptions(precision = 2)
plt.rcParams['figure.figsize'] = (10.0, 5.0)
plt.rcParams['image.interpolation'] = 'nearest'
plt.rcParams['image.cmap'] = 'gray'

def sigmoid(x):
    return 1.0/(1.0 + np.exp(-x))
```

## The vanilla RBM.

   * weights $W_{ji}$ means the weight between the j-th hidden unit and the i-th visible unit.
   * $W_{0i}$ is "bias" into the i-th visible unit.
   * $W_{j0}$ is "bias" into the j-th hidden unit.
The joint probability under the RBM factorisation is:
$$ P^\star(h,v) = \prod_i \prod_j e^{h_j W_{ji} v_i} \;\; \times \;\; \prod_{i^\prime} e^{W_{0i^\prime} v_{i^\prime}} \;\; \times \;\; \prod_{j^\prime} e^{W_{j^\prime 0} h_{j^\prime}} $$
and its logarithm is
$$ \log P^\star(h,v) = \sum_i  \sum_j h_j W_{ji} v_i \;\; + \;\; \sum_i W_{0i} v_i \;\; + \;\; \sum_j W_{j0} h_j $$

## Gibbs in vanilla RBM
To sample from this distribution we can figure out the Gibbs update step.

The probability $p(h_j=1|v)$ that the j-th hidden unit generates a 1 can be written as $ \sigma(\psi_j) $ with $\sigma(x)=1/(1+e^{-x})$ and $\psi_j = \log P^*(h,v | h_j =1 ) - \log P^*(h,v | h_j = 0)$.
And from the $\log P^\star(h,v)$ given above, this is easily seen to be
$ \psi_j(v) = \sum_i W_{ji} v_i + W_{j0}$

Similarly, for the visible units the probability $p(v_i=1|h)$ that the i-th visible generates a 1 can be written as
$\sigma(\phi_i(h)) $ with $\phi_i(h) = \log P^*(h,v | v_i =1 ) - \log P^*(h,v | v_i = 0)$.
And similarly this can be seen to be
$ \phi_i(h) = \sum_j W_{ji} h_j + W_{0i}$ for the i-th visible unit.

For convenience we will sometimes write $\sigma(\phi_i(h))$ as just $\sigma_i(h)$.

## Gibbs in an equivalent network

Note this is equivalent to a 3 layer network in which the top 2 layers form an RBM and the lower one is a sigmoid belief network.
Sampling from the latter model involves drawing Bernoulli variables repeatedly (eg. via alternating Gibbs sampling) from the RBM until equilibrium and then sampling $v$ from the visibles given the hiddens $h$, ie. "ancestral sampling" in the belief net, but from an $h$ generated by an RBM.

We know (from above) how to generate the $h$ sample: Gibbs sampling from the RBM will do it. Then, the conditional probability of $v$ under a sigmoid belief net is (by definition) $p(v_i=1|h) = \sigma_i(h)$.
Thus _Gibbs sampling from a simple RBM ending in a sample for $v$ is the same as sampling $h$ from the same RBM and then using a sigmoid belief net for the last step_.

However, there's another way to draw such samples. Write (product rule) $\log P^\star(h,v) = \log P^\star(h) + \log P(v|h)$. We have the second term already:
$$ \log P(v|h) = \sum_i v_i \log \sigma_i(h) + (1-v_i) \log (1 - \sigma_i(h)$$

To find $P^\star(h)$ we need to marginalise that joint over all $\mathbf{v}$ configurations:
$$\begin{align} P^\star(h) &= \sum_{v_1=0}^1 \cdots \sum_{v_n=0}^1 \exp \bigg[  \log P^{\star}(h,v) \bigg] \\
&= \sum_{v_1=0}^1 \cdots \sum_{v_n=0}^1 \exp \bigg[  \sum_i  \sum_j h_j W_{ji} v_i \;\; + \;\; \sum_i W_{0i} v_i \;\; + \;\; \sum_j W_{j0} h_j \bigg] \\
&= \sum_{v_1=0}^1 \cdots \sum_{v_n=0}^1 \exp \bigg[  \sum_i v_i \phi_i(h)  \;\; + \;\; \sum_j W_{j0} h_j \bigg] \\
\text{where } \phi_i(h) &= \sum_j W_{ji} h_j + W_{0i} \\
&= \exp\left[ \sum_j h_j  W_{j0} \right] \;\; \times \sum_{v_1=0}^1 \cdots \sum_{v_n=0}^1 \prod_i \exp\bigg[ v_i \phi_i(h) \bigg] \\
&= \exp\left[\sum_j h_j  W_{j0}\right] \;\; \times \prod_i \bigg( 1 + e^{\phi_i(h) } \bigg) \\
\text{and so}
\log P^\star(h) &= \sum_j h_j  W_{j0} \;\; +  \sum_i \log \bigg( 1 + e^{\phi_i(h) } \bigg)
\\
&= \sum_j h_j  W_{j0} \;\; + \; \sum_i \phi_i(h) \;  - \; \sum_i \log \sigma_i(h)
\end{align} $$

So far we've figured out $\log P^\star(h)$ for the RBM that is the "top layer".

Therefore another way to write $\log P^\star(h,v)$ is
$$
\log P^\star(h,v) = \underbrace{\sum_j h_j  W_{j0} \;\; + \; \sum_i \phi_i(h) \;  - \; \sum_i \log \sigma_i(h)}_{\log P^\star(h)} \;\;+\;\; \underbrace{\sum_i v_i \log \sigma_i(h) + (1-v_i) \log (1 - \sigma_i(h))}_{\log P(v \mid h)}
$$
By collecting terms and simplifying one can readily those that this matches the earlier form.

# a two-causes model

Now we'd like to change this slightly, so that 2 RBMs that are independent are used to model two causes, which are then combined at the last moment via a sigmoid belief net to form $v$.
Suppose that at a given moment the 2nd RBM is in a state which contributes an extra activation $\epsilon_i$ respectively to each of the visible units. We're interested in how this will affect the Gibbs updates to the hidden units $h$ in the first RBM.

Note: $\epsilon_i$ is in general going to be a weighted sum of inputs from the second RBM's hidden layer, _plus a new bias_ arising from the second RBM. Although the two biases both going into the visible units seems (and might be) redundant, in the generative model it seems sensible that visible activations under the two "causes" would have different background rates if taken separately (ie. different biases). So for now I think we should leave them in, but maybe hope to eliminate / merge if possible in future, if it helps anything...

Before going on to the Gibbs Sampler version in which visible units are clampled, consider "ancestral" sampling from the model: each RBM independently does alternating Gibbs Sampling for a (longish) period, and then both combine to generate a sample $v$ vector, by adding both their weighted sums ($\phi$) _and adding in both their visible biases too_ . That's a much more efficient way to do the "sleep" phase than doing what follows (which is mandatory for the "wake" phase samples however).

The Gibbs update step is given by $p_j = \sigma(\phi_j) $ with
$ \psi_j = \log P^*(h,v ; h_j =1 ) - \log P^*(h,v ; h_j = 0)$.

However this time we don't have exact correspondence with an RBM because only the final step involves $\epsilon$, not the reverberations in the RBM above it that generates $h$. So it's not enough to consider just the RBM alone, with it's joint being just the product of factors in the first line of math above. We need to incorporate the last step explicitly, with its slight difference in the form of $\phi^B_i$. We know the joint decomposes into this:
$$ \log P^* (h,v) = \log P^*(h) + \log P(v|h)$$
where the first term is the vanilla RBM probability but the second is the final layer's probability, now given by
$$ \log P(v|h^A,h^B) = \sum_i v_i \log \sigma (\phi^A_i(h) + \phi^B_i) + (1-v_i) \log (1 - \sigma(\phi^A_i(h) + \phi^B_i)$$

To carry out Gibbs sampling in the hidden layer of this architecture we need to calculate $\psi^A_j = \log P^*(h,v ; h^A_j =1 ) - \log P^*(h,v ; h^A_j = 0)$. We'll use the fact that $\phi^A_i(h ; h^A_j=1) = \phi^A_i(h ; h^A_j=0) + W^A_{ji}$.

And we'll abbreviate $\phi^A_i(h ; h_j=0)$ to $\phi^{Aj0}_i$.

We obtain:
$$\psi^A_j = \sum_i v_i \log \left( \frac{1+ e^{-\phi^{Aj0}_i - \phi^B_i}}{1+e^{-\phi^{Aj0}_i - W_{ji} -\phi^B_i}} \frac{1+ e^{\phi^{Aj0}_i + W_{ji} + \phi^B_i}}{1+e^{\phi_i^{Aj0} + \phi^B_i}}\right) \;\;+ \;\;\sum_i \log \left(\frac{1+e^{\phi_i^{Aj0} + W_{ji}}}{1+ e^{\phi_i^{Aj0}}}
\frac{1+e^{\phi_i^{Aj0} + \phi^B_i}}{1+ e^{\phi_i^{Aj0} + W_{ji} + \phi^B_i}} \right)$$

Now $\phi = \log \frac{1+e^{\phi}}{1+e^{-\phi}}$ (Marcus' magic identity), which is$ = \log \frac{\sigma(\phi)}{\sigma(-\phi)}$.
So the first term simplifies to
$ \sum_i v_i W_{ji}$, which is the same as that in a "vanilla RBM".

The second term can also be simplified, using the identity $\log(1-\sigma(\phi)) = \phi - \log(1+e^\phi)$.

This leads to the following Gibbs Sampler probability of the j-th hidden unit in network $A$ being 1: $p_j = \sigma(\psi_j^A)$ with

$$\psi_j^A = \underbrace{\sum_i W^A_{ji} v_i}_\text{vanilla RBM} \; + \; \underbrace{\sum_i C^A_{ji}}_\text{correction} $$
where
$$
\begin{align}
C^A_{ji} \; &= \;\log \bigg[ \frac{\sigma (\phi_i^{Aj0})}{\sigma (\phi_i^{Aj0} + W^A_{ji})} . \frac{\sigma (\phi_i^{Aj0} + W_{ji}^A + \phi_i^B) }{\sigma (\phi_i^{Aj0} + \phi_i^B)} \bigg]
\\
&= \log \sigma(\phi_i^{Aj0})  \; + \; \log \sigma (\phi_i^{Aj0} + W^A_{ji} + \phi_i^B) \;- \log \sigma (\phi_i^{Aj0} + W^A_{ji})  \; - \; \log \sigma ( \phi_i^{Aj0} + \phi_i^B)
\\
&= \log \bigg[ \frac{\sigma(\phi_i^{A} - h^A_i W^A_{ji})}{\sigma (\phi_i^{A} + (1-h^A_i) W^A_{ji})} \bigg]  \; - \; \log \bigg[ \frac{ \sigma ( \phi_i^{AB} - h^A_i W^A_{ji})}{\sigma (\phi_i^{AB} + (1-h^A_i) W^A_{ji})} \bigg]
\end{align}
$$
where $\phi_i^{AB} = \phi_i^{A} + \phi_i^{B}$.

Notice that, weirdly enough, $v$ plays no role in this correction!

It is clear that adding $\phi^B_i$ has introduced a dependency between the whole of $h$, which is "a bit of a worry".

### thinking about that computation

I want to think of of the correction "being computed somewhere" in the network - where? The natural place seems to be the connection itself.

Computation of $\psi^A_j$ seems to require the $ji$ connection knowing this:
   *  $W^A_{ji}$, the current weight
   *  $v_i$, the current output state

So far so good: the vanilla RBM needs those too, in order to compute the usual dot product. What is new is the requirement that the connection also needs to know...
   *  $h^A_j$, the activity on the hidden end
   * $\phi^A_i$
   * $\phi^{AB}_i$

On the face of it this isn't _so_ horrible. But it would be great if there were a simpler way to view this gruesome looking expression.

## What does the correction look like?

Thought: the "correction" is all hinge functions and should have an approximation as "if../else.." piecewise linear regimes, and this ought to have an intuitive hand-wave type explanation that makes sense.

So let's plot the correction contours on axes $\phi^A_i$ versus $\phi^{AB}_i$, for (say) a positive weight $W^A_{ij}$.
We'll need two plots for the two cases $h^A_j=0,1$

```python
def calc_psi_correction(phiA, phiAB, w, h):
    # note here the Phi is the full weighted sum into the visible node. We're explicitly taking care of the hidden activation too.
    correction = np.log(sigmoid(phiA - h*w)/sigmoid(phiA + (1-h)*w)) - np.log(sigmoid(phiAB - h*w)/sigmoid(phiAB + (1-h)*w))
    return correction

reach = 40
phiA, phiAB = np.mgrid[-reach:reach:100j, -reach:reach:100j]
wgt = 3.0
levels = np.arange(-abs(wgt)-0.5, abs(wgt)+0.5, abs(wgt)/10.)
cmap = cm.RdYlGn

plt.subplot(121)
psi_correction_h0 = calc_psi_correction(phiA, phiAB, wgt, 0)
C = plt.contourf(phiA, phiAB, psi_correction_h0, levels, origin='lower', cmap=cm.get_cmap(cmap, len(levels)-1))
#plt.colorbar()
plt.title('$h_j=0$')
plt.xlabel('$\phi^A_i$ (left network)')
plt.ylabel('$\phi^{AB}_i$ (both networks)')
plt.plot([0,0], [-reach, reach],'-k', [-reach, reach], [0,0], '-k', [-reach, reach], [reach,-reach], '-k')
plt.axis([-reach, reach, -reach, reach])
#plt.axis('off')
plt.axis('equal')
plt.xticks([-reach/2,reach/2])
plt.yticks([-reach/2,reach/2])

plt.subplot(122)
psi_correction_h1 = calc_psi_correction(phiA, phiAB, wgt, 1)
plt.title('$h_j=1$')
C = plt.contourf(phiA, phiAB, psi_correction_h1, levels, origin='lower', cmap=cm.get_cmap(cmap, len(levels)-1))
plt.plot([0,0], [-reach, reach],'-k', [-reach, reach], [0,0], '-k', [-reach, reach], [reach,-reach], '-k')
plt.axis('off')
plt.axis('equal')
plt.xticks([-reach/2,reach/2])
plt.yticks([-reach/2,reach/2])

plt.savefig('correction.png', dpi=200)
```

## Trying to find an approximation that is easier for a hidden unit to calculate
The true correction is composed of the sum of two terms of form $\log(\sigma(\phi)/\sigma(\phi+W))$.

The first term is exactly $\log(\sigma(\phi)/\sigma(\phi+W))$, which goes from being $-W$ at $\phi=-\infty$ to  $0$ at $\phi=+\infty$. The second term is $\log(\sigma(\phi+\epsilon+W)/\sigma(\phi+\epsilon))$, which goes from being $W$ at $\phi=-\infty$ to  $0$ at $\phi=+\infty$.

Each of these is "roughly sigmoid", so my approximation is going to use a sigmoid in its place.
Focussing on the 1st term, the best sigmoid has it's "switching point" at $\phi = -W/2$.
It should also have a "gain" of $\beta = \frac{4(2\sigma(W/2)-1)}{W}$ if you want its slope at the switching point to match that of the true correction.

The 2nd term has switching point at $\phi^A = -\phi^B - W/2$, and so the whole approximation becomes...
$$
\tilde{C}_{ji} = W^A_{ji} \bigg[ \sigma(\beta_{ji} . (\phi^{A}_i - \hat{h}^A_j W^A_{ji})) \;\; - \;\; \sigma(\beta_{ji} . (\phi_i^{AB} - \hat{h}^A_j W^A_{ji})) \bigg]
$$

where $\hat{h} = h- \small{\frac{1}{2}}$.

Interestingly, the fact that it's got a multiplication by $W$ means our correction can be thought of as equivalently a correction to $v_i$, the activity of the visible unit.

*With this approximation, computation of $\psi^A_j$ still requires the $ji$ connection knowing the same things* , namely:
   *  $W^A_{ji}$, the current weight
   *  $v_i$, the current output state
   *  $h^A_j$, the activity on the hidden end
   * $\phi^A_i$
   * $\phi^{AB}_i$

So one has to ask what the point is, other than the (possibly!) simpler story?

```python
# testing the approximation
W = -10.0
alpha = 4*(2*sigmoid(W/2)-1)/W # this is the scaler that would match the slope at the 'midpoint'
phi = np.linspace(-30, 30, 1001)
plt.plot(phi, np.log(sigmoid(phi)/sigmoid(phi+W)),'k')
plt.plot(phi, -W*sigmoid(-alpha*(phi+W/2)),'b')
print(alpha)
```

# The learning algorithm
Stochastic ascent of the log likelihood.

We have that
$$
\log P^\star(h^A, h^B, v) = \log P^\star(h^A) + \log P^\star(h^B) + \log P^\star(v \mid h^A, h^B)
$$

The first term is:
$ \log P^\star(h^A) = \sum_i \log (1 + e^{\phi^A_i}) = -\sum_i \log (\sigma(\phi^A_i))$

Second is the same...

Third is
$\sum_i v_i\log \sigma(\phi_i^\text{AB})  \; + \; (1-v_i) \log \sigma(- \phi_i^\text{AB}) $

So now we differentiate it w.r.t. some particular weight $W_{ml}^A$ like this:

First term:
$\frac{\partial}{\partial W_{ml}^A} \log P^\star(h^A)  = \sigma(\phi^A_l) h_m^A$ is the first term, which is just the average of the usual RBM Hebbian change.

The second term: is zero.

Third term:

$\frac{\partial}{\partial W_{ml}^A} \log P^\star(v \mid h^A, h^B)   = (v_l-\sigma(\phi_l^\text{AB})) h_m^A$
which is "just the perceptron rool".

## so the learning algorithm is...

### WAKE PHASE

use our Gibbs chain to get samples from $h$ for each $v \in \mathcal{D}$, and do

$\Delta W_{ji}^A \;\;\; \propto \;\;\; \underbrace{\sigma(\phi^A_i) h_j^A}_\text{mean field Hebbian} \; + \; \underbrace{(v_i - \sigma(\phi_i^\text{AB})) h_j^A}_\text{Perceptron learning rule}$

and similarly for the other RBM. Note that the $\phi$'s are not the same in the two terms: the first only sums input from the A side, but the second sums from both hidden layers.

### SLEEP PHASE
use "regular" Gibbs sampling _in each model separately_ to sample independently from the two RBMs, and do

$\Delta W_{ji}^A \;\;\; \propto \;\;\; - \sigma( \phi^{A}_i) \; h_j^A$

and similarly for the other RBM.

### notice the free phase is free
In the FREE PHASE the expected $\psi$ is simply:
$$\bar{\psi}_j = \sum_i W_{ji} \sigma_{\phi}$$
Does that mean that the free phase in this model corresponds to what it would be if the two RBMs were not even connected?
Looks like it (although this is just an approximation, isn't it?).
BUT OF COURSE IT DOES! In the "unrolled" version of the net, the visibles are not clamped in the free phase and hence there's no explaining away going on between the two RBMs: they're independent RBMs doing their own thing, in the free phase. It's *only the wake phase that needs any update to the vanilla Gibbs* $\psi$.

Thought: maybe even the wake phase could be implemented via samples instead of having to calculate and propogate those sigmoid floats? THINK ABOUT THIS SOME MORE.

```python
import sympy as sp
x,y,z = sp.symbols('x y z')
sp.simplify(sp.log( (1+sp.exp(-x-y)) * (1+sp.exp(x+y+z)) / (1+sp.exp(-x-y-z)) / (1+sp.exp(x+y))) )
sp.simplify(sp.log( (1+sp.exp(x)) * (1+sp.exp(x+y+z)) / (1+sp.exp(x+y)) / (1+sp.exp(x+z)) ))
```

    log((exp(x) + 1)*(exp(x + y + z) + 1)/((exp(x + y) + 1)*(exp(x + z) + 1)))


Green's function
==============

Fundamental solution
-------------------------------

```python
from sympy import *
init_printing()
```

```python
x1, x2, xi1, xi2 = symbols('x_1 x_2 xi_1 xi_2')
E = -1/(2*pi) * log(sqrt((x1-xi1)**2 + (x2-xi2)**2))
E
```

**Task**: Check that $\nabla^2_\xi E = 0$ for $x \neq \xi$.

*Hint*: https://docs.sympy.org/latest/tutorial/calculus.html#derivatives

```python
diff(E,x,2)
```

Directional derivative
------------------------------

```python
n1, n2 = symbols('n_1 n_2')
```

**Task**: Compute the directional derivative $\frac{\partial E}{\partial n}$.

```python

```

**Task** (optional): Write a function which returns the directional derivative of an expression.

```python
def ddn(expr):
    pass
```

Reflection principle
----------------------------

For simple geometries Green's function can sometimes be found by reflecting the fundamental solution at the boundary and linearly combining the fundamental solution with its reflection.

**Task**: Based on $E$, find the solution for the half-space problem
\begin{align*}
    \nabla^2G(x,\xi) &= -\delta(x-\xi), & \xi\in\Omega \\
    G(x,\xi) &= 0, & \xi\in\partial\Omega \\
    \Omega &= \{\xi\in\mathbb{R}^2 : \xi_2 > 0\}
\end{align*}

*Hint*: https://docs.sympy.org/latest/tutorial/basic_operations.html#substitution

```python

```

**Task**: Based on $E$, find the solution for the half-space problem
\begin{align*}
    \nabla^2G(x,\xi) &= -\delta(x-\xi), & \xi\in\Omega \\
    \frac{\partial G(x,\xi)}{\partial n} &= 0, & \xi\in\partial\Omega \\
    \Omega &= \{\xi\in\mathbb{R}^2 : \xi_2 > 0\}
\end{align*}

```python

```


# Exploring the static force model in Seaman

```python
%load_ext autoreload
%autoreload 2
%matplotlib inline
```

```python
import sympy as sp
from sympy.plotting import plot as plot
from sympy.plotting import plot3d as plot3d
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

import seaman.helpers
import seaman_symbol as ss
import total_equations as equations
import total_lambda_functions as lambda_functions

sp.init_printing()
```

## Coordinate system
In the static force model forces and moments are referred to a righthanded
coordinate system with 4 degrees of freedom (surge, sway,roll, yaw) with origin in L/2, in the centre line of the ship and
vertically in the mean water line:

$\left[\frac{L}{2},0,t_m\right]$

* X : forces in ship heading direction
* Y : forces perpendicular to ship heading direction on the water plane.
**Note!** Y direction is not changing with roll.
* P : moment around X
* N : moment around Z

Ship motions in time domain simulations are however referred to CG.

## Nomenclature
The following symbols are used in this documentation:

```python
import seaman_symbols
from IPython.core.display import HTML
```

```python
symbols = [item for key,item in seaman_symbols.__dict__.items()]
```

```python
HTML(ss.create_html_table(symbols=symbols))
```

## Bis system
The bis system is used in Seaman to make physical quantities nondimensional. The denominators in the table below is used. The quantities are denoted with a '' sign when they are in bis units, otherwise they are in SI units.

```python
from bis_system import BisSystem
```

```python

HTML(BisSystem.html_table())
```

## Bis example
Suppose that we have a force $Y''$ that can be calculated from linear velocity $x''$ multiplied with some kind of nondimensional coefficient $C$:

```python
l = ss.Symbol('l')
m = ss.Symbol('m')
C = ss.Coefficient('C')

x = ss.Bis('x',denominator=BisSystem.linear_velocity)
Y = ss.Bis('Y',denominator=BisSystem.force)

eq = sp.Eq(Y.bis,C*x.bis**2)
```

```python
eq
```

Relation between $Y$ (SI force [N]) and $Y''$ (bis force):

```python
Y.bis_eq
```

Relation between $x$ (SI linear velocity [m/s]) and $x''$ (bis linear velocity):

```python
x.bis_eq
```

**Note!** Linear velocity in bis system is the same thing as Froude Number.

The equation can be written in SI units by substitution of the bis relations above:

```python
eq3 = ss.expand_bis(equation = eq)
eq3
```

## Total forces
The total forces on the ship can be expressed as described below.
The underlying equations are explained in:
### Hull
* [Surge hull equations](05_seaman_surge_hull_equation.ipynb)
* [Sway hull equations](02_seaman_sway_hull_equation.ipynb)
* [Yaw hull equations](03_seaman_yaw_hull_equation.ipynb)

### Rudder:
* [Rudder equations](04_seaman_rudder_equation.ipynb)

## Surge

```python
equations.surge_equation
```

Expanding the surge equation (using equations for hull and rudder) and converting to SI units:

```python
equations.surge_equation_expanded_SI
```

## Sway

```python
equations.sway_equation
```

```python
equations.sway_equation_expanded_SI
```

## Yaw

```python
equations.yaw_equation
```

```python
equations.yaw_equation_expanded_SI
```

## Roll

```python
equations.roll_equation
```

```python
equations.roll_equation_expanded_SI
```

```python
import surge_hull_equations as surge_hull_equations
import sway_hull_equations as sway_hull_equations
import yaw_hull_equations as yaw_hull_equations
import roll_hull_equations as roll_hull_equations
import rudder_equations as rudder_equations
```

```python
from seaman_symbols import *
```

## Real seaman++
Run real seaman in C++ to verify that the documented model is correct.

```python
import generate_input
shipdict = seaman.ShipDict.load('../../tests/test_ship.ship')
import run_real_seaman

```

### Surge

```python
%connect_info
```

```python
df = pd.DataFrame()
df['v_w'] = np.linspace(-3,3,20)
df['delta'] = 0
df['T'] = 0
df['rho'] = 1025.0
df['g'] = 9.81
df['u_w'] = 5.0
df['r_w'] = 0.0
df['X_res'] = -np.interp(df['u_w'],shipdict.res_data['vres'],shipdict.res_data['res'])

result_comparison = run_real_seaman.compare_with_seaman(lambda_function=lambda_functions.total_surge_function,
                                                        shipdict = shipdict,
                                                        df = df,
                                                        label = 'fx',
                                                        seaman_function=run_real_seaman.calculate_static_ship
                                                       )

fig,ax = plt.subplots()
result_comparison.plot(x = 'v_w',y = ['fx','fx_seaman'],ax = ax)
ax.set_title('Drift angle variation');
```

Real seaman has a maximum effective rudder angle 0.61 rad for the rudder drag, which is why seaman gives different result for really large drift angles or yaw rates:

```python
df = pd.DataFrame()
df['delta'] = np.deg2rad(np.linspace(-45,45,50))
df['r_w'] = 0
df['T'] = 0
df['rho'] = 1025.0
df['g'] = 9.81
df['u_w'] = 5.0
df['v_w'] = 0.0
df['X_res'] = -np.interp(df['u_w'],shipdict.res_data['vres'],shipdict.res_data['res'])

result_comparison = run_real_seaman.compare_with_seaman(lambda_function=lambda_functions.total_surge_function,
                                                        shipdict = shipdict,
                                                        df = df,
                                                        label = 'fx',
                                                        seaman_function=run_real_seaman.calculate_static_ship
                                                       )

fig,ax = plt.subplots()
result_comparison.plot(x = 'delta',y = ['fx','fx_seaman'],ax = ax)
ax.set_title('Rudder angle variation');
```

```python
df = pd.DataFrame()
df['r_w'] = np.linspace(-0.05,0.05,20)
df['delta'] = 0
df['T'] = 0
df['rho'] = 1025.0
df['g'] = 9.81
df['u_w'] = 5.0
df['v_w'] = 0.0
df['X_res'] = -np.interp(df['u_w'],shipdict.res_data['vres'],shipdict.res_data['res'])

shipdict2 = shipdict.copy()
#shipdict2.design_particulars['lcg'] = 0.0

df_input = generate_input.add_shipdict_inputs(lambda_function=lambda_functions.total_surge_function,
                                                  shipdict=shipdict2,
                                                  df=df)

result_comparison = run_real_seaman.compare_with_seaman(lambda_function=lambda_functions.total_surge_function,
                                                        shipdict = shipdict2,
                                                        df = df,
                                                        label = 'fx',
                                                        seaman_function=run_real_seaman.calculate_static_ship
                                                       )

fig,ax = plt.subplots()
result_comparison.plot(x = 'r_w',y = ['fx','fx_seaman'],ax = ax)
ax.set_title('Yaw rate variation');
```

```python
result_comparison['fx']
```

```python
df_input.head()
```

### Sway

```python
df = pd.DataFrame()
df['v_w'] = np.linspace(-6,6,20)
df['delta'] = 0
df['T'] = 0
df['rho'] = 1025.0
df['g'] = 9.81
df['u_w'] = 5.0
df['r_w'] = 0.0

result_comparison = run_real_seaman.compare_with_seaman(lambda_function=lambda_functions.total_sway_function,
                                                        shipdict = shipdict,
                                                        df = df,
                                                        label = 'fy',
                                                        seaman_function=run_real_seaman.calculate_static_ship
                                                       )

fig,ax = plt.subplots()
result_comparison.plot(x = 'v_w',y = ['fy','fy_seaman'],ax = ax)
ax.set_title('Drift angle variation');
```

```python
%connect_info
```

```python
df = pd.DataFrame()
df['r_w'] = np.linspace(-0.1,0.1,20)
df['delta'] = 0
df['T'] = 0
df['rho'] = 1025.0
df['g'] = 9.81
df['u_w'] = 5.0
df['v_w'] = 0.0

shipdict2 = shipdict.copy()
#shipdict2.design_particulars['lcg'] = 0  # Something strange with lcg in Seaman?

result_comparison = run_real_seaman.compare_with_seaman(lambda_function=lambda_functions.total_sway_function,
                                                        shipdict = shipdict2,
                                                        df = df,
                                                        label = 'fy',
                                                        seaman_function=run_real_seaman.calculate_static_ship
                                                       )

fig,ax = plt.subplots()
result_comparison.plot(x = 'r_w',y = ['fy','fy_seaman'],ax = ax)
ax.set_title('Yaw rate variation');

fig,ax = plt.subplots()
result_comparison.plot(x = 'r_w',y = ['fy_rudders_seaman'],ax = ax)
ax.set_title('Yaw rate variation Rudder');

```

```python
df = pd.DataFrame()
df['delta'] = np.deg2rad(np.linspace(-45,45,20))
df['r_w'] = 0
df['T'] = 0
df['rho'] = 1025.0
df['g'] = 9.81
df['u_w'] = 5.0
df['v_w'] = 0.0

shipdict2 = shipdict.copy()
#shipdict2.rudder_coeff_data['s'] = 0

result_comparison = run_real_seaman.compare_with_seaman(lambda_function=lambda_functions.total_sway_function,
                                                        shipdict = shipdict2,
                                                        df = df,
                                                        label = 'fy',
                                                        seaman_function=run_real_seaman.calculate_static_ship
                                                       )

fig,ax = plt.subplots()
result_comparison.plot(x = 'delta',y = ['fy','fy_seaman'],ax = ax)
ax.set_title('Rudder angle variation');
```

### Yaw

```python
df = pd.DataFrame()
df['v_w'] = np.linspace(-2,2,20)
df['delta'] = 0
df['T'] = 0
df['rho'] = 1025.0
df['g'] = 9.81
df['u_w'] = 5.0
df['r_w'] = 0.0
df['Cd_lever'] = 0.0
shipdict2 = shipdict.copy()
#shipdict2.rudder_coeff_data['s'] = 0

result_comparison = run_real_seaman.compare_with_seaman(lambda_function=lambda_functions.total_yaw_function,
                                                        shipdict = shipdict2,
                                                        df = df,
                                                        label = 'mz',
                                                        seaman_function=run_real_seaman.calculate_static_ship
                                                       )

fig,ax = plt.subplots()
result_comparison.plot(x = 'v_w',y = ['mz','mz_seaman'],ax = ax)
ax.set_title('Drift angle variation');
ax.grid()
```

```python
df = pd.DataFrame()
df['delta'] = np.deg2rad(np.linspace(-20,20,20))
df['v_w'] = 0
df['T'] = 0
df['rho'] = 1025.0
df['g'] = 9.81
df['u_w'] = 5.0
df['r_w'] = 0.0
df['Cd_lever'] = 0

result_comparison = run_real_seaman.compare_with_seaman(lambda_function=lambda_functions.total_yaw_function,
                                                        shipdict = shipdict,
                                                        df = df,
                                                        label = 'mz',
                                                        seaman_function=run_real_seaman.calculate_static_ship
                                                       )

fig,ax = plt.subplots()
result_comparison.plot(x = 'delta',y = ['mz','mz_seaman'],ax = ax)
ax.set_title('Rudder angle variation');
```

### Roll

```python
%connect_info
```

```python
df = pd.DataFrame()
df['delta'] = np.deg2rad(np.linspace(-5,5,20))
df['v_w'] = 0
df['T'] = 0
df['rho'] = 1025.0
df['g'] = 9.81
df['u_w'] = 5.0
df['r_w'] = 0.0
df['p'] = 0
df['Cd_lever'] = 0

result_comparison = run_real_seaman.compare_with_seaman(lambda_function=lambda_functions.total_roll_function,
                                                        shipdict = shipdict,
                                                        df = df,
                                                        label = 'mx',
                                                        seaman_function=run_real_seaman.calculate_static_ship
                                                       )

fig,ax = plt.subplots()
result_comparison.plot(x = 'delta',y = ['mx','mx_seaman'],ax = ax)
ax.set_title('Rudder angle variation');
```

```python
df = pd.DataFrame()
df['beta'] = np.deg2rad(np.linspace(-20,20,20))
df['V'] = V = 5.0
df['u_w'] = V*np.cos(df['beta'])
df['v_w'] = -V*np.sin(df['beta'])

df['delta'] = 0
df['T'] = 0
df['rho'] = 1025.0
df['g'] = 9.81
df['r_w'] = 0.0
df['p'] = 0
df['Cd_lever'] = 0

shipdict2 = shipdict.copy()
#shipdict2.rudder_coeff_data['s'] = 0
#shipdict2.non_lin_coeff_data['cd'] = 3

result_comparison = run_real_seaman.compare_with_seaman(lambda_function=lambda_functions.total_roll_function,
                                                        shipdict = shipdict2,
                                                        df = df,
                                                        label = 'mx',
                                                        seaman_function=run_real_seaman.calculate_static_ship
                                                       )

fig,ax = plt.subplots()
result_comparison.plot(x = 'v_w',y = ['mx','mx_seaman'],ax = ax)
ax.set_title('Drift angle variation');
```

```python

```


# Computer Algebra System (CAS)

SymPy is a Python library for symbolic mathematics.
- [tutorial](https://docs.sympy.org/latest/tutorial/index.html)
- [documentation](https://docs.sympy.org/latest/tutorial/index.html)

Symbolic calculations begin by defining symbolic variables:

```python
import numpy as np
from sympy import *
x,y=symbols('x y')
```

Evaluate expressions containing symbolic variables:

```python
x+y
```

$\displaystyle x + y$

```python
(x+y)**2
```

$\displaystyle \left(x + y\right)^{2}$

```python
expand((x + y)**2)
```

$\displaystyle x^{2} + 2 x y + y^{2}$

Substitute variables in expressions: Substitution replaces all instances of something in an expression with something else. It is done using the `subs` method.

```python
z = x**2 + y**2
```

```python
z
```

$\displaystyle x^{2} + y^{2}$

replace y with a + b in the expression for z

```python
a,b=symbols('a b')
z.subs(y,a+b)
```

$\displaystyle x^{2} + \left(a + b\right)^{2}$

replace x with a + b in the expression for z

```python
z.subs(x,a+b)
```

$\displaystyle y^{2} + \left(a + b\right)^{2}$

Define and evaluate symbolic functions

```python
f = Function('f')
f(x)+1
```

$\displaystyle f{\left(x \right)} + 1$

```python
f = Function('f')(x,y)
f+1
```

$\displaystyle f{\left(x,y \right)} + 1$

To treat numbers symbolically rather than numerically, sym must be inserted:

```python
sin(np.pi)
```

$\displaystyle 1.22464679914735 \cdot 10^{-16}$

```python
sin(pi)
```

$\displaystyle 0$

Sometimes simplify needs to be called to simplify an expression:

```python
g = (1 + sqrt(Symbol('5')))/2  #golden ratio
g
```

$\displaystyle \frac{\sqrt{5}}{2} + \frac{1}{2}$

```python
g**2-g-1 #should vanish
```

$\displaystyle - \frac{\sqrt{5}}{2} + \left(\frac{\sqrt{5}}{2} + \frac{1}{2}\right)^{2} - \frac{3}{2}$

```python
simplify(g**2-g-1)
```

$\displaystyle \frac{5}{4} - \frac{5}{4}$

Conversion between symbolic and numeric quantities:

```python
nsimplify(0.25) #symbolic
```

$\displaystyle \frac{1}{4}$

```python
1/3  #numeric
```

    0.3333333333333333

```python
S(1)/3         #symbolic
```

$\displaystyle \frac{1}{3}$

```python
Rational(1,3)  #symbolic
```

$\displaystyle \frac{1}{3}$

```python
Rational(1,3).evalf()
```

$\displaystyle 0.333333333333333$

**Matrices**: A matrix can contain any symbolic expression.

```python
Matrix([[1,2],[3,4]])
```

$\displaystyle \left[\begin{matrix}1 & 2\\3 & 4\end{matrix}\right]$

```python
Matrix([1,2,3])
```

$\displaystyle \left[\begin{matrix}1\\2\\3\end{matrix}\right]$

```python
Matrix([1,2,3]).T
```

$\displaystyle \left[\begin{matrix}1 & 2 & 3\end{matrix}\right]$

```python
Matrix([[1,2,3]])
```

$\displaystyle \left[\begin{matrix}1 & 2 & 3\end{matrix}\right]$

```python
a,b,c=symbols('a b c')
Matrix([a,b,c])
```

$\displaystyle \left[\begin{matrix}a\\b\\c\end{matrix}\right]$

```python
Matrix([[1,2],[3,4]])@Matrix([a,b]) # Python 3.5+
```

$\displaystyle \left[\begin{matrix}a + 2 b\\3 a + 4 b\end{matrix}\right]$

```python
Matrix([[1,2],[3,4]])*Matrix([a,b])
```

$\displaystyle \left[\begin{matrix}a + 2 b\\3 a + 4 b\end{matrix}\right]$

calculating the jacobian

```python
x1,x2,x3 = symbols('x1 x2 x3')
f1,f2,f3 = symbols('f1 f2 f3', cls=Function)
M = Matrix([f1(x1)+f1(x2)+f1(x3),f2(x1)+f2(x2)+f2(x3),f3(x1)+f3(x2)+f3(x3)])
M.jacobian([x1,x2,x3])
```

$\displaystyle \left[\begin{matrix}\frac{d}{d x_{1}} \operatorname{f_{1}}{\left(x_{1} \right)} & \frac{d}{d x_{2}} \operatorname{f_{1}}{\left(x_{2} \right)} & \frac{d}{d x_{3}} \operatorname{f_{1}}{\left(x_{3} \right)}\\\frac{d}{d x_{1}} \operatorname{f_{2}}{\left(x_{1} \right)} & \frac{d}{d x_{2}} \operatorname{f_{2}}{\left(x_{2} \right)} & \frac{d}{d x_{3}} \operatorname{f_{2}}{\left(x_{3} \right)}\\\frac{d}{d x_{1}} \operatorname{f_{3}}{\left(x_{1} \right)} & \frac{d}{d x_{2}} \operatorname{f_{3}}{\left(x_{2} \right)} & \frac{d}{d x_{3}} \operatorname{f_{3}}{\left(x_{3} \right)}\end{matrix}\right]$

**Matrix Symbols**: operation on matrices of symbolic dimension $n\times m$

```python
n, m = symbols('n m', integer=True)
M = MatrixSymbol('M', n, m)
b = MatrixSymbol('b', m, 1)
M@b
```

$\displaystyle M b$

```python
(M@b).shape
```

    (n, 1)

```python
Transpose(M@b)
```

$\displaystyle \left(M b\right)^{T}$

```python
Transpose(M@b).doit()
#
#
# ## Algebraic equations
#
# solves the equation for y
```

$\displaystyle b^{T} M^{T}$

```python
x = Symbol('x')
y = Symbol('y')
```

```python

```

```python
simplify(1/3)
```

$\displaystyle 0.333333333333333$

```python
solve(x**2-y, x)
```

    [-sqrt(y), sqrt(y)]

### Linear systems

Example: $x +2y = u$, $4x + 5y = v$

```python
x, y, u, v = symbols('x y u v')
system = Matrix(( (1, 2, u), (4, 5, v)))
solve_linear_system(system, x, y)
```

    {x: -5*u/3 + 2*v/3, y: 4*u/3 - v/3}

```python
x, y, u, v = symbols('x y u v')
solve((x+2*y-u,4*x+5*y-v), x, y)
```

    {x: -5*u/3 + 2*v/3, y: 4*u/3 - v/3}

```python
x_1, x_2, u, v = symbols('x_1 x_2 u v')
A = Matrix(((1, 2), (4, 5)))
b = Matrix(((u), (v)))
x = Matrix(((x_1), (x_2)))
solve(A*x-b, x_1, x_2)
```

    {x_1: -5*u/3 + 2*v/3, x_2: 4*u/3 - v/3}

### Nonlinear systems

Example: $x^2 + 3y = 0$, $y^2+2x=0$

```python
x, y = symbols('x y')
solve([x**2 + 3*y, y**2 + 2*x])
```

    [{x: 0, y: 0},
     {x: -(12**(1/3)/2 + 2**(2/3)*3**(5/6)*I/2)**2/2,
      y: 12**(1/3)/2 + 2**(2/3)*3**(5/6)*I/2},
     {x: -(-3*2**(2/3)*3**(1/3)/4 - 12**(1/3)/4)**2/2,
      y: -3*2**(2/3)*3**(1/3)/4 - 12**(1/3)/4},
     {x: -(-12**(1/3)/4 + 3*2**(2/3)*3**(1/3)/4 - 2**(2/3)*3**(5/6)*I/2)**2/2,
      y: -12**(1/3)/4 + 3*2**(2/3)*3**(1/3)/4 - 2**(2/3)*3**(5/6)*I/2}]

### Exercise:

Find all matrices of the form

$$
\left( \begin{array}{ccc} a&b&c\\c&a&b\\b&c&a \end{array} \right)
$$

with $a,b,c\in\mathbf{R}$ whose square is the unit matrix.

```python
a, b, c = symbols('a b c')
M = Matrix(((a, b, c), (c, a, b), (b, c, a)))
solve(M**2-eye(3,3))
```

    [{a: -1, b: 0, c: 0},
     {a: -1/3, b: 2/3, c: 2/3},
     {a: -1/3, b: -1/3 - sqrt(3)*I/3, c: -1/3 + sqrt(3)*I/3},
     {a: -1/3, b: -1/3 + sqrt(3)*I/3, c: -1/3 - sqrt(3)*I/3},
     {a: 1/3, b: -2/3, c: -2/3},
     {a: 1/3, b: 1/3 - sqrt(3)*I/3, c: 1/3 + sqrt(3)*I/3},
     {a: 1/3, b: 1/3 + sqrt(3)*I/3, c: 1/3 - sqrt(3)*I/3},
     {a: 1, b: 0, c: 0}]

### Assumptions on symbolic variables

```python
x = symbols('x')
solve(x**3 + 1)
```

    [-1, 1/2 - sqrt(3)*I/2, 1/2 + sqrt(3)*I/2]

```python
x = symbols('x', real=True)
solve(x**3 + 1)
```

    [-1]

```python
x = symbols('x', positive=True)
solve(x**3 + 1)
```

    []

```python
x.is_real
```

    True

```python
x.is_positive
```

    True

reset assumptions

```python
x = symbols('x')
solve(x**3 + 1)
```

    [-1, 1/2 - sqrt(3)*I/2, 1/2 + sqrt(3)*I/2]

## Symbolic differentiation and integration

Differentiate a symbolic function:

```python
f = Function('f')
Derivative(f(x),x,x)
```

$\displaystyle \frac{d^{2}}{d x^{2}} f{\left(x \right)}$

```python
diff(x**2*y,x)
```

$\displaystyle 2 x y$

```python
diff(x**2*y,y)
```

$\displaystyle x^{2}$

Second order derivative

```python
diff(x**2*y,x,x)
```

$\displaystyle 2 y$

```python
diff(x**2*y,x,2)
```

$\displaystyle 2 y$

### Exercise:

Determine

$$
\lim_{x\to 0} \frac{2\tan(x) - \tan(2x)}{x(1 - \cos(3x))}
$$

Here we use derivatives to determine limits. This is in particular useful, if we obtain limits that end up in inteterminate or undefined forms such as $\infty / \infty$, $-\infty / \infty$, or $0/0$. The latter is the case here.

#### L'Hopital's Rule:

If $\lim_{x\to 0} f(x)=0$ and $\lim_{x\to 0} g(x)=0$ and $\lim_{x\to 0} \frac{f'(x)}{g'(x)}=L$ then we can say that the limit $\lim_{x\to 0} \frac{f(x)}{g(x)}=L$.

```python
x = symbols('x')
f=(2*tan(x) - tan(2*x))/(x*(1 - cos(3*x)))
f
```

$\displaystyle \frac{2 \tan{\left(x \right)} - \tan{\left(2 x \right)}}{x \left(1 - \cos{\left(3 x \right)}\right)}$

```python
z = 2*tan(x) - tan(2*x)  #numerator
z.subs(x, 0)
```

$\displaystyle 0$

```python
z1 = diff(z)
z1.subs(x,0)
```

$\displaystyle 0$

```python
z2 = diff(z1)
z2.subs(x,0)
```

$\displaystyle 0$

```python
z3 = diff(z2)
z3.subs(x,0)
```

$\displaystyle -12$

```python
n = x*(1 - cos(3*x))  #denominator
n.subs(x, 0)
```

$\displaystyle 0$

```python
n1 = diff(n)
n1.subs(x, 0)
```

$\displaystyle 0$

```python
n2 = diff(n1)
n2.subs(x, 0)
```

$\displaystyle 0$

```python
n3 = diff(n2)
n3.subs(x, 0)
```

$\displaystyle 27$

Applying [L'Hopital's rule](https://www.khanacademy.org/math/ap-calculus-ab/ab-diff-contextual-applications-new/ab-4-7/v/introduction-to-l-hopital-s-rule) three times yields the desired limit:

```python
(z1/n1).subs(x, 0)
```

$\displaystyle \text{NaN}$

```python
(z2/n2).subs(x, 0)
```

$\displaystyle \text{NaN}$

```python
(z3/n3).subs(x, 0)
```

$\displaystyle - \frac{4}{9}$

## Taylor expansion

A [Taylor series](https://mathworld.wolfram.com/TaylorSeries.html) is a series expansion of a function about a point. A one-dimensional Taylor series is an expansion of a real function $f(x)$ about a point $x=a$ is given by

$$f(x)=f(a)+f'(a)(x-a)+\frac{f''(a)}{2!}(x-a)^2+\frac{f'''(a)}{3!}(x-a)^3+\dots+\frac{f^{(n)}(a)}{n!}(x-a)^n+\dots$$

The Taylor series of an unknown function $f(x)$ about a point $a$ up to order $n$ may be found using `series`.

```python
x, h = symbols("x,h")
f = Function("f")
f(x).series(x, x0=h, n=3)
```

$\displaystyle f{\left(h \right)} + \left(- h + x\right) \left. \frac{d}{d \xi_{1}} f{\left(\xi_{1} \right)} \right|_{\substack{ \xi_{1}=h }} + \frac{\left(- h + x\right)^{2} \left. \frac{d^{2}}{d \xi_{1}^{2}} f{\left(\xi_{1} \right)} \right|_{\substack{ \xi_{1}=h }}}{2} + O\left(\left(- h + x\right)^{3}; x\rightarrow h\right)$

The Taylor series of the function $\sin(x)$ about a point $x_0=0$ up to order $n=6$:

```python
x = symbols("x")
sin(x).series(x, x0=0, n=6)
```

$\displaystyle x - \frac{x^{3}}{6} + \frac{x^{5}}{120} + O\left(x^{6}\right)$

The Taylor series of the function $\sin(x)$ about a point $x_0=0$ up to order $n=10$:

```python
x = symbols("x")
sin(x).series(x, x0=0, n=10)
```

$\displaystyle x - \frac{x^{3}}{6} + \frac{x^{5}}{120} - \frac{x^{7}}{5040} + \frac{x^{9}}{362880} + O\left(x^{10}\right)$

The Taylor series of the function $\log(x)$ about a point $x_0=1$ up to order $n=6$:

```python
x = symbols("x")
log(x).series(x, x0=1, n=6)
```

$\displaystyle -1 - \frac{\left(x - 1\right)^{2}}{2} + \frac{\left(x - 1\right)^{3}}{3} - \frac{\left(x - 1\right)^{4}}{4} + \frac{\left(x - 1\right)^{5}}{5} + x + O\left(\left(x - 1\right)^{6}; x\rightarrow 1\right)$

If you want a finite (forward) [difference approximation](https://docs.sympy.org/latest/special_topics/finite_diff_derivatives.html) to derivatives, you can for example write

```python
FW = f(x+h).series(x+h, x0=h, n=3)   #TODO
FW.subs(x+h,0)
```

$\displaystyle f{\left(2 h + x \right)}$

## Integration

SymPy implements methods to calculate definite and indefinite [integrals](https://docs.sympy.org/latest/modules/integrals/integrals.html) of expressions.
- `integrate(f,x)` returns the indefinite integral
- `integrate(f,(x,a))` returns the indefinite integral, with result given with $a$ replacing $x$
- `integrate(f,(x,a,b))` returns the definite integral

```python
x = symbols("x")
f = Function("f")
f(x).integrate(x)
```

$\displaystyle \int f{\left(x \right)}\, dx$

```python
x, a = symbols("x,a")
f = Function("f")
f(x).integrate((x,a))
```

$\displaystyle \int\limits^{a} f{\left(x \right)}\, dx$

```python
x, a, b = symbols("x,a,b")
f = Function("f")
f(x).integrate((x,a,b))
```

$\displaystyle \int\limits_{a}^{b} f{\left(x \right)}\, dx$

We start with a quadratic polynomical and take a derivative:

```python
x, a, b, c = symbols("x,a,b,c")
p = a*x**2 + b*x + c
dpdx = diff(p,x)
dpdx
```

$\displaystyle 2 a x + b$

Now we integrate to try and retrieve the function we started with:

```python
p1 = integrate(dpdx,x)
p1
```

$\displaystyle a x^{2} + b x$

**Note**: the integration constant is zero by default and thus needs to be added in by hand

```python
p1 = p1 + c
p1
```

$\displaystyle a x^{2} + b x + c$

#### Indefinite, definite and improper integrals

$$f(x)=\int \frac{1}{x^2}$$

```python
integrate(1/x**2,x)
```

$\displaystyle - \frac{1}{x}$

```python
plot(1/x**2, (x, 1, 2))
```

```python
integrate(1/x**2,(x,1,2))
```

$\displaystyle \frac{1}{2}$

```python
integrate(1/x**2,(x, 1, oo))
```

$\displaystyle 1$

$$\int_0^\inf \frac{1}{1 + x^4}$$

```python
plot(1/(1 + x**4), (x, 0, 10))
```

```python
integrate(1/(1 + x**4), (x, 0, oo))
```

$\displaystyle \frac{\sqrt{2} \pi}{4}$

Of course not all integrals can be computed symbolically:

```python
integrate(sin(sinh(x)), x)
```

$\displaystyle \int \sin{\left(\sinh{\left(x \right)} \right)}\, dx$

### Compute with Distributions
#### Heaviside Distribution

Heaviside function $\theta(x)$ has the following properties:
1. $\frac{d}{dx}\theta(x) = \delta(x)$
2. $\theta(x) = \begin{cases} 1 & x \geq 0\\ 0 & x < 0 \end{cases}$
3. $\frac{d}{dx}\max(x,0)=\theta(x)$

```python
Heaviside(9)
```

$\displaystyle 1$

```python
Heaviside(-9)
```

$\displaystyle 0$

There is some controversy about the behavior at $x=0$, Mathematica defines $\theta(0)=1$, but Maple uses $\theta(0)=\text{undefined}$. Different application areas may have specific conventions. For example, in control theory, it is common practice to assume $\theta(0)=0$ to match the Laplace transform of a DiracDelta distribution.

```python
Heaviside(0,)
```

$\displaystyle \theta\left(0\right)$

To specify the value of Heaviside at $x=0$, a second argument can be given. Omit this 2nd argument or pass None to recover the default behavior.

```python
Heaviside(0, S.Half)
```

$\displaystyle \frac{1}{2}$

```python
x = symbols("x ")
y = Heaviside(x)
plot(y, (x, -1, 1))
```

#### Dirac Distribution

DiracDelta function $\delta(x)$ has the following properties:

1. $\frac{d}{dx}\theta(x)=\delta(x)$
2. $\int_{a-\epsilon}^{a+\epsilon}\delta(x-a)f(x)=f(a)$
3. $\dots$

```python
dydx = diff(y)
dydx
```

$\displaystyle \delta\left(x\right)$

```python
integrate(dydx,(x,-.1,.1))
```

$\displaystyle 1$

```python
integrate(dydx,(x,.1,.2))
```

$\displaystyle 0$

```python
x = symbols("x ")
y = DiracDelta(x)
plot(y, (x, -1, 1))
```

```python
DiracDelta(0)
```

$\displaystyle \delta\left(0\right)$

## Ordinary differential equations (ODEs)

Example: $y''+4y=e^x$

```python
x = symbols("x ")
f = Function('f')
dsolve(Derivative(f(x), x, x) + 4*f(x) - exp(x), f(x))
```

$\displaystyle f{\left(x \right)} = C_{1} \sin{\left(2 x \right)} + C_{2} \cos{\left(2 x \right)} + \frac{e^{x}}{5}$

We can also add initial conditions, e.g. $y(0)=1$, $y'(0)=0$:

```python
dsolve(Derivative(f(x), x, x) + 4*f(x) - exp(x), f(x), ics={f(0): 1, f(x).diff(x).subs(x, 0): 0})
```

$\displaystyle f{\left(x \right)} = \frac{e^{x}}{5} - \frac{\sin{\left(2 x \right)}}{10} + \frac{4 \cos{\left(2 x \right)}}{5}$

**Exercise**: Solve the ODE of [Bernoulli type](https://en.wikipedia.org/wiki/Bernoulli_differential_equation) $y^3 - x^2 + x y^2 y' = 0$ with initial condition $y(1)=1$.

```python
x = Symbol('x')
f = Function('f')
dsolve(f(x)**4-x**2+x*f(x)**2*Derivative(f(x), x), f(x), ics={f(1): 1})
```

$\displaystyle f{\left(x \right)} = 1 + \left(x - 1\right)^{2} - \frac{5 \left(x - 1\right)^{3}}{3} + \frac{23 \left(x - 1\right)^{4}}{12} - \frac{26 \left(x - 1\right)^{5}}{15} + O\left(x^{6}\right)$

### Systems of ODEs

Example: $f'=3f + 4g$, $g'=-4f+3g$

```python
x = Symbol('x')
f = Function('f')
g = Function('g')
eq = (Eq(Derivative(f(x),x),3*f(x)+4*g(x)),Eq(Derivative(g(x),x),-4*f(x)+3*g(x)))
dsolve(eq)
```

    [Eq(f(x), C1*exp(3*x)*sin(4*x) + C2*exp(3*x)*cos(4*x)),
     Eq(g(x), C1*exp(3*x)*cos(4*x) - C2*exp(3*x)*sin(4*x))]

Again we can add initial conditions, e.g. $f(0)=0$, $g(0)=1$:

```python
S = dsolve(eq, ics={f(0): 0, g(0): 1})
```

```python
S[0].rhs
```

$\displaystyle e^{3 x} \sin{\left(4 x \right)}$

```python
S[1].rhs
```

$\displaystyle e^{3 x} \cos{\left(4 x \right)}$

## Code generation

SymPy allows one to [generate directly compilable and executable code](https://docs.sympy.org/latest/modules/codegen.html) in a variety of different programming languages from SymPy expressions. In addition, there are functions that generate Python importable objects that can evaluate SymPy expressions very efficiently.

- [tutorial: https://www.sympy.org/scipy-2017-codegen-tutorial/](https://www.sympy.org/scipy-2017-codegen-tutorial/)

### Code Printer

Code printer convert SymPy expressions into the target language. A list of supported languages is [here](https://docs.sympy.org/latest/modules/printing.html).

```python
pretty_print(S[1].rhs)
```

     3⋅x
    ℯ   ⋅cos(4⋅x)

```python
pprint(S[1].rhs)
```

     3⋅x
    ℯ   ⋅cos(4⋅x)

```python
print_latex(S[0])
```

    f{\left(x \right)} = e^{3 x} \sin{\left(4 x \right)}

```python
latex(S[0])
```

    'f{\\left(x \\right)} = e^{3 x} \\sin{\\left(4 x \\right)}'

```python
print_ccode(S[1].rhs)
```

    exp(3*x)*cos(4*x)

```python
ccode(S[1].rhs)
```

    'exp(3*x)*cos(4*x)'

```python
mathematica_code(S[1].rhs)
```

    'Exp[3*x]*Cos[4*x]'

```python
jscode(S[1].rhs)
```

    'Math.exp(3*x)*Math.cos(4*x)'

```python
fcode(S[1].rhs)
```

    '      exp(3.0d0*x)*cos(4.0d0*x)'

```python
julia_code(S[1].rhs)
```

    'exp(3*x).*cos(4*x)'

```python
octave_code(S[1].rhs)
```

    'exp(3*x).*cos(4*x)'

```python
rust_code(S[1].rhs)
```

    '(3*x).exp()*(4*x).cos()'

```python
pycode(S[1].rhs)
```

    'math.exp(3*x)*math.cos(4*x)'

```python
print_python(S[1].rhs)
```

    x = Symbol('x')
    e = exp(3*x)*cos(4*x)

**Exercise**: Plotting SymPy functions with JavaScript

In this call we load the [Chart.js library](https://www.chartjs.org):

%%javascript
# # require.config({
# #     paths: {
# #         'chartjs': '//cdnjs.cloudflare.com/ajax/libs/Chart.js/2.6.0/Chart'
# #     }
# # });

Here is some JavaScript to do the plotting. The code will take two mathematical expressions in JavaScript and plot the functions:

```python
js_template = """\
require(['chartjs'], function(chartjs){{
function f(x) {{
    return {top_function}
}};
function g(x) {{
    return {bottom_function}
}};
function linspace(a,b,n) {{
    // From: https://gist.github.com/joates/6584908
    if(typeof n === "undefined") n = Math.max(Math.round(b-a)+1,1);
    if(n<2) {{ return n===1?[a]:[]; }}
    var i,ret = Array(n);
    n--;
    for(i=n;i>=0;i--) {{ ret[i] = (i*b+(n-i)*a)/n; }}
    return ret;
}}
var ctx = document.getElementById("{chart_id}");
var data = {{
    labels: linspace(-7.5, 7.5, 500),
    datasets: [{{
        label: "top",
        function: f,
        borderColor: "rgba(75, 192, 192, 1)",
        data: [],
        fill: false,
        lineTension: 0,
    }},
    {{
        label: "bottom",
        function: g,
        borderColor: "rgba(153, 102, 255, 1)",
        data: [],
        fill: false,
        lineTension: 0,
    }}]
}};
chartjs.Chart.pluginService.register({{
    beforeInit: function(chart) {{
        var data = chart.config.data;
        for (var i = 0; i < data.datasets.length; i++) {{
            for (var j = 0; j < data.labels.length; j++) {{
                var fct = data.datasets[i].function,
                    x = data.labels[j],
                    y = fct(x);
                data.datasets[i].data.push(y);
            }}
        }}
    }}
}});
var myBarChart = new chartjs.Chart(ctx, {{
    type: 'line',
    data: data,
    options: {{
        scales: {{
            yAxes: [{{
                ticks: {{
                    beginAtZero:true
                }}
            }}]
        }}
    }}
}});
}});
element.append("<canvas id='{chart_id}' width='400'></canvas>");\
"""
```

```python
print(js_template.format(top_function='*** TODO ***',
                         bottom_function='*** TODO ***',
                         chart_id='*** TODO ***'))
```

    require(['chartjs'], function(chartjs){
    function f(x) {
        return *** TODO ***
    };
    function g(x) {
        return *** TODO ***
    };
    function linspace(a,b,n) {
        // From: https://gist.github.com/joates/6584908
        if(typeof n === "undefined") n = Math.max(Math.round(b-a)+1,1);
        if(n<2) { return n===1?[a]:[]; }
        var i,ret = Array(n);
        n--;
        for(i=n;i>=0;i--) { ret[i] = (i*b+(n-i)*a)/n; }
        return ret;
    }
    var ctx = document.getElementById("*** TODO ***");
    var data = {
        labels: linspace(-7.5, 7.5, 500),
        datasets: [{
            label: "top",
            function: f,
            borderColor: "rgba(75, 192, 192, 1)",
            data: [],
            fill: false,
            lineTension: 0,
        },
        {
            label: "bottom",
            function: g,
            borderColor: "rgba(153, 102, 255, 1)",
            data: [],
            fill: false,
            lineTension: 0,
        }]
    };
    chartjs.Chart.pluginService.register({
        beforeInit: function(chart) {
            var data = chart.config.data;
            for (var i = 0; i < data.datasets.length; i++) {
                for (var j = 0; j < data.labels.length; j++) {
                    var fct = data.datasets[i].function,
                        x = data.labels[j],
                        y = fct(x);
                    data.datasets[i].data.push(y);
                }
            }
        }
    });
    var myBarChart = new chartjs.Chart(ctx, {
        type: 'line',
        data: data,
        options: {
            scales: {
                yAxes: [{
                    ticks: {
                        beginAtZero:true
                    }
                }]
            }
        }
    });
    });
    element.append("<canvas id='*** TODO ***' width='400'></canvas>");

SymPy functions now can be plotted by filling in the two missing expressions and the id in the above boilerplate code. Then call the JavaScript display function on that code:

```python
from IPython.display import Javascript
```

```python
x=symbols('x')
f1=sin(x)
f2=cos(x)
Javascript(js_template.format(top_function=jscode(f1),
                              bottom_function=jscode(f2),
                              chart_id='sincos'))
```

    <IPython.core.display.Javascript object>

### Lamdify SymPy expressions

Here we convert SymPy expression evaluation to Python Lambda functions. SymPy evaluation is arbitrary precision and therefore very slow. In many cases it may be usefull to first create a Lambda function and then evaluate the function numerically.

Lets create a SymPy expression an record the time for the symbolic evaluation:

```python
y, y = symbols('x y')
expr = 3*x**2 + log(x**2 + y**2 + 1)
expr
```

$\displaystyle 3 x^{2} + \log{\left(x^{2} + y^{2} + 1 \right)}$

```python
expr.subs({x: 17, y: 42}).evalf()
```

$\displaystyle 874.627544390489$

```python
%timeit expr.subs({x: 17, y: 42}).evalf()
```

    296 µs ± 17.4 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

Now we create a python lambda function and compare the time for numeric evaluation.

```python
import math
f = lambda x, y: 3*x**2 + math.log(x**2 + y**2 + 1)
f(17,42)
```

    874.6275443904885

```python
%timeit f(17,42)
```

    2.07 µs ± 158 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

Unsuprisingly, the numeric evaluation is much faster. However, the advantave is deminished by the time we spend coding the python lambda function. Here comes in `lambdify`, that generates python code and execute `eval` on it.

```python
%timeit lambdify([x,y], expr, modules=['math'])(17,42)
```

    3.1 ms ± 364 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

```python
g=lambdify([x,y], expr, modules=['math'])
g(17,42)
```

    874.6275443904885

```python
%timeit g(17,42)
```

    2.59 µs ± 317 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

The code generation uses numpy by default (without `modules` parameter).

```python
g=lambdify([x,y], expr)
g(np.linspace(17,18,5),42)
```

    array([874.62754439, 900.31920442, 926.38590757, 952.82765322,
           979.64444076])

We can get a specific function signature from `lambdify`. In the following example, the function expects a tripple as third argument.

```python
z = z1, z2, z3 = symbols('z:3')
expr = x*y*(z1+z2+z3)
expr
```

$\displaystyle x y \left(z_{0} + z_{1} + z_{2}\right)$

```python
lambdify([x,y,z],expr)(1,2,[3,4,5])
```

    24

**Exercise**: Plot $z(x,y)=\frac{df(x,y)}{dxdy}$ from $f(x,y)=3 x^{2} + \log{\left(x^{2} + y^{2} + 1 \right)}$

```python
y, y = symbols('x y')
expr = 3*x**2 + log(x**2 + y**2 + 1)
expr
```

$\displaystyle 3 x^{2} + \log{\left(x^{2} + y^{2} + 1 \right)}$

```python
d2fdxdy = expr.diff(x,y)
xplot = np.outer(np.linspace(-5,5,100),np.ones(100))
yplot = xplot.T
zplot = lambdify([x, y],d2fdxdy)(xplot,yplot)
```

```python
from mpl_toolkits.mplot3d import Axes3D
import matplotlib.pyplot as plt
%matplotlib notebook
```

```python
fig = plt.figure()
ax = plt.axes(projection='3d')
ax.plot_surface(xplot,yplot,zplot,cmap=plt.cm.coolwarm)
ax.set_xlabel('x')
ax.set_ylabel('y')
ax.set_zlabel('$%s$'% latex(d2fdxdy))
```

    <IPython.core.display.Javascript object>

    Text(0.5, 0, '$- \\frac{4 x y}{\\left(x^{2} + y^{2} + 1\\right)^{2}}$')

```python
%matplotlib notebook
```

The [**competitive Lotka–Volterra equations**](https://en.wikipedia.org/wiki/Competitive_Lotka–Volterra_equations) are a simple model of the population dynamics of species competing for some common resource. They are a system of differential equations that describe the dynamics of a pair of species that interact, one as a predator and the other as a prey.
The coupled first-order non-linear differential equations are given by

$$\frac{dN_1}{dt}=\frac{N_{1} r_{1} \left(K_{1} - N_{1} - N_{2} \alpha\right)}{K_{1}}$$

$$\frac{dN_2}{dt}=\frac{N_{2} r_{2} \left(K_{2} - N_{1} \beta - N_{2}\right)}{K_{2}}$$

where
- $N_i$ = population size of species $i$
- $t$ = time
- $r_i$ = species $i$ growth rate
- $K_i$ = environment carrying capacity
- $\alpha,\beta$ = inter-species interaction parameters.

If species 2 is the predator and species 1 is the prey, then $\alpha$ must be positive (since the predator eats the prey reducing its population) and $\beta$ must be negative since the predator needs the prey to get food.

Define the variables:

```python
# We split it into two definitions to remind us that
# a set of variables is for species 1 and the other one for species 2

# Species 1
r1, K1, N1, alpha = symbols('r_1 K_1 N_1 alpha')

# Species 2
r2, K2, N2, beta = symbols('r_2 K_2 N_2 beta')
```

We can now define the differential equations in sympy

```python
dN1_dt = r1 * N1 * (K1 - N1 - alpha * N2) / K1
dN1_dt
```

$\displaystyle \frac{N_{1} r_{1} \left(K_{1} - N_{1} - N_{2} \alpha\right)}{K_{1}}$

```python
dN2_dt = r2 * N2 * (K2 - N2 - beta * N1) / K2
dN2_dt
```

$\displaystyle \frac{N_{2} r_{2} \left(K_{2} - N_{1} \beta - N_{2}\right)}{K_{2}}$

Usually one of the first things we do when analyzing a dynamical system is to find the steady state, i.e. finding when the derivative with respect to time equals to zero.

We define the equality using the `Eq`:

```python
ss_N1 = Eq(dN1_dt, 0)
ss_N1
```

$\displaystyle \frac{N_{1} r_{1} \left(K_{1} - N_{1} - N_{2} \alpha\right)}{K_{1}} = 0$

```python
ss_N2 = Eq(dN2_dt, 0)
ss_N2
```

$\displaystyle \frac{N_{2} r_{2} \left(K_{2} - N_{1} \beta - N_{2}\right)}{K_{2}} = 0$

Solve for the species population:

```python
ss_N1_sol = solve(ss_N1, N1)
ss_N2_sol = solve(ss_N2, N2)

print('The steady state solution for both species are:')
ss_N1_sol, ss_N2_sol
```

    The steady state solution for both species are:

    ([0, K_1 - N_2*alpha], [0, K_2 - N_1*beta])

So we find that there are two steady state solutions:
1. The trivial solutions in which both species are extinct $N_1=N_2=0$
2. A solution in which both species maintain their non-zero populations.

We can use sympy's solve function to solve simultaneously for both species. The solution is returned as a dictionary where each key is the name of the respective variable.

```python
ss_dict = solve((Eq(N1, ss_N1_sol[1]), Eq(N2, ss_N2_sol[1])),(N1, N2))
ss_dict
```

    {N_1: (-K_1 + K_2*alpha)/(alpha*beta - 1),
     N_2: (K_1*beta - K_2)/(alpha*beta - 1)}

We can use scipy's numerical solver to run a simulation for this system of equations.

**Note**: we use `lambdify` to numerically evaluate the diff. equations

```python
# Generate 'lambdify' functions to numerically evaluate the diff. equations
dN1_dt_num = lambdify([N1, N2, r1, K1, alpha], dN1_dt)
dN2_dt_num = lambdify([N2, N1, r2, K2, beta], dN2_dt)

def lotka_voltera_rhs(n1_n2, t, *p):
    '''
    Set up the right hand side (rhs) function for the system
    (necessary step to feed sympy's odeint function).
    '''
    # unpack the variables
    n1, n2 = n1_n2

    # unpack the parameters
    r_1, k1, a, r_2, k2, b = p

    # Define derivatices rom lambdified functions
    n1dot = dN1_dt_num(n1, n2, r_1, k1, a)
    n2dot = dN2_dt_num(n2, n1, r_2, k2, b)

    return np.array([n1dot, n2dot])
```

Now let's define the parameters and numerically solve the system

```python
from scipy.integrate import odeint
# let's set the parameter values.
r_1, r_2 = [0.5, 0.2]
k1, k2 = [100, 50]
a, b = [1.2, 0.2]
args = (r_1, k1, a, r_2, k2, b)

# Initial population size
n1_n2_0 = [2, 2]

# Time points we want to consider
t = np.linspace(0, 100, 1000)

# Now let's use odeint to solve the function
n1_n2 = odeint(lotka_voltera_rhs, n1_n2_0, t, args=args)
```

We will now plot the solution

```python
for i, N in enumerate([N1, N2]):
    plt.plot(t, n1_n2[:, i], label=r'${0:s}$'.format(latex(N)))
    plt.axhline(y=ss_dict[N].subs({K1: k1, K2: k2, alpha: a, beta: b}),
                linestyle='--',
                label=r'${0:s}$ steady state'.format(latex(N)))

plt.legend(loc=0, fontsize=15)
plt.xlabel('time (units consistent with $r_1$ and $r_2$)')
plt.ylabel('Population size')
```

    <IPython.core.display.Javascript object>

    Text(0, 0.5, 'Population size')

The [**generalised Lotka-Volterra equations**](https://en.wikipedia.org/wiki/Generalized_Lotka–Volterra_equation) model the dynamics of the populations $x_1, x_2, \dots, x_n$ of $n$ biological species ([lecture](https://stefanoallesina.github.io/Sao_Paulo_School/intro.html#basic-formulation)). Together, these populations can be considered as a vector $\mathbf{x}$. They are a set of ordinary differential equations given by

$$\frac{dx_i}{dt}=x_i f_i(\mathbf{x})$$

where the vector $\mathbf{f}$  is given by

$$\mathbf{f}=\mathbf{r}+\mathbf{A}\mathbf{x}$$

where $\mathbf{r}$ is a vector and $\mathbf{A}$ is a matrix known as the [community matrix](https://en.wikipedia.org/wiki/Community_matrix).

```python
y0 = Function('y0')
y1 = Function('y1')
y2 = Function('y2')
t = symbols('t')
C1, C2, C3 = symbols('C1, C2, C3')
b0,b1,b2 = symbols('b0:3')

y = Matrix([y0(t), y1(t), y2(t)])
B = Matrix([b0,b1,b2])

ode = (y.diff(t) - y).T.tolist()[0]
soln = dsolve(ode, y.T.tolist()[0])
soln
```

    [Eq(y0(t), C1*exp(t)), Eq(y1(t), C2*exp(t)), Eq(y2(t), C3*exp(t))]

```python
p1=plot(soln[0].subs({C1:1}).rhs,(t,0,10),show=False)
p2=plot(soln[1].subs({C2:2}).rhs,(t,0,10),show=False)
p3=plot(soln[2].subs({C3:3}).rhs,(t,0,10),show=False)
p1.append(p2[0])
p1.append(p3[0])
p1.show()
```

    <IPython.core.display.Javascript object>

Can this be written more general for any $n$?


# Maximum Likelihood Model Parameter Estimation

\begin{equation*}
D = \text{data} \\
\theta = \text{model parameters} \\
y(\theta, x) = \text{model}
\end{equation*}

A common parameter-estimation strategy is to minimize error,

\begin{equation*}
\sum_{i}(D_i - y_i(\theta, x))^2
\end{equation*}

E.g.

Create some phony straight line data with $\theta=(a, b)$ and normally distributed noise in $y$,

```
%pylab inline
import numpy as np
import matplotlib.pyplot as plt

n_points = 10
x = np.linspace(0,10,n_points)
def y(x, theta=(2, 4), noise=0):
    return  theta[0] * x + theta[1] + noise*np.random.randn(n_points)
D = y(x, noise=1)
plt.scatter(x,D)
```

We are trying to discover parameters, $\theta$, so write error in terms of only $theta$,

$SSE(\theta; y(x), D)$

```
# cheat, do this with global scope of y, x and D
def ssq_err(theta):
    return np.sum((D - y(x,theta))*(D - y(x,theta)).transpose())

from scipy.optimize import minimize
fit = minimize(ssq_err,[1,1])
print fit
plt.scatter(x, D)
plt.plot(x,y(x, fit.x))
print fit
```

Not amazing so far $\ldots$

__Leading quesiton__:  Is normally distributed noise "unusually compatible" with least squares errors minimization?

Maybe we should do something that takes into account the expected distibution of the model errors? What if the errors are distributed by Poisson, for example, does sum-squared-error still make sense?

```
from scipy.stats import norm, poisson
rvn, rvp = norm(5,1), poisson(5)
x = np.linspace(-2,10,140)
plot(x, rvn.pdf(x))
xd = np.arange(0, np.minimum(rvp.dist.b, 15))
vlines(xd, 0, rvp.pmf(xd), lw=2, color="red")
```

__New Idea__:  The parameter fitting problem can be framed as "Can we find the mostly likely value of the parameters $\theta$ of the model given the data?"

i.e. maximize $P(\theta|D)$

### Frequentist Statistics

Significance is specified by probability that
the data was drawn from the “wrong”
model (null hypothesis): p value

* Not the probability a model is correct
* Only random variables have probabilities

That is, from a frequentist perspective, the PDF for normal noise is in terms of $P(D|\theta)$,

E.g.
\begin{equation*}
P(D|\theta) \propto exp(\sum_i -\frac{(D_i - y_i(\theta))^2}{2\sigma^2})
\end{equation*}

We need to flip this around.

### Bayesian inference

* Frequentist: Probability is a frequency
(of a random variable)
* Bayesian: Probability is a measure of
belief or certainty

Some statistics Rules

\begin{equation}
P(A,B) = P(A|B)P(B) = P(B|A)P(A) \\
P(A) = \int_\Omega P(A,B)dB = \int_\Omega P(A|B)P(B)dB \\
\end{equation}

Bayesian statistics is the application of these rules

Back to data and parameters:

\begin{equation*}
P(\theta|D) = \frac{P(D|\theta)P(\theta)}{P(D)}
\end{equation*}

\begin{equation*}
P(D|\theta) : \text{Likelihood} \\
P(\theta) : \text{Prior} \\
P(D) : \text{Evidence} \\
P(\theta|D) : \text{Posterier} \\
\end{equation*}

\begin{equation*}
P(D) = \int_\Omega P(D|\theta)P(\theta)d\theta \\
\end{equation*}

Integration over the parameter space of the model.

## Simple Discrete Model: Coin Toss

In 10 trials, we get 7 heads: What's our best estimate of the probability of flipping heads with this coin?

The distribution of coin toss outcomes is the Binomial Distribution.

$pmf(y|t,w) = \binom{n}{k} p^k (1-p)^{(n-k)}$

Let's not forget how obvious this is and guess 0.7.

```
from scipy.stats import binom
rv = binom(10, 0.7)
x = np.arange(0, np.minimum(rv.dist.b, 10))
h = plt.vlines(x, 0, rv.pmf(x), lw=2, color="orange")
```

"Speciﬁcally, the PDF in Fig. 1 is a function of the data given a particular set of parameter values, deﬁned on the data scale. On the other hand, the likelihood function is a function of the parameter given a particular set of observed data, deﬁned on the
parameter scale. In short, Fig. 1 tells us the probability of a particular data value for a ﬁxed parameter, whereas Fig. 2 tells us the likelihood (‘‘unnormalized probability’’) of a particular parameter value for a ﬁxed data set." (.J. Myung / Journal of Mathematical Psychology 47 (2003) 90–100)

```
def f(w):
    rv = binom(10, w)
    return rv.pmf(7)
# Hold the data constant, vary the parameter
x = np.linspace(0,1,100)
plot(x, f(x))
```

It looks like the maximum likelihood on the data over the parameter is around 0.7?

Our data could have been (6,10) or (8,10) with a 0.7 coin.  But we started by asking about the coin, to make any headway, we need to flip the question around.

```
dat = [10, 7]
def L_binom(w):
    rv = binom(dat[0], w)
    x = dat[1]
    # -1 to make max problem a min problem
    return -rv.pmf(x)
```

```
x = np.linspace(0.1, 0.9, 9)
scatter(x, L_binom(x))
from scipy.optimize import minimize_scalar
minimize_scalar(L_binom, bounds=[0,1], method="bounded")
```

## Slightly More Complicated Model: Exponential with Normal Error

```
n_points = 10
x = linspace(0,100,n_points)
def y(x, theta=(10, 0.05), noise=0):
    return  theta[0] * np.exp(-x * theta[1]) + noise*randn(n_points)
D = y(x, noise=1)
scatter(x,D)
```

Likelihood:

\begin{equation}
P(D|\theta) \propto \prod_i {\exp(-\frac{(D_i - y_i(\theta))^2}{2\sigma^2})} \propto \prod_i {\exp(-\frac{(D_i - (10\exp(-wx)))^2}{2})}
\end{equation}

```
# use x, D from global scope
def L_cost(w):
    return np.prod(np.exp(((D - y(x,(10, w)))*(D - y(x,(10, w))).transpose())/2.))
w = np.linspace(0,0.1,10)
L_c = [L_cost(i) for i in w]
scatter(w,L_c)
```

Numerically troubling!  Turn the product into a sum with log!

```
# use x, D from global scope
def L_cost(w):
    return np.sum(((D - y(x,(10, w)))*(D - y(x,(10, w))).transpose())/2.)
w = np.linspace(0,0.1,10)
L_c = [L_cost(i) for i in w]
scatter(w,L_c)
from scipy.optimize import minimize_scalar
minimize_scalar(L_cost, bounds=[0,0.1], method="bounded")
```

So it might be useful to use the $\log$ likelihood:

\begin{equation}
\log(P(D|\theta)) \propto \sum_i \log(exp(-\frac{(D_i - (10\exp(-wx)))^2}{2})) = - \sum_i \frac{(D_i - (10\exp(-wx)))^2}{2}
\end{equation}

__Normally distributed errors + Minimum-Log-Likelihood is the same solution as Lease Squares.__

## Exponential Decay and Poisson Likelihood

\begin{equation}
\log(P) \propto \sum_i \log(\frac{\lambda^{x_i}e^{-\lambda}}{\Gamma(x_i)}) = \sum_i (x_i) \log\lambda + (-\lambda) - \log\Gamma(x_i)
\end{equation}

Sterling $\ldots$ or realize that $\Gamma(x_i)$ is a constant w.r.t. MLE estimation of $\lambda$.

\begin{equation}
\log(P(D|\theta)) \propto \sum_i \log(\frac{y_i(\theta))^{D_i}e^{-y_i(\theta))}}{\Gamma(D_i)}) = \sum_i D_i \log y_i(\theta) -y_i(\theta)
\end{equation}

```
from numpy.random import poisson
n_points = 20
x = linspace(0,300,n_points)
def y(x, theta=(110, 0.03), noise=0):
    ty = theta[0] * np.exp(-x * theta[1])
    if noise != 0:
        ty = [poisson(k)for k in ty]
    return ty
D = y(x, noise=5)
scatter(x,D)
plot(x,y(x))
```

```
# use x, D from global scope
def P_cost(*w):
    return -np.sum(D*np.log(y(x,*w)) - y(x,*w))
fit_ml = minimize(P_cost, (105,0.05))
print fit_ml
```

```
# cheat, do this with global scope of y, x and D
def ssq_err(*w):
    return np.sum((D - y(x,*w))*(D - y(x,*w)).transpose())
fit_ssq = minimize(ssq_err,[105,0.05])
print fit_ssq
scatter(x, D)
plot(x,y(x,fit_ml.x), color="green")
plot(x,y(x,fit_ssq.x), color="red")
```

Need more?

http://people.physics.anu.edu.au/~tas110/Teaching/Lectures/L3/Material/Myung03.pdf

http://sciencehouse.files.wordpress.com/2013/07/gordonconf13.pdf

http://www.math.utah.edu/~levin/M5080/mle.pdf

```

```


# Solving ODEs with SciPy

Now that we've learnt the basics of ODE solving we can look at using libraries. These libraries allow us to easy use methods that use adapative step size, explicit or implicit methods, and have been checked to work by many developers and tens of thousands of users.

We will look at the Python library SciPy, and imparticular the `solve_ivp` function for solving initial value problems. You can find the documentation for this function at https://docs.scipy.org/doc/scipy/reference/generated/scipy.integrate.solve_ivp.html.

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.integrate import solve_ivp

# The below commands make the font and image size bigger
plt.rcParams.update({'font.size': 22})
plt.rcParams["figure.figsize"] = (15,10)
```

## A simple first-order example

Consider the ODE $y'(x) = -\frac{1}{2} y(x)$ with $y(0) = 10$. First write a function for the derivative:

```python
def exponential_decay(t, y):
    return -0.5 * y
```

`solve_ivp` only needs three arguments (1) the function, (2) the range of the independent variable, and (3) an array with the initial values:

```python
sol = solve_ivp(exponential_decay, [0, 10], [10])
```

Explicitly print out the solutions:

```python
print(sol.t)
print(sol.y)
```

    [ 0.          0.11487213  1.26359346  3.06049939  4.85740531  6.65431124
      8.45121717 10.        ]
    [[10.          9.44182253  5.31648754  2.16609348  0.88253023  0.35956879
       0.14649891  0.06754689]]

Notice that the steps are not evenly spaced as the default function for the `solve_ivp` is an adaptive forth-order Runge-Kutta algorithm (RK4). Printing the the solution provides futher information, such as `nfev` the number of function evaluations:

```python
print(sol)
print(sol.nfev)
```

      message: 'The solver successfully reached the end of the integration interval.'
         nfev: 44
         njev: 0
          nlu: 0
          sol: None
       status: 0
      success: True
            t: array([ 0.        ,  0.11487213,  1.26359346,  3.06049939,  4.85740531,
            6.65431124,  8.45121717, 10.        ])
     t_events: None
            y: array([[10.        ,  9.44182253,  5.31648754,  2.16609348,  0.88253023,
             0.35956879,  0.14649891,  0.06754689]])
    44

Plot the steps and the analytic solution.

```python
t = np.linspace(0,10,100)
y = 10*np.exp(-0.5*t)

plt.grid(True)
plt.scatter(sol.t, sol.y[0], color='red', linewidth=5);
plt.plot(t, y);
```

## Second-order ODE example

Let's look at the second-order ODE: $y''(x) = -y(x)$ with $y[0] = 1, y'[0] = 0$. First we have to write this in first-order form:

$$\begin{align}
    y_0'(x) &= y_1\\
    y_1'(x) &= -y_0
\end{align}$$

Now we define a function for this:

```python
def oscilation(t, y):
    return [y[1], -y[0]]
```

Now let's solve the ODE. Notie we have to pass the two initial conditions. The code will internally use an adapative RK4 method, but let's output the results on a fixed grid by passing the `t_eval` option as a list of values.

```python
solOsc = solve_ivp(oscilation, [0, 10], [1,0], t_eval = np.linspace(0,10,50))
```

Plot the steps against the analytic solution

```python
tOsc = np.linspace(0,10,100)
yOsc = np.cos(tOsc)

plt.grid(True)
plt.scatter(solOsc.t, solOsc.y[0], color='red', linewidth=5);
plt.plot(tOsc, yOsc);
```

# Stiff ODE example

```python
lam = 300

def dydxStiff(x,y):
    global lam
    return lam*(-y + np.sin(x))

def yStiff(x):
    global lam
    C = lam/(1+lam**2)
    return C*np.exp(-lam*x) + (lam**2*np.sin(x) -lam*np.cos(x))/(1+lam**2)
```

The implicit methods often want/need the Jacobian matrix. This is an $n\times n$ matrix where the elements $a_{ij} = df_i/dy_j$

```python
def jacobian(x, y):
    global lam
    return [[-lam]]
```

```python
solStiffRK4      = solve_ivp(dydxStiff, [0, 2], [0], method='RK45')
solStiffImplicit = solve_ivp(dydxStiff, [0, 2], [0], method='BDF', jac=jacobian)
```

```python
plt.grid(True)
plt.scatter(solStiffRK4.t, solStiffRK4.y[0]);
plt.scatter(solStiffImplicit.t, solStiffImplicit.y[0]);
plt.legend(['Adapative RK4 (explicit) method', 'Adapative BDF (implicit) method']);
```

We see that the adaptive integrator forces the RK4 method to take many tiny steps, whereas the implicit `BDF` method can take much larger steps. Try playing with $\lambda$ above. The larger you make it the stiffer the ODE becomes and the more steps the adaptive RK4 method has to take in order to maintain accuracy. The implicit method though takes roughly the same number of steps regardless of the value of $\lambda$.

```python
print("Number of steps RK4 took: %d" % solStiffRK4.nfev)
print("Number of steps BDF took: %d" % solStiffImplicit.nfev)
```

    Number of steps RK4 took: 1382
    Number of steps BDF took: 86

```python

```


# Symbolic Mathematics in Python

There are times when you need to solve a difficult problem symbollically or analytically. If you have ever used Wolfram Alpha, then you have already done this. Sympy is a python library that allows you to do symbolic mathematics in python.

```python
import sympy as sym
```

## 1. Introduction

### Example 1.1

If you try to write the follwing in python by itself, you will get an error telling you x is undefined:
$$x-x$$

```python
x-x
```

(The error above is on purpose). Variables in python need to be defined before you can say something specific about them

```python
x=102
x-x
```

    0

If you are trying to show that $x-x=0$ is true for any $x$, the above answer would not be valid. Instead you can use a symbolic expression to show that it is true

**First we define the variable as a symmbolic expression**

```python
x = sym.symbols('x')
```

**Now we can use the variable in a symbolic expression**

```python
x-x
```

    0

### Example 1.2

Sympy can be used to perform algebreic operations (among other things). Consider the following expression: $$(3a-4b)^3$$

We can use symppy to expand the expression algebraically.

**First we need to define the variables as symbolic expressions**

```python
a,b = sym.symbols('a,b')
```

**Side note** Notice that the left hand side of the epression has two variables being defined. Python can define more than one variable at a time:

```python
x1,y1 =10,20
print(x1)
print(y1)
```

    10
    20

**Back to the expression** We can define an expression using the variables $a$ and $b$.

```python
expr = (3*a-4*b)**3
print(expr)
```

    (3*a - 4*b)**3

We can also make it look nicer in our notebook. This doesn't affect the math, but it makes our notebook more readable.

```python
sym.init_printing()
```

```python
expr
```

**Now we expand the function algebreically**

```python
expr.expand()
```

Sympy can also factor the equation

```python
sym.factor(26*a**3-108*a**2*b+144*a*b**2-64*b**3)
```

If you want to copy and paste a result, you print the result.

```python
print(sym.factor(26*a**3-108*a**2*b+144*a*b**2-64*b**3))
```

    2*(a - 2*b)*(13*a**2 - 28*a*b + 16*b**2)

You can also chain together functions

```python
expr.expand().factor()
```

### Exercise 1.1

Show that the following two expressions are true.
$$(2w-3z)(2w+3z)=4w^2-9z^2$$
$$(2w-3z)^2\ne4w^2-9z^2$$

```python
# Your code here
```

## 2. Solving Equations

Sympy can be used to symbolilically solve equations. As before, you need to define which variables are symbols

### Example 2.1

Use sympy to solve the following equation
$$ax^3+bx^2+cx+d=0$$

```python
# Define the variables
a,b,c,d,x = sym.symbols('a,b,c,d,x')
```

```python
# Define the expression
expr=a*x**3+b*x**2+c*x+d
expr
```

We can use the `solvset` function to solve this equation

```python
solutions=sym.solveset(expr,x)
```

```python
print(solutions)
```

    {-(-3*c/a + b**2/a**2)/(3*(sqrt(-4*(-3*c/a + b**2/a**2)**3 + (27*d/a - 9*b*c/a**2 + 2*b**3/a**3)**2)/2 + 27*d/(2*a) - 9*b*c/(2*a**2) + b**3/a**3)**(1/3)) - (sqrt(-4*(-3*c/a + b**2/a**2)**3 + (27*d/a - 9*b*c/a**2 + 2*b**3/a**3)**2)/2 + 27*d/(2*a) - 9*b*c/(2*a**2) + b**3/a**3)**(1/3)/3 - b/(3*a), -(-3*c/a + b**2/a**2)/(3*(-1/2 - sqrt(3)*I/2)*(sqrt(-4*(-3*c/a + b**2/a**2)**3 + (27*d/a - 9*b*c/a**2 + 2*b**3/a**3)**2)/2 + 27*d/(2*a) - 9*b*c/(2*a**2) + b**3/a**3)**(1/3)) - (-1/2 - sqrt(3)*I/2)*(sqrt(-4*(-3*c/a + b**2/a**2)**3 + (27*d/a - 9*b*c/a**2 + 2*b**3/a**3)**2)/2 + 27*d/(2*a) - 9*b*c/(2*a**2) + b**3/a**3)**(1/3)/3 - b/(3*a), -(-3*c/a + b**2/a**2)/(3*(-1/2 + sqrt(3)*I/2)*(sqrt(-4*(-3*c/a + b**2/a**2)**3 + (27*d/a - 9*b*c/a**2 + 2*b**3/a**3)**2)/2 + 27*d/(2*a) - 9*b*c/(2*a**2) + b**3/a**3)**(1/3)) - (-1/2 + sqrt(3)*I/2)*(sqrt(-4*(-3*c/a + b**2/a**2)**3 + (27*d/a - 9*b*c/a**2 + 2*b**3/a**3)**2)/2 + 27*d/(2*a) - 9*b*c/(2*a**2) + b**3/a**3)**(1/3)/3 - b/(3*a)}

```python
solutions
```

What if I need help. You can do this with any python function. `function?`

```python
# Run this command to see a help box
sym.solveset?
```

### Exercise 2.1

Use the `solveset` function to solve the following chemical problem.

Phosgene gas, $\text{COCl}_2$, dissociates at high temperatures according to the following equilibrium:

$$ \text{COCl}_2 \rightleftharpoons  \text{CO} + \text{Cl}_2 $$

At $\text{400 C}$, the equilibrium constant $K_c=8.05$.

If you start with a  $\text{0.250 M}$ phosgene sample at $\text{400 C}$, determine the concentrations of all species at equilibrium.

```python
# Your code here
```

Why did you pick your answer?

## 3. Calculus

We can use also Sympy to differentiate and integrate. Let us experiment with differentiating the following expression:

$$x ^ 2 - \cos(x)$$

```python
sym.diff(x ** 2 - sym.cos(x), x)
```

Similarly we can integrate:

```python
sym.integrate(x ** 2 - sym.cos(x), x)
```

We can also carry out definite integrals:

```python
sym.integrate(x ** 2 - sym.cos(x), (x, 0, 5))
```

### Exercise 3.1

Use Sympy to calculate the following:

1. $\frac{d(x ^2 + xy - \ln(y))}{dy}$
1. $\int_0^5 e^{2x}\;dx$

### Exercise 3.2
Solve the following definate integral
$$\int\limits_{ - \infty }^\infty  {\frac{1}{{\sigma \sqrt {2\pi } }}{e^{ - \frac{1}{2}{{\left( {\frac{{x - \mu }}{\sigma }} \right)}^2}}}}$$

Hint, the sympy symbol for infinity is `oo`

```python
# Your code here
```

Lookup Gaussian functions: https://en.wikipedia.org/wiki/Gaussian_function
Does your answer maake sense?

## 4. Plotting with Sympy

Finally Sympy can be used to plot functions. Note that this makes use of [matplotlib](http://matplotlib.org/).

Let us plot $x^2$:

```python
expr = x ** 2
p = sym.plot(expr)
```

### Exercise 4.1 Plot the following function:

1. $y=x + cos(x)$
1. ${\frac{1}{{ \sqrt {2\pi } }}{e^{ - \frac{x^2}{2}}}}$

```python
# Your code here
```

# Lecture

## L1. Hydrogen Atom

Sympy has built in modules for the eigenfunctions of the hydrogen atom.

```python
import sympy.physics.hydrogen
import numpy as np
```

You can caluclate the eigenvalues ($E$) in Hartrees

`sym.physics.hydrogen.E_nl(n,Z)`

```python
sym.physics.hydrogen.E_nl(1,1)
```

    -1/2

We can use a loop to print out many energies

```python
for n in range(1,5):
    print(sym.physics.hydrogen.E_nl(n,1))
```

    -1/2
    -1/8
    -1/18
    -1/32

We can plot the hydrogen radial wavefunction (1s orbital)

```python
sympy.symbols('r')
sympy.physics.hydrogen.R_nl(1, 0, r, 1)
```

    2*exp(-r)

```python
sym.plot(sympy.physics.hydrogen.R_nl(1, 0, r, 1),(r,0,10.50))
```

And the probablity distribution function

```python
sympy.symbols('r')
prob_1s=sympy.physics.hydrogen.R_nl(1, 0, r, 1)*sympy.physics.hydrogen.R_nl(1, 0, r, 1)
prob_1s
```

    4*exp(-2*r)

```python
sym.plot(prob_1s,(r,0,10))
```

Plot a 2s orbital

```python
sympy.symbols('r')
prob_2s=sympy.physics.hydrogen.R_nl(2, 0, r, 1)*sympy.physics.hydrogen.R_nl(2, 0, r, 1)
prob_2s
```

    (-r + 2)**2*exp(-r)/8

```python
sym.plot(prob_2s,(r,0,10))
```

We can change the range to see the node better.

```python
sym.plot(prob_2s,(r,1,8))
```

Notice the node!

### Exercise L1.1

Plot the radial distriubution function for a 2p, 3s, 4s, and 3d orbital.

```python
# Your code here
```

```python

```


# Mixed Integer Linear Programming (MILP)

## Introduction

* Some variables are restricted to be integers
* NP-complete
* Applications
    * Production planning
    * Scheduling
    * Many more...
* Exact Algorithms
    * Cutting plane methods
    * Variants of branch and bound
* Heuristic Methods
    * LP relaxation
    * Tabu search

## The standard form

\begin{align}
\text{maximize}\  & \mathbf{c}^T\mathbf{x} + \mathbf{k}^T\mathbf{y} \\
\text{subject to } & \\
& A\mathbf{x} &&\leq \mathbf{b} \\
& D\mathbf{y} &&\leq \mathbf{e} \\
& \mathbf{x},\mathbf{y} &&\geq 0 \\
& \mathbf{x} \in \mathbb{Z}^n
\end{align}
where $A, D \in \mathbb{R}^{m\times n}$ are matrices, $\mathbf{b}, \mathbf{e}\in\mathbb{R}^{m}$ are constants, $\mathbf{c}, \mathbf{k} \in \mathbb{R}^{n}$ objective function coefficients, and $\mathbf{x}, \mathbf{y} \in\mathbb{R}^{n}$ are the decision variables.

## Gurobi basics: Mixed Integer Programming Model
## Mathematical Model
\begin{align}
\text{maximize}\  & 2x + y + 3z \\
\text{subject to } & \\
& x+2y+z &&\leq 4 \\
& 2z + y &&\leq 5 \\
& x + y &&\geq 1 \\
& x &&\in \{0,1\} \\
& y, z \geq 0 \\
& z \in \mathbb{Z}
\end{align}

# Code in Python using gurobipy
## Step 1: Importing gurobipy package

```python
from gurobipy import *
```

## Step 2: Create an optimization model

```python
milp_model = Model("milp")
```

## Step 3: Add decision variables

```python
x = milp_model.addVar(vtype=GRB.BINARY, name="x")
y = milp_model.addVar(vtype=GRB.CONTINUOUS, lb=0, name="y")
z = milp_model.addVar(vtype=GRB.INTEGER, lb=0, name="z")
```

## Step 4: Define the objective function

```python
obj_fn = 2 * x + y + 3 * z
milp_model.setObjective(obj_fn, GRB.MAXIMIZE)
```

## Step 5: Add the constraints

```python
# Add constraint: x + 2 y + z <= 4
c1 = milp_model.addConstr(x + 2 * y + z <= 4, "c1")

# Add constraint: 2 z + y <= 5 \\
c2 = milp_model.addConstr(2 * z + y <= 5, "c2")

# Add constraint x + y >= 1
c3 = milp_model.addConstr(x + y >= 1, "c3")
```

## Step 6: Solve the model

```python
milp_model.optimize()
```

## Step 7: Output the result

```python
print('Objective Function Value: %.2f' % milp_model.objVal)
# Get values of the decision variables
for v in milp_model.getVars():
    print('%s: %g' % (v.varName, v.x))
```


# Introducción a Optimización con Cplex & Python

 Material preparado para la carrera de Ingeniería Civil Industrial | Universidad Católica del Norte | Escuela de Ingeniería - Campus Coquimbo.
© Sergio Correa Santis, Ingenerio Industrial UCN, Master of International Business UQ, Master Engeneering of Supply Chain and Logistics, MIT ZLC-Global Scale.                               Contacto: sergio.correa@ucn.cl o sergio.correa.santis@gmail.com

> El objetivo de este material es introducir a la optimización utilizando Python y Cplex.
Para mayor información puede visitar el siguiente link [Cplex IBM](https://ibmdecisionoptimization.github.io/tutorials/html/Linear_Programming.html)
.Para realizar gráficas puede revisar el siguiente link [Matplot](https://matplotlib.org)

(Ejemplo basado del Wiston)

### Gepeto Toys Inc. manufactura 2 tipos de jugestes: **Soldados** y **Trenes**.

Un soldado se vende por 27 dólares y usa 10 dólares en materias primas. Cada soldado que se fabrica aumenta la mano de obra variable y los costos generales de Giapetto en 14 dólares. Un tren se vende por 21 dólares y usa 9 dólares en materias primas. Cada tren construido aumenta los costos laborales y generales variables de Giapetto en 10 dólares. La fabricación de soldados y trenes de madera requiere dos tipos de mano de obra calificada: **carpintería** y **acabado**.

Un soldado requiere 2 horas de trabajo de acabado y 1 hora de trabajo de carpintería. Un tren requiere 1 hora de trabajo de acabado y 1 hora de trabajo de carpintería. Cada semana, Giapetto puede obtener toda la materia prima necesaria, pero solo 100 horas de acabado y 80 horas de carpintería. La demanda de trenes es ilimitada, pero se compran como máximo 40 soldados cada semana. Giapetto quiere maximizar la ganancia semanal (ingresos-costos).

**Formule un modelo matemático de la situación de Giapetto que pueda usarse para maximizar el beneficio semanal de Giapetto.**

###  Modelo de Programación Lineal
\begin{equation}
Max\;Z=27x_1+21x_2-(10x_1+9x_2)-(14x_1+10x_2)
\end{equation}
\begin{equation}
x_1+x_2\leq 80
\end{equation}
\begin{equation}
2x_1+x_2 \leq 100
\end{equation}
\begin{equation}
x_1\leq 40
\end{equation}
\begin{equation}
x_1,x_2\geq0
\end{equation}

Donde $x_1$ es la cantidad de soldados a producir y $x_2$ es la cantidad de trenes a producir.

```python
from IPython.display import YouTubeVideo
YouTubeVideo('lTcJlw0iaJ4')
```

```python
# Importanto Cplex.
from docplex.mp.model import Model
```

```python
# Inicializando el modelo
mdl = Model('modelo')
```

```python
# Creando las variables de decisión - continuas

x1 = mdl.continuous_var(name='x1')
x2 = mdl.continuous_var(name='x2')
```

### Creando las restricciones

\begin{equation}
x_1+x_2\leq 80
\end{equation}
\begin{equation}
2x_1+x_2 \leq 100
\end{equation}
\begin{equation}
x_1\leq 40
\end{equation}
\begin{equation}
x_1,x_2\geq0
\end{equation}

```python
# Escribiendo las restricciones

mdl.add_constraint(x1+x2 <= 80)
mdl.add_constraint(2*x1+x2 <= 100)
mdl.add_constraint(x1 <= 40)

```

    docplex.mp.LinearConstraint[](x1,LE,40)

### Creando la Función Objetivo
\begin{equation}
Max\;Z=27x_1+21x_2-(10x_1+9x_2)-(14x_1+10x_2)
\end{equation}

```python
# Creando la función objetivo
mdl.maximize(27*x1+21*x2-10*x1-9*x2-14*x1-10*x2)
```

```python
print(mdl.export_to_string())
```

    \ This file has been generated by DOcplex
    \ ENCODING=ISO-8859-1
    \Problem name: modelo

    Maximize
     obj: 3 x1 + 2 x2
    Subject To
     c1: x1 + x2 <= 80
     c2: 2 x1 + x2 <= 100
     c3: x1 <= 40

    Bounds
    End

```python
solucion = mdl.solve(log_output=True)
```

    CPXPARAM_Read_DataCheck                          1
    Tried aggregator 1 time.
    LP Presolve eliminated 1 rows and 0 columns.
    Reduced LP has 2 rows, 2 columns, and 4 nonzeros.
    Presolve time = 0.01 sec. (0.00 ticks)

    Iteration log . . .
    Iteration:     1   Dual infeasibility =             0.000000
    Iteration:     2   Dual objective     =           180.000000

```python
solucion.display()
```

    solution for: modelo
    objective: 180.000
    x1 = 20.000
    x2 = 60.000

```python
# Aprendiendo a graficar
import numpy as np
from matplotlib import pyplot as plt
from matplotlib.path import Path
from matplotlib.patches import PathPatch
import seaborn as sns

colores=sns.palplot(sns.color_palette("hls", ))
sns.set()
sns.set(style='white')
sns.despine()
sns.set_context('notebook')

# Crear un objeto gráfico.
fig, ax = plt.subplots(figsize=(8, 8))
x1 = np.linspace(0, 100)

# agregando la restricción de horas de capintería
plt.plot(x1, 80 - x1, lw=3, label='Carpinteria')
plt.fill_between(x1, 0, 80 - x1, alpha=0.1)

# agregando la restricción de horas de acabado
plt.plot(x1, 100 - 2 * x1, lw=3, label='Acabado')
plt.fill_between(x1, 0, 100 - 2 * x1, alpha=0.1)

# agregando la restricción de demanda
plt.plot(40 * np.ones_like(x1), x1, lw=3, label='Demanda')
plt.fill_betweenx(x1, 0, 40, alpha=0.1)

# add non-negativity constraints
plt.plot(np.zeros_like(x1), x1, lw=3, label='$x_2$ No Negatividad')
plt.plot(x1, np.zeros_like(x1), lw=3, label='$x_1$ No Negatividad')

plt.xlabel('Soldados', fontsize=16)
plt.ylabel('Trenes', fontsize=16)
plt.xlim(-0.5, 100)
plt.ylim(-0.5, 100)
plt.legend(fontsize=12)
plt.show()
```

```python
# Aprendiendo a graficar
import numpy as np
from matplotlib import pyplot as plt
from matplotlib.path import Path
from matplotlib.patches import PathPatch
import seaborn as sns

colores=sns.palplot(sns.color_palette("hls", ))
sns.set()
sns.set(style='white')
sns.despine()
sns.set_context('notebook')

# Crear un objeto gráfico.
fig, ax = plt.subplots(figsize=(8, 8))
x1 = np.linspace(0, 100)

# agregando la restricción de horas de capintería
plt.plot(x1, 80 - x1, lw=3, label='Carpinteria')
plt.fill_between(x1, 0, 80 - x1, alpha=0.1)

# agregando la restricción de horas de acabado
plt.plot(x1, 100 - 2 * x1, lw=3, label='Acabado')
plt.fill_between(x1, 0, 100 - 2 * x1, alpha=0.1)

# agregando la restricción de demanda
plt.plot(40 * np.ones_like(x1), x1, lw=3, label='Demanda')
plt.fill_betweenx(x1, 0, 40, alpha=0.1)

# add non-negativity constraints
plt.plot(np.zeros_like(x1), x1, lw=3, label='$x_2$ No Negatividad')
plt.plot(x1, np.zeros_like(x1), lw=3, label='$x_1$ No Negatividad')

# agregando la restricción función objetivo
plt.plot(x1, 50 - (3/2) * x1,color='magenta',linestyle='dashed',lw=4, label='$Z=100$')
plt.plot(x1, 75 - (3/2) * x1,color='magenta',linestyle='dashed',lw=4, label='$Z=150$')
plt.plot(x1, 90 - (3/2) * x1,color='magenta',linestyle='dashed',lw=4, label='$Z=180$')
plt.plot(x1, 100 - (3/2) * x1,color='magenta',linestyle='dashed',lw=4, label='$Z=200$')

plt.xlabel('Soldados', fontsize=16)
plt.ylabel('Trenes', fontsize=16)
plt.xlim(-0.5, 100)
plt.ylim(-0.5, 100)
plt.legend(fontsize=12)
plt.show()
```

 Material preparado para la carrera de Ingeniería Civil Industrial | Universidad Católica del Norte | Escuela de Ingeniería - Campus Coquimbo.
© Sergio Correa Santis, Ingenerio Industrial UCN, Master of International Business UQ, Master Engeneering of Supply Chain and Logistics, MIT ZLC-Global Scale.                               Contacto: sergio.correa@ucn.cl o sergio.correa.santis@gmail.com

```python

```

Intro/Introducción a Cplex & Python.ipynb


# The Gamma Function $\Gamma$

$\Gamma(p)$ is a generalization of the factorial function to real and complex $p$. If $p$ is a positive integer then $\Gamma(p) = (p-1)!$, the familar factorial __except with the argument offset by 1__ (widely described as an unfortunate historical accident).

More generally we can define

$$ \Gamma(z) = \int_0^{\infty} x^{z-1} e^{-x} dx \quad \text{for} \quad Re(z) > 0 $$

Gamma functions still obey the same recursion relation as factorials, $\Gamma(z+1) = z \Gamma(z)$, even for $Re(z) < 0$. Rearranging this slightly gives the definition for negative numbers:

$$ \Gamma(z) = \frac{1}{z} \Gamma(z+1) $$

However, the function is undefined (divergent) when $z$ is zero or a negative integer.

```python
import numpy as np
import matplotlib.pyplot as plt

plt.rcParams.update({'font.size': 16})
```

SciPy will evaluate Gamma functions very simply:

```python
import scipy.special as sp
xlims = (-4, 4)
x = np.linspace(xlims[0], xlims[1], 500)
gammas = sp.gamma(x)
# display(gammas)

plt.figure(figsize=(9, 9))
plt.plot(x, gammas)

plt.xlim(xlims)
plt.ylim((-10, 10))
plt.xlabel('$x$')
plt.ylabel('$\\Gamma(x)$')
plt.title('Plot of the Gamma function for real x')
plt.grid(True)
```

Complex values also work without fuss, but remember that $\sqrt{-1}$ is entered as `1j` in Python. Plots of $\Gamma(z)$ in the complex plane are not at all simple and quite pretty, so there are plenty of examples on the web.

```python
sp.gamma(1.1+0.5j)
```

    (0.7911433469580142-0.14123664385292767j)

SymPy will do all sorts of complicated things with Gamma and related functions, but I've not yet found anything simple and obviously useful to me.

```python
from sympy import gamma, pi, init_printing, diff
from sympy.abc import x
init_printing()

display(gamma(x))
display(diff(gamma(x),x))
```

## Beta functions

These are somewhat related to Gamma functions but take two arguments:

$$ B(p,q) = \int_0^1 x^{p-1} (1-x)^{q-1} dx, \quad p>0, \quad q>0 $$

They can be expressed in terms of Gamma functions:

$$ B(p,q) = \frac{\Gamma(p) \Gamma(q)}{\Gamma(p+q)} $$

## References:

- Boas, "Mathematical methods in the physical sciences", 3rd ed, sections 11.3-11.6
- MathWorld, http://mathworld.wolfram.com/GammaFunction.html and http://mathworld.wolfram.com/BetaFunction.html
- Wikipedia, https://en.wikipedia.org/wiki/Gamma_function and https://en.wikipedia.org/wiki/Beta_function

```python

```