# **Symbolic Regression**

## 1. What is symbolic regression?

**Symbolic regression** is a _machine learning_ technique that finds a **symbolic expression** that matches data from an unknown function. In other words, it is a machinery able to identify an underlying mathematical expression that best describes a relationship between one or more variables.

The symbolic regression problem for mathematical functions has been tackled with a variety of methods, including:

- **Sparse regression**: [Quade, et, al.](https://aip.scitation.org/doi/abs/10.1063/1.5027470?ai=1gvoi&mi=3ricys&af=R&), [Brunton, et al.](https://arxiv.org/pdf/1509.03580.pdf);
- **Genetic algorithms**: [GPTIPS](https://sites.google.com/site/gptips4matlab/) (`Matlab`), [gplearn](https://gplearn.readthedocs.io/en/stable/) (`Python`);
- [AI Feynman](https://advances.sciencemag.org/content/6/16/eaay2631/tab-pdf) (Tegmark).

## 2. How does it works?

It builds a set of random formulas to represent the relationship between known independent variables and their dependent variable targets in order to predict them.

Each successive **step** (_generation_) of programs is then **transformed** (_evolved_) from the one that came before it (_by selecting the fittest individuals_) from the **data** (_population_) to undergo **next** (_genetic_) operations.

### 2.1. Representation

For example, to write the following expression:

\begin{equation}
 y = X^2_0 - 3 X_1 + 0.5
\end{equation}
we can rewrite it as
\begin{equation}
 y = X_0 \times X_0 - 3 \times X_1 + 0.5 .
\end{equation}

But we can do more, we can use a _LISP symbolic expression_:
\begin{equation}
 y = (+ ( - (\times X_0 X_0)(\times 3 X_1)) 0.5 )
\end{equation}
or even, we can understand is as a **syntax tree**, where the **interior nodes** are the _functions_ and the _variables_ and _constants_ are the **terminal nodes**:

### 2.2 Fitness

It determines how well the program performs. As in other ML things, in GP we have to know whether the metric needs to be _maximized_ or _minimized_ in order to be able solve each specific problem:

- **Regression problems**:
    - `MSE`: mean squared error;
    - `RMSE`: root mean squared error;

- **Classification problems**:
    - `LOG LOSS`: logarithmic loss;
    - `BIN CROSS-ENTROPY`: binary cross-entropy loss;

### 2.3 Initialization

Compreehends the **parameters** that should be chosen to perform the symbolic operation:

- `population_size`: number of programs competing in the first generation and every generation thereafter;
- `function_set`: they are the available mathematical functions that  you want to pass;
- `generations`: the maximum number of steps until the programs stops;
- `stopping_criteria`:it defines a default number meaning a perfect score $\Rightarrow$ used to stop the program too;
- `p_crossover:` this crossover parameter is a percentage that takes the winner of a tournament and selects a random subtree from it to be replaced in the next generations;
- `p_subtree_mutation`: it is a more agressive operation. It basically is another percentage parameter that allows to reintroduce extinct functions and operators into the population to maintain diversity;
- `p_hoist_mutation`: another percentage parameter that is responsible to remove genetic material from tournament winners;
- `p_point_mutation`: crazy percentage parameter $\Rightarrow$ it takes the winner of a tournament and selects random nodes from it to be replaced;
- `max_samples:` increase the amount of subsampling performed on your data to get more diverse looks at individual programs from smaller portions of the data;
- `parsimony_coefficient`: controls the penalty given to fitness measure during selection.

## 2.4 Summarizing

## 3. What we are going to use?

`gplearn` implements Genetic Programming in `Python`, with a scikit-learn inspired and compatible API.

## 4. Symbolic Regressor Example

Here we will predict the Hublle evolution $H$ with the redshift $z$:

\begin{equation}
 H (z) = H_0 \sqrt{\Omega_k (1 + z)^2 + \Omega_m (1 + z)^3 + \Omega_r (1 + z)^4 + \Omega_{\Lambda}}
\end{equation}

where $H_0$ is the Hubble parameter, $\Omega_k$ is the curvature density, $\Omega_m$ is the matter density, $\Omega_r$ is the radiation density and $\Omega_{\Lambda}$ is the dark energy density of the Universe, all today.

### 4.1 Importing libraries

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from gplearn.genetic import SymbolicRegressor
import sklearn
from sklearn.ensemble import RandomForestRegressor
from sklearn.tree import DecisionTreeRegressor
from sympy import *
from sklearn.utils.random import check_random_state
import graphviz
import time
```

### 4.2 Initial dataset and data analysis

Initial dataset:

```python
nsample = 1000
sig = 0.2
```

```python
def Hz(z):
    H0 = 67.3
    Omega_m = 0.3115
    Omega_r = 2.473*10**(-5)
    Omega_de = 0.685
    Omega_k = 1. - Omega_m - Omega_r - Omega_de
    return H0*np.sqrt(Omega_k*(1 + z)**2 + Omega_m*(1 + z)**3 + Omega_r*(1 + z)**4 + Omega_de)
```

```python
rng = check_random_state(0)
z = rng.uniform(0, 10, nsample)
H = Hz(z) + sig*np.random.normal(size=nsample)
```

```python
data = np.array([z, H]).T
columns = ['z', 'H (z)']
df = pd.DataFrame(data = data, columns = columns)
df.head()
```

      z
      H (z)

      0
      5.488135
      624.287137

      1
      7.151894
      876.945822

      2
      6.027634
      702.550795

      3
      5.448832
      618.878539

      4
      4.236548
      454.219849

Data visualization:

```python
plt.figure(dpi = 100)
plt.title('Huble parameter evolution')
plt.scatter(df['z'], df['H (z)'])
plt.ylabel(r'$H (z)$')
plt.xlabel(r'$z$')
```

### 4.3 ML getting data:

```python
X = df[['z']]
y = df['H (z)']
y_true = y
X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(X, y, test_size=0.30, random_state=42)
X_train.shape, X_test.shape, y_train.shape, y_test.shape
```

    ((700, 1), (300, 1), (700,), (300,))

### 4.4 GPlearn implementation

#### 4.4.1 First test:

a) Choosing just some functions

```python
function_set = ['add', 'sub', 'mul']
```

```python
est_gp = SymbolicRegressor(population_size=5000,function_set=function_set,
                           generations=20, stopping_criteria=0.01,
                           p_crossover=0.7, p_subtree_mutation=0.1,
                           p_hoist_mutation=0.05, p_point_mutation=0.1,
                           max_samples=0.9, verbose=1,
                           parsimony_coefficient=0.01, random_state=0)
```

b) Fit:

```python
t0 = time.time()
est_gp.fit(X_train, y_train)
print('Time to fit:', time.time() - t0, 'seconds')
```

        |   Population Average    |             Best Individual              |
    ---- ------------------------- ------------------------------------------ ----------
     Gen   Length          Fitness   Length          Fitness      OOB Fitness  Time Left
       0    48.81      3.53838e+10       63          177.246          167.958      1.97m
       1    51.35           187098       63          143.405          141.207      1.61m
       2    63.31           197607      139          88.6634          83.3361      1.69m
       3    80.92      3.82144e+06      139          87.1473           80.188      1.70m
       4    73.40       1.7296e+06      143          85.3693          96.1901      1.51m
       5    88.35      2.55467e+06      135          82.2406          83.0213      1.59m
       6   123.16      6.42323e+09      129          58.0368          53.7706      1.70m
       7   142.64      8.53495e+09      129           49.992          52.5158      1.73m
       8   139.45      2.36822e+11      129          48.8323          44.3659      1.62m
       9   135.79        2.364e+08      145          41.4034          45.2658      1.44m
      10   132.55      7.76555e+07      149          40.9734          49.1365      1.31m
      11   136.44      4.68113e+06      151          39.3009           36.933      1.19m
      12   139.26      2.58968e+06      149          38.6996          42.3448      1.05m
      13   150.45      2.31207e+07      119          30.0367          32.4589     56.88s
      14   149.38      1.77535e+07      109          30.0065          32.7309     47.91s
      15   148.63       3.0615e+11      139           27.309          29.4107     40.99s
      16   153.20      7.27673e+06      139          26.9776           32.394     28.45s
      17   146.20      1.45254e+09      125          23.7784          24.5737     18.87s
      18   137.98      3.98801e+07      133          23.4379          26.5299      8.94s
      19   139.36       3.7941e+09      147          19.6209          20.0325      0.00s
    Time to fit: 163.70209121704102 seconds

c) Prediction

```python
t0 = time.time()
y_gp1 = est_gp.predict(X_test)
print('Time to predict:', time.time() - t0, 'seconds')
```

    Time to predict: 0.003470182418823242 seconds

d) Score

```python
score_gp1 = est_gp.score(X_test, y_test)
print('R2:', score_gp1)
```

    R2: 0.9967157328344243

##### 4.4.1.1. **Visualizing the symbolic function**

a) Equation

```python
converter = {
    'add': lambda x, y : x + y,
    'sub': lambda x, y : x - y,
    'mul': lambda x, y : x*y,
    'div': lambda x, y : x/y,
    'sqrt': lambda x : x**0.5,
    'log': lambda x : log(x),
    'abs': lambda x : abs(x),
    'neg': lambda x : -x,
    'inv': lambda x : 1/x,
    'max': lambda x, y : max(x, y),
    'min': lambda x, y : min(x, y),
    'sin': lambda x : sin(x),
    'cos': lambda x : cos(x),
    'pow': lambda x, y : x**y,
}
```

```python
next_e = sympify(str(est_gp._program), locals=converter)
next_e
```

$\displaystyle \left(3.928432 X_{0} + 7.036\right) \left(- X_{0}^{2} + X_{0} \left(0.806 X_{0} + 1.681\right) + 2.806 X_{0} + 3.35\right)$

b) Score

```python
y_gp = est_gp.predict(X_test)
score_gp1 = est_gp.score(X_test, y_test)
score_gp1
```

    0.9967157328344243

c) Plot

```python
fig = plt.figure(constrained_layout=False, dpi=100)
gs = fig.add_gridspec(nrows=7, ncols=1)
f_ax1 = fig.add_subplot(gs[0:5,0])
plt.title('H (z) comparison')
plt.scatter(X_test, y_test, label = 'True function')
plt.scatter(X_test, y_gp1, marker = 'v', s = 10, label = 'Symbolic function')
plt.legend()
plt.ylabel('H (z)')
f_ax2 = fig.add_subplot(gs[5:7, 0])
plt.scatter(X_test, 1. - y_gp1/y_test, marker = 's', s = 10)
plt.ylabel('Residual')
plt.xlabel('z')
```

d) Tree

```python
dot_data = est_gp._program.export_graphviz()
graph = graphviz.Source(dot_data)
graph.render('images/ex1', format='png', cleanup=True)
graph
```

#### 4.4.2 Second test:

a) Don't imposing any function

```python
est_gp = SymbolicRegressor(population_size=5000,
                           generations=20, stopping_criteria=0.01,
                           p_crossover=0.7, p_subtree_mutation=0.1,
                           p_hoist_mutation=0.05, p_point_mutation=0.1,
                           max_samples=0.9, verbose=1,
                           parsimony_coefficient=0.01, random_state=0)
```

b) Fit

```python
t0 = time.time()
est_gp.fit(X_train, y_train)
print('Time to fit:', time.time() - t0, 'seconds')
```

        |   Population Average    |             Best Individual              |
    ---- ------------------------- ------------------------------------------ ----------
     Gen   Length          Fitness   Length          Fitness      OOB Fitness  Time Left
       0    48.81      2.34608e+06        7          59.4318          66.2518      2.37m
       1    50.93          11966.8        7          59.0484          69.7024      2.06m
       2    58.86          5952.23       43          13.1229          14.6923      2.05m
       3    58.18          9087.44       43          11.0273          12.0974      1.90m
       4    62.73          11650.9       97          10.4827          10.7773      1.87m
       5    75.55          9385.07       47          10.1926          11.7112      1.97m
       6    56.27          6290.84       27          10.0121          13.1662      1.51m
       7    49.54          8367.19       49          9.88528          12.5177      1.30m
       8    41.17          14176.4       49          9.89063          12.4696      1.13m
       9    36.73          5736.22       39          9.37762          7.83287     59.69s
      10    33.50          15078.7       39          8.84874          12.5928     51.07s
      11    32.06          2281.44       43          8.93255          11.8385     45.60s
      12    35.45          1168.12       37          8.52854          10.3485     40.71s
      13    42.58          891.934       77          8.13283          10.0993     37.35s
      14    43.66          860.597       51          7.46917          8.47483     31.67s
      15    44.09          674.422       51          7.35701           9.4842     25.23s
      16    45.16          44730.3       55          7.17509          6.68837     19.05s
      17    48.63           108859       57          6.87769          7.53629     13.35s
      18    52.58          1407.36       49          6.37549          6.32367      6.75s
      19    54.80          822.332       53          5.82337          5.21556      0.00s
    Time to fit: 132.9189989566803 seconds

c) Prediction

```python
t0 = time.time()
y_gp2 = est_gp.predict(X_test)
print('Time to predict:', time.time() - t0, 'seconds')
```

    Time to predict: 0.002447843551635742 seconds

d) Score

```python
score_gp2 = est_gp.score(X_test, y_test)
print('R2:', score_gp2)
```

    R2: 0.9995898043073126

##### 4.4.2.1 **Visualizing the symbolic function**

a) Equation

```python
next_e = sympify(str(est_gp._program), locals=converter)
next_e
```

$\displaystyle 0.373 X_{0} \left(X_{0}^{2} - \frac{32.258064516129 \left(0.746 X_{0} - 1.072375\right)}{X_{0}^{2} \left(0.031 X_{0}^{2} + 0.001299272 + \frac{0.214}{X_{0}}\right)}\right) + 100.762601626016 X_{0} - 0.36909918699187$

b) Plot

```python
fig = plt.figure(constrained_layout=False, dpi=100)
gs = fig.add_gridspec(nrows=7, ncols=1)
f_ax1 = fig.add_subplot(gs[0:5,0])
plt.title('H (z) comparison')
plt.scatter(X_test, y_test, label = 'True function')
plt.scatter(X_test, y_gp2, marker = 'v', s = 10, label = 'Symbolic function')
plt.legend()
plt.ylabel('H (z)')
f_ax2 = fig.add_subplot(gs[5:7, 0])
plt.scatter(X_test, 1. - y_gp2/y_test, marker = 's', s = 10)
plt.ylabel('Residual')
plt.xlabel('z')
```

c) Tree

```python
dot_data = est_gp._program.export_graphviz()
graph = graphviz.Source(dot_data)
graph.render('images/ex2', format='png', cleanup=True)
graph
```

### 4.5 Comparing GPlearn to traditional ML approaches

#### 4.5.1 Decision Tree Regressor

a) Model and fit

```python
est_tree = DecisionTreeRegressor(max_depth=5)
t0 = time.time()
est_tree.fit(X_train, y_train)
print('Time to fit:', time.time() - t0, 'seconds')
```

    Time to fit: 0.014159202575683594 seconds

b) Prediction and score

```python
t0 = time.time()
y_tree = est_tree.predict(X_test)
print('Time to predict:', time.time() - t0, 'seconds')
```

    Time to predict: 0.0028460025787353516 seconds

```python
#Score
score_tree = est_tree.score(X_test, y_test)
print('DT:', score_tree)
```

    DT: 0.998831678221913

c) Plot

```python
fig = plt.figure(constrained_layout=False, dpi=100)
gs = fig.add_gridspec(nrows=7, ncols=1)
f_ax1 = fig.add_subplot(gs[0:5,0])
plt.title('H (z) comparison')
plt.scatter(X_test, y_test, label = 'True function')
plt.scatter(X_test, y_tree, marker = 'v', s = 10, label = 'Decision Tree')
plt.legend()
plt.ylabel('H (z)')
f_ax2 = fig.add_subplot(gs[5:7, 0])
plt.scatter(X_test, 1. - y_tree/y_test, marker = 's', s = 10)
plt.ylabel('Residual')
plt.xlabel('z')
```

#### 4.5.2 Random Forest Regressor

a) Model and Fit

```python
est_rf = RandomForestRegressor(n_estimators=100,max_depth=5)
t0 = time.time()
est_rf.fit(X_train, y_train)
print('Time to fit:', time.time() - t0, 'seconds')
```

    Time to fit: 0.22199487686157227 seconds

b) Prediction and Score

```python
t0 = time.time()
y_rf = est_rf.predict(X_test)
print('Time to predict:', time.time() - t0, 'seconds')
```

    Time to predict: 0.023862361907958984 seconds

```python
score_rf = est_rf.score(X_test, y_test)
print('RF:', score_rf)
```

    RF: 0.9998512936885466

c) Plot

```python
#Plot
fig = plt.figure(constrained_layout=False, dpi=100)
gs = fig.add_gridspec(nrows=7, ncols=1)
f_ax1 = fig.add_subplot(gs[0:5,0])
plt.title('H (z) comparison')
plt.scatter(X_test, y_test, label = 'True function')
plt.scatter(X_test, y_rf, marker = 'v', s = 10, label = 'Random Forest')
plt.legend()
plt.ylabel('H (z)')
f_ax2 = fig.add_subplot(gs[5:7, 0])
plt.scatter(X_test, 1. - y_rf/y_test, marker = 's', s = 10)
plt.ylabel('Residual')
plt.xlabel('z')
```

#### 4.5.3 All together

```python
fig = plt.figure(figsize=(12, 10))
for i, (y, score, title) in enumerate([(y_test, None, "Ground Truth"),
                                       (y_gp, score_gp2, r"Symbolic Regressor: Score $= {:.4f}$".format(score_gp2)),
                                       (y_tree, score_tree, r"DecisionTreeRegressor: Score $= {:.4f}$".format(score_tree)),
                                       (y_rf, score_rf, r"RandomForestRegressor: $Score = {:.4f}$".format(score_rf))]):
    ax = fig.add_subplot(2, 2, i+1)
    points = ax.scatter(X, y_true, alpha=0.5)
    test = ax.scatter(X_test,y, alpha=0.5)
    plt.title(title)
```

## 5. Symbolic Classifier

The **SymbolicClassifier** works in exactly the same way as the **SymbolicRegressor** in how the evolution takes place. The only _difference_ is that the **output** of the program is _transformed_ through a **sigmoid function** in order to transform the numeric output into probabilities of each class.

In essence this means that a negative output of a function means that the program is predicting one class, and a positive output predicts the other.

### 5.1 Importing libraries

```python
from gplearn.genetic import SymbolicClassifier
from sklearn.datasets import load_boston
from sklearn.metrics import roc_auc_score
from sklearn.metrics import roc_curve
```

### 5.2 Wisconsin breast cancer

a) Loading:

```python
from sklearn.datasets import load_breast_cancer
cancer = load_breast_cancer()
```

b) Description:

```python
print(cancer.DESCR)
```

    .. _breast_cancer_dataset:

    Breast cancer wisconsin (diagnostic) dataset
    --------------------------------------------

    **Data Set Characteristics:**

        :Number of Instances: 569

        :Number of Attributes: 30 numeric, predictive attributes and the class

        :Attribute Information:
            - radius (mean of distances from center to points on the perimeter)
            - texture (standard deviation of gray-scale values)
            - perimeter
            - area
            - smoothness (local variation in radius lengths)
            - compactness (perimeter^2 / area - 1.0)
            - concavity (severity of concave portions of the contour)
            - concave points (number of concave portions of the contour)
            - symmetry
            - fractal dimension ("coastline approximation" - 1)

            The mean, standard error, and "worst" or largest (mean of the three
            worst/largest values) of these features were computed for each image,
            resulting in 30 features.  For instance, field 0 is Mean Radius, field
            10 is Radius SE, field 20 is Worst Radius.

            - class:
                    - WDBC-Malignant
                    - WDBC-Benign

        :Summary Statistics:

        ===================================== ====== ======
                                               Min    Max
        ===================================== ====== ======
        radius (mean):                        6.981  28.11
        texture (mean):                       9.71   39.28
        perimeter (mean):                     43.79  188.5
        area (mean):                          143.5  2501.0
        smoothness (mean):                    0.053  0.163
        compactness (mean):                   0.019  0.345
        concavity (mean):                     0.0    0.427
        concave points (mean):                0.0    0.201
        symmetry (mean):                      0.106  0.304
        fractal dimension (mean):             0.05   0.097
        radius (standard error):              0.112  2.873
        texture (standard error):             0.36   4.885
        perimeter (standard error):           0.757  21.98
        area (standard error):                6.802  542.2
        smoothness (standard error):          0.002  0.031
        compactness (standard error):         0.002  0.135
        concavity (standard error):           0.0    0.396
        concave points (standard error):      0.0    0.053
        symmetry (standard error):            0.008  0.079
        fractal dimension (standard error):   0.001  0.03
        radius (worst):                       7.93   36.04
        texture (worst):                      12.02  49.54
        perimeter (worst):                    50.41  251.2
        area (worst):                         185.2  4254.0
        smoothness (worst):                   0.071  0.223
        compactness (worst):                  0.027  1.058
        concavity (worst):                    0.0    1.252
        concave points (worst):               0.0    0.291
        symmetry (worst):                     0.156  0.664
        fractal dimension (worst):            0.055  0.208
        ===================================== ====== ======

        :Missing Attribute Values: None

        :Class Distribution: 212 - Malignant, 357 - Benign

        :Creator:  Dr. William H. Wolberg, W. Nick Street, Olvi L. Mangasarian

        :Donor: Nick Street

        :Date: November, 1995

    This is a copy of UCI ML Breast Cancer Wisconsin (Diagnostic) datasets.
    https://goo.gl/U2Uwz2

    Features are computed from a digitized image of a fine needle
    aspirate (FNA) of a breast mass.  They describe
    characteristics of the cell nuclei present in the image.

    Separating plane described above was obtained using
    Multisurface Method-Tree (MSM-T) [K. P. Bennett, "Decision Tree
    Construction Via Linear Programming." Proceedings of the 4th
    Midwest Artificial Intelligence and Cognitive Science Society,
    pp. 97-101, 1992], a classification method which uses linear
    programming to construct a decision tree.  Relevant features
    were selected using an exhaustive search in the space of 1-4
    features and 1-3 separating planes.

    The actual linear program used to obtain the separating plane
    in the 3-dimensional space is that described in:
    [K. P. Bennett and O. L. Mangasarian: "Robust Linear
    Programming Discrimination of Two Linearly Inseparable Sets",
    Optimization Methods and Software 1, 1992, 23-34].

    This database is also available through the UW CS ftp server:

    ftp ftp.cs.wisc.edu
    cd math-prog/cpo-dataset/machine-learn/WDBC/

    .. topic:: References

       - W.N. Street, W.H. Wolberg and O.L. Mangasarian. Nuclear feature extraction
         for breast tumor diagnosis. IS&T/SPIE 1993 International Symposium on
         Electronic Imaging: Science and Technology, volume 1905, pages 861-870,
         San Jose, CA, 1993.
       - O.L. Mangasarian, W.N. Street and W.H. Wolberg. Breast cancer diagnosis and
         prognosis via linear programming. Operations Research, 43(4), pages 570-577,
         July-August 1995.
       - W.H. Wolberg, W.N. Street, and O.L. Mangasarian. Machine learning techniques
         to diagnose breast cancer from fine-needle aspirates. Cancer Letters 77 (1994)
         163-171.

### 5.3 Data pre-processing

#### 5.5.1 Shuffling data

```python
rng = check_random_state(0)
perm = rng.permutation(cancer.target.size)
cancer.data = cancer.data[perm]
cancer.target = cancer.target[perm]
```

#### 5.5.2 Spliting data

```python
X_train = cancer.data[:400]
y_train = cancer.target[:400]
X_test = cancer.data[400:]
y_test = cancer.target[400:]
X_train.shape, y_train.shape, X_test.shape, y_test.shape
```

    ((400, 30), (400,), (169, 30), (169,))

### 5.3 Classifying

```python
est = SymbolicClassifier(parsimony_coefficient=.01,
                         feature_names=cancer.feature_names,
                         random_state=1)
```

#### 5.3.1 Fit

```python
t0 = time.time()
est.fit(X_train, y_train)
print('Time to fit:', time.time() - t0, 'seconds')
```

    Time to fit: 16.150514125823975 seconds

#### 5.3.2 Predicting

```python
y_gpclas = est.predict_proba(X_test)[:,1]
y_gpclas.shape
```

    (169,)

#### 5.3.3 Scoring

```python
score = roc_auc_score(y_test, y_gpclas)
score
```

    0.9693786982248521

### 5.4 Visualizing

a) ROC curve:

```python
cutoff = 0.7
y_test_classes = np.zeros_like(y_test)
y_test_classes[y_test > cutoff] = 1

cutoff = 0.7
y_gpclas_classes = np.zeros_like(y_gpclas)
y_gpclas_classes[y_gpclas > cutoff] = 1
```

```python
fpr, tpr, thresholds = roc_curve(y_test_classes, y_gpclas_classes)
```

```python
plt.figure(dpi=100)
plt.plot(fpr, tpr, '-o', label = r'ROC curve ( area = {:.4})'.format(score))
plt.plot([0, 1], [0, 1], linestyle='--')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('ROC curve')
plt.legend()
```

b) Tree:

```python
dot_data = est._program.export_graphviz()
graph = graphviz.Source(dot_data)
graph
```

## 6. References

- [Python symbolic Regression with GPlearn](https://medium.com/analytics-vidhya/python-symbolic-regression-with-gplearn-cbc24dbbc271)
- [GPlearn documentation](https://gplearn.readthedocs.io/en/stable/index.html)

```python

```


Computational Seismology

Finite Differences - Taylor Operators and Numerical Differentiation

---

This notebook is part of the supplementary material
to [Computational Seismology: A Practical Introduction](https://global.oup.com/academic/product/computational-seismology-9780198717416?cc=de&lang=en&#),
Oxford University Press, 2016.

##### Authors:
* Lion Krischer ([@krischer](https://github.com/krischer))
* Heiner Igel ([@heinerigel](https://github.com/heinerigel))
---

This exercise covers the following aspects:

* Use the Taylor series to calculate finite-difference operators
* Learn how to define high-order finite-difference operators
* Demonstrate the accuracy improvement by testing the operators on (arbitrary) functions

---

## Basic Equations

The Taylor expansion of $f(x + dx)$ around $x$ is defined as

$$
f(x+dx)=\sum_{n=0}^\infty \frac{f^{(n)}(x)}{n!}dx^{n}
$$

Finite-difference operators can be calculated by seeking weights (here: $a$, $b$, $c$) with which function values have to be multiplied to obtain an interpolation or a derivative. Example:

$$
\begin{align}
a ~ f(x + dx) & \ = \ a ~ \left[ ~  f(x) +  f^{'} (x) dx +  \frac{1}{2!}  f^{''} (x) dx^2   + \dotsc  ~ \right] \\
b ~ f(x) & \  = \ b ~ \left[ ~  f(x)  ~ \right] \\
c ~ f(x - dx) & \ = \ c ~ \left[ ~  f(x) -  f^{'} (x) dx +  \frac{1}{2!}  f^{''} (x) dx^2   - \dotsc  ~ \right]
\end{align}
$$

This can  be expressed in matrix form by comparing coefficients, here seeking a 2nd derivative

$$
\begin{align}
&a  ~~+~~            ~~~~b           &+~~  c & = & 0 \\
&a  ~~\phantom{+}~~ \phantom{b}  &-~~  c & = & 0 \\
&a  ~~\phantom{+}~~ \phantom{b}  &+~~  c & = & \frac{2!}{\mathrm{d}x^2}
\end{align}
$$

which leads to

$$
\begin{pmatrix}
  1 & 1 & 1 \\
  1 & 0 & -1 \\
  1 & 0 & 1
 \end{pmatrix}
\begin{pmatrix}
  a\\
 b \\
 c
 \end{pmatrix}
 =
 \begin{pmatrix}
  0\\
 0 \\
 \frac{2!}{dx^2}
 \end{pmatrix}
$$

and using matrix inversion we obtain

$$
\begin{pmatrix}
a \\
b\\
c
\end{pmatrix}
=
\begin{pmatrix}
\frac{1}{2 \mathrm{d}x^2} \\
 - \frac{2}{2 \mathrm{d}x^2} \\
  \frac{1}{2 \mathrm{d}x^2}
\end{pmatrix}
$$

This is the the well known 3-point operator for the 2nd derivative. This can easily be generalized to higher point operators and higher order derivatives. Below you will find a routine that initializes the system matrix and solves for the Taylor operator.

---

## Calculating the Taylor operator

The subroutine `central_difference_coefficients()` initalizes the system matrix and solves for the difference weights assuming $dx=1$. It calculates the centered differences using an arbitrary number of coefficients, also for higher derivatives. The weights are defined at $x\pm i dx$ and $i=0,..,(nop-1)/2$, where $nop$ is the length of the operator. Careful! Because it is centered $nop$ has to be an odd number (3,5,...)!

It returns a central finite difference stencil (a vector of length $nop$)  for the `n`th derivative.

```python
import math
import numpy as np

def central_difference_coefficients(nop, n):
    """
    Calculate the central finite difference stencil for an arbitrary number
    of points and an arbitrary order derivative.

    :param nop: The number of points for the stencil. Must be
        an odd number.
    :param n: The derivative order. Must be a positive number.
    """
    m = np.zeros((nop, nop))
    for i in range(nop):
        for j in range(nop):
            dx = j - nop // 2
            m[i, j] = dx ** i

    s = np.zeros(nop)
    s[n] = math.factorial(n)

    # The following statement return oper = inv(m) s
    oper = np.linalg.solve(m, s)
    # Calculate operator
    return oper
```

---

## Plot Taylor operators

Investigate graphically the Taylor operators. Increase $nop$ for the first $n=1$ or higher order derivatives. Discuss the results and try to understand the interpolation operator (derivative order $n=0$).

```python
%matplotlib inline

# Some necessary initialization
import matplotlib.pyplot as plt
plt.style.use('ggplot')

# Give length of operator (odd)
nop = 13
# Give order of derivative (0 - interpolation, 1 - first derivative, 2 - second derivative)
n = 1

# Get operator from routine 'central_difference_coefficients'
oper = central_difference_coefficients(nop, n)

# Initialize location of operator
x = np.linspace(-(nop - 1) / 2, (nop - 1) / 2, nop)

# Simple plot with operator
plt.figure(figsize=(10, 4))
plt.plot(x, oper)
plt.show()
```

---

## Accuracy of centered finite differences

Define a simple function, e.g. sine, cosine, gaussian, some polynomial, ..., differentiate it using the stencils returned from the above function and compare the result to the analytically calculated derivative. Keep the physical domain (e.g., 3$\pi$) constant but change 1) the number of points nx (i.e., change dx) or 2) change the wavenumber k. In both cases you change the number of grid points per wavelength. Calculate the error of the solution (in the center of the domain) and look at the error as a function of grid points per wavelength.

```python
# Let us initialize space between 0 and 10 pi with 10000 points
# Note: here you can be creative, change the function as you like,
# as long as you know the analytical derivative
nx = 100        # Number of points in x
x = np.linspace(0, 3 * np.pi, nx)

# We initialite a function f as a sin curve with wavenumber k
k = 3
f = np.sin(k * x)

# The analytical derivative is given by
dfa = k * np.cos(k * x)

# We need to know the grid spacing dx
dx = x[2] - x[1]

# Now let's get the finite difference operator with paramters
nop = 5 # Length of operator
n = 1   #  1 st derivative
oper = central_difference_coefficients(nop, n)

# Calculate the derivative of f(x) by applying the finite-difference operator
df = np.zeros(nx)
# Loop over the central points where we calculate the derivatives
for i in range(nop // 2, nx - nop // 2):  # Avoid the boundaries
    # Loop over operator
    for j in range(0, nop):
       df[i] = df[i] + oper[j] * f[i - (nop - 1) // 2 + j]

# Scale by dx.
df /= dx ** n

plt.plot(df, label="Numerical derivative", lw=2)
plt.plot(dfa, ls="--",label="Analytic derivative")
plt.legend()
plt.show()
```

notebooks/Computational Seismology/The Finite-Difference Method/fd_taylor_operators.ipynb


# CSE 555 - Introduction to pattern recognition
## Problem Set 1: Bayesian Decision Theory

Dataloader to read the MNIST dataset. The dataset is read and is flattened. Hence each 28x28 image is now a vector of length 784.

```python
import numpy as np

def load_mnist(kind='train'):
    import os
    import gzip

    """Load MNIST data from `path`"""
    labels_path = ('%s-labels-idx1-ubyte.gz' % kind)
    images_path = ('%s-images-idx3-ubyte.gz' % kind)

    with gzip.open(labels_path, 'rb') as lbpath:
        labels = np.frombuffer(lbpath.read(), dtype=np.uint8,
                               offset=8)

    with gzip.open(images_path, 'rb') as imgpath:
        images = np.frombuffer(imgpath.read(), dtype=np.uint8,
                               offset=16).reshape(len(labels), 784)

    return images, labels

x_train, y_train = load_mnist(kind='train')
x_test, y_test = load_mnist(kind='t10k')
```

Segregating dataset into a dictionary consisting of 0-9 digits as key and the list of image pixels as values

```python
train_data_dict = dict()
for i in range(len(y_train)):
    if(y_train[i] in train_data_dict):
        train_data_dict[y_train[i]].append([x_train[i]])
    else:
        train_data_dict[y_train[i]] = list()
        train_data_dict[y_train[i]].append([x_train[i]])
```

### TASK 1:
Calculating the mean and standard deviation for each of the digit individially. Saving and visualizing the "mean digits" and "standard deviation digits" in current directory and by plotting them inline.

```python
import cv2
import os
import matplotlib.pyplot as plt

if not os.path.exists('./mean_digits'):
    os.makedirs('./mean_digits')
if not os.path.exists('./standard_deviation_digits'):
    os.makedirs('./standard_deviation_digits')
if not os.path.exists('./output_subplots'):
    os.makedirs('./output_subplots')

mean_list = list()
sd_list = list()

for i in range(10):
    fig, a =  plt.subplots(1, 2)
    mean_list.append(np.mean(np.asarray(train_data_dict[i]), axis=(0)).transpose())
    cv2.imwrite('./mean_digits/mean_'+str(i)+'.png', np.asarray(mean_list[i]).reshape(28, 28))
    sd_list.append(np.std(np.asarray(train_data_dict[i]), axis=(0)).transpose())
    cv2.imwrite('./standard_deviation_digits/sd_'+str(i)+'.png', np.asarray(sd_list[i]).reshape(28, 28))
    a[0].imshow(np.asarray(mean_list[i]).reshape(28, 28), cmap = 'gray')
    a[0].set_title('Mean of '+str(i))
    a[0].axis('off')
    a[1].imshow(np.asarray(sd_list[i]).reshape(28, 28), cmap = 'gray')
    a[1].set_title('Standard deviation of '+str(i))
    a[1].axis('off')
    plt.savefig('./output_subplots/mean_sd_'+str(i)+'.png', dpi=300)
    plt.show()
```

Setting up the dataset as pandas data series for the processing of task 2.

```python
import pandas as pd

train_data_frame = pd.DataFrame(x_train,y_train)
train_data_mean = list()
train_data_sd = list()
for i in range(10):
    mean = train_data_frame.loc[i].sum()/len(train_data_frame.loc[i])
    sd = np.sqrt(((train_data_frame.loc[i]-mean)**2).sum()/len(train_data_frame.loc[i]))
    train_data_mean.append(mean)
    train_data_sd.append(sd)
```

### Task 2:
Calculating the covariance and the inverse matrix as it is required in the implementation of Bayesian decision rule.

```python
covariance = list()
inverse = list()

for i in range(len(train_data_sd)):
    for j in range(len(train_data_sd[0])):
        train_data_sd[i][j] += 0.001

for i in range(len(train_data_sd)):
    a = np.diag(train_data_sd[i]**2)
    covariance.append(a)
    b = np.linalg.inv(covariance[i])
    inverse.append(b)
```

Implementation of Bayesian decision rule.

* Guassian Distribution - The LDA function is derived by the above equation.

\begin{align}
p(x)=\frac{1}{(2π)^{d/2}|\sum|^{1/2}}exp(−\frac{1}{2}(x−\mu)^{′}\sum^{−1}(x−\mu))
\end{align}

 * Then we take Log Likelyhood of the above equation to get the discriminant function.

 \begin{align}
\boldsymbol{g_{i}(x) = \log p(x|\omega_i) + \log P(\omega_i)}
\end{align}
\begin{align}
\newline = -\frac{1}{2}(x - \mu_{i})^{-1} \sum^{-1}(x - \mu_{i}) - \frac{d}{2} \log (2 \pi) -\frac{1}{2} \log |\sum_i| + \log P(\omega_i)
\end{align}

 * Now we take the case 3 of the LDA (arbitrary covariances)
   Case 3 is defined by:

\begin{align}
\boldsymbol{g_i = x^{t} W_i x + N_{i}^{t} x + B_{i0}}
\end{align}
\begin{align}
\newline
where, W_i = - \frac{1}{2} \sum_{i}^{-1} , N_{i} = \sum_{i}^{-1} \mu_{i} and B_{i0} = - \frac{1}{2} \mu_{i}^{t} \sum_{i}^{-1} \mu_{i} + ln P(w_i) - \frac{1}{2} ln |\sum_{i}|
\end{align}

```python
W = list()
N = list()
B = list()
g = list()
y_pred = list()

inverse = np.asarray(inverse)
train_data_mean = np.asarray(train_data_mean)
for i in range(len(train_data_sd)):
    W.append(inverse[i]*(-0.5))
    N.append(np.matmul(inverse[i], train_data_mean[i]))
    b1 = np.matmul(np.matmul(train_data_mean[i].T, inverse[i]), train_data_mean[i])
    B.append((-0.5)*b1)

N = np.asarray(N)
for i in range(len(x_test)):
    g = list()
    for j in range(len(train_data_sd)):
        g.append(B[j] + np.matmul(N[j].T, x_test[i]) + np.matmul(np.matmul(x_test[i].T, W[j]), x_test[i]))

    y_pred.append(np.argmax(g))
```

Calculation of accuracy using 0-1 loss function

```python
import sklearn.metrics
import seaborn as sn

print("Accuracy: ", (100-sklearn.metrics.zero_one_loss(y_test, y_pred)*100))
print("Confusion matrix:")
df_cm = pd.DataFrame(sklearn.metrics.confusion_matrix(y_test, y_pred), index = [i for i in "0123456789"],
                  columns = [i for i in "0123456789"])
sn.heatmap(df_cm, annot=True, fmt='d')
sn.set_xlabel('y_pred')
sn.set_ylabel('y_pred')
```

### Why it doesn't perform as good as many other methods on LeCun's web page?
1) One main reason is that the Discriminant Analysis makes the label prediction on the basis of similarity of the mean and standard deviation of images and not individual features themselves.
2) Another reason is the assumption that the feature vectors in each category have Gaussian distribution

```python

```


```python
from sympy import *
from sympy.geometry import *
from sympy.plotting.plot import Plot
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

def plot_segments(*segs, **kwargs):
    for seg in segs:
        x1,y1 = seg.p1
        x2,y2 = seg.p2
        plt.plot([x1,x2],[y1,y2], **kwargs)
```

```python
coupler_pos1 = Segment((0,100), (100,100))
coupler_pos2 = coupler_pos1.translate(200,-200)
coupler_pos2 = coupler_pos2.rotate(pi/6, coupler_pos2.p1)
segi = Segment(coupler_pos1.p1, coupler_pos2.p1)
segj = Segment(coupler_pos1.p2, coupler_pos2.p2)

iperp = segi.perpendicular_line(segi.midpoint)
jperp = segj.perpendicular_line(segj.midpoint)

pini = iperp.p2
pinj = jperp.p2

link2_pos1 = Segment(coupler_pos1.p1, pini)
link4_pos1 = Segment(coupler_pos1.p2, pinj)

link2_pos2 = Segment(coupler_pos2.p1, pini)
link4_pos2 = Segment(coupler_pos2.p2, pinj)
```

```python
plot_segments(link2_pos1, link4_pos1, coupler_pos1, color="r")
plot_segments(link2_pos2, link4_pos2, coupler_pos2, color="b")
plot_segments(iperp, jperp, segi, segj, ls="--", color="#aaaaaa")
plt.axis('equal')
```

```python

```


# Nonlinear Regression in [`SciPy`](https://docs.scipy.org/doc/scipy/reference/) and [R](https://www.r-project.org/about.html)

We often need to find a function $y=f(x,\beta)$ of variable $x$ and $p$ unknown parameters $\beta$ which fits a given set of $n$ predictor, {$x_1,...,x_n$}, and response, {$y_1,...,y_n$}, values.

The best fit can be obtained by minimizing the residual sums of squares (RSS) with respect to $\beta$,
\begin{equation}
\mathrm{RSS}(\beta) = \sum_{n=1}^{\infty} \left[ y_i - f(x_i,\beta) \right ] ^2.
\end{equation}
For nonlinear functions, numerical optimization methods are needed to minimize RSS and find the best-fit parameters, $\hat{\beta}$.

The most common numerical optimization method for nonlinear regression is the [Gauss-Newton method](https://en.wikipedia.org/wiki/Gauss%E2%80%93Newton_algorithm).

The assumption in nonlinear regression is that the measurement errors are standard normally distributed.

A summary measure of the model fit is the residual variance which is defined as the ratio of the minimum of RSS devided by the number of degrees of freedom:
\begin{equation}
\chi_R^2 = \frac{\mathrm{RSS}(\hat{\beta})}{n-p}.
\end{equation}
Sometimes, we have several curves that we want to fit with the same function $f$. These curves have some parameters in common that we want to be determined on all the available data.

In this Notebook, we show how to perform least-sqares fits in `scipy` and `R`.

```python
from scipy.optimize import least_squares
from scipy.optimize import curve_fit
import pandas as pd
import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt
plt.rcParams.update({'figure.figsize': [3.5, 3], 'font.size':8, 'legend.fontsize':8, 'figure.dpi': 150,
                     'legend.handlelength':1, 'legend.frameon':False,
                     'xtick.major.size':6,'ytick.major.size':6,
                     'xtick.minor.size':3,'ytick.minor.size':3,
                     'xtick.direction':'out','ytick.direction':'out',
                     'xtick.labelsize':8,'ytick.labelsize':8,
                     'xtick.major.pad':2,'ytick.major.pad':2,
                     'axes.linewidth':1,'axes.labelsize':10,})
from cryptography.fernet import Fernet
import base64
def encrypt(string, key):
    keygen = lambda x : base64.urlsafe_b64encode(x.encode() + b' '*(32 - len(x)))
    cipher = Fernet(keygen(key))
    return cipher.encrypt(string.encode())
def decrypt(string, key):
    keygen = lambda x : base64.urlsafe_b64encode(x.encode() + b' '*(32 - len(x)))
    cipher = Fernet(keygen(key))
    return print(cipher.decrypt(string.encode()).decode())
```

## Case Study: Time-Resolved Fluorescence Spectroscopy

### SDS Micelles in the Presence of a Hydrophobic Fluorophore and a Fluorescence Quencher

- **The micelle:**

    Sodium dodecyl sulfate (SDS) is an anionic surfactant with a hydrophobic tail of 12 carbon atoms and a hydrophilic sulphate head group. Due to its amphipatic nature, SDS self-assembles into micelles in aqueous solution. Micelle formation is a cooperative process which occurs in a narrow range of surfactant concentrations around the critical micellar concentration (CMC).
At concentrations below the CMC., surfactant molecules are mostly in the monomer state.
At the  (CMC), surfactant molecules form micelles with a well-defined aggregation number. The size distribution of the aggregates is rather narrow and depends on the relative dimensions of the head and tail groups.
Above the CMC, the number of micelles increases while the size of the micelles stays the same.

- **The fluorophore:**
Pyrene is a hydrophobic fluorescent molecule.
In an aqueous SDS solution at [SDS]>CMC, pyrene tends to partition into the hydrophobic interior of the micelles.

- **The quencher:**
Benzophenone is a hydrophobic compound which also tend to be in the micelles where it quenches the fluorescence of pyrene (it makes it decay faster).

- **The experiment:**
We measure the number of photons emitted by pyrene molecules in SDS micelles for increasing time intervals after the excitation pulse. The fluorescence intensity decay of pyrene following an excitation pulse is given by

\begin{equation}
F(t)=F(0)\exp(-t/\tau_0)
\end{equation}

where $\tau_0$ is the lifetime of the fluorophore.
In the presence of $j$ benzophenone molecules per micelle, the lifetime of pyrene decreases to
\begin{equation}
\tau_j = \frac{\tau_0}{1 + j k_q \tau_0}.
\end{equation}
where $k_q$ is the quenching constant.

The distribution of quenchers in the micelles follows Poisson's statistics, therefore the fraction of micelles containing $j$ benzophenone molecules is given by
\begin{equation}
P(j) = \frac{m^j}{j!}\exp{(-m)}
\end{equation}
where $m$ is the average number of benzophenone molecules per micelle.

The fluorescence intensity decay of pyrene in the presence of benzophenone is given by the weighted average over the difstribution of micelles with different numbers of quenchers

\begin{equation}
F(t)=F(0)\sum_{j=0}^{\infty} P(j) \exp{(-t/\tau_j)}=F(0)\exp\left\{-t/\tau_0-m\left [1-\exp{(-k_qt)}\right ]\right\},
\end{equation}

where we have used the converging series
\begin{equation}
\sum_{j=0}^{\infty} \frac{x^j}{j!} = \exp{(x)}.
\end{equation}

We used a solution 70 mM SDS and 10 $\mu$M pyrene in 5mM phosphate buffer at pH 7.4. The emission wavelength of the spectrometer is set to 391 nm, corresponding to the maximum fluorescence intensity for pyrene.
- First, we measure the intensity decay of the solution in the absence of quencher. A least-squares fit on this curve yields $F(0)$ and the pyrene lifetime, $\tau_0$.
- Then, we measure intensity decays for increasing concentrations of benzophenone, _i.e._, $[$quencher$]$ $=$ 0.1, 0.2, 0.5, 1 e 1.5 mM.
We perform a global fit on these five curves. Each corve has a different average number of quencher molecules per micelle, $m$, and common parameters $F(0)$, $\tau_0$, and $k_q$.

Since the excitation pulse occurs in a shorter time scale compared to the decay of the fluorescence signal, we disregard the width of the excitation pulse in our analysis. In the least-squares fitting procedures, we weight each photon count, $F(t)$, by its error which is simply $\sqrt{F(t)}$ because fluorescence counts follow Poisson's statistics.

### The Data

The data are tab-separated values (TSV) files (collected at [Prof. L. Stella lab](http://www.stc.uniroma2.it/physchem/stella.html))

```python
%cat 'data/fluorescence/quencher_0.0.txt'
```

```python
%ls 'data/fluorescence'
```

Let's plot the intensity decays for increasing concentrations of quencher.

The intensity signal decays faster with increasing concentration of benzophenone.

```python
concs = [0.0,0.2,0.5,1.0,1.5]
for conc in concs:
    t,F = np.loadtxt('data/fluorescence/quencher_{:1.1f}.txt'.format(conc), unpack=True, skiprows=9)
    plt.errorbar(t, F, np.sqrt(F), lw=0, marker='o', label='{:1.1f}'.format(conc)+' mM',
                 ms=2, elinewidth=1., capsize=2, capthick=1., mfc='w')
plt.legend(title='[quencher]',labelspacing=.7,loc=1)
plt.ylabel('$F(t)$  /  counts'); plt.xlabel('$t$  /  ns');
```

Here, we use boolean masking to select the data points after the excitation pulse and we shift the x values so that they measure the time intervals from the excitation pulse.

```python
for conc in concs:
    d = np.loadtxt('data/fluorescence/quencher_{:1.1f}.txt'.format(conc), unpack=False, skiprows=9)
    d = d[ d[:,0] > d[d[:,1]==d[:,1].max(), 0] ] # points before the intesity max are discarded
    plt.errorbar(d[:,0]-d[0,0], d[:,1], np.sqrt(d[:,1]), lw=0, marker='o', label='{:1.1f}'.format(conc)+' mM',
                 ms=2, elinewidth=1., capsize=2, capthick=1., mfc='w')
plt.legend(title='[quencher]',labelspacing=.7)
plt.ylabel('$F(t)$ / counts'); plt.xlabel('$t$ / ns'); plt.show()
```

A convenient way to save data sets to file is to use [pickling](https://docs.python.org/3/library/pickle.html). We create a  `pandas` DataFrame and we save it as a binary file and also as a csv file.

```python
data = pd.DataFrame()
for conc in concs:
    t,F = np.loadtxt('data/fluorescence/quencher_{:1.1f}.txt'.format(conc), unpack=True, skiprows=9)
    # points before the intesity max are discarded
    t = t[F.argmax():]
    F = F[F.argmax():]
    t = t - t[0] # t at the intesity max is t zero
    d = pd.DataFrame(data=np.c_[t,F,np.sqrt(F)],columns=['t','F','errF'])
    d['conc'] = conc
    d.set_index('conc',inplace=True)
    data = data.append(d)
data.to_pickle('data/fluorescence.p')
data.to_csv('data/fluorescence.csv')
data.head()
```

The curve for $[$quencher$]$=0 mM is fitted to a single exponential decay function using [`optimize.least_square`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.least_squares.html) which requires the following arguments:
- the function which computes the vector of residuals
- the initial parameters
- the data
- the lower and the upper bounds of the fitting parameters

`
least_squares(function_to_minimize, initial_params,
    [x, y, yerr], bounds=[(lower_bounds),(upper_bounds)])
`

`least_squares` returns an OptimizeResult object containig the best-fit parameters (`x`), the residuals calculated for the best-fit parameters (`fun`), RSS/2 (`cost`), and the modified Jacobian matrix, $J$, at the solution (`jac`), $J^T J$ is a [Gauss-Newton approximation of the Hessian](https://reference.wolfram.com/language/tutorial/UnconstrainedOptimizationGaussNewtonMethods.html) of the residual function.

Since we know that the error on the counts follows Poisson's distribution, we use the following [nonlinear regression model](https://link.springer.com/book/10.1007%2F978-0-387-09616-2)
\begin{equation}
y_i = f(x_i,\beta) + \frac{\epsilon_i}{\sqrt{w_i}},
\end{equation}
where the weight is the inverse of the error on the counts, $1 / \sqrt{y_i}$.

We minimize, with respect to $\beta$, the residual sums of squares (RSS) weighted by $y_i$:
\begin{equation}
\mathrm{RSS}(\beta) = \sum_{n=1}^{\infty} \frac{\left[ y_i - f(x_i,\beta) \right ] ^2}{y_i}.
\end{equation}
For further details read chapter 6 of [Nonlinear Regression with R](https://link.springer.com/book/10.1007%2F978-0-387-09616-2) by Christian Ritz and Jens Carl Streibig.

```python
# we define the exponential decay
def exp_decay(x, param):
    return param[0] * np.exp( - x / param[1] )

# we define the function that calculates the residuals
def residuals(param, *args):
    return ( args[1] - exp_decay(args[0], param) ) / args[2] # residuals weighted by the error on y
```

```python
# we create a list with the data
args = [data.loc[0.0][col] for col in ['t','F','errF']]

# we fit the data
res = least_squares(residuals, x0=[3000,150], args=args, bounds=[(6,150),(3500,200)])

# we plot the data
plt.errorbar(args[0],args[1],args[2],lw=0,marker='o',ms=1,elinewidth=1.,capsize=3,capthick=1.,
             color='k',alpha=0.3)

# we plot the fitting curve
plt.plot(args[0],exp_decay(args[0],res['x']),lw=3,color='r')

# we set the log scale for the y-axis
plt.yscale('log')
plt.ylabel('$\ln [F(t)]$  /  counts'); plt.xlabel('$t$  /  ns');
```

```python
print('Is one of the convergence criteria satisfied?', res.success)
```

```python
print('The best-fit parameters are',res.x[0],'and',res.x[1])
```

```python
from IPython import display

table = r"""| Parameter     | Value         | Error |
| :---------------- |:-------------:| :----:|
| $F(0)$ (counts)   | %.2f          | ?     |
| $\tau_0$ (ns)     | %.2f          | ?     |"""

display.Markdown(table % tuple(res.x) )
```

\begin{equation}
\chi_R^2 = \frac{\mathrm{RSS}(\hat{\beta})}{n-p}
\end{equation}

```python
print('The residual variance is', 2 * res.cost / (res.fun.size - res.x.size))
```

We can check whether the model is appropriate and we have obtained a good fit by comparing the Gaussian with $\sigma=1$ and $\mu=0$ with the histogram of the residuals at solution.

We can see that the distribution of the residuals is compatible with a normal distribution. Would the comparison be this favorable if we didn't weight the error by the square root of the counts?

```python
bins = np.linspace(res.fun.min(),res.fun.max()+1,100)
plt.plot(bins, np.exp(-bins*bins/2)/np.sqrt(2*np.pi), label='Gaussian', color='k')
plt.hist(res.fun, bins, lw=2, histtype='step', density=True, label='residuals', color='r')
plt.xlabel('Residuals'); plt.ylabel('Probability Distribution'); plt.legend();
```

The uncertainties on the best-fit parameters are estimated from the variance-covariance matrix.

Here is the implementation in the function [`optimize.curve_fit`)](https://github.com/scipy/scipy/blob/2526df72e5d4ca8bad6e2f4b3cbdfbc33e805865/scipy/optimize/minpack.py#L739) which uses `least_squares` under the hood.
Further information [here](https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.curve_fit.html#scipy.optimize.curve_fit) and [here](http://scipy-cookbook.readthedocs.io/items/FittingData.html).

```python
from scipy.linalg import svd
# copied from curve_fit
def error_analysis(res):
    rss = 2 * res.cost  # res.cost is half sum of squares!
    s_sq = rss / (res.fun.size - res.x.size) # residual variance or reduced chi squared
    # Do Moore-Penrose inverse discarding zero singular values.
    _, s, VT = svd(res.jac, full_matrices=False)
    threshold = np.finfo(float).eps * max(res.jac.shape) * s[0]
    s = s[s > threshold]
    VT = VT[:s.size]
    pcov = np.dot(VT.T / s**2, VT) * s_sq # variance-covariance matrix
    return np.sqrt(np.diag(pcov)), s_sq # errors on the params are the diagonal elements of pcov
```

```python
table = r"""| Parameter     | Value         | Error |
| :---------------- |:-------------:| :----:|
| $F(0)$ (counts)   | %.0f          | %.0f  |
| $\tau_0$ (ns)     | %.1f          | %.1f  |"""
```

```python
p_and_err = np.repeat(res.x, 2)
p_and_err[1::2] = error_analysis(res)[0]
print('Reduced chi squared:',error_analysis(res)[1])
display.Markdown(table % tuple(p_and_err) )
```

##  Task 1
- Fit the data for $[$quencher$]$ $=$ 0 using [`optimize.curve_fit`)](https://github.com/scipy/scipy/blob/2526df72e5d4ca8bad6e2f4b3cbdfbc33e805865/scipy/optimize/minpack.py#L739).
- Plot the data and the function with the best-fit parameters.
- View a markdown table with the best-fit parameters and their uncertainties.

```python
answer = 'gAAAAABcBDlaRCOfnjUcmToYnEdOciuqiLV8TVxoLmyI1YiwAU8Ux_kcXVgFb-AZ87LV3DggpValGCTdAZ3kfijtjcB-271i1d5gF-7tQzM_XqQpLoChhrOE31dym4nOheTkkMafgMtu5JTS2Xg2WD4ziYF1H4ri2E0zlV_yIdmbJ5X14f6caG3NFES5Y8FLuKBpJpuSYDoCOg3DsRH-_Lo3ZHpLHOe2iKNMyroVgdQ-W44SzRqucnkRv4bK0L3EUcBUHrW-fFhj6veqYTi464yVkNuLGQza0QzEGFvQu8QuBQMaggNw97dllsHpfSCoLh4sCwS6oe9IJLfxHBgusgwQnNcpahJcINmnFMQnRNsQn7bujSnRpEUeORZeQq2S_7OuKKZl2tpJrWtpyRrIophfTmf7fV7daORiXU2W9KAS0Ym0Kft10BfIp_XK7MsbDWHjc4CCj-U6SEx4Mn091ds8Qy-wHsTGxlAAnCLOY2irqVQYuYljzh7oq7iRWYk_sxGCddwSxMX2AsotclVz677gQWZHSHn-2lRVcL8xGQiattT2MicC0Gk2ukG2WdXs_cZHB3J7iBU2ZmMv00ZvYK7QWdEW-CCUhxKPb9daAhCURpXyatMKuZnP8KtCO8nJ9ITluNGFy-6dZ0FdplO9Y7wG7MHoXWJKoSlbXzMMZto5LK1AC_DLEM0ImzLyiQThwBFBpU0DuD1QRyEs4AzF1k3BS_ifpSZ-NXaF2j_pJneJkt7k64tEH7D1Ea9NsD1tLYmU2itKQwHh9BVdFRBJwYgBEoQ-oJAvh91gQxgJLi9gzU4I_kVEZnvnH-THrQ2xOAWdKsfCxXE5gBYNG5thCCsBllUUcB9h4jlVsuBCnx2u1hbrLq7glmWZOrfxUgMjvsYMi_6o5S0qW0XdFz0CjdPJ_E44CDSG30i-4XUE4z4AyrXsZP-8ViBY2vJmiGPNzLhiEv3c6URuS6Qns_EPvGwZQEnLDTeAtLY2owHeEva_gIGkT1U2-KK4CZXESBf9v0yv5ABptx3l_WhxqPIEzCbWvv9j_AU8f6-QT8eQR7a-akKrEaEqR9s_2LAAEN09nAmDHBcIJaKlfp-Zw00MG5UGBKERA4v1V74QoRNHunD3Cq-ZS8ckAiq9F3Tt4Hs0hLghSeujwLgsiQ95G8ktYKIyhnUxgXbwNdMcH5ulCEEgeeUVa6U3XadtVli3KUtKd3AaDGtlpoOZ9LqH6NSkI_mOk2BXiL9ZlVSM4hE_fz3gEt7fCJWd0VnxKGLiKuBNWsj7yc6dSZG6sdHwmVhGHmeY1pMXCN8lN-D3D1yOyzY5eYDtpkhGeJeNSC1Qbklx__ZuHUSbgX94g4tBL9XFnj4m6uSmfntk9KupnX3FWXV_oPVEyBbhEzU='

```

```python
decrypt(answer,)
```

## Interfacing Python and [R](https://www.r-project.org/about.html)

This part of the Notebook requires the packages r-essentials and rpy2. To install them, issue the following commands in the terminal

```bash
source activate LUcompute
conda install -c conda-forge r-essentials
conda install -c conda-forge rpy2
```

It is also possible to install rpy2 using pip

```python
!pip install rpy2
```

```python
We load the extension [rpy2](https://rpy2.readthedocs.io/en/version_2.8.x/)
```

```python
%load_ext rpy2.ipython
```

```python
X = args[0]
Y = args[1]
errY = args[2]
```

We send the data from Python to R

```python
%Rpush X Y errY
```

We plot the data in R

```r
%%R
plot(X, Y, ylim = range(c(Y-errY, Y+errY)), xlab='t  /  ns', ylab='ln [F(t)]  /  counts', log='y')
arrows(X, Y-errY, X, Y+errY, length=0.05, angle=90, code=3)
```

We create a data frame in R

```r
%%R
df <- data.frame('X' = X, 'Y' = Y)
```

We fit the data without weighting by the error

```r
%%R
fit1 <- nls(Y ~ F0 * exp( - X / tau0 ), data=df, start = list(F0 = 2000, tau0 = 100))
summary(fit1)
```

We weight by the error and we compare the results of the two approaches

```r
%%R
plot(Y ~ X, data=df, xlab='t  /  ns', ylab='ln [F(t)]  /  counts', log='y' )

fit2 <- nls(Y ~ F0 * exp( - X / tau0 ), data=df, start = list(F0 = 2000, tau0 = 100), weights=1/Y)

curve(predict(fit2, newdata = data.frame(X=x)), add=TRUE, lwd=4, col='red')

curve(predict(fit1, newdata = data.frame(X=x)), add=TRUE, lwd=4, col='blue')

legend(400, 1500, legend=c('weighting','w/o weighting'),
       col=c("red", "blue"), lwd=4, cex=1.5)

summary(fit2)
```

We calculate the residual variance

```r
%%R
sum(residuals(fit2))/952
```

## Global Least-Squares Fit

Now we performa a global fit of the 5 curves with increasing [quencher]: $F(0)$, $\tau_0$, and $k_q$ are global parameters whereas $m$ depends on [quencher].
* We create 3 lists (for x, y, and yerr) each containing 5 `NumPy` arrays (for the 5 curves).
* We create a list called args containing the 3 lists (allx, ally, allyerr)

```python
# args is a list of three lists containing 5 numpy arrays
args = [ [data.loc[conc][col] for conc in concs] for col in ['t','F','errF'] ]
param0 = [3000, 160, 0.04, 0] + [0.5]*4 # initial parameters
lbound = [0, 0, 0] + [0]*5              # lower bound
ubound = [4000, 200, 1, 1e-5] + [3]*4   # upper bound
```

In the residuals function, we calculate the vector of residuals for each curve which appended to a single 1D `NumPy` array.

```python
# we define the function for the intensity decay
def decay(x, param):
    return param[0]*np.exp( - x/param[1] + param[3]*np.expm1(-param[2]*x) )
# we define the function that calculates the least squares
def residuals(p, *args):
    residuals = np.empty(0)
    for i,m in enumerate(p[3:]):
        param = p[0], p[1], p[2], m
        r = ( args[1][i] - decay(args[0][i], param) ) / args[2][i]
        residuals = np.append(residuals, r)
    return residuals
```

```python
res = least_squares(residuals, x0=param0, args=args, bounds=[lbound,ubound])
c = plt.rcParams['axes.prop_cycle'].by_key()['color']
p = res['x']
for i,m in enumerate(p[3:]):
        param = p[0], p[1], p[2], m
        plt.fill_between(args[0][i],args[1][i]-args[2][i],args[1][i]+args[2][i],alpha=.7)
        plt.plot(args[0][i],decay(args[0][i],param),lw=1,color=c[i],label='{:1.1f}'.format(concs[i])+' mM')
plt.yscale('log'); plt.legend(title='[quencher]',labelspacing=.7)
plt.ylabel('$\ln F(t)$ / counts'); plt.xlabel('$t$ / ns');
print('Reduced chi squared:',error_analysis(res)[1])
```

Instead of using markdown to view the results in a table, we create a `pandas` DataFrame.

```python
table = pd.DataFrame({'Value':res.x,'Error':error_analysis(res)[0]},
             index=['$F(0)$ (counts)','$\tau_0$ (ns)','$k_q$ (ns$^{-1}$)','$m_0$','$m_1$','$m_2$','$m_3$','$m_4$'],
            columns=['Value','Error'])
table
```

```python
print(table.to_latex())
```

## Task 2
Take the logarithm of each curve and perform a global fit to $\ln F(t) = \ln F(0) - t/\tau_0 + m [\exp(-k_q t) -1]$ using `least_square`.
1. Modify the `NumPy` arrays in args[1] by taking the `np.log`.
```.py
data = pd.read_pickle('data/fluorescence.p')
allx = []; ally = []; allyerr = [] # three empty lists
for conc in data.columns:
    allx.append(data[conc]['t']); ally.append(data[conc]['F']); allyerr.append(data[conc]['Ferr'])
args = allx, ally, allyerr # args is a list of three lists containing 5 numpy arrays
for i in range( data.columns.size ):
    args[1][i] = ?
    args[2][i] = ?
```
2. Modify the `NumPy` arrays in args[2] according to the error propagation rule: error$(\ln F)$ = error($F$) / $F$.
3. Define a new fitting function for the linearized equation.
4. Fitting and plotting can be carried out exactly as in the previous cell.

```python
answer = 'gAAAAABcBbngbWGpiSxC5RJ77Vi6yfGEXVT93gqOhJFIYQOPDYiy5ygTjDywMIFhjBoc40NOdmF7_xxNU8ZueTjXVWgNa-du0iBv0jp-gt0xP0a-CD9L4cDzi1UDCQvuS7koOuZu103K68ycdjrLP36WW-JmHGZL06hx0VqQ8DfwQ-BfKMTvk55QXER-o2_bGxqGZYlKvcRAvN6FDBqxWSj0PUxq-YGros566TcIrf45EHvxJiB3FrDoUlLDBGvi_7mOMzvnM4zTH8hgLL7CZam9vx-atRkuGQ29y9iTh7MQDhNdtMmJPLv9xMgzRqJ9hP5-X_cySV2tlG21pNcEvTJqYIC7UfWvdfSS430x40aY5unGeNEKUgBAhHWlnHxl6mf8oewmsLnGSYgDnd8MO0VvEpl7VY1IXFTghbUVbG2wfdA18bLQBMEzgf6uK21iMo60DLwJFfci6SwC2fQsQGcNSHJiPhNdyzCIRmKgSkFQDw-CXKF8hHpiAkX0SdcOpegPMuFlSvbfNR_teoTyJemwYosvtGShvSSsbtJj0juSP6fsuGpJ1iRkio8rm8VfdISbXstnFw8ZCiigMDd0DXJL9bfJxYSko2BRAYrzax2nkRGZjGuxsuo20Im_0_4zacxL-pX_VdaLvwsifqOrDQKW7QHLKK4bTEZ-iRoLKI0hd_2JYcrASNTcSE_hmnIDHke4ki2YYpfOU-F-KAHDpefJ2tNH2USCEzuW6Wx9khxquCMgB637Z3up_RSzNdONCyFyX69878CpYR6bRT6kP25MpiUidhYf3WFRTleCIdmx6vdU_G2ug7mbrKQvCW2lmVxlrY-HmkEepqDqJydmpv_4D7pXr28Kwv_EDk-MLbjzrunG5NmfoDPxqMgoXT8mT5Yf23a0bF2qkm1ZhUYdk-0BeIcQ4JZgv_xTCwcWOilQ7oUjAHJa1BpoJI02HAUVOPO643r744w2FNOypIGQRkw3lQoQt176aAbOc9k_MkPErdin5YqZBsK6kppqlAwx_K0mjuh-6uodl_LfVv4UnsE0OvA2uazrPx0qKwp2YSKJO6pUtROEF12K5vHbu_ydVLOaxE8miPshxZm4uVAZelKE1Z5rhKaqmmUNKTjZ9PdeiFns63O6vW2BtPsHJrQQabkXHNfDBeQXPwE-a7wiV3IaT6IXTSZIgWLMWft8QduMabUruM5brgN-bnoAA7AR1ezaLMGvlV7nppyX2F8Lnr8XMtGy2MTDGzchfwxjpf-p4ZEYdsy-R1yHPZXnWoUJv34N4F33heiqwsmzFELx0CPo5DO8X9TM3VRPWRKF2k2UYkwMVI6nKTJe5v2tHY2tJ4JC79zurCAVp7KXtvxcABeuBTAqReaRi7gqy9ASIWnsfribaK683FXes2sVTcX7WJQ9GAN1UOVBLzuBekkeuxi7nlPSg0CXEoofxhpLeautHhx_EMsn2D_lLJjKVlNOrUL_g-7gPj419sOKejraRMtho9Foa2xE-fN7FNww20T4UlVyNBjqQHy4sH9mTYHp4TMBPgvU_OLit6T0tdf_Npc89ns4Oz9Tv-oXqJYnJhfT22VdbHvrZnWPikoAjguOroUrW3zgLaYeJL-vVcbZBIyV6RZu8LKuIq2hEXbaFODdlLhP1q5jt8bEzBTml_VpyxjuVMxNiQSU9DemOuS0A-Q4SO-_w9EboghUOb7_OmNBRXIKYMO-szKcOoDzx2EQemFngHbZTunXpR5pszWBtIvMj1F7g-SGYT2N5FJeNLJmiiA7w3TK70iI7vreadGgrg3k0sGsdzuW7fAHXr4oNnxSLgUYjX-yGPezC3tvKaYZe34d8uilTUrU3JbHZdPjfxuDCbM8I000pA233pa9Lui6yOZxFjdBJXtNSagxLBXf77YAy7WCYjvzS83FfpjyMO4bDN3PVhYbKRy_A2MzCBHpvAgxtR-xHBkI8D-LqpDADwYbUtwEKoOatEnBrjvzpFgFQMPImoiYxDGRHoQQyAOhu59u4hf7jmTunUF_AgGfwT-tUno-Yq94cyg9qdrY1_WK732IrpD-Nud5Kz4EEL5cKw4As7fumVQtiLwgNETgakjswQu3J2budl61VUmaBIe490ZaClZn18Y5IQmm2L7mo94q8mU8mmY1FH3rLooJSMBSfSfDT5mUc5T_iibyEjFov499nEu1XowTEMw2wkT0AzoUXqNCLbMfTmLDOjCWSTeLNQYvIk4S8Uf9D-K-bs7kL5h8aGMQlVszl13HGnt5SlHNqxOVgF016ryD6zMHQPY6Zobu-0x1JvjDbYfJAFXd1yrcSvLDtOGgGd8bEG5D1EPmVnLYVX4G6I4v2bW4NQ1vO44_G3qG74to1NOTral8ONcXNym6EOWqShyYcFZBbJupAwDMyclYF2eUJWsRpzWxhCwKkDQCbdVFJRRlAbphltKqbMaBRO8lDvlkMsGdbdaumPhO-Bs8auANK5UktYjLfe362yvDGNPlceClQT3x_3tUe4I0pkphY-GSccfDcS0uhIZDGYtFDg9V5DOv6nA6Eh5h96NsiZkY2J0pRdKaLEJIcEobhpvOFQQUgMZGebHLsMXxp0kfCSdLakydSvxeud_Cxu6zU18OnuFIledn1MtjWwJbwui7O2OBR7_GxA5qtAWVfpdycrM1njr2FYip2wbR18eI9CJg7iRILo2UMcryPZFgF86ex6TB_rH1-A9pWrryGrJ70-FriNuklt6YI1VhCPbG9qCm79yembK_3zZ2tetPZBgh1dPyYSA4LAThcoEByTF0iUqx3mcjgROlKtjNkez0ZQCTddIMojR4BKkfg4Uc7JQcba3WY2xNiwsUy38l6KzOSDWHIjO_YDG13M_2KUWWDuckebG15KZUJJeKZi94'

```

```python
decrypt(answer,)
```


# The minimum jerk hypothesis

> Marcos Duarte
> Laboratory of Biomechanics and Motor Control ([http://demotu.org/](http://demotu.org/))
> Federal University of ABC, Brazil

Hogan and Flash (1984, 1985), based on observations of voluntary movements in primates, suggested that movements are performed (organized) with the smoothest trajectory possible. In this organizing principle, the endpoint trajectory is such that the mean squared-jerk across time of this movement is minimum.

Jerk is the derivative of acceleration and the observation of the minimum-jerk trajectory is for the endpoint in the extracorporal coordinates (not for joint angles) and according to Flash and Hogan (1985), the minimum-jerk trajectory of a planar movement is such that minimizes the following objective function:

$$ C=\frac{1}{2} \int\limits_{t_{i}}^{t_{f}}\;\left[\left(\frac{d^{3}x}{dt^{3}}\right)^2+\left(\frac{d^{3}y}{dt^{3}}\right)^2\right]\:\mathrm{d}t $$

Hogan (1984) found that the solution for this objective function is a fifth-order polynomial trajectory (see Shadmehr and Wise (2004) for a simpler proof):

$$ \begin{array}{l l}
x(t) = a_0+a_1t+a_2t^2+a_3t^3+a_4t^4+a_5t^5 \\
y(t) = b_0+b_1t+b_2t^2+b_3t^3+b_4t^4+b_5t^5
\end{array} $$

With the following boundary conditions for $ x(t) $ and $ y(t) $: initial and final positions are $ (x_i,y_i) $ and $ (x_f,y_f) $ and initial and final velocities and accelerations are zero.

Let's employ [Sympy](http://sympy.org/en/index.html) to find the solution for the minimum jerk trajectory using symbolic algebra.

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
from IPython.display import display, Math, Latex
from sympy import symbols, Matrix, latex, Eq, collect, solve, diff, simplify
from sympy.utilities.lambdify import lambdify
```

Using Sympy, the equation for minimum jerk trajectory for x is:

```python
# declare the symbolic variables
x, xi, xf, y, yi, yf, d, t = symbols('x, x_i, x_f, y, y_i, y_f, d, t')
a0, a1, a2, a3, a4, a5 = symbols('a_0:6')
x = a0 + a1*t + a2*t**2 + a3*t**3 + a4*t**4 + a5*t**5
display(Math(latex('x(t)=') + latex(x)))
```

$$x(t)=a_{0} + a_{1} t + a_{2} t^{2} + a_{3} t^{3} + a_{4} t^{4} + a_{5} t^{5}$$

Without loss of generality, consider $ t_i=0 $ and let's use $ d $ for movement duration ($ d=t_f $). The system of equations with the boundary conditions for $ x $ is:

```python
# define the system of equations
s = Matrix([Eq(x.subs(t,0)          , xi),
            Eq(diff(x,t,1).subs(t,0),  0),
            Eq(diff(x,t,2).subs(t,0),  0),
            Eq(x.subs(t,d)          , xf),
            Eq(diff(x,t,1).subs(t,d),  0),
            Eq(diff(x,t,2).subs(t,d),  0)])
display(Math(latex(s, mat_str='matrix', mat_delim='[')))
```

$$\left[\begin{matrix}a_{0} = x_{i}\\a_{1} = 0\\2 a_{2} = 0\\a_{0} + a_{1} d + a_{2} d^{2} + a_{3} d^{3} + a_{4} d^{4} + a_{5} d^{5} = x_{f}\\a_{1} + 2 a_{2} d + 3 a_{3} d^{2} + 4 a_{4} d^{3} + 5 a_{5} d^{4} = 0\\2 a_{2} + 6 a_{3} d + 12 a_{4} d^{2} + 20 a_{5} d^{3} = 0\end{matrix}\right]$$

Which gives the following solution:

```python
# algebraically solve the system of equations
sol = solve(s, [a0, a1, a2, a3, a4, a5])
display(Math(latex(sol)))
```

$$\left \{ a_{0} : x_{i}, \quad a_{1} : 0, \quad a_{2} : 0, \quad a_{3} : \frac{10}{d^{3}} \left(x_{f} - x_{i}\right), \quad a_{4} : \frac{15}{d^{4}} \left(- x_{f} + x_{i}\right), \quad a_{5} : \frac{6}{d^{5}} \left(x_{f} - x_{i}\right)\right \}$$

Substituting this solution in the fifth order polynomial trajectory equation, we have the actual displacement trajectories:

```python
# substitute the equation parameters by the solution
x2 = x.subs(sol)
x2 = collect(simplify(x2, ratio=1), xf-xi)
display(Math(latex('x(t)=') + latex(x2)))
y2 = x2.subs([(xi, yi), (xf, yf)])
display(Math(latex('y(t)=') + latex(y2)))
```

$$x(t)=x_{i} + \left(x_{f} - x_{i}\right) \left(\frac{10 t^{3}}{d^{3}} - \frac{15 t^{4}}{d^{4}} + \frac{6 t^{5}}{d^{5}}\right)$$

$$y(t)=y_{i} + \left(y_{f} - y_{i}\right) \left(\frac{10 t^{3}}{d^{3}} - \frac{15 t^{4}}{d^{4}} + \frac{6 t^{5}}{d^{5}}\right)$$

And for the velocity, acceleration, and jerk trajectories in x:

```python
# symbolic differentiation
vx = x2.diff(t, 1)
display(Math(latex('v_x(t)=') + latex(vx)))
ax = x2.diff(t, 2)
display(Math(latex('a_x(t)=') + latex(ax)))
jx = x2.diff(t, 3)
display(Math(latex('j_x(t)=') + latex(jx)))
```

$$v_x(t)=\left(x_{f} - x_{i}\right) \left(\frac{30 t^{2}}{d^{3}} - \frac{60 t^{3}}{d^{4}} + \frac{30 t^{4}}{d^{5}}\right)$$

$$a_x(t)=\frac{60 t}{d^{3}} \left(x_{f} - x_{i}\right) \left(1 - \frac{3 t}{d} + \frac{2 t^{2}}{d^{2}}\right)$$

$$j_x(t)=\frac{60}{d^{3}} \left(x_{f} - x_{i}\right) \left(1 - \frac{6 t}{d} + \frac{6 t^{2}}{d^{2}}\right)$$

Let's plot the minimum jerk trajectory for x and its velocity, acceleration, and jerk considering $x_i=0,x_f=1,d=1$:

```python
# substitute by the numerical values
x3  = x2.subs([(xi, 0), (xf, 1), (d, 1)])
#create functions for calculation of numerical values
xfu = lambdify(t, diff(x3, t, 0), 'numpy')
vfu = lambdify(t, diff(x3, t, 1), 'numpy')
afu = lambdify(t, diff(x3, t, 2), 'numpy')
jfu = lambdify(t, diff(x3, t, 3), 'numpy')
#plots using matplotlib
ts = np.arange(0, 1.01, .01)
fig, axs = plt.subplots(1, 4, figsize=(12, 5), sharex=True, squeeze=True)
axs[0].plot(ts, xfu(ts), linewidth=3)
axs[0].set_title('Displacement [$\mathrm{m}$]')
axs[1].plot(ts, vfu(ts), linewidth=3)
axs[1].set_title('Velocity [$\mathrm{m/s}$]')
axs[2].plot(ts, afu(ts), linewidth=3)
axs[2].set_title('Acceleration [$\mathrm{m/s^2}$]')
axs[3].plot(ts, jfu(ts), linewidth=3)
axs[3].set_title('Jerk [$\mathrm{m/s^3}$]')

for axi in axs:
    axi.set_xlabel('Time [s]', fontsize=14)
    axi.grid(True)

fig.suptitle('Minimum jerk trajectory kinematics', fontsize=20, y=1.03)
fig.tight_layout()
plt.show()
```

Note that for the minimum jerk trajectory, initial and final values of both velocity and acceleration are zero, but not for the jerk.

Read more about the minimum jerk trajectory hypothesis in the [Shadmehr and Wise's book companion site](http://www.shadmehrlab.org/book/minimum_jerk/minimumjerk.htm) and in [Paul Gribble's website](http://www.gribblelab.org/compneuro/4_Computational_Motor_Control_Kinematics.html#sec-5-1).

### The angular trajectory of a minimum jerk trajectory

Let's calculate the resulting angular trajectory given a minimum jerk linear trajectory, supposing it is from a circular motion of an elbow flexion. The length of the forearm is 0.5 m, the movement duration is 1 s, the elbow starts flexed at 90$^o$ and the flexes to 180$^o$.

First, the linear trajectories for this circular motion:

```python
# substitute by the numerical values
x3 = x2.subs([(xi, 0.5), (xf, 0), (d, 1)])
y3 = x2.subs([(xi, 0), (xf, 0.5), (d, 1)])
display(Math(latex('y(t)=') + latex(x3)))
display(Math(latex('x(t)=') + latex(y3)))
#create functions for calculation of numerical values
xfux = lambdify(t, diff(x3, t, 0), 'numpy')
vfux = lambdify(t, diff(x3, t, 1), 'numpy')
afux = lambdify(t, diff(x3, t, 2), 'numpy')
jfux = lambdify(t, diff(x3, t, 3), 'numpy')
xfuy = lambdify(t, diff(y3, t, 0), 'numpy')
vfuy = lambdify(t, diff(y3, t, 1), 'numpy')
afuy = lambdify(t, diff(y3, t, 2), 'numpy')
jfuy = lambdify(t, diff(y3, t, 3), 'numpy')
```

$$y(t)=- 3.0 t^{5} + 7.5 t^{4} - 5.0 t^{3} + 0.5$$

$$x(t)=3.0 t^{5} - 7.5 t^{4} + 5.0 t^{3}$$

```python
#plots using matplotlib
ts = np.arange(0, 1.01, .01)
fig, axs = plt.subplots(1, 4, figsize=(12, 5), sharex=True, squeeze=True)
axs[0].plot(ts, xfux(ts), 'b', linewidth=3)
axs[0].plot(ts, xfuy(ts), 'r', linewidth=3)
axs[0].set_title('Displacement [$\mathrm{m}$]')
axs[1].plot(ts, vfux(ts), 'b', linewidth=3)
axs[1].plot(ts, vfuy(ts), 'r', linewidth=3)
axs[1].set_title('Velocity [$\mathrm{m/s}$]')
axs[2].plot(ts, afux(ts), 'b', linewidth=3)
axs[2].plot(ts, afuy(ts), 'r', linewidth=3)
axs[2].set_title('Acceleration [$\mathrm{m/s^2}$]')
axs[3].plot(ts, jfux(ts), 'b', linewidth=3)
axs[3].plot(ts, jfuy(ts), 'r', linewidth=3)
axs[3].set_title('Jerk [$\mathrm{m/s^3}$]')

for axi in axs:
    axi.set_xlabel('Time [s]', fontsize=14)
    axi.grid(True)

fig.suptitle('Minimum jerk trajectory kinematics', fontsize=20, y=1.03)
fig.tight_layout()
plt.show()
```

Now, the angular trajectories for this circular motion:

```python
from sympy import atan2
ang = atan2(y3, x3)*180/np.pi
display(Math(latex('angle(t)=') + latex(ang)))
xang = lambdify(t, diff(ang, t, 0), 'numpy')
vang = lambdify(t, diff(ang, t, 1), 'numpy')
aang = lambdify(t, diff(ang, t, 2), 'numpy')
jang = lambdify(t, diff(ang, t, 3), 'numpy')
```

$$angle(t)=57.2957795130823 \operatorname{atan_{2}}{\left (3.0 t^{5} - 7.5 t^{4} + 5.0 t^{3},- 3.0 t^{5} + 7.5 t^{4} - 5.0 t^{3} + 0.5 \right )}$$

```python
ts = np.arange(0, 1.01, .01)
fig, axs = plt.subplots(1, 4, figsize=(12, 5), sharex=True, squeeze=True)
axs[0].plot(ts, xang(ts), linewidth=3)
axs[0].set_title('Displacement [$\mathrm{m}$]')
axs[1].plot(ts, vang(ts), linewidth=3)
axs[1].set_title('Velocity [$\mathrm{m/s}$]')
axs[2].plot(ts, aang(ts), linewidth=3)
axs[2].set_title('Acceleration [$\mathrm{m/s^2}$]')
axs[3].plot(ts, jang(ts), linewidth=3)
axs[3].set_title('Jerk [$\mathrm{m/s^3}$]')

for axi in axs:
    axi.set_xlabel('Time [s]', fontsize=14)
    axi.grid(True)

fig.suptitle('Minimum jerk trajectory angular kinematics', fontsize=20, y=1.03)
fig.tight_layout()
plt.show()
```

## Problems

1. What is your opinion on the the minimum jerk hypothesis? Do you think humans control movement based on this principle? (Think about what biomechanical and neurophysiological properties are not considered on this hypothesis.)
2. Calculate and plot the position, velocity, acceleration, and jerk trajectories for different movement speeds (for example, consider always a displacement of 1 m and movement durations of 0.5, 1, and 2 s).
3. For the data in the previous item, calculate the ratio peak speed to average speed. Shadmehr and  Wise (2004) argue that psychophysical experiments show that reaching movements with the hand have this ratio equals to 1.75. Compare with the calculated values.
4. Can you propose alternative hypotheses for the control of movement?

## References

- Flash T, Hogan N (1985) [The coordination of arm movements: an experimentally confirmed mathematical model](http://www.jneurosci.org/cgi/reprint/5/7/1688.pdf). Journal of Neuroscience, 5, 1688-1703.
- Hogan N (1984) [An organizing principle for a class of voluntary movements](http://www.jneurosci.org/content/4/11/2745.full.pdf). Journal of Neuroscience, 4, 2745-2754.
- Shadmehr R, Wise S (2004) [The Computational Neurobiology of Reaching and Pointing: A Foundation for Motor Learning](http://www.shadmehrlab.org/book/). A Bradford Book. [Companion site](http://www.shadmehrlab.org/book/).
- Zatsiorsky VM (1998) [Kinematics of Human Motion](http://books.google.com.br/books/about/Kinematics_of_Human_Motion.html?id=Pql_xXdbrMcC&redir_esc=y). Champaign, Human Kinetics.


```python
%matplotlib inline
import matplotlib.pyplot as plt
import sympy as sym
from sympy import oo
sym.init_printing()
# X = sym.laplace_transform(x, t, s)

def pole_zero_plot(poles, zeros, ax=None):
    mR = -2 #  valor inicial para minimo real
    MR = 2  #  valor inicial para maximo real
    mI = -2 #  valor inicial para minimo imaginario
    MI = 2  #  valor inicial para maximo imaginario
    for pole in poles:
        plt.plot(complex(pole).real, complex(pole).imag, 'rx', markersize=10)
        if complex(pole).real < mR:
            mR = complex(pole).real
        if complex(pole).real > MR:
            MR = complex(pole).real
        if complex(pole).imag < mI:
            mI = complex(pole).imag
        if complex(pole).imag > MI:
            MI = complex(pole).imag

    for zero in zeros:
        plt.plot(complex(zero).real, complex(zero).imag, 'bo', markersize=10)
        if complex(zero).real < mR:
            mR = complex(zero).real
        if complex(zero).real > MR:
            MR = complex(zero).real
        if complex(zero).imag < mI:
            mI = complex(zero).imag
        if complex(zero).imag > MI:
            MI = complex(zero).imag

    if ax is None:
        ax = plt.gca()
    ax.set_aspect(1)

    plt.axis([mR*1.1, MR*1.1, mI*1.1, MI*1.1])
    plt.xlabel(r'$\Re \{ s \}$')
    plt.ylabel(r'$\Im \{ s \}$')
    plt.grid()

```

## Definición

La [transformada de Laplace](https://es.wikipedia.org/wiki/Transformada_de_Laplace) está definida por

\begin{equation}
X(s) = \int_{-\infty}^{\infty} x(t) \, e^{-s t} \; dt
\end{equation}

En la mayoría de aplicaciones en ingeniería, las señales se definene para $t>0$, lo que permite expresar la transformada como:

\begin{equation}
X(s) = \int_{0}^{\infty} x(t) \, e^{-s t} \; dt
\end{equation}

donde $X(s) = \mathcal{L} \{ x(t) \}$ se usa como notación de la Transformada de Laplace de la señal $x(t)$.

La variable $s = \sigma + j \omega$ es compleja, de manera que en el kernel de transformación $e^{-s t}$ aparezcan los comportamientos:

- oscilatorios con $e^{-j \omega t}$
- exponenciales crecientes o decreientes con $e^{\sigma t}$
- constantes con $e^{0 t}$
- combinaciones de los anteriores.

Lo anterior permite analizar sistemas LTI con la transformada de Laplace.

El conjunto de valores de $s$ que hacen que la integral de transformación converja se llama Región de convergencia (ROC).

Observe que la forma de la transformada de Laplace mantiene la estructura de la Transformada de Fourier. De esta manera, podría interpretarse como el "parecido" entre la señal $x(t)$ y  $e^{\sigma} \cdot cos(\omega t)$.

La transformada inversa de Laplace es:

\begin{equation}
x(t) = \frac{1}{2 \pi j} \int_{\sigma - j \infty}^{\sigma + j \infty} X(s) \, e^{s t} \; ds
\end{equation}

### Transformada del delta de Dirac

\begin{equation}
\mathcal{L} \{ \delta(t) \} = \int_{-\infty}^{\infty} \delta(t) \, e^{- s t} dt = e^{- s t} \Big\vert_{t=0}  =  1
\end{equation}
para $s \in \mathbb{C}$. La ROC corresponde a todo el plano $S$.

Como $\mathcal{L} \{ \delta(t) \} = 1$, todas las frecuencias están presentes en la señal con igual peso.

```python
t = sym.symbols('t', real=True)
s = sym.symbols('s', complex=True)

X = sym.integrate(sym.DiracDelta(t)*sym.exp(-s*t), (t, -sym.oo, sym.oo))
X
```

### Transformada de una Exponencial Compleja y causal

La transformada $X(s) = \mathcal{L} \{ x(t) \}$ de una señal exponencial compleja y causal

\begin{equation}
x(t) = \epsilon(t) \cdot e^{-s_0 t}
\end{equation}

con frecuencia compleja $s_0 \in \mathbb{C}$ se obtiene de la expresión integral de la transfomada de Laplace

\begin{equation}
\begin{split}
X(s) &= \int_{0}^{\infty} e^{-s_0 t} \cdot e^{-s t} \; dt \\
&= \int_{0}^{\infty} e^{-(s_0+s) t} \; dt \\
&= \frac{-e^{-(s+s_0)t}}{s+s_0} \, \bigg\vert_{0}^{\infty} \\
&= \frac{-1}{s+s_0} \left[ \lim_{t \to \infty} \left( e^{- (s+s_0) t} \right) -1 \right] \\
&= \frac{1}{s + s_0}
\end{split}
\end{equation}

Observe que el límite anterior converge solo si $e^{- (s+s_0) t}$ converge, es decir que $\Re \{ s + s_0 \} > 0$. Por lo cual la ROC es:

\begin{equation}
\Re \{ s \} > \Re \{ - s_0 \}
\end{equation}

Entonces, la transformada de la señal exponencial compleja y causal es
\begin{equation}
\mathcal{L} \{ \epsilon(t) \cdot e^{-s_0 t} \} = \frac{1}{s + s_0} \qquad \forall s : \Re \{ s \} > \Re \{ - s_0 \}
\end{equation}

```python
t = sym.symbols('t', real=True)
s = sym.symbols('s', complex=True)

s0 = 2
x = sym.exp(-s0*t)*sym.Heaviside(t)
x
```

```python
X = sym.integrate(x*sym.exp(-s*t), (t, -sym.oo, sym.oo))
X
```

$\displaystyle \begin{cases} \frac{1}{2 \left(\frac{s}{2} + 1\right)} & \text{for}\: \left|{\arg{\left(s \right)}}\right| \leq \frac{\pi}{2} \\\int\limits_{-\infty}^{\infty} e^{- 2 t} e^{- s t} \theta\left(t\right)\, dt & \text{otherwise} \end{cases}$

```python
X, a, cond = sym.laplace_transform(x, t, s)
X, a, cond
```

$\displaystyle \left( \frac{1}{s + 2}, \  0, \  \text{True}\right)$

```python
help(sym.laplace_transform)
```

    Help on function laplace_transform in module sympy.integrals.transforms:

    laplace_transform(f, t, s, **hints)
        Compute the Laplace Transform `F(s)` of `f(t)`,

        .. math :: F(s) = \int_0^\infty e^{-st} f(t) \mathrm{d}t.

        For all "sensible" functions, this converges absolutely in a
        half plane  `a < \operatorname{Re}(s)`.

        This function returns ``(F, a, cond)``
        where ``F`` is the Laplace transform of ``f``, `\operatorname{Re}(s) > a` is the half-plane
        of convergence, and ``cond`` are auxiliary convergence conditions.

        If the integral cannot be computed in closed form, this function returns
        an unevaluated :class:`LaplaceTransform` object.

        For a description of possible hints, refer to the docstring of
        :func:`sympy.integrals.transforms.IntegralTransform.doit`. If ``noconds=True``,
        only `F` will be returned (i.e. not ``cond``, and also not the plane ``a``).

        >>> from sympy.integrals import laplace_transform
        >>> from sympy.abc import t, s, a
        >>> laplace_transform(t**a, t, s)
        (s**(-a)*gamma(a + 1)/s, 0, re(a) > -1)

        See Also
        ========

        inverse_laplace_transform, mellin_transform, fourier_transform
        hankel_transform, inverse_hankel_transform

```python
X
```

SymPy devuelve tres resutados:

- Primero la transformada de Laplace $X(s)$
- Segundo un valor $a$ que indica qué parte del semiplano $\Re \{ s \} > 0$ tiene convergencia en que $\Re \{ s \} > a$
- Tercero otras condiciones de convergencia.

Inicialmente se analizará solamente la transformada.

Tenga en cuenta que $s$ es una variable compleja, de modo que el "espectro" no puede representarse como una curva.

```python
sigma, omega = sym.symbols('sigma omega')

X1 = X.subs({s: sigma+sym.I*omega})
print(X1)

plt.rcParams['figure.figsize'] = 6,6
sym.plotting.plot3d(abs(X1),
                    (sigma, -20, 20),
                    (omega, -20, 20),
                    xlabel=r'$\Re\{s\}$',
                    ylabel=r'$\Im\{s\}$',
                    title=r'$|X(s)|$');
```

Los diagramas de polos y ceros son versiones simplificadas de la gráfica anterior.

```python
sym.denom(X) # El numerador de X(s)
```

```python
sym.roots(sym.denom(X), s) # Las raíces del numerador de X(s)
```

```python
plt.rcParams['figure.figsize'] = 4,4
pole_zero_plot(sym.roots(sym.denom(X), s), sym.roots(sym.numer(X), s))
plt.rcParams['figure.figsize'] = 4,4
sym.plot(sym.re(x),(t,-1.0,5.0))
plt.rcParams['figure.figsize'] = 4,4
sym.plot(sym.im(x),(t,-1.0,5.0))
```

## Propiedades

Dadas las propiedades presentadas para la transformada de Fourier y la estructura de las transformadas de Laplace y de Fourier, se obtienen las mismas propiedades.

### Invertible
\begin{equation}
x(t) = \mathcal{L}^{-1} \left\{ \mathcal{L} \{ x(t) \} \right\}
\end{equation}

### Linealidad

\begin{equation}
\mathcal{L} \{ A \cdot x_1(t) + B \cdot x_2(t) \} = A \cdot X_1(S) + B \cdot X_2(S)
\end{equation}

**Ejemplo - Transformada de Laplace de una señal senoidal causal**

\begin{equation}
x(t) = sin( \omega _0 t) \cdot \epsilon(t)
\end{equation}

con $\omega _0 \in \mathbb{R}^+$

De acuerdo con las expresiones de Euler:
\begin{align}
\cos(\omega_0 t) &= \frac{1}{2} \left( e^{j \omega_0 t} + e^{-j \omega_0 t} \right) \\
\sin(\omega_0 t) &= \frac{1}{2j} \left( e^{j \omega_0 t} - e^{-j \omega_0 t} \right)
\end{align}

Así, la transformada de $sin( \omega _0 t)$ debe ser igual a la de $\frac{1}{2j} ( e^{j \omega_0 t} - e^{-j \omega_0 t})$.

Dado que la transformada de Laplace de una exponencial compleja decreciente y causal es :

\begin{equation}
\mathcal{L} \{ \epsilon(t) \cdot e^{-s_0 t} \} = \frac{1}{s + s_0}
\end{equation}

Entonces:
\begin{equation}
\mathcal{L} \{ sin( \omega _0 t) \} = \frac{1}{2j} \left( \frac{1}{s - j \omega_0} - \frac{1}{s + j \omega_0} \right) = \frac{\omega_0}{s^2 + \omega_0^2}
\end{equation}

Aplicando el mismo razonamiento puede encontrarse la transformada del coseno.

```python
t = sym.symbols('t', real=True)
s = sym.symbols('s', complex=True)

w0 = 5
x = sym.sin(w0* t)*sym.Heaviside(t)
x
```

```python
X, a, cond = sym.laplace_transform(x, t, s)
X
```

```python
sigma, omega = sym.symbols('sigma omega')

X1 = X.subs({s: sigma+sym.I*omega})

plt.rcParams['figure.figsize'] = 6,6
sym.plotting.plot3d(abs(X1),
                    (sigma, -10, 10),
                    (omega, -10, 10),
                    xlabel=r'$\Re\{s\}$',
                    ylabel=r'$\Im\{s\}$',
                    title=r'$|X(s)|$');
```

\begin{equation}
s^2+25=0
\end{equation}
\begin{equation}
s^2=-25
\end{equation}

\begin{equation}
s = +- j 5
\end{equation}

```python
sym.denom(X)
```

```python
sym.roots(sym.denom(X), s)
```

```python
plt.rcParams['figure.figsize'] = 4,4

pole_zero_plot(sym.roots(sym.denom(X), s), sym.roots(sym.numer(X), s))

plt.rcParams['figure.figsize'] = 4,4
sym.plot(sym.re(x),(t,-1.0,5.0))
plt.rcParams['figure.figsize'] = 4,4
sym.plot(sym.im(x),(t,-1.0,5.0))
```

**Ejercicio**

Encuentre la transformada de Laplace de la señal $cos(\omega_0 t)  \cdot \epsilon(t)$

$$\cos(\omega_0 t) = \frac{1}{2} \left( e^{j \omega_0 t} + e^{-j \omega_0 t} \right)$$

Recuerde que:

$$ \mathcal{L} \{ \epsilon(t) \cdot e^{-s_0 t} \} = \frac{1}{s + s_0} \qquad \forall s : \Re \{ s \} > \Re \{ - s_0 \} $$

Desarrolle su análisis aquí

```python
# Escriba su código
```

### Lista de propiedades

A continuación se listan propiedades y teoremas:

|   | $x(t)$ | $X(s) = \mathcal{L} \{ x(t) \}$ | Región de convergencia ROC |
|:---|:---:|:---:|:---:|
| Linealidad | $A \, x_1(t) + B \, x_2(t)$ | $A \, X_1(s) + B \, X_2(s)$ | $\supseteq ROC\{x_1\} \cap ROC\{x_2\}$ |
| Señales reales | $x(t) = x^*(t)$ | $X(s) = X^*(s^*)$ | |
| Escalamiento | $x(a t)$ | $\frac{1}{\lvert a \rvert} X\left( \frac{s}{a} \right)$ | $s: \frac{s}{a} \in \text{ROC}\{x(t)\}$ |
| Convolución | $x(t) * h(t)$ | $X(s) \cdot H(s)$ | $\supseteq \text{ROC}\{x(t)\} \cap \text{ROC}\{h(t)\}$ |
| Desplazamiento | $x(t - \tau)$ | $e^{-s \tau} \cdot X(s)$ | $\text{ROC}\{x(t)\}$ |
| Modulación | $e^{s_0 t}\cdot x(t)$ | $X(s - s_0)$ | $s: s - \Re \{s_0\} \in \text{ROC}\{x(t)\}$ |
| Diferenciación | $\frac{d}{dt} x(t)$ | $s \cdot X(s) - x(0+)$ | $\supseteq \text{ROC}\{x(t)\}$ |
| Integración | $\int_{-\infty}^{t} x(t) \; dt$ | $\frac{1}{s} \cdot X(s)$ | $\supseteq \text{ROC}\{x(t)\} \cap \{s: \Re \{s\} > 0 \}$ |

donde $A, B, s_0 \in \mathbb{C}$, $a \in \mathbb{R} \setminus \{0\}$ y $\tau \in \mathbb{R}$.

## Algunas transformadas de Laplace

A continuación se listan las transformadas de Laplace de funciones usadas a menudo:

| $x(t)$                             | $X(s) = \mathcal{L} \{ x(t) \}$ | Región de Convergencia ROC |
|:----------------------------------:|:-------------------------------:|:--------------------------:|
| $\delta(t)$                        | $1$                             | $\mathbb{C}$               |
| $\epsilon(t)$                      | $\frac{1}{s}$                   | $\Re \{s\} > 0$            |
| $t \epsilon(t)$                    | $\frac{1}{s^2}$                 | $\Re \{s\} > 0$            |
| $e^{- s_0 t} \epsilon(t)$          | $\frac{1}{s + s_0}$             | $\Re \{s\} > \text{Re}\{-s_0\}$ |
| $\sin(\omega_0 t) \epsilon(t)$     | $\frac{\omega_0}{s^2 + \omega_0^2}$ | $\Re \{s\} > 0$        |
| $\cos(\omega_0 t) \epsilon(t)$     | $\frac{s}{s^2 + \omega_0^2}$    | $\Re \{s\} > 0$            |
| $t^n e^{-s_0 t} \epsilon(t)$       | $\frac{n!}{(s+s_0)^{n+1}}$      | $\Re \{s\} > \text{Re}\{-s_0\}$ |
| $e^{-s_0 t} \cos(\omega_0 t) \epsilon(t)$ | $\frac{s + s_0}{(s+s_0)^2 + \omega_0^2}$ | $\Re \{s\} > \Re \{-s_0\}$ |
| $e^{-s_0 t} \sin(\omega_0 t) \epsilon(t)$ | $\frac{\omega_0}{(s+s_0)^2 + \omega_0^2}$ | $\Re \{s\} > \Re \{-s_0\}$ |
| $t \cos(\omega_0 t) \epsilon(t)$ | $\frac{s^2 - \omega_0^2}{(s^2 + \omega_0^2)^2}$ | $\Re \{s\} > 0$ |
| $t \sin(\omega_0 t) \epsilon(t)$ | $\frac{2 \omega_0 s}{(s^2 + \omega_0^2)^2}$ | $\Re \{s\} > 0$ |

Donde $s_0 \in \mathbb{C}$, $\omega_0 \in \mathbb{R}$ and $n \in \mathbb{N}$. Otras transformadas se pueden encontrar [en línea](https://en.wikipedia.org/wiki/List_of_Laplace_transforms).

##  Circuitos
### Impedancias complejas y equivalencias

Los circuitos eléctricos pueden analizarse en el dominio transformado de Laplace enfocándose primero en el comportamiento de cada elemento. La siguiente tabla resume el comportamiento de los elementos pasivos de los circuitos eléctricos el el dominio del tiempo (Ecuación diferencial) y de la frecuencia (Laplace). La impedancia compleja se muestra en la última columna.

| Elemento | Dominio del tiempo  (EDO)| Dominio de la frecuencias con Laplace | $Z(s) = \frac{U(s)}{I(s)}$ |
|:---:|:---:|:---:|:---:|
|  | $u(t) = R \cdot i(t)$ | $U(s) = R \cdot I(s)$ | $R$ |
|  | $\begin{matrix} u(t) = L \frac{d}{dt} i(t) \\ i(t) = \frac{1}{L} \int_{0}^{t} u(\tau) d\tau + i(0+) \epsilon(t) \end{matrix}$ | $\begin{matrix} U(s) = s L I(s) - L i(0+) \\ I(s) = \frac{1}{s L} U(s) + \frac{1}{s} i(0+) \end{matrix}$ | $s L$ |
|  | $\begin{matrix} u(t) = \frac{1}{C} \int_{0}^{t} i(\tau) d\tau + u(0+) \epsilon(t) \\ i(t) = C \frac{d}{dt} u(t) \end{matrix}$ | $\begin{matrix} U(s) = \frac{1}{s C} I(s) + \frac{1}{s} u(0+) \\ I(s) = s C U(s) - C u(0+) \end{matrix}$ | $\frac{1}{s C}$

Considere el siguiente circuito suponiendo valores iniciales iguales a cero.

Para ilustrar, considere que $L = 0.5$, $R = 1$, $C = 0.4$ .

\begin{equation}
v_L(t) + v_R(t) + v_C(t) = x(t)
\end{equation}

\begin{equation}
L \frac{di}{dt} + R i(t) + \frac{1}{C} \int_{0}^{t} i(\tau) d\tau = x(t)
\end{equation}

La salida es el voltaje del capacitor

\begin{equation}
y(t) = \frac{1}{C} \int_{0}^{t} i(\tau) d\tau
\end{equation}

De donde se obtiene la corriente

\begin{equation}
C \frac{dy}{dt} = i(t)
\end{equation}

Incorporando la salida en la ecuación de la malla se obtiene

\begin{equation}
C L \frac{d^2 y(t)}{dt^2} + C R \frac{d y(t)}{dt} + y(t) = x(t)
\end{equation}

La ecuación anterior puede transformarse al dominio de Laplace.
\begin{equation}
C L s^2Y(s)  + C R sY(s) + Y(s) = X(s)
\end{equation}

Puede encontrarse una expresión para la salida en el dominio de Laplace como

\begin{equation}
Y(s) = \frac{1}{LC s^2 + RC s + 1} \cdot X(s)
\end{equation}

Observe que el circuito puede transformarse al dominio de Laplace directamente.

**Nota** Las mismas aproximaciones pueden realizarse en otros campos de conocimiento en donde se modelen fenómenos con Ecuaciones Diferenciales.

\begin{equation}
Y(s) = X(s) \frac{\frac{1}{sC}}{R + sL + \frac{1}{sC}}
\end{equation}

\begin{equation}
Y(s) = X(s) \frac{\frac{sC}{sC}}{RCs + sLCs + \frac{sC}{sC}}
\end{equation}

\begin{equation}
Y(s) = X(s) \frac{1}{RCs + LCs^2 + 1}
\end{equation}

```python
%matplotlib inline
import sympy as sym
sym.init_printing()

s = sym.symbols('s', complex=True)
t, R, L, C = sym.symbols('t R L C', real=True)
X = sym.Function('X')(s)

F = 1/(L*C*s**2 + R*C*s + 1)
Y = F * X
Y
```

Introduzca los valores de los elementos del circuito en un diccionario para usarlo después.

```python
RLC = {R: 1,
       L: 0.5,
       C: .4}
RLC
```

### Respuesta al escalón

La respuesta al escalón en el dominio de Laplace se obtiene al considerar que $X(s) = \frac{1}{s}$

```python
F = F.subs(RLC)
F
```

```python
Y
```

```python

Y_escalon = Y.subs(RLC).subs(X, 1/s)
Y_escalon
```

La respuesta al escalón en el dominio del tiempo se obtiene al considerar operar la respuesta en el doninio de la frecuencia con la Transformada inversa de Laplace.

```python
y_escalon = sym.inverse_laplace_transform(Y_escalon,s,t)
y_escalon
```

Ahora puede graficarse la señal de salida en el dominio del tiempo y el mapa de polos y ceros asociado a dicha señal.

```python
plt.rcParams['figure.figsize'] = 7, 2
sym.plot(y_escalon, (t, -1, 10), xlabel='$t$', ylabel='$y(t)$');
plt.rcParams['figure.figsize'] = 3, 3
pole_zero_plot(sym.roots(sym.denom(Y_escalon), s), sym.roots(sym.numer(Y_escalon), s))
```

La ubicación de las raíces en el plano $s$ determinan la forma de las señales y pueden asociarse con propiedades de las señales o sistemas que modelan.

```python

```


# [Demo] Bayesian filters: vehicle tracking

Wouter Kouw | Last update: 31-05-2022

We will be tracking a driving car based on noisy position observations. You can find this example in [Bayesian Filtering & Smoothing](https://www.cambridge.org/core/books/bayesian-filtering-and-smoothing/C372FB31C5D9A100F8476C1B23721A67) by Simon Särkkä (Ex 3.6).

## System

Consider a discrete-time dynamical system with noisy observations $y_k \in \mathbb{R}^{d_y}$ and latent states $x_k \in \mathbb{R}^{d_x}$. We can express the dynamics in the form of a state-space model:

$$\begin{align}
x_k =&\ Ax_{k-1} + q_{k-1} \\
y_k =&\ Cx_k + r_{k} \, ,
\end{align}$$

where $A$ is a state transition matrix, $C$ is an observation emission matrix, $q_{k-1}$ represents process noise and $r_{k}$ is measurement noise. Both sources of noise are Gaussian distributed with zero mean:

$$q_{k-1} \sim \mathcal{N}(0, Q) \, , \qquad r_k \sim \mathcal{N}(0, R) \, ,$$

with covariance matrices $Q$ and $R$.

The state transition has the following form:

$$\begin{align*}
\underbrace{\begin{bmatrix} x_{1,k} \\ x_{2,k} \\ x_{3,k} \\ x_{4,k} \end{bmatrix}}_{x_k} = \underbrace{\begin{bmatrix} 1 & 0 & \Delta t & 0 \\ 0 & 1 & 0 & \Delta t \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1\end{bmatrix}}_{A} \underbrace{\begin{bmatrix} x_{1,k-1} \\ x_{2,k-1} \\ x_{3,k-1} \\ x_{4,k-1} \end{bmatrix}}_{x_{k-1}} + q_{k-1}
\end{align*}$$

where $x_1$ and $x_2$ represent the 2-D position of the car and $x_3$ and $x_4$ represent its velocity.

The measurement function is described as:

$$\begin{align*}
\underbrace{\begin{bmatrix} y_{1,k} \\ y_{2,k} \end{bmatrix}}_{y_k} = \underbrace{\begin{bmatrix} 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \end{bmatrix}}_{C} \begin{bmatrix} x_{1,k} \\ x_{2,k} \\ x_{3,k} \\ x_{4,k} \end{bmatrix} + r_k
\end{align*}$$

We only get the noisy observations of the position of the car, but not its velocity.

#### Generate data

```julia
using LinearAlgebra
using ProgressMeter
using Plots
pyplot();
```

```julia
include("../kalman_filters.jl")
```

    robust_students_t_filter (generic function with 1 method)

```julia
# Length of time-series
T = 100
Δt = 0.1

# Dimensionalities
Dx = 4
Dy = 2

# Transition matrix
A = [1 0 Δt 0;
     0 1 0 Δt;
     0 0 1 0;
     0 0 0 1];

# Emission matrix
C = [1 0 0 0;
     0 1 0 0];

# Process noise covariance
qc = 0.1
Q = qc.*[Δt^3/3       0   Δt^2/2       0;
              0  Δt^3/3        0  Δt^2/2;
         Δt^2/2       0       Δt       0;
              0  Δt^2/2        0      Δt];

# Measurement noise variance
rc = 0.01
R = rc.*Matrix{Float64}(I,2,2)

# Initial states
state_0 = [0., 0., 1., -1.]

# Initialize data array
states = zeros(Dx,T)
observations = zeros(Dy,T)

# Initialize previous state variable
prev_state = state_0

for k = 1:T

    # State transition
    states[:,k] = A*prev_state + cholesky(Q).L*randn(Dx)

    # Observation with added measurement noise
    observations[:,k] = C*states[:,k] + cholesky(R).L*randn(Dy)

    # Update "previous state"
    prev_state = states[:,k]

end
```

```julia
# Inspect data
p1 = plot((1:T).*Δt, states[1,:], linewidth=3, color="blue", xlabel="", ylabel="position_1", label="states", size=(700,400))
scatter!((1:T).*Δt, observations[1,:], color="black", markersize=2, label="observed", legend=:topleft)
p2 = plot((1:T).*Δt, states[2,:], linewidth=3, color="blue", xlabel="time", ylabel="position_2", label="states", size=(600,400))
scatter!((1:T).*Δt, observations[2,:], color="black", markersize=2, label="observed", legend=:bottomleft)
plot(p1,p2, layout=(2,1), size=(900,400))
```

```julia
p1 = plot((1:T).*Δt, states[3,:], linewidth=3, color="blue", xlabel="", ylabel="velocity_1", label="")
p2 = plot((1:T).*Δt, states[4,:], linewidth=3, color="blue", xlabel="time", ylabel="velocity_2", label="")
plot(p1,p2, layout=(2,1), size=(900,400))
```

### Kalman filters

A Kalman filter is a special case of a larger class of Bayesian filters: it is the state estimator in linear Gaussian dynamical system where all parameters and coefficients are known (or fixed).

The system is captured by the following state-space model:

$$\begin{align}
p(x_k | x_{k-1}) &= \mathcal{N}(x_k | Ax_{k-1}, Q) \\
p(y_k | x_k) &= \mathcal{N}(y_k | Cx_{k-1}, R) \, .
\end{align}$$

If we start with the following state prior,

$$p(x_0) = \mathcal{N}(x_0 | m_0, P_0) \, ,$$

then we can obtain a posterior recursively through the Bayesian filtering equations. The update and correction steps at each time point are:

$$\begin{align}
p(x_k | y_{1:k-1}) &= \int p(x_k | x_{k-1}) p(x_{k-1} | y_{1:k-1}) dx_{k-1} \\
p(x_k | y_{1:k}) &= \frac{1}{p(y_k | y_{1:k-1})} p(y_k | x_k) p(x_k | y_{1:k-1}) \, .
\end{align}$$

Using Gaussians, all these computations are tractable and we get the optimal state estimates in analytic form.

```julia
# State prior
x_0 = (zeros(Dx), diagm(ones(Dx)))

# Kalman filter
mx, Px = kalman_filter(observations, A, C, Q, R, x_0, time_horizon=T);
```

```julia
# Inspect data
p1 = plot((1:T).*Δt, states[1,:], linewidth=3, color="blue", xlabel="", ylabel="position_1", label="states", size=(700,400))
plot!((1:T).*Δt, mx[1,:], ribbon=sqrt.(Px[1,1,:]), linewidth=3, color="purple", label="inferred", ylabel="position_1", size=(700,400))
scatter!((1:T).*Δt, observations[1,:], color="black", markersize=2, label="observed", legend=:topleft)
p2 = plot((1:T).*Δt, states[2,:], linewidth=3, color="blue", xlabel="time", ylabel="position_2", label="states", size=(600,400))
plot!((1:T).*Δt, mx[2,:], ribbon=sqrt.(Px[2,2,:]), linewidth=3, color="purple", label="inferred", ylabel="position_1", size=(700,400))
scatter!((1:T).*Δt, observations[2,:], color="black", markersize=2, label="observed", legend=:bottomleft)
plot(p1,p2, layout=(2,1), size=(900,400))
```

```julia
p1 = plot((1:T).*Δt, states[3,:], linewidth=3, color="blue", xlabel="", ylabel="velocity_1", label="")
p2 = plot((1:T).*Δt, states[4,:], linewidth=3, color="blue", xlabel="time", ylabel="velocity_2", label="")
plot(p1,p2, layout=(2,1), size=(900,400))
```

```julia

```


## Design Matrix:

* Imputations

* Design Matrix:
$$
\begin{align}
t\ | \ y_t \xrightarrow{f} v_t, | \ \ x^1_t, \ x^2_t, \ \ldots \ x^k_t \xrightarrow{g} \ g_1x^1_t, \ g_2x^2_t, \ldots, g_kx^k_t
\end{align}
$$

$$
\text{DM}_t = \text{DM}_t(y_t, f, f^{-1}, \{x^i_t\}_{i=1,k}, \{g^i_t\}_{i=1,k})
$$

### Exploratory analysis

* Autocorrelation:
    * $ \text{ACF}(x_t) $
    * $ \text{PACF}(x_t) $

* Scatter Plots:
    * $y_t$ next to  $v_t$  and $x_t^{k}$ next to $g_kx^k_t$
    * $y_t \ \text{vs.} \ x^j_{t} $ for $j=1,k$ with LOWESS for dependency shape analysis
    * $x_t \ \text{vs.} \ x_{t-h}$ with LOWESS for autocorrelation analysis
    * $y_t \ \text{vs.} \ x^k_{t-h}$ for given $k$ with LOWESS for lagged-leading relationship

## Calibrator:

$$
 \mathbb{C} ( {\mathcal{H}yperParams}) \rightarrow \mathbb{C}
$$

## Model:
$$
\text{M} = \mathbb{M}(\text{C}, \ \text{DM} ) \to  \{\hat{\theta}_l\}_{l=1,m}
$$

### Model Specification

$$
\{\hat{\theta}_l\} \xrightarrow[\text{GridSearch}]{I(\theta): \ AIC, \ AICc, \ BIC }  \{\hat{\theta^*_l}\}
$$

### Residuals Diagnostics:
$$
\hat{\varepsilon}_t = v_{{t}}-\hat{v}_t; \ \hat{\varepsilon}_t^{std}; \ \hat{\varepsilon}_t^{stu}
$$

$$
\text{RD} = \mathcal{RD}(\hat{\varepsilon}_t, \hat{\varepsilon}_t^{std}, \hat{\varepsilon}_t^{stu}  )
$$

### Model Selection:

#### Cross Validation

$$
\text{CV} = \mathbb{CV}(\text{M}, \text{Partitioning}, \text{ Performance Metric})
$$

$$
\mathbb{C} \xrightarrow{\varepsilon_{CV}} \mathbb{C}^*
$$

## Forecast:

$$
\text{C}^*(DM_{t+1}=g_i({x^i_{t+1}})_{i=1,k} \ | \ \{\hat{\theta^*}_l\}_{l=1,m}) = \hat{v}_{t+1} \xrightarrow{f^{-1} , \  y_t} \hat{y}_{t+1}
$$

```python

```