## Subsurface Data Analytics

### Feature Transformations for Subsurface Data Analytics in Python

#### Michael Pyrcz, Associate Professor, University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

### Subsurface Machine Learning: Feature Transformations for Subsurface Data Analytics

Here's a demonstration of feature transformations for subsurface modeling in Python. This is part of my Subsuface Machine Learning Course at the Cockrell School of Engineering at the University of Texas at Austin.

#### Feature Transformations

There are many reasons that we may want to perform feature transformations.

* the make the features consistent for visualization and comparison

* to avoid bias or impose feature weighting for methods (e.g. k nearest neighbours regression) that rely on distances calculated in predictor feature space

* the method requires the variables to have a specific range or distribution:

    * artificial neural networks may require all features to range from [-1,1]
    * parital correlation coefficients require a Gaussian distribution.
    * statistical tests may require a specific distribution
    * geostatistical sequential simulation requires an indicator or Gaussian transform

Feature transformations is a common basic building blocks in many machine learning workflows.

* Let's learn how to perform feature transformations.

#### Objective

In the Stochastic Machine Learning class, I want to provide hands-on experience with solving complicated subsurface modeling problems with data analytics, machine learning. Python provides an excellent vehicle to accomplish this. I have coded a package called GeostatsPy with GSLIB: Geostatistical Library (Deutsch and Journel, 1998) functionality that provides basic building blocks for building subsurface modeling workflows.

The objective is to remove the hurdles of subsurface modeling workflow construction by providing building blocks and sufficient examples. This is not a coding class per se, but we need the ability to 'script' workflows working with numerical methods.

#### Getting Started

Here's the steps to get setup in Python with the GeostatsPy package:

1. Install Anaconda 3 on your machine (https://www.anaconda.com/download/).
2. From Anaconda Navigator (within Anaconda3 group), go to the environment tab, click on base (root) green arrow and open a terminal.
3. In the terminal type: pip install geostatspy.
4. Open Jupyter and in the top block get started by copy and pasting the code block below from this Jupyter Notebook to start using the geostatspy functionality.

You will need to copy the data file to your working directory.  They are available here:

* Tabular data - unconv_MV_v4.csv at https://git.io/fhHLT.

#### Importing Packages

We will import the GeostatsPy package.

* we will use a convenient transformation method

There are examples below with GeostatsPy functions. You can go here to see a list of the available functions, https://git.io/fh4eX, and for other example workflows and source code.

```python
import geostatspy.GSLIB as GSLIB          # GSLIB utilies, visualization and wrapper
import geostatspy.geostats as geostats    # GSLIB methods convert to Python
```

We will also need some standard packages. These should have been installed with Anaconda 3.

```python
import numpy as np                        # ndarrys for gridded data
import pandas as pd                       # DataFrames for tabular data
import os                                 # set working directory, run executables
import matplotlib.pyplot as plt           # for plotting
from scipy import stats                   # summary statistics
import math                               # trigonometry etc.
import scipy.signal as signal             # kernel for moving window calculation
import random                             # for randon numbers
import seaborn as sns                     # for matrix scatter plots
from scipy import linalg                  # for linear regression
from sklearn import preprocessing
```

#### Declare Functions

Let's define a single function to streamline plotting correlation matrices.

```python
def histograms(df):

    pormin = min(df['Por']); pormax = max(df['Por']);
    porname = 'Porosity (%)'; portitle = 'Porosity' # user specified min and max values, and labels for plotting
    permmin = min(df['Perm']); permmax = max(df['Perm']);
    permname = 'Permeability (mD)'; permtitle = 'Permeability'
    AImin = min(df['AI']); AImax = max(df['AI']);
    AIname = 'Acoustic Impedance (kg/m2s*10^6)'; AItitle = 'Acoustic Impedance'
    brmin = min(df['Brittle']); brmax = max(df['Brittle']);
    brname = 'Brittleness Ratio (%)'; brtitle = 'Brittleness'
    TOCmin = min(df['TOC']); TOCmax = max(df['TOC']);
    TOCname = 'Total Organic Carbon (%)'; TOCtitle = 'Total Organic Carbon'
    VRmin = min(df['VR']); VRmax = max(df['VR']);
    VRname = 'Vitrinite Reflectance (%)'; VRtitle = 'Vitrinite Reflectance'
    prodmin = min(df['Prod']); prodmax = max(df['Prod']);
    prodname = 'Normalized Initial Production (MCFPD)'; prodtitle = 'Normalized Initial Production'

    plt.subplot(331)
    GSLIB.hist_st(df['Por'].values,pormin,pormax,log=False,cumul = False,bins=20,weights = None,xlabel=porname,title=portitle)

    plt.subplot(332)
    GSLIB.hist_st(df['Perm'].values,permmin,permmax,log=False,cumul = False,bins=20,weights = None,xlabel=permname,title=permtitle)

    plt.subplot(333)
    GSLIB.hist_st(df['AI'].values,AImin,AImax,log=False,cumul = False,bins=20,weights = None,xlabel=AIname,title=AItitle)

    plt.subplot(334)
    GSLIB.hist_st(df['Brittle'].values,brmin,brmax,log=False,cumul = False,bins=20,weights = None,xlabel=brname,title=brtitle)

    plt.subplot(335)
    GSLIB.hist_st(df['TOC'].values,TOCmin,TOCmax,log=False,cumul = False,bins=20,weights = None,xlabel=TOCname,title=TOCtitle)

    plt.subplot(336)
    GSLIB.hist_st(df['VR'].values,VRmin,VRmax,log=False,cumul = False,bins=20,weights = None,xlabel=VRname,title=VRtitle)

    plt.subplot(338)
    GSLIB.hist_st(df['Prod'].values,prodmin,prodmax,log=False,cumul = False,bins=20,weights = None,xlabel=prodname,title=prodtitle)

    plt.subplots_adjust(left=0.0, bottom=0.0, right=3.0, top=3.5, wspace=0.2, hspace=0.2)
    plt.savefig('hist_Porosity_Multiple_bins.tif',dpi=600,bbox_inches="tight")
    plt.show()
```

#### Set the Working Directory

I always like to do this so I don't lose files and to simplify subsequent read and writes (avoid including the full address each time).

```python
os.chdir("c:/PGE383")                     # set the working directory
```

#### Loading Tabular Data

Here's the command to load our comma delimited data file in to a Pandas' DataFrame object.

```python
df = pd.read_csv('unconv_MV_v4.csv')      # load our data table (wrong name!)
```

Visualizing the DataFrame would be useful and we already learned about these methods in this demo (https://git.io/fNgRW).

We can preview the DataFrame by utilizing the 'head' DataFrame member function (with a nice and clean format, see below). With the head command, add parameter 'n=13' to see the first 13 rows of the dataset.

```python
df.head(n=13)                             # we could also use this command for a table preview
```

      Well
      Por
      Perm
      AI
      Brittle
      TOC
      VR
      Prod

      0
      1
      12.08
      2.92
      2.80
      81.40
      1.16
      2.31
      1695.360819

      1
      2
      12.38
      3.53
      3.22
      46.17
      0.89
      1.88
      3007.096063

      2
      3
      14.02
      2.59
      4.01
      72.80
      0.89
      2.72
      2531.938259

      3
      4
      17.67
      6.75
      2.63
      39.81
      1.08
      1.88
      5288.514854

      4
      5
      17.52
      4.57
      3.18
      10.94
      1.51
      1.90
      2859.469624

      5
      6
      14.53
      4.81
      2.69
      53.60
      0.94
      1.67
      4017.374438

      6
      7
      13.49
      3.60
      2.93
      63.71
      0.80
      1.85
      2952.812773

      7
      8
      11.58
      3.03
      3.25
      53.00
      0.69
      1.93
      2670.933846

      8
      9
      12.52
      2.72
      2.43
      65.77
      0.95
      1.98
      2474.048178

      9
      10
      13.25
      3.94
      3.71
      66.20
      1.14
      2.65
      2722.893266

      10
      11
      15.04
      4.39
      2.22
      61.11
      1.08
      1.77
      3828.247174

      11
      12
      16.19
      6.30
      2.29
      49.10
      1.53
      1.86
      5095.810104

      12
      13
      16.82
      5.42
      2.80
      66.65
      1.17
      1.98
      4091.637316

This dataset has features from 200 unconventional wells including:

0. well index
1. well average porosity (%)
2. permeability (mD)
3. accoustic impedance (kg/m2s*10^6)
4. brittness ratio (%)
5. total organic carbon (%)
6. vitrinite reflectance (%)
8. normalized initial production 90 day average (MCFPD).

Note, the dataset is synthetic, but has realistic ranges and general multivariate relationships.

Ranking features is really an effort to understand the features and their relationships with eachother.  We will start with basic data visualization and move to more complicated methods such are partial correlation and recursive feature elimination.

### Summary Statistics

In any multivariate work we should start with the univariate analysis, summary statistics of one variable at a time. The summary statistic ranking method is qualitative, we are asking:

* are there data issues?
* do we trust the features? do we we trust the features all equally?
* are there issues that need to be taken care of before we develop any multivariate workflows?

There are a lot of efficient methods to calculate summary statistics from tabular data in DataFrames. The describe() command provides count, mean, minimum, maximum, and quartiles all in a compact data table. We use transpose() command to flip the table so that features are on the rows and the statistics are on the columns.

```python
df.describe().transpose()
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      Well
      200.0
      100.500000
      57.879185
      1.000000
      50.750000
      100.500000
      150.250000
      200.000000

      Por
      200.0
      14.991150
      2.971176
      6.550000
      12.912500
      15.070000
      17.402500
      23.550000

      Perm
      200.0
      4.330750
      1.731014
      1.130000
      3.122500
      4.035000
      5.287500
      9.870000

      AI
      200.0
      2.968850
      0.566885
      1.280000
      2.547500
      2.955000
      3.345000
      4.630000

      Brittle
      200.0
      48.161950
      14.129455
      10.940000
      37.755000
      49.510000
      58.262500
      84.330000

      TOC
      200.0
      0.990450
      0.481588
      -0.190000
      0.617500
      1.030000
      1.350000
      2.180000

      VR
      200.0
      1.964300
      0.300827
      0.930000
      1.770000
      1.960000
      2.142500
      2.870000

      Prod
      200.0
      3864.407081
      1553.277558
      839.822063
      2686.227611
      3604.303507
      4752.637556
      8590.384044

Let's remove the well index as we will not want to transform it.

```python
df = df.iloc[:,1:]
df.head()
```

      Por
      Perm
      AI
      Brittle
      TOC
      VR
      Prod

      0
      12.08
      2.92
      2.80
      81.40
      1.16
      2.31
      1695.360819

      1
      12.38
      3.53
      3.22
      46.17
      0.89
      1.88
      3007.096063

      2
      14.02
      2.59
      4.01
      72.80
      0.89
      2.72
      2531.938259

      3
      17.67
      6.75
      2.63
      39.81
      1.08
      1.88
      5288.514854

      4
      17.52
      4.57
      3.18
      10.94
      1.51
      1.90
      2859.469624

We should also take a look at the histograms. We will use our convenient function (declared above) to visualize all the distribuitons.

```python
histograms(df)                       # our function to visualize all feature histograms
```

There are just a couple slighly negative values, let's just truncate them at zero. We   can use this command below to set all TOC values in the DataFrame that are less than 0.0 as 0.0, otherwise we keep the original TOC value.

### Truncation

It is possible that the features may extend beyond the palusible range of values.

* truncation is simply assigning values outside the range with a specific value

* it is common to assign the minimum permissible value to outliers on the lower tail and visa versa

Truncation can be handled easily with numpy operators applied to the feature array within the Pandas DataFrame

```python
num = df._get_numeric_data()
num[num < data_min] = data_min
num[num > data_max] = data_max
```

```python
num = df._get_numeric_data()              # get the numerical values
num[num < 0] = 0                          # truncate negative values to 0.0
df.describe().transpose()                 # get the summary statistics of all features
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      Por
      200.0
      14.991150
      2.971176
      6.550000
      12.912500
      15.070000
      17.402500
      23.550000

      Perm
      200.0
      4.330750
      1.731014
      1.130000
      3.122500
      4.035000
      5.287500
      9.870000

      AI
      200.0
      2.968850
      0.566885
      1.280000
      2.547500
      2.955000
      3.345000
      4.630000

      Brittle
      200.0
      48.161950
      14.129455
      10.940000
      37.755000
      49.510000
      58.262500
      84.330000

      TOC
      200.0
      0.991950
      0.478264
      0.000000
      0.617500
      1.030000
      1.350000
      2.180000

      VR
      200.0
      1.964300
      0.300827
      0.930000
      1.770000
      1.960000
      2.142500
      2.870000

      Prod
      200.0
      3864.407081
      1553.277558
      839.822063
      2686.227611
      3604.303507
      4752.637556
      8590.384044

Let's look at the summary statistics again to confirm that we were successful in truncating TOC to $\ge 0$.

```python
df.describe().transpose()
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      Por
      200.0
      14.991150
      2.971176
      6.550000
      12.912500
      15.070000
      17.402500
      23.550000

      Perm
      200.0
      4.330750
      1.731014
      1.130000
      3.122500
      4.035000
      5.287500
      9.870000

      AI
      200.0
      2.968850
      0.566885
      1.280000
      2.547500
      2.955000
      3.345000
      4.630000

      Brittle
      200.0
      48.161950
      14.129455
      10.940000
      37.755000
      49.510000
      58.262500
      84.330000

      TOC
      200.0
      0.991950
      0.478264
      0.000000
      0.617500
      1.030000
      1.350000
      2.180000

      VR
      200.0
      1.964300
      0.300827
      0.930000
      1.770000
      1.960000
      2.142500
      2.870000

      Prod
      200.0
      3864.407081
      1553.277558
      839.822063
      2686.227611
      3604.303507
      4752.637556
      8590.384044

From the summary statistics you can see that the truncation was successful, we now have a minimum of 0.0.

### Affine Correction

The affine correction is the transform of the feature distribution to a new mean and variance.

* this is a shift and stretch / squeeze of the original property distribution

* no shape change is assumed

The following equation is applied to each sample in the original distribution

\begin{equation}
y = \frac{\sigma_y}{\sigma_x}\left( x - \overline{x} \right) + \overline{y}
\end{equation}

where $\sigma_x$ is the original standard deviation, $\sigma_y$ is the target standard deviation, $\overline{x}$ is the original mean and $\overline{y}$ is the target mean.

* there is a affine function in GeostatsPy that we may use

* the function is not set up to accept multiple features at once so we will just transform a single feature

Let's transform porosity to have an arbitrary mean and standard deviation ($\overline{x} = 1.0$ and $\sigma_x = 1.0$)

```python
df['aPor'] = GSLIB.affine(df['Por'].values,tmean = 1.0,tstdev = 1.0)
df.head(n = 13)
```

      Por
      Perm
      AI
      Brittle
      TOC
      VR
      Prod
      aPor

      0
      12.08
      2.92
      2.80
      81.40
      1.16
      2.31
      1695.360819
      0.017744

      1
      12.38
      3.53
      3.22
      46.17
      0.89
      1.88
      3007.096063
      0.118968

      2
      14.02
      2.59
      4.01
      72.80
      0.89
      2.72
      2531.938259
      0.672323

      3
      17.67
      6.75
      2.63
      39.81
      1.08
      1.88
      5288.514854
      1.903875

      4
      17.52
      4.57
      3.18
      10.94
      1.51
      1.90
      2859.469624
      1.853263

      5
      14.53
      4.81
      2.69
      53.60
      0.94
      1.67
      4017.374438
      0.844403

      6
      13.49
      3.60
      2.93
      63.71
      0.80
      1.85
      2952.812773
      0.493495

      7
      11.58
      3.03
      3.25
      53.00
      0.69
      1.93
      2670.933846
      -0.150962

      8
      12.52
      2.72
      2.43
      65.77
      0.95
      1.98
      2474.048178
      0.166205

      9
      13.25
      3.94
      3.71
      66.20
      1.14
      2.65
      2722.893266
      0.412516

      10
      15.04
      4.39
      2.22
      61.11
      1.08
      1.77
      3828.247174
      1.016483

      11
      16.19
      6.30
      2.29
      49.10
      1.53
      1.86
      5095.810104
      1.404506

      12
      16.82
      5.42
      2.80
      66.65
      1.17
      1.98
      4091.637316
      1.617075

Let's check the summary statistics of our new feature.

```python
df['aPor'].describe().transpose()
```

    count    200.000000
    mean       1.000000
    std        1.002509
    min       -1.848142
    25%        0.298639
    50%        1.026605
    75%        1.813617
    max        3.887855
    Name: aPor, dtype: float64

Let's remove the affine transformed feature.

```python
df = df.drop(columns = ['aPor'])
```

### Standardization

Standardization is the transform of the feature distribution to a mean of zero and a variance of one.

* this is a shift and stretch / squeeze of the original property distribution

* no shape change is assumed

The transform is effectively a specific case of the affine correction, with $\overline{y} = 0$ and $\sigma_y = 1.0$.

\begin{equation}
y = \frac{1}{\sigma_x}\left( x - \overline{x} \right)
\end{equation}

where $\sigma_x$ is the original standard deviation and $\overline{x}$ is the original mean.

Let's standardize the feature to have:

* mean = 0.0
* variance = standard deviation = 1.0

To do this we:

1. instantiate the StandardScaler from scikit learn. We assign it as 'scaler' so we can use it to conveniently reverse the transformation if we like. We will need to do that to get our predictions back into regular production units.

```python
scaler = StandardScaler()
```

2. we then extract all the values from our DataFrame and apply the by-column standardization.  The result is a 2D ndarray

```python
sfeatures = scaler.fit_transform(df_mv.values)
```
3. we make an new empty DataFrame

```python
df_nmv = pd.DataFrame()
```

4. then we add the transformed value to the new DataFrame while keeping the sample index and feature names from the old DataFramae

```python
df_nmv = pd.DataFrame(sfeatures, index=df_mv.index, columns=df_mv.columns)
```

```python
from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()                                   # instantiate the scaler
sfeatures = scaler.fit_transform(df.values)              # standardize all the values extracted from the DataFrame
df_st = pd.DataFrame()                                     # instantiate a new DataFrame
df_st = pd.DataFrame(sfeatures, index=df.index, columns=df.columns) # copy the standardized values into the new DataFrame
df_st.head()                                               # preview the the new DataFrame
```

      Por
      Perm
      AI
      Brittle
      TOC
      VR
      Prod

      0
      -0.982256
      -0.817030
      -0.298603
      2.358297
      0.352257
      1.152048
      -1.399936

      1
      -0.881032
      -0.463751
      0.444147
      -0.141332
      -0.213702
      -0.280931
      -0.553322

      2
      -0.327677
      -1.008148
      1.841224
      1.748113
      -0.213702
      2.518377
      -0.859996

      3
      0.903875
      1.401098
      -0.599240
      -0.592585
      0.184565
      -0.280931
      0.919141

      4
      0.853263
      0.138561
      0.373409
      -2.640962
      1.085907
      -0.214280
      -0.648602

Let's close the loop and reverse the transformation and confirm that we get back to the original data.

To do this we:

5. call the 'fit' features' scaler's inverse transform function transformed value to the new DataFrame while keeping the sample index and feature names from the old DataFramae

```python
rfeatures = scaler.inverse_transform(df_st.values)
```

the output from this is a 2D numpy array.

6. We will put it into a new DataFrame.

```python
df_reverse = pd.DataFrame(rfeatures, index=df.index, columns=df.columns)
```

```python
rfeatures = scaler.inverse_transform(df_st.values)
df_reverse = pd.DataFrame(rfeatures, index=df.index, columns=df.columns) # copy the standardized values into the new DataFrame
df_reverse.head()
```

      Por
      Perm
      AI
      Brittle
      TOC
      VR
      Prod

      0
      12.08
      2.92
      2.80
      81.40
      1.16
      2.31
      1695.360819

      1
      12.38
      3.53
      3.22
      46.17
      0.89
      1.88
      3007.096063

      2
      14.02
      2.59
      4.01
      72.80
      0.89
      2.72
      2531.938259

      3
      17.67
      6.75
      2.63
      39.81
      1.08
      1.88
      5288.514854

      4
      17.52
      4.57
      3.18
      10.94
      1.51
      1.90
      2859.469624

You can confirm the result is the same as the original, prior to standardization DatatFrame.

We were just testing, so let's get rid of (delete) the new DataFrame.  We can use the following to delete an instantiated object in Python.

```python
del df_reverse
```

For more complicated workflows it may be a good idea to remove intermediate products to save memmory and to prevent clutter!

```python
del df_reverse
```

### Normalization

Also known as the min / max transform, recales the features to have a minimum of 0 and a maximum of 1.

\begin{equation}
y = \frac{x - min(x)}{max(x) - min(x)}
\end{equation}

where $min(x)$ and $max(x)$ are the minimum and maximum values for each feature.

* scikit learn has a built in min / max transform method

```python
min_max_scaler = preprocessing.MinMaxScaler()
scaled_array = min_max_scaler.fit_transform(float_array)
```

```python
norm_scaler = preprocessing.MinMaxScaler()                 # instantiate the scaler
nfeatures = norm_scaler.fit_transform(df.values)           # standardize all the values extracted from the DataFrame
df_n = pd.DataFrame(nfeatures, index=df.index, columns=df.columns) # copy the standardized values into the new DataFrame
df_n.head()
```

      Por
      Perm
      AI
      Brittle
      TOC
      VR
      Prod

      0
      0.325294
      0.204805
      0.453731
      0.960076
      0.532110
      0.711340
      0.110384

      1
      0.342941
      0.274600
      0.579104
      0.480038
      0.408257
      0.489691
      0.279628

      2
      0.439412
      0.167048
      0.814925
      0.842894
      0.408257
      0.922680
      0.218322

      3
      0.654118
      0.643021
      0.402985
      0.393378
      0.495413
      0.489691
      0.573983

      4
      0.645294
      0.393593
      0.567164
      0.000000
      0.692661
      0.500000
      0.260581

Let's check the summary statistics.

```python
df_n.describe().transpose()
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      Por
      200.0
      0.496538
      0.174775
      0.0
      0.374265
      0.501176
      0.638382
      1.0

      Perm
      200.0
      0.366219
      0.198057
      0.0
      0.227975
      0.332380
      0.475686
      1.0

      AI
      200.0
      0.504134
      0.169220
      0.0
      0.378358
      0.500000
      0.616418
      1.0

      Brittle
      200.0
      0.507180
      0.192526
      0.0
      0.365377
      0.525548
      0.644809
      1.0

      TOC
      200.0
      0.455023
      0.219387
      0.0
      0.283257
      0.472477
      0.619266
      1.0

      VR
      200.0
      0.533144
      0.155066
      0.0
      0.432990
      0.530928
      0.625000
      1.0

      Prod
      200.0
      0.390241
      0.200408
      0.0
      0.238229
      0.356681
      0.504843
      1.0

Let's close the loop and reverse the transformation (back-transform) and confirm that we get back to the original data.

```python
rfeatures = norm_scaler.inverse_transform(df_n.values)
df_reverse = pd.DataFrame()                                     # instantiate a new DataFrame
df_reverse = pd.DataFrame(rfeatures, index=df.index, columns=df.columns) # copy the standardized values into the new DataFrame
df_reverse.head()
```

      Por
      Perm
      AI
      Brittle
      TOC
      VR
      Prod

      0
      12.08
      2.92
      2.80
      81.40
      1.16
      2.31
      1695.360819

      1
      12.38
      3.53
      3.22
      46.17
      0.89
      1.88
      3007.096063

      2
      14.02
      2.59
      4.01
      72.80
      0.89
      2.72
      2531.938259

      3
      17.67
      6.75
      2.63
      39.81
      1.08
      1.88
      5288.514854

      4
      17.52
      4.57
      3.18
      10.94
      1.51
      1.90
      2859.469624

You can confirm the result is the same as the original, prior to standardization DataFrame.

Once again we were just testing, so let's delete the back-transformed DataFrame.

```python
del df_reverse
```

#### L1 / L2 Normalizer

Another type of normalization is performed independently on each sample to force the $L1$ or $L2$ norm to be 1.0.

For the L1 norm:

\begin{equation}
\sum^m_{i,\alpha = 1} x_{i, \alpha} = 1.0, \quad i = 1, \ldots, n
\end{equation}

where we have $x_{i, \alpha}, \alpha = 1, \dots, m$ features over $i = 1, \dots, n$ samples.

For the L2 norm:

\begin{equation}
\sum^m_{i,\alpha = 1}\left( x_{i, \alpha} \right)^2 = 1.0, \quad i = 1, \ldots, n
\end{equation}

where we have $x_{i, \alpha}, \alpha = 1, \dots, m$ features over $i = 1, \dots, n$ samples.

* this may be applied in text classification or clustering

We demonstrate the L1 and L2 normalizer below.

* there is no reverse to this transform

Let's start with the $L2$ norm.

```python
from sklearn.preprocessing import Normalizer
l2normalizer = Normalizer(norm = 'l2')
l2features = l2normalizer.fit_transform(df)                                                             # standardize all the values extracted from the DataFrame
df_nL2 = pd.DataFrame()                                      # instantiate a new DataFrame
df_nL2 = pd.DataFrame(l2features, index=df.index, columns=df.columns) # copy the standardized values into the new DataFrame
df_nL2.head()
```

      Por
      Perm
      AI
      Brittle
      TOC
      VR
      Prod

      0
      0.007117
      0.001720
      0.001650
      0.047957
      0.000683
      0.001361
      0.998820

      1
      0.004116
      0.001174
      0.001071
      0.015352
      0.000296
      0.000625
      0.999872

      2
      0.005535
      0.001022
      0.001583
      0.028740
      0.000351
      0.001074
      0.999569

      3
      0.003341
      0.001276
      0.000497
      0.007527
      0.000204
      0.000355
      0.999965

      4
      0.006127
      0.001598
      0.001112
      0.003826
      0.000528
      0.000664
      0.999972

Now let's demonstrate the $L1$ norm.

```python
from sklearn.preprocessing import Normalizer
l1normalizer = Normalizer(norm = 'l1')
l1features = l2normalizer.fit_transform(df)                                                             # standardize all the values extracted from the DataFrame
df_nL1 = pd.DataFrame()                                      # instantiate a new DataFrame
df_nL1 = pd.DataFrame(l1features, index=df.index, columns=df.columns) # copy the standardized values into the new DataFrame
df_nL1.head()
```

      Por
      Perm
      AI
      Brittle
      TOC
      VR
      Prod

      0
      0.007117
      0.001720
      0.001650
      0.047957
      0.000683
      0.001361
      0.998820

      1
      0.004116
      0.001174
      0.001071
      0.015352
      0.000296
      0.000625
      0.999872

      2
      0.005535
      0.001022
      0.001583
      0.028740
      0.000351
      0.001074
      0.999569

      3
      0.003341
      0.001276
      0.000497
      0.007527
      0.000204
      0.000355
      0.999965

      4
      0.006127
      0.001598
      0.001112
      0.003826
      0.000528
      0.000664
      0.999972

### Binary or Indictor Transform

For the many problems that we need to perform a binary transform to convert our continuous feature to 0's and 1's based on a threshold, $x_t$

* for the binary transform, $x_i = 0$ if $<= x_t$ and $x_i = 1$ otherwise
* for the indicator transform, $x_i = 1$ if $<= x_t$ and $x_i = 0$ otherwise

There is a scikit-learn function for the binary transform

```python
from sklearn.preprocessing import Binarizer
binarizer = Binarizer(threshold = 13.0)
bPor = binarizer.fit_transform(df['Por'].values.reshape(-1, 1))                                                         # standardize all the values extracted from the DataFrame
bPor
```

    array([[0.],
           [0.],
           [1.],
           [1.],
           [1.],
           [1.],
           [1.],
           [0.],
           [0.],
           [1.],
           [1.],
           [1.],
           [1.],
           [1.],
           [1.],
           [0.],
           [1.],
           [1.],
           [1.],
           [0.],
           [1.],
           [1.],
           [0.],
           [0.],
           [1.],
           [0.],
           [1.],
           [1.],
           [0.],
           [0.],
           [1.],
           [1.],
           [0.],
           [1.],
           [0.],
           [0.],
           [1.],
           [1.],
           [1.],
           [1.],
           [1.],
           [1.],
           [0.],
           [0.],
           [1.],
           [1.],
           [0.],
           [0.],
           [1.],
           [1.],
           [1.],
           [1.],
           [1.],
           [0.],
           [1.],
           [1.],
           [1.],
           [1.],
           [1.],
           [0.],
           [1.],
           [1.],
           [1.],
           [1.],
           [1.],
           [1.],
           [1.],
           [1.],
           [1.],
           [0.],
           [0.],
           [1.],
           [1.],
           [1.],
           [1.],
           [1.],
           [1.],
           [0.],
           [1.],
           [1.],
           [1.],
           [1.],
           [1.],
           [1.],
           [0.],
           [1.],
           [1.],
           [1.],
           [0.],
           [1.],
           [1.],
           [0.],
           [1.],
           [0.],
           [1.],
           [1.],
           [1.],
           [1.],
           [0.],
           [1.],
           [1.],
           [1.],
           [0.],
           [1.],
           [1.],
           [1.],
           [1.],
           [1.],
           [1.],
           [1.],
           [1.],
           [0.],
           [1.],
           [1.],
           [1.],
           [1.],
           [0.],
           [1.],
           [0.],
           [1.],
           [1.],
           [0.],
           [1.],
           [1.],
           [1.],
           [1.],
           [1.],
           [1.],
           [0.],
           [1.],
           [1.],
           [1.],
           [0.],
           [1.],
           [0.],
           [1.],
           [1.],
           [1.],
           [0.],
           [1.],
           [1.],
           [1.],
           [1.],
           [0.],
           [1.],
           [1.],
           [1.],
           [1.],
           [1.],
           [1.],
           [0.],
           [0.],
           [1.],
           [1.],
           [1.],
           [1.],
           [1.],
           [1.],
           [1.],
           [1.],
           [1.],
           [1.],
           [1.],
           [0.],
           [1.],
           [1.],
           [1.],
           [0.],
           [1.],
           [0.],
           [0.],
           [1.],
           [0.],
           [0.],
           [1.],
           [1.],
           [1.],
           [0.],
           [1.],
           [0.],
           [1.],
           [1.],
           [0.],
           [1.],
           [1.],
           [1.],
           [1.],
           [0.],
           [1.],
           [1.],
           [1.],
           [0.],
           [1.],
           [0.],
           [1.],
           [0.],
           [1.],
           [0.],
           [1.],
           [1.]])

### K Bins Discretization

With K bins discretization we bin the range of the feature into K bins and then expand each sample for our continuous feature to k features with the assignment of a value of 1 if the sample is within a bin and 0 if outsize the bin

* strategies include uniform width bins (uniform) and uniform number of data in each bin (quantile)

Let's make 5 uniform bins and then concatenate the original porosity values so we can compare the original porosity values and our K bins discretization.

```python
from sklearn.preprocessing import KBinsDiscretizer
kbins = KBinsDiscretizer(n_bins=5, strategy='uniform')
kbins_por = kbins.fit_transform(df['Por'].values.reshape(-1, 1))                                                         # standardize all the values extracted from the DataFrame
np.concatenate((df['Por'].values.reshape(-1, 1),kbins_por.toarray()),axis=1)
```

    array([[12.08,  0.  ,  1.  ,  0.  ,  0.  ,  0.  ],
           [12.38,  0.  ,  1.  ,  0.  ,  0.  ,  0.  ],
           [14.02,  0.  ,  0.  ,  1.  ,  0.  ,  0.  ],
           ...,
           [12.12,  0.  ,  1.  ,  0.  ,  0.  ,  0.  ],
           [15.55,  0.  ,  0.  ,  1.  ,  0.  ,  0.  ],
           [20.89,  0.  ,  0.  ,  0.  ,  0.  ,  1.  ]])

Let's look at the bin boundaries to better understand the result above.

```python
kbins.bin_edges_
```

    array([array([ 6.55,  9.95, 13.35, 16.75, 20.15, 23.55])], dtype=object)

Spot check the first sample, $12.08\%$ porosity is $\in [9.95\%,13.35\%]$ so we have a 1 in the second bin (second column in our table) and zeros in the other bins.

### Gaussian Transform / Gaussian Anamorphosis

This is a quantile transformation of the feature to follow a Gaussian distribution. We actually map each of the quantiles from the original feature distribution to the target distribution. This is required for a wide range of statistical methods.

\begin{equation}
y = G_y^{-1}(F_x(x))
\end{equation}

where $G_y$ is the Gaussian cumulative distribution function and $F_x$ is the original feature cumulative distribution function.

```python
from sklearn.preprocessing import QuantileTransformer
nscore = QuantileTransformer(n_quantiles=100, random_state=73, output_distribution = 'normal')
nsfeatures = nscore.fit_transform(df)                                                             # standardize all the values extracted from the DataFrame
df_ns = pd.DataFrame()                                      # instantiate a new DataFrame
df_ns = pd.DataFrame(nsfeatures, index=df.index, columns=df.columns) # copy the standardized values into the new DataFrame
df_ns.head()
```

      Por
      Perm
      AI
      Brittle
      TOC
      VR
      Prod

      0
      -0.966707
      -0.816627
      -0.308666
      2.703111
      0.305481
      1.123308
      -1.771156

      1
      -0.870846
      -0.383621
      0.449229
      -0.203734
      -0.275431
      -0.348756
      -0.401986

      2
      -0.321971
      -1.077879
      1.746017
      2.051587
      -0.275431
      2.324341
      -0.837169

      3
      0.735470
      1.339654
      -0.559592
      -0.592008
      0.114185
      -0.348756
      0.829950

      4
      0.702820
      0.295415
      0.362601
      -5.199338
      1.036433
      -0.269066
      -0.532220

Let's visualize the Gaussian transformed feature histograms.

* we should see a nice bell shape centered on 0.0 with a range of about -4 to 4.

```python
histograms(df_ns)
```

Once again, let's check out the reverse transform.  We just reverse the previous quantile transform

\begin{equation}
x = F_x^{-1}(G_y(y))
\end{equation}

```python
rfeatures = nscore.inverse_transform(df_ns.values)
df_reverse = pd.DataFrame()                                     # instantiate a new DataFrame
df_reverse = pd.DataFrame(rfeatures, index=df.index, columns=df.columns) # copy the standardized values into the new DataFrame
df_reverse.head()
```

      Por
      Perm
      AI
      Brittle
      TOC
      VR
      Prod

      0
      12.08
      2.92
      2.80
      81.40
      1.16
      2.31
      1695.360819

      1
      12.38
      3.53
      3.22
      46.17
      0.89
      1.88
      3007.096063

      2
      14.02
      2.59
      4.01
      72.80
      0.89
      2.72
      2531.938259

      3
      17.67
      6.75
      2.63
      39.81
      1.08
      1.88
      5288.514854

      4
      17.52
      4.57
      3.18
      10.94
      1.51
      1.90
      2859.469624

We can also perform a uniform transform. Let's show the quntile transformation to a uniform distribution.

```python
from sklearn.preprocessing import QuantileTransformer
uniform = QuantileTransformer(n_quantiles=100, random_state=73, output_distribution = 'uniform')
unifeatures = uniform.fit_transform(df)                                                             # standardize all the values extracted from the DataFrame
df_uni = pd.DataFrame()                                      # instantiate a new DataFrame
df_uni = pd.DataFrame(unifeatures, index=df.index, columns=df.columns) # copy the standardized values into the new DataFrame
df_uni.head()
```

      Por
      Perm
      AI
      Brittle
      TOC
      VR
      Prod

      0
      0.166845
      0.207071
      0.378788
      0.996565
      0.620000
      0.869347
      0.038267

      1
      0.191919
      0.350630
      0.673367
      0.419281
      0.391493
      0.363636
      0.343847

      2
      0.373737
      0.140544
      0.959596
      0.979895
      0.391493
      0.989946
      0.201249

      3
      0.768973
      0.909821
      0.287879
      0.276923
      0.545455
      0.363636
      0.796716

      4
      0.758916
      0.616162
      0.641549
      0.000000
      0.850000
      0.393939
      0.297287

Let's visualize the uniform transformed feature histograms.

* we should see a uniform frequencies (with some noise due to limited sampling) $\in [0,1]$.

```python
histograms(df_uni)
```

We now have features with uniform distributions $[0,1]$.

### Custom Transformers

We can also create our own custom transform.  We can specify our own transform within a scikit learn preprocessing function

* this allows us to have a convenient method for forward and reverse transforms of our features as we have seen above

Let's demonstrate with the natural log for the foward transform and the exponential for the reverse transform.

\begin{equation}
y = log(x)
\end{equation}

\begin{equation}
x = exp(y)
\end{equation}

```python
custom_transformer = FunctionTransformer(func = np.log, inverse_func = np.exp)
```

```python
from sklearn.preprocessing import FunctionTransformer
custom_transformer = FunctionTransformer(func = np.log, inverse_func = np.exp, check_inverse = True, validate=True)
custom_features = custom_transformer.fit_transform(df['Perm'].values.reshape(-1, 1))                                                             # standardize all the values extracted from the DataFrame
df_custom = pd.DataFrame(custom_features, columns = ['LogPerm'])                 # instantiate a new DataFrame
df_custom.head()
```

      LogPerm

      0
      1.071584

      1
      1.261298

      2
      0.951658

      3
      1.909543

      4
      1.519513

Let's demonstrate that our custom transform is reversible.

```python
rfeatures = custom_transformer.inverse_transform(df_custom.values)
df_reverse = pd.DataFrame(rfeatures, index=df_custom.index, columns=['Perm']) # copy the standardized values into the new DataFrame
df_reverse.head()
```

      Perm

      0
      2.92

      1
      3.53

      2
      2.59

      3
      6.75

      4
      4.57

Compare the back-transformed permeability values to the original dataset.  The reverse transform that we speficied with our custom transformation works!

#### Comments

This was a basic demonstration of feature transformations. This might not seem very interesting at this time, but these transformations are important building blocks of workflow construction for data analytics and machine learning!

I have other demonstrations on the basics of working with DataFrames, ndarrays, univariate statistics, plotting data, declustering, data transformations, trend modeling and many other workflows available at https://github.com/GeostatsGuy/PythonNumericalDemos and https://github.com/GeostatsGuy/GeostatsPy.

I hope this was helpful,

*Michael*

#### The Author:

### Michael Pyrcz, Associate Professor, University of Texas at Austin
*Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions*

With over 17 years of experience in subsurface consulting, research and development, Michael has returned to academia driven by his passion for teaching and enthusiasm for enhancing engineers' and geoscientists' impact in subsurface resource development.

For more about Michael check out these links:

#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

#### Want to Work Together?

I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.

* Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I'd be happy to drop by and work with you!

* Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!

* I can be reached at mpyrcz@austin.utexas.edu.

I'm always happy to discuss,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin

#### More Resources Available at: [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

```python

```


# This notebook shows retrieve a model and data from an model dump file.
The model is then simulated and the results are displayed.

## Why specify a model in Latex?

Sometime the **implementation** of a model in software don't match the **specification** of the model in
the text in which the model is presented. It can be a chalange to make sure that the specification is
updated in order to reflect changes made in the implementation.

By extracting the model from a Latex script which describes and specify the model a one can always be sure that simulations reflect the model as described in the paper.

Also the author is forced to make a complete specification of the model, else it won't run.

## The Economic Credit Loss model
This jupyter notebook is inspired by IMF working paper (WP/20/111) The Expected Credit Loss Modeling from a Top-Down Stress Testing Perspective by Marco Gross, Dimitrios Laliotis, Mindaugas Leika, Pavel Lukyantsau. The working paper and the associated material is located https://www.imf.org/en/Publications/WP/Issues/2020/07/03/Expected-Credit-Loss-Modeling-from-a-Top-Down-Stress-Testing-Perspective-49545

from the abstract of the paper:
> The objective of this paper is to present an integrated tool suite for IFRS 9- and CECL compatible
estimation in top-down solvency stress tests. The tool suite serves as an
illustration for institutions wishing to include accounting-based approaches for credit risk
modeling in top-down stress tests.

This is a jupyter notebook build with the purpose of illustrating the use of ModelFlow. The purpose is testing so take very much care.

## Import libraries

```python
%load_ext autoreload
%autoreload 2
import pandas as pd
from IPython.core.display import HTML,Markdown,Latex
from ipywidgets import interact,Dropdown,Checkbox

from modelclass import model
import modeljupytermagic
import modeljupyter as mj

# some useful stuf
model.widescreen()
pd.set_option('display.max_rows', None, 'display.max_columns', 10, 'precision', 2)
sortdf = lambda df: df[sorted([c for c in df.columns])]
```

## Fetch the precooked model and baseline
The model and data has been specified in another Jupyter notebook:

[file/Model for economic credit loss with Z score transition nynamic - setup]

```python
ecl,baseline  = model.modelload('ecl.pcim')
```

## The modelspecification - in latex
This is the rendering of the latex input, which has specified the model.

So the model is extracted from the Latex script, which produces the output below

```python
Markdown(ecl.equations_latex)
```

### Loans can be in 3 stages
Loans can be in 3 stages, s1,s2 and s3.
New loans will be generated and loans will mature.

Two lists of stages are defined:

List $stage=\{s1, s2,s3\}$

List $stage\_from=\{s1, s2,s3\}$

### A share of the loans in each stage wil transition to the same or another stage in the next time frame:

\begin{equation}
\label{eq:loanfromto}
loan\_transition\_from\_to^{stage\_from,stage}_{t} = loan^{stage\_from}_{t-1} \times TR^{stage\_from,stage}_{t}
\end{equation}

\begin{equation}
\label{eq:transition}
loan\_transition\_to^{stage}_{t}  = \underbrace{\sum_{stage\_from}(loan\_transition\_from\_to^{stage\_from,stage}_{t})}_{Transition}
\end{equation}

### A share of the loans in each stage will mature another share will have to be written off

\begin{equation}
\label{eq:maturing}
loan\_maturing^{stage}_{t}  = M^{stage}_{t} \times loan^{stage}_{t-1}
\end{equation}

\begin{equation}
\label{eq:writeoff}
loan\_writeoff^{stage}_{t}  = WRO^{stage}_{t} \times loan^{stage}_{t-1}
\end{equation}

### So loans in a stage will reflect the inflow and outflow

\begin{equation}
\label{eq:E}
loan^{stage}_{t}  = \underbrace{loan\_transition\_to^{stage}_{t} }_{Transition}
-\underbrace{loan\_maturing^{stage}_{t}}_{Maturing}
-\underbrace{loan\_writeoff^{stage}_{t}}_{Writeoff}
+\underbrace{loan\_new^{stage}_{t}}_{New loans}
\end{equation}

\begin{equation}
\label{eq:new}
loan\_new^{stage}_{t}  = new^{stage}_{t} \times loan^{stage}_{t-1}
\end{equation}

\begin{equation}
\label{eq:g}
loan\_total_{t}  =  \sum_{stage}(loan^{stage}_{t})
\end{equation}

### New loans are only generated in stage 1.

\begin{equation}
\label{eq:E2}
new\_s1_{t} = \frac{loan\_growth_{t} \times loan\_total_{t-1} +
                \sum_{stage}((M^{stage}_{t}+WRO^{stage}_{t})\times loan^{stage}_{t-1})}{(loan\_s1_{t-1})}
\end{equation}

### Performing Loans

\begin{equation}
\label{eq:Performing}
loan\_performing_{t} = loan\_s1_{t}+loan\_s2_{t}
\end{equation}

### Cure

\begin{equation}
\label{eq:cure}
loan\_cure = loan\_transition\_from\_to\_s3\_s1+loan\_transition\_from\_to\_s3\_s2
\end{equation}

### Probability of default (PD)

The point in time PD is the fraction of loans in stage s1 and s2 going into stage s3
\begin{equation}
\label{eq:PDPIT}
PD\_pit= \frac{loan\_transition\_from\_to\_s1\_s3+loan\_transition\_from\_to\_s2\_s3}{loan\_s1+loan\_s2}
\end{equation}

The Troug The  Cycle PD is a slow mowing average of the Point in time PD. The

\begin{equation}
\label{eq:PDTTC}
PD\_TTC = logit^{-1}(logit(PD\_TTC(-1)) + alfa \times \Delta{PD\_pit})
\end{equation}

### And we can specify the dynamic of the transition matrix, based on Z score

Let $\Phi$ be the normal cumulative distribution $\frac{1}{\sqrt{2\pi}}
\int_{-\infty}^x e^{
-\frac{t^2}{2}}dt$

\begin{equation}
\label{eq:tr}
TR^{stage\_from,stage}_{t} = \Phi{\left(\frac{bound\_upper^{stage\_from,stage}-\sqrt{\rho}\times Z_{t}}{\sqrt{1-\rho}}\right)}-\Phi{\left(\frac{bound\_lower^{stage\_from,stage}-\sqrt{\rho}\times Z_{t}}{\sqrt{1-\rho}}\right)}
\end{equation}

## Input values for the adverse scenario

```python
%%dataframe projection_adverse show
LOAN_GROWTH	M_S1	M_S2	WRO_S3	Z
0	1.00e-02	0.04	0.03	0.06	0.00
1	-1.00e-02	0.04	0.03	0.06	-0.65
2	-8.00e-03	0.04	0.03	0.06	-0.84
3	-6.00e-03	0.04	0.03	0.06	-0.99
4	-4.00e-03	0.04	0.03	0.06	-0.69
5	-2.00e-03	0.04	0.03	0.06	-0.39
6	-0.00e+00	0.04	0.03	0.06	-0.24
```

      LOAN_GROWTH
      M_S1
      M_S2
      WRO_S3
      Z

      index

      2021
      1.00e-02
      0.04
      0.03
      0.06
      0.00

      2022
      -1.00e-02
      0.04
      0.03
      0.06
      -0.65

      2023
      -8.00e-03
      0.04
      0.03
      0.06
      -0.84

      2024
      -6.00e-03
      0.04
      0.03
      0.06
      -0.99

      2025
      -4.00e-03
      0.04
      0.03
      0.06
      -0.69

      2026
      -2.00e-03
      0.04
      0.03
      0.06
      -0.39

      2027
      -0.00e+00
      0.04
      0.03
      0.06
      -0.24

## Create an adverse Dataframe

```python
adverse = baseline.copy()
adverse.update(projection_adverse)
```

## Run Model
For the Baseline and the adverse scenario

```python
base_result  = ecl(baseline,keep='Baseline')
adverse_result = ecl(adverse,keep = 'Adverse')
```

    Will start calculating: testmodel
    2022  solved
    2023  solved
    2024  solved
    2025  solved
    2026  solved
    2027  solved
    testmodel calculated
    Will start calculating: testmodel
    2022  solved
    2023  solved
    2024  solved
    2025  solved
    2026  solved
    2027  solved
    testmodel calculated

## Inspect Results
The ecl instance of the model class has a number of methods and properties wich allows the analysis and display of results.

### The difference in exogeneous variables (the input variable)

```python
ecl.exodif()
```

      LOAN_GROWTH
      M_S1
      M_S2
      WRO_S3
      Z

      2021
      0.00
      -0.01
      -8.00e-03
      -0.01
      0.00

      2022
      -0.02
      -0.01
      -8.00e-03
      -0.01
      -0.18

      2023
      -0.02
      -0.01
      -8.00e-03
      -0.01
      -0.42

      2024
      -0.02
      -0.01
      -8.00e-03
      -0.01
      -0.61

      2025
      -0.01
      -0.01
      -8.00e-03
      -0.01
      -0.33

      2026
      -0.01
      -0.01
      -8.00e-03
      -0.01
      -0.05

      2027
      -0.01
      -0.01
      -8.00e-03
      -0.01
      0.09

### Loan growth

```python
with ecl.set_smpl('2021','2027'):
    ecl.keep_plot('loan_total',showtype='growth',legend=0);
```

### Transition probabilities

```python
ecl.keep_plot('tr_*   ',showtype='level',legend=False);
```

### We can look at the structure of the model

```python
ecl.drawmodel(HR=0)
```

    Error in generation graph picture:
    Error: graph\A_model_graph.gv: syntax error in line 1 near 'Digraph'

    Perhaps it is already open - then close the application
    Error in generation graph picture:
    Error: graph\A_model_graph.gv: syntax error in line 1 near 'Digraph'

    Perhaps it is already open - then close the application

### Or just one variable and trace the variable preceding it
And look at their values

```python
with ecl.set_smpl(2021,2027):
    ecl.loan_s3.tracepre(all=1,HR=0,up=0)
```

### Or use an interactive inspector to find find the attribution to individual

```python
@interact
def explain(Variable=Dropdown(options = sorted(ecl.endogene),value='LOAN_TOTAL'),
            Levels=[0,1,2,3]):
    mj.get_frml_latex(ecl,Variable)
    _ = ecl.explain(Variable,up=Levels,svg=1,dec=2,size=(10,10),des=1)
```

### What exogenous input drives Probability of Default (point in time - PD_PIT)

We want to find out how much of the difference in the results from the baseline and adverse
which can be explained by the different exogenous variables.

This requires shocking each exogenous variable each period, simulating the model for each experiment and save the results.

All done by calling one method:

```python
ecl.totexplain('PD_PIT');
```

Examples/Economic Credit Loss/ECL Z - just run.ipynb


IPython

A multi-language architecture for interactive computing

<p class="gap05"

[ipython.org](http://ipython.org)

<p class="gap05"

 Fernando Pérez

[fperez.org](http://fperez.org), [@fperez_org](http://twitter.com/fperez_org)

U.C. Berkeley

<p class="gap2"

```
%pylab inline
%run talktools
```

    Populating the interactive namespace from numpy and matplotlib

Why IPython?

*"The purpose of computing is insight, not numbers"*

Hamming '62

## The Lifecycle of a Scientific Idea (schematically)

1. **Individual** exploratory work
2. **Collaborative** development
3. **Parallel** production runs (HPC, cloud, ...)
4. **Publication** (with reproducible results!)
5. **Education**
6. Goto 1.

From a better shell...

## 2001, instead of a Physics dissertation

... to a fancier shell (and a protocol!)...

## 2010, E Patterson, Enthought (key support!)

... 6 notebook attempts over 6 years...

## T. Matev, T. Alatalo, R. Kern, Min RK, J. Gao, B. Granger

# ... up to today's IPython Notebook

## Everyday exploratory computing, with annotations

Let's plot the Bessel functions $J_i$ for $i=0,3,6,9$:

```
import scipy.special as spec
x = linspace(0, 20, 200)
for n in range(0,13,3):
    plot(x, spec.jn(n, x), label=r'$J_{%i}(x)$' % n)
grid()
legend()
title('Bessel Functions are neat');
```

# Explore parameters interactively

```
@interactive(x=(1, 10))
def f(x):
    print 'X is:', x
```

    X is: 5

```
from sklearn import datasets

digits = datasets.load_digits()
n = len(digits.images)

@interactive(i=(0,n-1))
def view_image(i):
    print 'Classif. Label:', digits.target[i]
    plt.matshow(digits.images[i], cmap=cm.gray_r)
    plt.show()
```

Displaying results: a rich protocol

```
Image('fig/logo.png')
```

```
Video('fig/animation.m4v')  # Credit: Chris Kees, Army ERDC; created with Proteus.
```

# Browser multimedia + scientific visualization

```
plot_audio('voice.wav')
Audio('voice.wav')
```

# LaTeX and symbolic mathematics support

```
Math(r'F(k) = \int_{-\infty}^{\infty} f(x) e^{2\pi i k} dx')
```

$$F(k) = \int_{-\infty}^{\infty} f(x) e^{2\pi i k} dx$$

```
from sympy import symbols, Eq, factor, init_printing, expand
init_printing(use_latex=True)
x, y = symbols("x y")
```

```
eq = ((x+y)**3 * (x+1))
display(Eq(eq, expand(eq)))
```

```
@interactive(n=(1,10))
def _(n):
    eq = ((x+y)**n * (x+1))
    display(Eq(eq, expand(eq)))
```

# Dynamic graph visualizations with D3

# Talk to other languages: R (or Octave)

```
X = np.array([0,1,2,3,4])
Y = np.array([3,5,4,6,7])
# Now, load R support
%load_ext rmagic
```

```r
%%R -i X,Y -o XYcoef
XYlm = lm(Y~X)
XYcoef = coef(XYlm)
print(summary(XYlm))
par(mfrow=c(2,2))
plot(XYlm)
```

# Julia: seamless 2-way communication

```
%load_ext julia.magic
%julia @pyimport matplotlib.pyplot as plt
%julia @pyimport numpy as np
```

    Initializing Julia interpreter. This may take some time...

```julia
%%julia
# Note how we mix numpy and julia:
x = linspace(0,2*pi,1000); # use the julia linspace
y = sin(3*x + 4*np.cos(2*x)); # use the numpy cosine and julia sine
plt.plot(x, y, color="red", linewidth=2.0, linestyle="--")
```

# Julia and Python
## The call stack as a layer cake

```
jfib = %julia jfib(n, fib) = n < 2 ? n : fib(n-1, jfib) + fib(n-2, jfib)

def pyfib(n, fib):
    if n < 2:
         return n
    return fib(n-1, pyfib) + fib(n-2, pyfib)

pyfib(20, jfib)   # Credit: Steven Johnson, MIT.
```

# Fortran integration

```
%load_ext fortranmagic
```

```
%%fortran
subroutine f1(x, y, n)
    real, intent(in), dimension(n) :: x
    real, intent(out), dimension(n) :: y
    !intent(hide) :: n
    y = sin(x**2)
end subroutine f1
```

```
x = np.linspace(0, 2*pi, 300)
plot(x, f1(x));
```

nbconvert: notebook manipulation tools

* Library for format conversions
* HTML, LaTeX, PDF output.
* Basic command-line usage.
* Rich customization possibilities.

NBViewer: Share notebooks online

## Matthias Bussonnier, 2012

```
website('nbviewer.ipython.org')
```

nbviewer.ipython.org

Reproducible research...

## Min RK, Brian, Greg Caporaso, Rob Knight (group), Justin Riley

```
website('http://www.nature.com/ismej/journal/v7/n3/full/ismej2012123a.html', 'Paper:')
```

Paper:

# Notebooks and AMI for reproducibility:

```
website('http://qiime.org/home_static/nih-cloud-apr2012', 'Companion Website:')
```

Companion Website:

Notebook-based technical blogs

## Jake VanderPlas (astronomy/ML at UW)

```
website('http://jakevdp.github.io/blog/2013/12/05/static-interactive-widgets',
        'Pythonic Perambulations')
```

Pythonic Perambulations

Training IBM Watson

## Using Notebook and `IPython.parallel`

* **From:** 8000 lines of Java, JS and HTML5, 2min/query
* **To:** 200 lines of Python, 2sec/query

```
YouTubeVideo('tlontoyWX70',start='800', width=600, height=400)
```

Books: "Literate Computing"

Python for Signal Processing, By [Jose Unpingco](https://github.com/unpingco/Python-for-Signal-Processing)

Springer hardcover book

Each chapter is an IPython Notebook

Posted as a blog entry

And all available as a Github repo

... "Probabilistic Programming and Bayesian Methods for Hackers"

By [Cam Davidson Pilon](http://www.camdp.com).

```
website('camdavidsonpilon.github.io/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers',
        'Probabilistic programming...')
```

Probabilistic programming...

... "Mining the Social Web"

By [Matthew Russell](https://github.com/ptwobrussell).

O'Reilly book

Companion Notebook collection

All available as a Github repo

# CS 109: Data Science at Harvard

```
website('http://cs109.org/homework/homework.php', 'CS 109 Homeworks')
```

CS 109 Homeworks

# A growing (and amazing) body of work

 For much more: [http://ipython.org/gallery](http://ipython.org/gallery)

A simple and generic architecture

# `IPython.parallel`

Image credit: Continuum Analytics.

A protocol that enables kernels in other languages

* [IJulia](http://nbviewer.ipython.org/url/jdj.mit.edu/~stevenj/IJulia%20Preview.ipynb)
* [IHaskell](http://nbviewer.ipython.org/github/gibiansky/IHaskell/blob/master/demo/IHaskell.ipynb)
* [IFSharp](http://nbviewer.ipython.org/github/BayardRock/IfSharp/blob/master/Feature%20Notebook.ipynb)
* [IRuby](http://nbviewer.ipython.org/github/minad/iruby/blob/master/IRuby-Example.ipynb)
* [IGo](https://github.com/takluyver/igo)
* [IScala](https://github.com/mattpap/IScala)
* In planning stages: R and Matlab.

The interactive widget architecture is also 100% language-agnostic

# A healthy ecosystem, OSS and commercial

# Emacs IPython Notebook Client
## Takafumi Arakaki

# Vim client
## Paul Ivanov @Berkeley

# Enthought Canopy

# Microsoft Python Tools for Visual Studio and Azure
### Shahrokh Mortazavi, Dino Viehland, Wenming Ye, Dennis Gannon. Thanks!!

Microsoft Azure (S. Mortazavi, W. Ye)

# StarCluster
## Justin Riley @ MIT

# Continuum Analytics: Wakari

# Cloud.sagemath.com: UW startup, W. Stein

# Plot.ly: interactive JS plots in Python

```
website('http://nbviewer.ipython.org/gist/jackparmer/7729584', 'Rosling Countries',
        width=900, height=600)
```

Rosling Countries

## The Core IPython team (2013)

**Plus** [Jonathan Frederic](https://github.com/jdfreder) and [Zach Sailer](https://github.com/Zsailer)!

## An incomplete cast of characters

* *Brian Granger* - Physics, Cal State San Luis Obispo
* *Min Ragan-Kelley* - Nuclear Engineering, UC Berkeley
* *Matthias Bussonnier* - Physics, Institut Curie, Paris
* *Jonathan March*- Enthought
* *Thomas Kluyver* - Biology, U. Sheffield
* *Jörgen Stenarson* - Elect. Engineering, Sweden.
* *Paul Ivanov* - Neuroscience, UC Berkeley.
* *Robert Kern* - Enthought
* *Evan Patterson* - Physics, Caltech/Enthought
* *Brad Froehle* - Mathematics, UC Berkeley
* Stefan van der Walt - UC Berkeley
* John Hunter - TradeLink Securities, Chicago.
* Prabhu Ramachandran - Aerospace Engineering, IIT Bombay.
* Satra Ghosh- MIT Neuroscience
* Gaël Varoquaux - Neurospin (Orsay, France)
* Ville Vainio - CS, Tampere University of Technology, Finland
* Barry Wark - Neuroscience, U. Washington.
* Ondrej Certik - Physics, LANL
* Darren Dale - Cornell
* Justin Riley - MIT
* Mark Voorhies - UC San Francisco
* Nicholas Rougier - INRIA Nancy Grand Est
* Thomas Spura - Fedora project

Many more! (~220 commit authors)

# Public "Lab meetings on air"

```
YouTubeVideo('UUjTAq8cCcs', width=600, height=500)
```

# Current IPython funding

**Note:** We're hiring! Machine learning web pipelines for time-series analysis (astro, geo, neuro). With Josh Bloom (UCB Astro), NSF funding. Talk to me!

# Prior IPython support, thanks!!

* **Enthought**, Austin, TX: Lots!
* **Microsoft**: WinHPC support, Visual Studio integration, Azure
* **DoD/DRC Inc**: 2011/12 (thanks to Jose Unpingco and Chris Keees).
* Indirect: NIH via NiPy grant, NSF via Sage grant.
* Google: summer of code 2005, 2010.
* Tech-X Corp., Boulder, CO: Parallel/notebook (previous versions)

# A new $37.8M initiative in Data Science

* Moore/Sloan Foundations, 5 year support for *UC Berkeley*, *U. Washington*, *NYU*.
* Open source scientific computing tools will be central to this effort.
* **We're hiring:** [Executive Director (now)](http://vcresearch.berkeley.edu/datascience/career-opportunities), Data Science Fellows (soon).


# Kinematics of particle

> Marcos Duarte, Renato Naville Watanabe
> [Laboratory of Biomechanics and Motor Control](http://pesquisa.ufabc.edu.br/bmclab)
> Federal University of ABC, Brazil

Contents

1  Python setup

2  Biomechanics & Mechanics

3  Kinematics

3.1  Vectors in Kinematics

4  Position

5  Basis

6  Displacement

7  Velocity

8  Acceleration

9  The antiderivative

10  Some cases of motion of a particle

10.0.1  Particle at rest

10.0.2  Particle at constant speed

10.0.3  Particle at constant acceleration

10.1  Visual representation of these cases

11  Symbolic programming

12  Kinematics of human movement

12.1  Kinematics of the 100-m race

13  Further reading

14  Video lectures on the Internet

15  Problems

16  References

## Python setup

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns
sns.set_context("notebook", font_scale=1.2, rc={"lines.linewidth": 2, "lines.markersize": 10})
```

## Biomechanics & Mechanics

**A good knowledge of Mechanics is a necessary condition, although not sufficient!, to master Biomechanics**

For this reason, we will review principles of Classical Mechanics in the context of Biomechanics.

The book [*Introduction to Statics and Dynamics*](http://ruina.tam.cornell.edu/Book/index.html) , written by Andy Ruina and Rudra Pratap, is an excellent reference (a rigorous and yet didactic presentation of Mechanics for undergraduate students) on Classical Mechanics and we will use this book as the main reference on Mechanics and Mathematics for this brief review. The preface and first chapter of the book are a good read on how someone should study Mechanics. You should read them!

As we argued in the notebook [Biomechanics](http://nbviewer.jupyter.org/github/demotu/BMC/blob/master/notebooks/Biomechanics.ipynb), we will start with a branch of Classical Mechanics that is simpler to measure its related quantities on biological systems, the Kinematics.

There are some relevant cases in the study of human movement where modeling the human body or one of its segments as a particle might be all we need to explore the phenomenon. The concept of kinematics of a particle, for instance, can be applied to study the performance in the 100-m race; to describe spatial and temporal characteristics of a movement pattern, and to conjecture about how voluntary movements are planned (the minimum jerk hypothesis).

Now, let's review the concept of kinematics of a particle and later apply to the study of human movement.

## Kinematics

**Kinematics** is the branch of Classical Mechanics that describes the motion of objects without consideration of the causes of motion ([Wikipedia](http://en.wikipedia.org/wiki/Kinematics)).

Kinematics of a particle is the description of the motion when the object is considered a particle.

A particle as a physical object does not exist in nature; it is a simplification to understand the motion of a body or it is a conceptual definition such as the center of mass of a system of objects.

### Vectors in Kinematics

Some mechanical quantities in Kinematics (position and its derivatives) are represented as vectors and others, such as time and distance, are scalars.
A vector in Mechanics is a physical quantity with magnitude, direction, and satisfies some elementary vector arithmetic, whereas a scalar is a physical quantity that is fully expressed by a magnitude (a number) only.

For a review about scalars and vectors, see chapter 1 of [Ruina and Rudra's book](http://ruina.tam.cornell.edu/Book/index.html).

 For how to use Python to work with scalars and vectors, see the notebook [Scalar and Vector](http://nbviewer.jupyter.org/github/BMCLab/BMC/blob/master/notebooks/ScalarVector.ipynb).

## Position

Consider a point in the three-dimensional Euclidean space described in a Cartesian coordinate system (see the notebook [Frame of reference](http://nbviewer.jupyter.org/github/demotu/BMC/blob/master/notebooks/ReferenceFrame.ipynb) for an introduction on coordinate systems in Mechanics and Biomechanics):

Figure. Representation of a point $\mathbf{P}$ and its position vector $\overrightarrow{\mathbf{r}}$ in a Cartesian coordinate system. The versors $\hat{\mathbf{i}},\, \hat{\mathbf{j}},\, \hat{\mathbf{k}}\,$  form a basis for this coordinate system and are usually represented in the color sequence RGB (red, green, blue) for easier visualization.

The position of this point in space can be represented as a triple of values each representing the coordinate at each axis of the Cartesian coordinate system following the $ \mathbf{X, Y, Z} $ convention order (which is omitted):

\begin{equation}
(x,\, y,\, z)
\label{eq_xyz}
\end{equation}

The position of a particle in space can also be represented by a vector in the Cartesian coordinate system, with the origin of the vector at the origin of the coordinate system and the tip of the vector at the point position:

\begin{equation}
\overrightarrow{\mathbf{r}}(t) = x\,\hat{\mathbf{i}} + y\,\hat{\mathbf{j}} + z\,\hat{\mathbf{k}}
\label{eq_rxyz}
\end{equation}

Where  $\hat{\mathbf{i}},\, \hat{\mathbf{j}},\, \hat{\mathbf{k}}\,$  are unit vectors in the directions of the axes $ \mathbf{X, Y, Z} $.

For a review on vectors, see the notebook [Scalar and vector](http://nbviewer.ipython.org/github/demotu/BMC/blob/master/notebooks/ScalarVector.ipynb).

With this new notation, the coordinates of a point representing the position of a particle that vary with time would be expressed by the following position vector $\overrightarrow{\mathbf{r}}(t)$:

\begin{equation}
\overrightarrow{\mathbf{r}}(t) = x(t)\,\hat{\mathbf{i}} + y(t)\,\hat{\mathbf{j}} + z(t)\,\hat{\mathbf{k}}
\label{eq_rxyzijk}
\end{equation}

A vector can also be represented in matrix form:

\begin{equation}
\overrightarrow{\mathbf{r}}(t) = \begin{bmatrix} x(t) \\y(t) \\z(t) \end{bmatrix}
\label{eq_rxyzmatrix}
\end{equation}

And the unit vectors in each Cartesian coordinate in matrix form are given by:

\begin{equation}
\hat{\mathbf{i}} = \begin{bmatrix}1\\0\\0 \end{bmatrix},\; \hat{\mathbf{j}} = \begin{bmatrix}0\\1\\0 \end{bmatrix},\; \hat{\mathbf{k}} = \begin{bmatrix} 0 \\ 0 \\ 1 \end{bmatrix}
\label{eq_ijk}
\end{equation}

## Basis

In [linear algebra](http://en.wikipedia.org/wiki/Linear_algebra), a set of unit linearly independent vectors as the three vectors above (orthogonal in the Euclidean space) that can represent any vector via [linear combination](http://en.wikipedia.org/wiki/Linear_combination) is called a basis. A basis is the foundation of creating a reference frame and we will study how to do that other time.

## Displacement

The shortest distance between two positions of a particle.

As the difference between two vectors, displacement is also a vector quantity.

\begin{equation}
\mathbf{\overrightarrow{d}} = \mathbf{\overrightarrow{r}_2} - \mathbf{\overrightarrow{r}_1}
\label{eq_distance}
\end{equation}

Figure. Representation of the displacement vector $\mathbf{\overrightarrow{d}}$ between two positions $\mathbf{\overrightarrow{r}_1}$ and $\mathbf{\overrightarrow{r}_2}$.

## Velocity

Velocity is the rate (with respect to time) of change of the position of a particle.

The average velocity between two instants is:

\begin{equation}
\overrightarrow{\mathbf{v}}(t) = \frac{\overrightarrow{\mathbf{r}}(t_2)-\overrightarrow{\mathbf{r}}(t_1)}{t_2-t_1} = \frac{\Delta \overrightarrow{\mathbf{r}}}{\Delta t}
\label{eq_velocity}
\end{equation}

The instantaneous velocity of the particle is obtained when $\Delta t$ approaches to zero, which from calculus is the first-order [derivative](http://en.wikipedia.org/wiki/Derivative) of the position vector:

\begin{equation}
\overrightarrow{\mathbf{v}}(t) = \lim_{\Delta t \to 0} \frac{\Delta \overrightarrow{\mathbf{r}}}{\Delta t} = \lim_{\Delta t \to 0} \frac{\overrightarrow{\mathbf{r}}(t+\Delta t)-\overrightarrow{\mathbf{r}}(t)}{\Delta t} = \frac{\mathrm{d}\overrightarrow{\mathbf{r}}}{dt}
\label{eq_velocityderiv}
\end{equation}

For the movement of a particle described with respect to an [inertial Frame of reference](http://nbviewer.jupyter.org/github/demotu/BMC/blob/master/notebooks/ReferenceFrame.ipynb), the derivative of a vector is obtained by differentiating each vector component of the Cartesian coordinates (since the base versors  $\hat{\mathbf{i}}, \hat{\mathbf{j}}, \hat{\mathbf{k}}$  are constant):

\begin{equation}
\overrightarrow{\mathbf{v}}(t) = \frac{\mathrm{d}\overrightarrow{\mathbf{r}}(t)}{dt} = \frac{\mathrm{d}x(t)}{\mathrm{d}t}\hat{\mathbf{i}} + \frac{\mathrm{d}y(t)}{\mathrm{d}t}\hat{\mathbf{j}} + \frac{\mathrm{d}z(t)}{\mathrm{d}t}\hat{\mathbf{k}}
\label{eq_velocityderiv2}
\end{equation}

Or in matrix form (and using the Newton's notation for differentiation):

\begin{equation}
\overrightarrow{\mathbf{v}}(t) = \begin{bmatrix}
\dot x(t) \\
\dot y(t) \\
\dot z(t)
\end{bmatrix}
\label{eq_velocityderiv3}
\end{equation}

## Acceleration

Acceleration is the rate (with respect to time) of change of the velocity of a particle, which can also be given by the second-order rate of change of the position.

The average acceleration between two instants is:

\begin{equation}
\overrightarrow{\mathbf{a}}(t) = \frac{\overrightarrow{\mathbf{v}}(t_2)-\overrightarrow{\mathbf{v}}(t_1)}{t_2-t_1} = \frac{\Delta \overrightarrow{\mathbf{v}}}{\Delta t}
\label{eq_acc}
\end{equation}

Likewise, instantaneous acceleration is the first-order derivative of the velocity or the second-order derivative of the position vector:

\begin{equation}
\overrightarrow{\mathbf{a}}(t) = \frac{\mathrm{d}\overrightarrow{\mathbf{v}}(t)}{\mathrm{d}t} = \frac{\mathrm{d}^2\overrightarrow{\mathbf{r}}(t)}{\mathrm{d}t^2} = \frac{\mathrm{d}^2x(t)}{\mathrm{d}t^2}\hat{\mathbf{i}} + \frac{\mathrm{d}^2y(t)}{\mathrm{d}t^2}\hat{\mathbf{j}} + \frac{\mathrm{d}^2z(t)}{\mathrm{d}t^2}\hat{\mathbf{k}}
\label{eq_accderiv}
\end{equation}

And in matrix form:

\begin{equation}
\mathbf{a}(t) = \begin{bmatrix}
\ddot x(t) \\
\ddot y(t) \\
\ddot z(t)
\end{bmatrix}
\label{eq_accderiv2}
\end{equation}

For curiosity, see [Notation for differentiation](https://en.wikipedia.org/wiki/Notation_for_differentiation) on the origin of the different notations for differentiation.

When the base versors change in time, for instance when the basis is attached to a rotating frame or reference, the components of the vector’s derivative is not the derivatives of its components; we will also have to consider the derivative of the basis with respect to time.

## The antiderivative

As the acceleration is the derivative of the velocity which is the derivative of position, the inverse mathematical operation is the [antiderivative](http://en.wikipedia.org/wiki/Antiderivative) (or integral):

\begin{equation}
\begin{array}{l l}
\mathbf{r}(t) = \mathbf{r}_0 + \int \mathbf{v}(t) \:\mathrm{d}t \\
\mathbf{v}(t) = \mathbf{v}_0 + \int \mathbf{a}(t) \:\mathrm{d}t
\end{array}
\label{eq_antiderivative}
\end{equation}

**This part of the kinematics is presented in chapter 12, pages 605-620, of the [Ruina and Rudra's book](http://ruina.tam.cornell.edu/Book/index.html).**

## Some cases of motion of a particle

To deduce some trivial cases of motion of a particle (at rest, at constant speed, and at constant acceleration), we can start from the equation for its position and differentiate it to obtain expressions for the velocity and acceleration or the inverse approach, start with the equation for acceleration, and then integrate it to obtain the velocity and position of the particle. Both approachs are valid in Mechanics. For the present case, it probaly makes more sense to start with the expression for acceleration.

#### Particle at rest

\begin{equation}
\begin{array}{l l}
\overrightarrow{\mathbf{a}}(t) = 0 \\
\overrightarrow{\mathbf{v}}(t) = 0 \\
\overrightarrow{\mathbf{r}}(t) = \overrightarrow{\mathbf{r}}_0
\end{array}
\label{eq_rest}
\end{equation}

#### Particle at constant speed

\begin{equation}
\begin{array}{l l}
\overrightarrow{\mathbf{a}}(t) = 0 \\
\overrightarrow{\mathbf{v}}(t) = \overrightarrow{\mathbf{v}}_0 \\
\overrightarrow{\mathbf{r}}(t) = \overrightarrow{\mathbf{r}}_0 + \overrightarrow{\mathbf{v}}_0t
\end{array}
\label{eq_constantspeed}
\end{equation}

#### Particle at constant acceleration

\begin{equation}
\begin{array}{l l}
\overrightarrow{\mathbf{a}}(t) = \overrightarrow{\mathbf{a}}_0 \\
\overrightarrow{\mathbf{v}}(t) = \overrightarrow{\mathbf{v}}_0 + \overrightarrow{\mathbf{a}}_0t \\
\overrightarrow{\mathbf{r}}(t) = \overrightarrow{\mathbf{r}}_0 + \overrightarrow{\mathbf{v}}_0t +
\frac{1}{2}\overrightarrow{\mathbf{a}}_0 t^2
\end{array}
\label{eq_constantacceleration}
\end{equation}

### Visual representation of these cases

```python
t = np.arange(0, 2.0, 0.02)
r0 = 1; v0 = 2; a0 = 4

plt.rc('axes',  labelsize=14,  titlesize=14)
plt.rc('xtick', labelsize=10)
plt.rc('ytick', labelsize=10)
f, axarr = plt.subplots(3, 3, sharex = True, sharey = True, figsize=(14,7))
plt.suptitle('Scalar kinematics of a particle', fontsize=20);

tones = np.ones(np.size(t))

axarr[0, 0].set_title('at rest', fontsize=14);
axarr[0, 0].plot(t, r0*tones, 'g', linewidth=4, label='$r(t)=1$')
axarr[1, 0].plot(t,  0*tones, 'b', linewidth=4, label='$v(t)=0$')
axarr[2, 0].plot(t,  0*tones, 'r', linewidth=4, label='$a(t)=0$')
axarr[0, 0].set_ylabel('r(t) [m]')
axarr[1, 0].set_ylabel('v(t) [m/s]')
axarr[2, 0].set_ylabel('a(t) [m/s$^2$]')

axarr[0, 1].set_title('at constant speed');
axarr[0, 1].plot(t, r0*tones+v0*t, 'g', linewidth=4, label='$r(t)=1+2t$')
axarr[1, 1].plot(t, v0*tones,      'b', linewidth=4, label='$v(t)=2$')
axarr[2, 1].plot(t,  0*tones,      'r', linewidth=4, label='$a(t)=0$')

axarr[0, 2].set_title('at constant acceleration');
axarr[0, 2].plot(t, r0*tones+v0*t+1/2.*a0*t**2,'g', linewidth=4,
                 label='$r(t)=1+2t+\\frac{1}{2}4t^2$')
axarr[1, 2].plot(t, v0*tones+a0*t,             'b', linewidth=4,
                 label='$v(t)=2+4t$')
axarr[2, 2].plot(t, a0*tones,                  'r', linewidth=4,
                 label='$a(t)=4$')

for i in range(3):
    axarr[2, i].set_xlabel('Time [s]');
    for j in range(3):
        axarr[i,j].set_ylim((-.2, 10))
        axarr[i,j].legend(loc = 'upper left', frameon=True, framealpha = 0.9, fontsize=14)

plt.subplots_adjust(hspace=0.09, wspace=0.07)
```

## Symbolic programming

We can use [Sympy](http://www.sympy.org/en/index.html), a Python library for symbolic mathematics, to deduce the expressions for the cases of motion of a particle we just visualized.
Let's show how to integrate with Sympy for the case of particle with constant acceleration:

```python
from sympy import Symbol, integrate, init_printing
init_printing(use_latex='mathjax')

t = Symbol('t', real=True, positive=True)
g = Symbol('g', real=True, positive=True)
v0 = Symbol('v0', real=True, positive=True, constant = True)
r0 = Symbol('r0', real=True, positive=True, constant = True)
```

```python
v = integrate(g, t) + v0  # a constant has to be added
v
```

$\displaystyle g t + v_{0}$

```python
r = integrate(v, t) + r0  # a constant has to be added
r
```

$\displaystyle \frac{g t^{2}}{2} + r_{0} + t v_{0}$

## Kinematics of human movement

### Kinematics of the 100-m race

An example where the analysis of some aspects of the human body movement can be reduced to the analysis of a particle is the study of the Biomechanics of the 100-m race.

A technical report with the kinematic data for the 100-m world record by Usain Bolt can be downloaded from the [website for Research Projects](http://www.iaaf.org/development/research) from the International Association of Athletics Federations.
[Here is a direct link for that report](http://www.iaaf.org/download/download?filename=76ade5f9-75a0-4fda-b9bf-1b30be6f60d2.pdf&urlSlug=1-biomechanics-report-wc-berlin-2009-sprint). In particular, the following table shows the data for the three medalists in that race:

Figure. Data from the three medalists of the 100-m dash in Berlin, 2009 (IAAF report).

The column **RT** in the table above refers to the reaction time of each athlete. The IAAF has a very strict rule about reaction time: any athlete with a reaction time less than 100 ms is disqualified from the competition! See the website [Reaction Times and Sprint False Starts](http://condellpark.com/kd/reactiontime.htm) for a discussion about this rule.

You can measure your own reaction time in a simple way visiting this website: [http://www.humanbenchmark.com/tests/reactiontime](http://www.humanbenchmark.com/tests/reactiontime).

The article [A Kinematics Analysis Of Three Best 100 M Performances Ever](http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3661886/) by Krzysztof and Mero presents a detailed kinematic analysis of 100-m races.

## Further reading

 - Read the preface and first chapter of the [Ruina and Rudra's book](http://ruina.tam.cornell.edu/Book/index.html) about how someone should study Mechanics.
 - See the notebook [Spatial and temporal characteristics](http://nbviewer.jupyter.org/github/demotu/BMC/blob/master/notebooks/SpatialTemporalCharacteristcs.ipynb) about how the simple measurement of spatial and temporal kinematic variables can be very useful to describe the human gait.
 - See the notebook [The minimum jerk hypothesis](http://nbviewer.jupyter.org/github/demotu/BMC/blob/master/notebooks/MinimumJerkHypothesis.ipynb) about the conjecture that movements are performed (organized) with the smoothest trajectory possible.

## Video lectures on the Internet

 - Khan Academy: [One-dimensional motion](https://www.khanacademy.org/science/ap-physics-1/ap-one-dimensional-motion)
 - [Powers of 10, Units, Dimensions, Uncertainties, Scaling Arguments](https://youtu.be/GtOGurrUPmQ)
 - [1D Kinematics - Speed, Velocity, Acceleration](https://youtu.be/q9IWoQ199_o)

## Problems

1. Answer the 12 questions of the [Khan Academy's test on one-dimensional motion](https://www.khanacademy.org/science/ap-physics-1/ap-one-dimensional-motion/test/ap-one-dimensional-motion-unit-test?modal=1).

2. Consider the data for the three medalists of the 100-m dash in Berlin, 2009, shown previously.
 a. Calculate the average velocity and acceleration.
 b. Plot the graphs for the displacement, velocity, and acceleration versus time.
 c. Plot the graphs velocity and acceleration versus partial distance (every 20m).
 d. Calculate the average velocity and average acceleration and the instants and values of the peak velocity and peak acceleration.

3. The article "Biomechanical Analysis of the Sprint and Hurdles Events at the 2009 IAAF World Championships in Athletics" by Graubner and Nixdorf lists the 10-m split times for the three medalists of the 100-m dash in Berlin, 2009:

 a. Repeat the same calculations performed in problem 2 and compare the results.

4. A body attached to a spring has its position (in cm) described by the equation $x(t) = 2\sin(4\pi t + \pi/4)$.
 a. Calculate the equation for the body velocity and acceleration.
 b. Plot the position, velocity, and acceleration in the interval [0, 1] s.

5. There are some nice free software that can be used for the kinematic analysis of human motion. Some of them are: [Kinovea](http://www.kinovea.org/), [Tracker](http://www.cabrillo.edu/~dbrown/tracker/), and [SkillSpector](http://video4coach.com/index.php?option=com_content&task=view&id=13&Itemid=45). Visit their websites and explore these software to understand in which biomechanical applications they could be used.

6. From Ruina and Rudra's book, solve the problems **12.1.1** to **12.1.14**.

## References

- Graubner R, Nixdorf E (2011) [Biomechanical Analysis of the Sprint and Hurdles Events at the 2009 IAAF World Championships in Athletics ](http://www.meathathletics.ie/devathletes/pdf/Biomechanics%20of%20Sprints.pdf). [New Studies in Athletics](http://www.iaaf.org/development/new-studies-in-athletics), 1/2, 19-53.
- Krzysztof M, Antti Mero A (2013) [A Kinematics Analysis Of Three Best 100 M Performances Ever](http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3661886/). Journal of Human Kinetics, 36, 149–160.
- [Research Projects](http://www.iaaf.org/development/research) from the International Association of Athletics Federations.
- Ruina A, Rudra P (2019) [Introduction to Statics and Dynamics](http://ruina.tam.cornell.edu/Book/index.html). Oxford University Press.


```python
import sys
import sympy
from sympy import *
from sympy import Rational as frac
init_printing()

from memoize import memoize

vartheta, varphi = sympy.symbols('vartheta, varphi', real=True)
v, nu, delta, R_L = sympy.symbols('v, nu, delta, R_L', real=True)
Sigma_n, Sigma_ell, Sigma_lambda = sympy.symbols('Sigma_n, Sigma_ell, Sigma_lambda', real=True)
S_n, S_ell, S_lambda = sympy.symbols('S_n, S_ell, S_lambda', real=True)

nHat = Matrix([1,0,0])
lambdaHat = Matrix([0,1,0])
ellHat = Matrix([0,0,1])

NHat = Matrix([sin(vartheta)*cos(varphi), sin(vartheta)*sin(varphi), cos(vartheta)])
PHat = Matrix([sin(varphi), -cos(varphi), 0])
QHat = Matrix([cos(vartheta)*cos(varphi), cos(vartheta)*sin(varphi), -sin(vartheta)])

Sigma = Matrix([Sigma_n, Sigma_lambda, Sigma_ell])
S = Matrix([S_n, S_lambda, S_ell])

def hComplex(jComponent, kComponent):
    return sympy.trigsimp((nu/R_L)*( (jComponent.dot(PHat)*kComponent.dot(PHat) - jComponent.dot(QHat)*kComponent.dot(QHat))
                                    - sympy.I*(jComponent.dot(PHat)*kComponent.dot(QHat) + jComponent.dot(QHat)*kComponent.dot(PHat)) ) )
```

```python
# Memoize (cache) this function for speed
@memoize
def sYlm(s, ell, m, vartheta, varphi):
    r = sympy.Symbol('r', integer=True)
    return sympy.simplify( sympy.sqrt(frac( factorial(ell+m)*factorial(ell-m),factorial(ell+s)*factorial(ell-s) )
                                      * (2*ell+1)/(4*sympy.pi) )
                           * exp(sympy.I*m*varphi)
                           * summation(binomial(ell-s,r) * binomial(ell+s,r+s-m) * (-1)**(ell-r-s-m)
                                       *sin(vartheta/2)**(2*ell-2*r-s+m) * cos(vartheta/2)**(2*r+s-m),
                                       (r,0,ell-s)) )
@memoize
def sYlmConjugate(s, ell, m, vartheta, varphi):
    r = sympy.Symbol('r', integer=True)
    return sympy.simplify( sympy.sqrt(frac( factorial(ell+m)*factorial(ell-m),factorial(ell+s)*factorial(ell-s) )
                                      * (2*ell+1)/(4*sympy.pi) )
                           * exp(-sympy.I*m*varphi)
                           * summation(binomial(ell-s,r) * binomial(ell+s,r+s-m) * (-1)**(ell-r-s-m)
                                       *sin(vartheta/2)**(2*ell-2*r-s+m) * cos(vartheta/2)**(2*r+s-m),
                                       (r,0,ell-s)) )
# Calculate these now, just so everything else will be fast
for ell in range(2,4+1):
    for m in range(-ell,ell+1):
        tmp = sYlm(-2,ell,m,vartheta,varphi)
        tmp = sYlmConjugate(-2,ell,m,vartheta,varphi)
```

The following are the spin contributions to the waveform from Kidder '95 (and the first-order nonspinning contribution, for comparison).

```python
h0NS = 2*v**2*sympy.simplify(hComplex(lambdaHat, lambdaHat)-hComplex(nHat, nHat))
```

```python
h2Ss = 2*v**4*hComplex(Sigma.cross(NHat), nHat)
```

```python
h3SO = -2*v**5*sympy.simplify((5*S_ell + 3*delta*Sigma_ell)*hComplex(lambdaHat, lambdaHat)
                              - 6*(2*S_ell + delta*Sigma_ell)*hComplex(nHat, nHat))
```

```python
h3Sp = -2*v**5*sympy.simplify(2*hComplex(lambdaHat, nHat.cross(S + delta*Sigma))
                              + hComplex(nHat, lambdaHat.cross(9*S + 5*delta*Sigma)) )
```

```python
h3Spv = -2*v**5*sympy.simplify(2*NHat.dot(lambdaHat)*hComplex((S + delta*Sigma).cross(NHat), nHat)
                               + NHat.dot(nHat)*hComplex((S + delta*Sigma).cross(NHat), lambdaHat) )
```

```python
def IntegrateFunctionOnTheSphere(func):
    return integrate( integrate( sympy.expand(func*sin(vartheta)), (varphi, 0, 2*pi)), (vartheta, 0, pi))
def IntegrateFunctionAgainstsYlm(func, ell, m, s=-2):
    func = sympy.expand(func*sYlmConjugate(s,ell,m,vartheta,varphi)*sin(vartheta))
    return integrate( integrate( func, (varphi, 0, 2*pi)), (vartheta, 0, pi))
```

This isn't going to work; `sympy` just can't handle the simple trig integrals necessary.

```python
integrate(sin(vartheta)*cos(vartheta/2)**8, vartheta)
```

```python

```

```python

```


# Obnova obrazu

Nasnímaný obraz je běžně zatížen šumem z několika zrdojů. Těmito zdroji jsou např. šum vzniklý na elektrických senzorech, chyby kanálů, zrnitost atd. šum vzniklý od na senzorch nabo na kanálech je ve formě izolovaných variancí hodnot pixelů které nejsou mezi sebou korelované a které jsou v obraze viditelné jako výrazně odlišné hodnoty sousedních pixelů.

Na následujícím obrázku je blokovým diagramem vizualizovaný proces vzniku šumu a degradace obrazu např. rozmazáním.

ve skutečnosti je obraz degradován jak šumem tak degradační funkcí najednou, pro jedoduchost budeme uvažovat jen šum.

## Modely šumu

Existuje mnoho modelů šumu, v této hodině budeme uvažovat čtyři následující:

### Šum s rovnoměrným rozdělením
Šum s rovnoměrným rozdělením je generován z rozmezí  $low$ a $high$ dle předpisu:

$
\begin{equation}
p(z) = \begin{cases}
\frac{1}{high-low}, & \text{if } low \leq z \leq high \\
0, & \text{otherwise} \\
\end{cases}
\end{equation}
$

kde $p(z)$ je pravděpodobnost že pixel nabyde hodnoty šumu $z$.

### Gaussovský šum

Je generovaný pomocí normálního rozdělení dle předpisu:

$
\begin{equation}
p(z) = \frac{1}{ \sqrt{2\pi \sigma^2}} \; \exp \left( - \frac{(z - \mu)^2}{2\sigma^2} \right),
\end{equation}
$

kde $\mu$ je střední hodnota a $\sigma$ je směrodatná odchylka.

### Salt&Peper

Šum ve formě impulzů častu zůsoběný chybou A/D převodníku nebo chybou přenosu. Způsobuje že určité množství pixelů má 0 nebo maximální hodnotu.

Pravděpodobnost že pixel bude mít intenzitu $z$ je dána předpisem

$
\begin{equation}
p(z) = \begin{cases}
pPepper, & \text{for } z = 0 \text{ (pepper)} \\
pSalt, & \text{for } z = 2^n - 1 \text{ (salt)} \\
1-(pPepper+pSalt), & \text{for } 0 < z < 2^n - 1, \\
\end{cases}
\end{equation}
$

kde $n$ je počet bitů ve kterých je hodnota pixelů kódována (8). Pro vygenrování tohoto filtru využijte funkci pro generování náhodných čísel z rovnoměrného rozdělení.

### Šum s exponencíálním rozdělením

V případě exponencílního šumu je pravděpodobnost genrována předpisem:

$
\begin{equation}
p(z) = \begin{cases}
0, & z < 0 \\
\lambda \exp(-\lambda z), & z \geq \lambda \\
\end{cases}
\end{equation}
$

```python
import cv2
import numpy as np
import matplotlib.pyplot as plt
```

```python
img =  cv2.imread("data/tower.jpg")
fig = plt.figure(figsize= (10,10))
plt.imshow(cv2.cvtColor(img, cv2.COLOR_BGR2RGB));
```

```python
def uniform_noise(img, low=-0.1, high=0.1):
    """
    Aplikace rovnomerneho sumu na obraz v rozmezi low az high

    Parameters
    ----------
    img : ndarray
        Obraz nacteny funkci z opencv
    low : float
        minimalni hodnota pro generovani rovnomerneho sumu
    high : float
        maximalni hodnota pro generovani rovnomerneho sumu

    Returns
    ----------
    tuple
        obraz se sumem, obraz sumu, aplikovane hodnoty sumu
    """

    noise_rnd = np.random.uniform(0, 1, img.shape) # Vygenerovani 3d matice nahodnych cisel v rozmezi 0, 1 o rozmerech obrazu nahodnych cisel
    mask = np.where(np.logical_and(noise_rnd>0.5+low, noise_rnd<0.5+high)) #maska - indexy kde hodnot jsou v rozmezi low a high (+0.5)
    noise = np.zeros(img.shape)
    noise[mask] = (noise_rnd[mask] - 0.5) * 255 # Vygenerovani matice sumu pro aplikaci na obraz
    img_noise = img + noise # aplikace matice sumu

    # normalizace noveho obrazu mezi 0 a 255
    img_noise[img_noise<0] = 0
    img_noise[img_noise>255] = 255

    return img_noise.astype(np.uint8), np.abs(noise).astype(np.uint8), noise_rnd.flatten() - 0.5
```

```python
fig = plt.figure(figsize= (40,6))
img_noise, noise, noise_raw = uniform_noise(img, -0.25, 0.25)
plt.subplot(141)
plt.imshow(cv2.cvtColor(img, cv2.COLOR_BGR2RGB));
plt.title('original')
plt.subplot(142)
plt.imshow(cv2.cvtColor(noise, cv2.COLOR_BGR2RGB));
plt.title('noise - uniform')
plt.subplot(143)
plt.imshow(cv2.cvtColor(img_noise, cv2.COLOR_BGR2RGB));
plt.title('original + noise')
plt.subplot(144)
plt.hist(noise_raw, density=True, bins=np.linspace(-0.25,0.25,100));
plt.title('noise histogram');
```

## Obnova obrazu

Pro obnovu degradovaného obrazu je nejprve potřeba odhadnou typ degradace. Budeme uvažovat pouze šum. Vizualizace procesu obnovy obrazu pomocí blokového schématu je na následujícím, obrázku:

## Odstranění šumu

Šum může být redukován pomocí statistických filtrů. Zde záleží jaký druh šumu je v obraze přítomen. Filtr je aplikován jako dané statistické operaci odpovídajcí konvoluční filtr. Předpokládejme obraz $g(s,t)$ kde $g$ a $s$ jsou souřadnice pixelu v obraze. Konvoluční filtr je pak $S_{xy}$.

### Mediánový filtr
Je vhodný pro redukci šumu typu Salt&Peper. Předpis pro výpočet nového obrazu je

$
\begin{equation}
\hat{f}(x,y) = \underset{(s,t) \in S_{xy}}{\mathrm{median}} \, g(s,t)
\end{equation}
$

### Průměrovací filtr

Je vhodný pro odstarněni Gaussovského šumu. Nový obraz vznikne dle předpisu

$
\begin{equation}
\hat{f}(x,y) = \frac{1}{mn}\sum_{(s,t) \in S_{xy}} g(s,t),
\end{equation}
$

kde $m$ a $n$ jsou rozměry konvolučního filtru.

** Pro mediánový a průměrovací filtr je možné využít následující funkce OpenCV2 https://docs.opencv.org/master/d4/d13/tutorial_py_filtering.html **

### MIN filtr
Odpovídá morfologické erozi.

$
\begin{equation}
\hat{f}(x,y) = \min_{(s,t) \in S_{xy}} g(s,t)
\end{equation}
$

### MAX filtr
Odpovídá morfologické dilataci.

$
\begin{equation}
\hat{f}(x,y) = \max_{(s,t) \in S_{xy}} g(s,t)
\end{equation}
$

** Pro MIN a MAX filtry je možné využít funkce OpenCV2 pro erozi a dilataci https://docs.opencv.org/3.4/db/df6/tutorial_erosion_dilatation.html https://docs.opencv.org/3.4/d4/d86/group__imgproc__filter.html **

```python

```