Copyright (c) Microsoft Corporation. All rights reserved.

Licensed under the MIT License.

In this notebook we will explore the univaraite time-series data to determine the settings for an automated ML experiment. We will follow the thought process depicted in the following diagram:

The objective is to answer the following questions:

Is there a seasonal pattern in the data?

 Importance: If we are able to detect regular seasonal patterns, the forecast accuracy may be improved by extracting these patterns and including them as features into the model.

Is the data stationary?

 Importance: In the absense of features that capture trend behavior, ML models (regression and tree based) are not well equiped to predict stochastic trends. Working with stationary data solves this problem.

Is there a detectable auto-regressive pattern in the stationary data?

 Importance: The accuracy of ML models can be improved if serial correlation is modeled by including lags of the dependent/target varaible as features. Including target lags in every experiment by default will result in a regression in accuracy scores if such setting is not warranted.

The answers to these questions will help determine the appropriate settings for the automated ML experiment.

```python
import os
import warnings
import pandas as pd

from statsmodels.graphics.tsaplots import plot_acf, plot_pacf
import matplotlib.pyplot as plt
from pandas.plotting import register_matplotlib_converters

register_matplotlib_converters()  # fixes the future warning issue

from helper_functions import unit_root_test_wrapper
from statsmodels.tools.sm_exceptions import InterpolationWarning

warnings.simplefilter("ignore", InterpolationWarning)

# set printing options
pd.set_option("display.max_columns", 500)
pd.set_option("display.width", 1000)
```

```python
# load data
main_data_loc = "data"
train_file_name = "S4248SM144SCEN.csv"

TARGET_COLNAME = "S4248SM144SCEN"
TIME_COLNAME = "observation_date"
COVID_PERIOD_START = "2020-03-01"

df = pd.read_csv(os.path.join(main_data_loc, train_file_name))
df[TIME_COLNAME] = pd.to_datetime(df[TIME_COLNAME], format="%Y-%m-%d")
df.sort_values(by=TIME_COLNAME, inplace=True)
df.set_index(TIME_COLNAME, inplace=True)
df.head(2)
```

```python
# plot the entire dataset
fig, ax = plt.subplots(figsize=(6, 2), dpi=180)
ax.plot(df)
ax.title.set_text("Original Data Series")
locs, labels = plt.xticks()
plt.xticks(rotation=45)
```

The graph plots the alcohol sales in the United States. Because the data is trending, it can be difficult to see cycles, seasonality or other interestng behaviors due to the scaling issues. For example, if there is a seasonal pattern, which we will discuss later, we cannot see them on the trending data. In such case, it is worth plotting the same data in first differences.

```python
# plot the entire dataset in first differences
fig, ax = plt.subplots(figsize=(6, 2), dpi=180)
ax.plot(df.diff().dropna())
ax.title.set_text("Data in first differences")
locs, labels = plt.xticks()
plt.xticks(rotation=45)
```

In the previous plot we observe that the data is more volatile towards the end of the series. This period coincides with the Covid-19 period, so we will exclude it from our experiment. Since in this example there are no user-provided features it is hard to make an argument that a model trained on the less volatile pre-covid data will be able to accurately predict the covid period.

# 1. Seasonality

#### Questions that need to be answered in this section:
1. Is there a seasonality?
2. If it's seasonal, does the data exhibit a trend (up or down)?

It is hard to visually detect seasonality when the data is trending. The reason being is scale of seasonal fluctuations is dwarfed by the range of the trend in the data. One way to deal with this is to de-trend the data by taking the first differences. We will discuss this in more detail in the next section.

```python
# plot the entire dataset in first differences
fig, ax = plt.subplots(figsize=(6, 2), dpi=180)
ax.plot(df.diff().dropna())
ax.title.set_text("Data in first differences")
locs, labels = plt.xticks()
plt.xticks(rotation=45)
```

For the  next plot, we will exclude the Covid period again. We will also shorten the length of data because plotting a very long time series may prevent us from seeing seasonal patterns, if there are any, because the plot may look like a random walk.

```python
# remove COVID period
df = df[:COVID_PERIOD_START]

# plot the entire dataset in first differences
fig, ax = plt.subplots(figsize=(6, 2), dpi=180)
ax.plot(df["2015-01-01":].diff().dropna())
ax.title.set_text("Data in first differences")
locs, labels = plt.xticks()
plt.xticks(rotation=45)
```

 Conclusion

Visual examination does not suggest clear seasonal patterns. We will set the STL_TYPE = None, and we will move to the next section that examines stationarity.

Say, we are working with a different data set that shows clear patterns of seasonality, we have several options for setting the settings:is hard to say which option will work best in your case, hence you will need to run both options to see which one results in more accurate forecasts.

 If the data does not appear to be trending, set DIFFERENCE_SERIES=False, TARGET_LAGS=None and STL_TYPE = "season"

 If the data appears to be trending, consider one of the following two settings:

 DIFFERENCE_SERIES=True, TARGET_LAGS=None and STL_TYPE = "season", or

 DIFFERENCE_SERIES=False, TARGET_LAGS=None and STL_TYPE = "trend_season"

 In the first case, by taking first differences we are removing stochastic trend, but we do not remove seasonal patterns. In the second case, we do not remove the stochastic trend and it can be captured by the trend component of the STL decomposition. It is hard to say which option will work best in your case, hence you will need to run both options to see which one results in more accurate forecasts.

# 2. Stationarity
If the data does not exhibit seasonal patterns, we would like to see if the data is non-stationary. Particularly, we want to see if there is a clear trending behavior. If such behavior is observed, we would like to first difference the data and examine the plot of an auto-correlation function (ACF) known as correlogram. If the data is seasonal, differencing it will not get rid off the seasonality and this will be shown on the correlogram as well.

 Question: What is stationarity and how to we detect it?

 This is a fairly complex topic. Please read the following  link  for a high level discussion on this subject.

 Simply put, we are looking for scenario when examining the time series plots the mean of the series is roughly the same, regardless which time interval you pick to compute it. Thus, trending and seasonal data are examples of non-stationary series.

 Question: Why do want to work with stationary data?

 In the absence of features that capture stochastic trends, the ML models that use (deterministic) time based features (hour of the day, day of the week, month of the year, etc) cannot capture such trends, and will over or under predict depending on the behavior of the time series. By working with stationary data, we eliminate the need to predict such trends, which improves the forecast accuracy. Classical time series models such as Arima and Exponential Smoothing handle non-stationary series by design and do not need such transformations. By differencing the data we are still able to run the same family of models.

#### Questions that need to be answered in this section:

 Is the data stationary?

 Does the stationarized data (either the original or the differenced series) exhibit a clear auto-regressive pattern?

To answer the first question, we run a series of tests (we call them unit root tests).

```python
# unit root tests
test = unit_root_test_wrapper(df[TARGET_COLNAME])
print("---------------", "\n")
print("Summary table", "\n", test["summary"], "\n")
print("Is the {} series stationary?: {}".format(TARGET_COLNAME, test["stationary"]))
print("---------------", "\n")
```

In the previous cell, we ran a series of unit root tests. The summary table contains the following columns:

 test_name is the name of the test.

 ADF: Augmented Dickey-Fuller test

 KPSS: Kwiatkowski-Phillips–Schmidt–Shin test

 PP: Phillips-Perron test

 ADF GLS: Augmented Dickey-Fuller using generalized least squares method

 AZ: Andrews-Zivot test

 statistic: test statistic

 crit_val: critical value of the test statistic

 p_val: p-value of the test statistic. If the p-val is less than 0.05, the null hypothesis is rejected.

 stationary: is the series stationary based on the test result?

 Null hypothesis: what is being tested. Notice, some test such as ADF and PP assume the process has a unit root and looks for evidence to reject this hypothesis. Other tests, ex.g: KPSS, assumes the process is stationary and looks for evidence to reject such claim.

Each of the tests shows that the original time series is non-stationary. The final decision is based on the majority rule. If, there is a split decision, the algorithm will claim it is stationary. We run a series of tests because each test by itself may not be accurate. In many cases when there are conflicting test results, the user needs to make determination if the series is stationary or not.

Since we found the series to be non-stationary, we will difference it and then test if the differenced series is stationary.

```python
# unit root tests
test = unit_root_test_wrapper(df[TARGET_COLNAME].diff().dropna())
print("---------------", "\n")
print("Summary table", "\n", test["summary"], "\n")
print("Is the {} series stationary?: {}".format(TARGET_COLNAME, test["stationary"]))
print("---------------", "\n")
```

Four out of five tests show that the series in first differences is stationary. Notice that this decision is not unanimous. Next, let's plot the original series in first-differences to illustrate the difference between non-stationary (unit root) process vs the stationary one.

```python
# plot original and stationary data
fig = plt.figure(figsize=(10, 10))
ax1 = fig.add_subplot(211)
ax1.plot(df[TARGET_COLNAME], "-b")
ax2 = fig.add_subplot(212)
ax2.plot(df[TARGET_COLNAME].diff().dropna(), "-b")
ax1.title.set_text("Original data")
ax2.title.set_text("Data in first differences")
```

If you were asked a question "What is the mean of the series before and after 2008?", for the series titled "Original data" the mean values will be significantly different. This implies that the first moment of the series (in this case, it is the mean) is time dependent, i.e., mean changes depending on the interval one is looking at. Thus, the series is deemed to be non-stationary. On the other hand, for the series titled "Data in first differences" the means for both periods are roughly the same. Hence, the first moment is time invariant; meaning it does not depend on the interval of time one is looking at. In this example it is easy to visually distinguish between stationary and non-stationary data. Often this distinction is not easy to make, therefore we rely on the statistical tests described above to help us make an informed decision.

 Conclusion

Since we found the original process to be non-stationary (contains unit root), we will have to model the data in first differences. As a result, we will set the DIFFERENCE_SERIES parameter to True.

# 3 Check if there is a clear autoregressive pattern
We need to determine if we should include lags of the target variable as features in order to improve forecast accuracy. To do this, we will examine the ACF and partial ACF (PACF) plots of the stationary series. In our case, it is a series in first diffrences.

 Question: What is an Auto-regressive pattern? What are we looking for?

 We are looking for a classical profiles for an AR(p) process such as an exponential decay of an ACF and a the first $p$ significant lags of the PACF. For a more detailed explanation of ACF and PACF please refer to the appendix at the end of this notebook. For illustration purposes, let's examine the ACF/PACF profiles of the simulated data that follows a second order auto-regressive process, abbreviated as an AR(2).

            The lag order is on the x-axis while the auto- and partial-correlation coefficients are on the y-axis. Vertical lines that are outside the shaded area represent statistically significant lags. Notice, the ACF function decays to zero and the PACF shows 2 significant spikes (we ignore the first spike for lag 0 in both plots since the linear relationship of any series with itself is always 1).

 Question: What do I do if I observe an auto-regressive behavior?

 If such behavior is observed, we might improve the forecast accuracy by enabling the target lags feature in AutoML. There are a few options of doing this

 Set the target lags parameter to 'auto', or

 Specify the list of lags you want to include. Ex.g: target_lags = [1,2,5]

 Next, let's examine the ACF and PACF plots of the stationary target variable (depicted below). Here, we do not see a decay in the ACF, instead we see a decay in PACF. It is hard to make an argument the the target variable exhibits auto-regressive behavior.

```python
# Plot the ACF/PACF for the series in differences
fig, ax = plt.subplots(1, 2, figsize=(10, 5))
plot_acf(df[TARGET_COLNAME].diff().dropna().values.squeeze(), ax=ax[0])
plot_pacf(df[TARGET_COLNAME].diff().dropna().values.squeeze(), ax=ax[1])
plt.show()
```

 Conclusion

Since we do not see a clear indication of an AR(p) process, we will not be using target lags and will set the TARGET_LAGS parameter to None.

 AutoML Experiment Settings

Based on the analysis performed, we should try the following settings for the AutoML experiment and use them in the "2_run_experiment" notebook.

 STL_TYPE=None

 DIFFERENCE_SERIES=True

 TARGET_LAGS=None

# Appendix: ACF, PACF and Lag Selection
To do this, we will examine the ACF and partial ACF (PACF) plots of the differenced series.

 Question: What is the ACF?

 To understand the ACF, first let's look at the correlation coefficient $\rho_{xz}$
            \begin{equation}
            \rho_{xz} = \frac{\sigma_{xz}}{\sigma_{x} \sigma_{zy}}
            \end{equation}

        where $\sigma_{xzy}$ is the covariance between two random variables $X$ and $Z$; $\sigma_x$ and $\sigma_z$ is the variance for $X$ and $Z$, respectively. The correlation coefficient measures the strength of linear relationship between two random variables. This metric can take any value from -1 to 1.

 The auto-correlation coefficient $\rho_{Y_{t} Y_{t-k}}$ is the time series equivalent of the correlation coefficient, except instead of measuring linear association between two random variables $X$ and $Z$, it measures the strength of a linear relationship between a random variable $Y_t$ and its lag $Y_{t-k}$ for any positive interger value of $k$.

 To visualize the ACF for a particular lag, say lag 2, plot the second lag of a series $y_{t-2}$ on the x-axis, and plot the series itself $y_t$ on the y-axis. The autocorrelation coefficient is the slope of the best fitted regression line and can be interpreted as follows. A one unit increase in the lag of a variable one period ago leads to a $\rho_{Y_{t} Y_{t-2}}$ units change in the variable in the current period. This interpreation can be applied to any lag.

 In the interpretation posted above we need to be careful not to confuse the word "leads" with "causes" since these are not the same thing. We do not know the lagged value of the varaible causes it to change. Afterall, there are probably many other features that may explain the movement in $Y_t$. All we are trying to do in this section is to identify situations when the variable contains the strong auto-regressive components that needs to be included in the model to improve forecast accuracy.

 Question: What is the PACF?

 When describing the ACF we essentially running a regression between a partigular lag of a series, say, lag 4, and the series itself. What this implies is the regression coefficient for lag 4 captures the impact of everything that happens in lags 1, 2 and 3. In other words, if lag 1 is the most important lag and we exclude it from the regression, naturally, the regression model will assign the importance of the 1st lag to the 4th one. Partial auto-correlation function fixes this problem since it measures the contribution of each lag accounting for the information added by the intermediary lags. If we were to illustrate ACF and PACF for the fourth lag using the regression analogy, the difference is a follows:
        \begin{align}
            Y_{t} &= a_{0} + a_{4} Y_{t-4} + e_{t} \\
            Y_{t} &= b_{0} + b_{1} Y_{t-1} + b_{2} Y_{t-2} + b_{3} Y_{t-3} + b_{4} Y_{t-4} + \varepsilon_{t} \\
        \end{align}

            Here, you can think of $a_4$ and $b_{4}$ as the auto- and partial auto-correlation coefficients for lag 4. Notice, in the second equation we explicitely accounting for the intermediate lags by adding them as regrerssors.

 Question: Auto-regressive pattern? What are we looking for?

 We are looking for a classical profiles for an AR(p) process such as an exponential decay of an ACF and a the first $p$ significant lags of the PACF. Let's examine the ACF/PACF profiles of the same simulated AR(2) shown in Section 3, and check if the ACF/PACF explanation are refelcted in these plots.

 The autocorrelation coefficient for the 3rd lag is 0.6, which can be interpreted that a one unit increase in the value of the target varaible three periods ago leads to 0.6 units increase in the current period.  However, the PACF plot shows that the partial autocorrealtion coefficient is zero (from a statistical point of view since it lies within the shaded region). This is happening because the 1st and 2nd lags are good predictors of the target variable. Ommiting these two lags from the regression results in the misleading conclusion that the third  lag is a good prediciton.

 This is why it is important to examine both the ACF and the PACF plots when tring to determine the auto regressive order for the variable in question.


# Time-varying frame

> Renato Naville Watanabe, Marcos Duarte
> [Laboratory of Biomechanics and Motor Control](http://pesquisa.ufabc.edu.br/bmclab)
> Federal University of ABC, Brazil

Contents

1  Python setup

2  Cartesian coordinate system

3  Determination of a coordinate system

4  Time varying basis

4.1  Tangential versor

4.2  Normal versor

4.3  Binormal versor

5  Velocity and Acceleration in a time-varying frame

5.1  Velocity

5.2  Acceleration

6  Example

6.1  Solving numerically

6.2  Symbolic solution (extra reading)

7   Leitura Adicional

8   Vídeo aulas na Internet

9  Problems

10  References

## Python setup

```python
import numpy as np
import sympy as sym
from sympy.vector import CoordSys3D
import matplotlib.pyplot as plt
sym.init_printing()
from sympy.plotting import plot_parametric
from sympy.physics.mechanics import ReferenceFrame, Vector, dot
from matplotlib.patches import FancyArrowPatch
plt.rcParams.update({'figure.figsize':(8, 5), 'lines.linewidth':2})
```

## Cartesian coordinate system

As we perceive the surrounding space as three-dimensional, a convenient coordinate system is the [Cartesian coordinate system](http://en.wikipedia.org/wiki/Cartesian_coordinate_system) in the [Euclidean space](http://en.wikipedia.org/wiki/Euclidean_space) with three orthogonal axes as shown below. The axes directions are commonly defined by the [right-hand rule](http://en.wikipedia.org/wiki/Right-hand_rule) and attributed the letters X, Y, Z. The orthogonality of the Cartesian coordinate system is convenient for its use in classical mechanics, most of the times the structure of space is assumed having the [Euclidean geometry](http://en.wikipedia.org/wiki/Euclidean_geometry) and as consequence, the motion in different directions are independent of each other.

Figure. Representation of a point and its position vector in a Cartesian coordinate system.

## Determination of a coordinate system

In Biomechanics, we may use different coordinate systems for convenience and refer to them as global, laboratory, local, anatomical, or technical reference frames or coordinate systems.

From [linear algebra](http://en.wikipedia.org/wiki/Linear_algebra), a set of unit linearly independent vectors (orthogonal in the Euclidean space and each with norm (length) equals to one) that can represent any vector via [linear combination](http://en.wikipedia.org/wiki/Linear_combination) is called a basis (or **orthonormal basis**). The figure below shows a point and its position vector in the Cartesian coordinate system and the corresponding versors (**unit vectors**) of the basis for this coordinate system. See the notebook [Scalar and vector](http://nbviewer.ipython.org/github/demotu/BMC/blob/master/notebooks/ScalarVector.ipynb) for a description on vectors.

Figure. Representation of a point $\mathbf{P}$ and its position vector $\overrightarrow{\mathbf{r}}$ in a Cartesian coordinate system. The versors $\hat{\mathbf{i}},\, \hat{\mathbf{j}},\, \hat{\mathbf{k}}\,$  form a basis for this coordinate system and are usually represented in the color sequence RGB (red, green, blue) for easier visualization.

One can see that the versors of the basis shown in the figure above have the following coordinates in the Cartesian coordinate system:

\begin{equation}
\hat{\mathbf{i}} = \begin{bmatrix}1\\0\\0 \end{bmatrix}, \quad \hat{\mathbf{j}} = \begin{bmatrix}0\\1\\0 \end{bmatrix}, \quad \hat{\mathbf{k}} = \begin{bmatrix} 0 \\ 0 \\ 1 \end{bmatrix}
\label{eq_1}
\end{equation}

Using the notation described in the figure above, the position vector $\overrightarrow{\mathbf{r}}$  can be expressed as:

\begin{equation}
\overrightarrow{\mathbf{r}} = x\hat{\mathbf{i}} + y\hat{\mathbf{j}} + z\hat{\mathbf{k}}
\label{eq_2}
\end{equation}

However, to use a fixed basis can lead to very complex expressions.

## Time varying basis

Consider that we have the position vector of a particle, moving in the path described by the parametric curve $s(t)$, described in a fixed reference frame as:

\begin{equation}
{\bf\hat{r}}(t) = {x}{\bf\hat{i}}+{y}{\bf\hat{j}} + {z}{\bf\hat{k}}
\label{eq_3}
\end{equation}

Figure. Position vector of a moving particle in relation to a coordinate system.

### Tangential versor

Often we describe all the kinematic variables in this fixed reference frame. However, it is often useful to define a time-varying basis, attached to some point of interest. In this case, what is usually done is to choose as one of the basis vector a unitary vector in the direction of the velocity of the particle. Defining this vector as:

\begin{equation}
{\bf\hat{e}_t} = \frac{{\bf\vec{v}}}{\Vert{\bf\vec{v}}\Vert}
\label{eq_4}
\end{equation}

### Normal versor

For the second vector of the basis, we define first a vector of curvature of the path (the meaning of this curvature vector will be seeing in another notebook):

\begin{equation}
{\bf\vec{C}} = \frac{d{\bf\hat{e}_t}}{ds}
\label{eq_5}
\end{equation}

Note that $\bf\hat{e}_t$ is a function of the path $s(t)$. So, by the chain rule:

\begin{equation}
\frac{d{\bf\hat{e}_t}}{dt} = \frac{d{\bf\hat{e}_t}}{ds}\frac{ds}{dt} \longrightarrow \frac{d{\bf\hat{e}_t}}{ds} = \frac{\frac{d{\bf\hat{e}_t}}{dt}}{\frac{ds}{dt}} \longrightarrow {\bf\vec{C}} = \frac{\frac{d{\bf\hat{e}_t}}{dt}}{\frac{ds}{dt}}\longrightarrow {\bf\vec{C}} = \frac{\frac{d{\bf\hat{e}_t}}{dt}}{\Vert{\bf\vec{v}}\Vert}
\label{eq_6}
\end{equation}

Now we can define the second vector of the basis, ${\bf\hat{e}_n}$:

\begin{equation}
{\bf\hat{e}_n} = \frac{{\bf\vec{C}}}{\Vert{\bf\vec{C}}\Vert}
\label{eq_7}
\end{equation}

Figure. A moving particle and a corresponding time varying basis.

### Binormal versor

The third vector of the basis is obtained by the cross product between ${\bf\hat{e}_n}$ and ${\bf\hat{e}_t}$:

\begin{equation}
{\bf\hat{e}_b} = {\bf\hat{e}_t} \times {\bf\hat{e}_n}
\label{eq_8}
\end{equation}

Note that the vectors  ${\bf\hat{e}_t}$ ,  ${\bf\hat{e}_n}$  and  ${\bf\hat{e}_b}$  vary together with the particle movement.

## Velocity and Acceleration in a time-varying frame

### Velocity

Given the expression of $r(t)$ in a fixed frame we can write the velocity  ${\bf\vec{v}(t)}$ as a function of the fixed frame of reference  ${\bf\hat{i}}$,  ${\bf\hat{j}}$ and  ${\bf\hat{k}}$ (see http://nbviewer.jupyter.org/github/BMClab/bmc/blob/master/notebooks/KinematicsParticle.ipynb)).

\begin{equation}
{\bf\vec{v}}(t) = \dot{x}{\bf\hat{i}}+\dot{y}{\bf\hat{j}}+\dot{z}{\bf\hat{k}}
\label{eq_9}
\end{equation}

However, this can lead to very complex functions. So it is useful to use the basis find previously ${\bf\hat{e}_t}$,  ${\bf\hat{e}_n}$ and ${\bf\hat{e}_b}$.

The velocity  ${\bf\vec{v}}$  of the particle is, by the definition of ${\bf\hat{e}_t}$, in the direction of ${\bf\hat{e}_t}$:

\begin{equation}
{\bf\vec{v}}={\Vert\bf\vec{v}\Vert}.{\bf\hat{e}_t}
\label{eq_10}
\end{equation}

### Acceleration

The acceleration can be written in the fixed frame of reference as:

\begin{equation}
{\bf\vec{a}}(t) = \ddot{x}{\bf\hat{i}}+\ddot{y}{\bf\hat{j}}+\ddot{z}{\bf\hat{k}}
\label{eq_11}
\end{equation}

But for the same reasons of the velocity vector, it is useful to describe the acceleration vector in the time varying basis. We know that the acceleration is the time derivative of the velocity:

\begin{align}
{\bf\vec{a}} =& \frac{{d\bf\vec{v}}}{dt}=\\
    =&\frac{{d({\Vert\bf\vec{v}\Vert}{\bf\hat{e}_t}})}{dt}=\\
    =&\dot{\Vert\bf\vec{v}\Vert}{\bf\hat{e}_t}+{\Vert\bf\vec{v}\Vert}\dot{{\bf\hat{e}_t}}=\\
    =&\dot{\Vert\bf\vec{v}\Vert}{\bf\hat{e}_t}+{\Vert\bf\vec{v}\Vert}\frac{d{\bf\hat{e}_t}}{ds}\frac{ds}{dt}=\\
    =&\dot{\Vert\bf\vec{v}\Vert}{\bf\hat{e}_t}+{\Vert\bf\vec{v}\Vert}^2\frac{d{\bf\hat{e}_t}}{ds}=\\
    =&\dot{\Vert\bf\vec{v}\Vert}{\bf\hat{e}_t}+{\Vert\bf\vec{v}\Vert}^2\Vert{\bf\vec{C}} \Vert {\bf\hat{e}_n}
\label{eq_12}
\end{align}

## Example
For example, consider that a particle follows the path described by the parametric curve below:

\begin{equation}
{\bf\vec{r}(t)} = (10t+100){\bf{\hat{i}}} + \left(-\frac{9,81}{2}t^2+50t+100\right){\bf{\hat{j}}}
\label{eq_13}
\end{equation}

This curve could be, for example, from a projectile motion. See http://nbviewer.jupyter.org/github/BMClab/bmc/blob/master/notebooks/ProjectileMotion.ipynb for an explanation on projectile motion.

### Solving numerically

Now we will obtain the time-varying basis numerically. This method is useful when it is not available a mathematical expression of the path. This often happens when you have available data collected experimentally (most of the cases in Biomechanics).

First, data will be obtained from the expression of $r(t)$. This is done to replicate the example above. You could use data collected experimentally, for example.

```python
t = np.linspace(0, 10, 1000).reshape(-1,1)
r = np.hstack((10*t + 100, -9.81/2*t**2 + 50*t + 100))
```

Now, to obtain the  $\bf{\hat{e_t}}$  versor, we can use Equation (4).

```python
dt = t[1]
v = np.diff(r,axis=0)/dt
vNorm  = np.linalg.norm(v, axis=1, keepdims=True)

et = v/vNorm
```

And to obtain the versor $\bf{\hat{e_n}}$, we can use Equation (8).

```python
C = np.diff(et,axis=0)/dt
C = C/vNorm[1:]

CNorm = np.linalg.norm(C, axis=1, keepdims=True)
en = C/CNorm
```

```python
fig = plt.figure()
plt.plot(r[:, 0], r[:, 1], '.')
ax = fig.add_axes([0, 0, 1, 1])
time = np.linspace(0, 10, 10)
for i in np.arange(0, len(t)-2, 50):
    vec1 = FancyArrowPatch(r[i, :], r[i, :]+10*et[i, :], mutation_scale=20, color='r')
    vec2 = FancyArrowPatch(r[i, :], r[i, :]+10*en[i, :], mutation_scale=20, color='g')
    ax.add_artist(vec1)
    ax.add_artist(vec2)
plt.xlim((80, 250))
plt.ylim((80, 250))
plt.legend([vec1, vec2], [r'$\vec{e_t}$', r'$\vec{e_{n}}$'])
plt.show()
```

```python
v  = vNorm*et
vNormDot  = np.diff(vNorm, axis=0)/dt

a = vNormDot*et[1:,:] + CNorm*en*vNorm[1:]**2
```

```python
fig = plt.figure()
plt.plot(r[:, 0], r[:, 1], '.')
ax = fig.add_axes([0, 0, 1, 1])
for i in range(0, len(t)-2, 50):
    vec1 = FancyArrowPatch(r[i, :], r[i, :]+v[i, :], mutation_scale=10, color='r')
    vec2 = FancyArrowPatch(r[i, :], r[i, :]+a[i, :], mutation_scale=10, color='g')
    ax.add_artist(vec1)
    ax.add_artist(vec2)
plt.xlim((80, 250))
plt.ylim((80, 250))
plt.legend([vec1, vec2], [r'$\vec{v}$', r'$\vec{a}$'])
plt.show()
```

### Symbolic solution (extra reading)

The computation here will be performed symbolically, with the symbolic math package of Python, Sympy. Below,a reference frame, called O,  and a varible for time (t) are defined.

```python
O = sym.vector.CoordSys3D(' ')
t = sym.symbols('t')
```

Below the vector $r(t)$ is defined symbolically.

```python
r = (10*t+100)*O.i + (-9.81/2*t**2+50*t+100)*O.j+0*O.k
r
```

```python
p1 = plot_parametric(r.dot(O.i), r.dot(O.j), (t, 0, 10), line_width=4, aspect_ratio=(2, 1))
```

```python
v = sym.diff(r)
v
```

```python
et = v/sym.sqrt(v.dot(v))
et
```

```python
C = sym.diff(et)/sym.sqrt(v.dot(v))
C
```

```python
en = C/(sym.sqrt(C.dot(C)))
sym.simplify(en)
```

```python
fig = plt.figure()
ax = fig.add_axes([0, 0, 1, 1])
time = np.linspace(0, 10, 30)
for instant in time:
    vt = FancyArrowPatch([float(r.dot(O.i).subs(t,instant)),float(r.dot(O.j).subs(t,instant))],
                         [float(r.dot(O.i).subs(t,instant))+10*float(et.dot(O.i).subs(t,instant)),
                          float(r.dot(O.j).subs(t, instant))+10*float(et.dot(O.j).subs(t,instant))],
                         mutation_scale=20,
                         arrowstyle="->",color="r",label='${\hat{e_t}}$', lw=2)
    vn = FancyArrowPatch([float(r.dot(O.i).subs(t, instant)),float(r.dot(O.j).subs(t,instant))],
                         [float(r.dot(O.i).subs(t, instant))+10*float(en.dot(O.i).subs(t, instant)),
                          float(r.dot(O.j).subs(t, instant))+10*float(en.dot(O.j).subs(t, instant))],
                         mutation_scale=20,
                         arrowstyle="->",color="g",label='${\hat{e_n}}$', lw=2)
    ax.add_artist(vn)
    ax.add_artist(vt)
plt.xlim((90, 250))
plt.ylim((90, 250))
plt.xlabel('x')
plt.legend(handles=[vt, vn], fontsize=20)
plt.show()
```

Now we can find the vectors ${\bf\vec{v}}$ and ${\bf\vec{a}}$ described in the time varying frame.

```python
v = sym.sqrt(v.dot(v))*et
```

```python
a = sym.diff(sym.sqrt(v.dot(v)))*et+v.dot(v)*sym.sqrt(C.dot(C))*en
sym.simplify(sym.simplify(a))
```

```python
fig = plt.figure()
ax = fig.add_axes([0, 0, 1, 1])
time = np.linspace(0, 10, 10)
for instant in time:
    vt = FancyArrowPatch([float(r.dot(O.i).subs(t,instant)),float(r.dot(O.j).subs(t,instant))],
                         [float(r.dot(O.i).subs(t,instant))+float(v.dot(O.i).subs(t,instant)),
                          float(r.dot(O.j).subs(t, instant))+float(v.dot(O.j).subs(t,instant))],
                         mutation_scale=20,
                         arrowstyle="->",color="r",label='${{v}}$', lw=2)
    vn = FancyArrowPatch([float(r.dot(O.i).subs(t, instant)),float(r.dot(O.j).subs(t,instant))],
                         [float(r.dot(O.i).subs(t, instant))+float(a.dot(O.i).subs(t, instant)),
                          float(r.dot(O.j).subs(t, instant))+float(a.dot(O.j).subs(t, instant))],
                         mutation_scale=20,
                         arrowstyle="->",color="g",label='${{a}}$', lw=2)
    ax.add_artist(vn)
    ax.add_artist(vt)
plt.xlim((60, 250))
plt.ylim((60, 250))
plt.legend(handles=[vt, vn], fontsize=20)
plt.show()
```

## Further Reading

 - Read pages 932-971 of the 18th chapter of the [Ruina and Rudra's book](http://ruina.tam.cornell.edu/Book/index.html) about time-varying basis vectors.

## Video lectures on the Internet

 - [How curvy is a curve?](https://youtu.be/si6ka6HeUdY) and [Tangential and Normal components of Acceleration](https://youtu.be/LQNC7klG0F8)
 - [Curvilinear Motion: Normal and Tangential Components](https://youtu.be/Zfy3FYZrsWo)

## Problems

+ 1. Obtain the vectors  $\hat{e_n}$  and  $\hat{e_t}$  for the problem 17.1.1 from Ruina and Rudra's book.
+ 2. Solve the problem 17.1.9 from Ruina and Rudra's book.
+ 3. Write a Python program to solve the problem 17.1.10 (only the part of  $\hat{e_n}$  and  $\hat{e_t}$).
+ 4. A particle has its position vector along time equal to ${\bf\vec{r}(t)} = R\cos(\omega t){\bf\hat{i}} + R\sin(\omega t){\bf\hat{j}}$. Find the curvature vector  ${\bf\vec{C}}$  as a function of time. Draw the vector ${\bf\vec{C}}$ at some points of the particle path.

## References

+ Ruina A, Rudra P (2019) [Introduction to Mechanics for Engineers](http://ruina.tam.cornell.edu/Book/index.html). Oxford University Press.


#**Machine Learning 101 with Python**

Dr Jack Yang

jianliang.yang1@unsw.edu.au

*School of Material Science and Engineering, UNSW*

*Advanced Materials and Manufacturing Futures Institute, UNSW*

July 2020

#*Aims*

Machine learning (ML) and artificial intelligence (AI) are definitively some of the words in hype across the society at the moment. However, views on ML/AI are quite divided. While some are astonished with the power devlivered by the complex neural network models that can solve real problems once seems only solvable through human reasoning, such as AlphaGo, others criticize ML is simply a 'glorified version of curve fitting' and being largely a 'black box'.

At the end of previous workshop ([Visualisation of Covid-19 Data Using Python](https://colab.research.google.com/drive/1gtIWqmfnp4TsXGiEHOiOZ061MNsak9LX#scrollTo=ZRJzanK8aley)), we demonstrate how Python has made ML easily doable with a few lines of Python. In this workshop, we want to build on top of what we demonstrated previously, by using some simple and easy-to-visualize examples to walk through a proper process of optimizing a ML model.

We hope that, this workshop can open up some of the 'black boxes' in ML to help you gain more understanding on how ML work, and why it is more than just a simple 'curve-fitting'.

#**Imports**

As usual, we will import some of the existing packages for doing the heavy-liftings. We will use ***matplotlib*** for plotting, and ***scikit-learn*** for ML.

```
import matplotlib
import matplotlib.pyplot as plt
import sklearn
import math
```

#**Supervised Learning (Regression) - An Introduction**

At the first glance, the purpose of ML regression is very similar to curve-fitting. Given a set of $x$'s and $y$'s, we want to *interpolate* the relationship between them in order to obtain a function $y=f^{\mbox{pred}}(x)$, which should be as close to the true function  $y=f^{\mbox{true}}(x)$ as possible. From which we can predict the corresponding $y_{new}$ for a new observable $x_{new}$ as   $y_{new}=f^{\mbox{pred}}(x_{new})$.

However, being called machine **learning**, there is at least one key difference from a standard curve fitting:

In ML, we generally have **no knowledge** what $f(x)$ is needed to fit the observed data, because this is what we are trying to learn. What we have is a set of **probablistic believes** on how the world looks like (i.e. what functions can possibly fit to the data, which is called ***prior***), and the ML algorithm helps us to find out which subset of thses functions have the **highest probability to reproduce the observed relationship** (which is also called the ***posterior***). This is schematically illustrated as follow and we will demonstrate that using an ML method called **Gaussian Processes**.

#**Setting the Stage - Playing with the Lennard Jones Potential**

In this workshop, we will use Gaussian Process to demonstrate how an ML model can be optimized to learn a very simpe mathematical relationships from a set of obserbales. The true relationship is governed by the physics of a Lennard-Jones potential. Don't worry if you don't know what it is. In a nutshell, it describes the interaction energies between two particles separated at distance $x$, and has a very simple mathematical form:
\begin{equation}
y=-\frac{2a}{x^6}+\frac{b}{x^{12}}.
\end{equation}

Nevertheless, it is sufficient for demonstrating a few key ideas in ML for people who simply wants to get a crack on it!

To set this up, we construct a Python class (or object) to handle a few things in a more pragmatic way.

```
class LennardJonesian(object):
  """
  This is a class object that helps to define the functional form of the Lennard Jones
  potential, and allows you to calculate the value of the potential given a distances
  between the two particles.

  To construct a default object, simply call:
  lj = LennardJonesian()
  with the default parameters of a=1 and b=1 (see the __init__ function)

  If you would like to change these parameters, you can initialize a new object, such as
  lj2 = LennardJonesian(a=2,b=1.5)

  """

  def __init__(self,a=1,b=1):
    self.a = a
    self.b = b

  def potential(self,x):
    """
    This function calculate the value of the Lennard Jones potential at a given separation x.
    """
    return -2*self.a/math.pow(x,6)+self.b/math.pow(x,12)

  def plot_analytic_potential(self,n_samples=500,x_start=0.64,x_end=1.64,c='k',label=None):
    """
    This function plots the curve for Lennard Jones potential.

    Args:
      xs: a list of x values.
    """
    xs = [x_start+i*(x_end-x_start)/n_samples for i in range(n_samples)]
    plt.plot(xs,[self.potential(i) for i in xs],'-',c=c,label=label,lw=2.5,alpha=0.5)
    plt.plot(xs,[0 for _ in xs],'y:')
    plt.xlabel('$x$',fontsize=20)
    plt.ylabel('$y$',fontsize=20)

  def fake_samples(self,n_samples=500,x_start=0.64,x_end=1.64):
    """
    For the demonstration purpose only, we will generate a set of random samples (x,y)
    that will roughly follow the relationship given by the LJ potential. A random noise
    is added to the y value to make it deviate from the true LJ potential. The idea
    here is to mimic an experimentally/computationally generated database, on which
    ML will be performed. In reality, the database generating process would be
    the most laborious and slow part of the process, which cannot be demonstrated
    in a live event.

    Args:
      n_samples: number of samples to be generated, default is 500.
    """
    import random

    #this draws a random float in the range given by [x_start,x_end]
    x_sample = [x_start+random.randrange(0,100000)/100000*(x_end-x_start) for _ in range(n_samples)]
    x_sample = list(sorted(x_sample))

    #this calculate the corresponding y at the x, plus a random noise
    y_sample = [self.potential(x)*(random.randrange(-50,50)/100+1) for x in x_sample]
    return x_sample,y_sample
```

First of all, let's plot the true function to see what it looks like!

```
#Set up the aspect ratio of the lot, so I have a big plot to look at
plt.figure(figsize=(6,6))

#This is how I instantiate a new LennardJonesian object. I gave a set of parameters so
#the graph looks more curely (i.e. hase some interesting feature).
#Also notice that I assign the created object to a new variable called lj so I can
#get access to different attributes/methods available to the object.
lj=LennardJonesian(a=10,b=2)

#This is how you call the function *plot_analytic_potential* in the object to
#plot out how this function looks like. The idea of setting up an object, in this
#case, is that I dont need to pass the variables a and b to all other functions
#such as calculating y's from x's and plot data etc etc, because they are known
#throughout the object once it is constructed. This makes the code easier to read.
lj.plot_analytic_potential(c='b',label='True Function')

plt.legend()
```

You can probably see that this function, even though it is very simple, it has a part where $y$ changes very rapidly with respect to $x$, and a part where $y$ varies slowly with respect to $x$. This basically reflects the landscape of most mathematical functions that we want to learn/fit, whether or not we are using ML techniques or not. Being able to appreciate how your underlying function 'looks' like will help you a lot with properly tuning the ML model, and this is why we are using a simple function here so we can visualize what is going on.

Now let us generate some data that overall behaves like the Lennard-Jones system, but does not follow a simple curve as shown above.

This could be the case where:
1. data is measured experimentally where noises are inevitable.
2. data is generated in a high-dimensional space, when it is projected onto $x$, it shows a dominating LJ behaviors but their are additional contributing factors from other dimensions that makes the data deviate from a perfect LJ.

*REMEMBER*

**In real life, it is this set of data that we will see first and try to learn, not the underlying Lennard-Jones potential, which is the target knowledge to be learned. We only do this in the reverse order here so you can understand the data easier, which, as mentioned about, is quite important for ML to work for you.**

```
n_samples=500 # this is how many sample we have on our hand, from which we
              # will learn the underlying behaviors

plt.figure(figsize=(12,12))
plt.subplot(2,2,1)
lj.plot_analytic_potential(c='#f1e0d6',label='True Function')
x_sample,y_sample=lj.fake_samples(n_samples=n_samples)
#show what are the points being sampled
plt.scatter(x_sample,y_sample,marker='o',s=7,facecolor='#525b56',label='experimental observations')
plt.legend()

#We also want to have some understanding on the distributions of the data points
#It may come useful when we are trying to understand the accuracy of ML later.
plt.subplot(2,2,2)
plt.hist(y_sample,bins=50,orientation='horizontal',color='#525b56',alpha=0.6,density=True);
plt.xlabel('Frequency',fontsize=20);
plt.ylabel('y',fontsize=20);

plt.subplot(2,2,3)
plt.hist(x_sample,bins=50,color='#525b56',alpha=0.6,density=True);
plt.ylabel('Frequency',fontsize=20);
plt.xlabel('x',fontsize=20);
```

#**Initial Learning - Getting Some Idea Where We are Heading**

```
from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import RBF
import numpy as np
```

# The training set

The first step of the ML process is to extract a portion of the sample for constructing a training set to train the model, where the rest are left for testing/validating the trained model to see how well the model can predict the results of *previously unseen* data.

How to appropriately select the training data has very important implication on the quality of ML-model, which we cannot go into too much details here. For the moment, the training data is simply selected as ****a fraction ($t$%) of the total sample evenly spreaded across $x$****, so $t$ is one of the parameters in our ML model

```
training_pt_fraction=0.2
num_traing_pts=n_samples*training_pt_fraction
gap=int(n_samples/num_traing_pts)
print(gap)

#these are the x's and the corresponding y's for training the model
x_train=np.array(x_sample[::gap]).reshape(-1, 1)
y_train=np.array(y_sample[::gap]).reshape(-1, 1)

#Plot the training set out and have a look
plt.figure(figsize=(12,12))
plt.subplot(2,2,1)
lj.plot_analytic_potential(c='#f1e0d6',label='True Function')
plt.scatter(x_sample,y_sample,marker='o',s=7,facecolor='#525b56',alpha=0.5,label='experimental observations')
plt.scatter(x_train,y_train,marker='o',s=15,facecolor='b',label='training points')
plt.legend()
```

# The Kernel (or the covariance matrix)

*   This is the most critical ingredient in Gaussian Processes.
*   Given two points $(x_1,y_1)$ and $(x_2,y_2)$ in space, the kernel describes the similarity between two points $\mathcal{K}(x_1,x_2)$ based on the distance between them $\mathcal{K}(x_1,x_2)=\mathcal{K}(d(x_1,x_2))$. The idea is that if $x_1$ and $x_2$ are close together, $y_1$ should also be close to $y_2$ (in theory).
*   The kernel therefore encodes *our prior belief on how two data points are similar to each other*, controlled by some parameters.

Here we will be using the Radial Basis Function (RBF) as our kernel, which is a Gaussian function of the distance between two points:
\begin{equation}
\mathcal{K}(x_i,x_j)=\exp\left(-\frac{d(x_i,x_j)^2}{2\ell^2}\right)
\end{equation}
in which $\ell$ is the parameter controlling how correlated are two points in space.

**Note**

* While simple kernel may serve as a starting point for understanding the basics, real problems often require more complex kernels, often tailored designed to better capture the physics behind.
* It is possible to combine multiple kernels to model more complex behaviors in the data.

Now let's first get some impression on what our prior will look like drawn by the RBF kernel with length-scale parameter $\ell$.

```
def draw_prior(length_scale):
  #length_scale: variable, the value of which will be given to define
  #              the kernel

  #define a kernel and Gaussian process object by giving it a kernel
  kernel = RBF(length_scale=length_scale)
  gp = GaussianProcessRegressor(kernel=kernel)

  X_ = np.linspace(0.6, 1.6 ,100) #this is the range of x in which we want to model
  y_mean, y_std = gp.predict(X_[:, np.newaxis], return_std=True)

  #drawing sample functions from the prior probability distribution function, which
  #simply says all functions are equal likely to be drawn
  y_samples = gp.sample_y(X_[:, np.newaxis], 3)

  return X_, y_samples

length_scales=[0.1,0.05,0.001,0.0001]
plt.figure(figsize=(20,4))
for i in range(4):
  x, y = draw_prior(length_scales[i])
  plt.subplot(1,4,i+1)
  plt.plot(x,y,'-')
  plt.xlabel('x',fontsize=20)
  if i==0:
    plt.ylabel('y',fontsize=20)
  plt.title('RBF l='+str(length_scales[i]))
```

# Initial Fit

Let's first use the four kernels listed above to try fitting our data, using 0.2 as the fraction of the total data to be used as training points. This will allows us to get some idea on what might be a reasonable starting parameter for the RBF kernel.

Here we will fix the training points across all four parameters, and plot the machine--learned relationships between $x$ and $y$'s.

```
def play_around_with_GP(alpha=None):
  length_scales=[1,0.05,0.001,0.0001]
  plt.figure(figsize=(20,5))

  for i in range(4):
    #As above, set up a Gaussian Process object with the kernel we want
    kernel = RBF(length_scale=length_scales[i])
    if alpha is None:
      gp = GaussianProcessRegressor(kernel=kernel)
    else:
      gp = GaussianProcessRegressor(kernel=kernel,alpha=alpha)

    #Train our ML models by letting it observing our training set
    gp.fit(x_train,y_train)

    #Plot routines carried out from above, plot the raw data, training points etc.
    plt.subplot(1,4,i+1)
    lj.plot_analytic_potential(c='#f1e0d6',label='True Function')
    plt.scatter(x_sample,y_sample,marker='o',s=7,facecolor='#525b56',alpha=0.5,label='experimental observations')
    plt.scatter(x_train,y_train,marker='o',s=15,facecolor='b',label='training points')

    #now we see what the learned model are by evaluating the functional value at
    #a range of x's
    y_mean, y_std = gp.predict(np.array(x_sample)[:, np.newaxis], return_std=True)
    plt.plot(x_sample,y_mean,'r-',lw=2,label='ML Approximation')

    plt.title('RBF $l$='+str(length_scales[i]))

    plt.legend()
  plt.tight_layout()

play_around_with_GP(alpha=None)
```

As you can see from this initial trial, the parameter $\ell$ plays a rather significant role in how well ML can ***surrogate*** a good mathematical realtionship between $x$ and $y$. For the case where 20% of our data are used to train the model, $\ell=0.001$ seems to be a reasonable choice that gives a function which more or less follows the behaviour of a LJ potential.

However, you can also notice that the learned function is very oscillatory compared to the true LJ function, as the surrogated model tries to go pass the training points ***exactly***. But remember, when we generated the data, we introduced some ***random noises*** into the data to mimick what happens under realistic experimental situations. So we need to tell our ''machine'' that there are noises in the data, and don't try to force itself to reproduce the exact values of the training point. This can be controlled by specifying the noise level through the parameter $\alpha$.

```
play_around_with_GP(alpha=0.2)
```

#**Optimising the model**

# The Error Function

Now, for most of the machine learning models that we will actually deal with, we cannot really inspect the machine-learned model as above to judge the quality of the model. This is because the problem we are trying to deal with are in *high-dimensional space* and often difficult to visualize. (Although we can get some help from **Unsupervised Machine Learning**, we won't go into this and make the problem too complicated).

The most common way for us to optimise the model is to monitor the change of **error function** as we scan through a set of parameters to help us finding the best choice. The  **error function** simply measures the difference between the predicted value from the ML model and the true value.

Here we use a slightly more complicated error function to make it numerically more tractable.

\begin{equation}
\mbox{Error Function}=\frac{\sum_{i=1}^{N_{test}}\left|E^{ref}_{i}-E^{GP}_{i}\right|}{\sum_{i=1}^{N_{test}}\left|E^{ref}_{i}-\overline{E^{ref}}\right|},
\end{equation}

Moreover, for each $\ell$, we will train over 5 set of different training set randomly sampled from the data and take the average of it.

# Optimising the length scale $\ell$ for RBF kernel

```
import warnings
warnings.filterwarnings('ignore')

#Define a series of parameters that we are going to scan through
length_scales=np.logspace(-4, 1, num=150)

def get_train_test_set(x_sample,y_sample,training_pt_fraction=0.2):
  """
  Define a helper function to randomly select a set of points for training
  and validation. The portion of the training points is specified by
  the training_pt_fraction.
  """
  import random
  ids=[i for i in range(len(x_sample))]
  train_ids = random.sample(ids, int(training_pt_fraction*len(x_sample)))
  test_ids = [i for i in ids if i not in train_ids]

  training_set_x = [x_sample[i] for i in train_ids]
  training_set_y = [y_sample[i] for i in train_ids]
  test_set_x =  [x_sample[i] for i in test_ids]
  test_set_y =  [y_sample[i] for i in test_ids]
  return np.array(training_set_x).reshape(-1, 1), np.array(training_set_y).reshape(-1, 1), np.array(test_set_x).reshape(-1, 1), np.array(test_set_y).reshape(-1, 1)

y_mean = np.mean(y_sample)

mean_error = []

for i in range(len(length_scales)):
  kernel = RBF(length_scale=length_scales[i])

  #note that we fix alpha=0.2 here, of course, it may also be a parameter to be optimized
  gp = GaussianProcessRegressor(kernel=kernel)

  this_error = []
  for j in range(5): # test the same length scale 5 times to collect some statistics
    x_train,y_train,x_test,y_test = get_train_test_set(x_sample,y_sample)
    gp.fit(x_train,y_train)
    y_predict = gp.predict(x_test, return_std=False)

    #calculate the error function
    nominator = sum([abs(y_predict[k]-y_test[k]) for k in range(len(y_predict))])
    denominator = sum([abs(y_predict[k]-y_mean) for k in range(len(y_predict))])
    error = nominator/denominator
    this_error.append(error)

  mean_error.append(np.mean(error))

#plot the training curve for optimizing ell
plt.plot(length_scales,mean_error,'ro')
plt.xscale('log')

plt.xlabel("length_scale")
plt.ylabel("Error function")
```

# How much data should we use to train the model?

One of the most important parameter to choose for ML, which is independent from what exactly the ML method is, would be the number of data points that is needed to properly train an ML model.

It may be natural to think that the more data to train the model, the better the quality it would be. What this belief reflects is really a competition between two of the followings:

* The error on training set (i.e. how well does the model reproduce the data that it was trained upon?), and
* The error on validation/test set (i.e. how well can the model predict the values of the unseen data?)

Let's look at how our model behaves when we change the training set fraction while keeping $\ell$ and $\alpha$ fixed.

```
training_set_fractions = [0.1+0.0125*i for i in range(66)]

mean_error_training=[]
mean_error_validation=[]

y_mean = np.mean(y_sample)

for tf in training_set_fractions:
  kernel = RBF(length_scale=length_scales[i])

  #note that we fix alpha=0.2 here, of course
  gp = GaussianProcessRegressor(kernel=kernel,alpha=0.2)

  this_training_error = []
  this_validation_error = []
  for j in range(5): # test the same length scale 5 times to collect some statistics
    x_train,y_train,x_test,y_test = get_train_test_set(x_sample,y_sample,training_pt_fraction=tf)
    gp.fit(x_train,y_train)

    #error for the training set
    y_predict = gp.predict(x_train, return_std=False)
    nominator = sum([abs(y_predict[k]-y_train[k]) for k in range(len(y_predict))])
    denominator = sum([abs(y_predict[k]-y_mean) for k in range(len(y_predict))])
    error = nominator/denominator
    this_training_error.append(error)

    #error for the validation set
    y_predict = gp.predict(x_test, return_std=False)
    nominator = sum([abs(y_predict[k]-y_test[k]) for k in range(len(y_predict))])
    denominator = sum([abs(y_predict[k]-y_mean) for k in range(len(y_predict))])
    error = nominator/denominator
    this_validation_error.append(error)

  mean_error_training.append(np.mean(this_training_error))
  mean_error_validation.append(np.mean(this_validation_error))

plt.plot(training_set_fractions,mean_error_training,'bo-',label='Training Error')
plt.plot(training_set_fractions,mean_error_validation,'ro-',label='Validation Error')
plt.legend()
plt.xlabel("Training Set Fraction")
plt.ylabel("Error function")
plt.yscale('log')
```

Note that here we are only using a simple code to demonstrate why you should be testing your train/test set partitions to tune the performances of your ML model. In real work, this is more involved, as there are statistical concerns on how to choose them such that the evaulations are statically significant.

[Cross Validation](https://scikit-learn.org/stable/modules/cross_validation.html#cross-validation) is a pragmatic way on how this can be properly done to avoid under-/over-training an ML model. Practically it will be too involved to go through C.V. in this workshop, and I encourage you to read the documentations if you are seriously thinking about doing ML!

#**Enhaced Sampling - A More Advanced Topic**

You may have noticed that when we first generated the samples $\left\{(x_{i},y_{i})\vert y_{i}=\mbox{LJ} (x_{i})+\varepsilon_{i}\right\}$, where $\varepsilon_{i}$ is a random noise, all the $x$'s are rather uniformly smapled, but the $y$'s are heavily peaked at $y\approx 0$, because of the unique shape of the Lennard-Jones potential.

For many applications, it is often the $y$'s that we want to sample more evenly. A typical application scenario will be in the experiments, where we can only control the experimental parameters $x$ to get some results $y$. After performing $N$ experiments, we saw overwhelmingly results concentrated on a few points, with a few outliers. The question we want to ask is then, how can we finely tune our input parameter $x$ so we can observe more data around the outliers?

This is an extremely difficult task if we dont really know how $y$ is related to $x$ from random experimental observations. But if we can use ML to learn the ***inverse relationship from $y$ back to $x$***, then it gives us a handle to more uniformly sample $y$ and then somehow infer back to a probable $x$ from which the desired $y$ can be obtained.

We will try to demonstrate this with the code below.

```
x_predict_y_uniform=[0 for _ in range(10000)]
for i in range(10):
  gp = GaussianProcessRegressor(kernel=kernel,alpha=0.2)
  x_train,y_train,x_test,y_test = get_train_test_set(x_sample,y_sample,training_pt_fraction=0.5)

  #We train a model to learn x from y rather than to learn y from x
  gp.fit(y_train,x_train)
  x_predict = gp.predict(np.array(y_sample).reshape(-1, 1) , return_std=False)

  import random

  #now we draw a uniform set of samples from y and try to predict the corresponding x
  y_uniform=[-60+random.randrange(0,210) for _ in range(10000)]
  _x_predict_y_uniform = gp.predict(np.array(y_uniform).reshape(-1, 1) , return_std=False)

  x_predict_y_uniform += _x_predict_y_uniform[:,0]/10

  if i==0:
    plt.figure(figsize=(8,8))
    plt.subplot(2,2,1)
    plt.scatter(x_sample,y_sample,marker='o',s=7,facecolor='#525b56',alpha=0.5,label='experimental observations')
    plt.xlabel('$x$')
    plt.ylabel('$y$')
    plt.xlim([0.4,1.2])
    plt.plot(_x_predict_y_uniform,y_uniform,'b.',label='ML learned relationship')
    plt.legend()

plt.subplot(2,2,2)

plt.hist(y_uniform,bins=50,orientation='horizontal',color='#525b56',alpha=0.6,density=True);
plt.title("Probability Distribution in $y$")

plt.subplot(2,2,3)

#Here we will plot the distribution of x, if, we drawn samples of x according to
#   such a distribution, then we will have some confidence that it will lead to
#   a more uniformly distributed y
plt.hist(list(x_predict_y_uniform),bins=50,color='#525b56',alpha=0.6,density=True);
plt.title("Probability Distribution in $x$ averaged 10 times")
plt.xlim([0.4,1.2])

plt.tight_layout()
```

As you can see from above, after averaging 10 trained GP models, the optimum distribution of $x$ that leads to a uniform samling of $y$ is a Gaussian type distribution centered around 0.75. If now we draw samples of $x$ from this learned distribution, we would have a better chance to sample the region of $y$ where the data changes most rapidly.

# **Wrapping Up**

So, we have come to the end of this quick introductory tutorial on supervised machine-learning using scikit-learn. We have gone through the process of training the ML model, which is more or less common to many mature ML techniques that has been developed. However, to really make ML work for you, there are still many nity-gritties that you need to consider/work on, so you can get obtained the best outcomes. Some of the things that you need to think about include:

* What kind of data do I have, how do I get the data?
* How do I represent the data mathematically so something useful can be learned from it? (This itself is a big research topic in ML itself)
* Have I trained the model properly? Is there anything I can do to train the model better?
* And lastly, what can I get out from the ML model? Canit provide me with some new insights that were difficult to be discovered otherwise?

```

```


```python
from sympy import *
init_printing(use_unicode=True)
```

```python
import gc
gc.collect()
```

```python
x = Symbol('x', commutative=False)
y = Symbol('y', commutative=False)
X = Symbol('X', commutative=False)
Y = Symbol('Y', commutative=False)
eps = Symbol('\epsilon')
```

```python
def comm(a,b):
    return a*b*a**-1*b**-1
```

```python
_series_dict = {}
_magnus_dict = {}
def magnus(w):
    if w in _magnus_dict:
        return _magnus_dict[w]

    expr = (w.subs(x,1+eps*X).subs(y,1+eps*Y) - 1).series(eps, n=None)
    return expr
```

```python
w = comm(x,y)
```

```python
((1 + eps*X)**-1).series?
```

```python
((1 + eps*X)**-1).series
```

```python
def magnus(w, n=None):
    srs = (w.subs({x:1+eps*X, y:1+eps*Y}) - 1).series(eps,n = None if n is None else n+1)
    if n is None:
        return next(srs).subs(eps,1)
    return srs.removeO() / eps**n
```

```python
w = comm(w,y)
```

```python
%timeit magnus(w,3)
%timeit magnus(w)
```

    689 ms ± 7.16 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
    849 ms ± 6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

```python
Expr.subs?
```

```python
theta = lambda expr : expr.subs({x:x**-1, y:y**-1})
```

```python
w = comm(w,theta(w))
w
```

```python
a = [x]
b = [y]
for n in range(20):
    a_ = a[-1]
    b_ = b[-1]
    a.append(a_ * b_)
    b.append(a_**-1 * b_**-1)
```

```python
n=3
a[n],b[n]
```

Claim:
a[n] = x..x if n == 0 (3)
a[n] = x..y if n == 1 (3)
a[n] = x..y^-1 if n == 2 (3)
and
b[n] = y..y if n == 0 (3)
b[n] = x^-1..y^-1 if n == 1 (3)
b[n] = y^-1..x if n == 2 (3)
and there is never any cancellation

```python
for n in range(10):
    %timeit -n1 -r1 print(magnus(a[n]))
```

```python
import numpy as np
np.log((84/18.1, 191/84, 6.16*1000 / 191, 28*60/6.16)) / np.log(2)
```

    array([ 2.21439963,  1.18511141,  5.01128581,  8.09131517])

```python
def bch(w, n):
    expr = w.subs(x,exp(eps*X)).subs(y,exp(eps*Y)).series(eps, n=n+1) - 1
    return expr.removeO() / eps**n
```

```python
%timeit magnus(w,2)
%timeit bch(w,2)
```

    167 ms ± 64.6 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
    546 ms ± 63.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

```python
def degree(w):
    if w == 1:
        return 0 # should be inf actually
    d = 1
    while magnus(w,d) == 0:
        d += 1
    return d
```

```python
def degree_var(w):
    if w == 1:
        return 0
    d = 1
    while magnus_var(w,d) == 0:
        d += 1
    return d
```

```python
def decode_word(n):
    letters = [x, y, x**-1, y**-1]
    w = x
    i = 0 # index of letter
    while n > 0:
        i = (i + n % 3 - 1) % 4 # change by 1, 0, or -1
        w *= letters[i]
        n //= 3
    return w
```

```python
def length_word(n):
    count = 0
    while n > 0:
        n //= 3
        count += 1
    return count
```

```python
w = comm(comm(x,y),y)
magnus_var(w,3)
```

```python
%timeit magnus(w,3)
%timeit magnus_var(w,3)
```

    929 ms ± 45.3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
    525 µs ± 5.58 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

```python
%timeit degree_var(decode_word(10))
```

    302 µs ± 47.8 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)

```python
%timeit degree(decode_word(10))
```

    108 ms ± 2.73 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

```python
for n in log_progress(range(1000), every=1):
    magnus(decode_word(n))
```

```python
for n in log_progress(range(1000), every=10):
    degree(decode_word(n))
```

```python
from log_progress import log_progress

alpha = {}
for n in log_progress(range(1000000), every=1000):
    if n not in deg_dict:
        deg_dict[n] = degree_var(decode_word(n))
    l = length_word(n)
    d = deg_dict[n]
    if d not in alpha:
        alpha[d] = l
        print(d, l, decode_word(n))
```


```python
import networkx as nx
import matplotlib.pyplot as plt
import pprint
pp = pprint.PrettyPrinter(indent=4)
%matplotlib inline
```

Community detection

Communities are structures within networks where the links within a community are denser than the links out of the community. The problem of community detection is of particular interest because they often correspond to structurally interesting properties of the network, e.g. factions in social networks, similar functional groups in metabolic networks

Examples

Political books purchased on Amazon

(Newman 2003) Applying community detection algorithms to the co-purchasing network of political books on Amazon divides the network into two groups that nearly exactly reproduce the political affiliations of the authors.

Karate Network

(Zachary 1977) A karate club split into two factions after a conflict between members. This graph is often used as a benchmark test for community detection algorithms; we will use it in the following examples.

Dolphin Network

(Lusseau, et al. 2003) A similar division happened in a dolphin social network, after one member of the group left, the dolphin community separated into two separate, non-interacting pods.

Example network: Karate Club

```python
karate = nx.karate_club_graph()
#Coloring factions
(faction1_col, faction2_col) = ('yellow','lightblue')
#Building list of colors
node_color = [''] * len(karate.nodes())
node_dict = dict(karate.nodes(data=True))
#Assign colors based on factions
for n in karate.nodes():
    if node_dict[n]['club'] == 'Mr. Hi':
        node_color[n] = faction1_col
    elif node_dict[n]['club'] == 'Officer':
        node_color[n] = faction2_col
    else:
        print "Something is broken"
        break;
# Initialize the position
pos = nx.spring_layout(karate,scale=0.2)
# Label nodes
nx.draw_networkx_labels(karate,pos,font_size=10,font_color='black')
# Highlight the leaders
nx.draw_networkx_nodes(karate,pos,{0,33},node_color=['black','black'],node_size=700)
# Now draw all the nodes, including leaders, using faction color scheme.
nx.draw_networkx_nodes(karate,pos,node_color=node_color,node_size=500)
nx.draw_networkx_edges(karate,pos)
plt.show()
```

Algorithms

Algorithm: $k$-clique-community detection

$k$-clique detection finds communities of cliques of minimum degree $k$ that share at least $k-1$ nodes. The algorithm uses a percolation method to find these communities: a $k$-clique topological "template" is fitted over a clique, and one node of the clique is switched with another connected node outside of the clique. If this group of nodes is also a clique, that node is added to the $k$-clique-community. This is repeated until all cliques are visited.

```python
from networkx.algorithms.community import k_clique_communities
```

```python
c = list(k_clique_communities(karate,3))
pp.pprint(c)
```

    [   frozenset([   0,
                      1,
                      2,
                      3,
                      7,
                      8,
                      12,
                      13,
                      14,
                      15,
                      17,
                      18,
                      19,
                      20,
                      21,
                      22,
                      23,
                      26,
                      27,
                      28,
                      29,
                      30,
                      31,
                      32,
                      33]),
        frozenset([24, 25, 31]),
        frozenset([0, 4, 5, 6, 10, 16])]

```python
nx.draw_networkx(karate,pos)
nx.draw_networkx_nodes(karate,pos, with_labels=True,nodelist=c[0],node_color=['yellow'])
nx.draw_networkx_nodes(karate,pos, with_labels=True,nodelist=c[1],node_color=['blue'])
nx.draw_networkx_nodes(karate,pos, with_labels=True,nodelist=c[2],node_color=['green'])
nx.draw_networkx_edges(karate,pos)
plt.show()
```

Algorithm: Girvan-Newman

This method finds communities by iteratively removing edges with maximum betweenness centrality, recalculating centrality for the affected nodes, then removing the max-betweenness edge, etc.

Betweenness centrality for edge $e$ is calculated as:
\begin{align}
g(e) = \sum_{s \neq t} \frac{\sigma_{st}(e)}{\sigma_{st}}
\end{align}
where $s,t$ are the starting and terminal nodes, and $\sigma_{st}$ is the number of shortest paths from $s\rightarrow t$, and $\sigma_{st}(e)$ is the number of those shortest paths that contain edge $e$

Determining when to stop removing edges is a free parameter in the algorithm. Typical choices include: reaching a predefined number of separate remaining subgraphs or maximization of modularity (discussed later).

```python
from networkx.algorithms.community.centrality import girvan_newman
import itertools
```

```python
c = list(girvan_newman(karate))
#pp.pprint(c)
k = 6 ### maximun number of clusters
divisions =0;
comp = girvan_newman(karate)

limited = itertools.takewhile(lambda c: len(c) <= k, comp)
for communities in limited:
    divisions+=1
    print(tcuple(sorted(c) for c in communities))

print 'Number of hierarchies:',divisions
print 'Selecting',k,'clusters:',c[divisions-1]
print 'First division: ',c[0]
print 'Second Division: ',c[1]

```

    ([0, 1, 3, 4, 5, 6, 7, 10, 11, 12, 13, 16, 17, 19, 21], [2, 8, 9, 14, 15, 18, 20, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33])
    ([0, 1, 3, 4, 5, 6, 7, 10, 11, 12, 13, 16, 17, 19, 21], [2, 8, 14, 15, 18, 20, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33], [9])
    ([0, 1, 3, 7, 11, 12, 13, 17, 19, 21], [2, 8, 14, 15, 18, 20, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33], [4, 5, 6, 10, 16], [9])
    ([0, 1, 3, 7, 11, 12, 13, 17, 19, 21], [2, 24, 25, 27, 28, 31], [4, 5, 6, 10, 16], [8, 14, 15, 18, 20, 22, 23, 26, 29, 30, 32, 33], [9])
    ([0, 1, 3, 7, 12, 13, 17, 19, 21], [2, 24, 25, 27, 28, 31], [4, 5, 6, 10, 16], [8, 14, 15, 18, 20, 22, 23, 26, 29, 30, 32, 33], [9], [11])
    Number of hierarchies: 5
    Selecting 6 clusters: (set([0, 1, 3, 7, 12, 13, 17, 19, 21]), set([2, 24, 25, 27, 28, 31]), set([16, 10, 4, 5, 6]), set([32, 33, 8, 14, 15, 18, 20, 22, 23, 26, 29, 30]), set([9]), set([11]))
    First division:  (set([0, 1, 3, 4, 5, 6, 7, 10, 11, 12, 13, 16, 17, 19, 21]), set([32, 33, 2, 8, 9, 14, 15, 18, 20, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31]))
    Second Division:  (set([0, 1, 3, 4, 5, 6, 7, 10, 11, 12, 13, 16, 17, 19, 21]), set([32, 33, 2, 8, 14, 15, 18, 20, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31]), set([9]))

```python
## Let's draw the second Division which has 3 clusters
cd = list(c[1])
```

```python
nx.draw_networkx(karate,pos)
nx.draw_networkx_nodes(karate,pos, with_labels=True,nodelist=cd[0],node_color=['yellow'])
nx.draw_networkx_nodes(karate,pos, with_labels=True,nodelist=cd[1],node_color=['blue'])
nx.draw_networkx_nodes(karate,pos, with_labels=True,nodelist=cd[2],node_color=['green'])
nx.draw_networkx_edges(karate,pos)
plt.show()
```

Metric: Modularity

Modularity is a network metric that measures how much community structure a network has. The modularity of a network calculated against a null random graph generated via the configuration model. The configuration model takes a prescribed degree distribution but has a random rewiring of edges. One can view the null configuration model graph as cutting all the edges of the original graph, but preserving the "stubs" on the vertices, and randomly rewiring all the stubs.

The modularity of a graph with $c$ communities is then calculated as:
\begin{align}
Q = \frac{1}{2m} \sum_{vw} \big[\frac{A_{vw}}{2m} - \frac{k_v k_w}{(2m)^2}\big]\delta(c_v, c_w)
\end{align}
where $m$ is the total number of edges, $v,w$ are nodes in the graph, $A_{vw}$ is the actual number of edges between $v$ and $w$, $k_v$ is the degree of the node, and $\delta$ is the dirac delta function.

The modularity thus measures how much more within-community connectivity we observe in the network over the null model (where connectivity is proportional simply to the degree of the node). By choosing which communities the vertices belong to, we can try to maximize $Q$.

Algorithm: Modularity Maximization via spectral methods

(Newman 2006) shows that the eigenvector corresponding to the least-negative eigenvalue of the modularity matrix with elements $B_{vw} = A_{vw} - \frac{k_v k_w}{2m}$ can be used to partition the network into two communities that maximizes modularity. The process can be repeated on each subgraph to further divide the group into more communities.

The leading eigenvector $V$ has length equal to the number of nodes in the network. Based on the sign of the $i^{th}$ element of $V$, node $i$ is assigned to group 1 if it's positive and 2 if it's negative.

Algorithm: Modularity Maximization Heuristics

(Louvain 2008) offers a percolation-inspired heuristic for community detection. The algorithm has two phases that are applied iteratively: the first phase is a greedy algorithm that places all nodes in their own community, then each node is moved to the community that gives the maximum gain in modularity. This is repeated for all nodes until no more modularity improvement can be made.

The second phase builds a new graph where the communities from phase 1 are collapsed into a single node, and the edges between communities collapse into single edges between the respective nodes with equivalent weight. Then phase 1 is repeated on this reduced graph, and the passes continue until no changes occur over a pass.

```python
pip install python-louvain
```

```python

```