```python
%matplotlib inline
import matplotlib.pyplot as plt

# Some nice default configuration for plots
plt.rcParams['figure.figsize'] = 10, 7.5
plt.rcParams['axes.grid'] = True
plt.gray()

import numpy as np
import pandas as pd
```

    <matplotlib.figure.Figure at 0x116eae710>

# 6. Linear Regression and Its Cousins

We will dicuss several models here, all of which are akin to linear regression in that each can directly or indirectly be written in the form
$$y_i = \beta_0 + \beta_1 x_{i1} + \cdots + \beta_p x_{ip} + \epsilon_i,$$
where $y_i$ represents the numeric response for the $i$th sample, $\beta_0$ represents the estimated intercept, $\beta_j$ represents the estimated coefficient for the $j$th predictor, $x_{ij}$ represents the value of the $j$th predictor for the $i$th sample, and $\epsilon_i$ represents random error that cannot be explained by the model. In addition to ordinary linear regression, these types of models include partial least square (PLS) and penalized models such as ridge regression, the lasso, and the elastic net.

Each of these models seek to find estimates of the parameters so that the sum of squared errors or a function of the sum of squared errors is minimized. The objectives of the methods find parameter estimates that fall along the spectrum of the bias-variance trade-off. Ordinary linear regression, at one extreme, finds parameter estimates that have minimum bias, where ridge regression, the lasso, and the elastic net find estimates that have lower variance.

A distinct advantage of linear regression models is that they are highly interpretable. Furthermore, relationships among predictors can be further interpreted through the estimated coefficients. Another advantage is that their mathematrical nature enables us to compute standard errors of the coefficients, provided that we make certain assumptions about the distribution of the model residuals.

While linear regression-type models are highly interpretable, they can be limited in their usefulness. First, these models are appropriate when the relationship between the predictors and response falls along a hyperplane. If there is a curvilinear relationship between the predictors and response, then linear regression models can be augmented with additional predictors that are functions of the original predictors in an attempt to capture these relationships.

## 6.1 Case Study: Quantitative Structure-Acitivity Relationship Modeling

The study investigated a set of compounds with corresponding experimental solubility values using complex sets of descriptors. We will use 1,267 compounds (951 training and 316 test by random sampling) and a set of more understandable descriptors:
- Two hundred and eight binary "fingerprints" that indicate the presence or absence of a particular chemical substructure.
- Sixteen count descriptors, such as the number of bonds or the number of bromine atoms.
- Four continuous descriptors, such as molecular weight or surface area.

The training set will be used to tune and estimate models, as well as to determine initial estimates of performance using repeated 10-fold cross-validation. The test set will be used for a final characterization of the model of interest.

```python
!ls -l ../datasets/solubility/
```

    total 3016
    -rw-r--r--  1 leigong  staff  153513 Nov 24 15:58 solTestX.csv
    -rw-r--r--  1 leigong  staff  226232 Nov 24 15:58 solTestXtrans.csv
    -rw-r--r--  1 leigong  staff    2996 Nov 24 15:58 solTestY.csv
    -rw-r--r--  1 leigong  staff  459104 Nov 24 15:58 solTrainX.csv
    -rw-r--r--  1 leigong  staff  679218 Nov 24 15:58 solTrainXtrans.csv
    -rw-r--r--  1 leigong  staff    9202 Nov 24 15:58 solTrainY.csv

```python
# training
trainX = pd.read_csv("../datasets/solubility/solTrainX.csv").drop("Unnamed: 0", axis=1)
trainY = pd.read_csv("../datasets/solubility/solTrainY.csv").drop("Unnamed: 0", axis=1)

# test
testX = pd.read_csv("../datasets/solubility/solTestX.csv").drop("Unnamed: 0", axis=1)
testY = pd.read_csv("../datasets/solubility/solTestY.csv").drop("Unnamed: 0", axis=1)
```

```python
trainX.head(5)
```

      FP001
      FP002
      FP003
      FP004
      FP005
      FP006
      FP007
      FP008
      FP009
      FP010
      ...
      NumCarbon
      NumNitrogen
      NumOxygen
      NumSulfer
      NumChlorine
      NumHalogen
      NumRings
      HydrophilicFactor
      SurfaceArea1
      SurfaceArea2

      0
       0
       1
       0
       0
       1
       0
       0
       1
       0
       0
      ...
       14
       2
       0
       0
       0
       0
       3
      -0.856
       25.78
       25.78

      1
       0
       1
       0
       1
       1
       1
       1
       1
       0
       0
      ...
       21
       3
       1
       1
       0
       0
       4
      -0.370
       52.19
       80.43

      2
       1
       1
       1
       1
       1
       0
       0
       1
       0
       1
      ...
       13
       0
       2
       0
       0
       0
       1
      -0.330
       37.30
       37.30

      3
       0
       0
       1
       0
       0
       0
       1
       0
       0
       0
      ...
       10
       0
       0
       0
       0
       0
       1
      -0.960
        0.00
        0.00

      4
       0
       0
       1
       1
       1
       1
       0
       0
       1
       0
      ...
        9
       5
       0
       0
       1
       1
       1
      -0.069
       53.94
       53.94

5 rows × 228 columns

```python
trainY.head(5)
```

      0

      0
      -3.97

      1
      -3.98

      2
      -3.99

      3
      -4.00

      4
      -4.06

```python
print "training set size: {0} \ntest set size: {1}".format(trainX.shape, testX.shape)
```

    training set size: (951, 228)
    test set size: (316, 228)

```python
print "The range of the outcome is {0} with an avearge of {1}.".format((round(np.min(trainY)[0], 2),
                                                                        round(np.max(trainY)[0], 2)),
                                                                       round(np.mean(trainY)[0], 2))
```

    The range of the outcome is (-11.62, 1.58) with an avearge of -2.72.

Relationship between the outcome and two types of descriptors.

```python
fig, (ax1, ax2) = plt.subplots(1, 2)

ax1.scatter(trainX['MolWeight'], trainY)
ax1.set_xlabel("Molecular Weight")
ax1.set_ylabel("Solubility (log)")

trainY_strPresent = trainY.iloc[np.where(trainX['FP003'] == 1)[0]]
trainY_strAbsent = trainY.iloc[np.where(trainX['FP003'] == 0)[0]]
trainY_str = [trainY_strPresent, trainY_strAbsent]
ax2.boxplot(trainY_str)
ax2.set_xticklabels(['structure present', 'structure absent'])
ax2.set_ylabel("Solubility (log)")
```

It is useful to explore the training set to understand the characteristics of the data prior to modeling.

```python
# evaluate the continuous predictors for skewness
trainX_c = trainX[trainX.columns[-20:]] # continuous predictors

from scipy.stats import skew
trainX_skewness = trainX_c.apply(skew)
print "Summary of skewness: range {0}; mean {1}".format((round(np.min(trainX_skewness), 2),
                                                         round(np.max(trainX_skewness), 2)),
                                                        round(np.mean(trainX_skewness), 2))
```

    Summary of skewness: range (0.67, 3.84); mean 1.65

It indicates that these predictors have a propensity to be right skewed (skewness = 0 if normally distributed). To correct for this skewness, a Box-Cox transformation was applied to all predictors (add a shift parameter to non-positive predictors).

#### Side note

One-parameter Box-Cox transformations (for x > 0)
\begin{equation}
x^* = \begin{cases} {x^{\lambda}-1 \over \lambda} & \text{if} \ \lambda \neq 0 \\ log(x) & \text{if} \ \lambda = 0 \end{cases}
\end{equation}

Two-parameter Box-Cox transformations (for x > $\lambda_2$)
\begin{equation}
x^* = \begin{cases} {(x+\lambda_2)^{\lambda_1}-1 \over \lambda_1} & \text{if} \ \lambda_1 \neq 0 \\ log(x+\lambda_2) & \text{if} \ \lambda_1 = 0 \end{cases}
\end{equation}

```python
# Box-Cox transformed predictors
trainX_trans = pd.read_csv("../datasets/solubility/solTrainXtrans.csv").drop("Unnamed: 0", axis=1)
testX_trans = pd.read_csv("../datasets/solubility/solTestXtrans.csv").drop("Unnamed: 0", axis=1)
```

###### Q1: Is it safet to assume that the relationship between the predictors and outcome is linear?

```python
trainX_trans_c = trainX_trans[trainX_trans.columns[-20:]] # 20 continuous predictors

# lowess (local smoother)
import statsmodels.api as sm
lowess = sm.nonparametric.lowess
```

```python
def plotContinuousPredictors(X, y, nrows=4, ncols=5):
    '''plot the continous predictors (default 20) versus the outcome'''
    fig, axarr = plt.subplots(nrows, ncols)
    fig.tight_layout()

    for r in range(nrows):
        for c in range(ncols):
            plotOnePredictor(axarr[r, c], X.iloc[:, (r+1)*(c+1)-1], y)

def plotOnePredictor(ax, X, y):
    '''plot one predictor versus the outcome'''
    z = lowess(np.squeeze(y), np.squeeze(X), 0.88) # return sorted X and y_fit
    ax.scatter(X, y, alpha=0.4)
    ax.plot(z[:, 0], z[:, 1], 'r', linewidth=2)
    ax.set_title(X.name)

plotContinuousPredictors(trainX_trans_c, trainY)
```

The smoothed regression lines indicate that there are some linear and non-linear relationship between the predictors and the outcome. Because of this, we might consider augmenting the predictor set with quadratic terms for some variables.

###### Q2: Are there significant between-predictor correlations?

```python
# PCA scree plot
from sklearn.decomposition import PCA
from sklearn.preprocessing import scale

# scale before PCA
trainX_trans_scale = scale(trainX_trans)

pca = PCA()
pca.fit(trainX_trans_scale)

# generate scree plot
plt.plot(pca.explained_variance_ratio_)
plt.xlim((-5, None))
plt.ylim((-0.005, None))
plt.xlabel('Percent of Total Variance')
plt.ylabel('Component')
```

Notice that the amount of variability summarized by components drops sharply, with no one component accounting for more than 13% of the variance. This profile indicates that the structure of the data is contained in a much smaller number of dimensions than the number of dimensions of the original space; this is often due to a large number of collinearities among the predictors.

```python
# correlation matrix for continuous predictors
corr_dataframe_c = trainX_trans_c.corr()

# compute hierarchical cluster on both rows and columns for correlation matrix and plot heatmap
def corr_heatmap(corr_dataframe):
    import scipy.cluster.hierarchy as sch

    corr_matrix = np.array(corr_dataframe)
    col_names = corr_dataframe.columns

    Y = sch.linkage(corr_matrix, 'single', 'correlation')
    Z = sch.dendrogram(Y, color_threshold=0, no_plot=True)['leaves']
    corr_matrix = corr_matrix[Z, :]
    corr_matrix = corr_matrix[:, Z]
    col_names = col_names[Z]
    im = plt.imshow(corr_matrix, interpolation='nearest', aspect='auto', cmap='bwr')
    plt.colorbar()
    plt.xticks(range(corr_matrix.shape[0]), col_names, rotation='vertical', fontsize=12)
    plt.yticks(range(corr_matrix.shape[0]), col_names[::-1], fontsize=12)

# plot
corr_heatmap(corr_dataframe_c)
```

There are many strong positive correlations (dard red). As discussed in Chapter 3, this could create problems in developing some models and appropriate pre-processing steps will need to be taken to account for this problem.

## 6.2 Linear Regression

The objective of ordinary least square is find the plane that minimize the sum-of-squared errors (SSE) between the observed and predicted response: $$\text{SSE} = \sum_{i=1}^n (y_i - \hat{y}_i)^2,$$ where $y_i$ is the outcome and $\hat{y}_i$ is the model prediction of that sample's outcome. Mathematically, the optimal plane can be shown to be $$\hat{\beta} = (\pmb{X}^T \pmb{X})^{-1} \pmb{X}^T y,$$ where $\pmb{X}$ is the matrix of predictors, $y$ is the response vector, and $\hat{\beta}$ is the parameter estimates for each predictor. Making some assumptions about the distribution of the residuals, it is straightforward to show these estimates minimize the bias component of the bias-variance trade-off.

Notice that a unique inverse of $\pmb{X}^T \pmb{X}$ exists when (1) no predictor can be determined from a combination of one or more of the other predictors and (2) the number of samples is greater than the number of predictors. Note that linear regression can still be used for prediction when collinearity exists within the data. But we lose the ability to meaningfully interpret the coefficients. In practice, highly correlated predictors might be managed manually by removing one of the offending predictors. However, if the number of predictors is large, this may be difficult. Also, on many occasions, relationships among predictors can be complex and involve many predictors. In these cases, manual removal of specific predictors may not be possible and models that can tolerate collinearity may be more useful.

When condition (2) is true for a data set, we suggest using pre-processing techniques, such as removing pairwise correlated predictors and checking *variance inflation factor* (VIF). After pre-processing the data, if the number of predictors still outnumbers the number of observations, we need to consider PCA pre-processing, or simultaneous dimension reduction and regression via PLS or parameters shrinking estimates such as ridge regression, the lass or the elastic net.

Another drawback of multiple linear regression is that its solution is linear in the paramters. One visual clue to understanding if the relationship between predictors and the response is not linear is to examine the basic diagnostic plots. Curvature in the predicted-versus-residual plot is a primary indicator that the underlying relationship is not linear. Easily identifiable quadratic, cubic, or interactions between predictors can be accommodated in regression by add quadratic, cubic, and interactions of the original predictors.

A third notable problem with multiple linear regression is that it is prone to chasing observations that are away from the overall trend of the majority of the data. Observations that cause significant changes in the parameter estimates are called *influential*. One common approach to address these problems is to use an alternative metric to SSE that is less sensitive to large outliers, such as absolute errors and Huber function.

There are no tuning parameters for multiple linear regression. This fact, however, dose not impugn the practitioner from using rigorous model validation tools. In fact, we must use the same training and validation techniques to understand the predictive ability of this model on data which the model has not seen.

### Linear Regression for Solubility Data

Recall that we split the solubility data into training and test sets and applied a Box-Cox transformation to the continuous predictors in order to remove skewness. The next step is to identify predictors that have high pairwise correlations and to remove predictors so that no absolute pairwise correlation is greater than some pre-specified level.

In this case, we remove predictors that have pairwise correlations greater than 0.9 using a heuristic approach introduced in Chapter 3.

```python
corr_dataframe = trainX_trans.corr()

def remove_high_corr(corr_dataframe, thresh = 0.9):
    '''remove predictors with high pairwise correlation'''
    abs_corr = np.abs(corr_dataframe).as_matrix() # absolute correlation matrix
    col_names = list(corr_dataframe.columns)

    # set up diagonal to 0
    np.fill_diagonal(abs_corr, 0)

    print "Removed predictors (in order): \n"
    while np.max(abs_corr) >= thresh:
        i, j = np.unravel_index(abs_corr.argmax(), abs_corr.shape) # find maximum element
        # print abs_corr[i, j]
        rdx = which_to_remove(i, j, abs_corr)
        # remove corresponding predictor
        print col_names.pop(rdx)
        abs_corr = np.delete(abs_corr, rdx, 0)
        abs_corr = np.delete(abs_corr, rdx, 1)

    return col_names

def which_to_remove(i, j, abs_corr):
    '''compare two predictors and remove the one with higher abs correlation with other predictors'''
    i_absmean = np.mean(abs_corr[i, np.where(abs_corr[i,:] == 0)])
    j_absmean = np.mean(abs_corr[j, np.where(abs_corr[j,:] == 0)])

    return i if i_absmean > j_absmean else j

# remained predictors
col_remained = remove_high_corr(corr_dataframe)
trainX_trans_filtered = trainX_trans[col_remained]
testX_trans_filtered = testX_trans[col_remained]
```

    Removed predictors (in order):

    NumBonds
    NumSulfer
    FP203
    FP208
    NumNonHBonds
    NumChlorine
    NumHalogen
    FP183
    FP014
    FP194
    NumNitrogen
    NumAromaticBonds
    FP033
    FP166
    FP163
    FP180
    SurfaceArea2
    FP024
    FP175
    FP167
    FP141
    FP144
    FP155
    FP052
    FP056
    FP110
    NumCarbon
    FP005
    FP179
    FP136
    FP162
    FP087
    FP092
    FP112
    FP067
    FP051

Upon removing these predictors, a linear model was fit to the training data. The linear model was resampled using 10-fold cross-validation and the estimated RMSE and $\text{R}^2$ is reported.

```python
from sklearn.linear_model import LinearRegression
from sklearn.cross_validation import cross_val_score, ShuffleSplit

cv = ShuffleSplit(trainX_trans_filtered.shape[0], n_iter=10, random_state=0)

lm = LinearRegression()
train_scores_mse = cross_val_score(lm, trainX_trans_filtered, trainY.values, cv=cv,
                                   scoring = 'mean_squared_error')
train_scores_rmse = np.sqrt(-1.0 * train_scores_mse)

train_scores_r2 = cross_val_score(lm, trainX_trans_filtered, trainY.values, cv=cv,
                                  scoring = 'r2')
print "CV estimated RMSE: {0} \nCV estimated R2: {1}".format(np.mean(train_scores_rmse), np.mean(train_scores_r2))
```

    CV estimated RMSE: 0.707044315368
    CV estimated R2: 0.874723530558

```python
# apply to the test set
lm.fit(trainX_trans_filtered, trainY.values)
testY_pred = lm.predict(testX_trans_filtered)

from sklearn.metrics import r2_score, mean_squared_error
test_score_r2 = r2_score(testY.values, testY_pred)
test_score_rmse = np.sqrt(mean_squared_error(testY.values, testY_pred))

print "Test set RMSE: {0} \nTest set R2: {1}".format(test_score_rmse, test_score_r2)
```

    Test set RMSE: 0.742917835704
    Test set R2: 0.871834926473

```python
# regression diagnostic
fig, (ax1, ax2) = plt.subplots(1, 2)

ax1.scatter(testY_pred, testY.values)
ax1.set(xlim=(-11, 2), ylim=(-11, 2))
ax1.plot(ax1.get_xlim(), ax1.get_ylim(), ls="--", c=".3", linewidth=2)
ax1.set_xlabel("Predicted")
ax1.set_ylabel("Observed")

ax2.scatter(testY_pred, testY.values-testY_pred)
ax2.set(xlim=(-11, 2))
ax2.plot(ax1.get_xlim(), (0, 0), ls="--", c=".3", linewidth=2)
ax2.set_xlabel("Predicted")
ax2.set_ylabel("Residual")
```

There does not appear to be any bias in the prediction, and the distribution between the predicted values and residuals appears to be random about zero.

## 6.3 Partial Least Squares

If the correlation among predictors is high, then the ordinary least squares solution for multiple linear regression will have high variability and will become unstable. One the other hand, if the number of predictors may be greater than the number of observations, ordinary least squares in its usual form will be unable to find a unique set of regression coefficients that minimize the SSE.

A couple of common solutions include pre-processing the predictor by either (1) removal of the highly correlated predictors or (2) conducting PCA on the predictors. Removing highly correlated predictors ensures that pairwise correlations among predictors are below a pre-specified threshold. However, this process does not necessarily ensure that linear combinations of predictors are uncorrelated with each other, i.e., it may not guarantee a stable least squares solution. Alternatively, using PCA for pre-processing guarantees that the resulting predictors, or combinations thereof, will be uncorrelated. The trade-off in using PCA is that the new predictors are linear combinations of the original predictors, and thus, the practical understanding of the new predictors can become murky.

Pre-processing predictors via PCA prior to performing regression is known as principal component regression (PCR). While this two-step regression approach (dimension reduction, then regression) has been widely applied to problems with inherently highly correlated predictors or problems with more predictors than observations, it can easily be misled. Specifically, dimension reduction via PCA does not necessarily produce new predictors that explain the response, since PCA does not consider any aspects of teh response when it selects its components. Instead, it simply chases the variability present throughout the predictor space. Because of this inherent problem with PCR, we recommend using PLS when there are correlated predictors and a linear regression-type solution is desired.

PLS find linear combinations of the predictors just like PCA. While the PCA linear combinations are chosen to maximally summarize predictor space variability, the PLS linear combinations of predictors are chosen to maximally summarize covariance with the response. This means that PLS finds components that maximally summarize the variation of the predictors while simultaneously requiring these components to have maximum correlation with the response. It strikes a compromise between the objectives of predictor space dimension reduction and a predictive relationship with the reponse. In other words, PLS is a *supervised* dimension reduction procedure; PCR is an *unsupervised* procedure.

Prior to performing PLS, the predictors should be centered and scaled, especially if the predictors are on scale of differing magnitude. The PLS has one tuning parameter: the number of components to retain. Resampling techniques can be used to determine the optimal number of components.

### PCR and PLSR for Solubility Data

Return to our previous example, 10-fold cross-validation was used to determine the optimal number of components to retain that minimize RMSE in both PCR and PLSR.

```python
# RMSEs for 50 components were examined
pcr_rmse = np.zeros([50, 2])
plsr_rmse = np.zeros([50, 2])

# 10-fold CV
cv = ShuffleSplit(trainX_trans.shape[0], n_iter=10)
```

```python
def optimal_with_osr(rmse):
    '''Return the minimum RMSE and its # of components using "one-standard error" rule'''
    n_best = np.argmin(rmse[:, 0])
    rmse_min = rmse[n_best, 0] + rmse[n_best, 1]
    n_best = np.min(np.where(rmse[:, 0] <= rmse_min))
    return (n_best+1), rmse_min
```

```python
# PCR
from sklearn.decomposition import PCA

for n_compo in range(len(pcr_rmse)):
    pca = PCA(n_components= n_compo+1)
    pca_compo = pca.fit_transform(trainX_trans_scale)
    train_scores_mse = cross_val_score(LinearRegression(), pca_compo, trainY.values, cv=cv, scoring = 'mean_squared_error')
    pcr_rmse[n_compo, 0] = np.mean(np.sqrt(-1.0 * train_scores_mse))
    pcr_rmse[n_compo, 1] = np.std(np.sqrt(-1.0 * train_scores_mse))

n_best, rmse_min = optimal_with_osr(pcr_rmse)
print "With one-standard error rule, the minimum PCR RMSE is {0} with {1} components".format(rmse_min, n_best)
```

    With one-standard error rule, the minimum PCR RMSE is 0.860014338691 with 43 components

```python
# PLSR
from sklearn.cross_decomposition import PLSRegression

for n_compo in range(len(plsr_rmse)):
    plsr= PLSRegression(n_components= n_compo+1, scale=False)
    train_scores_mse = cross_val_score(plsr, trainX_trans_scale, trainY.values, cv=cv, scoring = 'mean_squared_error')
    plsr_rmse[n_compo, 0] = np.mean(np.sqrt(-1.0 * train_scores_mse))
    plsr_rmse[n_compo, 1] = np.std(np.sqrt(-1.0 * train_scores_mse))

n_best, rmse_min = optimal_with_osr(plsr_rmse)
print "With one-standard error rule, the minimum PLSR RMSE is {0} with {1} components".format(rmse_min, n_best)
```

    With one-standard error rule, the minimum PLSR RMSE is 0.726648755663 with 12 components

```python
plt_pcr, = plt.plot(pcr_rmse[:,0], '-x')
plt_plsr, = plt.plot(plsr_rmse[:,0], '-o')
plt.xlim((-1, None))
plt.xlabel("# Components")
plt.ylabel("RMSE (Cross-Validation)")
plt.legend([plt_pcr, plt_plsr], ['PCR', 'PLSR'])
```

With these data, we see the supervised dimension reduction finds a minimum RMSE with significantly fewer components than unsupervised dimension reduction.

```python
from scipy.stats import pearsonr

# compoare the first two components
pca2 = PCA(n_components=2)
pca2.fit(trainX_trans_scale)

plsr2 = PLSRegression(n_components=2, scale=False)
plsr2.fit(trainX_trans_scale, trainY.values)

# visulization
fig, axarr = plt.subplots(2, 2, sharex=True, sharey=True)
#fig.tight_layout()

axarr[0,0].scatter(pca2.transform(trainX_trans_scale)[:, 0], trainY.values)
axarr[0,0].set_title("PCA Component 1")
corr = pearsonr(pca2.transform(trainX_trans_scale)[:, 0], trainY.values[:,0])[0]
axarr[0,0].text(10, -12, 'corr:' + str(round(corr, 2)))

axarr[0,1].scatter(pca2.transform(trainX_trans_scale)[:, 1], trainY.values)
axarr[0,1].set_title("PCA Component 2")
corr = pearsonr(pca2.transform(trainX_trans_scale)[:, 1], trainY.values[:,0])[0]
axarr[0,1].text(10, -12, 'corr:' + str(round(corr, 2)))

axarr[1,0].scatter(plsr2.transform(trainX_trans_scale)[:, 0], trainY.values)
axarr[1,0].set_title("PLS Component 1")
corr = pearsonr(plsr2.transform(trainX_trans_scale)[:, 0], trainY.values[:,0])[0]
axarr[1,0].text(10, -12, 'corr:' + str(round(corr, 2)))

axarr[1,1].scatter(plsr2.transform(trainX_trans_scale)[:, 1], trainY.values)
axarr[1,1].set_title("PLS Component 2")
corr = pearsonr(plsr2.transform(trainX_trans_scale)[:, 0], trainY.values[:,0])[0]
axarr[1,1].text(10, -12, 'corr:' + str(round(corr, 2)))

fig.text(0.5, 0.06, 'Component Score', ha='center', va='center')
fig.text(0.06, 0.5, 'Solubility', va='center', rotation='vertical')
```

It is not surprising that the correlation between these components and the response is greater for PLS than PCR.

Prediction of the test set using the optimal PCR and PLS models is compared.

```python
from sklearn.preprocessing import scale
testX_trans_scale = scale(testX_trans)

# PCR
pca = PCA(n_components=34) # optimal
pca.fit(trainX_trans_scale)
trainX_pca = pca.transform(trainX_trans_scale)
testX_pca = pca.transform(testX_trans_scale)
lm = LinearRegression()
lm.fit(trainX_pca, trainY.values)
testY_pca_pred = lm.predict(testX_pca)

# PLSR
plsr = PLSRegression(n_components=7, scale=False)
plsr.fit(trainX_trans_scale, trainY.values)
testY_plsr_pred = plsr.predict(testX_trans_scale)
```

```python
# regression diagnostic
fig, axarr = plt.subplots(2, 2)

axarr[0, 0].scatter(testY_pca_pred, testY)
axarr[0, 0].set(xlim=(-11, 2), ylim=(-11, 2))
axarr[0, 0].plot(axarr[0,0].get_xlim(), axarr[0,0].get_ylim(), ls="--", c=".3", linewidth=2)
axarr[0, 0].set_xlabel("Predicted")
axarr[0, 0].set_ylabel("Observed")

axarr[0, 1].scatter(testY_pca_pred, testY-testY_pca_pred)
axarr[0, 1].set(xlim=(-11, 2))
axarr[0, 1].plot(axarr[0,1].get_xlim(), (0,0), ls="--", c=".3", linewidth=2)
axarr[0, 1].set_xlabel("Predicted")
axarr[0, 1].set_ylabel("Residual")

axarr[1, 0].scatter(testY_plsr_pred, testY)
axarr[1, 0].set(xlim=(-11, 2), ylim=(-11, 2))
axarr[1, 0].plot(axarr[0,0].get_xlim(), axarr[1,0].get_ylim(), ls="--", c=".3", linewidth=2)
axarr[1, 0].set_xlabel("Predicted")
axarr[1, 0].set_ylabel("Observed")

axarr[1, 1].scatter(testY_plsr_pred, testY-testY_plsr_pred)
axarr[1, 1].set(xlim=(-11, 2))
axarr[1, 1].plot(axarr[1,1].get_xlim(), (0,0), ls="--", c=".3", linewidth=2)
axarr[1, 1].set_xlabel("Predicted")
axarr[1, 1].set_ylabel("Residual")
```

The predictive ability of each method is good, and the residuals appear to be randomly scattered about zero. Although the predictive ability of these models is close, PLS finds a simpler model that uses far fewer components than PCR.

## 6.4 Penalized Models

Under standard assumptions, the coefficients produced by ordinary least squares regression are unbiased and, of all unbiased linear techniques, this model also has the lowest variance. However, given that the MSE is a combination of variane and bias, it is very possible to produce models with smaller MSEs by allowing the parameter estimates to be biased. Combatting collinearity by using biased models may result in regression models where the overall MSE is competitive.

One method of creating biased regression models is to add a penalty to the sum of the squared errors. Recall that original least squares regression found parameter estimates to minimize the sum of the squared errors: $$\text{SSE} = \sum_{i=1}^n (y_i - \hat{y}_i)^2.$$ When the model over-fits the data, or when there are issues with collinearity, the linear regression parameter estimates may become inflated. Controlling (or *regularizing*) the parameter estimates can be accomplished by adding a penalty to the SSE if the estimates become large.

*Ridge regression* adds a second-order penalty on the SSE: $$\text{SSE}_{L_2} = \sum_{i=1}^n (y_i - \hat{y}_i)^2 + \lambda \sum_{j=1}^P \beta_j^2.$$ In effect, this method *shrinks* the estimates towards 0 as the $\lambda$ penalty become large.

By adding the penalty, we are making a trade-off between the model variance and bias. By sacrificing some bias, we can often reduce the variance enough to make the overall MSE lower than unbiased models.

```python
# path of the ridge regression coefficients
from sklearn.linear_model import Ridge

lambda_range = np.linspace(0.0, 0.01, 600)
ridgeModel = Ridge(normalize=True)
coeff_path = []

for a in lambda_range:
    ridgeModel.set_params(alpha = a)
    ridgeModel.fit(trainX_trans, trainY)
    coeff_path.append(ridgeModel.coef_[0])

ax = plt.gca()
ax.plot(lambda_range, coeff_path)
ax.set_xlim((-0.0005, 0.0105))
plt.xlabel("Penalty")
plt.ylabel("Standardized Coefficient")
```

When there is no penalty, many parameters have reasonable values, but some parameter estimates are abnormally large. These large values are indicative of collinearity issues. As the penalty is increased, the paramter estimates move closer to 0 at diferent rate.

Use cross-validation, the penalty value was optimized (over-fit -> under-fit).

```python
# penalty value versus RMSE
lambda_range = np.linspace(0.0, 0.1, 15)
score_rmse = np.zeros(15)

ridgeModel = Ridge(normalize=True)

for adx, a in enumerate(lambda_range):
    temp_scores = cross_val_score(ridgeModel.set_params(alpha = a),
                                  trainX_trans, trainY.values,
                                  cv=10, scoring="mean_squared_error")
    temp_scores = np.sqrt(-1.0 * temp_scores)
    score_rmse[adx] = np.mean(temp_scores)

plt.plot(lambda_range, score_rmse, '-o')
plt.xlabel("Penalty")
plt.ylabel("RMSE (Cross-Validation)")
plt.xlim((-0.005, 0.105))
```

While ridge regression shrinks the parameter estimates towards 0, the model does not set the values to absolute 0 for any value of the penalty. Even though some parameter estimates become negligibly small, this model does not conduct *feature selection*.

A popular alternative to ridge regression is the *least absolute shrinkage and selection operator* model, known as the *lasso*. It uses a similar penalty to ridge regression $$\text{SSE}_{L_1} = \sum_{i=1}^n (y_i - \hat{y}_i)^2 + \lambda \sum_{j=1}^P |\beta_j|.$$ While this may seem like a small modification, the practical implications are significant. While the regression coefficients are still shrunk towards 0, a consequence of penalizing the absolute values is that some parametes are actually set to 0 for some values of $\lambda$. Thus the lasso yields models that simultaneously use regularization to improve the model and to conduct feature selection.

```python
# path of the lasso regression coefficients
from sklearn.linear_model import lars_path

# compute regularization path using the LARS
lambdas, _, coefs = lars_path(trainX_trans.values, trainY.values[:,0], method = 'lasso', verbose=True)

xx = np.sum(np.abs(coefs.T), axis=1)
xx /= xx[-1]

plt.plot(xx, coefs.T)
plt.xlabel('Fraction of Full Solution')
plt.ylabel('Standardized Coefficient')
```

The x-axis is the fraction of the full solution (i.e., ordinary least squares with no penalty). Smaller values on the x-axis indicate that a large penalty has been used. When the penalty is large, many of the regression coefficients are set to 0.

A significant advancement for this model, the least angle regression (LARS), is a broad framework that encompassess the lasso and similar models. The LARS model can be used to fit lasso models more efficiently, especially in high-dimensional problems.

A generalization of the lasso model is the *elastic net*. The model combines te two types of penalties: $$\text{SSE}_{Enet} = \sum_{i=1}^n (y_i - \hat{y}_i)^2 + \lambda_1 \sum_{j=1}^P |\beta_j| + \lambda_2 \sum_{j=1}^P \beta_j^2.$$ The advantage of this model is that it enables effective regularization via the ridge-type penalty with the feature selection quality of the lasso penalty. Both the penalties require tuning to achieve optimal performance.


# Latent Semantic Indexing

Here, we apply the technique *Latent Semantic Indexing* to capture the similarity of words. We are given a list of words and their frequencies in 9 documents, found on [GitHub](https://github.com/ppham27/MLaPP-solutions/blob/master/chap12/lsiDocuments.pdf).

```python
%matplotlib inline

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn import preprocessing

plt.rcParams['font.size'] = 16

words_list = list()
with open('lsiWords.txt') as f:
    for line in f:
        words_list.append(line.strip())
words = pd.Series(words_list, name="words")
word_frequencies = pd.read_csv('lsiMatrix.txt', sep=' ', index_col=False,
                               header=None, names=words)
word_frequencies.T.head(20)
```

      0
      1
      2
      3
      4
      5
      6
      7
      8

      000
      2
      0
      4
      0
      0
      0
      0
      0
      0

      100
      0
      0
      2
      0
      0
      0
      0
      0
      0

      1913
      0
      0
      0
      3
      0
      0
      0
      0
      0

      1977
      0
      0
      0
      0
      0
      0
      2
      0
      0

      2001
      0
      0
      0
      0
      0
      0
      0
      2
      0

      4
      0
      0
      2
      0
      0
      0
      0
      0
      0

      5
      0
      0
      2
      0
      0
      0
      0
      0
      0

      500
      2
      0
      0
      0
      0
      0
      0
      0
      0

      "
      0
      27
      13
      23
      19
      48
      56
      35
      22

      (
      4
      2
      3
      7
      4
      0
      5
      8
      4

      )
      4
      2
      3
      6
      4
      0
      5
      8
      4

      *
      2
      0
      0
      0
      0
      0
      0
      0
      2

      ,
      34
      30
      27
      83
      26
      74
      39
      49
      38

      -
      6
      0
      12
      2
      20
      5
      15
      16
      7

      .
      38
      41
      39
      41
      25
      36
      30
      31
      38

      :
      0
      0
      4
      0
      3
      0
      3
      3
      2

      ;
      0
      2
      0
      0
      3
      0
      3
      3
      0

      ?
      0
      4
      7
      0
      0
      3
      0
      0
      4

      [
      0
      0
      0
      0
      0
      2
      0
      0
      0

      ]
      0
      0
      0
      0
      0
      2
      0
      0
      0

Now as per part (a), we compute the SVD and use the first two singular values. Recall the model is that

\begin{equation}
\mathbf{x} \sim \mathcal{N}\left(W\mathbf{z},\Psi\right),
\end{equation}

where $\Psi$ is diagonal. If the SVD is $X = UDV^\intercal,$ $W$ will be the first two columns of $V$.

```python
X = word_frequencies.as_matrix().astype(np.float64)
U, D, V = np.linalg.svd(X.T)  # in matlab the matrix is read in as its transpose
```

In this way, we let $Z = UD$, so $X = ZV^\intercal$. Now, let $\tilde{Z}$ be the approximation from using 2 singular values, so $\tilde{X} = \tilde{Z}W^\intercal$, so $\tilde{Z} = \tilde{U}\tilde{D}$. For some reason, the textbook chooses not to scale by $\tilde{D}$, so we just have $\tilde{U}$. Recall that all the variables are messed up because we used the tranpose.

```python
Z = V.T[:,:2]
Z
```

    array([[-0.29579069, -0.51925267],
           [-0.3133317 , -0.33578154],
           [-0.27364718, -0.45865789],
           [-0.34873574,  0.47093077],
           [-0.22977276,  0.04358919],
           [-0.44143778,  0.28243982],
           [-0.3498604 ,  0.22347862],
           [-0.35250449,  0.17379354],
           [-0.35143587, -0.1538357 ]])

Now, let's plot these results.

```python
plt.figure(figsize=(8,8))
def plot_latent_variables(Z, ax=None):
    if ax == None:
        ax = plt.gca()
    ax.plot(Z[:,0], Z[:,1], 'o', markerfacecolor='none')
    for i in range(len(Z)):
        ax.text(Z[i,0] + 0.005, Z[i,1], i,
                verticalalignment='center')
    ax.set_xlabel('$z_1$')
    ax.set_ylabel('$z_2$')
    ax.set_title('PCA with $L = 2$ for Alien Documents')
    ax.grid(True)
plot_latent_variables(Z)
plt.show()
```

I, respectfully, disagree with the book for this reason. The optimal latent representation $Z = XW$ (observations are rows here), should be chosen such that

\begin{equation}
J(W,Z) = \frac{1}{N}\left\lVert X - ZW^\intercal\right\rVert^2
\end{equation}

is minimized, where $W$ is orthonormal.

```python
U, D, V = np.linalg.svd(X)
V = V.T # python implementation of SVD factors X = UDV (note that V is not tranposed)
```

By section 12.2.3 of the book, $W$ is the first $2$ columns of $V$. Thus, our actual plot should be below.

```python
W = V[:,:2]
Z = np.dot(X, W)
plt.figure(figsize=(8,8))
ax = plt.gca();
plot_latent_variables(Z, ax=ax)
ax.set_aspect('equal')
plt.show()
```

Note that this is very similar with the $y$-axis flipped. That part does not actually matter. What matters is the scaling by eigenvalues for computing. Before that scaling the proximity of points may not mean much if the eigenvalue is actually very large.

Now, the second part asks us to see if we can properly identify documents related to abductions by using a document with the single word *abducted* as a probe.

```python
probe_document = np.zeros_like(words, dtype=np.float64)
abducted_idx = (words=='abducted').as_matrix()
probe_document[abducted_idx] = 1
X[0:3,abducted_idx]
```

    array([[ 0.],
           [ 6.],
           [ 3.]])

Note that despite the first document being about abductions, it doesn't contain the word *abducted*.

Let's look at the latent variable representation. We'll use cosine similarity to account for the difference in magnitude.

```python
from scipy.spatial import distance
z = np.dot(probe_document, W)
similarities = list(map(lambda i : (i, 1 - distance.cosine(z,Z[i,:])), range(len(Z))))
similarities.sort(key=lambda similarity_tuple : -similarity_tuple[1])
similarities
```

    [(0, 0.59127529086736064),
     (2, 0.57730256865377172),
     (1, 0.45835648369851389),
     (8, 0.31235652085315346),
     (4, 0.15438034299529768),
     (7, 0.076686002588720159),
     (6, 0.039651047230294334),
     (5, 0.039384780001176511),
     (3, -0.13308361596893792)]

Indeed, we find the three alien abduction documents, $0$, $2$, and $1$ are most similar to our probe.


# Unitary Learning with qgrad

In this example, we shall try to learn an abitrary $8 \times 8$
unitary matrix $U$, via gradient descent. We shall start with a
random parametrized unitary matrix $U(\vec{\theta}, \vec{\phi}, \vec{\omega})$.

Parametrized unitaries in `qgrad` are available via `Unitary` class
with  $\vec{\theta}, \vec{\phi}, \vec{\omega}$ as parameter
vectors with $\vec{\theta}, \vec{\phi}$ being $\frac{(N) (N-1)}{2}$
dimensional and $\vec{\omega}$ being $N$-dimensional.

Here the input dataset consists of $8 \times 1$ random
kets, call them
$| \psi_{i} \rangle$ and output dataset is the action of the
target unitary $U$ on these kets, $U |\psi_{i} \rangle$. The
maximum value of $i$ is $80$, meaning that we merely use 80
data points (kets in this case) to efficiently learn the
target unitary, $U$.

This tutorial is different from the
[Qubit Rotation](https://github.com/qgrad/qgrad/blob/master/examples/QubitRotation.py),
in that it learns the unitary matrix to not only take a fixed _specific_
state to another _fixed_ state. Here the unitary
$U( \vec{\theta}, \vec{\phi}, \vec{\omega})$ is learnt to evolve _any_ same
dimensional ket as the target unitary, $U$ would evolve it.

**Note**: Another version of this tutorial is implemented
without `qgrad` that uses the parametrization used in
[Seth Lloyd and Reevu Maity, 2020](https://arxiv.org/pdf/1901.03431.pdf)
and reproduces part of the results of that paper. This tutorial
shows similar results, only with different unitary paramterization
$U(\vec{\theta}, \vec{\phi}, \vec{\omega})$  since the
parametrization used in the original paper
uses hamiltonians
in the powers of exponents, whose autodifferentiation is
not currently supported in JAX. For further reading
on this autodifferentiation incompatibility and unitary learning,
please refer to companion blogs
[here](https://araza6.github.io/posts/hamiltonian-differentiation/)
and [here](https://araza6.github.io/posts/unitary-learning/)

```python
import jax.numpy as jnp
from jax import grad
from jax.experimental import optimizers
from jax.random import PRNGKey, uniform

import numpy as onp

#Visualization
import matplotlib.pyplot as plt

from qgrad.qgrad_qutip import fidelity, Unitary

from qutip import rand_ket # only to make the dataset

from scipy.stats import unitary_group
```

```python
def make_dataset(m, d):
    """Prepares a dataset of input and output
    kets to be used for training.

    Args:
        m (int): Number of data points, 80% of
            which would be used for training
        d (int): Dimension of a (square) unitary
            matrix to be approximated

    Returns:
        tuple: tuple of lists containing (JAX Device
            Arrays of) input and output kets
            respectively
    """
    ket_input = []
    ket_output = []
    for i in range(m):
        ket_input.append(jnp.array(rand_ket(d, seed=300).full()))
        #Output data -- action of unitary on a ket states
        ket_output.append(jnp.dot(tar_unitr, ket_input[i]))

    return (ket_input, ket_output)

m = 100 # number of training data points
N = 8 # Dimension of the unitary to be learnt
train_len = int(m * 0.8)
# tar_unitr gives a different unitary each time
tar_unitr = jnp.asarray(unitary_group.rvs(N))
ket_input, ket_output = make_dataset(m, N)
```

## Cost Function

We use the same cost function as the authors
[Seth Lloyd and Reevu Maity, 2020](https://arxiv.org/pdf/1901.03431.pdf)
define

\begin{equation}
E = 1 - (\frac{1}{M})\sum_{i} \langle \psi_{i}|U^{\dagger} U(\vec{\theta}, \vec{\phi}, \vec{\omega})|\psi_{i}\rangle
\end{equation}

where $|\psi_{i}\rangle$ is the training (or testing)
data points -- in this case, kets, $U$ and
$U(\vec{\theta}, \vec{\phi}, \vec{\omega})$ are the target and
parameterized unitaries respectively and $M$ is the total
number of training data points, which in our example is
$80$

```python
def cost(params, inputs, outputs):
    r"""Calculates the cost on the whole
        training dataset.

    Args:
        params (obj:`jnp.ndarray`): parameter vectors
            :math:`\vec{\theta}, \vec{\phi},
            \vec{\omega}`
        inputs (obj:`jnp.ndarray`): input kets
            :math:`|\psi_{i} \rangle`in the dataset
        outputs (obj:`jnp.ndarray`): output kets
            :math:`U(\vec{\theta}, \vec{\phi},
            \vec{\omega})|ket_{input} \rangle`
            in the dataset

    Returns:
        float: cost (evaluated on the entire dataset)
            of parametrizing :math:`U(\vec{\theta},
            \vec{\phi}, \vec{\omega})` with `params`
    """
    loss = 0.0
    thetas, phis, omegas = params
    unitary = Unitary(N)(thetas, phis, omegas)
    for k in range(train_len):
        pred = jnp.dot(unitary, inputs[k])
        loss += jnp.absolute(jnp.real(jnp.dot(outputs[k].conjugate().T, pred)))

    loss = 1 - (1 / train_len) * loss
    return loss[0][0]
```

## Performance Metric -- Fidelity

While cost is a valid metric to judge the learnability.
We introduce another commonly used metric, the _average_
fidelity between the predicted and the output (label)
states, as another metric to track during training. Average
fidelity over the dataset over a particular set of
parameters is defined as:

\begin{equation}
F_{avg} = \frac{1}{M}\sum_{i}| \langle \psi_{in} | \psi_{pred} \rangle |^2
\end{equation}

where $\psi_{label}$
represents the resulting (or the output)
ket evolved under the target unitary,
$U$ as $U|\psi_{i}\rangle$ and
$\psi_{pred}$ represents the ket $\psi_{i}$
evolved under
$U(\vec{\theta}, \vec{\phi}, \vec{\omega})$
as $U(\vec{\theta}, \vec{\phi}, \vec{\omega})|\psi_{i}\rangle$.

```python
def test_score(params, inputs, outputs):
    """Calculates the average fidelity between the
        predicted and output kets for given parameters
        (averaged over the whole training set).

       Args:
           params (obj:`jnp.ndarray`): parameter vectors
               :math:`\vec{\theta}, \vec{\phi}, \vec{\omega}`
           inputs (obj:`jnp.ndarray`): input kets
               :math:`|\psi_{l} \rangle`in the dataset
           outputs (obj:`jnp.ndarray`): output kets
               :math:`U(\vec{t}, \vec{\tau})|ket_{input} \rangle`
               in the dataset

       Returns:
           float: fidelity between :math:`U(\vec{\theta},
               \vec{\phi}, \vec{\omega})|ket_{input} \rangle`
               and the output (label) kets for given `params`

       """
    fidel = 0
    thetas, phis, omegas = params
    unitary = Unitary(N)(thetas, phis, omegas)
    for i in range(train_len):
        pred = jnp.dot(unitary, inputs[i])
        step_fidel = fidelity(pred, outputs[i])
        fidel += step_fidel

    return (fidel / train_len)[0][0]
```

```python
# Fixed PRNGKeys to pick the same starting params
params = uniform(PRNGKey(0), (N**2, ),
                        minval=0.0, maxval=2 * jnp.pi)
thetas = params[:N * (N-1) // 2]
phis = params[N * (N - 1) // 2 : N * (N - 1)]
omegas = params[N * (N - 1):]
params = [thetas, phis, omegas]

opt_init, opt_update, get_params = optimizers.adam(step_size=1e-1)
opt_state = opt_init(params)

def step(i, opt_state, opt_update):
    params = get_params(opt_state)
    g = grad(cost)(params, ket_input, ket_output)
    return opt_update(i, g, opt_state)

epochs = 40

loss_hist = []
params_hist = []
fidel_hist = []

for i in range(epochs):
    opt_state = step(i, opt_state, opt_update)
    params = get_params(opt_state)
    params_hist.append(params)
    loss = cost(params, ket_input, ket_output)
    loss_hist.append(loss)
    avg_fidel = test_score(params, ket_input, ket_output)
    fidel_hist.append(avg_fidel)
    progress = [i+1, loss, avg_fidel]
    if (i % 10 == 9):
        print("Epoch: {:2f} | Loss: {:3f} | Fidelity: {:3f}".
              format(*jnp.asarray(progress)))
```

    Epoch: 10.000000 | Loss: 0.061839 | Fidelity: 0.888217
    Epoch: 20.000000 | Loss: 0.014260 | Fidelity: 0.975306
    Epoch: 30.000000 | Loss: 0.006706 | Fidelity: 0.986648
    Epoch: 40.000000 | Loss: 0.002017 | Fidelity: 0.995984

## Analyzing Learning Routine

We see that we _efficiently_ (with $ \sim 99 \%$
fidelity) reconstruct the target unitary $U$ starting
from a random initial guess. We merely use $80$
examples for training, and in $40$ crude gradient
steps, we almost perfectly approximate the target
unitary, $U$. Below is a plot of how fidelity increases
and loss gets down to zero in the training schedule.

In the graph below, $M$ represents
the total size of the train set, $\psi_{label}$
represents the resulting (or the output)
ket evolved under the target unitary,
$U$ as $U |\psi_{i} \rangle$ and
$\psi_{pred}$ represents the ket $\psi_{i}$
evolved under
$U(\vec{\theta}, \vec{\phi}, \vec{\omega})$
as $U(\vec{\theta}, \vec{\phi}, \vec{\omega})|\psi_{i}\rangle$.

Each marker on the graph is the fidelity between
the predicted and the target/label kets averaged over the
whole train set and the cost on whole training set
respectively.

```python
plt.figure(figsize=(9, 6))

plt.plot(fidel_hist, marker = 'o',
         label=r"$F_{avg} = \frac{1}{M}\sum_{i}| \langle \psi_{label} | \psi_{pred} \rangle |^2$")

plt.plot(loss_hist, marker = 'x',
        label=r'''$L = 1 - (\frac{1}{M})\sum_{i}\langle \psi_{i} | U ^{\dagger} U( \vec{\theta}, \vec{\phi}, \vec{\omega)} | \psi_{i} \rangle$''')

plt.title("Fidelity and Cost Trends", fontweight = "bold")
plt.legend(["Fidelity","Loss"])
plt.xlabel("epoch")
plt.legend(loc=0, prop = {'size': 15})
```

## Testing on unseen kets

We reserved the last $20$ (which is $20 \%$ of the total dataset)
kets for testing.
Now we shall apply our learned unitary matrix, call it
$U_{opt}(\vec{\theta}, \vec{\phi}, \vec{\omega})$
to the unseen kets and measure the fidelity of the evolved ket
under $U_{opt}(\vec{\theta}, \vec{\phi}, \vec{\omega})$
with those that evolved under the target unitary, $U$.

```python
theta_opt, phi_opt, omega_opt = params_hist[-1]
opt_unitary = Unitary(N)(theta_opt, phi_opt, omega_opt)
fidel = []
for i in range(train_len, m): # unseen data
    pred = jnp.dot(opt_unitary, ket_input[i])
    fidel.append(fidelity(pred, ket_output[i])[0][0])
fidel
```

    [DeviceArray(0.98841643, dtype=float32),
     DeviceArray(0.98841643, dtype=float32),
     DeviceArray(0.98841643, dtype=float32),
     DeviceArray(0.98841643, dtype=float32),
     DeviceArray(0.98841643, dtype=float32),
     DeviceArray(0.98841643, dtype=float32),
     DeviceArray(0.98841643, dtype=float32),
     DeviceArray(0.98841643, dtype=float32),
     DeviceArray(0.98841643, dtype=float32),
     DeviceArray(0.98841643, dtype=float32),
     DeviceArray(0.98841643, dtype=float32),
     DeviceArray(0.98841643, dtype=float32),
     DeviceArray(0.98841643, dtype=float32),
     DeviceArray(0.98841643, dtype=float32),
     DeviceArray(0.98841643, dtype=float32),
     DeviceArray(0.98841643, dtype=float32),
     DeviceArray(0.98841643, dtype=float32),
     DeviceArray(0.98841643, dtype=float32),
     DeviceArray(0.98841643, dtype=float32),
     DeviceArray(0.98841643, dtype=float32)]

## Conclusion

We see that the testing fidelity is
$\sim 98 \%$, as opposed to training
fidelity $\sim 99 \%$. One would expect
this since drop as the unitary now
acts on unseen data. We, however, note
that we generalize well with
$\sim 98 \%$ accuracy, if you will.

This learnt unitary
$U_{opt}(\vec{\theta}, \vec{\phi}, \vec{\omega})$
can now be used to emulate the original
target unitary, $U$, for more general
datasets as well.

## References
1. Lloyd, Seth, and Reevu Maity. "Efficient implementation of unitary transformations." arXiv preprint arXiv:1901.03431 (2019).


```python
from sympy import *
from sympy.plotting import plot3d
init_printing(use_unicode = True)
```

```python
x,y = symbols('x y')
f = x**3 - x + y**3 - y
i = .5
j = 10
```

```python
x0 = -1/(3**.5)
y0 = 1/(3**.5)
```

```python
fx = diff(f,x).subs(x,x0).subs(y,y0)
fy = diff(f,y).subs(x,x0).subs(y,y0)
fxx = diff(f,x,2).subs(x,x0).subs(y,y0)
fxy = diff(f,x,y).subs(x,x0).subs(y,y0)
fyy = diff(f,y,2).subs(x,x0).subs(y,y0)
H = Matrix([[fxx,fxy],[fxy,fyy]])
grad_f = Matrix([[fx,fy]])
dX = Matrix([x-x0,y-y0])
Q = 0.5*dX.T*H*dX
L = grad_f*dX
F = f.subs(x,x0).subs(y,y0) + Q.det() + L.det()
expand(F)
```

```python
c1=plot3d(f,(x,x0-j,x0+j),(y,y0-j,y0+j))
f
```

```python
c2=plot3d(F,(x,x0-j,x0+j),(y,y0-j,y0+j))
expand(F)
```

```python
c1.extend(c2)
c1.show()
print("together")
```

```python
plot3d(F,(x,x0-i,x0+i),(y,y0-i,y0+i))
print(f"approx at x,y = {x0},{y0}")
```

```python
plot3d(f,(x,x0-i,x0+i),(y,y0-i,y0+i))
print("original")
```

```python
H
```

$\displaystyle \left[\begin{matrix}-3.46410161513776 & 0\\0 & 3.46410161513776\end{matrix}\right]$

```python
val = H.eigenvals()
val
```

```python
21650635094611/6250000000000
```


# Final Project: Method of fokas and its implementation for solving linear partial differential equations

>**AMATH 581 Raphael Liu**  raph651@uw.edu

## Background of Research

The research interest is originated from the amath course 567: *applied complex analysis*, taught by Prof. Bernard Deconinck. I'm taking this course this autumn quarter. The main concept of the course is to introduce complex variables and its applications. Contour Integration, Cauchy's theorem, Riemann surface, etc,. are covered in the course.

This project's topic, method of fokas, is introduced at the end of the quarter but only the beginning of it is covered in class. In this notebook, I'll introduce and go through the procedure of fokas's method, and develop an implementation script that solves an example of linear PDEs system. More can be learned in Prof. Deconinck's paper here. {https://depts.washington.edu/bdecon/papers/pdfs/trogdon_deconinck_linear_pdes.pdf}

Classically, given a boundary value problem for linear PDEs with constant coefficient, we rely on separation of variables and specific integral transforms to find the explicit solution. The standard methods are a in collection of situation-wise approaches targeting specific boundary equations and boundary conditions. These approaches involve separation of variables, Fourier seires and transforms, Laplace's transforms, Sturm–Liouville theory, and other integral transforms. It becomes difficult, however, for the standard methods to solve higher order PDEs. On the other hand, the method of Fokas gives the equivalent solution as the standard methods provide, but the generality surpasses.

## Method of Fokas
### The Local Relation

 Consider the one-dimensional heat equation with advection: $u_t = \gamma u_{xx}+cu_x $, for $x > 0$. Here $\gamma > 0$ and $c \neq 0$ are real parameters

We wish to rewrite the equation in divergence forms for $x$ and $t$ independent on the two sides, specifically:

$$(e^{-ikx+Wt}u)_t = J_x\label{eq:2.1}$$

where $W(k)$ is only $k$-dependent, $J(x,t)$ is $x$ and $t$ dependent.

To find $W(k)$ and $J(x,t)$, we note the chain rule on the left side,

\begin{align}
\label{}
(e^{-ikx+Wt}u)_t &= We^{-ikx+Wt}u + e^{-ikx+Wt}u_t\\
&=We^{-ikx+Wt}u + e^{-ikx+Wt}(\gamma u_{xx}+cu_x)
\end{align}

substituting inverse chain rule, we have

\begin{align}
(e^{-ikx+Wt}u)_t &=We^{-ikx+Wt}u + (e^{-ikx+Wt}\gamma u_{x})_x+ike^{-ikx+Wt}\gamma u_x+(e^{-ikx+Wt}cu)_x+ike^{-ikx+Wt}cu\\
&=We^{-ikx+Wt}u + (e^{-ikx+Wt}\gamma u_{x})_x+ik[(e^{-ikx+Wt}\gamma u)_x+ike^{-ikx+Wt}\gamma u]+(e^{-ikx+Wt}cu)_x+ike^{-ikx+Wt}cu\\
&=(W-\gamma k^2+ikc)e^{-ikx+Wt}u+(e^{-ikx+Wt}(\gamma u_x+ik\gamma u+cu))_x
\end{align}

let  $\;\;$  $W-\gamma k^2+ikc =0,\label{eq:2.2}$

we are allowed to obtain the divergence forms for $x$ and $t$ on the two sides of equation:

\begin{equation}\label{eq:2.3}
(e^{-ikx+Wt}u)_t=(e^{-ikx+Wt}(\gamma u_x+ik\gamma u+cu))_x
\end{equation}

This is the **Local Relation**, which is equivalent to the original equation, but contains a free variable $k$. The Local Relation holds for all values for $k$, including complex ones. Referring to the quantity $W(k)$ as **the symbol of the operator** $L=\gamma \partial^2 x+c\partial x$. So we arrive at $u_t=Lu=\gamma u_{xx} +cu_x$.

 In a simpler case of heat equation, $u_t=u_{xx}$ , $W(k)$ is obtained by replacing each $\partial x \to ik$, and the result $W=k^2$ is familiar to us.

### Half Line Scheme

We now focus on the same heat equation with advection on the half line, says $x,t>0$, with given initial condition $u_0(x)$ and Dirichlet boundary condition $f_0(t)$:

\begin{align}\label{}
u_t&=\gamma u_{xx}+cu_x, \quad\quad\quad &x>0,\;\; t>0,\\
u(x,0)&=u_0(x),  \quad\quad\quad &x>0,\\
u(0,t)&=f_0(t), \quad\quad\quad &t>0.
\end{align}

To fully consider the situation for $x<0$, one might use the *mirror principle*. The equation is invariant under the transform $x\to-x$, and the Fourier sine transform can be utilized to solve for the negative $x$ region, at least with homogeneous boundary conditions. If the boundary condition is non-homogeneous, a two-steps process is required. First, solve for the general solution to homogeneous boundary conditions. Second, solve for the particular solution to non-homogeneous ones using variation of parameters.

>Figure 2.1: Integral along the contour in region $\mathcal{D}$ where the vertical dashed line indicates $x=\infty$

### The Global Relation

Integrating the Local Relation over the region $\mathcal{D}$ (see Fig. 2.1), we have

\begin{equation}\label{}
\iint_\mathcal{D}[(e^{-ikx+Wt}u)_t-(e^{-ikx+Wt}(\gamma u_x+ik\gamma u+cu))_x]dxdt=0\\
\end{equation}
Appling Green's Theorem,

\begin{equation}\label{eq:2.4}
\oint_{\partial \mathcal{D}}e^{-ikx+Wt}(\gamma u_x+ik\gamma u+cu)dt+e^{-ikx+Wt}udx=0
\end{equation}

Since $\partial \mathcal{D}$ consists of 4 line segments, we examine the contribution from each line segment.

(I) On the positive real axis, $t=0$ holds, thus $dt=0$ and $x$ goes from $0$ to $\infty$. The contribution reduces to

\begin{equation}\label{}
I = \int_0^\infty e^{-ikx}u(x,0)dx =\int_0^\infty e^{-ikx}u_0(x)dx
\end{equation}

(II) On the dashed line *II*, assuming $u$ and its derivatives vanish as $x\to \infty$,

\begin{equation}\label{}
II=0
\end{equation}

(III) On line *III*, $t=T$, $dt=0$. $x$ goes from $\infty \to 0$. The contribution is

\begin{equation}\label{}
III=\int_\infty^0e^{-ikx+WT}u(x,T)dx=-e^{WT}\int_0^\infty e^{-ikx}u(x,T)dx
\end{equation}

(IV) On line segment *IV*, $x=0$,$dx=0$ and $t$ goes from $T$ to $0$,

\begin{align}\label{}
IV&=\int_T^0 e^{Wt}(\gamma u_x(0,t)+ik\gamma u(0,t)+cu(0,t))dt\\&=-\int_0^Te^{Wt}(\gamma f_1(t)+(c+ik\gamma)f_0(t))dt\\
\end{align}
for $f_1(t)=u_x(0,t)$, $f_0(t)=u(0,t)$.

Summing up all the contribution, we find

\begin{align}\label{eq:2.5}
0&=I+II+III+IV\\
&=\int_0^\infty e^{-ikx}u_0(x)dx-e^{WT}\int_0^\infty e^{-ikx}u(x,T)dx-\int_0^Te^{Wt}(\gamma f_1(t)+(c+ik\gamma)f_0(t))dt
\end{align}

Here, note that these are the Fourier series transform, defining:

\begin{align}\label{}
\hat{u}_0(k)&=\int_0^\infty e^{-ikx}u_0(x)dx,\\\hat{u}(k,T)&=\int_0^\infty e^{-ikx}u(x,T)dx.
\end{align}

Consequently, we have the inverse Fourier transform,

\begin{align}\label{}
u_0(x)&=\frac{1}{2\pi}\int_{-\infty}^\infty e^{ikx}\hat{u}_0(k)dk,\\
u(x,T)&=\frac{1}{2\pi}\int_{-\infty}^\infty e^{ikx}\hat{u}(k,T)dk
\end{align}

Next, let

\begin{align}\label{eq:2.6}
F_j(W,T)=\int_0^T e^{Wt}u_{jx}(0,t)dt,
\end{align}

so that for j=0 and j=1,

\begin{equation}\label{}
F_0(W,T)=\int_0^T e^{Wt}f_0(t)dt, \quad\quad F1(W,T)=\int_0^Te^{Wt}f_1(t)dt
\end{equation}

These definations allow us to rewrite (eq. 2.5) as

\begin{equation}\label{eq:2.7}
0=\hat{u}_0(k)-e^{WT}\hat{u}(k,T)-(\gamma F_1(W,T)+(c+ik\gamma)F_0(W,T))
\end{equation}

This is the **Global Relation** for the heat equation on the positive half line. It links the solution at time $T>0$ to the initial and boundary conditions.

To determine where in the complex $k$-plane the Global Relation is valid, other than $k\in \mathbb{R}$, note that

$$k=k_R+ik_I, \quad\quad k_R,k_I \neq 0$$

\begin{equation}\label{}
\hat{u}_0(k)=\int_0^\infty e^{-ikx}u_0(x)dx=\int_0^\infty e^{-ik_Rx}e^{k_Ix}u_0(x)dx
\end{equation}

The first exponential term leads to oscillations and is harmless. The second exponential should be decaying as $x\to \infty$. Otherwise the integrand has arbitrarily fast exponential growth. Since $x>0$, this requires $k_I<0$. For $k_I=0$, the integral is still defined and $\hat{u}_0(k)$ is analytic. Thus,  the Global Relation is valid in the lower half $k$ plane, $k_I\leq 0$.

### The "solution" formula

From the Global Relation (eq. 2.7),

\begin{align}\label{}
e^{WT}\hat{u}(k,T)&=\hat{u}_0(k)-(\gamma F_1(W,T)+(c+ik\gamma)F_0(W,T))\\
\Rightarrow \quad\quad\quad\quad \hat{u}(k,T)&=e^{-WT}\hat{u}_0(k)-e^{-WT}(\gamma F_1(W,T)+(c+ik\gamma)F_0(W,T))\\
\Rightarrow \quad\quad\quad\quad u(x,T)&=\mathcal{F}^{-1}[\hat{u}(k,T)](x)\\
&=\frac{1}{2\pi}\int_{-\infty}^\infty e^{ikx}[e^{-WT}\hat{u}_0(k)-e^{-WT}(\gamma F_1(W,T)+(c+ik\gamma)F_0(W,T))]dk\\
&=\frac{1}{2\pi}\int_{-\infty}^\infty e^{ikx-WT}\hat{u}_0(k)dk-\frac{1}{2\pi}\int_{-\infty}^\infty e^{ikx-WT}(\gamma F_1(W,T)+(c+ik\gamma)F_0(W,T))dk\label{eq: 2.8)}
\end{align}

This seems like a fine expression for the solution. However, it is also useless because it expresses the solution in terms of the initial condition $u_0(x)$ and the boundary conditions $u(0,t)$ and $u_x(0,t)$. Although the first of the boundary conditions is known, the second is not. Thus we still lack a bit information, to avoid this we consider the next section.

### Deforming the path of intergration

We proceed to deform the current path of integration, the real line, to region far away. Since we do not know the Neumann boundary condition $f_1(t)$, the $F_1(W,T)$ term is unknown. Our goal is to eliminate the dependence on this Neumann boundary condition.

Let us then get back to the "solution" formula (eq. 2.8), and rewrite the integral of $F_1$ term as:

\begin{align}\label{}
\frac{1}{2\pi}\int_{-\infty}^\infty e^{ikx-WT} F_1(W,T)dk&=\frac{1}{2\pi}\int_{-\infty}^\infty e^{ikx-WT}\int_0^T e^{Wt}f_1(t)dtdk\\
&=\frac{1}{2\pi}\int_{-\infty}^\infty \int_0^Te^{ikx-W(T-t)}f_1(t)dtdk\\
&=\frac{1}{2\pi}\int_0^T(\int_{-\infty}^\infty e^{i(k_R+ik_I)x-(W_R+iW_I)(T-t)}f_1(t)dk)dt
\end{align}

where we denote \quad $W =W_R+iW_I $ and  $k=k_R+ik_I$.

Therefore, in order for the inside integral to be defined, the exponential term needs to be bounded. Based on that the $x$ and $T$ are independent variables, both exponentials need to be controlled separately. For the $x$ exponential term $e^{i(k_R+ik_I)x}$, we require

$$k_I\ge 0,$$

The second exponential term involving $W(k)$ requires that $e^{-(W_R+iW_I)(T-t)}$ is bounded. Since $T-t>0$, we requires $W_R>0$. For (eq. 2.2) gives us,

\begin{align}\label{}
W(k)=\gamma k^2-ikc&=\gamma(k_R+ik_I)^2-i(k_R+ik_I)c\\
&=\gamma(k_R^2-k_I^2)+ck_I+i(2\gamma k_Rk_I-k_R)
\end{align}

and as

\begin{align}\label{eq:2.9}
W_R>0 \quad\quad \Rightarrow \quad\quad \gamma(k_R^2-k_I^2)+ck_I&>0\\
k_R^2&>k_I^2-\frac{c}{\gamma}k_I=(k_I-\frac{c}{2\gamma})^2-\frac{c^2}{4\gamma^2}
\end{align}

Denote $\mathbb{C^+}$ as the upper half plane and the region

$$\Omega = \{k\in \mathbb{C}: W_R(k)<0\}=\{k\in \mathbb{C}: k_R^2<k_I^2-\frac{c}{\gamma}k_I\}.$$

The area depicted by the inequality is shown in (Figure 2.2) as $\mathbb{C^+}\setminus\Omega$.

>Figure 2.2: Integral along the contours labeled in letters. The shaded region represents $\Omega$. The dashed contours indicate $|k|=\infty$.

Figure 2.2 introduces the contours labeled in letters, where the contour B and B' are assumed to be at $|k|=\infty$. Now consider the second integral term in the "solution" formula (eq. 2.8) which is integrating over the real $k$ line from $-\infty$ to $\infty$.

The contour integrals on the left suggest $\int_A=\int_B+\int_C+\int_D$. Using **Jordan's Lemma**, we know $\int_B \;,\int_{B'}=0$. For the contour integrals on the right, we have $\int_{A'}=\int_{B'}+\int_{C'}-\int_{D'}$. It is then obvious to see  $\int_A+\int_{A'}=\int_C+\int_{C'} \quad \Rightarrow \quad \int_{\mathbb{R}}=\int_{\partial\Omega}.$ Thus, rewrite our "solution" formula:

\begin{equation}\label{eq: 2.10}
u(x,T)  =\frac{1}{2\pi}\int_{-\infty}^\infty e^{ikx-WT}\hat{u}_0(k)dk-\frac{1}{2\pi}\int_{\partial\Omega} e^{ikx-WT}(\gamma F_1(W,T)+(c+ik\gamma)F_0(W,T))dk
\end{equation}

In this way, the original path of integral is deformed as far away from the real line as possible.

### Deploy Symmetries of *W(k)*

So far the solution equation $u(x,T))$ is still trivial, because the dependence on $F_1(W,T)$ is not eliminated. To proceed, we start looking at the $W(k)$ quantity. Consider we find the symmetry of $W(k)$ in the manner of transform $k\to v(k)$ that leaves $W(k)$ invariant:

\begin{equation}\label{}
W(k)=W(v(k)).
\end{equation}

Writing

\begin{align}\label{}
W(k)=\gamma k^2-ikc&=\gamma (k^2-\frac{ic}{\gamma}k+(\frac{ic}{2\gamma}^2)-(\frac{ic}{2\gamma}^2))\\
&=\gamma[(k-\frac{ic}{2\gamma})^2+(\frac{ic}{2\gamma})^2]\\\\
W(v(k))&=\gamma[(v-\frac{ic}{2\gamma})^2+(\frac{ic}{2\gamma})^2].
\end{align}

Let $v(k)=\frac{ic}{\gamma}-k,$

\begin{align}\label{}
\quad \rightarrow \quad W(v(k)) &=\gamma[(\frac{ic}{\gamma}-k-\frac{ic}{2\gamma})^2+(\frac{ic}{2\gamma})^2]\\
&=\gamma[(\frac{ic}{2\gamma}-k)^2+(\frac{ic}{2\gamma})^2]\\
&=W(k).
\end{align}

If we apply it to the Global Relation (eq. 2.7), we find

\begin{align}
0&=\hat{u}_0(\frac{ic}{\gamma}-k)-e^{WT}\hat{u}(\frac{ic}{\gamma}-k,T)-[\gamma F_1(W,T)+(c+i(\frac{ic}{\gamma}-k)\gamma)F_0(W,T)]\\
&=\hat{u}_0(\frac{ic}{\gamma}-k)-e^{WT}\hat{u}(\frac{ic}{\gamma}-k,T)-\gamma F_1(W,T)+ik\gamma F_0(W,T)\label{eq: 2.11}
\end{align}

Solving this new Global Relation gives us

\begin{equation}\label{eq:2.12}
\gamma F_1(W,T)=\hat{u}_0(\frac{ic}{\gamma}-k)-e^{WT}\hat{u}(\frac{ic}{\gamma}-k,T)+ik\gamma F_0(W,T)
\end{equation}

This new Global Relation is valid in condition that $e^{-i(\frac{ic}{\gamma}-k)x}$ is decaying. The exponential terms comes from the inverse Fourier transform of $\hat{u},\hat{u}_0$. Therefore, it follows that

\begin{align}\label{}
e^{-i(\frac{ic}{\gamma}-k)x}=e^{(\frac{ic}{\gamma}+ik_R-k_I)x}=e^{(\frac{c}{\gamma}-k_I)x}e^{ik_Rx}
\end{align}

\begin{align}\label{}
\rightarrow \quad\quad\quad\frac{c}{\gamma}-k_I&\le0\\k_I\ge\frac{c}{\gamma}
\end{align}

Since this expression is valid in the upper half $k$ plane above $k_I=\frac{ic}{\gamma}$, we may substitute it in (eq. 2.10). This results in

\begin{align}\label{}
u(x,T)  &=\frac{1}{2\pi}\int_{-\infty}^\infty e^{ikx-WT}\hat{u}_0(k)dk-\frac{1}{2\pi}\int_{\partial\Omega} e^{ikx-WT}[\hat{u}_0(\frac{ic}{\gamma}-k)-e^{WT}\hat{u}(\frac{ic}{\gamma}-k,T)+ik\gamma F_0(W,T)+(c+ik\gamma)F_0(W,T)]dk\\\\
&=\frac{1}{2\pi}\int_{-\infty}^\infty e^{ikx-WT}\hat{u}_0(k)dk-\frac{1}{2\pi}\int_{\partial\Omega} e^{ikx-WT}\hat{u}_0(\frac{ic}{\gamma}-k)dk+\frac{1}{2\pi}\int_{\partial\Omega}e^{ikx}\hat{u}(\frac{ic}{\gamma}-k,T)dk -\frac{1}{2\pi}\int_{\partial\Omega} e^{ikx-WT}(c+2ik\gamma)F_0(W,T)dk
\end{align}

However, this is not the end. Note that we have an unknown term $\hat{u}(\frac{ic}{\gamma}-k,T)$.

### The solution formula

We again examine the unwanted term $\hat{u}(\frac{ic}{\gamma}-k,T)$. It is unknown, however, we know that it is analytic in the upper half plane above $k_I=\frac{ic}{\gamma}$. Note that the intergral $\frac{1}{2\pi}\int_{\partial\Omega}e^{ikx}\hat{u}(\frac{ic}{\gamma}-k,T)dk$ does not contain the $WT$ term exponential term. As a consequence, the region $\Omega$ is not relevant anymore. We could take integration paths through it, as will.

From Cauchy's Theorem, as depicted in the Figure below,

>Figure 2.3: Integral along the contours labeled in letters. The shaded region represents $\Omega$. The dashed contours indicate $|k|=\infty$.

\begin{equation}\label{}
(\int_A+\int_B+\int_C)e^{ikx}\hat{u}(\frac{ic}{\gamma}-k,T)dk=0 \quad \Rightarrow \quad \int_{\partial\Omega}e^{ikx}\hat{u}(\frac{ic}{\gamma}-k,T)dk=-\int_C
e^{ikx}\hat{u}(\frac{ic}{\gamma}-k,T)dk=0
\end{equation}

by Jordan's Lemma. Thus, the unwanted term is zero and we finally obtain the solution formula without quotation mark:

\begin{equation}\label{eq:2.13}
u(x,T)=\frac{1}{2\pi}\int_{-\infty}^\infty e^{ikx-WT}\hat{u}_0(k)dk-\frac{1}{2\pi}\int_{\partial\Omega} e^{ikx-WT}\hat{u}_0(\frac{ic}{\gamma}-k)dk -\frac{1}{2\pi}\int_{\partial\Omega} e^{ikx-WT}(c+2ik\gamma)F_0(W,T)dk
\end{equation}

Finally, from this example we showed that how method of fokas is capable of solving the heat equation with acceptable calculation costs. We first find the local relation and the global relation. Following that, we do a semi-infinite Fourier transform to obtain $\hat{u}_0(k)$, $\hat{u}_0(v(k))$, and $F_0(W,T)$. Then from analysis of exponential term, we obtain the integral path $\partial \Omega$. The final step is simply calculate the integral values out.

It is noted that for general heat equation with advection, $u_t=\gamma u_{xx}+cu_x+\alpha u$, nothing changes except for $W(k)\to W(k)-\gamma k^2+ikc+\alpha=0$. The symmetry of $W(k)$ allows us to eliminate $F_1(W,T)$. In a similar way with substitution for $F_0(W,T)$, one can eliminate the dependence on $F_0(W,T)$. For higher order PDEs, this method of fokas especially exhibits a generality to solutions.

## Implementation

In this section, I'll implement the Matlab scripts for solving $u(x,t)$ for the heat equation with adevction in 3 different methods: **Spectral Method**, **Time-Stepping Method**, and the method we just covered the **Fokas' Method**. You may note that the previous 2 methods are difficult to solve the linear PDEs system on the half line, because both requires specific boundary conditions. Now let us restrict the boundary conditions to be homogeneous and the heat equation does not cross the $x=0$, by that meaning:

 $u(0,t)=f_0(t)=0$, and $u(x,0)=u_0(x)$ such that $u(x,t)$ is always 0 for $x<0, \; 0\le t<T'$ where $T'$ is a fixed positive finite number.

Specifically, the example we are considering is:

\begin{align}\label{eq:}
u_t&=\gamma u_{xx} +\beta u_x+\alpha u, \quad\quad x>0, \; 0\le t<T'\\
u(x,0)&=u_0(x)=e^{-\frac{(x-16)^2}{10}}\\
u(0,t)&=f_0(t)=0\\
\end{align}

The coefficients are $\gamma=0.01, \;\beta=0.1, \;\alpha=0$, respectively. For the spectral method, we still use the Fourier Transform, in the interval $[-L,L]$,taking $L$ sufficiently large. Then we focus on the $[0,L]$ interval, provided that the solution of heat equation has $0$ value on the negative $x$ axis. So the periodic boundary condition $u(0,t)=u(L,t)=0$ still holds. For the Time-stepping method, we are using the interval $[0,L]$. ode45 solver will be used and the fourth-order center  difference scheme is used to calculate $u_{xx},\; u_x$. Finally, for the method of fokas: because $f_0(t)=0$, the last term in solution formula (eq. 2.13) is dropped. The integral with function-handle is used, and the complex path $\partial \Omega$ is set as "waypoints" in Matlab integral options.

## Conclusion and improvement

The animated plot shows that the method of fokas gives the result very close to the result from time-stepping method, while the spectral method using Fourier Transform produces an obvious discrepancy from the other two.

I guess the for the spectral method, our boundary is not symmetric nor periodic. Even though $L$ is took to be 30, it is not sufficiently large to assume the b.c holds. Some improvement might be try to use Fourier Sine Transform in the $[0,L]$ interval. Additionally, by increasing the $L$ we should explicitly make our region extend to infinity.

I haven't figured out how to solve the heat equation with non-homogeneous boundary conditions ($u(0,t)=f_0(t)\ne 0$). Generally, it requires to use Green's function to obtain the exact solution. However, I didn't manage to solve that numerically. Thus, I'm only using this homogeneous boundary condition for simplicity. One should then note the effectiveness and generality of method of fokas.

## Reference

Deconinck, B.; Trogdon, T.; Vasan, V. (2014-01-01). "The Method of Fokas for Solving Linear Partial Differential Equations". SIAM Review. 56 (1): 159–186. CiteSeerX 10.1.1.454.8462. doi:10.1137/110821871. ISSN 0036-1445.

final project method of fokas/.ipynb_checkpoints/final_project_RL-checkpoint.ipynb


## Task 14 - Motor Control
### Introduction to modeling and simulation of human movement
https://github.com/BMClab/bmc/blob/master/courses/ModSim2018.md

Desiree Miraldo

* Implement a simulation of the ankle joint model using the parameters from Thelen (2003) and Elias (2014)

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import math

%matplotlib notebook
```

### Muscle parameters

From Nigg & Herzog (2006):

```python
Umax = .04 # SEE strain at Fmax
width = .63#*Lce_o
```

From Thelen(2003) - dorsiflexors:

```python
Lce_o = .09 #optimal l (dorsiflexors) from Thelen(2003)
Lslack = 2.4*Lce_o # slack length (dorsiflexors)  from Thelen(2003)
alpha = 7*np.pi/180 #pennation angle (rad) (dorsiflexors) from Thelen(2003)
Fmax = 1400 #maximal isometric force (dorsiflexors)  from Thelen(2003)

Vmax = 10  # young adults

A = np.array([30.60,-7.44e-2,-1.41e-4,2.42e-6,1.50e-8])*1e-2 #parameters from Elias(2014) (meter/deg^ind)
B = np.array([4.30,1.66e-2,-3.89e-4,-4.45e-6,-4.34e-8])*1e-2 #parameters from Elias(2014) (meter/deg^ind)
```

Using BSP values from Dempster's model adapted by Winter (2009), available in [BodySegmentParameters.ipynb](https://github.com/BMClab/bmc/blob/master/notebooks/BodySegmentParameters.ipynb)

```python
m = 75 #kg
g = 9.81 #(m/s^2)
Lfoot = 26e-2 #(m)
mfoot = 0.0145*m #0.0145 from column Mass (kg)
Rcm = 0.5* Lfoot #0.5 from column CM prox (m)
Ifoot = mfoot*(.69*Lfoot)**2 #0.690 from column Rg prox (kg*m^2)

```

### Initial conditions

```python
phi = 0 #start as 0 degree flexion (rad)
phid = 0 #zero velocity
#Lnorm_ce = .087/Lce_o #norm
Lnorm_ce = ((.306-Lslack)/np.cos(alpha))/Lce_o
t0 = 0 #Initial time
tf = .5 #Final Time
h = 1e-4 #integration step size and step counter
```

```python
t = np.arange(t0,tf,h)
F = np.empty(t.shape)
Fkpe = np.empty(t.shape)
FiberLen = np.empty(t.shape)
TendonLen = np.empty(t.shape)
a_dynamics = np.empty(t.shape)
phi_dynamics = np.empty(t.shape)
fiberVelocity = np.empty(t.shape)
Moment     = np.empty(t.shape)
```

Activation dynamics parameters

```python
a = 1 #inital conditional for ativation
```

```python
# defining u (Initial conditional for Brain's activation)
form = 'step'

def createinput_u(form,t,h=.01,plot=True):
    if (form == 'sinusoid'):
        u = .2*np.sin(np.pi*t) +.7
    elif (form == 'step'):
        u = np.ones(t.shape)*h
        u[:int(1/h)] = 0
        u[int(1/h):int(3/h)] = 1

    elif (form == 'pulse'):
        u = np.ones(t.shape)*h
        u[int(1/h):int(3/h)] = 1

    if plot:
        plt.figure()
        plt.plot(u)
        plt.title('u wave form')
    return u

#u = createinput_u(form,h)
u = np.ones(t.shape)/1
```

## Functions

```python
def momentArm(phi,B):
    '''
    Compute moment arm of muscle - Eq. 9 from Elias(2014)

    Inputs:
        A = parameters from Elias(2014) (meter/deg^ind)
        B = parameters from Elias(2014) (meter/deg^ind)
    Output:
        Lm = length Muscle+tendon
    '''

    phi = phi*180/np.pi

    Rf = 0
    for i in range(len(B)):
        Rf += B[i]*(phi**i) #Eq. 9 from Elias(2014)

    return Rf

```

```python
def totalMuscleLength(phi,A):
    '''
    Compute length Muscle+tendon - Eq. 8 from Elias(2014)

    Inputs:
        A = parameters from Elias(2014) (meter/deg^ind)
        thetaAnkle = angle of ankle
    Output:
        Lm = length Muscle+tendon

    '''
    phi = phi*180/np.pi

    Lm = 0
    for i in range(len(A)):
        Lm += A[i]*(phi**i)

    return Lm

```

```python
def tendonLength(Lm,Lce_o,Lnorm_ce, alpha):
    '''
    Compute tendon length

    Inputs:
        Lm =
        Lce_o = optimal length of the fiber
        Lnorm_ce = normalized contractile element length

    Output:
        Lnorm_see = normalized tendon length
    '''
    Lnorm_see = Lm/Lce_o - Lnorm_ce*np.cos(alpha)

    return Lnorm_see
```

```python
def TendonForce (Lnorm_see,Lce_o,Lslack=0.216):
    '''
    Compute tendon force

    Inputs:
        Lnorm_see = normalized tendon length
        Lslack = slack length of the tendon (non-normalized)
        Lce_o = optimal length of the fiber

    Output:
        Fnorm_tendon = normalized tendon force

    '''
    Umax = .04

    if Lnorm_see<Lslack/Lce_o:
        Fnorm_tendon = 0
    else:
        Fnorm_tendon = ((Lnorm_see-Lslack/Lce_o)/(Umax*Lslack/Lce_o))**2

    return Fnorm_tendon
```

```python
def ParallelElementForce (Lnorm_ce):
    '''
    Compute parallel element force

    Inputs:
        Lnorm_ce = normalized contractile element length

    Output:
        Fnorm_kpe = normalized parallel element force

    '''
    Umax = 1

    if Lnorm_ce< 1:
        Fnorm_kpe = 0
    else:
        Fnorm_kpe = ((Lnorm_ce-1)/(Umax*1))**2

    return Fnorm_kpe
```

```python
def ForceLengthCurve (Lnorm_ce,width):
    F0 = max([0, (1-((Lnorm_ce-1)/width)**2)])
    return F0
```

```python
def ContractileElementDot(F0, Fnorm_ce, a,Vmax=10, FMlen = 1.4, Af = 0.25):

    '''
    Compute Contractile Element Derivative

    Inputs:
        F0 = Force-Length Curve
        Fce = Contractile element force
        Vmax = maximum muscle velocity for concentric activation
        FMlen = normalized maximum force generated at the lengthening phase
        Af = shape factor

    Output:
        Lnorm_cedot = normalized contractile element length derivative

    '''

    Fnorm_ce = min(FMlen*a*F0 - 0.001, Fnorm_ce)

    if  Fnorm_ce > a*F0:

        b = ((2 + 2/Af)*(a*F0*FMlen - Fnorm_ce))/(FMlen-1)

    elif Fnorm_ce <= a*F0:

        b = a*F0 + Fnorm_ce/Af

    Lnorm_cedot = (.25 + .75*a)*Vmax*((Fnorm_ce - a*F0)/b)

    return Lnorm_cedot
```

```python
def fvce_T03_symb():
    # Thelen (2003) velocity of the force-velocity relationship as function of CE force

    from sympy import symbols, solve, collect, Eq
    a, F0, Fnorm_ce, Af, FMlen, Vmax = symbols('a, F0, Fnorm_ce, Af, FMlen, Vmax', positive=True)
    vel_ce = symbols('vel_ce', real=True)

    b = a*F0 + Fnorm_ce/Af
    vel_ce_eq = Eq(vel_ce- (0.25 + 0.75*a)*Vmax*(Fnorm_ce - a*F0)/b)
    sol = solve(vel_ce_eq, Fnorm_ce)
    print('Fnorm_ce <= a*F0:\n', collect(sol[0], Vmax),'\n')

    b = (2 + 2/Af)*(a*F0*FMlen - Fnorm_ce)/(FMlen - 1)
    vel_ce_eq = Eq(vel_ce- (0.25 + 0.75*a)*Vmax*(Fnorm_ce - a*F0)/b)
    sol = solve(vel_ce_eq, Fnorm_ce)
    print('Fnorm_ce > a*F0:\n', collect(sol[0], (Vmax*Af, FMlen, vel_ce)))

fvce_T03_symb()
```

    Fnorm_ce <= a*F0:
     Af*F0*a*(Vmax*(3.0*a + 1) + 4.0*vel_ce)/(Vmax*(3.0*Af*a + Af) - 4.0*vel_ce)

    Fnorm_ce > a*F0:
     F0*a*(Af*Vmax*(3.0*FMlen*a + FMlen - 3.0*a - 1) + FMlen*(8.0*Af*vel_ce + 8.0*vel_ce))/(Af*Vmax*(3.0*FMlen*a + FMlen - 3.0*a - 1) + vel_ce*(8.0*Af + 8.0))

```python
def ForceVelocityCurve(vel_ce,a, F0, Fnorm_ce, Vmax=10, FMlen = 1.4, Af = 0.25):
    """'''
    Compute Force Velocity curve

    Inputs:
        vel_ce = contractile element velocity
        a = muscle activation level
        F0 = Force-Length Curve
        Fnorm_ce = Contractile element force
        Vmax = maximum muscle velocity for concentric activation
        FMlen = normalized maximum force generated at the lengthening phase
        Af = shape factor

    Output:
        FVnorm_ce = normalized force of the muscle contractile element

    '''

    """

    Vmax = Vmax*Lce_o
    if vel_ce <= 0:  # isometric and concentric activation
        FVnorm_ce = Af*F0*a*(Vmax*(3.0*a + 1) + 4.0*vel_ce)/(Vmax*(3.0*Af*a + Af) - 4.0*vel_ce)
    else:        # eccentric activation
        FVnorm_ce = F0*a*(Af*Vmax*(3.0*FMlen*a + FMlen - 3.0*a - 1) + FMlen*(8.0*Af*vel_ce + 8.0*vel_ce))/(Af*Vmax*(3.0*FMlen*a + FMlen - 3.0*a - 1) + vel_ce*(8.0*Af + 8.0))

    return FVnorm_ce
```

```python
def ContractileElementForce(Fnorm_tendon,Fnorm_kpe, alpha):
    '''
    Compute Contractile Element force

    Inputs:
        Fnorm_tendon = normalized tendon force
        Fnorm_kpe = normalized parallel element force

    Output:
        Fnorm_CE = normalized contractile element force
    '''
    Fnorm_CE = Fnorm_tendon/np.cos(alpha) - Fnorm_kpe
    return Fnorm_CE
```

```python
def activation(a,u,dt):
    '''
    Compute activation

    Inputs:
        u = idealized muscle excitation signal, 0 <= u <= 1
        a = muscular activation
        dt = time step

    Output:
        a = muscular activation
    '''

    tau_deact = 50e-3 #young adults
    tau_act = 15e-3

    if u>a:
        tau_a = tau_act*(0.5+1.5*a)
    elif u <=a:
        tau_a = tau_deact/(0.5+1.5*a)

    #-------
    dadt = (u-a)/tau_a # euler

    a = a + dadt*dt
    #-------
    return a
```

def totalLenght(Lm0,phi,Rf,Rcm):
    '''
    Inputs:
        Lm0 = initial lenght of the muscle
        Phi = degree flexion of the joint
        RF = Moment arm
        Lce_o = optimal size of the muscle
    Output:
        Lm = total muscle lenght
    '''
    Lm = Lm0 - (phi-(math.pi/2))*Rf #total muscle-tendon length from joint angle
    return Lm

```python
def momentJoint(Rf,Fnorm_tendon,Fmax,m,g,phi):
    '''
    Inputs:
        Rf = Moment arm
        Fnorm_tendon = Normalized tendon force
        m = Segment Mass
        g = Acelleration of gravity
        Fmax= maximal isometric force
        phi = angle (deg)
    Output:
        M = Total moment with respect to joint
    '''
    M=Rf*Fnorm_tendon*Fmax - m*g*Rcm*np.sin(np.pi/2-phi)
    return M
```

```python
def angularAcelerationJoint (M,I):
    '''
    Inputs:
        M = Total moment with respect to joint
        I = Moment of Inertia
    Output:
        phidd= angular aceleration of the joint
    '''
    phidd = M/I
    return phidd
```

### Check initial conditions

```python
def muscle_plot(a=1):
    """Plot muscle-tendon relationships with length and velocity."""

    import matplotlib.pyplot as plt

    fig, axs = plt.subplots(nrows=1, ncols=3, figsize=(12, 4))

    lmopt   = Lce_o
    ltslack = Lslack
    vmmax   = Vmax
    alpha0  = alpha
    fm0     = Fmax
    lm0     = Lnorm_ce*Lce_o
    lmt0    = totalMuscleLength(phi,A)
    lt0     = lmt0 - lm0*np.cos(alpha0)

    lm  = np.linspace(0, 2, 101)
    lt  = np.linspace(0, 1, 101)*0.05 + 1
    vm  = np.linspace(-1, 1, 101)*vmmax*lmopt
    fl  = np.zeros(lm.size)
    fpe = np.zeros(lm.size)
    fse = np.zeros(lt.size)
    fvm = np.zeros(vm.size)

    fl_lm0  = ForceLengthCurve(lm0/lmopt,width)
    fpe_lm0 = ParallelElementForce(lm0/lmopt)
    fm_lm0  = fl_lm0 + fpe_lm0
    ft_lt0  = TendonForce(lt0,lmopt, ltslack)*fm0

    for i in range(101):
        fl[i]  = ForceLengthCurve(lm[i],width)
        fpe[i] = ParallelElementForce(lm[i])
        fse[i] = TendonForce(lt[i]/lmopt,lmopt,Lslack=1)
        fvm[i] = ForceVelocityCurve(vm[i],a, fl_lm0, lm0/lmopt)

    lm  = lm*lmopt
    lt  = lt*ltslack
    fl  = fl
    fpe = fpe
    fse = fse*fm0
    fvm = fvm*fm0

    xlim = lm
    #axs[0].set_xlim(xlim)
    ylim = [0, 2]
    axs[0].set_ylim(ylim)
    axs[0].plot(lm, fl, 'b', label='Active')
    axs[0].plot(lm, fpe, 'b--', label='Passive')
    axs[0].plot(lm, fl+fpe, 'b:', label='')
    axs[0].plot([lm0, lm0], [ylim[0], fm_lm0], 'k:', lw=2, label='')
    axs[0].plot([xlim[0], lm0], [fm_lm0, fm_lm0], 'k:', lw=2, label='')
    axs[0].plot(lm0, fm_lm0, 'o', ms=6, mfc='r', mec='r', mew=2, label='fl(LM0)')
    axs[0].legend(loc='best', frameon=True, framealpha=.5)
    axs[0].set_xlabel('Length [m]')
    axs[0].set_ylabel('Scale factor')
    axs[0].xaxis.set_major_locator(plt.MaxNLocator(4))
    axs[0].yaxis.set_major_locator(plt.MaxNLocator(4))
    axs[0].set_title('Muscle F-L (a=1)')

    xlim = [0, np.min(vm), np.max(vm)]
    #axs[1].set_xlim(xlim)
    ylim = [0, fm0*1.2, np.max(fvm)*1.5]
    #axs[1].set_ylim(ylim)
    axs[1].plot(vm, fvm, label='')
    axs[1].set_xlabel('$\mathbf{^{CON}}\;$ Velocity [m/s] $\;\mathbf{^{EXC}}$')
    axs[1].plot([0, 0], [ylim[0], fvm[50]], 'k:', lw=2, label='')
    axs[1].plot([xlim[0], 0], [fvm[50], fvm[50]], 'k:', lw=2, label='')
    axs[1].plot(0, fvm[50], 'o', ms=6, mfc='r', mec='r', mew=2, label='FM0(LM0)')
    axs[1].plot(xlim[0], fm0, '+', ms=10, mfc='r', mec='r', mew=2, label='')
    axs[1].text(vm[0], fm0, 'FM0')
    axs[1].legend(loc='upper right', frameon=True, framealpha=.5)
    axs[1].set_ylabel('Force [N]')
    axs[1].xaxis.set_major_locator(plt.MaxNLocator(4))
    axs[1].yaxis.set_major_locator(plt.MaxNLocator(4))
    axs[1].set_title('Muscle F-V (a=1)')

    xlim = [lt0, ltslack, np.min(lt), np.max(lt)]
    #axs[2].set_xlim(xlim)
    ylim = [-100, np.max(fse)]
    #axs[2].set_ylim(ylim)
    axs[2].plot(lt, fse, label='')
    axs[2].set_xlabel('Length [m]')
    axs[2].plot([lt0, lt0], [ylim[0], ft_lt0], 'k:', lw=2, label='')
    axs[2].plot([xlim[0], lt0], [ft_lt0, ft_lt0], 'k:', lw=2, label='')
    axs[2].plot(lt0, ft_lt0, 'o', ms=6, mfc='r', mec='r', mew=2, label='FT(LT0)')
    axs[2].legend(loc='upper left', frameon=True, framealpha=.5)
    axs[2].set_ylabel('Force [N]')
    axs[2].xaxis.set_major_locator(plt.MaxNLocator(4))
    axs[2].yaxis.set_major_locator(plt.MaxNLocator(4))
    axs[2].set_title('Tendon')
    plt.suptitle('Muscle-tendon mechanics', fontsize=14, y=1)
    plt.tight_layout(w_pad=.1)
    plt.show()
```

```python
muscle_plot(a=1)
```

    <IPython.core.display.Javascript object>

## Simulation - Parallel

```python
#Normalizing
for i in range (len(t)):

    Lm =  totalMuscleLength(phi,A)

    Lnorm_see = tendonLength(Lm,Lce_o,Lnorm_ce, alpha)

    Fnorm_tendon = TendonForce (Lnorm_see,Lce_o,Lslack)

    Fnorm_kpe = ParallelElementForce (Lnorm_ce)

    #isometric force at Lce from CE force length relationship
    F0 = ForceLengthCurve (Lnorm_ce,width)

    Fnorm_CE = ContractileElementForce(Fnorm_tendon,Fnorm_kpe, alpha) #Fnorm_CE = ~Fm

    #computing activation
    a = activation(a,u[i],h)

    #calculate CE velocity from Hill's equation
    Lnorm_cedot = ContractileElementDot(F0, Fnorm_CE,a)

    #Compute MomentJoint
    Rf = momentArm(phi,B) #moment arm
    M = momentJoint(Rf,Fnorm_tendon,Fmax,mfoot,g,phi)

    #Compute Angular Aceleration Joint
    phidd = angularAcelerationJoint (M,Ifoot)

    # Euler integration steps
    Lnorm_ce = Lnorm_ce + h*Lnorm_cedot
    phid= phid + h*phidd
    phi  = phi  +h*phid

    # Store variables in vectors
    F[i] = Fnorm_tendon*Fmax
    Fkpe[i] = Fnorm_kpe*Fmax
    FiberLen[i] = Lnorm_ce*Lce_o
    TendonLen[i] = Lnorm_see*Lce_o
    a_dynamics[i] = a
    phi_dynamics[i] = phi
    fiberVelocity[i] = Lnorm_cedot*Lce_o
    #moment[i] =
```

## Plots

```python
fig, ax = plt.subplots(1, 1, figsize=(6,6), sharex=True)

ax.plot(t,fiberVelocity,c='purple')
plt.grid()
plt.xlabel('time (s)')
plt.ylabel('fiber Velocity (m/s)')
```

    <IPython.core.display.Javascript object>

    <matplotlib.text.Text at 0xb737a20>

```python
fig, ax = plt.subplots(1, 1, figsize=(6,6), sharex=True)

ax.plot(t,phi_dynamics ,c='orange')
plt.grid()
plt.xlabel('time (s)')
plt.ylabel('Joint angle (deg)')
```

    <IPython.core.display.Javascript object>

    <matplotlib.text.Text at 0xb75a0b8>

```python
fig, ax = plt.subplots(1, 1, figsize=(6,6), sharex=True)

ax.plot(t,a_dynamics,c='magenta')
plt.grid()
plt.xlabel('time (s)')
plt.ylabel('Activation dynamics')
```

    <IPython.core.display.Javascript object>

    <matplotlib.text.Text at 0xb7ab128>

```python
fig, ax = plt.subplots(1, 1, figsize=(6,6), sharex=True)

ax.plot(t,F,c='red')
plt.grid()
plt.xlabel('time (s)')
plt.ylabel('Force (N)')
```

    <IPython.core.display.Javascript object>

    <matplotlib.text.Text at 0xb7ebba8>

```python
fig, ax = plt.subplots(1, 1, figsize=(6,6), sharex=True)

ax.plot(t,FiberLen, label = 'fiber')
ax.plot(t,TendonLen, label = 'tendon')
plt.grid()
plt.xlabel('time (s)')
plt.ylabel('Length (m)')
ax.legend(loc='best')

fig, ax = plt.subplots(1, 3, figsize=(9,4), sharex=True, sharey=True)
plt.subplots_adjust(wspace=.12, hspace=None,left=None, bottom=None, right=None, top=None,)
ax[0].plot(t,FiberLen, label = 'fiber')
ax[1].plot(t,TendonLen, label = 'tendon')
ax[2].plot(t,FiberLen + TendonLen, label = 'muscle (tendon + fiber)')

ax[1].set_xlabel('time (s)')
ax[0].set_ylabel('Length (m)')
#ax[i].legend(loc='best')
```

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    <matplotlib.text.Text at 0xb892828>

```python

```