## Subsurface Data Analytics

### The Lasso for Subsurface Data Analytics in Python

#### Michael Pyrcz, Associate Professor, University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

### PGE 383 Exercise: The Lasso for Subsurface Modeling in Python

Here's a simple workflow, demonstration of the lasso for regression for subsurface modeling workflows. This should help you get started with building subsurface models with data analytics and machine learning. Here's some basic details about the lasso.

The lasso is an extension of linear regresion, and is closely related to ridge regression so let's review some basic details about linear regression first and then related the lasso to ridge regression.

Here's complete workflows and more details on these:

[Linear Regression](https://github.com/GeostatsGuy/PythonNumericalDemos/blob/master/SubsurfaceDataAnalytics_LinearRegression.ipynb)

[Ridge Regression](https://github.com/GeostatsGuy/PythonNumericalDemos/blob/master/SubsurfaceDataAnalytics_RidgeRegression.ipynb)

#### Linear Regression

Linear regression for prediction.  Here are some key aspects of linear regression:

**Parametric Model**

* the fit model is a simple weighted linear additive model based on all the available features, $x_1,\ldots,x_m$.

* the parametric model takes the form of:

\begin{equation}
y = \sum_{\alpha = 1}^m b_{\alpha} x_{\alpha} + b_0
\end{equation}

**Least Squares**

* least squares optimization is applied to select the model parameters, $b_1,\ldots,b_m,b_0$

* we minize the error, residual sum of squares (RSS) over the training data:

\begin{equation}
RSS = \sum_{i=1}^n (y_i - \left(\sum_{\alpha = 1}^m b_{\alpha} x_{\alpha} + b_0)\right)^2
\end{equation}

* this could be simplified as the sum of square error over the training data,

\begin{equation}
\sum_{i=1}^n (\Delta y_i)^2
\end{equation}

**Assumptions**

* **Error-free** - predictor variables are error free, not random variables
* **Linearity** - response is linear combination of feature(s)
* **Constant Variance** - error in response is constant over predictor(s) value
* **Independence of Error** - error in response are uncorrelated with each other
* **No multicollinearity** - none of the features are redundant with other features

#### Other Resources

In $Python$, the $SciPy$ package, specifically the $Stats$ functions (https://docs.scipy.org/doc/scipy/reference/stats.html) provide excellent tools for efficient use of statistics.
I have previously provided linear regression demonstration in R:

1. [R](https://github.com/GeostatsGuy/geostatsr/blob/master/linear_regression_demo_v2.R)
2. [R Markdown](https://github.com/GeostatsGuy/geostatsr/blob/master/linear_regression_demo_v2.Rmd)
3. [knit as an HTML document](https://github.com/GeostatsGuy/geostatsr/blob/master/linear_regression_demo_v2.html)

#### The Lasso

With the lasso we add a hyperparameter, $\lambda$, to our minimization, with a shrinkage penalty term.

\begin{equation}
\sum_{i=1}^n \left(y_i - \left(\sum_{\alpha = 1}^m b_{\alpha} x_{\alpha} + b_0 \right) \right)^2 + \lambda \sum_{j=1}^m |b_{\alpha}|
\end{equation}

As a result the lasso has 2 criteria:

1. set the model parameters to minimize the error with training data

2. shrink the estimates of the slope parameters towards zero. Note: the intercept is not affected by the lambda, $\lambda$, hyperparameter.

Note the only difference between the lasso and ridge regression is:

* for the lasso the shrinkage term is posed as an $\ell_1$ penalty ($\lambda \sum_{\alpha=1}^m |b_{\alpha}|$)

* for ridge regression the shrinkage term is posed as an $\ell_2$ penalty ($\lambda \sum_{\alpha=1}^m \left(b_{\alpha}\right)^2$).

While both ridge regression and the lasso shrink the model parameters ($b_{\alpha}, \alpha = 1,\ldots,m$) towards zero:

* the lasso parameters reach zero at different rates for each predictor feature as the lambda, $\lambda$, hyperparameter increases.

* as a result the lasso provides a method for feature ranking and selection!

The lambda, $\lambda$, hyperparameter controls the degree of fit of the model and may be related to the model variance and bias trade-off.

* for $\lambda \rightarrow 0$ the prediction model approaches linear regression, there is lower model bias, but the model variance is higher

* as $\lambda$ increases the model variance decreases and the model bias increases

* for $\lambda \rightarrow \infty$ the coefficients all become 0.0 and the model is the global mean

#### Workflow Goals

Learn the basics of the lasso in Python to for analysis, modeling and prediction of porosity from density. This includes:

* Basic Python workflows and data preparation

* Training / fitting a the lasso model and comparison to ridge regression

* Checking the model and learning about the impact of hyperparameters

#### Objective

In the PGE 383: Stochastic Subsurface Modeling class I want to provide hands-on experience with building subsurface modeling workflows. Python provides an excellent vehicle to accomplish this. I have coded a package called GeostatsPy with GSLIB: Geostatistical Library (Deutsch and Journel, 1998) functionality that provides basic building blocks for building subsurface modeling workflows.

The objective is to remove the hurdles of subsurface modeling workflow construction by providing building blocks and sufficient examples. This is not a coding class per se, but we need the ability to 'script' workflows working with numerical methods.

#### Getting Started

Here's the steps to get setup in Python with the GeostatsPy package:

1. Install Anaconda 3 on your machine (https://www.anaconda.com/download/).
2. From Anaconda Navigator (within Anaconda3 group), go to the environment tab, click on base (root) green arrow and open a terminal.
3. In the terminal type: pip install geostatspy.
4. Open Jupyter and in the top block get started by copy and pasting the code block below from this Jupyter Notebook to start using the geostatspy functionality.

There are examples below with these functions. You can go here to see a list of the available functions, https://git.io/fh4eX, other example workflows and source code.

#### Import Required Packages

Let's import the GeostatsPy package.

```python
import os                                                   # to set current working directory
import numpy as np                                          # arrays and matrix math
import scipy.stats as st                                    # statistical methods
import pandas as pd                                         # DataFrames
import matplotlib.pyplot as plt                             # for plotting
from sklearn.metrics import mean_squared_error, r2_score    # specific measures to check our models
from sklearn.linear_model import Ridge                      # ridge regression implemented in scikit learn
from sklearn.linear_model import Lasso                      # the lasso implemented in scikit learn
from sklearn.model_selection import cross_val_score         # multi-processor K-fold crossvalidation
from sklearn.model_selection import train_test_split        # train and test split
from sklearn.preprocessing import StandardScaler            # standardize the features
from sklearn import metrics                                 # measures to check our models
from sklearn.linear_model import LinearRegression           # linear regression implemented in scikit learn
```

If you get a package import error, you may have to first install some of these packages. This can usually be accomplished by opening up a command window on Windows and then typing 'python -m pip install [package-name]'. More assistance is available with the respective package docs.

#### Set the working directory

I always like to do this so I don't lose files and to simplify subsequent read and writes (avoid including the full address each time).  Also, in this case make sure to place the required (see below) data file in this working directory.

```python
os.chdir("C:\PGE383")                                       # set the working directory
```

#### Loading Data

Let's load the provided dataset. 'Density_Por_data.csv' is available at https://github.com/GeostatsGuy/GeoDataSets. It is a comma delimited file with 20 density ($\frac{g}{cm^3}$) and porosity (as a fraction) measures from the subsurface. We load the data file with the pandas 'read_csv' function into a data frame we called 'df' and then separate it into train and test datasets.  The smaples are in random order so we just split the dataset at the 80th sample.  We preview each with the head function from Pandas DataFrames.

```python
df = pd.read_csv("Density_Por_data.csv")                    # read a .csv file in as a DataFrame
df_mv = pd.read_csv("unconv_MV.csv")                        # load a multivariate dataset that we will use later
df_train = df.iloc[0:80,:]                                  # extract a training set, note samples are random ordered
df_train.head()                                             # preview the DataFrame
```

      Density
      Porosity

      0
      1.281391
      16.610982

      1
      1.404932
      13.668073

      2
      2.346926
      9.590092

      3
      1.348847
      15.877907

      4
      2.331653
      4.968240

```python
df_test = df.iloc[80:]                                      # extract a testing set, note samples are random ordered
df_test.head()                                              # preview the DataFrame
```

      Density
      Porosity

      80
      1.750352
      11.325941

      81
      1.666285
      15.609445

      82
      1.466517
      17.066529

      83
      1.650921
      13.876841

      84
      0.996736
      20.964941

It is useful to review the summary statistics of our loaded DataFrame.  That can be accomplished with the 'describe' DataFrame member function.  We transpose to switch the axes for ease of visualization.  We will summarize over the training and testing subsets separately.

```python
df_train.describe().transpose()
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      Density
      80.0
      1.759945
      0.295260
      1.067960
      1.552707
      1.767908
      1.953262
      2.410560

      Porosity
      80.0
      12.187127
      3.125755
      4.966421
      10.011526
      12.170687
      14.063670
      19.600717

```python
df_test.describe().transpose()
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      Density
      25.0
      1.667429
      0.257608
      0.996736
      1.568988
      1.718085
      1.791432
      2.339324

      Porosity
      25.0
      13.632564
      2.948887
      9.489298
      11.325941
      13.767060
      15.051759
      20.964941

Here we extract the ndarrays with porsity and density, training and testing datasets separate arrays for convenience.

```python
por_train = df_train['Porosity'].values                     # make a shallow copy of the features for convenvience
den_train = df_train['Density'].values
por_test = df_test['Porosity'].values
den_test = df_test['Density'].values
n_train = len(df_train); n_test = len(df_test)              # get the number of data in training and testing
```

#### Linear Regression Model

Let's first calculate the linear regression model. We use scikit learn and then extend the same workflow to ridge regression.

```python
# Step 1. Instantiate the Model
linear_reg = LinearRegression()                             # instantiate the linear regression model

# Step 2: Fit the Data on Training Data
linear_reg.fit(df_train["Density"].values.reshape(n_train,1), df_train["Porosity"]) # fit model
density_model = np.linspace(1.2,2.4,10)

# Print the model parameters
porosity_model = linear_reg.predict(density_model.reshape(10,1)) # predict with the fit model
print('Coefficients: ', str(round(linear_reg.coef_[0],3)) + ', Intercept: ', str(round(linear_reg.intercept_,3)))

# Plot model fit
plt.subplot(111)
plt.scatter(df_train["Density"].values, df_train["Porosity"],  color='black', s = 20, alpha = 0.3)
plt.plot(density_model,porosity_model, color='red', linewidth=1)
plt.title('Linear Regression Porosity from Density with Training Data'); plt.xlabel('Density (g/cm^3)'); plt.ylabel('Porosity (%)')
plt.xlim(1.,2.6)#; plt.ylim(0,1500000)
plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.0, wspace=0.2, hspace=0.2)
plt.show()
```

Let's run some quick model checks.  Much more could be done, but I limit this for breviety here.

```python
# Step 3: - Make predictions using the testing dataset
y_pred = linear_reg.predict(df_test['Density'].values.reshape(n_test,1))

# Report the goodness of fit
print('Variance explained: %.2f' % r2_score(df_test['Porosity'].values, y_pred))

# Plot testing diagnostics
plt.subplot(121)
plt.scatter(df_test['Density'].values, df_test['Porosity'].values,  color='black', s = 20, alpha = 0.3)
plt.scatter(df_test['Density'], y_pred, color='blue', s = 20, alpha = 0.3)
plt.title('Linear Regression Model Testing - Production from Porosity'); plt.xlabel('Density (g/cm^3)'); plt.ylabel('Porosity (%)')
plt.xlim(1.0,2.6); plt.ylim(5,24)

y_res = y_pred - df_test['Porosity'].values

print('Residual: mean = ' + str(round(np.average(y_res),2)) + ', standard deviation = ' + str(round(np.var(y_res),2)))

plt.subplot(122)
plt.hist(y_res, alpha = 0.2, color = 'red', edgecolor = 'black', bins=20)
plt.title('Linear Regression Model Prediction Error - Porosity form Density'); plt.xlabel('Porosity Estimation Error (%) (Estimate - Truth)'); plt.ylabel('Frequency')
plt.xlim(-4,4)#; plt.ylim(0,1500000)

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.2, top=1.2, wspace=0.3, hspace=0.2)
plt.show()
```

#### Ridge Regression

Let's replace the scikit learn linear regression method with the scikit learn ridge regression method.  Note, we must now set the lambda hyperparameter.

* the hyperparameter is set with the instantiation of the model

```python
lam = 1.0

# Step 1. Instantiate the Model
ridge_reg = Ridge(alpha=lam)

# Step 2: Fit the Data on Training Data
ridge_reg.fit(df_train["Density"].values.reshape(n_train,1), df_train["Porosity"]) # fit model
density_model = np.linspace(1.2,2.4,10)

# Print the model parameters
porosity_ridge_model = ridge_reg.predict(density_model.reshape(10,1)) # predict with the fit model
print('Coefficients: ', str(round(linear_reg.coef_[0],3)) + ', Intercept: ', str(round(linear_reg.intercept_,3)))

# Plot model fit
plt.subplot(111)
plt.scatter(df_train["Density"].values, df_train["Porosity"],  color='black', s = 20, alpha = 0.3)
plt.plot(density_model,porosity_ridge_model, color='red', linewidth=1)
plt.title('Ridge Regression Porosity from Density with Training Data'); plt.xlabel('Density (g/cm^3)'); plt.ylabel('Porosity (%)')
plt.xlim(1.,2.6); plt.ylim(8,17)
plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.0, wspace=0.2, hspace=0.2)
plt.show()
```

Let's repeat the simple model checks that we applied with our linear regression model.

```python
# Step 3: - Make predictions using the testing dataset
y_pred = ridge_reg.predict(df_test['Density'].values.reshape(n_test,1))

# Report the goodness of fit
print('Variance explained: %.2f' % r2_score(df_test['Porosity'].values, y_pred))

# Plot testing diagnostics
plt.subplot(121)
plt.scatter(df_test['Density'].values, df_test['Porosity'].values,  color='black', s = 20, alpha = 0.3)
plt.scatter(df_test['Density'], y_pred, color='blue', s = 20, alpha = 0.3)
plt.title('Ridge Regression Model Testing - Production from Porosity'); plt.xlabel('Density (g/cm^3)'); plt.ylabel('Porosity (%)')
plt.xlim(1.0,2.6); plt.ylim(5,24)

y_res = y_pred - df_test['Porosity'].values

print('Residual: mean = ' + str(round(np.average(y_res),2)) + ', standard deviation = ' + str(round(np.var(y_res),2)))

plt.subplot(122)
plt.hist(y_res, alpha = 0.2, color = 'red', edgecolor = 'black', bins=20)
plt.title('Ridge Regression Model Prediction Error - Porosity form Density'); plt.xlabel('Porosity Estimation Error (%) (Estimate - Truth)'); plt.ylabel('Frequency')
plt.xlim(-4,4)#; plt.ylim(0,1500000)

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.2, top=1.2, wspace=0.3, hspace=0.2)
plt.show()
```

Interesting, we explained less variance and have a larger residual standard deviation (more error).

* we reduced both testing variance explained and accuracy in this case!

#### the Lasso

Let's replace the scikit learn linear regression and ridge regression methods with the scikit learn the lasso method.  Note, we must now set the lambda hyperparameter.

* the lambda hyperparameter, $\lambda$, is set with the instantiation of the model

```python
lam = 1.0

# Step 1. Instantiate the Model
lasso_reg = Lasso(alpha=lam)

# Step 2: Fit the Data on Training Data
lasso_reg.fit(df_train["Density"].values.reshape(n_train,1), df_train["Porosity"]) # fit model
density_model = np.linspace(1.2,2.4,10)

# Print the model parameters
porosity_lasso_model = lasso_reg.predict(density_model.reshape(10,1)) # predict with the fit model
print('Coefficients: ', str(round(linear_reg.coef_[0],3)) + ', Intercept: ', str(round(linear_reg.intercept_,3)))

# Plot model fit
plt.subplot(111)
plt.scatter(df_train["Density"].values, df_train["Porosity"],  color='black', s = 20, alpha = 0.3)
plt.plot(density_model,porosity_lasso_model, color='red', linewidth=1)
plt.title('the Lasso Porosity from Density with Training Data'); plt.xlabel('Density (g/cm^3)'); plt.ylabel('Porosity (%)')
plt.xlim(1.,2.6); plt.ylim(8,17)
plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.0, wspace=0.2, hspace=0.2)
plt.show()
```

Let's repeat the simple model checks that we applied with our linear regression model.

```python
# Step 3: - Make predictions using the testing dataset
y_pred = lasso_reg.predict(df_test['Density'].values.reshape(n_test,1))

# Report the goodness of fit
print('Variance explained: %.2f' % r2_score(df_test['Porosity'].values, y_pred))

# Plot testing diagnostics
plt.subplot(121)
plt.scatter(df_test['Density'].values, df_test['Porosity'].values,  color='black', s = 20, alpha = 0.3)
plt.scatter(df_test['Density'], y_pred, color='blue', s = 20, alpha = 0.3)
plt.title('Linear Regression Model Testing - Production from Porosity'); plt.xlabel('Density (g/cm^3)'); plt.ylabel('Porosity (%)')
plt.xlim(1.0,2.6); plt.ylim(5,24)

y_res = y_pred - df_test['Porosity'].values

print('Residual: mean = ' + str(round(np.average(y_res),2)) + ', standard deviation = ' + str(round(np.var(y_res),2)))

plt.subplot(122)
plt.hist(y_res, alpha = 0.2, color = 'red', edgecolor = 'black', bins=20)
plt.title('the Lasso Model Prediction Error - Porosity form Density'); plt.xlabel('Porosity Estimation Error (%) (Estimate - Truth)'); plt.ylabel('Frequency')
plt.xlim(-4,4)#; plt.ylim(0,1500000)

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.2, top=1.2, wspace=0.3, hspace=0.2)
plt.show()
```

What happenned? With our set lambda level

```python
lam = 1.0
```

the lasso model removed the density feature and estimated with the global mean.  The lasso attempts to remove features as a function of the lambda level, in other words, the lasso performs feature selection! Let's investigate this model behavoir.

#### Investigating the Lambda Hyperparameter

Let's look at the multivariate dataset that we already loaded. This way we can observe the model behavoir over a range of features, for a range of lambda hyperparameter values.

We will first remove the well index and preview the multivariate unconventional dataset.

```python
df_mv = df_mv.drop('WellIndex',axis = 1)                    # remove the well index feature
df_mv.head()                                                # load the comma delimited data file
```

      Por
      LogPerm
      AI
      Brittle
      TOC
      VR
      Production

      0
      15.91
      1.67
      3.06
      14.05
      1.36
      1.85
      177.381958

      1
      15.34
      1.65
      2.60
      31.88
      1.37
      1.79
      1479.767778

      2
      20.45
      2.02
      3.13
      63.67
      1.79
      2.53
      4421.221583

      3
      11.95
      1.14
      3.90
      58.81
      0.40
      2.03
      1488.317629

      4
      19.53
      1.83
      2.57
      43.75
      1.40
      2.11
      5261.094919

Let's calculate the summary statistics for our data.

```python
df_mv.describe().transpose()
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      Por
      1000.0
      14.950460
      3.029634
      5.400000
      12.85750
      14.98500
      17.080000
      24.65000

      LogPerm
      1000.0
      1.398880
      0.405966
      0.120000
      1.13000
      1.39000
      1.680000
      2.58000

      AI
      1000.0
      2.982610
      0.577629
      0.960000
      2.57750
      3.01000
      3.360000
      4.70000

      Brittle
      1000.0
      49.719480
      15.077006
      -10.500000
      39.72250
      49.68000
      59.170000
      93.47000

      TOC
      1000.0
      1.003810
      0.504978
      -0.260000
      0.64000
      0.99500
      1.360000
      2.71000

      VR
      1000.0
      1.991170
      0.308194
      0.900000
      1.81000
      2.00000
      2.172500
      2.90000

      Production
      1000.0
      2247.295809
      1464.256312
      2.713535
      1191.36956
      1976.48782
      3023.594214
      12568.64413

Let's standardize the feature to have:

* mean = 0.0
* variance = standard deviation = 1.0

To do this we:

1. instantiate the StandardScaler from scikit learn. We assign it as 'scaler' so we can use it to conveniently reverse the transformation if we like. We will need to do that to get our predictions back into regular production units.

```python
scaler = StandardScaler()
```

2. we then extract all the values from our DataFrame and apply the by-column standardization.  The result is a 2D ndarray

```python
sfeatures = scaler.fit_transform(df_mv.values)
```
3. we make an new empty DataFrame

```python
df_nmv = pd.DataFrame()
```

4. then we add the transformed value to the new DataFrame while keeping the sample index and feature names from the old DataFramae

```python
df_nmv = pd.DataFrame(sfeatures, index=df_mv.index, columns=df_mv.columns)
```

```python
scaler = StandardScaler()                                   # instantiate the scaler
sfeatures = scaler.fit_transform(df_mv.values)              # standardize all the values extracted from the DataFrame
df_nmv = pd.DataFrame()                                     # instantiate a new DataFrame
df_nmv = pd.DataFrame(sfeatures, index=df_mv.index, columns=df_mv.columns) # copy the standardized values into the new DataFrame
df_nmv.head()                                               # preview the the new DataFrame
```

      Por
      LogPerm
      AI
      Brittle
      TOC
      VR
      Production

      0
      0.316877
      0.668174
      0.134046
      -2.367004
      0.705711
      -0.458285
      -1.414335

      1
      0.128641
      0.618884
      -0.662712
      -1.183816
      0.725523
      -0.653065
      -0.524438

      2
      1.816157
      1.530747
      0.255292
      0.925747
      1.557659
      1.749222
      1.485405

      3
      -0.990866
      -0.638008
      1.588995
      0.603241
      -1.196314
      0.126055
      -0.518596

      4
      1.512338
      1.062493
      -0.714674
      -0.396131
      0.784962
      0.385762
      2.059275

Let's check the summary statistics.

```python
df_nmv.describe().transpose()                               # summary statistics from the new DataFrame
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      Por
      1000.0
      1.186273e-16
      1.0005
      -3.153925
      -0.691175
      0.011406
      0.703255
      3.203157

      LogPerm
      1000.0
      -2.957079e-16
      1.0005
      -3.151793
      -0.662653
      -0.021885
      0.692819
      2.910864

      AI
      1000.0
      4.085621e-16
      1.0005
      -3.503327
      -0.701684
      0.047442
      0.653670
      2.974660

      Brittle
      1000.0
      1.703082e-16
      1.0005
      -3.996126
      -0.663393
      -0.002620
      0.627130
      2.903256

      TOC
      1000.0
      2.389755e-16
      1.0005
      -2.503956
      -0.720808
      -0.017455
      0.705711
      3.380433

      VR
      1000.0
      -3.831380e-16
      1.0005
      -3.542302
      -0.588138
      0.028665
      0.588658
      2.950365

      Production
      1000.0
      -4.163336e-17
      1.0005
      -1.533683
      -0.721496
      -0.185038
      0.530431
      7.052394

Success, we have all features standardized.  We are ready to build our model.  Let's extract training and testing datasets.

```python
X_train, X_test, y_train, y_test = train_test_split(df_nmv.iloc[:,:6], pd.DataFrame({'Production':df_nmv['Production']}), test_size=0.33, random_state=73073)
print('Number of training data = ' + str(len(X_train)) + ' and number of testing data = ' + str(len(X_test)))
```

    Number of training data = 670 and number of testing data = 330

Now let's observe the model coefficients ($b_{\alpha}, \alpha = 1,\ldots,m$) for a range of $\lambda$ hyperparameter values.

```python
nbins = 1000                                                # number of bins to explore the hyperparameter
df_nmv.describe().transpose()                               # summary statistics from the new DataFrame
lams = np.linspace(0.001,1.0,nbins)                         # make a list of lambda values
coefs = np.ndarray((nbins,6))

index = 0
for lam in lams:
    lasso_reg = Lasso(alpha=lam)                            # instantiate the model
    lasso_reg.fit(X_train, y_train)                         # fit model
    coefs[index,:] = lasso_reg.coef_                        # retreive the coefficients
    index = index + 1

color = ['black','blue','green','red','orange','grey']
plt.subplot(111)                                            # plot the results
for ifeature in range(0,6):
    plt.semilogx(lams,coefs[:,ifeature], label = df_mv.columns[ifeature], c = color[ifeature], linewidth = 3.0)

plt.title('Standardized Model Coefficients vs. Lambda Hyperparameter'); plt.xlabel('Lambda Hyperparameter'); plt.ylabel('Standardized Model Coefficients')
plt.xlim(0.001,1); plt.ylim(-1.0,1.0); plt.grid(); plt.legend(loc = 'lower right')

plt.subplots_adjust(left=0.0, bottom=0.0, right=1.5, top=1., wspace=0.2, hspace=0.2)
plt.show()
```

What do we see?

* for a very low lambda value, all features are included

* as we increase the lambda hyperparameter, total organic carbon is the first predictor feature to be removed

* then acoustic impedance, vitrinite reflectance, brittleness, log perm and finally porosity.

* at $\lambda \ge 0.8$ all features are removed.

Let's repeat this workflow with ridge regression for contrast.

```python
nbins = 5000                                                # number of bins to explore the hyperparameter
lams = np.logspace(-10,7,nbins)
ridge_coefs = np.ndarray((nbins,6))

index = 0
for lam in lams:
    ridge_reg = Ridge(alpha=lam)
    ridge_reg.fit(X_train, y_train) # fit model
    ridge_coefs[index,:] = ridge_reg.coef_
    index = index + 1

color = ['black','blue','green','red','orange','grey']
plt.subplot(111)
for ifeature in range(0,6):
    plt.semilogx(lams,ridge_coefs[:,ifeature], label = df_mv.columns[ifeature], c = color[ifeature], linewidth = 3.0)

plt.title('Standardized Model Coefficients vs. Lambda Hyperparameter'); plt.xlabel('Lambda Hyperparameter'); plt.ylabel('Standardized Model Coefficients')
plt.xlim(1.0e-10,1.0e7); plt.ylim(-1.0,1.0); plt.grid(); plt.legend(loc = 'lower right')

plt.subplots_adjust(left=0.0, bottom=0.0, right=1.5, top=1., wspace=0.2, hspace=0.2)
plt.show()
```

Ridge regression is quite different in the response of predictor feature to change in the lambda hyperparameter.

* there is no selective removal of predictor features as the lambda hyperparameter increases

* a major component is uniform shrinkage of all coefficients towards zero for $\lambda \in [10^1, 10^5]$

Let's check the mode performance for both models over the hyperparameters. We check the mean squared error and the variance explained for:

* linear regression

* ridge regression

* the lasso

```python
nbins = 1000                                               # number of bins
lams = np.logspace(-14,6,nbins)                           # assign equal bins in log space
ridge_error = []; lasso_error = []; linear_error = []
ridge_r2 = []; lasso_r2 = []; linear_r2 = []

linear_reg = LinearRegression()
linear_reg.fit(X_train, y_train) # fit model
linear_predict = linear_reg.predict(X_test)
linear_err = metrics.mean_squared_error(y_test["Production"],linear_predict)
linear_r = metrics.r2_score(y_test["Production"],linear_predict)

index = 0
for lam in lams:
    ridge_reg = Ridge(alpha=lam)
    ridge_reg.fit(X_train, y_train) # fit model
    ridge_predict = ridge_reg.predict(X_test)
    ridge_error.append(metrics.mean_squared_error(y_test["Production"],ridge_predict))
    ridge_r2.append(metrics.r2_score(y_test["Production"],ridge_predict))
    lasso_reg = Lasso(alpha=lam)
    lasso_reg.fit(X_train, y_train) # fit model
    lasso_predict = lasso_reg.predict(X_test)
    lasso_error.append(metrics.mean_squared_error(y_test["Production"],lasso_predict))
    lasso_r2.append(metrics.r2_score(y_test["Production"],lasso_predict))
    linear_error.append(linear_err)
    linear_r2.append(linear_r)
    index = index + 1

color = ['black','blue','green','red','orange','grey']

plt.subplot(121)
plt.semilogx(lams,linear_error, label = 'Linear Regression', c = 'black', linewidth = 3.0)
plt.semilogx(lams,ridge_error, label = 'Ridge Regression', c = 'blue', linewidth = 3.0)
plt.semilogx(lams,lasso_error, label = 'the Lasso', c = 'red', linewidth = 3.0)
plt.title('Testing MSE vs. Lambda Hyperparameter'); plt.xlabel('Lambda Hyperparameter'); plt.ylabel('Mean Square Error')
plt.xlim(1.0e-14,1.0e6); plt.ylim(0.0,1.0); plt.grid(); plt.legend(loc = 'lower right')

plt.subplot(122)
plt.semilogx(lams,linear_r2, label = 'Linear Regression', c = 'black', linewidth = 3.0)
plt.semilogx(lams,ridge_r2, label = 'Ridge Regression', c = 'blue', linewidth = 3.0)
plt.semilogx(lams,lasso_r2, label = 'the Lasso', c = 'red', linewidth = 3.0)
plt.title('Testing Variance Explained vs. Lambda Hyperparameter'); plt.xlabel('Lambda Hyperparameter'); plt.ylabel('Variance Explained (R2)')
plt.xlim(1.0e-14,1.0e6); plt.ylim(0.0,1.0); plt.grid(); plt.legend(loc = 'upper right')

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.5, wspace=0.2, hspace=0.2)
plt.show()
```

This is quite interesting. Let's make some observations.

* linear regression does not have the shrinkage term, nor the lambda parameter so the performance is constant

* for this problem linear regression is the best performer, lowest testing mean square error and higher testing variance explained

* at a low lambda value, ridge regression and the lasso approach linear regression

#### Making Predictions with Our Model

Here's the method to make predictions with our model for a specific lambda hyperparameter.

1. Instantiate and fit the model.

```python
lam = 0.05
lasso_reg = Lasso(alpha=lam)
lasso_reg.fit(X_train, y_train)                        # fit model
print(lasso_reg.coef_)
```

    [ 0.64745141  0.04203637 -0.          0.05811881  0.          0.00616774]

2. Standardize the predictor features with our StandardScalar object.

```python
por = 23.2; logperm = 1.2; AI = -0.4; Brittle = 45.2; TOC = 1.3; VR = 2.2; production = -9999.9
pred_features = np.array([por, logperm, AI, Brittle, TOC, VR, production]).reshape(1, -1)
spred_features = scaler.transform(pred_features)
```

3. Predict with the standardize predictor features.

```python
spred_features[0][6] = lasso_reg.predict(spred_features[0][:6].reshape(1, -1))
```

```python
predict = scaler.inverse_transform(spred_features[0])
predict
```

2. Now we can use the predictor features in our model (we will remove the production dummy).

```python
pred_features = np.array([por, logperm, AI, Brittle, TOC, VR, production]).reshape(1, -1)
scaler.transform(pred_features)
```

```python
np.array([por, logperm, AI, Brittle, TOC, VR]).reshape(1, -1)
```

```python
scaler
```

```python
df_mv.describe().transpose()
```

```python

```

```python
# Arrays to store the results
ncases = 100
lamd_mat = np.linspace(0.0,100.0,ncases)
density_model = np.linspace(1.2,2.4,10)
var_explained_train = np.zeros(ncases); var_explained_test = np.zeros(ncases)
mse_train = np.zeros(ncases); mse_test = np.zeros(ncases)

for ilam in range(0,len(lamd_mat)):                         # Loop over all lambda values
    ridge_reg = Ridge(alpha=lamd_mat[ilam])
    ridge_reg.fit(df_train["Density"].values.reshape(n_train,1), df_train["Porosity"]) # fit model

    porosity_model = ridge_reg.predict(density_model.reshape(10,1)) # predict with the fit model
    porosity_pred_train = ridge_reg.predict(df_train['Density'].values.reshape(n_train,1)) # predict with the fit model
    var_explained_train[ilam] = r2_score(df_train['Porosity'].values, porosity_pred_train)
    mse_train[ilam] = mean_squared_error(df_train['Porosity'].values, porosity_pred_train)

    porosity_pred_test = ridge_reg.predict(df_test['Density'].values.reshape(n_test,1))
    var_explained_test[ilam] = r2_score(df_test['Porosity'].values, porosity_pred_test)
    mse_test[ilam] = mean_squared_error(df_test['Porosity'].values, porosity_pred_test)

    if ilam <= 7:
        plt.subplot(4,2,ilam+1)
        plt.scatter(df_train["Density"].values, df_train["Porosity"],  color='black', s = 20, alpha = 0.3)
        plt.plot(density_model,porosity_model, color='red', linewidth=1)
        plt.title('Ridge Regression Porosity from Density with Training Data - Lambda = ' + str(round(lamd_mat[ilam],2))); plt.xlabel('Density (g/cm^3)'); plt.ylabel('Porosity (%)')
        plt.xlim(1.,2.6); plt.ylim(5,24)

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=4.2, wspace=0.2, hspace=0.3)
plt.show()
```

We can observed from the first 8 cases above of ridge regression model fit that increase in the lambda hyper parameter decreases the slope of the linear fit.

Let's plot the MSE and variance explained over training and testing datasets.

```python
plt.subplot(121)
plt.plot(lamd_mat, var_explained_train,  color='blue', linewidth = 2, label = 'Training')
plt.plot(lamd_mat, var_explained_test,  color='red', linewidth = 2, label = 'Test')
plt.title('Variance Explained vs. Lambda'); plt.xlabel('Lambda'); plt.ylabel('Variance Explained')
plt.xlim(0.,100.); plt.ylim(0,1.0)
plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.0, wspace=0.2, hspace=0.2)
plt.legend()

plt.subplot(122)
plt.plot(lamd_mat, mse_train,  color='blue', linewidth = 2, label = 'Training')
plt.plot(lamd_mat, mse_test,  color='red', linewidth = 2, label = 'Test')
plt.title('MSE vs. Lambda'); plt.xlabel('Lambda'); plt.ylabel('Mean Square Error')
plt.xlim(0.,100.); plt.ylim(0,10.0)
plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.0, wspace=0.2, hspace=0.2)
plt.legend()

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.2, wspace=0.2, hspace=0.3)
plt.show()
```

We observe that as we increase the lambda parameter the variance explained decreases and the mean square error increases.

* this makes sense as the data has a consistent linear trend and as the slope 'shrinks' to zero the error increases and the variance explained decreases

* there could be other cases where the reduced slope actually performs better in testing.  For example with sparce and noisy data.

#### Model Variance

Now let's explore the concept of model variance, an important part of machine learning accuracy in testing.

* the sensitivity of the model to the specfic training data

* as lambda increases the sensitivity to the training data, model variance decreases

Let's demonstrate this with this workflow:

* loop over multiple lambda values
    * loop over multiple bootstrap samples of the data
        * calculate the ridge regression fit (slope)
    * calculate the variance of these bootstrap results

```python
L = 200                                                     # the number of bootstrap realizations
nsamples = 20                                               # the number of samples in each bootstrap realization
nlambda = 100                                               # number of lambda values to evaluate

coef_mat = np.zeros(L)                                      # declare arrays to store the results
variance_coef = np.zeros(nlambda)

lamd_mat = np.linspace(0.0,100.0,nlambda)
df = pd.read_csv("Density_Por_data.csv")
for ilam in range(0,len(lamd_mat)):                         # loop over all lambda values
    for l in range(0, L):                                   # loop over all bootstrap realizations
        df_sample = df.sample(n = nsamples)                 # random sample (1 bootstrap)
        ridge_reg = Ridge(alpha=lamd_mat[ilam])             # instatiate model
        ridge_reg.fit(df_sample["Density"].values.reshape(nsamples,1), df_sample["Porosity"]) # fit model
        coef_mat[l] = ridge_reg.coef_[0]                    # get the slope parameter

    variance_coef[ilam] = np.var(coef_mat)                  # calculate the variance of the slopes over the L bootstraps
```

Now let's plot the result.

```python
plt.subplot(111)
plt.plot(lamd_mat, variance_coef,  color='black', linewidth = 2, label = 'Slope Variance')
plt.title('Model Fit Variance vs. Lambda'); plt.xlabel('Lambda'); plt.ylabel('Model Fit Variance')
plt.xlim(0.,100.); plt.ylim(0.001,10.0); plt.yscale('log')
plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.0, wspace=0.2, hspace=0.2)
plt.legend()
```

The result is as expected, with increase in lambda hyperparameter the sensitivity of the model to the training data is decreased.

#### k-fold Cross Validation

It would be useful to conduct a complete k-fold validation to evaluate the testing error vs. the hyperparameter lambda for model tuning.

* the following code should do this

* but with a single feature as input for fitting the fit function requires a reshape

```python
my_array.reshape((nsample,1))
```

* this is not included in the scikit learn function 'cross_val_score' so we will skip this for now

I have left the code commented out below for reference:

```python
#score = []                                                  # code modified from StackOverFlow by Dimosthenis
#nlambda = 1
#lambd_mat = np.linspace(0.0,100.0,nlambda)
#for ilam in range(0,nlambda):
#    ridge_reg = Ridge(alpha=lambd_mat[ilam])
#    scores = cross_val_score(estimator=ridge_reg, X= df['Density'].values, y=df['Porosity'].values, cv=10, n_jobs=4, scoring = "neg_mean_squared_error") # Perform 10-fold cross validation
#    score.append(abs(scores.mean()))
```

#### Comments

Ridge regression is a variant of linear regression that includes a hyperparameter to constrain the degree of model fit.  This allow us to tune the variance-bias trade-off of our model. I hope this was helpful,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin
On twitter I'm the @GeostatsGuy.

***

#### More on Michael Pyrcz and the Texas Center for Geostatistics:

### Michael Pyrcz, Associate Professor, University of Texas at Austin
*Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions*

With over 17 years of experience in subsurface consulting, research and development, Michael has returned to academia driven by his passion for teaching and enthusiasm for enhancing engineers' and geoscientists' impact in subsurface resource development.

For more about Michael check out these links:

#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

#### Want to Work Together?

I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.

* Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I'd be happy to drop by and work with you!

* Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!

* I can be reached at mpyrcz@austin.utexas.edu.

I'm always happy to discuss,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin

#### More Resources Available at: [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

```python

```


# A simple demonstration of coregionalisation

This notebook shows how to construct a multi-output GP model using GPflow. We will consider a regression problem for functions $f: \mathbb{R}^D \rightarrow \mathbb{R}^P$. We assume that the dataset is of the form $(X_1, f_1), \dots, (X_P, f_P)$, that is, we do not necessarily observe all the outputs for a particular input location (for cases where there are fully observed outputs for each input, see [Multi-output Gaussian processes in GPflow](./multioutput.ipynb) for a more efficient implementation).

For this problem, we model $f$ as a *coregionalised* Gaussian process, which assumes a kernel of the form:

$$\textrm{cov}(f_i(x), f_j(y)) = k_1(x, y) \times B[i, j].$$

The covariance of the $i$th function at $x$ and the $j$th function at $y$ is a kernel applied at $x$ and $y$, times the $i, j$th entry of a positive definite matrix $B$. This is known as the **intrinsic model of coregionalisation (ICM)** (Bonilla and Williams, 2008).

To make sure that B is positive-definite, we parameterise it as:

$$B = W W^\top + \textrm{diag}(\kappa).$$

To build such a model in GPflow, we need to perform the following two steps:

 * Create the kernel function defined previously, using the `Coregion` kernel class.
 * Augment the training data X with an extra column that contains an integer index to indicate which output an observation is associated with. This is essential to make the data work with the `Coregion` kernel.

```python
import gpflow
import numpy as np
import matplotlib
%matplotlib inline

matplotlib.rcParams['figure.figsize'] = (12, 6)
plt = matplotlib.pyplot
from gpflow.test_util import notebook_niter
np.random.seed(123)
```

## Data preparation
We start by generating some training data to fit the model with. For this example, we choose the following two correlated functions for our outputs:
$$
\begin{align}
y_1 &= \sin(6x) + \epsilon_1, \qquad \epsilon_1 \sim \mathcal{N}(0, 0.009) \\
y_2 &= \sin(6x + 0.7) + \epsilon_2, \qquad \epsilon_2 \sim \mathcal{N}(0, 0.01) \\
\end{align}
$$

```python
# make a dataset with two outputs, correlated, heavy-tail noise. One has more noise than the other.
X1 = np.random.rand(100, 1) # Observed locations for first output
X2 = np.random.rand(50, 1) * 0.5 # Observed locations for second output

Y1 = np.sin(6*X1) + np.random.randn(*X1.shape) * 0.03
Y2 = np.sin(6*X2+ 0.7) + np.random.randn(*X2.shape) * 0.1

plt.figure(figsize=(8, 4))
plt.plot(X1, Y1, 'x', mew=2)
plt.plot(X2, Y2, 'x', mew=2);
```

## Data formatting for the coregionalised model
We add an extra column to our training dataset that contains an index that specifies which output is observed.

```python
# Augment the input with ones or zeros to indicate the required output dimension
X_augmented = np.vstack((np.hstack((X1, np.zeros_like(X1))), np.hstack((X2, np.ones_like(X2)))))

# Augment the Y data
Y_augmented = np.vstack((np.hstack((Y1, np.zeros_like(Y1))), np.hstack((Y2, np.ones_like(Y2)))))
```

## Building the coregionalisation kernel
We build a coregionalisation kernel with the Matern 3/2 kernel as the base kernel. This acts on the leading ([0]) data dimension of the augmented X values. The `Coregion` kernel indexes the outputs, and acts on the last ([1]) data dimension (indices) of the augmented X values. To specify these dimensions, we use the built-in `active_dims` argument in the kernel constructor. To construct the full multi-output kernel, we take the product of the two kernels (for a more in-depth tutorial on kernel combination, see [Manipulating kernels](./kernels.ipynb)).

```python
output_dim = 2 # Number of outputs
rank = 1 # Rank of W
k1 = gpflow.kernels.Matern32(1, active_dims=[0]) # Base kernel
coreg = gpflow.kernels.Coregion(1, output_dim=output_dim, rank=rank, active_dims=[1]) # Coregion kernel
coreg.W = np.random.rand(output_dim, rank) # Initialise the W matrix
kern = k1 * coreg
```

**Note:** By default, the `W` matrix is initialised with zeros; however, this is a saddle point in the objective, so the value of `W` is not optimised to fit the data. Hence, re-initialising the matrix to random entries should give a more accurate result.

## Constructing the model
The final element in building the model is to specify the likelihood for each output dimension. To do this, use the `SwitchedLikelihood` object in GPflow.

```python
# This likelihood switches between Gaussian noise with different variances for each f_i:
lik = gpflow.likelihoods.SwitchedLikelihood([gpflow.likelihoods.Gaussian(), gpflow.likelihoods.Gaussian()])

# now build the GP model as normal
m = gpflow.models.VGP(X_augmented, Y_augmented, kern=kern, likelihood=lik, num_latent=1)
# Here we specify num_latent=1 to avoid getting two outputs when predicting as Y_augmented is 2-dimensional

# fit the covariance function parameters
gpflow.train.ScipyOptimizer().minimize(m, maxiter=notebook_niter(1000))
```

That's it: the model is trained. Let's plot the model fit to see what's happened.

```python
def plot_gp(x, mu, var, color, label):
    plt.plot(x, mu, color=color, lw=2, label=label)
    plt.fill_between(x[:, 0],
                     (mu - 2*np.sqrt(var))[:, 0],
                     (mu + 2*np.sqrt(var))[:, 0],
                     color=color, alpha=0.4)

def plot(m):
    plt.figure(figsize=(8, 4))
    xtest = np.linspace(0, 1, 100)[:,None]
    line, = plt.plot(X1, Y1, 'x', mew=2)
    mu, var = m.predict_f(np.hstack((xtest, np.zeros_like(xtest))))
    plot_gp(xtest, mu, var, line.get_color(), 'Y1')

    line, = plt.plot(X2, Y2, 'x', mew=2)
    mu, var = m.predict_f(np.hstack((xtest, np.ones_like(xtest))))
    plot_gp(xtest, mu, var, line.get_color(), 'Y2')

    plt.legend()

plot(m)
```

From the plots, we can see:

 - The first function (blue) has low posterior variance everywhere because there are so many observations, and the noise variance is small.
 - The second function (orange) has higher posterior variance near the data, because the data are more noisy, and very high posterior variance where there are no observations (x > 0.5).
 - The model has done a reasonable job of estimating the noise variance and lengthscales.
 - The model recognises the correlation between the two functions and is able to suggest (with uncertainty) that because x > 0.5 the orange curve should follow the blue curve (which we know to be the truth from the data-generating procedure).

The covariance matrix between outputs is as follows:

```python
B = coreg.W.value @ coreg.W.value.T + np.diag(coreg.kappa.value)
print('B =', B)
plt.imshow(B);
```

## References

Bonilla, Edwin V., Kian M. Chai, and Christopher Williams. "Multi-task Gaussian process prediction." _Advances in neural information processing systems_. 2008.


# Module 2: Inversion

In the previous module we started with a continuous distribution of a physical property and discretized it into many cells, then we performed a forward simulation that created data from known model parameters. Inversion, of course, is exactly the opposite process. Imagine each model parameter that we had represents a layer in a 1D layered earth. At the surface of the earth we measure the data, and when we invert we do so for the model parameters. Our goal is to take the observed data and recover models that emulate the real Earth as closely as possible.

You may have noticed that the act of discretizing our problem created more cells than data values. In our latter example we produced 20 data points from 1000 model parameters, which is only a few data points and many model parameters. While this was not much of a problem in the forward simulation, when we want to do the inverse process, that is, obtain the model parameters from the data, it is clear that we have many more unknowns than knowns. In short, we have an underdetermined problem, and therefore infinite possible solutions. In mathematical terms, geophysical surveys represent what are called "ill-posed" problems.

An "ill-posed" problem is any problem that does not satisfy the requirements for the definition of "well-posed" problem. A *well-posed* problem is a problem in mathematics that must satisfy all three of the following criteria:

 A solution exists.

 The solution is unique.

 The solution's behaviors change continuously with continuously changing initial conditions.

Any mathematical formulation that does not satisfy all three of the above is, by definition, an ill-posed problem. Since we are dealing with an underdetermined system, I hope that it is clear that we are dealing with an ill-posed problem (i.e., we have no unique solution), and we are going to have to come up with  a method (or methods) that can help us choose from the available solutions. We need to devise an algorithm that can choose the "best" model from the infinitely many that are available to us.

In short, we are going to have to find an optimum model. More specifically, in the context of most geophysics problems, we are going to use gradient-based optimization. This process involves building an *objective function*, which is a function that casts our inverse problem as an optimization problem. We will build an objective function consisting of two parts:

$$
\phi = \phi_d + \beta \phi_m
$$

Where the terms on the right hand side are (1) a data misfit (denoted as $\phi_d$) and (2) a model regularization (denoted as $\phi_m$). These two parts will be elaborated in detail below.

Once we have formulated the objective function, we will take derivatives and obtain a recovered model. This module will flesh out the details of the model objective function, and then take first and second derivatives to derive an expression that gives us a solution for our model parameters.

## The Data Misfit, $\phi_d$

A *misfit* describes how close synthetic data matches measurements that are made in the field. Traditionally this term refers to the difference between the measured data and the predicted data. If these two quantities are sufficiently close, then we consider the model to be a viable candidate for the solution to our problem. Because the data are inaccurate, a model that reproduces those data exactly is not feasible. A realistic goal, rather, is to find a model whose predicted data are consistent with the errors in the observations, and this requires incorporating knowledge about the noise and uncertainties. The concept of fitting the data means that some estimate of the “noise” be available. Unfortunately “noise” within the context of inversion is everything that cannot be accounted for by a compatible relationship between the model and the data. More specifically, noise refers to (1) noise from data aquisition in the field, (2) uncertainty in source and receiver locations, (3) numerical error, (4) physical assumptions about our model that do not capture all of the physics.

A standard approach is to assume that each datum, $d_i$, contains errors that can be described as Gaussian with a standard deviation $\epsilon_i$. It is important to give a significant amount of thought towards assigning standard deviations in the data, but a reasonable starting point is to assign each $\epsilon_i$ as $\epsilon_i= floor +\%|d_i|$.

Incorporating both the differences between predicted and measured data and a measure of the uncertainties in the data yields our misfit function, $\phi_d$:

$$
\phi_d (m) = \frac{1}{2} \sum_{i=1}^N \left( \frac{F[m] -d_i^{obs} }{\epsilon_i}\right)^2 = \frac{1}{2} \|W_d(F[m] - d^{obs}) \|_2^2
$$

Note that the right hand size of the equation is written as a matrix-vector product, with each $\epsilon_i$ in the denominator placed as elements on a diagonal matrix $W_d$, as follows:

\begin{equation}
\begin{split}
W_d =
\begin{bmatrix}
    \frac{1}{\epsilon_1} & 0 & 0 & \cdots & 0\\
    0 & \frac{1}{\epsilon_2} & 0 & \cdots & 0\\
    0 & 0 & \frac{1}{\epsilon_3} & \cdots & \vdots\\
    0 & 0 & 0 & \ddots & \frac{1}{\epsilon_M}\\
\end{bmatrix}
\end{split}
\end{equation}

If we return to linear problem from the previous section where our forward operator was simply a matrix of kernel functions, we can substitute $F[m]$ with $G$ and obtain
$$
\phi_d (m) = \frac{1}{2} \sum_{i=1}^N \left( \frac{(Gm)_i -d_i^{obs} }{\epsilon_i}\right)^2 = \frac{1}{2} \|W_d(Gm - d^{obs}) \|_2^2
$$

Now that we have defined a measure of misfit, the next task is to determine a tolerance value, such that if the misfit is about equal to that value, then we have an acceptable fit. Suppose that the standard deviations are known and that errors are Gaussian, then $\phi_d$ becomes a $\chi_N^2$ variable with $N$ degrees of freedom. This is a well-known quantity with an expected value $E[\chi_N^2]=N$ and a standard deviation of $\sqrt{2N}$. Basically, what this means is that computing $\phi_d$ should give us a value that is close to the number of data, $N$.

## The Model Regularization, $\phi_m$

There are many options for choosing a model regularization, but the goal in determining a model regularization is the same: given that we have no unique solution, we must make assumptions in order to recast the problem in such a way that a solution exists. A general function used in 1D is as follows:

$$
\phi_m = \alpha_s \int (m)^2 dx + \alpha_x \int \left( \frac{dm}{dx} \right)^2 dx
$$

Each term in the above expression is a norm that measures characteristics about our model. The first term is a representation of the square of the Euclidean length for a continuous function, and therefore measures the length of the model, while the second term uses derivative information to measure the model's smoothness. Usually  the model regularization is defined with respect to a reference model. In the above, the reference model would simply be zero, but choosing a non-zero reference model $m_{ref}$, yields the following:
$$
\phi_m = \alpha_s \int (m-m_{ref})^2 dx + \alpha_x \int \left( \frac{d}{dx} (m-m_{ref}) \right)^2 dx
$$

As before, we will discretize this expression. It is easiest to break up each term and treat them separately, at first.
We will denote each term of $\phi_m$ as $\phi_s$ and $\phi_x$, respectively. Consider the first term. Translating the integral to a sum yields:

$$
\phi_s = \alpha_s \int (m)^2 dx \rightarrow \sum_{i=1}^N \int_{x_{i-1}}^{x_i} (m_i)^2 dx = \sum_{i=1}^N m_i^2 (x_i - x_{i-1})
$$

Each spatial "cell" is $x_i - x_{i-1}$, which is the distance between nodes, as you may recall from the previous module. To simplify notation, we will use $\Delta x_{n_i}$ to denote the *ith* distance between nodes:

We can then write $\phi_s$ as:

$$
\phi_s = \alpha_s \sum_{i=1}^N m_i^2 \Delta x_{n_i} = \alpha_s m^T W_s^T W_s m = \alpha_s \|W_s m\|_2^2
$$

with:

\begin{equation}
\begin{split}
W_s =
\begin{bmatrix}
    {\sqrt{\Delta x_{n_1}}} & 0 & 0 & \cdots & 0\\
    0 & {\sqrt{\Delta x_{n_2}}} & 0 & \cdots & 0\\
    0 & 0 & {\sqrt{\Delta x_{n_3}}} & \cdots & \vdots\\
    0 & 0 & 0 & \ddots & {\sqrt{\Delta x_{n_N}}}\\
\end{bmatrix}
\end{split}
\end{equation}

For the second term, we will do a similar process. First, we will delineate $\Delta x_{c_i}$ as the distance between cell centers:

A discrete approximation to the integral can be made by evaluating the derivative of the model based on how much it changes between the cell-centers, that is, we will take the average gradient between the *ith* and *i+1th* cells:

$$
\phi_x = \alpha_x \int \left( \frac{dm}{dx} \right)^2 dx \rightarrow \sum_{i=1}^{N-1} \left( \frac{m_{i+1}-m_i}{h_k}\right) \Delta x_{c_i} = m^T W_x^T W_x m = \|W_x m\|_2^2
$$

The matrix $W_x$ is a finite difference matrix constructed thus:

\begin{equation}
\begin{split}
D_x =
\begin{bmatrix}
    -\frac{1}{{\Delta x_{c_1}}} & \frac{1}{{\Delta x_{c_1}}} & 0 & \cdots & 0\\
    0 & -\frac{1}{{\Delta x_{c_2}}} & \frac{1}{{\Delta x_{c_2}}} & \cdots & 0\\
    0 & 0 & \ddots & \ddots & \vdots\\
    0 & 0 & 0 &  -\frac{1}{{\Delta x_{c_{M-1}}}} & \frac{1}{{\Delta x_{c_{M-1}}}}\\
\end{bmatrix}
\end{split}
\end{equation}

and then we need to account for the integration, so we multiply by a diagonal matrix $\rm diag(\sqrt{v})$

\begin{equation}
W_x = D_x  \rm diag(\sqrt{v})
\end{equation}

So to summarize, we have $\phi_m = \phi_s + \phi_x$ with

\begin{equation}
\begin{split}
    \phi_m & = \phi_s + \phi_x \\[0.4em]
           & = \alpha_s \|W_s (m - m_{ref})\|_2^2 + \alpha_x \|W_x (m - m_{ref})\|_2^2 \\[0.4em]
\end{split}
\end{equation}

Next, we will write this more compactly by stacking $W_s$ and $W_x$ into a matrix $W_m$ as follows

\begin{equation}
\begin{split}
W_m =
\begin{bmatrix}
    \sqrt{\alpha_s} W_s\\
    \sqrt{\alpha_x} W_x
\end{bmatrix}
\end{split}
\end{equation}

## Model Objective Function

If we go back and recall what was discussed in the introduction, the model objective function casts the inverse problem as an optimization problem, and as mentioned, we will be using gradient-based optimization, so we will need to take derivatives. The complete model objective function that we are dealing with will contain both the data misfit and the model regularization. This means that we can write it as $\phi$ as the sum of the two and then differentiate:

$$
\phi = \phi_d + \beta \phi_m
$$
For the linear problem we are considering
$$
\phi_d = \frac{1}{2}\| W_d (Gm-d^{obs})\|_2^2 = \frac{1}{2}(Gm-d^{obs})^T W_d^T W_d (Gm-d^{obs})
$$
and
$$
\phi_m = \frac{1}{2} \|W_m (m-m_{ref}) \|^2_2 = \frac{1}{2}(m-m_{ref})^T W_m^T W_m (m-m_{ref})
$$

To simplify the terms and see the math a little more clearly, let's note that $W_d(Gm-d^{obs})$, and $\beta W_m(m-m_{ref})$ are simply vectors. And since we are taking the square of the 2-norm, all that we are really doing is taking the dot product of each vector with itself. So let $z=W_d(Gm-d^{obs})$, and let $y=W_m(m-m_{ref})$ where both $z$ and $y$ vectors are functions of $m$. So then:

$$
\phi_d =  \frac{1}{2}\|z\|_2^2 = \frac{1}{2}z^T z
$$

$$
\phi_m =  \frac{1}{2}\|y\|_2^2 =\frac{1}{2}y^T y
$$

To minimize this, we want to look at $\nabla \phi$. Using our compact expressions:
$$
\phi = \phi_d + \beta \phi_m = \frac{1}{2}z^Tz + \beta \frac{1}{2}y^Ty \\
$$

Taking the derivative with respect to $m$ yields:
\begin{equation}
\begin{split}
\frac{d \phi}{dm}&  = \frac{1}{2}  \left(z^T \frac{dz}{dm} +  z^T \frac{dz}{dm} + \beta y^T \frac{dy}{dm} + \beta y^T \frac{dy}{dm}\right)\\\\[0.6em]
& = z^T \frac{dz}{dm} + \beta y^T \frac{dy}{dm}
\end{split}
\end{equation}

Note that
$$\frac{dz}{dm} = \frac{d}{dm}(W_d(Gm-d^{obs})) = W_d G $$

and

$$ \frac{dy}{dm} = \frac{d}{dm}(W_m (m-m_{ref})) = W_m $$

Next, let's substitute both derivatives, our expressions for $z$ and $y$, apply the transposes, and rearrange:

\begin{equation}
\begin{split}
\frac{d \phi}{dm} & = z^T \frac{dz}{dm} + \beta y^T \frac{dy}{dm} \\[0.6em]
 & = (W_d(Gm-d^{obs}))^T W_d G + \beta (W_m (m-m_{ref}))^T W_m\\[0.6em]
 & = (Gm-d^{obs})^T W_d^T W_d G + \beta (m-m_{ref})^T W_m^T W_m \\[0.6em]
 & = ((Gm)^T - d^T) W_d^T W_d G + \beta (m^T-m_{ref}^T)W_m^T W_m \\[0.6em]
 & = (m^T G^T - d^T) W_d^T W_d G + \beta m^T W_m^T W_m - \beta  m_{ref}^T W_m^T W_m \\[0.6em]
 & = m^T G^T W_d^T W_d G  - d^T W_d^T W_d G + \beta m^T W_m^T W_m - \beta  m_{ref}^T W_m^T W_m\\[0.6em]
 & = m^T G^T W_d^T W_d G  + \beta m^T W_m^T W_m - d^T W_d^T W_d G - \beta  m_{ref}^T W_m^T W_m
 \end{split}
\end{equation}

Now we have an expression for the derivative of our equation that we can work with. Setting the gradient to zero and gathering like terms gives:

\begin{equation}
\begin{split}
 m^T G^T W_d^T W_d G  + \beta m^T W_m^T W_m = d^T W_d^T W_d G + \beta  m_{ref}^T W_m^T W_m\\[0.6em]
 (G^T W_d^T W_d G  + \beta W_m^T W_m)m = G^T W_d^T W_d d + \beta  W_m^T W_m m_{ref}\\[0.6em]
\end{split}
\end{equation}

From here we can do two things. First, we can solve for $m$, our recovered model:

\begin{equation}
\begin{split}
 m = (G^T W_d^T W_d G  + \beta W_m^T W_m)^{-1} (G^T W_d^T W_d d + \beta  W_m^T W_m m_{ref})\\[0.6em]
\end{split}
\end{equation}

Second, we can get the second derivative simply from the bracketed terms on the left hand side of the equation above:
\begin{equation}
\frac{d^2 \phi}{dm^2} = G^T W_d^T W_d G  + \beta W_m^T W_m
\end{equation}

In the model problem that we are solving, second derivative information is not required to obtain a solution, however, in non-linear problems or situations when higher order information is required, it is useful to have this available when we need it.

## Solving for $m$ in Python

Before we solve for $m$, we will recreate what we had in the first module. First, install the appropriate packages:

```python
# Import Packages
import numpy as np
import matplotlib.pyplot as plt
```

Here is the model that we had previously:

```python
# Begin by creating a ficticious set of model data

n_cells = 1000  # Set number of model parameters
n_nodes = n_cells + 1
xn = np.linspace(0, 1, n_nodes) # Define 1D domain on nodes
xc = 0.5*(xn[1:] + xn[:-1])  # Define 1D domain on cell centers
```

```python
# Define Gaussian function:
def gauss(x, amplitude, mean, std):
    """Define a gaussian function"""
    return amplitude * np.exp(-((x-mean)/std)**2 / 2)

# Choose parameters for Gaussian, evaluate, and store in an array, f.
std = 6e-2
mean = 0.7
amplitude_gaussian = 1
gaussian = gauss(xc, amplitude_gaussian, mean, std)

fig, ax = plt.subplots(1, 1)
ax.plot(xc, gaussian)
ax.set_title("Gaussian")
```

```python
# Define a boxcar function:
x_boxcar = np.r_[0.2, 0.35]
amplitude_boxcar = 1
boxcar = np.zeros(n_cells)  # initialize an array of all zeros
boxcar_inds = (xc >= x_boxcar.min()) & (xc <= x_boxcar.max())  # find the indices of the boxcar
boxcar[boxcar_inds] = amplitude_boxcar

# construct the model
mtrue = gaussian + boxcar

# Plot
fig, ax = plt.subplots(1, 1)
ax.plot(xc, mtrue)
ax.set_xlabel('x')
ax.set_ylabel('m(x)')
ax.set_title('Model, $m(x)$')
```

Again, we define out kernel functions and averaging and volume matrices as before:

```python
# Make the set of kernel functions

def kernel_functions(x, j, p, q):
    return np.exp(-p*j*x) * np.cos(2*np.pi*q*j*x)

p = 0.01  # Set values for p, q
q = 0.15
n_data = 20  # specify number of output data

j_min = 0
j_max = n_data
j_values = np.linspace(j_min, j_max, n_data)

Gn = np.zeros((n_nodes, n_data))

for i, j in enumerate(j_values):
    Gn[:, i] = kernel_functions(xn, j, p, q)

# Plot
fig, ax = plt.subplots(1, 1)
ax.plot(xn, Gn)
ax.set_xlabel('x')
ax.set_ylabel('g(x)')
ax.set_title('Kernel functions, $g(x)$')
```

```python
# Make Averaging Matrix
Av = np.zeros((n_cells, n_nodes)) # Create a matrix of zeros of the correct dimensions

# and fill in with elements usin the loop below (note the 1/2 is included in here).
for i in range(n_cells):
    Av[i, i] = 0.5
    Av[i, i+1] = 0.5

print(Av.shape)
```

    (1000, 1001)

```python
# make the Volume, "delta x" array
delta_x = np.diff(xn)  # set x-spacings
V = np.diag(delta_x)  # create diagonal matrix

print(V.shape)
```

    (1000, 1000)

Last, we produce our data:

```python
G = Gn.T @ Av.T @ V
```

```python
d = G @ mtrue
```

```python
# Plot
fig, ax = plt.subplots(1, 1)
ax.plot(d, '-o')
ax.set_title('Synthetic Data $d$')
```

## Introducing noise to the data

This is where we stood at the end of the last module. Next, to simulate taking data in the field, we are going to add a noise to the data before we perform our inversion. We will do this by defining a lambda function that assigns a floor value and percent scaling factor. Also, we will assume that the noise is Gaussian. We then add the noise to the original data to make a simulated vector of observed data. The superposition of our noise and original data is plotted below.

```python
# Add noise to our synthetic data

add_noise = False  # set to true if you want to add noise to the data

if add_noise is True:
    relative_noise = 0.04
    noise_floor = 1e-2
    noise = (
        relative_noise * np.random.randn(n_data) * np.abs(d) +  # percent of data
        noise_floor * np.random.randn(n_data)
    )
    dobs = d + noise
else:
    dobs = d

fig, ax = plt.subplots(1, 1)
ax.plot(d, '-o', label="d clean")
ax.plot(dobs, '-s', label="dobs")
ax.set_title("synthetic data")
ax.legend()
```

# Setting up the inverse problem

Now we will assemble the pieces for constructing an objective function to be minimized in the inversion.

Throughout we use L2 norms, so the first thing we will do is define a simple function for computing a weighted L2 norm.

```python
def weighted_l2_norm(W, v):
    """
    A function that returns a weighted L2 norm. The parameter W is a weighting matrix
    and v is a vector.
    """
    Wv = W @ v
    return Wv.T @ Wv
```

## Calculating $\phi_d$

We are now in a position to build up the data misfit term, $\phi_d$. We will need a function to compute the 2-norm, so constructing a function to do this is useful. Next we will make the matrix $W_d$, which is a diagonal matrix that contains the inverses of the uncertainty in our data. Again, we will define a floor and percent error for each datum. Last, we calculate $\phi_d$ using our 2-norm function that we created. It is insightful to see what values have been assigned to our floor and misfit, so they are printed below.

```python
# Calculate the data misfit, phi_d

noise_floor = 1e-3
relative_error = 0.05
standard_deviation = noise_floor + relative_error * np.abs(dobs)

# construct Wd
Wd = np.diag(1/standard_deviation)

fig, ax = plt.subplots(1, 1)
img = ax.imshow(Wd, "Greys")
plt.colorbar(img, ax=ax)
ax.set_title("Wd")
```

## Calculating $\phi_m$

As discussed above, we are going to first need to make our $W_m$ matrix, which is a partitioned matrix from two other matrices, $W_s$ and $W_x$, each scaled by a separate parameter $\alpha_s$ and $\alpha_x$. We are going to discuss the manner in which $\alpha_s$ and $\alpha_x$ are selected in more detail during the next module. But for the moment, we will set them as they are defined below. Once this matrix is built up, calculating $\phi_m$ is a simple matter, given that we have made a function to compute the 2-norm already. For the sake of illustration, I compute and print $\phi_m$ from the residual of our reference model $m_{ref}$ and our true model $m$. However, of interest to us will be the residual of the model that we recover $m_{rec}$ and our reference model.

```python
# Start with Ws
sqrt_vol = np.sqrt(delta_x)   # in 1D - the "Volume" = length of each cell (delta_x)
Ws = np.diag(sqrt_vol)

# and now Wx
Dx = np.zeros((n_cells-1, n_cells))  # differencing matrix

for i, dx in enumerate(delta_x[:-1]):
    Dx[i, i] = -1/dx
    Dx[i, i+1] = 1/dx

Wx = Dx @ np.diag(sqrt_vol)

print(Ws.shape, Wx.shape)
```

    (1000, 1000) (999, 1000)

```python
# plot both
fig, ax = plt.subplots(1, 2, figsize=(10, 4))
plot_up_to = 10 # plot 10 entries

# plot Ws
img = ax[0].imshow(Ws[:plot_up_to, :plot_up_to], "Greys")
plt.colorbar(img, ax=ax[0])
ax[0].set_title("Ws")

# plot Wx
img = ax[1].imshow(Wx[:plot_up_to, :plot_up_to+1], "bwr")
plt.colorbar(img, ax=ax[1])
ax[1].set_title("Wx")

plt.tight_layout()
```

### Stack Ws, Wx to make a single regularization matrix Wm

```python
alpha_s = 1e-6
alpha_x = 1

Wm = np.vstack([
    np.sqrt(alpha_s)*Ws,
    np.sqrt(alpha_x)*Wx
])

print(Wm.shape)
```

    (1999, 1000)

## Inverting for our recovered model

At last we can invert to find our recovered model and see how it compares with the true model. First we will assign a value for $\beta$. As with the $\alpha$ parameters from before, we will assign  a value, but the choice of beta will be a topic that we explore more fully in the next module. Once our $\beta$ value is assigned, we will define yet another lambda function to obtain the recovered model, plot it against our true model, and then output our results for $\phi_d$ and $\phi_m$.

```python
beta = 1e-1  # Set beta value
mref = 0.5 * np.ones(n_cells)  # choose a reference model

WdG = Wd @ G
mrec = (
    np.linalg.inv(WdG.T @ WdG + beta * Wm.T @ Wm) @
    (WdG.T @ Wd @ dobs + beta * Wm.T @ Wm @ mref)
)
```

```python
fig, ax = plt.subplots(1, 1)

ax.plot(xc, mtrue, label="true")
ax.plot(xc, mrec, label="recovered")
ax.legend()
```

```python
dpred = G @ mrec

fig, ax = plt.subplots(1, 1)
ax.plot(dobs, '-o', label="observed")
ax.plot(dpred, '-s', label="predicted")
ax.legend()
```

This concludes the current module. For the next module, we will examine the constraints on our choice for $\alpha_s$ and $\alpha_x$, and then introduce the Tikhonov curve and a method for choosing $\beta$.


# Lagrangian mechanics in generalized coordinates

> Marcos Duarte
> Laboratory of Biomechanics and Motor Control ([http://demotu.org/](http://demotu.org/))
> Federal University of ABC, Brazil

## Generalized coordinates

The direct application of Newton's laws to mechanical systems results in a set of equations of motion in terms of Cartesian coordinates of each of the particles that make up the system. In many cases, this is not the most convenient coordinate system to solve the problem or describe the movement of the system. For example, in problems involving many particles, it may be convenient to choose a system that includes the coordinate of the center of mass. Another example is a serial chain of rigid links, such as a member of the human body or from a robot manipulator, it may be simpler to describe the positions of each link by the angles between links.

Coordinate systems such as these are referred as [generalized coordinates](https://en.wikipedia.org/wiki/Generalized_coordinates). Generalized coordinates uniquely specify the positions of the particles in a system. Although there may be several generalized coordinates to describe a system, usually a judicious choice of generalized coordinates provides the minimum number of independent coordinates that define the configuration of a system (which is the number of degrees of freedom of the system), turning the problem simpler to solve.

Being a little more technical, according to [Wikipedia](https://en.wikipedia.org/wiki/Configuration_space_(physics)):
"In classical mechanics, the parameters that define the configuration of a system are called generalized coordinates, and the vector space defined by these coordinates is called the configuration space of the physical system. It is often the case that these parameters satisfy mathematical constraints, such that the set of actual configurations of the system is a manifold in the space of generalized coordinates. This manifold is called the configuration manifold of the system."

In problems where it is desired to use generalized coordinates, one can write Newton's equations of motion in terms of Cartesian coordinates and then transform them into generalized coordinates. However, it would be desirable and convenient to have a general method that would directly establish the equations of motion in terms of a set of convenient generalized coordinates. In addition, general methods for writing, and perhaps solving, the equations of motion in terms of any coordinate system would also be desirable. The [Lagrangian mechanics](https://en.wikipedia.org/wiki/Lagrangian_mechanics) is such a method.

When describing a system of particles using any set of generalized coordinates, $q_1,\dotsc,q_{3N}$, these are related to, for example, the Cartesian coordinates by:

\begin{equation} \begin{array}{rcl}
q_i =q_i (x_1,\dotsc,x_{3N} ) \quad i=1,\dotsc,3N \\
x_i =x_i (q_1,\dotsc,q_{3N} ) \quad i=1,\dotsc,3N
\label{eq24}
\end{array} \end{equation}

The Cartesian components of velocity as a function of generalized coordinates are:

\begin{equation}
\dot{x}_i =\frac{\mathrm d x_i (q_1, q_2,\dotsc,q_{3N}
)}{\mathrm d t}=\sum\limits_{j=1}^{3N} {\frac{\partial x_i }{\partial q_j }}
\frac{\mathrm d q_j }{\mathrm d t}
\label{eq26}
\end{equation}

where for simplicity we omitted the explicit mention of the temporal dependence of each coordinate.

That is, any Cartesian component of the particle velocity as a function of generalized coordinates is a function of all the components of position and velocity in the generalized coordinates:

\begin{equation}
\dot{x}_i = \dot{x}_i (q_1,\dotsc,q_{3N} ,\dot{q}_1,\dotsc,\dot{q}_{3N} ) \quad i=1,\dotsc,3N
\label{eq27}
\end{equation}

## Lagrange's equation

In analogy to Newtonian mechanics, one can think that the equations of motion can be obtained by equating the generalized force, $F_i$, to the temporal rate of change of each generalized momentum, $p_i$:

\begin{equation}
F_i =\frac{\partial p_i }{\partial t}
\label{eq28}
\end{equation}

In the formula above, let's substitute the quantity $p_i$ by its definition in terms of the kinetic energy:

\begin{equation}
\frac{\partial p_i }{\partial t} =\frac{\partial }{\partial t}\left( {\frac{\partial T}{\partial
\dot{q}_i }} \right)=\frac{\partial }{\partial t}\left(
{\sum\limits_{j=1}^{3N} {m_j \dot{x}_j \frac{\partial \dot{x}_j
}{\partial \dot{q}_i }} } \right)
\label{eq29}
\end{equation}

where we used:

\begin{equation}
\frac{\partial T}{\partial \dot{q}_i }=\sum\limits_{j=1}^{3N}
{\frac{\partial T}{\partial \dot{x}_j }\frac{\partial \dot{x}_j
}{\partial \dot{q}_i }}
\label{eq30}
\end{equation}

Using the [product rule](https://en.wikipedia.org/wiki/Product_rule), the derivative of the product in Eq. (\ref{eq29}) is:

\begin{equation}
\frac{\partial p_i }{\partial t}=\sum\limits_{j=1}^{3N} {m_j
\ddot{x}_j \frac{\partial \dot{x}_j }{\partial \dot{q}_i }}
+\sum\limits_{j=1}^{3N} {m_j \dot{x}_j \frac{\mathrm d }{\mathrm d t}\left(
{\frac{\partial \dot{x}_j }{\partial \dot{q}_i }} \right)}
\label{eq31}
\end{equation}

But:

\begin{equation}
\frac{\partial \dot{x}_i }{\partial \dot{q}_j }=\frac{\partial x_i
}{\partial q_j } \quad because \quad \frac{\partial
\dot{x}_i }{\partial \dot{q}_j }=\frac{\partial x_i }{\partial
t}\frac{\partial t}{\partial q_j }=\frac{\partial x_i }{\partial q_j}
\label{eq32}
\end{equation}

Then:

\begin{equation}
\frac{\partial p_i }{\partial t}=\sum\limits_{j=1}^{3N} {m_j
\ddot{x}_j \frac{\partial x_j }{\partial q_i }}
+\sum\limits_{j=1}^{3N} {m_j \dot{x}_j \frac{\mathrm d }{\mathrm d t}\left(
{\frac{\partial x_j }{\partial q_i }} \right)}
\label{eq33}
\end{equation}

The first term on the right side of the equation above is proportional to $m_j
\ddot{x}_j$ and we will define as the generalized force, $Q_i$. But, different from Newtonian mechanics, the temporal variation of the generalized momentum is equal to the generalized force plus another term, which will investigate now. The last part of this second term can be derived as:

\begin{equation}
\frac{\mathrm d }{\mathrm d t}\left( {\frac{\partial x_j }{\partial q_i }} \right) =
\sum\limits_{k=1}^{3N} {\frac{\mathrm d }{\mathrm d q_k }\left( {\frac{\partial
x_j }{\partial q_i }} \right)\frac{\mathrm d q_k }{\mathrm d t}} =\sum\limits_{k=1}^{3N}
{\frac{\partial^2 x_j }{\partial q_k \partial q_i }\dot{q}_k }
\label{eq34}
\end{equation}

where we used the [chain rule](https://en.wikipedia.org/wiki/Chain_rule) for the differentiation:
\begin{equation}
\frac{\mathrm d }{\mathrm d t}\Big( {f\big({g(t)}\big)}\Big) = \frac{\partial f}{\partial g}\frac{\partial g}{\partial t}
\label{eq35}
\end{equation}

But if we look at Eq. (\ref{eq26}) we see that the term at the right side of the Eq. (\ref{eq34}) can be obtained by:

\begin{equation}
\frac{\partial \dot{x}_j }{\partial q_i } = \frac{\partial }{\partial q_i }\left(\sum\limits_{k=1}^{3N} \frac{\partial
x_j }{\partial q_i }\dot{q}_k \right) = \sum\limits_{k=1}^{3N}
{\frac{\partial^2 x_j }{\partial q_k \partial q_i }\dot{q}_k }
\label{eq36}
\end{equation}

Comparing the Eq. (\ref{eq34}) and Eq. (\ref{eq36}) we have:

\begin{equation}
\frac{\mathrm d }{\mathrm d t}\left( {\frac{\partial x_j }{\partial q_i }} \right) =
\frac{\mathrm d }{\mathrm d q_i}\left( {\frac{\partial x_j }{\partial t }} \right)
\label{eq37}
\end{equation}

On the other hand, it is possible to relate the term $\partial \dot{x}_j / \partial q_i$ to the derivative of kinetic energy with respect to the coordinate $q_i$:

\begin{equation}
\frac{\partial T}{\partial q_i }=\frac{\partial }{\partial q_i }\left(
{\sum\limits_{j=1}^{3N} {\frac{1}{2}m_j \dot{x}_j^2} }
\right)=\sum\limits_{j=1}^{3N} {m_j \dot{x}_j } \frac{\partial
\dot{x}_j }{\partial q_i }
\label{eq38}
\end{equation}

where once again we used the chain rule for the differentiation.

Using Eq. (\ref{eq37}), Eq. (\ref{eq38}) becomes

\begin{equation}
\frac{\partial T}{\partial q_i }=\sum\limits_{j=1}^{3N} {m_j
\dot{x}_j } \frac{\mathrm d }{\mathrm d t}\left( {\frac{\partial x_j }{\partial q_i }}
\right)
\label{eq39}
\end{equation}

Returning to Eq. (\ref{eq33}), it can be rewritten as:

\begin{equation}
\frac{\mathrm d }{\mathrm d t}\left( {\frac{\partial T}{\partial \dot{q}_i }} \right) = Q_i + \frac{\partial T}{\partial q_i }
\label{eq40}
\end{equation}

and

\begin{equation}
\frac{\mathrm d }{\mathrm d t}\left( {\frac{\partial T}{\partial \dot{q}_i }} \right) - \frac{\partial T}{\partial q_i } = Q_i
\label{eq41}
\end{equation}

Now let's look at $Q_i$, the generalized force. It can be decomposed into two terms:

The first term, composed of the conservative forces, i.e. forces that can be written as potential gradients:

\begin{equation}
Q_C =-\frac{\partial V}{\partial q_i } \quad , \quad V=V\left( {q_1,\dotsc,q_{3N} } \right)
\label{eq42}
\end{equation}

An example of conservative force is the gravitational force.

And the second term, encompassing all non-conservative forces, such as a frictional force, $Q_{NC}$.

Then:

\begin{equation} Q_i =-\frac{\partial V}{\partial q_i }+Q_{NCi} \quad , \quad V=V\left( {q_1,\dotsc,q_{3N} } \right) \end{equation}

The Eq. (\ref{eq41}) becomes

\begin{equation}
\frac{\mathrm d }{\mathrm d t}\left( {\frac{\partial T}{\partial \dot{q}_i }}
\right)-\frac{\partial T}{\partial q_i }=-\frac{\partial V}{\partial q_i} + Q_{NCi}
\label{eq43}
\end{equation}

Rearranging, we have:

\begin{equation} \frac{\mathrm d }{\mathrm d t}\left( {\frac{\partial \left( {T-V} \right)}{\partial
\dot{q}_i }} \right)-\frac{\partial \left( {T-V} \right)}{\partial q_i} = Q_{NCi}
\label{eq44}
\end{equation}

This is possible because:

\begin{equation}
\frac{\partial V}{\partial \dot{q}_i} = 0
\label{eq45}
\end{equation}

Defining:

\begin{equation}
\mathcal{L} \equiv \mathcal{L}(q_1,\dotsc,q_{3N} ,\dot{q}_1,\dotsc,\dot{q}_{3N} ) = T - V
\label{eq46}
\end{equation}

as the Lagrange or Lagrangian function, we have the Lagrange's equation:

\begin{equation}
\frac{\mathrm d }{\mathrm d t}\left( {\frac{\partial \mathcal{L}}{\partial \dot{q}_i }}
\right)-\frac{\partial \mathcal{L}}{\partial q_i } = Q_{NCi} \quad i=1,\dotsc,3N
\label{eq47}
\end{equation}

Once all derivatives of the Lagrangian function are calculated, this equation will be the equation of motion for each particle. If there are  $N$ independent particles in a three-dimensional space, there will be $3N$ equations for the system.

The set of equations above for a system are known as Euler–Lagrange equations, or Lagrange's equations of the second kind.

### Constraints

An important class of problems in mechanics, in which the Lagrangian equations are particularly useful, are composed of constrained systems. A constraint is a restriction on the freedom of movement of a particle or a system of particles (a constraint decreases the number of degrees of freedom of a system). A rigid body, or the movement of a pendulum, are examples of constrained systems. It can be shown, in a similar way, that the Lagrange equation, deduced here for a system of free particles, is also valid for a system of particles under the action of constraints. The Lagrange's equation, for a system of $3N$ particles and with $k$ constraints, is then defined as:

\begin{equation}
\frac{\mathrm d }{\mathrm d t}\left( {\frac{\partial \mathcal{L}}{\partial \dot{q}_i}} \right)-\frac{\partial \mathcal{L}}{\partial q_i } = Q_{NCi} \quad i=1,\dotsc,3N-k
\label{eq48}
\end{equation}

## References

- Goldstein H (1980) [Classical Mechanics](https://books.google.com.br/books?id=tJCuQgAACAAJ), 3rd ed., Addison-Wesley.
- Marion JB (1970) [Classical Dynamics of particles and systems](https://books.google.com.br/books?id=Ss43BQAAQBAJ), 2nd ed., Academic Press.
- Synge JL (1949) [Principles of Mechanics](https://books.google.com.br/books?id=qsYfENCRG5QC), 2nd ed., McGraw-hill.
- Taylor J (2005) [Classical Mechanics](https://archive.org/details/JohnTaylorClassicalMechanics). University Science Books.


# ***Introduction to Radar Using Python and MATLAB***
## Andy Harrison - Copyright (C) 2019 Artech House

# Ovals of Cassini
***

Referring to Figure 4.8, The Cassini ovals are a family of quartic curves, sometimes referred to as Cassini ellipses, described by a point such that the product of its distances from two fixed points a distance apart is a constant [8]. For bistatic systems, the system performance may be analyzed by plotting the Cassini ovals for various signal-to-noise ratios.  The Cassini ovals are governed by (Equation 4.65)

\begin{equation}\label{eq:bistatic_range_product_polar}
    (r_t\, r_r)^2 = \Big(\rho^2 + (D/2)^2\Big)^2 - \rho^2\, D^2\, \cos^2\theta
\end{equation}
***

Begin by getting the library path

```python
import lib_path
```

Set the separation distance, ***D***, between the transmitting and receiving radars (m)

```python
separation_distance = 10e3
```

Set the system temperature (K), bandwidth (Hz), noise_figure (dB), transmitting and receiving losses (dB), peak transmit power (W), transmitting and receiving antenna gain (dB), operating frequency (Hz), and bistatic target RCS (dBsm)

```python
system_temperature = 290

bandwidth = 10e6

noise_figure = 3

transmit_losses = 4

receive_losses = 6

peak_power = 100e3

transmit_antenna_gain = 30

receive_antenna_gain = 28

frequency = 1e9

bistatic_target_rcs = 10
```

Set the number of points for plotting the Cassini ovals

```python
number_of_points = 100000
```

Set the parameters for the Cassini ovals equation
$$ r ^ 4 + a ^ 4 - 2 a ^ 2 r ^ 2(1 + cos(2 \theta)) = b ^ 4 $$

Import the `linspace` and `log10` routines along with some constants from `scipy` for the angle sweep

```python
from numpy import linspace, log10
from scipy.constants import pi, c, k
```

```python
# Parameter "a"

a = 0.5 * separation_distance

# Full angle sweep

t = linspace(0, 2.0 * pi, number_of_points)
```

Calculate the bistatic range factor and use this along with the separation distance to calculate SNR0 (where the factors ***a*** and ***b*** are equal)

```python
# Calculate the wavelength (m)

wavelength = c / frequency

# Calculate the bistatic radar range factor

bistatic_range_factor = (peak_power * transmit_antenna_gain * receive_antenna_gain * wavelength ** 2 * 10.0 ** (bistatic_target_rcs / 10.0)) / ((4.0 * pi) ** 3 * k * system_temperature * bandwidth * 10.0 ** (noise_figure / 10.0) * transmit_losses * receive_losses)

# Calculate the signal to noise ratio at which a = b

SNR_0 = 10.0 * log10(16.0 * bistatic_range_factor / separation_distance ** 4)
```

Create a list of the signal to noise ratios to plot

```python
SNR = [SNR_0 - 6, SNR_0 - 3, SNR_0, SNR_0 + 3]
```

Import the `matplotlib` routines for plotting the Cassini ovals

```python
from matplotlib import pyplot as plt
```

Import `sqrt`, `sin`, `cos`, `real`, and `imag` from `scipy` for plotting the Cassini ovals

```python
from numpy import sqrt, sin, cos, real, imag
```

Display the resulting Cassini ovals

```python
# Set the figure size

plt.rcParams["figure.figsize"] = (15, 10)

# Loop over all the desired signal to noise ratios

for s in SNR:

    # Convert to linear units

    snr = 10.0 ** (s / 10.0)

    # Parameter for Cassini ovals

    b = (bistatic_range_factor / snr) ** 0.25

    if a > b:

         # Calculate the +/- curves

         r1 = sqrt(a ** 2 * (cos(2.0 * t) + sqrt(cos(2 * t) ** 2 - 1.0 + (b / a) ** 4)))

         r2 = sqrt(a ** 2 * (cos(2.0 * t) - sqrt(cos(2 * t) ** 2 - 1.0 + (b / a) ** 4)))

         # Find the correct indices for imaginary parts = 0

         i1 = imag(r1) == 0

         i2 = imag(r2) == 0

         r1 = real(r1)

         r2 = real(r2)

         # Plot both parts of the curve

         label_text = "SNR = {:.1f}".format(s)

         plt.plot(r1[i1] * cos(t[i1]), r1[i1] * sin(t[i1]), 'k.', label=label_text)

         plt.plot(r2[i2] * cos(t[i2]), r2[i2] * sin(t[i2]), 'k.')

    else:

         # Calculate the range for the continuous curves

         r = sqrt(a ** 2 * cos(2 * t) + sqrt(b ** 4 - a ** 4 * sin(2.0 * t) ** 2))

         # Plot the continuous parts

         label_text = "SNR = {:.1f}".format(s)

         plt.plot(r * cos(t), r * sin(t), '.', label=label_text)

    # Add the text for Tx/Rx locations

    plt.text(-a, 0, 'Tx')

    plt.text(a, 0, 'Rx')

    # Set the plot title and labels

    plt.title('Ovals of Cassini', size=14)

    plt.xlabel('Range (km)', size=12)

    plt.ylabel('Range (km)', size=12)

    # Set the tick label size

    plt.tick_params(labelsize=12)

    # Turn on the grid

    plt.grid(linestyle=':', linewidth=0.5)

    # Add the legend

    plt.legend(loc='upper left', prop={'size': 10})
```