## Subsurface Data Analytics

### The Lasso for Subsurface Data Analytics in Python

#### Michael Pyrcz, Associate Professor, University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

### PGE 383 Exercise: The Lasso for Subsurface Modeling in Python

Here's a simple workflow, demonstration of the lasso for regression for subsurface modeling workflows. This should help you get started with building subsurface models with data analytics and machine learning. Here's some basic details about the lasso.

The lasso is an extension of linear regresion, and is closely related to ridge regression so let's review some basic details about linear regression first and then related the lasso to ridge regression.

Here's complete workflows and more details on these:

[Linear Regression](https://github.com/GeostatsGuy/PythonNumericalDemos/blob/master/SubsurfaceDataAnalytics_LinearRegression.ipynb)

[Ridge Regression](https://github.com/GeostatsGuy/PythonNumericalDemos/blob/master/SubsurfaceDataAnalytics_RidgeRegression.ipynb)

#### Linear Regression

Linear regression for prediction.  Here are some key aspects of linear regression:

**Parametric Model**

* the fit model is a simple weighted linear additive model based on all the available features, $x_1,\ldots,x_m$.

* the parametric model takes the form of:

\begin{equation}
y = \sum_{\alpha = 1}^m b_{\alpha} x_{\alpha} + b_0
\end{equation}

**Least Squares**

* least squares optimization is applied to select the model parameters, $b_1,\ldots,b_m,b_0$

* we minize the error, residual sum of squares (RSS) over the training data:

\begin{equation}
RSS = \sum_{i=1}^n (y_i - \left(\sum_{\alpha = 1}^m b_{\alpha} x_{\alpha} + b_0)\right)^2
\end{equation}

* this could be simplified as the sum of square error over the training data,

\begin{equation}
\sum_{i=1}^n (\Delta y_i)^2
\end{equation}

**Assumptions**

* **Error-free** - predictor variables are error free, not random variables
* **Linearity** - response is linear combination of feature(s)
* **Constant Variance** - error in response is constant over predictor(s) value
* **Independence of Error** - error in response are uncorrelated with each other
* **No multicollinearity** - none of the features are redundant with other features

#### Other Resources

In $Python$, the $SciPy$ package, specifically the $Stats$ functions (https://docs.scipy.org/doc/scipy/reference/stats.html) provide excellent tools for efficient use of statistics.
I have previously provided linear regression demonstration in R:

1. [R](https://github.com/GeostatsGuy/geostatsr/blob/master/linear_regression_demo_v2.R)
2. [R Markdown](https://github.com/GeostatsGuy/geostatsr/blob/master/linear_regression_demo_v2.Rmd)
3. [knit as an HTML document](https://github.com/GeostatsGuy/geostatsr/blob/master/linear_regression_demo_v2.html)

#### The Lasso

With the lasso we add a hyperparameter, $\lambda$, to our minimization, with a shrinkage penalty term.

\begin{equation}
\sum_{i=1}^n \left(y_i - \left(\sum_{\alpha = 1}^m b_{\alpha} x_{\alpha} + b_0 \right) \right)^2 + \lambda \sum_{j=1}^m |b_{\alpha}|
\end{equation}

As a result the lasso has 2 criteria:

1. set the model parameters to minimize the error with training data

2. shrink the estimates of the slope parameters towards zero. Note: the intercept is not affected by the lambda, $\lambda$, hyperparameter.

Note the only difference between the lasso and ridge regression is:

* for the lasso the shrinkage term is posed as an $\ell_1$ penalty ($\lambda \sum_{\alpha=1}^m |b_{\alpha}|$)

* for ridge regression the shrinkage term is posed as an $\ell_2$ penalty ($\lambda \sum_{\alpha=1}^m \left(b_{\alpha}\right)^2$).

While both ridge regression and the lasso shrink the model parameters ($b_{\alpha}, \alpha = 1,\ldots,m$) towards zero:

* the lasso parameters reach zero at different rates for each predictor feature as the lambda, $\lambda$, hyperparameter increases.

* as a result the lasso provides a method for feature ranking and selection!

The lambda, $\lambda$, hyperparameter controls the degree of fit of the model and may be related to the model variance and bias trade-off.

* for $\lambda \rightarrow 0$ the prediction model approaches linear regression, there is lower model bias, but the model variance is higher

* as $\lambda$ increases the model variance decreases and the model bias increases

* for $\lambda \rightarrow \infty$ the coefficients all become 0.0 and the model is the global mean

#### Workflow Goals

Learn the basics of the lasso in Python to for analysis, modeling and prediction of porosity from density. This includes:

* Basic Python workflows and data preparation

* Training / fitting a the lasso model and comparison to ridge regression

* Checking the model and learning about the impact of hyperparameters

#### Objective

In the PGE 383: Stochastic Subsurface Modeling class I want to provide hands-on experience with building subsurface modeling workflows. Python provides an excellent vehicle to accomplish this. I have coded a package called GeostatsPy with GSLIB: Geostatistical Library (Deutsch and Journel, 1998) functionality that provides basic building blocks for building subsurface modeling workflows.

The objective is to remove the hurdles of subsurface modeling workflow construction by providing building blocks and sufficient examples. This is not a coding class per se, but we need the ability to 'script' workflows working with numerical methods.

#### Getting Started

Here's the steps to get setup in Python with the GeostatsPy package:

1. Install Anaconda 3 on your machine (https://www.anaconda.com/download/).
2. From Anaconda Navigator (within Anaconda3 group), go to the environment tab, click on base (root) green arrow and open a terminal.
3. In the terminal type: pip install geostatspy.
4. Open Jupyter and in the top block get started by copy and pasting the code block below from this Jupyter Notebook to start using the geostatspy functionality.

There are examples below with these functions. You can go here to see a list of the available functions, https://git.io/fh4eX, other example workflows and source code.

#### Import Required Packages

Let's import the GeostatsPy package.

```python
import os                                                   # to set current working directory
import numpy as np                                          # arrays and matrix math
import scipy.stats as st                                    # statistical methods
import pandas as pd                                         # DataFrames
import matplotlib.pyplot as plt                             # for plotting
from sklearn.metrics import mean_squared_error, r2_score    # specific measures to check our models
from sklearn.linear_model import Ridge                      # ridge regression implemented in scikit learn
from sklearn.linear_model import Lasso                      # the lasso implemented in scikit learn
from sklearn.model_selection import cross_val_score         # multi-processor K-fold crossvalidation
from sklearn.model_selection import train_test_split        # train and test split
from sklearn.preprocessing import StandardScaler            # standardize the features
from sklearn import metrics                                 # measures to check our models
from sklearn.linear_model import LinearRegression           # linear regression implemented in scikit learn
```

If you get a package import error, you may have to first install some of these packages. This can usually be accomplished by opening up a command window on Windows and then typing 'python -m pip install [package-name]'. More assistance is available with the respective package docs.

#### Set the working directory

I always like to do this so I don't lose files and to simplify subsequent read and writes (avoid including the full address each time).  Also, in this case make sure to place the required (see below) data file in this working directory.

```python
#os.chdir("C:\PGE383")                                       # set the working directory
```

#### Loading Data

Let's load the provided dataset. 'Density_Por_data.csv' is available at https://github.com/GeostatsGuy/GeoDataSets. It is a comma delimited file with 20 density ($\frac{g}{cm^3}$) and porosity (as a fraction) measures from the subsurface. We load the data file with the pandas 'read_csv' function into a data frame we called 'df' and then separate it into train and test datasets.  The smaples are in random order so we just split the dataset at the 80th sample.  We preview each with the head function from Pandas DataFrames.

```python
#df = pd.read_csv("Density_Por_data.csv")                    # read a .csv file in as a DataFrame
df = pd.read_csv(r"https://raw.githubusercontent.com/GeostatsGuy/GeoDataSets/master/Density_Por_data.csv")
#df_mv = pd.read_csv("unconv_MV.csv")                        # load a multivariate dataset that we will use later
df_mv = pd.read_csv(r"https://raw.githubusercontent.com/GeostatsGuy/GeoDataSets/master/unconv_MV.csv")
df_train = df.iloc[0:80,:]                                  # extract a training set, note samples are random ordered
df_train.head()                                             # preview the DataFrame
```

      Density
      Porosity

      0
      1.281391
      16.610982

      1
      1.404932
      13.668073

      2
      2.346926
      9.590092

      3
      1.348847
      15.877907

      4
      2.331653
      4.968240

```python
df_test = df.iloc[80:]                                      # extract a testing set, note samples are random ordered
df_test.head()                                              # preview the DataFrame
```

      Density
      Porosity

      80
      1.750352
      11.325941

      81
      1.666285
      15.609445

      82
      1.466517
      17.066529

      83
      1.650921
      13.876841

      84
      0.996736
      20.964941

It is useful to review the summary statistics of our loaded DataFrame.  That can be accomplished with the 'describe' DataFrame member function.  We transpose to switch the axes for ease of visualization.  We will summarize over the training and testing subsets separately.

```python
df_train.describe().transpose()
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      Density
      80.0
      1.759945
      0.295260
      1.067960
      1.552707
      1.767908
      1.953262
      2.410560

      Porosity
      80.0
      12.187127
      3.125755
      4.966421
      10.011526
      12.170687
      14.063670
      19.600717

```python
df_test.describe().transpose()
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      Density
      25.0
      1.667429
      0.257608
      0.996736
      1.568988
      1.718085
      1.791432
      2.339324

      Porosity
      25.0
      13.632564
      2.948887
      9.489298
      11.325941
      13.767060
      15.051759
      20.964941

Here we extract the ndarrays with porsity and density, training and testing datasets separate arrays for convenience.

```python
por_train = df_train['Porosity'].values                     # make a shallow copy of the features for convenvience
den_train = df_train['Density'].values
por_test = df_test['Porosity'].values
den_test = df_test['Density'].values
n_train = len(df_train); n_test = len(df_test)              # get the number of data in training and testing
```

#### Linear Regression Model

Let's first calculate the linear regression model. We use scikit learn and then extend the same workflow to ridge regression.

```python
# Step 1. Instantiate the Model
linear_reg = LinearRegression()                             # instantiate the linear regression model

# Step 2: Fit the Data on Training Data
linear_reg.fit(df_train["Density"].values.reshape(n_train,1), df_train["Porosity"]) # fit model
density_model = np.linspace(1.2,2.4,10)

# Print the model parameters
porosity_model = linear_reg.predict(density_model.reshape(10,1)) # predict with the fit model
print('Coefficients: ', str(round(linear_reg.coef_[0],3)) + ', Intercept: ', str(round(linear_reg.intercept_,3)))

# Plot model fit
plt.subplot(111)
plt.scatter(df_train["Density"].values, df_train["Porosity"],  color='black', s = 20, alpha = 0.3)
plt.plot(density_model,porosity_model, color='red', linewidth=1)
plt.title('Linear Regression Porosity from Density with Training Data'); plt.xlabel('Density (g/cm^3)'); plt.ylabel('Porosity (%)')
plt.xlim(1.,2.6)#; plt.ylim(0,1500000)
plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.0, wspace=0.2, hspace=0.2)
plt.show()
```

Let's run some quick model checks.  Much more could be done, but I limit this for breviety here.

```python
# Step 3: - Make predictions using the testing dataset
y_pred = linear_reg.predict(df_test['Density'].values.reshape(n_test,1))

# Report the goodness of fit
print('Variance explained: %.2f' % r2_score(df_test['Porosity'].values, y_pred))

# Plot testing diagnostics
plt.subplot(121)
plt.scatter(df_test['Density'].values, df_test['Porosity'].values,  color='black', s = 20, alpha = 0.3)
plt.scatter(df_test['Density'], y_pred, color='blue', s = 20, alpha = 0.3)
plt.title('Linear Regression Model Testing - Production from Porosity'); plt.xlabel('Density (g/cm^3)'); plt.ylabel('Porosity (%)')
plt.xlim(1.0,2.6); plt.ylim(5,24)

y_res = y_pred - df_test['Porosity'].values

print('Residual: mean = ' + str(round(np.average(y_res),2)) + ', standard deviation = ' + str(round(np.var(y_res),2)))

plt.subplot(122)
plt.hist(y_res, alpha = 0.2, color = 'red', edgecolor = 'black', bins=20)
plt.title('Linear Regression Model Prediction Error - Porosity form Density'); plt.xlabel('Porosity Estimation Error (%) (Estimate - Truth)'); plt.ylabel('Frequency')
plt.xlim(-4,4)#; plt.ylim(0,1500000)

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.2, top=1.2, wspace=0.3, hspace=0.2)
plt.show()
```

#### Ridge Regression

Let's replace the scikit learn linear regression method with the scikit learn ridge regression method.  Note, we must now set the lambda hyperparameter.

* the hyperparameter is set with the instantiation of the model

```python
lam = 1.0

# Step 1. Instantiate the Model
ridge_reg = Ridge(alpha=lam)

# Step 2: Fit the Data on Training Data
ridge_reg.fit(df_train["Density"].values.reshape(n_train,1), df_train["Porosity"]) # fit model
density_model = np.linspace(1.2,2.4,10)

# Print the model parameters
porosity_ridge_model = ridge_reg.predict(density_model.reshape(10,1)) # predict with the fit model
print('Coefficients: ', str(round(linear_reg.coef_[0],3)) + ', Intercept: ', str(round(linear_reg.intercept_,3)))

# Plot model fit
plt.subplot(111)
plt.scatter(df_train["Density"].values, df_train["Porosity"],  color='black', s = 20, alpha = 0.3)
plt.plot(density_model,porosity_ridge_model, color='red', linewidth=1)
plt.title('Ridge Regression Porosity from Density with Training Data'); plt.xlabel('Density (g/cm^3)'); plt.ylabel('Porosity (%)')
plt.xlim(1.,2.6); plt.ylim(8,17)
plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.0, wspace=0.2, hspace=0.2)
plt.show()
```

Let's repeat the simple model checks that we applied with our linear regression model.

```python
# Step 3: - Make predictions using the testing dataset
y_pred = ridge_reg.predict(df_test['Density'].values.reshape(n_test,1))

# Report the goodness of fit
print('Variance explained: %.2f' % r2_score(df_test['Porosity'].values, y_pred))

# Plot testing diagnostics
plt.subplot(121)
plt.scatter(df_test['Density'].values, df_test['Porosity'].values,  color='black', s = 20, alpha = 0.3)
plt.scatter(df_test['Density'], y_pred, color='blue', s = 20, alpha = 0.3)
plt.title('Ridge Regression Model Testing - Production from Porosity'); plt.xlabel('Density (g/cm^3)'); plt.ylabel('Porosity (%)')
plt.xlim(1.0,2.6); plt.ylim(5,24)

y_res = y_pred - df_test['Porosity'].values

print('Residual: mean = ' + str(round(np.average(y_res),2)) + ', standard deviation = ' + str(round(np.var(y_res),2)))

plt.subplot(122)
plt.hist(y_res, alpha = 0.2, color = 'red', edgecolor = 'black', bins=20)
plt.title('Ridge Regression Model Prediction Error - Porosity form Density'); plt.xlabel('Porosity Estimation Error (%) (Estimate - Truth)'); plt.ylabel('Frequency')
plt.xlim(-4,4)#; plt.ylim(0,1500000)

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.2, top=1.2, wspace=0.3, hspace=0.2)
plt.show()
```

Interesting, we explained less variance and have a larger residual standard deviation (more error).

* we reduced both testing variance explained and accuracy in this case!

#### the Lasso

Let's replace the scikit learn linear regression and ridge regression methods with the scikit learn the lasso method.  Note, we must now set the lambda hyperparameter.

* the lambda hyperparameter, $\lambda$, is set with the instantiation of the model

```python
lam = 1.0

# Step 1. Instantiate the Model
lasso_reg = Lasso(alpha=lam)

# Step 2: Fit the Data on Training Data
lasso_reg.fit(df_train["Density"].values.reshape(n_train,1), df_train["Porosity"]) # fit model
density_model = np.linspace(1.2,2.4,10)

# Print the model parameters
porosity_lasso_model = lasso_reg.predict(density_model.reshape(10,1)) # predict with the fit model
print('Coefficients: ', str(round(linear_reg.coef_[0],3)) + ', Intercept: ', str(round(linear_reg.intercept_,3)))

# Plot model fit
plt.subplot(111)
plt.scatter(df_train["Density"].values, df_train["Porosity"],  color='black', s = 20, alpha = 0.3)
plt.plot(density_model,porosity_lasso_model, color='red', linewidth=1)
plt.title('the Lasso Porosity from Density with Training Data'); plt.xlabel('Density (g/cm^3)'); plt.ylabel('Porosity (%)')
plt.xlim(1.,2.6); plt.ylim(8,17)
plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.0, wspace=0.2, hspace=0.2)
plt.show()
```

Let's repeat the simple model checks that we applied with our linear regression model.

```python
# Step 3: - Make predictions using the testing dataset
y_pred = lasso_reg.predict(df_test['Density'].values.reshape(n_test,1))

# Report the goodness of fit
print('Variance explained: %.2f' % r2_score(df_test['Porosity'].values, y_pred))

# Plot testing diagnostics
plt.subplot(121)
plt.scatter(df_test['Density'].values, df_test['Porosity'].values,  color='black', s = 20, alpha = 0.3)
plt.scatter(df_test['Density'], y_pred, color='blue', s = 20, alpha = 0.3)
plt.title('Linear Regression Model Testing - Production from Porosity'); plt.xlabel('Density (g/cm^3)'); plt.ylabel('Porosity (%)')
plt.xlim(1.0,2.6); plt.ylim(5,24)

y_res = y_pred - df_test['Porosity'].values

print('Residual: mean = ' + str(round(np.average(y_res),2)) + ', standard deviation = ' + str(round(np.var(y_res),2)))

plt.subplot(122)
plt.hist(y_res, alpha = 0.2, color = 'red', edgecolor = 'black', bins=20)
plt.title('the Lasso Model Prediction Error - Porosity form Density'); plt.xlabel('Porosity Estimation Error (%) (Estimate - Truth)'); plt.ylabel('Frequency')
plt.xlim(-4,4)#; plt.ylim(0,1500000)

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.2, top=1.2, wspace=0.3, hspace=0.2)
plt.show()
```

What happenned? With our set lambda level

```python
lam = 1.0
```

the lasso model removed the density feature and estimated with the global mean.  The lasso attempts to remove features as a function of the lambda level, in other words, the lasso performs feature selection! Let's investigate this model behavoir.

#### Investigating the Lambda Hyperparameter

Let's look at the multivariate dataset that we already loaded. This way we can observe the model behavoir over a range of features, for a range of lambda hyperparameter values.

We will first remove the well index and preview the multivariate unconventional dataset.

```python
df_mv = df_mv.drop('WellIndex',axis = 1)                    # remove the well index feature
df_mv.head()                                                # load the comma delimited data file
```

      Por
      LogPerm
      AI
      Brittle
      TOC
      VR
      Production

      0
      15.91
      1.67
      3.06
      14.05
      1.36
      1.85
      177.381958

      1
      15.34
      1.65
      2.60
      31.88
      1.37
      1.79
      1479.767778

      2
      20.45
      2.02
      3.13
      63.67
      1.79
      2.53
      4421.221583

      3
      11.95
      1.14
      3.90
      58.81
      0.40
      2.03
      1488.317629

      4
      19.53
      1.83
      2.57
      43.75
      1.40
      2.11
      5261.094919

Let's calculate the summary statistics for our data.

```python
df_mv.describe().transpose()
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      Por
      1000.0
      14.950460
      3.029634
      5.400000
      12.85750
      14.98500
      17.080000
      24.65000

      LogPerm
      1000.0
      1.398880
      0.405966
      0.120000
      1.13000
      1.39000
      1.680000
      2.58000

      AI
      1000.0
      2.982610
      0.577629
      0.960000
      2.57750
      3.01000
      3.360000
      4.70000

      Brittle
      1000.0
      49.719480
      15.077006
      -10.500000
      39.72250
      49.68000
      59.170000
      93.47000

      TOC
      1000.0
      1.003810
      0.504978
      -0.260000
      0.64000
      0.99500
      1.360000
      2.71000

      VR
      1000.0
      1.991170
      0.308194
      0.900000
      1.81000
      2.00000
      2.172500
      2.90000

      Production
      1000.0
      2247.295809
      1464.256312
      2.713535
      1191.36956
      1976.48782
      3023.594214
      12568.64413

Let's standardize the feature to have:

* mean = 0.0
* variance = standard deviation = 1.0

To do this we:

1. instantiate the StandardScaler from scikit learn. We assign it as 'scaler' so we can use it to conveniently reverse the transformation if we like. We will need to do that to get our predictions back into regular production units.

```python
scaler = StandardScaler()
```

2. we then extract all the values from our DataFrame and apply the by-column standardization.  The result is a 2D ndarray

```python
sfeatures = scaler.fit_transform(df_mv.values)
```
3. we make an new empty DataFrame

```python
df_nmv = pd.DataFrame()
```

4. then we add the transformed value to the new DataFrame while keeping the sample index and feature names from the old DataFramae

```python
df_nmv = pd.DataFrame(sfeatures, index=df_mv.index, columns=df_mv.columns)
```

```python
scaler = StandardScaler()                                   # instantiate the scaler
sfeatures = scaler.fit_transform(df_mv.values)              # standardize all the values extracted from the DataFrame
df_nmv = pd.DataFrame()                                     # instantiate a new DataFrame
df_nmv = pd.DataFrame(sfeatures, index=df_mv.index, columns=df_mv.columns) # copy the standardized values into the new DataFrame
df_nmv.head()                                               # preview the the new DataFrame
```

      Por
      LogPerm
      AI
      Brittle
      TOC
      VR
      Production

      0
      0.316877
      0.668174
      0.134046
      -2.367004
      0.705711
      -0.458285
      -1.414335

      1
      0.128641
      0.618884
      -0.662712
      -1.183816
      0.725523
      -0.653065
      -0.524438

      2
      1.816157
      1.530747
      0.255292
      0.925747
      1.557659
      1.749222
      1.485405

      3
      -0.990866
      -0.638008
      1.588995
      0.603241
      -1.196314
      0.126055
      -0.518596

      4
      1.512338
      1.062493
      -0.714674
      -0.396131
      0.784962
      0.385762
      2.059275

Let's check the summary statistics.

```python
df_nmv.describe().transpose()                               # summary statistics from the new DataFrame
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      Por
      1000.0
      1.186273e-16
      1.0005
      -3.153925
      -0.691175
      0.011406
      0.703255
      3.203157

      LogPerm
      1000.0
      -2.957079e-16
      1.0005
      -3.151793
      -0.662653
      -0.021885
      0.692819
      2.910864

      AI
      1000.0
      4.085621e-16
      1.0005
      -3.503327
      -0.701684
      0.047442
      0.653670
      2.974660

      Brittle
      1000.0
      1.703082e-16
      1.0005
      -3.996126
      -0.663393
      -0.002620
      0.627130
      2.903256

      TOC
      1000.0
      2.389755e-16
      1.0005
      -2.503956
      -0.720808
      -0.017455
      0.705711
      3.380433

      VR
      1000.0
      -3.831380e-16
      1.0005
      -3.542302
      -0.588138
      0.028665
      0.588658
      2.950365

      Production
      1000.0
      -4.163336e-17
      1.0005
      -1.533683
      -0.721496
      -0.185038
      0.530431
      7.052394

Success, we have all features standardized.  We are ready to build our model.  Let's extract training and testing datasets.

```python
X_train, X_test, y_train, y_test = train_test_split(df_nmv.iloc[:,:6], pd.DataFrame({'Production':df_nmv['Production']}), test_size=0.33, random_state=73073)
print('Number of training data = ' + str(len(X_train)) + ' and number of testing data = ' + str(len(X_test)))
```

    Number of training data = 670 and number of testing data = 330

Now let's observe the model coefficients ($b_{\alpha}, \alpha = 1,\ldots,m$) for a range of $\lambda$ hyperparameter values.

```python
nbins = 1000                                                # number of bins to explore the hyperparameter
df_nmv.describe().transpose()                               # summary statistics from the new DataFrame
lams = np.linspace(0.001,1.0,nbins)                         # make a list of lambda values
coefs = np.ndarray((nbins,6))

index = 0
for lam in lams:
    lasso_reg = Lasso(alpha=lam)                            # instantiate the model
    lasso_reg.fit(X_train, y_train)                         # fit model
    coefs[index,:] = lasso_reg.coef_                        # retreive the coefficients
    index = index + 1

color = ['black','blue','green','red','orange','grey']
plt.subplot(111)                                            # plot the results
for ifeature in range(0,6):
    plt.semilogx(lams,coefs[:,ifeature], label = df_mv.columns[ifeature], c = color[ifeature], linewidth = 3.0)

plt.title('Standardized Model Coefficients vs. Lambda Hyperparameter'); plt.xlabel('Lambda Hyperparameter'); plt.ylabel('Standardized Model Coefficients')
plt.xlim(0.001,1); plt.ylim(-1.0,1.0); plt.grid(); plt.legend(loc = 'lower right')

plt.subplots_adjust(left=0.0, bottom=0.0, right=1.5, top=1., wspace=0.2, hspace=0.2)
plt.show()
```

What do we see?

* for a very low lambda value, all features are included

* as we increase the lambda hyperparameter, total organic carbon is the first predictor feature to be removed

* then acoustic impedance, vitrinite reflectance, brittleness, log perm and finally porosity.

* at $\lambda \ge 0.8$ all features are removed.

Let's repeat this workflow with ridge regression for contrast.

```python
nbins = 5000                                                # number of bins to explore the hyperparameter
lams = np.logspace(-10,7,nbins)
ridge_coefs = np.ndarray((nbins,6))

index = 0
for lam in lams:
    ridge_reg = Ridge(alpha=lam)
    ridge_reg.fit(X_train, y_train) # fit model
    ridge_coefs[index,:] = ridge_reg.coef_
    index = index + 1

color = ['black','blue','green','red','orange','grey']
plt.subplot(111)
for ifeature in range(0,6):
    plt.semilogx(lams,ridge_coefs[:,ifeature], label = df_mv.columns[ifeature], c = color[ifeature], linewidth = 3.0)

plt.title('Standardized Model Coefficients vs. Lambda Hyperparameter'); plt.xlabel('Lambda Hyperparameter'); plt.ylabel('Standardized Model Coefficients')
plt.xlim(1.0e-10,1.0e7); plt.ylim(-1.0,1.0); plt.grid(); plt.legend(loc = 'lower right')

plt.subplots_adjust(left=0.0, bottom=0.0, right=1.5, top=1., wspace=0.2, hspace=0.2)
plt.show()
```

Ridge regression is quite different in the response of predictor feature to change in the lambda hyperparameter.

* there is no selective removal of predictor features as the lambda hyperparameter increases

* a major component is uniform shrinkage of all coefficients towards zero for $\lambda \in [10^1, 10^5]$

Let's check the mode performance for both models over the hyperparameters. We check the mean squared error and the variance explained for:

* linear regression

* ridge regression

* the lasso

```python
nbins = 1000                                               # number of bins
lams = np.logspace(-14,6,nbins)                           # assign equal bins in log space
ridge_error = []; lasso_error = []; linear_error = []
ridge_r2 = []; lasso_r2 = []; linear_r2 = []

linear_reg = LinearRegression()
linear_reg.fit(X_train, y_train) # fit model
linear_predict = linear_reg.predict(X_test)
linear_err = metrics.mean_squared_error(y_test["Production"],linear_predict)
linear_r = metrics.r2_score(y_test["Production"],linear_predict)

index = 0
for lam in lams:
    ridge_reg = Ridge(alpha=lam)
    ridge_reg.fit(X_train, y_train) # fit model
    ridge_predict = ridge_reg.predict(X_test)
    ridge_error.append(metrics.mean_squared_error(y_test["Production"],ridge_predict))
    ridge_r2.append(metrics.r2_score(y_test["Production"],ridge_predict))
    lasso_reg = Lasso(alpha=lam)
    lasso_reg.fit(X_train, y_train) # fit model
    lasso_predict = lasso_reg.predict(X_test)
    lasso_error.append(metrics.mean_squared_error(y_test["Production"],lasso_predict))
    lasso_r2.append(metrics.r2_score(y_test["Production"],lasso_predict))
    linear_error.append(linear_err)
    linear_r2.append(linear_r)
    index = index + 1

color = ['black','blue','green','red','orange','grey']

plt.subplot(121)
plt.semilogx(lams,linear_error, label = 'Linear Regression', c = 'black', linewidth = 3.0)
plt.semilogx(lams,ridge_error, label = 'Ridge Regression', c = 'blue', linewidth = 3.0)
plt.semilogx(lams,lasso_error, label = 'the Lasso', c = 'red', linewidth = 3.0)
plt.title('Testing MSE vs. Lambda Hyperparameter'); plt.xlabel('Lambda Hyperparameter'); plt.ylabel('Mean Square Error')
plt.xlim(1.0e-14,1.0e6); plt.ylim(0.0,1.0); plt.grid(); plt.legend(loc = 'lower right')

plt.subplot(122)
plt.semilogx(lams,linear_r2, label = 'Linear Regression', c = 'black', linewidth = 3.0)
plt.semilogx(lams,ridge_r2, label = 'Ridge Regression', c = 'blue', linewidth = 3.0)
plt.semilogx(lams,lasso_r2, label = 'the Lasso', c = 'red', linewidth = 3.0)
plt.title('Testing Variance Explained vs. Lambda Hyperparameter'); plt.xlabel('Lambda Hyperparameter'); plt.ylabel('Variance Explained (R2)')
plt.xlim(1.0e-14,1.0e6); plt.ylim(0.0,1.0); plt.grid(); plt.legend(loc = 'upper right')

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.5, wspace=0.2, hspace=0.2)
plt.show()
```

This is quite interesting. Let's make some observations.

* linear regression does not have the shrinkage term, nor the lambda parameter so the performance is constant

* for this problem linear regression is the best performer, lowest testing mean square error and higher testing variance explained

* at a low lambda value, ridge regression and the lasso approach linear regression

#### Model Variance

Now let's explore the concept of model variance, an important part of machine learning accuracy in testing.

* the sensitivity of the model to the specfic training data

* as lambda increases the sensitivity to the training data, model variance decreases

Let's demonstrate this with this workflow:

* loop over multiple lambda values
    * loop over multiple bootstrap samples of the data
        * calculate the ridge regression fit (slope)
    * calculate the variance of these bootstrap results

```python
L = 200                                                     # the number of bootstrap realizations
nsamples = 20                                               # the number of samples in each bootstrap realization
nlambda = 100                                               # number of lambda values to evaluate

coef_mat = np.zeros(L)                                      # declare arrays to store the results
variance_coef = np.zeros(nlambda)

lamd_mat = np.linspace(0.0,100.0,nlambda)
for ilam in range(0,len(lamd_mat)):                         # loop over all lambda values
    for l in range(0, L):                                   # loop over all bootstrap realizations
        df_sample = df.sample(n = nsamples)                 # random sample (1 bootstrap)
        ridge_reg = Ridge(alpha=lamd_mat[ilam])             # instatiate model
        ridge_reg.fit(df_sample["Density"].values.reshape(nsamples,1), df_sample["Porosity"]) # fit model
        coef_mat[l] = ridge_reg.coef_[0]                    # get the slope parameter

    variance_coef[ilam] = np.var(coef_mat)                  # calculate the variance of the slopes over the L bootstraps
```

Now let's plot the result.

```python
plt.subplot(111)
plt.plot(lamd_mat, variance_coef,  color='black', linewidth = 2, label = 'Slope Variance')
plt.title('Model Fit Variance vs. Lambda'); plt.xlabel('Lambda'); plt.ylabel('Model Fit Variance')
plt.xlim(0.,100.); plt.ylim(0.001,10.0); plt.yscale('log')
plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.0, wspace=0.2, hspace=0.2)
plt.legend()
```

The result is as expected, with increase in lambda hyperparameter the sensitivity of the model to the training data is decreased.

#### Comments

LASSO regression is a variant of linear regression that includes a hyperparameter to constrain the degree of model fit.  This allow us to tune the variance-bias trade-off of our model. I hope this was helpful,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin
On twitter I'm the @GeostatsGuy.

***

#### More on Michael Pyrcz and the Texas Center for Geostatistics:

### Michael Pyrcz, Associate Professor, University of Texas at Austin
*Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions*

With over 17 years of experience in subsurface consulting, research and development, Michael has returned to academia driven by his passion for teaching and enthusiasm for enhancing engineers' and geoscientists' impact in subsurface resource development.

For more about Michael check out these links:

#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

#### Want to Work Together?

I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.

* Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I'd be happy to drop by and work with you!

* Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!

* I can be reached at mpyrcz@austin.utexas.edu.

I'm always happy to discuss,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin

#### More Resources Available at: [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

```python

```


```python
%matplotlib notebook
import numpy as np
from mpl_toolkits.mplot3d import Axes3D
# Axes3D import adds feature, it enables using projection='3d' in add_subplot
import matplotlib.pyplot as plt
import random
```

```python
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import axes3d
import matplotlib.cm as cm
from IPython.display import display, Math, clear_output

import sympy
from sympy import *
from sympy.physics.vector import ReferenceFrame, CoordinateSym
from sympy.vector import CoordSys3D, divergence, curl

import ipyvolume as ipv
import time
from ipywidgets import Output, interact
import ipywidgets as widgets
np.seterr(divide='ignore', invalid='ignore')
init_printing()
```

## Earnshaw's Theorem: 8 pts along a cube, position a charge in the center. Is it stable?

```python
# create a X,Y grid
x = np.linspace(0, 100, 300)
y = np.linspace(0, 100, 300)
X, Y = np.meshgrid(x, y)
```

```python
# calculate the electric potential at a x,y,z point due to 8 point charges
# at the vertices of a cube (100x100x100)
def V(x,y,z):
    v = 0
    for crds in [(0,0,0), (0,0,100), (0, 100, 0), (100,0,0), (100,100,0), (100,0,100), (0,100,100), (100,100,100)]:
        v += 1/np.sqrt( (x-crds[0])**2 + (y-crds[1])**2  + (z-crds[2])**2 )
    return v
```

```python
# set z fixed in the center of the cube, get potential versus x,y
v_res = V(X,Y,50)
```

```python
from mpl_toolkits import mplot3d
fig = plt.figure()
ax = plt.axes(projection='3d')
ax.plot_surface(X, Y, v_res, rstride=1, cstride=1,
                cmap='viridis', edgecolor='none')
```

    <IPython.core.display.Javascript object>

    <mpl_toolkits.mplot3d.art3d.Poly3DCollection at 0xb27c29438>


# The Fourier Transform

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Communications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Theorems

The theorems of the Fourier transform relate basic time-domain operations to their equivalents in the Fourier domain. They are of use for the computation of Fourier transforms of signals composed from modified [standard signals](../continuous_signals/standard_signals.ipynb) and for the computation of the response of systems to an input signal. The theorems allow further to predict the consequences of modifying a signal or system by certain operations.

### Temporal Scaling Theorem

A signal $x(t)$ for which the Fourier transform $X(j \omega) = \mathcal{F} \{ x(t) \}$ is existing and known is given. The Fourier transform of the [temporally scaled signal](../continuous_signals/operations.ipynb#Temporal-Scaling) $x(a t)$ with $a \in \mathbb{R} \setminus \{0\}$ reads

\begin{equation}
\mathcal{F} \{ x(a t) \} = \frac{1}{|a|} \cdot X \left( \frac{j \omega}{a} \right)
\end{equation}

The Fourier transformation of a temporally scaled signal is given by inverse scaling of the Fourier transform of the unscaled signal and weighting with $\frac{1}{|a|}$. This relation is known as scaling theorem of the Fourier transform. It can be proven by introducing the scaled signal $x(a t)$ into the definition of the Fourier transformation together with the substitution $t' = a t$. For $a>0$ this reads

\begin{equation}
\begin{split}
\mathcal{F} \{ x(a t) \} &= \int_{-\infty}^{\infty} x(a t) e^{-j \omega t} dt \\
&= \frac{1}{a} \int_{-\infty}^{\infty} x(t') e^{-j \frac{\omega}{a} t'} dt' \\
&= \frac{1}{a} X \left( \frac{\omega}{a} \right)
\end{split}
\end{equation}

Note that the cases $a<0$ and $a>0$ have to be regarded separately.

**Example**

According to the scaling theorem and the [Fourier transform of the rectangular signal](definition.ipynb#Transformation-of-the-Rectangular-Signal) $\mathcal{F} \{ \text{rect}(t) \} = \text{sinc} \left( \frac{\omega}{2} \right)$, the transform of the scaled rectangular signal $\text{rect}(a t)$ is given as

\begin{equation}
\mathcal{F} \{ \text{rect}(a t) \} = \frac{1}{|a|} \cdot \text{sinc} \left( \frac{\omega}{2 a} \right)
\end{equation}

This can be confirmed by explicitly calculating the Fourier transformation of $\text{rect}(a t)$

```python
import sympy as sym
sym.init_printing()

def fourier_transform(x):
    return sym.integrals.transforms._fourier_transform(x, t, w, 1, -1, 'Fourier')

t, w = sym.symbols('t omega')
a = sym.symbols('a', positive=True)

x = sym.Heaviside(t + sym.S.Half/a) - sym.Heaviside(t - sym.S.Half/a)
X = fourier_transform(x)
X
```

For illustration both the signal $x(t)$ and its Fourier transform $X(j \omega)$ are plotted for a specific value of $a$

```python
values = {a: 1/2}

sym.plot(x.subs(values), (t, -5, 5), xlabel=r'$t$',
         ylabel=r'$\mathrm{rect}(a t)$')
sym.plot(X.subs(values), (w, -20, 20), xlabel=r'$\omega$',
         ylabel=r'$\mathcal{F} \{ \mathrm{rect}(a t) \}$');
```

**Exercise**

* Change the width $1/a$ of the rectangular signal. How does its spectrum change?
* On a qualitative level, what happens when $1/a$ increases towards infinity?

### Convolution Theorem

The [convolution theorem](https://en.wikipedia.org/wiki/Convolution_theorem) states that the Fourier transform of the convolution of two signals $x(t)$ and $y(t)$ is equal to the scalar multiplication of their Fourier transforms $X(j \omega) = \mathcal{F} \{ x(t) \}$ and $Y(j \omega) = \mathcal{F} \{ y(t) \}$

\begin{equation}
\mathcal{F} \{ x(t) * y(t) \} = X(j \omega) \cdot Y(j \omega)
\end{equation}

The theorem can be proven by introducing the [definition of the convolution](../systems_time_domain/convolution.ipynb) into the [definition of the Fourier transform](definition.ipynb) and changing the order of integration

\begin{align}
\mathcal{F} \{ x(t) * y(t) \} &= \int_{-\infty}^{\infty} \left( \int_{-\infty}^{\infty} x(\tau) \cdot y(t-\tau) \; d \tau \right) e^{-j \omega t} \; dt \\
&= \int_{-\infty}^{\infty} \left( \int_{-\infty}^{\infty} y(t-\tau) \, e^{-j \omega t} \; dt \right) x(\tau) \; d\tau \\
&= Y(j \omega) \cdot \int_{-\infty}^{\infty} x(\tau) \, e^{-j \omega \tau} \; d \tau \\
&= Y(j \omega) \cdot X(j \omega)
\end{align}

under the assumption that both Fourier transforms $X(j \omega)$ and $Y(j \omega)$ exist.

The convolution theorem is very useful in the context of linear time-invariant (LTI) systems. The output signal $y(t)$ of an LTI system is given as the convolution of the input signal $x(t)$ with the impulse response $h(t)$. Hence, the signals and the system can be represented equivalently in the time and frequency domain

where $H(j \omega) = \mathcal{F} \{ h(t) \}$.
Calculation of the system response by transforming the problem into the Fourier domain can be beneficial since this replaces the evaluation of the convolution integral by a scalar multiplication. The (inverse) Fourier transforms are known for many signals or can be derived by applying the properties and theorems to standard signals and their transforms. This procedure simplifies the calculation of the system response significantly in many cases.

The convolution theorem can also be useful to derive an unknown Fourier transform. The key is here to express the signal as convolution of two other signals for which the Fourier transforms are known. This is illustrated by the following example.

#### Transformation of the triangular signal

A prominent example for the derivation of a Fourier transform by the convolution theorem is the [trianguar signal](https://en.wikipedia.org/wiki/Triangular_function)

\begin{equation}
\Lambda(t) = \begin{cases} 1- |t| & \text{for } |t| < 1 \\ 0 & \text{otherwise} \end{cases}
\end{equation}

which can be expressed as the convolution of two rectangular signals

\begin{equation}
\Lambda(t) = \text{rect}(t) * \text{rect}(t)
\end{equation}

Applying the convolution theorem yields

\begin{equation}
\mathcal{F} \{ \Lambda(t) \} = \text{sinc}\left( \frac{\omega}{2} \right) \cdot \text{sinc}\left( \frac{\omega}{2} \right) = \text{sinc}^2 \left( \frac{\omega}{2} \right)
\end{equation}

Both the signal and its transform are plotted for illustration

```python
x = sym.Piecewise((1 - abs(t), abs(t) < 1), (0, True))
X = sym.sinc(w/2)**2

sym.plot(x, (t, -2, 2), xlabel=r'$t$', ylabel=r'$\Lambda(t)$')
sym.plot(X, (w, -20, 20), xlabel=r'$\omega$',
         ylabel=r'$\mathcal{F} \{ \Lambda(t) \}$');
```

**Exercise**

* Extend above example by temporal scaling of the triangular signal.
* Derive the Fourier transform of a trapezoidal shaped signal by convolving two rectangular signals of different width.

### Temporal Shift Theorem

The [temporal shift](../continuous_signals/operations.ipynb#Temporal-Shift) $x(t-\tau)$ of a signal $x(t)$ can be expressed by a convolution with a shifted Dirac impulse

\begin{equation}
x(t - \tau) = x(t) * \delta(t - \tau)
\end{equation}

for $\tau \in \mathbb{R}$. This follows from the sifting property of the Dirac impulse. Applying a Fourier transform to the left- and right-hand side and exploiting the convolution theorem yields

\begin{equation}
\mathcal{F} \{ x(t - \tau) \} = X(j \omega) \cdot e^{- j \omega \tau}
\end{equation}

where $X(j \omega) = \mathcal{F} \{ x(t) \}$. Note that $\mathcal{F} \{ \delta(t - \tau) \} = e^{- j \omega \tau}$ can be derived from the definition of the Fourier transform together with the sifting property of the Dirac impulse. Above relation is known as shift theorem of the Fourier transform.

Expressing $X(j \omega)$ by its absolute value $|X(j \omega)|$ and phase $\varphi(j \omega)$ results in

\begin{equation}
\mathcal{F} \{ x(t - \tau) \} = | X(j \omega) | \cdot e^{j (\varphi(j \omega) - \omega \tau)}
\end{equation}

Temporal shifting of a signal does not change the absolute value of its spectrum but it subtracts the linear contribution $\omega \tau$ from the phase of its spectrum.

### Differentiation Theorem

Let's assume a signal $x(t)$ whose temporal derivative $\frac{d x(t)}{dt}$ exists. Using the [derivation property of the Dirac impulse](../continuous_signals/standard_signals.ipynb#Dirac-Impulse), the derivative of the signal can be expressed by the convolution

\begin{equation}
\frac{d x(t)}{dt} = \frac{d \delta(t)}{dt} * x(t)
\end{equation}

Fourier transformation of the left- and right-hand side together with the [convolution theorem](#Convolution-Theorem) yields the Fourier transform of the derivative of $x(t)$

\begin{equation}
\mathcal{F} \left\{ \frac{d x(t)}{dt} \right\} = j \omega \cdot X(j \omega)
\end{equation}

where $X(j \omega) = \mathcal{F} \{ x(t) \}$. The Fourier transform $\mathcal{F} \{ \frac{d \delta(t)}{dt} \} = j \omega$ can be derived by applying the definition of the Fourier transform together with the derivation property of the Dirac impulse. Above result is known as differentiation theorem of the Fourier transform. It states that the differentiation of a signal in the time domain is equivalent to a multiplication of its spectrum by $j \omega$. Splitting the spectrum of the differentiated signal into its magnitude and phase

\begin{equation}
j \omega \cdot X(j \omega) = |\omega \, X(j \omega)| \; e^{j (\varphi(j \omega) + \frac{\pi}{2} )}
\end{equation}

reveals that differentiation of a signal results in an attenuation of the higher frequencies of its magnitude spectrum and adds a phase shift of $\frac{\pi}{2}$.

The main applications of the differentiation theorem are the transformation of differential equations or the derivation of transforms of signals which can be expressed as derivatives of other signals. This latter is illustrated by the following example.

#### Transformation of the sign signal

The derivative of the [sign signal](../continuous_signals/standard_signals.ipynb#Sign-Signal) $\text{sgn}(t)$ [is given by the Dirac impulse](https://en.wikipedia.org/wiki/Sign_function#Properties)

\begin{equation}
\frac{d}{dt} \text{sgn}(t) = 2 \cdot \delta(t)
\end{equation}

Fourier transformation of the left- and right-hand side and application of the differentiation theorem yields

\begin{equation}
\mathcal{F} \{ \text{sgn}(t) \} = \frac{2}{j \omega}
\end{equation}

#### Transformation of the Heaviside signal

The transformation of the [Heaviside signal](../continuous_signals/standard_signals.ipynb#Heaviside-Signal) $\epsilon(t)$ is derived by expressing it in terms of the sign signal

\begin{equation}
\epsilon(t) = \frac{1}{2} ( \text{sgn}(t) + 1 )
\end{equation}

Using the linearity of the Fourier transform yields

\begin{equation}
\mathcal{F} \{ \epsilon(t) \} = \frac{1}{j \omega} + \pi \cdot \delta(\omega)
\end{equation}

The Fourier transformation of the Heaviside signal plays an important role in the context of [analytic signals](https://en.wikipedia.org/wiki/Analytic_signal) and their generation by the [Hilbert transform](https://en.wikipedia.org/wiki/Hilbert_transform).

### Integration Theorem

Let's assume a signal $x(t)$ for which the integral $\int_{-\infty}^{t} x(\tau) \; d\tau$ exists. The integration can be regarded as inverse operation to the temporal derivative. The integration can be represented as convolution with the rectangular signal $\epsilon(t)$ ([see integration theorem of the Laplace transform](../laplace_transform/theorems.ipynb#Integration-Theorem))

\begin{equation}
\int_{-\infty}^{t} x(\tau) \; d\tau = \int_{-\infty}^{\infty} x(\tau) \cdot \epsilon(t - \tau) \; d\tau = x(t) * \epsilon(t)
\end{equation}

Fourier transformation of the left- and right-hand side, application of the convolution theorem and using the Fourier transform of the Heaviside signal $\epsilon(t)$ yields the integration theorem of the Fourier transform

\begin{equation}
\mathcal{F} \left\{ \int_{-\infty}^{t} x(\tau) \; d\tau \right\}
= X(j \omega) * \left( \frac{1}{j \omega} + \pi \delta(\omega) \right)
= \frac{1}{j \omega} X(j \omega) + \pi X(0) \delta(\omega)
\end{equation}

**Example**

A signal whose amplitude increases linear over a finite temporal interval to a finite value can be defined as

\begin{equation}
x(t) = \begin{cases}
0 & \text{for } t < - \frac{1}{2} \\
t + \frac{1}{2} & \text{for } - \frac{1}{2} < t < \frac{1}{2} \\
1 & \text{for } t > \frac{1}{2}
\end{cases}
\end{equation}

Such functions are known as [ramp functions](https://en.wikipedia.org/wiki/Ramp_function). In order to derive the Fourier transform of the ramp signal $x(t)$ it is important to note that it can be expressed as integration over the rectangular signal $\text{rect}(t)$

\begin{equation}
x(t) = \int_{-\infty}^{t} \text{rect}(\tau) \; d\tau
\end{equation}

Now its straightforward to apply the integration theorem to yield the desired Fourier transform

\begin{equation}
X(j \omega) = \frac{1}{j \omega} \text{sinc}\left( \frac{\omega}{2} \right) + \pi \cdot \text{sinc}(0) \cdot \delta(\omega)
\end{equation}

### Multiplication Theorem

The Fourier transform of a multiplication of two signals $x(t)$ and $y(t)$ is given as

\begin{equation}
\mathcal{F} \{ x(t) \cdot y(t) \} = \frac{1}{2 \pi} X(j \omega) * Y(j \omega)
\end{equation}

The Fourier transform of a multiplication of two signals $x(t) \cdot y(t)$ is given by the convolution of their Fourier transforms $X(j \omega) = \mathcal{F} \{ x(t) \}$ and $Y(j \omega) = \mathcal{F} \{ y(t) \}$ weighted by $\frac{1}{2 \pi}$. Note, the convolution is performed with respect to the angular frequency $\omega$.

The multiplication theorem can be proven in a similar way as the convolution theorem by computing the inverse Fourier transform of the convolved spectra

\begin{equation}
\begin{split}
\mathcal{F}^{-1} \left\{ \frac{1}{2 \pi} X(j \omega) * Y(j \omega) \right\} &= \frac{1}{2 \pi} \int_{-\infty}^{\infty} \left( \frac{1}{2 \pi} \int_{-\infty}^{\infty} X(j \nu) \cdot Y(j (\omega-\nu)) \; d \nu \right) e^{j \omega t} \; d\omega \\
&= \frac{1}{2 \pi} \int_{-\infty}^{\infty} \left( \frac{1}{2 \pi} \int_{-\infty}^{\infty} Y(j (\omega-\nu)) \, e^{j \omega t} \; d\omega \right) X(j \nu) \; d\nu \\
&= y(t) \cdot \frac{1}{2 \pi} \int_{-\infty}^{\infty} X(j \nu) \, e^{j \nu t} \; d \nu \\
&= y(t) \cdot x(t)
\end{split}
\end{equation}

Alternatively the duality of the Fourier transform can be applied to the [convolution theorem](#Convolution-Theorem). Applications of the multiplication theorem include the modulation and windowing of signals. The former leads to the modulation theorem introduced later, the latter is illustrated by the following example.

**Example**

Windowing of signals is often applied to derive signals of finite duration from signals of infinite duration. Here the signal $x(t)$ is multiplied by a weighting function $w(t)$ in order to derive the finite length signal

\begin{equation}
y(t) = w(t) \cdot x(t)
\end{equation}

The following example considers the Fourier transform of a finite-length cosine signal. Following the concept of windowing, the signal can be defined as

\begin{equation}
y(t) = \text{rect}(a t) \cdot \cos(\omega_0 t)
\end{equation}

where $\text{rect}(a t)$ denotes the scaled [rectangular signal](../continuous_signals/standard_signals.ipynb#Rectangular-Signal) which is zero for $|t| > \frac{1}{2 a}$. The Fourier transform $Y(j \omega) = \mathcal{F} \{ y(t) \}$ is derived by the multiplication theorem, the [Fourier transform of the rectangular signal](definition.ipynb#Transformation-of-the-Rectangular-Signal), and the [Fourier transform of the cosine signal](properties.ipynb#Transformation-of-the-cosine-and-sine-signal) as

\begin{align}
Y(j \omega) &= \frac{1}{2 \pi} \cdot \frac{1}{|a|} \text{sinc}\left( \frac{\omega}{2 a} \right) * \pi \left( \delta(\omega + \omega_0) + \delta(\omega - \omega_0) \right) \\
&= \frac{1}{2 |a|} \left( \text{sinc}\left( \frac{\omega + \omega_0}{2 a} \right) + \text{sinc}\left( \frac{\omega - \omega_0}{2 a} \right) \right)
\end{align}

The latter identity results from the sifting property of the Dirac impulse. The convolution is computed in the following in `SymPy` by symbolic evaluation of the convolution integral.

```python
def convolve(x, y, var):
    tau = sym.symbols('tau')
    return sym.integrate(x.subs(var, tau) * y.subs(var, var - tau), (tau, -sym.oo, sym.oo))
```

Using above function, both the Fourier transform of the cosine signal and the rectangular window are convolved which each other

```python
w0, a = sym.symbols('omega0 a')

class rect(sym.Function):

    @classmethod
    def eval(cls, arg):
        return sym.Heaviside(arg + sym.S.Half) - sym.Heaviside(arg - sym.S.Half)

y = rect(a*t) * sym.cos(w0*t)
W = 1/abs(a) * sym.sinc(w/(2*a))
X = sym.pi*(sym.DiracDelta(w + w0) + sym.DiracDelta(w - w0))

Y = 1/(2*sym.pi) * convolve(W, X, w)
Y
```

Both the signal and its Fourier transform are plotted for specific values $\omega_0$, $a$

```python
values = {w0: 10, a: 1/2}

sym.plot(y.subs(values), (t, -5, 5), xlabel=r'$t$', ylabel=r'$y(t)$')
sym.plot(Y.subs(values), (w, -20, 20),
         xlabel=r'$\omega$', ylabel=r'$Y(j \omega)$');
```

**Exercise**

* Change the length $1/a$ of the signal by modifying the substitution in the example. How does the Fourier transform change if you decrease or increase the length $1/a$?

* What happens if you change the angular frequency $\omega_0$ of the signal?

* Assume a signal that is composed from a superposition of two finite length cosine signals with different frequencies. What qualitative condition has to hold that you can derive these frequencies from the spectrum of the superposition?

### Modulation Theorem

The complex modulation of a signal $x(t)$ is defined as $e^{j \omega_0 t} \cdot x(t)$ with $\omega_0 \in \mathbb{R}$. Its Fourier transform can be derived by applying the multiplication theorem

\begin{equation}
\mathcal{F} \left\{ e^{j \omega_0 t} \cdot x(t) \right\} = \frac{1}{2 \pi} \cdot 2 \pi \cdot \delta(\omega - \omega_0) * X(j \omega)
= X \big( j \, (\omega - \omega_0) \big)
\end{equation}

where $X(j \omega) = \mathcal{F} \{ x(t) \}$.  Above result states that the complex modulation of a signal leads to a shift of its spectrum. For $\omega_0 > 0$ the spectrum is shifted to higher frequencies. This lays the ground for wireless transmission of signals. Above result is known as modulation theorem.

**Example**

Lets assume a signal $x(t)$ which has a triangular spectrum $X(j \omega) = 2 \pi \cdot \Lambda(\omega)$. Due to the definition of the triangular function, the spectrum has only non-zero contributions for $|\omega| < 1$. Applying the duality principle to the [Fourier transform of $\Lambda(t)$ derived above](#Transformation-of-the-triangular-signal) yields the corresponding signal $x(t) = \mathcal{F}^{-1} \{ X(j \omega) \}$

\begin{equation}
x(t) = \text{sinc}^2 \left( \frac{t}{2} \right)
\end{equation}

Now lets consider the complex modulation $y(t) = e^{-j \omega_0 t} \cdot x(t)$ of the signal $x(t)$. The Fourier transform of the modulated signal is given as

\begin{equation}
\mathcal{F} \{ e^{j \omega_0 t} \cdot x(t) \} = 2 \pi \cdot \Lambda(\omega - \omega_0)
\end{equation}

The spectrum of the modulated signal has non-zero contributions for $\omega_0 -1 < \omega < \omega_0 + 1$. For large $\omega_0$ it only occupies a small part of the spectrum. This is an important property of signals used for wireless transmission. The signal $y(t)$ is defined

```python
y = sym.exp(sym.I*w0*t) * sym.sinc(t/2)**2
y
```

and the real part of the complex modulated signal is plotted for illustration

```python
values = {w0: 5}
sym.plot(sym.re(y.subs(values)), (t, -20, 20),
         xlabel=r'$t$', ylabel=r'$\Re \{ y(t) \}$');
```

**Exercise**

* Derive the spectrum of the signal $x(t)$ modulated by $\cos(\omega_0 t)$. Hint: Express $\cos(\omega_0 t)$ by Euler's formula and exploit the linearity of the Fourier transform. What is different in the resulting spectrum when comparing to above result?

### Parseval's Theorem

[Parseval's theorem](https://en.wikipedia.org/wiki/Parseval's_theorem) relates the energy of a signal in the time domain to its spectrum. The squared absolute value of a signal $x(t)$ represents its instantaneous power. It can be expressed as

\begin{equation}
| x(t) |^2 = x(t) \cdot x^*(t)
\end{equation}

where $x^*(t)$ denotes the complex conjugate of $x(t)$. Fourier transformation of the right-hand side and application of the multiplication theorem results in

\begin{equation}
\mathcal{F} \{ x(t) \cdot x^*(t) \} = \frac{1}{2 \pi} \cdot X(j \omega) * X^*(-j \omega)
\end{equation}

Introducing the definition of the Fourier transform and the convolution

\begin{equation}
\int_{-\infty}^{\infty} x(t) \cdot x^*(t) \, e^{-j \omega t} \; dt =
\frac{1}{2 \pi} \int_{-\infty}^{\infty} X(j \eta) \cdot X^*(-j (\omega - \eta)) \, d\eta
\end{equation}

and setting $\omega = 0$ yields the desired relation

\begin{equation}
\int_{-\infty}^{\infty} | x(t) |^2 \; dt = \frac{1}{2 \pi} \int_{-\infty}^{\infty} | X(j \omega) |^2 \; d\omega
\end{equation}

The integral over the squared absolute signal in the time-domain is equal to the integral over its squared absolute spectrum divided by $2 \pi$. Since the left-hand side represents the energy $E$ of the signal $x(t)$, Parseval's theorem states that the energy can be computed either in the time or spectral domain by integrating over the squared absolute value of the signal/spectrum. Parseval's theorem can be generalized to the [Plancherel theorem](https://en.wikipedia.org/wiki/Plancherel_theorem) which considers the product of two signals $x(t) \cdot y^*(t)$.

**Example**

The energy of the rectangular signal $\text{rect}(t)$ is computed in the time and spectral domain by evaluation of the respective integrals. First the time domain is evaluated

\begin{equation}
E_t = \int_{-\infty}^{\infty} | \text{rect}(t) |^2 \; dt = \int_{- \frac{1}{2}}^{\frac{1}{2}} 1 \; dt
\end{equation}

```python
E_t = sym.integrate(1, (t, -sym.S.Half, sym.S.Half))
E_t
```

The energy in the spectral domain is computed by evaluating

\begin{equation}
E_f = \frac{1}{2 \pi} \int_{-\infty}^{\infty} \left| \text{sinc}\left(\frac{\omega}{2}\right) \right|^2 \; d\omega
\end{equation}

```python
E_f = 1/(2*sym.pi) * sym.integrate(sym.sinc(w/2)**2, (w, -sym.oo, sym.oo))
E_f
```

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Continuous- and Discrete-Time Signals and Systems - Theory and Computational Examples*.


# _QECC:_ Shor's 9-qubit algorithm

In the following cells we would perform the Shor's 9-qubit algorithm, a quantum error correction circuit which can diagnose and repair one bit and/or phase-flip error in a 9-qubit circuit, based on the concatenation of 3-qubit bit and phase-flip QECC.

As every unitary error applied to a state can be described as a linear combination of the *Pauli group matrices*, i.e. $\{I, \sigma_i\}$, we can correct our errors using $X$ and $Z$-gates. See that $Y = iXZ$, so a $Y$-error can be expressed as a concatenation of $X$ and $Z$-errors plus a $\pi/2$ complex global phase.

---

**Author: Sebastián V. Romero** ([sebastian.vidal@rai.usc.es](mailto:sebastian.vidal@rai.usc.es))

*Last update:* 4th April 2021

## Package and function imports

```python
%run QECC_Shor9Q_functions.ipynb
```

## Let's draw the circuit!

### Initialization

We start registering our 9 qubits and 8 ancillas for the error correction. Also, we initialize our entangled initial vector state $|\phi\rangle = a |0\rangle + b |1\rangle$, defining the constants and normalizing the created state.

```python
# Qubits for our entangled state
q0 = QuantumRegister(3, name = 'q0')
q1 = QuantumRegister(3, name = 'q1')
q2 = QuantumRegister(3, name = 'q2')

# Ancillas for bit-flip errors
bf0 = AncillaRegister(2, name = 'bf0')
bf1 = AncillaRegister(2, name = 'bf1')
bf2 = AncillaRegister(2, name = 'bf2')

# Ancillas for phase-flip errors
pf = AncillaRegister(2, name = 'pf')

circuit = QuantumCircuit(q0, bf0, q1, bf1, q2, bf2, pf)

# Define our initial normalized state
initial_state = normalize_state(1.0 + 1.0j, 2.0 + 0.0j)
circuit.initialize(initial_state, 0)

# Draws the circuit
# circuit.draw(output = 'mpl', plot_barriers = False)
```

    <qiskit.circuit.instructionset.InstructionSet at 0x7fa39c366ca0>

### Obtain the initial 9-qubit state

As a first part, we add a $CNOTNOT$ in qubits 0, 3 and 6 in order to extend our *physical basis* $\{ |0\rangle, |1\rangle \}$ to the *logical basis* spanning the *codespace* $\mathcal{C} = \mathrm{span}\{ |0\rangle_L = |000\rangle, |1\rangle_L = |111\rangle \}$. Explicitly,

\begin{equation}
|\phi\rangle = a|0\rangle + b|1\rangle \rightarrow |\phi\rangle_L = a|000\rangle + b|111\rangle.
\end{equation}

Then, we add in the same qubits three Hadamard gates to go from $\{ |0\rangle_L, |1\rangle_L \}$ basis to $\{ |+\rangle_L = |+\rangle|+\rangle|+\rangle, |-\rangle_L = |-\rangle|-\rangle|-\rangle \}$, being $|\pm\rangle = \frac{1}{\sqrt{2}} \left( |0\rangle \pm |1\rangle \right)$, the $\sigma_x$ basis. It's more convenient to use this basis since $X |\pm\rangle = \pm |\pm\rangle$ ($X$ acts like a diagonal matrix in its own basis) and $Z |\pm\rangle = |\mp\rangle$ (bit-flip operation). Then, we have as final state of this part

\begin{equation}
\begin{aligned}
|\psi\rangle = &\frac{a}{2\sqrt{2}} (|000\rangle + |111\rangle) \otimes (|000\rangle + |111\rangle) \otimes (|000\rangle + |111\rangle) \\
+&\frac{b}{2\sqrt{2}} (|000\rangle - |111\rangle) \otimes (|000\rangle - |111\rangle) \otimes (|000\rangle - |111\rangle) \\
= & a |{+++}\rangle \otimes |{+++}\rangle \otimes |{+++}\rangle + b |{---}\rangle \otimes |{---}\rangle \otimes |{---}\rangle.
\end{aligned}
\end{equation}

```python
# Add controlled-NOTNOT to qubits 0, 3 and 6
circuit.append(cnotnot(), [q0[0], q1[0], q2[0]])

# Add Hadamard gates to qubits 0, 3 and 6
circuit.h(q0[0])
circuit.h(q1[0])
circuit.h(q2[0])

# Add controlled-NOTNOT in our three packs of three qubits
circuit.append(cnotnot(), [q0[0], q0[1], q0[2]])
circuit.append(cnotnot(), [q1[0], q1[1], q1[2]])
circuit.append(cnotnot(), [q2[0], q2[1], q2[2]])

circuit.barrier(q0, bf0, q1, bf1, q2, bf2, pf)

circuit.draw(output = 'mpl', plot_barriers = False)
```

```python
_ = get_state(circuit, fancy_print = True)
```

    (0.4330127019+0.1443375673j) |00000000000000000>   (-0.1443375673+0.1443375673j) |11100000000000000>   (-0.1443375673+0.1443375673j) |00000111000000000>   (0.4330127019+0.1443375673j) |11100111000000000>   (-0.1443375673+0.1443375673j) |00000000001110000>   (0.4330127019+0.1443375673j) |11100000001110000>   (0.4330127019+0.1443375673j) |00000111001110000>   (-0.1443375673+0.1443375673j) |11100111001110000>

### Add some noise to our circuit

We add a random noise gate in order to apply some errors to the $|\psi\rangle$ state.

```python
# Add a source of random noise in our channels
circuit.append(random_noise(reveal_error = True),
                            [q0[0], q0[1], q0[2],
                             q1[0], q1[1], q1[2],
                             q2[0], q2[1], q2[2]])

circuit.barrier(q0, bf0, q1, bf1, q2, bf2, pf)

circuit.draw(output = 'mpl', plot_barriers = False)
```

```python
_ = get_state(circuit, fancy_print = True)
```

    (-0.1443375673+0.4330127019j) |00000000001000000>   (-0.1443375673-0.1443375673j) |11100000001000000>   (-0.1443375673-0.1443375673j) |00000111001000000>   (-0.1443375673+0.4330127019j) |11100111001000000>   (0.1443375673+0.1443375673j) |00000000000110000>   (0.1443375673-0.4330127019j) |11100000000110000>   (0.1443375673-0.4330127019j) |00000111000110000>   (0.1443375673+0.1443375673j) |11100111000110000>

### Bit-flip error correction

The two initial Hadamard gates $H$ let change $|0\rangle \rightarrow |+\rangle$, having our controlls in the ancillas and the *syndrome detectors* in our qubit channels. We reverse that with another two Hadamard gates. It's important to see the different ancillas states possibilities, as we summarize in the next table for the first pack of three qubits (it's just the same for the another two packs). The subindices denote in which qubit of the pack the error is applied.

| Error | Syndrome | Ancilla | $$Z_0 Z_1$$ | $$Z_1 Z_2$$ |
| :-: | :-: | :-: | :-: | :-: |
| $$I$$   | $$a |000\rangle + b |111\rangle$$ | $$|00\rangle$$ | + | + |
| $$X_0$$ | $$a |100\rangle + b |011\rangle$$ | $$|10\rangle$$ | - | + |
| $$X_1$$ | $$a |010\rangle + b |101\rangle$$ | $$|11\rangle$$ | - | - |
| $$X_2$$ | $$a |001\rangle + b |110\rangle$$ | $$|01\rangle$$ | + | - |

```python
# Add Hadamard gates for the bit-flip ancillas
for index in range(2):
    circuit.h(bf0[index])
    circuit.h(bf1[index])
    circuit.h(bf2[index])

# Add controlled-ZZ gates in our ancillas
# First pack of three qubits
circuit.append(czz(), [bf0[0], q0[0], q0[1]])
circuit.append(czz(), [bf0[1], q0[1], q0[2]])

# Second pack of three qubits
circuit.append(czz(), [bf1[0], q1[0], q1[1]])
circuit.append(czz(), [bf1[1], q1[1], q1[2]])

# Third pack of three qubits
circuit.append(czz(), [bf2[0], q2[0], q2[1]])
circuit.append(czz(), [bf2[1], q2[1], q2[2]])

circuit.barrier(q0, bf0, q1, bf1, q2, bf2, pf)

# Add Hadamard gates for the bit-flip ancillas
for index in range(2):
    circuit.h(bf0[index])
    circuit.h(bf1[index])
    circuit.h(bf2[index])

# Add double controlled gates to correct bit-flip errors
# First pack of three qubits
circuit.append(invccnot(), [bf0[1], bf0[0], q0[0]])
circuit.ccx(bf0[0], bf0[1], q0[1])
circuit.append(invccnot(), [bf0[0], bf0[1], q0[2]])

# Second pack of three qubits
circuit.append(invccnot(), [bf1[1], bf1[0], q1[0]])
circuit.ccx(bf1[0], bf1[1], q1[1])
circuit.append(invccnot(), [bf1[0], bf1[1], q1[2]])

# Third pack of three qubits
circuit.append(invccnot(), [bf2[1], bf2[0], q2[0]])
circuit.ccx(bf2[0], bf2[1], q2[1])
circuit.append(invccnot(), [bf2[0], bf2[1], q2[2]])

circuit.barrier(q0, bf0, q1, bf1, q2, bf2, pf)

# BIT-FLIP ERRORS SOLVED! Let's go with the phase-flip errors

circuit.draw(output = 'mpl', plot_barriers = False)
```

```python
_ = get_state(circuit, fancy_print = True)
```

    (-0.1443375673+0.4330127019j) |00000000000001000>   (-0.1443375673-0.1443375673j) |11100000000001000>   (-0.1443375673-0.1443375673j) |00000111000001000>   (-0.1443375673+0.4330127019j) |11100111000001000>   (0.1443375673+0.1443375673j) |00000000001111000>   (0.1443375673-0.4330127019j) |11100000001111000>   (0.1443375673-0.4330127019j) |00000111001111000>   (0.1443375673+0.1443375673j) |11100111001111000>

### Phase-flip error correction

The idea is pretty similar to the bit-flip error correction part. The two initial Hadamard gates $H$ let change $|0\rangle \rightarrow |+\rangle$, having our controlls in the ancillas and the *syndrome detectors* in our qubit channels. We reverse that with another two Hadamard gates. It's important to see the different ancillas states possibilities, as we summarize in the next table for the three packs of qubits. Now, the subindices are used to denote in which pack of qubits the phase-flip error is applied.

| Error | Syndrome | Ancilla | $$X_0 X_1 X_2 X_3 X_4 X_5$$ | $$X_3 X_4 X_5 X_6 X_7 X_8$$ |
| :-: | :-: | :-: | :-: | :-: |
| $$I$$              | $$a |000\rangle + b |111\rangle$$ | $$|00\rangle$$ | + | + |
| $$Z_\mathrm{1st}$$ | $$a |100\rangle + b |011\rangle$$ | $$|10\rangle$$ | - | + |
| $$Z_\mathrm{2nd}$$ | $$a |010\rangle + b |101\rangle$$ | $$|11\rangle$$ | - | - |
| $$Z_\mathrm{3rd}$$ | $$a |001\rangle + b |110\rangle$$ | $$|01\rangle$$ | + | - |

```python
# Add Hadamard gates for the phase-flip ancillas
for index in range(2):
    circuit.h(pf[index])

# Add controlled-6X gates in our ancillas
circuit.append(c6x(), [pf[0], q0[0], q0[1], q0[2], q1[0], q1[1], q1[2]])
circuit.append(c6x(), [pf[1], q1[0], q1[1], q1[2], q2[0], q2[1], q2[2]])

# Add Hadamard gates for the phase-flip ancillas
for index in range(2):
    circuit.h(pf[index])

# Add double controlled gates to correct phase-flip errors
circuit.append(invccz(), [pf[1], pf[0], q0[0]])
circuit.append(ccz(), [pf[1], pf[0], q1[0]])
circuit.append(invccz(), [pf[0], pf[1], q2[0]])

circuit.barrier(q0, bf0, q1, bf1, q2, bf2, pf)

# PHASE-FLIP ERRORS SOLVED! Let's go for the final manipulations in our state

circuit.draw(output = 'mpl', plot_barriers = False, fold = 40)
```

```python
_ = get_state(circuit, fancy_print = True)
```

    (-0.1443375673+0.4330127019j) |00000000000001001>   (-0.1443375673-0.1443375673j) |11100000000001001>   (-0.1443375673-0.1443375673j) |00000111000001001>   (-0.1443375673+0.4330127019j) |11100111000001001>   (-0.1443375673-0.1443375673j) |00000000001111001>   (-0.1443375673+0.4330127019j) |11100000001111001>   (-0.1443375673+0.4330127019j) |00000111001111001>   (-0.1443375673-0.1443375673j) |11100111001111001>

### Let's get our corrected final state!

We "reverse" the steps that we did in our first block of gates, where we prepare our $|\psi\rangle$ state.

```python
# Add controlled-NOTNOT in our three packs of three qubits
circuit.append(cnotnot(), [q0[0], q0[1], q0[2]])
circuit.append(cnotnot(), [q1[0], q1[1], q1[2]])
circuit.append(cnotnot(), [q2[0], q2[1], q2[2]])

# Add Hadamard gates to qubits 0, 3 and 6
circuit.h(q0[0])
circuit.h(q1[0])
circuit.h(q2[0])

# Add controlled-NOTNOT to qubits 0, 3 and 6
circuit.append(cnotnot(), [q0[0], q1[0], q2[0]])

circuit.draw(output = 'mpl', plot_barriers = False, fold = 40)
```

### Get the state

```python
# Get the state vector
final_state = get_state(circuit, fancy_print = True)
```

    (-0.4082482905+0.4082482905j) |00000000000001001>   0.8164965809j |10000000000001001>

We can check if our expected ancillas correspond to our *final_state* ancillas

```python
check_error_to_ancilla(final_state)
```

    Error  Anc. expected  Anc. state  Equal?
    ----------------------------------------
     X0     00             00          True
     X1     00             00          True
     X2     10             10          True
     Z      01             01          True

### Latex code of the circuit

Prints and saves the latex source to a file. Uncomment the following line of code if you want to use it.

```python
# print(circuit.draw(output = 'latex_source', filename = '/tmp/circuit.tex'))
```

## Version information

```python
import qiskit.tools.jupyter
%qiskit_version_table
```

Version Information

Qiskit Software	Version

Qiskit	0.24.1

Terra	0.16.4

Aer	0.7.6

Ignis	0.5.2

Aqua	0.8.2

IBM Q Provider	0.12.2

System information

Python	3.8.8 (default, Feb 24 2021, 21:46:12)
[GCC 7.3.0]

OS	Linux

CPUs	4

Memory (Gb)	10.91162109375

Sun Apr 04 17:30:45 2021 CEST

```python

```


# Hückel Method
*Roberto Di Remigio*, *Luca Frediani*

With the [Hückel method] we aim at calculating the energy levels of conjugated $\pi$ systems.
The method is limited to planar, conjugated hydrocarbons and includes only the $\pi$ electrons in the system in the quantum mechanical treatment. The $\sigma$ electrons are ignored, as they are at much lower energies and will not affect molecular properties significantly.
The molecular orbitals (MOs) are built as a linear combination of atomic orbitals (LCAO),
where the atomic orbitals (AOs) in question are the $p_z$ orbitals on each of the carbon atoms.

[Hückel method]: https://en.wikipedia.org/wiki/H%C3%BCckel_method

## Variation principle and secular equations

If $N$ is the number of carbon atoms in the conjugated hydrocarbon, the trial wavefunction is thus written as:
\begin{equation}
\Psi = \sum_{i=1}^{N}c_i\chi_i
\end{equation}
where the index $i$ runs over carbon atoms and $\chi_i$ is the $p_z$ orbital on the $i$-th atom in the molecule.
The energy, i.e. the expectation value of the many-electron Hamiltonian, is:
\begin{equation}
E(\mathbf{c}^\dagger, \mathbf{c}) = \frac{\langle \Psi | H | \Psi \rangle}{\langle \Psi | \Psi \rangle}
= \frac{\sum_{i,j=1}^{N} c_i^*H_{ij}c_j }{\sum_{i, j=1}^{N}c_i^*S_{ij}c_j}
= \frac{\mathbf{c}^\dagger\mathbf{H}\mathbf{c}}{\mathbf{c}^\dagger\mathbf{S}\mathbf{c}}
\end{equation}
where the Hamiltonian and overlap matrices were introduced:
\begin{alignat}{2}
H_{ij} = \langle \chi_i | H |\chi_j\rangle \quad& S_{ij} = \langle \chi_i | \chi_j \rangle
\end{alignat}
The variation principle guarantees that minimization of the energy function will lead to the optimal value of the coefficients, in the chosen LCAO basis.
This reduces to the generalized eigenvalue problem:
\begin{equation}
\mathbf{H}\mathbf{c} = \mathbf{S}\mathbf{c}E
\end{equation}
where the **eigenvector** $\mathbf{c}$ is the set of coefficients in the linear combination describing the MO with energy **eigenvalue** $E$.
Finding the eigenvalues can be achieved by calculating the determinant of $\mathbf{H} - E\mathbf{S}$ and setting it to zero:
\begin{equation}
\det(\mathbf{H} - E\mathbf{S}) = 0
\end{equation}

## The eigenvalue problem in Hückel method

As already mentions, the molecular orbitals (MOs) are built as a linear combination of atomic orbitals (LCAO),
where the atomic orbitals (AOs) in question are the $p_z$ orbitals on each of the carbon atoms.
A schematic view of the model is given in the Figure below.

The Hamiltonian and overlap matrices are constructed from a simple set of rules:

* all overlap integrals between distinct orbitals are zero:
\begin{equation}
S_{ii} = 1 \quad S_{ij} = 0 \,\,\,\text{for}\, i\neq j
\end{equation}
* all diagonal Hamiltonian matrix elements are equal:
\begin{equation}
H_{ii} = \alpha \,\,\,\forall i
\end{equation}
where $\alpha$ is a parameter of the model. It is usually set to the energy of an isolated electron in a $2p_z$ orbital. $\alpha$ is thus a **negative number**.
* all off-diagonal Hamiltonian matrix elements are equal. They are nonzero for **adjacent** carbon atoms and zero otherwise:
\begin{equation}
H_{i, i+1} = \beta \quad H_{i, i+n} = 0 \,\,\,\text{for}\,n\geq 2
\end{equation}
where $\beta$ is a **negative** parameter that can be determined from experiment ($\beta = -75 \,\mathrm{kJ}\cdot\mathrm{mol}^{-1}$)

Once the connectivity and the values of the $\alpha$ and $\beta$ parameters are known one can set up the Hückel matrix and the correspoding secular problem.

### A simple case: ethene

Ethene has two carbon atoms. The Hückel matrix is:
\begin{equation}
\begin{pmatrix}
\alpha & \beta \\
\beta  & \alpha
\end{pmatrix}
\end{equation}
and its secular problem:
\begin{equation}
\det \begin{pmatrix}
(\alpha - E) & \beta \\
\beta  & (\alpha - E)
\end{pmatrix}
= (\alpha - E)^2 - \beta^2 = 0
\end{equation}
The solution of the second order equation gives the energies for the highest occupied molecular orbital $E_-$ (HOMO) and the lowest unoccupied molecular orbital $E_+$ (LUMO):
\begin{equation}
E_- = \alpha -\beta \quad E_+ = \alpha + \beta
\end{equation}
Inserting these energies in the eigenvalue equation will lead to two different systems of equations whose solution determines the coefficient in the MOs linear combinations.
Notice that the coefficients are determined up to normalization, meaning that only by imposing a normalization condition the solution to the eigenvalue problem will be unique.

## Exercise: a small script for Hückel method

Write a Python function to calculate eigenvalues and eigenvectors for the Hückel matrix of a generic conjugated hydrocarbon.
The function should take four parameters:
* the number of carbon atoms in the hydrocarbon
* the value of the $\alpha$ parameter
* the value of the $\beta$ parameter
* an array specifying the connectivity of the carbon atoms

The connectivity array describes which carbon atoms are connected _via_ a double bond. This is essential, since it determines which entries in the Hückel matrix are nonzero!
As an example, for the ethene molecule $[1, 2]$ would be the connectivity array. For the butadiene molecule one would instead have $[1, 2, 2, 3, 3, 4]$ which means that atom $1$ is connected to atom $2$ by a double bond, atom $2$ and $3$ are connected by a single bond and atom $3$ and $4$ are again connected by a double bond.
Why so? Because butadiene has 4 carbon atoms, hence only two double bonds can be present. Since these have to be _alternating_ carbon $2$ and carbon $3$ must be connected by a single bond.
To help with determining the connectivity array, draw the molecule and number all carbon atoms.
The function will, of course, use NumPy for manipulating matrices and vectors.
The module `linalg` includes the functions necessary for linear algebra manipulations. In our case, we're interested in finding eigenvalues and eigenvectors of the Hückel matrix with the [`eig` function]

```Python
def huckel(n, alpha, beta, connectivity):
    """ Huckel method

    n -- number of carbon atoms
    alpha -- Coulomb integral
    beta --  resonance integral
    connectivity -- an array of integers defining the connectivity
    """
    import numpy as np
    from numpy import linalg as LA

    H = np.zeros((n, n))
    # Fill the matrix!
    ## Fill diagonal

    ## Fill off-diagonal

    # Print H
    print('Huckel matrix is {}'.format(H))

    # Diagonalize H
    energies, orbitals = LA.eig(H)
    # Sort eigenvalues and eigenvectors in ascending order
    sort_perm = energies.argsort()
    energies.sort()     # <-- This sorts the list in place.
    orbitals = evecs[:, sort_perm]

    # Return energies and orbitals
    return energies, orbitals
```

[`eig` function]: http://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.eig.html#numpy.linalg.eig

```python
def huckel(n, alpha, beta, connectivity):
    """ Huckel method

    n -- number of carbon atoms
    alpha -- Coulomb integral
    beta --  resonance integral
    connectivity -- an array of integers defining the connectivity
    """
    import numpy as np
    from numpy import linalg as LA

    H = np.zeros((n, n))
    # Fill the matrix!
    ## Fill diagonal

    ## Fill off-diagonal

    # Print H
    print('Huckel matrix is \n{}'.format(H))

    # Diagonalize H
    energies, evecs = LA.eig(H)
    # Sort eigenvalues and eigenvectors in ascending order:q
    sort_perm = energies.argsort()
    energies.sort()     # <-- This sorts the list in place.
    orbitals = evecs[:, sort_perm]

    # Return energies and orbitals
    return energies, orbitals

energies, orbitals = huckel(4, -5.0, -75.0, [1, 2, 2, 3, 3, 4])
print('Huckel model energies \n{}'.format(energies))
print('Huckel model orbitals \n{}'.format(orbitals))
```

    Huckel matrix is
    [[ -5. -75.   0.   0.]
     [-75.  -5. -75.   0.]
     [  0. -75.  -5. -75.]
     [  0.   0. -75.  -5.]]
    Huckel model energies
    [-126.35254916  -51.35254916   41.35254916  116.35254916]
    Huckel model orbitals
    [[ 0.37174803 -0.60150096  0.60150096 -0.37174803]
     [ 0.60150096 -0.37174803 -0.37174803  0.60150096]
     [ 0.60150096  0.37174803 -0.37174803 -0.60150096]
     [ 0.37174803  0.60150096  0.60150096  0.37174803]]

## Visualizing the molecular orbitals

We will use the packages [`chemview`] and [`chemlab`] to visualize the molecular orbitals, as in [this example]. The coefficients are calculated from our small function for the Hückel method. The technicalities of the definition of the basis functions are not important in this context. You can find more details [here].

[`chemview`]: http://chemview.readthedocs.org/en/latest/
[`chemlab`]: http://chemlab.readthedocs.org/en/latest/index.html
[this example]: http://gabrielelanaro.github.io/blog/2015/01/14/visualizing-molecular-orbitals-ipython-notebook.html
[here]: https://en.wikipedia.org/wiki/Basis_set_(chemistry)

```python
import numpy as np

from chemlab.db.cirdb import CirDB
from chemlab.qc import molecular_orbital

from chemview import enable_notebook, MolecularViewer
enable_notebook()

# To get molecule by name
ethene = CirDB().get("molecule", "ethene")
# Huckel's method results for ethene
orbitals = np.array([[ 0.70710678,  0.70710678],
 [-0.70710678,  0.70710678]])

# Initialize molecular viewer
mv = MolecularViewer(ethene.r_array, { 'atom_types': ethene.type_array,
                                         'bonds': ethene.bonds })
# Define rendering type
mv.wireframe()

# Define basis set
# p-functions for carbon from STO-3G basis set
sto3g = [
    # atom 1
[
('P', [
(2.941249, 0.155916), (0.683483, 0.607684), (0.222290, 0.391957), ]),
],
    # atom 2
[
('P', [
(2.941249, 0.155916), (0.683483, 0.607684), (0.222290, 0.391957), ]),
]
]

# Define molecular orbital to be plotted
# Notice that we pass only the coordinates of the carbon atoms!
coords = np.array([ethene.r_array[0], ethene.r_array[1]])
# We need to fudge the coefficient arrays with zeros (p_x and p_y are absent!)
HOMO_coeff = np.array([0.0, 0.0, orbitals[0][0], 0.0, 0.0, orbitals[0][1]])
LUMO_coeff = np.array([0.0, 0.0, orbitals[1][0], 0.0, 0.0, orbitals[1][1]])
HOMO = molecular_orbital(coords, HOMO_coeff, sto3g)
LUMO = molecular_orbital(coords, LUMO_coeff, sto3g)
# Add isosurface of MO with positive isovalue
mv.add_isosurface(HOMO, isolevel=0.3, color=0xff0000)
# Add isosurface of MO with negative isovalue
mv.add_isosurface(HOMO, isolevel=-0.3, color=0x0000ff)
# This is necessary to actually get a picture of the molecule
mv
```

    HDF backend is not available because pytables is not installed
    cclib not found. Install cclib for more handlers.

    /home/roberto/.virtualenvs/py3/lib/python3.4/site-packages/traitlets/traitlets.py:565: FutureWarning: comparison to `None` will result in an elementwise object comparison in the future.
      silent = bool(old_value == new_value)

```python

```