<p align="center">

</p>

## Machine Learning Introduction

### Predictive Machine Learning Modeling in Python with scikit-learn

#### Michael Pyrcz, Associate Professor, University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1) | [GeostatsPy](https://github.com/GeostatsGuy/GeostatsPy)

#### Machine Learning-based Prediction

Here's a simple workflow, demonstration of training and tuning a machine learning model with a simple dataset.

* I provide this as the most basic, minimum Python workflow to assist those learning predictive machine learning

* I demonstration model training and tuning (by-hand and automatically), including the pipeline approach

* Given the 1 predictor feature and 1 response for ease of data and model visualization

This provides an opportunity to run and visualize a variety of machine learning models for experiential learning.

#### Machine Learning-based Prediction

We build predictive machine learning models with supervised learning:

**Prediction**

* non-parametric method for regression (continuous response) and classification (categorical response)

* a function $\hat{f}$ of the nearest $k$ training data in predictor feature space such that we predict a response feature $Y$ from a set of predictor features $X_1,\ldots,X_m$.

* the prediction is of the form:

\begin{equation}
\hat{Y} = \hat{f}(X_1,\ldots,X_m) + \epsilon
\end{equation}

where $\epsilon$ is an error term.

**Suppervised Learning**

* the response feature, $Y$, is available over the training and testing data

#### The Machine Learning Models

This workflow includes the following model:

* decision tree regression

For more information about decision trees check out this [decision tree lecture](https://github.com/GeostatsGuy/PythonNumericalDemos/blob/master/SubsurfaceDataAnalytics_DecisionTree.ipynb) and the associated demonstration [decision tree workflow](https://github.com/GeostatsGuy/PythonNumericalDemos/blob/master/SubsurfaceDataAnalytics_DecisionTree.ipynb).

#### Getting Started

Here's the steps to get setup to run this demonstration:

1. **Install Anaconda 3** on your machine from https://www.anaconda.com/download/.
2. **Open Jupyter Notebook**, look for the Jupyter app on your system after installing Anaconda 3.
3. **Load this Workflow** found here [PythonDataBasics_PredictiveMachineLearning.ipynb](https://github.com/GeostatsGuy/PythonNumericalDemos/blob/master/PythonDataBasics_PedictiveMachineLearning.ipynb).
4. **Load the data**, this workflow retreives the data from my GitHub [GeoDataSets Repository](https://github.com/GeostatsGuy/GeoDataSets). If you want to work locally, you will need to first download the data file to your working directory. The data file is found here, [1D_Porosity.csv](https://github.com/GeostatsGuy/GeoDataSets/blob/master/1D_Porosity.csv). Code is provided below to set the working directory and to load the data locally.

#### Import Required Packages

We will also need some standard packages. These should have been installed with Anaconda 3.

```python
import os                                                   # to set current working directory
import pandas as pd                                         # DataFrames and plotting
import numpy as np                                          # arrays and matrix math
import matplotlib.pyplot as plt                             # plotting
```

If you get a package import error, you may have to first install some of these packages. This can usually be accomplished by opening up a command window on Windows and then typing 'python -m pip install [package-name]'. More assistance is available with the respective package docs.

#### Set the Random Number Seed

These workflows and models may use random processes, e.g.:

* random train and test split with cross validation and k-fold cross validation
* random selection of a subset of features to consider for the next split

```python
seed = 73073                                                # random number seed
np.random.seed(seed)                                        # scikit-learn uses the NumPy seed
```

#### Declare functions

Let's define a couple of functions to streamline plotting correlation matrices and visualization of a decision tree regression model.

```python
#none needed
```

#### Set the working directory

I always like to do this so I don't lose files and to simplify subsequent read and writes (avoid including the full address each time).

```python
# not needed currently
#os.chdir("c:/PGE383")                                       # set the working directory
```

You will have to update the part in quotes with your own working directory and the format is different on a Mac (e.g. "~/PGE").

#### Read the data table

We use a data "1D_Porosity.csv" comma delimited file from my GitHub [GeoDataSets Respository](https://github.com/GeostatsGuy/GeoDataSets).

* this code loads the data directly from my GitHub repository.
* if you have the data file available locally (i.e. you are not connected to the internet) set the working directory to the location with the data file and use the second line below

```python
my_data = pd.read_csv("https://raw.githubusercontent.com/GeostatsGuy/GeoDataSets/master/1D_Porosity.csv") # load the comma delimited data file
#my_data = pd.read_csv("1D_Porosity.csv")                   # load the comma delimited data file locally
X = pd.DataFrame(data = my_data.loc[:,'Depth'])             # ensure X and y features are DataFrames
y = pd.DataFrame(data = my_data.loc[:,'Nporosity'])
X.head(n=2); y.head(n=2)                                    # preview the predictor feature
print('Loaded ' + str(len(my_data)) + ' samples, with features = ' + str(my_data.columns.values) + '.')
```

    Loaded 40 samples, with features = ['Depth' 'Nporosity'].

#### Plot the Available Data

Let's do a simple scatter plot to check the data.

```python
plt.scatter(X,y,color='red',s=30,edgecolor='black',alpha=0.2); plt.xlabel('Depth (m)'); plt.ylabel('Standardized Porosity'); plt.title('All Data'); plt.xlim([0,10.25]); plt.ylim([-2.5,2.5])
plt.subplots_adjust(left=0.0, bottom=0.0, right=1.5, top=0.7, wspace=0.2, hspace=0.2);
```

#### Fast Forward Through Data Analytics

This workflow is focussed on building predictive machine learning models

* for brevity we skip data checking, data analytics, statistical analysis, feature transformations etc.

* we skip all the way to model building

* we don't even discuss model selection, feature selection, nor robust hyperparameter tuning

This is the most simple predictive machine learning workflow possible!

* basics for machine learning with scikit-learn in Python. Nothing more.

#### Let's Build a Predictive Machine Learning Model

We use the following steps to build our model

1. **instantiate** the machine learning model with hyperparameters
2. **fit**, train the machine learing model to the training data
3. **predict**, check the trained machine learning models to predict at withheld testing data
4. **select** the optimum hyperparameters, hyperparameter tuning
5. **refit** the model by training with all data given the tuned hyperparmaeters

We will use the train - test approach, an alternative is the train, validate and test approach.

* we will start simple and cover k-fold cross validation and scikit-learn pipeline class later

#### Optional: Add Random Error to the Response Feature, $y$

For more experiential learning consider repeating this workflow with different amounts of additional error

* to observe issues of model overfit consider adding some additional error,

```python
error_stdev = 0.0                                           # standard deviation of additional random error
```

```python
error_stdev = 0.0                                           # standard deviation of additional random error
y_orig = y.copy(deep = True)                                # make a deep copy of original dataset
y['Nporosity'] = y['Nporosity'] + np.random.normal(loc = 0, scale = error_stdev, size = y.shape[0])
plt.scatter(X,y,color='blue',s=30,edgecolor='black',alpha=0.2,label='with added noise'); plt.xlabel('Depth (m)'); plt.ylabel('Standardized Porosity'); plt.title('All Data, Original and with Added Noise'); plt.xlim([0,10.25]); plt.ylim([-2.5,2.5])
plt.scatter(X,y_orig,color='red',s=30,edgecolor='black',alpha=0.2,label='original data'); plt.xlabel('Depth (m)'); plt.ylabel('Standardized Porosity'); plt.legend(loc='upper left')
plt.subplots_adjust(left=0.0, bottom=0.0, right=1.5, top=0.7, wspace=0.2, hspace=0.2);
```

#### Split Train and Test

We will demonstrate the following hyperparameter tuning methods:

* **cross validation** - split 25% of data as withheld testing data, only 25% of data are tested
* **k-fold cross validation** - calculate average error over k folds, all data are tested

Let's start with cross validation and then demonstrate k-fold cross validation.

```python
from sklearn.model_selection import train_test_split        # random train and test data split
X_train, X_test, y_train, y_test = train_test_split(X,y, test_size=0.25, random_state=seed) # train and test split
plt.scatter(X_train,y_train,color='green',s=30,edgecolor='black',alpha=0.4,label='train'); plt.xlabel('Depth (m)'); plt.ylabel('Standardized Porosity'); plt.title('Train and Test Data'); plt.xlim([0,10.25]); plt.ylim([-2.5,2.5]);
plt.scatter(X_test,y_test,color='gray',s=30,edgecolor='black',alpha=0.2,label='test'); plt.xlabel('Depth (m)'); plt.ylabel('Standardized Porosity');plt.legend(loc='upper right')
plt.subplots_adjust(left=0.0, bottom=0.0, right=1.5, top=0.7, wspace=0.2, hspace=0.2);
```

#### Build a Decision Tree Model

Let's instantiate (set the hyperparameter(s)) and fit (train with training data) a decision tree regression model to our data.

```python
from sklearn import tree                                    # tree program from scikit learn (package for machine learning)
max_leaf_nodes = 6                                          # set the hyperparameter
our_tree = tree.DecisionTreeRegressor(max_leaf_nodes = max_leaf_nodes) # instantiate the model
our_tree = our_tree.fit(X_train, y_train)                   # fit the model to the trainin data
```

#### Predict With a Decision Tree Model

Now let's make a prediction with our trained decision tree model.

```python
itest = 2; depth = X_test['Depth'].values[itest]            # set the predictor value for our prediction
spor = our_tree.predict([[depth]])                          # predict with our trained model
plt.scatter(X_train,y_train,color='green',s=30,edgecolor='black',alpha=0.4,label='train'); plt.xlabel('Depth (m)'); plt.ylabel('Standardized Porosity'); plt.title('Train and Test Data and Model Prediction'); plt.xlim([0,10.25]); plt.ylim([-2.5,2.5])
plt.scatter(X_test,y_test,color='gray',s=30,edgecolor='black',alpha=0.2,label='test'); plt.xlabel('Depth (m)'); plt.ylabel('Standardized Porosity');plt.legend(loc='upper right')
plt.scatter(depth,spor,color='black',s=30,marker='s',edgecolor='black',alpha=0.6,label='predict'); plt.xlabel('Depth (m)'); plt.ylabel('Standardized Porosity');plt.legend(loc='upper right')
plt.subplots_adjust(left=0.0, bottom=0.0, right=1.5, top=0.7, wspace=0.2, hspace=0.2);
```

#### Predict Over a Range of Predictor Feature Values with a Decision Tree Model

Now, let's make predictions over a range of predictor feature values so we can visualize the model.

* this is easy to do since we have just 1 predictor feature

```python
depths = np.linspace(0,10.25,1000)                          # set the predictor values for our prediction
spors = our_tree.predict(depths.reshape(-1, 1))             # predict with our trained model
plt.scatter(X_train,y_train,color='green',s=30,edgecolor='black',alpha=0.4,label='train'); plt.xlabel('Depth (m)'); plt.ylabel('Standardized Porosity'); plt.title('Train and Test Data and Model Predictions'); plt.xlim([0,10.25]); plt.ylim([-2.5,2.5]); plt.scatter(X_test,y_test,color='gray',s=30,edgecolor='black',alpha=0.2,label='test'); plt.xlabel('Depth (m)'); plt.ylabel('Standardized Porosity');plt.legend(loc='upper right')
plt.plot(depths,spors,color='black',alpha=0.6,label='predict',linestyle='--'); plt.xlabel('Depth (m)'); plt.ylabel('Standardized Porosity');plt.legend(loc='upper right'); plt.subplots_adjust(left=0.0, bottom=0.0, right=1.5, top=0.7, wspace=0.2, hspace=0.2);
```

#### Changing the Model Hyperparameter(s)

Let's try changing the model complexity and refitting the model.

```python
max_leaf_nodes = 10                                         # set the hyperparameter
our_tree = tree.DecisionTreeRegressor(max_leaf_nodes = max_leaf_nodes) # instantiate the model
our_tree = our_tree.fit(X_train, y_train)                   # fit the model to the trainin data
spors = our_tree.predict(depths.reshape(-1, 1))             # predict with our trained model
plt.scatter(X_train,y_train,color='green',s=30,edgecolor='black',alpha=0.4,label='train'); plt.xlabel('Depth (m)'); plt.ylabel('Standardized Porosity'); plt.xlim([0,10.25]); plt.ylim([-2.5,2.5]); plt.scatter(X_test,y_test,color='gray',s=30,edgecolor='black',alpha=0.2,label='test'); plt.xlabel('Depth (m)'); plt.ylabel('Standardized Porosity');plt.legend(loc='upper right'); plt.plot(depths,spors,color='black',alpha=0.6,label='predict',linestyle='--'); plt.xlabel('Depth (m)'); plt.ylabel('Standardized Porosity');plt.legend(loc='upper right'); plt.subplots_adjust(left=0.0, bottom=0.0, right=1.5, top=0.7, wspace=0.2, hspace=0.2);
```

#### Tuning the Model Hyperparameter(s)

To tune the model hyperparameters we need a metric to summarize model error over the withheld testing data

* let's predict at the testing data and calculate our metric, testing mean square error.

```python
from sklearn import metrics                                 # measures to check our models
leaf_nodes = 2                                              # set the hyperparameter
our_tree = tree.DecisionTreeRegressor(max_leaf_nodes = leaf_nodes) # instantiate the model
our_tree = our_tree.fit(X_train, y_train)                   # fit the model to the trainin data
y_predict = our_tree.predict(X_test)                        # predict at the testing data locations
mse = metrics.mean_squared_error(y_test,y_predict)
print('Testing MSE is ' + str(round(mse,2)) + '.')
```

    Testing MSE is 0.5.

#### Tuning the Model Hyperparameter(s)

To tune the model hyperparameters we need to rebuild the model with a variable level of complexity

* let's loop over the number of leaf nodes and plot the performance over the withheld testing data

```python
nodes = []; scores = []; max_leaf_nodes = 30
for i, nnode in enumerate(np.arange(2,max_leaf_nodes,1)):
    our_tree = tree.DecisionTreeRegressor(max_leaf_nodes = nnode).fit(X_train, y_train) # instantiate / fit
    y_predict = our_tree.predict(X_test)                    # predict at the testing data locations
    nodes.append(nnode); scores.append(metrics.mean_squared_error(y_test,y_predict))
plt.scatter(nodes,scores,color='red',s=30,edgecolor='black',alpha=0.4,label='train'); plt.plot(nodes,scores,color='red',alpha=0.2,linestyle='--'); plt.xlabel('Depth (m)'); plt.ylabel('Standardized Porosity'); plt.xlim([1,max_leaf_nodes]); plt.xlabel('Number of Leaf Nodes'); plt.ylabel('Testing Mean Square Error'); plt.subplots_adjust(left=0.0, bottom=0.0, right=1.5, top=0.7, wspace=0.2, hspace=0.2);
```

#### Tuning the Model Hyperparameter(s)

To tune the model hyperparameters we need to rebuild the model with variable level of complexity

* for a more robust assessment of performance let's use scikit-learns k-fold cross validation

```python
from sklearn.model_selection import cross_val_score as kfold       # cross validation methods
nodes = []; scores = []; max_leaf_nodes = 30
for i, nnode in enumerate(np.arange(2,max_leaf_nodes,1,dtype = int)):
    our_tree = tree.DecisionTreeRegressor(max_leaf_nodes = nnode) # instantiate / fit
    nodes.append(nnode);
    scores.append(abs(kfold(estimator=our_tree,X=X,y=y,cv=4,n_jobs=4,scoring = "neg_mean_squared_error").mean()))
plt.scatter(nodes,scores,color='red',s=30,edgecolor='black',alpha=0.4,label='train'); plt.plot(nodes,scores,color='red',alpha=0.2,linestyle='--'); plt.xlabel('Depth (m)'); plt.ylabel('Standardized Porosity'); plt.xlim([1,max_leaf_nodes]); plt.xlabel('Number of Leaf Nodes'); plt.ylabel('Testing Mean Square Error'); plt.subplots_adjust(left=0.0, bottom=0.0, right=1.5, top=0.7, wspace=0.2, hspace=0.2);
```

#### Select the Best Hyperparameter(s) and Re-train on All Data

The main product from all of the previous work is the tuned hyperparameter, the optimum model complexity.

* now we refit the tuned model, train the model on all the data and our tuned model is ready for real-world use

* once again we are assuming a train and test workflow, not a train, validate and test workflow

```python
tuned_nodes = nodes[np.argmin(scores)]
our_tuned_tree = tree.DecisionTreeRegressor(max_leaf_nodes = tuned_nodes).fit(X, y) # instantiate / fit
tuned_spors = our_tuned_tree.predict(depths.reshape(-1, 1))             # predict with our trained model
plt.scatter(X_train,y_train,color='green',s=30,edgecolor='black',alpha=0.4,label='train'); plt.xlabel('Depth (m)'); plt.ylabel('Standardized Porosity'); plt.title('Train and Test Data and Tuned Model with ' + str(tuned_nodes) + ' Leaf Nodes'); plt.xlim([0,10.25]); plt.ylim([-2.5,2.5]); plt.scatter(X_test,y_test,color='gray',s=30,edgecolor='black',alpha=0.2,label='test'); plt.xlabel('Depth (m)'); plt.ylabel('Standardized Porosity');plt.legend(loc='upper right')
plt.plot(depths,tuned_spors,color='black',alpha=0.6,label='tuned model',linestyle='--'); plt.xlabel('Depth (m)'); plt.ylabel('Standardized Porosity');plt.legend(loc='upper right'); plt.subplots_adjust(left=0.0, bottom=0.0, right=1.5, top=0.7, wspace=0.2, hspace=0.2);
```

#### Same Complete Workflow with Pipelines

Let's repeat the above workflow with a pipline from scikit-learn. Here some general comments:

* **pipeline workflow steps** - list of steps, custom labels and the associated scikit-learn classes
* **choices and hyperparameters** - as a dictionary with combined step custom labels and hyperparameter names
* **workflow scenarios** - we can specify lists of choices and hyperparameters
* **consistent iteration** - the entire workflow is repeated for the full combinatorial
* **best model selected** - best combination of model hyperparameters and choices are refit with all data

```python
from sklearn.pipeline import Pipeline                       # machine learning modeling pipeline
pipe = Pipeline([                                           # the machine learning workflow as a pipeline object
    ('tree', tree.DecisionTreeRegressor())
])

params = {                                                  # the machine learning workflow method's parameters
    'tree__max_leaf_nodes': np.arange(2,max_leaf_nodes,1,dtype = int)
}

from sklearn.model_selection import GridSearchCV            # model hyperparameter grid search
grid_cv_tuned = GridSearchCV(pipe, params, scoring = 'neg_mean_squared_error', # grid search cross validation
                             cv=4,refit = True);

grid_cv_tuned.fit(X,y);                                      # fit model with tuned hyperparameters
print('Tuned maximum number of leaf nodes = ' + str(grid_cv_tuned.best_params_['tree__max_leaf_nodes']) + '.')
depth = 2.5
print('Prediction at depth = ' + str(depth) + ' m is ' + str(np.round(grid_cv_tuned.predict([[depth]]),2))) # predict with our trained model) + '.')
```

    Tuned maximum number of leaf nodes = 14.
    Prediction at depth = 2.5 m is [-0.97]

#### Comments

This was a very simple demonstration of predictive machine learning model building with Python's scikit-learn.

* this is a basic introduction, much more could be done

I hope this was helpful,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin
On twitter I'm the @GeostatsGuy.

***

#### More on Michael Pyrcz and the Texas Center for Geostatistics:

### Michael Pyrcz, Associate Professor, University of Texas at Austin
*Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions*

With over 17 years of experience in subsurface consulting, research and development, Michael has returned to academia driven by his passion for teaching and enthusiasm for enhancing engineers' and geoscientists' impact in subsurface resource development.

For more about Michael check out these links:

#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

#### Want to Work Together?

I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.

* Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I'd be happy to drop by and work with you!

* Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!

* I can be reached at mpyrcz@austin.utexas.edu.

I'm always happy to discuss,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin

#### More Resources Available at: [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)


# Grafičke metode analize stabilnosti sustava

U grafičke metode analize stabilnosti sustava spadaju Nyquistov i Bodeov dijagram te očitavanje faznog i amplitudnog osiguranja putem dobivenih dijagrama. Također pole-zero plot koji nam prikazuje položaj polova i nula sustava.
Radi problema u iscrtavanju Nyquistovog dijagrama putem funkcije iz ```Python Control Systems Library``` knjižnice možemo uzeti samo Bodeov dijagram kao pouzanu metodu za grafičku analizu stabilnosti.

Python funkcije koje se koriste za ove operacije dio su ```Python Control Systems Library``` knjižnice.
```python
* tf
* bode_plot
* impulse_response
* step_response
* margin
* pole
* zero
* pzmap
```
Popis svih funkcija dostupan je na https://python-control.readthedocs.io/en/0.8.2/control.html#function-ref

te dio ```Sympy``` knjižnice.
```python
* symbols
* Poly
* simplify
* expand
* fraction
```

i ```Matplotlib``` knjižnice.
```python
* pyplot
```

Prije početka potrebno je instalirati ```Python Control Systems Library``` tako da otvorite ```Jupyter QtConsole``` iz početnog izbornika ```Anaconde``` te upišete
```python
conda install -c conda-forge control
```

Za početak potrebno je obrisati sve varijable iz memorije.

```python
import os
clear = lambda: os.system('cls')
clear()
```

    0

Potom importati sve funkcije i knjižnice koje koristimo.

```python
from sympy.interactive import printing
printing.init_printing(use_latex=True)

import scipy
from scipy import signal
import matplotlib
import matplotlib.pyplot as plt
import control
from control import *
import sympy
from sympy import symbols, Poly, simplify, expand, fraction
```

Potom definirati sve varijable koje koristimo kao Sympy simbole.

```python
s = symbols('s')
```

### Primjer: Sustav drugog reda s nulom u ishodistu

```python
tf = (s / ((s+6)*(s+2)))
display(tf)
```

Prijenosnu funkciju je kao u vježbi "Koeficijenti prijenosne funkcije" potrebno razbiti na brojnik i nazivnik te dobiti zapis u obliku matrice koeficijenata sustava.

```python
tf = expand(tf)
display(tf)
```

```python
brojnik, nazivnik = fraction(tf)
display(brojnik)
display(nazivnik)
```

```python
brojnik = Poly(brojnik, s)
nazivnik = Poly(nazivnik, s)
```

```python
brojnik = brojnik.coeffs()
nazivnik = nazivnik.coeffs()
display(brojnik)
display(nazivnik)
```

Sustav je sada potrebno definirati kao prijenosnu funkciju koristeći funkciju ```control.tf(SUSTAV)``` no to ne možemo izravno izvršiti već moramo pretvoriti ```Sympy``` matricu u ```Numpy``` array putem ```np.array()``` funkcije.
Ukoliko se radi o prijenosnoj funkciji s nulom ili polom u iskodistu (s^1); radi kompatibilnosti potrebno je dodati nulu na kraj arraya putem ```np.append(ARRAY, 0)``` funkcije. To je potrebno iz razloga što ```poly()``` i ```.coefs()``` funkcija iz gornjeg postupka za stvaranje matrica ```brojnik``` i ```nazivnik``` ignorira koeficijent 0 uz s^0 ukoliko postoji koeficijent višeg eksponenta s, to predstavlja problem za sustave gdje je u brojniku samo s^1 ili sustave gdje imamo astatizam prvog reda (s^1 u nazivniku)

I dalje je naj pametnije ponovno kreirati ```Control Systems Library``` prijenosnu funkciju ```tf([brojnik],[nazivnik])``` kako bi se eliminirala moguća pogreška u konverziji.
Također je potrebno uz pomoć ```display()``` funkcije provjeriti je li izraz prijenosne funkcije točan.

```python
brojnik = np.array(brojnik).astype(np.float64)
brojnik = np.append(brojnik, 0)
nazivnik = np.array(nazivnik).astype(np.float64)
display(brojnik)
display(nazivnik)
```

    array([1., 0.])

    array([ 1.,  8., 12.])

Sada smo u mogućnosti kreirati ```Control Systems Library``` prijenosnu funkciju.

```python
sys = control.tf(brojnik, nazivnik)
display(sys)
```

$$\frac{s}{s^2 + 8 s + 12}$$

Bodeov dijagram iscrtavamo putem funkcije
```python
control.bode_plot(sys)
```

No potrebno je definirati što ce se iscrtati, u ovom slucaju magnitudna i fazna karakteristika u frekvencijskoj domeni, sto definiramo kao ```mag, phase, omega```

```python
mag, phase, omega = control.bode_plot(sys)
```

U gornjem prikazu su frekvencije prikazane u ```rad/sec```, ukoliko želimo prikazati frekvenciju u Hz potrebno je u funkciju dodati ```, Hz=True```
```python
control.bode_plot(sys, Hz=True)
```

```python
mag, phase, Hz = control.bode_plot(sys, Hz=True)
```

Također u standardnim prikazima magnituda je prikazana u apsolutnoj vrijednosti, ukoliko želimo magnitudu prikazati u dB potrebno je u funkciju dodati ```, dB=True```
```python
control.bode_plot(sys, Hz=True, dB=True)
```

```python
mag, phase, Hz = control.bode_plot(sys, Hz=True, dB=True)
```

Kao što vidimo gornji Bode dijagrami su maleni, da bi smo povećali prikaz potrebno je pozvati sljedeće funkcije:
```python
fig = matplotlib.pyplot.gcf()
fig.set_size_inches(SIRINA, VISINA)
fig.savefig('IME_SLIKE.png', dpi=DPI)
```

```fig=gcf()``` sprema zadnji, tj. trenutni prikaz kao varijablu fig (get current figure)
```set_size_inches(SIRINA,VISINA)``` postavlja širinu i visinu prikaza na ekranu.
```savefig('IME_SLIKE.png, dpi=DPI)``` sprema prikazanu sliku u direktorij gdje se nalazi python skripta u rezoluciji prema zadanom DPI, preporucljivo 300-600 DPI.

```python
mag, phase, Hz = control.bode_plot(sys, Hz=True, dB=True)
fig = matplotlib.pyplot.gcf()
fig.set_size_inches(15, 10)
fig.savefig('BODE.png', dpi=300)
```

Za prikaz polova koristi se funkcija ```control.pole(SUSTAV)```

```python
control.pole(sys)
```

    array([-6., -2.])

Za prikaz nula koristi se funkcija ```control.zero(SUSTAV)```

```python
control.zero(sys)
```

    array([0.])

Za grafički prikaz polova i nula u Re-Im koordinatnom sustavu koristi se funkcija ```control.pzmap(SUSTAV)```

```python
control.pzmap(sys)
```

Odziv na ```delta``` funkciju dobivamo putem ```control.impulse_response(SUSTAV)``` funkcije.
No pozivanje ove funkcije samo izračuna vrijednosti odziva u vremenskoj domeni, za grafički prikaz potrebno je putem ```matplotlib``` ```pyplot``` funkcije ```plot(X,Y)```, u ovom slucaju definirane kao ```plt.plot(X,Y)``` iscrtati graf funkcije.
Putem funkcije ```plt.ylabel('TEKST')``` te ```plt.xlabel('TEKST')``` možemo imenovati X i Y osi.

```python
T, yout = control.step_response(sys)

plt.plot(T, yout)
plt.xlabel('t')
plt.ylabel('Y(t)')
```

```python
T, yout = control.impulse_response(sys)

plt.plot(T, yout)
plt.xlabel('t')
plt.ylabel('Y(t)')
```

```Control Systems Library``` kao impulsni odziv i odziv na step daje samo grafički prikaz, tj. numeričke vrijednosti, za analitički prikaz potrebno je početnu prijenosnu funkciju u ```Sympy``` pomnožiti s laplaceovom transformacijom željene funkcije, time dobivamo odziv u frekvencijskoj domeni, za odziv u vremenskoj domeni potrebno je izvršiti inverznu Laplaceovu transformaciju dobivenog odziva putem funkcije ```inverse_laplace_transform(ODZIV)``` koja je dio ```Sympy``` knjižnice.

Laplaceova transformacija delta funkcije je = 1
Laplaceova transformacija step funkcije je = 1/s

Simbol ```t``` je potrebno definirati kao pozitivan, iz razloga što radimo s LTI sustavima koji su po definiciji kauzalni. Ukoliko nije definiran kao pozitivna vrijednost tada nećemo dobiti dobar oblik inverzne Laplaceove transformacije s kakvim inače radimo.

```python
from sympy.integrals.transforms import inverse_laplace_transform
t = symbols('t', positive=True)
```

```python
impulse = 1 * tf
display(impulse)
```

```python
impulse_time = inverse_laplace_transform(impulse, s, t)
display(impulse_time)
```

```python
step = (1/s) * tf
display(step)
```

Za rastav funkcije na parcijalne razlomke koristimo ```apart(FUNKCIJA)``` funkciju iz ```Sympy``` knjiznice.

```python
from sympy import apart

step = apart(step)
display(step)
```

```python
step_time = inverse_laplace_transform(step, s, t)
display(step_time)

step_time = apart(step_time)
display(step_time)
```

Za graficki prikaz odziva u vremenskoj domeni mozemo koristiti ```plot(FUNKCIJA)``` funkciju iz ```sympy.plotting```

```python
from sympy.plotting import plot
```

```python
plot(step_time)
```

Kao sto je vidljivo prikaz ne lici na gore dobiveni prikaz iz ```Control Systems Library``` funkcije, no to je iz razloga sto granice prikaza nisu postavljene, ukoliko granice postavimo kao sto su gore automatski postavljene dobiti cemo ispravan prikaz.
```python
plot(FUNKCIJA, (t, POCETAK, KRAJ))
```

```python
plot(step_time, (t, 0, 3.5))
```

Kako je gornji sustav strukturno stabilan, te na nijednoj frekvenciji karakteristika nema gain=0 niti fazno kašnjenje od 180 stupnjeva ```margin(SUSTAV)``` funkcija nema rezultat.

```python
margin(sys)
```

### Primjer: Sustav 4og reda

```python
tf2 = ((10*(s+3)) / (s * (s+2) * ((s**2) + s + 2)))
tf2 = expand(tf2)
display(tf2)

sys2 = control.tf([10,30],[1,3,4,4,0])

print("=")
display(sys2)
```

```python
mag2, phase2, Hz2 = control.bode_plot(sys2, Hz=True, dB=True)
fig = matplotlib.pyplot.gcf()
fig.set_size_inches(15, 10)
```

Putem ```margin(SUSTAV)``` funkcije mozemo odrediti amplitudno i fazno osiguranje, rezultat su:
* gm (Gain Margin) - Amplitudno osiguranje
* wg (omega-gain) - Frekvencija na kojoj ocitavamo amplitudno osiguranje
* pm (Phase Margin) - Fazno osiguranje
* wp (omega-phase) - Frekvencija na kojoj ocitavamo fazno osiguranje

```python
margin(sys2)
```

```python
gm, pm, wg, wp = margin(sys2)
print("Amplitudno osiguranje = ")
display(gm)
print("Na frekvenciji w =")
display(wg)
print("")
print("Fazno osiguranje = ")
display(pm)
print("Na frekvenciji w =")
display(wp)

```

```python
pzmap(sys2)
```


# Neuromatch Academy: Week 1, Day 3, Tutorial 4
# Model Fitting: Multiple linear regression and polynomial regression

**Content creators**: Pierre-Étienne Fiquet, Anqi Wu, Alex Hyafil with help from Byron Galbraith, Ella Batty

**Content reviewers**: Lina Teichmann, Saeed Salehi, Patrick Mineault, Michael Waskom

---
#Tutorial Objectives

This is Tutorial 4 of a series on fitting models to data. We start with simple linear regression, using least squares optimization (Tutorial 1) and Maximum Likelihood Estimation (Tutorial 2). We will use bootstrapping to build confidence intervals around the inferred linear model parameters (Tutorial 3). We'll finish our exploration of regression models by generalizing to multiple linear regression and polynomial regression (Tutorial 4). We end by learning how to choose between these various models. We discuss the bias-variance trade-off (Tutorial 5) and Cross Validation for model selection (Tutorial 6).

In this tutorial, we will generalize the regression model to incorporate multiple features.
- Learn how to structure inputs for regression using the 'Design Matrix'
- Generalize the MSE for multiple features using the ordinary least squares estimator
- Visualize data and model fit in multiple dimensions
- Fit polynomial regression models of different complexity
- Plot and evaluate the polynomial regression fits

```python
#@title Video 1: Multiple Linear Regression and Polynomial Regression
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="d4nfTki6Ejc", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
```

    Video available at https://youtube.com/watch?v=d4nfTki6Ejc

---
# Setup

```python
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import axes3d
```

```python
#@title Figure Settings
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")
```

```python
#@title Helper Functions

def plot_fitted_polynomials(x, y, theta_hat):
  """ Plot polynomials of different orders

  Args:
    x (ndarray): input vector of shape (n_samples)
    y (ndarray): vector of measurements of shape (n_samples)
    theta_hat (dict): polynomial regression weights for different orders
  """

  x_grid = np.linspace(x.min() - .5, x.max() + .5)

  plt.figure()

  for order in range(0, max_order + 1):
    X_design = make_design_matrix(x_grid, order)
    plt.plot(x_grid, X_design @ theta_hat[order]);

  plt.ylabel('y')
  plt.xlabel('x')
  plt.plot(x, y, 'C0.');
  plt.legend([f'order {o}' for o in range(max_order + 1)], loc=1)
  plt.title('polynomial fits')
  plt.show()

```

---
# Section 1: Multiple Linear Regression

Now that we have considered the univariate case and how to produce confidence intervals for our estimator, we turn to the general linear regression case, where we can have more than one regressor, or feature, in our input.

Recall that our original univariate linear model was given as

\begin{align}
y = \theta x + \epsilon
\end{align}

where $\theta$ is the slope and $\epsilon$ some noise. We can easily extend this to the multivariate scenario by adding another parameter for each additional feature

\begin{align}
y = \theta_0 + \theta_1 x_1 + \theta_1 x_2 + ... +\theta_d x_d + \epsilon
\end{align}

where $\theta_0$ is the intercept and $d$ is the number of features (it is also the dimensionality of our input).

We can condense this succinctly using vector notation for a single data point

\begin{align}
y_i = \boldsymbol{\theta}^{\top}\mathbf{x}_i + \epsilon
\end{align}

and fully in matrix form

\begin{align}
\mathbf{y} = \mathbf{X}\boldsymbol{\theta} + \mathbf{\epsilon}
\end{align}

where $\mathbf{y}$ is a vector of measurements, $\mathbf{X}$ is a matrix containing the feature values (columns) for each input sample (rows), and $\boldsymbol{\theta}$ is our parameter vector.

This matrix $\mathbf{X}$ is often referred to as the "[design matrix](https://en.wikipedia.org/wiki/Design_matrix)".

For this tutorial we will focus on the two-dimensional case ($d=2$), which allows us to easily visualize our results. As an example, think of a situation where a scientist records the spiking response of a retinal ganglion cell to patterns of light signals that vary in contrast and in orientation. Then contrast and orientation values can be used as features / regressors to predict the cells response.

In this case our model can be writen as:

\begin{align}
y = \theta_0 + \theta_1 x_1 + \theta_2 x_2 + \epsilon
\end{align}

or in matrix form where

\begin{align}
\mathbf{X} =
\begin{bmatrix}
1 & x_{1,1} & x_{1,2} \\
1 & x_{2,1} & x_{2,2} \\
\vdots & \vdots & \vdots \\
1 & x_{n,1} & x_{n,2}
\end{bmatrix},
\boldsymbol{\theta} =
\begin{bmatrix}
\theta_0 \\
\theta_1 \\
\theta_2 \\
\end{bmatrix}
\end{align}

For our actual exploration dataset we shall set $\boldsymbol{\theta}=[0, -2, -3]$ and draw $N=40$ noisy samples from $x \in [-2,2)$. Note that setting the value of $\theta_0 = 0$ effectively ignores the offset term.

```python
#@title
#@markdown Execute this cell to simulate some data

# Set random seed for reproducibility
np.random.seed(1234)

# Set parameters
theta = [0, -2, -3]
n_samples = 40

# Draw x and calculate y
n_regressors = len(theta)
x0 = np.ones((n_samples, 1))
x1 = np.random.uniform(-2, 2, (n_samples, 1))
x2 = np.random.uniform(-2, 2, (n_samples, 1))
X = np.hstack((x0, x1, x2))
noise = np.random.randn(n_samples)
y = X @ theta + noise

ax = plt.subplot(projection='3d')
ax.plot(X[:,1], X[:,2], y, '.')

ax.set(
    xlabel='$x_1$',
    ylabel='$x_2$',
    zlabel='y'
)
plt.tight_layout()
```

Now that we have our dataset, we want to find an optimal vector of paramters $\boldsymbol{\hat\theta}$. Recall our analytic solution to minimizing MSE for a single regressor:

\begin{align}
\hat\theta = \frac{\sum_{i=1}^N x_i y_i}{\sum_{i=1}^N x_i^2}.
\end{align}

The same holds true for the multiple regressor case, only now expressed in matrix form

\begin{align}
\boldsymbol{\hat\theta} = (\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{X}^\top\mathbf{y}.
\end{align}

This is called the [ordinary least squares](https://en.wikipedia.org/wiki/Ordinary_least_squares) (OLS) estimator.

### Exercise 1: Ordinary Least Squares Estimator

In this exercise you will implement the OLS approach to estimating $\boldsymbol{\hat\theta}$ from the design matrix $\mathbf{X}$ and measurement vector $\mathbf{y}$. You can use the `@` symbol for matrix multiplication, `.T` for transpose, and `np.linalg.inv` for matrix inversion.

```python
def ordinary_least_squares(X, y):
  """Ordinary least squares estimator for linear regression.

  Args:
    x (ndarray): design matrix of shape (n_samples, n_regressors)
    y (ndarray): vector of measurements of shape (n_samples)

  Returns:
    ndarray: estimated parameter values of shape (n_regressors)
  """
  ######################################################################
  ## TODO for students: solve for the optimal parameter vector using OLS
  # Fill out function and remove
#   raise NotImplementedError("Student exercise: solve for theta_hat vector using OLS")
  ######################################################################

  # Compute theta_hat using OLS
  theta_hat = np.linalg.inv(X.T@X)@X.T@y

  return theta_hat

# Uncomment below to test your function
theta_hat = ordinary_least_squares(X, y)
print(theta_hat)
```

    [ 0.13861386 -2.09395731 -3.16370742]

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D3_ModelFitting/solutions/W1D3_Tutorial4_Solution_3dc448db.py)

After filling in this function, you should see that $\hat{\theta}$ = [ 0.13861386, -2.09395731, -3.16370742]

Now that we have our $\mathbf{\hat\theta}$, we can obtain $\mathbf{\hat y}$ and thus our mean squared error.

```python
# Compute predicted data
theta_hat = ordinary_least_squares(X, y)
y_hat = X @ theta_hat

# Compute MSE
print(f"MSE = {np.mean((y - y_hat)**2):.2f}")
```

    MSE = 0.91

Finally, the following code will plot a geometric visualization of the data points (blue) and fitted plane. You can see that the residuals (green bars) are orthogonal to the plane.

Extra: how would you check that the residuals are orthogonal to the fitted plane? (this is sometimes called the orthogonality principle, see [least squares notes](https://www.cns.nyu.edu/~eero/NOTES/leastSquares.pdf))

```python
#@title
#@markdown Execute this cell to visualize data and predicted plane

theta_hat = ordinary_least_squares(X, y)
xx, yy = np.mgrid[-2:2:50j, -2:2:50j]
y_hat_grid = np.array([xx.flatten(), yy.flatten()]).T @ theta_hat[1:]
y_hat_grid = y_hat_grid.reshape((50, 50))

ax = plt.subplot(projection='3d')
ax.plot(X[:, 1], X[:, 2], y, '.')
ax.plot_surface(xx, yy, y_hat_grid, linewidth=0, alpha=0.5, color='C1',
                cmap=plt.get_cmap('coolwarm'))

for i in range(len(X)):
  ax.plot((X[i, 1], X[i, 1]),
          (X[i, 2], X[i, 2]),
          (y[i], y_hat[i]),
          'g-', alpha=.5)

ax.set(
    xlabel='$x_1$',
    ylabel='$x_2$',
    zlabel='y'
)
plt.tight_layout()

```

---
# Section 2:  Polynomial Regression

So far today, you learned how to predict outputs from inputs by fitting a linear regression model. We can now model all sort of relationships, including in neuroscience!

One potential problem with this approach is the simplicity of the model. Linear regression, as the name implies, can only capture a linear relationship between the inputs and outputs. Put another way, the predicted outputs are only a weighted sum of the inputs. What if there are more complicated computations happening? Luckily, many more complex models exist (and you will encounter many more over the next 3 weeks). One model that is still very simple to fit and understand, but captures more complex relationships, is **polynomial regression**, an extension of linear regression.

Since polynomial regression is an extension of linear regression, everything you learned so far will come in handy now! The goal is the same: we want to predict the dependent variable $y_{n}$ given the input values $x_{n}$. The key change is the type of relationship between inputs and outputs that the model can capture.

Linear regression models predict the outputs as a weighted sum of the inputs:

$$y_{n}= \theta_0 + \theta x_{n} + \epsilon_{n}$$

With polynomial regression, we model the outputs as a polynomial equation based on the inputs. For example, we can model the outputs as:

$$y_{n}= \theta_0 + \theta_1 x_{n} + \theta_2 x_{n}^2 + \theta_3 x_{n}^3 + \epsilon_{n}$$

We can change how complex a polynomial is fit by changing the order of the polynomial. The order of a polynomial refers to the highest power in the polynomial. The equation above is a third order polynomial because the highest value x is raised to is 3. We could add another term ($+ \theta_4 x_{n}^4$) to model an order 4 polynomial and so on.

First, we will simulate some data to practice fitting polynomial regression models. We will generate random inputs $x$ and then compute y according to $y = x^2 - x - 2 $, with some extra noise both in the input and the output to make the model fitting exercise closer to a real life situation.

```python
#@title
#@markdown Execute this cell to simulate some data

# setting a fixed seed to our random number generator ensures we will always
# get the same psuedorandom number sequence
np.random.seed(121)
n_samples = 30
x = np.random.uniform(-2, 2.5, n_samples)  # inputs uniformly sampled from [-2, 2.5)
y =  x**2 - x - 2   # computing the outputs

output_noise = 1/8 * np.random.randn(n_samples)
y += output_noise  # adding some output noise

input_noise = 1/2 * np.random.randn(n_samples)
x += input_noise  # adding some input noise

fig, ax = plt.subplots()
ax.scatter(x, y)  # produces a scatter plot
ax.set(xlabel='x', ylabel='y');
```

## Section 2.1: Design matrix for polynomial regression

Now we have the basic idea of polynomial regression and some noisy data, let's begin! The key difference between fitting a linear regression model and a polynomial regression model lies in how we structure the input variables.

For linear regression, we used $X = x$ as the input data. To add a constant bias (a y-intercept in a 2-D plot), we use $X = \big[ \boldsymbol 1, x \big]$, where $\boldsymbol 1$ is a column of ones.  When fitting, we learn a weight for each column of this matrix. So we learn a weight that multiples with column 1 - in this case that column is all ones so we gain the bias parameter ($+ \theta_0$). We also learn a weight for every column, or every feature of x, as we learned in Section 1.

This matrix $X$ that we use for our inputs is known as a **design matrix**. We want to create our design matrix so we learn weights for $x^2, x^3,$ etc. Thus, we want to build our design matrix $X$ for polynomial regression of order $k$ as:

$$X = \big[ \boldsymbol 1 , x^1, x^2 , \ldots , x^k \big],$$

where $\boldsymbol{1}$ is the vector the same length as $x$ consisting of of all ones, and $x^p$ is the vector or matrix $x$ with all elements raised to the power $p$. Note that $\boldsymbol{1} = x^0$ and $x^1 = x$

### Exercise 2: Structure design matrix

Create a function (`make_design_matrix`) that structures the design matrix given the input data and the order of the polynomial you wish to fit. We will print part of this design matrix for our data and order 5.

```python
def make_design_matrix(x, order):
  """Create the design matrix of inputs for use in polynomial regression

  Args:
    x (ndarray): input vector of shape (n_samples)
    order (scalar): polynomial regression order

  Returns:
    ndarray: design matrix for polynomial regression of shape (samples, order+1)
  """
  ########################################################################
  ## TODO for students: create the design matrix ##
  # Fill out function and remove
#   raise NotImplementedError("Student exercise: create the design matrix")
  ########################################################################

  # Broadcast to shape (n x 1) so dimensions work
  if x.ndim == 1:
    x = x[:, None]

  #if x has more than one feature, we don't want multiple columns of ones so we assign
  # x^0 here
  design_matrix = np.ones((x.shape[0], order+1))

  # Loop through rest of degrees and stack columns (hint: np.hstack)
  for degree in range(1, order + 1):
      design_matrix[:,degree] = x.T**degree

  return design_matrix

# Uncomment to test your function
order = 5
X_design = make_design_matrix(x, order)
print(X_design[0:2, 0:2])
```

    [[ 1.         -1.51194917]
     [ 1.         -0.35259945]]

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D3_ModelFitting/solutions/W1D3_Tutorial4_Solution_35b1e60c.py)

You should see that the printed section of this design matrix is `[[ 1.         -1.51194917]
 [ 1.         -0.35259945]]`

## Section 2.2: Fitting polynomial regression models

Now that we have the inputs structured correctly in our design matrix, fitting a polynomial regression is the same as fitting a linear regression model! All of the polynomial structure we need to learn is contained in how the inputs are structured in the design matrix. We can use the same least squares solution we computed in previous exercises.

### Exercise 3: Fitting polynomial regression models with different orders

Here, we will fit polynomial regression models to find the regression coefficients ($\theta_0, \theta_1, \theta_2,$ ...) by solving the least squares problem. Create a function `solve_poly_reg` that loops over different order polynomials (up to `max_order`), fits that model, and saves out the weights for each. You may invoke the `ordinary_least_squares` function.

We will then qualitatively inspect the quality of our fits for each order by plotting the fitted polynomials on top of the data. In order to see smooth curves, we evaluate the fitted polynomials on a grid of $x$ values (ranging between the largest and smallest of the inputs present in the dataset).

```python
def solve_poly_reg(x, y, max_order):
  """Fit a polynomial regression model for each order 0 through max_order.

  Args:
    x (ndarray): input vector of shape (n_samples)
    y (ndarray): vector of measurements of shape (n_samples)
    max_order (scalar): max order for polynomial fits

  Returns:
    dict: fitted weights for each polynomial model (dict key is order)
  """

  # Create a dictionary with polynomial order as keys, and np array of theta_hat
  # (weights) as the values
  theta_hats = {}

  # Loop over polynomial orders from 0 through max_order
  for order in range(max_order + 1):

    ##################################################################################
    ## TODO for students: Create design matrix and fit polynomial model for this order
    # Fill out function and remove
#     raise NotImplementedError("Student exercise: fit a polynomial model")
    ##################################################################################

    # Create design matrix
    X_design = make_design_matrix(x,order)

    # Fit polynomial model
    this_theta = ordinary_least_squares(X_design,y)

    theta_hats[order] = this_theta

  return theta_hats

# Uncomment to test your function
max_order = 5
theta_hats = solve_poly_reg(x, y, max_order)
plot_fitted_polynomials(x, y, theta_hats)

```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D3_ModelFitting/solutions/W1D3_Tutorial4_Solution_54b1e4e3.py)

*Example output:*

## Section 2.4: Evaluating fit quality

As with linear regression, we can compute mean squared error (MSE) to get a sense of how well the model fits the data.

We compute MSE as:

$$ MSE = \frac 1 N ||y - \hat y||^2 = \sum_{i=1}^N (y_i - \hat y_i)^2 $$

where the predicted values for each model are given by $ \hat y = X \hat \theta$.

*Which model (i.e. which polynomial order) do you think will have the best MSE?*

### Exercise 4: Compute MSE and compare models

We will compare the MSE for different polynomial orders with a bar plot.

```python
mse = np.zeros((max_order + 1))
for order in range(0, max_order + 1):

  X_design = make_design_matrix(x, order)

  ############################################################################
  ## TODO for students: Compute MSE
  #############################################################################

  ## Uncomment below and fill in with your code
  theta_hat = ordinary_least_squares(X_design,y)
  # Get prediction for the polynomial regression model of this order
  y_hat = X_design@theta_hat

  # Compute the residuals
  residuals = y-y_hat

  # Compute the MSE
  mse[order] = np.mean(residuals**2)

  pass

fig, ax = plt.subplots()

# Uncomment once above exercise is complete
ax.bar(range(max_order + 1), mse)

ax.set(title='Comparing Polynomial Fits', xlabel='Polynomial order', ylabel='MSE')
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D3_ModelFitting/solutions/W1D3_Tutorial4_Solution_d8d00e00.py)

*Example output:*

---
# Summary

* Linear regression generalizes naturally to multiple dimensions
* Linear algebra affords us the mathematical tools to reason and solve such problems beyond the two dimensional case

* To change from a linear regression model to a polynomial regression model, we only have to change how the input data is structured

* We can choose the complexity of the model by changing the order of the polynomial model fit

* Higher order polynomial models tend to have lower MSE on the data they're fit with

**Note**: In practice, multidimensional least squares problems can be solved very efficiently (thanks to numerical routines such as LAPACK).

**Suggested readings**

[Introduction to Applied Linear Algebra – Vectors, Matrices, and Least Squares](http://vmls-book.stanford.edu/)
Stephen Boyd and Lieven Vandenberghe


```python
import open3d as o3d
import open3d.core as o3c

if o3d.__DEVICE_API__ == 'cuda':
    import open3d.cuda.pybind.t.pipelines.registration as treg
else:
    import open3d.cpu.pybind.t.pipelines.registration as treg

import numpy as np
import sys
import os
import time

# monkey patches visualization and provides helpers to load geometries
sys.path.append('..')
import open3d_tutorial as o3dtut
# change to True if you want to interact with the visualization windows
o3dtut.interactive = not "CI" in os.environ
```

# ICP registration
This tutorial demonstrates the ICP (Iterative Closest Point) registration algorithm. It has been a mainstay of geometric registration in both research and industry for many years. The inputs are two point clouds and an initial transformation that roughly aligns the source point cloud to the target point cloud. The output is a refined transformation that tightly aligns the two point clouds. A helper function `draw_registration_result` visualizes the alignment during the registration process. In this tutorial, we show differnt ICP variants, and the API for using them.

## Helper visualization function

The function below visualizes a target point cloud and a source point cloud transformed with an alignment transformation. The target point cloud and the source point cloud are painted with cyan and yellow colors respectively. The more and tighter the two point-clouds overlap with each other, the better the alignment result.

```python
def draw_registration_result(source, target, transformation):
    source_temp = source.clone()
    target_temp = target.clone()

    source_temp.transform(transformation)

    # This is patched version for tutorial rendering.
    # Use `draw` function for you application.
    o3d.visualization.draw_geometries(
        [source_temp.to_legacy(),
         target_temp.to_legacy()],
        zoom=0.4459,
        front=[0.9288, -0.2951, -0.2242],
        lookat=[1.6784, 2.0612, 1.4451],
        up=[-0.3402, -0.9189, -0.1996])
```

## Understanding ICP Algorithm

In general, the ICP algorithm iterates over two steps:

1. Find correspondence set $\mathcal{K}=\{(\mathbf{p}, \mathbf{q})\}$ from target point cloud $\mathbf{P}$, and source point cloud $\mathbf{Q}$ transformed with current transformation matrix $\mathbf{T}$.
2. Update the transformation $\mathbf{T}$ by minimizing an objective function $E(\mathbf{T})$ defined over the correspondence set $\mathcal{K}$.

Different variants of ICP use different objective functions $E(\mathbf{T})$ [\[BeslAndMcKay1992\]](../reference.html#beslandmckay1992) [\[ChenAndMedioni1992\]](../reference.html#chenandmedioni1992) [\[Park2017\]](../reference.html#park2017).

## Understanding ICP API

**Note:**

The `Tensor based ICP implementation` API is slightly different than the [Eigen based ICP implementation](../pipelines/icp_registration.rst), to support more functionalities.

### Input

`PointClouds` between which the `Transformation` is to be estimated. [open3d.t.PointCloud]

- Source Tensor PointCloud. [Float32 or Float64 dtypes are supported].
- Target Tensor PointCloud. [Float32 or Float64 dtypes are supported].

**Note:**

The initial alignment is usually obtained by a global registration algorithm.

```python
demo_icp_pcds = o3d.data.DemoICPPointClouds()
source = o3d.t.io.read_point_cloud(demo_icp_pcds.paths[0])
target = o3d.t.io.read_point_cloud(demo_icp_pcds.paths[1])

# For Colored-ICP `colors` attribute must be of the same dtype as `positions` and `normals` attribute.
source.point["colors"] = source.point["colors"].to(
    o3d.core.Dtype.Float32) / 255.0
target.point["colors"] = target.point["colors"].to(
    o3d.core.Dtype.Float32) / 255.0

# Initial guess transform between the two point-cloud.
# ICP algortihm requires a good initial allignment to converge efficiently.
trans_init = np.asarray([[0.862, 0.011, -0.507, 0.5],
                         [-0.139, 0.967, -0.215, 0.7],
                         [0.487, 0.255, 0.835, -1.4], [0.0, 0.0, 0.0, 1.0]])

draw_registration_result(source, target, trans_init)
```

---

### Parameters

#### Max correspondence Distances
- This is the radius of distance from each point in the source point-cloud in which the neighbour search will try to find a corresponding point in the target point-cloud.
- It is a `double` for `icp`, and `utility.DoubleVector` for `multi-scale-icp`.
- One may typically keep this parameter between `1.0x - 3.0x` `voxel-size` for each scale.
- This parameter is most important for performance tuning, as a higher radius will take larger time (as the neighbour search will be performed over a larger radius).

```python
# For Vanilla ICP (double)

# Search distance for Nearest Neighbour Search [Hybrid-Search is used].
max_correspondence_distance = 0.07
```

```python
# For Multi-Scale ICP (o3d.utility.DoubleVector):

# `max_correspondence_distances` is proportianal to the resolution or the `voxel_sizes`.
# In general it is recommended to use values between 1x - 3x of the corresponding `voxel_sizes`.
# We may have a higher value of the `max_correspondence_distances` for the first coarse
# scale, as it is not much expensive, and gives us more tolerance to initial allignment.
max_correspondence_distances = o3d.utility.DoubleVector([0.3, 0.14, 0.07])
```

#### Initial Transform from Source to Target [open3d.core.Tensor]
- Initial estimate for transformation from source to target.
- Transformation matrix Tensor of shape [4, 4] of type `Float64` on `CPU:0` device
- The initial alignment is usually obtained by a global registration algorithm. See [Global registration](../pipelines/global_registration.rst) for examples.

```python
# Initial alignment or source to target transform.
init_source_to_target = np.asarray([[0.862, 0.011, -0.507, 0.5],
                                    [-0.139, 0.967, -0.215, 0.7],
                                    [0.487, 0.255, 0.835, -1.4],
                                    [0.0, 0.0, 0.0, 1.0]])
```

#### Estimation Method
- This sets the ICP method to compute the transformation between two point-clouds given the correspondences.

Options:

- **o3d.t.pipelines.registration.TransformationEstimationPointToPoint()**
    - Point to Point ICP.
- **o3d.t.pipelines.registration.TransformationEstimationPointToPlane(robust_kernel)**
    - Point to Plane ICP.
    - Requires `target point-cloud` to have `normals` attribute (of same dtype as `position` attribute).
- **o3d.t.pipelines.registration.TransformationEstimationForColoredICP(robust_kernel, lambda)**
    - Colored ICP.
    - Requires `target` point-cloud to have `normals` attribute (of same dtype as `position` attribute).
    - Requires `source` and `target` point-clouds to have `colors` attribute (of same dtype as `position` attribute).
- **o3d.t.pipelines.registration.TransformationEstimationForGeneralizedICP(robust_kernel, epsilon)** [To be added].
    - Generalized ICP.

```python
# Select the `Estimation Method`, and `Robust Kernel` (for outlier-rejection).
estimation = treg.TransformationEstimationPointToPlane()
```

Estimation Method also supports `Robust Kernels`: Robust kernels are used for outlier rejection. More on this in `Robust Kernel` section.

`robust_kernel = o3d.t.pipelines.registration.robust_kernel.RobustKernel(method, scale, shape)`

Method options:

- robust_kernel.RobustKernelMethod.L2Loss
- robust_kernel.RobustKernelMethod.L1Loss
- robust_kernel.RobustKernelMethod.HuberLoss
- robust_kernel.RobustKernelMethod.CauchyLoss
- robust_kernel.RobustKernelMethod.GMLoss
- robust_kernel.RobustKernelMethod.TukeyLoss
- robust_kernel.RobustKernelMethod.GeneralizedLoss

#### ICP Convergence Criteria [relative rmse, relative fitness, max iterations]
- This sets the condition for termination or when the scale iterations can be considered to be converged.
- If the relative (of change in value from the last iteration) rmse and fitness are equal or less than the specified value, the iterations for that scale will be considered as converged/completed.
- For `Multi-Scale ICP` it is a `list` of `ICPConvergenceCriteria`, for each scale of ICP, to provide more fine control over performance.
- One may keep the initial values of `relative_fitness` and `relative_rmse` low as we just want to get an estimate transformation, and high for later iterations to fine-tune.
- Iterations on higher-resolution is more costly (takes more time), so we want to do fewer iterations on higher resolution.

```python
# Convergence-Criteria for Vanilla ICP:

criteria = treg.ICPConvergenceCriteria(relative_fitness=0.000001,
                                       relative_rmse=0.000001,
                                       max_iteration=50)
```

```python
# List of Convergence-Criteria for Multi-Scale ICP:

# We can control `ConvergenceCriteria` of each `scale` individually.
# We want to keep `relative_fitness` and `relative_rmse` high (more error tolerance)
# for initial scales, i.e. we will be happy to consider ICP converged, when difference
# between 2 successive iterations for that scale is smaller than this value.
# We expect less accuracy (more error tolerance) initial coarse-scale iteration,
# and want our later scale convergence to be more accurate (less error tolerance).
criteria_list = [
    treg.ICPConvergenceCriteria(relative_fitness=0.0001,
                                relative_rmse=0.0001,
                                max_iteration=20),
    treg.ICPConvergenceCriteria(0.00001, 0.00001, 15),
    treg.ICPConvergenceCriteria(0.000001, 0.000001, 10)
]
```

#### Voxel Sizes
- It is the voxel size (lower voxel size corresponds to higher resolution), for each scale of multi-scale ICP.
- We want to perform initial iterations on a coarse point-cloud (low-resolution or small voxel size) (as it is more time-efficient, and avoids local-minima), and then move to a dense point-cloud (high-resolution or small voxel size. Therefore the voxel sizes must be strictly decreasing order.

```python
# Vanilla ICP
voxel_size = 0.025
```

```python
# Lower `voxel_size` is equivalent to higher resolution,
# and we want to perform iterations from coarse to dense resolution,
# therefore `voxel_sizes` must be in strictly decressing order.
voxel_sizes = o3d.utility.DoubleVector([0.1, 0.05, 0.025])
```

#### Save Loss Log
When `True`, it saves the iteration-wise values of `fitness`, `inlier_rmse`, `transformaton`, `scale`, `iteration` in `loss_log_` in `regsitration_result`. Default: `False`.

```python
save_loss_log = True
```

---

## Vanilla ICP Example

#### 1. Set Inputs and Parameters

```python
# Input point-clouds
demo_icp_pcds = o3d.data.DemoICPPointClouds()
source = o3d.t.io.read_point_cloud(demo_icp_pcds.paths[0])
target = o3d.t.io.read_point_cloud(demo_icp_pcds.paths[1])
```

```python
# Search distance for Nearest Neighbour Search [Hybrid-Search is used].
max_correspondence_distance = 0.07

# Initial alignment or source to target transform.
init_source_to_target = np.asarray([[0.862, 0.011, -0.507, 0.5],
                                    [-0.139, 0.967, -0.215, 0.7],
                                    [0.487, 0.255, 0.835, -1.4],
                                    [0.0, 0.0, 0.0, 1.0]])

# Select the `Estimation Method`, and `Robust Kernel` (for outlier-rejection).
estimation = treg.TransformationEstimationPointToPlane()

# Convergence-Criteria for Vanilla ICP
criteria = treg.ICPConvergenceCriteria(relative_fitness=0.000001,
                                       relative_rmse=0.000001,
                                       max_iteration=50)
# Down-sampling voxel-size.
voxel_size = 0.025

# Save iteration wise `fitness`, `inlier_rmse`, etc. to analyse and tune result.
save_loss_log = True
```

#### 2. Get Registration Result from ICP

```python
s = time.time()

registration_icp = treg.icp(source, target, max_correspondence_distance,
                            init_source_to_target, estimation, criteria,
                            voxel_size, save_loss_log)

icp_time = time.time() - s
print("Time taken by ICP: ", icp_time)
print("Inlier Fitness: ", registration_icp.fitness)
print("Inlier RMSE: ", registration_icp.inlier_rmse)

draw_registration_result(source, target, registration_icp.transformation)
```

---
Now let's try with poor initial initialisation

```python
init_source_to_target = o3d.core.Tensor.eye(4, o3d.core.Dtype.Float32)
max_correspondence_distance = 0.07
```

```python
s = time.time()

registration_icp = treg.icp(source, target, max_correspondence_distance,
                            init_source_to_target, estimation, criteria,
                            voxel_size, save_loss_log)

icp_time = time.time() - s
print("Time taken by ICP: ", icp_time)
print("Inlier Fitness: ", registration_icp.fitness)
print("Inlier RMSE: ", registration_icp.inlier_rmse)

draw_registration_result(source, target, registration_icp.transformation)
```

As we can see, poor initial allignment might fail ICP convergence

Having large `max_correspondence_distance` might resolve this issue. But it will take longer to process.

```python
init_source_to_target = o3d.core.Tensor.eye(4, o3c.float32)
max_correspondence_distance = 0.5
```

```python
s = time.time()
# It is highly recommended to down-sample the point-cloud before using
# ICP algorithm, for better performance.

registration_icp = treg.icp(source, target, max_correspondence_distance,
                            init_source_to_target, estimation, criteria,
                            voxel_size, save_loss_log)

icp_time = time.time() - s
print("Time taken by ICP: ", icp_time)
print("Inlier Fitness: ", registration_icp.fitness)
print("Inlier RMSE: ", registration_icp.inlier_rmse)

draw_registration_result(source, target, registration_icp.transformation)
```

We may resolve the above issues and get even better accuracy by using `Multi-Scale ICP`

---

## Multi-Scale ICP Example

Problems with using Vanilla-ICP (previous version):

- Running the ICP algorithm on dense point-clouds is very slow.
- It requires good initial alignment:
    - If the point-cloud is not well aligned, the convergence might get stuck in local-minima in initial iterations.
    - We need to have a larger `max_correspondence_distance` if the aligned point cloud does not have sufficient overlaps.
- If point-cloud is heavily down-sampled (coarse), the obtained result will not be accurate.

These drawbacks can be solved using Multi-Scale ICP.
In Multi-Scale ICP, we perform the initial iterations on coarse point-cloud to get a better estimate of initial alignment and use this alignment for convergence on a more dense point cloud. ICP on coarse point cloud is in-expensive, and allows us to use a larger `max_correspondence_distance`. It is also less likely for the convergence to get stuck in local minima. As we get a good estimate, it takes fewer iterations on dense point-cloud to converge to a more accurate transform.

It is recommended to use `Multi-Scale ICP` over `ICP`, for efficient convergence, especially for large point clouds.

#### 1. Set Inputs and Parameters

```python
# Input point-clouds
demo_icp_pcds = o3d.data.DemoICPPointClouds()
source = o3d.t.io.read_point_cloud(demo_icp_pcds.paths[0])
target = o3d.t.io.read_point_cloud(demo_icp_pcds.paths[1])
```

```python
voxel_sizes = o3d.utility.DoubleVector([0.1, 0.05, 0.025])

# List of Convergence-Criteria for Multi-Scale ICP:
criteria_list = [
    treg.ICPConvergenceCriteria(relative_fitness=0.0001,
                                relative_rmse=0.0001,
                                max_iteration=20),
    treg.ICPConvergenceCriteria(0.00001, 0.00001, 15),
    treg.ICPConvergenceCriteria(0.000001, 0.000001, 10)
]

# `max_correspondence_distances` for Multi-Scale ICP (o3d.utility.DoubleVector):
max_correspondence_distances = o3d.utility.DoubleVector([0.3, 0.14, 0.07])

# Initial alignment or source to target transform.
init_source_to_target = o3d.core.Tensor.eye(4, o3d.core.Dtype.Float32)

# Select the `Estimation Method`, and `Robust Kernel` (for outlier-rejection).
estimation = treg.TransformationEstimationPointToPlane()

# Save iteration wise `fitness`, `inlier_rmse`, etc. to analyse and tune result.
save_loss_log = True
```

#### 2. Get Registration Result from Multi-Scale ICP

```python
# Setting Verbosity to Debug, helps in fine-tuning the performance.
# o3d.utility.set_verbosity_level(o3d.utility.VerbosityLevel.Debug)

s = time.time()

registration_ms_icp = treg.multi_scale_icp(source, target, voxel_sizes,
                                           criteria_list,
                                           max_correspondence_distances,
                                           init_source_to_target, estimation,
                                           save_loss_log)

ms_icp_time = time.time() - s
print("Time taken by Multi-Scale ICP: ", ms_icp_time)
print("Inlier Fitness: ", registration_ms_icp.fitness)
print("Inlier RMSE: ", registration_ms_icp.inlier_rmse)

draw_registration_result(source, target, registration_ms_icp.transformation)
```

### Plotting Convergence Graph
We can use the `registration_result.loss_log`, to plot convergence and fine-tune our application.

```python
from matplotlib import pyplot as plt

def plot_rmse(registration_result):
    fig, axes = plt.subplots(nrows=1, ncols=1, figsize=(20, 5))
    axes.set_title("Inlier RMSE vs Iteration")
    axes.plot(registration_result.loss_log["index"].numpy(),
              registration_result.loss_log["inlier_rmse"].numpy())

def plot_scale_wise_rmse(registration_result):
    scales = registration_result.loss_log["scale"].numpy()
    iterations = registration_result.loss_log["iteration"].numpy()

    num_scales = scales[-1][0] + 1

    fig, axes = plt.subplots(nrows=1, ncols=num_scales, figsize=(20, 5))

    masks = {}
    for scale in range(0, num_scales):
        masks[scale] = registration_result.loss_log["scale"] == scale

        rmse = registration_result.loss_log["inlier_rmse"][masks[scale]].numpy()
        iteration = registration_result.loss_log["iteration"][
            masks[scale]].numpy()

        title_prefix = "Scale Index: " + str(scale)
        axes[scale].set_title(title_prefix + " Inlier RMSE vs Iteration")
        axes[scale].plot(iteration, rmse)
```

```python
print("Vanilla ICP")
plot_rmse(registration_icp)
```

```python
print("Multi Scale ICP")
plot_rmse(registration_ms_icp)
plot_scale_wise_rmse(registration_ms_icp)
```

```python
fig, axes = plt.subplots(nrows=1, ncols=1, figsize=(20, 5))

axes.set_title("Vanilla ICP and Multi-Scale ICP `Inlier RMSE` vs `Iteration`")

if len(registration_ms_icp.loss_log["index"]) > len(
        registration_icp.loss_log["inlier_rmse"]):
    axes.plot(registration_ms_icp.loss_log["index"].numpy(),
              registration_ms_icp.loss_log["inlier_rmse"].numpy(),
              registration_icp.loss_log["inlier_rmse"].numpy())
else:
    axes.plot(registration_icp.loss_log["index"].numpy(),
              registration_icp.loss_log["inlier_rmse"].numpy(),
              registration_ms_icp.loss_log["inlier_rmse"].numpy())
```

---

### Multi-Scale ICP on CUDA device Example

```python
# The algorithm runs on the same device as the source and target point-cloud.
source_cuda = source.cuda(0)
target_cuda = target.cuda(0)
```

```python
s = time.time()

registration_ms_icp = treg.multi_scale_icp(source_cuda, target_cuda,
                                           voxel_sizes, criteria_list,
                                           max_correspondence_distances,
                                           init_source_to_target, estimation,
                                           save_loss_log)

ms_icp_time = time.time() - s
print("Time taken by Multi-Scale ICP: ", ms_icp_time)
print("Inlier Fitness: ", registration_ms_icp.fitness)
print("Inlier RMSE: ", registration_ms_icp.inlier_rmse)

draw_registration_result(source.cpu(), target.cpu(),
                         registration_ms_icp.transformation)
```

---

## Case of `no correspondences`.
In case of `no_correspondences` the `fitness` and `inlier_rmse` is `0`.

```python
max_correspondence_distance = 0.02

init_source_to_target = np.asarray([[1.0, 0.0, 0.0, 5], [0.0, 1.0, 0.0, 7],
                                    [0.0, 0.0, 1.0, 10], [0.0, 0.0, 0.0, 1.0]])

registration_icp = treg.icp(source, target, max_correspondence_distance,
                            init_source_to_target)

print("Inlier Fitness: ", registration_icp.fitness)
print("Inlier RMSE: ", registration_icp.inlier_rmse)
print("Transformation: \n", registration_icp.transformation)

if registration_icp.fitness == 0 and registration_icp.inlier_rmse == 0:
    print("ICP Convergence Failed, as no correspondence were found")
```

---

## Information Matrix

`Information Matrix` gives us futher information about how well the point-clouds are alligned.

```python
information_matrix = treg.get_information_matrix(
    source, target, max_correspondence_distances[2],
    registration_ms_icp.transformation)

print(information_matrix)
```

---

Now that we have a basic understanding of the ICP algorithm and the API, let's experiment with the different versions to understand the difference

## Initial Allignment

```python
demo_icp_pcds = o3d.data.DemoICPPointClouds()
source = o3d.t.io.read_point_cloud(demo_icp_pcds.paths[0])
target = o3d.t.io.read_point_cloud(demo_icp_pcds.paths[1])

# Initial guess transform between the two point-cloud.
# ICP algortihm requires a good initial allignment to converge efficiently.
trans_init = np.asarray([[0.862, 0.011, -0.507, 0.5],
                         [-0.139, 0.967, -0.215, 0.7],
                         [0.487, 0.255, 0.835, -1.4], [0.0, 0.0, 0.0, 1.0]])

draw_registration_result(source, target, trans_init)
```

```python
# Search distance for Nearest Neighbour Search [Hybrid-Search is used].
max_correspondence_distance = 0.02

print("Initial alignment")
evaluation = treg.evaluate_registration(source, target,
                                        max_correspondence_distance, trans_init)

print("Fitness: ", evaluation.fitness)
print("Inlier RMSE: ", evaluation.inlier_rmse)
```

---

## Point-To-Point ICP Registration

We first show a point-to-point ICP algorithm [\[BeslAndMcKay1992\]](../reference.html#beslandmckay1992) using the objective

\begin{equation}
E(\mathbf{T}) = \sum_{(\mathbf{p},\mathbf{q})\in\mathcal{K}}\|\mathbf{p} - \mathbf{T}\mathbf{q}\|^{2}
\end{equation}

The class `TransformationEstimationPointToPoint` provides functions to compute the residuals and Jacobian matrices of the point-to-point ICP objective.

#### 1. Set Inputs and Parameters

```python
# Input point-clouds
demo_icp_pcds = o3d.data.DemoICPPointClouds()
source = o3d.t.io.read_point_cloud(demo_icp_pcds.paths[0])
target = o3d.t.io.read_point_cloud(demo_icp_pcds.paths[1])
```

```python
# Select the `Estimation Method`, and `Robust Kernel` (for outlier-rejection).
estimation = treg.TransformationEstimationPointToPoint()
```

```python
# Search distance for Nearest Neighbour Search [Hybrid-Search is used].
max_correspondence_distance = 0.02

# Initial alignment or source to target transform.
init_source_to_target = trans_init

# Convergence-Criteria for Vanilla ICP
criteria = treg.ICPConvergenceCriteria(relative_fitness=0.0000001,
                                       relative_rmse=0.0000001,
                                       max_iteration=30)

# Down-sampling voxel-size. If voxel_size < 0, original scale is used.
voxel_size = -1

# Save iteration wise `fitness`, `inlier_rmse`, etc. to analyse and tune result.
save_loss_log = True
```

```python
print("Apply Point-to-Point ICP")
s = time.time()

reg_point_to_point = treg.icp(source, target, max_correspondence_distance,
                              init_source_to_target, estimation, criteria,
                              voxel_size, save_loss_log)

icp_time = time.time() - s
print("Time taken by Point-To-Point ICP: ", icp_time)
print("Fitness: ", reg_point_to_point.fitness)
print("Inlier RMSE: ", reg_point_to_point.inlier_rmse)

draw_registration_result(source, target, reg_point_to_point.transformation)
```

The fitness score increases from `0.174722` to `0.372474`. The inlier_rmse reduces from `0.011771` to `0.007761`.
By default, icp runs until convergence or reaches a maximum number of iterations (30 by default).
It can be changed to allow more computation time and to improve the results further.

```python
criteria = treg.ICPConvergenceCriteria(relative_fitness=0.0000001,
                                       relative_rmse=0.0000001,
                                       max_iteration=1000)
```

```python
print("Apply Point-to-Point ICP")
s = time.time()

reg_point_to_point = treg.icp(source, target, max_correspondence_distance,
                              init_source_to_target, estimation, criteria,
                              voxel_size, save_loss_log)

icp_time = time.time() - s
print("Time taken by Point-To-Point ICP: ", icp_time)
print("Fitness: ", reg_point_to_point.fitness)
print("Inlier RMSE: ", reg_point_to_point.inlier_rmse)

draw_registration_result(source, target, reg_point_to_point.transformation)
```

The final alignment is tight. The fitness score improves to `0.620972`. The inlier_rmse reduces to `0.006581`.

---

## Point-to-Plane ICP Registration
The point-to-plane ICP algorithm [\[ChenAndMedioni1992\]](../reference.html#chenandmedioni1992) uses a different objective function

\begin{equation}
E(\mathbf{T}) = \sum_{(\mathbf{p},\mathbf{q})\in\mathcal{K}}\big((\mathbf{p} - \mathbf{T}\mathbf{q})\cdot\mathbf{n}_{\mathbf{p}}\big)^{2},
\end{equation}

where $\mathbf{n}_{\mathbf{p}}$ is the normal of point $\mathbf{p}$. [\[Rusinkiewicz2001\]](../reference.html#rusinkiewicz2001) has shown that the point-to-plane ICP algorithm has a faster convergence speed than the point-to-point ICP algorithm.

The class `TransformationEstimationPointToPlane` provides functions to compute the residuals and Jacobian matrices of the point-to-plane ICP objective.

```python
estimation = treg.TransformationEstimationPointToPlane()

criteria = treg.ICPConvergenceCriteria(relative_fitness=0.0000001,
                                       relative_rmse=0.0000001,
                                       max_iteration=30)
```

```python
print("Apply Point-to-Plane ICP")
s = time.time()

reg_point_to_plane = treg.icp(source, target, max_correspondence_distance,
                              init_source_to_target, estimation, criteria,
                              voxel_size, save_loss_log)

icp_time = time.time() - s
print("Time taken by Point-To-Plane ICP: ", icp_time)
print("Fitness: ", reg_point_to_plane.fitness)
print("Inlier RMSE: ", reg_point_to_plane.inlier_rmse)

draw_registration_result(source, target, reg_point_to_plane.transformation)
```

The point-to-plane ICP reaches tight alignment within 30 iterations (a `fitness` score of 0.620972 and an `inlier_rmse` score of 0.006581).

---

## Colored ICP Registration
This tutorial demonstrates an ICP variant that uses both geometry and color for registration. It implements the algorithm of [\[Park2017\]](../reference.html#park2017). The color information locks the alignment along the tangent plane. Thus this algorithm is more accurate and more robust than prior point cloud registration algorithms, while the running speed is comparable to that of ICP registration.

```python
# Overriding visualization function, according to best camera view for colored-icp sample data.
def draw_registration_result(source, target, transformation):
    source_temp = source.clone()
    target_temp = target.clone()

    source_temp.transform(transformation)

    # This is patched version for tutorial rendering.
    # Use `draw` function for you application.
    o3d.visualization.draw_geometries(
        [source_temp.to_legacy(),
         target_temp.to_legacy()],
        zoom=0.5,
        front=[-0.2458, -0.8088, 0.5342],
        lookat=[1.7745, 2.2305, 0.9787],
        up=[0.3109, -0.5878, -0.7468])
```

```python
print("1. Load two point clouds and show initial pose")
demo_cicp_pcds = o3d.data.DemoColoredICPPointClouds()
source = o3d.t.io.read_point_cloud(demo_cicp_pcds.paths[0])
target = o3d.t.io.read_point_cloud(demo_cicp_pcds.paths[1])

# For Colored-ICP `colors` attribute must be of the same dtype as `positions` and `normals` attribute.
source.point["colors"] = source.point["colors"].to(
    o3d.core.Dtype.Float32) / 255.0
target.point["colors"] = target.point["colors"].to(
    o3d.core.Dtype.Float32) / 255.0

# draw initial alignment
current_transformation = np.identity(4)
draw_registration_result(source, target, current_transformation)
```

## Setting baseline with point-to-plane registration
We first run Point-to-plane ICP as a baseline approach. The visualization below shows misaligned green triangle textures. This is because a geometric constraint does not prevent two planar surfaces from slipping.

```python
estimation = treg.TransformationEstimationPointToPlane()
```

```python
max_correspondence_distance = 0.02
init_source_to_target = np.identity(4)
```

```python
print("Apply Point-to-Plane ICP")
s = time.time()

reg_point_to_plane = treg.icp(source, target, max_correspondence_distance,
                              init_source_to_target, estimation)

icp_time = time.time() - s
print("Time taken by Point-To-Plane ICP: ", icp_time)
print("Fitness: ", reg_point_to_plane.fitness)
print("Inlier RMSE: ", reg_point_to_plane.inlier_rmse)

draw_registration_result(source, target, reg_point_to_plane.transformation)
```

## Colored Registration
The core function for colored point cloud registration is `registration_colored_icp`. Following [\[Park2017\]](../reference.html#park2017), it runs ICP iterations (see [Point-to-point ICP](../pipelines/icp_registration.ipynb#Point-to-point-ICP) for details) with a joint optimization objective

\begin{equation}
E(\mathbf{T}) = (1-\delta)E_{C}(\mathbf{T}) + \delta E_{G}(\mathbf{T})
\end{equation}

where $\mathbf{T}$ is the transformation matrix to be estimated. $E_{C}$ and $E_{G}$ are the photometric and geometric terms, respectively. $\delta\in[0,1]$ is a weight parameter that has been determined empirically.

The geometric term $E_{G}$ is the same as the [Point-to-plane ICP](../pipelines/icp_registration.ipynb#Point-to-plane-ICP) objective

\begin{equation}
E_{G}(\mathbf{T}) = \sum_{(\mathbf{p},\mathbf{q})\in\mathcal{K}}\big((\mathbf{p} - \mathbf{T}\mathbf{q})\cdot\mathbf{n}_{\mathbf{p}}\big)^{2},
\end{equation}

where $\mathcal{K}$ is the correspondence set in the current iteration. $\mathbf{n}_{\mathbf{p}}$ is the normal of point $\mathbf{p}$.

The color term $E_{C}$ measures the difference between the color of point $\mathbf{q}$ (denoted as $C(\mathbf{q})$) and the color of its projection on the tangent plane of $\mathbf{p}$.

\begin{equation}
E_{C}(\mathbf{T}) = \sum_{(\mathbf{p},\mathbf{q})\in\mathcal{K}}\big(C_{\mathbf{p}}(\mathbf{f}(\mathbf{T}\mathbf{q})) - C(\mathbf{q})\big)^{2},
\end{equation}

where $C_{\mathbf{p}}(\cdot)$ is a precomputed function continuously defined on the tangent plane of $\mathbf{p}$. Function$\mathbf{f}(\cdot)$ projects a 3D point to the tangent plane. For more details, refer to [\[Park2017\]](../reference.html#park2017).

To further improve efficiency, [\[Park2017\]](../reference.html#park2017) proposes a multi-scale registration scheme.

```python
estimation = treg.TransformationEstimationForColoredICP()
```

```python
current_transformation = np.identity(4)

criteria_list = [
    treg.ICPConvergenceCriteria(relative_fitness=0.0001,
                                relative_rmse=0.0001,
                                max_iteration=50),
    treg.ICPConvergenceCriteria(0.00001, 0.00001, 30),
    treg.ICPConvergenceCriteria(0.000001, 0.000001, 14)
]

max_correspondence_distances = o3d.utility.DoubleVector([0.08, 0.04, 0.02])

voxel_sizes = o3d.utility.DoubleVector([0.04, 0.02, 0.01])
```

```python
# colored pointcloud registration
# This is implementation of following paper
# J. Park, Q.-Y. Zhou, V. Koltun,
# Colored Point Cloud Registration Revisited, ICCV 2017

print("Colored point cloud registration")
s = time.time()

reg_multiscale_icp = treg.multi_scale_icp(source, target, voxel_sizes,
                                          criteria_list,
                                          max_correspondence_distances,
                                          init_source_to_target, estimation)

icp_time = time.time() - s
print("Time taken by Colored ICP: ", icp_time)
print("Fitness: ", reg_point_to_plane.fitness)
print("Inlier RMSE: ", reg_point_to_plane.inlier_rmse)

draw_registration_result(source, target, reg_multiscale_icp.transformation)
```