<p align="center">

</p>

## Interactive Spatial Estimation with Strings of Data Demonstration

### Michael Pyrcz, Associate Professor, University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

### The Interactive Workflow

Here's a simple workflow for calculating and visualizing the simple kriging weights for a string of data.

#### Spatial Estimation

Consider the case of making an estimate at some unsampled location, $𝑧(\bf{u}_0)$, where $z$ is the property of interest (e.g. porosity etc.) and $𝐮_0$ is a location vector describing the unsampled location.

How would you do this given data, $𝑧(\bf{𝐮}_1)$, $𝑧(\bf{𝐮}_2)$, and $𝑧(\bf{𝐮}_3)$?

It would be natural to use a set of linear weights to formulate the estimator given the available data.

\begin{equation}
z^{*}(\bf{u}) = \sum^{n}_{\alpha = 1} \lambda_{\alpha} z(\bf{u}_{\alpha})
\end{equation}

We could add an unbiasedness constraint to impose the sum of the weights equal to one.  What we will do is assign the remainder of the weight (one minus the sum of weights) to the global average; therefore, if we have no informative data we will estimate with the global average of the property of interest.

\begin{equation}
z^{*}(\bf{u}) = \sum^{n}_{\alpha = 1} \lambda_{\alpha} z(\bf{u}_{\alpha}) + \left(1-\sum^{n}_{\alpha = 1} \lambda_{\alpha} \right) \overline{z}
\end{equation}

We will make a stationarity assumption, so let's assume that we are working with residuals, $y$.

\begin{equation}
y^{*}(\bf{u}) = z^{*}(\bf{u}) - \overline{z}(\bf{u})
\end{equation}

If we substitute this form into our estimator the estimator simplifies, since the mean of the residual is zero.

\begin{equation}
y^{*}(\bf{u}) = \sum^{n}_{\alpha = 1} \lambda_{\alpha} y(\bf{u}_{\alpha})
\end{equation}

while satisfying the unbaisedness constraint.

#### Kriging

Now the next question is what weights should we use?

We could use equal weighting, $\lambda = \frac{1}{n}$, and the estimator would be the average of the local data applied for the spatial estimate. This would not be very informative.

We could assign weights considering the spatial context of the data and the estimate:

* **spatial continuity** as quantified by the variogram (and covariance function)
* **redundancy** the degree of spatial continuity between all of the available data with themselves
* **closeness** the degree of spatial continuity between the avaiable data and the estimation location

The kriging approach accomplishes this, calculating the best linear unbiased weights for the local data to estimate at the unknown location.  The derivation of the kriging system and the resulting linear set of equations is available in the lecture notes.  Furthermore kriging provides a measure of the accuracy of the estimate!  This is the kriging estimation variance (sometimes just called the kriging variance).

\begin{equation}
\sigma^{2}_{E}(\bf{u}) = C(0) - \sum^{n}_{\alpha = 1} \lambda_{\alpha} C(\bf{u}_0 - \bf{u}_{\alpha})
\end{equation}

What is 'best' about this estimate? Kriging estimates are best in that they minimize the above estimation variance.

For lectures on kriging, start here - [spatial estimation](https://www.youtube.com/watch?v=CVkmuwF8cJ8&list=PLG19vXLQHvSB-D4XKYieEku9GQMQyAzjJ&index=42&t=808s).

#### Properties of Kriging

Here are some important properties of kriging:

* **Exact interpolator** - kriging estimates with the data values at the data locations
* **Kriging variance** can be calculated before getting the sample information, as the kriging estimation variance is not dependent on the values of the data nor the kriging estimate, i.e. the kriging estimator is homoscedastic.
* **Spatial context** - kriging takes into account, furthermore to the statements on spatial continuity, closeness and redundancy we can state that kriging accounts for the configuration of the data and structural continuity of the variable being estimated.
* **Scale** - kriging may be generalized to account for the support volume of the data and estimate. We will cover this later.
* **Multivariate** - kriging may be generalized to account for multiple secondary data in the spatial estimate with the cokriging system. We will cover this later.
* **Smoothing effect** of kriging can be forecast. We will use this to build stochastic simulations later.

#### Balancing Closeness and Redudancy

The simple kriging system balances:

* **closeness** - spatial correlation between the data and the unknown location

* **redundancy** - spatial correlation between the data (all possible pairs)

This demonstration will illustrate this trade-off by observing the weights assigned to a string of data as the you interactively change the:

* **data density** - spacing of the data on the string

* **spatial continuity** - through the anisotropic variogram model (single structure with nugget effect)

* **spatial estimate location** - relative to the string of data

Check out the behavoir of this system, see if you can cause a string effect!

* **string effect** - majority of weight assigned to the peripheral data along a string due to perceived lower redudancy.  This is due to the assumption of kriging that the estimation does not have a boundary (i.e. it is embedded in an infinit domain).

* observe this with high data redundancy (large along string continuity relative to data spacing)

#### Spatial Continuity

**Spatial Continuity** is the correlation between values over distance.

* No spatial continuity – no correlation between values over distance, random values at each location in space regardless of separation distance.

* Homogenous phenomenon have perfect spatial continuity, since all values as the same (or very similar) they are correlated.

We need a statistic to quantify spatial continuity! A convenient method is the Semivariogram.

#### Semivariogram Recall

Function of difference over distance.

* The expected (average) squared difference between values separated by a lag distance vector (distance and direction), $h$:

\begin{equation}
\gamma(\bf{h}) = \frac{1}{2 N(\bf{h})} \sum^{N(\bf{h})}_{\alpha=1} (z(\bf{u}_\alpha) - z(\bf{u}_\alpha + \bf{h}))^2
\end{equation}

where $z(\bf{u}_\alpha)$ and $z(\bf{u}_\alpha + \bf{h})$ are the spatial sample values at tail and head locations of the lag vector respectively.

* Calculated over a suite of lag distances to obtain a continuous function.

* the $\frac{1}{2}$ term converts a variogram into a semivariogram, but in practice the term variogram is used instead of semivariogram.
* We prefer the semivariogram because it relates directly to the covariance function, $C_x(\bf{h})$ and univariate variance, $\sigma^2_x$:

\begin{equation}
C_x(\bf{h}) = \sigma^2_x - \gamma(\bf{h})
\end{equation}

Note the correlogram is related to the covariance function as:

\begin{equation}
\rho_x(\bf{h}) = \frac{C_x(\bf{h})}{\sigma^2_x}
\end{equation}

The correlogram provides of function of the $\bf{h}-\bf{h}$ scatter plot correlation vs. lag offset $\bf{h}$.

\begin{equation}
-1.0 \le \rho_x(\bf{h}) \le 1.0
\end{equation}

For lectures on the semivariogram, start here - [spatial continuity lecture](https://www.youtube.com/watch?v=j0I5SGFm00c&list=PLG19vXLQHvSB-D4XKYieEku9GQMQyAzjJ&index=32&t=516s).

#### Objective

In the PGE 383: Stochastic Subsurface Modeling class I want to provide hands-on experience with building subsurface modeling workflows. Python provides an excellent vehicle to accomplish this. I have coded a package called GeostatsPy with GSLIB: Geostatistical Library (Deutsch and Journel, 1998) functionality that provides basic building blocks for building subsurface modeling workflows.

The objective is to remove the hurdles of subsurface modeling workflow construction by providing building blocks and sufficient examples. This is not a coding class per se, but we need the ability to 'script' workflows working with numerical methods.

#### Getting Started

Here's the steps to get setup in Python with the GeostatsPy package:

1. Install Anaconda 3 on your machine (https://www.anaconda.com/download/).
2. From Anaconda Navigator (within Anaconda3 group), go to the environment tab, click on base (root) green arrow and open a terminal.
3. In the terminal type: pip install geostatspy.
4. Open Jupyter and in the top block get started by copy and pasting the code block below from this Jupyter Notebook to start using the geostatspy functionality.

No external data is required for this demonstration.

#### Load the required libraries

The following code loads the required libraries.

```python
import geostatspy.GSLIB as GSLIB                       # GSLIB utilies, visualization and wrapper
import geostatspy.geostats as geostats                 # GSLIB methods convert to Python
```

We will also need some standard packages. These should have been installed with Anaconda 3.

```python
%matplotlib inline
import os                                               # to set current working directory
import sys                                              # supress output to screen for interactive variogram modeling
import io
import numpy as np                                      # arrays and matrix math
import pandas as pd                                     # DataFrames
import matplotlib.pyplot as plt                         # plotting
from matplotlib.pyplot import cm                        # color maps
from matplotlib.patches import Ellipse                  # plot an ellipse
import matplotlib.lines as lines
import math                                             # sqrt operator
import random                                           # random simulation locations
from copy import copy                                   # copy a colormap
from scipy.stats import norm
from ipywidgets import interactive                      # widgets and interactivity
from ipywidgets import widgets
from ipywidgets import Layout
from ipywidgets import Label
from ipywidgets import VBox, HBox
from scipy.stats import norm                            # Gaussian distribution
```

If you get a package import error, you may have to first install some of these packages. This can usually be accomplished by opening up a command window on Windows and then typing 'python -m pip install [package-name]'. More assistance is available with the respective package docs.

#### Simple, Simple Kriging Function

Let's write a fast Python function to take data points and unknown location and provide the:

* **simple kriging estimate**

* **simple kriging variance / estimation variance**

* **simple kriging weights**

This provides a fast method for small datasets, with less parameters (no search parameters) and the ability to see the simple kriging weights.

* we use it here for fast, flexible application of sequential simulation

* the method will not work with only ones simulation location so we send 2 and only use the first result (the 2nd is always a dummy location in the workflow below.

```python
def simple_simple_krige(df,xcol,ycol,vcol,dfl,xlcol,ylcol,vario,skmean):
# load the variogram
    nst = vario['nst']; pmx = 9999.9
    cc = np.zeros(nst); aa = np.zeros(nst); it = np.zeros(nst)
    ang = np.zeros(nst); anis = np.zeros(nst)
    nug = vario['nug']; sill = nug
    cc[0] = vario['cc1']; sill = sill + cc[0]
    it[0] = vario['it1']; ang[0] = vario['azi1'];
    aa[0] = vario['hmaj1']; anis[0] = vario['hmin1']/vario['hmaj1'];
    if nst == 2:
        cc[1] = vario['cc2']; sill = sill + cc[1]
        it[1] = vario['it2']; ang[1] = vario['azi2'];
        aa[1] = vario['hmaj2']; anis[1] = vario['hmin2']/vario['hmaj2'];

# set up the required matrices
    rotmat, maxcov = geostats.setup_rotmat(nug,nst,it,cc,ang,pmx)
    ndata = len(df); a = np.zeros([ndata,ndata]); r = np.zeros(ndata); s = np.zeros(ndata); rr = np.zeros(ndata)
    nest = len(dfl)

    est = np.zeros(nest); var = np.full(nest,sill); weights = np.zeros([nest,ndata])

# Make and solve the kriging matrix, calculate the kriging estimate and variance
    for iest in range(0,nest):
        for idata in range(0,ndata):
            for jdata in range(0,ndata):
                a[idata,jdata] = geostats.cova2(df[xcol].values[idata],df[ycol].values[idata],df[xcol].values[jdata],df[ycol].values[jdata],
                                        nst,nug,pmx,cc,aa,it,ang,anis,rotmat,maxcov)
            r[idata] = geostats.cova2(df[xcol].values[idata],df[ycol].values[idata],dfl[xlcol].values[iest],dfl[ylcol].values[iest],
                                        nst,nug,pmx,cc,aa,it,ang,anis,rotmat,maxcov)
            rr[idata] = r[idata]

        s = geostats.ksol_numpy(ndata,a,r)
        sumw = 0.0
        for idata in range(0,ndata):
            sumw = sumw + s[idata]
            weights[iest,idata] = s[idata]
            est[iest] = est[iest] + s[idata]*df[vcol].values[idata]
            var[iest] = var[iest] - s[idata]*rr[idata]
        est[iest] = est[iest] + (1.0-sumw)*skmean
    return est,var,weights
```

Now let's set up our dash board.

```python
import warnings; warnings.simplefilter('ignore')

# dashboard: number of simulation locations and variogram parameters
style = {'description_width': 'initial'}
l = widgets.Text(value='         Spatial Estimation with Simple Kriging: Closeness, Redundancy Demonstration, Michael Pyrcz, Associate Professor, The University of Texas at Austin',layout=Layout(width='950px', height='30px'))
spacing = widgets.FloatSlider(min = 0.1, max = 5, value = 1, step =0.1, description = 'data spacing',orientation='vertical',
                          layout=Layout(width='100px', height='200px'),continuous_update=False)
spacing.style.handle_color = 'gray'
nug = widgets.FloatSlider(min = 0, max = 1.0, value = 0.0, step = 0.1, description = 'nugget effect',orientation='vertical',
                          layout=Layout(width='100px', height='200px'),continuous_update=False)
nug.style.handle_color = 'gray'
it1 = widgets.Dropdown(options=['Spherical', 'Exponential', 'Gaussian'],value='Spherical',
    description='Type1:',disabled=False,layout=Layout(width='180px', height='30px'), style=style,continuous_update=False)

dip = widgets.FloatSlider(min=-90, max = 90, value = 0, step = 0.10, description = 'continuity dip',
                        orientation='vertical',layout=Layout(width='100px', height='200px'),continuous_update=False)
dip.style.handle_color = 'gray'
hmaj1 = widgets.FloatSlider(min=0.01, max = 10000.0, value = 5000.0, step = 100.0, description = 'horizontal range',
                        orientation='vertical',layout=Layout(width='100px', height='200px'),continuous_update=False)
hmaj1.style.handle_color = 'gray'
hmin1 = widgets.FloatSlider(min = 0.01, max = 200.0, value = 3.0, step = 0.1, description = 'vertical range',
                        orientation='vertical',layout=Layout(width='100px', height='200px'),continuous_update=False)
hmin1.style.handle_color = 'gray'
uikvar = widgets.HBox([spacing,nug,it1,dip,hmaj1,hmin1],)

# dashboard: data locations
x1 = widgets.FloatSlider(min=0.0, max = 1000.0, value = 0.0, step = 4.0, description = 'x1',orientation='horizontal',
                         layout=Layout(width='180px', height='30px'),readout_format = '.0f',style=style,continuous_update=False)
x1.style.handle_color = 'blue'
z1 = widgets.FloatSlider(min=0.0, max = 10.0, value = 5.0, step = 1.0, description = 'z1',orientation='vertical',
                         layout=Layout(width='90px', height='180px'),readout_format = '.0f',style=style,continuous_update=False)
z1.style.handle_color = 'blue'
uik1 = widgets.VBox([x1,z1],)

uipars = widgets.HBox([uikvar,uik1],)
uik = widgets.VBox([l,uipars],)

def convert_type(it):
    if it == 'Spherical':
        return 1
    elif it == 'Exponential':
        return 2
    else:
        return 3

def f_make_krige(spacing,nug,it1,dip,hmaj1,hmin1,x1,z1): # function to take parameters, make sample and plot
    text_trap = io.StringIO()                           # suppress all text function output to dashboard to avoid clutter
    sys.stdout = text_trap
    cmap = cm.inferno
    np.random.seed(seed = 73073)                        # ensure same results for all runs
    it1 = convert_type(it1)
    nst = 1; xlag = 10; nlag = int(hmaj1/xlag); c1 = 1.0-nug; azi = 90.0+dip
    vario = GSLIB.make_variogram(nug,nst,it1,c1,azi,hmaj1,hmin1) # make model object
    index_maj,h_maj,gam_maj,cov_maj,ro_maj = geostats.vmodel(nlag,xlag,azi,vario)   # project the model in the major azimuth                                                  # project the model in the 135 azimuth
    index_min,h_min,gam_min,cov_min,ro_min = geostats.vmodel(nlag,xlag,azi+90.0,vario) # project the model in the minor azimuth

    seed = 73073

# make static hard data dataframe and hard code the data values
    number = int(10.0/spacing + 1)
    z = np.linspace(0,10,number)
    x = np.zeros(len(z)); value = np.zeros(len(z))
    df = pd.DataFrame({'X':x,'Z':z,'Value':value})
    ndata = len(df); skmean = np.average(df['Value'].values)

# make the estimate locations (we will need some dummy locations, the code doesn't work with 1 location)
    dfl = pd.DataFrame({'X':[x1,-99999,99999],'Z':[z1,-99999,99999], 'Value':[0.0,-99999,-99999]},dtype=np.single)
    sk_est_temp, sk_var_temp, sk_weights_temp = simple_simple_krige(df,'X','Z','Value',dfl,'X','Z',vario,skmean=skmean)
    sk_est = sk_est_temp[0]
    sk_var = sk_var_temp[0]
    df['weights'] = sk_weights_temp[0,:]  # get the data weights for the first unknown location, the others are dummy locations
    print('weights'); print(sk_weights_temp[0,:])
# plot the variogram model
    xlag = 10.0; nlag = int(hmaj1/xlag)
    plt.subplot(1,3,1)
    plt.plot([0,hmaj1*1.5],[1.0,1.0],color = 'black')
    plt.plot(h_maj,gam_maj,color = 'black',label = 'Horizontal at Dip ' + str(azi-90))
    plt.plot(h_min,gam_min,color = 'gray',label = 'Vertical, Orthogonal to Dip')
    deltas = [75.0,60.0,45.0,30.0,15.0];
    ndelta = len(deltas); hd = np.zeros(ndelta); gamd = np.zeros(ndelta);
    color=iter(cm.plasma(np.linspace(0,1,ndelta)))
    for delta in deltas:
        index,hd,gamd,cov,ro = geostats.vmodel(nlag,xlag,dip+delta,vario);
        c=next(color)
        plt.plot(hd,gamd,color = c,label = 'Dip ' + str(round(dip+delta-90.0,2)))
    plt.xlabel(r'Lag Distance $\bf(h)$, (m)')
    plt.ylabel(r'$\gamma \bf(h)$')
    plt.title('Interpolated Directional Variogram Models')
    plt.xlim([0,hmaj1*1.5])
    plt.ylim([0,1.4])
    plt.legend(loc='lower right')
    max_weight = np.max(df['weights'].values)

# plot the data and simulated values on a scatter plot
    plt.subplot(1,3,2)
    # plot the data colored by their weights
    my_plot = plt.scatter(df['X'],df['Z'],marker='o',
            c = df['weights'], cmap = cmap, vmin = 0.0, vmax = 1.2, linewidths = 0.01, edgecolors = 'black',
            s = 500/number,label = 'Original Data', alpha = 0.6)
    # make a custom line chart on the scatter plot for the weights
    pos = df['X'].values + df['weights'].values/1.0 * 500
    print('pos'); print(pos)
    line = lines.Line2D([0, 500], [0, 0], color = 'grey')
    line2 = lines.Line2D([500, 500], [0, -0.2], color = 'grey')
    line3 = lines.Line2D([400, 400], [0, -0.2], color = 'grey')
    line4 = lines.Line2D([300, 300], [0, -0.2], color = 'grey')
    line5 = lines.Line2D([200, 200], [0, -0.2], color = 'grey')
    line6 = lines.Line2D([100, 100], [0, -0.2], color = 'grey')
    ax = plt.gca()
    ax.add_line(line); ax.add_line(line2); ax.add_line(line3); ax.add_line(line4); ax.add_line(line5); ax.add_line(line6)
    ax.annotate('Weights', (220.0,-1.8)); ax.annotate('1.0', (480.0,-0.9))
    ax.annotate('0.8', (380.0,-0.9)); ax.annotate('0.6', (280.0,-0.9));
    ax.annotate('0.4', (180.0,-0.9)); ax.annotate('0.2', (80.0,-0.9))
    plt.plot(pos,df['Z'].values,'-',color = 'red',alpha = 0.6)
    plt.scatter(x1,z1,marker='^', color = 'red', alpha = 0.6,edgecolors = 'black')
    ax.annotate('?', (x1+10,z1+0.10))
    # add labels and colorbar
    plt.xlabel('X(m)'); plt.ylabel('Z(m)')
    plt.title('Spatial String of Data with Weights and Unknown Location')
    plt.xlim([-10,1000])
    plt.ylim([-2,12])
    cbar = plt.colorbar(my_plot, orientation="vertical", ticks=np.linspace(0, 1.2, 13))
    cbar.set_label('Simple Kriging Weights', rotation=270, labelpad=20)

    # add continuity ranges
    ellipse = Ellipse((x1, z1),width=hmin1*2.0,height=hmaj1*2.0,angle = 360-azi,facecolor='gray',alpha = 0.1)
    ax.add_patch(ellipse)

    plt.subplots_adjust(left=0.0, bottom=0.0, right=3.2, top=0.9, wspace=0.3, hspace=0.3)
    plt.show()

# connect the function to make the samples and plot to the widgets
interactive_plot = widgets.interactive_output(f_make_krige, {'spacing':spacing,'nug':nug, 'it1':it1, 'dip':dip, 'hmaj1':hmaj1, 'hmin1':hmin1,
                                                      'x1':x1, 'z1':z1,})
#interactive_plot.clear_output(wait = True)               # reduce flickering by delaying plot updating
```

### Interactive Kriging String Effect

* select the variogram model and the data locations and observe data weights along a string of data

#### Michael Pyrcz, Associate Professor, University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1) | [GeostatsPy](https://github.com/GeostatsGuy/GeostatsPy)

### The Inputs

Select the variogram model and the data locations:

* **nug**: nugget effect

* **c1**: contributions of the sill

* **hmaj1** / **hmin1**: range in the major and minor direction

* **(x1, y1)**: spatial data location

```python
display(uik, interactive_plot)                            # display the interactive plot
```

    VBox(children=(Text(value='         Spatial Estimation with Simple Kriging: Closeness, Redundancy Demonstratio…

    Output(outputs=({'output_type': 'display_data', 'data': {'text/plain': '<Figure size 432x288 with 3 Axes>', 'i…

#### Comments

This was an interactive demonstration of simple kriging with strings of data. Much more could be done, I have other demonstrations on the basics of working with DataFrames, ndarrays, univariate statistics, plotting data, declustering, data transformations and many other workflows available at https://github.com/GeostatsGuy/PythonNumericalDemos and https://github.com/GeostatsGuy/GeostatsPy.

#### The Author:

### Michael Pyrcz, Associate Professor, University of Texas at Austin
*Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions*

With over 17 years of experience in subsurface consulting, research and development, Michael has returned to academia driven by his passion for teaching and enthusiasm for enhancing engineers' and geoscientists' impact in subsurface resource development.

For more about Michael check out these links:

#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

#### Want to Work Together?

I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.

* Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I'd be happy to drop by and work with you!

* Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!

* I can be reached at mpyrcz@austin.utexas.edu.

I'm always happy to discuss,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin

#### More Resources Available at: [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)


# Maximum Mean Discrepancy drift detector on CIFAR-10

### Method

The [Maximum Mean Discrepancy (MMD)](http://jmlr.csail.mit.edu/papers/v13/gretton12a.html) detector is a kernel-based method for multivariate 2 sample testing. The MMD is a distance-based measure between 2 distributions *p* and *q* based on the mean embeddings $\mu_{p}$ and $\mu_{q}$ in a reproducing kernel Hilbert space $F$:

\begin{align}
MMD(F, p, q) & = || \mu_{p} - \mu_{q} ||^2_{F} \\
\end{align}

We can compute unbiased estimates of $MMD^2$ from the samples of the 2 distributions after applying the kernel trick. We use by default a [radial basis function kernel](https://en.wikipedia.org/wiki/Radial_basis_function_kernel), but users are free to pass their own kernel of preference to the detector. We obtain a $p$-value via a [permutation test](https://en.wikipedia.org/wiki/Resampling_(statistics)) on the values of $MMD^2$. This method is also described in [Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift](https://arxiv.org/abs/1810.11953).

### Dataset

[CIFAR10](https://www.cs.toronto.edu/~kriz/cifar.html) consists of 60,000 32 by 32 RGB images equally distributed over 10 classes. We evaluate the drift detector on the CIFAR-10-C dataset ([Hendrycks & Dietterich, 2019](https://arxiv.org/abs/1903.12261)). The instances in
CIFAR-10-C have been corrupted and perturbed by various types of noise, blur, brightness etc. at different levels of severity, leading to a gradual decline in the classification model performance. We also check for drift against the original test set with class imbalances.

```python
import matplotlib.pyplot as plt
import os
import numpy as np
import tensorflow as tf
from tensorflow.keras.layers import Conv2D, Dense, Flatten, InputLayer, Reshape

from alibi_detect.cd import MMDDrift
from alibi_detect.cd.preprocess import UAE, HiddenOutput
from alibi_detect.models.resnet import scale_by_instance
from alibi_detect.utils.fetching import fetch_tf_model
from alibi_detect.utils.kernels import gaussian_kernel
from alibi_detect.utils.prediction import predict_batch
from alibi_detect.utils.saving import save_detector, load_detector
from alibi_detect.datasets import fetch_cifar10c, corruption_types_cifar10c
```

### Load data

Original CIFAR-10 data:

```python
(X_train, y_train), (X_test, y_test) = tf.keras.datasets.cifar10.load_data()
X_train = X_train.astype('float32') / 255
X_test = X_test.astype('float32') / 255
y_train = y_train.astype('int64').reshape(-1,)
y_test = y_test.astype('int64').reshape(-1,)
```

For CIFAR-10-C, we can select from the following corruption types at 5 severity levels:

```python
corruptions = corruption_types_cifar10c()
print(corruptions)
```

    ['brightness', 'contrast', 'defocus_blur', 'elastic_transform', 'fog', 'frost', 'gaussian_blur', 'gaussian_noise', 'glass_blur', 'impulse_noise', 'jpeg_compression', 'motion_blur', 'pixelate', 'saturate', 'shot_noise', 'snow', 'spatter', 'speckle_noise', 'zoom_blur']

Let's pick a subset of the corruptions at corruption level 5. Each corruption type consists of perturbations on all of the original test set images.

```python
corruption = ['gaussian_noise', 'motion_blur', 'brightness', 'pixelate']
X_corr, y_corr = fetch_cifar10c(corruption=corruption, severity=5, return_X_y=True)
X_corr = X_corr.astype('float32') / 255
```

We split the original test set in a reference dataset and a dataset which should not be rejected under the *H0* of the MMD test. We also split the corrupted data by corruption type:

```python
np.random.seed(0)
n_test = X_test.shape[0]
idx = np.random.choice(n_test, size=n_test // 2, replace=False)
idx_h0 = np.delete(np.arange(n_test), idx, axis=0)
X_ref,y_ref = X_test[idx], y_test[idx]
X_h0, y_h0 = X_test[idx_h0], y_test[idx_h0]
print(X_ref.shape, X_h0.shape)
```

    (5000, 32, 32, 3) (5000, 32, 32, 3)

```python
# check that the classes are more or less balanced
classes, counts_ref = np.unique(y_ref, return_counts=True)
counts_h0 = np.unique(y_h0, return_counts=True)[1]
print('Class Ref H0')
for cl, cref, ch0 in zip(classes, counts_ref, counts_h0):
    assert cref + ch0 == n_test // 10
    print('{}     {} {}'.format(cl, cref, ch0))
```

    Class Ref H0
    0     472 528
    1     510 490
    2     498 502
    3     492 508
    4     501 499
    5     495 505
    6     493 507
    7     501 499
    8     516 484
    9     522 478

```python
X_c = []
n_corr = len(corruption)
for i in range(n_corr):
    X_c.append(X_corr[i * n_test:(i + 1) * n_test])
```

We can visualise the same instance for each corruption type:

```python
i = 4

n_test = X_test.shape[0]
plt.title('Original')
plt.axis('off')
plt.imshow(X_test[i])
plt.show()
for _ in range(len(corruption)):
    plt.title(corruption[_])
    plt.axis('off')
    plt.imshow(X_corr[n_test * _+ i])
    plt.show()
```

We can also verify that the performance of a classification model on CIFAR-10 drops significantly on this perturbed dataset:

```python
dataset = 'cifar10'
model = 'resnet32'
clf = fetch_tf_model(dataset, model)
acc = clf.evaluate(scale_by_instance(X_test), y_test, batch_size=128, verbose=0)[1]
print('Test set accuracy:')
print('Original {:.4f}'.format(acc))
clf_accuracy = {'original': acc}
for _ in range(len(corruption)):
    acc = clf.evaluate(scale_by_instance(X_c[_]), y_test, batch_size=128, verbose=0)[1]
    clf_accuracy[corruption[_]] = acc
    print('{} {:.4f}'.format(corruption[_], acc))
```

    Test set accuracy:
    Original 0.9278
    gaussian_noise 0.2208
    motion_blur 0.6339
    brightness 0.8913
    pixelate 0.3666

Given the drop in performance, it is important that we detect the harmful data drift!

### Detect drift

We are trying to detect data drift on high-dimensional (*32x32x3*) data using a multivariate MMD permutation test. It therefore makes sense to apply dimensionality reduction first. Some dimensionality reduction methods also used in [Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift](https://arxiv.org/pdf/1810.11953.pdf) are readily available: **UAE** (Untrained AutoEncoder), **BBSDs** (black-box shift detection using the classifier's softmax outputs) and **PCA**.

#### Untrained AutoEncoder

First we try UAE:

```python
tf.random.set_seed(0)

# define encoder
encoding_dim = 32
encoder_net = tf.keras.Sequential(
  [
      InputLayer(input_shape=(32, 32, 3)),
      Conv2D(64, 4, strides=2, padding='same', activation=tf.nn.relu),
      Conv2D(128, 4, strides=2, padding='same', activation=tf.nn.relu),
      Conv2D(512, 4, strides=2, padding='same', activation=tf.nn.relu),
      Flatten(),
      Dense(encoding_dim,)
  ]
)
uae = UAE(encoder_net=encoder_net)
preprocess_kwargs = {'model': uae, 'batch_size': 128}

# initialise drift detector
cd = MMDDrift(
    p_val=.05,          # p-value for permutation test
    X_ref=X_ref,        # reference data to test against
    preprocess_kwargs=preprocess_kwargs,
    kernel=gaussian_kernel,  # use the default Gaussian kernel in MMD
    kernel_kwargs={'sigma': np.array([1.])},
    chunk_size=1000,
    n_permutations=5    # nb of permutations in the test, set to 5 for runtime
)                       # purposes; should be much higher for a real test

# we can also save/load an initialised detector
filepath = 'my_path'  # change to directory where detector is saved
save_detector(cd, filepath)
cd = load_detector(filepath, **{'preprocess_kwargs': preprocess_kwargs})
```

    WARNING:alibi_detect.utils.saving:Directory my_path does not exist and is now created.

The optional `chunk_size` variable will be used to compute the maximum mean discrepancy distance between the 2 samples in chunks using [dask](https://dask.org/) to avoid potential out-of-memory errors. In terms of speed, the optimal `chunk_size` is application and hardware dependent, so it is often worth to test a few different values, including *None*. *None* means that the computation is done in-memory in NumPy.

Let's check whether the detector thinks drift occurred within the original test set:

```python
preds_h0 = cd.predict(X_h0, return_p_val=True)
labels = ['No!', 'Yes!']
print('Drift? {}'.format(labels[preds_h0['data']['is_drift']]))
```

    Drift? No!

As expected, no drift occurred. The p-value of the permutation test is above the $0.05$ threshold:

```python
print(preds_h0['data']['p_val'])
```

    0.8

Let's now check the predictions on the perturbed data:

```python
for x, c in zip(X_c, corruption):
    preds = cd.predict(x, return_p_val=True)
    print(f'Corruption type: {c}')
    print('Drift? {}'.format(labels[preds['data']['is_drift']]))
    print('Feature-wise p-values:')
    print(preds['data']['p_val'])
    print('')
```

    Corruption type: gaussian_noise
    Drift? Yes!
    Feature-wise p-values:
    0.0

    Corruption type: motion_blur
    Drift? Yes!
    Feature-wise p-values:
    0.0

    Corruption type: brightness
    Drift? Yes!
    Feature-wise p-values:
    0.0

    Corruption type: pixelate
    Drift? Yes!
    Feature-wise p-values:
    0.0

#### BBSDs

For **BBSDs**, we use the classifier's softmax outputs for black-box shift detection. This method is based on [Detecting and Correcting for Label Shift with Black Box Predictors](https://arxiv.org/abs/1802.03916). The ResNet classifier is trained on data standardised by instance so we need to rescale the data.

```python
X_train = scale_by_instance(X_train)
X_test = scale_by_instance(X_test)
for i in range(n_corr):
    X_c[i] = scale_by_instance(X_c[i])
X_ref = scale_by_instance(X_ref)
X_h0 = scale_by_instance(X_h0)
```

Initialisation of the drift detector. Here we use the output of the softmax layer to detect the drift, but other hidden layers can be extracted as well by setting *'layer'* to the index of the desired hidden layer in the model:

```python
# use output softmax layer
preprocess_kwargs = {'model': HiddenOutput(model=clf, layer=-1), 'batch_size': 128}

cd = MMDDrift(
    p_val=.05,
    X_ref=X_ref,
    preprocess_kwargs=preprocess_kwargs,
    kernel_kwargs={'sigma': np.array([1.])},
    chunk_size=1000,
    n_permutations=5
)
```

There is no drift on the original held out test set:

```python
preds_h0 = cd.predict(X_h0)
print('Drift? {}'.format(labels[preds_h0['data']['is_drift']]))
print(preds_h0['data']['p_val'])
```

    Drift? No!
    0.4

We compare this with the perturbed data:

```python
for x, c in zip(X_c, corruption):
    preds = cd.predict(x)
    print(f'Corruption type: {c}')
    print('Drift? {}'.format(labels[preds['data']['is_drift']]))
    print('Feature-wise p-values:')
    print(preds['data']['p_val'])
    print('')
```

    Corruption type: gaussian_noise
    Drift? Yes!
    Feature-wise p-values:
    0.0

    Corruption type: motion_blur
    Drift? Yes!
    Feature-wise p-values:
    0.0

    Corruption type: brightness
    Drift? Yes!
    Feature-wise p-values:
    0.0

    Corruption type: pixelate
    Drift? Yes!
    Feature-wise p-values:
    0.0

### Kernel bandwidth

So far we have defined a specific bandwidth `sigma` for the Gaussian kernel. We can however also sum over a number of different kernel bandwidths or infer `sigma` from *X_ref* and *X* using the following heuristic: compute the pairwise distances between each of the instances in *X_ref* and *X*, and set `sigma` to the median distance.

Let's first try a range of bandwidths:

```python
cd = MMDDrift(
    p_val=.05,
    X_ref=X_ref,
    preprocess_kwargs=preprocess_kwargs,
    kernel_kwargs={'sigma': np.array([.5, 1., 5.])},
    chunk_size=1000,
    n_permutations=5
)
```

```python
preds_h0 = cd.predict(X_h0)
print('Original test set sample')
print('Drift? {}'.format(labels[preds_h0['data']['is_drift']]))
print(preds_h0['data']['p_val'])
print('')

for x, c in zip(X_c, corruption):
    preds = cd.predict(x)
    print(f'Corruption type: {c}')
    print('Drift? {}'.format(labels[preds['data']['is_drift']]))
    print('Feature-wise p-values:')
    print(preds['data']['p_val'])
    print('')
```

    Original test set sample
    Drift? No!
    0.2

    Corruption type: gaussian_noise
    Drift? Yes!
    Feature-wise p-values:
    0.0

    Corruption type: motion_blur
    Drift? Yes!
    Feature-wise p-values:
    0.0

    Corruption type: brightness
    Drift? Yes!
    Feature-wise p-values:
    0.0

    Corruption type: pixelate
    Drift? Yes!
    Feature-wise p-values:
    0.0

A bandwidth can also be inferred from *X_ref* and *X* using the heuristic:

```python
cd = MMDDrift(
    p_val=.05,
    X_ref=X_ref,
    preprocess_kwargs=preprocess_kwargs,
    chunk_size=1000,
    n_permutations=5
)
```

```python
preds_h0 = cd.predict(X_h0)
print('Drift? {}'.format(labels[preds_h0['data']['is_drift']]))
print(preds_h0['data']['p_val'])
```

    Drift? No!
    0.2

```python
print('Inferred bandwidth: {:.4f}'.format(cd.permutation_test.keywords['sigma'].item()))
```

    Inferred bandwidth: 1.4132

### Label drift

We can also check what happens when we introduce class imbalances between the reference data *X_ref* and the tested data *X_imb*. The reference data will use $75$% of the instances of the first 5 classes and only $25$% of the last 5. The data used for drift testing then uses respectively $25$% and $75$% of the test instances for the first and last 5 classes.

```python
np.random.seed(0)
# get index for each class in the test set
num_classes = len(np.unique(y_test))
idx_by_class = [np.where(y_test == c)[0] for c in range(num_classes)]
# sample imbalanced data for different classes for X_ref and X_imb
perc_ref = .75
perc_ref_by_class = [perc_ref if c < 5 else 1 - perc_ref for c in range(num_classes)]
n_by_class = n_test // num_classes
X_ref = []
X_imb, y_imb = [], []
for _ in range(num_classes):
    idx_class_ref = np.random.choice(n_by_class, size=int(perc_ref_by_class[_] * n_by_class), replace=False)
    idx_ref = idx_by_class[_][idx_class_ref]
    idx_class_imb = np.delete(np.arange(n_by_class), idx_class_ref, axis=0)
    idx_imb = idx_by_class[_][idx_class_imb]
    assert idx_ref != idx_imb
    X_ref.append(X_test[idx_ref])
    X_imb.append(X_test[idx_imb])
    y_imb.append(y_test[idx_imb])
X_ref = np.concatenate(X_ref)
X_imb = np.concatenate(X_imb)
y_imb = np.concatenate(y_imb)
print(X_ref.shape, X_imb.shape, y_imb.shape)
```

    (5000, 32, 32, 3) (5000, 32, 32, 3) (5000,)

Update reference dataset for the detector and make predictions:

```python
cd.X_ref = cd.preprocess_fn(X_ref)
```

```python
preds_imb = cd.predict(X_imb)
print('Drift? {}'.format(labels[preds_imb['data']['is_drift']]))
print(preds_imb['data']['p_val'])
```

    Drift? Yes!
    0.0


```python
import torch
import torch.nn as nn
import torch.nn.functional as F
import torchvision
from torchvision import models, datasets, transforms
import torchvision.transforms as transforms
import pennylane_qiskit
from torch.autograd import Variable
from torch.distributions import Categorical
from torchvision.transforms import Resize
import torch.optim as optim
from torch.optim import Optimizer # 'required' was removed here
from ipywidgets import *
print('Doe we get CUDA Power? ',torch.cuda.is_available())
print("CUDA WHO? CUDA:",torch.cuda.current_device() )
print("CUDA who WHO? Cuda : ", torch.cuda.get_device_name(torch.cuda.current_device()))
import matplotlib.image as img
# Pennylane
import pennylane as qml
from pennylane import numpy as np
import scipy
import os
import pyzx as zx
import qiskit
import sys; sys.path.append('..')
import random, math, os
import pyzx as zx
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
from matplotlib import rc
import progress
import time
import csv
import pandas as pd
import pylatexenc
import qiskit

from qiskit import *
from qiskit import IBMQ
from qiskit.providers.aer import QasmSimulator
from qiskit.visualization import plot_histogram
%config InlineBackend.figure_format = "svg"
from qiskit.tools.parallel import parallel_map
import qiskit.tools.jupyter
%qiskit_job_watcher
%qiskit_progress_bar
%matplotlib inline
print('libraries Imported')
```

    Doe we get CUDA Power?  True
    CUDA WHO? CUDA: 0
    CUDA who WHO? Cuda :  NVIDIA GeForce 940MX

    Accordion(children=(VBox(layout=Layout(max_width='710px', min_width='710px')),), layout=Layout(max_height='500…

    <IPython.core.display.Javascript object>

    libraries Imported

# SUGGESTIONS

* On the right hand side of screen, the jupyter file is available through nb viewer. Click in it to see full rendering, It looks better that way!

## Install whichever library is not installed in your system by :
* conda install *missing library*
* pip install   *missing library*

Here we define the circuit parameters for a random circuit. You may design your own circuit.
* I designed my circuit in Pennylane > Converted it into qasm language by PyZX > then also used the QASM Code to develop PyZX circuits and in Qiskit.

Tutorial below:

```python

n_qubits = 4             # Number of qubits
q_depth = 1                 # Depth of the quantum circuit (number of variational layers)
q_delta = 0.01              # Initial spread of random quantum weights
start_time = time.time()    # Start of the computation timer
random_seed = 1
torch.backends.cudnn.enabled = True
torch.manual_seed(random_seed)
```

    <torch._C.Generator at 0x7f5a30193b10>

```python
def H_layer(nqubits):
    """Layer of single-qubit Hadamard gates.
    """
    for idx in range(nqubits):
        qml.Hadamard(wires=idx)

def RY_layer(w):
    """Layer of parametrized qubit rotations around the y axis.
    """
    for idx, element in enumerate(w):
        qml.RY(element, wires=idx)

def entangling_layer(nqubits):
    """Layer of CNOTs followed by another shifted layer of CNOT.
    """
    # In other words it should apply something like :
    # CNOT  CNOT  CNOT  CNOT...  CNOT
    #   CNOT  CNOT  CNOT...  CNOT
    for i in range(0, nqubits - 1, 2):  # Loop over even indices: i=0,2,...N-2
        qml.CNOT(wires=[i, i + 1])
    for i in range(1, nqubits - 1, 2):  # Loop over odd indices:  i=1,3,...N-3
        qml.CNOT(wires=[i, i + 1])
```

```python
dev = qml.device("qiskit.aer", wires=n_qubits,  shots=1000)
@qml.qnode(dev, interface="torch", diff_method="best")

def quantum_net(q_input_features, q_weights_flat):
    """
    The variational quantum circuit.
    """

    # Reshape weights
    q_weights = q_weights_flat.reshape(q_depth, n_qubits) #(9,4)

    # Start from state |+> , unbiased w.r.t. |0> and |1>
    H_layer(n_qubits)

    # Embed features in the quantum node
    RY_layer(q_input_features)

    # Sequence of trainable variational layers
    for k in range(q_depth):
        entangling_layer(n_qubits)
        RY_layer(q_weights[k])

    # Expectation values in the Z basis
    exp_vals = [qml.expval(qml.PauliZ(position)) for position in range(n_qubits)]
    return tuple(exp_vals)
```

```python
q_input_features = torch.tensor([np.pi*.1, np.pi*0.2, np.pi*0.3], requires_grad=True)
q_weights_flat = torch.tensor([np.pi*.1, np.pi*0.2, np.pi*0.3, np.pi*0.4 ], requires_grad=True)
result = quantum_net(q_input_features, q_weights_flat)
print(quantum_net.draw())
```

     0: ──H──RY(0.314)──╭C───RY(0.314)─────────────┤ ⟨Z⟩
     1: ──H──RY(0.628)──╰X──╭C──────────RY(0.628)──┤ ⟨Z⟩
     2: ──H──RY(0.942)──╭C──╰X──────────RY(0.942)──┤ ⟨Z⟩
     3: ──H─────────────╰X───RY(1.26)──────────────┤ ⟨Z⟩

Pennylane offers to convert its circuits into QASM which is the Open Quantum Assembly Language.
It may be helpful when you are trying to work with various Quantum Libraries.

```python
s=dev._circuit.qasm(formatted=True)
```

    OPENQASM 2.0;
    include "qelib1.inc";
    qreg q[4];
    creg c[4];
    h q[0];
    h q[1];
    h q[2];
    h q[3];
    ry(pi/10) q[0];
    ry(pi/5) q[1];
    ry(3*pi/10) q[2];
    cx q[0],q[1];
    cx q[2],q[3];
    cx q[1],q[2];
    ry(pi/10) q[0];
    ry(pi/5) q[1];
    ry(3*pi/10) q[2];
    ry(2*pi/5) q[3];
    measure q[0] -> c[0];
    measure q[1] -> c[1];
    measure q[2] -> c[2];
    measure q[3] -> c[3];

PyZX offers Rotation Function for RX, RZ , but not for RY. For example, in the code example a below, since I couldn't find RY, I tested the above required QASM code with RZ Function.

```python
a = zx.qasm("""
            qreg q[4];
            creg c[4];
            h q[0];
            h q[1];
            h q[2];
            h q[3];
            rz(pi*1/10) q[0];
            rz(pi*1/5) q[1];
            rz(3*pi/10) q[2];
            cx q[0],q[1];
            cx q[2],q[3];
            cx q[1],q[2];
            rz(pi*1/10) q[0];
            rz(pi*1/5) q[1];
            rz(3*pi/10) q[2];
            rz(2*pi/5) q[3];
            measure q[0] -> c[0];
            measure q[1] -> c[1];
            measure q[2] -> c[2];
            measure q[3] -> c[3];""" )
zx.draw_d3(a)

```

The Nodes of the PyZX circuit are movable if You drag them by mouse. Come on try it!

A Y rotation of alpha should be :
   RY(alpha) =  RZ(-pi/2)*RX(alpha)*RZ(pi/2)   -Aleks Kissinger(The Creater)

```python
qc = zx.qasm("""
            qreg q[4];
            creg c[4];

            h q[0];
            h q[1];
            h q[2];
            h q[3];

            rz(-pi/2) q[0];
            h q[0];
            rx(pi*1/10) q[0];
            h q[0];
            rz(pi/2) q[0];

            rz(-pi/2) q[1];
            h q[1];
            rx(pi*1/10) q[1];
            h q[1];
            rz(pi/2) q[1];

            rz(-pi/2) q[2];
            h q[2];
            rx(pi*1/10) q[2];
            h q[2];
            rz(pi/2) q[2];

            cx q[0],q[1];
            cx q[2],q[3];
            cx q[1],q[2];

            rz(-pi/2) q[0];
            h q[0];
            rx(pi*1/10) q[0];
            h q[0];
            rz(pi/2) q[0];

            rz(-pi/2) q[1];
            h q[1];
            rx(pi*1/10) q[1];
            h q[1];
            rz(pi/2) q[1];

            rz(-pi/2) q[2];
            h q[2];
            rx(pi*1/10) q[2];
            h q[2];
            rz(pi/2) q[2];

            rz(-pi/2) q[3];
            h q[3];
            rx(pi*1/10) q[3];
            h q[3];
            rz(pi/2) q[3];

            measure q[0] -> c[0];
            measure q[1] -> c[1];
            measure q[2] -> c[2];
            measure q[3] -> c[3];""" )
zx.draw_d3(qc)

```

So, the required circuit should look like this in ZX Calculus. Why It looks like this? You need to study ZX Calculus. Its application is in Error Correction, Reducting Cliffords.

```python
zx.draw_matplotlib(qc,labels=True,h_edge_draw='box', figsize=(8, 5))
```

If You also want to picturise the position of Hadamard gates, try the code above.

You May also list the gate information of your circuit

```python
qc.gates
```

    [HAD(0),
     HAD(1),
     HAD(2),
     HAD(3),
     ZPhase(0,phase=-1/2),
     HAD(0),
     XPhase(0,phase=1/10),
     HAD(0),
     ZPhase(0,phase=1/2),
     ZPhase(1,phase=-1/2),
     HAD(1),
     XPhase(1,phase=1/10),
     HAD(1),
     ZPhase(1,phase=1/2),
     ZPhase(2,phase=-1/2),
     HAD(2),
     XPhase(2,phase=1/10),
     HAD(2),
     ZPhase(2,phase=1/2),
     CNOT(0,1),
     CNOT(2,3),
     CNOT(1,2),
     ZPhase(0,phase=-1/2),
     HAD(0),
     XPhase(0,phase=1/10),
     HAD(0),
     ZPhase(0,phase=1/2),
     ZPhase(1,phase=-1/2),
     HAD(1),
     XPhase(1,phase=1/10),
     HAD(1),
     ZPhase(1,phase=1/2),
     ZPhase(2,phase=-1/2),
     HAD(2),
     XPhase(2,phase=1/10),
     HAD(2),
     ZPhase(2,phase=1/2),
     ZPhase(3,phase=-1/2),
     HAD(3),
     XPhase(3,phase=1/10),
     HAD(3),
     ZPhase(3,phase=1/2)]

```python
l= zx.print_matrix(qc) #I dont know why its so zoomed out
l
```

    Label(value='\\begin{equation}\n2^{-2}\n\\begin{pmatrix}\n1 & 1 & 1 & 1 & 1 & 1 & 1 & 1 & 1 & 1 & 1 & 1 & 1 & …

This is how the whole QuantumCircuit can be prepresented as mathematical matrix

```python
k = qc.to_matrix(preserve_scalar=True)
k
```

    array([[ 0.25      -0.j        ,  0.25      -0.j        ,
             0.25      -0.j        ,  0.25      -0.j        ,
             0.25      -0.j        ,  0.25      -0.j        ,
             0.25      -0.j        ,  0.25      -0.j        ,
             0.25      -0.j        ,  0.25      -0.j        ,
             0.25      -0.j        ,  0.25      -0.j        ,
             0.25      -0.j        ,  0.25      -0.j        ,
             0.25      -0.j        ,  0.25      -0.j        ],
           [ 0.23776413+0.07725425j, -0.23776413-0.07725425j,
             0.23776413+0.07725425j, -0.23776413-0.07725425j,
             0.23776413+0.07725425j, -0.23776413-0.07725425j,
             0.23776413+0.07725425j, -0.23776413-0.07725425j,
             0.23776413+0.07725425j, -0.23776413-0.07725425j,
             0.23776413+0.07725425j, -0.23776413-0.07725425j,
             0.23776413+0.07725425j, -0.23776413-0.07725425j,
             0.23776413+0.07725425j, -0.23776413-0.07725425j],
           [ 0.20225425+0.14694631j, -0.20225425-0.14694631j,
            -0.20225425-0.14694631j,  0.20225425+0.14694631j,
             0.20225425+0.14694631j, -0.20225425-0.14694631j,
            -0.20225425-0.14694631j,  0.20225425+0.14694631j,
             0.20225425+0.14694631j, -0.20225425-0.14694631j,
            -0.20225425-0.14694631j,  0.20225425+0.14694631j,
             0.20225425+0.14694631j, -0.20225425-0.14694631j,
            -0.20225425-0.14694631j,  0.20225425+0.14694631j],
           [ 0.14694631+0.20225425j,  0.14694631+0.20225425j,
            -0.14694631-0.20225425j, -0.14694631-0.20225425j,
             0.14694631+0.20225425j,  0.14694631+0.20225425j,
            -0.14694631-0.20225425j, -0.14694631-0.20225425j,
             0.14694631+0.20225425j,  0.14694631+0.20225425j,
            -0.14694631-0.20225425j, -0.14694631-0.20225425j,
             0.14694631+0.20225425j,  0.14694631+0.20225425j,
            -0.14694631-0.20225425j, -0.14694631-0.20225425j],
           [ 0.14694631+0.20225425j, -0.14694631-0.20225425j,
            -0.14694631-0.20225425j,  0.14694631+0.20225425j,
            -0.14694631-0.20225425j,  0.14694631+0.20225425j,
             0.14694631+0.20225425j, -0.14694631-0.20225425j,
             0.14694631+0.20225425j, -0.14694631-0.20225425j,
            -0.14694631-0.20225425j,  0.14694631+0.20225425j,
            -0.14694631-0.20225425j,  0.14694631+0.20225425j,
             0.14694631+0.20225425j, -0.14694631-0.20225425j],
           [ 0.07725425+0.23776413j,  0.07725425+0.23776413j,
            -0.07725425-0.23776413j, -0.07725425-0.23776413j,
            -0.07725425-0.23776413j, -0.07725425-0.23776413j,
             0.07725425+0.23776413j,  0.07725425+0.23776413j,
             0.07725425+0.23776413j,  0.07725425+0.23776413j,
            -0.07725425-0.23776413j, -0.07725425-0.23776413j,
            -0.07725425-0.23776413j, -0.07725425-0.23776413j,
             0.07725425+0.23776413j,  0.07725425+0.23776413j],
           [ 0.14694631+0.20225425j,  0.14694631+0.20225425j,
             0.14694631+0.20225425j,  0.14694631+0.20225425j,
            -0.14694631-0.20225425j, -0.14694631-0.20225425j,
            -0.14694631-0.20225425j, -0.14694631-0.20225425j,
             0.14694631+0.20225425j,  0.14694631+0.20225425j,
             0.14694631+0.20225425j,  0.14694631+0.20225425j,
            -0.14694631-0.20225425j, -0.14694631-0.20225425j,
            -0.14694631-0.20225425j, -0.14694631-0.20225425j],
           [ 0.07725425+0.23776413j, -0.07725425-0.23776413j,
             0.07725425+0.23776413j, -0.07725425-0.23776413j,
            -0.07725425-0.23776413j,  0.07725425+0.23776413j,
            -0.07725425-0.23776413j,  0.07725425+0.23776413j,
             0.07725425+0.23776413j, -0.07725425-0.23776413j,
             0.07725425+0.23776413j, -0.07725425-0.23776413j,
            -0.07725425-0.23776413j,  0.07725425+0.23776413j,
            -0.07725425-0.23776413j,  0.07725425+0.23776413j],
           [ 0.14694631+0.20225425j,  0.14694631+0.20225425j,
             0.14694631+0.20225425j,  0.14694631+0.20225425j,
            -0.14694631-0.20225425j, -0.14694631-0.20225425j,
            -0.14694631-0.20225425j, -0.14694631-0.20225425j,
            -0.14694631-0.20225425j, -0.14694631-0.20225425j,
            -0.14694631-0.20225425j, -0.14694631-0.20225425j,
             0.14694631+0.20225425j,  0.14694631+0.20225425j,
             0.14694631+0.20225425j,  0.14694631+0.20225425j],
           [ 0.07725425+0.23776413j, -0.07725425-0.23776413j,
             0.07725425+0.23776413j, -0.07725425-0.23776413j,
            -0.07725425-0.23776413j,  0.07725425+0.23776413j,
            -0.07725425-0.23776413j,  0.07725425+0.23776413j,
            -0.07725425-0.23776413j,  0.07725425+0.23776413j,
            -0.07725425-0.23776413j,  0.07725425+0.23776413j,
             0.07725425+0.23776413j, -0.07725425-0.23776413j,
             0.07725425+0.23776413j, -0.07725425-0.23776413j],
           [ 0.        +0.25j      , -0.        -0.25j      ,
            -0.        -0.25j      ,  0.        +0.25j      ,
            -0.        -0.25j      ,  0.        +0.25j      ,
             0.        +0.25j      , -0.        -0.25j      ,
            -0.        -0.25j      ,  0.        +0.25j      ,
             0.        +0.25j      , -0.        -0.25j      ,
             0.        +0.25j      , -0.        -0.25j      ,
            -0.        -0.25j      ,  0.        +0.25j      ],
           [-0.07725425+0.23776413j, -0.07725425+0.23776413j,
             0.07725425-0.23776413j,  0.07725425-0.23776413j,
             0.07725425-0.23776413j,  0.07725425-0.23776413j,
            -0.07725425+0.23776413j, -0.07725425+0.23776413j,
             0.07725425-0.23776413j,  0.07725425-0.23776413j,
            -0.07725425+0.23776413j, -0.07725425+0.23776413j,
            -0.07725425+0.23776413j, -0.07725425+0.23776413j,
             0.07725425-0.23776413j,  0.07725425-0.23776413j],
           [ 0.07725425+0.23776413j, -0.07725425-0.23776413j,
            -0.07725425-0.23776413j,  0.07725425+0.23776413j,
             0.07725425+0.23776413j, -0.07725425-0.23776413j,
            -0.07725425-0.23776413j,  0.07725425+0.23776413j,
            -0.07725425-0.23776413j,  0.07725425+0.23776413j,
             0.07725425+0.23776413j, -0.07725425-0.23776413j,
            -0.07725425-0.23776413j,  0.07725425+0.23776413j,
             0.07725425+0.23776413j, -0.07725425-0.23776413j],
           [ 0.        +0.25j      ,  0.        +0.25j      ,
            -0.        -0.25j      , -0.        -0.25j      ,
             0.        +0.25j      ,  0.        +0.25j      ,
            -0.        -0.25j      , -0.        -0.25j      ,
            -0.        -0.25j      , -0.        -0.25j      ,
             0.        +0.25j      ,  0.        +0.25j      ,
            -0.        -0.25j      , -0.        -0.25j      ,
             0.        +0.25j      ,  0.        +0.25j      ],
           [ 0.07725425+0.23776413j,  0.07725425+0.23776413j,
             0.07725425+0.23776413j,  0.07725425+0.23776413j,
             0.07725425+0.23776413j,  0.07725425+0.23776413j,
             0.07725425+0.23776413j,  0.07725425+0.23776413j,
            -0.07725425-0.23776413j, -0.07725425-0.23776413j,
            -0.07725425-0.23776413j, -0.07725425-0.23776413j,
            -0.07725425-0.23776413j, -0.07725425-0.23776413j,
            -0.07725425-0.23776413j, -0.07725425-0.23776413j],
           [ 0.        +0.25j      , -0.        -0.25j      ,
             0.        +0.25j      , -0.        -0.25j      ,
             0.        +0.25j      , -0.        -0.25j      ,
             0.        +0.25j      , -0.        -0.25j      ,
            -0.        -0.25j      ,  0.        +0.25j      ,
            -0.        -0.25j      ,  0.        +0.25j      ,
            -0.        -0.25j      ,  0.        +0.25j      ,
            -0.        -0.25j      ,  0.        +0.25j      ]])

```python
g = qc.to_graph()
g

```

    Graph(53 vertices, 52 edges)

```python
g.normalize() # Reposition nodes horizontally to look nicer
zx.draw(g)
```

Following two code examples Show the count of Edges and vertex of Circuit connections.

```python
print(qc)
print("All edges: ", list(g.edges()))
print("\nThe neighbors of a particular vertex: ", list(g.neighbors(14)))
```

    Circuit(4 qubits, 42 gates)
    All edges:  [(0, 4), (1, 5), (2, 6), (3, 7), (4, 8), (5, 13), (6, 18), (7, 25), (8, 9), (9, 10), (10, 11), (11, 12), (12, 24), (13, 14), (14, 15), (15, 16), (16, 17), (17, 23), (18, 19), (19, 20), (20, 21), (21, 22), (22, 26), (23, 24), (23, 28), (24, 29), (25, 26), (25, 44), (26, 27), (27, 28), (27, 39), (28, 34), (29, 30), (30, 31), (31, 32), (32, 33), (33, 49), (34, 35), (35, 36), (36, 37), (37, 38), (38, 50), (39, 40), (40, 41), (41, 42), (42, 43), (43, 51), (44, 45), (45, 46), (46, 47), (47, 48), (48, 52)]

    The neighbors of a particular vertex:  [13, 15]

```python
print("vertex count", len(list(g.vertices())))
#print("Combinations of vertices: ", list(g.vertices())
print("edges count", len(list(g.edges())))
#print("Combinations of edges: ", list(g.edges())

```

    vertex count 53
    edges count 52

```python
from ipywidgets import widgets
from IPython.display import display, Markdown
qubit_amount =4
from pyzx.simplify import clifford_iter

graph = qc.to_basic_gates().to_graph().copy()
graphs = [zx.draw_matplotlib(graph ,labels=False,h_edge_draw='box')]
names = ["start"]
for gs, n in clifford_iter(graph):
    graphs.append(zx.draw_matplotlib(gs))
    names.append(n)
zx.drawing.pack_circuit_nf(gs,'grg')
graphs.append(zx.draw_matplotlib(gs))
names.append("reposition")

def plotter(rewrite):
    display(Markdown("Rewrite step: " + names[rewrite]))
    display(graphs[rewrite])

w = widgets.interactive(plotter, rewrite=(0,len(graphs)-1))
slider = w.children[0]
slider.layout.width = "{!s}px".format(min(800,50*len(graphs)))
output = w.children[-1]
output.layout.height = "{!s}px".format(200+qubit_amount*20)
slider.value = 0
w
```

    interactive(children=(IntSlider(value=0, description='rewrite', layout=Layout(width='700px'), max=13), Output(…

The above example may help you visualize step by step reduction of Quantum Circuit, if reduced through ZX calculus. The steps are called Iterations. You may slide the slider to see.

Following code is the 3 teration simplification of the original circuit.

```python
g = qc.to_graph()
g
zx.simplify.spider_simp(g)
zx.draw_d3(g)
```

    spider_simp: 11. 2. 1.  3 iterations

But what if you want to reduce the Circuit completely? Do as follows:

```python
g = qc.to_graph()
g

```

    Graph(53 vertices, 52 edges)

```python
zx.full_reduce(g)
zx.draw_d3(g)
```

```python
g = qc.to_graph()
g

```

    Graph(53 vertices, 52 edges)

## In the Quantum circuit, every Qubit is mapped to another Qubit. Proof? Its given follows:

```python
g1 = g.copy()
# The numbers in the paranthesis are the qubit identities 0, 1 , 2, 3 for 4 qubits of our Circuit.
g1.map_qubits([
    (0,0),(1,0),(2,0),(3,0),
    (0,1),(1,1),(2,1),(3,1),
    (0,2),(1,2),(2,2),(3,2),
    (0,3),(1,3),(2,3),(3,3),
    ])
zx.draw_d3(g1)
```

```python
e=zx.hsimplify.hpivot_simp(g)
```

    spider_simp: 11. 2. 1.  3 iterations
    id_simp: 8.  1 iterations
    spider_simp: 7. 6. 1.  3 iterations

Just another representation of the above Normalized circuit with representaion of Hadamard gates in Yellow boxes.

```python
zx.draw_matplotlib(g1,labels=True,h_edge_draw='box')
```

You May obtain the Gate Stats as follows. It might be important to understand your circuits's components

```python
print(qc.to_basic_gates().stats())

```

    Circuit  on 4 qubits with 42 gates.
            7 is the T-count
            35 Cliffords among which
            3 2-qubit gates (3 CNOT, 0 other) and
            18 Hadamard gates.

```python
print("Tcount: ",zx.tcount(g)) #Returns the amount of nodes in g that have a non-Clifford phase.
%timeit zx.simulate.calculate_path_sum(g)
#zx.spider_simp(g3,quiet=True)
#zx.d3.draw(g3)
```

    Tcount:  6
    866 µs ± 50.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

We conclude the tutorial by finally coding the same Circuit in Qiskit:

```python
from qiskit import QuantumRegister, ClassicalRegister, QuantumCircuit
from numpy import pi

qreg_q = QuantumRegister(4, 'q')
creg_c = ClassicalRegister(4, 'c')
circuit = QuantumCircuit(qreg_q, creg_c)

circuit.h(qreg_q[0])
circuit.h(qreg_q[1])
circuit.h(qreg_q[2])
circuit.h(qreg_q[3])
circuit.ry(pi/10, qreg_q[0])
circuit.ry(pi/5, qreg_q[1])
circuit.ry(3*pi/10, qreg_q[2])
circuit.cx(qreg_q[0], qreg_q[1])
circuit.cx(qreg_q[2], qreg_q[3])
circuit.cx(qreg_q[1], qreg_q[2])
circuit.ry(pi/10, qreg_q[0])
circuit.ry(pi/5, qreg_q[1])
circuit.ry(3*pi/10, qreg_q[2])
circuit.ry(2*pi/5, qreg_q[3])
circuit.measure(qreg_q[0], creg_c[0])
circuit.measure(qreg_q[1], creg_c[1])
circuit.measure(qreg_q[2], creg_c[2])
circuit.measure(qreg_q[3], creg_c[3])
circuit.draw('mpl', cregbundle=(False) )
```

```python
#overview of all available backends
from qiskit import IBMQ
total_qubits = 4
IBMQ.save_account('Insert Your own Token here', overwrite= True)
IBMQ.load_account()
IBMQ.providers()
%matplotlib inline

provider= IBMQ.get_provider(hub='ibm-q')
provider.backends()
backend= provider.backends(filters = lambda  x: x.configuration().n_qubits >= total_qubits and not x.configuration().simulator and x.status().operational == True)

print('least busy backend: ', backend)
```

    least busy backend:  [<IBMQBackend('ibmq_santiago') from IBMQ(hub='ibm-q', group='open', project='main')>, <IBMQBackend('ibmq_bogota') from IBMQ(hub='ibm-q', group='open', project='main')>, <IBMQBackend('ibmq_lima') from IBMQ(hub='ibm-q', group='open', project='main')>, <IBMQBackend('ibmq_belem') from IBMQ(hub='ibm-q', group='open', project='main')>, <IBMQBackend('ibmq_quito') from IBMQ(hub='ibm-q', group='open', project='main')>, <IBMQBackend('ibmq_manila') from IBMQ(hub='ibm-q', group='open', project='main')>]

```python
backend = provider.get_backend('ibmq_belem')
result = execute(circuit, backend= backend, shots = 1024).result()
counts = result.get_counts(circuit)
print(counts)
plot_histogram(counts, figsize= (8,5))

```

    {'0000': 1, '0001': 21, '0010': 12, '0011': 11, '0100': 5, '0101': 19, '0110': 4, '0111': 16, '1000': 19, '1001': 79, '1010': 93, '1011': 93, '1100': 17, '1101': 320, '1110': 118, '1111': 196}

```python
circuit.decompose()
```

    <qiskit.circuit.quantumcircuit.QuantumCircuit at 0x7f59847debb0>

```python

```


>>> Work in Progress (Following are the lecture notes of Prof Andrew Ng - CS229 - Stanford. This is my interpretation of his excellent teaching and I take full responsibility of any misinterpretation/misinformation provided herein.)

## Lecture 18

#### Outline
- Continuous State MDPs
  - Discretization
  - Models/Simulation
  - Fitted value iteration

### Continuous state space
- In the last lecture we learnt about finite state models.
- The state space representation of car
  - kinematic model of car
    - x - units in x direction
    - y - units in y direction
    - $\theta$ - orientation wrt some direction say North
  - dynamic model of car
    - $\dot{x}$ - linear velocity in x direction
    - $\dot{y}$ - linear velocity in y direction
    - $\dot{\theta}$ - angular velocity
- it depends on how we want to model the car, if we are interested in modeling race car, we need to include the temperature as well, we will need to model the 4 car tires separately

- the state space representation of helicopter will have 3-dimensions instead of 2
- inverted pendulum
- the dimensions for the above problem statement will be $\mathbb R^{6}(\text{car}), \mathbb R^{12}(\text{helicopter}), \mathbb R^{4}(\text{pendulum})$

- one easy way would be discretize continuous state space and solve it as discrete state space problem
- its fine to solve low dimensional state space problem using discretization, but for higher dimensional problem we need to use fitted value iteration
- disadvantages of discretization:
  - 1) Naive representation for $V^{*}$ and $\pi^{*}$
    - in the house price prediction problem, fit a constant function
    - the buckets will represent the price of a house as a function of the size
    - same applies to state-space problem, where we approximate the value function as a staircase function, as a function of set of stairs
    - this can be applied if there is lot of data and very few input features, but it does not allow to fit a smoother function
    - its not a very good representation

  - 2) Curse of dimensionality
    - if state space $s \in \mathbb R^{n}$, and if we discretize each dimension into k values, then we get $k^{n}$ discrete states which grows exponentially
    - this is not a good representation in large dimensions say more than 8-10
    - way of reducing/taking control of exponential blow up would be to use _unequal grid_ for state space
      - define finer or more number of state for more sensitive state-space and lesser number of state for less sensitive state-space

$\tiny{\text{YouTube-Stanford-CS229-Andrew Ng}}$

### Alternative approach
- approximate $V^{*}$ directly without using discretization directly
  - in linear regression problem
    - approximate the value function where we learn V as a function of s
    - approximate $y \approx \theta^{T}x$ or $\theta^{T} \phi(x)$
      - where $\phi(x)$ is the representation of features of x
      - in the housing price problem, $\phi(x)$ is vector of x's the housing price $\{x_{1}, x_{2}, x_{1}^{2}, x_{1}x_{2}\}$, either as a function of real feature or complex set of features of house
  - in fitted value iteration problem
    - is a model where we will approximate $V^{*}(s)$, as a linear function of features of the state, with lot of modifications to approximate the value function
    - approximate $V^{*}(s) \approx \theta^{T} \phi(s)$
    - once we have a good approximation of value function, we can compute the optimal action for every state

- fitted value iteration problem works best with a model with simulator of the MDP
- what does that mean?
- Model (simulator) of a robot is a function that takes an input state s, an action a and outputs state s', drawn from the state space probability $P_{sa}$
- for most of the MDP problems, the state space will be very high dimensional and the action space is much lower dimensional than the state space
  - for car state s will be 6 dimensional, and action a will be 2 dimensional - steering and wheel
  - for helicopter state s will be 12 dimensional, and action a will be 4 dimensional - left and right sticks with 2 controls each
  - for pendulum state space s will be 4 dimensional and action a will be 1 - left or right
- later we will see we will not need to discretize action space

$\tiny{\text{YouTube-Stanford-CS229-Andrew Ng}}$

### How do we make a model
- One way - use physics simulator package
  - in pendulum case,
    - state space is 4 dim, use physics simulator package, plug in the dimension details, mass and other details, and the simulator will spit out how the state evolves from one time step to another time step
- Second way - learn model from data
  - more generally used
  - lets say we have a helicopter and we want to build an autonomous controller for it
    - start helicopter in some state $s_{0}$
    - a human pilot will command the helicopter with some action $a_{0}$
    - say in few seconds, the helicopter reaches a new position and orientation as $s_{1}$
    - human pilot will keep on taking actions say every 10 times a second and we record its action and state. repeat for t seconds
    - record multiple trajectories say m times
    - apply supervised learning to predict $s_{t+1}$($s^{'}$) as function of state $s_{t}$($s$) and action $a_{t}$($a$)
    - in linear regression version
      - this model is fine for helicopter moving at slow speed
      > $s_{t+1} = As_{t} + Ba_{t}$
        - we can also plug in non-linear form of model depending on your choice of features of A as $\phi(s)$ and B as $\phi(a)$
      - to fit the parameters A and B, that minimizes the squared difference of left and right hand side
      > $\min\limits_{A,B}\sum\limits_{i=1}^{m}\sum\limits_{t=0}^{T}\|s_{t+1}^{(i)}-(As_{t}^{(i)} + Ba_{t}^{(i)}) \|^{2}$
      - we can either model it as
        - deterministic
          > $s_{t+1} = As_{t} + Ba_{t}$
        - stochastic
          > $s_{t+1} = As_{t} + Ba_{t} + \epsilon_{t}$
          - where $\epsilon_{t} \sim N(0, \sigma^{2}I)$
          - in this case, we will be sampling $\epsilon$ from a Gaussian vector and add it to the prediction of linear model
          - if we simulate the helicopter flying around, the simulator will generate random noise that adds or subtracts to the state space of helicopter as if there were little wind gusts blowing the helicopter around
      - for high speed helicopters, this model will not work
  - this approach is called __model based reinforcement learning__
    - to build a model of robot, we train the reinforcement learning algorithm in the simulator
    - then take the policy we learn in simulation
    - apply back on the real robot
    - this field is evolving quickly in robotics
  - alternative is __model-free reinforcement learning__,
    - this is generally not much used as we cannot afford to run actual helicopter and learn from it
    - instead model-free rl is generally used in video games

- if we design a deterministic simulators for robots and use it in a real robot, the odds of its success is very low
- instead if we design a stochastic simulators and add noise term into it, the chances of its success increases for robotic control problems

### How do you approximate the value function
- in order to apply, fitted value iteration
- first step is to choose features $\phi(s)$ of the state s
- approximate V(s)
> $V(s) = \theta^{T}\phi(s)$
- determine the features of $\phi(s)$ as nonlinear features that we suppose might be representative of value/how well the robot is doing
- value of state is the expected payoff/discounted rewards from that state
- the value function captures if the robot starts off in that state, how well would it do

### Fitted value algorithm
- Value iteration
> $\begin{equation}\\
\begin{aligned}\\
V(s) &= R(s) + \gamma\max\limits_{a}\sum\limits_{s'\in S}P_{sa}(s')V(s')\\
&= R(s) + \gamma\max\limits_{a}E_{s'\sim P_{sa}}[V(s')]\\
\end{aligned}\\
\end{equation}\\
$

- Fitted value iteration
  - Choose set of states randomly $\{ s^{(1)}, s^{(2)}, ..., s^{(m)}\} \in S$
  - Initialize $\theta := 0$
    - in general
      - we map $x \rightarrow y$,
      - as we have a finite set of examples that we see as values of x in the training set for predicting housing prices
    - similarly
      - here we map $s \rightarrow V(s)$
      - we will use a certain set of states, and use that finite set of examples s in a linear regression to fit V(s)
  - that is the use of sample above
  - Repeat { - _Loop1_
    - for i = 1,2,..,m {  - _Loop2_
      - for each action $a \in A$ { - _Loop3_
        - sample $\{ s'^{(1)}, s'^{(2)}, ..., s^{(k)}\} \sim P_{s^{(i)},a}$ - _Step4_
        - set $q(a) = \frac{1}{k}\sum\limits_{j=1}^{k}[R(s^{(i)}) + \gamma V(s'_{j})]$ - which is the estimate of expectation - _Step5_
      - }
      - set $y^{(i)} = \max\limits_{a}q(a)$ - _Step6_
    - }
    - $\theta := \arg\min\limits_{\theta}\frac{1}{2}\sum\limits_{i=1}^{m}(\theta^{T}\phi(s^{(i)}) - y^{(i)})^{2}$ - _Step7_
  - }

  - In _Loop2_, m represents how many times we fly the helicopter in order to build a model
  - in _Step4_, we took k samples from $V(s'_{j})$ distribution, average V(s') in _Step5_
    - we sampled k times because we use stochastic simulator here
    - if we use a deterministic simulator instead, for a given state and action, it will always result in same state $s^{(i)},a \rightarrow s'$
    - for deterministic simulator, we set k=1
  - in _Step5_, we get $V(s'_{j})$ from using the parameters of $\theta^{T}\phi(s_{j}')$ from the last iteration of fitted value iteration

> $\begin{equation}\\
\begin{aligned}\\
V(s) &= R(s) + \gamma\max\limits_{a}\sum\limits_{s'\in S}P_{sa}(s')V(s')\\
&= R(s) + \gamma\max\limits_{a}E_{s'\sim P_{sa}}[V(s')]\\
&= \max\limits_{a}E_{s'\sim P_{sa}}[R(s) + \gamma V(s')]\\
&= \max\limits_{a}E_{s'\sim P_{sa}}[q(a)]\\
\end{aligned}\\
\end{equation}\\
$

- In _Step6_, we set $y^{(i)} = \max\limits_{a}q(a)$ - is the estimate at right-hand side of value iteration
  - in the original value iteration
    - $V(s^{(i)}) := y^{(i)}$
  - in fitted value iteration
    - we want $V(s^{(i)}) \approx y^{(i)}$
    - i.e., $\theta^{T}\phi(s^{(i)}) \approx y^{(i)}$

- In _Step7_: run linear regression to choose the parameters $\theta$ that minimizes the squared error
> $\theta := \arg\min\limits_{\theta}\frac{1}{2}\sum\limits_{i=1}^{m}(\theta^{T}\phi(s^{(i)}) - y^{(i)})^{2}$

### Some practical aspects
- After we learn all the parameters, fitted VI gives approximation to $V^{*}$, which implicitly defines $\pi^{*}$
> $\pi^{*}(s) = \arg\max\limits_{a}E_{s'\in P_{s,a}}[V^{*}(s')]$
- in real world helicopter fly, we will have to choose control actions at 10Hertz, i.e., 10 times a second we are given the state, and we have to choose an action
- same would be with a self driving car, where we choose a new action 10 times a second
- but how do we compute this expectation and maximize this 10 times per second?
  - we used samples $\{ s'^{(1)}, s'^{(2)}, ..., s^{(k)}\} \sim P_{s^{(i)},a}$ to approximate expectation
- Tricks
  - simulator is of the form
    - $s_{t+1} = f(s_{t}, a_{t}) + \epsilon_{t}$, i.e., $(As_{t} + Ba_{t} + \epsilon_{t})$ - previous state and action plus some noise
  - for deployment/runtime
    - set $\epsilon_{t}=0$ and $k=1$
    - in simulator we need to have noise and randomness
    - but for real practical purposes, we can use discretized state and action V