## Interactive Variogram Calculation and Modeling Demonstration

### Michael Pyrcz, Associate Professor, University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

### The Interactive Workflow

Here's an interactive workflow for calculating directional experimental variograms in 2D.

* setting the variogram calculation parameters for identifying spatial data pairs

This approach is essential for quantifying spatial continuity with sparsely sampled, irregular spatial data.

I have more comprehensive workflows for variogram calculation:

* [Experimental Variogram Calculation in Python with GeostatsPy](https://github.com/GeostatsGuy/PythonNumericalDemos/blob/master/GeostatsPy_variogram_calculation.ipynb)

* [Determination of Major and Minor Spatial Continuity Directions in Python with GeostatsPy](https://github.com/GeostatsGuy/PythonNumericalDemos/blob/master/GeostatsPy_spatial_continuity_directions.ipynb)

#### Spatial Continuity

**Spatial Continuity** is the correlation between values over distance.

* No spatial continuity – no correlation between values over distance, random values at each location in space regardless of separation distance.

* Homogenous phenomenon have perfect spatial continuity, since all values as the same (or very similar) they are correlated.

We need a statistic to quantify spatial continuity! A convenient method is the Semivariogram.

#### The Semivariogram

Function of difference over distance.

* The expected (average) squared difference between values separated by a lag distance vector (distance and direction), $h$:

\begin{equation}
\gamma(\bf{h}) = \frac{1}{2 N(\bf{h})} \sum^{N(\bf{h})}_{\alpha=1} (z(\bf{u}_\alpha) - z(\bf{u}_\alpha + \bf{h}))^2
\end{equation}

where $z(\bf{u}_\alpha)$ and $z(\bf{u}_\alpha + \bf{h})$ are the spatial sample values at tail and head locations of the lag vector respectively.

* Calculated over a suite of lag distances to obtain a continuous function.

* the $\frac{1}{2}$ term converts a variogram into a semivariogram, but in practice the term variogram is used instead of semivariogram.
* We prefer the semivariogram because it relates directly to the covariance function, $C_x(\bf{h})$ and univariate variance, $\sigma^2_x$:

\begin{equation}
C_x(\bf{h}) = \sigma^2_x - \gamma(\bf{h})
\end{equation}

Note the correlogram is related to the covariance function as:

\begin{equation}
\rho_x(\bf{h}) = \frac{C_x(\bf{h})}{\sigma^2_x}
\end{equation}

The correlogram provides of function of the $\bf{h}-\bf{h}$ scatter plot correlation vs. lag offset $\bf{h}$.

\begin{equation}
-1.0 \le \rho_x(\bf{h}) \le 1.0
\end{equation}

#### Variogram Observations

The following are common observations for variograms that should assist with their practical use.

##### Observation \#1 - As distance increases, variability increase (in general).

This is common since in general, over greater distance offsets, there is often more difference between the head and tail samples.

In some cases, such as with spatial cyclicity of the hole effect variogram model the variogram may have negative slope over somelag distance intervals

Negative slopes at lag distances greater than half the data extent are often caused by too few pairs for a reliable variogram calculation

##### Observation \#2 - Calculated with over all possible pairs separated by lag vector, $\bf{𝐡}$.

We scan through the entire data set, searching for all possible pair combinations with all other data.  We then calculate the variogram as one half the expectation of squared difference between all pairs.

More pairs results in a more reliable measure.

##### Observation \#3 - Need to plot the sill to know the degree of correlation.

**Sill** is the variance, $\sigma^2_x$

Given stationarity of the variance, $\sigma^2_x$, and variogram $\gamma(\bf{h})$:

we can define the covariance function:

\begin{equation}
C_x(\bf{h}) = \sigma^2_x - \gamma(\bf{h})
\end{equation}

The covariance measure is a measure of similarity over distance (the mirror image of the variogram as shown by the equation above).

Given a standardized distribution $\sigma^2_x = 1.0$, the covariance, $C_x(\bf{h})$, is equal to the correlogram, $\rho_x(\bf{h})$:

\begin{equation}
\rho_x(\bf{h}) = \sigma^2_x - \gamma(\bf{h})
\end{equation}

##### Observation \#4 - The lag distance at which the variogram reaches the sill is know as the range.

At the range, knowing the data value at the tail location provides no information about a value at the head location of the lag distance vector.

##### Observation \#5 - The nugget effect, a discontinuity at the origin

Sometimes there is a discontinuity in the variogram at distances less than the minimum data spacing.  This is known as **nugget effect**.

The ratio of nugget / sill, is known as relative nugget effect (%). Modeled as a discontinuity with no correlation structure that at lags, $h \gt \epsilon$, an infinitesimal lag distance, and perfect correlation at $\bf{h} = 0$.
Caution when including nuggect effect in the variogram model as measurement error, mixing populations cause apparent nugget effect

This exercise demonstrates the semivariogram calculation with GeostatsPy. The steps include:

1. generate a 2D model with sequential Gaussian simulation
2. sample from the simulation
3. calculate and visualize experimental semivariograms

#### Variogram Calculation Parameters

The variogram calculation parameters include:

* **azimuth** is the azimuth of the lag vector

* **azimuth tolerance** is the maximum allowable departure from the azimuth (isotropic variograms are calculated with an azimuth tolerance of to 90.0)

* **unit lag distance** the size of the bins in lag distance, usually set to the minimum data spacing

* **lag distance tolerance** - the allowable tolerance in lage distance, commonly set to 50% of unit lag distanceonal smoothing

* **number of lags** - set based on the spatial extent of the dataset, we can typically calculate reliable variograms up to 1/2 the extent of the dataset

* **bandwidth** is the maximum offset allowable from the lag vector

#### Variogram Modeling

Spatial continuity can be modeled with nested, positive definate variogram structures:

\begin{equation}
\Gamma_x(\bf{h}) = \sum_{i=1}^{nst} \gamma_i(\bf{h})
\end{equation}

where $\Gamma_x(\bf{h})$ is the nested variogram model resulting from the summation of $nst$ nested variograms  $\gamma_i(\bf{h})$.

The types of structure commonly applied include:

* spherical

* exponential

* gaussian

* nugget

Other less common models include:

* hole effect

* dampenned hole effect

* power law

these will not be covered here.

Each one of these variogram structures, $\gamma_i(\bf{h})$, is based on a geometric anisotropy model parameterized by the orientation and range in the major and minor directions.  In 2D this is simply an azimuth and ranges, $azi$, $a_{maj}$ and $a_{min}$. Note, the range in the minor direction (orthogonal to the major direction).

The geometric anisotropy model assumes that the range in all off-diagonal directions is based on an ellipse with the major and minor axes alligned with and set to the major and minor for the variogram.

\begin{equation}
\bf{h}_i = \sqrt{\left(\frac{r_{maj}}{a_{maj_i}}\right)^2 + \left(\frac{r_{maj}}{a_{maj_i}}\right)^2}
\end{equation}

Therefore, if we know the major direction, range in major and minor directions, we may completely describe each nested componnent of the complete spatial continuity of the variable of interest, $i = 1,\dots,nst$.

Some comments on modeling nested variograms:

* we can capture nugget, short and long range continuity structures

* we rely on the geometric anisotropy model, so all structures must inform the same level of contribution (porportion of the sill) in all directions.

* the geometric anisotropy model is based on azimuth of the major direction of continuity, range in the major direction and range in the minor direction (orthogonal to the major direction).  The range is interpolated between the major and minor azimuths with a ellipse model

* we can vary the type of variogram, direction or azimuth of the major direction, and major and minor ranges by structure

In this workflow we will explore methods to:

1. detect directionality from a spatial dataset
2. calculate the directional variograms in the major and minor directions
3. build a consistent 2D model fit to the major and minor directions

#### Objective

In the PGE 383: Stochastic Subsurface Modeling class I want to provide hands-on experience with building subsurface modeling workflows. Python provides an excellent vehicle to accomplish this. I have coded a package called GeostatsPy with GSLIB: Geostatistical Library (Deutsch and Journel, 1998) functionality that provides basic building blocks for building subsurface modeling workflows.

The objective is to remove the hurdles of subsurface modeling workflow construction by providing building blocks and sufficient examples. This is not a coding class per se, but we need the ability to 'script' workflows working with numerical methods.

#### Getting Started

Here's the steps to get setup in Python with the GeostatsPy package:

1. Install Anaconda 3 on your machine (https://www.anaconda.com/download/).
2. From Anaconda Navigator (within Anaconda3 group), go to the environment tab, click on base (root) green arrow and open a terminal.
3. In the terminal type: pip install geostatspy.
4. Open Jupyter and in the top block get started by copy and pasting the code block below from this Jupyter Notebook to start using the geostatspy functionality.

You will need to copy the data file to your working directory.  They are available here:

* Tabular data - sample_data.csv at https://git.io/fh4gm.

There are exampled below with these functions. You can go here to see a list of the available functions, https://git.io/fh4eX, other example workflows and source code.

#### Load the required libraries

The following code loads the required libraries.

```python
import geostatspy.GSLIB as GSLIB                       # GSLIB utilies, visualization and wrapper
import geostatspy.geostats as geostats                 # GSLIB methods convert to Python
```

We will also need some standard packages. These should have been installed with Anaconda 3.

```python
%matplotlib inline
import os                                               # to set current working directory
import sys                                              # supress output to screen for interactive variogram modeling
import io
import numpy as np                                      # arrays and matrix math
import pandas as pd                                     # DataFrames
import matplotlib.pyplot as plt                         # plotting
from matplotlib.pyplot import cm                        # color maps
from ipywidgets import interactive                      # widgets and interactivity
from ipywidgets import widgets
from ipywidgets import Layout
from ipywidgets import Label
from ipywidgets import VBox, HBox
```

If you get a package import error, you may have to first install some of these packages. This can usually be accomplished by opening up a command window on Windows and then typing 'python -m pip install [package-name]'. More assistance is available with the respective package docs.

#### Set the working directory

I always like to do this so I don't lose files and to simplify subsequent read and writes (avoid including the full address each time).  Also, in this case make sure to place the required (see above) GSLIB executables in this directory or a location identified in the environmental variable *Path*.

```python
os.chdir("c:/PGE383")                                   # set the working directory
```

#### Loading Tabular Data

Here's the command to load our comma delimited data file in to a Pandas' DataFrame object.

```python
df = pd.read_csv("sample_data_MV_biased.csv")           # read a .csv file in as a DataFrame
df.head()                                               # we could also use this command for a table preview
```

      Index
      X
      Y
      Facies
      Porosity
      Perm
      AI

      0
      259
      190
      19
      0
      0.041122
      0.120862
      6367.442357

      1
      501
      270
      49
      0
      0.043853
      0.094627
      6906.176965

      2
      273
      270
      219
      0
      0.062169
      0.408779
      6244.965113

      3
      159
      380
      259
      0
      0.064295
      0.322764
      7113.679894

      4
      96
      870
      449
      0
      0.065537
      0.672005
      5672.957304

The features:

* **X** - x coordinate in meters
* **Y** - y coordinate in meters
* **Porosity** - rock porosity averaged over a specific rock unit from a vertical well
* **Perm** - rock permeability averaged (scaled up) over a specific rock unit from a vertical well
* **AI** - acoustic impedance from a seismic cube assigned at a specific rock unit and at the location of a vertical well
* **facies** - facies, 0 - shale, 1 - sandstone

Concerning facies:

We will work with all facies pooled together. I wanted to simplify this workflow and focus more on spatial continuity direction detection. Finally, by not using facies we do have more samples to support our statistical inference. Most often facies are essential in the subsurface model. Don't worry we will check if this is reasonable in a bit.

You are welcome to repeat this workflow on a by-facies basis.  The following code could be used to build DataFrames ('df_sand' and 'df_shale') for each facies.

```p
df_sand = pd.DataFrame.copy(df[df['Facies'] == 1]).reset_index()  # copy only 'Facies' = sand records
df_shale = pd.DataFrame.copy(df[df['Facies'] == 0]).reset_index() # copy only 'Facies' = shale records
```

Let's look at summary statistics for all facies combined:

```python
df.describe().transpose()                               # summary table of sand only DataFrame statistics
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      Index
      367.0
      292.926431
      169.167110
      0.000000
      150.000000
      295.000000
      440.000000
      586.000000

      X
      367.0
      500.108992
      289.978309
      0.000000
      240.000000
      500.000000
      765.000000
      990.000000

      Y
      367.0
      520.403270
      277.752407
      9.000000
      269.000000
      539.000000
      769.000000
      999.000000

      Facies
      367.0
      0.596730
      0.491224
      0.000000
      0.000000
      1.000000
      1.000000
      1.000000

      Porosity
      367.0
      0.126874
      0.030546
      0.041122
      0.103398
      0.125616
      0.148323
      0.210258

      Perm
      367.0
      83.313352
      224.350475
      0.094627
      2.288119
      10.294907
      50.219914
      1991.097723

      AI
      367.0
      4790.458884
      975.582304
      1981.177309
      4108.843701
      4712.673154
      5466.957467
      7561.250336

#### Set the Model Parameters

See the the following model parameters:

* **xmin**, **xmax**, **ymin** and **ymax** - extents of the dataset for plotting
* **feature** and **feature_units** - feature of interest and associated units
* **vmin** and **vmax** - minimum and maximum of the feature of interest

```python
xmin = 0.0; xmax = 1000.0                                # spatial extents in x and y
ymin = 0.0; ymax = 1000.0
feature = 'Porosity'; feature_units = 'percentage'         # name and units of the feature of interest
vmin = 0.0; vmax = 0.25                                  # min and max of the feature of interest
cmap = plt.cm.inferno                                    # set the color map
```

Let's transform the feature to standard normal (mean = 0.0, standard deviation = 1.0, Gaussian shape). This is required for sequential Gaussian simulation (common target for our variogram models) and the Gaussian transform assists with outliers and provides more interpretable variograms.

Let's look at the inputs for the GeostatsPy nscore program.  Note the output include an ndarray with the transformed values (in the same order as the input data in Dataframe 'df' and column 'vcol'), and the transformation table in original values and also in normal score values.

```python
geostats.nscore                                         # see the input parameters required by the nscore function
```

    <function geostatspy.geostats.nscore(df, vcol, wcol=None, ismooth=False, dfsmooth=None, smcol=0, smwcol=0)>

The following command will transform the Porosity and Permeabilty to standard normal.

```python
#Transform to Gaussian by Facies
df['N' + feature], tvPor, tnsPor = geostats.nscore(df, feature) # nscore transform for all facies porosity
```

Let's look at the updated DataFrame to make sure that we now have the normal score porosity and permeability.

```python
df.head()                                               # preview sand DataFrame with nscore transforms
```

      Index
      X
      Y
      Facies
      Porosity
      Perm
      AI
      NPorosity

      0
      259
      190
      19
      0
      0.041122
      0.120862
      6367.442357
      -2.671293

      1
      501
      270
      49
      0
      0.043853
      0.094627
      6906.176965
      -2.644781

      2
      273
      270
      219
      0
      0.062169
      0.408779
      6244.965113
      -2.467028

      3
      159
      380
      259
      0
      0.064295
      0.322764
      7113.679894
      -2.344090

      4
      96
      870
      449
      0
      0.065537
      0.672005
      5672.957304
      -2.248832

That looks good! One way to check is to see if the relative magnitudes of the normal score transformed values match the original values.  e.g. that the normal score transform of 0.10 porosity normal score is less than the normal score transform of 0.14 porsity.  Also, the normal score transform of values close to the mean value should be close to 0.0

Let's also check the original and transformed sand and shale porosity distributions.

```python
plt.subplot(121)                                        # plot original sand and shale porosity histograms
plt.hist(df[feature], facecolor='red',bins=np.linspace(vmin,vmax,1000),histtype="stepfilled",alpha=0.2,density=True,cumulative=True,edgecolor='black')
plt.xlim([vmin,vmax]); plt.ylim([0,1.0])
plt.xlabel(feature + '(' + feature_units + ')'); plt.ylabel('Frequency'); plt.title('Porosity')
plt.grid(True)

plt.subplot(122)
plt.hist(df['N'+feature], facecolor='blue',bins=np.linspace(-3.0,3.0,1000),histtype="stepfilled",alpha=0.2,density=True,cumulative=True,edgecolor='black')
plt.xlim([-3.0,3.0]); plt.ylim([0,1.0])
plt.xlabel('Gaussian Transformed ' + feature); plt.ylabel('Frequency'); plt.title('Guassian Transformed ' + feature)
plt.grid(True)

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.2, wspace=0.2, hspace=0.3)
plt.show()
```

We can see that the normal score transform has correctly transformed the feature to standard normal.

#### Inspection of Posted Data

Data visualization is very useful to detect patterns. Our brains are very good at pattern detection. I promote quantitative methods and recognize issues with cognitive bias, but it is important to recognize the value is expert intepretation based on data visualization.

* This data visualization will also be important to assist with parameter selection for the variogram calculation search template.

Let's plot the location maps of the original feature and the normal score transforms of the feature.

```python
plt.subplot(121)                                        # location map of normal score transform of porosity
GSLIB.locmap_st(df,'X','Y',feature,0,1000,0,1000,vmin,vmax,feature,'X (m)','Y (m)',feature,cmap)

plt.subplot(122)                                        # location map of normal score transform of porosity
GSLIB.locmap_st(df,'X','Y','N'+feature,0,1000,0,1000,-3,3,'Gaussian Transformed ' + feature,'X (m)','Y (m)','Gaussian Transformed ' + feature,cmap)

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.0, wspace=0.5, hspace=0.3)
plt.show()
```

What do you see?  Here's my observations:

* there is a high degree of spatial agreement between porosity and permeability, this is supported by the high correlation evident in the cross plot.
* there are no discontinuities that could suggest that facies represent a distinct change, rather the porosity and permeability seem continuous and the assigned facies are a truncation of their continous behavoir, we doing 'ok' with no facies
* suspect a 045 azimuth major direction of continuity (up - right)
* there may be cycles in the 135 azimuth
* there will not likely be a nugget effect, but there is an hint of some short scale discontinuity?

**Do you agree?** If you have a different observations, drop me a line at mpyrcz@austin.utexas.edu and I'll add to this lesson with credit!

#### Experimental Variograms

We can use the location maps to help determine good variogram calculation parameters. For example:

```p
tmin = -9999.; tmax = 9999.;
lag_dist = 100.0; lag_tol = 50.0; nlag = 7; bandh = 9999.9; azi = azi; atol = 22.5; isill = 1
```
* **tmin**, **tmax** are trimming limits - set to have no impact, no need to filter the data
* **lag_dist**, **lag_tol** are the lag distance, lag tolerance - set based on the common data spacing (100m) and tolerance as 100% of lag distance for additonal smoothing
* **nlag** is number of lags - set to extend just past 50 of the data extent
* **bandh** is the horizontal band width - set to have no effect
* **azi** is the azimuth -  it has not effect since we set atol, the azimuth tolerance, to 90.0
* **isill** is a boolean to standardize the distribution to a variance of 1 - it has no effect since the previous nscore transform sets the variance to 1.0

#### Dashboard for Interactive Variogram Calculation and Modeling

Below we make a dashboard with the ipywidgets and matplotlib Python packages for calculating and modeling experimental variograms.

* allowing you to calculate and model the variogram of the normal score transformed variogram interactively while changing (and exploring) the search template parameters.

* first calculate the isotropic or directional variogram(s)

* then fit the same isotropic or directional variogram(s)

```python
# interactive calculation of the experimental variogram
l = widgets.Text(value='                              Variogram Calculation Interactive Demonstration, Michael Pyrcz, Associate Professor, The University of Texas at Austin',layout=Layout(width='950px', height='30px'))
lag = widgets.FloatSlider(min = 10, max = 500, value = 10, step = 10, description = 'lag',orientation='vertical',layout=Layout(width='90px', height='200px'))
lag.style.handle_color = 'gray'

lag_tol = widgets.FloatSlider(min = 5, max = 500, value = 5, step = 10, description = 'lag tolerance',orientation='vertical',layout=Layout(width='90px', height='200px'))
lag_tol.style.handle_color = 'gray'

nlag = widgets.IntSlider(min = 1, max = 100, value = 100, step = 1, description = 'number of lags',orientation='vertical',layout=Layout(width='90px', height='200px'))
nlag.style.handle_color = 'gray'

azi = widgets.FloatSlider(min = 0, max = 360, value = 0, step = 5, description = 'azimuth',orientation='vertical',layout=Layout(width='90px', height='200px'))
azi.style.handle_color = 'gray'

azi_tol = widgets.FloatSlider(min = 10, max = 90, value = 10, step = 5, description = 'azimuth tolerance',orientation='vertical',layout=Layout(width='120px', height='200px'))
azi_tol.style.handle_color = 'gray'

bandwidth = widgets.FloatSlider(min = 100, max = 2000, value = 2000, step = 100, description = 'bandwidth',orientation='vertical',layout=Layout(width='90px', height='200px'))
azi_tol.style.handle_color = 'gray'

ui1 = widgets.HBox([lag,lag_tol,nlag,azi,azi_tol,bandwidth],) # basic widget formatting
ui = widgets.VBox([l,ui1],)

def f_make(lag,lag_tol,nlag,azi,azi_tol,bandwidth):     # function to take parameters, calculate variogram and plot
#    text_trap = io.StringIO()
#    sys.stdout = text_trap
    global lags,gammas,npps,lags2,gammas2,npps2
    tmin = -9999.9; tmax = 9999.9
    lags, gammas, npps = geostats.gamv(df,"X","Y","N"+feature,tmin,tmax,lag,lag_tol,nlag,azi,azi_tol,bandwidth,isill=1.0)
    lags2, gammas2, npps2 = geostats.gamv(df,"X","Y","N"+feature,tmin,tmax,lag,lag_tol,nlag,azi+90.0,azi_tol,bandwidth,isill=1.0)

    plt.subplot(111)                                    # plot experimental variogram
    plt.scatter(lags,gammas,color = 'black',s = npps*0.03,label = 'Major Azimuth ' +str(azi), alpha = 0.8)
    plt.scatter(lags2,gammas2,color = 'red',s = npps*0.03,label = 'Minor Azimuth ' +str(azi+90.0), alpha = 0.8)
    plt.plot([0,2000],[1.0,1.0],color = 'black')
    plt.xlabel(r'Lag Distance $\bf(h)$, (m)')
    plt.ylabel(r'$\gamma \bf(h)$')
    if azi_tol < 90.0:
        plt.title('Directional NSCORE ' + feature + ' Variogram - Azi. ' + str(azi) + ', Azi. Tol.' + str(azi_tol))
    else:
        plt.title('Omni Directional NSCORE ' + feature + ' Variogram ')
    plt.xlim([0,1000]); plt.ylim([0,1.8])
    plt.legend(loc="lower right")
    plt.grid(True)

    plt.subplots_adjust(left=0.0, bottom=0.0, right=1.5, top=1.0, wspace=0.3, hspace=0.3)
    plt.show()

    return

# connect the function to make the samples and plot to the widgets
interactive_plot = widgets.interactive_output(f_make, {'lag':lag,'lag_tol':lag_tol,'nlag':nlag,'azi':azi,'azi_tol':azi_tol,'bandwidth':bandwidth})
interactive_plot.clear_output(wait = True)               # reduce flickering by delaying plot updating
```

### Interactive Variogram Calculation Demonstration

* calculate omnidirectional and direction experimental variograms

#### Michael Pyrcz, Associate Professor, University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1) | [GeostatsPy](https://github.com/GeostatsGuy/GeostatsPy)

### The Problem

Calculate interpretable experimental variograms for sparse, irregularly-space spatial data. Note, size of the experimental point is scaled by the number of pairs.

* **azimuth** is the azimuth of the lag vector

* **azimuth tolerance** is the maximum allowable departure from the azimuth

* **unit lag distance** the size of the bins in lag distance

* **lag distance tolerance** - the allowable tolerance in lage distance

* **number of lags** - number of lags in the experimental variogram

* **bandwidth** - maximum departure from the lag vector

```python
display(ui, interactive_plot)                           # display the interactive plot
```

    VBox(children=(Text(value='                              Variogram Calculation Interactive Demonstration, Mich…

    Output(outputs=({'output_type': 'display_data', 'data': {'text/plain': '
', 'i…

```python
# interactive calculation of the sample set (control of source parametric distribution and number of samples)
l = widgets.Text(value='                              Variogram Modeling, Michael Pyrcz, Associate Professor, The University of Texas at Austin',layout=Layout(width='950px', height='30px'))
nug = widgets.FloatSlider(min = 0, max = 1.0, value = 0.0, step = 0.1, description = 'nug',orientation='vertical',layout=Layout(width='60px', height='200px'))
nug.style.handle_color = 'gray'
it1 = widgets.Dropdown(options=['Spherical', 'Exponential', 'Gaussian'],value='Spherical',
    description='Type1:',disabled=False,layout=Layout(width='200px', height='30px'))
c1 = widgets.FloatSlider(min=0.0, max = 1.0, value = 0.1, description = 'c1',orientation='vertical',layout=Layout(width='60px', height='200px'))
c1.style.handle_color = 'gray'
hmaj1 = widgets.FloatSlider(min=0.01, max = 10000.0, value = 0.01, step = 25.0, description = 'hmaj1',orientation='vertical',layout=Layout(width='60px', height='200px'))
hmaj1.style.handle_color = 'black'
hmin1 = widgets.FloatSlider(min = 0.01, max = 10000.0, value = 0.01, step = 25.0, description = 'hmin1',orientation='vertical',layout=Layout(width='60px', height='200px'))
hmin1.style.handle_color = 'red'

it2 = widgets.Dropdown(options=['Spherical', 'Exponential', 'Gaussian'],value='Spherical',
    description='Type2:',disabled=False,layout=Layout(width='200px', height='30px'))
c2 = widgets.FloatSlider(min=0.0, max = 1.0, value = 0.0, description = 'c2',orientation='vertical',layout=Layout(width='60px', height='200px'))
c2.style.handle_color = 'gray'
hmaj2 = widgets.FloatSlider(min=0.01, max = 10000.0, value = 0.01, step = 100.0, description = 'hmaj2',orientation='vertical',layout=Layout(width='60px', height='200px'))
hmaj2.style.handle_color = 'black'
hmin2 = widgets.FloatSlider(min = 0.01, max = 10000.0, value = 0.01, step = 100.0, description = 'hmin2',orientation='vertical',layout=Layout(width='60px', height='200px'))
hmin2.style.handle_color = 'red'

ui1 = widgets.HBox([nug,it1,c1,hmaj1,hmin1,it2,c2,hmaj2,hmin2],)                   # basic widget formatting
#ui2 = widgets.HBox([it2,c2,hmaj2,hmin2],)                   # basic widget formatting
ui = widgets.VBox([l,ui1],)

def convert_type(it):
    if it == 'Spherical':
        return 1
    elif it == 'Exponential':
        return 2
    else:
        return 3

def f_make(nug,it1,c1, hmaj1, hmin1, it2, c2, hmaj2, hmin2):                       # function to take parameters, make sample and plot
    text_trap = io.StringIO()
    sys.stdout = text_trap

    it1 = convert_type(it1); it2 = convert_type(it2)
    if c2 > 0.0:
        nst = 2
    else:
        nst = 1

    vario = GSLIB.make_variogram(nug,nst,it1,c1,0.0,hmaj1,hmin1,it2,c2,0.0,hmaj2,hmin2) # make model object
    nlag = 100; xlag = 10;
    index_maj,h_maj,gam_maj,cov_maj,ro_maj = geostats.vmodel(nlag,xlag,0.0,vario)   # project the model in the major azimuth                                                  # project the model in the 135 azimuth
    index_min,h_min,gam_min,cov_min,ro_min = geostats.vmodel(nlag,xlag,90.0,vario)

    plt.subplot(111)                                    # plot experimental variogram
    plt.scatter(lags,gammas,color = 'black',s = npps*0.03,label = 'Major Azimuth ' +str(azi.value), alpha = 0.8)
    plt.plot(h_maj,gam_maj,color = 'black')
    plt.scatter(lags2,gammas2,color = 'red',s = npps*0.03,label = 'Minor Azimuth ' +str(azi.value+90.0), alpha = 0.8)
    plt.plot(h_min,gam_min,color = 'red')
    plt.plot([0,2000],[1.0,1.0],color = 'black')
    plt.xlabel(r'Lag Distance $\bf(h)$, (m)')
    plt.ylabel(r'$\gamma \bf(h)$')
    if azi_tol.value < 90.0:
        plt.title('Directional NSCORE ' + feature + ' Variogram - Azi. ' + str(azi.value) + ', Azi. Tol.' + str(azi_tol.value))
    else:
        plt.title('Omni Directional NSCORE ' + feature + ' Variogram ')
    plt.xlim([0,1000]); plt.ylim([0,1.8])
    plt.legend(loc="lower right")
    plt.grid(True)

    plt.subplots_adjust(left=0.0, bottom=0.0, right=2.2, top=1.5, wspace=0.3, hspace=0.3)
    plt.show()

# connect the function to make the samples and plot to the widgets
interactive_plot = widgets.interactive_output(f_make, {'nug':nug, 'it1':it1,'c1':c1, 'hmaj1':hmaj1, 'hmin1':hmin1, 'it2':it2, 'c2':c2, 'hmaj2':hmaj2, 'hmin2':hmin2})
interactive_plot.clear_output(wait = True)               # reduce flickering by delaying plot updating
```

### Interactive Nested Variogram Modeling Demostration

* select the nested structures and their types, contributions and major and minor ranges

#### Michael Pyrcz, Associate Professor, University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1) | [GeostatsPy](https://github.com/GeostatsGuy/GeostatsPy)

### The Problem

Fit a positive definite variogram model based on the addition of multiple structures each describing spatial components of the feature variance

* **nug**: nugget effect

* **c1 / c2**: contributions of the sill

* **hmaj1 / hmaj2**: range in the major direction

* **hmin1 / hmin2**: range in the minor direction

```python
display(ui, interactive_plot)                           # display the interactive plot
```

    VBox(children=(Text(value='                              Variogram Modeling, Michael Pyrcz, Associate Professo…

    Output(outputs=({'output_type': 'display_data', 'data': {'text/plain': '
', 'i…

#### Comments

This was a basic demonstration / exercise of variogram calculation and modeling for spatial continuity analysis. Much more could be done, I have other demonstrations on the basics of working with DataFrames, ndarrays, univariate statistics, plotting data, declustering, data transformations and many other workflows available at https://github.com/GeostatsGuy/PythonNumericalDemos and https://github.com/GeostatsGuy/GeostatsPy.

#### The Author:

### Michael Pyrcz, Associate Professor, University of Texas at Austin
*Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions*

With over 17 years of experience in subsurface consulting, research and development, Michael has returned to academia driven by his passion for teaching and enthusiasm for enhancing engineers' and geoscientists' impact in subsurface resource development.

For more about Michael check out these links:

#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

#### Want to Work Together?

I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.

* Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I'd be happy to drop by and work with you!

* Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!

* I can be reached at mpyrcz@austin.utexas.edu.

I'm always happy to discuss,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin

#### More Resources Available at: [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)


EMG Tutorial

This short tutorial is a hands-on introduction to electromyography (EMG) signal processing and analyses. During this exercise, we will explore a single differential EMG's amplitude and spectral features recorded on two biceps brachii contractions. This practical example is part of the Graz University of Technology lecture series *Applied Electrophysiology and Sensors*.

I wish everyone a great dive into the topic, and please do not hesitate to contact me in case of **any** questions!

yours,

Christoph

Free EMG Ressources:

*   Practical Introduction into EMG from Carlo de Luca (provided open-source by Delsys)
*   Lecture slides and other relevant EMG resources provided by Roberto Merletti
*   Publication on a tattoo-based EMG system [1] - one recording of this study is used in this tutorial.

Contact

Christoph Leitner

e christoph.leitner@tugraz.at

g github.com/luuleitner/EMG_tutorial

---

Getting Started

```
[#] Areas shown like this are executable code. Use the mousover play button to run these cells.
```

First we need to install the EMG tutorial package from the the GitHub repository.

```python
!pip install git+https://github.com/luuleitner/EMG_tutorial
```

We load all necessary libraries into our Colab notebook in this next step.

```python
import numpy as np # numpy scientific computing package
import pandas as pd # pandas data analysis library
import matplotlib.pyplot as plt # matplotlib library to plot graphs and data
from scipy import signal # package providing fundamental algorithms for scientific computing

from emg_processing.filter import Butter # Butterworth filter function from the EMG-tutorial package
from emg_processing.analyses import rms # Root Mean Square function from the EMG-tutorial package
```

We download an EMG sample file from the repository and plot the time trace. The EMG data were recorded using two differential electrodes on the biceps brachii. Please have a look at our publication of the tattoo-based EMG system to get an impression of the applied test setup and electrode placement.

```python
url = 'https://raw.githubusercontent.com/luuleitner/EMG_tutorial/master/data/bbcontractions.csv'
EMG_data = pd.read_csv(url, sep=',').to_numpy()
EMG_data = np.ravel(EMG_data[:, 0])
```

```python
fig = plt.figure(figsize=(6,4), dpi = 100)
ax_1 = fig.add_subplot(111)
ax_1.plot(EMG_data,
          markersize = 1,
          color='blue',
          label = 'x')
ax_1.set_ylabel('EMG raw [uV]')
ax_1.grid()
ax_1.set_xlabel('Samples')
plt.show()
```

Assignment

1. Filtering

We recorded our data with a sampling frequency of 1kHz. To exclude unwanted frequency components in our raw EMG signal we want to apply a fourth-order (n=4) bandpass filter (BPF) with cutoff frequencies at 20Hz (LOW) and 320Hz (HIGH). For the filtering, you will use the provided function `Butter()`, which requires inputs in the following order: `Butter(raw data, cutoff low, cutoff high, sampling rate, order= order number)`

Enter the necessary filter parameters.

```python
ARG_SAMPLING_RATE = ## --> enter data values <--

# Bandpass Filter (BPF)
ARG_BPF_N = ## --> enter data values <--
ARG_BPF_LOW = ## --> enter data values <--
ARG_BPF_HIGH = ## --> enter data values <--
```

Apply the provided Butterworth bandpass filter to process the EMG raw data.

```python
EMG_data_filtered = #--> enter function to filter the raw EMG signal <--
```

Plot the raw and the filtered EMG data. Attention: the `Butter()` class returns a nested structure - use `EMG_data_filtered.data_filtered` to access the filtered signal.

```python
fig = plt.figure(figsize=(6,4), dpi = 100)
ax_1 = fig.add_subplot(211)
ax_1.plot(#--> enter raw data array <--,
          markersize = 1,
          color='blue',
          label = 'x')
ax_1.set_ylabel('EMG raw [uV]')
ax_1.grid()

ax_2 = fig.add_subplot(212)
ax_2.plot(#--> enter filtered data array<--,
          markersize = 1,
          color='red',
          label = 'x')
ax_2.set_ylabel('EMG filtered [uV]')
ax_2.set_xlabel('Samples')
ax_2.grid()

plt.show()
```

2. Amplitude analyses

We first rectify the filtered EMG signal for the amplitude analyses using the numpy function `np.abs()` on the filtered EMG signal. Be aware that the output of `Butter()` has a nested structure. To access the filtered EMG signal you need to enter `name_of_your_output_variable.data_filtered`.

For ease of use, I have already implemented a function called `rms()` to calculate the root mean square (RMS) of the EMG signal. However, this is not magic and could easily be implemented by yourself using the formula for the RMS:

\begin{align}
        RMS = \sqrt (1/n \sum_{i} x_{i}^{2})
\end{align}

RMS...root mean square

n    ...number of samples

$x_i$...EMG value

Rectify the EMG data.

```python
EMG_data_filtered.data_filtered_rect = #--> enter function to rectify the filtered EMG signal <--
```

Plot the rectified EMG signal.

```python
fig = plt.figure(figsize=(6,4), dpi = 100)
ax_1 = fig.add_subplot(211)
ax_1.plot(#--> enter filtered and rectified data array <--,
          markersize = 1,
          color='red',
          label = 'x')
ax_1.set_ylabel('EMG raw [uV]')
ax_1.set_xlabel('Samples')
ax_1.grid()

plt.show()
```

Analyse the EMG signal amplitudes using RMS.

```python
RMS = #--> enter function to calculate the rms <--
print(f'RMS = {RMS} [uV]')
```

3. Power spectral analyses

For the spectral analyses of the EMG signal, we will use Welch's method. For more details on the computation of the power spectrum using Welch's method have a look at some of the resources here: Video tutorial by Mike X. Cohan, Formalism provided by Stanford Center for Computer Research in Music and Acoustics.

This computation is already implemented in python's `scipy` library. You can calculate the power spectrum from the filtered EMG data using the command `signal.welch()`. The function requires the following inputs: `signal.welch(filtered EMG data, sampling rate, nperseg=1024)`. The `nperseg=1024` defines the length of the used window (1024 samples).

Evaluate the power spectral density using Welch's method.

```python
f, P_den = #--> enter function to calculate the power spectrum using Welch's method <--
```

Plot the rectified power spectrum.

```python
fig1 = plt.figure(figsize=(6,3), dpi = 300)
ax_1 = fig1.add_subplot(111)
ax_1.plot(f, P_den,
          markersize = 1,
          color='blue',
          label = 'x')
ax_1.set_xlabel('Frequency [Hz]')
ax_1.set_ylabel('Power [W/Hz]')
plt.ticklabel_format(style='sci', axis='y', scilimits=(0,0))
ax_1.yaxis.major.formatter._useMathText = True
plt.show()
```

Frequency domain features such as mean and median frequencies are frequently used to assess muscular fatigue [2]. In this next exercise we calculate these characteristics of the power spectrum and plot the results using the following formalism:

\begin{align}
        MNF = \sum_{i=1}^{n} f_{i}*P_{i} / \sum_{i=1}^{n} P_{i}
\end{align}

\begin{align}
      MDF = \sum_{i=1}^{n_{MDF}}P_{i} = \frac{1}{2} \sum_{i=1}^{n}P_{i}
\end{align}

MNF...mean frequency

MDF...median frequency

$P_{i}$...power spectrum at frequency of of $i^{th} sample$

$f_{i}$...frequency of $i^{th}$ sample

n...number of samples

$n_{MDF}$...sample number at median frequency

```python
mean_frequency = np.sum(f.T * P_den / np.sum(P_den))
median_frequency = np.squeeze(f[np.argwhere(np.diff(np.cumsum(P_den)<=(0.5 * np.sum(P_den))))])
```

```python
fig1 = plt.figure(figsize=(6,3), dpi = 300)
ax_1 = fig1.add_subplot(111)
ax_1.plot(f, P_den,
          markersize = 1,
          color='blue',
          label = 'x')
ax_1.set_xlabel('Frequency [Hz]')
ax_1.set_ylabel('Power [W/Hz]')
plt.ticklabel_format(style='sci', axis='y', scilimits=(0,0))
ax_1.yaxis.major.formatter._useMathText = True

ax_1.vlines(mean_frequency, 0, np.max(P_den), linestyles ="dotted", colors ="r")
ax_1.annotate(f'Mean frequency = {np.round(mean_frequency,2)} [Hz]', xy=(mean_frequency, np.max(P_den)),  xycoords='data',
            xytext=(0.8, 0.95), textcoords='axes fraction',
            arrowprops=dict(facecolor='black', shrink=0.01, width=1, headwidth=5),
            horizontalalignment='right', verticalalignment='top',
            )

ax_1.vlines(median_frequency, 0, np.max(P_den), linestyles ="dashed", colors ="r")
ax_1.annotate(f'Median frequency = {np.round(median_frequency,2)} [Hz]', xy=(median_frequency, np.max(P_den)/2),  xycoords='data',
            xytext=(0.8, 0.5), textcoords='axes fraction',
            arrowprops=dict(facecolor='black', shrink=0.01, width=1, headwidth=5),
            horizontalalignment='right', verticalalignment='top',
            )

plt.show()
```

# Great job!
## You have completed the EMG tutorial

For more open-source tools visit my github account at: github.com/luuleitner

References

[1] Leitner, C. et al. (2021). UStEMG: an Ultrasound Transparent Tattoo-based sEMG System for Unobtrusive Parallel Acquisitions of Muscle Electro-mechanics. in Proc. 43nd Conferences of the IEEE Engineering in Medicine and Biology Society, Guadalajara, Mexico. DOI: 10.1109/EMBC46164.2021.9630034

[2] Phinyomark, A. et al. (2012). The Usefulness of Mean and Median Frequencies in Electromyography Analysis. In Computational Intelligence in Electromyography Analysis – A Perspective on Current Applications and Future Challenges, edited by Ganesh R. Naik. InTech. DOI: 10.5772/50639


# Calculation of control fields for QFT gate on two qubits using L-BFGS-B algorithm

Alexander Pitchford (agp1@aber.ac.uk)

```python
cd /Users/benrosand/oct-qiskit-pulse
```

    /Users/benrosand/oct-qiskit-pulse

### Imports

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import datetime
```

```python
from qutip import Qobj, identity, sigmax, sigmay, sigmaz, tensor, hadamard_transform, basis
from qutip.qip.algorithms import qft
import qutip.logging_utils as logging
logger = logging.get_logger()
#Set this to None or logging.WARN for 'quiet' execution
log_level = logging.INFO
#QuTiP control modules
import qutip.control.pulseoptim as cpo
import qutip.control.pulsegen as pulsegen
from src.qutip_helper import convert_qutip_ham
from src.helper import *
from qiskit import IBMQ
from qutip.qip.device import Processor
from qiskit.pulse import Play, SamplePulse

from qiskit.providers.aer.pulse.system_models.hamiltonian_model import HamiltonianModel

example_name = 'QFT'
```

```python

```

### Defining the physics

Note here that there are two controls acting on each qubit.

```python
# subsystem_list=[0,1]
subsystem_list=[0,1]
single_q = True
IBMQ.load_account()

# provider = IBMQ.get_provider(hub='ibm-q', group='open', project='main')
provider = IBMQ.get_provider(hub='ibm-q-internal', group='deployed', project='default')
backend = provider.get_backend('ibmq_athens')

```

    /Users/benrosand/anaconda3/envs/qoc/lib/python3.8/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.
      and should_run_async(code)

```python
# HamiltonianModel.from_dict(backend.configuration().hamiltonian, subsystem_list=[0])._system[1][0].full()
print(backend.configuration().hamiltonian['h_latex'])

```

    \begin{align} \mathcal{H}/\hbar = & \sum_{i=0}^{4}\left(\frac{\omega_{q,i}}{2}(\mathbb{I}-\sigma_i^{z})+\frac{\Delta_{i}}{2}(O_i^2-O_i)+\Omega_{d,i}D_i(t)\sigma_i^{X}\right) \\ & + J_{1,2}(\sigma_{1}^{+}\sigma_{2}^{-}+\sigma_{1}^{-}\sigma_{2}^{+}) + J_{3,4}(\sigma_{3}^{+}\sigma_{4}^{-}+\sigma_{3}^{-}\sigma_{4}^{+}) + J_{0,1}(\sigma_{0}^{+}\sigma_{1}^{-}+\sigma_{0}^{-}\sigma_{1}^{+}) + J_{2,3}(\sigma_{2}^{+}\sigma_{3}^{-}+\sigma_{2}^{-}\sigma_{3}^{+}) \\ & + \Omega_{d,0}(U_{0}^{(0,1)}(t))\sigma_{0}^{X} + \Omega_{d,1}(U_{1}^{(1,0)}(t)+U_{2}^{(1,2)}(t))\sigma_{1}^{X} \\ & + \Omega_{d,2}(U_{3}^{(2,1)}(t)+U_{4}^{(2,3)}(t))\sigma_{2}^{X} + \Omega_{d,3}(U_{6}^{(3,4)}(t)+U_{5}^{(3,2)}(t))\sigma_{3}^{X} \\ & + \Omega_{d,4}(U_{7}^{(4,3)}(t))\sigma_{4}^{X} \\ \end{align}
    /Users/benrosand/anaconda3/envs/qoc/lib/python3.8/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.
      and should_run_async(code)

\begin{align} \mathcal{H}/\hbar = & \sum_{i=0}^{4}\left(\frac{\omega_{q,i}}{2}(\mathbb{I}-\sigma_i^{z})+\frac{\Delta_{i}}{2}(O_i^2-O_i)+\Omega_{d,i}D_i(t)\sigma_i^{X}\right) \\ & + J_{1,2}(\sigma_{1}^{+}\sigma_{2}^{-}+\sigma_{1}^{-}\sigma_{2}^{+}) + J_{3,4}(\sigma_{3}^{+}\sigma_{4}^{-}+\sigma_{3}^{-}\sigma_{4}^{+}) + J_{0,1}(\sigma_{0}^{+}\sigma_{1}^{-}+\sigma_{0}^{-}\sigma_{1}^{+}) + J_{2,3}(\sigma_{2}^{+}\sigma_{3}^{-}+\sigma_{2}^{-}\sigma_{3}^{+}) \\ & + \Omega_{d,0}(U_{0}^{(0,1)}(t))\sigma_{0}^{X} + \Omega_{d,1}(U_{1}^{(1,0)}(t)+U_{2}^{(1,2)}(t))\sigma_{1}^{X} \\ & + \Omega_{d,2}(U_{3}^{(2,1)}(t)+U_{4}^{(2,3)}(t))\sigma_{2}^{X} + \Omega_{d,3}(U_{6}^{(3,4)}(t)+U_{5}^{(3,2)}(t))\sigma_{3}^{X} \\ & + \Omega_{d,4}(U_{7}^{(4,3)}(t))\sigma_{4}^{X} \\ \end{align}

```python
backend.configuration().hamiltonian['vars']['wq0'] = 0
backend.configuration().hamiltonian['vars']['wq1'] = 0
ham = convert_qutip_ham(backend, subsystem_list, False)
H_d = ham['H_d']
# ham['H_c'].pop('U2')
H_c = list(ham['H_c'].values())

n_ctrls = len(H_c)
n_ctrls
```

```python
len(H_c)
```

### Define the 3 level operators

```python
h = hadamard_transform(1).full()

hadamard_3 = Qobj([[h[0][0],h[0][1],0],
                   [h[1][0],h[1][1],0],
                   [0,      0,      1]])
sigmax_3 =  Qobj([[0,1,0],
                  [1,0,0],
                  [0,0,1]])

# cnot = Qobj(d_sum(identity(3), sigmax_3))

# cnot = Qobj(tensor(identity(3), sigmax_3)
cnot = d_sum(Qobj(d_sum(identity(3), sigmax_3)), identity(3))

# U_targ = Qobj(tensor(sigmax_3, identity(3)))
# U_targ = Qobj(tensor(identity(3), hadamard_3))
# U_targ = Qobj(tensor(identity(3),sigmax_3))
U_0 = tensor(identity(3), identity(3))
U_targ = Qobj(cnot)
```

    /Users/benrosand/anaconda3/envs/qoc/lib/python3.8/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.
      and should_run_async(code)
    <ipython-input-8-a34fe8f3629a>:1: DeprecationWarning: Importing functions/classes of the qip submodule directly from the namespace qutip is deprecated. Please import them from the submodule instead, e.g.
    from qutip.qip.operations import cnot
    from qutip.qip.circuit import QubitCircuit

      h = hadamard_transform(1).full()

### Defining the time evolution parameters

Multiple total evolution times will be tried. Using this approach, the minimum evolution time required to achieve the target fidelity could be determined (iteratively).

Note that the timeslot duration dt is fixed, and so the number of timeslots depends on the evo_time

```python
# Duration of each timeslot
dt = backend.configuration().dt * 1e9
# dt = 0.02
n_ts_list = list(np.arange(320,1200,160))
# n_ts_list = [512]#, 480, 560,640]
evo_times = [dt * n for n in n_ts_list]
# List of evolution times to try
# evo_times = [1, 3, 6]
n_evo_times = len(evo_times)
evo_time = evo_times[0]
n_ts = int(float(evo_time) / dt)
#Empty list that will hold the results for each evolution time
# results = list()
```

    /Users/benrosand/anaconda3/envs/qoc/lib/python3.8/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.
      and should_run_async(code)

### Set the conditions which will cause the pulse optimisation to terminate

### Set the initial pulse type

Here the linear initial pulse type is used, simply because it results in smooth final pulses

```python
# pulse type alternatives: RND|ZERO|LIN|SINE|SQUARE|SAW|TRIANGLE|
p_type = 'LIN'
```

    /Users/benrosand/anaconda3/envs/qoc/lib/python3.8/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.
      and should_run_async(code)

### Give an extension for output files

```python
#Set to None to suppress output files
# f_ext = "{}_n_ts{}_ptype{}.txt".format(example_name, n_ts, p_type)
f_ext = None
```

    /Users/benrosand/anaconda3/envs/qoc/lib/python3.8/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.
      and should_run_async(code)

### Create the optimiser objects

Here is the main difference between this and the Hadamard example. In this case we use a different pulseoptim function that just creates the objects that can be used to set the physics and configure the optimisation algorithm. This gives greater flexibility (shown here by seting different initial pulse parameters for each control) and is also more efficient when running multiple optimisations on the same system.

```python
# Fidelity error target
fid_err_targ = 1e-30
# Maximum iterations for the optisation algorithm
max_iter = 2000
# Maximum (elapsed) time allowed in seconds
max_wall_time = 120
# Minimum gradient (sum of gradients squared)
# as this tends to 0 -> local minima has been found
min_grad = 1e-30
```

    /Users/benrosand/anaconda3/envs/qoc/lib/python3.8/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.
      and should_run_async(code)

```python
optim = cpo.create_pulse_optimizer(H_d, H_c, U_0, U_targ, n_ts, evo_time,
                amp_lbound=-.7, amp_ubound=.7,
                fid_err_targ=fid_err_targ, min_grad=min_grad,
                max_iter=max_iter, max_wall_time=max_wall_time,
                optim_method='fmin_l_bfgs_b',
                method_params={'max_metric_corr':20, 'accuracy_factor':1e8},
                dyn_type='UNIT',
                fid_params={'phase_option':'PSU'},
                init_pulse_type=p_type,
                log_level=log_level, gen_stats=True)

# **** get handles to the other objects ****
optim.test_out_files = 0
dyn = optim.dynamics
dyn.test_out_files = 0
p_gen = optim.pulse_generator

```

### Optimise the pulse for each of the different evolution times

Here a loop is used to perform the optimisation for each of the evo_times given in the list above. The first optimisation is completed using the timeslot parameters passed when the optimisation objects are created. For the subsequent runs, the Dynamics object 'dyn' is used to set the timeslot parameters before the initial pulses are generated and optimisation is completed. Note that using this method, the dyn.initialize_controls method must be called with an array of the initial amplitudes before the optim.run_optimization method is called.

```python
dt
```

    /Users/benrosand/anaconda3/envs/qoc/lib/python3.8/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.
      and should_run_async(code)

    0.2222222222222222

```python
results = []
for i in range(n_evo_times):
    # Generate the tau (duration) and time (cumulative) arrays
    # so that it can be used to create the pulse generator
    # with matching timeslots
    dyn.init_timeslots()
    if i > 0:
        # Create a new pulse generator for the new dynamics
        p_gen = pulsegen.create_pulse_gen(p_type, dyn)

    #Generate different initial pulses for each of the controls
    init_amps = np.zeros([n_ts, n_ctrls])
    if (p_gen.periodic):
        phase_diff = np.pi / n_ctrls
        for j in range(n_ctrls):
            init_amps[:, j] = p_gen.gen_pulse(start_phase=phase_diff*j)
    elif (isinstance(p_gen, pulsegen.PulseGenLinear)):
        for j in range(n_ctrls):
            p_gen.scaling = float(j) - float(n_ctrls - 1)/2
            init_amps[:, j] = p_gen.gen_pulse()
    elif (isinstance(p_gen, pulsegen.PulseGenZero)):
        for j in range(n_ctrls):
            p_gen.offset = sf = float(j) - float(n_ctrls - 1)/2
            init_amps[:, j] = p_gen.gen_pulse()
    else:
        # Should be random pulse
        for j in range(n_ctrls):
            init_amps[:, j] = p_gen.gen_pulse()

    dyn.initialize_controls(init_amps)

    # Save initial amplitudes to a text file
    if f_ext is not None:
        pulsefile = "ctrl_amps_initial_" + f_ext
        dyn.save_amps(pulsefile)
        print("Initial amplitudes output to file: " + pulsefile)

    print("***********************************")
    print("\n+++++++++++++++++++++++++++++++++++")
    print("Starting pulse optimisation for T={}".format(evo_time))
    print("+++++++++++++++++++++++++++++++++++\n")
    result = optim.run_optimization()
    # Save final amplitudes to a text file
    if f_ext is not None:
        pulsefile = "ctrl_amps_final_" + f_ext
        dyn.save_amps(pulsefile)
        print("Final amplitudes output to file: " + pulsefile)

    # Report the results
    result.stats.report()
    print("Final evolution\n{}\n".format(result.evo_full_final))
    print("********* Summary *****************")
    print("Final fidelity error {}".format(result.fid_err))
    print("Final gradient normal {}".format(result.grad_norm_final))
    print("Terminated due to {}".format(result.termination_reason))
    print("Number of iterations {}".format(result.num_iter))
    print("Completed in {} HH:MM:SS.US".format(
            datetime.timedelta(seconds=result.wall_time)))
    results.append(result)
    if i+1 < len(evo_times):
        # reconfigure the dynamics for the next evo time
        evo_time = evo_times[i+1]
        n_ts = int(float(evo_time) / dt)
        dyn.tau = None
        dyn.evo_time = evo_time
        dyn.num_tslots = n_ts
```

```python
evo = results[0].evo_full_final
evo.tidyup(1e-3)
# len(H_c)

```

### Plot the initial and final amplitudes

```python
fig1 = plt.figure(figsize=(30,8))
for i in range(n_evo_times):
    #Initial amps
    ax1 = fig1.add_subplot(2, n_evo_times, i+1)
    ax1.set_title("Init amps T={}".format(evo_times[i]))
    # ax1.set_xlabel("Time")
    ax1.get_xaxis().set_visible(False)
    if i == 0:
        ax1.set_ylabel("Control amplitude")
    for j in range(n_ctrls):
        ax1.step(results[i].time,
             np.hstack((results[i].initial_amps[:, j],
                        results[i].initial_amps[-1, j])),
                 where='post')

    ax2 = fig1.add_subplot(2, n_evo_times, i+n_evo_times+1)
    ax2.set_title("Final amps T={}".format(evo_times[i]))
    ax2.set_xlabel("Time")
    #Optimised amps
    if i == 0:
        ax2.set_ylabel("Control amplitude")
    for j in range(n_ctrls):
        ax2.step(results[i].time,
             np.hstack((results[i].final_amps[:, j],
                        results[i].final_amps[-1, j])),
                 where='post')

plt.tight_layout()
plt.show()
```

### Versions

```python
pulse_seq = results[0].final_amps
pulse_seq
```

### Run QuTip pulse simulation

```python
processor = Processor(N=1, dims=[3 ** len(subsystem_list)])

dt = backend.configuration().dt * 1e9

# H_d = wq0 * (1 - sigmaz()) / 2
tlist = np.array([dt * i for i in range(len(pulse_seq) + 1)])
for i,control in enumerate(H_c):
    processor.add_control(control, targets=0)
    coef = [a[i] for a in pulse_seq]

    processor.pulses[i].coeff =  coef
    processor.pulses[i].tlist = tlist

# processor.pulses[0].coeff = seq_x
# processor.pulses[0].tlist = np.array([dt * i for i in range(len(seq_x)+1)])
# processor.add_control(H_c[0], targets=[0], label='sigmax')
# processor.add_control(H_d, targets=[0], label="drift")
processor.add_drift(H_d, targets=[0])

basis0 = basis(3 ** len(subsystem_list), 0)
# basis0 = basis(4,0) + basis(4,2)
result = processor.run_state(init_state=basis0)

result.states[-1].tidyup(1.e-3)
```

### Create Channels and convert qoc pulses to Qiskit pulses

```python
q1=0
q2=1

drive_chan1 = pulse.DriveChannel(q1)
meas_chan1 = pulse.MeasureChannel(q1)
acq_chan1 = pulse.AcquireChannel(q1)
con_chan1 = pulse.ControlChannel(q1)

drive_chan2 = pulse.DriveChannel(q2)
meas_chan2 = pulse.MeasureChannel(q2)
acq_chan2 = pulse.AcquireChannel(q2)
con_chan2 = pulse.ControlChannel(q2)

```

    /Users/benrosand/anaconda3/envs/qoc/lib/python3.8/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.
      and should_run_async(code)

```python
pulse_seqs = []
for i,control in enumerate(H_c):
    coef = [a[i] for a in pulse_seq]
    pulse_seqs.append(coef)
```

```python
pulse_seq
D0 = []
D1 = []
for a in pulse_seq:
    # amp1 = derotate(a[0])
    # amp2 = derotate(a[1])
    D0.append(complex(a[0], a[1]))
    # D0.append(amp1, amp2)
    D1.append(complex(a[2], a[3]))

```

```python
schedule = pulse.Schedule(name='Frequency sweep')

# schedule += init_x

later = schedule.duration

# schedule += def_cx << later
schedule += Play(SamplePulse(D0),drive_chan1) << later
schedule += Play(SamplePulse(D1),drive_chan2) << later

from qiskit.pulse.macros import measure_all
schedule += measure_all(backend) << schedule.duration
```

```python
# Create the base schedule

# schedule.draw(plot_range=[0, 1000])

```

    /Users/benrosand/anaconda3/envs/qoc/lib/python3.8/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.
      and should_run_async(code)

### Run the assembled pulse schedule

```python
from qiskit import assemble

num_shots_per_frequency = 1024
frequency_sweep_program = assemble(schedule,
                                   backend=backend,
                                   meas_level=2,
                                   meas_return='single',
                                #    qubit_lo_freq=backend.defaults().qubit_freq_est,
                                   shots=num_shots_per_frequency,
                                   )
                                #    schedule_los=schedule_frequencies)

```

    /Users/benrosand/anaconda3/envs/qoc/lib/python3.8/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.
      and should_run_async(code)

```python
# job = backend.run(frequency_sweep_program)
```

    /Users/benrosand/anaconda3/envs/qoc/lib/python3.8/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.
      and should_run_async(code)

```python
from qiskit.tools.monitor import job_monitor
job_monitor(job)

# job.result().get_counts()
```

```python
from src.helper import qubit_distribution
qubit_distribution(job.result().get_counts())
```

```python
from qiskit import QuantumCircuit
```

    /Users/benrosand/anaconda3/envs/qoc/lib/python3.8/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.
      and should_run_async(code)

```python
circ = QuantumCircuit(2)
circ.cnot(0,1)
```

    /Users/benrosand/anaconda3/envs/qoc/lib/python3.8/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.
      and should_run_async(code)

```python

transp_circ = qiskit.transpile(circ, backend=backend)
transp_circ.data[0][0]
```

```python
processor = Processor(N=1, dims=[9])

dt = backend.configuration().dt * 1e9

# H_d = wq0 * (1 - sigmaz()) / 2
tlist = np.array([dt * i for i in range(len(def_seq[0]) + 1)])
for i,control in enumerate(H_c):
    processor.add_control(control, targets=0)
    # coef = [a[i] for a in pulse_seq]

    processor.pulses[i].coeff = def_seq[i]
    processor.pulses[i].tlist = tlist

# processor.pulses[0].coeff = seq_x
# processor.pulses[0].tlist = np.array([dt * i for i in range(len(seq_x)+1)])
# processor.add_control(H_c[0], targets=[0], label='sigmax')
# processor.add_control(H_d, targets=[0], label="drift")
processor.add_drift(H_d, targets=[0])

basis0 = basis(9, 0)
# basis0 = basis(4,0) + basis(4,2)
result = processor.run_state(init_state=basis0)

result.states[-1].tidyup(1.e-3)
```


# 朴素贝叶斯

作者：杨岱川

时间：2019年10月

github：https://github.com/DrDavidS/basic_Machine_Learning

开源协议：[MIT](https://github.com/DrDavidS/basic_Machine_Learning/blob/master/LICENSE)

## 朴素贝叶斯简介

朴素贝叶斯（naïve Bayes）法是基于贝叶斯定理和特征条件独立假设的分类方法。

朴素贝叶斯在20世纪60年代得到了广泛的研究，在60年代初被引入到文本检索领域（可能有的人会看过一些比较老的文章或者书，里面经常提到“使用朴素贝叶斯法来判断垃圾邮件”）。在统计学和计算机科学文献中，朴素贝叶斯模型也经常被叫做**简单贝叶斯（simple Bayes）**或者**独立贝叶斯（independence Bayes）**。

## 朴素贝叶斯的基本方法

### 基本定义

输入空间 $\Bbb X \subseteq \mathbf R^n$ 为 $n$ 维向量的集合，输出空间 $ \Bbb Y = \left \{ c_1,c_2,\cdots,c_K \right \} $ 为类标记的集合。

输入为特征向量 $x \in \Bbb X $，可以类比 iris 数据集的 features（特征），有多个维度；

输出为类标记（class label） $ y \in \Bbb Y$，可以类比为 iris 数据集的 labels（标签），可以是多分类的。

$X$ 是定义在输入空间 $\Bbb X $ 上的随机向量（一个实例的特征）， $Y$ 是定义在输出空间 $\Bbb Y $ 上的随机变量（一个实例的标签）。$P(X,Y)$ 是 $X$ 和 $Y$ 的联合概率分布。

训练数据集

$$\large T=\left \{(x_1,y_1),(x_2,y_2),\cdots,(x_N,y_N) \right \} $$

由 $P(X,Y)$ 独立同分布产生。

所以，朴素贝叶斯的基本思想就是通过训练数据集学习这个联合概率分布 $P(X,Y)$。

### 先验概率

**先验概率（prior probability）分布**如下：

$$\large P(Y=c_k), k=1,2,\cdots,K $$

何谓**先验概率分布**？查一下可以知道，先验概率是指根据以往经验和分析得到的概率。例如：

- 自然状况下，生男孩和生女孩的概率均是50%，这里，50%就是先验概率，不管是历史统计也好，生物学染色体理论解释也好，都指出这个概率是50%，这不需要我们再进行计算，可以直接使用。

- 根据历史统计，学校历届招生，男女比例均为 4:6，这时候来了一个新生报到，我们有理由猜测，这名新生是女生的概率为60%，这也是先验概率。

条件概率分布如下：

$$\large P(X=x|Y=c_k) = P(X^{(1)}=x^{(1)},\cdots,X^{(n)}=x^{(n)}|Y=c_k) ，\quad k = 1,2,\cdots,K$$

其中 $X^{(n)}$ 指的是实例X第$n$个特征，比如，在iris数据集中，$X^{(1)}$ 表示鸢尾花 $X$ 的第1个特征。

于是就学习到了联合概率分布$P(X,Y)$。

条件概率分布有指数级数量的参数，其估计实际是不可行的。在实际应用中，假设单独一个特征 $ x^{(j)} $ 可取值有 $S_j$个，$j=1,2,\cdots,n$，而标签 $Y$ 的取值有 $K$ 个，那么参数个数为 $K \cdot S_1 \cdot S_2 \cdots S_n$，即 $K \prod_{j=1}^n S_j$ 个参数。

## 何谓“朴素”

### 条件独立性假设

朴素贝叶斯对条件概率分布做了**条件独立性**的假设。由于这条假设非常重要，因此得名朴素贝叶斯。

>回顾一下**独立性**的定义：设 $A,B$ 为两个事件，如果其中任何一个事件发生的概率不受另外一个事件发生与否的影响，则称事件 $A$ 与 $B$ **相互独立**。

具体的，条件独立性假设是：

$$\large P(X=x|Y=c_k) = P(X^{(1)}=x^{(1)},\cdots,X^{(n)}=x^{(n)}|Y=c_k)=\prod_{j=1}^n P(X^{(j)}=x^{(j)}|Y=c_k)$$

即各个特征之间的分布都是独立的，或者说其特征之间的相关性为0。前面一节（2.4节）我们已经就特征之间的相关性做了分析，这里就不再赘述了。

### 优缺点

条件独立假设相当于用于分类的各个特征在类别确定的条件下都是条件独立的。这一假设使朴素贝叶斯法变得很简单，但是有时也会牺牲一定的分类准确率。

因为我们都知道很多情况下变量之间的相关性也很强，比如正常人的身高越高，体重一般也会越重，两者是有一定相关性的。

## 后验概率计算

朴素贝叶斯法分类时，对给定的输入 $x$ ，通过学习到的模型计算后验概率分布 $P(Y=c_k|X=x)$，然后将后验概率最大的类作为 $x$ 的类输出。

>考研时候概率论与数理统计肯定学过的，现在再复习一下**贝叶斯定理**（也叫**逆概率公式**）：
>
>如果 $ \bigcup_{i=1}^{n}A_i=\Omega $，$ A_iA_j=\emptyset(i \ne j;i,j=1,2,\cdots,n),P(A_i)>0 $，则对任一事件 $B$，只要 $P(B)>0$，就有
>
>$$\large P(A_j|B)=\frac{P(B|A_j)P(A_j)}{\sum_{i=1}^n P(B|A_i)P(A_i)}, \quad j=1,2,\cdots,n $$
>
>其中，
>
>- $A_iA_j=\emptyset(i \ne j;i,j=1,2,\cdots,n)$，指 $A_i$ 和 $A_j$ 事件**互不相容**，也叫**互斥**;
>
>- $\Omega$是**样本空间**，又叫**基本事件空间**，指的是随机试验的样本点 $\omega$ 的全体组成的集合；
>
>- 称“有限个事件$A_1,A_2,\cdots,A_n$至少有一个发生”的事件为事件$A_1,A_2,\cdots,A_n$的并（或和），记为$\bigcup^n_{i=1}A_i$;
>
>- 如果 $\bigcup^n_{i=1}A_i=\Omega$，则对一切 $(i \ne j;i,j=1,2,\cdots,n)$，有 $ A_iA_j=\emptyset $。
>

由贝叶斯定理：

$$ \large P(Y=c_k|X=x)=\frac{P(X=x|Y=c_k)P(Y=c_k)}{\sum_k P(X=x|Y=c_k)P(Y=c_k)} $$

其中，$ P(X=x|Y=c_k) $ 是条件概率（条件独立性假设中已经描述），$ P(Y=c_k) $是先验概率。

将条件独立性假设代入上式中，有：

$$ \large P(Y=c_k|X=x)=\frac{P(Y=c_k)\prod_j P(X^{(j)}=x^{(j)}|Y=c_k)}{\sum_k P(Y=c_k)\prod_j P(X^{(j)}=x^{(j)}|Y=c_k)}，\quad k = 1,2,\cdots,K $$

这就是朴素贝叶斯法分类的基本公式。故朴素贝叶斯分类器表示为：

$$ \large y=f(x)= {\rm argmax_{c_k}} \frac{P(Y=c_k)\prod_j P(X^{(j)}=x^{(j)}|Y=c_k)}{\sum_k P(Y=c_k)\prod_j P(X^{(j)}=x^{(j)}|Y=c_k)}
 $$

注意，在计算所有类别的后验概率中，分母都是相同的，只需要找令分子最大的参数值即可，故有：

$$ \large y=f(x)= {\rm argmax_{c_k}} {P(Y=c_k)\prod_j P(X^{(j)}=x^{(j)}|Y=c_k)} $$

## 极大似然估计

在朴素贝叶斯法中，学习（训练）就意味着估计先验概率 $ P(Y=c_k) $ 和条件概率分布 $ P(X^{(j)}=x^{(j)}|Y=c_k) $。

这里介绍极大似然估计法估计相应的概率。

---------

**先验概率** $P(Y=C_K)$ 的极大似然估计是

$$\large P(Y=c_k)=\frac{\sum^N_{i=1}I(y_i = c_k)}{N}, \quad k=1,2,\cdots,K $$

其中 $N$ 可以看做是有标签的训练样本的总数；$I$ 为指示函数。

>指示函数 $I(y_i=c_k)$，表示当 $y_i=c_k$ 的时候 $I$ 为1，否则 $I$ 为0。

举个通俗的例子，比如一个班学生 $N=50$ 人，男生有 $30$ 人，则 $\sum^{N}_{i=1}I(y_i = 男)=30$，则

$$ P(Y=男)=\frac{30}{N}=\frac{30}{50}=0.6 $$

---------

假设第 $j$ 个特征（第 $j$ 列）$x^{(j)}$ 可能的取值的集合为 $\{a_{j1},a_{j2},\cdots,a_{jS_j}\}$ ,**条件概率** $P(X^{(j)}=a_{jl}|Y=c_k)$ 的极大似然估计是

$$\large P(X^{(j)}=a_{jl}|Y=c_k)=\frac{\sum_{i=1}^N I(x_i^{(j)}=a_{jl},y_i=c_k)}{\sum_{i=1}^N I(y_i=c_k)} $$

$$\large j=1,2,\cdots,n; \quad l=1,2,\cdots,S_j; \quad k=1,2,\cdots,K$$

其中，$x_i^{(j)}$ 是第 $i$ 个样本的第 $j$ 个特征；$a_{jl}$ 是第 $j$ 个特征可能取的第 $l$ 个值；$I$ 为指示函数。

接上面的例子，比如学生的第一个特征$x^{(1)}$为“是否戴眼镜”，取值范围就是“$a_{11}=戴眼镜$”或者“$a_{12}=不戴眼镜$”，实际上班上有40个学生戴眼镜，其中25个是男生，则：

$$
\begin{equation}\begin{split}
\large P(X^{(1)}=戴眼镜|Y=男)
&=\large \frac{\sum_{i=1}^N I(x_i^{(1)}=戴眼镜,y_i=男)}{\sum_{i=1}^N I(y_i=男)}\\
&=\large \frac{25}{30}\\
&=\large \frac{5}{6}
\end{split}\end{equation}
$$

## 算法流程

输入：训练数据 $T=\left \{(x_1,y_1),(x_2,y_2),\cdots,(x_N,y_N) \right \}$，其中：

- $x_i=(x_i^{(1)},x_i^{(2)},\cdots,x_i^{(n)})^T$，$x_i^{(j)}$ 是第 $i$ 个样本的第 $j$ 个特征；
- $x_i^{(j)}\in \{ a_{j1},a_{j2},\cdots,a_{js_j}\}$，$a_{jl}$是第 $j$ 个特征可能取的第 $l$ 个值，$j=1,2,\cdots,n$，$l=1,2,\cdots,S_j$；
- 类别 $y_i\in {c_1,c_2,\cdots,c_K}$，比如有鸢尾花数据集的类别有三种；
- $x$ 是实例。

输出：实例 $x$ 得分类。

---------

首先，计算先验概率：

$$\large P(Y=c_k)=\frac{\sum^N_{i=1}I(y_i = c_k)}{N}, \quad k=1,2,\cdots,K $$

再拿我们熟悉的iris数据集举例，总共150个鸢尾花实例，所以 $N=150$，

那么属于第一类鸢尾花的先验概率就是 $P(Y=c_1)=\frac{50}{150}=0.333$

---------

其次，计算条件概率：

$$\large P(X^{(j)}=a_{jl}|Y=c_k)=\frac{\sum_{i=1}^N I(x_i^{(j)}=a_{jl},y_i=c_k)}{\sum_{i=1}^N I(y_i=c_k)} $$

$$\large j=1,2,\cdots,n; \quad l=1,2,\cdots,S_j, \quad k=1,2,\cdots,K$$

这里指的是，在花的种类是第一类的情况下，数据集第1类（列）特征 $X^{(1)}$，假设是花萼长度，取值为 $a_{jl}$ 的概率。这里花萼长度是一个实数，单位是厘米。

--------

然后，对于给定的实例 $x=(x_i^{(1)},x_i^{(2)},\cdots,x_i^{(n)})^T$，计算

$$\large {P(Y=c_k)\prod_{j=1}^n P(X^{(j)}=x^{(j)}|Y=c_k)}, \quad k=1,2,\cdots,K$$

这里计算的相当于是某一朵鸢尾花 $x$ 分类到每个类别 $c_k, \quad k=1,2,\cdots,K$ 的概率。

--------

最后，确定实例 $x$ 的类：

$$ \large y=f(x)= {\rm argmax_{c_k}} {P(Y=c_k)\prod_j P(X^{(j)}=x^{(j)}|Y=c_k)} $$

相当于选择出这一朵花 $x$ 对应的概率最大的分类。

## 贝叶斯估计

简单说一下贝叶斯估计。

极大似然估计由于数据的问题，可能存在所要估计的概率值为 $0$ 的情况，因此会影响到后验概率的计算结果，有可能让分类产生偏差。

具体的：

$$ \large P(X^{(j)}=a_{jl}|Y=c_k)=\frac{\sum_{i=1}^N I(x_i^{(j)}=a_{jl},y_i=c_k) + \lambda}{\sum_{i=1}^N I(y_i=c_k) + S_j\lambda}  $$

式中，$ \lambda \ge 0$。

明显地，当 $\lambda = 0$ 时候，就是极大似然估计。而我们常常取 $\lambda = 1$，称之为拉普拉斯平滑。

## 朴素贝叶斯的Sklearn实现

依旧使用我们熟悉的 Sklearn 来实现朴素贝叶斯方法。

Sklearn中内置了多种朴素贝叶斯实现，例如：

- **Gaussian Naive Bayes** : 假设特征是符合高斯分布的。
- **Multinomial Naive Bayes** : 多项式分布数据的朴素贝叶斯，多用于文本分类。
- **Bernoulli Naive Bayes** ： 针对多元伯努利分布数据实现的朴素贝叶斯训练和分类算法，即多个特征均为bool形式的变量。

在常见领域，比如故障诊断或者前面最常用的鸢尾花数据集中，大多数情况下采集的数据都是符合高斯分布的，所以我们就使用 `GaussianNB` 作为例子。

### 导入数据

和以前一样，导入鸢尾花数据集。

```python
import matplotlib.pyplot as plt  # 绘图
import numpy as np  # numpy
import pandas as pd
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
%matplotlib inline

# 导入iris数据集，转换为 dataframe 格式
iris = load_iris()
df = pd.DataFrame(iris.data, columns=iris.feature_names)
df['label'] = iris.target
df.columns = ['sepal length', 'sepal width', 'petal length', 'petal width', 'label']
```

```python
df
```

      sepal length
      sepal width
      petal length
      petal width
      label

      0
      5.1
      3.5
      1.4
      0.2
      0

      1
      4.9
      3.0
      1.4
      0.2
      0

      2
      4.7
      3.2
      1.3
      0.2
      0

      3
      4.6
      3.1
      1.5
      0.2
      0

      4
      5.0
      3.6
      1.4
      0.2
      0

      ...
      ...
      ...
      ...
      ...
      ...

      145
      6.7
      3.0
      5.2
      2.3
      2

      146
      6.3
      2.5
      5.0
      1.9
      2

      147
      6.5
      3.0
      5.2
      2.0
      2

      148
      6.2
      3.4
      5.4
      2.3
      2

      149
      5.9
      3.0
      5.1
      1.8
      2

150 rows × 5 columns

### 导入高斯朴素贝叶斯模块

模块如下，参考文档点[这里](https://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.GaussianNB.html#sklearn.naive_bayes.GaussianNB)

```python
class sklearn.naive_bayes.GaussianNB(priors=None, var_smoothing=1e-09)
```

同时准备训练集和测试集，开始训练

```python
from sklearn.naive_bayes import GaussianNB

# 筛选两类鸢尾花
data = np.array(df.iloc[:100, [0, 1, -1]])

# 拆分为 features 和 labels
X, y = data[:,:-1], data[:,-1]

# 随机划分训练集和测试集
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)  # 二八划分
```

```python
# 模型训练
gnb = GaussianNB()

gnb.fit(X_train, y_train)
```

    GaussianNB(priors=None, var_smoothing=1e-09)

### 模型预测与评估

使用刚刚训练好的 `gnb` 实例进行预测。

```python
y_pred = gnb.predict(X_test)
```

```python
y_pred
```

    array([0., 0., 0., 1., 1., 1., 0., 1., 0., 1., 0., 0., 0., 1., 1., 1., 0.,
           0., 0., 1.])

```python
y_test
```

    array([0., 0., 0., 1., 1., 1., 0., 1., 0., 1., 0., 0., 0., 1., 1., 1., 0.,
           0., 0., 1.])

`GaussianNB` 自带的 `score` 评估模型准确率：

```python
gnb.score(X_test, y_test)
```

    1.0

Sklearn 也有其他准确率评估方法，比如 `sklearn.metrics.accuracy_score` 方法：

```python
from sklearn.metrics import accuracy_score

accuracy_score(y_test, y_pred)
```

    1.0

```python

```


```python
import sys
sys.executable
```

    '/anaconda2/bin/python'

```python
%matplotlib inline

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

torch.manual_seed(1)

lin = nn.Linear(5, 3)  # maps from R^5 to R^3, parameters A, b
# data is 2x5.  A maps from 5 to 3... can we map "data" under A?
data = torch.randn(2, 5)
print(lin(data))  # yes
```

```python

```

Non-Linearities
~~~~~~~~~~~~~~~

First, note the following fact, which will explain why we need
non-linearities in the first place. Suppose we have two affine maps
$f(x) = Ax + b$ and $g(x) = Cx + d$. What is
$f(g(x))$?

\begin{align}f(g(x)) = A(Cx + d) + b = ACx + (Ad + b)\end{align}

$AC$ is a matrix and $Ad + b$ is a vector, so we see that
composing affine maps gives you an affine map.

From this, you can see that if you wanted your neural network to be long
chains of affine compositions, that this adds no new power to your model
than just doing a single affine map.

If we introduce non-linearities in between the affine layers, this is no
longer the case, and we can build much more powerful models.

There are a few core non-linearities.
$\tanh(x), \sigma(x), \text{ReLU}(x)$ are the most common. You are
probably wondering: "why these functions? I can think of plenty of other
non-linearities." The reason for this is that they have gradients that
are easy to compute, and computing gradients is essential for learning.
For example

\begin{align}\frac{d\sigma}{dx} = \sigma(x)(1 - \sigma(x))\end{align}

A quick note: although you may have learned some neural networks in your
intro to AI class where $\sigma(x)$ was the default non-linearity,
typically people shy away from it in practice. This is because the
gradient *vanishes* very quickly as the absolute value of the argument
grows. Small gradients means it is hard to learn. Most people default to
tanh or ReLU.

```python
# In pytorch, most non-linearities are in torch.functional (we have it imported as F)
# Note that non-linearites typically don't have parameters like affine maps do.
# That is, they don't have weights that are updated during training.
data = torch.randn(2, 2)
print(data)
print(F.relu(data))
```

Softmax and Probabilities
~~~~~~~~~~~~~~~~~~~~~~~~~

The function $\text{Softmax}(x)$ is also just a non-linearity, but
it is special in that it usually is the last operation done in a
network. This is because it takes in a vector of real numbers and
returns a probability distribution. Its definition is as follows. Let
$x$ be a vector of real numbers (positive, negative, whatever,
there are no constraints). Then the i'th component of
$\text{Softmax}(x)$ is

\begin{align}\frac{\exp(x_i)}{\sum_j \exp(x_j)}\end{align}

It should be clear that the output is a probability distribution: each
element is non-negative and the sum over all components is 1.

You could also think of it as just applying an element-wise
exponentiation operator to the input to make everything non-negative and
then dividing by the normalization constant.

```python
# Softmax is also in torch.nn.functional
data = torch.randn(5)
print(data)
print(F.softmax(data, dim=0))
print(F.softmax(data, dim=0).sum())  # Sums to 1 because it is a distribution!
print(F.log_softmax(data, dim=0))  # theres also log_softmax
```

Objective Functions
~~~~~~~~~~~~~~~~~~~

The objective function is the function that your network is being
trained to minimize (in which case it is often called a *loss function*
or *cost function*). This proceeds by first choosing a training
instance, running it through your neural network, and then computing the
loss of the output. The parameters of the model are then updated by
taking the derivative of the loss function. Intuitively, if your model
is completely confident in its answer, and its answer is wrong, your
loss will be high. If it is very confident in its answer, and its answer
is correct, the loss will be low.

The idea behind minimizing the loss function on your training examples
is that your network will hopefully generalize well and have small loss
on unseen examples in your dev set, test set, or in production. An
example loss function is the *negative log likelihood loss*, which is a
very common objective for multi-class classification. For supervised
multi-class classification, this means training the network to minimize
the negative log probability of the correct output (or equivalently,
maximize the log probability of the correct output).

Optimization and Training
=========================

So what we can compute a loss function for an instance? What do we do
with that? We saw earlier that Tensors know how to compute gradients
with respect to the things that were used to compute it. Well,
since our loss is an Tensor, we can compute gradients with
respect to all of the parameters used to compute it! Then we can perform
standard gradient updates. Let $\theta$ be our parameters,
$L(\theta)$ the loss function, and $\eta$ a positive
learning rate. Then:

\begin{align}\theta^{(t+1)} = \theta^{(t)} - \eta \nabla_\theta L(\theta)\end{align}

There are a huge collection of algorithms and active research in
attempting to do something more than just this vanilla gradient update.
Many attempt to vary the learning rate based on what is happening at
train time. You don't need to worry about what specifically these
algorithms are doing unless you are really interested. Torch provides
many in the torch.optim package, and they are all completely
transparent. Using the simplest gradient update is the same as the more
complicated algorithms. Trying different update algorithms and different
parameters for the update algorithms (like different initial learning
rates) is important in optimizing your network's performance. Often,
just replacing vanilla SGD with an optimizer like Adam or RMSProp will
boost performance noticably.

Creating Network Components in PyTorch
======================================

Before we move on to our focus on NLP, lets do an annotated example of
building a network in PyTorch using only affine maps and
non-linearities. We will also see how to compute a loss function, using
PyTorch's built in negative log likelihood, and update parameters by
backpropagation.

All network components should inherit from nn.Module and override the
forward() method. That is about it, as far as the boilerplate is
concerned. Inheriting from nn.Module provides functionality to your
component. For example, it makes it keep track of its trainable
parameters, you can swap it between CPU and GPU with the ``.to(device)``
method, where device can be a CPU device ``torch.device("cpu")`` or CUDA
device ``torch.device("cuda:0")``.

Let's write an annotated example of a network that takes in a sparse
bag-of-words representation and outputs a probability distribution over
two labels: "English" and "Spanish". This model is just logistic
regression.

Example: Logistic Regression Bag-of-Words classifier
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Our model will map a sparse BoW representation to log probabilities over
labels. We assign each word in the vocab an index. For example, say our
entire vocab is two words "hello" and "world", with indices 0 and 1
respectively. The BoW vector for the sentence "hello hello hello hello"
is

\begin{align}\left[ 4, 0 \right]\end{align}

For "hello world world hello", it is

\begin{align}\left[ 2, 2 \right]\end{align}

etc. In general, it is

\begin{align}\left[ \text{Count}(\text{hello}), \text{Count}(\text{world}) \right]\end{align}

Denote this BOW vector as $x$. The output of our network is:

\begin{align}\log \text{Softmax}(Ax + b)\end{align}

That is, we pass the input through an affine map and then do log
softmax.

```python
data = [("me gusta comer en la cafeteria".split(), "SPANISH"),
        ("Give it to me".split(), "ENGLISH"),
        ("No creo que sea una buena idea".split(), "SPANISH"),
        ("No it is not a good idea to get lost at sea".split(), "ENGLISH")]

test_data = [("Yo creo que si".split(), "SPANISH"),
             ("it is lost on me".split(), "ENGLISH")]

# word_to_ix maps each word in the vocab to a unique integer, which will be its
# index into the Bag of words vector
word_to_ix = {}
for sent, _ in data + test_data:
    for word in sent:
        if word not in word_to_ix:
            word_to_ix[word] = len(word_to_ix)
print(word_to_ix)

VOCAB_SIZE = len(word_to_ix)
NUM_LABELS = 2

class BoWClassifier(nn.Module):  # inheriting from nn.Module!

    def __init__(self, num_labels, vocab_size):
        # calls the init function of nn.Module.  Dont get confused by syntax,
        # just always do it in an nn.Module
        super(BoWClassifier, self).__init__()

        # Define the parameters that you will need.  In this case, we need A and b,
        # the parameters of the affine mapping.
        # Torch defines nn.Linear(), which provides the affine map.
        # Make sure you understand why the input dimension is vocab_size
        # and the output is num_labels!
        self.linear = nn.Linear(vocab_size, num_labels)

        # NOTE! The non-linearity log softmax does not have parameters! So we don't need
        # to worry about that here

    def forward(self, bow_vec):
        # Pass the input through the linear layer,
        # then pass that through log_softmax.
        # Many non-linearities and other functions are in torch.nn.functional
        return F.log_softmax(self.linear(bow_vec), dim=1)

def make_bow_vector(sentence, word_to_ix):
    vec = torch.zeros(len(word_to_ix))
    for word in sentence:
        vec[word_to_ix[word]] += 1
    return vec.view(1, -1)

def make_target(label, label_to_ix):
    return torch.LongTensor([label_to_ix[label]])

model = BoWClassifier(NUM_LABELS, VOCAB_SIZE)

# the model knows its parameters.  The first output below is A, the second is b.
# Whenever you assign a component to a class variable in the __init__ function
# of a module, which was done with the line
# self.linear = nn.Linear(...)
# Then through some Python magic from the PyTorch devs, your module
# (in this case, BoWClassifier) will store knowledge of the nn.Linear's parameters
for param in model.parameters():
    print(param)

# To run the model, pass in a BoW vector
# Here we don't need to train, so the code is wrapped in torch.no_grad()
with torch.no_grad():
    sample = data[0]
    bow_vector = make_bow_vector(sample[0], word_to_ix)
    log_probs = model(bow_vector)
    print(log_probs)
```

Which of the above values corresponds to the log probability of ENGLISH,
and which to SPANISH? We never defined it, but we need to if we want to
train the thing.

```python
label_to_ix = {"SPANISH": 0, "ENGLISH": 1}
```

So lets train! To do this, we pass instances through to get log
probabilities, compute a loss function, compute the gradient of the loss
function, and then update the parameters with a gradient step. Loss
functions are provided by Torch in the nn package. nn.NLLLoss() is the
negative log likelihood loss we want. It also defines optimization
functions in torch.optim. Here, we will just use SGD.

Note that the *input* to NLLLoss is a vector of log probabilities, and a
target label. It doesn't compute the log probabilities for us. This is
why the last layer of our network is log softmax. The loss function
nn.CrossEntropyLoss() is the same as NLLLoss(), except it does the log
softmax for you.

```python
# Run on test data before we train, just to see a before-and-after
with torch.no_grad():
    for instance, label in test_data:
        bow_vec = make_bow_vector(instance, word_to_ix)
        log_probs = model(bow_vec)
        print(log_probs)

# Print the matrix column corresponding to "creo"
print(next(model.parameters())[:, word_to_ix["creo"]])

loss_function = nn.NLLLoss()
optimizer = optim.SGD(model.parameters(), lr=0.1)

# Usually you want to pass over the training data several times.
# 100 is much bigger than on a real data set, but real datasets have more than
# two instances.  Usually, somewhere between 5 and 30 epochs is reasonable.
for epoch in range(100):
    for instance, label in data:
        # Step 1. Remember that PyTorch accumulates gradients.
        # We need to clear them out before each instance
        model.zero_grad()

        # Step 2. Make our BOW vector and also we must wrap the target in a
        # Tensor as an integer. For example, if the target is SPANISH, then
        # we wrap the integer 0. The loss function then knows that the 0th
        # element of the log probabilities is the log probability
        # corresponding to SPANISH
        bow_vec = make_bow_vector(instance, word_to_ix)
        target = make_target(label, label_to_ix)

        # Step 3. Run our forward pass.
        log_probs = model(bow_vec)

        # Step 4. Compute the loss, gradients, and update the parameters by
        # calling optimizer.step()
        loss = loss_function(log_probs, target)
        loss.backward()
        optimizer.step()

with torch.no_grad():
    for instance, label in test_data:
        bow_vec = make_bow_vector(instance, word_to_ix)
        log_probs = model(bow_vec)
        print(log_probs)

# Index corresponding to Spanish goes up, English goes down!
print(next(model.parameters())[:, word_to_ix["creo"]])
```

We got the right answer! You can see that the log probability for
Spanish is much higher in the first example, and the log probability for
English is much higher in the second for the test data, as it should be.

Now you see how to make a PyTorch component, pass some data through it
and do gradient updates. We are ready to dig deeper into what deep NLP
has to offer.