## Data Analytics

### Interactive Distribution Transformations in Python

#### Michael Pyrcz, Associate Professor, University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

### Data Analytics: Distribution Transformations

Here's a demonstration of making and general use of distribution transformations in Python. This demonstration is part of the resources that I include for my courses in Spatial / Subsurface Data Analytics at the Cockrell School of Engineering at the University of Texas at Austin.

#### Distribution Transformations

Why do we do this?

* **Inference**: variable has expected shape

* **Data Preparation / Cleaning**: correcting for too few data and outliers

* **Theory**: a specific distribution assumption required for a method

How do we do it?

We apply this to all sample data, $x_{\alpha}$ $\forall$ $\alpha = 1,\ldots,n$.

\begin{equation}
y_{\alpha} = G^{-1}_Y\left(F_X(x_{\alpha})\right)
\end{equation}

were $X$ is the original feature with a $F_X$ original cumulative distribution function and $Y$ is transformed feature with a $G_Y$ transformed cumulative distribution function.

* Mapping from one distribution to another through percentiles

* This may be applied to any parametric or nonparametric distributions

* This is a rank preserving transform, e.g. P50 of 𝑋 is P50 of 𝑌

I have a lecture on distribution transformations available on [YouTube](https://www.youtube.com/watch?v=ZDIpE3OkAIU&list=PLG19vXLQHvSB-D4XKYieEku9GQMQyAzjJ&index=14).

#### Getting Started

Here's the steps to get setup in Python with the GeostatsPy package:

1. Install Anaconda 3 on your machine (https://www.anaconda.com/download/).
2. From Anaconda Navigator (within Anaconda3 group), go to the environment tab, click on base (root) green arrow and open a terminal.
3. In the terminal type: pip install geostatspy.
4. Open Jupyter and in the top block get started by copy and pasting the code block below from this Jupyter Notebook to start using the geostatspy functionality.

You will need to copy the data file to your working directory.  They are available here:

* Tabular data - [sample_data.csv](https://github.com/GeostatsGuy/GeoDataSets/blob/master/sample_data.csv).

#### Importing Packages

We will need some standard packages. These should have been installed with Anaconda 3.

```python
import numpy as np                        # ndarrys for gridded data
import pandas as pd                       # DataFrames for tabular data
import matplotlib.pyplot as plt           # plotting
from scipy import stats                   # summary statistics
import math                               # trigonometry etc.
import random                             # randon numbers
from scipy.stats import norm              # Gaussian parametric distribution
import geostatspy.GSLIB as GSLIB
from ipywidgets import interactive                      # widgets and interactivity
from ipywidgets import widgets
from ipywidgets import Layout
from ipywidgets import Label
from ipywidgets import VBox, HBox
```

#### Set the Random Number Seed

Set the random number seed so that we have a repeatable workflow

```python
seed = 73073
```

#### Loading Tabular Data

Here's the command to load our comma delimited data file in to a Pandas' DataFrame object.  For fun try misspelling the name. You will get an ugly, long error.

```python
data_url = "https://raw.githubusercontent.com/GeostatsGuy/GeoDataSets/master/sample_data.csv"
df = pd.read_csv(data_url       )     # load our data table (wrong name!)
```

It worked, we loaded our file into our DataFrame called 'df'. But how do you really know that it worked? Visualizing the DataFrame would be useful and we already leard about these methods in this demo (https://git.io/fNgRW).

We can preview the DataFrame by printing a slice or by utilizing the 'head' DataFrame member function (with a nice and clean format, see below). With the slice we could look at any subset of the data table and with the head command, add parameter 'n=13' to see the first 13 rows of the dataset.

```python
df.head(n=6)                           # we could also use this command for a table preview
```

      X
      Y
      Facies
      Porosity
      Perm
      AI

      0
      100.0
      900.0
      1.0
      0.100187
      1.363890
      5110.699751

      1
      100.0
      800.0
      0.0
      0.107947
      12.576845
      4671.458560

      2
      100.0
      700.0
      0.0
      0.085357
      5.984520
      6127.548006

      3
      100.0
      600.0
      0.0
      0.108460
      2.446678
      5201.637996

      4
      100.0
      500.0
      0.0
      0.102468
      1.952264
      3835.270322

      5
      100.0
      400.0
      0.0
      0.110579
      3.691908
      5295.267191

#### Calculating and Plotting a CDF by Hand

Let's demonstrate the calculation and plotting of a non-parametric CDF by hand

1. make a copy of the feature as a 1D array (ndarray from NumPy)
2. sort the data in ascending order
3. assign cumulative probabilities based on the tail assumptions
4. plot cumuative probability vs. value

```python
por = df['Porosity'].copy(deep = True).values # make a deepcopy of the feature from the DataFrame
print('The ndarray has a shape of ' + str(por.shape) + '.')

por = np.sort(por)                           # sort the data in ascending order
n = por.shape[0]                             # get the number of data samples

cprob = np.zeros(n)
for i in range(0,n):
    index = i + 1
    cprob[i] = index / n                     # known upper tail
    # cprob[i] = (index - 1)/n               # known lower tail
    # cprob[i] = (index - 1)/(n - 1)         # known upper and lower tails
    # cprob[i] = index/(n+1)                 # unknown tails

plt.subplot(111)
plt.plot(por,cprob, alpha = 0.2, c = 'black') # plot piecewise linear interpolation
plt.scatter(por,cprob,s = 10, alpha = 1.0, c = 'red', edgecolor = 'black') # plot the CDF points
plt.grid(); plt.xlim([0.05,0.25]); plt.ylim([0.0,1.0])
plt.xlabel("Porosity (fraction)"); plt.ylabel("Cumulative Probability"); plt.title("Non-parametric Porosity Cumulative Distribution Function")

plt.subplots_adjust(left=0.0, bottom=0.0, right=1.0, top=1.2, wspace=0.1, hspace=0.2)
plt.show()
```

#### Transformation to a Parametric Distribution

We can transform our data feature distribution to any parametric distribution with this workflow.

1. Calculate the cumulative probability value of each of our data values, $p_{\alpha} = F_x(x_\alpha)$, $\forall$ $\alpha = 1,\ldots, n$.

2. Apply the inverse of the target parametric cumulative distribution function (CDF) to calculate the transformed values. $y_{\alpha} = G_y^{-1}\left(F_x(x_\alpha)\right)$, $\forall$ $\alpha = 1,\ldots, n$.

```python
y = np.zeros(n)

for i in range(0,n):
    y[i] = norm.ppf(cprob[i],loc=0.0,scale=1.0)

plt.subplot(121)
plt.plot(por,cprob, alpha = 0.2, c = 'black') # plot piecewise linear interpolation
plt.scatter(por,cprob,s = 10, alpha = 1.0, c = 'red', edgecolor = 'black') # plot the CDF points
plt.grid(); plt.xlim([0.05,0.25]); plt.ylim([0.0,1.0])
plt.xlabel("Porosity (fraction)"); plt.ylabel("Cumulative Probability"); plt.title("Non-parametric Porosity Cumulative Distribution Function")

plt.subplot(122)
plt.plot(y,cprob, alpha = 0.2, c = 'black') # plot piecewise linear interpolation
plt.scatter(y,cprob,s = 10, alpha = 1.0, c = 'red', edgecolor = 'black') # plot the CDF points
plt.grid(); plt.xlim([-3.0,3.0]); plt.ylim([0.0,1.0])
plt.xlabel("Porosity (fraction)"); plt.ylabel("Cumulative Probability"); plt.title("After Distribution Transformation to Gaussian")

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.2, wspace=0.2, hspace=0.2)
plt.show()

```

Let's make an interactive version of this plot to visualize the transformation.

```python
# widgets and dashboard
l = widgets.Text(value='                                                  Data Analytics, Distribution Transformation, Prof. Michael Pyrcz, The University of Texas at Austin',layout=Layout(width='950px', height='30px'))

data_index = widgets.IntSlider(min=1, max = n-1, value=1.0, step = 10.0, description = 'Data Index, $\\alpha$',orientation='horizontal', style = {'description_width': 'initial'}, continuous_update=False)

ui = widgets.VBox([l,data_index],)

def run_plot(data_index):                       # make data, fit models and plot
    plt.subplot(131)
    plt.plot(por,cprob, alpha = 0.2, c = 'black') # plot piecewise linear interpolation
    plt.scatter(por,cprob,s = 10, alpha = 1.0, c = 'red', edgecolor = 'black') # plot the CDF points
    plt.grid(); plt.xlim([0.05,0.25]); plt.ylim([0.0,1.0])
    plt.xlabel("Original Feature, $x$"); plt.ylabel("Cumulative Probability"); plt.title("Original Cumulative Distribution Function")
    plt.plot([por[data_index-1],por[data_index-1]],[0.0,cprob[data_index-1]],color = 'red',linestyle='dashed')
    plt.plot([por[data_index-1],3.0],[cprob[data_index-1],cprob[data_index-1]],color = 'red',linestyle='dashed')
    plt.annotate('x = ' + str(round(por[data_index-1],2)), xy=(por[data_index-1]+0.003, 0.01))
    plt.annotate('p = ' + str(round(cprob[data_index-1],2)), xy=(0.225, cprob[data_index-1]+0.02))

    plt.subplot(132)
    plt.plot(y,cprob, alpha = 0.2, c = 'black') # plot piecewise linear interpolation
    plt.scatter(y,cprob,s = 10, alpha = 1.0, c = 'red', edgecolor = 'black') # plot the CDF points
    plt.grid(); plt.xlim([-3.0,3.0]); plt.ylim([0.0,1.0])
    plt.xlabel("Gaussian Transformed Feature, $y$"); plt.ylabel("Cumulative Probability"); plt.title("After Distribution Transformation to Gaussian")
    plt.plot([-3.0,y[data_index-1]],[cprob[data_index-1],cprob[data_index-1]],color = 'red',linestyle='dashed')
    plt.plot([y[data_index-1],y[data_index-1]],[0.0,cprob[data_index-1]],color = 'red',linestyle='dashed')
    #plt.arrow(y[data_index-1],cprob[data_index-1],0.0,-1.0*(cprob[data_index-1]-0.01),color = 'red',width = 0.02, head_width = 0.1, linestyle='dashed', head_length = 0.01)
    plt.annotate('p = ' + str(round(cprob[data_index-1],2)), xy=(-2.90, cprob[data_index-1]+0.02))
    plt.annotate('y = ' + str(round(y[data_index-1],2)), xy=(y[data_index-1]+0.1, 0.01))

    plt.subplot(133)
    plt.plot(por,y, alpha = 0.2, c = 'black') # plot piecewise linear interpolation
    plt.grid(); plt.xlim([0.05,0.25]); plt.ylim([-3.0,3.0])
    plt.xlabel("Original Porosity (fraction)"); plt.ylabel("Gaussian Transformed Porosity (N[fraction])"); plt.title("Parametric Distribution Transformation, Q-Q Plot")
    #plt.plot([0.05,0.25],[0.05,0.25],color = 'red',linestyle='dashed', alpha = 0.4)
    plt.scatter(por[data_index-1],y[data_index-1],s = 50, c = 'red', edgecolor = 'black', alpha = 1.0, zorder=200) # plot the CDF points
    plt.scatter(por,y,s = 20, c = 'red', edgecolor = 'black', alpha = 0.1, zorder=100) # plot the CDF points

    plt.subplots_adjust(left=0.0, bottom=0.0, right=3.0, top=1.2, wspace=0.2, hspace=0.2)
    plt.show()

# connect the function to make the samples and plot to the widgets
interactive_plot = widgets.interactive_output(run_plot, {'data_index':data_index})
interactive_plot.clear_output(wait = True)               # reduce flickering by delaying plot updating
```

### Interactive Data Analytics Distribution Transformation Demonstration

#### Michael Pyrcz, Associate Professor, The University of Texas at Austin

Select any data value and observe the distribution transform by mapping through cumulative probability.

### The Inputs

* **data_index** - the data index from 1 to n in the sorted ascending order

```python
display(ui, interactive_plot)                           # display the interactive plot
```

    VBox(children=(Text(value='                                                  Data Analytics, Distribution Tran…

    Output(outputs=({'output_type': 'display_data', 'data': {'text/plain': '
', 'i…

#### Distribution Transform to a Non-Parametric Distribution

We can apply the mapping through cumulative probabilities to transform from any distribution to any other distribution.

* let's make a new data set by randomly sampling from the previous one and adding error

Then we can demonstrate transforming this dataset to match the original distribution

* this is mimicking the situation where we transform a dataset to match the distribution of a better sampled analog distribution

```python
n_sample = 30
df_sample = df.sample(n_sample,random_state = seed)

df_sample = df_sample.copy(deep = True) # make a deepcopy of the feature from the DataFrame

df_sample['Porosity'] = df_sample['Porosity'].values + np.random.normal(loc = 0.0, scale = 0.01, size = n_sample)

df_sample = df_sample.sort_values(by = 'Porosity')                # sort the DataFrame
por_sample = df_sample['Porosity'].values
print('The sample ndarray has a shape of ' + str(por_sample.shape) + '.')

cprob_sample = np.zeros(n_sample)
for i in range(0,n_sample):
    index = i + 1
    cprob_sample[i] = index / n_sample       # known upper tail
    # cprob[i] = (index - 1)/n               # known lower tail
    # cprob[i] = (index - 1)/(n - 1)         # known upper and lower tails
    # cprob[i] = index/(n+1)                 # unknown tails

plt.subplot(121)
plt.plot(por_sample,cprob_sample, alpha = 0.2, c = 'black') # plot piecewise linear interpolation
plt.scatter(por_sample,cprob_sample,s = 10, alpha = 1.0, c = 'red', edgecolor = 'black') # plot the CDF points
plt.grid(); plt.xlim([0.05,0.25]); plt.ylim([0.0,1.0])
plt.xlabel("Porosity (fraction)"); plt.ylabel("Cumulative Probability"); plt.title("Sparse Sample with Noise Cumulative Distribution Function")

plt.subplot(122)
plt.plot(por,cprob, alpha = 0.2, c = 'black') # plot piecewise linear interpolation
plt.scatter(por,cprob,s = 10, alpha = 1.0, c = 'red', edgecolor = 'black') # plot the CDF points
plt.grid(); plt.xlim([0.05,0.25]); plt.ylim([0.0,1.0])
plt.xlabel("Porosity (fraction)"); plt.ylabel("Cumulative Probability"); plt.title("Non-parametric Porosity Cumulative Distribution Function")

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.2, wspace=0.2, hspace=0.2)
plt.show()
```

Let's transform the values and show them on the target distribution.

```python
y_sample = np.zeros(n_sample)

for i in range(0,n_sample):
    y_sample[i] = np.percentile(por,cprob_sample[i]*100, interpolation = 'linear') # piecewise linear interpolation of inverse of target CDF

plt.subplot(121)
plt.plot(por_sample,cprob_sample, alpha = 0.2, c = 'black') # plot piecewise linear interpolation
plt.scatter(por_sample,cprob_sample,s = 30, alpha = 1.0, c = 'green', edgecolor = 'black', zorder = 100) # plot the CDF points
plt.grid(); plt.xlim([0.05,0.25]); plt.ylim([0.0,1.0])
plt.xlabel("Porosity (fraction)"); plt.ylabel("Cumulative Probability"); plt.title("Sparse Sample with Noise Cumulative Distribution Function")

plt.subplot(122)
plt.plot(por,cprob, alpha = 0.2, c = 'black') # plot piecewise linear interpolation
plt.scatter(por,cprob,s = 10, c = 'red', edgecolor = 'black', alpha = 0.3) # plot the CDF points
plt.scatter(y_sample,cprob_sample,s = 30, c = 'green', edgecolor = 'black', alpha = 1.0, zorder = 100) # plot the CDF points
plt.grid(); plt.xlim([0.05,0.25]); plt.ylim([0.0,1.0])
plt.xlabel("Porosity (fraction)"); plt.ylabel("Cumulative Probability"); plt.title("Non-parametric Porosity Cumulative Distribution Function")

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.2, wspace=0.2, hspace=0.2)
plt.show()
```

Let's make an interactive version of this plot to visualize the transformation.

```python
# widgets and dashboard
l_sample = widgets.Text(value='                                                  Data Analytics, Distribution Transformation, Prof. Michael Pyrcz, The University of Texas at Austin',layout=Layout(width='950px', height='30px'))

data_index_sample = widgets.IntSlider(min=1, max = n_sample, value=1.0, step = 1.0, description = 'Data Sample Index, $\\beta$',orientation='horizontal', style = {'description_width': 'initial'}, continuous_update=False)

ui_sample = widgets.VBox([l_sample,data_index_sample],)

def run_plot_sample(data_index_sample):                       # make data, fit models and plot
    plt.subplot(131)
    plt.plot(por_sample,cprob_sample, alpha = 0.2, c = 'black') # plot piecewise linear interpolation
    plt.scatter(por_sample,cprob_sample,s = 30, alpha = 1.0, c = 'green', edgecolor = 'black',zorder = 100) # plot the CDF points
    plt.grid(); plt.xlim([0.05,0.25]); plt.ylim([0.0,1.0])
    plt.xlabel("Porosity (fraction)"); plt.ylabel("Cumulative Probability"); plt.title("Original Sparse Sample with Noise, Cumulative Distribution Function")
    plt.plot([por_sample[data_index_sample-1],por_sample[data_index_sample-1]],[0.0,cprob_sample[data_index_sample-1]],color = 'red',linestyle='dashed')
    plt.plot([por_sample[data_index_sample-1],3.0],[cprob_sample[data_index_sample-1],cprob_sample[data_index_sample-1]],color = 'red',linestyle='dashed')
    plt.annotate('x = ' + str(round(por_sample[data_index_sample-1],2)), xy=(por_sample[data_index_sample-1]+0.003, 0.01))
    plt.annotate('p = ' + str(round(cprob_sample[data_index_sample-1],2)), xy=(0.225, cprob_sample[data_index_sample-1]+0.02))

    plt.subplot(132)
    plt.plot(por,cprob, alpha = 0.2, c = 'black') # plot piecewise linear interpolation
    plt.scatter(por,cprob,s = 10, c = 'red', edgecolor = 'black', alpha = 1.0) # plot the CDF points
    plt.grid(); plt.xlim([0.05,0.25]); plt.ylim([0.0,1.0])
    plt.xlabel("Porosity (fraction)"); plt.ylabel("Cumulative Probability"); plt.title("Non-parametric Target Porosity Cumulative Distribution Function")
    plt.plot([0.0,y_sample[data_index_sample-1]],[cprob_sample[data_index_sample-1],cprob_sample[data_index_sample-1]],color = 'red',linestyle='dashed')
    plt.plot([y_sample[data_index_sample-1],y_sample[data_index_sample-1]],[0.0,cprob_sample[data_index_sample-1]],color = 'red',linestyle='dashed')
    plt.annotate('p = ' + str(round(cprob_sample[data_index_sample-1],2)), xy=(0.053, cprob_sample[data_index_sample-1]+0.02))
    plt.annotate('y = ' + str(round(y_sample[data_index_sample-1],2)), xy=(y_sample[data_index_sample-1]+0.003, 0.01))
    plt.scatter(y_sample[data_index_sample-1],cprob_sample[data_index_sample-1],s = 50, c = 'green', edgecolor = 'black', alpha = 1.0, zorder=100) # plot the CDF points

    plt.subplot(133)
    plt.plot(por_sample,y_sample, alpha = 0.2, c = 'black') # plot piecewise linear interpolation
    plt.grid(); plt.xlim([0.05,0.25]); plt.ylim([0.05,0.25])
    plt.xlabel("Original Porosity (fraction)"); plt.ylabel("Transformed Porosity (fraction)"); plt.title("Non-parametric Distribution Transformation, Q-Q Plot")
    plt.plot([0.05,0.25],[0.05,0.25],color = 'red',linestyle='dashed', alpha = 0.4)
    plt.scatter(por_sample[data_index_sample-1],y_sample[data_index_sample-1],s = 50, c = 'green', edgecolor = 'black', alpha = 1.0, zorder=200) # plot the CDF points
    plt.scatter(por_sample,y_sample,s = 20, c = 'green', edgecolor = 'black', alpha = 0.3, zorder=100) # plot the CDF points

    plt.subplots_adjust(left=0.0, bottom=0.0, right=3.0, top=1.2, wspace=0.2, hspace=0.2)
    plt.show()

# connect the function to make the samples and plot to the widgets
interactive_plot_s = widgets.interactive_output(run_plot_sample, {'data_index_sample':data_index_sample})
#interactive_plot_sample.clear_output(wait = True)               # reduce flickering by delaying plot updating
```

### Interactive Data Analytics Distribution Transformation Demonstration

#### Michael Pyrcz, Associate Professor, The University of Texas at Austin

Select any data value and observe the distribution transform by mapping through cumulative probability.

#### The Inputs

* **data_index** - the data index from 1 to n in the sorted ascending order

```python
display(ui_sample, interactive_plot_s)                           # display the interactive plot
```

    VBox(children=(Text(value='                                                  Data Analytics, Distribution Tran…

    Output(outputs=({'output_type': 'display_data', 'data': {'text/plain': '
', 'i…

To summarize let's look at a DataFrame with the original noisey sample and the transformed to match the original distribution.

* we're making and showing a table of original values, $x_{\beta}$ $\forall$ $\beta = 1, \ldots, n_{sample}$, and the transformed values, $y_{\beta}$ $\forall$ $\beta = 1, \ldots, n_{sample}$.

```python
df_sample['Transformed_Por'] = y_sample
df_sample.head(n=n_sample)
```

      X
      Y
      Facies
      Porosity
      Perm
      AI
      Transformed_Por

      80
      900.0
      100.0
      0.0
      0.095041
      1.280257
      4573.656072
      0.081044

      3
      100.0
      600.0
      0.0
      0.096817
      2.446678
      5201.637996
      0.085867

      218
      251.0
      416.0
      0.0
      0.103819
      1.003374
      5822.467914
      0.091834

      226
      211.0
      396.0
      0.0
      0.103983
      6.368529
      5725.334803
      0.095628

      72
      900.0
      900.0
      0.0
      0.105751
      12.433996
      6242.704810
      0.099378

      47
      600.0
      700.0
      0.0
      0.107279
      12.384496
      3595.586977
      0.101987

      71
      800.0
      100.0
      0.0
      0.109215
      7.739105
      5274.532660
      0.103970

      41
      500.0
      400.0
      0.0
      0.109631
      6.312198
      5515.918646
      0.106704

      189
      201.0
      456.0
      0.0
      0.109739
      0.546396
      5018.355476
      0.108460

      207
      201.0
      426.0
      0.0
      0.109890
      0.400658
      5263.542112
      0.111491

      210
      231.0
      426.0
      0.0
      0.116357
      5.584040
      4919.074871
      0.113887

      5
      100.0
      400.0
      0.0
      0.119589
      3.691908
      5295.267191
      0.117984

      53
      600.0
      100.0
      1.0
      0.144072
      42.396044
      4204.150893
      0.121592

      93
      955.0
      549.0
      1.0
      0.171594
      374.298925
      3181.557281
      0.127131

      84
      955.0
      559.0
      1.0
      0.178382
      74.215058
      3386.182722
      0.137062

      151
      995.0
      489.0
      1.0
      0.182994
      460.494986
      2792.804322
      0.153759

      111
      955.0
      529.0
      1.0
      0.184800
      1113.971076
      3177.635737
      0.176536

      174
      955.0
      459.0
      1.0
      0.184864
      45.002088
      3394.563038
      0.185270

      158
      975.0
      479.0
      1.0
      0.192515
      69.336576
      2493.128177
      0.188136

      136
      935.0
      499.0
      1.0
      0.197534
      523.287810
      2579.032897
      0.191655

      150
      985.0
      489.0
      1.0
      0.199802
      73.133040
      2672.294567
      0.195998

      165
      955.0
      469.0
      1.0
      0.204886
      26.197239
      2889.196647
      0.198182

      113
      975.0
      529.0
      1.0
      0.206625
      1548.094062
      3167.185377
      0.199315

      99
      925.0
      539.0
      1.0
      0.209065
      211.163296
      3442.885245
      0.201943

      245
      690.0
      529.0
      1.0
      0.210129
      316.905689
      4271.013148
      0.206934

      141
      985.0
      499.0
      1.0
      0.210587
      68.276148
      2547.526113
      0.209051

      149
      975.0
      489.0
      1.0
      0.216690
      21.109085
      2412.875330
      0.211638

      181
      935.0
      449.0
      1.0
      0.216750
      368.507601
      4249.477923
      0.215686

      125
      1005.0
      519.0
      1.0
      0.221394
      54.667195
      2577.714678
      0.224214

      129
      955.0
      509.0
      1.0
      0.231752
      1525.247066
      2512.061434
      0.242298

It would be straitforward to modify the code above to perform distribution transformations:

* to a parametric distribution like Gaussian

* to a non-parametric distribution from actual data (build a CDF and interpolate between the data samples)

#### Comments

This was a basic demonstration of distribution transformations.

I have other demonstrations on the basics of working with DataFrames, ndarrays, univariate statistics, plotting data, declustering, data transformations, trend modeling and many other workflows available at [Python Demos](https://github.com/GeostatsGuy/PythonNumericalDemos) and a Python package for data analytics and geostatistics at [GeostatsPy](https://github.com/GeostatsGuy/GeostatsPy).

I hope this was helpful,

*Michael*

#### The Author:

### Michael Pyrcz, Associate Professor, University of Texas at Austin
*Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions*

With over 17 years of experience in subsurface consulting, research and development, Michael has returned to academia driven by his passion for teaching and enthusiasm for enhancing engineers' and geoscientists' impact in subsurface resource development.

For more about Michael check out these links:

#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

#### Want to Work Together?

I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.

* Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I'd be happy to drop by and work with you!

* Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!

* I can be reached at mpyrcz@austin.utexas.edu.

I'm always happy to discuss,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin

#### More Resources Available at: [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

```python

```


**Quantum data Preparation**

In this recent work author [(Huang et al.)](https://arxiv.org/abs/2011.01938) shows some problems that are classically hard to compute can be predicted easily with classical machines that learn from data. This work also showcases an empirical performance separation between classical and quantum machine learning model via a carefully crafted dataset.

1.   Prepare a reduced dimension [MNIST](http://yann.lecun.com/exdb/mnist/) dataset.
2.   Use quantum circuits to re-label the dataset and compute Projected Quantum Kernel features (PQK).
3.   Train a classical neural network on the modified level dataset and compare the performance.

**Install**

```
pip -q install tensorflow==2.3.1 tensorflow-quantum
```

```
#import necessary libraries
import cirq
import sympy
import numpy as np
import tensorflow as tf
import tensorflow_quantum as tfq
#Visualization library tools
%matplotlib inline
import matplotlib.pyplot as plt
from cirq.contrib.svg import SVGCircuit
np.random.seed(1234)
```

**Data Preparation** - Preparing MNIST dataset

```
# Downloading MNIST dataset
(x_train, y_train),(x_test, y_test) = tf.keras.datasets.mnist.load_data()

# Rescale the image range from decimal range[0-255] to binary range[0.0-1.0]
x_train, x_test = x_train[..., np.newaxis]/255.0, x_test[..., np.newaxis]/255.0

print("Total number of actual training examples-", len(x_train))
print("Total number of actual test examples-", len(x_test))
```

Removing the other classes except the Digit-9 and Digit-0. At the same time converting the label, y, to boolean: True for 9 and False for 0.

```
def filter_03(x, y):
    keep = (y == 9) | (y == 0)
    x, y = x[keep], y[keep]
    y = y == 0
    return x,y
```

```
x_train, y_train = filter_03(x_train, y_train)
x_test, y_test = filter_03(x_test, y_test)

print("Total number of filtered training examples-", len(y_train))
print("Total number of filtered test examples-", len(y_test))
```

Display sample-

```
print(y_train[0])
plt.imshow(x_train[0, :, :, 0])
```

Using Principal Component Analysis [(PCA)](https://en.wikipedia.org/wiki/Principal_component_analysis) transformation to reduce the dimensions-

```
def truncate_x(x_train, x_test, n_components=10):
#Performing PCA on image dataset with the top 'n_components=10'
  n_points_train = tf.gather(tf.shape(x_train), 0)
  n_points_test = tf.gather(tf.shape(x_test), 0)

  # Flatten the dimension into 1D
  x_train = tf.reshape(x_train, [n_points_train, -1])
  x_test = tf.reshape(x_test, [n_points_test, -1])

  # Normalize.
  feature_mean = tf.reduce_mean(x_train, axis=0)
  x_train_normalized = x_train - feature_mean
  x_test_normalized = x_test - feature_mean

  # Truncate.
  e_values, e_vectors = tf.linalg.eigh(
      tf.einsum('ji,jk->ik', x_train_normalized, x_train_normalized))
  return tf.einsum('ij,jk->ik', x_train_normalized, e_vectors[:,-n_components:]), \
    tf.einsum('ij,jk->ik', x_test_normalized, e_vectors[:, -n_components:])
```

```
DATASET_DIM = 10
x_train, x_test = truncate_x(x_train, x_test, n_components=DATASET_DIM)
print('New datapoint dimension:', len(x_train[0]))
```

Reducing the size of the Training and Testing sets

```
N_TRAIN = 1000
N_TEST = 200
x_train, x_test = x_train[:N_TRAIN], x_test[:N_TEST]
y_train, y_test = y_train[:N_TRAIN], y_test[:N_TEST]
```

```
print("New number of training examples:", len(x_train))
print("New number of test examples:", len(x_test))
```

**Releveling and Computing PQK features-**

To get the seperation between Quantum and classical methods, first prepare the PQK features and then relabel outputs based on their values.

```
#Generate the wall of single qubit rotations:
def single_qubit_wall(qubits, rotations):
  """Prepare a single qubit X,Y,Z rotation wall on `qubits`."""
  wall_circuit = cirq.Circuit()
  for i, qubit in enumerate(qubits):
    for j, gate in enumerate([cirq.X, cirq.Y, cirq.Z]):
      wall_circuit.append(gate(qubit) ** rotations[i][j])

  return wall_circuit
```

```
#Verify how it works
SVGCircuit(single_qubit_wall(
    cirq.GridQubit.rect(1,4), np.random.uniform(size=(4, 3))))
```

```
#Prepare V(theta)
def v_theta(qubits):
  ref_paulis = [
      cirq.X(q0) * cirq.X(q1) + \
      cirq.Y(q0) * cirq.Y(q1) + \
      cirq.Z(q0) * cirq.Z(q1) for q0, q1 in zip(qubits, qubits[1:])
  ]
  exp_symbols = list(sympy.symbols('ref_0:'+str(len(ref_paulis))))
  return tfq.util.exponential(ref_paulis, exp_symbols), exp_symbols
```

Examine a two qubit case from the bigger circuit for cross checking Output-

```
test_circuit, test_symbols = v_theta(cirq.GridQubit.rect(1, 2))
print(f'Symbols found in circuit:{test_symbols}')
SVGCircuit(test_circuit)
```

```
#Prepare the pqk feature circuits around a dataset
def prepare_pqk_circuits(qubits, classical_source, n_trotter=10):
  n_qubits = len(qubits)
  n_points = len(classical_source)

  # Prepare random single qubit rotation wall.
  random_rots = np.random.uniform(-2, 2, size=(n_qubits, 3))
  initial_U = single_qubit_wall(qubits, random_rots)

  # Prepare parametrized V
  V_circuit, symbols = v_theta(qubits)
  exp_circuit = cirq.Circuit(V_circuit for t in range(n_trotter))

  # Convert to `tf.Tensor`
  initial_U_tensor = tfq.convert_to_tensor([initial_U])
  initial_U_splat = tf.tile(initial_U_tensor, [n_points])

  full_circuits = tfq.layers.AddCircuit()(
      initial_U_splat, append=exp_circuit)
  # Replace placeholders in circuits with values from `classical_source`.
  return tfq.resolve_parameters(
      full_circuits, tf.convert_to_tensor([str(x) for x in symbols]),
      tf.convert_to_tensor(classical_source*(n_qubits/3)/n_trotter))
```

```
qubits = cirq.GridQubit.rect(1, DATASET_DIM + 1)
q_x_train_circuits = prepare_pqk_circuits(qubits, x_train)
q_x_test_circuits = prepare_pqk_circuits(qubits, x_test)
```

```
#PQK features based on above construction.
def get_pqk_features(qubits, data_batch):
  ops = [[cirq.X(q), cirq.Y(q), cirq.Z(q)] for q in qubits]
  ops_tensor = tf.expand_dims(tf.reshape(tfq.convert_to_tensor(ops), -1), 0)
  batch_dim = tf.gather(tf.shape(data_batch), 0)
  ops_splat = tf.tile(ops_tensor, [batch_dim, 1])
  exp_vals = tfq.layers.Expectation()(data_batch, operators=ops_splat)
  rdm = tf.reshape(exp_vals, [batch_dim, len(qubits), -1])
  return rdm
```

```
x_train_pqk = get_pqk_features(qubits, q_x_train_circuits)
x_test_pqk = get_pqk_features(qubits, q_x_test_circuits)
print('New PQK training dataset has shape:', x_train_pqk.shape)
print('New PQK testing dataset has shape:', x_test_pqk.shape)
```

**Re-labeling based on PQK features**

```
def compute_kernel_matrix(vecs, gamma):
  """Computes d[i][j] = e^ -gamma * (vecs[i] - vecs[j]) ** 2 """
  scaled_gamma = gamma / (
      tf.cast(tf.gather(tf.shape(vecs), 1), tf.float32) * tf.math.reduce_std(vecs))
  return scaled_gamma * tf.einsum('ijk->ij',(vecs[:,None,:] - vecs) ** 2)

#Compute the eigenvalues and eigenvectors of the kernel of datapoints
def get_spectrum(datapoints, gamma=1.0):
  KC_qs = compute_kernel_matrix(datapoints, gamma)
  S, V = tf.linalg.eigh(KC_qs)
  S = tf.math.abs(S)
  return S, V
```

```
S_pqk, V_pqk = get_spectrum(
    tf.reshape(tf.concat([x_train_pqk, x_test_pqk], 0), [-1, len(qubits) * 3]))

S_original, V_original = get_spectrum(
    tf.cast(tf.concat([x_train, x_test], 0), tf.float32), gamma=0.005)

print('Eigenvectors of pqk kernel matrix:', V_pqk)
print('Eigenvectors of original kernel matrix:', V_original)
```

```
#Preparing new labels to maximize geometric distance between kernels
def get_stilted_dataset(S, V, S_2, V_2, lambdav=1.1):
  S_diag = tf.linalg.diag(S ** 0.5)
  S_2_diag = tf.linalg.diag(S_2 / (S_2 + lambdav) ** 2)
  scaling = S_diag @ tf.transpose(V) @ \
            V_2 @ S_2_diag @ tf.transpose(V_2) @ \
            V @ S_diag

  # Generate new lables using the largest eigenvector.
  _, vecs = tf.linalg.eig(scaling)
  new_labels = tf.math.real(
      tf.einsum('ij,j->i', tf.cast(V @ S_diag, tf.complex64), vecs[-1])).numpy()
  # Create new labels and add some small amount of noise.
  final_y = new_labels > np.median(new_labels)
  noisy_y = (final_y ^ (np.random.uniform(size=final_y.shape) > 0.95))
  return noisy_y
```

```
y_relabel = get_stilted_dataset(S_pqk, V_pqk, S_original, V_original)
y_train_new, y_test_new = y_relabel[:N_TRAIN], y_relabel[N_TRAIN:]
```

**Comparing Models-**

Creating two small feedforward neural networks and comparing their performance analysis

1) PQK enhanced model

```
def create_pqk_model():
    model = tf.keras.Sequential()
    model.add(tf.keras.layers.Dense(32, activation='sigmoid', input_shape=[len(qubits) * 3,]))
    model.add(tf.keras.layers.Dense(16, activation='sigmoid'))
    model.add(tf.keras.layers.Dense(1))
    return model

pqk_model = create_pqk_model()
pqk_model.compile(loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
              optimizer=tf.keras.optimizers.Adam(learning_rate=0.003),
              metrics=['accuracy'])

pqk_model.summary()
```

```
pqk_history = pqk_model.fit(tf.reshape(x_train_pqk, [N_TRAIN, -1]),
          y_train_new,
          batch_size=32,
          epochs=1000,
          verbose=0,
          validation_data=(tf.reshape(x_test_pqk, [N_TEST, -1]), y_test_new))
```

2) Classical model

```
def create_fair_classical_model():
    model = tf.keras.Sequential()
    model.add(tf.keras.layers.Dense(32, activation='sigmoid', input_shape=[DATASET_DIM,]))
    model.add(tf.keras.layers.Dense(16, activation='sigmoid'))
    model.add(tf.keras.layers.Dense(1))
    return model

model = create_fair_classical_model()
model.compile(loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
              optimizer=tf.keras.optimizers.Adam(learning_rate=0.03),
              metrics=['accuracy'])

model.summary()
```

```
classical_history = model.fit(x_train,
          y_train_new,
          batch_size=32,
          epochs=1000,
          verbose=0,
          validation_data=(x_test, y_test_new))
```

```
plt.figure(figsize=(10,5))
plt.plot(classical_history.history['accuracy'], label='accuracy_classical')
plt.plot(classical_history.history['val_accuracy'], label='val_accuracy_classical')
plt.plot(pqk_history.history['accuracy'], label='accuracy_quantum')
plt.plot(pqk_history.history['val_accuracy'], label='val_accuracy_quantum')
plt.xlabel('Epoch')
plt.ylabel('Accuracy')
plt.legend()
```

Accuaracy on both models will achieve > 0.9  on the training data.

```

```


```python
# For reading data sets from the web.
import pandas
# For lots of great things.
import numpy as np
# To make our plots.
import matplotlib.pyplot as plt
%matplotlib inline
# Because sympy and LaTeX make
# everything look wonderful!
from sympy import *
init_printing(use_latex=True)
from IPython.display import display
# We will use this to check our implementation...
from sklearn.decomposition import PCA
# We will grab another data set using Keras
# after we finish up with Iris...
import keras
# Need this for LabelEncoder
from sklearn import preprocessing
```

    Using TensorFlow backend.

```python
student_data = np.array(pandas.read_table("./student-por.csv",
delimiter=";", header=0))
```

```python
student_data.shape
```

```python
le = preprocessing.LabelEncoder()
```

# Encoding non-numeric data to integers

```python
# School (binomial)
le.fit(student_data[:,0])
student_data[:,0] = le.transform(student_data[:,0])
```

```python
# Sex (binomial)
le.fit(student_data[:,1])
student_data[:,1] = le.transform(student_data[:,1])
```

```python
# address (binomial)
le.fit(student_data[:,3])
student_data[:,3] = le.transform(student_data[:,3])
```

```python
# Famsize (binomial)
le.fit(student_data[:,4])
student_data[:,4] = le.transform(student_data[:,4])
```

```python
# Pstatus (binomial)
le.fit(student_data[:,5])
student_data[:,5] = le.transform(student_data[:,5])
```

```python
# Mjob (nominal)
le.fit(student_data[:,8])
student_data[:,8] = le.transform(student_data[:,8])
```

```python
# Fjob (nominal)
le.fit(student_data[:,9])
student_data[:,9] = le.transform(student_data[:,9])
```

```python
# Reason (nominal)
le.fit(student_data[:,10])
student_data[:,10] = le.transform(student_data[:,10])
```

```python
# Guardian (nominal)
le.fit(student_data[:,11])
student_data[:,11] = le.transform(student_data[:,11])
```

```python
# schoolsup (binomial)
le.fit(student_data[:,15])
student_data[:,15] = le.transform(student_data[:,15])
```

```python
# famsup (binomial)
le.fit(student_data[:,16])
student_data[:,16] = le.transform(student_data[:,16])
```

```python
# paid (binomial)
le.fit(student_data[:,17])
student_data[:,17] = le.transform(student_data[:,17])
```

```python
# activities (binomial)
le.fit(student_data[:,18])
student_data[:,18] = le.transform(student_data[:,18])
```

```python
# nursery (binomial)
le.fit(student_data[:,19])
student_data[:,19] = le.transform(student_data[:,19])
```

```python
# higher (binomial)
le.fit(student_data[:,20])
student_data[:,20] = le.transform(student_data[:,20])
```

```python
# internet (binomial)
le.fit(student_data[:,21])
student_data[:,21] = le.transform(student_data[:,21])
```

```python
# romantic (binomial)
le.fit(student_data[:,22])
student_data[:,22] = le.transform(student_data[:,22])
```

```python
student_data
```

    array([[0, 0, 18, ..., 0, 11, 11],
           [0, 0, 17, ..., 9, 11, 11],
           [0, 0, 15, ..., 12, 13, 12],
           ...,
           [1, 0, 18, ..., 11, 12, 9],
           [1, 1, 17, ..., 10, 10, 10],
           [1, 1, 18, ..., 10, 11, 11]], dtype=object)

# Encoding 0's to -1 for binomial data.

### Surely there's a one line method to do this... right, Keras?

```python
school = student_data[:,0]
school = np.where(school==0, -1, school)
student_data[:,0] = school
```

```python
sex = student_data[:,1]
sex = np.where(sex==0, -1, sex)
student_data[:,1] = sex
```

```python
address = student_data[:,3]
address = np.where(address==0, -1, address)
student_data[:,3] = address
```

```python
famsize = student_data[:,4]
famsize = np.where(famsize==0, -1, famsize)
student_data[:,4] = famsize
```

```python
Pstatus = student_data[:,5]
Pstatus = np.where(Pstatus==0, -1, Pstatus)
student_data[:,5] = Pstatus
```

```python
schoolsup = student_data[:,15]
schoolsup = np.where(schoolsup==0, -1, schoolsup)
student_data[:,15] = schoolsup
```

```python
famsup = student_data[:,16]
famsup = np.where(famsup==0, -1, famsup)
student_data[:,16] = famsup
```

```python
paid = student_data[:,17]
paid = np.where(paid==0, -1, paid)
student_data[:,17] = paid
```

```python
activities = student_data[:,18]
activities = np.where(activities==0, -1, activities)
student_data[:,18] = activities
```

```python
nursery = student_data[:,19]
nursery = np.where(nursery==0, -1, nursery)
student_data[:,19] = nursery
```

```python
higher = student_data[:,20]
higher = np.where(higher==0, -1, higher)
student_data[:,20] = higher
```

```python
internet = student_data[:,21]
internet = np.where(internet==0, -1, internet)
student_data[:,21] = internet
```

```python
romantic = student_data[:,22]
romantic = np.where(romantic==0, -1, romantic)
student_data[:,22] = romantic
```

# Standardizing the nominal and numerical data.

```python
scaler = preprocessing.StandardScaler()
```

### I don't think this warning is THAT concerning

```python
temp = student_data[:,[2,6,7,8,9,10,11,12,13,14,23,24,25,26,27,28,29]]
Standardized = scaler.fit_transform(temp)
```

    /nfshome/apps/python-3.6.7/lib/python3.6/site-packages/sklearn/utils/validation.py:590: DataConversionWarning: Data with input dtype object was converted to float64 by StandardScaler.
      warnings.warn(msg, DataConversionWarning)
    /nfshome/apps/python-3.6.7/lib/python3.6/site-packages/sklearn/utils/validation.py:590: DataConversionWarning: Data with input dtype object was converted to float64 by StandardScaler.
      warnings.warn(msg, DataConversionWarning)

### Not sure about the -0.0 for the mean...

```python
print('Mean:', round(Standardized.mean()))
print('Standard deviation:', Standardized.std())
```

    Mean: -0.0
    Standard deviation: 1.0

```python
student_data[:,[2,6,7,8,9,10,11,12,13,14,23,24,25,26,27,28,29]] = Standardized
```

```python
x = student_data[:,0:30]
```

```python
x[0,0:]
```

    array([-1, -1, 1.0316951635937113, 1, -1, -1, 1.310215632680141,
           1.5407154371084224, -1.556452535644832, 2.057247821444129,
           -0.9339740507872134, 0.3326475069742609, 0.5767179914572501,
           0.08365294968062233, -0.3743051181921297, 1, -1, -1, -1, 1, 1, -1,
           -1, 0.07260614867809903, -0.17164653548813819, 0.6937849605682166,
           -0.5435553304269218, -0.9976946599159401, -0.3710421971530383,
           0.0734333641512146], dtype=object)

```python
results = student_data[:,32] # output vector
```

### Convert results to one-hot encoding
#### the actual output vector doesn't include at least one example of each possible grade so I made a vector of all possibilities for to_categorical

```python
possible_results = [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]
```

```python
y = keras.utils.to_categorical(results, len(possible_results))
```

# Data is now preprocessed

### Note: columns 31 and 32 are not used. I do not believe they are useful for our net.

```python
model = keras.Sequential()
```

```python
input_size = x.shape[1]
output_size = y.shape[1]
```

```python
# create the hidden layer
model.add(keras.layers.Dense(400,input_dim=input_size,activation='relu'))
```

```python
model.add(keras.layers.Dense(200,activation='relu'))
```

```python
# output layer
model.add(keras.layers.Dense(output_size,activation='softmax'))
```

```python
# compile
model.compile(loss='categorical_crossentropy',optimizer='adam', metrics=['accuracy'])
```

```python
model.summary()
```

    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    dense_18 (Dense)             (None, 400)               12400
    _________________________________________________________________
    dense_19 (Dense)             (None, 200)               80200
    _________________________________________________________________
    dense_20 (Dense)             (None, 20)                4020
    =================================================================
    Total params: 96,620
    Trainable params: 96,620
    Non-trainable params: 0
    _________________________________________________________________

# Model Training
## This is some results from training the model
## Loss seems to be going down slowly
## Interestingly val_loss seems to be increasing
## I have ploted the loss below
## I could not plot accuracy which is weird and I will have to look into what is causing that
# Austin

```python
history = model.fit(x,y,batch_size = 32, epochs = 160, verbose = 0, validation_split = 0.25)
```

```python
plt.plot(history.history['loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train'], loc='upper left')
plt.show()

```

```python
score = model.evaluate(x,y, verbose=0)
print('Test loss:', score[0])
print('Test accuracy:', score[1])
```

    Test loss: 1.9049611709701997
    Test accuracy: 0.7781201849113261

```python

```


```python
%pylab inline
%config InlineBackend.figure_format = 'retina'

```

    Populating the interactive namespace from numpy and matplotlib

```python
from __future__ import print_function
import argparse
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torchvision import datasets, transforms
from torch.optim.lr_scheduler import StepLR

```

```python
shift = Q = lambda d,z: sum([1/d * cos(2*pi*p/d *z) for p in arange(-d/2+1,d/2)], axis=0)
L0 = lambda d,z: sum([-2*pi*p/d**2 * sin(2*pi*p/d *z) for p in arange(-d/2+1,d/2)], axis=0)

# cosine correlation
coscorr = lambda x,y: trace(x.T@y)/norm(x)/norm(y)
```

```python
d = 20
m = 3
z = arange(d)[:,newaxis]-arange(d)
imshow(Q(d,z-m), cmap = 'seismic',vmin=-1, vmax=1)
colorbar()
```

```python
d = 20
m = 2
ns = array([4,8])*m
labs = '0abcde'

mx = 2
z = arange(d)[:,newaxis]-arange(d)
ii=0
figure(figsize=(13,2.7))
ii+=1
subplot(1,len(ns)+2,ii)
text(-3,-1.5,labs[ii], fontdict={'size':16,'weight':1})

a=Q(d,z-m)
mx=max(abs(a.ravel()))
imshow(a, cmap = 'seismic',vmin=-1, vmax=1)
colorbar()
title(r'Ground truth $g(z=%d)$'%m, size=13)
xticks([])
yticks([])

ii+=1
subplot(1,len(ns)+2,ii)
text(-4,-1.5,labs[ii], fontdict={'size':16,'weight':1})

a=L0(d,z)
mx=max(abs(a.ravel()))
imshow(a, cmap = 'seismic',vmin=-mx, vmax=mx)
colorbar()
title(r'Ground truth $L$',size=13)
xticks([])
yticks([])

for n in ns:
    ii+=1
    subplot(1,len(ns)+2,ii)
    text(-4,-1.5,labs[ii], fontdict={'size':16,'weight':1})

    a=matrix_power(eye(d)-m/n* L0(d,z),n)
    mx = max(abs(a.ravel()))
    imshow(a, cmap = 'seismic',vmin=-mx, vmax=mx)
    colorbar()
#     overlap = trace(a.T@Q(d,z-m))/norm(a)/norm(Q(d,z-m))
    overlap = coscorr(a,Q(d,z-m))
    title(r'$\tilde{g}=(I+(%d/%d)L)^{%d}$'%(m,n,n)+'\nCorr'+r'$(g,\tilde{g})=%.2f$'%(overlap),size=13)
    xticks([])
    yticks([])

tight_layout()
# savefig('figs2/shift-1D-example.pdf')
# savefig('figs2/shift-1D-example.png',dpi=300)

```

### Correlation with random

```python
r = randn(*a.shape)
# trace(r.T@Q(d,z-m))/norm(r)/norm(Q(d,z-m))
coscorr(r, Q(d,z-m))
```

    -0.03139264271799015

```python
# trace(L0(d,z).T@Q(d,z-m))/norm(L0(d,z))/norm(Q(d,z-m))
coscorr(L0(d,z), Q(d,z-m))
```

    0.2957397716093291

# Exact rotation generator
$\newcommand{\ro}{\partial}$
We can use $L$ from 1D translations as $\ro_x$ and $\ro_y$.
We use this to construct the groun truth rotation generator $L_\theta = x\ro_y - y\ro_x$.

```python
L0 = lambda d,z: sum([2*pi*p/d**2 * sin(2*pi*p/d *z) for p in arange(-d/2+1,d/2)], axis=0)
```

### Image dimensions

```python
DIM = 7
```

### double the width and height
Rotation in pytorch double the $h,w$ (height and width) extent to perform the affine transform.
Therefore, we need to use $2h$ and $2w$ in $L$.

```python
d = 20
z = arange(d)[:,newaxis]-arange(d)
imshow(L0(2*d,z), cmap = 'seismic',vmin=-1, vmax=1)
colorbar()
```

```python
d = DIM
coords = mgrid[:d,:d] - d/2
x,y = coords.reshape((2,-1))

dx = (x[:,newaxis]-x) * (y[:,newaxis]==y) # make sure distance calculated only for the same y
dy = (y[:,newaxis]-y) * (x[:,newaxis]==x) # ... same x

figure(figsize=(10,4))
subplot(121)

Lx = L0(2*d,dx)
imshow(Lx,cmap = 'seismic',vmin=-1, vmax=1)
colorbar()
title(r'$L_x\approx \partial_x$',size=14)

subplot(122)

Ly = L0(2*d,dy)
imshow(Ly,cmap = 'seismic',vmin=-1, vmax=1)
colorbar()
title(r'$L_y\approx \partial_y$',size=14);

```

```python

figure(figsize=(10,4))

yLx = diag(y) @ Lx
xLy = diag(x) @ Ly

L_rot = xLy - yLx
imshow(L_rot,cmap = 'seismic')#,vmin=-1, vmax=1)
colorbar()
title(r'Ground truth $L_\theta=x\partial_y-y\partial_x$', size = 14);
```

# Trainer

```python
from time import time
class trainer():
    def __init__(self, model, device, optimizer, dataset_class = None,
                 train_loader=None, test_loader=None,
                 batch_size = 64, test_batch_size = 1000,loss_func = F.nll_loss,
                ):
        """
        usage:
            t = trainer(...)
            t.fit(epochs)

        methods:
            .fit(epochs) : train + test; print results; stores results in trainer.history <dict>
            .train(epoch)
            .test()
        """
        self.device = device #torch.device(device)
        self.optimizer = optimizer
        self.model = model
        self.loss_func = loss_func

        self.scheduler = StepLR(optimizer, step_size=1, gamma=0.7)
        self.history = {'train loss':[], 'test loss':[], 'train acc':[], 'test acc':[], 'train time':[]}

        if dataset_class:
            self.make_dataloaders(dataset_class, batch_size, test_batch_size)
        else:
            self.train_loader = train_loader
            self.test_loader = test_loader

    def make_dataloaders(self, dataset_class, batch_size, test_batch_size):
        train_kwargs = {'batch_size': batch_size}
        test_kwargs = {'batch_size': test_batch_size}
        if self.device.type =='cuda':
            cuda_kwargs = {'num_workers': 1,
                           'pin_memory': True,
                           'shuffle': True}
            train_kwargs.update(cuda_kwargs)
            test_kwargs.update(cuda_kwargs)

        print('Creating data loaders...',end='')
        dataset1 = dataset_class('../data', train=True, download=True,)
        dataset2 = dataset_class('../data', train=False,)

        self.train_loader = torch.utils.data.DataLoader(dataset1,**train_kwargs)
        self.test_loader = torch.utils.data.DataLoader(dataset2, **test_kwargs)
        print('Done')

#         self.optimizer = optimizer
#         self.model, self.device, self.train_loader,  = model, device, train_loader,
#         self.test_loader = test_loader

    def progbar(self,percent, N=10):
        n = int(percent//N)
        return '[' + '='*n + '>' +'.'*(N-n-1) +']'

    def train(self,epoch):
        self.model.train()
        training_loss = 0
        correct = 0
        t0 = time()
        for batch_idx, (data, target) in enumerate(self.train_loader):
            if type(data)==list:
                data = [d.to(self.device) for d in data]
            else:
                data = data.to(self.device)
            if type(target)==list:
                target = [d.to(self.device) for d in target]
            else:
                target = target.to(self.device)

#             data, target = data.to(self.device), target.to(self.device)
            self.optimizer.zero_grad()
            output = self.model(data)
            loss = self.loss_func(output, target)
            training_loss += loss.sum().item() # sum up batch loss
            pred = output.argmax(dim=1, keepdim=True)  # get the index of the max log-probability
            correct += pred.eq(target.view_as(pred)).sum().item()

            loss.backward(retain_graph=True)
            self.optimizer.step()
            if batch_idx % 10 == 0:
                perc = 100. * batch_idx / len(self.train_loader)
                t1 = time()
                print('Train Epoch: {} {} {:.1f}s [{}/{} ({:.0f}%)]\tLoss: {:.4g}'.format(
                    epoch, self.progbar(perc), t1-t0,
                    batch_idx * len(data), len(self.train_loader.dataset), # n/N
                    perc, # % passed
                    loss.item()), end='\r')

        training_loss /= len(self.train_loader.dataset)
        acc = correct / len(self.train_loader.dataset)
        print('\nTraining: loss: {:.4g}, Acc: {:.2f}%'.format(training_loss, 100.*acc))

        return {'loss':training_loss, 'acc':acc , 'time':t1-t0}

    def test(self):
        self.model.eval()
        test_loss = 0
        correct = 0
        with torch.no_grad():
            for data, target in self.test_loader:
                if type(data)==list:
                    data = [d.to(self.device) for d in data]
                else:
                    data = data.to(self.device)
                if type(target)==list:
                    target = [d.to(self.device) for d in target]
                else:
                    target = target.to(self.device)
                #data, target = data.to(self.device), target.to(self.device)
                output = self.model(data)
                test_loss += self.loss_func(output, target, reduction='sum').item()  # sum up batch loss
                pred = output.argmax(dim=1, keepdim=True)  # get the index of the max log-probability
                correct += pred.eq(target.view_as(pred)).sum().item()

        test_loss /= len(self.test_loader.dataset)
        test_acc = correct / len(self.test_loader.dataset)

        print('Test loss: {:.4g}, Test acc.: {:.2f}%'.format( test_loss, 100.*test_acc))
        return {'loss':test_loss, 'acc':test_acc}

    def fit(self,epochs=1):
        for epoch in range(1, epochs + 1):
            r = self.train(epoch)
            self.history['train loss'] += [r['loss']]
            self.history['train acc'] += [r['acc']]
            self.history['train time'] += [r['time']]

            r = self.test()
            self.history['test loss'] += [r['loss']]
            self.history['test acc'] += [r['acc']]
            self.scheduler.step()
```

# L-conv funcs

```python
# includes residual

class Lconv_core(nn.Module):
    """ L-conv layer with full L """
    def __init__(self,d,num_L=1,cin=1,cout=1,rank=8):
        """
        L:(num_L, d, d)
        Wi: (num_L, cout, cin)
        """
        super().__init__()
        self.L = nn.Parameter(torch.Tensor(num_L, d, d))
        self.Wi = nn.Parameter(torch.Tensor(num_L, cout, cin))

        # initialize weights and biases
        nn.init.kaiming_normal_(self.L)
        nn.init.kaiming_normal_(self.Wi)

    def forward(self, x):
        # x:(batch, channel, flat_d)
        # h = (x + Li x Wi) W0
        y = torch.einsum('kdf,bcf,koc->bod', self.L, x, self.Wi ) +x #+ self.b
        return y

```

```python
# includes residual

class Lconv_grid(nn.Module):
    """ L-conv using an inferred grid in the data.
    """
    def __init__(self,idx, k,d,cin,cout):
        super().__init__()
        self.idx = idx
        self.L_sparse = nn.Parameter(torch.Tensor(*idx.shape, k))
        self.W = nn.Parameter(torch.Tensor(k, cin, cout))
        self.b = nn.Parameter(torch.Tensor(1, cout, 1))
        # the bias is not complete, but more efficient
#         self.b = nn.Parameter(torch.Tensor(1,k, cout, 1))

        # initialize weights and biases
        nn.init.kaiming_normal_(self.L_sparse, mode='fan_in')
        nn.init.kaiming_normal_(self.W)
        nn.init.kaiming_normal_(self.b)

    def forward(self, x):
        # x:(batch, channel, flat_d)
        return torch.einsum('bcdi,dik,kco->bod',x[:,:,self.idx], self.L_sparse, self.W ) + self.b

```

```python
class Reshape(nn.Module):
    def __init__(self,shape=None):
        self.shape = shape
        super().__init__()
    def forward(self,x):
        return x.view(-1,*self.shape)
```

# Datasets

```python
class FixedRot(datasets.VisionDataset):
    num_targets = 10
    def __init__(self,*args,angle =pi/3,N=50000,size=(7,7),
                 train=True,dataseed=0,**kwargs):
        super().__init__(*args,**kwargs)
        if not train:
            dataseed += 1
            N = int(0.2*N)
        torch.manual_seed(dataseed)
        angles = torch.ones(N)*angle # torch.rand(N)*2*np.pi
        self.data = torch.rand(N,1,*size)-.5
        print(N, self.data.shape)
        with torch.no_grad():
            # Build affine matrices for random translation of each image
            affineMatrices = torch.zeros(N,2,3)
            affineMatrices[:,0,0] = angles.cos()
            affineMatrices[:,1,1] = angles.cos()
            affineMatrices[:,0,1] = angles.sin()
            affineMatrices[:,1,0] = -angles.sin()
            # affineMatrices[:,0,2] = -2*np.random.randint(-self.max_trans, self.max_trans+1, bs)/w
            # affineMatrices[:,1,2] = 2*np.random.randint(-self.max_trans, self.max_trans+1, bs)/h

            flowgrid = F.affine_grid(affineMatrices, size = self.data.size())
            self.data_rot = F.grid_sample(self.data, flowgrid)
    def __getitem__(self,idx):
        return self.data[idx], self.data_rot[idx]

    def __len__(self):
        return len(self.data)

    def default_aug_layers(self):
        return RandomRotateTranslate(0)# no translation

```

# Experiment dataloader

```python

# dataset = CIFAR100
dataset = FixedRot #AugRotMNIST__0
# dataset = AugRotMNIST__0
num_targets = dataset.num_targets

batch_size = 32

test_batch_size = 1000
device = torch.device("cuda")
# device = torch.device("cpu")

train_kwargs = {'batch_size': batch_size}
test_kwargs = {'batch_size': test_batch_size}
# if use_cuda:
cuda_kwargs = {'num_workers': 1,
               'pin_memory': True,
               'shuffle': True}
train_kwargs.update(cuda_kwargs)
test_kwargs.update(cuda_kwargs)

ang_n = 10
ang = pi/ang_n
dataset1 = dataset('../data', train=True, size=(DIM,DIM), angle=ang)
dataset2 = dataset('../data', train=False, size=(DIM,DIM), angle=ang)

train_loader = torch.utils.data.DataLoader(dataset1,**train_kwargs)
test_loader = torch.utils.data.DataLoader(dataset2, **test_kwargs)

```

    50000 torch.Size([50000, 1, 7, 7])
    10000 torch.Size([10000, 1, 7, 7])

    /home/nima/pyenvs/ML_tests-1/lib/python3.8/site-packages/torch/nn/functional.py:3448: UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details.
      warnings.warn("Default grid_sample and affine_grid behavior has changed "
    /home/nima/pyenvs/ML_tests-1/lib/python3.8/site-packages/torch/nn/functional.py:3385: UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details.
      warnings.warn("Default grid_sample and affine_grid behavior has changed "

```python
r=0 # r = 1e-2*torch.randn(*dataset1.data.shape)
X,Y = (dataset1.data-127.5)/255+r, (dataset1.data_rot-127.5)/255 +r
X,Y = X.view(X.shape[0], -1), Y.view(Y.shape[0], -1)
```

### numerical estimate for $L_\theta$
Using the solution to linear regression
$$ R = (X^TX)^{-1}X^TY$$

```python
n = 20000
R0 = torch.inverse(X[:n].t()@X[:n]) @ (X[:n].t()@Y[:n])
L_linreg = (R0.numpy() - eye(R0.shape[0]))/ang
```

```python
title('Exact numerical')
imshow(L_linreg, cmap='seismic')#, vmin = -m, vmax = m)
colorbar()
```

# Learning $L_\theta$ from small rotations
Now we learn $L_\theta$ from pairs of images where one has a small, fixed rotation relative to the other.
This is simply using L-conv for linear regression, trained via SGD.
\begin{align}
    F(h) &= h+ L_ih\bar{\epsilon}^i %= (I+L_ihM^i)\cr
    %M^i &= W^i [\tilde{W}^0]^{-1}
\end{align}

```python
s = dataset1[0][0].shape
d = prod(s)
lc = Lconv_core(d=d,num_L=1, cin=1, cout=1)
model = nn.Sequential(
    nn.Flatten(2),
    lc,
    Reshape(s)
).to(device)
```

```python
optimizer = optim.Adam(model.parameters(),lr = 1e-2)

```

```python
t = trainer(model, device, optimizer, train_loader=train_loader, test_loader=test_loader,
            #dataset_class=AugRotMNIST__0, #dataset,
            loss_func=F.mse_loss)
```

```python
t.fit(20)
```

    Train Epoch: 1 [=========>] 1.7s [49920/50000 (100%)]	Loss: 5.285e-11
    Training: loss: 1.912e-13, Acc: 0.00%
    Test loss: 2.148e-09, Test acc.: 0.00%
    Train Epoch: 2 [=========>] 1.7s [49920/50000 (100%)]	Loss: 9.992e-12
    Training: loss: 5.941e-14, Acc: 0.00%
    Test loss: 5.974e-10, Test acc.: 0.00%
    Train Epoch: 3 [=========>] 1.7s [49920/50000 (100%)]	Loss: 3.099e-13
    Training: loss: 9.645e-15, Acc: 0.00%
    Test loss: 1.043e-11, Test acc.: 0.00%
    Train Epoch: 4 [=========>] 1.7s [49920/50000 (100%)]	Loss: 8.616e-15
    Training: loss: 2.998e-16, Acc: 0.00%
    Test loss: 4.285e-13, Test acc.: 0.00%
    Train Epoch: 5 [=========>] 1.7s [49920/50000 (100%)]	Loss: 1.787e-15
    Training: loss: 5.932e-17, Acc: 0.00%
    Test loss: 9.503e-14, Test acc.: 0.00%
    Train Epoch: 6 [=========>] 1.7s [49920/50000 (100%)]	Loss: 8.681e-16
    Training: loss: 3.328e-17, Acc: 0.00%
    Test loss: 6.067e-14, Test acc.: 0.00%
    Train Epoch: 7 [=========>] 1.8s [49920/50000 (100%)]	Loss: 6.981e-16
    Training: loss: 3.963e-17, Acc: 0.00%
    Test loss: 3.212e-14, Test acc.: 0.00%
    Train Epoch: 8 [=========>] 1.7s [49920/50000 (100%)]	Loss: 9.245e-16
    Training: loss: 2.211e-17, Acc: 0.00%
    Test loss: 2.758e-14, Test acc.: 0.00%
    Train Epoch: 9 [=========>] 1.7s [49920/50000 (100%)]	Loss: 4.676e-16
    Training: loss: 1.795e-17, Acc: 0.00%
    Test loss: 2.343e-14, Test acc.: 0.00%
    Train Epoch: 10 [=========>] 1.7s [49920/50000 (100%)]	Loss: 4.437e-16
    Training: loss: 1.661e-17, Acc: 0.00%
    Test loss: 2.154e-14, Test acc.: 0.00%
    Train Epoch: 11 [=========>] 1.8s [49920/50000 (100%)]	Loss: 4.101e-16
    Training: loss: 1.452e-17, Acc: 0.00%
    Test loss: 1.985e-14, Test acc.: 0.00%
    Train Epoch: 12 [=========>] 1.7s [49920/50000 (100%)]	Loss: 3.921e-16
    Training: loss: 1.233e-17, Acc: 0.00%
    Test loss: 1.923e-14, Test acc.: 0.00%
    Train Epoch: 13 [=========>] 1.7s [49920/50000 (100%)]	Loss: 3.933e-16
    Training: loss: 1.206e-17, Acc: 0.00%
    Test loss: 1.873e-14, Test acc.: 0.00%
    Train Epoch: 14 [=========>] 1.7s [49920/50000 (100%)]	Loss: 3.751e-16
    Training: loss: 1.189e-17, Acc: 0.00%
    Test loss: 1.853e-14, Test acc.: 0.00%
    Train Epoch: 15 [=========>] 1.7s [49920/50000 (100%)]	Loss: 3.631e-16
    Training: loss: 1.176e-17, Acc: 0.00%
    Test loss: 1.826e-14, Test acc.: 0.00%
    Train Epoch: 16 [=========>] 1.7s [49920/50000 (100%)]	Loss: 3.411e-16
    Training: loss: 1.168e-17, Acc: 0.00%
    Test loss: 1.82e-14, Test acc.: 0.00%
    Train Epoch: 17 [=========>] 1.8s [49920/50000 (100%)]	Loss: 3.636e-16
    Training: loss: 1.162e-17, Acc: 0.00%
    Test loss: 1.81e-14, Test acc.: 0.00%
    Train Epoch: 18 [=========>] 1.8s [49920/50000 (100%)]	Loss: 3.616e-16
    Training: loss: 1.159e-17, Acc: 0.00%
    Test loss: 1.803e-14, Test acc.: 0.00%
    Train Epoch: 19 [=========>] 1.7s [49920/50000 (100%)]	Loss: 3.657e-16
    Training: loss: 1.157e-17, Acc: 0.00%
    Test loss: 1.799e-14, Test acc.: 0.00%
    Train Epoch: 20 [=========>] 1.8s [49920/50000 (100%)]	Loss: 4.13e-16
    Training: loss: 1.154e-17, Acc: 0.00%
    Test loss: 1.799e-14, Test acc.: 0.00%

```python
for k in ['train loss', 'test loss']:
    plot(t.history[k], label=k)

title(f'min test loss:{min(t.history["test loss"]):.3g}')
legend()
grid()
yscale('log')
ylabel('Loss')
xlabel('Epochs')
```

```python
L_small = (lc.L*lc.Wi).detach().cpu().numpy()[0] / ang
lc.Wi
```

    Parameter containing:
    tensor([[[-1.1799]]], device='cuda:0', requires_grad=True)

```python
m = 3

figure(figsize=(10,4))
subplot(121)
title(r'Learned $L$ (fixed $\theta=\pi/%d$)'%(ang_n)+\
      '\n'+r'Corr($L_\theta,L$)=%.2f'% coscorr(L_rot, L_small))
imshow(L_small, cmap='seismic', vmin = -m, vmax = m)
colorbar()

subplot(122)
title('Lin Reg $L_{LR}$'+\
      '\n'+r'Corr($L_\theta,L_{LR}$)=%.2f'% coscorr(L_rot, L_linreg.T))
imshow(L_linreg.T, cmap='seismic', vmin = -m, vmax = m)
colorbar()

# savefig('./figs2/L-so2-pi-10.pdf')
# savefig('./figs2/L-so2-pi-10.png', dpi=300)
```

# Mutiangle augmentation
In this task we predict the angle by comparing an unrotated input $x$ with a rotated one $y$.
Let $F(x)$ denote the action of the L-conv layers on $x$.
Comparing $F(x)$ with the rotated input $y$ should yield the correct angle $t$.
We do so by having multiple channels for $W$.
Each output channel $F(x)_a$ is one rotated version of the input using shared $L$.
These outputs should be compared with $y$, which can be done as $f_a = y^TF(x)_a$.

## Experiment: Learning the rotation angle
We feed a pair of images to the model, original and rotated.
We want L-conv to learn the rotation angle.

We can start by having cxc matrices for $W^i$ and first pass copies of the input to it.
In the end we take the inner product with the rotated input and the outputs of the network.
We then pass this through a classification or regression layer to read the angle.

```python
class RandRot(datasets.VisionDataset):
    def __init__(self,*args,dataseed=0,N=50000,size=(7,7),max_angle = np.pi,
                 train=True,**kwargs):
        super().__init__(*args,**kwargs)
        if not train:
            dataseed += 1
            N = int(0.2*N)
        torch.manual_seed(dataseed)
        angles = torch.rand(N)*max_angle
        self.angles = angles[:,newaxis] #
        self.data = torch.rand(N,1,*size)-.5
        print(N, self.data.shape)
        with torch.no_grad():
            # Build affine matrices for random translation of each image
            affineMatrices = torch.zeros(N,2,3)
            affineMatrices[:,0,0] = angles.cos()
            affineMatrices[:,1,1] = angles.cos()
            affineMatrices[:,0,1] = angles.sin()
            affineMatrices[:,1,0] = -angles.sin()
            # affineMatrices[:,0,2] = -2*np.random.randint(-self.max_trans, self.max_trans+1, bs)/w
            # affineMatrices[:,1,2] = 2*np.random.randint(-self.max_trans, self.max_trans+1, bs)/h

            flowgrid = F.affine_grid(affineMatrices, size = self.data.size())
            self.data_rot = F.grid_sample(self.data, flowgrid)
    def __getitem__(self,idx):
        return (self.data[idx], self.data_rot[idx]), self.angles[idx]

    def __len__(self):
        return len(self.data)

    def default_aug_layers(self):
        return RandomRotateTranslate(0)# no translation

```

```python
class Net(nn.Module):
    def __init__(self,d,c,device,k=1, rec=1, hid = 5):
        super().__init__()
        self._rec = rec
        self._c = c
#         with torch.no_grad():
        self.ones = torch.ones((1,c,1)).to(device)
        self.LC = Lconv_core(d=d,num_L=k,cin=c,cout=c)
        self.flat = nn.Flatten(2)
        self.lin1 = nn.Linear(c,hid)
        self.out = nn.Linear(hid,1)

    def forward(self, inp):
        x,y = inp
        # x,y->(batch, channel, flat_d)
        x = self.flat(x)
        y = self.flat(y)
        # copy input to c channels
        x = x * self.ones
        # pass through L-conv
        for _ in range(self._rec):
            x = self.LC(x)
        x = torch.einsum('bcd,bad->bc', x, y)
        x = F.tanh(x)
        x = self.lin1(x)
        x = F.tanh(x)
        return self.out(x)
```

```python
dataset = RandRot
batch_size = 16
test_batch_size = 1000
device = torch.device("cuda")
# device = torch.device("cpu")

train_kwargs = {'batch_size': batch_size}
test_kwargs = {'batch_size': test_batch_size}
# if use_cuda:
cuda_kwargs = {'num_workers': 1,
               'pin_memory': True,
               'shuffle': True}
train_kwargs.update(cuda_kwargs)
test_kwargs.update(cuda_kwargs)

kws = dict(size=(7,7),
          max_angle = np.pi/8)

dataset1 = dataset('../data', train=True, **kws)
dataset2 = dataset('../data', train=False, **kws)

train_loader = torch.utils.data.DataLoader(dataset1,**train_kwargs)
test_loader = torch.utils.data.DataLoader(dataset2, **test_kwargs)

```

    50000 torch.Size([50000, 1, 7, 7])
    10000 torch.Size([10000, 1, 7, 7])

    /home/nima/pyenvs/ML_tests-1/lib/python3.8/site-packages/torch/nn/functional.py:3448: UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details.
      warnings.warn("Default grid_sample and affine_grid behavior has changed "
    /home/nima/pyenvs/ML_tests-1/lib/python3.8/site-packages/torch/nn/functional.py:3385: UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details.
      warnings.warn("Default grid_sample and affine_grid behavior has changed "

```python
for (x,y),th in train_loader:
    x = x.to(device)
    y = y.to(device)
    break
```

```python
x.shape, y.shape, th.shape
```

    (torch.Size([16, 1, 7, 7]), torch.Size([16, 1, 7, 7]), torch.Size([16, 1]))

```python
s = dataset1.data[0].shape
d = prod(s)
# lc = Lconv_2(k=2,d=d, cin=1, cout=1, rank=50)
rec = 3
model = Net(d,10,k=1,device=device,rec=rec).to(device)

# model = nn.Sequential(
#     nn.Flatten(2),
#     *(rec*(lc,)),
#     Reshape(s)
# ).to(device)
```

```python
optimizer = optim.Adam(model.parameters(),lr = 1e-3)

```

```python
t = trainer(model, device, optimizer, train_loader=train_loader, test_loader=test_loader,
            #dataset_class=AugRotMNIST__0, #dataset,
            loss_func=F.mse_loss)
```

```python
dataset1.data.device
```

    device(type='cpu')

```python
t.fit(10)
```

    /home/nima/pyenvs/ML_tests-1/lib/python3.8/site-packages/torch/nn/functional.py:1628: UserWarning: nn.functional.tanh is deprecated. Use torch.tanh instead.
      warnings.warn("nn.functional.tanh is deprecated. Use torch.tanh instead.")

    Train Epoch: 1 [=========>] 8.0s [6240/50000 (100%)]	Loss: 0.0002899
    Training: loss: 0.0003859, Acc: 0.00%
    Test loss: 0.0004376, Test acc.: 0.00%
    Train Epoch: 2 [=========>] 7.9s [6240/50000 (100%)]	Loss: 0.0002815
    Training: loss: 1.503e-05, Acc: 0.00%
    Test loss: 0.0001817, Test acc.: 0.00%
    Train Epoch: 3 [=========>] 7.7s [6240/50000 (100%)]	Loss: 0.0001618
    Training: loss: 9.241e-06, Acc: 0.00%
    Test loss: 0.0001674, Test acc.: 0.00%
    Train Epoch: 4 [=========>] 7.5s [6240/50000 (100%)]	Loss: 3.136e-05
    Training: loss: 7.437e-06, Acc: 0.00%
    Test loss: 0.0001137, Test acc.: 0.00%
    Train Epoch: 5 [=========>] 7.5s [6240/50000 (100%)]	Loss: 0.0002264
    Training: loss: 6.523e-06, Acc: 0.00%
    Test loss: 0.0001023, Test acc.: 0.00%
    Train Epoch: 6 [=========>] 7.6s [6240/50000 (100%)]	Loss: 8.259e-05
    Training: loss: 5.921e-06, Acc: 0.00%
    Test loss: 9.492e-05, Test acc.: 0.00%
    Train Epoch: 7 [=========>] 7.6s [6240/50000 (100%)]	Loss: 4.221e-05
    Training: loss: 5.508e-06, Acc: 0.00%
    Test loss: 9.566e-05, Test acc.: 0.00%
    Train Epoch: 8 [=========>] 7.6s [6240/50000 (100%)]	Loss: 0.0001364
    Training: loss: 5.219e-06, Acc: 0.00%
    Test loss: 9.08e-05, Test acc.: 0.00%
    Train Epoch: 9 [=========>] 7.7s [6240/50000 (100%)]	Loss: 3.893e-05
    Training: loss: 5.008e-06, Acc: 0.00%
    Test loss: 8.726e-05, Test acc.: 0.00%
    Train Epoch: 10 [=========>] 7.6s [6240/50000 (100%)]	Loss: 9.347e-05
    Training: loss: 4.865e-06, Acc: 0.00%
    Test loss: 8.446e-05, Test acc.: 0.00%

```python
for k in ['train loss', 'test loss']:
    plot(t.history[k], label=k)

title(f'min test loss:{min(t.history["test loss"]):.3g}')
legend()
grid()
yscale('log')
```

```python
l = model.LC
# L1 = (1*torch.eye(len(l.L[0])).to(device) + l.L*l.Wi[0,0] ).detach().cpu().numpy()[0]
Ls = l.L.detach().cpu().numpy()
# L= Ls[0]
```

```python
figure(figsize=(10,4))
for i in range(len(Ls)):
    subplot(1,len(Ls), i+1)
    imshow(Ls[i].T,cmap='seismic', vmin = -1, vmax = 1)
    colorbar()

```

```python
coscorr(Ls[0], L_rot)
```

    -0.7020861964579547

```python
m = 3

figure(figsize=(14,3))

dat = [[L_rot, r'Ground truth $L_\theta=x\partial_y-y\partial_x$'],
       [Ls[0].T,'Learned $L_{rec}$ (recur=%d)'%rec
        +'\n'+r'Corr($L_\theta,L_{rec}$)=%.2f'% coscorr(L_rot, Ls[0].T)],
       [L_small,r'Learned $L$ (fixed $\theta=\pi/%d$)'%(ang_n)
        +'\n'+r'Corr($L_\theta,L$)=%.2f'% coscorr(L_rot, L_small)],
       [L_linreg.T,'Lin Reg $L_{LR}$'
        +'\n'+r'Corr($L_\theta,L_{LR}$)=%.2f'% coscorr(L_rot, L_linreg.T)],
      ]

labs = 'abcd'
for ii in arange(len(dat)):
    subplot(1,len(dat),ii+1)
    l,txt = dat[ii]
    m = max(abs(l.ravel()))
    imshow(l,cmap = 'seismic',vmin=-m, vmax=m)
    colorbar()
    title(txt, size = 13);
    # abcd
    text(-8,-1.8,labs[ii], fontdict={'size':16,'weight':1})
    xticks([])
    yticks([])

tight_layout()
savefig('./figs2/L-so2-full.pdf')
savefig('./figs2/L-so2-full.png', dpi=300)

```

```python
m = 3

figure(figsize=(10,8))

dat = [[L_rot, r'Ground truth $L_\theta=x\partial_y-y\partial_x$'],
       [Ls[0].T,'Learned $L_{rec}$ (recur=%d)'%rec
        +'\n'+r'Corr($L_\theta,L_{rec}$)=%.2f'% coscorr(L_rot, Ls[0].T)],
       [L_small,r'Learned $L$ (fixed $\theta=\pi/%d$)'%(ang_n)
        +'\n'+r'Corr($L_\theta,L$)=%.2f'% coscorr(L_rot, L_small)],
       [L_linreg.T,'Lin Reg $L_{LR}$'
        +'\n'+r'Corr($L_\theta,L_{LR}$)=%.2f'% coscorr(L_rot, L_linreg.T)],
      ]

labs = 'abcd'
for ii in arange(len(dat)):
    subplot(2,len(dat)//2,ii+1)
    l,txt = dat[ii]
    m = max(abs(l.ravel()))
    imshow(l,cmap = 'seismic',vmin=-m, vmax=m)
    colorbar()
    title(txt, size = 14);
    # abcd
    text(-8,-1.8,labs[ii], fontdict={'size':16,'weight':1})
    xticks([])
    yticks([])

tight_layout()
savefig('./figs2/L-so2-full-2.pdf')
savefig('./figs2/L-so2-full-2.png', dpi=300)

```


# Re-creating [Capillary Hysteresis in Neutrally Wettable Fibrous Media: A Pore Network Study of a Fuel Cell Electrode](http://link.springer.com/10.1007/s11242-017-0973-2)
# Part C: Purcell Meniscus Model

## Introduction

In the final part of this series we take a deeper look at the Purcell meniscus model, which is central to the whole study. The fundamental concept is that the fibers form throats with properties similar to that of a torus:

As the fluid invades throaugh the center of the torus the meniscus is pinned to the surface and the "effective" contact angle becomes influenced by the converging diverging geometry and is a function of the filling angle $\alpha$. The shape of the meniscus as the invading phase moves upwards through the torus with key model parameters is shown below.

Different intrinsic contact angles through invading phase: (a) 60$^\circ$, (b) 90$^\circ$ and (c) 120$^\circ$. All scenarios clearly show an inflection of the meniscus curvature signifying a switch in the sign of the capillary pressure from negative to positive. This inflection is predicted to occur for all contact angles by the model with varying filling angle. The capillary pressure can be shown to be:

$P_C = -2\sigma cos(\theta-\alpha))/(r+R(1-cos(\alpha))$

## Set up
We will set up a trivially small network with one throat to demonstrate the use of the meniscus model. Here we do the imports and define a few functions for plotting.

```python
import numpy as np
import sympy as syp
import matplotlib.pyplot as plt
from ipywidgets import interact
%matplotlib inline
%pylab inline
pylab.rcParams['figure.figsize'] = (5, 5)

theta = 90
fiberRad = 5e-6
throatRad = 2e-5
max_bulge = 1e-5

# Parameters for plotting fibers
x, R, rt, s, t = syp.symbols('x, R, rt, s, t')
points = np.linspace(-fiberRad, fiberRad, 1001)
y = R*syp.sqrt(1- (x/R)**2)
r = rt + (R-y)
rx = syp.lambdify((x, R, rt), r, 'numpy')

def plot_arc(ax, arc_cen, arc_rad, arc_angle, num_points=100):
    r'''
    plot the meniscus arc from neg to pos angle
    '''
    angles = np.linspace(-arc_angle, arc_angle, num_points)
    x = arc_cen - arc_rad*np.cos(angles)
    y = arc_rad*np.sin(angles)
    print(x)
    print(y)
    ax.plot(x, y, 'b-')
```

    Populating the interactive namespace from numpy and matplotlib

Now we define our two pore network and add the meniscus model in several modes: 'max' returns the maximum pressure experienced by the meniscus as it transitions through the throat, i.e. the burst entry pressure. 'touch' is the pressure at which the meniscus has protruded past the throat center a distance defined by the 'touch_length' dictionary key. In network simulations this could be set to the pore_diameter. Finally the 'men' mode accepts a target_Pc parameter and returns all the mensicus information required for assessing cooperative filling or plotting.

```python
import openpnm as op
import openpnm.models.physics as pm
net = op.network.Cubic(shape=[2, 1, 1], spacing=5e-5)
geo = op.geometry.StickAndBall(network=net,
                               pores=net.pores(),
                               throats=net.throats())
phase = op.phases.Water(network=net)
phase['pore.contact_angle'] = theta
phys = op.physics.Standard(network=net,
                           phase=phase,
                           geometry=geo)
geo['throat.diameter'] = throatRad*2
geo['throat.touch_length'] = max_bulge
phys.add_model(propname='throat.max',
               model=pm.meniscus.purcell,
               mode='max',
               r_toroid=fiberRad)
phys.add_model(propname='throat.touch',
               model=pm.meniscus.purcell,
               mode='touch',
               r_toroid=fiberRad)
phys.add_model(propname='throat.meniscus',
               model=pm.meniscus.purcell,
               mode='men',
               r_toroid=fiberRad,
               target_Pc=1000)
touch_Pc = phys['throat.touch'][0]
print(touch_Pc)
max_Pc = phys['throat.max'][0]
print(max_Pc)
```

    4114.018827899374
    5878.390833049412

We define a plotting function that uses the meniscus data:
$\alpha$ is filling angle as defined above,
$radius$ is the radius of curvature of the mensicus,
$center$ is the position of the centre of curvature relative to the throat center along the axis of the throat,
$\gamma$ is the angle between the throat axis and the line joining the meniscus center and meniscus contact point.

```python
def plot_phys(target_Pc, print_data=False):
    phys.add_model(propname='throat.meniscus',
               model=pm.meniscus.purcell,
               mode='men',
               r_toroid=fiberRad,
               target_Pc=target_Pc)
    throatRad = geo['throat.diameter'][0]/2
    theta = np.deg2rad(phys['pore.contact_angle'][0])
    men_data = {}
    men_data['alpha'] = phys['throat.meniscus.alpha']
    men_data['gamma'] = phys['throat.meniscus.gamma']
    men_data['radius'] = phys['throat.meniscus.radius']
    men_data['center'] = phys['throat.meniscus.center']
    fig, ax = plt.subplots()
    ax.plot(points, rx(points, fiberRad, throatRad), 'k-')
    ax.plot(points, -rx(points, fiberRad, throatRad), 'k-')
    arc_cen = men_data['center']
    arc_rad = men_data['radius']
    arc_angle = men_data['gamma']
    angles = np.linspace(-arc_angle, arc_angle, 100)
    arcx = arc_cen - arc_rad*np.cos(angles)
    arcy = arc_rad*np.sin(angles)
    ax.plot(arcx, arcy, 'b-')
    ax.axis('equal')
    ax.ticklabel_format(style='sci', axis='both', scilimits=(-6,-6))
    if print_data:
        print(men_data)
    return ax
```

We can see that the touch_Pc calculated earlier, corresponds with the tip of the meniscus exceeding the max_bulge parameter. Try changing this and re-running to see what happens.

```python
# NBVAL_IGNORE_OUTPUT
ax = plot_phys(target_Pc=touch_Pc, print_data=True);
ax.plot([max_bulge, max_bulge], [-throatRad, throatRad], 'r--');
```

We can interact with the mensicus model by changing the target_Pc parameter.

```python
# NBVAL_IGNORE_OUTPUT
interact(plot_phys, target_Pc=(-max_Pc, max_Pc, 10));
```

    interactive(children=(FloatSlider(value=-8.390833049412322, description='target_Pc', max=5878.390833049412, mi…

examples/paper_recreations/Tranter et al. (2017)/Tranter et al. (2017) - Part C.ipynb