# Dimensionality Reduction

## The Problem

There is an interesting tradeoff between model performance and a feature's dimensionality:

>*If the amount of available training data is fixed, then overfitting occurs if we keep adding dimensions. On the other hand, if we keep adding dimensions, the amount of **training data needs to grow exponentially fast to maintain the same coverage** and to avoid overfitting* ([Computer Vision for Dummies](http://www.visiondummy.com/2014/04/curse-dimensionality-affect-classification/)).

### Multi-Collinearity

In many cases, there is a high degree of correlation between many of the features in a dataset. For instance, suppose that you

## Sparsity

- High dimensionality increases the sparsity of your features (**what NLP techniques have we used that illustrate this point?**)
- The density of the training samples decreases when dimensionality increases:
- Distance measures (Euclidean, for instance) start losing their effectiveness, because there isn't much difference between the max and min distances in higher dimensions.
- Many models that rely upon assumptions of Gaussian distributions (like OLS linear regression), Gaussian mixture models, Gaussian processes, etc. become less and less effective since their distributions become flatter and "fatter tailed".

What is the amount of data needed to maintain **20% coverage** of the feature space? For 1 dimension, it is **20% of the entire population's dataset**. For a dimensionality of $D$:

$$
X^{D} = .20
$$
$$
(X^{D})^{\frac{1}{D}} = .20^{\frac{1}{D}}
$$
$$
X = \sqrt[D]{.20}
$$
You can approximate this as
```python
def coverage_requirement(requirement, D):
    return requirement ** (1 / D)

x = []
y = []
for d in range(1,20):
    y.append(coverage_requirement(0.10, d))
    x.append(d)

import matplotlib.pyplot as plt

plt.plot(x,y)
plt.xlabel("Number of Dimensions")
plt.ylabel("Appromximate % of Population Dataset")
plt.title("% of Dataset Needed to Maintain 10% Coverage of Feature Space")
plt.show()
```

```python
import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer
reviews = pd.read_csv("mcdonalds-yelp-negative-reviews.csv", encoding='latin-1')
reviews = open("poor_amazon_toy_reviews.txt", encoding='latin-1')

#text = reviews["review"].values
text = reviews.readlines()

vectorizer = CountVectorizer(ngram_range=(3,3), min_df=0.01, max_df=0.75, max_features=200)
# tokenize and build vocab
vectorizer.fit(text)
vector = vectorizer.transform(text)
features = vector.toarray()
features_df = pd.DataFrame(features, columns=vectorizer.get_feature_names())

correlations = features_df.corr()
correlations_stacked = correlations.stack().reset_index()
#set column names
correlations_stacked.columns = ['Bi-Gram 1','Bi-Gram 2','Correlation']
correlations_stacked = correlations_stacked[correlations_stacked["Correlation"] < 1]
correlations_stacked = correlations_stacked.sort_values(by=['Correlation'], ascending=False)
correlations_stacked.head()
```

      Bi-Gram 1
      Bi-Gram 2
      Correlation

      43
      don waste your
      waste your money
      0.777888

      197
      waste your money
      don waste your
      0.777888

      82
      of the box
      out of the
      0.609369

      110
      out of the
      of the box
      0.609369

      123
      this for my
      my year old
      0.078176

```python
import seaborn as sns
import matplotlib.pyplot as plt
plt.rcParams["figure.figsize"] = (15,15)
sns.heatmap(correlations)
```

# Principle Component Analysis

If you have an original matrix $Z$, you can decompose this matrix into two smaller matrices $X$ and $Q$.

## Important Points:

- Multiplying a vector by a matrix typically changes the direction of the vector. For instance:

Lazy Programmer-
        Tutorial to PCA

However, there are eigenvalues λ and eigenvectors $v$ such that

$$
\sum{X}v = \lambda v
$$

Multiplying the eigenvectors $v$ with the eigenvalue $\lambda$ does not change the direction of the eigenvector.

Multiplying the eigenvector $v$ by the covariance matrix $\sum{X}$ also does not change the direction of the eigenvector.

If our data $X$ is of shape $N \times D$, it turns out that we have $D$ eigenvalues and $D$ eigenvectors. This means we can arrange the eigenvalues $\lambda$ in decreasing order so that

$$
\lambda_3 > \lambda_2 > \lambda_5
$$

In this case, $\lambda_3$ is the largest eigenvalue, followed by $\lambda_2$, and then $\lambda_5$. Then, we can arrange

We can also rearrange the eigenvectors the same: $v_3$ will be the first column, $v_2$ will be the second column, and $v_5$ will be the third column.

We'll end up with two matrices $V$ and $\Lambda$:

Lazy Programmer-
        Tutorial to PCA

```python
# what is the shape of our features?
features.shape
```

    (12700, 15)

```python
from sklearn.decomposition import PCA

pca = PCA(n_components=4)

Z = pca.fit_transform(features)

# what is the shape of Z?
Z.shape
```

    (12700, 4)

```python
# what will happen if we take the correlation matrix and covariance matrix of our new reduced features?
import numpy as np
covariances = pd.DataFrame(np.cov(Z.transpose()))
plt.rcParams["figure.figsize"] = (5,5)
sns.heatmap(covariances)
```

```python
pca = PCA(n_components=2)
Z_two_dimensions = pca.fit_transform(features)
Z_two_dimensions
```

    array([[ 0.19584061, -0.05193457],
           [-0.03890385, -0.02632155],
           [-0.03890385, -0.02632155],
           ...,
           [-0.03890385, -0.02632155],
           [-0.03890385, -0.02632155],
           [-0.03890385, -0.02632155]])

```python
import matplotlib.pyplot as plt

plt.scatter(Z_two_dimensions[:,0], Z_two_dimensions[:, 1])
reduced_features_df = pd.DataFrame(Z_two_dimensions, columns=["x1", "x2"])
reduced_features_df["text"] = text
reduced_features_df.to_csv("reduced_features.csv")
```

# Singular Value Decomposition

Given an input matrix $A$, we want to try to represent it instead as three smaller matrices $U$, $\sum$, and $V$. Instead of **$n$ original terms**, we want to represent each document as **$r$ concepts** (other referred to as **latent dimensions**, or **latent factors**):

        Mining of Massive Datasets - Dimensionality Reduction: Singular Value Decomposition by Leskovec, Rajaraman, and Ullman (Stanford University)

Here, **$A$ is your matrix of word vectors** - you could use any of the word vectorization techniques we have learned so far, include one-hot encoding, word count, TF-IDF.

- $\sum$ will be a **diagonal matrix** with values that are positive and sorted in decreasing order. Its value indicate the **variance (information encoded on that new dimension)**- therefore, the higher the value, the stronger that dimension is in capturing data from A, the original features. For our purposes, we can think of the rank of this $\sum$ matrix as the number of desired dimensions. Instance, if we want to reduce $A$ from shape $1020 x 300$ to $1020 x 10$, we will want to reduce the rank of $\sum$ from 300 to 10.

- $U^T U = I$ and $V^T V = I$

## Measuring the Quality of the Reconstruction

A popular metric used for measuring the quality of the reconstruction is the [Frobenius Norm](https://en.wikipedia.org/wiki/Matrix_norm#Frobenius_norm). When you explain your methodology for reducing dimensions, usually managers / stakeholders will want to some way to compare multiple dimensionality techniques' ability to quantify its ability to retain information but trim dimensions:

$$
\begin{equation}
||A_{old}-A_{new}||_{F} = \sqrt{\sum_{ij}{(A^{old}_{ij}- A^{new}_{ij}}})^2
\end{equation}
$$

## Heuristic Step for How Many Dimensions to Keep

1. Sum the $\sum$ matrix's diagonal values:
$$
\begin{equation}
\sum_{i}^{m}\sigma_{i}
\end{equation}
$$

2. Define your threshold of "information" (variance) $\alpha$ to keep: usually 80% to 90%.

3. Define your cutoff point $C$: $$
\begin{equation}
C = \sum_{i}^{m}\sigma_{i} \alpha
\end{equation}
$$

4. Beginning with your largest singular value, sum your singular values $\sigma_{i}$ until it is greater than C. Retain only those dimensions.

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.linalg import svd
x = np.linspace(1,20, 20) # create the first dimension
x = np.concatenate((x,x))
y = x + np.random.normal(0,1, 40) # create the second dimension
z = x + np.random.normal(0,2, 40) # create the third dimension
a = x + np.random.normal(0,4, 40) # create the fourth dimension
plt.scatter(x,y) # plot just the first two dimensions
plt.show()
```

```python
A = np.stack([x,y,z,a]).T
A
```

    array([[ 1.        ,  0.77950175,  0.7677888 , -0.56338469],
           [ 2.        ,  2.37168063,  2.68471589,  5.07651543],
           [ 3.        ,  4.23624726,  5.41206404,  6.0099617 ],
           [ 4.        ,  4.46378345,  4.44011107,  9.75428485],
           [ 5.        ,  6.50663634,  4.66200827,  4.11785887],
           [ 6.        ,  5.94101546, 10.02615748,  9.45905519],
           [ 7.        ,  6.08581691,  0.47947543,  9.33775111],
           [ 8.        ,  8.50611795,  4.76759185,  2.58313411],
           [ 9.        ,  8.85808728, 11.924234  ,  3.16806057],
           [10.        ,  9.81238686, 11.75431427,  9.3688049 ],
           [11.        ,  8.91639482, 14.3247862 , 12.86762692],
           [12.        , 10.82815483,  9.81102502, 17.17389241],
           [13.        , 11.97544391, 12.74739555, 13.85953691],
           [14.        , 14.44142881, 13.73807306, 12.38912932],
           [15.        , 14.6098057 , 16.54791873, 13.84012328],
           [16.        , 15.33512682, 14.80703823, 20.31090617],
           [17.        , 15.95991959, 19.17082756, 11.31018836],
           [18.        , 17.41056707, 20.24597257, 20.76009469],
           [19.        , 19.07420736, 21.06709273, 11.94927062],
           [20.        , 19.68485709, 15.5035568 , 24.61855472],
           [ 1.        ,  0.55202803,  1.53319735,  6.05005041],
           [ 2.        ,  3.69366161,  4.41677271, -1.49068716],
           [ 3.        ,  3.21751898,  4.63878563,  5.32230317],
           [ 4.        ,  4.24055608,  1.04304654, -0.28281646],
           [ 5.        ,  5.88922848,  3.28261049, 10.7436995 ],
           [ 6.        ,  6.30668612,  4.44229131,  3.6406705 ],
           [ 7.        ,  7.56914998,  7.3755215 ,  8.3623776 ],
           [ 8.        , 10.54384416, 13.32846563, 10.57344076],
           [ 9.        , 10.04319803,  9.45062888, 14.89627194],
           [10.        , 10.66212523, 10.99819923, 12.36232314],
           [11.        , 11.14953649,  8.04430148, 10.37330748],
           [12.        , 12.63366622, 12.89333678, 15.37158399],
           [13.        , 14.08764033, 14.53184093,  9.81763298],
           [14.        , 13.41368703, 13.51218414, 11.62458517],
           [15.        , 14.64728374, 14.92915007, 14.00188219],
           [16.        , 17.24241005, 16.48392888, 11.60676132],
           [17.        , 18.13595467, 17.78644608, 13.5988243 ],
           [18.        , 19.60109224, 20.26131041, 17.23827819],
           [19.        , 18.24049712, 21.70792017, 16.70115288],
           [20.        , 19.38794542, 19.53899593, 22.5304716 ]])

```python
D = 1
U, s, V = svd(A)
print(f"s is {s}\n")
print(f"U is {U}\n")
print(f"V is {V}")
```

    s is [152.86974397  21.23629493  10.8533152    4.10757824]

    U is [[-0.0065041   0.05469218 -0.03365372 ... -0.22239603 -0.25909183
      -0.25717117]
     [-0.03966881 -0.1058881   0.07421117 ... -0.26639667  0.15604784
       0.0815931 ]
     [-0.06113708 -0.05823958  0.14883772 ... -0.10575752 -0.16214804
       0.03279713]
     ...
     [-0.24577592  0.08393435  0.02374991 ...  0.87896565 -0.02911555
      -0.03335981]
     [-0.24769877  0.12976656  0.13040048 ... -0.03325364  0.88036094
      -0.06124731]
     [-0.26632272 -0.1296538  -0.01417942 ... -0.02760683 -0.0678463
       0.90535574]]

    V is [[-0.49363198 -0.4960859  -0.51307612 -0.49696996]
     [ 0.16544881  0.22035026  0.45036767 -0.84925934]
     [-0.45544321 -0.47763446  0.73066685  0.17482211]
     [ 0.72216733 -0.69080378  0.00691485 -0.0348808 ]]

```python
s[D:] = 0
S = np.zeros((A.shape[0], A.shape[1]))
S[:A.shape[1], :A.shape[1]] = np.diag(s)
A_reconstructed = U.dot(S.dot(V))
np.sum((A_reconstructed - A) ** 2) ** (1/2) # Frobenius norm
# reconstruct matrix
U.dot(S)
```

# Exercise (20 minutes)

**1. Use the Amazon Fire Dataset**:

- Vectorize the documents with either word embeddings or TF-IDF
- Reduce the # of dimensions to 2
- Visualize the results on a scatter plot and identify clusters of similar product reviews

**2. Challenge:** In the example we have just done, we modelled the relationship between `x` and `y` as a linear relationship. Your manager does not think SVD works as well with non-linear relationships, and is skeptical because many of the social media marketing data she sees has non-linear relationships (ie. the number of times a post was seen versus the number of reactions it got). Investigate what the impact of non-linear relationships on SVD performance is in terms of:

- **# of dimensions needed to capture 90% of the variance**
- **Frobenius Norm** reconstruction error

If SVD is not significantly affected by non-linearity, **explain to your manager why**.

If SVD is significantly affected, **evaluate whether or not SVD can still be used as a dimensionality reduction technique**.

```python
# define a matrix
A = np.array([[1, 2], [3, 4], [5, 6]])
print(A)
# Singular-value decomposition
U, s, VT = svd(A)
# create m x n Sigma matrix
Sigma = np.zeros((A.shape[0], A.shape[1]))
# populate Sigma with n x n diagonal matrix
Sigma[:A.shape[1], :A.shape[1]] = np.diag(s)
Sigma
```

    [[1 2]
     [3 4]
     [5 6]]

    array([[9.52551809, 0.        ],
           [0.        , 0.51430058],
           [0.        , 0.        ]])


```python
import grouptesting
from grouptesting.model import *
from grouptesting.algorithms import *
import autograd.numpy as np
from autograd import grad
import matplotlib
import matplotlib.pyplot as plt
import seaborn as sns
import math
from scipy.stats import bernoulli
from scipy.optimize import minimize, rosen, rosen_der
from sympy import *

def nCr(n,k):
    return math.factorial(n) // math.factorial(k) // math.factorial(n - k)

np.random.seed(10)

```

## Plot ideas:

1. NCOMP vs. NDD for some q, or for three different qs (just to show empirical performance of NDD vs NCOMP) x
2. NCOMP with an achievability bound for some q x
4. NDD with achievability found for some q x

### 2. NCOMP with an achievability bound for some q

```python
# Define initial parameters
num_tests = 100
n = 1000 # population size
theta = 0.5
C = 2
n_theta = C * (n**theta)
k = round(n_theta) # number of infected
alpha = 0.5
p = alpha/k # Bernoulli test design probability parameter
eta = 0.001
q_results = []

############ RUN EXPERIMENT #####################
for q in [0.1]:
    NCOMP_ber_acc = []

    ############## Achievable bound 1 for T ################
    ## Choose delta according to z
    T_array = np.linspace(1, n, 25)
    z = ((1- alpha/(k) * (1-q))**(k-1) + (1/q)*(1 - alpha/(k) * (1-q))**k) / 2
    l = (1 - alpha / k * (1-q))**(k-1)
    u = (1 - alpha / k * (1-q))**(k) * (1/q)
    print("l-1, u-1:", l-1, u-1)
    delta = z - 1
    assert (l-1) < delta < (u-1)

    ########### Achievability Bound 2 Method 2 #######
    eta_1, alpha_1, theta_1, q_1 = eta, alpha, theta, q
    n_1 = n
    n_theta_1 = n_theta
    d_1 = round(n_theta_1)
    eps = 1e-12

    # Objective
    Tminus = lambda delta_1: ((1 + eta_1) * theta_1 * (1/(q_1**2)) * (d_1) * (np.log(n_1))) / (alpha_1 * (1 - np.exp(-2)) * (eps + (1+delta_1) - (1- alpha_1/(d_1) * (1-q_1))**(d_1-1))**2)
    Tplus = lambda delta_1: ((1 + eta_1) * (1/(q_1**2)) * (d_1) * (np.log(n_1))) / (alpha_1 * (1 - np.exp(-2)) * (eps + (1+delta_1) - (1/q_1) * (1- alpha_1/(d_1) * (1-q_1))**d_1)**2)
    ff = lambda delta_1: max(Tminus(delta_1), Tplus(delta_1))

    # Constraints
    l = (1 - alpha_1 / d_1 * (1-q_1))**(d_1-1)
    u = (1 - alpha_1 / d_1 * (1-q_1))**(d_1) * (1/q_1)
    bound = (l-1, u-1)

    x0 = delta
    options={'disp': None, 'maxcor': 100, 'ftol': 1e-14, 'gtol': 1e-012, 'eps': 1e-08, 'maxfun': 15000, 'maxiter': 15000, 'iprint': 1, 'maxls': 20}
    gff = grad(ff)
    res = minimize(ff, x0, tol=1e-12, bounds=[bound], jac=gff)
    achiev_bound = res.fun[0]

    ###### Main Experiment loop #####

    for T in T_array:
        NCOMP_ber_error = []
        NDD_ber_error = []
        COMP_ber_error = []
        print("T: ", int(round(T)))
        for test in range(num_tests):
            sigma = D(n, k) # Generate the vector of defectives
            X_ber = Ber(n, int(round(T)), p)
            y_ber = Y(dilution_noise(X_ber, q), sigma)

            # NCOMP - Bernoulli
            sigma_hat_ber = NCOMP(X_ber, y_ber, q=q, delta=delta)
            err = error(sigma, sigma_hat_ber)
            NCOMP_ber_error.append(error(sigma, sigma_hat_ber))

        acc = (num_tests - np.sum(np.array(NCOMP_ber_error))) / num_tests
        NCOMP_ber_acc.append(acc)

    ################ Plot ####################
    PAL= ['#f2f0f7','#cbc9e2','#9e9ac8','#756bb1','#54278f']
    plt.plot(T_array, NCOMP_ber_acc,'o-', c=PAL[-2], label="NCOMP - q = " + str(round(q, 2)))

#     use LaTeX fonts in the plot
    plt.rc('text', usetex=True)
    plt.rc('font', family='serif')
    matplotlib.rcParams['text.usetex'] = True

    plt.axvline(x=float(achiev_bound), ymin=0, ymax=1, color='black', linestyle='dotted', label="Achievability Bound", linewidth=3)
    plt.xticks(ticks=[10, round(float(achiev_bound)), round(n)])
    plt.yticks(ticks=[0.0, 1.0])
    plt.xlabel(r'Number of tests (T)', fontsize=11)
    plt.ylabel(r'Success Probability', fontsize=11)
    plt.legend()
    plt.savefig('plots/NCOMP Report Gabriel 2' + str(round(C, 2)) + 'qs, q = ' + str(round(q, 2)) + ', delta = ' + str(round(delta, 2)) + ', n = ' + str(n) + ', theta = ' + str(round(theta, 2)) + '.png')
    plt.savefig('plots/NCOMP Report Gabriel 2' + str(round(C, 2)) + 'qs, q = ' + str(round(q, 2)) + ', delta = ' + str(round(delta, 2)) + ', n = ' + str(n) + ', theta = ' + str(round(theta, 2)) + '.eps')
    plt.savefig('plots/NCOMP Report Gabriel 2' + str(round(C, 2)) + 'qs, q = ' + str(round(q, 2)) + ', delta = ' + str(round(delta, 2)) + ', n = ' + str(n) + ', theta = ' + str(round(theta, 2)) + '.pdf')
    plt.show()
```

```python
## Redo the plot because the latex font only comes in the second time you save it for some reason.
PAL= ['#f2f0f7','#cbc9e2','#9e9ac8','#756bb1','#54278f']
plt.plot(T_array, NCOMP_ber_acc,'o-', c=PAL[-2], label="NCOMP - q = " + str(round(q, 2)))

#     use LaTeX fonts in the plot
plt.rc('text', usetex=True)
plt.rc('font', family='serif')
matplotlib.rcParams['text.usetex'] = True

plt.axvline(x=float(achiev_bound), ymin=0, ymax=1, color='black', linestyle='dotted', label="Achievability Bound", linewidth=3)
plt.xticks(ticks=[10, round(float(achiev_bound)), round(n)])
plt.yticks(ticks=[0.0, 1.0])
plt.xlabel(r'Number of tests (T)', fontsize=11)
plt.ylabel(r'Success Probability', fontsize=11)
plt.legend()
plt.savefig('plots/NCOMP Report Gabriel 2' + str(round(C, 2)) + 'qs, q = ' + str(round(q, 2)) + ', delta = ' + str(round(delta, 2)) + ', n = ' + str(n) + ', theta = ' + str(round(theta, 2)) + '.png')
plt.savefig('plots/NCOMP Report Gabriel 2' + str(round(C, 2)) + 'qs, q = ' + str(round(q, 2)) + ', delta = ' + str(round(delta, 2)) + ', n = ' + str(n) + ', theta = ' + str(round(theta, 2)) + '.eps')
plt.savefig('plots/NCOMP Report Gabriel 2' + str(round(C, 2)) + 'qs, q = ' + str(round(q, 2)) + ', delta = ' + str(round(delta, 2)) + ', n = ' + str(n) + ', theta = ' + str(round(theta, 2)) + '.pdf')
plt.show()
```

### 4. NDD with Achievability bound for some q

```python
# Define initial parameters
num_tests = 100
n = 1000 # population size
theta = 0.15
C = 2
n_theta = C * (n**theta)
k = round(n_theta) # number of infected
eta = 0.001
q_results = []

############ RUN EXPERIMENT #####################
for q in [0.1]:
#     alpha = np.log(2)/(1-q) # 0.77
    alpha = 0.5 # 0.77
    p = alpha/k # Bernoulli test design probability parameter
    print("q: ", q)
    NDD_ber_acc = []
    NCOMP_ber_acc = []

    T_array = np.linspace(1, n, 25)
    pi_init = (q+0.07)

    ########### Achievability Bound 2 Method 2 #######
    eta_1, alpha_1, theta_1, q_1 = eta, alpha, theta, q
    n_1 = n
    n_theta_1 = n_theta
    d_1 = round(n_theta_1)
    eps = 1e-12

    # Objective
    D_ = lambda _eps: (_eps)*np.log(_eps) - _eps + 1
    T1 = lambda pi: (d_1 * np.log(d_1) * np.exp(alpha_1 * (1-q_1)) / ((alpha_1 * q_1)*D_(pi / (q_1 * np.exp(-alpha_1 * (1-q_1))))))
    T2 = lambda pi: ((1 - (theta_1 - eps)) * d_1 * np.log(n) * np.exp(alpha_1*(1-q_1)) / (alpha_1 * D_(pi * np.exp(alpha_1 * (1-q_1)))))
    T3 = lambda pi: (np.exp(alpha_1) * d_1 * np.log(d_1) / (alpha_1 * (1-q_1)))
    ff = lambda pi: max(T1(pi), T2(pi), T3(pi))

    bound = (q, np.exp(-alpha * (1-q))-0.01)

    x0 = pi_init
    gff = grad(ff)
    res = minimize(ff, x0, tol=1e-12, bounds=[bound], jac=gff)
    achiev_bound = res.fun[0]
    pi_NDD = res.x[0]
    assert q <= pi_NDD <= 1

    ###### Main Experiment loop #####
    for T in T_array:
        NDD_ber_error = []
        DD_ber_error = []
        NCOMP_ber_error = []
        print("T: ", int(round(T)))
        for test in range(num_tests):
            sigma = D(n, k) # Generate the vector of defectives
            X_ber = Ber(n, int(round(T)), p)
            y_ber = Y(dilution_noise(X_ber, q), sigma)

            # NCOMP - Bernoulli
            # Choose delta according to z
            z = ((1- alpha/(k) * (1-q))**(k-1) + (1/q)*(1 - alpha/(k) * (1-q))**k) / 2
            l = (1 - alpha / k * (1-q))**(k-1)
            u = (1 - alpha / k * (1-q))**(k) * (1/q)
            delta_NCOMP = z - 1
            assert (l-1) < delta_NCOMP < (u-1)
            sigma_hat_ber = NCOMP(X_ber, y_ber, q=q, delta=delta_NCOMP)
            err = error(sigma, sigma_hat_ber)
            NCOMP_ber_error.append(error(sigma, sigma_hat_ber))

            # NDD - Bernoulli
            sigma_hat_ber = NDD(X_ber, y_ber, pi=(pi_NDD), alpha=alpha, T = T, d = k)
            err = error(sigma, sigma_hat_ber)
            NDD_ber_error.append(error(sigma, sigma_hat_ber))

        acc = (num_tests - np.sum(np.array(NDD_ber_error))) / num_tests
        NDD_ber_acc.append(acc)
        acc = (num_tests - np.sum(np.array(NCOMP_ber_error))) / num_tests
        NCOMP_ber_acc.append(acc)

    ################ Plot ####################
    PAL = ['#edf8e9','#bae4b3','#74c476','#31a354','#006d2c']
    plt.plot(T_array, NDD_ber_acc,'o-', c=PAL[-2], label="NDD - q = " + str(round(q, 2)))

    # use LaTeX fonts in the plot
    plt.rc('text', usetex=True)
    plt.rc('font', family='serif')
    matplotlib.rcParams['text.usetex'] = True

    plt.axvline(x=ff(pi_NDD), ymin=0, ymax=1, color='black', linestyle='dotted', label="Achievability Bound", linewidth=3)
    plt.xticks(ticks=[round(10.0), round(ff(pi_NDD)), round(n)])
    plt.yticks(ticks=[0.0, 1.0])
    plt.xlabel(r'Number of tests (T)', fontsize=11)
    plt.ylabel(r'Success Probability', fontsize=11)
    plt.legend()
    plt.savefig('plots/NDD Report Gabriel 2' + str(round(C, 2)) + 'qs, q = ' + str(round(q, 2)) + ', pi = ' + str(round(pi_NDD, 2)) + ', n = ' + str(n) + ', theta = ' + str(round(theta, 2)) + '.png')
    plt.savefig('plots/NDD Report Gabriel 2' + str(round(C, 2)) + 'qs, q = ' + str(round(q, 2)) + ', pi = ' + str(round(pi_NDD, 2)) + ', n = ' + str(n) + ', theta = ' + str(round(theta, 2)) + '.eps')
    plt.savefig('plots/NDD Report Gabriel 2' + str(round(C, 2)) + 'qs, q = ' + str(round(q, 2)) + ', pi = ' + str(round(pi_NDD, 2)) + ', n = ' + str(n) + ', theta = ' + str(round(theta, 2)) + '.pdf')
    plt.show()
```

### 1. NDD vs NCOMP for some/different qs

```python
# Define initial parameters
num_tests = 100
n = 1000 # population size
theta = 0.25
C = 2
n_theta = C * (n**theta)
k = round(n_theta) # number of infected
eta = 0.001
q_results = []

############ RUN EXPERIMENT #####################
for q in [0.000001, 0.15, 0.35, 0.50]:
    alpha = 0.5
    p = alpha/k # Bernoulli test design probability parameter
    print("q: ", q)
    NDD_ber_acc = []
    DD_ber_acc = []
    NCOMP_ber_acc = []

    pi_init = (q+0.07)

    ########### Achievability Bound 2 Method 2 #######
    eta_1, alpha_1, theta_1, q_1 = eta, alpha, theta, q
    n_1 = n
    n_theta_1 = n_theta
    d_1 = round(n_theta_1)
    eps = 1e-12

    # Objective
    D_ = lambda _eps: (_eps)*np.log(_eps) - _eps + 1
    T1 = lambda pi: (d_1 * np.log(d_1) * np.exp(alpha_1 * (1-q_1)) / ((alpha_1 * q_1)*D_(pi / (q_1 * np.exp(-alpha_1 * (1-q_1))))))
    T2 = lambda pi: ((1 - (theta_1 - eps)) * d_1 * np.log(n) * np.exp(alpha_1*(1-q_1)) / (alpha_1 * D_(pi * np.exp(alpha_1 * (1-q_1)))))
    T3 = lambda pi: (np.exp(alpha_1) * d_1 * np.log(d_1) / (alpha_1 * (1-q_1)))
    ff = lambda pi: max(T1(pi), T2(pi), T3(pi))

    bound = (q, np.exp(-alpha * (1-q))-0.01)
    x0 = pi_init
    gff = grad(ff)
    res = minimize(ff, x0, tol=1e-6, bounds=[bound], jac=gff)
    achiev_bound = res.fun[0]
    pi_NDD = res.x[0]
    assert q <= pi_NDD <= 1

    T_array = np.linspace(1, 2000, 25)

    ###### Main Experiment loop #####
    for T in T_array:
        NDD_ber_error = []
        DD_ber_error = []
        NCOMP_ber_error = []
        print("T: ", int(round(T)))
        for test in range(num_tests):
            sigma = D(n, k) # Generate the vector of defectives
            X_ber = Ber(n, int(round(T)), p)
            y_ber = Y(dilution_noise(X_ber, q), sigma)

            # NCOMP - Bernoulli
            # Choose delta according to z
            z = ((1- alpha/(k) * (1-q))**(k-1) + (1/q)*(1 - alpha/(k) * (1-q))**k) / 2
            l = (1 - alpha / k * (1-q))**(k-1)
            u = (1 - alpha / k * (1-q))**(k) * (1/q)
            delta_NCOMP = z - 1
            assert (l-1) < delta_NCOMP < (u-1)
            sigma_hat_ber = NCOMP(X_ber, y_ber, q=q, delta=delta_NCOMP)
            err = error(sigma, sigma_hat_ber)
            NCOMP_ber_error.append(error(sigma, sigma_hat_ber))

            # NDD - Bernoulli
            sigma_hat_ber = NDD(X_ber, y_ber, pi=(pi_NDD), alpha=alpha, T = T, d = k)
            err = error(sigma, sigma_hat_ber)
            NDD_ber_error.append(error(sigma, sigma_hat_ber))

        acc = (num_tests - np.sum(np.array(NDD_ber_error))) / num_tests
        NDD_ber_acc.append(acc)
        acc = (num_tests - np.sum(np.array(NCOMP_ber_error))) / num_tests
        NCOMP_ber_acc.append(acc)

    q_results.append([q, T_array, NCOMP_ber_acc, NDD_ber_acc])
```

    q:  1e-06
    T:  1
    T:  84
    T:  168
    T:  251
    T:  334
    T:  417
    T:  501
    T:  584
    T:  667
    T:  751
    T:  834
    T:  917
    T:  1000
    T:  1084
    T:  1167
    T:  1250
    T:  1334
    T:  1417
    T:  1500
    T:  1584
    T:  1667
    T:  1750
    T:  1833
    T:  1917
    T:  2000
    q:  0.15
    T:  1
    T:  84
    T:  168
    T:  251
    T:  334
    T:  417
    T:  501
    T:  584
    T:  667
    T:  751
    T:  834
    T:  917
    T:  1000
    T:  1084
    T:  1167
    T:  1250
    T:  1334
    T:  1417
    T:  1500
    T:  1584
    T:  1667
    T:  1750
    T:  1833
    T:  1917
    T:  2000
    q:  0.35
    T:  1
    T:  84
    T:  168
    T:  251
    T:  334
    T:  417
    T:  501
    T:  584
    T:  667
    T:  751
    T:  834
    T:  917
    T:  1000
    T:  1084
    T:  1167
    T:  1250
    T:  1334
    T:  1417
    T:  1500
    T:  1584
    T:  1667
    T:  1750
    T:  1833
    T:  1917
    T:  2000
    q:  0.5
    T:  1
    T:  84
    T:  168
    T:  251
    T:  334
    T:  417
    T:  501
    T:  584
    T:  667
    T:  751
    T:  834
    T:  917
    T:  1000
    T:  1084
    T:  1167
    T:  1250
    T:  1334
    T:  1417
    T:  1500
    T:  1584
    T:  1667
    T:  1750
    T:  1833
    T:  1917
    T:  2000

```python
PAL1 = ['#edf8e9','#bae4b3','#74c476','#31a354','#006d2c']
PAL2 = ['#f2f0f7','#cbc9e2','#9e9ac8','#756bb1','#54278f']
for i in range(len(q_results)):
    print(i)
    q, T_array, NCOMP_ber_acc, NDD_ber_acc = q_results[i]

    if (i==0):
        plt.plot(T_array, NDD_ber_acc,'o-', c=PAL1[-1-i], label="NDD - q = 0.0 to 0.5", linewidth=2)
        plt.plot(T_array, NCOMP_ber_acc,'--', c=PAL2[-1-i], label="NCOMP - q = 0.0 to 0.5", linewidth=2)
    else:
        plt.plot(T_array, NDD_ber_acc,'o-', c=PAL1[-1-i], linewidth=2)
        plt.plot(T_array, NCOMP_ber_acc,'--', c=PAL2[-1-i], linewidth=2)

    # use LaTeX fonts in the plot
    plt.rc('text', usetex=True)
    plt.rc('font', family='serif')
    matplotlib.rcParams['text.usetex'] = True

    plt.yticks(ticks=[0.0, 1.0])
    plt.xlabel(r'Number of tests (T)', fontsize=11)
    plt.ylabel(r'Success Probability', fontsize=11)

plt.legend(loc='lower right')
plt.savefig('plots/NCOMP vs NDD Report Gabriel 2' + str(round(C, 2)) + 'qs, q = ' + str(round(q, 2)) + ', pi_NDD = ' + str(round(pi_NDD, 2)) + ', n = ' + str(n) + ', theta = ' + str(round(theta, 2)) + '.png')
plt.savefig('plots/NCOMP vs NDD Report Gabriel 2' + str(round(C, 2)) + 'qs, q = ' + str(round(q, 2)) + ', pi_NDD = ' + str(round(pi_NDD, 2)) + ', n = ' + str(n) + ', theta = ' + str(round(theta, 2)) + '.pdf')
plt.show()
```

```python

```

notebooks/Semester Project Plots Gabriel.ipynb


# QISKit の利用方法

```python
#!pip install qiskit -U
```

```python
# 必要なライブラリをインポートします．
from qiskit import QuantumRegister, ClassicalRegister, QuantumCircuit
from qiskit import compile, BasicAer
# QISKit Aer が量子シミュレータを提供します．
backend = BasicAer.get_backend("qasm_simulator")
# Aer に備わっている "qasm_simulator" を指定します．
# 量子ビットと古典ビットを準備します．
q = QuantumRegister(3, "q")
c0 = ClassicalRegister(1, "c0")
c1 = ClassicalRegister(1, "c1")
c2 = ClassicalRegister(1, "c2")
# 作成する量子ゲート回路を宣言します．
qc = QuantumCircuit(q, c0, c1, c2,name="teleport")
# 量子ビット q[0] を初期状態に準備します．
qc.u3(0.3, 0.2, 0.1, q[0])
# 量子ビット q[1], q[2] でBellペアを作ります．
qc.h(q[1])
qc.cx(q[1], q[2])
qc.barrier(q) # 状態準備の終わりを示します．
# Bell基底で q[0], q[1] を測定します．
qc.cx(q[0], q[1])
qc.h(q[0])
qc.measure(q[0], c0[0])
qc.measure(q[1], c1[0])
# q[0], q[1] の測定結果に応じたゲート操作を q[2] に施します．
qc.barrier(q)  # Bell基底を測定するまでを示します．
qc.z(q[2]).c_if(c0, 1)
qc.x(q[2]).c_if(c1, 1)
# q[2] を測定します．
qc.measure(q[2], c2[0])
initial_layout = {("q", 0): ("q", 0), ("q", 1): ("q", 1),("q", 2): ("q", 2)}
qobj = compile(qc, backend=backend, coupling_map=None, shots=1024, initial_layout=initial_layout)
job = backend.run(qobj)
print(job.result().get_counts(qc))
# shots で指定した回数測定した結果の分布が得られます．
```

    {'0 0 0': 235, '1 0 1': 4, '0 1 1': 278, '0 0 1': 257, '1 1 0': 6, '1 1 1': 5, '0 1 0': 233, '1 0 0': 6}

```python
%matplotlib inline
from qiskit.tools.visualization import circuit_drawer
circuit_drawer(qc, output="mpl")
```

# Cirq の利用方法

```python
#!pip install cirq
```

```python
# 必要なライブラリをインポートします．
import cirq

# 量子ビットと回路を準備します．
qubit = cirq.LineQubit.range(3)
qc = cirq.Circuit()
# ゲート操作（Operation）を追加します．
qc.append([
  cirq.H.on(qubit[1]),
  cirq.CNOT(qubit[1], qubit[2]),
  cirq.H.on(qubit[0]),
  cirq.Z.on(qubit[0])
  ])
qc.append([
  cirq.CNOT(qubit[0], qubit[1]),
  cirq.H.on(qubit[0]),
  cirq.measure(qubit[0]),
  cirq.measure(qubit[1])
  ])
print(qc)     # 回路を表示します．
simulator = cirq.google.XmonSimulator()
result = simulator.run(qc, repetitions=20)
print(result)
```

    0: ───────H───Z───@───H───M───
                      │
    1: ───H───@───────X───────M───
              │
    2: ───────X───────────────────
    0=11001001111111011101
    1=00011110010001110011

# pyQuil の利用方法

```python
#!pip install pyquil
```

```python
#!qvm -S
```

```python
# 必要なライブラリをインポートします．
import pyquil.quil as pq
from pyquil.gates import *
from pyquil.api import WavefunctionSimulator
# 量子プログラムの準備をします．
qvm = WavefunctionSimulator()
ins = pq.Program()
ro = ins.declare(name='ro', memory_type='BIT', memory_size=3)
ins.inst(H(1), CNOT(1,2)) # Bellペアを作ります．
ins.inst(H(0),Z(0)) # 初期状態を作ります．
# Bell基底で測定し，測定結果に応じたゲート操作します．
ins.inst(CNOT(0,1),H(0))
ins.measure(0,ro[0]).measure(1,ro[1]).if_then(ro[1], X(2)).if_then(ro[0], Z(2))
ins.measure(2,ro[2])
# 量子プログラムを実行します．
for i in range(10):
    q = qvm.wavefunction(ins)
    print(q)
```

    (1+0j)|010>
    (1+0j)|010>
    (1+0j)|001>
    (-1+0j)|111>
    (-1+0j)|110>
    (1+0j)|001>
    (-1+0j)|111>
    (1+0j)|010>
    (1+0j)|011>
    (-1+0j)|101>

# SymPy の利用方法

```python
from sympy import *    # SymPy の便利な関数など
init_printing()        # Jupyter Notebook で SymPy 出力を良い感じに出力します

from sympy.physics.quantum import *    # 量子計算のための関数など
from sympy.physics.quantum.qubit import * # 量子ビットのブラ・ケットベクトル、測定に関する関数
from sympy.physics.quantum.gate import *  # 量子ゲート各種
from sympy.physics.quantum.gate import IdentityGate as Id  # Python で I はほかの意味に割当たっているため使えません。恒等変換操作を Id として使えるようにします
get_ipython().run_line_magic('matplotlib', 'inline')
import matplotlib.pyplot as plt
from sympy.physics.quantum.circuitplot import CircuitPlot,labeller, Mz,CreateOneQubitGate

# Alice と Bob が対象としている量子ビットを測定する操作を準備します。
def alice(qbit): return measure_partial(qbit,(0,1))
def bob(qbit):  return measure_partial(qbit,(2,))

def U(x): return T(x)*X(x)*H(x)
ini_alice = U(0) * Qubit('000')
print(measure_partial(qapply(ini_alice),(0,)))

# Alice と Bob は量子もつれの状態を共有します。
pairs = CNOT(1,2)*H(1)

# Bell測定を行います。
bell_meas = CNOT(0,2)*CPHASE(1,2)*H(0)*CNOT(0,1)*pairs
CircuitPlot(bell_meas,nqubits=3, labels = ['alice','alice','bob'])
teleportated = qapply(bell_meas*ini_alice)

alice(teleportated)
```

```python

```


# Customer Assignment Problem

## Objective and Prerequisites

In this notebook you will learn how:

- To select facility locations, based on customer proximity.
- Machine learning can be leveraged to cope with big datasets. To this end, the reader should be familiar with clustering techniques, such as the k-means algorithm.

This modeling example is at the intermediate level, where we assume that you know Python and are familiar with the Gurobi Python API. In addition, you should have some knowledge about building mathematical optimization models.

**Download the Repository**

You can download the repository containing this and other examples by clicking [here](https://github.com/Gurobi/modeling-examples/archive/master.zip).

**Gurobi License**

In order to run this Jupyter Notebook properly, you must have a Gurobi license. If you do not have one, you can request an [evaluation license](https://www.gurobi.com/downloads/request-an-evaluation-license/?utm_source=3PW&utm_medium=OT&utm_campaign=WW-MU-LOG-OR-O_LEA-PR_NO-Q3_FY20_WW_JPME_customer-assignment_COM_EVAL_GITHUB_&utm_term=customer-assignment-problem&utm_content=C_JPM) as a *commercial user*, or download a [free license](https://www.gurobi.com/academia/academic-program-and-licenses/?utm_source=3PW&utm_medium=OT&utm_campaign=WW-MU-LOG-OR-O_LEA-PR_NO-Q3_FY20_WW_JPME_customer-assignment_ACADEMIC_EVAL_GITHUB_&utm_term=customer-assignment-problem&utm_content=C_JPM) as an *academic user*.

---
## Motivation

Many companies in various industries must, at some point, make strategic decisions about where to build facilities to support their operations. For example:

- Producers of goods need to decide how to design their supply chains – which encompass factories, distribution centers, warehouses, and retail stores.
- Healthcare providers need to determine where to build hospitals to maximize their population coverage.

These are strategic decisions that are difficult to implement and costly to change because they entail long-term investments. Furthermore, these decisions have a significant impact, both in terms of customer satisfaction and cost management. One of the critical factors to consider in this process is the location of the customers that the company is planning to serve.

---
## Problem Description

The Customer Assignment Problem is closely related to the Facility Location Problem, which is concerned with the optimal placement of facilities (from a set of candidate locations) in order to minimize the distance between the company's facilities and the customers. When the facilities have unlimited capacity, customers are assumed to be served by the closest facility.

In cases where the number of customers considered is too big, the customers can be grouped into clusters. Then, the cluster centers can be used in lieu of the individual customer locations. This pre-processing makes the assumption that all customers belonging to a given cluster will be served by the facility assigned to that cluster. The k-means algorithm can be used for this task, which aims to partition $n$ objects into $k$ distinct and non-overlapping clusters.

---
## Solution Approach

Mathematical optimization is a declarative approach where the modeler formulates a mathematical optimization model that captures the key aspects of a complex decision problem. The Gurobi Optimizer solves such models using state-of-the-art mathematics and computer science.

A mathematical optimization model has five components, namely:

- Sets and indices.
- Parameters.
- Decision variables.
- Objective function(s).
- Constraints.

We now present a Binary Integer Programming (BIP) formulation:

### Sets and Indices
$i \in I$: Set of customer clusters.

$j \in J$: Set of potential facility locations.

$\text{Pairings}= \{(i,j) \in I \times J: \text{dist}_{i,j} \leq \text{threshold}\}$: Set of allowed pairings

### Parameters
$\text{threshold} \in \mathbb{R}^+$: Maximum distance for a cluster-facility pairing to be considered.

$\text{max_facilities} \in \mathbb{N}$: Maximum number of facilities to be opened.

$\text{weight}_i \in \mathbb{N}$: Number of customers in cluster $i$.

$\text{dist}_{i,j} \in \mathbb{R}^+$: Distance from cluster $i$ to facility location $j$.

### Decision Variables
$\text{select}_j \in \{0,1\}$: 1 if facility location $j$ is selected; 0 otherwise.

$\text{assign}_{i,j} \in \{0,1\}$: 1 if cluster $i$ is assigned to facility location $j$; 0 otherwise.

### Objective Function

- **Total distance**: Minimize the total distance from clusters to their assigned facility:

\begin{equation}
\text{Min} \quad Z = \sum_{(i,j) \in \text{Pairings}}\text{weight}_i \cdot \text{dist}_{i,j} \cdot \text{assign}_{i,j}
\tag{0}
\end{equation}

### Constraints

- **Facility limit**: The number of facilities built cannot exceed the limit:

\begin{equation}
\sum_{j}\text{select}_j \leq \text{max_facilities}
\tag{1}
\end{equation}

- **Open to assign**: Cluster $i$ can only be assigned to facility $j$ if we decide to build that facility:

\begin{equation}
\text{assign}_{i,j} \leq \text{select}_{j} \quad \forall (i,j) \in \text{Pairings}
\tag{2}
\end{equation}

- **Closest store**: Cluster $i$ must be assigned to exactly one facility:

\begin{equation}
\sum_{j:(i,j) \in \text{Pairings}}\text{assign}_{i,j} = 1 \quad \forall i \in I
\tag{3}
\end{equation}

---
## Python Implementation

### Dataset Generation

In this simple example, we choose random locations for customers and facility candidates. Customers are distributed using Gaussian distributions around a few randomly chosen population centers, whereas facility locations are uniformly distributed.

```python
%matplotlib inline
import random

import gurobipy as gp
from gurobipy import GRB

import matplotlib.pyplot as plt
import numpy as np
from sklearn.cluster import MiniBatchKMeans

# Tested with Gurobi v9.0.0 and Python 3.7.0

seed = 10101
num_customers = 50000
num_candidates = 50
max_facilities = 8
num_clusters = 1000
num_gaussians = 10
threshold = 0.99

random.seed(seed)
customers_per_gaussian = np.random.multinomial(num_customers,
                                               [1/num_gaussians]*num_gaussians)
customer_locs = []
for i in range(num_gaussians):
    # each center coordinate in [-0.5, 0.5]
    center = (random.random()-0.5, random.random()-0.5)
    customer_locs += [(random.gauss(0,.1) + center[0], random.gauss(0,.1) + center[1])
                  for i in range(customers_per_gaussian[i])]
# each candidate coordinate in [-0.5, 0.5]
facility_locs = [(random.random()-0.5, random.random()-0.5)
              for i in range(num_candidates)]
print('First customer location:', customer_locs[0])
```

    First customer location: (0.33164437091949245, -0.2809884943538464)

### Preprocessing
**Clustering**

To limit the size of the optimization model, we group individual customers into clusters and optimize on these clusters. Clusters are computed using the K-means algorithm, as implemented in the scikit-learn package.

```python
kmeans = MiniBatchKMeans(n_clusters=num_clusters, init_size=3*num_clusters,
                         random_state=seed).fit(customer_locs)
memberships = list(kmeans.labels_)
centroids = list(kmeans.cluster_centers_) # Center point for each cluster
weights = list(np.histogram(memberships, bins=num_clusters)[0]) # Number of customers in each cluster
print('First cluster center:', centroids[0])
print('Weights for first 10 clusters:', weights[:10])
```

    First cluster center: [ 0.17488411 -0.22886943]
    Weights for first 10 clusters: [54, 51, 95, 20, 92, 84, 40, 28, 44, 61]

***Viable Customer-Store Pairings***

Some facilities are just too far away from a cluster center to be relevant, so let's heuristically filter all distances that exceed a given `threshold`:

```python
def dist(loc1, loc2):
    return np.linalg.norm(loc1-loc2, ord=2) # Euclidean distance

pairings = {(facility, cluster): dist(facility_locs[facility], centroids[cluster])
            for facility in range(num_candidates)
            for cluster in range(num_clusters)
            if  dist(facility_locs[facility], centroids[cluster]) < threshold}
print("Number of viable pairings: {0}".format(len(pairings.keys())))
```

    Number of viable pairings: 45908

###  Model Deployment

Build facilities from among candidate locations to minimize total distance to cluster centers:

```python
m = gp.Model("Facility location")

# Decision variables: select facility locations
select = m.addVars(range(num_candidates), vtype=GRB.BINARY, name='select')
# Decision variables: assign customer clusters to a facility location
assign = m.addVars(pairings.keys(), vtype=GRB.BINARY, name='assign')

# Deploy Objective Function
# 0. Total distance
obj = gp.quicksum(weights[cluster]
               *pairings[facility, cluster]
               *assign[facility, cluster]
               for facility, cluster in pairings.keys())
m.setObjective(obj, GRB.MINIMIZE)

# 1. Facility limit
m.addConstr(select.sum() <= max_facilities, name="Facility_limit")

# 2. Open to assign
m.addConstrs((assign[facility, cluster] <= select[facility]
             for facility, cluster in pairings.keys()),
            name="Open2assign")

# 3. Closest store
m.addConstrs((assign.sum('*', cluster) == 1
             for cluster in range(num_clusters)),
            name="Closest_store")

# Find the optimal solution
m.optimize()
```

    Using license file c:\gurobi\gurobi.lic
    Set parameter TokenServer to value SANTOS-SURFACE-
    Gurobi Optimizer version 9.0.0 build v9.0.0rc2 (win64)
    Optimize a model with 46909 rows, 45958 columns and 137774 nonzeros
    Model fingerprint: 0xe9e43325
    Variable types: 0 continuous, 45958 integer (45958 binary)
    Coefficient statistics:
      Matrix range     [1e+00, 1e+00]
      Objective range  [1e-01, 1e+02]
      Bounds range     [1e+00, 1e+00]
      RHS range        [1e+00, 8e+00]
    Presolve time: 0.20s
    Presolved: 46909 rows, 45958 columns, 137774 nonzeros
    Variable types: 0 continuous, 45958 integer (45958 binary)
    Found heuristic solution: objective 14701.441666

    Deterministic concurrent LP optimizer: primal and dual simplex
    Showing first log only...

    Concurrent spin time: 0.00s

    Solved with dual simplex

    Root relaxation: objective 6.559923e+03, 4450 iterations, 0.77 seconds

        Nodes    |    Current Node    |     Objective Bounds      |     Work
     Expl Unexpl |  Obj  Depth IntInf | Incumbent    BestBd   Gap | It/Node Time

    *    0     0               0    6559.9228888 6559.92289  0.00%     -    1s

    Explored 0 nodes (4450 simplex iterations) in 1.29 seconds
    Thread count was 8 (of 8 available processors)

    Solution count 2: 6559.92 14701.4

    Optimal solution found (tolerance 1.00e-04)
    Best objective 6.559922888758e+03, best bound 6.559922888758e+03, gap 0.0000%

## Analysis

Let's plot a map with:

- Customer locations represented as small pink dots.
- Customer cluster centroids represented as large red dots.
- Facility location candidates represented as green dots. Notice that selected locations have black lines emanating from them towards each cluster that is likely to be served by that facility.

```python
plt.figure(figsize=(8,8), dpi=150)
plt.scatter(*zip(*customer_locs), c='Pink', s=0.5)
plt.scatter(*zip(*centroids), c='Red', s=10)
plt.scatter(*zip(*facility_locs), c='Green', s=10)
assignments = [p for p in pairings if assign[p].x > 0.5]
for p in assignments:
    pts = [facility_locs[p[0]], centroids[p[1]]]
    plt.plot(*zip(*pts), c='Black', linewidth=0.1)
```

---
## Conclusions

We learned how mathematical optimization can be used to solve the Customer Assignment Problem. Moreover, it has been shown how machine learning can be used in the pre-processing so as to reduce the computational burden of big datasets. Of course, this comes at a cost, as using fewer clusters will result in coarser approximations to the global optimal solution.

---
## References

1. Drezner, Z., & Hamacher, H. W. (Eds.). (2001). Facility location: applications and theory. Springer Science & Business Media.
2. James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning. New York: springer.
3. Klose, A., & Drexl, A. (2005). Facility location models for distribution system design. European journal of operational research, 162(1), 4-29.

Copyright © 2020 Gurobi Optimization, LLC


## GeostatsPy: Variogram Calculation for Subsurface Data Analytics in Python

### Michael Pyrcz, Associate Professor, University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

### Basic Variography in Python with GeostatsPy

Here's a simple workflow with some basic variogram calcuation with irregularly sampled data. This should help you get started with variogram caculation in subsurface modeling.

#### Spatial Continuity

**Spatial Continuity** is the correlation between values over distance.

* No spatial continuity – no correlation between values over distance, random values at each location in space regardless of separation distance.

* Homogenous phenomenon have perfect spatial continuity, since all values as the same (or very similar) they are correlated.

We need a statistic to quantify spatial continuity! A convenient method is the Semivariogram.

#### The Semivariogram

Function of difference over distance.

* The expected (average) squared difference between values separated by a lag distance vector (distance and direction), $h$:

\begin{equation}
\gamma(\bf{h}) = \frac{1}{2 N(\bf{h})} \sum^{N(\bf{h})}_{\alpha=1} (z(\bf{u}_\alpha) - z(\bf{u}_\alpha + \bf{h}))^2
\end{equation}

where $z(\bf{u}_\alpha)$ and $z(\bf{u}_\alpha + \bf{h})$ are the spatial sample values at tail and head locations of the lag vector respectively.

* Calculated over a suite of lag distances to obtain a continuous function.

* the $\frac{1}{2}$ term converts a variogram into a semivariogram, but in practice the term variogram is used instead of semivariogram.
* We prefer the semivariogram because it relates directly to the covariance function, $C_x(\bf{h})$ and univariate variance, $\sigma^2_x$:

\begin{equation}
C_x(\bf{h}) = \sigma^2_x - \gamma(\bf{h})
\end{equation}

Note the correlogram is related to the covariance function as:

\begin{equation}
\rho_x(\bf{h}) = \frac{C_x(\bf{h})}{\sigma^2_x}
\end{equation}

The correlogram provides of function of the $\bf{h}-\bf{h}$ scatter plot correlation vs. lag offset $\bf{h}$.

\begin{equation}
-1.0 \le \rho_x(\bf{h}) \le 1.0
\end{equation}

#### Variogram Observations

The following are common observations for variograms that should assist with their practical use.

##### Observation \#1 - As distance increases, variability increase (in general).

This is common since in general, over greater distance offsets, there is often more difference between the head and tail samples.

In some cases, such as with spatial cyclicity of the hole effect variogram model the variogram may have negative slope over somelag distance intervals

Negative slopes at lag distances greater than half the data extent are often caused by too few pairs for a reliable variogram calculation

##### Observation \#2 - Calculated with over all possible pairs separated by lag vector, $\bf{𝐡}$.

We scan through the entire data set, searching for all possible pair combinations with all other data.  We then calculate the variogram as one half the expectation of squared difference between all pairs.

More pairs results in a more reliable measure.

##### Observation \#3 - Need to plot the sill to know the degree of correlation.

**Sill** is the variance, $\sigma^2_x$

Given stationarity of the variance, $\sigma^2_x$, and variogram $\gamma(\bf{h})$:

we can define the covariance function:

\begin{equation}
C_x(\bf{h}) = \sigma^2_x - \gamma(\bf{h})
\end{equation}

The covariance measure is a measure of similarity over distance (the mirror image of the variogram as shown by the equation above).

Given a standardized distribution $\sigma^2_x = 1.0$, the covariance, $C_x(\bf{h})$, is equal to the correlogram, $\rho_x(\bf{h})$:

\begin{equation}
\rho_x(\bf{h}) = \sigma^2_x - \gamma(\bf{h})
\end{equation}

##### Observation \#4 - The lag distance at which the variogram reaches the sill is know as the range.

At the range, knowing the data value at the tail location provides no information about a value at the head location of the lag distance vector.

##### Observation \#5 - The nugget effect, a discontinuity at the origin

Sometimes there is a discontinuity in the variogram at distances less than the minimum data spacing.  This is known as **nugget effect**.

The ratio of nugget / sill, is known as relative nugget effect (%). Modeled as a discontinuity with no correlation structure that at lags, $h \gt \epsilon$, an infinitesimal lag distance, and perfect correlation at $\bf{h} = 0$.
Caution when including nuggect effect in the variogram model as measurement error, mixing populations cause apparent nugget effect

This exercise demonstrates the semivariogram calculation with GeostatsPy. The steps include:

1. generate a 2D model with sequential Gaussian simulation
2. sample from the simulation
3. calculate and visualize experimental semivariograms

#### Objective

In the PGE 383: Stochastic Subsurface Modeling class I want to provide hands-on experience with building subsurface modeling workflows. Python provides an excellent vehicle to accomplish this. I have coded a package called GeostatsPy with GSLIB: Geostatistical Library (Deutsch and Journel, 1998) functionality that provides basic building blocks for building subsurface modeling workflows.

The objective is to remove the hurdles of subsurface modeling workflow construction by providing building blocks and sufficient examples. This is not a coding class per se, but we need the ability to 'script' workflows working with numerical methods.

#### Getting Started

Here's the steps to get setup in Python with the GeostatsPy package:

1. Install Anaconda 3 on your machine (https://www.anaconda.com/download/).
2. From Anaconda Navigator (within Anaconda3 group), go to the environment tab, click on base (root) green arrow and open a terminal.
3. In the terminal type: pip install geostatspy.
4. Open Jupyter and in the top block get started by copy and pasting the code block below from this Jupyter Notebook to start using the geostatspy functionality.

You will need to copy the data file to your working directory.  They are available here:

* Tabular data - sample_data.csv at https://git.io/fh4gm.

There are exampled below with these functions. You can go here to see a list of the available functions, https://git.io/fh4eX, other example workflows and source code.

#### Load the required libraries

The following code loads the required libraries.

```python
import geostatspy.GSLIB as GSLIB                       # GSLIB utilies, visualization and wrapper
import geostatspy.geostats as geostats                 # GSLIB methods convert to Python
```

We will also need some standard packages. These should have been installed with Anaconda 3.

```python
import os                                               # to set current working directory
import numpy as np                                      # arrays and matrix math
import pandas as pd                                     # DataFrames
import matplotlib.pyplot as plt                         # plotting
```

If you get a package import error, you may have to first install some of these packages. This can usually be accomplished by opening up a command window on Windows and then typing 'python -m pip install [package-name]'. More assistance is available with the respective package docs.

#### Set the working directory

I always like to do this so I don't lose files and to simplify subsequent read and writes (avoid including the full address each time).  Also, in this case make sure to place the required (see above) GSLIB executables in this directory or a location identified in the environmental variable *Path*.

```python
#os.chdir("c:/PGE383")                                   # set the working directory
```

#### Loading Tabular Data

Here's the command to load our comma delimited data file in to a Pandas' DataFrame object.

```python
#df = pd.read_csv("sample_data.csv")                     # read a .csv file in as a DataFrame
df = pd.read_csv(r"https://raw.githubusercontent.com/GeostatsGuy/GeoDataSets/master/sample_data.csv") # load data from Dr. Pyrcz's GitHub respository
#print(df.iloc[0:5,:])                                  # display first 4 samples in the table as a preview
df.head()                                               # we could also use this command for a table preview
```

      X
      Y
      Facies
      Porosity
      Perm
      AI

      0
      100.0
      900.0
      1.0
      0.100187
      1.363890
      5110.699751

      1
      100.0
      800.0
      0.0
      0.107947
      12.576845
      4671.458560

      2
      100.0
      700.0
      0.0
      0.085357
      5.984520
      6127.548006

      3
      100.0
      600.0
      0.0
      0.108460
      2.446678
      5201.637996

      4
      100.0
      500.0
      0.0
      0.102468
      1.952264
      3835.270322

We will work by-facies, that is separating sand and shale facies and working with them separately.  This command extracts the sand and shale 'Facies" into new DataFrames for our analysis.

```python
# Separate Sample Data by-facies - Deepcopy
df_sand = pd.DataFrame.copy(df[df['Facies'] == 1]).reset_index() # copy only 'Facies' = sand records
df_shale = pd.DataFrame.copy(df[df['Facies'] == 0]).reset_index() # copy only 'Facies' = shale records
df_sand.head()                                          # preview the sand only DataFrame
```

      index
      X
      Y
      Facies
      Porosity
      Perm
      AI

      0
      0
      100.0
      900.0
      1.0
      0.100187
      1.363890
      5110.699751

      1
      8
      100.0
      100.0
      1.0
      0.137453
      5.727603
      5823.241783

      2
      9
      200.0
      900.0
      1.0
      0.137062
      14.771314
      5621.146994

      3
      10
      200.0
      800.0
      1.0
      0.125984
      10.675436
      4292.700500

      4
      18
      300.0
      900.0
      1.0
      0.111516
      27.999817
      4183.466773

#### Summary Statistics for Tabular Data

The table includes X and Y coordinates (meters), Facies 1 and 0 (1 is sandstone and 0 interbedded sand and mudstone), Porosity (fraction), and permeability as Perm (mDarcy).

There are a lot of efficient methods to calculate summary statistics from tabular data in DataFrames. The 'describe' command provides count, mean, minimum, maximum, and quartiles all in a nice data table. We use transpose just to flip the table so that features are on the rows and the statistics are on the columns.

Let's look at and compare the sumamry statistics for sand and shale.

```python
df_sand.describe().transpose()                          # summary table of sand only DataFrame statistics
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      index
      162.0
      117.981481
      59.058816
      0.000000
      75.250000
      117.500000
      157.750000
      260.000000

      X
      162.0
      831.111111
      238.857269
      50.000000
      800.000000
      945.000000
      975.000000
      1005.000000

      Y
      162.0
      526.987654
      142.707797
      100.000000
      469.000000
      509.000000
      549.000000
      939.000000

      Facies
      162.0
      1.000000
      0.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000

      Porosity
      162.0
      0.181001
      0.037196
      0.083842
      0.155735
      0.194823
      0.207754
      0.242298

      Perm
      162.0
      293.798999
      399.989890
      0.381032
      31.996547
      155.888123
      385.177730
      2642.999829

      AI
      162.0
      3441.701689
      969.924695
      1844.166880
      2748.296631
      3179.596509
      3998.567914
      6197.834381

```python
df_shale.describe().transpose()                         # summary table of shale only DataFrame statistics
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      index
      99.0
      149.666667
      93.588330
      1.000000
      41.500000
      201.000000
      225.500000
      259.000000

      X
      99.0
      300.444444
      196.365820
      40.000000
      201.000000
      231.000000
      300.000000
      970.000000

      Y
      99.0
      425.111111
      183.665784
      29.000000
      376.000000
      416.000000
      456.000000
      989.000000

      Facies
      99.0
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000

      Porosity
      99.0
      0.100212
      0.014483
      0.058871
      0.091902
      0.101987
      0.109846
      0.141657

      Perm
      99.0
      3.568463
      6.782476
      0.033611
      0.723788
      1.530878
      3.743835
      52.500870

      AI
      99.0
      5450.493543
      728.871517
      3595.586977
      4897.828718
      5570.604405
      5951.816775
      7881.898531

The facies have strong differences in their summary statistics.

Let's transform the data grouped overall both facies (sand and shale) and separated by facies to normal score values (Gaussian distributed with a mean of 0.0 and variance of 1.0). This is required for sequential Gaussian simulation (common target for our variogram models) and the Gaussian transform assists with outliers and provides more interpretable variograms.

Let's look at the inputs for the GeostatsPy nscore program.  Note the output include an ndarray with the transformed values (in the same order as the input data in Dataframe 'df' and column 'vcol'), and the transformation table in original values and also in normal score values.

```python
geostats.nscore                                         # see the input parameters required by the nscore function
```

    <function geostatspy.geostats.nscore(df, vcol, wcol=None, ismooth=False, dfsmooth=None, smcol=0, smwcol=0)>

The following command will transform the Porosity and Permeabilty to standard normal. Note: we perform the transform for porsity and permeability and for each for the cases of:

1. all facies together
2. sand facies only
3. shale facies only

```python
#Transform to Gaussian by Facies
df['NPor'], tvPor, tnsPor = geostats.nscore(df, 'Porosity') # nscore transform for all facies porosity
df_sand['NPor'], tvPorSand, tnsPorSand = geostats.nscore(df_sand, 'Porosity')  # nscore transform for sand facies porosity
df_shale['NPor'], tvPorShale, tnsPorShale = geostats.nscore(df_shale, 'Porosity')  # nscore transform for shale facies porosity
df['NPerm'], tvPermSand, tnsPermSand = geostats.nscore(df, 'Perm')  # nscore transform for all facies permeability
df_sand['NPerm'], tvPermSand, tnsPermSand = geostats.nscore(df_sand, 'Perm')  # nscore transform for sand facies permeability
df_shale['NPerm'], tvPermShale, tnsPermShale = geostats.nscore(df_shale, 'Perm')  # nscore transform for shale facies permeability
```

Let's look at a previous of one of the DataFrames to make sure that we now have the normal score porosity and permeability.

```python
df_sand.head()                                          # preview sand DataFrame with nscore transforms
```

      index
      X
      Y
      Facies
      Porosity
      Perm
      AI
      NPor
      NPerm

      0
      0
      100.0
      900.0
      1.0
      0.100187
      1.363890
      5110.699751
      -2.158819
      -2.158819

      1
      8
      100.0
      100.0
      1.0
      0.137453
      5.727603
      5823.241783
      -0.817609
      -1.468475

      2
      9
      200.0
      900.0
      1.0
      0.137062
      14.771314
      5621.146994
      -0.839418
      -0.955141

      3
      10
      200.0
      800.0
      1.0
      0.125984
      10.675436
      4292.700500
      -1.031153
      -1.237102

      4
      18
      300.0
      900.0
      1.0
      0.111516
      27.999817
      4183.466773
      -1.621370
      -0.694044

That looks good! One way to check is to see if the relative magnitudes of the normal score transformed values match the original values.  e.g. that the normal score transform of 0.10 porosity normal score is less than the normal score transform of 0.14 porsity.  Also, the normal score transform of values close to the mean value should be close to 0.0

Let's also check the original and transformed sand and shale porosity distributions.

```python
plt.subplot(121)                                        # plot original sand and shale porosity histograms
plt.hist(df_sand['Porosity'], facecolor='red',bins=np.linspace(0.0,0.4,50),alpha=0.2,density=True,edgecolor='black',label='Sand')
plt.hist(df_shale['Porosity'], facecolor='blue',bins=np.linspace(0.0,0.4,50),alpha=0.2,density=True,edgecolor='black',label = 'Shale')
plt.xlim([0.05,0.25]); plt.ylim([0,35.0])
plt.xlabel('Porosity (fraction)'); plt.ylabel('Frequency'); plt.title('Porosity Sand and Shale')
plt.legend(loc='upper left')
plt.grid(True)

plt.subplot(122)                                        # plot nscore transformed sand and shale histograms
plt.hist(df_sand['NPor'], facecolor='red',bins=np.linspace(-3.0,3.0,40),histtype="stepfilled",alpha=0.2,density=True,cumulative=False,edgecolor='black',label='Sand')
plt.hist(df_shale['NPor'], facecolor='blue',bins=np.linspace(-3.0,3.0,40),histtype="stepfilled",alpha=0.2,density=True,cumulative=False,edgecolor='black',label='Shale')
plt.xlim([-3.0,3.0]); plt.ylim([0,0.50])
plt.xlabel('Nscore Porosity'); plt.ylabel('Density'); plt.title('Nscore Porosity Sand and Shale')
plt.legend(loc='upper left')
plt.grid(True)

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.2, wspace=0.2, hspace=0.3)
plt.show()
```

The normal score transform has correctly transformed the porosity over sand and shale facies to standard normal.  Let's plot the location maps of normal score transforms of porosity and permeability for all facies, sand facies and shale facies.

```python
cmap = plt.cm.plasma                    # color map
plt.subplot(231)
GSLIB.locmap_st(df,'X','Y','NPor',0,1000,0,1000,-3,3,'Nscore Porosity - All Facies','X (m)','Y (m)','Nscore Porosity',cmap)

plt.subplot(232)
GSLIB.locmap_st(df_sand,'X','Y','NPor',0,1000,0,1000,-3,3,'Nscore Porosity - Sand Facies','X (m)','Y (m)','Nscore Porosity',cmap)

plt.subplot(233)
GSLIB.locmap_st(df_shale,'X','Y','NPor',0,1000,0,1000,-3,3,'Nscore Porosity - Shale Facies','X (m)','Y (m)','Nscore Porosity',cmap)

plt.subplot(234)
GSLIB.locmap_st(df,'X','Y','NPerm',0,1000,0,1000,-3,3,'Nscore Permeability - All Facies','X (m)','Y (m)','Nscore Permeability',cmap)

plt.subplot(235)
GSLIB.locmap_st(df_sand,'X','Y','NPerm',0,1000,0,1000,-3,3,'Nscore Permeability - Sand Facies','X (m)','Y (m)','Nscore Permeability',cmap)

plt.subplot(236)
GSLIB.locmap_st(df_shale,'X','Y','NPerm',0,1000,0,1000,-3,3,'Nscore Permeability - Shale Facies','X (m)','Y (m)','Nscore Permeability',cmap)

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.8, wspace=0.4, hspace=0.3)
plt.show()
```

Let's see what the parameters are for the gamv, irregular data, experimental variogram calculation program.

```python
geostats.gamv                                           # see the input parameters required by the gamv function
```

    <function geostatspy.geostats.gamv(df, xcol, ycol, vcol, tmin, tmax, xlag, xltol, nlag, azm, atol, bandwh, isill)>

We can use the location maps to help determine good variogram calculation parameters.

We are ready to calculate variogram! Let's calculate isotropic variograms for the transformed normal score porosity and permeability for sand, shale and mixed (without separating sand and shale).  Some information on the parameters that I chose:

```p
tmin = -9999.; tmax = 9999.;
lag_dist = 100.0; lag_tol = 50.0; nlag = 7; bandh = 9999.9; azi = 0; atol = 90.0; isill = 1
```
* tmin, tmax are trimming limits - set to have no impact, no need to filter the data
* lag_dist, lag_tol are the lag distance, lag tolerance - set based on the common data spacing (100m) and tolerance as 100% of lag distance for additonal smoothing
* nlag is number of lags - set to extend just past 50 of the data extent
* bandh is the horizontal band width - set to have no effect
* azi is the azimuth -  it has not effect since we set atol, the azimuth tolerance, to 90.0
* isill is a boolean to standardize the distribution to a variance of 1 - it has no effect since the nscore transform sets the variance to 1.0

Let's try running these variograms and visualizing them.

```python
# Calculate Sample Data Isotropic Variograms
tmin = -9999.; tmax = 9999.;
lag_dist = 100.0; lag_tol = 100.0; nlag = 7; bandh = 9999.9; azi = 0; atol = 90.0; isill = 1

lag, por_sand_gamma, por_sand_npair = geostats.gamv(df_sand,"X","Y","NPor",tmin,tmax,lag_dist,lag_tol,nlag,azi,atol,bandh,isill)
lag, por_shale_gamma, por_shale_npair = geostats.gamv(df_shale,"X","Y","NPor",tmin,tmax,lag_dist,lag_tol,nlag,azi,atol,bandh,isill)
lag, por_gamma, por_npair = geostats.gamv(df,"X","Y","NPor",tmin,tmax,lag_dist,lag_tol,nlag,azi,atol,bandh,isill)

lag, perm_sand_gamma, perm_sand_npair = geostats.gamv(df_sand,"X","Y","NPerm",tmin,tmax,lag_dist,lag_tol,nlag,azi,atol,bandh,isill)
lag, perm_shale_gamma, perm_shale_npair = geostats.gamv(df_shale,"X","Y","NPerm",tmin,tmax,lag_dist,lag_tol,nlag,azi,atol,bandh,isill)
lag, perm_gamma, perm_npair = geostats.gamv(df,"X","Y","NPerm",tmin,tmax,lag_dist,lag_tol,nlag,azi,atol,bandh,isill)

plt.subplot(121)
plt.plot(lag,por_gamma,'x',color = 'black',label = 'All')
plt.plot(lag,por_sand_gamma,'x',color = 'orange',label = 'Sand')
plt.plot(lag,por_shale_gamma,'x',color = 'brown',label = 'Shale')
plt.plot([0,2000],[1.0,1.0],color = 'black')
plt.xlabel(r'Lag Distance $\bf(h)$, (m)')
plt.ylabel(r'$\gamma \bf(h)$')
plt.title('Isotropic NSCORE Porosity Variogram')
plt.xlim([0,700])
plt.ylim([0,1.8])
plt.legend(loc='upper left')
plt.grid(True)

plt.subplot(122)
plt.plot(lag,perm_gamma,'x',color = 'black',label = 'All')
plt.plot(lag,perm_sand_gamma,'x',color = 'orange',label = 'Sand')
plt.plot(lag,perm_shale_gamma,'x',color = 'brown',label = 'Shale')
plt.plot([0,2000],[1.0,1.0],color = 'black')
plt.xlabel(r'Lag Distance $\bf(h)$, (m)')
plt.ylabel(r'$\gamma \bf(h)$')
plt.title('Isotropic NSCORE Permeaiblity Variogram')
plt.xlim([0,700])
plt.ylim([0,1.8])
plt.legend(loc='upper left')
plt.grid(True)

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.2, wspace=0.2, hspace=0.3)
plt.show()
```

The experiental variograms have some interesting features:

* the range of the sand porosity and permeability is greater than the shale porosity and permeability range
* although the shale short range experimental points may be nosy due to sparce shale data

There is much more that you could try out, e.g. direction variograms.

I hope you find this code and demonstration useful. I'm always happy to discuss geostatistics, statistical modeling, uncertainty modeling and machine learning,

*Michael*

**Michael Pyrcz**, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin
On Twitter I'm the **GeostatsGuy** and on YouTube my lectures are on the channel, **GeostatsGuy Lectures**.


```python
# -*- coding: utf-8 -*-
#   This work is part of the Core Imaging Library (CIL) developed by CCPi
#   (Collaborative Computational Project in Tomographic Imaging), with
#   substantial contributions by UKRI-STFC and University of Manchester.

#   Licensed under the Apache License, Version 2.0 (the "License");
#   you may not use this file except in compliance with the License.
#   You may obtain a copy of the License at

#   http://www.apache.org/licenses/LICENSE-2.0

#   Unless required by applicable law or agreed to in writing, software
#   distributed under the License is distributed on an "AS IS" BASIS,
#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#   See the License for the specific language governing permissions and
#   limitations under the License.

#   Copyright 2019 UKRI-STFC, The University of Manchester
#   Authored by:    Evangelos Papoutsellis (UKRI-STFC)
```

Primal Dual Hybrid Gradient Algorithm

In this demo, we learn how to use the **Primal Dual Hybrid Algorithm (PDHG)** introduced by [Chambolle & Pock](https://hal.archives-ouvertes.fr/hal-00490826/document) for Tomography Reconstruction. We will solve the following minimisation problem under three different regularisation terms, i.e.,

* $\|\cdot\|_{1}$ or
* Tikhonov regularisation or
* with $L=\nabla$ and Total variation:

$$\begin{equation}
u^{*} =\underset{u}{\operatorname{argmin}} \frac{1}{2} \| \mathcal{A} u - g\|^{2} +
\underbrace{
\begin{cases}
\alpha\,\|u\|_{1}, & \\[10pt]
\alpha\,\|\nabla u\|_{2}^{2}, & \\[10pt]
\alpha\,\mathrm{TV}(u) + \mathbb{I}_{\{u\geq 0\}}(u).
\end{cases}}_{Regularisers}
\tag{1}
\end{equation}$$

where,

1. $g$ is the Acqusisition data obtained from the detector.

1. $\mathcal{A}$ is the projection operator ( _Radon transform_ ) that maps from an image-space to an acquisition space, i.e., $\mathcal{A} : \mathbb{X} \rightarrow \mathbb{Y}, $ where $\mathbb{X}$ is an __ImageGeometry__ and $\mathbb{Y}$ is an __AcquisitionGeometry__.

1. $\alpha$: regularising parameter that measures the trade-off between the fidelity and the regulariser terms.

1. The total variation (isotropic) is defined as $$\mathrm{TV}(u) = \|\nabla u \|_{2,1} = \sum \sqrt{ (\partial_{y}u)^{2} + (\partial_{x}u)^{2} }$$

1. $\mathbb{I}_{\{u\geq 0\}}(u) : =
\begin{cases}
0, & \mbox{ if } u\geq 0\\
\infty , & \mbox{ otherwise}
\,
\end{cases}
$, $\quad$ a non-negativity constraint for the minimiser $u$.

 Learning objectives

- Load the data using the CIL reader: `TXRMDataReader`.
- Preprocess the data using the CIL processors:  `Binner`, `TransmissionAbsorptionConverter`.
- Run FBP and SIRT reconstructions.
- Setup PDHG for 3 different regularisers: $L^{1}$, Tikhonov and Total variation.

We first import all the necessary libraries for this notebook.

```python
# Import libraries

from cil.framework import BlockDataContainer

from cil.optimisation.functions import L2NormSquared, L1Norm, BlockFunction, MixedL21Norm, IndicatorBox, TotalVariation
from cil.optimisation.operators import GradientOperator, BlockOperator
from cil.optimisation.algorithms import PDHG, SIRT

from cil.plugins.astra.operators import ProjectionOperator
from cil.plugins.astra.processors import FBP

from cil.plugins.ccpi_regularisation.functions import FGP_TV

from cil.utilities.display import show2D, show_geometry
from cil.utilities.jupyter import islicer

from cil.io import TXRMDataReader

from cil.processors import Binner, TransmissionAbsorptionConverter, Slicer

import matplotlib.pyplot as plt

import numpy as np

import os
```

# Data information

In this demo, we use the **Walnut** found in [Jørgensen_et_all](https://zenodo.org/record/4822516#.YLXyAJMzZp8). In total, there are 6 individual micro Computed Tomography datasets in the native Zeiss TXRM/TXM format. The six datasets were acquired at the 3D Imaging Center at Technical University of Denmark in 2014 (HDTomo3D in 2016) as part of the ERC-funded project High-Definition Tomography (HDTomo) headed by Prof. Per Christian Hansen.

# Load walnut data

```python
reader = TXRMDataReader()

pathname = os.path.abspath("/mnt/materials/SIRF/Fully3D/CIL/Walnut/valnut_2014-03-21_643_28/tomo-A")
data_name = "valnut_tomo-A.txrm"
filename = os.path.join(pathname,data_name )

reader.set_up(file_name=filename, angle_unit='radian')
data3D = reader.read()

# reorder data to match default order for Astra/Tigre operator
data3D.reorder('astra')

# Get Image and Acquisition geometries
ag3D = data3D.geometry
ig3D = ag3D.get_ImageGeometry()
```

## Acquisition and Image geometry information

```python
print(ag3D)
```

```python
print(ig3D)
```

# Show Acquisition geometry and full 3D sinogram.

```python
show_geometry(ag3D)
```

```python
show2D(data3D, slice_list = [('vertical',512), ('angle',800), ('horizontal',512)], cmap="inferno", num_cols=3, size=(15,15))
```

# Slice through projections

```python
islicer(data3D, direction=1, cmap="inferno")
```

## For demonstration purposes, we extract the central slice and select only 160 angles from the total 1601 angles.

1. We use the `Slicer` processor with step size of 10.
1. We use the `Binner` processor to crop and bin the acquisition data in order to reduce the field of view.
1. We use the `TransmissionAbsorptionConverter` to convert from transmission measurements to absorption based on the Beer-Lambert law.

**Note:** To avoid circular artifacts in the reconstruction space, we subtract the mean value of a background Region of interest (ROI), i.e., ROI that does not contain the walnut.

```python
# Extract vertical slice
data2D = data3D.subset(vertical='centre')

# Select every 10 angles
sliced_data = Slicer(roi={'angle':(0,1601,10)})(data2D)

# Reduce background regions
binned_data = Binner(roi={'horizontal':(120,-120,2)})(sliced_data)

# Create absorption data
absorption_data = TransmissionAbsorptionConverter()(binned_data)

# Remove circular artifacts
absorption_data -= np.mean(absorption_data.as_array()[80:100,0:30])
```

```python
# Get Image and Acquisition geometries for one slice
ag2D = absorption_data.geometry
ag2D.set_angles(ag2D.angles, initial_angle=0.2, angle_unit='radian')
ig2D = ag2D.get_ImageGeometry()
```

```python
print(" Acquisition Geometry 2D: {} with labels {}".format(ag2D.shape, ag2D.dimension_labels))
print(" Image Geometry 2D: {} with labels {}".format(ig2D.shape, ig2D.dimension_labels))
```

## Define Projection Operator
We can define our projection operator using our __astra__ __plugin__ that wraps the Astra-Toolbox library.

```python
A = ProjectionOperator(ig2D, ag2D, device = "gpu")
```

## FBP and SIRT reconstuctions

Now, let's perform simple reconstructions using the **Filtered Back Projection (FBP)** and **Simultaneous Iterative Reconstruction Technique [SIRT](../appendix.ipynb/#SIRT) .**

Recall, for FBP we type

```python

    fbp_recon = FBP(ig, ag, device = 'gpu')(absorption_data)

```

For SIRT, we type

```python

    x_init = ig.allocate()
    sirt = SIRT(initial = x_init, operator = A, data=absorption_data,
            max_iteration = 50, update_objective_interval=10)
    sirt.run(verbose=1)
    sirt_recon = sirt.solution

```

**Note**: In SIRT, a non-negative constraint can be used with

```python

    constraint=IndicatorBox(lower=0)

```

## Exercise 1: Run FBP and SIRT reconstructions

Use the code blocks described above and run FBP (`fbp_recon`) and SIRT (`sirt_recon`) reconstructions.

**Note**: To display the results, use

```python

    show2D([fbp_recon,sirt_recon], title = ['FBP reconstruction','SIRT reconstruction'], cmap = 'inferno')

```

```python
# Setup and run the FBP algorithm
fbp_recon = FBP(..., ...,  device = 'gpu')(absorption_data)

# Setup and run the SIRT algorithm, with  non-negative constraint
x_init = ig2D.allocate()
sirt = SIRT(initial = x_init,
            operator = ...,
            data= ...,
            constraint = ...,
            max_iteration = 300,
            update_objective_interval=100)
sirt.run(verbose=1)
sirt_recon = sirt.solution

# Show reconstructions
show2D([fbp_recon,sirt_recon],
       title = ['FBP reconstruction','SIRT reconstruction'],
       cmap = 'inferno', fix_range=(0,0.05))
```

## Exercise 1: Solution

```python
# Setup and run the FBP algorithm
fbp_recon = FBP(ig2D, ag2D,  device = 'gpu')(absorption_data)
```

```python
# Setup and run the SIRT algorithm, with  non-negative constraint
x_init = ig2D.allocate()
sirt = SIRT(initial = x_init,
            operator = A ,
            data = absorption_data,
            constraint = IndicatorBox(lower=0),
            max_iteration = 300,
            update_objective_interval=100)
sirt.run(verbose=1)
sirt_recon = sirt.solution
```

```python
# Show reconstructions
show2D([fbp_recon,sirt_recon],
       title = ['FBP reconstruction','SIRT reconstruction'],
       cmap = 'inferno', fix_range=(0,0.05))
```

 Why PDHG?

In the previous notebook, we presented the __Tikhonov regularisation__ for tomography reconstruction, i.e.,

$$\begin{equation}
u^{*} =\underset{u}{\operatorname{argmin}} \frac{1}{2} \| \mathcal{A} u - g\|^{2} + \alpha\|L u\|^{2}_{2}
\tag{Tikhonov}
\end{equation}$$

where we can use either the `GradientOperator` ($L = \nabla) $ or the `IdentityOperator` ($L = \mathbb{I}$). Due to the $\|\cdot\|^{2}_{2}$ terms, one can observe that the above objective function is differentiable. As shown in the previous notebook, we can use the standard `GradientDescent` algorithm namely

```python

    f1 = LeastSquares(A, absorption_data)
    D = GradientOperator(ig2D)
    f2 = OperatorCompositionFunction(L2NormSquared(),D)
    f = f1 + alpha_tikhonov*f2

    gd = GD(x_init=ig2D.allocate(), objective_function=f, step_size=None,
            max_iteration=1000, update_objective_interval = 10)
    gd.run(100, verbose=1)

```

However, this is not always the case. Consider for example an $L^{1}$ norm for the fidelity, i.e., $\|\mathcal{A} u - g\|_{1}$ or an $L^{1}$ norm of the regulariser i.e., $\|u\|_{1}$ or a non-negativity constraint $\mathbb{I}_{\{u>0\}}(u)$. An alternative is to use **Proximal Gradient Methods**, discused in the previous notebook, e.g., the `FISTA` algorithm, where we require one of the functions to be differentiable and the other to have a __simple__ proximal method, i.e., "easy to solve". For more information, we refer to [Parikh_Boyd](https://web.stanford.edu/~boyd/papers/pdf/prox_algs.pdf#page=30).

Using the __PDHG algorithm__, we can solve minimisation problems where the objective is not differentiable, and the only required assumption is convexity with __simple__ proximal problems.

 $L^{1}$ regularisation

Let $L=$`IdentityOperator` in [Tikhonov regularisation](#Tikhonov) and replace the

$$\alpha^{2}\|L u\|^{2}_{2}\quad\mbox{  with  }\quad \alpha\|u\|_{1}, $$

which results to a non-differentiable objective function. Hence, we have

$$\begin{equation}
u^{*} =\underset{u}{\operatorname{argmin}} \frac{1}{2} \| \mathcal{A} u - g\|^{2} + \alpha\|u\|_{1}
\tag{$L^{2}-L^{1}$}
\end{equation}$$

 How to setup and run PDHG?

In order to use the PDHG algorithm for the problem above, we need to express our minimisation problem into the following form:

$$\begin{equation}
\min_{u\in\mathbb{X}} \mathcal{F}(K u) + \mathcal{G}(u)
\label{PDHG_form}
\tag{2}
\end{equation}$$

where we assume that:

1. $\mathcal{F}$, $\mathcal{G}$ are __convex__ functionals:

    - $\mathcal{F}: \mathbb{Y} \rightarrow \mathbb{R}$

    - $\mathcal{G}: \mathbb{X} \rightarrow \mathbb{R}$

1. $K$ is a continuous linear operator acting from a space $\mathbb{X}$ to another space $\mathbb{Y}$ :

    $$K : \mathbb{X} \rightarrow \mathbb{Y} \quad $$

    with operator norm  defined as $$\| K \| = \max\{ \|K x\|_{\mathbb{Y}} : \|x\|_{\mathbb{X}}\leq 1 \}.$$

We can write the problem [($L^{2}-L^{1})$](#Lasso) into [(2)](#PDHG_form), if we let

1. $K = \mathcal{A} \quad \Longleftrightarrow \quad $ `K = A`

1. $\mathcal{F}: Y \rightarrow \mathbb{R}, \mbox{ with } \mathcal{F}(z) := \frac{1}{2}\| z - g \|^{2}, \quad \Longleftrightarrow \quad$ ` F = 0.5 * L2NormSquared(absorption_data)`

1. $\mathcal{G}: X \rightarrow \mathbb{R}, \mbox{ with } \mathcal{G}(z) := \alpha\|z\|_{1}, \quad \Longleftrightarrow \quad$ ` G = alpha * L1Norm()`

Hence, we can verify that with the above setting we have that [($L^{2}-L^{1})$](#Lasso)$\Rightarrow$[(2)](#PDHG_form) for $x=u$,  $$\underset{u}{\operatorname{argmin}} \frac{1}{2}\|\mathcal{A} u - g\|^{2}_{2} + \alpha\|u\|_{1} =
\underset{u}{\operatorname{argmin}} \mathcal{F}(\mathcal{A}u) + \mathcal{G}(u) = \underset{x}{\operatorname{argmin}} \mathcal{F}(Kx) + \mathcal{G}(x) $$

The algorithm is described in the [Appendix](../appendix.ipynb/#PDHG) and for every iteration, we solve two (proximal-type) subproblems, i.e., __primal & dual problems__ where
$\mbox{prox}_{\tau \mathcal{G}}(x)$ and $\mbox{prox}_{\sigma \mathcal{F^{*}}}(x)$ are the **proximal operators** of $\mathcal{G}$ and $\mathcal{F}^{*}$ (convex conjugate of $\mathcal{F}$), i.e.,

$$\begin{equation}
\mbox{prox}_{\lambda \mathcal{F}}(x) = \underset{z}{\operatorname{argmin}} \frac{1}{2}\|z - x \|^{2} + \lambda
\mathcal{F}(z) \end{equation}
$$

One application of the proximal operator is similar to a gradient step but is defined for convex and not necessarily differentiable functions.

To setup and run PDHG in CIL:

```python

    pdhg = PDHG(f = F, g = G, operator = K,
            max_iterations = 500, update_objective_interval = 100)
    pdhg.run(verbose=1)

```

**Note:** To monitor convergence, we use `pdhg.run(verbose=1)` that prints the objective value of the primal problem, or `pdhg.run(verbose=2)` that prints the objective value of the primal and dual problems, as well as the primal dual gap. Nothing is printed with `verbose=0`.

### Define operator $K$, functions $\mathcal{F}$  and $\mathcal{G}$

```python
K = A
F = 0.5 * L2NormSquared(b=absorption_data)
alpha = 0.01
G = alpha * L1Norm()
```

### Setup and run PDHG

```python
# Setup and run PDHG
pdhg_l1 = PDHG(f = F, g = G, operator = K,
            max_iteration = 500,
            update_objective_interval = 100)
pdhg_l1.run(verbose=1)
```

```python
# Show reconstuction and ground truth
show2D([pdhg_l1.solution,fbp_recon], fix_range=(0,0.05), title = ['L1 regularisation', 'FBP'], cmap = 'inferno')

# Plot middle line profile
plt.figure(figsize=(30,8))
plt.rcParams.update({'font.size': 15})
plt.rcParams.update({'lines.linewidth': 5})
plt.plot(fbp_recon.subset(horizontal_y = int(ig2D.voxel_num_y/2)).as_array(), label = 'FBP')
plt.plot(pdhg_l1.solution.subset(horizontal_y = int(ig2D.voxel_num_y/2)).as_array(), label = 'L1 regularisation')
plt.legend()
plt.title('Middle Line Profiles')
plt.show()
```

 PDHG for Total Variation Regularisation

Now, we continue with the setup of the PDHG algorithm using the Total variation regulariser appeared in [(1)](#all_reg).

Similarly, to the [($L^{2}-L^{1}$)](#Lasso) problem, we need to express [($L^{2}-TV$)](#all_reg) in the general form of [PDHG](#PDHG_form). This can be done using two different formulations:

1. Explicit formulation: All the subproblems in the PDHG algorithm have a closed form solution.
1. Implicit formulation: One of the subproblems in the PDHG algorithm is not solved explicitly but an inner solver is used.

---

 ($L^{2}-TV$) with Explicit PDHG

For the setup of the **($L^{2}-TV$) Explicit PDHG**, we let

$$\begin{align}
& f_{1}: \mathbb{Y} \rightarrow \mathbb{R}, \quad f_{1}(z_{1}) = \alpha\,\|z_{1}\|_{2,1}, \mbox{ ( the TV term ) }\\
& f_{2}: \mathbb{X} \rightarrow \mathbb{R}, \quad f_{2}(z_{2}) = \frac{1}{2}\|z_{2} - g\|_{2}^{2}, \mbox{ ( the data-fitting term ). }
\end{align}$$

```python

                                f1 = alpha * MixedL21Norm()
                                f2 = 0.5 * L2NormSquared(b=absorption_data)

```

For $z = (z_{1}, z_{2})\in \mathbb{Y}\times \mathbb{X}$, we define a separable function, e.g., [BlockFunction,](../appendix.ipynb/#BlockFunction)

$$\mathcal{F}(z) : = \mathcal{F}(z_{1},z_{2}) = f_{1}(z_{1}) + f_{2}(z_{2})$$

```python

                                         F = BlockFunction(f1, f2)

```

In order to obtain an element $z = (z_{1}, z_{2})\in \mathbb{Y}\times \mathbb{X}$, we need to define a `BlockOperator` $K$, using the two operators involved in [$L^{2}-TV$](#TomoTV), i.e., the `GradientOperator` $\nabla$ and the `ProjectionOperator` $\mathcal{A}$.

$$ \mathcal{K} =
\begin{bmatrix}
\nabla\\
\mathcal{A}
\end{bmatrix}
$$

```python

                                        Grad = GradientOperator(ig)
                                        K = BlockOperator(Grad, A)

```

Finally, we enforce a non-negativity constraint by letting $\mathcal{G} = \mathbb{I}_{\{u>0\}}(u)$ $\Longleftrightarrow$ `G = IndicatorBox(lower=0)`

Again, we can verify that with the above setting we can express our problem into [(2)](#PDHG_form), for $x=u$

$$
\begin{align}
\underset{u}{\operatorname{argmin}}\alpha\|\nabla u\|_{2,1} + \frac{1}{2}\|\mathcal{A} u - g\|^{2}_{2} + \mathbb{I}_{\{u>0\}}(u) =  \underset{u}{\operatorname{argmin}} f_{1}(\nabla u) + f_{2}(\mathcal{A}u) + \mathbb{I}_{\{u>0\}}(u) \\ = \underset{u}{\operatorname{argmin}} F(
\begin{bmatrix}
\nabla \\
\mathcal{A}
\end{bmatrix}u) + \mathbb{I}_{\{u>0\}}(u) =
\underset{u}{\operatorname{argmin}} \mathcal{F}(Ku) + \mathcal{G}(u) = \underset{x}{\operatorname{argmin}} \mathcal{F}(Kx) + \mathcal{G}(x)
\end{align}
$$

```python
# Define BlockFunction F
alpha_tv = 0.0003
f1 = alpha_tv * MixedL21Norm()
f2 = 0.5 * L2NormSquared(b=absorption_data)
F = BlockFunction(f1, f2)

# Define BlockOperator K
Grad = GradientOperator(ig2D)
K = BlockOperator(Grad, A)

# Define Function G
G = IndicatorBox(lower=0)

# Setup and run PDHG
pdhg_tv_explicit = PDHG(f = F, g = G, operator = K,
            max_iteration = 1000,
            update_objective_interval = 200)
pdhg_tv_explicit.run(verbose=1)
```

```python
# Show reconstuction and ground truth
show2D([pdhg_tv_explicit.solution,fbp_recon], fix_range=(0,0.055), title = ['TV regularisation','FBP'], cmap = 'inferno')

# Plot middle line profile
plt.figure(figsize=(30,8))
plt.rcParams.update({'font.size': 15})
plt.rcParams.update({'lines.linewidth': 5})
plt.plot(fbp_recon.subset(horizontal_y = int(ig2D.voxel_num_y/2)).as_array(), label = 'FBP')
plt.plot(pdhg_tv_explicit.solution .subset(horizontal_y = int(ig2D.voxel_num_y/2)).as_array(), label = 'TV regularisation')
plt.legend()
plt.title('Middle Line Profiles')
plt.show()
```

## Speed of PDHG convergence

The PDHG algorithm converges when $\sigma\tau\|K\|^{2}<1$, where the variable $\sigma$, $\tau$ are called the _primal and dual stepsizes_. When we setup the PDHG algorithm, the default values of $\sigma$ and $\tau$ are used:

- $\sigma=1.0$
- $\tau = \frac{1.0}{\sigma\|K\|^{2}}$,

and are not passed as arguments in the setup of PDHG. However, **the speed of the algorithm depends heavily on the choice of these stepsizes.** For the following, we encourage you to use different values, such as:

- $\sigma=\frac{1}{\|K\|}$
- $\tau =\frac{1}{\|K\|}$

where $\|K\|$ is the operator norm of $K$.

```python

normK = K.norm()
sigma = 1./normK
tau = 1./normK

PDHG(f = F, g = G, operator = K, sigma=sigma, tau=tau,
            max_iteration = 2000,
            update_objective_interval = 500)

```

The operator norm is computed using the [Power Method](https://en.wikipedia.org/wiki/Power_iteration) to approximate the greatest eigenvalue of $K$.

## Exercise 2: Setup and run PDHG algorithm for Tikhonov regularisation

Use exactly the same code as above and replace:

$$f_{1}(z_{1}) = \alpha\,\|z_{1}\|_{2,1} \mbox{ with } f_{1}(z_{1}) = \alpha\,\|z_{1}\|_{2}^{2}.$$

```python
# Define BlockFunction F
alpha_tikhonov = 0.05
f1 = ...
F = BlockFunction(f1, f2)

# Setup and run PDHG
pdhg_tikhonov_explicit = PDHG(f = F, g = G, operator = K,
            max_iteration = 500,
            update_objective_interval = 100)
pdhg_tikhonov_explicit.run(verbose=1)
```

## Exercise 2: Solution

```python
# Define BlockFunction F
alpha_tikhonov = 0.05
f1 = alpha_tikhonov * L2NormSquared()
F = BlockFunction(f1, f2)

# Setup and run PDHG
pdhg_tikhonov_explicit = PDHG(f = F, g = G, operator = K,
            max_iteration = 1000,
            update_objective_interval = 200)
pdhg_tikhonov_explicit.run(verbose=1)
```

```python
# Show reconstuction and ground truth
show2D([pdhg_tikhonov_explicit.solution,fbp_recon], fix_range=(0,0.055), title = ['Tikhonov regularisation','FBP'], cmap = 'inferno')

# Plot middle line profile
plt.figure(figsize=(30,8))
plt.rcParams.update({'font.size': 15})
plt.rcParams.update({'lines.linewidth': 5})
plt.plot(fbp_recon.subset(horizontal_y = int(ig2D.voxel_num_y/2)).as_array(), label = 'FBP')
plt.plot(pdhg_tikhonov_explicit.solution .subset(horizontal_y = int(ig2D.voxel_num_y/2)).as_array(), label = 'Tikhonov regularisation')
plt.legend()
plt.title('Middle Line Profiles')
plt.show()
```

---

 ($L^{2}-TV$) with Implicit PDHG

In the implicit PDHG, one of the proximal subproblems, i.e., $\mathrm{prox}_{\tau\mathcal{F}^{*}}$ or $\mathrm{prox}_{\sigma\mathcal{G}}$ are not solved exactly and an iterative solver is used. For the setup of the **Implicit PDHG**, we let

$$\begin{align}
& \mathcal{F}: \mathbb{Y} \rightarrow \mathbb{R}, \quad \mathcal{F}(z_{1}) = \frac{1}{2}\|z_{1} - g\|_{2}^{2}\\
& \mathcal{G}: \mathbb{X} \rightarrow \mathbb{R}, \quad \mathcal{G}(z_{2}) = \alpha\, \mathrm{TV}(z_{2}) = \|\nabla z_{2}\|_{2,1}
\end{align}$$

For the function $\mathcal{G}$, we can use the `TotalVariation` `Function` class from `CIL`. Alternatively, we can use the `FGP_TV` `Function` class from our `cil.plugins.ccpi_regularisation` that wraps regularisation routines from the [CCPi-Regularisation Toolkit](https://github.com/vais-ral/CCPi-Regularisation-Toolkit). For these functions, the `proximal` method implements an iterative solver, namely the **Fast Gradient Projection (FGP)** algorithm that solves the **dual** problem of

$$\begin{equation}
\mathrm{prox}_{\tau G}(u) = \underset{z}{\operatorname{argmin}} \frac{1}{2} \| u - z\|^{2} + \tau\,\alpha\,\mathrm{TV}(z) +  \mathbb{I}_{\{z>0\}}(z),
\end{equation}
$$

for every PDHG iteration. Hence, we need to specify the number of iterations for the FGP algorithm. In addition, we can enforce a non-negativity constraint using `lower=0.0`. For the `FGP_TV` class, we can either use `device=cpu` or `device=gpu` to speed up this inner solver.

```python

    G = alpha * FGP_TV(max_iteration=100, nonnegativity = True, device = 'gpu')

    G = alpha * TotalVariation(max_iteration=100, lower=0.)

```

## Exercise 3: Setup and run implicit PDHG algorithm with the Total variation regulariser

- Using the TotalVariation class, from CIL. This solves the TV denoising problem (using the FGP algorithm) in CPU.

- Using the FGP_TV class from the CCPi regularisation plugin.

    **Note:** In the FGP_TV implementation no pixel size information is included when in the forward and backward of the finite difference operator. Hence, we need to divide our regularisation parameter by the pixel size, e.g., $$\frac{\alpha}{\mathrm{ig2D.voxel\_size\_y}}$$

## $(L^{2}-TV)$ Implicit PDHG: using FGP_TV

```python
F = 0.5 * L2NormSquared(b=absorption_data)
G = (alpha_tv/ig2D.voxel_size_y) * ...
K = A

# Setup and run PDHG
pdhg_tv_implicit_regtk = PDHG(f = F, g = G, operator = K,
            max_iteration = 1000,
            update_objective_interval = 200)
pdhg_tv_implicit_regtk.run(verbose=1)
```

## Exercise 3: Solution

```python
F = 0.5 * L2NormSquared(b=absorption_data)
G = (alpha_tv/ig2D.voxel_size_y) * FGP_TV(max_iteration=100, device='gpu')
K = A

# Setup and run PDHG
pdhg_tv_implicit_regtk = PDHG(f = F, g = G, operator = K,
            max_iteration = 1000,
            update_objective_interval = 200)
pdhg_tv_implicit_regtk.run(verbose=1)
```

```python
# Show reconstuction and ground truth
show2D([pdhg_tv_implicit_regtk.solution,pdhg_tv_explicit.solution,
        (pdhg_tv_explicit.solution-pdhg_tv_implicit_regtk.solution).abs()],
       fix_range=[(0,0.055),(0,0.055),(0,1e-3)],
       title = ['TV (Implicit CCPi-RegTk)','TV (Explicit)', 'Absolute Difference'],
       cmap = 'inferno', num_cols=3)

# Plot middle line profile
plt.figure(figsize=(30,8))
plt.rcParams.update({'font.size': 15})
plt.rcParams.update({'lines.linewidth': 5})
plt.plot(pdhg_tv_explicit.solution.subset(horizontal_y = int(ig2D.voxel_num_y/2)).as_array(), label = 'TV (explicit)')
plt.plot(pdhg_tv_implicit_regtk.solution.subset(horizontal_y = int(ig2D.voxel_num_y/2)).as_array(), label = 'TV (implicit)')
plt.legend()
plt.title('Middle Line Profiles')
plt.show()
```

In the above comparison between explicit and implicit TV reconstructions, we observe some differences in the reconstructions and in the middle line profiles. This is due to a) the number of iterations and b) $\sigma, \tau$ values used in both the explicit and implicit setup of the PDHG algorithm. You can try more iterations with different values of $\sigma$ and $\tau$ for both cases in order to be sure that converge to the same solution.

For example, you can use:

* max_iteration = 2000
* $\sigma=\tau=\frac{1}{\|K\|}$

## $(L^{2}-TV)$ Implicit PDHG: using TotalVariation

```python
G = alpha_tv * TotalVariation(max_iteration=100, lower=0.)

# Setup and run PDHG
pdhg_tv_implicit_cil = PDHG(f = F, g = G, operator = K,
            max_iteration = 500,
            update_objective_interval = 100)
pdhg_tv_implicit_cil.run(verbose=1)
```

```python
# Show reconstuction and ground truth
show2D([pdhg_tv_implicit_regtk.solution,
        pdhg_tv_implicit_cil.solution,
        (pdhg_tv_implicit_cil.solution-pdhg_tv_implicit_regtk.solution).abs()],
       fix_range=[(0,0.055),(0,0.055),(0,1e-3)], num_cols=3,
       title = ['TV (CIL)','TV (CCPI-RegTk)', 'Absolute Difference'],
       cmap = 'inferno')

# Plot middle line profile
plt.figure(figsize=(30,8))
plt.rcParams.update({'font.size': 15})
plt.rcParams.update({'lines.linewidth': 5})
plt.plot(pdhg_tv_implicit_regtk.solution.subset(horizontal_y = int(ig2D.voxel_num_y/2)).as_array(), label = 'TV (CCPi-RegTk)')
plt.plot(pdhg_tv_implicit_cil.solution.subset(horizontal_y = int(ig2D.voxel_num_y/2)).as_array(), label = 'TV (CIL)')
plt.legend()
plt.title('Middle Line Profiles')
plt.show()
```

# FBP reconstruction with all the projection angles.

```python
binned_data3D = Binner(roi={'horizontal':(120,-120,2)})(data3D)
absorption_data3D = TransmissionAbsorptionConverter()(binned_data3D.subset(vertical=512))
absorption_data3D -= np.mean(absorption_data3D.as_array()[80:100,0:30])
ag3D = absorption_data3D.geometry
ag3D.set_angles(ag3D.angles, initial_angle=0.2, angle_unit='radian')
ig3D = ag3D.get_ImageGeometry()

fbp_recon3D = FBP(ig3D, ag3D)(absorption_data3D)
```

# Show all reconstructions

- FBP (1601 projections)
- FBP (160 projections)
- SIRT (160 projections)
- $L^{1}$ regularisation (160 projections)
- Tikhonov regularisation (160 projections)
- Total variation regularisation (160 projections)

```python
show2D([fbp_recon3D,
        fbp_recon,
        sirt_recon,
        pdhg_l1.solution,
        pdhg_tikhonov_explicit.solution,
        pdhg_tv_explicit.solution],
       title=['FBP 1601 projections', 'FBP', 'SIRT','$L^{1}$','Tikhonov','TV'],
                cmap="inferno",num_cols=3, size=(25,20), fix_range=(0,0.05))
```

## Zoom ROIs

```python
show2D([fbp_recon3D.as_array()[175:225,150:250],
        fbp_recon.as_array()[175:225,150:250],
        sirt_recon.as_array()[175:225,150:250],
        pdhg_l1.solution.as_array()[175:225,150:250],
        pdhg_tikhonov_explicit.solution.as_array()[175:225,150:250],
        pdhg_tv_implicit_regtk.solution.as_array()[175:225,150:250]],
       title=['FBP 1601 projections', 'FBP', 'SIRT','$L^{1}$','Tikhonov','TV'],
                cmap="inferno",num_cols=3, size=(25,20), fix_range=(0,0.05))
```

Conclusions

In the PDHG algorithm, the step-sizes $\sigma, \tau$ play a significant role in terms of the convergence speed. In the above problems, we used the default values:

* $\sigma = 1.0$, $\tau = \frac{1.0}{\sigma\|K\|^{2}}$

and we encourage you to try different values provided that $\sigma\tau\|K\|^{2}<1$ is satisfied. Certainly, these values are not the optimal ones and there are sevelar accelaration methods in the literature to tune these parameters appropriately, see for instance [Chambolle_Pock2010](https://hal.archives-ouvertes.fr/hal-00490826/document), [Chambolle_Pock2011](https://ieeexplore.ieee.org/document/6126441), [Goldstein et al](https://arxiv.org/pdf/1305.0546.pdf), [Malitsky_Pock](https://arxiv.org/pdf/1608.08883.pdf).

In the following notebook, we are going to present a stochastic version of PDHG, namely **SPDHG** introduced in [Chambolle et al](https://arxiv.org/pdf/1706.04957.pdf) which is extremely useful to reconstruct large datasets, e.g., 3D walnut data. The idea behind SPDHG is to split our initial dataset into smaller chunks and apply  forward and backward operations to these randomly selected subsets of the data. SPDHG has been used for different imaging applications and produces significant computational improvements
over the PDHG algorithm, see [Ehrhardt et al](https://arxiv.org/abs/1808.07150) and [Papoutsellis et al](https://arxiv.org/pdf/2102.06126.pdf).

```python

```