Table of Contents

1  Singular Value Decomposition (SVD)

1.1  Interpretation of SVD

1.1.1  Geometric Interpretation

1.1.2  Factor Interpretation

1.2  Worked Example Full SVD

1.3  Relationships with PCA

1.4  Applications

1.4.1  Dimensionality Reduction

1.4.2  Information Retrieval

1.4.3  Collaborative Filtering

2  Reference

```python
from jupyterthemes import get_themes
from jupyterthemes.stylefx import set_nb_theme
themes = get_themes()
set_nb_theme(themes[1])
```

```python
# 1. magic for inline plot
# 2. magic to print version
# 3. magic so that the notebook will reload external python modules
# 4. magic to enable retina (high resolution) plots
# https://gist.github.com/minrk/3301035
%matplotlib inline
%load_ext watermark
%load_ext autoreload
%autoreload 2
%config InlineBackend.figure_format = 'retina'

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from scipy.linalg import svd as scipy_svd
from sklearn.pipeline import Pipeline
from sklearn.datasets import load_iris
from sklearn.decomposition import PCA, TruncatedSVD
from sklearn.preprocessing import normalize, StandardScaler
from sklearn.feature_extraction.text import TfidfVectorizer

%watermark -a 'Ethen' -d -t -v -p numpy,scipy,pandas,sklearn,matplotlib
```

    Ethen 2017-11-18 21:13:40

    CPython 3.5.2
    IPython 6.2.1

    numpy 1.13.3
    scipy 0.19.1
    pandas 0.20.3
    sklearn 0.19.1
    matplotlib 2.1.0

# Singular Value Decomposition (SVD)

When conducting data analysis project, it's very common to encounter dataset that contains some useful information to the task at hand, but also contains low quality information that do not contribute too much to the end goal. When facing this issue, there are numerous ways to isolating the signal from the noise. e.g. We can employ regularization methods such as [Lasso regression](http://nbviewer.jupyter.org/github/ethen8181/machine-learning/blob/master/regularization/regularization.ipynb) to perform constrained optimization, automatically dropping uninformative features from the model or use [tree-based methods](https://github.com/ethen8181/machine-learning#trees--20161210) to identify the features that were most often used for constructing the tree.

Or use a variance maximization method such as [Principal Component Analysis (PCA)](http://nbviewer.jupyter.org/github/ethen8181/machine-learning/blob/master/dim_reduct/PCA.ipynb) that aims to transform the data into a new set of orthogonal components, ensuring that the first component aligns to the maximum variance in the dataset, and the subsequent component aligns with the next maximum component and so on. In other words, it makes a dataset more compact while preserving information. The convention method for calculating PCA requires us to compute the full covariance matrix, making it suffer from extensive use of memory and can be numerically unstable. It turns out, SVD is a method that can be used to compute PCA and obtain the principal component to transform our raw dataset.

**Singular Value Decomposition (SVD)** is a particular decomposition method that decomposes an arbitrary matrix $A$ with $m$ rows and $n$ columns (assuming this matrix also has a rank of $r$, i.e. $r$ columns of the matrix $A$ are linear independent) into a set of related matrices:

$$
\begin{align}
A = U \Sigma V^{T}
\end{align}
$$

where:

- $\Sigma$ (Sigma) is a $r * r$ non-negative, decreasing order diagonal matrix. All elements not on the main diagonal are
0 and the elements of $\Sigma$ are called the singular values. Another common notation that is used for this matrix is $S$. Thus in the following post, we'll use these two symbols interchangeably.

- $U$ is a $m * r$ orthonormal matrix and $V$ is a $n * r$ orthonormal matrix.
    - Orthogonal matrix refers to a square matrix where the columns are 90 degrees between each other and its inner dot product is zero, i.e. Given a orthogonal matrix $Q$, $Q^T Q = Q Q^T = I$ and $Q^T = Q^{-1}$.
    - Orthonormal matrix: orthogonal matrix where columns are unit vectors.

A classic pictorial representation of SVD.

## Interpretation of SVD

### Geometric Interpretation

We'll use a 2 dimensional dataset for the geometric interpretation for ease of visualization. Transformation of a matrix by $U \Sigma V^T$ can be visualized as a rotation and reflection, scaling, rotation and reflection. We'll see this as a step-by-step visualization.

Given a matrix $x = \begin{bmatrix} -10 & -10 & 20 & 20\\ -10 & 20 & 20 & -10 \end{bmatrix}$ and a transformation matrix $A = \begin{bmatrix} 1 & 0.3 \\ 0.45 & 1.2 \end{bmatrix}$.

- $V^T x$ We can see that multiplying by $V^T$ rotates and reflects the input matrix $x$. Notice the swap of colors red-blue and green-yellow indicating a reflection along the x-axis.

- $S V^T x$ Since $S$ only contains values on the diagonal, it scales the matrix. $V$ rotates the matrix to a position where the singular values now represent the scaling factor along the V-basis. In the picture below $V^Tx$ is dashed and $SV^Tx$ is solid.

- $U S V^T$ Finally, $U$ rotates and reflects the matrix back to the standard basis.

Putting all three steps into one picture below, the dashed square shows $x$ as the corners and the transformed matrix $Ax$ as the solid shape.

The most useful property of the SVD is that the axes in the new space, which represent new latent attributes, are orthogonal. Hence original attributes are expressed in terms of new attributes that are independent of each other.

```python
# we can confirm this with code
A = np.array([[1, 0.3], [0.45, 1.2]])
U, S, V = scipy_svd(A)
print('singular values:', S)

# the toy 2d matrix
x = np.array([[-10, -10, 20, 20], [-10, 20, 20, -10]]).T
x
```

    singular values: [ 1.49065822  0.71444949]

    array([[-10, -10],
           [-10,  20],
           [ 20,  20],
           [ 20, -10]])

```python
# change default font size
plt.rcParams['font.size'] = 12

# the plot is not as pretty as the diagram above,
# but hopefully it gets the point across
fig, ax = plt.subplots(1, 4, figsize = (20, 4))

ax[0].scatter(x[:, 0], x[:, 1])
ax[0].set_title('Original matrix')

temp = x @ V.T
ax[1].scatter(temp[:, 0], temp[:, 1])
ax[1].set_title('$V^Tx$')

temp = temp @ np.diag(S)
ax[2].scatter(temp[:, 0], temp[:, 1])
ax[2].set_title('$SV^Tx$')

temp = temp @ U
ax[3].scatter(temp[:, 0], temp[:, 1])
ax[3].set_title('$USV^Tx$')

plt.tight_layout()
plt.show()
```

### Factor Interpretation

Here we are given a rank 3 matrix $A$, representing ratings of movies by users.

| Name  | Matrix | Alien | Star Wars | Casablanca | Titanic |
| ----- | ------ | ----- | --------- | ---------- | ------- |
| Joe   | 1      | 1     | 1         | 0          | 0       |
| Jim   | 3      | 3     | 3         | 0          | 0       |
| John  | 4      | 4     | 4         | 0          | 0       |
| Jack  | 5      | 5     | 5         | 0          | 0       |
| Jill  | 0      | 2     | 0         | 4          | 4       |
| Jenny | 0      | 0     | 0         | 5          | 5       |
| Jane  | 0      | 1     | 0         | 2          | 2       |

Applying SVD to this matrix will give us the following decomposition:

The key to understanding what SVD offers is viewing the $r$ columns of $U$, $\Sigma$, and $V$ as representing concepts that are hidden in the original matrix. In our contrived example, we can imagine there are two concepts underlying the movies, scientific fiction and romance.

To be explicit:

- The matrix $U$ connects people to concept. For example, looking at Joe (the first row in the original matrix). The value 0.14 in the first row and first column of $U$ is smaller than some of the other entries in that column. The rationale for this is because while Joe watches only science fiction, he doesn’t rate those movies highly.
- The matrix $V$ relates movies to concept. The approximately 0.58 in each of the first three columns of the first row of $V^T$ indicates that the first three movies – The Matrix, Alien and Star Wars – each are of science-fiction genre.
- Matrix $\Sigma$ gives the strength of each of the concepts. In our example, the strength of the science-fiction concept is 12.4, while the strength of the romance concept is 9.5. Intuitively, the science-fiction concept is stronger because the data provides more information about the movies of that genre and the people who like them.
- The third concept is a bit harder to interpret, but it doesn't matter that much, because its weight, given by the third nonzero diagonal entry in $\Sigma$ is relatively low compared to the first two concepts.
- Note that the matrix decomposition doesn't know the meaning of any column in the dataset, it discovers the underlying concept and it is up to us to interpret these latent factors.

## Worked Example Full SVD

Let's step through a worked example of "Full" SVD. In practice the full version is computationally expensive, since we must calculate the full matrices $U_{mr}$, $S_{rr}$, and $V_{nr}^{T}$. The "truncated" versions of SVD are usually preferred, where we can preselect the top $k < r$ dimensions of interest and calculate $U_{mk}$, $S_{kk}$ and $V_{nk}^{T}$. But the truncated version is a topic for another day.

```python
# matrix form of the table above
rank = 3
A = np.array([
    [1, 1, 1, 0, 0],
    [3, 3, 3, 0, 0],
    [4, 4, 4, 0, 0],
    [5, 5, 5, 0, 0],
    [0, 2, 0, 4, 4],
    [0, 0, 0, 5, 5],
    [0, 1, 0, 2, 2]])

# we'll use a library to perform the svd
# so we can confirm our result with it
U, S, V = scipy_svd(A, full_matrices = False)

# we'll just print out S, a.k.a Sigma to show the numbers
# are identical to the results shown earlier
print(S)
```

    [  1.24810147e+01   9.50861406e+00   1.34555971e+00   3.04642685e-16
       0.00000000e+00]

```python
# the following cell verifies some properties of SVD

# Verify calculation of A=USV^T
print(np.allclose(A, U @ np.diag(S) @ V))

# orthonormal, columns are unit vectors (length = 1)
print(np.allclose(np.round(np.sum(U * U, axis = 0)), np.ones(S.size)))

# orthogonal, dot product of itself is equivalent to
# the identity matrix U^T U = I
print(np.allclose(U.T @ U, np.eye(S.size)))
```

    True
    True
    True

The SVD of a matrix $A$ is strongly connected to the eigenvalues of the symmetric matrices $A^{T}A$ and $AA^{T}$. We'll start with the expression for SVD: $A = U \Sigma V^T$.

$$
\begin{align}
A^T
&= (U \Sigma V^T)^T \\
&= (V^T)^T \Sigma^T U^T \\
&= V \Sigma^T U^T \\
&= V \Sigma U^T
\end{align}
$$

In the second step we use the matrix property that $(BA)^T = A^T B^T$ and in the final step $\Sigma$ is a diagonal matrix, thus $\Sigma^T = \Sigma$. Next:

$$
\begin{align}
A^T A
&= (V \Sigma U^T)(U \Sigma V^T)  \\
&= V \Sigma I \Sigma V^T \\
&= V \Sigma \Sigma V^T
\end{align}
$$

In the second step, we use the fact that $U$ is a orthonormal matrix, so $U^T U$ is an identity matrix of the appropriate size.

We now multiply both sides of this equation by $V$ to get:

$$
\begin{align}
A^T A V
&= V \Sigma^2 V^T V \\
&= V \Sigma^2 I \\
&= V \Sigma^2
\end{align}
$$

Here we use the fact that $V$ is also a orthonormal matrix, so $V^T V$ is an identity matrix of the appropriate size. Looking at the equation $A^T A V = V \Sigma^2$, we now see that $V$ is the eigenvector of the matrix $A^T A$ and $\Sigma^2$ is the diagonal matrix whose entries are the corresponding eigenvalues. i.e. $V = eig(A^T A)$

```python
AtA = A.T @ A
_, V1 = np.linalg.eig(AtA)

# Note that the possible non-uniqueness of the decomposition means
# that an axis can be flipped without changing anything fundamental,
# thus we compare whether the absolute values are relatively close
# instead of the raw value
print(np.allclose(np.abs(V1[:, :rank]), np.abs(V1[:, :rank])))
```

    True

Only $U$ remains to be computed, but it can be found in the same way we found $V$. Instead this time, we'll be starting with $A A^T$

$$
\begin{align}
A A^T U
&= (U \Sigma V^T)(V \Sigma U^T) U \\
&= V \Sigma I \Sigma V^T U \\
&= U \Sigma \Sigma U^T U \\
&= U \Sigma \Sigma I \\
&= U \Sigma^2
\end{align}
$$

In other words: $U = eig(A A^T)$

```python
AAt = A @ A.T
_, U1 = np.linalg.eig(AAt)
np.allclose(np.abs(U1[:, :rank]), np.abs(U[:, :rank]))
```

    True

```python
# notice that since this is a rank 3 matrix
# only the first 3 values of 3 contains non-zero values
np.round(S, 0)
```

    array([ 12.,  10.,   1.,   0.,   0.])

To sum this section up:

- $U$ is a $m * r$ orthonormal matrix of "left-singular" (eigen)vectors of $AA^{T}$.
- $V$ is a $n * r$ orthonormal matrix of 'right-singular' (eigen)vectors of $A^{T}A$.
- $\Sigma$ is a $r * r$ non-negative, decreasing order diagonal matrix. All elements not on the main diagonal are
0 and the elements of $\Sigma$ are called the singular values, which is the square root of nonzero eigenvalues.

For those interested, the following link contains a detail walkthrough of the computation by hand. [Notes: Singular Value Decomposition Tutorial](https://datajobs.com/data-science-repo/SVD-Tutorial-[Kirk-Baker].pdf)

## Relationships with PCA

This usage of SVD is very similar to Principal Components Analysis (PCA) and in fact several numerical software libraries actually use SVD under the hood for their PCA routines, for example `sklearn.decomposition.PCA` within scikit-learn. This is due to the fact that it is more numerically stable and it's also possible to perform a truncated SVD, which only needs us to calculate $U \Sigma V^T$ for the first $k<n$ features; this makes it far quicker to compute than the full covariance matrix as computed within PCA.

In the following section, we'll take a look at the relationship between these two methods, PCA and SVD. Recall from the documentation on [PCA](http://nbviewer.jupyter.org/github/ethen8181/machine-learning/blob/master/dim_reduct/PCA.ipynb), given the input matrix $\mathbf X$ the math behind the algorithm is to solve the eigendecomposition for the correlation matrix (assuming we standardized all features) $\mathbf C = \mathbf X^T \mathbf X / (n - 1)$. It turns out, we can represent $\mathbf C$ by a product of its eigenvectors $\mathbf W$ and diagonalized eigenvalues $\mathbf L$.

$$
\begin{align}
\mathbf C &= \mathbf W \mathbf L \mathbf W^T
\end{align}
$$

```python
# use some toy dataset
iris = load_iris()
X = iris['data']

# construct the pipeline
standardize = StandardScaler()
pca = PCA()
pipeline = Pipeline([
    ('standardize', standardize),
    ('pca', pca)
])
X_pca = pipeline.fit_transform(X)
```

```python
standardize = pipeline.named_steps['standardize']
X_std = standardize.transform(X)

# confirm the WLW^T
X_cov = np.cov(X_std.T)
eigen_values, eigen_vecs = np.linalg.eig(X_cov)
reconstructed_X = eigen_vecs @ np.diag(eigen_values) @ np.linalg.inv(eigen_vecs)
print(np.allclose(X_cov, reconstructed_X))
```

    True

After obtaining the eigenvectors, i.e. principal direrctions, we can project our raw data onto the principal axes, which are called principal component scores via the operation $\mathbf{XW}$.

As for singular decompostion, $\mathbf X = \mathbf U \mathbf \Sigma \mathbf V^T$. We can write out the correlation matrix using this form:

$$
\begin{align}
\mathbf C
&= \mathbf V \mathbf \Sigma \mathbf U^T \mathbf U \mathbf \Sigma \mathbf V^T / (n - 1) \\
&= \mathbf V \frac{\mathbf \Sigma^2}{n - 1}\mathbf V^T
\end{align}
$$

Meaning thte right singular vectors $\mathbf V$ are principal directions and that singular values are related to the eigenvalues of correlation matrix via $\mathbf L = \mathbf \Sigma^2 / (n - 1)$ And the principal component scores can be computed by: $\mathbf X \mathbf V = \mathbf U \mathbf \Sigma \mathbf V^T \mathbf V = \mathbf U \mathbf \Sigma$.

```python
# here we'll print out the eigenvectors
# learned from PCA and the V learned from svd
pca.components_
```

    array([[ 0.52237162, -0.26335492,  0.58125401,  0.56561105],
           [ 0.37231836,  0.92555649,  0.02109478,  0.06541577],
           [-0.72101681,  0.24203288,  0.14089226,  0.6338014 ],
           [-0.26199559,  0.12413481,  0.80115427, -0.52354627]])

```python
# we can do X @ V to obtain the principal component score
U, S, V = scipy_svd(X_std)
V
```

    array([[ 0.52237162, -0.26335492,  0.58125401,  0.56561105],
           [-0.37231836, -0.92555649, -0.02109478, -0.06541577],
           [ 0.72101681, -0.24203288, -0.14089226, -0.6338014 ],
           [ 0.26199559, -0.12413481, -0.80115427,  0.52354627]])

Notice that some of the signs are flipped, this is normal due to the previously stated non-uniqueness of the decomposition. We'll now wrap up this section with a diagram of PCA versus SVD:

## Applications

This is by no means an exhaustive list of SVD's application.

### Dimensionality Reduction

Due to its relationships with PCA, we can imagine a very frequent use of SVD is feature reduction. By only selecting the top $k$ singular values, we have in effect compressed the original information and represented it using fewer features. Note that because SVD is also a numerical algorithm, it is important to standardize the features to ensure the magnitude of the entries are of similar range.

### Information Retrieval

SVD has also been used extensively in information retrieval, in this particular application, it is also known as Latent Semantic Analysis (LSA) or Latent Semantic Indexing (LSI). As we'll soon see, the idea is very similar to [topic modeling](http://nbviewer.jupyter.org/github/ethen8181/machine-learning/blob/master/clustering/topic_model/LDA.ipynb). The fundamental problem in information retrieval is: given some search terms, retrieve all of the documents that contain those search terms or, perhaps more usefully, return documents whose content is semantically related to the search terms. For example, if one of the search terms was "automobile" it might be appropriate to return also documents that contain the search term "car".

One approach to this problem is: Given an information repository, we might convert a raw text to document-term matrix with one row per document and one column per word. Then convert the search term as a vector in the same space, and retrieving document vectors that are close to the search vector. There are several problems with vector-based retrieval.

- First, the space is very high dimensional. For example, a typical collection of documents can easily mention more than 100,000 words even if stemming is used (i.e., "skip", "skipping", "skipped" are all treated as the same word). This creates problems for distance measurement due to the curse of dimensionality.
- Second, it treats each word as independent, whereas in languages like English, the same word can mean two different things ("bear" a burden versus "bear" in the woods), and two different words can mean the same thing ("car" and "automobile").

By applying SVD, we can reduce the dimension to speed up the search, words with similar meanings will get mapped to a similar truncated space. We'll take at this application in the following quick example:

```python
example = [
    'Machine learning is super fun',
    'Python is super, super cool',
    'Statistics is cool, too',
    'Data science is fun',
    'Python is great for machine learning',
    'I like football',
    'Football is great to watch']

# a two-staged model pipeline,
# first convert raw words to a tfidf document-term matrix
# and apply svd decomposition after that
tfidf = TfidfVectorizer(stop_words = 'english')
svd = TruncatedSVD(n_components = 2)
pipeline = Pipeline([
    ('tfidf', tfidf),
    ('svd', svd)
])
X_lsa = pipeline.fit_transform(example)
X_lsa
```

    array([[ 0.82714832, -0.20216821],
           [ 0.64317518, -0.27989764],
           [ 0.19952711, -0.19724375],
           [ 0.24907097, -0.13828783],
           [ 0.7392593 ,  0.14892526],
           [ 0.1162772 ,  0.73645697],
           [ 0.28427388,  0.79260792]])

```python
# mapping of words to latent factors/concepts,
# i.e. each concept is a linear combination of words
tfidf = pipeline.named_steps['tfidf']
vocab = tfidf.get_feature_names()
pd.DataFrame(svd.components_, index = ['concept1', 'concept2'], columns = vocab)
```

      cool
      data
      football
      fun
      great
      learning
      like
      machine
      python
      science
      statistics
      super
      watch

      concept1
      0.211903
      0.082524
      0.123490
      0.293206
      0.283966
      0.425531
      0.048611
      0.425531
      0.343490
      0.082524
      0.083414
      0.510029
      0.100157

      concept2
      -0.175362
      -0.061554
      0.654756
      -0.124878
      0.365768
      -0.019431
      0.413619
      -0.019431
      -0.029054
      -0.061554
      -0.110779
      -0.240595
      0.375162

```python
svd = pipeline.named_steps['svd']
print('total variance explained:', np.sum(svd.explained_variance_))

# mapping of document to latent factors/concepts,
# i.e. Eech document is a linear combination of the concepts
pd.DataFrame(X_lsa, index = example, columns = ['concept1', 'concept2'])
```

    total variance explained: 0.252606886963

      concept1
      concept2

      Machine learning is super fun
      0.827148
      -0.202168

      Python is super, super cool
      0.643175
      -0.279898

      Statistics is cool, too
      0.199527
      -0.197244

      Data science is fun
      0.249071
      -0.138288

      Python is great for machine learning
      0.739259
      0.148925

      I like football
      0.116277
      0.736457

      Football is great to watch
      0.284274
      0.792608

After applying LSA, we can use the compressed features to see which documents are more similar to a particular document. The following code chunk shows the pairwise cosine similarity of all the documents.

```python
X_normed = normalize(X_lsa, axis = 1)
similarity = X_normed @ X_normed.T
pd.DataFrame(similarity, index = example, columns = example)
```

      Machine learning is super fun
      Python is super, super cool
      Statistics is cool, too
      Data science is fun
      Python is great for machine learning
      I like football
      Football is great to watch

      Machine learning is super fun
      1.000000
      0.985458
      0.857746
      0.964535
      0.905386
      -0.083026
      0.104459

      Python is super, super cool
      0.985458
      1.000000
      0.932623
      0.995359
      0.820075
      -0.251150
      -0.066049

      Statistics is cool, too
      0.857746
      0.932623
      1.000000
      0.963019
      0.558322
      -0.583514
      -0.421662

      Data science is fun
      0.964535
      0.995359
      0.963019
      1.000000
      0.761204
      -0.343126
      -0.161758

      Python is great for machine learning
      0.905386
      0.820075
      0.558322
      0.761204
      1.000000
      0.347952
      0.516841

      I like football
      -0.083026
      -0.251150
      -0.583514
      -0.343126
      0.347952
      1.000000
      0.982423

      Football is great to watch
      0.104459
      -0.066049
      -0.421662
      -0.161758
      0.516841
      0.982423
      1.000000

### Collaborative Filtering

This post is getting quite long, thus we'll just list it here that similar to the movie rating matrix in a couple of sections back, SVD can be applied to implementing recommendation system, namely collaborative filtering. I personally haven't checked it yet, but the following post seems to contain a walkthrough of SVD applied to collaborative filtering for people who are interested in diving deeper. [Blog: Matrix Factorization for Movie Recommendations in Python](https://beckernick.github.io/matrix-factorization-recommender/)

# Reference

- [Notes: Latent Semantic Analysis](http://www.datascienceassn.org/sites/default/files/users/user1/lsa_presentation_final.pdf)
- [Notes: Singular Value Decomposition Tutorial](https://datajobs.com/data-science-repo/SVD-Tutorial-[Kirk-Baker].pdf)
- [Blog: Feature Reduction using SVD](http://blog.applied.ai/feature-reduction-using-svd/)
- [Blog: Singular Value Decomposition Demystified](http://makeyourowntextminingtoolkit.blogspot.co.uk/2017/02/singular-value-decomposition-demystified.html)
- [Blog: Singular Value Decomposition (SVD) Visualisation](https://alyssaq.github.io/2015/singular-value-decomposition-visualisation/)
- [Blog: Reducing Dimensionality from Dimensionality Reduction Techniques](https://towardsdatascience.com/reducing-dimensionality-from-dimensionality-reduction-techniques-f658aec24dfe)
- [Online book: Mining Massive Dataset: Chapter 11 Dimensionality Reduction](http://infolab.stanford.edu/~ullman/mmds/ch11.pdf)
- [Online book: Understanding Complex Datasets - Data Mining with Matrix Decomposition Chapter 3: Singular Value Decomposition (SVD)](http://lnfm1.sai.msu.ru/~rastor/Books/Skillicorn-Understanding_complex_datasets_data_mining_with_matrix_decompositions.pdf)
- [StackExchange: Relationship between SVD and PCA. How to use SVD to perform PCA?](https://stats.stackexchange.com/questions/134282/relationship-between-svd-and-pca-how-to-use-svd-to-perform-pca)


# Sparse Gaussian Process by Varitional Inference
+ $p(u,f|y,X)$ are approximated by $q(u,f|y,X)$, where
$$ q(u,f|y,X) = \argmin_{\tilde{q}} \int \tilde{q}(u,f|y,X) \log \frac{\tilde{q}(u,f|y,X)}{p(y|f,X) p(u|f,X) p(u)} df du , \equiv \argmin_{\tilde{q}} \mathcal{F}(\tilde{q}) $$
$$ \tilde{q}(u,f|y,X) = \tilde{q}(u) \tilde{q}(f|u) = \tilde{q}(u) \prod_{i=1}^n \tilde{q}(f_i|u)$$
+ When we minimize $q(f|u)$, then it is a same form with $p(f|u)$. Thus, let $q(u) = N(u|\hat{u}, \hat{\Sigma}_u)$, then eventually,
\begin{equation} \mathcal{F} = \int \tilde{q}(u,f|y,X) \log \frac{1}{p(y|f,X)} df du + KL(q(u) || p(u)) \\
 = \sum_{i=1}^n \Bigl\{ \frac{1}{2\sigma^2} (y_i - a_i^T \hat{u})^2 + \frac{1}{2\sigma^2}\hat{\sigma}_i^2 + \frac{1}{2\sigma^2} a_i^T \hat{\Sigma}_u a_i \Bigr\} + \frac{1}{2} \log \frac{|K_{M,M}|}{|\hat{\Sigma}_u|} -\frac{M}{2} + \frac{1}{2} tr(K_{M,M}^{-1} \hat{\Sigma}_u) + \frac{1}{2} \hat{u}^T \hat{\Sigma}_u \hat{u}, \end{equation}
 where $a_i = K_{M,M}^{-1}K_{M,i}, \hat{\sigma}_i = K_{i,i} - K_{i,M} K_{M,M}^{-1} K_{M,i}$
+ Hence, to obtain the optimal $q(\cdot)$, we need to search an optimal $\hat{u}$ and $\hat{\Sigma}_u$ for $\mathcal{F}$:
    + By using gradient for both $\hat{u}$ and $\hat{\Sigma}_u$, we can obtain a local minimum. The gradients are
    \begin{equation} \frac{\partial}{\partial \hat{u}} \mathcal{F} = (K_{M,M}^{-1} + \frac{1}{\sigma^2} a^T a)\hat{u} - \frac{1}{\sigma^2}ay, \end{equation}
    \begin{equation} \frac{\partial}{\partial \hat{\Sigma}_u} \mathcal{F} = -\frac{1}{2}\hat{\Sigma}_u + \frac{1}{2}K_{M,M}^{-1} + \frac{1}{2\sigma^2} a^T a, \end{equation}
    where $a = (a_1, a_2, \ldots a_n)^T \in \mathrm{R}^{n \times M}$.

```python
%matplotlib inline
```

```python
import numpy as np
import matplotlib.pyplot as plt
```

# Preparation for learning

## Setting for data

```python
n = 100
M = 1
data_seed = 20190727
N = 100

x_domain = (-5, 5)
true_func = lambda x:(np.sin(4*x)/x).sum(axis = 1)
true_sigma = 1
```

## Generating data

```python
np.random.seed(data_seed)
train_X = np.random.uniform(low = x_domain[0], high = x_domain[1], size = (n,M))
train_Y = true_func(train_X) + np.random.normal(scale = true_sigma, size = n)

test_X = np.random.uniform(low = x_domain[0], high = x_domain[1], size = (N,M))
test_Y = true_func(test_X) + np.random.normal(scale = true_sigma, size = N)
```

```python
plt.scatter(train_X[:,0], train_Y)
plt.scatter(train_X[:,0], true_func(train_X))
plt.show()
```

## Setting for learning

```python
### used library
from scipy.spatial import distance_matrix
from scipy.stats import wishart
```

```python
iteration = 1000
learning_seed = 20190729
M = 10
np.random.seed(learning_seed)

theta1 = 1; theta2 = 2; theta3 = 0.0001
ln_sigma = true_sigma

cronecker_delta_sigma = 0.0001
cronecker_delta = lambda x,y: np.exp(- distance_matrix(x,y)**2/cronecker_delta_sigma)
gauss_kernel = lambda x,y,theta1, theta2, theta3: theta1 * np.exp(- distance_matrix(x,y)**2/theta2) + theta3 * cronecker_delta(x,y)
exp_kernel = lambda x,y,theta1, theta2, theta3: theta1 * np.exp(- distance_matrix(x, y, p=1)/theta2) + theta3 * cronecker_delta(x,y)
used_kernel = gauss_kernel
```

```python
sub_train_ind = np.argsort(np.random.dirichlet(alpha = np.ones(n), size = 1)).squeeze()[:M]
sub_train_X = train_X[sub_train_ind,:]
sub_train_Y = train_Y[sub_train_ind]
```

```python
### kernel calculation
train_sub_kernel = used_kernel(train_X, sub_train_X, theta1, theta2, theta3)
sub_sub_kernel = used_kernel(sub_train_X, sub_train_X, theta1, theta2, theta3)
inv_sub_sub_kernel = np.linalg.inv(sub_sub_kernel)
proj_train_sub = train_sub_kernel @ inv_sub_sub_kernel
```

```python
true_u = true_func(sub_train_X)
```

```python
### initialization
est_u = np.random.normal(size = M)
est_sigma = wishart.rvs(df = M+2, scale = np.eye(M), size = 1)
```

```python
obj_func = (np.linalg.slogdet(est_sigma)[1] - np.linalg.slogdet(inv_sub_sub_kernel)[1])/2 - M/2 +
```

```python
obj_func
```

    (1.0, 12.674021310700981)

```python
### gradient based calculation
eta0 = 1
for ite in range(iteration):
    du = (inv_sub_sub_kernel + proj_train_sub.T @ proj_train_sub) @ est_u - proj_train_sub.T @ train_Y/ln_sigma
    dsigma = -hat_sigma/2 + inv_sub_sub_kernel/2 + proj_train_sub.T @ proj_train_sub/(2*ln_sigma)
    est_u += -eta0 * du
    est_sigma += -eta0 * dsigma
    break
    pass
```

```python
true_u
```

    array([-0.29589615, -0.17757156,  2.36998793, -0.21701056, -0.18563475,
            2.22302033,  3.65597805,  0.13709608, -0.34575016,  0.02731914])

```python
du
```

    array([    76.00698253,    -59.18450109, -10073.68517859,     70.16417959,
               28.64476509,   9746.27515242,    494.2322649 ,    -40.4673718 ,
               20.35227654,    -64.63921854])

```python
est_u
```

    array([-1.50274774,  1.89437592, -0.08993591,  2.08477826, -0.85014173,
            1.64895204,  2.39740451, -0.89936495,  0.27741541, -0.4933787 ])


# Materials Project Database

[](https://colab.research.google.com/github/rhennig/EMA6938/blob/main/Notebooks/6.Materials_Databases.ipynb)

This notebook also illustrates how we can interface with the Materials Project (MP) database. We will use the MP data retrieval tool `retrieve_MP.py` to retrieve computed bulk moduli in the form of a pandas dataframe, then apply matminer's tools to populate the dataframe with descriptors/features from pymatgen, and finally fit a linear regression model from the scikit-learn library to the dataset.

### Overview

In this notebook, we will:
1. Load and examine a dataset in a pandas dataframe
2. Add descriptors to the dataframe using matminer
3. Train and visualize a linear regression machine learning methods with scikit-learn and Plotly.

```python
# Install the matminer package using pip in the current Jupyter kernel
# To use them, you may need to restart the kernel
# Under Windows, you may need to install or update the Visual Studio Tools: https://visualstudio.microsoft.com/downloads/
import sys
!{sys.executable} -m pip install matminer
```

    Requirement already satisfied: matminer in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (0.7.5)
    Requirement already satisfied: pymongo>=4.0.1 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from matminer) (4.0.1)
    Requirement already satisfied: tqdm>=4.62.3 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from matminer) (4.62.3)
    Requirement already satisfied: monty>=2022.1.12.1 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from matminer) (2022.1.12.1)
    Requirement already satisfied: scikit-learn>=1.0.1 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from matminer) (1.0.1)
    Requirement already satisfied: numpy>=1.22.0 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from matminer) (1.22.1)
    Requirement already satisfied: pymatgen>=2022.1.9 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from matminer) (2022.1.9)
    Requirement already satisfied: pandas>=1.3.5 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from matminer) (1.3.5)
    Requirement already satisfied: future>=0.18.2 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from matminer) (0.18.2)
    Requirement already satisfied: sympy>=1.9 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from matminer) (1.9)
    Requirement already satisfied: jsonschema>=4.4.0 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from matminer) (4.4.0)
    Requirement already satisfied: six>=1.16.0 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from matminer) (1.16.0)
    Requirement already satisfied: requests>=2.27.1 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from matminer) (2.27.1)
    Requirement already satisfied: attrs>=17.4.0 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from jsonschema>=4.4.0->matminer) (20.3.0)
    Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from jsonschema>=4.4.0->matminer) (0.18.0)
    Requirement already satisfied: importlib-resources>=1.4.0 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from jsonschema>=4.4.0->matminer) (5.4.0)
    Requirement already satisfied: python-dateutil>=2.7.3 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from pandas>=1.3.5->matminer) (2.8.2)
    Requirement already satisfied: pytz>=2017.3 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from pandas>=1.3.5->matminer) (2021.3)
    Requirement already satisfied: uncertainties>=3.1.4 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from pymatgen>=2022.1.9->matminer) (3.1.6)
    Requirement already satisfied: networkx>=2.2 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from pymatgen>=2022.1.9->matminer) (2.6.3)
    Requirement already satisfied: Cython>=0.29.23 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from pymatgen>=2022.1.9->matminer) (0.29.26)
    Requirement already satisfied: palettable>=3.1.1 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from pymatgen>=2022.1.9->matminer) (3.3.0)
    Requirement already satisfied: matplotlib>=1.5 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from pymatgen>=2022.1.9->matminer) (3.4.3)
    Requirement already satisfied: plotly>=4.5.0 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from pymatgen>=2022.1.9->matminer) (5.5.0)
    Requirement already satisfied: tabulate in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from pymatgen>=2022.1.9->matminer) (0.8.9)
    Requirement already satisfied: pybtex in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from pymatgen>=2022.1.9->matminer) (0.24.0)
    Requirement already satisfied: ruamel.yaml>=0.15.6 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from pymatgen>=2022.1.9->matminer) (0.17.20)
    Requirement already satisfied: spglib>=1.9.9.44 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from pymatgen>=2022.1.9->matminer) (1.16.3)
    Requirement already satisfied: scipy>=1.5.0 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from pymatgen>=2022.1.9->matminer) (1.7.3)
    Requirement already satisfied: urllib3<1.27,>=1.21.1 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from requests>=2.27.1->matminer) (1.26.7)
    Requirement already satisfied: certifi>=2017.4.17 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from requests>=2.27.1->matminer) (2021.10.8)
    Requirement already satisfied: charset-normalizer~=2.0.0 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from requests>=2.27.1->matminer) (2.0.9)
    Requirement already satisfied: idna<4,>=2.5 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from requests>=2.27.1->matminer) (3.1)
    Requirement already satisfied: threadpoolctl>=2.0.0 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from scikit-learn>=1.0.1->matminer) (3.0.0)
    Requirement already satisfied: joblib>=0.11 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from scikit-learn>=1.0.1->matminer) (0.14.1)
    Requirement already satisfied: mpmath>=0.19 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from sympy>=1.9->matminer) (1.2.1)
    Requirement already satisfied: zipp>=3.1.0 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from importlib-resources>=1.4.0->jsonschema>=4.4.0->matminer) (3.6.0)
    Requirement already satisfied: kiwisolver>=1.0.1 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from matplotlib>=1.5->pymatgen>=2022.1.9->matminer) (1.3.2)
    Requirement already satisfied: pillow>=6.2.0 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from matplotlib>=1.5->pymatgen>=2022.1.9->matminer) (8.3.2)
    Requirement already satisfied: pyparsing>=2.2.1 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from matplotlib>=1.5->pymatgen>=2022.1.9->matminer) (2.4.7)
    Requirement already satisfied: cycler>=0.10 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from matplotlib>=1.5->pymatgen>=2022.1.9->matminer) (0.11.0)
    Requirement already satisfied: tenacity>=6.2.0 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from plotly>=4.5.0->pymatgen>=2022.1.9->matminer) (8.0.1)
    Requirement already satisfied: ruamel.yaml.clib>=0.2.6 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from ruamel.yaml>=0.15.6->pymatgen>=2022.1.9->matminer) (0.2.6)
    Requirement already satisfied: latexcodec>=1.0.4 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from pybtex->pymatgen>=2022.1.9->matminer) (2.0.1)
    Requirement already satisfied: PyYAML>=3.01 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from pybtex->pymatgen>=2022.1.9->matminer) (5.4.1)

## 1. Load and process data set

## 1.1 Use Matminer to Obtain Data

We use matminer to load a data set of computed elastic properties of materials from MaterialsProject automatically in a pandas dataframe.

To download data from MaterialsProject, you need to create an account at https://materialsproject.org. Simply go the page, and "Sign in or Register." Then you click on "Dashboard" in the upper left of the screen and copy your API key.

You can either set the environment variable MAPI_KEY to your API key or simply add the API key in Python. To set the environment variable MAPI_KEY in Anaconda:

`conda env config vars set MAPI_KEY="api_key_from_materialsproject"`

To activate the environment variable, you need to restart Anaconda.

```python
from matminer.data_retrieval.retrieve_MP import MPDataRetrieval

# Set your MP API key here. If set as an environment variable 'MAPI_KEY', set it to 'None'
api_key = None
mpr = MPDataRetrieval(api_key)     # Create an adapter to the MP Database.

# criteria is to get all entries with elasticity (K_VRH is bulk modulus) data
criteria = {'elasticity.K_VRH': {'$ne': None}}

# properties are the materials attributes we want
# See https://github.com/materialsproject/mapidoc for available properties you can specify
properties = ['pretty_formula', 'spacegroup.symbol', 'elasticity.K_VRH', 'formation_energy_per_atom', 'band_gap',
              'e_above_hull', 'density', 'volume', 'nsites']

# get the data!
df_mp = mpr.get_dataframe(criteria=criteria, properties=properties)
print('Number of bulk moduli extracted = ', len(df_mp))
```

    100%|██████████| 13172/13172 [00:12<00:00, 1085.81it/s]

    Number of bulk moduli extracted =  13172

## 1.2 Explore the Dataset

The data set comes as a pandas DataFrame, which is a kind of "spreadsheet" object in Python. DataFrames have several useful methods you can use to explore and clean the data, some of which we'll explore below.

```python
df_mp.head()
```

      pretty_formula
      spacegroup.symbol
      elasticity.K_VRH
      formation_energy_per_atom
      band_gap
      e_above_hull
      density
      volume
      nsites

      material_id

      mp-1
      Cs
      Im-3m
      2.0
      0.038770
      0.0000
      0.038770
      1.935039
      114.051805
      1

      mp-1002083
      CsKICl
      P4/mmm
      3.0
      -1.834990
      4.1193
      0.062262
      2.968000
      187.068706
      4

      mp-1002227
      RbSb
      Fm-3m
      6.0
      0.322214
      0.0000
      0.741339
      3.139285
      109.614091
      2

      mp-1006112
      Na3Co
      Fm-3m
      9.0
      0.754160
      0.0000
      0.754160
      2.156369
      98.492949
      4

      mp-10074
      GeSe2
      I-42d
      10.0
      -0.254001
      1.6631
      0.000000
      4.305992
      177.823768
      6

A pandas DataFrame includes a function called `describe()` that helps determine statistics for the various numerical / categorical columns in the data.

```python
df_mp.describe()
```

      elasticity.K_VRH
      formation_energy_per_atom
      band_gap
      e_above_hull
      density
      volume
      nsites

      count
      13172.000000
      13172.000000
      13172.000000
      13172.000000
      13172.000000
      13172.000000
      13172.000000

      mean
      101.763210
      -0.847532
      0.671362
      0.096432
      6.998474
      141.718257
      7.763134

      std
      72.728987
      1.013787
      1.405085
      0.265915
      3.385482
      135.430947
      7.881010

      min
      0.000000
      -4.513010
      0.000000
      0.000000
      0.023670
      5.602384
      1.000000

      25%
      45.000000
      -1.187357
      0.000000
      0.000000
      4.474429
      66.173948
      4.000000

      50%
      85.000000
      -0.536274
      0.000000
      0.000000
      6.613811
      103.096526
      5.000000

      75%
      146.000000
      -0.241757
      0.556400
      0.049605
      8.842227
      168.836892
      9.000000

      max
      591.000000
      4.412093
      17.652600
      4.412093
      22.165012
      3390.625722
      160.000000

Sometimes, the `describe()` function will reveal outliers that indicate mistakes in the data. For example, negative hence unphysical minimum bulk/shear moduli or maximum bulk/shear moduli that are too high.

The data looks ok at first glance; meaning that there are no clear problems with the ranges of the various properties. Therefore, and we won't filter out any data.

Note that the `describe()` function only describes numerical columns by default.

## 1.3 Filter Out Unstable Materials and Negative Bulk Moduli

The data set above has some entries that correspond to thermodynamically or mechanically unstable materials. We filter these materials out using the distance from the convex hull and `K_VRH` (the Voight-Reuss-Hill average of the bulk modulus).

```python
df = df_mp

# Remove materials with negative bulk moduli
df = df[df['elasticity.K_VRH'] > 0]

# Filter materials that are unstable by 100 meV/atom or more
# against decomposition into other phases
df = df[df['e_above_hull'] < 0.1]

df.describe()
```

      elasticity.K_VRH
      formation_energy_per_atom
      band_gap
      e_above_hull
      density
      volume
      nsites

      count
      10801.000000
      10801.000000
      10801.000000
      10801.000000
      10801.000000
      10801.000000
      10801.000000

      mean
      98.768355
      -0.956631
      0.746245
      0.011081
      7.029388
      151.915624
      8.283585

      std
      68.417595
      0.958849
      1.465732
      0.021934
      3.340361
      142.600369
      8.305099

      min
      1.000000
      -4.513010
      0.000000
      0.000000
      0.107073
      6.926148
      1.000000

      25%
      45.000000
      -1.248292
      0.000000
      0.000000
      4.501844
      71.102107
      4.000000

      50%
      84.000000
      -0.591223
      0.000000
      0.000000
      6.679733
      110.953545
      6.000000

      75%
      142.000000
      -0.321101
      0.859500
      0.010991
      8.880615
      179.641389
      10.000000

      max
      551.000000
      0.099251
      17.652600
      0.099937
      22.087251
      3390.625722
      160.000000

## 1.4 Create a New Descriptor

We create a new desciptor for the volume per atom and add it to the pandas data frame.

```python
# Add a new column to the pandas dataframe for the volume per atom as a new descriptor
df['vpa'] = df['volume']/df['nsites']

# explore columns
df.head()
```

      pretty_formula
      spacegroup.symbol
      elasticity.K_VRH
      formation_energy_per_atom
      band_gap
      e_above_hull
      density
      volume
      nsites
      vpa

      material_id

      mp-1
      Cs
      Im-3m
      2.0
      0.038770
      0.0000
      0.038770
      1.935039
      114.051805
      1
      114.051805

      mp-1002083
      CsKICl
      P4/mmm
      3.0
      -1.834990
      4.1193
      0.062262
      2.968000
      187.068706
      4
      46.767177

      mp-10074
      GeSe2
      I-42d
      10.0
      -0.254001
      1.6631
      0.000000
      4.305992
      177.823768
      6
      29.637295

      mp-1009490
      O2
      C2/m
      3.0
      0.001712
      1.6301
      0.001712
      1.140970
      46.570259
      2
      23.285130

      mp-1014013
      P
      Pmna
      2.0
      0.045053
      0.8924
      0.045053
      0.384568
      534.970620
      4
      133.742655

## 1.5 Add More Descriptors

We use MatMiner’s pymatgen descriptor getter tools to add some more descriptors to our dataset.

```python
from matminer.featurizers.composition import ElementProperty
from matminer.utils.data import PymatgenData
from pymatgen.core import Composition

df["composition"] = df['pretty_formula'].map(lambda x: Composition(x))

#from matminer.featurizers.composition import ElementProperty
#from matminer.utils.data import PymatgenData
#from pymatgen.ext.matproj import MPRester

#df["composition"] = df['pretty_formula'].map(lambda x: Composition(x))

dataset = PymatgenData()
descriptors = ['row', 'group', 'atomic_mass',
               'atomic_radius', 'boiling_point', 'melting_point', 'X']
stats = ["mean", "std_dev"]

ep = ElementProperty(data_source=dataset, features=descriptors, stats=stats)
df = ep.featurize_dataframe(df, "composition")

#Remove NaN values
df = df.dropna()
df.head()
```

## 2. Fit a Linear Regression Model Using SciKitLearn

We now have a sufficiently detailed dataset to fit a linear regression model that predicts the bulk modulus. The linear model is given by
$$
y(x) = \beta_0 + \sum_{i=1}^n \beta_i x_i,
$$
where $x_i$ denotes the $n$ descriptors.

## 2.1 Define the Target Output and Relevant Descriptors

The data set above has many columns - we won't need all this data for our modeling. We try to predict `K_VRH`, the Voight-Reuss-Hill average of the bulk. We can drop most of the other output data.

```python
# Target output column
y = df['elasticity.K_VRH'].values

# Possible descriptor columns
excluded = ["elasticity.K_VRH", "pretty_formula",
            "volume", "nsites", "spacegroup.symbol", "e_above_hull", "composition"]
X = df.drop(excluded, axis=1)
print("There are {} possible descriptors:\n\n{}".format(X.shape[1], X.columns.values))
```

## 2.2 Fit the Linear Regression Model

```python
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
from sklearn.preprocessing import PolynomialFeatures
import numpy as np

lr = LinearRegression()

lr.fit(X, y)

# get fit statistics
print('Training R2   = %.3f' % round(lr.score(X, y), 3))
print('Training RMSE = %.3f' % np.sqrt(mean_squared_error(y_true=y, y_pred=lr.predict(X))))
```

```python
from sklearn.model_selection import KFold, cross_val_score, cross_val_predict

# Use 10-fold cross validation (90% training, 10% test)
crossvalidation = KFold(n_splits=10, shuffle=True)

# compute cross validation scores for random forest model
r2_scores = cross_val_score(lr, X, y, scoring='r2', cv=crossvalidation, n_jobs=1)
print(r2_scores)
mse_scores = cross_val_score(lr, X, y, scoring='neg_mean_squared_error', cv=crossvalidation, n_jobs=1)
rmse_scores = [np.sqrt(abs(s)) for s in mse_scores]

print('Cross-validation results:')
print('Folds: %i, mean R2   = %.3f' % (len(r2_scores), np.mean(np.abs(r2_scores))))
print('Folds: %i, mean RMSE = %.3f' % (len(rmse_scores), np.mean(np.abs(rmse_scores))))
```

## 2.4 Scatter Density Plot the Results with Plotly and Kernel Density Estimate

```python
import plotly.graph_objects as PlotlyFig
from scipy import stats

# a line to represent a perfect model with 1:1 prediction
xy_params = {'x_col': [0, 400],
             'y_col': [0, 400],
             'color': 'black',
             'mode': 'lines',
             'legend': None,
             'text': None,
             'size': None}

xx=y
yy=lr.predict(X)

# Calculate the point density
kde = stats.gaussian_kde([xx,yy])
zz = kde([xx,yy])

# Sort the points by density, so that the densest points are plotted last
idx = zz.argsort()
xx, yy, z = xx[idx], yy[idx], zz[idx]

fig = PlotlyFig.Figure(data=PlotlyFig.Scattergl(
    x=xx,
    y=yy,
    mode='markers',
    marker=dict(
        size=5,
        color=z, #set color equal to a variable
        colorscale='Viridis', # one of plotly colorscales
    ),
    text=df['pretty_formula']
))

fig.update_layout(xaxis_title='DFT (MP) bulk modulus (GPa)',
                  yaxis_title='Predicted bulk modulus (GPa)',
                  title='Linear regression',
                  width=800,
                  height=800,
                  font=dict(family="Helvetica",
                            size=18, color="black")
    )

fig.update_yaxes(scaleanchor="x")

fig.add_trace(PlotlyFig.Scatter(x=[0,400], y=[0,400],
                    mode='lines'))

fig.update_layout(xaxis=dict(range=[0, 400]),
                  yaxis=dict(range=[0, 400]),
                  showlegend=False)

fig.show()

# Save image, can change format by simply changing the file suffix
# Requires kaleido package, pip install -U kaleido
#fig.write_image("images/LinearRegression.jpeg")
```

```python

```


```
%matplotlib inline
from scipy import *
from IPython.display import SVG, display, Latex
import matplotlib
import matplotlib.pylab as pl
```
\iffalse

```
%config InlineBackend.figure_format = 'svg'
import json ; s = json.load( open("mplrc.json") ) ; del json
matplotlib.rcParams.update(s)
matplotlib.rcParams['figure.figsize'] = 10,4

```

```
from IPython.core.display import HTML
def css_styling():
    styles = open("custom.css", "r").read()
    return HTML(styles)
css_styling()
```

<style>
    @font-face {
        font-family: "Computer Modern";
        src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
    }
    div.cell{
        width:800px;
        margin-left:10% !important;
        margin-right:auto;
    }
    h1 {
        font-family: Candara, Cambria, serif;
    }
    h2 { text-align:center; }
    h4{
        margin-top:12px;
        margin-bottom: 3px;
       }
    div.text_cell_render{
        font-family: Candara, Calibri, Trebuchet, Geneva, sans-serif;
        line-height: 155%;
        font-size: 130%;
        width:800px;
        margin-left:auto;
        margin-right:auto;
    }
    .CodeMirror{
            background-color: #FDFCFF;
            font-family:"Aurulent Sans Mono","Source Code Pro", source-code-pro,Consolas, monospace;
    }
    .prompt{
        display: None;
    }
    .text_cell_render h5 {
        font-weight: 300;
        font-size: 16pt;
        color: #3035A1;
        font-style: italic;
        margin-bottom: .5em;
        margin-top: 0.5em;
        display: block;
    }

    .warning{
        color: rgb( 240, 20, 20 )
        }

    BODY {background-color: rgb( 252, 255, 220 )}

</style>

\fi
# MDOF System, Modal Analysis

We have to study the structure in the figure below (btw, one mass is yellow and other is blue because they're M&M's).

```
display(SVG(filename="mdof.svg"))
```

## Flexibility matrix

We'll use the Principle of Virtual Displacements, and we start drawing the diagrams of
bending moments for the three different load conditions we have loading the structure
with a unit force in the location and direction of each one of the dynamic degrees of
freedom.

```
display(SVG(filename="bending.svg"))
```

The bending moments on each stretch of beam can be represented using polynomials,
so we give a short name to the polynomial object that is defined  by the `scipy`
library, and then fill a table `m` with `NDOF` rows and as many columns as the number of different intervals of definition of the bending moments (here 6).

Note that the _single_ argument needed to create a
polynomial object is a sequence, containing all the coefficients from the highest power
to the lowest one --- e.g., in `p((1,0))` the sequence `(1,0)` that is used to create
the polynomial object has length 2, so the polynomial is ` 1*x**1 + 0*x**0`.

```
p = poly1d

m = [[p((1,0)),p((0.0,-1)),p((-0.50,0)),p((-0.5, -0.5)),p((0.50,0)),p((1.0,0))],
     [p((0,0)),p((1.0,+0)),p((+0.50,0)),p(( 0.5,  0.5)),p((0.50,0)),p((1.0,0))],
     [p((0,0)),p((0.5,+0)),p((+0.75,0)),p((-0.25,0.75)),p((0.25,0)),p((0.5,0))]]

```

Now that we have a data structure with a representation of the bending moments we

  1. initialise a matrix,
  2. create a sequence containing the normalised lengths of the beam stretches
     (only one has a length 2L)
  3. compute the `i` and `j` terms of the flexibility matrix
    - `i` and `j` are the indices used to access the rows of `m`
    - zero out `s`
    - start a cycle over the columns - aka beam stretches - of `m`
    - for each stretch, compute $M_i(x)M_j(x)$ using the nice properties that the
      multiplication of two poly objects gives a new poly object
    - compute the indefinite integral, taking advantge of a library functions that
      does this for polynomial objects
    - compute the definite integral over the (normalized) length of the current stretch
      and increment `s` accordingly
    - assign `s` to the element `i,j` of the flexibility matrix.

```
F = zeros((3,3))
l = [1,1,1,1,2,1]

for i in range(3):
    for j in range(3):
        s = 0
        for n in range(6):
            integrand = m[i][n]*m[j][n]
            integral = polyint(integrand)
            s = s + integral(l[n]) - integral(0)
        F[i,j] = s

print F*12,"/ 12"
```

    [[ 36.  -2.  -4.]
     [ -2.  24.  15.]
     [ -4.  15.  11.]] / 12

## Eigenvalues and Eigenvectors

Now we have `F`, that is an `array` object in the speech of `scipy`,
but it's easier to do what we want to do if we transform it into a `matrix` object,
that has a lot of useful properties that directly mimics the properties of a matrix
as we know them from algebra.

```
F = matrix(F)
```

The three most important properties are, the `.I` attribute that gives
the matrix inverse, the `.T` attribute that gives the matrix transpose and last but not least the overloading of the `*` operator to have matrix product between two matrices.

So we compute `K` by inversion of `F`, we construct the mass matrix `M` directly and
at his point we can import a library function and compute the eigenvalues' and eigenvectors' arrays.

```
K = F.I
print K*304/3

M = matrix("1 0 0;0 1 0;0 0 1")

from scipy.linalg import eigh

evals, evecs = eigh(K,M)
evecs = matrix(evecs)
evecs[:,0] *= -1
evecs[:,2] *= -1
```

    [[  39.  -38.   66.]
     [ -38.  380. -532.]
     [  66. -532.  860.]]

Are the eignvalues correctly sorted? Are the eigenvectors orthonormal with respect
to the structural matrices?

```
print evals
print
print evecs
print
print evecs.T*M*evecs
print
print evecs.T*K*evecs
```

    [  0.30751138   0.38746035  11.9267388 ]

    [[ 0.80015337  0.59627453  0.06489431]
     [-0.47377838  0.69467934 -0.54125287]
     [-0.36781604  0.40233978  0.83835199]]

    [[  1.00000000e+00   5.55111512e-17  -5.55111512e-17]
     [  5.55111512e-17   1.00000000e+00  -1.11022302e-16]
     [ -5.55111512e-17  -1.11022302e-16   1.00000000e+00]]

    [[  3.07511376e-01   1.15879528e-15  -5.13478149e-16]
     [  8.74300632e-16   3.87460352e-01   4.71844785e-16]
     [ -4.44089210e-16   0.00000000e+00   1.19267388e+01]]

The largest absolute off-diagonal term in `evecs.T*M*evecs` is rather small:
if `1` were an AU (iu.e., the mean Earth-Sun distance) then `1.39E-16*`would
be about 21 micron.

## Excitation

Our plan is to use an adimensional time, $\alpha=\omega_ot$, and an adimensional load,
$\bm\rho(\alpha)=p(\alpha)/p_o\begin{Bmatrix}1\\0\\0\end{Bmatrix}$.

```
def r(a):
    return where(a<4*pi, sin(0.5*a), 0.0)
```

```
a = linspace(0,6*pi, 2401)
pl.plot(a, r(a))
pl.xlabel(r'$\alpha = \omega_ot$')
pl.ylabel(r'$p(t)/p_o$')
n = 200
pl.xticks(a[::n], [r"$%5.2f\pi$"%(x/pi,) for x in a[::n]])
pl.xlim(0,6*pi)
pl.ylim(-1.05,1.05);
```

Note that the circular frq. of the excitation, $0.5\omega_o$, is quite close to the circular frequencies of the first two modes, especially the 1st one.

```
print "   modal circular frequencies:", sqrt(evals)
print "dynamic amplification factors:", 1/(1-0.25/evals)
```

       modal circular frequencies: [ 0.55453708  0.62246313  3.45351108]
    dynamic amplification factors: [ 5.34696608  2.81870624  1.02141009]

## Modal equations of motion

With $M_i = m$, $K_i = \omega_i^2 m  = \Lambda_i^2\omega_o^2 m$ and
$p_o = \delta_o \tfrac{EJ}{L^3}$, the modal eom is

$$  m\ddot q_i(t) + \Lambda^2\omega_o^2 m q_i(t) =
    \Gamma_i \frac{EJ}{L^3} \delta_o \sin (0.5 \omega_o t)$$

dividing both members by $m$, as by definition it is $\omega = \tfrac{EJ}{mL^3}$ we have

$$ \ddot q_i(t) + \Lambda_i^2\omega_o^2 q_i(t) = \Gamma_i \omega_o^2 \delta_o \sin (0.5 \omega_o t).$$

The coefficients $\Gamma_i$ are given by

$$\bm\Gamma = \bm\Psi^\text{T}\begin{Bmatrix}1\\0\\0\end{Bmatrix}.$$

```
Gamma = evecs.T*matrix('1;0;0')
print evecs
print
print Gamma
```

    [[ 0.80015337  0.59627453  0.06489431]
     [-0.47377838  0.69467934 -0.54125287]
     [-0.36781604  0.40233978  0.83835199]]

    [[ 0.80015337]
     [ 0.59627453]
     [ 0.06489431]]

For our undamped systems, subjected to a sine excitation not in resonance,
with $\Lambda_o=0.5$, the general integrals can be written

$$ q_i(a) = A_i\sin(\Lambda_ia) + B_i\cos(\Lambda_ia) + C_i\sin(\Lambda_oa).$$

### Particular Integral

With $\xi_i = C_i \sin(\Lambda_o a)$ and $\ddot\xi_i = - \omega_0^2\Lambda_o^2 C_i \sin(\Lambda_o a)$, substituting in the eom gives

$$ C_i\,\omega_o^2\left(\Lambda_i^2-\Lambda_o^2\right) \sin(\Lambda_o a) =
   \Gamma_i \omega_o^2 \delta_o \sin(\Lambda_o a)
   \qquad\Rightarrow\qquad C_i = \frac{\Gamma_i}{\Lambda_i^2-\Lambda_o^2}\delta_o.$$

```
L_o = 0.50
L   = sqrt(evals)
C = ravel(Gamma)/(evals-L_o**2)
print C
```

    [  1.39129582e+01   4.33779285e+00   5.55757144e-03]

### Forced response

$$q_i(a) = C_i\left(
    \sin(\Lambda_oa) - \frac{\Lambda_o}   {\Lambda_i} \sin(\Lambda_ia)
    \right).$$

We still miss the coefficients $\beta_i = \frac{\Lambda_o}{\Lambda_i}$, so we compute
them and immediately define two arrays of functions, the modal velocities and the modal velocities. Note that the modal displacements are normalized with respect to $\delta_o$,
while the velocities are normalized with respect to the unit velocity $\delta_o\omega_o$.

```
B = L_o/sqrt(evals)

q0_f = [lambda a,i=i: C[i] * (sin(L_o*a) - B[i]*sin(L[i]*a)) for i in range (3)]
q1_f = [lambda a,i=i: C[i] * L_o * (cos(L_o*a) - cos(L[i]*a)) for i in range (3)]
```

It's about time to display our results, first in a textual representation

```
format_q = r'$\displaystyle\frac{q_%d(a)}{\delta_o} = %+10G  \sin( 0.500 a) %+10G \sin(%9G a)$'
format_v = r'$\displaystyle\frac{\dot{q}_%d(a)}{\delta_o\omega_o} = %+10G \left(\cos( 0.500 a) - \cos(%9G a)\right)$'
for i in range(3):
    display(Latex(format_q % (i+1, C[i], -C[i]*B[i], L[i])))
    display(Latex(format_v % (i+1, C[i]*L_o, L[i])))
    print
```

$\displaystyle\frac{q_1(a)}{\delta_o} =    +13.913  \sin( 0.500 a)   -12.5447 \sin( 0.554537 a)$

$\displaystyle\frac{\dot{q}_1(a)}{\delta_o\omega_o} =   +6.95648 \left(\cos( 0.500 a) - \cos( 0.554537 a)\right)$

$\displaystyle\frac{q_2(a)}{\delta_o} =   +4.33779  \sin( 0.500 a)   -3.48438 \sin( 0.622463 a)$

$\displaystyle\frac{\dot{q}_2(a)}{\delta_o\omega_o} =    +2.1689 \left(\cos( 0.500 a) - \cos( 0.622463 a)\right)$

$\displaystyle\frac{q_3(a)}{\delta_o} = +0.00555757  \sin( 0.500 a) -0.000804626 \sin(  3.45351 a)$

$\displaystyle\frac{\dot{q}_3(a)}{\delta_o\omega_o} = +0.00277879 \left(\cos( 0.500 a) - \cos(  3.45351 a)\right)$

and eventually in a graphical rendering.

Here I have chosen to display separately, mode by mode, the modal displacement
and the modal velocity, so the plotting commands are inside a loop, as well as a final
`pl.show()` command that is necessary to obtain three separate plots (try to comment
the last line and re-execute).

```
a = linspace(0,4*pi, 1601)
n = 200

for i in range(3):
    pl.plot(a,q0_f[i](a), label=r'$q_%d(a)$'%(i+1,))
    pl.plot(a,q1_f[i](a), label=r'$\dot{q}_%d(a)$'%(i+1,))
    pl.legend(loc=0)
    pl.xlabel(r'$\alpha = \omega_ot$')
    pl.ylabel(r'$q_%d(t)/\delta_o,\quad \dot q_%d(t)/(\delta_o\omega_o)}$'%(i+1,i+1))
    pl.xticks(a[::n], [r"$%5.2f\pi$"%(x/pi,) for x in a[::n]])
    pl.xlim(0,4*pi)
    pl.show();
```

For plotting the DOF displacements, I put inside the loop only the `pl.plot` command,
so that the three components can be appreciated side by side.

```
for i in range(3):
    pl.plot(a,
            evecs[i,0]*q0_f[0](a)+
            evecs[i,1]*q0_f[1](a)+
            evecs[i,2]*q0_f[2](a),
            label=r'$x_%d(a)$'%(i+1,))
pl.xlabel(r'$\alpha = \omega_ot$')
pl.ylabel(r'$x_i(t)/\delta_o$')
pl.xticks(a[::n], [r"$%5.2f\pi$"%(x/pi,) for x in a[::n]])
pl.xlim(0,4*pi)
pl.legend(loc=8);
```

### Free Response

```
a_a = 4*pi # from a in forced range to a in free response range
for i in range(3):
    print "%d\t%+12G\t%+12G" % (i+1, q0_f[i](a_a), q1_f[i](a_a))
```

    1	     -7.9399	    +1.57072
    2	    -3.48261	    +2.09977
    3	+0.000443775	+0.000460847

```
q  = [] ; qf = [] ; pp = []

for i in range(3):
    l = L[i]
    M = matrix((( sin(l*a_a),    + cos(l*a_a)),
                ( l*cos(l*a_a),  - l*sin(l*a_a))))
    xvT = matrix(( (q0_f[i](a_a),), (q1_f[i](a_a),)))
    Af, Bf = ravel(M.I*xvT)
    def qf0(a, i=i, l=l, Af=Af, Bf=Bf):
        return Af*sin(l*a)+Bf*cos(l*a)
    qf.append(qf0)
    qf1 = lambda a: l*Af*cos(l*a) - l*Bf*sin(l*a)
    print "%d\t%+12G\t%+12G" % (i+1, qf0(4*pi), qf1(4*pi))
    format_qf = r'\frac{q_%d(a)}{\delta_o} &= %+10G \sin(%10G a) %+10G \cos(%10G a)'
    pp.append(format_qf%(i+1, Af, L[i], Bf, L[i]))
    def qi(a, i=i):
        return where(a<a_a, q0_f[i](a), qf[i](a))
    q.append(qi)
print
display(Latex(r'\begin{align}'+r'\\'.join(pp)+r'\end{align}'))
```

    1	     -7.9399	    +1.57072
    2	    -3.48261	    +2.09977
    3	+0.000443775	+0.000460847

\begin{align}\frac{q_1(a)}{\delta_o} &=   -2.83248 \sin(  0.554537 a)    -7.9399 \cos(  0.554537 a)\\\frac{q_2(a)}{\delta_o} &=   -3.37332 \sin(  0.622463 a)   -3.48261 \cos(  0.622463 a)\\\frac{q_3(a)}{\delta_o} &= -0.000133443 \sin(   3.45351 a) +0.000443775 \cos(   3.45351 a)\end{align}

### Closing down

To close this soo loong exercise, here are the plots for the modal
displacements (note the scaling factor applied to $q_3$, that would be
otherwise represented as a horizontal line)

```
a = linspace(0,8*pi,3201)

for i in range(3):
    pl.plot(a,(1,1,100)[i]*q[i](a), label=r'$%sq_%d(a)$'%(('','','100\\,')[i],i+1))
pl.xlabel(r'$\alpha = \omega_ot$')
pl.ylabel(r'$q_i(t)/\delta_o$')
pl.xticks(a[::n], [r"$%5.2f\pi$"%(x/pi,) for x in a[::n]])
pl.xlim(0,8*pi)
pl.legend(loc=0,ncol=2);
```

and for the DOF displacements.

```
for i in range(3):
    pl.plot(a,evecs[i,0]*q[0](a)+evecs[i,1]*q[1](a)+evecs[i,2]*q[2](a),
            label=r'$x_%d(a)$'%(i+1,))
pl.xlabel(r'$\alpha = \omega_ot$')
pl.ylabel(r'$x_i(t)/\delta_o$')
pl.xticks(a[::n], [r"$%5.2f\pi$"%(x/pi,) for x in a[::n]])
pl.xlim(0,8*pi)
pl.legend(loc=3,ncol=2);
```


```python
%matplotlib inline
import sympy
import numpy as np
import astropy.units as u
import uncertainties as unc
import uncertainties.unumpy as unp
import matplotlib.pyplot as plt
from scipy.optimize import minimize_scalar, fsolve
from fact.analysis import li_ma_significance
sympy.init_printing()
```

## Sensitivity Calculation
The sensitvity of an IACT is defined as the minimum flux needed to observe a (point) source with a significance of least $\sigma_{target}$ within a fixed amount of time $t_{ref}$.
After analysing the high-level IACT data you're left with the number of events from the source region from the sky and the number of events from the background region in the sky. Namely $N_{on}$ and  $N_{off}$. Usually, if not always, the source and background regions have different sizes. The ration between these is called $\alpha$.

The good old Li&Ma formula calculates the significance of an observation based on the numbers above. To calculate the sensitivity we scale the number of 'gamma' events by the relative flux.

All other counts are scaled taking into account the reference time (i.e. 50 hours)
and the actual observation time (which of course only really defined for real observations)

The target significance is pre defined and fixed (usually set to 5 sigma). So we create an equality between the target sigma and the scaled Li&Ma formula and solve for the flux.

```python
#target_significance = sympy.Symbol('\sigma_\mathrm{target}')
#eq = sympy.Equality(scaled_li_ma, target_significance)
```

Calling `sympy.solve(eq, relative_flux)`, to solve for the relative flux, yields no solution unfortunately. So we use a numerical minimizer to solve the problem. To help find a good minimizer lets plot the Li&Ma significance and the target function. Obviously the minimizer doesnt work in this case.

```python
fig, (ax1, ax2) = plt.subplots(1,2, figsize=(15, 5))

off=np.ones(200) * 10 # keep off counts constant

parameter = np.linspace(-1, 10, 200)
s = li_ma_significance(10 * parameter, off, alpha=1)

ax1.plot(parameter, s, '.')
ax1.set_ylabel('Li&Ma Significance')
ax1.set_xlabel('Relative Flux')

target_significance = 5
f = lambda relative_flux: (target_significance - li_ma_significance(150 * relative_flux, off, alpha=1))**2
ax2.plot(parameter, f(parameter), '.')

# # lets eee if a minimizer finds the minimum
minimum = minimize_scalar(f)
ax2.axvline(minimum.x, color='red', linestyle='dashed')
```

We need a significance function that is continuos and differentiable. Lets try to modifiy the Li&Ma method accordingly.

```python
# def differentiable_li_ma_significance(n_on, n_off, alpha=1):
#     scalar = np.isscalar(n_on)

#     n_on = np.array(n_on, copy=False, ndmin=1)
#     n_off = np.array(n_off, copy=False, ndmin=1)

#     # find the indeces in the array for which the sign should be flipped
#     m = n_on < n_off*alpha

#     p_on = n_on / (n_on + n_off)
#     p_off = n_off / (n_on + n_off)

#     t1 = n_on * np.log(((1 + alpha) / alpha) * p_on)
#     t2 = n_off * np.log((1 + alpha) * p_off)

#     ts = (t1 + t2)
#     significance = np.sqrt(ts * 2)
#     if len(m)> 0:
#         significance[m] = -significance[m]
#     if scalar:
#         return significance[0]
#     return significance

```

Or simply add bounds to the optimizer.

```python
on = 200
off = 10
parameter = np.linspace(-3, 3, 200)
f = lambda relative_flux: (target_significance - li_ma_significance((on - off) * relative_flux + off, off, alpha=1))**2
plt.plot(parameter, f(parameter), '.')
plt.ylabel('target function value')

# lets see if a minimizer finds the minimum
minimum = minimize_scalar(f)
plt.axvline(minimum.x, color='red', linestyle='dashed')

minimum = minimize_scalar(f, bounds=(1e-10, 100), method='bounded')
plt.axvline(minimum.x, color='green', linestyle='dashed')
```

Below you will find a function implementing the Li&Ma formula and a function minimizing the function versus the target significance of 5 sigma.

```python
@u.quantity_input(t_obs=u.hour, t_ref=u.hour)
def relative_sensitivity(
        n_on,
        n_off,
        alpha,
        target_significance=5,
        ):
    '''
    Calculate the relative sensitivity defined as the flux
    relative to the reference source that is detectable with
    significance in t_ref.

    Parameters
    ----------
    n_on: int or array-like
        Number of signal-like events for the on observations
    n_off: int or array-like
        Number of signal-like events for the off observations
    alpha: float
        Scaling factor between on and off observations.
        1 / number of off regions for wobble observations.
    target_significance: float
        Significance necessary for a detection

    Returns
    ----------
    The relative flux neccessary to detect the source with the given target significance.
    '''
    scale = []
    for on, off in zip(n_on, n_off):
        if on < off*alpha or off == 0:
            scale.append(np.inf)
            continue
        def f(relative_flux):
            s = (target_significance - li_ma_significance((on - off) * relative_flux + off, off, alpha=alpha))**2
            return s

        s = minimize_scalar(f, bounds=(1e-12, 100), method='bounded')

        scale.append(s.x)
    return scale
```

```python
n_on =  [4, 1, 200, 400, 50, 200, 20000]
n_off = [2, 0, 10, 10, 10, 10, 10]

s = relative_sensitivity(n_on, n_off, alpha=1, target_significance=5)
print('relative sensitivity is: {}'.format(s))
```

    relative sensitivity is: [12.239475768994886, inf, 0.18904943918602754, 0.09210194632997426, 0.89798928277450285, 0.18904943918602754, 0.0017985381089989318]


# ForestDML, ForestDRLearner, OrthoForest and CausalForest: Basic Example

We depict the performance of our `ForestDML`, `ForestDRLearner`, `OrthoForest` and `CausalForest` estimators on the same data generating process as the one used in the tutorial page of the grf package (see https://github.com/grf-labs/grf#usage-examples). This is mostly for qualitative comparison and verification purposes among our implementation of variants of Causal Forests and the implementation in the grf R package.

```python
# Helper imports
import numpy as np
import matplotlib.pyplot as plt
import matplotlib
%matplotlib inline
```

# DGP

We use the following data generating process (DGP) from [here](https://github.com/grf-labs/grf#usage-examples):

\begin{align}
X \sim& \text{Normal}(0,\, I_{p})\\
T =& \text{Binomial}(1, .4 + .2 \cdot 1\{X[0] > 0\})\\
Y =& (X[0] \cdot 1\{X[0] > 0\}) \cdot T + X[1] + X[2] \cdot 1\{X[2] < 0\} + \epsilon, &\; \epsilon \sim \text{Normal}(0, 1)\\
\end{align}

We use $p=10$ and draw $n=2000$ samples from this DGP.

```python
import numpy as np
import scipy.special
np.random.seed(123)
n = 2000
p = 10
X = np.random.normal(size=(n, p))
true_propensity = lambda x: .4 + .2 * (x[:, 0] > 0)
true_effect = lambda x: (x[:, 0] * (x[:, 0] > 0))
true_conf = lambda x: x[:, 1] + np.clip(x[:, 2], - np.inf, 0)
T = np.random.binomial(1, true_propensity(X))
Y =  true_effect(X) * T + true_conf(X) + np.random.normal(size=(n,))
```

### Cross-Validated Forest Nuisance Models

We use forest based estimators (Gradient Boosted Forests or Random Forests) as nuisance models. For the meta-learner versions of our forest based estimators, we also use a generic forest estimator even as a final model. The hyperparameters of the forest models (e.g. number of estimators, max depth, min leaf size) is chosen via cross validation. We also choose among Gradient or Random Forests via cross validation

```python
from econml.sklearn_extensions.model_selection import GridSearchCVList
from sklearn.linear_model import Lasso, LogisticRegression
from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.base import clone
from econml.sklearn_extensions.linear_model import WeightedLasso

def first_stage_reg():
    return GridSearchCVList([Lasso(),
                             RandomForestRegressor(n_estimators=100, random_state=123),
                             GradientBoostingRegressor(random_state=123)],
                             param_grid_list=[{'alpha': [.001, .01, .1, 1, 10]},
                                               {'max_depth': [3, None],
                                               'min_samples_leaf': [10, 50]},
                                              {'n_estimators': [50, 100],
                                               'max_depth': [3],
                                               'min_samples_leaf': [10, 30]}],
                             cv=5,
                             scoring='neg_mean_squared_error')

def first_stage_clf():
    return GridSearchCVList([LogisticRegression(),
                             RandomForestClassifier(n_estimators=100, random_state=123),
                             GradientBoostingClassifier(random_state=123)],
                             param_grid_list=[{'C': [0.01, .1, 1, 10, 100]},
                                              {'max_depth': [3, 5],
                                               'min_samples_leaf': [10, 50]},
                                              {'n_estimators': [50, 100],
                                               'max_depth': [3],
                                               'min_samples_leaf': [10, 30]}],
                             cv=5,
                             scoring='neg_mean_squared_error')

def final_stage():
    return GridSearchCVList([WeightedLasso(),
                             RandomForestRegressor(n_estimators=100, random_state=123)],
                             param_grid_list=[{'alpha': [.001, .01, .1, 1, 10]},
                                              {'max_depth': [3, 5],
                                               'min_samples_leaf': [10, 50]}],
                             cv=5,
                             scoring='neg_mean_squared_error')
```

```python
model_y = clone(first_stage_reg().fit(X, Y).best_estimator_)
model_y
```

    GradientBoostingRegressor(min_samples_leaf=30, n_estimators=50,
                              random_state=123)

```python
model_t = clone(first_stage_clf().fit(X, T).best_estimator_)
model_t
```

    RandomForestClassifier(max_depth=3, min_samples_leaf=10, random_state=123)

# DML Estimators

```python
from econml.dml import CausalForestDML
from sklearn.ensemble import RandomForestRegressor, RandomForestClassifier
from sklearn.ensemble import GradientBoostingRegressor, GradientBoostingClassifier
from sklearn.dummy import DummyRegressor, DummyClassifier

n_samples, n_features = X.shape
subsample_fr_ = (n_samples/2)**(1-1/(2*n_features+2))/(n_samples/2)
est = CausalForestDML(model_y=model_y,
                      model_t=model_t,
                      discrete_treatment=True,
                      cv=3,
                      n_estimators=4000,
                      random_state=123)
est.tune(Y, T, X=X).fit(Y, T, X=X, cache_values=True)
```

    <econml.dml.causal_forest.CausalForestDML at 0x1a07d0cd188>

```python
from econml.dml import NonParamDML
est2 = NonParamDML(model_y=model_y,
                   model_t=model_t,
                   cv=3,
                   discrete_treatment=True,
                   model_final=final_stage())
est2.fit(Y, T, X=X)
```

    <econml.dml.dml.NonParamDML at 0x1a07cee4708>

```python
X_test = np.zeros((100, p))
X_test[:, 0] = np.linspace(-2, 2, 100)
```

```python
pred = est.effect(X_test)
lb, ub = est.effect_interval(X_test, alpha=0.01)
```

```python
pred2 = est2.effect(X_test)
```

```python
plt.figure(figsize=(15, 5))
plt.subplot(1, 2, 1)
plt.plot(X_test[:, 0], true_effect(X_test), '--')
plt.plot(X_test[:, 0], pred2, label='nonparamdml')
plt.plot(X_test[:, 0], pred, label='forestdml (causal forest)')
plt.fill_between(X_test[:, 0], lb, ub, alpha=.4, label='honestrf_ci')
plt.legend()
plt.show()
```

```python
np.mean((true_effect(X) - est.effect(X))**2)
```

```python
np.mean((true_effect(X) - est2.effect(X))**2)
```

### First Stage Learned Models

```python
# Model T
plt.figure(figsize=(15, 5))
plt.subplot(1, 2, 1)
plt.title('honestrf')
for mdls in est.models_t:
    for mdl in mdls:
        plt.plot(X_test[:, 0], mdl.predict_proba(X_test)[:, 1])
plt.plot(X_test[:, 0], true_propensity(X_test), '--', label='truth')
plt.legend()

plt.subplot(1, 2, 2)
plt.title('rf')
for mdls in est2.models_t:
    for mdl in mdls:
        plt.plot(X_test[:, 0], mdl.predict_proba(X_test)[:, 1])
plt.plot(X_test[:, 0], true_propensity(X_test), '--', label='truth')
plt.legend()
plt.show()
```

```python
# Model Y
plt.figure(figsize=(15, 5))
plt.subplot(1, 2, 1)
plt.title('honestrf')
for mdls in est.models_y:
    for mdl in mdls:
        plt.plot(X_test[:, 0], mdl.predict(X_test))
plt.plot(X_test[:, 0], true_effect(X_test) * true_propensity(X_test) + true_conf(X_test), '--', label='truth')
plt.legend()

plt.subplot(1, 2, 2)
plt.title('rf')
for mdls in est2.models_y:
    for mdl in mdls:
        plt.plot(X_test[:, 0], mdl.predict(X_test))
plt.plot(X_test[:, 0], true_effect(X_test) * true_propensity(X_test) + true_conf(X_test), '--', label='truth')
plt.legend()
plt.show()
```

### Interpretability of CATE Model of NonParamDML with SHAP

```python
import shap
import string

feature_names = list(string.ascii_lowercase)[:X.shape[1]]
# explain the model's predictions using SHAP values
shap_values = est.shap_values(X[:100], feature_names=feature_names)
```

     97%|=================== | 97/100 [00:41<00:01]

```python
# visualize the first prediction's explanation (use matplotlib=True to avoid Javascript)
shap.force_plot(shap_values["Y0"]["T0_1"][0], matplotlib=True)
```

```python
shap.summary_plot(shap_values["Y0"]["T0_1"])
```

# DRLearner

```python
model_regression = clone(first_stage_reg().fit(np.hstack([T.reshape(-1, 1), X]), Y).best_estimator_)
model_regression
```

    GradientBoostingRegressor(min_samples_leaf=30, n_estimators=50,
                              random_state=123)

```python
from econml.dr import ForestDRLearner
from sklearn.dummy import DummyRegressor, DummyClassifier

est = ForestDRLearner(model_regression=model_y,
                      model_propensity=model_t,
                      cv=3,
                      n_estimators=4000,
                      min_samples_leaf=10,
                      verbose=0,
                      min_weight_fraction_leaf=.005)
est.fit(Y, T, X=X)
```

    <econml.dr._drlearner.ForestDRLearner at 0x1a07f5b6248>

```python
from econml.dr import DRLearner
est2 = DRLearner(model_regression=model_y,
                 model_propensity=model_t,
                 model_final=final_stage(),
                 cv=3)
est2.fit(Y, T.reshape((-1, 1)), X=X)
```

    <econml.dr._drlearner.DRLearner at 0x1a006e87688>

```python
X_test = np.zeros((100, p))
X_test[:, 0] = np.linspace(-2, 2, 100)
```

```python
pred = est.effect(X_test)
lb, ub = est.effect_interval(X_test, alpha=0.01)
```

```python
pred2 = est2.effect(X_test)
```

```python
plt.figure(figsize=(15, 5))
plt.subplot(1, 2, 1)
plt.plot(X_test[:, 0], true_effect(X_test), '--')
plt.plot(X_test[:, 0], pred2, label='nonparamdml')
plt.plot(X_test[:, 0], pred, label='forestdml (causal forest)')
plt.fill_between(X_test[:, 0], lb, ub, alpha=.4, label='honestrf_ci')
plt.legend()
plt.show()
```

### First stage nuisance models

```python
# Model T
plt.figure(figsize=(15, 5))
plt.subplot(1, 2, 1)
plt.title('honestrf')
for mdls in est.models_propensity:
    for mdl in mdls:
        plt.plot(X_test[:, 0], mdl.predict_proba(X_test)[:, 1])
plt.plot(X_test[:, 0], true_propensity(X_test), '--', label='truth')
plt.legend()

plt.subplot(1, 2, 2)
plt.title('rf')
for mdls in est2.models_propensity:
    for mdl in mdls:
        plt.plot(X_test[:, 0], mdl.predict_proba(X_test)[:, 1])
plt.plot(X_test[:, 0], true_propensity(X_test), '--', label='truth')
plt.legend()
plt.show()
```

```python
# Model Y
plt.figure(figsize=(15, 5))
plt.subplot(1, 2, 1)
plt.title('honestrf')
for mdls in est.models_regression:
    for mdl in mdls:
        plt.plot(X_test[:, 0], mdl.predict(np.hstack([X_test, np.ones((X_test.shape[0], 1))])))
plt.plot(X_test[:, 0], true_effect(X_test) + true_conf(X_test), '--', label='truth')
plt.legend()

plt.subplot(1, 2, 2)
plt.title('rf')
for mdls in est2.models_regression:
    for mdl in mdls:
        plt.plot(X_test[:, 0], mdl.predict(np.hstack([X_test, np.ones((X_test.shape[0], 1))])))
plt.plot(X_test[:, 0], true_effect(X_test) + true_conf(X_test), '--', label='truth')
plt.legend()
plt.show()
```

### Interpretability of CATE Model of DRLearner with SHAP

```python
# explain the model's predictions using SHAP values
shap_values = est.shap_values(X[:100], feature_names=feature_names)
```

     98%|===================| 98/100 [00:24<00:00]

```python
# visualize the first prediction's explanation (use matplotlib=True to avoid Javascript)
shap.force_plot(shap_values["Y0"]["T0_1"][0], matplotlib=True)
```

```python
shap.summary_plot(shap_values["Y0"]["T0_1"])
```

# OrthoForest

```python
from econml.orf import DROrthoForest
from sklearn.linear_model import Lasso, LassoCV, LogisticRegression, LogisticRegressionCV
from econml.sklearn_extensions.linear_model import WeightedLassoCV

est3 = DROrthoForest(model_Y=Lasso(alpha=0.01),
                     propensity_model=LogisticRegression(C=1),
                     model_Y_final=WeightedLassoCV(cv=3),
                     propensity_model_final=LogisticRegressionCV(cv=3),
                     n_trees=1000, min_leaf_size=10)
est3.fit(Y, T, X=X)
```

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:   45.3s
    [Parallel(n_jobs=-1)]: Done 112 tasks      | elapsed:   48.1s
    [Parallel(n_jobs=-1)]: Done 272 tasks      | elapsed:   53.0s
    [Parallel(n_jobs=-1)]: Done 496 tasks      | elapsed:  1.0min
    [Parallel(n_jobs=-1)]: Done 784 tasks      | elapsed:  1.2min
    [Parallel(n_jobs=-1)]: Done 1000 out of 1000 | elapsed:  1.4min finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.9s
    [Parallel(n_jobs=-1)]: Done 112 tasks      | elapsed:    5.6s
    [Parallel(n_jobs=-1)]: Done 272 tasks      | elapsed:   12.9s
    [Parallel(n_jobs=-1)]: Done 496 tasks      | elapsed:   25.0s
    [Parallel(n_jobs=-1)]: Done 784 tasks      | elapsed:   44.6s
    [Parallel(n_jobs=-1)]: Done 1000 out of 1000 | elapsed:  1.1min finished

    <econml.orf._ortho_forest.DROrthoForest at 0x1a00ac3db48>

```python
pred3 = est3.effect(X_test)
```

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:  1.2min
    [Parallel(n_jobs=-1)]: Done 100 out of 100 | elapsed:  1.4min finished

```python
plt.figure(figsize=(15, 5))
plt.subplot(1, 2, 1)
plt.plot(X_test[:, 0], true_effect(X_test), '--')
plt.plot(X_test[:, 0], pred, label='forestdr')
plt.plot(X_test[:, 0], pred2, label='nonparamdr')
plt.plot(X_test[:, 0], pred3, label='discreteorf')
plt.fill_between(X_test[:, 0], lb, ub, alpha=.4, label='forest_dr_ci')
plt.legend()
plt.show()
```

```python

```