# Basic Feature Extraction Methods
>Feature Extraction from raw text
- toc: true
- badges: true
- comments: true
- image:https://miro.medium.com/max/1400/1*ns1YWa4cU78B2bpkSOky7w.png
- author: Uday Paila
- categories: [NLP, feature extraction, bow, tfidf, hashing vectorizer]

## Document Term Matrix
It is a matrix with rows contains unique documents and the column contain the unique words/tokens. Let's take sample documents and store them in the `sample_documents`.

```python
sample_documents = ['This is the NLP notebook',
                    'This is basic NLP. NLP is easy',
                    'NLP is awesome']
```

In the above sample_documents, we have 3 documents and 8 unique words. The Document Term matrix contains 3 rows and 8 columns as below.

There are many ways to determine the value(content) in the above matrix. I will discuss some of the ways below. After filling those values, we can use each row as vector representation of documents.

### Bag of Words
In this, we will fill with the number of times that word occurred in the same document.

If you check the above matrix, "`nlp`" occurred two times in the document-2 so value corresponding to that is 2.  If it occurs `n times` in the document, the value corresponding is `n`. We can do the same in the using `CountVectorizer` in  `sklearn`.

```python
##import count vectorizer
from sklearn.feature_extraction.text import CountVectorizer
#creating CountVectorizer instance
bow_vec = CountVectorizer(lowercase=True, ngram_range=(1,1), analyzer='word')
#fitting with our data
bow_vec.fit(sample_documents)
#transforming the data to the vector
sample_bow_metrix = bow_vec.transform(sample_documents)
#printing
print("Unique words -->", bow_vec.get_feature_names())
print("BOW Matrix -->",sample_bow_metrix.toarray())
print("vocab to index dict -->", bow_vec.vocabulary_)
```

    Unique words --> ['awesome', 'basic', 'easy', 'is', 'nlp', 'notebook', 'the', 'this']
    BOW Matrix --> [[0 0 0 1 1 1 1 1]
     [0 1 1 2 2 0 0 1]
     [1 0 0 1 1 0 0 0]]
    vocab to index dict --> {'this': 7, 'is': 3, 'the': 6, 'nlp': 4, 'notebook': 5, 'basic': 1, 'easy': 2, 'awesome': 0}

> Note: How CountVectorizer gets the unique words? -- It first splits the documents into words and then it gets the unique words. `CountVectorizer` uses `token_pattern` or `tokenizer`, we can give our custom `tokenization` algorithm to get words from a sentence. Please try to read the documentation of the `sklearn` to know more about it.

We can also get the n-gram words as vocab. please check the below code. That was written for unigrams and bi-grams.

>Note: N-grams are simply all combinations of adjacent words of length n that you can find in your source text.

```python
#creating CountVectorizer instance with ngram_range = (1,2) i.e uni-gram and bi-gram
bow_vec = CountVectorizer(lowercase=True, ngram_range=(1,2), analyzer='word')
#fitting with our data
bow_vec.fit(sample_documents)
#transforming the data to the vector
sample_bow_metrix = bow_vec.transform(sample_documents)
#printing
print("Unique words -->", bow_vec.get_feature_names())
print("BOW Matrix -->",sample_bow_metrix.toarray())
print("vocab to index dict -->", bow_vec.vocabulary_)
```

    Unique words --> ['awesome', 'basic', 'basic nlp', 'easy', 'is', 'is awesome', 'is basic', 'is easy', 'is the', 'nlp', 'nlp is', 'nlp nlp', 'nlp notebook', 'notebook', 'the', 'the nlp', 'this', 'this is']
    BOW Matrix --> [[0 0 0 0 1 0 0 0 1 1 0 0 1 1 1 1 1 1]
     [0 1 1 1 2 0 1 1 0 2 1 1 0 0 0 0 1 1]
     [1 0 0 0 1 1 0 0 0 1 1 0 0 0 0 0 0 0]]
    vocab to index dict --> {'this': 16, 'is': 4, 'the': 14, 'nlp': 9, 'notebook': 13, 'this is': 17, 'is the': 8, 'the nlp': 15, 'nlp notebook': 12, 'basic': 1, 'easy': 3, 'is basic': 6, 'basic nlp': 2, 'nlp nlp': 11, 'nlp is': 10, 'is easy': 7, 'awesome': 0, 'is awesome': 5}

### TF-IDF
In this, we will fill with `TF*IDF`.
#### Term Frequency
\begin{align}
TF_K = \frac{\text{No of times word K occurred in that document}}{\text{Total number of words in that document}}
\end{align}

>Note: `TF` of a word is only dependent on a particular document. It won't depend on the total corpus of documents. `TF` value of word changes from document to document

#### Inverse Document Frequency
\begin{align}
IDF_K = log(\frac{\text{Total number of documents}}{\text{Number of documents with word K}} )
\end{align}

>Note: `IDF` of a word dependent on total corpus of documents. `IDF` value of word is constant for total corpus.

You can think `IDF` as the information content of the word.

\begin{align}
\text{Information Content} = -log(\text{Probability of Word}) \\
\\
\text{Probability of Word K} = \frac{\text{Number of documents with word K}}{\text{Total number of documents}}
\end{align}

We can calculate the `TFIDF` vectors using `TfidfVectorizer` in sklearn.

```python
from sklearn.feature_extraction.text import TfidfVectorizer
#creating TfidfVectorizer instance
tfidf_vec = TfidfVectorizer()
#fitting with our data
tfidf_vec.fit(sample_documents)
#transforming the data to the vector
sample_tfidf_metrix = tfidf_vec.transform(sample_documents)
#printing
print("Unique words -->", tfidf_vec.get_feature_names())
print("TFIDF Matrix -->", '\n',sample_tfidf_metrix.toarray())
print("vocab to index dict -->", tfidf_vec.vocabulary_)
```

    Unique words --> ['awesome', 'basic', 'easy', 'is', 'nlp', 'notebook', 'the', 'this']
    TFIDF Matrix -->
     [[0.         0.         0.         0.32630952 0.32630952 0.55249005
      0.55249005 0.42018292]
     [0.         0.43157129 0.43157129 0.50978591 0.50978591 0.
      0.         0.32822109]
     [0.76749457 0.         0.         0.45329466 0.45329466 0.
      0.         0.        ]]
    vocab to index dict --> {'this': 7, 'is': 3, 'the': 6, 'nlp': 4, 'notebook': 5, 'basic': 1, 'easy': 2, 'awesome': 0}

With the `TfidfVectorizer` also we can get the n-grams and we can give our own `tokenization` algorithm.

## What if we have so much vocab in our corpus?

If we have many unique words, our `BOW/TFIDF` vectors will be very high dimensional that may cause `curse of dimensionality` problem.  We can solve this with the below methods.

### Limiting the number of vocab in BOW/TFIDF

In `CountVectorize`, we can do this using max_features, min_df, max_df. You can use vocabulary parameter to get specific words only. Try to read the documentation of `CountVectorize` to know better about those. You can check the sample code below.

```python
#creating CountVectorizer instance, limited to 4 features only
bow_vec = CountVectorizer(lowercase=True, ngram_range=(1,1),
                                analyzer='word', max_features=4)
#fitting with our data
bow_vec.fit(sample_documents)
#transforming the data to the vector
sample_bow_metrix = bow_vec.transform(sample_documents)
#printing
print("Unique words -->", bow_vec.get_feature_names())
print("BOW Matrix -->",sample_bow_metrix.toarray())
print("vocab to index dict -->", bow_vec.vocabulary_)
```

    Unique words --> ['awesome', 'is', 'nlp', 'this']
    BOW Matrix --> [[0 1 1 1]
     [0 2 2 1]
     [1 1 1 0]]
    vocab to index dict --> {'this': 3, 'is': 1, 'nlp': 2, 'awesome': 0}

You can do similar thing with `TfidfVectorizer` with same parameters. Please read the documentation.

## Some of the problems with the CountVectorizer and TfidfVectorizer
- If we have a large corpus, vocabulary will also be large and for `fit` function, you have to get all documents into RAM. This may be impossible if you don't have sufficient RAM.
- building the `vocab` requires a full pass over the dataset hence it is not possible to fit text classifiers in a strictly online manner.
- After the `fit`, we have to store the `vocab dict`, which takes so much memory. If we want to deploy in `memory-constrained` environments like amazon lambda, IoT devices, mobile devices, etc.., these maybe not useful.

>Important: We can solve the first problem with an iterator over the total data and building the `vocab` then, using that `vocab`, we can create the `BOW` matrix in the sparse format and then `TFIDF` vectors using `TfidfTransformer`. The sparse matrix won't take much space so, we can store the BOW sparse matrix in our RAM to create the TFIDF sparse matrix.

I have written a sample code to do that for the same data. I have iterated over the data,  created vocab, and using that vocab, created BOW.  We can write a much more optimized version of the code, This is just a sample to show.

```python
##for tokenization
import nltk
#vertical stack of sparse matrix
from scipy.sparse import vstack
#vocab set
vocab_set = set()
#looping through the points(for huge data, you will get from your disk/table)
for data_point in sample_documents:
    #getting words
    for word in nltk.tokenize.word_tokenize(data_point):
        if word.isalpha():
            vocab_set.add(word.lower())

vectorizer_bow = CountVectorizer(vocabulary=vocab_set)

bow_data = []
for data_point in sample_documents: # use a generator
    ##if we give the vocab, there will be no data lekage for fit_transform so we can use that
    bow_data.append(vectorizer_bow.fit_transform([data_point]))

final_bow = vstack(bow_data)

print("Unique words -->", vectorizer_bow.get_feature_names())
print("BOW Matrix -->",final_bow.toarray())
print("vocab to index dict -->", vectorizer_bow.vocabulary_)
```

    Unique words --> ['awesome', 'basic', 'easy', 'is', 'nlp', 'notebook', 'the', 'this']
    BOW Matrix --> [[0 0 0 1 1 1 1 1]
     [0 1 1 2 2 0 0 1]
     [1 0 0 1 1 0 0 0]]
    vocab to index dict --> {'awesome': 0, 'basic': 1, 'easy': 2, 'is': 3, 'nlp': 4, 'notebook': 5, 'the': 6, 'this': 7}

The above result is similar to the one we printed while doing the BOW, you can check that.

Using the above `BOW` sparse matrix and the `TfidfTransformer`, we can create the `TFIDF` vectors. you can check below code.

```python
#importing
from sklearn.feature_extraction.text import TfidfTransformer
#instanciate the class
vec_tfidftransformer = TfidfTransformer()
#fit with the BOW sparse data
vec_tfidftransformer.fit(final_bow)
vec_tfidf = vec_tfidftransformer.transform(final_bow)
print(vec_tfidf.toarray())
```

    [[0.         0.         0.         0.32630952 0.32630952 0.55249005
      0.55249005 0.42018292]
     [0.         0.43157129 0.43157129 0.50978591 0.50978591 0.
      0.         0.32822109]
     [0.76749457 0.         0.         0.45329466 0.45329466 0.
      0.         0.        ]]

The above result is similar to the one we printed while doing the TFIDF, you can check that.

>Important: Other than our own iterator/generator, if we have data in one file or multiple files, we can directly give `input` parameter as `file/filename` and while `fit` function, we can give file path. Please read the documentation.

Another way to solve all the above problems are hashing. We can convert a word into fixed index number using the hash function. so, there will be no training process to get the vocabulary and no need to save the vocab. It was implemented in sklearn with `HashingVectorizer`. In `HashingVectorizer`, you have to mention number of features you need, by default it takes  $2^{20}$. below you can see some code to use `HashingVectorizer`.

```python
#importing the hashvectorizer
from sklearn.feature_extraction.text import HashingVectorizer
#instanciating the HashingVectorizer
hash_vectorizer = HashingVectorizer(n_features=5, norm=None, alternate_sign=False)
#transforming the data, No need to fit the data because, it is stateless
hash_vector = hash_vectorizer.transform(sample_documents)
#printing the output
print("Hash vectors -->",hash_vector.toarray())
```

    Hash vectors --> [[0. 1. 3. 1. 0.]
     [0. 1. 5. 1. 0.]
     [0. 0. 3. 0. 0.]]

>Note: You can normalize your vectors using norm. Since the hash function might cause collisions between (unrelated) features, a signed hash function is used and the sign of the hash value determines the sign of the value stored in the output matrix for a feature. This way, collisions are likely to cancel out rather than accumulate error, and the expected mean of any output feature’s value is zero. This mechanism is enabled by default with `alternate_sign=True` and is particularly useful for small hash table sizes (`n_features < 10000`).

We can convert above vector to `TFIDF` using `TfidfTransformer`. check the below code

```python
#instanciate the class
vec_idftrans = TfidfTransformer()
#fit with the hash BOW sparse data
vec_idftrans.fit(hash_vector)
##transforming the data
vec_tfidf2 = vec_idftrans.transform(hash_vector)
print("tfidf using hash BOW -->",vec_tfidf2.toarray())
```

    tfidf using hash BOW --> [[0.         0.36691832 0.85483442 0.36691832 0.        ]
     [0.         0.2419863  0.93961974 0.2419863  0.        ]
     [0.         0.         1.         0.         0.        ]]

This vectorizer is memory efficient but there are some cons for this as well, some of them are
- There is no way to compute the inverse transform of the Hashing so there will be no interpretability of the model.
- There can be collisions in the hashing.

References:
1. https://scikit-learn.org/stable/

```python

```


```python
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

import sys
sys.path.append('../../pyutils')
import metrics
import utils
```

# Introduction

In an undirected graph model, each vertex represent a random variable, and the absence of edge between two vertices means that the 2 randoms variables are conditionaly independent, given the other. The graph gives a visual way of understanding the joint distribution of the entire set of random variables.
This graph is also called a Markov Random Field.
Sparge graphs have a small number of edges, and are convenient for interpretation.

Each edge is parametrized by it's value (or potiential) that encode the strength of the conditional dependence between the random variables.
Challenges of grahical models are:
- model selection (graph structure)
- estimation of the edges parameters from data (learning)
- compute marginal random variables propabilities and expectations. (inference)

# Markov Graphs and Their Properties

Let a graph $\mathcal{G}$ a pair $(V,E)$ with $V$ a set of vertices and $E$ a set of edges.
Two vertices $X$ and $Y$ are adjacent is there is an edge between them: $X \sim Y$.
A path $X_1,\text{...},X_n$ is a set of joined vertices; $X_{i-1} \sim X_i$ for $i=2,\text{...},n$.
A complete graph is a graph with every pair of vertices joined by an edge.
A subgraph $U \in V$ is a subject of vertices with their edges.

In a markov graph, the absence of an edge implies that the random variables are conditionally independant given the other variables:
$$\text{No edge joining $X$ and $Y$} \iff X \perp Y | \text{rest}$$

$A,B,C$ subgraphs. $C$ separate $A$ and $B$ if every path between $A$ and $B$ intersects $C$.
$$\text{$C$ separates $A$ and $B$} \implies A \perp B | C$$

A clique is a complete subgraph. A clique is said maximal is no other vertices can be added to it and still yield a clique.

A probability density function $f$ over $\mathcal{G}$ can be represented as:
$$f(x) = \frac{1}{Z} \prod_{C \in \mathcal{C}} \psi_C(x_C)$$
with $\mathcal{C}$ set of maximal cliques, and $\psi_C$ clique potentials. These are not really density functions, but capture dependence in $X_C$ by scoring certains $x_C$ higher than others.
$Z$ is the normilazing constant, also called the partition function:
$$Z = \sum_{x \in \mathcal{X}} \prod_{C \in \mathcal{C}} \Psi_C (x_C)$$

This chapter focus on pairwise Markov Graph. There is a potential function for each edge, and at-most second-order interactions are represented. They need fewer parameters, and are easier to work with.

# Undirected Graphical Models for Continuous Variables

We assume the observations have a multivariate Gaussian distribution with mean $\mu$ and covariance $\Sigma$. Since Gaussian distribution represents at most second-order relationships, it encodes a pairwise Markov Graph.

Th Gaussian distribution has the property that all conditionals distributions are also Gaussian.
Let $\Theta = \Sigma^{-1}$. If $\Theta_{ij}=0$, then variables $i$ and $j$ are conditionally independant given the other variables.

## Estimation of the Parameters when the Graph Structure is Known

Given some observations of $X$, let estimate the parameters of the joint distribution ($\mu$ and $\Sigma$). We suppose that the graph is complete.
Let's define the empirical covariance matrix $S$:
$$S = \frac{1}{N} \sum_{i=1}^N (x_i - \bar{x}) (x_i - \bar{x})^T$$
with $\bar{x}$ the sample mean vector.

The log-likelihood of the data can be written as:
$$l(\Theta) = \log \det \theta - \text{trace}(S\theta)$$
The maximum likelihood estimate of $\Sigma$ is $S$.

Now is some edges are missing, we are trying to maximize $l(\Theta)$ under the constraint that somes entries of $\Theta$ are 0.
We add Lagrange constants for all missing edges:
$$l_C(\Theta) = \log \det \theta - \text{trace}(S\theta) - \sum_{(j,k) \notin E} \gamma_{jk}\theta_{jk}$$
This can be maximized with the followigng equation:
$$\Theta^{-1} - S - \Gamma = 0$$

with $\Gamma$ matrix of Lagrange parameters.

We can patition the matrices into 2 parts: part 1 the first $p-1$ rows and columns, and part 2 the $pth$ row and column.
The equation can be rewrtieen as:
$$W_{11}\beta - s_{12} - \gamma_{12} = 0$$
We can remove all non-zeros elements from $\gamma_12$, corresponding to edges constrained to be $0$, because they carry no information. We can also reduce the same way $\beta$ and $W_11$, giving us the new equation:
$$W^*_{11}\beta^* - s^*_{12} = 0$$
with solution:
$$\hat{\beta}^* = W_{11}^{*-1} s^*_{12}$$
the solution is padded with zeros to give $\hat{\beta}$

Algorithm $17.1$ page 634

```python
def part_mats(W, S, j):
    W_11 = np.delete(W, j, axis=0)
    W_11 = np.delete(W_11, j, axis=1)
    s_12 = np.delete(S, j, axis=0)[:,j]
    return W_11, s_12

def regroup_mat(W, w_12, j):
    w_12 = np.insert(w_12, j, W[j,j])
    W[:,j] = w_12
    return W

def edit_mats(G, j, W_11, s_12):
    N = len(G)
    suppr = [i for i in range(N) if G[i,j] == 0]
    suppr = [x if x < j else x - 1 for x in suppr]
    s_12 = np.delete(s_12, suppr)
    W_11 = np.delete(W_11, suppr, axis=0)
    W_11 = np.delete(W_11, suppr, axis=1)
    return W_11, s_12

def extend_beta(G, j, betar):
    N = len(G)
    suppr = [i for i in range(N) if G[i,j] == 0]
    suppr = [x if x < j else x - 1 for x in suppr]
    beta = np.insert(betar, suppr, 0)
    return beta

def estim_gauss(S, G, max_iters = 100, tol=1e-16):

    N = len(S)
    W = S.copy()

    for it in range(max_iters):

        W_old = W.copy()

        for j in range(N):
            W_11, s_12 = part_mats(W, S, j)
            #print(W_11)
            #print(s_12)

            W_11r, s_12r = edit_mats(G, j, W_11, s_12)

            #print(W_11r)
            #print(s_12r)

            betar = np.linalg.inv(W_11r) @ s_12r
            beta = extend_beta(G, j, betar)
            #print(betar)
            #print(beta)

            w_12 = W_11 @ beta
            #print(w_12)

            W = regroup_mat(W, w_12, j)
            #W[:N-1, -1] = w_12

            #print(W)

        if np.linalg.norm(W - W_old) < tol:
            break

    print('Iterations:', it)
    return W

S = np.array([
    [10, 1, 5, 4],
    [1., 10, 2, 6],
    [5, 2, 10, 3],
    [4, 6, 3, 10]
])

G = np.array([
    [1, 1, 0, 1],
    [1, 1, 1, 0],
    [0, 1, 1, 1],
    [1, 0, 1, 1]
])

W = estim_gauss(S, G)
print(np.around(W,2))
print(np.around(np.linalg.inv(W),2))
```

    Iterations: 10
    [[10.    1.    1.31  4.  ]
     [ 1.   10.    2.    0.87]
     [ 1.31  2.   10.    3.  ]
     [ 4.    0.87  3.   10.  ]]
    [[ 0.12 -0.01 -0.   -0.05]
     [-0.01  0.1  -0.02 -0.  ]
     [ 0.   -0.02  0.11 -0.03]
     [-0.05  0.   -0.03  0.13]]

## Estimation of the Graph Structure

Sparse inverse covariance estimation with the graphical lasso - Friedman, J., Hastie, T. and Tibshirani, R. (2008) -[PDF](file:///home/aiw/docs/eosl-refs.pdf)

We can use the lasso regularization to estimate $\Sigma$ in a way that tries to insert zeroes in $\Theta$, giving us the graph strcture.
Let's maximizing the penalized log-likelihood:
$$\log \det \Theta - \text{trace}(S\Theta) - \lambda ||\Theta||_1$$

The gradient equation is:
$$\Theta^{-1} - S - \lambda \text{sign}(\Theta) = 0$$

Similary to the algorithm above, we reach the equation:
$$W_{11}\beta - s_{12} + \lambda \text{sign}(\beta) = 0$$

This problem is similar to linear regression with lasso, and can be solved using the pathwise coordinate descent method.
Let $V = W_{11}$, the update has the form:
$$\hat{\beta}_j \leftarrow \frac{1}{V_jj} S(s_{12j} - \sum_{k \neq j} V_{kj} \hat{\beta}_k, \lambda)$$
with $S$ the soft-threshold operator:
$$S(x,t) = \text{sign}(x)(|x| - t)_+$$

Algorithm $17.2$ page $636$

## Undirected Graphical Models for Discrete Variables

Pairwise Markov Networks with binary variables are veary common. They are called Ising models, or Botlzmann machines.
Each vertices are referred to as nodes or units. The values at each node can either be obversed (visible) or unobserved (hidden).

We consider first the case where all $p$ nodes are visible with edge pairs $(j,k) \in E$. Their joint probability is given by:
$$p(X,\Theta) = \exp \left[ \sum_{(j,k) \in E} \theta_{jk} X_j X_k - \Phi(\Theta) \right]$$

with $\Phi(\Theta)$ the log of the partition function:
$$\Phi(\Theta) = \log \sum_{x \in \mathcal{X}} \left[ \exp \sum_{(j,k) \in E} \theta_{jk} x_j x_k \right]$$

The model requires a constant node $X_0=1$.
The Ising model implies a logistic form for each node conditional on the others:
$$P(X_j=1| X_{-j} = x_{-j}) = \left( 1 + \exp ( -\theta_{j0} - \sum_{(j,k) \in E} \theta_{jk} x_k) \right) ^{-1}$$

## Estimations of the Parameters when the Graph Structure is known

Given N observations $x_i$, we can estimate the parameters by maximizing the log-likelihood:
$$l(\Theta) = \sum_{i=1}^N \log P_\Theta(X = x_i)$$

$$
\begin{equation}
\begin{split}
l(\Theta) & = \sum_{i=1}^N \log P_\Theta(X = x_i) \\
& = \sum_{i=1}^N \left( \sum_{(j,k) \in E} \theta_{jk} x_{ij} x_{ik} - \Phi(\Theta) \right)
\end{split}
\end{equation}
$$

Setting the gradient to $0$ gives:
$$\hat{E}(X_j,X_k) - E_\Theta(X_j,X_k) = 0$$
with $\hat{E}(X_j,X_k)$ the expecation taken with respect to the empirical distribution of the data:
$$\hat{E}(X_j, X_k) = \frac{1}{N} \sum_{i=1}^N x_{ij}x_{ik}$$

We can find the maximum likelihood estimates using gradient search or Netwon methods, but computing $E_\Theta(X_j,X_k)$ involves the enumeration of $p(X,\Theta)$ over the $|\mathcal{X}|=2^p$ values of $X$, and it not feasible for large $p$ (over 30).

When $p$ is large, the gradient is approximated using other methods, like mean field approximation or Gibbs sampling

## Hidden Nodes

Let's suppose we have a subset of visible variables $X_V$, and the remaining are the hidden $X_H$. The log-likelihood become:

$$
\begin{equation}
\begin{split}
l(\Theta) & = \sum_{i=1}^N \log P_\Theta(X_V = x_{iV}) \\
& = \sum_{i=1}^N \left( \log \sum_{x_h \in \mathcal{X}_H} \exp \sum_{(j,k) \in E} \theta_{jk} x_{ij} x_{ik} - \Phi(\Theta) \right)
\end{split}
\end{equation}
$$

The gradient becames:
$$\frac{d l(\Theta)}{d\theta_{jk}} = \hat{E}_V E_\Theta(X_k,X_K|X_V) - E\Theta(X_j,X_k)$$

It can be computed using Gibbs sampling, but the method can be very slow, even for moderate-sized models. We can add more restrictions to make those computations manageable

## Estimation of the Graph Structure

As for continuous random variables, we can use lasso to remove edges.

One solution is to use a penalized log-likelihood, but the gradient computation for dense graphs is not manageable.

Another solution fit an $L_1$ penalized logistic regression model to each node as a functon of the other nodes, and symmetrize the edge parameter estimates.

## Restricted Boltzmann Machines

An RBM consts of one layer of visible units, and one layer of hidden units , with no connections within a layer.
The restricted form simplifies the Gibbs Sampling to compute the gradient of the log-likelihhod.
Using the contrastive-divergence algorithm, they can be trained rapidly.

RBM can learn to extract interesting features from data.
We can stack several RBM togethers and train the whole joint density model.


# Julia で連分数

Fresnelの余弦積分，Fresnelの正弦積分をそれぞれ次で定義する:
$$C(x)=\int_{0}^{x} \cos\left(\frac{\pi}{2}t^2\right)dt , \quad S(x)=\int_{0}^{x} \sin\left(\frac{\pi}{2}t^2\right)dt.$$
このとき
$$C(x)+iS(x)=\int_{0}^{x} e^{i\pi t^2/2}dt$$
となる．[2] では $x\geq 2$ の時に次の式を用いて計算している：
$$\int_{0}^{x} e^{i\pi t^2/2}dt=\frac{1+i}{2}-\frac{i}{\pi x} e^{i\pi x^2/2}H(z).$$
ここで$z=i/\pi x^2$ であり $H(z)$ は連分数で定まる次の形をしている関数である．
$$H(z)=\frac{1}{1+\frac{z}{1+\frac{2z}{1+\frac{3z}{1+...}}}}$$

というわけでJuliaでFresnel積分の（スクラッチによる）実装をしてみたかったときに連分数の計算が必要になった.

## 方法１(Reference [1])

```julia
using SymPy
```

```julia
function f(z::Number, n::Integer)
    cf::typeof(inv(z))=1
    for k in n:-1:1
        cf = 1+k*z/cf
    end
    return inv(cf)
end
```

    f (generic function with 1 method)

```julia
display("text/latex",f(Sym(:z),6))
```

$$\frac{1}{\frac{z}{\frac{2 z}{\frac{3 z}{\frac{4 z}{\frac{5 z}{6 z + 1} + 1} + 1} + 1} + 1} + 1}$$

## 方法２(Reference [2])

\begin{align}
&A_{-1}=1,B_{-1}=0, \\
&A_{0}=1,B_{0}=1, \\
&A_k=A_{k-1}+kzA_{k-2} \quad (k=1,2,3,...)\\
&B_k=B_{k-1}+kzB_{k-2} \quad (k=1,2,3,...)
\end{align}

で定まる数列 $A_N, B_N$ を計算して $1/(A_N/B_N)$ とする．

```julia
function g(z::Number, n::Integer)
    a_old=1 ; b_old=0
    a_now=1 ; b_now=1
    a=0
    b=0
    for k in 1:n
        a=a_now+k*z*a_old
        b=b_now+k*z*b_old
        a_old=a_now
        b_old=b_now
        a_now=a
        b_now=b
    end
    print("a=$a b=$b")
    return inv(a/b)
end
```

    g (generic function with 1 method)

```julia
display("text/latex",g(Sym(:z),6))
```

$$\frac{4 z \left(2 z + 1\right) + 5 z \left(5 z + 1\right) + 6 z \left(4 z \left(2 z + 1\right) + 5 z + 1\right) + 5 z + 1}{3 z \left(z + 1\right) + 4 z \left(3 z + 1\right) + 5 z \left(3 z \left(z + 1\right) + 3 z + 1\right) + 6 z \left(3 z \left(z + 1\right) + 4 z \left(3 z + 1\right) + 3 z + 1\right) + 3 z + 1}$$

    a=3*z*(z + 1) + 4*z*(3*z + 1) + 5*z*(3*z*(z + 1) + 3*z + 1) + 6*z*(3*z*(z + 1) + 4*z*(3*z + 1) + 3*z + 1) + 3*z + 1 b=4*z*(2*z + 1) + 5*z*(5*z + 1) + 6*z*(4*z*(2*z + 1) + 5*z + 1) + 5*z + 1

## 数値での比較

```julia
a=rand()
n=100
f(a,n)-g(a,n) <1e-15
```

    a=1.72468885394813e64 b=1.3750788773870185e64

    true

```julia
@time f(a,n)
```

      0.000006 seconds (84 allocations: 6.107 KiB)

    0.7972909862780239

```julia
@time g(a,n)
```

    a=1.72468885394813e64 b=1.3750788773870185e64  0.000238 seconds (1.30 k allocations: 21.016 KiB)

    0.7972909862780234

## g について
-  n=1000とかにするとa,bともに発散してどうしようもなくなる
- 上のtime測定でもわかるようにメモリの使用量もちがうっぽい

```julia
a=5.5
n=1000
@time g(a,n)
```

    a=Inf b=Inf  0.000308 seconds (13.07 k allocations: 204.641 KiB)

    NaN

# Reference:
- [1] :https://twitter.com/genkuroki/status/898008651033489408
- [2] :田口俊弘著 Fortan ハンドブック 270頁 Key Element6.1 連分数の計算方法 技術評論社より
- [3] :http://www.aip.de/groups/soe/local/numres/bookcpdf/c6-9.pdf

```julia

```


```python
%matplotlib inline
```

# Bayesian optimization with `skopt`

Gilles Louppe, Manoj Kumar July 2016.
Reformatted by Holger Nahrstaedt 2020

.. currentmodule:: skopt

## Problem statement

We are interested in solving

\begin{align}x^* = arg \min_x f(x)\end{align}

under the constraints that

- $f$ is a black box for which no closed form is known
  (nor its gradients);
- $f$ is expensive to evaluate;
- and evaluations of $y = f(x)$ may be noisy.

**Disclaimer.** If you do not have these constraints, then there
is certainly a better optimization algorithm than Bayesian optimization.

This example uses :class:`plots.plot_gaussian_process` which is available
since version 0.8.

## Bayesian optimization loop

For $t=1:T$:

1. Given observations $(x_i, y_i=f(x_i))$ for $i=1:t$, build a
   probabilistic model for the objective $f$. Integrate out all
   possible true functions, using Gaussian process regression.

2. optimize a cheap acquisition/utility function $u$ based on the
   posterior distribution for sampling the next point.
   $x_{t+1} = arg \min_x u(x)$
   Exploit uncertainty to balance exploration against exploitation.

3. Sample the next observation $y_{t+1}$ at $x_{t+1}$.

## Acquisition functions

Acquisition functions $u(x)$ specify which sample $x$: should be
tried next:

- Expected improvement (default):
  $-EI(x) = -\mathbb{E} [f(x) - f(x_t^+)]$
- Lower confidence bound: $LCB(x) = \mu_{GP}(x) + \kappa \sigma_{GP}(x)$
- Probability of improvement: $-PI(x) = -P(f(x) \geq f(x_t^+) + \kappa)$

where $x_t^+$ is the best point observed so far.

In most cases, acquisition functions provide knobs (e.g., $\kappa$) for
controlling the exploration-exploitation trade-off.
- Search in regions where $\mu_{GP}(x)$ is high (exploitation)
- Probe regions where uncertainty $\sigma_{GP}(x)$ is high (exploration)

```python
print(__doc__)

import numpy as np
np.random.seed(237)
import matplotlib.pyplot as plt
from skopt.plots import plot_gaussian_process
```

## Toy example

Let assume the following noisy function $f$:

```python
noise_level = 0.1

def f(x, noise_level=noise_level):
    return np.sin(5 * x[0]) * (1 - np.tanh(x[0] ** 2))\
           + np.random.randn() * noise_level
```

**Note.** In `skopt`, functions $f$ are assumed to take as input a 1D
vector $x$: represented as an array-like and to return a scalar
$f(x)$:.

```python
# Plot f(x) + contours
x = np.linspace(-2, 2, 400).reshape(-1, 1)
fx = [f(x_i, noise_level=0.0) for x_i in x]
plt.plot(x, fx, "r--", label="True (unknown)")
plt.fill(np.concatenate([x, x[::-1]]),
         np.concatenate(([fx_i - 1.9600 * noise_level for fx_i in fx],
                         [fx_i + 1.9600 * noise_level for fx_i in fx[::-1]])),
         alpha=.2, fc="r", ec="None")
plt.legend()
plt.grid()
plt.show()
```

Bayesian optimization based on gaussian process regression is implemented in
:class:`gp_minimize` and can be carried out as follows:

```python
from skopt import gp_minimize

res = gp_minimize(f,                  # the function to minimize
                  [(-2.0, 2.0)],      # the bounds on each dimension of x
                  acq_func="EI",      # the acquisition function
                  n_calls=15,         # the number of evaluations of f
                  n_random_starts=5,  # the number of random initialization points
                  noise=0.1**2,       # the noise level (optional)
                  random_state=1234)   # the random seed
```

Accordingly, the approximated minimum is found to be:

```python
"x^*=%.4f, f(x^*)=%.4f" % (res.x[0], res.fun)
```

For further inspection of the results, attributes of the `res` named tuple
provide the following information:

- `x` [float]: location of the minimum.
- `fun` [float]: function value at the minimum.
- `models`: surrogate models used for each iteration.
- `x_iters` [array]:
  location of function evaluation for each iteration.
- `func_vals` [array]: function value for each iteration.
- `space` [Space]: the optimization space.
- `specs` [dict]: parameters passed to the function.

```python
print(res)
```

Together these attributes can be used to visually inspect the results of the
minimization, such as the convergence trace or the acquisition function at
the last iteration:

```python
from skopt.plots import plot_convergence
plot_convergence(res);
```

Let us now visually examine

1. The approximation of the fit gp model to the original function.
2. The acquisition values that determine the next point to be queried.

```python
plt.rcParams["figure.figsize"] = (8, 14)

def f_wo_noise(x):
    return f(x, noise_level=0)
```

Plot the 5 iterations following the 5 random points

```python
for n_iter in range(5):
    # Plot true function.
    plt.subplot(5, 2, 2*n_iter+1)

    if n_iter == 0:
        show_legend = True
    else:
        show_legend = False

    ax = plot_gaussian_process(res, n_calls=n_iter,
                               objective=f_wo_noise,
                               noise_level=noise_level,
                               show_legend=show_legend, show_title=False,
                               show_next_point=False, show_acq_func=False)
    ax.set_ylabel("")
    ax.set_xlabel("")
    # Plot EI(x)
    plt.subplot(5, 2, 2*n_iter+2)
    ax = plot_gaussian_process(res, n_calls=n_iter,
                               show_legend=show_legend, show_title=False,
                               show_mu=False, show_acq_func=True,
                               show_observations=False,
                               show_next_point=True)
    ax.set_ylabel("")
    ax.set_xlabel("")

plt.show()
```

The first column shows the following:

1. The true function.
2. The approximation to the original function by the gaussian process model
3. How sure the GP is about the function.

The second column shows the acquisition function values after every
surrogate model is fit. It is possible that we do not choose the global
minimum but a local minimum depending on the minimizer used to minimize
the acquisition function.

At the points closer to the points previously evaluated at, the variance
dips to zero.

Finally, as we increase the number of points, the GP model approaches
the actual function. The final few points are clustered around the minimum
because the GP does not gain anything more by further exploration:

```python
plt.rcParams["figure.figsize"] = (6, 4)

# Plot f(x) + contours
_ = plot_gaussian_process(res, objective=f_wo_noise,
                          noise_level=noise_level)

plt.show()
```


# NRPy+ SymPy LaTeX Interface (NRPyLaTeX)

## Author: Ken Sible

### The following notebook will demonstrate LaTeX to SymPy conversion, including [Einstein notation](https://en.wikipedia.org/wiki/Einstein_notation).

# Table of Contents
$$\label{toc}$$

- [Step 1](#step_1): Lexical Analysis and Syntax Analysis
- [Step 2](#step_2): Grammar Demonstration and Sandbox
- [Step 3](#step_3): Tensor Support with Einstein Notation
    - [Example 1](#example_1): Tensor Contraction
    - [Example 2](#example_2): Index Raising
    - [Example 3](#example_3): Cross Product
    - [Example 4](#example_4): Covariant Derivative
    - [Example 5 (1)](#example_5_1): Schwarzschild Metric
    - [Example 5 (2)](#example_5_2): Kretschmann Scalar
    - [Example 6 (1)](#example_6_1): Extrinsic Curvature (ADM Formalism)
    - [Example 6 (2)](#example_6_2): Hamiltonian/Momentum Constraint
- [Step 4](#step_4): Exception Handling and Index Checking
- [Step 5](#step_5): Output Notebook to PDF

Further Reading: [Parsing BSSN (Cartesian) Notebook](Tutorial-LaTeX_Interface_Example-BSSN_Cartesian.ipynb)

## Step 1: Lexical Analysis and Syntax Analysis [ [^](#top) ]

In the following section, we discuss [lexical analysis](https://en.wikipedia.org/wiki/Lexical_analysis) (lexing) and [syntax analysis](https://en.wikipedia.org/wiki/Parsing) (parsing). In lexical analysis, a lexical analyzer (or lexer) can tokenize a character string, called a sentence, using substring pattern matching. In syntax analysis, a syntax analyzer (or parser) can construct a parse tree, containing all syntactic information of the language (specified by a [formal grammar](https://en.wikipedia.org/wiki/Formal_grammar)), after receiving a token iterator from the lexical analyzer.

For LaTeX to SymPy conversion, we implemented a [recursive descent parser](https://en.wikipedia.org/wiki/Recursive_descent_parser) that can construct a parse tree in [preorder traversal](https://en.wikipedia.org/wiki/Tree_traversal#Pre-order_(NLR)), starting from the root [nonterminal](https://en.wikipedia.org/wiki/Terminal_and_nonterminal_symbols), using a [right recursive](https://en.wikipedia.org/wiki/Left_recursion) grammar (partially shown below in the canonical (extended) [BNF](https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form) notation).

```
<EXPRESSION>    -> <TERM> { ( '+' | '-' ) <TERM> }*
<TERM>          -> <FACTOR> { [ '/' ] <FACTOR> }*
<FACTOR>        -> <BASE> { '^' <EXPONENT> }*
<BASE>          -> [ '-' ] ( <ATOM> | <SUBEXPR> )
<EXPONENT>      -> <BASE> | '{' <BASE> '}' | '{' '{' <BASE> '}' '}'
<ATOM>          -> <COMMAND> | <OPERATOR> | <NUMBER> | <TENSOR>
<SUBEXPR>       -> '(' <EXPRESSION> ')' | '[' <EXPRESSION> ']' | '\' '{' <EXPRESSION> '\' '}'
<COMMAND>       -> <FUNC> | <FRAC> | <SQRT> | <NLOG> | <TRIG>
    ⋮            ⋮
```

**Source**: Robert W. Sebesta. Concepts of Programming Languages. Pearson Education Limited, 2016.

```python
import sympy as sp
try: import nrpylatex
except ImportError:
    !pip install nrpylatex
!pip freeze | grep nrpylatex
from nrpylatex import *
```

    nrpylatex==1.0.7

```python
lexer = Lexer(); lexer.initialize(r'(1 + x/n)^n')
print(', '.join(token for token in lexer.tokenize()))
```

    LPAREN, INTEGER, PLUS, LETTER, DIVIDE, LETTER, RPAREN, CARET, LETTER

```python
expr = parse_latex(r'(1 + x/n)^n')
print(expr, '\n  >>', sp.srepr(expr))
```

    (1 + x/n)**n
      >> Pow(Add(Integer(1), Mul(Pow(Symbol('n', real=True), Integer(-1)), Symbol('x', real=True))), Symbol('n', real=True))

`Grammar Derivation: (1 + x/n)^n`
```
<EXPRESSION> -> <TERM>
             -> <FACTOR>
             -> <BASE>^<EXPONENT>
             -> <SUBEXPR>^<EXPONENT>
             -> (<EXPRESSION>)^<EXPONENT>
             -> (<TERM> + <TERM>)^<EXPONENT>
             -> (<FACTOR> + <TERM>)^<EXPONENT>
             -> (<BASE> + <TERM>)^<EXPONENT>
             -> (<ATOM> + <TERM>)^<EXPONENT>
             -> (<NUMBER> + <TERM>)^<EXPONENT>
             -> (<INTEGER> + <TERM>)^<EXPONENT>
             -> (1 + <TERM>)^<EXPONENT>
             -> (1 + <FACTOR> / <FACTOR>)^<EXPONENT>
             -> ...
```

## Step 2: Grammar Demonstration and Sandbox [ [^](#top) ]

In the following section, we demonstrate the process for extending the parsing module to include a (previously) unsupported LaTeX command.

1. update the `grammar` dictionary in the `Lexer` class with the mapping `regex` $\mapsto$ `token`
1. write a grammar abstraction in BNF notation (similar to a regular expression) for the command
1. implement a private method for the nonterminal (command name) to parse the grammar abstraction

```<SQRT> -> <SQRT_CMD> [ '[' <INTEGER> ']' ] '{' <EXPRESSION> '}'```
```
def _sqrt(self):
    self.expect('SQRT_CMD')
    if self.accept('LBRACK'):
        integer = self.lexer.lexeme
        self.expect('INTEGER')
        root = Rational(1, integer)
        self.expect('RBRACK')
    else: root = Rational(1, 2)
    self.expect('LBRACE')
    expr = self._expression()
    self.expect('RBRACE')
    if root == Rational(1, 2):
        return sqrt(expr)
    return Pow(expr, root)
```

In addition to expression parsing, we included support for equation parsing, which can produce a dictionary mapping `LHS` $\mapsto$ `RHS`, where `LHS` must be a symbol, and insert that mapping into the global namespace of the previous stack frame, as demonstrated below.

$$ \mathit{s_n} = \left(1 + \frac{1}{n}\right)^n $$

```python
parse_latex(r'\text{s_n} = \left(1 + \frac{1}{n}\right)^n')
```

    ('s_n',)

```python
print('s_n =', s_n)
```

    s_n = (1 + 1/n)**n

Furthermore, we implemented robust error messaging using the custom `ParseError` exception, which should handle every conceivable case to identify, as detailed as possible, invalid syntax inside of a LaTeX sentence. The following are some runnable examples of possible error messages.

```python
try: parse_latex(r'5x^{{4$}}')
except ParseError as e:
    print(type(e).__name__ + ': ' + str(e))
```

    ParseError: 5x^{{4$}}
                      ^
    unexpected '$' at position 6

```python
try: parse_latex(r'\sqrt[0.1]{5x^{{4}}}')
except ParseError as e:
    print(type(e).__name__ + ': ' + str(e))
```

    ParseError: \sqrt[0.1]{5x^{{4}}}
                      ^
    expected token INTEGER at position 6

```python
try: parse_latex(r'\int_0^5 5x^{{4}}dx')
except ParseError as e:
    print(type(e).__name__ + ': ' + str(e))
```

    ParseError: \int_0^5 5x^{{4}}dx
                ^
    unsupported command '\int' at position 0

In the sandbox code cell below, you can experiment with converting LaTeX to SymPy using the wrapper function `parse(sentence)`, where `sentence` must be a Python [raw string](https://docs.python.org/3/reference/lexical_analysis.html) to interpret a backslash as a literal character rather than an [escape sequence](https://en.wikipedia.org/wiki/Escape_sequence). You could, alternatively, use the supported cell magic `%%parse_latex` to automatically escape every backslash and parse the cell (more convenient than `parse(sentence)` in a notebook format).

```python
# Write Sandbox Code Here
```

## Step 3: Tensor Support with Einstein Notation [ [^](#top) ]

In the following section, we demonstrate parsing tensor notation using the Einstein summation convention. In each example, every tensor should appear either on the LHS of an equation or on the RHS of a `vardef` macro before appearing on the RHS of an equation. Furthermore, an exception will be raised upon violation of the Einstein summation convention, i.e. the occurrence of an invalid free or bound index.

**Configuration Grammar**

```
<MACRO>     -> <PARSE> | <SREPL> | <VARDEF> | <KEYDEF> | <ASSIGN> | <IGNORE>
<PARSE>     -> <PARSE_MACRO> <ASSIGNMENT> { ',' <ASSIGNMENT> }*
<SREPL>     -> <SREPL_MACRO> <STRING> <ARROW> <STRING> { ',' <STRING> <ARROW> <STRING> }*
<VARDEF>    -> <VARDEF_MACRO> { '-' <OPTION> }* <VARIABLE> { ',' <VARIABLE> }* [ '(' <DIMENSION> ')' ]
<KEYDEF>    -> <KEYDEF_MACRO> <BASIS_KWRD> <BASIS> | <INDEX_KWRD> <INDEX>
<ASSIGN>    -> <ASSIGN_MACRO> { '-' <OPTION> }* <VARIABLE> { ',' <VARIABLE> }*
<IGNORE>    -> <IGNORE_MACRO> <STRING> { ',' <STRING> }*
<OPTION>    -> <DRV_TYPE> [ <PRIORITY> ] | <SYMMETRY> | <WEIGHT> '=' <NUMBER>
<BASIS>     -> <BASIS_KWRD> '{' <LETTER> { ',' <LETTER> }* '}'
<INDEX>     -> ( <LETTER> | '[' <LETTER> '-' <LETTER> ']' ) '(' <DIMENSION> ')'
```

### Example 1. [Tensor Contraction](https://en.wikipedia.org/wiki/Tensor_contraction) [ [^](#top) ]

```python
parse_latex(r"""
    % vardef 'hUD' (4D)
    h = h^\mu{}_\mu
""", reset=True, verbose=True)
```

    (Tensor(hUD, 4D), Scalar(h))

```python
print('h =', h)
```

    h = hUD00 + hUD11 + hUD22 + hUD33

### Example 2. [Index Raising](https://en.wikipedia.org/wiki/Raising_and_lowering_indices) [ [^](#top) ]

```python
parse_latex(r"""
    % vardef -metric 'gUU' (3D)
    % vardef 'vD' (3D)
    v^\mu = g^{\mu\nu} v_\nu
""", reset=True)
```

    ('gUU', 'epsilonDDD', 'gdet', 'gDD', 'GammaUDD', 'vD', 'vU')

```python
print('vU =', vU)
```

    vU = [gUU00*vD0 + gUU01*vD1 + gUU02*vD2, gUU01*vD0 + gUU11*vD1 + gUU12*vD2, gUU02*vD0 + gUU12*vD1 + gUU22*vD2]

### Example 3. [Cross Product](https://en.wikipedia.org/wiki/Cross_product) [ [^](#top) ]

```python
parse_latex(r"""
    % vardef 'vU' (3D), 'wU' (3D)
    u_i = \epsilon_{ijk} v^j w^k
""", reset=True)
```

    ('vU', 'wU', 'epsilonDDD', 'uD')

```python
print('uD =', uD)
```

    uD = [vU1*wU2 - vU2*wU1, -vU0*wU2 + vU2*wU0, vU0*wU1 - vU1*wU0]

### Example 4. [Covariant Derivative](https://en.wikipedia.org/wiki/Covariant_derivative) [ [^](#top) ]

The following are contextually inferred, dynamically generated, and injected into the global namespace for expansion of the covariant derivative $\nabla_\nu F^{\mu\nu}$
$$
\begin{align*}
    \Gamma^\mu_{ba} &= \frac{1}{2} g^{\mu c}(\partial_b\,g_{a c} + \partial_a\,g_{c b} - \partial_c\,g_{b a}) \\
    \Gamma^\nu_{ba} &= \frac{1}{2} g^{\nu c}(\partial_b\,g_{a c} + \partial_a\,g_{c b} - \partial_c\,g_{b a}) \\
    \nabla_a F^{\mu \nu} &= \partial_a F^{\mu \nu} + \Gamma^\mu_{b a} F^{b \nu} + \Gamma^\nu_{b a} F^{\mu b}
\end{align*}
$$

```python
parse_latex(r"""
    % vardef -diff_type=dD -symmetry=anti01 'FUU' (4D)
    % vardef -diff_type=dD -metric 'gDD' (4D)
    % vardef -const 'k'
    J^\mu = (4\pi k)^{-1} \nabla_\nu F^{\mu\nu}
""", reset=True)
```

    ('FUU',
     'gDD',
     'epsilonUUUU',
     'gdet',
     'gUU',
     'gDD_dD',
     'GammaUDD',
     'FUU_dD',
     'FUU_cdD',
     'JU')

```python
parse_latex(r"""
    % vardef -diff_type=dD -symmetry=anti01 'FUU' (4D)
    % vardef -diff_type=dD -metric 'ghatDD' (4D)
    % vardef -const 'k'
    J^\mu = (4\pi k)^{-1} \vphantom{dD} \hat{\nabla}_\nu F^{\mu\nu}
""", reset=True)
```

    ('FUU',
     'ghatDD',
     'epsilonUUUU',
     'ghatdet',
     'ghatUU',
     'ghatDD_dD',
     'GammahatUDD',
     'FUU_dD',
     'FUU_cdhatD',
     'JU')

### Example 5 (1). [Schwarzschild Metric](https://en.wikipedia.org/wiki/Schwarzschild_metric) [ [^](#top) ]

```python
%load_ext nrpylatex.extension
```

```python
%%parse_latex --reset --ignore-warning

% keydef basis [t, r, \theta, \phi]
% vardef -zero 'gDD' (4D)
% vardef -const 'G', 'M'
\begin{align}
    g_{t t} &= -\left(1 - \frac{2GM}{r}\right) \\
    g_{r r} &=  \left(1 - \frac{2GM}{r}\right)^{-1} \\
    g_{\theta \theta} &= r^2 \\
    g_{\phi \phi} &= r^2 \sin^2\theta
\end{align}
% assign -metric 'gDD'
```

\[
% keydef basis [t, r, \theta, \phi]
% vardef -zero 'gDD' (4D)
% vardef -const 'G', 'M'
\begin{align}
    g_{t t} &= -\left(1 - \frac{2GM}{r}\right) \\
    g_{r r} &=  \left(1 - \frac{2GM}{r}\right)^{-1} \\
    g_{\theta \theta} &= r^2 \\
    g_{\phi \phi} &= r^2 \sin^2\theta
\end{align}
% assign -metric 'gDD'
\]

```python
sp.Matrix(gDD)
```

$\displaystyle \left[\begin{matrix}\frac{2 G M}{r} - 1 & 0 & 0 & 0\\0 & \frac{1}{- \frac{2 G M}{r} + 1} & 0 & 0\\0 & 0 & r^{2} & 0\\0 & 0 & 0 & r^{2} \sin^{2}{\left(\theta \right)}\end{matrix}\right]$

### Example 5 (2). [Kretschmann Scalar](https://en.wikipedia.org/wiki/Kretschmann_scalar) [ [^](#top) ]

```python
%%parse_latex

\begin{align}
    R^\alpha{}_{\beta\mu\nu} &= \partial_\mu \Gamma^\alpha_{\beta\nu} - \partial_\nu \Gamma^\alpha_{\beta\mu}
        + \Gamma^\alpha_{\mu\gamma}\Gamma^\gamma_{\beta\nu} - \Gamma^\alpha_{\nu\sigma}\Gamma^\sigma_{\beta\mu} \\
    K &= R^{\alpha\beta\mu\nu} R_{\alpha\beta\mu\nu} \\
    R_{\beta\nu} &= R^\alpha{}_{\beta\alpha\nu} \\
    R &= g^{\beta\nu} R_{\beta\nu} \\
    G_{\beta\nu} &= R_{\beta\nu} - \frac{1}{2}g_{\beta\nu}R
\end{align}
```

\[
\begin{align}
    R^\alpha{}_{\beta\mu\nu} &= \partial_\mu \Gamma^\alpha_{\beta\nu} - \partial_\nu \Gamma^\alpha_{\beta\mu}
        + \Gamma^\alpha_{\mu\gamma}\Gamma^\gamma_{\beta\nu} - \Gamma^\alpha_{\nu\sigma}\Gamma^\sigma_{\beta\mu} \\
    K &= R^{\alpha\beta\mu\nu} R_{\alpha\beta\mu\nu} \\
    R_{\beta\nu} &= R^\alpha{}_{\beta\alpha\nu} \\
    R &= g^{\beta\nu} R_{\beta\nu} \\
    G_{\beta\nu} &= R_{\beta\nu} - \frac{1}{2}g_{\beta\nu}R
\end{align}
\]

```python
sp.simplify(sp.Matrix(RDD))
```

$\displaystyle \left[\begin{matrix}0 & 0 & 0 & 0\\0 & 0 & 0 & 0\\0 & 0 & 0 & 0\\0 & 0 & 0 & 0\end{matrix}\right]$

```python
display(sp.Matrix(GammaUDD[0][:][:]))
```

$\displaystyle \left[\begin{matrix}0 & - \frac{G M}{r^{2} \left(\frac{2 G M}{r} - 1\right)} & 0 & 0\\- \frac{G M}{r^{2} \left(\frac{2 G M}{r} - 1\right)} & 0 & 0 & 0\\0 & 0 & 0 & 0\\0 & 0 & 0 & 0\end{matrix}\right]$

```python
display(sp.Matrix(GammaUDD[1][:][:]))
```

$\displaystyle \left[\begin{matrix}\frac{G M \left(- \frac{2 G M}{r} + 1\right)}{r^{2}} & 0 & 0 & 0\\0 & - \frac{G M}{r^{2} \left(- \frac{2 G M}{r} + 1\right)} & 0 & 0\\0 & 0 & - r \left(- \frac{2 G M}{r} + 1\right) & 0\\0 & 0 & 0 & - r \left(- \frac{2 G M}{r} + 1\right) \sin^{2}{\left(\theta \right)}\end{matrix}\right]$

```python
display(sp.Matrix(GammaUDD[2][:][:]))
```

$\displaystyle \left[\begin{matrix}0 & 0 & 0 & 0\\0 & 0 & \frac{1}{r} & 0\\0 & \frac{1}{r} & 0 & 0\\0 & 0 & 0 & - \sin{\left(\theta \right)} \cos{\left(\theta \right)}\end{matrix}\right]$

```python
display(sp.Matrix(GammaUDD[3][:][:]))
```

$\displaystyle \left[\begin{matrix}0 & 0 & 0 & 0\\0 & 0 & 0 & \frac{1}{r}\\0 & 0 & 0 & \frac{\cos{\left(\theta \right)}}{\sin{\left(\theta \right)}}\\0 & \frac{1}{r} & \frac{\cos{\left(\theta \right)}}{\sin{\left(\theta \right)}} & 0\end{matrix}\right]$

For the Schwarzschild metric, the Kretschmann scalar $K$ has the property that $K\to\infty$ as $r\to 0$, and hence the metric and spacetime itself are undefined at the point of infinite curvature $r=0$, indicating the presence of a physical singularity since the Kretschmann scalar is an [invariant quantity](https://en.wikipedia.org/wiki/Curvature_invariant_(general_relativity)) in general relativity.

```python
display(sp.simplify(K))
```

$\displaystyle \frac{48 G^{2} M^{2}}{r^{6}}$

In a [vacuum region](https://en.wikipedia.org/wiki/Vacuum_solution_(general_relativity)#:~:text=In%20general%20relativity%2C%20a%20vacuum,non%2Dgravitational%20fields%20are%20present.), such as the spacetime described by the Schwarzschild metric, $T_{\mu\nu}=0$ and hence $G_{\mu\nu}=0$ since $G_{\mu\nu}=8\pi G\,T_{\mu\nu}$ ([Einstein Equations](https://en.wikipedia.org/wiki/Einstein_field_equations)).

```python
sp.simplify(sp.Matrix(GDD))
```

$\displaystyle \left[\begin{matrix}0 & 0 & 0 & 0\\0 & 0 & 0 & 0\\0 & 0 & 0 & 0\\0 & 0 & 0 & 0\end{matrix}\right]$

### Example 6 (1). [Extrinsic Curvature](https://en.wikipedia.org/wiki/Curvature) ([ADM Formalism](https://en.wikipedia.org/wiki/ADM_formalism)) [ [^](#top) ]

```python
%%parse_latex --ignore-warning

% keydef basis [r, \theta, \phi]
\begin{align}
    \gamma_{ij} &= g_{ij} \\
    % assign -metric 'gammaDD'
    \beta_i &= g_{r i} \\
    \alpha &= \sqrt{\gamma^{ij}\beta_i\beta_j - g_{r r}} \\
    K_{ij} &= \frac{1}{2\alpha}\left(\nabla_i \beta_j + \nabla_j \beta_i\right) \\
    K &= \gamma^{ij} K_{ij}
\end{align}
```

\[
% keydef basis [r, \theta, \phi]
\begin{align}
    \gamma_{ij} &= g_{ij} \\
    % assign -metric 'gammaDD'
    \beta_i &= g_{r i} \\
    \alpha &= \sqrt{\gamma^{ij}\beta_i\beta_j - g_{r r}} \\
    K_{ij} &= \frac{1}{2\alpha}\left(\nabla_i \beta_j + \nabla_j \beta_i\right) \\
    K &= \gamma^{ij} K_{ij}
\end{align}
\]

For the Schwarzschild metric (defined in the previous example), the extrinsic curvature in the ADM formalism should evaluate to zero.

```python
display(sp.Matrix(KDD))
```

$\displaystyle \left[\begin{matrix}0 & 0 & 0\\0 & 0 & 0\\0 & 0 & 0\end{matrix}\right]$

### Example 6 (2). [Hamiltonian/Momentum Constraint](https://en.wikipedia.org/wiki/Hamiltonian_constraint) [ [^](#top) ]

```python
%%parse_latex --ignore-warning

\begin{align}
    R_{ij} &= \partial_k \Gamma^k_{ij} - \partial_j \Gamma^k_{ik}
        + \Gamma^k_{ij}\Gamma^l_{kl} - \Gamma^l_{ik}\Gamma^k_{lj} \\
    R &= \gamma^{ij} R_{ij} \\
    E &= \frac{1}{16\pi}\left(R + K^{{2}} - K_{ij}K^{ij}\right) \\
    p_i &= \frac{1}{8\pi}\left(D_j \gamma^{jk} K_{ki} - D_i K\right)
\end{align}
```

\[
\begin{align}
    R_{ij} &= \partial_k \Gamma^k_{ij} - \partial_j \Gamma^k_{ik}
        + \Gamma^k_{ij}\Gamma^l_{kl} - \Gamma^l_{ik}\Gamma^k_{lj} \\
    R &= \gamma^{ij} R_{ij} \\
    E &= \frac{1}{16\pi}\left(R + K^{{2}} - K_{ij}K^{ij}\right) \\
    p_i &= \frac{1}{8\pi}\left(D_j \gamma^{jk} K_{ki} - D_i K\right)
\end{align}
\]

Every solution to the Einstein Equations, including Schwarzschild, must satisfy the Hamiltonian constraint ($E=0$) and the Momentum constraint ($p_i=0$).

```python
print('E = %s, pD = %s' % (sp.simplify(E), pD))
```

    E = 0, pD = [0, 0, 0]

## Step 4: Exception Handling and Index Checking ( [^](#top) )

We extended our robust error messaging using the custom `TensorError` exception, which should handle any inconsistent tensor dimension and any violation of the Einstein summation convention, specifically that a bound index must appear exactly once as a superscript and exactly once as a subscript in any single term and that a free index must appear in every term with the same position and cannot be summed over in any term.

```python
%%parse_latex --reset

% vardef 'TUD' (4D), 'uD' (4D)
v^\mu = T^\mu_\nu u_\nu
```

    TensorError: illegal bound index 'nu' in vU

```python
%%parse_latex --reset

% vardef 'TUD' (4D), 'uD' (4D)
v^\mu = T^\mu_\nu u_\mu
```

    TensorError: unbalanced free index {'mu', 'nu'} in vU

```python
%%parse_latex --reset

% vardef 'TUD' (4D), 'uD' (3D)
v^\mu = T^\mu_\nu u_\mu
```

    ParseError: inconsistent dimension for index 'mu'

```python
%%parse_latex --reset

% vardef 'vD' (4D)
T_{\mu\nu} = v_\mu w_\nu
```

    ParseError: T_{\mu\nu} = v_\mu w_\nu
                                   ^
    cannot index undefined tensor 'wD' at position 38

```python
%%parse_latex --reset

% vardef -symmetry=anti01 'FUU' (4D)
% vardef -const 'k'
J^\mu = (4\pi k)^{-1} \nabla_\nu F^{\mu\nu}
```

    ParseError: J^\mu = (4\pi k)^{-1} \nabla_\nu F^{\mu\nu}
                                      ^
    cannot generate covariant derivative without defined metric 'g'

## Step 5: Output Notebook to PDF ( [^](#top) )

The following code cell converts this Jupyter notebook into a proper, clickable $\LaTeX$-formatted PDF file. After the cell is successfully run, the generated PDF may be found in the root NRPy+ tutorial directory, with filename
[Tutorial-SymPy_LaTeX_Interface.pdf](Tutorial-SymPy_LaTeX_Interface.pdf) (Note that clicking on this link may not work; you may need to open the PDF file through another means.)

```python
import cmdline_helper as cmd    # NRPy+: Multi-platform Python command-line interface
cmd.output_Jupyter_notebook_to_LaTeXed_PDF("Tutorial-SymPy_LaTeX_Interface")
```

    Created Tutorial-SymPy_LaTeX_Interface.tex, and compiled LaTeX file to PDF
        file Tutorial-SymPy_LaTeX_Interface.pdf


## Histograms of Oriented Gradients (HOG)

As we saw with the ORB algorithm, we can use keypoints in images to do keypoint-based matching to detect objects in images. These type of algorithms work great when you want to detect objects that have a lot of consistent internal features that are not affected by the background. For example, these algorithms work well for facial detection because faces have a lot of consistent internal features that don’t get affected by the image background, such as the eyes, nose, and mouth. However, these type of algorithms don’t work so well when attempting to do more general object recognition, say for example, pedestrian detection in images. The reason is that people don’t have consistent internal features, like faces do, because the body shape and style of every person is different (see Fig. 1). This means that every person is going to have a different set of internal features, and so we need something that can more generally describe a person.

Fig. 1. - Pedestrians.

One option is to try to detect pedestrians by their contours instead.  Detecting objects in images by their contours (boundaries) is very challenging because we have to deal with the difficulties brought about by the contrast between the background and the foreground. For example, suppose you wanted to detect a pedestrian in an image that is walking in front of a white building and she is wearing a white coat and black pants (see Fig. 2). We can see in Fig. 2, that since the background of the image is mostly white, the black pants are going to have a very high contrast, but the coat, since it is white as well, is going to have very low contrast. In this case, detecting the edges of pants is going to be easy but detecting the edges of the coat is going to be very difficult. This is where **HOG** comes in. HOG stands for **Histograms of Oriented Gradients** and it was first introduced by Navneet Dalal and Bill Triggs in 2005.

Fig. 2. - High and Low Contrast.

The HOG algorithm works by creating histograms of the distribution of gradient orientations in an image and then normalizing them in a very special way. This special normalization is what makes HOG so effective at detecting the edges of objects even in cases where the contrast is very low. These normalized histograms are put together into a feature vector, known as the HOG descriptor, that can be used to train a machine learning algorithm, such as a Support Vector Machine (SVM), to detect objects in images based on their boundaries (edges). Due to its great success and reliability, HOG has become one of the most widely used algorithms in computer vison for object detection.

In this notebook, you will learn:

* How the HOG algorithm works
* How to use OpenCV to create a HOG descriptor
* How to visualize the HOG descriptor.

# The HOG Algorithm

As its name suggests, the HOG algorithm, is based on creating histograms from the orientation of image gradients. The HOG algorithm is implemented in a series of steps:

1. Given the image of particular object, set a detection window (region of interest) that covers the entire object in the image (see Fig. 3).

2. Calculate the magnitude and direction of the gradient for each individual pixel in the detection window.

3. Divide the detection window into connected *cells* of pixels, with all cells being of the same size (see Fig. 3). The size of the cells is a free parameter and it is usually chosen so as to match the scale of the features that want to be detected. For example, in a 64 x 128 pixel detection window, square cells 6 to 8 pixels wide are suitable for detecting human limbs.

4. Create a Histogram for each cell, by first grouping the gradient directions of all pixels in each cell into a particular number of orientation (angular) bins; and then adding up the gradient magnitudes of the gradients in each angular bin (see Fig. 3). The number of bins in the histogram is a free parameter and it is usually set to 9 angular bins.

5. Group adjacent cells into *blocks* (see Fig. 3). The number of cells in each block is a free parameter and all blocks must be of the same size. The distance between each block (known as the stride) is a free parameter but it is usually set to half the block size, in which case you will get overlapping blocks (*see video below*).  The HOG algorithm has been shown empirically to work better with overlapping blocks.

6. Use the cells contained within each block to normalize the cell histograms in that block (see Fig. 3). If you have overlapping blocks this means that most cells will be normalized with respect to different blocks (*see video below*). Therefore, the same cell may have several different normalizations.

7. Collect all the normalized histograms from all the blocks into a single feature vector called the HOG descriptor.

8. Use the resulting HOG descriptors from many images of the same type of object to train a machine learning algorithm, such as an SVM, to detect those type of objects in images. For example, you could use the HOG descriptors from many images of pedestrians to train an SVM to detect pedestrians in images. The training is done with both positive a negative examples of the object you want detect in the image.

9. Once the SVM has been trained, a sliding window approach is used to try to detect and locate objects in images.  Detecting an object in the image entails finding the part of the image that looks similar to the HOG pattern learned by the SVM.

Fig. 3. - HOG Diagram.

Vid. 1. - HOG Animation.

# Why The HOG Algorithm Works

As we learned above, HOG creates histograms by adding the magnitude of the gradients in particular orientations in localized portions of the image called *cells*.  By doing this we guarantee that stronger gradients will contribute more to the magnitude of their respective angular bin, while the effects of weak and randomly oriented gradients resulting from noise are minimized. In this manner the histograms tell us the dominant gradient orientation of each cell.

### Dealing with contrast

Now, the magnitude of the dominant orientation can vary widely due to variations in local illumination and the contrast between the background and the foreground.

To account for the background-foreground contrast differences, the HOG algorithm tries to detect edges locally.  In order to do this, it defines groups of cells, called **blocks**, and normalizes the histograms using this local group of cells. By normalizing locally, the HOG algorithm can detect the edges in each block very reliably; this is called **block normalization**.

In addition to using block normalization, the HOG algorithm also uses overlapping blocks to increase its performance. By using overlapping blocks, each cell contributes several independent components to the final HOG descriptor, where each component corresponds to a cell being normalized with respect to a different block. This may seem redundant but, it has been shown empirically that by normalizing each cell several times with respect to different local blocks, the performance of the HOG algorithm increases dramatically.

### Loading Images and Importing Resources

The first step in building our HOG descriptor is to load the required packages into Python and to load our image.

We start by using OpenCV to load an image of a triangle tile. Since, the `cv2.imread()` function loads images as BGR we will convert our image to RGB so we can display it with the correct colors. As usual we will convert our BGR image to Gray Scale for analysis.

```python
import cv2
import numpy as np
import matplotlib.pyplot as plt

# Set the default figure size
plt.rcParams['figure.figsize'] = [17.0, 7.0]

# Load the image
image = cv2.imread('./images/triangle_tile.jpeg')

# Convert the original image to RGB
original_image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)

# Convert the original image to gray scale
gray_image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

# Print the shape of the original and gray scale images
print('The original image has shape: ', original_image.shape)
print('The gray scale image has shape: ', gray_image.shape)

# Display the images
plt.subplot(121)
plt.imshow(original_image)
plt.title('Original Image')
plt.subplot(122)
plt.imshow(gray_image, cmap='gray')
plt.title('Gray Scale Image')
plt.show()
```

    The original image has shape:  (250, 250, 3)
    The gray scale image has shape:  (250, 250)

# Creating The HOG Descriptor

We will be using OpenCV’s `HOGDescriptor` class to create the HOG descriptor. The parameters of the HOG descriptor are setup using the `HOGDescriptor()` function. The parameters of the `HOGDescriptor()` function and their default values are given below:

`cv2.HOGDescriptor(win_size = (64, 128),
                  block_size = (16, 16),
                  block_stride = (8, 8),
                  cell_size = (8, 8),
                  nbins = 9,
                  win_sigma = DEFAULT_WIN_SIGMA,
                  threshold_L2hys = 0.2,
                  gamma_correction = true,
                  nlevels = DEFAULT_NLEVELS)`

Parameters:

* **win_size** – *Size*
Size of detection window in pixels (*width, height*). Defines the region of interest. Must be an integer multiple of cell size.

* **block_size** – *Size*
Block size in pixels (*width, height*). Defines how many cells are in each block. Must be an integer multiple of cell size and it must be smaller than the detection window. The smaller the block the finer detail you will get.

* **block_stride** – *Size*
Block stride in pixels (*horizontal, vertical*). It must be an integer multiple of cell size. The `block_stride` defines the distance between adjecent blocks, for example, 8 pixels horizontally and 8 pixels vertically. Longer `block_strides` makes the algorithm run faster (because less blocks are evaluated) but the algorithm may not perform as well.

* **cell_size** – *Size*
Cell size in pixels (*width, height*). Determines the size fo your cell. The smaller the cell the finer detail you will get.

* **nbins** – *int*
Number of bins for the histograms. Determines the number of angular bins used to make the histograms. With more bins you capture more gradient directions. HOG uses unsigned gradients, so the angular bins will have values between 0 and 180 degrees.

* **win_sigma** – *double*
Gaussian smoothing window parameter. The performance of the HOG algorithm can be improved by smoothing the pixels near the edges of the blocks by applying a Gaussian spatial window to each pixel before computing the histograms.

* **threshold_L2hys** – *double*
L2-Hys (Lowe-style clipped L2 norm) normalization method shrinkage. The L2-Hys method is used to normalize the blocks and it consists of an L2-norm followed by clipping and a renormalization. The clipping limits the maximum value of the descriptor vector for each block to have the value of the given threshold (0.2 by default). After the clipping the descriptor vector is renormalized as described in *IJCV*, 60(2):91-110, 2004.

* **gamma_correction** – *bool*
Flag to specify whether the gamma correction preprocessing is required or not. Performing gamma correction slightly increases the performance of the HOG algorithm.

* **nlevels** – *int*
Maximum number of detection window increases.

As we can see, the  `cv2.HOGDescriptor()`function supports a wide range of parameters. The first few arguments (`block_size, block_stride, cell_size`, and `nbins`) are probably the ones you are most likely to change. The other parameters can be safely left at their default values and you will get good results.

In the code below, we will use the `cv2.HOGDescriptor()`function to set the cell size, block size, block stride, and the number of bins for the histograms of the HOG descriptor. We will then use `.compute(image)`method to compute the HOG descriptor (feature vector) for the given `image`.

```python
# Specify the parameters for our HOG descriptor

# Cell Size in pixels (width, height). Must be smaller than the size of the detection window
# and must be chosen so that the resulting Block Size is smaller than the detection window.
cell_size = (6, 6)

# Number of cells per block in each direction (x, y). Must be chosen so that the resulting
# Block Size is smaller than the detection window
num_cells_per_block = (2, 2)

# Block Size in pixels (width, height). Must be an integer multiple of Cell Size.
# The Block Size must be smaller than the detection window
block_size = (num_cells_per_block[0] * cell_size[0],
              num_cells_per_block[1] * cell_size[1])

# Calculate the number of cells that fit in our image in the x and y directions
x_cells = gray_image.shape[1] // cell_size[0]
y_cells = gray_image.shape[0] // cell_size[1]

# Horizontal distance between blocks in units of Cell Size. Must be an integer and it must
# be set such that (x_cells - num_cells_per_block[0]) / h_stride = integer.
h_stride = 1

# Vertical distance between blocks in units of Cell Size. Must be an integer and it must
# be set such that (y_cells - num_cells_per_block[1]) / v_stride = integer.
v_stride = 1

# Block Stride in pixels (horizantal, vertical). Must be an integer multiple of Cell Size
block_stride = (cell_size[0] * h_stride, cell_size[1] * v_stride)

# Number of gradient orientation bins
num_bins = 9

# Specify the size of the detection window (Region of Interest) in pixels (width, height).
# It must be an integer multiple of Cell Size and it must cover the entire image. Because
# the detection window must be an integer multiple of cell size, depending on the size of
# your cells, the resulting detection window might be slightly smaller than the image.
# This is perfectly ok.
win_size = (x_cells * cell_size[0] , y_cells * cell_size[1])

# Print the shape of the gray scale image for reference
print('\nThe gray scale image has shape: ', gray_image.shape)
print()

# Print the parameters of our HOG descriptor
print('HOG Descriptor Parameters:\n')
print('Window Size:', win_size)
print('Cell Size:', cell_size)
print('Block Size:', block_size)
print('Block Stride:', block_stride)
print('Number of Bins:', num_bins)
print()

# Set the parameters of the HOG descriptor using the variables defined above
hog = cv2.HOGDescriptor(win_size, block_size, block_stride, cell_size, num_bins)

# Compute the HOG Descriptor for the gray scale image
hog_descriptor = hog.compute(gray_image)
```

    The gray scale image has shape:  (250, 250)

    HOG Descriptor Parameters:

    Window Size: (246, 246)
    Cell Size: (6, 6)
    Block Size: (12, 12)
    Block Stride: (6, 6)
    Number of Bins: 9

# Number of Elements In The HOG Descriptor

The resulting HOG Descriptor (feature vector), contains the normalized histograms from all cells from all blocks in the detection window concatenated in one long vector. Therefore, the size of the HOG feature vector will be given by the total number of blocks in the detection window, multiplied by the number of cells per block, times the number of orientation bins:

\begin{equation}
\mbox{total_elements} = (\mbox{total_number_of_blocks})\mbox{ } \times \mbox{ } (\mbox{number_cells_per_block})\mbox{ } \times \mbox{ } (\mbox{number_of_bins})
\end{equation}

If we don’t have overlapping blocks (*i.e.* the `block_stride`equals the `block_size`), the total number of blocks can be easily calculated by dividing the size of the detection window by the block size. However, in the general case we have to take into account the fact that we have overlapping blocks. To find the total number of blocks in the general case (*i.e.* for any `block_stride` and `block_size`), we can use the formula given below:

\begin{equation}
\mbox{Total}_i = \left( \frac{\mbox{block_size}_i}{\mbox{block_stride}_i} \right)\left( \frac{\mbox{window_size}_i}{\mbox{block_size}_i} \right) - \left [\left( \frac{\mbox{block_size}_i}{\mbox{block_stride}_i} \right) - 1 \right]; \mbox{  for  } i = x,y
\end{equation}

Where Total$_x$, is the total number of blocks along the width of the detection window, and Total$_y$, is the total number of blocks along the height of the detection window. This formula for Total$_x$ and Total$_y$, takes into account the extra blocks that result from overlapping. After calculating Total$_x$ and Total$_y$, we can get the total number of blocks in the detection window by multiplying Total$_x$  $\times$ Total$_y$. The above formula can be simplified considerably because the `block_size`, `block_stride`, and `window_size`are all defined in terms of the `cell_size`. By making all the appropriate substitutions and cancelations the above formula reduces to:

\begin{equation}
\mbox{Total}_i = \left(\frac{\mbox{cells}_i - \mbox{num_cells_per_block}_i}{N_i}\right) + 1\mbox{  }; \mbox{  for  } i = x,y
\end{equation}

Where cells$_x$ is the total number of cells along the width of the detection window, and cells$_y$, is the total number of cells along the height of the detection window. And $N_x$ is the horizontal block stride in units of `cell_size` and $N_y$ is the vertical block stride in units of `cell_size`.

Let's calculate what the number of elements for the HOG feature vector should be and check that it matches the shape of the HOG Descriptor calculated above.

```python
# Calculate the total number of blocks along the width of the detection window
tot_bx = np.uint32(((x_cells - num_cells_per_block[0]) / h_stride) + 1)

# Calculate the total number of blocks along the height of the detection window
tot_by = np.uint32(((y_cells - num_cells_per_block[1]) / v_stride) + 1)

# Calculate the total number of elements in the feature vector
tot_els = (tot_bx) * (tot_by) * num_cells_per_block[0] * num_cells_per_block[1] * num_bins

# Print the total number of elements the HOG feature vector should have
print('\nThe total number of elements in the HOG Feature Vector should be: ',
      tot_bx, 'x',
      tot_by, 'x',
      num_cells_per_block[0], 'x',
      num_cells_per_block[1], 'x',
      num_bins, '=',
      tot_els)

# Print the shape of the HOG Descriptor to see that it matches the above
print('\nThe HOG Descriptor has shape:', hog_descriptor.shape)
print()
```

    The total number of elements in the HOG Feature Vector should be:  40 x 40 x 2 x 2 x 9 = 57600

    The HOG Descriptor has shape: (57600, 1)

# Visualizing The HOG Descriptor

We can visualize the HOG Descriptor by plotting the histogram associated with each cell as a collection of vectors. To do this, we will plot each bin in the histogram as a single vector whose magnitude is given by the height of the bin and its orientation is given by the angular bin that its associated with. Since any given cell might have multiple histograms associated with it, due to the overlapping blocks, we will choose to average all the histograms for each cell to produce a single histogram for each cell.

OpenCV has no easy way to visualize the HOG Descriptor, so we have to do some manipulation first in order to visualize it. We will start by reshaping the HOG Descriptor in order to make our calculations easier.  We will then compute the average histogram of each cell and finally we will convert the histogram bins into vectors. Once we have the vectors, we plot the corresponding vectors for each cell in an image.

The code below produces an interactive plot so that you can interact with the figure. The figure contains:
* the grayscale image,
* the HOG Descriptor (feature vector),
* a zoomed-in portion of the HOG Descriptor, and
* the histogram of the selected cell.

**You can click anywhere on the gray scale image or the HOG Descriptor image to select a particular cell**. Once you click on either image a *magenta* rectangle will appear showing the cell you selected. The Zoom Window will show you a zoomed in version of the HOG descriptor around the selected cell; and the histogram plot will show you the corresponding histogram for the selected cell. The interactive window also has buttons at the bottom that allow for other functionality, such as panning, and giving you the option to save the figure if desired. The home button returns the figure to its default value.

**NOTE**: If you are running this notebook in the Udacity workspace, there is around a 2 second lag in the interactive plot. This means that if you click in the image to zoom in, it will take about 2 seconds for the plot to refresh.

```python
%matplotlib notebook

import copy
import matplotlib.patches as patches

# Set the default figure size
plt.rcParams['figure.figsize'] = [9.8, 9]

# Reshape the feature vector to [blocks_y, blocks_x, num_cells_per_block_x, num_cells_per_block_y, num_bins].
# The blocks_x and blocks_y will be transposed so that the first index (blocks_y) referes to the row number
# and the second index to the column number. This will be useful later when we plot the feature vector, so
# that the feature vector indexing matches the image indexing.
hog_descriptor_reshaped = hog_descriptor.reshape(tot_bx,
                                                 tot_by,
                                                 num_cells_per_block[0],
                                                 num_cells_per_block[1],
                                                 num_bins).transpose((1, 0, 2, 3, 4))

# Print the shape of the feature vector for reference
print('The feature vector has shape:', hog_descriptor.shape)

# Print the reshaped feature vector
print('The reshaped feature vector has shape:', hog_descriptor_reshaped.shape)

# Create an array that will hold the average gradients for each cell
ave_grad = np.zeros((y_cells, x_cells, num_bins))

# Print the shape of the ave_grad array for reference
print('The average gradient array has shape: ', ave_grad.shape)

# Create an array that will count the number of histograms per cell
hist_counter = np.zeros((y_cells, x_cells, 1))

# Add up all the histograms for each cell and count the number of histograms per cell
for i in range (num_cells_per_block[0]):
    for j in range(num_cells_per_block[1]):
        ave_grad[i:tot_by + i,
                 j:tot_bx + j] += hog_descriptor_reshaped[:, :, i, j, :]

        hist_counter[i:tot_by + i,
                     j:tot_bx + j] += 1

# Calculate the average gradient for each cell
ave_grad /= hist_counter

# Calculate the total number of vectors we have in all the cells.
len_vecs = ave_grad.shape[0] * ave_grad.shape[1] * ave_grad.shape[2]

# Create an array that has num_bins equally spaced between 0 and 180 degress in radians.
deg = np.linspace(0, np.pi, num_bins, endpoint = False)

# Each cell will have a histogram with num_bins. For each cell, plot each bin as a vector (with its magnitude
# equal to the height of the bin in the histogram, and its angle corresponding to the bin in the histogram).
# To do this, create rank 1 arrays that will hold the (x,y)-coordinate of all the vectors in all the cells in the
# image. Also, create the rank 1 arrays that will hold all the (U,V)-components of all the vectors in all the
# cells in the image. Create the arrays that will hold all the vector positons and components.
U = np.zeros((len_vecs))
V = np.zeros((len_vecs))
X = np.zeros((len_vecs))
Y = np.zeros((len_vecs))

# Set the counter to zero
counter = 0

# Use the cosine and sine functions to calculate the vector components (U,V) from their maginitudes. Remember the
# cosine and sine functions take angles in radians. Calculate the vector positions and magnitudes from the
# average gradient array
for i in range(ave_grad.shape[0]):
    for j in range(ave_grad.shape[1]):
        for k in range(ave_grad.shape[2]):
            U[counter] = ave_grad[i,j,k] * np.cos(deg[k])
            V[counter] = ave_grad[i,j,k] * np.sin(deg[k])

            X[counter] = (cell_size[0] / 2) + (cell_size[0] * i)
            Y[counter] = (cell_size[1] / 2) + (cell_size[1] * j)

            counter = counter + 1

# Create the bins in degress to plot our histogram.
angle_axis = np.linspace(0, 180, num_bins, endpoint = False)
angle_axis += ((angle_axis[1] - angle_axis[0]) / 2)

# Create a figure with 4 subplots arranged in 2 x 2
fig, ((a,b),(c,d)) = plt.subplots(2,2)

# Set the title of each subplot
a.set(title = 'Gray Scale Image\n(Click to Zoom)')
b.set(title = 'HOG Descriptor\n(Click to Zoom)')
c.set(title = 'Zoom Window', xlim = (0, 18), ylim = (0, 18), autoscale_on = False)
d.set(title = 'Histogram of Gradients')

# Plot the gray scale image
a.imshow(gray_image, cmap = 'gray')
a.set_aspect(aspect = 1)

# Plot the feature vector (HOG Descriptor)
b.quiver(Y, X, U, V, color = 'white', headwidth = 0, headlength = 0, scale_units = 'inches', scale = 5)
b.invert_yaxis()
b.set_aspect(aspect = 1)
b.set_facecolor('black')

# Define function for interactive zoom
def onpress(event):

    #Unless the left mouse button is pressed do nothing
    if event.button != 1:
        return

    # Only accept clicks for subplots a and b
    if event.inaxes in [a, b]:

        # Get mouse click coordinates
        x, y = event.xdata, event.ydata

        # Select the cell closest to the mouse click coordinates
        cell_num_x = np.uint32(x / cell_size[0])
        cell_num_y = np.uint32(y / cell_size[1])

        # Set the edge coordinates of the rectangle patch
        edgex = x - (x % cell_size[0])
        edgey = y - (y % cell_size[1])

        # Create a rectangle patch that matches the the cell selected above
        rect = patches.Rectangle((edgex, edgey),
                                  cell_size[0], cell_size[1],
                                  linewidth = 1,
                                  edgecolor = 'magenta',
                                  facecolor='none')

        # A single patch can only be used in a single plot. Create copies
        # of the patch to use in the other subplots
        rect2 = copy.copy(rect)
        rect3 = copy.copy(rect)

        # Update all subplots
        a.clear()
        a.set(title = 'Gray Scale Image\n(Click to Zoom)')
        a.imshow(gray_image, cmap = 'gray')
        a.set_aspect(aspect = 1)
        a.add_patch(rect)

        b.clear()
        b.set(title = 'HOG Descriptor\n(Click to Zoom)')
        b.quiver(Y, X, U, V, color = 'white', headwidth = 0, headlength = 0, scale_units = 'inches', scale = 5)
        b.invert_yaxis()
        b.set_aspect(aspect = 1)
        b.set_facecolor('black')
        b.add_patch(rect2)

        c.clear()
        c.set(title = 'Zoom Window')
        c.quiver(Y, X, U, V, color = 'white', headwidth = 0, headlength = 0, scale_units = 'inches', scale = 1)
        c.set_xlim(edgex - cell_size[0], edgex + (2 * cell_size[0]))
        c.set_ylim(edgey - cell_size[1], edgey + (2 * cell_size[1]))
        c.invert_yaxis()
        c.set_aspect(aspect = 1)
        c.set_facecolor('black')
        c.add_patch(rect3)

        d.clear()
        d.set(title = 'Histogram of Gradients')
        d.grid()
        d.set_xlim(0, 180)
        d.set_xticks(angle_axis)
        d.set_xlabel('Angle')
        d.bar(angle_axis,
              ave_grad[cell_num_y, cell_num_x, :],
              180 // num_bins,
              align = 'center',
              alpha = 0.5,
              linewidth = 1.2,
              edgecolor = 'k')

        fig.canvas.draw()

# Create a connection between the figure and the mouse click
fig.canvas.mpl_connect('button_press_event', onpress)
plt.show()
```

    The feature vector has shape: (57600, 1)
    The reshaped feature vector has shape: (40, 40, 2, 2, 9)
    The average gradient array has shape:  (41, 41, 9)

    <IPython.core.display.Javascript object>

# Understanding The Histograms

Let's take a look at a couple of snapshots of the above figure to see if the histograms for the selected cell make sense. Let's start looking at a cell that is inside a triangle and not near an edge:

Fig. 4. - Histograms Inside a Triangle.

In this case, since the triangle is nearly all of the same color there shouldn't be any dominant gradient in the selected cell. As we can clearly see in the Zoom Window and the histogram, this is indeed the case. We have many gradients but none of them clearly dominates over the other.

Now let’s take a look at a cell that is near a horizontal edge:

Fig. 5. - Histograms Near a Horizontal Edge.

Remember that edges are areas of an image where the intensity changes abruptly. In these cases, we will have a high intensity gradient in some particular direction. This is exactly what we see in the corresponding histogram and Zoom Window for the selected cell.  In the Zoom Window, we can see that the dominant gradient is pointing up, almost at 90 degrees, since that’s the direction in which there is a sharp change in intensity. Therefore, we should expect to see the 90-degree bin in the histogram to dominate strongly over the others. This is in fact what we see.

Now let’s take a look at a cell that is near a vertical edge:

Fig. 6. - Histograms Near a Vertical Edge.

In this case we expect the dominant gradient in the cell to be horizontal, close to 180 degrees, since that’s the direction in which there is a sharp change in intensity. Therefore, we should expect to see the 170-degree bin in the histogram to dominate strongly over the others. This is what we see in the histogram but we also see that there is another dominant gradient in the cell, namely the one in the 10-degree bin. The reason for this, is because the HOG algorithm is using unsigned gradients, which means 0 degrees and 180 degrees are considered the same. Therefore, when the histograms are being created, angles between 160 and 180 degrees, contribute proportionally to both the 10-degree bin and the 170-degree bin. This results in there being two dominant gradients in the cell near the vertical edge instead of just one.

To conclude let’s take a look at a cell that is near a diagonal edge.

Fig. 7. - Histograms Near a Diagonal Edge.

To understand what we are seeing, let’s first remember that gradients have an *x*-component, and a *y*-component, just like vectors. Therefore, the resulting orientation of a gradient is going to be given by the vector sum of its components. For this reason, on vertical edges the gradients are horizontal, because they only have an x-component, as we saw in Figure 4. While on horizontal edges the gradients are vertical, because they only have a y-component, as we saw in Figure 3. Consequently, on diagonal edges, the gradients are also going to be diagonal because both the *x* and *y* components are non-zero.  Since the diagonal edges in the image are close to 45 degrees, we should expect to see a dominant gradient orientation in the 50-degree bin. This is in fact what we see in the histogram but, just like in Figure 4., we see there are two dominant gradients instead of just one. The reason for this is that when the histograms are being created, angles that are near the boundaries of bins, contribute proportionally to the adjacent bins.  For example, a gradient with an angle of 40 degrees, is right in the middle of the 30-degree and 50-degree bin. Therefore, the magnitude of the gradient is split evenly into the 30-degree and 50-degree bin. This results in there being two dominant gradients in the cell near the diagonal edge instead of just one.

Now that you know how HOG is implemented, in the workspace you will find a notebook named *Examples*. In there, you will be able set your own paramters for the HOG descriptor for various images. Have fun!

```python

```

```python

```


```python
%matplotlib inline
import matplotlib.pylab as plt
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.utils.data import DataLoader
from torchvision import transforms, datasets

from torchmin import minimize_constr
```

```python
device = torch.device('cuda:0')

root = '/path/to/data'  # fill in torchvision dataset path
train_data = datasets.MNIST(root, train=True, transform=transforms.ToTensor())
train_loader = DataLoader(train_data, batch_size=128, shuffle=True)
```

# Train CNN classifier

```python
def CNN():
    return nn.Sequential(
        nn.Conv2d(1, 10, kernel_size=5),
        nn.SiLU(),
        nn.AvgPool2d(2),
        nn.Conv2d(10, 20, kernel_size=5),
        nn.SiLU(),
        nn.AvgPool2d(2),
        nn.Dropout(0.2),
        nn.Flatten(1),
        nn.Linear(320, 50),
        nn.Dropout(0.2),
        nn.Linear(50, 10),
        nn.LogSoftmax(1)
    )
```

```python
torch.manual_seed(382)
net = CNN().to(device)
optimizer = optim.Adam(net.parameters())
for epoch in range(10):
    epoch_loss = 0
    for (x, y) in train_loader:
        x = x.to(device, non_blocking=True)
        y = y.to(device, non_blocking=True)
        logits = net(x)
        loss = F.nll_loss(logits, y)
        optimizer.zero_grad(set_to_none=True)
        loss.backward()
        optimizer.step()
        epoch_loss += loss.item() * x.size(0)
    print('epoch %2d - loss: %0.4f' % (epoch+1, epoch_loss / len(train_data)))
```

    epoch  1 - loss: 0.4923
    epoch  2 - loss: 0.1428
    epoch  3 - loss: 0.1048
    epoch  4 - loss: 0.0883
    epoch  5 - loss: 0.0754
    epoch  6 - loss: 0.0672
    epoch  7 - loss: 0.0626
    epoch  8 - loss: 0.0578
    epoch  9 - loss: 0.0524
    epoch 10 - loss: 0.0509

# set up adversarial example environment

```python
# evaluation mode settings
net = net.requires_grad_(False).eval()

# move net to CPU
# Note: using CUDA-based inputs and objectives is allowed
# but inefficient with trust-constr, as the data will be
# moved back-and-forth from CPU
net = net.cpu()
```

```python
def nll_objective(x, y):
    assert x.numel() == 28**2
    assert y.numel() == 1
    x = x.view(1, 1, 28, 28)
    return F.nll_loss(net(x), y.view(1))
```

```python
# select a random image from the dataset
torch.manual_seed(338)
x, y = next(iter(train_loader))
img = x[0]
label = y[0]
```

```python
nll_objective(img, label)
```

    tensor(1.4663e-05)

```python
# minimization objective for adversarial examples
#   goal is to maximize NLL of perturbed image (image + perturbation)
fn = lambda eps: - nll_objective(img + eps, label)
```

```python
# plotting utility

def plot_distortion(img, eps, y):
    assert img.numel() == 28**2
    assert eps.numel() == 28**2
    img = img.view(28, 28)
    img_ = img + eps.view(28, 28)
    fig, axes = plt.subplots(1,2,figsize=(4,2))
    for i, x in enumerate((img, img_)):
        axes[i].imshow(x.cpu(), cmap=plt.cm.binary)
        axes[i].set_xticks([])
        axes[i].set_yticks([])
        axes[i].set_title('nll: %0.4f' % nll_objective(x, y))
    plt.show()
```

# craft adversarial example

We will use our constrained optimizer to find the optimal unit-norm purturbation $\epsilon$

\begin{equation}
\max_{\epsilon} NLL(x + \epsilon) \quad \text{s.t.} \quad ||\epsilon|| = 1
\end{equation}

```python
torch.manual_seed(227)
eps0 = torch.randn_like(img)
eps0 /= eps0.norm()
```

```python
fn(eps0).item()
```

    -2.2291887944447808e-05

```python
res = minimize_constr(
    fn, eps0,
    max_iter=100,
    constr=dict(
        fun=lambda x: x.square().sum(),
        lb=1, ub=1
    ),
    disp=1
)
```

    `xtol` termination condition is satisfied.
    Number of iterations: 32, function evaluations: 50, CG iterations: 52, optimality: 1.02e-04, constraint violation: 0.00e+00, execution time: 0.57 s.

```python
eps = res.x
print(eps.norm())
```

    tensor(1.)

```python
plot_distortion(img.detach(), eps, label)
```

```python

```