```python
%pylab inline
import pandas as pd

import numpy as np
from __future__ import division
import itertools

import matplotlib.pyplot as plt
import seaborn as sns

import logging
logger = logging.getLogger()
```

    Populating the interactive namespace from numpy and matplotlib

3 Finding Similar Items
===================
Problem:
+ "similar": finding sets with a relatively large intersection.
    - shingling
    - minhashing
+ too many pairs of items to test
    - locality-sentitive hashing

### 3.1 Applications of Near-Neighbor Search

The *Jaccard Similarity* of sets $S$ and $T$:
\begin{equation}
SIM(S,T) = \frac{| S \cup T |}{| S \cap T |}
\end{equation}

##### Similar-sets Problem
1. Similar Documents
In many applicatons, the documents are not identical, yet they share large portions of their text.
eg: Plagiarism, Mirror Pages, Articles from the Same Source.
2. Collaborative Filtering(Similar Tastes)
eg:
    + Online Purchases
    + Movie Ratings
        When data is not binary:
        - Ignore low-rated customer/movie pairs
        - "liked" VS "hated"
        - 1-to-5-starts, put it in a set $n$ times. <= _Jaccard similarity for bags_

##### Exercises

```python
#Exercise 3.1.1
logger.setLevel(logging.WARNING)

data = [
    set([1, 2, 3, 4]),
    set([2, 3, 5, 7]),
    set([2, 4, 6]),
]

def Jaccard_similarity_calc(set_a, set_b):
    """calculate the Jaccard similarity of two sets

    res = \frac{a \cap b}{a \cup b}
    """

    assert isinstance(set_a, set), '{} is not a set'.format(set_a)
    assert isinstance(set_b, set), '{} is not a set'.format(set_b)

    logging.debug('a:{}, b:{}'.format(set_a, set_b))
    logging.debug('inter:{}, union:{}'.format(set_a.intersection(set_b), set_a.union(set_b)))
    return len(set_a.intersection(set_b)) / len(set_a.union(set_b))

for comb in list(itertools.combinations(range(3),2)):
    set_a, set_b = data[comb[0]], data[comb[1]]
    print('a:{}, b:{}, SIM:{:.3f}'.format(set_a, set_b, Jaccard_similarity_calc(set_a, set_b)))

```

    a:set([1, 2, 3, 4]), b:set([2, 3, 5, 7]), SIM:0.333
    a:set([1, 2, 3, 4]), b:set([2, 4, 6]), SIM:0.400
    a:set([2, 3, 5, 7]), b:set([2, 4, 6]), SIM:0.167

```python
#Exercise 3.1.2
logger.setLevel(logging.WARNING)

data = [
    [1, 1, 1, 2],
    [1, 1, 2, 2, 3],
    [1, 2, 3, 4],
]

def Jaccard_bag_similarity_calc(a, b):
    """calculate the Jaccard bag similarity of two sets

    see page 76, movie ratings 3.
    intersecton = min(times of element in the two sets)
    union = sum of length of the two sets
    res = intersection / union
    """

    from collections import Counter

    count_a = Counter(a)
    count_b = Counter(b)
    logging.debug('count_a:{}\n count_b:{}'.format(count_a,count_b))

    inter = [min(count_a[x], count_b[x]) for x in count_a if x in count_b]
    logging.debug('intersection:{}'.format(inter))

    return sum(inter) / (len(a) + len(b))

for comb in list(itertools.combinations(range(3),2)):
    set_a, set_b = data[comb[0]], data[comb[1]]
    print('a:{}, b:{}, JbSIM:{:.3f}'.format(set_a, set_b, Jaccard_bag_similarity_calc(set_a, set_b)))

```

    a:[1, 1, 1, 2], b:[1, 1, 2, 2, 3], JbSIM:0.333
    a:[1, 1, 1, 2], b:[1, 2, 3, 4], JbSIM:0.250
    a:[1, 1, 2, 2, 3], b:[1, 2, 3, 4], JbSIM:0.333

_exercise 3.1.3_

引理：
定义

\begin{equation}
    I\{A\} = \begin{cases}
                 1 & \text{if $A$ occurs.} \\
                 0 & \text{otherwise.}
             \end{cases}
\end{equation}

令 $X_A = I\{A\}$，则有 $E[X_A] = \sum I\{A\}P[A] = P[A]$

令 $x_i$ 表示 S 和 T 中有 $i$ 个相同元素，则：

\begin{align}
    E[x_i] & = P[\text{S 和 T 中有 $i$ 个相同元素}] \\
           & = \frac{C_n^i \, C_{n-i}^{2(m-i)}}{C_n^{m} C_n^{m}}
\end{align}

所以

\begin{align}
    E[J] & = \sum_{i=0}^m E[x_i] \, J_i \\
         & = \frac{\sum_{i=0}^m \, C_n^i \, C_{n-i}^{2(m-i)} \, \frac{i}{2(m-i)}}{C_n^{m} C_n^{m}}
\end{align}

`#todo 推导最终式`

### 3.2 Shingling of Documents

**k-shingle**: any substring of length $k$ found within the document.
+ variation: set -> bag
+ multiple white-space => one white-space
+ choosing k
    - **large enough** that prob. of **any given shingle** appearing in **any given document** is **low**.
      k 越大，对相似性要求越高
    - estimate the number of k-shingles as $20^k$, instead of 27 char, 因为各个字符的出现频率并非一致
    - in general,
      email: k = 5;
      large documents, such as research articles: k = 9
+ stop words, such as "and", "you", "to"
    - ignore them in many application
    - **"stop words + two words"** used as shingle in finding similar **news articles**.

**hashing Shingles**: h(shingle) = bucket
+ 9-shingles, then hash to 4 char > 4-shingles

```python
logger.setLevel(logging.WARNING)

def to_alphanumeric_string(s):
    """remove any non-alphanumeric characte in string s.
    """
    import re

    s = s.lower()
    s = re.sub('\W', ' ', s)
    logging.debug('after: del special char: {}'.format(s))
    s = re.sub('\s+', ' ', s).strip()
    logging.debug('after: del multiple whitespace: {}'.format(s))

    return s

def k_shingle_extrac(document, k):
    """extract all k-shingles from document.

    """
    document = to_alphanumeric_string(document)

    assert len(document) >= k, 'k should be less than the length of document'

    k_shingles = []
    while len(document) >= k:
        shingle = document[0:k]
        if shingle not in k_shingles:
            k_shingles.append(shingle)
        document = document[1:]

    return k_shingles

print('Example 3.3: {}'.format(k_shingle_extrac('abcdabd', 2)))
print('other example: {}'.format(k_shingle_extrac("Hello, I'm So-So.  ", 2)))
```

    Example 3.3: ['ab', 'bc', 'cd', 'da', 'bd']
    other example: ['he', 'el', 'll', 'lo', 'o ', ' i', 'i ', ' m', 'm ', ' s', 'so']

##### exercise

```python
#exercise 3.2.1

data = "The most effective way to represent documents as sets, for the purpose of iden- tifying lexically similar documents is to construct from the document the set of short strings that appear within it. "

print('first 10 3-shingles:{}'.format(k_shingle_extrac(data,3)[0:10]))
```

    first 10 3-shingles:['the', 'he ', 'e m', ' mo', 'mos', 'ost', 'st ', 't e', ' ef', 'eff']

```python
#exercise 3.2.2
logger.setLevel(logging.WARNING)

def stopwords_shingles_extract(document, k):
    """extract the stopwords-shingle.

    stropwords-shingle = stop-words + k words
    """
    stop_words_list = ['the', 'you', 'to', 'as', 'for', 'of', 'is', 'that', 'it', 'from']

    document = to_alphanumeric_string(document)
    document = document.split()
    logging.debug('split:{}'.format(document))

    shingles = []
    k = k + 1 #len(shingle) = 1 stop-word + k words
    while(document):
        try:
            logging.debug('check:{}'.format(document[0]))
            if document[0] in stop_words_list:
                shingle = ' '.join(document[0:k])
                logging.debug('hit: {}'.format(shingle))

                if shingle not in shingles:
                    shingles.append(shingle)

        except IndexError:
            logging.debug('Index Error: no of char:{}, k: {}'.format(len(document), k))
            k = len(document)
            continue

        document = document[1:]

    return shingles

print('stop-shingles:{}'.format(stopwords_shingles_extract(data, 2)))

```

    stop-shingles:['the most effective', 'to represent documents', 'as sets for', 'for the purpose', 'the purpose of', 'of iden tifying', 'is to construct', 'to construct from', 'from the document', 'the document the', 'the set of', 'of short strings', 'that appear within', 'it']

`#exercise 3.2.3`

k-shingles 最大数量：所有提取的 shingle 均不同
故从左向右依次取片，有：
the largest number = n - (k - 1) = n - k + 1

### 3.3 Similarity-Preserving Summaries of Sets

##### Matrix Representation of Sets:
(r,c) = 1: the element for row r is a member of the set for column c.
(r,c) = 0: otherwise

+ matrices are almost always **sparse**

```python
#Example 3.6

logger.setLevel(logging.WARNING)

my_database = ['a', 'b', 'c', 'd', 'e']
my_features_dict = {
    'S1':['a', 'd'],
    'S2':['c'],
    'S3':['b', 'd', 'e'],
    'S4':['a', 'c', 'd']
    }

def matrix_representation_create(database, features_dict):
    """create the matrix representation of one database.

    """
    matrix_ = np.zeros((len(database), len(features_dict)), dtype=np.int)
    matrix = pd.DataFrame(matrix_, index=database, columns=sorted(features_dict.keys()))

    for feature_name, values in features_dict.iteritems():
        for value in values:
            matrix.loc[value, feature_name] = 1

    return matrix

my_matrix = matrix_representation_create(my_database, my_features_dict)
my_matrix
```

      S1
      S2
      S3
      S4

      a
      1
      0
      0
      1

      b
      0
      0
      1
      0

      c
      0
      1
      0
      1

      d
      1
      0
      1
      1

      e
      0
      0
      1
      0

##### Minhashing

_target_: sets of shingles are large =====> much smaller representations called "**signatures**"

**minhash**
1. pick a permutation of the rows
2. h(columu c) = the row which has the first "1"

```python
#Example 3.7

logger.setLevel(logging.WARNING)

def minhash(matrix, row_orders):
    """calculate the minhash value of matrix according to element permutation in Fig 3.3.

    """
    hash_fun_names = ['h{}'.format(i) for i in range(1, len(row_orders)+1)]
    hash_table = pd.DataFrame(np.zeros((len(row_orders), matrix.shape[1])), index=hash_fun_names, columns=matrix.columns)

    for row_order, hash_fun_name in zip(row_orders, hash_fun_names):
        matrix_p = matrix.loc[row_order,:]
        logging.debug('after permutation: \n{}'.format(matrix_p))

        for c in matrix_p.columns:
            first_one_index = next((i for i, x in enumerate(matrix_p.loc[:,c]) if x), None)
            hash_table.loc[hash_fun_name, c] = row_order[first_one_index]

    return hash_table

minhash(my_matrix, [['b', 'e', 'a', 'd', 'c']])

```

      S1
      S2
      S3
      S4

      h1
      a
      c
      b
      a

**minhash and Jaccard Similarity**: $P[h(S1) = h(S2)] = SIM(S1, S2)$

*Proof*:
rows can be divided into three classes:

+ Type X rows have 1 in both columns (means both S1 and S2 have these features)
+ Type Y rows have 1 in either S1 or S2
+ Type Z rows have 0 in both columns (means both S1 and S2 don't have them)

\begin{align}
    | S1 \cap S2 | & = len(X) \\
    | S1 \cap S2 | & = len(X+Y) \\
    SIM(S1, S2) &= \frac{len(X)}{len(X+Y)}
\end{align}

randomly permute rows, there is at least one colunm whose value is 1:
$$ P[h(S1) = h(S2)] = \frac{len(x)}{len(X+Y)} $$

**minhash signature**:
use vector $[h_1(S), h_2(S), ... , h_n(S)]$ to represent matrix M
namely, reduce the dim: columns(M) -----> n

**Computing Minhash Signatures**
picking a random permutation of large rows and find its hash value both are **time-consuming** ==> simulation

```python
logger.setLevel(logging.WARNING)

my_matrix.index = range(my_matrix.shape[0])
print('matrix:\n{}\n'.format(my_matrix))

h_rows = [
    [1, 2, 3, 4, 0],
    [1, 4, 2, 0, 3]
]

def minhash_by_rows(matrix, row_orders):
    """calculate the minhash value of matrix according to row_permutation in Fig 3.4.

    """
    hash_fun_names = ['h{}'.format(i) for i in range(1, len(row_orders)+1)]
    hash_table = pd.DataFrame(np.zeros((len(row_orders), matrix.shape[1])), index=hash_fun_names, columns=matrix.columns)

    for row_order, hash_fun_name in zip(row_orders, hash_fun_names):
        logging.debug('row_order:{}, h:{}'.format(row_order, hash_fun_name))
        matrix_p = matrix.copy()
        matrix_p.index = row_order #new rows permutation
        matrix_p.sort_index(inplace=True) #adjust rows orders

        logging.debug('after permutation: \n{}'.format(matrix_p))

        for c in matrix_p.columns:
            first_one_index = next((i for i, x in enumerate(matrix_p.loc[:,c]) if x), None)
            hash_table.loc[hash_fun_name, c] = first_one_index

    return hash_table

my_minhash_res = minhash_by_rows(my_matrix, h_rows)
print('minhash: 5dim -> 2dim \n\
       true hash res:\n{}\n'.format(my_minhash_res))

print('Minhashing')

for comb in list(itertools.combinations(range(4),2)):
    s_a, s_b = 'S{}'.format(comb[0]+1), 'S{}'.format(comb[1]+1)
    print('{}-{}'.format(s_a, s_b)),

    set_a, set_b = set(my_minhash_res.iloc[:,comb[0]]), set(my_minhash_res.iloc[:,comb[1]])
    print('minhash:{:.3f}'.format(Jaccard_similarity_calc(set_a, set_b))),
    print('true SIM:{:.3f}'.format(Jaccard_similarity_calc(set(my_features_dict[s_a]), set(my_features_dict[s_b]))))

```

    matrix:
       S1  S2  S3  S4
    0   1   0   0   1
    1   0   0   1   0
    2   0   1   0   1
    3   1   0   1   1
    4   0   0   1   0

    minhash: 5dim -> 2dim
           true hash res:
        S1  S2  S3  S4
    h1   1   3   0   1
    h2   0   2   0   0

    Minhashing
    S1-S2 minhash:0.000 true SIM:0.000
    S1-S3 minhash:0.500 true SIM:0.250
    S1-S4 minhash:1.000 true SIM:0.667
    S2-S3 minhash:0.000 true SIM:0.000
    S2-S4 minhash:0.000 true SIM:0.333
    S3-S4 minhash:0.500 true SIM:0.200

对 $M$ 中的所有行做一次随机组合得到 $\bar{M}$，非常耗时。
解决方法是不直接操作矩阵 $M$，而是分两步操作：

1) 选择合适的哈希函数生成行号序列来模拟随机排列结果，如上个代码区块中 `h_rows` 即可用 $h_1(x) = (x + 1) \% 5$ 和 $h_2(x) = (3x + 1) \% 5$ 生成对应的行号

```python
def add_hash_func(a, b, c):
    return lambda x: (a*x + b) % c

h_funcs = [
    add_hash_func(1, 1, 5),
    add_hash_func(3, 1, 5)
    ]

h_rows = []
for h_func in h_funcs:
    h_rows.append(map(h_func, range(5)))

print('h_rows:{}'.format(h_rows))
```

    h_rows:[[1, 2, 3, 4, 0], [1, 4, 2, 0, 3]]

2) 再按照行号重新对行排序，生成新矩阵 $\bar{M}$
但第 2 步排序仍然需要操作矩阵，见函数 `minhash_by_rows` 中的语句 `matrix_p.sort_index(inplace=True)`。

```python
#Fig 3.4

df_matrix = my_matrix
df_h_rows = pd.DataFrame(np.array(h_rows).T)
df_h_rows.columns = ['h{}'.format(x+1) for x in df_h_rows.columns]

print('Hash functions computed for the matrix:\n{}\n'.format(pd.concat([df_matrix, df_h_rows], axis=1)))
print('signature matrix\n(SIG):\n{}'.format(my_minhash_res))
```

    Hash functions computed for the matrix:
       S1  S2  S3  S4  h1  h2
    0   1   0   0   1   1   1
    1   0   0   1   0   2   4
    2   0   1   0   1   3   2
    3   1   0   1   1   4   0
    4   0   0   1   0   0   3

    signature matrix
    (SIG):
        S1  S2  S3  S4
    h1   1   3   0   1
    h2   0   2   0   0

Let SIG(i,c) be the element of the signature matrix for the ith hash function and column c.

替代方法是借助思路：$ SIG(i,c) = min(h_i(S_c=1)) $
即对于 $h_i(S_c)$ 值来说，它只可能是 “1” 对应的行号，而第一个 “1” 肯定是最小的行号。

比如，我们想得到 SIG(1,1) 的值，
$S1 = [1, 0, 0, 1, 0]^T$，对应的 $h1 = [1, 2, 3, 4, 0]^T$，
SIG(1,1) 只能取 “1” 的行号，即 1 或 4，
又 SIG 是第一个遇到的 “1”，所以 SIG(1,1) 肯定是 1。

**思路总结**：
我们对每个 SIG(i,c) 都可以通过遍历 $S_i$ 中 "1" 位置，取出 $h_c$ 中相应位置的行号，行号中最小的值就是解。

而这个思路是可以并行化处理的，有两种方法：

1. 列遍历：即从左至右遍历 $S_i$ 得到 "1" 位置时，我们可以据此取出所有 $h_c$ 相应的行号，分别取最小。
2. 行遍历：书中 Example3.8 例子，从上往下遍历，逐行比较。
   步骤如下：
   1. Compute $h_1(r), h_2(r), ... , h_n(r)$.
   2. For each column $c$ do the following:
       + If $c$ has $0$ in row $r$, do nothing
       + else, if $c$ has $1$ in row $r$,
       for each $i = 1,2, ..., n$,
       $SIG(i,c) = min \left( SIG(i,c), \, h_i(r) \right)$

行遍历的方法，粗略想，计算量应该更少。
`#todo: 验证计算量`

```python
#Example 3.8

logger.setLevel(logging.WARNING)

def minhash_signatures_calc(df_M, hash_funcs, nagging=False):
    """computing minhash signatures by the way in Example 3.8.

    """
    logging.debug('data matrix:\n{}\n'.format(df_M))

    h = []

    for hash_func in hash_funcs:
        h.append(map(hash_func, range(df_M.shape[0])))

    df_h = pd.DataFrame(np.array(h).T)
    df_h.columns = ['h{}'.format(x+1) for x in df_h.columns]
    logging.debug('hash matrix:\n{}\n'.format(df_h))

    if nagging:
        print('hash matrix:\n{}\n'.format(pd.concat([df_matrix, df_h], axis=1)))

    df_signatures = pd.DataFrame(np.ones((df_h.shape[1], df_M.shape[1]))*np.inf, index=df_h.columns, columns=df_M.columns)
    logging.debug('signatures matrix:\ninit\n{}\n'.format(df_signatures))

    for r in df_M.index:
        for c in df_h.columns:
            r_1_loc = df_M.loc[r,:] == 1
            logging.debug('r:{}, c:{}, 1 loc:\n{}\n'.format(r,c, r_1_loc))

            sig_c = df_signatures.loc[c,:]
            line_bigger_loc = sig_c > df_h.loc[r, c]
            logging.debug('bigger row loc:\n{}\n'.format(line_bigger_loc))

            sig_c[line_bigger_loc & r_1_loc] = df_h.loc[r, c]
            logging.debug('modified:\n{}\n'.format(sig_c))

            df_signatures.loc[c,:] = sig_c
        if nagging:
            print('row:{},\n signature matrix:\n{}\n'.format(r, df_signatures))

    return df_signatures

minhash_signatures_calc(df_matrix, h_funcs, nagging=True)
```

    hash matrix:
       S1  S2  S3  S4  h1  h2
    0   1   0   0   1   1   1
    1   0   0   1   0   2   4
    2   0   1   0   1   3   2
    3   1   0   1   1   4   0
    4   0   0   1   0   0   3

    row:0,
     signature matrix:
        S1   S2   S3  S4
    h1   1  inf  inf   1
    h2   1  inf  inf   1

    row:1,
     signature matrix:
        S1   S2  S3  S4
    h1   1  inf   2   1
    h2   1  inf   4   1

    row:2,
     signature matrix:
        S1  S2  S3  S4
    h1   1   3   2   1
    h2   1   2   4   1

    row:3,
     signature matrix:
        S1  S2  S3  S4
    h1   1   3   2   1
    h2   0   2   0   0

    row:4,
     signature matrix:
        S1  S2  S3  S4
    h1   1   3   0   1
    h2   0   2   0   0

      S1
      S2
      S3
      S4

      h1
      1
      3
      0
      1

      h2
      0
      2
      0
      0

##### exercise 3.3

```python
#exercise 3.3.1

#generate 120 permutations
h_rows = list(itertools.permutations(range(5),5))
my_minhash_res = minhash_by_rows(my_matrix, h_rows)

for comb in list(itertools.combinations(range(4),2)):
    s_a, s_b = 'S{}'.format(comb[0]+1), 'S{}'.format(comb[1]+1)
    print('{}-{}'.format(s_a, s_b)),
    #calc Jaccard similarity
    print('true SIM:{:.3f}'.format(Jaccard_similarity_calc(set(my_features_dict[s_a]), set(my_features_dict[s_b])))),
    #calc the fraction of the 120 permutations in which the value is same
    print('fraction:{:.3f}'.format(sum(my_minhash_res.loc[:,s_a] == my_minhash_res.loc[:,s_b])/120))

```

    S1-S2 true SIM:0.000 fraction:0.000
    S1-S3 true SIM:0.250 fraction:0.250
    S1-S4 true SIM:0.667 fraction:0.667
    S2-S3 true SIM:0.000 fraction:0.000
    S2-S4 true SIM:0.333 fraction:0.333
    S3-S4 true SIM:0.200 fraction:0.200

```python
#exercise 3.3.2
h_funcs[2:4] = [
    add_hash_func(2, 4, 5),
    add_hash_func(3, -1, 5)
]

minhash_signatures_calc(df_matrix, h_funcs, nagging=True)
```

    hash matrix:
       S1  S2  S3  S4  h1  h2  h3  h4
    0   1   0   0   1   1   1   4   4
    1   0   0   1   0   2   4   1   2
    2   0   1   0   1   3   2   3   0
    3   1   0   1   1   4   0   0   3
    4   0   0   1   0   0   3   2   1

    row:0,
     signature matrix:
        S1   S2   S3  S4
    h1   1  inf  inf   1
    h2   1  inf  inf   1
    h3   4  inf  inf   4
    h4   4  inf  inf   4

    row:1,
     signature matrix:
        S1   S2  S3  S4
    h1   1  inf   2   1
    h2   1  inf   4   1
    h3   4  inf   1   4
    h4   4  inf   2   4

    row:2,
     signature matrix:
        S1  S2  S3  S4
    h1   1   3   2   1
    h2   1   2   4   1
    h3   4   3   1   3
    h4   4   0   2   0

    row:3,
     signature matrix:
        S1  S2  S3  S4
    h1   1   3   2   1
    h2   0   2   0   0
    h3   0   3   0   0
    h4   3   0   2   0

    row:4,
     signature matrix:
        S1  S2  S3  S4
    h1   1   3   0   1
    h2   0   2   0   0
    h3   0   3   0   0
    h4   3   0   1   0

      S1
      S2
      S3
      S4

      h1
      1
      3
      0
      1

      h2
      0
      2
      0
      0

      h3
      0
      3
      0
      0

      h4
      3
      0
      1
      0

```python
#exercise 3.3.3

my_database = range(6)
my_features_dict = {
    'S1':[2, 5],
    'S2':[0, 1],
    'S3':[3, 4],
    'S4':[0, 2, 4]
    }

df_fig_3_5 = matrix_representation_create(my_database, my_features_dict)
print('Fig 3.5:\n{}\n'.format(df_fig_3_5))

#(a)
h_funcs = [
    add_hash_func(2, 1, 6),
    add_hash_func(3, 2, 6),
    add_hash_func(5, 2, 6)
]
df_matrix = df_fig_3_5
my_minhash_res = minhash_signatures_calc(df_matrix, h_funcs, nagging=True)
my_minhash_res
```

    Fig 3.5:
       S1  S2  S3  S4
    0   0   1   0   1
    1   0   1   0   0
    2   1   0   0   1
    3   0   0   1   0
    4   0   0   1   1
    5   1   0   0   0

    hash matrix:
       S1  S2  S3  S4  h1  h2  h3
    0   0   1   0   1   1   2   2
    1   0   1   0   0   3   5   1
    2   1   0   0   1   5   2   0
    3   0   0   1   0   1   5   5
    4   0   0   1   1   3   2   4
    5   1   0   0   0   5   5   3

    row:0,
     signature matrix:
         S1  S2   S3  S4
    h1  inf   1  inf   1
    h2  inf   2  inf   2
    h3  inf   2  inf   2

    row:1,
     signature matrix:
         S1  S2   S3  S4
    h1  inf   1  inf   1
    h2  inf   2  inf   2
    h3  inf   1  inf   2

    row:2,
     signature matrix:
        S1  S2   S3  S4
    h1   5   1  inf   1
    h2   2   2  inf   2
    h3   0   1  inf   0

    row:3,
     signature matrix:
        S1  S2  S3  S4
    h1   5   1   1   1
    h2   2   2   5   2
    h3   0   1   5   0

    row:4,
     signature matrix:
        S1  S2  S3  S4
    h1   5   1   1   1
    h2   2   2   2   2
    h3   0   1   4   0

    row:5,
     signature matrix:
        S1  S2  S3  S4
    h1   5   1   1   1
    h2   2   2   2   2
    h3   0   1   4   0

      S1
      S2
      S3
      S4

      h1
      5
      1
      1
      1

      h2
      2
      2
      2
      2

      h3
      0
      1
      4
      0

```python
#(b) h_3 is a true permutation.

#(c)

for comb in list(itertools.combinations(range(4),2)):
    s_a, s_b = 'S{}'.format(comb[0]+1), 'S{}'.format(comb[1]+1)
    print('{}-{}'.format(s_a, s_b)),

    set_a, set_b = set(my_minhash_res.iloc[:,comb[0]]), set(my_minhash_res.iloc[:,comb[1]])
    print('minhash:{:.3f}'.format(Jaccard_similarity_calc(set_a, set_b))),
    print('true SIM:{:.3f}'.format(Jaccard_similarity_calc(set(my_features_dict[s_a]), set(my_features_dict[s_b]))))

```

    S1-S2 minhash:0.250 true SIM:0.000
    S1-S3 minhash:0.200 true SIM:0.000
    S1-S4 minhash:0.500 true SIM:0.250
    S2-S3 minhash:0.667 true SIM:0.000
    S2-S4 minhash:0.667 true SIM:0.250
    S3-S4 minhash:0.500 true SIM:0.250

`#exercise 3.3.4`
不是很明白意思，大概想法是：
从 $n$ 中抽取 $m$ 个元素集合 $S$ 和 $T$，则从 $S$ 和 $T$ 中各抽一次，元素相同的概率是多少？

`#exercise 3.3.5`
元素相同的概率是 0

### 3.4 Locality-Sensitive Hashing for Documents

minhash 降低了维度，但计算耗时问题仍然存在：
令样本集有 $k$ 个元素，两两比较次数是 $C_k^2 = O(k^2)$。
若 $k$ 很大，则比较次数成乘方倍增长。

解决思路：
1. 并行处理
2. 只比较可能相似的样本
   often we want only the most similar pairs or all paris that are above some lower bound in similarity.
   ----> Locality-sensitive hasing (LSH) or near-neighbor search.

**LSH**:
"hash" items several times, in such a way: $$P[\text{similar items in the same bucket}] >> P[\text{dissimilar ...}]$$

candidate pair: any pair hashed to the same bucket.
false postive: dissimilar items in the candidate pair.
false negtive: similar items NOT in the cnadidate pair.

if we have minhash signatures for the items, an effective way to choose the hashings is:
1. divide the signature matrix into $b$ bands consisting of $r$ rows each.
2. for each band $b_i$, hash each vector in $b_i$
   hash function $h$: same vectors -> same bucket

也就是说，将两个样本向量分割，只要有一个节点哈希到一起，就认为是候选对。

注意，统计的是每个节点各自哈希的情况，所以每个节点使用相同的哈希函数，但使用独立的哈希表存储中间结果。如下图：

matrix | --> | hash table
:-------:|:------:|:------------:
band 1 | h(v) | hash_talbe_1
band 2 | h(v) | hash_talbe_2
....   | h(v) | ....

**Analysis** the probabilty of two pairs to become a candidate pairs:
Given the pairs $s_a$ and $s_b$ have Jaccard similarity $s$, we use $b$ bands of $r$ row each to split their signature matrix $M_a$ and $M_b$.
1. $\forall \, i^{th} \, \text{row}, \, prob_{rowY} = P[M_a[i,:] == M_b[i,:]] = \text{Jaccard similarity} = s$
2. the possibility that one band($r$ row) is the same is: $$prob_{bandY} = prob_{rowY}^r = s^r$$
   .......................................not the same is: $$prob_{bandN} = 1 - prob_{bandY} = 1 - s^r$$
3. $s_a$ and $s_b$ are totally different: $$prob_{pairN} = prob_{bandN}^b = (1 - s^r)^b$$
   $s_a$ and $_b$ **at least agree in one band**: $$prob_{pairY} = 1 - prob_{pairN} = 1 - (1 - s^r)^b$$

**threshold**: the value of $s$ at which the probability of becoming a candidate is 1/2.
namely, $1 - (1 - s^r)^b = \frac{1}{2}$, the solution $s \approx (1/b)^{1/r}$

```python
#Example 3.11

b = 20
r = 5
s = np.linspace(0, 1, 100)

def p(s, r, b):
    return 1 - (1 - s**r)**b

plt.plot(s,p(s, r, b))

s = np.arange(0.2, 0.9, 0.1)

zip(s, p(s, r, b))
```

if we consider two documents with 80% similarity, then their false negative is $1 - 0.99964... \approx 0.00035$, which means that only roughly 1 in 3000 pairs that are as high as 80% similar will fail to become a candidate pair.

##### Combining the Techniques
整合前面讲过的方法，找到相似文本的方法如下：

1. k-shingles
   [opt] hash k-shignles to shorter bucket numbers

2. Sort the document-shigles pairs
   不用生成完成的矩阵，而是依据 shingles 顺序后面用到时逐行生成（节省内存）

3. Pick n, compute the minhash signatures by feed the sorted list line by line
   逐行遍历

4. Choose a threshold $t$
   $t \approx (1/b)^{1/r}$ and $n = b \times n$ to determine appropriate $b, n$.

5. Construct candidate pairs using LSH

6. Examine each candidate pair's signatures, $SIM > t$?

7. [opt] check document themselve

`#todo: 找个例子，代码实现`

##### exercise 3.4
`#maybe:`

`#exercise 3.4.3`

`#exercise 3.4.4`

### 3.5 Distance Measures

**LSH** 本身的意义：
就是为一个特定的距离 $l$，找到合适的哈希函数 $h$ 来实现 "S" 曲线的效果：
即此距离下越近，哈希到同一区块的可能性越高。

这章介绍距离，为下一章找哈希函数打基础。

##### 距离的定义

定义集合 $X$ 上的度量為一函數距离 $d: X \times X \to \mathbf{R}$
对于 $\forall x, y, z \in X$，应满足：

1. 正定性
$d(x, y) \ge 0$
$d(x, y) = 0$ if and only if $x = y$

2. 交换性
$d(x, y) = d(y, x)$

3. 三角不等式
$d(x, y) \le d(x, z) + d(z, y)$

##### 几种距离
1. Euclidean Distance: $\| \cdot \|_2$
   $L_r\text{-norm} = (\sum \| \cdot \|_1^r)^{1/r}$
   $L_\infty\text{-norm} = max(\| \cdot \|_1)$

2. Jaccard Distance: $d(x, y) = 1 - SIM(x, y)$

3. Cosine Distance:
   向量夹角
   $d(x, y) = arc cos\{ \frac{x \cdot y}{\|x\|_2 + \|y\|_2} \}$

4. Edit Distance:
   1. 直接计算
   Given $x = x_1 x_2 ...$, $y = y_1 y_2 ...$.
   $d(x, y)$ is the smallest number of insertion and deletion of single char($x \to y$).
   有[动态规划算法](http://blog.csdn.net/xanxus46/article/details/38678335)，

   \begin{equation}
       \operatorname{lev}_{a,b}(i,j) = \begin{cases}
          \max(i,j) & \text{ if} \min(i,j)=0, \\
          \min \begin{cases}
                  \operatorname{lev}_{a,b}(i-1,j) + 1 \\
                  \operatorname{lev}_{a,b}(i,j-1) + 1 \\
                  \operatorname{lev}_{a,b}(i-1,j-1) + 1_{(a_i \neq b_j)}
                \end{cases} &\text{ otherwise.}
          \end{cases}
    \end{equation}

    具体内容见维基[Levenshtein distance](https://en.wikipedia.org/wiki/Levenshtein_distance)。

   2. 间接计算
   $d(x, y) = len(x) + len(y) - 2 LCS(x, y)$
   LCS is the longest common subsequence of $x$ and $y$.

5. Hamming Distance: $d(x, y) = sum( \, diff(x, y) \,)$

### 3.6 The Theory of Locality-Sensitive Function

对于每种距离定义，需要找到合适的哈希函数使其满足 $P-s$ 是 "S" 型曲线（越相似，概率越高）。

a family of functions $\{h_i(x)\}$ should statify 3 conditions:

1) $P[\text{close pairs become candidate pairs}] > P[\text{dissimilar pairs...}]$
   补充: "close" - distance measure, "P" 是概率。 这里就说明 "Prob(x, y) - d(x, y)" 应该是 ”S" 型曲线。
   eg: Probility using minhash to Jaccard distance is "S" curve.

```python
s = np.linspace(0, 1, 100)
plt.plot(1-s, p(s, r, b))
plt.xlabel('Jaccard distance')
plt.ylabel('Probility')
```

2) $h_i(x)$ should be statistically independent, because we need estimate the possbility by using the product rule for independent events.
   eg: $s^r$ in $1 - (1 - s^r)^b$ for Jaccard similarity.

3) efficient in two ways:
+ identify candidate pairs in time much less than the time it takes to look at all pairs.
+ They must be combinable to build functions that are better at avoiding false positives and negatives, and the combined functions must also take time that is much less than the number of pairs.

当我们使用 “band, then hash" 这种方法来组合哈希函数达到概率 $1 - (1 - s^r)^b$ 变成 "S" 曲线（其实是倒”S"型，如上图），
所以我们需要找到的哈希方法，它应该有将距离接近的候选对更高概率匹配的特性，即 $P-d(x,y)$ 是倒 ”S"。这就引出了接下的内容：local sensitive.

**$(d_1, d_2, p_1, p_2)$-sensitive**：
1. if $d(x, y) \leq d_1$, then $P[f(x) = f(y)] \geq p_1$.
2. if $d(x, y) \geq d_2$, then $P[f(x) = f(y)] \leq p_2$.

eg: For Jaccard distance, the family of minhash functinos is a $(d_1, d_2, 1-d_1, 1-d_2)$-sensitive family for any $d_1$ and $d_2$ where $0 \leq d_1 < d_2 \leq 1$.

```python
#minhash for Jaccard distance
d = 1 - s
d_1, d_2 = 0.4, 0.6

plt.plot(d[d<d_1], p(s, r, b)[d<d_1], d[d>d_2], p(s, r, b)[d>d_2])
plt.xlabel('d(x, y)')
plt.ylabel('Probility of being a candidate')
```

**Amplifing a Locality-Sensitive Family**
Given a $(d_1, d_2, p_1, p_2)$-sensitive family $F = {f_1, ..., f_r}$, we can construct a new family $F'$:
1. **AND**-construction:
    $f(x) = f(y)$ when **all** $f_i(x) = f_i(y)$
    $$(d_1, d_2, p_1^r, p_2^r)$$

2. **OR**-construction:
    $f(x) = f(y)$ when **any** $f_i(x) = f_i(y)$
    $$(d_1, d_2, 1-(1-p_1)^r, 1-(1-p_2)^r)$$

```python
#AND-construciton and OR-constuction

dis = d
pos = p(s, r, b)

r = 4
AND_pos = (pos**r)
OR_pos = 1 - (1 - pos)**r

plt.plot(dis, pos, label='origin')
plt.plot(dis, AND_pos, label='AND-cons')
plt.plot(dis, OR_cons, label='OR-cons')
plt.legend(loc='upper right')
plt.xlabel('d(x, y)')
plt.ylabel('P[f(x) = f(y)]')
```

As shown in the figure above,
+ AND-construction: low probability close to 0, namely, false negative$\uparrow$, false postive$\downarrow$.

+ OR-construction: high probabiity close to 1, namely, false negative$\downarrow$, false postive$\uparrow$.

用 AND- 和 OR- 可以组合成需要的 $(d_1, d_2, p_1, p_2)$。

**LSH families for Distance measures**
1. For **Jaccard distance**,
   the family of **minhash functinos** is a $(d_1, d_2, 1-d_1, 1-d_2)$-sensitive family for any $d_1$ and $d_2$ where $0 \leq d_1 < d_2 \leq 1$.

2. For **Hamming Distance**,
   given d-dim vector $x$ adn $y$, their hamming distance is $h(x, y)$.
   define:  $f_i(x)$ to be the $i_{th}$ bit of $x$,
   then for a randomly chosen $i$, $P[f_i(x) = f_i(y)] = 1 - h(x,y)/d$.

   $F={f_i(x)}$ is $(d_1, d_2, 1 - d_1/d, 1 - d_2/d)$-sensitive for any $d_1 < d_2$
   + $h(x, y) \in [0, d]$
   + $max \, |F| = d$,
     函数个数有限，若 $d$ 特别小，会限制了使用 AND- 和 OR- 组合的可能性。

4. For **Euclidean Distance**,
   $f$ is associated with a randomly chosen line which is divided into segments of length $a$, and the segments of the line are the buckets into which function $f$ hashed points. as shown in Fig (3.13).
   $(a/2, 2a, 1/2, 1/3)$

3. For **Cosine Distance**,
   $x$ 和 $y$ 的夹角是 $h(x,y$.
   任选一超平面 $S_i$，取其法向量 $v_i$，
   则当 $S_i$ 位于夹角内时，$v_i \cdot x$ 和 $v_i \cdot y$ 符号不同；夹角外时，符号相同。具体见书中图 3.12。

   定义:
   $f(x) = f(y)$ 有且仅有 $v_f \cdot x$ 和 $v_f \cdot y$ 符号相同。

   $(d_1, d_2, (180-d_1)/180, (180-d_2)/180)$

   + 感觉这个函数计算量，未必比直接计算余弦距离小
     **sketches**:
     有随机向量集合 $V = \{v_1, ..., v_n\}$，其中 $\forall \, v_i \in \{+1, -1\}^d$
     则 sketches of $x$ is

     \begin{align}
         sx & = [sx_1, ..., sx_n] \\
            & = [I(v_1 \cdot x), ..., I(v_n \cdot x)]
     \end{align}

     其中

     \begin{equation}
         I(x) = \begin{cases}
                     +1, & \quad x > 0 \\
                     -1, & \quad x < 0 \\
                     +1, \, \text{or} \, -1 & \quad x = 0
                \end{cases}
     \end{equation}

     比较两个向量 $x$ 和 $y$ 的相似性，可转化为 $$P[f(x) = f(y)] = \frac{len(\text{intersection}(sx, sy))}{len(sx)}$$

```python
#example 3.21
x = np.arange(3, 7)
y = np.arange(4, 0, -1)
print('x:{}, y:{}'.format(x,y))

from scipy.spatial.distance import cosine

print('angle between x and y: {:.3f}\n'.format(np.rad2deg(np.arccos(1-cosine(x,y)))))

def sketch_calc(x, v):
    sketch = np.dot(x, v)
    sketch[sketch>0] = 1
    sketch[sketch<=0] = -1
    return sketch

v = [[1, -1, 1, 1], [-1, 1, -1, 1], [1, 1, -1, -1]]
v = np.array(v).T
print('v:{}'.format(v))

x_s = sketch_calc(x, v)
print('sketch of x: {}'.format(x_s))
y_s = sketch_calc(y, v)
print('sketch of y: {}'.format(y_s))
print('p = same elemets / total elements: {} / {}'. format(sum((x_s-y_s)==0), len(x_s)))

def sketch_angle_calc(x_s, y_s):
    """calculate angle when similar possibily of two vectors are determined.

       p = (180 - d) / 180
    """
    p = sum((x_s-y_s)==0) / len(x_s)
    d = 180 - 180 * p
    return d

print('angle(sketch): {}\n'.format(sketch_angle_calc(x_s, y_s)))

v = list(itertools.product([-1,1],repeat=len(x)))
v = np.array(v).T
print('v:{}'.format(v))

x_s = sketch_calc(x, v)
print('sketch of x: {}'.format(x_s))
y_s = sketch_calc(y, v)
print('sketch of y: {}'.format(y_s))
print('p = same elemets / total elements: {} / {}'. format(sum((x_s-y_s)==0), len(x_s)))
print('angle(sketch): {}'.format(sketch_angle_calc(x_s, y_s)))
```

    x:[3 4 5 6], y:[4 3 2 1]
    angle between x and y: 38.048

    v:[[ 1 -1  1]
     [-1  1  1]
     [ 1 -1 -1]
     [ 1  1 -1]]
    sketch of x: [ 1  1 -1]
    sketch of y: [ 1 -1  1]
    p = same elemets / total elements: 1 / 3
    angle(sketch): 120.0

    v:[[-1 -1 -1 -1 -1 -1 -1 -1  1  1  1  1  1  1  1  1]
     [-1 -1 -1 -1  1  1  1  1 -1 -1 -1 -1  1  1  1  1]
     [-1 -1  1  1 -1 -1  1  1 -1 -1  1  1 -1 -1  1  1]
     [-1  1 -1  1 -1  1 -1  1 -1  1 -1  1 -1  1 -1  1]]
    sketch of x: [-1 -1 -1  1 -1  1 -1  1 -1 -1 -1  1 -1  1  1  1]
    sketch of y: [-1 -1 -1 -1 -1 -1 -1  1 -1 -1  1  1  1  1  1  1]
    p = same elemets / total elements: 12 / 16
    angle(sketch): 45.0

### 3.8 Applications of Locality-Sensitive Hashing
##### 1. Entity Resolution
to match data records that refer to the same real-world entity.
找到同一对象在不同数据库里的数据

eg: A 和 B 公司对账（记录方式不尽相同）
1. 相似度比较：
   使用"姓名“，”电话“，”地址信息“，三个分别用编辑距离计算，评分（d=0-1, score=100-0)，累积总值就是相似度评分 $s$。
   + certain publicly available tables were used to reduce the penalty in appropriate situations, like "Bill" and "William" are treated as 1 letter.

2. 全部比较太耗时，所以用 LSH 先找出潜在配对。
   对三组分别哈希，用 AND- 构建。
   实际中的方法是直接排序：名字，电话，地址均按升序排列，相似的就聚焦在一起。

3. 评价匹配正确率 $f$。
   使用其它未使用的记录来做为评据。
   如这里使用两个数据库配对项目的创建时间差 $t$ 做为度量：
   1. $\bar{t} = E[\text{random_pick}(t)]$: 随机配对的时间差期望
   2. $t_p = E[t_{xy}: argmax_{x,y} s(x,y)]$: 最大 $s$ 数据集的时间差期望（假设全是正确匹配，$f=1$）。
   3. 假设现在用 $s_i$ 做阈值，得到的匹配对算出均值 $t_i$，其正确匹配率是 $f_i$，
      则有 $t_i = t_p * f_i + \bar{t} * (1 - f_i)$
      解得：
      $$f_i = \frac{\bar{t} - t_i}{\bar{t} - t_p}$$

      即，我们可以用结果中的其他数据项，组成评价匹配正确率的函数。

##### 2. Matching Fingerprints
特征描述： minutiae 构建指纹的特征向量。

两类问题：
    1. 一对多：找出某个指纹的匹配
    2. 多对多：找出潜在相同的匹配

假设随机抽取一个指纹的特征向量的一位，其中有 minutiae 的概率是 $P[x] 20\%$。
   同一来源的两个指纹在特征向量的一位，一个有 minutiae 时，另一个也有的概率是 $P[Y|X] = 80\%$。
定义**哈希函数** $f$ 为：
   抽取特征向量的三位，若两个指纹的每一位均含有 minutiae，则做为候选对。

分析：随机对候选的概率是 $0.2^6 = 0.000064$, 同一来源候选的概率是 $0.2^3 \times 0.8^3 = 0.004096$，

我们生成 1024 个 $f_i$ 哈希函数，使用 OR- 构建，再用 AND- 构建，结果见下图。

```python
#Example 3.22
p = np.array([0.2**6, (0.2**3)*(0.8**3)])
print('dissimilar:{:.4f}, similar:{:.4f}'.format(p[0], p[1]))

#1024 OR-cons
p = 1 - (1 - p)**1024
print('dissimilar:{:.4f}, similar:{:.4f}'.format(p[0], p[1]))

#then 2 AND-cons
p = p**2
print('dissimilar:{:.4f}, similar:{:.4f}'.format(p[0], p[1]))

print 1/p[0]
```

    dissimilar:0.0001, similar:0.0041
    dissimilar:0.0634, similar:0.9850
    dissimilar:0.0040, similar:0.9703
    248.496189026

##### 3. Matching Newspaper Articles
difference of text in prose and in ads/headlines:
Prose has a much greater frequency of stop words  ====> define a _shingle_ to be a stop word followd by the next two words.

### 3.9 Methods for High Degrees of Similarity

**LSH-based methods** appear most **effective** when the degree of similarity we accept is **relatively low**.

##### 1. Finding Identical Items
methods:
1. hash the first few char and compare.
   + cons: some doc has the same header, such as html.
2. hash the whole document.
   + cons: examine every char of every doc.
3. pick some fixed random pos, then hash and compare.
   + prob: exclude the docs whose length are different.

##### 2. Finding highly similar items
all pairs in the sets have a high Jaccard similarity, say at least 0.9.

**representing** sets as strings:
represent a set by **sorting** the elements of the universal set in some **fixed order** and constructing a string of "characters".
+ No char appears > 1 time
+ if 2 char in 2 different strings, they are in the same order in both string.
+ For documents, the element is shingle.

eg: 26 lower-case letters, normal alphabetical order: {d, a, b, b} => 'abd'

1. Length-Based Filtering: sort the strings by length
    each string $s$ is compared with those strings $t$ that follow $s$ in the list ($L_s \leq L_t: L_x = len(x)$).
    $J_u$ is the upper bound on Jaccard distance between two strings.

    SIM(s, t) is at most $L_s/L_t$, and lower bound = $1 - L_s/L_t < J_u$ = upper bound.
    hence, $$L_t \leq \frac{L_s}{1 - J_u}$$.

2. Prefix Indexing: create an index for the firs $p$-symbols of $s$.
    $p$: make sure that at least 1 shared symbol in two strings when $SIM(s, t) \geq J$

    从相反情况讨论，假设 $SIM(s, t) \geq J$ 但前 $p$ 字符无一相同，则此时 $max \, SIM(s, t) = (L_s - p) / L_s$。那么只要 $J > (L_s - p) / L_s$ 则能保证前 $p$ 字符至少有一个相同。有：
    $$p = \lfloor (1 - J)L_s \rfloor + 1$$

    1. Using Position Information: 使用位置信息来判定是否需要比较
       索引是 $(x, j)$
       若 $s[1:i-1] \cap t[1:i-1] = \emptyset$ 并且 $s[i:-1] = t[j:-1]$
       则其  $$SIM(s, t) = \frac{L_s-(i-1)}{L_s+(j-1)} \geq J$$
       可解得 $$j \leq (L_s(1 - J) - i + 1 + J)/J$$

    2. Using Position and Length in Indexs:
       索引是 (x, j, suffix length).
       若有 (s_i, i, p) 和 (t_j, j, q)，前面部分无一相同，后面部分两者是子集关系。
       则分情况讨论：
       + 若 $p \geq q$，则  $$\frac{L_s - (i-1) - (p-q)}{L_s + j - 1} = \frac{q + 1}{L_s + j - 1} \geq J$$
       + 若 $p < q$, 则  $$\frac{L_s - (i-1)}{L_s + (j-1) + (q-p)} = \frac{L_s - i + 1}{i + j - 1  + q} \geq J$$

```python

```


# GIAO 的 RHF 磁化率数值导数计算方式简述

> 创建时间：2020-08-30

在这篇文档中，我们会讨论使用 PySCF 以及其作为 libcint 的接口，计算 GIAO 的 RHF 数值磁化率的程序。该文档大量参考 PySCF 的代码 [magnetizability/rhf.py](https://github.com/pyscf/pyscf/blob/master/pyscf/prop/magnetizability/rhf.py) 与 [nmr/rhf.py](https://github.com/pyscf/pyscf/blob/master/pyscf/prop/nmr/rhf.py)。一篇公式记号比较清晰的文章是 Laasner, Blum, et al. [^Laasner-Blum.arXiv.2018]。

与上一篇文档一样，我们的讨论中所使用到的分子体系 `mol` 会是非对称的氨分子，并且取用最小基组。其 RHF 计算放在实例 `mf`，而磁化率计算实例会放在 `mf_mag`。

在这篇文档中，我们仍然需要保留规范原点 (Gauge Origin) 的概念。规范原点会取在坐标原点上 `coord_orig`。

```python
from pyscf import gto, scf, dft
from pyscf.prop import nmr, magnetizability
import numpy as np

np.set_printoptions(precision=5, linewidth=150, suppress=True)
np.random.seed(0)
```

```python
mol = gto.Mole()
mol.atom = """
N  0.  0.  0.
H  0.  1.  0.2
H  0.1 0.3 1.5
H  0.9 0.4 -.2
"""
mol.basis = "STO-3G"
mol.verbose = 0
mol.build()
coord_orig = np.zeros(3)
```

```python
nocc, nao, nmo = mol.nelec[0], mol.nao, mol.nao
```

其自洽场能量为

```python
mf = scf.RHF(mol).run()
mf.e_tot
```

    -55.253540514686556

我们定义磁化率计算类为 `mf_mag`。其磁化率张量 $\xi_{ts}$ 为：

```python
mf_mag = magnetizability.RHF(mf)
mf_mag.kernel()
```

    array([[-3.72497, -0.02158, -0.07514],
           [-0.02158, -2.88877,  0.20345],
           [-0.07514,  0.20345, -3.57485]])

但留意到，与上一份文档不同地，如果我们对分子作平移操作 (譬如下述平移操作是将原点移动 $(x, y, z) = (10, -10, 5) \, \mathsf{a.u.}$)，其磁化率仍然保持不变：

```python
mol_trans = mol.copy()
mol_trans.set_geom_(mol.atom_coords() + np.array([[10, -10, 5]]), unit="AU")
mol_trans.build()
```

    <pyscf.gto.mole.Mole at 0x7fa032502d00>

```python
magnetizability.RHF(scf.RHF(mol_trans).run()).kernel()
```

    array([[-3.72498, -0.0216 , -0.07515],
           [-0.0216 , -2.8888 ,  0.20345],
           [-0.07515,  0.20345, -3.57482]])

## GIAO 规范不变原子轨道

### 基本概念

我们从上一篇文档中，得知对分子作平移操作后，磁化率可能会改变。

GIAO 规范不变原子轨道 (Gauge Invariant Atomic Orbital) 是一种解决方案，它可以让分子在平移操作后，保证磁化率不变。事实上，分子旋转操作下磁化率也不变，但这会额外引入三维旋转矩阵，为了方便我们就不讨论分子旋转的情形。

GIAO 的基本思路是对于外磁场引入的微扰，其波函数更换为

$$
\phi^\mathrm{GIAO}_\mu (\boldsymbol{r}) = e^{- \frac{i}{2} (\boldsymbol{R}_\mu \times \boldsymbol{r}) \cdot \boldsymbol{\mathscr{B}}} \phi_\mu (\boldsymbol{r})
$$

其中，$\phi_\mu (\boldsymbol{r})$ 是普通的原子轨道基组，$\boldsymbol{R}_\mu$ 是原子轨道 $\mu$ 作为 Gaussian 基组的中心坐标相对于规范原点 (Gauge Origin) 向量，$\boldsymbol{r}$ 是电子坐标，$\boldsymbol{\mathscr{B}}$ 是外加微扰磁场。根据其定义，我们知道，GIAO 方法必须要使用原子轨道基组下使用。需要注意，这是一个复函数轨道。

使用该转换后的原子轨道就可以达到平移操作后磁化率不变的结果。关于这一点的证明，参考 Pople [^Pople-Pople.MP.1958] eq 2.5 附近的讨论。

事实上，规范 (原点) **不变** 原子轨道的称呼可能是不恰当的。Pople [^Pople-Pople.DFS.1962.34] 曾在文章里使用了规范 (原点) **依赖** 原子轨道 (Gauge Dependent Atomic Orbital)；这是因为 $\phi^\mathrm{GIAO}_\mu (\boldsymbol{r})$ 实际上是依规范原点位置不同而不同的 (注意 $\boldsymbol{R}_\mu$ 的定义)。当然，我们仍然沿用 GIAO 的约定俗成。

### 实例分析：GIAO 重叠矩阵在外磁场下的微扰

**概念与程序准备**

我们拿一个具体的例子来说明 GIAO 的计算；最简单的例子是重叠矩阵：

$$
S_{\mu \nu}^\mathrm{GIAO} = \langle \mu | e^{- \frac{i}{2} (\boldsymbol{R}_{\mu \nu} \times \boldsymbol{r}) \cdot \boldsymbol{\mathscr{B}}} | \nu \rangle
$$

其中，$\boldsymbol{R}_{\mu \nu} = \boldsymbol{R}_\nu - \boldsymbol{R}_\mu$，它表示原子轨道 $\mu$ 与 $\nu$ 作为 Gaussian 基组的中心坐标的向量之差。之所以会出现 $- \boldsymbol{R}_\mu$ 的负号，是因为 $\phi^\mathrm{GIAO}_\mu (\boldsymbol{r})$ 出现在左矢时，其虚数使得在作复共轭时，指数项应当乘以负号。

我们在实际计算中，只关心其一阶算符的矩阵形式：

$$
\nabla_{\boldsymbol{\mathscr{B}}} S_{\mu \nu}^\mathrm{GIAO} = \langle \mu | - \frac{i}{2} \boldsymbol{R}_{\mu \nu} \times \boldsymbol{r} | \nu \rangle
$$

其分量形式不太容易写出，但我们下面会用程序来具体地求取其分量。

我们先使用格点积分来计算矩阵。借助 PySCF 的 DFT 格点，我们写出下述代码：

- `ni` 格点积分引擎

- `grids` (50, 194) 大小的格点

- `ao` $\phi_\mu (\boldsymbol{r}_g)$ 即处于格点 $\boldsymbol{r}_g$ 上 $\phi_\mu$ 基轨道的函数值，维度表示为 $(g, \mu)$

```python
ni = dft.numint.NumInt()

grids = dft.Grids(mol)
grids.atom_grid = (50, 194)
grids.build()

ao = ni.eval_ao(mol, grids.coords)
ao.shape
```

    (26896, 8)

**数值格点积分**

我们预先已经知道，第 $\mu = 3$ 根基轨道是第 0 个原子 (N 原子)，第 $\nu = 6$ 跟轨道是第 2 个原子 (H 原子) (按照 0-index 计数)，那么向量 $\boldsymbol{R}_{\mu \nu} = \boldsymbol{R}_{36} = \boldsymbol{R}_{6} - \boldsymbol{R}_{3}$：

```python
R_36 = (mol.atom_coord(2) - coord_orig) - (mol.atom_coord(0) - coord_orig)
R_36
```

    array([0.18897, 0.56692, 2.83459])

上述代码中，看起来在 `coord_orig` 上有多余的代码；这是因为 $\boldsymbol{R}_\mu$ 本身应当是原子核坐标相对于规范原点的距离；这里我们的规范原点恰好是原点 `coord_orig`。

下面我们就求取积分值 $\nabla_{\boldsymbol{\mathscr{B}}} S_{36}^\mathrm{GIAO}$：

$$
\nabla_{\boldsymbol{\mathscr{B}}} S_{36}^\mathrm{GIAO} = \int \phi_3 (\boldsymbol{r}) \left( - \frac{i}{2} \boldsymbol{R}_{36} \times \boldsymbol{r} \right) \phi_6 (\boldsymbol{r}) \, \mathrm{d} \boldsymbol{r} \simeq
- \frac{i}{2} \sum_g w_g \phi_3 (\boldsymbol{r}_g) \phi_6 (\boldsymbol{r}_g) \cdot \boldsymbol{R}_{36} \times \boldsymbol{r}_g
$$

```python
- 0.5j * np.einsum("g, g, g, gr -> r", grids.weights, ao[:, 3], ao[:, 6], np.cross(R_36, grids.coords))
```

    array([0.+0.22657j, 0.-0.j     , 0.-0.0151j ])

上述结果是一个纯虚数向量，其三个维度分别相当于 $(\mathscr{B}_x, \mathscr{B}_y, \mathscr{B}_z)$ 的维度。

最后，我们指出，在 PySCF/libcint 中，上述积分的虚部负值可以通过输入 `int1e_igovlp` 字符串实现。其意义可以参考 [auto_intor.cl](https://github.com/sunqm/libcint/blob/abf6948fa17e5b4ecbd26de05bf4b1d7b2b2fe3c/scripts/auto_intor.cl#L17) 与 [README](https://github.com/sunqm/libcint/blob/master/README)。留意在 libcint 中，这类积分被表示为 $i \langle \boldsymbol{U}_\mathrm{g} \mu | \nu \rangle$，其 $\boldsymbol{U}_\mathrm{g} = (U_\mathrm{g}^x, U_\mathrm{g}^y, U_\mathrm{g}^z)$ 的意义相当于三维向量算符

$$
\boldsymbol{U}_\mathrm{g} = - \frac{i}{2} \boldsymbol{R}_{\mu \nu} \times \boldsymbol{r}
$$

```python
mol.intor("int1e_igovlp")[:, 3, 6]
```

    array([-0.22658, -0.     ,  0.01511])

PySCF 导出是实 (反对称) 矩阵 $i \langle \boldsymbol{U}_\mathrm{g} \mu | \nu \rangle$；因此在实际使用该积分时，我们就需要乘以 $- i$，让该矩阵变为复厄米矩阵，成为真正的 GIAO 变换算符一阶的矩阵形式 $\langle \boldsymbol{U}_\mathrm{g} \mu | \nu \rangle$。

```python
- 1j * mol.intor("int1e_igovlp")[:, 3, 6]
```

    array([0.+0.22658j, 0.+0.j     , 0.-0.01511j])

后文我们也会用 $\boldsymbol{U}_\mathrm{g}$ 来简化一阶导数下的 GIAO 变换算符。

当然，我们需要知道这并不是一个很严格的写法，因为 $\boldsymbol{U}_\mathrm{g}$ 的等式右边与 $\mu, \nu$ 有关；因此，使用这种简写时一定需要留意哪些轨道与该 GIAO 变换算符一阶导数有关。拿双电子积分 $( U_\mathrm{g}^t \mu \lambda | \kappa \nu )$ 来说，由于 $U_\mathrm{g}^t$ 作用在 $\mu$ 上，并且 $\lambda$ 与 $\mu$ 在积分过程中使用相同的电子坐标，因此 $U_\mathrm{g}^t$ 的作用对象就是 $\mu$ 与 $\lambda$。

**从偶极积分出发给出微扰的 GIAO 重叠矩阵**

实际上，

$$
\begin{align}
\nabla_{\boldsymbol{\mathscr{B}}} S_{\mu \nu}^\mathrm{GIAO}
&= \langle \mu | - \frac{i}{2} \boldsymbol{R}_{\mu \nu} \times \boldsymbol{r} | \nu \rangle \\
&= - \frac{i}{2} \boldsymbol{R}_{\mu \nu} \times \langle \mu | \boldsymbol{r} | \nu \rangle
\end{align}
$$

我们仍然拿 $(\mu, \nu) = (3, 6)$ 的情形讨论。我们应当注意到，$\langle \mu | \boldsymbol{r} | \nu \rangle$ 与偶极积分形式几乎一致 (依照不同的定义方式，偶极积分可能是该值或其相反数)，可以用字符串表示为 `int1e_r`。因此，$\nabla_{\boldsymbol{\mathscr{B}}} S_{36}^\mathrm{GIAO}$ 还可以程序化为

```python
- 0.5j * np.cross(R_36, mol.intor("int1e_r")[:, 3, 6])
```

    array([0.+0.22658j, 0.-0.j     , 0.-0.01511j])

**微扰的 GIAO 重叠矩阵并非“规范不变”**

我们知道“规范不变”的意义是，作为最终结果的磁化率在任意的规范原点下值相同。但这并不意味着中间过程的矩阵或数值结果也相同。重叠矩阵就不满足这种“规范不变”性质：

```python
- 1j * mol.intor("int1e_igovlp")[:, 3, 6]
```

    array([0.+0.22658j, 0.+0.j     , 0.-0.01511j])

```python
- 1j * mol_trans.intor("int1e_igovlp")[:, 3, 6]
```

    array([0.-0.52235j, 0.-0.65814j, 0.+0.16645j])

PySCF 所使用的默认规范原点就是坐标系原点，因此我们这里都没有严格地使用 `with mol.with_common_orig(coord_orig)` 语句。当然，由于 GIAO 下的磁化率应当不受规范原点的选取变化而受到影响，因此仅从最终结果上来说，也不需要刻意地规定规范原点。

## Core Hamiltonian 的程序实现

### 一阶 Hamiltonian Core

前一篇文档中，所有的一阶微扰是

$$
\hat h {}^{(1)} (\boldsymbol{\mathscr{B}}) = \frac{1}{2} \boldsymbol{\mathscr{B}} \cdot \boldsymbol{r} \times \boldsymbol{\hat{p}}
$$

但当考虑到 GIAO，一阶微扰算符则应当写作

$$
\hat h {}^{(1)} (\boldsymbol{\mathscr{B}}, \boldsymbol{U}_\mathrm{g}) | \nu \rangle = \frac{1}{2} \boldsymbol{\mathscr{B}} \cdot (\boldsymbol{r} - \boldsymbol{R}_\nu) \times \boldsymbol{\hat{p}} + \boldsymbol{\mathscr{B}} \cdot \boldsymbol{U}_\mathrm{g} \hat f {}^{(0)} | \nu \rangle
$$

需要注意到，这里使用到了零阶 Fock 算符 $\hat f {}^{(0)}$，该算符需要代入未受外场微扰的密度矩阵才能获得，而不是单纯地由分子构型与基组就能决定的。该项贡献的来源是 $\langle \Psi^{(0), \mathrm{GIAO}} | \hat H^{(0)} | \Psi^{(0), \mathrm{GIAO}} \rangle$。

因此，顺磁项对应的积分可以公式表达为

$$
\begin{align}
h_{t \mu \nu}^{(1)}
&=
\frac{1}{2} \langle \mu | \hat l_t | \nu \rangle_{\mathrm{Gauge} \rightarrow \boldsymbol{R}_\nu}
\\ & \quad
+ \langle U_\mathrm{g}^t \mu | \hat t | \nu \rangle
+ \langle U_\mathrm{g}^t \mu | \hat v_\mathrm{nuc} | \nu \rangle
\\ & \quad
+ \sum_{\kappa \lambda} ( U_\mathrm{g}^t \mu \nu | \kappa \lambda ) D_{\kappa \lambda}^{(0)}
- \frac{1}{2} \sum_{\kappa \lambda} ( U_\mathrm{g}^t \mu \lambda | \kappa \nu ) D_{\kappa \lambda}^{(0)}
- \frac{1}{2} \sum_{\kappa \lambda} ( U_\mathrm{g}^t \kappa \nu | \mu \lambda ) D_{\kappa \lambda}^{(0)}
\end{align}
$$

其中，积分字符与公式表达之间的关系为

- `int1e_cg_irxp` $i \langle \mu | \hat l_t | \nu \rangle_{\mathrm{Gauge} \rightarrow \boldsymbol{R}_\nu}$

- `int1e_igkin` $i \langle \boldsymbol{U}_\mathrm{g} \mu | \hat t | \nu \rangle$

- `int1e_ignuc` $i \langle \boldsymbol{U}_\mathrm{g} \mu | \hat v_\mathrm{nuc} | \nu \rangle$

- `int2e_ig1` $i ( \boldsymbol{U}_\mathrm{g} \mu \nu | \kappa \lambda )$

需要留意 $i$，因为这会使得上式中很多项的正负号在编写程序时是相反的。

因此，一阶 Core Hamiltonian `hcore_1` $h_{t \mu \nu}^{(1)}$ 的表达式可以用下述程序表示：

```python
dm_guess = mf.make_rdm1()
hcore_1 = 1j * (
    - 0.5 * mol.intor("int1e_giao_irjxp")
    - mol.intor("int1e_igkin")
    - mol.intor("int1e_ignuc")
    - np.einsum("tuvkl, kl -> tuv", mol.intor("int2e_ig1"), dm_guess)
    + 0.5 * np.einsum("tulkv, kl -> tuv", mol.intor("int2e_ig1"), dm_guess)
    + 0.5 * np.einsum("tkvul, kl -> tuv", mol.intor("int2e_ig1"), dm_guess)
)
```

PySCF 中有对应的函数，生成 GIAO 下的一阶 Core Hamiltonian 矩阵 (事实上上述程序块就是从下述函数中获得的)：

```python
np.allclose(
    hcore_1,
    1j * nmr.rhf.make_h10(mol, dm_guess)
)
```

    True

最后，我们指出，关于库伦积分，实际上应当表示为

$$
h_{t \mu \nu}^{(1)} \leftarrow
- i \sum_{\kappa \lambda} ( U_\mathrm{g}^t \mu \nu | \kappa \lambda ) D_{\kappa \lambda}^{(0)}
- i \sum_{\kappa \lambda} ( \mu \nu | U_\mathrm{g}^t \kappa \lambda ) D_{\kappa \lambda}^{(0)}
$$

但由于反对称性质 $( \mu \nu | U_\mathrm{g}^t \kappa \lambda ) = - ( \mu \nu | U_\mathrm{g}^t \lambda \kappa )$ (反映到复数的情形其实是复共轭)，导致它与对称的零阶密度矩阵 $D_{\kappa \lambda}^{(0)}$ 相乘并求和后必为零值。

```python
np.allclose(mol.intor("int2e_ig1"), - mol.intor("int2e_ig1").swapaxes(-3, -4))
```

    True

在程序初步编写时，ERI 积分上很容易遇到角标顺序应该怎么写的问题。这是因为在实对称矩阵情形下，$\sum_{\kappa \lambda} (\mu \kappa | \nu \lambda) D_{\kappa \lambda}$ 与 $\sum_{\kappa \lambda} (\mu \lambda | \kappa \nu) D_{\kappa \lambda}$ 是完全相同的，但当 $\boldsymbol{U}_g$ 作用于 ERI 后，这种性质就不满足了。一个比较容易达成正确程序编写与公式推导的技巧是，保证 $\mu, \kappa$ 处在 ERI 积分的复共轭位上，而 $\nu, \lambda$ 处在普通位上 (或者用物理的记号来讲，当需要交换 $\langle \mu \kappa | \nu \lambda \rangle$ 的角标时，只能在竖线左或者竖线右相互交换，不能跨线)。

### 二阶 Hamiltonian Core

在前一篇文档中，所有的二阶微扰是

$$
\hat h {}^{(2)} (\boldsymbol{\mathscr{B}}) = \frac{1}{8} \big( \boldsymbol{\mathscr{B}}^2 \boldsymbol{r}^2 - (\boldsymbol{\mathscr{B}} \cdot \boldsymbol{r}) (\boldsymbol{\mathscr{B}} \cdot \boldsymbol{r})^\mathrm{T} \big)
$$

但当考虑到 GIAO，二阶微扰算符则应当写作

$$
\begin{align}
\hat h {}^{(2)} (\boldsymbol{\mathscr{B}}, \boldsymbol{U}_\mathrm{g}) | \nu \rangle
&=
\frac{1}{8} \big( \boldsymbol{\mathscr{B}}^2 (\boldsymbol{r} - \boldsymbol{R}_\nu)^2 - \boldsymbol{\mathscr{B}}^\mathrm{T} \big( (\boldsymbol{r} - \boldsymbol{R}_\nu) (\boldsymbol{r} - \boldsymbol{R}_\nu)^\mathrm{T} \big) \boldsymbol{\mathscr{B}} | \nu \rangle \\
& \quad
+ \frac{i}{4} \boldsymbol{\mathscr{B}}^\mathrm{T} \big( (\boldsymbol{r} - \boldsymbol{R}_\nu) \times \boldsymbol{\hat{p}} \big) \boldsymbol{U}_\mathrm{g}^\mathrm{T} \boldsymbol{\mathscr{B}} | \nu \rangle
+ \frac{i}{4}  \boldsymbol{\mathscr{B}}^\mathrm{T} \boldsymbol{U}_\mathrm{g} \big( (\boldsymbol{r} - \boldsymbol{R}_\nu) \times \boldsymbol{\hat{p}} \big)^\mathrm{T} \boldsymbol{\mathscr{B}} | \nu \rangle \\
& \quad
+ \frac{1}{2} \boldsymbol{\mathscr{B}}^\mathrm{T} \big( \boldsymbol{U}_\mathrm{g} \boldsymbol{U}_\mathrm{g}^\mathrm{T} \big) \boldsymbol{\mathscr{B}}  \hat f^{(0)} | \nu \rangle
\end{align}
$$

其中，$\mathbf{T}$ 是向量算符的转置，这种转置会生成矩阵形式的磁化率。

落实到程序中，则可以写为

$$
\begin{align}
h_{ts \mu \nu}^{(2)} \cdot \mathscr{B}_t \mathscr{B}_s
&= \frac{1}{8} \big( \delta_{ts} \langle \mu | x^2 + y^2 + z^2 | \nu \rangle_{\mathrm{Gauge} \rightarrow \boldsymbol{R}_\nu} - \langle \mu | ts | \nu \rangle_{\mathrm{Gauge} \rightarrow \boldsymbol{R}_\nu} \big) \\
&\quad + \frac{1}{4} \langle U_g^t \mu | \hat l_s | \nu \rangle_{\mathrm{Gauge} \rightarrow \boldsymbol{R}_\nu} + \frac{1}{4} \langle U_g^s \mu | \hat l_t | \nu \rangle_{\mathrm{Gauge} \rightarrow \boldsymbol{R}_\nu} \\
&\quad + \frac{1}{2} \langle U_g^t U_g^s \mu | \hat t | \nu \rangle + \frac{1}{2} \langle U_g^t U_g^s \mu | \hat v_\mathrm{nuc} | \nu \rangle \\
&\quad
+ \frac{1}{4} \sum_{\kappa \lambda} \big( (U_g^t U_g^s \mu \nu | \kappa \lambda) + (U_g^t \mu \nu | U_g^s \kappa \lambda) \big) D_{\kappa \lambda}^{(0)}
+ \frac{1}{4} \sum_{\kappa \lambda} \big( (U_g^t U_g^s \kappa \lambda | \mu \nu) + (U_g^t \kappa \lambda | U_g^s \mu \nu) \big) D_{\kappa \lambda}^{(0)} \\
&\quad
- \frac{1}{8} \sum_{\kappa \lambda} \big( (U_g^t U_g^s \mu \lambda | \kappa \nu) + (U_g^t \mu \lambda | U_g^s \kappa \nu) \big) D_{\kappa \lambda}^{(0)}
- \frac{1}{8} \sum_{\kappa \lambda} \big( (U_g^t U_g^s \kappa \nu | \mu \lambda) + (U_g^t \kappa \nu | U_g^s \mu \lambda) \big) D_{\kappa \lambda}^{(0)} \\
\end{align}
$$

其中，积分字符与公式表达之间的关系为

- `int1e_rr_origj` $\langle \mu | ts | \nu \rangle_{\mathrm{Gauge} \rightarrow \boldsymbol{R}_\nu}$

- `int1e_grjxp` $\langle U_g^t \mu | \hat l_s | \nu \rangle_{\mathrm{Gauge} \rightarrow \boldsymbol{R}_\nu}$

- `int1e_ggkin` $\langle U_g^t \mu | \hat t | \nu \rangle_{\mathrm{Gauge} \rightarrow \boldsymbol{R}_\nu}$

- `int1e_ggnuc` $\langle U_g^s \mu | \hat v_\mathrm{nuc} | \nu \rangle_{\mathrm{Gauge} \rightarrow \boldsymbol{R}_\nu}$

- `int2e_gg1` $(U_g^t U_g^s \mu \nu | \kappa \lambda)$

- `int2e_g1g2` $(U_g^t \mu \nu | U_g^s \kappa \lambda)$

```python
int1e_rr_origj = mol.intor("int1e_rr_origj").reshape(3, 3, nao, nao)
int1e_grjxp = mol.intor('int1e_grjxp').reshape(3, 3, nao, nao)
int1e_ggkin = mol.intor('int1e_ggkin').reshape(3, 3, nao, nao)
int1e_ggnuc = mol.intor('int1e_ggnuc').reshape(3, 3, nao, nao)
int2e_gg1   = mol.intor("int2e_gg1")  .reshape(3, 3, nao, nao, nao, nao)
int2e_g1g2  = mol.intor("int2e_g1g2") .reshape(3, 3, nao, nao, nao, nao)
int2e_gg    = int2e_gg1 + int2e_g1g2
```

因此，二阶 Core Hamiltonian `hcore_2` $h_{ts \mu \nu}^{(2)}$ 的表达式可以用下述程序表示：

```python
hcore_2 = (
    + 1/8 * (np.einsum("ts, uv -> tsuv", np.eye(3), int1e_rr_origj.diagonal(0, 0, 1).sum(-1)) - int1e_rr_origj)
    + 1/4 * mol.intor('int1e_grjxp').reshape(3, 3, nao, nao)
    + 1/4 * mol.intor('int1e_grjxp').reshape(3, 3, nao, nao).swapaxes(0, 1)
    + 1/2 * mol.intor('int1e_ggkin').reshape(3, 3, nao, nao)
    + 1/2 * mol.intor('int1e_ggnuc').reshape(3, 3, nao, nao)
    + 1/4 * np.einsum("tsuvkl, kl -> tsuv", int2e_gg, dm_guess)
    + 1/4 * np.einsum("tskluv, kl -> tsuv", int2e_gg, dm_guess)
    - 1/8 * np.einsum("tsulkv, kl -> tsuv", int2e_gg, dm_guess)
    - 1/8 * np.einsum("tskvul, kl -> tsuv", int2e_gg, dm_guess)
)
```

## 重叠矩阵的程序实现

在 GIAO 下，除了 Hamiltonian Core 会发生改变，重叠矩阵也同样会产生变化。

一阶重叠矩阵程序中表示为

$$
S_{t \mu \nu}^{(1)} = \langle U_g^t \mu | \nu \rangle
$$

二阶重叠矩阵程序中表示为

$$
S_{t \mu \nu}^{(2)} = \frac{1}{2} \langle U_g^t U_g^s \mu | \nu \rangle
$$

其中，积分字符与公式表达之间的关系为

- `int1e_igovlp` $i \langle U_g^t \mu | \nu \rangle$

- `int1e_ggovlp` $\langle U_g^t U_g^s \mu | \nu \rangle$

```python
ovlp_1 = - 1j * mol.intor("int1e_igovlp")
```

```python
ovlp_2 = 0.5 * mol.intor('int1e_ggovlp').reshape(3, 3, mol.nao, mol.nao)
```

## 数值导数求磁化率

与上一篇文档一样，我们已经获得了原子轨道形式的一阶、二阶 Core Hamiltonian、重叠积分。通过在 `scf.RHF` 的自洽场过程中重载 (override) 函数 `get_hcore` 与 `get_ovlp`，就可以得到受外磁场微扰的分子能量 $E_\mathrm{tot} (\mathscr{B}_x, \mathscr{B}_y, \mathscr{B}_z)$：

```python
dm_guess = mf.make_rdm1()

def hcore_mag_field(dev_xyz):
    mf = scf.RHF(mol)
    def get_hcore(mol_=mol):
        hcore_total  = np.asarray(scf.rhf.get_hcore(mol_), dtype=np.complex128)
        hcore_total += np.einsum("tuv, t -> uv", hcore_1, dev_xyz)
        hcore_total += np.einsum("tsuv, t, s -> uv", hcore_2, dev_xyz, dev_xyz)
        return hcore_total
    def get_ovlp(mol_):
        ovlp_total  = np.asarray(scf.rhf.get_ovlp(mol_), dtype=np.complex128)
        ovlp_total += np.einsum("tuv, t -> uv", ovlp_1, dev_xyz)
        ovlp_total += np.einsum("tsuv, t, s -> uv", ovlp_2, dev_xyz, dev_xyz)
        return ovlp_total
    mf.get_hcore = get_hcore
    mf.get_ovlp  = get_ovlp
    return mf.kernel(dm=dm_guess)
```

我们可以用与上一篇文档相同的代码实现二阶梯度：

$$
\xi_{ts} = - \frac{\partial^2 E_\mathrm{tot} (\boldsymbol{\mathscr{B}})}{\partial \mathscr{B}_t \partial \mathscr{B}_s}
$$

```python
eng_origin = hcore_mag_field((0, 0, 0))
interval = 1e-4
num_polar = np.zeros((3, 3))
for t in range(3):
    for s in range(3):
        if t != s:
            dev_xyzs = np.zeros((4, 3))
            dev_xyzs[0, t] = dev_xyzs[0, s] = dev_xyzs[1, t] = dev_xyzs[2, s] =  interval
            dev_xyzs[3, t] = dev_xyzs[3, s] = dev_xyzs[2, t] = dev_xyzs[1, s] = -interval
            num_polar[t, s] = (
                + hcore_mag_field(dev_xyzs[0])
                - hcore_mag_field(dev_xyzs[1])
                - hcore_mag_field(dev_xyzs[2])
                + hcore_mag_field(dev_xyzs[3])
            ) / (4 * interval**2)
        else:
            dev_xyzs = np.zeros((2, 3))
            dev_xyzs[0, t], dev_xyzs[1, t] = interval, -interval
            num_polar[t, t] = (
                + hcore_mag_field(dev_xyzs[0])
                + hcore_mag_field(dev_xyzs[1])
                - eng_origin * 2
            ) / (interval ** 2)
num_polar *= -1
```

```python
num_polar
```

    array([[-3.72496, -0.02158, -0.07515],
           [-0.02158, -2.88877,  0.20344],
           [-0.07514,  0.20345, -3.57484]])

我们最后与 PySCF 所给出的结果进行核验：

```python
mf_mag.kernel()
```

    array([[-3.72497, -0.02158, -0.07514],
           [-0.02158, -2.88877,  0.20345],
           [-0.07514,  0.20345, -3.57485]])

[^Laasner-Blum.arXiv.2018]: Laasner, R.; Huhn, W.; Colell, J.; Theis, T.; Yu, V.; Warren, W.; Blum, V. Molecular NMR Shieldings, J-Couplings, and Magnetizabilities from Numeric Atom-Centered Orbital Based Density-Functional Calculations. *arXiv*: [1805.12225v1](http://arxiv.org/abs/1805.12225v1).

[^Pople-Pople.MP.1958]: Pople, J. A. Molecular Orbital Theory of Aromatic Ring Currents. *Mol. Phys.* **1958**, *1* (2), 175–180. doi: [10.1080/00268975800100211](https://doi.org/10.1080/00268975800100211).

[^Pople-Pople.DFS.1962.34]: Pople, J. A. Nuclear Magnetic Resonance in Diamagnetic Materials. the Theory of Chemical Shifts. *Discuss. Faraday Soc.* **1962**, 34, 7. doi: [10.1039/df9623400007](https://doi.org/10.1039/df9623400007).


# The problem

The product development department at Dream-Inc is suggesting a software upgrade to a cleaning robot, claiming to decrease its failure rate. To quantify this claim, a test was conducted. Software
version (“treatment”), run-id (“id”), and outcome (“failure”) was recorded and provided to you. You are tasked to help senior management with guidance on an informed decision.

# Tasks

Specifically, you are asked to answer the following questions:
1. Is software version B decreasing the error rate. If yes, by how much?
2. Please quantify your confidence in statement (1)?
3. Each failed run costs Dream-Inc $2.32, the software update \$43k. Which software version should be used?

Please communicate your answers in a short presentation. Feel free to prepare back-up-slides for relevant technical details and interesting observations.

# Solution

First things first: the libraries we need.

```python
# Manipulation of dataframes
import pandas as pd

# Matrix/vector calculations
import numpy as np

# Normal distribution
import scipy
from scipy.stats import norm

# Charts
import matplotlib.pyplot as plt
%matplotlib inline
```

Then we check versions. This is useful whenever we need to consult documentation.

```python
print(f'Pandas version: {pd.__version__}')
print(f'Numpy version: {np.__version__}')
print(f'Scipy version: {scipy.__version__}')
```

    Pandas version: 1.1.5
    Numpy version: 1.19.5
    Scipy version: 1.4.1

Mount my Google Drive account so I can access the datasets hosted in `DATA_PATH`.

```python
from google.colab import drive
drive.mount('/content/drive')

DATA_PATH = 'drive/MyDrive/data'
```

    Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount("/content/drive", force_remount=True).

Load the data. Notice that, because I already know the datasets, I can instruct Pandas to cast the `treatment` feature as strings instead of floats, which is appropriate because this feature is indeed categorical, not numerical.

```python
labels = pd.read_csv(f'{DATA_PATH}/label.csv', sep='|', dtype={'treatment': str})
outcomes = pd.read_csv(f'{DATA_PATH}/outcome.csv', sep='|')
```

# Checking NA (AKA blanks)

First we implement an utility function to help us spot NA in the datasets:

```python
def check_na(df):
  for (column_name, data) in df.iteritems():
    n_blanks = len(data) - len(data.dropna())
    if n_blanks > 0:
      print(f'Column "{column_name}" has {n_blanks} NA')
```

Then we run it against both datasets:

```python
check_na(outcomes)
check_na(labels)
```

    Column "failure" has 3 NA

Now that we know where are those NA, we can safely drop them:

```python
outcomes = outcomes.dropna()
```

## Cast types

```python
print(labels.dtypes)
print(outcomes.dtypes)
```

    id           object
    treatment    object
    dtype: object
    id          object
    failure    float64
    dtype: object

```python
dataset = outcomes.merge(labels, on='id')
```

```python
dataset.head()
```

      id
      failure
      treatment

      0
      5d6c2b66-27b8-4c31-8670-c9e0cc1c9a3d
      0.0
      1.4

      1
      d628fd57-c4cd-4bfd-855e-59da588a3925
      0.0
      1.4

      2
      e185f40d-e88d-41bd-a2fd-2cd0fbaf7ce1
      0.0
      1.5

      3
      0351b627-06b4-451e-8bbc-842e318ec84b
      1.0
      1.4

      4
      1ab70e83-0947-4ef7-9a36-2367c0c68f5a
      0.0
      1.4

```python
id_counts = dataset.id.value_counts()
id_counts[id_counts > 1]
```

    aa9eed90-8ba9-4d14-aeae-24f23069c739    4
    Name: id, dtype: int64

```python
dataset.failure.value_counts()
```

     0.0      2201
     1.0        59
     100.0       1
    -1.0         1
    Name: failure, dtype: int64

```python
dataset.treatment.value_counts()
```

    1.4    1507
    1.5     755
    Name: treatment, dtype: int64

## Drop noise on `failure`

```python
from math import isclose

@np.vectorize
def not_noise(failure):
  return isclose(failure, 0) or isclose(failure, 1)

df = dataset[not_noise(dataset.failure)]
```

```python
df.failure.value_counts()
```

    0.0    2201
    1.0      59
    Name: failure, dtype: int64

## Convert `failure` to `success` (and boolean)

```python
@np.vectorize
def success(failure):
  return not bool(failure)

df = pd.DataFrame(data={
    'id': dataset.id,
    'success': success(dataset.failure),
    'version': dataset.treatment
})
```

```python
df.head()
```

      id
      success
      version

      0
      5d6c2b66-27b8-4c31-8670-c9e0cc1c9a3d
      True
      1.4

      1
      d628fd57-c4cd-4bfd-855e-59da588a3925
      True
      1.4

      2
      e185f40d-e88d-41bd-a2fd-2cd0fbaf7ce1
      True
      1.5

      3
      0351b627-06b4-451e-8bbc-842e318ec84b
      False
      1.4

      4
      1ab70e83-0947-4ef7-9a36-2367c0c68f5a
      True
      1.4

```python
df.dtypes
```

    id         object
    success      bool
    version    object
    dtype: object

## Assure independent events and samples

Remove duplicates of any cleaning robot (ID).

```python
df = df.drop_duplicates(subset='id')
```

```python
id_duplicates = not len(set(df.id)) == len(df.id)

print(f'There are duplicate ID in dataset: {id_duplicates}')
```

    There are duplicate ID in dataset: False

## Statistical test

The appropriate test for this problem is the two-proportions hypothesis test, which is implemented by the `proportions_ztest` module in the `statsmodels` package.

### Hypotheses

$
\begin{align}
H_0&: p_B - p_A = 0 \\
H_a&: p_B - p_A > 0
\end{align}
$

Without any reason to argue in favor of type-I or type-II errors, we **choose** the standard **relevance level** $\alpha = 5\%$, which implies **confidence level** of 95%.

```python
𝛼 = 0.05
```

Moreover, the **inequality** above implies we must use a single-tail test. In this case the critical statistics $z_\alpha$ is:

```python
z_alpha = norm.ppf(1-𝛼)
print(f'Critical value, z_alpha = {z_alpha}')
```

    Critical value, z_alpha = 1.6448536269514722

### Proportions

Now we need to calculate success proportion for each sample, as well as for both samples together (joint sample). We do this in Pandas by filtering the dataframe based on the feature `version`:

```python
version_A = df[df.version == '1.4']
version_B = df[df.version == '1.5']
```

```python
n_success_A = len(version_A[version_A.success]) # Amount of successes for 1.4
n_A = len(version_A) # Sample size for 1.4
p_A = n_success_A / n_A # Success rate (or proportion) for 1.4

n_success_B = len(version_B[version_B.success])
n_B = len(version_B)
p_B = n_success_B / n_B

# Joint successes, sample size and success proportion
n_success = n_success_A + n_success_B
n = n_A + n_B
p = n_success / n
```

```python
print(n_success_A)
print(n_A-n_success_A)
print(n_A)
print(n_success_B)
print(n_B-n_success_B)
print(n_B)
print(n_success)
print(n-n_success)
print(n)
```

    1461
    44
    1505
    737
    17
    754
    2198
    61
    2259

```python
print(f'p_A = {p_A}')
print(f'p_B = {p_B}')
print(f'p = {p}')

print(f'{1 - p_A}')
print(f'{1 - p_B}')
print(f'{p_B-p_A}')

```

    p_A = 0.9707641196013289
    p_B = 0.9774535809018567
    p = 0.9729969012837538
    0.029235880398671088
    0.022546419098143256
    0.0066894613005278325

```python
# This is a quite complex technique used just to simplify the
# calculation for the amount of decimal places to ensure only one significative
# figure in joint error for success proportion, below.
def compose(*functions):
  def composition(x):
    ans = x
    for f in reversed(functions):
      ans = f(ans)
    return ans

  return composition
```

```python
stdev_A = np.sqrt(p_A*(1-p_A))
stdev_B = np.sqrt(p_B*(1-p_B))

standard_error_A = stdev_A / np.sqrt(n_A)
standard_error_B = stdev_B / np.sqrt(n_B)
```

```python
# Joint standard error on joint success proportion
joint_standard_error = np.sqrt(stdev_A**2/n_A + stdev_B**2/n_B)

# Joint error is the standard joint error scaled by the choosed confidence level (two-tail)
joint_error = norm.ppf(0.975) * joint_standard_error

# The next lines truncate the joint error to just one significative figure.
decimal_places = compose(int, np.abs, np.floor, np.log10)
dp = decimal_places(joint_error)
truncated_joint_error = round(joint_error, dp)
print(f'Joint error: {truncated_joint_error}')

bottom_limit = (p_A - p_B) - truncated_joint_error
top_limit = (p_A - p_B) + truncated_joint_error
print(f'{round(bottom_limit, dp)} < p_A - p_B < {round(top_limit, dp)}')
```

    Joint error: 0.01
    -0.02 < p_A - p_B < 0.0

Notice that, according to confidence interval, we might have $\hat p_A > \hat p_B$ as well as $\hat p_A < \hat p_B$.

### Checking assumptions

The assumptions under which the proportion test is valid are:
1. Independent samples.
1. $n_Ap_A > 5$ and $n_A(1-p_A) > 5$
1. $n_Bp_B > 5$ and $n_B(1-p_B) > 5$

Concerning assumption 1, we argue that, as long the same cleaning robot is not present in both samples, and there are no repetitions of the same ID within each sample and in both samples, then the samples are independent, as are each event within each sample. However, it remains to check if there are any correlation among cleaning robots and the area they clean.

Concerning assumptions 2 and 3, we can compute and check explicitly:

```python
print(f'Condition (2) is satisfied: {n_A * p_A > 5 and n_A * (1-p_A) > 5}')
print(f'Condition (3) is satisfied: {n_B * p_B > 5 and n_B * (1-p_B) > 5}')
```

    Condition (2) is satisfied: True
    Condition (3) is satisfied: True

### The test

The function `proportions_ztest` implements the test for proportion. So we need to import it:

```python
from statsmodels.stats.proportion import proportions_ztest
```

```python
z_stat, p_value = proportions_ztest(
  [n_success_B, n_success_A],
  [n_B, n_A],
  alternative='larger'
)

print(f'Sample A (version 1.4): {n_success_A} successes in {n_A} tries.')
print(f'Sample B (version 1.5): {n_success_B} successes in {n_B} tries.')
print(f'Significance level (𝛼): {𝛼}')
print(f'z-statistic: {z_stat}')
print(f'p-value: {p_value}')
```

    Sample A (version 1.4): 1461 successes in 1505 tries.
    Sample B (version 1.5): 737 successes in 754 tries.
    Significance level (𝛼): 0.05
    z-statistic: 0.9249631316745907
    p-value: 0.1774925427443519

### Interpretation

The (absolute value of) $z$ statistics is smaller than $z_\alpha$, which means the result is not too extreme to be meaningful. In statistical language: the data we have does not provide enough evidence to reject the null hypothesis that $p_B - p_A = 0$. This can also be checked with $p$-value, which is greater than the relevance level $\alpha = 5\%$. Indeed, the $p$-value means there are 18% chance that we observe what we observed, given the two samples come from the same population or, equivalently, that null hypothesis is true. In other words, the difference $p_A - p_B$ is not statistically meaningful. In practice it means there is no advantage (or disadvantage) to upgrade software from version 1.4 to 1.5, as it does not provide any meaningul difference in the long run.

### Graphical interpretation

First we define the probability density functions for the sampling proportion, to be reused. Note that, according to central limit theorem, the sampling proportion $\bar P \sim N\left(p, \sqrt{p(1-p)/n}\right)$. i.e., it follows gaussian distribution with mean equal to proportion $p$ and standard deviation $\sqrt{p(1-p)/n}$, where $n$ is the sample size.

```python
# The probability density function for sample A (version 1.4)
@np.vectorize
def pdf_A(x):
  return norm.pdf(x, loc=p_A, scale=standard_error_A)

# The probability density function for sample B (version 1.5)
@np.vectorize
def pdf_B(x):
  return norm.pdf(x, loc=p_B, scale=standard_error_B)
```

```python
fig, ax = plt.subplots()
plt.xlabel('P')
plt.ylabel('Probability density')

# Define the range of the chart
xs = np.linspace(0.95, 1, 100)

# Plot probability density functions
ax.plot(xs, pdf_A(xs), label='1.4')
ax.plot(xs, pdf_B(xs), label='1.5')

# Add vertical line and label for p_A
plt.vlines(p_A, 0, pdf_A(p_A), linestyles='dotted')
plt.text(p_A, 50, '$\hat p_A$', ha='right', fontsize=15)

# Add vertical line and label for p_B
plt.vlines(p_B, 0, pdf_B(p_B), linestyles='dotted')
plt.text(p_B, 50, '$\hat p_B$', ha='left', fontsize=15)

# Add legend
ax.legend();
```

As we can see above, the overlap between probability density functions is not negligible. This is a graphical evidence of the hypothesis test result.

The following chart presents a simplified view of the above result, for didatic purposes only:

```python
fig, ax = plt.subplots()
plt.xlabel('P')
plt.ylabel('Probability density')

# Define the range of the chart
xs = np.linspace(0.95, 1, 100)

# Plot probability density functions
ax.plot(xs, pdf_A(xs), label='1.4')

error = norm.ppf(1-𝛼) * standard_error_A
xs = np.linspace(0.95, p_A+error, 100)
ys = pdf_A(xs)
plt.fill_between(xs, ys, alpha=0.2)

# Add vertical line and label for p_A
plt.vlines(p_A, 0, pdf_A(p_A), linestyles='dotted')
plt.text(p_A, 50, '$\hat p_A$', ha='right', fontsize=15)

# Add vertical line and label for p_B
plt.vlines(p_B, 0, pdf_B(p_B), linestyles='dotted')
plt.text(p_B, 50, '$\hat p_B$', ha='left', fontsize=15)

plt.show()
```

In picture above, were $\hat p_B$ a population parameter, its value would be almost beyond the critical $P$ (the end of shaded area). This means obtaining $\hat p_A$ as far from $\hat p_B$ as seen above is just above 5% (relevance level): the blank area.

### Hand calculation

```python
p = (n_success_A + n_success_B) / (n_A + n_B)
p
```

    0.9729969012837538

```python
z_statistic = (p_B - p_A) / np.sqrt(p*(1-p)*(1/n_A + 1/n_B))
z_statistic
```

    0.9249631316745907

```python
p_value = norm.sf(z_statistic) # sf = Survival function. Check documentation: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bernoulli.html
p_value
```

    0.1774925427443519

### Resampling

The framework used to test the hypothesis does not impose any restriction to the difference in samples size. In any case we can perform the test with two samples of the same size. For this we only need to undersample sample A, which is greater than B.

```python
undersample_A = pd.DataFrame(data={
    'success': np.random.choice(version_A.success, size=len(version_B))
})
```

```python
n_success_A_undersample = len(undersample_A[undersample_A.success])
n_A_undersample = len(undersample_A)

z_stat, p_value = proportions_ztest(
  [n_success_A_undersample, n_success_B],
  [n_A_undersample, n_B],
  alternative='smaller'
)

print(f'Sample A (version 1.4): {n_success_A_undersample} successes in {n_A_undersample} tries.')
print(f'Sample B (version 1.5): {n_success_B} successes in {n_B} tries.')
print(f'Significance level (𝛼): {𝛼}')
print(f'z-statistic: {z_stat}')
print(f'p-value: {p_value}')
```

    Sample A (version 1.4): 728 successes in 754 tries.
    Sample B (version 1.5): 737 successes in 754 tries.
    Significance level (𝛼): 0.05
    z-statistic: -1.3924837638628371
    p-value: 0.0818879774369966

Conclusion is the same as before: $p$-value is greater than significance level $\alpha$, which means the observed result is quite probable, given both samples came from the same population. In other words, there is no distinction between success rate for samples A and B.

Equivalent analysis can be done with z-statistic: it is smaller (absolute value) than critical value ($\sim 1.6$).

### Statistical power

Cohen's d metric is a measure of effect size:

```python
cohen_effect_size = (p_B - p_A) / joint_standard_error
cohen_effect_size
```

    0.9646755334156003

The value ~0.4 for Cohen measure means we are facing a small to medium effect size.

```python
from statsmodels.stats.power import zt_ind_solve_power
```

```python
n_B
```

    754

```python
zt_ind_solve_power(effect_size=cohen_effect_size, alpha=𝛼, nobs1=n_A, ratio=n_B/n_A, alternative='larger')
```

    1.0

The above result is the _statistical power_ for the test. It means the probability of correctly reject the null hypothesis is nearly 100%, supposing the alternative hypothesis is true. That means we can safely state the conclusion (not rejecting the null hypothesis) of the above test is right.

The statistical power is also $1-\beta$, where $\beta$ is the probability of type II errors. In words, there are nearly 0% chance of not reject the null hypothesis, in the case there is an effect (difference between $p_A$ and $p_B$).

# Answers

## Question 1
_Is software version B decreasing the error rate. If yes, by how much?_

**Answer**: no. Software version B does not improve (decrease) error rate in the long run.

## Question 2

_Please quantify your confidence in statement (1)._

**Answer**: the _confidence level_ for the assertion that $p_A = p_B$ is 95%.

## Question 3

_Each failed run costs Dream-Inc $2.32, the software update \$43k. Which software version should be used?_

**Answer**: because there is no statistically significant difference on success (or failure) rates between versions 1.4 and 1.5, the recommendation is to **not** upgrade. The average loss per run is $0.06.

```python
average_loss_per_event = 2.32 * (1 - p) + 0 * p
average_loss_per_event
```

    0.06264718902169113


```python
import numpy as np

# Importing standard Qiskit libraries
from qiskit import QuantumCircuit, transpile, Aer, IBMQ,  assemble
from qiskit.tools.jupyter import *
from qiskit.visualization import *
from ibm_quantum_widgets import *
from qiskit.providers.aer import QasmSimulator

# Loading your IBM Quantum account(s)
provider = IBMQ.load_account()
```

```python
print("Available backends:")
provider.backends()

```

    Available backends:

    [<IBMQSimulator('ibmq_qasm_simulator') from IBMQ(hub='ibm-q', group='open', project='main')>,
     <IBMQBackend('ibmq_armonk') from IBMQ(hub='ibm-q', group='open', project='main')>,
     <IBMQBackend('ibmq_santiago') from IBMQ(hub='ibm-q', group='open', project='main')>,
     <IBMQBackend('ibmq_bogota') from IBMQ(hub='ibm-q', group='open', project='main')>,
     <IBMQBackend('ibmq_lima') from IBMQ(hub='ibm-q', group='open', project='main')>,
     <IBMQBackend('ibmq_belem') from IBMQ(hub='ibm-q', group='open', project='main')>,
     <IBMQBackend('ibmq_quito') from IBMQ(hub='ibm-q', group='open', project='main')>,
     <IBMQSimulator('simulator_statevector') from IBMQ(hub='ibm-q', group='open', project='main')>,
     <IBMQSimulator('simulator_mps') from IBMQ(hub='ibm-q', group='open', project='main')>,
     <IBMQSimulator('simulator_extended_stabilizer') from IBMQ(hub='ibm-q', group='open', project='main')>,
     <IBMQSimulator('simulator_stabilizer') from IBMQ(hub='ibm-q', group='open', project='main')>,
     <IBMQBackend('ibmq_manila') from IBMQ(hub='ibm-q', group='open', project='main')>]

```python
backend = provider.get_backend('ibmq_lima')
```

```python
backend
```

    VBox(children=(HTML(value="<h1 style='color:#ffffff;background-color:#000000;padding-top: 1%;padding-bottom: 1…

    <IBMQBackend('ibmq_lima') from IBMQ(hub='ibm-q', group='open', project='main')>

```python
#def initialization in superposition
def initialization(qc, qubits):
    for qubit in qubits:
        qc.h(qubit)
    return qc
```

```python
example = QuantumCircuit(3)
example_ = initialization(example, range(3))
f = example_.draw('mpl')
f.savefig('Images/QiskitGInit.png')
f
```

```python
#def oracle marking state [0,0,.....1]
# 1 state marked
num_q = 3

from numpy import pi
qc = QuantumCircuit(num_q)

for i in range(num_q -2):
    qc.x(i)
qc.h(num_q -1)
qc.mct(list(range(num_q -2)), num_q -1)  # multi-controlled-toffoli
qc.h(num_q -2)
#o = [0, 1, 2, 4]
qc.mct(list(range(num_q -2)), num_q -2)
#qc.mct(o, num_q -2)
qc.h(num_q -2)
#temp
#qc.h(num_q -2)
#qc.mct(list(range(num_q -2)), num_q -2)  # multi-controlled-toffoli
#qc.h(num_q -2)
#fine temp

qc.h(num_q -1)
for i in range(num_q -2):
    qc.x(i)

oracle_ = qc.to_gate()
oracle_.name = "Oracle"
f = qc.draw('mpl')
f.savefig('Images/QiskitGOracle.png')
f
```

```python
num_q = 3

from numpy import pi
qc = QuantumCircuit(num_q)

qc.x(0)
qc.cz(0, 2)
qc.cz(0, 1)
qc.x(0)
oracle_ = qc.to_gate()
oracle_.name = "Oracle"
f = qc.draw('mpl')
#f.savefig('Images/QiskitGOracle.png')
f
```

```python
#diffuser
def diffuser(nqubits):
    qc = QuantumCircuit(nqubits)
    # Apply transformation |s> -> |00..0> (H-gates)
    for qubit in range(nqubits):
        qc.h(qubit)
    # Apply transformation |00..0> -> |11..1> (X-gates)
    for qubit in range(nqubits):
        qc.x(qubit)
    # Do multi-controlled-Z gate
    qc.h(nqubits-1)
    qc.mct(list(range(nqubits-1)), nqubits-1)  # multi-controlled-toffoli
    qc.h(nqubits-1)
    # Apply transformation |11..1> -> |00..0>
    for qubit in range(nqubits):
        qc.x(qubit)
    # Apply transformation |00..0> -> |s>
    for qubit in range(nqubits):
        qc.h(qubit)
    # We will return the diffuser as a gate
    U_s = qc.to_gate()
    U_s.name = "Diff"
    return U_s
```

```python
a = diffuser(num_qubits)
f = a.draw()
f.savefig('Images/QiskitGDiffuser.png')
f
```

    Traceback (most recent call last):
      File "<ipython-input-46-5b5d3cf5e42c>", line 2, in <module>
        f = a.draw()
    AttributeError: 'Gate' object has no attribute 'draw'

    Use %tb to get the full traceback.

<button class='button iqx-button'>Search for solution online</button>

```python
num_qubits = 3
grover_circ = QuantumCircuit(num_qubits)
grover_circ = initialization(grover_circ, range(num_qubits))
for i in range(1):
    grover_circ.append(oracle_, range(num_qubits))
    grover_circ.append(diffuser(num_qubits), range(num_qubits))
grover_circ.measure_all()
f = grover_circ.draw()
#f.savefig('Images/QiskitGAll.png')
f
```

```python

```

```python
aer_sim = Aer.get_backend('aer_simulator')
transpiled_grover_circuit = transpile(grover_circ, aer_sim)
qobj = assemble(transpiled_grover_circuit)
results = aer_sim.run(qobj).result()
counts = results.get_counts()
f = plot_histogram(counts)
#f.savefig('Images/QiskitGSimulator.png')
f
```

```python
from qiskit.tools.monitor import job_monitor
device = backend
transpiled_grover_circuit = transpile(grover_circ, device, optimization_level=3)
job = device.run(transpiled_grover_circuit, shots = 8000)
job_monitor(job, interval=2)
```

    Job Status: job has successfully run

```python
results = job.result()
answer = results.get_counts(grover_circ)
f = plot_histogram(answer)
f.savefig('Images/QiskitGIBMQlima.png')
f
```

```python

```