    Principal Component Analysis (PCA)

# Introduction

In the previous lessons you've learned the core idea behind **Principal Component Analysis (PCA)** and leanred about eigenvectors and eigenvalues. Before we apply PCA to Risk Factor Models, in this notebook, we will see how we can use PCA for **Dimensionality Reduction**. In short, dimensionality reduction is the process of reducing the number of variables used to explain your data.

We will start by giving a brief overview of dimensionality reduction, we will then use Scikit-Learn's implementation of PCA to reduce the dimension of random correlated data and visualize its principal components.

# Dimensionality Reduction

One of the main applications of Principal Component Analysis is to reduce the dimensionality of highly correlated data. For example, suppose your data looks like this:

Fig 1. - Highly Correlated Data.

We can see that this 2-Dimesnional data is described by two variables, $X$ and $Y$. However, notice that all the data points lie close to a straight line:

Fig 2. - Direction of Biggest Variation.

We can see that most of the variation in the data occurs along this particular purple line. This means, that we could explain most of the variation of the data by only looking at how the data is distributed along this particular line. Therefore, we could reduce the data from 2D to 1D data by projecting the data points onto this straight line:

Fig 3. - Projected Points.

This will reduce the number of variables needed to describe the data from 2 to 1 since you only need one number to specify a data point's position on a straight line. Therefore, the 2 variables that describe the 2D plot will be replaced by a new single variable that encodes the 1D linear relation.

Fig 4. - Data Reduced to 1D.

It is important to note, that this new variable and dimension don't need to have any particular meaning attached to them. For example, in the original 2D plot, $X$ and $Y$ may represent stock returns, however, when we perform dimensionality reduction, the new variables and dimensions don't need to have any such meaning attach to them. The new variables and dimensions are just abstract tools that allow us to express the data in a more compact form. While in some cases these new variables and dimensions may represent a real-world quantities, it is not necessary that they do.

Dimensionality reduction of correlated data works on any number of dimensions, *i.e.* you can use it to reduce $N$-Dimensional data to $k$-Dimensional data, where $k < N$. As mentioned earlier, PCA is one of the main tools used to perform such dimensionality reduction. To see how this is done, we will apply PCA to some random corelated data. In the next section, we create random data with a given amount of correlation.

# Create a Dataset

In this section we will learn how to create random correlated data. In the code below we will use the `utils.create_corr_data()` function from the `utils` module to create our random correlated data. The `utils` module was created specifically for this notebook and contains some useful functions. You are welcome to take a look it to see how the fucntions work. In the code below, you can choose the data range and the amount of correlation you want your data to have. These parameters are then passed to the `utils.create_corr_data()` function to create the data. Finally we use the `utils.plot_data()` function from the `utils` module to plot our data. You can explore how the data changes depending on the amount of correlation. Remember the correlation is a number ranging from -1 to 1. A correlation `0` indicates that there is no correlation between the data points, while a correlation of `1` and `-1` indicate the data points display full positive and negative correlation, respectively. This means, that for `corr = +/-1` all the points will lie in a straight line.

```python
%matplotlib inline

import matplotlib.pyplot as plt

import utils

# Set the default figure size
plt.rcParams['figure.figsize'] = [10.0, 6.0]

# Set data range
data_min = 10
data_max = 80

# Set the amount of correlation. The correlation is anumber in the closed interval [0,1].
corr = 0.8

# Create correlated data
X = utils.create_corr_data(corr, data_min, data_max)

# Plot the correlated data
utils.plot_data(X, corr)
```

# Mean Normalization

Before we can apply PCA to our data, it is important that we mean normalize the data. Mean normalization will evenly distribute our data in some small interval around zero. Consequently, the average of all the data points will be close to zero. The code below uses the `utils.mean_normalize_data()` function from the `utils` module to mean normalize the data we created above. We will again use the `utils.plot_data()` function from the `utils` module to plot our data. When we plot the mean normalized data, we will see that now the data is evenly distributed around the origin with coordinates`(0,0)`. This is expected because the average of all points should be zero.

```python
%matplotlib inline

import matplotlib.pyplot as plt

import utils

# Set the default figure size
plt.rcParams['figure.figsize'] = [10.0, 6.0]

# Mean normalize X
X_norm = utils.mean_normalize_data(X)

# Plot the mean normalized correlated data
utils.plot_data(X_norm, corr)
```

# PCA In Brief

Let's go back to the example we saw at the beginning. We had some 2D data that lies close to a straight line and we would like to reduce this data from 2D to 1D by projecting the data points onto a straight line. But how do we find the best straight line to project our data onto? In fact, how do we define the best straight line? We define the best line as the line such that the sum of the squares of the distances of the data points to their projected counterparts is minimized. It is important to note, that these projected distances are orthogonal to the straight line, not vertical as in linear regression. Also, we refer to the distances from the data points to their projected counterparts as *projection errors*. Now that we have defined what the best straight line should be, how do we find it? This is where PCA comes in. For this particular example, PCA will find a straight line on which to project the data such that the sum of squares of the projection errors is minimized. So we can use PCA to find the best straight line to project our data onto.

In general, for $N$-Dimensional data, PCA will find the lower dimensional surface on which to project the data so as to minimize the projection error. The lower dimensional surface is going to be determined by a set of vectors $v^{(1)}, v^{(2)}, ...v^{(k)}$, where $k$ is the dimension of the lower dimensional surface, with $k<N$. So for our example above, where we were reducing 2D data to 1D data, so $k=1$, and hence the lower dimensional surface, a straight line in this case, will be determined by only one vector $v^{(1)}$. This makes sense because you only need one vector to describe a straight line. Similarly in the case of reducing 3D data to 2D data, $k=2$, and hence we will have two vectors to determine a plane (a 2D surface) on which to project our data.

Therefore, what the PCA algorithm is really doing is finding the vectors that determine the lower dimensional surface that minimizes the projection error. As you learned previously, these vectors correspond to a subset of the eigenvectors of the data matrix $X$. We call this subset of eigenvectors the *Principal Components* of $X$. We also define the first principal component to be the eigenvector corresponding to the largest eigenvalue of $X$; the second principal component as the eigenvector corresponding to the second largest eigenvalue of $X$, and so on. If $v^{(1)}, v^{(2)}, ...v^{(N)}$ is the set of eigenvectors of $X$ then the principal components of $X$ will be determined by the subset $v^{(1)}, v^{(2)}, ...v^{(k)}$, for some chosen value of $k$, where $k<N$. Remember that $k$ determines the dimension of the lower dimensional surface were projecting our data to.

You can program the PCA algorithm by hand, but luckily many packages, such as Scikit-Learn, already contain built-in functions that perfrom the PCA algorithm for you. In the next section we will take a look at how we can implement the PCA algorithm using Scikit-Learn.

# PCA With Scikit-Learn

We can perform principal component analysis on our data using Scikit-Learn's [`PCA()` class](http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html). Scikit-Learn's `PCA()` class uses a technique called **Singular Value Decomposition (SVD)** to compute the eigenvectors and eigenvalues of a given set of data. Given a matrix $X$ of shape $(M, N)$, the SVD algorithm consists of factorizing $X$ into 3 matrices $U, S$, and $V$ such that:

\begin{equation}
X = U S V
\end{equation}

The shape of the $U$ and $V$ matrices depends on the implementation of the SVD algorithm. When using Scikit-Learn's `PCA()` class, the $U$ and $V$ matrices have dimensions $(M, P)$ and $(P,N)$, respectevely, where $P = \min(M, N)$. The $V$ matrix contains the eigenvectors of $X$ as rows and the $S$ matrix is a diagonal $(P,P)$ matrix and contains the eigenvalues of $X$ arranged in decreasing order, *i.e.* the largest eigenvalue will be the element $S_{11}$, the second largest eigenvalue will be the element $S_{22}$, and so on. The eigenvectors in $V$ are arranged such that the first row of $V$ holds the eigenvector correspoding to the eigenvalue in $S_{11}$, the second row of $V$ will hold the eigenvector corresponding to eigenvalue in $S_{22}$, and so on.

Once the eigenvectors and eigenvalues have been calculated using SVD, the next step in dimensionality reduction using PCA is to choose the size of the dimension we are going to project our data onto. The size of this dimension is determined by $k$, which tells us the number of principal components we want to use. We can tell the `PCA()` class the number of principal components to return, by setting the parameter `n_components=k`, for some chosen value of $k$, like in the code below:

```python
from sklearn.decomposition import PCA

pca = PCA(n_components = 2)

print('\nPCA Parameters:', pca, '\n')
```

    PCA Parameters: PCA(copy=True, iterated_power='auto', n_components=2, random_state=None,
      svd_solver='auto', tol=0.0, whiten=False)

As we can see `pca`contains the parameters we are going to use for our PCA algorithm. Since the random correlated data we created above is 2D, then in this case, $X$ will have a maximum of 2 eigenvectors. We have chosen $k=2$, so that the `PCA()` class will return both principal components (eigenvectors). We chose $k=2$ because we want to visulaize both principal components in the next section. If we didn't want to visulaize both pricncipal components, but rather perform dimensionality reduction directly, we would have chosen $k=1$, to reduce our 2D data to 1D, as we showed at the beginnig.

After we have set the paramters of our PCA algorithm, we now have to pass the data to the `PCA()` class. This is done via the `.fit()` method as shown in the code below:

```python
pca.fit(X_norm);
```

Once the `PCA()` class fits the data through the `.fit()` method it returns an array containing the principal components in the attribute `.components_`, and the corresponding eigenvalues in a 1D array in the attribute `.singular_values_`. Other attributes of the `PCA()` class include `.explained_variance_ratio_` which gives the percentage of variance explained by each of the principal components. In the code below we access the above attributes and display their contents:

```python
print('\nArray Containing all Principal Components:\n', pca.components_)
print('\nFirst Principal Component:', pca.components_[0])
print('Second Principal Component:', pca.components_[1])
print('\nEigenvalues:', pca.singular_values_)
print('\nPercentage of Variance Explained by Each Principal Component:', pca.explained_variance_ratio_)
```

    Array Containing all Principal Components:
     [[-0.70710678 -0.70710678]
     [ 0.70710678 -0.70710678]]

    First Principal Component: [-0.70710678 -0.70710678]
    Second Principal Component: [ 0.70710678 -0.70710678]

    Eigenvalues: [42.39465018 14.23705152]

    Percentage of Variance Explained by Each Principal Component: [0.89865318 0.10134682]

We can see that the first principal component has a corresponding eigenvalue of around 42, while the second principal component has an eigenvalue of around 14. We can also see from the `.explained_variance_ratio_` attribute that the first principal component explains approximately 90% of the variance in the data, while the second principal components only explains around 10%. In general, the principal components with the largest eigenvalues, explain the mayority of the variance. In dimensionality reduction, it is custumary to project your data onto the principal components that explain the mayority of the variance. In this case for example, we would like to project our data onto the first principal component, since it exaplans 90% of the variance in the data. For more inforamtion on Scikit-Learn's `PCA()` class please see the [documentation](http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html).

# Visualizing The Principal Components

Now that we have our principal components, let's visualize them. In the code below we use the `utils.plot_data_with_pca_comp()` function from the `utils` module to calculate the principal components of our random correlated data. The function performs all the steps we have seen above and then plots the resulting principal components along with the data. In order to prevent you from scrolling up and down the notebook to create new random correlated data, I have copied the same code as before so you can change the parameters of the random data here as well.

```python
%matplotlib inline

import matplotlib.pyplot as plt

import utils

# Set the default figure size
plt.rcParams['figure.figsize'] = [10.0, 6.0]

# Set data range
data_min = 10
data_max = 80

# Set the amount of correlation
corr = 0.8

# Plot the data and the principal components
utils.plot_data_with_pca_comp(corr, data_min, data_max)
```

# Choosing The Number of Components

As we saw, the dimension of the lower dimensional surface, $k$, is a free parameter of the PCA algorithm. When working with low dimensional data, choosing the value of $k$ can be easy, for example, when working with 2D data, you can choose $k=1$, to reduce your 2D data to 1D. However, when working with high dimensional data, a suitable value for $k$ is not that clear. For example, suppose you had 1,000-dimensional data, what will be the best choice for $k$? Should we choose $k=500$ to reduce our data from 1,000D to 500D, or we could do better than that and reduce our data to 100D by choosing $k=100$.

Usually, the number of principal components, $k$, is chosen depending on how much of the variance of the original data you want to retain. Typically, you choose $k$ such that anywhere from 80% to 99% of the variance of the original data is retained, however you can choose a lower percentage if that is what you desire for your particular application. You check the percentage of the variance of your data that is explained for a given value of $k$ using the `.explained_variance_ratio_` attribute as we saw before. So, in practice what you will do, is to have the `PCA()`class return all the eigenvectors and then add up the elements in the array returned the `.explained_variance_ratio_` attribute until the desired retained variance is reached. For example, if we wanted to retain 98% of the variance in our data we choose $k$ such that the following condition is true:

\begin{equation}
\sum_{i=1}^k P_{i} \geq 0.98
\end{equation}

where $P$ is the array returned the `.explained_variance_ratio_` attribute. So, if you are choosing the value of $k$ manually, you can use the above formula to figure out what percentage of variance was retained for that particular value of $k$. For highly correlated data, it is possible to reduce the dimensions of the data significantly, even if we retain 99% of the variance.

# Projected Data

Now that we have seen what all the principal components look like, we will now use the `PCA()` class with `n_components = 1` so that we can perform dimensionality reduction. Once we find the vectors (principal component) that span our lower dimensional space, the next part of the dimensionality redcution algorithm is to find the projected values of our data onto that space. We can use the `.transform()` method from the `PCA()` class to apply dimensionalty reduction and project our data points onto the lower dimensional space. In our simple example, $k=1$, so we will only have a principal component to project our data onto.

In the code below, we apply PCA with `n_components = 1`  and use the `.transform()` method to project our data points onto a straight line. Finally we plot our projected data.

```python
import numpy as np

from sklearn.decomposition import PCA

pca = PCA(n_components = 1)

pca.fit(X_norm);

transformed_data = pca.transform(X_norm)

yvals = np.zeros(1000)

# Plot the data
plt.scatter(transformed_data, yvals, color = 'white', alpha = 0.5, linewidth = 0)
ax = plt.gca()
ax.set_facecolor('lightslategray')
plt.grid()

plt.show()
```


```python
# 그래프, 수학 기능 추가
# Add graph and math features
import pylab as py
import numpy as np
import numpy.linalg as nl
# 기호 연산 기능 추가
# Add symbolic operation capability
import sympy as sy

```

```python
sy.init_printing()

```

# 2차 적분
Second Order Numerical Integral

다시 면적 1인 반원을 생각해 보자.

Again, let's think about a half circle with area of 1.

```python
import plot_num_int as pi

```

```python
r = pi.radius_of_half_circle_area(1)

```

```python
pi.plot_a_half_circle_of_area(1)
pi.axis_equal_grid_True()

```

이번에는 3 지점에서의 함수값을 이용하는 심슨규칙을 이용해서 구해 보기로 하자.

This time, let's integrate by the Simpson's rule using function values at three points.

## 심슨 규칙
Simpson's Rule

마찬가지로 일정 간격으로 $x$ 좌표를 나누어 보자.

Same as before, let's divide $x$ coordinates in a constant interval.

```python
n = 10

pi.plot_half_circle_with_stems(n, 1)

```

마지막 두 구간을 생각해 보기로 하자.

Let's just think about the last two segments.

```python
n = 10

pi.plot_half_circle_with_stems(n, 1)
x_array, y_plus = pi.get_half_circle_xy_theta_space(1)
x_array_bar, y_array_bar = pi.get_half_circle_xy_linspace(n, 1)

# 마지막 두 구간에 해당하는 x y 값을 선택
# Choose x y values of the last two intervals
x_last_two_array = x_array[x_array_bar[-3] < x_array]
y_last_two_array = y_plus[x_array_bar[-3] < x_array]

py.fill_between(x_last_two_array, y_last_two_array)

py.axis('equal')
py.grid(True)

```

해당 넓이를 구하기 위해, 이 세 점을 지나는 2차 다항식을 찾아서 적분할 수 있을 것이다

To get the area, we would be able to find a second order polynomal passing through these three points and integrate.

문제를 좀 더 쉽게 만들기 위해 해당 면적을 원점 주위로 평행 이동 시켜 보자.

To make the problem simpler, let's translate the area around the origin.

```python
delta_x = x_array_bar[1]-x_array_bar[0]

py.plot(x_array, y_plus, alpha=0.0)
py.plot(x_array_bar[-3:], y_array_bar[-3:], '.')

# 마지막 두 구간을 표시
# Indicate last two intervals
py.fill_between(x_last_two_array, y_last_two_array)

# x 좌표 표시
# Indicate x coordinates
py.text(x_last_two_array[0], -0.1, '$x_{n-2}$', horizontalalignment='center')
py.text(x_last_two_array[-1], -0.1, '$x_{n}$', horizontalalignment='center')

# y 좌표 표시
# Indicate x coordinates
py.text(x_array_bar[-3], y_array_bar[-3], '$f(x_{n-2})$', horizontalalignment='center', verticalalignment='bottom')
py.text(x_array_bar[-2], y_array_bar[-2], '$f(x_{n-1})$', horizontalalignment='center', verticalalignment='bottom')
py.text(x_array_bar[-1], y_array_bar[-1], '$f(x_{n})$', horizontalalignment='center', verticalalignment='bottom')

# 평행이동한 면적
# Translated Area
py.plot(x_array_bar[-3:]-x_array_bar[-2], y_array_bar[-3:], '.')
py.fill_between(x_last_two_array-x_array_bar[-2], y_last_two_array)

# x 좌표 표시
# Indicate x coordinates
py.text(-delta_x, -0.1, '$-\Delta x$', horizontalalignment='center')
py.text(delta_x, -0.1, '$+\Delta x$', horizontalalignment='center')

# y 좌표 표시
# Indicate x coordinates
py.text(-delta_x, y_array_bar[-3], '$y_0$', horizontalalignment='center', verticalalignment='bottom')
py.text(          0, y_array_bar[-2], '$y_1$', horizontalalignment='center', verticalalignment='bottom')
py.text(+delta_x, y_array_bar[-1], '$y_2$', horizontalalignment='center', verticalalignment='bottom')

py.axis('equal')
py.grid(True)

```

$$
y=a_0 x^2 + a_1 x + a_2
$$

원래 위치의 면적과 평행이동한 면적은 같다.
The translate area and the original area are equivalent.

평행이동한 면적의 세 점을 살펴 보자.
Let's take a look at the three points of the translated area.

$$
\begin{align}
    p_0&=\left(-\Delta x, y_0\right) \\
    p_1&=\left(0, y_1\right) \\
    p_2&=\left(\Delta x, y_2\right)
\end{align}
$$

```python
delta_x, y_m, y_0, y_p = sy.symbols('Delta_x, y_0, y_1, y_2', real=True)

```

```python
points = (-delta_x, y_m), (0, y_0), (delta_x, y_p)

```

```python
points

```

2차 다항식은 다음과 같은 형태를 가진다.

A second order polynomial would take following form.

```python
a0, a1, a2, x = sy.symbols('a0, a1, a2, x', real=True)
f = a0 * x**2 + a1 * x + a2

```

```python
f

```

위 세 점을 모두 지나는 2차 곡선을 생각해 보자.
Let's think about a second order polynomal passing all three points above.

$$
\begin{align}
    y_0&=a_0 \left(-\Delta x\right)^2 + a_1 \left(-\Delta x\right) + a_2 \\
    y_1&=a_2 \\
    y_2&=a_0 \left(\Delta x\right)^2 + a_1 \left(\Delta x\right) + a_2
\end{align}
$$

```python
eq_points = [sy.Eq(p[-1], f.subs(x, p[0])) for p in points]

```

```python
eq_points

```

계수 $a_i$에 관하여 풀어 보자.
Let's try to solve for the coefficients $a_i$.

```python
a_sol = sy.solve(eq_points, (a0, a1, a2))

```

```python
a_sol

```

## 2차 다항식의 정적분
Definite Integral of a Second Order Polynomial

이제 $f(x)$를 $x$에 관하여 $-\Delta x$ 부터 $\Delta x$까지 적분해 보자.
Now let's integrate $f(x)$ about $x$ from $-\Delta x$ to $\Delta x$.

```python
integral = sy.integrate(f, (x, -delta_x, delta_x))

```

```python
integral

```

계수를 대입하고 정리해 보자.
Let's substitute the coefficients and simplfy.

```python
simpson = sy.simplify(integral.subs(a_sol))

```

```python
simpson

```

예를 들어 C 언어 코드로는 다음과 같이 가능하다
For example, in C programming language, following expression would be possible.

```python
sy.ccode(simpson)

```

## 심슨 규칙 구현
Implementing Simpson's Rule

한번에 두 구간의 면적을 계산한다.

In one iteration, calculate the area of two intervals.

$$
    Area = F_0 + F_2 + \ldots + F_{n-2}
$$

$$
    F_k = \frac{\Delta x}{3}\left[f(x_k)+4 \cdot f(x_{k+1}) + f(x_{k+2})\right]
$$

```python
def get_delta_x(xi, xe, n):
    return (xe - xi)/n

```

```python
def num_int_2(f, xi, xe, n_partition, b_verbose=False):
    """
    f : function to indegrate f(x)
    xi : start of integration
    xe : end of integration
    n_partition : number of partitions within the interval
    """
    # 구간의 갯수를 항상 짝수로 한다.
    # Always use even number of intervals

    if n_partition % 2:

        n_partition += 1

    delta_x = get_delta_x(xi, xe, n_partition)

    # delta_x 값이 너무 작은 경우
    # if delta_x is too small
    if 1e-7 > abs(delta_x):
        raise ValueError(f'delta_x(delta_x:g) too small')

    x_array = py.linspace(xi, xe, n_partition+1)

    assert 1e-3 > abs((abs(x_array[1] - x_array[0]) - delta_x)/delta_x), (
        f"\ndelta_x = {delta_x} "
        f"\nx_array[1] - x_array[0] = {x_array[1] - x_array[0]}"
    )

    delta_x_third = delta_x / 3.0

    integration_result = 0.0
    xp = x_array[0]
    y0 = f(xp)

    for i in range(1, n_partition, 2):
        x1 = x_array[i]
        x2 = x_array[i+1]

        y1 = f(x1)
        y2 = f(x2)

        area_i = delta_x_third * (y0 + 4*y1 + y2)

        if b_verbose: print('i = %2d, area_i = %g' % (i-1, area_i))

        xp, y0 = x2, y2
        integration_result += area_i

    return integration_result

```

```python
n = 10
result = num_int_2(pi.half_circle, -r, r, n, b_verbose=True)

```

```python
n = 100
result = num_int_2(pi.half_circle, -r, r, n)
print('result =', result)

```

```python
%timeit -n 100 result = num_int_2(pi.half_circle, -r, r, n)

```

```python
n = 2**8
result_256 = num_int_2(pi.half_circle, -r, r, n)
print('result =', result_256)

```

```python
%timeit -n 100 result = num_int_2(pi.half_circle, -r, r, n)

```

### $cos \theta$의 반 주기
Half period of $cos \theta$

```python
n = 10
result_cos = num_int_2(py.cos, 0, py.pi, n, b_verbose=True)
print('result =', result_cos)

```

```python
n = 100
result_cos = num_int_2(py.cos, 0, py.pi, n)
print('result =', result_cos)

```

### 1/4 원
A quarter circle

```python
n = 10
result_quarter = num_int_2(pi.half_circle, -r, 0, n, b_verbose=True)
print('result =', result_quarter)

```

```python
n = 100
result_quarter = num_int_2(pi.half_circle, -r, 0, n)
print('result =', result_quarter)

```

## 연습문제
Exercises

도전 과제 1 : 넓이 1인 반원의 예로 0차, 1차 적분과의 오차를 비교하시오.
Using the example of half circle with area 1, compare errors with zeroth and first order integrations.

```python

```

도전 과제 2 : 긴 지름 4, 짧은 지름 2인 타원의 면적의 절반을 심슨법으로 계산하시오. [[위키피디아](https://ko.wikipedia.org/wiki/%ED%83%80%EC%9B%90)]
Try this 2 : Calculate the half of area of an ellipse with long diameter 4 and short diameter 2 using the Simpson's rule. [[wikipedia](https://en.wikipedia.org/wiki/Ellipse)]

$$
    \frac{x^2}{4^2} + \frac{y^2}{2^2} = 1
$$

```python

```

## 함수형 프로그래밍
Functional programming

$n$ 개의 간격에 대해 심슨 규칙 적용을 생각해 보자.

Let's think about applying Simpson's rule over $n$ intervals.

$$
    Area = F_0 + F_2 + \ldots + F_{n-2}
$$

$$
    F_k = \frac{\Delta x}{3}\left[f(x_k)+4 \cdot f(x_{k+1}) + f(x_{k+2})\right]
$$

$$
\begin{align}
    Area &= \frac{\Delta x}{3}\left[f(x_0)+4 \cdot f(x_{1}) + f(x_{2})\right] \\
        &+ \frac{\Delta x}{3}\left[f(x_2)+4 \cdot f(x_{3}) + f(x_{4})\right] \\
        &+ \frac{\Delta x}{3}\left[f(x_4)+4 \cdot f(x_{5}) + f(x_{6})\right] \\
        & \ldots \\
        &+ \frac{\Delta x}{3}\left[f(x_{n-4})+4 \cdot f(x_{n-3}) + f(x_{n-2})\right] \\
        &+ \frac{\Delta x}{3}\left[f(x_{n-2})+4 \cdot f(x_{n-1}) + f(x_{n})\right] \\
\end{align}
$$

$$
\begin{align}
    Area &= \frac{\Delta x}{3}\left[f(x_0)+f(x_{n})\right] \\
        &+ \frac{4}{3}\Delta x \left[f(x_{1}) + f(x_{3}) + f(x_{5}) + \ldots + f(x_{n-3}) + f(x_{n-1})\right] \\
        &+ \frac{2}{3}\Delta x \left[f(x_{2}) + f(x_{4}) + f(x_{6}) + \ldots + f(x_{n-4}) + f(x_{n-2})\right] \\
\end{align}
$$

```python
def even_sum_func(f, xi, xe, delta_x):
    return sum(
        map(
            f,
            py.arange(xi+delta_x, xe-delta_x*0.5, delta_x*2),
        )
    )

```

```python
def odd_sum_func(f, xi, xe, delta_x):
    return sum(
        map(
            f,
            py.arange(xi+(delta_x*2) , xe-delta_x*0.5, delta_x*2),
        )
    )

```

```python
def num_int_2_functional(f, xi, xe, n):
    return (
        (get_delta_x(xi, xe, n) * (1.0/3)) * (
            f(xi) + f(xe)
            + 4 * even_sum_func(f, xi, xe, get_delta_x(xi, xe, n))
            + 2 * odd_sum_func(f, xi, xe, get_delta_x(xi, xe, n))
        )
    )

```

```python
n = 100
result_func = num_int_2_functional(pi.half_circle, -r, r, n)
print('result_func =', result_func)

```

```python
assert 1e-7 > abs(result - result_func), f"result = {result}, result_func = {result_func}"

```

```python
%timeit -n 100 result_func = num_int_2_functional(pi.half_circle, -r, r, n)

```

## 시험
Test

아래는 함수가 맞게 작동하는지 확인함

Following cells verify whether the functions work correctly.

```python
import pylab as py
r = py.sqrt(1.0 / py.pi)
n = 10
delta_x = r/n

def half_circle(x):
    return py.sqrt(r**2 - x ** 2)

assert 0.25 > num_int_2(half_circle, -r, 0, n)
assert 0.25 > num_int_2(half_circle, 0, r, n)
assert 0.25 > num_int_2_functional(half_circle, -r, 0, n)
assert 0.25 > num_int_2_functional(half_circle, 0, r, n)

```

```python
assert 0.1 > (abs(num_int_2(half_circle, -r, 0, n) - 0.25) * 4)
assert 0.1 > (abs(num_int_2(half_circle, 0, r, n) - 0.25) * 4)
assert 0.1 > (abs(num_int_2_functional(half_circle, -r, 0, n) - 0.25) * 4)
assert 0.1 > (abs(num_int_2_functional(half_circle, 0, r, n) - 0.25) * 4)

```

## Final Bell
마지막 종

```python
# stackoverfow.com/a/24634221
import os
os.system("printf '\a'");

```

```python

```


## Interactive Variogram Modeling Demonstration

### Michael Pyrcz, Associate Professor, University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

### The Interactive Workflow

Here's a simple workflow on detecting the major spatial continuity directions in a spatial dataset with variogram analysis and then building a consistent. positive definate variogram model in 2D.

* this model will provide consistent variogram values for any offset (azimuth and distance) in 2D

This information is essential to optimum well placement and prectiction away from wells.  First let's explain the concept of spatial continuity and the variogram.

#### Spatial Continuity

**Spatial Continuity** is the correlation between values over distance.

* No spatial continuity – no correlation between values over distance, random values at each location in space regardless of separation distance.

* Homogenous phenomenon have perfect spatial continuity, since all values as the same (or very similar) they are correlated.

We need a statistic to quantify spatial continuity! A convenient method is the Semivariogram.

#### The Semivariogram

Function of difference over distance.

* The expected (average) squared difference between values separated by a lag distance vector (distance and direction), $h$:

\begin{equation}
\gamma(\bf{h}) = \frac{1}{2 N(\bf{h})} \sum^{N(\bf{h})}_{\alpha=1} (z(\bf{u}_\alpha) - z(\bf{u}_\alpha + \bf{h}))^2
\end{equation}

where $z(\bf{u}_\alpha)$ and $z(\bf{u}_\alpha + \bf{h})$ are the spatial sample values at tail and head locations of the lag vector respectively.

* Calculated over a suite of lag distances to obtain a continuous function.

* the $\frac{1}{2}$ term converts a variogram into a semivariogram, but in practice the term variogram is used instead of semivariogram.
* We prefer the semivariogram because it relates directly to the covariance function, $C_x(\bf{h})$ and univariate variance, $\sigma^2_x$:

\begin{equation}
C_x(\bf{h}) = \sigma^2_x - \gamma(\bf{h})
\end{equation}

Note the correlogram is related to the covariance function as:

\begin{equation}
\rho_x(\bf{h}) = \frac{C_x(\bf{h})}{\sigma^2_x}
\end{equation}

The correlogram provides of function of the $\bf{h}-\bf{h}$ scatter plot correlation vs. lag offset $\bf{h}$.

\begin{equation}
-1.0 \le \rho_x(\bf{h}) \le 1.0
\end{equation}

#### Variogram Observations

The following are common observations for variograms that should assist with their practical use.

##### Observation \#1 - As distance increases, variability increase (in general).

This is common since in general, over greater distance offsets, there is often more difference between the head and tail samples.

In some cases, such as with spatial cyclicity of the hole effect variogram model the variogram may have negative slope over somelag distance intervals

Negative slopes at lag distances greater than half the data extent are often caused by too few pairs for a reliable variogram calculation

##### Observation \#2 - Calculated with over all possible pairs separated by lag vector, $\bf{𝐡}$.

We scan through the entire data set, searching for all possible pair combinations with all other data.  We then calculate the variogram as one half the expectation of squared difference between all pairs.

More pairs results in a more reliable measure.

##### Observation \#3 - Need to plot the sill to know the degree of correlation.

**Sill** is the variance, $\sigma^2_x$

Given stationarity of the variance, $\sigma^2_x$, and variogram $\gamma(\bf{h})$:

we can define the covariance function:

\begin{equation}
C_x(\bf{h}) = \sigma^2_x - \gamma(\bf{h})
\end{equation}

The covariance measure is a measure of similarity over distance (the mirror image of the variogram as shown by the equation above).

Given a standardized distribution $\sigma^2_x = 1.0$, the covariance, $C_x(\bf{h})$, is equal to the correlogram, $\rho_x(\bf{h})$:

\begin{equation}
\rho_x(\bf{h}) = \sigma^2_x - \gamma(\bf{h})
\end{equation}

##### Observation \#4 - The lag distance at which the variogram reaches the sill is know as the range.

At the range, knowing the data value at the tail location provides no information about a value at the head location of the lag distance vector.

##### Observation \#5 - The nugget effect, a discontinuity at the origin

Sometimes there is a discontinuity in the variogram at distances less than the minimum data spacing.  This is known as **nugget effect**.

The ratio of nugget / sill, is known as relative nugget effect (%). Modeled as a discontinuity with no correlation structure that at lags, $h \gt \epsilon$, an infinitesimal lag distance, and perfect correlation at $\bf{h} = 0$.
Caution when including nuggect effect in the variogram model as measurement error, mixing populations cause apparent nugget effect

This exercise demonstrates the semivariogram calculation with GeostatsPy. The steps include:

1. generate a 2D model with sequential Gaussian simulation
2. sample from the simulation
3. calculate and visualize experimental semivariograms

#### Variogram Modeling

Spatial continuity can be modeled with nested, positive definate variogram structures:

\begin{equation}
\Gamma_x(\bf{h}) = \sum_{i=1}^{nst} \gamma_i(\bf{h})
\end{equation}

where $\Gamma_x(\bf{h})$ is the nested variogram model resulting from the summation of $nst$ nested variograms  $\gamma_i(\bf{h})$.

The types of structure commonly applied include:

* spherical

* exponential

* gaussian

* nugget

Other less common models include:

* hole effect

* dampenned hole effect

* power law

these will not be covered here.

Each one of these variogram structures, $\gamma_i(\bf{h})$, is based on a geometric anisotropy model parameterized by the orientation and range in the major and minor directions.  In 2D this is simply an azimuth and ranges, $azi$, $a_{maj}$ and $a_{min}$. Note, the range in the minor direction (orthogonal to the major direction).

The geometric anisotropy model assumes that the range in all off-diagonal directions is based on an ellipse with the major and minor axes alligned with and set to the major and minor for the variogram.

\begin{equation}
\bf{h}_i = \sqrt{\left(\frac{r_{maj}}{a_{maj_i}}\right)^2 + \left(\frac{r_{maj}}{a_{maj_i}}\right)^2}
\end{equation}

Therefore, if we know the major direction, range in major and minor directions, we may completely describe each nested componnent of the complete spatial continuity of the variable of interest, $i = 1,\dots,nst$.

Some comments on modeling nested variograms:

* we can capture nugget, short and long range continuity structures

* we rely on the geometric anisotropy model, so all structures must inform the same level of contribution (porportion of the sill) in all directions.

* the geometric anisotropy model is based on azimuth of the major direction of continuity, range in the major direction and range in the minor direction (orthogonal to the major direction).  The range is interpolated between the major and minor azimuths with a ellipse model

* we can vary the type of variogram, direction or azimuth of the major direction, and major and minor ranges by structure

In this workflow we will explore methods to:

1. detect directionality from a spatial dataset
2. calculate the directional variograms in the major and minor directions
3. build a consistent 2D model fit to the major and minor directions

#### Objective

In the PGE 383: Stochastic Subsurface Modeling class I want to provide hands-on experience with building subsurface modeling workflows. Python provides an excellent vehicle to accomplish this. I have coded a package called GeostatsPy with GSLIB: Geostatistical Library (Deutsch and Journel, 1998) functionality that provides basic building blocks for building subsurface modeling workflows.

The objective is to remove the hurdles of subsurface modeling workflow construction by providing building blocks and sufficient examples. This is not a coding class per se, but we need the ability to 'script' workflows working with numerical methods.

#### Getting Started

Here's the steps to get setup in Python with the GeostatsPy package:

1. Install Anaconda 3 on your machine (https://www.anaconda.com/download/).
2. From Anaconda Navigator (within Anaconda3 group), go to the environment tab, click on base (root) green arrow and open a terminal.
3. In the terminal type: pip install geostatspy.
4. Open Jupyter and in the top block get started by copy and pasting the code block below from this Jupyter Notebook to start using the geostatspy functionality.

You will need to copy the data file to your working directory.  They are available here:

* Tabular data - sample_data.csv at https://git.io/fh4gm.

There are exampled below with these functions. You can go here to see a list of the available functions, https://git.io/fh4eX, other example workflows and source code.

#### Load the required libraries

The following code loads the required libraries.

```python
import geostatspy.GSLIB as GSLIB                       # GSLIB utilies, visualization and wrapper
import geostatspy.geostats as geostats                 # GSLIB methods convert to Python
```

We will also need some standard packages. These should have been installed with Anaconda 3.

```python
%matplotlib inline
import os                                               # to set current working directory
import sys                                              # supress output to screen for interactive variogram modeling
import io
import numpy as np                                      # arrays and matrix math
import pandas as pd                                     # DataFrames
import matplotlib.pyplot as plt                         # plotting
from matplotlib.pyplot import cm                        # color maps
from ipywidgets import interactive                      # widgets and interactivity
from ipywidgets import widgets
from ipywidgets import Layout
from ipywidgets import Label
from ipywidgets import VBox, HBox
from matplotlib.patches import Ellipse                  # plot an ellipse
```

If you get a package import error, you may have to first install some of these packages. This can usually be accomplished by opening up a command window on Windows and then typing 'python -m pip install [package-name]'. More assistance is available with the respective package docs.

#### Set the working directory

I always like to do this so I don't lose files and to simplify subsequent read and writes (avoid including the full address each time).  Also, in this case make sure to place the required (see above) GSLIB executables in this directory or a location identified in the environmental variable *Path*.

```python
os.chdir("c:/PGE383")                                   # set the working directory
```

#### Loading Tabular Data

Here's the command to load our comma delimited data file in to a Pandas' DataFrame object.

```python
df = pd.read_csv("sample_data_MV_biased.csv")                     # read a .csv file in as a DataFrame
#print(df.iloc[0:5,:])                                  # display first 4 samples in the table as a preview
df.head()                                               # we could also use this command for a table preview
```

      Index
      X
      Y
      Facies
      Porosity
      Perm
      AI

      0
      259
      190
      19
      0
      0.041122
      0.120862
      6367.442357

      1
      501
      270
      49
      0
      0.043853
      0.094627
      6906.176965

      2
      273
      270
      219
      0
      0.062169
      0.408779
      6244.965113

      3
      159
      380
      259
      0
      0.064295
      0.322764
      7113.679894

      4
      96
      870
      449
      0
      0.065537
      0.672005
      5672.957304

We will work with all facies pooled together. I wanted to simplify this workflow and focus more on spatial continuity direction detection. Finally, by not using facies we do have more samples to support our statistical inference. Most often facies are essential in the subsurface model. Don't worry we will check if this is reasonable in a bit.

You are welcome to repeat this workflow on a by-facies basis.  The following code could be used to build DataFrames ('df_sand' and 'df_shale') for each facies.

```p
df_sand = pd.DataFrame.copy(df[df['Facies'] == 1]).reset_index()  # copy only 'Facies' = sand records
df_shale = pd.DataFrame.copy(df[df['Facies'] == 0]).reset_index() # copy only 'Facies' = shale records
```

Let's look at summary statistics for all facies combined:

```python
df.describe().transpose()                          # summary table of sand only DataFrame statistics
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      Index
      367.0
      292.926431
      169.167110
      0.000000
      150.000000
      295.000000
      440.000000
      586.000000

      X
      367.0
      500.108992
      289.978309
      0.000000
      240.000000
      500.000000
      765.000000
      990.000000

      Y
      367.0
      520.403270
      277.752407
      9.000000
      269.000000
      539.000000
      769.000000
      999.000000

      Facies
      367.0
      0.596730
      0.491224
      0.000000
      0.000000
      1.000000
      1.000000
      1.000000

      Porosity
      367.0
      0.126874
      0.030546
      0.041122
      0.103398
      0.125616
      0.148323
      0.210258

      Perm
      367.0
      83.313352
      224.350475
      0.094627
      2.288119
      10.294907
      50.219914
      1991.097723

      AI
      367.0
      4790.458884
      975.582304
      1981.177309
      4108.843701
      4712.673154
      5466.957467
      7561.250336

Let's transform the porosity and permeaiblity data to standard normal (mean = 0.0, standard deviation = 1.0, Gaussian shape). This is required for sequential Gaussian simulation (common target for our variogram models) and the Gaussian transform assists with outliers and provides more interpretable variograms.

Let's look at the inputs for the GeostatsPy nscore program.  Note the output include an ndarray with the transformed values (in the same order as the input data in Dataframe 'df' and column 'vcol'), and the transformation table in original values and also in normal score values.

```python
geostats.nscore                                         # see the input parameters required by the nscore function
```

    <function geostatspy.geostats.nscore(df, vcol, wcol=None, ismooth=False, dfsmooth=None, smcol=0, smwcol=0)>

The following command will transform the Porosity and Permeabilty to standard normal.

```python
#Transform to Gaussian by Facies
df['NPor'], tvPor, tnsPor = geostats.nscore(df, 'Porosity') # nscore transform for all facies porosity
df['NPerm'], tvPermSand, tnsPermSand = geostats.nscore(df, 'Perm')  # nscore transform for all facies permeability
```

Let's look at the updated DataFrame to make sure that we now have the normal score porosity and permeability.

```python
df.head()                                               # preview sand DataFrame with nscore transforms
```

      Index
      X
      Y
      Facies
      Porosity
      Perm
      AI
      NPor
      NPerm

      0
      259
      190
      19
      0
      0.041122
      0.120862
      6367.442357
      -2.671293
      -2.644781

      1
      501
      270
      49
      0
      0.043853
      0.094627
      6906.176965
      -2.644781
      -2.775560

      2
      273
      270
      219
      0
      0.062169
      0.408779
      6244.965113
      -2.467028
      -1.789282

      3
      159
      380
      259
      0
      0.064295
      0.322764
      7113.679894
      -2.344090
      -1.945032

      4
      96
      870
      449
      0
      0.065537
      0.672005
      5672.957304
      -2.248832
      -1.316486

That looks good! One way to check is to see if the relative magnitudes of the normal score transformed values match the original values.  e.g. that the normal score transform of 0.10 porosity normal score is less than the normal score transform of 0.14 porsity.  Also, the normal score transform of values close to the mean value should be close to 0.0

Let's also check the original and transformed sand and shale porosity distributions.

```python
plt.subplot(221)                                        # plot original sand and shale porosity histograms
plt.hist(df['Porosity'], facecolor='red',bins=np.linspace(0.0,0.25,1000),histtype="stepfilled",alpha=0.2,density=True,cumulative=True,edgecolor='black',label='Original')
plt.xlim([0.05,0.25]); plt.ylim([0,1.0])
plt.xlabel('Porosity (fraction)'); plt.ylabel('Frequency'); plt.title('Porosity')
plt.legend(loc='upper left')
plt.grid(True)

plt.subplot(222)
plt.hist(df['NPor'], facecolor='blue',bins=np.linspace(-3.0,3.0,1000),histtype="stepfilled",alpha=0.2,density=True,cumulative=True,edgecolor='black',label = 'Trans')
plt.xlim([-3.0,3.0]); plt.ylim([0,1.0])
plt.xlabel('Porosity (fraction)'); plt.ylabel('Frequency'); plt.title('Nscore Porosity')
plt.legend(loc='upper left')
plt.grid(True)

plt.subplot(223)                                        # plot nscore transformed sand and shale histograms
plt.hist(df['Perm'], facecolor='red',bins=np.linspace(0.0,1000.0,100000),histtype="stepfilled",alpha=0.2,density=True,cumulative=True,edgecolor='black',label='Original')
plt.xlim([0.0,1000.0]); plt.ylim([0,1.0])
plt.xlabel('Porosity (fraction)'); plt.ylabel('Frequency'); plt.title('Permeability')
plt.legend(loc='upper left')
plt.grid(True)

plt.subplot(224)                                        # plot nscore transformed sand and shale histograms
plt.hist(df['NPerm'], facecolor='blue',bins=np.linspace(-3.0,3.0,100000),histtype="stepfilled",alpha=0.2,density=True,cumulative=True,edgecolor='black',label = 'Trans')
plt.xlim([-3.0,3.0]); plt.ylim([0,1.0])
plt.xlabel('Permeability (mD)'); plt.ylabel('Frequency'); plt.title('Nscore Permeability')
plt.legend(loc='upper left')
plt.grid(True)

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=2.2, wspace=0.2, hspace=0.3)
plt.show()
```

The normal score transform has correctly transformed the porosity and permeability to standard normal.

#### Method \#1: Ocular Inspection of Posted Data

Data visualization is very useful to detect patterns. Our brains are very good at pattern detection. I promote quantitative methods and recognize issues with cognitive bias, but it is important to recognize the value is expert intepretation based on data visualization.

* This data visualization will also be important to assist with parameter selection for the quantitative methods later.

Let's plot the location maps of normal score transforms of porosity and permeability for all facies. We will also include a cross plot of the nscore permeability vs. porosity colored by facies to aid with comparison in spatial features between the porosity and permeability data.

```python
cmap = plt.cm.plasma                    # color map
plt.subplot(131)
GSLIB.locmap_st(df,'X','Y','NPor',0,1000,0,1000,-3,3,'Nscore Porosity - All Facies','X (m)','Y (m)','Nscore Porosity',cmap)

plt.subplot(132)
GSLIB.locmap_st(df,'X','Y','NPerm',0,1000,0,1000,-3,3,'Nscore Permeability - All Facies','X (m)','Y (m)','Nscore Permeability',cmap)

plt.subplot(133)
facies = df['Facies'].values +0.01
plt.scatter(df['NPor'],df['NPerm'],c = facies,edgecolor = 'black',cmap = plt.cm.inferno)
#plt.plot([-3,3],[-3,3],color = 'black')
plt.xlabel(r'Nscore Porosity')
plt.ylabel(r'Nscore Permeability')
plt.title('Nscore Permeability vs. Porosity')
plt.xlim([-3,3])
plt.ylim([-3,3])
plt.grid(True)

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=0.8, wspace=0.5, hspace=0.3)
plt.show()
```

What do you see?  Here's my observations:

* there is a high degree of spatial agreement between porosity and permeability, this is supported by the high correlation evident in the cross plot.
* there are no discontinuities that could suggest that facies represent a distinct change, rather the porosity and permeability seem continuous and the assigned facies are a truncation of their continous behavoir, we doing 'ok' with no facies
* suspect a 045 azimuth major direction of continuity (up - right)
* there may be cycles in the 135 azimuth
* there will not likely be a nugget effect, but there is an hint of some short scale discontinuity?

**Do you agree?** If you have a different observations, drop me a line at mpyrcz@austin.utexas.edu and I'll add to this lesson with credit.

#### Method \#2: Variogram Maps

Let's try out variogram maps.

* I realized that I had not coded variogram maps in Python, so I just did it and added it to GeostatsPy.  I have added if here temporarily to support my students completely their assignments. I'm teaching this week at a company and it is safer not to update the package until I have time to test it.

The inputs include:

* **input data** - DataFrame, 'df', and columns for x, y and property of interest, 'x', 'y' and 'vcol',
* **variogram map parameters** - number of cells in each direction to search, 'nxlag', nylag', the cell size / lag distance, 'dxlag' and 'dylag'''
* **search** - the minimum number of pairs reuqired to assign a result, 'mnpairs'
* **normalization** - 1 for standardize variance to 1.0 and 0 for not

The output is a 2D ndarray with the variogram map and the number of pairs.

```python
vmap, npmap = geostats.varmapv(df,'X','Y','NPor',tmin=-999,tmax=999,nxlag=11,nylag=11,dxlag=50,dylag=50,minnp=1,isill=1)

plt.subplot(121)
GSLIB.pixelplt_st(vmap,-575,575,-575,575,50.0,0,1.6,'Nscore Porosity Variogram Map','X(m)','Y(m)','Nscore Variogram',cmap)

#plt.subplot(122)
#GSLIB.pixelplt_st(npmap,-575,575,-575,575,50.0,0,900,'Nscore Porosity Variogram Map Number of Pairs','X(m)','Y(m)','Number of Pairs',cmap)

plt.subplots_adjust(left=0.0, bottom=0.0, right=3.0, top=1.2, wspace=0.2, hspace=0.2)
plt.show()

print('The shape of the output is ' + str(vmap.shape))
```

Notice that the output ndarray is 23, 23 cells? We asked for the number of cells to extend in each direction, 11 and 11 in x and y.  The map has an origin (zero distance) cell in the middle and extends 11 in both positive and negative directions.  So we have 2*$nx$ + 1, 2*$ny$ + 1 cells in the resulting variogram map and the $xmin = -1 * (nx * x_{cellsize} + \frac{1}{2} x_{cellsize})$ and the $xmax = nx * x_{cellsize} + \frac{1}{2} x_{cellsize}$

What do you think of this variogram map? These are my observations:

* major continuity direction is at azimuth 045
* there is a high degree of geometric anisotropy
* there is cyclicity in the 135 direction
* there may be some cyclicity in the 045 direction

From this variogram map we can immediately see directionality in our spatial data.

#### Method \#3: Experimental Variograms

Another method for exploring spatial data directionality is the calculation of multiple experimental variograms for a variety of directions.

We can use the location maps to help determine good variogram calculation parameters.

```p
tmin = -9999.; tmax = 9999.;
lag_dist = 100.0; lag_tol = 50.0; nlag = 7; bandh = 9999.9; azi = azi; atol = 22.5; isill = 1
```
* **tmin**, **tmax** are trimming limits - set to have no impact, no need to filter the data
* **lag_dist**, **lag_tol** are the lag distance, lag tolerance - set based on the common data spacing (100m) and tolerance as 100% of lag distance for additonal smoothing
* **nlag** is number of lags - set to extend just past 50 of the data extent
* **bandh** is the horizontal band width - set to have no effect
* **azi** is the azimuth -  it has not effect since we set atol, the azimuth tolerance, to 90.0
* **isill** is a boolean to standardize the distribution to a variance of 1 - it has no effect since the nscore transform sets the variance to 1.0

```python
tmin = -9999.; tmax = 9999.                             # no trimming
lag_dist = 100.0; lag_tol = 100.0; nlag = 8;            # maximum lag is 700m and tolerance > 1/2 lag distance for smoothing
bandh = 9999.9; atol = 22.5                             # no bandwidth, directional variograms
isill = 1                                               # standardize sill
azi_mat = [0,22.5,45,67.5,90,112.5,135,157.5]           # directions in azimuth to consider
```

Let's try running these variograms and visualizing them on separate plots.  I'll demonstrate a method to promgramtically loop over each direction for efficiency (and code brevity).

* we have the direction in the list called 'azi_mat'
* we use the command:
```p
for iazi in range(0,len(azi_mat)):
```
to loop over all the elements in the list with index 'iazi'

* we run the variogram calculation and store the reuslts in 2D arrays, iy is direction, ix is the lag
* we use subplots with the 'iazi' index to add each plot

```p
    plt.subplot(4,2,iazi+1)
```
we add one because the plot index must be $1,\ldots,n$, but arrays / list index as $0,\ldots,n-1$ in Python.

```python
# Arrays to store the results
lag = np.zeros((len(azi_mat),nlag+2)); gamma = np.zeros((len(azi_mat),nlag+2)); npp = np.zeros((len(azi_mat),nlag+2));

for iazi in range(0,len(azi_mat)):                      # Loop over all directions
    lag[iazi,:], gamma[iazi,:], npp[iazi,:] = geostats.gamv(df,"X","Y","NPor",tmin,tmax,lag_dist,lag_tol,nlag,azi_mat[iazi],atol,bandh,isill)
    plt.subplot(4,2,iazi+1)
    plt.plot(lag[iazi,:],gamma[iazi,:],'x',color = 'black',label = 'Azimuth ' +str(azi_mat[iazi]))
    plt.plot([0,2000],[1.0,1.0],color = 'black')
    plt.xlabel(r'Lag Distance $\bf(h)$, (m)')
    plt.ylabel(r'$\gamma \bf(h)$')
    plt.title('Directional NSCORE Porosity Variogram')
    plt.xlim([0,700])
    plt.ylim([0,1.8])
    plt.legend(loc='upper left')
    plt.grid(True)

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=4.2, wspace=0.2, hspace=0.3)
plt.show()
```

The directional variograms provide a very clear image of directionality. The ranges vary from 300m, 500m to zonal anisotropy, to 500m, 350m, 280m, 250m and finally 280m.  We are observing the actually spatial continuity ellipse by exploring a variety of directions!

* We can observe that Azimuth 045 is the major direction and Azimuth 135 is the minor direction.

This is a very powerful tool for exploring directionality in spatial datasets.

#### Variogram Modeling

From above we can identify 045 as the major direction and 135 as the minor direction. We must build a single 2D nested variogram model based on the experiential variograms in each 045 and 135 directions.  First, let's look at these two experimental variograms again.

Below edit the following code to select the major and minor directions from the plots above.

```python
# Select the plot above for the major and minor
imajor = 2
iminor = 6
```

Note the first plot is 0 and then step through $0, 1, \ldots, n$ where $n$ is the number of directional variograms calculated above.  For the provided example we selected the plots, 3rd (045 Azimuth) for the major direction and the 7th (135 Azimuth) for the minor direction.

```python
# Select the plot above for the major and minor
imajor = 2
iminor = 6

print('Major direction is ' + str(azi_mat[imajor]) + ' azimuth.')
print('Minor direction is ' + str(azi_mat[iminor]) + ' azimuth.')

if not abs(azi_mat[imajor] - azi_mat[iminor]) == 90.0:
    print('Major and minor directions must be orthogonal to each other.')
    sys.exit()

plt.subplot(1,2,1)
plt.plot(lag[imajor,:],gamma[imajor,:],'x',color = 'black',label = 'Azimuth ' + str(azi_mat[imajor]))
plt.plot([0,2000],[1.0,1.0],color = 'black')
plt.xlabel(r'Lag Distance $\bf(h)$, (m)')
plt.ylabel(r'$\gamma \bf(h)$')
plt.title('Directional NSCORE Porosity Variogram - Major ' + str(azi_mat[imajor]) + ' Azimuth')
plt.xlim([0,700])
plt.ylim([0,1.8])
plt.legend(loc='upper left')

plt.subplot(1,2,2)
plt.plot(lag[iminor,:],gamma[iminor,:],'x',color = 'black',label = 'Azimuth ' +str(azi_mat[iminor]))
plt.plot([0,2000],[1.0,1.0],color = 'black')
plt.xlabel(r'Lag Distance $\bf(h)$, (m)')
plt.ylabel(r'$\gamma \bf(h)$')
plt.title('Directional NSCORE Porosity Variogram  - Minor ' + str(azi_mat[iminor]) + ' Azimuth')
plt.xlim([0,700])
plt.ylim([0,1.8])
plt.legend(loc='upper left')

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.2, top=1.6, wspace=0.2, hspace=0.3)
plt.show()
```

We will assume these models will be applied in sequential Gaussian simulation; therefore, we must model to the sill (or the global distribution will not be reproduced over the realizations).  We will also not use cyclicity for now, as we are just getting started.

* Let's build a reasonable model to the sill.

##### make_variogram function

We use the make_variogram function to make a variogram model

* a dictionary for compact storage of the variogram model parameters to pass into plotting (below), kriging and simulation methods

The variogram model parameter include:

* **nug** - nugget effect contribution to sill
* **nst** - number of nested structures (1 or 2)
* **it** - type for this nested structure (1 - spherical, 2 - exponential, 3 - Gaussian)
* **cc** - contribution of each nested structure (contributions + nugget must sum to the sill)
* **azi** - the azimuth for this nested structure of the major direction, the minor is orthogonal
* **hmaj** - the range for this nested structure in the major direction
* **hmin** - the range for this nested structure in the minor direction

We increment it, cc, azi, hmaj, and hmin for the 1st and 2nd structures

* for only 1 structure plus optional nugget, omit the 2nd structure parameters and they will default to $cc2 = 0$, no contribution to the model

Here's my model:

```p
nug = 0.0; nst = 2
it1 = 1; cc1 = 0.6; azi1 = 45; hmaj1 = 350; hmin1 = 350                # first structure
it2 = 1; cc2 = 0.4; azi2 = 45; hmaj2 = 9999.9; hmin2 = 400             # second structure
```

Some comments on our model:

* we model to the sill of 1.0, since we applied the normal score transform ($nug + cc1 + cc2 = 1.0$)

* we used 2 spherical structures to capture zontal anisotropy in the 045 azimuth

* since the experimental variogram exceeds the sill with trend or cyclicity we could have attempted trend modeling and then worked with the residual, but we will not do this for workflow brevity and simplicity

We input these model parameters to make a variogram model dictionary with the make_variogram function as follows:

```p
vario = GSLIB.make_variogram(nug,nst,it1,cc1,azi1,hmaj1,hmin1,it2,cc2,azi2,hmaj2,hmin2)
```

##### vmodel function

To plot the variogram we use the vmodel function to project the model in the major and minor directions

The inputs for vmodel are:

* **nlag** - the number of points along the variogram to calculate for the projection

* **xlag** - the size of a lag for the projection

* **azm** - the direction of the projection in azimuth (this is all we need since we are working in 2D)

* **vario** - the variogram model dictionary from the make_variogram function (above)

Note: this function is just for visualization by projecting the variogram model in a direction, so the convention is to use a very small **xlag** and large **nlag** for a high resolution display of the variogram model

The outputs from the vmodel program include:

* **index** - the lag number for the projection

* **lag distance** - the distance offset along the projection (the **h** in the variogram plot)

* **variogram** - the variogram value at the lag distance for the projection (the $\gamma$(**h**) in the variogram plot)

* **covariance function** - the covariance function at the lag distance for the projection (for the C(**h**) plot)

* **correlogram** - the correlogram at the lag distance for the projection (for the $\rho$(**h**) plot)

We have 2 structures and no nugget effect.  We needed the 2nd structure to capture the zonal anisotropy in the 045 direction.  Let's calculate the variogram model in these directions and plot them with the experimental variograms.

```python
nug = 0.0; nst = 2                                             # 2 nest structure variogram model parameters
it1 = 1; cc1 = 0.6; azi1 = azi_mat[imajor]; hmaj1 = 350; hmin1 = 350
it2 = 1; cc2 = 0.4; azi2 = azi_mat[imajor]; hmaj2 = 9999.9; hmin2 = 400

vario = GSLIB.make_variogram(nug,nst,it1,cc1,azi1,hmaj1,hmin1,it2,cc2,azi2,hmaj2,hmin2) # make model object
nlag = 70; xlag = 10;                                          # project the model in the 045 azimuth
index_maj,h_maj,gam_maj,cov_maj,ro_maj = geostats.vmodel(nlag,xlag,azi_mat[imajor],vario)                                                     # project the model in the 135 azimuth
index_min,h_min,gam_min,cov_min,ro_min = geostats.vmodel(nlag,xlag,azi_mat[iminor],vario)

plt.subplot(1,2,1)
plt.plot(lag[imajor,:],gamma[imajor,:],'x',color = 'black',label = 'Azimuth ' +str(azi_mat[imajor]))
plt.plot([0,2000],[1.0,1.0],color = 'black')
plt.plot(h_maj,gam_maj,color = 'black')
plt.xlabel(r'Lag Distance $\bf(h)$, (m)')
plt.ylabel(r'$\gamma \bf(h)$')
plt.title('Directional NSCORE Porosity Variogram - Major ' + str(azi_mat[imajor]) + ' Azimuth')
plt.xlim([0,700])
plt.ylim([0,1.8])
plt.legend(loc='upper left')

plt.subplot(1,2,2)
plt.plot(lag[iminor,:],gamma[iminor,:],'x',color = 'black',label = 'Azimuth ' +str(azi_mat[6]))
plt.plot([0,2000],[1.0,1.0],color = 'black')
plt.plot(h_min,gam_min,color = 'black')
plt.xlabel(r'Lag Distance $\bf(h)$, (m)')
plt.ylabel(r'$\gamma \bf(h)$')
plt.title('Directional NSCORE Porosity Variogram  - Minor ' + str(azi_mat[iminor]) + ' Azimuth')
plt.xlim([0,700])
plt.ylim([0,1.8])
plt.legend(loc='upper left')

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.2, top=1.6, wspace=0.2, hspace=0.3)
plt.show()
```

#### Interactive Variogram Modeling

The following code uses the ipywidgets and matplotlib packages to build a method for interactive variogram modeling.

```python
# interactive calculation of the sample set (control of source parametric distribution and number of samples)
l = widgets.Text(value='                              Variogram Modeling, Michael Pyrcz, Associate Professor, The University of Texas at Austin',layout=Layout(width='950px', height='30px'))
nug = widgets.FloatSlider(min = 0, max = 1.0, value = 0.0, step = 0.1, description = 'nug',orientation='vertical',layout=Layout(width='80px', height='200px'))
nug.style.handle_color = 'gray'
it1 = widgets.Dropdown(options=['Spherical', 'Exponential', 'Gaussian'],value='Spherical',
    description='Type1:',disabled=False,layout=Layout(width='190px', height='30px'))
c1 = widgets.FloatSlider(min=0.0, max = 1.0, value = 0.01, description = 'c1',orientation='vertical',layout=Layout(width='80px', height='200px'))
c1.style.handle_color = 'blue'
hmaj1 = widgets.FloatSlider(min=0.01, max = 10000.0, value = 0.01, step = 100.0, description = 'hmaj1',orientation='vertical',layout=Layout(width='80px', height='200px'))
hmaj1.style.handle_color = 'red'
hmin1 = widgets.FloatSlider(min = 0.01, max = 10000.0, value = 0.01, step = 100.0, description = 'hmin1',orientation='vertical',layout=Layout(width='80px', height='200px'))
hmin1.style.handle_color = 'green'

it2 = widgets.Dropdown(options=['Spherical', 'Exponential', 'Gaussian'],value='Spherical',
    description='Type2:',disabled=False,layout=Layout(width='190px', height='30px'))
c2 = widgets.FloatSlider(min=0.0, max = 1.0, value = 0.0, description = 'c2',orientation='vertical',layout=Layout(width='80px', height='200px'))
c2.style.handle_color = 'blue'
hmaj2 = widgets.FloatSlider(min=0.01, max = 10000.0, value = 0.01, step = 100.0, description = 'hmaj2',orientation='vertical',layout=Layout(width='80px', height='200px'))
hmaj2.style.handle_color = 'red'
hmin2 = widgets.FloatSlider(min = 0.01, max = 10000.0, value = 0.01, step = 100.0, description = 'hmin2',orientation='vertical',layout=Layout(width='80px', height='200px'))
hmin2.style.handle_color = 'green'

ui1 = widgets.HBox([nug,it1,c1,hmaj1,hmin1,it2,c2,hmaj2,hmin2],)                   # basic widget formatting
#ui2 = widgets.HBox([it2,c2,hmaj2,hmin2],)                   # basic widget formatting
ui = widgets.VBox([l,ui1],)

def convert_type(it):
    if it == 'Spherical':
        return 1
    elif it == 'Exponential':
        return 2
    else:
        return 3

def f_make(nug,it1,c1, hmaj1, hmin1, it2, c2, hmaj2, hmin2):                       # function to take parameters, make sample and plot
    text_trap = io.StringIO()
    sys.stdout = text_trap

    it1 = convert_type(it1); it2 = convert_type(it2)
    if c2 > 0.0:
        nst = 2
    else:
        nst = 1

    vario = GSLIB.make_variogram(nug,nst,it1,c1,azi_mat[imajor],hmaj1,hmin1,it2,c2,azi_mat[imajor],hmaj2,hmin2) # make model object
    nlag = 70; xlag = 10;
    index_maj,h_maj,gam_maj,cov_maj,ro_maj = geostats.vmodel(nlag,xlag,azi_mat[imajor],vario)   # project the model in the major azimuth                                                  # project the model in the 135 azimuth
    index_min,h_min,gam_min,cov_min,ro_min = geostats.vmodel(nlag,xlag,azi_mat[iminor],vario) # project the model in the minor azimuth

    plt.subplot(2,2,1)
    plt.plot(lag[imajor,:],gamma[imajor,:],'x',color = 'black',label = 'Azimuth ' +str(azi_mat[2]))
    plt.plot([0,2000],[1.0,1.0],color = 'black')
    plt.plot(h_maj,gam_maj,color = 'black')
    plt.xlabel(r'Lag Distance $\bf(h)$, (m)')
    plt.ylabel(r'$\gamma \bf(h)$')
    plt.title('Horizontal Major Directional NSCORE Porosity Variogram - Major ' + str(azi_mat[imajor]) + 'Azimuth')
    plt.xlim([0,700])
    plt.ylim([0,1.8])
    plt.legend(loc='upper left')

    plt.subplot(2,2,2)
    plt.plot(lag[iminor,:],gamma[iminor,:],'x',color = 'black',label = 'Azimuth ' +str(azi_mat[6]))
    plt.plot([0,2000],[1.0,1.0],color = 'black')
    plt.plot(h_min,gam_min,color = 'black')
    plt.xlabel(r'Lag Distance $\bf(h)$, (m)')
    plt.ylabel(r'$\gamma \bf(h)$')
    plt.title('Horizontal Minor Directional NSCORE Porosity Variogram - Minor ' + str(azi_mat[iminor]) + 'Azimuth')
    plt.xlim([0,700])
    plt.ylim([0,1.8])
    plt.legend(loc='upper left')

    plt.subplot(2,2,3)
    plt.plot([0,2000],[1.0,1.0],color = 'black')
    plt.plot(h_maj,gam_maj,color = 'black',label = 'Azimuth ' + str(azi_mat[imajor]))
    plt.plot(h_min,gam_min,color = 'black',label = 'Azimuth ' + str(azi_mat[iminor]))
    deltas = [22.5, 45, 67.5];
    ndelta = len(deltas); hd = np.zeros(ndelta); gamd = np.zeros(ndelta);
    color=iter(cm.plasma(np.linspace(0,1,ndelta)))
    for delta in deltas:
        index,hd,gamd,cov,ro = geostats.vmodel(nlag,xlag,azi_mat[imajor]+delta,vario);
        c=next(color)
        plt.plot(hd,gamd,color = c,label = 'Azimuth ' + str(azi_mat[imajor]+delta))
    plt.xlabel(r'Lag Distance $\bf(h)$, (m)')
    plt.ylabel(r'$\gamma \bf(h)$')
    plt.title('Interpolated Azimuth NSCORE Porosity Variogram Models')
    plt.xlim([0,700])
    plt.ylim([0,1.8])
    plt.legend(loc='upper left')

    plt.subplot(2,2,4)
    GSLIB.locmap_st(df,'X','Y','NPor',0,1000,0,1000,-3,3,'Nscore Porosity - All Facies','X (m)','Y (m)','Nscore Porosity',cmap)

#     plt.scatter(x1,y1,color = 'blue', edgecolors = 'black', s = sk_weights[0,0]*1000)
#     plt.scatter(x2,y2,color = 'red', edgecolors = 'black', s = sk_weights[0,1]*1000)
#     plt.scatter(x3,y3,color = 'green', edgecolors = 'black', s = sk_weights[0,2]*1000)
#     scatter = plt.scatter(500,500,color = 'gray', edgecolors = 'black', s = (1-sk_std)*1000)
    ax = plt.gca()
    plt.xlabel('X(m)'); plt.ylabel('Y(m)')
    plt.title('Data and Variogram Ranges for Each Structure')
    plt.xlim([0,1000])
    plt.ylim([0,1000])

    ellipse1 = Ellipse((500, 500),width=hmin1*2.0,height=hmaj1*2.0,angle = 360-azi_mat[imajor],facecolor='orange',alpha = 0.3,edgecolor='black',label='Structure 1')
    ellipse2 = Ellipse((500, 500),width=hmin2*2.0,height=hmaj2*2.0,angle = 360-azi_mat[imajor],facecolor='red',alpha = 0.3,edgecolor='black',label='Structure 2')
    ax = plt.gca()
    ax.add_patch(ellipse1); ax.add_patch(ellipse2)
    plt.legend()

    plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=2.1, wspace=0.3, hspace=0.3)
    plt.show()

# connect the function to make the samples and plot to the widgets
interactive_plot = widgets.interactive_output(f_make, {'nug':nug, 'it1':it1,'c1':c1, 'hmaj1':hmaj1, 'hmin1':hmin1, 'it2':it2, 'c2':c2, 'hmaj2':hmaj2, 'hmin2':hmin2})
interactive_plot.clear_output(wait = True)               # reduce flickering by delaying plot updating
```

### Interactive Nested Variogram Modeling Demostration

* select the nested structures and their types, contributions and major and minor ranges

#### Michael Pyrcz, Associate Professor, University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1) | [GeostatsPy](https://github.com/GeostatsGuy/GeostatsPy)

### The Problem

Fit a positive definite variogram model based on the addition of multiple structures each describing spatial components of the feature variance

* **nug**: nugget effect

* **c1 / c2**: contributions of the sill

* **hmaj1 / hmaj2**: range in the major direction

* **hmin1 / hmin2**: range in the minor direction

```python
display(ui, interactive_plot)                            # display the interactive plot
```

    VBox(children=(Text(value='                              Variogram Modeling, Michael Pyrcz, Associate Professo…

    Output(outputs=({'output_type': 'display_data', 'data': {'text/plain': '
', 'i…

#### Comments

This was a basic demonstration of vairogram modeling for spatial continuity analysis. Much more could be done, I have other demonstrations on the basics of working with DataFrames, ndarrays, univariate statistics, plotting data, declustering, data transformations and many other workflows available at https://github.com/GeostatsGuy/PythonNumericalDemos and https://github.com/GeostatsGuy/GeostatsPy.

#### The Author:

### Michael Pyrcz, Associate Professor, University of Texas at Austin
*Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions*

With over 17 years of experience in subsurface consulting, research and development, Michael has returned to academia driven by his passion for teaching and enthusiasm for enhancing engineers' and geoscientists' impact in subsurface resource development.

For more about Michael check out these links:

#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

#### Want to Work Together?

I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.

* Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I'd be happy to drop by and work with you!

* Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!

* I can be reached at mpyrcz@austin.utexas.edu.

I'm always happy to discuss,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin

#### More Resources Available at: [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

```python

```

```python

```


## 5. Linear ensemble filtering Lorenz-96 problem with localization

In this notebook, we apply the stochastic ensemble Kalman filter to the Lorenz-96 problem.

To regularize the inference problem, we use a localization radius `L` to cut-off long-range correlations and improve the conditioning of the covariance matrix. We refers readers to Asch et al. [2] for further details.

[1] Evensen, G., 1994. Sequential data assimilation with a nonlinear quasi‐geostrophic model using Monte Carlo methods to forecast error statistics. Journal of Geophysical Research: Oceans, 99(C5), pp.10143-10162.

[2] Asch, M., Bocquet, M. and Nodet, M., 2016. Data assimilation: methods, algorithms, and applications. Society for Industrial and Applied Mathematics.

[3] Bishop, C.H., Etherton, B.J. and Majumdar, S.J., 2001. Adaptive sampling with the ensemble transform Kalman filter. Part I: Theoretical aspects. Monthly weather review, 129(3), pp.420-436.

[4] Lorenz, E.N., 1963. Deterministic nonperiodic flow. Journal of atmospheric sciences, 20(2), pp.130-141.

[5] Spantini, A., Baptista, R. and Marzouk, Y., 2019. Coupling techniques for nonlinear ensemble filtering. arXiv preprint arXiv:1907.00389.

[6] Evensen, G. (2003). The ensemble Kalman filter: Theoretical formulation and practical implementation. Ocean dynamics, 53(4), 343-367.

[7] Tsitouras, C. (2011). Runge–Kutta pairs of order 5 (4) satisfying only the first column simplifying assumption. Computers & Mathematics with Applications, 62(2), 770-775.

```julia
using Revise
using LinearAlgebra
using AdaptiveTransportMap
using Statistics
using Distributions
```

    ┌ Info: Precompiling AdaptiveTransportMap [bdf749b0-1400-4207-80d3-e689c0e3f03d]
    └ @ Base loading.jl:1278
    ┌ Warning: Type annotations on keyword arguments not currently supported in recipes. Type information has been discarded
    └ @ RecipesBase ~/.julia/packages/RecipesBase/92zOw/src/RecipesBase.jl:116
    ┌ Warning: Type annotations on keyword arguments not currently supported in recipes. Type information has been discarded
    └ @ RecipesBase ~/.julia/packages/RecipesBase/92zOw/src/RecipesBase.jl:116

Load some packages to make nice figures

```julia
using Plots
default(tickfont = font("CMU Serif", 9),
        titlefont = font("CMU Serif", 14),
        guidefont = font("CMU Serif", 12),
        legendfont = font("CMU Serif", 10),
        grid = false)
pyplot()

using LaTeXStrings
PyPlot.rc("text", usetex = "true")
PyPlot.rc("font", family = "CMU Serif")
# gr()
using ColorSchemes
```

The Lorenz-96 model  is a famous problem used in data assimilation and weather prediction. It was derived from first principles as a one-dimensional model for the response of the mid-latitude atmosphere to forcing input. For certain forcing input, it can exhibit a chaotic behavior: sensitivity to initial conditions, strong mixing... The forty-dimensional state $\boldsymbol{x} = (x_1, \ldots, x_{40})$ at time $t$ is governed by the following set of ordinary differential equations:
\begin{equation}
\label{eqn:lorenz96}
\frac{\mathrm{d} x_i}{\mathrm{d} t} = (x_{i+1} - x_{i-2}) x_{i-1} - x_{i} + F, \quad \mbox{for } i = 1, \ldots, 40,
\end{equation}
with periodic boundary conditions, setting the forcing input $F=8.0$ leads to chaos. We reproduce the hard configuration case of Spantini et al. [5]. We integrate this system of ODEs with time step $\Delta t_{dyn} = 0.05$. We observe the state with $\Delta t_{obs}=0.4$, every two components ($x_1, x_3, x_5, \ldots, x_{39})$. The large time interval between two assimilation makes the problem particularly challenging and enhances the non-Gaussianity. The initial distribution $\pi_{\mathsf{X}_0}$ is the standard Gaussian. We assume that there is no process noise. The measurement noise has a Gaussian distribution with zero mean and covariance $\theta^2\boldsymbol{I}_{20}$ where $\theta^2 = 0.5$.

### Simple twin-experiment

Define the dimension of the state and observation vectors

```julia
Nx = 40
Ny = 20
```

    20

Define the time steps $\Delta t_{dyn}, \Delta t_{obs}$  of the dynamical and observation models. Observations from the truth are assimilated every $\Delta t_{obs}$.

```julia
Δtdyn = 0.05
Δtobs = 0.4
```

    0.4

Define the time span of interest

```julia
t0 = 0.0
tf = 100.0
Tf = ceil(Int64, (tf-t0)/Δtobs)
```

    250

Define the properties of the initial condition

```julia
m0 = zeros(Nx)
C0 = Matrix(1.0*I, Nx, Nx);
```

We construct the state-space representation `F` of the system composed of the deterministic part of the dynamical and observation models.

The dynamical model is provided by the right hand side of the ODE to solve. For a system of ODEs, we will prefer an in-place syntax `f(du, u, p, t)`, where `p` are parameters of the model.
We rely on `OrdinaryDiffEq` to integrate the dynamical system with the Tsitouras 5/4 Runge-Kutta method adaptive time marching [7].

We assume that the state is fully observable, i.e. $h(t,x) = x$.

```julia
h(t,x) = x[2:2:end]
h(t,x,idx) = x[idx]

F = StateSpace(lorenz96!, h)
```

    StateSpace(AdaptiveTransportMap.lorenz96!, h)

`ϵx` defines the additive process noise applied between the forecast step and the analysis step. The process noise is applied before to sample form the likelihood.

`ϵy` defines the additive observation noise.

We assume that these noises have Gaussian distribution.

```julia
σx = 1e-16
σy = 0.5

ϵx = AdditiveInflation(Nx, zeros(Nx), σx)
ϵy = AdditiveInflation(Ny, zeros(Ny), σy)
```

    AdditiveInflation(20, [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.25 0.0 … 0.0 0.0; 0.0 0.25 … 0.0 0.0; … ; 0.0 0.0 … 0.25 0.0; 0.0 0.0 … 0.0 0.25], [0.5 0.0 … 0.0 0.0; 0.0 0.5 … 0.0 0.0; … ; 0.0 0.0 … 0.5 0.0; 0.0 0.0 … 0.0 0.5])

```julia
model = Model(Nx, Ny, Δtdyn, Δtobs, ϵx, ϵy, m0, C0, 0, 0, 0, F);
```

Set initial condition of the true system

```julia
x0 = model.m0 + sqrt(model.C0)*randn(Nx);
```

Run dynamics and generate data

```julia
@time data = generate_lorenz96(model, x0, Tf);
```

      0.005344 seconds (19.17 k allocations: 3.892 MiB)

Define a stochastic ensemble Kalman filter

```julia
senkf = StochEnKF(model.ϵy, model.Δtdyn, model.Δtobs)
```

    Stochastic EnKF  with filtered = false

Define a ensemble transform Kalman filter

```julia
etkf = ETKF(model.ϵy, model.Δtdyn, model.Δtobs, 20*model.Δtobs)
```

    ETKF  with filtered = false

Initialize the ensemble matrix `X` $\in \mathbb{R}^{(N_y + N_x) \times N_e}$.

```julia
Ne = 100 #ensemble size
X = zeros(model.Ny + model.Nx, Ne)

# Generate the initial conditions for the state.
X[model.Ny+1:model.Ny+model.Nx,:] .= sqrt(model.C0)*randn(model.Nx, Ne) .+ model.m0;
```

Apply the sequential filter over the time window

The function `seqassim` provides a friendly API to experiment with the different ensemble filters, the tuning of the different inflation parameters...

Without localization

```julia
@time Xsenkf = seqassim(F, data, Tf, model.ϵx, senkf, deepcopy(X), model.Ny, model.Nx, t0);
```

      0.984651 seconds (2.00 M allocations: 440.674 MiB, 12.74% gc time)

Localization can easily be added as an additional regularization of the analysis step as follows....

```julia
@time Xsenkfloc = seqassim(F, data, Tf, model.ϵx, senkf, deepcopy(X), model.Ny, model.Nx, t0);
```

      0.922576 seconds (2.00 M allocations: 440.674 MiB, 8.93% gc time)

`mean_hist` stacked together the ensemble mean over the assimilation window.

```julia
# Plot the first component of the state over time
nb = 1
ne = 100#size(Xsenkf,1)-1
Δ = 1
plt = plot(xlim = (-Inf, Inf), ylim = (-Inf, Inf), xlabel = L"t", ylabel = L"x_1")
plot!(plt, data.tt[nb:Δ:ne], data.xt[1,nb:Δ:ne], linewidth =  3, color = :teal, label = "True")
plot!(plt, data.tt[nb:Δ:ne], mean_hist(Xsenkf)[1,1+nb:Δ:1+ne], linewidth = 3, grid = false,
     color = :orangered2, linestyle = :dash, label = "sEnKF")
scatter!(plt, data.tt[nb:Δ:ne], data.yt[1,nb:Δ:ne], linewidth = 3, color = :grey, markersize = 5, alpha = 0.5, label = "Observation")
plt
```

```julia
nb = 100
ne = size(Xsenkf,1)-1
Δ = 5

plt = plot(layout = grid(3,1))
contourf!(plt[1,1], data.xt[:,nb:Δ:ne], color = :gnuplot2)
contourf!(plt[2,1], mean_hist(Xsenkf)[:,1+nb:Δ:1+ne], color = :plasma)
contourf!(plt[3,1], mean_hist(Xsenkf)[:,1+nb:Δ:1+ne] .- data.xt[:,nb:Δ:ne], color = :plasma)
```

```julia
# Plot the different component of the state over time
nb = 1
ne = size(Xsenkf,1)-1
Δ = 1
plt = plot(layout = grid(3,1), xlim = (-Inf, Inf), ylim = (-Inf, Inf), xlabel = L"t",
           size = (900, 1000))

for i =1:3
    plot!(plt[i,1], data.tt[nb:Δ:ne], data.xt[i,nb:Δ:ne], linewidth =  2, color = :teal,
          ylabel = latexstring("x_"*string(i)), legend = (i == 1), label = "True")
    plot!(plt[i,1], data.tt[nb:Δ:ne], mean_hist(Xsenkf)[i,1+nb:Δ:1+ne], linewidth = 2, grid = false,
          color = :orangered2, linestyle = :dash, label = "sEnKF")
    scatter!(plt[i,1], data.tt[nb:Δ:ne], data.yt[i,nb:Δ:ne], linewidth = 3, color = :grey,
          markersize = 5, alpha = 0.5, label  = "Observation")
end

plt
```

```julia

```

notebooks/Linear ensemble filtering Lorenz 96 with localization.ipynb


### VQE（Variational quantum eigensolver）
パラメータ付き量子回路で変分的に基底状態を求めましょう。

### 必要なライブラリをインポート

```python
from sympy import *
from sympy.physics.quantum import *
from sympy.physics.quantum.qubit import Qubit,QubitBra,measure_all,measure_partial
from sympy.physics.quantum.gate import X,Y,Z,H,CNOT,SWAP,CPHASE,CGateS
from sympy.physics.quantum.gate import IdentityGate as _I
from sympy.physics.quantum.gate import UGate as U

import matplotlib.pyplot as plt
%matplotlib inline
```

```python
import scipy.optimize
import scipy.linalg
import numpy as np
import sys
```

### 変分量子回路で利用するためのゲートを定義

```python
def Rxi(n,t): return U(n,represent(cos(t)*_I(1)*_I(0)-I*sin(t)*X(1)*_I(0),nqubits=2))
def Rix(n,t): return U(n,represent(cos(t)*_I(1)*_I(0)-I*sin(t)*_I(1)*X(0),nqubits=2))
def Rzi(n,t): return U(n,represent(cos(t)*_I(1)*_I(0)-I*sin(t)*Z(1)*_I(0),nqubits=2))
def Riz(n,t): return U(n,represent(cos(t)*_I(1)*_I(0)-I*sin(t)*_I(1)*Z(0),nqubits=2))

print(Rxi((0,1),pi/4).get_target_matrix())
print(Rix((0,1),pi/4).get_target_matrix())
print(Rzi((0,1),pi/4).get_target_matrix())
print(Riz((0,1),pi/4).get_target_matrix())
```

    Matrix([[sqrt(2)/2, 0, -sqrt(2)*I/2, 0], [0, sqrt(2)/2, 0, -sqrt(2)*I/2], [-sqrt(2)*I/2, 0, sqrt(2)/2, 0], [0, -sqrt(2)*I/2, 0, sqrt(2)/2]])
    Matrix([[sqrt(2)/2, -sqrt(2)*I/2, 0, 0], [-sqrt(2)*I/2, sqrt(2)/2, 0, 0], [0, 0, sqrt(2)/2, -sqrt(2)*I/2], [0, 0, -sqrt(2)*I/2, sqrt(2)/2]])
    Matrix([[sqrt(2)/2 - sqrt(2)*I/2, 0, 0, 0], [0, sqrt(2)/2 - sqrt(2)*I/2, 0, 0], [0, 0, sqrt(2)/2 + sqrt(2)*I/2, 0], [0, 0, 0, sqrt(2)/2 + sqrt(2)*I/2]])
    Matrix([[sqrt(2)/2 - sqrt(2)*I/2, 0, 0, 0], [0, sqrt(2)/2 + sqrt(2)*I/2, 0, 0], [0, 0, sqrt(2)/2 - sqrt(2)*I/2, 0], [0, 0, 0, sqrt(2)/2 + sqrt(2)*I/2]])

### ハミルトニアンの定義

$ H = \frac{1}{2} \left( S_{ii} \mathbb{1} + S_{ix} \sigma_{x}^{1} + S_{iz} \sigma_{z}^{1} + S_{xi} \sigma_{x}^{0}  + S_{zi} \sigma_{z}^{0}
+ S_{xx} \sigma_{x}^{0} \sigma_{x}^{1} + S_{xz} \sigma_{x}^{0} \sigma_{z}^{1} + S_{zx} \sigma_{z}^{0} \sigma_{x}^{1}
+ S_{zz} \sigma_{z}^{0} \sigma_{z}^{1} \right) $

```python
Sii,Six,Sxi,Sxx,Szz,Siz,Szi,Sxz,Szx = symbols('Sii Six Sxi Sxx Szz Siz Szi Sxz Szx')
# 【設問2-1】定義にあるハミルトニアンを完成してください。
Hamiltonian = (   Sii *_I(0) *_I(1)
                + Six *_I(0) * X(1)
                + Siz *_I(0) * Z(1)
                + Sxi * X(0) *_I(1)
                + Szi * Z(0) *_I(1)
                + Sxx # ***回答欄*** #
                + Sxz # ***回答欄*** #
                + Szx # ***回答欄*** #
                + Szz * Z(0) * Z(1) )/2
h = represent(Hamiltonian,nqubits=2)
Hamiltonian
```

$ HeH^+ $ 分子のハミルトニアンの数値設定（パラメータはNature Communication 2014 より)

```python
H_valued = h.subs([
    (Sii,-3.8505),
    (Six,-0.2288),
    (Sxi,-0.2288),
    (Siz,-1.0466),
    (Szi,-1.0466),
    (Sxx, 0.2613),
    (Sxz, 0.2288),
    (Szx, 0.2288),
    (Szi,-1.0466),
    (Szz,0.2356)])
H_valued
```

    Matrix([
    [-2.85405,        0,        0,  0.13065],
    [       0, -2.04305,  0.13065,  -0.2288],
    [       0,  0.13065, -2.04305,  -0.2288],
    [ 0.13065,  -0.2288,  -0.2288, -0.76085]])

### 答えを事前に計算
このような小さいサイズのMatrixであれば、厳密対角化も計算で、固有値は求められます。

```python
# sympy は数値計算が苦手なので対角化は時間がかかります.
# ↓ sympy で提供されている対角化
# P, D = H_valued.diagonalize()
# ↓ sympy で提供されている固有値, 固有ベクトルの求め方
E = H_valued.eigenvects()
M = np.argmin([re(E[i][0]) for i in range(len(E))])
pprint(E[M])
print(re(E[M][0]))
```

    ⎛                                 ⎡⎡-15.2436817570853 - 7.50558636857843e-28⋅ⅈ
    ⎜                                 ⎢⎢
    ⎜                                 ⎢⎢0.240786150597669 + 1.14522305828383e-30⋅ⅈ
    ⎜-2.86262076407668 - 0.e-22⋅ⅈ, 1, ⎢⎢
    ⎜                                 ⎢⎢0.240786150597669 - 1.97436109465294e-26⋅ⅈ
    ⎜                                 ⎢⎢
    ⎝                                 ⎣⎣                   1.0

    ⎤⎤⎞
    ⎥⎥⎟
    ⎥⎥⎟
    ⎥⎥⎟
    ⎥⎥⎟
    ⎥⎥⎟
    ⎦⎦⎠
    -2.86262076407668

```python
# ↓数値計算は, 固有値を求めるのが得意な numpy でも試します.
l, p = np.linalg.eig( np.array( H_valued.tolist(), dtype=np.complex128 ))
v = np.transpose(p)
mini = np.argmin(l)
E_answer = l[mini]
#print(p)
#print(l)
print(np.array([v[mini]]))
print(E_answer)
```

    [[ 0.99760736+0.j -0.01575801+0.j -0.01575801+0.j -0.06544399+0.j]]
    (-2.8626207640766825+0j)

```python
def dice(n): return [np.random.rand() for i in range(n)]
def vqe_trial(phi):
    global count
    global f
    count += 1
    trial_func = Rxi((0,1), phi[0]).get_target_matrix() \
        * Riz((0,1), phi[1]).get_target_matrix() \
        * represent(CNOT(0,1),nqubits=2) \
        * Riz((0,1), phi[2]).get_target_matrix() \
        * Rix((0,1), phi[3]).get_target_matrix() \
        * Rzi((0,1), phi[4]).get_target_matrix() \
        * Rxi((0,1), phi[5]).get_target_matrix()
    # 【設問2-2】試行関数マトリックス（trial_func）の複素共役を設定してください。
    trial_func_dag = # ***回答欄*** #
    trial = trial_func_dag * H_valued * trial_func
    pr = -1*abs(((qapply(trial).tolist())[0])[0])   # *(-1) をつけなければなりません
    # print(pr)
    f.write(str(count)+ ' ' +str(pr)+ ' ' +str(pr/E_answer)+ '\n')
    f.flush()
    return pr
```

```python
count = 0
f = sys.stdout
# 【設問2-3】上記で定義した vqe_trial を試してみましょう。何回か呼び出して結果を表示してください。
# ***回答欄*** #
```

    1 -2.00938731271756 0.701939753226682
    2 -2.26243027367454 0.790335311636807
    3 -1.71447257643126 0.598917117470241
    4 -1.39939629922598 0.488851445775544
    5 -2.24611667062403 0.784636476759611
    6 -2.07523617734658 0.724942752944761
    7 -1.95482185353686 0.682878388247623
    8 -2.20604916952604 0.770639686964467
    9 -1.83745878938148 0.641879920819389
    10 -1.94575607616102 0.679711438056522

```python
count = 0
f = open('result_VQE.txt', 'w')
# 【設問2-4】scipy.optimize.minimize() 内で vqe_trial を使って、最低エネルギー固有値を探索してください。
res = # ***回答欄*** #
f.close()
```

```python
# 結果を表示します。
# pprint(res)
print(res["fun"])
```

```python
# 【設問2-5】結果ファイルをグラフで表示してみましょう。
dat = np.loadtxt("result_VQE.txt")
# 回答欄 #
```


# ベイズ推定

## ベルヌーイ分布のベイズ推定
具体例 コイントスの確率推定

```python
import numpy as np
import sympy
import matplotlib.pyplot as plt
from scipy.special import gamma
%matplotlib inline
```

```python
mu = sympy.Symbol("u")
def posterior(D, prior):
    global mu
    # 尤度
    likelihood = mu**D[0] * (1-mu)**(D[1]-D[0])
    # 事後確率
    post = prior * likelihood
    # 正規化
    post /= sympy.integrate(post, (mu, 0,1))
    return post
```

```python
fig = plt.figure(0)
# 事前分布
prior = 1

# 分布のプロット
x = np.linspace(0, 1, 100)
y = [1 for j in x]
plt.plot(x, y)

# 分布の更新 (n: 更新回数)
# 0<=th_min<=th_max<=100
n = 5
th_min = 30
th_max = 60
for i in range(n):
    # サンプリング（コインの表の確率[%]の範囲）
    data = np.random.randint(th_min,th_max)
    # 事後分布の計算
    post = posterior((data, 100), prior)

    # 事後分布のプロット
    y = [post.subs(mu, j) for j in x]
    plt.plot(x, y)
    prior = post
```

---

## １変数ガウスのベイズ推定

```python
# ガウス分布を作成
def makeGaussian(mu, sig):
    def gaussian(x):
        return np.exp(-(x - mu)**2 / (2*sig)) / np.sqrt(2*np.pi*sig)
    return gaussian

# ガンマ分布を作成
def makeGammaDist(a, b):
    def gammaDist(x):
        return b**a*x**(a-1)*np.exp(-b*x) / gamma(a)
    return gammaDist

```

データの分布のパラメータ（正規分布）

```python
mu_D, sig_D, N = 2, 9, 10000
```

### 平均$\mu$未知， 分散$\sigma^2$既知の正規分布の$\mu$についてのベイズ推定

平均の事前分布を正規分布とする．
データの確率モデルが正規分布とする時，平均値の共役事前分布は正規分布となる

```python
# 見本のサンプル
x = np.linspace(-10, 10, 100)
Xm = np.random.normal(mu_D, sig_D**0.5, N)
fig, ax1 = plt.subplots()
ax1.hist(Xm, bins=x, normed=True, color="c")  # 見本サンプルのヒストグラム
ax2 = ax1.twinx()

# 事前分布のパラメータを更新
mu_init, sig_init = np.random.uniform(-5, 5), np.random.uniform(1, 5)**2
Ns, n = 10, 5  # サンプル数

# 事後分布計算 → 事前分布の更新
mu_prior, sig_prior = mu_init, sig_init
for i in range(n):
    # サンプリング
    Xs = np.random.normal(mu_D, sig_D**0.5, Ns)

    # 事後分布計算
    mu_post = sig_D*mu_prior/(Ns*sig_prior+sig_D) + Ns*sig_prior*Xs.mean()/(Ns*sig_prior+sig_D)
    sig_post = (1/sig_prior + Ns/sig_D)**-1
    dist_post = makeGaussian(mu_post, sig_post)

    # 事後分布をプロット
    ax2.plot(x, dist_post(x), linewidth=2)

    # 事前分布のパラメータを更新
    mu_prior = mu_post
    sig_prior = sig_post

_=ax2.set_ylim(0, )

# 推定分布
ax1.plot(x, makeGaussian(mu_post, sig_D)(x), "m", linewidth=2)
print("mu", mu_init,  "==>", mu_post, "<=>", mu_D)
print("sig", sig_init,  "==>", sig_post)
```

### 平均$\mu$既知， 分散$\sigma^2$未知の正規分布の$\mu$についてのベイズ推定
分散の事前分布をガンマ分布とする
データの確率モデルが正規分布とする時，分散値の共役事前分布はガンマ分布となる

```python
# 見本のサンプル
x = np.linspace(-10, 10, 100)
Xm = np.random.normal(mu_D, sig_D**0.5, N)
fig, ax1 = plt.subplots()
ax1.hist(Xm, bins=x, normed=True, color="c")  # 見本サンプルのヒストグラム

# 事前分布のパラメータ
a_init, b_init = 0, 0  # 無情報事前分布のパラメータ
Ns, n = 10, 10  # サンプル数 and 繰り返し回数

# 事後分布計算 → 事前分布の更新
a_prior, b_prior = a_init, b_init
for i in range(n):
    # サンプリング
    Xs = np.random.normal(mu_D, sig_D**0.5, Ns)

    # 事後分布計算
    a_post = a_prior + Ns/2
    b_post = b_prior + Ns*Xs.var()/2;
    dist_post = makeGammaDist(a_post, b_post)

    ax1.plot(x, makeGaussian(mu_D, b_post/a_post)(x), linewidth=2)

    # 事前分布のパラメータを更新
    a_prior = a_post
    b_prior = b_post

_=ax2.set_ylim(0, )

ax1.plot(x, makeGaussian(mu_D, b_post/a_post)(x), "m", linewidth=2)
print("sig:", b_post/a_post, "<==>", sig_D)
```

### 平均$\mu$未知， 分散$\sigma^2$未知の正規分布の$\mu$についてのベイズ推定
平均の事前分布を正規分布，分散の事前分布をガンマ分布とする
データの確率モデルが正規分布とする時，平均値と分散値の共役事前分布は正規-ガンマ分布となる

```python
# 見本のサンプル
x = np.linspace(-20, 20, 100)
Xs = np.random.normal(mu_D, sig_D**0.5, N)
fig, ax1 = plt.subplots()
ax1.hist(Xs, bins=x, normed=True, color="c")  # 見本サンプルのヒストグラム

# 事前分布のパラメータ
a_init, b_init, mu_init, l_init = 0,0,0,0
Ns, n = 10, 10   # サンプル数 and 繰り返し回数

a_prior, b_prior, mu_prior, l_prior = a_init, b_init, mu_init, l_init
Xs = np.zeros(0)
for i in range(n):
    Xs = np.random.normal(mu_D, sig_D**0.5, Ns)

    a_post = a_prior + Ns/2
    b_post =  b_prior + ((Xs**2).sum() - Ns*(mu_prior**2)) / 2
    mu_post = (Xs.sum() + l_prior*mu_prior) / (Ns + l_prior)
    l_post = Ns + l_prior

    #plt.subplot(122)
    model_dist = makeGaussian(mu_post, b_post/a_post)
    plt.plot(np.linspace(-20, 20, 100), model_dist(np.linspace(-20, 20, 100)), )

    a_prior = a_post
    b_prior = b_post
    mu_prior = mu_post
    l_prior = l_post

plt.plot(x, makeGaussian(mu_post, b_post/a_post)(x), "m", linewidth=2)
print(mu_D, sig_D)
print(mu_post, b_post/a_post)
```