# Evaluation of Clusters

**CS5483 Data Warehousing and Data Mining**
$\def\abs#1{\left\lvert #1 \right\rvert}
\def\Set#1{\left\{ #1 \right\}}
\def\mc#1{\mathcal{#1}}
\def\M#1{\boldsymbol{#1}}
\def\R#1{\mathsf{#1}}
\def\RM#1{\boldsymbol{\mathsf{#1}}}
\def\op#1{\operatorname{#1}}
\def\E{\op{E}}
\def\d{\mathrm{\mathstrut d}}
$
___

$\def\abs#1{\left\lvert #1 \right\rvert}
\def\Set#1{\left\{ #1 \right\}}
\def\mc#1{\mathcal{#1}}
\def\M#1{\boldsymbol{#1}}
\def\R#1{\mathsf{#1}}
\def\RM#1{\boldsymbol{\mathsf{#1}}}
\def\op#1{\operatorname{#1}}
\def\E{\op{E}}
\def\d{\mathrm{\mathstrut d}}
$

```python
%reset -f
%matplotlib inline
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn import datasets, preprocessing
from sklearn.pipeline import make_pipeline
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_samples, silhouette_score
# produce vector inline graphics
from IPython.display import set_matplotlib_formats, IFrame
set_matplotlib_formats('svg')
from ipywidgets import interact
import ipywidgets as widgets
from scipy.optimize import linear_sum_assignment
```

In this notebook, we will consider different methods of evaluating clustering solutions.

## Intrinsic measure

The intrinsic measures of cluster quality is useful when the ground truth is not available or should not be used. For example, to determine the number of clusters, we can compare the intrinsic measures of the clustering solutions for different numbers of clusters.

### Elbow Method

To determine the number $k$ of clusters for a centroid-based clustering algorithm, the elbow method uses the within cluster sum of squared $\operatorname{WSS}(k)$ errors as the intrinsic quality measure. Instead of minimizing $\operatorname{WSS}(k)$, it maximizes the marginal decrease.

The following diagram shows the KnowledgeFlow interface of Weka that applies `SimpleKMeans` to the `iris.2D.arff` dataset for different choices of $k$.

Implement the above using Weka. As the following video shows, it is now possible to run Weka GUI in a web browser through the JupyterHub server:

```python
IFrame(src="https://cityuhk-lms.ap.panopto.com/Panopto/Pages/Embed.aspx?id=e38e4edb-816e-4535-b0d2-acf40085c0e4&autoplay=false&offerviewer=true&showtitle=true&showbrand=false&start=0&interactivity=all", height=450, width=805)
```

- Under the `Files` tab, click the `new` button on the right and choose `xpra desktop`.
- In the terminal, run the command `weka-stable-gui` to launch Weka.

To access the weka data files under your home directory as `~/wekadata`, execute the following cell, which creates a symbolic link to the data folder:

```python
if input('Create symlink to weka data file? [Y/n]').lower() != 'n':
    !ln -s /home/course/cs5483/wekadata ~/
```

**Exercise** Complete the following `DataFrame` `df_WSS` by recording $\operatorname{WSS}(k)$ for $k$ from 1 to 4. Your answer may look like

```Python
...
df_WSS['WSS'] = [__, __, __, __]
...
```

```python
df_WSS = pd.DataFrame(columns=['k', 'WSS'])
df_WSS['k'] = np.arange(1, 5, dtype=int)
df_WSS['WSS'] = df_WSS['WSS'].astype(float)
# YOUR CODE HERE
raise NotImplementedError()

# plot WSS as a function of k
df_WSS.plot(x='k', y='WSS')
plt.xlabel('k')
plt.ylabel('WSS')
plt.show()
df_WSS
```

```python
# tests
```

### Silouette analysis

An alternative method to the elbow method is the silouette analysis. We will use the existing implementation in `sklearn`. First, import the iris dataset from `sklearn.datasets` and store it as a `DataFrame`:

```python
# load the dataset from sklearn
dataset = datasets.load_iris()

# create a DataFrame to help further analysis
df = pd.DataFrame(data=dataset.data, columns=dataset.feature_names)
df['target'] = dataset.target
df.target = df.target.astype('category')
df.target.cat.categories = dataset.target_names
df  # display an overview of the data
```

      sepal length (cm)
      sepal width (cm)
      petal length (cm)
      petal width (cm)
      target

      0
      5.1
      3.5
      1.4
      0.2
      setosa

      1
      4.9
      3.0
      1.4
      0.2
      setosa

      2
      4.7
      3.2
      1.3
      0.2
      setosa

      3
      4.6
      3.1
      1.5
      0.2
      setosa

      4
      5.0
      3.6
      1.4
      0.2
      setosa

      ...
      ...
      ...
      ...
      ...
      ...

      145
      6.7
      3.0
      5.2
      2.3
      virginica

      146
      6.3
      2.5
      5.0
      1.9
      virginica

      147
      6.5
      3.0
      5.2
      2.0
      virginica

      148
      6.2
      3.4
      5.4
      2.3
      virginica

      149
      5.9
      3.0
      5.1
      1.8
      virginica

150 rows × 5 columns

To cluster the data using $k$-means clustering:

```python
from sklearn.cluster import KMeans

kmeans_minmax_normalized = make_pipeline(preprocessing.MinMaxScaler(),
                                         KMeans(n_clusters=3, random_state=0))
kmeans_minmax_normalized

feature1, feature2 = 'petal length (cm)', 'petal width (cm)'
labels = kmeans_minmax_normalized.fit_predict(df[[feature1, feature2]])

plt.figure(figsize=(10, 5))

plt.subplot(121)
plt.scatter(df[feature1], df[feature2], c=labels)
plt.title("Cluster assignment")
plt.xlabel(feature1)
plt.ylabel(feature2)

plt.subplot(122)
plt.scatter(df[feature1], df[feature2], c=dataset['target'])
plt.title("Class (ground truth)")
plt.xlabel(feature1)

plt.show()
```

**Exercise** Complete the following `DataFrame` `df_silouette` by recording the [`silouette_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.silhouette_score.html) for different number $k$ of clusters from 2 to 10. Your answer may look like

```Python
        ...
        df_silouette.loc[i, 's'] = silhouette_score(___, ___)
        ...
```
A more elaborate silhouette analysis is given [here](https://scikit-learn.org/stable/auto_examples/cluster/plot_kmeans_silhouette_analysis.html#sphx-glr-auto-examples-cluster-plot-kmeans-silhouette-analysis-py).

```python
from sklearn.metrics import silhouette_score

df_silouette = pd.DataFrame(columns=['k', 's'])
df_silouette['k'] = np.arange(2, 11, dtype=int)
df_silouette['s'] = df_silouette['s'].astype(float)
for i in range(len(df_silouette)):
    labels_ = make_pipeline(
        preprocessing.MinMaxScaler(),
        KMeans(n_clusters=df_silouette.loc[i, 'k'],
               random_state=0)).fit_predict(df[[feature1, feature2]])
    # YOUR CODE HERE
    raise NotImplementedError()

# plot WSS as a function of k
df_silouette.plot(x='k', y='s')
plt.xlabel('k')
plt.ylabel('silouette score')
plt.show()
df_silouette
```

```python
# tests
```

### Extrinsic measures of cluster quality

Suppose $L(\M{p})$ and $C(\M{p})$ are the class and cluster labels of each sample $\M{p}\in D$. An extrinsic measure compares how similar the corresponding partitions $\Set{L_i}$ and $\Set{C_i}$ are, where

\begin{align}
L_i &:=\Set{\M{p}|L(\M{p})=i}\\
C_i &:=\Set{\M{p}|C(\M{p})=i}.
\end{align}

#### Pairwise correctness

Define the indicator of correct clustering for a pair $\M{p}, \M{q}\in D$ as

\begin{align}
\op{correctness}(\M{p},\M{q}) :=
\begin{cases}
1 & L(\M{p})=L(\M{q}) \iff C(\M{p})=C(\M{q})\\
0 & \text{otherwise.}
\end{cases}
\end{align}

In other words, the clustering for the pair of samples is correct when
- the samples have equal class labels and equal cluster labels, or
- the samples have different class labels and different cluster labels.

The following function returns a boolean matrix of $[\op{correctness}(\M{p},\M{q})]$, with rows an columns indexed by $\M{p}$ and $\M{q}$ respectively.

```python
def correctness(class_labels, cluster_labels):
    '''Returns the pairwise correctness matrix for the class and cluster labels.

    Parameters:
    -----------
    class_labels (array): non-negative integer class labels for certain samples
    cluster_labels (array): corresponding non-negative integer cluster labels

    Returns:
    --------
    A matrix (2D array) of correctness(p,q) with rows and columns indexed by
    p and q respectively in the samples. correctness(p,q) indicates whether
      - p and q have equal class labels and equal cluster labels, or
      - p and q have different class labels and different cluster labels.
    '''
    class_labels = np.asarray(class_labels)
    cluster_labels = np.asarray(cluster_labels)

    eq_class = class_labels.reshape(-1, 1) == class_labels.reshape(1, -1)
    eq_cluster = cluster_labels.reshape(-1, 1) == cluster_labels.reshape(1, -1)

    return (eq_class & eq_cluster) | ~(eq_class | eq_cluster)
```

For instance, consider the following class and cluster labels:

|index $i$|class label $L(\M{p}_i)$ | cluster label $C(\M{p}_i)$|
|:-:|:----------:|:----------:|
| 0 | 0 | 1 |
| 1 | 0 | 1 |
| 2 | 1 | 0 |

The correctness matrix and the fraction correctly clustered pairs are:

```python
correctness_matrix = correctness(class_labels=[0, 0, 1],
                                 cluster_labels=[1, 1, 0])
correctness_accuracy = correctness_matrix.mean()
print(f'Correctness matrix:\n {correctness_matrix}')
print(f'Accuracy: {correctness_matrix.mean():.3g}')
```

**Exercise** Assign the accuracy to `correctness_accuracy` the accuracy for the $k$-means clustering solution on the iris dataset. Try to have as many samples as possible.

```python
# YOUR CODE HERE
raise NotImplementedError()
correctness_accuracy
```

```python
# tests
```

**Exercise** Give a choice of `class_labels` and `cluster_labels` that give the smallest possible accuracy computed using `correctness`. Assign the accuracy to `correctness_accuracy`. Try to have as many samples as possible.

```python
# YOUR CODE HERE
raise NotImplementedError()
correctness_accuracy
```

```python
# tests
```

#### B-Cubed metrics

The accuracy computed from the pairwise correctness matrix can be misleading as it is dominated by true negatives. Similar to the class-imbalance problem, precision and recall can be used instead.

For every point $\M{p}\in D$, the B-Cubed precision and recall are defined as

\begin{align}
\op{precision}(\M{p}) &:= \frac{\abs{\Set{\M{q}\in D|C(\M{p})=C(\M{q}), L(\M{p})=L(\M{q})}}}{\abs{\Set{\M{q}\in D|C(\M{p})=C(\M{q})}}}\\
\op{recall}(\M{p}) &:= \frac{\abs{\Set{\M{q}\in D|C(\M{p})=C(\M{q}), L(\M{p})=L(\M{q})}}}{\abs{\Set{\M{q}\in D|L(\M{p})=L(\M{q})}}}.
\end{align}

The overall precision and recall are the average precisions and recalls of each point.

For instance, consider the following class and cluster labels:

|index $i$|class label $L(\M{p}_i)$ | cluster label $C(\M{p}_i)$|
|:-:|:----------:|:----------:|
| 0 | 0 | 0 |
| 1 | 0 | 1 |
| 2 | 1 | 2 |
| 3 | 1 | 2 |

The precisions and recalls for each point are

|index $i$|precision for $\M{p}_i$ | recall for $\M{p}_i$|
|:-:|:----------:|:----------:|
| 0 | 1 | 0.5 |
| 1 | 1 | 0.5 |
| 2 | 1 | 1 |
| 3 | 1 | 1 |

and so the overall precision an recall are $1$ and $0.75$ respectively.

**Exercise** Complete the following class that computes the B-Cubed precisions and recalls for each sample and their averages. Your solution may look like:
```Python
        ...
        FPs = (___ & ___).sum(axis=1)
        FNs = (___ & ___).sum(axis=1)

        self.precisions = TPs / (___ + ___)
        self.recalls = ___
        ...
```

```python
class BCubed:
    ''' Compute B-Cubed precision and recall.

    Parameters:
    -----------
    class_labels (array): non-negative integer class labels for certain samples.
    cluster_labels (array): corresponding non-negative integer cluster labels.

    Attributes:
    -----------
    precisions (array of float): B-Cubed precisions for each sample.
    recalls (array of float): B-Cubed recalls for each sample.
    precision (float): Overall B-Cubed precision.
    recall (float): Overall B-Cubed recall.
    '''
    def __init__(self, class_labels, cluster_labels):
        self.class_labels = np.asarray(class_labels)
        self.cluster_labels = np.asarray(cluster_labels)

        eq_class = self.class_labels[:, None] == self.class_labels[None, :]
        eq_cluster = self.cluster_labels[:, None] == self.cluster_labels[None, :]

        TPs = (eq_class & eq_cluster).sum(axis=1)
        # YOUR CODE HERE
        raise NotImplementedError()
        self.precision = self.precisions.mean()
        self.recall = self.recalls.mean()

    def __repr__(self):
        return f'Precision: {self.precision:.3g}\n' + f'Recall: {self.recall:.3g}'
```

```python
# tests
# simple case
bcubed_eval = BCubed(class_labels=[0, 0, 1, 1], cluster_labels=[0, 1, 2, 2])
assert np.isclose(bcubed_eval.precisions, [1, 1, 1, 1], rtol=1e-3).all()
assert np.isclose(bcubed_eval.recalls, [0.5, 0.5, 1, 1], rtol=1e-3).all()
# test on iris
bcubed_eval = BCubed(dataset['target'], labels)
print(bcubed_eval)
assert np.isclose(bcubed_eval.precision, 0.9252136752136751, rtol=1e-3)
assert np.isclose(bcubed_eval.recall, 0.9253333333333335, rtol=1e-3)
```

#### Classes to Clusters Evaluation

Instead of using pairwise comparison, Weka uses the classes-to-clusters evaluation. The computation of accuracy can be cast as a linear sum assignment problem (maximum weight matching in bipartite graphs). First, note that a classes-to-clusters assignment is a matching between the class and cluster labels, which can be represented by a boolean matrix $[\delta_{ij}]$ for class label $i$ and cluster label $j$, where

\begin{align}
\delta_{ij}
&= \begin{cases}
1 & \text{if class $i$ is assigned to cluster $j$,}\\
0 & \text{otherwise.}
\end{cases}
\end{align}

The accuracy for classes-to-clusters evaluation is then given by

\begin{align}
\text{accuracy} &=
\max_{[\delta_{ij}]} \sum_{i} \sum_{j}
\overbrace{\left|\{\boldsymbol{p}\in D|L(\boldsymbol{p})=i, C(\boldsymbol{p})=j\}\right|}^{n_{ij}:=}
\end{align}

where $C(\boldsymbol{p})$ and $L(\boldsymbol{p})$ are the class and cluster labels respectively for the data point $\boldsymbol{p}\in D$.

It will be useful to compute a matrix of the counts $n_{ij}$ of samples with class label $i$ as row index and cluster label $j$ as column index. The following function implements such computation.

```python
def class_cluster_counts(class_labels, cluster_labels):
    '''Returns the class-cluster count matrix with rows and columns indexed by
    class and cluster labels respectively.

    Parameters:
    -----------
    class_labels (array): non-negative integer class labels for certain samples
    cluster_labels (array): corresponding non-negative integer cluster labels

    Returns:
    --------
    counts: a matrix of counts of samples with row indexed by class labels and
            columns index by cluster labels.
    '''
    counts = np.zeros((class_labels.max() + 1, cluster_labels.max() + 1),
                      dtype=int)
    for i, j in np.column_stack((class_labels, cluster_labels)):
        counts[i, j] += 1
    return counts
```

For the $k$-means clustering on the iris dataset, the matrix $[n_{ij}]$ of class-cluster counts is:

```python
counts = class_cluster_counts(dataset['target'], labels)
df_counts = pd.DataFrame(counts)
df_counts
```

      0
      1
      2

      0
      50
      0
      0

      1
      0
      2
      48

      2
      0
      46
      4

We can use [`linear_sum_assignment`][lsa] from `scipy.optimize` module to find the optimal assignment as follows.

[lsa]: https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.linear_sum_assignment.html

```python
from scipy.optimize import linear_sum_assignment

classes, clusters = linear_sum_assignment(counts, maximize=True)
```

The following highlights the optimal assignment on the class-cluster count matrix.

```python
def highlight(data):
    attr = 'background-color: yellow'
    return pd.DataFrame(
        [[attr if i == j else '' for j in range(len(data.columns))]
         for i in range(len(data.index))],
        index=data.index,
        columns=data.columns)

df_counts.style.apply(highlight, axis=None, subset=(classes, clusters))
```

        	        0	        1	        2

                        0
                        50
                        0
                        0

                        1
                        0
                        2
                        48

                        2
                        0
                        46
                        4

**Exercise** Complete the following class that implements the classes-to-cluster evaluation using `linear_sum_assignment` from `scipy.optimize` module. You solution may look like:
```Python
        ...
        # counts of samples indexed by class labels followed by cluster labels
        counts = class_cluster_counts(____, ____)

        # compute the best assignment of class labels to cluster labels
        # that maximizes accuracy
        self.classes, self.clusters = linear_sum_assignment(_____, maximize=____)
        self.accuracy = float(counts[____, ____].sum()) / counts.sum()
        ...
```

```python
from scipy.optimize import linear_sum_assignment

class Classes2ClustersEvaluation:
    ''' Classes-to-clusters evaluation.

    Parameters:
    -----------
    class_labels (array): non-negative integer class labels for certain samples.
    cluster_labels (array): corresponding non-negative integer cluster labels.

    Attributes:
    -----------
    accuracy: fraction of correctly clustered instances.
    classes: int array of assigned class labels sorted in ascending order.
    clusters: int array of cluster labels corresponding to the assigned class labels.
    '''
    def __init__(self, class_labels, cluster_labels):
        self.class_labels = np.asarray(class_labels)
        self.cluster_labels = np.asarray(cluster_labels)
        # YOUR CODE HERE
        raise NotImplementedError()

    def __repr__(self):
        s = f'Accuracy: {self.accuracy:.3g}\n'
        for i, j in np.column_stack((self.classes, self.clusters)):
            s += f'Class #{i} --> Cluster #{j}\n'
        return s
```

```python
# tests
# simple case
c2c_eval = Classes2ClustersEvaluation(np.array([0, 0, 1, 1]),
                                      np.array([0, 1, 2, 2]))
assert (c2c_eval.classes == [0, 1]).all()
assert (c2c_eval.clusters == [0, 2]).all()
assert np.isclose(c2c_eval.accuracy, 0.75, rtol=1e-3)
# test on iris
c2c_eval = Classes2ClustersEvaluation(dataset['target'], labels)
print(c2c_eval)
assert np.isclose(c2c_eval.accuracy, 0.96, rtol=1e-3)
```


+ This notebook is part of lecture 6 *Columnspace and nullspace* in the OCW MIT course 18.06 by Prof Gilbert Strang [1]
+ Created by me, Dr Juan H Klopper
    + Head of Acute Care Surgery
    + Groote Schuur Hospital
    + University Cape Town
    + Email me with your thoughts, comments, suggestions and corrections

Linear Algebra OCW MIT18.06 IPython notebook [2] study notes by Dr Juan H Klopper is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

+ [1] OCW MIT 18.06
+ [2] Fernando Pérez, Brian E. Granger, IPython: A System for Interactive Scientific Computing, Computing in Science and Engineering, vol. 9, no. 3, pp. 21-29, May/June 2007, doi:10.1109/MCSE.2007.53. URL: http://ipython.org

```python
from IPython.core.display import HTML, Image
css_file = 'style.css'
HTML(open(css_file, 'r').read())
```

```python
#import numpy as np
from sympy import init_printing, Matrix, symbols
#import matplotlib.pyplot as plt
#import seaborn as sns
#from IPython.display import Image
from warnings import filterwarnings

init_printing(use_latex = 'mathjax')
%matplotlib inline
filterwarnings('ignore')
```

# Columnspace and nullspace of a matrix

## Columnspaces of matrices

* We saw in the previous lecture that columns of a matrix can form vectors
* Consider now the LU-decomposition of *A*
$$  PA = PLU $$

* The union P∪L (all vectors in P or L or both) is NOT a subspace
* The intersection P∩L (or vectors in P and L) is a subspace (because their intersection is only the zero vector)

* The intersection of any two subspaces is a subspace

* Consider the following example matrix

```python
A = Matrix([[1, 1, 2], [2, 1, 3], [3, 1, 4], [4, 1, 5]])
A
```

* Each of the column spaces are vectors (column space) in ℝ4

* The linear combinations of all the column vectors form a subspace
* Is it the whole *V* = ℝ4, though?

* The reason why we ask is because we want to bring it back to a system of linear equations and ask the question: Is there (always) a solution to the following:
$$ {A} \overline {x}= \overline {b} $$

* Thus, which right-hand sides *b* are allowed?
* In our example above we are in ℝ4 and we ask if linear combination of all of them fill ℝ4

* From our example above some right-hand sides will be allowed (they form a subspace)
* Let's look at an example for **b**

```python
x1, x2, x3 = symbols('x1, x2, x3')
vec_x = Matrix([x1, x2, x3])
b = Matrix([1, 2, 3, 4])
A, vec_x, b
```

```python
A * vec_x
```

* You can do the row multiplication, but it's easy to see from above we are asking about linear combinations of the columns, i.e. how many (*x*1) of column 1 plus how many (*x*2) of column 2 plus how many (*x*3) of column 3 equals **b**?
* Well, since **b** is the same as the first column, **x** would be
$$ \begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix} $$

* So we can solve for all values of **b** if **b** is in the column space

### Linear independence

* We really need to know if the columns above are linearly independent
* We note that column three above is a linear combination of the first two, so adds nothing new
* Actually, we could also throw away the first one because it is column 3 plus -1 times column 2
* Same for column 2
* We thus have two columns left and we say that the column space is of dimension 2 (a 2-dimensional subspace of ℝ4)

## The nullspace

* It contains all solutions **x** for A**x**=0
* This solution(s) is in ℝ3

```python
zero_b = Matrix([0, 0, 0, 0])
A, vec_x, zero_b
```

* Some solutions would be
$$ \begin{bmatrix} 0 \\ 0 \\ 0 \end{bmatrix} $$
$$ \begin{bmatrix} 1 \\ 1 \\ -1 \end{bmatrix} $$
$$ \begin{bmatrix} 2 \\ 2 \\ -2 \end{bmatrix} $$
* In fact, we have:
$$ {c} \begin{bmatrix} 1 \\ 1 \\ -1 \end{bmatrix} $$
* It is thus a line
* The nullspace is a line in ℝ3

* **PLEASE** remember, for any space the rules of addition and scalar multiplication must hold for vectors to remain in that space

```python

```


# Determinant Quantum Monte Carlo

## 1 Hubbard model

The Hubbard model is defined as

\begin{align}
    \label{eq:ham} \tag{1}
    H &= -\sum_{ij\sigma} t_{ij} \left( \hat{c}_{i\sigma}^\dagger \hat{c}_{j\sigma} + hc \right)
    + \sum_{i\sigma} (\varepsilon_i - \mu) \hat{n}_{i\sigma}
    + U \sum_{i} \left( \hat{n}_{i\uparrow} - \tfrac{1}{2} \right) \left( \hat{n}_{i\downarrow} - \tfrac{1}{2} \right)
\end{align}

where $U$ is the interaction strength, $\varepsilon_i$ the on-site energy at site $i$ and $t_{ij}$ the hopping energy between the sites $i$ and $j$. The chemical potential is defined to be $\mu = 0$ for a half filled Hubabrd model since the total chemical potential is given as $\mu + \tfrac{U}{2}$:

\begin{equation}
    H = -\sum_{ij\sigma} t_{ij} \left( \hat{c}_{i\sigma}^\dagger \hat{c}_{j\sigma} + hc \right)
    + \sum_{i\sigma} \left(\varepsilon_i - \mu - \tfrac{U}{2}\right) \hat{n}_{i\sigma}
    + U \sum_{i} \hat{n}_{i\uparrow} \hat{n}_{i\downarrow}.
\end{equation}

The non-interacting of the Hubbard Hamiltonian is quadratic in the creation and annihilation operators,

\begin{equation}
    K = -\sum_{ij\sigma} t_{ij} \left( \hat{c}_{i\sigma}^\dagger \hat{c}_{j\sigma} + hc \right)
    + \sum_{i\sigma} (\varepsilon_i - \mu) \hat{n}_{i\sigma},
    \label{eq:ham_kin} \tag{2}
\end{equation}

while the interaction part is quartic in the fermion operators:

\begin{equation}
    V = U \sum_{i} \left( \hat{n}_{i\uparrow} - \tfrac{1}{2} \right) \left( \hat{n}_{i\downarrow} - \tfrac{1}{2} \right).
    \label{eq:ham_inter} \tag{3}
\end{equation}

## 2 Distribution operator

The expectation value of a observable $O$ is given by

\begin{equation}
    \langle O \rangle = \text{Tr}(O \mathcal{P}),
\end{equation}

where $\mathcal{P}$ is the distribution operator,

\begin{equation}
    \mathcal{P} = \frac{1}{\mathcal{Z}} e^{-\beta H},
    \label{eq:distop} \tag{4}
\end{equation}

$\mathcal{Z}$ is the partition function,

\begin{equation}
    \mathcal{Z} = \text{Tr}(e^{-\beta H}),
    \label{eq:partition}
\end{equation}

and $\beta = 1/k_B T$ is the inverse of the temperature. The trace is taken over the Hilbert space describing all occupied states of the system:

\begin{equation}
    \text{Tr}(e^{-\beta H}) = \sum_i \langle \psi_i | e^{-\beta H} | \psi_i \rangle.
\end{equation}

In order to obtain a computable approximation of the distribution operator the partition function has to be approximated. Since the quadratic term $ K $ and quartic term $ V $ of the Hubbard Hamiltonian do not commute a Trotter-Suzuki decomposition has to be used to approximate $\mathcal{Z}$. By dividing the imaginary-time interval from $0$ to $\beta$ into $L$ intervals of size $\Delta \tau = \beta / L$, the partition function can be written as

\begin{equation}
    \label{eq:partitionTrotterDecomp}
    \mathcal{Z} = \text{Tr}(e^{-\beta H }) = \text{Tr}( \prod_{l=1}^{L} e^{-\Delta \tau H})
    = \text{Tr}( \prod_{l=1}^{L} e^{-\Delta \tau K} e^{-\Delta \tau V}) + \mathcal{O}(\Delta \tau^2).
\end{equation}

The spin-up and spin-down operators of the quadratic kinetic energy term are independent and can be written as

\begin{equation}
    e^{-\Delta \tau K} = e^{-\Delta \tau K_\uparrow} e^{-\Delta \tau K_\downarrow}.
\end{equation}

The particle number operators $\hat{n}_{i\sigma}$ in the interacting term of the Hubbard Hamiltonian are independent of the site index $i$:

\begin{equation}
    e^{-\Delta \tau V} = e^{- U \Delta\tau \sum_{i=1}^N
    \left( \hat{n}_{i\uparrow} - \tfrac{1}{2} \right) \left( \hat{n}_{i\downarrow} - \tfrac{1}{2} \right)}
    = \prod_{i=1}^N e^{- U \Delta\tau \left( \hat{n}_{i\uparrow} - \tfrac{1}{2} \right) \left( \hat{n}_{i\downarrow} - \tfrac{1}{2} \right)}
\end{equation}

The quartic contributions of the interacting term need to be represented in a quadratic form. This can be achieved by using  the discrete \emph{Hubbard-Stratonovich} transformation, which replaces the term $\left( \hat{n}_{i\uparrow} - \tfrac{1}{2} \right) \left( \hat{n}_{i\downarrow} - \tfrac{1}{2} \right)$ by a quadratic term of the form $\left( \hat{n}_{i\uparrow} - \hat{n}_{i\downarrow} \right)$. For $U>0$, this yields

\begin{equation}
    \label{eq:hubbardStratanovichInteractionTerm}
    e^{- U \Delta\tau \left( \hat{n}_{i\uparrow} - \tfrac{1}{2} \right) \left( \hat{n}_{i\downarrow} - \tfrac{1}{2} \right)} = C \sum_{h_i = \pm 1} e^{\nu h_i \left( \hat{n}_{i\uparrow} - \hat{n}_{i\downarrow} \right)},
\end{equation}

where $C=\frac{1}{2} e^{-\frac{U \Delta\tau}{4}}$ and the constant $\nu$ is defined by
\begin{equation}
     \label{eq:lambda}
    \cosh \nu = e^{-\frac{U \Delta\tau}{2}}.
\end{equation}

The set of auxiliary variables $\lbrace h_i \rbrace$ is called the *Hubbard-Stratonovich field* or *configuration*. The variables $h_i$ take the values $\pm 1$ for a up- or down-spin, respectively. Using the Hubbard-Stratanovich transformation the interaction term can be formulated as

\begin{equation}
\begin{split}
    \label{eq:hubbardStratanovichInteractionFull}
    e^{-\Delta\tau V} &= \prod_{i=1}^N \left(C \sum_{h_i = \pm 1} e^{\nu h_i \left( \hat{n}_{i\uparrow} - \hat{n}_{i\downarrow} \right)}\right) \\
    &= C^N \sum_{h_i = \pm 1} e^{\sum_{i=1}^N \nu h_i \left( \hat{n}_{i\uparrow} - \hat{n}_{i\downarrow} \right)} \\
    &= C^N \text{Tr}_h  e^{\sum_{i=1}^N \nu h_i \left( \hat{n}_{i\uparrow} - \hat{n}_{i\downarrow} \right)} \\
    &= C^N \text{Tr}_h  e^{\sum_{i=1}^N \nu h_i \hat{n}_{i\uparrow}} e^{-\sum_{i=1}^N \nu h_i \hat{n}_{i\downarrow}} \\
    &= C^N \text{Tr}_h  e^{V_\uparrow} e^{V_\downarrow}
\end{split}
\end{equation}
where
\begin{equation}
    V_\sigma = \sum_{i=1}^N \nu h_i \hat{n}_{i\sigma} = \sigma \nu \boldsymbol{\hat{c}}_\sigma^\dagger V(h) \boldsymbol{\hat{c}}_\sigma
\end{equation}

and $V(h)$ is a diagonal matrix of the configurations $V(h) = \text{diag}(h_1, h_2, \dots, h_N)$. Taking into account the $L$ imaginary time slices, the Hubbard-Stratonovich variables are expanded to have two indices $h_{i, l}$, where $i$ represents the site index and $l$ the imaginary time slice:

\begin{equation}
    h_i \longrightarrow h_{il}, \quad V(h) \longrightarrow V_l(h_l), \quad V_\sigma \longrightarrow V_{l\sigma}.
\end{equation}

The Hubbard-Stratonovich field or configuration now is a $N \times L$ matrix for a system of $N$ sites with $L$ time steps.
Therefore, the partition function can be approximated by

\begin{equation}
    \label{eq:partitionApproximation} \tag{5}
    \mathcal{Z} = \eta_d \text{Tr}_h \text{Tr} \left( \prod_{l=1}^L e^{-\Delta\tau K_\uparrow} e^{V_{l\uparrow}} \right) \left( \prod_{l=1}^L e^{-\Delta\tau K_\downarrow} e^{V_{l\downarrow}} \right),
\end{equation}

where $\eta_d = C^{NL}$ is a normalization constant. At this point, all operators are quadratic in the fermion operators. For any quadratic operator

\begin{equation}
    H_l = \sum_{ij} \hat{c}_i^\dagger (H_l)_{ij} \hat{c}_j
\end{equation}

the trace can be computed via the a determinant:

\begin{equation}
    \text{Tr}(e^{-H_1}e^{-H_2} \dots e^{-H_L}) =  \det(I + e^{-H_L} e^{-H_{L-1}} \dots e^{-H_1} )
\end{equation}

Using this identity, the trace in the expression of the partition function \eqref{eq:partitionApproximation} can be turned into a computable form:

\begin{equation}
    \label{eq:partitionComputable} \tag{6}
    \mathcal{Z}_h = \eta_d \text{Tr}_h \det[M_\uparrow(h)]  \det[M_\downarrow(h)],
\end{equation}

where for $\sigma = \pm1$ and $h=(h_1, h_2, \dots, h_L)$ the matrix

\begin{equation}
    \label{eq:mMatrix} \tag{7}
    M_\sigma(h) = I + B_{L,\sigma}(h_L) B_{L-1,\sigma}(h_{L-1}) \dots B_{1,\sigma}(h_1)
\end{equation}

consist of the time step matrices $B_{l,\sigma}(h_l)$, which are associated with the operators $e^{-\Delta\tau K_\sigma} e^{V_{l\sigma}}$:

\begin{equation}
    \label{eq:bMatrix}
    B_{l,\sigma}(h_l) = e^{-\Delta\tau K_\sigma} e^{\sigma \nu V_l(h_l)}.
\end{equation}

With the approximation \eqref{eq:partitionComputable} the distribution operator $\mathcal{P}$, defined in \eqref{eq:distop}, can be expressed as the computable approximation

\begin{equation}
    \label{eq:distopComputable} \tag{8}
    \mathcal{P}(h) = \frac{\eta_d}{\mathcal{Z}_h} \det[M_\uparrow(h)]  \det[M_\downarrow(h)].
\end{equation}

The Green's function $G$ associated with the configuration $h$ is defined as the inverse of the matrix $M_\sigma(h)$:

\begin{equation}
    G_\sigma(h) = \left[M_\sigma(h)\right]^{-1}
\end{equation}

## 3 Determinant Quantum Monte Carlo algorithm

The simulation of the Hubbard model is a classical Monte Carlo problem. The Hubbard-Stratanovich variables or configurations $h$ are sampled such that the follow the probability distribution $\mathcal{P}(h)$. The determinant QMC (DQMC) algorithm can be summarized by the following steps:

First, the configuration $h$ is initialized with an array of randomly distributed values of $\pm 1$. Starting from the time slice $l=1$, a change in the Hubbard-Stratanovich field on the lattice site $i=1$ is proposed:
\begin{equation}
    h^\prime_{1, 1} = -h_{1, 1}.
\end{equation}

With the new configuration $h^\prime$ the Metropolis ratio
\begin{equation}
    d_{1, 1} = \frac{\det[M_\uparrow(h^\prime)]  \det[M_\downarrow(h^\prime)]}{\det[M_\uparrow(h)]  \det[M_\downarrow(h)]},
\end{equation}

can be computed. A random number generator is then used to sample a uniformly distributed random number $r$. If $r < d_{1, 1}$, the proposed update to the configuration is accepted:
\begin{equation}
    h = h^\prime.
\end{equation}

After all lattice sites $i = 1, \dots, N$ of the imaginary time slice $l=1$ have been updated, the procedure is continued with the next time slice $l=2$ until all imaginary time slices $l=1, \dots, L$ are updated. This concludes one \emph{sweep} through the Hubbard-Stratanovich field. After a few hundred "warmup-sweeps" have been performed, measurements can be made after completing an entire set of updates to all space-time points of the system (see section \ref{sec:measurements}). The measurement results have to be normalized with the number of "measurement-sweeps". One iteration (sweep) of the DQMC algorithm can be summarized as

1. Set $l=1$ and $i=1$
2. Propose new coonfiguration $h^\prime$ by flipping spin:
    \begin{equation}
        h^\prime_{i, l} = -h_{i, l}.
    \end{equation}
3. Compute Metropolis ratio:
    \begin{equation}
        d_{i, l} = \frac{\det[M_\uparrow(h^\prime)]  \det[M_\downarrow(h^\prime)]}{\det[M_\uparrow(h)]  \det[M_\downarrow(h)]},
    \end{equation}
    where
    \begin{equation}
        M_\sigma(h) = I + B_{l-1,\sigma} \dots B_{1, \sigma} B_{L,\sigma}B_{L-1,\sigma} \dots B_{l,\sigma}.
    \end{equation}

4. Sample random number $r$
5. Accept new configuration, if $r < d_{i, l}$:
    \begin{equation}
        h_{i, l} = \begin{cases} h^\prime_{i, l} &\text{if } r < d_{i, l} \\
        h_{i, l} &\text{else }
        \end{cases}
    \end{equation}
6. Increment site $i = i+1$ if $i < N$
7. Increment time slice $l = l+1$ if $i=N$

### 3.1 Rank-one update scheme

The one-site update of the inner DQMC loop leads to a simple rank-one update of the matrix $M_\sigma(h)$, which leads to an efficient method of computing the Metropolis ratio $d_{i,l}$ and Green's function $G_\sigma$.

The DQMC simulation requires the computation of $NL$ Metropolis ratios

\begin{equation}
    d = \frac{\det[M_\uparrow(h^\prime)]  \det[M_\downarrow(h^\prime)]}{\det[M_\uparrow(h)]  \det[M_\downarrow(h)]}
\end{equation}

for the configurations $h = (h_1, h_2, \dots, h_l)$ and $h^\prime = (h^\prime_1, h^\prime_2, \dots, h^\prime_L)$ per sweep. The matrix $M_\sigma(h)$, defined in \eqref{eq:mMatrix}, is given by

\begin{equation}
    M_\sigma = I + B_{L,\sigma} B_{L-1,\sigma} \dots B_{1,\sigma}.
\end{equation}

with the time step matrices
\begin{equation}
    B_{l,\sigma} = e^{-\Delta\tau K_\sigma} e^{\sigma \nu V_l(h_l)}.
\end{equation}

The Green's function of the configuration $h$ is defined as
\begin{equation}
    G_\sigma(h) = M_\sigma^{-1}.
\end{equation}

The elements of the configuration $h$ and $h^\prime$ differ only by one element at a specific time slice $l$ and spatial site $i$, which gets inverted by a proposed update:
\begin{equation}
    h^\prime_{i,l} = - h_{i, l}.
\end{equation}

During one sweep of the QMC iteration the inner loops run over the $l=1, \dots, L$ imaginary time slices and $i=1, \dots, N$ lattice sites. Starting with the first time slice, $l=1$, the Metropolis ratio $d_{i, 1}$ for each lattice site $i$ is given by
\begin{equation}
    d_{i, 1} = d_\uparrow d_\downarrow,
\end{equation}
where for $\sigma = \pm 1$
\begin{equation}
    d_\sigma = 1 + \alpha_{i,\sigma} \left[ 1 - \boldsymbol{e}_i^T M_\sigma^{-1}(h) \boldsymbol{e}_i \right] = 1 + \alpha_{i, \sigma} \left[ 1 - G_{ii}^\sigma(h) \right]
\end{equation}
and
\begin{equation}
    \alpha_{i,\sigma} = e^{-2\sigma \nu h_{i,1}} - 1.
\end{equation}

The Metropolis ration $d_{i, 1}$ can therefore be obtained by computing the inverse of the matrix $M_\sigma(h)$, which corresponds to the Green's function $G_\sigma(h)$. If $G_\sigma(h)$ has already been computed in a previous step, then it is essentially free to compute the Metropolis ratio.\\
If the proposed update $h^\prime$ to the configuration is accepted, the Green's function needs to be updated by a rank-1 matrix update:
\begin{equation}
    G_\sigma(h^\prime) = G_\sigma(h) - \frac{\alpha_{i, \sigma}}{d_{i, 1}} \boldsymbol{u}_\sigma \boldsymbol{w}_\sigma^T,
\end{equation}
where
\begin{equation}
    \boldsymbol{u}_\sigma = \left[I - G_\sigma(h)\right] \boldsymbol{e}_i, \qquad \boldsymbol{w}_\sigma = \left[G_\sigma(h)\right]^T \boldsymbol{e}_i.
\end{equation}

After all spatial sites $i=1, \dots, N$ have been updated, we can move to the next time slice $l=2$. The matrices $M_\sigma(h)$ and $M_\sigma(h^\prime)$ can be written as
\begin{equation}
\begin{split}
    M_\sigma(h) &= B_{1, \sigma}^{-1}(h_1) \hat{M}_\sigma(h) B_{1, \sigma}(h_1)\\
    M_\sigma(h^\prime) &= B_{1, \sigma}^{-1}(h_1^\prime) \hat{M}_\sigma(h^\prime) B_{1, \sigma}(h_1^\prime)
\end{split}
\end{equation}
where
\begin{equation}
\begin{split}
    \hat{M}_\sigma(h) &= I + B_{1, \sigma}(h_1) B_{L, \sigma}(h_L) B_{L-1, \sigma}(h_{L-1}) \dots B_{2, \sigma}(h_2)\\
    \hat{M}_\sigma(h^\prime) &= I + B_{1, \sigma}(h_1^\prime) B_{L, \sigma}(h_L^\prime) B_{L-1, \sigma}(h_{L-1}^\prime) \dots B_{2, \sigma}(h_2^\prime)
\end{split}
\end{equation}

are obtained by a cyclic permutation of the time step matrices $B_{l,\sigma}(h_l)$. The Metropolis ratios $d_{i, 2}$ corresponding to the time slice $l=2$ can therefore be written as
\begin{equation}
    d_{i,2} = \frac{\det\big[M_\uparrow(h^\prime)\big] \det\big[M_\downarrow(h^\prime)\big]}{\det\big[M_\uparrow(h)\big]  \det\big[M_\downarrow(h)\big]}
    = \frac{\det\big[\hat{M}_\uparrow(h^\prime)\big]  \det\big[\hat{M}_\downarrow(h^\prime)\big]}{\det\big[\hat{M}_\uparrow(h)\big]  \det\big[\hat{M}_\downarrow(h)\big]}.
\end{equation}

The associated Green's functions are given by "wrapping":
\begin{equation}
    \hat{G}_\sigma(h) = B^{-1}_{1,\sigma}(h_1) G_\sigma(h) B_{1,\sigma}(h_1).
\end{equation}

Wrapping the Green's function ensures that the configurations $h_2$ and $h_2^\prime$ associated with the time slice $l=2$ appear at the same location of the matrices $\hat{M}_\sigma(h)$ and $\hat{M}_\sigma(h^\prime)$ as the configurations $h_1$ and $h_1^\prime$ at the time slice $l=1$. Therefore the same formulation can be used for the time slice $l=2$ as for the time slice $l=1$ to compute the Metropolis ratio and update the Green's functions. This procedure can be repeated for all the remaining time slices $l=3, 4, \dots, L$.

```python

```


```python
import sympy as sp

sp.init_printing()
```

# 03_StellarDynamics_HW_Chap04

In class, we discussed the formation of a protostar, Hayashi track.
There are three different state of Hayashi track.Your mission is
- A)The relationship among surface temperature, stellar mass and stellar radius in different state of Hayashi track. (45%)
- B)A)The relationship among luminosity, stellar mass and stellar radius in different state of Hayashi track. (45%)
- C)Explain the mechanism of Hayashi track, why we can roughly disassemble the Hayashi track and what is the difference among the three states?(10%)

Hint: The answers of A and B are polynomials. What you have to do is to find out the power of mass and radius.

```python
L, T, M, P, kappa = sp.symbols('L T M P kappa')

# expression of luminosity
exprL = sp.Eq(L, sp.simplify('4 * pi * R^2 * T^4 * c'))
exprL
```

```python
exprDp = sp.Derivative(P, sp.simplify('r'))
exprDp = sp.Eq(kappa * P, exprDp)
exprDp
```

```python
# gavitation
exprGrav = sp.simplify('- G * m * rho / R^2')
exprGrav = sp.Eq(exprDp.rhs, exprGrav)
exprGrav
```

```python
# substitute dP / dr
exprDp = exprDp.subs({exprDp.rhs: exprGrav.rhs})
exprDp
```

```python
# substitute kappa and rho
exprDp = exprDp.subs({kappa : sp.simplify('kappa_0 * P^a * T^b'),
                      sp.simplify('rho') : P / T})
exprDp
```

```python
# simplify the expression
exprDp = sp.simplify(exprDp)
exprDp
```

```python
# Adiabatic process
gamma = sp.Symbol('gamma')
exprP = (T**(-gamma))**(1 / (1 - gamma))
sp.Eq(P, exprP)
```

```python
# for gamma = 5/3
exprP = sp.powdenest(exprP.subs(gamma, 5 / 3), force=1)
sp.Eq(P, exprP)
```

```python
# substitute back into the exprDp
exprDp = exprDp.subs(P, exprP)
exprDp
```

```python
# solve for T
exprSolveT = sp.solve(exprDp, T, rational=0)
exprSolveT
```

```python
# it seems we only need second solution
exprSolveT = exprSolveT[1]
exprSolveT
```

```python
# expand powers based on bases
exprSolveT = sp.expand_power_base(exprSolveT, force=1)
sp.Eq(T, exprSolveT)
```

The above equation stands for the relation between $T$ and $R, m$ based on power law.

```python
# recall this equation
exprL
```

```python
# substitute T with exprSolveT, the T relation based on R and m
exprSolveL = exprL.subs(T, exprSolveT)
exprSolveL
```

Let's simplify both L and T expressions for taking all constants to be 1.

```python
# simplify L
exprSolveL = exprSolveL.subs({'4 * pi' : 1, 'G' : 1, 'c' : 1,
                 'kappa_0' : 1})
sp.powsimp(exprSolveL, deep=1, force=1)
```

Oh... this is the expression the program thinks is the most simplistic...

```python
# simplify T
exprSolveT = exprSolveT.subs({'G' : 1, 'kappa_0' : 1})
exprSolveT = sp.powsimp(exprSolveT, deep=1, force=1)
exprSolveT = sp.Eq(T, exprSolveT)
exprSolveT
```

# Question 1
Use expression of $T$ with $a, b$ in different stages:

|stages| `a` | `b` |
| ---- | --- | --- |
| (1)  | 0.7 | 0.6 |
| (2)  | 5.3 | 9.4 |

(BYW, did 老闕 provided `a` `b` for stage (3)?)

```python
# 1) low-density stage

```


```python
%matplotlib inline
```

# Korteweg-deVries equation -- Interaction of solitons

This example solves the Korteweg-deVries (KdV) equation, given by

\begin{align}\partial_t u = - \delta^2 \partial_x^3 u  -u \partial_x u,\end{align}

wherein the evolution of the real-valued field $u = u(t, x)$ is
determined by the parameter $\delta=0.022$.

The propagation scenario below considers the initial condition

\begin{align}u_0(x) = \cos(\pi x), \quad 0\leq x \leq 2,\end{align}

and uses a pseudospectral integrating factor method, i.e. the "Runge-Kutta in
the interaction picture" method implemented in the `py-fmas` libragy, to
reproduce the data shown in Fig. 2 of Ref. [1].

References:
    [1] N. J. Zabusky, M. D. Kruskal, Interaction of "Solitons" in a
    Collisionless Plasma and the Recurrence of Initial States, Phys. Rev. Lett.
    15 (1965) 240, https://doi.org/10.1103/PhysRevLett.15.240.

.. codeauthor:: Oliver Melchert <melchert@iqo.uni-hannover.de>

```python
import fmas
import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt
import matplotlib.colors as col
from fmas.config import FTFREQ, FT, IFT, C0
from fmas.solver import IFM_RK4IP

def plot_evolution_KdV(t, x, u):

    def _setColorbar(im, refPos):
        """colorbar helper"""
        x0, y0, w, h = refPos.x0, refPos.y0, refPos.width, refPos.height
        cax = f.add_axes([x0, y0+1.02*h, w, 0.03*h])
        cbar = f.colorbar(im, cax=cax, orientation='horizontal')
        cbar.ax.tick_params(color='k',
                            labelcolor='k',
                            bottom=False,
                            direction='out',
                            labelbottom=False,
                            labeltop=True,
                            top=True,
                            size=4,
                            pad=0
                            )

        cbar.set_ticks((-1,-0.5,0,0.5,1))
        cbar.ax.tick_params(which="minor", bottom=False, top=False )
        return cbar

    f, ax1 = plt.subplots(1, 1, sharey=True, figsize=(2.5,4))
    plt.subplots_adjust(left=0.2, right=0.9, bottom=0.12, top=0.84)
    cmap=mpl.cm.get_cmap('coolwarm')

    im1 = ax1.pcolorfast(x, t, u[:-1,:-1],
                         norm=col.Normalize(vmin=-1,vmax=1),
                         cmap=cmap
                         )
    cbar1 = _setColorbar(im1,ax1.get_position())
    cbar1.ax.set_title(r"$u(x,t)$",color='k',y=3.5)
    ax1.xaxis.set_ticks_position('bottom')
    ax1.yaxis.set_ticks_position('left')
    ax1.set_xlim(x[0],x[-1])
    ax1.set_xticks((0,0.5,1.,1.5,2))
    ax1.set_ylim([0.,t.max()])
    ax1.set_xlabel(r"Distance $x$")
    ax1.set_ylabel(r"Time $t$")

    plt.show()

def main():
    # -- DEFINE SIMULATION PARAMETERS
    x_min = 0.
    x_max = 2.
    Nx = 512
    t_min = 0.
    t_max = 6.0
    Nt = 30000
    n_skip = 10
    delta = 0.022

    # -- INITIALIZATION STAGE
    # ... COMPUTATIONAL DOMAIN
    x = np.linspace(x_min, x_max, Nx, endpoint=False)
    k = FTFREQ(x.size,d=x[1]-x[0])*2*np.pi
    # ... KORTEWEG DEVRIES MODEL
    Lk = -1j*k*k*k*delta*delta
    Nk_fun = lambda uk: 0.5j*k*FT(IFT(uk)**2)
    # ... SOLVER BASED ON INTEGRATING FACTOR METHOD
    solver = IFM_RK4IP(Lk, Nk_fun)
    # ... INITIAL CONDITION
    u_0x = np.cos(np.pi*x)
    solver.set_initial_condition(k, FT(u_0x))

    # -- RUN SIMULATION
    solver.propagate(z_range = t_max, n_steps = Nt, n_skip = n_skip)

    plot_evolution_KdV( solver.z, x, np.real(solver.utz))

if __name__=='__main__':
    main()
```


###### Content under Creative Commons Attribution license CC-BY 4.0, code under MIT license (c)2014 L.A. Barba, G.F. Forsyth, C.D. Cooper. Partly based on content  by David Ketcheson, also under CC-BY.

# Phugoid model: bonus!

_The phugoid model of glider flight_ has been such a fun problem to showcase the power of numerical solution of differential equations, we thought you'd enjoy a bonus notebook. The previous lessons were:

* [Phugoid motion](http://nbviewer.ipython.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/01_phugoid/01_01_Phugoid_Theory.ipynb) —Lays the groundwork for our fun problem, with some context, a little history and a description of the physics of phugoids: curves representing the trajectory of a glider exchanging potential and kinetic energy, with no drag.
* [Phugoid oscillation](http://nbviewer.ipython.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/01_phugoid/01_02_Phugoid_Oscillation.ipynb) —Develops the simple harmonic motion of an aircraft experiencing a small perturbation from the horizontal trajectory: our opportunity to introduce Euler's method, and study its convergence via an exact solution.
* [Full phugoid motion](http://nbviewer.ipython.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/01_phugoid/01_03_PhugoidFullModel.ipynb) —The full model takes into account the force of drag and results in a system of two nonlinear equations. We obtain the trajectories using Euler's method in vectorized form, introduce grid-convergence analysis and finish with the paper-airplane challenge!

That is a fantastic foundation for numerical methods. It's a good time to complement it with some theory: the first screencast of the course uses Taylor series to show that _Euler's method is a first-order method_, and we also show you graphical interpretations. Many problems require a more accurate method, though: second order or higher. Among the most popular higher-order methods that we can mention are the _Runge-Kutta methods_, developed around 1900: more than 100 years after Euler published his book containing the method now named after him!

### Euler's method is a first-order method

In this screencast, we use a Taylor series expansion to analyze Euler's method and show that it incurs a truncation error of first order. We also use a graphical interpretation to motivate the _modified_ Euler method, which achieves second order.

```
from IPython.display import YouTubeVideo
YouTubeVideo('6i6qhqDCViA')
```

## Second-order methods

The notebook on phugoid oscillation (lesson 2) included a study of the accuracy obtained with Euler's method, using the exact solution for the simple harmonic motion. We made a _convergence plot_ and saw that as $\Delta t$ gets smaller, the error also gets smaller.

We could have drawn a line with a slope equal to 1 on that log-log plot, and you would have seen that it was parallel to the convergence line. A slope equal to 1 on a log-log convergence plot is an indication that we have a first-order method: the error scales as ${\mathcal O}(\Delta t)$.

In lesson 3, using the full phugoid model (which is nonlinear and does not have an exact solution), we did a _grid-convergence study_ with three different grids, and obtained the _observed_ order of convergence—it was very close to 1, indicating a slope of 1 on a log-log plot.

Another way to look at an ${\mathcal O}(\Delta t)$ method is to say that the error scales _linearly_ with the step size, or that they are proportional:

$$ e \propto \Delta t.$$

where $e$ stands for the error. To get more accuracy, we could use a _second-order_ method, in which the error is ${\mathcal O}(\Delta t^2)$. In general, we say that a method is of order $p$ when the error is proportional to $(\Delta t)^p$.

In the screencast titled "Euler's method is a first-order method," we used a graphical interpretation to get an idea for improving it: by estimating an intermediate point, like the **midpoint**, we can get a better approximation of the area under the curve of $u^\prime$. The scheme has two steps and is written as:

\begin{align}
u_{n+1/2}   & = u_n + \frac{\Delta t}{2} f(u_n) \\
u_{n+1} & = u_n + \Delta t \,\, f(u_{n+1/2}).
\end{align}

This method is known as the *explicit midpoint method* or the *modified Euler method*, and it is a second-order method. Notice that we had to apply the right-hand side, $~f(u)$, twice. This idea can be extended: we could imagine estimating additional points between $u_{n}$ and $u_{n+1}$ and evaulating $~f(u)$ at the intermediate points to get higher accuracy—that's the idea behind Runge-Kutta methods.

### Runge-Kutta methods

In the modified Euler method, we improve the accuracy over Euler's method by evaluating the right-hand side of the differential equation at an intermediate point: the midpoint. The same idea can be applied again, and the function $f(u)$ can be evaluated at more intermediate points, improving the accuracy even more. This is the basis of the famous *Runge-Kutta (RK) methods*, going back to Carl Runge and Martin Kutta. The modified Euler method corresponds to _second-order_ Runge-Kutta.

Here's a bit of historical coincidence that will blow your mind: Carl Runge's daughter Iris—an accomplished applied mathematician in her own right—worked assiduously over the summer of 1909 to translate Lanchester's _"Aerodonetics."_ She also reproduced his graphical method to draw the phugoid curves (Tobies, 2012).

### Phugoid model with 2nd-order RK

Let's compute the motion of a glider under the full phugoid model using the second-order Runge-Kutta method. We'll build on the _paper airplane challenge_ of lesson 3 now, and look for the horizontal distance that the plane travels until the moment it touches the ground.

As usual, let's start by importing the libraries and modules that we need, and setting up the model parameters. We also set some default plotting formats using the [`rcParams`](http://matplotlib.org/api/matplotlib_configuration_api.html#matplotlib.rcParams) module.

```
from math import sin, cos, log
import numpy
from matplotlib import pyplot
%matplotlib inline
from matplotlib import rcParams
rcParams['font.family'] = 'serif'
rcParams['font.size'] = 16
```

In the paper-airplane challenge of lesson 3, we suggested an $L/D=5.0$ as a realistic value for paper airplanes, according to experiments, and a trim velocity of 4.9 m/s. Let's start with those values, but you could experiment changing these a bit. _What do you think will happen if you make $L/D$ higher?_

```
# model parameters:
g = 9.8      # gravity in m s^{-2}
v_t = 4.9    # trim velocity in m s^{-1}
C_D = 1/5.0  # drag coefficient --- or D/L if C_L=1
C_L = 1.0    # for convenience, use C_L = 1

### set initial conditions ###
v0 = 6.5     # start at the trim velocity (or add a delta)
theta0 = -0.1 # initial angle of trajectory
x0 = 0.0     # horizotal position is arbitrary
y0 = 2.0     # initial altitude
```

Among the initial parameters that we suggest for your first experiment, we are starting with a velocity a little higher than the trim velocity, launch the paper airplane with a negative initial angle, and take the initial height to be 2 meters—all sound like reasonable choices.

Now, we can define a few functions to carry out the computation:
* The right-hand side of the phugoid model from [Lesson 3](http://nbviewer.ipython.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/01_phugoid/01_03_PhugoidFullModel.ipynb),
* One step of the Euler's method that we learned in [Lesson 2](http://nbviewer.ipython.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/01_phugoid/01_02_Phugoid_Oscillation.ipynb), and
* Differences with respect to a fine grid, as in [Lesson 3](http://nbviewer.ipython.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/01_phugoid/01_03_PhugoidFullModel.ipynb).

```
def f(u):
    """Returns the right-hand side of the phugoid system of equations.

    Parameters
    ----------
    u : array of float
        array containing the solution at time n.

    Returns
    -------
    dudt : array of float
        array containing the RHS given u.
    """

    v = u[0]
    theta = u[1]
    x = u[2]
    y = u[3]
    return numpy.array([-g*sin(theta) - C_D/C_L*g/v_t**2*v**2,
                      -g*cos(theta)/v + g/v_t**2*v,
                      v*cos(theta),
                      v*sin(theta)])

def euler_step(u, f, dt):
    """Returns the solution at the next time-step using Euler's method.

    Parameters
    ----------
    u : array of float
        solution at the previous time-step.
    f : function
        function to compute the right hand-side of the system of equation.
    dt : float
        time-increment.

    Returns
    -------
    u_n_plus_1 : array of float
        approximate solution at the next time step.
    """

    return u + dt * f(u)

def get_diffgrid(u_current, u_fine, dt):
    """Returns the difference between one grid and the fine one using L-1 norm.

    Parameters
    ----------
    u_current : array of float
        solution on the current grid.
    u_finest : array of float
        solution on the fine grid.
    dt : float
        time-increment on the current grid.

    Returns
    -------
    diffgrid : float
        difference computed in the L-1 norm.
    """

    N_current = len(u_current[:,0])
    N_fine = len(u_fine[:,0])

    grid_size_ratio = numpy.ceil(N_fine/float(N_current))

    diffgrid = dt * numpy.sum( numpy.abs(\
            u_current[:,2]- u_fine[::grid_size_ratio,2]))

    return diffgrid

```

Next, we also need to define the function `rk2_step()` that computes the next time step using the *modified Euler* method of equations $(1)$ and $(2)$, above, otherwise known as 2nd-order Runge-Kutta or RK2. This function will be called over and over again within the time loop.

```
def rk2_step(u, f, dt):
    """Returns the solution at the next time-step using 2nd-order Runge-Kutta.

    Parameters
    ----------
    u : array of float
        solution at the previous time-step.
    f : function
        function to compute the right hand-side of the system of equation.
    dt : float
        time-increment.

    Returns
    -------
    u_n_plus_1 : array of float
        solution at the next time step.
    """
    u_star = u + 0.5*dt*f(u)
    return u + dt*f(u_star)
```

Like in [Lesson 3](http://nbviewer.ipython.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/01_phugoid/01_03_PhugoidFullModel.ipynb), we first need to set up the time discretization, then initialize arrays to save the solution and we are set to go! The only difference this time is that we are using _both_ Euler's method and 2nd-order Runge-Kutta to get a solution, to compare the two.

```
# set time-increment and discretize the time
T  = 15.0                           # final time
dt = 0.01                           # set time-increment
N  = int(T/dt) + 1                  # number of time-steps

# set initial conditions
u_euler = numpy.empty((N, 4))
u_rk2   = numpy.empty((N, 4))

# initialize the array containing the solution for each time-step
u_euler[0] = numpy.array([v0, theta0, x0, y0])
u_rk2[0]   = numpy.array([v0, theta0, x0, y0])

# use a for loop to call the function rk2_step()
for n in range(N-1):

    u_euler[n+1] = euler_step(u_euler[n], f, dt)
    u_rk2[n+1]   = rk2_step(u_rk2[n], f, dt)
```

Now we can get the position of the glider in time, according to both Euler's method and the 2nd-order Runge-Kutta method, by extracting the appropriate portions of the solution arrays:

```
x_euler = u_euler[:,2]
y_euler = u_euler[:,3]
x_rk2 = u_rk2[:,2]
y_rk2 = u_rk2[:,3]
```

##### How far will it fly before touching the ground?

As the $y$-axis measures the vertical coordinate with respect to the ground, negative values of $y$ don't have any physical meaning: the glider would have hit the ground by then! To find out if there are any negative $y$ values we can use the handy function [`numpy.where`](http://docs.scipy.org/doc/numpy/reference/generated/numpy.where.html). This function returns the **indices** of the elements in an array that match a given condition. For example, `numpy.where(y_euler<0)[0]` gives an array of the indices `i` where `y_euler[i]<0` (the `[0]` is necessary as `numpy.where` returns an array, which in this case contains a single line). If no elements of the array match the condition, the array of indices comes out empty.

From the physical problem, we know that once there is one negative value, the glider has hit the ground and all the remaining time-steps are unphysical. Therefore, we are interested in finding the _first_ index where the condition applies, given by `numpy.where(y_euler<0)[0][0]`—do read the  documentation of the function if you need to!

```
# get the index of element of y where altitude becomes negative
idx_negative_euler = numpy.where(y_euler<0.0)[0]
if len(idx_negative_euler)==0:
    idx_ground_euler = N-1
    print ('Euler integration has not touched ground yet!')
else:
    idx_ground_euler = idx_negative_euler[0]

idx_negative_rk2 = numpy.where(y_rk2<0.0)[0]
if len(idx_negative_rk2)==0:
    idx_ground_rk2 = N-1
    print ('Runge-Kutta integration has not touched ground yet!')
else:
    idx_ground_rk2 = idx_negative_rk2[0]
```

##### Do Euler and RK2 produce the same solution?

An easy way to compare the numerical results obtained with the Euler and 2nd-order Runge-Kutta methods is using [`numpy.allclose`](http://docs.scipy.org/doc/numpy/reference/generated/numpy.allclose.html). This function compares each element of two arrays and returns `True` if each comparison is within some relative tolerance. Here, we use the default tolerance: $10^{-5}$.

```
# check to see if the paths match
print('Are the x-values close? {}'.format(numpy.allclose(x_euler, x_rk2)))
print('Are the y-values close? {}'.format(numpy.allclose(y_euler, y_rk2)))
```

    Are the x-values close? False
    Are the y-values close? False

Hmmm, they do differ. Maybe $10^{-5}$ is too tight a tolerance, considering we're using a somewhat coarse grid with first- and second-order methods. Perhaps we can assess this visually, by plotting the glider's path? Study the code below, where we are plotting the path twice, taking a closer look in the second plot by "zooming in" to the beginning of the flight.

```
# plot the glider path
pyplot.figure(figsize=(10,6))
pyplot.subplot(121)
pyplot.grid(True)
pyplot.xlabel('$x$')
pyplot.ylabel('$y$')
pyplot.plot(x_euler[:idx_ground_euler], y_euler[:idx_ground_euler], 'k-', label='Euler')
pyplot.plot(x_rk2[:idx_ground_rk2], y_rk2[:idx_ground_rk2], 'r--', label='RK2')
pyplot.title('distance traveled: {:.3f}'.format(x_rk2[idx_ground_rk2-1]))
pyplot.legend();

# Let's take a closer look!
pyplot.subplot(122)
pyplot.grid(True)
pyplot.xlabel('$x$')
pyplot.ylabel('$y$')
pyplot.plot(x_euler, y_euler, 'k-', label='Euler')
pyplot.plot(x_rk2, y_rk2, 'r--', label='RK2')
pyplot.xlim(0,5)
pyplot.ylim(1.8,2.5);

```

From far away, the Euler and RK2 methods seem to be producing similar answers. However, if we take a closer look, small differences become evident. Keep in mind that we are solving the same equation and both methods will converge to the same solution as we refine the grid. However, they converge to that solution at different rates: RK2 gets more accurate faster, as you make $\Delta t$ smaller.

### Grid-convergence

Just like in [Lesson 3](http://nbviewer.ipython.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/01_phugoid/01_03_PhugoidFullModel.ipynb), we want to do a grid-convergence study with RK2, to see if we indeed observe the expected rate of convergence. It is always an important step in a numerical solution to investigate whether the method is behaving the way we expect it to: this needs to be confirmed experimentally for every new problem we solve and for every new method we apply!

In the code below, a `for`-loop computes the solution on different time grids, with the coarsest and finest grid differing by 100x. We can use the difference between solutions to investigate convergence, as before.

```
# use a for-loop to compute the solution on different grids
dt_values = numpy.array([0.1, 0.05, 0.01, 0.005, 0.001])

u_values = numpy.empty_like(dt_values, dtype=numpy.ndarray)

for i, dt in enumerate(dt_values):

    N = int(T/dt)+1    # number of time-steps

    ### discretize the time t ###
    t = numpy.linspace(0.0, T, N)

    # initialize the array containing the solution for each time-step
    u = numpy.empty((N, 4))
    u[0] = numpy.array([v0, theta0, x0, y0])

    # time loop
    for n in range(N-1):
        u[n+1] = rk2_step(u[n], f, dt)

    # store the value of u related to one grid
    u_values[i] = u
```

Once those runs are done, we compute the difference between each numerical solution and the fine-grid solution.

```
# compute diffgrid
diffgrid = numpy.empty_like(dt_values)
for i, dt in enumerate(dt_values):

    diffgrid[i] = get_diffgrid(u_values[i], u_values[-1], dt)
```

And now we plot!

```
# plot using the matplotlib function loglog()
pyplot.figure(figsize=(6,6))
pyplot.grid(True)
pyplot.xlabel(r'$\Delta t$', fontsize=18)
pyplot.ylabel(r'$L_1$-norm of the grid differences', fontsize=18)
pyplot.xlim(1e-4,1)
pyplot.ylim(1e-4,1)
pyplot.axis('equal')
pyplot.loglog(dt_values[:-1], diffgrid[:-1], color='k', ls='--', lw=2, marker='o');
```

This is looking good! The difference relative to our fine-grid solution is decreasing with the mesh size at a faster rate than in [Lesson 3](http://nbviewer.ipython.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/01_phugoid/01_03_PhugoidFullModel.ipynb), but *how much faster?* When we computed the observed order of convergence with Euler's method, we got a value close to 1—it's a first-order method. Can you guess what we'll get now with RK2?

To compute the observed order of convergence, we use three grid resolutions that are refined at a constant rate, in this case $r=2$.

```
# check convergence rate
r = 2
h = 0.001

dt_values = numpy.array([h, r*h, r**2*h])

u_values = numpy.empty_like(dt_values, dtype=numpy.ndarray)

for i, dt in enumerate(dt_values):

    N = int(T/dt)+1    # number of time-steps

    ### discretize the time t ###
    t = numpy.linspace(0.0, T, N)

    # initialize the array containing the solution for each time-step
    u = numpy.empty((N, 4))
    u[0] = numpy.array([v0, theta0, x0, y0])

    # time loop
    for n in range(N-1):
        ### call rk2_step() ###
        u[n+1] = rk2_step(u[n], f, dt)

    # store the value of u related to one grid
    u_values[i] = u

# calculate the order of convergence
alpha = (log(get_diffgrid(u_values[2], u_values[1], dt_values[2]))
         - log(get_diffgrid(u_values[1], u_values[0], dt_values[1]))) / log(r)

print('The order of convergence is alpha = {:.3f}'.format(alpha))
```

    The order of convergence is alpha = 1.983

Probably you're not too surprised to see that the observed order of convergence is close to $2$. Because we used a second-order method! This means that the numerical solution is converging with the grid resolution twice as fast compared with Euler's method in [Lesson 3](http://nbviewer.ipython.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/01_phugoid/01_03_PhugoidFullModel.ipynb), or in other words, the error scales as ${\mathcal O}(\Delta t^2)$. That is a lot faster! However, we are paying a price here: second-order Runge-Kutta requires more computations per iteration.

##### Challenge task

How much longer does it take to get the solution with RK2, compared to Euler's method? Run the same solution (same time grid, same parameters), but find a way to *time* the calculation with Python, and compare the runtimes.

## Multi-step methods

The screencast *"Euler's method is a first-order method"* motivated graphically an idea to get increased accuracy: using intermediate points between $u_{n}$ and $u_{n+1}$ and evaluating the right-hand side of the differential equation at those intermediate points. The idea is to somehow get a better approximation using more data from the function $f(u)$.

Another way to bring more information about $f(u)$ into the numerical solution is to look at time data $t\lt t_{n}$. For example, we can involve in the calculation of the solution $u_{n+1}$ the known solution at $u_{n-1}$, in addition to $u_{n}$. Schemes that use this idea are called _multi-step methods_.

A classical multi-step method achieves second order by applying a _centered difference_ approximation of the derivative $u'$:

$$ u'(t) \approx \frac{u_{n+1} - u_{n-1}}{2\Delta t}.$$

Isolate the future value of the solution $u_{n+1}$ and apply the differential equation $u'=f(u)$, to get the following formula for this method:

$$ u_{n+1} = u_{n-1} + 2\Delta t \, f(u_n),$$

This scheme is known as the **leapfrog method**. Notice that it is using the right-hand side of the differential equation, $f(u)$, evaluated at the _midpoint_ between $u_{n-1}$ and $u_{n+1}$, where the time interval between these two solutions is $2\Delta t$. Why is it called "leapfrog"? If you imagine for a moment all of the _even_ indices $n$ of the numerical solution, you notice that these solution values are computed using the slope estimated from _odd_ values $n$, and vice-versa.

Let's define a function that computes the numerical solution using the leapfrog method:

```
def leapfrog_step(unm1, u, f, dt):
    """Returns the solution time-step n+1) using Euler's method.

    Parameters
    ----------
    unm1 : array of float
        solution at time-step n-1.
    u : array of float
        solution at time-step n.
    f : function
        function to compute the right hand-side of the system of equation.
    dt : float
        time-increment.

    Returns
    -------
    u_n_plus_1 : array of float
        solution at time-step n+1.
    """
    return unm1 + 2.0*dt*f(u)
```

But wait ... what will we do at the _initial_ time step, when we don't have information for $u_{n-1}$? This is an issue with all multi-step methods: we say that they are _not self-starting_. In the first time step, we need to use another method to get the first "kick"—either Euler's method or 2nd-order Runge Kutta could do: let's use RK2, since it's also second order.

For this calculation, we are going to reenter the model parameters in the code cell below, so that later on we can experiment here using the leapfrog method and different starting values. At the end of this notebook, we'll give you some other model parameters to try that will create a very interesting situation!

```
# model parameters:
g = 9.8      # gravity in m s^{-2}
v_t = 4.9    # trim velocity in m s^{-1}
C_D = 1/5.0  # drag coefficient --- or D/L if C_L=1
C_L = 1.0    # for convenience, use C_L = 1

### set initial conditions ###
v0 = 6.5     # start at the trim velocity (or add a delta)
theta0 = -0.1 # initial angle of trajectory
x0 = 0.0     # horizotal position is arbitrary
y0 = 2.0     # initial altitude

# set time-increment and discretize the time
T  = 15.0                           # final time
dt = 0.01                             # set time-increment
N  = int(T/dt) + 1                   # number of time-steps

# set initial conditions
u_leapfrog = numpy.empty((N, 4))

# initialize the array containing the solution for each time-step
u_leapfrog[0] = numpy.array([v0, theta0, x0, y0])

# first step using RK2
u_leapfrog[1] = rk2_step(u_leapfrog[0], f, dt)
```

Now we have all the required information to loop in time using the leapfrog method. The code cell below calls the leapfrog function for each time step.

```
# use a for loop to call the function leapfrog_step()
for n in range(1,N-1):

    u_leapfrog[n+1] = leapfrog_step(u_leapfrog[n-1], u_leapfrog[n], f, dt)
```

Like before, we extract from the solution array the information about the glider's position in time and find where it reaches the ground.

```
# get the glider position in time
x_leapfrog = u_leapfrog[:,2]
y_leapfrog = u_leapfrog[:,3]

# get the index of element of y where altitude becomes negative
idx_negative_leapfrog = numpy.where(y_leapfrog<0.0)[0]

if len(idx_negative_leapfrog)==0:
    idx_ground_leapfrog = N-1
    print ('The glider has not reached the ground yet!')
else:
    idx_ground_leapfrog = idx_negative_leapfrog[0]
```

Plotting the glider's trajectory with both the leapfrog and RK2 methods, we find that the solutions are very close to each other now: we don't see the differences that were apparent when we compared Euler's method and RK2.

```
# plot the glider path
pyplot.figure(figsize=(11,8))
pyplot.subplot(121)
pyplot.grid(True)
pyplot.xlabel('$x$')
pyplot.ylabel('$y$')
pyplot.plot(x_leapfrog[:idx_ground_leapfrog], y_leapfrog[:idx_ground_leapfrog], color='k', ls='-', lw=2)
pyplot.title('distance traveled: {:.3f}'.format(x_leapfrog[idx_ground_leapfrog-1]), fontsize=18);

# Let's take a closer look!
pyplot.subplot(122)
pyplot.grid(True)
pyplot.xlabel('$x$')
pyplot.ylabel('$y$')
pyplot.plot(x_leapfrog[:idx_ground_leapfrog], y_leapfrog[:idx_ground_leapfrog], color='k', ls=':', lw=2)
pyplot.plot(x_rk2, y_rk2, 'r--', label='RK2')
pyplot.xlim(0,5)
pyplot.ylim(1.8,2.5);
```

What about the observed order of convergence? We'll repeat the process we have used before, with a grid-refinement ratio $r=2$ ... here we go:

```
# check convergence rate
r = 2
h = 0.001

dt_values = numpy.array([h, r*h, r**2*h])

u_values = numpy.empty_like(dt_values, dtype=numpy.ndarray)

for i, dt in enumerate(dt_values):

    N = int(T/dt) + 1   # number of time-steps

    ### discretize the time t ###
    t = numpy.linspace(0.0, T, N)

    # initialize the array containing the solution for each time-step
    u = numpy.empty((N, 4))
    u[0] = numpy.array([v0, theta0, x0, y0])

    # time loop
    u[1] = rk2_step(u[0], f, dt)
    for n in range(1, N-1):
        u[n+1] = leapfrog_step(u[n-1], u[n], f, dt)

    # store the value of u related to one grid
    u_values[i] = u

# calculate the order of convergence
alpha = (log(get_diffgrid(u_values[2], u_values[1], dt_values[2]))
         - log(get_diffgrid(u_values[1], u_values[0], dt_values[1]))) / log(r)

print('The order of convergence is alpha = {:.3f}'.format(alpha))
```

    The order of convergence is alpha = 2.186

We now have numerical evidence that our calculation with the leapfrog method indeed exhibits second-order convergence, i.e., the method is ${\mathcal O}(\Delta t^2)$. _The leapfrog method is a second-order method_. Good job!

### But chew on this ...

Go back to the cell that re-enters the model parameters, just above the leapfrog-method time loop, and change the following: the initial height `y0` to 25, and the final time `T` to 36. Now re-run the leapfrog calculation and the two code cells below that, which extract the glider's position and plot it.

_What is going on?_

## Reference

Tobies, R. "Iris Runge: A life at the crossroads of mathematics, science and industry," Springer Basel, 1st ed. (2012). [Read on Google books, page 73](http://books.google.com/books?id=EDm0eQqFUQ4C&lpg=PA73&dq=%22I%20have%20been%20making%20good%20progress%20with%20Lanchester.%20The%20second%20chapter%20is%20already%20on%20your%20desk%22&pg=PA73#v=onepage&q=%22I%20have%20been%20making%20good%20progress%20with%20Lanchester.%20The%20second%20chapter%20is%20already%20on%20your%20desk%22&f=false).

---

######The cell below loads the style of the notebook.

```
from IPython.core.display import HTML
css_file = '../../styles/numericalmoocstyle.css'
HTML(open(css_file, "r").read())
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Nixie+One' rel='stylesheet' type='text/css'>
<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: rgb(245,245,245);
}

div.cell { /* set cell width */
    width: 750px;
}

div #notebook { /* centre the content */
    background: #fff; /* white background for content */
    width: 1000px;
    margin: auto;
    padding-left: 0em;
}

#notebook li { /* More space between bullet points */
margin-top:0.8em;
}

/* draw border around running cells */
div.cell.border-box-sizing.code_cell.running {
    border: 1px solid #111;
}

/* Put a solid color box around each cell and its output, visually linking them*/
div.cell.code_cell {
    background-color: rgb(256,256,256);
    border-radius: 0px;
    padding: 0.5em;
    margin-left:1em;
    margin-top: 1em;
}

div.text_cell_render{
    font-family: 'Alegreya Sans' sans-serif;
    line-height: 140%;
    font-size: 125%;
    font-weight: 400;
    width:600px;
    margin-left:auto;
    margin-right:auto;
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Nixie One', serif;
    font-style:regular;
    font-weight: 400;
    font-size: 45pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}
.text_cell_render h2 {
    font-family: 'Nixie One', serif;
    font-weight: 400;
    font-size: 30pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.1em;
    margin-top: 0.3em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Nixie One', serif;
    margin-top:16px;
	font-size: 22pt;
    font-weight: 600;
    margin-bottom: 3px;
    font-style: regular;
    color: rgb(102,102,0);
}

.text_cell_render h4 {    /*Use this for captions*/
    font-family: 'Nixie One', serif;
    font-size: 14pt;
    text-align: center;
    margin-top: 0em;
    margin-bottom: 2em;
    font-style: regular;
}

.text_cell_render h5 {  /*Use this for small titles*/
    font-family: 'Nixie One', sans-serif;
    font-weight: 400;
    font-size: 16pt;
    color: rgb(163,0,0);
    font-style: italic;
    margin-bottom: .1em;
    margin-top: 0.8em;
    display: block;
}

.text_cell_render h6 { /*use this for copyright note*/
    font-family: 'PT Mono', sans-serif;
    font-weight: 300;
    font-size: 9pt;
    line-height: 100%;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

.CodeMirror{
        font-family: "PT Mono";
        font-size: 90%;
}

</style>


# Stiffness matrix for a perfectly square bilinear element.

### Summary
This notebook describes the computational steps required in the computation of the displacement based finite element stiffness matrix for a perfectly square element of side $2h$.

### Stiffness matrix
The displacement based finite elment stiffness matrix can be written like:

$${K^{QP}} = \int\limits_V {B_{ij}^Q{C_{ijkl}}B_{kl}^PdV} $$

where:

$$C = \frac{{E(1 - \nu )}}{{(1 + \nu )(1 - 2\nu )}}\left[ {\begin{array}{*{20}{c}}
1&{\frac{\nu }{{1 - \nu }}}&0\\
{\frac{\nu }{{1 - \nu }}}&1&0\\
0&0&{\frac{{1 - 2\nu }}{{2(1 - \nu )}}}
\end{array}} \right]$$

is the elstic tensor and $B_{ij}^Q$ is the contribution to the strain-displacement interpolator from the $Q$ degree of freedom.

The 4-noded perfectly square element of side $2h$ is shown in the figure below:

 with each node having 2 degrees of freedom corresponding to the rectangular components of the displacement vector.

## Finite element approach

```python
%matplotlib notebook
from __future__ import division
import numpy as np
import sympy as sym
import matplotlib.pyplot as plt
from IPython.display import Image
```

In the displacement based finite element method we assume that the displacemnts vector at any point $x_i$ over the element are expressed, via interpolation, in terms of the nodal displacements like:

$${u_i} = N_i^Q(r){{\hat u}^Q}$$

and where $N_i^K(r)$ is the shape function associated to the $k$-th degree of freedom.

For the computation of the stiffness matrix ${K^{QP}}$ we require the term $B_{ij}$ relating nodal displacements to strains. This interpolator can be written in terms of drivativs of the shape functions as follows:

$$B_{ij}^Q = \frac{1}{2}\left( {\frac{{\partial N_i^Q}}{{\partial {x_j}}} + \frac{{\partial N_j^Q}}{{\partial {x_i}}}} \right).$$

In the implementation this operator is computed by the subroutine strain-displacement matrix for a 4-noded element stdm4() as listed blow:

```python
def shape4(x , y , h):
    """Shape functions for the bi-lineal element.
    Parameters
    ----------
    x , y: Space variables.
    h    : Element halwidth.

    Returns
    -------
    N : Array

    """
    N=sym.zeros(4)
    N = 1/(4*h**2)*sym.Matrix([(h + x)*(h + y),
         (h - x)*(h + y),
         (h - x)*(h - y),
         (h + x)*(h - y)])

    return N
```

```python
def stdm4(x , y , h ):
    """Strain-displacement interpolator for the bi-lineal element.
    Parameters
    ----------
    x , y: Space variables.
    h    : Element halwidth.

    Returns
    -------
    B : Array

    """
    B = sym.zeros(3,8)
    N = shape4(x , y , h)
    dhdx=sym.zeros(2,4)
    for i in range(4):
        dhdx[0,i]=sym.diff(N[i],x)
        dhdx[1,i]=sym.diff(N[i],y)
#
    for i in range(4):
        B[0, 2*i]   = dhdx[0, i]
        B[1, 2*i+1] = dhdx[1, i]
        B[2, 2*i]   = dhdx[1, i]
        B[2, 2*i+1] = dhdx[0, i]
#
    return B
```

We also need to evaluate the elastic constitutive matrix:

$$C = \frac{{E(1 - \nu )}}{{(1 + \nu )(1 - 2\nu )}}\left[ {\begin{array}{*{20}{c}}
1&{\frac{\nu }{{1 - \nu }}}&0\\
{\frac{\nu }{{1 - \nu }}}&1&0\\
0&0&{\frac{{1 - 2\nu }}{{2(1 - \nu )}}}
\end{array}} \right]$$

by the subroutine umat().

```python
def umat(nu,E):
    """Plane stress constitutive tensor.
    Parameters
    ----------
    nu : Posisson ratio.
    E  : Young modulus.

    Returns
    -------
    C : Array. Constitutive matrix.

    """
#
    C=sym.zeros(3,3)
    G=E/(1-nu**2)
    mnu=(1-nu)/2.0
    C[0,0]=G
    C[0,1]=nu*G
    C[1,0]=C[0,1]
    C[1,1]=G
    C[2,2]=G*mnu
#
    return C
```

### Computation of the stiffness matrix

```python
C = sym.zeros(3,3)
B = sym.zeros(3,8)
K = sym.zeros(8,8)
```

```python
x , y = sym.symbols('x y')
nu, E = sym.symbols('nu, E')
h     = sym.symbols('h')
```

```python
C = umat(nu,E)
B = stdm4(x , y , h)
K_int = B.T*C*B
```

```python
nuu = 1.0/3.0
EE  = 8.0/3.0
```

```python
for i in range(8):
    for j in range(8):
        K[i,j] = sym.integrate(K_int[i,j], (x,-h,h), (y,-h,h))
kk=K.subs([(E, EE), (nu, nuu), (h, 1.00)])
```

```python
print(sym.N(kk , 3))
```

    Matrix([
    [  1.33,    0.5, -0.833,      0, -0.667,   -0.5,  0.167,      0],
    [   0.5,   1.33,      0,  0.167,   -0.5, -0.667,      0, -0.833],
    [-0.833,      0,   1.33,   -0.5,  0.167,      0, -0.667,    0.5],
    [     0,  0.167,   -0.5,   1.33,      0, -0.833,    0.5, -0.667],
    [-0.667,   -0.5,  0.167,      0,   1.33,    0.5, -0.833,      0],
    [  -0.5, -0.667,      0, -0.833,    0.5,   1.33,      0,  0.167],
    [ 0.167,      0, -0.667,    0.5, -0.833,      0,   1.33,   -0.5],
    [     0, -0.833,    0.5, -0.667,      0,  0.167,   -0.5,   1.33]])

```python
from IPython.core.display import HTML
def css_styling():
    styles = open('../styles/custom_barba.css', 'r').read()
    return HTML(styles)
css_styling()
```

<link href='http://fonts.googleapis.com/css?family=Fenix' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>

```python

```