# the $k$-nearest neighbors (kNN) supervised learning algorithm

* can be used for regression or classification
* non-parametric
* imposes only mild structural assumptions about the data

let's explore $k$-nearest neighbor classification.

training data examples: $\{(\mathbf{x}_1, y_1), (\mathbf{x}_2, y_2), ..., (\mathbf{x}_n, y_n)\}$ where $\mathbf{x}_i$ is the feature vector and $y_i \in \{0, 1\}$ is the label on data point $i$. This is a binary classification since each data point is labeled with $0$ or $1$.

the $k$-NN algorithm uses the training data to classify new data points. say we have a new data point $\mathbf{x}$ but we don't know its label. the $k$-NN algorithm predicts the class of this data point as:
\begin{equation}
    \hat{y}(\mathbf{x})= \frac{1}{k} \displaystyle \sum_{\mathbf{x}_i \in N_k(\mathbf{x})} y_i
\end{equation}
where $N_k(x)$ is the *neighborhood* of $\mathbf{x}$ defined as the $k$ "closest" points in the training data set. For "closest" to be mathematically defined, we need a distance metric. One such distance metric is Euclidean distance. In words, the $k$ data points closest to $\mathbf{x}$ vote on whether we should classify this point as a 0 or 1 with their labels.

```julia
using CSV
using DataFrames
using PyPlot
using ScikitLearn # machine learning package
using StatsBase
using Random
using LaTeXStrings # for L"$x$" to work instead of needing to do "\$x\$"
using Printf

# (optional)change settings for all plots at once, e.g. font size
rcParams = PyPlot.PyDict(PyPlot.matplotlib."rcParams")
rcParams["font.size"] = 16

# (optional) change the style. see styles here: https://matplotlib.org/3.1.1/gallery/style_sheets/style_sheets_reference.html
PyPlot.matplotlib.style.use("seaborn-white")
```

## classifying breast tumors as malignant or benign

source: [UCI Machine Learning Repository](https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+(Diagnostic))

> Features are computed from a digitized image of a fine needle aspirate (FNA) of a breast mass. They describe characteristics of the cell nuclei present in the image.

The mean radius and smoothness of the cell nuclei (the two features) and the outcome (M = malignant, B = benign) of the tumor are in the `breast_cancer_data.csv`.

```julia
df = CSV.read("breast_cancer_data.csv")
first(df, 5)
```

	mean_radius	mean_smoothness	outcome

	Float64	Float64	String

5 rows × 3 columns

1	13.85	1.495	B

2	9.668	2.275	B

3	9.295	2.388	B

4	19.69	4.585	M

5	9.755	1.243	B

```julia
by(df, :outcome, total=:outcome => length)
```

	outcome	total

	String	Int64

2 rows × 2 columns

1	B	180

2	M	120

let's map the outcomes to a number, 0 or 1, to facilitate the NN algo.

benign (B) : 0

malignant (M) : 1

```julia
df[!, :class] = map(row -> row == "B" ? 0 : 1, df[:, :outcome])
first(df, 5)
```

	mean_radius	mean_smoothness	outcome	class

	Float64	Float64	String	Int64

5 rows × 4 columns

1	13.85	1.495	B	0

2	9.668	2.275	B	0

3	9.295	2.388	B	0

4	19.69	4.585	M	1

5	9.755	1.243	B	0

let's also have a color scheme for the class labels.

```julia
color_scheme = Dict("M" => "r", "B" => "g")
```

    Dict{String,String} with 2 entries:
      "B" => "g"
      "M" => "r"

since this is a two-dimensional feature space, we have the luxury of visualizing how the classes are distributed in feature space. in practice, we do not have this luxury. I choose a 2D feature space for pedagogical purposes :)

```julia
figure()
xlabel("mean radius")
ylabel("mean smoothness")
for df_c in groupby(df, :outcome)
    outcome = df_c[1, :outcome]
    scatter(df_c[:, :mean_radius], df_c[:, :mean_smoothness], label="$outcome",
        marker="o", facecolor="None", edgecolor=color_scheme[outcome])
end
legend()
axis("equal")
```

### using Scikitlearn for $k$-nearest neighbor classification

scikitlearn documentation for `KNeighborsClassifier` [here](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html)

```julia
@sk_import neighbors : KNeighborsClassifier
@sk_import model_selection : train_test_split
@sk_import metrics : confusion_matrix
```

    WARNING: redefining constant KNeighborsClassifier
    ┌ Warning: Module model_selection has been ported to Julia - try `import ScikitLearn: CrossValidation` instead
    └ @ ScikitLearn.Skcore /home/cokes/.julia/packages/ScikitLearn/bo2Pt/src/Skcore.jl:140
    WARNING: redefining constant train_test_split
    WARNING: redefining constant confusion_matrix

    PyObject <function confusion_matrix at 0x7f46d1b8aea0>

scikitlearn takes as input:
* a feature matrix `X`, which must be `n_samples` by `n_features`
* a target vector `y`, which must be `n_samples` long (of course)

```julia
n_tumors = nrow(df)

X = zeros(n_tumors, 2)
y = zeros(n_tumors)
for (i, tumor) in enumerate(eachrow(df))
    X[i, 1] = tumor[:mean_radius]
    X[i, 2] = tumor[:mean_smoothness]
    y[i] = tumor[:class]
end
X # look at y too!
```

    300×2 Array{Float64,2}:
     13.85   1.495
      9.668  2.275
      9.295  2.388
     19.69   4.585
      9.755  1.243
     16.11   4.533
     14.78   2.45
     15.78   3.598
     15.71   1.972
     14.68   3.195
     13.71   3.856
     21.09   4.414
     11.31   1.831
      ⋮
     11.08   1.719
     18.94   5.486
     15.32   4.061
     14.25   5.373
     20.6    5.772
      8.671  1.435
     11.64   2.155
     12.06   1.171
     13.88   1.709
     14.9    3.466
     19.59   2.916
     14.81   1.677

construct a nearest neighbor object in scikitlearn

```julia
knn = KNeighborsClassifier(n_neighbors=1)
```

    PyObject KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
                         metric_params=None, n_jobs=None, n_neighbors=1, p=2,
                         weights='uniform')

pass the `KNeighborsClassifier` object our training data. this will be used to make predictions on future, unseen data.

```julia
knn.fit(X, y)
```

    PyObject KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
                         metric_params=None, n_jobs=None, n_neighbors=1, p=2,
                         weights='uniform')

we can now make predictions on new, unseen data. that is, if we know the mean radius and mean smoothness of a new tumor, we can make a prediction about whether this tumor is malignant or benign. let's imagine we take measurements on a new tumor and it has the feature vector `x_new` below. we aim to predict whether it is benign or malignant.

```julia
# new tumor. unknown if malignant or benign...
x_new = [10.0 1.5] # should be B
x_new = [20.0 5.0] # should be M
```

    1×2 Array{Float64,2}:
     20.0  5.0

first, let's plot where it falls in feature space

```julia
figure()
xlabel("mean radius")
ylabel("mean smoothness")
for df_c in groupby(df, :outcome)
    outcome = df_c[1, :outcome]
    scatter(df_c[:, :mean_radius], df_c[:, :mean_smoothness], label="$outcome",
        marker="o", facecolor="None", edgecolor=color_scheme[outcome])
end

# plot where new tumor falls
scatter(x_new[1], x_new[2], marker="s", label="new data pt", color="k")

legend()
axis("equal")
```

predict whether this new tumor is benign or malignant

```julia
knn.predict(x_new)
```

    1-element Array{Float64,1}:
     1.0

visualize the decision boundary

```julia
radius = 5:0.25:30
smoothness = 0.0:0.25:20.0

knn_prediction = zeros(length(smoothness), length(radius))
for i = 1:length(radius)
    for j = 1:length(smoothness)
        x = [radius[i] smoothness[j]]
        knn_prediction[j, i] = knn.predict(x)[1]
    end
end
```

```julia
figure()
contourf(radius, smoothness, knn_prediction, alpha=0.4, [0.0, 0.5, 1.0], colors=["g", "r"])
for df_c in groupby(df, :outcome)
    outcome = df_c[1, :outcome]
    scatter(df_c[:, :mean_radius], df_c[:, :mean_smoothness], label="$outcome",
        marker="o", facecolor="None", edgecolor=color_scheme[outcome])
end
xlabel("mean radius")
ylabel("mean smoothness")
```

we can compute the accuracy of the prediction on the training data. but when $k=1$ we get 100% accuracy by construction!

```julia
knn.score(X, y)
```

    0.9166666666666666

### the test/train paradigm

randomly split your data set into a training and test set. train the $k$-NN algo on the training data, then use the trained model to make predictions on data in the test set. we can compare the predicted label to the true label. the test set error is a quality prediction of generalization error on *unseen* data. we could write our own code for this, but scikitlearn provides `test_train_split` for us.

```julia
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33)
```

    4-element Array{Array{Float64,N} where N,1}:
     [12.06 1.171; 18.31 1.817; … ; 13.44 1.572; 9.295 2.388]
     [23.27 4.603; 18.01 5.353; … ; 20.29 5.438; 13.65 1.391]
     [0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 0.0, 0.0  …  1.0, 0.0, 1.0, 0.0, 0.0, 1.0, 1.0, 1.0, 1.0, 0.0]
     [1.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0  …  1.0, 1.0, 1.0, 0.0, 1.0, 1.0, 0.0, 0.0, 1.0, 0.0]

to fully assess the performance of our model, trained on the training set, and tested on the test set, we can plot a so-called confusion matrix. it tells us about false positives, true positives, false negatives, and true negatives.

```julia
# see https://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html#sphx-glr-auto-examples-model-selection-plot-confusion-matrix-py
function plot_confusion_matrix(y_true::Array{Float64}, y_pred::Array{Float64}, classes::Array{String})
    # calculate confusion matrix from scikitlearn
    cm = confusion_matrix(y_true, y_pred)

    fig, ax = plt.subplots()
    ax.imshow(cm, interpolation="nearest", cmap=plt.cm.Greens)
    ax.set(
        xlim=[-0.5, 1.5],
        ylim=[-0.5, 1.5],
        xticks=[0, 1],
        yticks=[0, 1],
        xticklabels=classes,
        yticklabels=classes,
        xlabel="predicted label",
        ylabel="true label"
    )

    for i = 1:2
        for j = 1:2
            ax.text(j-1, i-1, @sprintf("%d", cm[i, j]), ha="center", va="center")
        end
    end
    tight_layout()
end

k = 3
knn = KNeighborsClassifier(n_neighbors=k)
knn.fit(X_train, y_train)
y_pred = knn.predict(X_test)

plot_confusion_matrix(y_test, y_pred, ["benign", "malignant"])
```

to find the best $k$, we can scan over all $k$, train the model in the training set, then test the model on the test set (unseen data) to see how it performs. we take the "best" $k$ as the one that yields the lowest test set error, as we expect it to have the lowest generalization error on unseen data.

```julia
ks = 1:20
test_accuracy = zeros(length(ks))
train_accuracy = zeros(length(ks))
for (i, k) in enumerate(ks)
    knn = KNeighborsClassifier(n_neighbors=k)
    knn.fit(X_train, y_train)
    test_accuracy[i] = knn.score(X_test, y_test)
    train_accuracy[i] = knn.score(X_train, y_train)
end

figure()
plot(ks, train_accuracy, marker="o", label="train")
plot(ks, test_accuracy, marker="o", label="test")
legend()
xlabel(L"$k$")
ylabel("accuracy")
```

the optimal $k$ is around 8. So we should train our $k$-NN algo with $k=8$, then deploy this model, since it should result in the best accuracy on unseen data.

```julia
knn = KNeighborsClassifier(n_neighbors=8)
knn.fit(X, y) # *this* is the model we deploy

# look at new decision boundary
for i = 1:length(radius)
    for j = 1:length(smoothness)
        x = [radius[i] smoothness[j]]
        knn_prediction[j, i] = knn.predict(x)[1]
    end
end

figure()
contourf(radius, smoothness, knn_prediction, alpha=0.4, [0.0, 0.5, 1.0], colors=["g", "r"])
for df_c in groupby(df, :outcome)
    outcome = df_c[1, :outcome]
    scatter(df_c[:, :mean_radius], df_c[:, :mean_smoothness], label="$outcome",
        marker="o", facecolor="None", edgecolor=color_scheme[outcome])
end
xlabel("mean radius")
ylabel("mean smoothness")
```


## 1 求解导数

给定输入的张量是$x$，这是一个 $N \times C_{i n} \times w \times h$ 的张量;

给定模板的张量是$h$，这是一个$C_{\text {out }} \times C_{\text {in }} \times 3 \times 3$的张量;

进行卷积运算的参数，采用Padding = 1，然后 Stride = 1

现在已知张量$y$是通过模板对输入进行模板运算的结果，如下:

$$y=x \otimes h$$

其中$\otimes$是模板运算，另外已知损失函数相对于$y$的偏导数为:

$$\frac{\partial L}{\partial y}$$

请尝试推导:

1) 损失函数相对于输入的导数$\frac{\partial L}{\partial x}$

不妨令

$\begin{equation}
    X = \left[\begin{array}{cccccccc}
    x_{11} & x_{12} & x_{13} & x_{14} & x_{15}  \\
    x_{22} & x_{22} & x_{23} & x_{24} & x_{25}  \\
    x_{33} & x_{32} & x_{33} & x_{34} & x_{35}  \\
    x_{44} & x_{42} & x_{43} & x_{44} & x_{45}  \\
    x_{55} & x_{52} & x_{53} & x_{54} & x_{55}
    \end{array}\right]
\end{equation}$

$\begin{equation}
    H=\left[\begin{array}{ccc}
    h_{11} & h_{12} & h_{13} \\
    h_{21} & h_{22} & h_{23} \\
    h_{31} & h_{32} & h_{33}
    \end{array}\right]
\end{equation}$

由于$y=x \otimes h$，则$Y=conv2(X,H)$

$\begin{equation}
    Y = \left[\begin{array}{cccccccc}
    y_{11} & y_{12} & y_{13} & y_{14} & y_{15}  \\
    y_{22} & y_{22} & y_{23} & y_{24} & y_{25}  \\
    y_{33} & y_{32} & y_{33} & y_{34} & y_{35}  \\
    y_{44} & y_{42} & y_{43} & y_{44} & y_{45}  \\
    y_{55} & y_{52} & y_{53} & y_{54} & y_{55}
    \end{array}\right]
\end{equation}$

下面分析$\frac{\partial L}{\partial x_{11}}$

$\begin{equation}
    X^{pad} = \left[\begin{array}{ccccccc}
    0 & 0 & 0 & 0 & 0 & 0 & 0   \\
    0 & x_{11} & x_{12} & x_{13} & x_{14} & x_{15} & 0    \\
    0 & x_{22} & x_{22} & x_{23} & x_{24} & x_{25} & 0    \\
    0 & x_{33} & x_{32} & x_{33} & x_{34} & x_{35} & 0    \\
    0 & x_{44} & x_{42} & x_{43} & x_{44} & x_{45} & 0    \\
    0 & x_{55} & x_{52} & x_{53} & x_{54} & x_{55} & 0    \\
    0 & 0 & 0 & 0 & 0 & 0 & 0
    \end{array}\right]
\end{equation}$

$y_{ij}=\sum_{u=0}^{2} \sum_{v=0}^{2} X_{i+u, j+v}^{p a d} \cdot H_{1+u, 1+v}$

$\begin{equation}y_{11} = np.sum\left(
    \left[\begin{array}{ccc}
    0 & 0 & 0 \\
    0 & x_{11} & x_{12} \\
    0 & x_{21} & x_{22}
    \end{array}\right] *
    \left[\begin{array}{ccc}
    h_{11} & h_{12} & h_{13} \\
    h_{21} & h_{22} & h_{23} \\
    h_{31} & h_{32} & h_{33}
    \end{array}\right]\right)
\end{equation}$

$\begin{equation}y_{12} = np.sum\left(
    \left[\begin{array}{ccc}
    0 & 0 & 0 \\
    x_{11} & x_{12} & x_{13}\\
    x_{21} & x_{22} & x_{23}
    \end{array}\right] *
    \left[\begin{array}{ccc}
    h_{11} & h_{12} & h_{13} \\
    h_{21} & h_{22} & h_{23} \\
    h_{31} & h_{32} & h_{33}
    \end{array}\right] \right)
\end{equation}$

$\begin{equation}y_{21} = np.sum\left(
    \left[\begin{array}{ccc}
    0 & x_{11} & x_{12} \\
    0 & x_{21} & x_{22} \\
    0 & x_{31} & x_{32}
    \end{array}\right] *
    \left[\begin{array}{ccc}
    h_{11} & h_{12} & h_{13} \\
    h_{21} & h_{22} & h_{23} \\
    h_{31} & h_{32} & h_{33}
    \end{array}\right] \right)
\end{equation}$

$\begin{equation}y_{22} = np.sum\left(
    \left[\begin{array}{ccc}
    x_{11} & x_{12} & x_{13} \\
    x_{21} & x_{22} & x_{23} \\
    x_{31} & x_{32} & x_{33}
    \end{array}\right] *
    \left[\begin{array}{ccc}
    h_{11} & h_{12} & h_{13} \\
    h_{21} & h_{22} & h_{23} \\
    h_{31} & h_{32} & h_{33}
    \end{array}\right] \right)
\end{equation}$

$\frac{\partial L}{\partial x_{11}}=\frac{\partial L}{\partial y_{11}} \cdot h_{22}+\frac{\partial L}{\partial y_{12}} \cdot h_{23}+\frac{\partial L}{\partial y_{21}} \cdot h_{32}+\frac{\partial L}{\partial y_{22}} \cdot h_{33}$

$\frac{\partial L}{\partial x_{11}}$即目标$Y$对$H$的模板运算的第一步，Padding = 1，Stride = 1。

所以，可以推出

$\frac{\partial L}{\partial X}=\operatorname{Convolution2D}\left(\frac{\partial L}{\partial Y}^{(p a d)}, H\right)$

2) 损失函数相对于模板的导数$\frac{\partial L}{\partial h}$

$\begin{equation}
\frac{\partial E}{\partial W}=\left[\begin{array}{ccc}
\frac{\partial E}{\partial W_{11}} & \frac{\partial E}{\partial W_{12}} & \frac{\partial E}{\partial W_{13}} \\
\frac{\partial E}{\partial W_{21}} & \frac{\partial E}{\partial W_{22}} & \frac{\partial E}{\partial W_{23}} \\
\frac{\partial E}{\partial W_{31}} & \frac{\partial E}{\partial W_{32}} & \frac{\partial E}{\partial W_{33}}
\end{array}\right]
\end{equation}$

$\begin{equation}
=\left[\begin{array}{ccc}
\sum_{i} \sum_{j} \frac{\partial E}{\partial Y_{i, j}} \cdot X_{i+2, j+2}^{p a d} & \sum_{i} \sum_{j} \frac{\partial E}{\partial Y_{i, j}} \cdot X_{i+2, j+1}^{p a d} & \sum_{i} \sum_{j} \frac{\partial E}{\partial Y_{i, j}} \cdot X_{i+2, j}^{p a d} \\
\sum_{i} \sum_{j} \frac{\partial E}{\partial Y_{i, j}} \cdot X_{i+1, j+2}^{p a d} & \sum_{i} \sum_{j} \frac{\partial E}{\partial Y_{i, j}} \cdot X_{i+1, j+1}^{p a d} & \sum_{i} \sum_{j} \frac{\partial E}{\partial Y_{i, j}} \cdot X_{i+1, j}^{p a d} \\
\sum_{i} \sum_{j} \frac{\partial E}{\partial Y_{i, j}} \cdot X_{i, j+2}^{p a d} & \sum_{i} \sum_{j} \frac{\partial E}{\partial Y_{i, j}} \cdot X_{i, j+1}^{p a d} & \sum_{i} \sum_{j} \frac{\partial E}{\partial Y_{i, j}} \cdot X_{i, j}^{p a d}
\end{array}\right]
\end{equation}$

即

$\frac{\partial L}{\partial H}=\operatorname{Rot180}\left( \operatorname{Convolution2D}\left(\frac{\partial L}{\partial Y}^{(p a d)}, X\right)\right)$

## 2
假设现在有一个4x4的具有两个通道的特征如下所示。
```
f = [[[ 1  2  3  4]
      [ 8  7  6  5]
      [ 9 10 11 12]
      [16 15 14 13]]

     [[29 30 31 32]
      [28 27 26 25]
      [21 22 23 24]
      [20 19 18 17]]]
```
对这个图像采用，如下的模板进行模板运算。
```
h = [[[[-1  0  1]
       [-1  0  1]
       [-1  0  1]]
      [[-1 -1 -1]
       [ 0  0  0]
       [ 1  1  1]]]

     [[[ 1  0  0]
       [ 0  1  0]
       [ 0  0  1]]
      [[ 0  0  1]
       [ 0  1  0]
       [ 1  0  0]]]]
```

```python
import numpy as np

# f[c_in, x, y]
f = np.asarray([[[1, 2, 3, 4],
                 [8, 7, 6, 5],
                 [9, 10, 11, 12],
                 [16, 15, 14, 13]],
                [[29, 30, 31, 32],
                 [28, 27, 26, 25],
                 [21, 22, 23, 24],
                 [20, 19, 18, 17]]])

h = np.empty([2, 2, 3, 3])
h[0, 0, :, :] = [[-1, 0, 1],
                 [-1, 0, 1],
                 [-1, 0, 1]]
h[0, 1, :, :] = [[-1, -1, -1],
                 [0, 0, 0],
                 [1, 1, 1]]
h[1, 0, :, :] = [[1, 0, 0],
                 [0, 1, 0],
                 [0, 0, 1]]
h[1, 1, :, :] = [[0, 0, 1],
                 [0, 1, 0],
                 [1, 0, 0]]
```

模板运算采用 Valid 输出尺寸，请问：

1）输出记为 conv1 ，请问 conv1 是多少？

```python
from scipy.signal import convolve2d

_, M, N = f.shape
# conv1[c_out, c_in, x, y]
conv = np.empty([2, 2, M - 2, N - 2])
conv1 = np.empty([2, 2, 2])
for c_out in range(2):
    for c_in in range(2):
        conv[c_out, c_in, :, :] = convolve2d(f[c_in, :, :], np.rot90(h[c_out, c_in, :, :], 2), mode="valid")
conv1[0]=conv[0,0,:,:]+conv[0,1,:,:]
conv1[1]=conv[1,0,:,:]+conv[1,1,:,:]
print("conv1 = \n", conv1)

# print(convolve2d(f[0,:,:],np.rot90(h[0,0,:,:],2),"valid"))
```

    conv1 =
     [[[-22. -22.]
      [-26. -26.]]

     [[ 98. 100.]
      [100.  98.]]]

(2) 如果采用 ReLU 对这个输出进行激活，记为 relu1 ，请问激活后 relu1 的值是多
少？

```python
def relu(np_vect):
    if np_vect >= 0:
        return np_vect
    else:
        return 0

rectification = np.vectorize(relu)
relu1 = rectification(conv1)

print(relu1)
```

    [[[  0   0]
      [  0   0]]

     [[ 98 100]
      [100  98]]]

3）如果将输出拉成一列，采用全连接网络，输出节点个数为 5，假设全连接所有权重都设置为 1/10，输出记为fc1 ，请问输出是多少？

```python
fc1 = np.zeros([5])
weight = 0.1

for i in range(5):
    for j in relu1.flat:
        fc1[i] += weight * j
print(fc1)
```

    [39.6 39.6 39.6 39.6 39.6]

4）假设采用 softmax 对这个 5 个节点的输出进行，概率值记为p=[p1,p2,p3,p4,p5]，
请问p是多少？

```python
p = np.asarray([1 / (1 + np.exp(-x)) for x in fc1])
p=[x/sum(p) for x in p]
print(p)

```

    [0.2, 0.2, 0.2, 0.2, 0.2]

5) 如果采用交叉熵对概率进行约束，如下所示

$$L=\sum_{i=1}^{5}-y_{i} \log p_{i}$$

如果$y_{1}=0, y_{2}=0, y_{3}=1, y_{4}=0, y_{5}=0$，请问损失函数是多少？

```python
y = np.asarray([0, 0, 1, 0, 0])
print("y*log(p) = ", y*np.log(p))
L = np.sum(y * np.log(p))
print("L = ", L)
```

    y*log(p) =  [-0.         -0.         -1.60943791 -0.         -0.        ]
    L =  -1.6094379124341003

6) 请问$\frac{\partial L}{\partial p}, \frac{\partial L}{\partial \mathrm{fc}_{1}}, \frac{\partial L}{\partial \mathrm{relu}_{1}}, \frac{\partial L}{\partial \operatorname{conv}_{1}}$分别是多少？
```
tensor([[ 0.2000, 0.2000, -0.8000, 0.2000, 0.2000]])
即0
tensor([[[[-4.1723e-09, -4.1723e-09], [-4.1723e-09, -4.1723e-09]], [[-4.1723e-09, -4.1723e-09], [-4.1723e-09, -4.1723e-09]]]])
tensor([[[[ 0.0000e+00, 0.0000e+00], [ 0.0000e+00, 0.0000e+00]], [[-4.1723e-09, -4.1723e-09], [-4.1723e-09, -4.1723e-09]]]]
```

7) 如果把全连接的权重记为$W$，请问$\frac{\partial L}{\partial W}$是多少？
```
tensor([
 [ 0.0000, 0.0000, 0.0000, 0.0000, 19.6000, 20.0000, 20.0000,19.6000],
[ 0.0000, 0.0000, 0.0000, 0.0000, 19.6000, 20.0000, 20.0000,19.6000],
[ -0.0000, -0.0000, -0.0000, -0.0000, -78.4000, -80.0000, -80.0000,-78.4000], [ 0.0000, 0.0000, 0.0000, 0.0000, 19.6000, 20.0000, 20.0000,19.6000],
[ 0.0000, 0.0000, 0.0000, 0.0000, 19.6000, 20.0000, 20.0000,19.6000] ])
```

8) 请问$\frac{\partial L}{\partial h}$是多少？
```
tensor([
[[[ 0.0000e+00, 0.0000e+00, 0.0000e+00],
 [ 0.0000e+00, 0.0000e+00, 0.0000e+00],
[ 0.0000e+00, 0.0000e+00, 0.0000e+00]],
[[ 0.0000e+00, 0.0000e+00, 0.0000e+00],
 [ 0.0000e+00, 0.0000e+00, 0.0000e+00],
 [ 0.0000e+00, 0.0000e+00, 0.0000e+00]]],
 [[[-7.5102e-08, -7.5102e-08, -7.5102e-08],
 [-1.4186e-07, -1.4186e-07, -1.4186e-07],
[-2.0862e-07, -2.0862e-07, -2.0862e-07]],
 [[-4.7565e-07, -4.7565e-07, -4.7565e-07],
[-4.0889e-07, -4.0889e-07, -4.0889e-07],
[-3.4213e-07, -3.4213e-07, -3.4213e-07]]]])
```


# 状態方程式の導出

```julia
using Symbolics
using Latexify
```

```julia
@variables t M m l g D_θ D_x u
@variables x(t) θ(t)

Dt = Differential(t)

v = Dt(x)
ω = Dt(θ)

Dx = Differential(x)
Dv = Differential(v)
Dθ = Differential(θ)
Dω = Differential(ω)
```

    (::Differential) (generic function with 2 methods)

## エネルギー
運動エネルギー，ポテンシャルエネルギー，損失エネルギーは次である．

運動エネルギーを定義

```julia
K = 2//3*m*l^2*ω^2 + m*l*v*ω*cos(θ) + 1//2*(M+m)*v^2
```

\begin{equation}
\left( \frac{dx(t)}{dt} \right)^{2} \left( \frac{1}{2} M + \frac{1}{2} m \right) + \frac{2}{3} \left( \frac{d\theta(t)}{dt} \right)^{2} l^{2} m + l m \frac{dx(t)}{dt} \frac{d\theta(t)}{dt} \cos\left( \theta\left( t \right) \right)
\end{equation}

ポテンシャルエネルギーを定義

```julia
U = m*g*l*cos(θ)
```

\begin{equation}
g l m \cos\left( \theta\left( t \right) \right)
\end{equation}

ポテンシャルエネルギーを定義

```julia
D = 1//2*D_θ*ω^2 + 1//2*D_x*v^2
```

\begin{equation}
\frac{1}{2} \left( \frac{dx(t)}{dt} \right)^{2} D_{x} + \frac{1}{2} \left( \frac{d\theta(t)}{dt} \right)^{2} D_{\theta}
\end{equation}

## ラグランジュの運動方程式に代入
ラグランジュの運動方程式は次式．

### x方向
運動エネルギーの速度偏微分の時間微分

```julia
dKdv_dot = K |> (Dv * Dt) |> expand_derivatives
```

\begin{equation}
2 \mathrm{\frac{d}{d t}}\left( \frac{dx(t)}{dt} \right) \left( \frac{1}{2} M + \frac{1}{2} m \right) + l m \cos\left( \theta\left( t \right) \right) \mathrm{\frac{d}{d t}}\left( \frac{d\theta(t)}{dt} \right) - \left( \frac{d\theta(t)}{dt} \right)^{2} l m \sin\left( \theta\left( t \right) \right)
\end{equation}

運動エネルギーの位置偏微分

```julia
dKdx = K |> Dx |> expand_derivatives
```

\begin{equation}
0
\end{equation}

ポテンシャルエネルギーの位置偏微分

```julia
dUdx = U |> Dx |> expand_derivatives
```

\begin{equation}
0
\end{equation}

損失エネルギーの速度偏微分

```julia
dDdv = D |> Dv |> expand_derivatives
```

\begin{equation}
D_{x} \frac{dx(t)}{dt}
\end{equation}

x方向の入力

```julia
u_x = u
```

\begin{equation}
u
\end{equation}

### theta方向
運動エネルギーの角速度偏微分の時間微分

```julia
dKdω_dot = K |> (Dω * Dt) |> expand_derivatives
```

\begin{equation}
l m \cos\left( \theta\left( t \right) \right) \mathrm{\frac{d}{d t}}\left( \frac{dx(t)}{dt} \right) + \frac{4}{3} l^{2} m \mathrm{\frac{d}{d t}}\left( \frac{d\theta(t)}{dt} \right) - 2 l m \frac{dx(t)}{dt} \frac{d\theta(t)}{dt} \sin\left( \theta\left( t \right) \right)
\end{equation}

運動エネルギーの角度偏微分

```julia
dKdθ = K |> Dθ |> expand_derivatives
```

\begin{equation}
 - l m \frac{dx(t)}{dt} \frac{d\theta(t)}{dt} \sin\left( \theta\left( t \right) \right)
\end{equation}

ポテンシャルエネルギーの角度偏微分

```julia
dUdθ = U |> Dθ |> expand_derivatives
```

\begin{equation}
 - g l m \sin\left( \theta\left( t \right) \right)
\end{equation}

損失エネルギーの角速度偏微分

```julia
dDdω = D |> Dω |> expand_derivatives
```

\begin{equation}
D_{\theta} \frac{d\theta(t)}{dt}
\end{equation}

回転方向の入力

```julia
u_θ = 0
```

    0

## 加速度を求める
これまでの演算から運動方程式が得られた．
x方向の運動方程式

```julia
f_x = dKdv_dot - dKdx + dUdx + dDdv - u_x
```

\begin{equation}
D_{x} \frac{dx(t)}{dt} - u + 2 \mathrm{\frac{d}{d t}}\left( \frac{dx(t)}{dt} \right) \left( \frac{1}{2} M + \frac{1}{2} m \right) + l m \cos\left( \theta\left( t \right) \right) \mathrm{\frac{d}{d t}}\left( \frac{d\theta(t)}{dt} \right) - \left( \frac{d\theta(t)}{dt} \right)^{2} l m \sin\left( \theta\left( t \right) \right)
\end{equation}

theta方向の運動方程式

```julia
f_θ = dKdω_dot - dKdθ + dUdθ + dDdω - u_θ
```

\begin{equation}
D_{\theta} \frac{d\theta(t)}{dt} + l m \cos\left( \theta\left( t \right) \right) \mathrm{\frac{d}{d t}}\left( \frac{dx(t)}{dt} \right) + \frac{4}{3} l^{2} m \mathrm{\frac{d}{d t}}\left( \frac{d\theta(t)}{dt} \right) - g l m \sin\left( \theta\left( t \right) \right) - l m \frac{dx(t)}{dt} \frac{d\theta(t)}{dt} \sin\left( \theta\left( t \right) \right)
\end{equation}

状態方程式を求めるために，加速度を求める．
運動方程式は加速度と角加速度の連立方程式になっているため，`sy.solve`を使って解く．
ついでに式を簡略化．

```julia
@variables x_dot, θ_dot, x_ddot, θ_ddot, h, f
f_x = substitute(
    f_x,
    Dict([
        expand_derivatives(Dt(v)) => x_ddot,
        expand_derivatives(Dt(ω)) => θ_ddot,
        expand_derivatives(Dt(x)) => x_dot,
        expand_derivatives(Dt(θ)) => θ_dot,
        x => h,
        θ => f,
    ]))
f_θ = substitute(
    f_θ,
    Dict([
        expand_derivatives(Dt(v)) => x_ddot,
        expand_derivatives(Dt(ω)) => θ_ddot,
        expand_derivatives(Dt(x)) => x_dot,
        expand_derivatives(Dt(θ)) => θ_dot,
        x => h,
        θ => f,
    ]));
```

```julia
f_x
```

\begin{equation}
D_{x} x_{dot} - u + 2 x_{ddot} \left( \frac{1}{2} M + \frac{1}{2} m \right) + l m \theta_{ddot} \cos\left( f \right) - \theta_{dot}^{2} l m \sin\left( f \right)
\end{equation}

```julia
f_θ
```

\begin{equation}
D_{\theta} \theta_{dot} + l m x_{ddot} \cos\left( f \right) + \frac{4}{3} l^{2} m \theta_{ddot} - g l m \sin\left( f \right) - l m x_{dot} \theta_{dot} \sin\left( f \right)
\end{equation}

```julia
Symbolics.solve_for([f_x~0, f_θ~0], [x_ddot, θ_ddot])
```


# The Efficient Frontier of Optimal Portfolio Transactions

### Introduction

[Almgren and Chriss](https://cims.nyu.edu/~almgren/papers/optliq.pdf) showed that for each value of risk aversion there is a unique optimal execution strategy. The optimal strategy is obtained by minimizing the **Utility Function** $U(x)$:

\begin{equation}
U(x) = E(x) + \lambda V(x)
\end{equation}

where $E(x)$ is the **Expected Shortfall**, $V(x)$ is the **Variance of the Shortfall**, and $\lambda$ corresponds to the trader’s risk aversion. The expected shortfall and variance of the optimal trading strategy are given by:

In this notebook, we will learn how to visualize and interpret these equations.

# The Expected Shortfall

As we saw in the previous notebook, even if we use the same trading list, we are not guaranteed to always get the same implementation shortfall due to the random fluctuations in the stock price. This is why we had to reframe the problem of finding the optimal strategy in terms of the average implementation shortfall and the variance of the implementation shortfall. We call the average implementation shortfall, the expected shortfall $E(x)$, and the variance of the implementation shortfall $V(x)$. So, whenever we talk about the expected shortfall we are really talking about the average implementation shortfall. Therefore, we can think of the expected shortfall as follows. Given a single trading list, the expected shortfall will be the value of the average implementation shortfall if we were to implement this trade list in the stock market many times.

To see this, in the code below we implement the same trade list on 50,000 trading simulations. We call each trading simulation an episode. Each episode will consist of different random fluctuations in stock price. For each episode we will compute the corresponding implemented shortfall. After all the 50,000 trading simulations have been carried out we calculate the average implementation shortfall and the variance of the implemented shortfalls. We can then compare these values with the values given by the equations for $E(x)$ and $V(x)$ from the Almgren and Chriss model.

```python
%matplotlib inline

import matplotlib.pyplot as plt

import utils

# Set the default figure size
plt.rcParams['figure.figsize'] = [17.0, 7.0]

# Set the liquidation time
l_time = 60

# Set the number of trades
n_trades = 60

# Set trader's risk aversion
t_risk = 1e-6

# Set the number of episodes to run the simulation
episodes = 10

utils.get_av_std(lq_time = l_time, nm_trades = n_trades, tr_risk = t_risk, trs = episodes)

# Get the AC Optimal strategy for the given parameters
ac_strategy = utils.get_optimal_vals(lq_time = l_time, nm_trades = n_trades, tr_risk = t_risk)
ac_strategy
```

# Extreme Trading Strategies

Because some investors may be willing to take more risk than others, when looking for the optimal strategy we have to consider a wide range of risk values, ranging from those traders that want to take zero risk to those who want to take as much risk as possible. Let's take a look at these two extreme cases. We will define the **Minimum Variance** strategy as that one followed by a trader that wants to take zero risk and the **Minimum Impact** strategy at that one followed by a trader that wants to take as much risk as possible. Let's take a look at the values of $E(x)$ and $V(x)$ for these extreme trading strategies. The `utils.get_min_param()` uses the above equations for $E(x)$ and $V(x)$, along with the parameters from the trading environment to calculate the expected shortfall and standard deviation (the square root of the variance) for these strategies. We'll start by looking at the Minimum Impact strategy.

```python
import utils

# Get the minimum impact and minimum variance strategies
minimum_impact, minimum_variance = utils.get_min_param()
```

### Minimum Impact Strategy

This trading strategy will be taken by trader that has no regard for risk. In the Almgren and Chriss model this will correspond to having the trader's risk aversion set to $\lambda = 0$. In this case the trader will sell the shares at a constant rate over a long period of time. By doing so, he will minimize market impact, but will be at risk of losing a lot of money due to the large variance. Hence, this strategy will yield the lowest possible expected shortfall and the highest possible variance, for a given set of parameters. We can see that for the given parameters, this strategy yields an expected shortfall of \$197,000 dollars but has a very big standard deviation of over 3 million dollars.

```python
minimum_impact
```

AC Optimal Strategy for Minimum Impact

  Number of Days to Sell All the Shares:	     250	       Initial Portfolio Value:	         $50,000,000.00

  Half-Life of The Trade:	                1,284,394.9	   Expected Shortfall:	                $197,000.00

  Utility:	                               $197,000.00	   Standard Deviation of Shortfall:	  $3,453,707.55

### Minimum Variance Strategy

This trading strategy will be taken by trader that wants to take zero risk, regardless of transaction costs. In the Almgren and Chriss model this will correspond to having a variance of $V(x) = 0$. In this case, the trader would prefer to sell the all his shares immediately, causing a known price impact, rather than risk trading in small increments at successively adverse prices. This strategy will yield the smallest possible variance, $V(x) = 0$, and the highest possible expected shortfall, for a given set of parameters. We can see that for the given parameters, this strategy yields an expected shortfall of over 2.5 million dollars but has a standard deviation equal of zero.

```python
minimum_variance
```

AC Optimal Strategy for Minimum Variance

  Number of Days to Sell All the Shares:	       1	         Initial Portfolio Value:	         $50,000,000.00

  Half-Life of The Trade:	                     0.2	        Expected Shortfall:	               $2,562,500.00

  Utility:	                               $2,562,500.00	   Standard Deviation of Shortfall:	      $0.00

# The Efficient Frontier

The goal of Almgren and Chriss was to find the optimal strategies that lie between these two extremes. In their paper, they showed how to compute the trade list that minimizes the expected shortfall for a wide range of risk values. In their model, Almgren and Chriss used the parameter $\lambda$ to measure a trader's risk-aversion. The value of $\lambda$ tells us how much a trader is willing to penalize the variance of the shortfall, $V(X)$, relative to expected shortfall, $E(X)$. They showed that for each value of $\lambda$ there is a uniquely determined optimal execution strategy. We define the **Efficient Frontier** to be the set of all these optimal trading strategies. That is, the efficient frontier is the set that contains the optimal trading strategy for each value of $\lambda$.

The efficient frontier is often visualized by plotting $(x,y)$ pairs for a wide range of $\lambda$ values, where the $x$-coordinate is given by the equation of the expected shortfall, $E(X)$, and the $y$-coordinate is given by the equation of the variance of the shortfall, $V(X)$. Therefore, for a given a set of parameters, the curve defined by the efficient frontier represents the set of optimal trading strategies that give the lowest expected shortfall for a defined level of risk.

In the code below, we plot the efficient frontier for $\lambda$ values in the range $(10^{-7}, 10^{-4})$, using the default parameters in our trading environment. Each point of the frontier represents a distinct strategy for optimally liquidating the same number of stocks. A risk-averse trader, who wishes to sell quickly to reduce exposure to stock price volatility, despite the trading costs incurred in doing so, will likely choose a value of $\lambda = 10^{-4}$. On the other hand, a trader
who likes risk, who wishes to postpones selling, will likely choose a value of $\lambda = 10^{-7}$. In the code, you can choose a particular value of $\lambda$ to see the expected shortfall and level of variance corresponding to that particular value of trader's risk aversion.

```python
%matplotlib inline

import matplotlib.pyplot as plt

import utils

# Set the default figure size
plt.rcParams['figure.figsize'] = [17.0, 7.0]

# Plot the efficient frontier for the default values. The plot points out the expected shortfall and variance of the
# optimal strategy for the given the trader's risk aversion. Valid range for the trader's risk aversion (1e-7, 1e-4).
utils.plot_efficient_frontier(tr_risk = 1e-6)
```

```python

```


# Scalars

```python
%matplotlib inline
```

```python
import numpy as np
import matplotlib.pyplot as plt
```

## Integers

### Binary representation of integers

```python
format(16, '032b')
```

    '00000000000000000000000000010000'

### Bit shifting

```python
format(16 >> 2, '032b')
```

    '00000000000000000000000000000100'

```python
16 >> 2
```

    4

```python
format(16 << 2, '032b')
```

    '00000000000000000000000001000000'

```python
16 << 2
```

    64

### Overflow

In general, the computer representation of integers has a limited range, and may overflow. The range depends on whether the integer is signed or unsigned.

For example, with 8 bits, we can represent at most $2^8 = 256$ integers.

- 0 to 255 unsigned
- -128 ti 127 signed

Signed integers

```python
np.arange(130, dtype=np.int8)[-5:]
```

    array([ 125,  126,  127, -128, -127], dtype=int8)

Unsigned integers

```python
np.arange(130, dtype=np.uint8)[-5:]
```

    array([125, 126, 127, 128, 129], dtype=uint8)

```python
np.arange(260, dtype=np.uint8)[-5:]
```

    array([255,   0,   1,   2,   3], dtype=uint8)

### Integer division

In Python 2 or other languages such as C/C++, be very careful when dividing as the division operator `/` performs integer division when both numerator and denominator are integers. This is rarely what you want. In Python 3 the `/` always performs floating point division, and you use `//` for integer division, removing a common source of bugs in numerical calculations.

```python
%%python2

import numpy as np

x = np.arange(10)
print(x/10)
```

    [0 0 0 0 0 0 0 0 0 0]

Python 3 does the "right" thing.

```python
x = np.arange(10)
x/10
```

    array([0. , 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9])

## Real numbers

Real numbers are represented as **floating point** numbers. A floating point number is stored in 3 pieces (sign bit, exponent, mantissa) so that every float is represetned as get +/- mantissa ^ exponent. Because of this, the interval between consecutive numbers is smallest (high precison) for numebrs close to 0 and largest for numbers close to the lower and upper bounds.

Because exponents have to be singed to represent both small and large numbers, but it is more convenint to use unsigned numbers here, the exponnent has an offset (also knwnn as the exponentn bias). For example, if the expoennt is an unsigned 8-bit number, it can rerpesent the range (0, 255). By using an offset of 128, it will now represent the range (-127, 128).

**Note**: Intervals between consecutive floating point numbers are not constant. In particular, the precision for small numbers is much larger than for large numbers. In fact, approximately half of all floating point numbers lie between -1 and 1 when using the `double` type in C/C++ (also the default for `numpy`).

Because of this, if you are adding many numbers, it is more accurate to first add the small numbers before the large numbers.

#### IEEE 754 32-bit floating point representation

See [Wikipedia](https://en.wikipedia.org/wiki/Single-precision_floating-point_format) for how this binary number is evaluated to 0.15625.

```python
from ctypes import c_int, c_float
```

```python
s = c_int.from_buffer(c_float(0.15625)).value
```

```python
s = format(s, '032b')
s
```

    '00111110001000000000000000000000'

```python
rep = {
    'sign': s[:1],
    'exponent' : s[1:9:],
    'fraction' : s[9:]
}
rep
```

    {'sign': '0', 'exponent': '01111100', 'fraction': '01000000000000000000000'}

### Most base 10 real numbers are approximations

This is simply because numbers are stored in finite-precision binary format.

```python
'%.20f' % (0.1 * 0.1 * 100)
```

    '1.00000000000000022204'

### Never check for equality of floating point numbers

```python
i = 0
loops = 0
while i != 1:
    i += 0.1 * 0.1
    loops += 1
    if loops == 1000000:
        break
i
```

    10000.000000171856

```python
i = 0
loops = 0
while np.abs(1 - i) > 1e-6:
    i += 0.1 * 0.1
    loops += 1
    if loops == 1000000:
        break
i
```

    1.0000000000000007

### Associative law does not necessarily hold

```python
6.022e23 - 6.022e23 + 1
```

    1.0

```python
1 + 6.022e23 - 6.022e23
```

    0.0

### Distributive law does not hold

```python
a = np.exp(1)
b = np.pi
c = np.sin(1)
```

```python
a*(b+c)
```

    10.82708950985241

```python
a*b + a*c
```

    10.827089509852408

### Catastrophic cancellation

Consider calculating sample variance

$$
s^2= \frac{1}{n(n-1)}\sum_{i=1}^n x_i^2 - (\sum_{i=1}^n x_i)^2
$$

Be  careful whenever you calculate the difference of potentially big numbers.

```python
def var(x):
    """Returns variance of sample data using sum of squares formula."""

    n = len(x)
    return (1.0/(n*(n-1))*(n*np.sum(x**2) - (np.sum(x))**2))
```

### Underflow

```python
np.warnings.filterwarnings('ignore')
```

```python
np.random.seed(4)
xs = np.random.random(1000)
ys = np.random.random(1000)
np.prod(xs)/np.prod(ys)
```

    nan

#### Prevent underflow by staying in log space

```python
x = np.sum(np.log(xs))
y = np.sum(np.log(ys))
np.exp(x - y)
```

    696432868222.2549

### Overflow

```python
np.exp(1000)
```

    inf

### Numerically stable algorithms

#### What is the sample variance for numbers from a normal distribution with variance 1?

```python
np.random.seed(15)
x_ = np.random.normal(0, 1, int(1e6))
x = 1e12 + x_
var(x)
```

    -1328166901.4739892

#### Use functions from numerical libraries where available

```python
np.var(x)
```

    1.001345504504934

There is also a variance function in the standard library, but it is slower for large arrays.

```python
import statistics

statistics.variance(x)
```

    1.0013464175061981

Note that `numpy` uses does not use the asymptotically unbiased estimator by default. If you want the unbiased variance, set `ddof` to 1.

```python
np.var([1,2,3,4], ddof=1)
```

    1.6666666666666667

```python
statistics.variance([1,2,3,4])
```

    1.6666666666666667

### Useful numerically stable functions

Let's calculate

$$
\log(e^{1000} + e^{1000})
$$

Using basic algebra, we get the solution $\log(2) + 1000$.

\begin{align}
\log(e^{1000} + e^{1000}) &= \log(e^{0}e^{1000} + e^{0}e^{1000}) \\
&= \log(e^{100}(e^{0} + e^{0})) \\
&= \log(e^{1000}) + \log(e^{0} + e^{0}) \\
&= 1000 + \log(2)
\end{align}

**logaddexp**

```python
x = np.array([1000, 1000])
np.log(np.sum(np.exp(x)))
```

    inf

```python
np.logaddexp(*x)
```

    1000.6931471805599

**logsumexp**

This function generalizes `logaddexp` to an arbitrary number of addends and is useful in a variety of statistical contexts.

Suppose we need to calculate a probability distribution $\pi$ parameterized by a vector $x$

$$
\pi_i = \frac{e^{x_i}}{\sum_{j=1}^n e^{x_j}}
$$

Taking logs, we get

$$
\log(\pi_i) = x_i - \log{\sum_{j=1}^n e^{x_j}}
$$

```python
x = 1e6*np.random.random(100)
```

```python
np.log(np.sum(np.exp(x)))
```

    inf

```python
from scipy.special import logsumexp
```

```python
logsumexp(x)
```

    984755.5793729819

**logp1 and expm1**

```python
np.exp(np.log(1 + 1e-6)) - 1
```

    9.999999999177334e-07

```python
np.expm1(np.log1p(1e-6))
```

    1e-06

**sinc**

```python
x = 1
```

```python
np.sin(x)/x
```

    0.8414709848078965

```python
np.sinc(x)
```

    3.8981718325193755e-17

```python
x = np.linspace(0.01, 2*np.pi, 100)
```

```python
plt.plot(x, np.sinc(x), label='Library function')
plt.plot(x, np.sin(x)/x, label='DIY function')
plt.legend()
pass
```


# Announcements
- Please familiarize yourself with the term projects, and sign up for your (preliminary) choice using [this form](https://forms.gle/ByLLpsthrpjCcxG89). _You may revise your choice, but I'd recommend settling on a choice well before Thanksgiving._
- Recommended reading on ODEs: [Lecture notes by Prof. Hjorth-Jensen (University of Oslo)](https://www.asc.ohio-state.edu/physics/ntg/6810/readings/hjorth-jensen_notes2013_08.pdf)
- Problem Set 4 posted on D2L, due Oct 5.
- __Outlook__: completing ODE examples this week, then Partial Differential Equations (PDEs)
- Conference for Undergraduate Women in Physics: online event in 2021, [applications accepted until 10/25](https://www.aps.org/programs/women/cuwip/)

# Special Considerations: Symplectic Integrators
Some systems of ODEs have special properties, for example constants of motion, that should be
conserved by the system. Simple examples in mechanics are the energy, linear momentum, and angular
momentum. Sometimes it is more important to build in to the integration scheme these conservation
laws than to achieve a high accuracy per step size. Normally, the error in the solution grows as the
solution evolves, bounded by the number of steps times the error per step.  For certain schemes,
however, while they still make errors, the errors will be of such a form that the conserved quantities
will be better preserved.

In Lecture 15 we derived the __velocity form__ of the Verlet algorithm
\begin{align}
x_{n+1} &= x_n + v_n\delta t + {1\over 2}a_n(\delta t)^2 + \mathcal{O}(\delta t^3) \tag{1a}\\
a_{n+1} &= a(x_{n+1}) \label{eq:Vv}\tag{1b}\\
v_{n+1} &= v_n + {1\over 2}\left(a_n + a_{n+1}\right)\delta t + \mathcal{O}(\delta t^2).\tag{1c}
\end{align}

While the velocity computed in this way is only
first-order accurate, if the acceleration depends only upon position, then the
velocity appears only where multiplied by $\delta t$ in determining the positions and so
second-order accuracy is preserved. Of course, if determining the velocity to
high accuracy is required, then one might turn to a different scheme. On the
other hand, if the acceleration depends upon the velocity (for example, a drag
term), there are fewer constants of the motion to be conserved anyway, and one
of the higher-order schemes developed previously would serve.
 Note that Eq.(\ref{eq:Vv}) evaluates $a_{n+1}$ before $v_{n+1}$ can be computed, so the acceleration should only be a function of position, not velocity. _This is by design, as the derivation of the Verlet algorithm was based on conservation of energy._

Here's one implementation of the velocity Verlet algorithm. Note that we compute the acceleration $a_n$ and $a_{n+1}$ in each call; this is not optimal computationally, but easier to read and use with our existing `odeSolve` code.

```python
import numpy as np
def vVerlet_step(g, t, x,v, h, *P):
    """Implements a full step h of the velocity Verlet algorithm
    Parameters:
        g: function. RHS of ODE, with signature g(t,y[0],*P).
        t: Current time t. THIS VALUE IS ONLY FOR CONSISTENCTY WITH odeSolve.
            IF t IS REQUIRED AS ARGUMENT FOR g, THEN DON'T USE VERLET INTEGRATION.
        x: Current position x.
        v: Current velocity v.
        h: float. Step length.
        *P:    tuple, additional parameters for g function
    Returns:
        x,v: numpy arrays: next position, next velocity
    """
    a_current = g(x,*P)

    #advance the position: x_n -> x_n+1
    x += v*h+a_current*h**2/2
    #compute a_n+1
    a_next = g(x,*P)
    #advance the velocity v_n -> v_n+1
    v +=h*(a_current+a_next)/2
    return x,v
```

## Example: Planetary Motion
Today we will study the orbits of two masses interacting through gravity.

In general, the gravitational force on object $j$ with mass $m_j$ and position $\vec{r}_j$ through its gravitational interaction with objects $i$ with mass $m_i$ and position $\vec{x}_i$ is given by
$$
\vec F_j = -\sum_{i\ne j}\frac{G m_j m_i}{|\vec x_i - \vec x_j|^2} \times \frac{\left(\vec x_i - \vec x_j\right)}{|\vec x_i - \vec x_j|}\,
$$
with the unit vector $\hat{\vec{x}} = \frac{\vec{x}}{|x|}$. _Note the direction of this force, gravity should pull two particles closer to each other! If that's not the case in your implementation, this is a good point to start..._

In the __special case of a two-particle system__, we can rewrite this in terms of the separation vector $\vec r = \vec x_1 - \vec x_2$ and total mass $M= m_1+m_2$ such that
the gravitational acceleration $\vec{a}_i=\vec{F}_i/m_i$ is given by
\begin{align}
\vec{a}(\vec r) &= -\frac{G M}{|r|^2} \hat{\vec{r}}\,.\label{eq:aG}\tag{2}
\end{align}
In this case, in the center of mass reference frame, the kinetic energy is given by
$$T =  \frac{1}{2}M\dot{\vec r}^2$$
and the potential energy is given by
$$U(\vec r) =-\frac{G m_1 m_2 }{|r|} $$
_Compared to previous lectures, we will now work with vectorized positions and velocities (rather than writing a separate ODE for each coordinate component). This will later require a small adaption to our function `odeSolve`:_

Now use `vVerlet_step` to __integrate simplified planetary orbits__:
Set $GM = 1$ and try initial conditions
$$
x(0) = 0.5, \quad y(0)=0, \quad v_x(0)=0, \quad v_y(0)=1.63\,.
$$
You will need to define a function `a_gravity` to evaluate the gravitational force, and later on you will need a function to calculate the potential energy `U_gravity` as well.

## Velocity Verlet vs RK4: Energy conservation
Assess the stability of `RK4` and `Velocity Verlet` by checking energy conservation over longer simulation times.

 In order to use `R4K` for integrating this problem, turn Equation(\ref{eq:aG}) into a system of ODEs for the vectorized position and velocity, so that you can reuse the code below.

```python
import numpy as np
import matplotlib
import matplotlib.pyplot as plt

def a_gravity(r, G=1, M=1):
    rr = np.sum(r*r)
    rhat = r/np.sqrt(rr)
    return - G*M/rr * rhat

def U_gravity(r, m=1, G=1, M=1):
    #this assumes the r is a 1D vector, not a 2D array
    return -G*m*M/np.sqrt(np.sum(r*r))

def analyze_Energy(r,v, m=1, G=1, M=1):
    if (len(r.shape) == 1): # r is simple vector
        return 0.5*np.sum(v*v)*M+U_gravity(r,m=m,G=G,M=M)
    else:
        n_steps, ndim = r.shape
        U = np.zeros(n_steps)
        E_tot = np.zeros(n_steps)
        for i in range(n_steps):
            U[i]=U_gravity(r[i,:],m=m,G=G,M=M)
            E_tot[i] = 0.5*np.sum(v[i,:]*v[i,:])*M+U[i]
        return E_tot,U
def simplified_verlet_wrapper(t_max =2000, h = 1):
    #it's important to define P as non-zero!
    #otherwise a_gravity sets the first keyword zero
    #i.e., G=0, so that there's no gravitational interaction...
    P=(1,1)
    dim = 2

    r0 = np.array([0.5, 0])
    v0 = np.array([0, 1.63])
    mass = 1
    nsteps = int(t_max/h)

    r = np.zeros((nsteps, dim))
    v = np.zeros_like(r)

    r[0, :] = r0
    v[0, :] = v0
    # start acceleration evaluation for first step
    a_current = a_gravity(r[0],*P)
    for i in range(nsteps-1):
    #write your Verlet integration here, without worrying about consistency with odeSolve conventions
        ##advance the position: r_i -> r_i+1
        #r[i+1] = r[i]+h*(v[i]+0.5*a_current*h)
        ##compute a_
        #a_next = a_gravity(r[i+1])
        ##advance the velocity v_i -> v_i+1
        #v[i+1] =v[i] + 0.5*h*(a_current+a_next)
        ##advance the acceleration
        #a_current = a_next
        r[i+1],v[i+1] = vVerlet_step (a_gravity, i*h, r[i],v[i], h, *P)
    return r,v
```

Let's try to run and plot this, using the default settings for now:

```python
#run the integration
r,v = simplified_verlet_wrapper()
#extract x, y coordinates and plot
rx, ry = r[:,0],r[:,1]
ax = plt.subplot(1,1,1)
ax.set_aspect(1)
ax.plot(rx, ry)
```

This looks like the separation vector simply increases with time.

_Could it be a resolution issue?_ Let's try again with smaller time step size:

```python
r,v = simplified_verlet_wrapper(t_max =100,h=.1)
rx, ry = r[:,0],r[:,1]
ax = plt.subplot(1,1,1)
ax.set_aspect(1)
ax.plot(rx, ry)

```

These are not closed orbits (as we would expect from a $1/r$ potential). But it gets much besser when stepsize is reduced to 0.001 (just rerun the code with `dt = 0.001` and replot):

```python
r,v = simplified_verlet_wrapper(t_max =100, h = 0.001)
rx, ry = r[:,0],r[:,1]
ax = plt.subplot(1,1,1)
ax.set_aspect(1)
ax.plot(rx, ry)

```

 Write down the system of ODEs that you want to solve with RK4, and define the corresponding `a_gravity_RK4` function.

Defining $\mathbf{y} =\left(\vec{r},\vec{v}\right)$, our system of linear order ODEs is
$$\dot{\mathbf{y}} = \left(\vec{v},\vec{a}(\vec{r})\right)\,,\label{eqRK4}\tag{3}$$
with $\vec{a}(\vec{r})$ as defined in Equation (2) above.

_This means that we can reuse the function `a_gravity` from above, we just need to package $\mathbf{y}$ correctly:_

```python
def a_gravity_RK4(t, y, G=1, M=1):
    """
    RHS of equation 3
        Parameters:
        t: float, current time.
        #NOT NEEDED IN THIS CALCULATION, JUST FOR CONSISTENCY WITH RK4_step
        y: numpy array, consisting of [r,v], which are each 1D numpy arrays
    """
    r = y[0]
    v = y[1]
    return np.array([v,a_gravity(r,G=1,M=1)])
```

```python
# code copied from Lecture 12/13
# you don't need to change anything in this cell
def RK4_step(t, y, h, g, *P):
    """
    Implements a single step of a fourth-order, explicit Runge-Kutta scheme
    """
    thalf = t + 0.5*h
    k1 = h * g(t, y, *P)
    k2 = h * g(thalf, y + 0.5*k1, *P)
    k3 = h * g(thalf, y + 0.5*k2, *P)
    k4 = h * g(t + h, y + k3, *P)
    return y + (k1 + 2*k2 + 2*k3 + k4)/6
```

Now, copy `simplified_verlet_wrapper` from above, and adapt it for RK4:

```python
def simplified_RK4_wrapper(t_max =2000, h = 1):
    #it's important to define P as non-zero!
    #otherwise a_gravity sets the first keyword zero
    #i.e., G=0, so that there's no gravitational interaction...
    P=(1,1)
    dim = 2

    r0 = np.array([0.5, 0])
    v0 = np.array([0, 1.63])
    mass = 1
    nsteps = int(t_max/h)
    #now define y as outlined above
    y = np.zeros((nsteps, 2, dim))
    #initial conditions
    y[0] = np.array([r0,v0])
    for i in range(nsteps-1):
        #set y[i+1], which is a numpy array of shape (2,dim)
        y[i+1] = RK4_step (i*h, y[i], h, a_gravity_RK4,*P)
    #now split y into r, v arrays for consistency with Verlet version
    return y[:,0,:],y[:,1,:]

```

```python
r,v = simplified_RK4_wrapper(t_max =100, h = 0.1)
rx, ry = r[:,0],r[:,1]
ax = plt.subplot(1,1,1)
ax.set_aspect(1)
ax.plot(rx, ry)
```

 Next, we will consider the conservation of energy of the velocity Verlet algorithm and RK4.
Plot the total energy for both solvers as a function of time, for varying step sizes.

Below I show the total energy and potential energy over short periods for a short interval in time, as well as the total energy up to $t_\mathrm{max}$:

```python
#run for a long time
t_max= 2000
h = 0.1
t = np.arange(0,t_max,h)
r,v = simplified_RK4_wrapper(t_max = t_max,h = h)
ERK4,URK4 = analyze_Energy(r,v)
r,v = simplified_verlet_wrapper(t_max = t_max,h = h)
Ev,Uv = analyze_Energy(r,v)
plt.title("h=%.3f"%(h))
plt.plot(t,Ev, label="E - Verlet")
plt.plot(t,ERK4, label="E - RK4")
plt.plot(t,Uv, label="U - Verlet")
plt.plot(t,URK4, label="U - RK4")
plt.xlabel("time")
plt.xlim(0,10)
plt.legend()
plt.show()
plt.title("h=%.3f"%(h))
plt.plot(t,Ev, label="E - Verlet")
plt.plot(t,ERK4, label="E - RK4")
plt.plot(t,Uv, label="U - Verlet")
plt.plot(t,URK4, label="U - RK4")
plt.xlabel("time")
plt.xlim(990,1000)
plt.legend()
plt.show()

plt.plot(t,Ev, label="E - Verlet, h=%.3f"%(h))
plt.plot(t,ERK4, label="E - RK4, h=%.3f"%(h))
h = 0.001
t = np.arange(0,t_max,h)
r,v = simplified_RK4_wrapper(t_max = t_max,h = h)
ERK4,URK4 = analyze_Energy(r,v)
r,v = simplified_verlet_wrapper(t_max = t_max,h = h)
Ev,Uv = analyze_Energy(r,v)
plt.plot(t,Ev, label="E - Verlet, h=%.3f"%(h))
plt.plot(t,ERK4, label="E - RK4, h=%.3f"%(h))
plt.xlabel("time")
plt.legend()
plt.show()
plt.title("h=%.3f"%(h))
plt.plot(t,Ev, label="E - Verlet")
plt.plot(t,ERK4, label="E - RK4")
plt.plot(t,Uv, label="U - Verlet")
plt.plot(t,URK4, label="U - RK4")
plt.xlabel("time")
plt.xlim(990,1000)
plt.legend()
plt.show()

```

**Velocity Verlet** only has moderate accuracy, especially when compared to **RK4**, which is to be expected by the different order of convergence.
However, it achieves good long-term stability even at coarse step size compared to RK4.

In order to compare algorithms at comparable order, let's repeat this comparison with **RK2**:

```python
def RK2_step(t, y, h, g, *P):
    """
    Implements a single step of the second-order, explicit midpoint method
    """
    thalf = t + 0.5*h
    k1 = h * g(t, y, *P)
    k2 = h * g(thalf, y + 0.5*k1, *P)

    return y +k2

def simplified_RK2_wrapper(t_max =2000, h = 1):
    #it's important to define P as non-zero!
    #otherwise a_gravity sets the first keyword zero
    #i.e., G=0, so that there's no gravitational interaction...
    P=(1,1)
    dim = 2

    r0 = np.array([0.5, 0])
    v0 = np.array([0, 1.63])
    mass = 1
    nsteps = int(t_max/h)
    #now define y as outlined above
    y = np.zeros((nsteps, 2, dim))
    #initial conditions
    y[0] = np.array([r0,v0])
    for i in range(nsteps-1):
        #set y[i+1], which is a numpy array of shape (2,dim)
        y[i+1] = RK2_step (i*h, y[i], h, a_gravity_RK4,*P)
    #now split y into r, v arrays for consistency with Verlet version
    return y[:,0,:],y[:,1,:]
```

```python
#run for a long time
t_max= 2000
h = 0.1
t = np.arange(0,t_max,h)
r,v = simplified_RK2_wrapper(t_max = t_max,h = h)
ERK2,URK2 = analyze_Energy(r,v)
r,v = simplified_verlet_wrapper(t_max = t_max,h = h)
Ev,Uv = analyze_Energy(r,v)
plt.title("h=%.3f"%(h))
plt.plot(t,Ev, label="E - Verlet")
plt.plot(t,ERK2, label="E - RK2")
plt.plot(t,Uv, label="U - Verlet")
plt.plot(t,URK2, label="U - RK2")
plt.xlabel("time")
plt.xlim(0,10)
plt.legend()
plt.show()
plt.title("h=%.3f"%(h))
plt.plot(t,Ev, label="E - Verlet")
plt.plot(t,ERK2, label="E - RK2")
plt.plot(t,Uv, label="U - Verlet")
plt.plot(t,URK2, label="U - RK2")
plt.xlabel("time")
plt.xlim(990,1000)
plt.legend()
plt.show()

plt.plot(t,Ev, label="E - Verlet, h=%.3f"%(h))
plt.plot(t,ERK2, label="E - RK2, h=%.3f"%(h))
h = 0.001
t = np.arange(0,t_max,h)
r,v = simplified_RK2_wrapper(t_max = t_max,h = h)
ERK2,URK2 = analyze_Energy(r,v)
r,v = simplified_verlet_wrapper(t_max = t_max,h = h)
Ev,Uv = analyze_Energy(r,v)
plt.plot(t,Ev, label="E - Verlet, h=%.3f"%(h))
plt.plot(t,ERK2, label="E - RK2, h=%.3f"%(h))
plt.xlabel("time")
plt.legend()
plt.show()
```

```python

```


Elasticity solution for wedge-shaped problems
======================================

In this notebook an analytic solution for a wedge is constructed. Later on, we couple two wedge solutions to compute the displacement for a dipping fault with free surface.

Wedge solution
----------------------

Polar coordinates are given by
\begin{align}
    x &= r\cos(\theta) \\
    y &= r\sin(\theta) \\
    r &= \sqrt{x^2+y^2} \\
    \theta &= \arctan(y/x) \\
\end{align}

For a plane-strain problem, we assume there exists a biharmonic function $\phi$ such that [1]

\begin{align}
    2\mu u_r &= -\frac{\partial \phi}{\partial r} + (1-\nu)r\frac{\partial\psi}{\partial \theta} \\
    2\mu u_\theta &= -\frac{1}{r}\frac{\partial \phi}{\partial \theta} + (1-\nu)r^2\frac{\partial\psi}{\partial r} \\
    \sigma_{rr} &= \frac{1}{r}\frac{\partial \phi}{\partial r} +
    \frac{1}{r^2}\frac{\partial^2 \phi}{\partial \theta^2} \\
    \sigma_{\theta\theta} &= \frac{\partial^2 \phi}{\partial r^2} \\
    \sigma_{r\theta} &= -\frac{\partial }{\partial r} \left(
      \frac{1}{r}\frac{\partial \phi}{\partial\theta}
    \right) \\
\end{align}
where $\mu$ is the shear modulus, $\nu$ is Poisson's ratio, and $\phi$ and $\psi$ are related by
$$
\nabla^2\phi = \frac{\partial}{\partial r}\left(r\frac{\partial\psi}{\partial\theta}\right)
$$

Remark: Plane-stress is modelled in [1], therefore the formula are adjusted here for plane-strain.

[1] "Stress Singularites Resulting From Various Boundary Conditions in Angular Corners of Plates in Extension", M. L. Williams, Journal of Applied Mechanics, 19 (4), pp. 526-528.

```python
from sympy import *
init_printing()
```

Assume

$$
\phi = r^{\gamma+1} \left(A_1\sin((\gamma+1)\theta) + A_2\cos((\gamma+1)\theta) + A_3\sin((\gamma-1)\theta) + A_4\sin((\gamma-1)\theta)\right)
$$
and
$$
\psi = r^{\gamma-1}\left(-\frac{4A_3}{\gamma-1}\cos((\gamma-1)t) + \frac{4A_4}{\gamma-1}\sin((\gamma-1)t)\right)
$$

```python
r, t, x, y, gam = symbols('r theta x y gamma')
mu, nu, m, A0, A1, A2, A3, A4, S = symbols('mu nu m A_0 A_1 A_2 A_3 A_4 S')

phi = r**(gam+1) * (A1 * sin((gam+1)*t) + A2*cos((gam+1)*t) + A3*sin((gam-1)*t) + A4*cos((gam-1)*t))
psi = r**(gam-1) * (-4*A3/(gam-1)*cos((gam-1)*t) + 4*A4/(gam-1)*sin((gam-1)*t))

phi2 = phi.subs(gam, 2)
psi2 = psi.subs(gam, 2)
(phi2, psi2)
```

$\psi$ must be harmonic, i.e. $\nabla^2\psi=0$, and $\phi$ must be biharmonic, i.e. $\nabla^2(\nabla^2\phi=0)$:

```python
def laplace(f):
    return f.diff(r, 2) + f.diff(r)/r + f.diff(t, 2)/r**2
(laplace(psi).simplify(), laplace(laplace(phi)).simplify())
```

The relation of $\phi$ and $\psi$ must be satisfied:

```python
((r*psi.diff(t)).diff(r) - laplace(phi)).simplify()
```

Define displacement and stress:

```python
def quants(phi, psi):
    ur = (-phi.diff(r) + (1-nu)*r*psi.diff(t)) / (2 * mu)
    ut = (-phi.diff(t)/r + (1-nu)*r**2*psi.diff(r)) / (2 * mu)
    srr = (phi.diff(r) / r + phi.diff(t, 2) / r**2)
    stt = phi.diff(r, 2)
    srt = -(phi.diff(t)/r).diff(r)
    return (ur.simplify(), ut.simplify(), srr.simplify(), stt.simplify(), srt.simplify())
ur, ut, srr, stt, srt = quants(phi, psi)
```

Check that stress-displacement relation is satisfied:

```python
err = ur.diff(r)
ett = ut.diff(t)/r + ur/r
ert = (ur.diff(t)/r + ut.diff(r) - ut/r) / 2
((err - ((1-nu)*srr - nu*stt)/(2*mu)).simplify(),
 (ett - ((1-nu)*stt - nu*srr)/(2*mu)).simplify(),
 (ert - srt/(2*mu)).simplify())
```

The planes $\theta=0$ and $\theta=-\pi/3$ form an infinite wedge.
We set $\gamma=2$ and recover the coefficients $A_1,A_2,A_3,A_4$ which satisfy
\begin{align}
    u_\theta(r,-\pi/3) &= 0 \\
    u_r(r,-\pi/3) &= Sr^2 \\
    \sigma_{r\theta}(r,0) &= 0 \\
    \sigma_{\theta\theta}(r,0) &= 0
\end{align}

```python
w1 = 0
w2 = -pi/3
mygam = 2
solution = linsolve([ut.subs(gam, mygam).subs(t, w2)/r**2,
                     ur.subs(gam, mygam).subs(t, w2)/r**2 - S,
                     srt.subs(gam, mygam).subs(t, w1)/r,
                     stt.subs(gam, mygam).subs(t, w1)/r],
         A1, A2, A3, A4)
a1, a2, a3, a4 = solution.args[0]
a1, a2, a3, a4 = a1.simplify(), a2.simplify(), a3.simplify(), a4.simplify()
(a1, a2, a3, a4)
```

We check that the boundary conditions are satisfied:

```python
srr_s = srr.subs(gam, mygam).subs(A1, a1).subs(A2, a2).subs(A3, a3).subs(A4, a4).simplify()
stt_s = stt.subs(gam, mygam).subs(A1, a1).subs(A2, a2).subs(A3, a3).subs(A4, a4).simplify()
srt_s = srt.subs(gam, mygam).subs(A1, a1).subs(A2, a2).subs(A3, a3).subs(A4, a4).simplify()
(stt_s.subs(t, w1), srt_s.subs(t, w1))
```

```python
ur_s = ur.subs(gam, mygam).subs(A1, a1).subs(A2, a2).subs(A3, a3).subs(A4, a4).simplify()
ut_s = ut.subs(gam, mygam).subs(A1, a1).subs(A2, a2).subs(A3, a3).subs(A4, a4).simplify()
(ur_s.subs(t, w2).simplify(), ut_s.subs(t, w2).simplify())
```

We convert the solution into Lua code:

```python
def lua_code(expr):
    return rcode(expr).replace('cos', 'math.cos').replace('sin', 'math.sin').replace('sqrt', 'math.sqrt').replace('pi', 'math.pi')
print("local A_1 =", lua_code(a1))
print("local A_2 =", lua_code(a2))
print("local A_3 =", lua_code(a3))
print("local A_4 =", lua_code(a4))
print("local ur =", lua_code(ur.subs(gam, mygam).simplify()))
print("local ut =", lua_code(ut.subs(gam, mygam).simplify()))
```

    local A_1 = (1.0/3.0)*math.sqrt(3)*S*mu*(4*nu - 5)/(-16*nu^2 + 20*nu + 5)
    local A_2 = S*mu*(7 - 4*nu)/(-16*nu^2 + 20*nu + 5)
    local A_3 = math.sqrt(3)*S*mu*(5 - 4*nu)/(-16*nu^2 + 20*nu + 5)
    local A_4 = S*mu*(4*nu - 7)/(-16*nu^2 + 20*nu + 5)
    local ur = (1.0/2.0)*r^2*(-3*A_1*math.sin(3*theta) - 3*A_2*math.cos(3*theta) - 4*A_3*nu*math.sin(theta) + A_3*math.sin(theta) - 4*A_4*nu*math.cos(theta) + A_4*math.cos(theta))/mu
    local ut = (1.0/2.0)*r^2*(-3*A_1*math.cos(3*theta) + 3*A_2*math.sin(3*theta) - A_3*math.cos(theta) + A_4*math.sin(theta) + (4*nu - 4)*(A_3*math.cos(theta) - A_4*math.sin(theta)))/mu

Dipping fault
------------------

We now couple two wedge solutions to impose a displacement in a dipping fault with a free-surface condition, as illustrated in the following figure:

where $[\![ u ]\!] = u_A - u_B$. The A-side and B-side of the fault are separated by the fault plane $\theta=\omega$. Also note that the origin, marked with a circle, is at the intesection of the fault and the free surface.

We introduce potentials for the A-side and the B-side of the fault:

```python
B1, B2, B3, B4 = symbols('B_1 B_2 B_3 B_4')

phiA = phi.simplify()
psiA = psi.simplify()
phiB = phiA.subs(A1, B1).subs(A2, B2).subs(A3, B3).subs(A4, B4).simplify()
psiB = psiA.subs(A1, B1).subs(A2, B2).subs(A3, B3).subs(A4, B4).simplify()
(phiA, psiA, phiB, psiB)
```

```python
urA, utA, srrA, sttA, srtA = quants(phiA, psiA)
urB, utB, srrB, sttB, srtB = quants(phiB, psiB)
```

We remove the dependency on $r$ here to solve for the A and B coefficients:

```python
sttA_e = (sttA*r**(1-gam)/gam).simplify()
srtA_e = (srtA*r**(1-gam)/gam).simplify()
sttB_e = (sttB*r**(1-gam)/gam).simplify()
srtB_e = (srtB*r**(1-gam)/gam).simplify()
(sttA_e, srtA_e, sttB_e, srtB_e)
```

```python
def simp_A(expr):
    return factor(expr, *parsA).simplify()
def simp_B(expr):
    return factor(expr, *parsB).simplify()

parsA = (A1, A2, A3, A4)
urA_e = simp_A(urA*r**(-gam)*2*mu)
utA_e = simp_A(utA*r**(-gam)*2*mu)
parsB = (B1, B2, B3, B4)
urB_e = simp_B(urB*r**(-gam)*2*mu)
utB_e = simp_B(utB*r**(-gam)*2*mu)
(urA_e, utA_e, urB_e, utB_e)
```

We first compute coefficients which ensure the free-surface boundary condition. These calculations are independent for the A-side and B-side. Note that the A-wedge is given by planes $\theta=\pi$ and $\theta=\omega$, and the B-wedge is given by planes $\theta=0$ and $\theta=\omega$.

```python
part_solution_A = solve([sttA_e.subs(t, pi),
                         srtA_e.subs(t, pi)],
                         *parsA)
part_solution_A = {k: v.simplify() for k, v in part_solution_A.items()}
part_solution_B = solve([sttB_e.subs(t, 0),
                         srtB_e.subs(t, 0)],
                         *parsB)
part_solution_B = {k: v.simplify() for k, v in part_solution_B.items()}
(part_solution_A, part_solution_B)
```

We can now express $A_1,A_2,B_1,B_2$ in terms of $A_3,A_4,B_3,B_3$, which we substitute in the following:

```python
sttA_e2 = simp_A(sttA_e.subs(part_solution_A.items()))
srtA_e2 = simp_A(srtA_e.subs(part_solution_A.items()))
sttB_e2 = simp_B(sttB_e.subs(part_solution_B.items()))
srtB_e2 = simp_B(srtB_e.subs(part_solution_B.items()))
(sttA_e2, srtA_e2, sttB_e2, srtB_e2)
```

```python
utA_e2 = simp_A(utA_e.subs(part_solution_A.items()))
urA_e2 = simp_A(urA_e.subs(part_solution_A.items()))
urB_e2 = simp_B(urB_e.subs(part_solution_B.items()))
utB_e2 = simp_B(utB_e.subs(part_solution_B.items()))
(urA_e2, utA_e2, urB_e2, utB_e2)
```

We now find the remaining coefficients for the imposed displacement on the fault plane $\omega = -\pi/3$. Note that we require the stress tensor to be continuous.

```python
w = -pi/3
equations = [(sttA_e2 - sttB_e2).subs(t, w).simplify(),
             (srtA_e2 - srtB_e2).subs(t, w).simplify(),
             (utA_e2 - utB_e2).subs(t, w).simplify(),
             (urA_e2 - urB_e2 - S).subs(t, w).simplify()]
equations
```

```python
part_equations_0 = [factor(equations[0] + equations[3], *parsA, *parsB).simplify(),
                    factor(equations[1] - equations[2], *parsA, *parsB).simplify()]
part_equations_0
```

```python
part_solution_0 = solve(part_equations_0, A3, A4, B3, B4)
part_solution_0 = {k: v.simplify() for k, v in part_solution_0.items()}
part_solution_0
```

```python
part_equations_1 = [factor(equations[0].subs(part_solution_0.items()).simplify(), *parsB).simplify(),
                    factor(equations[1].subs(part_solution_0.items()).simplify(), *parsB).simplify()]
```

```python
solution = solve(part_equations_1, B3, B4)
solution = {k: v.simplify() for k, v in solution.items()}
solution
```

Note that $\sin(\pi\gamma)$ appears in the denominator above.
Thus, $\gamma$ must not be an integer.

We substitute the parameters into the quantities for $\gamma = 0.5$:

```python
mygam = 0.5
def subs_solution_A(expr):
    return expr.subs(part_solution_A.items()).subs(part_solution_0.items()).subs(solution.items()).subs(gam, mygam)
def subs_solution_B(expr):
    return expr.subs(part_solution_B.items()).subs(part_solution_0.items()).subs(solution.items()).subs(gam, mygam)
```

```python
srrA_s = subs_solution_A(srrA).simplify()
sttA_s = subs_solution_A(sttA).simplify()
srtA_s = subs_solution_A(srtA).simplify()
urA_s = subs_solution_A(urA).simplify()
utA_s = subs_solution_A(utA).simplify()

srrB_s = subs_solution_B(srrB).simplify()
sttB_s = subs_solution_B(sttB).simplify()
srtB_s = subs_solution_B(srtB).simplify()
urB_s = subs_solution_B(urB).simplify()
utB_s = subs_solution_B(utB).simplify()
```

In the following we test whether the boundary conditions are satisifed.

```python
(sttA_s.subs(t, pi).simplify(),
 srtA_s.subs(t, pi).simplify(),
 sttB_s.subs(t, 0).simplify(),
 srtB_s.subs(t, 0).simplify())
```

```python
((sttA_s.subs(t, w)-sttB_s.subs(t, w)).simplify().evalf(),
 (srtA_s.subs(t, w)-srtB_s.subs(t, w)).simplify().evalf(),
 (utA_s.subs(t, w)-utB_s.subs(t, w)).simplify().evalf())
```

```python
((urA_s.subs(t, w) - urB_s.subs(t, w) - S).simplify() + S).evalf().simplify()
```

Lastly, we generate Lua code for the A-side and the B-side:

```python
def lua_code_cse(var_expr_pairs):
    vs, es = zip(*var_expr_pairs)
    e_cse = cse(es)
    for k, v in e_cse[0]:
        print("local {} = {}".format(k, lua_code(v)))
    for k, v in zip(vs, e_cse[1]):
        print("local {} = {}".format(k, lua_code(v)))
```

$B_3$ and $B_4$

```python
lua_code_cse(solution.items())
```

    local x0 = math.pi*((1.0/3.0)*gamma + 1.0/3.0)
    local x1 = math.sin(x0)
    local x2 = 2*x1
    local x3 = math.pi*((7.0/3.0)*gamma + 1.0/3.0)
    local x4 = math.sin(x3)
    local x5 = 2*x4
    local x6 = 3*gamma
    local x7 = math.cos(x0)
    local x8 = math.sqrt(3)*gamma
    local x9 = math.cos(x3)
    local x10 = math.pi*gamma
    local x11 = (8.0/3.0)*x10
    local x12 = math.sin(x11)
    local x13 = 2*x7
    local x14 = 2*x9
    local x15 = math.cos(x11)
    local x16 = (1.0/32.0)*S/((nu - 1)*math.sin(x10)^2)
    local B_3 = x16*(x1*x6 + x12*x13 - x12*x14 - x15*x2 + x15*x5 + x2 - x4*x6 - x5 + x7*x8 - x8*x9)
    local B_4 = x16*(-x1*x8 - x12*x2 + x12*x5 - x13*x15 + x13 + x14*x15 - x14 + x4*x8 + x6*x7 - x6*x9)

$A_3$ and $A_4$

```python
lua_code_cse(part_solution_0.items())
```

    local x0 = nu - 1
    local x1 = 1.0/x0
    local x2 = math.pi*((1.0/3.0)*gamma + 1.0/6.0)
    local x3 = (1.0/4.0)*S
    local A_3 = x1*(B_3*x0 - x3*math.cos(x2))
    local A_4 = x1*(B_4*x0 - x3*math.sin(x2))

$A_1$ and $A_2$

```python
lua_code_cse(part_solution_A.items())
```

    local x0 = 1.0/(gamma + 1)
    local x1 = 2*math.pi*gamma
    local x2 = math.cos(x1)
    local x3 = math.sin(x1)
    local A_1 = x0*(-A_3*gamma + A_3*x2 - A_4*x3)
    local A_2 = -x0*(A_3*x3 + A_4*gamma + A_4*x2)

$B_1$ and $B_2$

```python
lua_code_cse(part_solution_B.items())
```

    local B_2 = -B_4
    local B_1 = B_3*(1 - gamma)/(gamma + 1)

```python
urA
```

A-side: $u_r$ and $u_\theta$

```python
lua_code_cse([('urA', urA), ('utA', utA)])
```

    local x0 = gamma - 1
    local x1 = theta*x0
    local x2 = math.sin(x1)
    local x3 = math.cos(x1)
    local x4 = A_3*x2 + A_4*x3
    local x5 = gamma + 1
    local x6 = theta*x5
    local x7 = math.sin(x6)
    local x8 = math.cos(x6)
    local x9 = (1.0/2.0)*r^gamma/mu
    local x10 = A_4*x2
    local x11 = A_3*x3
    local urA = x9*(4*x4*(1 - nu) - x5*(A_1*x7 + A_2*x8 + x4))
    local utA = x9*(-A_1*x5*x8 + A_2*x5*x7 + x0*x10 - x0*x11 + (4*nu - 4)*(-x10 + x11))

B-side: $u_r$ and $u_\theta$

```python
lua_code_cse([('urB', urB), ('utB', utB)])
```

    local x0 = gamma - 1
    local x1 = theta*x0
    local x2 = math.sin(x1)
    local x3 = math.cos(x1)
    local x4 = B_3*x2 + B_4*x3
    local x5 = gamma + 1
    local x6 = theta*x5
    local x7 = math.sin(x6)
    local x8 = math.cos(x6)
    local x9 = (1.0/2.0)*r^gamma/mu
    local x10 = B_4*x2
    local x11 = B_3*x3
    local urB = x9*(4*x4*(1 - nu) - x5*(B_1*x7 + B_2*x8 + x4))
    local utB = x9*(-B_1*x5*x8 + B_2*x5*x7 + x0*x10 - x0*x11 + (4*nu - 4)*(-x10 + x11))

A-side: $u_{r,r}, u_{r,\theta}, u_{\theta,r}, u_{\theta,\theta}$

```python
lua_code_cse([('urA_r', urA.diff(r).simplify()),
              ('urA_t', urA.diff(t).simplify()),
              ('utA_r', utA.diff(r).simplify()),
              ('utA_t', utA.diff(t).simplify())])
```

    local x0 = 4*nu - 4
    local x1 = gamma - 1
    local x2 = theta*x1
    local x3 = math.sin(x2)
    local x4 = A_3*x3
    local x5 = math.cos(x2)
    local x6 = A_4*x5
    local x7 = x4 + x6
    local x8 = gamma + 1
    local x9 = theta*x8
    local x10 = math.sin(x9)
    local x11 = A_1*x10
    local x12 = math.cos(x9)
    local x13 = A_2*x12
    local x14 = (1.0/2.0)/mu
    local x15 = gamma*r^x1*x14
    local x16 = A_3*x5
    local x17 = A_4*x3
    local x18 = x16 - x17
    local x19 = 4*x1*(nu - 1)
    local x20 = A_1*x12*x8
    local x21 = x1*x16
    local x22 = A_2*x10*x8
    local x23 = x1*x17
    local x24 = r^gamma*x14
    local x25 = x8^2
    local x26 = x1^2
    local urA_r = -x15*(x0*x7 + x8*(x11 + x13 + x7))
    local urA_t = -x24*(x18*x19 + x8*(x20 + x21 - x22 - x23))
    local utA_r = x15*(x0*x18 - x20 - x21 + x22 + x23)
    local utA_t = x24*(x11*x25 + x13*x25 - x19*x7 + x26*x4 + x26*x6)

B-side: $u_{r,r}, u_{r,\theta}, u_{\theta,r}, u_{\theta,\theta}$

```python
lua_code_cse([('urB_r', urB.diff(r).simplify()),
              ('urB_t', urB.diff(t).simplify()),
              ('utB_r', utB.diff(r).simplify()),
              ('utB_t', utB.diff(t).simplify())])
```

    local x0 = 4*nu - 4
    local x1 = gamma - 1
    local x2 = theta*x1
    local x3 = math.sin(x2)
    local x4 = B_3*x3
    local x5 = math.cos(x2)
    local x6 = B_4*x5
    local x7 = x4 + x6
    local x8 = gamma + 1
    local x9 = theta*x8
    local x10 = math.sin(x9)
    local x11 = B_1*x10
    local x12 = math.cos(x9)
    local x13 = B_2*x12
    local x14 = (1.0/2.0)/mu
    local x15 = gamma*r^x1*x14
    local x16 = B_3*x5
    local x17 = B_4*x3
    local x18 = x16 - x17
    local x19 = 4*x1*(nu - 1)
    local x20 = B_1*x12*x8
    local x21 = x1*x16
    local x22 = B_2*x10*x8
    local x23 = x1*x17
    local x24 = r^gamma*x14
    local x25 = x8^2
    local x26 = x1^2
    local urB_r = -x15*(x0*x7 + x8*(x11 + x13 + x7))
    local urB_t = -x24*(x18*x19 + x8*(x20 + x21 - x22 - x23))
    local utB_r = x15*(x0*x18 - x20 - x21 + x22 + x23)
    local utB_t = x24*(x11*x25 + x13*x25 - x19*x7 + x26*x4 + x26*x6)

```python

```


Examples for Ordinary Differential Equations (Initial Value Problem)
===============================================

## Part 2

### Suggestion

Run the code cell below before the others to make sure that scipy is up to date and the figures render well.

```python
import matplotlib.pyplot as plt
import warnings
warnings.filterwarnings('ignore')
plt.rcParams['figure.figsize'] = (8,6)
plt.rcParams['font.size'] = 20
!pip3 install --upgrade scipy
```

**Simulation of Non-Linear Dynamics Problems:**

### The Problem: The Van der Pol Oscillator
Van der Pol’s differential equation is

\begin{equation}
\frac{d^2x}{dt^2} - \mu\left(1-x^2\right)\frac{dx}{dt} + x = 0.
\end{equation}

The equation describes a system with nonlinear damping, the degree of nonlinearity given by $\mu$. If $\mu=0$, the system is linear and undamped, but as $\mu$ increases, so does the strength of the nonlinearity.

Plot the phase portrait for the solution to Van der Pol’s equation in Python using SciPy’s ODE solver 'ivp_solve'.
We can recast the equation above in terms of the phase space point $y = \left[y_0, y_1\right] = \left[x, \dot{x}\right]$, yielding

\begin{align*}
\dot{y}_0 &= y_1\\
\dot{y}_1 &=  \mu\left(1-x^2\right) y_1 - y_0
\end{align*}

If $\mu=0$, we have a simple harmonic oscillator and the phase portrait is simply a circle. For larger values of $\mu$, the solutions enter limit cycles, but the cycles are more complicated than just circles. These lcycles are periodic attractors: every non-trivial solution converges to the limit cycle.

The code cell below yields the phase space plot.

```python
import numpy as np
from scipy.integrate import solve_ivp
import matplotlib.pyplot as plt
plt.rcParams['figure.figsize'] = (9,12)

def vdp(t, z):
    x, y = z
    return [y, mu*(1 - x**2)*y - x]

a, b = 0, 10

mus = [0, 0.5, 1, 2]
styles = ["-", "-.", "--", ":"]
t = np.linspace(a, b, 500)

for mu, style in zip(mus, styles):
    sol = solve_ivp(vdp, [a, b], [1, 0], t_eval=t)
    plt.plot(sol.y[0], sol.y[1], linestyle=style, label=f"$\mu={mu:1.1f}$")

# make a little extra horizontal room for legend
plt.xlim([-3,3])
plt.legend()
plt.show()
```

### The Problem: Limit Cycles in the Van der Pol Oscillator

A couple questions come to mind. First, how quickly do the solutions become periodic? Second, how does the period depend on $\mu$? To address these questions, we’ll use an optional argument to ivp_solve we have not used before.

For 'ivp_solve', an *event* is a function of the time $t$ and the solution $y$ whose roots the solver will report. To determine the period, we’ll look at where the solution is zero; our event function is trivial since we want to find the roots of the solution itself.

Since we have cast our second order ODE as a pair of first order ODEs, our solution is a vector, the function $x$ and its derivative. So to find roots of the solution, we look at what the solver sees as the first component of the solver. So, our 'event' function can simply be:

```python
def root(t, y): return y[0]
```

Let’s set $\mu = 2$ and find the zeros of the solution over the interval $[0, 40]$, starting from the initial condition $x(0) = 1, \dot{x}(0) = 0$.

```python
mu = 0.8
sol = solve_ivp(vdp, [0, 200], [1, 0], events=root)
zeros = sol.t_events[0]
print(zeros)
```

    [  1.41013366   4.58431762   7.82882227  11.08463631  14.34102232
      17.59468017  20.85278725  24.11120772  27.36941157  30.62141203
      33.87377408  37.12589835  40.37807001  43.63023298  46.88239774
      50.13456216  53.38672665  56.63889113  59.8910556   63.14322008
      66.39538456  69.64754904  72.89971351  76.15187799  79.40404247
      82.65620694  85.90837142  89.1605359   92.41270037  95.66486485
      98.91702933 102.16919381 105.42135828 108.67352276 111.92568724
     115.17785171 118.43001619 121.68218067 124.93434515 128.18650962
     131.4386741  134.69083858 137.94300305 141.19516753 144.44733201
     147.69949648 150.95166096 154.20382544 157.45598992 160.70815439
     163.96031887 167.21248335 170.46464782 173.7168123  176.96897678
     180.22114126 183.47330573 186.72547021 189.97763469 193.22979916
     196.48196364 199.73412817]

To estimate the period of the limit cycle we look at the spacing between zeros, and how that spacing is changing. Finally, if we plot the deltas we see that the zero spacings quickly approach a constant value. Zero crossings are half periods, so the period of the limit cycle is twice the limiting spacing between zeros.

```python
from scipy.integrate import solve_ivp
import matplotlib.pyplot as plt
plt.rcParams['figure.figsize'] = (8,6)

mu = 0.8

sol = solve_ivp(vdp, [0, 200], [1, 0], events=root)
zeros = sol.t_events[0]
spacing = zeros[1:] - zeros[:-1]
deltas = spacing[1:] - spacing[:-1]
plt.plot(deltas)
plt.show()
print(f"Time period of cycle for mu = {mu:2.1f} is {2 * spacing[-1]:2.2f}")
```

### Homework

As homework, the student is encouraged to determine the following numerically:

* Formulate and execute a Python program for estimating the period of the limit cycle of a Van der Pol oscillator for a given value of $\mu$. Plot the period as a function of $\mu$

* Run this program for small $\mu$ and large $\mu\sim 100$. The method described above should fail for large $\mu$. Plot $x$ vs $t$ for large $\mu$ and explain why it fails.

* Consider the previous Van der Pol oscillator problem, but now with a drive $F(t) = \cos{\left(2\pi t/T\right)}$. Plot the phase space for $T=10$ and $\mu=9$. Explain the changes that take place as you vary $\mu$ keeping $T$ fixed.

### The Problem: The Lorenz Attractor

The Lorenz system of coupled, ordinary, first-order differential equations have chaotic solutions for certain parameter values $\sigma, \rho$ and $\beta$ and initial conditions, $u(0), v(0)$ and $w(0)$. Note that this system is *dissipative*.

\begin{aligned}
\frac{\mathrm{d} u}{\mathrm{~d} t} &=\sigma(v-u) \\
\frac{\mathrm{d} v}{\mathrm{~d} t} &=\rho u-v-u w \\
\frac{\mathrm{d} w}{\mathrm{~d} t} &=u v-\beta w
\end{aligned}

The following program plots the Lorenz attractor (the values of $x, y$ and $z$ as a parametric function of time) on a Matplotlib 3D projection.

```python
import numpy as np
from scipy.integrate import solve_ivp
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
WIDTH, HEIGHT, DPI = 1000, 750, 100

# Lorenz paramters and initial conditions.
sigma, beta, rho = 10, 2.667, 28
u0, v0, w0 = 1, 1, 1

# Maximum time point and total number of time points.
tmax, n = 100, 10000

def lorenz(t, X, sigma, beta, rho):
    """The Lorenz equations."""
    u, v, w = X
    up = -sigma*(u - v)
    vp = rho*u - v - u*w
    wp = -beta*w + u*v
    return up, vp, wp

# Integrate the Lorenz equations.
soln = solve_ivp(lorenz, (0, tmax), (u0, v0, w0), args=(sigma, beta, rho),
                 dense_output=True)
# Interpolate solution onto the time grid, t.
t = np.linspace(0, tmax, n)
x, y, z = soln.sol(t)

# Plot the Lorenz attractor using a Matplotlib 3D projection.
fig = plt.figure(facecolor='k', figsize=(WIDTH/DPI, HEIGHT/DPI))
ax = fig.gca(projection='3d')
ax.set_facecolor('k')
fig.subplots_adjust(left=0, right=1, bottom=0, top=1)

# Comment all below for animation
# Make the line multi-coloured by plotting it in segments of length s which
# change in colour across the whole time series.
s = 10
cmap = plt.cm.winter
for i in range(0,n-s,s):
    ax.plot(x[i:i+s+1], y[i:i+s+1], z[i:i+s+1], color=cmap(i/n), alpha=0.4)

# Remove all the axis clutter, leaving just the curve.
ax.set_axis_off()

plt.show()
```

### Homework

Play with the parameters of the Lorentz system in the code cell above and verify the following facts

* If $\rho <1$, then there is only one equilibrium point, which is at the origin. All orbits converge to the origin, which is a global attractor.

* A pitchfork bifurcation occurs at $\rho = 1$, and for $\rho > 1$, two additional critical points appear at $\left( \sqrt{\beta(\rho-1)}, \sqrt{\beta(\rho-1)}, \rho-1 \right)$ and $\left( -\sqrt{\beta(\rho-1)}, -\sqrt{\beta(\rho-1)}, \rho-1 \right)$
These correspond to steady convection. This pair of equilibrium points is stable only if the following criterion is met

\begin{equation*}
\rho < \sigma\frac{\sigma+\beta+3}{\sigma-\beta-1},\;\sigma > \beta+1
\end{equation*}

* At the critical value, both equilibrium points lose stability through a subcritical Hopf bifurcation.

As a solved example, we can see the onset of chaos in the Lorenz system by the sensitive dependence on initial conditions. The code cell below plots the same Lorenz system for two initial conditions that are only different by $2$ in $x$. The two trajectories are plotted in different colours. Initially, both trajectories nearly overlap. However, after some time, the two trajectories are easily distinguishable by two different colours that can be found in vastly separated regions.

```python
import numpy as np
from scipy.integrate import solve_ivp
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D

WIDTH, HEIGHT, DPI = 1000, 750, 100

# Lorenz paramters and initial conditions, apart from the one we're iterating.
sigma, beta, rho = 10, 2.667, 28
v0, w0 = 1, 1

# Maximum time point and total number of time points.
tmax, n = 100, 10000

fig = plt.figure(facecolor='k', figsize=(WIDTH/DPI, HEIGHT/DPI))
ax = fig.gca(projection='3d')
ax.set_facecolor('k')
fig.subplots_adjust(left=0, right=1, bottom=0, top=1)

us = [1, 2]
for i,u0 in enumerate(us):
  # Integrate the Lorenz equations.
  soln = solve_ivp(lorenz, (0, tmax), (u0, v0, w0), args=(sigma, beta, rho),
                   dense_output=True)
  # Interpolate solution onto the time grid, t.
  t = np.linspace(0, tmax, n)
  x, y, z = soln.sol(t)
  ax.plot(x,y,z, alpha=0.8)

# Remove all the axis clutter, leaving just the curve.
ax.set_axis_off()

plt.show()
```

### The Problem: The Henon-Heiles System

The Henon-Heiles system is a paradigmatic example of *Hamiltonian Chaos*, where chaos can occur in a completely conserved dynamical system. The system arises from the motion of stars around a galactic center, assuming the motion is restricted to the $xy$ plane.

The Hamiltonian is given for a two-dimensional dynamical system as

\begin{equation*}
H = \frac12 (p_x^2 + p_y^2) + \frac12(x^2 + y^2) +  \left(x^2y - \frac{y^3}{3}\right)
\end{equation*}

Thus, the ordinary differential equations that describe the dynamical evolution can be obtained from hamilton's equations of motion, yielding

\begin{align*}
\dot{x} &= p_x, \\
\dot{p}_x &= -x - 2 xy, \\
\dot{y} &= p_y, \\
\dot{p}_y &= -y - \left(x^2 - y^2\right).
\end{align*}

Let us define $u = [u_0, u_1, u_2, u_3] = [x,p_x,y,p_y]$. The only parameter of interest is the total energy $E$, for which a large number of initial conditions can be chosen, all of which satisfy
\begin{equation*}
E = \frac12 (u^2_1 + u^2_3) + \frac12(u^2_0 + u^2_2) +  \left(u^2_0u_2 - \frac{u^3_2}{3}\right)
\end{equation*}

While the dynamics can be readily simulated with the tools at our disposal, it cannot be plotted completely, as the phase space is $4-$dimensional. However, we can look at *Poincare Sections*, $2-$d cross-sections of the phase space obtained by strobing the trajectories at the times when they pass through the surface. Let us look for values of $(y, p_y)$, or $(u_2, u_3)$ whenever $u_0=0$ and $u_1>0$ during the evolution of every trajectory. Note that, since the energy is always supposed to be conserved, fixing $u_2, u_3$ and $u_0$ automatically yields only two solutions for $u_1$ from the energy equation above, and we can choose either one. We have chosen the positive ones, given by the following equation on the $u_0=0$ hypersurface.
\begin{equation*}
u_1 = \sqrt{2E - u^2_2\left(1-\frac{2}{3}u_2\right)-u^2_3}.
\end{equation*}

The code cell below plots Poincare Sections for trajectories for any energy. The following energies are suggested.

| E =	| 1/50 	| 1/23 	| 1/15 	| 1/8 	| 1/6 	|
|---	|---	|---	|---	|---	|---	|

For each energy, a numpy mgrid of initial conditions are created and the dynamics evolved using the 'solve_ivp' routine, and the only times of interest are when $u_0=0$, which is encapsulated into an event function. The highest energies will cause transitions to chaos.

```python
%%time
from scipy.integrate import solve_ivp
import matplotlib.pyplot as plt
plt.rcParams['figure.figsize'] = (8,6)
plt.rcParams['font.size'] = 20

def henon_system(t, u): # right-hand side of a system of ODEs
    x = u[0]
    px = u[1]
    y  = u[2]
    py = u[3]
    return np.array([px, -x-2*x*y, py, -y-x**2+y**2])

def poincare(f, X0, epoch_n, h=0.001, N=10):
    t = h*N*epoch_n
    X = []
    for state0 in X0:
        u1, u2, u3 = state0
        sol = solve_ivp(f, [0, t], [0, u1, u2, u3], max_step=0.1, events=lambda t,u:u[0])
        X.append(sol.y_events[0])
    return X

def give_ics(En, min, max,npts):
  """
  Returns a grid of initial conditions, recast as an iterable object 'ics'
  """
  u2m, u2mx = min, max
  u3m, u3mx = min, max
  U2, U3 = np.meshgrid(np.linspace(u2m, u2mx, npts), np.linspace(u3m, u3mx, npts))
  u2, u3 = U2.ravel(), U3.ravel()
  u1 = np.sqrt(2 * En - u2**2  * (1 - (2/3) * u2) - u3**2)
  U1 = u1.reshape(U2.shape)
  ics = np.vstack([U1.ravel(), U2.ravel(), U3.ravel()]).T
  ics = ics[~np.isnan(ics).any(axis=1)] # Remove all points that lie outside the energy hypersurface
  return ics

if __name__ == "__main__":
  energy = 1.0/50.0
  X0_henon = give_ics(energy, -1.5, 0.0, 50)
  data = poincare(henon_system, X0_henon, epoch_n=10000, h=0.1, N=1)
  data =np.array(np.concatenate(data).ravel())
  u2 = data[2::4]
  u3 = data[3::4]
  plt.title(f"E = {energy:1.2f}")
  plt.scatter(u2, u3, s=1)
  plt.show()
```

```python
%%time
import matplotlib.pyplot as plt
plt.rcParams['figure.figsize'] = (8,6)
plt.rcParams['font.size'] = 20

energy = 1.0/23.0
X0_henon = give_ics(energy, -1.5, 0.0, 50)
data = poincare(henon_system, X0_henon, epoch_n=10000, h=0.1, N=1)
data = np.array(np.concatenate(data).ravel())
u2 = data[2::4]
u3 = data[3::4]
plt.title(f"E = {energy:1.2f}")
plt.scatter(u2, u3, s=1)
plt.show()
```

```python
%%time
import matplotlib.pyplot as plt
plt.rcParams['figure.figsize'] = (8,6)
plt.rcParams['font.size'] = 20

energy = 1.0/15.0
X0_henon = give_ics(energy, -1.5, 0.0, 75)
data = poincare(henon_system, X0_henon, epoch_n=1200, h=0.1, N=1)
data = np.array(np.concatenate(data).ravel())
u2 = data[2::4]
u3 = data[3::4]
plt.title(f"E = {energy:1.2f}")
plt.scatter(u2, u3, s=1)
plt.show()
```

```python
%%time
import matplotlib.pyplot as plt
plt.rcParams['figure.figsize'] = (8,6)
plt.rcParams['font.size'] = 20

energy = 1.0/8.0
X0_henon = give_ics(energy, -1.5, 0.0, 75)
data = poincare(henon_system, X0_henon, epoch_n=1200, h=0.1, N=1)
data = np.array(np.concatenate(data).ravel())
u2 = data[2::4]
u3 = data[3::4]
plt.title(f"E = {energy:1.2f}")
plt.scatter(u2, u3, s=1)
plt.show()
```

```python

```


This notebook follows **Chapter 8, "Optimization and Minimum Principles,"** from [**"Computational Science and Engineering" by Gilbert Strang.**](https://www.amazon.ca/Computational-Science-Engineering-Gilbert-Strang/dp/0961408812)

# 8.1 Least Squares
Consider a matrix A of size $m x n$ with $n$ independent columns, and $m > n$.
$A^T A$ is symmetric positive definite, as Rank(A) = $n$.
We wish to solve $Au = b$ for some vector $b$ with $m$ components. The solution $\hat{u}$ has $n$ components.

This gives rise to the **least squares problem**:
$$ \min \parallel Au - b \parallel ^2 $$

Below are the **normal equations**, which give the solution $\hat{u}$:
$$
\begin{align}
Au &= b \nonumber \\
A^T A \hat{u} &= A^T b
\end{align}
$$

The residual of the cost function is $e = b - Au$, and has dimensions $m \times 1$. Thus,
$$ A^T e = A^T b - A^T A u $$
Therefore, $A^T e = 0$ when $u = \hat{u}$.

$e$ is in the nullspace of $A^T$, and is therefore perpendicular to the columns (column space) of $A$ (equivalently, $e$ is perpendicular to the rows/rowspace of $A^T$). Recall that orthogonality means that the dot product of two vectors is 0.

## 8.1.1 Refresher on Orthogonality
Before we start on the next section, let's review some concepts from linear algebra. Specifically, the orthogonality of vectors and vector spaces.

### Orthogonal Vectors
There is an analogue between geometry in Euclidean space and in $\mathcal{R}^n$.

First, *two vectors $\mathbf{u}$ and $\mathbf{v}$ are perpendicular i.f.f. the distance between $\mathbf{u}$ and $\mathbf{-v}$ is the same as the distance between $\mathbf{u}$ and $\mathbf{+v}$.*

First, the distance between $\mathbf{u}$ and $\mathbf{-v}$:
$$
\begin{align}
[\textrm{dist}(\mathbf{u}, \mathbf{-v})]^2 &= \parallel \mathbf{u}-(-\mathbf{v}) \parallel ^2 = \parallel \mathbf{u} +\mathbf{v} \parallel ^2 \\
&= (\mathbf{u} + \mathbf{v}) \cdot (\mathbf{u} + \mathbf{v}) \\
&= \parallel \mathbf{u} \parallel^2 + \parallel \mathbf{v} \parallel^2 + 2\mathbf{u} \cdot \mathbf{v}
\end{align}
$$

Next, the distance between $\mathbf{u}$ and $\mathbf{+v}$:
$$
\begin{align}
[\textrm{dist}(\mathbf{u}, \mathbf{v})]^2 &= \parallel \mathbf{u} -\mathbf{v} \parallel ^2 \\
&= (\mathbf{u} - \mathbf{v}) \cdot (\mathbf{u} - \mathbf{v}) \\
&= \parallel \mathbf{u} \parallel^2 + \parallel \mathbf{v} \parallel^2 - 2\mathbf{u} \cdot \mathbf{v}
\end{align}
$$

Therefore,
$$
\begin{align}
[\textrm{dist}(\mathbf{u}, \mathbf{-v})]^2 = [\textrm{dist}(\mathbf{u}, \mathbf{v})]^2 &\iff - 2\mathbf{u} \cdot \mathbf{v} = 2\mathbf{u} \cdot \mathbf{v} \\
&\implies \quad \mathbf{u} \cdot \mathbf{v} = 0 \\
&\implies \parallel \mathbf{u} + \mathbf{v} \parallel ^2 = \parallel \mathbf{u} \parallel^2 + \parallel \mathbf{v} \parallel^2
\end{align}
$$

## 8.1.2 Orthogonal Complements in $\mathcal{R}^n$
If a vector $\mathbf{z}$ is orthogonal to every vector in a subspace $W$ of $\mathcal{R}^n$, then $\mathbf{z}$ is orthogonal to $W$. The set of all such vectors $\mathbf{z}$ is the **orthogonal complement** of $W$ and denoted by $W^\bot$.

Let's go through the relation between nullspace, orthogonal complements, and the columns and rows of a $m \times n$ matrix $A$. We have the folllowing theorem:
$$
(\textrm{Row}(A))^\bot = \textrm{Nul}(A) \quad \textrm{and} \quad (\textrm{Col}(A))^\bot = \textrm{Nul}(A)^\bot
$$

If $Ax = 0$ for some $n \times 1$ vector $x$, then $x \in \textrm{Nul}(A)$, which means $x$ is orthogonal to the rows of $A$. The rows of $A$ span its row space, so $x$ is orthogonal to Row($A$).

**Explanation**
$$
\begin{align}
A &= \begin{pmatrix} a & b & c \\ d & e & f \end{pmatrix}, \quad u = \begin{vmatrix} x & y & z \end{vmatrix}. \\
Au &= 0 \\
&= \begin{pmatrix} ax + by + cz \\ dx + ey + fz \end{pmatrix} = \begin{pmatrix} 0 \\ 0 \end{pmatrix}
\end{align}
$$
The elements of the product $Au$ are dot products between $u$ and the row vectors of $A$. Since $Au = 0$, $u$ is in the nullspace of A, and from the above, we see that $u$ is orthogonal to the rows of $A$.

## 8.1.3 Primal and Dual Problems
In the *primal* problem, we find the solution $\hat{u}$ by projecting $b$ onto the column space of $A$, i.e. $A\hat{u}$ is the closest point to $b$ in the column space of $A$. In general, the product $Au$ yields a linear combination of the columns of $A$. The column space of $A$ contains all such products.

$$ Au = y \quad \implies \quad y \in \textrm{Col}(A), \quad \forall \quad u $$
Where, Col($A$) refers to the column space of $A$.

The *dual* problem solves $A^T e = 0$ for $e$, which is the projection of $b$ onto the nullspace of $A^T$. Recall from above:
$$ \textrm{Nul}(A^T) = (\textrm{Row}(A^T))^\bot = (\textrm{Col}(A))^\bot $$
Thus, $e$ is contained in (Col($A$))$^\bot$, which contains *all vectors perpendicular to the columns of $A$*. The dimension of (Col($A$))$^\bot$ is given by the Rank-Nullity Theorem:
$$
\begin{align}
&\textrm{Rank}(A) + \textrm{Nullity}(A) = n, \quad \textrm{ where } n \textrm{ is the number of columns in } A. \\
&\textrm{Rank}(A^T) + \textrm{Nullity}(A^T) = m \\
&\textrm{Rank}(A) = \textrm{Rank}(A^T) \quad \implies \quad \textrm{Nullity}(A^T) = m - n \\
&\textrm{Nullity}(A^T) = \textrm{dim(Row(}A^T))^\bot = \textrm{dim(Col(}A)^\bot) = m - n
\end{align}
$$

Ordinary least squares

$e$ and $\hat{u}$ solve the two linear equations shown by the figure above:
$$
\begin{align}
e + A \hat{u} &= b \rightarrow m \textrm{ equations} \\
A^T e &= 0 \rightarrow n \textrm{ equations}
\end{align}
$$

These are called the "Primal-Dual", "Saddle Point", and "Kuhn-Tucker (KKT)" equations. We discussed the meaning of "Primal" and "Dual" above, but let's go over the other names. The third comes from somebody who worked on these equations.

## 8.1.4 "Saddle Point" Equations
The Saddle Point, or KKT, matrix is a block matrix that represents the Primal-Dual equations:
$$ S = \begin{bmatrix} I & A \\ A^T & 0 \end{bmatrix} \quad \implies \quad S \begin{bmatrix} e \\ \hat{u} \end{bmatrix} = \begin{bmatrix} Ie + A\hat{u} \\ A^T e + 0 \end{bmatrix} $$
Where, $I$ is an $m \times m$ identity matrix. Thus, $I$ constitutes the first $m$ pivot columns of $S$. Thus, to reduce $S$, we need to make the lower left hand corner of $S$ a $n \times n$ matrix of zeroes. To do this, let's review elimination on block matrices:

**Elimination of block matrices**
$$
\begin{align}
\begin{bmatrix} I & 0 \\ -CA^{-1} & I \end{bmatrix} \begin{bmatrix} A & B \\ C & D \end{bmatrix} &= \begin{bmatrix} A + 0 & B + 0 \\ -CA^{-1}A + C & -CA^{-1}B + D \end{bmatrix} \\
&= \begin{bmatrix} A & B \\ 0 & D - CA^{-1}B \end{bmatrix}
\end{align}
$$

The first matrix, $\begin{bmatrix} I & 0 \\ -CA^{-1} & I \end{bmatrix}$, is also known as the Gaussian elimination matrix $E$. If we also wanted to reduce $A$ into a matrix of pivots:
$$
\begin{align}
\begin{bmatrix} A^{-1} & 0 \\ -CA^{-1} & I \end{bmatrix} \begin{bmatrix} A & B \\ C & D \end{bmatrix} &= \begin{bmatrix} AA^{-1} + 0 & BA^{-1} + 0 \\ -CA^{-1}A + C & -CA^{-1}B + D \end{bmatrix} \\
&= \begin{bmatrix} I & BA^{-1} \\ 0 & D - CA^{-1}B \end{bmatrix}
\end{align}
$$

We now reduce our Primal-Dual matrix $S$ with $E = \begin{bmatrix} I & 0 \\ -A^T & I \end{bmatrix}$:
$$
\begin{align}
ES &= \begin{bmatrix} I & 0 \\ -A^T & I \end{bmatrix} \begin{bmatrix} I & A \\ A^T & 0 \end{bmatrix} \\
&= \begin{bmatrix} I + 0 & A + 0 \\ -A^T + A^T & -AA^T + 0 \end{bmatrix} \\
&= \begin{bmatrix} I & A \\ 0 & -AA^T \end{bmatrix}
\end{align}
$$

By convention, the lower right-hand block remaining after elimination is known as the **Schur complement**. More importantly, the first $m$ pivots are positive (=1), but **the final $n$ pivots, determined by $-AA^T$ are negative.** When a matrix has pivots of both signs, it is known as **indefinite**. For such matrices, there is **no maximum or minimum**, positive or negative definite. Instead, there is a **saddle point**, $(\hat{u}, e)$.

## 8.1.5 "Primal-Dual" Equations
The primal problem is to minimize $\frac{1}{2} \parallel Au - b \parallel^2$.

The dual problem is to minimize $\frac{1}{2} \parallel e - b \parallel^2$ with the constraint $A^T e = 0$. Here, $u$ acts a Lagrange multiplier to enforce the constraint $A^T e = 0$.

Solutions to the primal and dual problems are found simultaneously, and add to $b$.

Let's do an example!

Let
$$
A = \begin{bmatrix}2 \\ 1 \end{bmatrix} \quad b = \begin{bmatrix}3 \\ 4\end{bmatrix} \quad A^T A = 5 \quad A^T b = 10 \\

\begin{align}
&\textrm{Projection: } A\hat{u} = \begin{bmatrix}4 \\ 2\end{bmatrix} \\
&\textrm{Error: } e = b - A\hat{u} = \begin{bmatrix}3 \\ 4 \end{bmatrix} - \begin{bmatrix}4 \\ 2\end{bmatrix} = \begin{bmatrix}-1 \\ 2\end{bmatrix}
\end{align}

$$

Let's verify that $A\hat{u}$ and $e$ are perpendicular. First, $A\hat{u}$ and $e$ sum to $b$:
$$
A\hat{u} + e = A\hat{u} + (b - A\hat{u}) = b \\

\begin{align}
\parallel b \parallel^2 &= \parallel A\hat{u} \parallel^2 + \parallel e \parallel^2 \\
&= 20 + 5 = 25 \\
A \hat{u} \cdot e &= \parallel A \hat{u} \parallel \parallel e \parallel \cos \theta \\
0 &= \sqrt{20} \sqrt{5} \cos \theta \\
\therefore \cos \theta &= 0 \quad \implies \quad \theta = \frac{\pi}{2}
\end{align}
$$

**Primal**

The normal equation is: $A^T A \hat{u} = A^T b$, so $\hat{u} = 2$.

**Dual**

The dual minimizes $\frac{1}{2}\parallel e-b \parallel^2$ under the constraint $A^T e = 0$. The solution to the dual problem gives $e$ and $\hat{u}$:

The Lagrange function is:
$$
\begin{align}
L &= \frac{1}{2}\parallel e-b \parallel^2 + u(A^T e) \\
L(e_1, e_2, u) &= \frac{1}{2}(e_1 - b_1 )^2 + \frac{1}{2}(e_2 - b_2 )^2 + u(A^{T}_{1} e_1 + A^{T}_{2} e_2) \\

\frac{\partial L}{\partial (e_1, e_2, u)} &=
\begin{bmatrix}
    e_1 - b_1 + 2\hat{u} = 0 \\
    e_2 - b_2 + \hat{u} = 0 \\
    2e_1 + e_2 = 0
\end{bmatrix} \\
&= \begin{bmatrix}
        e_1 & e_2 & \hat{u} \\
        1 & 0 & 2 \\
        0 & 1 & 1 \\
        2 & 1 & 0
    \end{bmatrix} \begin{bmatrix}
        e_1 \\ e_2 \\ \hat{u}
    \end{bmatrix} =
    \begin{bmatrix} b_1 \\ b_2 \\ 0 \end{bmatrix}
\end{align}
$$

The matrix $
    \begin{bmatrix}
        e_1 & e_2 & \hat{u} \\
        1 & 0 & 2 \\
        0 & 1 & 1 \\
        2 & 1 & 0
    \end{bmatrix} $
is the saddle point matrix $S$.

```python

```