## What is Ensemble?

A supervised *meta*-learning algorithm:
1. It uses multiple learners (also called estimators)
1. Combines them to produce *better* results

## How do they work?

We're going to say that F is a learner if, for given a training set, $ X = {(x_i, y_i) \in \mathbb{R}^n \times \mathbb{R}} $, F gives us an estimation, $f$, of the underlying distribution, $P$.

In other words, $f:\mathbb{R}^n \rightarrow \mathbb{R}$ is our *estimator* or *model*, trained on $X$ using the learner $F$. Some examples of learners:
* Linear Regression: takes in $(x_i, y_i)$ pairs and outputs a estimator in the form $f(x) = \beta_0 + \beta_1\cdot x$. The *learning* consists in attempting to find $\beta$ such that the $L_2$ error norm is minimized over $X$.
* Multilayer Perceptron: takes in $(x_i, y_i)$ pairs and, for a given architecture of a network, $f$, tries to minimize some loss function over our training set.

Suppose you have several learners:
\begin{equation}
F_1, F_2, \ldots, F_k
\end{equation}

### How could you combine them to get something better?

There are essentially 3 ways in which to do this.
1. Bagging: average them to reduce variance
1. Boosting: exploit inter-dependence
1. Stacking: combine them using another learner

## 1. Bagging

Bagging is short for **Bootstrap Aggregation:**

**Bootstrap method** is a statistical technique for estimating quantities about a population by averaging estimates from multiple small data samples

1. Repeat the following m times:
    * Bootstrap your training set (i.e. take a sub-sample with replacement)
    * Train one of your learners on this sample, producing $f_i$
2. Combine all of your models into a single model:
\begin{equation}
f(x) = \frac{1}{m}\sum_{i = 1}^m {f_i(x)}
\end{equation}

Bagging can be used with any base learners.
* It reduces variance
* Turns unstable learners (such as decision trees) into stable learners
* It does not help as much when the base learners are already stable

One example of a famous bagging technique is Random Forest. Let's see how it works:
1. Take as learner a Decision Tree - e.g. ID3.
1. Repeat M times:
    * Generate a bagging sample, while also sampling the features
    * Train ID3 on it, producing one decision tree
1. The resulting model is the average of all the decision trees (for classification, you would use **voting**)

```python
from sklearn.exceptions import ConvergenceWarning
%matplotlib inline

import itertools
import pydotplus  # you can install pydotplus with: pip install pydotplus
from IPython.display import Image
from sklearn.metrics import roc_auc_score
import matplotlib.gridspec as gridspec
import matplotlib.pyplot as plt
import numpy as np
from mlxtend.classifier import StackingClassifier
from mlxtend.plotting import plot_decision_regions
from sklearn import datasets
from sklearn.ensemble import AdaBoostClassifier
from sklearn.ensemble import BaggingClassifier
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.naive_bayes import GaussianNB
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score, train_test_split
from mlxtend.plotting import plot_learning_curves
from mlxtend.plotting import plot_decision_regions
from sklearn.tree import DecisionTreeClassifier, export_graphviz
import pandas as pd

from sklearn.svm import SVC, LinearSVC
from sklearn.naive_bayes import GaussianNB
from sklearn.linear_model import LogisticRegression
from sklearn.neural_network import MLPClassifier
from sklearn.kernel_approximation import Nystroem
from sklearn.kernel_approximation import RBFSampler
from sklearn.pipeline import make_pipeline

# Graphviz path for Windows
import os
os.environ["PATH"] += os.pathsep + 'C:/Program Files (x86)/Graphviz2.38/bin/'

np.random.seed(0)

# Import data
df = pd.read_csv('..\data\input.csv.gz')

import warnings
warnings.filterwarnings("ignore", category=FutureWarning)
warnings.filterwarnings("ignore", category=ConvergenceWarning)
```

```python
iris = datasets.load_iris()
X, y = iris.data[:, 0:2], iris.target

tree_clf = DecisionTreeClassifier(criterion='entropy', max_depth=1) # an optimised version of the CART
knn_clf = KNeighborsClassifier(n_neighbors=1)

tree_bagging_clf = BaggingClassifier(base_estimator=tree_clf, n_estimators=10, max_samples=0.8, max_features=0.8)
knn_bagging_cls = BaggingClassifier(base_estimator=knn_clf, n_estimators=10, max_samples=0.8, max_features=0.8)
label = ['Decision Tree', 'K-NN', 'Bagging Tree', 'Bagging K-NN']
clf_list = [tree_clf, knn_clf, tree_bagging_clf, knn_bagging_cls]

fig = plt.figure(figsize=(10, 8))
gs = gridspec.GridSpec(2, 2)
grid = itertools.product([0,1],repeat=2)

for clf, label, grd in zip(clf_list, label, grid):
    scores = cross_val_score(clf, X, y, cv=3, scoring='accuracy')
    print("Accuracy: %.2f (+/- %.2f) [%s]" %(scores.mean(), scores.std(), label))

    clf.fit(X, y)
    ax = plt.subplot(gs[grd[0], grd[1]])
    fig = plot_decision_regions(X=X, y=y, clf=clf, legend=2)
    plt.title(label)

plt.show()
```

The figure above shows the decision boundary of a decision tree and k-NN classifiers along with their bagging ensembles applied to the Iris dataset. The decision tree shows axes parallel boundaries while the $k=1$ nearest neighbors fits closely to the data points. The bagging ensembles were trained using $10$ base estimators with $0.8$ subsampling of training data and $0.8$ subsampling of features. The decision tree bagging ensemble achieved higher accuracy in comparison to k-NN bagging ensemble because k-NN are less sensitive to perturbation on training samples and therefore they are called *stable learners*. Combining stable learners is less advantageous since the ensemble will not help improve generalization performance.

## 2. Boosting

**Boosting** refers to a family of algorithms which converts weak learner to strong learners.

The idea of boosting is to train weak learners, that have high bias, sequentially, each trying to correct its predecessor.

#### Boossting for binary class and regression
Train learners one on top of the other:
1. Learn using the existing training set, where we weigh each sample identically
1. Repeat m times:
    * Weigh samples based on the current error so far
    * Apply a new learner to the weighted train set
1. Output as model the sum of all trained models

#### Boosting for multi-class
The main flow of the algorithm is similar to the binary case, but a measure of the joint training error shall be defined in advance.

During each iteration the algorithm chooses a classifier of a single feature (features that can be shared by more categories shall be encouraged). This can be done via converting multi-class classification into a binary one (a set of categories versus the rest), or by introducing a penalty error from the categories that do not have the feature of the classifier.

Boosting helps reduce bias, but increases variance:
* It works well with stable learners
* It converts weak learners into strong learners
* Idea is that the family of weak learners should have a minimum correlation between them

The main differences between boosting and bagging:
1. Bagging trains in parallel. Boosting trains sequentially
1. Boosting exploits learner inter-dependence. Bagging exploits learner independence.

Examples of algorithms: AdaBoost, Gradient Boosting, XGBoost

```python
# We'll use AdaBoost here
iris = datasets.load_iris()
X, y = iris.data[:, 0:2], iris.target
np.random.seed(0)

clf = DecisionTreeClassifier(criterion='entropy', max_depth=1)

num_est = [1, 2, 3, 16]
label = ['AdaBoost (n_est=1)', 'AdaBoost (n_est=2)', 'AdaBoost (n_est=3)', 'AdaBoost (n_est=10)']
fig = plt.figure(figsize=(10, 8))
gs = gridspec.GridSpec(2, 2)
grid = itertools.product([0,1],repeat=2)

for n_est, label, grd in zip(num_est, label, grid):
    boosting = AdaBoostClassifier(base_estimator=clf, n_estimators=n_est)
    boosting.fit(X, y)

    scores = cross_val_score(boosting, X, y, cv=3, scoring='accuracy')
    print("Accuracy: %.2f (+/- %.2f) [%s]" %(scores.mean(), scores.std(), label))

    ax = plt.subplot(gs[grd[0], grd[1]])
    fig = plot_decision_regions(X=X, y=y, clf=boosting, legend=2)
    plt.title(label)

plt.show()
```

The AdaBoost algorithm is illustrated in the figure above. Each **base learner** consists of a **decision tree** with depth __$1$__, thus classifying the data based on a feature threshold that partitions the space into two regions separated by a linear decision surface that is parallel to one of the axes.

```python
#Ensemble Size
num_est = list(map(int, np.linspace(1,100,20)))
bg_clf_cv_mean = []
bg_clf_cv_std = []
for n_est in num_est:
    ada_clf = AdaBoostClassifier(base_estimator=clf, n_estimators=n_est)
    scores = cross_val_score(ada_clf, X, y, cv=3, scoring='accuracy')
    bg_clf_cv_mean.append(scores.mean())
    bg_clf_cv_std.append(scores.std())

plt.figure()
(_, caps, _) = plt.errorbar(num_est, bg_clf_cv_mean, yerr=bg_clf_cv_std, c='blue', fmt='-o', capsize=5)
for cap in caps:
    cap.set_markeredgewidth(1)
plt.ylabel('Accuracy'); plt.xlabel('Ensemble Size'); plt.title('AdaBoost Ensemble');
plt.show()
```

The figure above shows how the test accuracy improves with the size of the ensemble.

## 3. Stacking

__Stacking__ is a Meta Modeling Technique introduced by [Wolpert in 1992](http://www.machine-learning.martinsewell.com/ensembles/stacking/Wolpert1992.pdf).

Combines the two methods, using two levels:
1. First, train your base learners similarly to bagging, obtaining $f_1, f_2, \ldots, f_m$
1. Enrich your data with the predictions made by these base learners:
\begin{equation}
    x = (x^1,\ldots,x^d) \rightarrow (x^1,\ldots,x^d, f_1(x), f_2(x), \ldots, f_m(x))
\end{equation}
1. Apply another learner on top of the extended data

It's important to note:
1. The final learner will be able to compute the residuals (errors) for each base learner.
1. It might decide to average several base learners for a prediction (bagging), or offset their prediction for several examples (boosting)

The input **X**  is either the original data or the previous layer’s output;

**Tr(j)** represents preprocessing pipelines that transform the input to its associated base learners **f(i)**.

The __Ft__ operation propagates specified features **s** from input to output.

Base learner predictions __pj(i)__ are concatenated to propagated features __X:,s__ to form the output matrix __P__

```python
iris = datasets.load_iris()
X, y = iris.data[:, 1:3], iris.target

clf1 = KNeighborsClassifier(n_neighbors=1)
clf2 = RandomForestClassifier(random_state=1)
clf3 = GaussianNB()
lr = LogisticRegression()
sclf = StackingClassifier(classifiers=[clf1, clf2, clf3],
                          meta_classifier=lr)
```

```python
label = ['KNN', 'Random Forest', 'Naive Bayes', 'Stacking Classifier']
clf_list = [clf1, clf2, clf3, sclf]

fig = plt.figure(figsize=(10,8))
gs = gridspec.GridSpec(2, 2)
grid = itertools.product([0,1],repeat=2)

clf_cv_mean = []
clf_cv_std = []
for clf, label, grd in zip(clf_list, label, grid):

    scores = cross_val_score(clf, X, y, cv=3, scoring='accuracy')
    print("Accuracy: %.2f (+/- %.2f) [%s]" %(scores.mean(), scores.std(), label))
    clf_cv_mean.append(scores.mean())
    clf_cv_std.append(scores.std())

    clf.fit(X, y)
    ax = plt.subplot(gs[grd[0], grd[1]])
    fig = plot_decision_regions(X=X, y=y, clf=clf)
    plt.title(label)

plt.show()
```

## SuperLearner

There is an issue with all the algorithms presented thus far:
* All model errors get evaluated on the same data the model is trained
* This can be corrected somewhat through sub-sampling
* But sub-sampling reduces training set size, which means you loose on efficiency

The idea for SuperLearner was initially hinted at in van der Laan and Dudoit's 2003 paper and developed in van der Laan's 2007 paper (https://pdfs.semanticscholar.org/4e81/f4c95dde9a327026c584b91a3fe691595d1f.pdf). Essentially, it states that **cross-fold validation** gives a good estimate for the error, so let's use that.

1. We define the CrossFold Validation operator $CV_F$:
    * Do k-fold on X: $K = \{ (X_T^i, X_V^i)| X_T^i \cup X_V^i = X, X_T^i\cap X_V^i = 0 \}$ such that $\cup_i X_V^i = X$ and $X_V^i \cap X_V^j = 0$
    * Apply $F_i$ on $X_T^i$
    * Apply $f_i$ on $X_V^i$: $CV_F(x) = f_i(x)$ where $x\in X_V^i$
1. We enrich our data using the CV operator:
\begin{equation}
x = (x^1,\ldots,x^d) \rightarrow (x^1,\ldots,x^d, CV_{F_1}(x), CV_{F_2}(x), \ldots, CV_{F_m}(x))
\end{equation}
1. We train on top of this another model, also known as the meta-model, $mf$.
1. Apply each base learner on $X$ and produce $f_1, \ldots, f_m$.
1. Output the model
\begin{equation}
f(x) = mf(x, f_1(x), \ldots, f_m(x))
\end{equation}

#### Van der Laan **shows** that SuperLearner behaves **at worst** as the **best** base learner.

## Why should I use it?

**You need to use cross-fold validation anyway!!**
* by partitioning the dataset, they scale better
* Any ML pipeline should contain a CV fold stage that searches for the *best* learner
* You're already supposed to spend the resources on CV folding
* SuperLearner will simply squeeze that extra performance from the base learners
* by creating subset-specific base learners, they can harness local information better, possibly leading to a richer prediction input set for the meta learner.

## Following on an example: Political Donation Data

Let's follow through our proposed examples on a larger dataset.

Note that we split the data into train and test.

```python
# Always good to set a seed for reproducibility
SEED = 222
np.random.seed(SEED)

def get_train_test(test_size=0.95):
    """Split Data into train and test sets."""
    y = 1 * (df.cand_pty_affiliation == "REP")
    X = df.drop(["cand_pty_affiliation"], axis=1)
    X = pd.get_dummies(X, sparse=True)
    X.drop(X.columns[X.std() == 0], axis=1, inplace=True)
    return train_test_split(X, y, test_size=test_size, random_state=SEED)

xtrain, xtest, ytrain, ytest = get_train_test()

# A look at the data
print("\nExample data:")
df.head()
```

    Example data:

      cand_pty_affiliation
      cand_office_st
      cand_office
      cand_status
      rpt_tp
      transaction_tp
      entity_tp
      state
      classification
      cycle
      transaction_amt

      0
      REP
      US
      P
      C
      Q3
      15
      IND
      NY
      Engineer
      2016.0
      500.0

      1
      DEM
      US
      P
      C
      M5
      15E
      IND
      OR
      Math-Stat
      2016.0
      50.0

      2
      DEM
      US
      P
      C
      M3
      15
      IND
      TX
      Scientist
      2008.0
      250.0

      3
      DEM
      US
      P
      C
      Q2
      15E
      IND
      IN
      Math-Stat
      2016.0
      250.0

      4
      REP
      US
      P
      C
      12G
      15
      IND
      MA
      Engineer
      2016.0
      184.0

```python
df.cand_pty_affiliation.value_counts(normalize=True).plot(
    kind="bar", title="Share of No. donations")
plt.show()
```

# Play with it

We can try different parameters for it and see if it performs better.

Let's fit a decision tree with a single node (decision rule) on our training data and see how it perform on the test set:

```python
t1 = DecisionTreeClassifier(max_depth=1, random_state=SEED)
t1.fit(xtrain, ytrain)
p = t1.predict_proba(xtest)[:, 1]

print("Decision tree ROC-AUC score: %.3f" % roc_auc_score(ytest, p))
print_graph(t1, xtrain.columns)
```

Each of the two leaves register their share of training samples, the class distribution within their share, and the class label prediction. Our decision tree bases its prediction on whether the the size of the contribution is above #101.5: but it makes the same prediction regardless! This is not too surprising given that __75% of all donations are to Democrats__. But it's not making use of the data we have. Let's use __three levels__ of decision rules and see what we can get:

```python
t2 = DecisionTreeClassifier(max_depth=3, random_state=SEED)
t2.fit(xtrain, ytrain)
p = t2.predict_proba(xtest)[:, 1]

print("Decision tree ROC-AUC score: %.3f" % roc_auc_score(ytest, p))
def print_graph(clf, feature_names):
    """Print decision tree."""
    graph = export_graphviz(
        clf,
        label="root",
        proportion=True,
        impurity=False,
        out_file=None,
        feature_names=feature_names,
        class_names={0: "D", 1: "R"},
        filled=True,
        rounded=True
    )
    graph = pydotplus.graph_from_dot_data(graph)
    return Image(graph.create_png())

print_graph(t2, xtrain.columns)
```

This model is not much better than the simple decision tree: a measly __5%__ of all donations are predicted to go to Republicans–far short of the __25% we would expect__.

A closer look tells us that the decision tree uses some __dubious splitting__ rules. A whopping 47.3% of all observations end up in the left-most leaf, while another 35.9% end up in the leaf second to the right. The vast majority of leaves are therefore __irrelevant__.

Making the model __deeper__ just causes it to __overfit__.

## What if we drop features?
A decision tree can be made more complex by increasing __"width"__, that is, creating several decision trees and combining them.

In other words, an ensemble of decision trees.

To see why such a model would help, consider how we may force a decision tree to investigate other patterns than those in the above tree. The simplest solution is to remove features that appear early in the tree. Suppose for instance that we remove the transaction amount feature (transaction_amt), the root of the tree. Our new decision tree would look like this:

Projecting the data to a subset of features might not change much in the overall performance.

```python
drop = ["transaction_amt"]

xtrain_slim = xtrain.drop(drop, 1)
xtest_slim = xtest.drop(drop, 1)

t3 = DecisionTreeClassifier(max_depth=3, random_state=SEED)
t3.fit(xtrain_slim, ytrain)
p = t3.predict_proba(xtest_slim)[:, 1]

print("Decision tree ROC-AUC score: %.3f" % roc_auc_score(ytest, p))
print_graph(t3, xtrain_slim.columns)
```

#### But if you look at the correlation of the two, you will see that their errors are not very well correlated.

```python
p1 = t2.predict_proba(xtest)[:, 1]
p2 = t3.predict_proba(xtest_slim)[:, 1]

pd.DataFrame({"full_data": p1,
              "red_data": p2}).corr()
```

The second tree has learned something different from the first, with the same performance overall.
The share of Republican donation __increased to 7.3%__.
We now have two models that by themselves have __similar__ predictive power, but operate on __different rules__. Because of this, they are likely to make different prediction errors, which we can average out with an __ensemble__.

#### This means that the learner is not very stable.

**A chance for bagging to shine!**

```python
p1 = t2.predict_proba(xtest)[:, 1]
p2 = t3.predict_proba(xtest_slim)[:, 1]
p = np.mean([p1, p2], axis=0)
print("Average of decision tree ROC-AUC score: %.3f" % roc_auc_score(ytest, p))
```

#### Take it all the way

Let's do a full-on Random Forest and see what happens.

```python
rf = RandomForestClassifier(
    n_estimators=100,
    max_features=5,
    random_state=SEED
)

rf.fit(xtrain, ytrain)
p = rf.predict_proba(xtest)[:, 1]
print("Average of decision tree ROC-AUC score: %.3f" % roc_auc_score(ytest, p))
```

## Errors Correlation

So we see that when errors don't correlate, we can use bagging to improve performance:
    * Different learners act well on different regions

Let's look at several learners and see how their errors correlate.

We first define a set of learners below.

```python
import time

def get_models():
    """Generate a library of base learners."""
    nb = GaussianNB()
    svc = SVC(C=100, probability=True)
    knn = KNeighborsClassifier(n_neighbors=3)
    lr = LogisticRegression(C=100, random_state=SEED, solver='lbfgs')
    nn = MLPClassifier((80, 10), early_stopping=False, random_state=SEED)
    gb = GradientBoostingClassifier(n_estimators=100, random_state=SEED) #regression trees
    rf = RandomForestClassifier(n_estimators=10, max_features=3, random_state=SEED)

    models = {'svm': svc,
              'knn': knn,
              'naive bayes': nb,
              'mlp-nn': nn,
              'random forest': rf,
              'gbm': gb,
              'logistic': lr,
              }

    return models

def train_predict(model_list):
    """Fit models in list on training set and return preds"""
    P = np.zeros((ytest.shape[0], len(model_list)))
    P = pd.DataFrame(P)

    print("Fitting models.")
    cols = list()
    for i, (name, m) in enumerate(models.items()):
        print("%s..." % name, end=" ", flush=False)
        _t0 = time.time()
        m.fit(xtrain, ytrain)
        dt = time.time()-_t0
        P.iloc[:, i] = m.predict_proba(xtest)[:, 1]
        cols.append(name)
        print("done in %.2f sec"%dt)

    P.columns = cols
    print("Done.\n")
    return P

def score_models(P, y):
    """Score model in prediction DF"""
    print("Scoring models.")
    for m in P.columns:
        score = roc_auc_score(y, P.loc[:, m])
        print("%-26s: %.3f" % (m, score))
    print("Done.\n")
```

#### We look at their individual performance

```python
models = get_models()
P = train_predict(models)
score_models(P, ytest)
```

That's our baseline. The Gradient Boosting Machine (GBM) does best, followed by a simple logistic regression. For our ensemble strategy to work, prediction errors must be relatively uncorrelated. Checking that this holds is our first order of business:
#### Let's see those correlations

```python
# You need ML-Ensemble for this figure: you can install it with: pip install mlens
from mlens.visualization import corrmat

corrmat(P.corr(), inflate=False)
plt.show()
```

Errors are significantly correlated, which is to be expected for models that perform well, since it's typically the outliers that are hard to get right. Yet most correlations are in the 50-80% span, so there is decent room for improvement.
#### How will the bagging ensemble work?

```python
print("Ensemble ROC-AUC score: %.3f" % roc_auc_score(ytest, P.mean(axis=1)))
```

#### Let's compare their ROC AUCs

```python
from sklearn.metrics import roc_curve

def plot_roc_curve(ytest, P_base_learners, P_ensemble, labels, ens_label):
    """Plot the roc curve for base learners and ensemble."""
    plt.figure(figsize=(10, 8))
    plt.plot([0, 1], [0, 1], 'k--')

    cm = [plt.cm.rainbow(i)
      for i in np.linspace(0, 1.0, P_base_learners.shape[1] + 1)]

    for i in range(P_base_learners.shape[1]):
        p = P_base_learners[:, i]
        fpr, tpr, _ = roc_curve(ytest, p)
        plt.plot(fpr, tpr, label=labels[i], c=cm[i + 1])

    fpr, tpr, _ = roc_curve(ytest, P_ensemble)
    plt.plot(fpr, tpr, label=ens_label, c=cm[0])

    plt.xlabel('False positive rate')
    plt.ylabel('True positive rate')
    plt.title('ROC curve')
    plt.legend(frameon=False)
    plt.show()

plot_roc_curve(ytest, P.values, P.mean(axis=1), list(P.columns), "ensemble")
```

```python
p = P.apply(lambda x: 1*(x >= 0.5).value_counts(normalize=True))
p.index = ["DEM", "REP"]
p.loc["REP", :].sort_values().plot(kind="bar")
plt.axhline(0.25, color="k", linewidth=0.5)
plt.text(0., 0.23, "True share republicans")
plt.show()
```

### Let's try to use Boosting
There's strength in diversity!

The only thing to consider is that the more models we add, the slower the ensemble will be.

#### Step 1: define a library of base learners

Here, we'll use our set of models from before

```python
base_learners = get_models()
```

#### Step 2: define a meta learner

Which meta learner to use is not obvious, but popular choices are linear models, kernel-based models (SVMs and KNNS) and decision tree based models.

But you could also use another ensemble as "meta learner": in this special case, you end up with a two-layer ensemble, akin to a feed-forward neural network.

To ensure the GBM explores local patterns, we are restricting each of the 1000 decision trees to train on a random subset of 4 base learners and 50% of input data. This way, the GBM will be exposed to each base learner's strength in different neighborhoods of the input space.

```python
meta_learner = GradientBoostingClassifier(
    n_estimators=1000,
    loss="exponential",
    max_features=4,
    max_depth=3,
    subsample=0.5,
    learning_rate=0.005,
    random_state=SEED
)
```

#### Step 3: define a procedure for generating train and test sets
To keep things simple, we split the full training set into a training and prediction set of the base learners. This method is sometimes referred to as Blending. Unfortunately, the terminology differs between communities, so it's not always easy to know what type of cross-validation the ensemble is using.

```python
xtrain_base, xpred_base, ytrain_base, ypred_base = train_test_split(
    xtrain, ytrain, test_size=0.5, random_state=SEED)
```

#### Step 4: train the base learners on a training set

```python
def train_base_learners(base_learners, inp, out, verbose=True):
    """Train all base learners in the library."""
    if verbose: print("Fitting models.")
    for i, (name, m) in enumerate(base_learners.items()):
        if verbose: print("%s..." % name, end=" ", flush=False)
        m.fit(inp, out)
        if verbose: print("done")

train_base_learners(base_learners, xtrain_base, ytrain_base)
```

#### Step 5: generate base learners predictions

```python
def predict_base_learners(pred_base_learners, inp, verbose=True):
    """Generate a prediction matrix."""
    _P = np.zeros((inp.shape[0], len(pred_base_learners)))

    if verbose: print("Generating base learner predictions.")
    for i, (name, m) in enumerate(pred_base_learners.items()):
        if verbose: print("%s..." % name, end=" ", flush=False)
        p = m.predict_proba(inp)
        # With two classes, need only predictions for one class
        _P[:, i] = p[:, 1]
        if verbose: print("done")

    return _P
P_base = predict_base_learners(base_learners, xpred_base)
```

#### Step 6: Train the meta learner
The prediction matrix P_base reflects test-time performance and can be used to train the meta learner

```python
meta_learner.fit(P_base, ypred_base)
```

Now that we have a firm understanding of ensemble learning, it's time to see what it can do to improve our prediction performance on the political contributions data set:

```python
def ensemble_predict(base_learners, meta_learner, inp, verbose=True):
    """Generate predictions from the ensemble."""
    P_pred = predict_base_learners(base_learners, inp, verbose=verbose)
    return P_pred, meta_learner.predict_proba(P_pred)[:, 1]

P_pred, p = ensemble_predict(base_learners, meta_learner, xtest)
print("\nEnsemble ROC-AUC score: %.3f" % roc_auc_score(ytest, p))
```

As expected, the ensemble __beats the best estimator__ from our previous benchmark, but it doesn't beat the __simple average__ ensemble. That's because we trained the base learners and the meta learner __on only half the data__, so a lot of information is lost. To prevent this, we need to use a cross-validation strategy.

```python
from sklearn.base import clone

def stacking(base_learners, meta_learner, X, y, generator):
    """Simple training routine for stacking."""

    # Train final base learners for test time
    print("Fitting final base learners...", end="")
    train_base_learners(base_learners, X, y, verbose=False)
    print("done")

    # Generate predictions for training meta learners
    # Outer loop:
    print("Generating cross-validated predictions...")
    cv_preds, cv_y = [], []
    for i, (train_idx, test_idx) in enumerate(generator.split(X)):

        fold_xtrain, fold_ytrain = X[train_idx, :], y[train_idx]
        fold_xtest, fold_ytest = X[test_idx, :], y[test_idx]

        # Inner loop: step 4 and 5
        fold_base_learners = {name: clone(model)
                              for name, model in base_learners.items()}
        train_base_learners(
            fold_base_learners, fold_xtrain, fold_ytrain, verbose=False)

        fold_P_base = predict_base_learners(
            fold_base_learners, fold_xtest, verbose=False)

        cv_preds.append(fold_P_base)
        cv_y.append(fold_ytest)
        print("Fold %i done" % (i + 1))

    print("CV-predictions done")

    # Be careful to get rows in the right order
    cv_preds = np.vstack(cv_preds)
    cv_y = np.hstack(cv_y)

    # Train meta learner
    print("Fitting meta learner...", end="")
    meta_learner.fit(cv_preds, cv_y)
    print("done")

    return base_learners, meta_learner

from sklearn.model_selection import KFold

# Train with stacking
cv_base_learners, cv_meta_learner = stacking(
    get_models(), clone(meta_learner), xtrain.values, ytrain.values, KFold(2))

P_pred, p_stacking_ens = ensemble_predict(cv_base_learners, cv_meta_learner, xtest, verbose=False)
print("\nEnsemble ROC-AUC score: %.3f" % roc_auc_score(ytest, p_stacking_ens))
```

The upshot is that you should use a unit-tested package and focus on building your machine learning pipeline. In fact, once you've settled on a ensemble package, building ensembles becomes really easy: all you need to do is specify the base learners, the meta learner, and a method for training the ensemble.

```python
from mlens.ensemble import SuperLearner

# Instantiate the ensemble with 10 folds
sl = SuperLearner(
    folds=10,
    random_state=SEED,
    verbose=2
)

# Add the base learners and the meta learner
sl.add(list(base_learners.values()), proba=True)
sl.add_meta(meta_learner, proba=True)

# Train the ensemble
sl.fit(xtrain, ytrain)

# Predict the test set
p_sl = sl.predict_proba(xtest)

print("\nSuper Learner ROC-AUC score: %.3f" % roc_auc_score(ytest, p_sl[:, 1]))
```

```python
plot_roc_curve(ytest, p_sl[:, 1].reshape(-1, 1), P.mean(axis=1), ["Super Learner"], "Simple average")
```

### Where to go from here
There are many other types of ensembles than those presented here. However the basic ingredients are always the same: a library of base learners, a meta learner, and a training procedure. By playing around with these components, various specialized forms of ensembles can be created.

| Language  | Name         | Comment                                                                         |
|-----------|--------------|---------------------------------------------------------------------------------|
| Python    | ML-Ensemble  | General                                                                         |
| Python    | Scikit-learn | Bagging,                                                                        |
| Python    | mlxtend      | Regression                                                                      |
| Multiple  | H2O          | Distributed stacked ensemble learning. AutoML: Automatic Machine Learning       |
| Java      | StackNet     | Empowered by H20                                                                |
| Web-based | xcessiv      | Web-based ensemble learning                                                     |

#### Example of ensembles from a $10K Kaggle winning solution

Our solution is based in a 3-layer learning architecture as shown in the picture attached.

* 1st level: there are about 33 models that we used their predictions as meta features for the 2nd level, also there are 8 engineered features.
* 2nd level: there are 3 models trained using 33 meta features + 7 features from 1st level: XGBOOST, Neural Network(NN) and ADABOOST with ExtraTrees.
* 3rd level: it's composed by a weighted mean of 2nd level predictions.

All models in 1st layers are trained using a 5 fold cross-validation technique using always the same fold indices.

The 2nd level we trainned using 4 Kfold random indices. It provided us the ability to calculate the score before submitting to the leader board. All our cross-validate scores are extremely correlated with LB scores, so we have a good estimate of performance locally and it enabled us the ability to discard useless models for the 2nd learning level.

Models and features used for 2nd level training:

X = Train and test sets

* Model 1: RandomForest(R). Dataset: X
* Model 2: Logistic Regression(scikit). Dataset: Log(X+1)
* Model 3: Extra Trees Classifier(scikit). Dataset: Log(X+1) (but could be raw)
* Model 4: KNeighborsClassifier(scikit). Dataset: Scale( Log(X+1) )
* Model 5: libfm. Dataset: Sparse(X). Each feature value is a unique level.
* Model 6: H2O NN. Bag of 10 runs. Dataset: sqrt( X + 3/8)
* Model 7: Multinomial Naive Bayes(scikit). Dataset: Log(X+1)
* Model 8: Lasagne NN(CPU). Bag of 2 NN runs. First with Dataset Scale( Log(X+1) ) and second with Dataset Scale( X )
* Model 9: Lasagne NN(CPU). Bag of 6 runs. Dataset: Scale( Log(X+1) )
* Model 10: T-sne. Dimension reduction to 3 dimensions. Also stacked 2 kmeans features using the T-sne 3 dimensions. Dataset: Log(X+1)
* Model 11: Sofia(R). Dataset: one against all with learner_type="logreg-pegasos" and loop_type="balanced-stochastic". Dataset: Scale(X)
* Model 12: Sofia(R). Trainned one against all with learner_type="logreg-pegasos" and loop_type="balanced-stochastic". Dataset: Scale(X, T-sne Dimension, some 3 level interactions between 13 most important features based in randomForest importance )
* Model 13: Sofia(R). Trainned one against all with learner_type="logreg-pegasos" and loop_type="combined-roc". Dataset: Log(1+X, T-sne Dimension, some 3 level interactions between 13 most important features based in randomForest importance )
* Model 14: Xgboost(R). Trainned one against all. Dataset: (X, feature sum(zeros) by row ). Replaced zeros with NA.
* Model 15: Xgboost(R). Trainned Multiclass Soft-Prob. Dataset: (X, 7 Kmeans features with different number of clusters, rowSums(X==0), rowSums(Scale(X)>0.5), rowSums(Scale(X)< -0.5) )
* Model 16: Xgboost(R). Trainned Multiclass Soft-Prob. Dataset: (X, T-sne features, Some Kmeans clusters of X)
* Model 17: Xgboost(R): Trainned Multiclass Soft-Prob. Dataset: (X, T-sne features, Some Kmeans clusters of log(1+X) )
* Model 18: Xgboost(R): Trainned Multiclass Soft-Prob. Dataset: (X, T-sne features, Some Kmeans clusters of Scale(X) )
* Model 19: Lasagne NN(GPU). 2-Layer. Bag of 120 NN runs with different number of epochs.
* Model 20: Lasagne NN(GPU). 3-Layer. Bag of 120 NN runs with different number of epochs.
* Model 21: XGboost. Trained on raw features. Extremely bagged (30 times averaged).
* Model 22: KNN on features X + int(X == 0)
* Model 23: KNN on features X + int(X == 0) + log(X + 1)
* Model 24: KNN on raw with 2 neighbours
* Model 25: KNN on raw with 4 neighbours
* Model 26: KNN on raw with 8 neighbours
* Model 27: KNN on raw with 16 neighbours
* Model 28: KNN on raw with 32 neighbours
* Model 29: KNN on raw with 64 neighbours
* Model 30: KNN on raw with 128 neighbours
* Model 31: KNN on raw with 256 neighbours
* Model 32: KNN on raw with 512 neighbours
* Model 33: KNN on raw with 1024 neighbours
* Feature 1: Distances to nearest neighbours of each classes
* Feature 2: Sum of distances of 2 nearest neighbours of each classes
* Feature 3: Sum of distances of 4 nearest neighbours of each classes
* Feature 4: Distances to nearest neighbours of each classes in TFIDF space
* Feature 5: Distances to nearest neighbours of each classed in T-SNE space (3 dimensions)
* Feature 6: Clustering features of original dataset
* Feature 7: Number of non-zeros elements in each row
* Feature 8: X (That feature was used only in NN 2nd level training)

The 2nd level we start training cross-validated just to choose best models, tune hyperparameters and find optimum weights to average 3rd level.

After we found some good parameters, we trained 2nd level using entire trainset and bagged results.

The final model is a very stable 2nd level bagging of:

XGBOOST: 250 runs.

NN: 600 runs.

ADABOOST: 250 runs.

The average for the 3rd level we found better using a geometric mean of XGBOOST and NN. For ET we did an aritmetic mean with previous result: 0.85 * [XGBOOST^0.65 * NN^0.35] + 0.15 * [ET].

We tried a lot of training algorithms in first level as Vowpal Wabbit(many configurations), R glm, glmnet, scikit SVC, SVR, Ridge, SGD, etc... but none of these helped improving performance on second level.

Also we tried some preprocessing like PCA, ICA and FFT without improvement.

Also we tried Feature Selection without improvement. It seems that all features have positive prediction power.

Also we tried semi-supervised learning without relevant improvement and we discarded it due the fact that it have great potential to overfit our results.

Definetely the best algorithms to solve this problem are: Xgboost, NN and KNN. T-sne reduction also helped a lot. Other algorithm have a minor participation on performance. So we learn not to discard low performance algorithms, since it have enough predictive power to improve performance in a 2nd level training.

Our final cross-validated solution scored around 0.3962. LB(Public): 0.38055 and LB(Private): 0.38243.

Thanks for the ideas:
* https://pdfs.semanticscholar.org/4e81/f4c95dde9a327026c584b91a3fe691595d1f.pdf
* https://www.stat.berkeley.edu/users/laan/Class/Class_subpages/BASS_sec1_3.1.pdf
* https://blog.statsbot.co/ensemble-learning-d1dcd548e936
* http://web.engr.oregonstate.edu/~tgd/publications/mcs-ensembles.pdf

Thanks for the code:
* https://www.dataquest.io/blog/introduction-to-ensembles/
* http://web.engr.oregonstate.edu/~tgd/publications/mcs-ensembles.pdf
* https://github.com/vsmolyakov/experiments_with_python/blob/master/chp01/ensemble_methods.ipynb

```python

```


# Transformation

\begin{align}
\sum_{j=1}^{M}{\left\| S_j - T_j \right\|}_F^2
&= \sum_{j=1}^{M}{\left\| S_j - \widetilde{V}_jV_j^{-1} \right\|}_F^2
\\
&= \sum_{j=1}^{M}{\left\| \left( S_j - \widetilde{V}_jV_j^{-1} \right)^T \right\|}_F^2
\\
&= \sum_{j=1}^{M}{\left\| S_j^T - {V_j^{-1}}^T \widetilde{V}_j^T \right\|}_F^2
\\
&= \sum_{j=1}^{M}{\left\| S_j^T - {\hat{V}_j^{-1}}^T x^T \right\|}_F^2
\\
&= {\left\|
\begin{pmatrix}
S_1^T
\\
S_2^T
\\
...
\\
S_M^T
\end{pmatrix} -
\begin{pmatrix}
{\hat{V_1}^{-1}}^T
\\
{\hat{V_2}^{-1}}^T
\\
...
\\
{\hat{V_M}^{-1}}^T
\end{pmatrix} x^T
\right\|}_F^2
\end{align}


# Simulating classical orbits using the RK4 method

The Jupyter notebook below will walk you through how to simulate classical two-body orbits using the 4th-order Runge-Kutta (RK4) numerical integration technique. Since this is our first interaction with Python, we will step through these items carefully.

**Note:** this is strongly related to material from week 2 of the course, themed as [*What time is it on Mars?*](https://alurban.github.io/black-hole-astrophysics/week-2/)

## Step 1: import useful packages

As we've discussed before, Python is an *interpreted* language (rather than a *compiled* one). This means you can interact directly with the Python interpreter to run your code in real-time, without having to compile it. There are many, many, many third-party software applications available through the Python Package Index ([PyPI](https://pypi.org)), which in general can be **imported** so you can use them in your code.

In this example we will make heavy use of [NumPy](https://numpy.org), a fundamental package for scientific computing with Python.

```python
import numpy
```

We will also make use of [Matplotlib](https://matplotlib.org), a standard and phenomenally useful data visualization library with native support for Jupyter. Here and in general, we will include the rule `matplotlib inline` so that plots can be rendered directly within the notebook.

```python
import matplotlib.pyplot as plt
%matplotlib inline
```

## Step 2: define global variables

We will now define a few global-scope variables that will be used across the notebook, particularly in some functions defined below. By convention these global variables should always be named in all-caps, in order to distinguish them from other, local-scope variables.

In general, the global variables defined here correspond either to physical constants or to general properties of the binary system we're trying to simulate. The values I've entered below correspond to the orbit of the planet [Mercury](https://solarsystem.nasa.gov/planets/mercury/overview/) around the Sun, but if you wanted to visualize a different system, you can change those values here.

**Note:** Mercury is a very important example in the history of general relativity, so it is one we will be returning to.

```python
# global variables
PI = numpy.pi
ECC = 0.21  # orbital eccentricity
G = 6.67408e-11  # Newton's constant in m^3 / kg / s
MSUN = 1.989e30  # Solar mass in kg
M1 = MSUN  # primary mass in kg
M2 = 3.285e23  # Mercury mass in kg
MU = M1 * M2 / (M1 + M2)  # reduced mass in kg
M = M1 + M2  # total mass in kg
AU = 1.496e11  # 1 astronomical unit in m
L = 9e38  # Mercury angular momentum in J*s
RL = L**2 / (G * M * MU**2)  # semi-latus rectum in meters
E = (MU**3 / 2) * (G*M / L)**2 * (ECC**2 - 1)  # total energy in J
```

## Step 3: define useful helper functions

In the course of doing our simulation, it'll be useful to outsource the heavy lifting to a few robustly-defined functions that can then be called at each step. We will define one function for the right-hand side of our differential equation for $\varphi(t)$,

$$ \dot{\varphi} = \frac{d\varphi}{dt} = F(\varphi) = \frac{L}{\mu r_L^2} \left( 1 + e\cos\varphi \right)^2 $$

where $r_L$ is the semi-latus rectum for a given angular momentum. A second function will approximate the solution to this equation using the RK4 method, which is determined by a four-step process at each point:

\begin{align}
k_1 &= h\,F(\varphi_i) \\
k_2 &= h\,F\left(\varphi_i + \frac{k_1}{2}\right) \\
k_3 &= h\,F\left(\varphi_i + \frac{k_2}{2}\right) \\
k_4 &= h\,F(\varphi_i + k_3) \\
\varphi_{i+1} &= \varphi_i + \frac{1}{6} \left( k_1 + 2k_2 + 2k_3 + k_4 \right)
\end{align}

where $h$ is the step size in time and $\varphi_i$ the value of $\varphi$ at the previous point.

The functions `rhs` and `rk4` below implement these equations numerically. A third function, `_format_scientific`, is provided in order to render numerical values nicely for plots using $\LaTeX$.

**Note:** The name of `_format_scientific` begins with an underscore (`_`) because it is intended as a *private function*. What this means is that, if you were to convert this notebook to a standard Python module and then try to import it, `rhs` and `rk4` would be within your purview to use, but `_format_scientific` is intended for internal use only.

```python
# -- functions ----------------------------------------------------------------

def _format_scientific(x):
    """Format an arbitrary floating-point number in scientific notation
    """
    n = int(numpy.floor(numpy.log10(x)))
    return '%.3g' % x if (n >= -1 and n <= 4) else \
        r'%.3g \times 10^{%d}' % (x / 10**n, n)

def rhs(y):
    """Returns the right-hand side of the equation of motion at a single
    point, given the following parameters:

        y: solution value at the previous point
    """
    return L * (1 + ECC * numpy.cos(y))**2 / (MU * RL**2)

def rk4(y, h):
    """Returns the estimated integral at a new point using the RK4 method (an
    extension of Simpson's rule) given the following parameters:

        y: the function value at the previous point
        h: the integration step size
    """
    k1 = h * rhs(y)
    k2 = h * rhs(y + k1/2)
    k3 = h * rhs(y + k2/2)
    k4 = h * rhs(y + k3)
    return y + (1/6) * (k1 + 2*k2 + 2*k3 + k4)
```

## Step 4: set initial conditions

We're now in a position to stage the simulation by setting up its initial conditions. This includes an initial value for $\varphi(0) = \varphi_0$, which we will set to 0 radians so that motion starts on the positive $x$-axis.

We will also need to create an array of discrete timestamps. We can be somewhat clever in doing this, using a combination of elliptical geometry and Kepler's third law to predict the orbital period, $T$, in terms of the semi-major axis, $a$:

\begin{align}
a = \frac{r_L}{1 - e^2} \\
\left(\frac{T}{2\pi}\right)^2 = \frac{a^3}{GM}
\end{align}

Thus will simulate exactly one orbital period, with a relatively small step size of $h = T/10^2$.

```python
# set initial conditions
a = RL / (1 - ECC**2)  # orbital semi-major axis
T = numpy.sqrt(4 * PI**2 * a**3 / (G * M))  # orbital period
dt = T / 1e2  # step size, determined as 1% of T
t = numpy.arange(0, T + dt, dt)  # time samples
phi = [0]  # orbital phase initial condition
```

## Step 5: simulation!

Now that we've laid all the groundwork, we're in a position to perform the actual simulation. This is going to look remarkably simple because of the way we've defined helper functions: we're just going to have a simple `for` loop that ranges over all timestamps and, for each one, calls a function to iterate the numerical integration and then appends the result to `phi`.

```python
# do the simulation
for i in range(t.size - 1):
    phi.append(rk4(phi[i], dt))
```

## Step 6: get the radial position

We will now reconstruct the radial position from the standard formula for a conic section that we derived in class,

$$ r[\varphi(t)] = \frac{r_L}{1 + e\cos[\varphi(t)]} $$

To do this efficiently we will first convert `phi` to a NumPy array object, which makes mathematical operations easier to handle internally.

```python
# get radial position
phi = numpy.array(phi)  # NumPy arrays are easier to work with
r = RL / (1 + ECC * numpy.cos(phi))  # from the parametric equation for r
```

## Step 7: plot the orbital track

We can now visualize the orbital track by plotting $(r(t), \varphi(t))$ in polar coordinates. Fortunately, Matplotlib makes it very easy to do this, with support for fine-tuning various features of the plot such as axis tick marks and color schemes.

```python
# plot the equivalent-one-body orbital track
fig = plt.figure(figsize=(6, 6))
ax = fig.gca(projection='polar')  # use polar projection
ax.plot(phi, r/AU, '#4ba6ff',
        label='$m_1 = %s\,M_{\odot}$\n$m_2 = %s\,M_{\odot}$' % (
            _format_scientific(M1/MSUN), _format_scientific(M2/MSUN)))
ax.grid(True)

ax.legend(framealpha=1)
ax.set_rticks([0.2, 0.4, 0.6])
ax.set_title(r'$L=%s$ J$\cdot$s, $e=%.3g$' % (
    _format_scientific(L), ECC), va='bottom')
plt.show()
```

## Step 8: check that energy is conserved

Huzzah, we have simulated Mercury's orbit! Now we'll want to do a couple of sanity checks to make sure our integration scheme did it accurately and faithfully. To examine this, recall that we already know the energy we *expect* the system to have is the constant `E`, a global variable. We can directly estimate the percentage error by numerically differentiating $r$, re-calculating the energy by hand for each data point using the condition

$$ E = \frac{1}{2} \,\mu \dot{r}^2 - \frac{GM\mu}{r} + \frac{L^2}{2\mu r^2}, $$

and then comparing these values to `E`.

**Note:** to take a derivative numerically, we will use the convenience function [`numpy.gradient`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.gradient.html). This is based on an algorithm that is second-order accurate everywhere except the first and last data points, where it is first-order accurate.

```python
# check that the energy is conserved
rdot = numpy.gradient(r, dt)  # time derivative of r
energy = 0.5*MU*rdot**2 - G*M*MU/r + L**2/(2*MU*r**2)
error = numpy.abs((energy - E)/E)

# set up a figure
ty = t / 3.1536e7  # convert time to years
fig = plt.figure(figsize=(8, 6))

# plot the total energy
ax1 = fig.add_subplot(211)
ax1.plot(ty, energy/1e32, 'Orange', linewidth=2)
ax1.plot([ty[0], ty[-1]], [E/1e32, E/1e32], 'k--')
ax1.set_ylim([1.1*E/1e32, 0.9*E/1e32])
ax1.set_ylabel('$E$ [$10^{32}$ J]')

# plot the percentage error on a separate graph
ax2 = fig.add_subplot(212, sharex=ax1)
ax2.plot(ty, 100*error, 'DarkSlateGray', linewidth=2)
ax2.set_xlim([ty.min(), ty.max()])
ax2.set_xlabel('$t$ [years]')
ax2.set_ylim([1e-5, 100])
ax2.set_yscale('log')
ax2.set_ylabel(r'Percentage error [%]')

# show the figure
plt.show()
```

## Step 9: check that angular momentum is conserved

Finally, we will do the same with the magnitude of orbital angular momentum, based on the condition that

$$ L = \mu r^2 \dot{\varphi}. $$

We will compare this to `L`, our global variable giving the expected angular momentum.

```python
# check that the angular momentum is conserved
phidot = numpy.gradient(phi, dt)  # time derivative of phi
angmomentum = MU * r**2 * phidot
Lerror = numpy.abs((angmomentum - L)/L)

# set up a figure
fig = plt.figure(figsize=(8, 6))

# plot the orbital angular momentum
ax1 = fig.add_subplot(211)
ax1.plot(ty, angmomentum/1e38, 'DeepSkyBlue', linewidth=2)
ax1.plot([ty[0], ty[-1]], [L/1e38, L/1e38], 'k--')
ax1.set_ylim([0.9*L/1e38, 1.1*L/1e38])
ax1.set_ylabel('$L$ [$10^{38}$ J$\cdot$s]')

# plot the percentage error on a separate graph
ax2 = fig.add_subplot(212, sharex=ax1)
ax2.plot(ty, 100*Lerror, 'DarkSlateGray', linewidth=2)
ax2.set_xlim([ty.min(), ty.max()])
ax2.set_xlabel('$t$ [years]')
ax2.set_ylim([1e-5, 100])
ax2.set_yscale('log')
ax2.set_ylabel(r'Percentage error [%]')

# show the figure
plt.show()
```

## Digging deeper

Now that we've simulated a complete orbit, consider the following:

* Can you plot `r` and `phi` as a function of time? (Try to do it in the same figure but on separate graphs, like we did for the percentage error plots above.)
* If you think of $r$ and $\varphi$ as oscillators, do they have the same frequency? (Remember: they should, because there can be no apsidal precession in the classical two-body problem.)
* What does the motion look like when you transform back to the two-body model? Does it match what you expected?
* What do the orbital features look like if you try to model different Solar System planets? How well are the energy and angular momentum conserved, numerically speaking?
* Do you have to change anything about your initial conditions if you want to model parabolic or hyperbolic orbits? How well are the energy and angular momentum conserved, numerically?


# Travel Times in 1D Models

## Name:

## Date:

```python
# If using Anaconda3 on your machine you can do without this.  This is for Azure people.
#!pip install obspy # TODO Uncomment if on Azure
```

## Computing the $X$ offset and travel time $T$ of the downgoing ray in a linear velocity gradient

To begin, recall that the offset, $X$, and travel time $T$ of the down-going ray path in $1D$ media is very generally given by
\begin{equation}
  x(p) = \int_0^{h} \frac{p v(h)}{\sqrt{1 - p^2 v^2(h)}} dh
\end{equation}
\begin{equation}
  t(p) = \int_0^{h} \frac{1}{v(h) \sqrt{1 - p^2 v^2(h)}} dh
\end{equation}
where $h$ is the thickness of the model, $v$ the velocity, and $p$ the ray parameter.  We now introduce the velocity gradient
\begin{equation}
  v = v(h) = v_0 + a h
\end{equation}
where $v_0$ is the velocity at $h=0$ and $a$ the velocity gradient.  Inserting the velocity gradient into the offset and travel time equations
\begin{equation}
  x(p) = \int_0^{h} \frac{p (v_0 + ah)}{\sqrt{1 - p^2(v_0 + ah)^2}} dh
\end{equation}
\begin{equation}
  t(p) = \int_0^{h} \frac{1}{(v_0 + a h) \sqrt{1 - p^2 (v_0 + ah)^2} } dh
\end{equation}
Let's now change the integrals to be functions of $v$.  To do this we use the change of variables $v = v_0 + a h$.  The derivative requires that $dv = a dh$ or
\begin{equation}
  dh = \frac{dv}{a}
\end{equation}
The limits of the integrands must also be reworked so that
\begin{equation}
  v(h=0) = v_0
\end{equation}
and
\begin{equation}
  v(h=h) = v_0 + \frac{(v_1 - v_0)}{h} h = v_1
\end{equation}
the offset and travel time integrals are then transformed into
\begin{equation}
  x(p) = \frac{1}{a} \int_{v_0}^{v_1} \frac{pv}{\sqrt{1 - p^2 v^2}} dv
\end{equation}
\begin{equation}
  t(p) = \frac{1}{a} \int_{v_0}^{v_1} \frac{1}{v \sqrt{1 - p^2 v^2} } dv
\end{equation}
The first integral has, for this entire time, corresponded to a fairly obvious u substitution.  The second integral is less obviously the inverse hyperbolic cos.  Let's make the mechanics of both integrals a little more clear by making one more substitution, $s = pv$.  This requires that
\begin{equation}
  dv = \frac{ds}{p}
\end{equation}
and new integral limits $s_1 = pv_0$ and $s_2 = pv_1$.
\begin{equation}
  x(p) = \frac{1}{a} \int_{pv_0}^{pv_1} \frac{s}{\sqrt{1 - s^2}} \frac{ds}{p}
       = \frac{1}{a p} \int_{pv_0}^{pv_1} \frac{s}{\sqrt{1 - s^2}} ds
\end{equation}
\begin{equation}
  t(p) = \frac{1}{a} \int_{pv_0}^{pv_1} \frac{p}{s \sqrt{1 - s^2} } \frac{ds}{p}
       = \frac{1}{a} \int_{pv_0}^{pv_1} \frac{1}{s \sqrt{1 - s^2} } ds
\end{equation}
The first integral is
\begin{equation}
  x(p) =-\frac{1}{ap} \left . \sqrt{1 - s^2} \right |_{p v_0}^{p v_1}
       = \frac{1}{ap} \left . \sqrt{1 - s^2} \right |_{p v_1}^{p v_0}
       = \frac{1}{ap} \left . \sqrt{1 - v^2 p^2} \right |_{v_1}^{v_0}
       = \frac{v}{a p} \left . \sqrt{\frac{1}{v^2} - p^2} \right |_{v_1}^{v_0}
\end{equation}
The second integral is a little more complicated so I'll write down the punchline.
<!--
In this case, you would say that $u=\sqrt{1 - s^2}$.  Thus, $du =-\frac{s}{\sqrt{1 - s^2}} ds$ or $\frac{ds}{\sqrt{1 - s^2}} =-\frac{du}{s}$.
\begin{equation}
  t(p) =-\frac{1}{a} \int \frac{du}{s^2}
       =-\frac{1}{a} \int \frac{du}{1 - u^2}
       = \frac{1}{a} \int \frac{du}{u^2 - 1}
       = \frac{1}{a} \int \frac{1}{1 - u} - \frac{1}{u + 1} \, du
       = \frac{1}{a} \left [ \log(1 - u) - \log(u + 1) \right ]
       = \frac{1}{a} \left [ \log(1 - \sqrt{1-s^2}) - \log(\sqrt{1-s^2} + 1)  \right ]
       = \frac{1}{a} \left [ \frac{1 - \sqrt{1 - s^2}}{\sqrt{1 - s^2} + 1} \right ]
\end{equation}
-->
\begin{equation}
  t(p) =-\frac{1}{a} \log \left ( \frac{1 + \sqrt{1 - s^2}}{s} \right )_{p v_0}^{p v_1}
       = \frac{1}{a} \log \left ( \frac{1 + \sqrt{1 - p^2 v^2}}{p v} \right )_{v_1}^{v_0}
       = \frac{1}{a} \log \left ( \frac{1}{vp} +  \frac{1}{p} \sqrt{ \frac{1}{v^2} - p^2 } \right )_{v_1}^{v_0}
\end{equation}
Defining the vertical slowness $\eta = \sqrt{\frac{1}{v^2} - p^2}$ and $u=\frac{1}{v}$ we have that the offset is simply given by
\begin{equation}
  x(p) = \left . \frac{1}{a u p} \eta \right |_{u_1}^{u_0}
\end{equation}
\begin{equation}
  t(p) = \frac{1}{a} \log \left ( \frac{u + \eta}{p} \right )_{v_1}^{v_0}
\end{equation}
You would think that here Shearer calls it a day.  But no.  Let's further add by zero so that
\begin{equation}
  t(p) = \frac{1}{a} \log \left ( \frac{u + \eta}{p} \right )_{v_1}^{v_0}
       - p \left . \frac{1}{a u p} \eta \right |_{u_1}^{u_0}
       + p \left . \frac{1}{a u p} \eta \right |_{u_1}^{u_0}
       = \frac{1}{a} \left (  \log \left ( \frac{u + \eta}{p} \right ) - \frac{\eta}{u} \right )_{v_1}^{v_0}
       + p x(p)
\end{equation}
## What Needs to be Programmed
And there you have it - the equations to implement for computing offset and travel time are
\begin{equation}
  x(p) = \left . \frac{1}{a u p} \eta \right |_{u_1}^{u_0}
\end{equation}
\begin{equation}
  t(p) = \frac{1}{a} \left (  \log \left ( \frac{u + \eta}{p} \right ) - \frac{\eta}{u} \right )_{v_1}^{v_0}
       + p x(p)
\end{equation}
In general we have to be somewhat careful.  For example, if the slowness becomes undefined at the bottom of the layer we would say the ray has turned and only keep the first contribution.  Additionally, we are not counting the upgoing contribution, thus, $x(p)$ would have to be multiplied by $2$ to obtain $X(p)$.

# Computing the Travel Time Integral

For the travel time integral we substitute $u=\sqrt{1 - s^2}$.  The corresponding derivative is
$du =-\frac{s}{\sqrt{1 - s^2}} ds$ or
$$
  \frac{ds}{\sqrt{1 - s^2}} =-\frac{du}{s}
$$

\begin{equation}
  t(p) =-\frac{1}{a} \int \frac{du}{s^2}
       =-\frac{1}{a} \int \frac{du}{1 - u^2}
\end{equation}
Using an integration table, $\int \frac{du}{1 - u^2}  = \tanh^{-1}(u)$, so our integral becomes
\begin{equation}
  t(p) =-\frac{1}{a} \tanh^{-1}(u)
\end{equation}
Now back-substitute
\begin{equation}
  t(p) =-\frac{1}{a} \tanh^{-1} \left ( \sqrt{1 - s^2} \right )
\end{equation}
We now apply the definition of the inverse hyperbolic arctangent that
$$
  \tanh^{-1} x = \frac{1}{2} \left [ \log (1 + x) - \log(1 - x) \right ]
$$
so that
$$
  \tanh^{-1} s
= \frac{1}{2}
  \left [ \log \left (1 + \sqrt{1 - s^2} \right ) - \log \left (1 - \sqrt{1 - s^2} \right ) \right ]
= \frac{1}{2}
  \left [
    \log \left (\frac{1 + \sqrt{1 - s^2}}{1 - \sqrt{1 - s^2}} \right )
  \right ]
= \frac{1}{2}
  \left [
    \log
    \left (
      \left (\frac{1 + \sqrt{1 - s^2}}{1 - \sqrt{1 - s^2}} \right )
      \left (\frac{1 + \sqrt{1 - s^2}}{1 + \sqrt{1 - s^2}} \right )
    \right )
  \right ]
$$
Notice the denominator is now $1^2 - \sqrt{1 - s^2}^2 = 1 - (1 - s^2) = s^2$.  And the numerator is just
$\left ( 1 + \sqrt{1 - s^2} \right )^2$. We now apply the property of logarithms that $a \log x = \log x^a$.
$$
  \tanh^{-1} s
= \log \left ( \sqrt { \left ( \frac{(1 + \sqrt{1 - s^2})^2}{s^2} \right ) } \right )
= \log \left ( \frac{1 + \sqrt{1 - s^2}}{s} \right )
%= \log \left (1 + \sqrt{1 - s^2} \right ) - \log s
$$
Therefore, we obtain
\begin{equation}
 t(p)
=-\frac{1}{a} \int_{p v_0}^{p v_1} \frac{1}{s\sqrt{1 - s^2}} ds
=\left . -\frac{1}{a}  \log \left ( \frac{1 + \sqrt{1 - s^2}}{s} \right ) \right |_{p v_0}^{p v_1}
\end{equation}

## Additional Notes

The cleanest derivations I've seen for the the linear gradient problem are from Slotnick's Lessons in Seismic Computing and Sergey Fomel's 2008 paper - Fast sweeping method for the factored eikonal equation.  I ultimately used Slotnick's derivation because it is simpler but working from the ray-path equation defined as a first order system does open more potential doors.  I've tried to rework Slotnick's work to match the results given in the Shearer's textbook.

<!---

If you want to have a go then uncomment the lines in this cell
$$
\cosh^{-1}(x) = \ln(x + \sqrt{x^2 -1})
$$
$$
\int \frac{1}{\sqrt{x^2 - 1}}
$$
$$
\ln \left ( \frac{1}{v(z)} + \sqrt{\frac{1}{v(z)^2} - p^2} \right )
- \frac{1}{v(z)} \sqrt{\frac{1}{v(z)^2} - p^2} - \ln(p)
$$
Using Wolfram this is really close to the integral of
$$
I=\int \sqrt{x^2 - p^2} dx =-\frac{p^2}{2} \ln(x^2 - p^2) + x\sqrt{x^2 - p^2} + constant
$$
which would be great because I can write
$$
t(p) = px(p) + \int \sqrt{\gamma(z)^2 - p^2} dz
$$
anyway...
$$
\ln \left ( x + \sqrt{x^2 - 1} \right )
-\frac{p}{v(z)} \sqrt{\frac{1}{p^2 v(z)^2} - 1}
$$
$$
\ln \left ( x + \sqrt{x^2 - 1} \right )
-\frac{p^2}{v(z)} \sqrt{\frac{1}{p^2 v(z)} - 1}
$$
$$
t(x) = \ln \left ( x + \sqrt{x^2 - 1} \right ) - p^2 x \sqrt{x^2 - 1}
$$
Differentiate and get
$$
  \frac{dt}{dx}
 =\frac{1}{\sqrt{x^2 - 1}} \frac{dx}{dv} \frac{dv}{dz}
- p^2 \sqrt{x^2 - 1} \frac{dx}{dv} \frac{dv}{dz}
- p^2 x^2 \frac{1}{\sqrt{x^2 - 1}} \frac{dx}{dv} \frac{dv}{dz}
$$
$$
\frac{1 - p^2 x^2}{\sqrt{x^2 - 1}} \frac{dx}{dz}  - p^2 \sqrt{x^2 - 1} \frac{dx}{dz}
$$
$$
\frac{1 - p^2 x^2}{\sqrt{x^2 - 1}} \frac{dx}{dz}  - \frac{p^2 (x^2 - 1)}{\sqrt{x^2 - 1}} \frac{dx}{dz}
$$
$$
\frac{1 + p^2(1 - 2 x^2)}{\sqrt{x^2 - 1}} \frac{dx}{dz}
$$
Maybe this is the time to evaluate $\frac{dv}{dz} = b$ and $\frac{dx}{dv} =-\frac{1}{p v(z)^2}$ so
that
$$
  \frac{dx}{dv} \frac{dv}{dz} =-\frac{b}{p v(z)^2}
$$
And
$$
 -\frac{1 + p^2(1 - 2 x^2)}{\sqrt{\frac{1}{pv(z)^2} - 1}} \frac{b}{p v(z)^2}
=-\frac{b + b p^2 \left (1 - 2 \frac{1}{p^2 v(z)^2} \right )}{v(z) \sqrt{1 - v(z)^2 p^2}}
=-\frac{b(1 + p^2)  - 2 \frac{1}{v(z)^2} }{v(z) \sqrt{1 - v(z)^2 p^2}}
$$
--->

```python
from numpy import sqrt
from numpy import log
from numpy import linspace
from numpy import zeros
from numpy import asarray

def layerxt(p, h, vtop, vbottom):
    """
    Calcualtes the change in position and change in time for a ray in a layer with a linear
    velocity gradient.  This is a Python translation of Shearer's layerxt which was heavily
    modified from a portion of Chris Chapman's WKBJ program.

    Input
    -----
    p : float
        Horizontal slowness (s/km).
    h : float
        Layer thickness (km).
    vtop : float
        Velocity (km/s) at the top of the layer.
    vbottom : float
        Velocity (km/s) at the bottom of the layer.

    Returns
    -------
    dx : float
         Range offset (km).
    dt : float
         Travel time change (seconds).
    irtr : integer
        This is a return code that can mean:
          -1 -> zero thickness layer.
           0 -> ray turned above the layer.
           1 -> ray passed through the layer.
           2 -> ray turned in the layer and only one leg in the dx and dt is counted.

    """
    # Checks
    irtr =-1
    dx = None
    dt = None
    if (h < 0.0):
        print("Layer thickness cannot be negative")
        return dx, dt, irtr
    if (vtop <= 0.0 or vbottom <= 0.0):
        print("Velocities must be positive")
        return dx, dt, irtr
    # Compute slownesses
    u1 = 1.0/vtop    # Slowness at top of layer
    u2 = 1.0/vbottom # Slowness at bottom of layer
    dx = 0.0
    dt = 0.0
    # Ray turned above this layer
    if (p >= u1):
        irtr = 0
        return dx, dt, irtr
    # Zero layer thickness
    if (h == 0):
        irtr = -1
        return dx, dt, irtr
    b = (vtop - vbottom)/h # Velocity gradient; Eqn 4.20 of Shearer
    eta1 = sqrt(u1**2 - p**2)

    # Constant velocity layer - ray must pass through
    if (abs(b) == 0.0):
        dx = h*p/eta1
        dt = h*u1**2/eta1
        irtr = 1
        return dx, dt, irtr

    x1 = eta1/(u1*b*p) # Eqn 4.21 of Shearer
    tau1 = (log((u1 + eta1)/p) - eta1/u1)/b # Eqn 4.22 of Shearer
    # Ray turns within layer - no contribution to integral from bottom point
    # Ray turned in the layer
    if (p >= u2):
        dx = x1
        dtau = tau1
        dt = dtau + p*dx
        irtr = 2
        return dx, dt, irtr

    # Ray passed through the layer
    irtr = 1
    eta2 = sqrt(u2**2 - p**2)
    x2 = eta2/(u2*b*p) # Eqn 4.21 of Shearer
    tau2 = (log((u2 + eta2)/p) - eta2/u2)/b # Eqn 4.22 of Shearer

    dx = x1 - x2
    dtau = tau1 - tau2

    dt = dtau + p*dx

    return dx, dt, irtr

```

```python
def computeXTInLayeredGradientModel(rayParameters, deps, vp, redvel = 8.0):
    """
    Computes the travel-time distance in a layered model where each model
    consists of velocity gradients.  Here, the structural models increase
    positively with depth.

    Input
    -----
    rayParameters : array_like
        Apparent slownesses (s/km)
    deps : array_like
        Depth to each interface (km).  Note, there are len(h) - 1 layers and each
        layer thickness is h[layer+1] - h[layer].
    vp : array_like
        Compressional velocity (km/s) at the layer interfaces.
    redvel : float
        Reduction velocity (km/s).  Shearer uses 8 km/s in the HW.

    Returns
    -------
    offset : array_like
        The offsets in x (km), i.e, distance traveled for each ray parameter.
    ttimes : array_like
        The redued travel times (s) for each ray parameter.

    Copyright
    ---------
    Ben Baker distributed under the MIT license.

    """
    # Loop on ray parameters
    xtimes = zeros(len(rayParameters))
    ttimes = zeros(len(rayParameters))
    for ip in range(len(rayParameters)):
        # Loop on model depths
        x = 0.0 # Offset
        t = 0.0 # Travel time
        p = rayParameters[ip] # Ray parameter
        for depth in range(len(deps)-1):
            h = deps[depth+1] - deps[depth]
            vtop    = vp[depth]
            vbottom = vp[depth+1]
            dx, dt, irtr = layerxt(p, h, vtop, vbottom)
            # Typically the ray continues through the layer
            if (irtr == 1):
                x = x + dx*2.0 # Up and downgoing leg
                t = t + dt*2.0 # Up and downgoing leg
            elif (irtr == 2):
                x = x + dx*2.0 # Up and downgoing leg
                t = t + dt*2.0 # Up and downgoing leg
                break
            # Ray turned in layer above - stop calculating (I should hit this)
            if (irtr == 0):
                break
        # Loop on depths
        xtimes[ip] = x # Tabulate distance
        ttimes[ip] = t # Tabulate time
    # Loop on ray parameters
    #TODO - Something screwy - but x and t are negative.  I need to implement from Chapman's book
    offset =-xtimes # Make xtimes positive
    ttimes =-ttimes
    if (redvel > 0.0):
        reducedTimes = ttimes - offset/redvel # Reduce travel times with reduction velocity
    else:
        reducedTimes = ttimes
    return offset, reducedTimes

```

## Programming Question - Shearer Question 8 From Section 4.11

Compute the P-wave traveltime, $T(X)$ curve for the velocity model

| Depth (km)  | $\alpha$ (km/s) | $\beta$ (km/s)  | $\rho$ (g/cm^3) |
| ----------- |-------------:| -----:|----:|
| 0.0         | 4.50         | 2.40  | 2.0 |
| 1.5         | 6.80         | 3.75  | 2.8 |
| 6.0         | 7.00         | 3.85  | 2.9 |
| 6.5         | 8.00         | 4.60  | 3.1 |
| 10.0        | 8.10         | 4.70  | 3.1 |

for $100$ ray parameters from $0.1236$ s/km to $0.2217$ s/km.  Use a reduction velocity of $8$ km/s.

As a hint, your function call should look something like:

    offset, ttimes = computeXTInLayeredGradientModel(rayParameters, deps, velocities, redvel)

You'll probably want to program something like:

    deps = [0.0, 1.5,  6.0,  6.5, 10.0] # Layer interfaces
    vs   = [2.4, 3.75, 3.85, 4.6, 4.7]  # Shear velocity (km/s) - THIS IS OF NO USE TO YOU - CHANGE TO VP
    redvel = 8.0  # Reduction velocity (km/s).  This trick increases the resolution of the curve.
    pmin = 0.1236 # Minimum slowness in x (s/km)
    pmax = 0.2217 # Maximum slowness in x (s/km)
    np = 100 # Number of ray parameters
    rayParameters = linspace(pmin, pmax, np)
    offset, ttimes = computeXTInLayeredGradientModel(rayParameters, deps, vp, redvel)

## Insert Your Code Here

```python
# Put your code here

```

## Question

Caustics are where $\frac{dp}{dx} = \infty$.  Given this, plot a travel-time vs. offset curve from distance $0$ to $\max\{offset\}$ and identify the offsets at which the caustics are located.  Be sure to fix the code so that your axes are correctly labeled.

```python
import matplotlib.pyplot as plt
plt.plot(offset, ttimes) # Plot travel time as a function of offset
plt.ylabel('label') # TODO put the correct label name
plt.xlabel('label') # TODO put the correct label name
plt.title('label')  # TODO put an appropriate title for this type of plot
plt.xlim(0,max(offset))
plt.xticks(linspace(0,100,11))
plt.grid(True)
plt.show()

```

WRITE YOUR ANSWER HERE

## Question

We expect large amplitudes where rays are focused.  A way to identify an abundance of rays is to look at places where the change in ray parameter to the change in distance quickly changes to $0$ - i.e., places where the ray parameter/distance curve becomes flat.  At which distances would you expect the greatest amount of enregy to occur?  As a hint, don't pick ~72 km - though there is energy focusing here it won't be the most significant because the ray density will decrease as distance increases (this is a way to describe geometrical spreading).

```python
import matplotlib.pyplot as plt
plt.plot(offset, rayParameters) # Plot ray-parameter as a function of offset
plt.ylabel('label') # TODO - put an appropriate label here
plt.xlabel('label') # TODO - put an appropriate label here
plt.title('label')  # TODO - put an appropriate title here
plt.xlim(0,max(offset))
plt.xticks(linspace(0,100,11))
plt.grid(True)
plt.show()
```

WRITE YOUR ANSWER HERE

# Depth Phases and Event Depth Estimation

The ability to predict travel time as a function of distance is useful for a variety of tasks.  One particularly important task is to locate earthquakes in $1D$ models.  Here we will use as a-priori information the event epicenter and optimize for the a depth and origin time.  At teleseismic distances, like in local events, we face a trade-off.  This trade-off is that we could decrease the event depth by decreasing the origin-time.  A way to eliminate this trade-off is to use depth phases that first travel up from the source and then reflect onto receiver location thereby emplacing a constraint on the depth.  For this activity we'll get some data from a deep earthquake in Alaska that has $P$ phases and $pP$ depth phases.  Then, we will generate theoretical times using a global theoretical travel time table.  Finally, we will tabulate an objective function and see how depth and origin time trade-off at teleseismic distances with $P$ phases and with $P$ phases and $pP$ phases.

## Data

Here we will use data collected by some broadband Pacific Northwest Seismic Network stations.  The information we extract is an estimate of the phase arrival time, or a travel time *pick*.

```python
# If using Anaconda3 on your machine you can do without this.  This is for Azure people.
#!pip install obspy # TODO Uncomment if on Azure
```

```python
# A relatively deep earthquake in Alaska
from obspy import UTCDateTime
from obspy.clients.fdsn import Client
from obspy.geodetics.base import gps2dist_azimuth
from obspy import read
from obspy import Stream
from obspy.signal.trigger import recursive_sta_lta # Help with picking
from obspy.signal.trigger import plot_trigger      # Visualizing what the picker
from obspy.taup import TauPyModel # This will let us compute global travel times
from numpy import meshgrid # Plotting stuff
from numpy import sqrt
import matplotlib.cm as cm

# Initialize some obspy things
model = TauPyModel(model="ak135") # For computing travel times
client = Client("IRIS")
# Pull waveforms BC earthquake from https://earthquake.usgs.gov/earthquakes/eventpage/us2000b2vq#executive
originTime = UTCDateTime("2017-10-08T22:34:33")
eventLat = 52.391   # Event latitude from catalog
eventLon =-176.769  # Event longitude from catalog
eventDepth = 119.0  # Event depth from catalog

def getStationLocation(network="IU", station="KEV", channel="BHZ", location="00"):
    """
    A convenience function for getting the station latitude and longitude from the given station, network,
    channel, and location.

    Returns
    stationLatitude : float
       Station latitude in degrees.
    stationLongitude : float
       Station longitud in degrees.
    """
    # It's much more convenient to wildcard and get ZNE all at once
    stRaw = client.get_waveforms(network, station, location, channel, originTime, originTime+3000) #t, t + 60 * 60)
    # Very awkward - but get the station lat/lon (I don't why obspy isn't lifting this from the miniSEED header)
    inventory = client.get_stations(network=network, station=station,
                                    starttime=originTime, endtime=originTime+400)
    for net in inventory:
        for sta in net:
            stationLatitude = sta.latitude
            stationLongitude = sta.longitude
            return stationLatitude, stationLongitude
    print("Error getting station location")
    return None, None

# Pull data from some puget sound stations
windowStart = 410
windowEnd   = windowStart + 70
nets     = ["UW",  "UW",  "UW",   "UW",   "UW"]
stations = ["GNW", "SP2", "RATT", "DOSE", "TOLT"]
channels = ["BHZ", "BHZ", "BHZ",  "BHZ",  "BHZ"]
locs     = ["--",  "--",  "--",   "--",   "--"]
# Get the data
dataList = []
for k in range(len(nets)):
    # Fetch the waveform data
    st = client.get_waveforms(nets[k], stations[k], locs[k],  channels[k],
                              originTime + windowStart, originTime + windowEnd, attach_response=True)[0]
    # Fetch the station lat/lon
    stla, stlo = getStationLocation(network=nets[k], station=stations[k], channel=channels[k], location=locs[k])
    # Compute the dsitance
    d, az, baz = gps2dist_azimuth(eventLat, eventLon, stla, stlo)
    distDeg = d*1.e-3/111.195
    st.stats.coordinates = {'latitude': stla, 'longitude': stlo, 'distance': distDeg}
    # Append the data to the data list
    dataList.append(st)
# Make an obspy stream
data = Stream(traces=dataList)

```

```python
# Plot the picks and get my attention use an automatic picker
for k in range(len(data)):
    cft = recursive_sta_lta(data[k].data, int(6 * data[k].stats.sampling_rate), int(20* data[k].stats.sampling_rate))
    #plot_trigger(data[k].data, cft, 1.2, 0.5)
    plot_trigger(data[k], cft, 2., 2.2)
```

```python
# Call these the pick times
from numpy import array
# Create an array of observed arrival times.  Recall I'm requesting waveforms windowStart seconds after the
# catalog origin time so I need to add that back in for the full pick time.
observed_P_times  = array([18, 22, 24, 16, 22]) + windowStart - 35.0 # I think NEIC's origin time is screwy
observed_pP_times = array([52, 56, 52, 48, 52]) + windowStart - 35.0 # Or it's possible ObsPy is grabbing waveforms awkwardly
print("The observed P pick times are:", observed_P_times, "seconds")
print("The observed pP pick times are:", observed_pP_times, "seconds")
```

    The observed P pick times are: [ 393.  397.  399.  391.  397.] seconds
    The observed pP pick times are: [ 427.  431.  427.  423.  427.] seconds

```python
def getPhasePickTime(eventLat, eventLon, eventDepth, stationLat, stationLon, phaseName=['P']):
    """
    Convenience function to get the desired phase pick time from the ak135 travel time table.

    Input
    -----
    eventLat : float
       Event latitude in degrees
    eventLon : float
       Event longitude in degrees
    eventDepth : float
       Event depth in degrees
    stationLat : float
       Station latitude in degrees
    stationLon : float
       Station longitude in degrees
    phaseName : list
       Phase name - e.g., ['P'] for a P phase or ['pP'] for a P depth phase.

    Result
    ------
    phaseTime : float
       Theoretical travel time for phase in seconds

    Copyright
    ---------
    Ben Baker distributed under the MIT license.

    """
    # Compute the source/receiver distance in meters
    d, az, baz = gps2dist_azimuth(eventLat, eventLon, stationLat, stationLon)
    # Convert kilometers to degrees where 111.195 is the number of kilometers per degree
    distDeg = d*1.e-3/111.195 # Convert to kilometers then to degrees
    # Compute the travel time for the phase - for example look at Figure 4.20 of Shearer
    pArrival = model.get_travel_times(source_depth_in_km=eventDepth,
                                      distance_in_degree=distDeg,
                                      phase_list=phaseName) #["P"])
    return pArrival[0].time

# This is some debugging code
#k = 0
#for k in range(len(data)):
#    stationLat = data[k].stats.coordinates.latitude
#    stationLon = data[k].stats.coordinates.longitude
#    P_time  = getPhasePickTime(eventLat, eventLon, eventDepth, stationLat, stationLon, phaseName=['P'])
#    pP_time = getPhasePickTime(eventLat, eventLon, eventDepth, stationLat, stationLon, phaseName=['pP'])
#    #print(P_time, pP_time, pP_time - P_time)
```

## Algorithm

Now that we've made observed travel time picks and can make theoretical picks we will tabulate an objective functions for each candidate origin time, $t_0$ and depth $d$ as the residual squared of the observed and estimate travel times for the $P$ phase only
\begin{equation}
  \mathcal{C}_{P}(t_0,d)
 =\sum_{i=1}^{n_{obs}} (P_i^{obs}  - P_i^{est}(d, \Delta_i) + t_0)^2
\end{equation}
and the combined $P$ and $pP$ phases
\begin{equation}
  \mathcal{C}_{pP}(t_0,d)
 =\sum_{i=1}^{n_{obs}} (P_i^{obs}  - P_i^{est}(d, \Delta_i) + t_0)^2
                     + (pP_i^{obs} - pP_i^{est}(d, \Delta_i) + t_0)^2
\end{equation}
Here $i$ is the observation counter and $\Delta_i$ the great-circle distance between the source and i'th observation.

```python
# Now we have the observed the estimate pick times - let's make an objective function.  To do this, let's
# grid search over candidate depths from 80 - 160 and origin times from [-20s, 20s]
nobs = len(data)
nt = 31
nd = 81
relativeOriginTime = linspace(-30, 30, nt)
depths = linspace(60, 180, nd)
# Build up a travel time table at each depth and distance distance
print("Computing travel times...")
P_travelTimeTable = zeros([nd, nobs])
pP_travelTimeTable = zeros([nd, nobs])
for idep in range(nd):
    for iobs in range(nobs):
        # Compute the P theoretical travel time at this event depth/station distance
        stationLat = data[k].stats.coordinates.latitude
        stationLon = data[k].stats.coordinates.longitude
        P_travelTimeTable[idep, iobs] = getPhasePickTime(eventLat, eventLon, depths[idep],
                                                        stationLat, stationLon, phaseName=['P'])
        # Compute the pP theoretical travel time at this event depth/station distance
        pP_travelTimeTable[idep, iobs] = getPhasePickTime(eventLat, eventLon, depths[idep],
                                                         stationLat, stationLon, phaseName=['pP'])
    # Loop on observations
# Loop on depths
print("Tabulating objective function...")
# Now tabulate two objective functions.  The first objective function
sumResidualsSquared_P_phase = zeros([nt, nd])
sumResidualsSquared_pPandP_phase = zeros([nt, nd])
for idep in range(nd):
    for it in range(nt):
        P_estimate  = P_travelTimeTable[idep,:]  + relativeOriginTime[it] # P estimate travel time
        pP_estimate = pP_travelTimeTable[idep,:] + relativeOriginTime[it] # pP estimate travel time
        # Tabulate the residuals
        P_residuals   = observed_P_times  - P_estimate
        pP_residuals  = observed_pP_times - pP_estimate
        # Compute the sum squared of the residuals
        sumResidualsSquared_P_phase[it, idep]      = sum(P_residuals**2)
        sumResidualsSquared_pPandP_phase[it, idep] = sum(P_residuals**2) + sum(pP_residuals**2)
    # Loop on travel times
# Loop on depths
# Compute the square root of the objective functions to make them l2 norms
sumResidualsSquared_P_phase      = 0.5*sqrt(sumResidualsSquared_P_phase)
sumResidualsSquared_pPandP_phase = 0.5*sqrt(sumResidualsSquared_pPandP_phase)
```

    Computing travel times...
    Tabulating objective function...

```python
# Plot the results
[X, Y] = meshgrid(relativeOriginTime, depths)
extent = [min(relativeOriginTime), max(relativeOriginTime), min(depths), max(depths)]
im = plt.imshow(sumResidualsSquared_P_phase, interpolation='bilinear', cmap=cm.viridis,
                extent=extent, aspect=1./2., origin='lower')
plt.title("P Phase Objective Function")
plt.xlabel('Relative Origin Time (s)')
plt.ylabel('Depths (km)')
plt.show()

im = plt.imshow(sumResidualsSquared_pPandP_phase, interpolation='bilinear', cmap=cm.viridis,
                extent=extent, aspect=1/2., origin='lower')
plt.title("pP and P Phase Objective Function")
plt.xlabel('Relative Origin Time (s)')
plt.ylabel('Depths (km)')
plt.show()

```

## Question

In this color scheme dark-blue is good and bright yellow is bad.  Which plot shows a clear trade-off with origin time and depth?

## Question

If the hypocenter of an earthquake observed at teleseismic distances only included $P$ primary phases in the inversion - would you have high-confidence in the recovered depth parameter?

## Question

At local distances we see the same trade-off with origin time and depth.  Can you identify a phase in a regional network like the PNSN that would help to resolve this depth/origin time ambiguity?  Note that all-phases leave upwards from the source so depth-phases are not a valid option.

```python

```


Energy Spectrum of Prompt Fission Neutrons
=======================

The energy spectrum of the prompt fission neutrons, which we'll call as the
prompt fission spectrum for simplicity, represents the probability of neutrons
of different energies to be born in a fission event — a very important
parameter.

This quantity, usually denoted by $\chi_p$, is a probability distribution.
So let's check some simple approaches to it.

```python
import numpy as np
from math import pi, sqrt
import matplotlib as mpl
import matplotlib.pyplot as plt
import sympy as sym
from pyne import ace
from sympy.plotting import plot

mpl.style.use('seaborn')
```

We'll define 2 common distributions used to represent the fission spectrum:
the Maxwell distribution and the Watt distribution.
At fission spectrum energies we are quite far from thermal equilibrium, so both are fittings.
We'll talk just briefly about the modern theory in the end.

The Maxwell distribution **in energy** is of the form:

$ P(E') = \frac{2\pi}{(\pi kT)^{3/2}} \sqrt{E'} \ \mathrm{e}^{\frac{-E'}{kT}} $

For convenience, we can separate the coefficients in such a way:

$ P(E') = c_1 \sqrt{E'} \ \mathrm{e}^{-c_2 E'} $; where:

$ c_1(kT) = \frac{2\pi}{(\pi kT)^{3/2}} $ and $ c_2(kT) = {\frac{1}{kT}} $

In these coefficients $ k $ is the Boltzmann constant and $ T $ is the absolute temperature, however it is convenient to
leave the product $ kT $, which is a measure of temperature in units of energy, because $ T $ itself is unknown, but
$ kT $ is a quantity called "nuclear temperature" (also found as $ T(E) $), which is tabulated for incoming neutron
energies on ENDF files.

We can find the approximate Prompt Fission Neutron Spectrum (PFNS), represented by $ \chi_p(E') $, taking
$ kT = 1.2887 $ MeV, giving a $ c_1 = 0.771 $ and $ c_2 = 0.776 $.

$ \chi_p(E') = 0.771 \sqrt{E'} \ \mathrm{e}^{-0.776E'} $

```python

def maxwell(c1, c2):
    P = c1 * sym.sqrt(Ep) * sym.exp(-c2 * Ep)
    return P

def maxwell_coe(kT):
    c1 = 2*pi / pow(pi*kT, 3/2)
    c2 = 1/kT
    return c1, c2

xmin = 0
xmax = 10
points = 10000000
fig_size = (5,3)

xlabel = "$E'$ (MeV)"
ylabel = "$ \chi_{\mathrm{p}} (\mathrm{MeV}^{-1}) $"
```

Let's start by checking how the Maxwell distribution does.

```python
kT = 1.2887 # MeV

print("Coefficients c1 and c2 are:", maxwell_coe(kT))

Ep = sym.symbols('Ep')

maxwell_chi = maxwell(*maxwell_coe(kT))
maxwell_lin = plot(maxwell_chi, (Ep, xmin, xmax),
                   adaptive = False,
                   nb_of_points = points,
                   size= fig_size,
                   xlabel=xlabel,
                   ylabel=ylabel,
                   axis_center=(-0.5,-0.01),
                   show=False)

maxwell_lin.save('chi_maxwell.pdf')
```

```python
x_maxwellPFNS, y_maxwellPFNS = maxwell_lin._series[0].get_points()

fig1, ax1 = plt.subplots(figsize=fig_size)
# ax1.set_xscale('log')
ax1.semilogx(x_maxwellPFNS, y_maxwellPFNS, label='Maxwell PFNS')
ax1.set_xlabel(xlabel)
ax1.set_ylabel(ylabel)

plt.savefig('chi_maxwell_logx.pdf', bbox_inches='tight')

fig2, ax2 = plt.subplots(figsize=fig_size)
ax2.semilogy(x_maxwellPFNS, y_maxwellPFNS, label='Maxwell PFNS')
ax2.set_xlabel(xlabel)
ax2.set_ylabel(ylabel)

plt.savefig('chi_maxwell_logy.pdf', bbox_inches='tight')

fig3, ax3 = plt.subplots(figsize=fig_size)
ax3.loglog(x_maxwellPFNS, y_maxwellPFNS, label='Maxwell PFNS')
ax3.set_xlabel(xlabel)
ax3.set_ylabel(ylabel)
ax3.set_xlim(1E-4, 10)
ax3.set_ylim(0.005, 1)

plt.savefig('chi_maxwell_logxy.pdf', bbox_inches='tight')
```

The Maxwell distribution is unimodal (i.e., has only one mode), so when we
calculate the mode, we can simply pick the only value in the list.
We can find this mode as one would expect: take the derivative and solve for the 0s.

```python
def dist_modes(f_x, x):
    fp_x = sym.diff(f_x, x)
    modes = sym.solve(fp_x, x)

    return modes

# modes = dist_modes(maxwell_chi, Ep)
#
# max_Ep = modes[0]
# max_chi = maxwell_chi.subs(Ep, max_Ep)
#
# print('Most probable fission promp neutron energy: ', max_Ep)
# print('With probability of: ', max_chi)
```

With this, we found that the most probable prompt neutron from fission has
energy of around 0.64 MeV. (at least according to Maxwell)

But for such a skewed distribution, it is quite obvious that the average energy
of the prompt neutrons $ \bar{E'} $ is quite different.
We can find it by calculating the mean of the distribution.

$ \bar{E'} = \frac{\int_{0}^{\infty} E'\  \chi(E') dE'}{\int_{0}^{\infty} \chi(E') dE'} $

```python
def dist_mean(f_x, x):
    interval = (x, 0, sym.oo)

    num = sym.integrate(f_x*x, interval)
    den = sym.integrate(f_x, interval)

    return num / den

mean_Ep = dist_mean(maxwell_chi, Ep)
mean_chi = maxwell_chi.subs(Ep, mean_Ep)

print('Fission promp neutron mean energy: ', mean_Ep)
print('With probability of : ', mean_chi)
```

    Fission promp neutron mean energy:  1.93305000000000
    With probability of :  0.239280406306773

So, according to the Maxwellian distribution, the average fission prompt
neutron energy is around 1.93 MeV.
Usually, the value is simply rounded and accepted to be around 2 MeV.

Now let's repeat everything for the Watt distribution and see how it does.
The Watt's distribution is empirical, of the form:

$ \chi_p(E') = c \sinh{\sqrt{b(E)\ E'}} \ \mathrm{e}^{\frac{-E'}{a(E)}} $

where $E$ is the energy of the incoming neutron (the neutron that causes
fission) and $E'$ is the energy of the outgoing prompt fission neutron and
the parameters $ a b c $ are tabulated, and $ a $ seems to represent something
like "nuclear temperature" as in the Maxwellian distribution.

The idea is that it approximates the Maxwellian spectrum corrected for the
transformation from the center-of-mass system to the laboratory system.

We can already expect the Watt function to perform a bit better, if for no
other reason than by having more tuning parameters (assuming that the functions
were well-chosen — and they were)

```python
def watt_function(c, b, a):
    P = c * sym.sinh(sym.sqrt(b * Ep)) * sym.exp(-Ep/a)
    return P
```

```python
# I actually forgot where I got these, but I know that they are tabulated on ENDF
c = 0.4865
b = 2
a = 1

watt_chi = watt_function(c, b, a)
watt_lin = plot(watt_chi, (Ep, xmin, xmax),
                adaptive = False,
                nb_of_points = points,
                size=fig_size,
                xlabel=xlabel,
                ylabel=ylabel,
                axis_center=(-0.5,-0.01),
                show=False)

watt_lin.save ('chi_watt.pdf')

# modes = dist_modes(watt_chi, Ep)
#
# max_Ep = modes[0]
# max_chi = watt_chi.subs(Ep, max_Ep)

# print('Most probable fission promp neutron energy: ', max_Ep)
# print('With probability of: ', max_chi)

mean_Ep = dist_mean(watt_chi, Ep)
mean_chi = watt_chi.evalf(subs={Ep: mean_Ep})

print('Fission promp neutron mean energy: ', mean_Ep)
print('With probability of : ', mean_chi)
```

As an empirical formulation, the Watt distribution nails the mean fission prompt
neutron energy at 2 MeV.
Naturally, that happens because the parameters were carefully chosen.

Now let's plot both distributions together to see how they differ qualitatively.

```python
x_wattPFNS, y_wattPFNS = watt_lin._series[0].get_points()

fig, ax = plt.subplots(figsize=fig_size)
ax.plot(x_maxwellPFNS, y_maxwellPFNS, label='Maxwell PFNS')
ax.plot(x_wattPFNS, y_wattPFNS, label='Watt PFNS')
ax.legend()
ax.set_xlabel(xlabel)
ax.set_ylabel(ylabel)

plt.savefig('maxwell_watt_comparison.pdf', bbox_inches='tight')

fig, ax = plt.subplots(figsize=fig_size)
ax.semilogy(x_maxwellPFNS, y_maxwellPFNS, label='Maxwell PFNS')
ax.plot(x_wattPFNS, y_wattPFNS, label='Watt PFNS')
ax.legend()
ax.set_xlabel(xlabel)
ax.set_ylabel(ylabel)

plt.savefig('maxwell_watt_comparison_logy.pdf', bbox_inches='tight')
```

We can also plot using a log scale on the energy axis to investigate emission
of neutrons with lower energy.
The distribution shows that some do appear, but the vast majority is clearly on
the MeV range.
Considering that fission cross-sections become much bigger in the eV range, we
can develop intuitively the idea of moderation.

```python
fig, ax = plt.subplots(figsize=fig_size)
ax.set_xscale('log')
ax.plot(x_maxwellPFNS, y_maxwellPFNS, label='Maxwell PFNS')
ax.plot(x_wattPFNS, y_wattPFNS, label='Watt PFNS')
ax.legend()
ax.set_xlabel(xlabel)
ax.set_ylabel(ylabel)

plt.savefig('maxwell_watt_comparison_logx.pdf', bbox_inches='tight')

ymin = 8E-4

fig, ax = plt.subplots(figsize=fig_size)
ax.loglog(x_maxwellPFNS, y_maxwellPFNS, label='Maxwell PFNS')
ax.plot(x_wattPFNS, y_wattPFNS, label='Watt PFNS')
ax.legend()
ax.set_xlabel(xlabel)
ax.set_ylabel(ylabel)
ax.set_ylim(ymin=ymin)

plt.savefig('maxwell_watt_comparison_logxy.pdf', bbox_inches='tight')
```

The modern take on fission spectrum of prompt neutrons is based on the
Madland-Nix Spectrum.
This model is based on the standard nuclear evaporation theory and utilizes
an isospin-dependent optical potential for the inverse process of compound
nucleus formation in neutron-rich fission fragments.
Quite impressive keywords there you have to admit!
The intention of this elaborate model is to represent the fission products in
full acceleration in the laboratory system in order to get the right spectrum
for the neutrons that evaporate from the primary fission products.

This is enough about fission spectrum for our purposes.
It allows us to know that there are different models available, with different
accuracies and sophistication behind.

But more importantly, now you know that the mean fission neutron energy is
around 2 MeV, so now when we plot some other figures, you know where the
fission neutron is located.

Below, we do even better by plotting the PFNS with the fission cross-section
for U-235, which shows how the overlap between both is minimal.

```python
xlims = (1E-9, 10)
ylims = np.array([8E-1, 4E3])

xlabel_xs = 'Neutron energy (MeV)'
ylabel_xs = '$ \sigma_{\mathrm{f}} $ (barn)'

U235 = ace.Library("/Users/rodrigo/opt/Serpent2/xsdata/jeff311/acedata/92235JEF311.ace")
U235.read()
U235_12 = U235.tables['92235.12c']

fig, ax = plt.subplots()
ax.loglog(U235_12.energy, U235_12.reactions[18].sigma,
          linewidth=0.5,
          label='U235(n,f)')
ax.set_xlim(*xlims)
ax.set_ylim(*ylims)
ax.set(xlabel=xlabel_xs,
       ylabel=ylabel_xs,
       title='U-235 fission XS at 1200 K')

h1, l1 = ax.get_legend_handles_labels()

plt.savefig('U235_1200K_MT18.pdf', bbox_inches='tight')

ax.set(title='U-235 fission XS at 1200 K and Watt PFNS')

ax2 = ax.twinx()
ax2.loglog(x_wattPFNS, y_wattPFNS, color='red', label='Watt PFNS')
ax2.set_ylabel(ylabel)
ax2.set_ylim(*ylims/1E3)
ax2.grid(False)

h2, l2 = ax2.get_legend_handles_labels()

plt.legend(h1+h2, l1+l2)

plt.savefig('U235_1200K_MT18_chip.pdf', bbox_inches='tight')
```

In order to improve the chance of fission, we would like to shift the neutron
spectrum to somewhere on the eV range, which is the idea of adding a moderator
to the reactors.

So we'll check that idea here by studying energy and speed at reactor operating
temperatures.

```python
#From NIST in 14/2/2022
k_B = 8.617333262E-5 # eV/K
kB_J = 1.380649E-23 # J/K

neutron_mass = 1.67492749804E-27 # kg
eV2Joule = 1.602176565E-19

def E_mean(T):
    return 3/2 * k_B * T

def particle_speed(m, E):
    return sqrt(2*E/m)

def report_neutron_data(T):
    E = T*k_B
    v = particle_speed(neutron_mass, E*eV2Joule)
    print(f"{T} K represents", E, "eV")
    print(f"Neutron speed", v, "m/s")

    Em = E_mean(T)
    vm = particle_speed(neutron_mass, Em*eV2Joule)
    print(f"Mean energy", Em, "eV")
    print(f"Mean speed:", vm, "m/s\n")

# test energy and idea
report_neutron_data(300)
report_neutron_data(600)
```

    300 K represents 0.025851999786 eV
    Neutron speed 2223.920462567311 m/s
    Mean energy 0.038777999679 eV
    Mean speed: 2723.735180912124 m/s

    600 K represents 0.051703999572 eV
    Neutron speed 3145.098479801738 m/s
    Mean energy 0.077555999358 eV
    Mean speed: 3851.9432331586618 m/s

So we can see that with a moderator at 300 K (+- room temperature) we have a neutron
energy of just around 0.025 eV, which intuitively already looks promising if we look
at the cross-section values.

But we can reach better conclusions by plotting the Maxwellian spectrum at this
temperature and see what happens. Unlikk when we wanted to calculate the PFNS, which
required a tabulated "nuclear temperature", here we can calculate everything from the
desired temperature.

```python
T1 = 300
T2 = 600 # reactor coolant temperature in K

kT1_thermal = T1*k_B # eV
kT2_thermal = T2*k_B

print(f"Coefficients c1 and c2 at {T1} K:", maxwell_coe(kT1_thermal))
print(f"Coefficients c1 and c2 at {T2} K:", maxwell_coe(kT2_thermal))
maxwell_T1 = maxwell(*maxwell_coe(kT1_thermal))
maxwell_T2 = maxwell(*maxwell_coe(kT2_thermal))

xlabel_thermal = "$E'$ (eV)"
ylabel_thermal = "Spectrum $(\mathrm{eV}^{-1})$"
maxwell_T_lin = plot(maxwell_T1, maxwell_T2, (Ep, 1E-9, 1),
                     adaptive = False,
                     nb_of_points = 1000000,
                     size= fig_size,
                     xlabel=xlabel_thermal,
                     ylabel=ylabel_thermal,
                     axis_center=(-0.04,-0.3),
                     show=False)

x_maxwellT1, y_maxwellT1 = maxwell_T_lin._series[0].get_points()
x_maxwellT2, y_maxwellT2 = maxwell_T_lin._series[1].get_points()

fig, ax = plt.subplots(figsize=fig_size)
ax.plot(x_maxwellT1, y_maxwellT1, label=f'{T1} K')
ax.plot(x_maxwellT2, y_maxwellT2, label=f'{T2} K')
ax.set_xlabel(xlabel_thermal)
ax.set_ylabel(ylabel_thermal)
ax.set_xlim(-0.01, 0.4)
ax.legend()

plt.savefig('maxwell_thermal.pdf', bbox_inches='tight')

fig, ax = plt.subplots(figsize=fig_size)
ax.semilogx(x_maxwellT1, y_maxwellT1, label=f'{T1} K')
ax.semilogx(x_maxwellT2, y_maxwellT2, label=f'{T2} K')
ax.set_xlabel(xlabel_thermal)
ax.set_ylabel(ylabel_thermal)
ax.set_xlim(1E-6, 1)
ax.legend()

plt.savefig('maxwell_thermal_logx.pdf', bbox_inches='tight')

fig, ax = plt.subplots(figsize=fig_size)
ax.loglog(x_maxwellT1, y_maxwellT1, label=f'{T1} K')
ax.loglog(x_maxwellT2, y_maxwellT2, label=f'{T2} K')
ax.set_xlabel(xlabel_thermal)
ax.set_ylabel(ylabel_thermal)
ax.set_xlim(1E-6, 1)
ax.set_ylim(1E-4, 50)
ax.legend()

plt.savefig('maxwell_thermal_logxy.pdf', bbox_inches='tight')
```

The distributions appear to be in the right place. We can put that together with
the fission cross-section and PFNS and observe that we shifted the spectrum to
the left of the resonance range.

This is much better from the point of view of maximizing fission as long as we can
achieve that effectively, which means that we have to reduce the neutron energy as
fast as possible to avoid the resonance range that is in between the PFNS and the
Maxwellian distributions.

```python
x_maxwellT1 = x_maxwellT1*10E-6 # shift x from eV to MeV
x_maxwellT2 = x_maxwellT2*10E-6

y_maxwellT1 = y_maxwellT1/10E-6 # shift x from eV to MeV
y_maxwellT2 = y_maxwellT2/10E-6
```

```python
fig, ax = plt.subplots()
ax.loglog(U235_12.energy, U235_12.reactions[18].sigma,
          linewidth=0.5,
          label='U235(n,f)')

ax.set(xlabel='Neutron energy (MeV)',
       ylabel='$ \sigma_{\mathrm{f}} $ (barn)',
       title='U-235 fission XS at 1200 K with PFNS and thermal spectra')
ax.set_xlim(*xlims)
ax.set_ylim(*ylims)
h1, l1 = ax.get_legend_handles_labels()

ax2 = ax.twinx()
ax2.grid(False)
ax2.set_ylabel(ylabel)
ax2.set_ylim(*ylims/1E3)
ax2.loglog(x_wattPFNS, y_wattPFNS, color='red', label='Watt PFNS')

h2, l2 = ax2.get_legend_handles_labels()

ax3 = ax.twinx()
ax3.spines['right'].set_position(('outward', 40))
ax3.grid(False)
ax3.set_ylabel("Spectrum $(\mathrm{MeV}^{-1})$")
ax3.set_ylim(*ylims*1E3)
ax3.loglog(x_maxwellT1, y_maxwellT1, color='green', label=f'Maxwell {T1} K')

h3, l3 = ax3.get_legend_handles_labels()
plt.legend(h1+h2+h3, l1+l2+l3)

plt.savefig('U235_1200K_MT18_chip_maxwellT1.pdf', bbox_inches='tight')

ax3.loglog(x_maxwellT2, y_maxwellT2, color='orange', label=f'Maxwell {T2} K')
h3, l3 = ax3.get_legend_handles_labels()
plt.legend(h1+h2+h3, l1+l2+l3)

plt.savefig('U235_1200K_MT18_chip_maxwellT12.pdf', bbox_inches='tight')
```