```python
### Requirements: PyDotPlus, Matplotlib, Scikit-Learn, Pandas, Numpy, IPython (and possibly GraphViz)

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.preprocessing import Imputer
from sklearn import preprocessing
from sklearn.tree import DecisionTreeRegressor, DecisionTreeClassifier
from sklearn.ensemble import AdaBoostClassifier
from sklearn.ensemble import BaggingRegressor
from sklearn.ensemble import BaggingClassifier
from sklearn import tree
import sklearn
import sklearn.metrics as skm
from scipy import misc
from sklearn.externals.six import StringIO
import pydotplus
from IPython.display import Image, YouTubeVideo

def visualize_tree(tree, feature_names, class_names):
    dot_data = StringIO()
    sklearn.tree.export_graphviz(tree, out_file=dot_data,
                         filled=True, rounded=True,
                         feature_names=feature_names,
                         class_names=class_names,
                         special_characters=True)
    graph = pydotplus.graph_from_dot_data(dot_data.getvalue())
    return graph.create_png()
```

# EECS 445: Machine Learning
## Lecture 12: Bagging and Boosting
- Instructor: **Jacob Abernethy**
- Date: October 19, 2016

Lecture Exposition Credit: Valliappa Chockalingam

## Announcements

- **Midterm** in class on Monday, October 24
- Come to the section in which *you are enrolled*
- You can bring notes, no more than 3 pieces of paper (double sided)
- Bring an ID. No calculators allowed
- Topics: lectures 1-11, see Piazza post for details
- Note: **No Lecture on Wednesday October 26**

## Quick review of information gain and decision trees

### Metrics: Information Gain (Mutual Information)
#### Used by the ID3, C4.5 and C5.0 tree-generation algorithms.

Assume the true binary labels $\{y_i : i =1\ldots m\}$ are distribution according to $P(y)$. But when we observe the value of a decision stump $A = T\text{ or }F$, then we obtain two new distributions, $P(y \mid a = T)$ and $P(y \mid a = F)$. We use *information gain* to measure how much the distribution on $y$ changes when we observe $a$.

\begin{align*}
\text{Information Gain } & = \text{ Entropy(Parent) - } \text{ Weighted Sum of Entropy(Children)} \\
 & = IG(P,A) = H(P) - H(P|A) \\
 & = H(P) - \sum_{A = T,F} Pr(A) H(P(\cdot | A))
\end{align*}

### A question to illustrate the Information Gain metric

Note:
 - The $[x+, y-]$ indicate the number of samples belonging to the two classes, say positive and negative.
 - The topmost one denotes the number of positive and negative samples in the dataset before "any partitioning."

## Good Notes on Information Gain and DTs

[See thse notes from CMU](http://www.cs.cmu.edu/~awm/10701/slides/DTreesAndOverfitting-9-13-05.pdf)

### So, what is Information Gain?

- Intuitively, Information Gain captures:
 - The mutual information that there is between an attribute and the class labels, or, equivalently,
 - The reduction in entropy gained by observing an attribute.
- Another intersting note:
 - Mutual information (i.e., Information Gain) and KL-divergence are connected: $IG(X, Y) = D_{KL}(p(x, y) \mid \mid p(x)p(y))$.

### Alternative: Misclassification Error

If we think of a node $A$ as simply "guessing" $A(x)$ for the label $y$ of $x$, then the Misclassification Error (ME) is essentially the probability $P(A(x) = y)$

$$\text{ME}(A) = \frac{\sum_{i=1}^N \mathbb{I}[ A(x_i) = y_i ] }{N}$$

- Often, we imagine that the misclassification error of a node is either the error rate of $A$ **or** the error rate of $\neg A$. We might call this $\text{ME}^*$.

$$\text{ME}^*(A) = \min(\text{ME}(A), \text{ME}(\neg A))$$

### Generalizations of Decision Trees (1)
- Decision Trees in their simplest form involve:
 - A 2-class scenario, with,
 - Binary Features, and use a,
 - Binary Tree (every node has at most 2 children).

However, generalizations are possible.

### Generalizations of Decision Trees (2)
- Categorical Features
 - Can use $n$-ary trees rather than binary search trees
   - Fruits Tree Example: Color? asked at the root had three choices: Red, Yellow or Green (Ternary Tree)
 - Can use questions such as "$x_i$ = $l$?" or even $"2 \leq x_i \leq l$?", where $x_i \in \{1, ..., l, ..., K\}$?
   - The Mushroom Classification Tree was an example of this (Binary Tree with comparative rather than only equality checking conditions).

### Generalizations of Decision Trees (3)
- Categorical Output/Multiclass Scenario (the Fruits Example)
 - One way to handle categorical outputs is to pose a question such as "Is it a bird?" when "Yes" is expected. This keeps the outputs binary.
 - Another way is to simply use a **one-hot encoding** for the output (Ex: Bird corresponds to $[0, 0, 0, 0, 0, 1, 0]$)
- Real-Valued Response (Output). Decision Trees are typically designed for binary problems, i.e. classification. But they can be used for regression! However, applying DTs for regression usually involves discretizing the output space either in some way.
- [More on DTs for classification and regression](http://machinelearningmastery.com/classification-and-regression-trees-for-machine-learning/)

### Decision Tree Computation Limitations

Decision Trees in general perform well with lots of data, are robust to violations of assumptions, and probably most strikingly are easy to understand and interpret. However:
 - The problem of Learning an optimal Decision Trees is NP-Complete under several definitions of optimal.
  - Standard algs are "greedy", make myopic decisions that may not be globally optimal.
 - There are concepts that are hard to learn using Decision Trees, which are also generally hard for other linear classifiers, as the Decision Tree learned is prohibitively large. These include "toy" problems such as:
  - XOR, Parity or Multiplexer based Problems

## Decision Tree overfitting

With a decision tree it is easy to overfit your data!

## Decision Tree overfitting

We need to control the "complexity" of the hypothesis

One straightforward way to do this: limit the depth of the tree!

### Inductive Bias

- What is the policy by which a particular decision tree algorithm generalizes from observed training examples to classify unseen instances?

- ***Definition:*** The set of assumptions that, together with the training data, deductively justify the classifications assigned by the learner to future instances.

- We can also think of this bias as an algorithm's "preference" over possibly hypotheses.

[More here](https://en.wikipedia.org/wiki/Inductive_bias)

### Inductive Bias of Decision Tree Algorithms

- When a decision tree is built, it is almost always not the only tree that will perfectly classify the training set!
- Finding the inductive bias of a decision tree algorithm is basically trying to find the type of tree that the algorithm favors in general.
- It turns out that two of the common decision tree algorithms (ID3 and C4.5) have the same approximate inductive bias:
 - Prefers shorter trees over larger trees, and,
 - Trees that place high information gain attributes close to the root over those that do not.

### Occam's Razor: "the simpler the better."

- If a larger tree classifies the training set just as well as the shorter hypothesis, then we would logically want to use the shorter tree as it performs the same, is smaller, and is quicker to build.
- But is it always the case that shorter, more simpler hypotheses are preferred over larger ones?
- Occam's razor is a heuristic, that we should be biased in preferring simpler hypotheses to complex ones. But this idea is in some sense a foundational principle in Machine Learning.

### Dealing with Overfitting

- Simple Method: Grow the tree and check error iteratively, stop at a point where error rate is satisfactory or tree has reached some limit.
- **Pruning**: Construct a large decision tree. Use a method such as cross-validation and prune nodes from the leaves upwards. If removing a node does not change performance, make the change permanent. This can also be done for entire subtrees.
- Use Ensemble Methods!

### Ensemble Methods
- In supervised ML, our goal is to find a hypothesis that performs well on **unseen data**.
- Finding a single hypothesis within the hypothesis space that gives good predictions can be hard.
- Idea of Ensemble Methods: "Combine" multiple hypotheses to form a (hopefully) better hypothesis.
- The notion of "combine" is important, and we'll discuss this
- ***Note***: The hypothesis represented by an Ensemble Model is not necessarily contained within the hypothesis space of the constituent models!

### Bagging (**B**ootstrap **Agg**regat**ing**)

- Given a datset $\mathcal{D}$, $|\mathcal{D}| = n$.
- Create multiple ***bootstrap samples*** $\mathcal{D}^{'}_i, i \in \{1, ..., m\}$ such that $\forall i, |\mathcal{D}^{'}_i| = n^{'}$ using ***sampling with replacement.***
- Fit $m$ models using the above $m$ bootstrap samples
 - ***Note:*** No pruning or stopping is used. Bagging helps when the models are unstable and can hurt if they are not.
- Given a new input $\mathbf{x}$, run each of the $m$ classifiers and use a **majority vote** to classify $\mathbf{x}$.

***Note:*** Bagging can also be applied for regression but instead of using majority vote, the average is used.

### Why does Bagging make sense / What's the intuition?

- With one large Decision Tree (or more generally a single complex hypothesis), the model will likely have Low Bias and High Variance (Overfits to the "random noise" in the data).
- (Large) Decision Trees are unstable (using slightly different datasets causes a big change in the model learned).
- So, once we train multiple Decision Trees (or in general, multiple unstable ML models), with the bootstrap samples, we can get a much more stable model that performs better by say, using majority voting.

```python
%matplotlib inline

# Author: Gilles Louppe <g.louppe@gmail.com>
# License: BSD 3 clause

# Settings
n_repeat = 50       # Number of iterations for computing expectations
n_train = 50        # Size of the training set
n_test = 1000       # Size of the test set
noise = 0.1         # Standard deviation of the noise
np.random.seed(0)

# Change this for exploring the bias-variance decomposition of other
# estimators. This should work well for estimators with high variance (e.g.,
# decision trees or KNN), but poorly for estimators with low variance (e.g.,
# linear models).
estimators = [("Tree", DecisionTreeRegressor()),
              ("Bagging(Tree)", BaggingRegressor(DecisionTreeRegressor()))]

n_estimators = len(estimators)

# Generate data
def f(x):
    x = x.ravel()

    return np.exp(-x ** 2) + 1.5 * np.exp(-(x - 2) ** 2)

def generate(n_samples, noise, n_repeat=1):
    X = np.random.rand(n_samples) * 10 - 5
    X = np.sort(X)

    if n_repeat == 1:
        y = f(X) + np.random.normal(0.0, noise, n_samples)
    else:
        y = np.zeros((n_samples, n_repeat))

        for i in range(n_repeat):
            y[:, i] = f(X) + np.random.normal(0.0, noise, n_samples)

    X = X.reshape((n_samples, 1))

    return X, y

def bias_variance_example():
    X_train = []
    y_train = []

    for i in range(n_repeat):
        X, y = generate(n_samples=n_train, noise=noise)
        X_train.append(X)
        y_train.append(y)

    X_test, y_test = generate(n_samples=n_test, noise=noise, n_repeat=n_repeat)

    # Loop over estimators to compare
    for n, (name, estimator) in enumerate(estimators):
        # Compute predictions
        y_predict = np.zeros((n_test, n_repeat))

        for i in range(n_repeat):
            estimator.fit(X_train[i], y_train[i])
            y_predict[:, i] = estimator.predict(X_test)

        # Bias^2 + Variance + Noise decomposition of the mean squared error
        y_error = np.zeros(n_test)

        for i in range(n_repeat):
            for j in range(n_repeat):
                y_error += (y_test[:, j] - y_predict[:, i]) ** 2

        y_error /= (n_repeat * n_repeat)

        y_noise = np.var(y_test, axis=1)
        y_bias = (f(X_test) - np.mean(y_predict, axis=1)) ** 2
        y_var = np.var(y_predict, axis=1)

        print("{0}: {1:.4f} (error) = {2:.4f} (bias^2) "
              " + {3:.4f} (var) + {4:.4f} (noise)".format(name,
                                                          np.mean(y_error),
                                                          np.mean(y_bias),
                                                          np.mean(y_var),
                                                          np.mean(y_noise)))

        # Plot figures
        from pylab import rcParams
        rcParams['figure.figsize'] = 9, 9
        plt.subplot(2, n_estimators, n + 1)
        plt.plot(X_test, f(X_test), "b", label="$f(x)$")
        plt.plot(X_train[0], y_train[0], ".b", label="LS ~ $y = f(x)+noise$")

        for i in range(n_repeat):
            if i == 0:
                plt.plot(X_test, y_predict[:, i], "r", label="$\^y(x)$")
            else:
                plt.plot(X_test, y_predict[:, i], "r", alpha=0.05)

        plt.plot(X_test, np.mean(y_predict, axis=1), "c",
                 label="$\mathbb{E}_{LS} \^y(x)$")

        plt.xlim([-5, 5])
        plt.title(name)

        if n == 0:
            plt.legend(loc="upper left", prop={"size": 11})

        plt.subplot(2, n_estimators, n_estimators + n + 1)
        plt.plot(X_test, y_error, "r", label="$error(x)$")
        plt.plot(X_test, y_bias, "b", label="$bias^2(x)$"),
        plt.plot(X_test, y_var, "g", label="$variance(x)$"),
        plt.plot(X_test, y_noise, "c", label="$noise(x)$")

        plt.xlim([-5, 5])
        plt.ylim([0, 0.1])

        if n == 0:
            plt.legend(loc="upper left", prop={"size": 11})
    plt.show()
```

```python
# Bias-Variance of Bagging with
# Decision Tree Regressors Illustration (Adapted from ELSII, 2009)
# (Note: LS refers to a bootstrap sample)
bias_variance_example()
```

### Informal Bias-Variance Reasoning when using Bagging

- In general, the Bias remains about the same as we are performing model averaging and as long as the bootstrap samples represent the dataset well, the bias stays about the same.
- Variance reduces by a factor of at most the size of the bootstrap samples ($n^{'}$).
- In reality, bagging reduces variance (often by less than a factor of $n^{'}$) and tends to slightly increase bias.

### Random Forests

- Extends Bagging and in practice performs generally better.
- The only difference: When constructing the trees, best splits are found on only **a subset of the features**, not all.
 - Rule of thumb: $m = \sqrt{p}$ (rounded down) is recommended for classification problems and $\frac{p}{3}$ (rounded down) is recommended for regression where $p$ is the number of features.
- For each tree grown on a bootstrap sample, we can measure the error rate on a test set.
 - This is called the "out-of-bag" error rate.
 - This can be regarded as a generalization error and can provide a ranking of the importance of features.

## Random Forests work **very** well in practice

Many ML competitions are won using some version of Random Forest. Hard to overstate the value of this algorithm on real-world problems.

### Limitations of Bagging

- Loss of interpretability: the final bagged classifier is not a tree, and so we forfeit the clear interpretative ability of a classification tree.

- Computational complexity: we are essentially multiplying the work of growing a single tree by $m$. Can be a lot of work!

## Break time!

### Boosting

- Bagging can help in reducing Variance. Are there methods that reduce both Bias and Variance? Yes! Boosting is one of them.
- General Ideas:
 - Weighted Majority Vote (unlike Bagging)
 - Elements of Ensemble built Sequentially (unlike Bagging where the models could be built parallely)

### Boosting (2 Class Scenario)

- Assume class labels are -1 and +1.
- The final classifier then has the form:
 - $h_T(\mathbf{x}) = \text{sgn}\left(\sum \limits_{t = 1}^T \alpha_t f_t(\mathbf{x})\right)$ where $f_1, ..., f_T$ are called base (or weak) classifiers and $\alpha_1, ..., \alpha_T > 0$ reflect the confidence of the various base classifiers.

### Base/Weak Learners

- Let $(\mathbf{x}_1, y_1), ..., (\mathbf{x}_n, y_n)$ be the training data.
- Let $\mathscr{F}$ be a fixed set of classifiers called the base class.
- A base learner for $\mathscr{F}$ is a rule that takes as input a set of weights $\mathbf{w} = (w_1, ..., w_n)$ such that $w_i \geq 0, \sum w_i = 1$, and outputs a classifier $f \in \mathscr{F}$ such that the weighted empirical risk $$e_w(f) = \sum \limits_{i = 1}^n w_i \mathbb{1}_{\{f(\mathbf{x}_i) \neq y_i\}}$$ is (approximately) minimized.

### Examples of Base (Weak) Learners

- Decision Stumps, i.e., decision trees with depth 1
- Decision Trees
- Polynomial thresholds, i.e., $$f(\vec{x}) = \pm \text{sign}((\vec{w}^\top \vec{x})^2 - b)$$ where $b \in \mathbb{R}$ and $\vec{w} \in \mathbb{R}^d$ is a radial kernel.

### AdaBoost (Adaptive Boosting)

- The first concrete algorithm to successfully realize the boosting principle.

### AdaBoost Algorithm

An *iterative* algorithm for "ensembling" base learners

- Input: $\{(\mathbf{x}_i, y_i)\}_{i = 1}^n, T, \mathscr{F}$, base learner
- Initialize: $\mathbf{w}^{1} = (\frac{1}{n}, ..., \frac{1}{n})$
- For $t = 1, ..., T$
 - $\mathbf{w}^{t} \rightarrow \boxed{\text{base learner}} \rightarrow f_t$
 - $\alpha_t = \frac{1}{2}\text{ln}\left(\frac{1 - r_t}{r_t}\right)$
     - where $r_t := e_{\mathbf{w}^t}(f_t) = \frac 1 n \sum \limits_{i = 1}^n \mathbf{w}_i \mathbf{1}_{\{f(\mathbf{x}_i) \neq y_i\}} $
 - $w_i^{t + 1} = \frac{\mathbf{w}_i^t \exp \left(- \alpha_ty_if_t(\mathbf{x}_i)\right)}{z_t}$ where $z_t$ normalizes.
- Output: $h_T(\mathbf{x}) = \text{sign}\left(\sum \limits_{t = 1}^T \alpha_t f_t(\mathbf{x})\right)$

## AdaBoost in Action

```python
YouTubeVideo('k4G2VCuOMMg')
```

## Intuition behind Boosting

- Suppose you have a bunch of friends that give you advice
- But frequently their advice is bad advice; in fact, each of them only gives good advice 53% of the time!
- Good news: at least this is better than 50/50 :-)
- Can we use such poor advice? Yes! Just combine their opinions using a majority vote!
    - (Of course this only works when the advice they give is "independent")
- Take home message: **combining lots of weak predictions can produce a strong prediction**

## Boosting for face detection

In the context of face detection, what makes a good set of "weak learners"? Apparently a good choice are these [haar-like features](https://en.wikipedia.org/wiki/Haar-like_features). You sum up the pixel values in the white patches, minus the pixel values in the black patches.

### Strong and Weak Learnability

- Boosting's roots are in the Probably Approximately Correct "PAC" (Leslie Valiant) learning model
- Get random examples from an unknown, arbitrary distribution.
- For ***any*** distribution, given polynomially many examples (and polynomial time), a ***Strong PAC learning algorithm*** can, with high probability, find a classifier with ***arbitrarily small*** generalization error.
- Weak PAC Learning Algorithm can do the same except the generalization error only needs to be ***slightly better than random guessing*** $\left(\frac{1}{2} - \gamma\right)$.

### Weak Learning

- Adaboost is justified by the following result.
- Let $\gamma_t = \frac{1}{2} - r_t$. Recall that $r_t = e_{\mathbf{w}^t}(f_t)$ the weighted empirical risk.
- Note that we may assume $\gamma_t \geq 0 \leftrightarrow r_t \leq \frac{1}{2}$.
- If not, just replace $f_t$ with $-f_t$ and note that for any $f$ and $\mathbf{w}$, $$e_\mathbf{w}(f) + e_\mathbf{w}(-f) = 1$$

### Theorem
(Proof in Mohri et. al, Foundations of Machine Learning, 2012)
- The training error of Adaboost satisfies $\frac{1}{n} \sum_{i=1}^n \mathbb{1}_{\{h_T(\mathbf{x}_i) \neq y_i\}} \leq \exp(-2\sum \limits_{t = 1}^T \gamma_t^2)$
- In particular, if $\forall t, \gamma_t \geq \gamma > 0$ then $\frac{1}{n} \mathbb{1}_{\{h_T(\mathbf{x}_i) \neq y_i\}} \leq \exp(-2\sum \limits_{t = 1}^T \gamma^2)$

### Weak Learning Hypothesis

- We may interpret $r_t = \frac{1}{2}$ as corresponding to a base classifier $f_t$ that randomly guesses.
- Thus, $\gamma_t \geq \gamma > 0$ means that $f_t$ is at least slightly better than randomly guessing.
- If the base learner is guaranteed to satisfy $\gamma_t \geq \gamma > 0, \forall t$, it is said to satisfy the weak learning hypothesis.

### Interpretation of the Theorem

- The theorem says that under the weak learning hypothesis, the Adaboost training error converges to zero ***exponentially*** fast.
- Note: To avoid overfitting, the parameters $T$ should be chosen carefully. Example: Cross Validation.

### Remarks about the Adaptibility of AdaBoost
#### 1. Can exploit $\gamma_t \gg \gamma$

- If $r_t = 0$, then $\alpha_t = \frac{1}{2}\ln\left(\frac{1 - r_t}{r_t}\right) = \lim_{r_t \rightarrow 0^{+}}\frac{1}{2}\text{ln}\left(\frac{1 - r_t}{r_t}\right) = +\infty$.
- In other words, if $\exists$ a classifier in $\mathscr{F}$ that perfectly separates the data, AdaBoost says to just use that classifier.

####  2. $\gamma$ and $T$ do not need to be known a Priori

### Towards Generalized Boosting

- It turns out that AdaBoost can be viewed as an iterative algorithm for minimizing the empirical risk corresponding to the exponential loss.
- By generalizing the loss, we get different boosting algorithms with different properties.

### Span of a base class $\mathscr{F}$

For a fixed base class $\mathscr{F}$, define $$\text{span}(\mathscr{F}) = \{\sum \limits_{t = 1}^T \alpha_tf_t \mid T \geq 1, \alpha_t \in \mathbb{R}, f_t \in \mathscr{F}\}$$

### A Minimization Problem

- Consider the following problem $$\min_{F \in \text{span}(\mathscr{F})} \frac{1}{n} \sum \limits_{i = 1}^n \mathbb{1}_{\{\text{sign}(F(\mathbf{x}_i)) \neq y_i\}}$$

- Now, minimizing the zero-one loss is computationally infeasible.

### Minimizing with surrogate losses

- We can use a surrogate loss function $\phi$ instead to give the following optimization problem $$\min_{F \in \text{span}(\mathscr{F})} \frac{1}{n} \sum \limits_{i = 1}^n \phi(y_i F(\mathbf{x}_i))$$

- Examples of surrogate losses:
 - Exponential loss: $\phi(t) = \exp(-t)$
 - Logistic Loss: $\phi(t) = log(1 + \exp(-t))$
 - Hinge Loss: $\phi(t) = \max(0, 1 - t)$
- Note: We will assume $\phi$ is differentiable and $\phi' < 0$ everywhere.

### Functional Gradient Descent
To solve the optimization problem, we can apply gradient descent on a space consisting of functions.

### An FGD Iteration (1)

- Consider the $t^{\text{th}}$ iteration of FGD. The current iterate is $F_{t - 1} = \sum \limits_{s = 1}^{t - 1} \alpha_s f_s$.
- The next iterate will have the form $F_{t - 1} + \alpha_tf_t$

### An FGD Iteration (2)

- Now, we can view $\alpha_1, f_1, ..., \alpha_{t - 1}, f_{t - 1}$ as fixed.
- Define $B_t(\alpha, f) = \frac{1}{n}\sum \limits_{i = 1}^n \phi(y_iF_{t - 1}(\mathbf{x}_i) + y_i\alpha f(\mathbf{x}_i))$
- $f_t$ can then be chosen as the function $f \in \mathscr{F}$ for which the directional derivative of $B_t$ in the direction $f$ is minimized.
- $\alpha_t$ can be chosen as a stepsize $\alpha > 0$ in the direction $f_t$ for which $B_t(\alpha, f_t)$ is minimized.

### Mathematical Details for the choice of $f_t$ (1)

- $\left.\frac{\partial B(\alpha, f)}{\partial \alpha}\right\vert_{\alpha = 0} = \frac{1}{n}\sum \limits_{i = 1}^n y_i f(\mathbf{x}_i)\phi'(y_i F_{t - 1}(\mathbf{x}_i))$

- Minimizing the above with respect to $f$ is equivalent to minimizing $-\sum \limits_{i = 1}^n y_i f(\mathbf{x}_i)\frac{\phi'(y_iF_{t - 1}(\mathbf{x}_i))}{\sum \limits_{j = 1}^n \phi'(y_j F_{t - 1}(x_j)}$ (Note, a minus sign is used as $\phi' < 0$)

### Mathematical Details for the choice of $f_t$ (2)

- Setting $w_i^t = \frac{\phi'(y_iF_{t - 1}(\mathbf{x}_i))}{\sum \limits_{j = 1}^n \phi'(y_j F_{t - 1}(x_j)}$, the minimization problem reduces to $\sum \limits_{i = 1}^n w_i^t\mathbb{1}_{\{f(\mathbf{x}_i) \neq y_i)\}} - \sum \limits_{i = 1}^n w_i^t\mathbb{1}_{\{f(\mathbf{x}_i) = y_i\}}$

- Finally, we then get the minimization as $2\left(\sum \limits_{i = 1}^n w_i^t \mathbb{1}_{\{f(\mathbf{x}_i) = y_i\}}\right) - 1$

- Thus, to solve the first step (choose $f_t$) we just apply the base learner.

### Mathematical Details for the choice of $\alpha_t$

$$\begin{align}
\alpha_t &= \underset{\alpha}{\arg\min} \hspace{0.2cm} B_t(\alpha, f_t)\\
         &= \underset{\alpha}{\arg\min} \frac{1}{n}\sum \limits_{i = 1}^n \phi(y_iF_{t - 1}(\mathbf{x}_i) + y_i\alpha f_t(\mathbf{x}_i))
   \end{align}$$

The above is just a scalar minimization problem that can be solved numerically, e.g., via Newton's method, if no closed form solution is available.

### The Generalized Boosting Algorithm

- Input: $\{(\mathbf{x}_i, y_i)\}_{i = 1}^n, T, \mathscr{F}$, base learner, surrogate loss $\phi$ (differentiable, $\phi^{'} < 0$ everywhere)
- Initialize: $\mathbf{w}^{1} = (\frac{1}{n}, ..., \frac{1}{n}), F_0 = 0$
- For $t = 1, ..., T$
 - $\mathbf{w}^{t} \rightarrow \boxed{\text{base learner}} \rightarrow f_t$
 - $\alpha_t = \underset{\alpha}{\arg\min} \frac{1}{n}\sum \limits_{i = 1}^n \phi(y_iF_{t - 1}(\mathbf{x}_i) + y_i\alpha f_t(\mathbf{x}_i)$
 - $F_t = F_{t - 1} + \alpha_t f_t$
 - $w_i^{t + 1} = \frac{\phi^{'}(y_iF_t(\mathbf{x}_i))}{\sum \limits_{i = 1}^n \phi^{'}(y_iF_y(\mathbf{x}_j))}$
- End
- Output: $h_T(\mathbf{x}) = \text{sign}\left(F_T(\mathbf{x}\right)$


# Possible Binaries among Low-Mass Stars in NGC 2516

Exploring how the data lines up against the study of Irwin et al. (2007), who identified a binary sequence in various CMDs.

```python
%matplotlib inline
import matplotlib.pyplot as plt
from scipy.interpolate import interp1d
import numpy as np
```

```python
ngc2516 = np.genfromtxt('data/ngc2516_Christophe_v3.dat') # data for this study from J&J (2012)
irwin01 = np.genfromtxt('data/irwin2007.phot')            # data from Irwin+ (2007)
```

Plot $(V - I_C)$ CMDs for both data sets. Jackson, Jeffries, & Maxted (2009) apply a correction to the Irwin I-band data to homogenize zero points between the two studies. This correction takes the form $\Delta I = 0.080-0.0075\cdot I$. It was also noted in Jackson & Jeffries (2012) that a separate correction was made to colors, $\Delta(V-I_C) = 0.300 - 0.153\cdot(V-I_C)$.

```python
fig, ax = plt.subplots(1, 2, figsize=(16.0, 8.0))

for axis in ax:
    axis.set_xlabel('$(V - I_C)$', fontsize=20.)
    axis.tick_params(which='major', axis='both', length=10., labelsize=16.)
ax[0].set_ylabel('$V$', fontsize=20.)
ax[1].set_ylabel('$I_C$', fontsize=20.)
ax[0].set_ylim(23., 15.)
ax[1].set_ylim(19.5, 14.)

# approximate "binary" sequence borders from Irwin+ (2007)
binary_domain = np.array([ 1.4,  1.8,  2.1,  2.4,  2.6,  3.0,  3.1,  3.3,  3.6])
binary_range  = np.array([15.8, 16.4, 17.1, 17.9, 18.5, 19.8, 20.2, 20.9, 21.9])

# testing curve smoothing
icurve = interp1d(binary_domain, binary_range, kind='slinear')
binary_domain = np.arange(1.4, 3.6, 0.02)
binary_range  = icurve(binary_domain)

# zero point corrections
DeltaI  = 0.080 - 7.6e-3*irwin01[:,8]
DeltaVI = 0.300 - 0.153*(irwin01[:,7] - irwin01[:,8])
binary_domain = binary_domain + (0.300 - 0.153*binary_domain)

ax[0].plot((irwin01[:,7] - irwin01[:,8] + DeltaVI), irwin01[:,7], 'o', lw=2, ms=5.0, c='#999999')
ax[0].plot((ngc2516[:,1] - ngc2516[:,2]), ngc2516[:,1], 'o', lw=2, ms=8.0, c='#008B8B')
ax[0].plot(binary_domain, binary_range, '-', lw=3, dashes=(20., 20.), c='#800000')

ax[1].plot((irwin01[:,7] - irwin01[:,8] + DeltaVI), irwin01[:,8] + DeltaI, 'o', lw=2, ms=5.0, c='#999999')
ax[1].plot((ngc2516[:,1] - ngc2516[:,2]), ngc2516[:,2], 'o', lw=2, ms=8.0, c='#008B8B')
```

While the binary sequence definition is tenuous, it does indicate that several of the stars in our sample are potentially unrecognized binaries. Plotting the above CMD without our data and only the Irwin et al. data, our approximate binary sequence provide a very clear divide between what is clear the single star sequence and a population of more luminous (or redder) stars. The idea that they are simply redder is explored with the addition of spots. Here, we are only concerned with possible affects of binarity.

If we now plot the differences between the individual data points and the approximate binary sequence, we get

```python
# redefine binary sequence
icurve = interp1d(binary_domain, binary_range, kind='slinear')

# trim the dataset to reasonable limits
irwin01 = np.column_stack((irwin01, DeltaVI))
ngc2516_new = np.array([star for star in ngc2516 if 1.4 < (star[1]-star[2]) < 3.33])
irwin01_new = np.array([star for star in irwin01 if 1.4 < (star[7]-star[8]+star[-1]) < 3.33])

# interpolate data points onto the binary curve
ngc2516_binary_expected = icurve(ngc2516_new[:,1] - ngc2516_new[:,2])
irwin01_binary_expected = icurve(irwin01_new[:,7] - irwin01_new[:,8] + irwin01_new[:,-1])

# calculate differences
ngc2516_diff = ngc2516_binary_expected - ngc2516_new[:,1]
irwin01_diff = irwin01_binary_expected - irwin01_new[:,7]

# plot differences
fig, ax = plt.subplots(2, 1, figsize=(8.0, 8.0), sharex=True)

ax[1].set_xlabel('$(V - I_C)$', fontsize=20.)
for axis in ax:
    axis.set_xlim(1.4, 3.5)
    axis.set_ylabel('$\\Delta V$', fontsize=20.)
    axis.tick_params(which='major', axis='both', length=10., labelsize=16.)

# showing only the Irwin data
ax[0].plot([1.4, 3.6], [0.0, 0.0], '-', lw=3, dashes=(20., 20.), c='#800000')
ax[0].plot((irwin01_new[:,7] - irwin01_new[:,8] + irwin01_new[:,-1]), irwin01_diff, 'o', ms=5.0, c='#999999')

# now both data sets
ax[1].plot([1.4, 3.6], [0.0, 0.0], '-', lw=3, dashes=(20., 20.), c='#800000')
ax[1].plot((irwin01_new[:,7] - irwin01_new[:,8] + irwin01_new[:,-1]), irwin01_diff, 'o', ms=5.0, c='#999999')
ax[1].plot((ngc2516_new[:,1] - ngc2516_new[:,2]), ngc2516_diff, 'o', ms=8.0, c='#008B8B')

fig.tight_layout()
```

There appear to be small offsets in the adopted $V$ magnitudes between the two studies, despite Jackson & Jeffries explicitly stating that they adopt the photometry of Irwin et al. (2007). We can, instead, simply adjust the cut-off criterion if we wish to remove potential binaries by incorporating the small offset uncertainty into the total ambiguity surrounding the binary sequence.

If we select out only the stars that lie +0.1 mag above the zero point in the above figure,

```python
# flag binaries in J&J data
ngc2516_new = np.column_stack((ngc2516_new, ngc2516_diff))
ngc2516_bin = np.array([star for star in ngc2516_new if star[-1] >  0.0])
ngc2516_new = np.array([star for star in ngc2516_new if star[-1] <=  0.0])

# plot differences
fig, ax = plt.subplots(1, 1, figsize=(8.0, 4.0))

ax.set_xlabel('$(V - I_C)$', fontsize=20.)
ax.set_xlim(1.4, 3.5)
ax.set_ylabel('$\\Delta V$', fontsize=20.)
ax.tick_params(which='major', axis='both', length=10., labelsize=16.)

# now both data sets
ax.plot([1.4, 3.6], [0.0, 0.0], '-', lw=3, dashes=(20., 20.), c='#800000')
ax.plot((irwin01_new[:,7] - irwin01_new[:,8] + irwin01_new[:,-1]), irwin01_diff, 'o', ms=5.0, c='#999999')
ax.plot((ngc2516_new[:,1] - ngc2516_new[:,2]), ngc2516_new[:,-1], 'o', ms=7.0, c='#008B8B')
ax.plot((ngc2516_bin[:,1] - ngc2516_bin[:,2]), ngc2516_bin[:,-1], '*', ms=14.0, c='#800000')
```

A selection criterion of 0.2 mag above the zero point corresponds to, approximately, a $3\sigma$ threshold. We could be conservative (in the sense of testing models) and remove stars beginning below 0.1 mag. The impact on the resulting mean radii can be probed later. ~~For now, we'll stick with our initial 0.2 mag criterion.~~ There may be better reason to adopt 0.0 as the threshold. Need to review the binary sequence definition later in Irwin+ (2007).

How does this affect the appearance of color magnitude diagrams?

```python
fig, ax = plt.subplots(1, 2, figsize=(16.0, 8.0))

for axis in ax:
    axis.set_xlabel('$(V - I_C)$', fontsize=20.)
    axis.tick_params(which='major', axis='both', length=10., labelsize=16.)
ax[0].set_ylabel('$V$', fontsize=20.)
ax[1].set_ylabel('$I_C$', fontsize=20.)
ax[0].set_ylim(23., 15.)
ax[1].set_ylim(19.5, 14.)

ax[0].plot((ngc2516_bin[:,1] - ngc2516_bin[:,2]), ngc2516_bin[:,1], '*', lw=2, ms=12.0, c='#800000')
ax[1].plot((ngc2516_bin[:,1] - ngc2516_bin[:,2]), ngc2516_bin[:,2], '*', lw=2, ms=12.0, c='#800000')
ax[0].plot((ngc2516_new[:,1] - ngc2516_new[:,2]), ngc2516_new[:,1], 'o', lw=2, ms=8.0,  c='#008B8B')
ax[1].plot((ngc2516_new[:,1] - ngc2516_new[:,2]), ngc2516_new[:,2], 'o', lw=2, ms=8.0,  c='#008B8B')
```

This has a strong affect on the CMDs of NGC 2516. Both display tighter sequences, as expected. Notably, removing binaries based on $V$-band magnitude removes suspected binaries flagged from visual inspection of the $I_C$-band magnitude.

How, then, are the radii (and mean radii) affected by the removal of binaries? First, radius-color figures.

```python
# load data with *OLD* mean values
mean_radii_VI = np.genfromtxt('data/ngc2516_avg_radii_V-I_v2.dat')
mean_radii_IK = np.genfromtxt('data/ngc2516_avg_radii_I-K_v2.dat')

fig, ax = plt.subplots(1, 2, figsize=(16.0, 8.0))

for axis in ax:
    axis.set_ylabel('Radius ($R_{\\odot}$)', fontsize=20.)
    axis.tick_params(which='major', axis='both', length=10., labelsize=16.)
ax[0].set_xlabel('$(V - I_C)$', fontsize=20.)
ax[1].set_xlabel('$(I_C - K)$', fontsize=20.)

ax[0].plot((ngc2516_bin[:,1] - ngc2516_bin[:,2]), ngc2516_bin[:,10], '*', lw=2, ms=12.0, c='#800000')
ax[0].plot((ngc2516_new[:,1] - ngc2516_new[:,2]), ngc2516_new[:,10], 'o', lw=2, ms=8.0,  c='#008B8B')
ax[0].plot(mean_radii_VI[:,0], mean_radii_VI[:,1], '-', lw=3, c = '#444444')
ax[0].fill_between(mean_radii_VI[:,0], mean_radii_VI[:,2], mean_radii_VI[:,3], edgecolor='#eeeeee', facecolor='#eeeeee')
ax[1].plot((ngc2516_bin[:,2] - ngc2516_bin[:,3]), ngc2516_bin[:,10], '*', lw=2, ms=12.0, c='#800000')
ax[1].plot((ngc2516_new[:,2] - ngc2516_new[:,3]), ngc2516_new[:,10], 'o', lw=2, ms=8.0,  c='#008B8B')
ax[1].plot(mean_radii_IK[:,0], mean_radii_IK[:,1], '-', lw=3, c = '#444444')
ax[1].fill_between(mean_radii_IK[:,0], mean_radii_IK[:,2], mean_radii_IK[:,3], edgecolor='#eeeeee', facecolor='#eeeeee')
```

Looking at the radius-color diagrams, there is a fairly even division of the binary stars about the previously determined mean values. This suggests that removing binaries would have only a minor impact on the mean radii as a function of photometric color.

And radius-magnitude,

```python
# load data with *OLD* mean values
mean_radii_I = np.genfromtxt('data/ngc2516_avg_radii_I_v2.dat')
mean_radii_K = np.genfromtxt('data/ngc2516_avg_radii_K_v2.dat')

fig, ax = plt.subplots(1, 2, figsize=(16.0, 8.0))

for axis in ax:
    axis.set_ylabel('Radius ($R_{\\odot}$)', fontsize=20.)
    axis.tick_params(which='major', axis='both', length=10., labelsize=16.)
ax[0].set_xlabel('$I_C$ (mag)', fontsize=20.)
ax[1].set_xlabel('$K$ (mag)', fontsize=20.)

ax[0].plot(ngc2516_bin[:,2], ngc2516_bin[:,10], '*', lw=2, ms=12.0, c='#800000')
ax[0].plot(ngc2516_new[:,2], ngc2516_new[:,10], 'o', lw=2, ms=8.0,  c='#008B8B')
ax[0].plot(mean_radii_I[:,0], mean_radii_I[:,1], '-', lw=3, c = '#444444')
ax[0].fill_between(mean_radii_I[:,0], mean_radii_I[:,2], mean_radii_I[:,3], edgecolor='#eeeeee', facecolor='#eeeeee')
ax[1].plot(ngc2516_bin[:,3], ngc2516_bin[:,10], '*', lw=2, ms=12.0, c='#800000')
ax[1].plot(ngc2516_new[:,3], ngc2516_new[:,10], 'o', lw=2, ms=8.0,  c='#008B8B')
ax[1].plot(mean_radii_K[:,0], mean_radii_K[:,1], '-', lw=3, c = '#444444')
ax[1].fill_between(mean_radii_K[:,0], mean_radii_K[:,2], mean_radii_K[:,3], edgecolor='#eeeeee', facecolor='#eeeeee')
```

This is quite interesting. Binaries appear to have estimated radii consistent with those from the single star population. It's therefore possible that removing binaries will increase the mean radii as a function of both magnitude, since the stars with the largest radii are considered to be among the single star population.

Does it make sense that binaries lie on the smaller side? Recall,

\begin{equation}
    \frac{R\sin i}{R_{\odot}} = 0.124 \frac{P}{2\pi} v\sin i,
\end{equation}

so the radius is proportional to both the rotational velocity and the rotational period. At a given magnitude, one would expect the average measured $v\sin i$ of a binary to be greater than for single stars. That would tend to increase the measurement of $R\sin i$ for a fixed orbital period. But, if the star has a brighter magnitude than it would if the two stars are single, then the adopted template spectrum for the binary system may be too early of a spectral type, leading to intrinsically broader spectral lines. This would tend to produce lower $v\sin i$ values.

```python
fig, ax = plt.subplots(1, 2, figsize=(16.0, 8.0))

for axis in ax:
    axis.tick_params(which='major', axis='both', length=10., labelsize=16.)
ax[0].set_xlabel('$I_C$ (mag)', fontsize=20.)
ax[1].set_xlabel('$I_C$ (mag)', fontsize=20.)
ax[0].set_ylabel('Measured Period (day)', fontsize=20.)
ax[1].set_ylabel('Measured $v \\sin i$ (km s$^{-1}$)', fontsize=20.)

ax[0].plot(ngc2516_bin[:,2], ngc2516_bin[:,0], '*', lw=2, ms=12.0, c='#800000')
ax[0].plot(ngc2516_new[:,2], ngc2516_new[:,0], 'o', lw=2, ms=8.0,  c='#008B8B')
ax[1].plot(ngc2516_bin[:,2], ngc2516_bin[:,7], '*', lw=2, ms=12.0, c='#800000')
ax[1].plot(ngc2516_new[:,2], ngc2516_new[:,7], 'o', lw=2, ms=8.0,  c='#008B8B')
```

```python
fig, ax = plt.subplots(1, 2, figsize=(16.0, 8.0))

for axis in ax:
    axis.tick_params(which='major', axis='both', length=10., labelsize=16.)
ax[0].set_xlabel('$K$ (mag)', fontsize=20.)
ax[1].set_xlabel('$K$ (mag)', fontsize=20.)
ax[0].set_ylabel('Measured Period (day)', fontsize=20.)
ax[1].set_ylabel('Measured $v \\sin i$ (km s$^{-1}$)', fontsize=20.)

ax[0].plot(ngc2516_bin[:,3], ngc2516_bin[:,0], '*', lw=2, ms=12.0, c='#800000')
ax[0].plot(ngc2516_new[:,3], ngc2516_new[:,0], 'o', lw=2, ms=8.0,  c='#008B8B')
ax[1].plot(ngc2516_bin[:,3], ngc2516_bin[:,7], '*', lw=2, ms=12.0, c='#800000')
ax[1].plot(ngc2516_new[:,3], ngc2516_new[:,7], 'o', lw=2, ms=8.0,  c='#008B8B')
```

Jackson et al. (2009) used two spectral templates: a K4.5 and an M6. If the switching of spectral templates was the root cause of lower $v\sin i$ values for some of the binary systems, we expect that there would be a noticeable distinction across the transition threshold, biasing $v \sin i$ measurements toward lower values for binaries _above_ the threshold magnitude where the template transition occurs. However, Jackson et al. note that
> The $v \sin i$ values found using the two templates did not differ significantly, either on average of as a function of color.

Yet, there does appear to be a difference between the population of stars above and below 14.5 mag in $K$ and approximately 17.0 mag in $I_C$.


 Tutorial 2

 Physics Informed Neural Networks Part 2

 1D Heat Equation PINNs Equation Example

# Overview

This notebook is based on two papers: *[Physics-Informed Neural Networks:  A Deep LearningFramework for Solving Forward and Inverse ProblemsInvolving Nonlinear Partial Differential Equations](https://www.sciencedirect.com/science/article/pii/S0021999118307125)* and *[Hidden Physics Models:  Machine Learning of NonlinearPartial Differential Equations](https://www.sciencedirect.com/science/article/pii/S0021999117309014)* with the help of  Fergus Shone and Michael Macraild.

These tutorials will go through solving Partial Differential Equations using Physics Informed Neuaral Networks focusing on the 1D Heat Equation and a more complex example using the Navier Stokes Equation

**This introduction section is replicated in all PINN tutorial notebooks (please skip if you've already been through)**

If you have not already then in your gitbash or terminal please run the following code in the LIFD_ENV_ML_NOTEBOOKS directory via the terminal(mac or linux)  or git bash (windows)

```bash
git submodule init
git submodule update --init --recursive
```

**If this does not work please clone the [PINNs](https://github.com/maziarraissi/PINNs) repository into your Physics_Informed_Neural_Networks folder**

Physics Informed Neural Networks

For a typical Neural Network using algorithims like gradient descent to look for a hypothesis, data is the only guide, however if the data is noisy or sparse and we already have governing physical models we can use the knowledge we already know to optamize and inform the algoithms. This can be done via [feature enginnering]() or by adding a physicall inconsistency term to the loss function.

## The very basics

If you know nothing about neural networks there is a [toy neural network python code example](https://github.com/cemac/LIFD_ENV_ML_NOTEBOOKS/tree/main/ToyNeuralNetwork) included in the [LIFD ENV ML Notebooks Repository]( https://github.com/cemac/LIFD_ENV_ML_NOTEBOOKS). Creating a 2 layer neural network to illustrate the fundamentals of how Neural Networks work and the equivlent code using the python machine learning library [tensorflow](https://keras.io/).

## Recommended reading

The in-depth theory behind neural networks will not be covered here as this tutorial is focusing on application of machine learning methods. If you wish to learn more here are some great starting points.

* [All you need to know on Neural networks](https://towardsdatascience.com/nns-aynk-c34efe37f15a)
* [Introduction to Neural Networks](https://victorzhou.com/blog/intro-to-neural-networks/)
* [Physics Guided Neural Networks](https://towardsdatascience.com/physics-guided-neural-networks-pgnns-8fe9dbad9414)
* [Maziar Rassi's Physics informed GitHub web Page](https://maziarraissi.github.io/PINNs/)

 Machine Learning Theory

## Physics informed Neural Networks

Neural networks work by using lots of data to calculate weights and biases from data alone to minimise the loss function enabling them to act as universal fuction approximators. However these loose their robustness when data is limited. However by using know physical laws or empirical validated relationships the solutions from neural networks can be sufficiently constrianed by disregardins no realistic solutions.

A Physics Informed Nueral Network considers a parameterized and nonlinear partial differential equation in the genral form;

\begin{align}
u_t + \mathcal{N}[u; \lambda]  = 0,  x \in \Omega, t \in [0,T],\\
\end{align}

where $\mathcal{u(t,x)}$ denores the hidden solution, $\mathcal{N}$ is a nonlinear differential operator acting on $u$, $\mathcal{\lambda}$ and $\Omega$ is a subset of $\mathbb{R}^D$ (the perscribed data). This set up an encapuslate a wide range of problems such as diffusion processes, conservation laws,  advection-diffusion-reaction  systems,  and  kinetic  equations and conservation laws.

Here we will go though this for the 1D headt equation and Navier stokes equations

 Python

## Tensorflow

There are many machine learning python libraries available, [TensorFlow](https://www.tensorflow.org/) a is one such library. If you have GPUs on the machine you are using TensorFlow will automatically use them and run the code even faster!

## Further Reading

* [Running Jupyter Notebooks](https://jupyter.readthedocs.io/en/latest/running.html#running)
* [Tensorflow optimizers](https://www.tutorialspoint.com/tensorflow/tensorflow_optimizers.htm)

 Requirements

These notebooks should run with the following requirements satisfied

 Python Packages:

* Python 3
* tensorflow > 2
* numpy
* matplotlib
* scipy

 Data Requirements

This notebook referes to some data included in the git hub repositroy

**Contents:**

1. [1D Heat Equation Non ML Example](PINNs_1DHeatEquations_nonML.ipynb)
2. **[1D Heat Equation PINN Example](PINNs_1DEquationExample.ipynb)**
    * [1D Heat Equation Forwards](#1D-Heat-Equation-Forwards)
    * [1D Heat Equation Inverse](#1D-Heat-Equation-Inverse)
3. [Navier-Stokes PINNs discovery of PDE’s](PINNs_NavierStokes_example.ipynb)
4. [Navier-Stokes PINNs Hidden Fluid Mechanics](PINNs_NavierStokes_HFM.ipynb)

Load in all required modules (includig some auxillary code) and turn off warnings.

```python
# For readability: disable warnings
import warnings
warnings.filterwarnings('ignore')
```

```python
import sys
import os
sys.path.insert(0, 'PINNs/Utilities/')
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
import scipy.io
from scipy.interpolate import griddata
import time
from itertools import product, combinations
from mpl_toolkits.mplot3d import Axes3D
from mpl_toolkits.mplot3d.art3d import Poly3DCollection
from mpl_toolkits.axes_grid1 import make_axes_locatable
import matplotlib.gridspec as gridspec
from time import time
import scipy.sparse as sp
import scipy.sparse.linalg as la
import subprocess
subprocess.check_call([sys.executable, "-m", "pip", "install", "pyDOE"])
from pyDOE import lhs
```

# Solving 1D heat equations via Neural Networks

# 1D Heat Equation Forwards

**Model Problem: 1D Heat Equation**

We begin by describing the first model problem - the one-dimensional heat equation.

The heat equation is the prototypical parabolic partial differential equation and can be applied to modelling the diffusion of heat through a given region, hence its name. Read more about the heat equation here: https://en.wikipedia.org/wiki/Heat_equation.

In 1D, the heat equation can be written as:

\begin{equation}
\frac{\partial u}{\partial t} = k \frac{\partial^2 u}{\partial x^2 },
\end{equation}

where $k$ is a material parameter called the coefficient of thermal diffusivitiy.

This equation can be solved using numerical methods, such as finite differences or finite elements. For this notebook, we have solved the above equation numerically on a domain of $x \in [0,1]$ and $t \in [0, 0.25]$. Solving this equation numerically gives us a spatiomtemporal domain $(x,t)$ and corresponding values of the solution $u$.

Here we will describe the architecture of the PINN we use to solve the 1D heat equation in this notebook.

Net U in the above diagram approximations a function that maps from $(x,t) \mapsto u$. $\sigma$ represents the biases and weights for the each neuron of the network. These $\sigma$ values are the network parameters that are updated after each iteration. AD means Automatic Differentiation - this is the chain rule-based differentiation procedure that allows for differentiation of network outputs with respect to its inputs, e.g. differenting $u$ with respect to $x$, or calculating $\frac{\partial u}{\partial x}$. The I node in the AD section represents the identity operation, i.e. keeping $u$ fixed without applying any differentiation.

After the automatic differentiation part of the network, we have two separate loss function components - the data loss and the PDE loss. The data loss term is calculated by finding the difference between the network outputs/predictions $u$ and the ground truth values of $u$, which could come from simulation or experiment. The data loss term enforces the network outputs to match known data points, which are represented by the pink box labelled "Data". The PDE loss term is where we add the "physics-informed" part of the network. Using automatic differentiation, we are able to calculate derivatives of our network outputs, and so we are able to construct a loss function that enforces the network to match the PDE that is known to govern the system. In this case, the PDE loss term is defined as:

\begin{equation}
f = \frac{\partial u}{\partial t} - k \frac{\partial^2 u}{\partial x^2 },
\end{equation}

where f is the residual of the 1D heat equation. By demanding that $f$ is minimised as our network train, we ensure that the network outputs obey the underlying PDE that governs the system. We then calculate the total loss of the system as a sum of the data loss and the PDE loss.

The loss is calculated after each pass through the network and when it is above a certain tolerance, the weights and biases are updated using a gradient descent step. When the loss falls below the tolerance the network is trained. In inference mode, we can then input a fine mesh of spatiomteporal coordinates and the network will find the solution at each of these points.

**$u(x,t)$** can then be defined below as the function `net_u` and the physics informed neural network **$f(x,t)$** is outline in function `net_f`

`neural_net()` constructs the network U where X is a matrix containing the input and output coordinates, i.e. x,t,u
and X is normalised so that all values lie between -1 and 1, this improves training

`net_u()` constructs a network that takes input x,t and outputs the solution u

`net_f()`  the f network is where the PDE is encoded:

1. we read in the value of k first so that it can be included in the equations
2. then we evaluate u for the X_f input coordinates (collocation points)
3. then we use tensorflow differentiation to calculate the derivatives of the solution
4. finally we encode the PDE in residual form, as f->0, u_t = k*u_xx, which is the governing eq

```python

def neural_net(X, weights, biases,lb,ub):
    num_layers = len(weights) + 1

    H = 2.0*(X - lb)/(ub - lb) - 1.0
    for l in range(0,num_layers-2):
        W = weights[l]
        b = biases[l]
        H = tf.tanh(tf.add(tf.matmul(H, W), b))
    W = weights[-1]
    b = biases[-1]
    Y = tf.add(tf.matmul(H, W), b)
    return Y

def net_u(x_tf, t_tf, weights, biases, lb, ub):
    u = neural_net(tf.concat([x_tf,t_tf],1), weights, biases, lb, ub)
    return u

def net_f(x_tf, t_tf, weights, biases,lb, ub,k):
    u = net_u(x_tf,t_tf,weights, biases, lb, ub)
    u_t = tf.gradients(u, t_tf)[0]
    u_x = tf.gradients(u, x_tf)[0]
    u_xx = tf.gradients(u_x, x_tf)[0]
    f = k*u_xx
    return f
```

###  Intialise everying

the `init` function will take our gridded data X and U initialised it building our neural networks from the functions defined above ready to train the model

Variables to be deffined here:

`X_u`: Input coordinates, e.g. spatial and temporal coordinates.

`u`: Output corresponding to each input coordinate.

`X_f`: Collocation points at which the governing equations are satisfied. These coordinates will have the same format as the X_u coordinates, e.g. $(x,t)$.

layers: Specifies the structure of the u network.

`lb`: Vector containing the lower bound of all of the coordinate variables, e.g. $x_{min}$, $t_{min}$.

`ub`: Vector containing the upper bound of all of the coordinate variables, e.g. $x_{max}$, $t_{max}$.

`k`: This is the constant material parameter for this specific problem. For this problem, the heat equation, $k$ represents thermal diffusivity.

# Advanced

Once you have run through the notebook once you may wish to alter the optamizer used in the `init()` function to see the large effect optamizer choice may have.

We've highlighted in the comments a number of possible optamizers to use from the [tf.compat.v1.train](https://www.tensorflow.org/api_docs/python/tf/compat/v1/train) module.
*This method was chosen to limit tensorflow version modifications required from the original source code*

You can learn more about different optamizers [here](https://towardsdatascience.com/optimizers-for-training-neural-network-59450d71caf6)

# init

```python
## the initialisation type can be played with, however we kept it as in the original code
def xavier_init( size):
    in_dim = size[0]
    out_dim = size[1]
    xavier_stddev = np.sqrt(2/(in_dim + out_dim))
    return tf.Variable(tf.random.truncated_normal([in_dim, out_dim], stddev=xavier_stddev), dtype=tf.float32)

def initialize_NN( layers):
    weights = []
    biases = []
    num_layers = len(layers)
    for l in range(0,num_layers-1):
        W = xavier_init(size=[layers[l], layers[l+1]])
        b = tf.Variable(tf.zeros([1,layers[l+1]], dtype=tf.float32), dtype=tf.float32)
        weights.append(W)
        biases.append(b)
    return weights, biases

def init(X, u, layers, lb, ub, k):
        # This line of code is required to prevent some tensorflow errors arrising from the
        # inclusion of some tensorflw v 1 code
        tf.compat.v1.disable_eager_execution()

        ## lb and ub denote lower and upper bounds on the inputs to the network
        # these bounds are used to normalise the network variables

        ## the first two columns of X contain x_u and t_u
        x = X[:,0:1]
        t = X[:,1:2]

        layers = layers

        # Initialize NN
        weights, biases = initialize_NN(layers)

        # tf placeholders and graph
        ## This converts the data into a Tensorflow format
        sess = tf.compat.v1.Session(config=tf.compat.v1.ConfigProto(allow_soft_placement=True,
                                                     log_device_placement=True))

        x_tf = tf.compat.v1.placeholder(tf.float32, shape=[None, x.shape[1]])
        t_tf = tf.compat.v1.placeholder(tf.float32, shape=[None, t.shape[1]])
        u_tf = tf.compat.v1.placeholder(tf.float32, shape=[None, u.shape[1]])

        # u predictions take the input coordinates x_u and t_u and calculate the corresponding u value
        u_pred = net_u(x_tf, t_tf,weights, biases, lb, ub)
        # f predictions take the input coordinates x_f and t_f and calculate the corresponding f value
        f_pred = net_f(x_tf, t_tf,weights, biases,lb, ub,k)

        ## the loss function is defined by the sum of the prediction loss: u-u_pred
        ## and the PDE loss: f_pred
        ## the PDE loss only requires one term, as it is a residual and tends to zero
        loss_PDE = tf.reduce_mean(tf.square(f_pred))
        loss_data = tf.reduce_mean(tf.square(u_tf - u_pred))
        loss = loss_PDE + 5*loss_data

        ##############################################################################################
        #                                                                                #
        ## the optimizer is something that can be tuned to different requirements                    #
        ## we have not investigated using different optimizers, the orignal code uses L-BFGS-B which #
        ## is not tensorflow 2 compatible                                                            #
        #                                                                                #
        #  SELECT OPTAMIZER BY UNCOMMENTING OUT one of the below lines AND RERUNNING CODE            #
        #  You can alsoe edit the learning rate to see the effect of that                            #
        #                                                                                #
        ##############################################################################################

        learning_rate = 0.001
        optimizer = tf.compat.v1.train.MomentumOptimizer(learning_rate, 0.9)
        # optimizer = tf.compat.v1.train.AdagradOptimizer(learning_rate) # 8 %
        # optimizer = tf.compat.v1.train.ProximalGradientDescentOptimizer(learning_rate)
        # optimizer = tf.compat.v1.train.GradientDescentOptimizer(learning_rate)
        # optimizer = tf.compat.v1.train.AdadeltaOptimizer(learning_rate) # yeilds poor results
        # ptimizer = tf.compat.v1.train.FtrlOptimizer(learning_rate)

        # LEAVE THESE OPIMISERS ALONE
        optimizer_Adam = tf.compat.v1.train.AdamOptimizer()
        train_op_Adam = optimizer_Adam.minimize(loss)

        init = tf.compat.v1.global_variables_initializer()
        sess.run(init)
        xvars = [X, lb, ub, x, t, u]
        NNvars = [layers, weights, biases]
        tfvars = [sess, x_tf, t_tf ,u_tf]
        preds = [u_pred, f_pred]
        optvars = [loss, optimizer,optimizer_Adam,train_op_Adam]
        return xvars, NNvars, tfvars, preds, optvars
```

# Load data and set input parameters

A feedforward neural network of the following structure is assumed:
- the input is scaled elementwise to lie in the interval $[-1, 1]$,
- followed by 8 fully connected layers each containing 20 neurons and each followed by a hyperbolic tangent activation function,
- one fully connected output layer.

This setting results in a network with a first hidden layer: $2 \cdot 20 + 20 = 60$; $9$ intermediate layers: each $20 \cdot 20 + 20 = 540$; output layer: $20 \cdot 1 + 1 = 21$).

# Number of collocation points

`2000` colloction points is the default setting for this example this can be increased to improve results at cost of computational speed. The original work set this `N_u=10000` running on GPU's in a few minutes.

The network takes in data in coordinate pairs: $(x,t) \mapsto u$.

Once you have run through the notebook once you may wish to alter any the following

- number of data training points `N_u`
- number of collocation training points `N_f`
- number of layers in the network `layers`
- number of neurons per layer `layers`

```python
k = 1
N_u = 100 #100 # number of data points
N_f = 2000 # Coloaction points
# structure of network: two inputs (x,t) and one output u
# 8 fully connected layers with 20 nodes per layer
layers = [2, 20, 20, 20, 20, 20, 20, 20, 20, 1]
```

```python
data = scipy.io.loadmat("Data/heatEquation_data.mat")
t = data['t'].flatten()[:,None] # read in t and flatten into column vector
x = data['x'].flatten()[:,None] # read in x and flatten into column vector
 # Exact represents the exact solution to the problem, from the Matlab script provided
Exact = np.real(data['usol']).T # Exact has structure of nx times nt

print("usol shape = ", Exact.shape)

# We need to find all the x,t coordinate pairs in the domain
X, T = np.meshgrid(x,t)

# Flatten the coordinate grid into pairs of x,t coordinates
X_star = np.hstack((X.flatten()[:,None], T.flatten()[:,None])) # coordinates x,t
u_star = Exact.flatten()[:,None]   # corresponding solution value with each coordinate

print("X has shape ", X.shape, ", X_star has shape ", X_star.shape)

# Doman bounds (-1,1)
lb = X_star.min(0)
ub = X_star.max(0)

print("Lower bounds of x,t: ", lb)
print("Upper bounds of x,t: ", ub)

## train using internal points
X_u_train = X_star
u_train = u_star

## Generate collocation points using Latin Hypercube sampling within the bounds of the spationtemporal coordinates
# Generate N_f x,t coordinates within range of upper and lower bounds
X_f_train = lb + (ub-lb)*lhs(2, N_f) # the 2 denotes the number of coordinates we have - x,t

## In addition, we add the X_u_train coordinats from the boundaries to the X_f coordinate set
X_f_train = np.vstack((X_f_train, X_u_train)) # stack up all training x,t coordinates for u and f

## We downsample the boundary data to leave N_u randomly distributed points
## This makes the training more difficult -
## if we used all the points then there is not much for the network to do!
idx = np.random.choice(X_star.shape[0], N_u, replace=False)
X_u_train = X_star[idx,:]
u_train = u_star[idx,:]

```

```python
## making a plot to show the distribution of training data
plt.scatter(X_f_train[:,1], X_f_train[:,0], marker='x', color='red',alpha=0.1)
plt.scatter(X_u_train[:,1], X_u_train[:,0], marker='x', color='black')
plt.xlabel('x')
plt.ylabel('t')
plt.title('Data points and collocation points (red crosses)')
plt.legend(['Collocation Points', 'Data Points'])
plt.show()
```

# Initalise the neural network

`init` is called passing in the training data `X_u_train` and `u_train` with information about the neural network layers and bounds `lb` `ub`

# Extract vars

`init` reformats some of the data and outputs model features that we need to pass into the training function `train`

```python
xvars, NNvars, tfvars, preds, optvars = init(X_u_train, u_train, layers, lb, ub,k)
X, lb, ub, x, t, u = xvars
layers, weights, biases=NNvars
sess, x_tf, t_tf ,u_tf=tfvars
u_pred, f_pred,=preds
loss, optimizer,optimizer_Adam,train_op_Adam = optvars
```

```python

def train(sess, nIter,x_tf, t_tf, u_tf,x, t,u_train, loss, train_op_Adam, optimizer_Adam):
    tf_dict = {x_tf: x,  t_tf: t, u_tf: u}

    start_time = time()
    for it in range(nIter):
        sess.run(train_op_Adam, tf_dict)

        # Print
        if it % 50 == 0:
            elapsed = time() - start_time
            loss_value = sess.run(loss, tf_dict)

            print('It: %d, Loss: %.3e, l1: %.3f, l2: %.5f, Time: %.2f' %
                (it, loss_value, elapsed))
            start_time = time()

    optimizer.minimize(loss)

```

**Training might take a while depending on value of Train_iterations**

If you set Train_iterations too low the end results will be garbage. 50000 was used to achieve excellent results.

* If you are using a machine with GPUs please set `Train_iterations` to 50000 and this will run quickly
* If you are using a well spec'ed laptop/computer and can leave this setting `Train_iterations=50000` but it will take upto 10 mins
* If you are using a low spec'ed laptop/computer or cannont leave the code running `Train_iterations=20000` is the reccomended value (this solution may not be accurate)

```python
# Training
Train_iterations=50000
```

```python
train(sess, Train_iterations,x_tf, t_tf, u_tf,x, t,u_train, loss, train_op_Adam, optimizer_Adam)

```

# Use trained model to predict from data sample

`predict` will predict `u` using the trained model

```python

def predict(sess, x_star,u_star, u_pred, f_pred):
    tf_dict = {x_tf: x_star, t_tf: u_star}
    u_star = sess.run(u_pred, tf_dict)
    f_star = sess.run(f_pred, tf_dict)

    return u_star, f_star
```

```python
u_pred, f_pred=preds
u_pred, f_pred = predict(sess,X_star[:,0:1],X_star[:,1:2], u_pred, f_pred)

error_u = np.linalg.norm(u_star-u_pred,2)/np.linalg.norm(u_star,2)
```

# Calculate Errors

if you have set the number of training iterations large enough the errors should be small.

```python
print("f_pred mean = ", np.mean(f_pred))
print('Error u: %e' % (error_u))
print('Percent error u: ',  100*error_u)

```

```python
# Set grid values back to full data set size for plotting

t = data['t'].flatten()[:,None]
x = data['x'].flatten()[:,None]
X, T = np.meshgrid(x,t)

U_pred = griddata(X_star, u_pred.flatten(), (X,T), method='cubic')
Error = np.abs(Exact - U_pred)
percentError = 100*np.divide(Error, Exact)
```

# Plot Exact and Precticed $(u,t)$

```python
fig, ax = plt.subplots(figsize=(15,15))
ax.axis('off')

print("--------- Errors ---------")
print('Percent error u: ',  100*error_u)
print("--------------------------")

####### Row 0: u(t,x) ##################
gs0 = gridspec.GridSpec(3, 2)
gs0.update(top=1-0.06, bottom=1-1/3, left=0.15, right=0.85, wspace=0, hspace=1)

########## Prediction ##################
ax = plt.subplot(gs0[0, :])
h = ax.imshow(U_pred.T, interpolation='nearest', cmap='rainbow',
              extent=[t.min(), t.max(), x.min(), x.max()],
              origin='lower', aspect='auto', vmin = 0, vmax = 1)
divider = make_axes_locatable(ax)
cax = divider.append_axes("right", size="5%", pad=0.05)
fig.colorbar(h, cax=cax)

ax.plot(X_u_train[:,1], X_u_train[:,0], 'kx', label = 'Data (%d points)' % (u_train.shape[0]), markersize = 4, clip_on = False)

line = np.linspace(x.min(), x.max(), 2)[:,None]
ax.plot(t[2]*np.ones((2,1)), line, 'w-', linewidth = 1)
ax.plot(t[5]*np.ones((2,1)), line, 'w-', linewidth = 1)
ax.plot(t[10]*np.ones((2,1)), line, 'w-', linewidth = 1)

ax.set_xlabel('$t$')
ax.set_ylabel('$x$')
ax.legend(frameon=False, loc = 'best')
ax.set_title('$u(t,x) - Prediction$', fontsize = 10)

########## Exact ##################
ax = plt.subplot(gs0[1, :])
i = ax.imshow(Exact.T, interpolation='nearest', cmap='rainbow',
              extent=[t.min(), t.max(), x.min(), x.max()],
              origin='lower', aspect='auto', vmin = 0, vmax = 1)
divider = make_axes_locatable(ax)
cax = divider.append_axes("right", size="5%", pad=0.05)
fig.colorbar(i, cax=cax)

ax.plot(X_u_train[:,1], X_u_train[:,0], 'kx', label = 'Data (%d points)' % (u_train.shape[0]), markersize = 4, clip_on = False)

line = np.linspace(x.min(), x.max(), 2)[:,None]
ax.plot(t[2]*np.ones((2,1)), line, 'w-', linewidth = 1)
ax.plot(t[5]*np.ones((2,1)), line, 'w-', linewidth = 1)
ax.plot(t[10]*np.ones((2,1)), line, 'w-', linewidth = 1)

ax.set_xlabel('$t$')
ax.set_ylabel('$x$')
ax.legend(frameon=False, loc = 'best')
ax.set_title('$u(t,x)$ - Exact', fontsize = 10)

########## Error ##################
ax = plt.subplot(gs0[2, :])
j = ax.imshow(percentError.T, interpolation='nearest', cmap='rainbow',
              extent=[t.min(), t.max(), x.min(), x.max()],
              origin='lower', aspect='auto', vmin = 0, vmax = 10)
divider = make_axes_locatable(ax)
cax = divider.append_axes("right", size="5%", pad=0.05)
fig.colorbar(j, cax=cax)

ax.plot(X_u_train[:,1], X_u_train[:,0], 'kx', label = 'Data (%d points)' % (u_train.shape[0]), markersize = 4, clip_on = False)

line = np.linspace(x.min(), x.max(), 2)[:,None]
ax.plot(t[2]*np.ones((2,1)), line, 'w-', linewidth = 1)
ax.plot(t[5]*np.ones((2,1)), line, 'w-', linewidth = 1)
ax.plot(t[10]*np.ones((2,1)), line, 'w-', linewidth = 1)

ax.set_xlabel('$t$')
ax.set_ylabel('$x$')
ax.legend(frameon=False, loc = 'best')
ax.set_title('$u(t,x)$ - Percent Error', fontsize = 10)

####### Row 1: u(t,x) slices ##################

fig, ax = plt.subplots(figsize=(15,15))
ax.axis('off')
gs1 = gridspec.GridSpec(1, 3)
gs1.update(top=1-1/3, bottom=0, left=0.1, right=0.9, wspace=0.5)

ax = plt.subplot(gs1[:, 0])
ax.plot(x,Exact[2,:], 'b-', linewidth = 2, label = 'Exact')
ax.plot(x,U_pred[2,:], 'r--', linewidth = 2, label = 'Prediction')
ax.set_xlabel('$x$')
ax.set_ylabel('$u(t,x)$')
ax.set_title('$t = ' + str(t[2,0]) + '$', fontsize = 10)
ax.axis('square')
ax.set_xlim([0,1])
ax.set_ylim([0,1])

ax = plt.subplot(gs1[:, 1])
ax.plot(x,Exact[5,:], 'b-', linewidth = 2, label = 'Exact')
ax.plot(x,U_pred[5,:], 'r--', linewidth = 2, label = 'Prediction')
ax.set_xlabel('$x$')
ax.set_ylabel('$u(t,x)$')
ax.axis('square')
ax.set_xlim([0,1])
ax.set_ylim([0,1])
ax.set_title('$t = ' + str(t[5,0]) + '$', fontsize = 10)
ax.legend(loc='upper center', bbox_to_anchor=(0.5, -0.35), ncol=5, frameon=False)

ax = plt.subplot(gs1[:, 2])
ax.plot(x,Exact[10,:], 'b-', linewidth = 2, label = 'Exact')
ax.plot(x,U_pred[10,:], 'r--', linewidth = 2, label = 'Prediction')
ax.set_xlabel('$x$')
ax.set_ylabel('$u(t,x)$')
ax.axis('square')
ax.set_xlim([0,1])
ax.set_ylim([0,1])
ax.set_title('$t = ' + str(t[10,0]) + '$', fontsize = 10);

```

**Results**

Above are the results of the PINN. The error for recreating the full solution field is $\approx 10 \%$, despite using only $N_u = 100$ data points. This shows the power of PINNs to learn from sparse measurements by augmanting the available observational data with knowledge of the underlying physics (i.e. governing equations).

The three colourmaps show the PINN prediction, the exact solution from the numerical method and the relative error between these two fields. We can see that the errors are largest near $t=0$ and $x=0$, but that overall the agreement is very good.

On the colourmap, we can see three vertical white lines, which show the location in time of the three profile plots of $u$ against $x$. The three heat profiles at these times are plotted against the exact solution found using numerical methods. The profiles can be seen to be in very good agreement, but show worse agreement.

**Further Work**

Congratulations, you have now trained your first physics-informed neural network!

This network contains a number of hyper-parameters that could be tuned to give better results. Various hyper-parameters include:
- number of data training points N_u
- number of collocation training points N_f
- number of layers in the network
- number of neurons per layer
- weightings for the data and PDE loss terms in the loss function (currently we use loss = loss_PDE + 5*loss_data)

It is also possible to use different sampling techniques for training data points. We randomly select $N_u$ data points, but alternative methods could be choosing only boundary points or choosing more points near the $t=0$ boundary. Choosing boundary points for training could help to reduce the errors seen in these regions.

Feel free to try out some of these changes if you like!

There are 3 subsequent PINNs notebooks to follow, which look at the inverse heat equation problem and two Navier-Stokes fluid flow problems.

# 1D Heat Equation Inverse

Remembering that in 1D, the heat equation can be written as:

\begin{equation}
\frac{\partial u}{\partial t} = k \frac{\partial^2 u}{\partial x^2 }
\end{equation}

where $k$ is a material parameter called the coefficient of thermal diffusivitiy. For this notebook, we have solved the above equation numerically on a domain of $x \in [0,1]$ and $t \in [0, 0.25]$. Solving this equation numerically gives us a spatiomtemporal domain $(x,t)$ and corresponding values of the solution $u$.

**Now we will let $k$ be an unknown input parameter in the PINN**. In reality, we know the value of $k$, as we set it when solving the system numerically, but for the sake of this example let's imagine we do not know the value of $k$ when we come to use the PINN. This corresponds to real-world problems where we may have observational data, knowledge of the governing equations, but little information for some input parameters for the system.

The network architecture for this example is the same as for the previous [example](#1D-Heat-Equation-Forwards). The only difference is that this time we do not know the value for $k$, and so in each training iteration we do not only updates the network weights and biases, but also the value of $k$. Through training, the network will then optimise the value of $k$ such that it fits with the observed data.

```python
## this is the k value used to generate the data
## we use this to compare to the value found by the PINN
k_exact = 1
```

Once you have run through the notebook once you may wish to alter any the following

- number of data training points `N_u`
- number of collocation training points `N_f`
- number of layers in the network `layers`
- number of neurons per layer `layers`

```python
N_u = 100 #100 # number of data points
N_f = 2000 # Coloaction points
# structure of network: two inputs (x,t) and one output u
# 8 fully connected layers with 20 nodes per layer
layers = [2, 20, 20, 20, 20, 20, 20, 20, 20, 1]
```

```python
# This code duplicated from above incase you have been playing with parameters
data = scipy.io.loadmat("Data/heatEquation_data.mat")
t = data['t'].flatten()[:,None] # read in t and flatten into column vector
x = data['x'].flatten()[:,None] # read in x and flatten into column vector
 # Exact represents the exact solution to the problem, from the Matlab script provided
Exact = np.real(data['usol']).T # Exact has structure of nx times nt

# print("t = ", t.transpose())
# print("x = ", x.transpose())
print("usol shape = ", Exact.shape)

# We need to find all the x,t coordinate pairs in the domain
X, T = np.meshgrid(x,t)

# Flatten the coordinate grid into pairs of x,t coordinates
X_star = np.hstack((X.flatten()[:,None], T.flatten()[:,None])) # coordinates x,t
u_star = Exact.flatten()[:,None]   # corresponding solution value with each coordinate

print("X has shape ", X.shape, ", X_star has shape ", X_star.shape)

# Doman bounds (-1,1)
lb = X_star.min(0)
ub = X_star.max(0)

print("Lower bounds of x,t: ", lb)
print("Upper bounds of x,t: ", ub)

## train using internal points
X_u_train = X_star
u_train = u_star

## Generate collocation points using Latin Hypercube sampling within the bounds of the spationtemporal coordinates
# Generate N_f x,t coordinates within range of upper and lower bounds
X_f_train = lb + (ub-lb)*lhs(2, N_f) # the 2 denotes the number of coordinates we have - x,t

## In addition, we add the X_u_train coordinats from the boundaries to the X_f coordinate set
X_f_train = np.vstack((X_f_train, X_u_train)) # stack up all training x,t coordinates for u and f

## We downsample the boundary data to leave N_u randomly distributed points
## This makes the training more difficult -
## if we used all the points then there is not much for the network to do!
idx = np.random.choice(X_star.shape[0], N_u, replace=False)
X_u_train = X_star[idx,:]
u_train = u_star[idx,:]
```

now we will use all the same fuctions as before except we will modify `k` and the train function to handle a changing `k` value

```python
# k as tensorflow variable with inital value set
k = tf.Variable([1.0], dtype=tf.float32)
xvars, NNvars, tfvars, preds, optvars = init(X_u_train, u_train, layers, lb, ub,k)
X, lb, ub, x, t, u = xvars
layers, weights, biases = NNvars
sess, x_tf, t_tf ,u_tf = tfvars
u_pred, f_pred,=preds
loss, optimizer,optimizer_Adam,train_op_Adam = optvars
```

```python
def train(sess, nIter,x_tf, t_tf, u_tf,x, t,u_train, loss, train_op_Adam, optimizer_Adam):
    tf_dict = {x_tf: x,  t_tf: t, u_tf: u}

    start_time = time()
    for it in range(nIter):
        sess.run(train_op_Adam, tf_dict)

        # Print
        if it % 50 == 0:
            elapsed = time() - start_time
            loss_value = sess.run(loss, tf_dict)
            k_value = self.sess.run(self.k)
            print('It: %d, Loss: %.3e, k: %.3f, Time: %.2f' %
                          (it, loss_value, k_value, elapsed))
            start_time = time.time()

    optimizer_Adam.minimize(loss)
```

**Training might take a while depending on value of Train_iterations**

If you set Train_iterations too low the end results will be garbage. 50000 was used to achieve excellent results.

* If you are using a machine with GPUs please set `Train_iterations` to 50000 and this will run quickly
* If you are using a well spec'ed laptop/computer and can leave this setting `Train_iterations=50000` but it will take upto 10 mins
* If you are using a low spec'ed laptop/computer or cannont leave the code running `Train_iterations=20000` is the reccomended value (this solution may not be accurate)

```python
# Training
Train_iterations=50000
train(sess, Train_iterations,x_tf, t_tf, u_tf,x, t,u_train, loss, train_op_Adam, optimizer_Adam)
```

```python
u_pred, f_pred=preds
u_pred, f_pred = predict(sess, X_star[:,0:1], X_star[:,1:2], u_pred, f_pred)

error_u = np.linalg.norm(u_star - u_pred, 2)/np.linalg.norm(u_star, 2)
```

```python
print("f_pred mean = ", np.mean(f_pred))
print('Error u: %e' % (error_u))
print('Percent error u: ',  100*error_u)

```

```python
# Set grid values back to full data set size for plotting

t = data['t'].flatten()[:,None]
x = data['x'].flatten()[:,None]
X, T = np.meshgrid(x,t)

U_pred = griddata(X_star, u_pred.flatten(), (X,T), method='cubic')
Error = np.abs(Exact - U_pred)
percentError = 100*np.divide(Error, Exact)
```

```python
fig, ax = plt.subplots(figsize=(15,15))
ax.axis('off')

print("--------- Errors ---------")
print('Percent error u: ',  100*error_u)
print("--------------------------")

####### Row 0: u(t,x) ##################
gs0 = gridspec.GridSpec(3, 2)
gs0.update(top=1-0.06, bottom=1-1/3, left=0.15, right=0.85, wspace=0, hspace=1)

########## Prediction ##################
ax = plt.subplot(gs0[0, :])
h = ax.imshow(U_pred.T, interpolation='nearest', cmap='rainbow',
              extent=[t.min(), t.max(), x.min(), x.max()],
              origin='lower', aspect='auto', vmin = 0, vmax = 1)
divider = make_axes_locatable(ax)
cax = divider.append_axes("right", size="5%", pad=0.05)
fig.colorbar(h, cax=cax)

ax.plot(X_u_train[:,1], X_u_train[:,0], 'kx', label = 'Data (%d points)' % (u_train.shape[0]), markersize = 4, clip_on = False)

line = np.linspace(x.min(), x.max(), 2)[:,None]
ax.plot(t[2]*np.ones((2,1)), line, 'w-', linewidth = 1)
ax.plot(t[5]*np.ones((2,1)), line, 'w-', linewidth = 1)
ax.plot(t[10]*np.ones((2,1)), line, 'w-', linewidth = 1)

ax.set_xlabel('$t$')
ax.set_ylabel('$x$')
ax.legend(frameon=False, loc = 'best')
ax.set_title('$u(t,x) - Prediction$', fontsize = 10)

########## Exact ##################
ax = plt.subplot(gs0[1, :])
i = ax.imshow(Exact.T, interpolation='nearest', cmap='rainbow',
              extent=[t.min(), t.max(), x.min(), x.max()],
              origin='lower', aspect='auto', vmin = 0, vmax = 1)
divider = make_axes_locatable(ax)
cax = divider.append_axes("right", size="5%", pad=0.05)
fig.colorbar(i, cax=cax)

ax.plot(X_u_train[:,1], X_u_train[:,0], 'kx', label = 'Data (%d points)' % (u_train.shape[0]), markersize = 4, clip_on = False)

line = np.linspace(x.min(), x.max(), 2)[:,None]
ax.plot(t[2]*np.ones((2,1)), line, 'w-', linewidth = 1)
ax.plot(t[5]*np.ones((2,1)), line, 'w-', linewidth = 1)
ax.plot(t[10]*np.ones((2,1)), line, 'w-', linewidth = 1)

ax.set_xlabel('$t$')
ax.set_ylabel('$x$')
ax.legend(frameon=False, loc = 'best')
ax.set_title('$u(t,x)$ - Exact', fontsize = 10)

########## Error ##################
ax = plt.subplot(gs0[2, :])
j = ax.imshow(percentError.T, interpolation='nearest', cmap='rainbow',
              extent=[t.min(), t.max(), x.min(), x.max()],
              origin='lower', aspect='auto', vmin = 0, vmax = 10)
divider = make_axes_locatable(ax)
cax = divider.append_axes("right", size="5%", pad=0.05)
fig.colorbar(j, cax=cax)

ax.plot(X_u_train[:,1], X_u_train[:,0], 'kx', label = 'Data (%d points)' % (u_train.shape[0]), markersize = 4, clip_on = False)

line = np.linspace(x.min(), x.max(), 2)[:,None]
ax.plot(t[2]*np.ones((2,1)), line, 'w-', linewidth = 1)
ax.plot(t[5]*np.ones((2,1)), line, 'w-', linewidth = 1)
ax.plot(t[10]*np.ones((2,1)), line, 'w-', linewidth = 1)

ax.set_xlabel('$t$')
ax.set_ylabel('$x$')
ax.legend(frameon=False, loc = 'best')
ax.set_title('$u(t,x)$ - Percent Error', fontsize = 10)

####### Row 1: u(t,x) slices ##################

fig, ax = plt.subplots(figsize=(15,15))
ax.axis('off')
gs1 = gridspec.GridSpec(1, 3)
gs1.update(top=1-1/3, bottom=0, left=0.1, right=0.9, wspace=0.5)

ax = plt.subplot(gs1[:, 0])
ax.plot(x,Exact[2,:], 'b-', linewidth = 2, label = 'Exact')
ax.plot(x,U_pred[2,:], 'r--', linewidth = 2, label = 'Prediction')
ax.set_xlabel('$x$')
ax.set_ylabel('$u(t,x)$')
ax.set_title('$t = ' + str(t[2,0]) + '$', fontsize = 10)
ax.axis('square')
ax.set_xlim([0,1])
ax.set_ylim([0,1])

ax = plt.subplot(gs1[:, 1])
ax.plot(x,Exact[5,:], 'b-', linewidth = 2, label = 'Exact')
ax.plot(x,U_pred[5,:], 'r--', linewidth = 2, label = 'Prediction')
ax.set_xlabel('$x$')
ax.set_ylabel('$u(t,x)$')
ax.axis('square')
ax.set_xlim([0,1])
ax.set_ylim([0,1])
ax.set_title('$t = ' + str(t[5,0]) + '$', fontsize = 10)
ax.legend(loc='upper center', bbox_to_anchor=(0.5, -0.35), ncol=5, frameon=False)

ax = plt.subplot(gs1[:, 2])
ax.plot(x,Exact[10,:], 'b-', linewidth = 2, label = 'Exact')
ax.plot(x,U_pred[10,:], 'r--', linewidth = 2, label = 'Prediction')
ax.set_xlabel('$x$')
ax.set_ylabel('$u(t,x)$')
ax.axis('square')
ax.set_xlim([0,1])
ax.set_ylim([0,1])
ax.set_title('$t = ' + str(t[10,0]) + '$', fontsize = 10);
```

**Results**

Above are the results of the PINN. The error for recreating the full solution field is $\approx 5 \%$, despite using only $N_u = 100$ data points. This shows the power of PINNs to learn solution fields from sparse measurements, even when some of the input parameters are unknown.

The three colourmaps show the PINN prediction, the exact solution from the numerical method and the relative error between these two fields. We can see that the errors are largest near $t=0$ and $x=0$, but that overall the agreement is very good.

On the colourmap, we can see three vertical white lines, which show the location in time of the three profile plots of $u$ against $x$. The three heat profiles at these times are plotted against the exact solution found using numerical methods. The profiles can be seen to be in very good agreement, but show worse agreement.

**Further Work**

Congratulations, you have now trained your second physics-informed neural network!

This network contains a number of hyper-parameters that could be tuned to give better results. Various hyper-parameters include:
- number of data training points N_u
- number of collocation training points N_f
- number of layers in the network
- number of neurons per layer
- weightings for the data and PDE loss terms in the loss function (currently we use loss = loss_PDE + 5*loss_data)
- initialisation value for k
- optimisation

It is also possible to use different sampling techniques for training data points. We randomly select $N_u$ data points, but alternative methods could be choosing only boundary points or choosing more points near the $t=0$ boundary.

return [here](#1D-Heat-Equation) to try out some of these changes if you like, or [here](#init) to alter optimization method used

# Next Steps

Next we move on to a more complex example using the Navier Stokes Equation in the notebook linked below

[Navier-Stokes PINNs discovery of PDE’s](PINNs_NavierStokes_example.ipynb)

**Contact Details**

For any questions, corrections or comments, please contact either:

Michael MacRaild - scmm@leeds.ac.uk

Fergus Shone - mm16f2s@leeds.ac.uk

```python

```


Table of Contents

1  WARP (Weighted Approximate Rank Pairwise Loss)

1.1  Geometric Distribution

1.2  Building Intuition for WARP

1.3  Introducing WARP

2  Implementation

3  Reference

```python
# code for loading the format for the notebook
import os

# path : store the current path to convert back to it later
path = os.getcwd()
os.chdir(os.path.join('..', 'notebook_format'))
from formats import load_style
load_style(css_style = 'custom2.css', plot_style = False)
```

```python
os.chdir(path)

# 1. magic to print version
# 2. magic so that the notebook will reload external python modules
%load_ext watermark
%load_ext autoreload
%autoreload 2

import numpy as np
from time import time
from scipy.stats import geom
from lightfm import LightFM
from lightfm.evaluation import auc_score
from lightfm.evaluation import precision_at_k
from lightfm.datasets import fetch_movielens

%watermark -a 'Ethen' -d -t -v -p scipy,numpy,lightfm
```

    Ethen 2018-04-07 14:21:10

    CPython 3.6.4
    IPython 6.2.1

    scipy 1.0.0
    numpy 1.14.2
    lightfm 1.14

# WARP (Weighted Approximate Rank Pairwise Loss)

## Geometric Distribution

We do a quick recap on the **Geometric Distribution** as we'll be leveraging its property later. Geometric Distribution measures number of trials needed to get a first success in repeated Bernoulli trials. So suppose we have some independent trials and each trial results in one of two possible outcomes success or failure. And our probability of success is denoted by P(success) = $p$, this number stays constant from trial to trial and $X$ represents the number of trials needed to get the first success.

For the first success to occur on the $x_{th}$ trial:

- The first $x - 1$ trial must be failures.
- The $x_{th}$ trial must be a success.

This gives us the probability mass function of the geometric distribution:

\begin{align}
P(X = x) = (1 - p)^{x - 1}p
\end{align}

For the geometric distribution the minimum number $x$ can take is 1, however, there is no upper bound to that number.

> Example: In a large populations of adults, 30% have received CPR training, if adults from this population are randomly selected, what is the probability that the 6th person sampled is the first that has received CPR training?

```python
# we could do it manually
p = 0.3
prob = (1 - p) ** 5 * p
print(prob)

# or use scipy stats' geometric distribution
# the first success occuring on the 6th trial
print(geom(p = p).pmf(k = 6))
```

    0.05042099999999998
    0.05042099999999998

Last but not least, one useful property to know about the geometric distribution is variable X's mean/expectation is:

\begin{align}
\frac{1}{p}
\end{align}

## Building Intuition for WARP

Like the Bayesian Personalized Ranking (BPR) model, **WARP** deals with (user, positive item, negative item) triplets. Unlike BPR, the negative items in the triplet are not chosen by random sampling: they are chosen from among those negative items which would violate the desired item ranking given the state of the model. This approximates a form of active learning where the model selects those triplets that it cannot currently rank correctly.

This procedure yields roughly the following algorithm:

- For a given user, positive item pair, sample a negative item at random from all the remaining items. Compute predictions for both items; if the negative item's prediction exceeds that of the positive item plus a margin, perform a gradient update so we can rank the positive item higher and the negative item lower. If there is no rank violation, continue sampling negative items until a violation is found.
- If we found a violating negative example at the first try, make a large gradient update: this indicates that a lot of negative items are ranked higher than positives items in current state of the model, and the model should be updated by a large amount. If it took a lot of sampling to find a violating example, perform a small update since the model is likely close to the optimum and should be updated at a low rate.

Given this intuition, we can formalize this idea with some notations in the next section.

## Introducing WARP

In the context of making recommendation using implicit feedback data, we are interested in ranking items $i$ given a user $u$. And when a scoring function $f_u(i)$ give the score of item $i$ for user $u$, the item with the highest score is recommended.

We let $f(u) \in \mathbb{R}^Y$ be a vector function providing a score for each of the items, where $f_i(u)$ is the value for item $i$. At the end of the day, after a scoring function $f_i(u)$ gives a score for all the items $i$, the item with the highest score is recommended.

The class of ranking loss function we'll be looking at today is defined as:

\begin{align}
\sum_{i \in D_u} L \big( rank_i(f_u) \big)
\end{align}

Where:

- $D_u$ represents users $u$'s positive feedback.
- $rank_i(f_u)$ is the margin-based rank of item $i$. i.e.

\begin{align}
rank_i(f_u) = \sum_{j \notin D_u} I \big( f_u(j) \ge f_u(i) \big)
\end{align}

- $I$ is the indicator function.
- This $rank_i(f_u)$ tells us that for an item that does not have a user's positive feedback (denoted as $j$), if its score is higher than that of an item that the user showed a positive feedback for, then it will be counted towards the loss function.

Now going back to $L$, $L(\cdot)$ is in charge of transforming this rank into the actual loss:

\begin{align}
L(k) = \sum_{t = 1}^k a_t, \enspace a_1 \ge a_2 \ge \ldots \ge 0
\end{align}

Different settings of $a_t$ allows the loss function to optimize for different objectives. e.g.

- When $a_1 = 1$ and $a_{t > 1} = 0$, then precision at 1 is optimized for.
- To generalize this a bit, when $a_k = 1$ and $a_{k > 1} = 0$, then precision at $k$ is optimized for.
- On the other hand, $a_{t} = 1 / t$ becomes a smooth weighting over various positions, with most of the weights given to the top position and rapidly decaying weight for lower positions. This is approximately the natural logarithm and we'll be using this value later in our implementation.

This property of optimizing for the top-k ranked items is often times a desired property in recommendation system since in these scenarios, we're often times only interested in recommending a few items to the users and as such we often do not care about the ranking of the entire set of items.

The problem with this version of the loss function is that, while it does depend on the model's parameter, this dependence is not continuous (our rank being integer value), hence we can't derive gradients to directly optimize for this loss function. Hence, the approach that the original authors took is to derive a differentiable approximation to the logarithm of the rank. The rank loss for a specific item $i \in D_u$ is defined as:

\begin{align}
L \big( rank_i(f_u) \big)
&= log \big( rank_i(f_u) \big) \frac{rank_i(f_u)}{rank_i(f_u)} \\
&= log \big( rank_i(f_u) \big) \frac{\sum_{j \notin D_u} I \big( f_u(j) \ge f_u(i) \big)}{rank_i(f_u)} \\
&= \sum_{j \notin D_u} log \big( rank_i(f_u) \big) \frac{I \big( f_u(j) \ge f_u(i) \big)}{rank_i(f_u)} \\
&= \sum_{j \notin D_u} log \big( rank_i(f_u) \big) \frac{\big| 1 - f_u(i) + f_u(j) \big|_+}{rank_i(f_u)}
\end{align}

- Notice that during the last step, we've replaced the indicator function with the hinge loss so the gradient descent based algorithms can be adopted to learn the parameters.
- Based on the equation above, after randomly sampling a positive item $i \in D_u$, we would need to calculate the margin rank for all its negative items. Exactly computing this, however, can be very time-consuming when the number of items is very large. Thus, we instead uniformly sample a negative feedback instance $j \notin D_u$ until a pairwise violation is found, that is, until $1 - f_u(i) + f_u(j) > 0$.
- Using the procedure above, we would have a probability of $1 / rank_i(f_u)$ of drawing $j$ during the sampling for negative feedback step, which accounts for the denominator of equation $9$.

Now our equation becomes:

\begin{align}
L \big( rank_i(f_u) \big)
&= \sum_{j \notin D_u} log \big( rank_i(f_u) \big) \big| 1 - f_u(i) + f_u(j) \big|_+
\end{align}

The next brilliant idea is how to estimate for $rank_i(f_u)$, the rank of item $i$. Recall during the sampling procedure mentioned above, we were sampling negative items uniformly with replacement until we find a negative item in which the pairwise violation is found. If we were to re-read the last paragraph a few more times, hopefully, we can see that this steps follows a geometric distribution. To formalize this, during the sampling for negative items step, say we needed to sample $Q$ times before we were able to find such pairwise violation, then that means in our negative items pool, the probability of sampling a negative item whose score violates the pairwise comparison is:

\begin{align}
\frac{Q}{\big|D_u'\big|}
\end{align}

where $\big|D_u'\big|$ denotes the number of negative items. Utilizing geometric distribution's mean/expectation property, this suggests the value of $rank_i(f_u)$ can be approximated by $\big\lfloor{\frac{|D_u'|}{Q}}\big\rfloor$. Where $\lfloor{\cdot}\rfloor$ is the floor function.

Given all of that, the final formula for estimating the WARP loss for a given item $i$ becomes:

\begin{align}
L \big( rank_i(f_u) \big)
&= \sum_{j \notin D_u} log \big( \big\lfloor{\frac{|D_u'|}{Q}}\big\rfloor \big) \big| 1 - f_u(i) + f_u(j) \big|_+
\end{align}

# Implementation

For the implementation part, we'll leverage the [`lightfm`](https://github.com/lyst/lightfm) library,

LightFM provides a function for fetching the [MovieLens 100K dataset](https://grouplens.org/datasets/movielens/100k/), which is a small recommender dataset, consisting of around 950 users, 1700 movies, and 100,000 ratings. The ratings are on a scale from 1 to 5, but we'll all treat them as implicit positive feedback in this example.

```python
# movielens is a dictionary and we
# can access the train and test key
# to obtain the training and test set
movielens = fetch_movielens()
train = movielens['train']
test = movielens['test']
train
```

    <943x1682 sparse matrix of type '<class 'numpy.int32'>'
    	with 90570 stored elements in COOrdinate format>

The next two code chunks compares the models' performance between a model that uses the BPR loss as its objective/loss function and another one using the WARP loss.

We'll use two metrics to evaluate the performance: precision@k and ROC AUC. Both are ranking metrics: to compute them, we'll be constructing recommendation lists for all of our users, and checking the ranking of known positive movies. For precision at k we'll be looking at whether they are within the first k results on the list; for AUC, we'll be calculating the probability that any known positive is higher on the list than a random negative example.

```python
start = time()
model = LightFM(learning_rate = 0.05, loss = 'bpr')
model.fit(train, epochs = 50)
print(time() - start)

train_precision = precision_at_k(model, train, k = 10).mean()
test_precision = precision_at_k(model, test, k = 10).mean()

train_auc = auc_score(model, train).mean()
test_auc = auc_score(model, test).mean()

print('Precision: train %.2f, test %.2f.' % (train_precision, test_precision))
print('AUC: train %.2f, test %.2f.' % (train_auc, test_auc))
```

    2.9854891300201416
    Precision: train 0.62, test 0.09.
    AUC: train 0.92, test 0.87.

```python
start = time()
model = LightFM(learning_rate = 0.05, loss = 'warp')
model.fit(train, epochs = 50)
print(time() - start)

train_precision = precision_at_k(model, train, k = 10).mean()
test_precision = precision_at_k(model, test, k = 10).mean()

train_auc = auc_score(model, train).mean()
test_auc = auc_score(model, test).mean()

print('Precision: train %.2f, test %.2f.' % (train_precision, test_precision))
print('AUC: train %.2f, test %.2f.' % (train_auc, test_auc))
```

    3.049008846282959
    Precision: train 0.65, test 0.11.
    AUC: train 0.95, test 0.91.

From the example above, we can see that despite the fact that WARP optimizes for precision@k its performance, it is able to obtain both a higher precision and ROC AUC score. Note that this is not saying this will be the case for every single example we've encountered, but based on this result, we can infer that taking the extra step to ensure the sampled negative item is violating the pairwise comparison and penalizing the loss function more if we were able to find a violating negative item quickly does in fact gives performance boost.

# Reference

- [Youtube: An Introduction to the Geometric Distribution](https://www.youtube.com/watch?v=zq9Oz82iHf0)
- [Blog: WARP loss for implicit-feedback recommendation](http://building-babylon.net/2016/03/18/warp-loss-for-implicit-feedback-recommendation/)
- [Jupyter Notebook: Learning-to-rank using the WARP loss](http://nbviewer.jupyter.org/github/lyst/lightfm/blob/master/examples/movielens/warp_loss.ipynb)
- [Jupyter Notebook: An implicit feedback recommender for the Movielens dataset](http://nbviewer.jupyter.org/github/lyst/lightfm/blob/master/examples/movielens/example.ipynb)
- [Paper: J. Weston, S. Bengio, N. Usunier - Wsabie: Scaling up to large vocabulary image annotation (2011)](http://www.thespermwhale.com/jaseweston/papers/wsabie-ijcai.pdf)