```python
from IPython.display import Image
from IPython.core.display import HTML
from sympy import *; x,h,y,t = symbols("x h y t")
Image(url= "https://i.imgur.com/4TwpNlR.png")
```

```python
# source https://nathancarter.github.io/how2data/site/how-to-find-the-critical-numbers-of-a-function-in-python-using-sympy/
f = t**4 - 12*t**3 + 16*t**2
d = diff(f)
solve(Eq(d,0))

```

    [0, 1, 8]

```python
Image(url= "https://i.imgur.com/EyFbCXI.png")
```


https://github.com/sachinruk/KerasQuantileModel

# Deep Quantile Regression

One area that Deep Learning has not explored extensively is the uncertainty in estimates. However, as far as decision making goes, most people actually require quantiles as opposed to true uncertainty in an estimate. eg. For a given age the weight of an individual will vary. What would be interesting is the (for arguments sake) the 10th and 90th percentile. The uncertainty of _the estimate_ of an individuals weight is less interesting.

```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Activation
import tensorflow.keras.backend as K

%matplotlib inline

mcycle = pd.read_csv('mcycle',delimiter='\t')
```

Standardise the inputs and outputs so that it is easier to train. I have't saved the mean and standard deviations, but you should.

```python
mcycle.times = (mcycle.times - mcycle.times.mean())/mcycle.times.std()
mcycle.accel = (mcycle.accel - mcycle.accel.mean())/mcycle.accel.std()
```

```python
mcycle
```

      times
      accel
      strata
      v

      0
      -1.684002
      0.430638
      1
      3.7

      1
      -1.669670
      0.404939
      1
      3.7

      2
      -1.626675
      0.377263
      1
      3.7

      3
      -1.598011
      0.430638
      1
      3.7

      4
      -1.569347
      0.377263
      1
      3.7

      ...
      ...
      ...
      ...
      ...

      89
      1.870317
      0.642161
      3
      138.0

      90
      1.956309
      0.140041
      3
      138.0

      91
      2.085296
      0.377263
      3
      138.0

      92
      2.113960
      0.377263
      3
      138.0

      93
      2.271612
      0.642161
      3
      138.0

94 rows × 4 columns

```python
model = Sequential()
model.add(Dense(units=10, input_dim=1,activation='relu'))
model.add(Dense(units=10, input_dim=1,activation='relu'))
model.add(Dense(1))
model.compile(loss='mae', optimizer='adadelta')
model.fit(mcycle.times.values, mcycle.accel.values, epochs=2000, batch_size=32, verbose=0)
model.evaluate(mcycle.times.values, mcycle.accel.values)
```

    3/3 [==============================] - 0s 332us/step - loss: 0.8030

    0.8030017018318176

```python
model.summary()
```

    Model: "sequential_1"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    dense_3 (Dense)              (None, 10)                20
    _________________________________________________________________
    dense_4 (Dense)              (None, 10)                110
    _________________________________________________________________
    dense_5 (Dense)              (None, 1)                 11
    =================================================================
    Total params: 141
    Trainable params: 141
    Non-trainable params: 0
    _________________________________________________________________

```python
t_test = np.linspace(mcycle.times.min(),mcycle.times.max(),200)
y_test = model.predict(t_test)

plt.scatter(mcycle.times, mcycle.accel)
plt.plot(t_test, y_test,'r')
plt.show()
```

## Quantiles 0.1, 0.5, 0.9

The loss for an individual data point is defined as:
$$
\begin{align}
\mathcal{L}(\xi_i|\alpha)=\begin{cases}
\alpha \xi_i &\text{if }\xi_i\ge 0, \\
(\alpha-1) \xi_i &\text{if }\xi_i< 0.
\end{cases}
\end{align}
$$
where $\alpha$ is the required quantile and $\xi_i = y_i - f(\mathbf{x}_i)$ and, $f(\mathbf{x}_i)$ is the predicted (quantile) model. The final overall loss is defines as:
$$\mathcal{L}(\mathbf{y},\mathbf{f}|\alpha)=\frac{1}{N} \sum_{i=1}^N \mathcal{L}(y_i-f(\mathbf{x}_i)|\alpha)$$

The following function defines the loss function for a quantile model.

**Note**: The following 4 lines is ALL that you change in comparison to a normal Deep Learning method, i.e. The loss function is all that changes.

```python
def tilted_loss(q,y,f):
    e = (y-f)
    return K.mean(K.maximum(q*e, (q-1)*e), axis=-1)
```

```python
def mcycleModel():
    model = Sequential()
    model.add(Dense(units=32, input_dim=1,activation='relu'))
    model.add(Dense(units=32,activation='relu'))
    model.add(Dense(1))

    return model
```

```python
qs = [0.1, 0.5, 0.9]

t_test = np.linspace(mcycle.times.min(),mcycle.times.max(),200)
plt.scatter(mcycle.times,mcycle.accel)

for q in qs:
    model = mcycleModel()
    model.compile(loss=lambda y,f: tilted_loss(q,y,f), optimizer='adadelta')
    model.fit(mcycle.times.values, mcycle.accel.values, epochs=2000, batch_size=32, verbose=0)

    # Predict the quantile
    y_test = model.predict(t_test)
    plt.plot(t_test, y_test, label=q) # plot out this quantile

plt.legend()
plt.show()
```

## Final Notes

1. Note that the quantile 0.5 is the same as median, which you can attain by minimising Mean Absolute Error, which you can attain in Keras regardless with `loss='mae'`.
2. Uncertainty and quantiles are **not** the same thing. But most of the time you care about quantiles and not uncertainty.
3. If you really do want uncertainty with deep nets checkout http://mlg.eng.cam.ac.uk/yarin/blog_3d801aa532c1ce.html

```python

```


*This notebook is part of  course materials for CS 345: Machine Learning Foundations and Practice at Colorado State University.
Original versions were created by Asa Ben-Hur.
The content is availabe [on GitHub](https://github.com/asabenhur/CS345).*

*The text is released under the [CC BY-SA license](https://creativecommons.org/licenses/by-sa/4.0/), and code is released under the [MIT license](https://opensource.org/licenses/MIT).*

<a href="https://colab.research.google.com/github//asabenhur/CS345/blob/master/notebooks/module08_01_neural_networks_mlp.ipynb">

</a>

```python
import numpy as np
import matplotlib.pyplot as plt
%autosave 0
%matplotlib inline
```

    Autosave disabled

# Neural Networks

Neural networks are models inspired by the workings of the human brain and the ability of brains to learn and adapt.  Neural networks build on the foundations of humble linear models like the perceptron and logistic regression.

### The XOR problem

Linear models do not always give acceptable levels of accuracy.  As a toy example consider the XOR problem, which is a canonical example of a problem that is not linearly separable:

```python
X = np.array([[0, 0], [1, 0], [0, 1], [1, 1]])
y = np.array([1, 0, 0, 1])

plt.figure(figsize=(3,3))
plt.scatter(X[:,0], X[:,1], c=y, s=100, alpha=0.5, cmap='bwr');
plt.xlabel('$x_1$')
plt.ylabel('$x_2$');
```

There is no line we can draw in the plane to successfully seperate the positive and negative examples.

### Recap: the perceptron

The perceptron was our first linear classifier.  It used the following rule:

$$f(\textbf{x}) = \text{step}(\textbf{w}^T \textbf{x} + b),$$

where $\text{step}$ refers to the step function

$$\text{step}(x) = \begin{cases} 1 & x \geq 0 \\ 0 & x < 0. \end{cases} $$

Previously we used the sign function because for convenience we used $\pm 1$ as the labels.

Think of the perceptron as a neural network with a single computational unit.  That neuron collects its input from the *input layer*, which also includes a constant unit that represents the bias.  The connections of the neuron have weights associated with them; those connections act to multiply the inputs received over those connections resulting in $\textbf{w}^T \textbf{x} + b$ as the input received by the neuron.  Whenever the input received by the neuron exceeds a certain threshold (e.g. 0), the neuron fires.  This is represented by the application of the step function to the input received by the neuron.
The step function is an example of an **activation function**, which specifies the response of the neuron to the input it receives.

Given features that are "good enough" - i.e. features that render the problem linearly separable, the perceptron can solve any classification problem.

Unfortunately, finding such features is not easy, and features that render a problem linearly separable are often not available.  This was a major criticism of perceptrons in Minsky and Papert's 1969 book *Perceptrons*, which demonstrated that perceptrons are not capable of addressing even simple problems like the XOR problem.  It took until the 1986 work of McClelland, Rumelhart, and Hinton on "multi-layer perceptrons" to turn things around.

### Multi-layer perceptrons

A multi-layer perceptron (MLP) is a classifier that consists of at least three layers of nodes:  an **input layer**, a **hidden layer** and an **output layer**.
The input layer receives the input, while each subsequent layer computes a function of the previous layer (usually nonlinear).  The output layer produces the resulting prediction of the MLP.  Here is a simple example:

First we'll consider an architecture where each node computes a function in exactly the same way as the perceptron. Here, the two red nodes, $h_1$ and $h_2$, represent perceptrons with

$$ h_1(\textbf{x}) = \text{step}(\textbf{w}_1^\top \textbf{x} + b_1) $$
and
$$ h_2(\textbf{x}) = \text{step}(\textbf{w}_2^\top \mathbf{x} + b_2). $$

Next, $h_1$ and $h_2$ are used as inputs to another perceptron, $f$.  For this perceptron, we'll drop the step function:

$$ f(\textbf{x}) = v_1 h_1(\textbf{x}) + v_2 h_2(\textbf{x}) + c $$
or
$$ f(\textbf{x}) = v_1 \text{step}(\textbf{w}_1^T \textbf{x} + b_1) + v_2 \text{step}(\textbf{w}_2^T \textbf{x} + b_2) + c. $$

It is useful to write this in vector-matrix form as

$$ f(\mathbf{x}) = \mathbf{v}^T \text{step} \left( W \mathbf{x} + \mathbf{b} \right) + c. $$

Here $\mathbf{b} = (b_1, b_2)^\top$ and the matrix $W$ is the result of stacking $\mathbf{w}_1$ and $\mathbf{w}_2$:

$$
W = \begin{pmatrix}
\mathbf{w}_1^\top \\
\mathbf{w}_2^\top
\end{pmatrix}
$$

As an example, here is such a multilayer perceptron that solves the XOR problem:

This perceptron uses the following values for the parameters as shown in the figure:
\begin{align}
\textbf{W} &= \begin{pmatrix} 1 & 1 \\ 1 & 1 \end{pmatrix} \\
\textbf{b} &= \begin{pmatrix} -1.5 \\ -0.5 \end{pmatrix} \\
\textbf{v} &= \begin{pmatrix} 1 \\ -1 \end{pmatrix} \\
c &= 1
\end{align}

To illustrate that it indeed solves the XOR problem let's plot the decision boundary of this network:

```python
from mpl_toolkits.mplot3d import Axes3D

fig = plt.figure(figsize=(8,4))
ax = fig.add_subplot(121)
ax2 = fig.add_subplot(122, projection='3d')

W = np.array([[1, 1], [1, 1]])
b = np.array([[-1.5], [-0.5]])
v = np.array([1, -1])
c = 1

delta = 0.01
xmin=-0.1;xmax=1.1;ymin=-0.1;ymax=1.1
xs = np.arange(xmin, xmax, delta)
ys = np.arange(ymin, ymax, delta)
x_grid, y_grid = np.meshgrid(xs, ys)

xs = x_grid.reshape(-1)
ys = y_grid.reshape(-1)
x = np.vstack((xs, ys))

Z = np.dot(v, np.heaviside(np.matmul(W, x) + b, 1)) + c
Z = Z.reshape(x_grid.shape)

im = ax.imshow(Z, origin='lower',
                cmap=plt.cm.bwr, alpha=0.5, extent=(xmin,xmax,ymin,ymax) )

X = np.array([[0, 0], [1, 0], [0, 1], [1, 1]])
y = np.array([1, 0, 0, 1])

ax.scatter(X[:,0], X[:,1], c=y, s=100, cmap='bwr', edgecolors='w');
ax.set_xlabel('$x_1$')
ax.set_ylabel('$x_2$')

ax2.plot_wireframe(x_grid, y_grid, Z, alpha=0.5);
ax2.scatter(X[:,0], X[:,1], y, c=y, s=100, cmap='bwr', edgecolors='w')
ax2.set_xlabel('$x_1$')
ax2.set_ylabel('$x_2$')
ax2.set_zlabel('f($x_1$, $x_2$)');
```

You can verify that this multilayer perceptron computes $\mathrm{XOR}(x_1, x_2)$ as $\mathrm{OR}(x_1, x_2)-\mathrm{AND}(x_1, x_2)$.

## More complex architectures

The architecture described earlier, with two hidden nodes is perhaps  the simplest multi-layer neural network architecture that can be constructed.  For added complexity you can add more hidden nodes:

In this architecture visualization $\textbf{x}$ represents any number of inputs to the network.

The set of nodes before the output is known as the **hidden layer**.  The number of nodes in that hidden layer is known as the "width" of the network.

Another way to allow the network to model more complex problems is to add more hidden layers to increase the "depth" of the network.  Here's an example of a network with two hidden layers:

This structure of successive layers is called a **multilayer perceptron**, and is also frequently referred to as a **fully-connected feedforward neural network**.  In general we can have any number of hidden layers and each hidden layer can have any number of neurons:

Modern neural networks can be very large both in depth and number of neurons per layer, which can add up to many parameters!

## Universal approximation

Neural networks being able to solve the XOR problem is great, but it doesn't mean they will be useful for problems in general.

It has been shown that neural networks can serve as "universal approximators", i.e. can serve to approximate any function.  Therefore they should be able to solve arbitrarily complex machine learning problems.
If you are interested, [chapter 4 of Neural networks and Deep Learning](http://neuralnetworksanddeeplearning.com/chap4.html) shows a relatively simple visual proof of this property.
It is worth noting that a network with a single hidden layer is a universal approximator given sufficient width.

### Training a neural network

To train a network, we need a method to update its weights.   Applying the perceptron update rule will no longer work: it will successfully update weights in the final layer, but the hidden neurons don't have a sense of a "correct" or "incorrect" classification, so they cannot be updated in that way.

The method that is used instead is gradient descent.  However, the output of the perceptron which uses the step function as the "activation function" is not differentiable.  So we need to choose an alternative to the step function.

### Activation functions

As mentioned above, the activation function describes how a neuron responds to its input.
The activation function that was used most commonly from the mid 1980s until the arrival of deep learning was the sigmoid and its sibling function, *tanh*, which is essentially a rescaling of the sigmoid to have a range from -1 to 1 instead of 0 to 1.  These can be considered as smooth versions of the step function, as they have similar behavior except near zero.

```python
def sigmoid(s):
    return 1/(1+np.exp(-s))

def sigmoid_derivative(s):
    return sigmoid(s) *(1-sigmoid(s))

def step(s) :
    return s > 0
```

```python
plt.figure(figsize=(6,4))
s = np.linspace(-10,10, 100)
plt.plot(s, sigmoid(s), label = r'$\sigma(s)$')
plt.plot(s, sigmoid_derivative(s), label = r"$\sigma'(s)$")
plt.plot(s, step(s), label = r'$step(s)$')

plt.xlabel(r"$s$", fontsize=20)
plt.legend(fontsize=15);
```

The sigmoid function has a severe problem: "vanishing gradients."  As you can see in the plot of the derivative, as the input to the sigmoid increases in magnitude, the derivative approaches zero very rapidly.  This means that if a hidden neuron in the network recieves a large input, the derivative to the weights resulting from that input will be very small.  This isn't typically a significant issue for small networks, but for modern networks that have many layers, it can become difficult to train the network effectively.

With the advent of "deep learning," the activation function that is used more commonly is the Rectified Linear Unit or ReLU.  ReLU is defined as

$$ \text{ReLU}(x) = \begin{cases} x, & x \geq 0 \\ 0, & x < 0 \end{cases} $$

Its derivative is

$$ \text{ReLU}'(x) = \begin{cases} 1, & x > 0 \\ 0, & x < 0 \end{cases} $$

with the derivative at zero being somewhat arbitrary (typically a value of 0, 0.5, or 1 is chosen) due to the piecewise nature of the function.

In addition to not suffering from the vanishing gradient problem, the ReLU function is faster to evaluate.

```python
def ReLU(x):
    return np.maximum(0, x)

def ReLU_derivative(x):
    dy = np.ones(x.shape)
    dy[x<0] = 0
    return dy
```

```python
fig = plt.figure(figsize=(8, 4))
ax1 = fig.add_subplot(121)
ax2 = fig.add_subplot(122)
plt.subplots_adjust(wspace=0.3)

x = np.linspace(-10, 10, 1000)
y = ReLU(x)

dy = ReLU_derivative(x)

ax1.plot(x, y)
ax1.set_xlabel('$x$', fontsize=15)
ax1.set_ylabel('ReLU$(x)$', fontsize=15)
ax1.set_title('ReLU')
ax2.plot(x, dy)
ax2.set_xlabel('$x$', fontsize=15)
ax2.set_ylabel("ReLU$'(x)$", fontsize=15)
ax2.set_title("ReLU derivative");
```

### The forward function

In feed forward neural networks the computation of the network output is often referred to as a *forward pass*.
For our simple network with a single hidden layer which has two neurons the output of the network can be written as:

$$
f(\textbf{x}) = v_1 \sigma \underbrace{\left(\textbf{w}_1^T \textbf{x} + b_1\right)}_{h_1(\mathbf{x})} + v_2 \sigma\underbrace{\left(\textbf{w}_2^T \textbf{x} + b_2\right)}_{h_2(\mathbf{x})} + c.
$$

Let's express that in code using Numpy:

```python
# first, create some random values for the parameters:
from numpy.random import default_rng
rng = default_rng(42)
w1 = rng.normal(0, 0.1, 2)
w2 = rng.normal(0, 0.1, 2)
b = rng.normal(0, 0.1, 2)
v = rng.normal(0, 0.1, 2)
c = rng.normal(0, 0.1, 1)[0]
```

```python
# we'll apply the network to the second element in the dataset:
x = X[1]
h1 = ReLU(np.dot(w1, x) + b[0])
h2 = ReLU(np.dot(w2, x) + b[1])

output = v[0] * h1 + v[1] * h2 + c
output
```

    -0.0016801157504288797

We can write that in a single line:

```python
output = (v[0] * ReLU(np.dot(w1, x) + b[0]) +
          v[1] * ReLU(np.dot(w2, x) + b[1]) + c)
output
```

    -0.0016801157504288797

By stacking together the weight vectors of the two hidden units we can express the output more succinctly:

```python
rng = default_rng(42)
W = rng.normal(0, 0.1, (2, 2))
b = rng.normal(0, 0.1, (2,1))
v = rng.normal(0, 0.1, (2,1))
c = rng.normal(0, 0.1, 1)[0]
```

We can do this either using `np.dot`

```python
np.dot(v.T, ReLU(np.dot(W, X.T) + b)) + c
```

    array([[-0.00168012, -0.00168012, -0.00168012, -0.00290978]])

or using the matrix multiplication operator:

```python
v.T @ ReLU(W @ X.T + b) + c
```

    array([[-0.00168012, -0.00168012, -0.00168012, -0.00290978]])

### Exercise

Write the forward function for a neural network with two hidden layers, each of which has two neurons. In other words, compute the output of a neural network with this architecture.

### Gradient Descent

Our two layer network with two hidden neurons is defined as

$$
f(\textbf{x}) = v_1 \sigma(\textbf{w}_1^T \textbf{x} + b_1) + v_2 \sigma(\textbf{w}_2^T \textbf{x} + b_2) + c.
$$

where $\sigma$ refers to any activation function, not necessarily the sigmoid function.
We can express this more explicitly as:

$$
f(\textbf{x}) = v_1 \sigma(w_{11}x_1 + w_{12}x_2 + b_1) + v_2 \sigma(w_{21}x_1 + w_{22}x_2 + b_2) + c.
$$

Next, we need a loss function that defines the errors of our network.  For a network with a single output we can use the squared loss, like we did in linear regression:

$$ L(\textbf{x}, y) = (f(\textbf{x}) - y)^2. $$

To be able to optimize the parameters using gradient descent, we need to compute the derivative of the loss function with respect to the parameters of the network.

First, we'll compute the derivatives with respect to the parameters of the output layer.  For example, the derivative of this with respect to $v_1$ is

\begin{align*}
\frac{\partial L(\textbf{x}, y)}{\partial v_1} &= \frac{\partial}{\partial v_1} (f(\textbf{x}) - y)^2 \\
                                               &= 2(f(\textbf{x}) - y) * \frac{\partial}{\partial v_1} f(\textbf{x}) \qquad \text{from the chain rule} \\
                                               &= 2(f(\textbf{x}) - y) * \frac{\partial}{\partial v_1} \left(v_1 \sigma(\textbf{w}_1^T \textbf{x} + b_1) + v_2 \sigma(\textbf{w}_2^T \textbf{x} + b_2) + c\right) \qquad \text{substituting in the network definition}\\
                                               &= 2(f(\textbf{x}) - y) * \frac{\partial}{\partial v_1} v_1 \sigma(\textbf{w}_1^T \textbf{x} + b_1) \qquad \text{removing terms that don't depend on $v_1$}\\
                                               &= 2(f(\textbf{x}) - y) * \sigma(\textbf{w}_1^T \textbf{x} + b_1)
\end{align*}

The derivatives for $v_2$ and $c$ are computed similarly, so we have:

\begin{align*}
\frac{\partial L(\textbf{x}, y)}{\partial v_1} &= 2(f(\textbf{x}) - y) * \sigma(\textbf{w}_1^T \textbf{x} + b_1) \\
\frac{\partial L(\textbf{x}, y)}{\partial v_2} &= 2(f(\textbf{x}) - y) * \sigma(\textbf{w}_2^T \textbf{x} + b_2) \\
\frac{\partial L(\textbf{x}, y)}{\partial c} &= 2(f(\textbf{x}) - y)
\end{align*}

Next, we'll compute the derivatives with respect to the parameters of the hidden layer.
The derivative of the loss with respect to $w_{11}$ is

\begin{align*}
\frac{\partial L(\textbf{x}, y)}{\partial w_{11}} &= \frac{\partial}{\partial w_{11}} (f(\textbf{x}) - y)^2 \\
                                               &= 2(f(\textbf{x}) - y) * \frac{\partial}{\partial w_{11}} f(\textbf{x})\\
                                               &= 2(f(\textbf{x}) - y) * \frac{\partial}{\partial w_{11}} \left(v_1 \sigma(w_{11}x_1 + w_{12}x_2 + b_1) + v_2 \sigma(w_{21}x_1 + w_{22}x_2 + b_2) + c\right) \\
                                               &= 2(f(\textbf{x}) - y) * \frac{\partial}{\partial w_{11}} v_1 \sigma(w_{11}x_1 + w_{12}x_2 + b_1) \\
                                               &= 2(f(\textbf{x}) - y) * v_1 \frac{\partial}{\partial w_{11}}  \sigma(w_{11}x_1 + w_{12}x_2 + b_1) \\
                                               &= 2(f(\textbf{x}) - y) * v_1 * \sigma'(w_{11}x_1 + w_{12}x_2 + b_1) * x_1
\end{align*}

The derivatives for the other parameters in this layer proceeds in a similar way. Overall we have:

\begin{align*}
\frac{\partial L(\textbf{x}, y)}{\partial v_1} &= 2(f(\textbf{x}) - y) * \sigma(\textbf{w}_1^T \textbf{x} + b_1) \\
\frac{\partial L(\textbf{x}, y)}{\partial v_2} &= 2(f(\textbf{x}) - y) * \sigma(\textbf{w}_2^T \textbf{x} + b_2) \\
\frac{\partial L(\textbf{x}, y)}{\partial c} &= 2(f(\textbf{x}) - y) \\
\frac{\partial L(\textbf{x}, y)}{\partial w_{11}} &= 2(f(\textbf{x}) - y) * v_1 * \sigma'(\textbf{w}_1^T \textbf{x} + b_1) * x_1 \\
\frac{\partial L(\textbf{x}, y)}{\partial w_{12}} &= 2(f(\textbf{x}) - y) * v_1 * \sigma'(\textbf{w}_1^T \textbf{x} + b_1) * x_2 \\
\frac{\partial L(\textbf{x}, y)}{\partial b_{1}} &= 2(f(\textbf{x}) - y) * v_1 * \sigma'(\textbf{w}_1^T \textbf{x} + b_1) \\
\frac{\partial L(\textbf{x}, y)}{\partial w_{21}} &= 2(f(\textbf{x}) - y) * v_2 * \sigma'(\textbf{w}_2^T \textbf{x} + b_2) * x_1 \\
\frac{\partial L(\textbf{x}, y)}{\partial w_{22}} &= 2(f(\textbf{x}) - y) * v_2 * \sigma'(\textbf{w}_2^T \textbf{x} + b_2) * x_2 \\
\frac{\partial L(\textbf{x}, y)}{\partial b_{2}} &= 2(f(\textbf{x}) - y) * v_2 * \sigma'(\textbf{w}_2^T \textbf{x} + b_2) \\
\end{align*}

This process of updating the weights is known as "backpropagation," as the derivative is propagated backwards through the network to determine the update rule for the weights.

### A simple neural network implementation

The following `neural_network` class initializes, trains, and handles prediction for a network with single hidden layer with two units in its hidden layer.

```python
from numpy.random import default_rng
import math

def ReLU(x):
    return max(0, x)

def ReLU_derivative(x):
    if x >= 0 :
        return 1
    else :
        return 0

def sigmoid(x):
    return 1/(1+math.exp(-x))

def sigmoid_derivative(x):
    return sigmoid(x)*(1-sigmoid(x))

class neural_network:
    def __init__(self, lr=0.05, epochs=100,
                 activation = ReLU,
                 activation_derivative = ReLU_derivative,
                 random_state=42):
        self.lr = lr
        self.epochs = epochs
        # set the network's activation function
        self.activation = activation
        self.activation_derivative = activation_derivative
        # create an instance of a random number generator for
        # initializing the weights randomly
        self.rng = default_rng(random_state)
        # save a history of the loss values
        self.loss_history = []

    def initialize_weights(self, input_dim):
        # weights are initialized to small random numbers
        self.w1 = self.rng.normal(0, 0.1, input_dim)
        self.w2 = self.rng.normal(0, 0.1, input_dim)
        self.b = self.rng.normal(0, 0.1, 2)
        self.v = self.rng.normal(0, 0.1, 2)
        self.c = self.rng.normal(0, 0.1, 1)[0]

    def decision_function(self, x):
        h1 = self.activation(
                np.dot(x, self.w1) + self.b[0])
        h2 = self.activation(
                np.dot(x, self.w2) + self.b[1])
        activation = self.v[0] * h1 + self.v[1] * h2 + self.c
        return activation

    def gradient_step(self, X, y):
        gradient_w1 = np.zeros_like(self.w1)
        gradient_w2 = np.zeros_like(self.w2)
        gradient_b = np.zeros_like(self.b)
        gradient_v = np.zeros_like(self.v)
        gradient_c = np.zeros_like(self.c)
        for i in range(len(X)):
            x = X[i]
            pred = self.decision_function(x)
            loss_derivative = 2*(pred - y[i])
            h1 = self.activation(
                np.dot(self.w1, x) + self.b[0])
            h2 = self.activation(
                np.dot(self.w2, x) + self.b[1])
            h1_derivative = self.activation_derivative(
                np.dot(self.w1, x) + self.b[0])
            h2_derivative = self.activation_derivative(
                np.dot(self.w2, x) + self.b[1])

            gradient_c += loss_derivative
            gradient_v += loss_derivative*np.array([h1, h2])
            gradient_b += loss_derivative*np.array(
                [self.v[0]*h1_derivative, self.v[1]*h2_derivative])
            gradient_w1 += loss_derivative*self.v[0]*h1_derivative*x
            gradient_w2 += loss_derivative*self.v[1]*h2_derivative*x
        self.v -= self.lr * gradient_v
        self.c -= self.lr * gradient_c
        self.w1 -= self.lr * gradient_w1
        self.w2 -= self.lr * gradient_w2
        self.b -= self.lr * gradient_b

    def loss(self, X, y):
        return np.mean([(self.decision_function(X[i]) - y[i])**2 for
                        i in range(len(X))])

    def fit(self, X, y):
        self.initialize_weights(X.shape[1])
        self.loss_history.append(self.loss(X, y))
        for _ in range(self.epochs):
            self.gradient_step(X, y)
            self.loss_history.append(self.loss(X, y))

    def predict(self, X):
        pred = np.array([self.decision_function(x) for x in X])
        return np.where(pred >= .5, 1, 0)

```

While testing this notebook, this training method found a correct solution for the XOR problem about 20-30% of the time.
There are ways to increase that significantly - standardizing the features, or using a different loss function than the sum-squared loss.

Let's run the network on the XOR problem:

```python
X = np.array([[0, 0], [1, 0], [0, 1], [1, 1]])
y = np.array([1, 0, 0, 1])

for i in range(10):
    network = neural_network(lr=0.05, epochs=300, random_state=i)
    network.fit(X, y)
    print('training set accuracy: ', np.mean(network.predict(X) == y))
    if np.mean(network.predict(X) == y) == 1.0 :
        break
print('It took {} runs for a correct solution to be found.'.format(i+1))
```

    training set accuracy:  0.5
    training set accuracy:  1.0
    It took 2 runs for a correct solution to be found.

```python
plt.plot(network.loss_history);
plt.xlabel('epochs')
plt.ylabel('loss');
```

Next, we'll see how the decision boundary looks like:

```python
fig = plt.figure(figsize=(8,4))
ax = fig.add_subplot(121)
ax2 = fig.add_subplot(122, projection='3d')

delta = 0.01
xmin=-0.1;xmax=1.1;ymin=-0.1;ymax=1.1
xs = np.arange(xmin, xmax, delta)
ys = np.arange(ymin, ymax, delta)
x_grid, y_grid = np.meshgrid(xs, ys)

xs = x_grid.reshape(-1)
ys = y_grid.reshape(-1)
x = np.vstack((xs, ys)).T

Z = np.array([network.decision_function(xx) for xx in x])
Z = Z.reshape(x_grid.shape)

im = ax.imshow(Z, origin='lower',
               cmap=plt.cm.bwr, extent=(xmin,xmax,ymin,ymax),
               vmin=0, vmax=1)

X = np.array([[0, 0], [1, 0], [0, 1], [1, 1]])
y = np.array([1, 0, 0, 1])

plt.style.use('default')
ax.scatter(X[:,0], X[:,1], c=y, s=50, cmap='bwr', edgecolors='w');
ax.plot([], [], 'or', label='0')
ax.plot([], [], 'ob', label='1')
ax.set_xlabel('$x_1$')
ax.set_ylabel('$x_2$')
ax.legend()

ax2.view_init(30, 120)
ax2.plot_wireframe(x_grid, y_grid, Z, alpha=0.5);
ax2.scatter(X[:,0], X[:,1], y, c=y, s=30, alpha=0.7, cmap='bwr')
ax2.set_xlabel('$x_1$')
ax2.set_ylabel('$x_2$')
ax2.set_zlabel('f($x_1$, $x_2$)');
```

```python
from sklearn.datasets import load_boston
from sklearn.preprocessing import StandardScaler

X_boston,y_boston = load_boston(return_X_y=True)
scaler = StandardScaler()
X_scale = scaler.fit_transform(X_boston)
```

```python
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X_scale, y_boston, test_size=0.3, random_state=42)
```

```python
network = neural_network(lr=0.001, epochs=10,
                         activation = sigmoid,
                         activation_derivative = sigmoid_derivative,
                         random_state=1)
network.fit(X_train, y_train)
```

```python
plt.plot(network.loss_history);
plt.xlabel('epochs')
plt.ylabel('loss');
```

```python
# note that the network's "predict" function is designed for
# producing a classification, so we use "decision_function"
# for obtaining predictions for regression
y_pred = np.array([network.decision_function(x) for x in X_test])
```

```python
np.sqrt(np.mean( (y_pred - y_test)**2 ))
```

    7.195766596558915

For comparison here are results using linear regression:

```python
from sklearn.linear_model import LinearRegression
lr = LinearRegression()
lr.fit(X_train, y_train)
y_pred_lr = lr.predict(X_test)
np.sqrt(np.mean( (y_pred_lr - y_test)**2 ))
```

    4.638689926172827

Clearly there is room for improvement since we didn't do as well as linear regression.

### Exercises

* Update the `neural_network` class to have one more node in its hidden layer.
* With this more flexible network, how does the frequency with which training finds solutions that correctly classify all examples in the XOR problem?

### Comments

1. Libraries such as PyTorch and TensorFlow perform automatic differentiation, making it unnecessary to perform differentiation yourself.  These libraries allow for very complex architectures that would have been close to impossible otherwise.  Keras, which is an easy to use interface to TensorFlow, will be presented in a future notebook.

2. There are many more types of neural network layers.  We will discuss a few in upcoming notebooks.

### Pros and cons of feed forward networks

#### Pros

- Highly flexible - you can choose almost every aspect of the architecture:  number of layers, number of neurons per layer, activation function, loss function, initialization, and many more.
- Can provide high accuracy if you choose the right network architecture

#### Cons

- Training on a CPU can take a long time.  Solution:  GPUs/TPUs
- Difficult to interpret
- Many options to tweak means that selecting hyperparameters can be a challenge
- Difficult to get better performance than random forests or support-vector machines.

## References

Rosenblatt, Frank. "The perceptron: a probabilistic model for information storage and organization in the brain." Psychological review 65.6 (1958): 386.

Minsky, Marvin and Seymour Papert. Perceptrons: An Introduction to Computational Geometry.  The MIT Press, 1969.

Rumelhart, David E.; Hinton, Geoffrey E.; Williams, Ronald J. (1986). "Learning Internal Representations by Error Propagation". In Rumelhart, David E.; McClelland, James L. (eds.). Parallel Distributed Processing : Explorations in the Microstructure of Cognition. Volume 1 : Foundations. Cambridge: MIT Press.

### A more vectorized implementation

Below you will find another implementation of the same simple neural network that introduces more vectorized computations.

```python
def ReLU2(x):
    return np.maximum(0, x)

def ReLU_derivative2(x):
    dy = np.ones(x.shape)
    dy[x<0] = 0
    return dy

class neural_network2 :
    def __init__(self, lr=0.01, epochs=100, random_state=42):
        self.lr = lr
        self.epochs = epochs
        self.rng = default_rng(random_state)
        # Initialize the weights to small random values
        self.initialize_weights()
        self.loss_history = []
        # Use ReLU for the network's activation function
        self.activation_function = ReLU2
        self.activation_deriv = ReLU_derivative2

    def initialize_weights(self):
        self.W = self.rng.normal(0, 0.1, (2, 2))
        self.b = self.rng.normal(0, 0.1, (2, 1))
        self.v = self.rng.normal(0, 0.1, (2))
        self.c = self.rng.normal(0, 0.1, 1)

    def decision_function(self, X):
        return np.dot(self.v,
            self.activation_function(
                np.dot(self.W, X.T) + self.b)) + self.c

    def gradient_step(self, X, y):
        # we are using gradient descent and not SGD
        # so we keep track of the old parameter values
        # in order to avoid updating the weights
        W = np.copy(self.W)
        b = np.copy(self.b)
        v = np.copy(self.v)
        c = np.copy(self.c)

        for i in range(X.shape[0]):
            x = X[i]
            # calculate quantities of interest
            pred = np.dot(v, self.activation_function(
                np.dot(W, x).reshape(2, -1) + b)) + c
            loss_derivative = 2*(pred - y[i])
            h1 = self.activation_function(np.dot(W[0], x) + b[0])
            h1_derivative = self.activation_deriv(np.dot(W[0], x) + b[0])
            h2 = self.activation_function(np.dot(W[1], x) + b[1])
            h2_derivative = self.activation_deriv(np.dot(W[1], x) + b[1])

            # update weights using the contribution of example i to the
            # gradient
            self.v[0] -= self.lr*loss_derivative*h1
            self.v[1] -= self.lr*loss_derivative*h2
            self.c -= self.lr*loss_derivative
            self.W[0] -= self.lr*loss_derivative*v[0]*h1_derivative*x
            self.b[0] -= (self.lr*loss_derivative*v[0]*
                          h1_derivative).reshape(-1)
            self.W[1] -= self.lr*loss_derivative*v[1]*h2_derivative*x
            self.b[1] -= (self.lr*loss_derivative*v[1]*
                          h2_derivative).reshape(-1)

    def loss(self, X, y):
        return np.mean((self.decision_function(X) - y)**2)

    def fit(self, X, y):
        loss = self.loss(X, y)
        self.loss_history.append(loss)
        for _ in range(self.epochs):
            self.gradient_step(X, y)
            self.loss_history.append(self.loss(X, y))

    def predict(self, X):
        pred_prob = self.decision_function(X)
        pred_class = np.where(pred_prob >= .5, 1, 0)
        return np.squeeze(pred_class)

```

```python
X = np.array([[0, 0], [1, 0], [0, 1], [1, 1]])
y = np.array([1, 0, 0, 1])

for i in range(10):
    network = neural_network2(lr=0.05, epochs=300, random_state=i)
    network.fit(X, y)
    print('training set accuracy: ', np.mean(network.predict(X) == y))
    if np.mean(network.predict(X) == y) == 1.0 :
        break
print('It took {} runs for a correct solution to be found.'.format(i+1))
```

    training set accuracy:  0.5
    training set accuracy:  1.0
    It took 2 runs for a correct solution to be found.


# Differentiation vs. integration approach

# Purpose
The paper claims that the integration approch is better than the differentiation approach as a Parameter Identification Technique. This notebook investigates this statement.

# Methodology
PIT using the two methods is used on a simulated signal with Gaus distributed measurement noise.

# Setup

```python
# %load imports.py
"""
These is the standard setup for the notebooks.
"""

%matplotlib inline
%load_ext autoreload
%autoreload 2

import sys
sys.path.append("../../")

import pandas as pd
pd.options.display.max_rows = 999
pd.options.display.max_columns = 999
pd.set_option("display.max_columns", None)
import numpy as np
import os
import matplotlib.pyplot as plt
from collections import OrderedDict
import copy
from sklearn.pipeline import Pipeline
from rolldecayestimators.transformers import CutTransformer, LowpassFilterDerivatorTransformer, ScaleFactorTransformer, OffsetTransformer
from rolldecayestimators.direct_estimator_cubic import EstimatorLinear
import src.equations as equations
import rolldecayestimators.lambdas as lambdas
from rolldecayestimators.substitute_dynamic_symbols import lambdify, run
import rolldecayestimators.symbols as symbols
import sympy as sp

from sympy.physics.vector.printing import vpprint, vlatex
from IPython.display import display, Math, Latex

from sklearn.metrics import r2_score
import shipflowmotionshelpers.shipflowmotionshelpers as helpers
import src.visualization.visualize as visualize
import scipy
from copy import deepcopy
import joblib
```

    Duplicate key in file WindowsPath('C:/Users/maa/.matplotlib/stylelib/paper.mplstyle'), line 462 ('figure.figsize   : 5, 3   ## figure size in inches')
    Duplicate key in file WindowsPath('C:/Users/maa/.matplotlib/stylelib/paper.mplstyle'), line 463 ('figure.dpi       : 100        ## figure dots per inch')

```python
from rolldecayestimators.direct_estimator_cubic import EstimatorCubic,EstimatorQuadraticB
from sklearn.base import clone
from copy import deepcopy
from numpy.random import normal

import reports.examples.analytical_numerical
```

```python
eq_phi1d = sp.Eq(symbols.phi_dot_dot,
      sp.solve(equations.roll_decay_equation_quadratic_A,symbols.phi_dot_dot)[0])

accelaration_lambda = lambdify(sp.solve(eq_phi1d,symbols.phi_dot_dot)[0])
```

```python
omega0_eq = equations.omega0_equation.subs(symbols.C,symbols.C_1)
omega0_lambda = lambdify(sp.solve(omega0_eq, symbols.omega0)[0])
```

```python

df = reports.examples.analytical_numerical.show(omega0_lambda=omega0_lambda,
                                                accelaration_lambda=accelaration_lambda, plot=False)
df_true = df.drop(columns=['phi1d'])
```

```python
df_true.head()
```

      phi

      0.00
      0.174533

      0.01
      0.174489

      0.02
      0.174359

      0.03
      0.174142

      0.04
      0.173838

```python
df_measure = df_true.copy()
np.random.seed(42)
df_measure['noise'] = normal(loc=0, scale=np.deg2rad(0.5), size=len(df_measure))
df_measure['phi']+=df_measure['noise']
```

## Analyze the damping

```python
np.random.seed(42)
lowpass_filter = LowpassFilterDerivatorTransformer(cutoff=1, minimum_score=0.0)
cutter = CutTransformer(phi_max=np.deg2rad(15), phi_min=np.deg2rad(0), phi1d_start_tolerance=0.015)
offset_transformer = OffsetTransformer()
```

```python
fig,ax=plt.subplots()

df.plot(y='phi1d',style='-', color='grey', label='true', ax=ax, alpha=0.75)

lowpass_filter.fit(df_measure).transform(df_measure).plot(y='phi1d',style='b-', label='filtered', ax=ax)
```

```python
np.random.seed(42)
estimator_diff = EstimatorLinear(fit_method='derivation', maxfev=100000)

steps = [
    ('filter',lowpass_filter),
    #('cutter', cutter),
    ('estimator', estimator_diff)
]

pipline_estimator_diff = Pipeline(steps=steps)
pipline_estimator_diff.fit(X=df_measure[['phi']])
df_diff = pipline_estimator_diff.predict(df_true)
pipline_estimator_diff.score(df_true)
```

    0.9262226087157728

```python
pipline_estimator_diff['estimator'].parameters
```

    {'C_1A': 4.528247565167409, 'B_1A': 0.3300282442976332}

```python
estimator_int= EstimatorLinear(fit_method='integration', maxfev=100000, p0=estimator_diff.parameters)

steps = [
    ('filter',lowpass_filter),
    #('cutter', cutter),
    ('estimator', estimator_int)
]

pipline_estimator_int = Pipeline(steps=steps)
pipline_estimator_int.fit(X=df_measure[['phi']])
df_int = pipline_estimator_int.predict(df_true)
pipline_estimator_int.score(df_true)
```

    0.9802886393850015

```python
pipline_estimator_int['estimator'].parameters
```

    {'C_1A': 4.74725600798353, 'B_1A': 0.3261232770509625}

```python
fig,ax=plt.subplots()

df_measure.plot(y='phi',style='-', color='grey', label='measurement', ax=ax, alpha=0.5)
df_diff.plot(y='phi',style='r-', label='differentiation', ax=ax)
df_int.plot(y='phi',style='g-', label='integration', ax=ax)
df_true.plot(y='phi',style='k:', label='true', ax=ax)

ax.set_ylabel(r'$\phi$ $[rad]$')
ax.set_xlabel(r'time $[s]$')
ax.grid(True)
ax.legend()
```

```python
df_compare = df_true.copy()

frames = {
    'measure' : df_measure,
    'diff' : df_diff,
    'int' : df_int,
}

for name, df_ in frames.items():

    df_compare = pd.merge(left=df_compare, right=df_[['phi']], how='left',
                      left_index=True, right_index=True, suffixes=('','_%s' % name))

df_compare_diff = df_compare.sub(df_compare['phi'], axis='rows')
```

```python
df_compare_diff.plot()
```

## Vary filter:

```python
def vary_cutoff(cutoff:float, fit_method:str):

    lowpass_filter = LowpassFilterDerivatorTransformer(cutoff=cutoff, minimum_score=0.0)

    estimator = EstimatorLinear(fit_method='derivation', maxfev=100000)
    steps = [
        ('filter',lowpass_filter),
        ('estimator', estimator)
    ]

    pipeline = Pipeline(steps=steps)
    pipeline.fit(X=df_measure[['phi']])

    if fit_method is 'integration':
        estimator = EstimatorLinear(fit_method='integration', maxfev=100000,
                                         p0=estimator.parameters)
        steps = [
            ('filter',lowpass_filter),
            ('cutter', cutter),
            ('estimator', estimator)
        ]
        pipeline = Pipeline(steps=steps)
        pipeline.fit(X=df[['phi']])

    return pipeline['estimator'].parameters
```

```python
N = 10
np.random.seed(42)

cutoffs = np.linspace(0.7,6,N)
df_vary = pd.DataFrame()
for cutoff in cutoffs:

    for method in ['differentiation','integration']:
        s = pd.Series(vary_cutoff(cutoff=cutoff, fit_method=method))
        s['method'] = method
        s['cutoff'] = cutoff
        df_vary = df_vary.append(s, ignore_index=True)

```

```python
s_true = pd.Series()
A_44 = 1.0
B_1 = 0.3
C_1 = 5.0
s_true['B_1A'] = B_1/A_44
s_true['B_2A'] = 0

s_true['C_1A'] = C_1/A_44
s_true['C_3A'] = 0
s_true['C_5A'] = 0

```

```python
s_true
```

    B_1A    0.3
    B_2A    0.0
    C_1A    5.0
    C_3A    0.0
    C_5A    0.0
    dtype: float64

```python
df_error = df_vary.select_dtypes(exclude='object').sub(s_true).abs()
df_error['method'] = df_vary['method']
df_error['cutoff'] = df_vary['cutoff']
```

```python
fig,ax=plt.subplots()
for method, df_ in df_error.groupby(by='method'):
    df_.plot(x='cutoff', y='B_1A', ax=ax, label=method)

```

```python
fig,ax=plt.subplots()
for method, df_ in df_vary.groupby(by='method'):
    df_.plot(x='cutoff', y='B_1A', style='.-', ax=ax, label=method)

ax.set_ylabel(r'$B_1$ $[Nm \cdot s]$')
ax.set_xlabel(r'Low pass filter cutoff frequency [1/s]')
ax.grid(True)
```

```python
x,y,z = sp.symbols('x y z')
expression = x + y - z
expression
```

$\displaystyle x + y - z$

```python
expression.func
```

    sympy.core.add.Add

```python
expression.args
```

    (x, y, -z)

```python
X = lowpass_filter.fit(df_measure).transform(df_measure)
X.head()
```

      phi
      noise
      phi_filtered
      phi1d
      phi2d

      0.00
      0.178868
      0.004335
      0.183017
      -0.110489
      -0.172108

      0.01
      0.173283
      -0.001207
      0.181912
      -0.112210
      -0.259432

      0.02
      0.180011
      0.005652
      0.180773
      -0.115677
      -0.349288

      0.03
      0.187432
      0.013291
      0.179599
      -0.119195
      -0.354329

      0.04
      0.171795
      -0.002043
      0.178389
      -0.122764
      -0.359324

```python
subs = [
    (equations.new_symbols[symbols.B_2],0),
    (equations.new_symbols[symbols.B_3],0),
    (equations.new_symbols[symbols.C_3],0),
    (equations.new_symbols[symbols.C_5],0),
]
eq = equations.roll_decay_equation_cubic_A.subs(subs)
eq
```

$\displaystyle B_{1A} \frac{d}{d t} \phi{\left(t \right)} + C_{1A} \phi{\left(t \right)} + \frac{d^{2}}{d t^{2}} \phi{\left(t \right)} = 0$

```python
lhs = eq.lhs
lhs.free_symbols
```

    {B_1A, C_1A, t}

```python
for arg in lhs.args:
    lambdify(arg)
```

```python
arg
```

$\displaystyle \frac{d^{2}}{d t^{2}} \phi{\left(t \right)}$

```python
def remove_free(arg):
    new_arg = arg.copy()
    for symbol in arg.free_symbols:
        if symbol.name == symbols.t.name:
            continue
        new_arg = new_arg.subs(symbol,1)
    return new_arg
```

```python
def create_features_and_label(X, lhs):

    args = []
    free_symbols = []

    for arg in lhs.args:
        new_arg = remove_free(arg)

        f = list(arg.free_symbols - set([sp.symbols('t')]))
        if len(f) == 0:
            y_arg = new_arg
        elif len(f) == 1:
            free_symbol = f[0]  # assuming one...
            free_symbols.append(free_symbol)
            args.append(new_arg)
        elif len(f) > 1:
            raise ValueError('More than one free symbol')

    datas = []
    for arg in args:
        func = lambdify(arg)
        datas.append(run(func, X))
    X_ = np.array(datas).T

    y_func = lambdify(y_arg)
    y_ = -run(y_func, X)

    return X_, y_, free_symbols
```

```python
X_, y_, free_symbols = create_features_and_label(X=X, lhs=eq.lhs)
```

```python
p, res, rnk, s = scipy.linalg.lstsq(X_, y_)
p
```

    array([0.33002824, 4.52824757])

```python
free_symbols
```

    [B_1A, C_1A]

```python
eq2 = sp.Eq(symbols.phi,
            sp.solve(eq,symbols.phi)[0])
eq2
```

$\displaystyle \phi{\left(t \right)} = - \frac{B_{1A} \frac{d}{d t} \phi{\left(t \right)} + \frac{d^{2}}{d t^{2}} \phi{\left(t \right)}}{C_{1A}}$

```python
def simulate(X,p):
    B_1A_ = p[0]
    C_1A_ = p[1]

    phi = -B_1A_/C_1A_*X['phi1d'] -X['phi2d']/C_1A_
    return phi
```

```python
#phi = X_2.dot(p)
phi = simulate(X=X,p=p)

fig,ax=plt.subplots()
df_true.plot(y='phi', ax=ax)
ax.plot(df_true.index, phi)
```

```python
def vary_cutoff2(df:pd.DataFrame, cutoff:float, eq):
    lowpass_filter = LowpassFilterDerivatorTransformer(cutoff=cutoff, minimum_score=0.0)
    X = lowpass_filter.fit(df).transform(df)

    X_, y_, free_symbols = create_features_and_label(X=X, lhs=eq.lhs)

    return scipy.linalg.lstsq(X_, y_)

```

```python
df_vary2 = pd.DataFrame()

for cutoff in cutoffs:

    p, res, rnk, s = vary_cutoff2(df=df_measure, cutoff=cutoff, eq=eq)

    s = pd.Series(name=cutoff)
    s['B_1A'] = p[0]
    s['C_1A'] = p[1]
    df_vary2 = df_vary2.append(s)
```

```python
fig,ax=plt.subplots()
df_vary2.plot(y='B_1A', style='.-', ax=ax)
ax.grid(True)
```

```python

```


# Water heating

An insulated, rigid tank contains 4 kg of water at 100 kPa, where initially 0.25 of the mass is liquid. An electric heater turns on and operates until all of the liquid has vaporized. (Neglect the heat capacity of the tank and heater.)

Figure: Rigid tank containting water and electric heater

**Problem:**
- Determine the final temperature and pressure of the water.
- Determine the electrical work required by this process.
- Determine the total change in entropy associated with this process.
- Plot the state points for the water on a temperature-specific entropy diagram.

First, load the necessary modules and specify the known/initial conditions.

```python
%matplotlib notebook
import matplotlib.pyplot as plt
import numpy as np
import cantera as ct

from pint import UnitRegistry
ureg = UnitRegistry()
Q_ = ureg.Quantity
```

```python
mass = Q_(4, 'kg')
pressure_initial = Q_(100, 'kPa')
quality_initial = 0.25

quality_final = 1.0

# specify the initial state using pressure and quality
state_initial = ct.Water()
state_initial.PX = pressure_initial.to('Pa').magnitude, quality_initial
state_initial()
```

      water:

           temperature         372.809  K
              pressure          100000  Pa
               density         2.35669  kg/m^3
      mean mol. weight          18.016  amu
        vapor fraction            0.25

                              1 kg            1 kmol
                           -----------      ------------
              enthalpy    -1.49887e+07         -2.7e+08     J
       internal energy    -1.50312e+07       -2.708e+08     J
               entropy         6337.06        1.142e+05     J/K
        Gibbs function    -1.73512e+07       -3.126e+08     J
     heat capacity c_p    inf              inf              J/K
     heat capacity c_v         20936.9        3.772e+05     J/K

## Find final temperature and pressure

Due to conservation of mass, since the mass and volume of the system are fixed, the specific volume and density must be constant:
\begin{align}
v_2 &= v_1 \\
\rho_2 &= \rho_1
\end{align}
Therefore the final state is fixed by the density and quality, where $x_2 = 1$:

```python
state_final = ct.Water()
# rather than use
state_final.DX = state_initial.density, quality_final
```

Hmm, what happened here? It looks like Cantera unfortunately does not support specifying the thermodynamic state using density and quality. (With quality as one property, it only supports temperature or pressure as the other property.)

Fortunately, CoolProp *does* support specifying the state the way we need to solve this problem, so let's use that for the final state:

```python
from CoolProp.CoolProp import PropsSI

temp_final = PropsSI(
    'T', 'D', state_initial.density, 'Q', quality_final, 'water'
    ) * ureg.kelvin
pres_final = PropsSI(
    'P', 'D', state_initial.density, 'Q', quality_final, 'water'
    ) * ureg.pascal

print(f'Final temperature: {temp_final: .2f}')
print(f'Final pressure: {pres_final: .2f}')

# We can then set the final state using the Cantera object,
# now that we know temperature
state_final = ct.Water()
state_final.TX = temp_final.magnitude, quality_final
```

    Final temperature: 420.08 kelvin
    Final pressure: 438257.38 pascal

## Find electrical work required

To find the work required, we can do an energy balance on the (closed) system:
\begin{equation}
W_{\text{in}} = m (u_2 - u_1)
\end{equation}

```python
work = mass * (Q_(state_final.u, 'J/kg') - Q_(state_initial.u, 'J/kg'))
print(f'Electrical work required: {work.to(ureg.megajoule): .2f}')
```

    Electrical work required: 6.47 megajoule

## Find entropy change

The total entropy change is the change in entropy of the system plus that of the surroundings:
\begin{align}
\Delta S_{\text{total}} &= \Delta S_{\text{system}} + \Delta S_{\text{surr}} \\
\Delta S_{\text{total}} &= \Delta S_{\text{system}} = m (s_2 - s_1)
\end{align}
since the entropy change of the surroundings is zero.

```python
entropy_change = mass * (Q_(state_final.s, 'J/kg') - Q_(state_initial.s, 'J/kg'))
print(f'Entropy change: {entropy_change: .2f}')
```

    Entropy change: 16195.98 joule

This process is irreversible, associated with a positive increase in total entropy.

## Plot the state points for water

We can construct the saturated liquid and saturated vapor lines in a temperature–specific entropy diagram (T–s diagram), and then plot the initial and final states locations along with the process line (of constant density):

```python
f = ct.Water()

# Array of temperatures from fluid minimum temperature to critical temperature
temps = np.arange(np.ceil(f.min_temp) + 0.15, f.critical_temperature, 0.1)

def get_sat_entropy_fluid(T):
    '''Gets entropy for temperature along saturated liquid line'''
    f = ct.Water()
    f.TX = T, 0.0
    return f.s
def get_sat_entropy_gas(T):
    '''Gets entropy for temperature along saturated vapor line'''
    f = ct.Water()
    f.TX = T, 1.0
    return f.s

# calculate entropy values associated with temperatures along
# saturation lines
entropies_f = np.array([get_sat_entropy_fluid(T) for T in temps])
entropies_g = np.array([get_sat_entropy_gas(T) for T in temps])

# critical point
f.TP = f.critical_temperature, f.critical_pressure

fig, ax = plt.subplots(figsize=(5, 3))

# Plot the saturated liquid line, critical point,
# and saturated vapor line
ax.plot(entropies_f, temps)
ax.plot(f.s, f.T, 'o')
ax.plot(entropies_g, temps)
plt.xlabel('Specific entropy (J/kg⋅K)')
plt.ylabel('Temperature (K)')

# Plot the initial and final states, and label them
ax.plot(state_initial.s, state_initial.T, 's')
ax.annotate('(1)', xy=(state_initial.s, state_initial.T),
            xytext=(0, -20), textcoords='offset points',
            ha='right', va='bottom'
            )

ax.plot(state_final.s, state_final.T, 's')
ax.annotate('(2)', xy=(state_final.s, state_final.T),
            xytext=(20, 0), textcoords='offset points',
            ha='right', va='bottom'
            )

# show process line of constant density
temps = np.arange(state_initial.T, state_final.T, 0.1)
def get_entrophy(T, density):
    f = ct.Water()
    f.TD = T, density
    return f.s
entropies = np.array([get_entrophy(T, state_initial.density) for T in temps])
ax.plot(entropies, temps, '--')

plt.grid(True)
fig.tight_layout()
plt.show()
```

    <IPython.core.display.Javascript object>

```python

```


```python
!pip install nangs
```

    Requirement already satisfied: nangs in /usr/local/lib/python3.7/dist-packages (0.1.2)

```python
# autoreload nangs
%reload_ext autoreload
%autoreload 2
%matplotlib inline
```

```python

```

```python
#imports

import numpy as np
import matplotlib.pyplot as plt
import torch
import pandas as pd
import nangs
from nangs import *

device = "cuda" if torch.cuda.is_available() else "cpu"

nangs.__version__, torch.__version__
```

    ('0.1.2', '1.9.0+cu102')

```python
class Sine(torch.nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, x):
        return torch.sin(x)

def block(i, o):
    fc = torch.nn.Linear(i, o)
    return torch.nn.Sequential(
        #Sine(),
        torch.nn.GELU(),
        torch.nn.Linear(i, o)
    )

class MLP(torch.nn.Module):
    def __init__(self, inputs, outputs, layers, neurons):
        super().__init__()
        fc_in = torch.nn.Linear(inputs, neurons)
        fc_hidden = [
            block(neurons, neurons)
            for layer in range(layers-1)
        ]
        fc_out = block(neurons, outputs)

        self.mlp = torch.nn.Sequential(
            fc_in,
            *fc_hidden,
            fc_out
        )

    def forward(self, x):
        return self.mlp(x)
```

```python
device

```

    'cuda'

```python
alpha=10
class Difussion2d(PDE):
    def computePDELoss(self, inputs, outputs):
        p = outputs[:, 0]
        grads = self.computeGrads(p, inputs)
        dpdx, dpdt = grads[:, 0], grads[:, 1]
        # second order derivatives
        dp2dx2 = self.computeGrads(dpdx, inputs)[:, 0]
        #dp2dy2 = self.computeGrads(dpdy, inputs)[:, 1]
        return {'pde':dpdt - dp2dx2}
```

```python
class NeumannX(Neumann):
    def computeBocoLoss(self, inputs, outputs):
        p = outputs[:, 0]
        grads = self.computeGrads(p, inputs)
        dpdx = grads[:, 0]
        return {'gradX': dpdx}
```

```python
class PBC0(Neumann):
    def computeBocoLoss(self, inputs, outputs):
        pbc = 0
        p = outputs[:, 0]
        return {'PWF': 100*(p-pbc)}
```

```python
pde = Difussion2d(inputs=('x', 't'), outputs='p')
```

```python
def append(arr, values, axis=None):
    arr = np.asanyarray(arr)
    if axis is None:
        if arr.ndim != 1:
            arr = arr.ravel()
        values = np.ravel(values)
        axis = arr.ndim-1
    return np.concatenate((arr, values), axis=axis)
```

```python
# mesh

x = np.linspace(0,1,100)

t = np.linspace(0,1,100)
#t = append(t,np.linspace(0.2,1,10))
#t=np.array([0])
#tt=np.logspace(-2,0,20)
#t = append(t,tt)
mesh = Mesh({'x': x, 't':t}, device=device)
pde.set_mesh(mesh)
```

```python
from nangs import Dirichlet
```

```python
t0 = np.array([0])
```

```python
_x = np.meshgrid(x)
#p0 = np.sin(2*np.pi*_x)*np.sin(2*np.pi*_y)
```

```python
_x[0].shape
```

    (100,)

```python
p0=np.zeros((100))
p0[:]=1
p0[0]=0
#p0[47:53,47:53]=1
```

```python
p0.shape
```

    (100,)

```python
from mpl_toolkits.mplot3d import Axes3D
import matplotlib.pyplot as plt
from matplotlib import cm
fig = plt.figure()
#ax = fig.gca(projection='3d')
plt.plot(_x[0], p0)
#fig.colorbar(surf, shrink=0.5, aspect=5)
#plt.show()
```

```python
initial_condition = Dirichlet({'x': x,'t':t0}, {'p': p0.reshape(-1)}, device=device, name="initial")
```

```python
p0.shape
```

    (100,)

```python
pde.add_boco(initial_condition)
#p0=np.ones(len(y))
#p1=np.zeros(len(y))
```

```python
pwf = PBC0({'x': np.array([0]),'t':t}, name='pwf', device=device)
#left = Dirichlet({'x': x, 'y': y}, {'p': p0.reshape(-1)}, device=device, name="left")
```

```python
pde.add_boco(pwf)
```

```python
#right = PBC0({'x': np.array([1]), 'y': y, 't':t}, name='right', device=device)
#right = Dirichlet({'x': x[-1], 'y': y}, {'p': p1.reshape(-1)}, device=device, name="right")
right =  NeumannX({'x': x[-1],'t':t}, name='right', device=device)
```

```python
pde.add_boco(right)
```

```python
#from nangs import MLP

BATCH_SIZE = 128
LR = 1e-2
EPOCHS = 2
NUM_LAYERS = 3
NUM_HIDDEN = 50

mlp = MLP(len(pde.inputs), len(pde.outputs), NUM_LAYERS, NUM_HIDDEN).to(device)
optimizer = torch.optim.Adam(mlp.parameters())
scheduler = torch.optim.lr_scheduler.OneCycleLR(optimizer, max_lr=LR, pct_start=0.1, total_steps=EPOCHS)
```

```python
mlp
```

    MLP(
      (mlp): Sequential(
        (0): Linear(in_features=2, out_features=50, bias=True)
        (1): Sequential(
          (0): GELU()
          (1): Linear(in_features=50, out_features=50, bias=True)
        )
        (2): Sequential(
          (0): GELU()
          (1): Linear(in_features=50, out_features=50, bias=True)
        )
        (3): Sequential(
          (0): GELU()
          (1): Linear(in_features=50, out_features=1, bias=True)
        )
      )
    )

```python

scheduler = torch.optim.lr_scheduler.OneCycleLR(optimizer, max_lr=LR, pct_start=0.1, total_steps=EPOCHS)
```

```python
pde.compile(mlp, optimizer, scheduler)
```

```python

%time hist = pde.solve(EPOCHS, BATCH_SIZE)
```

```python
t = 0.0
x = np.linspace(0,1,100)
#y = np.linspace(0,1,100)

_x, _t = np.meshgrid(x, t)
eval_mesh = Mesh({'x': x,  't':t}, device=device)
p = pde.eval(eval_mesh)
p = p.cpu().numpy()
#p5 = p.reshape((_y.shape[0],_x.shape[1]))
```

```python
ppp=[]
x = np.linspace(0.0,1,100)
for t in np.linspace(0.1,1.,11):
  #_x,_t = np.meshgrid(x,t)
  eval_mesh = Mesh({'x': x,  't':t}, device=device)
  pp = pde.eval(eval_mesh)
  pp = pp.cpu().numpy()
  ppp.append(pp)
```

```python
for p in ppp:
  plt.plot(p)
```

```python
from sympy import Sum, factorial, oo, IndexedBase, Function, sin,exp
from sympy.abc import n
```

```python
def diff_sol(x,t,alpha=0.1):
    sol= Sum(4/((2*n + 1)*np.pi)*sin((2*n+1)*np.pi/2*x)*exp(-((2*n + 1)*np.pi/2)**2*t*alpha),(n,0,50))

    return sol #,sol_n
```

```python
xxx=np.linspace(0,1,100)
for t in np.linspace(0.1,1,11):
    d=[]
    for xx in xxx:
        d.append(diff_sol(xx,t,alpha=1).doit())
    plt.plot(d)
    eval_mesh = Mesh({'x': x,  't':t}, device=device)
    pp = pde.eval(eval_mesh)
    pp = pp.cpu().numpy()
    plt.plot(p)
```

```python
xxx=np.linspace(0,1,10)
x = np.linspace(0,1,10)
for t in np.linspace(0.02,1,11):
    d=[]
    for xx in xxx:
        d.append(diff_sol(xx,t,alpha=1).doit())
    plt.plot(d)
    eval_mesh = Mesh({'x': x,  't':t}, device=device)
    pp = pde.eval(eval_mesh)
    pp = pp.cpu().numpy()
    plt.plot(pp,marker='o',linestyle='None')
```

```python

out=pde.eval_with_grad(eval_mesh)
```

```python
out_np=out[1].detach().cpu().numpy()
```

```python
out_np.shape
```

    (100, 2)

```python
xxx=np.linspace(0,1,10)
x = np.linspace(0,1,10)
for t in np.linspace(0.015,1,11):
    d=[]
    for xx in xxx:
        d.append(diff_sol(xx,t,alpha=1).doit())
    plt.plot(d)
    eval_mesh = Mesh({'x': x,  't':t}, device=device)
    pp = pde.eval(eval_mesh)
    pp = pp.cpu().numpy()
    plt.plot(pp,marker='o',linestyle='None')
```

```python
class PDE():

    def __init__(self, inputs, outputs):

        # check lists of unique strings, non-repeated
        if isinstance(inputs, str):
            inputs = tuple(inputs)
        if isinstance(outputs, str):
            outputs = tuple(outputs)

        checkIsListOfStr(inputs)
        checkIsListOfStr(outputs)
        checkUnique(inputs)
        checkUnique(outputs)
        checkNoRepeated(inputs, outputs)

        self.inputs = inputs
        self.outputs = outputs
        self.mesh = None
        self.bocos = []

    def set_mesh(self, mesh):
        assert mesh.vars == self.inputs, "your data does not match the PDE inputs"
        self.mesh = mesh

    def add_boco(self, boco):
        assert boco.name not in [
            boco.name for boco in self.bocos], f'Boco {boco.name} already exists, use another name'
        boco.validate(self.inputs, self.outputs)
        self.bocos.append(boco)

    def compile(self, model, optimizer, scheduler=None, criterion=None):
        self.model = model
        self.optimizer = optimizer
        self.criterion = criterion if criterion else torch.nn.MSELoss()
        self.scheduler = scheduler

    def computePDELoss(self, vars, grads):
        print("This function need to be overloaded !!!")

    def solve(self, epochs=50, batch_size=None, shuffle=True, graph=True):
        dataloaders = self.set_dataloaders(batch_size, shuffle)
        if graph:
            self.graph_fig, (self.graph_ax1, self.graph_ax2) = plt.subplots(
                1, 2, figsize=(15, 5))
            self.graph_out = display(self.graph_fig, display_id=True)
        # solve PDE
        history = History()
        mb = master_bar(range(1, epochs+1))
        for epoch in mb:
            history.add({'lr': get_lr(self.optimizer)})
            # iterate over the internal points in batches
            for batch in progress_bar(dataloaders['inner'], parent=mb):
                X = batch
                self.optimizer.zero_grad()
                # optimize for boundary points
                for boco in self.bocos:
                    for batch in dataloaders['bocos'][boco.name]:
                        loss = boco.computeLoss(
                            batch, self.model, self.criterion)
                        for name, l in loss.items():
                            l.backward()
                            history.add_step({name: l.item()})
                # optimize for internal points
                X.requires_grad = True
                p = self.model(X)
                loss = self.computePDELoss(X, p)
                assert isinstance(
                    loss, dict), "you should return a dict with the name of the equation and the corresponding loss"
                for name, l in loss.items():
                    l = self.criterion(l, torch.zeros(
                        l.shape).to(self.mesh.device))
                    l.backward(retain_graph=True)
                    history.add_step({name: l.item()})
                self.optimizer.step()
                mb.child.comment = str(history.average())
            history.step()
            mb.main_bar.comment = str(history)
            if graph:
                self.plot_history(history)
            # mb.write(f"Epoch {epoch}/{epochs} {history}")
            if self.scheduler:
                self.scheduler.step()
        if graph:
            plt.close()
        return history.history

    def plot_history(self, history):
        self.graph_ax1.clear()
        self.graph_ax2.clear()
        for name, metric in history.history.items():
            if name != 'lr':
                self.graph_ax1.plot(metric, label=name)
            else:
                self.graph_ax2.plot(metric, label=name)
        self.graph_ax1.legend(loc='upper right')
        self.graph_ax2.legend(loc='upper right')
        self.graph_ax1.grid(True)
        self.graph_ax2.grid(True)
        self.graph_ax1.set_yscale("log")
        self.graph_out.update(self.graph_fig)

    def set_dataloaders(self, batch_size, shuffle):
        dataloaders = {
            'inner': self.mesh.build_dataloader(batch_size, shuffle),
            'bocos': {}
        }
        for boco in self.bocos:
            dataloaders['bocos'][boco.name] = boco.build_dataloader(
                batch_size, shuffle)
        return dataloaders

    def computeGrads(self, outputs, inputs):
        grads, = torch.autograd.grad(outputs, inputs,
                                     grad_outputs=outputs.data.new(
                                         outputs.shape).fill_(1),
                                     create_graph=True, only_inputs=True)
        return grads

    def eval(self, mesh, batch_size=None):
        dataloader = mesh.build_dataloader(batch_size, shuffle=False)
        outputs = torch.tensor([]).to(mesh.device)
        self.model.eval()
        with torch.no_grad():
            for batch in dataloader:
                outputs = torch.cat([outputs, self.model(batch)])
        return outputs

    def eval_with_grad(self, mesh, batch_size=None):
        dataloader = mesh.build_dataloader(batch_size, shuffle=False)
        outputs = torch.tensor([]).to(mesh.device)
        self.model.eval()
        #with torch.no_grad():
        for batch in dataloader:
            batch.requires_grad = True
            outputs = self.model(batch)
        #for batch in dataloader:

            #outputs = self.model(batch)
            p=outputs
            X=batch
            X.requires_grad = True
            grad , = torch.autograd.grad(p, X,
                    grad_outputs=p.data.new(p.shape).fill_(1),
                    create_graph=True, only_inputs=True)
        return outputs,grad
```

```python
def checkIsListOfStr(l):
    "Make sure that l is a list containing only strings"
    if isinstance(l, tuple):
        for i in l:
            if not isinstance(i, str):
                raise Exception(str(i) + ' must be a string')

def checkUnique(l):
    "Make sure that l does not contain repeated elements"
    for i, item1 in enumerate(l):
        for j, item2 in enumerate(l):
            if i != j and item1 == item2:
                raise Exception('Repeated item ' + str(item1))

def checkNoRepeated(l1, l2):
    "Make sure there are no repeated elements in both lists"
    for i in l1:
        if i in l2:
            raise Exception('Repeated item ' + str(i))
```

```python
import numpy as np

class History():
    def __init__(self, precision=5):
        self.history = {}
        self.current = {}
        self.precision = precision

    def add(self, d):
        for name, metric in d.items():
            if not name in self.history:
                self.history[name] = []
            self.history[name].append(metric)

    def add_step(self, d):
        for name, metric in d.items():
            if name not in self.current:
                self.current[name] = []
            self.current[name].append(metric)

    def average(self):
        return {name: round(np.mean(self.current[name]), self.precision) for name in self.current}

    def step(self):
        for name in self.current:
            self.add({name: np.mean(self.current[name])})
        self.current = {}

    def __str__(self):
        s = ''
        for name, value in self.history.items():
            s += f' | {name} {round(value[-1], self.precision)}'
        return s
```

```python
from fastprogress import master_bar, progress_bar
```

```python
def get_lr(optimizer):
    for param_group in optimizer.param_groups:
        return param_group['lr']
```

```python

```