# The strange attractor

- toc:false
- branch: master
- badges: true
- comments: false
- categories: [mathematics, numerical recipes]
- hide: true

------
Questions:
- How do I solve simultaneous ODEs?
- How do I solve second-order ODEs (and higher)?

Objectives:
- Use Euler's method, implemented in Python, to solve a set of simultaneous ODEs
- Use Euler's method, implemented in Python, to solve a second-order ODE
- Understand how the same method could be applied to higher order ODEs

-----

### Computers don't care so much about the type of differential equation

In the previous lesson we used Euler's method to model radioactive decay. To demonstrate the method we deliberately chose a simple example (a linear, first order, separable ODE with one dependent variable). We have seen that this equation can also be solved analytically, so really there is no need for approximate numerical methods in this case.

However there are a large number of physical equations that cannot be solved analytically, and that rely on numerical methods for their modelling.

For example, the Lotka-Volterra equations for studying predator-prey interactions have multiple dependent variables and the Cahn-Hilliard equation for modelling phase separation in fluids is non-linear. These equations can be solved analytically for particular, special cases only.

Computers don't care whether an equation is linear or non-linear, multi-variable or single-variable, the numerical method for studying it is much the same - with the caveat that numerical methods are approximate and so we need to think about the accuracy of the methods used.

### Simultaneous ODEs can be solved using numerical methods

One class of problems that are difficult to solve analytically are <bold>simultaenous ODEs</bold>. These are equations where the derivative of each dependent variable can depend on any of the variables (dependent or independent). For example,

\begin{eqnarray}
\frac{\mathrm{d}x}{\mathrm{d}t} &=& xt + y \\
\frac{\mathrm{d}y}{\mathrm{d}t} &=& \mathrm{sin}^2\omega t - xy
\end{eqnarray}

A famous set of simulataneous ODEs are the <bold>Lorenz equations</bold>, which are the subject of this lesson.

\begin{eqnarray}
\frac{\mathrm{d}x}{\mathrm{d}t} &=& \sigma(y-x) \\
\frac{\mathrm{d}y}{\mathrm{d}t} &=& rx-y-xz \\
\frac{\mathrm{d}z}{\mathrm{d}t} &=& xy-bz
\end{eqnarray}

There three dependent variables - $x$, $y$ and $z$, and one independent variable $t$. There are also three constants -  $\sigma$, $r$ and $b$.

For particular values of $\sigma$, $r$ and $b$ the Lorenz systems has <bold>chaotic behaviour</bold> (a strong dependence on the initial conditions) and for a subset of these, there is also fractal structure called the <bold>strange attractor</bold>.

### Euler's method is easily extended to simultaneous ODEs

In the previous lesson we were introduced to Euler's method for the single variable case:

\begin{equation}
N(t+h) = N(t) + hf(N,t).
\end{equation}

This can be easily extended to the multi-variable case using vector notation:

\begin{equation}
\mathbf{r}(t+h) = \mathbf{r}(t) + h\mathbf{f}(\mathbf{r},t).
\end{equation}

We have seen that arrays can be easily represented in Python using the NumPy library. This allows us to do arithmetic with vectors directly (rather than using verbose workarounds such as `for` loops), so the code is not much more complicated than the one-variable case.

First, let's import the standard scientific libraries we will be using - Numpy and  Matplotlib:

```python
import numpy as np
import matplotlib.pyplot as plt
```

Now we define the function that describe the Lorenz system.

> Note: When the operands are NumPy arrays, the operations are automatically element-wise vector operations.

```python
# define the Lorenz system

def Lorenz(sigma,r,b,xyz):

    x = xyz[0]
    y = xyz[1]
    z = xyz[2]

    fx = sigma*(y-x)
    fy = (r*x)-y-(x*z)
    fz = (x*y)-(b*z)

    return np.array([fx,fy,fz],float)
```

Next we'll list the simulation parameters (start time, end time, number of time steps and step size), the initial values for $x$, $y$ and $z$, and the constants $\sigma$, $r$ and $b$.

```python
# calculation parameters
start = 0                  # start time
end = 50                 # end time
num_steps = 3000         # number of time steps
h = (end-start) / num_steps  # time step size

# intitial conditions: x=0, y=1, z=0
xyz = np.array([0,1,0],float)

# constants
sigma = 10
r = 28
b = 8/3
```

We use the Numpy `arange` function to generate a list of evenly spaced times at which to evaluate the system. We also create  an empty list to hold the values for $x$, $y$ and $z$ that we are yet to calculate.

```python
# use the Numpy arange function to generate a list of evenly spaced times at which to evaluate the number of atoms N.
time_list = np.arange(start,end,h)

# create empty arrays to hold the calculated values
x_points = []
y_points = []
z_points = []
```

Finally, we apply Euler's method using a `For` loop. Note that the order of operations in the loop body is important.

```python
for time in time_list:

    x_points.append(xyz[0])
    y_points.append(xyz[1])
    z_points.append(xyz[2])
    xyz += h*Lorenz(sigma,r,b,xyz)
```

To visualise the strange attractor we can plot the values of $z$ against $y$:

```python
plt.plot(x_points,z_points)
```

### Higher order ODEs can be re-cast as simultaneous ODEs and solved in the same way

Many physical equations are second-order or higher. The general form for a second-order differential equation with one dependent variable is:

\begin{equation}
\frac{\mathrm{d}^2x}{\mathrm{d}t^2} = f\left(x,\frac{\mathrm{d}x}{\mathrm{d}t},t\right)
\end{equation}

Luckily, we can re-cast a higher order equation as a set of simultaneous equations, and then solve in the same way as above.

Let's use the non-linear pendulum as an example. For a pendulum with an art of length $l$ and a bob of mass $m$, Newton's second law ($F=ma$) provides the following equation of motion:

\begin{equation}
ml\frac{\mathrm{d}^2\theta}{\mathrm{d}t^2} = -mg\sin(\theta),
\end{equation}

where $\theta$ is the angle of displacement of the arm from the vertical and $l\frac{\mathrm{d}^2\theta}{\mathrm{d}t^2}$ is the acceleration of the mass in the tangential direction. The exact solution to this equation is unknown, but we now have the knowledge needed to find a numerical approximation.

The equation can be re-written as:

\begin{equation}
\frac{\mathrm{d}^2\theta}{\mathrm{d}t^2} = -\frac{g}{l}\sin(\theta)
\end{equation}

We define a new variable $\omega$:

\begin{equation}
\frac{\mathrm{d}\theta}{\mathrm{d}t} = \omega
\end{equation}

and substitute this into the equation of motion:

\begin{equation}
\frac{\mathrm{d}\omega}{\mathrm{d}t} = -\frac{g}{l}\sin(\theta).
\end{equation}

These two expressions form a set of simultaneous equations that can be solved numerically using the method outlined above.

-----
Keypoints:
- Computers don't care so much about the type of differential equation
- Simulatenous ODEs can also be solved using numerical methods
- Euler's method is easily extended to the multi-variable case
- Higher order ODEs can be re-cast as simultaneous ODEs and solved the same way
-----

---

Do [the quick-test](https://nu-cem.github.io/CompPhys/2021/08/02/Strange-Attractor-Qs.html).

Back to [Modelling with Ordinary Differential Equations](https://nu-cem.github.io/CompPhys/2021/08/02/ODEs.html).

---

```python

```


# Network Architecture

**DIVE into Deep Learning**
___

```python
%matplotlib inline
from util import *
```

## Mathematical Definition

```python
%%html

```

As shown below, a neural network is organized into layers of computation units called the *neurons*.$\def\abs#1{\left\lvert #1 \right\rvert}
\def\Set#1{\left\{ #1 \right\}}
\def\mc#1{\mathcal{#1}}
\def\M#1{\boldsymbol{#1}}
\def\R#1{\mathsf{#1}}
\def\RM#1{\boldsymbol{\mathsf{#1}}}
\def\op#1{\operatorname{#1}}
\def\E{\op{E}}
\def\d{\mathrm{\mathstrut d}}
$

For $\ell\in \{0,\dots,L\}$ and integer $L\geq 1$, let
- $\M{a}^{(\ell)}$ be the output of the $\ell$-th layer of the neural network, and
- $a^{(\ell)}_i$ be the $i$-th element of $\M{a}^{(\ell)}$. The element is computed from the output $\M{a}^{(\ell-1)}$ of its previous layer except for $\ell=0$.

The $0$-th layer is called the *input layer*, i.e.,
$$\M{a}^{(0)}:=\M{x}.$$

The $L$-th layer $\M{a}^{(L)}$ is called the *output layer*. All other layers are called the *hidden layers*.

**What should be the neural network output?**

The goal is to train a *classifier* that predicts a label $\R{y}$ for an input feature $\RM{x}$:

- Instead of a hard-decision classifier is a function $f:\mc{X}\to \mc{Y}$ such that
$f(\RM{x})$ predicts $\R{y}$,

- we train a probabilistic classifier $q_{\R{y}|\RM{x}}$ that estimates $p_{\R{y}|\RM{x}}$, i.e.,

$$
\begin{align}
[q_{\R{y}|\RM{x}}(y|\M{x})]_{y\in \mc{Y}} &:= \M{a}^{(L)}.
\end{align}
$$

For MNIST, the goal is to classify the digit type of a handwritten digit.  When given a handwritten digit,
- a hard-decision classifier returns a digit type, and
- a probabilistic classifier returns a distribution of the digit types.

**Why train a probabilistic classifier?**

- A probabilistic classifer is more general and can give a hard decision as well

  $$f(\RM{x}):=\arg\max_{y\in \mc{Y}} q_{\R{y}|\RM{x}}(y|\RM{x})$$
  by returning the estimated most likely digit type.

- A neural network can model the distribution $p_{\R{y}|\RM{x}}(\cdot|\RM{x})$ better than $\R{y}$ because its output is continuous.

**How to ensure $\M{a}^{(L)}$ is a valid probability vector?**

The soft-max activation function is often used for the last layer:

$$ \sigma^{(L)}\left(\left[\begin{smallmatrix}z^{(\ell)}_1 \\ \vdots \\ z^{(\ell)}_k\end{smallmatrix}\right]\right) := \frac{1}{\sum_{i=1}^k e^{z^{(\ell)}_i}} \left[\begin{smallmatrix}e^{z^{(\ell)}_1} \\ \vdots \\ e^{z^{(\ell)}_k}\end{smallmatrix}\right] $$

where $k:=\abs{\mc{Y}}=10$ is the number of distinct class labels.

It follows that:

$$\sum_{i=1}^k a_i^{(L)}  = 1\kern1em \text{and} \kern1em a_i^{(L)}\geq 0\qquad \forall i\in \{1,\dots,k\}.$$

**How are the different layers related?**

$$
\begin{align}
\M{a}^{(\ell)}&:=\begin{cases}
\M{x} & \ell=0\\
\sigma^{(\ell)}(\overbrace{\M{W}^{(\ell)}\M{a}^{(\ell-1)}+\M{b}^{(\ell)}}^{\RM{z}^{(\ell)}:=})& \ell>0;
\end{cases}\tag{net}
\end{align}
$$

- $\M{W}^{(\ell)}$ is a matrix of weights;
- $\M{b}^{(\ell)}$ is a vector called bias; and
- $\sigma^{(\ell)}$ is a reveal-valued function called the *activation function*.

The activation functions $\sigma^{(\ell)}$ for other layers $1\leq \ell<L$ is often the vectorized version of
-  sigmoid:

  $$\sigma_{\text{sigmoid}}(z) = \frac{1}{1+e^{-z}}$$
-  rectified linear unit (ReLU):

  $$ \sigma_{\text{ReLU}}(z) = \max\{0,z\}. $$

```python
%%html
<b>Activation function:</b><br>

```

The following plots the ReLU activation function.

```python
def ReLU(z):
    return np.max([np.zeros(z.shape), z], axis=0)

z = np.linspace(-5, 5, 100)
plt.plot(z, ReLU(z))
plt.xlim(-5, 5)
plt.title(r'ReLU: $\max\{0,z\}$')
plt.xlabel(r'$z$')
plt.show()
```

**Exercise** Complete the vectorized function `sigmoid` using the vectorized exponentiation `np.exp`.

```python
def sigmoid(z):
    # YOUR CODE HERE
    raise NotImplementedError()

plt.plot(z, ReLU(z))
plt.xlim(-5, 5)
plt.title(r'Sigmoid function: $\frac{1}{1+e^{-z}}$')
plt.xlabel(r'$z$')
plt.show()
```

```python
# tests
```

## Implementation

The following uses the [`keras`](https://keras.io/) library to define the basic neural network achitecture.

`keras` runs on top of `tensorflow` and offers a higher-level abstraction to simplify the construction and training of a neural network. ([`tflearn`](https://github.com/tflearn/tflearn) is another library that provides a higher-level API for `tensorflow`.)

```python
def create_simple_model():
    tf.keras.backend.clear_session() # clear keras cache.
                        # See https://github.com/keras-team/keras/issues/7294
    model = tf.keras.models.Sequential([
        tf.keras.layers.Flatten(input_shape=(28, 28, 1)),
        tf.keras.layers.Dense(16, activation=tf.keras.activations.relu),
        tf.keras.layers.Dense(16, activation=tf.keras.activations.relu),
        tf.keras.layers.Dense(10, activation=tf.keras.activations.softmax)
    ], 'Simple_sequential')
    return model

model = create_simple_model()
model.summary()
```

The above defines [a linear stack](https://www.tensorflow.org/api_docs/python/tf/keras/layers) of [fully-connected layers](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dense) after [flattening the input](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Flatten). The method `summary` is useful for [debugging in Keras](https://keras.io/examples/keras_recipes/debugging_tips/).

**Execise** Assign to `n_hidden_layers` the number of hidden layers for the above simple sequential model.

*Hint:* The layer `Flatten` does not count as a hidden layer since it simply reshapes the input without using any trainable parameters. The output layer also does not count as a hidden layer since its output is the output of the neural network, not intermediate (hidden) values that require further processing by the neurons.

```python
# YOUR CODE HERE
raise NotImplementedError()
n_hidden_layers
```

```python
# tests
```