# The strange attractor

- toc:false
- branch: master
- badges: true
- comments: false
- categories: [mathematics, numerical recipes]
- hide: true

------
Questions:
- How do I solve simultaneous ODEs?
- How do I solve second-order ODEs (and higher)?

Objectives:
- Use Euler's method, implemented in Python, to solve a set of simultaneous ODEs
- Use Euler's method, implemented in Python, to solve a second-order ODE
- Understand how the same method could be applied to higher order ODEs

-----

### Computers don't care so much about the type of differential equation

In the previous lesson we used Euler's method to model radioactive decay. To demonstrate the method we deliberately chose a simple example (a linear, first order, separable ODE with one dependent variable). We have seen that this equation can also be solved analytically, so really there is no need for approximate numerical methods in this case.

However there are a large number of physical equations that cannot be solved analytically, and that rely on numerical methods for their modelling.

For example, the Lotka-Volterra equations for studying predator-prey interactions have multiple dependent variables and the Cahn-Hilliard equation for modelling phase separation in fluids is non-linear. These equations can be solved analytically for particular, special cases only.

Computers don't care whether an equation is linear or non-linear, multi-variable or single-variable, the numerical method for studying it is much the same - with the caveat that numerical methods are approximate and so we need to think about the accuracy of the methods used.

### Simultaneous ODEs can be solved using numerical methods

One class of problems that are difficult to solve analytically are <bold>simultaenous ODEs</bold>. These are equations where the derivative of each dependent variable can depend on any of the variables (dependent or independent). For example,

\begin{eqnarray}
\frac{\mathrm{d}x}{\mathrm{d}t} &=& xt + y \\
\frac{\mathrm{d}y}{\mathrm{d}t} &=& \mathrm{sin}^2\omega t - xy
\end{eqnarray}

A famous set of simulataneous ODEs are the <bold>Lorenz equations</bold>, which are the subject of this lesson.

\begin{eqnarray}
\frac{\mathrm{d}x}{\mathrm{d}t} &=& \sigma(y-x) \\
\frac{\mathrm{d}y}{\mathrm{d}t} &=& rx-y-xz \\
\frac{\mathrm{d}z}{\mathrm{d}t} &=& xy-bz
\end{eqnarray}

There three dependent variables - $x$, $y$ and $z$, and one independent variable $t$. There are also three constants -  $\sigma$, $r$ and $b$.

For particular values of $\sigma$, $r$ and $b$ the Lorenz systems has <bold>chaotic behaviour</bold> (a strong dependence on the initial conditions) and for a subset of these, there is also fractal structure called the <bold>strange attractor</bold>.

### Euler's method is easily extended to simultaneous ODEs

In the previous lesson we were introduced to Euler's method for the single variable case:

\begin{equation}
N(t+h) = N(t) + hf(N,t).
\end{equation}

This can be easily extended to the multi-variable case using vector notation:

\begin{equation}
\mathbf{r}(t+h) = \mathbf{r}(t) + h\mathbf{f}(\mathbf{r},t).
\end{equation}

We have seen that arrays can be easily represented in Python using the NumPy library. This allows us to do arithmetic with vectors directly (rather than using verbose workarounds such as `for` loops), so the code is not much more complicated than the one-variable case.

First, let's import the standard scientific libraries we will be using - Numpy and  Matplotlib:

```python
import numpy as np
import matplotlib.pyplot as plt
```

Now we define the function that describe the Lorenz system.

> Note: When the operands are NumPy arrays, the operations are automatically element-wise vector operations.

```python
# define the Lorenz system

def Lorenz(sigma,r,b,xyz):

    x = xyz[0]
    y = xyz[1]
    z = xyz[2]

    fx = sigma*(y-x)
    fy = (r*x)-y-(x*z)
    fz = (x*y)-(b*z)

    return np.array([fx,fy,fz],float)
```

Next we'll list the simulation parameters (start time, end time, number of time steps and step size), the initial values for $x$, $y$ and $z$, and the constants $\sigma$, $r$ and $b$.

```python
# calculation parameters
start = 0                  # start time
end = 50                 # end time
num_steps = 501         # number of time steps
h = (b-a) / num_steps  # time step size

# intitial conditions: x=0, y=1, z=0
xyz = np.array([0,1,0],float)

# constants
sigma = 10
r = 28
b = 8/3
```

We use the Numpy `arange` function to generate a list of evenly spaced times at which to evaluate the system. We also create  an empty list to hold the values for $x$, $y$ and $z$ that we are yet to calculate.

```python
# use the Numpy arange function to generate a list of evenly spaced times at which to evaluate the number of atoms N.
time_list = np.arange(start,end,h)

# create empty arrays to hold the calculated values
x_points = []
y_points = []
z_points = []
```

Finally, we apply Euler's method using a `For` loop. Note that the order of operations in the loop body is important.

```python
for time in time_list:

    x_points.append(xyz[0])
    y_points.append(xyz[1])
    z_points.append(xyz[2])
    xyz += h*Lorenz(sigma,r,b,xyz)
```

To visualise the strange attractor we can plot the values of $z$ against $y$:

```python
plt.plot(x_points,z_points)
```

### Higher order ODEs can be re-cast as simultaneous ODEs and solved in the same way

Many physical equations are second-order or higher. The general form for a second-order differential equation with one dependent variable is:

\begin{equation}
\frac{\mathrm{d}^2x}{\mathrm{d}t^2} = f\left(x,\frac{\mathrm{d}x}{\mathrm{d}t},t\right)
\end{equation}

Luckily, we can re-cast a higher order equation as a set of simultaneous equations, and then solve in the same way as above.

Let's use the non-linear pendulum as an example. For a pendulum with an art of length $l$ and a bob of mass $m$, Newton's second law ($F=ma$) provides the following equation of motion:

\begin{equation}
ml\frac{\mathrm{d}^2\theta}{\mathrm{d}t^2} = -mg\sin(\theta),
\end{equation}

where $\theta$ is the angle of displacement of the arm from the vertical and $l\frac{\mathrm{d}^2\theta}{\mathrm{d}t^2}$ is the acceleration of the mass in the tangential direction. The exact solution to this equation is unknown, but we now have the knowledge needed to find a numerical approximation.

The equation can be re-written as:

\begin{equation}
\frac{\mathrm{d}^2\theta}{\mathrm{d}t^2} = -\frac{g}{l}\sin(\theta)
\end{equation}

We define a new variable $\omega$:

\begin{equation}
\frac{\mathrm{d}\theta}{\mathrm{d}t} = \omega
\end{equation}

and substitute this into the equation of motion:

\begin{equation}
\frac{\mathrm{d}\omega}{\mathrm{d}t} = -\frac{g}{l}\sin(\theta).
\end{equation}

These two expressions form a set of simultaneous equations that can be solved numerically using the method outlined above.

-----
Keypoints:
- Computers don't care so much about the type of differential equation
- Simulatenous ODEs can also be solved using numerical methods
- Euler's method is easily extended to the multi-variable case
- Higher order ODEs can be re-cast as simultaneous ODEs and solved the same way
-----

---

Do [the quick-test](https://nu-cem.github.io/CompPhys/2021/08/02/Strange-Attractor-Qs.html).

Back to [Modelling with Ordinary Differential Equations](https://nu-cem.github.io/CompPhys/2021/08/02/ODEs.html).

---

```python

```


# PRÁCTICA 02.
# Perceptrón Simple.

### OBJETIVO:

Que el alumno implemente el perceptrón simple y lo aplique en distintos conjuntos de datos aplicando variaciones en la forma de aprendizaje del perceptrón.

```python
%matplotlib inline

import matplotlib.pyplot as plt
import matplotlib.cm as cm
from mpl_toolkits.mplot3d import Axes3D

import json, matplotlib
from IPython.core.pylabtools import figsize
figsize(11, 5)

from ipywidgets import interact, interact_manual, interactive, fixed
import ipywidgets as widgets
from IPython.display import display
import numpy as np

```

```python
class Perceptron:
    def __init__(self, num_inputs, weights = None, seed = 0):
        if weights: self.weights = np.copy(weights)
        else:
            np.random.seed(seed)
            self.weights = np.random.rand(num_inputs)

    def predict(self, input):
        ponderate_sum = np.dot(input, self.weights)
        result = 1.0 if ponderate_sum > 0 else 0.0

        return np.float64(result), ponderate_sum

    def train(self, input, desired_output, learning_rate):
        prediction, _ = self.predict(input)
        if prediction == desired_output: return

        add_or_substract = 1.0 if (prediction == 0.0 and desired_output == 1.0)  else -1.0
        self.weights = self.weights + (add_or_substract * input)

    def train_epoch(self, input_set, output_set, learning_rate, may_print = True):
        for index, input in enumerate(input_set):
            output = output_set[index]
            if may_print: print(f"input={input} index={index}")

            prediction, ponderate = self.predict(input)
            if may_print: print(f"prediction={prediction} real={output} ponderate={ponderate} ")

            self.train(input, output, learning_rate)

            prediction, ponderate = self.predict(input)
            if may_print: print(f"prediction={prediction} real={output} ponderate={ponderate} ")
            if may_print: print()

```

### PERCEPTRÓN

Un perceptrón simple es una unidad computacional con un umbral $\theta$, que al recibir $X \text{ de }n$ entradas reales $X = x_1, x_2, \cdots, x_n$ a través de sus pesos asociados $W = w_1, w_2, \cdots, w_n$, da una salida de valor 1 si la desigualdad $\sum_{i=1}^{n}w_ix_i \geq \theta$ se cumple, en otro caso la salida es 0.

La ley de aprendizaje se define como: $w_i(t+1) = w_i (t) + \alpha · d(x) · x_i$

$$
\
d(x)=
\begin{cases}
1, & \text{si } x \in A;\\
-1, & \text{si } x \in B
\end{cases}
\
$$

y $\alpha$ la tasa de aprendizaje. Los pesos no se modifican si $x$ está bien clasificado.

1.- Implementa la clase Perceptron que contenga lo siguiente:
   * Inicialización del perceptrón dados los pesos iniciales.
   * Función **evalua_escalon** que reciba X y evalúe la salida utilizando la función escalón en el intervalo [0,1].
   * Función **entrena**, debe entrenar al perceptrón: modificación de los pesos en una iteración cada que se llame a esta función dada una tasa de aprendizaje.

2.- Crea un *Perceptron* de pesos iniciales aleatorios y entrénalo con el conjunto de datos datosAND con inicialización de pesos aleatorios.

 * Completa la función **entrenaPerceptronAND** y utiliza tu función **entrena** para que en cada paso de entrenamiento se dibuje el conjunto de datos y el plano que divide al conjunto, tal como se muestra en la figura (utiliza colores para diferenciar al conjunto de datos)

HINT1: Recuerda que el vector de pesos W contiene el vector normal al plano.

HINT2: Puedes utilizar la función scatter y plot_surface

```python
data_and = np.array([[1, 0, 0, 0],
                     [1, 0 ,0, 1],
                     [1, 0, 1, 0],
                     [1, 0, 1, 1],
                     [1, 1, 0, 0],
                     [1, 1, 0, 1],
                     [1, 1, 1, 0],
                     [1, 1, 1, 1]]).astype(np.float64)

answers_and = np.array([0,0,0,0,0, 0, 0, 1]).astype(np.float64)

num_inputs = 4
perceptron = Perceptron(num_inputs)

for i in range(6):
    perceptron.train_epoch(data_and, answers_and, 0.08)

x = np.linspace(-0.1, 1.1, 30)
y = np.linspace(-0.1, 1.1, 30)

X,Y = np.meshgrid(x,y)

@interact_manual()
def entrenaPerceptronAND():
    input_1 = data_and[:, 1]
    input_2 = data_and[:, 2]
    input_3 = data_and[:, 3]

    d = perceptron.weights[0]
    a, b, c = perceptron.weights[1:]

    x = np.linspace(-1,1,10)
    y = np.linspace(-1,1,10)

    X,Y = np.meshgrid(x,y)
    Z = (d - a*X - b*Y) / c

    fig = plt.figure()
    plane = fig.add_subplot(111, projection='3d')

    plane.set_xlabel('input_1')
    plane.set_ylabel('input_2')
    plane.set_zlabel('input_3')
    plane.plot_surface(X, Y, Z)
    plane.scatter(*data_and[:, 1:].T, c=answers_and)

```

    input=[1. 0. 0. 0.] index=0
    prediction=1.0 real=0.0 ponderate=0.5488135039273248
    prediction=0.0 real=0.0 ponderate=-0.45118649607267525

    input=[1. 0. 0. 1.] index=1
    prediction=1.0 real=0.0 ponderate=0.09369668692422162
    prediction=0.0 real=0.0 ponderate=-1.9063033130757785

    input=[1. 0. 1. 0.] index=2
    prediction=0.0 real=0.0 ponderate=-0.8484231200010314
    prediction=0.0 real=0.0 ponderate=-0.8484231200010314

    input=[1. 0. 1. 1.] index=3
    prediction=0.0 real=0.0 ponderate=-1.3035399370041345
    prediction=0.0 real=0.0 ponderate=-1.3035399370041345

    input=[1. 1. 0. 0.] index=4
    prediction=0.0 real=0.0 ponderate=-0.7359971297002558
    prediction=0.0 real=0.0 ponderate=-0.7359971297002558

    input=[1. 1. 0. 1.] index=5
    prediction=0.0 real=0.0 ponderate=-1.191113946703359
    prediction=0.0 real=0.0 ponderate=-1.191113946703359

    input=[1. 1. 1. 0.] index=6
    prediction=0.0 real=0.0 ponderate=-0.1332337536286119
    prediction=0.0 real=0.0 ponderate=-0.1332337536286119

    input=[1. 1. 1. 1.] index=7
    prediction=0.0 real=1.0 ponderate=-0.588350570631715
    prediction=1.0 real=1.0 ponderate=3.411649429368285

    input=[1. 0. 0. 0.] index=0
    prediction=0.0 real=0.0 ponderate=-0.45118649607267525
    prediction=0.0 real=0.0 ponderate=-0.45118649607267525

    input=[1. 0. 0. 1.] index=1
    prediction=1.0 real=0.0 ponderate=0.09369668692422162
    prediction=0.0 real=0.0 ponderate=-1.9063033130757785

    input=[1. 0. 1. 0.] index=2
    prediction=1.0 real=0.0 ponderate=0.15157687999896874
    prediction=0.0 real=0.0 ponderate=-1.8484231200010313

    input=[1. 0. 1. 1.] index=3
    prediction=0.0 real=0.0 ponderate=-2.3035399370041345
    prediction=0.0 real=0.0 ponderate=-2.3035399370041345

    input=[1. 1. 0. 0.] index=4
    prediction=0.0 real=0.0 ponderate=-0.7359971297002557
    prediction=0.0 real=0.0 ponderate=-0.7359971297002557

    input=[1. 1. 0. 1.] index=5
    prediction=0.0 real=0.0 ponderate=-1.191113946703359
    prediction=0.0 real=0.0 ponderate=-1.191113946703359

    input=[1. 1. 1. 0.] index=6
    prediction=0.0 real=0.0 ponderate=-0.13323375362861167
    prediction=0.0 real=0.0 ponderate=-0.13323375362861167

    input=[1. 1. 1. 1.] index=7
    prediction=0.0 real=1.0 ponderate=-0.5883505706317148
    prediction=1.0 real=1.0 ponderate=3.411649429368285

    input=[1. 0. 0. 0.] index=0
    prediction=0.0 real=0.0 ponderate=-1.4511864960726752
    prediction=0.0 real=0.0 ponderate=-1.4511864960726752

    input=[1. 0. 0. 1.] index=1
    prediction=0.0 real=0.0 ponderate=-0.9063033130757784
    prediction=0.0 real=0.0 ponderate=-0.9063033130757784

    input=[1. 0. 1. 0.] index=2
    prediction=1.0 real=0.0 ponderate=0.15157687999896874
    prediction=0.0 real=0.0 ponderate=-1.8484231200010313

    input=[1. 0. 1. 1.] index=3
    prediction=0.0 real=0.0 ponderate=-1.3035399370041345
    prediction=0.0 real=0.0 ponderate=-1.3035399370041345

    input=[1. 1. 0. 0.] index=4
    prediction=1.0 real=0.0 ponderate=0.26400287029974434
    prediction=0.0 real=0.0 ponderate=-1.7359971297002557

    input=[1. 1. 0. 1.] index=5
    prediction=0.0 real=0.0 ponderate=-1.191113946703359
    prediction=0.0 real=0.0 ponderate=-1.191113946703359

    input=[1. 1. 1. 0.] index=6
    prediction=0.0 real=0.0 ponderate=-1.1332337536286117
    prediction=0.0 real=0.0 ponderate=-1.1332337536286117

    input=[1. 1. 1. 1.] index=7
    prediction=0.0 real=1.0 ponderate=-0.5883505706317148
    prediction=1.0 real=1.0 ponderate=3.411649429368285

    input=[1. 0. 0. 0.] index=0
    prediction=0.0 real=0.0 ponderate=-2.4511864960726752
    prediction=0.0 real=0.0 ponderate=-2.4511864960726752

    input=[1. 0. 0. 1.] index=1
    prediction=0.0 real=0.0 ponderate=-0.9063033130757785
    prediction=0.0 real=0.0 ponderate=-0.9063033130757785

    input=[1. 0. 1. 0.] index=2
    prediction=0.0 real=0.0 ponderate=-0.8484231200010313
    prediction=0.0 real=0.0 ponderate=-0.8484231200010313

    input=[1. 0. 1. 1.] index=3
    prediction=1.0 real=0.0 ponderate=0.6964600629958655
    prediction=0.0 real=0.0 ponderate=-2.3035399370041345

    input=[1. 1. 0. 0.] index=4
    prediction=0.0 real=0.0 ponderate=-0.7359971297002557
    prediction=0.0 real=0.0 ponderate=-0.7359971297002557

    input=[1. 1. 0. 1.] index=5
    prediction=0.0 real=0.0 ponderate=-0.1911139467033589
    prediction=0.0 real=0.0 ponderate=-0.1911139467033589

    input=[1. 1. 1. 0.] index=6
    prediction=0.0 real=0.0 ponderate=-0.13323375362861167
    prediction=0.0 real=0.0 ponderate=-0.13323375362861167

    input=[1. 1. 1. 1.] index=7
    prediction=1.0 real=1.0 ponderate=0.4116494293682851
    prediction=1.0 real=1.0 ponderate=0.4116494293682851

    input=[1. 0. 0. 0.] index=0
    prediction=0.0 real=0.0 ponderate=-3.4511864960726752
    prediction=0.0 real=0.0 ponderate=-3.4511864960726752

    input=[1. 0. 0. 1.] index=1
    prediction=0.0 real=0.0 ponderate=-2.9063033130757785
    prediction=0.0 real=0.0 ponderate=-2.9063033130757785

    input=[1. 0. 1. 0.] index=2
    prediction=0.0 real=0.0 ponderate=-2.8484231200010313
    prediction=0.0 real=0.0 ponderate=-2.8484231200010313

    input=[1. 0. 1. 1.] index=3
    prediction=0.0 real=0.0 ponderate=-2.3035399370041345
    prediction=0.0 real=0.0 ponderate=-2.3035399370041345

    input=[1. 1. 0. 0.] index=4
    prediction=0.0 real=0.0 ponderate=-0.7359971297002557
    prediction=0.0 real=0.0 ponderate=-0.7359971297002557

    input=[1. 1. 0. 1.] index=5
    prediction=0.0 real=0.0 ponderate=-0.1911139467033589
    prediction=0.0 real=0.0 ponderate=-0.1911139467033589

    input=[1. 1. 1. 0.] index=6
    prediction=0.0 real=0.0 ponderate=-0.13323375362861167
    prediction=0.0 real=0.0 ponderate=-0.13323375362861167

    input=[1. 1. 1. 1.] index=7
    prediction=1.0 real=1.0 ponderate=0.4116494293682851
    prediction=1.0 real=1.0 ponderate=0.4116494293682851

    input=[1. 0. 0. 0.] index=0
    prediction=0.0 real=0.0 ponderate=-3.4511864960726752
    prediction=0.0 real=0.0 ponderate=-3.4511864960726752

    input=[1. 0. 0. 1.] index=1
    prediction=0.0 real=0.0 ponderate=-2.9063033130757785
    prediction=0.0 real=0.0 ponderate=-2.9063033130757785

    input=[1. 0. 1. 0.] index=2
    prediction=0.0 real=0.0 ponderate=-2.8484231200010313
    prediction=0.0 real=0.0 ponderate=-2.8484231200010313

    input=[1. 0. 1. 1.] index=3
    prediction=0.0 real=0.0 ponderate=-2.3035399370041345
    prediction=0.0 real=0.0 ponderate=-2.3035399370041345

    input=[1. 1. 0. 0.] index=4
    prediction=0.0 real=0.0 ponderate=-0.7359971297002557
    prediction=0.0 real=0.0 ponderate=-0.7359971297002557

    input=[1. 1. 0. 1.] index=5
    prediction=0.0 real=0.0 ponderate=-0.1911139467033589
    prediction=0.0 real=0.0 ponderate=-0.1911139467033589

    input=[1. 1. 1. 0.] index=6
    prediction=0.0 real=0.0 ponderate=-0.13323375362861167
    prediction=0.0 real=0.0 ponderate=-0.13323375362861167

    input=[1. 1. 1. 1.] index=7
    prediction=1.0 real=1.0 ponderate=0.4116494293682851
    prediction=1.0 real=1.0 ponderate=0.4116494293682851

    interactive(children=(Button(description='Run Interact', style=ButtonStyle()), Output()), _dom_classes=('widge…

### MÍNIMOS CUADRADOS

La meta es encontrar los pesos $w$ que mejor mapean los datos de entrada en los datos objetivo. Con ese fin, se plantea la ecuación de error:

\begin{equation}
    E(w) = \frac{1}{2} \| \bar{y} − \bar{t} \|^2
\end{equation}

Con $\bar{t}$ el vector de datos objetivo, $\bar{y}$ el vector de salidas asociadas $ y(x) = X^Tw $

A partir de la ecuación anterior obtenemos:

\begin{equation}
E(w) = \frac{1}{2}(\bar{y}-\bar{t})^T(\bar{y}-\bar{t})
\end{equation}

Derivando respecto a $w$

\begin{equation}
\frac{dE(w)}{dw} = \phi(\bar{y}-\bar{t})
\end{equation}

con $\phi= [x^1  \cdots x^N] \text{ y }x^i$ el i−ésimo vector de datos de entrada a la red. $N$ el número de datos/registros de entrenamiento.

De lo anterior tenemos:

\begin{equation}
w = (\phi \phi^T)^{-1}\phi \bar{t}
\end{equation}

### ENTRENAMIENTO Y PRUEBA

Hay dos fases en el modelado con redes neuronales:
* **Fase de entrenamiento:** Se utiliza un conjunto de datos o patrones de entrenamiento
para determinar los pesos que definen el modelo. Se calculan de manera iterativa con el objeto de minimizar el error cometido entre la salida obtenida por la red neuronal y la salida deseada.

* **Fase de prueba:** Se utiliza para estimar el error de generalización del modelo.

1. Dado el conjunto de datos datos.txt anexo, entrena un perceptrón con el 70% de los datos (elegidos aleatoriamente). El otro 30% será el conjunto de prueba.

2. Para el mismo conjunto de entrenamiento, calcula los pesos $w$ con el método de mínimos cuadrados.

2. Grafica y compara los resultados de $w$ obtenidos del entrenamiento del perceptrón y el de mínimos cuadrados. La salida dece ser similar a la figura siguiente:

3. Utiliza el conjunto de prueba para obtener la exactitud de los modelos.

OBS:  El archivo datos.txt contiene 3 columnas separadas por comas, las dos primeras columnas son el valor del conjunto $X = \{x_1, x_2\}$; la tercer columna son los datos objetivo $\bar{t}$

El criterio de parada tómalo como un número fijo de iteraciones o hasta que el conjunto esté  totalmente clasificado y fija la tasa de aprendizaje a una constante.

```python
# Split data
data = np.loadtxt(open("data.csv", "rb"), delimiter=",").astype(np.float64)
np.random.shuffle(data)
number_of_points = len(data)
split_index = int(0.7 * number_of_points)

training, test = data[ : split_index], data[split_index : ]

training_input, training_output = training[:, [0, 1]], training[:, 2]
test_input, test_output = test[:, [0, 1]], test[:, 2]

bias = np.ones((len(training_input), 1))
training_input = np.concatenate( (bias, training_input), axis=1)

bias = np.ones((len(test_input), 1))
test_input = np.concatenate( (bias, test_input), axis=1)

training_pure_data = training_input[:, 1:].T
test_pure_data = test_input[:, 1:].T
```

```python
num_inputs = 3
perceptron2 = Perceptron(num_inputs)

for i in range(20):
    perceptron2.train_epoch(training_input, training_output, 0.08, False)
```

```python
b, w1, w2 = perceptron2.weights
x = -b / w1
y = -b / w2

d = y
c = -y / x

line_x_coords = np.array([-1, x])
line_y_coords = c * line_x_coords + d

plt.plot(line_x_coords, line_y_coords)
plt.scatter(*test_pure_data, c=test_output)

plt.show()
```

```python
# 𝑤=(𝜙𝜙𝑇)−1𝜙𝑡¯
phi = np.matrix(test_pure_data)
t = np.matrix(test_output)
y, b = (phi * phi.T)**-1 * phi * t.T
```

```python
line_x_coords = np.array([-1, 4])
line_y_coords = y * line_x_coords + b

plt.plot(line_x_coords, line_y_coords)

plt.show()
```


#Chapter 4
______

##The greatest theorem never told

This chapter focuses on an idea that is always bouncing around our minds, but is rarely made explicit outside books devoted to statistics. In fact, we've been used this simple idea in every example thus far.

###The Law of Large Numbers

Let $Z_i$ be $N$ independent samples from some probability distribution. According to *the Law of Large numbers*,  so long as the expected value $E[Z]$ is finite, the following holds,

$$\frac{1}{N} \sum_{i=1}^N Z_i \rightarrow E[ Z ],  \;\;\; N \rightarrow \infty.$$

In words:

>   The average of a sequence of random variables from the same distribution converges to the expected value of that distribution.

This may seem like a boring result, but it will be the most useful tool you use.

### Intuition

If the above Law is somewhat surprising,  it can be made more clear be examining a simple example.

Consider a random variable $Z$ that can take only two values, $c_1$ and $c_2$. Suppose we have a large number of samples of $Z$, denoting a specific sample $Z_i$. The Law says that we can approximate the expected value of $Z$ by averaging over all samples. Consider the average:

$$ \frac{1}{N} \sum_{i=1}^N \;Z_i $$

By construction, $Z_i$ can only take on $c_1$ or $c_2$, hence we can partition the sum over these two values:

\begin{align}
\frac{1}{N} \sum_{i=1}^N \;Z_i
& =\frac{1}{N} \big(  \sum_{ Z_i = c_1}c_1 + \sum_{Z_i=c_2}c_2 \big) \\\\[5pt]
& = c_1 \sum_{ Z_i = c_1}\frac{1}{N} + c_2 \sum_{ Z_i = c_2}\frac{1}{N} \\\\[5pt]
& = c_1 \times \text{ (approximate frequency of $c_1$) } \\\\
& \;\;\;\;\;\;\;\;\; + c_2 \times \text{ (approximate frequency of $c_2$) } \\\\[5pt]
& \approx c_1 \times P(Z = c_1) + c_2 \times P(Z = c_2 ) \\\\[5pt]
& = E[Z]
\end{align}

Equality holds in the limit, but we can get closer and closer by using more and more samples in the average. This Law holds for *any distribution*, minus some pathological examples that only mathematicians have fun with.

##### Example
____

Below is a diagram of the Law of Large numbers in action for three different sequences of Poisson random variables.

 We sample `sample_size= 100000` Poisson random variables with parameter $\lambda = 4.5$. (Recall the expected value of a Poisson random variable is equal to it's parameter.) We calculate the average for the first $n$ samples, for $n=1$ to `sample_size`.

```
%pylab inline

figsize( 12.5, 5 )
import pymc as mc

sample_size = 100000
expected_value = lambda_ = 4.5
poi = mc.rpoisson
N_samples = range(1,sample_size,100)

for k in range(3):

    samples = poi( lambda_, size = sample_size )

    partial_average = [ samples[:i].mean() for i in N_samples ]

    plt.plot( N_samples, partial_average, lw=1.5,label="average \
    of  $n$ samples; seq. %d"%k)

plt.plot( N_samples, expected_value*np.ones_like( partial_average), \
    ls = "--", label = "true expected value", c = "k" )

plt.ylim( 4.35, 4.65)
plt.title( "Convergence of the average of \n random variables to its \
expected value" )
plt.ylabel( "average of $n$ samples" )
plt.xlabel( "# of samples, $n$")
plt.legend()
```

Looking at the above plot, it is clear that when the sample size is small, there is greater variation in the average (compare how *jagged and jumpy* the average is initially, then *smooths* out). All three paths *approach* the value 4.5, but just flirt with it as $N$ gets large. Mathematicians and statistician have another name for *flirting*: convergence.

Another very relevant question we can ask is *how quickly am I converging to the expected value?* Let's plot something new. For a specific $N$, let's do the above trials thousands of times and compute how far away we are from the true expected value, on average. But wait — *compute on average*? This simply the law of large numbers again! For example, we are interested in, for a specific $N$, the quantity:

$$D(N) = \sqrt{ \;E\left[\;\; \left( \frac{1}{N}\sum_{i=1}^NZ_i  - 4.5 \;\right)^2 \;\;\right] \;\;}$$

(We take the square root so the dimensions of the above quantity and our random variables are the same). As the above is an expected value, it can be approximated using the law of large numbers: instead of averaging $Z_i$, we calculate the following multiple times and average them:

$$ Y_k = \left( \;\frac{1}{N}\sum_{i=1}^NZ_i  - 4.5 \; \right)^2 $$

i.e., we consider the average

$$ \frac{1}{N_Y} \sum_{k=1}^{N_Y} Y_k \approx D(N) $$ where $N_Y$ is some suitably large number.

```
figsize( 12.5, 4)
N_Y = 250
D_N_results = []
N_array = np.arange( 0, 50000, 2500 )
lambda_ = 4.5
expected_value = 4.5

def D_N( n ):
    Z = poi( lambda_, size = (n, N_Y) )
    average_Z = Z.mean(axis=0)
    return np.sqrt( (  (average_Z - expected_value)**2  ).mean() )

for n in N_array:
    D_N_results.append( D_N(n) )

plt.xlabel( "$N$" )
plt.ylabel( "expected squared-distance from true value" )
plt.plot(N_array, D_N_results, lw = 1,
            label="expected distance between\n\
expected value and \naverage of $N$ random variables.")
plt.plot( N_array, np.sqrt(expected_value)/np.sqrt(N_array), lw = 3, ls = "--",
        label = r"$\frac{\sqrt{\lambda}}{\sqrt{N}}$" )
plt.legend()
plt.title( "How 'fast' is the sample average converging? " )
```

As expected, the expected distance between our sample average and the actual expected value shrinks as $N$ grows large. But also notice that the *rate* of convergence decreases, that is, we need only 10 000 additional samples to move from 0.020 to 0.015, a difference of 0.005, but *20 000* more samples to again decrease from 0.015  to 0.010, again only a 0.005 decrease.

It turns out we can measure this rate of convergence. Above I have plotted a second line, the function $\sqrt{\lambda}/\sqrt{N}$. This was not chosen arbitrarily. In most cases, given a sequence of random variable distributed like $Z$, the rate of converge to $E[Z]$ of the Law of Large Numbers is

$$ \frac{ \sqrt{ \; Var(Z) \; } }{\sqrt{N} }$$

This is useful to know: for a given large $N$, we know (on average) how far away we are from the estimate. On the other hand, in a Bayesian setting, this can seem like a useless result: Bayesian analysis is OK with uncertainty so what's the *statistical* point of adding extra precise digits? Though drawing samples can be so computationally cheap that having a *larger* $N$ is fine too.

### How do we compute $Var(Z)$ though?

The variance is simply another expected value that can be approximated! Consider the following, once we have the expected value (by using the Law of Large Numbers to estimate it, denote it $\mu$), we can estimate the variance:

$$ \frac{1}{N}\sum_{i=1}^N \;(Z_i - \mu)^2 \rightarrow E[ \;( Z - \mu)^2 \;] = Var( Z )$$

### Expected values and probabilities
There is an even less explicit relationship between expected value and estimating probabilities. Define the *indicator function*

$$\mathbb{1}_A(x) =
\begin{cases} 1 &  x \in A \\\\
              0 &  else
\end{cases}
$$
Then, by the law of large numbers, if we have many samples $X_i$, we can estimate the probability of an event $A$, denoted $P(A)$, by:

$$ \frac{1}{N} \sum_{i=1}^N \mathbb{1}_A(X_i) \rightarrow E[\mathbb{1}_A(X)] =  P(A) $$

Again, this is fairly obvious after a moments thought: the indicator function is only 1 if the event occurs, so we are summing only the times the event occurs and dividing by the total number of trials  (consider how we usually approximate probabilities using frequencies). For example, suppose we wish to estimate the probability that a $Z \sim Exp(.5)$ is greater than 10, and we have many samples from a $Exp(.5)$ distribution.

$$ P( Z > 10 ) =  \sum_{i=1}^N \mathbb{1}_{z > 10 }(Z_i) $$

```
import pymc as mc

N = 10000
print np.mean( [ mc.rexponential( 0.5 )>10 for i in range(N) ] )
```

    0.0058

### What does this all have to do with Bayesian statistics?

*Point estimates*, to be introduced in the next chapter, in Bayesian inference are computed using expected values. In more analytical Bayesian inference, we would have been required to evaluate complicated expected values represented as multi-dimensional integrals. No longer. If we can sample from the posterior distribution directly, we simply need to evaluate averages. Much easier. If accuracy is a priority, plots like the ones above show how fast you are converging. And if further accuracy is  desired, just take more samples from the posterior.

When is enough enough? When can you stop drawing samples from the posterior? That is the practitioners decision, and also dependent on the variance of the samples (recall from above a high variance means the average will converge slower).

We also should understand when the Law of Large Numbers fails. As the name implies, and comparing the graphs above for small $N$, the Law is only true for large sample sizes. Without this, the asymptotic result is not reliable. Knowing in what situations the Law fails can give use *confidence in how unconfident we should be*. The next section deals with this issue.

## The Disorder of Small Numbers

The Law of Large Numbers is only valid as $N$ gets *infinitely* large: never truly attainable.  While the law is a powerful tool, it is foolhardy to apply it liberally. Our next example illustrates this.

##### Example: Aggregated geographic data

Often data comes in aggregated form. For instance, data may be grouped by state, county, or city level. Of course, the population numbers vary per geographic area. If the data is an average of some characteristic of each the geographic areas, we must be conscious of the Law of Large Numbers and how it can *fail* for areas with small populations.

We will observe this on a toy dataset. Suppose there are five thousand counties in our dataset. Furthermore,  population number in each state are uniformly distributed between 100 and 1500. The way the population numbers are generated is irrelevant to the discussion, so we do not justify this. We are interested in measuring the average height of individuals per county. Unbeknownst to the us, height does **not** vary across county, and each individual, regardless of the county he or she is currently living in, has the same distribution of what their height may be:

$$ \text{height} \sim \text{Normal}(150, 15 ) $$

We aggregate the individuals at the county level, so we only have data for the *average in the county*. What might our dataset look like?

```
figsize( 12.5, 4)
std_height = 15
mean_height = 150

n_counties = 5000
pop_generator = mc.rdiscrete_uniform
norm = mc.rnormal

#generate some artificial population numbers
population = pop_generator(100, 1500, size = n_counties )

average_across_county = np.zeros( n_counties )
for i in range( n_counties ):
    #generate some individuals and take the mean
    average_across_county[i] = norm(mean_height, 1./std_height**2,
                                        size=population[i] ).mean()

#where are the extreme populations?
i_min = np.argmin( average_across_county )
i_max = np.argmax( average_across_county )

#plot population vs. average
plt.scatter( population, average_across_county, alpha = 0.5, c="#7A68A6")
plt.scatter( [ population[i_min], population[i_max] ],
           [average_across_county[i_min], average_across_county[i_max] ],
           s = 60, marker = "o", facecolors = "none",
           edgecolors = "#A60628", linewidths = 1.5,
            label="extreme heights")

plt.xlim( 100, 1500 )
plt.title( "Average height vs. County Population")
plt.xlabel("County Population")
plt.ylabel("Average height in county")
plt.plot( [100, 1500], [150, 150], color = "k", label = "true expected \
height", ls="--" )
plt.legend(scatterpoints = 1)
```

What do we observe? *Without accounting for population sizes* we run the risk of making an enormous inference error: if we ignored population size, we would say that the county with the shortest and tallest individuals have been correctly circled. But this inference is wrong for the following reason. These two counties do *not* necessarily have the most extreme heights. The error is that the calculated average of the small population is not a good reflection of the true expected value of the population (which should be $\mu =150$). The sample size/population size/$N$, whatever you wish to call it,  is simply too small to invoke the Law of Large Numbers effectively.

We provide more damning evidence against this inference. Recall the population numbers were uniformly distributed over 100 to 1500. Our intuition should tell us that the counties with the most extreme population heights should also be uniformly spread over 100 to 4000, and certainly independent of the county's population. Not so. Below are the population sizes of the counties with the most extreme heights.

```
print "Population sizes of 10 'shortest' counties: "
print population[ np.argsort( average_across_county )[:10] ]
print
print "Population sizes of 10 'tallest' counties: "
print population[ np.argsort( -average_across_county )[:10] ]
```

    Population sizes of 10 'shortest' counties:
    [181 168 229 110 156 123 222 154 498 375]

    Population sizes of 10 'tallest' counties:
    [105 114 111 236 373 244 183 278 234 268]

Not at all uniform over 100 to 1500. This is an absolute failure of the Law of Large Numbers.

##### Example:  Kaggle's *U.S. Census Return Rate Challenge*

Below is data from the 2010 US census, which partitions populations beyond counties to the level of block groups (which are aggregates of city blocks or equivalents). The dataset is from a Kaggle machine learning competition some colleagues and I participated in. The objective was to predict the census letter mail-back rate of a group block, measured between 0 and 100, using census variables (median income, number of females in the block-group, number of trailer parks, average number of children etc.). Below we plot the census mail-back rate versus block group population:

```
figsize( 12.5, 6.5 )
data = np.genfromtxt( "./data/census_data.csv", skip_header=1,
                        delimiter= ",")
plt.scatter( data[:,1], data[:,0], alpha = 0.5, c="#7A68A6")
plt.title("Census mail-back rate vs Population")
plt.ylabel("Mail-back rate")
plt.xlabel("population of block-group")
plt.xlim(-100, 15e3 )
plt.ylim( -5, 105)

i_min = np.argmin(  data[:,0] )
i_max = np.argmax(  data[:,0] )

plt.scatter( [ data[i_min,1], data[i_max, 1] ],
             [ data[i_min,0],  data[i_max,0] ],
             s = 60, marker = "o", facecolors = "none",
             edgecolors = "#A60628", linewidths = 1.5,
             label="most extreme points")

plt.legend(scatterpoints = 1);

```

The above is a classic phenomenon in statistics. I say *classic* referring to the "shape" of the scatter plot above. It follows a classic triangular form, that tightens as we increase the sample size (as the Law of Large Numbers becomes more exact).

I am perhaps overstressing the point and maybe I should have titled the book *"You don't have big data problems!"*, but here again is an example of the trouble with *small datasets*, not big ones. Simply, small datasets cannot be processed using the Law of Large Numbers. Compare with applying the Law without hassle to big datasets (ex. big data). I mentioned earlier that paradoxically big data prediction problems are solved by relatively simple algorithms. The paradox is partially resolved by understanding that the Law of Large Numbers creates solutions that are *stable*, i.e. adding or subtracting a few data points will not affect the solution much. On the other hand, adding or removing data points to a small dataset can create very different results.

For further reading on the hidden dangers of the Law of Large Numbers, I would highly recommend the excellent manuscript [The Most Dangerous Equation](http://nsm.uh.edu/~dgraur/niv/TheMostDangerousEquation.pdf).

##### Example: How Reddits ranks comments

You may have disagreed with the original statement that the Law of Large numbers is known to everyone, but only implicitly in our subconscious decision making. Consider ratings on online products: how often do you trust an average 5-star rating if there is only 1 reviewer? 2 reviewers? 3 reviewers? We implicitly understand that with such few reviewers that the average rating is **not** a good reflection of the true value of the product.

This has created flaws in how we sort items, and more generally, how we compare items. Many people have realized that sorting online search results by their rating, whether the objects be books, videos, or online comments, return poor results. Often the seemingly top videos or comments have perfect ratings only from a few enthusiastic fans, and truly more quality videos or comments are hidden in later pages with *falsely-substandard* ratings of around 4.8. How can we correct this?

Consider the popular site Reddit (I purposefully did not link to the website as you would never come back). The site hosts links to stories or images, and a very popular part of the site are the comments associated with each link. Redditors can vote up or down on each comment (called upvotes and downvotes). Reddit, by default, will sort comments by Top, that is, the best comments.

How would you determine which comments are the best? There are a number of ways to achieve this:

1. *Popularity*: A comment is considered good if it has many upvotes. A problem with this model is that a comment with hundreds of upvotes, but thousands of downvotes. While being very *popular*, the comment is likely more controversial than best.
2. *Difference*: Using the *difference* of upvotes and downvotes. This solves the above problem, but fails when we consider the temporal nature of comments. Comments can be posted many hours after the original link submission. The difference method will bias the *Top* comments to be the oldest comments, which have accumulated more upvotes than newer comments, but are not necessarily the best.
3. *Time adjusted*:  Consider using Difference divided by the age of the comment. This creates a *rate*, something like *difference per second*, or *per minute*. An immediate counter example is, if we use per second, a 1 second old comment with 1 upvote would be better than a 100 second old comment with 99 upvotes. One can avoid this by only considering at least t second old comments. But what is a good t value? Does this mean no comment younger than t is good? We end up comparing unstable quantities with stable quantities (young vs. old comments).
3. *Ratio*: Rank comments by the ratio of upvotes to total number of votes (upvotes plus downvotes). This solves the temporal issue, such that new comments who score well can be considered Top just as likely as older comments, provided they have many upvotes to total votes. The problem here is that a comment with a single upvote (ratio = 1.0) will beat a comment with 999 upvotes and 1 downvote (ratio = 0.999), but clearly the latter comment is *more likely* to be better.

I used the phrase *more likely* for good reason. It is possible that the former comment, with a single upvote, is in fact a better comment than the later with 999 upvotes. The hesitation to agree with this is because we have not seen the other 999 potential votes the former comment might get. Perhaps it will achieve an additional 999 upvotes and 0 downvotes and be considered better than the latter, though not likely.

What we really want is an estimate of the *true upvote ratio*. Note that the true upvote ratio is not the same as the observed upvote ratio: the true upvote ratio is hidden, and we only observe upvotes vs. downvotes (one can think of the true upvote ratio as "what is the underlying probability someone gives this comment a upvote, versus a downvote"). So the 999 upvote/1 downvote comment probably has a true upvote ratio close to 1, which we can assert with confidence thanks to the Law of Large Numbers, but on the other hand we are much less certain about the true upvote ratio of the comment with only a single upvote. Sounds like a Bayesian problem to me.

One way to determine a prior on the upvote ratio is that look at the historical distribution of upvote ratios. This can be accomplished by scrapping Reddit's comments and determining a distribution. There are a few problems with this technique though:

1. Skewed data:  The vast majority of comments have very few votes, hence there will be many comments with ratios near the extremes (see the "triangular plot" in the above Kaggle dataset), effectively skewing our distribution to the extremes. One could try to only use comments with votes greater than some threshold. Again, problems are encountered. There is a tradeoff between number of comments available to use and a higher threshold with associated ratio precision.
2. Biased data: Reddit is composed of different subpages, called subreddits. Two examples are *r/aww*, which posts pics of cute animals, and *r/politics*. It is very likely that the user behaviour towards comments of these two subreddits are very different: visitors are likely friend and affectionate in the former, and would therefore upvote comments more, compared to the latter, where comments are likely to be controversial and disagreed upon. Therefore not all comments are the same.

In light of these, I think it is better to use a `Uniform` prior.

With our prior in place, we can find the posterior of the true upvote ratio. The Python script `comments_for_top_reddit_pic.py` will scrap the comments from the current top picture on Reddit. Below is the picture, and some comments:

```
from IPython.core.display import Image
#adding a number to the end of the %run call with get the ith top photo.
%run top_pic_comments.py 2

Image(top_post_url)

```

    Version 2.0.14 of praw is outdated. Version 2.0.15 was released Saturday April 06, 2013.
    Title of submission:
    3 Wolves in Quebec, Canada
    http://i.imgur.com/aHes4aF.jpg

```
"""
contents: an array of the text from all comments on the pic
votes: a 2d numpy array of upvotes, downvotes for each comment.
"""
n_comments = len(contents )
comments = np.random.randint( n_comments, size=4)
print "Some Comments (out of %d total) \n-----------"%n_comments
for i in comments:
    print '"' + contents[i] + '"'
    print"upvotes/downvotes: ",votes[i,:]
    print
```

    Some Comments (out of 49 total)
    -----------
    "Shaggydog, Nymeria, and Ghost?"
    upvotes/downvotes:  [134  17]

    "I call bullshit. Not one of them is howling, nor is there a moon present."
    upvotes/downvotes:  [80 13]

    "That would make a badass t shirt."
    upvotes/downvotes:  [10  0]

    "That is the alpha male and he is alerting his pack that he has found a Tim Hortons."
    upvotes/downvotes:  [2 0]

 For a given true upvote ratio $p$ and $N$ votes, the number of upvotes will look like a Binomial random variable with parameters $p$ and $N$. (This is because of the equivalence between upvote ratio and probability of upvoting versus downvoting, out of $N$ possible votes/trials). We create a function that performs Bayesian inference on $p$, for a particular comment's upvote/downvote pair.

```
import pymc as mc

def posterior_upvote_ratio( upvotes, downvotes, samples = 20000):
    N = upvotes + downvotes
    upvote_ratio = mc.Uniform( "upvote_ratio", 0, 1 )
    observations = mc.Binomial( "obs",  N, upvote_ratio, value = upvotes, observed = True)

    model = mc.Model( [upvote_ratio, observations ])
    map_ = mc.MAP(model).fit()
    mcmc = mc.MCMC(model)

    mcmc.sample(samples, samples/4)

    return mcmc.trace("upvote_ratio")[:]

```

Below are the resulting posterior distributions.

```
figsize( 11., 8)
posteriors = []
colours = ["#348ABD", "#A60628", "#7A68A6", "#467821", "#CF4457"]
for i in range(len(comments)):
    j = comments[i]
    posteriors.append( posterior_upvote_ratio( votes[j, 0], votes[j,1] ) )
    plt.hist( posteriors[i], bins = 18, normed = True, alpha = .9,
            histtype="step",color = colours[i%5], lw = 3,
            label = '(%d up:%d down)\n%s...'%(votes[j, 0], votes[j,1], contents[j][:50]) )
    plt.hist( posteriors[i], bins = 18, normed = True, alpha = .2,
            histtype="stepfilled",color = colours[i], lw = 3, )
    #plt.title( 'upvotes:downvotes: %d:%d'%(votes[j,0], votes[j,1]) )

plt.legend(loc="upper left")
plt.xlim( 0, 1)
plt.title("Posterior distributions of upvote ratios on different comments");

```

Some distributions are very tight, others have very long tails (relatively speaking), expressing our uncertainty with what the true upvote ratio might be.

### Sorting!

We have been ignoring the goal of this exercise: how do we sort the comments from *best to worst*? Of course, we cannot sort distributions, we must sort scalar numbers. There are many ways to distill a distribution down to a scalar: expressing the distribution through its expected value, or mean, is one way. Choosing the mean bad choice though. This is because the mean does not take into account the uncertainty of distributions.

I  suggest using the *95% least plausible value*, defined as the value such that there is only a 5% chance the true parameter is lower (think of the lower bound on the 95% credible region). Below are the posterior distributions with the 95% least-plausible value plotted:

```
N = posteriors[0].shape[0]
lower_limits = []

for i in range(len(comments)):
    j = comments[i]
    plt.hist( posteriors[i], bins = 20, normed = True, alpha = .9,
            histtype="step",color = colours[i], lw = 3,
            label = '(%d up:%d down)\n%s...'%(votes[j, 0], votes[j,1], contents[j][:50]) )
    plt.hist( posteriors[i], bins = 20, normed = True, alpha = .2,
            histtype="stepfilled",color = colours[i], lw = 3, )
    v = np.sort( posteriors[i] )[ int(0.05*N) ]
    #plt.vlines( v, 0, 15 , color = "k", alpha = 1, linewidths=3 )
    plt.vlines( v, 0, 10 , color = colours[i], linestyles = "--",  linewidths=3  )
    lower_limits.append(v)
    plt.legend(loc="upper left")

plt.legend(loc="upper left")

plt.title("Posterior distributions of upvote ratios on different comments");
order = argsort( -np.array( lower_limits ) )
print order, lower_limits

```

The best comments, according to our procedure, are the comments that are *most-likely* to score a high percentage of upvotes. Visually those are the comments with the 95% least plausible value close to 1.

Why is sorting based on this quantity a good idea? By ordering by the 95% least plausible value, we are being the most conservative with what we think is best. That is, even in the worst case scenario, when we have severely overestimated the upvote ratio, we can be sure the best comments are still on top. Under this ordering, we impose the following very natural properties:

1. given two comments with the same observed upvote ratio, we will assign the comment with more votes as better (since we are more confident it has a higher ratio).
2. given two comments with the same number of votes, we still assign the comment with more upvotes as *better*.

### But this is too slow for real-time!

I agree, computing the posterior of every comment takes a long time, and by the time you have computed it, likely the data has changed. I delay the mathematics to the appendix, but I suggest using the following formula to compute the lower bound very fast.

$$ \frac{a}{a + b} - 1.65\sqrt{ \frac{ab}{ (a+b)^2(a + b +1 ) } }$$

where
\begin{align}
& a = 1 + u \\\\
& b = 1 + d \\\\
\end{align}

$u$ is the number of upvotes, and $d$ is the number of downvotes. The formula is a shortcut in Bayesian inference, which will be further explained in Chapter 6 when we discuss priors in more detail.

```
def intervals(u,d):
    a = 1. + u
    b = 1. + d
    mu = a/(a+b)
    std_err = 1.65*np.sqrt( (a*b)/( (a+b)**2*(a+b+1.) ) )
    return ( mu, std_err )

print "Approximate lower bounds:"
posterior_mean, std_err  = intervals(votes[:,0],votes[:,1])
lb = posterior_mean - std_err
print lb
print

print "Top 40 Sorted according to approximate lower bounds:"
print
order = np.argsort( -lb )
ordered_contents = []
for i in order[:40]:
    ordered_contents.append( contents[i] )
    print  votes[i,0], votes[i,1], contents[i]
    print "-------------"

```

    Approximate lower bounds:
    [ 0.85943711  0.83951434  0.8458571   0.84536604  0.86388258  0.83651439
      0.89239065  0.7929375   0.85447643  0.70806405  0.79018506  0.82545948
      0.70408053  0.79198214  0.60100251  0.6931046   0.6931046   0.6530083
      0.7137931   0.60091591  0.60091591  0.60091591  0.60091591  0.60091591
      0.52182581  0.60100251  0.53055613  0.53055613  0.53055613  0.56085485
      0.51184301  0.4722123   0.43047887  0.43047887  0.43047887  0.53055613
      0.43047887  0.43047887  0.43047887  0.53055613  0.43047887  0.43047887
      0.43047887  0.43047887  0.43047887  0.43047887  0.43047887  0.43047887
      0.43047887]

    Top 40 Sorted according to approximate lower bounds:

    22 0 Being as an ocean?
    -------------
    354 43 Ok, three-wolf-joke-in-my-second-language time:

    Un jours trois loups chassent ensemble dans la forêt: un Québécois, un Franco-Ontarien et un Français. Soudainement, **PAF!**, ils tombent dans un piège à loup et ont chacun une patte poigner.

    "Oh ben *tabarnak*" dit le Franco-Ontarien. "Là on fait quoi, crisse? Les chasseurs vont nous tuer!"

    "J'ai une idée!" exclame le Québécois. "Si on s'mange une patte on pourrait nous libérer. C'est ben plate, mais on n'a pas le choix." Aussitôt dit, aussitôt fait, et Québécois et le Franco-Ontarien s'en sortent. Alors qu'ils allaient sortir de la forêt, le Franco-Ontarien dit "Hey! Y'est où le Français d'abord?"

    "Osti d'sappé," répond le Québécois, "là on va avoir à le sauver!" Les deux retournent sur le lieu de leur capture et voient le Français, toujours emprisonné, qui leur dit

    "Bordel, vous avez fait comment les mecs? Je m'suis déjà bouffé trois pattes et je suis encore bloqué!"
    -------------
    362 46 This was done by photographer Daniel Parent. [[Source](http://500px.com/photo/7505677?from=set/285118)]  -- [Check out his collection of wolf photos here](http://500px.com/djlparent99/sets/wolves)

    Here are some of my favorites from the collection:

    - OP's one, called ["Three Weary Wolves"](http://i.imgur.com/dW2bsks.jpg) [707 x 900 Version]

    - ["The Crossing" (with 12 wolves)](http://i.imgur.com/J5bKJsu.jpg)

    - ["Above and Beyond"](http://i.imgur.com/bJHbrNa.jpg)

    - ["On Patrol"](http://i.imgur.com/TKyST9h.jpg)
    -------------
    44 3 Are they also in a Christian hard rock band?
    -------------
    33 2 Princess Mononoke?
    -------------
    805 126             / \      _-'
              _/|  \-''- _ /    /\__/\
         __~' { |          \   /      \
             /              \  | -  - |
             /       "ಠ.  |ಠ } \     /|
             |            \ ;   \ T / |\
                           ',         |
                \_         __\
                  ''-_    \.//
                    / '-____'
                   /           n,   Sorry.
                  /          _/ | _
                _'         /'  `'/
              _-'         <~    .'
                         .'    |
                      _/      |
                   _/      `.`.
                     / '   \__ | |
           ___/      /__\ \ \
          (___.'\_______)\_|_|

    -------------
    134 17 Shaggydog, Nymeria, and Ghost?
    -------------
    106 13 [Was this picture taken in Rivière-du-Loup?](http://www.badumtss.net/)
    -------------
    21 1 Dire Wolves. Fucking sexy ass Dire Wolves.
    -------------
    80 13 I call bullshit. Not one of them is howling, nor is there a moon present.
    -------------
    17 1 Yayy Quebec on the front page :)
    -------------
    10 0 That would make a badass t shirt.
    -------------
    23 4 where's moon moon?
    -------------
    11 1 Good Ol' Quebec, Canada. Land of wolves, maple syrup and Labatt 50.
    -------------
    22 4 No retarded comment from me. This is a nice picture!
    -------------
    6 0 I recognize this from a Being As An Ocean band t-shirt.

    Such a cool picture.
    -------------
    6 0 Looks like a boy band posing for their new album cover.
    -------------
    5 0 Is that Wolf+1's new album cover?
    -------------
    7 1 On a tu pris ces photos à Rivière-du-Loup  ?? :-)
    -------------
    7 1 Mon tabarnac c'est dont ben des beaux loups esti!

    -------------
    4 0 This image has actually been reprinted into a REALLY nice shirt by the band Being As An Ocean

    http://merchnow.com/products/149014/alpine-wolf-white
    -------------
    4 0 where, specifically in Quebec was that photo taken?
    -------------
    4 0 I like this one, one wolf is facing one way another facing the other way and the wolf in the back is like whaddya want from me
    -------------
    4 0 reminds me of princess mononoke
    -------------
    4 0 [As a Magic: The Gathering card](http://i.imgur.com/KI74xCc.jpg)
    -------------
    6 1 Winter is coming.
    -------------
    3 0 I suppose this is the time to post [this](http://www.reddit.com/r/pics/comments/1d1qs4/i_wrestled_those_wolves_in_quebec_canada/) picture.

    Me, in Quebec, joking around with some wolves.
    -------------
    3 0 Looks like the Christ + 1 album cover.
    -------------
    3 0 Where exactly was it taken?
    -------------
    3 0 J'adore!
    -------------
    3 0 Looks like Cartman tried to make an christian rock album cover
    -------------
    9 3 Did Eric Cartman stage this for an album cover?
    -------------
    5 1 [3 Wolf River](http://i.imgur.com/W01N0G4.png) saved my marriage! I used to wear boring 3-piece suits and platinum cufflinks, but my lady got tired of my buttoned-down looks; she wanted a wildman.

    Cue 3 Wolf River: this magnificent fucking shirt has not one, NOT TWO, BUT *THREE* stunning, predatory killing machines featured in a sharp art-deco arrangement. Also there's an awesome river, full of water and freshwater crabs and shit. My lady saw me sporting this bad boy and we had to send the kids away. Not for the weekend; *forever.*

    The amount of sheer manliness contained within these threads was not meant for mortal men, and yet *you* could own one of these majestic chest-hiders for *only $24.99!*
    -------------
    9 4 I hope those Stark kids pick up after them.
    -------------
    2 0 Dude i love how quebec looks, lived there nearly my entire life
    -------------
    2 0 3 wolf canada...
    -------------
    2 0 MAJESTIC AS FUCK!
    -------------
    2 0 For those of you who were wondering where those pictures came from, a lot of them were taken at Park Omega (http://www.parc-omega.com/)
    -------------
    2 0 That is the alpha male and he is alerting his pack that he has found a Tim Hortons.
    -------------
    2 0 Like an all wolf [Christian rock band](http://imgur.com/9yR3V6R.jpg)
    -------------

We can view the ordering visually by plotting the posterior mean and bounds, and sorting by the lower bound. In the plot below, notice that the left error-bar is sorted (as we suggested this is the best way to determine an ordering), so the means, indicated by dots, do not follow any strong pattern.

```
r_order = order[::-1][:40]
plt.errorbar( posterior_mean[r_order], np.arange( len(r_order) ),
               xerr=std_err[r_order],xuplims=True, capsize=0, fmt="o",
                color = "#7A68A6")
plt.xlim( 0.3, 1)
plt.yticks( np.arange( len(r_order)-1,-1,-1 ), map( lambda x: x[:30], ordered_contents) );

```

In the graphic above, you can see why sorting by mean would be sub-optimal.

### Conclusion

While the Law of Large Numbers is cool, it is only true so much as its name implies: with large sample sizes only. We have seen how our inference can be affected by not considering *how the data is shaped*.

1. By (cheaply) drawing many samples from the posterior distributions, we can ensure that the Law of Large Number applies as we approximate expected values (which we will do in the next chapter).

2. Bayesian inference understands that with small sample sizes, we can observe wild randomness. Our posterior distribution will reflect this by being more spread rather than tightly concentrated. Thus, our inference should be correctable.

3. There are major implications of not considering the sample size, and trying to sort objects that are unstable leads to pathological orderings. The method provided above solves this problem.

### Appendix

##### Derivation of sorting comments formula

Basically what we are doing is using a Beta prior (with parameters $a=1, b=1$, which is a uniform distribution), and using a Binomial likelihood with observations $u, N = u+d$. This means our posterior is a Beta distribution with parameters $a' = 1 + u, b' = 1 + (N - u) = 1+d$. We then need to find the value, $x$, such that 0.05 probability is less than $x$. This is usually done by inverting the CDF ([Cumulative Distribution Function](http://en.wikipedia.org/wiki/Cumulative_Distribution_Function)), but the CDF of the beta, for integer parameters, is known but is a large sum [3].

We instead using a Normal approximation. The mean of the Beta is $\mu = a'/(a'+b')$ and the variance is

$$\sigma^2 = \frac{a'b'}{ (a' + b')^2(a'+b'+1) }$$

Hence we solve the following equation for $x$ and have an approximate lower bound.

$$ 0.05 = \Phi\left( \frac{(x - \mu)}{\sigma}\right) $$

$\Phi$ being the [cumulative distribution for the normal distribution](http://en.wikipedia.org/wiki/Normal_distribution#Cumulative_distribution)

##### Exercises

1\. How would you estimate the quantity $E\left[ \cos{X} \right]$, where $X \sim \text{Exp}(4)$? What about $E\left[ \cos{X} | X \lt 1\right]$, i.e. the expected value *given* we know $X$ is less than 1? Would you need more samples than the original samples size to be equally as accurate?

```
## Enter code here
import scipy.stats as stats
exp = stats.expon( 4 )
N = 1e5
X = exp.rvs( N )
## ...
```

2\. The following table was located in the paper "Going for Three: Predicting the Likelihood of Field Goal Success with Logistic Regression" [2]. The table ranks football field-goal kickers by there percent of non-misses. What mistake have the researchers made?

-----

####  Kicker Careers Ranked by Make Percentage
<table><tbody><tr><th>Rank </th><th>Kicker </th><th>Make % </th><th>Number  of Kicks</th></tr><tr><td>1 </td><td>Garrett Hartley </td><td>87.7 </td><td>57</td></tr><tr><td>2</td><td> Matt Stover </td><td>86.8 </td><td>335</td></tr><tr><td>3 </td><td>Robbie Gould </td><td>86.2 </td><td>224</td></tr><tr><td>4 </td><td>Rob Bironas </td><td>86.1 </td><td>223</td></tr><tr><td>5</td><td> Shayne Graham </td><td>85.4 </td><td>254</td></tr><tr><td>… </td><td>… </td><td>…</td><td> </td></tr><tr><td>51</td><td> Dave Rayner </td><td>72.2 </td><td>90</td></tr><tr><td>52</td><td> Nick Novak </td><td>71.9 </td><td>64</td></tr><tr><td>53 </td><td>Tim Seder </td><td>71.0 </td><td>62</td></tr><tr><td>54 </td><td>Jose Cortez </td><td>70.7</td><td> 75</td></tr><tr><td>55 </td><td>Wade Richey </td><td>66.1</td><td> 56</td></tr></tbody></table>

### References

1. Wainer, Howard. *The Most Dangerous Equation*. American Scientist, Volume 95.
2. Clarck, Torin K., Aaron W. Johnson, and Alexander J. Stimpson. "Going for Three: Predicting the Likelihood of Field Goal Success with Logistic Regression." (2013): n. page. Web. 20 Feb. 2013.
3. http://en.wikipedia.org/wiki/Beta_function#Incomplete_beta_function

```
from IPython.core.display import HTML
def css_styling():
    styles = open("../styles/custom.css", "r").read()
    return HTML(styles)
css_styling()
```

<style>
    @font-face {
        font-family: "Computer Modern";
        src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
    }
    div.cell{
        width:800px;
        margin-left:auto;
        margin-right:auto;
    }
    h1 {
        font-family: Helvetica, serif;
    }
    h4{
        margin-top:12px;
        margin-bottom: 3px;
       }
    div.text_cell_render{
        font-family: Computer Modern, "Helvetica Neue", Arial, Helvetica, Geneva, sans-serif;
        line-height: 145%;
        font-size: 130%;
        width:800px;
        margin-left:auto;
        margin-right:auto;
    }
    .CodeMirror{
            font-family: "Source Code Pro", source-code-pro,Consolas, monospace;
    }
    .prompt{
        display: None;
    }
    .text_cell_render h5 {
        font-weight: 300;
        font-size: 16pt;
        color: #4057A1;
        font-style: italic;
        margin-bottom: .5em;
        margin-top: 0.5em;
        display: block;
    }

    .warning{
        color: rgb( 240, 20, 20 )
        }

</style>

<style>
    img{
        max-width:800px}
</style>


# Solving orbital equations with different algorithms

This notebook was adapted from `Orbit_games.ipynb`.

We consider energy plots and orbital solutions in polar coordinates for the general potential energy

$\begin{align}
   U(r) = k r^n
\end{align}$

for different ODE solution algorithms.  The `solve_ivp` function can itself be specified to use different solution methods (with the `method` keyword).  Here we will set it by default to use 'RK23', which is a variant on the Runge-Kutta second-order algorithm.  Second-order in this context means that the accuracy of a calculation will improve by a factor of $10^2 = 100$ if $\Delta t$ is reduced by a factor of ten.

We will compare it with the crudest algorithm, Euler's method, which is first order, and a second-order algorithm called Leapfrog, which is designed to be precisely time-reversal invariant.  This property guarantees conservation of energy, which is not true of the other algorithms we will consider.

To solve the differential equations for orbits, we have defined the $\mathbf{y}$
and $d\mathbf{y}/dt$ vectors as

$\begin{align}
  \mathbf{y} = \left(\begin{array}{c} r(t) \\ \dot r(t) \\ \phi(t)  \end{array} \right)
  \qquad
  \frac{d\mathbf{y}}{dt}
       = \left(\begin{array}{c} \dot r(t) \\ \ddot r(t) \\ \dot\phi(t) \end{array} \right)
       = \left(\begin{array}{c} \dot r(t) \\
                                 -\frac{1}{\mu}\frac{dU_{\rm eff}(r)}{dr} \\
                                 \frac{l}{\mu r^2} \end{array} \right)
\end{align}$

where we have substituted the differential equations for $\ddot r$ and $\dot\phi$.

Then Euler's method can be written as a simple prescription to obtain $\mathbf{y}_{i+1}$
from $\mathbf{y}_i$, where the subscripts label the elements of the `t_pts` array:
$\mathbf{y}_{i+1} = \mathbf{y}_i + \left(d\mathbf{y}/dt\right)_i \Delta t$, or, by components:

$\begin{align}
   r_{i+1} &= r_i + \frac{d\mathbf{y}_i[0]}{dt}  \Delta t  \\
   \dot r_{i+1} &= \dot r_{i} + \frac{d\mathbf{y}_i[1]}{dt}  \Delta t \\
   \phi_{i+1} &= \phi_i + \frac{d\mathbf{y}_i[2]}{dt} \Delta t
\end{align}$

**Look at the** `solve_ode_Euler` **method below and verify the algorithm is correctly implemented.**

The leapfrog method does better by evaluating $\dot r$ at a halfway time step before and after the $r$ evaluation,
which is both more accurate and incorporates time reversal:

$\begin{align}
   \dot r_{i+1/2} &= \dot r_{i} + \frac{d\mathbf{y}_i[1]}{dt}  \Delta t/2 \\
   r_{i+1} &= r_i +  \dot r_{i+1/2}  \Delta t  \\
   \dot r_{i+1} &= \dot r_{i+1/2} + \frac{d\mathbf{y}_{i+1}[1]}{dt}  \Delta t/2 \\
   \phi_{i+1} &= \phi_i + \frac{d\mathbf{y}_i[2]}{dt} \Delta t
\end{align}$

**Look at the** `solve_ode_Leapfrog` **method below and verify the algorithm is correctly implemented.**

A third method is the second-order Runge-Kutta algorithm, which we invoke from `solve_ivp` as `RK23`.
It does not use a fixed time-step as in our "homemade" implementations, so there is not a direct
comparison, but we can still check if it conserves energy.

**Run the notebook.  You are to turn in and comment on the "Change in energy with time" plot at the end.
Where do you see energy conserved or not conserved?  Show that Euler is first order and leapfrog is second
order by changing $\Delta t$; describe what you did and what you found.**

**Try another potential to see if you get the same general conclusions.**

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.integrate import solve_ivp
```

```python
# Change the common font size
font_size = 14
plt.rcParams.update({'font.size': font_size})
```

```python
class Orbit:
    """
    Potentials and associated differential equations for central force motion
    with the potential U(r) = k r^n.  Several algorithms for integration of
    ordinary differential equations are now available.
    """

    def __init__(self, m_1=1., m_2=1., G=1,):
        self.m_1 = m_1
        self.m_2 = m_2
        self.G = G

    def dz_dt(self, t, z):
        """
        This function returns the right-hand side of the diffeq:
        [dz/dt d^2z/dt^2]

        Parameters
        ----------
        t : float
            time
        z : float
            8-component vector with
            z[0] = x_1(t), z[1] = x_dot_1(t)
            z[2] = y_1(t), z[3] = y_dot_1(t)
            z[4] = x_2(t), z[5] = x_dot_2(t)
            z[6] = y_2(t), z[7] = y_dot_2(t)

        """
        r_12=np.sqrt( (z[0]-z[4])**2 + (z[2]-z[6])**2 )

        return [ z[1], self.G  * self.m_2 * (z[4] - z[0]) / r_12 ** 3,\
                 z[3], self.G  * self.m_2 * (z[6] - z[2]) / r_12 ** 3,\
                 z[5], -self.G * self.m_1 * (z[4] - z[0]) / r_12 ** 3,\
                 z[7], -self.G * self.m_1 * (z[6] - z[2]) / r_12 ** 3 ]

    def solve_ode(self, t_pts, z_0,
                  abserr=1.0e-8, relerr=1.0e-8):
        """
        Solve the ODE given initial conditions.
        Use solve_ivp with the option of specifying the method.
        Specify smaller abserr and relerr to get more precision.
        """
        solution = solve_ivp(self.dz_dt, (t_pts[0], t_pts[-1]),
                             z_0, t_eval=t_pts, method='RK23',
                             atol=abserr, rtol=relerr)
        x_1, x_dot_1, y_1, y_dot_1, x_2, x_dot_2, y_2, y_dot_2 = solution.y
        return x_1, x_dot_1, y_1, y_dot_1, x_2, x_dot_2, y_2, y_dot_2

    def solve_ode_Leapfrog(self, t_pts, z_0):
        """
        Solve the ODE given initial conditions with the Leapfrog method.
        """
        delta_t = t_pts[1] - t_pts[0]

        x_1_0, x_dot_1_0, y_1_0, y_dot_1_0, x_2_0, x_dot_2_0, y_2_0, y_dot_2_0 = z_0

        # initialize the arrays with zeros
        num_t_pts = len(t_pts)

        x_1 = np.zeros(num_t_pts)
        x_dot_1 = np.zeros(num_t_pts)
        x_dot_1_half = np.zeros(num_t_pts)

        y_1 = np.zeros(num_t_pts)
        y_dot_1 = np.zeros(num_t_pts)
        y_dot_1_half = np.zeros(num_t_pts)

        x_2 = np.zeros(num_t_pts)
        x_dot_2 = np.zeros(num_t_pts)
        x_dot_2_half = np.zeros(num_t_pts)

        y_2 = np.zeros(num_t_pts)
        y_dot_2 = np.zeros(num_t_pts)
        y_dot_2_half = np.zeros(num_t_pts)

        # initial conditions
        x_1[0] = x_1_0
        x_dot_1[0] = x_dot_1_0

        y_1[0] = y_1_0
        y_dot_1[0] = y_dot_1_0

        x_2[0] = x_2_0
        x_dot_2[0] = x_dot_2_0

        y_2[0] = y_2_0
        y_dot_2[0] = y_dot_2_0

        # step through the differential equation
        for i in np.arange(num_t_pts - 1):
            t = t_pts[i]

            z = [x_1[i], x_dot_1[i], y_1[i], y_dot_1[i],\
                 x_2[i], x_dot_2[i], y_2[i], y_dot_2[i] ]

            out=self.dz_dt(t,z)

            x_dot_1_half[i] = x_dot_1[i] + out[1] * delta_t/2.
            x_1[i+1] = x_1[i] + x_dot_1_half[i] * delta_t

            y_dot_1_half[i] = y_dot_1[i] + out[3] * delta_t/2.
            y_1[i+1] = y_1[i] + y_dot_1_half[i] * delta_t

            x_dot_2_half[i] = x_dot_2[i] + out[5] * delta_t/2.
            x_2[i+1] = x_2[i] + x_dot_2_half[i] * delta_t

            y_dot_2_half[i] = y_dot_2[i] + out[7] * delta_t/2.
            y_2[i+1] = y_2[i] + y_dot_2_half[i] * delta_t

            z = [x_1[i+1], x_dot_1[i], y_1[i+1], y_dot_1[i],\
                 x_2[i+1], x_dot_2[i], y_2[i+1], y_dot_2[i] ]
            out = self.dz_dt(t,z)

            x_dot_1[i+1] = x_dot_1_half[i] + out[1] * delta_t/2.
            y_dot_1[i+1] = y_dot_1_half[i] + out[3] * delta_t/2.
            x_dot_2[i+1] = x_dot_2_half[i] + out[5] * delta_t/2.
            y_dot_2[i+1] = y_dot_2_half[i] + out[7] * delta_t/2.

        return x_1, x_dot_1, y_1, y_dot_1, x_2, x_dot_2, y_2, y_dot_2
```

```python
def start_stop_indicies(t_pts, plot_start, plot_stop):
    start_index = (np.fabs(t_pts-plot_start)).argmin()  # index in t_pts array
    stop_index = (np.fabs(t_pts-plot_stop)).argmin()  # index in t_pts array
    return start_index, stop_index
```

```python
def plot_y_vs_x(x, y, axis_labels=None, label=None, title=None,
                color=None, linestyle=None, semilogy=False, loglog=False,
                ax=None):
    """
    Generic plotting function: return a figure axis with a plot of y vs. x,
    with line color and style, title, axis labels, and line label
    """
    if ax is None:        # if the axis object doesn't exist, make one
        ax = plt.gca()
    if (semilogy):
        line, = ax.semilogy(x, y, label=label,
                            color=color, linestyle=linestyle)

    elif (loglog):
        line, = ax.loglog(x, y, label=label,
                          color=color, linestyle=linestyle)
    else:
        line, = ax.plot(x, y, label=label,
                    color=color, linestyle=linestyle)

    if label is not None:    # if a label if passed, show the legend
        ax.legend()
    if title is not None:    # set a title if one if passed
        ax.set_title(title)
    if axis_labels is not None:  # set x-axis and y-axis labels if passed
        ax.set_xlabel(axis_labels[0])
        ax.set_ylabel(axis_labels[1])

    return ax, line
```

# Start The Plots
The first one is a normal plot using 'RK23', the second one is using Leapfrog method, the thrid one is a simple plot, setting the initial conditions so that they move in the opposite directions.

```python
#Labels for individual plot axes
orbit_labels=(r'$x$',r'$y$')

#common plotting time
t_start=0.
t_end=10.
delta_t=0.01
t_pts=np.arange(t_start, t_end+delta_t,delta_t)

G = 1.
m_1 = 1.
m_2 = 5.

o1 = Orbit(m_1, m_2, G)

#Initial conditions with the velocity of the center of mass to be 0
x_1_0, x_dot_1_0 = 1., -1.
y_1_0, y_dot_1_0 = 1., 1.
x_2_0, x_dot_2_0 = -(m_1 / m_2) * x_1_0, -(m_1 / m_2) * x_dot_1_0
y_2_0, y_dot_2_0 = -(m_1 / m_2) * y_1_0, -(m_1 / m_2) * y_dot_1_0

z_0 = [x_1_0, x_dot_1_0, y_1_0, y_dot_1_0,\
       x_2_0, x_dot_2_0, y_2_0, y_dot_2_0]

x_1,x_dot_1, y_1, y_dot_1, x_2, x_dot_2, y_2, y_dot_2 = o1.solve_ode(t_pts,z_0)

```

```python
fig = plt.figure(figsize=(7,5))

ax = fig.add_subplot(1,1,1)

ax.plot(x_1, y_1,  color='blue', label=r'$m_1$')
ax.plot(x_2, y_2,  color='red', label=r'$m_2$')

start, stop = start_stop_indicies(t_pts, t_start, t_end)

ax.set_title('Simple gravitational orbit')

ax.legend()
ax.set_aspect(1)

fig.tight_layout()

fig.savefig('Gravitation_orbit.png')

```

## Plot orbit and check energy conservation

```python
#common plotting time
t_start=0.
t_end=20.
delta_t=0.001
t_pts=np.arange(t_start, t_end+delta_t,delta_t)

G = 10.
m_1 = 10.
m_2 = 1.

o2 = Orbit(m_1, m_2, G)

#Initial conditions with the velocity of the center of mass to be 0
x_1_0, x_dot_1_0 = 0.1, 0.
y_1_0, y_dot_1_0 = 0., 0.75
x_2_0, x_dot_2_0 = -(m_1 / m_2) * x_1_0, -(m_1 / m_2) * x_dot_1_0
y_2_0, y_dot_2_0 = -(m_1 / m_2) * y_1_0, -(m_1 / m_2) * y_dot_1_0

z_0 = [x_1_0, x_dot_1_0, y_1_0, y_dot_1_0,\
       x_2_0, x_dot_2_0, y_2_0, y_dot_2_0]

x_1,x_dot_1, y_1, y_dot_1, x_2, x_dot_2, y_2, y_dot_2 = o2.solve_ode_Leapfrog(t_pts,z_0)
```

```python
fig = plt.figure(figsize=(5,5))

ax = fig.add_subplot(1,1,1)

ax.plot(x_1, y_1,  color='blue', label=r'$m_1$')
ax.plot(x_2, y_2,  color='red', label=r'$m_2$')

start, stop = start_stop_indicies(t_pts, t_start, t_end)

ax.set_title('Simple gravitational orbit')

ax.legend()
ax.set_aspect(1)

fig.tight_layout()

fig.savefig('Gravitation_orbit_LeapFrog.png')

```

```python
#common plotting time
t_start=0.
t_end=50.
delta_t=0.01
t_pts=np.arange(t_start, t_end+delta_t,delta_t)

G = 10.
m_1 = 1.
m_2 = 1.

o2 = Orbit(m_1, m_2, G)

#Initial conditions with the velocity of the center of mass to be 0
x_1_0, x_dot_1_0 = 1., 0.
y_1_0, y_dot_1_0 = 0., 1.
x_2_0, x_dot_2_0 = -(m_1 / m_2) * x_1_0, -(m_1 / m_2) * x_dot_1_0
y_2_0, y_dot_2_0 = -(m_1 / m_2) * y_1_0, -(m_1 / m_2) * y_dot_1_0

z_0 = [x_1_0, x_dot_1_0, y_1_0, y_dot_1_0,\
       x_2_0, x_dot_2_0, y_2_0, y_dot_2_0]

x_1,x_dot_1, y_1, y_dot_1, x_2, x_dot_2, y_2, y_dot_2 = o2.solve_ode(t_pts,z_0)
```

```python
fig = plt.figure(figsize=(5,5))

ax = fig.add_subplot(1,1,1)

ax.plot(x_1, y_1,  color='blue', label=r'$m_1$')
ax.plot(x_2, y_2,  color='red', label=r'$m_2$')

start, stop = start_stop_indicies(t_pts, t_start, t_end)

ax.set_title('Simple gravitational orbit')

ax.legend()
ax.set_aspect(1)

fig.tight_layout()

fig.savefig('Simple orbits.png')

```

```python

```

```python

```


### Getting started with functions

```python
# Define a really simple function
# the keyword "def" denotes the start of a function definition
# it is followed by function name and arguments and then a :
#
# Function definition is all the indented lines after the :
#
def printhi():
    print("hi")

```

```python
# To invoke a function use its name followed by parentheses.
# This executes the code in the function
printhi()
printhi()
```

##### Functions can also take arguments

```python
# Here there is a required argument name, that is used in the print statement.
# It can be different for each call of the function.
def printhiname(name):
    print("hi", name)
```

```python
printhiname("Chris")
printhiname("Tom")
printhiname()
```

##### Argments are required by default, but this can be altered by providing a default value

```python
# Here is an example of adding a default value for a function argument
def printhiname(name="NO NAME"):
    print("hi", name)
```

```python
printhiname()
printhiname("Chris")
printhiname(name="Chris")
```

##### Functions can return computation results

```python
def calc_sum(arr):
    asum=0.
    for aval in arr:
        asum=asum+aval
    return asum
```

```python
calc_sum([0,1,2])
```

```python
#
# Now lets define a function for plotting sea-level that we can use for any location
#
def plot_sl(scode="WOODS"):
    """ A function to plot the time-series of tide-gauge water level measurments (relative
    to local solid Earth) for historical data at the tide gauge archive https://www.psmsl.org.
    """
    import pandas
    import matplotlib.pyplot as plt
    station_list=pandas.read_html("https://www.psmsl.org/data/obtaining/")[0]

    matched_stations=station_list[station_list['Station Name'].str.contains(scode)==True]
    ls=len(matched_stations)
    print(ls,"matching stations found.")

    if ls == 0:
        return

    if ls > 1:
        for i,r in matched_stations.iterrows():
            print("%-30.30s %s"%(r["Station Name"],r["Country"]))
        return

    # Here we only have 1 station so we can make a plot
    snum=matched_stations["ID"]
    surl="http://www.psmsl.org/data/obtaining/rlr.monthly.data/%d.rlrdata"%(snum)
    df = pandas.read_csv(surl,delimiter=';')
    df=df[df.iloc[:,1]>=-1000]
    npdat =df.to_numpy()
    t=npdat[:,0];h=npdat[:,1]
    plt.rcParams['figure.figsize'] = [20, 10]
    plt.plot(t,(h-h[0])/10);
    return
```

```python
plot_sl("BOSTON")
```

```python
# Lets show the doc string
plot_sl?
```

```python
# One final piece of function syntax
def print_vars(*args):
    for a in args:
        print(a)
    return

print_vars(1)
print("")
print_vars(1,2)

print("")

def print_vars_with_kw(*args,**kwargs):
    for a in args:
        print(a)
    print(kwargs.keys())
    print(kwargs["x"])

    return

print_vars_with_kw(1,x=7)

```

#### Now lets look functions for solving some equations

```python
# First a simple recursive function
```

```python
# This function calls itself recursvely with n reduced by 1 each time until n is 1.
def factorial(n):
    if n==1 or n==0:
        return 1
    else:
        return n * factorial(n-1)
```

```python
factorial(23)
```

#### Using functions to make a Lorenz 63 "butterfly" plot
( https://en.wikipedia.org/wiki/Lorenz_system )

Lorenz 63 is a system of 3 equations devepoed by Ed Lorenz ( https://en.wikipedia.org/wiki/Edward_Norton_Lorenz ) in build 54 in the 1960's, with computhing help from Ellen Fetter ( https://en.wikipedia.org/wiki/Ellen_Fetter ). The system is famous for being a very simple demonstration of a "chaotic" system.

A chaotic system is a system in which a very small change in the initial conditions can get amplified in an unpredictable way so that two "trajectories" that are initially close separate to become very different. In a chaotic system, after some time, the difference between two similar but slight different initial trajectories is not proportional to the initial difference.

Lorenz defined a system of 3 hypothetical variables, $x,y,z$ that vary in time, $t$ in relation to one another according to the equations

$$\begin{align}
\frac{dx}{dt} & =  \sigma(x-y)  \\
\frac{dy}{dt} & =  x(\rho-z)-y  \\
\frac{dz}{dt} & = xy-\beta z
\end{align}$$

with parameters, $\sigma, \rho, \beta$, constant in time.

```python
# Lets write a function to evaluate the Lorenz 63 equations
def lorenz63( x, y, z, σ=10., ρ=28., β=8./3. ):
    """
    Function to evaluate the Lorenz 63 time derivative equation for a given current
    state and parameters.
    Arguments:
    x, y, z: x,y,z values.
    σ, ρ, β: static parameters α, ρ and β.
    """
    xt=σ*(y-x)
    yt=x*ρ-x*z-y
    zt=x*y-β*z
    return xt,yt,zt

```

```python
x,y,z=1,1,1;
dxdt,dydt,dzdt=lorenz63( x, y, z )
print(dxdt,dydt,dzdt)
```

```python
lorenz63?
```

For given initial conditions, we can simulate the time evolution of the Lorenz 63 system using the same sort
of timestepping we looked at in Lecture 1.
$$
\phi^{n+1}=\phi^{n}+\Delta t f(\phi^{n})
$$
this is not a very accurate numerical scheme (it is called an Euler forward scheme), so we will look at other schemes later. For illustrating programming concepts here it is usable.

In the Lorenz 63 case $\phi$ will be our array of time evolving values $x,y$ and $z$, and the function $f()$ can be our ``lorenz63`` Python function we just looked at.

```python
# Lets try this
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits import mplot3d

x0,y0,z0=0.,1.,1.05; Δt=0.01 # Initial conditions and paramteres (note we need a discrete timestep)
nsteps=10000;
x=np.zeros(nsteps+1);x[0]=x0
y=np.zeros(nsteps+1);y[0]=y0
z=np.zeros(nsteps+1);z[0]=z0
for i in range(nsteps):
        dxdt,dydt,dzdt=lorenz63( x[i], y[i], z[i] )
        x[i+1]=x[i]+dxdt*Δt
        y[i+1]=y[i]+dydt*Δt
        z[i+1]=z[i]+dzdt*Δt

ax=plt.axes(projection='3d')
ax.scatter3D(x,y,z,s=0.5);
```

```python
plt.plot(x);
```

```python
# We can go a bit further with functions by defining a "timestepping" function for the Euler forward scheme we are using
# and passing the time-derivative function as an argument to a generic stepper.
def euler_forward_stepper(f, u0,nt,dt, params={}):
    """
    Function euler_forward_stepper steps forward for n steps a set of discrete ODE equations
    using an Euler forward scheme.
    Arguments:
    f: Function that returns time-derivative at current value of the state, u.
       f is expected to have each component of u as a separate argument, followed
       by keyword arguments for named parameters.
    u0: Initial values for the components of u
    nsteps: number of steps to take
    dt: time-step
    params: Dictionary of parameter settings, set using keywords.
    """
    # Set up initial state
    nf=len(u0)
    u=np.zeros( (nt+1,nf) )
    dudt=np.zeros( nf )
    for i in range(nf):
        u[0,i]=u0[i]

    # Step forward
    for n in range(nt):
        dudt=f(*u[n,:],**params)
        u[n+1,:]=u[n,:]+np.array(dudt)*dt

    # return result
    return u
```

```python
nsteps=10000;
dt=0.01;
eps=1.e-4;
u0=euler_forward_stepper(lorenz63, np.array([0.,1.,1.05]), nsteps, dt);
u1=euler_forward_stepper(lorenz63, np.array([0.+eps,1.,1.05]), nsteps, dt);
```

```python
plt.plot(u0[:,0]);plt.plot(u1[:,0]);
```

##### This plot demonstrates why computer weather forecasts get less accurate with time. Can you think why the plot shows this!

```python

```