# Chapter 9

*Modeling and Simulation in Python*

Copyright 2021 Allen Downey

License: [Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International](https://creativecommons.org/licenses/by-nc-sa/4.0/)

```python
# check if the libraries we need are installed

try:
    import pint
except ImportError:
    !pip install pint

try:
    from modsim import *
except ImportError:
    !pip install modsimpy
```

In this chapter we express the models from previous chapters as
difference equations and differential equations, solve the equations,
and derive the functional forms of the solutions. We also discuss the
complementary roles of mathematical analysis and simulation.

## Recurrence relations

The population models in the previous chapter and this one are simple
enough that we didn't really need to run simulations. We could have
solved them mathematically. For example, we wrote the constant growth
model like this:

```
results[t+1] = results[t] + annual_growth
```

In mathematical notation, we would write the same model like this:

$$x_{n+1} = x_n + c$$

where $x_n$ is the population during year $n$,
$x_0$ is a given initial population, and $c$ is constant annual growth.
This way of representing the model is a **recurrence relation**; see
<http://modsimpy.com/recur>.

Sometimes it is possible to solve a recurrence relation by writing an
equation that computes $x_n$, for a given value of $n$, directly; that
is, without computing the intervening values from $x_1$ through
$x_{n-1}$.

In the case of constant growth we can see that $x_1 = x_0 + c$, and
$x_2 = x_1 + c$. Combining these, we get $x_2 = x_0 + 2c$, then
$x_3 = x_0 + 3c$, and it is not hard to conclude that in general

$$x_n = x_0 + nc$$

So if we want to know $x_{100}$ and we don't care
about the other values, we can compute it with one multiplication and
one addition.

We can also write the proportional model as a recurrence relation:

$$x_{n+1} = x_n + \alpha x_n$$

Or more conventionally as:

$$x_{n+1} = x_n (1 + \alpha)$$

Now we can see that

$x_1 = x_0 (1 + \alpha)$, and $x_2 = x_0 (1 + \alpha)^2$, and in general

$$x_n = x_0 (1 + \alpha)^n$$

This result is a **geometric progression**;
see <http://modsimpy.com/geom>. When $\alpha$ is positive, the factor
$1+\alpha$ is greater than 1, so the elements of the sequence grow
without bound.

Finally, we can write the quadratic model like this:

$$x_{n+1} = x_n + \alpha x_n + \beta x_n^2$$

or with the more
conventional parameterization like this:

$$x_{n+1} = x_n + r x_n (1 - x_n / K)$$

There is no analytic solution to
this equation, but we can approximate it with a differential equation
and solve that, which is what we'll do in the next section.

## Differential equations

Starting again with the constant growth model

$$x_{n+1} = x_n + c$$

If we define $\Delta x$ to be the change in $x$ from one time step to the next, we can write:

$$\Delta x = x_{n+1} - x_n = c$$

If we define
$\Delta t$ to be the time step, which is one year in the example, we can
write the rate of change per unit of time like this:

$$\frac{\Delta x}{\Delta t} = c$$

This model is **discrete**, which
means it is only defined at integer values of $n$ and not in between.
But in reality, people are born and die all the time, not once a year,
so a **continuous** model might be more realistic.

We can make this model continuous by writing the rate of change in the
form of a derivative:

$$\frac{dx}{dt} = c$$

This way of representing the model is a **differential equation**; see <http://modsimpy.com/diffeq>.

We can solve this differential equation if we multiply both sides by
$dt$:

$$dx = c dt$$

And then integrate both sides:

$$x(t) = c t + x_0$$

Similarly, we can write the proportional growth model like this:

$$\frac{\Delta x}{\Delta t} = \alpha x$$

And as a differential equation
like this:

$$\frac{dx}{dt} = \alpha x$$

If we multiply both sides by
$dt$ and divide by $x$, we get

$$\frac{1}{x}~dx = \alpha~dt$$

Now we
integrate both sides, yielding:

$$\ln x = \alpha t + K$$

where $\ln$ is the natural logarithm and $K$ is the constant of integration.

Exponentiating both sides, we have

$$\exp(\ln(x)) = \exp(\alpha t + K)$$

The exponential function can be written $\exp(x)$ or $e^x$. In this book I use the first form because it resembles the Python code.

We can rewrite the previous equation as

$$x = \exp(\alpha t) \exp(K)$$

Since $K$ is an arbitrary constant,
$\exp(K)$ is also an arbitrary constant, so we can write

$$x = C \exp(\alpha t)$$

where $C = \exp(K)$. There are many solutions
to this differential equation, with different values of $C$. The
particular solution we want is the one that has the value $x_0$ when
$t=0$.

When $t=0$, $x(t) = C$, so $C = x_0$ and the solution we want is
$$x(t) = x_0 \exp(\alpha t)$$ If you would like to see this derivation
done more carefully, you might like this video:
<http://modsimpy.com/khan1>.

## Analysis and simulation

Once you have designed a model, there are generally two ways to proceed: simulation and analysis. Simulation often comes in the form of a computer program that models changes in a system over time, like births and deaths, or bikes moving from place to place. Analysis often comes in the form of algebra; that is, symbolic manipulation using mathematical notation.

Analysis and simulation have different capabilities and limitations.
Simulation is generally more versatile; it is easy to add and remove
parts of a program and test many versions of a model, as we have done in the previous examples.

But there are several things we can do with analysis that are harder or impossible with simulations:

-   With analysis we can sometimes compute, exactly and efficiently, a
    value that we could only approximate, less efficiently, with
    simulation. For example, in the quadratic model we plotted growth rate versus population and saw net crossed through zero when the population is
    near 14 billion.  We could estimate the crossing point using a
    numerical search algorithm (more about that later). But with the
    analysis in Section xxx, we get the general result that
    $K=-\alpha/\beta$.

-   Analysis often provides "computational shortcuts", that is, the
    ability to jump forward in time to compute the state of a system
    many time steps in the future without computing the intervening
    states.

-   We can use analysis to state and prove generalizations about models;
    for example, we might prove that certain results will always or
    never occur. With simulations, we can show examples and sometimes
    find counterexamples, but it is hard to write proofs.

-   Analysis can provide insight into models and the systems they
    describe; for example, sometimes we can identify regimes of
    qualitatively different behavior and key parameters that control
    those behaviors.

When people see what analysis can do, they sometimes get drunk with
power, and imagine that it gives them a special ability to see past the veil of the material world and discern the laws of mathematics that govern the universe. When they analyze a model of a physical system, they talk about "the math behind it" as if our world is the mere shadow of a world of ideal mathematical entities (I am not making this up; see <http://modsimpy.com/plato>.).

This is, of course, nonsense. Mathematical notation is a language
designed by humans for a purpose, specifically to facilitate symbolic
manipulations like algebra. Similarly, programming languages are
designed for a purpose, specifically to represent computational ideas
and run programs.

Each of these languages is good for the purposes it was designed for and less good for other purposes. But they are often complementary, and one of the goals of this book is to show how they can be used together.

## Analysis with WolframAlpha

Until recently, most analysis was done by rubbing graphite on wood
pulp, a process that is laborious and error-prone. A useful
alternative is symbolic computation. If you have used services like
WolframAlpha, you have used symbolic computation.

For example, if you go to <https://www.wolframalpha.com/> and type

```
df(t) / dt = alpha f(t)
```

WolframAlpha infers that `f(t)` is a function of `t` and `alpha` is a
parameter; it classifies the query as a "first-order linear ordinary
differential equation\", and reports the general solution:

$$f(t) = c_1 \exp(\alpha t)$$

If you add a second equation to specify
the initial condition:

```
df(t) / dt = alpha f(t),  f(0) = p_0
```

WolframAlpha reports the particular solution:

$$f(t) = p_0 \exp(\alpha t)$$

WolframAlpha is based on Mathematica, a powerful programming language
designed specifically for symbolic computation.

## Analysis with SymPy

Python has a library called SymPy that provides symbolic computation
tools similar to Mathematica. They are not as easy to use as
WolframAlpha, but they have some other advantages.

Before we can use SymPy, we have to import it:

SymPy defines a `Symbol` object that represents symbolic variable names,
functions, and other mathematical entities.

The `symbols` function takes a string and returns `Symbol` objects. So
if we run this assignment:

```python
from sympy import symbols

t = symbols('t')
```

Python understands that `t` is a symbol, not a numerical value. If we
now run

```python
expr = t + 1
expr
```

$\displaystyle t + 1$

Python doesn't try to perform numerical addition; rather, it creates a
new `Symbol` that represents the sum of `t` and `1`. We can evaluate the
sum using `subs`, which substitutes a value for a symbol. This example
substitutes 2 for `t`:

```python
expr.subs(t, 2)
```

$\displaystyle 3$

Functions in SymPy are represented by a special kind of `Symbol`:

```python
from sympy import Function

f = Function('f')
f
```

    f

Now if we write `f(t)`, we get an object that represents the evaluation of a function, $f$, at a value, $t$.

```python
f(t)
```

$\displaystyle f{\left(t \right)}$

But again SymPy doesn't actually
try to evaluate it.

## Differential equations in SymPy

SymPy provides a function, `diff`, that can differentiate a function. We can apply it to `f(t)` like this:

```python
from sympy import diff

dfdt = diff(f(t), t)
dfdt
```

$\displaystyle \frac{d}{d t} f{\left(t \right)}$

The result is a `Symbol` that represents the derivative of `f` with
respect to `t`. But again, SymPy doesn't try to compute the derivative
yet.

To represent a differential equation, we use `Eq`:

```python
from sympy import Eq

alpha = symbols('alpha')
eq1 = Eq(dfdt, alpha*f(t))
eq1
```

$\displaystyle \frac{d}{d t} f{\left(t \right)} = \alpha f{\left(t \right)}$

The result is an object that represents an equation.  Now
we can use `dsolve` to solve this differential equation:

```python
from sympy import dsolve

solution_eq = dsolve(eq1)
solution_eq
```

$\displaystyle f{\left(t \right)} = C_{1} e^{\alpha t}$

The result is the **general
solution**, which still contains an unspecified constant, $C_1$. To get the **particular solution** where $f(0) = p_0$, we substitute `p_0` for `C1`. First, we have to create two more symbols:

```python
C1, p_0 = symbols('C1 p_0')
```

Now we can perform the substitution:

```python
particular = solution_eq.subs(C1, p_0)
particular
```

$\displaystyle f{\left(t \right)} = p_{0} e^{\alpha t}$

The result is called the **exponential growth curve**; see
<http://modsimpy.com/expo>.

## Solving the quadratic growth model

We'll use the (r, K) parameterization, so we'll need two more symbols:

```python
r, K = symbols('r K')
```

Now we can write the differential equation.

```python
eq2 = Eq(diff(f(t), t), r * f(t) * (1 - f(t)/K))
eq2
```

$\displaystyle \frac{d}{d t} f{\left(t \right)} = r \left(1 - \frac{f{\left(t \right)}}{K}\right) f{\left(t \right)}$

And solve it.

```python
solution_eq = dsolve(eq2)
solution_eq
```

$\displaystyle f{\left(t \right)} = \frac{K e^{C_{1} K + r t}}{e^{C_{1} K + r t} - 1}$

The result, `solution_eq`, contains `rhs`, which is the right-hand side of the solution.

```python
general = solution_eq.rhs
general
```

$\displaystyle \frac{K e^{C_{1} K + r t}}{e^{C_{1} K + r t} - 1}$

We can evaluate the right-hand side at $t=0$

```python
at_0 = general.subs(t, 0)
at_0
```

$\displaystyle \frac{K e^{C_{1} K}}{e^{C_{1} K} - 1}$

Now we want to find the value of `C1` that makes `f(0) = p_0`.

So we'll create the equation `at_0 = p_0` and solve for `C1`.  Because this is just an algebraic identity, not a differential equation, we use `solve`, not `dsolve`.

```python
from sympy import solve

solutions = solve(Eq(at_0, p_0), C1)
```

The result from `solve` is a list of solutions.  In this case, [we have reason to expect only one solution](https://en.wikipedia.org/wiki/Picard%E2%80%93Lindel%C3%B6f_theorem), but we still get a list, so we have to use the bracket operator, `[0]`, to select the first one.

```python
type(solutions), len(solutions)
```

    (list, 1)

```python
value_of_C1 = solutions[0]
value_of_C1
```

$\displaystyle \frac{\log{\left(- \frac{p_{0}}{K - p_{0}} \right)}}{K}$

Now in the general solution, we want to replace `C1` with the value of `C1` we just figured out.

```python
particular = general.subs(C1, value_of_C1)
particular
```

$\displaystyle - \frac{K p_{0} e^{r t}}{\left(K - p_{0}\right) \left(- \frac{p_{0} e^{r t}}{K - p_{0}} - 1\right)}$

The result is complicated, but SymPy provides a function that tries to simplify it.

```python
particular.simplify()
```

$\displaystyle \frac{K p_{0} e^{r t}}{K + p_{0} e^{r t} - p_{0}}$

This function is called the **logistic growth curve**; see
<http://modsimpy.com/logistic>. In the context of growth models, the
logistic function is often written like this:

$$f(t) = \frac{K}{1 + A \exp(-rt)}$$

where $A = (K - p_0) / p_0$.

We can use SymPy to confirm that these two forms are equivalent.  First we represent the alternative version of the logistic function:

```python
A = (K - p_0) / p_0
A
```

$\displaystyle \frac{K - p_{0}}{p_{0}}$

```python
from sympy import exp

logistic = K / (1 + A * exp(-r*t))
logistic
```

$\displaystyle \frac{K}{1 + \frac{\left(K - p_{0}\right) e^{- r t}}{p_{0}}}$

To see whether two expressions are equivalent, we can check whether their difference simplifies to 0.

```python
(particular - logistic).simplify()
```

$\displaystyle 0$

This test only works one way: if SymPy says the difference reduces to 0, the expressions are definitely equivalent (and not just numerically close).

But if SymPy can't find a way to simplify the result to 0, that doesn't necessarily mean there isn't one.  Testing whether two expressions are equivalent is a surprisingly hard problem; in fact, there is no algorithm that can solve it in general.

If you would like to see this differential equation solved by hand, you might like this video: <http://modsimpy.com/khan2>

## Summary

In this chapter we wrote the growth models from the previous chapters in terms of difference and differential equations.

What I called the constant growth model is more commonly called **linear growth** because the solution is a line.  If we model time as continuous, the solution is

$$f(t) = p_0 + \alpha t$$

where $\alpha$ is the growth rate.

Similarly, the proportional growth model is usually called **exponential growth** because the solution is exponential:

$$f(t) = p_0 \exp{\alpha t}$$

Finally, the quadratic growth model is called **logistic growth** because the solution is the logistic function:

$$f(t) = \frac{K}{1 + A \exp(-rt)}$$

where $A = (K - p_0) / p_0$.

I avoided these terms until now because they are based on results we had not derived yet.

## Exercises

**Exercise:** Solve the quadratic growth equation using the alternative parameterization

$\frac{df(t)}{dt} = \alpha f(t) + \beta f^2(t) $

```python
# Solution

alpha, beta = symbols('alpha beta')
```

```python
# Solution

eq3 = Eq(diff(f(t), t), alpha*f(t) + beta*f(t)**2)
eq3
```

$\displaystyle \frac{d}{d t} f{\left(t \right)} = \alpha f{\left(t \right)} + \beta f^{2}{\left(t \right)}$

```python
# Solution

solution_eq = dsolve(eq3)
solution_eq
```

$\displaystyle f{\left(t \right)} = \frac{\alpha e^{\alpha \left(C_{1} + t\right)}}{\beta \left(1 - e^{\alpha \left(C_{1} + t\right)}\right)}$

```python
# Solution

general = solution_eq.rhs
general
```

$\displaystyle \frac{\alpha e^{\alpha \left(C_{1} + t\right)}}{\beta \left(1 - e^{\alpha \left(C_{1} + t\right)}\right)}$

```python
# Solution

at_0 = general.subs(t, 0)
```

```python
# Solution

solutions = solve(Eq(at_0, p_0), C1)
value_of_C1 = solutions[0]
value_of_C1
```

$\displaystyle \frac{\log{\left(\frac{\beta p_{0}}{\alpha + \beta p_{0}} \right)}}{\alpha}$

```python
# Solution

particular = general.subs(C1, value_of_C1)
particular.simplify()
```

$\displaystyle \frac{\alpha p_{0} e^{\alpha t}}{\alpha - \beta p_{0} e^{\alpha t} + \beta p_{0}}$

**Exercise:**  Use [WolframAlpha](https://www.wolframalpha.com/) to solve the quadratic growth model, using either or both forms of parameterization:

    df(t) / dt = alpha f(t) + beta f(t)^2

or

    df(t) / dt = r f(t) (1 - f(t)/K)

Find the general solution and also the particular solution where `f(0) = p_0`.


# Binary classification from 2 features

Classification using "raw" python or libraries to implement logistic regression.

The classification is on a single boundary defined by a continuous function and added white noise.

Linear regression is about finding the line, plan or hyper plan that has the best fit with the model. The predicted samples is a value.

Binary classification is about splitting the space into two halves with the best line/plan/hyper plan. The predicted sample is a probability that is transformed into a boolean using the threshold of 0.5.

```python
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.colors as pltcolors
from sklearn import metrics
from sklearn.linear_model import LogisticRegression as SkLinReg
import seaborn as sns
```

## Model

Quadratic function a boundary between positive and negative values

Adding some unknown as a Gaussian noise

The values of X are uniformly distributed and independent

```python
# Two features, Gaussian noise
nFeatures = 2
def generateBatch(N):
    #
    xMin = 0
    xMax = 1
    b = 0.1
    std = 0.1
    #
    x = np.random.uniform(xMin, xMax, (N, nFeatures))
    # 4th degree relation to shape the boundary
    boundary = 2*(x[:,0]**4 + (x[:,0]-0.3)**3 + b)
    # Adding some gaussian noise
    labels = boundary + np.random.normal(0, std, N) > x[:,1]
    return (x, labels)
```

### Training data

```python
N = 2000
# x has 1 dim in R, label has 1 dim in B
xTrain, labelTrain = generateBatch(N)

colors = ['blue','red']

fig = plt.figure(figsize=(15,4))
plt.subplot(1,3,1)
plt.scatter(xTrain[:,0], xTrain[:,1], c=labelTrain, cmap=pltcolors.ListedColormap(colors), marker=',', alpha=0.1)
plt.xlabel('$x_0$')
plt.ylabel('$x_1$')
plt.title('Generated train data')
plt.grid()
cb = plt.colorbar()
loc = np.arange(0,1,1/float(len(colors)))
cb.set_ticks(loc)
cb.set_ticklabels([0,1])
plt.subplot(1,3,2)
plt.scatter(xTrain[:,0], labelTrain, marker=',', alpha=0.01)
plt.xlabel('$x_0$')
plt.ylabel('label')
plt.grid()
plt.subplot(1,3,3)
plt.scatter(xTrain[:,1], labelTrain, marker=',', alpha=0.01)
plt.xlabel('$x_1$')
plt.ylabel('label')
plt.grid()
```

```python
count, bins, ignored = plt.hist(labelTrain*1.0, 10, density=True, alpha=0.5)
p = np.mean(labelTrain)
print('Bernouilli parameter of the distribution:', p)
```

### Test data for verification of the model

```python
xTest, labelTest = generateBatch(N)
testColors = ['navy', 'orangered']
```

### Logistic and log of Logistic functions

The Logistic function is used to transform a linear separation of the plan into a probability in the range \[0,1\]

\begin{align}
 logistic(x) = \frac{1}{1+e^{-x}}
\end{align}

```python
def logistic(X):
    return (1+(np.exp(-(X))))**-1
```

```python
xx = np.linspace(-10, 10)
xlogistic = logistic(xx)
plt.figure(figsize=(10,5))
plt.subplot(1, 2, 1)
plt.plot(xx, xlogistic)
plt.grid()
plt.subplot(1, 2, 2)
plt.plot(xx, np.log(xlogistic))
plt.grid()
```

# Logistic regression

$$y \in \left\{ 0, 1 \right\}$$

$$\begin{align}
P (Y=1 \mid x) & = \frac{1}{1+e^{-f(x)}} \\
P (Y=0 \mid x) & = 1 - \frac{1}{1+e^{-f_\theta(x)}} \\
f_\theta(x) & = b + w x
\end{align}$$

Let's define $\theta=\{b, w\}$

We are looking for the value of $\theta$ that maximizes the likelyhood:
$$\begin{align}
\hat{\theta} & = \underset{\theta \in \mathbb{R}}{argmax} \prod_{i=0}^N{P(y_i \mid  x_i, \theta)} \\
& = \underset{\theta \in \mathbb{R}}{argmax} \sum_{i=0}^N{log \bigl(p(y_i \mid x_i, \theta)\bigr)}  \\
& = \underset{\theta \in \mathbb{R}}{argmax} \sum_{i=0}^N{log \biggl(\bigl(\frac{1}{1+e^{-f_\theta(x_i)}}\bigr)^{y_i}\bigl(1-\frac{1}{1+e^{-f_\theta(x_i)}}\bigr)^{1-y_i}\biggr)}  \\
& = \underset{\theta \in \mathbb{R}}{argmax} \sum_{i=0}^N{log \biggl(y_i * \bigl(\frac{1}{1+e^{-f_\theta(x_i)}}\bigr) + \bigl(1-y_i\bigr) * \bigl(1-\frac{1}{1+e^{-f_\theta(x_i)}}\bigr) \biggr)}  \\
\end{align}$$

Using the fact that $y_i$ is either 0 or 1

If using an iterative algorithm, the cost function to minimize is taken as the negative maximum log likelyhood:
$$
J_\theta = - \sum_{i=0}^N{log \left(y_i * \left(\frac{1}{1 + e^{-f_\theta(x_i)}}\right) + \left(1 -y_i\right) * \left(1 - \frac{1}{1+e^{-f_\theta(x_i)}}\right) \right)}
$$

Reference:
- https://en.wikipedia.org/wiki/Logistic_regression

## Helpers

```python
def plotTestResult(xTest, labelTest, yEst, labelEst):
    fig, axes = plt.subplots(1, 3, figsize=(16, 5))
    im0 = axes[0].scatter(xTest[:,0], xTest[:,1], c=labelEst, cmap=pltcolors.ListedColormap(testColors), marker='x', alpha=0.2);
    axes[0].set_xlabel('$x_0$')
    axes[0].set_ylabel('$x_1$')
    axes[0].grid()
    axes[0].set_title('Estimated')
    cb = plt.colorbar(im0, ax=axes[0])
    loc = np.arange(0, 1, 1./len(testColors))
    cb.set_ticks(loc)
    cb.set_ticklabels([0,1]);

    axes[1].hist(yEst, 10, density=True, alpha=0.5)
    axes[1].set_title('Bernouilli parameter =' + str(np.mean(labelEst)))

    im2 = axes[2].scatter(xTest[:,0], xTest[:,1], c=labelTest, cmap=pltcolors.ListedColormap(colors), marker='x', alpha=0.2);
    axes[2].set_xlabel('$x_0$')
    axes[2].set_ylabel('$x_1$')
    axes[2].grid()
    axes[2].set_title('Generator')
    cb = plt.colorbar(im2, ax=axes[2])
    loc = np.arange(0, 1, 1./len(colors))
    cb.set_ticks(loc)
    cb.set_ticklabels([0,1])
    return axes

# ROC curve, obtained varying the decision threshold to predict a True given a probability
def plotRocCurve(labelTest, yEstTrue):
    logit_roc_auc = metrics.roc_auc_score(labelTest, yEstTrue)
    fpr, tpr, thresholds = metrics.roc_curve(labelTest, yEstTrue)
    plt.plot(fpr, tpr, label='Logistic Regression (area = %0.2f)' % logit_roc_auc)
    plt.plot([0, 1], [0, 1],'r--')
    plt.xlim([0.0, 1.0])
    plt.ylim([0.0, 1.05])
    plt.xlabel('False Positive Rate')
    plt.ylabel('True Positive Rate')
    plt.title('Receiver operating characteristic')
    plt.legend(loc="lower right");

def plotHeatMap(X, classes, title=None, fmt='.2g', ax=None, xlabel=None, ylabel=None):
    """ Fix heatmap plot from Seaborn with pyplot 3.1.0, 3.1.1
        https://stackoverflow.com/questions/56942670/matplotlib-seaborn-first-and-last-row-cut-in-half-of-heatmap-plot
    """
    ax = sns.heatmap(X, xticklabels=classes, yticklabels=classes, annot=True, fmt=fmt, cmap=plt.cm.Blues, ax=ax) #notation: "annot" not "annote"
    bottom, top = ax.get_ylim()
    ax.set_ylim(bottom + 0.5, top - 0.5)
    if title:
        ax.set_title(title)
    if xlabel:
        ax.set_xlabel(xlabel)
    if ylabel:
        ax.set_ylabel(ylabel)

def plotConfusionMatrix(yTrue, yEst, classes, title=None, fmt='.2g', ax=None):
    plotHeatMap(metrics.confusion_matrix(yTrue, yEst), classes, title, fmt, ax, \
                xlabel='Estimations', ylabel='True values');
```

## Homemade fitting of $b$ and then $w$

Suboptimal fitting:
- Taking some assumption on $w$ to fit $b$ as $\hat{b}$
- and then fitting $w$ with the $\hat{b}$ estimate using a simple grid search

```python
bs = np.linspace(-20, 20)
ws = np.array([0,0]);
px = np.zeros(len(bs))
for i in range(len(bs)):
    fx = logistic(bs[i] + np.matmul(xTrain, ws))
    px[i] = 1/N * np.sum(np.log(labelTrain * fx + (1 - labelTrain)*(1 - fx)))
plt.plot(bs, px);
plt.xlabel('$b$')
plt.ylabel('$l(b, X)$')
plt.grid()
bHat = bs[np.argmax(px)]
print('Estimated b =', bHat)
```

### Fitting $w$ with a simple grid search

```python
w0 = np.linspace(-20, 20)
w1 = np.linspace(-20, 20)
px = np.zeros((len(w0), len(w1)))
for i in range(len(w0)):
    for j in range(len(w1)):
        fx = logistic(bHat + np.matmul(xTrain, np.array([w0[i], w1[j]])))
        px[i, j] = 1/N * np.sum(np.log(np.multiply(labelTrain, fx) + np.multiply((1-labelTrain),(1-fx))))
m = np.argmax(px)
wHat = np.array([w0[m//len(w0)], w1[m%len(w0)]])
print('Estimated w =', wHat)
```

    Estimated w = [ 11.83673469 -10.20408163]

```python
plt.contourf(w0, w1, px, levels=128, cmap="RdBu_r");
```

## Testing of model 0 (homemade)

```python
yEst0 = logistic(bHat + np.matmul(xTest, wHat))
labelEst0 = yEst0 > 0.5
```

Classes centroids (center of gravity)

```python
xBarEst = np.array([np.mean(xTest[labelEst0 == c], axis = 0) for c in [0, 1]])
xBarGen = np.array([np.mean(xTest[labelTest == c], axis = 0) for c in [0, 1]])
```

Classes separation frontiere in the 2D plan

```python
xx0 = np.linspace(0, 1)
xx1 = -(bHat + wHat[0] * xx0) / wHat[1]
```

```python
axes = plotTestResult(xTest, labelTest, yEst0, labelEst0)
axes[0].plot(xBarEst[:,0], xBarEst[:,1], marker='o', label='Classes centroids')
axes[0].plot(xx0, xx1, color='g', label='Frontiere');
axes[0].legend(loc='upper left')
axes[2].plot(xBarGen[:,0], xBarGen[:,1], marker='o', label='Classes centroids')
axes[2].legend(loc='upper left');
```

The separation of the space by a straight line is a mere approximation of the polynomial function of the generator. However, performance is not that bad as shown below.

```python
accuracy0 = np.sum(labelTest == labelEst0)/N
print('Accuracy =', accuracy0)
```

    Accuracy = 0.8805

### Precision
$P(y = 1 \mid \hat{y} = 1)$

```python
print('Precision =', np.sum(labelTest[labelEst0 == 1])/np.sum(labelEst0))
```

    Precision = 0.8620689655172413

### Recall
$P(\hat{y} = 1 \mid y = 1)$

```python
print('Recall =', np.sum(labelTest[labelEst0 == 1])/np.sum(labelTest))
```

    Recall = 0.9104330708661418

### Confusion matrix

```python
plotConfusionMatrix(labelTest, labelEst0, np.array(['Blue', 'Red']));
```

```python
print(metrics.classification_report(labelTest, labelEst0))
```

                  precision    recall  f1-score   support

           False       0.90      0.85      0.87       984
            True       0.86      0.91      0.89      1016

        accuracy                           0.88      2000
       macro avg       0.88      0.88      0.88      2000
    weighted avg       0.88      0.88      0.88      2000

### ROC Curve

```python
plotRocCurve(labelTest, yEst0)
```

# Using SciKit Learn's linear regression

References:
- SciKit documentation
- https://www.geeksforgeeks.org/ml-logistic-regression-using-python/

```python
model1 = SkLinReg(solver='lbfgs')
model1.fit(xTrain, labelTrain)
print('Y = logistic({1} + X * {0})'.format(model1.coef_, model1.intercept_[0]))
```

    Y = logistic(-1.1819649092679545 + X * [[ 9.15042414 -6.74323194]])

```python
labelEst1 = model1.predict(xTest)
print('Mean accuracy =', model1.score(xTest, labelTest))
```

    Mean accuracy = 0.8935

```python
plt.hist(labelEst1*1.0, 10, density=True, alpha=0.5)
plt.title('Bernouilli parameter =' + str(np.mean(labelEst1)));
```

### Confusion matrix (plot)

```python
plotConfusionMatrix(labelTest, labelEst1, np.array(['Blue', 'Red']));
```

### Classification report

```python
print(metrics.classification_report(labelTest, labelEst1))
```

                  precision    recall  f1-score   support

           False       0.89      0.89      0.89       984
            True       0.89      0.90      0.90      1016

        accuracy                           0.89      2000
       macro avg       0.89      0.89      0.89      2000
    weighted avg       0.89      0.89      0.89      2000

References :
- https://towardsdatascience.com/building-a-logistic-regression-in-python-step-by-step-becd4d56c9c8
- https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html#sklearn.linear_model.LogisticRegression.get_params

### ROC curve

```python
yEst1True = model1.predict_proba(xTest)[:,1]
plotRocCurve(labelTest, yEst1True)
```

# Using SciKitLearn's gradient descent classifier

References:
- https://scikit-learn.org/stable/modules/sgd.html#classification

```python
from sklearn.linear_model import SGDClassifier as skSGDClassifier

model2 = skSGDClassifier(alpha=0.0001, average=False, class_weight=None,
           early_stopping=True, epsilon=0.1, eta0=0.0, fit_intercept=True,
           l1_ratio=0.15, learning_rate='optimal', loss='hinge', max_iter=1000,
           n_iter_no_change=5, n_jobs=None, penalty='l2', power_t=0.5,
           random_state=None, shuffle=True, tol=0.001,
           validation_fraction=0.1, verbose=0, warm_start=False)
```

Notes:
- Regularizer is called 'penalty' and parameterized by 'alpha' (and 'l1_ratio')
- Early stopping is available and parameterized by 'early_stopping', 'max_iter', 'tol' and 'n_iter_no_change'
- Shuffling between epochs enabled by 'shuffle'

```python
model2.fit(xTrain, labelTrain);
print('Y = logistic({1} + X * {0})'.format(model2.coef_, model2.intercept_[0]))
```

    Y = logistic(-1.399356875305446 + X * [[ 12.52565336 -10.43100322]])

```python
labelEst2 = model2.predict(xTest)
print('Mean accuracy =', model2.score(xTest, labelTest))
```

    Mean accuracy = 0.878

```python
plt.hist(labelEst2*1.0, 10, density=True, alpha=0.5)
plt.title('Bernouilli parameter =' + str(np.mean(labelEst2)));
```

### Confusion matrix (plot)

```python
plotConfusionMatrix(labelTest, labelEst2, np.array(['Blue', 'Red']));
```

### Classification report

```python
print(metrics.classification_report(labelTest, labelEst2))
```

                  precision    recall  f1-score   support

           False       0.86      0.90      0.88       984
            True       0.90      0.85      0.88      1016

        accuracy                           0.88      2000
       macro avg       0.88      0.88      0.88      2000
    weighted avg       0.88      0.88      0.88      2000

# Where to go from here ?

- Other linear implementations and simple neural nets using TensorFlow ([HTML](ClassificationContinuous2Features-TensorFlow.html) / [Jupyter](ClassificationContinuous2Features-TensorFlow.ipynb)), using Keras ([HTML](ClassificationContinuous2Features-Keras.html) / [Jupyter](ClassificationContinuous2Features-Keras.ipynb)) or the K Nearest Neighbors classifier ([HTML](ClassificationContinuous2Features-KNN.html) / [Jupyter](ClassificationContinuous2Features-KNN.ipynb))

- Non linear problem solving with Support Vector Machine (SVM) ([HTML](ClassificationSVM.html) / [Jupyter](ClassificationSVM.ipynb))

- More complex multi-class models on the Czech and Norways flags using Keras ([HTML](ClassificationMulti2Features-Keras.html) / [Jupter](ClassificationMulti2Features-Keras.ipynb)), showing one of the main motivations to neural networks.

- Compare with the two feature linear regression using simple algorithms ([HTML](../linear/LinearRegressionBivariate.html) / [Jupyter](LinearRegressionBivariate.ipynb])), or using Keras ([HTML](LinearRegressionBivariate-Keras.html) / [Jupyter](LinearRegressionUnivariate-Keras.ipynb))


# Numerical Computing in Python

(c) 2019 [Steve Phelps](mailto:sphelps@sphelps.net)

## Overview

- Floating-point representation
- Arrays and Matrices with `numpy`
- Basic plotting with `matplotlib`
- Pseudo-random variates with `numpy.random`

## Representing continuous values

- Digital computers are inherently *discrete*.

- Real numbers $x \in R$ cannot always be represented exactly in a digital computer.

- They are stored in a format called *floating-point*.

- [IEEE Standard 754](http://steve.hollasch.net/cgindex/coding/ieeefloat.html) specifies a universal format across different implementations.
    - As always there are deviations from the standard.

- There are two standard sizes of floating-point numbers: 32-bit and 64-bit.

- 64-bit numbers are called *double precision*, are sometimes called *double* values.

- IEEE floating-point calculations are performed in *hardware* on modern computers.

- How can we represent aribitrary real values using only 32 bits?

## Fixed-point verses floating-point

- One way we could discretise continuous values is to represent them as two integers $x$ and $y$.

- The final value is obtained by e.g. $r = x + y \times 10^{-5}$.

- So the number $500.4421$ would be represented as the tuple $x = 500$, $y = 44210$.

- The exponent $5$ is fixed for all computations.

- This number represents the *precision* with which we can represent real values.

- It corresponds to the where we place we place the decimal *point*.

- This scheme is called fixed precision.

- It is useful in certain circumstances, but suffers from many problems, in particular it can only represent a very limited *range* of values.

- In practice, we use variable precision, also known as *floating* point.

## Scientific Notation

- Humans also use a form of floating-point representation.

- In [Scientific notation](https://en.wikipedia.org/wiki/Scientific_notation), all numbers are written
in the form $m \times 10^n$.

- When represented in ASCII, we abbreviate this `<m>e<n>`, for example `6.72e+11` = $6.72 \times 10^{11}$.

- The integer $m$ is called the *significand* or *mantissa*.

- The integer $n$ is called the *exponent*.

- The integer $10$ is the *base*.

## Scientific Notation in Python

- Python uses Scientific notation when it displays floating-point numbers:

```python
print(672000000000000000.0)
```

    6.72e+17

- Note that internally, the value is not *represented* exactly like this.

- Scientific notation is a convention for writing or rendering numbers, *not* representing them digitally.

## Floating-point representation

- Floating point numbers use a base of $2$ instead of $10$.

- Additionally, the mantissa are exponent are stored in binary.

- Therefore we represent floating-point numbers as $m \times 2 ^ e$.

- The integers $m$ (mantissa) and $e$ (exponent) are stored in binary.

- The mantissa uses [two's complement](https://en.wikipedia.org/wiki/Two's_complement) to represent positive and negative numbers.
    - One bit is reserved as the sign-bit: 1 for negative, 0 for positive values.

- The mantissa is normalised, so we assume that it starts with the digit $1$ (which is not stored).

## Bias

- We also need to represent signed *exponents*.

- The exponent does *not* use two's complement.

- Instead a *bias* value is subtracted from the stored exponent ($s$) to obtain the final value ($e$).

- Double-precision values use a bias of $b = 1023$, and single-precision uses a bias value of $b = 127$.

- The actual exponent is given by $e = s - b$ where $s$ is the stored exponent.

- The stored exponent values $s=0$ and $s=1024$ are reserved for special values- discussed later.

- The stored exponent $s$ is represented in binary *without using a sign bit*.

## Double and single precision formats

The number of bits allocated to represent each integer component of a float is given below:

| **Format**   | **Sign** |   **Exponent** | **Mantissa** |  **Total** |
| ------------ | -------- | -------------- | ------------ | ---------- |
| **single**   | 1      | 8        | 23        | 32
| **double**   | 1      | 11       | 52        | 64

- By default, Python uses 64-bit precision.

- We can specify alternative precision by using the [numpy numeric data types](https://docs.scipy.org/doc/numpy/user/basics.types.html).

## Loss of precision

- We cannot represent every value in floating-point.

- Consider single-precision (32-bit).

- Let's try to represent $4,039,944,879$.

## Loss of precision

- As a binary integer we write $4,039,944,879$ as:

`11110000 11001100 10101010 10101111`

- This already takes up 32-bits.

- The mantissa only allows us to store 24-bit integers.

- So we have to *round*.  We store it as:

`+1.1110000 11001100 10101011e+31`

- Which gives us

`+11110000 11001100 10101011 0000000`

$= 4,039,944,960$

## Ranges of floating-point values

In single precision arithmetic, we *cannot* represent the following values:

- Negative numbers less than $-(2-2^{-23}) \times 2^{127}$

- Negative numbers greater than $-2^{-149}$

- Positive numbers less than $2^{-149}$

- Positive numbers greater than $(2-2^{-23}) \times 2^{127}$

Attempting to represent these numbers results in *overflow* or *underflow*.

## Effective floating-point range

| Format | Binary | Decimal |
| ------ | ------ | ------- |
| single | $\pm (2-2^{-23}) \times 2^{127}$ | $\approx \pm 10^{38.53}$ |
| double | $\pm (2-2^{-52}) \times 2^{1023}$ | $\approx \pm 10^{308.25}$ |

```python
import sys
sys.float_info.max
```

    1.7976931348623157e+308

```python
sys.float_info.min
```

    2.2250738585072014e-308

```python
sys.float_info
```

    sys.float_info(max=1.7976931348623157e+308, max_exp=1024, max_10_exp=308, min=2.2250738585072014e-308, min_exp=-1021, min_10_exp=-307, dig=15, mant_dig=53, epsilon=2.220446049250313e-16, radix=2, rounds=1)

## Range versus precision

- With a fixed number of bits, we have to choose between:
    - maximising the range of values (minimum to maximum) we can represent,
    - maximising the precision with-which we can represent each indivdual value.
- These are conflicting objectives:
    - we can increase range, but only by losing precision,
    - we can increase precision, but only by decreasing range.

- Floating-point addresses this dilemma by allowing the precision to *vary* ("float") according to the magnitude of the number we are trying to represent.

## Floating-point density

- Floating-point numbers are unevenly-spaced over the line of real-numbers.

- The precision *decreases* as we increase the magnitude.

## Representing Zero

- Zero cannot be represented straightforwardly because we assume that all mantissa values start with the digit 1.

- Zero is stored as a special-case, by setting mantissa *and* exponent both to zero.

- The sign-bit can either be set or unset, so there are distinct positive and negative representations of zero.

## Zero in Python

```python
x = +0.0
x

```

    0.0

```python
y = -0.0
y

```

    -0.0

- However, these are considered equal:

```python
x == y
```

    True

## Infinity

- Positive overflow results in a special value of infinity (in Python `inf`).

- This is stored with an exponent consiting of all 1s, and a mantissa of all 0s.

- The sign-bit allows us to differentiate between negative and positive overflow: $-\infty$ and $+\infty$.

- This allows us to carry on calculating past an overflow event.

## Infinity in Python

```python
x = 1e300 * 1e100
x

```

    inf

```python
x = x + 1
x
```

    inf

## Negative infinity in Python

```python
x > 0
```

    True

```python
y = -x
y
```

    -inf

```python
y < x
```

    True

## Not A Number (NaN)

- Some mathematical operations on real numbers do not map onto real numbers.

- These results are represented using the special value to `NaN` which represents "not a (real) number".

- `NaN` is represented by an exponent of all 1s, and a non-zero mantissa.

## `NaN` in Python

```python
from numpy import sqrt, inf, isnan, nan
x = sqrt(-1)
x
```

    /home/sphelps/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:2: RuntimeWarning: invalid value encountered in sqrt

    nan

```python
y = inf - inf
y
```

    nan

## Comparing `nan` values in Python

- Beware of comparing `nan` values

```python
x == y
```

    False

- To test whether a value is `nan` use the `isnan` function:

```python
isnan(x)
```

    True

## `NaN` is not the same as `None`

- `None` represents a *missing* value.

- `NaN` represents an *invalid* floating-point value.

- These are fundamentally different entities:

```python
nan is None
```

    False

```python
isnan(None)
```

## Beware finite precision

```python
x = 0.1 + 0.2
```

```python
x == 0.3
```

    False

```python
x
```

    0.30000000000000004

## Relative and absolute error

- Consider a floating point number $x_{fp}$ which represents a real number $x \in \mathbf{R}$.

- In general, we cannot precisely represent the real number; that is $x_{fp} \neq x$.

- The absolute error $r$ is $r = x - x_{fp}$.

- The relative error $R$ is:

\begin{equation}
R = \frac{x - x_{fp}}{x}
\end{equation}

## Numerical Methods

- In e.g. simulation models or quantiative analysis we typically repeatedly update numerical values inside long loops.

- Programs such as these implement *numerical algorithms*.

- It is very easy to introduce bugs into code like this.

## Numerical stability

- The round-off error associated with a result can be compounded in a loop.

- If the error increases as we go round the loop, we say the algorithm is
numerically *unstable*.

- Mathematicians design numerically stable algorithms using [numerical analysis](https://en.wikipedia.org/wiki/Numerical_analysis).

## Catastrophic Cancellation

- Suppose we have two real values $x$, and $y = x + \epsilon$.

- $\epsilon$ is very small and $x$ is very large.

- $x$ has an _exact_ floating point representation

- However, because of lack of precision $x$ and $y$ have the same floating
point representation.

  - i.e. they are represented as the same sequence of 64-bits

- Consider wheat happens when we compute $y - x$ in floating-point.

## Catestrophic Cancellation and Relative Error

- Catestrophic cancellation results in very large relative error.

- If we calculate $y - x$ in floating-point we will obtain the result 0.

- The correct value is $(x + \epsilon) - x = \epsilon$.

- The relative error is

\begin{equation}
\frac {\epsilon - 0} {\epsilon} = 1
\end{equation}

- That is, the relative error is $100\%$.

- This can result in [catastrophy](http://www-users.math.umn.edu/~arnold/disasters/).

## Catastrophic Cancellation in Python

```python
x = 3.141592653589793
x
```

    3.141592653589793

```python
y = 6.022e23
x = (x + y) - y
```

```python
x
```

    0.0

### Cancellation versus addition

- Addition, on the other hand, is not catestrophic.

```python
z = x + y
z
```

    6.022e+23

- The above result is still inaccurate with an absolute error $r \approx \pi$.

- However, let's examine the *relative* error:

\begin{equation}
R = \frac {1.2044 \times 10^{24} - (1.2044 \times 10^{24} + \pi)} {1.2044 \times 10^{24} + \pi} \\
\approx 10^{-24}
\end{equation}

- Here we see that that the relative error from the addition is miniscule compared with the cancellation.

### Floating-point arithemetic is nearly always inaccurate.

- You can hardly-ever eliminate absolute rounding error when using floating-point.

- The best we can do is to take steps to minimise error, and prevent it from increasing as your calculation progresses.

- Cancellation can be *catastrophic*, because it can greatly increase the *relative* error in your calculation.

## Use a well-tested library for numerical algorithms.

- Avoid subtracting two nearly-equal numbers.

- Especially in a loop!

- Better-yet use a well-validated existing implementation in the form of a numerical library.

## Importing numpy

- Functions for numerical computiing are provided by a separate _module_ called [`numpy`](http://www.numpy.org/).

- Before we use the numpy module we must import it.

- By convention, we import `numpy` using the alias `np`.

- Once we have done this we can prefix the functions in the numpy library using the prefix `np.`

```python
import numpy as np
```

- We can now use the functions defined in this package by prefixing them with `np`.

## Arrays

- Arrays represent a collection of values.

- In contrast to lists:
    - arrays typically have a *fixed length*
        - they can be resized, but this involves an expensive copying process.
    - and all values in the array are of the *same type*.
        - typically we store floating-point values.

- Like lists:
    - arrays are *mutable*;
    - we can change the elements of an existing array.

## Arrays in `numpy`

- Arrays are provided by the `numpy` module.

- The function `array()` creates an array given a list.

```python
import numpy as np
x = np.array([0, 1, 2, 3, 4])
x
```

    array([0, 1, 2, 3, 4])

## Array indexing

- We can index an array just like a list

```python
x[4]
```

    4

```python
x[4] = 2
x
```

    array([0, 1, 2, 3, 2])

## Arrays are not lists

- Although this looks a bit like a list of numbers, it is a fundamentally different type of value:

```python
type(x)
```

    numpy.ndarray

- For example, we cannot append to the array:

```python
x.append(5)
```

## Populating Arrays

- To populate an array with a range of values we use the `np.arange()` function:

```python
x = np.arange(0, 10)
print(x)
```

    [0 1 2 3 4 5 6 7 8 9]

- We can also use floating point increments.

```python
x = np.arange(0, 1, 0.1)
print(x)
```

    [0.  0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9]

## Functions over arrays

- When we use arithmetic operators on arrays, we create a new array with the result of applying the operator to each element.

```python
y = x * 2
y
```

    array([0. , 0.2, 0.4, 0.6, 0.8, 1. , 1.2, 1.4, 1.6, 1.8])

- The same goes for numerical functions:

```python
x = np.array([-1, 2, 3, -4])
y = abs(x)
y
```

    array([1, 2, 3, 4])

## Vectorized functions

- Note that not every function automatically works with arrays.

- Functions that have been written to work with arrays of numbers are called *vectorized* functions.

- Most of the functions in `numpy` are already vectorized.

- You can create a vectorized version of any other function using the higher-order function `numpy.vectorize()`.

## `vectorize` example

```python
def myfunc(x):
    if x >= 0.5:
        return x
    else:
        return 0.0

fv = np.vectorize(myfunc)
```

```python
x = np.arange(0, 1, 0.1)
x
```

    array([0. , 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9])

```python
fv(x)
```

    array([0. , 0. , 0. , 0. , 0. , 0.5, 0.6, 0.7, 0.8, 0.9])

## Testing for equality

- Because of finite precision we need to take great care when comparing floating-point values.

- The numpy function [`allclose()`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.allclose.html) can be used to test equality of floating-point numbers within a relative tolerance.

- It is a vectorized function so it will work with arrays as well as single floating-point values.

```python
x = 0.1 + 0.2
y = 0.3
x == y
```

    False

```python
np.allclose(x, y)
```

    True

## Plotting with `matplotlib`

- We will use a module called `matplotlib` to plot some simple graphs.

- This module has a nested module called `pyplot`.

- By convention we import this with the alias `plt`.

- This module provides functions which are very similar to [MATLAB plotting commands](http://uk.mathworks.com/help/matlab/ref/plot.html).

```python
import matplotlib.pyplot as plt

```

### A simple linear plot

```python
x = np.arange(0, 1, 0.1)
y = x*2 + 5
plt.plot(x, y)
plt.xlabel('$x$')
plt.ylabel('$y = 2x + 5$')
plt.title('Linear plot')
plt.show()
```

### Plotting a sine curve

```python
from numpy import pi, sin

x = np.arange(0, 2*pi, 0.01)
y = sin(x)
plt.plot(x, y)
plt.xlabel('x')
plt.ylabel('sin(x)')
plt.show()
```

## Multi-dimensional data

- Numpy arrays can hold multi-dimensional data.

- To create a multi-dimensional array, we can pass a list of lists to the `array()` function:

```python
import numpy as np

x = np.array([[1,2], [3,4]])
x
```

    array([[1, 2],
           [3, 4]])

### Arrays containing arrays

- A multi-dimensional array is an array of an arrays.

- The outer array holds the rows.

- Each row is itself an array:

```python
x[0]
```

    array([1, 2])

```python
x[1]
```

    array([3, 4])

- So the element in the second row, and first column is:

```python
x[1][0]
```

    3

### Matrices

- We can create a matrix from a multi-dimensional array.

```python
M = np.matrix(x)
M
```

    matrix([[1, 2],
            [3, 4]])

### Plotting multi-dimensional with matrices

- If we supply a matrix to `plot()` then it will plot the y-values taken from the *columns* of the matrix (notice the transpose in the example below).

```python
from numpy import pi, sin, cos

x = np.arange(0, 2*pi, 0.01)
y = sin(x)
ax = plt.plot(x, np.matrix([sin(x), cos(x)]).T)
plt.show()
```

### Performance

- When we use `numpy` matrices in Python the corresponding functions are linked with libraries written in C and FORTRAN.

- For example, see the [BLAS (Basic Linear Algebra Subprograms) library](http://www.netlib.org/blas/).

- These libraries are very fast.

- Vectorised code can be more easily ported to frameworks like [TensorFlow](https://www.tensorflow.org/) so that operations are performed in parallel using GPU hardware.

### Matrix Operators

- Once we have a matrix, we can perform matrix computations.

- To compute the [transpose](http://mathworld.wolfram.com/MatrixTranspose.html) and [inverse](http://mathworld.wolfram.com/MatrixInverse.html) use the `T` and `I` attributes:

To compute the transpose $M^{T}$

```python
M.T

```

    matrix([[1, 3],
            [2, 4]])

To compute the inverse $M^{-1}$

```python
M.I
```

    matrix([[-2. ,  1. ],
            [ 1.5, -0.5]])

### Matrix Dimensions

- The total number of elements, and the dimensions of the array:

```python
M.size
```

    4

```python
M.shape
```

    (2, 2)

```python
len(M.shape)
```

    2

### Creating Matrices from strings

- We can also create arrays directly from strings, which saves some typing:

```python
I2 = np.matrix('2 0; 0 2')
I2
```

    matrix([[2, 0],
            [0, 2]])

- The semicolon starts a new row.

### Matrix Multiplication

Now that we have two matrices, we can perform [matrix multiplication](https://en.wikipedia.org/wiki/Matrix_multiplication):

```python
M * I2
```

    matrix([[2, 4],
            [6, 8]])

### Matrix Indexing

- We can [index and slice matrices](http://docs.scipy.org/doc/numpy/user/basics.indexing.html) using the same syntax as lists.

```python
M[:,1]
```

    matrix([[2],
            [4]])

### Slices are references

- If we use this is an assignment, we create a *reference* to the sliced elements, *not* a copy.

```python
V = M[:,1]  # This does not make a copy of the elements!
V
```

    matrix([[2],
            [4]])

```python
M[0,1] = -2
V
```

    matrix([[-2],
            [ 4]])

### Copying matrices and vectors

- To copy a matrix, or a slice of its elements, use the function `np.copy()`:

```python
M = np.matrix('1 2; 3 4')
V = np.copy(M[:,1])  # This does copy the elements.
V
```

    array([[2],
           [4]])

```python
M[0,1] = -2
V
```

    array([[2],
           [4]])

## Sums

One way we _could_ sum a vector or matrix is to use a `for` loop.

```python
vector = np.arange(0.0, 100.0, 10.0)
vector

```

    array([ 0., 10., 20., 30., 40., 50., 60., 70., 80., 90.])

```python
result = 0.0
for x in vector:
    result = result + x
result
```

    450.0

- This is not the most _efficient_ way to compute a sum.

## Efficient sums

- Instead of using a `for` loop, we can use a numpy function `sum()`.

- This function is written in the C language, and is very fast.

```python
vector = np.array([0, 1, 2, 3, 4])
print(np.sum(vector))
```

    10

## Summing rows and columns

- When dealing with multi-dimensional data, the 'sum()' function has a named-argument `axis` which allows us to specify whether to sum along, each rows or columns.

```python
matrix = np.matrix('1 2 3; 4 5 6; 7 8 9')
print(matrix)
```

    [[1 2 3]
     [4 5 6]
     [7 8 9]]

### To sum along rows:

```python
np.sum(matrix, axis=0)
```

    matrix([[12, 15, 18]])

### To sum along columns:

```python
np.sum(matrix, axis=1)
```

    matrix([[ 6],
            [15],
            [24]])

## Cumulative sums

- Suppose we want to compute $y_n = \sum_{i=1}^{n} x_i$ where $\vec{x}$ is a vector.

```python
import numpy as np
x = np.array([0, 1, 2, 3, 4])
y = np.cumsum(x)
print(y)
```

    [ 0  1  3  6 10]

## Cumulative sums along rows and columns

```python
x = np.matrix('1 2 3; 4 5 6; 7 8 9')
print(x)
```

    [[1 2 3]
     [4 5 6]
     [7 8 9]]

```python
y = np.cumsum(x)
np.cumsum(x, axis=0)
```

    matrix([[ 1,  2,  3],
            [ 5,  7,  9],
            [12, 15, 18]])

```python
np.cumsum(x, axis=1)
```

    matrix([[ 1,  3,  6],
            [ 4,  9, 15],
            [ 7, 15, 24]])

## Cumulative products

- Similarly we can compute $y_n = \Pi_{i=1}^{n} x_i$ using `cumprod()`:

```python
import numpy as np
x = np.array([1, 2, 3, 4, 5])
np.cumprod(x)
```

    array([  1,   2,   6,  24, 120])

- We can compute cummulative products along rows and columns using the `axis` parameter, just as with the `cumsum()` example.

## Generating (pseudo) random numbers

- The nested module `numpy.random` contains functions for generating random numbers from different probability distributions.

```python
from numpy.random import normal, uniform, exponential, randint
```

- Suppose that we have a random variable $\epsilon \sim N(0, 1)$.

- In Python we can draw from this distribution like so:

```python
epsilon = normal()
epsilon
```

    0.1465312427787133

- If we execute another call to the function, we will make a _new_ draw from the distribution:

```python
epsilon = normal()
epsilon
```

    -2.3135050810408773

## Pseudo-random numbers

- Strictly speaking, these are not random numbers.

- They rely on an initial state value called the *seed*.

- If we know the seed, then we can predict with total accuracy the rest of the sequence, given any "random" number.

- Nevertheless, statistically they behave like independently and identically-distributed values.
    - Statistical tests for correlation and auto-correlation give insignificant results.

- For this reason they called *pseudo*-random numbers.

- The algorithms for generating them are called Pseudo-Random Number Generators (PRNGs).

- Some applications, such as cryptography, require genuinely unpredictable sequences.
    - never use a standard PRNG for these applications!

## Managing seed values

- In some applications we need to reliably reproduce the same sequence of pseudo-random numbers that were used.

- We can specify the seed value at the beginning of execution to achieve this.

- Use the function `seed()` in the `numpy.random` module.

## Setting the seed

```python
from numpy.random import seed

seed(5)

```

```python
normal()
```

    0.44122748688504143

```python
normal()
```

    -0.33087015189408764

```python
seed(5)
```

```python
normal()
```

    0.44122748688504143

```python
normal()
```

    -0.33087015189408764

## Drawing multiple variates

- To generate more than number, we can specify the `size` parameter:

```python
normal(size=10)
```

    array([ 2.43077119, -0.25209213,  0.10960984,  1.58248112, -0.9092324 ,
           -0.59163666,  0.18760323, -0.32986996, -1.19276461, -0.20487651])

- If you are generating very many variates, this will be *much* faster than using a for loop

- We can also specify more than one dimension:

```python
normal(size=(5,5))
```

    array([[-0.35882895,  0.6034716 , -1.66478853, -0.70017904,  1.15139101],
           [ 1.85733101, -1.51117956,  0.64484751, -0.98060789, -0.85685315],
           [-0.87187918, -0.42250793,  0.99643983,  0.71242127,  0.05914424],
           [-0.36331088,  0.00328884, -0.10593044,  0.79305332, -0.63157163],
           [-0.00619491, -0.10106761, -0.05230815,  0.24921766,  0.19766009]])

## Histograms

- We can plot a histograms of randomly-distributed data using the `hist()` function from matplotlib:

```python
import matplotlib.pyplot as plt

data = normal(size=10000)
ax = plt.hist(data)
plt.title('Histogram of normally distributed data ($n=10^5$)')
plt.show()
```

## Computing histograms as matrices

- The function `histogram()` in the `numpy` module will count frequencies into bins and return the result as a 2-dimensional array.

```python
import numpy as np
np.histogram(data)
```

    (array([  23,  136,  618, 1597, 2626, 2635, 1620,  599,  130,   16]),
     array([-3.59780883, -2.87679609, -2.15578336, -1.43477063, -0.71375789,
             0.00725484,  0.72826758,  1.44928031,  2.17029304,  2.89130578,
             3.61231851]))

## Descriptive statistics

- We can compute the descriptive statistics of a sample of values using the numpy functions `mean()` and `var()` to compute the sample mean $\bar{X}$ and sample [variance](https://en.wikipedia.org/wiki/Variance) $\sigma_{X}^2$ .

```python
np.mean(data)
```

    -0.00045461080333497925

```python
np.var(data)
```

    1.0016048722546331

- These functions also have an `axis` parameter to compute mean and variances of columns or rows of a multi-dimensional data-set.

## Descriptive statistics with `nan` values

- If the data contains `nan` values, then the descriptive statistics will also be `nan`.

```python
from numpy import nan
import numpy as np
data = np.array([1, 2, 3, 4, nan])
np.mean(data)
```

    nan

- To omit `nan` values from the calculation, use the functions `nanmean()` and `nanvar()`:

```python
np.nanmean(data)
```

    2.5

## Discrete random numbers

- The `randint()` function in `numpy.random` can be used to draw from a uniform discrete probability distribution.

- It takes two parameters: the low value (inclusive), and the high value (exclusive).

- So to simulate one roll of a die, we would use the following Python code.

```python
die_roll = randint(0, 6) + 1
die_roll
```

    4

- Just as with the `normal()` function, we can generate an entire sequence of values.

- To simulate a [Bernoulli process](https://en.wikipedia.org/wiki/Bernoulli_process) with $n=20$ trials:

```python
bernoulli_trials = randint(0, 2, size = 20)
bernoulli_trials
```

    array([1, 1, 0, 1, 1, 1, 0, 1, 0, 0, 1, 1, 1, 0, 0, 0, 0, 1, 1, 0])

__Acknowledgements__

The early sections of this notebook were adapted from an online article by [Steve Hollasch](https://steve.hollasch.net/cgindex/coding/ieeefloat.html).


```julia
using Pkg
if isfile("../Project.toml") && isfile("../Manifest.toml")
    Pkg.activate("..");
end

using Random
using Distributions
using Plots
using StatsPlots
using prml
using prml: pdf
Random.seed!(1234);
gr();
```

## Bernoulli Distributions

$$
\mathrm{Bern}(x \mid \mu) = x^{\mu} \cdot (1 - x)^{1 - \mu}
$$

```julia
mu = [0.0, 1.0, 1.0, 1.0];
bernoulli = BernoulliDist(mu);
pdf(bernoulli, 0)
```

## Beta Distributions

$$
\mathrm{Beta}(\mu \mid a, b) = \dfrac{\Gamma(a+b)}{\Gamma(a) \Gamma(b)} \mu^{a-1}(1-\mu)^{b-1}
$$

```julia
plots = [];
l = @layout [a b; c d]

x = collect(range(0, stop=1.0, length=100));
for (index, values) in enumerate([[0.1, 0.1], [1, 1], [2, 3], [8, 4]])
    a, b = values[1], values[2];
    beta = BetaDist(a, b);
    y = pdf(beta, x);
    p = plot(x, y, xlims=(0, 1), ylims=(0, 3), label="a=$a, b=$b", lw=2);
    push!(plots, p);
end

plot(plots[1], plots[2], plots[3], plots[4], layout=l)
```

### ML and MAP estimation

```julia
plots = [];
l = @layout [a; b]

x = collect(range(0, stop=1.0, length=100));

beta = BetaDist(2, 2);
p = plot(x, pdf(beta, x), xlim=(0, 1), ylim=(0, 2), label="prior", lw=2);
push!(plots, p);

bern = BernoulliDist(beta);
fitting(bern, [1.0])
p = plot(x, pdf(bern._beta, x), xlim=(0, 1), ylim=(0, 2), label="posterior", lw=2);
push!(plots, p);

plot(plots[1], plots[2], layout=l)
```

```julia
println("Maximum likehood estimation");

model = BernoulliDist([1.0]);
count = draw(model, 10000);
println("$(count) out of 10000 is label-1");

println("Beysian estimation");
model = BernoulliDist(BetaDist(1.0, 1.0));
fitting(model, [1.0]);
count = draw(model, 10000);
println("$(count) out of 10000 is label-1");
```

## Categorical variables

```julia
model = CategoricalDist([0.0]);
fitting(model, [[0. 1. 0.]; [1. 0. 0.]; [0. 1. 0.]]);
println(model._mu);

mu = DirichletDist(ones(3));
model = CategoricalDist(mu);
println("prior");
println(model._dirichlet._alpha);
trials = [[1. 0. 0.]; [1. 0. 0.]; [0. 1. 0.]];
fitting(model, trials);
println("posterior");
println(model._dirichlet._alpha);
```

### Dirichlet distribution

```julia
dirichlet1 = DirichletDist([0.1, 0.1, 0.1]);
dirichlet2 = DirichletDist([1., 1., 1.]);
dirichlet3 = DirichletDist([10., 10., 10.]);
```

## Gaussian Distribution

### Central Limit Theorem

```julia
n_sample = 10000;
plots = [];
for N in [1, 2, 10]
    # a matrix of size (n_samples, N)
    samples = rand(Uniform(0, 1), n_sample, N)
    mean_samples = mean(samples, dims=2);
    p =　StatsPlots.histogram(mean_samples, bins=20, legend=:topright);
    push!(plots, p);
end

plot(plots[1], plots[2], plots[3], layout=(1, 3))
```

### MAP estimation for the mean of Gaussian distribution

For a Gaussian distribution $x \sim \mathcal{N}(\mu, \sigma^2)$ with $\mu \sim \mathcal{N}(\mu_0, \sigma^{2}_0)$, given $N$ observations of $x$, denoted as  $\boldsymbol{x} = [x_0, x_1, \cdots, x_N]$, the posterior distribution of $\mu$ is given by $\mu \sim \mathcal{N}(\mu_N, \sigma^{2}_N)$ where

$$\begin{align} \mu_N &= \dfrac{\sigma^{2}_N}{N\sigma_{0}^2 + \sigma^{2}_N}\mu_0 + \dfrac{N \sigma_{0}^2}{N\sigma_{0}^2 + \sigma^{2}_N}\mu_{\mathrm{ML}} \\ \dfrac{1}{\sigma^{2}_N} &= \dfrac{1}{\sigma_{0}^2} + \dfrac{N}{\sigma^2}
\end{align}$$

```julia
mu_prior = GaussianDist(0.0, 0.1);
model = GaussianBayesMeanDist(mu_prior, 0.1);

x = collect((range(-1, stop=1.0, length=200)));
plot(x, pdf(model._mu_gauss, x), label="N=0", xlim=(-1, 1), ylim=(0, 5), lw=2, color="steelblue")

fitting(model, [rand(Normal(0.8, 0.01))]);
plot!(x, pdf(model._mu_gauss, x), label="N=1", xlim=(-1, 1), ylim=(0, 5), lw=2, color="orange")

fitting(model, [rand(Normal(0.8, 0.01))]);
plot!(x, pdf(model._mu_gauss, x), label="N=2", xlim=(-1, 1), ylim=(0, 5), lw=2, color="green");

fitting(model, rand(Normal(0.8, 0.01), 8));
plot!(x, pdf(model._mu_gauss, x), label="N=10", xlim=(-1, 1), ylim=(0, 5), lw=2, color="red", legend=:topleft)
```

### Gamma distribution

$$
\begin{align} \mathrm{Gam}(\lambda \mid a, b) = (b^a / \Gamma(a)) \lambda^{a-1} \exp(-b\lambda) \end{align}
$$

```julia
x = collect(range(0, stop=2.0, length=100));
plots = [];
for (index, values) in enumerate([[0.1, 0.1], [1., 1.], [2., 3.], [4., 6.]])
    gamma = GammaDist(values[1], values[2]);
    p = plot(x, pdf(gamma, x), xlim=(0, 2), ylim=(0, 2), label="a=$(values[1]), b=$(values[2])", lw=2)
    push!(plots, p);
end

plot(plots[1], plots[2], plots[3], plots[4], layout=(2, 2))
```

### MAP estimation for the variance of Gaussian distribution

For a Gaussian distribution $x \sim \mathcal{N}(\mu, \lambda = \sigma^2)$ with $\lambda \sim \mathrm{Gam}(\lambda \mid a, b)$, given $N$ observations of $x$, denoted as  $\boldsymbol{x} = [x_0, x_1, \cdots, x_N]$, the posterior distribution of $\lambda$ is given by $\lambda \sim \mathrm{Gam}(\lambda \mid a_N, b_N)$ where

$$
\begin{align} a_N &=\dfrac{N}{2} + a  = \dfrac{N}{2} + \dfrac{2a}{2} \\ b_N &= b + \dfrac{N}{2} \sigma^2_{\mathrm{ML}}  = \dfrac{2a}{2}\dfrac{b}{a} + \dfrac{N}{2} \sigma^2_{\mathrm{ML}} \end{align}
$$

```julia
tau = GammaDist(1.0, 1.0);
model = GaussianBayesVarDist(tau);
println("a=$(model._var_gamma._a), b=$(model._var_gamma._b)");

fitting(model, rand(Normal(0, 1.414), 100));

println("a=$(model._var_gamma._a), b=$(model._var_gamma._b)");
```

## Multivariate Gaussian Distribution

```julia
X = rand(Uniform(1, 3), 2, 100);
gaussian = MultivariateGaussianDist(2);
fitting(gaussian, X);

x = range(-5, stop=10, length=100);
y = range(-5, stop=10, length=100);
grids = [[i, j] for i in x, j in y];
vals = [pdf(gaussian, grids[i, j]) for i in 1:100, j in 1:100];
contour(x, y, vals)
plot!(X[1,:], X[2, :], seriestype=:scatter, legend=nothing)
```


# Maximum Likelihood Model Parameter Estimation

\begin{equation*}
D = \text{data} \\
\theta = \text{model parameters} \\
y(\theta, x) = \text{model}
\end{equation*}

A common parameter-estimation strategy is to minimize error,

\begin{equation*}
\sum_{i}(D_i - y_i(\theta, x))^2
\end{equation*}

E.g.

Create some phony straight line data with $\theta=(a, b)$ and normally distributed noise in $y$,

```
n_points = 10
x = linspace(0,10,n_points)
def y(x, theta=(2, 4), noise=0):
    return  theta[0] * x + theta[1] + noise*randn(n_points)
D = y(x, noise=1)
scatter(x,D)
```

We are trying to discover parameters, $\theta$, so write error in terms of only $theta$,

$SSE(\theta; y(x), D)$

```
# cheat, do this with global scope of y, x and D
def ssq_err(theta):
    return np.sum((D - y(x,theta))*(D - y(x,theta)).transpose())

from scipy.optimize import minimize
fit = minimize(ssq_err,[1,1])
print fit
scatter(x, D)
plot(x,y(x, fit.x))
print fit
```

Not amazing so far $\ldots$

__Leading quesiton__:  Is normally distributed noise "unusually compatible" with least squares errors minimization?

Maybe we should do something that takes into account the expected distibution of the model errors? What if the errors are distributed by Poisson, for example, does sum-squared-error still make sense?

```
from scipy.stats import norm, poisson
rvn, rvp = norm(5,1), poisson(5)
x = np.linspace(-2,10,140)
plot(x, rvn.pdf(x))
xd = np.arange(0, np.minimum(rvp.dist.b, 15))
vlines(xd, 0, rvp.pmf(xd), lw=2, color="red")
```

__New Idea__:  The parameter fitting problem can be framed as "Can we find the mostly likely value of the parameters $\theta$ of the model given the data?"

i.e. maximize $P(\theta|D)$

### Frequentist Statistics

Significance is specified by probability that
the data was drawn from the “wrong”
model (null hypothesis): p value

* Not the probability a model is correct
* Only random variables have probabilities

That is, from a frequentist perspective, the PDF for normal noise is in terms of $P(D|\theta)$,

E.g.
\begin{equation*}
P(D|\theta) \propto exp(\sum_i -\frac{(D_i - y_i(\theta))^2}{2\sigma^2})
\end{equation*}

We need to flip this around.

### Bayesian inference

* Frequentist: Probability is a frequency
(of a random variable)
* Bayesian: Probability is a measure of
belief or certainty

Some statistics Rules

\begin{equation}
P(A,B) = P(A|B)P(B) = P(B|A)P(A) \\
P(A) = \int_\Omega P(A,B)dB = \int_\Omega P(A|B)P(B)dB \\
\end{equation}

Bayesian statistics is the application of these rules

Back to data and parameters:

\begin{equation*}
P(\theta|D) = \frac{P(D|\theta)P(\theta)}{P(D)}
\end{equation*}

\begin{equation*}
P(D|\theta) : \text{Likelihood} \\
P(\theta) : \text{Prior} \\
P(D) : \text{Evidence} \\
P(\theta|D) : \text{Posterier} \\
\end{equation*}

\begin{equation*}
P(D) = \int_\Omega P(D|\theta)P(\theta)d\theta \\
\end{equation*}

Integration over the parameter space of the model.

## Simple Discrete Model: Coin Toss

In 10 trials, we get 7 heads: What's our best estimate of the probability of flipping heads with this coin?

The distribution of coin toss outcomes is the Binomial Distribution.

$pmf(y|t,w) = \binom{n}{k} p^k (1-p)^{(n-k)}$

Let's not forget how obvious this is and guess 0.7.

```
from scipy.stats import binom
rv = binom(10, 0.7)
x = np.arange(0, np.minimum(rv.dist.b, 10))
h = plt.vlines(x, 0, rv.pmf(x), lw=2, color="orange")
```

"Speciﬁcally, the PDF in Fig. 1 is a function of the data given a particular set of parameter values, deﬁned on the data scale. On the other hand, the likelihood function is a function of the parameter given a particular set of observed data, deﬁned on the
parameter scale. In short, Fig. 1 tells us the probability of a particular data value for a ﬁxed parameter, whereas Fig. 2 tells us the likelihood (‘‘unnormalized probability’’) of a particular parameter value for a ﬁxed data set." (.J. Myung / Journal of Mathematical Psychology 47 (2003) 90–100)

```
def f(w):
    rv = binom(10, w)
    return rv.pmf(7)
# Hold the data constant, vary the parameter
x = np.linspace(0,1,100)
plot(x, f(x))
```

It looks like the maximum likelihood on the data over the parameter is around 0.7?

Our data could have been (6,10) or (8,10) with a 0.7 coin.  But we started by asking about the coin, to make any headway, we need to flip the question around.

```
dat = [10, 7]
def L_binom(w):
    rv = binom(dat[0], w)
    x = dat[1]
    # -1 to make max problem a min problem
    return -rv.pmf(x)
```

```
x = np.linspace(0.1, 0.9, 9)
scatter(x, L_binom(x))
from scipy.optimize import minimize_scalar
minimize_scalar(L_binom, bounds=[0,1], method="bounded")
```

## Slightly More Complicated Model: Exponential with Normal Error

```
n_points = 10
x = linspace(0,100,n_points)
def y(x, theta=(10, 0.05), noise=0):
    return  theta[0] * np.exp(-x * theta[1]) + noise*randn(n_points)
D = y(x, noise=1)
scatter(x,D)
```

Likelihood:

\begin{equation}
P(D|\theta) \propto \prod_i {\exp(-\frac{(D_i - y_i(\theta))^2}{2\sigma^2})} \propto \prod_i {\exp(-\frac{(D_i - (10\exp(-wx)))^2}{2})}
\end{equation}

```
# use x, D from global scope
def L_cost(w):
    return np.prod(np.exp(((D - y(x,(10, w)))*(D - y(x,(10, w))).transpose())/2.))
w = np.linspace(0,0.1,10)
L_c = [L_cost(i) for i in w]
scatter(w,L_c)
```

Numerically troubling!  Turn the product into a sum with log!

```
# use x, D from global scope
def L_cost(w):
    return np.sum(((D - y(x,(10, w)))*(D - y(x,(10, w))).transpose())/2.)
w = np.linspace(0,0.1,10)
L_c = [L_cost(i) for i in w]
scatter(w,L_c)
from scipy.optimize import minimize_scalar
minimize_scalar(L_cost, bounds=[0,0.1], method="bounded")
```

So it might be useful to use the $\log$ likelihood:

\begin{equation}
\log(P(D|\theta)) \propto \sum_i \log(exp(-\frac{(D_i - (10\exp(-wx)))^2}{2})) = - \sum_i \frac{(D_i - (10\exp(-wx)))^2}{2}
\end{equation}

__Normally distributed errors + Minimum-Log-Likelihood is the same solution as Lease Squares.__

## Exponential Decay and Poisson Likelihood

\begin{equation}
\log(P) \propto \sum_i \log(\frac{\lambda^{x_i}e^{-\lambda}}{\Gamma(x_i)}) = \sum_i (x_i) \log\lambda + (-\lambda) - \log\Gamma(x_i)
\end{equation}

Sterling $\ldots$ or realize that $\Gamma(x_i)$ is a constant w.r.t. MLE estimation of $\lambda$.

\begin{equation}
\log(P(D|\theta)) \propto \sum_i \log(\frac{y_i(\theta))^{D_i}e^{-y_i(\theta))}}{\Gamma(D_i)}) = \sum_i D_i \log y_i(\theta) -y_i(\theta)
\end{equation}

```
from numpy.random import poisson
n_points = 20
x = linspace(0,300,n_points)
def y(x, theta=(110, 0.03), noise=0):
    ty = theta[0] * np.exp(-x * theta[1])
    if noise != 0:
        ty = [poisson(k)for k in ty]
    return ty
D = y(x, noise=5)
scatter(x,D)
plot(x,y(x))
```

```
# use x, D from global scope
def P_cost(*w):
    return -np.sum(D*np.log(y(x,*w)) - y(x,*w))
fit_ml = minimize(P_cost, (105,0.05))
print fit_ml
```

      status: 2
     success: False
        njev: 40
        nfev: 172
         fun: -901.2355508250871
           x: array([  1.09423618e+02,   2.74569336e-02])
     message: 'Desired error not necessarily achieved due to precision loss.'
        hess: array([[  1.84852921e-05,  -1.78246740e-05],
           [ -1.78246740e-05,   1.78203684e-05]])
         jac: array([  2.28881836e-05,   2.34222412e-03])

```
# cheat, do this with global scope of y, x and D
def ssq_err(*w):
    return np.sum((D - y(x,*w))*(D - y(x,*w)).transpose())
fit_ssq = minimize(ssq_err,[105,0.05])
print fit_ssq
scatter(x, D)
plot(x,y(x,fit_ml.x), color="green")
plot(x,y(x,fit_ssq.x), color="red")
```

Need more?

http://people.physics.anu.edu.au/~tas110/Teaching/Lectures/L3/Material/Myung03.pdf

http://sciencehouse.files.wordpress.com/2013/07/gordonconf13.pdf

http://www.math.utah.edu/~levin/M5080/mle.pdf

```

```


```python
import sympy as sy
from sympy.plotting import plot, plot_parametric
```

```python
sy_t = sy.symbols('t')
sy_c = (sy.cos(sy_t), sy.sin(sy_t))
sy_o = 0.5*sy_t
```

```python
plot_parametric(sy_c, (sy_t, 0,1), aspect_ratio=(1,1), ylim=(0,1), xlim=(0,1));
plot(sy_o, (sy_t, 0,1), aspect_ratio=(1,1), ylim=(0,1), xlim=(0,1));
```

```python
sy_c_d = tuple([sy.diff(x) for x in sy_c])
sy_c_n = (-sy_c_d[1], sy_c_d[0])
sy_cp = tuple([sy_c[i] + (sy_o * sy_c_n[i]) for i in range(2)])
print(sy_cp)
```

    (-0.5*t*cos(t) + cos(t), -0.5*t*sin(t) + sin(t))

```python
p1 = plot_parametric(sy_cp, (sy_t, 0,1), aspect_ratio=(1,1), ylim=(0,1), xlim=(0,1), show=False)
p2 = plot_parametric(sy_c, (sy_t, 0,1), aspect_ratio=(1,1), ylim=(0,1), xlim=(0,1), show=False)
p1.append(p2[0])
p1.show()
```

```python

```


```python
from sympy import init_session
init_session()
```

    IPython console for SymPy 1.0 (Python 3.5.3-64-bit) (ground types: gmpy)

    These commands were executed:
    >>> from __future__ import division
    >>> from sympy import *
    >>> x, y, z, t = symbols('x y z t')
    >>> k, m, n = symbols('k m n', integer=True)
    >>> f, g, h = symbols('f g h', cls=Function)
    >>> init_printing()

    Documentation can be found at http://docs.sympy.org/1.0/

## Linear axisymmetric conservative interpolation

We want to construct a linear polynomal through that points $r_i$, $r_{i+1}$ that gives the correct averages,
$r_i$, and $r_{i+1}$ when integrated over the volume, e.g.

$$\frac{1}{r_i \Delta r} \int_{r_{i-1/2}}^{r_{i+1/2}} f(r) r dr = f_i$$

There are 2 unknowns and two constraints

Define the polynomial

```python
a, b = symbols("a b")
r, r0 = symbols("r r0")
f = a*(r-r0) + b
f
```

```python
dr = symbols("\Delta")
```

### constraints

Define the constraint equations---here we set them up construct $A$ and $B$ integrals over the 2 control volumes

```python
f0, fp = symbols("f_i f_{i+1}")

rm12 = r0 - Rational(1,2)*dr
rp12 = r0 + Rational(1,2)*dr
rp32 = r0 + Rational(3,2)*dr

r1 = r0 + dr
```

interfaces

```python
rm12, rp12, rp32
```

centers

```python
r0, r1
```

```python
A = simplify(integrate(f*r/(r0*dr), (r, rm12, rp12)))
B = simplify(integrate(f*r/(r1*dr), (r, rp12, rp32)))
```

The analytic forms of the integrals

```python
A, B
```

Our linear system is now:

$$A = f_i$$
$$B = f_{i+1}$$

Now find the coefficients of the polynomial

```python
coeffs = solve([A-f0, B-fp], [a,b])
coeffs
```

And in pretty form, here's the polynomial

```python
l = f.subs(a,coeffs[a]).subs(b,coeffs[b])
l
```

Now, evaluate it on the interface

```python
a = l.subs(r, rp12)
```

```python
simplify(a)
```

```python

```


# Absolute Orientation of Two 3D Point Sets

## Goal

Given two 3D point sets $P$ and $Q$ as $3{\times}N$ matrices, the absolute orientation is a problem that finds rotation $R$, translation $t$, and scale $s$ satisfying
\begin{equation}
Q = s R P + t .
\end{equation}

Here, each corresponding columns of $P$ and $Q$ are assumed to be a corresponding point pair in 3D.

* Input
  * $P$, $Q$
* Output
  * $R, t, s$

## References
* Horn. Closed-form solution of absolute orientation using unit quaternions, JOSA 1987

## Libraries

```python
%matplotlib notebook
import numpy as np
import os
import sys
import cv2
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D

module_path = os.path.abspath(os.path.join('..'))
if module_path not in sys.path:
    sys.path.append(module_path)

from pycalib.calib import absolute_orientation

```

## Synthetic data

```python
N = 10
P = np.random.random((3, N))
R, _ = cv2.Rodrigues(np.random.random(3))
t = np.random.random(3)
s = np.random.random(1)
Q = s * (R @ P) + t[:, None]

fig = plt.figure()
ax = Axes3D(fig)
ax.plot(P[0,:], P[1,:], P[2,:], 'o')
ax.plot(Q[0,:], Q[1,:], Q[2,:], 'x')
fig.show()
```

    <IPython.core.display.Javascript object>

## Absolute Orientation

```python
R2, t2, s2 = absolute_orientation(P, Q)

P2 = s2 * (R2 @ P) + t2[:, None]

fig = plt.figure()
ax = Axes3D(fig)
ax.plot(P2[0,:], P2[1,:], P2[2,:], 'o')
ax.plot(Q[0,:], Q[1,:], Q[2,:], 'x')
fig.show()

assert np.allclose(R, R2)
assert np.allclose(t, t2)
assert np.allclose(s, s2)

```

    <IPython.core.display.Javascript object>


```python
# check convergence rate of Fourier series and
# global polynomials on either an exponential
# function or a Gaussian bell.
# Plot log-log and do regression / least square .
# Discuss the problems with using regression / least square

import sympy as sym
from approx1D import least_squares_non_verbose
import scipy
import scipy.integrate
import time

def Lagrange_series(N):
  psi = []
  h = 1.0/N
  points = [i*h for i in range(N+1)]
  for i in range(len(points)):
    p = 1
    for k in range(len(points)):
      if k != i:
        p *= (x - points[k])/(points[i] - points[k])
    psi.append(p)
  return psi

def Bernstein_series(N):
  psi = []
  for k in range(0,N+1):
    psi_k = sym.binomial(N, k)*x**k*(1-x)**(N-k)
    psi.append(psi_k)
  return psi

def Sinusoidal_series(N):
  psi = []
  for k in range(1,N):
    psi_k = sym.sin(sym.pi*k*x)
    psi.append(psi_k)
  return psi

def Taylor_series(N):
  psi = []
  for k in range(1,N):
    psi_k = x**k
    psi.append(psi_k)
  return psi

def series(series_type, N):
  if   series_type== "Taylor"     : return Taylor_series(N)
  elif series_type== "Sinusoidal" : return Sinusoidal_series(N)
  elif series_type== "Bernstein"  : return Bernstein_series(N)
  elif series_type== "Lagrange"   : return Lagrange_series(N)
  else: print("series type unknown ") # sys.exit(0)

def convergence_rate_analysis(series_type, func):
  Ns =[2, 4, 8, 16]
  norms = []
  cpu_times = []
  for N in Ns:

    psi = series(series_type, N)
    t0 = time.time()
    u, c = least_squares_non_verbose(gauss_bell, psi, Omega, False)
    t1 = time.time()

    error2 = sym.lambdify([x], (func - u)**2)
    L2_norm = scipy.integrate.quad(error2, Omega[0], Omega[1])
    L2_norm = scipy.sqrt(L2_norm)
    norms.append(L2_norm[0])
    cpu_times.append(t1-t0)

  return Ns, norms, cpu_times

Omega = [0, 1]
x = sym.Symbol("x")
gauss_bell = sym.exp(-(x-0.5)**2) - sym.exp(-0.5**2)
step = sym.Piecewise( (1, 0.25 < x), (0, True)  )- sym.Piecewise( (1, 0.75 < x), (0, True)  )
func = gauss_bell

import matplotlib.pyplot as plt
series_types = ["Taylor", "Sinusoidal", "Bernstein", "Lagrange"]
for series_type in series_types:
  Ns, norms, cpu_times = convergence_rate_analysis(series_type, func)
  plt.loglog(Ns, norms)

  print((series_type,  "Ns ", Ns))
  print((" norms ", norms))
  print((" cpu_time ", cpu_times))
  print("")

plt.legend(series_types, loc="lower left")
plt.savefig("Bell_convergence_loglog.png")
plt.savefig("Bell_convergence_loglog.pdf")
plt.hold(False)

for series_type in series_types:
#  plt.loglog(Ns, cpu_times)
  Ns, norms, cpu_times = convergence_rate_analysis(series_type, func)
  plt.loglog(Ns, cpu_times)
  plt.hold(True)

plt.legend(series_types, loc="lower right")
#plt.show()
plt.savefig("Bell_computations_loglog.png")
plt.savefig("Bell_computations_loglog.pdf")
plt.hold(False)

for series_type in series_types:
  Ns, norms, cpu_times = convergence_rate_analysis(series_type, func)
  plt.semilogy(Ns, norms)
  plt.hold(True)
  print((series_type,  "Ns ", Ns))
  print((" norms ", norms))
  print((" cpu_time ", cpu_times))
  print("")

plt.legend(series_types, loc="lower left")
#plt.show()
plt.savefig("Bell_convergence_semilog.png")
plt.savefig("Bell_convergence_semilog.pdf")
plt.hold(False)

```

Data Science and Machine Learning/Machine-Learning-In-Python-THOROUGH/EXAMPLES/FINITE_ELEMENTS/INTRO/SRC/10_CONVERGENCE_RATE_GLOBAL.ipynb


# Problem Statement

Given an inverted Pendulum of length=1, mass=1, design a controller
so that the damping ratio, $\zeta=0.707$, and the 2\% settling time, $t_s=1$ seconds.

# Dynamics

```python
import control
import sympy
import sympy.physics.mechanics as mech
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

mech.init_vprinting()
```

```python
# symbols
theta, Q, T = mech.dynamicsymbols('theta, Q, T')
m, l, g, t = sympy.symbols('m, l, g, t')

# frames
frame_e = mech.ReferenceFrame('e')
frame_p = frame_e.orientnew('p', 'Axis', (theta, frame_e.z))
#frame_p.set_ang_vel(frame_e, Q*frame_e.z)

# points
point_a = mech.Point('a')
point_a.set_vel(frame_e, 0)
point_p = point_a.locatenew('p', l*frame_p.x)
point_p.v2pt_theory(point_a, frame_e, frame_p)

# bodies
pendulum = mech.Particle('pendulum', point_p, m)
bodies = [pendulum]

# forces/moments
loads = [
    (point_p, -m*g*frame_e.x),
    (frame_p, T*frame_e.z - 0*Q*frame_e.z)
]

KM = mech.KanesMethod(frame=frame_e, q_ind=[theta], u_ind=[Q], kd_eqs=[theta.diff(t) - Q])
KM.kanes_equations(bodies, loads)
```

```python
KM.rhs()
```

```python
KM.linearize(A_and_B=True)
```

```python
op_point={m: 1, l: 1, g:9.8, theta: 0, Q: 0}
op_point
```

```python
A, B, u = KM.linearize(A_and_B=True, op_point=op_point)
A, B, u
```

```python
C = np.eye(2)
D = np.zeros((2, 1))
G_simo = control.ss2tf(A, B, C, D)
G_simo
```

$$\begin{bmatrix}\frac{1}{s^2 - 9.8}&\\\frac{s}{s^2 - 9.8}&\\ \end{bmatrix}$$

```python
G = G_simo[0, 0]
G
```

$$\frac{1}{s^2 - 9.8}$$

# Control Design

```python
def rlocus(G, kvect=np.logspace(-2, 0, 1000), kwargs={}):
    """
    A custom rlocus command to improve some of the plotting
    """
    rlist, klist = control.rlocus(G, kvect=kvect, Plot=False)
    for key in ['rlocus', 'zero', 'pole']:
        if key not in kwargs.keys():
            kwargs[key] = {}

    for z in G.zero():
        plt.plot(np.real(z), np.imag(g), 'o', markeredgewidth=3,  markerfacecolor='none', markersize=20, **kwargs['zero'])

    for z in G.pole():
        plt.plot(np.real(z), np.imag(g), 'x', markeredgewidth=3, markersize=20, **kwargs['pole'])

    for i in range(rlist.shape[1]):
        plt.plot(np.real(rlist[:, i]), np.imag(rlist[:, i]), **kwargs['rlocus'])

    plt.xlabel('Real')
    plt.ylabel('Imaginary')
```

```python
# requirements
zeta = np.sqrt(2)/2
ts = 1

sigma_d = 4/ts
a = np.arccos(zeta)
wd = sigma_d*np.tan(a)
sd = -sigma_d + wd*1j
sd
```

    (-4+3.9999999999999996j)

```python
plt.figure(figsize=(10, 10))
rlocus(G, kvect=np.logspace(-2, 2, 1000), kwargs={'rlocus': {'color': 'r'}})
plt.vlines(-sigma_d, -6, 6, color='g', linestyle='--', label='$t_s$')
plt.plot([0, -10*np.cos(a)], [0, 10*np.sin(a)],  'b--', label='$\zeta$')
plt.grid()
plt.legend()
plt.title('Uncompensated Root Locus');
```

```python
angle_deficiency = np.pi - np.angle(G(sd))
print('angle deficiency: {:4.1f} deg'.format(np.rad2deg(angle_deficiency)))
```

    angle deficiency: 73.0 deg

```python
s = control.tf([1, 0], [1])
z = -np.imag(sd)/np.tan(angle_deficiency) + np.real(sd)
C = (s-z)
K = 1/np.abs((G*C)(sd))
H = C*K
print('K:', K, 'z:', z)
```

    K: 8.0 z: -5.2250000000000005

# Proportional Derivative Controller

```python
H
```

$$\frac{8 s + 41.8}{1}$$

```python
control.rlocus((s+1.5)/(s*(s+2)*(s+1)));
```

```python
plt.figure(figsize=(10, 10))
rlocus(G*H, kvect=np.linspace(0, 4, 5000))
plt.vlines(-sigma_d, -6, 6, color='g', linestyle='--', label='$t_s$')
plt.plot([0, -10*np.cos(a)], [0, 10*np.sin(a)],  'b--', label='$\zeta$')
plt.plot(np.real(sd), np.imag(sd), '*', label='$s_d$', markersize=20, color='orange')
plt.grid()
plt.legend()
plt.axis([-12, 4, -8, 8]);
plt.title('Compensated Root Locus');
```

```python
L = G*H
Kp = np.real(L(0))
ess = 1/(1 + Kp)
Gc = G*H/(1 + G*H)
Gc
```

$$\frac{8 s^3 + 41.8 s^2 - 78.4 s - 409.6}{s^4 + 8 s^3 + 22.2 s^2 - 78.4 s - 313.6}$$

This is the step response of the system, notice that there is a steady state error, to get rid of this, we should have specified a steady state error design requirement. This will require the use of an integrator, which is a full PID controller instead of just a PD controller.

```python
t_step, y_step = control.step_response(Gc, np.linspace(0,2, 1000));
plt.figure(figsize=(10, 5))
plt.plot(t_step, y_step, label='y')
yss = 1 - ess
plt.hlines(1, 0, 2, linestyle='-', color='r', label='$r$')
plt.hlines(0.02 + yss, 0, 2, linestyle='-', color='g', label='$2\%$')
plt.hlines(-0.02 + yss, 0, 2, linestyle='-', color='g')
plt.vlines(ts, 0, 1.5, label='$t_s$')
plt.hlines(1 - ess, 0, 2, linestyle='--', color='k', label='$y_{ss}$')
plt.xlabel('t, sec')
plt.ylabel('y')
plt.grid()
plt.legend(loc='lower right', ncol=3)
plt.title('unit step response')
```

# Simulation

```python
class Controller:
    """
    Implements a discrete state space model.
    """

    def __init__(self, A, B, C, D, dt):
        self.A = A
        self.B = B
        self.C = C
        self.D = D
        self.x = np.zeros((np.shape(A)[0], 1))

    def update(self, u):
        self.x = self.A*self.x + self.B*u
        return self.C*self.x + self.D*u

    def reset(self):
        self.x = 0
```

```python
KM.rhs()
```

```python
import scipy.integrate

def do_sim(r, H, dt):
    x_vect = sympy.Matrix([theta, Q])
    u_vect = sympy.Matrix([T])
    p_vect = sympy.Matrix([m, g, l])
    f_pend = sympy.lambdify([x_vect, u_vect, p_vect], list(KM.rhs()))

    Hd = control.tf2ss(control.c2d(H*100/(s+100), dt))
    my_controller = Controller(Hd.A, Hd.B, Hd.C, Hd.D, Hd.dt)

    tf = 2
    p = [1, 9.8, 1]
    data = {
        't': [],
        'x': [],
        'u': []
    }
    x0 = np.array([0, 0])
    t0 = 0
    u0 = 0
    for i in range(int(tf/dt)):
        # save data
        data['t'].append(t0)
        data['x'].append(x0)
        data['u'].append(u0)

        # calculate control
        u0 = my_controller.update(r - x0[0])[0, 0]
        # saturation of motors
        if u0 > 100:
            u0 = 100
        elif u0 < -100:
            u0 = -100

        # integrate equations of motion
        res = scipy.integrate.solve_ivp(lambda t, x: f_pend(x, [u0], p), [t0, t0+dt], x0)
        x0 = res['y'][:, -1]
        t0 = res['t'][-1]

    for k in data:
        data[k] = np.array(data[k])
    return data

def plot_sim(r, t_step, y_step, sim_data):
    plt.figure(figsize=(20, 10))

    plt.subplot(121)
    plt.plot(sim_data['t'], np.rad2deg(sim_data['x'][:, 0]), label='nonlinear');
    plt.plot(t_step, np.rad2deg(r*y_step), label='linear approx')
    plt.hlines(np.rad2deg(r), 0, 2)
    plt.legend()
    plt.grid()
    plt.title('simulated step response')
    plt.xlabel('t, sec')
    plt.ylabel('$\\theta$, deg')

    plt.subplot(122)
    plt.plot(sim_data['t'], sim_data['u']);
    plt.xlabel('t, sec')
    plt.ylabel('T, N-m')
    plt.grid()
    plt.title('input')

    #plt.plot(sim_data['t'], sim_data['u']);
```

For small angles, the linear approximation let's us design a decent controll for the non-linear system, but the non-linear system has less damping than we would like.

```python
r = 0.1
sim_data = do_sim(r, H, 0.001)
plot_sim(r, t_step, y_step, sim_data)
```

When the motors saturate, or we are far from the equilibrium point, the linear approximation is very poor.

```python
r = 1
sim_data = do_sim(r, H, 0.001)
plot_sim(r, t_step, y_step, sim_data)
```