# Functions
So far in this course we've explored equations that perform algebraic operations to produce one or more results. A *function* is a way of encapsulating an operation that takes an input and produces exactly one ouput.

For example, consider the following function definition:

\begin{equation}f(x) = x^{2} + 2\end{equation}

This defines a function named ***f*** that accepts one input (***x***) and returns a single value that is the result calculated by the expression *x2 + 2*.

Having defined the function, we can use it for any input value. For example:

\begin{equation}f(3) = 11\end{equation}

You've already seen a few examples of Python functions, which are defined using the **def** keyword. However, the strict definition of an algebraic function is that it must return a single value. Here's an example of defining and using a Python function that meets this criteria:

```R
# define a function to return x^2 + 2
f = function(x){x**2 + 2}

# call the function
f(3)
```

11

You can use functions in equations, just like any other term. For example, consider the following equation:

\begin{equation}y = f(x) - 1\end{equation}

To calculate a value for ***y***, we take the ***f*** of ***x*** and subtract 1. So assuming that ***f*** is defined as previously, given an ***x*** value of 4, this equation returns a ***y*** value of **17** (*f*(4) returns 42 + 2, so 16 + 2 = 18; and then the equation subtracts 1 to give us 17). Here it is in Python:

```R
x = 4
y = f(x) - 1
cat(y)
```

    17

Of course, the value returned by a function depends on the input; and you can graph this with the iput (let's call it ***x***) on one axis and the output (***f(x)***) on the other.

```R
# Create an array of x values from -100 to 100
df = data.frame(x = seq(-100, 100))
df$y = f(df$x)

library(ggplot2)
library(repr)
options(repr.plot.width=4, repr.plot.height=3)
ggplot(df, aes(x,y)) + geom_line(color = 'magenta', size = 1) +
       xlab('x') + ylab('f(x)')
```

As you can see (if you hadn't already figured it out), our function is a *quadratic function* - it returns a squared value that results in a parabolic graph when the output for multiple input values are plotted.

## Bounds of a Function
Some functions will work for any input and may return any output. For example, consider the function ***u*** defined here:

\begin{equation}u(x) = x + 1\end{equation}

This function simply adds 1 to whatever input is passed to it, so it will produce a defined output for any value of ***x*** that is a *real* number; in other words, any "regular" number - but not an *imaginary* number like √-1, or ∞ (infinity). You can specify the set of real numbers using the symbol ${\rm I\!R}$ (note the double stroke). The values that can be used for ***x*** can be expressed as a *set*, which we indicate by enclosing all of the members of the set in "{...}" braces; so to indicate the set of all possible values for x such that x is a member of the set of all real numbers, we can use the following expression:

\begin{equation}\{x \in \rm I\!R\}\end{equation}

### Domain of a Function
We call the set of numbers for which a function can return value it's *domain*, and in this case, the domain of ***u*** is the set of all real numbers; which is actually the default assumption for most functions.

Now consider the following function ***g***:

\begin{equation}g(x) = (\frac{12}{2x})^{2}\end{equation}

If we use this function with an ***x*** value of **2**, we would get the output **9**; because (12 ÷ (2•2))2 is 9. Similarly, if we use the value **-3** for ***x***, the output will be **4**. However, what happens when we apply this function to an ***x*** value of **0**? Anything divided by 0 is undefined, so the function ***g*** doesn't work for an ***x*** value of 0.

So we need a way to denote the domain of the function ***g*** by indicating the input values for which a defined output can be returned. Specifically, we need to restrict ***x*** to a specific list of values - specifically any real number that is not 0. To indicate this, we can use the following notation:

\begin{equation}\{x \in \rm I\!R\;\;|\;\; x \ne 0 \}\end{equation}

This is interpreted as *Any value for x where x is in the set of real numbers such that x is not equal to 0*, and we can incorporate this into the function's definition like this:

\begin{equation}g(x) = (\frac{12}{2x})^{2}, \{x \in \rm I\!R\;\;|\;\; x \ne 0 \}\end{equation}

Or more simply:

\begin{equation}g(x) = (\frac{12}{2x})^{2},\;\; x \ne 0\end{equation}

When you plot the output of a function, you can indicate the gaps caused by input values that are not in the function's domain by plotting an empty circle to show that the function is not defined at this point:

```R
g = function(x){
    ## Use vectorized ifelse to return the value or the missing value, NA
    ifelse(x != 0,(12.2*x)^2, NA)
}

## Construct the data frame.
df2 = data.frame(x = seq(-100,100))
df2$y = g(df2$x) ## Call g(x) with the vector df2$x

## Make the plot
ggplot(df2, aes(x,y)) + geom_line(color = 'magenta', size = 1) +
       annotate("text", x = 0, y = 0, label = "O") + ## Put a symbol at the origin
       xlab('x') + ylab('g(x)')
```

Note that the function works for every value other than 0; so the function is defined for x = 0.000000001, and for x = -0.000000001; it only fails to return a defined value for exactly 0.

OK, let's take another example. Consider this function:

\begin{equation}h(x) = 2\sqrt{x}\end{equation}

Applying this function to a non-negative ***x*** value returns a meaningful output; but for any value where ***x*** is negative, the output is undefined.

We can indicate the domain of this function in its definition like this:

\begin{equation}h(x) = 2\sqrt{x}, \{x \in \rm I\!R\;\;|\;\; x \ge 0 \}\end{equation}

This is interpreted as *Any value for x where x is in the set of real numbers such that x is greater than or equal to 0*.

Or, you might see this in a simpler format:

\begin{equation}h(x) = 2\sqrt{x},\;\; x \ge 0\end{equation}

Note that the symbol ≥ is used to indicate that the value must be *greater than **or equal to*** 0; and this means that **0** is included in the set of valid values. To indicate that the value must be *greater than 0, **not including 0***, use the > symbol. You can also use the equivalent symbols for *less than or equal to* (≤) and *less than* (<).

When plotting a function line that marks the end of a continuous range, the end of the line is shown as a circle, which is filled if the function includes the value at that point, and unfilled if it does not.

Here's the Python to plot function ***h***:

```R
options(warn=-1) ## Turn off warnings from attempts to plot NAs

h = function(x){
    ## Use vectorized ifelse to return the value or the missing value, NA
    ifelse(x >= 0,(2 * sqrt(x)), NA)
}

## Construct the data frame.
df3 = data.frame(x = seq(-100,100))
df3$y = h(df3$x) ## Call g(x) with the vector df2$x

## Make the plot
ggplot(df3, aes(x,y)) + geom_line(color = 'magenta', size = 1) +
        annotate("text", x = 0, y = 0, label = "O") + # Put a symbol at the origin
        xlim(-1,101) + # Limit the range of x values displayed
        xlab('x') + ylab('h(x)')
```

Sometimes, a function may be defined for a specific *interval*; for example, for all values between 0 and 5:

\begin{equation}j(x) = x + 2,\;\; x \ge 0 \text{ and } x \le 5\end{equation}

In this case, the function is defined for ***x*** values between 0 and 5 *inclusive*; in other words, **0** and **5** are included in the set of defined values. This is known as a *closed* interval and can be indicated like this:

\begin{equation}\{x \in \rm I\!R\;\;|\;\; 0 \le x \le 5 \}\end{equation}

It could also be indicated like this:

\begin{equation}\{x \in \rm I\!R\;\;|\;\; [0,5] \}\end{equation}

If the condition in the function was **x > 0 and x < 5**, then the interval would be described as *open* and 0 and 5 would *not* be included in the set of defined values. This would be indicated using one of the following expressions:

\begin{equation}\{x \in \rm I\!R\;\;|\;\; 0 \lt x \lt 5 \}\end{equation}
\begin{equation}\{x \in \rm I\!R\;\;|\;\; (0,5) \}\end{equation}

Here's function ***j*** in Python:

```R
j = function(x){
    ## Use vectorized ifelse to return the value or the missing value, NA
    ifelse(x >= 0 & x <= 5, x + 2, NA)
}

## Construct the data frame.
df4 = data.frame(x = seq(-100,100))
df4$y = j(df4$x) ## Call g(x) with the vector df2$x

## Make the plot
suppressWarnings( # Suppress the warnings from attempts to plot NAs
    ggplot(df4, aes(x,y)) + geom_line(color = 'magenta', size = 1) +
           # Put a symbols at the ends of the lines
           geom_point(data = data.frame(x = c(0,5), y =c(2,7)), aes(x,y), color = 'magenta', size = 2) +
           xlim(-1,6) + # Limit the range of x values displayed
           xlab('x') + ylab('j(x)')
    )
```

Now, suppose we have a function like this:

\begin{equation}
k(x) = \begin{cases}
  0, & \text{if } x = 0, \\
  1, & \text{if } x = 100
\end{cases}
\end{equation}

In this case, the function has highly restricted domain; it only returns a defined output for 0 and 100. No output for any other ***x*** value is defined. In this case, the set of the domain is:

\begin{equation}\{0,100\}\end{equation}

Note that this does not include all real numbers, it only includes 0 and 100.

When we use Python to plot this function, note that it only makes sense to plot a scatter plot showing the individual values returned, there is no line in between because the function is not continuous between the values within the domain.

```R
k = function(x){
    ## Use vectorized ifelse to return the value or the missing value, NA
    ifelse(x == 0, x, ifelse(x ==100, x, NA))
}

## Construct the data frame.
df5 = data.frame(x = seq(-100,100))
df5$y = k(df4$x) ## Call g(x) with the vector df2$x

## Make the plot
suppressWarnings( # Suppress the warnings from attempts to plot NAs
    ggplot(df5, aes(x,y)) + geom_point(color = 'magenta', size = 2) +
           xlim(-1,101) + # Limit the range of x values displayed
           xlab('x') + ylab('k(x)')
    )
```

### Range of a Function
Just as the domain of a function defines the set of values for which the function is defined, the *range* of a function defines the set of possible outputs from the function.

For example, consider the following function:

\begin{equation}p(x) = x^{2} + 1\end{equation}

The domain of this function is all real numbers. However, this is a quadratic function, so the output values will form a parabola; and since the function has no negative coefficient or constant, it will be an upward opening parabola with a vertex that has a y value of 1.

So what does that tell us? Well, the minimum value that will be returned by this function is 1, so it's range is:

\begin{equation}\{p(x) \in \rm I\!R\;\;|\;\; p(x) \ge 1 \}\end{equation}

Let's create and plot the function for a range of ***x*** values in Python:

```R
p = function(x){x**2 + 1}

# Create an array of x values from -100 to 100
df6 = data.frame(x = seq(-100, 100))
df6$y = p(df$x)

# Plot the function
ggplot(df6, aes(x,y)) + geom_line(color = 'magenta', size = 1) +
       xlab('x') + ylab('p(x)')
```

Note that the ***p(x)*** values in the plot drop exponentially for ***x*** values that are negative, and then rise exponentially for positive ***x*** values; but the minimum value returned by the function (for an *x* value of 0) is **1**.


```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
matplotlib.rcParams['figure.figsize'] = (10.0, 8.0)
```

# Phase plane analysis of dynamical systems
When we have a system of two differential equations in two variables, $\mathbf{x}' = \mathbf{f}(\mathbf{x})$ we have a very useful analysis technique available to us, that of the *phase plane*. The phase plane is a two-dimensional plot with the two state variables of the ODE system on the axes. Each point on the plane corresponds to a particular choice of $\mathbf{x}$ at which we can compute a velocity vector $\mathbf{x}'$ and plot it on the graph (if we like) as an arrow originating at $\mathbf{x}$.

### Example: Fitzhugh-Nagumo equations
We're going to use the Fitzhugh-Nagumo equations as our example:
\begin{equation}
  \begin{split}
     \epsilon \frac{dv}{dt} &= f(v) - w + I_{\text{app}} \\
     \frac{dw}{dt} &= v - \gamma w
  \end{split}
\end{equation}
where
\begin{equation*}
  f(v) = v(1 - v)(v - \alpha)
\end{equation*}
and $0 < \alpha < 1$, $\epsilon \ll 1$. We'll use $\alpha = 0.1$, $\gamma = 0.5$, and $\epsilon = 0.01$ for this example.

Below we're going to define our equations in code so we can use them to illustrate various things about the system.

```python
# Parameters
alpha = 0.1
gamma = 0.5
epsilon = 0.1
I_app = 0

# Define the f(v) function separately
def f(v):
    return v*(1 - v)*(v - alpha)

# Fitzhugh-Nagumo equations - these will pick up the parameter values above
def fhn(x):
    v, w = x[0], x[1]
    dv = 1/epsilon*(f(v) - w + I_app)
    dw = v - gamma*w
    return array([dv, dw])

def plot_vector_field():
    V, W = meshgrid(np.linspace(-1, 2, 20), np.linspace(-1, 2, 20))
    dV, dW = fhn([V, W])
    plt.quiver(V, W, dV, dW)
    plt.title('Vector field for Fitzhugh-Nagumo equations', fontsize=16)
    plt.xlabel('$v$', fontsize=16)
    plt.ylabel('$w$', fontsize=16)

plot_vector_field()
```

If we simulate our system numerically for a given choice of initial conditions. e.g. $v(0) = -1$, $w(0) = -0.5$, we can plot the resulting $v$, $w$ against each other to give a *trajectory*

```python
from scipy.integrate import odeint
# initial conditions and simulation time

v0, w0 = -1, -0.5
t = np.linspace(0, 10, 2000)
def compute_trajectory(v0, w0, t):
    # scipy's odeint does the heavy lifting for us. Its ode function requires an additional
    # t input, so that's what the lambda is doing. We could have just added this to fhn at the
    # start
    X = odeint(lambda x, t: fhn(x), [v0, w0], t)

    # Solution comes out as a big array with two columns, so pull v and w out
    v, w = X[:, 0], X[:, 1]
    return [v, w]

plot_vector_field()
v, w = compute_trajectory(v0, w0, t)
plt.plot(v, w)
```

Note that the trajectory is tangent to the vector field at all points.

## Nullclines
The first thing we do in the phase plane method is to compute the *nullclines* of our system. These are the curves were either $\frac{dv}{dt} = 0$ or $\frac{dw}{dt} = 0$. For our example:
\begin{equation*}
  \frac{dv}{dt} = 0 \implies w = f(v) + I_{\text{app}} = 0
\end{equation*}
and
\begin{equation*}
  \frac{dw}{dt} = 0 \implies w = \frac{1}{\gamma} v.
\end{equation*}

Referring to our example, the curve $\frac{dv}{dt} = 0$ corresponds to all points where the velocity vector has no horizontal component, trajectories can only cross this nullcline vertically. Moreover, the sign of $\frac{dv}{dt}$ is fixed on either side of the nullcline, as the sign cannot change without passing back through it. For our example, $\frac{dv}{dt}$ is *positive* when we are below the nullcline, meaning trajectories move right when we are beneath it, and *negative* when we are above the nullcline, meaning trajectories will move left.

```python
def null_v(v):
    return f(v) + I_app
def null_w(v):
    return 1/gamma*v
v_trajectory, w_trajectory = compute_trajectory(v0, w0, t)

v = np.linspace(-1, 2, 200)
plot_vector_field()
plt.plot(v, null_v(v), label='dv/dt = 0')
plt.plot(v, null_w(v), label='dw/dt=0')

plt.plot(v_trajectory, w_trajectory, label='sample trajectory')
plt.ylim([-1, 2])

text(-1, -0.2, 'dv/dt > 0 \ndw/dt < 0', fontsize=16, bbox=dict(alpha=1, facecolor='white'))
text(0.5, -0.8, 'dv/dt > 0 \ndw/dt > 0', fontsize=16, bbox=dict(alpha=1, facecolor='white'))
text(1.5, 1, 'dv/dt < 0 \ndw/dt > 0', fontsize=16, bbox=dict(alpha=1, facecolor='white'))
text(-0.1, 1.5, 'dv/dt < 0 \ndw/dt < 0', fontsize=16, bbox=dict(alpha=1, facecolor='white'))
plt.legend(fontsize=16)
```

You can see at this point that we can already understand quite a bit about how the system behaves. For our given trajectory, it starts in a regime where it is moving right and downward. When the trajectory crosses the $w$ nullcline (orange), it begins to move upward, but continues moving right until it hits the $v$ nullcline(blue). It crosses this nullcline vertically, and then begins to move left, still moving upward. It starts to move down after crossing the orange nullcline again, and then gradually spirals into the origin.

This simple piece of analysis (figuring out the nullclines) already gives us a lot of qualitative insight into how the ODE system behaves. Now, the next thing we have to do is look at the *steady states*, the points where the nullclines intersect.

## Steady states
The next thing that we want to do is to understand what kind of long term behaviour the system can exhibit. For our example we see that the trajectory we plotted spirals into the origin and seems to stay there. The origin is an example of what we call a *stable steady state*, i.e. is a point $\mathbf{x}^\star$ satisfying
\begin{equation}
    \mathbf{f}(\mathbf{x}^*) = \mathbf{0}.
\end{equation}
and for which trajectories starting near $\mathbf{x}^\star$ do not depart a small neighbourhood of it.

Some steady states are stable, some are unstable, and some look like a mixture. We're going to take a small digression and look at linear systems now.

## Digression: Linear autonomous systems
Forget our complicated example for a second, we're going to consider some simpler systems. Consider the system
\begin{equation}
    \mathbf{x}' = A \mathbf{x}
\end{equation}
where $A$ is a 2x2 matrix. You may recall from your earlier studies that assuming $A$ has two linearly independent eigenvectors that the solution to this system is
\begin{equation}
    \mathbf{x}(t) = c_1 \mathbf{v}_1 \exp(\lambda_1 t) + c_2 \mathbf{v}_2 \exp(\lambda_2 t)
\end{equation}
where $\lambda_1, \lambda_2$ are the eigenvalues of $A$ (may be duplicate), and $\mathbf{v}_1, \mathbf{v}_2$ are the two corresponding eigenvectors. The constants are found by considering the initial conditions:
\begin{equation*}
    \begin{bmatrix} x_1(0) \\ x_2(0) \end{bmatrix} =
    c_1 \mathbf{v_1} + c_2 \mathbf{v}_2
\end{equation*}
i.e. $c_1, c_2$ are the coefficients of $\mathbf{v}_1$ and $\mathbf{v}_2$ when expressing the initial condition using the eigenvectors as a basis.

If the eigenvectors are real, note that if $\mathbf{x}(0)$ is lined up with an eigenvector, then the other coefficient will be zero, and the trajectory will track this eigenvector into the origin (if $\lambda < 0$) or out to infinity (if $\lambda > 0$) or just stay put (if $\lambda = 0$).

There are a few typical cases that we frequently see (we'll ignore the zero real part possibilities):

### Two negative real eigenvalues: stable node
Considering the form of the solution, it is evident that if both eigenvalues have negative real part that $\mathbf{x}(t) \to \mathbf{0}$ as $t \to \infty$. In this case, the system is *stable* and we call the origin a *stable node*

### Two positive real eigenvalues: unstable node
This situation is the opposite of the previous one, no matter the initial condition the exponentials in the solution will blow up and the long term solution is infinite. We say the origin is an *unstable node*

### One positive, one negative real eigenvalue: saddle node
This one is interesting. If we start on the eigenvector corresponding to the negative eigenvalue, then the trajectory will hone in on the origin. This eigenvector is called the *stable manifold* of the steady state, the set of points for which the steady state is stable.

If we start anywhere else, the coefficient of the positive eigenvalue term will be nonzero and so eventually the positive exponential term will dominate.

### Complex (conjugate) eigenvalues, negative real parts: stable spiral
If $\lambda = a \pm ib$ with $a < 0$, then the solution takes the form
\begin{equation*}
    \mathbf{x}(t) = \exp(at)\left(\mathbf{u}_1 \cos bt + \mathbf{u}_2 \sin bt\right)
\end{equation*}
In this case the solution oscillates with decaying amplitude, and so in the phase plane it spirals into the origin. We call this a *stable spiral*.

### Complex (conjugate) eigenvalues, positive real parts: unstable spiral
Same as previously, except the exponential term corresponds to a growing exponential, and so trajectories spiral away from the origin. We call this an *unstable spiral*.

## Back to steady states: Hartman-Grobman Theorem
So why did we look at linear systems? Well, turns out that if we approximate our nonlinear system by a linear system near around a steady state, then due to a cool theorem called the *Hartman-Grobman Theorem*, that the nonlinear system is guaranteed to behave qualitatively the same as the linear system in some neighbourhood of the fixed point, so long as the real parts of the eigenvalues are nonzero.

### Approximating our nonlinear system by a linear one
So, how do we do this? Well, we can expand our system around the steady state in a 2D Taylor series. Let's assume $\mathbf{x}^\star$ is a steady state, i.e. $\mathbf{f}(\mathbf{x}^\star) = \mathbf{0}$. Then
\begin{align*}
    \mathbf{f}(\mathbf{x} - \mathbf{x}^\star) &= \mathbf{f}(\mathbf{x}^\star) + J(\mathbf{x}^\star)(\mathbf{x} - \mathbf{x}^\star) + \mathcal{O}(\lVert \mathbf{x} - \mathbf{x}^\star \rVert^2) \\
    &= J(\mathbf{x}^\star)(\mathbf{x} - \mathbf{x}^\star) + \mathcal{O}(\lVert \mathbf{x} - \mathbf{x}^\star \rVert^2)
\end{align*}
where $J(\mathbf{x})$ is the Jacobian of the system, evaluated at $\mathbf{x}$.

So, defining new coordinates $\mathbf{u} = \mathbf{x} - \mathbf{x}^\star$ so that the origin in $\mathbf{u}$-space is the steady state, we have that
\begin{align*}
    \mathbf{u}' &= J(\mathbf{x}^\star)\mathbf{u} + \text{higher order terms} \\
    &\approx J(\mathbf{x}^\star)\mathbf{u}.
\end{align*}
This approximation is a linear system, and so long as the eigenvalues of $J(\mathbf{x}^\star)$ have nonzero real parts, this approximation behaves the same way as the nonlinear system near the steady state.

### Fitzhugh-Nagumo equations
So let's look at the steady state of the Fitzhugh-Nagumo equations. We can see that the only steady state is at the origin by inspection, so we don't need to do any fancy variable translations. We need the Jacobian. If $\mathbf{x} = \begin{bmatrix} v & w \end{bmatrix}^T$, and express our system as
\begin{equation}
  \begin{split}
     \frac{dv}{dt} &= \frac{1}{\epsilon}\left(f(v) - w + I_{\text{app}}\right) = g(v, w) \\
     \frac{dw}{dt} &= v - \gamma w = h(v, w)
  \end{split}
\end{equation}

\begin{equation*}
    J(\mathbf{0}) = \begin{bmatrix}
        \frac{\partial g}{\partial v} & \frac{\partial g}{\partial w} \\
        \frac{\partial h}{\partial v} & \frac{\partial h}{\partial w}
    \end{bmatrix}(\mathbf{0}) =
    \begin{bmatrix}
        \frac{1}{\epsilon}\left(-3v^2 + 2(\alpha + 1)v - \alpha\right) & -\frac{1}{\epsilon} \\
        1 & -\gamma
    \end{bmatrix}(\mathbf{0}) = \begin{bmatrix}
    -\frac{\alpha}{\epsilon} & -\frac{1}{\epsilon} \\
    1 & -\gamma
    \end{bmatrix}
\end{equation*}

For our choice of parameters, this matrix has eigenvalues $\lambda \approx -5.25 \pm 8.8i$, which makes the steady state of the linear system, and hence locally the nonlinear system, a *stable spiral*.


```python
import numpy as np
import matplotlib.pyplot as plt
```

# Lab 6 Monte Carlo Methods

## Introduction: Random numbers and statistics

### Normal distribution

A nomral distribution can be generated by `numpy.eandom.normal`
See details [here](https://docs.scipy.org/doc/numpy-1.15.0/reference/generated/numpy.random.normal.html).

The probability density function of the normal distribution follows a Gaussian function
\begin{equation}
    f(x|\mu,\sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{(x-\mu)^2}{2\sigma^2}}
\end{equation}

```python
# mean
mu = 0
# Standard deviation
sigma = 1
# size of variables of the normal distribution, n can beint or tuple
# depending on if x is a vector, or a matrix, etc.
n = 10000
# n = (100, 100)
# Normal distribution
x_normal = np.random.normal(mu, sigma, n)
# plot
fig, axs = plt.subplots(2, 1, figsize=(8, 8))
axs[0].plot(x_normal, 'ro')
axs[0].set_title(r'Normal Distribution with $\mu=$'+str(mu)+' and $\sigma=$'+str(sigma))

count, bins, ignored = axs[1].hist(x_normal, 100, density=True)
axs[1].plot(bins, 1/(sigma * np.sqrt(2 * np.pi)) *
            np.exp( - (bins - mu)**2 / (2 * sigma**2) ),
            linewidth=2, color='r')
axs[1].set_title('Probability Density')

for i in range(2): axs[i].autoscale(enable=True, axis='both', tight=True)
```

### Uniform distribution

Similarly, a uniform distribution can be constructed by `numpy.random.uniform`.
See [here](https://docs.scipy.org/doc/numpy-1.15.0/reference/generated/numpy.random.uniform.html).

```python
# the half-open interval [low, high)
low, high = 0, 1
# size of variables of the normal distribution, n can beint or tuple
# depending on if x is a vector, or a matrix, etc.
n = 500
# uniform distribution
x_uni = np.random.uniform(low, high, n)

# plot
fig, axs = plt.subplots(2, 1, figsize=(8, 8))
axs[0].plot(x_uni, 'ro')
axs[0].set_title(r'Uniform Distribution in $[$'+str(low)+','+str(high)+r'$)$')

count, bins, ignored = axs[1].hist(x_uni, 100, density=True)
axs[1].plot(bins, np.ones_like(bins), linewidth=2, color='r')
axs[1].set_title('Probability Density')
for i in range(2): axs[i].autoscale(enable=True, axis='both', tight=True)
```

## A simulation: Throw dice

The uniform distribution random function `numpy.random.uniform` generate real numbers.
In order to create only integers, one need to round the random value to the nearest integer with `numpy.floor` or `numpy.ceil`.

Define the following function to simulate dice throw

```python
def throwDice(N):
    '''
    To simulate throwing dice N times
    '''
    return np.floor(1 + 6*np.random.uniform(0, 1, size=N))
```

```python
N = 1000
Nrepeat = 10000
r = [np.mean(throwDice(N)) for i in range(Nrepeat)]

# plot
fig, ax = plt.subplots(1, 1, figsize=(8, 4))
ax.hist(r, 100, density=True)
ax.autoscale(enable=True, axis='both', tight=True)
```

__NOTE__: There is a random integers generator in `numpy.random.randint` which create discrete uniform random integers from _low (inclusive)_ to __high (exclusive)__.
See [here](https://docs.scipy.org/doc/numpy-1.15.0/reference/generated/numpy.random.randint.html).

```python
# Range of the dice
# NOTE!! high value is not included, for a dice with six sides, high=7
low, high = 1, 7
# Number of throws
n = 1000
x_dice = np.random.randint(low, high, n)

```

Then, we can simulate the same event with the following code

```python
N = 1000
Nrepeat = 10000
r = [np.mean(np.random.randint(low, high, n)) for i in range(Nrepeat)]

# plot
fig, ax = plt.subplots(1, 1, figsize=(8, 4))
ax.hist(r, 100, density=True)
ax.autoscale(enable=True, axis='both', tight=True)
```

## Application: Computing an integral with Monte Carlo

An example use Monte Carlo method to compute the integration of multivariate normal distribution funciton.

```python
def mcint(func, domain, N, M=30):
    """ Numerical integration using Monte Carlo method
    Parameters
    ----------
    func : function, function handler of the integrand;
    domain : numpy.ndarray, the domain of computation,
            domain = array([[-5, 5],
                            [-5, 5],
                            [-5, 5]])
            The dimensions of the domain is given by domain.shape[0];
    N : integer, the number of points in each realization;
    M : integer, the number of repetitions used for error estimation,
        (Recommendation, M = 30+).
        Total number of points used is thus M*N
    Returns
    -------
    r.mean() : the integral value of func in the domain
    r.std() : the error in the result (the standard deviation)
    """
    # Get the dimensions
    dim = domain.shape[0]
    # volume of the domain
    V = abs(domain.T[0] - domain.T[1]).prod()

    r = np.zeros(M)
    for i in range(M):
        # generate uniform distributed random numbers within the domain
        x = np.random.uniform(domain.T[0], domain.T[1], (N, dim))
        r[i] = V * np.mean(func(x), axis=0)

    return r.mean(), r.std()

def fnorm(x):
    """ Normal distribution function in d-dimensions
    Parameters
    ----------
    x : numpy.ndarray, of shape (N, d), where d is the dimension and
        N is the number of realizations
    Returns
    -------
    y : numpy.ndarray, of the shape (N, 1)
    """
    d = x.shape[1]
    y = 1/((2*np.pi)**(d/2))*np.exp(-0.5*np.sum(x**2, axis=1))
    return y
```

Take the domain in $[-4,4]\times[-4,4]$, compute the integral using funtion `fnorm`

```python
# numbers of samples
N = 1000
M = 50
# domain
domain = np.array([[-4,4],[-4,4]])
# integrate
intF, err = mcint(fnorm, domain, N, M)
print('The result of the integral with N=', str(N), 'is', '{:.5f}'.format(intF),',')
print('with standard deviation', '{:.5f}'.format(err), 'for', str(M), 'realizations.')
```

### Check the order of accuracy $p$ for the Monte Carlo method.

```python
# Change the dimension to see different results
dim = 3

# domain [-5, 5]^dim
domain = np.array([[-5, 5] for i in range(dim)])

# take an array of N
n = 8
NList = 500 * 2**np.array(range(n))
M = 50

# Save the error
errList = np.zeros_like(NList, dtype=float)
for i in range(n):
    intF, err = mcint(fnorm, domain, NList[i], M)
    errList[i] = err

# Plot
fig, ax = plt.subplots(1, 1, figsize=(8, 4))
ax.loglog(NList, errList)
ax.autoscale(enable=True, axis='both', tight=True)
ax.set_xlabel('N')
ax.set_ylabel('Error')

# Compute the order
a = np.polyfit(np.log(NList), np.log(errList),1)
p = np.round(a[0], 1)
print('Order of accuracy is N^p, with p=', p)
```

## Programming: Brownian motion

```python
def brownian(x0, tEnd, dt):
    """
    Generate an instance of Brownian motion
    Parameters
    ----------
    x0 : float or numpy array (or something that can be converted to a numpy array
         using numpy.asarray(x0)).
         The initial condition(s) (i.e. position(s)) of the Brownian motion.
    tEnd : float, the final time.
    dt : float, the time step.
    Returns
    -------
    x: A numpy array of floats with shape `x0.shape + (n,)`.
    """
    x0 = np.asarray(x0)
    n = int(tEnd/dt)

    # For each element of x0, generate a sample of n numbers from a
    # normal distribution.
    r = np.random.normal(size=x0.shape + (n,), scale=(dt**0.5))

    # This computes the Brownian motion by forming the cumulative sum of
    # the random samples.
    x = np.cumsum(r, axis=-1)

    # Add the initial condition.
    x += np.expand_dims(x0, axis=-1)

    return x
```

```python
# Total time.
T = 10.0
# Number of steps.
N = 500
# Time step size
dt = T/N
# Initial values of x.
x = np.empty((2,N+1))
x[:, 0] = 0.0

# Brownian motion
x[:, 1:] = brownian(x[:,0], T, dt)

# Plot the 2D trajectory.
fig, ax = plt.subplots(1, 1, figsize=(8, 8))
ax.plot(x[0],x[1])
ax.plot(x[0,0],x[1,0], 'go')
ax.plot(x[0,-1], x[1,-1], 'ro')
ax.set_title('2D Brownian Motion')
ax.set_xlabel('x')
ax.set_ylabel('y')
ax.axis('equal')
ax.grid(True)
```


```python
import numpy as np
import scipy.stats as stats
import scipy.special as special
```

### Binomial Distribution

Take the example of the flipping of a coin. Let us label the outcome as $\gamma$ which can take two values $0$ or $1$. We then define a parameter $\theta$ which will give us the probability of the outcome being $\gamma$. This is written as:

\begin{equation}
P\left(\gamma|\theta\right) = \theta^{\gamma}\left(1-\theta\right)^{1-\gamma}
\end{equation}

The above distribution is known as the Binomial distribution or the Bernoulli distribution. This particular distribution can also be inferred in a different manner. We can consider that $\gamma$ to be fixed by an observation and the parameter $\theta$ to be a variable. The above equation then gives the probability of getting $\gamma$ output for different values of $\theta$. In this scenario, this function is called the likelihood function of the parameter $\theta$

In Bayesian inference, $P\left(\gamma|\theta\right)$ is usually thought of with the data $\gamma$ being fixed and certain and the parameter $\theta$ to be a variable and uncertain. In this particular case, this is known as the Bernoulli likelihood function of $\theta$.

#### Fixed set of outcomes

Consider the case of multiple flips. Each flip can be considered independent of each other. The joint probability would be thus a product of all individual probabilities.

\begin{align}
P\left(\{\gamma_i\} |\theta\right) &=& \Pi_i p\left(\gamma_i|\theta\right)\\
                                   &=& \theta^{\Sigma_i \gamma_i}\left(1-\theta\right)^{\Sigma_i\left(1-\gamma_i\right)}\\
                                   &=& \theta^{z}\left(1-\theta\right)^{N-z}
\end{align}

where $N$ is the total number of tosses and $z = \Sigma_i\gamma_i$ is the number of heads

### Conjugate Priors

The Baye's rule is given by :

\begin{equation}
P\left(\gamma|\theta\right) = \frac{P\left(\theta|\gamma\right)P\left(\theta\right)}{\int d\theta' P\left(\theta'|\gamma\right)P\left(\theta'\right)}
\end{equation}

Bayesian statistics involve the idea of a prior and a posterior distribution of probabilities. In the Bayes's rule given above, $P\left(\theta\right)$ is the prior probability and the posterior probability upon the outcome of an experiment or an observation is given by the term on the left hand side $P\left(\gamma|\theta\right)$.

Ideally we can assume any form of function for the prior distribution as long as the outcome lies in the range $[0, 1]$. However it would be mathematically far more simpler if on multiplication by the likelihood function, the posterior distribution has the same functional form as the prior. This would indeed make future posterior probaility functions very easy to caluculate. It would also be helpful if the denominator $\int d\theta' P\left(\theta'|\gamma\right)P\left(\theta'\right)$ be analytically solvable. In the particular scenarios where the prior and posterior distributions has the same functional form, the posterior is said to be conjugate of the prior.

##### Note: A conjugate prior is related to the likelihood function under consideration. If you change the likelihood function will probably need a different conjugate prior.

### Conjugate function for a Bernoulli likelihood function

The outcome of a single coin toss or a multiple coin toss can be decsribed by the Bernoulli function as depicted in the previous function. Bernoulli function has the form $\theta^{\gamma}\left(1-\theta\right)^{1-\gamma}$. An appropriate prior function should look like $\theta^\left(a-1\right)\left(1-\theta\right)^\left(b-1\right)$ which when multiplied by the Bernoulli likelihood will have the same functional form.

The Beta distribution function given by
\begin{equation}
\beta\left(\theta, a, b\right) = \frac{\theta^\left(a-1\right)\left(1-\theta\right)^\left(b-1\right)}{B\left(a, b\right)}
\end{equation}

will be a possible conjugate prior to the Bernoulli likelihood function, here $B\left(a, b\right)$ is a normalizing factor so that the area under the integration goes to 1. In effect we will have the prior probaility as $P\left(\theta\right) = \beta\left(\theta, a, b\right)$

Also the normalizer $B\left(\theta\right)$ is given by:
\begin{equation}
B\left(a, b\right) = \int d\theta\text{ } \theta^a\left(1-\theta\right)^b
\end{equation}

which is the beta function.

#### Choosing a prior
The choice of a prior depends on the information available to us. For example, if we do not know anything about the coin that is about to be tossed, then probable we will assume that it is equally likely to have all values of the parameter $\theta$. That would correspond to $P\left(\theta\right) = \frac{\theta^a\left(1-\theta\right)^b}{\beta\left(a, b\right)}$ with $a=1, b=1$

```python
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
from scipy.stats import beta
```

```python
a, b = 1, 1
x = np.linspace(0, 1, 100)

```

```python
fig, ax = plt.subplots(1, 1)
ax.plot(x, beta.pdf(x, a, b)/special.beta(a,b),'r-', lw=5, alpha=0.6, label='a = {}, b = {}'.format(a,b))
ax.legend(loc = 'best')
ax.set_xlabel(r'$\theta$')
ax.set_ylabel(r'$P\left(\theta\right)$')
```

Let us look at the various shapes of priors we can choose from depending upn the previous information that might be provided to us:

```python
a,  b = [[0.1,1,2,3,4], [0.1,1,2,3,4]]
fig1, ax1 = plt.subplots(len(a), len(b), sharex = 'all', sharey = 'all', figsize = (12, 9))
for i in range(len(a)):
    for j in range(len(b)):
        ax1[i, j].plot(x, beta.pdf(x, a[i], b[j]),  label='a = {}, b = {}'.format(a[i],b[j]))
        ax1[i, j].legend(loc = 'best')
        ax1[i, j].set_xlabel(r'$\theta$')
        ax1[i, j].set_ylabel(r'$P\left(\theta|a,b\right)$')
        ax1[i, j].set_ylim(0,3)
fig1.tight_layout()

```

The above plots shows the various shape of the prior distribution functions (not normalized) that might be available to us depending on our experience and available information. In the figures shown above the total no. of tosses are $n = a + b$. We can see that for $a=4, b=4$ the distrinution is narrow around $\theta = 0.5$ compared to the case of $a=2, b = 2$. This means that with more information we can choose a prior that has more confidence.

### Some more properties of the beta distribution function

The choice of a prior is quite often dependent on the information that we have acquired and our own biases. For e.g. if we know that a coin has been minted in a government facility, then we would reasonably assume that the parameter $\theta $ is centred around $0.5$. Depending upon our confidence in the government minting facilty we could assume prior to performing any experiment that if we had performed 200 experiments then half of it would have turned up heads and the other half tails. That would correspond to a very narrow distribution of the probability $P\left(\theta|a,b\right)$ centered around $\theta = 0.5$. Thus it would be useful to know the central tendencies of this distribution (or as matter of fact any prior distribution one might assume.)

The mean of the distribution function $B\left(\theta|a, b\right)$ is :
\begin{equation}
\mu = \frac{a}{a+b}
\end{equation}
The mode is given by :

\begin{equation}
\omega = \frac{a-1}{a+b-2}
\end{equation}
for $a>1$ and $b>1$. As expected from the plots of the distribution function when $a=b$, both the mean and the mode are the same and equal to $0.5$.

Another important parameter is the spread of the beta function. This is denoted by $\kappa = a+b$. The higher the value of $\kappa$, narrower the spread of the distribution around the mean.

The following equations are useful to know the relation ships between the parameters and the choice of $a, b$.
\begin{align}
a & = & \mu\kappa\\
b & = & \left(1-\mu\right)\kappa\\
a & = & \omega\left(\kappa - 2\right)+1\\
b & = & \left(1 - \omega \right)\left(\kappa - 2\right)+1
\end{align}

In terms of standard deviation, $\sigma$ the parameters $a, b$ can be written as:

\begin{align}
a &=& \mu\left( \frac{\mu\left(1-\mu\right)}{\sigma^2} -1\right)\\
b &=& \left(1-\mu\right)\left( \frac{\mu\left(1-\mu\right)}{\sigma^2} -1\right)
\end{align}

### Choice of prior or how to choose $\kappa$

Suppose I borrow coins from a friend. I have the previous information from someone else that he is susceptible to give you biased coins with $\theta = 0.8$.  However you are not sure how much this information is reliable. You this want a few tosses to sway the distribution towards a correct one. This can be achieved by assuming a smaller value of $\kappa$. Choosing a very large value would $\kappa$ is advisable only if your source of information is really reliable. Otherwise it will take a large number of tosses to sway the model towards the correct distribution.

In order to see how distribution varies with $\kappa$ let us redefine a the beta function in terms of $\kappa, \mu, \omega \text{ and } \sigma$.

```python
def betaABfromMeanKappa(x, mean, kappa):
    a = mean*kappa
    b = (1-mean)*kappa
    return beta.pdf(x, a, b)
def betaABfromModeKappa(x, mode, kappa):
    a = mode*(kappa-2)+1
    b = (1-mode)*(kappa-2)+1
    return beta.pdf(x, a, b)
def betaABfromSDMean(x, sd, mean):
    a = mean*(mean*(1-mean)/sd**2-1)
    b = (1-mean)*(mean*(1-mean)/sd**2-1)
    return beta.pdf(x, a, b)
```

```python
mean, kappa = 0.8, 7.0
fig, ax = plt.subplots(1,1, figsize = (5,4))
ax.plot(x, betaABfromMeanKappa(x, mean, kappa), label = r'$\mu=${}, $\kappa=${}'.format(mean, kappa))
ax.set_xlabel(r'\theta')
ax.set_ylabel(r'$P\left(\theta\right)$')
ax.legend(loc = 'best')
```

```python
mode, kappa = 0.8, 7
fig, ax = plt.subplots(1,1, figsize = (5,4))
ax.plot(x, betaABfromModeKappa(x, mode, kappa), label = r'$\omega=${}, $\kappa=${}'.format(mode, kappa))
ax.set_xlabel(r'\theta')
ax.set_ylabel(r'$P\left(\theta\right)$')
ax.legend(loc = 'best')
```

### The Posterior beta distribution

The posterior distribution is obtained by applying the Bayes' rule to the prior distribution. Suppose we performed an experiment, consisting of $N$ coin flips out of which $z$ where heads. The likelihood function is thus given by:
\begin{equation}
P\left(z, N|\theta\right) = \theta^z\left(1-\theta\right)^\left(N-z\right)
\end{equation}
Let us assume a beta distribution prior given by:

\begin{equation}
P\left(\theta|a, b\right) = \frac{\theta^\left(a-1\right)\left(1-\theta\right)^\left(b-1\right)}{\beta\left(a, b\right)}
\end{equation}

Now let's start from the Bayes' rule:
\begin{align}
P\left(\theta|z,N\right) &=& \frac{P\left(z, N|\theta\right) P\left(\theta\right)}{P\left(z, N\right)}\\
                         &=& \frac{P\left(z, N|\theta\right) P\left(\theta\right)}{\int_{\theta '}d\theta 'P\left(z, N|\theta ' \right)P\left(\theta '\right)}\\
                         &=& \frac{\theta^{\left(z+a\right)-1}\left(1-\theta\right)^{\left(N-z+b\right)-1}}{\beta\left(z+a, N-z+b\right)}
\end{align}

The posterior distribution as we can see is also a beta distribution function. However we have new value for the parameters of the beta distribution function $a'= a+z, b'= N-z+b$, which changes the overall model.

### The significance of the posterior

Let us consider a prior such with $a=5$ and $b=5$. Now suppose we do $N = 1$ flips out of which the number of heads $z=1$. Let us look at the prior probability, likelihood function and the posterior probability. The prior is given by the function:

\begin{equation}
P\left(\theta|5, 5\right) = \frac{\theta^\left(5-1\right)\left(1-\theta\right)^\left(5-1\right)}{\beta\left(5, 5\right)}
\end{equation}

```python
#prior
a,b = 5, 5
fig, ax = plt.subplots(1, 1)
ax.plot(x, beta.pdf(x, a, b), 'b-',  label='a = {}, b = {}'.format(a,b))
ax.legend(loc = 'best', fontsize = 15)
ax.set_xlabel(r'$\theta$', fontsize = 15)
ax.set_ylabel(r'$dbeta\left(\theta|a,b\right)$', fontsize = 15)
ax.fill(x, beta.pdf(x, a, b), 'b')
ax.set_facecolor('0.8')

```

### Likelihood function
Now for the likelihood function let us assume that we performed and experiment where we had $N = 10$ flips and got only $z = 1$ heads. Thus we will have a Bernoulli likelihood given by:
\begin{equation}
P\left(D|\theta\right) = \theta\left(1-\theta\right)^9
\end{equation}

```python
def likelihood(theta, N, z):
    return ((theta)**z)*(1-theta)**(N-z)
fig, ax = plt.subplots(1, 1)
ax.plot(x, likelihood(x, 10, 1), 'b-', label = 'z = 1, N = 10')
ax.legend(loc = 'best', fontsize = 15)
ax.set_xlabel(r'$\theta$', fontsize = 15)
ax.set_ylabel(r'$P\left(D|\theta\right)$', fontsize = 15)
ax.fill(x, likelihood(x, 10, 1), 'b')
ax.set_facecolor('0.8')
```

### The posterior distribution
The posterior distribution can now be obtained from the function:
\begin{align}
P\left(\theta|z,N\right) &=& \frac{P\left(z, N|\theta\right) P\left(\theta\right)}{P\left(z, N\right)}\\
                         &=& \frac{P\left(z, N|\theta\right) P\left(\theta\right)}{\int_{\theta '}d\theta 'P\left(z, N|\theta ' \right)P\left(\theta '\right)}\\
                         &=& \frac{\theta^{\left(z+a\right)-1}\left(1-\theta\right)^{\left(N-z+b\right)-1}}{\beta\left(z+a, N-z+b\right)}
\end{align}

Replacing $a=5, b=5$ and $z=1, N=10$ we will have :
\begin{equation}
P\left(\theta|z=1,N=10\right) = \frac{\theta^{\left(6-1\right)}\left(1-\theta\right)^{14-1}}{\beta\left(6,14\right)}
\end{equation}

```python
a,b = 6, 14
fig, ax = plt.subplots(1, 1)
ax.plot(x, beta.pdf(x, a, b), 'b-',  label='a = {}, b = {}'.format(a,b))
ax.legend(loc = 'best', fontsize = 15)
ax.set_xlabel(r'$\theta$', fontsize = 15)
ax.set_ylabel(r'$dbeta\left(\theta|a,b\right)$', fontsize = 15)
ax.fill(x, beta.pdf(x, a, b), 'xkcd:blue')
ax.set_facecolor('0.8')

```

### Prior knowledge that cannot be expressed as a beta distribution

Suppose a company specializes in manufacturing coins of two types - one that has a probability of heads being 25% and the other having probability of head being 75 %. Or prior model thus has a distribution that is bimodal with peaks around 0.25 and 0.75. We can construct to be composed of two Gaussians with peaks at 0.25 and 0.75 respectively.

```python
# Creating the prior
def prior(x):
    return  (np.exp(-(x - 0.25)**2/.002)/np.sqrt(2*np.pi*.001) + np.exp(-(x - 0.75)**2/.002)/np.sqrt(2*np.pi*.001))/2
```

```python
x =  np.linspace(0, 1, 1000)
```

```python
fig, ax = plt.subplots(1, 1)
ax.plot(x, prior(x), 'b-',  label='Double Peaked Prior Distribution')
ax.legend(loc = 'best', fontsize = 10)
ax.set_xlabel(r'$\theta$', fontsize = 15)
ax.set_ylabel(r'$P\left(\theta\right)$', fontsize = 15)
ax.fill(x, prior(x), 'xkcd:blue')
ax.set_facecolor('lightsteelblue')
```

Let us check whether our distrbution integrates to unity or not when intergrated all over the theta space as it should.

```python
from scipy.integrate import simps
simps(prior(x),x)
```

    0.9999999999999987

Which is close to unity so this prior distribution is valid. Now suppose we flip coins 27 times and we get 14 heads and 13 tails. The likelihood function is thus given by:

```python
fig, ax = plt.subplots(1, 1)
ax.plot(x, likelihood(x, 27, 14), 'b-', label = 'z = {}, N = {}'.format(27, 14) )
ax.legend(loc = 'best', fontsize = 10)
ax.set_xlabel(r'$\theta$', fontsize = 15)
ax.set_ylabel(r'$P\left(D|\theta\right)$', fontsize = 15)
ax.fill(x, likelihood(x, 27, 14), 'b')
ax.set_facecolor('0.8')
```

Now we will calculate the posterior distribution:

```python
fig, ax = plt.subplots(1, 1)
ax.plot(x, prior(x)*likelihood(x, 27, 14)/simps(prior(x)*likelihood(x, 27, 14),x), 'r-',  label='Posterior Distribution')
ax.plot(x, prior(x), 'b-',  label='Prior Distribution')
ax.legend(loc = 'best', fontsize = 10)
ax.set_xlabel(r'$\theta$', fontsize = 15)
ax.set_ylabel(r'$P\left(\theta\right)$', fontsize = 15)
ax.fill(x, prior(x)*likelihood(x, 27, 14)/simps(prior(x)*likelihood(x, 27, 14),x), 'xkcd:orange' )
ax.fill(x, prior(x), 'xkcd:blue', alpha = 0.8)
ax.set_facecolor('lightsteelblue')
```

As can be seen the posterior distribution has been shifted inwards when compared to the prior distribution. This makes sense as the experiment performed has nearly equal number of heads and tails which comes up, making the central tendency of the theta distribution move towards a central peak at $theta = 0.5$. However that would need a really large number of experiments as we have a relatively strong prior. We can do that above calulcation using a relatively weak prior and see what happens

```python
# Creating the prior
def prior_weak(x):
    return  (np.exp(-(x - 0.25)**2/.02)/np.sqrt(2*np.pi*.01) + np.exp(-(x - 0.75)**2/.02)/np.sqrt(2*np.pi*.01))/2
```

```python
fig, ax = plt.subplots(1, 1)
ax.plot(x, prior_weak(x), 'b-',  label='Double Peaked Prior Distribution')
ax.legend(loc = 'best', fontsize = 10)
ax.set_xlabel(r'$\theta$', fontsize = 15)
ax.set_ylabel(r'$P\left(\theta\right)$', fontsize = 15)
ax.fill(x, prior_weak(x), 'xkcd:blue')
ax.set_facecolor('lightsteelblue')
```

Let's us calculate the posterior:

```python
fig, ax = plt.subplots(1, 1)
ax.plot(x, prior_weak(x)*likelihood(x, 27, 14)/simps(prior(x)*likelihood(x, 27, 14),x), 'r-',  label='Posterior Distribution')
ax.plot(x, prior_weak(x), 'b-',  label='Prior Distribution')
ax.legend(loc = 'best', fontsize = 10)
ax.set_xlabel(r'$\theta$', fontsize = 15)
ax.set_ylabel(r'$P\left(\theta\right)$', fontsize = 15)
ax.fill(x, prior_weak(x)*likelihood(x, 27, 14)/simps(prior(x)*likelihood(x, 27, 14),x), 'xkcd:orange' )
ax.fill(x, prior_weak(x), 'xkcd:blue', alpha = 0.8)
ax.set_facecolor('lightsteelblue')
```

```python

```


# Numerical integrals

What about when we cannot integrate a function analytically? In other words, when there is no (obvious) closed-form solution. In these cases, we can use **numerical methods** to solve the problem.

Let's use this problem:
\begin{align}
\frac{dy}{dx} &= e^{-x^2} \\
y(x) &= \int e^{-x^2} dx + C
\end{align}

(You may recognize this as leading to the error function, $\text{erf}$:
$\frac{1}{2} \sqrt{\pi} \text{erf}(x) + C$,
so the exact solution to the integral over the range $[0,1]$ is 0.7468.)

```matlab
x = linspace(0, 1);
f = @(x) exp(-x.^2);
plot(x, f(x))
axis([0 1 0 1])
```

## Numerical integration: Trapezoidal rule

In such cases, we can find the integral by using the **trapezoidal rule**, which finds the area under the curve by creating trapezoids and summing their areas:
\begin{equation}
\text{area under curve} = \sum \left( \frac{f(x_{i+1}) + f(x_i)}{2} \right) \Delta x
\end{equation}

Let's see what this looks like with four trapezoids ($\Delta x = 0.25$):

```matlab
hold off
x = linspace(0, 1);
plot(x, f(x)); hold on
axis([0 1 0 1])

x = 0 : 0.25 : 1;

% plot the trapezoids
for i = 1 : length(x)-1
    xline = [x(i), x(i)];
    yline = [0, f(x(i))];
    line(xline, yline, 'Color','red','LineStyle','--')
    xline = [x(i+1), x(i+1)];
    yline = [0, f(x(i+1))];
    line(xline, yline, 'Color','red','LineStyle','--')
    xline = [x(i), x(i+1)];
    yline = [f(x(i)), f(x(i+1))];
    line(xline, yline, 'Color','red','LineStyle','--')
end
hold off
```

Now, let's integrate using the trapezoid formula given above:

```matlab
dx = 0.1;
x = 0.0 : dx : 1.0;

area = 0.0;
for i = 1 : length(x)-1
    area = area + (dx/2)*(f(x(i)) + f(x(i+1)));
end

fprintf('Numerical integral: %f\n', area)
exact = 0.5*sqrt(pi)*erf(1);
fprintf('Exact integral: %f\n', exact)
fprintf('Error: %f %%\n', 100.*abs(exact-area)/exact)
```

    Numerical integral: 0.746211
    Exact integral: 0.746824
    Error: 0.082126 %

We can see that using the trapezoidal rule, a numerical integration method, with an internal size of $\Delta x = 0.1$ leads to an approximation of the exact integral with an error of 0.08%.

You can make the trapezoidal rule more accurate by:

- using more segments (that is, a smaller value of $\Delta x$, or
- using higher-order polynomials (such as with Simpson's rules) over the simpler trapezoids.

First, how does reducing the segment size (step size) by a factor of 10 affect the error?

```matlab
dx = 0.01;
x = 0.0 : dx : 1.0;

area = 0.0;
for i = 1 : length(x)-1
    area = area + (dx/2)*(f(x(i)) + f(x(i+1)));
end

fprintf('Numerical integral: %f\n', area)
exact = 0.5*sqrt(pi)*erf(1);
fprintf('Exact integral: %f\n', exact)
fprintf('Error: %f %%\n', 100.*abs(exact-area)/exact)
```

    Numerical integral: 0.746818
    Exact integral: 0.746824
    Error: 0.000821 %

So, reducing our step size by a factor of 10 (using 100 segments instead of 10) reduced our error by a factor of 100!

## Numerical integration: Simpson's rule

We can increase the accuracy of our numerical integration approach by using a more sophisticated interpolation scheme with each segment. In other words, instead of using a straight line, we can use a polynomial. **Simpson's rule**, also known as Simpson's 1/3 rule, refers to using a quadratic polynomial to approximate the line in each segment.

Simpson's rule defines the definite integral for our function $f(x)$ from point $a$ to point $b$ as
\begin{equation}
\int_a^b f(x) \approx \frac{1}{6} \Delta x \left( f(a) + 4 f \left(\frac{a+b}{2}\right) + f(b) \right)
\end{equation}
where $\Delta x = b - a$.

That equation comes from interpolating between points $a$ and $b$ with a third-degree polynomial, then integrating by parts.

```matlab
hold off
x = linspace(0, 1);
plot(x, f(x)); hold on
axis([-0.1 1.1 0.2 1.1])

plot([0 1], [f(0) f(1)], 'Color','black','LineStyle',':');

% quadratic polynomial
a = 0; b = 1; m = (b-a)/2;
p = @(z) (f(a).*(z-m).*(z-b)/((a-m)*(a-b))+f(m).*(z-a).*(z-b)/((m-a)*(m-b))+f(b).*(z-a).*(z-m)/((b-a)*(b-m)));
plot(x, p(x), 'Color','red','LineStyle','--');

xp = [0 0.5 1];
yp = [f(0) f(m) f(1)];
plot(xp, yp, 'ok')
hold off
legend('exact', 'trapezoid fit', 'polynomial fit', 'points used')
```

We can see that the polynomial fit, used by Simpson's rule, does a better job of of approximating the exact function, and as a result Simpson's rule will be more accurate than the trapezoidal rule.

Next let's apply Simpson's rule to perform the same integration as above:

```matlab
dx = 0.1;
x = 0.0 : dx : 1.0;

area = 0.0;
for i = 1 : length(x)-1
    area = area + (dx/6.)*(f(x(i)) + 4*f((x(i)+x(i+1))/2.) + f(x(i+1)));
end

fprintf('Simpson rule integral: %f\n', area)
exact = 0.5*sqrt(pi)*erf(1);
fprintf('Exact integral: %f\n', exact)
fprintf('Error: %f %%\n', 100.*abs(exact-area)/exact)
```

    Simpson rule integral: 0.746824
    Exact integral: 0.746824
    Error: 0.000007 %

Simpson's rule is about three orders of magnitude (~1000x) more accurate than the trapezoidal rule.

In this case, using a more-accurate method allows us to significantly reduce the error while still using the same number of segments/steps.


### Example 4: Burgers' equation

Now that we have seen how to construct the non-linear convection and diffusion examples, we can combine them to form Burgers' equations. We again create a set of coupled equations which are actually starting to form quite complicated stencil expressions, even if we are only using a low-order discretisations.

Let's start with the definition fo the governing equations:
$$ \frac{\partial u}{\partial t} + u \frac{\partial u}{\partial x} + v \frac{\partial u}{\partial y} = \nu \; \left(\frac{\partial ^2 u}{\partial x^2} + \frac{\partial ^2 u}{\partial y^2}\right)$$

$$ \frac{\partial v}{\partial t} + u \frac{\partial v}{\partial x} + v \frac{\partial v}{\partial y} = \nu \; \left(\frac{\partial ^2 v}{\partial x^2} + \frac{\partial ^2 v}{\partial y^2}\right)$$

The discretized and rearranged form then looks like this:

\begin{aligned}
u_{i,j}^{n+1} &= u_{i,j}^n - \frac{\Delta t}{\Delta x} u_{i,j}^n (u_{i,j}^n - u_{i-1,j}^n) - \frac{\Delta t}{\Delta y} v_{i,j}^n (u_{i,j}^n - u_{i,j-1}^n) \\
&+ \frac{\nu \Delta t}{\Delta x^2}(u_{i+1,j}^n-2u_{i,j}^n+u_{i-1,j}^n) + \frac{\nu \Delta t}{\Delta y^2} (u_{i,j+1}^n - 2u_{i,j}^n + u_{i,j+1}^n)
\end{aligned}

\begin{aligned}
v_{i,j}^{n+1} &=  v_{i,j}^n - \frac{\Delta t}{\Delta x} u_{i,j}^n (v_{i,j}^n - v_{i-1,j}^n) - \frac{\Delta t}{\Delta y} v_{i,j}^n (v_{i,j}^n - v_{i,j-1}^n) \\
&+ \frac{\nu \Delta t}{\Delta x^2}(v_{i+1,j}^n-2v_{i,j}^n+v_{i-1,j}^n) + \frac{\nu \Delta t}{\Delta y^2} (v_{i,j+1}^n - 2v_{i,j}^n + v_{i,j+1}^n)
\end{aligned}

Great. Now before we look at the Devito implementation, let's re-create the NumPy-based implementation form the original.

```python
from examples.cfd import plot_field, init_hat
import numpy as np
%matplotlib inline

# Some variable declarations
nx = 41
ny = 41
nt = 120
c = 1
dx = 2. / (nx - 1)
dy = 2. / (ny - 1)
sigma = .0009
nu = 0.01
dt = sigma * dx * dy / nu
```

```python
#NBVAL_IGNORE_OUTPUT

# Assign initial conditions
u = np.empty((nx, ny))
v = np.empty((nx, ny))

init_hat(field=u, dx=dx, dy=dy, value=2.)
init_hat(field=v, dx=dx, dy=dy, value=2.)

plot_field(u)
```

```python
#NBVAL_IGNORE_OUTPUT

for n in range(nt + 1): ##loop across number of time steps
    un = u.copy()
    vn = v.copy()

    u[1:-1, 1:-1] = (un[1:-1, 1:-1] -
                     dt / dx * un[1:-1, 1:-1] *
                     (un[1:-1, 1:-1] - un[1:-1, 0:-2]) -
                     dt / dy * vn[1:-1, 1:-1] *
                     (un[1:-1, 1:-1] - un[0:-2, 1:-1]) +
                     nu * dt / dx**2 *
                     (un[1:-1,2:] - 2 * un[1:-1, 1:-1] + un[1:-1, 0:-2]) +
                     nu * dt / dy**2 *
                     (un[2:, 1:-1] - 2 * un[1:-1, 1:-1] + un[0:-2, 1:-1]))

    v[1:-1, 1:-1] = (vn[1:-1, 1:-1] -
                     dt / dx * un[1:-1, 1:-1] *
                     (vn[1:-1, 1:-1] - vn[1:-1, 0:-2]) -
                     dt / dy * vn[1:-1, 1:-1] *
                    (vn[1:-1, 1:-1] - vn[0:-2, 1:-1]) +
                     nu * dt / dx**2 *
                     (vn[1:-1, 2:] - 2 * vn[1:-1, 1:-1] + vn[1:-1, 0:-2]) +
                     nu * dt / dy**2 *
                     (vn[2:, 1:-1] - 2 * vn[1:-1, 1:-1] + vn[0:-2, 1:-1]))

    u[0, :] = 1
    u[-1, :] = 1
    u[:, 0] = 1
    u[:, -1] = 1

    v[0, :] = 1
    v[-1, :] = 1
    v[:, 0] = 1
    v[:, -1] = 1

plot_field(u)
```

Nice, our wave looks just like the original. Now we shall attempt to write our entire Burgers' equation operator in a single cell - but before we can demonstrate this, there is one slight problem.

The diffusion term in our equation requires a second-order space discretisation on our velocity fields, which we set through the `TimeFunction` constructor for $u$ and $v$. The `TimeFunction` objects will store this dicretisation information and use it as default whenever we use the shorthand notations for derivative, like `u.dxl` or `u.dyl`. For the advection term, however, we want to use a first-order discretisation, which we now have to create by hand when combining terms with different stencil discretisations. To illustrate let's consider the following example:

```python
from devito import Grid, TimeFunction, first_derivative, left

grid = Grid(shape=(nx, ny), extent=(2., 2.))
x, y = grid.dimensions
t = grid.stepping_dim

u1 = TimeFunction(name='u1', grid=grid, space_order=1)
print("Space order 1:\n%s\n" % u1.dxl)

u2 = TimeFunction(name='u2', grid=grid, space_order=2)
print("Space order 2:\n%s\n" % u2.dxl)

# We use u2 to create the explicit first-order derivative
u1_dx = first_derivative(u2, dim=x, side=left, order=1)
print("Explicit space order 1:\n%s\n" % u1_dx)
```

    Space order 1:
    u1(t, x, y)/h_x - u1(t, x - h_x, y)/h_x

    Space order 2:
    3*u2(t, x, y)/(2*h_x) + u2(t, x - 2*h_x, y)/(2*h_x) - 2*u2(t, x - h_x, y)/h_x

    Explicit space order 1:
    u2(t, x, y)/h_x - u2(t, x - h_x, y)/h_x

Ok, so by constructing derivative terms explicitly we again have full control of the spatial discretisation - the power of symbolic computation. Armed with that trick, we can now build and execute our advection-diffusion operator from scratch in one cell.

```python
#NBVAL_IGNORE_OUTPUT
from sympy import solve
from devito import Operator, Constant, Eq, INTERIOR

# Define our velocity fields and initialise with hat function
u = TimeFunction(name='u', grid=grid, space_order=2)
v = TimeFunction(name='v', grid=grid, space_order=2)
init_hat(field=u.data[0], dx=dx, dy=dy, value=2.)
init_hat(field=v.data[0], dx=dx, dy=dy, value=2.)

# Write down the equations with explicit backward differences
a = Constant(name='a')
u_dx = first_derivative(u, dim=x, side=left, order=1)
u_dy = first_derivative(u, dim=y, side=left, order=1)
v_dx = first_derivative(v, dim=x, side=left, order=1)
v_dy = first_derivative(v, dim=y, side=left, order=1)
eq_u = Eq(u.dt + u*u_dx + v*u_dy, a*u.laplace, region=INTERIOR)
eq_v = Eq(v.dt + u*v_dx + v*v_dy, a*v.laplace, region=INTERIOR)

# Let SymPy rearrange our stencils to form the update expressions
stencil_u = solve(eq_u, u.forward)[0]
stencil_v = solve(eq_v, v.forward)[0]
update_u = Eq(u.forward, stencil_u)
update_v = Eq(v.forward, stencil_v)

# Create Dirichlet BC expressions using the low-level API
bc_u = [Eq(u.indexed[t+1, 0, y], 1.)]  # left
bc_u += [Eq(u.indexed[t+1, nx-1, y], 1.)]  # right
bc_u += [Eq(u.indexed[t+1, x, ny-1], 1.)]  # top
bc_u += [Eq(u.indexed[t+1, x, 0], 1.)]  # bottom
bc_v = [Eq(v.indexed[t+1, 0, y], 1.)]  # left
bc_v += [Eq(v.indexed[t+1, nx-1, y], 1.)]  # right
bc_v += [Eq(v.indexed[t+1, x, ny-1], 1.)]  # top
bc_v += [Eq(v.indexed[t+1, x, 0], 1.)]  # bottom

# Create the operator
op = Operator([update_u, update_v] + bc_u + bc_v)

# Execute the operator for a number of timesteps
op(time=nt, dt=dt, a=nu)

plot_field(u.data[0])
```


## Few examples from sympy.stats

```python
from sympy import *
init_session(quiet=True)
%matplotlib inline
```

```python
from sympy.stats import Die, P, E
```

Let's play with two random variables, each representing a throw with a 6-sided die.

```python
X, Y = Die("X"), Die("Y")
```

Probability that we get 6 on both dice

```python
P(Eq(X, 6) & Eq(Y, 6))
```

```python
P(X>Y)
```

You can ask about conditional probabilities

```python
P(X+Y>3, Y<6)
```

and conditional expectations

```python
E(X, X+Y>3)
```

There are many more statistical distributions in the `sympy.stats` module

```python
import sympy.stats
" ".join([p for p in dir(sympy.stats) if p[0].isupper() & (len(p)>1)])
```

    'Arcsin Benini Bernoulli Beta BetaPrime Binomial Cauchy Chi ChiNoncentral ChiSquared Coin ContinuousRV Covariance Dagum Die DiscreteUniform Erlang Expectation Exponential FDistribution FiniteRV FisherZ Frechet Gamma GammaInverse Geometric Gompertz Gumbel Hypergeometric Kumaraswamy Laplace LogNormal Logarithmic Logistic Maxwell Nakagami NegativeBinomial Normal Pareto Poisson Probability QuadraticU Rademacher RaisedCosine Rayleigh ShiftedGompertz StudentT Trapezoidal Triangular Uniform UniformSum Variance VonMises Weibull WignerSemicircle YuleSimon Zeta'

```python
from sympy.stats import Normal, sample, std, cdf
mu1, mu2 = symbols("mu1, mu2")
sigma1, sigma2 = symbols("sigma1, sigma2", positive=True)
```

```python
X, Y = Normal("X", mu1, sigma1), Normal("Y", mu2, sigma2)
```

```python
cdf(X)
```

Probability of encounterin random value larger than $\mu+\sigma$

```python
simplify(P(X>mu1+sigma1))
```

its numerical value

```python
simplify(P(X>mu1+sigma1)).evalf()
```

and the expression used to calculate it:

```python
P(X>mu1+sigma1, evaluate=False)
```

Random samples from the distribution can be generated

```python
sample(X)
```

Standard deviation of sum of two independent normally distributed random variables...

```python
simplify(std(X+Y))
```

If the integrals cannot be calculated analytically in SymPy,

```python
X, Y = Normal("X", 0, 1), Normal("Y", 1, 1)
P(X>Y, evaluate=False)
```

the probabilities can be estimated by Monte Carlo method (random sampling)

```python
P(X>Y, numsamples=100000)
```


# Effect of cancelling a process zero
The following exercise is taken from Åström & Wittenmark (problem 5.3)

Consider the system with pulse-transfer function
$$ H(z) = \frac{z+0.7}{z^2 - 1.8z + 0.81}.$$
Use polynomial design to determine a controller such that the closed-loop system has the characteristic polynomial $$ A_c = z^2 -1.5z + 0.7. $$
Let the observer polynomial have as low order as possible, and place all observer poles in the origin (dead-beat observer). Consider the following two cases

(a) The process zero is cancelled

(b) The process zero is not cancelled.

Simulate the two cases and discuss the differences between the two controllers. Which one should be preferred?

(c) Design an incremental controller for the system

```python
import numpy as np
import matplotlib.pyplot as plt
import control
import sympy as sy
```

## Checking the poles
Before solving the problem, let's look at the location of the poles of the plant and the desired closed-loop system.

```python
H = control.tf([1, 0.7], [1, -1.8, 0.81], 1)
control.pzmap(H)
```

```python
z = sy.symbols("z", real=False)
Ac = sy.Poly(z**2 - 1.5*z + 0.7,z)
sy.roots(Ac)
```

    {0.75 + 0.370809924354783*I: 1, 0.75 - 0.370809924354783*I: 1}

So, the plant has a double pole in $z=0.9$, and the desired closed-loop system has complex-conjugated poles in $z=0.75 \pm i0.37$.

## (a)
### The feedback controller $F_b(z)$
The plant has numerator polynomial $B(z) = z+0.7$ and denominator polynomial $A(z) = z^2 - 1.8z + 0.81$. With the feedback controller $$F_b(z) = \frac{S(z)}{R(z)}$$ and feedforward $$F_f(z) = \frac{T(z)}{R(z)}$$ the closed-loop pulse-transfer function from the command signal to the output becomes
$$ H_{c}(z) = \frac{\frac{T(z)}{R(z)} \frac{B(z)}{A(z)}}{1 + \frac{B(z)}{A(z)}\frac{S(z)}{R(z)}} = \frac{T(z)(z+0.7)}{A(z)R(z) + S(z)(z+0.7)}.$$
To cancel the process zero, $z+0.7$ should be a factor of $R(z)$. Write $R(z)= \bar{R}(z)(z+0.7)$ to obtain the Diophantine equation
$$ A(z)\bar{R}(z) + S(z) = A_c(z)A_o(z).$$

Let's try to find a minimum-order controller that solves the Diophantine equation. The degree of the left hand side (and hence also of the right-hand side) is
$$ \deg (A\bar{R} + S) = \deg A + \deg \bar{R} = 2 + \deg\bar{R}.$$
The number of equations obtained when setting the coefficients of the left- and right-hand side equal is the same as the degree of the polynomials on each side (taking into account that the leading coefficient is 1, by convention).

The feedback controller can be written
$$ F_b(z) = \frac{S(z)}{R(z)} = \frac{s_0z^n + s_1z^{n-1} + \cdots + s_n}{(z+0.7)(z^{n-1} + r_1z^{n-2} + \cdots + r_{n-1}}, $$
which has $(n-1) + (n+1) = 2n$ unknown parameters, where $n = \deg\bar{R} + 1$.
So to obtain a Diophantine equation which gives exactly as many equations in the coefficients as unknowns, we must have
$$ 2 + \deg\bar{R} = 2\deg\bar{R} + 2 \quad \Rightarrow \quad \deg\bar{R} = 0.$$

Thus, the controller becomes
$$ F_b(z) = \frac{s_0z + s_1}{z+0.7}, $$
and the Diophantine equation
$$ z^2 - 1.8z + 0.81 + (s_0z + s_1) = z^2 - 1.5z + 0.7$$
$$ z^2 - (1.8-s_0)z + (0.81 + s_1) = z^2 - 1.5z + 0.7, $$
with solution
$$ s_0 = 1.8 - 1.5 = 0.3, \qquad s_1 = 0.7-0.81 = -0.11. $$
The right hand side of the Diophantine equation consists only of the desired characteristic polynomial $A_c(z)$, and the observer polynomial is $A_o(z) = 1$, in order for the degrees of the left- and right hand side to be the same.

Let's verify by calculation using SymPy.

```python
s0,s1 = sy.symbols("s0, s1")
A = sy.Poly(z**2 -1.8*z + 0.81, z)
B = sy.Poly(z + 0.7, z)
S = sy.Poly(s0*z + s1, z)
Ac = sy.Poly(z**2 - 1.5*z + 0.7, z)
Ao = sy.Poly(1, z)

# Diophantine equation
Dioph = A + S - Ac*Ao

# Extract the coefficients
Dioph_coeffs = Dioph.all_coeffs()

# Solve for s0 and s1,
sol = sy.solve(Dioph_coeffs, (s0,s1))
print('s_0 = %f' % sol[s0])
print('s_1 = %f' % sol[s1])
```

    s_0 = 0.300000
    s_1 = -0.110000

### The feedforward controller $F_f(z)$
Part of the methodology of the polynomial design, is that the forward controller $F_f(z) = \frac{T(z)}{R(z)}$ should cancel the observer poles, so we set $T(z) = t_0A_o(z)$. In case (a) the observer poynomial is simply $A_o(z)=1$. However, since $R(z)=z+0.7$, we can choose $T(z) = t_0z$ and still have a causal controller $F_f(z)$.
The scalar factor $t_0$ is chosen to obtain unit DC-gain of  $H_c(z)$, hence
$$ H_c(1) = \frac{t_0}{A_c(1)} = 1 \quad \Rightarrow \quad t_0 = A_c(1) = 1-1.5+0.7 = 0.2$$

```python

```

### Simulate
Let's simulate a step-responses from the command signal, and plot both the output and the control signal.

```python
t0 = float(Ac.eval(1))
Scoeffs = [float(sol[s0]), float(sol[s1])]
Rcoeffs = [1, 0.7]
Fb = control.tf(Scoeffs, Rcoeffs, 1)
Ff = control.tf([t0], Rcoeffs, 1)
Hc = Ff * control.feedback(H, Fb) # From command-signal to output
Hcu = Ff * control.feedback(1, Fb*H)
```

```python
tvec = np.arange(40)
(t1, y1) = control.step_response(Hc,tvec)
plt.figure(figsize=(14,4))
plt.step(t1, y1[0])
plt.xlabel('k')
plt.ylabel('y')
plt.title('Output')
```

```python
(t1, y1) = control.step_response(Hcu,tvec)
plt.figure(figsize=(14,4))
plt.step(t1, y1[0])
plt.xlabel('k')
plt.ylabel('y')
plt.title('Control signal')
```

## Solve (b) on your own

```python

```

## Solve (c) on your own

```python

```


# Deutsch's algorithm

```python
import numpy as np

from qiskit import QuantumRegister, ClassicalRegister
from qiskit import QuantumCircuit
from qiskit import execute, BasicAer

import qiskit.tools.visualization as qvis

import warnings
warnings.filterwarnings("ignore", category=DeprecationWarning)
```

Deutsch's algorithm allows us to perform a task using fewer queries quantumly than are needed classically.

Suppose we have some function $f$, that we know to be one of the 4 following functions:

| Function ($f$) |   $f$(0)    | $f$(1) |
| --- | -------- | -------- |
| `constant_zero`| 0 | 0|
| `constant_one` | 1| 1 |
| `balanced_id` | 0| 1 |
| `balanced_not`| 1 | 0 |

The two "constant" functions output the same value no matter the input, whereas the balanced functions output different values for each input.

## Our task

We are given a black box that we know implements one of: `constant_zero`, `constant_one`, `balanced_id`, `balanced_not`. We are allowed to query the box (i.e. ask for $f(x)$ for either $x= 0, 1$) as many times as we need. _How many queries are needed to determine with certainty which function the box implements_?

Let's choose one of the four functions randomly.

```python
function_names = ['constant_zero', 'constant_one', 'balanced_id', 'balanced_not']
oracle = function_names[np.random.randint(4)]
```

### Classical solution

We _must_ query twice. For example, if we query $f(0)$ and receive $0$, we know we must have `constant_zero` or `balanced_id`, but we still can't pin down the exact function without querying a second time.

### Quantum solution

We can perform this task with only a _single_ query using two qubits.

Our oracle $U_f$ is going to implement the following transformation:
\begin{equation}
 U_f|x\rangle|y\rangle = |x\rangle|y\oplus f(x) \rangle
\end{equation}
where $f$ will be replaced with one of the four functions above, and $\oplus$ indicates addition modulo 2.

For example, if $f$ is `constant_one`,
\begin{eqnarray}
 U_f|00\rangle &=& |01\rangle \\
 U_f|01\rangle &=& |00\rangle \\
 U_f|10\rangle &=& |11\rangle \\
 U_f|11\rangle &=& |10\rangle
\end{eqnarray}

From this you can see that each function will produce a different permutation of the computational basis states.

**Exercise**: Compute the unitary matrix associated with each function, and write out the quantum circuit that implements it. (The answers are shown further down because we need them for the demo, but you can still compute them yourself to understand how they work.)

To execute Deutsch's algorithm, we'll follow the steps as shown in class. We begin with the state
\begin{equation} |+-\rangle =
 \left(\frac{|0\rangle + |1\rangle}{\sqrt{2}}\right) \otimes \left(\frac{|0\rangle - |1\rangle}{\sqrt{2}}\right) = \frac{1}{2}\left(|00\rangle - |01\rangle + |10\rangle - |11\rangle \right)
\end{equation}

Let's set up the quantum and classical registers and prepare it:

```python
q = QuantumRegister(2)
c = ClassicalRegister(1)

circ = QuantumCircuit(q, c)
```

```python
# Prepares the state |+-> starting from |00>
circ.h(q[0])
circ.x(q[1])
circ.h(q[1]);
```

Let's now apply the oracle circuit:

```python
# Here are the quantum circuits for each of our functions
# If the circuit is 'constant_zero', we do nothing.
if oracle == 'constant_one':
    circ.x(q[1])
elif oracle == 'balanced_id':
    circ.cx(q[0], q[1])
elif oracle == 'balanced_not':
    circ.x(q[0])
    circ.cx(q[0], q[1])
```

We saw in the lectures that after calling the oracle, if the function is constant, we get a $|+\rangle$ on the first qubit, but when it is balanced, we get $|-\rangle$. We can combine all these results into a compact form:

\begin{equation}
 U_f|+\rangle |- \rangle =  \left( \frac{|0\rangle + (-1)^{f(0) \oplus f(1)}|1\rangle}{\sqrt{2}} \right) \otimes \left( \frac{|0\rangle - |1\rangle}{\sqrt{2}} \right)
 \end{equation}

We can then determine the value of $f(0) \oplus f(1)$, by applying a Hadamard to the first qubit to perform a basis change back to the computational basis, and then measuring. We should obtain:
\begin{equation}
 |f(0)\oplus f(1) \rangle \otimes |-\rangle
\end{equation}

If the function is constant, $f(0) \oplus f(1) = 0$ and so the first qubit will be in state $|0\rangle$. Similarly, if it is balanced, the first qubit will be in state $|1\rangle$. Does this work?

```python
# Convert the first qubit back to the computational basis and measure
circ.h(q[0])
circ.measure(q[0], c[0])

# Execute the circuit a single time to get the measurement outcome
backend = BasicAer.get_backend('qasm_simulator')
result = execute(circ, backend, shots = 1).result()
counts = result.get_counts()

if list(counts.keys())[0] == '0': # c[0] is a tuple of the register and the measurement value
    print("Measured first qubit in state |0>, function is constant.")
else:
    print("Measured first qubit in state |1>, function is balanced.")
```

Were we correct? Let's see which function the oracle had chosen to implement.

```python
print(f"The function implemented by the oracle is '{oracle}'")
```

Deutsch's algorithm can also be generalized to the Deutsch-Josza algorithm when $f$ is a function on $n$ bits. No matter the value of $n$, it is still possible to determine whether $f$ is constant or balanced with a single query.


# Least Squares for Response Surface Work

Link: [http://charlesreid1.com/wiki/Empirical_Model-Building_and_Response_Surfaces#Chapter_3:_Least_Squares_for_Response_Surface_Work](http://charlesreid1.com/wiki/Empirical_Model-Building_and_Response_Surfaces#Chapter_3:_Least_Squares_for_Response_Surface_Work)

## Method of Least Squares

Least squares helps you to understand a model of the form:

$$
y = f(x,t) + \epsilon
$$

where:

$$
E(y) = \eta = f(x,t)
$$

is the mean level of the response $y$ which is affected by $k$ variables $(x_1, x_2, ..., x_k) = \mathbf{x}$

It also involves $p$ parameters $(t_1, t_2, ..., t_p) = \mathbf{t}$

$\epsilon$ is experimental error

To examine this model, experiments would run at n different sets of conditions, $x_1, x_2, ..., x_n$

would then observe corresponding values of response $y_1, y_2, ..., y_n$

Two important questions:

1. does postulated model accurately represent the data?

2. if model does accurately represent data, what are best estimates of parameters t?

start with second question first

-----

Given: function $f(x,t)$ for each experimental run

$n$ discrepancies:

$$
{y_1 - f(x_1,t)}, {y_2 - f(x_2,t)}, ..., {y_n - f(x_n,t)}
$$

Method of least squares selects best value of t that make the sum of squares smallest:

$$
S(t) = \sum_{u=1}^{n} \left[ y_n - f \left( x_u, t \right) \right]^2
$$

$S(t)$ is the sum of squares function

Minimizing choice of $t$ is denoted

$$
\hat{t}
$$

are least-squares estimates of $t$ good?

Their goodness depends on the nature of the distribution of their errors

Least-squares estimates are appropriate if you can assume that experimental errors:

$$
\epsilon_u = y_u - \eta_u
$$

are statistically independent and with constant variance, and are normally distributed

these are "standard assumptions"

## Linear models

This is a limiting case, where

$$
\eta = f(x,t) = t_1 z_1 + t_2 z_2 + ... + t_p z_p
$$

adding experimental error $\epsilon = y - \eta$:

$$
y = t_1 z_1 + t_2 z_2 + ... + t_p z_p + \epsilon
$$

model of this form is linear in the parameters

### Algorithm

Formulate a problem with $n$ observed responses, $p$ parameters...

This yields $n$ equations of the form:

$$
y_1 = t_1 z_{11} + t_2 z_{21} + ... \\
y_2 = t_1 z_{21} + t_2 z_{22} + ...
$$

etc...

This can be written in matrix form:

$$
\mathbf{y} = \mathbf{Z t} + \boldsymbol{\epsilon}
$$

and the dimensions of each matrix are:

* $y = n \times 1$
* $Z = n \times p$
* $t = p \times 1$
* $\epsilon = n \times 1$

the sum of squares function is given by:

$$
S(\mathbf{t}) = \sum_{u=1}^{n} \left( y_u - t_1 z_{1u} - t_2 z_{2u} - ... - t_p z_{pu} \right)^2
$$

or,

$$
S(t) = ( y - Zt )^{\prime} ( y - Zt )
$$

this can be rewritten as:

$$
\mathbf{ Z^{\prime} Z t = Z^{\prime} y }
$$

### Rank of Z

If there are relationships between the different input parameters $(z's)$, then the matrix $\mathbf{Z}$ can become singular

e.g. if there is a relationship $z_2 = c z_1$, then you can only estimate the linear combination $z_1 + c z_2$

reason: when $z_2 = c z_1$, changes in $z_1$ can't be distinguished from changes in $z_2$

$Z$ (an $n \times p$ matrix) is said to be full rank $p$ if there are no linear relationships of the form:

$$
a_1 z_1 + a_2 z_2 + ... + a_p z_p l= 0
$$

if there are $q > 0$ independent linear relationships, then $Z$ has rank $p - q$

## Analysis of Variance: 1 regressor

Assume simple model $y = \beta + \epsilon$

This states that $y$ is varying about an unknown mean $\beta$

Suppose we have 3 observations of $y$, $\mathbf{y} = (4, 1, 1)' $

Then the model can be written as $y = z_1 t + \epsilon$

and $z_1 = (1, 1, 1) '$

and $t = \beta$

so that

```
[ 4 ]   [ 1 ]     [ \epsilon_1 ]
[ 1 ] = [ 1 ] t + [ \epsilon_2 ]
[ 1 ]   [ 1 ]     [ \epsilon_3 ]
```

Supposing the linear model posited a value of one of the regressors t, e.g. $t_0 = 0.5$

Then you could check the null hypothesis, e.g. $H_0 : t = t_0 = 0.5$

If true, the mean observation vector given by $\eta_0 = z_1 t_0$

or,

```
[ 0.5 ]   [ 1 ]
[ 0.5 ] = [ 1 ] 0.5
[ 0.5 ]   [ 1 ]
```

and the appropriate "observation breakdown" (whatever that means?) is:

$$
y - \eta_0 = ( \hat{y} - \eta_0 ) + ( y - \hat{y} )
$$

Associated with this observation breakdown is an analysis of variance table:

{|
|Source
|Degrees of freedom (df)
|Sum of squares (square of length), SS
|Mean square, MS
|Expected value of mean square, E(MS)
|-
|Model
|1
|$\vert \hat{y} - \eta_0 \vert^2 = ( \hat{t} - t_0 )^2 \sum z_1^2$
|6.75
|$\sigma^2 + ( t - t_0 )^2 \sum z_1^2$

|-
|Residual
|2
|$\vert y - \hat{y} \vert^2 = \sum ( y - \hat{t} z_1 )^2$
|3.00
|$\sigma^2$

|-
|Total
|3
|$\vert y - \eta_0 \vert^2 = \sum ( y - \eta_0 )^2 = 12.75$
|
|
|}

Sum of squares: squared lengths of vectors

Degrees of freedom: number of dimensions in which vector can move (geometric interpretation)

The model $y = z_1 t + \epsilon$ says whatever the data is, the systematic part $\hat{y} - \eta_0 = ( \hat{t} - t_0) z_1$ of $y - \eta_0$ must lie in the direction of $z_1$, which gives $\hat{y} - \eta_0$ only one degree of freedom.

Whatever the data, the residual vector must be perpendicular to $z_1$ (why?), and so it can move in 2 directions and has 2 degrees of freedom

Now, looking at the null hypothesis:

The component $\vert \hat{y} - \eta_0 \vert^2 = ( \hat{t} - t_0 )^2 \sum z^2$ is a measure of discrepancy between POSTULATED model $\eta_0 = z_1 t_0$ and ESTIMATED model $\hat{y} = z_1 \hat{t}$

Making "standard assumptions" (earlier), expected value of sum of squares, assuming model is true, is $( t - t_0 )^2 \sum z_1^2 + \sigma^2$

For the residual component it is $2 \sigma^2$ (or, in general, $\nu_2 \sigma^2$, where $\nu_2$ is number of degrees of freedom of residuals)

Thus a measure of discrepancy from the null hypothesis $t = t_0$ is $F = \frac{ \vert \hat{y} - \eta_0 \vert^2 / 1 }{ \vert y - \hat{y} \vert^2 / 2 }$

if the null hypothesis were true, then the top and bottom would both estimate the same $\sigma^2$

So if $F$ is different from 1, that indicates departure from null hypothesis

The MORE $F$ differs from 1, the more doubtful the null hypothesis becomes

## Least squares: 2 regressors

Previous model, $y = \beta + \epsilon$, said $y$ was represented with a mean $t$ plus an error.

Instead, suppose that there are systematic deviations from the mean, associated with an external variable (e.g. humidity in the lab).

Now equation is for straight line: $ y = \beta_0 + \beta_1 x + \epsilon$

or, $y = z_1 t_1 + z_2 t_2 + \epsilon$

So now the revised least-squares model is: $\eta = z_1 t_1 + z_2 t_2$

$\eta = E(y)$ - i.e. $\eta$ is in the plane defined by linear combinations of vectors $z_1, z_2$

because $z_1^{\prime} z_2 = \sum z_1 z_2 \neq 0$, these two vectors are NOT at right angles

The least-squares values $\hat{t_1}, \hat{t_2}$ produce a vector $\hat{\hat{y}} = z_1 \hat{t_1} + z_2 \hat{t_2}$

These least-squares values make the squared length $\sum ( y - \hat{\hat{y}} )^2 = \vert y - \hat{\hat{y}} \vert^2$ of the residual vector as small as possible

The normal equations express fact that residual vector must be perpendicular to both $z_1$ and $z_2$:

$$
z_1^{\prime} ( y - \hat{\hat{y}} ) = 0 \\
z_2^{\prime} ( y - \hat{\hat{y}} ) = 0
$$

also written as:

$$
\begin{align}
\sum z_1 ( y - \hat{t_1} z_1 - \hat{t_2} z_2 ) &=& 0 \\
\sum z_2 ( y - \hat{t_1} z_1 - \hat{t_2} z_2 ) &=& 0
\end{align}
$$

also written (in matrix form) as:

$$
\mathbf{Z^{\prime}} ( \mathbf{y - Z \hat{t} } ) = 0
$$

Now suppose the null hypothesis was investigated for $t_1 = t_{10} = 0.5$ and $t_2 = t_{20} = 1.0$

Then the mean observation vector $\eta_0$ is represented as $\eta_0 = t_{10} z_1 + t_{20} z_2$

$$
y - \eta_0 = \left( \hat{\hat{y}} - \eta_0 \right) + \left( y - \hat{\hat{y}} \right)
$$

and so

$$F_0 = \frac{ \vert \hat{\hat{y}} - \eta_0 \vert / 2 }{ \vert y - \hat{\hat{y}} \vert^2 / 1 } = 2.23
$$

## Orthogonalizing second regressor

In the above example, $z_1$ and $z_2$ are not orthogonal

One can find the vectors $z_1$ and $z_{2 \cdot 1}$ that are orthogonal

To do this, use least squares property that residual vector is orthogonal to space in which the predictor variables lie

Regard $z_2$ as "response" vector and $z_1$ as predictor variable

You then obtain $\hat{z_2} = 0.2 z_1$ (how?)

so the residual vector is $z_{2 \cdot 1} = z_2 - \hat{z_2} = z_2 - 0.2 z_1$

now the model can be rewritten as $\eta = \left( t_1 + 0.2 t_2 \right) z_1 + t_2 \left( z_2 - 0.2 z_1 \right) = t z_1 + t_2 z_{2 \cdot 1}$

This gives three least-squares equations:

1. $\hat{y} = 2 z_1$
2. $\hat{y} = 1.5 z_1 + 2.5 z_2$
3. $\hat{y} = 2.0 z_1 + 2.5 z_{2 \cdot 1}$

The analysis of variance becomes:

---

Source: Response function with $z_1$ only

DoF: 1

Sum of Squares (SS): $\vert \hat{y} - \eta_0 |vert^2 = \left( \hat{t} - t_0 \right)^2 \sum z_1^2 = 12.0$

Source: Extra due to $z_2$ (given $z_1$)

DoF: 1

SS: $\vert \hat{\hat{y}} - \hat{y} \vert^2 = \hat{t}_2^2 \sum z_{2 \cdot 1}^2 = 4.5$

Source: Residual

DoF: 1

SS: $\vert y - \hat{\hat{y}} \vert^2 = \sum \left( y - \hat{\hat{y}} \right)^2 = 1.5$

Source: Total

DoF: 3

SS: $\vert y - \eta_0 \vert^2 = \sum \left( y - \eta_0 \right)^2 = 18.0$

## Generalization to p regressors

With n observations and p parameters:

n relations implicit in response function can be written

$$
\boldsymbol{\eta} = \mathbf{Z t}
$$

Assuming $Z$ is full rank, and letting $\hat{\mathbf{t}}$ be the vector of estimates given by normal equations

$$
\left( \mathbf{ y - \hat{y} } \right)^{\prime} \mathbf{Z} = \left( y - Z \hat{t} \right)^{\prime} Z = 0
$$

Sum of squares function is $S(t) = (y - \eta)^{\prime} (y - \eta) = (y - \hat{y})^{\prime} (y - \hat{y}) + ( \hat{y} - \eta )^{\prime} (\hat{y} - \eta)$

Because cross-product is zero from the normal equations

$$
S(t) = S(\hat{t}) + (\hat{t} - t)^{\prime} \mathbf{Z^{\prime} Z} ( \hat{t} - t )
$$

Furthermore, because $\mathbf{Z^{\prime} Z}$ is positive definite, $S(t)$ minimized when $t = \hat{t}$

So the solution to the normal equations producing the least squares estimate is the one where $t = \hat{t}$:

$$
\hat{t} = ( \mathbf{Z^{\prime} Z} )^{-1} \mathbf{Z^{\prime} y}
$$

----

Source: Response function

DoF: $p$

SS: $\vert \hat{y} - \eta \vert^2 = (\hat{t} - t)^{\prime} \mathbf{Z^{\prime} Z} ( \hat{t} - t )$

Source: Residual

DoF: $n - p$

SS: $\vert y - \hat{y} \vert^2 = \sum ( y - \hat{y} )^2 $

Source: Total

DoF: $n$

SS: $\vert y - \eta \vert^2 = \sum ( y - \eta )^2 $

## Bias in Least-Squares Estimators if Inadequate Model

Say data was being fit with a model $y = Z_1 t_1 + \epsilon$,

but the true model that should have been used is $y = Z_1 t_1 + Z_2 t_2 + \epsilon$

$t_1$ would be estimated by $\hat{t_1} = (\mathbf{ Z_1^{\prime} Z_1 } )^{-1} \mathbf{ Z_1^{\prime} y }$

but using true model,

$$
\begin{array}{rcl}
E( \hat{t_1} ) &=& ( \mathbf{Z_1^{\prime} Z_1} )^{-1} \mathbf{Z_1^{\prime}} E(\mathbf{y}) \\
&=& ( \mathbf{ Z_1^{\prime} Z_1 } )^{-1} \mathbf{Z_1^{\prime}} (\mathbf{Z_1 t_1} + \mathbf{Z_2 t_2} ) \\
&=& \mathbf{t_1 + A t_2}
\end{array}
$$

The matrix A is the bias or alias matrix

$$
A = \left( \mathbf{ Z_1^{\prime} Z_1 } \right)^{-1} \mathbf{ Z_1^{\prime} Z_2 }
$$

Unless $A = 0$, $\hat{t_1}$ will represent $t_1$ AND $t_2$, not just $t_1$

$A = 0$ when $\mathbf{Z_1^{\prime} Z_2} = 0$, which happens if regressors in $\mathbf{Z_1}$ are orthogonal to regressors in $\mathbf{Z_2}$

```python

```


**Exercise set 3**
==============

>The goal of this exercise is to investigate some theoretical properties of the solution to
>the (multiple) linear regression problem, and to perform a **least-squares regression**.
>We will also see how we can evaluate our regression model by investigating **residuals**.

**Exercise 3.1**

Multiple linear regression (MLR) solves the regression equation $\mathbf{Y} = \mathbf{X}\mathbf{B}$ with
$\mathbf{B} = (\mathbf{X}^\mathrm{T} \mathbf{X})^{-1} \mathbf{X}^\mathrm{T} \mathbf{Y}$
where the
dimensions of the data matrix $\mathbf{X}$ is $[N \times M]$.
Will this solution work when (please explain why/why not):

**(a)** $\det \left( \mathbf{X}^\mathrm{T} \mathbf{X} \right) = 0$?

**(b)** $\det \left( \mathbf{X}^\mathrm{T} \mathbf{X} \right) > 0$?

**(c)** $\det \left( \mathbf{X}^\mathrm{T} \mathbf{X} \right) \neq 0$?

**(d)** The variables in $\mathbf{X}$ are correlated?

**(e)** The columns in $\mathbf{X}$ are orthogonal?

**(f)** The rank of $\mathbf{X}$ is $\frac{\min(N, M)}{2}$?

**(g)** We have more variables than samples/objects (more columns than rows in $\mathbf{X}$)?

**(h)** When we have more samples/objects than variables (more rows than columns in $\mathbf{X}$)?

**Your answer to question 3.1:** *Double click here*

**Exercise 3.2**

As stated in the previous problem, the MLR solution of the equation
$\mathbf{Y} = \mathbf{X}\mathbf{B}$ is

\begin{equation}
\mathbf{B} = (\mathbf{X}^\mathrm{T} \mathbf{X})^{-1} \mathbf{X}^\mathrm{T} \mathbf{Y}.
\end{equation}

If we let $\hat{\mathbf{Y}}$ be the values calculated by the MLR solution, we can write
this as,
\begin{equation}
\hat{\mathbf{Y}} = \mathbf{X}\mathbf{B} =
\mathbf{X} \left[ (\mathbf{X}^\mathrm{T} \mathbf{X})^{-1} \mathbf{X}^\mathrm{T} \mathbf{Y} \right] =
\left[\mathbf{X} (\mathbf{X}^\mathrm{T} \mathbf{X})^{-1} \mathbf{X}^\mathrm{T}\right] \mathbf{Y} =
\mathbf{H} \mathbf{Y},
\tag{1}\end{equation}

where we have defined the projection matrix, $\mathbf{H}$, as,
\begin{equation}
\mathbf{H} = \mathbf{X} \left(\mathbf{X}^\mathrm{T} \mathbf{X}\right)^{-1} \mathbf{X}^\mathrm{T}.
\label{eq:projectionmatrix}
\tag{2}\end{equation}

This means that we can write the residual, $\mathbf{E}$, as,
\begin{equation}
\mathbf{E} = \mathbf{Y} - \hat{\mathbf{Y}} = \mathbf{Y} - \mathbf{H} \mathbf{Y} =
(\mathbf{I} -\mathbf{H}) \mathbf{Y},
\tag{3}\end{equation}

where $\mathbf{I}$ is the identity matrix.

In this exercise, we will show two properties of $\mathbf{H}$ that enables us to simplify
the squared error, $\mathbf{E}^\mathrm{T} \mathbf{E}$, as follows,

\begin{equation}
\mathbf{E}^\mathrm{T} \mathbf{E} = \mathbf{Y}^\mathrm{T} (\mathbf{I} -\mathbf{H})^\mathrm{T}
(\mathbf{I} -\mathbf{H}) \mathbf{Y} =
\mathbf{Y}^\mathrm{T} (\mathbf{I} -\mathbf{H}) \mathbf{Y}.
\tag{4}\end{equation}

In this equation, the last equality follows from the following two properties of $\mathbf{H}$:

**(a)**  $\mathbf{H}$ is *symmetric*: $\mathbf{H}^\mathrm{T} = \mathbf{H}$.

**(b)**  $\mathbf{H}$ is *idempotent*: $\mathbf{H}^{k} = \mathbf{H}$ where $k > 0$ is an integer.

Show these two properties for $\mathbf{H}$. (Hint: For the idempotency, begin by showing that $\mathbf{H}^{2} = \mathbf{H}$.)

**Your answer to question 3.2:** *Double click here*

**Exercise 3.3**

In the regression problem $\mathbf{y} = \mathbf{X}\mathbf{b}$
we find the least-squares solution assuming that $\mathbf{X}^\mathrm{T} \mathbf{X}$ is
non-singular. If you are given the information that $\mathbf{X}$ is symmetric
and non-singular, is there another
simpler formula for estimating the regression coefficients ($\mathbf{b}$)?

**Your answer to question 3.3:** *Double click here*

**Exercise 3.4**

Assume that we have recorded data as shown in Fig. 1.

**Fig. 1:** Example data.

To model this data, we suggest a third-order polynomial in $x$:
\begin{equation}
\hat{y} = b_0 + b_1 x + b_2 x^2 + b_3 x^3 .
\end{equation}

Explain how you can formulate this on a form suitable for least-squares regression,
$\mathbf{y} = \mathbf{X} \mathbf{b}$.
What do the vectors $\mathbf{y}$ and $\mathbf{b}$ contain? What does the matrix $\mathbf{X}$ contain?

**Your answer to question 3.4:** *Double click here*

**Exercise 3.5**

The temperature (°C) is measured continuously over time at a high altitude
in the atmosphere using a
weather balloon. Every hour a measurement is made and sent to an on-board computer.
The measurements are
shown in Fig. 2 and can be found in [the data file](Data/temperature.txt) (located at 'Data/temperature.txt').

**Fig. 2:** Measured temperature as a function of time.

**(a)**  Create a Python script that performs polynomial
fitting to the data using a first, second, third, fourth,
and fifth order polynomial model. Hint: Make use of `numpy`, `matplotlib`
and `pandas`.

```python
# Your code here
```

**(b)**  Plot the fitted curves for the five models to the raw data.

```python
# Your code here
```

**(c)** Plot the residual curves for the five models and determine,
from a visual inspection, the best polynomial order to use for modeling the
temperature as a function of time.

```python
# Your code here
```

**Your answer to question 3.5(c):** *Double click here*

**(d)**  Obtain the sum of squared residuals for each polynomial. Plot this as a function
of the degree of the polynomial and determine from visual inspection
the best polynomial order to use for modeling the
temperature as a function of time. Does this agree with your conclusion in point **3.5(c)**?

```python
# Your code here
```

**Your answer to question 3.5(d):** *Double click here*