# Multivariate Gaussians, K-means and Gaussian Mixture Models

```python
%matplotlib inline
import matplotlib
import numpy as np
import matplotlib.pyplot as plt
import scipy.linalg
import sklearn.datasets
import pylab
from matplotlib.patches import Ellipse

```

# Generate data
We'll generate data from a mixture of Gaussians model by generating data from three separate Gaussians and concatenating the data. We'll do everything in 2 dimensions so that we can visualise things easily.

In this tutorial, we'll sometimes avoid using a built in function, and instead do things the "long way" to get some extra exposure to some basic maths and statistics.

Using the `np.random.randn` command we can generate data from a standard multivariate normal, and using `matplotlib` we can visualise this.

```python
# Generate data
num_data = 10000
dimensions = 2
data = np.random.randn(num_data,dimensions)

# Plot the data
plt.figure(figsize=(6, 6))
plt.scatter(data[:,0], data[:,1])
plt.axis('equal')
plt.show()
```

Just as a quick sanity check, we can make sure that the mean and covariance of the data we just generated is what we expect:

```python
print np.mean(data[:,0])
print np.mean(data[:,1])
print np.cov(data.T)
```

    -0.0123484576753
    -0.0148022033307
    [[ 0.99082741  0.00706382]
     [ 0.00706382  0.99911906]]

Great! These numbers are roughly what we would expect, given that the population mean is `0` and the population covariance is the `2x2` identity matrix. You can try increasing `num_data` and verify that the empirical mean and covariance get closer to the population values.

Now, let's use the function `np.random.randn` to generate data from non-standard Gaussians.

Let's suppose that $Z \sim \mathcal{N}\left(0, I_n\right)$ is a standard Gaussian, and we'll define a new variable $X$ as:

$$ X = \Sigma^{1/2} Z + \mu $$

where $\Sigma$ is a _symmetric_ $n\times n$ matrix and $\mu$ is a vector of size $n$.

By closure of Gaussians under linear operations, $X$ is still Gaussian and its distribution is therefore determined by its mean and covariance, which we can simply calculate.

$$
\begin{align}
\mathbb{E}[X] &= \Sigma^{1/2}\mathbb{E}[Z] + \mu \\
              &= \mu \\
\mathbb{C}[X] &= \mathbb{E}[XX^\intercal] - \mathbb{E}[X]\mathbb{E}[X]^{\intercal} \\
              &= \Sigma^{1/2}\mathbb{E}[ZZ^\intercal]\Sigma^{1/2} + \mu \mu^\intercal - \mu \mu^\intercal\\
              &= \Sigma
\end{align}
$$

Therefore, $X \sim \mathcal{N}\left(\mu, \Sigma\right)$. Using this fact we can write a function to generate samples from arbitrary Gaussians.

Note that because we store our data in a matrix where each _row_ corresponds to a single observation, we multiply by the square root of the covariance matrix on the right rather than the left in the function below.

```python
def sample_gaussian(mean, covariance, n_data, dimensions=2):
    assert mean.shape == (dimensions,)
    assert covariance.shape == (dimensions, dimensions)
    sqrt_cov = scipy.linalg.sqrtm(covariance)
    data = np.random.randn(n_data, dimensions)
    return np.dot(data, sqrt_cov) + mean
```

```python
mean1 = np.array([3,1])
cov1 = np.array([[3,1],[1,2]])
data1 = sample_gaussian(mean1, cov1, 1000)

plt.figure(figsize=(6, 6))
plt.scatter(data1[:,0], data1[:,1])
plt.axis('equal')
plt.show()
```

Again as a sanity check we can calculate the empirical mean and covariances of the data we just generated to see that they match what we expect.

```python
print np.mean(data1[:,0])
print np.mean(data1[:,1])
print np.cov(data1.T)
```

    3.07562386296
    1.02746883564
    [[ 2.92964863  0.94634928]
     [ 0.94634928  1.9995487 ]]

Let's generate a few different clusters and plot them together.

```python
mean1 = np.array([3,1])
cov1 = np.array([[3,1],[1,2]])
data1 = sample_gaussian(mean1, cov1, 1000)

mean2 = np.array([7,-2])
cov2 = np.array([[1,1],[1,5]])
data2 = sample_gaussian(mean2, cov2, 1000)

mean3 = np.array([-1,-4])
cov3 = np.array([[3,-1],[-1,3]])
data3 = sample_gaussian(mean3, cov3, 1000)

plt.figure(figsize=(6, 6))
plt.scatter(data1[:,0], data1[:,1])
plt.scatter(data2[:,0], data2[:,1])
plt.scatter(data3[:,0], data3[:,1])
plt.axis('equal')
plt.show()
```

# K-means
Now that we have some data generated, let's do some clustering. We're going to implement one of the most basic clustering algorithms: K-means.

The basic idea here is that we are going to start with an initial seed of $K$ points in the data-space. These are the centres of $K$ distinct clusters. We then iterate, first assigning each datum to the cluster whose centre is nearest to it, and then updating the $K$ points by setting each to be the average of all of the data assigned to it.

More formally, we start with $K$ points $m_1, \ldots, m_K$ and some data $x_1,\ldots, x_N$. Then, we iteratively apply each of the following steps:

$1)$ Find, for each $k$, $S_k= \lbrace x_n : d(x_n,m_k) \leq d(x_n, m_i) \: \forall i \rbrace$

$2)$ Update the cluster centres: $m_k = \frac{1}{|S_k|} \sum_{x_n \in S_k} x_n$

We apply these steps until convergence occurs. We could define this, for instance, as occurring when cluster assignments do not change.

We'll write two functions, one for each of the steps above.

```python
def update_cluster_assignments(data, m):
    # returns a dictionary mapping index i to numpy array of data that are nearest to m[i]
    clusters = {}
    for i in range(len(m)):
        clusters[i] = []
    for x in data:
        # find index i of nearest m[i] and append it to cluster
        clusters[np.sum((m-x)**2, 1).argmin()].append(x)
    for i in range(len(m)):
        clusters[i] = np.array(clusters[i])
    return clusters

def update_cluster_means(clusters):
    # returns the updated cluster means
    n_clusters = len(clusters)
    dimension = len(clusters[0][0])
    m = np.zeros([n_clusters, dimension])
    for i in clusters:
        # calculate the mean of points assigned to cluster i
        m[i] = np.mean(clusters[i], 0)
    return m

```

Let's try out the functions that we've written on the data we generated earlier. We'll concatenate all our data together into one big matrix, try some initial seeds for the centres and find the corresponding clusters. We can visualise this by slightly modifying the code to plot from before (we'll put it in a function as we'll use it again later).

```python
data = np.concatenate([data1, data2, data3])
m = np.array([[0,0],[10,10],[2,0.5],[-5,-5]])
clusters = update_cluster_assignments(data, m)

def plotter(clusters):
    # Plot data, colored by cluster
    plt.figure(figsize=(6, 6))
    for i in range(len(clusters)):
        plt.scatter(clusters[i][:,0], clusters[i][:,1])
    plt.axis('equal')
    plt.show()

plotter(clusters)
```

Let's update the mean once and the update the clusters.

```python
m = update_cluster_means(clusters)
clusters = update_cluster_assignments(data, m)
plotter(clusters)
```

The clusters changed! Great. $k$-means works by repeatedly performing the two update functions we just called, so let's write a wrapper function to do this, terminating when the cluster centers have stopped changing.

```python
def kmeans(data, k, eps=None):
    # If initial cluster centres are not provided, randomly select k data points to be the centres
    m = data[np.random.choice(range(len(data)), k, replace=False)]
    m_old = m
    clusters = update_cluster_assignments(data, m)
    m = update_cluster_means(clusters)

    # Stop once the cluster centres stop changing
    while (m_old != m).any():
        m_old = m
        clusters = update_cluster_assignments(data, m)
        m = update_cluster_means(clusters)

    return clusters
```

```python
plotter(kmeans(data, 3))
plotter(kmeans(data, 5))
plotter(kmeans(data, 7))
```

## Trying this out on some real data
We'll run our freshly written $k$-means algorithm on the Iris dataset

```python
iris = sklearn.datasets.load_iris()
iris_data = iris.data
iris_clusters = kmeans(iris_data,3)
```

Since the iris dataset is 4-dimensional, we can't visualise all of it at once. We can compare each of the 6 pairs of dimensions separately:

```python
number_of_subplots=6
v=0
plt.figure(figsize=(12, 22))
plt.axis("equal")
for j in range(3):
    for k in range(j+1, 4):
        v += 1

        ax1 = pylab.subplot(number_of_subplots,3,v)
        for i in range(len(iris_clusters)):
            ax1.scatter(iris_clusters[i][:,j], iris_clusters[i][:,k])

plt.show()
```

# Principal Component Analysis (PCA)

Plotting each pair of dimensions separately is not very satisfying, as it's hard to piece together what each plot shows separately into a single explanation. We can use Principal Component Analysis (PCA) to instead find the most informative 2D subspace to project our data onto before visualising.

PCA is a useful pre-processing algorithm that projects data to a lower dimensional subspace while maintaining the maximum variance within the data. We'll apply this here, projecting the 4-dimensional Iris dataset to a 2-dimensional space so that we can see our clusters better.

Before applying PCA, we have to standardise our data, since different measurement units in different dimensions can mess things up.

```python
iris_data_std = iris_data.copy()
for i in range(len(iris_data[0])):
    iris_data_std[:,i] = (iris_data_std[:,i] - iris_data[:,i].mean()) / iris_data[:,i].std()
```

Next, we calculate the directions of maximal variance within the data by finding the eigenvectors of the covariance matrix. We'll then stack the two eigenvectors corresponding to the largest eigenvalues, giving us a matrix that will project our 4D data onto the 2D subspace with largest variance.

The next plot shows the data in the projected space coloured by the true class label.

```python
# Calculate eigenvalues
eig_vals, eig_vecs = np.linalg.eig(np.cov(iris_data_std.T))

# Project data onto lower dimensional subspace
proj_matrix = eig_vecs[:,0:2]
proj_data = iris_data_std.dot(proj_matrix)

# Plot all data coloured by class label
plt.figure(figsize=(6, 6))
for col in range(3):
    plt.scatter(proj_data[:,0][iris.target==col], proj_data[:,1][iris.target==col])
plt.axis('equal')
plt.show()
```

Now let's take the clusters we found using $k$-means, project each of them to the 2D subspace, and plot.

```python
# Plot data, colored by cluster
plt.figure(figsize=(6, 6))
for i in range(len(iris_clusters)):
    cluster_data = iris_clusters[i].copy()
    # we have to apply the same standardisation transformation as we did to the original data before doing PCA
    for j in range(len(cluster_data[0])):
        cluster_data[:,j] = (cluster_data[:,j] - iris_data[:,j].mean()) / iris_data[:,j].std()
    proj_cluster = cluster_data.dot(proj_matrix)
    plt.scatter(proj_cluster[:, 0], proj_cluster[:, 1])
plt.axis('equal')
plt.show()
```

As you can see, we do a fairly good job at separating the data into the correct classes!

# Gaussian Mixture Model

Let's now work towards building up to a probabilistic generalisation of $k$-means known as a _Gaussian Mixture Model_. $k$-means is great for its simplicity, but it suffers from (at least) the following two problems:

1) Each datum is given a "hard" assignment to a cluster. If for example a datum straddles the boundary of two clusers, we might in fact want to express an uncertain belief of which cluster the datum belongs to.

2) The only thing that is important in determining which cluster a datum belongs to is which centre it is nearest to. The shape of the cluster is not taken into account at all, leading to somewhat strage clusterings such as the one below.

```python
mean1 = np.array([0,0])
cov1 = np.array([[1,0],[0,1]])
data1 = sample_gaussian(mean1, cov1, 100)

mean2 = np.array([0,10])
cov2 = np.array([[1,0],[0,1]])
data2 = sample_gaussian(mean2, cov2, 100)

mean3 = np.array([10,5])
cov3 = np.array([[1,0],[0,50]])
data3 = sample_gaussian(mean3, cov3, 100)

clusters = kmeans(np.concatenate([data1, data2, data3]), 3)
plotter(clusters)
```

In the case above, we'd really want to have three clusters - the two circles on the left, and the long sausage on the right. However, because the sausage is very long, the points near the ends will be closer to the circles than the sausage centre. The Gaussian Mixture Model will come to our rescue.

## The generative model

In a Gaussian Mixture Model (henceforth GMM) we assume that there are $K$ distinct classes, each of which follow different Gaussian distributions with parameters $\mu_k$ and $\Sigma_k$. We assume a prior discrete distribution $Dis(\pi)$ with parameter $\pi$ over the class $s$ for each datum.

$$
\begin{align*}
s_i & \sim Dis(\pi) \\
X_i & \sim \mathcal{N}\left( \mu_{s_i}, \Sigma_{s_i}\right)
\end{align*}
$$

Here's some code that can generate data from this model.

```python
def sample_GMM(mu, sigma, pi, num_data):
    # mu, sigma and pi are lists of length k
    assert len(mu) == len(sigma)
    assert len(mu) == len(pi)

    K = len(mu)
    D = len(mu[0])
    X = np.zeros([num_data, D])

    # random sample from multinomial distribution to see how many samples we should generate for each class
    class_nums = np.random.multinomial(num_data, pi)
    for k in range(K):
        # Fill in X in order of classes.... we do this so that we only have to call sample_gaussian K times
        lower = sum(class_nums[0:k])
        upper = sum(class_nums[0:k+1])
        X[lower:upper,:] = \
                   sample_gaussian(covariance=sigma[k], mean=mu[k], n_data=class_nums[k])

    # Shuffle rows of X so that classes are not in order
    np.random.shuffle(X)
    return X

```

```python
mu = [np.array([-20,-20]), np.array([10,10]), np.array([-10,10])]
sigma = [np.array([[10,0],[0,10]]), np.array([[10,0],[0,1]]), np.array([[1,0],[0,10]])]
pi = [0.7, 0.2, 0.1]

X = sample_GMM(mu, sigma, pi, 1000)

plt.scatter(X[:,0], X[:,1])
plt.show()
```

Given any observation $X_i$ and values for the parameters $\pi, \mu_k, \Sigma_k$ for all $k$, we can calculate the posterior probability that $X_i$ belongs to class $k$ using Bayes' rule:

$$
\begin{align*}
P(s_i = k | X_i) &= \frac{P(X_i | s_i = k) P(s_i = k)}{P(X_i)} \\
                 &\propto \mathcal{N}\left(X_i | \mu_k, \Sigma_k \right) \pi_k
\end{align*}
$$

How should we choose the values for the parameters $\pi, \mu_k$ and $\Sigma_k$? We'll optimise the margingal likelihood (the probability of the data given the parameters). To make the maths a bit simpler, we'll actually consider the marginal log-likelihood. This has the advantage of products into sums (making things more numerically stable when we would otherwise have the product of lots of small probabilities), and since $log$ is a monotonically increasing function, maximising the marginal log-likelihood will give the same parameters as maximising the marginal likelihood.

$$
\begin{align*}
\log P(X |\pi, \mu, \Sigma) &=  \sum_{i=1}^{N} \log P(X_i |\pi, \mu, \Sigma) \\
                           &=  \sum_{i=1}^{N} \log \left( \sum_k P(X_i |\mu_k, \Sigma_k) P(s_i = k)  \right) \\
                           &=  \sum_{i=1}^{N} \log \left( \sum_k \mathcal{N}(X_i |\mu_k, \Sigma_k) \pi_k  \right)
\end{align*}
$$

Unfortunately, this would be difficult to directly optimise. Instead, we can use the **EM algorithm**  to optimise a lower bound on this.

# The EM algorithm

We'll breifly explain the EM algorithm here. For models with latent varibles such as the above GMM, it can be difficult to optimise the marginal log-likelihood with respect to the parameters. However, if we knew the latent variables for each observation, this would be easy. Similarly, if we knew the parameters, then finding the latent variables for each observations would be easy. EM works by iteratively performing these two updates, optimising a lower bound on the log-likelihood called the _Free Energy_. The Free Energy is a function of the parameters $\theta$ of the model and an approximating distribution $Q(Y)$ for the posteriors over the latents:

$$
\begin{align*}
\log P(X | \theta) &= \log \int P(X, Y | \theta) dY \\
                   &= \log \int  \frac{P(X, Y| \theta)}{ Q(Y)} Q(Y) dY \\
                   &\geq \int  \log \left(\frac{P(X, Y| \theta)}{ Q(Y)}\right) Q(Y) dY =: F(Q,\theta)
\end{align*}
$$

In an __Expectation__ step, we maximise $F$ with respect to $Q$, keeping $\theta$ fixed. Noting that

$$
\begin{align*}
F(Q,\theta) &= \int  \log \left(\frac{P(X, Y| \theta)}{ Q(Y)}\right) Q(Y) dY \\
            &= \int  \log \left( \frac{P(Y|X, \theta) P(X|\theta)}{Q(Y)} \right) Q(Y) dY \\
            &= \log P(X|\theta) + \int  \log \left( \frac{P(Y|X, \theta)}{Q(Y)} \right) Q(Y) dY \\
            &= \log P(X|\theta) - KL\left[Q(Y) || P(Y|X, \theta) \right]\\
\end{align*}
$$

Since KL divergences are always greater than zero, maximising $F$ with respect to $Q$ reduces to minimising the KL on the right hand side. This term equals zero if and only if $Q(Y)$ is set to be the posteriors of the latents given the current parameters.

In a __Maximisation__ step, we maximise $F$ with respect to $\theta$, keeping $Q$ fixed. Noting that

$$
\begin{align*}
F(Q,\theta) &= \int  \log \left(\frac{P(X, Y| \theta)}{ Q(Y)}\right) Q(Y) dY \\
            &= \int  \log \left( P(X, Y| \theta) \right) Q(Y) dY - \int  \log \left(Q(Y)\right) Q(Y) dY \\
            &= \mathbb{E}_{Y\sim Q} \left[\log P(X, Y| \theta)\right]  + H[Q]
\end{align*}
$$

where $H[Q]$ is the _entropy_ of $Q$, we see maximising $F$ with respect to $\theta$ reduces to maximising $\mathbb{E}_{Y\sim Q} \left[\log P(X, Y| \theta)\right]$

## EM for GMM

It's relatively straightforward to derive the E and M updates for the Gaussian Mixture Model.

__E-step__:

For the $i$th datum $X_i$, the posterior probability it belongs to class $k$ is

$$
\begin{align*}
P(s_i = k | X_i) &= \frac{P(X_i | s_i = k) P(s_i = k)}{P(X_i)} \\
                 &\propto \mathcal{N}\left(X_i | \mu_k, \Sigma_k \right) \pi_k
\end{align*}
$$

We'll write $r_{ik} = P(s_i = k | X_i)$ for shorthand (r stands for _responsibility)

__M-step__:

Given the posteriors over the latents, we have that.... TODO: tidy this up...
$$
\begin{align*}
\mathbb{E}_{Y\sim Q} \left[\log P(X, Y| \theta)\right] &= \sum_{i,k} r_{ik} \log\left( \pi_k \mathcal{N}(X_i| \mu_k, \Sigma_k) \right) \\
&= \sum_{i,k} r_{ik} \left[ \log(\pi_k) - \log( + \left( (X_i - \mu_k) \Sigma_k^{-1} (X_i - \mu_k)^{\intercal} \right) \right]\\
\end{align*}
$$

$$\pi_k = \frac{\sum_i r_{ik}}{\sum_{ik} r_{ik}}$$

$$ \mu_k = \frac{\sum_i r_{ik} X_i}{\sum_i r_{ik}} $$

$$\Sigma_k = \frac{\sum_i r_{ik} \left(X_i - \mu_k \right)\left(X_i - \mu_k \right)^\intercal}{\sum_i r_{ik}} $$

The following code implements these updates, which we can wrap together into a function to iteratively perform these updates until all of the parameters have stopped changing.

```python

```

```python
def E_step(X, pi, mu, Sigma):
    K = len(pi)
    N = len(X)
    r = np.zeros([N,K])

    # for each component, calculate probability density for all X...
    for k in range(K):
        r[:,k] = scipy.stats.multivariate_normal.pdf(cov=Sigma[k], mean=mu[k], x=X)
    # ... then normalise each row
    r = r / r.sum(axis=1)[:,None]
    return r

def M_step(X, r):
    N = len(r)
    K = len(r[0])

    pi = list(r.sum(axis=0) / r.sum())
    mu = list((r.T).dot(X)/r.sum(axis=0)[:,None])
    Sigma = [None]*K

    for k in range(K):
        mu_k = mu[k]
        Sigma[k] = ((r[:,k][:,None]*(X - mu[k])).T).dot(X-mu[k])/ r[:,k].sum()
    return pi, mu, Sigma

def fit_GMM(X, K):
    # K = number of components
    # Initialise means to be random K data points
    mu = list(X[np.random.choice(len(X),K,replace=False)])
    sigma = [5*np.identity(len(X[0]))] * K
    pi = [1./K] * K

    r = E_step(X, pi, mu, sigma)

    mu_old = mu
    sigma_old = sigma
    pi_old = pi
    r_old = r

    pi, mu, sigma = M_step(X, r)

    while ((np.array(mu_old) - np.array(mu))**2).sum() + \
                ((np.array(sigma_old) - np.array(sigma))**2).sum() + \
                ((np.array(pi_old) - np.array(pi))**2).sum() + \
                ((np.array(r_old) - np.array(r))**2).sum()> 1e-25:
        mu_old = mu
        sigma_old = sigma
        pi_old = pi
        r_old = r

        r = E_step(X, pi, mu, sigma)
        pi, mu, sigma = M_step(X, r)

    return pi, mu, sigma

```

We'll also write a plotting function that will help us visualise the clusters that we find.

```python
def plotter_GMM(X, mu, sigma):
    fig = plt.figure(figsize=(6, 6))
    ax = fig.add_subplot(111, aspect='equal')
    ax.scatter(X[:,0], X[:,1])
    for k in range(len(mu)):
        eigvals, eigvecs = np.linalg.eig(sigma[k])
        eigvals = np.sqrt(eigvals)
        ell = Ellipse(xy=mu[k],
                      width=eigvals[0]*4, height=eigvals[1]*4,
                      angle=np.rad2deg(np.arccos(eigvecs[0, 0])),
                      color='black')
        ell.set_facecolor('none')
        ax.add_artist(ell)

    plt.axis('equal')
    plt.show()
```

```python
np.random.seed(0)
pi, mu, sigma = fit_GMM(X, 3)
print pi

plotter_GMM(X, mu, sigma)
```

Note that the clusters that our algorithm finds are highly dependent on the initial choice of parameters. In the above case, it would appear that two of the initial cluster seeds were chosen to be from one of the clusters, and since EM is vulnerable to be stuck in local minimima, it is not able to move either of the centres away.

A sensible idea would be to try multiple choices for initial values. At the end of each iteration, we could calculate the marginal log-likelihood corresponding to the choice of parameters we found, and choose the final parameters that give the largest marginal log-likelihood.

When we have a good choice of initial parameters though, the algorithm appears to perform well:

```python
np.random.seed(7)
pi, mu, sigma = fit_GMM(X, 3)
print pi

plotter_GMM(X, mu, sigma)
```

Also, we see that the earlier problem with $k$-means is resolved using our more advanced GMM:

```python
mean1 = np.array([0,0])
cov1 = np.array([[1,0],[0,1]])
data1 = sample_gaussian(mean1, cov1, 100)

mean2 = np.array([0,10])
cov2 = np.array([[1,0],[0,1]])
data2 = sample_gaussian(mean2, cov2, 100)

mean3 = np.array([10,5])
cov3 = np.array([[1,0],[0,50]])
data3 = sample_gaussian(mean3, cov3, 100)

data = np.concatenate([data1, data2, data3])
pi, mu, sigma = fit_GMM(data, 3)
plotter_GMM(data, mu, sigma)
```

```python

```


# Radioactive decay

- toc: false
- branch: master
- badges: false
- comments: false
- categories: [mathematics, numerical recipes]
- hide: true

----
Questions:
- How can I describe radioactive decay using a first-order ODE?
- What are initial conditions and why are  they important?
----

----
Objectives:
- Map between physical notation for a particular problem and the more general notation for all differential equations
- Solve a linear, first order, separable ODE using integration
- Understand the physical importance of initial conditions
----

### Radioactive decay can be modelled a linear, first-order ODE

As our first example of an ODE we will model radioactive decay using a differential equation.

We know that the decay rate is proportional to the number of atoms present.  Mathematically, this relationship can be expressed as:

\begin{equation}
\frac{d N}{d t} = -\lambda N
\end{equation}

Note that we could choose to use different variables, for example:

\begin{equation}
\frac{d y}{d x} = cy
\end{equation}

However we try to use variables connected to the context of the problem. For example $N$ for the Number of atoms.

For example,  if we know that 10% of atoms will decay per second we could write:

\begin{equation}
\frac{d N}{d t} = -0.1 N
\end{equation}

where $N$ is the number of atoms and $t$ is time measured in seconds.

This equation is linear and first-order.

| physical notation | generic notation |
|-----|-----|
|number of  atoms $N$ | dependent  variable $y$|
| time $t$ | independent variable  $x$|
| decay rate $\frac{dN}{dt}$ | differential $\frac{dy}{dx}$|
| constant of proportionality $\lambda=0.1 $ | parameter $c$ |

### The equation for radioactive decay is separable and has an analytic solution

The radioactive decay equation is separable. For example,

\begin{equation}
\frac{d N}{d t} = -\lambda N
\end{equation}

Can be seperated as

\begin{equation}
\frac{dN}{N} = -\lambda dt.
\end{equation}

We can then integrate each side:

\begin{equation}
\ln N = -\lambda t + const.
\end{equation}

and solve for N:

\begin{equation}
N = e^{-\lambda t}e^{\textrm{const.}}
\end{equation}

> Note: Remember that $\int \frac{1}{x} dx = \ln x + \textrm{const.}$

### To model a physical system an initial value has to be provided

At the beginning (when $t=0$):

\begin{equation}
N = e^{-\lambda t}e^{\textrm{const.}} = e^{0}e^{\textrm{const.}} = e^{\textrm{const.}}
\end{equation}

So we can identify $e^{\textrm{const.}}$ as the amount of radioactive material that was present in the beginning. We denote this starting amount as $N_0$.

Substituting this back into Equation 4, the final solution can be more meaningfully written as:

\begin{equation}
N = N_0 e^{-\lambda t}
\end{equation}

We now have  not  just one solution, but a whole class  of solutions that are dependent on the initial amount of radioactive material $N_0$.

Remember that not all mathematical solutions make physical sense. To model a physical system, this initial value (also known as initial condition) has to be provided alongside the constant of proportionality $\lambda$.

### ODEs can have initial values or boundary values

ODEs have either initial values or boundary values. For example, using Newton's second laws we could calculate the distance $x$ an object travels under the influence of gravity over time $t$

\begin{equation}
\frac{\mathrm{d}^2x}{\mathrm{d}t^2} = -g
\end{equation}

An initial value problem would be where we know the starting position and velocity. A boundary value problem would be where we specify the position of the ball at times $t=t_0$ and $t=t_1$.

In this course we will only study ODEs with initial values. ODEs with boundary values are more difficult to solve, but you can related materials listed under `External resources`.

### The number of  initial conditions depends on the order of  the differential equation

Our radioactive decay example is a first-order ODE and so we only had to provide a single initial condition. For second-order ODEs (such as acceleration under gravity) we need to provide two initial/boundary conditions, for third-order ODEs we would need to provide three, and so on.

-----
Keypoints:
- Radioactive decay can be modelled a linear, first-order ODE
- The equation for radioactive decay is separable and has an analytic solution
- To model a physical system an initial value has to be provided
- The number of  initial conditions depends on the order of  the differential equation
-----

---

Do [the quick-test](https://nu-cem.github.io/CompPhys/2021/08/02/Radioactive-Decay-Qs.html).

Back to [Modelling with Ordinary Differential Equations](https://nu-cem.github.io/CompPhys/2021/08/02/ODEs.html).

---


```python
import sympy as sm
from scipy.misc import derivative
```

```python
x, y, z = sm.symbols('x y z')
```

```python
#Function - 3x**2
sm.diff(3*x**2)
```

$\displaystyle 6 x$

```python
sm.diff(2*x)
```

$\displaystyle 2$

```python
def myFunc(x):
    return 2*x
```

```python
derivative(myFunc,4)
```

    2.0

```python
#Function 3x^2+5x
sm.diff(3*x**2+5*x)
```

$\displaystyle 6 x + 5$

```python
#Product Rule
sm.diff((x**2 + 1) * sm.cos(x))
```

$\displaystyle 2 x \cos{\left(x \right)} - \left(x^{2} + 1\right) \sin{\left(x \right)}$

```python
#Chain Rule
sm.diff((x**2 - 3*x + 5) ** 3)
```

$\displaystyle \left(6 x - 9\right) \left(x^{2} - 3 x + 5\right)^{2}$

```python
#Partial Derivative
f = x**2 * y * z**5
```

```python
sm.diff(f,x)
```

$\displaystyle 2 x y z^{5}$

```python
sm.diff(f,y)
```

$\displaystyle x^{2} z^{5}$

```python
sm.diff(f,z)
```

$\displaystyle 5 x^{2} y z^{4}$

```python

```


# Introduction to Graph Matching

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

The graph matching problem (GMP), is meant to find an alignment of nodes between two graphs that minimizes the number of edge disagreements between those two graphs. Therefore, the GMP can be formally written as an optimization problem:

\begin{equation}
\begin{aligned}
\min & {\;-trace(APB^T P^T)}\\
\text{s.t. } & {\;P \: \epsilon \: \mathcal{P}} \\
\end{aligned}
\end{equation}

Where $\mathcal{P}$ is the set of possible permutation matrices.

The Quadratic Assignment problem is a combinatorial opimization problem, modeling following the real-life problem:

"Consider the problem of allocating a set of facilities to a set of locations, with the
cost being a function of the distance and flow between the facilities, plus costs associated
with a facility being placed at a certain location. The objective is to assign each facility
to a location such that the total cost is minimized." [1]

When written as an optimization problem, the QAP is represented as:

\begin{equation}
\begin{aligned}
\min & {\; trace(APB^T P^T)}\\
\text{s.t. } & {\;P \: \epsilon \: \mathcal{P}} \\
\end{aligned}
\end{equation}

Since the GMP objective function is the negation of the QAP objective function, any algorithm that solves one can solve the other.

This class is an implementation of the Fast Approximate Quadratic Assignment Problem (FAQ), an algorithm designed to efficiently and accurately solve the QAP, as well as GMP.

[1] Optimierung, Diskrete & Er, Rainer & Ela, A & Burkard, Rainer & Dragoti-Cela, Eranda & Pardalos, Panos & Pitsoulis, Leonidas. (1998). The Quadratic Assignment Problem. Handbook of Combinatorial Optimization. 10.1007/978-1-4613-0303-9_27.

```python
from graspy.match import GraphMatch as GMP
from graspy.simulations import er_np
```

For the sake of this tutorial, we will use FAQ to solve the GMP for two graphs where we know a solution exists.
Below, we sample a binary graph (undirected and no self-loops) $G_1 \sim ER_{NP}(50, 0.3)$.
Then, we randomly shuffle the nodes of $G_1$ to initiate $G_2$.
The number of edge disagreements as a result of the node shuffle is printed below.

```python
n = 50
p = 0.3

np.random.seed(1)
G1 = er_np(n=n, p=p)
node_shuffle_input = np.random.permutation(n)
G2 = G1[np.ix_(node_shuffle_input, node_shuffle_input)]
print("Number of edge disagreements: ", np.sum(abs(G1-G2)))
```

## Visualize the graphs using heat mapping

```python
from graspy.plot import heatmap
heatmap(G1, cbar=False, title = 'G1 [ER-NP(50, 0.3) Simulation]')
heatmap(G2, cbar=False, title = 'G2 [G1 Randomly Shuffled]')
```

Below, we create a model to solve GMP. The model is then fitted for the two graphs $G_1$ and $G_2$. One of the option for the algorithm is the starting position of $P$. In this case, the class default of barycenter intialization is used, or the flat doubly stochastic matrix. The number of edge disagreements is printed below. With zero edge disagreements, we see that FAQ is successful in unshuffling the graph.

```python
gmp = GMP()
gmp = gmp.fit(G1,G2)
G2 = G2[np.ix_(gmp.perm_inds_, gmp.perm_inds_)]
print("Number of edge disagreements: ", np.sum(abs(G1-G2)))
```

```python
heatmap(G1, cbar=False, title = 'G1[ER-NP(50, 0.3) Simulation]')
heatmap(G2, cbar=False, title = 'G2[ER-NP(50, 0.3) Randomly Shuffled] unshuffled')
```


<form action="javascript:code_toggle()"><input type="submit" id="toggleButton" value="Show Code"></form>

```python
# Importing some python libraries.
import numpy as np
from numpy.random import randn,rand
import matplotlib.pyplot as pl

from matplotlib.pyplot import plot

import seaborn as sns
%matplotlib inline

# Fixing figure sizes
from pylab import rcParams
rcParams['figure.figsize'] = 10,5

sns.set_palette('Reds_r')
```

# Reaction Network Homework

In this homework, we will study a very simple set of reactions by modelling it through three different ways. First, we shall employ an ODE model called the **Reaction Rate Equation**.  Then, we will solve the **Chemical Langevin Equation** and, finally, we will simulate the exact model by "solving" the **Chemical  Master Equation**.

The reaction network of choice shall be a simple birth-death process, described by the relations :

$$
\begin{align}
\emptyset \stackrel{a}{\to} X,\\
X \stackrel{\mu X}{\to} \emptyset.
\end{align}
$$

$X$ here is the population number.

Throughout, we shall use $a=10$ and $\mu=1.0$.

## Reaction Rate Equation

The reaction rate equation corresponding to the system is

$$
\begin{align}
\dot{x}=a-\mu\cdot x,\\
x(0)=x_0.
\end{align}
$$

As this is a linear equation, we can solve it exactly, with solution

$$
x(t)=x(t) = a/\mu+(x_0-a/\mu) e^{\mu (-t)}
$$

```python
# Solution of the RRE
def x(t,x0=3,a=10.0,mu=1.0):
    return (x0-a/mu)*np.exp(-t*mu)+a/mu

```

We note that there is a stationary solution, $x(t)=a/\mu$. From the exponential in the solution, we can see that this is an attracting fixed point.

```python
t = np.linspace(0,3)
x0list = np.array([0.5,1,15])

sns.set_palette("Reds",n_colors=3)

for x0 in x0list:
 pl.plot(t,x(t,x0),linewidth=4)

pl.title('Population numbers for different initial conditions.', fontsize=20)
pl.xlabel('Time',fontsize=20)
```

# Chemical Langevin Equation

Next, we will model the system by using the CLE. For our particular birth/death process, this will be

$$
dX_t=(a-\mu\cdot X_t)dt+(\sqrt{a}-\sqrt{\mu\cdot X_t})dW.
$$

To solve this, we shall use the Euler-Maruyama scheme from the previous homework. We fix a $\Delta t$ positive. Then, the scheme shall be :

$$
X_{n+1}=X_n+(a-\mu\cdot X_n)\Delta t+(\sqrt{a}-\sqrt{\mu\cdot X_t})\cdot \sqrt{\Delta t}\cdot z,\ z\sim N(0,1).
$$

```python
def EM(xinit,T,Dt=0.1,a=1,mu=2):
    '''
        Returns the solution of the CLE with parameters a, mu

        Arguments
        =========
        xinit : real, initial condition.
        Dt    : real, stepsize of the Euler-Maruyama.
        T     : real, final time to reach.
        a     : real, parameter of the RHS.
        mu    : real, parameter of the RHS.

    '''

    n = int(T/Dt) # number of steps to reach T
    X = np.zeros(n)
    z = randn(n)

    X[0] = xinit # Initial condition

    # EM method
    for i in xrange(1,n):
        X[i] = X[i-1] + Dt* (a-mu*X[i-1])+(np.sqrt(a)-np.sqrt(mu*X[i-1]))*np.sqrt(Dt)*z[i]

    return X

```

Similarly to the previous case, here is a run with multiple initial conditions.

```python
T = 10 # final time to reach
Dt = 0.01 # time-step for EM

# Set the palette to reds with ten colors
sns.set_palette('Reds',10)

def plotPaths(T,Dt):
    n = int(T/Dt)
    t = np.linspace(0,T,n)

    xinitlist = np.linspace(10,15,10)

    for x0 in xinitlist :
        path = EM(xinit=x0,T=T,Dt=Dt,a=10.0,mu=1.0)
        pl.plot(t, path,linewidth=5)

    pl.xlabel('time', fontsize=20)
    pl.title('Paths for initial conditions between 1 and 10.', fontsize=20)

    return path

path = plotPaths(T,Dt)

print 'Paths decay towards', path[np.size(path)-1]
print 'The stationary point is', 1.0
```

We notice that the asymptotic behavior of the CLE is the same as that of the RRE. The only notable difference is the initial random kicks in the paths, all because of the stochasticicity.

## Chemical Master Equation

Finally, we shall simulate the system exactly by using the Stochastic Simulation Algorithm (SSA).

```python
def SSA(xinit, nsteps, a=10.0, mu=1.0):
    '''
        Using SSA to exactly simulate the death/birth process starting
        from xinit and for nsteps.

        a and mu are parameters of the propensities.

        Returns
        =======
        path : array-like, the path generated.
        tpath: stochastic time steps
    '''

    path = np.zeros(nsteps)
    tpath= np.zeros(nsteps)

    path[0] = xinit # initial population

    u = rand(2,nsteps) # pre-pick all the uniform variates we need

    for i in xrange(1,nsteps):

        # The propensities will be normalized
        tot_prop = path[i-1]*mu+a
        prob = path[i-1]*mu/tot_prop # probability of death

        if(u[0,i]<prob):
            # Death
            path[i] = path[i-1]-1
        else:
            # Birth
            path[i] = path[i-1]+1

        # Time stayed at current state
        tpath[i] = -np.log(u[1,i])*1/tot_prop

    tpath = np.cumsum(tpath)
    return path, tpath
```

Now that we have SSA setup, we can run multiple paths and compare the results to the previous cases.

```python
# Since the paths below are not really related
# let's use a more interesting palette
# for the plot.

sns.set_palette('hls',1)

for _ in xrange(1):
    path, tpath = SSA(xinit=1,nsteps=100)

    # Since this is the path of a jump process
    # I'm switching from "plot" to "step"
    # to get the figure right. :)
    pl.step(tpath,path,linewidth=5,alpha=0.9)

pl.title('One path simulated with SSA, $a<\mu$. ', fontsize=20);
pl.xlabel('Time', fontsize=20)

```

```python
# Since the paths below are not really related
# let's use a more interesting palette
# for the plot.

sns.set_palette('hls',3)

for _ in xrange(3):
    path, tpath = SSA(xinit=1,nsteps=100)

    # Since this is the path of a jump process
    # I'm switching from "plot" to "step"
    # to get the figure right. :)
    pl.step(tpath,path,linewidth=5,alpha=0.9)

pl.title('Three paths simulated with SSA, $a<\mu$. ', fontsize=20);
pl.xlabel('Time', fontsize=20)

```

We can see three chains above, all starting from $X_0=1$, and simulated with the SSA.

```python
npaths = 1
nsteps = 30000

path = np.zeros([npaths,nsteps])
for i in xrange(npaths):
    path[i,:], tpath = SSA(xinit=1,nsteps=nsteps)

skip = 20000
sum(path[0,skip:nsteps-1]*tpath[skip:nsteps-1])/sum(tpath[skip:nsteps-1])
```

    10.550243180622136


# How do distributions transform under a change of variables ?

Kyle Cranmer, March 2016

```python
%pylab inline --no-import-all
```

    Populating the interactive namespace from numpy and matplotlib

We are interested in understanding how distributions transofrm under a change of variables.
Let's start with a simple example. Think of a spinner like on a game of twister.

<!---->

We flick the spinner and it stops. Let's call the angle of the pointer $x$. It seems a safe assumption that the distribution of $x$ is uniform between $[0,2\pi)$... so $p_x(x) = 1/\sqrt{2\pi}$

Now let's say that we change variables to $y=\cos(x)$ (sorry if the names are confusing here, don't think about x- and y-coordinates, these are just names for generic variables).  The question is this:
**what is the distribution of y?**  Let's call it $p_y(y)$

Well it's easy to do with a simulation, let's try it out

```python
# generate samples for x, evaluate y=cos(x)
n_samples = 100000
x = np.random.uniform(0,2*np.pi,n_samples)
y = np.cos(x)
```

```python
# make a histogram of x
n_bins = 50
counts, bins, patches = plt.hist(x, bins=50, density=True, alpha=0.3)
plt.plot([0,2*np.pi], (1./2/np.pi, 1./2/np.pi), lw=2, c='r')
plt.xlim(0,2*np.pi)
plt.xlabel('x')
plt.ylabel('$p_x(x)$')
```

Ok, now let's make a histogram for $y=\cos(x)$

```python
counts, y_bins, patches = plt.hist(y, bins=50, density=True, alpha=0.3)
plt.xlabel('y')
plt.ylabel('$p_y(y)$')
```

It's not uniform! Why is that?  Let's look at the $x-y$ relationship

```python
# make a scatter of x,y
plt.scatter(x[:300],y[:300]) #just the first 300 points

xtest = .2
plt.plot((-1,xtest),(np.cos(xtest),np.cos(xtest)), c='r')
plt.plot((xtest,xtest),(-1.5,np.cos(xtest)), c='r')
xtest = xtest+.1
plt.plot((-1,xtest),(np.cos(xtest),np.cos(xtest)), c='r')
plt.plot((xtest,xtest),(-1.5,np.cos(xtest)), c='r')

xtest = 2*np.pi-xtest
plt.plot((-1,xtest),(np.cos(xtest),np.cos(xtest)), c='g')
plt.plot((xtest,xtest),(-1.5,np.cos(xtest)), c='g')
xtest = xtest+.1
plt.plot((-1,xtest),(np.cos(xtest),np.cos(xtest)), c='g')
plt.plot((xtest,xtest),(-1.5,np.cos(xtest)), c='g')

xtest = np.pi/2
plt.plot((-1,xtest),(np.cos(xtest),np.cos(xtest)), c='r')
plt.plot((xtest,xtest),(-1.5,np.cos(xtest)), c='r')
xtest = xtest+.1
plt.plot((-1,xtest),(np.cos(xtest),np.cos(xtest)), c='r')
plt.plot((xtest,xtest),(-1.5,np.cos(xtest)), c='r')

xtest = 2*np.pi-xtest
plt.plot((-1,xtest),(np.cos(xtest),np.cos(xtest)), c='g')
plt.plot((xtest,xtest),(-1.5,np.cos(xtest)), c='g')
xtest = xtest+.1
plt.plot((-1,xtest),(np.cos(xtest),np.cos(xtest)), c='g')
plt.plot((xtest,xtest),(-1.5,np.cos(xtest)), c='g')

plt.ylim(-1.5,1.5)
plt.xlim(-1,7)
```

The two sets of vertical lines are both separated by $0.1$. The probability  $P(a < x < b)$ must equal the probability of $P( cos(b) < y < cos(a) )$. In this example there are two different values of $x$ that give the same $y$ (see green and red lines), so we need to take that into account. For now, let's just focus on the first part of the curve with $x<\pi$.

So we can write (this is the important equation):

\begin{equation}
\int_a^b p_x(x) dx = \int_{y_b}^{y_a} p_y(y) dy
\end{equation}
where $y_a = \cos(a)$ and $y_b = \cos(b)$.

and we can re-write the integral on the right by using a change of variables (pure calculus)

\begin{equation}
\int_a^b p_x(x) dx = \int_{y_b}^{y_a} p_y(y) dy = \int_a^b p_y(y(x))  \left| \frac{dy}{dx}\right| dx
\end{equation}

notice that the limits of integration and integration variable are the same for the left and right sides of the equation, so the integrands must be the same too. Therefore:

\begin{equation}
p_x(x) = p_y(y)  \left| \frac{dy}{dx}\right|
\end{equation}
and equivalently
\begin{equation}
p_y(y) = p_x(x) \,/ \,\left| \, {dy}/{dx}\, \right |
\end{equation}

The factor $\left|\frac{dy}{dx} \right|$ is called a Jacobian. When it is large it is stretching the probability in $x$ over a large range of $y$, so it makes sense that it is in the denominator.

```python
plt.plot((0.,1), (0,.3))
plt.plot((0.,1), (0,0), lw=2)
plt.plot((1.,1), (0,.3))
plt.ylim(-.1,.4)
plt.xlim(-.1,1.6)
plt.text(0.5,0.2, '1', color='b')
plt.text(0.2,0.03, 'x', color='black')
plt.text(0.5,-0.05, 'y=cos(x)', color='g')
plt.text(1.02,0.1, '$\sin(x)=\sqrt{1-y^2}$', color='r')
```

In our case:
\begin{equation}
\left|\frac{dy}{dx} \right| =  \sin(x)
\end{equation}

Looking at the right-triangle above you can see $\sin(x)=\sqrt{1-y^2}$ and finally there will be an extra factor of 2 for $p_y(y)$ to take into account $x>\pi$. So we arrive at
\begin{equation}
p_y(y) = 2 \times \frac{1}{2 \pi} \frac{1}{\sin(x)} = \frac{1}{\pi} \frac{1}{\sin(\arccos(y))} = \frac{1}{\pi} \frac{1}{\sqrt{1-y^2}}
\end{equation}

  Notice that when $y=\pm 1$ the pdf is diverging. This is called a [caustic](http://www.phikwadraat.nl/huygens_cusp_of_tea/) and you see them in your coffee and rainbows!

|   |   |
|---|---|
|   |   |

**Let's check our prediction**

```python
counts, y_bins, patches = plt.hist(y, bins=50, density=True, alpha=0.3)
pdf_y = (1./np.pi)/np.sqrt(1.-y_bins**2)
plt.plot(y_bins, pdf_y, c='r', lw=2)
plt.ylim(0,5)
plt.xlabel('y')
plt.ylabel('$p_y(y)$')
```

Perfect!

## A trick using the cumulative distribution function (cdf) to generate random numbers

Let's consider a different variable transformation now -- it is a special one that we can use to our advantage.
\begin{equation}
y(x) = \textrm{cdf}(x) = \int_{-\infty}^x p_x(x') dx'
\end{equation}

Here's a plot of a distribution and cdf for a Gaussian.

(NOte: the axes are different for the pdf and the cdf http://matplotlib.org/examples/api/two_scales.html

```python
from scipy.stats import norm
```

```python
x_for_plot = np.linspace(-3,3, 30)
fig, ax1 = plt.subplots()

ax1.plot(x_for_plot, norm.pdf(x_for_plot), c='b')
ax1.set_ylabel('p(x)', color='b')
for tl in ax1.get_yticklabels():
    tl.set_color('b')

ax2 = ax1.twinx()
ax2.plot(x_for_plot, norm.cdf(x_for_plot), c='r')
ax2.set_ylabel('cdf(x)', color='r')
for tl in ax2.get_yticklabels():
    tl.set_color('r')
```

Ok, so let's use our result about how distributions transform under a change of variables to predict the distribution of $y=cdf(x)$. We need to calculate

\begin{equation}
\frac{dy}{dx} = \frac{d}{dx} \int_{-\infty}^x p_x(x') dx'
\end{equation}

Just like particles and anti-particles, when derivatives meet anti-derivatives they annihilate. So $\frac{dy}{dx} = p_x(x)$, which shouldn't be a surprise.. the slope of the cdf is the pdf.

So putting these together we find the distribution for $y$ is:

\begin{equation}
p_y(y) = p_x(x) \, / \, \frac{dy}{dx} = p_x(x) /p_x(x) = 1
\end{equation}

So it's just a uniform distribution from $[0,1]$, which is perfect for random numbers.

We can turn this around and generate a uniformly random number between $[0,1]$, take the inverse of the cdf and we should have the distribution we want for $x$.

Let's try it for a Gaussian. The inverse of the cdf for a Gaussian is called [ppf](http://docs.scipy.org/doc/scipy-0.15.1/reference/generated/scipy.stats.norm.html)

```python
norm.ppf.__doc__
```

    '\n        Percent point function (inverse of `cdf`) at q of the given RV.\n\n        Parameters\n        ----------\n        q : array_like\n            lower tail probability\n        arg1, arg2, arg3,... : array_like\n            The shape parameter(s) for the distribution (see docstring of the\n            instance object for more information)\n        loc : array_like, optional\n            location parameter (default=0)\n        scale : array_like, optional\n            scale parameter (default=1)\n\n        Returns\n        -------\n        x : array_like\n            quantile corresponding to the lower tail probability q.\n\n        '

```python
#check it out
norm.cdf(0), norm.ppf(0.5)
```

    (0.5, 0.0)

Ok, let's use CDF trick to generate Normally-distributed (aka Gaussian-distributed) random numbers

```python
rand_cdf = np.random.uniform(0,1,10000)
rand_norm = norm.ppf(rand_cdf)
```

```python
_ = plt.hist(rand_norm, bins=30, density=True, alpha=0.3)
plt.xlabel('x')
```

**Pros**: The great thing about this technique is it is very efficient. You only generate one random number per random $x$.

**Cons**: the downside is you need to know how to compute the inverse cdf for $p_x(x)$ and that can be difficult. It works for a distribution like a Gaussian, but for some random distribution this might be even more computationally expensive than the accept/reject approach. This approach also doesn't really work if your distribution is for more than one variable.

## Going full circle

Ok, let's try it for our distribution of $y=\cos(x)$ above.  We found

\begin{equation}
p_y(y) = \frac{1}{\pi} \frac{1}{\sqrt{1-y^2}}
\end{equation}

So the CDF is  (see Wolfram alpha for [integral](http://www.wolframalpha.com/input/?i=integrate%5B1%2Fsqrt%5B1-x%5E2%5D%2FPi%5D) )
\begin{equation}
cdf(y') = \int_{-1}^{y'} \frac{1}{\pi} \frac{1}{\sqrt{1-y^2}} = \frac{1}{\pi}\arcsin(y') + C
\end{equation}
and we know that for $y=-1$ the CDF must be 0, so the constant is $1/2$ and by looking at the plot or remembering some trig you know that it's also $cdf(y') = (1/\pi) \arccos(y')$.

So to apply the trick, we need to generate uniformly random variables $z$ between 0 and 1, and then take the inverse of the cdf to get $y$. Ok, so what would that be:
\begin{equation}
y = \textrm{cdf}^{-1}(z) = \cos(\pi z)
\end{equation}

**Of course!** that's how we started in the first place, we started with a uniform $x$ in $[0,2\pi]$ and then defined $y=\cos(x)$. So we just worked backwards to get where we started.  The only difference here is that we only evaluate the first half: $\cos(x < \pi)$

```python

```


# Week 2
## Introduction to Object-Oriented Programming

These weeks exercises starts you working with classes. If you want a gentler introduction, exercises for Chapter 7 in Langtangen is recommended.

## Exercise 1 — Quadratic functions

In this exercise, we will build on the example given in the lectures of implementing 2nd degree polynomials as objects of a custom defined class. A general 2nd degree polynomial, aka, quadratic function, can be written as:

$$ f(x) = a_2x^2 + a_1x + a_0,$$

where the coefficients, $a_2$, $a_1$, and $a_0$ uniquely defines the polynomial.

### Exercise 1a) Defining the `Quadratic` class

Create a class, `Quadratic`, that represents a general 2nd degree polynomial. Define the following methods:
* A constructor (`__init__`)
* A call method (`__call__`)
The constructor should take in the three coefficients in order: `a_2`, `a_1`, and `a_0`, and the call method should take the free variable `x`.

You class should be able to handle the following test script:
***
```Python
f = Quadratic(1, -2, 1)
x = np.linspace(-5, 5, 101)
plt.plot(x, f(x))
plt.show()
```
***
Implement your solution here:

```python

```

Use this to test your implementation:

```python
def test_Quadratic():
    f = Quadratic(1, -2, 1)
    assert abs(f(-1) - 4) < 1e-8
    assert abs(f(0)  - 1) < 1e-8
    assert abs(f(1)  - 0) < 1e-8

# test_Quadratic()  # uncomment this line when testing
```

### Exercise 1b) Pretty printing

Extend your `Quadratic` class with a string special method (`__str__`) so that you can print a Polynomial object  and get the polynomial written out on a readable form. Test by creating a polynomial object and printing it out.

#### Exercise 1c) Adding together polynomials

Adding together two general quadratic functions:

$$f(x) = a_2x^2 + a_1 x + a_0, \qquad g(x) = b_2x^2 + b_1 x + b_0,$$
gives a new quadratic function:

$$(f + g)(x) = (a_2 + b_2)x^2 + (a_1 + b_1)x + (a_0 + b_0)$$

Implement this functionality using the addition special method (`__add__`). This method should return a new Quadratic-object, without changing the two that are added together. Your new class should be able to handle the following test script:
***
```Python
f = Quadratic(1, -2, 1)
g = Quadratic(-1, 6, -3)

h = f + g
print(h)

x = np.linspace(-5, 5, 101)
plt.plot(x, h(x))
plt.show()
```
***
(Because $a_2 + b_2 = 0$, the resulting plot should be a straight line.)

Implement your solution here:

```python

```

Use this to test your implementation:

```python
def test_Quadratic_add():
    f = Quadratic(1, -2, 1)
    g = Quadratic(-1, 6, -3)
    h = f + g
    a2, a1, a0 = h.coeffs
    assert a2 == 0
    assert a1 == 4
    assert a0 == -2

# test_Quadratic_add()  # uncomment this line when testing
```

### Exercise 1d) Finding the roots

The roots of a general quadratic function,
$$f(x) = ax^2+bx+c = 0,$$
are given by the quadratic formula
$${\displaystyle x={\frac {-b\pm {\sqrt {b^{2}-4ac}}}{2a}}.}$$

Extend your `Quadratic` function with a method `.roots()` that finds and returns the real roots of the function (ignore the imaginary ones). Return the result as a tuple with 0, 1, or 2 elements.

Test your method on the three polynomials:
* $2x^2 -2x + 2$
* $x^2 - 2x + 1$
* $x^2 -3x + 2$

Implement your solution here:

```python

```

Use this to test your implementation:

```python
def test_Quadratic_root():
    f1 = Quadratic(2, -2, 2)
    f2 = Quadratic(1, -2, 1)
    f3 = Quadratic(1, -3, 2)

    assert f1.roots() == ()
    assert abs(f2.roots()[0] - 1) < 1e-8
    assert abs(f3.roots()[0] - 1) < 1e-8 and abs(f3.roots()[1] - 2) < 1e-8

# test_Quadratic_root()  # uncomment this line when testing
```

### Exercise 1e) Finding the intersection of two quadratic functions

Extend your class with a method that finds and returns the intersection points (if any) between two `Quadratic`-objects. It should work as follows:

***
```Python
f = Quadratic(1, -2, 1)
g = Quadratic(2, 3, -2)

print(f.intersect(g))
```
***
**Hint:** The intersections are all points solving $f(x) = g(x)$, which can be written as $(f-g)(x) = 0$.

Test your solution by plotting the two functions and their intersections:

$$f(x) = x^2 -2x + 1, \qquad g(x) = 2x^2 + 3x - 2.$$

## Exercise 2 — A class for general polynomials

We now turn to looking at general polynomials of degree $n$. These can be written as

$$f(x) = a_{n}x^{n}+a_{n-1}x^{n-1}+\dotsb +a_{2}x^{2}+a_{1}x+a_{0},$$
or more compactly as
$${\displaystyle \sum _{k=0}^{n}a_{k}x^{k}}.$$

We want to make a class that represents such a polynomial, and can take any number of coefficients in. The constructor of such a class could for example take in a list of the coefficients: `[a0, a1, ..., aN]`. However, this list will always have to be of length $N$, and say we want to specify the polynomial $x^{1000} + 1$, it is highly inefficient to pass in such a long list, as most coefficients are actually 0.

A better approach is to use a dictionary, where we use the index as the key and the coefficient as the value. Doing this, we can then specify only the non-zero coefficients, and simply skip those that are 0. So defining $x^{1000} + 1$ would simply be: `Polynomial({0: 1, 1000: 1})`.

### Exercise 2a) Defining the Polynomial class

Define the `Polynomial` class with the following methods
* A constructor (`__init__`) that takes in the coefficients of the polynomial as a dictionary
* A call method (`__call__`) that computes f(x) for a given x
* A string method (`__str__`) for informative printing of the polynomial

Your class should be able to handle the following test script

***
```Python
coeffs = {0: 1, 5:-1, 10:1}
f = Polynomial(coeffs)

print(f)

x = linspace(-1, 1, 101)
plt.plot(x, f(x))
plt.show()
```
****

Implement your solution here:

```python

```

### Exercise 2b): Adding general polynomials together

We now want to be able to add together two general polynomial objects, which should produce a new general polynomial object. Mathematically, this is just an extension of the 2nd degree polynomial case which we saw in exercise (1). If we have

$$f(x) = {\displaystyle \sum _{k=0}^{m}a_{k}x^{k}}, \qquad g(x)={\displaystyle \sum _{k=0}^{n}b_{k}x^{k}},$$
the sum will be defined by
$$(f + g)(x) = {\displaystyle \sum _{k=0}^{\max(m, n)}(a_{k} + b_{k})x^{k}}.$$

Thus, if we add together two polynomials of degree $m$ and $n$, then the sum will have degree $\max(m, n)$, i.e., the largest of the two.

Extend your class to add this functionality using the addition special method (`__add__`).

The class should handle the following test case:
***
```Python

f = Polynomial({0:1, 5:-7, 10:1})
g = Polynomial({5:7, 10:1, 15:-3})

print(f+g)
```
***
Which should produce the output: $-3x^{15} + 2x^{10} + 1$

Implement your solution here:

```python

```

**Hint:** You will need to create a new coefficient dictionary for the new polynomial and add in the coefficients from the two polynomials. This can be slightly tricky getting the keys right. Here `collections.defaultdict` can be useful, but it isn't necessary.

### Exercise 2c) Defining a `AddableDictionary` class

The previous exercise would have been a lot simpler, if we could simply add two dictionary objects together as follows:
```Python
a = {0: 2, 1: 3, 2: 4}
b = {0: -1, 1:3, 2: 3, 3: 2}
c = a + b
```
However, if you try to do this, you get an exception:
> `TypeError: unsupported operand type(s) for +: 'dict' and 'dict'`

This means that there is no addition special method defined for dictionaries. However, we can extend the normal dictionary class to include this by adding a special method as follows
```Python
class AddableDict(dict):
    def __add__(self, other):
        ...
```

Add the necessary code, so that our new `AddableDict` class can add two dictionaries together as follows:
```Python
a = AddableDict({0: 2, 1: 3, 2: 4})
b = AddableDict({0: -1, 1:3, 2: 3, 3: 2})
print(a + b)
```
And give the ouput: `{0: 1, 1: 6, 2: 7, 3: 2}`.

Implement your solution here:

```python

```

Use this to test your implementation:

```python
def test_AddableDict():
    a = AddableDict({0: 2, 1: 3, 2: 4})
    b = AddableDict({0: -1, 1:3, 2: 3, 3: 2})
    c = a + b
    assert c[0] == 1
    assert c[1] == 6
    assert c[2] == 7
    assert c[3] == 2

# test_AddableDict()  # uncomment this line when testing
```

Having made the `AddableDict class`, go back and change the Polynomial constructor, so that even if the user sends in the coefficients as a normal dictionary, it is converted to an `AddableDict` inside the Polynomial. Having done this, rewrite `Polynomial.__add__`, which should be trivial.

### Exercise 2d) Derivative of a polynomial

It is also the case that the derivative of a polynomial is a polynomial, if we have

$$f(x) = {\displaystyle \sum _{k=0}^{m}a_{k}x^{k}},$$
then we get
$$f'(x) = {\displaystyle \sum _{k=1}^{m} (a_{k}\cdot k)x^{k-1}},$$
which can be written as
$$f'(x) = {\displaystyle \sum _{k=0}^{m-1} b_k x^{k}},$$
where $b_{k} = (k+1)a_{k+1}$.

Implement a method, `derivative`, that returns the function $f'(x)$ as a new Polynomial object. Test your function by finding the derivative of

$$f(x) = x^{10} - 3x^6 + 2x^2 + 1.$$

Implement your solution here:

```python

```

Use this to test your implementation:

```python
def test_derivative():
    f = Polynomial({10:1, 6:-3, 2:2, 0:1})
    f_deriv = f.derivative()
    assert f_deriv.coeffs == {9:10, 5:-18, 1:4}

# test_derivative()  # uncomment this line when testing
```

### Exercise 2e) Multiplying polynomials

It is also the case that the *product* of two polynomials form a new polynomial. If we again define

$$f(x) = {\displaystyle \sum _{k=0}^{m}a_{k}x^{k}}, \qquad g(x)={\displaystyle \sum _{k=0}^{n}b_{k}x^{k}},$$
then the product is given by
$$(f \cdot g)(x) = \left(\sum _{i = 0}^m a_{i}x^{i}\right) \cdot \left(\sum _{j=0}^{n} b_{j}x^{j}\right) = \sum_{i=0}^m\sum_{j=0}^n a_i b_j x^{i + j}$$

Implement this functionality using the multiplication special method (`__mul__`). To acomplish this, you will need two nested for-loops over the coefficient dictionaries.

Test your implementation with the code block
***
```Python
f = Polynomial({2: 4, 1: 1})
g = Polynomial({3: 3, 0: 1})
print(f*g)
```
***
Which should give the output:
$$(4x^2 + x)(3x^3 + 1) = 12x^5 + 3x^4 + 4x^2 + x$$

Implement your solution here:

```python

```

Use this to test your implementation:

```python
def test_Polynomial_mul():
    f = Polynomial({2: 4, 1: 1})
    g = Polynomial({3: 3, 0: 1})
    h = f*g
    assert h.coeffs == {5:12, 4:3, 2:4, 1:1}

# test_Polynomial_mul()   # uncomment this line when testing
```

## Exercise 3 - Quantum Harmonic Oscillator in One Dimension

The quantum harmonic oscillator wave function, $\psi_n (x)$ is a solution to the time-independent Scrödinger equation
$$\hat{H} \psi_n (x)  = E_n \psi_n (x)$$
where $\hat{H}$ is the quantum harmonic oscillator hamiltonian and $E_n$ is the energy at the $n$-th level ($n$ must be a positive integer, $n = 0, 1, 2,$...).

In this excersise you will implement a `class HOWF` which represents a quantum harmonic oscillator wave function in one dimension.

### Exercise 3a) Hermite polynomials

To implement the harmonic oscillator wave function, a Hermite Polynomial is needed. The Hermite Polynomials are given by the recursive formula

\begin{align*}
    H_n(x) &= 2 x H_{n - 1}(x) - 2(n - 1) H_{n - 2}(x) ,
\end{align*}
for $n = 2, 3, 4, ...$, and inital conditions are
\begin{align}
    H_0(x) &= 1 \\
    H_1(x) &= 2 x .
\end{align}

Make a `class HOWF`. Start by implementing the constructor and the recursive private method `_compute_Hermite(self, n)`.

The method `_compute_Hermite(self, n)` should return the hermite polynomial as an object of the class `Polynomial`, which you implemented in the previous exercise. The method should be implemented using recursion.
*Hint: You will easily get error messages if you try to multiply numbers with your polynomial class. Remember that you can consider a constant as a polynomial of the zero-th degree!

The constructor must take the level $n$ as a paramterer. The constructor should make a call the method `_compute_Hermite(self, n)` to obtain and store the hermite polynomial of the $n$-th order. Name the variable of the hermite polynomial `H`.

Implement your solution here:

```python

```

```python
def test_Hermite():
    H0 = lambda x: 1
    H1 = lambda x: 2*x
    H2 = lambda x: 4*x*x - 2
    H3 = lambda x: 8*x**3 - 12*x
    H4 = lambda x: 16*x**4 - 48*x**2 + 12
    H5 = lambda x: 32*x**5 - 160*x**3 + 120*x
    H_table = [H0, H1, H2, H3, H4, H5]

    tol = 1e-12
    for n in range(0, 6):
        H = HOWF(n).H
        for x in [0, 0.5, 1.0/3, 1, 3/2, 2]:
            expected = H_table[n](x)
            computed = H(x)
            msg = "The implemented Hermite Polynomial yields unexpected result for n = %d\
            \n\tH(%.2f) = %.13g != %.13g" %(n, x, expected, computed)
            assert abs(expected - computed) < tol, msg

# test_Hermite()  # uncomment this line when testing
```

### Exercise 3b) The quantum harmonic wave function

In this exercise you will be asked to implement two functions. The expressions would include the mass of the particle $m$, plancks constant $\hbar$ and the angular frequency $\omega$. When computing, it is an advantage to simplify the expressions as much as possible. Therefore we simplifiy the calculations by using dimensionless variables, defined by
$\chi = \sqrt{\frac{m \omega}{\hbar}} \cdot x$. This will make all the calculation dimensionless. Consequently, the dimensionless energy (in this case scale) is then given by $\epsilon_n = \frac{2 E_n}{\hbar \omega}$.

The quantum harmonic wave function of the $n$-th energy level is then given by

\begin{align*}
    \psi_n (\chi) = \pi^{-\frac{1}{4}} \frac{1}{\sqrt{2^n n!}} H_n(\chi)\exp\left(\frac{-\chi^2}{2}\right)
\end{align*}

where $H_n$ is the Hermite polynimial of the $n$-th order. Let `HOWF` a callable class by implementing the formula above in the special method  `__call__(self, chi)`.

The energy of the $n$-th level is then given by
\begin{align*}
    \epsilon_n = 2n + 1 .
\end{align*}

Implement a method in `HOWF` which calculates the energy of the wave function. Define this method as a property.

Implement your solution here:

```python
from numpy import exp, pi
from math import factorial, sqrt
```

### Exercise 3c) Visualising the wave functions

You will now reproduce a quite iconic figure of the quantum harmonic oscillator wave function. Plot the wave functions in relative height to their energy, as in plot $\psi_n(\chi) + \epsilon_n$ for $n = 0, 1, 2, ..., N$ in the same plot.

Set $N$ to be at least 15. A sanity check for your implementation of the wave functions is that the number of nodes of $\psi_n$ should be $n$. (That means that $\psi_n$ should intersect $\epsilon_n$ $n$ times on the y-axis, since the wave function has been lifted by $\epsilon_n$ on the $y$-axis). An appropriate range for $\chi$ could be $\chi \in [-8, 8]$.

Implement your solution here:

```python

```

## Exercise 4 — Fibonnaci with Memoization

The Fibonnaci sequence is given by the numbers
$$1, 1, 2, 3, 5, 8, 13, 21, 34, \ldots$$
it is defined by the recursive formula
$$F_{i} = F_{i-1} + F_{i-2}.$$

For this defintion to make any sense, the sequence has to start somewhere, so we define that

$$F_0 = 0, \qquad F_1 = 1.$$

### Exercise 4a) A Fibonacci function

Write a *recursive* function `fibonacci(n)`, that returns $F_n$. A *recursive* function is a function that calls itself. To do this, you won't need any explicit loops. You will however, need to include the *base-case* of $F_0=0$ $F_1=1$, otherwise the recursion would just continue forever.

Verify your function by writing out $F_1, \ldots, F_{10}$ and comparing to the sequence above.

Implement your solution here:

```python

```

Use this to test your implementation:

```python
def test_fibonacci():
    computed = []
    for i in range(11):
        computed.append(fibonacci(i))
    assert computed == [0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55]

# test_fibonacci()  # uncomment this line when testing
```

### Exercise 4b) Improving our algorithm

We have now implemented a function that finds the $n$'th Fibonnaci number, which is great. However, it is very inefficient. Imagine for example we want to compute $F_{100}$. The first call to the function, creates two new calls: `F(99)` and `F(98)`. Both of these call the function twice again, so we have four function calls there. Each of those four lead to two new ones and so on all the way down to the base case of $n=1$. This means we make something on the order of $2^{100}$ function calls! This means the complexity of calulating $F(n)$ grows exponentially as $n$ grows, which is horrible. Try drawing a tree to represent the function calls for $n=100$ and you quickly see the absurdity of the situation.

To improve our Fibonnaci algorithm, we will use a dynamic programming technique called *memoization*. While the name is a bit weird, the idea is fairly simple, we simply make our program remember the answers it has already computed, so that we won't have to compute them again later.

Classes are perfect for making memoized functions, as we can use an internal dictionary to remember old solutions.

Fill in the skeleton code below so that the class computes Fibonacci numbers:

```python
class Fibonacci:
    def __init__(self):
        self.memory = {0: 1, 1: 1}

    def __call__(self, n):
        """
        if n is in memory
            - return it
        if n is not in memory
            - calcuate it recursively
            - put it into memory
            - return it
        """
        pass
```

If you program your class correctly, you should see a huge speed up. This is because we now avoid repeating the same calculations uneccesarily. For example, if you try computing $F(100)$ as follows:

```python
fib = Fibonacci()
print(fib(100))
```

The memoized function will only need to compute the numbers $F(2), \ldots F(100)$ once, meaning the number of function calls needed is now *linear* instead of exponential!

Implement your solution here:

```python

```

Same test to make sure the code still works:

```python
# test_fibonacci()  # uncomment this line when testing
```

### Exercise 4c) Maximum Recursion Depth

Try finding an even larger number, like $F_{100000}$. Does it work? It will probably fail due to a maximum recursion depth. Try to understand what this means and why it happens. Can you think of a work-around for this problem?

Hint: You need to build up the memoization dictionary from the ground up, and then it will be able to handle larger inputs. This can be done with a for-loop for example.

Implement your solution here:

```python

```

Same test to make sure the code still works:

```python
def test_fibonacci():
    computed = []
    for i in range(11):
        computed.append(fibonacci(i))
    assert computed == [0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55]

# test_fibonacci()  # uncomment this line when testing
```

### Exercise 4d) A memoized factorial class

The factorial is also defined recursively
$$N! = N \cdot (N-1)!,$$
with a base case of $F(0) = 1$.

Repeat the process of the Fibonacci sequence and create a memoized class, `Factorial`, that computes $N!$.

Implement your solution here:

```python

```

Use this to test your implementation:

```python
def test_Factorial():
    f = Factorial()
    computed = []
    for i in range(6):
        computed.append(f(i))

    assert computed == [1, 1, 2, 6, 24, 120]

# test_Factorial()  # uncomment this line when testing
```


```python
import sympy as sym
from sympy.polys import subresultants_qq_zz

sym.init_printing()
```

The Bezout matrix is a special square matrix associated with two polynomials, introduced by Sylvester (1853) and Cayley (1857) and named after Étienne Bézout. Bézoutian may also refer to the determinant of this matrix, which is equal to the resultant of the two polynomials.

The entries of Bezout matrix are bilinear functions of coefficients of the given polynomials. The Bezout formulation has gone over different generalizations. The most common one is the Cayley.. Cayley's matrix is given by,

$$ \left|
\begin{array}{cc}
p(x) & q(x)\\
p(a)& q(a)
\end{array}
\right| = \Delta(x, a)$$

where $\Delta(x, a)$ is the determinant.

We have the polynomial:

$$ \delta(x, a) = \frac{\Delta(x,a)}{x-a}$$

The matrix is then constructed from the coefficients of polynomial $\alpha$. Each coefficient is viewed as a polynomial of $x_1,..., x_n$.

The Bezout matrix is highly related to the Sylvester matrix and the greatest common divisor of polynomials. Unlike in Sylvester's formulation, where the resultant of $p$ and $q$ is the determinant of an $(m + n) \times (m + n)$ matrix, in the Cayley formulation, the resultant is obtained
as the determinant of a $n \times n$ matrix.

Example: Generic example
------------------------

```python
b_3, b_2, b_1, b_0 = sym.symbols("b_3, b_2, b_1, b_0")
x = sym.symbols('x')
```

```python
b = sym.IndexedBase("b")
```

```python
p = b_2 * x ** 2 + b_1 * x + b_0
q = sym.diff(p, x)
```

```python
subresultants_qq_zz.bezout(p, q, x)
```

Example: Existence of common roots
------------------------------------------

Note that if the system has a common root we are expecting the resultant/determinant to equal to zero.

**A commot root exists.**

```python
# example one
p = x ** 3 +1
q = x + 1
```

```python
subresultants_qq_zz.bezout(p, q, x)
```

```python
subresultants_qq_zz.bezout(p, q, x).det()
```

```python
# example two
p = x ** 2 - 5 * x + 6
q = x ** 2 - 3 * x + 2
```

```python
subresultants_qq_zz.bezout(p, q, x)
```

```python
subresultants_qq_zz.bezout(p, q, x).det()
```

**A common root does not exist.**

```python
z = x ** 2 - 7 * x + 12
h = x ** 2 - x
```

```python
subresultants_qq_zz.bezout(z, h, x).det()
```

Dixon's Resultant
-----------------

Dixon (1908) showed how to extend this formulation to $m = 3$ polynomials in $n = 2$ variables.

In a similar manner but this time,

$$ \left|
\begin{array}{cc}
p(x, y) & q(x, y) & h(x, y) \cr
p(a, y) & q(a, y) & h(b, y) \cr
p(a, b) & q(a, b) & h(a, b) \cr
\end{array}
\right| = \Delta(x, y, \alpha, \beta)$$

where $\Delta(x, y, \alpha, \beta)$ is the determinant.

Thus, we have the polynomial:

$$ \delta(x,y, \alpha, \beta) = \frac{\Delta(x, y, \alpha, \beta)}{(x-\alpha)(y - \beta)}$$

```python
from sympy.polys.multivariate_resultants import DixonResultant
```

Example: Generic example of Dixon $(n=2, m=3)$
---------------------------------------------------

```python
a_1, a_2, b_1, b_2, u_1, u_2, u_3 = sym.symbols('a_1, a_2, b_1, b_2, u_1, u_2, u_3')
```

```python
y = sym.symbols('y')
```

```python
p = a_1 * x ** 2 * y ** 2 + a_2 * x ** 2
q = b_1 * x ** 2 * y ** 2 + b_2 * y ** 2
h = u_1 * x + u_2 * y + u_3
```

```python
dixon = DixonResultant(variables=[x, y], polynomials=[p, q, h])
```

```python
poly = dixon.get_dixon_polynomial()
```

```python
poly
```

```python
matrix = dixon.get_dixon_matrix(poly)
```

```python
matrix
```

```python
matrix.det().factor()
```

Dixon's General Case
--------------------

[Yang et al.](https://rd.springer.com/chapter/10.1007/3-540-63104-6_11) generalized the Dixon resultant method of three polynomials with two variables to the system of $n+1$ polynomials with $n$ variables.

Example: Numerical example
--------------------

```python
p = x + y
q = x ** 2 + y ** 3
h = x ** 2 + y
```

```python
dixon = DixonResultant([p, q, h], (x, y))
```

```python
poly = dixon.get_dixon_polynomial()
poly.simplify()
```

```python
matrix = dixon.get_dixon_matrix(polynomial=poly)
matrix
```

```python
matrix.det()
```

Example: Generic example
---------

```python
a, b, c = sym.symbols('a, b, c')
```

```python
p_1 = a * x ** 2 + b * x * y + (b + c - a) * x + a * y + 3 * (c - 1)
p_2 = 2 * a ** 2 * x ** 2 + 2 * a * b * x * y + a * b * y + b ** 3
p_3 = 4 * (a - b) * x + c * (a + b) * y + 4 * a * b
```

```python
polynomials = [p_1, p_2, p_3]
```

```python
dixon = DixonResultant(polynomials, [x, y])
```

```python
poly = dixon.get_dixon_polynomial()
```

```python
size = len(poly.monoms())
size
```

```python
matrix = dixon.get_dixon_matrix(poly)
matrix
```

Example:
--------------------------------------------------------------------------------------------------
**From [Dixon resultant’s solution of systems of geodetic polynomial equations](https://rd.springer.com/content/pdf/10.1007%2Fs00190-007-0199-0.pdf)**

```python
z = sym.symbols('z')
```

```python
f = x ** 2 + y ** 2 - 1 + z * 0
g = x ** 2 + z ** 2 - 1 + y * 0
h = y ** 2 + z ** 2 - 1
```

```python
dixon = DixonResultant([f, g, h], [y, z])
```

```python
poly = dixon.get_dixon_polynomial()
```

```python
matrix = dixon.get_dixon_matrix(poly)
matrix
```

```python
matrix.det()
```


# Marginal Probability - Districts & Ages

> This document is written in *R*.
>
> ***GitHub***: https://github.com/czs108

## Background

> The following table gives the *percentage of pupils* in a certain school from different *districts* and at different *ages*.
>
> | *District \ Age* |  14  |  15  |  16  |  17  |
> | :--------------: | :--: | :--: | :--: | :--: |
> |    **North**     |  5%  | 10%  |  5%  | 10%  |
> |    **Center**    | 10%  | 15%  |  5%  |  5%  |
> |    **South**     | 15%  | 10%  |  5%  |  5%  |

## Question A

> Give values for the following probabilities.

### Question 1

\begin{align}
P(age = 15)
    &= \sum_{i = North,Center,South} P(age = 15\, \cap\, district = i) \\
    &= 10\% + 15\% + 10\% \\
    &= 35\%
\end{split}
\end{equation}

### Question 2

\begin{equation}
\begin{split}
P(district = Center)
    &= \sum_{i = 14}^{17} P(age = i\, \cap\, district = Center) \\
    &= 10\% + 15\% + 5\% + 5\% \\
    &= 35\%
\end{split}
\end{equation}

### Question 3

\begin{equation}
\begin{split}
P(age = 15\, \mid\, district = Center)
    &= \frac{P(age = 15\, \cap\, district = Center)}{P(district = Center)} \\
    &= \frac{15\%}{35\%}
\end{split}
\end{equation}

### Question 4

\begin{equation}
\begin{split}
P(age = 15\, \cap\, district = Center)
    &= P(district = Center) \cdot P(age = 15\, \mid\, district = Center) \\
    &= 35\% \times \frac{15\%}{35\%} \\
    &= 15\%
\end{split}
\end{equation}

### Question 5

\begin{equation}
\begin{split}
P(age = 15\, \cup\, district = Center)
    &= P(age = 15) + P(district = Center) - P(age = 15\, \cap\, district = Center) \\
    &= 35\% + 35\% - 15\% \\
    &= 55\%
\end{split}
\end{equation}

## Question B

> What are the *mean* and *variance* of *age*?

\begin{equation}
Mean = \sum_{i = 14}^{17} i \cdot P(age = i)
\end{equation}

\begin{equation}
Variance = \sum_{i = 14}^{17} (i - \overline{i})^2 \cdot P(age = i)
\end{equation}

```R
ages <- c(14:17)
probs <- c(0.30, 0.35, 0.15, 0.20)
mean <- sum(ages * probs)
var <- sum(((ages - mean) ^ 2) * probs)

print(sprintf("Mean = %.1f", mean), quote=FALSE)
print(sprintf("Variance = %.1f", var), quote=FALSE)
```

    [1] Mean = 15.2
    [1] Variance = 1.2

## Question C

> Use the `barplot` to visualize the table.

Store the data in a matrix.

```R
prcnt <- c(5, 10, 5, 10, 10, 15, 5, 5, 15, 10, 5, 5)
prcnt <- matrix(prcnt, nrow=3, byrow=TRUE)

prcnt
```

 5	10	5 	10

10	15	5 	 5

15	10	5 	 5

Add row names and column names.

```R
colnames(prcnt) <- c(14:17)
rownames(prcnt) <- c("North", "Center", "South")

prcnt
```

	14	15	16	17

North	 5	10	5 	10

Center	10	15	5 	 5

South	15	10	5 	 5

```R
barplot(prcnt, xlab="Age", ylab="Percentage", legend.text=TRUE)
```

exercises/Marginal Probability - Districts & Ages.ipynb