# Programming Assignment

## Naive Bayes and logistic regression

### Instructions

In this notebook, you will write code to develop a Naive Bayes classifier model to the Iris dataset using Distribution objects from TensorFlow Probability. You will also explore the connection between the Naive Bayes classifier and logistic regression.

Some code cells are provided you in the notebook. You should avoid editing provided code, and make sure to execute the cells in order to avoid unexpected errors. Some cells begin with the line:

`#### GRADED CELL ####`

Don't move or edit this first line - this is what the automatic grader looks for to recognise graded cells. These cells require you to write your own code to complete them, and are automatically graded when you submit the notebook. Don't edit the function name or signature provided in these cells, otherwise the automatic grader might not function properly.

### How to submit

Complete all the tasks you are asked for in the worksheet. When you have finished and are happy with your code, press the **Submit Assignment** button at the top of this notebook.

### Let's get started!

We'll start running some imports, and loading the dataset. Do not edit the existing imports in the following cell. If you would like to make further Tensorflow imports, you should add them here.

```python
from google.colab import drive
drive.mount('/content/drive',force_remount=True)
```

    Mounted at /content/drive

```python
import os
os.chdir('/content/drive/MyDrive/Colab_Notebooks/coursera/coursera_Probabilistic_Deep_Learning_TF2')
```

```python
!pip install tensorflow=='2.1.0'
```

    Collecting tensorflow==2.1.0
      Downloading tensorflow-2.1.0-cp37-cp37m-manylinux2010_x86_64.whl (421.8 MB)
         |████████████████████████████████| 421.8 MB 23 kB/s
    Requirement already satisfied: six>=1.12.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.1.0) (1.15.0)
    Requirement already satisfied: protobuf>=3.8.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.1.0) (3.17.3)
    Collecting gast==0.2.2
      Downloading gast-0.2.2.tar.gz (10 kB)
    Collecting tensorboard<2.2.0,>=2.1.0
      Downloading tensorboard-2.1.1-py3-none-any.whl (3.8 MB)
         |████████████████████████████████| 3.8 MB 35.8 MB/s
    Requirement already satisfied: keras-preprocessing>=1.1.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.1.0) (1.1.2)
    Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.1.0) (3.3.0)
    Requirement already satisfied: scipy==1.4.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.1.0) (1.4.1)
    Requirement already satisfied: google-pasta>=0.1.6 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.1.0) (0.2.0)
    Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.1.0) (0.37.1)
    Requirement already satisfied: numpy<2.0,>=1.16.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.1.0) (1.19.5)
    Requirement already satisfied: absl-py>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.1.0) (1.0.0)
    Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.1.0) (1.1.0)
    Collecting tensorflow-estimator<2.2.0,>=2.1.0rc0
      Downloading tensorflow_estimator-2.1.0-py2.py3-none-any.whl (448 kB)
         |████████████████████████████████| 448 kB 36.7 MB/s
    Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.1.0) (1.43.0)
    Collecting keras-applications>=1.0.8
      Downloading Keras_Applications-1.0.8-py3-none-any.whl (50 kB)
         |████████████████████████████████| 50 kB 4.5 MB/s
    Requirement already satisfied: astor>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.1.0) (0.8.1)
    Requirement already satisfied: wrapt>=1.11.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.1.0) (1.13.3)
    Requirement already satisfied: h5py in /usr/local/lib/python3.7/dist-packages (from keras-applications>=1.0.8->tensorflow==2.1.0) (3.1.0)
    Requirement already satisfied: requests<3,>=2.21.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (2.23.0)
    Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (57.4.0)
    Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (1.0.1)
    Requirement already satisfied: google-auth<2,>=1.6.3 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (1.35.0)
    Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (3.3.6)
    Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (0.4.6)
    Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.7/dist-packages (from google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (4.8)
    Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.7/dist-packages (from google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (0.2.8)
    Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (4.2.4)
    Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (1.3.1)
    Requirement already satisfied: importlib-metadata>=4.4 in /usr/local/lib/python3.7/dist-packages (from markdown>=2.6.8->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (4.10.1)
    Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (3.7.0)
    Requirement already satisfied: typing-extensions>=3.6.4 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (3.10.0.2)
    Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.7/dist-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (0.4.8)
    Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (1.24.3)
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (3.0.4)
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (2.10)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (2021.10.8)
    Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (3.2.0)
    Requirement already satisfied: cached-property in /usr/local/lib/python3.7/dist-packages (from h5py->keras-applications>=1.0.8->tensorflow==2.1.0) (1.5.2)
    Building wheels for collected packages: gast
      Building wheel for gast (setup.py) ... done
      Created wheel for gast: filename=gast-0.2.2-py3-none-any.whl size=7554 sha256=6b1fb0741501a258c25b65ecb1233b810c89700b29dcd20a347293592c79f73c
      Stored in directory: /root/.cache/pip/wheels/21/7f/02/420f32a803f7d0967b48dd823da3f558c5166991bfd204eef3
    Successfully built gast
    Installing collected packages: tensorflow-estimator, tensorboard, keras-applications, gast, tensorflow
      Attempting uninstall: tensorflow-estimator
        Found existing installation: tensorflow-estimator 2.7.0
        Uninstalling tensorflow-estimator-2.7.0:
          Successfully uninstalled tensorflow-estimator-2.7.0
      Attempting uninstall: tensorboard
        Found existing installation: tensorboard 2.7.0
        Uninstalling tensorboard-2.7.0:
          Successfully uninstalled tensorboard-2.7.0
      Attempting uninstall: gast
        Found existing installation: gast 0.4.0
        Uninstalling gast-0.4.0:
          Successfully uninstalled gast-0.4.0
      Attempting uninstall: tensorflow
        Found existing installation: tensorflow 2.7.0
        Uninstalling tensorflow-2.7.0:
          Successfully uninstalled tensorflow-2.7.0
    ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
    tensorflow-probability 0.15.0 requires gast>=0.3.2, but you have gast 0.2.2 which is incompatible.
    Successfully installed gast-0.2.2 keras-applications-1.0.8 tensorboard-2.1.1 tensorflow-2.1.0 tensorflow-estimator-2.1.0

```python
!pip install tensorflow_probability=='0.9.0'
```

    Collecting tensorflow_probability==0.9.0
      Downloading tensorflow_probability-0.9.0-py2.py3-none-any.whl (3.2 MB)
         |████████████████████████████████| 3.2 MB 4.3 MB/s
    Requirement already satisfied: numpy>=1.13.3 in /usr/local/lib/python3.7/dist-packages (from tensorflow_probability==0.9.0) (1.19.5)
    Requirement already satisfied: cloudpickle>=1.2.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow_probability==0.9.0) (1.3.0)
    Requirement already satisfied: gast>=0.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow_probability==0.9.0) (0.2.2)
    Requirement already satisfied: six>=1.10.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow_probability==0.9.0) (1.15.0)
    Requirement already satisfied: decorator in /usr/local/lib/python3.7/dist-packages (from tensorflow_probability==0.9.0) (4.4.2)
    Installing collected packages: tensorflow-probability
      Attempting uninstall: tensorflow-probability
        Found existing installation: tensorflow-probability 0.15.0
        Uninstalling tensorflow-probability-0.15.0:
          Successfully uninstalled tensorflow-probability-0.15.0
    Successfully installed tensorflow-probability-0.9.0

```python
#### PACKAGE IMPORTS ####

# Run this cell first to import all required packages. Do not make any imports elsewhere in the notebook
import tensorflow as tf
import tensorflow_probability as tfp
tfd = tfp.distributions
import numpy as np
import matplotlib.pyplot as plt
from sklearn.metrics import accuracy_score
from sklearn import datasets, model_selection
%matplotlib inline

# If you would like to make further imports from TensorFlow or TensorFlow Probability, add them here

```

#### The Iris dataset

In this assignment, you will use the [Iris dataset](https://scikit-learn.org/stable/auto_examples/datasets/plot_iris_dataset.html). It consists of 50 samples from each of three species of Iris (Iris setosa, Iris virginica and Iris versicolor). Four features were measured from each sample: the length and the width of the sepals and petals, in centimeters. For a reference, see the following papers:

- R. A. Fisher. "The use of multiple measurements in taxonomic problems". Annals of Eugenics. 7 (2): 179–188, 1936.

Your goal is to construct a Naive Bayes classifier model that predicts the correct class from the sepal length and sepal width features. Under certain assumptions about this classifier model, you will explore the relation to logistic regression.

#### Load and prepare the data

We will first read in the Iris dataset, and split the dataset into training and test sets.

```python
# Load the dataset

iris = datasets.load_iris()
```

```python
# Use only the first two features: sepal length and width

data = iris.data[:, :2]
targets = iris.target
```

```python
# Randomly shuffle the data and make train and test splits

x_train, x_test, y_train, y_test = model_selection.train_test_split(data, targets, test_size=0.2)
```

```python
# Plot the training data

labels = {0: 'Iris-Setosa', 1: 'Iris-Versicolour', 2: 'Iris-Virginica'}
label_colours = ['blue', 'orange', 'green']

def plot_data(x, y, labels, colours):
    for c in np.unique(y):
        inx = np.where(y == c)
        plt.scatter(x[inx, 0], x[inx, 1], label=labels[c], c=colours[c])
    plt.title("Training set")
    plt.xlabel("Sepal length (cm)")
    plt.ylabel("Sepal width (cm)")
    plt.legend()

plt.figure(figsize=(8, 5))
plot_data(x_train, y_train, labels, label_colours)
plt.show()
```

### Naive Bayes classifier

We will briefly review the Naive Bayes classifier model. The fundamental equation for this classifier is Bayes' rule:

$$
P(Y=y_k | X_1,\ldots,X_d) = \frac{P(X_1,\ldots,X_d | Y=y_k)P(Y=y_k)}{\sum_{k=1}^K P(X_1,\ldots,X_d | Y=y_k)P(Y=y_k)}
$$

In the above, $d$ is the number of features or dimensions in the inputs $X$ (in our case $d=2$), and $K$ is the number of classes (in our case $K=3$). The distribution $P(Y)$ is the class prior distribution, which is a discrete distribution over $K$ classes. The distribution $P(X | Y)$ is the class-conditional distribution over inputs.

The Naive Bayes classifier makes the assumption that the data features $X_i$ are conditionally independent give the class $Y$ (the 'naive' assumption). In this case, the class-conditional distribution decomposes as

$$
\begin{align}
P(X | Y=y_k) &= P(X_1,\ldots,X_d | Y=y_k)\\
&= \prod_{i=1}^d P(X_i | Y=y_k)
\end{align}
$$

This simplifying assumption means that we typically need to estimate far fewer parameters for each of the distributions $P(X_i | Y=y_k)$ instead of the full joint distribution $P(X | Y=y_k)$.

Once the class prior distribution and class-conditional densities are estimated, the Naive Bayes classifier model can then make a class prediction $\hat{Y}$ for a new data input $\tilde{X} := (\tilde{X}_1,\ldots,\tilde{X}_d)$ according to

$$
\begin{align}
\hat{Y} &= \text{argmax}_{y_k} P(Y=y_k | \tilde{X}_1,\ldots,\tilde{X}_d) \\
&= \text{argmax}_{y_k}\frac{P(\tilde{X}_1,\ldots,\tilde{X}_d | Y=y_k)P(Y=y_k)}{\sum_{k=1}^K P(\tilde{X}_1,\ldots,\tilde{X}_d | Y=y_k)P(Y=y_k)}\\
&= \text{argmax}_{y_k} P(\tilde{X}_1,\ldots,\tilde{X}_d | Y=y_k)P(Y=y_k)
\end{align}
$$

#### Define the class prior distribution

We will begin by defining the class prior distribution. To do this we will simply take the maximum likelihood estimate, given by

$$
P(Y=y_k) = \frac{\sum_{n=1}^N \delta(Y^{(n)}=y_k)}{N},
$$

where the superscript $(n)$ indicates the $n$-th dataset example, $\delta(Y^{(n)}=y_k) = 1$ if $Y^{(n)}=y_k$ and 0 otherwise, and $N$ is the total number of examples in the dataset. The above is simply the proportion of data examples belonging to class $k$.

You should now write a function that builds the prior distribution from the training data, and returns it as a `Categorical` Distribution object.

* The input to your function `y` will be a numpy array of shape `(num_samples,)`
* The entries in `y` will be integer labels $k=0, 1,\ldots, K-1$
* Your function should build and return the prior distribution as a `Categorical` distribution object
  * The probabilities for this distribution will be a length-$K$ vector, with entries corresponding to $P(Y = y_k)$ for $k=0,1,\ldots,K-1$
  * Your function should work for any value of $K\ge 1$
  * This Distribution will have an empty batch shape and empty event shape

```python

```

```python
#### GRADED CELL ####

# Complete the following function.
# Make sure to not change the function name or arguments.

def get_prior(y):
    """
    This function takes training labels as a numpy array y of shape (num_samples,) as an input.
    Your function should
    This function should build a Categorical Distribution object with empty batch shape
    and event shape, with the probability of each class given as above.
    Your function should return the Distribution object.
    """
    #labels = {0: 'Iris-Setosa', 1: 'Iris-Versicolour', 2: 'Iris-Virginica'}
    n_classes=len(np.unique(y))#len(labels)
    print(n_classes)
    counts=np.zeros(shape=n_classes)
    for c in range(n_classes):
      counts[c]=np.sum(np.where(y==c,1,0))
    priors=counts/np.sum(counts)
    print('The class priors are {}'.format(priors))

    priors=tfd.Categorical(probs=priors)
    return priors

```

```python
# Run your function to get the prior

prior = get_prior(y_train)
```

    3
    The class priors are [0.35       0.34166667 0.30833333]

```python
# Plot the prior distribution

labels = ['Iris-Setosa', 'Iris-Versicolour', 'Iris-Virginica']
plt.bar([0, 1, 2], prior.probs.numpy(), color=label_colours)
plt.xlabel("Class")
plt.ylabel("Prior probability")
plt.title("Class prior distribution")
plt.xticks([0, 1, 2], labels)
plt.show()
```

#### Define the class-conditional densities

We now turn to the definition of the class-conditional distributions $P(X_i | Y=y_k)$ for $i=0, 1$ and $k=0, 1, 2$. In our model, we will assume these distributions to be univariate Gaussian:

$$
\begin{align}
P(X_i | Y=y_k) &= N(X_i | \mu_{ik}, \sigma_{ik})\\
&= \frac{1}{\sqrt{2\pi\sigma_{ik}^2}} \exp\left\{-\frac{1}{2} \left(\frac{x - \mu_{ik}}{\sigma_{ik}}\right)^2\right\}
\end{align}
$$
with mean parameters $\mu_{ik}$ and standard deviation parameters $\sigma_{ik}$, twelve parameters in all. We will again estimate these parameters using maximum likelihood. In this case, the estimates are given by

$$
\begin{align}
\hat{\mu}_{ik} &= \frac{\sum_n X_i^{(n)} \delta(Y^{(n)}=y_k)}{\sum_n \delta(Y^{(n)}=y_k)} \\
\hat{\sigma}^2_{ik} &= \frac{\sum_n (X_i^{(n)} - \hat{\mu}_{ik})^2 \delta(Y^{(n)}=y_k)}{\sum_n \delta(Y^{(n)}=y_k)}
\end{align}
$$

Note that the above are just the means and variances of the sample data points for each class.

You should now write a function the computes the class-conditional Gaussian densities, using the maximum likelihood parameter estimates given above, and returns them in a single, batched `MultivariateNormalDiag` Distribution object.

* The inputs to the function are
  * a numpy array `x` of shape `(num_samples, num_features)` for the data inputs
  * a numpy array `y` of shape `(num_samples,)` for the target labels
* Your function should work for any number of classes $K\ge 1$ and any number of features $d\ge 1$

```python
np.mean(x_train)
```

    4.42

```python
#### GRADED CELL ####

# Complete the following function.
# Make sure to not change the function name or arguments.

def get_class_conditionals(x, y):
    """
    This function takes training data samples x and labels y as inputs.
    This function should build the class-conditional Gaussian distributions above.
    It should construct a batch of distributions for each feature and each class, using the
    parameter estimates above for the means and standard deviations.
    The batch shape of this distribution should be rank 2, where the first dimension corresponds
    to the number of classes and the second corresponds to the number of features.
    Your function should then return the Distribution object.
    """
    labels={label:label for label in y}
    labels=sorted(list(labels.keys()))
    class_in_x=[np.where(label==y) for label in labels]
    u=[]
    var=[]
    for class_i in class_in_x:
       u.append(list(np.mean(x[class_i],axis=0)))
    count=0
    for class_i in class_in_x:
      var.append(list(np.mean((x[class_i]-u[count])**2,axis=0)))
      count+=1
    std=np.sqrt(var)
    print(u,std)
    dist = tfd.MultivariateNormalDiag(loc=u,scale_diag=std)
    return dist

```

```python
# Run your function to get the class-conditional distributions

class_conditionals = get_class_conditionals(x_train, y_train)
```

    [[4.995238095238095, 3.4285714285714293], [5.834146341463413, 2.741463414634147], [6.6324324324324335, 2.972972972972973]] [[0.36836342 0.3953815 ]
     [0.45831598 0.32004238]
     [0.63843307 0.31421171]]

We can visualise the class-conditional densities with contour plots by running the cell below. Notice how the contours of each distribution correspond to a Gaussian distribution with diagonal covariance matrix, since the model assumes that each feature is independent given the class.

```python
# Plot the training data with the class-conditional density contours

def get_meshgrid(x0_range, x1_range, num_points=100):
    x0 = np.linspace(x0_range[0], x0_range[1], num_points)
    x1 = np.linspace(x1_range[0], x1_range[1], num_points)
    return np.meshgrid(x0, x1)

def contour_plot(x0_range, x1_range, prob_fn, batch_shape, colours, levels=None, num_points=100):
    X0, X1 = get_meshgrid(x0_range, x1_range, num_points=num_points)
    Z = prob_fn(np.expand_dims(np.array([X0.ravel(), X1.ravel()]).T, 1))
    Z = np.array(Z).T.reshape(batch_shape, *X0.shape)
    for batch in np.arange(batch_shape):
        if levels:
            plt.contourf(X0, X1, Z[batch], alpha=0.2, colors=colours, levels=levels)
        else:
            plt.contour(X0, X1, Z[batch], colors=colours[batch], alpha=0.3)

plt.figure(figsize=(10, 6))
plot_data(x_train, y_train, labels, label_colours)
x0_min, x0_max = x_train[:, 0].min(), x_train[:, 0].max()
x1_min, x1_max = x_train[:, 1].min(), x_train[:, 1].max()
contour_plot((x0_min, x0_max), (x1_min, x1_max), class_conditionals.prob, 3, label_colours)
plt.title("Training set with class-conditional density contours")
plt.show()
```

#### Make predictions from the model

Now the prior and class-conditional distributions are defined, you can use them to compute the model's class probability predictions for an unknown test input $\tilde{X} = (\tilde{X}_1,\ldots,\tilde{X}_d)$, according to

$$
P(Y=y_k | \tilde{X}_1,\ldots,\tilde{X}_d) = \frac{P(\tilde{X}_1,\ldots,\tilde{X}_d | Y=y_k)P(Y=y_k)}{\sum_{k=1}^K P(\tilde{X}_1,\ldots,\tilde{X}_d | Y=y_k)P(Y=y_k)}
$$

The class prediction can then be taken as the class with the maximum probability:

$$
\hat{Y} = \text{argmax}_{y_k} P(Y=y_k | \tilde{X}_1,\ldots,\tilde{X}_d)
$$

You should now write a function to return the model's class probabilities for a given batch of test inputs of shape `(batch_shape, 2)`, where the `batch_shape` has rank at least one.

* The inputs to the function are the `prior` and `class_conditionals` distributions, and the inputs `x`
* Your function should use these distributions to compute the probabilities for each class $k$ as above
  * As before, your function should work for any number of classes $K\ge 1$
* It should then compute the prediction by taking the class with the highest probability
* The predictions should be returned in a numpy array of shape `(batch_shape)`

```python
prior
```

    <tfp.distributions.Categorical 'Categorical' batch_shape=[] event_shape=[] dtype=int32>

```python
#### GRADED CELL ####

# Complete the following function.
# Make sure to not change the function name or arguments.

def predict_class(prior, class_conditionals, x):
    """
    This function takes the prior distribution, class-conditional distribution, and
    a batch of inputs in a numpy array of shape (batch_shape, 2).
    This function should compute the class probabilities for each input in the batch, using
    the prior and class-conditional distributions, according to the above equation.
    Note that the batch_shape of x could have rank higher than one!
    Your function should then return the class predictions by taking the class with the
    maximum probability in a numpy array of shape (batch_shape,).
    """
    result=[]
    print(x.shape)

    for i,sample in enumerate(x):
      cond_probs=class_conditionals.log_prob(sample)
      #print(cond_probs)
      #print(prior.probs)
      joint_likelihood=tf.add(np.log(prior.probs),cond_probs)
      norm_factor=tf.math.reduce_logsumexp(joint_likelihood,axis=-1,keepdims=True)
      log_prob=joint_likelihood - norm_factor
      result.append((np.argmax(log_prob)))
    return np.array(result)

```

```python
# Get the class predictions

predictions = predict_class(prior, class_conditionals, x_test)
```

    (30, 2)

```python
# Evaluate the model accuracy on the test set

accuracy = accuracy_score(y_test, predictions)
print("Test accuracy: {:.4f}".format(accuracy))
```

    Test accuracy: 0.6333

```python
# Plot the model's decision regions

plt.figure(figsize=(10, 6))
plot_data(x_train, y_train, labels, label_colours)
x0_min, x0_max = x_train[:, 0].min(), x_train[:, 0].max()
x1_min, x1_max = x_train[:, 1].min(), x_train[:, 1].max()
contour_plot((x0_min, x0_max), (x1_min, x1_max),
             lambda x: predict_class(prior, class_conditionals, x),
             1, label_colours, levels=[-0.5, 0.5, 1.5, 2.5],
             num_points=500)
plt.title("Training set with decision regions")
plt.show()
```

### Binary classifier

We will now draw a connection between the Naive Bayes classifier and logistic regression.

First, we will update our model to be a binary classifier. In particular, the model will output the probability that a given input data sample belongs to the 'Iris-Setosa' class: $P(Y=y_0 | \tilde{X}_1,\ldots,\tilde{X}_d)$. The remaining two classes will be pooled together with the label $y_1$.

```python
# Redefine the dataset to have binary labels

y_train_binary = np.array(y_train)
y_train_binary[np.where(y_train_binary == 2)] = 1

y_test_binary = np.array(y_test)
y_test_binary[np.where(y_test_binary == 2)] = 1
```

```python
# Plot the training data

labels_binary = {0: 'Iris-Setosa', 1: 'Iris-Versicolour / Iris-Virginica'}
label_colours_binary = ['blue', 'red']

plt.figure(figsize=(8, 5))
plot_data(x_train, y_train_binary, labels_binary, label_colours_binary)
plt.show()
```

We will also make an extra modelling assumption that for each class $k$, the class-conditional distribution $P(X_i | Y=y_k)$ for each feature $i=0, 1$, has standard deviation $\sigma_i$, which is the same for each class $k$.

This means there are now six parameters in total: four for the means $\mu_{ik}$ and two for the standard deviations $\sigma_i$ ($i, k=0, 1$).

We will again use maximum likelihood to estimate these parameters. The prior distribution will be as before, with the class prior probabilities given by

$$
P(Y=y_k) = \frac{\sum_{n=1}^N \delta(Y^{(n)}=y_k)}{N},
$$

We will use your previous function `get_prior` to redefine the prior distribution.

```python
# Redefine the prior

prior_binary = get_prior(y_train_binary)
```

    2
    The class priors are [0.35 0.65]

```python
# Plot the prior distribution

plt.bar([0, 1], prior_binary.probs.numpy(), color=label_colours_binary)
plt.xlabel("Class")
plt.ylabel("Prior probability")
plt.title("Class prior distribution")
plt.xticks([0, 1], labels_binary)
plt.show()
```

For the class-conditional densities, the maximum likelihood estimate for the means are again given by

$$
\hat{\mu}_{ik} = \frac{\sum_n X_i^{(n)} \delta(Y^{(n)}=y_k)}{\sum_n \delta(Y^{(n)}=y_k)} \\
$$

However, the estimate for the standard deviations $\sigma_i$ is updated. There is also a closed-form solution for the shared standard deviations, but we will instead learn these from the data.

You should now write a function that takes the training inputs and target labels as input, as well as an optimizer object, number of epochs and a TensorFlow Variable. This function should be written according to the following spec:

* The inputs to the function are:
  * a numpy array `x` of shape `(num_samples, num_features)` for the data inputs
  * a numpy array `y` of shape `(num_samples,)` for the target labels
  * a `tf.Variable` object `scales` of length 2 for the standard deviations $\sigma_i$
  * `optimiser`: an optimiser object
  * `epochs`: the number of epochs to run the training for
* The function should first compute the means $\mu_{ik}$ of the class-conditional Gaussians according to the above equation
* Then create a batched multivariate Gaussian distribution object using `MultivariateNormalDiag` with the means set to $\mu_{ik}$ and the scales set to `scales`
* Run a custom training loop for `epochs` number of epochs, in which:
  * the average per-example negative log likelihood for the whole dataset is computed as the loss
  * the gradient of the loss with respect to the `scales` variables is computed
  * the `scales` variables are updated by the `optimiser` object
* At each iteration, save the values of the `scales` variable and the loss
* The function should return a tuple of three objects:
   * a numpy array of shape `(epochs,)` of loss values
   * a numpy array of shape `(epochs, 2)` of values for the `scales` variable at each iteration
   * the final learned batched `MultivariateNormalDiag` distribution object

_NB: ideally, we would like to constrain the `scales` variable to have positive values. We are not doing that here, but in later weeks of the course you will learn how this can be implemented._

```python
#### GRADED CELL ####

# Complete the following function.
# Make sure to not change the function name or arguments.

def learn_stdevs(x, y, scales, optimiser, epochs):
    """
    This function takes the data inputs, targets, scales variable, optimiser and number of
    epochs as inputs.
    This function should set up and run a custom training loop according to the above
    specifications, by setting up the class conditional distributions as a MultivariateNormalDiag
    object, and updating the trainable variables (the scales) in a custom training loop.
    Your function should then return the a tuple of three elements: a numpy array of loss values
    during training, a numpy array of scales variables during training, and the final learned
    MultivariateNormalDiag distribution object.
    """
   #x=np.expand_dims(x,axis=1).astype(np.float32)
    #x=np.array(x).astype(np.float32)
    #y=tf.cast(y,tf.float32)
    #scales=tf.case(scales,tf.float32)

    #print(y)
    labels={label:label for label in y}
    labels=sorted(list(labels.keys()))
    #print(labels)
    class_in_x=[np.where(label==y) for label in labels]
    u=[]
    for class_i in class_in_x:
      #class_i=tf.cast(class_i,tf.int32)
      #print(class_i.shape)
      #print(x.shape)
      #print(x[class_i])
      u.append(list(np.mean(x[class_i],axis=0)))
      #print(u)
    u=np.array(u).astype(np.float32)
    print(u.shape)
    distribution = tfd.MultivariateNormalDiag(loc=u,scale_diag=scales)
    print(distribution)
    # Define the negative log likelihood
    def nll(x_train,distribution,y_train):
      y_train=tf.one_hot(y_train,len(tf.unique(y_train)))
      return -tf.reduce_mean(distribution.log_prob(x_train)*y_train)

    # Define a function to compute the loss and gradients
    @tf.function
    def get_loss_and_grads(x_train,distribution,y_train):
      with tf.GradientTape() as tape:
        tape.watch(distribution.trainable_variables)
        x_train=tf.cast(x_train,tf.float32)
        loss=nll(x_train,distribution,y_train)
        #print(loss)
        #loss=tf.cast(loss,tf.float32)
        grads=tape.gradient(loss,distribution.trainable_variables)
      return loss, grads
    print(u)
    nlls=[]
    scales_arr=[]
    x=np.expand_dims(x,axis=1).astype(np.float32)
    for i in range(epochs):
        loss,grads=get_loss_and_grads(x,distribution,y)
        #print(loss)
        optimiser.apply_gradients(zip(grads,distribution.trainable_variables))
        #rate_value=distribution.rate.value()
        nlls.append(loss)
        scales_arr.append(scales.numpy())

        print("Step {:03d}: Loss: {:.3f}".format(i,loss))

    return np.array(nlls),np.array(scales_arr),distribution
```

```python
# Define the inputs to your function

scales = tf.Variable([1., 1.])
opt = tf.keras.optimizers.Adam(learning_rate=0.01)
epochs = 500
```

```python
# Run your function to learn the class-conditional standard deviations

nlls, scales_arr, class_conditionals_binary = learn_stdevs(x_train, y_train_binary, scales, opt, epochs)
```

    (2, 2)
    tfp.distributions.MultivariateNormalDiag("MultivariateNormalDiag", batch_shape=[2], event_shape=[2], dtype=float32)
    [[4.9952383 3.4285715]
     [6.2128205 2.8512821]]
    Step 000: Loss: 1.038
    Step 001: Loss: 1.031
    Step 002: Loss: 1.023
    Step 003: Loss: 1.015
    Step 004: Loss: 1.008
    Step 005: Loss: 1.000
    Step 006: Loss: 0.992
    Step 007: Loss: 0.984
    Step 008: Loss: 0.976
    Step 009: Loss: 0.969
    Step 010: Loss: 0.961
    Step 011: Loss: 0.953
    Step 012: Loss: 0.945
    Step 013: Loss: 0.937
    Step 014: Loss: 0.929
    Step 015: Loss: 0.921
    Step 016: Loss: 0.913
    Step 017: Loss: 0.905
    Step 018: Loss: 0.897
    Step 019: Loss: 0.890
    Step 020: Loss: 0.882
    Step 021: Loss: 0.874
    Step 022: Loss: 0.866
    Step 023: Loss: 0.858
    Step 024: Loss: 0.850
    Step 025: Loss: 0.842
    Step 026: Loss: 0.835
    Step 027: Loss: 0.827
    Step 028: Loss: 0.819
    Step 029: Loss: 0.812
    Step 030: Loss: 0.804
    Step 031: Loss: 0.797
    Step 032: Loss: 0.790
    Step 033: Loss: 0.783
    Step 034: Loss: 0.775
    Step 035: Loss: 0.768
    Step 036: Loss: 0.762
    Step 037: Loss: 0.755
    Step 038: Loss: 0.748
    Step 039: Loss: 0.742
    Step 040: Loss: 0.736
    Step 041: Loss: 0.729
    Step 042: Loss: 0.723
    Step 043: Loss: 0.717
    Step 044: Loss: 0.712
    Step 045: Loss: 0.706
    Step 046: Loss: 0.701
    Step 047: Loss: 0.696
    Step 048: Loss: 0.690
    Step 049: Loss: 0.686
    Step 050: Loss: 0.681
    Step 051: Loss: 0.676
    Step 052: Loss: 0.672
    Step 053: Loss: 0.668
    Step 054: Loss: 0.664
    Step 055: Loss: 0.660
    Step 056: Loss: 0.657
    Step 057: Loss: 0.654
    Step 058: Loss: 0.651
    Step 059: Loss: 0.649
    Step 060: Loss: 0.647
    Step 061: Loss: 0.645
    Step 062: Loss: 0.644
    Step 063: Loss: 0.644
    Step 064: Loss: 0.643
    Step 065: Loss: 0.643
    Step 066: Loss: 0.644
    Step 067: Loss: 0.644
    Step 068: Loss: 0.644
    Step 069: Loss: 0.645
    Step 070: Loss: 0.645
    Step 071: Loss: 0.646
    Step 072: Loss: 0.646
    Step 073: Loss: 0.646
    Step 074: Loss: 0.646
    Step 075: Loss: 0.646
    Step 076: Loss: 0.646
    Step 077: Loss: 0.645
    Step 078: Loss: 0.645
    Step 079: Loss: 0.644
    Step 080: Loss: 0.644
    Step 081: Loss: 0.644
    Step 082: Loss: 0.643
    Step 083: Loss: 0.643
    Step 084: Loss: 0.643
    Step 085: Loss: 0.643
    Step 086: Loss: 0.643
    Step 087: Loss: 0.643
    Step 088: Loss: 0.643
    Step 089: Loss: 0.643
    Step 090: Loss: 0.643
    Step 091: Loss: 0.643
    Step 092: Loss: 0.643
    Step 093: Loss: 0.643
    Step 094: Loss: 0.643
    Step 095: Loss: 0.643
    Step 096: Loss: 0.643
    Step 097: Loss: 0.643
    Step 098: Loss: 0.643
    Step 099: Loss: 0.643
    Step 100: Loss: 0.643
    Step 101: Loss: 0.643
    Step 102: Loss: 0.643
    Step 103: Loss: 0.643
    Step 104: Loss: 0.643
    Step 105: Loss: 0.643
    Step 106: Loss: 0.643
    Step 107: Loss: 0.643
    Step 108: Loss: 0.643
    Step 109: Loss: 0.643
    Step 110: Loss: 0.643
    Step 111: Loss: 0.643
    Step 112: Loss: 0.643
    Step 113: Loss: 0.643
    Step 114: Loss: 0.643
    Step 115: Loss: 0.643
    Step 116: Loss: 0.643
    Step 117: Loss: 0.643
    Step 118: Loss: 0.643
    Step 119: Loss: 0.643
    Step 120: Loss: 0.643
    Step 121: Loss: 0.643
    Step 122: Loss: 0.643
    Step 123: Loss: 0.643
    Step 124: Loss: 0.643
    Step 125: Loss: 0.643
    Step 126: Loss: 0.643
    Step 127: Loss: 0.643
    Step 128: Loss: 0.643
    Step 129: Loss: 0.643
    Step 130: Loss: 0.643
    Step 131: Loss: 0.643
    Step 132: Loss: 0.643
    Step 133: Loss: 0.643
    Step 134: Loss: 0.643
    Step 135: Loss: 0.643
    Step 136: Loss: 0.643
    Step 137: Loss: 0.643
    Step 138: Loss: 0.643
    Step 139: Loss: 0.643
    Step 140: Loss: 0.643
    Step 141: Loss: 0.643
    Step 142: Loss: 0.643
    Step 143: Loss: 0.643
    Step 144: Loss: 0.643
    Step 145: Loss: 0.643
    Step 146: Loss: 0.643
    Step 147: Loss: 0.643
    Step 148: Loss: 0.643
    Step 149: Loss: 0.643
    Step 150: Loss: 0.643
    Step 151: Loss: 0.643
    Step 152: Loss: 0.643
    Step 153: Loss: 0.643
    Step 154: Loss: 0.643
    Step 155: Loss: 0.643
    Step 156: Loss: 0.643
    Step 157: Loss: 0.643
    Step 158: Loss: 0.643
    Step 159: Loss: 0.643
    Step 160: Loss: 0.643
    Step 161: Loss: 0.643
    Step 162: Loss: 0.643
    Step 163: Loss: 0.643
    Step 164: Loss: 0.643
    Step 165: Loss: 0.643
    Step 166: Loss: 0.643
    Step 167: Loss: 0.643
    Step 168: Loss: 0.643
    Step 169: Loss: 0.643
    Step 170: Loss: 0.643
    Step 171: Loss: 0.643
    Step 172: Loss: 0.643
    Step 173: Loss: 0.643
    Step 174: Loss: 0.643
    Step 175: Loss: 0.643
    Step 176: Loss: 0.643
    Step 177: Loss: 0.643
    Step 178: Loss: 0.643
    Step 179: Loss: 0.643
    Step 180: Loss: 0.643
    Step 181: Loss: 0.643
    Step 182: Loss: 0.643
    Step 183: Loss: 0.643
    Step 184: Loss: 0.643
    Step 185: Loss: 0.643
    Step 186: Loss: 0.643
    Step 187: Loss: 0.643
    Step 188: Loss: 0.643
    Step 189: Loss: 0.643
    Step 190: Loss: 0.643
    Step 191: Loss: 0.643
    Step 192: Loss: 0.643
    Step 193: Loss: 0.643
    Step 194: Loss: 0.643
    Step 195: Loss: 0.643
    Step 196: Loss: 0.643
    Step 197: Loss: 0.643
    Step 198: Loss: 0.643
    Step 199: Loss: 0.643
    Step 200: Loss: 0.643
    Step 201: Loss: 0.643
    Step 202: Loss: 0.643
    Step 203: Loss: 0.643
    Step 204: Loss: 0.643
    Step 205: Loss: 0.643
    Step 206: Loss: 0.643
    Step 207: Loss: 0.643
    Step 208: Loss: 0.643
    Step 209: Loss: 0.643
    Step 210: Loss: 0.643
    Step 211: Loss: 0.643
    Step 212: Loss: 0.643
    Step 213: Loss: 0.643
    Step 214: Loss: 0.643
    Step 215: Loss: 0.643
    Step 216: Loss: 0.643
    Step 217: Loss: 0.643
    Step 218: Loss: 0.643
    Step 219: Loss: 0.643
    Step 220: Loss: 0.643
    Step 221: Loss: 0.643
    Step 222: Loss: 0.643
    Step 223: Loss: 0.643
    Step 224: Loss: 0.643
    Step 225: Loss: 0.643
    Step 226: Loss: 0.643
    Step 227: Loss: 0.643
    Step 228: Loss: 0.643
    Step 229: Loss: 0.643
    Step 230: Loss: 0.643
    Step 231: Loss: 0.643
    Step 232: Loss: 0.643
    Step 233: Loss: 0.643
    Step 234: Loss: 0.643
    Step 235: Loss: 0.643
    Step 236: Loss: 0.643
    Step 237: Loss: 0.643
    Step 238: Loss: 0.643
    Step 239: Loss: 0.643
    Step 240: Loss: 0.643
    Step 241: Loss: 0.643
    Step 242: Loss: 0.643
    Step 243: Loss: 0.643
    Step 244: Loss: 0.643
    Step 245: Loss: 0.643
    Step 246: Loss: 0.643
    Step 247: Loss: 0.643
    Step 248: Loss: 0.643
    Step 249: Loss: 0.643
    Step 250: Loss: 0.643
    Step 251: Loss: 0.643
    Step 252: Loss: 0.643
    Step 253: Loss: 0.643
    Step 254: Loss: 0.643
    Step 255: Loss: 0.643
    Step 256: Loss: 0.643
    Step 257: Loss: 0.643
    Step 258: Loss: 0.643
    Step 259: Loss: 0.643
    Step 260: Loss: 0.643
    Step 261: Loss: 0.643
    Step 262: Loss: 0.643
    Step 263: Loss: 0.643
    Step 264: Loss: 0.643
    Step 265: Loss: 0.643
    Step 266: Loss: 0.643
    Step 267: Loss: 0.643
    Step 268: Loss: 0.643
    Step 269: Loss: 0.643
    Step 270: Loss: 0.643
    Step 271: Loss: 0.643
    Step 272: Loss: 0.643
    Step 273: Loss: 0.643
    Step 274: Loss: 0.643
    Step 275: Loss: 0.643
    Step 276: Loss: 0.643
    Step 277: Loss: 0.643
    Step 278: Loss: 0.643
    Step 279: Loss: 0.643
    Step 280: Loss: 0.643
    Step 281: Loss: 0.643
    Step 282: Loss: 0.643
    Step 283: Loss: 0.643
    Step 284: Loss: 0.643
    Step 285: Loss: 0.643
    Step 286: Loss: 0.643
    Step 287: Loss: 0.643
    Step 288: Loss: 0.643
    Step 289: Loss: 0.643
    Step 290: Loss: 0.643
    Step 291: Loss: 0.643
    Step 292: Loss: 0.643
    Step 293: Loss: 0.643
    Step 294: Loss: 0.643
    Step 295: Loss: 0.643
    Step 296: Loss: 0.643
    Step 297: Loss: 0.643
    Step 298: Loss: 0.643
    Step 299: Loss: 0.643
    Step 300: Loss: 0.643
    Step 301: Loss: 0.643
    Step 302: Loss: 0.643
    Step 303: Loss: 0.643
    Step 304: Loss: 0.643
    Step 305: Loss: 0.643
    Step 306: Loss: 0.643
    Step 307: Loss: 0.643
    Step 308: Loss: 0.643
    Step 309: Loss: 0.643
    Step 310: Loss: 0.643
    Step 311: Loss: 0.643
    Step 312: Loss: 0.643
    Step 313: Loss: 0.643
    Step 314: Loss: 0.643
    Step 315: Loss: 0.643
    Step 316: Loss: 0.643
    Step 317: Loss: 0.643
    Step 318: Loss: 0.643
    Step 319: Loss: 0.643
    Step 320: Loss: 0.643
    Step 321: Loss: 0.643
    Step 322: Loss: 0.643
    Step 323: Loss: 0.643
    Step 324: Loss: 0.643
    Step 325: Loss: 0.643
    Step 326: Loss: 0.643
    Step 327: Loss: 0.643
    Step 328: Loss: 0.643
    Step 329: Loss: 0.643
    Step 330: Loss: 0.643
    Step 331: Loss: 0.643
    Step 332: Loss: 0.643
    Step 333: Loss: 0.643
    Step 334: Loss: 0.643
    Step 335: Loss: 0.643
    Step 336: Loss: 0.643
    Step 337: Loss: 0.643
    Step 338: Loss: 0.643
    Step 339: Loss: 0.643
    Step 340: Loss: 0.643
    Step 341: Loss: 0.643
    Step 342: Loss: 0.643
    Step 343: Loss: 0.643
    Step 344: Loss: 0.643
    Step 345: Loss: 0.643
    Step 346: Loss: 0.643
    Step 347: Loss: 0.643
    Step 348: Loss: 0.643
    Step 349: Loss: 0.643
    Step 350: Loss: 0.643
    Step 351: Loss: 0.643
    Step 352: Loss: 0.643
    Step 353: Loss: 0.643
    Step 354: Loss: 0.643
    Step 355: Loss: 0.643
    Step 356: Loss: 0.643
    Step 357: Loss: 0.643
    Step 358: Loss: 0.643
    Step 359: Loss: 0.643
    Step 360: Loss: 0.643
    Step 361: Loss: 0.643
    Step 362: Loss: 0.643
    Step 363: Loss: 0.643
    Step 364: Loss: 0.643
    Step 365: Loss: 0.643
    Step 366: Loss: 0.643
    Step 367: Loss: 0.643
    Step 368: Loss: 0.643
    Step 369: Loss: 0.643
    Step 370: Loss: 0.643
    Step 371: Loss: 0.643
    Step 372: Loss: 0.643
    Step 373: Loss: 0.643
    Step 374: Loss: 0.643
    Step 375: Loss: 0.643
    Step 376: Loss: 0.643
    Step 377: Loss: 0.643
    Step 378: Loss: 0.643
    Step 379: Loss: 0.643
    Step 380: Loss: 0.643
    Step 381: Loss: 0.643
    Step 382: Loss: 0.643
    Step 383: Loss: 0.643
    Step 384: Loss: 0.643
    Step 385: Loss: 0.643
    Step 386: Loss: 0.643
    Step 387: Loss: 0.643
    Step 388: Loss: 0.643
    Step 389: Loss: 0.643
    Step 390: Loss: 0.643
    Step 391: Loss: 0.643
    Step 392: Loss: 0.643
    Step 393: Loss: 0.643
    Step 394: Loss: 0.643
    Step 395: Loss: 0.643
    Step 396: Loss: 0.643
    Step 397: Loss: 0.643
    Step 398: Loss: 0.643
    Step 399: Loss: 0.643
    Step 400: Loss: 0.643
    Step 401: Loss: 0.643
    Step 402: Loss: 0.643
    Step 403: Loss: 0.643
    Step 404: Loss: 0.643
    Step 405: Loss: 0.643
    Step 406: Loss: 0.643
    Step 407: Loss: 0.643
    Step 408: Loss: 0.643
    Step 409: Loss: 0.643
    Step 410: Loss: 0.643
    Step 411: Loss: 0.643
    Step 412: Loss: 0.643
    Step 413: Loss: 0.643
    Step 414: Loss: 0.643
    Step 415: Loss: 0.643
    Step 416: Loss: 0.643
    Step 417: Loss: 0.643
    Step 418: Loss: 0.643
    Step 419: Loss: 0.643
    Step 420: Loss: 0.643
    Step 421: Loss: 0.643
    Step 422: Loss: 0.643
    Step 423: Loss: 0.643
    Step 424: Loss: 0.643
    Step 425: Loss: 0.643
    Step 426: Loss: 0.643
    Step 427: Loss: 0.643
    Step 428: Loss: 0.643
    Step 429: Loss: 0.643
    Step 430: Loss: 0.643
    Step 431: Loss: 0.643
    Step 432: Loss: 0.643
    Step 433: Loss: 0.643
    Step 434: Loss: 0.643
    Step 435: Loss: 0.643
    Step 436: Loss: 0.643
    Step 437: Loss: 0.643
    Step 438: Loss: 0.643
    Step 439: Loss: 0.643
    Step 440: Loss: 0.643
    Step 441: Loss: 0.643
    Step 442: Loss: 0.643
    Step 443: Loss: 0.643
    Step 444: Loss: 0.643
    Step 445: Loss: 0.643
    Step 446: Loss: 0.643
    Step 447: Loss: 0.643
    Step 448: Loss: 0.643
    Step 449: Loss: 0.643
    Step 450: Loss: 0.643
    Step 451: Loss: 0.643
    Step 452: Loss: 0.643
    Step 453: Loss: 0.643
    Step 454: Loss: 0.643
    Step 455: Loss: 0.643
    Step 456: Loss: 0.643
    Step 457: Loss: 0.643
    Step 458: Loss: 0.643
    Step 459: Loss: 0.643
    Step 460: Loss: 0.643
    Step 461: Loss: 0.643
    Step 462: Loss: 0.643
    Step 463: Loss: 0.643
    Step 464: Loss: 0.643
    Step 465: Loss: 0.643
    Step 466: Loss: 0.643
    Step 467: Loss: 0.643
    Step 468: Loss: 0.643
    Step 469: Loss: 0.643
    Step 470: Loss: 0.643
    Step 471: Loss: 0.643
    Step 472: Loss: 0.643
    Step 473: Loss: 0.643
    Step 474: Loss: 0.643
    Step 475: Loss: 0.643
    Step 476: Loss: 0.643
    Step 477: Loss: 0.643
    Step 478: Loss: 0.643
    Step 479: Loss: 0.643
    Step 480: Loss: 0.643
    Step 481: Loss: 0.643
    Step 482: Loss: 0.643
    Step 483: Loss: 0.643
    Step 484: Loss: 0.643
    Step 485: Loss: 0.643
    Step 486: Loss: 0.643
    Step 487: Loss: 0.643
    Step 488: Loss: 0.643
    Step 489: Loss: 0.643
    Step 490: Loss: 0.643
    Step 491: Loss: 0.643
    Step 492: Loss: 0.643
    Step 493: Loss: 0.643
    Step 494: Loss: 0.643
    Step 495: Loss: 0.643
    Step 496: Loss: 0.643
    Step 497: Loss: 0.643
    Step 498: Loss: 0.643
    Step 499: Loss: 0.643

```python
# View the distribution parameters

print("Class conditional means:")
print(class_conditionals_binary.loc.numpy())
print("\nClass conditional standard deviations:")
print(class_conditionals_binary.stddev().numpy())
```

    Class conditional means:
    [[4.9952383 3.4285715]
     [6.2128205 2.8512821]]

    Class conditional standard deviations:
    [[2.876563  2.8765185]
     [2.876563  2.8765185]]

```python

```

```python
# Plot the loss and convergence of the standard deviation parameters

fig, ax = plt.subplots(1, 2, figsize=(14, 5))
ax[0].plot(nlls)
ax[0].set_title("Loss vs epoch")
ax[0].set_xlabel("Epoch")
ax[0].set_ylabel("Average negative log-likelihood")
for k in [0, 1]:
    ax[1].plot(scales_arr[:, k], color=label_colours_binary[k], label=labels_binary[k])
ax[1].set_title("Standard deviation ML estimates vs epoch")
ax[1].set_xlabel("Epoch")
ax[1].set_ylabel("Standard deviation")
plt.legend()
plt.show()
```

We can also plot the contours of the class-conditional Gaussian distributions as before, this time with just binary labelled data. Notice the contours are the same for each class, just with a different centre location.

```python
# Plot the training data with the class-conditional density contours

plt.figure(figsize=(10, 6))
plot_data(x_train, y_train_binary, labels_binary, label_colours_binary)
x0_min, x0_max = x_train[:, 0].min(), x_train[:, 0].max()
x1_min, x1_max = x_train[:, 1].min(), x_train[:, 1].max()
contour_plot((x0_min, x0_max), (x1_min, x1_max), class_conditionals_binary.prob, 2, label_colours_binary)
plt.title("Training set with class-conditional density contours")
plt.show()
```

We can also plot the decision regions for this binary classifier model, notice that the decision boundary is now linear.

```python
# Plot the model's decision regions

plt.figure(figsize=(10, 6))
plot_data(x_train, y_train_binary, labels_binary, label_colours_binary)
x0_min, x0_max = x_train[:, 0].min(), x_train[:, 0].max()
x1_min, x1_max = x_train[:, 1].min(), x_train[:, 1].max()
contour_plot((x0_min, x0_max), (x1_min, x1_max),
             lambda x: predict_class(prior_binary, class_conditionals_binary, x),
             1, label_colours_binary, levels=[-0.5, 0.5, 1.5],
             num_points=500)
plt.title("Training set with decision regions")
plt.show()
```

#### Link to logistic regression

In fact, we can see that our predictive distribution $P(Y=y_0 | X)$ can be written as follows:

$$
\begin{align}
P(Y=y_0 | X) =& ~\frac{P(X | Y=y_0)P(Y=y_0)}{P(X | Y=y_0)P(Y=y_0) + P(X | Y=y_1)P(Y=y_1)}\\
=&  ~\frac{1}{1 + \frac{P(X | Y=y_1)P(Y=y_1)}{P(X | Y=y_0)P(Y=y_0)}}\\
=& ~\sigma(a)
\end{align}
$$

where $\sigma(a) = \frac{1}{1 + e^{-a}}$ is the sigmoid function, and $a = \log\frac{P(X | Y=y_0)P(Y=y_0)}{P(X | Y=y_1)P(Y=y_1)}$ is the _log-odds_.

With our additional modelling assumption of a shared covariance matrix $\Sigma$, it can be shown (using the Gaussian pdf) that $a$ is in fact a linear function of $X$:

$$
a = w^T X + w_0
$$

where

$$
\begin{align}
w =& ~\Sigma^{-1} (\mu_0 - \mu_1)\\
w_0 =& -\frac{1}{2}\mu_0^T \Sigma^{-1}\mu_0 + \frac{1}{2}\mu_1^T\Sigma^{-1}\mu_1 + \log\frac{P(Y=y_0)}{P(Y=y_1)}
\end{align}
$$

The model therefore takes the form $P(Y=y_0 | X) = \sigma(w^T X + w_0)$, with weights $w\in\mathbb{R}^2$ and bias $w_0\in\mathbb{R}$. This is the form used by logistic regression, and explains why the decision boundary above is linear.

In the above we have outlined the derivation of the generative logistic regression model. The parameters are typically estimated with maximum likelihood, as we have done.

Finally, we will use the above equations to directly parameterise the output Bernoulli distribution of the generative logistic regression model.

You should now write the following function, according to the following specification:

* The inputs to the function are:
  * the prior distribution `prior` over the two classes
  * the (batched) class-conditional distribution `class_conditionals`
* The function should use the parameters of the above distributions to compute the weights and bias terms $w$ and $w_0$ as above
* The function should then return a tuple of two numpy arrays for $w$ and $w_0$

```python
#### GRADED CELL ####

# Complete the following function.
# Make sure to not change the function name or arguments.

def get_logistic_regression_params(prior, class_conditionals):
    """
    This function takes the prior distribution and class-conditional distribution as inputs.
    This function should compute the weights and bias terms of the generative logistic
    regression model as above, and return them in a 2-tuple of numpy arrays of shapes
    (2,) and () respectively.
    """
    sigma_inv=np.linalg.inv(class_conditionals.covariance().numpy())
    print(sigma_inv)
    u0=class_conditionals.loc.numpy()[0]
    u1=class_conditionals.loc.numpy()[1]

    w=np.dot(sigma_inv,(u0-u1))
    print(w)

    w0=np.dot(np.dot(-0.5*u0.T,sigma_inv),u0)+np.dot(np.dot(0.5*u1.T,sigma_inv),u1)+np.log(prior.prob(0)/prior.prob(1))
    print(w0)
    return w,w0

```

```python
# Run your function to get the logistic regression parameters

w, w0 = get_logistic_regression_params(prior_binary, class_conditionals_binary)
```

    [[0.80699   0.       ]
     [0.        3.7892613]]
    [-1.0832673  1.9806364]
    -0.8766242

We can now use these parameters to make a contour plot to display the predictive distribution of our logistic regression model.

```python
# Plot the training data with the logistic regression prediction contours

fig, ax = plt.subplots(1, 1, figsize=(10, 6))
plot_data(x_train, y_train_binary, labels_binary, label_colours_binary)
x0_min, x0_max = x_train[:, 0].min(), x_train[:, 0].max()
x1_min, x1_max = x_train[:, 1].min(), x_train[:, 1].max()
X0, X1 = get_meshgrid((x0_min, x0_max), (x1_min, x1_max))

logits = np.dot(np.array([X0.ravel(), X1.ravel()]).T, w) + w0
Z = tf.math.sigmoid(logits)
lr_contour = ax.contour(X0, X1, np.array(Z).T.reshape(*X0.shape), levels=10)
ax.clabel(lr_contour, inline=True, fontsize=10)
contour_plot((x0_min, x0_max), (x1_min, x1_max),
             lambda x: predict_class(prior_binary, class_conditionals_binary, x),
             1, label_colours_binary, levels=[-0.5, 0.5, 1.5],
             num_points=300)
plt.title("Training set with prediction contours")
plt.show()
```

Congratulations on completing this programming assignment! In the next week of the course we will look at Bayesian neural networks and uncertainty quantification.

```python

```


```python
import numpy
import sympy
from matplotlib import pyplot
%matplotlib inline
```

```python
# Set the font family and size to use for Matplotlib figures.
pyplot.rcParams['font.family'] = 'serif'
pyplot.rcParams['font.size'] = 16
```

## Symbolized equation

```python
sympy.init_printing()
```

```python
x, nu, t = sympy.symbols('x nu t')
phi = (sympy.exp(-(x - 4 * t)**2 / (4 * nu * (t + 1)))) + \
sympy.exp(-(x - 4 * t - 2 * sympy.pi)**2 / (4 * nu * (t + 1)))
```

```python
phi
```

```python
phiprime = phi.diff(x)
```

```python
phiprime
```

```python
from sympy.utilities.lambdify import lambdify

u = -2 * nu * (phiprime / phi) + 4
print(u)
```

    -2*nu*(-(-8*t + 2*x)*exp(-(-4*t + x)**2/(4*nu*(t + 1)))/(4*nu*(t + 1)) - (-8*t + 2*x - 4*pi)*exp(-(-4*t + x - 2*pi)**2/(4*nu*(t + 1)))/(4*nu*(t + 1)))/(exp(-(-4*t + x - 2*pi)**2/(4*nu*(t + 1))) + exp(-(-4*t + x)**2/(4*nu*(t + 1)))) + 4

```python
u_lamb = lambdify((t, x, nu) ,u) # input t,x,nu; output to u
print('The value of u at t=1, x=4, nu=3 is {}'.format(u_lamb(1, 4, 3)))
```

    The value of u at t=1, x=4, nu=3 is 3.49170664206445

## Back to equation

```python
# set parameters
nx = 101 # spatial grids
L = 2.0 * numpy.pi # length of domain
dx = L / (nx - 1) # step size
nu = 0.07 # viscosity
nt = 100 # time grids
sigma = 0.1 # CFL limit
dt = sigma * dx**2 / nu # calculate time step size

# Discretize domain
x = numpy.linspace(0.0, L, num=nx)
```

```python
# initial condition
t = 0.0
u0 = numpy.array([u_lamb(t, xi, nu) for xi in x])
```

```python
u0
```

    array([4.        , 4.06283185, 4.12566371, 4.18849556, 4.25132741,
           4.31415927, 4.37699112, 4.43982297, 4.50265482, 4.56548668,
           4.62831853, 4.69115038, 4.75398224, 4.81681409, 4.87964594,
           4.9424778 , 5.00530965, 5.0681415 , 5.13097336, 5.19380521,
           5.25663706, 5.31946891, 5.38230077, 5.44513262, 5.50796447,
           5.57079633, 5.63362818, 5.69646003, 5.75929189, 5.82212374,
           5.88495559, 5.94778745, 6.0106193 , 6.07345115, 6.136283  ,
           6.19911486, 6.26194671, 6.32477856, 6.38761042, 6.45044227,
           6.51327412, 6.57610598, 6.63893783, 6.70176967, 6.76460125,
           6.82742866, 6.89018589, 6.95176632, 6.99367964, 6.72527549,
           4.        , 1.27472451, 1.00632036, 1.04823368, 1.10981411,
           1.17257134, 1.23539875, 1.29823033, 1.36106217, 1.42389402,
           1.48672588, 1.54955773, 1.61238958, 1.67522144, 1.73805329,
           1.80088514, 1.863717  , 1.92654885, 1.9893807 , 2.05221255,
           2.11504441, 2.17787626, 2.24070811, 2.30353997, 2.36637182,
           2.42920367, 2.49203553, 2.55486738, 2.61769923, 2.68053109,
           2.74336294, 2.80619479, 2.86902664, 2.9318585 , 2.99469035,
           3.0575222 , 3.12035406, 3.18318591, 3.24601776, 3.30884962,
           3.37168147, 3.43451332, 3.49734518, 3.56017703, 3.62300888,
           3.68584073, 3.74867259, 3.81150444, 3.87433629, 3.93716815,
           4.        ])

```python
# Plot the initial conditions.
pyplot.figure(figsize=(6.0, 4.0))
pyplot.title('Initial conditions')
pyplot.xlabel('x')
pyplot.ylabel('u')
pyplot.grid()
pyplot.plot(x, u0, color='C0', linestyle='-', linewidth=2)
pyplot.xlim(0.0, L)
pyplot.ylim(0.0, 10.0);
```

## Push forward

```python
u = u0.copy()
for n in range(nt):
    un = u.copy()
    # update
    u[1:-1] = (un[1:-1] - \
              un[1:-1] * dt / dx * (un[1:-1] - un[:-2]) + \
              nu * dt / dx**2 * (un[2:] - 2 * un[1:-1] + un[:-2]))
    # update boundary points
    u[0] = (un[0] - un[0] * dt / dx * (un[0] - un[-1]) + \
           nu * dt / dx**2 * (un[1] - 2 * un[0] + un[-1]))
    u[-1] = (un[-1] - un[-1] * dt / dx * (un[-1] - un[-2]) + \
            nu * dt / dx**2 * (un[0] - 2 * un[-1] + un[-2]))
```

```python
# Compute the analytical solution.
u_analytical = numpy.array([u_lamb(nt * dt, xi, nu) for xi in x])
```

```python
# Plot the numerical solution along with the analytical solution.
pyplot.figure(figsize=(6.0, 4.0))
pyplot.xlabel('x')
pyplot.ylabel('u')
pyplot.grid()
pyplot.plot(x, u, label='Numerical',
            color='C0', linestyle='-', linewidth=2)
pyplot.plot(x, u_analytical, label='Analytical',
            color='C1', linestyle='--', linewidth=2)
pyplot.legend()
pyplot.xlim(0.0, L)
pyplot.ylim(0.0, 10.0);
```

## Animation

```python
from matplotlib import animation
from IPython.display import HTML
```

```python
def burgers(u0, dx, dt, nu, nt=20):
    """
    Computes the numerical solution of the 1D Burgers' equation
    over the time steps.

    Parameters
    ----------
    u0 : numpy.ndarray
        The initial conditions as a 1D array of floats.
    dx : float
        The grid spacing.
    dt : float
        The time-step size.
    nu : float
        The viscosity.
    nt : integer, optional
        The number of time steps to compute;
        default: 20.

    Returns
    -------
    u_hist : list of numpy.ndarray objects
        The history of the numerical solution.
    """
    u_hist = [u0.copy()]
    u = u0.copy()
    for n in range(nt):
        un = u.copy()
        # Update all interior points.
        u[1:-1] = (un[1:-1] -
                   un[1:-1] * dt / dx * (un[1:-1] - un[:-2]) +
                   nu * dt / dx**2 * (un[2:] - 2 * un[1:-1] + un[:-2]))
        # Update boundary points.
        u[0] = (un[0] -
                un[0] * dt / dx * (un[0] - un[-1]) +
                nu * dt / dx**2 * (un[1] - 2 * un[0] + un[-1]))
        u[-1] = (un[-1] -
                un[-1] * dt / dx * (un[-1] - un[-2]) +
                nu * dt / dx**2 * (un[0] - 2 * un[-1] + un[-2]))
        u_hist.append(u.copy())
    return u_hist
```

```python
# Compute the history of the numerical solution.
u_hist = burgers(u0, dx, dt, nu, nt=nt)
```

```python
# Compute the history of the analytical solution.
u_analytical = [numpy.array([u_lamb(n * dt, xi, nu) for xi in x])
                for n in range(nt)]
```

```python
fig = pyplot.figure(figsize=(6.0, 4.0))
pyplot.xlabel('x')
pyplot.ylabel('u')
pyplot.grid()
u0_analytical = numpy.array([u_lamb(0.0, xi, nu) for xi in x])
line1 = pyplot.plot(x, u0, label='Numerical',
                    color='C0', linestyle='-', linewidth=2)[0]
line2 = pyplot.plot(x, u0_analytical, label='Analytical',
                    color='C1', linestyle='--', linewidth=2)[0]
pyplot.legend()
pyplot.xlim(0.0, L)
pyplot.ylim(0.0, 10.0)
fig.tight_layout()
```

```python
def update_plot(n, u_hist, u_analytical):
    """
    Update the lines y-data of the Matplotlib figure.

    Parameters
    ----------
    n : integer
        The time-step index.
    u_hist : list of numpy.ndarray objects
        The history of the numerical solution.
    u_analytical : list of numpy.ndarray objects
        The history of the analytical solution.
    """
    fig.suptitle('Time step {:0>2}'.format(n))
    line1.set_ydata(u_hist[n])
    line2.set_ydata(u_analytical[n])
```

```python
# Create an animation.
anim = animation.FuncAnimation(fig, update_plot,
                               frames=nt, fargs=(u_hist, u_analytical),
                               interval=100)
```

```python
# Display the video.
HTML(anim.to_html5_video())
```

## Computing time difference

```python
%%timeit
# Set initial conditions.
u = numpy.array([u_lamb(t, x0, nu) for x0 in x])
# Integrate in time using a nested for loop.
for n in range(nt):
    un = u.copy()
    # Update all interior points and the left boundary point.
    for i in range(nx - 1):
        u[i] = (un[i] -
                un[i] * dt / dx *(un[i] - un[i - 1]) +
                nu * dt / dx**2 * (un[i + 1] - 2 * un[i] + un[i - 1]))
    # Update the right boundary.
    u[-1] = (un[-1] -
             un[-1] * dt / dx * (un[-1] - un[-2]) +
             nu * dt / dx**2 * (un[0]- 2 * un[-1] + un[-2]))
```

    20.3 ms ± 70.5 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

```python
%%timeit
# Set initial conditions.
u = numpy.array([u_lamb(t, xi, nu) for xi in x])
# Integrate in time using array operations.
for n in range(nt):
    un = u.copy()
    # Update all interior points.
    u[1:-1] = (un[1:-1] -
               un[1:-1] * dt / dx * (un[1:-1] - un[:-2]) +
               nu * dt / dx**2 * (un[2:] - 2 * un[1:-1] + un[:-2]))
    # Update boundary points.
    u[0] = (un[0] -
            un[0] * dt / dx * (un[0] - un[-1]) +
            nu * dt / dx**2 * (un[1] - 2 * un[0] + un[-1]))
    u[-1] = (un[-1] -
            un[-1] * dt / dx * (un[-1] - un[-2]) +
            nu * dt / dx**2 * (un[0] - 2 * un[-1] + un[-2]))
```

    2.29 ms ± 20.6 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

```python

```


# Solutions to Exercises (not Activities) in the Bohemian Unit

1: Write down as many questions as you can for this unit.

Maybe this is the most important one of these in this book (OER).  Your questions are as likely as ours to be productive.  But, here are some of ours.  Most have no answers that we know of.  In no particular order, here we go. For a given integer polynomial, which companion Bohemian matrix has _minimal_ height? Which matrix in the family has the maximum determinant?  Which matrix in the family has the maximum characteristic height? Minimum nonzero determinant? How many matrices are singular? How many are stable? How many have multiple eigenvalues? How many nilpotent matrices are there? How many non-normal matrices are there?  How many commuting pairs are there? What is the distribution of eigenvalue condition numbers? How many different eigenvalues are there?  How many matrices have a given characteristic polynomial? How many have nontrivial Jordan form? How many have nontrivial Smith form? How many are orthogonal? unimodular? How many matrices have inverses that are also Bohemian? With the same height? (In this case we say the matrix family has _rhapsody_). Given the eigenvalues, can we find a matrix in the family with those eigenvalues?

The colouring scheme we settled on uses an approximate inversion of the cumulative density function and maps that onto a _perceptually even_ sequential colour map like viridis or cividis.  We have only just begun to explore more sophisticated schemes, but this one has the advantage that you can sort of "see" the probability density in the colours.  Are there better methods?  Can we do this usefully in $3D$?  What about multi-dimensional problems? Tensors?

2: Looking back at the Fractals Unit, there seem to be clear connections with this unit.  Discuss them.

Some of the Bohemian eigenvalue pictures are suggestive of fractals, especially the upper Hessenberg zero diagonal Toeplitz ones.  In fact we have just managed to prove that Sierpinski-like triangles genuinely appear in some of these (when the population has three elements).  In some other cases we think we have also managed to prove that in the limit as $m\to\infty$ we get a genuine fractal.  We can't think of a way to connect the Julia set idea, though.

3: Looking back at the Rootfinding Unit, there seem to be clear connections with this unit.  Discuss them.

Mostly the connection is practical: we want a good way to find all the roots of the characteristic polynomial (when we go that route, possibly because of the compression factor).  Newton's method can be used (sometimes!) to "polish" the roots to greater accuracy, for instance.

4: Looking back at the Continued Fractions Unit, it seems a stretch to connect them.  Can you find a connection?

__One connection__: We thought of taking a general matrix and letting its entries be the partial quotients of the continued fraction of a number chosen "at random" in $[0,1)$.  To make that "at random" have the right distribution turned out to be simple enough, because the distribution of partial quotients is known (called the [Gauss-Kuzmin distribution](https://en.wikipedia.org/wiki/Gauss–Kuzmin_distribution)) and its relation to the probability distribution of the iterates $x_n$ of the Gauss map is also known (and called the "Gauss measure"). This is
\begin{equation}
F(x) = \frac{1}{\ln 2}\int_{t=0}^x \frac{1}{1+t}\,dt = \mathrm{lg}(1+x)
\end{equation}
where the symbol lg means "log to base 2" and is frequently seen in computer science.  We can invert this cumulative distribution by solving $u=\mathrm{lg}(1+x)$ by raising both sides to the power $2$, like so: $2^u = 1+x$ so $x=2^u-1$.  Therefore, sampling $u$ uniformly on $(0,1)$ will give $x$ distributed on $(0,1)$ according to the Gauss measure; then taking the fractional part of $1/x$ will give us the partial quotients with the correct distribution.

Doing this a large number of times and plotting the eigenvalue density we get a very interesting image; we're still thinking about it. We were surprised that there was a pattern, and we don't yet know how to explain it. This isn't _quite_ a Bohemian matrix problem, because the partial quotients of most numbers are unbounded.  Still, very large entries are not likely, even though this is quite definitely what is known as a "heavy-tailed distribution" or "fat-tailed distribution".  The expected value of a partial quotient is infinity!  The use of floating-point essentially bounds the largest entry in practice; we do not know what this does to the statistics.

__Another way to connect__ continued fractions to Bohemian matrices might be to compute the continued fractions of the eigenvalues of one of our earlier computations, and see if there was any correlation with the "holes" (there might be—we have not tried this).

```python
import itertools
import random
import numpy as np
from numpy.polynomial import Polynomial as Poly
import matplotlib as plt
import time
import csv
import math
from PIL import Image
import json
import ast
import sys
sys.path.insert(0,'../../code')
from bohemian_inheritance import *
from densityPlot import *
```

```python
rng = np.random.default_rng(2022)
# We will convert uniform floats u on (0,1) to
# floats that have the Gauss measure by x = 2^u - 1
# and this will get the right distribution of partial quotients

# This lambda function replaced by broadcast operations which are faster
# partial_quotient = lambda u: math.floor( 1/(2**u-1)) # u=0 is unlikely

# 5000 in 1.5 seconds, 50,000 in 12 seconds, 500,000 in 2 minutes, 1 million in 4 minutes
# six hours and 14 minutes for one hundred million matrices at mdim=13
Nsample = 1*10**5
mdim = 6
A = Bohemian(mdim)
sequencelength = A.getNumberOfMatrixEntries()
one = np.ones(sequencelength)
two = 2*one
start = time.time()
B =  0.6*mdim*(Nsample)**(0.25)
print( "Bounding box is width 2*{}".format(B))
bounds = [-B,B,-B,B] # Found by experiment; Depends on Nsample!
nrow = math.floor(9*B)  # These need to be adjusted depending on Nsample as well
ncol = math.floor(9*B)
image = DensityPlot(bounds, nrow, ncol)
for k in range(Nsample):
    u = rng.random(size=sequencelength)
    r = np.power(two,u)-one
    p = np.floor_divide(one,r)
    A.makeMatrix( p )
    image.addPoints(A.eig())

# We encode the population into a label which we will use for the filename.
poplabel = "ContinuedFraction"
cmap = 'viridis'
fname = '../Supplementary Material/Bohemian/dense/pop_{}_{}_{}N{}.png'.format(poplabel,cmap,mdim,Nsample)
image.makeDensityPlot(cmap, filename=fname, bgcolor=[0, 0, 0, 1], colorscale="cumulative")
finish = time.time()
print("Took {} seconds to compute and plot ".format(finish-start))

```

book/Solutions/Solutions to Exercises (not Activities) in the Bohemian Unit.ipynb