# Programming Assignment

## Naive Bayes and logistic regression

### Instructions

In this notebook, you will write code to develop a Naive Bayes classifier model to the Iris dataset using Distribution objects from TensorFlow Probability. You will also explore the connection between the Naive Bayes classifier and logistic regression.

Some code cells are provided you in the notebook. You should avoid editing provided code, and make sure to execute the cells in order to avoid unexpected errors. Some cells begin with the line:

`#### GRADED CELL ####`

Don't move or edit this first line - this is what the automatic grader looks for to recognise graded cells. These cells require you to write your own code to complete them, and are automatically graded when you submit the notebook. Don't edit the function name or signature provided in these cells, otherwise the automatic grader might not function properly.

### How to submit

Complete all the tasks you are asked for in the worksheet. When you have finished and are happy with your code, press the **Submit Assignment** button at the top of this notebook.

### Let's get started!

We'll start running some imports, and loading the dataset. Do not edit the existing imports in the following cell. If you would like to make further Tensorflow imports, you should add them here.

```python
#### PACKAGE IMPORTS ####

# Run this cell first to import all required packages. Do not make any imports elsewhere in the notebook
import tensorflow as tf
import tensorflow_probability as tfp
tfd = tfp.distributions

print("TF version:", tf.__version__)
print("TFP version:", tfp.__version__)

import numpy as np
import matplotlib.pyplot as plt
from sklearn.metrics import accuracy_score
from sklearn import datasets, model_selection
%matplotlib inline

# If you would like to make further imports from TensorFlow or TensorFlow Probability, add them here
```

    TF version: 2.3.0
    TFP version: 0.11.0

#### The Iris dataset

In this assignment, you will use the [Iris dataset](https://scikit-learn.org/stable/auto_examples/datasets/plot_iris_dataset.html). It consists of 50 samples from each of three species of Iris (Iris setosa, Iris virginica and Iris versicolor). Four features were measured from each sample: the length and the width of the sepals and petals, in centimeters. For a reference, see the following papers:

- R. A. Fisher. "The use of multiple measurements in taxonomic problems". Annals of Eugenics. 7 (2): 179–188, 1936.

Your goal is to construct a Naive Bayes classifier model that predicts the correct class from the sepal length and sepal width features. Under certain assumptions about this classifier model, you will explore the relation to logistic regression.

#### Load and prepare the data

We will first read in the Iris dataset, and split the dataset into training and test sets.

```python
# Load the dataset
iris = datasets.load_iris()
```

```python
# Use only the first two features: sepal length and width
data = iris.data[:, :2]
targets = iris.target
```

```python
# Randomly shuffle the data and make train and test splits
x_train, x_test, y_train, y_test = model_selection.train_test_split(data, targets, test_size=0.2)
```

```python
# Plot the training data
labels = {0: 'Iris-Setosa', 1: 'Iris-Versicolour', 2: 'Iris-Virginica'}
label_colours = ['blue', 'orange', 'green']

def plot_data(x, y, labels, colours):
    for c in np.unique(y):
        inx = np.where(y == c)
        plt.scatter(x[inx, 0], x[inx, 1], label=labels[c], c=colours[c])
    plt.title("Training set")
    plt.xlabel("Sepal length (cm)")
    plt.ylabel("Sepal width (cm)")
    plt.legend()

plt.figure(figsize=(8, 5))
plot_data(x_train, y_train, labels, label_colours)
plt.show()
```

### Naive Bayes classifier

We will briefly review the Naive Bayes classifier model. The fundamental equation for this classifier is Bayes' rule:

$$
P(Y=y_k | X_1,\ldots,X_d) = \frac{P(X_1,\ldots,X_d | Y=y_k)P(Y=y_k)}{\sum_{k=1}^K P(X_1,\ldots,X_d | Y=y_k)P(Y=y_k)}
$$

In the above, $d$ is the number of features or dimensions in the inputs $X$ (in our case $d=2$), and $K$ is the number of classes (in our case $K=3$). The distribution $P(Y)$ is the class prior distribution, which is a discrete distribution over $K$ classes. The distribution $P(X | Y)$ is the class-conditional distribution over inputs.

The Naive Bayes classifier makes the assumption that the data features $X_i$ are conditionally independent give the class $Y$ (the 'naive' assumption). In this case, the class-conditional distribution decomposes as

$$
\begin{align}
P(X | Y=y_k) &= P(X_1,\ldots,X_d | Y=y_k)\\
&= \prod_{i=1}^d P(X_i | Y=y_k)
\end{align}
$$

This simplifying assumption means that we typically need to estimate far fewer parameters for each of the distributions $P(X_i | Y=y_k)$ instead of the full joint distribution $P(X | Y=y_k)$.

Once the class prior distribution and class-conditional densities are estimated, the Naive Bayes classifier model can then make a class prediction $\hat{Y}$ for a new data input $\tilde{X} := (\tilde{X}_1,\ldots,\tilde{X}_d)$ according to

$$
\begin{align}
\hat{Y} &= \text{argmax}_{y_k} P(Y=y_k | \tilde{X}_1,\ldots,\tilde{X}_d) \\
&= \text{argmax}_{y_k}\frac{P(\tilde{X}_1,\ldots,\tilde{X}_d | Y=y_k)P(Y=y_k)}{\sum_{k=1}^K P(\tilde{X}_1,\ldots,\tilde{X}_d | Y=y_k)P(Y=y_k)}\\
&= \text{argmax}_{y_k} P(\tilde{X}_1,\ldots,\tilde{X}_d | Y=y_k)P(Y=y_k)
\end{align}
$$

#### Define the class prior distribution

We will begin by defining the class prior distribution. To do this we will simply take the maximum likelihood estimate, given by

$$
P(Y=y_k) = \frac{\sum_{n=1}^N \delta(Y^{(n)}=y_k)}{N},
$$

where the superscript $(n)$ indicates the $n$-th dataset example, $\delta(Y^{(n)}=y_k) = 1$ if $Y^{(n)}=y_k$ and 0 otherwise, and $N$ is the total number of examples in the dataset. The above is simply the proportion of data examples belonging to class $k$.

You should now write a function that builds the prior distribution from the training data, and returns it as a `Categorical` Distribution object.

* The input to your function `y` will be a numpy array of shape `(num_samples,)`
* The entries in `y` will be integer labels $k=0, 1,\ldots, K-1$
* Your function should build and return the prior distribution as a `Categorical` distribution object
  * The probabilities for this distribution will be a length-$K$ vector, with entries corresponding to $P(Y = y_k)$ for $k=0,1,\ldots,K-1$
  * Your function should work for any value of $K\ge 1$
  * This Distribution will have an empty batch shape and empty event shape

```python
#### GRADED CELL ####

# Complete the following function.
# Make sure to not change the function name or arguments.

def get_prior(y):
    """
    This function takes training labels as a numpy array y of shape (num_samples,) as an input.
    Your function should
    This function should build a Categorical Distribution object with empty batch shape
    and event shape, with the probability of each class given as above.
    Your function should return the Distribution object.
    """
    y = tf.sort(y)
    y_ = np.bincount(y)
    y_ = y_ / len(y)
    print(y_)
    dist = tfd.Categorical(probs=y_) # tf.cast(count, dtype=tf.float16)
    return dist
```

```python
# Run your function to get the prior

prior = get_prior(y_train)
```

    [0.34166667 0.3        0.35833333]

```python
# Plot the prior distribution

labels = ['Iris-Setosa', 'Iris-Versicolour', 'Iris-Virginica']
plt.bar([0, 1, 2], prior.probs.numpy(), color=label_colours)
plt.xlabel("Class")
plt.ylabel("Prior probability")
plt.title("Class prior distribution")
plt.xticks([0, 1, 2], labels)
plt.show()
```

#### Define the class-conditional densities

We now turn to the definition of the class-conditional distributions $P(X_i | Y=y_k)$ for $i=0, 1$ and $k=0, 1, 2$. In our model, we will assume these distributions to be univariate Gaussian:

$$
\begin{align}
P(X_i | Y=y_k) &= N(X_i | \mu_{ik}, \sigma_{ik})\\
&= \frac{1}{\sqrt{2\pi\sigma_{ik}^2}} \exp\left\{-\frac{1}{2} \left(\frac{x - \mu_{ik}}{\sigma_{ik}}\right)^2\right\}
\end{align}
$$
with mean parameters $\mu_{ik}$ and standard deviation parameters $\sigma_{ik}$, twelve parameters in all. We will again estimate these parameters using maximum likelihood. In this case, the estimates are given by

$$
\begin{align}
\hat{\mu}_{ik} &= \frac{\sum_n X_i^{(n)} \delta(Y^{(n)}=y_k)}{\sum_n \delta(Y^{(n)}=y_k)} \\
\hat{\sigma}^2_{ik} &= \frac{\sum_n (X_i^{(n)} - \hat{\mu}_{ik})^2 \delta(Y^{(n)}=y_k)}{\sum_n \delta(Y^{(n)}=y_k)}
\end{align}
$$

Note that the above are just the means and variances of the sample data points for each class.

You should now write a function the computes the class-conditional Gaussian densities, using the maximum likelihood parameter estimates given above, and returns them in a single, batched `MultivariateNormalDiag` Distribution object.

* The inputs to the function are
  * a numpy array `x` of shape `(num_samples, num_features)` for the data inputs
  * a numpy array `y` of shape `(num_samples,)` for the target labels
* Your function should work for any number of classes $K\ge 1$ and any number of features $d\ge 1$

```python
#### GRADED CELL ####

# Complete the following function.
# Make sure to not change the function name or arguments.

def get_class_conditionals(x, y):
    """
    This function takes training data samples x and labels y as inputs.
    This function should build the class-conditional Gaussian distributions above.
    It should construct a batch of distributions for each feature and each class, using the
    parameter estimates above for the means and standard deviations.
    The batch shape of this distribution should be rank 2, where the first dimension corresponds
    to the number of classes and the second corresponds to the number of features.
    Your function should then return the Distribution object.
    """
    idxs = [np.where(y==0), np.where(y==1), np.where(y==2)]
    mu = []
    sigma2 = []
    for idx in idxs:
        mu += [list(np.mean(x[idx], axis=0))]
    for ii, idx in enumerate(idxs):
        sigma2 += [list(np.mean((x[idx] - mu[ii])**2, axis=0))]
    sigma = np.sqrt(sigma2)
    print(mu, sigma)

    print(x.shape, y.shape)
    y = tf.sort(y)
    y_ = np.bincount(y)
    y_ = y_ / len(y)
    print("Categorical distribution:", y_)

    dist = tfd.MultivariateNormalDiag(loc=mu, scale_diag=sigma)
    print(dist)
    return dist
```

```python
# Run your function to get the class-conditional distributions

class_conditionals = get_class_conditionals(x_train, y_train)
```

    [[4.985365853658537, 3.409756097560976], [5.883333333333331, 2.75], [6.6000000000000005, 2.9906976744186045]] [[0.30886178 0.37857556]
     [0.52625511 0.3192874 ]
     [0.6509385  0.32906771]]
    (120, 2) (120,)
    Categorical distribution: [0.34166667 0.3        0.35833333]
    tfp.distributions.MultivariateNormalDiag("MultivariateNormalDiag", batch_shape=[3], event_shape=[2], dtype=float64)

We can visualise the class-conditional densities with contour plots by running the cell below. Notice how the contours of each distribution correspond to a Gaussian distribution with diagonal covariance matrix, since the model assumes that each feature is independent given the class.

```python
# Plot the training data with the class-conditional density contours

def get_meshgrid(x0_range, x1_range, num_points=100):
    x0 = np.linspace(x0_range[0], x0_range[1], num_points)
    x1 = np.linspace(x1_range[0], x1_range[1], num_points)
    return np.meshgrid(x0, x1)

def contour_plot(x0_range, x1_range, prob_fn, batch_shape, colours, levels=None, num_points=100):
    X0, X1 = get_meshgrid(x0_range, x1_range, num_points=num_points)
    Z = prob_fn(np.expand_dims(np.array([X0.ravel(), X1.ravel()]).T, 1))
    Z = np.array(Z).T.reshape(batch_shape, *X0.shape)
    for batch in np.arange(batch_shape):
        if levels:
            plt.contourf(X0, X1, Z[batch], alpha=0.2, colors=colours, levels=levels)
        else:
            plt.contour(X0, X1, Z[batch], colors=colours[batch], alpha=0.3)

plt.figure(figsize=(10, 6))
plot_data(x_train, y_train, labels, label_colours)
x0_min, x0_max = x_train[:, 0].min(), x_train[:, 0].max()
x1_min, x1_max = x_train[:, 1].min(), x_train[:, 1].max()
contour_plot((x0_min, x0_max), (x1_min, x1_max), class_conditionals.prob, 3, label_colours)
plt.title("Training set with class-conditional density contours")
plt.show()
```

#### Make predictions from the model

Now the prior and class-conditional distributions are defined, you can use them to compute the model's class probability predictions for an unknown test input $\tilde{X} = (\tilde{X}_1,\ldots,\tilde{X}_d)$, according to

$$
P(Y=y_k | \tilde{X}_1,\ldots,\tilde{X}_d) = \frac{P(\tilde{X}_1,\ldots,\tilde{X}_d | Y=y_k)P(Y=y_k)}{\sum_{k=1}^K P(\tilde{X}_1,\ldots,\tilde{X}_d | Y=y_k)P(Y=y_k)}
$$

The class prediction can then be taken as the class with the maximum probability:

$$
\hat{Y} = \text{argmax}_{y_k} P(Y=y_k | \tilde{X}_1,\ldots,\tilde{X}_d)
$$

You should now write a function to return the model's class probabilities for a given batch of test inputs of shape `(batch_shape, 2)`, where the `batch_shape` has rank at least one.

* The inputs to the function are the `prior` and `class_conditionals` distributions, and the inputs `x`
* Your function should use these distributions to compute the probabilities for each class $k$ as above
  * As before, your function should work for any number of classes $K\ge 1$
* It should then compute the prediction by taking the class with the highest probability
* The predictions should be returned in a numpy array of shape `(batch_shape)`

```python
#### GRADED CELL ####

# Complete the following function.
# Make sure to not change the function name or arguments.

def predict_class(prior, class_conditionals, x):
    """
    This function takes the prior distribution, class-conditional distribution, and
    a batch of inputs in a numpy array of shape (batch_shape, 2).
    This function should compute the class probabilities for each input in the batch, using
    the prior and class-conditional distributions, according to the above equation.
    Note that the batch_shape of x could have rank higher than one!
    Your function should then return the class predictions by taking the class with the
    maximum probability in a numpy array of shape (batch_shape,).
    """

    y = np.zeros((x.shape[0]), dtype=int)
    for ii, coords in enumerate(x):
        A = tf.cast(class_conditionals.prob(coords), dtype=tf.float32) * tf.cast(prior.probs, dtype=tf.float32)
        B = tf.reduce_sum(A)
        P = tf.math.divide(A, B)
        Y = tf.argmax(P)
        y[ii] = int(Y)
    return y
```

```python
# Get the class predictions

predictions = predict_class(prior, class_conditionals, x_test)
```

```python
# Evaluate the model accuracy on the test set

accuracy = accuracy_score(y_test, predictions)
print("Test accuracy: {:.4f}".format(accuracy))
```

    Test accuracy: 0.7000

```python
# Plot the model's decision regions

plt.figure(figsize=(10, 6))
plot_data(x_train, y_train, labels, label_colours)
x0_min, x0_max = x_train[:, 0].min(), x_train[:, 0].max()
x1_min, x1_max = x_train[:, 1].min(), x_train[:, 1].max()
contour_plot((x0_min, x0_max), (x1_min, x1_max),
             lambda x: predict_class(prior, class_conditionals, x),
             1, label_colours, levels=[-0.5, 0.5, 1.5, 2.5],
             num_points=500)
plt.title("Training set with decision regions")
plt.show()
```

### Binary classifier

We will now draw a connection between the Naive Bayes classifier and logistic regression.

First, we will update our model to be a binary classifier. In particular, the model will output the probability that a given input data sample belongs to the 'Iris-Setosa' class: $P(Y=y_0 | \tilde{X}_1,\ldots,\tilde{X}_d)$. The remaining two classes will be pooled together with the label $y_1$.

```python
# Redefine the dataset to have binary labels

y_train_binary = np.array(y_train)
y_train_binary[np.where(y_train_binary == 2)] = 1

y_test_binary = np.array(y_test)
y_test_binary[np.where(y_test_binary == 2)] = 1
```

```python
# Plot the training data

labels_binary = {0: 'Iris-Setosa', 1: 'Iris-Versicolour / Iris-Virginica'}
label_colours_binary = ['blue', 'red']

plt.figure(figsize=(8, 5))
plot_data(x_train, y_train_binary, labels_binary, label_colours_binary)
plt.show()
```

We will also make an extra modelling assumption that for each class $k$, the class-conditional distribution $P(X_i | Y=y_k)$ for each feature $i=0, 1$, has standard deviation $\sigma_i$, which is the same for each class $k$.

This means there are now six parameters in total: four for the means $\mu_{ik}$ and two for the standard deviations $\sigma_i$ ($i, k=0, 1$).

We will again use maximum likelihood to estimate these parameters. The prior distribution will be as before, with the class prior probabilities given by

$$
P(Y=y_k) = \frac{\sum_{n=1}^N \delta(Y^{(n)}=y_k)}{N},
$$

We will use your previous function `get_prior` to redefine the prior distribution.

```python
# Redefine the prior

prior_binary = get_prior(y_train_binary)
```

    [0.34166667 0.65833333]

```python
# Plot the prior distribution

plt.bar([0, 1], prior_binary.probs.numpy(), color=label_colours_binary)
plt.xlabel("Class")
plt.ylabel("Prior probability")
plt.title("Class prior distribution")
plt.xticks([0, 1], labels_binary)
plt.show()
```

For the class-conditional densities, the maximum likelihood estimate for the means are again given by

$$
\hat{\mu}_{ik} = \frac{\sum_n X_i^{(n)} \delta(Y^{(n)}=y_k)}{\sum_n \delta(Y^{(n)}=y_k)} \\
$$

However, the estimate for the standard deviations $\sigma_i$ is updated. There is also a closed-form solution for the shared standard deviations, but we will instead learn these from the data.

You should now write a function that takes the training inputs and target labels as input, as well as an optimizer object, number of epochs and a TensorFlow Variable. This function should be written according to the following spec:

* The inputs to the function are:
  * a numpy array `x` of shape `(num_samples, num_features)` for the data inputs
  * a numpy array `y` of shape `(num_samples,)` for the target labels
  * a `tf.Variable` object `scales` of length 2 for the standard deviations $\sigma_i$
  * `optimiser`: an optimiser object
  * `epochs`: the number of epochs to run the training for
* The function should first compute the means $\mu_{ik}$ of the class-conditional Gaussians according to the above equation
* Then create a batched multivariate Gaussian distribution object using `MultivariateNormalDiag` with the means set to $\mu_{ik}$ and the scales set to `scales`
* Run a custom training loop for `epochs` number of epochs, in which:
  * the average per-example negative log likelihood for the whole dataset is computed as the loss
  * the gradient of the loss with respect to the `scales` variables is computed
  * the `scales` variables are updated by the `optimiser` object
* At each iteration, save the values of the `scales` variable and the loss
* The function should return a tuple of three objects:
   * a numpy array of shape `(epochs,)` of loss values
   * a numpy array of shape `(epochs, 2)` of values for the `scales` variable at each iteration
   * the final learned batched `MultivariateNormalDiag` distribution object

_NB: ideally, we would like to constrain the `scales` variable to have positive values. We are not doing that here, but in later weeks of the course you will learn how this can be implemented._

```python
#### GRADED CELL ####

# Complete the following function.
# Make sure to not change the function name or arguments.

def learn_stdevs(x, y, scales, optimiser, epochs):
    """
    This function takes the data inputs, targets, scales variable, optimiser and number of
    epochs as inputs.
    This function should set up and run a custom training loop according to the above
    specifications, by setting up the class conditional distributions as a MultivariateNormalDiag
    object, and updating the trainable variables (the scales) in a custom training loop.
    Your function should then return the a tuple of three elements: a numpy array of loss values
    during training, a numpy array of scales variables during training, and the final learned
    MultivariateNormalDiag distribution object.
    """
    def compute_mu(x_train, y_train):
        mu = []
        for value in np.unique(y):
            idx = np.where(y==value)
            mu += [list(np.mean(x[idx], axis=0))]
        return np.array(mu).astype(np.float32)

    @tf.function
    def nll(x_train, y_train):
        log_probs = dist.log_prob(x_train)
        y_train = tf.one_hot(y_train, len(tf.unique(y_train)))
        return -tf.reduce_mean(log_probs * y_train)

    def compute_loss_grads(x_train, y_train, dist):
        with tf.GradientTape() as g:
            g.watch(dist.trainable_variables)
            loss = nll(x_train, y_train)
            grads = g.gradient(loss, dist.trainable_variables)
        return loss, grads

    print(x.shape)
    nlls, scales_arr = [], []
    mu = compute_mu(x, y)
    x = np.expand_dims(x, axis=1).astype(np.float32)
    dist = tfd.MultivariateNormalDiag(loc=mu, scale_diag=scales)
    for epoch in range(epochs):
        loss, grads = compute_loss_grads(x, y, dist)
        optimiser.apply_gradients(zip(grads, dist.trainable_variables))
        print(f"Epoch {epoch};\tLoss {loss}", end='\r')
        nlls += [loss]
        scales_arr += [scales.numpy()]
    nlls, scales_arr = np.array(nlls), np.array(scales_arr)
    return nlls, scales_arr, dist
```

```python
# Define the inputs to your function

scales = tf.Variable([1., 1.])
opt = tf.keras.optimizers.Adam(learning_rate=0.01)
epochs = 500
```

```python
# Run your function to learn the class-conditional standard deviations

nlls, scales_arr, class_conditionals_binary = learn_stdevs(x_train, y_train_binary, scales, opt, epochs)
```

    (120, 2)

```python
# View the distribution parameters

print("Class conditional means:")
print(class_conditionals_binary.loc.numpy())
print("\nClass conditional standard deviations:")
print(class_conditionals_binary.stddev().numpy())
```

    Class conditional means:
    [[4.985366  3.4097562]
     [6.273418  2.8810127]]

    Class conditional standard deviations:
    [[0.59277165 0.35751012]
     [0.59277165 0.35751012]]

```python
# Plot the loss and convergence of the standard deviation parameters

fig, ax = plt.subplots(1, 2, figsize=(14, 5))
ax[0].plot(nlls)
ax[0].set_title("Loss vs epoch")
ax[0].set_xlabel("Epoch")
ax[0].set_ylabel("Average negative log-likelihood")
for k in [0, 1]:
    ax[1].plot(scales_arr[:, k], color=label_colours_binary[k], label=labels_binary[k])
ax[1].set_title("Standard deviation ML estimates vs epoch")
ax[1].set_xlabel("Epoch")
ax[1].set_ylabel("Standard deviation")
plt.legend()
plt.show()
```

We can also plot the contours of the class-conditional Gaussian distributions as before, this time with just binary labelled data. Notice the contours are the same for each class, just with a different centre location.

```python
# Plot the training data with the class-conditional density contours

plt.figure(figsize=(10, 6))
plot_data(x_train, y_train_binary, labels_binary, label_colours_binary)
x0_min, x0_max = x_train[:, 0].min(), x_train[:, 0].max()
x1_min, x1_max = x_train[:, 1].min(), x_train[:, 1].max()
contour_plot((x0_min, x0_max), (x1_min, x1_max), class_conditionals_binary.prob, 2, label_colours_binary)
plt.title("Training set with class-conditional density contours")
plt.show()
```

We can also plot the decision regions for this binary classifier model, notice that the decision boundary is now linear.

```python
# Plot the model's decision regions

plt.figure(figsize=(10, 6))
plot_data(x_train, y_train_binary, labels_binary, label_colours_binary)
x0_min, x0_max = x_train[:, 0].min(), x_train[:, 0].max()
x1_min, x1_max = x_train[:, 1].min(), x_train[:, 1].max()
contour_plot((x0_min, x0_max), (x1_min, x1_max),
             lambda x: predict_class(prior_binary, class_conditionals_binary, x),
             1, label_colours_binary, levels=[-0.5, 0.5, 1.5],
             num_points=500)
plt.title("Training set with decision regions")
plt.show()
```

#### Link to logistic regression

In fact, we can see that our predictive distribution $P(Y=y_0 | X)$ can be written as follows:

$$
\begin{align}
P(Y=y_0 | X) =& ~\frac{P(X | Y=y_0)P(Y=y_0)}{P(X | Y=y_0)P(Y=y_0) + P(X | Y=y_1)P(Y=y_1)}\\
=&  ~\frac{1}{1 + \frac{P(X | Y=y_1)P(Y=y_1)}{P(X | Y=y_0)P(Y=y_0)}}\\
=& ~\sigma(a)
\end{align}
$$

where $\sigma(a) = \frac{1}{1 + e^{-a}}$ is the sigmoid function, and $a = \log\frac{P(X | Y=y_0)P(Y=y_0)}{P(X | Y=y_1)P(Y=y_1)}$ is the _log-odds_.

With our additional modelling assumption of a shared covariance matrix $\Sigma$, it can be shown (using the Gaussian pdf) that $a$ is in fact a linear function of $X$:

$$
a = w^T X + w_0
$$

where

$$
\begin{align}
w =& ~\Sigma^{-1} (\mu_0 - \mu_1)\\
w_0 =& -\frac{1}{2}\mu_0^T \Sigma^{-1}\mu_0 + \frac{1}{2}\mu_1^T\Sigma^{-1}\mu_1 + \log\frac{P(Y=y_0)}{P(Y=y_1)}
\end{align}
$$

The model therefore takes the form $P(Y=y_0 | X) = \sigma(w^T X + w_0)$, with weights $w\in\mathbb{R}^2$ and bias $w_0\in\mathbb{R}$. This is the form used by logistic regression, and explains why the decision boundary above is linear.

In the above we have outlined the derivation of the generative logistic regression model. The parameters are typically estimated with maximum likelihood, as we have done.

Finally, we will use the above equations to directly parameterise the output Bernoulli distribution of the generative logistic regression model.

You should now write the following function, according to the following specification:

* The inputs to the function are:
  * the prior distribution `prior` over the two classes
  * the (batched) class-conditional distribution `class_conditionals`
* The function should use the parameters of the above distributions to compute the weights and bias terms $w$ and $w_0$ as above
* The function should then return a tuple of two numpy arrays for $w$ and $w_0$

```python
#### GRADED CELL ####

# Complete the following function.
# Make sure to not change the function name or arguments.

def get_logistic_regression_params(prior, class_conditionals):
    """
    This function takes the prior distribution and class-conditional distribution as inputs.
    This function should compute the weights and bias terms of the generative logistic
    regression model as above, and return them in a 2-tuple of numpy arrays of shapes
    (2,) and () respectively.
    """
    cov = class_conditionals.covariance().numpy()
    cov_inv = np.linalg.inv(cov)
    print(cov)
    print(cov_inv, end='\n\n')

    mu = class_conditionals.loc.numpy()
    print(mu, end='\n\n')

    log_p = np.log(prior.prob(0) / prior.prob(1))
    print(log_p, end='\n\n')

    w = cov_inv @ (mu[0] - mu[1])
    print(w, end='\n\n')

    w1 = -0.5 * (mu[0].T @ cov_inv @ mu[0]) + 0.5 * (mu[1].T @ cov_inv @ mu[1]) + log_p
    print(w1)

    return w, w1
```

```python
# Run your function to get the logistic regression parameters

w, w0 = get_logistic_regression_params(prior_binary, class_conditionals_binary)
```

We can now use these parameters to make a contour plot to display the predictive distribution of our logistic regression model.

```python
# Plot the training data with the logistic regression prediction contours

fig, ax = plt.subplots(1, 1, figsize=(10, 6))
plot_data(x_train, y_train_binary, labels_binary, label_colours_binary)
x0_min, x0_max = x_train[:, 0].min(), x_train[:, 0].max()
x1_min, x1_max = x_train[:, 1].min(), x_train[:, 1].max()
X0, X1 = get_meshgrid((x0_min, x0_max), (x1_min, x1_max))

logits = np.dot(np.array([X0.ravel(), X1.ravel()]).T, w) + w0
Z = tf.math.sigmoid(logits)
lr_contour = ax.contour(X0, X1, np.array(Z).T.reshape(*X0.shape), levels=10)
ax.clabel(lr_contour, inline=True, fontsize=10)
contour_plot((x0_min, x0_max), (x1_min, x1_max),
             lambda x: predict_class(prior_binary, class_conditionals_binary, x),
             1, label_colours_binary, levels=[-0.5, 0.5, 1.5],
             num_points=300)
plt.title("Training set with prediction contours")
plt.show()
```

Congratulations on completing this programming assignment! In the next week of the course we will look at Bayesian neural networks and uncertainty quantification.


# Simulation of Ball drop and Spring mass damper system
> Simulation of dynamic systems for dummies. This is a very simple description of how to do time simulations of a dynamic system using SciPy ODE (Ordinary Differnetial Equation) Solver.
- toc: true
- branch: master
- badges: true
- comments: true
- categories: [machine learning, regression]
- image: images/for_dummies.jpg
- hide: false
- search_exclude: true
- metadata_key1: metadata_value1
- metadata_key2: metadata_value2

```python
#hide
import warnings
warnings.filterwarnings("ignore")
```

```python
#collapse
from scipy.integrate import odeint
import numpy as np
import matplotlib.pyplot as plt
```

## Simulation of a static system to introduce ODEint

Define a method that takes a system state and describe how this state will change in time. The method does this by returning time derivatives for each state. The ODE solver will use these time derivatives to calculate new states, for the next time step.

Here is a method that takes a system to simulate a train that travels with constant speed:

(The system has only one state, the position of the train)

```python
V_start = 150*10**3/3600  # [m/s] Train velocity at start

def train(states,t):

    # states:
    # [x]

    x = states[0]  # Position of train
    dxdt = V_start # The position state will change by the speed of the train

    # Time derivative of the states:
    d_states_dt = np.array([dxdt])

    return d_states_dt
```

```python
x_start = 0  # [m] Train position at start

# The states at start of the simulation, the train is traveling with constant speed V at position x = 0.
states_0 = np.array([x_start])

# Create a time vector for the simulation:
t = np.linspace(0,10,100)

# Simulate with the "train" method and start states for the times in t:
states = odeint(func = train,y0 = states_0,t = t)

# The result is the time series of the states:
x = states[:,0]

```

```python
fig,ax = plt.subplots()
ax.plot(t,x,label = 'Train position')

ax.set_title('Train traveling at constant speed')
ax.set_xlabel('time [s]')
ax.set_ylabel('x [m]')
a = ax.legend()

```

The speed can hower be a state too:

```python
def train_2_states(states,t):

    # states:
    # [x,V]

    x = states[0]  # Position of train
    V = states[1]  # Speed of train
    dxdt = V  # The position state will change by the speed of the train
    dVdt = 0  # The velocity will not change (No acceleration)

    # Time derivative of the states:
    d_states_dt = np.array([dxdt,dVdt])

    return d_states_dt
```

```python
# The states at start of the simulation, the train is traveling with constant speed V at position x = 0.
states_0 = np.array([x_start,V_start])

# Create a time vector for the simulation:
t = np.linspace(0,10,100)

# Simulate with the "train" method and start states for the times in t:
states = odeint(func = train_2_states,y0 = states_0,t = t)

# The result is the time series of the states:
x = states[:,0]
dxdt = states[:,1]

```

```python
fig,axes = plt.subplots(ncols = 2)
fig.set_size_inches(11,5)
ax = axes[0]
ax.plot(t,x,label = 'Train position')

ax.set_title('Train traveling at constant speed')
ax.set_xlabel('time [s]')
ax.set_ylabel('x [m]')
a = ax.legend()

ax = axes[1]
ax.plot(t,dxdt,label = 'Train speed')

ax.set_title('Train traveling at constant speed')
ax.set_xlabel('time [s]')
ax.set_ylabel('dx/dt [m/s]')
a = ax.legend()

```

## Ball drop
Here is a system where the speed is not constant.
A simulation of a ball drop under the influence of gravity force.

```python
g = 9.81
m = 1

def ball_drop(states,t):

    # states:
    # [x,v]
    # F = g*m = m*dv/dt
    # --> dv/dt = (g*m) / m

    x = states[0]
    dxdt = states[1]

    dvdt = (g*m) / m

    d_states_dt = np.array([dxdt,dvdt])

    return d_states_dt
```

```python
states_0 = np.array([0,0])
t = np.linspace(0,10,100)
states = odeint(func = ball_drop,y0 = states_0,t = t)
x = states[:,0]
dxdt = states[:,1]

fig,axes = plt.subplots(ncols = 2)
fig.set_size_inches(11,5)
ax = axes[0]
ax.plot(t,x,label = 'Ball position')

ax.set_title('Ball drop')
ax.set_xlabel('time [s]')
ax.set_ylabel('x [m]')
a = ax.legend()

ax = axes[1]
ax.plot(t,dxdt,label = 'Ball speed')

ax.set_title('Ball drop')
ax.set_xlabel('time [s]')
ax.set_ylabel('dx/dt [m/s]')
a = ax.legend()
```

Simulating in air, where the ball has a resistance due aerodynamic drag.

```python
cd = 0.01

def ball_drop_air(states,t):

    # states:
    # [x,u]
    # F = g*m - cd*u = m*du/dt
    # --> du/dt = (g*m - cd*u**2) / m

    x = states[0]
    u = states[1]
    dxdt = u

    dudt = (g*m - cd*u**2) / m

    d_states_dt = np.array([dxdt,dudt])

    return d_states_dt
```

```python
states = odeint(func = ball_drop_air,y0 = states_0,t = t)
x_air = states[:,0]
dxdt_air = states[:,1]

fig,axes = plt.subplots(ncols = 2)
fig.set_size_inches(11,5)
ax = axes[0]
ax.plot(t,x,label = 'Vacuum')
ax.plot(t,x_air,label = 'Air')

ax.set_title('Ball drop in vacuum and air')
ax.set_xlabel('time [s]')
ax.set_ylabel('x [m]')
a = ax.legend()

ax = axes[1]
ax.plot(t,dxdt,label = 'Vacuum')
ax.plot(t,dxdt_air,label = 'Air')

ax.set_title('Ball drop in vacuum and air')
ax.set_xlabel('time [s]')
ax.set_ylabel('dx/dt [m/s]')
a = ax.legend()

```

The very classical dynamic system with a spring, a mass and a damper.

```python
k = 3  # The stiffnes of the spring (relates to position)
c = 0.1  # Damping term (relates to velocity)
m = 0.1  # The mass (relates to acceleration)

def spring_mass_damp(states,t):

    # states:
    # [x,v]
    # F = -k*x -c*v = m*dv/dt
    # --> dv/dt = (-kx -c*v) / m

    x = states[0]
    dxdt = states[1]

    dvdt = (-k*x -c*dxdt) / m

    d_states_dt = np.array([dxdt,dvdt])

    return d_states_dt
```

```python
y0 = np.array([1,0])
t = np.linspace(0,10,100)
states = odeint(func = spring_mass_damp,y0 = y0,t = t)
x = states[:,0]
dxdt = states[:,1]
```

```python
fig,ax = plt.subplots()
ax.plot(t,x)
ax.set_title('Spring mass damper simulation')
ax.set_xlabel('time [s]')
a = ax.set_ylabel('x [m]')

```

Also add a gravity force

```python
g = 9.81

def spring_mass_damp_g(states,t):

    # states:
    # [x,v]
    # F = g*m -k*x -c*v = m*dv/dt
    # --> dv/dt = (g*m -kx -c*v) / m

    x = states[0]
    dxdt = states[1]

    dvdt = (g*m -k*x -c*dxdt) / m

    d_states_dt = np.array([dxdt,dvdt])

    return d_states_dt
```

```python
states_g = odeint(func = spring_mass_damp_g,y0 = y0,t = t)
x_g = states_g[:,0]
dxdt_g = states_g[:,1]
```

```python
fig,ax = plt.subplots()
ax.plot(t,x,label = 'No gravity force')
ax.plot(t,x_g,label = 'Gravity force')

ax.set_title('Spring mass damper simulation with and without gravity')
ax.set_xlabel('time [s]')
ax.set_ylabel('x [m]')
a = ax.legend()
```

## SymPy solution
(To be continued...)

```python
#collapse
import sympy as sym
import sympy.physics.mechanics as me
from sympy.physics.vector import init_vprinting
init_vprinting(use_latex='mathjax')
x, v = me.dynamicsymbols('x v')
m, c, k, g, t = sym.symbols('m c k g t')
ceiling  = me.ReferenceFrame('C')
O = me.Point('O')
P = me.Point('P')
O.set_vel(ceiling, 0)
P.set_pos(O, x * ceiling.x)
P.set_vel(ceiling, v * ceiling.x)
P.vel(ceiling)

damping = -c * P.vel(ceiling)
stiffness = -k * P.pos_from(O)
gravity = m * g * ceiling.x
forces = damping + stiffness + gravity
```


# Solution {-}

A stationary Gaussian process $X(t)$ has power spectral density function:

\begin{equation*}
  S_X(j\omega)=\frac{2}{\omega^4 + 1}
\end{equation*}

1. Find $E(X)$ and $E(X^2)$

```python
from numpy import linspace
import matplotlib.pyplot as plt

omega = linspace(-5, 5, 100)
Sx = 2/(omega**4 + 1)

plt.plot(omega, Sx)
plt.title("Spectral density")
plt.xlabel("Frequency")
plt.ylabel("Power")
plt.grid()
plt.show()
```

```python
from sympy import symbols, integrate, pi, oo

omega = symbols('omega')

Sx = 2/(omega**4 + 1)

EX = 0 # No impulse function at the origin of Sx
display(EX)

EX2 = 1/(2*pi)*integrate(Sx, (omega, -oo, oo))
display(EX2)
```

    0

$\displaystyle \frac{\sqrt{2}}{2}$


# Plotting a Torus

To plot a surface in 3D do you first need to find the
parametrization of the surface you wish to plot.

For a Torus the parametrization looks like this:
\begin{align}
x &= [c + a  \cos v] \cos u \\
y &= [c + a  \cos v] \sin u \\
z &= a \sin v
\end{align}

where $a$ represents the radius of the tube and $c$ is the radius of the
center hole of the torus tube. See https://mathworld.wolfram.com/Torus.html for further information.

Begin by activating matplotlib notebook and importing numpy and
matplotlib

```python
%matplotlib notebook
# This enables you to drag and rotate the figure

from mpl_toolkits.mplot3d import Axes3D # needed to do the 3D Plots
import numpy as np
import matplotlib.pyplot as plt
```

```python
fig = plt.figure()
ax = fig.gca(projection='3d')

u = np.linspace(0, 2*np.pi, 100)
v = np.linspace(0, 2*np.pi, 100)
u, v = np.meshgrid(u, v) # Make coordinate matrices from coordinate vectors.
a, c = 0.2, 1.0 # Set a and c from the torus parametrization

# Paramatrization
x = (c + a*np.cos(v))*np.cos(u)
y = (c + a*np.cos(v))*np.sin(u)
z = a*np.sin(v)

ax.plot_surface(x, y, z)

ax.set_xlim3d(-1, 1)
ax.set_ylim3d(-1, 1)
ax.set_zlim3d(-1, 1)
plt.show()
```

    <IPython.core.display.Javascript object>

# Make a surface plot of a sphere

Plot a sphere the parametrization is:

\begin{align}
x &= r \cos \theta \sin \phi  \\
y &= r \sin \theta \sin \phi  \\
z &= r \cos \phi
\end{align}

where $\theta$ is the loggitude coorinate running from $0$ to $2\pi$,
$\phi$ is the colatitude coordinate running from $0$ to $\pi$ and $r$ is the radius.

# Make a surface plot of a Mobius strip

The parametrization is:
\begin{align}
x &= \left(1 + \frac{v}{2}\cos \frac{u}{2} \right)\cos u  \\
y &= \left(1 + \frac{v}{2}\cos v \right)\sin u  \\
z &= \frac{v}{2}\sin \frac{u}{2}
\end{align}

where $u$ is running from $0$ to $2\pi$ and $v$ is running from $-1$ to $1$.


# We implement a Kalman Filter on a double integrator $\ddot{x} = a + v$ with GPS measurements $z = x + w$
We follow https://www.intechopen.com/books/introduction-and-implementations-of-the-kalman-filter/introduction-to-kalman-filter-and-its-applications section 2

This is a good paper for learning the theory and implementation, but honestly I find they chose very poor parameters which make the estimation problem trivial anyways...

\begin{align}
x_{k+1} = \pmatrix{ p_{k+1} \\ v_{k+1}}
&= \pmatrix{ p_k + v_k \Delta t + \frac{1}{2} a_k \Delta t^2 \\ v_k + a_k \Delta t} \\
&= \pmatrix{ I_{3 \times 3} & I_{3 \times 3} \Delta t \\ 0 & I_{3 \times 3}} \pmatrix{ p_k \\ v_k} +
   \pmatrix{ \frac{1}{2} I_{3 \times 3} \Delta t^2 \\  I_{3 \times 3} \Delta t} a_k \\
&= Ax_k + Ba_k
\end{align}

```python
import numpy as np
import matplotlib.pyplot as plt
from dymoesco.estimation.filters import KalmanFilter
from dymoesco.dynamics.doubleintegrator import DoubleIntegrator
np.set_printoptions(suppress=True, precision=3)
```

```python
# PARAMS
a_std = 0.3
p_std = 3
v_std = 0.03

x0 = np.array([2,-2,0,5,5.1,0.1])
P0 = np.block([[np.eye(3) * 4**2, np.zeros((3,3))],
               [np.zeros((3,3)), np.eye(3) * 0.4**2]])
dt = 1
N = 20
dims = ["$p_x$", "$p_y$", "$p_z$", "$v_x$", "$v_y$", "$v_z$"]
```

```python
# Building the Kalman Filter
Q = np.block([[1/4 * dt**4 * np.eye(3), 1/2 * dt**3 * np.eye(3)],
              [1/2 * dt**3 * np.eye(3), dt**2 * np.eye(3)]])
R = a_std**2 *np.block([[np.eye(3) * p_std**2, np.zeros((3,3))],
                        [np.zeros((3,3)), np.eye(3) * v_std**2]])
A = np.block([[np.eye(3), np.eye(3)*dt],
              [np.zeros((3,3)), np.eye(3)]])
B = np.block([[1/2 * np.eye(3) * dt**2],
              [np.eye(3) * dt]])
C = np.eye(6)
kf = KalmanFilter(A,B,C,Q,R)
```

```python
# Generate ground truth trajectory
di = DoubleIntegrator(dim=3)
def u(t):
    return np.random.normal(0, a_std, 3)

x0 = [0,0,0,5,5,0]
t_span = (0,N)
sol = di.simulate(u, t_span, x0, t_eval=np.arange(0,N,dt))
```

```python
# KF prediction only
preds = np.zeros((N,6))
preds[0] = x0
x = x0; P = P0;
for i in range(1,20):
    x, P = kf.predict(x, P, np.zeros(3))
    preds[i] = x
```

```python
# Generate noisy observations
x = sol.y
p_obs_noise = np.random.normal(0, p_std, (3,N))
v_obs_noise = np.random.normal(0, v_std, (3,N))
obs_noise = np.vstack([p_obs_noise, v_obs_noise])
obs = x + obs_noise
```

```python
# KF with pred + update
filtered = np.zeros((N,6))
filtered[0] = x0
x = x0; P = P0;
for i in range(1,20):
    x, P = kf.predict(x, P, np.zeros(3))
    x, P = kf.update(x, P, obs.T[i])
    filtered[i] = x
```

```python
from ipywidgets import interact
@interact(ground_truth=True, observation=True, pred=False, pred_and_update=False)
def f(ground_truth, observation, pred, pred_and_update):
    fig, axes = plt.subplots(1,2)
    for i, ax in enumerate(axes):
        ax.set_title(dims[i])
        if ground_truth:
            ax.plot(sol.y[i], label="ground_truth")
        if observation:
            ax.scatter(range(20), obs[i], label="observations")
        if pred:
            ax.plot(preds.T[i], label="pred only")
        if pred_and_update:
            ax.plot(filtered.T[i], label="pred+update")
        ax.legend()
```

    interactive(children=(Checkbox(value=True, description='ground_truth'), Checkbox(value=True, description='obse…

```python
# This is figure 1 from the paper
err = filtered.T - sol.y
fig, axes = plt.subplots(6,1)
fig.suptitle("Error per dimension")
for i,ax in enumerate(axes):
    ax.plot(err[i])
    ax.set_ylabel(dims[i])
```

```python

```


# Exploration of wireless network RF calculations

Free space loss equation: http://www.tapr.org/ve3jf.dcc97.html
Assuming isotropic receiving antenna

$$
L_p = 32.4 + 20 log \,f + 20 log\,d\,dB\\
f\,in\,MHz,\,d\,in\,km
$$

Now lets combine this with the antenna gains and cable losses

\begin{align}
P_r & = P_t - L_p + G_t + G_r - L_t - L_r\\
\\
where &\\
\\
P_t & = transmitter\,power\,output\,(dBm\,or\,dBW,\,same\,units\,as\,P_r)\\
...&\\
\end{align}

Not taking into account refraction, defraction and reflection we can compute the loss. From [here](https://www.signalboosters.com/blog/what-is-decibel-db-gain-and-how-does-it-relate-to-cell-phone-reception) we can see that mobile phones operate in ideal conditions at - 50 dB and stop operating at - 100 dB. This post also shows you how to check your network readings from you iPhone/Android.

```python
import math
```

```python
L_p = 32.4 + (20 * math.log10(915)) + (20 * math.log10(10))
P_t = 24
G_t = 25
G_r = 12
L_t = 2
L_r = 2
P_r = P_t - L_p + G_t + G_r - L_t - L_r
P_r
```

    -54.62842188132896

### Now lets explore the loss as we increase the frequency upto millimeter wave (mmWave)

```python
def calc_loss(freq, distance, tr_pwr=24, tr_gain=10, rec_gain=10, tr_loss=2, rec_loss=2):
    """ Calculate the path loss for a frequency and distance

    Keyword arguments:
    freq -- the frequnecy in MHz
    distance -- the distance in km
    tr_pwr -- the transmitter power in dB
    tr_gain -- the transmitter gain in dB
    rec_gain -- the receiver gain in dB
    tr_loss - the transmitter loss in dB
    rec_loss - the receiver loss in dB
    """
    free_space_loss = 32.4 + (20 * math.log10(freq)) + (20 * math.log10(distance))
    path_loss = tr_pwr - free_space_loss + tr_gain  + rec_gain - tr_loss - rec_loss
    return path_loss
```

```python
import numpy as np
```

```python
losses = []
freq = np.arange(500, 10000, step=500)
dist = np.arange(0.00001, 20, step=0.2)
for x in freq:
    row = []
    for y in dist:
        row.append(calc_loss(x, y))
    losses.append(row)

```

```python
l_a = np.array(losses)
```

```python
for i in range(len(l_a)):
    print(l_a[i].mean())
```

    -62.73381721669774
    -68.75441712997736
    -72.27624231109098
    -74.77501704325698
    -76.7132173034181
    -78.29684222437061
    -79.63577801698287
    -80.79561695653662
    -81.8186674054842
    -82.73381721669773
    -83.56167091986222
    -84.31744213765019
    -85.01268426283447
    -85.6563779302625
    -86.25564239781136
    -86.81621686981622
    -87.34279564426325
    -87.83926731876383
    -88.3088892357543

```python
% matplotlib inline
import matplotlib.pyplot as plt

top = plt.plot(dist, l_a[0], 'r', label='%i MHz'% freq[0])
mid = plt.plot(dist, l_a[4], 'b', label='%i MHz'% freq[4])
bottom = plt.plot(dist, l_a[-1], 'g', label='%i MHz'% freq[-1])
plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
plt.title("Propagation of RF frequnecy for distance (km)")
plt.ylabel('dB loss')
plt.xlabel('km')
plt.show()
```

Now we want to check the distance at which each frequency hits -50 and -75 dB loss

```python
def calc_loss_at_distance(loss, distance):

    hits = []

    for c in l_a:
        for i, l in enumerate(c):
            if l <= loss:

                hits.append(distance[i])
                break
    return hits

hits_fifty = calc_loss_at_distance(-50, dist)
hits_seventy = calc_loss_at_distance(-70, dist)

```

```python
neg_fifty = plt.plot(freq, hits_fifty, 'r', label='distance to hit loss -50 dB')
neg_seventy = plt.plot(freq, hits_seventy, 'b', label='distance to hit loss -70 dB')
plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
plt.title("dB loss over distance as frequency increases with 400m baseline")
plt.ylabel('distance (km)')
plt.xlabel('MHz')
plt.axhline(0.4, linestyle='--', color='k') # horizontal lines
plt.show()
```


```python
import numpy as np
from sympy import *
init_printing(use_latex='mathjax')
```

```python
z1, w1, a0, b1, a1, y, sigma = symbols('z1 w1 a0 b1 a1 y sigma')
a1f = sigma(z1)
z1f = w1 * a0 + b1
ck = (a1 - y) ** 2
```

```python
diff(ck, a1)
```

$$2 a_{1} - 2 y$$

```python
diff(z1f, w1)
```

$$a_{0}$$

```python
diff(z1f, b1)
```

$$1$$

```python
diff(a1f, z1)
```

$$\frac{d}{d z_{1}} \sigma{\left (z_{1} \right )}$$

```python

```

Certification 2/Week3.2 - Derivative for Neural Network.ipynb


Python for Everyone!<br/>[Oregon Curriculum Network](http://4dsolutions.net/ocn/)

## Extended Precision with the Native Decimal Type
### With LaTeX and Generator Functions

The Python Standard Library provides a *decimal* module containing the class Decimal.  A Decimal object behaves according to the base 10 algorithms we learn in school.

The precision i.e. number of decimal places to which computations are carried out, is set globally, or within the scope of a context manager.

Note also that Jupyter Notebooks are able to render LaTeX when commanded to do so with the %%latex magic command.  As a first example, here is an expression for the mathematical constant *e*.

```latex
%%latex
\begin{align}
e = lim_{n \to \infty} (1 + 1/n)^n
\end{align}
```

\begin{align}
e = lim_{n \to \infty} (1 + 1/n)^n
\end{align}

```python
from math import e, pi
print(e)  # as a floating point number
print(pi)
```

    2.718281828459045
    3.141592653589793

Lets show setting precision to a thousand places within a scope defined by decimal.localcontext.  We set precision internally to the scope.  By default, precision is to 28 places.  We set n to 1 followed by 102 zeros, so a very large number.  The resulting computation matches a published value for *e* to 100 decimal places.

```python
import decimal

with decimal.localcontext() as ctx:  # context manager
    ctx.prec = 1000
    n = decimal.Decimal(1e102)
    e = (1 + 1/n) ** n
```

```python
e_1000_places = """2.7182818284590452353602874713526624977572470936999595749669
6762772407663035354759457138217852516642742746639193200305992181741359662904357
2900334295260595630738132328627943490763233829880753195251019011573834187930702
1540891499348841675092447614606680822648001684774118537423454424371075390777449
9206955170276183860626133138458300075204493382656029760673711320070932870912744
3747047230696977209310141692836819025515108657463772111252389784425056953696770
7854499699679468644549059879316368892300987931277361782154249992295763514822082
6989519366803318252886939849646510582093923982948879332036250944311730123819706
8416140397019837679320683282376464804295311802328782509819455815301756717361332
0698112509961818815930416903515988885193458072738667385894228792284998920868058
2574927961048419844436346324496848756023362482704197862320900216099023530436994
1849146314093431738143640546253152096183690888707016768396424378140592714563549
0613031072085103837505101157477041718986106873969655212671546889570350354"""
e_1000_places = e_1000_places.replace("\n",str())
```

```python
str(e)[2:103] == e_1000_places[2:103]  # skipping "2." and going to 100 decimals
```

    True

In the context below, the value of n starts at 10 and then gets two more zeros every time around the loop.

The *yield* keyword is similar to *return* in handing back an object, however a generator function then pauses to pick up where it left off when nudged by next(), (which triggers \_\_next\_\_ internally).

Generator functions do not forget their internal state as they advance through next values.

Note that when a Decimal type object operates with an integer, that integer is coerced (cast) as a Decimal object.

```python
with decimal.localcontext() as ctx:  # context manager
    ctx.prec = 1000
    def converge():  # generator function
        n = decimal.Decimal('10')
        while True:
            yield (1 + 1/n) ** n
            n = n * 100 # two more zeros
    f = converge()
    for _ in range(9):
        next(f) # f.__next__() <--- not quite like Python 2.x (f.next())
```

```python
r = next(f)
r
```

    Decimal('2.718281828459045235224373380')

```python
str(r)[:20] == e_1000_places[:20]
```

    True

The fancier LaTeX below renders a famous equation by Ramanujan, which has been shown to converge to 1/π and therefore π very quickly, relative to many other algorithms.  I don't think anyone understands how some random guy could think up such a miraculaous equation.

```latex
%%latex
\begin{align}
\frac{1}{\pi} = \frac{2\sqrt{2}}{9801} \sum^\infty_{k=0} \frac{(4k)!(1103+26390k)}{(k!)^4 396^{4k}}
\end{align}
```

\begin{align}
\frac{1}{\pi} = \frac{2\sqrt{2}}{9801} \sum^\infty_{k=0} \frac{(4k)!(1103+26390k)}{(k!)^4 396^{4k}}
\end{align}

Lets break the above equation into three components, a constant A and a product of Bn and Cn that accumulate as a sum.  The latter two may be written as generators, with n increasing by one each time they're fed to next(). An outermost generator ties them all together and takes the reciprocal to get an approximation for π itself.

```latex
%%latex
\begin{align*}
\frac{1}{\pi} = \frac{2\sqrt{2}}{9801}\sum_{n = 0}^{\infty}\frac{(4n)!(1103 + 26390n)}{4^{4n}(n!)^{4}99^{4n}} = A\sum_{n = 0}^{\infty}B_{n}C_{n},\\
A = \frac{2\sqrt{2}}{9801},\,\\
B_{n} = \frac{(4n)!(1103 + 26390n)}{4^{4n}(n!)^{4}},\,\\
C_{n} = \frac{1}{99^{4n}}
\end{align*}
```

\begin{align*}
\frac{1}{\pi} = \frac{2\sqrt{2}}{9801}\sum_{n = 0}^{\infty}\frac{(4n)!(1103 + 26390n)}{4^{4n}(n!)^{4}99^{4n}} = A\sum_{n = 0}^{\infty}B_{n}C_{n},\\
A = \frac{2\sqrt{2}}{9801},\,\\
B_{n} = \frac{(4n)!(1103 + 26390n)}{4^{4n}(n!)^{4}},\,\\
C_{n} = \frac{1}{99^{4n}}
\end{align*}

```python
from math import factorial
from decimal import Decimal as D
decimal.getcontext().prec=100
A = (2 * D('2').sqrt()) / 9801
A
```

    Decimal('0.0002885855652225477091728780173879600201142070962915923014338699597981292681281720311907739076273118198')

```python
def B():
    n = 0
    while True:
        numerator = factorial(4 * n) * (D(1103) + 26390 * n)
        denominator = (4 ** (4*n))*(factorial(n))**4
        yield numerator / denominator
        n += 1

def C():
    n = 0
    while True:
        yield 1 / (D('99')**(4*n))
        n += 1

def Pi():
    Bn = B()
    Cn = C()
    the_sum = 0
    while True:
        the_sum += next(Bn) * next(Cn)
        yield 1/(A * the_sum)
```

```python
pi = Pi()
```

```python
next(pi)
next(pi)
next(pi)
```

    Decimal('3.141592653589793238462649065702758898156677480462334781168399595644739794558841580205059234965983146')

```python
pi_1000_places = """3.1415926535 8979323846 2643383279 5028841971 6939937510 5820974944
5923078164 0628620899 8628034825 3421170679 8214808651 3282306647 0938446095 5058223172
5359408128 4811174502 8410270193 8521105559 6446229489 5493038196 4428810975 6659334461
2847564823 3786783165 2712019091 4564856692 3460348610 4543266482 1339360726 0249141273
7245870066 0631558817 4881520920 9628292540 9171536436 7892590360 0113305305 4882046652
1384146951 9415116094 3305727036 5759591953 0921861173 8193261179 3105118548 0744623799
6274956735 1885752724 8912279381 8301194912 9833673362 4406566430 8602139494 6395224737
1907021798 6094370277 0539217176 2931767523 8467481846 7669405132 0005681271 4526356082
7785771342 7577896091 7363717872 1468440901 2249534301 4654958537 1050792279 6892589235
4201995611 2129021960 8640344181 5981362977 4771309960 5187072113 4999999837 2978049951
0597317328 1609631859 5024459455 3469083026 4252230825 3344685035 2619311881 7101000313
7838752886 5875332083 8142061717 7669147303 5982534904 2875546873 1159562863 8823537875
9375195778 1857780532 1712268066 1300192787 6611195909 2164201989"""
pi_1000_places = pi_1000_places.replace(" ","").replace("\n","")
```

```python
r = next(pi)
```

```python
str(r)[:20] == pi_1000_places[:20]
```

    True

*Resource*:

[Ballad about Ramanujan](https://archive.org/details/Ramanujan) by Mark Engelberg, 2005.

This I-Python Notebook is by Kirby Urner, copyleft MIT License, October 2016.


# Sentiment Analysis of Tweets in Russian using Multinomial Naive Bayes
Multinomial Naive Bayes classification algorithm tends to be a baseline solution for sentiment analysis task. The basic idea of Naïve Bayes technique is to find the probabilities of classes assigned to texts by using the joint probabilities of words and classes. Given the dependent feature vector $(x_1, \dots, x_n)$ and the class $C_k$. Bayes' theorem is stated mathematically as the following relationship:
$$
\begin{align}
P(C_k \mid x_1,\dots,x_n) = \frac{P(C_k)P(x_1,\dots,x_n \mid C_k)}{P(x_1,\dots,x_n)}
\end{align}
$$
According to the "naive" conditional independence assumptions, for the given class $C_k$ each feature $x_{i}$ is conditionally independent of every other feature $x_{j}$ for$j\neq i$.
$$
\begin{align}
P(x_i \mid C_k,x_1,\dots,x_n) = P(x_i \mid C_k)
\end{align}
$$
Thus, the relation can be simplified to
$$
\begin{align}
P(C_k \mid x_1,\dots,x_n) = \frac{P(C_k)\prod_{i=1}^{n}P(x_i \mid C_k)}{P(x_1,\dots,x_n)}
\end{align}
$$
Since $P(x_1,...x_n)$ is constant, if the values of the feature variables are known, the following classification rule can be used:
$$
\begin{align}
P(C_k \mid x_1,\dots,x_n) \propto  P(C_k)\prod_{i=1}^{n}P(x_i \mid C_k) \\ \Downarrow \\ \hat{y} = \underset{k}{\arg\max} P(C_k)\prod_{i=1}^{n}P(x_i \mid C_k)
\end{align}
$$
To avoid underflow, log probabilities can be used.
$$
\begin{align}
\hat{y} = \underset{k}{\arg\max}(\ln{P(C_k)}+\sum_{i=1}^{n}\ln{P(x_i \mid C_k)})
\end{align}
$$
The variaty of naive Bayes classifiers primarly differs between each other by the assumptions they make regarding the distribution of $P(x_i \mid C_k)$, while $P(C_k)$ is usually defined as the relative frequency of class $C_k$ in the training dataset.

The multinomial distribution is parametrized by vectors $\theta_k = (\theta_{k1},\ldots,\theta_{kn})$ for each class $C_k$, where $n$ is the number of features (i.e. the size of the vocabulary) and $\theta_{ki}$ is the probability $P(x_i \mid C_k)$ of feature $i$ appearing in a sample that belongs to the class $C_k$.

The parameters $\theta_y$ is estimated by a smoothed version of maximum likelihood, i.e. relative frequency counting:
$$
\begin{align}
\hat{\theta}_{ki} = \frac{ N_{ki} + \alpha}{N_k + \alpha n}
\end{align}
$$
where $N_{ki} = \sum_{x \in T} x_i$ is the number of times feature $i$ appears in a sample of class y in the training set $T$, and $N_{y} = \sum_{i=1}^{|T|} N_{ki}$ is the total count of all features for class $C_k$. The smoothing priors $\alpha \ge 0$ accounts for features not present in the learning samples and prevents zero probabilities in further computations. Setting $\alpha = 1$ is called Laplace smoothing, while $\alpha < 1$ is called Lidstone smoothing.

Thus, the final decision rule is defined as follows:
$$
\begin{align}
\hat{y} = \underset{k}{\arg\max}(\ln{P(C_k)}+\sum_{i=1}^{n}\ln{\frac{ N_{ki} + \alpha}{N_k + \alpha n}})
\end{align}
$$

## 1. Loading data
Sentiment dataset of Tweets in Russian [1] is avaialble at http://study.mokoron.com/. The files positive.csv and negative.csv contain positively labelled and negatively labelled tweets, respectively.

```python
import pandas as pd
import numpy as np
```

```python
# Read the data from CSV files
n = ['id', 'date','name','text','typr','rep','rtw','faw','stcount','foll','frien','listcount']
data_positive = pd.read_csv('positive.csv', sep=';',error_bad_lines=False, names=n, usecols=['text'])
data_negative = pd.read_csv('negative.csv', sep=';',error_bad_lines=False, names=n, usecols=['text'])
```

```python
# Create balanced dataset
sample_size = min(data_positive.shape[0], data_negative.shape[0])
raw_data = np.concatenate((data_positive['text'].values[:sample_size],
                           data_negative['text'].values[:sample_size]), axis=0)
labels = [1]*sample_size + [0]*sample_size
```

## 2. Preprocessing data
Texts generated by humans in social media sites contain lots of noise that can significantly affect the results of the sentiment classification process. Moreover, depending on the features generation approach, every new term seems to add at least one new dimension to the feature space. That makes the feature space more sparse and high-dimensional. Consequently, the task of the classifier has become more complex.

To prepare messages, such text preprocessing techniques as replacing URLs and usernames with keywords, removing punctuation marks and converting to lowercase were used in this program.

```python
import re

def preprocess_text(text):
    text = re.sub('((www\.[^\s]+)|(https?://[^\s]+))','URL', text)
    text = re.sub('@[^\s]+','USER', text)
    text = text.lower().replace("ё", "е")
    text = re.sub('[^a-zA-Zа-яА-Я1-9]+', ' ', text)
    text = re.sub(' +',' ', text)
    return text.strip()

data = [preprocess_text(t) for t in raw_data]
```

## 3. Training MultinomialNB
A Pipeline class was used to make the vectorizer => transformer => classifier easier to work with. Such hyper-parameters as n-grams range, IDF usage, TF-IDF normalization type and Naive Bayes alpha were tunned using grid search. The performance of the selected hyper-parameters was measured on a test set that was not used during the model training step.

```python
from sklearn.pipeline import Pipeline
from sklearn.naive_bayes import MultinomialNB
from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer
from sklearn.model_selection import train_test_split, GridSearchCV

text_clf = Pipeline([('vect', CountVectorizer()),
                     ('tfidf', TfidfTransformer()),
                     ('clf', MultinomialNB())])

tuned_parameters = {
    'vect__ngram_range': [(1, 1), (1, 2), (2, 2)],
    'tfidf__use_idf': (True, False),
    'tfidf__norm': ('l1', 'l2'),
    'clf__alpha': [1, 1e-1, 1e-2]
}
```

The dataset was plited into train and test subsets.

```python
x_train, x_test, y_train, y_test = train_test_split(data, labels, test_size=0.33, random_state=42)
```

```python
from sklearn.metrics import classification_report

score = 'f1_macro'
print("# Tuning hyper-parameters for %s" % score)
print()
np.errstate(divide='ignore')
clf = GridSearchCV(text_clf, tuned_parameters, cv=10, scoring=score)
clf.fit(x_train, y_train)

print("Best parameters set found on development set:")
print()
print(clf.best_params_)
print()
print("Grid scores on development set:")
print()
for mean, std, params in zip(clf.cv_results_['mean_test_score'],
                             clf.cv_results_['std_test_score'],
                             clf.cv_results_['params']):
    print("%0.3f (+/-%0.03f) for %r" % (mean, std * 2, params))
print()

print("Detailed classification report:")
print()
print("The model is trained on the full development set.")
print("The scores are computed on the full evaluation set.")
print()
print(classification_report(y_test, clf.predict(x_test), digits=4))
print()
```

    # Tuning hyper-parameters for f1_macro

    Best parameters set found on development set:

    {'clf__alpha': 1, 'tfidf__norm': 'l2', 'tfidf__use_idf': True, 'vect__ngram_range': (1, 2)}

    Grid scores on development set:

    0.732 (+/-0.006) for {'clf__alpha': 1, 'tfidf__norm': 'l1', 'tfidf__use_idf': True, 'vect__ngram_range': (1, 1)}
    0.741 (+/-0.006) for {'clf__alpha': 1, 'tfidf__norm': 'l1', 'tfidf__use_idf': True, 'vect__ngram_range': (1, 2)}
    0.705 (+/-0.008) for {'clf__alpha': 1, 'tfidf__norm': 'l1', 'tfidf__use_idf': True, 'vect__ngram_range': (2, 2)}
    0.728 (+/-0.004) for {'clf__alpha': 1, 'tfidf__norm': 'l1', 'tfidf__use_idf': False, 'vect__ngram_range': (1, 1)}
    0.730 (+/-0.005) for {'clf__alpha': 1, 'tfidf__norm': 'l1', 'tfidf__use_idf': False, 'vect__ngram_range': (1, 2)}
    0.702 (+/-0.006) for {'clf__alpha': 1, 'tfidf__norm': 'l1', 'tfidf__use_idf': False, 'vect__ngram_range': (2, 2)}
    0.734 (+/-0.006) for {'clf__alpha': 1, 'tfidf__norm': 'l2', 'tfidf__use_idf': True, 'vect__ngram_range': (1, 1)}
    0.753 (+/-0.006) for {'clf__alpha': 1, 'tfidf__norm': 'l2', 'tfidf__use_idf': True, 'vect__ngram_range': (1, 2)}
    0.711 (+/-0.007) for {'clf__alpha': 1, 'tfidf__norm': 'l2', 'tfidf__use_idf': True, 'vect__ngram_range': (2, 2)}
    0.738 (+/-0.004) for {'clf__alpha': 1, 'tfidf__norm': 'l2', 'tfidf__use_idf': False, 'vect__ngram_range': (1, 1)}
    0.750 (+/-0.005) for {'clf__alpha': 1, 'tfidf__norm': 'l2', 'tfidf__use_idf': False, 'vect__ngram_range': (1, 2)}
    0.712 (+/-0.008) for {'clf__alpha': 1, 'tfidf__norm': 'l2', 'tfidf__use_idf': False, 'vect__ngram_range': (2, 2)}
    0.726 (+/-0.005) for {'clf__alpha': 0.1, 'tfidf__norm': 'l1', 'tfidf__use_idf': True, 'vect__ngram_range': (1, 1)}
    0.748 (+/-0.008) for {'clf__alpha': 0.1, 'tfidf__norm': 'l1', 'tfidf__use_idf': True, 'vect__ngram_range': (1, 2)}
    0.707 (+/-0.009) for {'clf__alpha': 0.1, 'tfidf__norm': 'l1', 'tfidf__use_idf': True, 'vect__ngram_range': (2, 2)}
    0.738 (+/-0.005) for {'clf__alpha': 0.1, 'tfidf__norm': 'l1', 'tfidf__use_idf': False, 'vect__ngram_range': (1, 1)}
    0.752 (+/-0.006) for {'clf__alpha': 0.1, 'tfidf__norm': 'l1', 'tfidf__use_idf': False, 'vect__ngram_range': (1, 2)}
    0.711 (+/-0.008) for {'clf__alpha': 0.1, 'tfidf__norm': 'l1', 'tfidf__use_idf': False, 'vect__ngram_range': (2, 2)}
    0.717 (+/-0.006) for {'clf__alpha': 0.1, 'tfidf__norm': 'l2', 'tfidf__use_idf': True, 'vect__ngram_range': (1, 1)}
    0.737 (+/-0.007) for {'clf__alpha': 0.1, 'tfidf__norm': 'l2', 'tfidf__use_idf': True, 'vect__ngram_range': (1, 2)}
    0.699 (+/-0.008) for {'clf__alpha': 0.1, 'tfidf__norm': 'l2', 'tfidf__use_idf': True, 'vect__ngram_range': (2, 2)}
    0.734 (+/-0.005) for {'clf__alpha': 0.1, 'tfidf__norm': 'l2', 'tfidf__use_idf': False, 'vect__ngram_range': (1, 1)}
    0.750 (+/-0.007) for {'clf__alpha': 0.1, 'tfidf__norm': 'l2', 'tfidf__use_idf': False, 'vect__ngram_range': (1, 2)}
    0.707 (+/-0.007) for {'clf__alpha': 0.1, 'tfidf__norm': 'l2', 'tfidf__use_idf': False, 'vect__ngram_range': (2, 2)}
    0.710 (+/-0.006) for {'clf__alpha': 0.01, 'tfidf__norm': 'l1', 'tfidf__use_idf': True, 'vect__ngram_range': (1, 1)}
    0.729 (+/-0.008) for {'clf__alpha': 0.01, 'tfidf__norm': 'l1', 'tfidf__use_idf': True, 'vect__ngram_range': (1, 2)}
    0.692 (+/-0.008) for {'clf__alpha': 0.01, 'tfidf__norm': 'l1', 'tfidf__use_idf': True, 'vect__ngram_range': (2, 2)}
    0.727 (+/-0.006) for {'clf__alpha': 0.01, 'tfidf__norm': 'l1', 'tfidf__use_idf': False, 'vect__ngram_range': (1, 1)}
    0.744 (+/-0.007) for {'clf__alpha': 0.01, 'tfidf__norm': 'l1', 'tfidf__use_idf': False, 'vect__ngram_range': (1, 2)}
    0.700 (+/-0.008) for {'clf__alpha': 0.01, 'tfidf__norm': 'l1', 'tfidf__use_idf': False, 'vect__ngram_range': (2, 2)}
    0.706 (+/-0.005) for {'clf__alpha': 0.01, 'tfidf__norm': 'l2', 'tfidf__use_idf': True, 'vect__ngram_range': (1, 1)}
    0.715 (+/-0.007) for {'clf__alpha': 0.01, 'tfidf__norm': 'l2', 'tfidf__use_idf': True, 'vect__ngram_range': (1, 2)}
    0.686 (+/-0.009) for {'clf__alpha': 0.01, 'tfidf__norm': 'l2', 'tfidf__use_idf': True, 'vect__ngram_range': (2, 2)}
    0.721 (+/-0.006) for {'clf__alpha': 0.01, 'tfidf__norm': 'l2', 'tfidf__use_idf': False, 'vect__ngram_range': (1, 1)}
    0.731 (+/-0.006) for {'clf__alpha': 0.01, 'tfidf__norm': 'l2', 'tfidf__use_idf': False, 'vect__ngram_range': (1, 2)}
    0.694 (+/-0.008) for {'clf__alpha': 0.01, 'tfidf__norm': 'l2', 'tfidf__use_idf': False, 'vect__ngram_range': (2, 2)}

    Detailed classification report:

    The model is trained on the full development set.
    The scores are computed on the full evaluation set.

                 precision    recall  f1-score   support

              0     0.7397    0.7941    0.7659     37078
              1     0.7759    0.7183    0.7460     36792

    avg / total     0.7577    0.7564    0.7560     73870

## 4. Conslusion
The model, which was trained on the development set, demonstrated $F_1=0.765$ on the evaluation set.

## References
1. Y. Rubtsova, "Constructing a Corpus for Sentiment Classification Training", Software & Systems, vol. 109, no. 1, pp. 72-78, 2015.
2. "Naive Bayes", scikit-learn.org, 2018. [Online]. Available: http://scikit-learn.org/stable/modules/naive_bayes.html. [Accessed: 26- Aug- 2018].
3. "Working With Text Data", scikit-learn.org, 2018. [Online]. Available: http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html. [Accessed: 26- Aug- 2018].

```python

```

Sentiment Analysis of Tweets in Russian using Multinomial Naive Bayes.ipynb


# Norms
## Introduction

A norm is a way to measure the size of a vector, a matrix, a tensor, or a function.

Norms are any functions that are characterized by the following properties:

1. Norms are non-negative values. If you think of the norms as a length, you easily see why it can’t be negative.

2. Norms are 0
 if and only if the vector is a zero vector

3. Norms respect the triangle inequity. See bellow.

4. $\left\Vert k⋅u\right\Vert=|k|⋅\left\Vert u\right\Vert$. The norm of a vector multiplied by a scalar is equal to the absolute value of this scalar multiplied by the norm of the vector.

It is usually written with two horizontal bars: $\left\Vert x \right\Vert$

### The triangle inequity

The norm of the sum of some vectors is less than or equal to the sum of the norms of these vectors.

$$\Vert u+v \Vert \le \Vert u \Vert + \Vert v \Vert$$

```python
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
```

```python
u = np.array([1, 6])
v = np.array([4, 2])
print(r'𝑢+𝑣={}'.format(u+v))
print(r'‖𝑢+𝑣‖={}'.format(np.linalg.norm(u+v)))
print(r'‖𝑢‖+‖𝑣‖={}'.format(np.linalg.norm(u)+np.linalg.norm(v)))
```

    𝑢+𝑣=[5 8]
    ‖𝑢+𝑣‖=9.433981132056603
    ‖𝑢‖+‖𝑣‖=10.554898485297798

```python
u = [0,0,1,6]
v = [0,0,4,2]
u_bis = [1,6,v[2],v[3]]
w = [0,0,5,8]
plt.quiver([u[0], u_bis[0], w[0]],
           [u[1], u_bis[1], w[1]],
           [u[2], u_bis[2], w[2]],
           [u[3], u_bis[3], w[3]],
           angles='xy', scale_units='xy', scale=1, color=sns.color_palette())
# plt.rc('text', usetex=True)
plt.xlim(-2, 6)
plt.ylim(-2, 9)
plt.axvline(x=0, color='grey')
plt.axhline(y=0, color='grey')

plt.text(-1, 3.5, r'$||\vec{u}||$', color=sns.color_palette()[0], size=20)
plt.text(2.5, 7.5, r'$||\vec{v}||$', color=sns.color_palette()[1], size=20)
plt.text(2, 2, r'$||\vec{u}+\vec{v}||$', color=sns.color_palette()[2], size=20)

plt.show()
plt.close()
```

Vector illustration of the triangle inequity. Geometrically, this simply means that the shortest path between two points is a line.

### P-norms: general rules

Here is the recipe to get the p-norm of a vector:

1. Calculate the absolute value of each element
2. Take the power $p$ of these absolute values
3. Sum all these powered absolute values
4. Take the power $1/p$ of this result

This is more condensly expressed with the formula:

$$\Vert v \Vert_p = \left[\sum_i |v_i|^p\right]^{1/p}$$

This will be clear with examples using these widely used p-norms.

#### $l^0$-Norm ($p=0$)

$l^0$-Norm is the number of non zero components. This norm has the problem, that is violates the 2nd property from above.

#### $l^1$-Norm ($p=1$)

$l^1$-Norm is the sum of absolute values: $\Vert v \Vert_1 = |v_1|+ \dots + |v_n|$. It plays a significant role in compressed sensing, signal processing etc.

#### $l^2$-Norm, Euclidean Norm ($p=2$)

$l^2$-Norm is the sum of squares: $\Vert v \Vert_2 = \sqrt{v_1^2+ \dots + v_n^2}$.

#### $l^\infty$-Norm, Maximum Norm ($p=\infty$)

$l^\infty$-Norm is the maximum of the absolute values: $\Vert v \Vert_\infty = \max{|v_i|}$.

Generally, $p$ can be any positive value $p\ge 1$. Prof Strang suggests [in his lecture](https://ocw.mit.edu/courses/mathematics/18-065-matrix-methods-in-data-analysis-signal-processing-and-machine-learning-spring-2018/) to visualize p-Norms as follows:

```python
from IPython.display import YouTubeVideo
from datetime import timedelta

vid = YouTubeVideo("NcPUI7aPFhA", start=int(timedelta(hours=0, minutes=9, seconds=22).total_seconds()))
display(vid)
```

```python
from sympy import *
x, y = symbols('x y')
def plot_p_norm(p,rhs=1):
    return plot_implicit(Eq(abs(x)**p + abs(y)**p, rhs),
                         (x, -(rhs+.5), (rhs+.5)),
                         (y, -(rhs+.5), (rhs+.5)),
                         title=r"$l^{{{}}}$ Norm".format(p),
                         size=(5,5),
                         show=False
                        )
plot_p_norm(0.5).show()
plot_p_norm(1).show()
plot_p_norm(2).show()
plot_p_norm(100).show()
```

A true Norm has a convex unit ball $\Vert x \Vert \le 1$. The first plot for $p=.5$ is concave. It's not a norm!

## Vektor Norm
A generalisition of the p-Norm is
$$\Vert v \Vert_S=\sqrt{v^T S v}$$
where $S$ is a [positiv definit symmetric Matrix](https://ocw.mit.edu/courses/mathematics/18-065-matrix-methods-in-data-analysis-signal-processing-and-machine-learning-spring-2018/video-lectures/lecture-5-positive-definite-and-semidefinite-matrices). This can generate all Vector norms that are natural to use.

A special case for $S=\mathbb{1}$ (identity matrix) is the p2-norm.

```python
def plot_s_norm(S,rhs=1):
    v=Matrix([x, y])
    lhs=(v.T*S*v)[0]
    return plot_implicit(Eq(lhs, rhs),
                         (x, -(rhs+.5), (rhs+.5)),
                         (y, -(rhs+.5), (rhs+.5)),
                         title=r"$v^TSv={}$, for S={}: {}".format(rhs,S,lhs),
                         size=(5,5),
                         show=False
                        )

plot_s_norm(Matrix([[1, 0], [0, 1]])).show()
```

For $S=\big(\begin{smallmatrix}
  2 & 0\\
  0 & 3
\end{smallmatrix}\big)$
we would get $2x^2+3y^2$, and that would be a weighted norm.

If we plot all of those norms equal to 1 in 2d we will get the ellipse $2x^2+3y^2=1$.

```python
plot_s_norm(Matrix([[2, 0], [0, 3]])).show()
```

**Example:** Application in Optimization

We want to minimise the p-Norm of $x$, constrained by $Av=b$:
$$\min_{Av=b}\Vert v \Vert_p$$

One way of doing this is to blow up the norm (increase rhs) until it hits the constraint. The point where it hits is the minimum.

For our 2d example we use the constraint $y=-2/3*x+3$.

Which point $(x,y)$ minimises the $l^1$ norm? The $l^1$ norm is the sparcest vector, so it's always on one of the axes.

```python
p1=plot_implicit(Eq(-2/3*x+3, y),
                 title=r"$\min_{y=-2/3*x+3}\Vert v \Vert_1=3$",
                 size=(5,5),
                 show=False)
p1.extend(plot_p_norm(1, 3))
p1.show()
```

Which point $(x,y)$ minimises the $l^2$ norm? The $l^2$ norm is how far out you go, so the radius of a circle.

```python
p1=plot_implicit(Eq(-2/3*x+3, y),
                 title=r"$\min_{y=-2/3*x+3}\Vert v \Vert_2\approx 6$",
                 size=(5,5),
                 show=False)
p1.extend(plot_p_norm(2, (18/13)**2+(27/13)**2))
# p1.extend(plot_implicit(Eq(3/2*x, y),show=False))
p1.show()
```

## Matrix Norm

- $\Vert A \Vert_2=\sigma_1$: The **2 norm** of a matrix is the laggest [singular value of the matrix](https://ocw.mit.edu/courses/mathematics/18-065-matrix-methods-in-data-analysis-signal-processing-and-machine-learning-spring-2018/video-lectures/lecture-6-singular-value-decomposition-svd).
$$\Vert A \Vert_2=\max_{\text{all } x}\frac{\Vert Ax \Vert_2}{\Vert x \Vert_2}=\frac{\Vert Av_1 \Vert}{\Vert v_1 \Vert}=\sigma_1$$
where $v_1$ is the first right singular vektor, with $\Vert v_1 \Vert = 1$ and $Av_1=\sigma_1 u_1$.
- $\Vert A \Vert_F = \sqrt{\text{add all }\vert a_{ij} \vert^2}= \sqrt{\sigma_1^2+\dots+\sigma_r^2}$: The **Frobenius norm** of a matrix, considder singular value decomposition $A=U\Sigma V^T$, with $\Vert \Sigma \Vert_F=\sqrt{\sigma_1^2+\dots+\sigma_r^2}$.
- $\Vert A \Vert_N = \sigma_1+\dots\sigma_r$ The Nuckear Norm of a matrix, like $l_1$ norm

### Definition of a matrix norm:
The matrix norm of $A$ is the maximum over all unit vectors of the norm of $Ax$:
$$\Vert A \Vert = \max_{\Vert x\Vert=1}\Vert Ax \Vert$$
We have seen the visualization of the norm $\Vert x \Vert = 1$ above. Now we apply a martix transformation on $x$ and look for the maxima.

**Example:** counterclockwise rotation by $\theta$ with rotation matrix:
$$A=
  \begin{bmatrix}
    \cos{\theta} &  -\sin{\theta}\\
    \sin{\theta} & \cos{\theta}
  \end{bmatrix}$$

```python
def plot_Ax_s_norm(A,S,rhs=1):
    v=A*Matrix([x, y])
    lhs=(v.T*S*v)[0]
    return plot_implicit(Eq(lhs, rhs),
                         (x, -(rhs+.5), (rhs+.5)),
                         (y, -(rhs+.5), (rhs+.5)),
                         # title=r"$v^TSv={}$, for S={}: {}".format(rhs,S,lhs),
                         size=(5,5),
                         show=False
                        )
theta = np.pi/4 # 45 degree
p1 = plot_Ax_s_norm(Matrix([[cos(theta), -sin(theta)], [sin(theta), cos(theta)]]),
               Matrix([[2, 0], [0, 3]]))
p1.extend(plot_s_norm(Matrix([[2, 0], [0, 3]])))
p1.show()
```

stretch by a factor k along the x-axis is given by:
$$A=
  \begin{bmatrix}
    k &  0\\
    0 & 1
  \end{bmatrix}$$

```python
k = 2 # 2x strech along x-axis
p1 = plot_Ax_s_norm(Matrix([[k, 0], [0, 1]]),
               Matrix([[2, 0], [0, 3]]))
p1.extend(plot_s_norm(Matrix([[2, 0], [0, 3]])))
p1.show()
```

```python

```


## Programming Exercise 2: Logistic Regression
#### Author - Rishabh Jain

```python
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
import numpy as np
%matplotlib inline
```

### 1    Logistic Regression

##### Problem Statement
In this part of the exercise, we will build a logistic regression model to predict whether the student gets admitted into a university.

Suppose that we are the adminstrator of a department and want to determine each applicant's chance of admission based on their results in two exams. We have hisotrical data from previous applicants that we can use as a training set for logistic regression. For each applicant, we have their scores in two exams and their admission decision.

Our task is to build a classification model that estimates an applicant's probability of admission based on the score from those two exams.

```python
data=pd.read_csv('./ex2data1.csv')
print(f'TRAINING DATASET SHAPE : {data.shape[0]} X {data.shape[1]}')
data.sample(5)
```

    TRAINING DATASET SHAPE : 100 X 3

      exam1
      exam2
      decision

      3
      60.182599
      86.308552
      1

      37
      64.176989
      80.908061
      1

      79
      82.226662
      42.719879
      0

      85
      68.468522
      85.594307
      1

      64
      44.668262
      66.450086
      0

#### 1.1    Visualizing the data
Before starting to implement any learning algorithms, it is always good to visualize the data if possible.

```python
sns.scatterplot(x='exam1',y='exam2',data=data[data['decision']==0],label='Not Admitted');
sns.scatterplot(x='exam1',y='exam2',data=data[data['decision']==1],label='Admitted');
```

#### 1.2    Implementation
We will add another dimension to the Design Matrix to accomodate the $\theta_0$ intercept term. We will also initialize the fitting parameters to 0.

```python
m=data.shape[0]
X=np.ones(shape=(m,3))
X[:,1:]=data.values[:,:2]
print(f'X : {X.shape}')

y=data.values[:,2]
y=y[:,np.newaxis]
print(f'y : {y.shape}')

theta=np.zeros(shape=(1,X.shape[1]))
print(f'theta : {theta.shape}')
```

    X : (100, 3)
    y : (100, 1)
    theta : (1, 3)

##### 1.2.1    Sigmoid/Logistic Function

Before starting with the implementation, let's try to understand the sigmoid function.
For large positive values of $x$, sigmoid function is close to 1, while for large negative values, sigmoid function is close to 0. And sigmoid of 0 is exactly 0.5 .

$$ \sigma(x)=\frac{1}{1+e^{-x}} $$

```python
def sigmoid(z):
    sigma=1/(1+np.exp(-z))
    return sigma
```

```python
df=pd.DataFrame({'X':np.linspace(-10,10,100)})
df['Y']=sigmoid(df['X'])
sns.lineplot(x='X',y='Y',data=df,label='SIGMOID CURVE');
```

###### Model Representation

The hypothesis in Linear Regression (i.e. $h_\theta(x)=\theta^Tx$) cannot be used for the classification problems because the linear hypothesis output ranges from $-\infty$ to $\infty$, whereas $y$ (i.e. target variable) in classification problems are either 0 or 1. Hence the range of hypothesis output in Logistic Regression should be:

$$ 0\le h_\theta(x) \le 1 $$

In order to keep $h_\theta(x)$ in the range mentioned above, Sigmoid Function comes to rescue:

$$ h_\theta(x)=g(\theta^Tx) $$
$$ g(z)=\frac{1}{1+e^{-z}} $$

```python
def predict(X,theta):
    '''Predicts by applying logistic function on linear model'''
    z=np.dot(X,theta.T)
    h=sigmoid(z)
    return h
```

##### 1.2.2    Cost function
We learnt about the cost function $J(\theta)$ in **Linear Regression**, the cost function represents the optimization objective i.e. we create a cost function and minimize it to develope an accurate model with minimum error.
$$ J(\theta)=\frac{1}{2m}\sum_{i=0}^m(h_\theta(x^{(i)})-y^{(i)})^2 $$
On replacing $h_\theta(x)$ with $\frac{1}{1+e^{(-\theta^Tx)}}$ in the above equation,it becomes '**Non Convex**' which does not guarrante to converge and find the global minimum. So we need a new cost function for Logistic Regression.

The figure in left is a '**Convex**' function and has only one minimum called global minimum and will always converage whereas the figure in right is a '**Non Convex**' function and has mutiple minimums also called local minimums. And there is no guarrante that the non convex function will always converge.

For **Logistic Regression** cost function is defined as:

$$\begin{equation}
  Cost(h_\theta(x),y)=\begin{cases}
    -log(h_\theta(x)) & \text{if }y=1 \\
    -log(1-h_\theta(x)) & \text{if }y=0
  \end{cases}
\end{equation}$$

Let's try to understand the cost function.

For $y=1$
- $J(\theta)\approx\infty$ if $h_\theta(x)\approx0$ (Penalty for wrong prediction)
- $J(\theta)\approx0$ if $h_\theta(x)\approx1$

For $y=0$
- $J(\theta)\approx\infty$ if $h_\theta(x)\approx1$ (Penalty for wrong prediction)
- $J(\theta)\approx0$ if $h_\theta(x)\approx0$

```python
# In Logistic Regression, 0<h(x)<1
df=pd.DataFrame({'h':np.linspace(0.01,0.99,100)})
df['Cost_1']=-np.log(df['h'])
df['Cost_0']=-np.log(1-df['h'])
sns.lineplot(x='h',y='Cost_0',data=df,label='y=0');
sns.lineplot(x='h',y='Cost_1',data=df,label='y=1');
```

From the above graph and the set of equations, we can see that this new cost function captures the intuition of penalizing the learning algorithm by a very large cost if predictions are not even remotely close. Combinig both the equations, **Cost function of Logistic Regression** also called **Cross Entropy function** looks like this:

$$ J(\theta)= \frac{-1}{m}\sum_{i=0}^m[y^{(i)}log(h_\theta(x^{(i)}))+(1-y^{(i)})log(1-h_\theta(x^{(i)})]$$

```python
def computeCost(h,y):
    '''Computes the cost using Cross Entropy function'''
    m=y.shape[0]
    J=(-1/m)*np.sum(np.multiply(y,np.log(h))+np.multiply(1-y,np.log(1-h)))
    return J
```

```python
h=predict(X,theta)
computeCost(h,y)
```

    0.6931471805599453

##### 1.2.3    Learning Parameters

**Gradient Descent Optimization** :
$$ \theta_j := \theta_j - \alpha \frac{\partial}{\partial \theta_j}J(\theta) $$

**Lets calculate the $\frac{\partial}{\partial \theta_j}J(\theta)$ for Logistic Regression cost function.**

$ log(h_\theta(x))=log(\frac{1}{1+e^{-\theta^Tx}})=-log(1+e^{-\theta^Tx})\;\;\;\;\; (1) $
$ log(1-h_\theta(x))=log(1-\frac{1}{1+e^{-\theta^Tx}})=log(\frac{e^{-\theta^Tx}}{1+e^{-\theta^Tx}})=log(e^{-\theta^Tx})-log(\frac{1}{1+e^{-\theta^Tx}})=-\theta^Tx-log(1+e^{-\theta^Tx})\;\;\;\;\; (2) $

Replacing $(1)$ and $(2)$ in $J(\theta)$,

$ J(\theta)=\frac{-1}{m}\sum_{i=0}^m[-y^i(log(1+e^{(-\theta^Tx^i)}))+(1-y^i)(-\theta^Tx^i-log(1+e^{-\theta^Tx^i}))] $

$ \;\;\;\;\;\;\;=\frac{-1}{m}\sum_{i=0}^m[-y^ilog(1+e^{-\theta^Tx^i})-\theta^Tx^i-log(1+e^{-\theta^Tx^i})+y^i\theta^Tx^i+y^ilog(1+e^{-\theta^Tx^i})] $

$ \;\;\;\;\;\;\;=\frac{-1}{m}\sum_{i=0}^m[y^i\theta^Tx^i-(log(e^{\theta^Tx^i})+log(1+e^{-\theta^Tx^i}))]\;\;\;\;\;(using\; m=log(e^m)) $

$ \;\;\;\;\;\;\;=\frac{-1}{m}\sum_{i=0}^m[y^i\theta^Tx^i-log(1+e^{\theta^Tx^i})]\;\;\;\;\;(using\; log(m)+log(n)=log(mn)) $

$ \frac{\partial}{\partial \theta}(J(\theta))=\frac{-1}{m}\sum_{i=0}^m[\frac{\partial}{\partial \theta}(y^i\theta^Tx^i)-\frac{\partial}{\partial \theta}(log(1+e^{\theta^Tx^i}))] $

$ \;\;\;\;\;\;\;\;\;\;\;\;=\frac{-1}{m}\sum_{i=0}^m[y^ix^i-\frac{1.e^{\theta^Tx^i}.x^i}{1+e^{\theta^Tx^i}}]\;\;\;\;\;(using\;\frac{d}{dx}(uv)=u.\frac{dv}{dx}+v.\frac{du}{dx}) $

$ \;\;\;\;\;\;\;\;\;\;\;\;=\frac{-1}{m}\sum_{i=0}^m[y^ix^i-\frac{x^i}{\frac{1+e^{\theta^Tx^i}}{e^{\theta^Tx^i}}}] $

$ \;\;\;\;\;\;\;\;\;\;\;\;=\frac{-1}{m}\sum_{i=0}^m[y^ix^i-\frac{x^i}{1+e^{-\theta^Tx^i}}] $

$ \;\;\;\;\;\;\;\;\;\;\;\;=\frac{1}{m}\sum_{i=0}^m(h_\theta(x^i)-y^i)x^i\;\;\;\;\; (3) $

Replacing $(3)$ in update rule:

$$ \theta_j:=\theta_j-\alpha\frac{1}{m}\sum_{i=0}^m(h_\theta(x^{(i)})-y^{(i)})x_j^{(i)} $$

As we can see that the Gradient Descent update rule remains the same even though the underlying cost function has changed.

```python
def gradientDescent(X,y,theta,alpha,iterations):
    '''Minimizes the cost function using Gradient Descent Optimization Algorithm'''
    m=X.shape[0]
    jHistory=[]
    for i in range(iterations):
        h=predict(X,theta)
        # Computing cost
        J=computeCost(h,y)
        jHistory.append(J)
        # Parameters update rule
        gradient=(alpha/m)*np.dot((h-y).T,X)
        theta=theta-gradient
    return (theta,jHistory)
```

```python
def featureNormalize(X):
    '''Normalizes the Design Matrix assuming the first column is filled with ones for intrecept term.'''
    X[:,1:]=(X[:,1:]-X[:,1:].mean(axis=0))/X[:,1:].std(axis=0)
    return X
```

**Normalizing the features before training the model is important as it speeds up the optimization.**

```python
X=featureNormalize(X)
X[:10,:]
```

    array([[ 1.        , -1.60224763,  0.63834112],
           [ 1.        , -1.82625564, -1.2075414 ],
           [ 1.        , -1.53903969,  0.3612943 ],
           [ 1.        , -0.28210129,  1.0863683 ],
           [ 1.        ,  0.69152826,  0.49337794],
           [ 1.        , -1.06199728, -0.5357394 ],
           [ 1.        , -0.23437234,  1.63818413],
           [ 1.        ,  0.4845113 , -1.06373024],
           [ 1.        ,  0.53998666,  1.14651105],
           [ 1.        ,  0.97044832, -1.22709853]])

```python
alpha=0.3
iterations=1500

theta,jHistory=gradientDescent(X,y,theta,alpha,iterations)
df=pd.DataFrame({'Iterations':range(iterations),'Cost Function':jHistory})
print('FINAL COST : %.4f'%jHistory[iterations-1])
sns.lineplot(data=df,x='Iterations',y='Cost Function');
```

**Plotting Decision Boundary**
A decision boundary is hypersurface that partitions the underlying vector into two sets or classes.

Genral Equation of Line :
$$ ax+by=c $$
$$ \theta_1x_1+\theta_2x_2=-\theta_0 $$
$$ x_2=(\frac{-1}{\theta_2})(\theta_0+\theta_1x_1)$$

```python
sns.scatterplot(x='exam1',y='exam2',data=data[data['decision']==0],label='Not Admitted');
sns.scatterplot(x='exam1',y='exam2',data=data[data['decision']==1],label='Admitted');
# Only need two points to define a line, so choose two endpoints
x1=np.array([min(X[:,1]),max(X[:,1])])
x2=np.multiply((-1/theta[0][2]),(theta[0][0]+np.multiply(theta[0][1],x1)))
# De-normalizing the data
x1=x1*data['exam1'].std()+data['exam1'].mean()
x2=x2*data['exam2'].std()+data['exam2'].mean()
sns.lineplot(x=x1,y=x2,label='Decision Boundary',color='green');
```

##### 1.2.4    Evaluating Logistic Regression

```python
h=predict(X,theta).reshape(m)
result=pd.DataFrame({'actual':data['decision'],'prediction':h})
result.sample(10)
```

      actual
      prediction

      64
      0
      0.075530

      73
      1
      0.878757

      63
      0
      0.000157

      10
      0
      0.905205

      49
      1
      0.999988

      21
      1
      0.998391

      40
      1
      0.967775

      38
      0
      0.209379

      99
      1
      0.999695

      6
      1
      0.998787

Calculating the admission probability of student with Exam 1 score of 45 and Exam 2 score of 85.

```python
x=[1,45,85]
# Normalizing the data
x[1]=(x[1]-data['exam1'].mean())/data['exam1'].std()
x[2]=(x[2]-data['exam2'].mean())/data['exam2'].std()
print('ADMISSION PROBABILITY : %.3f'%predict(x,theta)[0])
```

    ADMISSION PROBABILITY : 0.770

Calculating the training accuracy.

```python
p=np.round(h).reshape(m,1)
accuracy=np.mean(y==p)*100
print(f'TRAINING ACCURACY : {accuracy}%')
```

    TRAINING ACCURACY : 89.0%

### 2    Regularized Logistic Regression

##### Problem Statement
In this part of the exercise, we will implement regularized logistic regression to predict whether microchips from a fabrication plant passes quality assurance (QA). During QA,e ach microchip goes through various tests to ensure it is functioning correctly.

Suppose we are the product manager of the factory and we have the rest results for some microchips on two different tests. From these two tests, we would like to determine whether the microchips should be accepted or rejected. To help us make a decision, we have a dataset of test results on past microchips, from which we can build a logistic regression model.

```python
data=pd.read_csv('./ex2data2.csv')
print(f'TRAINING DATASET SHAPE : {data.shape[0]} X {data.shape[1]}')
data.sample(5)
```

    TRAINING DATASET SHAPE : 118 X 3

      test1
      test2
      result

      114
      -0.593890
      0.49488
      0

      16
      0.166470
      0.53874
      1

      92
      0.103110
      0.77997
      0

      38
      0.062788
      -0.16301
      1

      79
      0.085829
      -0.75512
      0

#### 2.1    Visualizing the data

```python
sns.scatterplot(x='test1',y='test2',data=data[data['result']==0],label='Rejected');
sns.scatterplot(x='test1',y='test2',data=data[data['result']==1],label='Accepted');
```

Figure above shows that our dataset cannot be separated into two different classes by a straight line. Therefore, a straight forward application of logistic regression will not perform well on this dataset since logistic regression will only be able to find a linear decision boundary.

#### 2.2    Feature Mapping

One way to fit the data better is to create more features from each data point. We will map the features into all polynomial terms of $x_1$ and $x_2$ up to $n^{th}$ power. As a result of this mapping, our vector of two features will be transformed into higher-dimensional feature vector. A logistic regression classifier trained on higher-dimension feature vector will have a more complex decision boundary and will appear non-linear.

While the feature mapping allows us to build a more expressive classifier and  helps us solve the **Underfitting** problem, it is also more susceptible to **Overfitting** problem.

From $[x_1,x_2]$ features to $[1,x_1,x_2,x_1^2,x_1x_2,x_2^2,x_1^3...x_2^n]$ features.

```python
def mapFeature(x1,x2,degree):
    result=np.ones(shape=(x1.shape[0],1))
    for i in range(1,degree+1):
        for j in range(0,i+1):
            column=np.multiply(np.power(x1,i-j),np.power(x2,j))
            result=np.column_stack((result,column))
    return result
```

#### 2.3    Cost Function and Gradient

In this part of the exercise, we will implement the **regularized** logistic regression to fit the data and also see for ourselves how parameters regularization can help combat the overfitting problem.

If we have too many features, the learned hypothesis will have large parameters and fit the training set very well such that $J(\theta)\approx0$, but fail to generalize new examples. What are our options available?
1. Reduce the number of features. (we will be throwing away the important information here.)
2. **Regularization** - this process keeps all the features and adds a penalty to the **RSS (Sum of squared residuals)** to reduce the freedom of the model. Hence, the model will less likely to fit the noise of the training data and will improve the generalization abilities of the model. In layman terms, regularization tries to prevent the parameters from growing larger.

##### Types of Regularization
1. **L1 Regularization (used in Lasso Regression)**
LASSO stands for Least Absolute Shrinkage and Selection Operator.
This regularization will shrink some parameters to **Zero**. Hence, this type of regularization can also help in **Feature Selection** in the model. As $\lambda$ grows bigger, more shrinkage in parameters will occur.
$$ J(\theta)=RSS+\lambda \sum_{j=1}^n|\theta_j| $$
2. **L2 Regularization (used in Ridge Regression)**
Parameters decrease progressively as $\lambda$ grows larger, **but are never cut to zero.**
$$ J(\theta)=RSS+\lambda \sum_{j=1}^n(\theta_j^2) $$
3. **L1/L2 Regularization (used in Elastic Net Regression)**
Elastic-net is a mix of both L1 and L2 regularization. Lambda is a shared penalization parameter while alpha sets the ratio between L1 and L2 regularization in the Elastic Net Regularization.
$$ J(\theta)=RSS+\lambda[(1-\alpha)\sum_{j=1}^n(\theta_j^2)+\alpha\sum_{j=1}^n|\theta_j|]$$

##### Geometric Perspective on Regularization
Lasso, Ridge and Elastic-Net regularization can also be viewed as a **constraint** added to **optimization** process. Lasso error minimization can be written as :
$$ minimize(RSS)\;\;\;\;subject\;to\; \sum_{j=1}^n|\theta_j|<t$$
Ridge error minimization can be written as :
$$ minimize(RSS)\;\;\;\;subject\;to\; \sum_{j=1}^n(\theta_j)^2<t$$

For the L1 regularization, the constraint region ($|\theta_1|+|\theta_2|<t$) is a diamond, this means that the loss function(RSS) contours has some chances of touching the corners of diamond at $\beta_1$ or $\beta_2$ axis, **setting some of the lasso coeffcients to zero**. While for the L2 regularization the constraint reigon ($\theta_1^2+\theta_2^2<t$) is a circle, thus it has no corners and the coefficients can not be equal to zero.

**Regularized Cost Function :**
$$ J(\theta)= \frac{-1}{m}\sum_{i=0}^m[y^{(i)}log(h_\theta(x^{(i)}))+(1-y^{(i)})log(1-h_\theta(x^{(i)})]+\frac{\lambda}{2m}\sum_{j=1}^n\theta_j^2$$
**Regularized Gradient Descent :**
$$ \theta_j:=\theta_j-\frac{\alpha}{m}(\sum_{i=0}^m(h_\theta(x^{(i)})-y^{(i)})x_j^{(i)})\;\;\text{ , for}\;j=0 $$

$$ \theta_j:=\theta_j-\frac{\alpha}{m}(\sum_{i=0}^m(h_\theta(x^{(i)})-y^{(i)})x_j^{(i)}+\lambda\theta_j)\;\;\text{ , for}\;j\neq0 $$

##### 2.3.1    Learning Parameters

```python
def regularizedGradientDescent(X,y,theta,alpha,lmbda,iterations):
    '''Minimizes the cost function using Gradient Descent Optimization Algorithm'''
    m=X.shape[0]
    jHistory=[]
    for i in range(iterations):
        h=predict(X,theta)
        # Computing cost
        J=computeCost(h,y)
        # Adding the regularized term
        J=J+(lmbda/(2*m))*np.sum(np.power(theta[:,1:],2))
        jHistory.append(J)
        # Parameters update rule
        gradient=(alpha/m)*(np.dot((h-y).T,X))
        # Adding the regularized term
        gradient=gradient+(alpha/m)*lmbda*np.column_stack((np.zeros((1,1)),theta[:,1:]))
        theta=theta-gradient
    return (theta,jHistory)
```

#### 2.4    Plotting the decision boundary

```python
def plotDecisionBoundary(theta,data,y,axis):
    sns.scatterplot(x='test1',y='test2',data=data[data['result']==0],label='Rejected',ax=axis);
    sns.scatterplot(x='test1',y='test2',data=data[data['result']==1],label='Accepted',ax=axis);
    u=np.linspace(-1,1.5)
    v=np.linspace(-1,1.5)
    z=np.zeros(shape=(len(u),len(v)))
    # Evaluate z=X*theta over the grid
    for i in range(len(u)):
        for j in range(len(v)):
            X=mapFeature(np.reshape(u[i],(1,1)),np.reshape(v[j],(1,1)),degree)
            z[i,j]=np.dot(X,theta.T)
    z=z.T
    axis.contour(u,v,z,levels=[0])
```

#### Models with different $\lambda$ values

```python
degree=6
lmbdas=[0,1,100]
alphas=[10,0.03,0.03]
iterations=[40000,10000,10000]

labels=['No Regularization (Overfitting)','Regularization (Just Right)','Too much Regularization (Underfitting)']

nrows,ncols=2,3
fig,ax=plt.subplots(nrows,ncols,figsize=(18,8))
fig.suptitle('Models with different lambda values',size=16)

for i in range(ncols):
    X=mapFeature(data['test1'],data['test2'],degree)
    y=data.values[:,2]
    y=y[:,np.newaxis]
    theta=np.zeros(shape=(1,X.shape[1]))
    theta,jHistory=regularizedGradientDescent(X,y,theta,alphas[i],lmbdas[i],iterations[i])
    axis1=ax[0][i]
    axis1.set_title(f'{labels[i]}\nlambda = {lmbdas[i]} | Cost = {jHistory[iterations[i]-1]:.3f}')
    plotDecisionBoundary(theta,data,y,axis1)
    axis2=ax[1][i]
    c=sns.distplot(theta[0],ax=axis2,kde=False)
    c.set(xlabel=f'Theta{i+1} Values',ylabel='Frequency')
```


# Physics 256
## Mean Field Theory for the Ising Model

```python
import matplotlib.pyplot as plt
import numpy as np

%matplotlib inline
plt.style.use('notebook');
%config InlineBackend.figure_format = 'retina'
colors = ["#2078B5", "#FF7F0F", "#2CA12C", "#D72827", "#9467BE", "#8C574B",
            "#E478C2", "#808080", "#BCBE20", "#17BED0", "#AEC8E9", "#FFBC79",
            "#98E08B", "#FF9896", "#C6B1D6", "#C59D94", "#F8B7D3", "#C8C8C8",
           "#DCDC8E", "#9EDAE6"]
```

### We want to solve

\begin{equation}
M = \tanh \left(\frac{M z J}{T}\right)
\end{equation}

Let's begin by looking for a graphical solution.  Define $x = zJ/T$: at fixed $J$, $x\gg 1$ corresponds to low temperature while $x\ll 1$ is high temperature.

```python
M = np.linspace(-1.5,1.5,1000)
```

```python
plt.plot(M,M,color='k', label='$y=M$')
for cx in [2,1.25,1.0,0.5]:
    plt.plot(M,np.tanh(M*cx), label='x = %3.2f' % cx)
plt.xlabel(r'$M$')
plt.legend(loc='lower right')
plt.title('Mean Field Theory')
```

## Constructing the solution as a function of temperature

It appears that there is always a solution between 0 and 1 for any value of $x$.

```python
def mean_field_eqn(m,cx):
    '''The mean field equation for the magnetization.'''
    return m - np.tanh(m*cx)
```

```python
from scipy.optimize import fsolve
x = np.linspace(0.5,100,10000)
Mx = [fsolve(mean_field_eqn, 1.1, args=(cx)) for cx in x]
```

```python
plt.plot(1.0/x,Mx, linewidth=4)
plt.xlabel('Temperature  $T/zJ$')
plt.ylabel('Magnetization  $M$')
plt.ylim(-0.001,1.05);
```

```python

```


# Euler Method

```python
import matplotlib.pyplot as plt
import numpy as np
```

```python
step = 0.5
x = np.arange(-10, 10, step)
y = np.exp(0.1*x)
```

```python
plt.figure(figsize=(15, 5))
plt.plot(x, y)
plt.plot(x, y, "k+", markeredgewidth=2)
plt.bar(x[:-1], y[:-1], width=step, align="edge", color='red', edgecolor='black', hatch="/")
plt.title("Integration from the left side")
plt.grid()
```

We want to find a numerical solution (i.e. the shape of $x(t)$ given following equation.

\begin{align}
\dot{x}(t) + a_1x(t) = b_0 + b_1u(t)
\end{align}

We rewrite it to have the first derivative on the left side:

\begin{align}
\dot{x}(t) = b_0 + b_1u(t) - a_1x(t)
\end{align}

Given initial value for $x(t=0)$ we can derive $x(t)$ by integrating

\begin{align}
x(t) = \int_{\tau=0}^{t}[b_0 + b_1u(t) - a_1x(t)] \text{d}\tau
\end{align}

We can approximate above equation using small steps $T$:

\begin{align}
x(k+1) = T\sum_{i=0}^{k}b_0 + b_1u(i) - a_1x(i)
\end{align}

We can do that incrementally too (rectangle method from the left side):

\begin{align}
x(k+1) \approx x(k) + T\left[b_0 + b_1u(k) - a_1x(k)\right]
\end{align}

```python
# step function
def u(t):
    if t>0:
        return 1
    else:
        return 0
```

```python
T=1e-3
```

```python
b0, b1, a1 = 1, 2, 1
```

```python
t_sim = np.arange(0, 2, T)
x_t = np.zeros_like(t_sim)
n_steps = t_sim.size
```

```python
x_k = 0
for i in range(n_steps):
    x_t[i] = x_k + T*(b0 + b1*u(i)+x_k)
    x_k = x_t[i]
```

```python
fig, ax = plt.subplots(nrows=2, figsize=(15, 5), sharex=True)
ax[0].plot(t_sim, [u(t) for t in range(len(t_sim))])
ax[0].set_title("input $u(t)$ - step function")
ax[0].grid(True)
ax[1].plot(t_sim, x_t)
ax[1].set_title("output $x(t)$")
ax[-1].set_xlabel("t [s]")
ax[-1].set_xlim([-5e-3, 2])
ax[-1].set_ylim([0, 30])
ax[-1].grid(True)
```

```python

```