# Linear regression from

Let's implement different components of Linear Regression algorithm specifically **model inference, loss function, optimization and evaluation measures** in a vectorized form from scratch using basic python libraries like numpy.

We will demonstrate working of our implementation on a couple of synthetic datasets.

## Quick recap

A quick recap of vectorized operations for these components:

1. Training data contains features and label that is real number.
2. Model or inference: $\mathbf{y} = \mathbf{X}\mathbf{w}$
3. Loss function: $J(\mathbf{w}) = \frac{1}{2} (\mathbf{X}\mathbf{w} - \mathbf{y})^T (\mathbf{X}\mathbf{w} - \mathbf{y})$
4. Optimization:
  * Normal equation:
  * Gradient descent:
5. RMSE: $\sqrt{\frac{2}{n} J(\mathbf{w})}$

Let's first import necessary libraries

```python
from IPython.display import display, Math, Latex  # Imported for proper rendering of latex in colab.

import numpy as np

# Import for generating plots
import matplotlib.pyplot as plt
import seaborn as sns

%matplotlib inline
```

# C1: Training Data

```python
# Create a dataset of 100 examples with a single feature and a label.
# For this construction, we use the following three parameters:
w1 = 3
w0 = 4
n = 100

X = 10 * np.random.rand(
    n,
)

# Obtain y = 4 + 3*x + noise.  Noise is randomly sampled.
y = (
    w0
    + w1 * X
    + np.random.randn(
        n,
    )
)
```

Let's examine the shapes of training data for sanity check.

```python
print("Shape of the training data feature matrix:", X.shape)
print("Shape of label vector:", y.shape)
```

    Shape of the training data feature matrix: (100,)
    Shape of label vector: (100,)

Let's divide the data into training and test set.  We will set aside 20% examples for testing.

```python
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=42)
```

Let's do a quick sanity check to make sure the sizes of feature and labels sets are identical both in training and test sets:

```python
print("Shape of training feature matrix:", X_train.shape)
print("Shape of training label vector:", y_train.shape)

print("Shape of test feature matrix:", X_test.shape)
print("Shape of test label vector:", y_test.shape)
```

    Shape of training feature matrix: (80,)
    Shape of training label vector: (80,)
    Shape of test feature matrix: (20,)
    Shape of test label vector: (20,)

Let's quickly check the first few examples and labels

```python
X_train[:5]
```

    array([2.66264564, 5.2778843 , 7.14629389, 5.3417576 , 9.21518806])

```python
y_train[:5]
```

    array([14.41877402, 22.35106895, 25.05583962, 19.649562  , 32.52076056])

Let's visualize the training set.

```python
sns.set_style("white")
f = plt.figure(figsize=(8, 8))
sns.set_context("notebook", font_scale=1.5, rc={"lines.linewidth": 2.5})

plt.plot(X_train, y_train, "b.")
plt.title("Data Points")
plt.grid(True)
plt.xlabel("$x_1$", fontsize=18)
plt.ylabel("$y$", rotation=0, fontsize=18)
plt.axis([0, 10, 0, 40])

plt.show()
```

We have a training set consisting a single feature so we will fit a simple linear regression model with one feature.  It's form is: $y = w_0 + w_1 x_1$.

As discussed in the lectures, we add a special dummy feature $x_0$ and set it to 1.  We create a helper function for that.

```python
def add_dummy_feature(x):
    """Adds a dummry feature to the dataset.

    Args:
      x: Training dataset

    Returns:
      Training dataset with an addition of dummy feature.
    """
    # np.ones(x.shape[0]) create a vector of 1's having the same number of
    # rows as number of samples in dataset.
    return np.column_stack((np.ones(x.shape[0]), x))
```

Let's write a test case to test this function:

For that let's take two examples and three features.  The first example is a feature vector:
\begin{eqnarray}
\mathbf{x}^{(1)}_{3 \times 1} &=&  \begin{bmatrix}
      3 \\
      2 \\
      5 \\
    \end{bmatrix}
\end{eqnarray}

And the second example is:
\begin{eqnarray}
\mathbf{x}^{(2)}_{3 \times 1} &=&  \begin{bmatrix}
      9 \\
      4 \\
      7 \\
    \end{bmatrix}
\end{eqnarray}

And recall that a fecture matrix $\mathbf{X}$ has shape $(n, m+1)$ corresponding to features of all examples.
  \begin{equation}
    \mathbf{X}_{n \times (m+1)} = \begin{bmatrix}
        - \left(\mathbf{x}^{(1)}\right)^T - \\
        - \left(\mathbf{x}^{(2)}\right)^T - \\
        \vdots \\
        - \left(\mathbf{x}^{(n)}\right)^T - \\
      \end{bmatrix}
  \end{equation}

The corresponding feature matrix $\mathbf{X}$ appears as follows:
\begin{eqnarray}
  \mathbf{X}_{2 \times 3} &=&  \begin{bmatrix}
      3 & 2 & 5 \\
      9 & 4 & 7 \\
    \end{bmatrix} \\
\end{eqnarray}

Here the feature vectors are transposed and represented as rows:
* The first row corresponds to the first example $\left(\mathbf{x}^{(1)}\right)^T$ and
* The second row corresponds to the second example $\left(\mathbf{x}^{(2)}\right)^T$.

Once we add the dummy feature, the resulting matrix becomes:
\begin{eqnarray}
  \mathbf{X}_{2\times 4} &=&  \begin{bmatrix}
      1 & 3 & 2 & 5\\
      1 & 9 & 4 & 7\\
    \end{bmatrix}
\end{eqnarray}

```python
import unittest

class TestAddDummyFeature(unittest.TestCase):
    def test_add_dummy_feature(self):
        """Test case function for add_dummy_feature"""
        train_matrix = np.array([[3, 2, 5], [9, 4, 7]])
        train_matrix_with_dummy_feature = add_dummy_feature(train_matrix)

        # test the shape
        self.assertEqual(train_matrix_with_dummy_feature.shape, (2, 4))

        # and contents
        np.testing.assert_array_equal(train_matrix_with_dummy_feature, np.array([[1, 3, 2, 5], [1, 9, 4, 7]]))

unittest.main(argv=[""], defaultTest="TestAddDummyFeature", verbosity=2, exit=False)
```

    test_add_dummy_feature (__main__.TestAddDummyFeature)
    Test case function for add_dummy_feature ... ok

    ----------------------------------------------------------------------
    Ran 1 test in 0.004s

    OK

    <unittest.main.TestProgram at 0x7f6b64220290>

```python
add_dummy_feature(np.array([[3, 2], [5, 4]]))
```

    array([[1., 3., 2.],
           [1., 5., 4.]])

Let's preprocess the training set to add the dummy feature

```python
print("Before adding the dummy feature:\n", X_train[:5])
print("\n")

X_train_with_dummy = add_dummy_feature(X_train)

print("After adding the dummy feature:\n", X_train_with_dummy[:5, :])
```

    Before adding the dummy feature:
     [2.66264564 5.2778843  7.14629389 5.3417576  9.21518806]

    After adding the dummy feature:
     [[1.         2.66264564]
     [1.         5.2778843 ]
     [1.         7.14629389]
     [1.         5.3417576 ]
     [1.         9.21518806]]

# C2. Model

Linear regression model uses linear combination of features to obtain output labels.  In vectorized form, this can be written as $\mathbf{y} = \mathbf{X} \mathbf{w}$

**Note**

* Model is parameterized by its weight vector.
* It is described by its mathematical form and weight vector.

## Implementation

The general vectorized form is as follows:
$$
  \mathbf{y}_{n \times 1} = \mathbf{X}_{n \times (m+1)} \mathbf{w}_{(m+1) \times 1}
$$

where
* $n$ is number of examples in dataset (train/test/validation).
* $m$ is the number of features.
* $\mathbf{X}$ is a feature matrix contain $(m+1)$ features for $n$ examples along rows. (Notice capital case bold **X** used for matrix).
* $\mathbf{w}$ is a weight vector containing $(m+1)$ weights one for each feature. (notice small case bold **w**).
* $\mathbf{y}$ is a label vector containing labels for $n$ examples with shape $(n,)$.

```python
def predict(X, w):
    """Prediction of output label for a given input.

    Args:
      X: Feature matrix of shape (n, m+1).
      w: weight vector of shape (m+1, n)

    Returns:
      y: Predicted label vector of shape (n,).
    """
    # Check to make sure that feature matrix and weight vector have compatible
    # shapes.
    assert X.shape[-1] == w.shape[0], "X and w don't have compatible dimensions"
    return X @ w
```

We test this function with the following feature matrix $\mathbf{X}_{2 \times (3+1)}$:

\begin{eqnarray}
  \mathbf{X}_{2 \times 4} &=&  \begin{bmatrix}
      1 & 3 & 2 & 5 \\
      1 & 9 & 4 & 7 \\
    \end{bmatrix} \\
\end{eqnarray}

and weight vector $\mathbf{w}$
\begin{eqnarray}
  \mathbf{w}_{4 \times 1} &=&  \begin{bmatrix}
      1 \\
      1 \\
      1 \\
      1 \\
    \end{bmatrix} \\
\end{eqnarray}

<!-- In this example, we have three features and hence we use the following linear regression model:
\begin{eqnarray}
  \mathbf{w}: y &=& \sum_{j=0}^{3} w_j x_j \\
    &=& w_0 x_0 + w_1 x_1 + w_2 x_2 + w_3 x_3  = \\
    &=& 1 + 1 x_1 + 1 x_2 + 1 x_3\\
    &=& x_0 + x_1 + x_2 + x_3
\end{eqnarray}

The prediction
* for the first example is: $y_1 = 1 + 3 + 2 + 5 = 11$ and
* for the second example is: $y_2 = 1 + 9 + 4 + 7 = 21$ -->

Let's perform matrix-vector multiplication between the feature matrix $\mathbf{X}$ and a weight vector $\mathbf{w}$ to obtain labels for all examples:
\begin{eqnarray}
\mathbf{y} &=& \mathbf{X} \color{red}{\mathbf{w}} \\
  &=& \begin{bmatrix}
        \color{blue}{1} & \color{blue}{3} & \color{blue}{2} & \color{blue}{5} \\
        \color{purple}{1} & \color{purple}{9} & \color{purple}{4} & \color{purple}{7} \\
      \end{bmatrix}
    \times
    \begin{bmatrix}
      \color{red}{1} \\
      \color{red}{1} \\
      \color{red}{1} \\
      \color{red}{1} \\
    \end{bmatrix} \\
    &=& \begin{bmatrix}
        \color{blue}{1} \times \color{red}{1} + \color{blue}{3} \times \color{red}{1} + \color{blue}{2} \times \color{red}{1} + \color{blue}{5} \times \color{red}{1}  \\
        \color{purple}{1} \times \color{red}{1} + \color{purple}{3} \times \color{red}{1} + \color{purple}{2} \times \color{red}{1} + \color{purple}{5} \times \color{red}{1}
      \end{bmatrix} \\
    &=&
    \begin{bmatrix}
      11 \\
      21
    \end{bmatrix} \\
\end{eqnarray}

```python
import unittest

class TestPredict(unittest.TestCase):
    def test_predict(self):
        """Test case predict function of linear regression"""
        # set up
        train_matrix = np.array([[1, 3, 2, 5], [1, 9, 4, 7]])
        weight_vector = np.array([1, 1, 1, 1])
        expected_label_vector = np.array([11, 21])

        # call
        predicted_label_vector = predict(train_matrix, weight_vector)

        # asserts
        # test the shape
        self.assertEqual(predicted_label_vector.shape, (2,))

        # and contents
        np.testing.assert_array_equal(expected_label_vector, predicted_label_vector)

unittest.main(argv=[""], defaultTest="TestPredict", verbosity=2, exit=False)
```

    test_predict (__main__.TestPredict)
    Test case predict function of linear regression ... ok

    ----------------------------------------------------------------------
    Ran 1 test in 0.005s

    OK

    <unittest.main.TestProgram at 0x7f6b642984d0>

## Demonstration on synthetic dataset

```python
# @title Dataset: $n=100$, $m=1$, $[w_0, w_1] = [4, 3]$
def generate_data(n):
    """Generates a syntic data with $n$ points.

    The training features are generated with random samples. The
    label for each example is generated with the following
    relationship: y = 4 + 3*feature_value + random_noise.

    Args:
      n: Number of data points

    Returns:
      (X, y): pair of feature matrix and label vector.
    """
    w1 = 3
    w0 = 4
    X = 10 * np.random.rand(
        n,
    )

    # Obtain y = 4 + 3*x + noise.  Noise is randomly sampled.
    y = (
        w0
        + w1 * X
        + np.random.randn(
            n,
        )
    )
    return X, y

X, y = generate_data(n=100)
```

```python
# @title Preprocessing: Dummy feature and train-test split
from sklearn.model_selection import train_test_split

def add_dummy_feature(x):
    """Adds a dummy feature to the dataset.

    Args:
      x: Training dataset

    Returns:
      Training dataset with an addition of dummy feature.
    """
    # np.ones(x.shape[0]) create a vector of 1's having the same number of
    # rows as number of samples in dataset.
    return np.column_stack((np.ones(x.shape[0]), x))

def preprocess(X, y):
    """Preprocesses training set and splits it into train/test.

    Args:
      X: Training feature matrix
      y: Training labels

    Returns:
      (X_train, X_test, y_train, y_test)
    """
    X_with_dummy_feature = add_dummy_feature(X)
    return train_test_split(X_with_dummy_feature, y, test_size=0.20, random_state=42)

X_train, X_test, y_train, y_test = preprocess(X, y)
```

Since we have not yet trained our model, let's use a random weight vector to get predictions from our model for the given dataset:

```python
w = np.random.rand(
    2,
)
w
```

    array([0.7544478 , 0.99493283])

```python
y_hat = predict(X_train, w)
```

Let's compare the prediction with the actual value:

```python
y_hat[:10]
```

    array([8.40126361, 5.95166443, 1.87485789, 6.30904185, 7.08430428,
           3.92799696, 8.55891733, 1.96791787, 3.37120937, 7.3224093 ])

Actual labels are

```python
y_train[:10]
```

    array([26.61862727, 21.04746492,  7.96318521, 23.33656661, 24.17463552,
           13.04634202, 27.74065111,  7.14788236, 11.58211279, 23.87376769])

Since we used a random weight vector $\mathbf{w}$ here, most of the predicted labels do not match the actual labels.

## Comparison of vectorized and non-vectorized version of model inference

```python
def non_vectorized_predict(X, w):
    """Prediction of output label for a given input.

    Args:
      X: Feature matrix of shape (n, m+1).
      w: weight vector of shape (m+1, n)

    Returns:
      y: Predicted label vector of shape (n, ).
    """
    y = []
    for i in range(0, X.shape[0]):
        y_hat_i = 0
        for j in range(0, X.shape[1]):
            y_hat_i += X[i][j] * w[j]
        y.append(y_hat_i)
    return np.array(y)
```

Let's write a test for this function with the same set up as the vectorized implementation.

```python
import unittest

class TestPredictNonVectorized(unittest.TestCase):
    def test_predict_non_vectorized(self):
        """Test case predict function of linear regression"""
        # set up
        train_matrix = np.array([[1, 3, 2, 5], [1, 9, 4, 7]])
        weight_vector = np.array([1, 1, 1, 1])
        expected_label_vector = np.array([11, 21])

        # call
        predicted_label_vector = non_vectorized_predict(train_matrix, weight_vector)

        # asserts
        # test the shape
        self.assertEqual(predicted_label_vector.shape, (2,))

        # and contents
        np.testing.assert_array_equal(expected_label_vector, predicted_label_vector)

unittest.main(argv=[""], defaultTest="TestPredictNonVectorized", verbosity=2, exit=False)
```

    test_predict_non_vectorized (__main__.TestPredictNonVectorized)
    Test case predict function of linear regression ... ok

    ----------------------------------------------------------------------
    Ran 1 test in 0.004s

    OK

    <unittest.main.TestProgram at 0x7f6b641d6a50>

Let's compare run time of vectorized and non-vectorized versions on dataset with 100 examples.

```python
import time

start_time = time.time()
y_hat_vectorized = predict(X_train, w)
end_time = time.time()

print("Total time incurred in vectorized inference is: %0.6f s" % (end_time - start_time))

start_time = time.time()
y_hat_non_vectorized = non_vectorized_predict(X_train, w)
end_time = time.time()
print("Total time incurred in non-vectorized inference is: %0.6f s" % (end_time - start_time))

np.testing.assert_array_equal(y_hat_vectorized, y_hat_non_vectorized)
```

    Total time incurred in vectorized inference is: 0.000118 s
    Total time incurred in non-vectorized inference is: 0.000353 s

Let's try to check the difference in their performance on large dataset of 1 million data points.

```python
X, y = generate_data(n=1000000)
X_train, X_test, y_train, y_test = preprocess(X, y)

start_time = time.time()
y_hat_vectorized = predict(X_train, w)
end_time = time.time()
print("Total time incurred in vectorized inference is: %0.6f s" % (end_time - start_time))

start_time = time.time()
y_hat_non_vectorized = non_vectorized_predict(X_train, w)
end_time = time.time()
print("Total time incurred in non-vectorized inference is: %0.6f s" % (end_time - start_time))

np.testing.assert_array_equal(y_hat_vectorized, y_hat_non_vectorized)
```

    Total time incurred in vectorized inference is: 0.002506 s
    Total time incurred in non-vectorized inference is: 1.381545 s

Note that the time required for non-vectorized inference is order of magnitude more than the vectorized inference.

# C3. Loss function implementation

## Implementation

The loss function is calculated as follows:

\begin{equation}
J(\mathbf{w}) = \frac{1}{2} \left(\mathbf{X}\mathbf{w} - \mathbf{y} \right)^T \left(\mathbf{X}\mathbf{w} - \mathbf{y} \right)
\end{equation}

where
* $\mathbf{X}$ is a feature matrix contain $(m+1)$ features for $n$ examples along rows.
* $\mathbf{w}$ is a weight vector containing $(m+1)$ weights one for each feature.
* $\mathbf{y}$ is a label vector containing labels for $n$ examples in a vector of shape $(n,)$.

```python
def loss(X, y, w):
    """Calculates loss for a model based on known labels.

    Args:
      X: Feature matrix for given inputs.
      y: Output label vector as predicted by the given model.
      w: Weight vector

    Returns:
      Loss
    """
    e = predict(X, w) - y
    return (1 / 2) * (np.transpose(e) @ e)
```

We will test this function with the following configuration:

1. Feature matrix ($\mathbf{X}$):
\begin{eqnarray}
  \mathbf{X}_{2 \times 4} &=&  \begin{bmatrix}
      1 & 3 & 2 & 5 \\
      1 & 9 & 4 & 7 \\
    \end{bmatrix} \\
\end{eqnarray}

2. Weight vector ($\mathbf{w}$):
\begin{eqnarray}
  \mathbf{w}_{4 \times 1} &=&  \begin{bmatrix}
      1 \\
      1 \\
      1 \\
      1 \\
    \end{bmatrix} \\
\end{eqnarray}

3. Label vector $\mathbf{y}$:
  \begin{eqnarray}
  \mathbf{y}_{2 \times 1} &=&  \begin{bmatrix}
      6 \\
      11
    \end{bmatrix} \\
   \end{eqnarray}

Let's compute the loss $J(\mathbf{w})$ i.e.
\begin{eqnarray}
  J\left(\begin{bmatrix}
      1 \\
      1 \\
      1 \\
      1 \\
    \end{bmatrix} \right) &=& \frac{1}{2} \left(\mathbf{X}\mathbf{w} - \mathbf{y} \right)^T \left(\mathbf{X}\mathbf{w} - \mathbf{y} \right)  \\
     &=& \frac{1}{2} \left( \begin{bmatrix}
        \color{blue}{1} & \color{blue}{3} & \color{blue}{2} & \color{blue}{5} \\
        \color{purple}{1} & \color{purple}{9} & \color{purple}{4} & \color{purple}{7} \\
      \end{bmatrix}
    \times
    \begin{bmatrix}
      \color{red}{1} \\
      \color{red}{1} \\
      \color{red}{1} \\
      \color{red}{1}
    \end{bmatrix} - \begin{bmatrix}
      6 \\
      11
    \end{bmatrix}
      \right)^T  \left( \begin{bmatrix}
        \color{blue}{1} & \color{blue}{3} & \color{blue}{2} & \color{blue}{5} \\
        \color{purple}{1} & \color{purple}{9} & \color{purple}{4} & \color{purple}{7} \\
      \end{bmatrix}
    \times
    \begin{bmatrix}
      \color{red}{1} \\
      \color{red}{1} \\
      \color{red}{1} \\
      \color{red}{1}
    \end{bmatrix} - \begin{bmatrix}
      6 \\
      11
    \end{bmatrix} \right) \\
    &=& \frac{1}{2} \left(
      \begin{bmatrix}
      11 \\
      21
    \end{bmatrix} -
    \begin{bmatrix}
      6 \\
      11
    \end{bmatrix}
      \right)^T
      \left(
      \begin{bmatrix}
      11 \\
      21
    \end{bmatrix} -
    \begin{bmatrix}
      6 \\
      11
    \end{bmatrix}
      \right)\\
    &=& \frac{1}{2}
      \left(
      \begin{bmatrix}
      5 \\
      10
    \end{bmatrix}
      \right)^T \left(
      \begin{bmatrix}
      5 \\
      10
    \end{bmatrix}
      \right) \\
   &=& \frac{1}{2}
      \left(
      \begin{bmatrix}
      5 & 10
    \end{bmatrix}
    \begin{bmatrix}
      5 \\
      10
    \end{bmatrix}
      \right) \\
  &=& \frac{1}{2}
      \left(
      \begin{bmatrix}
      5 \times 5 + 10 \times 10
    \end{bmatrix}
      \right) \\
  &=& \frac{1}{2} [125] = [62.5]
\end{eqnarray}

```python
import unittest

class TestLossFunction(unittest.TestCase):
    def test_loss_function(self):
        """Test case for loss function of linear regression"""
        # set up
        feature_matrix = np.array([[1, 3, 2, 5], [1, 9, 4, 7]])
        weight_vector = np.array([1, 1, 1, 1])
        label_vector = np.array([6, 11])
        expected_loss = np.array([62.5])

        # call
        loss_value = loss(feature_matrix, label_vector, weight_vector)

        # asserts
        # test the shape
        self.assertEqual(loss_value.shape, ())

        # and contents
        np.testing.assert_array_equal(expected_loss, loss_value)

unittest.main(argv=[""], defaultTest="TestLossFunction", verbosity=2, exit=False)
```

    test_loss_function (__main__.TestLossFunction)
    Test case for loss function of linear regression ... ok

    ----------------------------------------------------------------------
    Ran 1 test in 0.003s

    OK

    <unittest.main.TestProgram at 0x7f6b64298fd0>

Since we have not yet trained our model, let's use a random weight vector to calculate loss for linear regression model with single feature on synthetic dataset.

```python
w = np.random.rand(
    2,
)
```

## Demonstration on synthetic dataset

```python
# @title [Generate training data]
def generate_data(n):
    """Generates a syntic data with $n$ points.

    The training features are generated with random samples. The
    label for each example is generated with the following
    relationship: y = 4 + 3*feature_value + random_noise.

    Args:
      n: Number of data points

    Returns:
      (X, y): pair of feature matrix and label vector.
    """
    w1 = 3
    w0 = 4
    X = 10 * np.random.rand(
        n,
    )

    # Obtain y = 4 + 3*x + noise.  Noise is randomly sampled.
    y = (
        w0
        + w1 * X
        + np.random.randn(
            n,
        )
    )
    return X, y
```

```python
# @title [Preprocessing: Dummy feature and train-test split]
from sklearn.model_selection import train_test_split

def add_dummy_feature(x):
    """Adds a dummy feature to the dataset.

    Args:
      x: Training dataset

    Returns:
      Training dataset with an addition of dummy feature.
    """
    # np.ones(x.shape[0]) create a vector of 1's having the same number of
    # rows as number of samples in dataset.
    return np.column_stack((np.ones(x.shape[0]), x))

def preprocess(X, y):
    """Preprocesses training set and splits it into train/test.

    Args:
      X: Training feature matrix
      y: Training labels

    Returns:
      (X_train, X_test, y_train, y_test)
    """
    X_with_dummy_feature = add_dummy_feature(X)
    return train_test_split(X_with_dummy_feature, y, test_size=0.20, random_state=42)
```

```python
X, y = generate_data(100)  # y = 4 + 3 x_1 + noise
X_train, X_test, y_train, y_test = preprocess(X, y)
```

```python
# @title [Visualize error for each data point]

def visualize_loss_for_single_feature_model(X, y, w):
    """Plots loss for the model for a given dataset with single feature.

    Args:
      X: Feature matrix for given inputs.
      y: Output label vector as predicted by the given model.
      w: Weight vector

    Returns:
      None
    """
    if X.shape[1] > 2:
        return

    sns.set_style("white")

    # f = plt.figure(figsize=(8, 8))
    sns.set_context("notebook", font_scale=1.5, rc={"lines.linewidth": 2.5})

    plt.plot(X[:, 1], predict(X, w), "b-", ms=12)
    plt.plot(X[:, 1], y, "g.", ms=12)

    for i in range(0, 80):
        plt.plot((X[i][1], X[i][1]), (predict(X[i], w), y[i]), "r-", ms=12)

    plt.xlabel("$x_1$")
    plt.ylabel("$y$")
    plt.title("Loss or Error")
    plt.grid(True)
```

```python
visualize_loss_for_single_feature_model(X_train, y_train, w)
```

Let's try with the actual weight vector used for data generation process and examine the loss.

```python
visualize_loss_for_single_feature_model(X_train, y_train, np.array([4, 3]))
```

# C4: Optimization

Let's implement optimization component of linear regression model.

It is implemented with one of the following two methods:
* Normal equation method, that sets the partial derivative of the loss function w.r.t. weight vector to 0 and solves the resulting equation to obtain the weight vector.
* Gradient descent method, that iteratively adjusts the weight vector based on the learning rate and the gradient of loss function at the current weight vector.

Let's generate training data with 100 examples with linear regression model
of known parameters.

We will use this data to test optimization procedure, where we compare the estimated weight vectors with the actual weight vector.

```python
X, y = generate_data(100)
X_train, X_test, y_train, y_test = preprocess(X, y)
```

## Normal equation

The weight vector is estimated by matrix multiplication of psuedo-inverse of feature matrix and the label vector.

The vectorized implementation is fairly straight forward.
*  We make use of `np.linalg.pinv` for calculating psuedoinverse of the feature matrix.

```python
def normal_equation(X, y):
    """Estimates parameters of the linear regression model with normal equation.

    Args:
      X: Feature matrix for given inputs.
      y: Actual label vector.

    Returns:
      Weight vector
    """
    return np.linalg.pinv(X) @ y
```

We test this function with the generated training set whose weight vector is known to us.

* We set up the test with feature matrix, label vector and the expected weight vectors.
* Next we estimate the weight vector with `normal_equation` function.
* We test (a) shape and (ii) match between expected and estimated weight vectors.

```python
import unittest

class TestNormalEquation(unittest.TestCase):
    def test_normal_equation(self):
        """
        Test case for weight estimation for linear regression with normal
         equation method.'
        """
        # set up
        feature_matrix = X_train
        label_vector = y_train
        expected_weight_vector = np.array([4.0, 3.0])

        # call
        estimated_weight_vector = normal_equation(feature_matrix, label_vector)

        # asserts
        # test the shape
        self.assertEqual(estimated_weight_vector.shape, (2,))

        # and contents
        np.testing.assert_array_almost_equal(estimated_weight_vector, expected_weight_vector, decimal=0)

unittest.main(argv=[""], defaultTest="TestNormalEquation", verbosity=2, exit=False)
```

    test_normal_equation (__main__.TestNormalEquation)
    Test case for weight estimation for linear regression with normal ... ok

    ----------------------------------------------------------------------
    Ran 1 test in 0.021s

    OK

    <unittest.main.TestProgram at 0x7f6b6176a5d0>

## Gradient descent (GD)

GD is implemented as follows:

* Randomly initialize $\mathbf{w}$ to $\mathbf{0}$.
* Iterate until convergence:
  * Calculate partial derivative of loss w.r.t. weight vector.
  * Calculate new values of weights.
  * Update weights to new values *simultaneously*.

We use number of epochs as a convergence criteria in this implementation.

### Partial derivative of loss function

Let's first implement a function to calculate partial derivative of loss function, which is obtained with the following equation:
$$
\frac{\partial}{\partial \mathbf{w}} J(\mathbf{w}) = \mathbf{X}^T \left(\mathbf{X}\mathbf{w} - \mathbf{y} \right)
$$

The multiplication of transpose of feature matrix with the difference of predicted and actual label vectors.

```python
def calculate_gradient(X, y, w):
    """Calculates gradients of loss function w.r.t weight vector on training set.

    Arguments:
        X: Feature matrix for training data.
        y: Label vector for training data.
        w: Weight vector

    Returns:
      A vector of gradients.
    """
    return np.transpose(X) @ (predict(X, w) - y)
```

Let's write a test case for gradient calculation with the following setup:

1. Feature matrix ($\mathbf{X}$):
\begin{eqnarray}
  \mathbf{X}_{2 \times 4} &=&  \begin{bmatrix}
      1 & 3 & 2 & 5 \\
      1 & 9 & 4 & 7 \\
    \end{bmatrix} \\
\end{eqnarray}

2. Weight vector ($\mathbf{w}$):
\begin{eqnarray}
  \mathbf{w}_{4 \times 1} &=&  \begin{bmatrix}
      1 \\
      1 \\
      1 \\
      1 \\
    \end{bmatrix} \\
\end{eqnarray}

3. Label vector $\mathbf{y}$:
  \begin{eqnarray}
  \mathbf{y}_{2 \times 1} &=&  \begin{bmatrix}
      6 \\
      11
    \end{bmatrix} \\
   \end{eqnarray}

Let's compute the partial derivative of loss $J(\mathbf{w})$ i.e.
\begin{eqnarray}
  \frac{\partial}{\partial \mathbf{w}} J(\mathbf{w}) &=& \mathbf{X}^T \left(\mathbf{X}\mathbf{w} - \mathbf{y} \right)  \\
     &=&  \begin{bmatrix}
      1 & 1 \\
      3 & 9 \\
      2 & 4 \\
      5 & 7 \\
    \end{bmatrix}
     \left( \begin{bmatrix}
        \color{blue}{1} & \color{blue}{3} & \color{blue}{2} & \color{blue}{5} \\
        \color{purple}{1} & \color{purple}{9} & \color{purple}{4} & \color{purple}{7} \\
      \end{bmatrix}
    \times
    \begin{bmatrix}
      \color{red}{1} \\
      \color{red}{1} \\
      \color{red}{1} \\
      \color{red}{1}
    \end{bmatrix} - \begin{bmatrix}
      6 \\
      11
    \end{bmatrix}
      \right) \\
    &=& \begin{bmatrix}
      1 & 1 \\
      3 & 9 \\
      2 & 4 \\
      5 & 7 \\
    \end{bmatrix} \left(
      \begin{bmatrix}
      11 \\
      21
    \end{bmatrix} -
    \begin{bmatrix}
      6 \\
      11
    \end{bmatrix}
      \right) \\
    &=& \begin{bmatrix}
      1 & 1 \\
      3 & 9 \\
      2 & 4 \\
      5 & 7 \\
    \end{bmatrix}
      \left(
      \begin{bmatrix}
      5 \\
      10
    \end{bmatrix}
      \right)  \\
  &=&
      \begin{bmatrix}
      1 \times 5 + 1 \times 10 \\
      3 \times 5 + 9 \times 10 \\
      2 \times 5 + 4 \times 10 \\
      5 \times 5 + 7 \times 10 \\
    \end{bmatrix} \\
  &=&
      \begin{bmatrix}
      15 \\
      105 \\
      50 \\
      95 \\
    \end{bmatrix} \\
\end{eqnarray}

```python
class TestCalculateGradient(unittest.TestCase):
    def test_calculate_gradient(self):
        """
        Test case for gradient calculation.'
        """
        # set up
        feature_matrix = np.array([[1, 3, 2, 5], [1, 9, 4, 7]])
        weight_vector = np.array([1, 1, 1, 1])
        label_vector = np.array([6, 11])
        expected_grad = np.array([15, 105, 50, 95])

        # call
        grad = calculate_gradient(feature_matrix, label_vector, weight_vector)

        # asserts
        # test the shape
        self.assertEqual(grad.shape, (4,))

        # and contents
        np.testing.assert_array_almost_equal(expected_grad, grad, decimal=0)

unittest.main(argv=[""], defaultTest="TestCalculateGradient", verbosity=2, exit=False)
```

    test_calculate_gradient (__main__.TestCalculateGradient)
    Test case for gradient calculation.' ... ok

    ----------------------------------------------------------------------
    Ran 1 test in 0.004s

    OK

    <unittest.main.TestProgram at 0x7f6b6193d4d0>

### Weight updates

Next let's implement the weight update part:
* We obtain the new weight from the old one by substracting gradient weighted by the learning rate.

```python
def update_weights(w, grad, lr):
    """Updates the weights based on the gradient of loss function.

    Weight updates are carried out with the following formula:
        w_new := w_old - lr * grad

    Args:
      1. w: weight vector
      2. grad: gradient of loss w.r.t. w
      3. lr: learning rate

    Returns:
      Updated weight vector
    """
    return w - lr * grad
```

Let's workout a weight update for:

1. Weight vector ($\mathbf{w}$):
\begin{eqnarray}
  \mathbf{w}_{4 \times 1} &=&  \begin{bmatrix}
      1 \\
      1 \\
      1 \\
      1 \\
    \end{bmatrix} \\
\end{eqnarray}

2. Grad vector $\mathbf{g}$:
  \begin{eqnarray}
  {\mathbf{g}}_{4 \times 1} &=&  \begin{bmatrix}
      15 \\
      105 \\
      50 \\
      95
    \end{bmatrix} \\
   \end{eqnarray}

3. Learning rate = 0.001

The updated weights are given by:\begin{eqnarray}
  \mathbf{w} &:=& \mathbf{w}^\mathrm{(old)} - \alpha \ \mathbf{g} \\
     &=&  \begin{bmatrix}
      1 \\
      1 \\
      1 \\
      1 \\
    \end{bmatrix} - 0.001 \times
    \begin{bmatrix}
      15 \\
      105 \\
      50 \\
      95 \\
      \end{bmatrix}\\
    &=& \begin{bmatrix}
      1 - 0.001 \times 15  \\
      1 - 0.001 \times 105 \\
      1 - 0.001 \times 50  \\
      1 - 0.001 \times 95  \\
    \end{bmatrix} \\
    &=& \begin{bmatrix}
      1 - 0.015 \\
      1 - 0.105 \\
      1 - 0.05 \\
      1 - 0.095 \\
    \end{bmatrix}\\
    &=&
      \begin{bmatrix}
      0.985 \\
      0.895 \\
      0.95 \\
      0.905 \\
    \end{bmatrix} \\
\end{eqnarray}

```python
class TestUpdateWeights(unittest.TestCase):
    def test_update_weights(self):
        """
        Test case for weight update in GD.'
        """
        # set up
        weight_vector = np.array([1, 1, 1, 1])
        grad_vector = np.array([15, 105, 50, 95])
        lr = 0.001
        expected_w_new = np.array([0.985, 0.895, 0.95, 0.905])

        # call
        w_new = update_weights(weight_vector, grad_vector, lr)

        # asserts
        # test the shape
        self.assertEqual(expected_w_new.shape, (4,))

        # and contents
        np.testing.assert_array_almost_equal(expected_w_new, w_new, decimal=1)

unittest.main(argv=[""], defaultTest="TestUpdateWeights", verbosity=2, exit=False)
```

    test_update_weights (__main__.TestUpdateWeights)
    Test case for weight update in GD.' ... ok

    ----------------------------------------------------------------------
    Ran 1 test in 0.005s

    OK

    <unittest.main.TestProgram at 0x7f6b61764310>

With these building blocks in place, let's implement gradient descent procedure:

```python
def gradient_descent(X: np.ndarray, y: np.ndarray, lr: float, num_epochs: int):
    """Estimates parameters of linear regression model through gradient descent.

    Arguments:
      X: Feature matrix for training data.
      y: Label vector for training data.
      lr: learning rate
      num_epochs: Number of training steps

    Returns:
      Weight vector: Final weight vector
      Error vector across different iterations
      Weight vectors across different iterations
    """
    w_all = []  # all parameters across iterations.
    err_all = []  # all errors across iterations.

    # Parameter vector initialized to [0, 0]
    w = np.zeros((X.shape[1]))

    # Gradient descent loop
    print()
    for i in np.arange(0, num_epochs):
        w_all.append(w)

        # Calculate error due to the current weigh vector: Note that here we use
        # loss function to calculate the loss.
        err_all.append(loss(X, y, w))

        # Gradient calculation
        dJdW = calculate_gradient(X, y, w)

        # Print stats every 100 iterations
        if (i % 100) == 0:
            print("Iteration #: %d, loss: %4.2f" % (i, err_all[-1]))

        # Weight vector update.
        w = update_weights(w, dJdW, lr)

    return w, err_all, w_all
```

In order to test this function, we will use the synthetic training data that was generated earlier.  We know the actual weights, that can be compared against the weights obtained from gradient descent procedure.

```python
class TestGradientDescent(unittest.TestCase):
    def test_gradient_descent(self):
        """
        Test case for weight estimation for linear regression with
        gradient descent method.'
        """
        # set up
        feature_matrix = X_train
        label_vector = y_train
        expected_weights = np.array([4.0, 3.0])

        # call
        w, err_all, w_all = gradient_descent(feature_matrix, label_vector, lr=0.0001, num_epochs=2000)

        # asserts
        # test the shape
        self.assertEqual(w.shape, (2,))

        # and contents
        np.testing.assert_array_almost_equal(expected_weights, w, decimal=0)

unittest.main(argv=[""], defaultTest="TestGradientDescent", verbosity=2, exit=False)
```

    test_gradient_descent (__main__.TestGradientDescent)
    Test case for weight estimation for linear regression with ...

    Iteration #: 0, loss: 20200.29
    Iteration #: 100, loss: 125.33
    Iteration #: 200, loss: 103.74
    Iteration #: 300, loss: 87.76
    Iteration #: 400, loss: 75.94
    Iteration #: 500, loss: 67.20
    Iteration #: 600, loss: 60.73
    Iteration #: 700, loss: 55.94
    Iteration #: 800, loss: 52.40
    Iteration #: 900, loss: 49.78
    Iteration #: 1000, loss: 47.84
    Iteration #: 1100, loss: 46.41
    Iteration #: 1200, loss: 45.35
    Iteration #: 1300, loss: 44.56
    Iteration #: 1400, loss: 43.98
    Iteration #: 1500, loss: 43.55
    Iteration #: 1600, loss: 43.23
    Iteration #: 1700, loss: 43.00
    Iteration #: 1800, loss: 42.83
    Iteration #: 1900, loss: 42.70

    ok

    ----------------------------------------------------------------------
    Ran 1 test in 0.043s

    OK

    <unittest.main.TestProgram at 0x7f6b61770fd0>

Since we store weights in each iteration, we can use them to plot intermediate models in order to understand the trajectory taken by GD.

In the following code:
* `err_all` contains loss values across all iterations and
* `w_all` contains weight vectors across all iterations.

```python
w, err_all, w_all = gradient_descent(X_train, y_train, lr=0.00001, num_epochs=200)
```

    Iteration #: 0, loss: 20200.29
    Iteration #: 100, loss: 178.86

```python
# @title [Plot model trajectory]
X_b = np.c_[np.ones((X.shape[0], 1)), X]  # add x0 = 1 to each instance
X_new = np.array([[0], [10]])
X_new_b = np.c_[np.ones((2, 1)), X_new]  # add x0 = 1 to each instance

for j in range(0, len(w_all)):
    if j % 10 != 0:
        continue
    y_hat = predict(X_new_b, w_all[j])
    style = "b-" if j > 0 else "r--"
    plt.plot(X_new_b[:, 1], y_hat, style)

plt.plot(X_train[:, 1], y_train, "b.")
plt.xlabel("$x_1$", fontsize=18)
plt.ylabel("$y$", rotation=0, fontsize=18)
plt.title("Gradient Descent", fontsize=18)
plt.axis([0, 10, 0, 40])
plt.show()
```

Let's plot the learning curves.

```python
def plot_learning_curves(err_all):
    plt.plot(err_all)
    plt.xlabel("iteration #")
    plt.ylabel("Loss: $J(\mathbf{w})$")
```

```python
w, err_all, w_all = gradient_descent(X_train, y_train, lr=0.0001, num_epochs=2000)
plot_learning_curves(err_all)
```

## Learning rate and convergence

Let's vary the learning rate and observe change in the convergence characteristics of GD.

We will use:
* $\alpha=\{1e-6, 1e-4, 1e-1\}$ to run GD for 2000 epochs each.
* Compare the convergence characteristics.

```python
w, err_all, w_all = gradient_descent(X_train, y_train, lr=1e-6, num_epochs=2000)
plot_learning_curves(err_all)
plt.title("lr=1e-6")
```

```python
w, err_all, w_all = gradient_descent(X_train, y_train, lr=1e-4, num_epochs=2000)
plot_learning_curves(err_all)

plt.title("lr=1e-4")
```

Let's try learning rate $\alpha=1e-1$ (Doesn't converge)

```python
w, err_all, w_all = gradient_descent(X_train, y_train, lr=1e-1, num_epochs=2000)
plot_learning_curves(err_all)
plt.title("lr=0.1")
```

# Linear regression: combining all components

Let's combine all components that we implemented in last few sections in a single place and make it available for training linear regression model on different datasets.

```python
from IPython.display import display, Math, Latex  # Imported for proper rendering of latex in colab.

import numpy as np

# Import for generating plots
import matplotlib.pyplot as plt
import seaborn as sns

%matplotlib inline
```

Let's combine implementation of different components of Linear Regression that we implemented so far into a single `LinearRegression` class.

```python
class LinReg(object):
    """
    Linear regression model
    -----------------------
    y = X@w
    X: A feature matrix
    w: weight vector
    y: label vector
    """

    def __init__(self):
        self.t0 = 200
        self.t1 = 100000

    def predict(self, X: np.ndarray) -> np.ndarray:
        """Prediction of output label for a given input.

        Args:
          X: Feature matrix for given inputs.

        Returns:
          y: Output label vector as predicted by the given model.
        """
        y = X @ self.w
        return y

    def loss(self, X: np.ndarray, y: np.ndarray) -> float:
        """Calculates loss for a model based on known labels.

        Args:
          X: Feature matrix for given inputs.
          y: Output label vector as predicted by the given model.

        Returns:
          Loss
        """
        e = y - self.predict(X)
        return (1 / 2) * (np.transpose(e) @ e)

    def rmse(self, X: np.ndarray, y: np.ndarray) -> float:
        """Calculates root mean squared error of prediction w.r.t. actual label.

        Args:
          X: Feature matrix for given inputs.
          y: Output label vector as predicted by the given model.

        Returns:
          Loss
        """
        return np.sqrt((2 / X.shape[0]) * self.loss(X, y))

    def fit(self, X: np.ndarray, y: np.ndarray) -> np.ndarray:
        """Estimates parameters of the linear regression model with normal equation.

        Args:
          X: Feature matrix for given inputs.
          y: Output label vector as predicted by the given model.

        Returns:
          Weight vector
        """
        self.w = np.linalg.pinv(X) @ y
        return self.w

    def calculate_gradient(self, X: np.ndarray, y: np.ndarray) -> np.ndarray:
        """Calculates gradients of loss function w.r.t weight vector on training set.

        Arguments:
            X: Feature matrix for training data.
            y: Label vector for training data.

        Returns:
          A vector of gradients.
        """
        return np.transpose(X) @ (self.predict(X) - y)

    def update_weights(self, grad: np.ndarray, lr: float) -> np.ndarray:
        """Updates the weights based on the gradient of loss function.

        Weight updates are carried out with the following formula:
            w_new := w_old - lr * grad

        Args:
          2. grad: gradient of loss w.r.t. w
          3. lr: learning rate

        Returns:
          Updated weight vector
        """
        return self.w - lr * grad

    def learning_schedule(self, t):
        return self.t0 / (t + self.t1)

    def gd(self, X: np.ndarray, y: np.ndarray, num_epochs: int, lr: float) -> np.ndarray:
        """Estimates parameters of linear regression model through gradient descent.

        Args:
          X: Feature matrix for training data.
          y: Label vector for training data.
          num_epochs: Number of training steps
          lr: Learning rate

        Returns:
          Weight vector: Final weight vector
        """
        self.w = np.zeros((X.shape[1]))
        self.w_all = []
        self.err_all = []
        for i in np.arange(0, num_epochs):
            dJdW = self.calculate_gradient(X, y)
            self.w_all.append(self.w)
            self.err_all.append(self.loss(X, y))
            self.w = self.update_weights(dJdW, lr)
        return self.w

    def mbgd(self, X: np.ndarray, y: np.ndarray, num_epochs: int, batch_size: int) -> np.ndarray:
        """Estimates parameters of linear regression model through gradient descent.

        Args:
          X: Feature matrix for training data.
          y: Label vector for training data.
          num_epochs: Number of training steps
          batch_size: Number of examples in a batch

        Returns:
          Weight vector: Final weight vector
        """
        self.w = np.zeros((X.shape[1]))
        self.w_all = []
        self.err_all = []
        mini_batch_id = 0

        for epoch in range(num_epochs):
            shuffled_indices = np.random.permutation(X.shape[0])
            X_shuffled = X[shuffled_indices]
            y_shuffled = y[shuffled_indices]
            for i in range(0, X.shape[0], batch_size):
                mini_batch_id += 1
                xi = X_shuffled[i : i + batch_size]
                yi = y_shuffled[i : i + batch_size]

                self.w_all.append(self.w)
                self.err_all.append(self.loss(xi, yi))

                dJdW = 2 / batch_size * self.calculate_gradient(xi, yi)
                self.w = self.update_weights(dJdW, self.learning_schedule(mini_batch_id))
        return self.w

    def sgd(self, X: np.ndarray, y: np.ndarray, num_epochs: int) -> np.ndarray:
        """Estimates parameters of linear regression model through gradient descent.

        Args:
          X: Feature matrix for training data.
          y: Label vector for training data.
          num_epochs: Number of training steps
          batch_size: Number of examples in a batch

        Returns:
          Weight vector: Final weight vector
        """
        # Parameter vector initialized to [0, 0]
        self.w = np.zeros((X.shape[1]))
        self.w_all = []
        self.err_all = []
        for epoch in range(num_epochs):
            for i in range(X.shape[0]):
                random_index = np.random.randint(X.shape[0])
                xi = X[random_index : random_index + 1]
                yi = y[random_index : random_index + 1]

                self.w_all.append(self.w)
                self.err_all.append(self.loss(xi, yi))

                gradients = 2 * self.calculate_gradient(xi, yi)
                lr = self.learning_schedule(epoch * X.shape[0] + i)
                self.w = self.update_weights(gradients, lr)

        return self.w
```

```python
X, y = generate_data(100)
X_train, X_test, y_train, y_test = preprocess(X, y)
```

Let's compare weight vectors obtained by normal equation, GD, MBGD and SGD:

```python
# normal equation
lin_reg = LinReg()
lin_reg.fit(X_train, y_train)
print("Weight vector (normal equation): ", lin_reg.w)
```

    Weight vector (normal equation):  [4.40391715 2.92509303]

```python
lin_reg.gd(X_train, y_train, 1000, lr=1e-4)
print("Weight vector (gradient descent): ", lin_reg.w)
```

    Weight vector (gradient descent):  [3.69922564 3.03252769]

```python
lin_reg.mbgd(X_train, y_train, 1000, 16)
print("Weight vector (mini-batch gradient descent): ", lin_reg.w)
```

    Weight vector (mini-batch gradient descent):  [4.34252109 2.93310427]

```python
lin_reg.sgd(X_train, y_train, 1000)
print("Weight vector (sgd): ", lin_reg.w)
```

    Weight vector (sgd):  [4.43994181 2.90445985]

## Linear regression on multiple features and single label

```python
# @title Plot learning curves.
def plot_learning_curves(err_all):
    plt.plot(err_all)
    plt.xlabel("iteration #")
    plt.ylabel("Loss: $J(\mathbf{w})$")
```

```python
from sklearn.datasets import make_regression

X, y, coef = make_regression(
    n_samples=200, n_features=10, n_informative=10, n_targets=1, shuffle=True, coef=True, noise=0.5, random_state=0
)
```

```python
print("Shape of feature matrix: ", X.shape)
print("Shape of label vector: ", y.shape)
print("Shape of coef vector: ", coef.shape)
```

    Shape of feature matrix:  (200, 10)
    Shape of label vector:  (200,)
    Shape of coef vector:  (10,)

```python
print("Weight vector used for data generation:", coef)
```

    Weight vector used for data generation: [40.05104636 10.32532207 51.90989393  6.18447832 41.09157343 28.46940664
     88.97621358  9.68032193 15.48694157 44.96944303]

We will compare it with weight vector estimated from different methods.

```python
X_train, X_test, y_train, y_test = preprocess(X, y)
```

```python
# Normal equation for weight vector estimation
lin_reg.fit(X_train, y_train)
print("Weight vector (normal equation): ", lin_reg.w)
```

    Weight vector (normal equation):  [4.30868690e-03 4.00335228e+01 1.03086790e+01 5.19091277e+01
     6.29286567e+00 4.11625781e+01 2.84422862e+01 8.89543306e+01
     9.75752481e+00 1.54723974e+01 4.49980952e+01]

```python
# check if these coefficients are close to the ones used for data generation.
np.testing.assert_almost_equal(coef, lin_reg.w[1:], decimal=0)
```

```python
# GD for weight vector estimation
lin_reg.gd(X_train, y_train, 1000, lr=1e-4)
print("Weight vector (gradient descent): ", lin_reg.w)
```

    Weight vector (gradient descent):  [4.23513204e-03 4.00335497e+01 1.03081869e+01 5.19092201e+01
     6.29241093e+00 4.11619488e+01 2.84422549e+01 8.89543222e+01
     9.75811687e+00 1.54722408e+01 4.49977615e+01]

```python
plot_learning_curves(lin_reg.err_all[:1000])
```

```python
# check if these coefficients are close to the ones used for data generation.
np.testing.assert_almost_equal(coef, lin_reg.w[1:], decimal=1)
```

```python
# MBGD for weight vector estimation
lin_reg.mbgd(X_train, y_train, 1000, 16)
print("Weight vector (mini-batch gradient descent): ", lin_reg.w)
```

    Weight vector (mini-batch gradient descent):  [4.32776422e-03 4.00334261e+01 1.03087000e+01 5.19090812e+01
     6.29292578e+00 4.11625561e+01 2.84423184e+01 8.89544085e+01
     9.75751021e+00 1.54725483e+01 4.49981559e+01]

```python
plot_learning_curves(lin_reg.err_all[:1000])
```

```python
# check if these coefficients are close to the ones used for data generation.
np.testing.assert_almost_equal(coef, lin_reg.w[1:], decimal=0)
```

```python
# SGD for weight vector estimation
lin_reg.sgd(X_train, y_train, 1000)
print("Weight vector (sgd): ", lin_reg.w)
```

    Weight vector (sgd):  [-1.76539732e-02  4.00182258e+01  1.03026464e+01  5.19197207e+01
      6.29637150e+00  4.11677078e+01  2.84447965e+01  8.89409755e+01
      9.73540047e+00  1.54983459e+01  4.49941318e+01]

```python
plot_learning_curves(lin_reg.err_all[:1000])
```

```python
# check if these coefficients are close to the ones used for data generation.
np.testing.assert_almost_equal(coef, lin_reg.w[1:], decimal=0)
```


---
author: Nathan Carter (ncarter@bentley.edu)
---

This answer assumes you have imported SymPy as follows.

```python
from sympy import *                   # load all math functions
init_printing( use_latex='mathjax' )  # use pretty math output
```

Let's consider the example of the unit circle, $x^2+y^2=1$.

To plot it, SymPy first expects us to move everything to the left-hand side
of the equation, so in this case, we would have $x^2+y^2-1=0$.

We then use that left hand side to represent the equation as a single formula,
and we can plot it with SymPy's `plot_implicit` function.

```python
var( 'x y' )
formula = x**2 + y**2 - 1   # to represent x^2+y^2=1
plot_implicit( formula )
```

database/tasks/How to graph curves that are not functions/Python, using SymPy.ipynb


```python

from sympy import *
init_printing(use_unicode=False, wrap_line=False)
x, i, j = symbols('x i j')
i = 2*x**3 + 3*x -2
j = x**3 + 2*x**2 - x +3
factor(2*x**3 + 3*x -2 )+ (x**3 + 2*x**2 - x +3)

```

```python
factor(x**4/2 + 5*x**3/12 - x**2/3)
```

```python
j = x**2 - 16
factor(j)
```

```python
expand((x-4)**2)
```

```python
g = expand((x-4)**2)
factor(g)

```

```python
from sympy import *

y = x**2 -4*x + 2
a = 1
b = -4
c = 2
h = -(b) /(2*a)
k = c - (b**2) / (4*a)
h
o = y.subs(x, h)

```

```python
h
```

```python
k
```

```python
o

```

```python

```

```python

```

```python

```

```python

```


# Logistic Regression

Logistic regression is a tool that we can use to solve classification problems with arbitrary number of classes. To simplify equations, we focus on binary regression problems.

Solving classification problems with linear regression models have some issues:
- for binary classification, the resulting estimates may lie outside the 0 to 1 interval because the model's outcome is continuously distributed. However, this does not make sense because we want to be able to find the probability that a given data instance belongs to a particular category.
- not useful if the number of classes exceeds 2. If we were to simply code the response as $1,2,\cdots,k$ for a number of classes, then the ordering here would be arbitrary. We could use a binary coding, and performing $k$ separate regressions, then using the strongest prediction at the end, given an input X. This is flawed too, however, as we would likely encounter a problem called **masking**. With this problem, there is some class $j$ that never ends up being predicted in favor of the others, regardless of the input.

Figure 4.2 [#] illustrates the difference between the linear function (left) and the logistic function (right). As our data points can only be one of two values on the $y$-axis, a linear regression model is not a good fit since it dips below zero. The logistic function is a better fit because it never estimates outside the $[0, 1]$ interval

Logistic regression models the probability that $y$ is class 1 given $x$ instead of modelling $y$ as a function $x$ directly like in linear regression. In other words, just like a Linear Regression model, a Logistic Regression first computes a weighted sum of the input features (plus the intercept) but instead of outputting the result directly, it outputs the logistic of this result.  The logistic model is:

\begin{equation}
p(x) = P( y = 1 \mid x) = \frac{\exp(\beta_0 + \beta_1 x)}{1 + \exp(\beta_0 + \beta_1 x)} = \frac{1}{1 + \exp(-\beta_0 - \beta_1 x)}
\end{equation}

We can rewrite the equation above to:

\begin{equation}
\frac{P( y = 1 \mid x)}{1 - P( y = 1 \mid x)} =  exp(\beta_0 + \beta_1 x)
\end{equation}

The left-hand side of the equality is called the *odds*. We get the *log-odds* or *logit* if we take the logarithm of both sides of the equality sign:

\begin{equation}
\log \left( \frac{P( y = 1 \mid x)}{1 - P( y = 1 \mid x)} \right) =  \beta_0 + \beta_1 x
\end{equation}

In general, $logit(a) = \log(a/(1 − a))$

If we have $p$ predictors $x_1, x_2, \cdots, x_p$, we can collect them in a vector $X=(1, x_1, x_2, \cdots, x_p)$. Similarly, the corresponding coefficients can put in a vector $\beta = (\beta_0, \beta_1, \beta_2, \cdots, \beta_p)$. Now, we can construct a linear transformation:

\begin{equation}
X^T \beta =  1 \cdot \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \cdots + \beta_p x_p
\end{equation}

We can write the logistic function as:

\begin{equation}
P( y = 1 \mid X) = \frac{\exp(\beta^T X)}{1 + \exp(\beta^T X)}
\end{equation}

The coefficients $\beta_0$ and $\beta_1$ can be learned using the maximum likelihood method.

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
```

The logistic function is characterised by the logistic (sigmoid or S) curve:

```python
X = np.linspace(-10, 10, 100)
y = 1/ (1 + np.exp(-1*X))
plt.plot(X, y)
```

```python

```


# ***Introduction to Radar Using Python and MATLAB***
## Andy Harrison - Copyright (C) 2019 Artech House
<br/>

# Probability Distributions
***

The probability density function for a Gaussian distribution is given by

\begin{equation}\label{eq:gaussian_pdf}
    p(x) = \frac{1}{\sigma \sqrt{2 \pi}} \exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right),
\end{equation}

where $\mu$ is the mean and $\sigma$ is the standard deviation. This probability density function may be calculated using the SciPy implementation ***scipy.stats.norm.pdf(x, loc, scale)***, where the keyword ***loc*** is the mean and keyword ***scale*** is the standard deviation.

Other probability functions may be found from the routines in `scipy.stats` as shown in this example.
***

Set the location, scale and shape factor for the distribution

```python
loc = 2.0

scale = 1.0

shape_factor = 10.0
```

Set the type of distribution (Gaussian, Weibull, Rayleigh, Rice or Chi-Squared)

```python
distribution_type = 'Gaussian'
```

Calculate the distributions from the `scipy` routines

```python
from scipy.stats import norm, weibull_min, rayleigh, rice, chi2

from numpy import linspace

if distribution_type == 'Gaussian':

    x = linspace(norm.ppf(0.001, loc, scale),  norm.ppf(0.999, loc, scale), 200)

    y = norm.pdf(x, loc, scale)

elif distribution_type == 'Weibull':

    x = linspace(weibull_min.ppf(0.001, shape_factor), weibull_min.ppf(0.999, shape_factor), 200)

    y = weibull_min.pdf(x, shape_factor, loc, scale)

elif distribution_type == 'Rayleigh':

    x = linspace(rayleigh.ppf(0.001, loc, scale), rayleigh.ppf(0.999, loc, scale), 200)

    y = rayleigh.pdf(x, loc, scale)

elif distribution_type == 'Rice':

    x = linspace(rice.ppf(0.001, shape_factor), rice.ppf(0.999, shape_factor), 200)

    y = rice.pdf(x, shape_factor, loc, scale)

elif distribution_type == 'Chi Squared':

    x = linspace(chi2.ppf(0.001, shape_factor), chi2.ppf(0.999, shape_factor), 200)

    y = chi2.pdf(x, shape_factor, loc, scale)
```

Display the resulting probability distribution using the `matplotlib` routines

```python
from matplotlib import pyplot as plt

# Set the figure size

plt.rcParams["figure.figsize"] = (15, 10)

# Display the results

plt.plot(x, y, '')

# Set the plot title and labels

plt.title('Probability Density Function', size=14)

plt.xlabel('x', size=12)

plt.ylabel('Probability p(x)', size=12)

# Set the tick label size

plt.tick_params(labelsize=12)

# Turn on the grid

plt.grid(linestyle=':', linewidth=0.5)
```

```python

```


```python
from sympy import *
init_printing()
```

```python
# Example 1
```

```python
hbar, m, L, g = symbols("hbar, m, L, g", positive=True)
```

```python
n, ell = symbols('n, ell', integer=True)
```

```python
x = symbols('x', real=True)
```

```python
def phi0(x,n):
    return sqrt(2/L)*sin(n*pi*x/L)
```

```python
phi0(x,n)
```

```python
def E0(n):
    return hbar**2*n**2*pi**2/(2*m*L**2)
```

```python
E0(n)
```

```python
#first order energy corrections
def E1(n):
    return integrate(phi0(x,n)*(m*g*x)*phi0(x,n),(x,0,L))
```

```python
E1(n) # the top line is the one that counts: n neq 0
```

```python
# first order state corrections
```

```python
# first, the integrals are
integrate(phi0(x,ell)*m*g*x*phi0(x,n),(x,0,L))
# The last line is the only good one: ell neq n and both are > 0
```

```python
_15.subs(n,1)
```

```python
summation(_16,(ell,2,oo))
```

```python
str(_15)
```

    'Piecewise((0, (Eq(ell, 0) & Eq(n, 0)) | (Eq(ell, 0) & Eq(ell, n) & Eq(n, 0)) | (Eq(ell, 0) & Eq(n, 0) & Eq(ell, -n)) | (Eq(ell, 0) & Eq(ell, n) & Eq(n, 0) & Eq(ell, -n))), (-L*g*m/2, Eq(ell, -n) | (Eq(ell, 0) & Eq(ell, -n)) | (Eq(ell, n) & Eq(ell, -n)) | (Eq(n, 0) & Eq(ell, -n)) | (Eq(ell, 0) & Eq(ell, n) & Eq(ell, -n)) | (Eq(ell, n) & Eq(n, 0) & Eq(ell, -n))), (L*g*m/2, Eq(ell, n) | (Eq(ell, 0) & Eq(ell, n)) | (Eq(ell, n) & Eq(n, 0))), (4*(-1)**ell*(-1)**n*L*ell*g*m*n/(pi**2*ell**4 - 2*pi**2*ell**2*n**2 + pi**2*n**4) - 4*L*ell*g*m*n/(pi**2*ell**4 - 2*pi**2*ell**2*n**2 + pi**2*n**4), True))'

```python
Hprimeelln = 4*(-1)**ell*(-1)**n*L*ell*g*m*n/(pi**2*ell**4 - 2*pi**2*ell**2*n**2 + pi**2*n**4) - 4*L*ell*g*m*n/(pi**2*ell**4 - 2*pi**2*ell**2*n**2 + pi**2*n**4)
```

```python
sumterm = Hprimeelln/(E0(n)-E0(ell))*phi0(ell,x)
```

```python
sumterm
```

```python
with assuming((Q.nonzero(n-ell)),Q.positive(n),Q.positive(ell)):
    summation(sumterm,(ell,1,n-1))+summation(sumterm,(ell,n+1,oo))
```

```python
theta = symbols('theta', real=True)
```

```python
Hmatrix = Matrix([[cos(theta),sin(theta)/sqrt(2),0],[sin(theta)/sqrt(2),0,sin(theta)/sqrt(2)],[0,sin(theta)/sqrt(2),-cos(theta)]])
```

```python
Hmatrix
```

```python
Hmatrix.eigenvects(simplify=True)
```

```python

```

```python

```


Bayesian interpretation of medical tests
-----------------------------------------

This notebooks explores several problems related to interpreting the results of medical tests.

Copyright 2016 Allen Downey

MIT License: http://opensource.org/licenses/MIT

```python
from __future__ import print_function, division

from thinkbayes2 import Pmf, Suite

from fractions import Fraction
```

## Medical tests

Suppose we test a patient to see if they have a disease, and the test comes back positive.  What is the probability that the patient is actually sick (that is, has the disease)?

To answer this question, we need to know:

*  The prevalence of the disease in the population the patient is from.  Let's assume the patient is identified as a member of a population where the known prevalence is `p`.

*  The sensitivity of the test, `s`, which is the probability of a positive test if the patient is sick.

*  The false positive rate of the test, `t`, which is the probability of a positive test if the patient is not sick.

Given these parameters, we can compute the probability that the patient is sick, given a positive test.

### Test class

To do that, I'll define a `Test` class that extends `Suite`, so it inherits `Update` and provides `Likelihood`.

The instance variables of `Test` are:

*  `p`, `s`, and `t`: Copies of the parameters.
*  `d`: a dictionary that maps from hypotheses to their probabilities.  The hypotheses are the strings `sick` and `notsick`.
*  `likelihood`: a dictionary that encodes the likelihood of the possible data values `pos` and `neg` under the hypotheses.

```python
class Test(Suite):
    """Represents beliefs about a patient based on a medical test."""

    def __init__(self, p, s, t, label='Test'):
        # initialize the prior probabilities
        d = dict(sick=p, notsick=1-p)
        super(Test, self).__init__(d, label)

        # store the parameters
        self.p = p
        self.s = s
        self.t = t

        # make a nested dictionary to compute likelihoods
        self.likelihood = dict(pos=dict(sick=s, notsick=t),
                               neg=dict(sick=1-s, notsick=1-t))

    def Likelihood(self, data, hypo):
        """
        data: 'pos' or 'neg'
        hypo: 'sick' or 'notsick'
        """
        return self.likelihood[data][hypo]
```

Now we can create a `Test` object with parameters chosen for demonstration purposes (most medical tests are better than this!):

```python
p = Fraction(1, 10)     # prevalence
s = Fraction(9, 10)     # sensitivity
t = Fraction(3, 10)     # false positive rate

test = Test(p, s, t)
test.Print()
```

    notsick 9/10
    sick 1/10

If you are curious, here's the nested dictionary that computes the likelihoods:

```python
test.likelihood
```

    {'neg': {'notsick': Fraction(7, 10), 'sick': Fraction(1, 10)},
     'pos': {'notsick': Fraction(3, 10), 'sick': Fraction(9, 10)}}

And here's how we update the `Test` object with a positive outcome:

```python
test.Update('pos')
test.Print()
```

    notsick 3/4
    sick 1/4

The positive test provides evidence that the patient is sick, increasing the probability from 0.1 to 0.25.

## Uncertainty about `t`

So far, this is basic Bayesian inference.  Now let's add a wrinkle.  Suppose that we don't know the value of `t` with certainty, but we have reason to believe that `t` is either 0.2 or 0.4 with equal probability.

Again, we would like to know the probability that a patient who tests positive actually has the disease.  As we did with the Red Die problem, we will consider several scenarios:

**Scenario A**: The patients are drawn at random from the relevant population, and the reason we are uncertain about `t` is that either (1) there are two versions of the test, with different false positive rates, and we don't know which test was used, or (2) there are two groups of people, the false positive rate is different for different groups, and we don't know which group the patient is in.

**Scenario B**: As in Scenario A, the patients are drawn at random from the relevant population, but the reason we are uncertain about `t` is that previous studies of the test have been contradictory.  That is, there is only one version of the test, and we have reason to believe that `t` is the same for all groups, but we are not sure what the correct value of `t` is.

**Scenario C**: As in Scenario A, there are two versions of the test or two groups of people.  But now the patients are being filtered so we only see the patients who tested positive and we don't know how many patients tested negative.  For example, suppose you are a specialist and patients are only referred to you after they test positive.

**Scenario D**: As in Scenario B, we have reason to think that `t` is the same for all patients, and as in Scenario C, we only see patients who test positive and don't know how many tested negative.

## Scenario A

We can represent this scenario with a hierarchical model, where the levels of the hierarchy are:

*  At the top level, the possible values of `t` and their probabilities.
*  At the bottom level, the probability that the patient is sick or not, conditioned on `t`.

To represent the hierarchy, I'll define a `MetaTest`, which is a `Suite` that contains `Test` objects with different values of `t` as hypotheses.

```python
class MetaTest(Suite):
    """Represents a set of tests with different values of `t`."""

    def Likelihood(self, data, hypo):
        """
        data: 'pos' or 'neg'
        hypo: Test object
        """
        # the return value from `Update` is the total probability of the
        # data for a hypothetical value of `t`
        return hypo.Update(data)
```

To update a `MetaTest`, we update each of the hypothetical `Test` objects.  The return value from `Update` is the normalizing constant, which is the total probability of the data under the hypothesis.

We use the normalizing constants from the bottom level of the hierarchy as the likelihoods at the top level.

Here's how we create the `MetaTest` for the scenario we described:

```python
q = Fraction(1, 2)
t1 = Fraction(2, 10)
t2 = Fraction(4, 10)

test1 = Test(p, s, t1, 'Test(t=0.2)')
test2 = Test(p, s, t2, 'Test(t=0.4)')

metatest = MetaTest({test1:q, test2:1-q})
metatest.Print()
```

    Test(t=0.2) 1/2
    Test(t=0.4) 1/2

At the top level, there are two tests, with different values of `t`.  Initially, they are equally likely.

When we update the `MetaTest`, it updates the embedded `Test` objects and then the `MetaTest` itself.

```python
metatest.Update('pos')
```

    Fraction(9, 25)

Here are the results.

```python
metatest.Print()
```

    Test(t=0.4) 5/8
    Test(t=0.2) 3/8

Because a positive test is more likely if `t=0.4`, the positive test is evidence in favor of the hypothesis that `t=0.4`.

This `MetaTest` object represents what we should believe about `t` after seeing the test, as well as what we should believe about the probability that the patient is sick.

### Marginal distributions

To compute the probability that the patient is sick, we have to compute the marginal probabilities of `sick` and `notsick`, averaging over the possible values of `t`.  The following function computes this distribution:

```python
def MakeMixture(metapmf, label='mix'):
    """Make a mixture distribution.

    Args:
      metapmf: Pmf that maps from Pmfs to probs.
      label: string label for the new Pmf.

    Returns: Pmf object.
    """
    mix = Pmf(label=label)
    for pmf, p1 in metapmf.Items():
        for x, p2 in pmf.Items():
            mix.Incr(x, p1 * p2)
    return mix
```

Here's the posterior predictive distribution:

```python
predictive = MakeMixture(metatest)
predictive.Print()
```

    notsick 3/4
    sick 1/4

After seeing the test, the probability that the patient is sick is 0.25, which is the same result we got with `t=0.3`.

### Two patients

Now suppose you test two patients and they both test positive.  What is the probability that they are both sick?

To answer that, I define a few more functions to work with Metatests:

```python
def MakeMetaTest(p, s, pmf_t):
    """Makes a MetaTest object with the given parameters.

    p: prevalence
    s: sensitivity
    pmf_t: Pmf of possible values for `t`
    """
    tests = {}
    for t, q in pmf_t.Items():
        label = 'Test(t=%s)' % str(t)
        tests[Test(p, s, t, label)] = q
    return MetaTest(tests)

def Marginal(metatest):
    """Extracts the marginal distribution of t.
    """
    marginal = Pmf()
    for test, prob in metatest.Items():
        marginal[test.t] = prob
    return marginal

def Conditional(metatest, t):
    """Extracts the distribution of sick/notsick conditioned on t."""
    for test, prob in metatest.Items():
        if test.t == t:
            return test
```

`MakeMetaTest` makes a `MetaTest` object starting with a given PMF of `t`.

`Marginal` extracts the PMF of `t` from a `MetaTest`.

`Conditional` takes a specified value for `t` and returns the PMF of `sick` and `notsick` conditioned on `t`.

I'll test these functions using the same parameters from above:

```python
pmf_t = Pmf({t1:q, t2:1-q})
metatest = MakeMetaTest(p, s, pmf_t)
metatest.Print()
```

    Test(t=1/5) 1/2
    Test(t=2/5) 1/2

Here are the results

```python
metatest = MakeMetaTest(p, s, pmf_t)
metatest.Update('pos')
metatest.Print()
```

    Test(t=2/5) 5/8
    Test(t=1/5) 3/8

Same as before.  Now we can extract the posterior distribution of `t`.

```python
Marginal(metatest).Print()
```

    1/5 3/8
    2/5 5/8

Having seen one positive test, we are a little more inclined to believe that `t=0.4`; that is, that the false positive rate for this patient/test is high.

And we can extract the conditional distributions for the patient:

```python
cond1 = Conditional(metatest, t1)
cond1.Print()
```

    notsick 2/3
    sick 1/3

```python
cond2 = Conditional(metatest, t2)
cond2.Print()
```

    notsick 4/5
    sick 1/5

Finally, we can make the posterior marginal distribution of sick/notsick, which is a weighted mixture of the conditional distributions:

```python
MakeMixture(metatest).Print()
```

    notsick 3/4
    sick 1/4

At this point we have a `MetaTest` that contains our updated information about the test (the distribution of `t`) and about the patient that tested positive.

Now, to compute the probability that both patients are sick, we have to know the distribution of `t` for both patients.  And that depends on details of the scenario.

In Scenario A, the reason we are uncertain about `t` is either (1) there are two versions of the test, with different false positive rates, and we don't know which test was used, or (2) there are two groups of people, the false positive rate is different for different groups, and we don't know which group the patient is in.

So the value of `t` for each patient is an independent choice from `pmf_t`; that is, if we learn something about `t` for one patient, that tells us nothing about `t` for other patients.

So if we consider two patients who have tested positive, the MetaTest we just computed represents our belief about each of the two patients independently.

To compute the probability that both patients are sick, we can convolve the two distributions.

```python
convolution = metatest + metatest
convolution.Print()
```

    Pmf({'notsicknotsick': Fraction(8, 15), 'sicksick': Fraction(1, 15), 'sicknotsick': Fraction(2, 15), 'notsicksick': Fraction(4, 15)}) 15/64
    Pmf({'notsicknotsick': Fraction(16, 25), 'sicksick': Fraction(1, 25), 'sicknotsick': Fraction(4, 25), 'notsicksick': Fraction(4, 25)}) 25/64
    Pmf({'notsicknotsick': Fraction(8, 15), 'sicksick': Fraction(1, 15), 'sicknotsick': Fraction(4, 15), 'notsicksick': Fraction(2, 15)}) 15/64
    Pmf({'notsicknotsick': Fraction(4, 9), 'sicksick': Fraction(1, 9), 'sicknotsick': Fraction(2, 9), 'notsicksick': Fraction(2, 9)}) 9/64

Then we can compute the posterior marginal distribution of sick/notsick for the two patients:

```python
marginal = MakeMixture(metatest+metatest)
marginal.Print()
```

    notsicknotsick 9/16
    notsicksick 3/16
    sicknotsick 3/16
    sicksick 1/16

So in Scenario A the probability that both patients are sick is 1/16.

As an aside, we could have computed the marginal distributions first and then convolved them, which is computationally more efficient:

```python
marginal = MakeMixture(metatest) + MakeMixture(metatest)
marginal.Print()
```

    notsicknotsick 9/16
    notsicksick 3/16
    sicknotsick 3/16
    sicksick 1/16

We can confirm that this result is correct by simulation.  Here's a generator that generates random pairs of patients:

```python
from random import random

def flip(p):
    return random() < p

def generate_pair_A(p, s, pmf_t):
    while True:
        sick1, sick2 = flip(p), flip(p)

        t = pmf_t.Random()
        test1 = flip(s) if sick1 else flip(t)

        t = pmf_t.Random()
        test2 = flip(s) if sick2 else flip(t)

        yield test1, test2, sick1, sick2
```

And here's a function that runs the simulation for a given number of iterations:

```python
def run_simulation(generator, iters=100000):
    pmf_t = Pmf([0.2, 0.4])
    pair_iterator = generator(0.1, 0.9, pmf_t)

    outcomes = Pmf()
    for i in range(iters):
        test1, test2, sick1, sick2 = next(pair_iterator)
        if test1 and test2:
            outcomes[sick1, sick2] += 1

    outcomes.Normalize()
    return outcomes
```

```python
outcomes = run_simulation(generate_pair_A)
outcomes.Print()
```

    (False, False) 0.5635400907715582
    (False, True) 0.18267776096822994
    (True, False) 0.19130105900151284
    (True, True) 0.062481089258698934

As we increase `iters`, the probablity of (True, True) converges on 1/16, which is what we got from the analysis.

Good so far!

## Scenario B

In Scenario B, we have reason to believe the `t` is the same for all patients, but we are not sure what it is.  So each time we see a positive test, we get some information about `t` for all patients.

The first time we see positive test we do the same update as in Scenario A:

```python
metatest1 = MakeMetaTest(p, s, pmf_t)
metatest1.Update('pos')
metatest1.Print()
```

    Test(t=2/5) 5/8
    Test(t=1/5) 3/8

And the marginal distribution of sick/notsick is the same:

```python
marginal = MakeMixture(metatest1)
marginal.Print()
```

    notsick 3/4
    sick 1/4

Now suppose the second patient arrives.  We need a new `MetaTest` that contains the updated information about the test, but no information about the patient other than the prior probability of being sick, `p`:

```python
metatest2 = MakeMetaTest(p, s, Marginal(metatest1))
metatest2.Print()
```

    Test(t=1/5) 3/8
    Test(t=2/5) 5/8

Now we can update this `MetaTest` with the result from the second test:

```python
metatest2.Update('pos')
metatest2.Print()
```

    Test(t=2/5) 25/34
    Test(t=1/5) 9/34

This distribution contains updated information about the test, based on two positive outcomes, and updated information about a patient who has tested positive (once).

After seeing two patients with positive tests, the probability that `t=0.4` has increased to 25/34, around 74%.

For either patient, the probability of being sick is given by the marginal distribution from `metatest2`:

```python
predictive = MakeMixture(metatest2)
predictive.Print()
```

    notsick 13/17
    sick 4/17

After two tests, the probability that the patient is sick is slightly lower than after one (4/17 is about 23.5%, compared to 25%).  That's because the second positive test increases our belief that the false positive rate is high (t=0.4), which decreases our belief that either patient is sick.

Now, to compute the probability that both are sick, we can't just convolve the posterior marginal distribution with itself, as we did in Scenario A, because the selection of `t` is not independent for the two patients.  Instead, we have to make a weighted mixture of conditional distributions.

If we know `t=t1`, we can compute the joint distribution for the two patients:

```python
cond_t1 = Conditional(metatest2, t1)
conjunction_t1 = cond_t1 + cond_t1
conjunction_t1.Print()
```

    notsicknotsick 4/9
    notsicksick 2/9
    sicknotsick 2/9
    sicksick 1/9

If we know that `t=t1`, the probability of `sicksick` is 0.111.  And for `t=t2`:

```python
cond_t2 = Conditional(metatest2, t2)
conjunction_t2 = cond_t2 + cond_t2
conjunction_t2.Print()
```

    notsicknotsick 16/25
    notsicksick 4/25
    sicknotsick 4/25
    sicksick 1/25

If we know that `t=t2`, the probability of `sicksick` is `0.04`.

The overall probability of `sicksick` is the weighted average of these probabilities:

```python
posterior_t = Marginal(metatest2)
posterior_t[t1] * conjunction_t1['sicksick'] + posterior_t[t2] * conjunction_t2['sicksick']
```

    Fraction(1, 17)

`1/17` is about `0.0588`, slightly smaller than in Scenario A (`1/16`, which is about `0.0667`).

To compute the probabilities for all four outcomes, I'll make a `Metapmf` that contains the two conditional distributions.

```python
metapmf = Pmf()
for t, prob in Marginal(metatest2).Items():
    cond = Conditional(metatest2, t)
    conjunction = cond + cond
    metapmf[conjunction] = prob

metapmf.Print()
```

    Pmf({'notsicknotsick': Fraction(16, 25), 'sicksick': Fraction(1, 25), 'sicknotsick': Fraction(4, 25), 'notsicksick': Fraction(4, 25)}) 25/34
    Pmf({'notsicknotsick': Fraction(4, 9), 'sicksick': Fraction(1, 9), 'sicknotsick': Fraction(2, 9), 'notsicksick': Fraction(2, 9)}) 9/34

And finally we can use `MakeMixture` to compute the weighted averages of the posterior probabilities:

```python
predictive = MakeMixture(metapmf)
predictive.Print()
```

    notsicknotsick 10/17
    notsicksick 3/17
    sicknotsick 3/17
    sicksick 1/17

To confirm that this result is correct, I'll use the simuation again with a different generator:

```python
def generate_pair_B(p, s, pmf_t):
    while True:
        sick1, sick2 = flip(p), flip(p)

        t = pmf_t.Random()
        test1 = flip(s) if sick1 else flip(t)

        #  Here's the difference
        #  t = pmf_t.Random()
        test2 = flip(s) if sick2 else flip(t)

        yield test1, test2, sick1, sick2
```

The difference between Scenario A and Scenario B is the line I commented out.  In Scenario B, we generate `t` once and it applies to both patients.

```python
outcomes = run_simulation(generate_pair_B)
outcomes.Print()
```

    (False, False) 0.5922922348213004
    (False, True) 0.17568537390555478
    (True, False) 0.17145112674034738
    (True, True) 0.06057126453279748

As `iters` increases, the results from the simulation converge on `1/17`.

### Summary so far

In summary:

                        P(sick|pos)             P(sicksick|pospos)
    Scenario A          1/4 = 25%               1/16 = 6.25%
    Scenario B          1/4 = 25%               1/17 ~= 5.88%

If we are only interested in one patient at a time, Scenarios A and B are the same.  But for collections of patients, they yield different probabilities.

A real scenario might combine elements of A and B; that is, the false positive rate might be different for different people, and we might have some uncertainty about what it is.  In that case, the most accurate probability for two patients might be anywhere between `1/16` and `1/17`.

### Scenario C

Scenario C is similar to Scenario A: we believe that the false positive rate `t` might be different for different people, or for different versions of the test.  The difference is that in Scenario A we see all patients, sick or not, positive test or not.

In Scenario C, we only see patients after they have tested positive, and we don't know how many tested negative.  For example, if you are a specialist and patients are referred to you only if they test positive, Scenario C might be a good model of your situation.

Before I analyze this scenario, I'll start with a simulation.  As a reminder, here's a generator that generates pairs of patients in Scenario A:

```python
def generate_pair_A(p, s, pmf_t):
    while True:
        sick1, sick2 = flip(p), flip(p)

        t = pmf_t.Random()
        test1 = flip(s) if sick1 else flip(t)

        t = pmf_t.Random()
        test2 = flip(s) if sick2 else flip(t)

        yield test1, test2, sick1, sick2
```

And here's the simulator that uses the generator to estimate the probability that two patients who test positive are both sick.

```python
def run_simulation(generator, iters=100000):
    pmf_t = Pmf([0.2, 0.4])
    pair_iterator = generator(0.1, 0.9, pmf_t)

    outcomes = Pmf()
    for i in range(iters):
        test1, test2, sick1, sick2 = next(pair_iterator)
        if test1 and test2:
            outcomes[sick1, sick2] += 1

    outcomes.Normalize()
    return outcomes
```

As we saw before, this probability converges on $1/16$.

```python
outcomes = run_simulation(generate_pair_A)
outcomes.Print()
```

    (False, False) 0.5723947125730094
    (False, True) 0.18567476175837688
    (True, False) 0.1796802951122041
    (True, True) 0.062250230556409464

Now here's a generator that generates pairs of patients in Scenario C.  The difference is that for each pair we check the outcome of the tests; if they are not both positive, we loop back and try again:

```python
def generate_pair_C(p, s, pmf_t):
    while True:
        sick1, sick2 = flip(p), flip(p)

        t = pmf_t.Random()
        test1 = flip(s) if sick1 else flip(t)

        t = pmf_t.Random()
        test2 = flip(s) if sick2 else flip(t)

        # here is the difference
        if test1 and test2:
            yield test1, test2, sick1, sick2
```

When we run it, it seems like the probability is still `1/16`:

```python
outcomes = run_simulation(generate_pair_C)
outcomes.Print()
```

    (False, False) 0.56137
    (False, True) 0.18819000000000002
    (True, False) 0.18706
    (True, True) 0.06338

If you examine the code, you see that the conditional in `generate_pair_C` makes no difference because it is redundant with the conditional in `run_simulation`.  In Scenarios A and C, we filter out pairs if they are not both positive; it doesn't matter whether the filtering happens in the generator or the simulator.

In fact, Scenarios A and C are identical.  In both scenarios, when we see a patient with a positive test, we learn something about the patient (more likely to be sick) and something about the particular test applied to the patient (more likely to generate false positives).

This is similar to what we saw in the Red Die problem.  In Scenario C, the reddish die is more likely to produce a red outcome, so a red outcome provides evidence that we rolled the reddish die.

However, that is not the case with Scenario D.

### Scenario D

As a reminder, Scenario D is similar to B: we have reason to think that `t` is either `0.2` or `0.4` for everyone.  The difference in Scenario D is that we only see patients if they test positive.

Here's a generator that generates single patients:

```python
def generate_patient_D(p, s, pmf_t):
    while True:
        # choose t
        t = pmf_t.Random()

        # generate patients until positive test
        while True:
            sick = flip(p)
            test = flip(s) if sick else flip(t)
            if test:
                yield test, sick
                break
```

And here's a simulator that counts the fraction of positive tests that turn out to be sick:

```python
def run_single_simulation(generator, iters=100000):
    pmf_t = Pmf([0.2, 0.4])
    iterator = generator(0.1, 0.9, pmf_t)

    outcomes = Pmf()
    for i in range(iters):
        test, sick = next(iterator)
        if test:
            outcomes[sick] += 1

    outcomes.Normalize()
    return outcomes
```

When we run the simulation, it doesn't look like it converges to `1/4` as it does in the other three scenarios.

```python
outcomes = run_single_simulation(generate_patient_D)
outcomes.Print()
```

    False 0.7319500000000001
    True 0.26805

So how can we analyze this scenario?

The key is to realize that, as in Scenario D of the Red Dice problem, if we roll until we get red, we don't learn anything about the die we rolled, and in this case, if we generate patients until we get a positive test, we don't learn anything about `t`.  The likelihood of the data (a positive test) is 1, regardless of `t`.

We can compute the probablity the patient is sick by creating a MetaTest and updating only the lower level (the Test objects) but not the upper level (the distribution of `t`).

```python
metatest = MakeMetaTest(p, s, pmf_t)
for hypo in metatest:
    hypo.Update('pos')
```

After the update, the marginal distribution of `t` is unchanged:

```python
Marginal(metatest).Print()
```

    1/5 1/2
    2/5 1/2

But the conditional probabilities have been updated:

```python
Conditional(metatest, t1).Print()
```

    notsick 2/3
    sick 1/3

```python
Conditional(metatest, t2).Print()
```

    notsick 4/5
    sick 1/5

We can use `MakeMixture` to compute the weighted average of the conditional distributions.

```python
MakeMixture(metatest).Print()
```

    notsick 11/15
    sick 4/15

So in Scenario D, a patient who tests positive has a probability of `4/15` of being sick, which is about 26.7%, and consistent with the simulation.

That's a little higher than in the other three Scenarios, because we have less reason to think that `t` is high.

### Scenario D, two patients

Now let's see what happens with two patients.  Here's a generator that generates pairs of patients:

```python
def generate_pair_D(p, s, pmf_t):
    while True:
        t = pmf_t.Random()
        while True:
            sick1, sick2 = flip(p), flip(p)

            test1 = flip(s) if sick1 else flip(t)
            test2 = flip(s) if sick2 else flip(t)

            if test1 and test2:
                yield test1, test2, sick1, sick2
                break
```

And here's what we get when we run the simulation:

```python
outcomes = run_simulation(generate_pair_D, iters=1000000)
outcomes.Print()
```

    (False, False) 0.541945
    (False, True) 0.191442
    (True, False) 0.191182
    (True, True) 0.075431

It looks like the probability that both patients are sick is higher than `1/16`.

We can compute the result exactly using the posterior distribution and the same method we used in Scenario B, computing the mixture of two conjunctions:

```python
def MixConjunctions(metatest):
    metapmf = Pmf()
    for t, prob in Marginal(metatest).Items():
        cond = Conditional(metatest, t)
        conjunction = cond + cond
        metapmf[conjunction] = prob

    return MakeMixture(metapmf)
```

```python
MixConjunctions(metatest).Print()
```

    notsicknotsick 122/225
    notsicksick 43/225
    sicknotsick 43/225
    sicksick 17/225

In Scenario D, the probability that both patients are sick is `17/225`, or about 0.0755, which is consistent with the simulation and, again, a little higher than in the other scenarios.

In summary:

                        P(sick|pos)             P(sicksick|pospos)
    Scenario A          1/4 = 25%               1/16 = 6.25%
    Scenario B          1/4 = 25%               1/17 ~= 5.88%
    Scenario C          1/4 = 25%               1/16 = 6.25%
    Scenario D          4/15 ~= 26.7%           17/225 ~= 7.55%

## Symbolic solutions

One nice thing about Python is that the same code can work with floating-point numbers, rational numbers (`Fraction` objects), and SymPy symbols.

The following functions solve the various scenarios:

```python
def scenario_a(p, s, pmf_t):
    metatest = MakeMetaTest(p, s, pmf_t)
    metatest.Update('pos')
    single = MakeMixture(metatest)
    pair = single + single
    return single, pair
```

```python
single, pair = scenario_a(p, s, pmf_t)
single.Print()
pair.Print()
```

    notsick 3/4
    sick 1/4
    notsicknotsick 9/16
    notsicksick 3/16
    sicknotsick 3/16
    sicksick 1/16

```python
def scenario_b(p, s, pmf_t):
    metatest1 = MakeMetaTest(p, s, pmf_t)
    metatest1.Update('pos')
    single = MakeMixture(metatest1)

    metatest2 = MakeMetaTest(p, s, Marginal(metatest1))
    metatest2.Update('pos')
    pair = MixConjunctions(metatest2)
    return single, pair
```

```python
single, pair = scenario_b(p, s, pmf_t)
single.Print()
pair.Print()
```

    notsick 3/4
    sick 1/4
    notsicknotsick 10/17
    notsicksick 3/17
    sicknotsick 3/17
    sicksick 1/17

```python
def scenario_d(p, s, pmf_t):
    metatest = MakeMetaTest(p, s, pmf_t)
    for hypo in metatest:
        hypo.Update('pos')
    single = MakeMixture(metatest)
    pair = MixConjunctions(metatest)
    return single, pair
```

```python
single, pair = scenario_d(p, s, pmf_t)
single.Print()
pair.Print()
```

    notsick 11/15
    sick 4/15
    notsicknotsick 122/225
    notsicksick 43/225
    sicknotsick 43/225
    sicksick 17/225

And here's the symbolic version:

```python
from sympy import symbols

p, s, q, t1, t2 = symbols(['p', 's', 'q', 't1', 't2'])
pmf_t = Pmf({t1:q, t2:1-q})
```

```python
def PrintSymSuite(suite):
    for hypo, prob in suite.Items():
        print(hypo, prob.simplify())
```

```python
single, pair = scenario_a(p, s, pmf_t)
PrintSymSuite(single)
PrintSymSuite(pair)
```

    notsick (p - 1)*(-q*t1 + t2*(q - 1))/(q*(p*s - t1*(p - 1)) - (q - 1)*(p*s - t2*(p - 1)))
    sick p*s/(q*(p*s - t1*(p - 1)) - (q - 1)*(p*s - t2*(p - 1)))
    notsicknotsick (p - 1)**2*(q*t1 - t2*(q - 1))**2/(q*(p*s - t1*(p - 1)) - (q - 1)*(p*s - t2*(p - 1)))**2
    sicksick p**2*s**2/(q*(p*s - t1*(p - 1)) - (q - 1)*(p*s - t2*(p - 1)))**2
    sicknotsick -p*s*(p - 1)*(q*t1 - t2*(q - 1))/(q*(p*s - t1*(p - 1)) - (q - 1)*(p*s - t2*(p - 1)))**2
    notsicksick -p*s*(p - 1)*(q*t1 - t2*(q - 1))/(q*(p*s - t1*(p - 1)) - (q - 1)*(p*s - t2*(p - 1)))**2

```python
single, pair = scenario_b(p, s, pmf_t)
PrintSymSuite(single)
PrintSymSuite(pair)
```

    notsick (p - 1)*(-q*t1 + t2*(q - 1))/(q*(p*s - t1*(p - 1)) - (q - 1)*(p*s - t2*(p - 1)))
    sick p*s/(q*(p*s - t1*(p - 1)) - (q - 1)*(p*s - t2*(p - 1)))
    notsicknotsick (p - 1)**2*(q*t1**2 + t2**2*(-q + 1))/(q*(p*s - t1*(p - 1))**2 - (q - 1)*(p*s - t2*(p - 1))**2)
    sicksick p**2*s**2/(q*(p*s - t1*(p - 1))**2 - (q - 1)*(p*s - t2*(p - 1))**2)
    sicknotsick p*s*(p - 1)*(-q*t1 + t2*(q - 1))/(q*(p*s - t1*(p - 1))**2 - (q - 1)*(p*s - t2*(p - 1))**2)
    notsicksick p*s*(p - 1)*(-q*t1 + t2*(q - 1))/(q*(p*s - t1*(p - 1))**2 - (q - 1)*(p*s - t2*(p - 1))**2)

```python
single, pair = scenario_d(p, s, pmf_t)
PrintSymSuite(single)
PrintSymSuite(pair)
```

    notsick (p - 1)*(-q*t1*(p*s - t2*(p - 1)) + t2*(q - 1)*(p*s - t1*(p - 1)))/((p*s - t1*(p - 1))*(p*s - t2*(p - 1)))
    sick p*s*(q*(p*s - t2*(p - 1)) - (q - 1)*(p*s - t1*(p - 1)))/((p*s - t1*(p - 1))*(p*s - t2*(p - 1)))
    notsicknotsick (p - 1)**2*(q*t1**2*(p*s - t2*(p - 1))**2 + t2**2*(-q + 1)*(p*s - t1*(p - 1))**2)/((p*s - t1*(p - 1))**2*(p*s - t2*(p - 1))**2)
    sicksick p**2*s**2*(q*(p*s - t2*(p - 1))**2 + (-q + 1)*(p*s - t1*(p - 1))**2)/((p*s - t1*(p - 1))**2*(p*s - t2*(p - 1))**2)
    sicknotsick p*s*(p - 1)*(-q*t1*(p*s - t2*(p - 1))**2 + t2*(q - 1)*(p*s - t1*(p - 1))**2)/((p*s - t1*(p - 1))**2*(p*s - t2*(p - 1))**2)
    notsicksick p*s*(p - 1)*(-q*t1*(p*s - t2*(p - 1))**2 + t2*(q - 1)*(p*s - t1*(p - 1))**2)/((p*s - t1*(p - 1))**2*(p*s - t2*(p - 1))**2)

```python

```