# Multi-Qubit System Measurements Tutorial Workbook

**What is this workbook?**
A workbook is a collection of problems, accompanied by solutions to them.
The explanations focus on the logical steps required to solve a problem; they illustrate the concepts that need to be applied to come up with a solution to the problem, explaining the mathematical steps required.

Note that a workbook should not be the primary source of knowledge on the subject matter; it assumes that you've already read a tutorial or a textbook and that you are now seeking to improve your problem-solving skills. You should attempt solving the tasks of the respective kata first, and turn to the workbook only if stuck. While a textbook emphasizes knowledge acquisition, a workbook emphasizes skill acquisition.

This workbook describes the solutions to the problems offered in the [Multi-Qubit System Measurements tutorial](./MultiQubitSystemMeasurements.ipynb).
Since the tasks are offered as programming problems, the explanations also cover some elements of Q# that might be non-obvious for a first-time user.

**What you should know for this workbook**

You should be familiar with the following concepts before tackling the Single-Qubit System Measurements tutorial (and this workbook):
1. Basic linear algebra
2. Single and multi-qubit systems
3. Single and multi-qubit gates
$\newcommand{\ket}[1]{\left|#1\right>}$
$\newcommand{\bra}[1]{\left<#1\right|}$

### Exercise 1: Multi-qubit measurement outcome probabilities I
Suppose that a two-qubit system is known to be in the following state:
$$\ket \psi =  \frac{1}{3}\ket {00} + \frac{2}{3} \ket {01} + \frac{2}{3}\ket {11}$$

If all the qubits are measured simultaneously in the computational basis, what are the outcome probabilities?

### Solution
The wave function $|\psi\rangle$ is normalized, since $\left(\frac{1}{3}\right)^2 + \left(\frac{2}{3}\right)^2 + \left(\frac{2}{3}\right)^2 = 1$. Hence, the probabilities of measuring each of the computational basis states is simply the square of the absolute value of the corresponding coefficients. That is, the probabilities of measuring $00$, $01$ and $11$ are $\frac{1}{9}$, $\frac{4}{9}$ and $\frac{4}{9}$, respectively, and the probability of measuring the basis state $10$ that is not part of the superposition is $0$:

    <col width=150>
    <col width=150>

        Measurement outcome
        Probability of outcome

        $00$
        $\left( \frac{1}{3}\right)^2 = \frac{1}{9}$

        $01$
        $\left( \frac{2}{3}\right)^2 = \frac{4}{9}$

        $10$
        $\left( 0\right)^2 = 0$

        $11$
        $\left( \frac{2}{3}\right)^2 = \frac{4}{9}$

[Return to exercise 1 of the Multi-Qubit System Measurements tutorial.](./MultiQubitSystemMeasurements.ipynb#Exercise-1:-Multi-qubit-measurement-outcome-probabilities-I)

### Exercise 2: Multi-qubit measurement outcome probabilities II
Suppose that a two-qubit system is known to be in the following state:
$$\ket \psi =  \frac{2}{3}\ket {00} + \frac{1}{3} \ket {01} + \frac{2}{3}\ket {11}$$.

If all the qubits are measured simultaneously in the Pauli X basis, i.e., in the $\{ \ket{++}, \ket{+-}, \ket{-+}, \ket{--}\}$ basis, what are the outcome probabilities?

### Solution
Using the expressions $|0\rangle = \frac{1}{\sqrt{2}} \big( |+\rangle + |-\rangle \big)$ and $|1\rangle = \frac{1}{\sqrt{2}} \big( |+\rangle - |-\rangle \big)$, we first express $|\psi\rangle$ in the Pauli X basis. This gives us
$$
|\psi \rangle =
\frac{2}{3} \cdot \frac{1}{\sqrt{2}} \big( |+\rangle + |-\rangle \big) \otimes \frac{1}{\sqrt{2}}\big( |+\rangle + |-\rangle \big) +
\frac{1}{3} \cdot \frac{1}{\sqrt{2}} \big( |+\rangle + |-\rangle \big) \otimes \frac{1}{\sqrt{2}} \big( |+\rangle - |-\rangle \big) +
\frac{2}{3} \cdot \frac{1}{\sqrt{2}} \big( |+\rangle - |-\rangle \big) \otimes \frac{1}{\sqrt{2}} \big( |+\rangle - |-\rangle \big) =
\\ = \frac{5}{6} |++\rangle - \frac{1}{6} |+-\rangle + \frac{1}{2} |-+\rangle + \frac{1}{6} |--\rangle
$$
After this, the probabilities of measuring each of the four basis vectors is given by the square of the absolute value of its amplitude in the superposition:

    <col width=150>
    <col width=150>

        Measurement outcome
        Probability of outcome

        $++$
        $\left( \frac{5}{6}\right)^2 = \frac{25}{36}$

        $+-$
        $\left( -\frac{1}{6}\right)^2 = \frac{1}{36}$

        $-+$
        $\left( \frac{1}{2}\right)^2 = \frac{1}{4}$

        $++$
        $\left( \frac{1}{6}\right)^2 = \frac{1}{36}$

[Return to exercise 2 of the Multi-Qubit System Measurements tutorial.](./MultiQubitSystemMeasurements.ipynb#Exercise-2:-Multi-qubit-measurement-outcome-probabilities-II)

### Exercise 3: Distinguish four basis states

**Input:** Two qubits (stored in an array of length 2) which are guaranteed to be in one of the four basis states ($|00\rangle$, $|01\rangle$, $|10\rangle$, or $|11\rangle$).

**Output:**

* 0 if the qubits were in the $|00\rangle$ state,
* 1 if they were in the $|01\rangle$ state,
* 2 if they were in the $|10\rangle$ state,
* 3 if they were in the $|11\rangle$ state.

In this task and the subsequent ones the order of qubit states in task description matches the order of qubits in the array (i.e., $|10\rangle$ state corresponds to `qs[0]` in state $|1\rangle$ and `qs[1]` in state $|0\rangle$).

The state of the qubits at the end of the operation does not matter.

### Solution

First, we measure both qubits in the input array and store the result in `m1` and `m2`. We can decode these results like this:
- `m1` is $|0\rangle$ and `m2` is $|0\rangle$: we return $0\cdot2+0 = 0$
- `m1` is $|0\rangle$ and `m2` is $|1\rangle$: we return $0\cdot2+1 = 1$
- `m1` is $|1\rangle$ and `m2` is $|0\rangle$: we return $1\cdot2+0 = 2$
- `m1` is $|1\rangle$ and `m2` is $|1\rangle$: we return $1\cdot2+1 = 3$

In other words, we treat the measurement results as the binary notation of the return value in [big endian notation](../MultiQubitSystems/MultiQubitSystems.ipynb#Endianness).

```qsharp
%kata T1_BasisStateMeasurement

operation BasisStateMeasurement (qs : Qubit[]) : Int {
    // Measurement on the first qubit gives the higher bit of the answer, on the second - the lower
    let m1 = M(qs[0]) == Zero ? 0 | 1;
    let m2 = M(qs[1]) == Zero ? 0 | 1;
    return m1 * 2 + m2;
}
```

We can generalize this code to read out an integer in big endian notation from a qubit array of arbitrary length using several library operations and functions:

* `MultiM` measures each of the qubits in the array in the computational basis and returns an array of `Result` data type.
* `Reversed` reverses the given array.
* `ResultArrayAsInt` converts an array of bits given as `Result` to an integer, assuming little-endian notation (that's why we have to reverse the array before converting it).

```qsharp
%kata T1_BasisStateMeasurement

open Microsoft.Quantum.Arrays;
open Microsoft.Quantum.Convert;
open Microsoft.Quantum.Measurement;

operation BasisStateMeasurement (qs : Qubit[]) : Int {
    return ResultArrayAsInt(Reversed(MultiM(qs)));
}
```

[Return to exercise 3 of the Multi-Qubit System Measurements tutorial.](./MultiQubitSystemMeasurements.ipynb##Exercise-3:-Distinguish-four-basis-states)

### Exercise 4: Partial measurement probabilities for the Hardy state

Consider a 2-qubit system in the state $\ket \psi = \frac{1}{\sqrt{12}} \big(3|00\rangle + |01\rangle + |10\rangle + |11\rangle\big)$.

If only the first qubit is measured in the computational basis, what are the probabilities of the outcomes, and the post-measurement states of the system?

### Solution

A measurement outcome of $0$ on the first qubit corresponds to the projection operator $P_0|0\rangle\langle 0| \otimes \mathbb{1}$. Applying it to the state $\ket \psi$ gives us
$$\big|P_0 \ket{\psi}\big|^2 = \big|\frac{1}{\sqrt{10}} \left(3\ket {00} + \ket{01}\right) \big|^2 = \frac{5}{6}$$
and
$$\frac{P_0 \ket{\psi}}{\big|P_0 \ket{\psi}\big|} = \frac{1}{\sqrt{10}} \left( 3\ket{00} + \ket{01}\right)$$

Similarly, $P_1 = |1\rangle \langle 1 | \otimes \mathbb{1}$ is the projector corresponding to a measurement outcome of $1$ on the first qubit. Applying $P_1$ on $\ket{\psi}$ gives us $\big|P_1 \ket{\psi}\big|^2 = \frac{1}{6}$ and

$$\frac{P_1 \ket{\psi}}{\big|P_1 \ket{\psi}\big|} = \frac{1}{\sqrt{2}} \left(\ket{10} + \ket{11}\right)$$

    <col width=150>
    <col width=150>
    <col width=150>

        Measurement outcome
        Probability of outcome
        Post-measurement state

        $0$
        $\frac{5}{6}$
        $\frac{1}{\sqrt{10}} \left( 3\ket{00} + \ket{01}\right)$

        $1$
        $\frac{1}{6}$
        $\frac{1}{\sqrt{2}} \left(\ket{10} + \ket{11}\right)$

[Return to exercise 4 of the Multi-Qubit System Measurements tutorial.](./MultiQubitSystemMeasurements.ipynb#Exercise-4:-Partial-measurement-probabilities-for-the-Hardy-state)

### Exercise 5: Distinguish orthogonal states using partial measurements

**Input**: A 3-qubit system which is guaranteed to be in either the $\ket{++-}$ state, or the $\ket{---}$ state.

**Output**:
- 0 if the qubit is in the $\ket{++-}$ state,
- 1 if the qubit is in the $\ket{---}$ state.

The state of the qubits at the end of the operation should be the same as the initial state.

### Solution
Since the state of the first qubit is different in these states ($\ket +$ and $\ket -$, respectively), it is sufficient to measure only the first qubit in the Pauli X basis in order to distinguish the two states. Furthermore, this measurement will not change the state of the system, since the post-projection state is identical to the original state regardless of the input state.

```qsharp
%kata T2_IsPlusPlusMinus

operation IsPlusPlusMinus (qs : Qubit[]) : Int {
    return Measure([PauliX], [qs[0]]) == Zero ? 0 | 1;
}
```

[Return to exercise 5 of the Multi-Qubit System Measurements tutorial.](./MultiQubitSystemMeasurements.ipynb#Exercise-5:-Distinguish-orthogonal-states-using-partial-measurements)

### Exercise 6: Sequential measurements on an entangled state and a separable state
Consider two two-qubit states:
- The Bell state $|\Phi^{+}\rangle = \frac{1}{\sqrt{2}} \big (|00\rangle + |11\rangle\big)$.
- A state $\ket \Theta = \frac{1}{2} \big( \ket{00} + \ket{01} + \ket{10} + \ket{11} \big)$.

For both states, consider a measurement on the first qubit, followed by a measurement on the second qubit, both done in the computational basis. For which state can we expect the measurement outcomes to be correlated? Verify by calculating the sequential measurement probabilities explicitly for both states.

### Solution
- *The Bell state*: If the measurement outcome on the first qubit is $0$, a subsequent measurement on the second qubit *always* results in an outcome of $0$, with probability $1$. Similarly, if the measurement outcome on the first qubit is $1$, then the second qubit measurement always results in $1$. Thus, sequential measurements are perfectly *correlated*.
- *Separable state* $\ket \Theta$: Irrespective of whether the first qubit measurement outcome is $0$ of $1$ (each of which occurs with a probability of $0.5$), a subsequent measurement on the second qubit results in an outcome of $0$ or $1$ (both with a probability of $0.5$). Thus, sequential measurements are perfectly *uncorrelated*.

This aligns with the fact that the Bell state is entangled, while the $\ket{\Theta}$ is separable and can be expressed as $\ket \Theta = \ket + \otimes \ket +$.

[Return to exercise 6 of the Multi-Qubit System Measurements tutorial.](./MultiQubitSystemMeasurements.ipynb#Exercise-6:-Sequential-measurements-on-an-entangled-state-and-a-separable-state)

### Exercise 7: State selection using partial measurements

**Input**:
1. A 2-qubit system in the state
$$
|\psi\rangle = \frac{1}{\sqrt{2}} |0\rangle \otimes ( a |0\rangle + b|1\rangle) + \frac{1}{\sqrt{2}} |1\rangle \otimes (b|0\rangle + a |1\rangle),
$$
where the constants $a$ and $b$ satisfying $|a|^2 + |b|^2 = 1$ are unknown.
2. An integer $ind$ which is either $0$ or $1$.

**Goal**:
- If $ind$ equals 0, the state of the state of the second qubit should be $a|0\rangle + b|1\rangle$
- If $ind$ equals 1, the state of the second qubit should be $b|0\rangle + a |1\rangle$.

The state of the first qubit at the end does not matter.

### Solution
We note that if we measure the first qubit in the computational basis, then an outcome of $0$ collapses the second qubit to the state $a\ket 0 + b \ket 1$, while an outcome of $1$ collapses the second qubit to the state $b\ket 0 + a \ket 1$.

Thus, if $ind=0$ and we measure $0$ or if $ind=1$ and we measure $1$, then after the measurement the second qubit will be in the desired state. On the other hand, if $ind=1$ and we measure $0$, or if $ind=0$ and we measure $1$, then the state of the second qubit after the measurement is not what we're looking for, but we can adjust it using the Pauli X gate.

```qsharp
%kata T3_StateSelction

operation StateSelction(qs : Qubit[], ind : Int) : Unit {
    // It is convenient to convert measurement outcome to an integer
    let outcome = M(qs[0]) == Zero ? 0 | 1;
    if outcome != ind {
        X(qs[1]);
    }
}
```

[Return to exercise 7 of the Multi-Qubit System Measurements tutorial.](./MultiQubitSystemMeasurements.ipynb#Exercise-7:-State-selection-using-partial-measurements)

### Exercise 8: State preparation using partial measurements

**Input:** Two qubits (in an array) which are in the state $\ket{00}$.

**Goal:**  Modify the qubits to the state $\frac{1}{\sqrt{3}} \big(|00\rangle + |01\rangle + |10\rangle\big)$ using post-selection.

### Solution
While it is possible to prepare the state $\ket \psi$ directly using unitary rotations, it is simpler to use post-selection for preparing it. Here, we describe the procedure in more detail below.

Initially we will prepare an equal superposition of all basis states corresponding to the first two qubits by applying the **H** gate to each of them:
$$\frac{1}{2} \big(|00\rangle + |01\rangle + |10\rangle + |11\rangle\big) \otimes \ket 0$$

This state is a superposition of the state we want to prepare, and the $|11\rangle$ state that we would like to discard.

Now, we want to separate the first three basis states from the last one and to store this separation in the extra qubit.
For example, we can keep the state of the extra qubit $|0\rangle$ for the basis states that we want to keep, and switch it to $|1\rangle$ for the basis states that we would like to discard.
A **CCNOT** gate can be used to accomplish this, with the first two qubits used as control qubits and the extra qubit as target.
When the gate is applied, the state of the extra qubit will only change to $|1\rangle$ if both control qubits are in the $|11\rangle$ state, which marks exactly the state that we want to discard:

$$\text{CCNOT}\frac{1}{2} \big(|00\textbf{0}\rangle + |01\textbf{0}\rangle + |10\textbf{0}\rangle + |11\textbf{0}\rangle\big) =
\frac{1}{2}\big(|00\rangle + |01\rangle + |10\rangle \big) \otimes |\textbf{0}\rangle + \frac{1}{2}|11\rangle \otimes |\textbf{1}\rangle $$

Finally we measure just the extra qubit; this causes a partial collapse of the system to the state defined by the measurement result:
* If the result is $|0\rangle$, the system collapses to a state that is a linear combination of basis states which had the extra qubit in state $|0\rangle$, i.e., the two qubits end up in the target state $\frac{1}{\sqrt3}\big(|00\rangle + |01\rangle + |10\rangle\big)$.
* If the result is $|1\rangle$, the system collapses to a state $|11\rangle$, so our goal is not achieved. The good thing is, this only happens in 25% of the cases, and we can just reset our qubits to the $|00\rangle$ state and try again.

> Q# has a built-in [repeat-until-success (RUS) loop](https://docs.microsoft.com/en-us/quantum/user-guide/using-qsharp/control-flow#repeat-until-success-loop), which comes in handy in this case.
> * We will describe the main operations (applying **H** and **CCNOT** gates and the measurement) in the `repeat` part of the loop, which specifies its body.
> * `until` section specifies the condition which will break the loop. In this case the result of the measurement needs to be `Zero` to indicate our success.
> * Finally, the `fixup` section allows us to clean up the results of the loop body execution before trying again if the success criteria is not met. In this case we reset the first two qubits back to the $|00\rangle$ state.

This technique is sometimes called post-selection.

```qsharp
%kata T4_PostSelection

open Microsoft.Quantum.Measurement;

operation PostSelection (qs : Qubit[]) : Unit {
    // Initialize the extra qubit
    use anc = Qubit();
    // Using the repeat-until-success pattern to prepare the right state
    repeat {
        ApplyToEach(H, qs);
        Controlled X(qs, anc);
        let res = MResetZ(anc);
    }
    until (res == Zero)
    fixup {
        ResetAll(qs);
    }
}
```

[Return to exercise 8 of the Multi-Qubit System Measurements tutorial.](./MultiQubitSystemMeasurements.ipynb#Exercise-8:-State-preparation-using-partial-measurements)

### Exercise 9: Two qubit parity measurement

**Inputs**: Two qubits stored in an array which are guaranteed to be either in a superposition of the states $|00\rangle$ and $|11\rangle$ or in a superposition of states $|01\rangle$ and $|10\rangle$.

**Output**: 0 if qubits were in the first superposition, 1 if they were in the second superposition.
*The state of the qubits at the end of the operation should be the same as the starting state.*

### Solution

If we were not asked to maintain the state of the qubits, one approach would be to measure both the qubits separately in the computational basis, and check if the result is the same for both the measurements. If the measurement results are equal, the input state must have been a superposition of $\ket{00}$ and $\ket{11}$, while different measurement outcomes will imply that the input state must have been a superposition of $\ket{01}$ and $\ket{10}$. However, in these measurements we will lose the information about the original superposition states: a state $\alpha |00\rangle + \beta |11\rangle$ will collapse to either $|00\rangle$ or $|11\rangle$, and we won't be able to recover the information about the coefficients $\alpha$ and $\beta$.

We need to measure the *parity* of the state without collapsing it all the way to the basis states. [Pauli measurements](https://docs.microsoft.com/en-us/quantum/concepts/pauli-measurements) can be used for joint measurements involving multiple qubits. For this task we apply the $Z \otimes Z$ measurement on both qubits.

A joint measurement using $Z \otimes Z$ operator can be thought as projecting the measured state to one of the two eigenspaces of $Z \otimes Z$ with $+1$ and $-1$ as the corresponding eigenvalues. The measurement returns `Zero` if the measured state is projected to the space with an eigenvalue of $+1$, and a result of `One` if projected to the space with an eigenvalue of $-1$.

As we've seen in the tutorial, the state $\alpha |00\rangle + \beta |11\rangle$ is an eigenstate of the $Z \otimes Z$ operator with the eigenvalue $+1$, and the state $\alpha |01\rangle + \beta |10\rangle$ is an eigenstate with the eigenvalue $-1$.
Hence, we can use this joint measurement to recognize which of the superposition states we were given while preserving the initial superposition state.

In Q#, the operation [`Measure`](https://docs.microsoft.com/en-us/qsharp/api/qsharp/microsoft.quantum.intrinsic.measure) can be used to measure multiple qubits using an array of [Pauli](https://docs.microsoft.com/en-us/quantum/user-guide/language/types?#primitive-types) constants that define the basis for measurement.

```qsharp
%kata T5_ParityMeasurement

operation ParityMeasurement (qs : Qubit[]) : Int {
    return Measure([PauliZ, PauliZ], qs) == Zero ? 0 | 1;
}
```

[Return to exercise 9 of the Multi-Qubit System Measurements tutorial.](./MultiQubitSystemMeasurements.ipynb#Exercise-9:-Two-qubit-parity-Measurement)

### Exercise 10: Parity measurement in different basis

Consider a system which is in a state $\alpha |00\rangle + \beta |01\rangle + \beta |10\rangle + \alpha |11\rangle$.

What are the possible outcomes and their associated probabilities, if a measurement in an $XX$ Pauli measurement is done?

## Solution

The first step towards identifying the outcomes and their probabilities for joint measurements is to identify the eigenvectors corresponding to eigenvalues $\pm1$ of the Pauli operator. We note that since $X\ket{\pm}= \pm\ket{\pm}$, we have
\begin{align}
XX \ket{++} &= \ket{++}, &XX \ket{--} &= \ket{--};\\
XX \ket{+-} &= -\ket{+-}, &XX \ket{-+} &= -\ket{-+}.
\end{align}
Thus, the $XX$ operator measures the parity in the Hadamard, or the $\ket{\pm}$ basis. That is, it distinguishes basis states with an even number of $+$'s from basis states which have an odd number of $+$'s.

The projector corresponding to a result of `Zero` is given by $P_{+1} = \ket{++}\bra{++} + \ket{--}\bra{--}$, while the projector corresponding to a result of `One` is given by $P_{-1} = \ket{+-}\bra{+-} + \ket{-+}\bra{-+}$. Then, we note that $P_{+1}$ annihilates states with odd parity, while leaving states with even parity unaffected. That is, for any values of the constants
\begin{align}
P_{+1} ( \gamma \ket{++} + \delta \ket{--} ) &= ( \gamma \ket{++} + \delta \ket{--} )\\
P_{+1} ( \mu \ket{-+} + \nu \ket{+-} ) &= 0.
\end{align}
Similarly, $P_{-1}$ annihilates states with even parity, while leaving states with odd parity unaffected.

Now we express the given state in the Hadamard basis. We note that it is possible to go from the computational basis to the Hadamard basis using the following relations
$$
\ket{0} = \frac{1}{\sqrt{2}} \left( \ket{+} + \ket{-} \right)\\
\ket{1} = \frac{1}{\sqrt{2}} \left( \ket{+} - \ket{-} \right).
$$
Using these, we obtain
$$ \alpha |00\rangle + \beta |01\rangle + \beta |10\rangle + \alpha |11\rangle = (\alpha + \beta) |++\rangle + (\alpha - \beta) |--\rangle.$$
Thus, this state has an even parity in the Hadamard basis. It follows that an $XX$ Pauli measurement will result in the outcome `Zero` with probability 1, leaving the state unchanged after the measurement.

[Return to exercise 10 of the Multi-Qubit System Measurements tutorial.](./MultiQubitSystemMeasurements.ipynb#Exercise-10:-Parity-measurement-in-different-basis)


# Logistic Regression

    Notebook version: 2.0 (Nov 21, 2017)
                      2.1 (Oct 19, 2018)

    Author: Jesús Cid Sueiro (jcid@tsc.uc3m.es)
            Jerónimo Arenas García (jarenas@tsc.uc3m.es)

    Changes: v.1.0 - First version
             v.1.1 - Typo correction. Prepared for slide presentation
             v.2.0 - Prepared for Python 3.0 (backcompmatible with 2.7)
                     Assumptions for regression model modified
             v.2.1 - Minor changes regarding notation and assumptions

```python
from __future__ import print_function
# To visualize plots in the notebook
%matplotlib inline

# Imported libraries
import csv
import random
import matplotlib
import matplotlib.pyplot as plt
import pylab

import numpy as np
from mpl_toolkits.mplot3d import Axes3D
from sklearn.preprocessing import PolynomialFeatures
from sklearn import linear_model

```

# Logistic Regression

## 1. Introduction

### 1.1. Binary classification and decision theory. The MAP criterion

The goal of a classification problem is to assign a *class* or *category* to every *instance* or *observation* of a data collection. Here, we will assume that every instance ${\bf x}$ is an $N$-dimensional vector in $\mathbb{R}^N$, and that the class $y$ of sample ${\bf x}$ is an element of a binary set ${\mathcal Y} = \{0, 1\}$. The goal of a classifier is to predict the true value of $y$ after observing ${\bf x}$.

We will denote as $\hat{y}$ the classifier output or *decision*. If $y=\hat{y}$, the decision is a *hit*, otherwise $y\neq \hat{y}$ and the decision is an *error*.

Decision theory provides a solution to the classification problem in situations where the relation between instance ${\bf x}$ and its class $y$ is given by a known probabilistic model: assume that every tuple $({\bf x}, y)$ is an outcome of a random vector $({\bf X}, Y)$ with joint distribution $p_{{\bf X},Y}({\bf x}, y)$. A natural criteria for classification is to select predictor $\hat{Y}=f({\bf x})$ in such a way that the probability or error, $P\{\hat{Y} \neq Y\}$ is minimum. Noting that

$$
P\{\hat{Y} \neq Y\} = \int P\{\hat{Y} \neq Y | {\bf x}\} p_{\bf X}({\bf x}) d{\bf x}
$$

the optimal decision is got if, for every sample ${\bf x}$, we make decision minimizing the conditional error probability:

\begin{align}
\hat{y}^* &= \arg\min_{\hat{y}} P\{\hat{y} \neq Y |{\bf x}\} \\
          &= \arg\max_{\hat{y}} P\{\hat{y} = Y |{\bf x}\} \\
\end{align}

Thus, the optimal decision rule can be expressed as

$$
P_{Y|{\bf X}}(1|{\bf x}) \quad\mathop{\gtrless}^{\hat{y}=1}_{\hat{y}=0}\quad  P_{Y|{\bf X}}(0|{\bf x})
$$

or, equivalently

$$
P_{Y|{\bf X}}(1|{\bf x}) \quad\mathop{\gtrless}^{\hat{y}=1}_{\hat{y}=0}\quad  \frac{1}{2}
$$

The classifier implementing this decision rule is usually named MAP (*Maximum A Posteriori*). As we have seen, the MAP classifier minimizes the error probability for binary classification, but the result can also be generalized to multiclass classification problems.

### 1.2. Parametric classification.

Classical decision theory is grounded on the assumption that the probabilistic model relating the observed sample ${\bf X}$ and the true hypothesis $Y$ is known. Unfortunately, this is unrealistic in many applications, where the only available information to construct the classifier is a dataset $\mathcal D = \{{\bf x}^{(k)}, y^{(k)}\}_{k=0}^{K-1}$ of instances and their respective class labels.

A more realistic formulation of the classification problem is the following: given a dataset $\mathcal D = \{({\bf x}^{(k)}, y^{(k)}) \in {\mathbb{R}}^N \times {\mathcal Y}, \, k=0,\ldots,{K-1}\}$ of independent and identically distributed (i.i.d.) samples from an ***unknown*** distribution $p_{{\bf X},Y}({\bf x}, y)$, predict the class $y$ of a new sample ${\bf x}$ with the minimum probability of error.

Since the probabilistic model generating the data is unknown, the MAP decision rule cannot be applied. However, many classification algorithms use the dataset to obtain an estimate of the posterior class probabilities, and apply it to implement an approximation to the MAP decision maker.

Parametric classifiers based on this idea assume, additionally, that the posterior class probabilty satisfies some parametric formula:

$$
P_{Y|X}(1|{\bf x},{\bf w}) = f_{\bf w}({\bf x})
$$

where ${\bf w}$ is a vector of parameters. Given the expression of the MAP decision maker, classification consists in comparing the value of $f_{\bf w}({\bf x})$ with the threshold $\frac{1}{2}$, and each parameter vector would be associated to a different decision maker.

In practice, the dataset ${\mathcal S}$ is used to select a particular parameter vector $\hat{\bf w}$ according to certain criterion. Accordingly, the decision rule becomes

$$
f_{\hat{\bf w}}({\bf x}) \quad\mathop{\gtrless}^{\hat{y}=1}_{\hat{y}=0}\quad  \frac{1}{2}
$$

In this lesson, we explore one of the most popular model-based parametric classification methods: **logistic regression**.

## 2. Logistic regression.

### 2.1. The logistic function

The logistic regression model assumes that the binary class label $Y \in \{0,1\}$ of observation $X\in \mathbb{R}^N$ satisfies the expression.

$$P_{Y|{\bf X}}(1|{\bf x}, {\bf w}) = g({\bf w}^\intercal{\bf x})$$
$$P_{Y|{\bf,X}}(0|{\bf x}, {\bf w}) = 1-g({\bf w}^\intercal{\bf x})$$

where ${\bf w}$ is a parameter vector and $g(·)$ is the *logistic* function, which is defined by

$$g(t) = \frac{1}{1+\exp(-t)}$$

It is straightforward to see that the logistic function has the following properties:

- **P1**: Probabilistic output: $\quad 0 \le g(t) \le 1$
- **P2**: Symmetry: $\quad g(-t) = 1-g(t)$
- **P3**: Monotonicity: $\quad g'(t) = g(t)·[1-g(t)] \ge 0$

In the following we define a logistic function in python, and use it to plot a graphical representation.

**Exercise 1**: Verify properties P2 and P3.

**Exercise 2**: Implement a function to compute the logistic function, and use it to plot such function in the inverval $[-6,6]$.

```python
# Define the logistic function
def logistic(t):
    #<SOL>
    return 1.0 / (1 + np.exp(-t))
    #</SOL>

# Plot the logistic function
t = np.arange(-6, 6, 0.1)
z = logistic(t)

plt.plot(t, z)
plt.xlabel('$t$', fontsize=14)
plt.ylabel('$g(t)$', fontsize=14)
plt.title('The logistic function')
plt.grid()
```

### 2.2. Classifiers based on the logistic model.

The MAP classifier under a logistic model will have the form

$$P_{Y|{\bf X}}(1|{\bf x}, {\bf w}) = g({\bf w}^\intercal{\bf x}) \quad\mathop{\gtrless}^{\hat{y}=1}_{\hat{y}=0} \quad \frac{1}{2} $$

Therefore

$$
2 \quad\mathop{\gtrless}^{\hat{y}=1}_{\hat{y}=0} \quad
1 + \exp(-{\bf w}^\intercal{\bf x}) $$

which is equivalent to

$${\bf w}^\intercal{\bf x}
\quad\mathop{\gtrless}^{\hat{y}=1}_{\hat{y}=0}\quad
0 $$

Therefore, the classifiers based on the logistic model are given by linear decision boundaries passing through the origin, ${\bf x} = {\bf 0}$.

```python
# Weight vector:
w = [4, 8]   # Try different weights

# Create a rectangular grid.
x_min = -1
x_max = 1
dx = x_max - x_min
h = float(dx) / 200
xgrid = np.arange(x_min, x_max, h)
xx0, xx1 = np.meshgrid(xgrid, xgrid)

# Compute the logistic map for the given weights
Z = logistic(w[0]*xx0 + w[1]*xx1)

# Plot the logistic map
fig = plt.figure()
ax = fig.gca(projection='3d')
ax.plot_surface(xx0, xx1, Z, cmap=plt.cm.copper)
ax.contour(xx0, xx1, Z, levels=[0.5], colors='b', linewidths=(3,))
plt.xlabel('$x_0$')
plt.ylabel('$x_1$')
ax.set_zlabel('P(1|x,w)')

plt.show()
```

The next code fragment represents the output of the same classifier, representing the output of the logistic function in the $x_0$-$x_1$ plane, encoding the value of the logistic function in the representation color.

```python
CS = plt.contourf(xx0, xx1, Z)
CS2 = plt.contour(CS, levels=[0.5],
                  colors='m', linewidths=(3,))
plt.xlabel('$x_0$')
plt.ylabel('$x_1$')

plt.colorbar(CS, ticks=[0, 0.5, 1])

plt.show()
```

### 3.3. Nonlinear classifiers.

The logistic model can be extended to construct non-linear classifiers by using non-linear data transformations. A general form for a nonlinear logistic regression model is

$$P_{Y|{\bf X}}(1|{\bf x}, {\bf w}) = g[{\bf w}^\intercal{\bf z}({\bf x})] $$

where ${\bf z}({\bf x})$ is an arbitrary nonlinear transformation of the original variables. The boundary decision in that case is given by equation

$$
{\bf w}^\intercal{\bf z} = 0
$$

** Exercise 2**: Modify the code above to generate a 3D surface plot of the polynomial logistic regression model given by

$$
P_{Y|{\bf X}}(1|{\bf x}, {\bf w}) = g(1 + 10 x_0 + 10 x_1 - 20 x_0^2 + 5 x_0 x_1 + x_1^2)
$$

```python
# Weight vector:
w = [1, 10, 10, -20, 5, 1]   # Try different weights

# Create a regtangular grid.
x_min = -1
x_max = 1
dx = x_max - x_min
h = float(dx) / 200
xgrid = np.arange(x_min, x_max, h)
xx0, xx1 = np.meshgrid(xgrid, xgrid)

# Compute the logistic map for the given weights
# Z = <FILL IN>
Z = logistic(w[0] + w[1]*xx0 + w[2]*xx1 + w[3]*np.multiply(xx0,xx0) + w[4]*np.multiply(xx0,xx1) + w[5]*np.multiply(xx1,xx1))

# Plot the logistic map
fig = plt.figure()
ax = fig.gca(projection='3d')
ax.plot_surface(xx0, xx1, Z, cmap=plt.cm.copper)
plt.xlabel('$x_0$')
plt.ylabel('$x_1$')
ax.set_zlabel('P(1|x,w)')

plt.show()
```

```python
CS = plt.contourf(xx0, xx1, Z)
CS2 = plt.contour(CS, levels=[0.5],
                  colors='m', linewidths=(3,))
plt.xlabel('$x_0$')
plt.ylabel('$x_1$')

plt.colorbar(CS, ticks=[0, 0.5, 1])

plt.show()
```

## 3. Inference

Remember that the idea of parametric classification is to use the training data set $\mathcal D = \{({\bf x}^{(k)}, y^{(k)}) \in {\mathbb{R}}^N \times \{0,1\}, k=0,\ldots,{K-1}\}$ to set the parameter vector ${\bf w}$ according to certain criterion. Then, the estimate $\hat{\bf w}$ can be used to compute the label prediction for any new observation as

$$\hat{y} = \arg\max_y P_{Y|{\bf X}}(y|{\bf x},\hat{\bf w}).$$

We need still to choose a criterion to optimize with the selection of the parameter vector. In the notebook, we will discuss two different approaches to the estimation of ${\bf w}$:

   * Maximum Likelihood (ML): $\hat{\bf w}_{\text{ML}} = \arg\max_{\bf w} P_{{\mathcal D}|{\bf W}}({\mathcal D}|{\bf w})$
   * Maximum *A Posteriori* (MAP): $\hat{\bf w}_{\text{MAP}} = \arg\max_{\bf w} p_{{\bf W}|{\mathcal D}}({\bf w}|{\mathcal D})$

For the mathematical derivation of the logistic regression algorithm, the following representation of the logistic model will be useful: noting that

$$P_{Y|{\bf X}}(0|{\bf x}, {\bf w}) = 1-g[{\bf w}^\intercal{\bf z}({\bf x})]
= g[-{\bf w}^\intercal{\bf z}({\bf x})]$$

we can write

$$P_{Y|{\bf X}}(y|{\bf x}, {\bf w}) = g[\overline{y}{\bf w}^\intercal{\bf z}({\bf x})]$$

where $\overline{y} = 2y-1$ is a *symmetrized label* ($\overline{y}\in\{-1, 1\}$).

### 3.1. Model assumptions

In the following, we will make the following assumptions:

- **A1**. (Logistic Regression): We assume a logistic model for the *a posteriori* probability of ${Y=1}$ given ${\bf X}$, i.e.,

$$P_{Y|{\bf X}}(1|{\bf x}, {\bf w}) = g[{\bf w}^\intercal{\bf z}({\bf x})].$$

- **A2**. All samples in ${\mathcal D}$ have been generated by the same distribution, $p_{{\bf X}, Y}({\bf x}, y)$.

- **A3**. Input variables $\bf x$ do not depend on $\bf w$. This implies that

$$p({\bf x}|{\bf w}) = p({\bf x})$$

- **A4**. Targets $y^{(0)}, \cdots, y^{(K-1)}$ are statistically independent given $\bf w$ and the inputs ${\bf x}^{(0)}, \cdots, {\bf x}^{(K-1)}$, that is:

$$p(y^{(0)}, \cdots, y^{(K-1)} | {\bf x}^{(0)}, \cdots, {\bf x}^{(K-1)}, {\bf w}) = \prod_{k=0}^{K-1} p(s^{(k)} | {\bf x}^{(k)}, {\bf w})$$

### 3.2. ML estimation.

The ML estimate is defined as

$$\hat{\bf w}_{\text{ML}} = \arg\max_{\bf w} P_{{\mathcal D}|{\bf W}}({\mathcal D}|{\bf w})$$

Ussing assumptions A2 and A3 above, we have that

\begin{align}
P_{{\mathcal D}|{\bf W}}({\mathcal D}|{\bf w}) & =  p(y^{(0)}, \cdots, y^{(K-1)},{\bf x}^{(0)}, \cdots, {\bf x}^{(K-1)}| {\bf w}) \\
& =  P(y^{(0)}, \cdots, y^{(K-1)}|{\bf x}^{(0)}, \cdots, {\bf x}^{(K-1)}, {\bf w}) \; p({\bf x}^{(0)}, \cdots, {\bf x}^{(K-1)}| {\bf w}) \\
& = P(y^{(0)}, \cdots, y^{(K-1)}|{\bf x}^{(0)}, \cdots, {\bf x}^{(K-1)}, {\bf w}) \; p({\bf x}^{(0)}, \cdots, {\bf x}^{(K-1)})\end{align}

Finally, using assumption A4, we can formulate the ML estimation of $\bf w$ as the resolution of the following optimization problem

\begin{align}
\hat {\bf w}_\text{ML} & = \arg \max_{\bf w} p(y^{(0)}, \cdots, y^{(K-1)}|{\bf x}^{(0)}, \cdots, {\bf x}^{(K-1)}, {\bf w}) \\
& = \arg \max_{\bf w} \prod_{k=0}^{K-1} P(y^{(k)}|{\bf x}^{(k)}, {\bf w}) \\
& = \arg \max_{\bf w} \sum_{k=0}^{K-1} \log P(y^{(k)}|{\bf x}^{(k)}, {\bf w}) \\
& = \arg \min_{\bf w} \sum_{k=0}^{K-1} - \log P(y^{(k)}|{\bf x}^{(k)}, {\bf w})
\end{align}

where the arguments of the maximization or minimization problems of the last three lines are usually referred to as the **likelihood**, **log-likelihood** $\left[L(\bf w)\right]$, and **negative log-likelihood** $\left[\text{NLL}(\bf w)\right]$, respectively.

Now, using A1 (the logistic model)

\begin{align}
\text{NLL}({\bf w})
    &= - \sum_{k=0}^{K-1}\log\left[g\left(\overline{y}^{(k)}{\bf w}^\intercal {\bf z}^{(k)}\right)\right]   \\
    &= \sum_{k=0}^{K-1}\log\left[1+\exp\left(-\overline{y}^{(k)}{\bf w}^\intercal {\bf z}^{(k)}\right)\right]
\end{align}

where ${\bf z}^{(k)}={\bf z}({\bf x}^{(k)})$.

It can be shown that $\text{NLL}({\bf w})$ is a convex and differentiable function of ${\bf w}$. Therefore, its minimum is a point with zero gradient.

\begin{align}
\nabla_{\bf w} \text{NLL}(\hat{\bf w}_{\text{ML}})
    &= - \sum_{k=0}^{K-1}
       \frac{\exp\left(-\overline{y}^{(k)}\hat{\bf w}_{\text{ML}}^\intercal {\bf z}^{(k)}\right) \overline{y}^{(k)} {\bf z}^{(k)}}
       {1+\exp\left(-\overline{y}^{(k)}\hat{\bf w}_{\text{ML}}^\intercal {\bf z}^{(k)}
       \right)} = \\
    &= - \sum_{k=0}^{K-1} \left[y^{(k)}-g(\hat{\bf w}_{\text{ML}}^T {\bf z}^{(k)})\right] {\bf z}^{(k)} = 0
\end{align}

Unfortunately, $\hat{\bf w}_{\text{ML}}$ cannot be taken out from the above equation, and some iterative optimization algorithm must be used to search for the minimum.

### 3.2. Gradient descent.

A simple iterative optimization algorithm is  gradient descent.

\begin{align}
{\bf w}_{n+1} = {\bf w}_n - \rho_n \nabla_{\bf w} L({\bf w}_n)
\end{align}

where $\rho_n >0$ is the *learning step*.

Applying the gradient descent rule to logistic regression, we get the following algorithm:

\begin{align}
{\bf w}_{n+1} &= {\bf w}_n
    + \rho_n \sum_{k=0}^{K-1} \left[y^{(k)}-g({\bf w}_n^\intercal {\bf z}^{(k)})\right] {\bf z}^{(k)}
\end{align}

Defining vectors

\begin{align}
{\bf y} &= [y^{(0)},\ldots,y^{(K-1)}]^\intercal \\
\hat{\bf p}_n &= [g({\bf w}_n^\intercal {\bf z}^{(0)}), \ldots, g({\bf w}_n^\intercal {\bf z}^{(K-1)})]^\intercal
\end{align}
and matrix
\begin{align}
{\bf Z} = \left[{\bf z}^{(0)},\ldots,{\bf z}^{(K-1)}\right]^\intercal
\end{align}

we can write

\begin{align}
{\bf w}_{n+1} &= {\bf w}_n
    + \rho_n {\bf Z}^\intercal \left({\bf y}-\hat{\bf p}_n\right)
\end{align}

In the following, we will explore the behavior of the gradient descend method using the Iris Dataset.

#### 3.2.1 Example: Iris Dataset.

As an illustration, consider the  Iris dataset , taken from the  UCI Machine Learning repository. This data set contains 3 classes of 50 instances each, where each class refers to a type of iris plant (*setosa*, *versicolor* or *virginica*). Each instance contains 4 measurements of given flowers: sepal length, sepal width, petal length and petal width, all in centimeters.

We will try to fit the logistic regression model to discriminate between two classes using only two attributes.

First, we load the dataset and split them in training and test subsets.

```python
# Adapted from a notebook by Jason Brownlee
def loadDataset(filename, split):
    xTrain = []
    cTrain = []
    xTest = []
    cTest = []

    with open(filename, 'r') as csvfile:
        lines = csv.reader(csvfile)
        dataset = list(lines)
    for i in range(len(dataset)-1):
        for y in range(4):
            dataset[i][y] = float(dataset[i][y])
        item = dataset[i]
        if random.random() < split:
            xTrain.append(item[0:4])
            cTrain.append(item[4])
        else:
            xTest.append(item[0:4])
            cTest.append(item[4])
    return xTrain, cTrain, xTest, cTest

xTrain_all, cTrain_all, xTest_all, cTest_all = loadDataset('iris.data', 0.66)
nTrain_all = len(xTrain_all)
nTest_all = len(xTest_all)
print('Train:', nTrain_all)
print('Test:', nTest_all)
```

    Train: 98
    Test: 52

Now, we select two classes and two attributes.

```python
# Select attributes
i = 0 # Try 0,1,2,3
j = 1 # Try 0,1,2,3 with j!=i

# Select two classes
c0 = 'Iris-versicolor'
c1 = 'Iris-virginica'

# Select two coordinates
ind = [i, j]

# Take training test
X_tr = np.array([[xTrain_all[n][i] for i in ind] for n in range(nTrain_all)
                  if cTrain_all[n]==c0 or cTrain_all[n]==c1])
C_tr = [cTrain_all[n] for n in range(nTrain_all)
          if cTrain_all[n]==c0 or cTrain_all[n]==c1]
Y_tr = np.array([int(c==c1) for c in C_tr])
n_tr = len(X_tr)

# Take test set
X_tst = np.array([[xTest_all[n][i] for i in ind] for n in range(nTest_all)
                 if cTest_all[n]==c0 or cTest_all[n]==c1])
C_tst = [cTest_all[n] for n in range(nTest_all)
         if cTest_all[n]==c0 or cTest_all[n]==c1]
Y_tst = np.array([int(c==c1) for c in C_tst])
n_tst = len(X_tst)
```

#### 3.2.2. Data normalization

Normalization of data is a common pre-processing step in many machine learning algorithms. Its goal is to get a dataset where all input coordinates have a similar scale. Learning algorithms usually show less instabilities and convergence problems when data are normalized.

We will define a normalization function that returns a training data matrix with zero sample mean and unit sample variance.

```python
def normalize(X, mx=None, sx=None):

    # Compute means and standard deviations
    if mx is None:
        mx = np.mean(X, axis=0)
    if sx is None:
        sx = np.std(X, axis=0)

    # Normalize
    X0 = (X-mx)/sx

    return X0, mx, sx
```

Now, we can normalize training and test data. Observe in the code that the same transformation should be applied to training and test data. This is the reason why normalization with the test data is done using the means and the variances computed with the training set.

```python
# Normalize data
Xn_tr, mx, sx = normalize(X_tr)
Xn_tst, mx, sx = normalize(X_tst, mx, sx)
```

The following figure generates a plot of the normalized training data.

```python
# Separate components of x into different arrays (just for the plots)
x0c0 = [Xn_tr[n][0] for n in range(n_tr) if Y_tr[n]==0]
x1c0 = [Xn_tr[n][1] for n in range(n_tr) if Y_tr[n]==0]
x0c1 = [Xn_tr[n][0] for n in range(n_tr) if Y_tr[n]==1]
x1c1 = [Xn_tr[n][1] for n in range(n_tr) if Y_tr[n]==1]

# Scatterplot.
labels = {'Iris-setosa': 'Setosa',
          'Iris-versicolor': 'Versicolor',
          'Iris-virginica': 'Virginica'}
plt.plot(x0c0, x1c0,'r.', label=labels[c0])
plt.plot(x0c1, x1c1,'g+', label=labels[c1])
plt.xlabel('$x_' + str(ind[0]) + '$')
plt.ylabel('$x_' + str(ind[1]) + '$')
plt.legend(loc='best')
plt.axis('equal')

plt.show()
```

In order to apply the gradient descent rule, we need to define two methods:
 - A `fit` method, that receives the training data and returns the model weights and the value of the negative log-likelihood during all iterations.
 - A `predict` method, that receives the model weight and a set of inputs, and returns the posterior class probabilities for that input, as well as their corresponding class predictions.

```python
def logregFit(Z_tr, Y_tr, rho, n_it):

    # Data dimension
    n_dim = Z_tr.shape[1]

    # Initialize variables
    nll_tr = np.zeros(n_it)
    pe_tr = np.zeros(n_it)
    Y_tr2 = 2*Y_tr - 1     # Transform labels into binary symmetric.
    w = np.random.randn(n_dim,1)

    # Running the gradient descent algorithm
    for n in range(n_it):

        # Compute posterior probabilities for weight w
        p1_tr = logistic(np.dot(Z_tr, w))

        # Compute negative log-likelihood
        # (note that this is not required for the weight update, only for nll tracking)
        nll_tr[n] = np.sum(np.log(1 + np.exp(-np.dot(Y_tr2*Z_tr, w))))

        # Update weights
        w += rho*np.dot(Z_tr.T, Y_tr - p1_tr)

    return w, nll_tr

def logregPredict(Z, w):

    # Compute posterior probability of class 1 for weights w.
    p = logistic(np.dot(Z, w)).flatten()

    # Class
    D = [int(round(pn)) for pn in p]

    return p, D
```

We can test the behavior of the gradient descent method by fitting a logistic regression model with ${\bf z}({\bf x}) = (1, {\bf x}^\intercal)^\intercal$.

```python
# Parameters of the algorithms
rho = float(1)/50    # Learning step
n_it = 200   # Number of iterations

# Compute Z's
Z_tr = np.c_[np.ones(n_tr), Xn_tr]
Z_tst = np.c_[np.ones(n_tst), Xn_tst]
n_dim = Z_tr.shape[1]

# Convert target arrays to column vectors
Y_tr2 = Y_tr[np.newaxis].T
Y_tst2 = Y_tst[np.newaxis].T

# Running the gradient descent algorithm
w, nll_tr = logregFit(Z_tr, Y_tr2, rho, n_it)

# Classify training and test data
p_tr, D_tr = logregPredict(Z_tr, w)
p_tst, D_tst = logregPredict(Z_tst, w)

# Compute error rates
E_tr = D_tr!=Y_tr
E_tst = D_tst!=Y_tst

# Error rates
pe_tr = float(sum(E_tr)) / n_tr
pe_tst = float(sum(E_tst)) / n_tst

# NLL plot.
plt.plot(range(n_it), nll_tr,'b.:', label='Train')
plt.xlabel('Iteration')
plt.ylabel('Negative Log-Likelihood')
plt.legend()

print('The optimal weights are:')
print(w)
print('The final error rates are:')
print('- Training:', pe_tr)
print('- Test:', pe_tst)
print('The NLL after training is', nll_tr[len(nll_tr)-1])
```

#### 3.2.3. Free parameters

Under certain conditions, the gradient descent method can be shown to converge asymptotically (i.e. as the number of iterations goes to infinity) to the ML estimate of the logistic model. However, in practice, the final estimate of the weights ${\bf w}$ depend on several factors:

- Number of iterations
- Initialization
- Learning step

**Exercise**: Visualize the variability of gradient descent caused by initializations. To do so, fix the number of iterations to 200 and the learning step, and execute the gradient descent 100 times, storing the training error rate of each execution. Plot the histogram of the error rate values.

Note that you can do this exercise with a loop over the 100 executions, including the code in the previous code slide inside the loop, with some proper modifications. To plot a histogram of the values in array `p` with `n`bins, you can use `plt.hist(p, n)`

##### 3.2.3.1. Learning step

The learning step, $\rho$, is a free parameter of the algorithm. Its choice is critical for the convergence of the algorithm. Too large values of $\rho$ make the algorithm diverge. For too small values, the convergence gets very slow and more iterations are required for a good convergence.

**Exercise 3**: Observe the evolution of the negative log-likelihood with the number of iterations for different values of $\rho$. It is easy to check that, for large enough $\rho$, the gradient descent method does not converge. Can you estimate (through manual observation) an approximate value of $\rho$ stating a boundary between convergence and divergence?

**Exercise 4**: In this exercise we explore the influence of the learning step more sistematically. Use the code in the previouse exercises to compute, for every value of $\rho$, the average error rate over 100 executions. Plot the average error rate vs. $\rho$.

Note that you should explore the values of $\rho$ in a logarithmic scale. For instance, you can take $\rho = 1, 1/10, 1/100, 1/1000, \ldots$

In practice, the selection of $\rho$ may be a matter of trial an error. Also there is some theoretical evidence that the learning step should decrease along time up to cero, and the sequence $\rho_n$ should satisfy two conditions:
- C1: $\sum_{n=0}^{\infty} \rho_n^2 < \infty$ (decrease slowly)
- C2: $\sum_{n=0}^{\infty} \rho_n = \infty$ (but not too slowly)

For instance, we can take $\rho_n= 1/n$. Another common choice is $\rho_n = \alpha/(1+\beta n)$ where $\alpha$ and $\beta$ are also free parameters that can be selected by trial and error with some heuristic method.

#### 3.2.4. Visualizing the posterior map.

We can also visualize the posterior probability map estimated by the logistic regression model for the estimated weights.

```python
# Create a regtangular grid.
x_min, x_max = Xn_tr[:, 0].min(), Xn_tr[:, 0].max()
y_min, y_max = Xn_tr[:, 1].min(), Xn_tr[:, 1].max()
dx = x_max - x_min
dy = y_max - y_min
h = dy /400
xx, yy = np.meshgrid(np.arange(x_min - 0.1 * dx, x_max + 0.1 * dx, h),
                     np.arange(y_min - 0.1 * dx, y_max + 0.1 * dy, h))
X_grid = np.array([xx.ravel(), yy.ravel()]).T

# Compute Z's
Z_grid = np.c_[np.ones(X_grid.shape[0]), X_grid]

# Compute the classifier output for all samples in the grid.
pp, dd = logregPredict(Z_grid, w)

# Paint output maps
pylab.rcParams['figure.figsize'] = 6, 6  # Set figure size

# Put the result into a color plot
plt.plot(x0c0, x1c0,'r.', label=labels[c0])
plt.plot(x0c1, x1c1,'g+', label=labels[c1])
plt.xlabel('$x_' + str(ind[0]) + '$')
plt.ylabel('$x_' + str(ind[1]) + '$')
plt.legend(loc='best')
plt.axis('equal')
pp = pp.reshape(xx.shape)
CS = plt.contourf(xx, yy, pp, cmap=plt.cm.copper)
plt.contour(xx, yy, pp, levels=[0.5],
                  colors='b', linewidths=(3,))

plt.colorbar(CS, ticks=[0, 0.5, 1])

plt.show()
```

#### 3.2.5. Polynomial Logistic Regression

The error rates of the logistic regression model can be potentially reduced by using polynomial transformations.

To compute the polynomial transformation up to a given degree, we can use the `PolynomialFeatures` method in `sklearn.preprocessing`.

```python
# Parameters of the algorithms
rho = float(1)/50    # Learning step
n_it = 500   # Number of iterations
g = 5 # Degree of polynomial

# Compute Z_tr
poly = PolynomialFeatures(degree=g)
Z_tr = poly.fit_transform(Xn_tr)
# Normalize columns (this is useful to make algorithms more stable).)
Zn, mz, sz = normalize(Z_tr[:,1:])
Z_tr = np.concatenate((np.ones((n_tr,1)), Zn), axis=1)

# Compute Z_tst
Z_tst = poly.fit_transform(Xn_tst)
Zn, mz, sz = normalize(Z_tst[:,1:], mz, sz)
Z_tst = np.concatenate((np.ones((n_tst,1)), Zn), axis=1)

# Convert target arrays to column vectors
Y_tr2 = Y_tr[np.newaxis].T
Y_tst2 = Y_tst[np.newaxis].T

# Running the gradient descent algorithm
w, nll_tr = logregFit(Z_tr, Y_tr2, rho, n_it)

# Classify training and test data
p_tr, D_tr = logregPredict(Z_tr, w)
p_tst, D_tst = logregPredict(Z_tst, w)

# Compute error rates
E_tr = D_tr!=Y_tr
E_tst = D_tst!=Y_tst

# Error rates
pe_tr = float(sum(E_tr)) / n_tr
pe_tst = float(sum(E_tst)) / n_tst

# NLL plot.
plt.plot(range(n_it), nll_tr,'b.:', label='Train')
plt.xlabel('Iteration')
plt.ylabel('Negative Log-Likelihood')
plt.legend()

print('The optimal weights are:')
print(w)
print('The final error rates are:')
print('- Training:', pe_tr)
print('- Test:', pe_tst)
print('The NLL after training is', nll_tr[len(nll_tr)-1])
```

Visualizing the posterior map we can se that the polynomial transformation produces nonlinear decision boundaries.

```python
# Compute Z_grid
Z_grid = poly.fit_transform(X_grid)
n_grid = Z_grid.shape[0]
Zn, mz, sz = normalize(Z_grid[:,1:], mz, sz)
Z_grid = np.concatenate((np.ones((n_grid,1)), Zn), axis=1)

# Compute the classifier output for all samples in the grid.
pp, dd = logregPredict(Z_grid, w)
pp = pp.reshape(xx.shape)

# Paint output maps
pylab.rcParams['figure.figsize'] = 6, 6  # Set figure size

plt.plot(x0c0, x1c0,'r.', label=labels[c0])
plt.plot(x0c1, x1c1,'g+', label=labels[c1])
plt.xlabel('$x_' + str(ind[0]) + '$')
plt.ylabel('$x_' + str(ind[1]) + '$')
plt.axis('equal')
plt.legend(loc='best')
CS = plt.contourf(xx, yy, pp, cmap=plt.cm.copper)
plt.contour(xx, yy, pp, levels=[0.5],
                  colors='b', linewidths=(3,))

plt.colorbar(CS, ticks=[0, 0.5, 1])

plt.show()
```

## 4. Regularization and MAP estimation.

An alternative to the ML estimation of the weights in logistic regression is Maximum A Posteriori estimation. Modelling the logistic regression weights as a random variable with prior distribution $p_{\bf W}({\bf w})$, the MAP estimate is defined as

$$
\hat{\bf w}_{\text{MAP}} = \arg\max_{\bf w} p({\bf w}|{\mathcal D})
$$

The posterior density $p({\bf w}|{\mathcal D})$ is related to the likelihood function and the prior density of the weights, $p_{\bf W}({\bf w})$ through the Bayes rule

$$
p({\bf w}|{\mathcal D}) =
    \frac{P\left({\mathcal D}|{\bf w}\right) \; p_{\bf W}({\bf w})}
         {p\left({\mathcal D}\right)}
$$

In general, the denominator in this expression cannot be computed analytically. However, it is not required for MAP estimation because it does not depend on ${\bf w}$. Therefore, the MAP solution is given by

\begin{align}
\hat{\bf w}_{\text{MAP}} & = \arg\max_{\bf w} P\left({\mathcal D}|{\bf w}\right) \; p_{\bf W}({\bf w}) \\
& = \arg\max_{\bf w} \left\{ L({\mathbf w}) + \log p_{\bf W}({\bf w})\right\} \\
& = \arg\min_{\bf w} \left\{ \text{NLL}({\mathbf w}) - \log p_{\bf W}({\bf w})\right\}
\end{align}

In the light of this expression, we can conclude that the MAP solution is affected by two terms:

   - The likelihood, which takes large values for parameter vectors $\bf w$ that fit well the training data
   - The prior distribution of weights $p_{\bf W}({\bf w})$, which expresses our *a priori* preference for some solutions. Usually, we recur to prior distributions that take large values when $\|{\bf w}\|$ is small (associated to smooth classification borders).

We can check that the MAP criterion adds a penalty term to the ML objective, that penalizes parameter vectors for which the prior distribution of weights takes small values.

### 4.1 MAP estimation with Gaussian prior

If we assume that ${\bf W}$ is a zero-mean Gaussian random variable with variance matrix $v{\bf I}$,

$$
p_{\bf W}({\bf w}) = \frac{1}{(2\pi v)^{N/2}} \exp\left(-\frac{1}{2v}\|{\bf w}\|^2\right)
$$

the MAP estimate becomes

\begin{align}
\hat{\bf w}_{\text{MAP}}
  &= \arg\min_{\bf w} \left\{L({\bf w}) + \frac{1}{C}\|{\bf w}\|^2
         \right\}
\end{align}

where $C = 2v$. Noting that

$$\nabla_{\bf w}\left\{L({\bf w}) + \frac{1}{C}\|{\bf w}\|^2\right\}
= - {\bf Z} \left({\bf y}-\hat{\bf p}_n\right) + \frac{2}{C}{\bf w},
$$

we obtain the following gradient descent rule for MAP estimation

\begin{align}
{\bf w}_{n+1} &= \left(1-\frac{2\rho_n}{C}\right){\bf w}_n
    + \rho_n {\bf Z} \left({\bf y}-\hat{\bf p}_n\right)
\end{align}

### 4.2 MAP estimation with Laplacian prior

If we assume that ${\bf W}$ follows a multivariate zero-mean Laplacian distribution given by

$$
p_{\bf W}({\bf w}) = \frac{1}{(2 C)^{N}} \exp\left(-\frac{1}{C}\|{\bf w}\|_1\right)
$$

(where $\|{\bf w}\|=|w_1|+\ldots+|w_N|$ is the $L_1$ norm of ${\bf w}$), the MAP estimate is

\begin{align}
\hat{\bf w}_{\text{MAP}}
  &= \arg\min_{\bf w} \left\{L({\bf w}) + \frac{1}{C}\|{\bf w}\|_1
         \right\}
\end{align}

The additional term introduced by the prior in the optimization algorithm is usually named the *regularization term*. It is usually very effective to avoid overfitting when the dimension of the weight vectors is high. Parameter $C$ is named the *inverse regularization strength*.

**Exercise 5**: Derive the gradient descent rules for MAP estimation of the logistic regression weights with Laplacian prior.

## 5. Other optimization algorithms

### 5.1. Stochastic Gradient descent.

Stochastic gradient descent (SGD) is based on the idea of using a single sample at each iteration of the learning algorithm. The SGD rule for ML logistic regression is

\begin{align}
{\bf w}_{n+1} &= {\bf w}_n
    + \rho_n {\bf z}^{(n)} \left(y^{(n)}-\hat{p}^{(n)}_n\right)
\end{align}

Once all samples in the training set have been applied, the algorith can continue by applying the training set several times.

The computational cost of each iteration of SGD is much smaller than that of gradient descent, though it usually needs more iterations to converge.

**Exercise 6**: Modify logregFit to implement an algorithm that applies the SGD rule.

### 5.2. Newton's method

Assume that the function to be minimized, $C({\bf w})$, can be approximated by its second order Taylor series expansion around ${\bf w}_0$

$$
C({\bf w}) \approx C({\bf w}_0)
+ \nabla_{\bf w}^\intercal C({\bf w}_0)({\bf w}-{\bf w}_0)
+ \frac{1}{2}({\bf w}-{\bf w}_0)^\intercal{\bf H}({\bf w}_0)({\bf w}-{\bf w}_0)
$$

where ${\bf H}({\bf w}_k)$ is the  *Hessian* matrix of $C$ at ${\bf w}_k$. Taking the gradient of $C({\bf w})$, and setting the result to ${\bf 0}$, the minimum of C around ${\bf w}_0$ can be approximated as

$$
{\bf w}^* = {\bf w}_0 - {\bf H}({\bf w}_0)^{-1} \nabla_{\bf w}^\intercal C({\bf w}_0)
$$

Since the second order polynomial is only an approximation to $C$, ${\bf w}^*$ is only an approximation to the optimal weight vector, but we can expect ${\bf w}^*$ to be closer to the minimizer of $C$ than ${\bf w}_0$. Thus, we can repeat the process, computing a second order approximation around ${\bf w}^*$ and a new approximation to the minimizer.

 Newton's method is based on this idea. At each optization step, the function to be minimized is approximated by a second order approximation using a Taylor series expansion around the current estimate. As a result, the learning rules becomes

$$\hat{\bf w}_{n+1} = \hat{\bf w}_{n} - \rho_n {\bf H}({\bf w}_k)^{-1} \nabla_{{\bf w}}C({\bf w}_k)
$$

For instance, for the MAP estimate with Gaussian prior, the *Hessian* matrix becomes

$$
{\bf H}({\bf w})
  = \frac{2}{C}{\bf I} + \sum_{k=1}^K f({\bf w}^T {\bf z}^{(k)}) \left(1-f({\bf w}^T {\bf z}^{(k)})\right){\bf z}^{(k)} ({\bf z}^{(k)})^\intercal
$$

Defining diagonal matrix

$$
{\mathbf S}({\bf w}) = \text{diag}\left(f({\bf w}^T {\bf z}^{(k)}) \left(1-f({\bf w}^T {\bf z}^{(k)})\right)\right)
$$

the Hessian matrix can be written in more compact form as

$$
{\bf H}({\bf w})
  = \frac{2}{C}{\bf I} + {\bf Z}^\intercal {\bf S}({\bf w}) {\bf Z}
$$

Therefore, the Newton's algorithm for logistic regression becomes

\begin{align}
\hat{\bf w}_{n+1} = \hat{\bf w}_{n} +
\rho_n
\left(\frac{2}{C}{\bf I} + {\bf Z}^\intercal {\bf S}(\hat{\bf w}_{n})
{\bf Z}
\right)^{-1}
{\bf Z}^\intercal \left({\bf y}-\hat{\bf p}_n\right)
\end{align}

Some variants of the Newton method are implemented in the  Scikit-learn  package.

```python
def logregFit2(Z_tr, Y_tr, rho, n_it, C=1e4):

    # Compute Z's
    r = 2.0/C
    n_dim = Z_tr.shape[1]

    # Initialize variables
    nll_tr = np.zeros(n_it)
    pe_tr = np.zeros(n_it)
    w = np.random.randn(n_dim,1)

    # Running the gradient descent algorithm
    for n in range(n_it):
        p_tr = logistic(np.dot(Z_tr, w))

        sk = np.multiply(p_tr, 1-p_tr)
        S = np.diag(np.ravel(sk.T))

        # Compute negative log-likelihood
        nll_tr[n] = - np.dot(Y_tr.T, np.log(p_tr)) - np.dot((1-Y_tr).T, np.log(1-p_tr))

        # Update weights
        invH = np.linalg.inv(r*np.identity(n_dim) + np.dot(Z_tr.T, np.dot(S, Z_tr)))

        w += rho*np.dot(invH, np.dot(Z_tr.T, Y_tr - p_tr))

    return w, nll_tr
```

```python
# Parameters of the algorithms
rho = float(1)/50    # Learning step
n_it = 500   # Number of iterations
C = 1000
g = 4

# Compute Z_tr
poly = PolynomialFeatures(degree=g)
Z_tr = poly.fit_transform(X_tr)
# Normalize columns (this is useful to make algorithms more stable).)
Zn, mz, sz = normalize(Z_tr[:,1:])
Z_tr = np.concatenate((np.ones((n_tr,1)), Zn), axis=1)

# Compute Z_tst
Z_tst = poly.fit_transform(X_tst)
Zn, mz, sz = normalize(Z_tst[:,1:], mz, sz)
Z_tst = np.concatenate((np.ones((n_tst,1)), Zn), axis=1)

# Convert target arrays to column vectors
Y_tr2 = Y_tr[np.newaxis].T
Y_tst2 = Y_tst[np.newaxis].T

# Running the gradient descent algorithm
w, nll_tr = logregFit2(Z_tr, Y_tr2, rho, n_it, C)

# Classify training and test data
p_tr, D_tr = logregPredict(Z_tr, w)
p_tst, D_tst = logregPredict(Z_tst, w)

# Compute error rates
E_tr = D_tr!=Y_tr
E_tst = D_tst!=Y_tst

# Error rates
pe_tr = float(sum(E_tr)) / n_tr
pe_tst = float(sum(E_tst)) / n_tst

# NLL plot.
plt.plot(range(n_it), nll_tr,'b.:', label='Train')
plt.xlabel('Iteration')
plt.ylabel('Negative Log-Likelihood')
plt.legend()

print('The final error rates are:')
print('- Training:', str(pe_tr))
print('- Test:', str(pe_tst))
print('The NLL after training is:', str(nll_tr[len(nll_tr)-1]))
```

## 6. Logistic regression in Scikit Learn.

The  scikit-learn  package includes an efficient implementation of  logistic regression. To use it, we must first create a classifier object, specifying the parameters of the logistic regression algorithm.

```python
# Create a logistic regression object.
LogReg = linear_model.LogisticRegression(C=1.0)

# Compute Z_tr
poly = PolynomialFeatures(degree=g)
Z_tr = poly.fit_transform(Xn_tr)
# Normalize columns (this is useful to make algorithms more stable).)
Zn, mz, sz = normalize(Z_tr[:,1:])
Z_tr = np.concatenate((np.ones((n_tr,1)), Zn), axis=1)

# Compute Z_tst
Z_tst = poly.fit_transform(Xn_tst)
Zn, mz, sz = normalize(Z_tst[:,1:], mz, sz)
Z_tst = np.concatenate((np.ones((n_tst,1)), Zn), axis=1)

# Fit model to data.
LogReg.fit(Z_tr, Y_tr)

# Classify training and test data
D_tr = LogReg.predict(Z_tr)
D_tst = LogReg.predict(Z_tst)

# Compute error rates
E_tr = D_tr!=Y_tr
E_tst = D_tst!=Y_tst

# Error rates
pe_tr = float(sum(E_tr)) / n_tr
pe_tst = float(sum(E_tst)) / n_tst

print('The final error rates are:')
print('- Training:', str(pe_tr))
print('- Test:', str(pe_tst))

# Compute Z_grid
Z_grid = poly.fit_transform(X_grid)
n_grid = Z_grid.shape[0]
Zn, mz, sz = normalize(Z_grid[:,1:], mz, sz)
Z_grid = np.concatenate((np.ones((n_grid,1)), Zn), axis=1)

# Compute the classifier output for all samples in the grid.
dd = LogReg.predict(Z_grid)
pp = LogReg.predict_proba(Z_grid)[:,1]
pp = pp.reshape(xx.shape)

# Paint output maps
pylab.rcParams['figure.figsize'] = 6, 6  # Set figure size

plt.plot(x0c0, x1c0,'r.', label=labels[c0])
plt.plot(x0c1, x1c1,'g+', label=labels[c1])
plt.xlabel('$x_' + str(ind[0]) + '$')
plt.ylabel('$x_' + str(ind[1]) + '$')
plt.axis('equal')

plt.contourf(xx, yy, pp, cmap=plt.cm.copper)
plt.legend(loc='best')

plt.contour(xx, yy, pp, levels=[0.5],
                  colors='b', linewidths=(3,))

plt.colorbar(CS, ticks=[0, 0.5, 1])

plt.show()
```


```python
#Importing the important libraries
%matplotlib notebook
#The Schrodinger solver by Matthew Srnec and Shiv Upadhyay will be used
from utils3 import * #This is another file with functions in it to clean up the plotting.
import numpy as np
from IPython.display import clear_output
import ipywidgets as widgets
# atomic units
hbar=1.0
m=1.0
```

## Atomic potential

The potential for an atom is the coulomb potential is

\begin{equation}
U_{atom} = -\frac{e^2}{4\pi\epsilon_0 r}
\end{equation}

in atomic units it is even easier

\begin{equation}
U_{atom} = -\frac{1}{r}
\end{equation}

The solutions to the coulomb potential in atomic units are

\begin{equation}
E_n = -\frac{1}{2n^2}
\end{equation}

with the radial wavefunctions

\begin{eqnarray}
\psi_{1s,\; l=0}=&2e^{-r} \\
\psi_{2s,\; l=0}=&\frac{1}{2\sqrt{2}}(2-r)e^{-r/2} \\
\psi_{2s,\; l=1}=&\frac{1}{2\sqrt{2}}e^{-r/2} \\
\psi_{3s,\; l=0}=&\frac{2}{81\sqrt{3}}(27-18r+2r^2)e^{-r/3} \\
\psi_{3s,\; l=1}=&\frac{4}{81\sqrt{6}}r(6-r)e^{-r/3} \\
\psi_{3s,\; l=2}=&\frac{4}{81\sqrt{30}}r^2e^{-r/2}
\end{eqnarray}

```python
x = np.linspace(-50,50,num=1000)
r = abs(x)
scale=0.15

U_atom = -1/r

plt.figure(figsize=(4,3))
plt.plot(x,U_atom,'k-')
plt.axhline(-1/2/1**2,linestyle='dashed')
plt.plot(x,scale*2*np.exp(-r)-(1/2/1**2))
plt.axhline(-1/2/2**2,linestyle='dashed')
plt.plot(x,scale*1/2/np.sqrt(2)*(2-r)*np.exp(-r/2)-(1/2/2**2))
plt.axhline(-1/2/3**2,linestyle='dashed')
plt.plot(x,scale*2/81/np.sqrt(3)*(27-18*r+2*r**2)*np.exp(-r/3)-(1/2/3**2))
plt.ylim(-1,0.2)
plt.show()
```

    <IPython.core.display.Javascript object>

## Quantum interference

Let's consider two atoms interacting with each other separted by one bohr radius

```python
x = np.linspace(-20,20,num=1000)
shift = 1

r = abs(x-shift)
U_atom_1 = -1/r

zeta = 1.0

psi_1 = (1/np.pi)**(0.5)*np.exp(-r)

shift = -1
r = abs(x-shift)
U_atom_2 = -1/r

psi_2 = (1/np.pi)**(0.5)*np.exp(-r)

plt.figure(figsize=(4,3))
plt.axhline(-1/2/1**2,linestyle='dashed')
plt.plot(x,psi_1+psi_2-(1/2/1**2))
#plt.plot(x,psi_STO_1**2+psi_STO_2**2-(1/2/1**2))
plt.plot(x,U_atom_1+U_atom_2)
plt.ylim(-3,0.2)
```

    <IPython.core.display.Javascript object>

    (-3.0, 0.2)

Let's compare the classical case of two charge densities interacting psi_1$^2$+psi_2$^2$ and the quantum case of constructively interfering and destructively interfering, (psi_1+psi_2)$^2$ and (psi_1-psi_2)$^2$.

```python
plt.figure(figsize=(6,3))
plt.plot(x,psi_1**2+psi_2**2,'k',label="Classical")
plt.plot(x,(psi_1+psi_2)**2,'b',label="Quantum +ve")
plt.plot(x,(psi_1-psi_2)**2,'r',label="Quantum -ve")
plt.legend()
plt.xlim((-5,5))
plt.show()
```

    <IPython.core.display.Javascript object>

Let's expand the equation for the constructive interfering case (psi_1+psi_2)$^2$ and explain how the equation gives the classical terms and interference term. Discuss how the increase in charge density in between the atomic nuclei enables bonding between the positively charged nuclei.

\begin{equation}
\psi_{tot} = N(\psi_1+\psi_2)^2
\end{equation}

\begin{equation}
\psi_{tot} = N(\psi_1+\psi_2)(\psi_1+\psi_2)
\end{equation}

\begin{equation}
\psi_{tot} = N(\psi_1\psi_1+\psi_2\psi_2+2\psi_1\psi_2)
\end{equation}

\begin{equation}
\psi_{tot} = N(\psi_1^2+\psi_2^2+2\psi_1\psi_2)
\end{equation}

The first two terms are the classical charge densities they are what you expectt if no quantum effects are occuring. The $2\psi_1\psi_2$ is the quantum interference term and means an increase in the electron density between the atoms.

```python

```


# Data Visualization Tutorial/Cheat Sheet

Python's *matplotlib* has amazing data visualization and charting capabilities. However, when I want to draw a chart, I often find it a struggle to plot the chart that I want. To make a good plot, not only do you need to know the charting commands, but you also need to know arrange your data intelligently. Since I don't want to become frustrated with two different problems to create the plot, I created this so that I can just copy paste the code I need for the plot and isntead to focus on the data massaging. I compiled codes from various souces including the matplotlib tutorials. I have added some additional notes or comments that was helpful for me to do my own plots. This is NOT original. It is a cheat sheet, so the contents have some copy/paste from other source, with modifications. The data in these charts are made up and not real data. To that end, drawing any inference from these plots are worthless.

```python
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
```

```python
#!wget https://matplotlib.org/stable/_images/anatomy.png
```

# Part of a Figure

**Figure:** Think of it as the canvas. This is where you can have one or more plots.

**Axes:** The Axes is the plot. The plot contains the labels, titles, axis labels, the data points, curves, etc.

**Axis:** This the number lines. For example, the x and y coordinates are called axis.

**Subplot**: A figure can have multiple sub-plots.

**Artists:** For some reason they call artists everything that is on the figure.

The definition I provided are a contextual or illustrative definition. These names also have a python object meaning.

All of plotting functions expect *numpy.array* or *numpy.ma.masked_array* as input. Classes that are 'array-like' such as pandas data objects and numpy.matrix may or may not work as intended. It is best to convert these to numpy.array objects prior to plotting.

```python
#For example, to convert a pandas.DataFrame
a = pd.DataFrame(np.random.rand(4, 5), columns = list('abcde'))
a_asarray = a.values
```

```python
#To convert a numpy.matrix
b = np.matrix([[1, 2], [3, 4]])
b_asarray = np.asarray(b)
```

## The OO-Style

```python
x = np.linspace(0, 2, 100)

# Note that even in the OO-style, we use `.pyplot.figure` to create the figure.
fig, ax = plt.subplots()  # Create a figure and an axes.
ax.plot(x, x, label='linear')  # Plot some data on the axes.
ax.plot(x, np.sin(x*np.pi), label='sine wave')  # Plot more data on the axes...
ax.plot(x, np.log(x**3), label='logarithmic')  # ... and some more.
ax.set_xlabel('x label')  # Add an x-label to the axes.
ax.set_ylabel('y label')  # Add a y-label to the axes.
ax.set_title("Simple Plot")  # Add a title to the axes.
ax.legend()  # Add a legend.
```

* plt.subplot(nrows, ncols, index, **kwargs)
* plt.plot(x,y, label="label_name") where y = f(x). Essentially, we are providing the x-values and the y-values for the plot.

## Pyplot style

```python
x = np.linspace(0, 2, 100)

# Note that even in the OO-style, we use `.pyplot.figure` to create the figure.
fig, ax = plt.subplots()  # Create a figure and an axes.
ax.plot(x, x, label='linear')  # Plot some data on the axes.
ax.plot(x, x**2, label='quadratic')  # Plot more data on the axes...
ax.plot(x, x**3, label='cubic')  # ... and some more.
ax.set_xlabel('x label')  # Add an x-label to the axes.
ax.set_ylabel('y label')  # Add a y-label to the axes.
ax.set_title("Simple Plot")  # Add a title to the axes.
ax.legend()  # Add a legend.
```

# Sample plots in Matplotlib

## 1. Line Plot

Line plots are perhaps the most common plots that we might be accustomed to seeing in mathematics. The Line plot, or as I like to think of it - **the continuous plot**, is used here to draw a consine wave.

```python
#Let's draw a line plot of a Cosine curve
# Data for plotting
t = np.arange(0.0, 2.0, 0.01)
s = 1 + np.cos(2 * np.pi * t) #1 + 2 * cos(2*pi*t)

fig, ax = plt.subplots()
ax.plot(t, s)

ax.set(xlabel='Time (s)', ylabel='Voltage (mV)', title='Cosine Plot')
ax.grid()

fig.savefig("cosine_line_plot.png")
plt.show()
```

## 2. Multiple subplots

Sometimes, it is useful to show multiple plots. We are going to draw two plots (or axes) within a figure.

```python
x1 = np.linspace(0.0, 5.0)
x2 = np.linspace(0.0, 2.0)

y1 = np.cos(2 * np.pi * x1) * np.exp(-x1)
y2 = np.cos(2 * np.pi * x2)

fig, (ax1, ax2) = plt.subplots(2, 1)
fig.suptitle('A tale of 2 subplots')

#The o- and .- are the format string. Refer to https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.plot.html?highlight=plot#matplotlib.pyplot.plot
# for the types of format strings. You can set color, line style and markers.
ax1.plot(x1, y1, 'o-')
ax1.set_ylabel('Damped oscillation')

ax2.plot(x2, y2, '.-')
ax2.set_xlabel('time (s)')
ax2.set_ylabel('Undamped')

plt.show()
```

Alternative Method:

```python
x1 = np.linspace(0.0, 5.0)
x2 = np.linspace(0.0, 2.0)

y1 = np.cos(2 * np.pi * x1) * np.exp(-x1)
y2 = np.cos(2 * np.pi * x2)

plt.subplot(2, 1, 1)
plt.plot(x1, y1, 'o-')
plt.title('A tale of 2 subplots')
plt.ylabel('Damped oscillation')

plt.subplot(2, 1, 2)
plt.plot(x2, y2, '.-')
plt.xlabel('time (s)')
plt.ylabel('Undamped')

plt.show()
```

**When should you use a single axes plot with multiple curves or multiple axes with single curves?**

This depends on what you want to represent and there is no absolute answer. When trying to compare different curves, a single plot on the same axes may be preferred. However, too many plots can make the axes too cluttered. Simpler plots are usually easier to understand. Consider using multiple plots where clarity can be improved. Multiple plots are particularly useful when you are trying to visualize some data or related information but from different angles or contexts.

## 3. Histogram

A dataset has a distribution. A dataset can have a mean $\mu$ and standard deviation $\sigma$.

\begin{equation}
y = \mu + \sigma * x
\end{equation}

This is the equation for a **normal distribution**.

We will also draw a best fit line.

\begin{equation}
z = \frac{1}{\sqrt{2*\pi}*\sigma} * e^{-0.5 * {\frac{1}{\sigma} * (bins - \mu)}^2}
\end{equation}

> *ax.hist(x, bins,range,density, weights)*

> **x:** array

> **bins:** If bins is an integer, it defines the number of equal-width bins in the range. If bins is a sequence, it defines the bin edges, including the left edge of the first bin and the right edge of the last bin.

> **density:** if True, draw and return a probability density.

> **Returns**

> **n:** array of the values of the histogram bins

> **bins:** the edges of the bins

> **patches:** contains the information to create the histogram.

```python
np.random.seed(19680801)

# example data
mu = 100  # mean of distribution
sigma = 15  # standard deviation of distribution
x = mu + sigma * np.random.randn(437)

num_bins = 50

fig, ax = plt.subplots()

# the histogram of the data

n, bins, patches = ax.hist(x, num_bins, density=True)

# add a 'best fit' line
z = ((1 / (np.sqrt(2 * np.pi) * sigma)) *
     np.exp(-0.5 * (1 / sigma * (bins - mu))**2))
ax.plot(bins, z, '--')
ax.set_xlabel('Smarts')
ax.set_ylabel('Probability density')
ax.set_title(r'Histogram of IQ: $\mu=100$, $\sigma=15$')

# Tweak spacing to prevent clipping of ylabel
fig.tight_layout()
plt.show()
```

**How is a histogram different from a bar chart?**

Histograms are used to show distributions of variables while bar charts are used to compare variables.

## 4. Bar Chart

The bar chart is useful when comparing variables. In this case, we look at fuel sources for electricity and how many Giga Joules (GJ) we are producing from each fuel source.

```python
x = ['Nuclear', 'Hydro', 'Gas', 'Oil', 'Coal', 'Biofuel', 'Solar', 'GeoThermal']
energy = [5, 6, 15, 22, 24, 8, 5, 0.5]

x_pos = [i for i, _ in enumerate(x)]

plt.bar(x_pos, energy, color='Red')
plt.xlabel("Energy Source")
plt.ylabel("Energy Output (GJ)")
plt.title("Energy output from various fuel sources")

plt.xticks(x_pos, x)

plt.show()
```

## 5. Error Bars

Consider the same bar chart as above, but with a variance of +/- 10%.

```python
x = ['Nuclear', 'Hydro', 'Gas', 'Oil', 'Coal', 'Biofuel', 'Solar', 'GeoThermal']
energy = [5, 6, 15, 22, 24, 8, 5, 0.5]
variance = [i * 0.1 for i in energy]  #a variance of 10% of energy
#Note: There is no need for variance to be strictly 10% for all items in
# energy. Here i Used a standardized value.

x_pos = [i for i, _ in enumerate(x)]

plt.bar(x_pos, energy, color='pink', yerr=variance)
plt.xlabel("Energy Source")
plt.ylabel("Energy Output (GJ)")
plt.title("Energy output from various fuel sources with Variance")

plt.xticks(x_pos, x)

plt.show()
```

## 6. Horizontal Bar Chart

**When should you use a vertical bar chart versus a horizontal bar chart?**

There are different benefits to the vertical bar chart versus horizontal bar chart.

* If one of the axis contains text, while the other is numeric, consider a horizontal bar chart with the text-based axis as the y-axis.

* If the data has (or can be organized as) a normal distribution or an inverted bell curve, then consider a vertical bar chart. It is more easier on the eyes to view it this way.

* If the data you wish to represents ought increaseor decreases in sequence, a horizontal bar chart is preferred.

```python
x = ['Nuclear', 'Hydro', 'Gas', 'Oil', 'Coal', 'Biofuel', 'Solar', 'GeoThermal']
energy = [5, 6, 15, 22, 24, 8, 5, 0.5]
variance = [1, 2, 7, 4, 2, 3, 4, 0.2]

x_pos = [i for i, _ in enumerate(x)]

plt.barh(x_pos, energy, color='orange', xerr=variance)   #OBSERVE xerr and barh
plt.ylabel("Energy Source")
plt.xlabel("Energy Output (GJ)")
plt.title("Energy output from various fuel sources with Variance")

plt.yticks(x_pos, x)  #OBSERVE yticks

plt.show()
```

```python
from matplotlib.ticker import MaxNLocator #Finds up to a max number of intervals with ticks at nice locations.
from collections import namedtuple  #factory function for creating tuple subclasses with named fields

np.random.seed(42)

Student = namedtuple('Student', ['name', 'grade', 'gender'])
Score = namedtuple('Score', ['score', 'percentile'])

# GLOBAL CONSTANTS
test_names = ['Pacer Test', 'Flexed Arm\n Hang', 'Mile Run', 'Agility',
              'Push Ups']
test_units = dict(zip(test_names, ['laps', 'sec', 'min:sec', 'sec', '']))

def attach_ordinal(num):
    """Convert an integer to an ordinal string, e.g. 2 -> '2nd'."""
    suffixes = {str(i): v
                for i, v in enumerate(['th', 'st', 'nd', 'rd', 'th',
                                       'th', 'th', 'th', 'th', 'th'])}
    v = str(num)
    # special case early teens
    if v in {'11', '12', '13'}:
        return v + 'th'
    return v + suffixes[v[-1]]

def format_score(score, test):
    """
    Create score labels for the right y-axis as the test name followed by the
    measurement unit (if any), split over two lines.
    """
    unit = test_units[test]
    if unit:
        return f'{score}\n{unit}'
    else:  # If no unit, don't include a newline, so that label stays centered.
        return score

def format_ycursor(y):
    y = int(y)
    if y < 0 or y >= len(test_names):
        return ''
    else:
        return test_names[y]

def plot_student_results(student, scores, cohort_size):
    fig, ax1 = plt.subplots(figsize=(9, 7))  # Create the figure
    fig.subplots_adjust(left=0.115, right=0.88)
    fig.canvas.manager.set_window_title('Eldorado K-8 Fitness Chart')

    pos = np.arange(len(test_names))

    rects = ax1.barh(pos, [scores[k].percentile for k in test_names],
                     align='center',
                     height=0.5,
                     tick_label=test_names)

    ax1.set_title(student.name)

    ax1.set_xlim([0, 100])
    ax1.xaxis.set_major_locator(MaxNLocator(11))
    ax1.xaxis.grid(True, linestyle='--', which='major',
                   color='grey', alpha=.25)

    # Plot a solid vertical gridline to highlight the median position
    ax1.axvline(50, color='grey', alpha=0.25)

    # Set the right-hand Y-axis ticks and labels
    ax2 = ax1.twinx()

    # Set the tick locations
    ax2.set_yticks(pos)
    # Set equal limits on both yaxis so that the ticks line up
    ax2.set_ylim(ax1.get_ylim())

    # Set the tick labels
    ax2.set_yticklabels([format_score(scores[k].score, k) for k in test_names])

    ax2.set_ylabel('Test Scores')

    xlabel = ('Percentile Ranking Across {grade} Grade {gender}s\n'
              'Cohort Size: {cohort_size}')
    ax1.set_xlabel(xlabel.format(grade=attach_ordinal(student.grade),
                                 gender=student.gender.title(),
                                 cohort_size=cohort_size))

    rect_labels = []
    # Lastly, write in the ranking inside each bar to aid in interpretation
    for rect in rects:
        # Rectangle widths are already integer-valued but are floating
        # type, so it helps to remove the trailing decimal point and 0 by
        # converting width to int type
        width = int(rect.get_width())

        rank_str = attach_ordinal(width)
        # The bars aren't wide enough to print the ranking inside
        if width < 40:
            # Shift the text to the right side of the right edge
            xloc = 5
            # Black against white background
            clr = 'black'
            align = 'left'
        else:
            # Shift the text to the left side of the right edge
            xloc = -5
            # White on magenta
            clr = 'white'
            align = 'right'

        # Center the text vertically in the bar
        yloc = rect.get_y() + rect.get_height() / 2
        label = ax1.annotate(
            rank_str, xy=(width, yloc), xytext=(xloc, 0),
            textcoords="offset points",
            horizontalalignment=align, verticalalignment='center',
            color=clr, weight='bold', clip_on=True)
        rect_labels.append(label)

    # Make the interactive mouse over give the bar title
    ax2.fmt_ydata = format_ycursor
    # Return all of the artists created
    return {'fig': fig,
            'ax': ax1,
            'ax_right': ax2,
            'bars': rects,
            'perc_labels': rect_labels}

student = Student('Johnny Doe', 2, 'boy')
scores = dict(zip(
    test_names,
    (Score(v, p) for v, p in
     zip(['7', '48', '12:52', '17', '14'],
         np.round(np.random.uniform(0, 100, len(test_names)), 0)))))
cohort_size = 62  # The number of other 2nd grade boys

arts = plot_student_results(student, scores, cohort_size)
plt.show()
```

## 7. Bar Chart with Multiple X’s

Bar chart with multiple X's is usual when comparing the strength of a metric for a few categories. In this example, the categories are (men,women) and the metrics are (strength, intelligence, fine-mobility, flexibility, Charisma)

```python
N = 5
men_means = (20, 35, 30, 35, 27)
women_means = (25, 32, 34, 20, 25)

ind = np.arange(N)
width = 0.35
plt.bar(ind, men_means, width, label='Men')
plt.bar(ind + width, women_means, width,
    label='Women')

plt.ylabel('Scores')
plt.title('Scores by group and gender')

plt.xticks(ind + width / 2, ('Strength', 'Intelligence', 'Fine-Mobility', 'Flexibility', 'Charisma'))
plt.legend(loc='best')
plt.show()
```

## 8. Stacked Bar Charts

Stacked Bar charts are useful when you want to show case for a given class, the individual components that make up the class as well as the total. In the example below, the class is USA, and the individual components are Gold, Silver and Bronze medals.

```python
countries = ['USA', 'GB', 'China', 'Russia', 'Germany']
bronzes = np.array([38, 17, 26, 19, 15])
silvers = np.array([37, 23, 18, 18, 10])
golds = np.array([46, 27, 26, 19, 17])
ind = [x for x, _ in enumerate(countries)]

plt.bar(ind, golds, width=0.8, label='golds', color='gold', bottom=silvers+bronzes)
plt.bar(ind, silvers, width=0.8, label='silvers', color='silver', bottom=bronzes)
plt.bar(ind, bronzes, width=0.8, label='bronzes', color='#CD853F')

plt.xticks(ind, countries)
plt.ylabel("Medals")
plt.xlabel("Countries")
plt.legend(loc="upper right")
plt.title("2012 Olympics Top Scorers")

plt.show()
```

If we wanted to view the same bar charts but as a proportion of the total medals won by that country, we can do the following:

```python
total = bronzes + silvers + golds
proportion_bronzes = np.true_divide(bronzes, total) * 100
proportion_silvers = np.true_divide(silvers, total) * 100
proportion_golds = np.true_divide(golds, total) * 100

plt.bar(ind, proportion_golds, width=0.8, label='golds', color='gold', bottom=proportion_bronzes+proportion_silvers)
plt.bar(ind, proportion_silvers, width=0.8, label='silvers', color='silver', bottom=proportion_bronzes)
plt.bar(ind, proportion_bronzes, width=0.8, label='bronzes', color='#CD853F')

plt.xticks(ind, countries)
plt.ylabel("Medals")
plt.xlabel("Countries")
plt.title("2012 Olympics Top Scorers' Medals by Proportion")
plt.ylim=1.0

# rotate axis labels
plt.setp(plt.gca().get_xticklabels(), rotation=45, horizontalalignment='right')

plt.show()
```

## 9. Pie Chart

A pie chart is fairly simple and self explanatory. All part of a pie chart should add up to a whole. In percentage terms, the sum of all parts total 100%.

```python
# Pie chart, where the slices will be ordered and plotted counter-clockwise:
labels = 'Nuclear', 'Hydro', 'Gas', 'Oil', 'Coal', 'Biofuel', 'Solar', 'GeoThermal'
#labels = 'Frogs', 'Hogs', 'Dogs', 'Logs'
sizes = [15, 30, 45, 10, 20, 10, 50, 5]
explode = (0, 0, 0, 0, 0, 0, 0.2, 0)  # only "explode" the 2nd slice (i.e. 'Solar')

fig1, ax1 = plt.subplots()
ax1.pie(sizes, explode=explode, labels=labels, autopct='%1.1f%%',
        shadow=True, startangle=90)
ax1.axis('equal')  # Equal aspect ratio ensures that pie is drawn as a circle.
plt.title("Electric Power Fuel Pie Chart")

plt.show()
```

## 10. Table Chart

```python
data = [[ 66386, 174296,  75131, 577908,  32015],
        [ 58230, 381139,  78045,  99308, 160454],
        [ 89135,  80552, 152558, 497981, 603535],
        [ 78415,  81858, 150656, 193263,  69638],
        [139361, 331509, 343164, 781380,  52269]]

columns = ('Freeze', 'Wind', 'Flood', 'Quake', 'Hail')
rows = ['%d year' % x for x in (100, 50, 20, 10, 5)]

values = np.arange(0, 2500, 500)
value_increment = 1000

# Get some pastel shades for the colors
colors = plt.cm.BuPu(np.linspace(0, 0.5, len(rows)))
n_rows = len(data)

index = np.arange(len(columns)) + 0.3
bar_width = 0.4

# Initialize the vertical-offset for the stacked bar chart.
y_offset = np.zeros(len(columns))

# Plot bars and create text labels for the table
cell_text = []
for row in range(n_rows):
    plt.bar(index, data[row], bar_width, bottom=y_offset, color=colors[row])
    y_offset = y_offset + data[row]
    cell_text.append(['%1.1f' % (x / 1000.0) for x in y_offset])
# Reverse colors and text labels to display the last value at the top.
colors = colors[::-1]
cell_text.reverse()

# Add a table at the bottom of the axes
the_table = plt.table(cellText=cell_text,
                      rowLabels=rows,
                      rowColours=colors,
                      colLabels=columns,
                      loc='bottom')

# Adjust layout to make room for the table:
plt.subplots_adjust(left=0.2, bottom=0.2)

plt.ylabel("Loss in ${0}'s".format(value_increment))
plt.yticks(values * value_increment, ['%d' % val for val in values])
plt.xticks([])
plt.title('Loss by Disaster')

plt.show()
```

##11. Scatter Plot

Scatter plots are useful when compared with a line plot if you want to draw distributed data points. This is in contrast to a continuous plot or line plot shown earlier.

```python
x = np.linspace(0, 10, 30)
y = np.sin(x)

plt.plot(x, y, 'o', color='black');
plt.title("Scatter Plot of Sine Curve")
```

## 12. Log Plots

Log scales can help visualize between large descrepancies of values on a single axis. For example, say you want to compare the relative work done by animal/inspect species. The y axis may be the work done dictaed in joules but the x-axis is their weight. A log plot along the x-axis might be quite useful here.

```python
# Data for plotting
t = np.arange(0.01, 20.0, 0.01)

# Create figure and allow for a 2x2=4 diagrams
fig, ((ax1, ax2), (ax3, ax4)) = plt.subplots(2, 2)

#The y axis is logarithmic. x-axis is normal
ax1.semilogy(t, np.exp(-t / 5.0))   #first argument is x axis, second is yaxis.
ax1.set(title='semilogy')
ax1.grid()

#The x axis is logarithmic. y-axis is normal
ax2.semilogx(t, np.sin(2 * np.pi * t))
ax2.set(title='semilogx')
ax2.grid()

#Both the x and y axis are logarithmic.
ax3.loglog(t, 20 * np.exp(-t / 10.0))
#ax3.set_xscale('log', base=2)
ax3.set_xscale('log')
ax3.set(title='loglog base 2 on x')
ax3.grid()

# With errorbars: clip non-positive values
# Use new data for plotting
x = 10.0**np.linspace(0.0, 2.0, 20)
y = x**2.0

#ax4.set_xscale("log", nonpositive='clip')
ax4.set_xscale("log")
#ax4.set_yscale("log", nonpositive='clip')
ax4.set_yscale("log")
ax4.set(title='Errorbars go negative')
ax4.errorbar(x, y, xerr=0.1 * x, yerr=5.0 + 0.75 * y)
# ylim must be set after errorbar to allow errorbar to autoscale limits
ax4.set_ylim(bottom=0.1)

fig.tight_layout()
plt.show()
```

## 13. Polar Plot

A polar chart is useful when you want to compare the magnitude of a quantity with respect to an angle.

Polar charts are often very useful for present data that uses a natural cycle such as daily temperatures or annual rainfall. They're also frequently used in scientific and mathematic applications. When you purchase a microphone, you will find polar charts for them.

```python
r = np.arange(0, 2, 0.01)
theta = 2 * np.pi * r

fig, ax = plt.subplots(subplot_kw={'projection': 'polar'})
ax.plot(theta, r)
ax.set_rmax(2)
ax.set_rticks([0.5, 1, 1.5, 2])  # Less radial ticks
ax.set_rlabel_position(-22.5)  # Move radial labels away from plotted line
ax.grid(True)

ax.set_title("A line plot on a polar axis", va='bottom')
plt.show()
```

## 14. ColorMesh

The colormesh (such as the one we show below) is useful when you want to represent 3 things and one of those 3 things is not as useful to represent as a number.

```python
np.random.seed(42)
Z = np.random.rand(6, 10)
x = np.arange(-0.5, 10, 1)  # len = 11
y = np.arange(4.5, 11, 1)  # len = 7

fig, ax = plt.subplots()
ax.pcolormesh(x, y, Z)
```

```python
age = [25, 30, 35, 40, 45, 50, 55, 60, 65]  #x-axis has 9 items
race = ["African", "Chinese", "Indian", "European", "American"] #y-axis has 5 items
weight = np.random.rand(5,9)  #9x5 = 45
#print(weight, weight.shape)

fig, ax = plt.subplots()
ax.pcolormesh(age, race, weight)
```

## 15. DateTime Chart

```python
#import matplotlib.dates as mdates
#import matplotlib.cbook as cbook

# Load a numpy structured array from yahoo csv data with fields date, open,
# close, volume, adj_close from the mpl-data/example directory.  This array
# stores the date as an np.datetime64 with a day unit ('D') in the 'date'
# column.
#data = cbook.get_sample_data('goog.npz', np_load=True)['price_data']

fig, ax = plt.subplots()
ax.plot('date', 'adj_close', data=data)

# Major ticks every 6 months.
fmt_half_year = mdates.MonthLocator(interval=6)
ax.xaxis.set_major_locator(fmt_half_year)

# Minor ticks every month.
fmt_month = mdates.MonthLocator()
ax.xaxis.set_minor_locator(fmt_month)

# Text in the x axis will be displayed in 'YYYY-mm' format.
ax.xaxis.set_major_formatter(mdates.DateFormatter('%Y-%m'))

# Round to nearest years.
datemin = np.datetime64(data['date'][0], 'Y')
datemax = np.datetime64(data['date'][-1], 'Y') + np.timedelta64(1, 'Y')
ax.set_xlim(datemin, datemax)

# Format the coords message box, i.e. the numbers displayed as the cursor moves
# across the axes within the interactive GUI.
ax.format_xdata = mdates.DateFormatter('%Y-%m')
ax.format_ydata = lambda x: f'${x:.2f}'  # Format the price.
ax.grid(True)

# Rotates and right aligns the x labels, and moves the bottom of the
# axes up to make room for them.
fig.autofmt_xdate()

plt.show()
```

## 16. 3D plots

```python
ax = plt.axes(projection='3d')

# Data for a three-dimensional line
zline = np.linspace(0, 15, 1000)
xline = np.sin(zline)
yline = np.cos(zline)
ax.plot3D(xline, yline, zline, 'gray')

# Data for three-dimensional scattered points
zdata = 15 * np.random.random(100)
xdata = np.sin(zdata) + 0.1 * np.random.randn(100)
ydata = np.cos(zdata) + 0.1 * np.random.randn(100)
ax.scatter3D(xdata, ydata, zdata, c=zdata, cmap='Greens');
```

```python
x = np.linspace(-6, 6, 30)
y = np.linspace(-6, 6, 30)

X, Y = np.meshgrid(x, y)
Z = np.sin(np.sqrt(X ** 2 + Y ** 2))

fig = plt.figure()
ax = plt.axes(projection='3d')
ax.contour3D(X, Y, Z, 50, cmap='binary')
ax.set_xlabel('x')
ax.set_ylabel('y')
ax.set_zlabel('z');
```

```python
ax.view_init(60, 35)
fig
```

```python
fig = plt.figure()
ax = plt.axes(projection='3d')
ax.plot_wireframe(X, Y, Z, color='black')
ax.set_title('wireframe');
```

```python
ax = plt.axes(projection='3d')
ax.plot_surface(X, Y, Z, rstride=1, cstride=1,
                cmap='viridis', edgecolor='none')
ax.set_title('surface');
```

# THE END

Data visualization is an art. And there are many, many charts/plots/illustrations/visualization. I have covered many of the common plots, but there are many more. I may add some more over time, but if you need more, refer to the matplotlib documentation: https://matplotlib.org/stable/tutorials/index.html