# pAPRika tutorial 7 - APR/NAMD with Colvars restraints

In this tutorial, we will perform APR calculations for the butane (BUT)--cucurbit[6]uril (CB6) host-guest system with the [Generalized-Born](https://en.wikipedia.org/wiki/Implicit_solvation#Generalized_Born) implicit solvent. This tutorial will use `NAMD` for the simulation and `Colvars` for the restraints. Although it is possible to run `NAMD` with `Plumed` we will not use `Plumed` in this tutorial. Users should refer to the previous tutorial to see how to use `Plumed` restraints.

🔵 Since we have a prepared the host-guest-dummy setup from the first tutorial, we will skip the initial tleap steps and go right into initializing the restraints. If you haven't completed [tutorial 1](01-tutorial-cb6-but.ipynb) please go back and run the notebook to generate the starting structure.

## Initialize

```python
import os
```

### Define names
We will store the files created in this tutorial in a folder called `namd` so we don't mix files with the previous tutorial.

```python
base_name = "cb6-but-dum"
work_dir = "namd"
complex_dir = "complex"
data = "../../../paprika/data/cb6-but"
```

## Configure APR Restraints

### Define anchor atoms
See [tutorial 1](01-tutorial-cb6-but.ipynb) for the choice of selection

```python
# Guest atoms
G1 = ":BUT@C"
G2 = ":BUT@C3"

# Host atoms
H1 = ":CB6@C"
H2 = ":CB6@C31"
H3 = ":CB6@C18"

# Dummy atoms
D1 = ":DM1"
D2 = ":DM2"
D3 = ":DM3"
```

### Determine the number of windows
Before we add the restraints, it is helpful to set the $\lambda$ fractions that control the strength of the force constants during attach and release, and to define the distances for the pulling phase.

The attach fractions go from 0 to 1 and we place more points at the bottom of the range to sample the curvature of $dU/d \lambda$. Next, we generally apply a distance restraint until the guest is ~18 Angstroms away from the host, in increments of 0.4 Angstroms. This distance should be at least twice the Lennard-Jones cutoff in the system. These values have worked well for us, but this is one aspect that should be carefully checked for new systems.

```python
import numpy as np
```

```python
attach_string = "0.00 0.40 0.80 1.60 2.40 4.00 5.50 8.65 11.80 18.10 24.40 37.00 49.60 74.80 100.00"
attach_fractions = [float(i) / 100 for i in attach_string.split()]

initial_distance = 6.0
pull_distances = np.arange(0.0 + initial_distance, 18.0 + initial_distance, 1.0)

release_fractions = []

windows = [len(attach_fractions), len(pull_distances), len(release_fractions)]
print(f"There are {windows} windows in this attach-pull-release calculation.")
```

    There are [15, 18, 0] windows in this attach-pull-release calculation.

### Load structure

```python
import parmed as pmd
```

* Load complex structure

```python
structure = pmd.load_file(
    os.path.join(complex_dir, f"{base_name}.prmtop"),
    os.path.join(complex_dir, f"{base_name}.rst7"),
    structure = True,
)
```

### Host Static Restraints
See [tutorial 1](01-tutorial-cb6-but.ipynb#host_static) for an explanation of the static restraints.

```python
import paprika.restraints as restraints
```

```python
static_restraints = []
```

```python
r = restraints.static_DAT_restraint(restraint_mask_list = [D1, H1],
                                    num_window_list = windows,
                                    ref_structure = structure,
                                    force_constant = 5.0,
                                    amber_index=True)

static_restraints.append(r)
```

```python
r = restraints.static_DAT_restraint(restraint_mask_list = [D2, D1, H1],
                                    num_window_list = windows,
                                    ref_structure = structure,
                                    force_constant = 100.0,
                                    amber_index=True)

static_restraints.append(r)
```

```python
r = restraints.static_DAT_restraint(restraint_mask_list = [D3, D2, D1, H1],
                                    num_window_list = windows,
                                    ref_structure = structure,
                                    force_constant = 100.0,
                                    amber_index=True)

static_restraints.append(r)
```

```python
r = restraints.static_DAT_restraint(restraint_mask_list = [D1, H1, H2],
                                    num_window_list = windows,
                                    ref_structure = structure,
                                    force_constant = 100.0,
                                    amber_index=True)

static_restraints.append(r)
```

```python
r = restraints.static_DAT_restraint(restraint_mask_list = [D2, D1, H1, H2],
                                    num_window_list = windows,
                                    ref_structure = structure,
                                    force_constant = 100.0,
                                    amber_index=True)

static_restraints.append(r)
```

```python
r = restraints.static_DAT_restraint(restraint_mask_list = [D1, H1, H2, H3],
                                    num_window_list = windows,
                                    ref_structure = structure,
                                    force_constant = 100.0,
                                    amber_index=True)

static_restraints.append(r)
```

### Guest translational and rotational restraints
See [tutorial 1](01-tutorial-cb6-but.ipynb#guest) for an explanation of the guest restraints

```python
guest_restraints = []
```

```python
r = restraints.DAT_restraint()
r.mask1 = D1
r.mask2 = G1
r.topology = structure
r.auto_apr = True
r.continuous_apr = True
r.amber_index = True

r.attach["target"] = pull_distances[0]          # Angstroms
r.attach["fraction_list"] = attach_fractions
r.attach["fc_final"] = 5.0                      # kcal/mol/Angstroms**2

r.pull["target_final"] = 24.0                   # Angstroms
r.pull["num_windows"] = windows[1]

r.initialize()
guest_restraints.append(r)
```

```python
r = restraints.DAT_restraint()
r.mask1 = D2
r.mask2 = D1
r.mask3 = G1
r.topology = structure
r.auto_apr = True
r.continuous_apr = True
r.amber_index = True

r.attach["target"] = 180.0                      # Degrees
r.attach["fraction_list"] = attach_fractions
r.attach["fc_final"] = 100.0                    # kcal/mol/radian**2

r.pull["target_final"] = 180.0                  # Degrees
r.pull["num_windows"] = windows[1]

r.initialize()
guest_restraints.append(r)
```

```python
r = restraints.DAT_restraint()
r.mask1 = D1
r.mask2 = G1
r.mask3 = G2
r.topology = structure
r.auto_apr = True
r.continuous_apr = True
r.amber_index = True

r.attach["target"] = 180.0                      # Degrees
r.attach["fraction_list"] = attach_fractions
r.attach["fc_final"] = 100.0                    # kcal/mol/radian**2

r.pull["target_final"] = 180.0                  # Degrees
r.pull["num_windows"] = windows[1]

r.initialize()
guest_restraints.append(r)
```

### Create APR windows
We use the guest restraints to create a list of windows with the appropriate names and then create the directories.

```python
from paprika.restraints.restraints import create_window_list
```

```python
window_list = create_window_list(guest_restraints)
```

```python
if not os.path.isdir(work_dir):
    os.makedirs(work_dir)

for window in window_list:
    folder = os.path.join(work_dir, window)
    if not os.path.isdir(folder):
        os.makedirs(os.path.join(work_dir, window))
```

### Write APR restraints to a Colvars module file
In this section we create an instance of `Colvars()` from `paprika.restraints.colvars`, which is a class to generate the `Colvars` restraint files. This class is analogous to the `Plumed` class. We need to specify the list of restraints used throughout the APR calculations and the corresponding windows list. In this tutorial we will print the **host static** restraints and the **guest** restraints. The `Colvars` class includes a method to add restraints to dummy atoms (`add_dummy_atoms_to_file`). We will add the positional restraints on the dummy atoms after generating the structures for each window.

**Note**: be careful when specifying the force constants in `DAT_restraints`. We follow the AMBER (and CHARMM) convention where the force constant is already multiplied by a factor of 1/2 but `Colvars` (and `Plumed`) requires the user to specify the force constant without this factor, i.e.

$$
\begin{align}
U_{amber} &= K_{amber} (r-r_{0})^2 \\
U_{colvars} &= \frac{1}{2} k_{colvars} (r - r_{0})^2
\end{align}
$$

thus $k_{colvars} = 2 \times  K_{amber}$. If AMBER force constants was used in generating the `DAT_restraints` (the case in this tutorial) we need to set the variable `uses_legacy_k` to `True` (this is on by default).

```python
from paprika.restraints.colvars import Colvars
```

```python
restraints_list = (static_restraints + guest_restraints)

colvars = Colvars()
colvars.file_name = "colvars.tcl"
colvars.path = work_dir
colvars.window_list = window_list
colvars.restraint_list = restraints_list
colvars_uses_legacy_k = True

colvars.dump_to_file()
```

## Prepare host-guest system

### Translate guest molecule
For the attach windows, we will use the initial, bound coordinates for the host-guest complex. Only the force constants change during this phase, so a single set of coordinates is sufficient. For the pull windows, we will translate the guest to the target value, and for the release windows, we will use the coordinates from the final pull window.

```python
import shutil
```

```python
for window in window_list:
    if window[0] == "a":
        shutil.copy(os.path.join(complex_dir, f"{base_name}.prmtop"),
                    os.path.join(work_dir, window, f"{base_name}.prmtop"))
        shutil.copy(os.path.join(complex_dir, f"{base_name}.rst7"),
                    os.path.join(work_dir, window, f"{base_name}.rst7"))

    elif window[0] == "p":
        structure = pmd.load_file(
            os.path.join(complex_dir, f"{base_name}.prmtop"),
            os.path.join(complex_dir, f"{base_name}.rst7"),
            structure = True
        )
        target_difference = guest_restraints[0].phase['pull']['targets'][int(window[1:])] -\
                            guest_restraints[0].pull['target_initial']
        print(f"In window {window} we will translate the guest {target_difference.magnitude:0.1f}.")

        for atom in structure.atoms:
            if atom.residue.name == "BUT":
                atom.xz += target_difference.magnitude

        structure.save(os.path.join(work_dir, window, f"{base_name}.prmtop"), overwrite=True)
        structure.save(os.path.join(work_dir, window, f"{base_name}.rst7"), overwrite=True)
```

    In window p000 we will translate the guest 0.0 Angstroms.
    In window p001 we will translate the guest 1.1 Angstroms.
    In window p002 we will translate the guest 2.1 Angstroms.
    In window p003 we will translate the guest 3.2 Angstroms.
    In window p004 we will translate the guest 4.2 Angstroms.
    In window p005 we will translate the guest 5.3 Angstroms.
    In window p006 we will translate the guest 6.4 Angstroms.
    In window p007 we will translate the guest 7.4 Angstroms.
    In window p008 we will translate the guest 8.5 Angstroms.
    In window p009 we will translate the guest 9.5 Angstroms.
    In window p010 we will translate the guest 10.6 Angstroms.
    In window p011 we will translate the guest 11.6 Angstroms.
    In window p012 we will translate the guest 12.7 Angstroms.
    In window p013 we will translate the guest 13.8 Angstroms.
    In window p014 we will translate the guest 14.8 Angstroms.
    In window p015 we will translate the guest 15.9 Angstroms.
    In window p016 we will translate the guest 16.9 Angstroms.
    In window p017 we will translate the guest 18.0 Angstroms.

### Add dummy atom restraints
Here we will add positional restraints on the dummy atoms. The reference positions are extracted from the structure files we just created in each window.

```python
for window in window_list:
    folder = os.path.join(work_dir, window)

    structure = pmd.load_file(
        os.path.join(folder, f"{base_name}.prmtop"),
        os.path.join(folder, f"{base_name}.rst7"),
        structure = True
    )

    colvars.add_dummy_atom_restraints(structure, window)
```

## Simulation

Now that we have prepared the structure and `colvars.tcl` file for each window, we have everything ready to go. **pAPRika** comes with a python wrapper for `NAMD` that can help set up default parameters for the simulation. There are some high level options that we set directly, like `simulation.path`, and then we call the function `config_gb_min()` to setup reasonable default simulation parameters for a minimization and production run. The temperature of the system will be controlled with the `Langevin` thermostat, which is the default. For running simulations with a different thermostat parse the thermostat of choice using the `Thermostat` Enum class to `config_gb_md()`. For example, if we want to run our simulations with the *Lowe-Anderson* thermostat:

```python
simulation = NAMD()
...
simulation.config_gb_md(NAMD.Thermostat.LoweAnderson)
simulation.run()
```

See the [pAPRika](<https://www.ks.uiuc.edu/Research/namd/2.14/ug/node38.html>) documentation for more details on the different temperature control algorithm implemented.

**Note**: `NAMD` version 2 is similar to `GROMACS` in that more than one thread for every GPU is preferred to deliver better speed. We set the number of processors with `simulation.n_threads = 8` and the GPU device with `simulation.gpu_devices = 0`. Starting with `NAMD` version 3 (currently in alpha phase) a smaller number of `n_threads` is required per GPU.

```python
from paprika.simulate import NAMD
```

Initialize logger

```python
import logging
from importlib import reload
reload(logging)

logger = logging.getLogger()
logging.basicConfig(
    format='%(asctime)s %(message)s',
    datefmt='%Y-%m-%d %I:%M:%S %p',
    level=logging.INFO
)
```

### Energy Minimization

```python
for window in window_list:
    simulation = NAMD()
    simulation.executable = "namd2"
    simulation.n_threads = 4
#     simulation.gpu_devices = 0
    simulation.path = os.path.join(work_dir, window)

    simulation.topology = f"{base_name}.prmtop"
    simulation.coordinates = f"{base_name}.rst7"
    simulation.prefix = "minimize"
    simulation.colvars_file = "colvars.tcl"

    simulation.config_gb_min()

    logging.info(f"Running minimization in window {window}...")
    simulation.run(overwrite=True)
```

    2020-10-09 11:31:02 AM Running minimization in window a000...
    2020-10-09 11:31:07 AM Running minimization in window a001...
    2020-10-09 11:31:13 AM Running minimization in window a002...
    2020-10-09 11:31:20 AM Running minimization in window a003...
    2020-10-09 11:31:25 AM Running minimization in window a004...
    2020-10-09 11:31:30 AM Running minimization in window a005...
    2020-10-09 11:31:34 AM Running minimization in window a006...
    2020-10-09 11:31:39 AM Running minimization in window a007...
    2020-10-09 11:31:43 AM Running minimization in window a008...
    2020-10-09 11:31:48 AM Running minimization in window a009...
    2020-10-09 11:31:53 AM Running minimization in window a010...
    2020-10-09 11:31:59 AM Running minimization in window a011...
    2020-10-09 11:32:03 AM Running minimization in window a012...
    2020-10-09 11:32:08 AM Running minimization in window a013...
    2020-10-09 11:32:12 AM Running minimization in window p000...
    2020-10-09 11:32:16 AM Running minimization in window p001...
    2020-10-09 11:32:21 AM Running minimization in window p002...
    2020-10-09 11:32:25 AM Running minimization in window p003...
    2020-10-09 11:32:30 AM Running minimization in window p004...
    2020-10-09 11:32:34 AM Running minimization in window p005...
    2020-10-09 11:32:39 AM Running minimization in window p006...
    2020-10-09 11:32:43 AM Running minimization in window p007...
    2020-10-09 11:32:48 AM Running minimization in window p008...
    2020-10-09 11:32:56 AM Running minimization in window p009...
    2020-10-09 11:33:01 AM Running minimization in window p010...
    2020-10-09 11:33:05 AM Running minimization in window p011...
    2020-10-09 11:33:10 AM Running minimization in window p012...
    2020-10-09 11:33:14 AM Running minimization in window p013...
    2020-10-09 11:33:19 AM Running minimization in window p014...
    2020-10-09 11:33:23 AM Running minimization in window p015...
    2020-10-09 11:33:28 AM Running minimization in window p016...
    2020-10-09 11:33:32 AM Running minimization in window p017...

### Thermalization
Here, we will do a slow heating process to demonstrate the `custom_run_command` feature of the `NAMD` wrapper. We will heat the system from 50 K to 300 K in steps of 50 K and will do a total of 500 MD steps for each temperature.

```python
for window in window_list:
    simulation = NAMD()
    simulation.executable = "namd2"
    simulation.n_threads = 4
#     simulation.gpu_devices = 0
    simulation.path = os.path.join(work_dir, window)

    simulation.topology = f"{base_name}.prmtop"
    simulation.coordinates = f"{base_name}.rst7"
    simulation.prefix = "thermalization"
    simulation.checkpoint = "minimize"
    simulation.colvars_file = "colvars.tcl"

    simulation.config_gb_md()

    # Slow heating
    simulation.custom_run_commands = [
        "for {set temp 50} {$temp <= 300} {incr temp 50} {",
        "langevinTemp $temp",
        "reinitvels $temp",
        "run 500",
        "}",
    ]

    logging.info(f"Running thermalization in window {window}...")
    simulation.run(overwrite=True)
```

    2020-10-09 11:33:37 AM Running thermalization in window a000...
    2020-10-09 11:33:39 AM Running thermalization in window a001...
    2020-10-09 11:33:40 AM Running thermalization in window a002...
    2020-10-09 11:33:42 AM Running thermalization in window a003...
    2020-10-09 11:33:44 AM Running thermalization in window a004...
    2020-10-09 11:33:46 AM Running thermalization in window a005...
    2020-10-09 11:33:47 AM Running thermalization in window a006...
    2020-10-09 11:33:49 AM Running thermalization in window a007...
    2020-10-09 11:33:51 AM Running thermalization in window a008...
    2020-10-09 11:33:53 AM Running thermalization in window a009...
    2020-10-09 11:33:54 AM Running thermalization in window a010...
    2020-10-09 11:33:56 AM Running thermalization in window a011...
    2020-10-09 11:33:58 AM Running thermalization in window a012...
    2020-10-09 11:34:00 AM Running thermalization in window a013...
    2020-10-09 11:34:02 AM Running thermalization in window p000...
    2020-10-09 11:34:06 AM Running thermalization in window p001...
    2020-10-09 11:34:09 AM Running thermalization in window p002...
    2020-10-09 11:34:13 AM Running thermalization in window p003...
    2020-10-09 11:34:15 AM Running thermalization in window p004...
    2020-10-09 11:34:17 AM Running thermalization in window p005...
    2020-10-09 11:34:18 AM Running thermalization in window p006...
    2020-10-09 11:34:20 AM Running thermalization in window p007...
    2020-10-09 11:34:22 AM Running thermalization in window p008...
    2020-10-09 11:34:24 AM Running thermalization in window p009...
    2020-10-09 11:34:25 AM Running thermalization in window p010...
    2020-10-09 11:34:27 AM Running thermalization in window p011...
    2020-10-09 11:34:29 AM Running thermalization in window p012...
    2020-10-09 11:34:30 AM Running thermalization in window p013...
    2020-10-09 11:34:32 AM Running thermalization in window p014...
    2020-10-09 11:34:34 AM Running thermalization in window p015...
    2020-10-09 11:34:36 AM Running thermalization in window p016...
    2020-10-09 11:34:37 AM Running thermalization in window p017...

We will skip the equilibration step and go straight to production

### Production Run
We will run the production phase for 5,000 steps.

**Note**: a proper production run would require much longer simulation.

```python
for window in window_list:
    simulation = NAMD()
    simulation.executable = "namd2"
    simulation.n_threads = 4
#     simulation.gpu_devices = 0
    simulation.path = os.path.join(work_dir, window)

    simulation.topology = f"{base_name}.prmtop"
    simulation.coordinates = f"{base_name}.rst7"
    simulation.prefix = "production"
    simulation.checkpoint = "thermalization"
    simulation.colvars_file = "colvars.tcl"

    simulation.config_gb_md()
    simulation.control["run"] = 5000

    logging.info(f"Running production in window {window}...")
    simulation.run(overwrite=True)
```

    2020-10-09 11:34:39 AM Running production in window a000...
    2020-10-09 11:34:41 AM Running production in window a001...
    2020-10-09 11:34:44 AM Running production in window a002...
    2020-10-09 11:34:46 AM Running production in window a003...
    2020-10-09 11:34:49 AM Running production in window a004...
    2020-10-09 11:34:52 AM Running production in window a005...
    2020-10-09 11:34:54 AM Running production in window a006...
    2020-10-09 11:34:57 AM Running production in window a007...
    2020-10-09 11:35:00 AM Running production in window a008...
    2020-10-09 11:35:02 AM Running production in window a009...
    2020-10-09 11:35:05 AM Running production in window a010...
    2020-10-09 11:35:10 AM Running production in window a011...
    2020-10-09 11:35:15 AM Running production in window a012...
    2020-10-09 11:35:17 AM Running production in window a013...
    2020-10-09 11:35:21 AM Running production in window p000...
    2020-10-09 11:35:25 AM Running production in window p001...
    2020-10-09 11:35:29 AM Running production in window p002...
    2020-10-09 11:35:33 AM Running production in window p003...
    2020-10-09 11:35:36 AM Running production in window p004...
    2020-10-09 11:35:40 AM Running production in window p005...
    2020-10-09 11:35:42 AM Running production in window p006...
    2020-10-09 11:35:46 AM Running production in window p007...
    2020-10-09 11:35:51 AM Running production in window p008...
    2020-10-09 11:35:54 AM Running production in window p009...
    2020-10-09 11:35:58 AM Running production in window p010...
    2020-10-09 11:36:06 AM Running production in window p011...
    2020-10-09 11:36:15 AM Running production in window p012...
    2020-10-09 11:36:21 AM Running production in window p013...
    2020-10-09 11:36:27 AM Running production in window p014...
    2020-10-09 11:36:32 AM Running production in window p015...
    2020-10-09 11:36:37 AM Running production in window p016...
    2020-10-09 11:36:41 AM Running production in window p017...

## Analysis

Once the simulation is completed, we can using the `Analysis` module to determine the binding free energy. We supply the location of the parameter information, a string or list for the file names (wildcards supported), the location of the windows, and the restraints on the guest.

In this example, we use the method `ti-block` which determines the free energy using **t**hermodynamic **i**integration and then estimates the standard error of the mean at each data point using blocking analysis. Bootstrapping it used to determine the uncertainty of the  full thermodynamic integral for each phase.

After running `compute_free_energy()`, a dictionary called `results` will be populated, that contains the free energy and SEM for each phase of the simulation.

```python
import paprika.analysis as analysis
```

```python
free_energy = analysis.fe_calc()
free_energy.topology = "cb6-but-dum.prmtop"
free_energy.trajectory = 'production*.dcd'
free_energy.path = work_dir
free_energy.restraint_list = guest_restraints
free_energy.collect_data()
free_energy.methods = ['ti-block']
free_energy.ti_matrix = "full"
free_energy.bootcycles = 1000
free_energy.compute_free_energy()
```

We also need to calculate the free-energy cost of releasing the restraints on the guest molecule.

```python
free_energy.compute_ref_state_work([
    guest_restraints[0], guest_restraints[1], None,
    None, guest_restraints[2], None
])
```

Then we add the free-energies together and combine the uncertainties to get the binding-free energy

```python
binding_affinity = -1 * (
    free_energy.results["attach"]["ti-block"]["fe"] + \
    free_energy.results["pull"]["ti-block"]["fe"] + \
    free_energy.results["ref_state_work"]
)

sem = np.sqrt(
    free_energy.results["attach"]["ti-block"]["sem"]**2 + \
    free_energy.results["pull"]["ti-block"]["sem"]**2
)
```

```python
print(f"The binding affinity of butane to cucurbit[6]uril = {binding_affinity.magnitude:0.2f} +/- {sem.magnitude:0.2f} kcal/mol")
```

    The binding affinity of butane to cucurbit[6]uril = -6.16 +/- 4.40 kcal/mol


>
Copyright 2021 DeepMind Technologies Limited.

>
 Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

>
 https://www.apache.org/licenses/LICENSE-2.0

>
 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

# **Unsupervised learning**

**Aim**
Provide you with the basics of the unsupervised learning. It is intended as a practical guide, so do not expect a solid theoretical background. You'll learn about the connection between neural networks and probability theory, how to build and train an autoencoder with only basic python knowledge, and how to compress an image using the $\mathrm{K\!-\!means}$ clustering algorithm. Lots of visualisations and exercises are included to make this journey fun.

**Disclaimer**

This code is intended for educational purposes, and in the name of readability for a non-technical audience does not always follow best practices for software engineering.

**Links to resources**
- [What is Colab?](https://colab.sandbox.google.com/notebooks/intro.ipynb) If you have never used Colab before, get started here!

# **Introduction**

Machine learning is often categorised into three types:
- **Supervised learning**, which provides the machine with input-output pairs, i.e. for each observation, the user defines the desired output which the machine needs to learn;
- **Reinforcement learning**, where instead of target outputs, the machine receives a more general feedback (the reward), which it tries to maximise (e.g. winning at chess);
- **Unsupervised learning**, which works solely with the observations.
The machine is expected to discover patterns in the data and create their compact representation.

Typical topics addressed in unsupervised learning are:
- **Density estimation**, which models the probability distribution of data, $p(x)$;
<image src="https://storage.googleapis.com/dm-educational/assets/unsupervised-learning/density_estimation.png" >
- **Clustering**, which seeks to discover (and characterise) groups of similar examples in the data (data clusters);
<image src="https://storage.googleapis.com/dm-educational/assets/unsupervised-learning/clustering.png" >
- **Dimensionality reduction / blind source separation / latent variable models**, which enable analysis and visualisation, as well as compression by extracting the most informative pieces of information from a dataset (e.g. Principle Component Analysis, Factor Analysis).

These goals are interrelated. For example, we will see how clustering can be cast as a way to estimate data density and how it can enable data compression.

**Probabilistic Models** and **Neural Networks** are two main families of Unsupervised Learning algorithms. There are many correspondences between the two and thus knowing the basics of probability theory helps in developing a deeper understanding of Unsupervised Learning.

**Famous Neural Networks for Unsupervised Learning**

One of the most famous networks for clustering is an Autoencoder (AE). If you have never come across the name, [this blog post](https://hackernoon.com/how-to-autoencode-your-pok%C3%A9mon-6b0f5c7b7d97) offers a fun introduction.

Neural networks for density estimation are called normalizing flows, they output $p(x)$ directly. Example neural networks for clustering are Self-Organising Maps (SOM) and Neural Gas. Probably the most famous latent variable model is the Variational Autoencoder (VAE). Similar to the classical autoencoder, VAE computes a compressed representation of the data - the latent code. However, unlike AE, VAE is in a position to produce new data samples, i.e. it is a generative model. Another famous generative model is Generative Adversarial Network (GAN), although it is less motivated by probabilistic modelling (and difficult to evaluate).

For more on neural networks in unsupervised learning, check out this great [INDABA tutorial](
https://github.com/deep-learning-indaba/indaba-pracs-2019/blob/master/3b_generative_models.ipynb)
which covers both VAEs and GANs.

**In this tutorial**, we cover the basics of [density estimation](#scrollTo=HU-jyOW-HTlQ&line=2&uniqifier=1), [clustering](#scrollTo=JuzhThGFPJ_M&line=4&uniqifier=1) and [latent variable modelling](#scrollTo=mT5bu54-pXnj). We also cover the neural network basics by coding up a simple [autoencoder](#scrollTo=nits21YIkI_h). We finish with an example of how clustering can be used for [image compression](#scrollTo=m7LBpsTSIhk_).

**Further reading:** [Chris Bishop. Pattern Recognition and Machine Learning](https://www.microsoft.com/en-us/research/people/cmbishop/prml-book/).
Many exercises in this tutorial were inspired by this excellent book.

## **Setup**

```
import numpy as np
# Basic plotting in style:
import matplotlib.pyplot as plt
import seaborn as sns
# Animations:
from matplotlib import animation
from IPython.display import HTML
```

```
#@title Data generating functions
# 0, 2, 5, 10, 80
#https://tall.life/height-percentile-calculator-age-country/
HIDDEN_MU = [49.3, 50, 86.2, 87.6, 108.1, 109.4, 139.1, 138, 155.9, 170.7]
HIDDEN_SIGMA = [1.88, 1.87, 3.24, 3.08, 4.91, 4.61, 6.97, 6.88,  6.32, 6.63]
def get_data_a(n=1000, k=0):
  return np.random.randn(n) * HIDDEN_SIGMA[k] + HIDDEN_MU[k]

HIDDEN_PI = np.random.dirichlet([1, 1, 1])
#https://www.ons.gov.uk/peoplepopulationandcommunity/populationandmigration/populationestimates/bulletins/annualmidyearpopulationestimates/mid2019estimates
HIDDEN_PI = np.array([.514, .486, .513, .487, .512, .488, .514, .486, .55, .45])
HIDDEN_PI /= np.sum(HIDDEN_PI)
def get_responsibilities(p, n=10000):
  n_per_cluster = np.random.multinomial(n, p)
  r = [[k] * n_k for k, n_k in enumerate(n_per_cluster)]
  r = np.concatenate(r)
  np.random.shuffle(r)
  return r

R = get_responsibilities(HIDDEN_PI)
def get_data_b():
  y = np.random.randn(len(R))
  for t, k in enumerate(R):
    y[t] *= HIDDEN_SIGMA[k]
    y[t] += HIDDEN_MU[k]
  return y

def get_data_c(n_x=6):
  return np.eye(n_x * 3).reshape((-1, n_x, 3))
```

```
#@title Plotting settings
plt.rcParams['figure.figsize'] = [12, 2.5]
sns.set_style('whitegrid')
sns.set_context('notebook', font_scale=1.3)
```

# **Density estimation**
Density estimation aims to describe observed data by approximating its probability density function.

In practice, datasets are finite, and thus we often consider probability mass rather than densities. For continous variables, this requires a binning process, which combines the number of possible values into discrete sets (bins).

The data itself forms an **empirical data distribution**: it reflects how many samples of a given kind were observed. An alternative estimate is the **parametric data distribution**, which refers to a set of parameters, $\Theta$, such as the mean and variance of a Gaussian. Parametric data distributions allow for an approximate, concise description of the data. They can also relate the underlying data generating process.

In this section, we will consider a continuous random variable. We will
- visualise the binning process with a histogram,
- normalise the binned counts to obtain the empirical data distribution and density estimate, and also
- approximate it with a parametric data distribution (a Gaussian).

But how do we find the best parameters to fit the data?
And what do we even mean by "the best"? We cover these important topics in the remainder of this section.

Let us look at the first dataset: the height of newborn baby girls. (Samples were generated according to statistics available [online](https://tall.life/height-percentile-calculator-age-country/)).

Height is a continuous variable, so we start by binning the data. This process is best visualised with a **histogram**. A histogram divides the domain into discrete, nonoverlapping sets (bins) and it measures how many samples fall within each group.

```
x_a = get_data_a()
counts, bins, _ = plt.hist(x_a, bins=20)
plt.title(f'Histogram of our data (N={len(x_a)})')
plt.ylabel('Counts (bin)')
plt.xlabel('x (cm)');
```

By default, `plt.hist()` function displays the number of counts per bin, $n_k$. For probability mass, we need to divide the bin counts by the number of all samples, $N$.
$$P(x\in bin_k) = \frac{n_k}{N}$$

```
p_bins = counts / np.sum(counts)

# Let's plot the distribution
dx = bins[1] - bins[0]
bin_centers = bins[:-1] + dx / 2
plt.bar(bin_centers, p_bins, dx)
plt.title('Empirical data distribution')
plt.ylabel('P(x in bin)')
plt.xlabel('x (cm)');
```

Thus, we arrived at an empirical probability distribution, $P(x\in bin_k)$ over a discrete number of bins.

To treat the continuous $x$ directly, we have to estimate probability density, $p(x)$.

One straightforward way to do this is to divide the probability mass by bin size, $|bin_k|$. By default `plt.hist()` uses bins of equal length, $\Delta x$, which simplifies the procedure:
$$p(x\in bin_k) = \frac{P(x\in bin_k)}{|bin_k|} = \frac{P(x\in bin_k)}{\Delta x}$$
This is what we get when setting `plt.hist(x, density=True)`.

```
p_bins_density = p_bins / dx
plt.step(bin_centers, p_bins_density,
         where='mid', c='tab:orange', lw=3, label=r'P(bin)/|bin|')
plt.hist(x_a, bins=20, density=True, label='hist(x, density=True)')

plt.title('Empirical data density')
plt.ylabel('p(x)')
plt.xlabel('x (cm)');
plt.legend()
```

Do you recognise this shape?

##**Gaussian distribution**

The **normal distribution** (**Gaussian distribution**) is one of the most important distributions describing real-valued observations, $x\in\mathbb{R}$. It is characterised by two parameters, the mean ($\mu\in\mathbb{R}$) and the variance ($\sigma^2 \in\mathbb{R_+}$).
$$p(x|\mu, \sigma^2) = \frac{1}{\sqrt{2 \pi \sigma^2}}
 e^{\left . -(x - \mu)^2\small \right / 2\sigma^2} $$

The constant term ensures that, as with all probability distributions:
$$\int_{-\infty}^\infty p(x|\mu, \sigma^2) dx = 1.$$
(For a discrete random variable, we would use a sum rather than an integral over all the possible values of $x$.)

Let's plot the Gaussian probability density. The shape of this curve is often referred to as the "bell curve".

```
def bell_curve(x, mu, sigma_2):
  y = np.exp(- (x - mu)**2 / 2 / sigma_2)
  y /= np.sqrt(2 * np.pi * sigma_2)
  return y
```

Let's use the empirical mean and variance as the parameters of our Gaussian:

```
x_mean, x_var, n_x = np.mean(x_a), np.var(x_a), len(x_a)
print('Mean of our data: {:.2g}cm, variance: {:.2g}cm^2 (N={})'.format(x_mean, x_var, n_x))
```

```
x_ = np.linspace(np.min(x_a), np.max(x_a), 100)
p_x = bell_curve(x_, x_mean, x_var)
plt.plot(x_, p_x)
plt.title(r'Gaussian probability density')
plt.ylabel('p(x)')
plt.xlabel('x (cm)');
```

Indeed, the shape is similar to our data distribution! Let's overlay the two:

```
plt.hist(x_a, bins=20, density=True, label='data')
plt.plot(x_, p_x, lw=3, label='fit')
plt.title('Empirical data density')
plt.ylabel('p(x)')
plt.xlabel('x (cm)');
plt.legend();
```

Note, that we could compare probability masses $P(x\in bin)$ rather than densities $p(x)$. We just need to integrate the density over the bins
$$P_{\mu, \sigma^2}(x\in bin) = \int_{x \in bin} p(x |\mu, \sigma^2) dx.$$
In practice, we will approximate it with $P_{\mu, \sigma^2}(x\in bin) \sim p(\hat x|\mu, \sigma^2) \Delta x$, with $\hat x$ corresponding to the center of the bin.

```
plt.bar(bin_centers, p_bins, dx, label='data')

p_gauss_bins = dx * bell_curve(bin_centers, x_mean, x_var)
plt.step(bin_centers, p_gauss_bins, where='mid', c='tab:orange', lw=3,
         label='fit')

plt.title('Empirical data distribution and its Gaussian fit')
plt.ylabel('P(x in bin)')
plt.xlabel('x (cm)');
plt.legend();
```

## **Maximum Likelihood Estimates (MLE)**
Our dataset is very well described by the normal distribution with parameters estimated from the dataset. This was for a good reason. The empirical mean and variance are obtained by maximising the likelihood of data under Gaussian probability:
$$\mathcal L (\mathcal D|\mu, \sigma^2)= \prod_{n=1}^N p(x_n|\mu, \sigma^2).$$

In general, for a chosen parametric distribution $P(X|\Theta)$ and some data $\mathcal D$, the **likelihood** is defined as
$$\mathcal L (\mathcal D|\Theta)= \prod_{x_n\in\mathcal D} p(x_n|\Theta).$$
$\mathcal L (\mathcal D|\Theta)$ is often used as a measure of how well $P(X|\Theta)$ fits the data.

The parameters that optimise this measure are called **maximum likelihood estimates (MLE)**,  $$\text{MLE}={\arg\max}_\Theta \mathcal L (\mathcal D|\Theta).$$

Likelihood is probably the most popular measure of the quality of fit in a probabilistic setting. However, it is numerically challenging: with a growing number of samples, its values quickly approach zero.

### **Log-Likelihood**
Instead of evaluating the likelihood, it's much more practical to evaluate its logarithm, $\log \left(\mathcal L(\mathcal D)\right)$. Log-likelihood makes it easier to keep track of values close to zero. It also turns the massive product $\prod_n$ into a sum:
$$\log\left(\mathcal \prod_n p (\mathcal x_n|\theta)\right)
 =\sum_n \log (p (\mathcal x_n|\theta)),$$
vastly simplifying differentiation.

Importantly, the logarithm is a monotonically increasing function, so the MLE will be the same for $\mathcal L$ and $\log (\mathcal L)$,
$${\arg\max}_\Theta \mathcal L (\mathcal D|\Theta) = {\arg\max}_\Theta \log(\mathcal D|\Theta ).$$

Let's compute the log-likelihood for our Gaussian distribution:
\begin{align}
\sum_n \log [p(x_n|\mu, \sigma^2)] = \sum_n  \left(\frac{-(x_n - \mu)^2 }{ 2\sigma^2 } - \frac 1 2 \log 2\pi\sigma^2\right)
\end{align}

### **Log-Loss**
Machine learning prefers to use the language of **cost** or **loss** functions that need to be minimised. Thus, you might come across **log-loss**, which is nothing else than a negative log-likelihood:
$$C(\mathcal D|\Theta) = -  \log(\mathcal L (\mathcal D|\Theta)).$$

Yet again, optimising log-loss recovers MLE:
$${\arg\max}_\Theta \mathcal L (\mathcal D|\Theta) = {\arg\min}_\Theta C(\mathcal D|\Theta)$$

This is the cost function used by most neural networks in the unsupervised setting.

###**Task: Optimise by hand**
Using the sliders, try to do the NN's job and minimise the cost function. How well can you fit our data distribution? What happens (and why) when the variance is zero?

```
def log_loss_gaussian(x_, mu, sigma_2):
  log_p = ((x_ - mu)**2 / sigma_2  + np.log(sigma_2) + np.log(2 * np.pi)) / 2
  return np.sum(log_p)
```

```
#@title Fit a Gaussian {run: "auto"}
mu = 50.1597 #@param{type:'slider', min:40, max:60, step:1e-4}
sigma_2 = 9.8342 #@param{type:'slider', min:0, max:100, step:1e-4}

plt.hist(x_a, bins=20, density=True, label='data',
         facecolor='.9', edgecolor='.3')
plt.plot(x_, bell_curve(x_, mu, sigma_2), lw=3, label='fit')
plt.title('Negative log-likelihood = {:.2f}'.format(log_loss_gaussian(x_a, mu, sigma_2)))
plt.ylabel('p(x)')
plt.xlabel('x (cm)');
plt.legend();
```

##**Extra reading: Derivation of MLE**
**1. Use log-likelihood**
As discussed above, maximising log-likelihood is equivalent to maximising likelihood. For a Gaussian:
\begin{align}
\sum_n \log [p(x_n|\mu, \sigma^2)] = \sum_n  \left(-\frac{(x_n - \mu)^2 }{ 2\sigma^2 } - \frac 1 2 \log 2\pi\sigma^2\right)
\end{align}

**2. At maximum, the gradients should be zero**

Now we need to compute the gradients of the log-likelihood with respect to the parameters $\mu$ and $\sigma^2$:

\begin{align}
\frac{d \log \left[\mathcal L(\mathcal D)\right]}{d\mu} &= \sum_n \frac{2(x_n - \mu)}{2\sigma^2} = \frac{1}{\sigma^2} \left(\sum_n x_n - N \mu \right)\\
\frac{d \log \left[\mathcal L(\mathcal D)\right]}{d\sigma^2} &=
\sum_n \left(\frac{(x_n - \mu)^2}{2} \frac{1}{(\sigma^2)^2} - \frac{1}{2 \sigma^2} \right)= \frac{1}{2(\sigma^2)^2} \left(\sum_n (x_n -\mu)^2 - N \sigma^2\right)
\end{align}

Setting these gradients to zero we get:
$$\mu_{\mathrm{MLE}}= \frac 1 N \sum_n (x_n) \qquad \sigma^2_{\mathrm{MLE}} = \frac 1 N \sum_n (x_n -\mu)^2$$
which you might recognise as definitions of empirical mean and variance.

**3. At maximum, the second derivative should be positive**

We leave it as an exercise.

# **Clustering with $\mathrm{K\!-\!means}$**
Our first dataset was quite simple, it was well approximated by a Gaussian distribution. Thus, mean (49cm) and variance (3.7cm$^2$) were statistics sufficient to succinctly describe it. But in real life, that's rarely the case.

In this section, we explore one of the most important algorithms for clustering: $\mathrm{K\!-\!means}$. For this purpose, we load another dataset.

```
x_b = get_data_b()
plt.hist(x_b, bins=200, density=True, histtype='stepfilled');
plt.title('Empirical data density')
plt.ylabel('p(x)')
plt.xlabel('x (cm)');
```

Clearly, the bell curve is no match for this dataset - its distribution is multimodal (it has many peaks). It seems appropriate to instead describe it by clusters.

##**$\mathrm{K\!-\!means}$**

The $\mathrm{K\!-\!means}$ algorithm assumes that data can be well described by clusters. The number of clusters ($K$) needs to be decided upfront. Each cluster is associated with a template, i.e. the mean of all samples that belong to the cluster. The $\mathrm{K\!-\!means}$ algorithm iteratively decides which cluster the data belongs to and what is the template most suitable for each cluster.

1. Input: Data ($\mathcal{D}\in \mathbb{R}^{d \times N}$) and number of clusters $K$.

  Here, we consider a one-dimensional case, $d=1$, with $N=3000$ samples. We make a guess $K=4$.
2. Initialise the templates for each cluster $k$:  $\mu_k\in\mathbb{R}^{d}$.
3. Repeat (for a pre-defined number of steps or until convergence):
  -  **$\mathrm{E}$-step**: For each sample $n$, decide which cluster it belongs to, $z_n\in\{1, \ldots, K\}$.   $z_n$ is a categorical variable which minimises distance between the sample and the cluster: $z_n = \arg \min_k D(x_n, \mu_k)$.

    In our 1-d case, measuring distance amounts to checking the absolute value of the scalar difference:  $D(x_n, \mu_k) = |x_n-\mu_k|$.

  - **$\mathrm{M}$-step**: update $\mu_k$ by computing the mean over all samples assigned to a given cluster: $\mu_k= \langle x_n\rangle_{\mathrm z_n=k}$.

**Why do we call the steps E and M?**

The reason we refer to the iterative steps as E and M is that $\mathrm{K\!-\!means}$ algorithm can be cast as a special case of the Expectation Maximisation algorithm.
- $\mathrm{E}$-step computes the expectation over our latent variable $z_n$ for each data sample $n$ (only one cluster can be assigned to any data sample, so our "expectation" is simply the id of the most likely cluster).
- $\mathrm{M}$-step maximises the likelihood of the data by adjusting cluster parameters ($\mu_k$), where we can assume that the cluster is well described by a Gaussian with a fixed variance (as shown [above](#scrollTo=NoSEEAtUIk-l&line=18&uniqifier=1), empirical mean is the maximum likelihood value for the $\mu$ parameter of a Gaussian).

We first explain how to implement every step of this algorithm and then write a little program to visualise the learning via $\mathrm{K\!-\!means}$.

##**Initialise the parameters**

```
# Init
n_c = 4 # n_c is the number of clusters (K in the algorithm description)
n_x = len(x_b)
mu_b = np.linspace(min(x_b), max(x_b), n_c)

plt.hist(x_b, bins=200, density=True, histtype='stepfilled',
         facecolor='.9', edgecolor='.3')
mu_colors = sns.color_palette(n_colors=n_c)
for k_c, mu_k in enumerate(mu_b):
  plt.axvline(mu_k, c=mu_colors[k_c])

plt.title('Empirical data density and cluster means')
plt.ylabel('p(x)')
plt.xlabel('x (cm)');
```

##**$\mathrm{E}$-step**
Assign each sample to one of the clusters.

We use different colours for each of the clusters. The vertical lines depict the current cluster templates and the shaded area shows the part of the input domain that was assigned to a given cluster. Assigned clusters (indexed by 0, 1, 2, 3) are plotted on the y-axis.

```
r = np.zeros(n_x, dtype='int16')
for k, x_k in enumerate(x_b):
  r[k] = np.argmin(np.abs(x_k - mu_b))

plt.plot(x_b, r, '.', alpha=.2, color='.2')
for k_c, mu_k in enumerate(mu_b):
  x_b_c = x_b[r == k_c]
  plt.axvspan(min(x_b_c), max(x_b_c),
              facecolor=mu_colors[k_c], alpha=0.2)
  plt.axvline(mu_k, c=mu_colors[k_c])

plt.title('E: Assigning responsibility')
plt.ylabel('r(x)')
plt.xlabel('x (cm)');
```

##**$\mathrm{M}$-step**
Given the assignement of samples to the clusters, compute the new templates.

```
# Maximise likelihood for each of the clusters:
for k_c in range(n_c):
  mu_b[k_c] = np.mean(x_b[r == k_c])

plt.hist(x_b, bins=200, density=True, histtype='stepfilled',
         facecolor='.9', edgecolor='.3')
for k_c, mu_k in enumerate(mu_b):
  plt.axvline(mu_k, c=mu_colors[k_c])
plt.title('M: recompute the means')
plt.ylabel('p(x)')
plt.xlabel('x (cm)');
```

##**Putting it all together**
Let's write functions for each step of the algorithm and let's call them iteratively.

We plot cluster templates for each iteration and use transparency to depict learning progress. (Transparency is commonly referred to as `alpha` parameter in `matplotlib` functions.) This gives us a quick and computationally inexpensive way to monitor the progress of training.

```
def e_step(x, mu):
  r = [np.nanargmin(np.abs(x_k - mu))
      for k, x_k in enumerate(x)]
  return np.array(r)

def m_step(x, r, cluster_ids):
  mu_c = np.ones_like(cluster_ids) * np.nan
  for k_c in cluster_ids:
    x_c = x[r == k_c]
    if np.any(x_c):
      mu_c[k_c] = np.mean(x_c)
  return mu_c

n_steps = 10
plt.hist(x_b, bins=200, density=True, histtype='stepfilled',
         facecolor='.9', edgecolor='.3')

for t in range(n_steps):
  r = e_step(x_b, mu_b)
  mu_b = m_step(x_b, r, range(n_c))
  for k_c, mu_k in enumerate(mu_b):
    plt.axvline(mu_k, c=mu_colors[k_c], alpha=max(.2, t/n_steps), lw=3)
```

Below we provide code to save our data and display them as an animation. Please, take a look (by double-clicking on the cell)! `matplotlib` library is truly powerful!

```
#@title Visualisation code
def k_means(x, n_clusters, n_steps=10):
  n_steps += 1 # Need one for the initial values.
  mu_over_time = np.zeros((n_steps, n_clusters))
  rlim_over_time = np.zeros((n_steps, n_clusters, 2)) * np.nan
  mu_over_time[0] = np.random.randn(n_clusters) * np.std(x) + np.mean(x)
  for t in range(1, n_steps):
    r = e_step(x, mu_over_time[t - 1])
    for k_c in range(n_clusters):
      x_b_k = x_b[r == k_c]
      if len(x_b_k):
        rlim_over_time[t, k_c] = [min(x_b_k), max(x_b_k)]
    mu_over_time[t] = m_step(x, r, range(n_clusters))
  return mu_over_time, rlim_over_time

def animate_kmeans(x, mu_over_time, rlim_over_time):
  n_steps, n_clusters = np.shape(mu_over_time)
  colors = sns.color_palette(n_colors=n_clusters)

  fig, ax = plt.subplots()
  # Initialise graphics:
  areas = [plt.axvspan(np.nan, np.nan, facecolor=colors[k_c], alpha=0.2)
          for k_c in range(n_clusters)]
  plt.hist(x, bins=200, density=True, histtype='stepfilled',
           facecolor='.9', edgecolor='.3')
  lines = [plt.axvline(0, c=colors[k_c])
          for k_c in range(n_clusters)]

  def animate_(frame_no):
    frame_type = frame_no % 2
    t = frame_no // 2
    for k_c, mu_k in enumerate(mu_over_time[t]):
      if frame_type or frame_no == 0:
        lines[k_c].set_data([mu_k, [0, 1]])
      else:
        xy = areas[k_c].get_xy()
        xy[:, 0] = rlim_over_time[t][k_c][0]
        xy[2:-1, 0] = rlim_over_time[t][k_c][1]
        areas[k_c].set_xy(xy)
    if t == 0:
      s_title = 'Initialise clusters'
    else:
      s_title = '{}. {}-Step'.format(t, ['E', 'M'][frame_type])
    ax.set_title(s_title)

  my_anim = animation.FuncAnimation(fig, animate_, frames=2 * n_steps,
                                interval=600)
  plt.close()
  return HTML(my_anim.to_jshtml())
```

```
params_over_time = k_means(x_b, n_clusters=4)
animate_kmeans(x_b, *params_over_time)
```

##**How do we decide the number of clusters?**

What happens when we try to run the $\mathrm{K\!-\!means}$ algorithm with more clusters? Here, we initialise the means randomly.

Feel free to modify the number of clusters or the number of steps to run the algorithm for.

```
params_over_time = k_means(x_b, n_clusters=10, n_steps=20)
animate_kmeans(x_b, *params_over_time)
```

We see that sometimes the clusters become irrelevant (when there is no data assigned to a cluster, it vanishes). Some seem superflouous.
Were they anywhere close to the truth?

##**Spoiler alert: the ground truth about our data**
The data actually came from 10 clusters.

```
x_ = np.linspace(np.min(x_b), np.max(x_b), 1000)
for k, c in enumerate(sns.color_palette('Paired', 10)):
  plt.hist(x_b[R == k], bins=50, density=True, histtype='stepfilled',
           edgecolor='none', color=c, alpha=.5)
  plt.plot(x_, bell_curve(x_, HIDDEN_MU[k], HIDDEN_SIGMA[k]**2), c=c,
           label=[0, 1, 5, 10, 80][k // 2])

plt.title('Distributiton of heights for different ages (in years).')
plt.ylabel('Probability density')
plt.xlabel('Height (cm)')
plt.legend(ncol=5);
```

The data reflect the [height statistics](https://tall.life/height-percentile-calculator-age-country/) for female and male genders across age. As you can see, there is a large overlap between these clusters. While $\mathrm{K\!-\!means}$ works well for separable patterns, it stood no chance to decipher the ground truth of our example. Are there better methods to address our problem?

Before you say "neural networks", notice that our data has a very compact description - it is a **mixture of (Gaussian) distributions**, with age and gender deciding parameters of each cluster. Because we had access to measurements only, age and gender can be considered as **latent variables** -- had we known them, they'd allow us to characterise this distribution well.

#**Latent variables in probabilistic modelling. $\mathrm{EM}$ algorithm**
In probabilistic modelling, latent variables enable a compact description of data. In our example above, such a latent variable was the id of the cluster from which our height data was drawn, $z_n\in\{1,\ldots, K\}$. However, our assignment was deterministic - only one cluster was assigned to each data sample.
A **mixture of distributions** is a probabilistic extension of this approach, which models the beliefs over cluster assignments.

##**Prior and Posterior beliefs**
Beliefs can come in different flavours.
1. We might have some **prior** belief about the clusters - some of them might be more common, some might be rare. This we express with $K$ positive values which sum to 1: $0 \le \pi_{k} \le 1$ and $\sum_k \pi_{k}=1$.
2. After seeing a data sample $x$, we can update our belief about the cluster assignment, calling it a **posterior** belief $P(z=k|x)$. According to the Bayes theorem,
$$P(z=k|x) = \frac{\pi_k p(x|z=k)}{\sum_k \pi_k p(x|z=k)}$$
where the denominator makes sure that our posterior belief sums to 1.

$p(x|z=k)$ is the probability density similar to the one we talked about above, except that now we assume it was generated by the $k$-th cluster. We refer to it as **likelihood** of the data **conditioned** on cluster id.

###**Notation**
Here's a quick recap of the notation relating to the Bayes theorem.
- $p(x|y)$ denotes conditioning, "probability of $x$ given $y$".
- $p(x,y)$ denotes a **joint** distribution, "probability of $x$ and $y$ co-occuring". It is simply $$p(x, y)=p(y|x)p(x).$$
- **Marginalising** means summing (/integrating) over all possible values of a random variable, e.g.
$$p(x)=\sum_{k=1}^K p(x, z_k)= \int_{y_{\min}}^{y_\max} p(x, y) dy$$
where we considered a discrete latent $z_k$ from a set of size K, and a continuous latent $y$ bounded by $y_\min$ and $y_\max$.

Considering the above, we can rewrite our posterior as:
$$p(z|x) = \frac{p(x|z)p(z)}{p(x)} = \frac{p(x, z)}{p(x)}$$
Bayes theorem is easiest to memorise as the following symmetrical expression:
$$p(x, z) = p(z|x)p(z) = p(x|z)p(x)$$

###**Task**
We observe a data point $x$, which could have come from either of two data sources. Source 1 generates similar samples with probability $p(x|z=1)=0.3$, source 2 rarely ever produces $x$ with that value, $p(x|z=2)=0.01$. However, source 2 is much more active than source 1, it produces 100x more data. What should be our posterior belief about how our data point was generated? What is the likelihood of our data point?

```
#@title Solution
# Prior beliefs: 1/11, 10/11 (need to sum to 1)
print('Prior beliefs p(z): 1/101, 100/101')
print('Likelihood p(x): 0.3/101 + 1/101 ~ .13')
print('Posterior p(z|x): (3/13, 10/13)')
print('According to Bayes, data comes from the 2nd source with ~76% probability')
```

###**Task**
Here's a task to improve familiarity of conditional, joint and marginal distributions. By manipulating the slider you can modify the prior on cluster freuqency $\pi=p(z)$ and the likelihood of data generated by the first cluster $p(y|z)$.
- How does the marginal change when cluster centers get closer together?
- When is the posterior on cluster assignment useful?
- What happens when you set the 1-st cluster prior to zero?
- Can you modify the code to control the variance of the distributions?

```
#@title Conditional, joint, marginal {run: "auto"}
pi_1 = 0.3695 #@param{type:'slider', min:0, max:1, step:1e-4}
mu_1 = 5.6 #@param{type:'slider', min:-20, max:20, step:1e-1}
mu_2 = 10
sigma_2 = 10
pi_2 = 1 - pi_1

x_ = np.linspace(-20, 20, 100)
p_1 = bell_curve(x_, mu_1, sigma_2)
p_2 = bell_curve(x_, mu_2, sigma_2)
lw = 3
_, axes = plt.subplots(1, 3, figsize=(18, 2.7))
plt.sca(axes[0])
plt.plot(x_, p_1, lw=lw, label='p(x|z=1)')
plt.plot(x_, p_2, lw=lw, label='p(x|z=2)')
plt.title('Conditional p(x|z)')
y_lim = plt.ylim()

plt.sca(axes[1])
p_zx_1 = p_1 * pi_1
p_zx_2 = p_2 * pi_2
p_x = p_zx_1 + p_zx_2
plt.plot(x_, p_zx_1, lw=lw, label='p(x,z=1)')
plt.plot(x_, p_zx_2, lw=lw, label='p(x,z=2)')
plt.plot(x_, p_x, lw=lw, label='p(x)')
plt.title('Joint p(x,z),  marginal p(x)')
y_lim = plt.ylim(y_lim)

plt.sca(axes[2])
plt.plot(x_, p_zx_1 / p_x, lw=lw, label='p(z=1|x)')
plt.plot(x_, p_zx_2 / p_x, lw=lw, label='p(z=2|x)')
plt.title('Conditional p(z|x)')

for ax in axes:
  plt.sca(ax)
  plt.legend(ncol=3, loc='upper center', bbox_to_anchor=[0.5, -.3]);
  plt.xlabel('x (cm)');
```

##**Generative model: mixture of Gaussians**
Formally, we assume the following probabilistic model:
\begin{eqnarray}
z &\sim& \mathrm{Categorical}(\mathbb{\pi})\\
x &\sim& \mathrm{Normal}(\mu_z, \sigma^2_z)
\end{eqnarray}
Which means that we believe our dataset was generated by the following process: For each sample $n$:
1. Draw the id of the cluster from a categorical distribution: $P(z_{n}=k)=\pi_k$.
2. Draw the sample from the Gaussian distribution associated with the $k$-th cluster: $p(x_n|z_n=k)=\frac{1}{\sqrt{2 \pi \sigma^2_k}} e^{\left . -(x_n - \mu_k)^2\small \right / 2\sigma^2_k}$

Taking the two together, we can write the likelihood of $x_n$ (data density according to the mixture of Gaussians) as:
$$p(x_n) = \sum_k \pi_k \frac{e^{\left . -(x_n - \mu_k)^2\small \right / 2\sigma^2_k}}{\sqrt{2 \pi \sigma^2_k}} $$

The likelihood of observing all our data $\mathcal{D}$:
$$\mathcal{L(D}|\Theta) = \prod_n\sum_k \pi_k \frac{e^{\left . -(x_n - \mu_k)^2\small \right / 2\sigma^2_k}}{\sqrt{2 \pi \sigma^2_k}} $$
where $\Theta$ now includes all $\pi_k$, $\mu_k$ and $\sigma^2_k$ for all K clusters.

We see that the logarithm of that likelihood does not yield a simple expression, like it did in the case of a single Gaussian:
$$\mathcal{C(D}|\Theta) = - \sum_n\log\left[\sum_k \pi_k \frac{e^{\left . -(x - \mu_k)^2\small \right / 2\sigma^2_k}}{\sqrt{2 \pi \sigma^2_k}} \right]$$

The summation over $k$ inside the logarithm makes the optimisation challenging.

##**$\mathrm{EM}$ algorithm**
Instead of optimising $\mathcal{C(D)}$, $\mathrm{EM}$ minimises $\langle\mathcal{C(D, Z)}\rangle_{p(Z|\mathcal{D})}$, i.e. expected value of the negative log-likelihood of the data and hidden variables, rather than just data, with expectation taken over the posterior on latent variables ($p(Z|\mathcal{D})$). In case of our mixture:
$$\langle\mathcal{C(D, Z)}\rangle_{p(Z|\mathcal{D})} = - \sum_n\sum_{k}P(z_n=k|x_n) \log\left[\pi_k \frac{e^{\left . -(x - \mu_k)^2\small \right / 2\sigma^2_k}}{\sqrt{2 \pi \sigma^2_k}}\right].$$
Note, that we got rid of the summation over $k$ inside the logarithm!

After deciding on the generative model and initialising its parameters, compute recursively.
1. $\mathrm{E}$-step: Compute the posterior over latent variables.

  For the mixture, the posterior of cluster ids, also referred to as the *responsibilities* is: $$r_{nk} \equiv P(z_n=k|x_n) = \frac{\pi_k p(x_n|z_n=k)}{\sum_k \pi_k p(x_n|z_n=k)}$$

2. $\mathrm{M}$-step: Maximise likelihood of the data according to the current posterior.

  For our mixture of Gaussians, we get the following set of equations:
  \begin{eqnarray}
  \mu_k &=& \frac {1}{N_k} \sum_n r_{nk} x_n\\
  \sigma_k^2 &=& \frac {1}{N_k} \sum_n r_{nk} (x_n - \mu_k)^2\\
  \pi_k &=& \frac{N_k}{N}
  \end{eqnarray}
where $N_k = \sum_n r_{nk}$ and $N$ is the number of all samples.

$\mathrm{EM}$ is an iterative algorithm that is sure to converge, although it might get stuck in local minima.

Despite minimising $\mathcal{C(D, Z|}\Theta)$, rather than the log-loss $\mathcal{C(D|}\Theta)$, it can be shown that it never increases the latter. At the end of this section, you can read about the variational-$\mathrm{EM}$ to further your understanding of this powerful optimisation technique.

```
def e_step_gaussian_mixture(x, mu, sigma_2, pi):
  # Posterior probability for each of data samples belonging to each cluster:
  r = np.array([pi_c * bell_curve(x, mu_c, sigma_2c)
       for pi_c, mu_c, sigma_2c in zip(pi, mu, sigma_2)]).T
  # normalising posterior to one:
  r /= np.sum(r, -1, keepdims=True)
  return np.array(r)

def m_step_gaussian_mixture(x, r):
  # Maximum likelihood:
  n_k = np.sum(r, 0)
  mu = np.sum(r * x[:, None], 0) / n_k
  sigma_2 = np.sum(r * (x[:, None] - mu[None])**2, 0) / n_k
  pi = n_k / np.sum(n_k)
  return mu, sigma_2, pi

def loss_gaussian_mixture(x, mu, sigma_2, pi):
  p = np.sum([pi_c * bell_curve(x, mu_c, sigma_2c)
       for pi_c, mu_c, sigma_2c in zip(pi, mu, sigma_2)], 0)
  return - np.sum(np.log(p))
```

Let's first initialise the algorithm deterministically. We spread our guess over the means evenly, we use a reasonable guess for $\sigma^2$ (based on the histogram above), and we use a flat prior over the frequency of clusters:

```
n_c = 10
n_x = len(x_b)
mu_b = np.linspace(min(x_b), max(x_b), n_c)
sigma_2b = [30] * n_c
pi = np.ones(n_c) / n_c
```

The responsibility variable $r_{nk}$ expresses the probability of $x_n$ being generated from the $k$-th distribution.

```
r_b = e_step_gaussian_mixture(x_b, mu_b, sigma_2b, pi)
mu_colors = sns.color_palette(n_colors=n_c)
for k_c, mu_k in enumerate(mu_b):
  plt.plot(x_b, r_b[:, k_c], '.', alpha=.4, c=mu_colors[k_c])
  plt.axvline(mu_k, c=mu_colors[k_c])
plt.title('E step: Responsibilities')
plt.xlabel('x (cm)')
plt.ylabel('p(z|x)');
```

We see that most of our data points fall between two clusters (for each $x$, there are at least two lines $p(z_k|x)$ significantly greater than 0).

Now, let's update the parameters ($\mathrm{M}$-step).

```
mu_b, sigma_2b, pi_b = m_step_gaussian_mixture(x_b, r_b)
x_ = np.linspace(min(x_b), max(x_b), 1000)
plt.hist(x_b, bins=200, density=True, histtype='stepfilled',
         facecolor='.9', edgecolor='.3')
for k_c, mu_k in enumerate(mu_b):
  y_ = bell_curve(x_, mu_b[k_c], sigma_2b[k_c]) * pi_b[k_c]
  plt.plot(x_, y_, c=mu_colors[k_c])
  plt.axvline(mu_k, c=mu_colors[k_c])
plt.title('M: recompute the means')
plt.ylabel('p(x)')
plt.xlabel('x (cm)');
```

Again, we can use transparency to create a static plot of the training progress:

```
plt.hist(x_b, bins=200, density=True, histtype='stepfilled',
           facecolor='.9', edgecolor='.3')
for k_ in range(n_steps):
  r_b = e_step_gaussian_mixture(x_b, mu_b, sigma_2b, pi)
  alpha = max((k_ / n_steps, .1))
  mu_b, sigma_2b, pi_b = m_step_gaussian_mixture(x_b, r_b)
  for k_c, mu_k in enumerate(mu_b):
    y_ = bell_curve(x_, mu_b[k_c], sigma_2b[k_c]) * pi_b[k_c]
    plt.plot(x_, y_, c=mu_colors[k_c], alpha=alpha)
    plt.axvline(mu_k, c=mu_colors[k_c], alpha=alpha)
```

Or code up a little movie clip using `matlplotlib.animation` library.

```
#@title Visualisation code for EM
def em_gaussian_mixture(x, n_clusters, n_steps=10):
  n_steps += 1 # Need one for the initial values.
  params_over_time = np.zeros((n_steps, 3, n_clusters))
  params_over_time[0][0] = np.random.randn(n_clusters) * np.std(x) + np.mean(x)
  params_over_time[0][1] = 50
  params_over_time[0][2] = 1 / n_clusters
  for t in range(1, n_steps):
    r = e_step_gaussian_mixture(x, *params_over_time[t - 1])
    params_over_time[t] = m_step_gaussian_mixture(x, r)
  return params_over_time

def animate_em_gaussian_mixture(x, params_over_time):
  n_steps, _, n_clusters = np.shape(params_over_time)
  colors = sns.color_palette(n_colors=n_clusters)

  x_ = np.linspace(min(x), max(x), 1000)
  fig, ax = plt.subplots()
  # Initialise graphics:
  pdfs = [plt.plot(x_, np.nan * x_, c=colors[k_c])[0]
          for k_c in range(n_clusters)]
  plt.hist(x, bins=200, density=True, histtype='stepfilled',
           facecolor='.9', edgecolor='.3')
  lines = [plt.axvline(np.nan, c=colors[k_c])
          for k_c in range(n_clusters)]

  def animate_(t):
    mu_t, sigma_2t, pi_t = params_over_time[t]
    for k_c, mu_k in enumerate(mu_t):
      lines[k_c].set_data([mu_k, [0, 1]])
      y_ = bell_curve(x_, mu_k, sigma_2t[k_c]) * pi_t[k_c]
      pdfs[k_c].set_data(x_, y_)

    if t == 0:
      s_title = 'Initialise clusters'
    else:
      s_title = '{}'.format(t)
    ax.set_title(s_title)

  my_anim = animation.FuncAnimation(fig, animate_, frames=n_steps,
                                interval=600)
  plt.close()
  return HTML(my_anim.to_jshtml())
```

```
params_over_time = em_gaussian_mixture(x_b, n_clusters=4, n_steps=20)
animate_em_gaussian_mixture(x_b, params_over_time)
```

**Task**: Commonly, ML researchers visualise the progress of training by plotting how the loss changes over time. Can you do it?

###**Extra reading: $\mathrm{EM}$ as a special case of variational-$\mathrm{EM}$**

So far, we used the cost function terminology. In this section, it will be easier to talk about the log-likelihood instead, $\log(P(D|\Theta))=-\mathcal{C(D|}\Theta)$, also referred to as the **evidence**.

Thus, we can say that $\mathrm{EM}$ maximises a variational evidence lower-bound,
$$\mathrm{ELBO}=\log(P(D|\Theta))-\mathrm{KL}[P(\mathcal{Z|D})||Q(\mathcal{Z})],$$
where $Q(\mathcal{Z})$ is some distribution and $\mathrm{KL}$ stands for the **Kullback-Leibler divergence**, which measures distances between distributions.

As with all distance measures, $\mathrm{KL}$ cannot be negative. Thus, $\mathrm{ELBO}$ is less or equal to the evidence.

To summarise:
1. $\mathrm{E}$-step maximises $\mathrm{ELBO}$ wrt $Q(\mathcal{Z})$.

  This means minimising $\mathrm{KL}[P(\mathcal{Z|D})||Q(\mathcal{Z})]$.

2. $\mathrm{M}$-step maximises $\mathrm{ELBO}$ assuming a fixed $Q(\mathcal{Z})$ distribution.

  By recalling definitions of the  $\mathrm{KL}$-divergence:
\begin{align}
\mathrm{KL}[P(\mathcal{Z|D})||Q(\mathcal{Z})]&=- \langle \log(P(\mathcal{Z|D}))\rangle_{Q(\mathcal{Z})} + \langle \log Q(\mathcal{Z})\rangle_{Q(\mathcal{Z})},
\end{align}
and of the joint distribution $P(\mathcal{Z,D})=P(\mathcal{Z|D})P(\mathcal{D})$, i.e.:
\begin{align}
\log(P(\mathcal{D})) = \log(P(\mathcal{Z,D})) - \log(P(\mathcal{Z|D}))
\end{align}
we get:
$$\mathrm{ELBO}= \langle \log(P(\mathcal{D,Z})|\Theta))\rangle_{Q(\mathcal{Z})} - \langle \log Q(\mathcal{Z})\rangle_{Q(\mathcal{Z})}.$$
  For a fixed $Q(\mathcal{Z})$, this means maximising
  $\langle \log(P(\mathcal{D,Z})|\Theta))\rangle_{Q(\mathcal{Z})}$.

Comparing to the $\mathrm{EM}$ algorithm introduced above, we note:
1. The minimum of $\mathrm{KL}$ is for $Q(\mathcal{Z}) = P(\mathcal{Z|D})$.
2. Maximising $\langle \log(P(\mathcal{Z|D})|\Theta))\rangle_{Q(\mathcal{Z})}$ is equivalent to minimising $\langle -\log(P(\mathcal{Z|D})|\Theta))\rangle_{Q(\mathcal{Z})}= \langle \mathcal{C(D, Z|}\Theta)\rangle_{Q(\mathcal{Z})}$. Thus, $\mathrm{M}$-step is identical in both algorithms, with $Q(\mathcal{Z})=P(\mathcal{Z|D})$.

The variational approach is more flexible than $\mathrm{EM}$ algorithm, as it doesn't require computing the true posterior $P(\mathcal{Z|D})$. While  $P(\mathcal{Z|D})$ is not computable for most practical application, it is always feasible to optimise parameters of its proxy, $Q(\mathcal{Z})$.

#**Latent code via Neural networks: Autoencoder.**

So far, we assumed the meaning for our latent variables ($z_n$ standing for the id of the cluster that generated $n$-th sample). In the next section, we talk about how the latent code can be used for compression. But before we move on to this task, let's talk about discovering the latent code using neural networks.

We are going to code up the classical **autoencoder**.

**Notation** We use bold symbols to represent vectors. For example,  $\mathbf{x}=(x_{1}, x_{2}, \ldots, x_i, \ldots, x_{D})$ represents a $D$-dimensional vector $\mathbf{x} \in\mathcal{X}^D$.

An autoencoder is composed of:
-  an **encoder**, which is a feedforward network computing a latent code $\mathbf{z(\mathbf{x})} \in\mathcal{Z}^K$ for input $\mathbf{x} \in\mathcal{X}^D$, and
- a **decoder**, which attempts to reconstruct the input $\mathbf{x}$ from the latent code, $\mathbf{\hat x}(\mathbf{z})\in\mathcal{X}^D$.

It is common to use the real space $\mathbb R$ for both $\mathcal{X}$ and $\mathcal{Z}$, but in the example below, we bound the activities between 0 and 1.

An autoencoder uses a square loss function as a measure of distance between the original $\mathbf{x}$ and its reconstruction $\mathbf{\hat x}$:
$$C(\mathbf{x}, \mathbf{\hat x}) = \frac 1 2 \sum_{i=1}^D (\hat x_{i} - x_i)^2$$

Indexing samples by $n$ and using $\mathbf{y}$, rather than $\mathbf{\hat x}$ as the network output, we get the full expression of the cost the neural network has to optimise:
$$C(\mathcal{D}) = \frac 1 2 \sum_{n=1}^N \sum_{i=1}^D (y_{i}(\mathbf{x}_{n}) - x_{ni})^2$$

**Task**: Do you recognise this loss? How can we interpret the autoencoder from the probabilistic modelling perspective?

Let us get our data. How many dimensions does each sample $\mathbf x_n$ have?

```
x_c = get_data_c()
x_c = np.reshape(x_c, (x_c.shape[0], -1, 3))
plt.imshow(x_c);
plt.grid(False);
plt.ylabel('Sample id');
```

If you answered $D=6$, then you got tricked! Colour is composed of RGB values, so the network needs to work with $D=6\times 3$ dimensions in this case. It is best to reshape our data to directly reflect that. This is how the network "sees" the data:

```
n_x = np.prod(np.shape(x_c)[1:])
x_c = np.reshape(x_c, (-1, n_x))
plt.imshow(x_c);
plt.grid(False)
plt.ylabel('Sample id')
```

##**Multi-layer perceptron (MLP)**

For the purpose of this exercise, we will construct one of the simplest autoencoders - a multi-layer perceptron (MLP) with a single hidden layer.

1. The input to our network is $D$-dimensional, $\mathbf{x}\in \mathbb{R}^D$.
2. Our first layer computes the latent code $\mathbf{z}\in \mathbb{R}^K_+$. In Machine Learning language, our hidden layer consists of K neurons, their activity is non-negative.

  A perceptron layer consist of:
  - a linear operation: the input is multiplied via a **weight matrix** $w^e\in \mathbb{R}^{K \times D}$, and a **bias vector**  $b^e\in\mathbb{R}^K$ is added,
$$z_k^{lin} = \sum_{j=1}^D w^e_{kj} x_j + b^e_k $$
  - followed by a nonlinear **activation function** $\sigma$ applied to every element of the resulting vector.
$$z_k = \sigma(z_k^{lin}) = \sigma(\sum_j w^e_{kj} x_j + b^e_k) $$
3. Our second layer is the output layer, which follows the same set of operations.
$$y_l = \sigma(\sum_k w^d_{lk} z_j + b^d_l) \equiv \sigma(y_l^{lin})$$

We choose the standard sigmoid function for our nonlinearity:
$$\sigma(x) = \frac{1}{1 + e^{-x}},$$ other popular choices are rectification (setting negative values to zero) or $\tanh$.

```
def sigmoid(x):
  return 1 / (1 + np.exp(-x))

def perceptron(x, w, b):
  return sigmoid(np.dot(w, x) + b)
```

```
n_hidden = 5
n_all_epochs = 0
w_e = np.random.randn(n_hidden, n_x) / np.sqrt(n_x)
w_d = np.random.randn(n_x, n_hidden) / np.sqrt(n_hidden)
b_e = np.zeros(n_hidden)
b_d = np.zeros(n_x)
```

```
def visualise_predictions(data):
  data = np.reshape(data, (-1, n_x))
  xhat = np.zeros_like(data)
  for n, x_ in enumerate(data):
    xhat[n] = perceptron(perceptron(x_, w_e, b_e), w_d, b_d)

  _, axes = plt.subplots(1, 2, figsize=(3, 4))
  for ax, x_, a_label in zip(axes, [data, xhat], ['Data', 'AE Output']):
    ax.imshow(x_.reshape(-1, int(n_x / 3), 3))
    ax.grid(False)
    ax.set_title(a_label)
  axes[0].set_ylabel('Sample id')
  axes[1].set_yticks([])
```

Let's visualise predictions in a randomly initialised network:

```
visualise_predictions(x_c)
```

##**Stochastic gradient descent ($\mathrm{SGD}$)**
We will now derive equations for learning.

According to the **gradient descent** approach, parameters of our network are modified in the direction opposite to the gradient of the cost with respect to these parameters. That is, for any parameter $\theta$, the update $\Delta \theta$ is computed as
$$\Delta \theta = - \lambda \frac{\partial \mathcal C(\mathcal D)}{\partial \theta}$$
where $\lambda$, the learning rate, is some small positive number and $\frac{\partial \mathcal C(\mathcal D)}{\partial \theta}$ is the gradient (the derivative) of the cost wrt $\theta$. In a single learning step, we add such updates to the current values of the parameters. For small enough updates, this guarantees that our loss decreases with every step.

In the famous **stochastic gradient descent ($\mathrm{SGD}$)** algorithm, instead of computing the full cost function $\mathcal C(\mathcal D)$, updates are computed and applied after a single input sample. That makes things a bit easier! However, it comes at a cost: We are losing the guarantee that every optimisation step will decrease the overall loss. The training curve might look a bit noisier.

###**Task**
Code up the square loss and its gradient for a single sample

```
def square_loss(x, y):
  return

def square_loss_gradient(x, y):
  return
```

```
#@title Solution
def square_loss(x, y):
  return (x - y)** 2 / 2

def square_loss_gradient(x, y):
  return x - y
```

If you coded up the square loss and its gradient correctly, you should get the following graph:

```
x_ = np.linspace(-5, 5, 100)
plt.plot(x_, square_loss(x_, 0), label=r'$L(x, 0)$')
plt.plot(x_, square_loss_gradient(x_, 0), label=r'$dL(x, 0)/dx$')
plt.xlabel('x')
plt.legend()
plt.title('Square loss and its gradient');
```

##**Chain rule for differentiation**

In order to compute the derivative $\frac{\partial \mathcal C(\mathbf x)}{\partial \theta}$, we need to apply the chain rule for differentation.

For a chain of functions in which $y()$ is applied to $z()$, which in turn is applied to $x$, $y(z(x))$, the gradient of $y$ with respect to $x$ can be expressed as:
$$\frac{d y(z(x))}{dx} = \frac{dy}{dz} \frac{dz}{dx}.$$

If $y$ is a function of more variables, we need to take all of them into account:
$$\frac{d y(z_1(x), z_2(x), \ldots, z_k(x)))}{dx} = \sum_k \frac{\partial y}{\partial z_k} \frac{dz_k}{dx}.$$
(The symbol $\partial$ is used whenever we compute a gradient of a multi-variable function.)

#####**Task**
Using the chain rule and the following basic derivatives:
$$\frac{d x^k}{dx} = kx^{k-1}\qquad \frac{d e^x}{dx}=e^x\qquad \frac{d (a+bx)}{dx} = b,$$ derive the gradient of $\sigma(x) = \frac{1}{1 + e^{-x}}$

```
def sigmoid_gradient(x):
  return
```

```
#@title **Solution**: `sigmoid_gradient`
def sigmoid_gradient(x):
  return sigmoid(x) * sigmoid(-x)
```

If you coded up the sigmoid correctly, our AE should be using the following activation function:

```
x_ = np.linspace(-10, 10, 100)
plt.plot(x_, sigmoid(x_), label=r'$\sigma(x)$')
plt.plot(x_, sigmoid_gradient(x_), label=r'$d\sigma(x)/dx$')
plt.xlabel('x')
plt.legend()
plt.title('Sigmoid activation function and its gradient');
```

##**$\mathrm{Backprop}$**
$\mathrm{Backprop}$ is nothing else but application of chain rule for differentation.

Let us start with the decoder:
\begin{align}
\frac{\partial C}{\partial b_j^d} &= \sum_k \frac{\partial C}{\partial y_k} \frac{\partial y_k}{b_j^d} = (y_j - x_j) \sigma'(y_j^{lin})\\
\frac{\partial C}{\partial w^d_{jl}} &=  \sum_k \frac{\partial C}{\partial y_k} \frac{\partial y_k}{\partial w^d_{jl}} = (y_j - x_j) \sigma'(y_j^{lin}) z_l
\end{align}
Notice, how the summation over $k$ yields only one non-zero component, because only $y_j$ depends on $b^d_j$ (or $w^d_{jl}$).

Let us now compute the gradients for the layer below, the encoding layer:
\begin{align}
\frac{\partial C}{\partial b^e_{j}} &=  \sum_k\frac{\partial C}{\partial y_k} \frac{\partial y_k}{\partial z_j} \frac{\partial z_j}{\partial b^e_{j}}
= \frac{\partial C}{\partial z_j^{lin}}\\
\frac{\partial C}{\partial w^e_{jl}} &=  \sum_k\frac{\partial C}{\partial y_k} \frac{\partial y_k}{\partial z_j} \frac{\partial z_j}{\partial w^e_{jl}}
= \frac{\partial C}{\partial z_j^{lin}}x_l
\end{align}
Notice how for neuron $j$, gradients over its parameters are expressed via gradients over its linear activity, $z_j^{lin}$. These can be easily computed by re-using computations from the layer above:
$$\frac{\partial C}{\partial z_j^{lin}}=\sum_k\frac{\partial C}{\partial y_k}\frac{\partial y_k}{\partial z_j}\sigma'(z^{lin})=\sum_k\frac{\partial C}{\partial y_k}w^d_{kj}\sigma'(z^{lin}_j).$$
(In our case, the layer above was the output layer, so we re-used $\frac{\partial C}{\partial y_k}$.)

That's all you need to know to understand backprop (and automated differentiation in JAX or TensorFlow). Oh, and also solution of the task above:
\begin{align}
\frac{d \sigma(z)}{dz} = \sigma(z)\sigma(-z)
\end{align}

##**Training**
Now that we have all the ingredients in place, we can train our network.

For the purpose of interepretability, we will not train the biases in this exercise.

```
learning_rate = 1e-2
n_epochs = 5_000
loss_c = np.zeros(n_epochs)
for epoch in range(n_epochs):
  loss_ = []
  for x_ in x_c:
    z = sigmoid(np.dot(w_e, x_) + b_e)
    y = sigmoid(np.dot(w_d, z) + b_d)
    dL_dylin =  (y - x_) * sigmoid_gradient(y)
    # Backprop uses gradient from the layer above:
    dL_dzlin = np.matmul(dL_dylin, w_d) * sigmoid_gradient(z)
    # Gradients for the weights are outer products
    dL_dw_d = dL_dylin[:, None] * z[None]
    dL_dw_e = dL_dzlin[:, None] * x_[None]
    # Learning:
    w_d -= learning_rate * dL_dw_d
    w_e -= learning_rate * dL_dw_e
    # For the purpose of analysis, we will not train the biases:
    # b_d -= learning_rate * dL_dylin
    # b_e -= learning_rate * dL_dzlin
    loss_.append(np.sum((x_ - y)**2 / 2))
  n_all_epochs += 1
  loss_c[epoch] = np.sum(loss_)

plt.plot(loss_c)
s_title = f'Training curve, last {n_epochs} epochs out of {n_all_epochs};'
plt.title(s_title + r' $\lambda$={}'.format(learning_rate))
plt.xlabel('Training steps')
plt.ylabel('Loss')

visualise_predictions(x_c)
```

####**Task**
Feel free to re-run the training cell, maybe change a learning rate? The loss is likely to keep decreasing forever (or at least until numerical instabilities kick in), as we are dealing with sigmoids $\sigma(x)$, which can obtain binary values only in the limit ($\lim_{x\rightarrow-\infty}\sigma(x)=0$, $\lim_{x\rightarrow\infty}\sigma(x)=1$). That would require infinite values for the weights in our network.

####**Extra Task: Linear encoder**

Using the code above, create a linear autoencoder. (You need to remove all non-linearities from the network and update the training accordingly.) The linear autoencoder is shown to be equivalent to the PCA (Principal Components Analysis). How well does it do encoding our 18 orthogonal samples?

##**Analysis**
We were able to train the network to encode our 18 orthogonal samples with 5-dimensional code. How did the network do that? Did the weights grow into inifity? Let's visualise them:

```
_, axes = plt.subplots(1, 2)
for ax, w_, l_ in zip(axes, [w_e, w_d.T], ['Encoder', 'Decoder']):
  sns.heatmap(w_, ax=ax, center=0, cmap='RdBu_r')
  ax.set_title(l_ + ' weights')
axes[0].set_ylabel('Latent code id')
```

Note, that for convenience, we plot the transpose of the decoder weights. The encoder appears to have more extreme values of the weights. Why?

It might be informative to pass the weights through the activation function used by the network. Each entry $\sigma(w_{jk})$ gives us insight into what the output of the neuron $j$ would be, if only its $k$-th input was active (and unitary). This is akin to how neuroscientists measure Receptive Fields (using spotlights to stimulate the retina and the down-stream neurons of the visual system).

```
_, axes = plt.subplots(1, 2)
for ax, w_, l_ in zip(axes, [w_e, w_d.T], ['Encoder', 'Decoder']):
  sns.heatmap(sigmoid(w_), vmin=0, vmax=1, ax=ax, center=.5, cmap='RdBu_r')
  ax.set_title(l_ + ' weights')
axes[0].set_ylabel('Latent code id')
```

This was informative! The encoder and decoder matrices start to look alike.

Now, if we were to go back to the RGB coding that is actually represented in our dataset:

```
w_e_rgb = sigmoid(w_e.reshape((-1, 6, 3), order='f' ))
w_d_rgb = sigmoid(w_d.T.reshape((-1, 6, 3), order='f'))
_, axes = plt.subplots(1, 2, figsize=(10, 3))
for ax, w_, l_ in zip(axes, [w_e_rgb, w_d_rgb], ['Encoder', 'Decoder']):
  ax.imshow(w_)
  ax.set_title(l_ + ' weights in RGB')
  ax.grid(False)
  ax.set_xticks([])
axes[0].set_ylabel('Latent code id');
```

Again, we can visually attest to the similarities between the weights encoding and decoding the latent code.

But hey, we can visualise the compressed code directly! It shows the feat we achieved with this encoder - representing 18 orthogonal samples with just 5 latent variables.

```
# Note: If you changed the code to the linear autoencoder,
# you need to update this line:
z = np.squeeze([perceptron(x_, w_e, b_e) for x_ in x_c])

_, axes = plt.subplots(1, 2, figsize=(5, 6))
ax = axes[0]
ax.imshow(x_c.reshape((-1, 6, 3)))
ax.grid(False)
ax.set_title('Input')
ax.set_ylabel('Sample id')
ax = axes[1]
sns.heatmap(z, ax=ax)
ax.set_title('Latent code');
```

#**Compression**
We talked a lot about using the latent code for compression. What do we mean by it?

Let's explain it on the simplest example considered [above](#scrollTo=JuzhThGFPJ_M&line=4&uniqifier=1), where the latent code (cluster ids) was deterministic, i.e. only one $z_n$ was assigned to each sample $x_n$.
- Now, we simply replace $x_n$ with its cluster id $z_n$.
- We also need to encode the clusters, but there is much fewer of them than samples $K\ll N$, which requires less memory.

How much memory do we save?

By default, `numpy` uses `float64` to encode numbers, i.e. your computer uses 64 bits per number. But to encode 4 clusters, we only need 2! The four possible clusters can be encoded by $00, 01, 10, 11$.  It's saving memory by a factor of $\mathbf{32}$!  (To be exact, it is $64N / (2N + 64K)$, which is almost $32$.)
And you get even more when your samples are high-dimenional. For $D$-dimenions, you can compress dataset taking $64ND$ bits of memory into $2N + 64KD$ bits. Saving such compressed data takes nearly $\mathbf{32D}$ less space!

We will now apply the compression to an image.

Our data is simply pixels. Thus, our samples are now vectors, rather than scalars ($\mathbf{x}\in\mathbb{R}^3$).

We use $\mathrm{K\!-\! means}$ to obtain $z_n$ of each of the $N$ pixels and cluster parameters $\mathbf{\mu}_k$ for each of the $K$ clusters.

```
from skimage import data
x = data.rocket()
# We normalise the data between (0, 1) for easier plotting
x = x / np.max(x)
plt.imshow(x); plt.axis('off')
x_shape = x.shape
# And we reshape it so that N = number of pixels.
N = np.prod(x_shape[:2])
x = np.reshape(x, (N, -1))
```

We need to adjust the `e_step` and `m_step` to be able to apply them to vectors. We use Euclidean distance to measure the similarity between samples and clusters.

```
from scipy.spatial import distance_matrix
def e_step(x, mu):
  d = distance_matrix(x, mu)
  return np.nanargmin(d, 1)

def m_step(x, r, cluster_ids):
  # We change the way we treat clusters that have no data assinged to them.
  # Rather than becoming irrelevant, they now represent a black pixel (0, 0, 0):
  mu_c = np.zeros((len(cluster_ids), x.shape[-1]))
  for k_c in cluster_ids:
    x_c = x[r == k_c]
    if np.any(x_c):
      mu_c[k_c] = np.mean(x_c, 0)
  return mu_c
```

Let's apply the $\mathrm{K\!-\!means}$ algorithm to our image, compute the responsibilities ($\mathrm{E}$-step) and cluster parameters ($\mathrm{M}$-step) and use both to compress our image!

Note: What we do to the image, replacing the original pixel values by a values from a discrete set of colours is also called **image segmentation**, or **vector quantisation**.

```
def show_mu(x, ax):
  # cluster parameters have shape (K, D), so if the last dim codes for RGB,
  # we need to reshape it into (1, K, 3)
  x = x[None] if x.shape[-1] == 3 else x.T
  ax.imshow(x)

n_c = 7
n_steps = 5
n_rgb = x.shape[-1]
cluster_ids = range(n_c)

mu_t = np.random.uniform(size=(n_c, n_rgb))
_, axes = plt.subplots(1, 2, figsize=(15, 2.7))
show_mu(mu_t, axes[0])
axes[0].set_title(r'Initial draw of $\mu$')
axes[1].imshow(x.reshape(x_shape))
axes[1].set_title(r'Original image')
[ax.axis(False) for ax in axes]

r_t = e_step(x, mu_t)
for t in range(n_steps):
  fig, axes = plt.subplots(1, 3, figsize=(15, 2.7), gridspec_kw={'top':.9})
  fig.suptitle(f'Step {t + 1}')
  axes[0].imshow(r_t.reshape(x_shape[:2]), cmap='Set1')
  axes[0].set_title(f'E step: $z_n$')
  mu_t = m_step(x, r_t, cluster_ids)
  show_mu(mu_t, axes[1])
  axes[1].set_title(f'M step: $\mu_k$')
  axes[2].imshow(mu_t[r_t].reshape(x_shape))
  axes[2].set_title(f'Compressed image')
  [ax.axis(False) for ax in axes]
  plt.show()
  r_t = e_step(x, mu_t)

```

####**Task**

Try out a different number of clusters, you can also let the algorithm run for longer. You could also try a different image from the `skimage.data` library, or maybe even upload your own photo?

```

```


## Elementify any strings

```python
elementList = [
    'H', 'D', 'T', 'He', 'Li', 'Be', 'B', 'C', 'N', 'O', 'F', 'Ne', 'Na', 'Mg', 'Al', 'Si', 'P',
    'S', 'Cl', 'Ar', 'K', 'Ca', 'Sc', 'Ti', 'V', 'Cr', 'Mn', 'Fe', 'Co', 'Ni', 'Cu', 'Zn',
    'Ga', 'Ge', 'As', 'Se', 'Br', 'Kr', 'Rb', 'Sr', 'Y', 'Zr', 'Nb', 'Mo', 'Tc', 'Ru',
    'Rh', 'Pd', 'Ag', 'Cd', 'In', 'Sn', 'Sb', 'Te', 'I', 'Xe', 'Cs', 'Ba', 'La', 'Ce',
    'Pr', 'Nd', 'Pm', 'Sm', 'Eu', 'Gd', 'Tb', 'Dy', 'Ho', 'Er', 'Tm', 'Yb', 'Lu', 'Hf',
    'Ta', 'W', 'Re', 'Os', 'Ir', 'Pt', 'Au', 'Hg', 'Tl', 'Pb', 'Bi', 'Po', 'At', 'Rn',
    'Fr', 'Ra', 'Ac', 'Th', 'Pa', 'U', 'Np', 'Pu', 'Am', 'Cm', 'Bk', 'Cf', 'Es', 'Fm',
    'Md', 'No', 'Lr', 'Rf', 'Db', 'Sg', 'Bh', 'Hs', 'Mt', 'Ds', 'Rg', 'Cn', 'Nh',
    'Fl', 'Mc', 'Lv', 'Ts', 'Og']
elementList = [x.upper() for x in elementList]
elementSet = set(elementList)

```

```python
import numpy as np
import itertools
# from sympy.utilities.iterables import multiset_permutations
import matplotlib.pyplot as plt
import matplotlib.patches as patches

def permutationWithoutRepeat12(inList):
    # Currently only supports 2 kinds, 1 and 2
    len2 = len([x for x in inList if x == 2])
    lenList = len(inList)
    combination2 = list(itertools.combinations(range(lenList), len2))
    combinations = []
    for comb in combination2:
        baseArray = [1] * lenList
        for ii in comb:
            baseArray[ii] = 2
        combinations.append(baseArray)
    return combinations

def zerosAndOnes(num):
    combinations = []
    for ii in range(2**num):
        combinations.append([ii//(2**(x)) % 2 for x in range(num)])
    return combinations

def getSepWords(inString, sep):
    currIdx = 0
    sepWords = []
    for ss in sep:
        sepWords.append(inString[currIdx:currIdx+ss])
        currIdx += ss
    return sepWords

def Fibonacci(n):
    F = np.ones(n,dtype=int)
    for ii in range(2, len(F)):
        F[ii] = F[ii-1] + F[ii-2]
    return F[-1]

def elementify(inString,verbose=True,printing=False,resolution=None):
    elementWordSet = []

    if len(inString) == 0:
        return None
    if len(inString) > 120:
        if verbose:
            print("Procssing string with length > 25 is very slow, so skip!")
        return None
    inString = inString.upper()
    if 'J' in inString or 'Q' in inString:
        if verbose:
            print("Can't have J and Q in the string - there's no symbol with these characters")
        return None
    # First check if there are irrepairable ones
    workAlone = []
    workWithPrev = [False]
    workWithNext = []
    for idx, ii in enumerate(inString):
        workAlone.append(ii in elementSet)
        if len(inString) == 1:
            workWithNext.append(False)
        if idx < len(inString)-1:
            workWithNext.append(inString[idx:idx+2] in elementSet)
        if idx > 0:
            workWithPrev.append(inString[idx-1:idx+1] in elementSet)
        if not workWithPrev[-1] and not workWithNext[-1] and not workAlone[-1]:
            if verbose:
                print(f'{inString} cannot be elementified. Ruled out by linear filtering')
            return None
    workWithNext.append(False)

    # Using YJF's method
    OKarray = np.array([workAlone, workWithNext],dtype=int)
    OKarraySum = OKarray.sum(0)
    whereTwoIsPossible = np.where(OKarraySum == 2)[0]
    twos = len(whereTwoIsPossible)
    twoCombinations = 2 ** twos

    twoCombinationsSequence = zerosAndOnes(twos)
    exhaustiveCombinations = Fibonacci(len(inString)+1)

    if twoCombinations < exhaustiveCombinations:
        for seq in twoCombinationsSequence:
            idxCounter = 0
            seqCounter = 0
            validDivision = False
            idxSeq = [0]
            while idxCounter <= len(inString) - 1:
                if OKarraySum[idxCounter] == 0:
                    break
                elif OKarraySum[idxCounter] == 1:
                    if OKarray[0, idxCounter] == 1:
                        idxCounter += 1
                        idxSeq.append(idxCounter)
                    elif OKarray[1, idxCounter] == 1:
                        idxCounter += 2
                        idxSeq.append(idxCounter)
                elif OKarraySum[idxCounter] == 2:
                    idxCounter += OKarray[seq[seqCounter], idxCounter]+seq[seqCounter]
                    seqCounter += 1
                    idxSeq.append(idxCounter)
                if idxCounter == len(inString):
                    validDivision = True
            if validDivision:
                sepWords = [inString[idxSeq[ii]:idxSeq[ii+1]] for ii in range(len(idxSeq)-1)]
                if sepWords not in elementWordSet:
                    elementWordSet.append(sepWords)
            else:
                pass
    else:
        numSep = len(inString)
        for ii in range(numSep//2+1):
            sepArray = [2]*ii + [1] * (numSep - 2 * ii)
            sepSet = permutationWithoutRepeat12(sepArray)
            for sep in sepSet:
                isGoodSep = True
                sepWords = getSepWords(inString, sep)
                for word in sepWords:
                    if word not in elementSet:
                        isGoodSep = False
                        break
                if isGoodSep:
                    elementWordSet.append(sepWords)

    if verbose:
        if len(elementWordSet) > 0:
            print(f'{inString} can be elementified in {len(elementWordSet)} ways as')
            for ii in elementWordSet:
                print('  '.join(ii))
        else:
            print(f'{inString} cannot be elementified.')
            return None

    if printing:
        for word in elementWordSet:
            printWord(word,resolution=resolution)

    return elementWordSet

def printWord(inWordList, resolution=None):
    # Use matplotlib to print element symbols
    fig, ax = plt.subplots(figsize=(2*len(inWordList), 2), dpi=resolution)
    fig.tight_layout(rect=[0, 0.00, 1, 1])
    ax.set_xlim([0, len(inWordList)])
    ax.set_ylim([0,1])

    ax.tick_params(
    axis='both',          # changes apply to the x-axis
    which='both',      # both major and minor ticks are affected
    bottom=False,      # ticks along the bottom edge are off
    top=False,         # ticks along the top edge are off
    left=False,
    right=False,
    labelbottom=False,
    labelleft=False)
    ax.set_box_aspect(1/len(inWordList))
    for ii in range(len(inWordList)):
        rect = patches.Rectangle((ii,0), 1, 1, linewidth=1, edgecolor='k', facecolor='w')
        ax.add_patch(rect)
        plt.text(ii+0.5,0.46,inWordList[ii].capitalize(),fontdict={'family':'Serif'},fontsize=50,
                 horizontalalignment='center',verticalalignment='center')

```

```python
printWord(['Ha','Ku','Na','','Ma','Ta','Ta'])
```

```python
elementify('SATURN',printing=True)
```

```python
elementify('Phagocytosis',printing=True)
```

```python
elementify('BANANAS',printing=True,resolution=100)
```

```python
elementify('shower',printing=True,resolution=100)
```

```python
elementify('MOOSE',printing=True)
```

```python
elementify('Montana',printing=True)
```

```python
elementify('HOHOHO',printing=True)
```

```python

```