# pAPRika tutorial 6 - APR/Gromacs with Plumed restraints

In this tutorial, we will perform APR calculations for the butane (BUT)--cucurbit[6]uril (CB6) host-guest system. This tutorial is similar to [tutorial 5](05-tutorial-cb6-but-plumed.ipynb) where `Plumed` will be used for the restraints but we will be using the `GROMACS` MD engine instead. One other difference in this tutorial is that we will be performing the simulation with periodic boundary condition (PBC) and explicit water molecules. This is because `GROMACS` does not support implicit solvent calculations.

## Initialize

### Before you start

We will run the simulations in this tutorial using `GROMACS` and `Plumed`. `GROMACS` will need to be installed before you can run the [simulation](#simulate) part of this tutorial. `Plumed`, however, should be installed in your `conda` environment if you installed *pAPRika* though the `conda` route. To use `Plumed` we first need to make sure the `PLUMED_KERNEL` environment variable is loaded (the library is called `libplumedKernel.so`). *pAPRika* should load the `Plumed` kernel automatically but let's make sure it is loaded and run the cell below.

```python
import os
'PLUMED_KERNEL' in os.environ.keys()
```

    True

If it does not exists we will load the environment in this Jupyter Notebook. Since `Plumed` is installed through `conda` the kernel will be located in your conda environment library folder. If you are running this on a MacOS, replace the kernel library in the cell below to `libplumedKernel.dylib`. If you compiled `Plumed` yourself and then you will need to change the path below.

```python
os.environ['PLUMED_KERNEL'] = f"{os.environ['CONDA_PREFIX']}/lib/libplumedKernel.so"
```

For running `Plumed` outside of this notebook it might be better to export the `PLUMED_KERNEL` variable into your `.bashrc` file.

**Note:** we can run `Plumed` with `GROMACS` *versions 2020.2* but the user is required to patch the source code. Unlike `AMBER` and `LAMMPS`, `GROMACS` does not come with a patched code out-of-the-box and the `conda-forge` repository also do not contain a patched version of `GROMACS`. See the `Plumed` documentation for more details on how to patch and recompile the `GROMACS` code https://www.plumed.org/doc-v2.6/user-doc/html/index.html.

**Note:** We will be running the CB6-BUT host-guest system in explicit solvent because `GROMACS` does not support implicit solvent calculation. Thus, we recommend running the simulation on a GPU.

🔵 Since we have a prepared the host-guest-dummy setup from the first tutorial, we will skip the initial tleap steps and go right into initializing the restraints. If you haven't completed [tutorial 1](01-tutorial-cb6-but.ipynb) please go back and run the notebook to generate the starting structure.

### Define names
We will store the files created in this tutorial in a folder called `gromacs-plumed` so we don't mix files with the previous tutorial.

```python
base_name = "cb6-but-dum"
work_dir = "gromacs-plumed"
complex_dir = "complex"
data = "../../../paprika/data/cb6-but"
```

## Configure APR Restraints

### Define anchor atoms
See [tutorial 1](01-tutorial-cb6-but.ipynb) for the choice of selection

```python
# Guest atoms
G1 = ":BUT@C"
G2 = ":BUT@C3"

# Host atoms
H1 = ":CB6@C"
H2 = ":CB6@C31"
H3 = ":CB6@C18"

# Dummy atoms
D1 = ":DM1"
D2 = ":DM2"
D3 = ":DM3"
```

### Determine the number of windows
Before we add the restraints, it is helpful to set the $\lambda$ fractions that control the strength of the force constants during attach and release, and to define the distances for the pulling phase.

The attach fractions go from 0 to 1 and we place more points at the bottom of the range to sample the curvature of $dU/d \lambda$. Next, we generally apply a distance restraint until the guest is ~18 Angstroms away from the host, in increments of 0.4 Angstroms. This distance should be at least twice the Lennard-Jones cutoff in the system. These values have worked well for us, but this is one aspect that should be carefully checked for new systems.

```python
import numpy as np
```

```python
attach_string = "0.00 0.40 0.80 1.60 2.40 4.00 5.50 8.65 11.80 18.10 24.40 37.00 49.60 74.80 100.00"
attach_fractions = [float(i) / 100 for i in attach_string.split()]

initial_distance = 6.0
pull_distances = np.arange(0.0 + initial_distance, 18.0 + initial_distance, 1.0)

release_fractions = []

windows = [len(attach_fractions), len(pull_distances), len(release_fractions)]
print(f"There are {windows} windows in this attach-pull-release calculation.")
```

    There are [15, 18, 0] windows in this attach-pull-release calculation.

### Load structure

```python
import parmed as pmd
```

* Load complex structure

```python
structure = pmd.load_file(
    os.path.join(complex_dir, f"{base_name}.prmtop"),
    os.path.join(complex_dir, f"{base_name}.rst7"),
    structure = True,
)
```

### Host Static Restraints
See [tutorial 1](01-tutorial-cb6-but.ipynb#host_static) for an explanation of the static restraints.

```python
import paprika.restraints as restraints
```

```python
static_restraints = []
```

```python
r = restraints.static_DAT_restraint(restraint_mask_list = [D1, H1],
                                    num_window_list = windows,
                                    ref_structure = structure,
                                    force_constant = 5.0,
                                    amber_index=True)

static_restraints.append(r)
```

```python
r = restraints.static_DAT_restraint(restraint_mask_list = [D2, D1, H1],
                                    num_window_list = windows,
                                    ref_structure = structure,
                                    force_constant = 100.0,
                                    amber_index=True)

static_restraints.append(r)
```

```python
r = restraints.static_DAT_restraint(restraint_mask_list = [D3, D2, D1, H1],
                                    num_window_list = windows,
                                    ref_structure = structure,
                                    force_constant = 100.0,
                                    amber_index=True)

static_restraints.append(r)
```

```python
r = restraints.static_DAT_restraint(restraint_mask_list = [D1, H1, H2],
                                    num_window_list = windows,
                                    ref_structure = structure,
                                    force_constant = 100.0,
                                    amber_index=True)

static_restraints.append(r)
```

```python
r = restraints.static_DAT_restraint(restraint_mask_list = [D2, D1, H1, H2],
                                    num_window_list = windows,
                                    ref_structure = structure,
                                    force_constant = 100.0,
                                    amber_index=True)

static_restraints.append(r)
```

```python
r = restraints.static_DAT_restraint(restraint_mask_list = [D1, H1, H2, H3],
                                    num_window_list = windows,
                                    ref_structure = structure,
                                    force_constant = 100.0,
                                    amber_index=True)

static_restraints.append(r)
```

### Guest translational and rotational restraints
See [tutorial 1](01-tutorial-cb6-but.ipynb#guest) for an explanation of the guest restraints

```python
guest_restraints = []
```

```python
r = restraints.DAT_restraint()
r.mask1 = D1
r.mask2 = G1
r.topology = structure
r.auto_apr = True
r.continuous_apr = True
r.amber_index = True

r.attach["target"] = pull_distances[0]          # Angstroms
r.attach["fraction_list"] = attach_fractions
r.attach["fc_final"] = 5.0                      # kcal/mol/Angstroms**2

r.pull["target_final"] = 24.0                   # Angstroms
r.pull["num_windows"] = windows[1]

r.initialize()
guest_restraints.append(r)
```

```python
r = restraints.DAT_restraint()
r.mask1 = D2
r.mask2 = D1
r.mask3 = G1
r.topology = structure
r.auto_apr = True
r.continuous_apr = True
r.amber_index = True

r.attach["target"] = 180.0                      # Degrees
r.attach["fraction_list"] = attach_fractions
r.attach["fc_final"] = 100.0                    # kcal/mol/radian**2

r.pull["target_final"] = 180.0                  # Degrees
r.pull["num_windows"] = windows[1]

r.initialize()
guest_restraints.append(r)
```

```python
r = restraints.DAT_restraint()
r.mask1 = D1
r.mask2 = G1
r.mask3 = G2
r.topology = structure
r.auto_apr = True
r.continuous_apr = True
r.amber_index = True

r.attach["target"] = 180.0                      # Degrees
r.attach["fraction_list"] = attach_fractions
r.attach["fc_final"] = 100.0                    # kcal/mol/radian**2

r.pull["target_final"] = 180.0                  # Degrees
r.pull["num_windows"] = windows[1]

r.initialize()
guest_restraints.append(r)
```

### Create APR windows
We use the guest restraints to create a list of windows with the appropriate names and then create the directories.

```python
from paprika.restraints.restraints import create_window_list
```

```python
window_list = create_window_list(guest_restraints)
```

```python
if not os.path.isdir(work_dir):
    os.makedirs(work_dir)

for window in window_list:
    folder = os.path.join(work_dir, window)
    if not os.path.isdir(folder):
        os.makedirs(os.path.join(work_dir, window))
```

### Write APR restraints to Plumed format
In this section we create an instance of `Plumed()` from `paprika.restraints.plumed`, which is a class to generate the `Plumed` restraint files. We need to specify the list of restraints used throughout the APR calculations and the corresponding windows list. In this tutorial we will print the **host static** restraints and the **guest** restraints. The `Plumed` class includes a method to add restraints to dummy atoms (`add_dummy_atoms_to_file`). We will add the positional restraints on the dummy atoms after solvation of the host-guest complex.

**Note**: be careful when specifying the force constants in `DAT_restraints`. We follow the AMBER (and CHARMM) convention where the force constant is already multiplied by a factor of 1/2 but `Plumed` requires the user to specify the force constant without this factor, i.e.

$$
\begin{align}
U_{amber} &= K_{amber} (r-r_{0})^2 \\
U_{plumed} &= \frac{1}{2} k_{plumed} (r - r_{0})^2
\end{align}
$$

thus $k_{plumed} = 2 \times  K_{amber}$. If AMBER force constants was used in generating the `DAT_restraints` (the case in this tutorial) we need to set the variable `uses_legacy_k` to `True` (this is on by default).

```python
from paprika.restraints.plumed import Plumed
```

```python
restraints_list = (static_restraints + guest_restraints)

plumed = Plumed()
plumed.file_name = 'plumed.dat'
plumed.path = work_dir
plumed.window_list = window_list
plumed.restraint_list = restraints_list
plumed.uses_legacy_k = True

plumed.dump_to_file()
```

## Prepare host-guest system

### Translate guest molecule
For the attach windows, we will use the initial, bound coordinates for the host-guest complex. Only the force constants change during this phase, so a single set of coordinates is sufficient. For the pull windows, we will translate the guest to the target value of the restraint before solvation, and for the release windows, we will use the coordinates from the final pull window.

```python
import shutil
```

```python
for window in window_list:
    if window[0] == "a":
        shutil.copy(os.path.join(complex_dir, f"{base_name}.prmtop"),
                    os.path.join(work_dir, window, f"{base_name}.prmtop"))
        shutil.copy(os.path.join(complex_dir, f"{base_name}.rst7"),
                    os.path.join(work_dir, window, f"{base_name}.rst7"))

    elif window[0] == "p":
        structure = pmd.load_file(
            os.path.join(complex_dir, f"{base_name}.prmtop"),
            os.path.join(complex_dir, f"{base_name}.rst7"),
            structure = True
        )
        target_difference = guest_restraints[0].phase['pull']['targets'][int(window[1:])] -\
                            guest_restraints[0].pull['target_initial']
        print(f"In window {window} we will translate the guest {target_difference.magnitude:0.1f}.")

        for atom in structure.atoms:
            if atom.residue.name == "BUT":
                atom.xz += target_difference.magnitude

        structure.save(os.path.join(work_dir, window, f"{base_name}.prmtop"), overwrite=True)
        structure.save(os.path.join(work_dir, window, f"{base_name}.rst7"), overwrite=True)
```

    In window p000 we will translate the guest 0.0 Angstroms.
    In window p001 we will translate the guest 1.1 Angstroms.
    In window p002 we will translate the guest 2.1 Angstroms.
    In window p003 we will translate the guest 3.2 Angstroms.
    In window p004 we will translate the guest 4.2 Angstroms.
    In window p005 we will translate the guest 5.3 Angstroms.
    In window p006 we will translate the guest 6.4 Angstroms.
    In window p007 we will translate the guest 7.4 Angstroms.
    In window p008 we will translate the guest 8.5 Angstroms.
    In window p009 we will translate the guest 9.5 Angstroms.
    In window p010 we will translate the guest 10.6 Angstroms.
    In window p011 we will translate the guest 11.6 Angstroms.
    In window p012 we will translate the guest 12.7 Angstroms.
    In window p013 we will translate the guest 13.8 Angstroms.
    In window p014 we will translate the guest 14.8 Angstroms.
    In window p015 we will translate the guest 15.9 Angstroms.
    In window p016 we will translate the guest 16.9 Angstroms.
    In window p017 we will translate the guest 18.0 Angstroms.

### Solvate the host-guest system
Here we will solvate the host-guest system in each window with 2000 water molecules. Since `Gromacs` cannot read in `Amber` topology and coordinates files we will have to convert them to the appropriate formats. `TLeap` has a built-in function called `TLeap.convert_to_gromacs()`, which will convert the `Amber` files to `Gromacs`. We also need to add positional restraints on dummy atoms to the `plumed.dat` file and this is done by invoking the function `plumed.add_dummy_atoms_to_file`

```python
from paprika.build.system import TLeap
```

```python
for window in window_list:
    print(f"Solvating system in {window}.")

    structure = pmd.load_file(
        os.path.join(work_dir, window, f"{base_name}.prmtop"),
        os.path.join(work_dir, window, f"{base_name}.rst7"),
        structure=True,
    )

    if not os.path.exists(os.path.join(work_dir, window, f"{base_name}.pdb")):
        structure.save(os.path.join(work_dir, window, f"{base_name}.pdb"))

    system = TLeap()
    system.output_path = os.path.join(work_dir, window)
    system.output_prefix = f"{base_name}-sol"

    system.target_waters = 2000
    system.neutralize = True
    system.add_ions = ["Na+", 6, "Cl-", 6]
    system.pbc_type = "rectangular"
    system.template_lines = [
        "source leaprc.gaff",
        "source leaprc.water.tip3p",
        f"loadamberparams {data}/cb6.frcmod",
        "loadamberparams ../../complex/dummy.frcmod",
        f"CB6 = loadmol2 {data}/cb6.mol2",
        f"BUT = loadmol2 {data}/but.mol2",
        "DM1 = loadmol2 ../../complex/dm1.mol2",
        "DM2 = loadmol2 ../../complex/dm2.mol2",
        "DM3 = loadmol2 ../../complex/dm3.mol2",
        "model = loadpdb cb6-but-dum.pdb",
    ]
    system.build()

    # Convert Amber to Gromacs
    system.convert_to_gromacs(overwrite=True)

    # Add dummy atom restraints - load in the Gromacs files instead
    # of the amber files because the coordinates a shifted.
    structure = pmd.load_file(
        os.path.join(work_dir, window, f"{base_name}-sol.top"),
        xyz=os.path.join(work_dir, window, f"{base_name}-sol.gro"),
        structure=True
    )
    plumed.add_dummy_atoms_to_file(structure,window=window)
```

    Solvating system in a000.
    Solvating system in a001.
    Solvating system in a002.
    Solvating system in a003.
    Solvating system in a004.
    Solvating system in a005.
    Solvating system in a006.
    Solvating system in a007.
    Solvating system in a008.
    Solvating system in a009.
    Solvating system in a010.
    Solvating system in a011.
    Solvating system in a012.
    Solvating system in a013.
    Solvating system in p000.
    Solvating system in p001.
    Solvating system in p002.
    Solvating system in p003.
    Solvating system in p004.
    Solvating system in p005.
    Solvating system in p006.
    Solvating system in p007.
    Solvating system in p008.
    Solvating system in p009.
    Solvating system in p010.
    Solvating system in p011.
    Solvating system in p012.
    Solvating system in p013.
    Solvating system in p014.
    Solvating system in p015.
    Solvating system in p016.
    Solvating system in p017.

## Simulation

Now that we have a solvated structure and a `plumed.dat` file with dummy atoms position restraints, we have everything ready to go. **pAPRika** has a `GROMACS` wrapper module that can help setting default parameters for the simulation. There are some high level options that we set directly, like `simulation.path`, and then we call the function `config_pbc_min()` to setup reasonable default simulation parameters for a minimization and production run. See the `pAPRika` documentation for more details.

`GROMACS`, unlike `AMBER` and `OpenMM`, have a higher dependency on CPUs, so more than one processor for every GPU is preferred to deliver better speed. Setting the number of processors requires using different keyword depending on how `GROMACS` was compiled. The cells below were run using `GROMACS` that is compiled with MPI and hence the suffix, i.e., `gmx_mpi` (for a non-MPI version, the executable is usually called `gmx`). We set the number of processors with `simulation.n_threads = 8` and the GPU device with `simulation.gpu_devices = 0`.

**Note**: For host-guest systems I find that the non-MPI version is considerably much faster. Also, 12 CPUs with a single GPU seems to be a good combination with a non-MPI version.

By default, `pAPRika` will use `-ntomp` to specify the number of CPUs when executing `mdrun` if the executable is `gmx_mpi` and `-nt` for `gmx`. If these default settings do not work with the `GROMACS` currently installed then we can use `simulation.custom_mdrun_command` to override the default, e.g.,
```python
simulation.custom_mdrun_command = "-ntmpi 8 -gpu_id 1"
```
or
```python
simulation.custom_mdrun_command = "mpirun -np 4 gmx_mpi mdrun -ntomp 8 -gpu_id 4"
```

**Note**: We do not need to run `grompp` beforehand because the simulation wrapper in `pAPRika` takes care of this automatically when we call the `simulation.run()` method. We also set `simulation.grompp_maxwarn = 5` to ignore warnings about the total charge of the system (the total charge is slightly off from neutral).

**Note**: as explained at the [start](#start) of this tutorial, make sure that the `PLUMED_KERNEL` environment variable is set otherwise the simulation will fail to run.

```python
from paprika.simulate import GROMACS
```

Initialize logger

```python
import logging
from importlib import reload
reload(logging)

logger = logging.getLogger()
logging.basicConfig(
    format='%(asctime)s %(message)s',
    datefmt='%Y-%m-%d %I:%M:%S %p',
    level=logging.INFO
)
```

### Energy Minimization

```python
for window in window_list:
    simulation = GROMACS()
    simulation.executable = "gmx_mpi"
    simulation.n_threads = 8
    simulation.gpu_devices = 0
#     simulation.custom_mdrun_command = "-ntomp 8 -gpu_id 6"
    simulation.path = os.path.join(work_dir, window)
    simulation.maxwarn = 5

    simulation.prefix = "minimize"
    simulation.topology = f"{base_name}-sol.top"
    simulation.coordinates = f"{base_name}-sol.gro"
    simulation.plumed_file = "plumed.dat"

    simulation.config_pbc_min()

    logging.info(f"Running minimization in window {window}...")
    simulation.run(overwrite=True)
```

    2020-09-25 05:22:17 PM Running minimization in window a000...
    2020-09-25 05:22:36 PM Running minimization in window a001...
    2020-09-25 05:22:56 PM Running minimization in window a002...
    2020-09-25 05:23:15 PM Running minimization in window a003...
    2020-09-25 05:23:37 PM Running minimization in window a004...
    2020-09-25 05:23:56 PM Running minimization in window a005...
    2020-09-25 05:24:16 PM Running minimization in window a006...
    2020-09-25 05:24:33 PM Running minimization in window a007...
    2020-09-25 05:24:52 PM Running minimization in window a008...
    2020-09-25 05:25:12 PM Running minimization in window a009...
    2020-09-25 05:25:32 PM Running minimization in window a010...
    2020-09-25 05:25:50 PM Running minimization in window a011...
    2020-09-25 05:26:10 PM Running minimization in window a012...
    2020-09-25 05:26:29 PM Running minimization in window a013...
    2020-09-25 05:26:49 PM Running minimization in window p000...
    2020-09-25 05:27:08 PM Running minimization in window p001...
    2020-09-25 05:27:28 PM Running minimization in window p002...
    2020-09-25 05:27:46 PM Running minimization in window p003...
    2020-09-25 05:28:05 PM Running minimization in window p004...
    2020-09-25 05:28:24 PM Running minimization in window p005...
    2020-09-25 05:28:46 PM Running minimization in window p006...
    2020-09-25 05:29:04 PM Running minimization in window p007...
    2020-09-25 05:29:23 PM Running minimization in window p008...
    2020-09-25 05:29:41 PM Running minimization in window p009...
    2020-09-25 05:30:01 PM Running minimization in window p010...
    2020-09-25 05:30:20 PM Running minimization in window p011...
    2020-09-25 05:30:39 PM Running minimization in window p012...
    2020-09-25 05:30:58 PM Running minimization in window p013...
    2020-09-25 05:31:18 PM Running minimization in window p014...
    2020-09-25 05:31:37 PM Running minimization in window p015...
    2020-09-25 05:31:56 PM Running minimization in window p016...
    2020-09-25 05:32:15 PM Running minimization in window p017...

### NVT - Equilibration
Here, we will perform 50,000 steps of equilibration with the NVT ensemble.

```python
for window in window_list:
    simulation = GROMACS()
    simulation.executable = "gmx_mpi"
    simulation.n_threads = 8
    simulation.gpu_devices = 0
#     simulation.custom_mdrun_command = "-ntomp 8 -gpu_id 6"
    simulation.path = os.path.join(work_dir, window)
    simulation.maxwarn = 5

    simulation.prefix = "equilibration"
    simulation.topology = f"{base_name}-sol.top"
    simulation.coordinates = "minimize.gro"
    simulation.plumed_file = "plumed.dat"

    simulation.config_pbc_md(ensemble='nvt')
    simulation.control["nsteps"] = 50000

    logging.info(f"Running equilibration in window {window}...")
    simulation.run(overwrite=True)
```

    2020-09-25 05:41:42 PM Running equilibration in window a000...
    2020-09-25 05:42:26 PM Running equilibration in window a001...
    2020-09-25 05:43:07 PM Running equilibration in window a002...
    2020-09-25 05:43:49 PM Running equilibration in window a003...
    2020-09-25 05:44:33 PM Running equilibration in window a004...
    2020-09-25 05:45:17 PM Running equilibration in window a005...
    2020-09-25 05:46:01 PM Running equilibration in window a006...
    2020-09-25 05:46:36 PM Running equilibration in window a007...
    2020-09-25 05:47:13 PM Running equilibration in window a008...
    2020-09-25 05:47:54 PM Running equilibration in window a009...
    2020-09-25 05:48:36 PM Running equilibration in window a010...
    2020-09-25 05:49:12 PM Running equilibration in window a011...
    2020-09-25 05:49:56 PM Running equilibration in window a012...
    2020-09-25 05:50:31 PM Running equilibration in window a013...
    2020-09-25 05:51:11 PM Running equilibration in window p000...
    2020-09-25 05:51:53 PM Running equilibration in window p001...
    2020-09-25 05:52:36 PM Running equilibration in window p002...
    2020-09-25 05:53:20 PM Running equilibration in window p003...
    2020-09-25 05:54:02 PM Running equilibration in window p004...
    2020-09-25 05:54:44 PM Running equilibration in window p005...
    2020-09-25 05:55:26 PM Running equilibration in window p006...
    2020-09-25 05:56:09 PM Running equilibration in window p007...
    2020-09-25 05:56:44 PM Running equilibration in window p008...
    2020-09-25 05:57:27 PM Running equilibration in window p009...
    2020-09-25 05:58:05 PM Running equilibration in window p010...
    2020-09-25 05:58:48 PM Running equilibration in window p011...
    2020-09-25 05:59:31 PM Running equilibration in window p012...
    2020-09-25 06:00:14 PM Running equilibration in window p013...
    2020-09-25 06:00:59 PM Running equilibration in window p014...
    2020-09-25 06:01:41 PM Running equilibration in window p015...
    2020-09-25 06:02:20 PM Running equilibration in window p016...
    2020-09-25 06:03:02 PM Running equilibration in window p017...

### Production Run
We will run the production phase for 50,000 steps. **Note**: a proper production run would require simulation in the tens of nanoseconds per window.

```python
for window in window_list:
    simulation = GROMACS()
    simulation.executable = "gmx_mpi"
    simulation.n_threads = 8
    simulation.gpu_devices = 0
#     simulation.custom_mdrun_command = "-ntomp 8 -gpu_id 6"
    simulation.path =  os.path.join(work_dir, window)
    simulation.maxwarn = 5

    simulation.prefix = "production"
    simulation.topology = f"{base_name}-sol.top"
    simulation.coordinates = "equilibration.gro"
    simulation.checkpoint = "equilibration.cpt"
    simulation.plumed_file = "plumed.dat"

    simulation.config_pbc_md(ensemble='npt')
    simulation.control["nsteps"] = 50000

    logging.info(f"Running production in window {window}...")
    simulation.run(overwrite=True)
```

    2020-09-25 06:04:07 PM Running production in window a000...
    2020-09-25 06:04:49 PM Running production in window a001...
    2020-09-25 06:05:33 PM Running production in window a002...
    2020-09-25 06:06:17 PM Running production in window a003...
    2020-09-25 06:07:02 PM Running production in window a004...
    2020-09-25 06:07:45 PM Running production in window a005...
    2020-09-25 06:08:30 PM Running production in window a006...
    2020-09-25 06:09:14 PM Running production in window a007...
    2020-09-25 06:09:56 PM Running production in window a008...
    2020-09-25 06:10:37 PM Running production in window a009...
    2020-09-25 06:11:22 PM Running production in window a010...
    2020-09-25 06:12:04 PM Running production in window a011...
    2020-09-25 06:12:49 PM Running production in window a012...
    2020-09-25 06:13:39 PM Running production in window a013...
    2020-09-25 06:14:23 PM Running production in window p000...
    2020-09-25 06:15:07 PM Running production in window p001...
    2020-09-25 06:15:50 PM Running production in window p002...
    2020-09-25 06:16:35 PM Running production in window p003...
    2020-09-25 06:17:20 PM Running production in window p004...
    2020-09-25 06:18:03 PM Running production in window p005...
    2020-09-25 06:18:55 PM Running production in window p006...
    2020-09-25 06:19:40 PM Running production in window p007...
    2020-09-25 06:20:24 PM Running production in window p008...
    2020-09-25 06:21:13 PM Running production in window p009...
    2020-09-25 06:21:57 PM Running production in window p010...
    2020-09-25 06:22:43 PM Running production in window p011...
    2020-09-25 06:23:28 PM Running production in window p012...
    2020-09-25 06:24:16 PM Running production in window p013...
    2020-09-25 06:24:58 PM Running production in window p014...
    2020-09-25 06:25:43 PM Running production in window p015...
    2020-09-25 06:26:24 PM Running production in window p016...
    2020-09-25 06:27:06 PM Running production in window p017...

## Analysis

Once the simulation is completed, we can using the `Analysis` module to determine the binding free energy. We supply the location of the parameter information, a string or list for the file names (wildcards supported), the location of the windows, and the restraints on the guest.

In this example, we use the method `ti-block` which determines the free energy using **t**hermodynamic **i**integration and then estimates the standard error of the mean at each data point using blocking analysis. Bootstrapping it used to determine the uncertainty of the  full thermodynamic integral for each phase.

After running `compute_free_energy()`, a dictionary called `results` will be populated, that contains the free energy and SEM for each phase of the simulation.

```python
import paprika.analysis as analysis
```

```python
free_energy = analysis.fe_calc()
free_energy.topology = "cb6-but-dum-sol.pdb"
free_energy.trajectory = 'production*.trr'
free_energy.path = work_dir
free_energy.restraint_list = guest_restraints
free_energy.collect_data()
free_energy.methods = ['ti-block']
free_energy.ti_matrix = "full"
free_energy.bootcycles = 1000
free_energy.compute_free_energy()
```

We also need to calculate the free-energy cost of releasing the restraints on the guest molecule.

```python
free_energy.compute_ref_state_work([
    guest_restraints[0], guest_restraints[1], None, None,
    guest_restraints[2], None
])
```

Then we add the free-energies together and combine the uncertainties to get the binding-free energy

```python
binding_affinity = -1 * (
    free_energy.results["attach"]["ti-block"]["fe"] + \
    free_energy.results["pull"]["ti-block"]["fe"] + \
    free_energy.results["ref_state_work"]
)

sem = np.sqrt(
    free_energy.results["attach"]["ti-block"]["sem"]**2 + \
    free_energy.results["pull"]["ti-block"]["sem"]**2
)
```

```python
print(f"The binding affinity of butane to cucurbit[6]uril = {binding_affinity.magnitude:0.2f} +/- {sem.magnitude:0.2f} kcal/mol")
```

    The binding affinity of butane to cucurbit[6]uril = -7.55 +/- 1.58 kcal/mol


```python
%matplotlib inline
```

# Kuka Reinforcement Learning (DQN)

This notebook shows how to train a Deep Q Learning (DQN) agent
on the pyBullet Kuka environment based on:
https://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html

**Task**
The agent has to decide between seven (2+2+2+1) actions - moving the manipulator
in $x$ (2), or $y$ (2) directions by vertical angle offset (2) or not moving at all (1) - so that the manipulator can grasp an object. We assume that the velocity for each directions equal and there is a "height hack" where the gripper automatically moves down for each action.

As the agent observes the current state of the environment and chooses
an action, the environment *transitions* to a new state, and also
returns a reward that indicates the consequences of the action. In this
task, the reward is 1 if one of the objects is above height .2 at the end
of the episode.

The pyBullet Kuka is designed so that the inputs to the agent are (48, 48, 3) images
representing the environment state.
Neural networks can solve the task purely by looking at the
scene, so we'll use stack of consecutive screens as an input. In this way, we are hoping to capture the dynamics of the environment.

**Packages**

First, let's import needed packages. Firstly, we need gym
(Install using `pip install gym`) and pyBullet (Install using `pip install pybullet`.
We'll also use the following from PyTorch:

-  neural networks (``torch.nn``)
-  optimization (``torch.optim``)
-  automatic differentiation (``torch.autograd``)
-  utilities for vision tasks (``torchvision`` - `a separate
   package <https://github.com/pytorch/vision>`__).

Also for display the average reward and agent progress we are going to use tensorboarX (`pip install tensorboardX`.) To run tensorboard web server, you need to install it using `pip install tensorboard`.

```python
!pip install pybullet
!pip install tensorboardX
import gym
import math
import random
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
from collections import namedtuple
import collections
from itertools import count
import timeit
from datetime import timedelta
from PIL import Image
from tensorboardX import SummaryWriter

import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
import torchvision.transforms as T

from pybullet_envs.bullet.kuka_diverse_object_gym_env import KukaDiverseObjectEnv
from gym import spaces
import pybullet as p

env = KukaDiverseObjectEnv(renders=False, isDiscrete=True, removeHeightHack=False, maxSteps=20)
env.cid = p.connect(p.DIRECT)

# set up matplotlib
is_ipython = 'inline' in matplotlib.get_backend()
if is_ipython:
    from IPython import display

plt.ion()

# if gpu is to be used
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
```

    Requirement already satisfied: pybullet in /home/monster/anaconda3/envs/python36/lib/python3.6/site-packages (2.6.1)
    Requirement already satisfied: tensorboardX in /home/monster/anaconda3/envs/python36/lib/python3.6/site-packages (2.0)
    Requirement already satisfied: six in /home/monster/anaconda3/envs/python36/lib/python3.6/site-packages (from tensorboardX) (1.13.0)
    Requirement already satisfied: numpy in /home/monster/anaconda3/envs/python36/lib/python3.6/site-packages (from tensorboardX) (1.17.4)
    Requirement already satisfied: protobuf>=3.8.0 in /home/monster/anaconda3/envs/python36/lib/python3.6/site-packages (from tensorboardX) (3.11.2)
    Requirement already satisfied: setuptools in /home/monster/anaconda3/envs/python36/lib/python3.6/site-packages (from protobuf>=3.8.0->tensorboardX) (42.0.2.post20191203)
    current_dir=/home/monster/anaconda3/envs/python36/lib/python3.6/site-packages/pybullet_envs/bullet

# Replay Memory

We'll be using experience replay memory for training our DQN. It stores
the transitions that the agent observes, allowing us to reuse this data
later. By sampling from it randomly, the transitions that build up a
batch are decorrelated. It has been shown that this greatly stabilizes
and improves the DQN training procedure.

For this, we're going to need two classses:

-  ``Transition`` - a named tuple representing a single transition in
   our environment. It essentially maps (state, action) pairs
   to their (next_state, reward) result, with the state being the
   consecutive stacked screens image as described later on.
-  ``ReplayMemory`` - a cyclic buffer of bounded size that holds the
   transitions observed recently. It also implements a ``.sample()``
   method for selecting a random batch of transitions for training.

```python
Transition = namedtuple('Transition',
                        ('state', 'action', 'next_state', 'reward'))

class ReplayMemory(object):

    def __init__(self, capacity):
        self.capacity = capacity
        self.memory = []
        self.position = 0

    def push(self, *args):
        """Saves a transition."""
        if len(self.memory) < self.capacity:
            self.memory.append(None)
        self.memory[self.position] = Transition(*args)
        self.position = (self.position + 1) % self.capacity

    def sample(self, batch_size):
        return random.sample(self.memory, batch_size)

    def __len__(self):
        return len(self.memory)
```

Now, let's define our model. But first, let's quickly recap what a DQN is.

# DQN algorithm

Our aim will be to train a policy that tries to maximize the discounted,
cumulative reward
$R_{t_0} = \sum_{t=t_0}^{\infty} \gamma^{t - t_0} r_t$, where
$R_{t_0}$ is also known as the *return*. The discount,
$\gamma$, should be a constant between $0$ and $1$
that ensures the sum converges. It makes rewards from the uncertain far
future less important for our agent than the ones in the near future
that it can be fairly confident about.

The main idea behind Q-learning is that if we had a function
$Q^*: State \times Action \rightarrow \mathbb{R}$, that could tell
us what our return would be, if we were to take an action in a given
state, then we could easily construct a policy that maximizes our
rewards:

\begin{align}\pi^*(s) = \arg\!\max_a \ Q^*(s, a)\end{align}

However, we don't know everything about the world, so we don't have
access to $Q^*$. But, since neural networks are universal function
approximators, we can simply create one and train it to resemble
$Q^*$.

For our training update rule, we'll use a fact that every $Q$
function for some policy obeys the Bellman equation:

\begin{align}Q^{\pi}(s, a) = r + \gamma Q^{\pi}(s', \pi(s'))\end{align}

The difference between the two sides of the equality is known as the
temporal difference error, $\delta$:

\begin{align}\delta = Q(s, a) - (r + \gamma \max_a Q(s', a))\end{align}

To minimise this error, we will use the `Huber
loss <https://en.wikipedia.org/wiki/Huber_loss>`__. The Huber loss acts
like the mean squared error when the error is small, but like the mean
absolute error when the error is large - this makes it more robust to
outliers when the estimates of $Q$ are very noisy. We calculate
this over a batch of transitions, $B$, sampled from the replay
memory:

\begin{align}\mathcal{L} = \frac{1}{|B|}\sum_{(s, a, s', r) \ \in \ B} \mathcal{L}(\delta)\end{align}

\begin{align}\text{where} \quad \mathcal{L}(\delta) = \begin{cases}
     \frac{1}{2}{\delta^2}  & \text{for } |\delta| \le 1, \\
     |\delta| - \frac{1}{2} & \text{otherwise.}
   \end{cases}\end{align}

## Q-network

Our model will be a convolutional neural network that takes in the
stacked screen patches. It has seven
outputs, representing $Q(s, d_x = d_v)$, $Q(s, d_x = -d_v)$, $Q(s, d_y = d_v)$, $Q(s, d_y = -d_v)$, $Q(s, d_a=0.25)$,
and $Q(s, d_a=-0.25)$ (where $d_v=0.06$ is the velocity in PyBullet, $d_a$ is vertical angle offset for the gripper, and $s$ is the input to the
network). This discretization of action is based on the continuous action which is a 5-vector parameterizing XYZ offset, vertical angle offset (radians), and grasp angle (radians). The continuous action (5-vector) is discretized as follows with "height hack" where the gripper automatically moves down for each action, and grasp angle in PyBullet is set to 0.3:
```
        dx = [0, -dv, dv, 0, 0, 0, 0][action]
        dy = [0, 0, 0, -dv, dv, 0, 0][action]
        dz = -dv
        da = [0, 0, 0, 0, 0, -0.25, 0.25][action]

        action_continuous = [dx, dy, dz, da, 0.3]
```
In effect, the network is trying to predict the *expected return* of
taking each action given the current input. In order to capture the dynamics of the environment, we stack 5 consecutive screens of the environment after each step together and then feed them to the network.

```python
STACK_SIZE = 5
```

```python
class DQN(nn.Module):
    def __init__(self, h, w, outputs):
        super(DQN, self).__init__()
        self.conv1 = nn.Conv2d(STACK_SIZE, 32, kernel_size=8, stride=4)
        self.bn1 = nn.BatchNorm2d(32)
        self.conv2 = nn.Conv2d(32, 64, kernel_size=4, stride=2)
        self.bn2 = nn.BatchNorm2d(64)
        self.conv3 = nn.Conv2d(64, 64, kernel_size=3, stride=1)

        # Number of Linear input connections depends on output of conv2d layers
        # and therefore the input image size, so compute it.
        def conv2d_size_out(size, kernel_size = 5, stride = 2):
            return (size - (kernel_size - 1) - 1) // stride  + 1
        convw = conv2d_size_out(conv2d_size_out(conv2d_size_out(w,8,4),4,2),3,1)
        convh = conv2d_size_out(conv2d_size_out(conv2d_size_out(h,8,4),4,2),3,1)
        linear_input_size = convw * convh * 64
        self.linear = nn.Linear(linear_input_size, 512)
        self.head = nn.Linear(512, outputs)

    # Called with either one element to determine next action, or a batch
    # during optimization. Returns tensor([[left0exp,right0exp]...]).
    def forward(self, x):
        x = F.relu(self.bn1(self.conv1(x)))
        x = F.relu(self.bn2(self.conv2(x)))
        x = F.relu(self.conv3(x))
        x = F.relu(self.linear(x.view(x.size(0), -1)))
        return self.head(x)
```

## Input extraction

The code below are utilities for extracting and processing rendered
images from the environment. It uses the ``torchvision`` package, which
makes it easy to compose image transforms. Once you run the cell it will
display an example patch that it extracted.

```python
preprocess = T.Compose([T.ToPILImage(),
                    T.Grayscale(num_output_channels=1),
                    T.Resize(40, interpolation=Image.CUBIC),
                    T.ToTensor()])

def get_screen():
    global stacked_screens
    # Returned screen requested by gym is 400x600x3, but is sometimes larger
    # such as 800x1200x3. Transpose it into torch order (CHW).
    screen = env._get_observation().transpose((2, 0, 1))
    # Convert to float, rescale, convert to torch tensor
    # (this doesn't require a copy)

    screen = np.ascontiguousarray(screen, dtype=np.float32) / 255
    screen = torch.from_numpy(screen)
    # Resize, and add a batch dimension (BCHW)
    return preprocess(screen).unsqueeze(0).to(device)

env.reset()
plt.figure()
plt.imshow(get_screen().cpu().squeeze(0)[-1].numpy(),cmap='Greys',
           interpolation='none')
plt.title('Example extracted screen')
plt.show()
```

# Training

## Hyperparameters and utility function

This cell instantiates our model and its optimizer, and defines the
utility function:

-  ``select_action`` - will select an action accordingly to an epsilon
   greedy policy. Simply put, we'll sometimes use our model for choosing
   the action, and sometimes we'll just sample one uniformly. The
   probability of choosing a random action will start at ``EPS_START``
   and will decay exponentially towards ``EPS_END``. ``EPS_DECAY``
   controls the rate of the decay.

```python
BATCH_SIZE = 32
GAMMA = 0.99
EPS_START = 0.9
EPS_END = 0.1
EPS_DECAY = 200
EPS_DECAY_LAST_FRAME = 10**4
TARGET_UPDATE = 1000
LEARNING_RATE = 1e-4

# Get screen size so that we can initialize layers correctly based on shape
# returned from pybullet (48, 48, 3).
init_screen = get_screen()
_, _, screen_height, screen_width = init_screen.shape

# Get number of actions from gym action space
n_actions = env.action_space.n

policy_net = DQN(screen_height, screen_width, n_actions).to(device)
target_net = DQN(screen_height, screen_width, n_actions).to(device)
target_net.load_state_dict(policy_net.state_dict())
target_net.eval()

optimizer = optim.Adam(policy_net.parameters(), lr=LEARNING_RATE)
memory = ReplayMemory(10000)

eps_threshold = 0

def select_action(state, i_episode):
    global steps_done
    global eps_threshold
    sample = random.random()
    eps_threshold = max(EPS_END, EPS_START - i_episode / EPS_DECAY_LAST_FRAME)
    if sample > eps_threshold:
        with torch.no_grad():
            # t.max(1) will return largest column value of each row.
            # second column on max result is index of where max element was
            # found, so we pick action with the larger expected reward.
            return policy_net(state).max(1)[1].view(1, 1)
    else:
        return torch.tensor([[random.randrange(n_actions)]], device=device, dtype=torch.long)

```

# Training loop

Finally, the code for training our model.

Here, you can find an ``optimize_model`` function that performs a
single step of the optimization. It first samples a batch, concatenates
all the tensors into a single one, computes $Q(s_t, a_t)$ and
$V(s_{t+1}) = \max_a Q(s_{t+1}, a)$, and combines them into our
loss. By defition we set $V(s) = 0$ if $s$ is a terminal
state. We also use a target network to compute $V(s_{t+1})$ for
added stability. The target network has its weights kept frozen most of
the time, but is updated with the policy network's weights every so often.
This is usually a set number of steps but we shall use episodes for
simplicity.

```python
def optimize_model():
    if len(memory) < BATCH_SIZE:
        return
    transitions = memory.sample(BATCH_SIZE)
    # Transpose the batch (see https://stackoverflow.com/a/19343/3343043 for
    # detailed explanation). This converts batch-array of Transitions
    # to Transition of batch-arrays.
    batch = Transition(*zip(*transitions))

    # Compute a mask of non-final states and concatenate the batch elements
    # (a final state would've been the one after which simulation ended)
    non_final_mask = torch.tensor(tuple(map(lambda s: s is not None,
                                          batch.next_state)), device=device, dtype=torch.bool)
    non_final_next_states = torch.cat([s for s in batch.next_state
                                                if s is not None])
    state_batch = torch.cat(batch.state)
    action_batch = torch.cat(batch.action)
    reward_batch = torch.cat(batch.reward)

    # Compute Q(s_t, a) - the model computes Q(s_t), then we select the
    # columns of actions taken. These are the actions which would've been taken
    # for each batch state according to policy_net
    state_action_values = policy_net(state_batch).gather(1, action_batch)

    # Compute V(s_{t+1}) for all next states.
    # Expected values of actions for non_final_next_states are computed based
    # on the "older" target_net; selecting their best reward with max(1)[0].
    # This is merged based on the mask, such that we'll have either the expected
    # state value or 0 in case the state was final.
    next_state_values = torch.zeros(BATCH_SIZE, device=device)
    next_state_values[non_final_mask] = target_net(non_final_next_states).max(1)[0].detach()
    # Compute the expected Q values
    expected_state_action_values = (next_state_values * GAMMA) + reward_batch

    # Compute Huber loss
    loss = F.smooth_l1_loss(state_action_values, expected_state_action_values.unsqueeze(1))

    # Optimize the model
    optimizer.zero_grad()
    loss.backward()
    for param in policy_net.parameters():
        param.grad.data.clamp_(-1, 1)
    optimizer.step()
```

Below, you can find the main training loop. At the beginning we reset
the environment and initialize the ``state`` Tensor. Then, we sample
an action, execute it, observe the next screen and the reward (always
1), and optimize our model once. When the episode ends (our model
fails), we restart the loop.

```python
PATH = 'policy_dqn.pt'
```

```python
num_episodes = 10000000
writer = SummaryWriter()
total_rewards = []
ten_rewards = 0
best_mean_reward = None
start_time = timeit.default_timer()
for i_episode in range(num_episodes):
    # Initialize the environment and state
    env.reset()
    state = get_screen()
    stacked_states = collections.deque(STACK_SIZE*[state],maxlen=STACK_SIZE)
    for t in count():
        stacked_states_t =  torch.cat(tuple(stacked_states),dim=1)
        # Select and perform an action
        action = select_action(stacked_states_t, i_episode)
        _, reward, done, _ = env.step(action.item())
        reward = torch.tensor([reward], device=device)

        # Observe new state
        next_state = get_screen()
        if not done:
            next_stacked_states = stacked_states
            next_stacked_states.append(next_state)
            next_stacked_states_t =  torch.cat(tuple(next_stacked_states),dim=1)
        else:
            next_stacked_states_t = None

        # Store the transition in memory
        memory.push(stacked_states_t, action, next_stacked_states_t, reward)

        # Move to the next state
        stacked_states = next_stacked_states

        # Perform one step of the optimization (on the target network)
        optimize_model()
        if done:
            reward = reward.cpu().numpy().item()
            ten_rewards += reward
            total_rewards.append(reward)
            mean_reward = np.mean(total_rewards[-100:])*100
            writer.add_scalar("epsilon", eps_threshold, i_episode)
            if (best_mean_reward is None or best_mean_reward < mean_reward) and i_episode > 100:
                # For saving the model and possibly resuming training
                torch.save({
                        'policy_net_state_dict': policy_net.state_dict(),
                        'target_net_state_dict': target_net.state_dict(),
                        'optimizer_policy_net_state_dict': optimizer.state_dict()
                        }, PATH)
                if best_mean_reward is not None:
                    print("Best mean reward updated %.1f -> %.1f, model saved" % (best_mean_reward, mean_reward))
                best_mean_reward = mean_reward
            break

    if i_episode%10 == 0:
            writer.add_scalar('ten episodes average rewards', ten_rewards/10.0, i_episode)
            ten_rewards = 0

    # Update the target network, copying all weights and biases in DQN
    if i_episode % TARGET_UPDATE == 0:
        target_net.load_state_dict(policy_net.state_dict())
    if i_episode>=200 and mean_reward>50:
        print('Environment solved in {:d} episodes!\tAverage Score: {:.2f}'.format(i_episode+1, mean_reward))
        break

print('Average Score: {:.2f}'.format(mean_reward))
elapsed = timeit.default_timer() - start_time
print("Elapsed time: {}".format(timedelta(seconds=elapsed)))
writer.close()
env.close()
```

    Best mean reward updated 36.0 -> 37.0, model saved
    Best mean reward updated 37.0 -> 38.0, model saved
    Best mean reward updated 38.0 -> 39.0, model saved
    Best mean reward updated 39.0 -> 40.0, model saved
    Best mean reward updated 40.0 -> 41.0, model saved
    Best mean reward updated 41.0 -> 42.0, model saved
    Best mean reward updated 42.0 -> 43.0, model saved
    Best mean reward updated 43.0 -> 44.0, model saved
    Best mean reward updated 44.0 -> 45.0, model saved
    Best mean reward updated 45.0 -> 46.0, model saved
    Best mean reward updated 46.0 -> 47.0, model saved
    Best mean reward updated 47.0 -> 48.0, model saved
    Best mean reward updated 48.0 -> 49.0, model saved
    Best mean reward updated 49.0 -> 50.0, model saved
    Best mean reward updated 50.0 -> 51.0, model saved
    Environment solved in 7412 episodes!	Average Score: 51.00
    Average Score: 51.00
    Elapsed time: 3:11:17.312433

Actions are chosen either randomly or based on a policy, getting the next
step sample from the gym environment. We record the results in the
replay memory and also run optimization step on every iteration.
Optimization picks a random batch from the replay memory to do training of the
new policy. "Older" target_net is also used in optimization to compute the
expected Q values; it is updated occasionally to keep it current.

## Evaluation

```python
episode = 10
scores_window = collections.deque(maxlen=100)  # last 100 scores
env = KukaDiverseObjectEnv(renders=False, isDiscrete=True, removeHeightHack=False, maxSteps=20, isTest=True)
env.cid = p.connect(p.DIRECT)
# load the model
checkpoint = torch.load(PATH)
policy_net.load_state_dict(checkpoint['policy_net_state_dict'])

# evaluate the model
for i_episode in range(episode):
    env.reset()
    state = get_screen()
    stacked_states = collections.deque(STACK_SIZE*[state],maxlen=STACK_SIZE)
    for t in count():
        stacked_states_t =  torch.cat(tuple(stacked_states),dim=1)
        # Select and perform an action
        action = policy_net(stacked_states_t).max(1)[1].view(1, 1)
        _, reward, done, _ = env.step(action.item())
        # Observe new state
        next_state = get_screen()
        stacked_states.append(next_state)
        if done:
            break
    print("Episode: {0:d}, reward: {1}".format(i_episode+1, reward), end="\n")
```

    Episode: 1, reward: 1
    Episode: 2, reward: 0
    Episode: 3, reward: 0
    Episode: 4, reward: 1
    Episode: 5, reward: 0
    Episode: 6, reward: 1
    Episode: 7, reward: 0
    Episode: 8, reward: 1
    Episode: 9, reward: 0
    Episode: 10, reward: 1


## Chapter 8

```python
import sys
sys.path.append("..")
from importlib import reload
```

```python
import Ch6.utilities as ch6_utils
reload(ch6_utils)
```

    <module 'Ch6.utilities' from '../Ch6/utilities.py'>

```python
import Ch7.utilities as ch7_utils
reload(ch7_utils)
```

    <module 'Ch7.utilities' from '../Ch7/utilities.py'>

```python
import Ch8.utilities as ch8_utils
reload(ch8_utils)
```

    <module 'Ch8.utilities' from '../Ch8/utilities.py'>

```python
import numpy as np
from sympy import *
init_printing(use_latex='mathjax')
import control
import matplotlib.pyplot as plt
```

### Problem 8.1

It is given that

\begin{eqnarray}
\Phi_c (s) \triangleq [ \Phi^{-1}(s) + BG ]^{-1}.
\end{eqnarray}

Do algebraic manipulations to get rid of the $\Phi^{-1}$ on the RHS:

\begin{eqnarray}
[ \Phi^{-1}(s) + BG ] \Phi_c (s) &=& I \\
\Phi(s) [ \Phi^{-1}(s) + BG ] \Phi_c(s) &=& \Phi(s) \\
[ I + \Phi(s) BG] \Phi_c(s) &=& \Phi(s) \\
\Phi_c(s) &=& [ I + \Phi(s) BG]^{-1} \Phi(s).
\end{eqnarray}

Now, it remains to show that:

\begin{eqnarray}
\Phi(s) - \Phi(s) B [ \Phi(s) + G \Phi(s) B]^{-1} G \Phi(s) &=& [ I + \Phi(s) BG]^{-1} \Phi(s) \\
\implies I - \Phi(s) B [ \Phi(s) + G \Phi(s) B]^{-1} G &=& [ I + \Phi(s) BG]^{-1}
\end{eqnarray}

Using the hint about block-diagram manipulations, it can be shown that $\Phi(s) B [ \Phi(s) + G \Phi(s) B]^{-1} = [ \Phi(s) + \Phi(s) B G]^{-1} \Phi(s) B$:

Now, plug in the above to the previous relation:
\begin{eqnarray}
I - [\Phi(s) + \Phi(s) B G]^{-1} \Phi(s) B G &=& [ I + \Phi(s) BG]^{-1}
\end{eqnarray}

Left-multiply both sides by $I + \Phi(s) BG$:
\begin{eqnarray}
(I + \Phi(s) BG) \bigl ( I - [\Phi(s) + \Phi(s) B G]^{-1} \Phi(s) B G \bigr ) &=& I \\
(I + \Phi(s) BG) - \Phi(s) B G &=& I \\
I &=& I
\end{eqnarray}

So, through algebraic reductions, a tautology is reached.  Therefore, it is true that

\begin{eqnarray}
\Phi_c (s) \triangleq [ \Phi^{-1}(s) + BG ]^{-1} = \Phi(s) - \Phi(s) B [ \Phi(s) + G \Phi(s) B]^{-1} G \Phi(s)
\end{eqnarray}

### Problem 8.2

Transform the system of equations from $\mathbf{\hat x}$ to $\mathbf{\hat w}$, where $\mathbf{\hat x} = P \mathbf{\hat w}_1 + Q \mathbf{\hat w}_2$, and $\mathbf{w}_1 = C \mathbf{x}$ and $\mathbf{w}_2 = M \mathbf{x}$.

The feedback law $u = -G \mathbf{\hat x}$ becomes:
\begin{eqnarray}
u &=& -G (P \mathbf{\hat w}_1 + Q \mathbf{\hat w}_2) \\
&=& -G P \mathbf{\hat w}_1 - G Q \mathbf{\hat w}_2 \\
&\triangleq& -\bar G_1 \mathbf{\hat w}_1 -  \bar G_2 \mathbf{\hat w}_2.
\end{eqnarray}

The dynamics of the closed-loop transformed system is:
\begin{eqnarray}
\mathbf{\dot w} &=& \bar A \mathbf{w} + \bar B u \\
&=& \bar A \mathbf{w} - \bar B ( \bar G_1 \mathbf{\hat w}_1 + \bar G_2 \mathbf{\hat w}_2 )
\end{eqnarray}

Now, follow the steps leading up to Eqn. 8.27:  $\mathbf{e} = \mathbf{w} - \mathbf{\hat w} \implies \mathbf{\hat w} = \mathbf{w} - \mathbf{e}$:

\begin{eqnarray}
\mathbf{\dot w} &=& \bar A \mathbf{w} - \bar B \bigl [ \bar G_1 (\mathbf{w}_1 - \mathbf{e}_1) + \bar G_2 (\mathbf{w}_2 - \mathbf{e}_2) \bigr ] \\
&=& (\bar A - \bar B \bar G) \mathbf{w} + \bar B( \bar G_1 \mathbf{e}_1 + \bar G_2 \mathbf{e_2}).
\end{eqnarray}

The remainder of the derivation for the transformed system follows _exactly_ the same steps in deriving Eqn. 8.27:

\begin{eqnarray}
\mathbf{w}(s) &=& (sI - \mathbf{\bar A}_c)^{-1} \bar B \bar G_2 \mathbf{e}_2(s) + (sI - \mathbf{\bar A}_c)^{-1} \mathbf{w}(0) \\
\mathbf{e}_2 &=& (sI - \bar F)^{-1} \mathbf{e}_2(0) \\
\implies \mathbf{w}(s) &=& (sI - \mathbf{\bar A}_c)^{-1} \bar B \bar G_2 (sI - \bar F)^{-1} \mathbf{e}_2(0) + (sI - \mathbf{\bar A}_c)^{-1} \mathbf{w}(0) \\
&=& \frac{\text{adj}(sI - \mathbf{\bar A}_c) \bar B \bar G_2 \text{adj}(sI - \bar F)}{|sI - \mathbf{\bar A}_c| |sI - \bar F|} \mathbf{e}_2(0) + (sI - \mathbf{\bar A}_c)^{-1} \mathbf{w}(0)
\end{eqnarray}

The characteristic equation is then $|sI - \mathbf{\bar A}_c||sI - \bar F| = 0$, with

\begin{eqnarray}
\bar A &=& \begin{pmatrix}
CAP & CAQ \\
MAP & MAQ
\end{pmatrix}\\
\bar B &=& \begin{pmatrix}
CB \\
MB
\end{pmatrix} \\
\bar F &=& (M - LC)AQ \\
\mathbf{\bar A}_c &=& \bar A - \bar B \bar G,
\end{eqnarray}

with $\bar G \triangleq [\bar G_1 | \bar G_2]$.

### Problem 8.3

#### a)
The reduced order observer is defined by the matrices $F, \bar {\bar G}, H$:
\begin{eqnarray}
F &\triangleq& A_{uu} - L C_a A_{au} = -\alpha - L \times 1 \times 1 = -\alpha - L \\
\bar {\bar G} &\triangleq& (A_{ua} - L C_a A_{aa}) C_a^{-1} = 0 - L \times 1 \times 0 \times 1 = 0 \\
H &\triangleq& B_u - L \times C_a \times B_a = \beta - L \times 1 \times 0 = \beta
\end{eqnarray}

The observability matrix for the available subsystem, $C_a$, is just unity ($=1$).

#### b)
The gains from example 8B are:
\begin{eqnarray}
G_1 &=& \frac{\bar a_2}{\beta} \\
G_2 &=& \frac{\bar a_1 - \alpha}{\beta}
\end{eqnarray}

The feedback law using these gains is:
\begin{eqnarray}
u &=& -G_1 \hat \theta - G_2 \hat \omega \\
&=& -G_1 y - G_2 (Ly + z),
\end{eqnarray}

The second equality above follows from the fact that $\theta$ is the measurement from which the reduced order observer for $\hat \omega$ is built.

Now, it remains to find the transfer function that takes measurement $y(s)$ to $z(s)$.  Plugging in the control found above into the equation for $z$, $\dot z = F z + \bar {\bar G} y + Hu$, leads to the closed-loop system:
\begin{eqnarray}
\dot z &=& (F - H G_2) z + (\bar {\bar G} - HG_1 C_a^{-1} - H G_2 L) y \\
&\equiv& \mathbf{F}_c z + \mathbf{\bar G} y
\end{eqnarray}

which has transfer function representation (see Ch. 3):
\begin{eqnarray}
z(s) = (sI - \mathbf{F}_c)^{-1} \mathbf{\bar G} y(s)
\end{eqnarray}

The feedback law can then be written in the Laplace domain as:
\begin{eqnarray}
u(s) &=& -(G_1 + G_2 L + (sI - \mathbf{F}_c)^{-1} \mathbf{\bar G}) y(s)
\end{eqnarray}

Putting this in the form $u(s) = -D(s)y(s)$ gives the compensator:
\begin{eqnarray}
D(s) = G_1 + G_2 L + (sI - \mathbf{F}_c)^{-1} \mathbf{\bar G}.
\end{eqnarray}

#### c)

Example 8B gives $\bar a_1 = 4$ and $\bar a_2 = 8$.  Plugging these values into $D(s)$ found in the previous section gives:

```python
# use sympy to do reduction
a1, a2, l, s = symbols('a1 a2 l s')
p = a2 + (a1 - 1)*l + ((a1 - 1)*(1-a1)*l - a2)/(s+l+a1)
pprint(simplify(p))
# the beta term cancels regardless of choice of L
a2 = 8.
a1 = 4.
q = (-a2 - l*(a1-1.)**2 + (a2 + l*(a1-1.))*(a1 + l + s)) / (s+l+a1)
pprint(simplify(q))
```

                    2
    -a₂ - l⋅(a₁ - 1)  + (a₂ + l⋅(a₁ - 1))⋅(a₁ + l + s)
    ──────────────────────────────────────────────────
                        a₁ + l + s
    -9.0⋅l + (3.0⋅l + 8.0)⋅(l + s + 4.0) - 8.0
    ──────────────────────────────────────────
                   l + s + 4.0

Use `sympy` reduction above:
\begin{eqnarray}
D(s) = \frac{(3L+8)s + 3L^2 + 3L + 32}{\beta (s + L + 4)}.
\end{eqnarray}

```python
G = control.TransferFunction(np.array([1.]), np.array([1., 1., 0.]))
L = 1.0
num = np.array([3.*L+8., 3.*L**2 + 3.*L+32.])
den = np.array([1., L+4])
D = control.TransferFunction(num, den)
control.root_locus(D*G, PrintGain=False, Plot=True)
#_, _, _ = control.nyquist_plot(D*G, Plot=True, PrintGain=False)
plt.show()
```

For $L \ge 0$, the system has infinite gain margin.  As hinted at in Example 8B, this seems to indicate that a compensator derived with a reduced order observer can have simplified form and better performance characteristics.  Nyquist plots don't make much sense to me, but here it is because the problem asked for it.

```python
_, _, _ = control.nyquist_plot(D*G, Plot=True)
plt.show()
```

### Problem 8.4

#### a)
The gains from Problems 6.1 (regulator) and 7.2 (full-order observer) are:
\begin{eqnarray}
G &=& \begin{pmatrix}
-326.53061224, \
-226.32653061, \
-812.09061224, \
-242.32653061
\end{pmatrix}\\
K &=& \begin{pmatrix}
-11.935 \\
394.510 \\
-452.006 \\
-1993.619
\end{pmatrix}
\end{eqnarray}

The form of the compensator is $D(s) = G (sI - A + BG + KC)^{-1} K$:

```python
A = np.array([[0., 1., 0., 0.], [0., -25., -0.98, 0.], [0., 0., 0., 1.], [0., 25., 10.78, 0.]]).astype('float')
B = np.array([[0.], [0.5], [0.], [-0.5]]).reshape((4,1)).astype('float')
C = np.array([[1., 0., 0., 0.]]).reshape((1, 4)).astype('float')
D = np.zeros((1, 1)).astype('float')
desiredPolesCtrl = np.array([-25., -4., np.complex(-2., 2.*np.sqrt(3)), np.complex(-2., -2.*np.sqrt(3))])
w0 = 5.
desiredPolesObsv = np.roots(np.array([1.0, 2.613*w0, (2.+np.sqrt(2))*w0**2, 2.613*w0**3, w0**4]))
tfD = ch8_utils.fullOrderCompensator(A, B, C, D, desiredPolesCtrl, desiredPolesObsv)
pprint('D(s) = {}'.format(tfD))
sysplant = control.StateSpace(A, B, C, D)
tfplant = control.ss2tf(sysplant)
pprint('H(s) = {}'.format(tfplant))
```

                            D(s) =
    7.648e+05 s^3 + 2.153e+07 s^2 + 5.963e+07 s - 2.041e+05
    -------------------------------------------------------
      s^4 + 21.07 s^3 + 232.7 s^2 - 3.807e+05 s - 1.197e+06
                      H(s) =
                0.5 s^2 + 2.012e-15 s - 4.9
    --------------------------------------------
    s^4 + 25 s^3 - 10.78 s^2 - 245 s + 1.981e-14

The return difference has the desired form for $K=1$ (unity loop gain).

#### b)
The return difference $T(s) = 1 + K D(s) H(s)$ is the denominator of the closed-loop transfer function. As shown above, the zeros of $T(s)$ will be the poles of the closed-loop system.  We can find the range of K for which the closed-loop system is stable by the Hurwitz criteria (see Ch. 4), or just use the `zero` method of `python-control` to find some numeric range for $K$.

```python
# T(s) = 1 + Kb*D(s)*H(s) (Kb != K from above!)
k = np.linspace(0.998, 1.002, 101)
kStable = ch8_utils.stabilityRange(tfD, tfplant, k)
if kStable is not None:
    print('{min} <= k <= {max} defines the range of stable gains.'.format(min=kStable[0], max=kStable[1]))
else:
    print('no stable gains were found in the range provided')
```

    0.99972 <= k <= 1.001 defines the range of stable gains.

#### c)

The closed-loop compensator was shown to achieve the desired closed-loop poles, but with very slight gain margin.  This is not a good sign. Hopefully, the reduced order observer results in a simpler system and more robust system.

### Problem 8.5
#### a)

```python
A = np.array([[0., 1., 0., 0.], [0., -25., -0.98, 0.], [0., 0., 0., 1.], [0., 25., 10.78, 0.]]).astype('float')
B = np.array([[0.], [0.5], [0.], [-0.5]]).reshape((4,1)).astype('float')
C = np.array([[1., 0., 0., 0.]]).reshape((1, 4)).astype('float')
D = np.zeros((1, 1)).astype('float')

### Controller ###
desiredPolesCtrl = np.array([-25., -4., np.complex(-2., 2.*np.sqrt(3)), np.complex(-2., -2.*np.sqrt(3))])
G = ch6_utils.bassGura(A, B, desiredPolesCtrl)

### Observer ###
desiredPolesObsv = np.roots(np.array([1.0, 2.*5., 2.*5.**2, 5.**3]))
Aaa = np.array(A[0, 0]).reshape((1, 1))
Aau = np.array(A[0, 1:]).reshape((1, 3))
Aua = np.array(A[1:, 0]).reshape((3, 1))
Auu = np.array(A[1:, 1:]).reshape((3, 3))
Ba = np.array(B[0]).reshape((1, 1))
Bu = np.array(B[1:]).reshape((3, 1))
Ca = np.array([[1]]).reshape((1, 1)).astype('float')
L, Gbb, H = ch7_utils.reducedOrderObserver(Aaa, Aau, Aua, Auu, Ba, Bu, Ca, desiredPolesObsv)

### Compensator ###
G1 = G[0, 0].reshape((1, 1))
G2 = G[0, 1:].reshape((1, 3))
F = Auu - L@Aau
Ar = F-H@G2
Br = Ar@L + Gbb - H@G1
Cr = G2
Dr = G1+G2@L
sysD = control.StateSpace(Ar, Br, Cr, Dr)
tfD = control.ss2tf(sysD)
pprint('D(s) = {}'.format(tfD))
sysplant = control.StateSpace(A, B, C, D)
tfplant = control.ss2tf(sysplant)
pprint('H(s) = {}'.format(tfplant))
```

                            D(s) =
    1.049e+05 s^3 + 2.955e+06 s^2 + 8.202e+06 s - 4.082e+04
    -------------------------------------------------------
                  s^3 + 18 s^2 - 5.23e+04 s - 1.648e+05
                      H(s) =
                0.5 s^2 + 2.012e-15 s - 4.9
    --------------------------------------------
    s^4 + 25 s^3 - 10.78 s^2 - 245 s + 1.981e-14

The return difference has the desired form for $K=1$ (unity loop gain).

#### b)

```python
# T(s) = 1 + Kb*D(s)*H(s) (Kb != K from above!)
k = np.linspace(0.998, 1.002, 101)
kStable = ch8_utils.stabilityRange(tfD, tfplant, k)
if kStable is not None:
    print('{min} <= k <= {max} defines the range of stable gains.'.format(min=kStable[0], max=kStable[1]))
else:
    print('no stable gains were found in the range provided')
```

    0.99948 <= k <= 1.00112 defines the range of stable gains.

#### c)
The compensated system has slightly better stability margins and with lower order.  This would be preferable to the controller from the previous problem.

### Problem 8.6

#### a)
I'm not drawing any more block diagrams.  As for the compensator:

The gains found in Problem 6.8 were $G = [0, \ 6.5, \ 6.5]$, and for Problem 7.8 was $K = [42.5, \ 16.5, \ 2.]$.

```python
R, _C = 2., 1.
A = np.array([[-3./(R*_C), 1./(R*_C), 0.],[1./(R*_C), -2./(R*_C), 1./(R*_C)], [0., 1./(R*_C), -3./(R*_C)]]).reshape((3, 3)).astype('float')
B = np.array([[2. / (R*_C)], [0.], [0.]]).reshape((3, 1)).astype('float')
C = np.array([[0., 0., 1.]]).reshape((1, 3)).astype('float')
D = np.zeros((1, 1)).astype('float')
desiredPolesCtrl = np.roots(np.array([1., 4., 8., 8.]))
w0 = 3.
desiredPolesObsv = np.roots(np.array([1.0, 2.*w0, 2.*w0**2, w0**3]))
tfD = ch8_utils.fullOrderCompensator(A, B, C, D, desiredPolesCtrl, desiredPolesObsv)
pprint('D(s) = {}'.format(tfD))
sysplant = control.StateSpace(A, B, C, D)
tfplant = control.ss2tf(sysplant)
pprint('H(s) = {}'.format(tfplant))
```

             D(s) =
    120.3 s^2 + 552.5 s + 624
    --------------------------
    s^3 + 6 s^2 + 21.25 s + 40
             H(s) =
                    0.25
    --------------------------
    s^3 + 4 s^2 + 4.75 s + 1.5

$0$ is not a zero of the return difference, so then it is not a pole of the closed-loop transfer function.  This means that the compensator does _not_ provide true integral control.

#### b)

_(omitted)_

#### c)

```python
H = tfD*tfplant
control.root_locus(H, Plot=True, PrintGain=False)
plt.show()
```

```python
control.bode_plot(H)
plt.show()
gm, pm, _, _ = control.margin(H)
print('Gain margin (dB): {}'.format(control.mag2db(gm)))
print('Phase margin (dB): {}'.format(control.mag2db(pm)))
```

### Problem 8.7

#### a)

```python
# with K, M, k, R, r = 40, 1, 2, 100, 0.02
A = np.array([[0, 0, 1, 0], [0, 0, 0, 1], [-40, 40, -100, 0], [40, -40, 0, -100]]).astype('float')
B = np.array([[0], [0], [1], [0]]).astype('float')
C = np.array([1, 0, 0, 0]).astype('float').reshape((1, 4))
D = np.zeros((1, 1)).astype('float')

## from Problem 6.3
closedLoopPoles = np.array([np.complex(-1., -1.), np.complex(-1., 1.), np.complex(-100., -100.), np.complex(-100., 100.)])
G = ch6_utils.bassGura(A, B, closedLoopPoles)

H = control.ss2tf(control.StateSpace(A, B, C, D))
N = control.zero(H)
poly1 = np.poly(N)

# place 2 remaining observer poles in a butterworth pattern order 2 -> z**2 + sqrt(2)*z + 1
# z = s/w0
w0 = 2
p2 = np.poly(np.roots([1., 1.414*w0, w0**2]))
p = np.polymul(poly1, p2)

K = ch7_utils.obsBassGura(A, C, np.roots(p))
pprint('K = ')
pprint(K)
```

    K =
    [[-9.7172000e+01]
     [ 1.8189894e-11]
     [ 9.6812000e+03]
     [ 4.0000000e+01]]

#### b) and c)

```python
Abarc = A - B.dot(G) - K.dot(C)
sysD = control.StateSpace(Abarc, K, G, np.zeros((1, 1)))
tfD = control.ss2tf(sysD)
pprint('D(s) = {}'.format(tfD))
sysplant = control.StateSpace(A, B, C, np.zeros((1, 1)))
tfplant = control.ss2tf(sysplant)
pprint('H(s) = {}'.format(tfplant))

H = tfD*tfplant

gm, pm, _, _ = control.margin(H)
print('Gain margin (dB): {}'.format(control.mag2db(gm)))
print('Phase margin: {}'.format(pm))

control.root_locus(H, Plot=True, PrintGain=False)
plt.show()
```

The gain and phase margins here suck!  Maybe the design will be better in the next problem.

### Problem 8.8

#### a)

```python
# with K, M, k, R, r = 40, 1, 2, 100, 0.02
A = np.array([[0, 0, 1, 0], [0, 0, 0, 1], [-40, 40, -100, 0], [40, -40, 0, -100]]).astype('float')
B = np.array([[0], [0], [1], [0]]).astype('float')
C = np.array([0, 1, 0, 0]).astype('float').reshape((1, 4))
D = np.zeros((1, 1)).astype('float')

## from Problem 6.3
closedLoopPoles = np.array([np.complex(-1., -1.), np.complex(-1., 1.), np.complex(-100., -100.), np.complex(-100., 100.)])
G = ch6_utils.bassGura(A, B, closedLoopPoles)

H = control.ss2tf(control.StateSpace(A, B, C, D))
N = control.zero(H)
poly1 = np.poly(N)

# place 4 remaining observer poles in a butterworth pattern order 4
# z = s/w0
w0 = 2
p2 = np.poly(np.roots([1., 2.613*w0, (2.+np.sqrt(2.))*w0**2, 2.613*w0**3, w0**4]))
p = np.polymul(poly1, p2)
K = ch7_utils.obsBassGura(A, C, np.roots(p))
pprint('K = ')
pprint(K)
```

    K =
     [[-2.35323455e+04]
      [-1.94774000e+02]
      [ 2.34382390e+06]
     [ 2.88884569e+04]]

#### b) and c)

```python
Abarc = A - B.dot(G) - K.dot(C)
sysD = control.StateSpace(Abarc, K, G, np.zeros((1, 1)))
tfD = control.ss2tf(sysD)
pprint('D(s) = {}'.format(tfD))
sysplant = control.StateSpace(A, B, C, np.zeros((1, 1)))
tfplant = control.ss2tf(sysplant)
pprint('H(s) = {}'.format(tfplant))

H = tfD*tfplant

gm, pm, _, _ = control.margin(H)
print('Gain margin (dB): {}'.format(control.mag2db(gm)))
print('Phase margin: {}'.format(pm))

control.root_locus(H, Plot=True, PrintGain=False)
#plt.axis([-10, 10, -10, 10])
plt.show()
```

##### Similarities to 8.7
* Compensators have similar form (3 zeros vs. 4 poles)

##### Differences to 8.7
* worse gain/phase margins
* greater freedom of placing observer poles
* qualitatively different pole/zero migration indicated by root locus plots

### Problem 8.9
#### a)
See Problem 4.4:

\begin{eqnarray}
A_{aa} &=& \begin{pmatrix}
-0.746 & 0.387 & 0 \\
0.024 & -0.0174 & 0 \\
0 & 0 & 0 \\
\end{pmatrix},
A_{au} = \begin{pmatrix}
-12.9 & 0 \\
4.31 & 0 \\
0 & 0 \\
\end{pmatrix}, \\
A_{ua} &=& \begin{pmatrix}
0.006 & 0.001 & 0 \\
-1 & 0 & 1 \\
\end{pmatrix},
A_{uu} = \begin{pmatrix}
-0.746 & 0.0369 \\
0 & 0 \\
\end{pmatrix},\\
B_{a} &=& \begin{pmatrix}
6.05 & 0.952 \\
-0.416 & -1.76 \\
0 & 0 \\
\end{pmatrix},\\
B_{u} &=& \begin{pmatrix}
0.0012 & 0.0092 \\
0 & 0 \\
\end{pmatrix}
\end{eqnarray}

#### b)

As I've said before, I am not doing any more block-diagrams.

#### c)

Condition 8.86 is $B_u - L C_a B_a = \mathbf{0}$.  $B_u$ is $2 \times 2$, while $C_a$ is $3 \times 3$ and $B_a$ is $3 \times 2$.  This means that $L$ must be $2 \times 3$ just by size requirements.  Let:
\begin{eqnarray}
L = \begin{pmatrix}
l_1 & l_2 & l_3 \\
l_4 & l_5 & l_6
\end{pmatrix},
\end{eqnarray}
and then see if is a way to satisfy Condition 8.86.

```python
Ba = Matrix([[6.05, .952], [-.416, -1.76], [0, 0]])
Bu = Matrix([[.0012, .0092], [0, 0]])
Ca = eye(3)
l = symbols("l1:7")
L = Matrix([[l[0], l[1], l[2]], [l[3], l[4], l[5]]])
pprint(L*Ba)
pprint(' = ')
pprint(Bu)
```

    ⎡6.05⋅l₁ - 0.416⋅l₂  0.952⋅l₁ - 1.76⋅l₂⎤
    ⎢                                      ⎥
    ⎣6.05⋅l₄ - 0.416⋅l₅  0.952⋅l₄ - 1.76⋅l₅⎦
     =
    ⎡0.0012  0.0092⎤
    ⎢              ⎥
    ⎣  0       0   ⎦

This is a linear system with $\mathbf{x} = [l_1 \ l_2 \ l_4 \ l_5]^T$.

```python
As = np.array([[6.05, -.416, 0, 0], [.952, 1.76, 0, 0], [0, 0, 6.05, -.416], [0, 0, 0.952, -1.76]]).astype('float').reshape((4, 4))
bs = np.array([[.0012], [.0092], [0], [0]]).astype('float').reshape((4, 1))
xs = np.linalg.solve(As, bs)
pprint('[[l1], [l2], [l4], [l5]]\' = ')
pprint(xs)
```

    [[l1], [l2], [l4], [l5]]' =
     [[ 0.00053777]
      [ 0.00493639]
      [ 0.        ]
     [-0.        ]]

The gain matrix $L$ can be made to meet Condition 8.86 when:
\begin{eqnarray}
L = \begin{pmatrix}
0.000538 & 0.00494 & l_3 \\
0 & 0 & l_6 \\
\end{pmatrix}
\end{eqnarray}

The parameters $l_3$ and $l_6$ can be chosen freely to meet other observer design objectives; e.g. pole-placement.

But what about the suitability of the pole-placement?  The observer poles are determined by the characteristic polynomial of $F$.

```python
Auu = Matrix([[-0.746, 0.0369], [0, 0]])
Aau = Matrix([[-12.9, 0], [4.31, 0], [0, 0]])
L = Matrix([[5.37777e-4, 4.93639e-3, l[2]], [0, 0, l[5]]])
F = Auu - L*Aau
pprint(F)
```

    ⎡-0.7603385176  0.0369⎤
    ⎢                     ⎥
    ⎣      0          0   ⎦

Clearly, $F$ has a pole at $s=0$, so the closed-loop dynamics will not be asymptotically stable.  Because of this one pole, every pole is not at a desirable location.

#### d)
We have two free gains, $l_3$ and $l_6$.  The reduced-order observer will have two singular values, because it is order 2.  Therefore, we can place the remaining poles to cancel transmission zeros of the (unavailable) process.

### Problem 8.10

#### a)
We have a reduced order observer from Problem 7.5, and a control law from 6.9.  At this point, the methods of Problem 8.5 above can be used to find each compensator in terms of the gains $g_i$, $l_i$ for $i = 1, 2, 3 ,4$.  This would be kind of tedious, so I'm not going to do it.

#### b)
Block-diagram omitted.

### Problem 8.11
_Omitted because of bogus exogenous method_

### Problem 8.12
#### a)
Get $G$ from Problem 6.6.  The problem specifically calls out a fourth-order observer, so the reference $h_0$ and the airspeed $V$ are discarded in the subsequent treatment.

```python
A = np.array([[-1, 1, 0, 0], [-5, -.5, 0, 0], [0, 1, 0, 0], [-1, 0, 1, 0]]).astype('float').reshape((4, 4))
B = np.array([[-.1], [-9], [0], [0]]).astype('float').reshape((4, 1))
C = np.array([[0., 0., 0., 1.]]).astype('float').reshape((1, 4))
D = np.zeros((1, 1))
closedLoopPoles = np.array(2.5*np.array([np.complex(-0.5, 0.5*np.sqrt(3.)), np.complex(-0.5, -0.5*np.sqrt(3.)),
                                         np.complex(-0.5*np.sqrt(3.), 0.5), np.complex(-0.5*np.sqrt(3.), -0.5)]))
desiredObsPoles = np.array([np.complex(-2., np.sqrt(3)*2.), np.complex(-2., -np.sqrt(3)*2.),
                      np.complex(-np.sqrt(3)*2., 2.), np.complex(-np.sqrt(3)*2., -2.)])
tfComp = ch8_utils.fullOrderCompensator(A, B, C, D, closedLoopPoles, desiredObsPoles)
print('D(s) = {}'.format(tfComp))
sysPlant = control.StateSpace(A, B, C, D)
tfPlant = control.ss2tf(sysPlant)
H = tfComp*tfPlant
```

    D(s) =
       -339.8 s^3 - 725.8 s^2 - 2096 s - 1176
    --------------------------------------------
    s^4 + 16.26 s^3 + 127.8 s^2 + 633.2 s + 1786

#### b)

```python
control.root_locus(H, Plot=True, PrintGain=False)
plt.show()
k = np.linspace(1, 2, 101)
kStable = ch8_utils.stabilityRange(tfComp, tfPlant, k)
if kStable is not None:
    print('{min} <= k <= {max} defines the range of stable gains.'.format(min=kStable[0], max=kStable[1]))
else:
    print('no stable gains were found in the range provided')
```

### Problem 8.13
#### a)
We can reuse a lot from Problem 8.12.  However, the system needs to be rearranged so that $h$. which is measurable, is the first state.

```python
A = np.array([[0, -1, 0, 1], [0, -1, 1, 0], [0, -5, -.5, 0], [0, 0, 1, 0]]).astype('float').reshape((4, 4))
B = np.array([[0], [-.1], [-9], [0]]).astype('float').reshape((4, 1))
C = np.array([[1, 0, 0, 0]]).astype('float').reshape((1, 4))
D = np.zeros((1, 1))
sysPlant = control.StateSpace(A, B, C, D)

### Controller ###
closedLoopPoles = np.array(2.5*np.array([np.complex(-0.5, 0.5*np.sqrt(3.)), np.complex(-0.5, -0.5*np.sqrt(3.)),
                                         np.complex(-0.5*np.sqrt(3.), 0.5), np.complex(-0.5*np.sqrt(3.), -0.5)]))
G = ch6_utils.bassGura(A, B, closedLoopPoles)

### Observer ###
desiredObsPoles = np.array([np.complex(-4., 0.), np.complex(-2.*np.sqrt(2), 2.*np.sqrt(2)),
                            np.complex(-2.*np.sqrt(2), -2.*np.sqrt(2))])
Aaa = np.array([A[0, 0]]).reshape((1, 1))
Aau = np.array(A[0, 1:]).reshape((1, 3))
Aua = np.array(A[1:, 0]).reshape((3, 1))
Auu = np.array(A[1:, 1:]).reshape((3, 3))
Ba = np.array([0]).reshape((1, 1))
Bu = np.array(B[1:]).reshape((3, 1))
Ca = np.array([[1]])
L, Gbb, H = ch7_utils.reducedOrderObserver(Aaa, Aau, Aua, Auu, Ba, Bu, Ca, desiredObsPoles)
F = Auu - L@Aau

### Compensator ###
G1 = np.array([G[0, 0]]).reshape((1, 1))
G2 = np.array(G[0, 1:]).reshape((1, 3))
Ar = F-H@G2
Br = Ar@L + Gbb - H@G1
Cr = G2
Dr = G1+G2@L
sysComp = control.StateSpace(Ar, Br, Cr, Dr)
tfComp = control.ss2tf(sysComp)
print('D(s) = {}'.format(tfComp))
tfPlant = control.ss2tf(sysPlant)
H = tfComp*tfPlant
```

    D(s) =
    -81.66 s^3 - 178.9 s^2 - 500.7 s - 294.1
    ----------------------------------------
        s^3 + 14.99 s^2 + 108.1 s + 373.2

     [-2.82842712+2.82842712j -2.82842712-2.82842712j -4.        +0.j
     -1.25      +2.16506351j -1.25      -2.16506351j -2.16506351+1.25j
                         -2.16506351-1.25j      ]

#### b)

```python
control.root_locus(H, Plot=True, PrintGain=False)
plt.show()
# T(s) = 1 + Kb*D(s)*H(s) (Kb != K from above!)
k = np.linspace(1, 2.5, 101)
kStable = ch8_utils.stabilityRange(tfComp, tfPlant, k)
if kStable is not None:
    print('{min} <= k <= {max} defines the range of stable gains.'.format(min=kStable[0], max=kStable[1]))
else:
    print('no stable gains were found in the range provided')
```

The margins for the compensator with reduced-order observer is significantly better.  The root locus indicates some nice pole-zero cancellation (dots are very near x's near the line $y=0$ above).  There is the same number of poles as zeros in the reduced order observer case.  This compensator is lower order with better closed-loop stability properties.  This is pretty cool, I think.


```python
from IPython.core.display import HTML, Image
css_file = 'style.css'
HTML(open(css_file, 'r').read())
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Philosopher:400,700,400italic,700italic' rel='stylesheet' type='text/css'>

```python
from sympy import init_printing, symbols, eye, Matrix, Rational
from warnings import filterwarnings
```

```python
init_printing(use_latex = 'mathjax')
filterwarnings('ignore')
```

# Equations for the inverse of a matrix

# Cramer's rule

# The volume of a box

## Deriving an equation for the inverse of a matrix

* The equation for the inverse of a matrix
$$ {A}^{-1}=\frac{1}{\left| A \right|}{C}^{T} $$
* With arithmetic alteration we have the following
$$ \therefore {A}{C}^{T}=\left|A\right|I $$
* Writing out the left-hand side we have
$$ \begin{bmatrix} {a}_{11} & {a}_{12} & \dots & {a}_{1n} \\ {a}_{21} & {a}_{22} & \dots & {a}_{2n} \\  \vdots & \vdots & \dots & \vdots \\ {a}_{n1} & {a}_{n2} & \dots & {a}_{nn} \end{bmatrix} \begin{bmatrix} { C }_{ 11 } & { C }_{ 21 } & \dots  & { C }_{ n1 } \\ { C }_{ 12 } & { C }_{ 22 } & \dots  & { C }_{ n2 } \\ \vdots  & \vdots  & \dots  & \vdots  \\ { C }_{ 1n } & { C }_{ 2n } & \dots  & { C }_{ nn } \end{bmatrix} $$

* From the previous lecture we had the equation for the determinant (using cofactors), which correlates with the above (looking at row 1 times column 1 (*i*=1)
$$ \left|A\right|={a}_{i1}{C}_{i1} + {a}_{i2}{C}_{i2} \dots {a}_{in}{C}_{in}$$
* Alas we have to get |A| only on the main diagonal for the right-hand side above
* It follows, though that i.e. for position row 1, column 2 we do get a zero
* Look at the 2×2 matrix
$$ { \begin{bmatrix} a & b \\ c & d \end{bmatrix} }^{ -1 }=\frac { 1 }{ \left| A \right|  } { C }^{ T }\\ { \begin{bmatrix} a & b \\ c & d \end{bmatrix} }^{ -1 }=\frac { 1 }{ \left| A \right|  } { \begin{bmatrix} d & -c \\ -b & a \end{bmatrix} }^{ T }\\ { \begin{bmatrix} a & b \\ c & d \end{bmatrix} }^{ -1 }=\frac { 1 }{ \left| A \right|  } { \begin{bmatrix} d & -b \\ -c & a \end{bmatrix} } $$
* So for ACT we would have the following (note, though what happens if we try and get row 1, column 2
$$ \begin{bmatrix} a & b \\ c & d \end{bmatrix}{ \begin{bmatrix} d & -b \\ -c & a \end{bmatrix} }=\begin{bmatrix} ad-bc & -ab+ab \\ cd-cd & ad-bc \end{bmatrix}=\begin{bmatrix} ad-bc & 0 \\ 0 & ad-bc \end{bmatrix}=\begin{bmatrix} \left| A \right|  & 0 \\ 0 & \left| A \right|  \end{bmatrix}=\left| A \right| \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} $$
* ... and that's so cool!

## Cramer's rule

* From A**x**=**b** we have **x**=A-1**b**, which gives us the following
$$ \underline { x } =\frac { 1 }{ \left| A \right|  } { C }^{ T }\underline { b } $$
$$ \therefore \begin{bmatrix} { x }_{ 1 } \\ { x }_{ 2 } \\ \vdots  \\ { x }_{ n } \end{bmatrix}=\frac { 1 }{ \left| A \right|  } \begin{bmatrix} { C }_{ 11 } & { C }_{ 21 } & \dots  & { C }_{ n1 } \\ { C }_{ 12 } & { C }_{ 22 } & \dots  & { C }_{ n2 } \\ \vdots  & \vdots  & \dots  & \vdots  \\ { C }_{ 1n } & { C }_{ 2n } & \dots  & { C }_{ nn } \end{bmatrix}\begin{bmatrix} { b }_{ 1 } \\ { b }_{ 2 } \\ \vdots  \\ { b }_{ n } \end{bmatrix} $$

* This is difficult to see, but we successively replace each column in A with the column vector **b**, whic creates a bunch of new matrices Bj, such that the following applies
$$  \begin{bmatrix} { x }_{ 1 } \\ { x }_{ 2 } \\ \vdots  \\ { x }_{ n } \end{bmatrix}=\begin{bmatrix} \frac { \left| { B }_{ 1 } \right|  }{ \left| A \right|  }  \\ \frac { \left| { B }_{ 2 } \right|  }{ \left| A \right|  }  \\ \vdots  \\ \frac { \left| { B }_{ n } \right|  }{ \left| A \right|  }  \end{bmatrix} $$

## The volume of a box (parallelepiped )

* Consider a box in three dimensions (each side is a parallelogram)
* Make one corner coincide with origin
* The vector coordinate of the three sides that emanate from this corner become the rows of a square matrix, 3×3 in this case
* The volume is then the determinant of this matrix

### Consider a square box of sides of unit length one

```python
A = eye(3)
A
```

$$\left[\begin{matrix}1 & 0 & 0\\0 & 1 & 0\\0 & 0 & 1\end{matrix}\right]$$

```python
A.det()
```

$$1$$

* This proves the first property of determinants

### What about the orthogonal matrix

* Here we have the three edges being orthonormal
* We know that QTQ = I

$$ \left| { Q }^{ T }Q \right| =\left| I \right| \\ \left| { Q }^{ T } \right| \left| Q \right| =\left| I \right| \\ \because \quad \left| { Q }^{ T } \right| =\left| Q \right| \\ \therefore \quad \left| Q \right| \left| Q \right| =\left| I \right| \\ { \left| Q \right|  }^{ 2 }=\left| I \right| =1\\ \therefore \quad \left| Q \right| =\pm 1 $$

### A rectangular box (edges square)

* Doubling an edge doubles the volume
* This is akin to a single row multiplied by a scalar
* Thus determinant will increase by this scalar (multiplication)

### Area of  parallelogram and a triangle

* The area of a parallelogram is just the determinant of a 2×2 matrix with the rows being row vectors of two sides from the origin
* The area of a triangle is simply half of this

* For the triangle that is not at the origin, with three angles at (*x*1, *y*1), (*x*2, *y*2) and (*x*3, *y*3), we simply subtract values along the axes from each other
* That is akin to getting the determinant of this matrix
$$ \begin{vmatrix} { x }_{ 1 } & { y }_{ 1 } & 1 \\ { x }_{ 2 } & { y }_{ 2 } & 1 \\ { x }_{ 3 } & { y }_{ 3 } & 1 \end{vmatrix} $$
* Simple row reduction is equivalent to moving the triangle by the subtraction above
$$ \begin{vmatrix} { x }_{ 1 } & { y }_{ 1 } & 1 \\ { x }_{ 2 }-{ x }_{ 1 } & { y }_{ 2 }-{ y }_{ 1 } & 0 \\ { x }_{ 3 }-{ x }_{ 1 } & { y }_{ 3 }-{ y }_{ 1 } & 0 \end{vmatrix} $$

## Example problems

### Example problem 1

* Calculate the volume of the tetrahedron with vertices being the following vectors
    * **a**1=(2,2,-1), **a**2=(1,3,0), **a**3=(-1,1,4)
* Also calculate the volume if **a**3=(-201,-199,104)

#### Solution

```python
A = Matrix([[2, 2, -1], [1, 3, 0], [-1, 1, 4]])
A
```

$$\left[\begin{matrix}2 & 2 & -1\\1 & 3 & 0\\-1 & 1 & 4\end{matrix}\right]$$

* The volume of a tetrahedron is a third times the area of the (any) base and the height from the (chosen) base
* The volume of a parallelepiped is the area of the base times the height
* If we keep the base of the two the same and the apex the same, we note that the base is twice the area of the triangle that forms the base of the tetrahedron
* We thus have that the volume of the tetrahedron is a 6th of the volume of the parallelepiped

```python
A.det()
```

$$12$$

```python
Rational(1, 6) * A.det()
```

$$2$$

```python
A_new = Matrix([[2, 2, -1], [1, 3, 0], [-201, -199, 104]])
A_new
```

$$\left[\begin{matrix}2 & 2 & -1\\1 & 3 & 0\\-201 & -199 & 104\end{matrix}\right]$$

```python
Rational(1, 6) * A_new.det()
```

$$2$$

* By the second part of the third property of determinants we know, though, that a constant multiple of a row subtracted from another (one of the elementary row operations) does not change the determinant
* In this case we subtracted 100 times row 1 from row 3

* In effect, the height is not changing; the apex is moving away parallel to **a**1

```python

```

Mathematics/Linear Algebra/0.0 MIT-18.06 - Jupyter/Lecture_20_Equations_for_the_inverse_Cramer_rule_Volume_of_a_box.ipynb


```python
%matplotlib inline
```

.. currentmodule:: dgl

Graph Classification Tutorial
=============================

**Author**: `Mufei Li <https://github.com/mufeili>`_,
`Minjie Wang <https://jermainewang.github.io/>`_,
`Zheng Zhang <https://shanghai.nyu.edu/academics/faculty/directory/zheng-zhang>`_.

In this tutorial, you learn how to use DGL to batch multiple graphs of variable size and shape. The
tutorial also demonstrates training a graph neural network for a simple graph classification task.

Graph classification is an important problem
with applications across many fields, such as bioinformatics, chemoinformatics, social
network analysis, urban computing, and cybersecurity. Applying graph neural
networks to this problem has been a popular approach recently. This can be seen in the following reserach references:
`Ying et al., 2018 <https://arxiv.org/abs/1806.08804>`_,
`Cangea et al., 2018 <https://arxiv.org/abs/1811.01287>`_,
`Knyazev et al., 2018 <https://arxiv.org/abs/1811.09595>`_,
`Bianchi et al., 2019 <https://arxiv.org/abs/1901.01343>`_,
`Liao et al., 2019 <https://arxiv.org/abs/1901.01484>`_,
`Gao et al., 2019 <https://openreview.net/forum?id=HJePRoAct7>`_).

Simple graph classification task
--------------------------------
In this tutorial, you learn how to perform batched graph classification
with DGL. The example task objective is to classify eight types of topologies shown here.

    :align: center

Implement a synthetic dataset :class:`data.MiniGCDataset` in DGL. The dataset has eight
different types of graphs and each class has the same number of graph samples.

```python
from dgl.data import MiniGCDataset
import matplotlib.pyplot as plt
import networkx as nx
# A dataset with 80 samples, each graph is
# of size [10, 20]
dataset = MiniGCDataset(80, 10, 20)
graph, label = dataset[0]
fig, ax = plt.subplots()
nx.draw(graph.to_networkx(), ax=ax)
ax.set_title('Class: {:d}'.format(label))
plt.show()
```

Form a graph mini-batch
-----------------------
To train neural networks efficiently, a common practice is to batch
multiple samples together to form a mini-batch. Batching fixed-shaped tensor
inputs is common. For example, batching two images of size 28 x 28
gives a tensor of shape 2 x 28 x 28. By contrast, batching graph inputs
has two challenges:

* Graphs are sparse.
* Graphs can have various length. For example, number of nodes and edges.

To address this, DGL provides a :func:`dgl.batch` API. It leverages the idea that
a batch of graphs can be viewed as a large graph that has many disjointed
connected components. Below is a visualization that gives the general idea.

    :width: 400pt
    :align: center

Define the following ``collate`` function to form a mini-batch from a given
list of graph and label pairs.

```python
import dgl
import torch

def collate(samples):
    # The input `samples` is a list of pairs
    #  (graph, label).
    graphs, labels = map(list, zip(*samples))
    batched_graph = dgl.batch(graphs)
    return batched_graph, torch.tensor(labels)
```

The return type of :func:`dgl.batch` is still a graph. In the same way,
a batch of tensors is still a tensor. This means that any code that works
for one graph immediately works for a batch of graphs. More importantly,
because DGL processes messages on all nodes and edges in parallel, this greatly
improves efficiency.

Graph classifier
----------------
Graph classification proceeds as follows.

From a batch of graphs, perform message passing and graph convolution
for nodes to communicate with others. After message passing, compute a
tensor for graph representation from node (and edge) attributes. This step might
be called readout or aggregation. Finally, the graph
representations are fed into a classifier $g$ to predict the graph labels.

Graph convolution layer can be found in the ``dgl.nn.<backend>`` submodule.

```python
from dgl.nn.pytorch import GraphConv
```

Readout and classification
--------------------------
For this demonstration, consider initial node features to be their degrees.
After two rounds of graph convolution, perform a graph readout by averaging
over all node features for each graph in the batch.

\begin{align}h_g=\frac{1}{|\mathcal{V}|}\sum_{v\in\mathcal{V}}h_{v}\end{align}

In DGL, :func:`dgl.mean_nodes` handles this task for a batch of
graphs with variable size. You then feed the graph representations into a
classifier with one linear layer to obtain pre-softmax logits.

```python
import torch.nn as nn
import torch.nn.functional as F

class Classifier(nn.Module):
    def __init__(self, in_dim, hidden_dim, n_classes):
        super(Classifier, self).__init__()
        self.conv1 = GraphConv(in_dim, hidden_dim)
        self.conv2 = GraphConv(hidden_dim, hidden_dim)
        self.classify = nn.Linear(hidden_dim, n_classes)

    def forward(self, g):
        # Use node degree as the initial node feature. For undirected graphs, the in-degree
        # is the same as the out_degree.
        h = g.in_degrees().view(-1, 1).float()
        # Perform graph convolution and activation function.
        h = F.relu(self.conv1(g, h))
        h = F.relu(self.conv2(g, h))
        g.ndata['h'] = h
        # Calculate graph representation by averaging all the node representations.
        hg = dgl.mean_nodes(g, 'h')
        return self.classify(hg)
```

Setup and training
------------------
Create a synthetic dataset of $400$ graphs with $10$ ~
$20$ nodes. $320$ graphs constitute a training set and
$80$ graphs constitute a test set.

```python
import torch.optim as optim
from torch.utils.data import DataLoader

# Create training and test sets.
trainset = MiniGCDataset(320, 10, 20)
testset = MiniGCDataset(80, 10, 20)
# Use PyTorch's DataLoader and the collate function
# defined before.
data_loader = DataLoader(trainset, batch_size=32, shuffle=True,
                         collate_fn=collate)

# Create model
model = Classifier(1, 256, trainset.num_classes)
loss_func = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)
model.train()

epoch_losses = []
for epoch in range(80):
    epoch_loss = 0
    for iter, (bg, label) in enumerate(data_loader):
        prediction = model(bg)
        loss = loss_func(prediction, label)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        epoch_loss += loss.detach().item()
    epoch_loss /= (iter + 1)
    print('Epoch {}, loss {:.4f}'.format(epoch, epoch_loss))
    epoch_losses.append(epoch_loss)
```

The learning curve of a run is presented below.

```python
plt.title('cross entropy averaged over minibatches')
plt.plot(epoch_losses)
plt.show()
```

The trained model is evaluated on the test set created. To deploy
the tutorial, restrict the running time to get a higher
accuracy ($80$ % ~ $90$ %) than the ones printed below.

```python
model.eval()
# Convert a list of tuples to two lists
test_X, test_Y = map(list, zip(*testset))
test_bg = dgl.batch(test_X)
test_Y = torch.tensor(test_Y).float().view(-1, 1)
probs_Y = torch.softmax(model(test_bg), 1)
sampled_Y = torch.multinomial(probs_Y, 1)
argmax_Y = torch.max(probs_Y, 1)[1].view(-1, 1)
print('Accuracy of sampled predictions on the test set: {:.4f}%'.format(
    (test_Y == sampled_Y.float()).sum().item() / len(test_Y) * 100))
print('Accuracy of argmax predictions on the test set: {:4f}%'.format(
    (test_Y == argmax_Y.float()).sum().item() / len(test_Y) * 100))
```

The animation here plots the probability that a trained model predicts the correct graph type.

To understand the node and graph representations that a trained model learned,
we use `t-SNE, <https://lvdmaaten.github.io/tsne/>`_ for dimensionality reduction
and visualization.

    :align: center

    :align: center

The two small figures on the top separately visualize node representations after one and two
layers of graph convolution. The figure on the bottom visualizes
the pre-softmax logits for graphs as graph representations.

While the visualization does suggest some clustering effects of the node features,
you would not expect a perfect result. Node degrees are deterministic for
these node features. The graph features are improved when separated.

What's next?
------------
Graph classification with graph neural networks is still a new field.
It's waiting for people to bring more exciting discoveries. The work requires
mapping different graphs to different embeddings, while preserving
their structural similarity in the embedding space. To learn more about it, see
`How Powerful Are Graph Neural Networks? <https://arxiv.org/abs/1810.00826>`_ a research paper
published for the International Conference on Learning Representations 2019.

For more examples about batched graph processing, see the following:

* Tutorials for `Tree LSTM <https://docs.dgl.ai/tutorials/models/2_small_graph/3_tree-lstm.html>`_ and `Deep Generative Models of Graphs <https://docs.dgl.ai/tutorials/models/3_generative_model/5_dgmg.html>`_
* An example implementation of `Junction Tree VAE <https://github.com/dmlc/dgl/tree/master/examples/pytorch/jtnn>`_