#  4 Trajectory Analysis

***

** Important Note **

Before proceeding to the rest of the analysis, it is a good time to define a path that points to the location of the MD simulation data, which we will analyze here.

If you successfully ran the MD simulation, the correct path should be:

```python
path="OxCompBio-Datafiles/run"
```

If however, you need/want to use the data from the simulation that has been already performed, uncomment the command below to instead define the path that points to the prerun simulation.

```python
#path="OxCompBio-Datafiles/prerun/run"
```

##  4.1 Visualize the simulation

The simplest and easiest type of analysis you should always do is to look at it with your eyes! Your eyes will tell you if something strange is happening immediately. A numerical analysis may not.

###   4.1.1 VMD

*Note: Again, this step is optional. If you don't have VMD, go to section 4.1.2 below to visualize the trajectory with NGLView instead.*

Let us look at the simulations on VMD.

Open your vmd, by typing on your terminal:

`% vmd`

When it has finished placing all the windows on the screen. Click on `File` in the VMD main menu window and select `New Molecule`. The Molecule File Browser window should appear. Click on `Browse...` then select the `OxCompBio-Datafiles` and then the `run` directory and finally select `em.gro` (i.e. the file you made that has protein system energy minimized). Click `OK` and then click `Load`. It should load up the starting coordinates into the main window. Then click `Browse...` in the Molecule File Browser window. Select again the `OxCompBio-Datafiles`, then the `run` directory and then `md.xtc`. Select `OK` and then hit `Load`. The trajectory should start loading into the main VMD window.

Although things will be moving, you can see that it is quite difficult to visualize the individual components. That is one of the problems with simulating such large and complicated systems. VMD makes it quite easy to look at individual components of a system. For example, let us consider the protein only. On the VMD Main menu, left-click on Graphics and select `Representations`. A new menu will appear (`Graphical Representations`). In the box entitled `Selected Atoms` type protein and hit enter. Only those atoms that form part of the protein are now selected. Various other selections and drawing methods will help to visualize different aspects of the simulation.

 **Questions**

* How would you say the protein behaves?

* Is it doing anything unexpected? What would you consider unexpected behaviour?

###  4.1.2 NGLView

You have already tested NGLView at the Python tutorial (Notebook `12_ProteinAnalysis`) and at the beginning of this tutorial. This time however, you can visualize the trajectory you generated after carrying out the MD simulation.

You should also be familiar now with the MDAnalysis Python library that we will use to analyze the MD trajectory. We will also use it below, to create a Universe and load it on NGLView.

```python
# Import MDAnalysis and NGLView
import MDAnalysis
import nglview

# Load the protein structure and the trajectory as a universe named protein
protein=MDAnalysis.Universe(f"{path}/em.gro", f"{path}/md_fit.xtc")
protein_view = nglview.show_mdanalysis(protein)
protein_view.gui_style = 'ngl'

#Color the protein based on its secondary structure
protein_view.update_cartoon(color='sstruc')
protein_view

```

    ThemeManager()

    NGLWidget(gui_style='ngl', max_frame=100)

 **Questions**

* How would you say the protein behaves?

* Is it doing anything unexpected? What would you consider unexpected behaviour?

Now that we are sure the simulation is not doing anything ridiculous, we can start to ask questions about the simulation. The first thing to establish is whether the simulation has equilibrated to some state. So what are some measures of the system
being equilibrated? And what can we use to test the reliability of the simulation?

##  4.2 System Equilibration

###  4.2.1 Temperature fluctuation

The system temperature as a function of time was calculated in the previous section, with the built-in GROMACS tool `gmx energy`, but we still have not looked at it. It is now time to plot the temperature *vs* time and assess the results.

 **Questions**

* Does the temperature fluctuate around an equilibrium  value?

* Does this value correspond to the temperature that we predefined in the `md.mdp` input file?

Import numpy and pyplot from matplotlib, required to read and plot the data, respectively.

```python
# We declare matplotlib inline to make sure it plots properly
%matplotlib inline
# We need to import numpy
import numpy as np
# We need pyplot from matplotlib to generate our plots
from matplotlib import pyplot
```

Now, using numpy, we can read the data from the `1hsg_temperature.xvg` file; the first column is the time (in ps) and the secong is the system temperature (in K).

```python
# Read the file that contains the system temperature for each frame
time=np.loadtxt(f"{path}/1hsg_temperature.xvg", comments=['#','@'])[:, 0]
temperature=np.loadtxt(f"{path}/1hsg_temperature.xvg", comments=['#','@'])[:, 1]
```

You can use numpy again to compute the average temperature and its standard deviation.

```python
# Calculate and print the mean temperature and the standard deviation
# Keep only two decimal points
mean_temperature=round(np.mean(temperature), 2)
std_temperature=round(np.std(temperature), 2)
print(f"The mean temperature is {mean_temperature} ± {std_temperature} K")
```

    The mean temperature is 300.01 ± 1.79 K

Finally, you can plot the temperature *vs* simulation time.

```python
# Plot the temperature
pyplot.plot(time, temperature, color='darkred')
pyplot.title("Temperature over time")
pyplot.xlabel("Time [ps]")
pyplot.ylabel("Temperature [K]")
pyplot.show()
```

###  4.2.2 Energy of the system

Another set of properties that is quite useful to examine is the various energetic contributions to the energy. The total
energy should be constant. but the various contributions can change and this can sometimes indicate something
interesting or strange happening in your simulation. Let us look at some energetic properties of the simulation.

We have already exctracted the Lennard-Jones energy, the Coulomb energy and the potential energy using again the GROMACS built-in tool `gmx energy`. The data of these three energetic components are saved in the same file called `1hsg_energies.xvg`; the first column contains the time (in ps) and the columns that follow contain the energies (in kJ/mol), in the same order as they were generated.

We can now read the data from the `1hsg_energies.xvg` file using numpy.

```python
# Read the file that contains the various energetic components for each frame
time=np.loadtxt(f"{path}/1hsg_energies.xvg", comments=['#','@'])[:, 0]
lennard_jones=np.loadtxt(f"{path}/1hsg_energies.xvg", comments=['#','@'])[:, 1]
coulomb=np.loadtxt(f"{path}/1hsg_energies.xvg", comments=['#','@'])[:, 2]
potential=np.loadtxt(f"{path}/1hsg_energies.xvg", comments=['#','@'])[:, 3]
```

And now that we read the data file, we can plot the energetic components *vs* simulation time in separate plots using matplotlib.

```python
# Plot the Lennard-Jones energy
pyplot.plot(time, lennard_jones, color='blue')
pyplot.title("Lennard Jones energy over time")
pyplot.xlabel("Time [ps]")
pyplot.ylabel("LJ energy [kJ/mol]")
pyplot.show()

# Plot the electrostatic energy
pyplot.plot(time, coulomb, color='purple')
pyplot.title("Electrostatic energy over time")
pyplot.xlabel("Time [ps]")
pyplot.ylabel("Coulomb energy [kJ/mol]")
pyplot.show()

# Plot the potential energy
pyplot.plot(time, potential, color='green')
pyplot.title("Potential energy over time")
pyplot.xlabel("Time [ps]")
pyplot.ylabel("Potential energy [kJ/mol]")
pyplot.show()
```

 **Questions**

* Can you plot the Coulomb energy and the potential energy, following the same steps as above?

* Is the total energy stable in this simulation?

* What is the dominant contribution to the potential energy?

##  4.3 Analysis of Protein

###  4.3.1 Root mean square deviation (RMSD) of 1HSG

The RMSD gives us an idea of how 'stable' our protein is when compared to our starting, static, structure. The lower the RMSD is, the more stable we can say our protein is.

The RMSD as a function of time, $\rho (t)$, can be defined by the following equation:

\begin{equation}
\\
\rho (t) = \sqrt{\frac{1}{N}\sum^N_{i=1}w_i\big(\mathbf{x}_i(t) - \mathbf{x}^{\text{ref}}_i\big)^2}
\end{equation}

Luckily MDAnalysis has its own built-in function to calculate this and we can import it.

```python
# Import built-in MDAnalysis tools for alignment and RMSD.
from MDAnalysis.analysis import align
from MDAnalysis.analysis.rms import RMSD as rmsd

# Define the simulation universe and the reference structure (protein structure at first frame)
protein = MDAnalysis.Universe(f"{path}/md.gro", f"{path}/md_fit.xtc")
protein_ref = MDAnalysis.Universe(f"{path}/em.gro", f"{path}/md_fit.xtc")
protein_ref.trajectory[0]

# Call the MDAnalysis align function to align the MD simulation universe to the reference (first frame) universe
align_strucs = align.AlignTraj(protein, protein_ref, select="backbone", weights="mass", in_memory=True, verbose=True)

R = align_strucs.run()
rmsd_data = R.rmsd

# Plot the RMSD
pyplot.plot(rmsd_data)
pyplot.title("RMSD over time")
pyplot.xlabel("Frame number")
pyplot.ylabel("RMSD (Angstrom)")
pyplot.show()

```

 **Questions**

* What does this tell you about the stability of the protein? Is it in a state of equilibrium and if so why and at what time?

* Can you think of a situation where this approach might not be a very good indication of stability?

###  4.3.2 Root mean square fluctuation (RMSF) of 1HSG

A similar property that is particularly useful is the root mean square fluctuation (RMSF), which shows how each residue flucuates over its average position.

The RMSF for an atom, $\rho_i$, is given by:

\begin{equation}
\rho_i = \sqrt{\sum^N_{i=1} \big\langle(\mathbf{x}_i - \langle \mathbf{x}_i \rangle )^2 \big\rangle }
\end{equation}

```python
from MDAnalysis.analysis.rms import RMSF as rmsf

# Define again the simulation universe, using however the renumbered .gro file that you had generated earlier
protein = MDAnalysis.Universe(f"{path}/em.gro", f"{path}/md_fit.xtc")

# Reset the trajectory to the first frame
protein.trajectory[0]

# We will need to select the alpha Carbons only
calphas = protein.select_atoms("name CA")

# Compute the RMSF of alpha carbons. Omit the first 20 frames,
# assuming that the system needs this amount of time (200 ps) to equilibrate
rmsf_calc = rmsf(calphas, verbose=True).run(start=20)

# Plot the RMSF
pyplot.plot(calphas.resindices+1, rmsf_calc.rmsf, color='darkorange' )
pyplot.title("Per-Residue Alpha Carbon RMSF")
pyplot.xlabel("Residue Number")
pyplot.ylabel("RMSF (Angstrom)")
pyplot.show()
```

 **Questions**

* Can you identify structural regions alone from this plot and does that fit in with the structure?

* Residues 43-58 form part of the flexible flap that covers the binding site. How does this region behave in the simulation?

###  4.3.3 Hydrogen Bond Formation

We can also use the simulation to monitor the formation of any hydrogen bonds that may be of interest.

In the case of HIV-1 protease, the hydrogen bonds (HB) that are formed between the ARG8', the ASP29 and the ARG87 amino acids at the interface of the two subunits act in stabilising the dimer.

We can analyse the trajectory and monitor the stability of these interactions *vs* simulation time.

```python
# Import the MDAnalysis built-in tool for HB Analysis
from MDAnalysis.analysis.hydrogenbonds.hbond_analysis import HydrogenBondAnalysis as HBA

# Define the protein universe
# Note that when using this tool, it is recommended to include the .tpr file instead of the .gro file,
# because it contains bond information, required for the identification of donors and acceptors.
protein = MDAnalysis.Universe(f"{path}/md.tpr", f"{path}/md.xtc")

# Define the atom selections for the HB calculation.
# In this case, the ARG hydrogens and the ASP oxygens, which act as the HB acceptors are specifically defined.
hbonds = HBA(universe=protein, hydrogens_sel='resname ARG and name HH21 HH22', acceptors_sel='resname ASP and name OD1 OD2')

# Perform the HB calculation
hbonds.run()

# Plot the total number of ASP-ARG HBs vs time
hbonds_time=hbonds.times
hbonds_data=hbonds.count_by_time()

pyplot.plot(hbonds_time, hbonds_data, color='darkorange')
pyplot.title("ASP-ARG Hydrogen Bonds")
pyplot.xlabel("Time [ps]")
pyplot.ylabel("# Hydrogen Bonds")
pyplot.show()

# Compute and print the average number of HBs and the standard deviation
aver_hbonds=round(np.mean(hbonds_data), 2)
std_hbonds=round(np.std(hbonds_data), 2)
print(f"The average number of ASP-ARG HBs is {aver_hbonds} ± {std_hbonds}")
```

 **Questions**

* How much variation is there in the number of hydrogen bonds?

* Do any break and not reform?

* Using VMD, can you observe the HB formation and breakage throughout the simulation?

***

This concludes the analysis section, but the aim was only to give you an idea of the numerous information that we can gain when analysing an MD trajectory. Feel free to ask and attempt to answer your own questions, utilising the tools that you were introduced to during the tutorial.

##  4.4 Further Reading

The texts recommended here are the same as those mentioned in the lecture:
* "Molecular Modelling. Principles and Applications". Andrew Leach. Publisher: Prentice Hall. ISBN: 0582382106. This book has rapidly become the defacto introductory text for all aspects of simulation.
* "Computer simulation of liquids". Allen, Michael P., and Dominic J. Tildesley. Oxford university press, 2017.
* "Molecular Dynamics Simulation: Elementary Methods". J.M. Haile. Publisher: Wiley. ISBN: 047118439X. This text provides a more focus but slightly more old-fashioned view of simulation. It has some nice simple examples of how to code (in fortran) some of the algorithms though.


```python
! pip install qeds
import pandas as pd
import numpy as np
!pip install pyarrow
import pyarrow.feather
```

    Requirement already satisfied: qeds in c:\users\asus\anaconda3\lib\site-packages (0.6.2)
    Requirement already satisfied: pyarrow in c:\users\asus\anaconda3\lib\site-packages (from qeds) (0.16.0)
    Requirement already satisfied: scipy in c:\users\asus\anaconda3\lib\site-packages (from qeds) (1.3.1)
    Requirement already satisfied: openpyxl in c:\users\asus\anaconda3\lib\site-packages (from qeds) (3.0.0)
    Requirement already satisfied: plotly in c:\users\asus\anaconda3\lib\site-packages (from qeds) (4.5.4)
    Requirement already satisfied: statsmodels in c:\users\asus\anaconda3\lib\site-packages (from qeds) (0.10.1)
    Requirement already satisfied: pandas-datareader in c:\users\asus\anaconda3\lib\site-packages (from qeds) (0.8.1)
    Requirement already satisfied: quandl in c:\users\asus\anaconda3\lib\site-packages (from qeds) (3.5.0)
    Requirement already satisfied: requests in c:\users\asus\anaconda3\lib\site-packages (from qeds) (2.22.0)
    Requirement already satisfied: seaborn in c:\users\asus\anaconda3\lib\site-packages (from qeds) (0.9.0)
    Requirement already satisfied: numpy in c:\users\asus\anaconda3\lib\site-packages (from qeds) (1.16.5)
    Requirement already satisfied: scikit-learn in c:\users\asus\anaconda3\lib\site-packages (from qeds) (0.21.3)
    Requirement already satisfied: quantecon in c:\users\asus\anaconda3\lib\site-packages (from qeds) (0.4.6)
    Requirement already satisfied: pandas in c:\users\asus\anaconda3\lib\site-packages (from qeds) (0.25.1)
    Requirement already satisfied: matplotlib in c:\users\asus\anaconda3\lib\site-packages (from qeds) (3.1.1)
    Requirement already satisfied: six>=1.0.0 in c:\users\asus\anaconda3\lib\site-packages (from pyarrow->qeds) (1.12.0)
    Requirement already satisfied: jdcal in c:\users\asus\anaconda3\lib\site-packages (from openpyxl->qeds) (1.4.1)
    Requirement already satisfied: et-xmlfile in c:\users\asus\anaconda3\lib\site-packages (from openpyxl->qeds) (1.0.1)
    Requirement already satisfied: retrying>=1.3.3 in c:\users\asus\anaconda3\lib\site-packages (from plotly->qeds) (1.3.3)
    Requirement already satisfied: patsy>=0.4.0 in c:\users\asus\anaconda3\lib\site-packages (from statsmodels->qeds) (0.5.1)
    Requirement already satisfied: lxml in c:\users\asus\anaconda3\lib\site-packages (from pandas-datareader->qeds) (4.4.1)
    Requirement already satisfied: python-dateutil in c:\users\asus\anaconda3\lib\site-packages (from quandl->qeds) (2.8.0)
    Requirement already satisfied: more-itertools in c:\users\asus\anaconda3\lib\site-packages (from quandl->qeds) (7.2.0)
    Requirement already satisfied: inflection>=0.3.1 in c:\users\asus\anaconda3\lib\site-packages (from quandl->qeds) (0.3.1)
    Requirement already satisfied: chardet<3.1.0,>=3.0.2 in c:\users\asus\anaconda3\lib\site-packages (from requests->qeds) (3.0.4)
    Requirement already satisfied: certifi>=2017.4.17 in c:\users\asus\anaconda3\lib\site-packages (from requests->qeds) (2019.9.11)
    Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in c:\users\asus\anaconda3\lib\site-packages (from requests->qeds) (1.24.2)
    Requirement already satisfied: idna<2.9,>=2.5 in c:\users\asus\anaconda3\lib\site-packages (from requests->qeds) (2.8)
    Requirement already satisfied: joblib>=0.11 in c:\users\asus\anaconda3\lib\site-packages (from scikit-learn->qeds) (0.13.2)
    Requirement already satisfied: sympy in c:\users\asus\anaconda3\lib\site-packages (from quantecon->qeds) (1.4)
    Requirement already satisfied: numba>=0.38 in c:\users\asus\anaconda3\lib\site-packages (from quantecon->qeds) (0.45.1)
    Requirement already satisfied: pytz>=2017.2 in c:\users\asus\anaconda3\lib\site-packages (from pandas->qeds) (2019.3)
    Requirement already satisfied: cycler>=0.10 in c:\users\asus\anaconda3\lib\site-packages (from matplotlib->qeds) (0.10.0)
    Requirement already satisfied: kiwisolver>=1.0.1 in c:\users\asus\anaconda3\lib\site-packages (from matplotlib->qeds) (1.1.0)
    Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in c:\users\asus\anaconda3\lib\site-packages (from matplotlib->qeds) (2.4.2)
    Requirement already satisfied: mpmath>=0.19 in c:\users\asus\anaconda3\lib\site-packages (from sympy->quantecon->qeds) (1.1.0)
    Requirement already satisfied: llvmlite>=0.29.0dev0 in c:\users\asus\anaconda3\lib\site-packages (from numba>=0.38->quantecon->qeds) (0.29.0)
    Requirement already satisfied: setuptools in c:\users\asus\anaconda3\lib\site-packages (from kiwisolver>=1.0.1->matplotlib->qeds) (41.4.0)
    Requirement already satisfied: pyarrow in c:\users\asus\anaconda3\lib\site-packages (0.16.0)
    Requirement already satisfied: six>=1.0.0 in c:\users\asus\anaconda3\lib\site-packages (from pyarrow) (1.12.0)
    Requirement already satisfied: numpy>=1.14 in c:\users\asus\anaconda3\lib\site-packages (from pyarrow) (1.16.5)

# Storage Formats - Exercises

## Exercise 1

**In the cell below, the variable `url` contains a web address to a csv file containing the result of all NFL games from September 1920 to February 2017.**

1. **Use `pd.read_csv` to read this file into a DataFrame named `nfl`.**

```python
url = "https://raw.githubusercontent.com/fivethirtyeight/nfl-elo-game/"
url = url + "3488b7d0b46c5f6583679bc40fb3a42d729abd39/data/nfl_games.csv"

nfl = pd.read_csv(url, index_col=0)
nfl
```

      season
      neutral
      playoff
      team1
      team2
      elo1
      elo2
      elo_prob1
      score1
      score2
      result1

      date

      1920-09-26
      1920
      0
      0
      RII
      STP
      1503.947000
      1300.000000
      0.824651
      48
      0
      1.0

      1920-10-03
      1920
      0
      0
      AKR
      WHE
      1503.420000
      1300.000000
      0.824212
      43
      0
      1.0

      1920-10-03
      1920
      0
      0
      RCH
      ABU
      1503.420000
      1300.000000
      0.824212
      10
      0
      1.0

      1920-10-03
      1920
      0
      0
      DAY
      COL
      1493.002000
      1504.908000
      0.575819
      14
      0
      1.0

      1920-10-03
      1920
      0
      0
      RII
      MUN
      1516.108000
      1478.004000
      0.644171
      45
      0
      1.0

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      2017-01-15
      2016
      0
      1
      DAL
      GB
      1617.794683
      1635.451172
      0.567714
      31
      34
      0.0

      2017-01-15
      2016
      0
      1
      KC
      PIT
      1681.926463
      1647.734179
      0.638993
      16
      18
      0.0

      2017-01-22
      2016
      0
      1
      ATL
      GB
      1664.127266
      1651.537731
      0.609840
      44
      21
      1.0

      2017-01-22
      2016
      0
      1
      NE
      PIT
      1747.160321
      1662.437215
      0.703052
      36
      17
      1.0

      2017-02-05
      2016
      1
      1
      ATL
      NE
      1688.081418
      1763.818168
      0.392700
      28
      34
      0.0

15740 rows × 11 columns

2. **Print the shape and column names of `nfl`.**

```python
print("nfl.shape=",nfl.shape)
print(nfl.columns)
```

    nfl.shape= (15740, 11)
    Index(['season', 'neutral', 'playoff', 'team1', 'team2', 'elo1', 'elo2',
           'elo_prob1', 'score1', 'score2', 'result1'],
          dtype='object')

3. **Save the DataFrame to a file named `nfl.xlsx`**

```python
nfl.to_excel("nfl.xlsx", "nfl")
```

4. **Open the spreadsheet using Excel on your computer.**

## Exercise 2

1. **Compute the average total points in each game (note, you will need to sum two of the columns to get total points).**

```python
#we compute the total points as score1 + score 2, then we take the average
nfl["total_points"] = nfl["score1"]+nfl["score2"]
nfl["total_points"].mean()
```

    39.95368487928844

2. **Repeat the above calculation, but only for playoff games.**

```python
#here we compute the average total points for the subset corresponding to playoff games
nfl["total_points"][nfl["playoff"]==1].mean()
```

    42.70404411764706

3. **Compute the average score for your favorite team (you’ll need to consider when they were team1 vs team2).**

```python
#I don't have a smarter solution

#in order to get the average for AKR:
    #I sum the total points obtained by AKR either as team1 or team2
    #I sum the total number of games of AKR either as team1 or team2
    #then I divide the total points by the total number of games

sum_teams12   = nfl["total_points"][nfl["team1"]=="AKR"].sum()   + nfl["total_points"][nfl["team2"]=="AKR"].sum()
count_teams12 = nfl["total_points"][nfl["team1"]=="AKR"].count() + nfl["total_points"][nfl["team2"]=="AKR"].count()
average = sum_teams12/count_teams12
average
```

    17.125

4. **Compute the ratio of “upsets” to total games played. An upset is defined as a team with a lower ELO winning the game.**

```python
#for all the matches
nfl.loc[((nfl['elo1']>nfl['elo2']) & (nfl['result1']==0)) | ((nfl['elo2']>nfl['elo1']) & (nfl['result1']==1))  , 'Upset?' ] =1
nfl.loc[(nfl['Upset?']!=1), 'Upset?']=0
nfl['Upset?'].mean()
```

    0.34358322744599745

```python
#only for the matches of AKR (as team1)
nfl.loc[((nfl['elo1']>nfl['elo2']) & (nfl['result1']==0)) | ((nfl['elo2']>nfl['elo1']) & (nfl['result1']==1))  , 'Upset?' ] =1
nfl.loc[(nfl['Upset?']!=1), 'Upset?']=0
nfl['Upset?'][nfl["team1"] == "AKR"].mean()
```

    0.21739130434782608