```python
print("Hello world")
```

    Hello world

### My First Notebook

```python
# simple code
2+2
```

    4

```python
# variable
x=3
```

```python
print(x)
```

    3

```python
# used keyboard short cut ECS + L  to display line numbers

```

```python
# use ! to run shell commands from jupiter

! python --version
```

    Python 3.6.5 :: Anaconda, Inc.

### Magic functions

##### magic function: matplotlib

```python
%matplotlib inline
```

```python
import matplotlib.pyplot as plt
plt.plot(range(100))
```

##### magic function: time

```python
%time x= range(10000)
```

    Wall time: 1.01 ms

##### magic function: timeit

```python
%%timeit x = range(10000)
max(x)
```

    347 µs ± 95.7 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

##### magic function: writefile

```python
%%writefile test.txt
This is written from jupiter notebook
A new file is created if no file exists with name test.txt
```

    Writing test.txt

```python
%%writefile test.txt
overwriting to test.txt 2
```

    Overwriting test.txt

##### magic function: ls

```python
%ls
```

     Volume in drive C is Windows
     Volume Serial Number is 7E28-0B6E

     Directory of C:\Users\Sunit\Documents\GitHub\Titanic

    10/22/2018  09:59 PM    <DIR>          .
    10/22/2018  09:59 PM    <DIR>          ..
    10/22/2018  09:39 PM    <DIR>          .ipynb_checkpoints
    10/22/2018  09:59 PM            16,314 HelloWorld.ipynb
    10/22/2018  09:31 PM             1,092 LICENSE
    10/22/2018  09:53 PM                23 test.txt
                   3 File(s)         17,429 bytes
                   3 Dir(s)  176,694,788,096 bytes free

##### magic function: HTML

```python
%%HTML
<i> Image in jupyter notebook </i>
</img>
```

 Image in jupyter notebook

##### magic function: latex

```latex
%%latex
\begin{align}
Gradient :  \nabla J = -2H^T (Y-HW)
\end{align}
```

\begin{align}
Gradient :  \nabla J = -2H^T (Y-HW)
\end{align}

##### magic function: loat_ext

```python
! pip install ipython-sql
```

    Requirement already satisfied: ipython-sql in c:\programdata\anaconda3\lib\site-packages (0.3.9)
    Requirement already satisfied: prettytable in c:\programdata\anaconda3\lib\site-packages (from ipython-sql) (0.7.2)
    Requirement already satisfied: ipython-genutils>=0.1.0 in c:\programdata\anaconda3\lib\site-packages (from ipython-sql) (0.2.0)
    Requirement already satisfied: sqlalchemy>=0.6.7 in c:\programdata\anaconda3\lib\site-packages (from ipython-sql) (1.2.7)
    Requirement already satisfied: sqlparse in c:\programdata\anaconda3\lib\site-packages (from ipython-sql) (0.2.4)
    Requirement already satisfied: ipython>=1.0 in c:\programdata\anaconda3\lib\site-packages (from ipython-sql) (6.4.0)
    Requirement already satisfied: six in c:\programdata\anaconda3\lib\site-packages (from ipython-sql) (1.11.0)
    Requirement already satisfied: simplegeneric>0.8 in c:\programdata\anaconda3\lib\site-packages (from ipython>=1.0->ipython-sql) (0.8.1)
    Requirement already satisfied: pickleshare in c:\programdata\anaconda3\lib\site-packages (from ipython>=1.0->ipython-sql) (0.7.4)
    Requirement already satisfied: pygments in c:\programdata\anaconda3\lib\site-packages (from ipython>=1.0->ipython-sql) (2.2.0)
    Requirement already satisfied: decorator in c:\programdata\anaconda3\lib\site-packages (from ipython>=1.0->ipython-sql) (4.3.0)
    Requirement already satisfied: setuptools>=18.5 in c:\programdata\anaconda3\lib\site-packages (from ipython>=1.0->ipython-sql) (39.1.0)
    Requirement already satisfied: prompt-toolkit<2.0.0,>=1.0.15 in c:\programdata\anaconda3\lib\site-packages (from ipython>=1.0->ipython-sql) (1.0.15)
    Requirement already satisfied: backcall in c:\programdata\anaconda3\lib\site-packages (from ipython>=1.0->ipython-sql) (0.1.0)
    Requirement already satisfied: colorama; sys_platform == "win32" in c:\programdata\anaconda3\lib\site-packages (from ipython>=1.0->ipython-sql) (0.3.9)
    Requirement already satisfied: jedi>=0.10 in c:\programdata\anaconda3\lib\site-packages (from ipython>=1.0->ipython-sql) (0.12.0)
    Requirement already satisfied: traitlets>=4.2 in c:\programdata\anaconda3\lib\site-packages (from ipython>=1.0->ipython-sql) (4.3.2)
    Requirement already satisfied: wcwidth in c:\programdata\anaconda3\lib\site-packages (from prompt-toolkit<2.0.0,>=1.0.15->ipython>=1.0->ipython-sql) (0.1.7)
    Requirement already satisfied: parso>=0.2.0 in c:\programdata\anaconda3\lib\site-packages (from jedi>=0.10->ipython>=1.0->ipython-sql) (0.2.0)

    launcher build: 64bit
    launcher executable: Console
    File 'C:\Users\Sunit\AppData\Local\py.ini' non-existent
    File 'C:\ProgramData\Anaconda3\Scripts\py.ini' non-existent
    Called with command line: install ipython-sql
    Using wrapped script file 'C:\ProgramData\Anaconda3\Scripts\pip-script.py'
    Running wrapped script with command line 'C:\ProgramData\Anaconda3\Scripts\pip-script.py install ipython-sql'
    maybe_handle_shebang: read 207 bytes
    maybe_handle_shebang: BOM not found, using UTF-8
    parse_shebang called without a valid shebang  (for argv0 C:\ProgramData\Anaconda3\Scripts\pip-script.py)
    invented shebang: #!C:\ProgramData\Anaconda3\python.exe
    parse_shebang: found command: C:\ProgramData\Anaconda3\python.exe
    run_child: about to run 'C:\ProgramData\Anaconda3\python.exe C:\ProgramData\Anaconda3\Scripts\pip-script.py install ipython-sql'
    child process exit code: 0

```python
%%!
pip install ipython-sql
```

    ['launcher build: 64bit',
     'launcher executable: Console',
     "File 'C:\\Users\\Sunit\\AppData\\Local\\py.ini' non-existent",
     "File 'C:\\ProgramData\\Anaconda3\\Scripts\\py.ini' non-existent",
     'Called with command line: install ipython-sql',
     "Using wrapped script file 'C:\\ProgramData\\Anaconda3\\Scripts\\pip-script.py'",
     "Running wrapped script with command line 'C:\\ProgramData\\Anaconda3\\Scripts\\pip-script.py install ipython-sql'",
     'maybe_handle_shebang: read 207 bytes',
     'maybe_handle_shebang: BOM not found, using UTF-8',
     'parse_shebang called without a valid shebang  (for argv0 C:\\ProgramData\\Anaconda3\\Scripts\\pip-script.py)',
     'invented shebang: #!C:\\ProgramData\\Anaconda3\\python.exe',
     'parse_shebang: found command: C:\\ProgramData\\Anaconda3\\python.exe',
     "run_child: about to run 'C:\\ProgramData\\Anaconda3\\python.exe C:\\ProgramData\\Anaconda3\\Scripts\\pip-script.py install ipython-sql'",
     'Requirement already satisfied: ipython-sql in c:\\programdata\\anaconda3\\lib\\site-packages (0.3.9)',
     'Requirement already satisfied: ipython>=1.0 in c:\\programdata\\anaconda3\\lib\\site-packages (from ipython-sql) (6.4.0)',
     'Requirement already satisfied: sqlparse in c:\\programdata\\anaconda3\\lib\\site-packages (from ipython-sql) (0.2.4)',
     'Requirement already satisfied: six in c:\\programdata\\anaconda3\\lib\\site-packages (from ipython-sql) (1.11.0)',
     'Requirement already satisfied: ipython-genutils>=0.1.0 in c:\\programdata\\anaconda3\\lib\\site-packages (from ipython-sql) (0.2.0)',
     'Requirement already satisfied: sqlalchemy>=0.6.7 in c:\\programdata\\anaconda3\\lib\\site-packages (from ipython-sql) (1.2.7)',
     'Requirement already satisfied: prettytable in c:\\programdata\\anaconda3\\lib\\site-packages (from ipython-sql) (0.7.2)',
     'Requirement already satisfied: pickleshare in c:\\programdata\\anaconda3\\lib\\site-packages (from ipython>=1.0->ipython-sql) (0.7.4)',
     'Requirement already satisfied: prompt-toolkit<2.0.0,>=1.0.15 in c:\\programdata\\anaconda3\\lib\\site-packages (from ipython>=1.0->ipython-sql) (1.0.15)',
     'Requirement already satisfied: decorator in c:\\programdata\\anaconda3\\lib\\site-packages (from ipython>=1.0->ipython-sql) (4.3.0)',
     'Requirement already satisfied: pygments in c:\\programdata\\anaconda3\\lib\\site-packages (from ipython>=1.0->ipython-sql) (2.2.0)',
     'Requirement already satisfied: colorama; sys_platform == "win32" in c:\\programdata\\anaconda3\\lib\\site-packages (from ipython>=1.0->ipython-sql) (0.3.9)',
     'Requirement already satisfied: jedi>=0.10 in c:\\programdata\\anaconda3\\lib\\site-packages (from ipython>=1.0->ipython-sql) (0.12.0)',
     'Requirement already satisfied: traitlets>=4.2 in c:\\programdata\\anaconda3\\lib\\site-packages (from ipython>=1.0->ipython-sql) (4.3.2)',
     'Requirement already satisfied: setuptools>=18.5 in c:\\programdata\\anaconda3\\lib\\site-packages (from ipython>=1.0->ipython-sql) (39.1.0)',
     'Requirement already satisfied: backcall in c:\\programdata\\anaconda3\\lib\\site-packages (from ipython>=1.0->ipython-sql) (0.1.0)',
     'Requirement already satisfied: simplegeneric>0.8 in c:\\programdata\\anaconda3\\lib\\site-packages (from ipython>=1.0->ipython-sql) (0.8.1)',
     'Requirement already satisfied: wcwidth in c:\\programdata\\anaconda3\\lib\\site-packages (from prompt-toolkit<2.0.0,>=1.0.15->ipython>=1.0->ipython-sql) (0.1.7)',
     'Requirement already satisfied: parso>=0.2.0 in c:\\programdata\\anaconda3\\lib\\site-packages (from jedi>=0.10->ipython>=1.0->ipython-sql) (0.2.0)',
     'child process exit code: 0']

```python
%load_ext sql
```

```python
%sql sqlite://
```

    'Connected: @None'

```sql
%%sql
CREATE TABLE classroom(name, age, totalmarks);
INSERT INTO classroom VALUES ('Sunith',20,89);
INSERT INTO classroom VALUES ('Sid',07,95);

```

     * sqlite://
    Done.
    1 rows affected.
    1 rows affected.

    []

```python
%sql SELECT * FROM classroom
```

     * sqlite://
    Done.

        name
        age
        totalmarks

        Sunith
        20
        89

        Sid
        7
        95

##### magic function: lsmagic

```python
%lsmagic
```

    Available line magics:
    %alias  %alias_magic  %autocall  %automagic  %autosave  %bookmark  %cd  %clear  %cls  %colors  %config  %connect_info  %copy  %ddir  %debug  %dhist  %dirs  %doctest_mode  %echo  %ed  %edit  %env  %gui  %hist  %history  %killbgscripts  %ldir  %less  %load  %load_ext  %loadpy  %logoff  %logon  %logstart  %logstate  %logstop  %ls  %lsmagic  %macro  %magic  %matplotlib  %mkdir  %more  %notebook  %page  %pastebin  %pdb  %pdef  %pdoc  %pfile  %pinfo  %pinfo2  %popd  %pprint  %precision  %profile  %prun  %psearch  %psource  %pushd  %pwd  %pycat  %pylab  %qtconsole  %quickref  %recall  %rehashx  %reload_ext  %ren  %rep  %rerun  %reset  %reset_selective  %rmdir  %run  %save  %sc  %set_env  %sql  %store  %sx  %system  %tb  %time  %timeit  %unalias  %unload_ext  %who  %who_ls  %whos  %xdel  %xmode

    Available cell magics:
    %%!  %%HTML  %%SVG  %%bash  %%capture  %%cmd  %%debug  %%file  %%html  %%javascript  %%js  %%latex  %%markdown  %%perl  %%prun  %%pypy  %%python  %%python2  %%python3  %%ruby  %%script  %%sh  %%sql  %%svg  %%sx  %%system  %%time  %%timeit  %%writefile

    Automagic is ON, % prefix IS NOT needed for line magics.

#### Extracting data from relation database

##### SQLite Database

###### import SQLite

```python
import sqlite3
```

###### connect to DB

```python
connection = sqlite3.connect("classroomDB.db")
connection.close()
```

###### create table

```python
connection = sqlite3.connect("classroomDB.db")
# open cursor
cursor=connection.cursor()
# query for creating table
create_table= """
                CREATE TABLE classroom(
                student_id INTEGER PRIMARY KEY,
                name VARCHAR(20),
                gender CHAR(1),
                mark INTEGER
                );"""
# execute query
cursor.execute(create_table)
# commit change
connection.commit()
# close connection
connection.close()
```

###### insert data

```python
classroom=[(1,"Sunith","M",92),
        (2,"Varsha","F",93),
        (3,"Jyotika","F",92),
        (4,"Sid","M",95)]

# open connection
connection = sqlite3.connect("classroomDB.db")
# open cursor
cursor=connection.cursor()
#insert each student record
for student in classroom:
    insert_statement="""INSERT INTO classroom
                    (student_id , name, gender, mark)
                    VALUES
                    ({0},"{1}","{2}",{3});""".format(student[0],student[1],student[2],student[3])
    #execute insert query
    cursor.execute(insert_statement)

# commit change
connection.commit()
# close connection
connection.close()
```

###### read data

```python
# open connection
connection = sqlite3.connect("classroomDB.db")
# open cursor
cursor=connection.cursor()

select_query = "SELECT * FROM classroom"
results = cursor.execute(select_query)

for student in results:
    print(student)

# close connection
connection.close()

```

    (1, 'Sunith', 'M', 92)
    (2, 'Varsha', 'F', 93)
    (3, 'Jyotika', 'F', 92)
    (4, 'Sid', 'M', 95)

```python

```


```python
from IPython.html.services.config import ConfigManager
from IPython.utils.path import locate_profile
cm = ConfigManager(profile_dir=locate_profile(get_ipython().profile))
cm.update('livereveal', {
              'theme': 'sky',
              'transition': 'zoom',
              'start_slideshow_at': 'selected',
})
```

    {u'start_slideshow_at': 'selected', u'theme': 'sky', u'transition': 'zoom'}

#Lecture 10. Integral equations: discretization

## Previous lecture
- Introduction into the integral equations: the basic one, the concept of non-locality
- Exterior and interior problems

## Todays lecture
- How to discretize: method of moments (MoM) and the Galerkin method
- Computation of the singular integrals and exponential sums
- Fast methods: the idea of the Barnes-Hut method

## Books
Books on integral equations

- [Integral equation methods in scattering theory, D. Colton R. Kress](https://www.google.com/books?hl=ru&lr=&id=Z9qmAgAAQBAJ&oi=fnd&pg=PR1&dq=Colton+Kress+integral+equation&ots=0H3pcRZBUs&sig=0RzExxkaSor_Sv9Ss7HIX3cd864)
- Sauter, Schwab, Boundary element methods (2011)

## Method of moments (collocation method)
Recall the basic integral equation from electrostatics:

$$\int_{\Omega} \frac{q(y) dy}{\Vert x - y \Vert} = f(x).$$

1. First we mesh the domain in simplices.
2. Then, we introduce the discrete **solution space** (piecewise-linear, piecewise-constant functions).
3. Then, we approximate the solution as a linear combination of the basis functions
  $$ q(y) = \sum_i q_i \phi_i(y).$$

## MoM (continued)
Then, we get the system of equations
$$
     f(x) = \sum_i q_i \int_{\Pi_i} \frac{\phi_i}{\Vert x - y \Vert } dy.
$$
The  method of moments  is obtained by the selection of
 collocation points (aka staggerred grids), typically in the middle of triangles/rectangles

## Piecewise-constant basis functions

For a piecewise constant basis functions, we have

$$\int_{\Pi} \frac{1}{\Vert x - y \Vert} dy, $$

where this is a two-dimensional integral.

## Computing the simplest integral
We can actually compute the anti-derivative of the integral in question, but taking successive integrals

$$
   \frac{1}{\sqrt{x^2 + y^2}}
$$

\begin{equation}
\begin{split}
F(x, y) = \int \int \frac{1}{\sqrt{x^2 + y^2}} dx dy = \\
-y +
y \log(x + \sqrt{x^2 + y^2}) + x \log(y + \sqrt{x^2 + y^2}),
\end{split}
\end{equation}

and the final integral is just
$$
   I(x_0, y_0, x_1, y_1) = F(x_0, y_0) + F(x_1, y_1) - F(x_0, y_1) - F(x_1, y_0),
$$

(generalization of the Newton-Leibniz formula)

```python
import sympy
from sympy import Symbol, sqrt
x = Symbol('x', real=True)
a = Symbol('a', real=True)
y = Symbol('y', real=True)
b = Symbol('b', real=True)
r = sqrt(x ** 2 + y ** 2)
sympy.init_printing()
z = sympy.integrate(1/r, x, conds='none')
print z
```

## Nystrom method
A Nystrom method is a popular choice: replace the integral by  quadrature

probably by the simplest one: the rectangle formula,

which in 1D has the form

$$I(f) = \sum_{k=0}^n w_k f(x_k), $$

where $x_k$ is the uniform grid on the interval, $w_0 = w_n = h/2, \quad w_k = h$ for all other $k$.

## For our equation, it looks like

  $$A_{ij} = \frac{1} {\Vert x_i - y_j \Vert} h^2,$$

  the set $x_i$ can be called  receivers  and the set $y_j$ can be called  sources ;

  A typical way is to shift the grid by half-step, i.e. in 1D it will be

  $$
  A_{ij} = \frac{1}{|i-j| + \frac{1}{2}}.
  $$
  By the way, do you remember, what is the rank of this matrix?

## Galerkin method
1. Collocation method gives non-symmetric matrices, even if the original operator was symmetric;
2. Nystrom method adds logarithmic factor into the accuracy estimate
3. Galerkin method is the method of choice in many cases

## Galerkin method
A Galerkin method is defined for an arbitrary linear (and even non-linear) equations

$$
    A u = f,
$$

you select the  test space  and  trial space .

For simplicity, let them be equal,

i.e.
$$
   u = \sum_i u_i \phi_i,
$$

and

$$A_{ij} = (A \phi_i, \phi_j) $$

## Integrals in the Galerkin method
We have to compute 4D integrals (here it is written for piecewise-constant functions)

$$
  A_{ij} = \int_{\Omega_i} \int_{\Omega_j}\frac{1}{\Vert x - y \Vert} dx dy
$$
Analytic expression are cumbersome!

## Semi-numerical method
You can be **a little bit** smarter: approximate $1/r$ by something that can be well-integrated.

By a sum of separable functions!

$$\frac{1}{\Vert x - y\Vert } \approx \sum_{\alpha=1}^r u_{\alpha}(x) u_{\alpha}(y)$$

We have used the symmetry.

But how to compute these functions $u_{\alpha}$?  (why it is useful?)

## Exponential sums

There is an identity:

$$\frac{1}{\sqrt{x}} = \frac{1}{\sqrt{\pi}} \int^{\infty}_0 \frac{e^{-px}}{\sqrt{p}} dp.$$

It can be very useful in separating variables.

## A quadrature

Suppose we have a quadrature for the integral:

$$ \frac{1}{\sqrt{x}} = \frac{1}{\sqrt{\pi}} \int^{\infty}_0 \frac{e^{-px}}{\sqrt{p}} dp. \approx \sum_{k} w_k e^{-p_k x}.$$

Then put

$$x := x^2 + y^2$$

and we get the sum of  Gaussians  on the right.

## Sum of Gaussians

$$\frac{1}{r} \approx \sum_{k} w_k e^{-p_k (x^2 + y^2)},$$

and the integrals reduce to the 1D integrals

$$
   \int_a^b e^{-p_k x^2} dx,
$$

and this can be expressed via the  Error functions  (and there is a standard numpy erfc function)

## But who you can approximate?

We reduced the problem to the problem of creating a good quadrature rule for the integral

$$\int^{\infty}_0 \frac{e^{-px}}{\sqrt{p}} dp$$

You have to do the $p = e^t$ (why do we do so?) and we the new integral

$$
\int^{\infty}_{-\infty} e^{-e^{t} x + t/2} dt
$$

and this integral is then approximate by **trapezoidal rule** $t_k = a_t + k h_t, \quad h_t = (b_t - a_t)/K$

The integral is doubly exponentially decaying, and also its Fourier transform is decaying exponentially fast.

Thus, trapezoidal rule has exponential convergence in this setting

A short demo...

```python
import numpy as np
import math
import matplotlib.pyplot as plt
import seaborn as sns
sns.set(style='ticks', palette='Set2')
sns.despine()
%matplotlib inline
a = -10
b = 20
n = 70
t = np.linspace(a, b, n)
h = t[1] - t[0]
w = np.ones(n) * h
w[0] = h/2
w[n-1] = h/2
w = w * np.exp(0.5 * t)/np.sqrt(math.pi)
ps = np.exp(t)
x = np.linspace(0.1, 1)
fun = 1.0/np.sqrt(x)
appr = 0 * x
for i in xrange(n):
    appr = appr + w[i] * np.exp(-(x) * ps[i])
#plt.plot(x, fun)
ax = plt.subplot(1, 1, 1)
ax.plot(x, appr - fun)
#ax.set_title('Approximation error by %d Gaussians' % n)
ax.set_title('Error')
plt.tight_layout(.8)
```

## A sidenote about fitting by sum of exponentials
- Approximation of a given function by a sum of exponentials is a ill-posed problem
- Original method by Prony (1795)
- Recent work by [G. Beylkin, et. al, Approximation by exponential sums revisited](https://www.google.ru/url?sa=t&rct=j&q=&esrc=s&source=web&cd=2&cad=rja&uact=8&ved=0CCYQFjAB&url=http%3A%2F%2Famath.colorado.edu%2Fpub%2Fwavelets%2Fpapers%2FBEY-MON-2010.pdf&ei=Cok1VZHkJKLnygO7hYCgBw&usg=AFQjCNGRcxece5lCuy9N7oheTBOzOpcSYw&sig2=Rf3DPGBAMmYO4parmQXzFQ&bvm=bv.91071109,d.bGQ)

## Efficient quadratures (papers)
- C. Schwab,  S. Sauter [Efficient automatic quadrature in 3-d Galerkin BEM](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.45.4312&rep=rep1&type=pdf)
- A. Polimeridis, Traianos V Yioultsis [On the direct evaluation of weakly singular integrals in Galerkin mixed potential integral equation formulations](http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=4618709)

## Green functions and fundamental solutions

Terminology:
- A fundamental solution is a "point charge" in free space: no boundary conditions
- Green function can be for the boundary conditions/layered media/etc. (i.e. "inverse" of the differential operator)

## Fundamental solution: Helmholtz
The Helmholtz equation has the form

$$\Delta u + k^2 u = f,$$

and it describes the propagation of  scalar waves  (acoustic wave).

$k$ is called **wavenumber**.

The fundamental solution (the one that gives $\delta$-function after the application of the operator)  has the form

$$K(r) = \frac{\exp(i k r)}{r}.$$

So it oscillates.

## Laplace in 2D

What is the fundamental solution to the Laplace equation in 2D?

$$G(x, y) = \log \Vert x - y \Vert.$$

It plays a role in the theory of  conformal mappings

Helmholtz in 2D is solved by the  Hankel function

## Stokes problem
Stokes equation define the laminar flow in low Reynolds number regime:

\begin{equation}
\begin{split}
- \nabla p + \mu \Delta v = -F \delta(r) \\
 \nabla \cdot p = 0
\end{split}
\end{equation}

## Maxwell equations

$$
\begin{split}
\nabla \times H = \frac{\partial D}{\partial t} + J, \\\quad \nabla \times E = -\frac{\partial B}{\partial t}, \quad \nabla \cdot D =  \rho, \quad \nabla \cdot B = 0. \\
D = \varepsilon E, \quad H = \mu B.
\end{split}
$$

## How to make it fast

The memory is $\mathcal{O}(N^2)$, the complexity is $\mathcal{O}(N^3)$.

Need to make it $\mathcal{O}(N \log N)$ both in memory and complexity of matrix-by-vector product.

## Matrix structure
The integral operator matrix:

1. Matrix is dense
2. Matrix is not of low-rank
3. Some blocks are of low rank!

A demo

```python
import numpy as np
import matplotlib.pyplot as plt
n = 128
a = [[1.0/(abs(i - j) + 0.5) for i in xrange(n)] for j in xrange(n)]
a = np.array(a)

plt.plot(np.linalg.svd(a)[1])
```

```python
#And for a block:
plt.semilogy(np.linalg.svd(a[:n/2, n/2:])[1])
```

## Off-diagonal blocks correspond to "far" interaction

$$y_i = \sum_{j} a_{ij} q_j,$$

and the sources are separated from the receivers.

This gives a hint: separated geometrically -- low-rank property.

## Geometric separability and low rank

For $\frac{1}{r}$ we can have the exponential sums to separate variables, if $r > r_0$.

What happens, if $r_0$ goes to zero?

The separability property vanishes!

## One-dimensional case

$$A = \begin{bmatrix} A_{11} & A_{12} \\ A_{21} & A_{22}\end{bmatrix},$$

where $A_{12}$ and $A_{21}$ correspond to the "far" interaction, and are of low-rank.

Remember, that a low-rank matrix can be factored as

$$A = U V^{\top},$$

where $U$ is $n \times r$, $V$ is $m \times r$,

## Memory estimate

Thus we have

$$Memory(n) = 2 Memory(n/2) + 2 n/2 r$$

and that gives

$Memory(n) = 2 n/2 * r + 4 n/4 * r + \ldots = \mathcal{O}( n \log n)$

Similar estimates hold for the matrix-by-vector product procedure (why do we care about that?)

## Two-dimensional case
In two dimension, the splitting is different.

```python
from numba import jit
n = 32
t = np.linspace(0, 1, n)
h = t[1] - t[0]
x_src, y_src  = np.meshgrid(t, t)
x_src, y_src = x_src.flatten(), y_src.flatten()
x_rec, y_rec = x_src + h * 0.5, y_src + 0.5 * h
N = n * n
mat = np.zeros((N, N))
@jit(nopython=True)
def compute_mat(mat, x_src, y_src, x_rec, y_rec):
    for i in range(N):
        for j in xrange(N):
            r = (x_src[i] - x_rec[j]) ** 2 + (y_src[i] - y_rec[j]) ** 2
            mat[i, j] = 1.0/np.sqrt(r)

%timeit compute_mat(mat, x_src, y_src, x_rec, y_rec)
#(x_rec - x_src)/h
```

    The slowest run took 20.23 times longer than the fastest. This could mean that an intermediate result is being cached
    100 loops, best of 3: 9.62 ms per loop

Plotting the singular values of the off-diagonal block shows that they do not decay that fast.

Why? Because they correspond to the non-separated sets of sources and receivers.

```python
plt.semilogy(np.linalg.svd(mat[:N/2, N/2:])[1])
```

## The general scheme
- Construct **trees** for the sources and receivers
- Check the sets for geometric separability.
- If they are not separated, call the recursion
- Otherwise, do low-rank approximation.

This is the basis for the **Barnes-Hut** method (original work in astronomy)

and the preliminary approach to the  Fast Multipole Method , proposed by Greengard and Rokhlin

```python
from IPython.core.display import HTML
def css_styling():
    styles = open("./styles/alex.css", "r").read()
    return HTML(styles)
css_styling()
```

<link href='http://fonts.googleapis.com/css?family=Fenix' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>

```python

```


```python
import kamodo
from readers.gitm.gitmK import GITM

import glob
import numpy as np
from plotly.offline import init_notebook_mode, iplot, plot
import plotly.io as pio
from kaleido.scopes.plotly import PlotlyScope
init_notebook_mode(connected = True)
```

```python
# Select the data file
runpath = "../../../data/"
#runname = "Sophia_Schwalbe_062220_IT_4"
#runname = "Aaron_Ridley_081017_IT_2"
runname = "GDC_GITM"
files = glob.glob(runpath+runname+'/*.bin')
files.sort()
file = files[-1]

model = GITM(file, runpath=runpath, runname=runname, debug=1)
model
```

    opening ../../../data/GDC_GITM/3DALL_t980106_002900.bin
    ... simulation time =  1998-01-06 00:29:00.000539
    ... raw data array size =  (112, 184, 54)
    ... range of altitudes is  96880.16186492516  to  567345.7542485126  meters.
    ... GITM code version  4.03
    Time loading file and kamodifying results: 2.6889 seconds

\begin{equation}\rho{\left(\vec{x} \right)} [kg/m^3] = \lambda{\left(\vec{x} \right)}\end{equation}\begin{equation}\operatorname{O3P}{\left(\vec{x} \right)} [kg/m^3] = \lambda{\left(\vec{x} \right)}\end{equation}\begin{equation}\operatorname{O_{2}}{\left(\vec{x} \right)} [kg/m^3] = \lambda{\left(\vec{x} \right)}\end{equation}\begin{equation}\operatorname{N_{2}}{\left(\vec{x} \right)} [kg/m^3] = \lambda{\left(\vec{x} \right)}\end{equation}\begin{equation}\operatorname{N4S}{\left(\vec{x} \right)} [kg/m^3] = \lambda{\left(\vec{x} \right)}\end{equation}\begin{equation}\operatorname{NO}{\left(\vec{x} \right)} [kg/m^3] = \lambda{\left(\vec{x} \right)}\end{equation}\begin{equation}\operatorname{He}{\left(\vec{x} \right)} [kg/m^3] = \lambda{\left(\vec{x} \right)}\end{equation}\begin{equation}\operatorname{N2D}{\left(\vec{x} \right)} [kg/m^3] = \lambda{\left(\vec{x} \right)}\end{equation}\begin{equation}\operatorname{N2P}{\left(\vec{x} \right)} [kg/m^3] = \lambda{\left(\vec{x} \right)}\end{equation}\begin{equation}H{\left(\vec{x} \right)} [kg/m^3] = \lambda{\left(\vec{x} \right)}\end{equation}\begin{equation}\operatorname{CO_{2}}{\left(\vec{x} \right)} [kg/m^3] = \lambda{\left(\vec{x} \right)}\end{equation}\begin{equation}\operatorname{O1D}{\left(\vec{x} \right)} [kg/m^3] = \lambda{\left(\vec{x} \right)}\end{equation}\begin{equation}\operatorname{Tn}{\left(\vec{x} \right)} [K] = \lambda{\left(\vec{x} \right)}\end{equation}\begin{equation}\operatorname{Vn_{east}}{\left(\vec{x} \right)} [m/s] = \lambda{\left(\vec{x} \right)}\end{equation}\begin{equation}\operatorname{Vn_{north}}{\left(\vec{x} \right)} [m/s] = \lambda{\left(\vec{x} \right)}\end{equation}\begin{equation}\operatorname{Vn_{up}}{\left(\vec{x} \right)} [m/s] = \lambda{\left(\vec{x} \right)}\end{equation}\begin{equation}\operatorname{Vn_{upO3P}}{\left(\vec{x} \right)} [m/s] = \lambda{\left(\vec{x} \right)}\end{equation}\begin{equation}\operatorname{Vn_{upO2}}{\left(\vec{x} \right)} [m/s] = \lambda{\left(\vec{x} \right)}\end{equation}\begin{equation}\operatorname{Vn_{upN2}}{\left(\vec{x} \right)} [m/s] = \lambda{\left(\vec{x} \right)}\end{equation}\begin{equation}\operatorname{Vn_{upN4S}}{\left(\vec{x} \right)} [m/s] = \lambda{\left(\vec{x} \right)}\end{equation}\begin{equation}\operatorname{Vn_{upNO}}{\left(\vec{x} \right)} [m/s] = \lambda{\left(\vec{x} \right)}\end{equation}\begin{equation}\operatorname{Vn_{upHe}}{\left(\vec{x} \right)} [m/s] = \lambda{\left(\vec{x} \right)}\end{equation}\begin{equation}\operatorname{O4SP+}{\left(\vec{x} \right)} [kg/m^3] = \lambda{\left(\vec{x} \right)}\end{equation}\begin{equation}\operatorname{O2+}{\left(\vec{x} \right)} [kg/m^3] = \lambda{\left(\vec{x} \right)}\end{equation}\begin{equation}\operatorname{N2+}{\left(\vec{x} \right)} [kg/m^3] = \lambda{\left(\vec{x} \right)}\end{equation}\begin{equation}\operatorname{N+}{\left(\vec{x} \right)} [kg/m^3] = \lambda{\left(\vec{x} \right)}\end{equation}\begin{equation}\operatorname{NO+}{\left(\vec{x} \right)} [kg/m^3] = \lambda{\left(\vec{x} \right)}\end{equation}\begin{equation}\operatorname{O2D+}{\left(\vec{x} \right)} [1/m^3] = \lambda{\left(\vec{x} \right)}\end{equation}\begin{equation}\operatorname{O2P+}{\left(\vec{x} \right)} [kg/m^3] = \lambda{\left(\vec{x} \right)}\end{equation}\begin{equation}\operatorname{H+}{\left(\vec{x} \right)} [kg/m^3] = \lambda{\left(\vec{x} \right)}\end{equation}\begin{equation}\operatorname{He+}{\left(\vec{x} \right)} [kg/m^3] = \lambda{\left(\vec{x} \right)}\end{equation}\begin{equation}\operatorname{e-}{\left(\vec{x} \right)} [1/m^3] = \lambda{\left(\vec{x} \right)}\end{equation}\begin{equation}\operatorname{Te}{\left(\vec{x} \right)} [K] = \lambda{\left(\vec{x} \right)}\end{equation}\begin{equation}\operatorname{Ti}{\left(\vec{x} \right)} [K] = \lambda{\left(\vec{x} \right)}\end{equation}\begin{equation}\operatorname{Vi_{east}}{\left(\vec{x} \right)} [m/s] = \lambda{\left(\vec{x} \right)}\end{equation}\begin{equation}\operatorname{Vi_{north}}{\left(\vec{x} \right)} [m/s] = \lambda{\left(\vec{x} \right)}\end{equation}\begin{equation}\operatorname{Vi_{up}}{\left(\vec{x} \right)} [m/s] = \lambda{\left(\vec{x} \right)}\end{equation}\begin{equation}\operatorname{MagneticLatitude}{\left(\vec{x} \right)} [degrees] = \lambda{\left(\vec{x} \right)}\end{equation}\begin{equation}\operatorname{MagneticLongitude}{\left(\vec{x} \right)} [degrees] = \lambda{\left(\vec{x} \right)}\end{equation}

```python
#Write list of variables to file 'kamodo_info'
model.write_variables()
```

# Interpolation

```python
print(model.rho([[70,30.,440000.],[70,40.,400000.],[90,40.,400000.],[120,40.,440000.]]))
```

    [2.68652554e-14 9.22378231e-14 1.12557827e-13 9.39359154e-14]

```python
var="rho"
grid = np.ndarray(shape=(4,3), dtype=np.float32)
grid[:,0] = [70, 70, 90, 120]
grid[:,1] = [30, 40, 40, 40]
grid[:,2] = [440000., 400000., 400000., 440000.]
units=model.variables[var]['units']
test = model.variables[var]['interpolator'](grid)
print(units)
print(test)
```

    kg/m^3
    [2.68652554e-14 9.22378231e-14 1.12557827e-13 9.39359154e-14]

# Plotting

```python
# Slice at given Altitude
fig=model.get_plot('rho', 400000., '2D-alt', colorscale='Viridis', log="T")
#iplot(fig)

#pio.write_image(fig, 'images/GITM_2D-alt.svg')
```

```python
# Slice at given latitude
fig=model.get_plot('Tn', 0., '2D-lat', colorscale='Rainbow')
iplot(fig)

#pio.write_image(fig, 'images/GITM_2D-lat.svg')
```

```python
# Slice at given longitude
fig=model.get_plot('rho', 180., '2D-lon', colorscale='Rainbow', log='T')
iplot(fig)

#pio.write_image(fig, 'images/GITM_2D-lon.svg')
```

```python
# 3D view at given Altitude
fig=model.get_plot('rho', 400000., '3D-alt', colorscale='Rainbow')
iplot(fig)

#pio.write_image(fig, 'images/GITM_3D-alt.svg')
```

```python
# Isosurface with slice at Lat=0.
fig=model.get_plot('Tn', 750., 'iso')
iplot(fig)

#scope = PlotlyScope()
#with open("images/GITM_iso.png", "wb") as f:
#    f.write(scope.transform(fig, format="png"))

#fig.write_html("GITM_iso.html",full_html=False)
```

```python

```


# 12. Analysing MD data

### Important note:
**This part of the tutorial was originally developed with the assumption that the reader had already gone through the Molecular Dynamics (MD) OxCompBio workshop. It makes some assumptions of prior knowledge about molecular dynamics and the trajectories generated from such simulations.**

**We therefore advise readers to first complete the MD tutorial and then return to this section afterwards.**

As previously discussed, there are many things for which python can be used. One such example is the analysis of Molecular Dynamics (MD) simulations.

There exist several MD analysis libraries, including (but not limited to):
- [MDAnalysis](https://www.mdanalysis.org/)
- [MDTraj](http://mdtraj.org/1.9.3/)
- [PyTraj](https://amber-md.github.io/pytraj/latest/index.html)
- [LOOS](http://grossfieldlab.github.io/loos/)

In this section, we will look at how we can use the MDAnalysis library to process simulation structures and trajectories. We will also look at how we can interface with the [NGLView](https://amber-md.github.io/pytraj/latest/index.html) to visualise our simulations.

This tutorial will aim to put together all you have learned so far into one final exercise.

## 12.1 The MDAnalysis Universe module

To work with structures and trajectories in MDAnalysis, you must first create an MDAnalysis `Universe`.

This is an object that contains all the necessary information about your system, including atomic coordinates, atom types, box vectors (if available), etc...

A `Universe` is a 'class' object, which stores all the information above your system, including;
- Atomic coordinates (i.e. positions).
- Atom names.
- Atom types.
- Bond information (if available).

.. and so on. Having access to this information is very useful for analysis, such as if you are trying to measure distances or dihedrals.

When we load a trajectory (as we will do later in the section), the `Universe` also stores information like the current frame number, etc.

First we import the MDAnalysis `Universe` module, we will also import `numpy` for later use:

```python
from MDAnalysis import Universe
import numpy
```

You then load the structure file into a `Universe` instance named `alanine` with:

```python
alanine = Universe('datafiles/ALA.pdb')
```

**Note 1**: We could have also used `import MDAnalysis`, in which case we would have to use `MDAnalysis.Universe` in place of `Universe`. As previously discussed, the `from ... import ...` syntax allows us to import only a sub-module from a library, rather than the full thing, which can be convenient in various circumstances.

**Note 2**: On loading the PDB file you will get a warning that element information is absent or missing. This is normal and occurs because the input PDB file we use does not have element information for all the atoms (which is expected in a normal PDB file). Therefore MDAnalysis warns us that there are no elements present for us to use.

You can check the different properties of your universe using MDAnalysis. The general syntax for this is `universename.attributename`; some properties (attributes) are themselves classes (e.g. `alanine.atoms` groups all the information about our atoms) and will have their own attributes. For example, if you want to check the names of the atoms of your structure, you can type:

```python
alanine.atoms.names
```

The atom "name" is essentially a type of tag that defines the atoms in your system. Quite often this will be used to label similar atoms, for example in the PDB convention alpha carbons are always named "CA". When analysing MD simulations, the naming scheme will vary depending on the force field used (see MD tutorial).

### How would you find the values of the Phi and Psi dihedral angles of alanine dipeptide, in degrees or radians?

Hint: the Phi dihedral is calculated from atoms CLP, NL, CA, and CRP, and the Psi dihedral is calculated using atoms NR, CRP, CA, and NL.

For this we can make use of the built-in functions associated with the Universe class (or 'methods'). We can use `select_atoms(selection)` to isolate a group of atoms that match `selection`; the syntax of the selections strings is generally the same as for VMD (which you may have use in the MD tutorial).

`alanine.select_atoms(selection)` will give us an 'atom group' class instance. We can then use this instance's method `dihedral` to turn four atoms into a 'dihedral' instance, and finally using `value()` on this instance will return the (current) value of that dihedral angle.

```python
print(type(alanine))
print(type(alanine.select_atoms('name CLP NL CA CRP')))
print(type(alanine.select_atoms('name CLP NL CA CRP').dihedral))

phi = alanine.select_atoms('name CLP NL CA CRP').dihedral.value()
psi = alanine.select_atoms('name NR CRP CA NL').dihedral.value()
print(f'The Phi dihedral is {phi:.2f} in degrees and {numpy.deg2rad(phi):.2f} in radians')
print(f'The Phi dihedral is {psi:.2f} in degrees and {numpy.deg2rad(psi):.2f} in radians')
```

### How would you plot the change in Phi and Psi over a trajectory?

In order to do this, let's break the process down into small steps.

You will first need to load the trajectory. The format to do this is:

universe_name = Universe(PDBfile, TRJfile)

In this case we have provided you a trajectory in the `datafiles` folder, the PDB file is called 'ALA.pdb' and the TRJ file is called 'ALA.xtc'. So we can call them as `./datafiles/ALA.pdb` and `./datafiles/ALA.xtc`.

```python
# Exercise 12.1.1: Create a Universe called 'ala_trajectory' using the PDB file 'ALA.pdb'
# and the trajectory file 'ALA.xtc'
ala_trajectory = Universe('./datafiles/ALA.pdb', './datafiles/ALA.xtc')
```

Now that you have a Universe with your trajectory, we can access another feature of the Universe: trajectory data.

First we should look at what the current trajectory frame number is.

The frame number is stored under `universename.trajectory.frame` and the total number of frames is found under `universename.trajectory.n_frames`.

```python
# Exercise 12.1.2: Print the current frame number and the total number of frames
print(ala_trajectory.trajectory.frame)
print(ala_trajectory.trajectory.n_frames)
```

A `Universe.trajectory` acts kind of like a list storing each frame in a simulation and various information about it, so we can loop through it the same way we did for lists in previous sections. As we go through each frame, the coordinates of each atom, stored in Universe.atoms, will be updated.

We can therefore quite a quick `for` loop that prints the frame number for each frame in the trajectory.

```python
# When using MDanalysis, we usually refer to each frame as a 'timestep' or 'ts'
# Here at each loop instance, we allocate the current frame to 'ts'
for ts in ala_trajectory.trajectory:
    # the comma at the end here will stop it printing on a new line every time,
    # so this doesn't take up too much space
    print(ts.frame)
```

Now, use the same logic as Exercise 11.1 to print the phi and psi dihedral angles at each frame:

```python
# Exercise 12.1.3: Print the frame number and at least one of the dihedrals.
# First create the phi and psi selections
phi_dihedral = ala_trajectory.select_atoms('name CLP NL CA CRP').dihedral
psi_dihedral = ala_trajectory.select_atoms('name NR CRP CA NL').dihedral

# We then loop over the the trajectory
for ts in ala_trajectory.trajectory:
    phi = phi_dihedral.value()
    psi = psi_dihedral.value()
    print(f"Current frame: {ts.frame}, phi: {phi}, psi: {psi}")
```

Now that you can access both the frame number and the phi and psi dihedrals, it's time to plot them.

First, we will need to import a plotting library. In this case we import pyplot from matplotlib.

We then create three empty lists (`frames`, `all_phi`, and `all_psi`) to hold our trajectory data.

Then we fill each of those lists with the relevant values.

```python
# We declare matplotlib inline to make sure it plots properly
%matplotlib inline
# We need pyplot from matplotlib to plot the dihedrals
from matplotlib import pyplot

# Create three empty lists
frames = []
all_phi = []
all_psi = []

# iterate through the trajectory
for frame in ala_trajectory.trajectory:
    # calculate phi and psi
    phi = ala_trajectory.select_atoms('name CLP or name NL or name CA or name CRP').dihedral.value()
    psi = ala_trajectory.select_atoms('name NR or name CRP or name CA or name NL').dihedral.value()
    # append frame number, phi, and psi to the lists
    frames.append(frame.frame)
    all_phi.append(phi)
    all_psi.append(psi)

# We then plot our data
pyplot.plot(frames, all_psi)
pyplot.plot(frames, all_phi)
pyplot.show()
```

As shown above, you can use the matplotlib.pyplot module to plot `x` and `y` values in the following manner:

```python
pyplot.plot(x_value, y_value)
```

We then use `pyplot.show()` to make the plot appear.

**Note:** You can adjust your matplotlib pyplot plot using various different options such as; `pyplot.title()`, `pyplot.xlim()`, `pyplot.ylim()`, `pyplot.xlabel()`, and `pyplot.ylabel()`.

For example, `pyplot.title()` will add a title.

Check the `help()` of these functions for more details!

## 12.2 Analysing a protein simulation

In the MD tutorial you will do some molecular dynamics simulations of a HIV-1 protease protein using gromacs.
Let's now look at how you could use MDAnalysis, and an associated python visulisation library NGLview, to analyse an MD trajectory of this system.

_Note: depending on whether you have already done the MD tutorial, you may find that you have already done the analysis shown here. You may still find this section useful as we provide a few extra details which explain how the MD trajectory analysis works._

### Visualising the trajectory using nglview

The nglview library is a python widget for visualising simulation trajectories, achieving a similar task to the VMD program that you may also use in the MD tutorial. One of the interesting advantages of nglview is that it interfaces directly with analysis packages such as MDAnalysis and runs within jupyter notebooks.

Let's see how we can use nglview to visualise an MDAnalysis universe object.

First we need to create a universe (let's call it `protein`) from the simulation output files "pre_md.pdb" and "md_cent.xtc" (which are present in the `datafiles` directory, hence will be passed to Universe as `./datafiles/pre_md.pdb` and `./datafiles/md_cent.xtc`).

_Note 1: We have pre-aligned the trajectory to the first frame for you so as to remove any motions related to translation._

_Note 2: When loading the trajectory you will get a warning that MDAnalysis needs to reload offsets, this is normal in this scenario._

```python
# Exercise 12.2.1: Let's load a universe named protein
protein = Universe('./datafiles/pre_md.pdb', './datafiles/md_cent.xtc')
```

Next let's load nglview and use it's `show_mdanalysis` function to load the MDAnalysis universe

```python
import nglview
protein_view = nglview.show_mdanalysis(protein)
```

By default this pre-sets the nglview to show the protein in the cartoon representation. Let's add a few options to colour the protein by secondary structure, show water oxygens and change the background colour

```python
# Let's update the cartoon representation to colour the protein by secondary structure
protein_view.update_cartoon(color='sstruc')

# We then add a transparent hyperball representation of the water oxygens
#(play with the opacity value, see what you get)
protein_view.add_hyperball('SOL and not hydrogen', opacity=0.4)

# Let's change the display a little bit
protein_view.parameters = dict(camera_type='orthographic', clip_dist=0)

# Set the background colour to black
protein_view.background = 'black'

# Call protein_view to visualise the trajectory
protein_view
```

The nglview output can be controlled in the following way:

- play / pause button: play the trajectory
- double click window: enter or exit full screen mode
- left mouse button: rotate system
- middle mouse wheel: zoom in/out
- right mouse button: translate system

As you can be seen from the trajectory, the HIV-1 protease structure does indeed move, but by how much? In the next section we will see how we can use MDAnalysis to quantify backbone fluctuations.

### Calculating the root-mean-square deviation

In order to gain a quantitative description of how the HIV-1 protease moves in our simulation we can calculate the root-mean-square deviation (RMSD) of the protein backbone.

The RMSD gives us an idea of how 'stable' our protein is when compared to our starting, static, structure. The lower the RMSD is the, more stable we can say our protein is.

The RMSD as a function of time, $\rho (t)$, can be defined by the following equation:

\begin{equation}
\\
\rho (t) = \sqrt{\frac{1}{N}\sum^N_{i=1}w_i\big(\mathbf{x}_i(t) - \mathbf{x}^{\text{ref}}_i\big)^2}
\end{equation}

Luckily MDAnalysis has its own built-in function to calcualte this, we can import it like we did before.

```python
from MDAnalysis.analysis.rms import RMSD as rmsd
```

In order to calculate the RMSD for every frame in our trajectory we will need:

- A reference structure
- A universe object
- A selection of atoms

In our case the reference structure will be the HIV-1 protease structure in the first frame.

Our universe object will be the 'protein' object we defined above.

For our selection we will use the backbone atoms.

```python
ref = Universe('datafiles/pre_md.pdb', 'datafiles/md_cent.xtc')

# Set the ref trajectory to the first frame
ref.trajectory[0]
```

Due to the way that GROMACS post processes the trajectory file we need to edit it slightly before running our RMSD.

This is done by aligning all frames to the reference structure.

```python
from MDAnalysis.analysis import align

# Create the MD simulation universe
protein = Universe('datafiles/pre_md.pdb', 'datafiles/md_cent.xtc')
# Call the MDAnalysis align function to align the MD simulation unvierse to the reference (first frame) universe
align_strucs = align.AlignTraj(protein, ref, select="backbone", weights="mass", in_memory=True, verbose=True)

R = align_strucs.run()
```

You will have noticed that running this function stores it in the variable 'R', we can now access the RMSD values:

```python
rmsd_data = R.rmsd
```

We'd like to visualise how the RMSD changes over time and this can be done in the same way you did in Excercise 11.1.5.

Take a look at the 'rmsd_data' variable (it's a numpy array) and try plotting it below.

You will need to access 'rmsd_data' (a numpy array) in order to plot both the time and the RMSD as a line plot.

```python
# Excercise 12.2.2: Plot the RMSD data for the HIV-1 protease system.

# Make sure to add appropriate axis titles.
# What does the RMSD tell you about the protein?
# [If you have time] What happens when you calculate the RMSD using more atoms (i.e. not just the backbone)
pyplot.plot(rmsd_data)
pyplot.title("RMSD over time")
pyplot.xlabel("Frame number")
pyplot.ylabel("RMSD (Angstroms)")
pyplot.show()
```

### Calculating the root-mean-square fluctuation

To look at how each residue flucuates over it's average postion we can use the closely related measurement of root-mean-square fluctuation (RMSF).

The RMSF for an atom, $\rho_i$, is given by:

\begin{equation}
\rho_i = \sqrt{\sum^N_{i=1} \big\langle(\mathbf{x}_i - \langle \mathbf{x}_i \rangle )^2 \big\rangle }
\end{equation}

```python
from MDAnalysis.analysis.rms import RMSF as rmsf
```

```python
# Reset the trajectory to the first frame
protein.trajectory[0]

# We will need to select the alpha Carbons only
calphas = protein.select_atoms("name CA")
rmsf_calc = rmsf(calphas, verbose=True).run()
```

```python
# Excercise 12.2.3: Plot the RMSF data for the HIV-1 protease system.
    # Tip, in order to plot the resids you will need to access them through the rmsf_calc object

# Make sure to add appropriate axis titles.
# What parts of the protein have a high RMSF, can you locate these on the protein structure?
# [If you have time] What happens when you calculate the RMSF using more atoms (i.e. not just the backbone)
pyplot.plot(calphas.resids, rmsf_calc.rmsf)
pyplot.title("Per-Residue Alpha Carbon RMSF")
pyplot.xlabel("Residue Number")
pyplot.ylabel("RMSF (Angstrom)")
pyplot.show()
```


# 3. Data Pre-processing

Data pre-processing techniques generally refer to the addition, deletion, or transformation of training set data. Different models have different sensitivities to the type of predictors in the model; *how* the predictors enter the model is also important.

The need for data pre-processing is determined by the type of model being used. Some procedures, such as tree-based models, are notably insensitive to the characteristics of the predictor data. Others, like linear regression, are not. In this chapter, a wide array of possible methodologies are discussed.

How the predictors are encoded, called *feature engineering*, can have a significant impact on model performance. Often the most effective encoding of the data is informed by the modeler's understanding of the problem and thus is not derived from any mathematical techniques.

## 3.1 Case Study: Cell Segmentation in High-Content Screening

Check if data exists.

```
!ls -l ../datasets/segmentationOriginal/
```

    total 4016
    -rw-r--r--  1 leigong  staff  2053006 Nov 24 15:58 segmentationOriginal.csv

This dataset is from Hill et al. (2007) that consists of 2019 cells. Of these cells, 1300 were judged to be poorly segmented (PS) and 719 were well segmented (WS); 1009 cells were reserved for the training set.

```
import numpy as np
import pandas as pd

cell_segmentation = pd.read_csv("../datasets/segmentationOriginal/segmentationOriginal.csv")
```

```
cell_segmentation.shape
```

    (2019, 120)

A first look at the dataset.

```
cell_segmentation.head(5)
```

      Unnamed: 0
      Cell
      Case
      Class
      AngleCh1
      AngleStatusCh1
      AreaCh1
      AreaStatusCh1
      AvgIntenCh1
      AvgIntenCh2
      ...
      VarIntenCh1
      VarIntenCh3
      VarIntenCh4
      VarIntenStatusCh1
      VarIntenStatusCh3
      VarIntenStatusCh4
      WidthCh1
      WidthStatusCh1
      XCentroid
      YCentroid

      0
       1
       207827637
        Test
       PS
       143.247705
       1
       185
       0
       15.711864
         3.954802
      ...
       12.474676
        7.609035
         2.714100
       0
       2
       2
       10.642974
       2
        42
        14

      1
       2
       207932307
       Train
       PS
       133.752037
       0
       819
       1
       31.923274
       205.878517
      ...
       18.809225
       56.715352
       118.388139
       0
       0
       0
       32.161261
       1
       215
       347

      2
       3
       207932463
       Train
       WS
       106.646387
       0
       431
       0
       28.038835
       115.315534
      ...
       17.295643
       37.671053
        49.470524
       0
       0
       0
       21.185525
       0
       371
       252

      3
       4
       207932470
       Train
       PS
        69.150325
       0
       298
       0
       19.456140
       101.294737
      ...
       13.818968
       30.005643
        24.749537
       0
       0
       2
       13.392830
       0
       487
       295

      4
       5
       207932455
        Test
       PS
         2.887837
       2
       285
       0
       24.275735
       111.415441
      ...
       15.407972
       20.504288
        45.450457
       0
       0
       0
       13.198561
       0
       283
       159

5 rows × 120 columns

This chapter will use the training set samples to demonstrate data pre-processing techniques.

```
cell_segmentation.groupby('Case').count()
```

      Unnamed: 0
      Cell
      Class
      AngleCh1
      AngleStatusCh1
      AreaCh1
      AreaStatusCh1
      AvgIntenCh1
      AvgIntenCh2
      AvgIntenCh3
      ...
      VarIntenCh1
      VarIntenCh3
      VarIntenCh4
      VarIntenStatusCh1
      VarIntenStatusCh3
      VarIntenStatusCh4
      WidthCh1
      WidthStatusCh1
      XCentroid
      YCentroid

      Case

      Test
       1010
       1010
       1010
       1010
       1010
       1010
       1010
       1010
       1010
       1010
      ...
       1010
       1010
       1010
       1010
       1010
       1010
       1010
       1010
       1010
       1010

      Train
       1009
       1009
       1009
       1009
       1009
       1009
       1009
       1009
       1009
       1009
      ...
       1009
       1009
       1009
       1009
       1009
       1009
       1009
       1009
       1009
       1009

2 rows × 119 columns

```
# separate training and test data
cell_train = cell_segmentation.ix[cell_segmentation['Case'] == 'Train']
cell_test = cell_segmentation.ix[cell_segmentation['Case'] == 'Test']

cell_train.head(5)
```

      Unnamed: 0
      Cell
      Case
      Class
      AngleCh1
      AngleStatusCh1
      AreaCh1
      AreaStatusCh1
      AvgIntenCh1
      AvgIntenCh2
      ...
      VarIntenCh1
      VarIntenCh3
      VarIntenCh4
      VarIntenStatusCh1
      VarIntenStatusCh3
      VarIntenStatusCh4
      WidthCh1
      WidthStatusCh1
      XCentroid
      YCentroid

      1
        2
       207932307
       Train
       PS
       133.752037
       0
       819
       1
       31.923274
       205.878517
      ...
       18.809225
       56.715352
       118.388139
       0
       0
       0
       32.161261
       1
       215
       347

      2
        3
       207932463
       Train
       WS
       106.646387
       0
       431
       0
       28.038835
       115.315534
      ...
       17.295643
       37.671053
        49.470524
       0
       0
       0
       21.185525
       0
       371
       252

      3
        4
       207932470
       Train
       PS
        69.150325
       0
       298
       0
       19.456140
       101.294737
      ...
       13.818968
       30.005643
        24.749537
       0
       0
       2
       13.392830
       0
       487
       295

      11
       12
       207932484
       Train
       WS
       109.416426
       0
       256
       0
       18.828571
       125.938776
      ...
       13.922937
       18.643027
        40.331747
       0
       0
       2
       17.546861
       0
       211
       495

      14
       15
       207932459
       Train
       PS
       104.278654
       0
       258
       0
       17.570850
       124.368421
      ...
       12.324971
       17.747143
        41.928533
       0
       0
       2
       17.660339
       0
       172
       207

5 rows × 120 columns

## 3.2 Data Transformation for Individual Predictors

Transformations of predictor variables may be needed for several reasons. Some modeling techniques may have strict requirements, such as the predictors having a commom scale. In other cases, creating a good model may be difficult due to specific characteristics of the data (e.g., outliers).

### Centering and Scaling

To center a predictor variable, the average predictor value is substracted from all the values. As a result of centering, the predictor has a zero mean. Similarly, to scale the data, each value of the predictor variable is divided by its standard deviation. Scaling the data coerce the values to have a common standard deviation of one. These manipulations are generally used to improve the numerical stability of some calculations, such as PLS. The only real downside to these transformation is a loss of interpretability of the individual values.

### Transformations to Resolve Skewness

An un-skewed distribution is one that is roughly symmetric. A rule of thumb to consider is that skewed data whose ratio of the highest value to the lowest value is greater than 20 have significant skewness. The sample skewness statistic is defined $$\text{skewness} = {\sum (x_i - \bar{x})^3 \over (n - 1) v^{3/2}},$$ where $$v = {\sum (x_i - \bar{x})^2 \over (n - 1)}.$$ Note that the skewness for a normal distribution is zero.

The cell segmentation data contain a predictor that measures the standard deviation of the intensity of the pixels in the actin filaments.

```
%matplotlib inline
import matplotlib.pyplot as plt

# Some nice default configuration for plots
plt.rcParams['figure.figsize'] = 10, 7.5
plt.rcParams['axes.grid'] = True
plt.gray()
```

    <matplotlib.figure.Figure at 0x10f0c2e10>

```
fig, (ax1, ax2, ax3) = plt.subplots(1, 3)

ax1.hist(cell_train['VarIntenCh3'].values, bins=20)
ax1.set_xlabel('Natural Units')
ax1.set_ylabel('Count')

ax2.hist(np.log(cell_train['VarIntenCh3'].values), bins=20)
ax2.set_xlabel('Log Units')

ax3.hist(np.sqrt(cell_train['VarIntenCh3'].values), bins=20)
ax3.set_xlabel('Square Root Units')
```

The histogram shows a strong right skewness. The log transformation seems to work well for this dataset. The ratio of the smallest to largest value and the sample skewness statistic all agree with the histogram under natural units.

```
from scipy.stats import skew

r = np.max(cell_train['VarIntenCh3'].values)/np.min(cell_train['VarIntenCh3'].values)
skewness = skew(cell_train['VarIntenCh3'].values)

print 'Ratio of the smallest to largest value is {0} \nSample skewness statistic is {1}'.format(r, skewness)
```

    Ratio of the smallest to largest value is 870.887247203
    Sample skewness statistic is 2.39518418124

Alternatively, statistical models can be used to empirically identify an appropriate transformation. One of the most famous transformations is the Box-Cox family, i.e.
\begin{equation}
x^* = \begin{cases} {x^{\lambda}-1 \over \lambda} & \text{if} \ \lambda \neq 0 \\ log(x) & \text{if} \ \lambda = 0 \end{cases}
\end{equation}
This family covers the log ($\lambda = 0$), square ($\lambda = 2$), square root ($\lambda = 0.5$), inverse ($\lambda = -1$), and others in-between. Using the training data, $\lambda$ can be estimated using maximum likelihood estimation (MLE). This procedure would be applied independently to each predictor data that contain values **greater than 0**.

The boxcox() in *scipy.stats* finds the estimated lambda and performs the transformation at the same time.

```
from scipy.stats import boxcox

print 'Estimated lambda is {0}'.format(boxcox(cell_train['VarIntenCh3'].values)[1])
```

    Estimated lambda is 0.121531937325

Take another predictor for example.

```
fig, (ax1, ax2) = plt.subplots(1, 2)

ax1.hist(cell_train['PerimCh1'].values, bins=20)
ax1.set_xlabel('Natural Units')
ax1.set_ylabel('Count')

ax2.hist(boxcox(cell_train['PerimCh1'].values)[0], bins=20)
ax2.set_xlabel('Transformed Data (lambda = {:1.4f})'.format(boxcox(cell_train['PerimCh1'].values)[1]))
```

## 3.3 Data Transformations for Multiple Predictors

These transformations act on groups of predictors, typically the entire set under consideration. Of primary importance are methods to resolve outliers and reducce the dimension of the data.

### Transformations to Resolve Outliers

We generally define outliers as samples that are exceptionally far from the mainstream of the data. Even with a thorough understanding of the data, outliers can be hard to define. However, we can often identify an unusual value by looking at a figure. When one or more samples are suspected to be outliers, the first step is to make sure that the values are scientifically valid and that no data recording errors have occured. Great care should be taken not to hastily remove or change values, especially if the sample size is small. With small sample sizes, apparent outliers might be a result of a skewed distribution where there are not yet enough data to see the skewness. Also, the outlying data may be an indication of a special part of the population under study that is just starting to be sampled. Depending on how the data were collected, a "cluster" of valid points that reside outside the mainstream of the data might belong to a different population than the other samples, e.g. *extrapolation* and *applicability domain*.

There are several predictive models that are resistant to outliers, e.g.
- Tree based classification models: creat splits of the training set.
- Support Vector Machines (SVM) for classification: disregard a portion if tge training set that may be far away from the decision boundary.

If a model is considered to be sensitive to outliers, one data transformation that can minimize the problem is the *spatial sign*. Mathematically, each sample is divided by its squared norm: $$x_{ij}^* = {x_{ij} \over \sqrt{\sum_{j=1}^p x_{ij}^2}}.$$ Since the denominator is intended to measure the squared distance to the center of the predictor's distribution, it is **important** to center and scale the predictor data prior to using this transformation. Note that, unlike centering and scaling, this manipulation of the predictors transform them as a group. Removing predictor variables after applying the spatial sign transformation may be problematic.

```
# toy example
beta0 = -2.3 # intercept
beta1 = 0.8 # slope
n = 1000
x1_true = np.random.normal(4, 2, n)
x2_true = np.zeros(n)

# generate a random sample
for i in xrange(n):
    x2_true[i] = beta0 + beta1*x1_true[i] + np.random.normal(size = 1)

# generate outliers
x1_outliers = np.random.uniform(-4, -3, 8)
x2_outliers = np.zeros(8)
for i in xrange(8):
    x2_outliers[i] = x1_outliers[i] + np.random.normal(size = 1)

plt.scatter(x1_true, x2_true)
plt.plot(x1_outliers, x2_outliers, 'ro', markersize=8)
```

```
from sklearn.preprocessing import scale
x1 = scale(np.concatenate([x1_true, x1_outliers]))
x2 = scale(np.concatenate([x2_true, x2_outliers]))
x = np.array(zip(x1, x2))

# spatial sign
dist = x[:, 0]**2 + x[:, 1]**2
x1 = x[:, 0]/np.sqrt(dist)
x2 = x[:, 1]/np.sqrt(dist)

plt.scatter(x1[:-8], x2[:-8])
plt.plot(x1[-7:], x2[-7:], 'ro', markersize=8)
```

The *spatial sign* transformation brings the outliers towards the majority of the data.

### Data Reduction and Feature Extraction

These methods reduce the data by generating a smaller set of predictors that seek to capture a majority of the information in the original variables. For most data reduction techniques, the new predictors are functions of the original predictors; therefore, all the original predictors are still needed to create the surrogate variables. This class of methods is often called *signal extraction* or *feature extraction* techniques.

Principal component analysis (PCA) seeks to find linear combinations of the predictors, known as principal components (PCs), which capture the most possible variance. The first PC is defined as the linear combination of the predictors that captures the most variability of all possible linear combinations. Then, subsequent PCs are derived such that these linear combinations capture the most remaining variability while also being uncorrelated with all previous PCs. Mathematically,
$$\text{PC}_j = (a_{j1} \times \text{Predictor 1}) + \cdots + (a_{jP} \times \text{Predictor P}).$$
P is the number of predictors. The coefficients $a_{j1}, \cdots, a_{jP}$ are called component weights and help us understand which predictors are most important to each PC.

Let us look at an example from the previous dataset.

```
cell_train_subset = cell_train[['Class', 'FiberWidthCh1', 'EntropyIntenCh1']]
```

```
colors = ['b', 'r']
markers = ['s', 'o']
c = ['PS', 'WS']
for k, m in enumerate(colors):
    i = (cell_train_subset['Class'] == c[k])
    if k == 0:
        plt.scatter(cell_train_subset['FiberWidthCh1'][i], cell_train_subset['EntropyIntenCh1'][i],
                    c=m, marker=markers[k], alpha=0.4, s=26, label='PS')
    else:
        plt.scatter(cell_train_subset['FiberWidthCh1'][i], cell_train_subset['EntropyIntenCh1'][i],
                    c=m, marker=markers[k], alpha=0.4, s=26, label='WS')

plt.title('Original Data')
plt.xlabel('Channel 1 Fiber Width')
plt.ylabel('Entropy intensity of Channel 1')
plt.legend(loc='upper right')
plt.show()
```

Calculate PCs

```
from sklearn.decomposition import PCA

pca = PCA()
pca.fit(cell_train_subset[['FiberWidthCh1', 'EntropyIntenCh1']])
print 'variance explained by PCs {0}'.format(pca.explained_variance_ratio_)
```

    variance explained by PCs [ 0.96900659  0.03099341]

The first PC summarizes 97% of the original variability, while the second summarizes 3%. Hence, it is reasonable to use only the first PC for modeling since it accounts for the majority of the information in the data.

```
cell_train_subset_pca = pca.transform(cell_train_subset[['FiberWidthCh1', 'EntropyIntenCh1']])

colors = ['b', 'r']
markers = ['s', 'o']
c = ['PS', 'WS']
for k, m in enumerate(colors):
    i = np.where(cell_train_subset['Class'] == c[k])[0]
    if k == 0:
        plt.scatter(cell_train_subset_pca[i, 0], cell_train_subset_pca[i, 1],
                    c=m, marker=markers[k], alpha=0.4, s=26, label='PS')
    else:
        plt.scatter(cell_train_subset_pca[i, 0], cell_train_subset_pca[i, 1],
                    c=m, marker=markers[k], alpha=0.4, s=26, label='WS')

plt.title('Transformed')
plt.xlabel('Principal Component #1')
plt.ylabel('Principal Component #2')
plt.legend(loc='upper right')
plt.show()
```

The primary advantage of PCA is that it creates components that are uncorrelated. PCA preprocessing creates new predictors with desirable characteristics for models that prefer predictors to be uncorrelated.

While PCA delivers new predictors with desirable characteristics, it must be used with understanding and care. PCA seeks predictor-set variation without regard to any further understanding of the predictors (i.e. measurement scales or distributions) or to knowledge of the modeling objectives (i.e. response variable). Hence, without proper guidance, PCA can generate components that summarize characteristics of the data that are irrelevant to the underlying structure of the data and also to the ultimate modeling objectives.

PCA was applied to the entire set of segmentation data predictors.

```
cell_train.head(5)
```

      Unnamed: 0
      Cell
      Case
      Class
      AngleCh1
      AngleStatusCh1
      AreaCh1
      AreaStatusCh1
      AvgIntenCh1
      AvgIntenCh2
      ...
      VarIntenCh1
      VarIntenCh3
      VarIntenCh4
      VarIntenStatusCh1
      VarIntenStatusCh3
      VarIntenStatusCh4
      WidthCh1
      WidthStatusCh1
      XCentroid
      YCentroid

      1
        2
       207932307
       Train
       PS
       133.752037
       0
       819
       1
       31.923274
       205.878517
      ...
       18.809225
       56.715352
       118.388139
       0
       0
       0
       32.161261
       1
       215
       347

      2
        3
       207932463
       Train
       WS
       106.646387
       0
       431
       0
       28.038835
       115.315534
      ...
       17.295643
       37.671053
        49.470524
       0
       0
       0
       21.185525
       0
       371
       252

      3
        4
       207932470
       Train
       PS
        69.150325
       0
       298
       0
       19.456140
       101.294737
      ...
       13.818968
       30.005643
        24.749537
       0
       0
       2
       13.392830
       0
       487
       295

      11
       12
       207932484
       Train
       WS
       109.416426
       0
       256
       0
       18.828571
       125.938776
      ...
       13.922937
       18.643027
        40.331747
       0
       0
       2
       17.546861
       0
       211
       495

      14
       15
       207932459
       Train
       PS
       104.278654
       0
       258
       0
       17.570850
       124.368421
      ...
       12.324971
       17.747143
        41.928533
       0
       0
       2
       17.660339
       0
       172
       207

5 rows × 120 columns

```
cell_train_feature = cell_train.iloc[:, 4:]
cell_train_feature.head(5)
```

      AngleCh1
      AngleStatusCh1
      AreaCh1
      AreaStatusCh1
      AvgIntenCh1
      AvgIntenCh2
      AvgIntenCh3
      AvgIntenCh4
      AvgIntenStatusCh1
      AvgIntenStatusCh2
      ...
      VarIntenCh1
      VarIntenCh3
      VarIntenCh4
      VarIntenStatusCh1
      VarIntenStatusCh3
      VarIntenStatusCh4
      WidthCh1
      WidthStatusCh1
      XCentroid
      YCentroid

      1
       133.752037
       0
       819
       1
       31.923274
       205.878517
       69.916880
       164.153453
       0
       0
      ...
       18.809225
       56.715352
       118.388139
       0
       0
       0
       32.161261
       1
       215
       347

      2
       106.646387
       0
       431
       0
       28.038835
       115.315534
       63.941748
       106.696602
       0
       0
      ...
       17.295643
       37.671053
        49.470524
       0
       0
       0
       21.185525
       0
       371
       252

      3
        69.150325
       0
       298
       0
       19.456140
       101.294737
       28.217544
        31.028070
       0
       0
      ...
       13.818968
       30.005643
        24.749537
       0
       0
       2
       13.392830
       0
       487
       295

      11
       109.416426
       0
       256
       0
       18.828571
       125.938776
       13.600000
        46.800000
       0
       0
      ...
       13.922937
       18.643027
        40.331747
       0
       0
       2
       17.546861
       0
       211
       495

      14
       104.278654
       0
       258
       0
       17.570850
       124.368421
       22.461538
        71.206478
       0
       0
      ...
       12.324971
       17.747143
        41.928533
       0
       0
       2
       17.660339
       0
       172
       207

5 rows × 116 columns

Because PCA seeks linear combinations of predictors that maximize variability, it will naturally first be drawn to summarizing predictors that have more variation. If the original predictors are on measurement scales that differ in orders of magnitude or have skewed distributions, PCA will be focusing its efforts on identifying the data structure based on measurement scales and distributional difference rather than based on the important relationships within the data for the current problem. Hence, it is best to first transform skewed predictors and then center and scale the predictors prior to performing PCA.

```
# Box-Cox transformation on positive predictors
# separate positive and non-positive predictors
pos_indx = np.where(cell_train_feature.apply(lambda x: np.all(x > 0)))[0]
cell_train_feature_pos = cell_train_feature.iloc[:, pos_indx]
print "# of positive features is {0}".format(pos_indx.shape[0])
cell_train_feature_nonpos = cell_train_feature.drop(cell_train_feature.columns[pos_indx], axis=1, inplace=False)
print "# of npn-positive features is {0}".format(cell_train_feature.shape[1] - pos_indx.shape[0])

cell_train_feature_pos_tr = cell_train_feature_pos.apply(lambda x: boxcox(x)[0])

cell_train_feature_tr = np.c_[cell_train_feature_pos_tr, cell_train_feature_nonpos]
print "The shape before/after transformation is {0} and {1}".format(cell_train_feature.shape, cell_train_feature_tr.shape)
```

    # of positive features is 47
    # of npn-positive features is 69
    The shape before/after transformation is (1009, 116) and (1009, 116)

```
# scale and center predictors
from sklearn.preprocessing import scale

cell_train_feature_tr = scale(cell_train_feature_tr, with_mean=True, with_std=True)
```

The second caveat of PCA is that it does not consider the modeling obejective or response variable when summarizing variability -- it is an *unsupervised technique*. If the predictive relationship between the predictors and response is not connected to the predictors' variability, then the derived PCs will not provide a suitable relationship with the response. In this case, a *supervised technique*, like PLS will derive components while simultaneously considering the corresponding response.

To decide how many components to retain after PCA, a heuristic approach is to create a scree plot, which contains the ordered component number (x-axis) and the ammount of summarized variability (y-axis). Generally, the component number prior to the tapering off of variation is the maximal component that is retained. In an automated model building process, the optimal number of components can be determined by cross-validation.

```
# conduct PCA to transformed predictors
from sklearn.decomposition import PCA

pca = PCA()
pca.fit(cell_train_feature_tr)

# generate scree plot
plt.plot(pca.explained_variance_ratio_)
plt.xlabel('Percent of Total Variance')
plt.ylabel('Component')
```

```
print "The first four components account for {0} of the total variance".format(pca.explained_variance_ratio_[:4])
print "All together they account for {0} of the total variance".format(np.sum(pca.explained_variance_ratio_[:4]))
```

    The first four components account for [ 0.14015262  0.12569721  0.09394676  0.06393814] of the total variance
    All together they account for 0.42373472472 of the total variance

Visually examining the principal components is a critical step for assessing data quality and gaining intuition for the problem. To do this, the first few PCs can be plotted against each other and the plot symbols can be colored by the relevant characteristics, such as the class labels. If PCA has captured a sufficient amount of the information in the data, this type of plot can demonstrate clusters of samples or outliers that may prompt a closer examination of the individual data points. Note that the scale of the components tend to become smaller as they account for less and less variation in the data. If axes are displayed on separate scales, there is the potential to over-interpret any patterns that might be seen for components that account for small amounts of variation.

```
# look at the first 3 PCs
pca = PCA(n_components=3)
cell_train_feature_pca = pca.fit_transform(cell_train_feature_tr)
```

```
colors = ['b', 'r']
markers = ['s', 'o']
c = ['PS', 'WS']

fig, axarr = plt.subplots(3, 3, sharex=True, sharey=True)

# PC1 vs PC3
for k, m in enumerate(colors):
    i = np.where(cell_train['Class'] == c[k])[0]
    if k == 0:
        line1= axarr[0,0].scatter(cell_train_feature_pca[i, 0], cell_train_feature_pca[i, 2],
                    c=m, marker=markers[k], alpha=0.4, s=26, label='PS')
    else:
        line2= axarr[0,0].scatter(cell_train_feature_pca[i, 0], cell_train_feature_pca[i, 2],
                    c=m, marker=markers[k], alpha=0.4, s=26, label='WS')

# PC2 vs PC3
for k, m in enumerate(colors):
    i = np.where(cell_train['Class'] == c[k])[0]
    if k == 0:
        axarr[0,1].scatter(cell_train_feature_pca[i, 1], cell_train_feature_pca[i, 2],
                    c=m, marker=markers[k], alpha=0.4, s=26, label='PS')
    else:
        axarr[0,1].scatter(cell_train_feature_pca[i, 1], cell_train_feature_pca[i, 2],
                    c=m, marker=markers[k], alpha=0.4, s=26, label='WS')

# PC1 vs PC2
for k, m in enumerate(colors):
    i = np.where(cell_train['Class'] == c[k])[0]
    if k == 0:
        axarr[1,0].scatter(cell_train_feature_pca[i, 0], cell_train_feature_pca[i, 1],
                    c=m, marker=markers[k], alpha=0.4, s=26, label='PS')
    else:
        axarr[1,0].scatter(cell_train_feature_pca[i, 0], cell_train_feature_pca[i, 1],
                    c=m, marker=markers[k], alpha=0.4, s=26, label='WS')

axarr[2,0].text(0.5, -1.0, 'PC1', ha='center', va='center', fontsize=24)
axarr[1,1].text(0.5, -1.0, 'PC2', ha='center', va='center', fontsize=24)
axarr[0,2].text(0.5, -1.0, 'PC3', ha='center', va='center', fontsize=24)
fig.legend([line1, line2], ('PS', 'WS'), loc='upper center', ncol=2, frameon=False)
fig.subplots_adjust(hspace=0.12, wspace=0.1)
fig.text(0.5, 0.06, 'Scatter Plot Matrix', ha='center', va='center', fontsize=18)
```

Since the percentages of variation explained are not large for the first three components, it is important not to over-interpret the resulting image. From this plot, there appears to be some separation between the classes when plotting the first and second components. However, the distribution of the well-segmented cells is roughly contained within the distribution of the poorly identified cells. One conclusion is that the cell types are not easily separated.

Another exploratory use of PCA is characterizing which predictors are associated with each component. Recall that each component is a linear combination of the predictors and the coefficient for each predictor is called the loading. Loadings close to zero indicate that the predictor variable did not contribute much to that component.

```
# loadings
pca.components_.shape
```

    (3, 116)

## 3.4 Dealing with Missing Values

In many cases, some predictors have no values for a given sample. It is important to understand *why* the values are missing. First and foremost, it is important to know if the pattern of missing data is related to the outcome. This is called *informative missingness* since the missing data pattern is instructional on its own. Informative missingness can induce significant bias in the model.

Missing data should not be confused with *censored* data where the exact value is missing but something is known about its value. When building traditional statistical models focused on interpretation or inference, the censoring is usually taken in to account in a formal manner by making assumptions about the censoring mechanism. For predictive models, it is more common to treat these data as simple missing data or use the censored value as the observed value.

Missing values are more often related to predictive variables than the sample. Because of this, amount of missing data may be concentrated in a subset of predictors rather than occuring randomly across all the predictors. In some cases, the percentage of missing data is substantial enough to remove this predictor from subsequent modeling activities.

There are cases where the missing values might be concentrated in specific samples. For large datasets, removal of samples based on missing values is not a problem, assuming that the missingness is not informative. In smaller datasets, there is a steep price in removing samples; some of alternative approaches described below may be more appropriate.

If we do not remove the missing data, there are two general approaches. First, a few predictive models, especially tree-based techniques, can specifically account for missing data. Alternatively, missing data can be imputed. In this case, we can use information in the training set predictors to, in essence, estimate the values of other predictors.

Imputation is just another layer of modeling where we try to estimate values of the predictor variables based on other predictor variables. The most relevant scheme for accomplishing this is to use the training set to built an imputation model for each predictor in the daa set. Prior to model training or the prediction of new samples, missing values are filled in using imputation. Note that this extra layer of models adds uncertainty. If we are using resampling to select tuning parameter values or to estimate performance, the imputation should be incorporated within the resampling. This will increase the computational time for building models, but it will also provide honest estimates of model performance.

If the number of predictors affected by missing values is small, an exploratory analysis of the relationships between the preditors is a good idea. For example, visulization or methods like PCA can be used to determine if there are strong relationships between the predictors. If a variable with missing values is highly correlated with another predictor that has few missing values, a focused model can often be effective for imputation.

One popular technique for imputation is a $K$-nearest neighbor model. A new sample is imputed by finding the samples in the training set "closest" to it and averages these nearby points to fill in the value. One advantage of this approach is that the imputed data are confined to be within the range of the training set values. One disadvantage is that the entire training set is required every time a missing value needs to be imputed. Also, the number of neighbors is a tuning parameter, as is the method for determining "closeness" of two points. However, Troyanskaya et al. (2001) found the nearest neighbor approach to be fairly robust to the tuning parameters, as well as the amount of missing data.

```
# randomly sample 50 test set
import random
cell_test_subset = cell_test.iloc[np.sort(random.sample(range(cell_test.shape[0]), 50))]

# separate features
cell_test_subset_f = cell_test_subset.iloc[:, 4:].drop('VarIntenCh3', 1)
cell_test_subset_v = cell_test_subset.iloc[:, 4:]['VarIntenCh3']
cell_train_f = cell_train_feature.drop('VarIntenCh3', 1)
cell_train_v = cell_train_feature['VarIntenCh3']
```

```
# scale and center before imputation
from sklearn.preprocessing import StandardScaler

# standardize based on training set
sc_f = StandardScaler()
cell_train_f_sc = sc_f.fit_transform(cell_train_f)
cell_test_subset_f_sc = sc_f.transform(cell_test_subset_f)

sc_v = StandardScaler()
cell_train_v_sc = sc_v.fit_transform(cell_train_v)
cell_test_subset_v_sc = sc_v.transform(cell_test_subset_v)
```

```
# use 5-nearest neighbor
from sklearn.neighbors import NearestNeighbors

nbrs = NearestNeighbors(n_neighbors = 5)
nbrs.fit(cell_train_f_sc) # based on training set
distance, indices = nbrs.kneighbors(cell_test_subset_f_sc) # neighbors for test set

# imputation
cell_test_subset_v_pred_knn = np.empty(50)
for idx, i in enumerate(indices):
    cell_test_subset_v_pred_knn[idx] = np.mean(cell_train_v_sc[i[1:]])
```

Find the predictor with highest correlation.

```
from scipy.stats.stats import pearsonr

print "corr('VarIntenCh3', 'DiffIntenDensityCh3') is {0}".format(pearsonr(cell_train_v, cell_train_f['DiffIntenDensityCh3'])[0])
```

    corr('VarIntenCh3', 'DiffIntenDensityCh3') is 0.894871504785

```
# use linear model
from sklearn.linear_model import LinearRegression

lm = LinearRegression()
lm.fit(cell_train_f_sc[:, cell_train_f.columns.get_loc('DiffIntenDensityCh3')][:, np.newaxis],
       cell_train_v_sc[:, np.newaxis]) # find the predictor with highest correlation
cell_test_subset_v_pred_lm = \
lm.predict(cell_test_subset_f_sc[:, cell_train_f.columns.get_loc('DiffIntenDensityCh3')][:, np.newaxis])
```

Correlation between the real and imputed values

```
print "kNN: {0}".format(pearsonr(cell_test_subset_v_sc, cell_test_subset_v_pred_knn)[0])
print "Linear Model: {0}".format(pearsonr(cell_test_subset_v_sc[:, np.newaxis], cell_test_subset_v_pred_lm)[0][0])
```

    kNN: 0.869781133553
    Linear Model: 0.858789107105

Note that the better performance of linear model is because of the high correlation (0.895) between these two predictors. kNN is generally more robust since it takes all predictors into consideration.

```
fig, (ax1, ax2) = plt.subplots(1, 2)

ax1.scatter(cell_test_subset_v_sc, cell_test_subset_v_pred_knn)
ax1.set(xlim=(-1.5, 3), ylim=(-1.5, 3))
ax1.plot(ax1.get_xlim(), ax1.get_ylim(), ls="--", c=".3")
ax1.set_title('5NN')

ax2.scatter(cell_test_subset_v_sc, cell_test_subset_v_pred_lm)
ax2.set(xlim=(-1.5, 3), ylim=(-1.5, 3))
ax2.plot(ax2.get_xlim(), ax2.get_ylim(), ls="--", c=".3")
ax2.set_title('Linear Model')

fig.text(0.5, 0.04, 'Original Value (centered and scaled)', ha='center', va='center')
fig.text(0.06, 0.5, 'Imputed', ha='center', va='center', rotation='vertical')
```

## 3.5 Removing Predictors

There are potential advantages to removing predictors prior to modeling. First, fewer predictors means decreased computational time and complexity. Second, if two predictors are highly correlated, this implies that they are measuring the same underlying information. Removing one should not compromise the performance of the model and might lead to a more parsimonious and interpretable model. Third, some models can be crippled by predictors with degenerate distributions, e.g. near-zero variance predictors. In these cases, there can be a significant improvement in model performance and/or stability without the problematic variables.

A rule of thumb for detecting near-zero variance predictors:
- The fraction of unique values over the sample size is low (say 10%)
- The ratio of the frequency of the most prevalent value to the frequency of the second most prevalent value is large (say around 20)

If both of these criteria are true and the model in question is susceptible to this type of predictor, it may be advantageous to remove the variable from the model.

### Between-Predictor Correlations

*Collinearity* is the technical term for the situation where a pair of predictor variables have a substantial correlation with each other. It is also possible to have relationships between multiple predictors at once (called *multicollinearity*).

A direct visualization of the correlation matrix from the training set.

```
# calculate the correlation matrix
corr_dataframe = cell_train_feature.corr()

# compute hierarchical cluster on both rows and columns for correlation matrix and plot heatmap
def corr_heatmap(corr_dataframe):
    import scipy.cluster.hierarchy as sch

    corr_matrix = np.array(corr_dataframe)
    col_names = corr_dataframe.columns

    Y = sch.linkage(corr_matrix, 'single', 'correlation')
    Z = sch.dendrogram(Y, color_threshold=0, no_plot=True)['leaves']
    corr_matrix = corr_matrix[Z, :]
    corr_matrix = corr_matrix[:, Z]
    col_names = col_names[Z]
    im = plt.imshow(corr_matrix, interpolation='nearest', aspect='auto', cmap='bwr')
    plt.colorbar()
    plt.xticks(range(corr_matrix.shape[0]), col_names, rotation='vertical', fontsize=4)
    plt.yticks(range(corr_matrix.shape[0]), col_names[::-1], fontsize=4)

# plot
corr_heatmap(corr_dataframe)
```

Note that the predictor variables have been grouped using a clustering technique so that collinear groups of predictors are adjacent to one another.

When the data set consists of too many predictors to examine visually, techniques such as PCA can be used to characterize the magnitude of the problem. For example, if the first principal component accounts for a large percentage of the variance, this implies that there is at least one group of predictors that represent the same information. The PCA loadings can be used to understand which predictors are associated with each component to tease out this relationship.

In general, there are good reasons to avoid data with highly correlated predictors. First, redundant predictors frequently add more complexity to the model than information they provide to the model. In situations where obtaining the predictor data is costly, fewer variables is obviously better. Using highly correlated predictors in techniques like linear regression can result in highly unstable models, numerical values, and degraded predictive performances.

Classical regression analysis has several tools to diagnose multicollinearity for linear regression. A statistic called the variance inflation factor (VIF) can be used to identify predictors that are impacted. A common rule of thumb is that if VIF > 5, then multicollinearity is high. Note that this method is developed for linear models, it requires more samples than predictor variables and it does not determine which should be removed to resolve the problem

A more heuristic approach is to remove the minimum number of predictors to ensure that all pairwise correlation are below a certain threshold. The algorithm is as follows:
- Calculate the correlation matrix of the predictors.
- Determine the two predictors associated with the largest absolute pairwise correlation (A and B).
- Determine the average absolute correlation between A and the other variables. Do the same for predictor B.
- If A has a larger average correlation, remove it; otherwise, remove predictor B.
- Repeat Steps 2-4 until no absolute correlations are above the threshold.

Suppose we wanted to use a model that is particularly sensitive to between predictor correlations, we might apply a threshold of 0.75.

As previously mentioned, feature extraction methods (e.g., principal components) are another technique for mitigating the effect of strong correlations between predictors. However, these techniques make the connection between the predictors and the outcome more complex. Additionally, since signal extraction methods are usually unsupervised, there is no guarantee that the resulting surrogate preditors have any relationship with the outcome.

## 3.6 Adding Predictors

When a predictor is categorical, it is common to decompose the predictor into a set of more specific variables.

Look at the following example for the credit scoring data.

```
!ls -l ../datasets/GermanCredit/
```

    total 480
    -rw-r--r--  1 leigong  staff  244776 Nov 24 15:58 GermanCredit.csv

```
credit_data = pd.read_csv("../datasets/GermanCredit/GermanCredit.csv")
credit_data.head(5)
```

      Unnamed: 0
      Duration
      Amount
      InstallmentRatePercentage
      ResidenceDuration
      Age
      NumberExistingCredits
      NumberPeopleMaintenance
      Telephone
      ForeignWorker
      ...
      OtherInstallmentPlans.Bank
      OtherInstallmentPlans.Stores
      OtherInstallmentPlans.None
      Housing.Rent
      Housing.Own
      Housing.ForFree
      Job.UnemployedUnskilled
      Job.UnskilledResident
      Job.SkilledEmployee
      Job.Management.SelfEmp.HighlyQualified

      0
       1
        6
       1169
       4
       4
       67
       2
       1
       0
       1
      ...
       0
       0
       1
       0
       1
       0
       0
       0
       1
       0

      1
       2
       48
       5951
       2
       2
       22
       1
       1
       1
       1
      ...
       0
       0
       1
       0
       1
       0
       0
       0
       1
       0

      2
       3
       12
       2096
       2
       3
       49
       1
       2
       1
       1
      ...
       0
       0
       1
       0
       1
       0
       0
       1
       0
       0

      3
       4
       42
       7882
       2
       4
       45
       1
       2
       1
       1
      ...
       0
       0
       1
       0
       0
       1
       0
       0
       1
       0

      4
       5
       24
       4870
       3
       4
       53
       2
       2
       1
       1
      ...
       0
       0
       1
       0
       0
       1
       0
       0
       1
       0

5 rows × 63 columns

```
credit_data.shape
```

    (1000, 63)

The predictor based on how much money was in the applicant's saving account is categorical coded into dummy variables.

```
credit_data_saving = credit_data[['SavingsAccountBonds.lt.100', 'SavingsAccountBonds.100.to.500',
                                  'SavingsAccountBonds.500.to.1000', 'SavingsAccountBonds.gt.1000',
                                  'SavingsAccountBonds.Unknown']]
credit_data_saving.head(10)
```

      SavingsAccountBonds.lt.100
      SavingsAccountBonds.100.to.500
      SavingsAccountBonds.500.to.1000
      SavingsAccountBonds.gt.1000
      SavingsAccountBonds.Unknown

      0
       0
       0
       0
       0
       1

      1
       1
       0
       0
       0
       0

      2
       1
       0
       0
       0
       0

      3
       1
       0
       0
       0
       0

      4
       1
       0
       0
       0
       0

      5
       0
       0
       0
       0
       1

      6
       0
       0
       1
       0
       0

      7
       1
       0
       0
       0
       0

      8
       0
       0
       0
       1
       0

      9
       1
       0
       0
       0
       0

```
credit_data_saving.apply(np.sum)
```

    SavingsAccountBonds.lt.100         603
    SavingsAccountBonds.100.to.500     103
    SavingsAccountBonds.500.to.1000     63
    SavingsAccountBonds.gt.1000         48
    SavingsAccountBonds.Unknown        183
    dtype: float64

| Value    |  n   | <100 | 100-500 | 500-1000 | >1000 | Unknown |
|:---------|:----:|:----:|:-------:|:--------:|:-----:|:-------:|
| < 100    | 603  |   1  |   0     |    0     |   0   |    0    |
| 100-500  | 100  |   0  |   1     |    0     |   0   |    0    |
| 500-1000 |  63  |   0  |   0     |    1     |   0   |    0    |
| >1000    |  48  |   0  |   0     |    0     |   1   |    0    |
| Unknown  | 183  |   0  |   0     |    0     |   0   |    1    |

Usually, each category gets its own dummy variable that is a zero/one indicator for that group. Only four dummy variables are needed here, the fifth can be inferred. However, the decision to include all of the dummy variables can depend on the choice of the model. Models that include an intercept term, such as simple linear model, would have numerical issues if each dummy variable was included in the model. The reason is that, for each sample, these variables all add up to one and this would provide the same information as the intercept. If the model is insensitive to this type of issue, using the complete set of dummy variables would help improve interpretation of the model.

Many of the advanced models automatically generate highly complex, nonlinear relationships between the predictors and the outcome. More simplistic models do not unless the user manually specifices which predictors should be nonlinear and in what way. Another technique to augment the prediction data for classification model is through the "*class centroids*", which are the centers of the predictor data for each class. For each predictor, the distance to each class centroid can be calculated and these distances can be added to the model.

## 3.7 Binning Predictors (to avoid)

There are many issues with the manual binning of continuous data. First, there can be a significant loss of performance in the model. Second, there is a loss of precision in the predictions when the predictors are categorized. Unfortunately, the predictive models that are most powerful are usually the least interpretable. The bottom line is that the perceived improvement in interpretability gained by manual categorization is usually offset by a significant loss in performance.

Note that the argument here is related to the *manual* categorization of predictors prior to model building. There are several models, such as classification/regression trees and multivariate adaptive regression splines, that estimate cut points in the process of model building. The difference between these methodologies and manual binning is that the models ues all the predictors to derive bins based on a single objective (such as maximizing accuracy). They evaluate many variable simultaneously and are usually based on statistically sound methodologies.