< [Biological Computing in Python I](05-Python_I.ipynb) | [Main Contents](Index.ipynb) | [Biological Computing in R](07-R.ipynb) >

# Biological Computing in Python II

>> ...some things in life are bad. They can really make you mad. Other things just make you swear and curse. When you're chewing on life's gristle, don't grumble; give a whistle, and this'll help things turn out for the best. And... always look on the bright side of life...
— Guess who?

Contents

1  Numerical computing in Python

1.1  Indexing and accessing arrays

1.2  Manipulating arrays

1.2.1  Replacing, adding or deleting elements

1.2.2  Flattening or reshaping arrays

1.3  Pre-allocating arrays

1.4  numpy matrices

1.4.1  Matrix-vector operations

2  Two particularly useful scipy sub-packages

2.1  sc.stats

2.2  Numerical integration using  scipy

2.2.1  The Lotka-Volterra model

3  Plotting in Python

4  Practicals

5  The need for speed: profiling code

5.1  Profiling in Python

5.2  Quick profiling with timeit

6  Practicals

6.1  Lotka-Volterra model problem

6.2  Extra Credit problems

7  Networks in Python

7.1  Food web network example

8  Practicals

9  Regular expressions in Python

9.1  Metacharacters vs. regular characters

9.2  regex elements

9.3  Regex in python

10  Practicals: Some RegExercises

10.1  Grouping regex patterns

11  Useful re commands

11.1  Finding all matches

11.2  Finding in files

11.3  Groups within multiple matches

11.4  Extracting text from webpages

11.5  Replacing text

12  Practicals

12.1  Blackbirds problem

13  Using Python to build workflows

13.1  Using subprocess

13.1.1  Running processes

13.2  Handling directory and file paths

13.3  Running R

14  Practicals

14.1  Using os problem 1

14.2  Using os problem 2

15  Readings and Resources

In this chapter, we will build on the [first Python Chapter](05-Python_I.ipynb). We cover some more advanced topics that will round-off your training in Biological Computing in Python.

## Numerical computing in Python

The python package `scipy` allows you to do serious number crunching, including:

* Linear algebra (matrix and vector operations)
* Numerical integration (solving ODEs)
* Fourier transforms
* Interpolation
* Calculating special functions (incomplete Gamma, Bessel, etc.)
* Generation of random numbers
* Using statistical functions and transformations

In the following, we will use the `numpy array` data structure for data manipulations and calculations. These
arrays are similar in some respects to python lists, but are more naturally multidimensional, homogeneous in type (the default is float), and allow efficient (fast) manipulations. Thus numpy arrays are analogous to the R `matrix` data object/structure. We will use the `scipy` package, which includes `numpy`, and lot more.

So let's try `scipy`:

```python
import scipy as sc
```

```python
a = sc.array(range(5)) # a one-dimensional array
a
```

    array([0, 1, 2, 3, 4])

```python
print(type(a))
```

    <class 'numpy.ndarray'>

```python
print(type(a[0]))
```

    <class 'numpy.int64'>

Thus the last two outputs tell you that firstly, there is a data structure type (and a class) called `numpy.ndarray`, and secondly, that at position `0` (remember, Python indexing starts at 0) it holds an [64 bit integer](https://en.wikipedia.org/wiki/9,223,372,036,854,775,807). All elements in `a` will be of type `int` because that is what `range()` returns (try `?range`).

            (Source: [http://pages.physics.cornell.edu/~myers/teaching/ComputationalMethods/python/arrays.html](http://pages.physics.cornell.edu/~myers/teaching/ComputationalMethods/python/arrays.html))

            A graphical depiction of numpy/numpy arrays, which can have multiple dimensions (even greater than 3).

You can also specify the data type of the array:

```python
a = sc.array(range(5), float)
a
```

    array([ 0.,  1.,  2.,  3.,  4.])

```python
a.dtype # Check type
```

    dtype('float64')

You can also get a 1-D arrays as follows:

```python
x = sc.arange(5)
x
```

    array([0, 1, 2, 3, 4])

```python
x = sc.arange(5.) #directly specify float using decimal
x
```

    array([ 0.,  1.,  2.,  3.,  4.])

As with other Python variables (e.g., created as a list or a dictionary), you can apply methods to variables created as numpy arrays. For example, type `x.` and hit TAB to see all methods you can apply to`x`. To see dimensions of `x`:

```python
x.shape
```

    (5,)

Remember, you can type `:?x.methodname` to get info on a particular method. For example, try `?x.shape`.

You can also convert to and from Python lists:

```python
b = sc.array([i for i in range(10) if i%2==1]) #odd numbersbetween 1 and `0
b
```

    array([1, 3, 5, 7, 9])

```python
c = b.tolist() #convert back to list
c
```

    [1, 3, 5, 7, 9]

To make a matrix, you need a 2-D numpy array:

```python
mat = sc.array([[0, 1], [2, 3]])
mat
```

    array([[0, 1],
           [2, 3]])

```python
mat.shape
```

    (2, 2)

### Indexing and accessing arrays

As with other Python data objects such as lists, numpy array elements can be accessed using square brackets (`[ ]`) with the usual `[row,column]` reference. Indexing of numpy arrays works like that for other data structures, with index values starting at 0. So, you can obtain all the elements of a particular row as:

```python
mat[1] # accessing whole 2nd row, remember indexing starts at  0
```

    array([2, 3])

```python
mat[:,1] #accessing whole second column
```

    array([1, 3])

And accessing particular elements:

```python
mat[0,0] # 1st row, 1st column element
```

    0

```python
mat[1,0] # 2nd row, 1st column element
```

    2

Note that (like all other programming languages) row index always comes before column index. That is, `mat[1]` is always going to mean "whole second row", and `mat[1,1]` means 1st row and 1st column element. Therefore, to access the whole second column, you need:

```python
mat[:,0] #accessing whole first column
```

    array([0, 2])

Python indexing also accepts negative values for going back to the start
from the end of an array:

```python
mat[0,1]
```

    1

```python
mat[0,-1] #interesting!
```

    1

```python
mat[0,-2] #very interesting, but rather useless for this simple matrix!
```

    0

### Manipulating arrays

Manipulating numpy arrays is pretty straightforward.

---
> **Why numpy arrays are computationally efficient:**  The data associated with a numpy array object (its metadata  – number of dimensions, shape, data type, etc – as well as the actual data) are stored in a homogeneous and contiguous block of memory (a "data buffer"), at a particular address in the system's RAM (Random Access Memory). This makes numpy arrays more efficient than a pure Python data structures like lists whose data are scattered across the system memory.

---

#### Replacing, adding or deleting elements

Let's look at how you can replace, add, or delete an array element (a single entry, or whole row(s) or whole column(s)):

```python
mat[0,0] = -1 #replace a single element
mat
```

    array([[-1,  1],
           [ 2,  3]])

```python
mat[:,0] = [12,12] #replace whole column
mat
```

    array([[12,  1],
           [12,  3]])

```python
sc.append(mat, [[12,12]], axis = 0) #append row, note axis specification
```

    array([[12,  1],
           [12,  3],
           [12, 12]])

```python
sc.append(mat, [[12],[12]], axis = 1) #append column
```

    array([[12,  1, 12],
           [12,  3, 12]])

```python
newRow = [[12,12]] #create new row
```

```python
mat = sc.append(mat, newRow, axis = 0) #append that existing row
mat
```

    array([[12,  1],
           [12,  3],
           [12, 12]])

```python
sc.delete(mat, 2, 0) #Delete 3rd row
```

    array([[12,  1],
           [12,  3]])

And concatenation:

```python
mat = sc.array([[0, 1], [2, 3]])
mat0 = sc.array([[0, 10], [-1, 3]])
sc.concatenate((mat, mat0), axis = 0)
```

    array([[ 0,  1],
           [ 2,  3],
           [ 0, 10],
           [-1,  3]])

#### Flattening or reshaping arrays

You can also "flatten" or "melt" arrays, that is, change array dimensions (e.g., from a matrix to a vector):

```python
mat.ravel() # NOTE: ravel is row-priority - happens row by row
```

    array([0, 1, 2, 3])

```python
mat.reshape((4,1)) # this is different from ravel - check ?sc.reshape
```

    array([[0],
           [1],
           [2],
           [3]])

```python
mat.reshape((1,4)) # NOTE: reshaping is also row-priority
```

    array([[0, 1, 2, 3]])

```python
mat.reshape((3, 1)) # But total elements must remain the same!
```

This is a bit different than how [`R` behaves](07-R.ipynb#Recycling), where you won't get an error (R "recycles" data), which can be dangerous!

### Pre-allocating arrays

As in other computer languages, it is usually more efficient to preallocate an array rather than append / insert / concatenate additional elements, rows, or columns. *why*? – because you might run out of contiguous space in the specific system memory (RAM) address where the current array is stored. Preallocation allocates all the RAM memory you need in one call, while resizing the array (through append,insert,concatenate, resize, etc) may require copying the array to a larger block of memory, slowing things down, and significantly so if the matrix/array is very large.

For example, if you know the size of your matrix or array, you can initialize it with ones or zeros:

```python
sc.ones((4,2)) #(4,2) are the (row,col) array dimensions
```

    array([[ 1.,  1.],
           [ 1.,  1.],
           [ 1.,  1.],
           [ 1.,  1.]])

```python
sc.zeros((4,2)) # or zeros
```

    array([[ 0.,  0.],
           [ 0.,  0.],
           [ 0.,  0.],
           [ 0.,  0.]])

```python
m = sc.identity(4) #create an identity matrix
m
```

    array([[ 1.,  0.,  0.,  0.],
           [ 0.,  1.,  0.,  0.],
           [ 0.,  0.,  1.,  0.],
           [ 0.,  0.,  0.,  1.]])

```python
m.fill(16) #fill the matrix with 16
m
```

    array([[ 16.,  16.,  16.,  16.],
           [ 16.,  16.,  16.,  16.],
           [ 16.,  16.,  16.,  16.],
           [ 16.,  16.,  16.,  16.]])

### `numpy` matrices

Scipy/Numpy also has a `matrix` data structure class. Numpy matrices are strictly 2-dimensional, while numpy arrays are N-dimensional. Matrix objects are a subclass of numpy arrays, so they inherit all the attributes and methods of numpy arrays (ndarrays).

The main advantage of scipy matrices is that they provide a convenient notation for matrix multiplication: if `a` and `b` are matrices, then `a * b` is their matrix product.

#### Matrix-vector operations

Now let's perform some common matrix-vector operations on arrays (you can also try the same using matrices instead of arrays):

```python
mm = sc.arange(16)
mm = mm.reshape(4,4) #Convert to matrix
mm
```

    array([[ 0,  1,  2,  3],
           [ 4,  5,  6,  7],
           [ 8,  9, 10, 11],
           [12, 13, 14, 15]])

```python
mm.transpose()
```

    array([[ 0,  4,  8, 12],
           [ 1,  5,  9, 13],
           [ 2,  6, 10, 14],
           [ 3,  7, 11, 15]])

```python
mm + mm.transpose()
```

    array([[ 0,  5, 10, 15],
           [ 5, 10, 15, 20],
           [10, 15, 20, 25],
           [15, 20, 25, 30]])

```python
mm - mm.transpose()
```

    array([[ 0, -3, -6, -9],
           [ 3,  0, -3, -6],
           [ 6,  3,  0, -3],
           [ 9,  6,  3,  0]])

```python
mm * mm.transpose() ## Note: Elementwise multiplication!
```

    array([[  0,   4,  16,  36],
           [  4,  25,  54,  91],
           [ 16,  54, 100, 154],
           [ 36,  91, 154, 225]])

```python
mm // mm.transpose()
```

    /usr/local/lib/python3.5/dist-packages/ipykernel_launcher.py:1: RuntimeWarning: divide by zero encountered in floor_divide
      """Entry point for launching an IPython kernel.

    array([[0, 0, 0, 0],
           [4, 1, 0, 0],
           [4, 1, 1, 0],
           [4, 1, 1, 1]])

Note that we used integer division `//`. Note also the warning you get(because of zero division). So let's avoid the divide by zero:

```python
mm // (mm+1).transpose()
```

    array([[0, 0, 0, 0],
           [2, 0, 0, 0],
           [2, 1, 0, 0],
           [3, 1, 1, 0]])

```python
mm * sc.pi
```

    array([[  0.        ,   3.14159265,   6.28318531,   9.42477796],
           [ 12.56637061,  15.70796327,  18.84955592,  21.99114858],
           [ 25.13274123,  28.27433388,  31.41592654,  34.55751919],
           [ 37.69911184,  40.8407045 ,  43.98229715,  47.1238898 ]])

```python
mm.dot(mm) # MATRIX MULTIPLICATION
```

    array([[ 56,  62,  68,  74],
           [152, 174, 196, 218],
           [248, 286, 324, 362],
           [344, 398, 452, 506]])

```python
mm = sc.matrix(mm) # convert to scipy matrix class
mm
```

    matrix([[ 0,  1,  2,  3],
            [ 4,  5,  6,  7],
            [ 8,  9, 10, 11],
            [12, 13, 14, 15]])

```python
print(type(mm))
```

    <class 'numpy.matrixlib.defmatrix.matrix'>

```python
mm * mm # now matrix multiplication is syntactically easier
```

    matrix([[ 56,  62,  68,  74],
            [152, 174, 196, 218],
            [248, 286, 324, 362],
            [344, 398, 452, 506]])

We can do a lot more by importing the `linalg` sub-package: `sc.linalg`. Try it.

## Two particularly useful `scipy` sub-packages

Two particularly useful `scipy` sub-packages are `sc.integrate` (*what will I need this for?*) and `sc.stats`. *Why not use `R` for this?* — because often you might just want to calculate some summary stats of your simulation results within Python.

### `sc.stats`

Let's take a quick spin in `sc.stats`.

```python
import scipy.stats
```

```python
scipy.stats.norm.rvs(size = 10) # 10 samples from N(0,1)
```

    array([-1.18737619,  0.97466982,  0.24744395, -0.13007556, -1.91779308,
            1.39418314,  0.66301452,  2.50881111,  0.79540968, -0.35377855])

```python
scipy.stats.randint.rvs(0, 10, size =7) # Random integers between 0 and 10
```

    array([3, 7, 4, 2, 1, 8, 1])

### Numerical integration using  `scipy`

Numerical integration is the approximate computation of an integral using numerical techniques. You need numerical integration whenever you have a complicated function that cannot be integrated analytically using anti-derivatives. For example, calculating the area under a curve A particularly useful application is solving ordinary differential equations (ODEs), commonly used for modelling biological systems.

#### The Lotka-Volterra model

Let's try numerical integration in Python for solving a classical model in biology — the Lotka-Volterra model for a predator-prey system in two-dimensional space (e.g., on land).

The Lotka-Volterra (LV) model is:

\begin{aligned}
    \frac{dR}{dt} &= r R - a C R \\
    \frac{dC}{dt} &= - z C + e a C R
\end{aligned}

where $C$ and $R$ are consumer (e.g., predator) and resource (e.g., prey) population abundance (either number $\times$ area$^{-1}$ ), $r$ is the intrinsic (per-capita) growth rate of the resource population (time$^{-1}$), $a$ is per-capita "search rate" for the resource ($\text{area}\times \text{time}^{-1}$)  multiplied by its attack success probability, which determines the encounter and consumption rate of the consumer on the resource, $z$ is mortality rate ($\text{time}^{-1}$) and $e$ is the consumer's efficiency (a fraction) in converting resource to consumer biomass.

We have already imported scipy above (`import scipy as sc`) so we can proceed to solve the LV model using numerical integration.

First, import `scipy`'s `integrate` submodule:

```python
import scipy.integrate as integrate
```

Now define a function that returns the growth rate of consumer and resource population at any given time step.

```python
def dCR_dt(pops, t=0):

    R = pops[0]
    C = pops[1]
    dRdt = r * R - a * R * C
    dCdt = -z * C + e * a * R * C

    return sc.array([dRdt, dCdt])
```

```python
type(dCR_dt)
```

    function

Assign some parameter values:

```python
r = 1.
a = 0.1
z = 1.5
e = 0.75
```

Define the time vector; let's integrate from time point 0 to 15, using 1000 sub-divisions of time:

```python
t = sc.linspace(0, 15,  1000)
```

Note that the units of time are arbitrary here.

Set the initial conditions for the two populations (10 resources and 5 consumers per unit area), and convert the two into an array (because our `dCR_dt` function take an array as input).

```python
R0 = 10
C0 = 5
RC0 = sc.array([R0, C0])
```

Now numerically integrate this system forward from those starting conditions:

```python
pops, infodict = integrate.odeint(dCR_dt, RC0, t, full_output=True)
```

```python
pops
```

    array([[ 10.        ,   5.        ],
           [ 10.07578091,   4.94421976],
           [ 10.1529783 ,   4.88948321],
           ...,
           [  9.99869712,  17.56204194],
           [  9.8872779 ,  17.3642589 ],
           [  9.78000354,  17.16658946]])

So `pops` contains the result (the population trajectories).  Also check what's in infodict (it's a  dictionary with additional information)

```python
type(infodict)
```

    dict

```python
infodict.keys()
```

    dict_keys(['mused', 'message', 'tsw', 'nst', 'imxer', 'nje', 'lenrw', 'nfe', 'tcur', 'hu', 'nqu', 'tolsf', 'leniw'])

Check what the `infodict` output is by reading the help documentation with `?scipy.integrate.odeint`. For example, you can return a message to screen about whether the integration was successful:

```python
infodict['message']
```

    'Integration successful.'

So it worked, great! But we would like to visualize the results. Let's do it using the `matplotlib` package.

## Plotting in Python

To visualize the results of your numerical simulations in Python (or for data exploration/analyses), you can use `matplotlib`, which uses Matlab like plotting syntax.

First let's import the package:

```python
import matplotlib.pylab as p
```

Now open empty figure object (analogous to an R graphics object).

```python
f1 = p.figure()
```

    <matplotlib.figure.Figure at 0x7fccb45185c0>

```python
p.plot(t, pops[:,0], 'g-', label='Resource density') # Plot
p.plot(t, pops[:,1]  , 'b-', label='Consumer density')
p.grid()
p.legend(loc='best')
p.xlabel('Time')
p.ylabel('Population density')
p.title('Consumer-Resource population dynamics')
p.show()# To display the figure
```

Finally, save the figure as a pdf:

```python
f1.savefig('../results/LV_model.pdf') #Save figure
```

You can use many any other output formats; check the documentation with `p.savefig`.

## Practicals

1. Create a self-standing script using the above example and save it as `LV1.py` in your code directory. In addition to generating the above figure, it should also generate the following figure:

            Generate this figure as part of the `LV1.py` script.

It should save both figures in pdf to the `results` directory, *without displaying them on screen*.

## The need for speed: profiling code

Donald Knuth says: *Premature optimization is the root of all evil*.

Indeed, computational speed may not be your initial concern. Also, you should focus on developing clean, reliable, reusable code rather than worrying first about how fast your code runs. However, speed will become an issue when and if your analysis or modeling becomes complex enough (e.g., food web or large network simulations). In that case, knowing which parts of your code take the most time is useful – optimizing those parts may save you lots of time.

To find out what is slowing down your code you need to "profile" your code: locate the sections of your code where  speed bottlenecks exist.

### Profiling in Python

Profiling is easy in `ipython` – simply use the command:

```python
%run -p your_function_name
```

Let's write an illustrative program (name it `profileme.py`):
```python
def my_squares(iters):
    out = []
    for i in range(iters):
        out.append(i ** 2)
    return out

def my_join(iters, string):
    out = ''
    for i in range(iters):
        out += string.join(", ")
    return out

def run_my_funcs(x,y):
    print(x,y)
    my_squares(x)
    my_join(x,y)
    return 0

run_my_funcs(10000000,"My string")
```
Now run it with `run -p profileme.py`, and you should see something like:

```bash
        20000063 function calls (20000062 primitive calls) in 9.026 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    3.335    3.335    3.732    3.732 profileme.py:1(my_squares)
        1    2.698    2.698    5.200    5.200 profileme.py:7(my_join)
 10000001    2.502    0.000    2.502    0.000 {method 'join' of 'str' objects}
 10000008    0.397    0.000    0.397    0.000 {method 'append' of 'list' objects}
        1    0.093    0.093    9.025    9.025 profileme.py:13(run_my_funcs)
[more output]
```

Now you can see that the `my_join` function  is hogging most of the time, followed by `my_squares`. Furthermore, its the string method `join` that is clearly slowing `my_join` down, and the list method `append` that is slowing `my_squares`down. In other words, `.join`ing the string again and again, and `.append`ing values to a list are both not particularly fast.

Can we do better? *Yes!*

Let's try this alternative approach to writing the program (save it as `profileme2.py`):

```python
def my_squares(iters):
    out = [i ** 2 for i in range(iters)]
    return out

def my_join(iters, string):
    out = ''
    for i in range(iters):
        out += ", " + string
    return out

def run_my_funcs(x,y):
    print(x,y)
    my_squares(x)
    my_join(x,y)
    return 0

run_my_funcs(10000000,"My string")
```

We did two things: converted the loop to a list comprehension, and replaced the `.join` with an explicit string concatenation.

Now profile this program (`run -p profileme2.py`), and you should get something like:

```bash
        64 function calls (63 primitive calls) in 4.585 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    2.497    2.497    2.497    2.497 profileme2.py:2(<listcomp>)
        1    1.993    1.993    1.993    1.993 profileme2.py:5(my_join)
        1    0.094    0.094    4.584    4.584 profileme2.py:11(run_my_funcs)
[more output]
```

Woo hoo! So we about halved the time! Not quite enough to grab a pint, but ah well...

Another approach would be to preallocate a `numpy` array instead of using a list for `my_squares`. Try it.

### Quick profiling with `timeit`

Alternatively,  you can use the `timeit` module if you want to figure out what the best way to do something specific as part of a larger program (say a particular command or a loop) might be.

Type an run the following code in a python script called `timeitme.py`:

```python
##############################################################################
# loops vs. list comprehensions: which is faster?
##############################################################################

iters = 1000000

import timeit

from profileme import my_squares as my_squares_loops

from profileme2 import my_squares as my_squares_lc

# %timeit my_squares_loops(iters)
# %timeit my_squares_lc(iters)

##############################################################################
# loops vs. the join method for strings: which is faster?
##############################################################################

mystring = "my string"

from profileme import my_join as my_join_join

from profileme2 import my_join as my_join

# %timeit(my_join_join(iters, mystring))
# %timeit(my_join(iters, mystring))
```
Note how we imported the functions using `from profileme import my_squares as my_squares_loops`, etc., which highlights the convenience of Python's elegant object-oriented approach.

Now run the two sets of comparisons using `timeit()` in ipython and make sure every line makes sense. Note that I have commented out the `%timeit()` commands because having a magic command inside a script will not work.

Of course, a simple approach would have been to time the functions like this:

```python
import time
start = time.time()
my_squares_loops(iters)
print("my_squares_loops takes %f s to run." % (time.time() - start))

start = time.time()
my_squares_lc(iters)
print("my_squares_lc takes %f s to run." % (time.time() - start))
```
But you'll notice that if you run it multiple times, the time taken changes each time. So `timeit` takes a sample of runs and returns the average, which is better.

*But remember, don't go crazy with profiling for the sake of shaving a couple of milliseconds, tempting as that may be!*

## Practicals

### Lotka-Volterra model problem

Copy and modify `LV1.py` into another script called `LV2.py` that does the following:

*  Take arguments for the four LV model parameters $r$, $a$, $z$ ,$e$ from the command line:
```
LV2.py arg1 arg2 ... etc
```
*  Runs the Lotka-Volterra model with prey density dependence $r R \left(1 - \frac{R} {K}\right)$, which changes the coupled ODEs to,
\begin{aligned}
    \frac{dR}{dt} &= r R \left(1 - \frac{R} {K}\right) - a C R \\
    \frac{dC}{dt} &= - z C + e a C R
\end{aligned}
*  Saves the plot as `.pdf` in an appropriate location.
*  The chosen parameter values should show in the plot (e.g., $r = 1, a = .5 $, etc) You can change time length $t$ too.
*  Include a script in `code` that will run both `LV1.py` and `LV2.py` with appropriate arguments. This script should also profile the two scripts and print the results to screen for each of the scripts using the `%run -p` approach. Look at and compare the speed bottlenecks in `LV1.py` and `LV2.py`. *Think about how you could further speed up the scripts.*

### Extra Credit problems

*Write every subsequent extra credit script file with a new name such as `LV3.py`,`LV4.py`, etc.*

* **Extra credit**: Choose appropriate values for the parameters such that both predator and prey persist with prey density dependence — the final (non-zero) population values should be printed to screen.

* **Extra-extra credit**: Write a discrete-time version of the LV model called `LV3.py`. The discrete-time model is:

\begin{align}
    R_{t+1} &= R_t (1 + r \left(1 - \frac{R_t}{K}\right) - a C_t)\\
    C_{t+1} &= C_t (1 - z + e a R_t)
\end{align}

Include this script in `run_LV.py`, and profile it as well.

* **Extra-extra-extra credit**: Write a version of the discrete-time model (which you implemented in `LV3.py`) simulation with a random gaussian fluctuation in resource's growth rate at each time-step:

\begin{aligned}
        R_{t+1} &= R_t (1 + (r + \epsilon) \left(1 - \frac{R_t}{K}\right)- a C_t)\\
        C_{t+1} &= C_t (1 - z + e a R_t)
\end{aligned}

where $\epsilon$ is a random fluctuation drawn from a gaussian distribution (use `sc.stats`). Include this
script in ` run_LV.py`, and profile it as well. You can also add fluctuations to both populations simultaneously this way:

\begin{aligned}
    R_{t+1} &= R_t (1 + \epsilon + r +  \left(1 - \frac{R_t}{K}\right) - a C_t)\\
    C_{t+1} &= C_t (1 - z + \epsilon + e a R_t)
\end{aligned}

*As always, test, add, commit and push all your new code and data to your git repository.*

## Networks in Python

ALL biological systems have a network representation, consisting of nodes for the biological entities of interest, and edges or links for the relationships between them. Here are some examples:
* Metabolic networks
* Gene regulatory networks
* Individual-Individual (e.g., social networks)
* Who-eats-whom (Food web) networks
* Mutualistic (e.g., plant-pollinator) networks

*Can you think of a few more examples from biology?*

You can easily simulate, analyze, and visualize biological networks in both `python` and `R` using some nifty packages. A full network analysis tutorial is out of the scope of our Python module's objectives, but let's try a simple visualization using the ` networkx` python package.

For this you need to first install the package, for example, by using:

```bash
sudo apt-get install python3-networkx
```

### Food web network example

As an example, let's plot a food web network.

The best way to store a food web dataset is as an "adjacency list" of who eats whom: a matrix with consumer name/id in 1st column, and resource name/id in 2nd column, and a separate matrix of species names/ids and properties such as biomass (node's abundance), or average body mass. You will see what these data structures look like below.

First, import the necessary modules:

```python
import networkx as nx
import scipy as sc
import matplotlib.pyplot as p
```

Let's generate a "synthetic" food web. We can do this with the following function that generates a random adjacency list of a $N$-species food web with "connectance probability" $C$: the probability of having a link between any pair of species in the food web.

```python
def GenRdmAdjList(N = 2, C = 0.5):
    """
    """
    Ids = range(N)
    ALst = []
    for i in Ids:
        if sc.random.uniform(0,1,1) < C:
            Lnk = sc.random.choice(Ids,2).tolist()
            if Lnk[0] != Lnk[1]: #avoid self (e.g., cannibalistic) loops
                ALst.append(Lnk)
    return ALst
```

Note that we are using a uniform random distribution between `[0,1]` to generate a connectance probability between each species pair.

Now, let's assign a body mass range (in log scale):

Now assign number of species (`MaxN`) and connectance (`C`):

```python
MaxN = 30
C = 0.75
```

Now generate an adjacency list representing a random food web:

```python
AdjL = sc.array(GenRdmAdjList(MaxN, C))
AdjL
```

    array([[18, 20],
           [15, 29],
           [ 0,  3],
           [12,  3],
           [ 3,  8],
           [ 8, 26],
           [ 8, 17],
           [ 4, 25],
           [ 1, 20],
           [22, 11],
           [19, 15],
           [28, 22],
           [ 8, 29],
           [15,  9],
           [ 7, 15],
           [27, 29],
           [21, 13],
           [14, 15],
           [12, 28],
           [21, 19],
           [11, 23],
           [19, 23]])

So that's what an adjacency list looks like. The two columns of numbers correspond to the consumer and resource ids, respectively.

Now generate species (node) data:

```python
Sps = sc.unique(AdjL) # get species ids
```

Now generate body sizes for the species. We will use a log$_{10}$ scale because species body sizes tend to be [log-normally distributed](08-Data_R.ipynb#Histograms).

```python
SizRan = ([-10,10]) #use log10 scale
Sizs = sc.random.uniform(SizRan[0],SizRan[1],MaxN)
Sizs
```

    array([ 7.9092565 ,  8.28890769,  1.93290082,  5.65741057,  3.20689454,
            2.70967583,  5.00598443,  0.90134135, -8.38467277,  2.64012453,
            4.97781765, -4.00465023, -9.93293439,  7.90936016, -8.22276944,
           -4.86729704,  1.7039879 ,  0.44887105,  0.20853699, -8.99008497,
           -4.74009949, -4.24718942, -9.93293894, -2.73320298,  0.04017755,
           -5.55501357, -6.83177169,  2.72087488, -6.51475447,  3.5965115 ])

Let's visualize the size distribution we have generated.

```python
p.hist(Sizs) #log10 scale
```

```python
p.hist(10 ** Sizs) #raw scale
```

Now let's plot the network, with node sizes proportional to (log) body size:

```python
p.close('all') # close all open plot objects
```

Let's use a circular configuration. For this, we need to calculate the coordinates, easily done using networkx:

```python
pos = nx.circular_layout(Sps)
```

See `networkx.layout` for inbuilt functions to compute other types of node coordinates.

Now generate a networkx graph object:

```python
G = nx.Graph()
```

Now add the nodes and links (edges) to it:

```python
G.add_nodes_from(Sps)
G.add_edges_from(tuple(AdjL)) # this function needs a tuple input
```

Generate node sizes that are proportional to (log) body sizes:

```python
NodSizs= 1000 * (Sizs-min(Sizs))/(max(Sizs)-min(Sizs))
```

Now render (plot) the graph:

```python
nx.draw_networkx(G, pos, node_size = NodSizs)
```

Some of you might get the warning above, or a different one. In that case, just try upgrading the networkx package.

## Practicals

1. Type the above code for plotting a food web network in a program file called `DrawFW.py`. This file should save the plotted network as a pdf.

2. (**Extra Credit**) You can also do nice network visualizations in R. Here you will convert a network visualization script written in `R` using the `igraph` package to a python script that does the same thing.

  * First copy the script file called `Nets.R` and the data files it calls and run it. This script visualizes the [QMEE CDT collaboration network](http://www.imperial.ac.uk/qmee-cdt), coloring the the nodes by the type of node (organization type: "University","Hosting Partner", "Non-hosting Partner").

 * Now, convert this script to a `python` script that does the same thing, including writing to an `.svg` file using the same QMEE CDT link and node data. You can use `networkx` or some other python network visualization package.

## Regular expressions in Python

Let's shift gears now, and look at a very important skill that you should learn, or at least be aware of — *Regular expressions*.

Regular expressions (regex) are a tool to find patterns (not just a particular sequence of characters) in strings. For example, `your@email.com` is a specific sequence of characters, but, in fact, all email addresses have such a pattern: alphanumeric characters, a "@", alphanumeric characters, a ".", alphanumeric characters. Using regex, you can search for all email addresses in a text file by searching for this pattern.

There are many uses of regex, such as:

* Parsing (reading) text files and finding and replacing or deleting specific patterns
* Finding DNA motifs in sequence data
* Navigating through files in a directory
* Extracting information from html and xml files

Thus, if you are interested in data mining, need to clean or process data in any other way, or convert a bunch of information into usable data, knowing regex is absolutely necessary.

            (Source: [www.xkcd.com](https://www.xkcd.com/208/))

            Regular expressions could change your life!

Regex packages are available for most programming languages (recall [`grep` in UNIX](01-Unix.ipynb#Using-`grep`); that is how regex first became popular).

### Metacharacters vs. regular characters

A regex may consist of a combination of "metacharacters" (modifiers) and "regular" or literal characters. There are 14 metacharacters:

[ ] { } ( ) \ ^ $ . | ? * +

These metacharacters do special things, for example:

* `[12]` means match target to *1* and if that does not match then match target to *2*
* `[0-9]` means match to any character in range *0* to *9*
* `[^Ff]` means anything except upper or lower case *f* and `[^a-z]` means everything except lower case *a* to *z*

Everything else is interpreted literally (e.g., *a* is matched by entering `a` in the regex).

`[` and `]`, specify a character "class" — the set of characters that you wish to match. Metacharacters are not active inside classes. For example, [a-z$] will match any of the characters `a` to `z`, but also $, because inside a character class it loses its special metacharacter status.

### regex elements

A useful (not exhaustive) list of regex elements is:

|Regex|Description|
|:-|:-|
|\ | inhibit the "specialness" of a (meta)character so that it can be interpreted literally. So, for example, use `\.` to match a period or `\\` to match a slash|
|`aX9`| match the character string *aX9* exactly (case sensitively)|
|`8`| match the number *8*|
|`\n`| match a newline|
|`\t`| match a tab |
|`\s`| match a whitespace |
|`.`| match any character except line break (newline)|
|`\w`|  match a single "word" character: any alphanumeric character (including underscore)|
|`\W`| match any character not covered by `\w`, i.e., match any non-alphanumeric character excluding underscore, such as `?`, `!`, `+`, `<`, etc. |
|`\d`| match a numeric (integer) character|
|`\D`| match any character not covered by ` \d` (i.e., match a non-digit)|
|`[atgc]` | match any character listed: `a`, `t`, `g`, `c`|
| at|gc  | match `at` or `gc`|
|`[^atgc]`| match any character not listed: any character except `a`, `t`, `g`, `c`|
|`?`| match the preceding pattern element zero or one times|
|*|match the preceding pattern element zero or more times|
|`+`| match the preceding pattern element one or more times|
|`{n}`| match the preceding pattern element exactly `n` times|
|`{n,}`| match the preceding pattern element at least `n` times|
|`{n,m}`| match the preceding pattern element at least `n` but not more than `m` times|
|`^`| match the start of a string|
|$| match the end of a string|

### Regex in `python`

Regex functions in python are in the module `re`.

Let's import it:

```python
import re
```

The simplest `python` regex function is `re.search`, which searches the string for match to a given pattern — returns a *match object* if a match is found and `None` if not. Thus, the command `match = re.search(pat, str)` finds matches of the pattern `pat` in the given string `str` and stores the search result in a variable named `match`.

> **Always** put `r` in front of your regex — it tells python to read the regex in its "raw" (literal) form. Without raw string notation (`r"text"`), every backslash (`\`) in a regular expression would have to be prefixed with another one to escape it. Read more about this [here](https://docs.python.org/3.5/library/re.html).

OK, let's try some regexes (type all that follows in `regexs.py`):

```python
my_string = "a given string"
```

Find a space in the string:

```python
match = re.search(r'\s', my_string)
print(match)
```

    <_sre.SRE_Match object; span=(1, 2), match=' '>

That's only telling you that a match was found (the object was created successfully).

To see the match, use:

```python
match.group()
```

    ' '

Now let's try another pattern:

```python
match = re.search(r'\d', my_string)
```

```python
print(match)
```

    None

No surprise, because there are no numeric characters in our string!

To know whether a pattern was matched, we can use an `if`:

```python
MyStr = 'an example'

match = re.search(r'\w*\s', MyStr) # what pattern is this?

if match:
    print('found a match:', match.group())
else:
    print('did not find a match')
```

    found a match: an

Here are some more regexes (add all that follows to `regexs.py`):

```python
match = re.search(r'2' , "it takes 2 to tango")
match.group()
```

    '2'

```python
match = re.search(r'\d' , "it takes 2 to tango")
match.group()
```

    '2'

```python
match = re.search(r'\d.*' , "it takes 2 to tango")
match.group()
```

    '2 to tango'

```python
match = re.search(r'\s\w{1,3}\s', 'once upon a time')
match.group()
```

    ' a '

```python
match = re.search(r'\s\w*$', 'once upon a time')
match.group()
```

    ' time'

Let's switch to a more compact syntax by directly returning the matched group (by directly appending `.group()` to the result).

```python
re.search(r'\w*\s\d.*\d', 'take 2 grams of H2O').group()
```

    'take 2 grams of H2'

```python
re.search(r'^\w*.*\s', 'once upon a time').group() # 'once upon a '
```

    'once upon a '

 Note that *, `+`, and `{ }` are all "greedy":  They repeat the previous regex token as many times as possible.

As a result, they may match more text than you want. To make it non-greedy and terminate at the first found instance of a pattern, use `?`:

```python
re.search(r'^\w*.*?\s', 'once upon a time').group()
```

    'once '

To further illustrate greediness in regexes, let's try matching an HTML tag:

```python
re.search(r'<.+>', 'This is a <EM>first</EM> test').group()
```

    'first'

But we wanted just `<EM>`!

It's because `+` is greedy. Instead, we can make `+` "lazy":

```python
re.search(r'<.+?>', 'This is a <EM>first</EM> test').group()
```

    ''

OK, moving on from greed and laziness...

```python
re.search(r'\d*\.?\d*','1432.75+60.22i').group()
```

    '1432.75'

Note `\` before the `.`, to be able to find a literal `.`

Otherwise, `re.search` will consider it to be a regex element (`.` means "match any character except newline").

A couple more examples:

```python
re.search(r'[AGTC]+', 'the sequence ATTCGT').group()
```

    'ATTCGT'

```python
re.search(r'\s+[A-Z]\w+\s*\w+', "The bird-shit frog's name is Theloderma asper.").group()
```

    ' Theloderma asper'

            In case you were wondering what *Theloderma asper*, the "bird-shit frog", looks like. I snapped this one in North-east India ages ago

How about looking for email addresses in a string? For example, let's try matching a string consisting of an academic's name, email address and research area or interest (no need to type this into any python file):

```python
MyStr = 'Samraat Pawar, s.pawar@imperial.ac.uk, Systems biology and ecological theory'
match = re.search(r"[\w\s]+,\s[\w\.@]+,\s[\w\s]+",MyStr)
match.group()
```

    'Samraat Pawar, s.pawar@imperial.ac.uk, Systems biology and ecological theory'

Note the use of `[ ]`'s: for example, `[\w\s]` ensures that any combination of word characters and spaces is found.

Let's see if this regex works on a different pattern of email addresses:

```python
MyStr = 'Samraat Pawar, s-pawar@imperial.ac.uk, Systems biology and ecological theory'
```

```python
match = re.search(r"[\w\s]+,\s[\w\.@]+,\s[\w\s&]+",MyStr)
match.group()
```

Nope! So let's make the email address part of the regex more robust:

```python
match = re.search(r"[\w\s]+,\s[\w\.-]+@[\w\.-]+,\s[\w\s&]+",MyStr)
match.group()
```

    'Samraat Pawar, s-pawar@imperial.ac.uk, Systems biology and ecological theory'

## Practicals: Some RegExercises

The following exercises are not for submission as part of your coursework, but we will discuss them in class on a subsequent day.

1. Try the regex we used above for finding names (`[\w\s]+`) for cases where the person's name has something unexpected, like a `?` or a `+`. Does it work? How can you make it more robust?
* Translate the following regular expressions into regular English:
    * `r'^abc[ab]+\s\t\d'`
    * `r'^\d{1,2}\/\d{1,2}\/\d{4}$'`
    * `r'\s*[a-zA-Z,\s]+\s*'`
* Write a regex to match dates in format YYYYMMDD, making sure that:
    * Only seemingly valid dates match (i.e., year greater than 1900)
    * First digit in month is either 0 or 1
    * First digit in day $\leq 3$

### Grouping regex patterns

You can group regex patterns into meaningful blocks using parentheses. Let's look again at the example of finding  email addresses.

```python
MyStr = 'Samraat Pawar, s.pawar@imperial.ac.uk, Systems biology and ecological theory'
match = re.search(r"[\w\s]+,\s[\w\.-]+@[\w\.-]+,\s[\w\s&]+",MyStr)
match.group()
```

    'Samraat Pawar, s.pawar@imperial.ac.uk, Systems biology and ecological theory'

Without grouping the regex:

```python
match.group(0)
```

    'Samraat Pawar, s.pawar@imperial.ac.uk, Systems biology and ecological theory'

Now create groups using `( )`:

```python
match = re.search(r"([\w\s]+),\s([\w\.-]+@[\w\.-]+),\s([\w\s&]+)",MyStr)
if match:
    print(match.group(0))
    print(match.group(1))
    print(match.group(2))
    print(match.group(3))
```

    Samraat Pawar, s.pawar@imperial.ac.uk, Systems biology and ecological theory
    Samraat Pawar
    s.pawar@imperial.ac.uk
    Systems biology and ecological theory

Nice! This is very handy for extracting specific patterns from text data. Note that we excluded the `,`'s and the `\s`'s from the grouping parentheses because we don't want them to be returned in the match group list.

Have a look at `re4.py` in the TheMulQuaBio's code repository for more on parsing email addresses using regexes.

## Useful `re` commands

Here are some important functions in the `re` module:

|Command|What it does|
|:-|:-|
| `re.search(reg, text)`| Scans the string and finds the first match of the pattern, returning a `match` object if successful and `None` otherwise.|
| `re.match(reg, text)`| Like `re.search`, but only matches the beginning of the string.|
| `re.compile(reg)`| Compiles (stores) a regular expression for repeated use, improving efficiency.|
| `re.split(ref, text)`| Splits the text by the occurrence of the pattern described by the regular expression.|
| `re.findall(ref, text)`| Like `re.search`, but returns a list of all matches. If groups are present, returns a list of groups.|
| `re.finditer(ref, text)`| Like `re.findall`, but returns an iterator containing the match objects over which you can iterate. Useful for "crawling" efficiently through text till you find all necessary number of matches.|
| `re.sub(ref, repl, text)`|  Substitutes each non-overlapping occurrence of the match with the text in `repl`.|
|||

Many of these commands also work on whole contents of files. We will look at an example of this below. Let us try some are particularly useful applications of some of these commands.

### Finding all matches

Above we used re.search() to find the first match for a pattern. In many scenarios, you will need to find *all* the matches of a pattern. The function `re.findall()` does precisely this and returns all matches as a list of strings, with each string representing one match.

Let's try this on an extension of the email example above for some data with multiple addresses:

```python
MyStr = "Samraat Pawar, s.pawar@imperial.ac.uk, Systems biology and ecological theory; Another academic, a-academic@imperial.ac.uk, Some other stuff thats equally boring; Yet another academic, y.a_academic@imperial.ac.uk, Some other stuff thats even more boring"
```

Now `re.findall()` returns a list of all the emails found:

```python
emails = re.findall(r'[\w\.-]+@[\w\.-]+', MyStr)
for email in emails:
    print(email)
```

    s.pawar@imperial.ac.uk
    a-academic@imperial.ac.uk
    y.a_academic@imperial.ac.uk

Nice!

### Finding in files

You will generally be wanting to apply regex searches to whole files. You might be tempted to write a loop to iterate over the lines of the file, calling `re.findall()` on each line. However, `re.findall()` can return a list of all the matches in a single step.

Let's try finding all species names that correspond to Oaks in a data file:

```python
f = open('../data/TestOaksData.csv', 'r')
found_oaks = re.findall(r"Q[\w\s].*\s", f.read())

found_oaks
```

    ['Quercus, robur\n', 'Quercus, cerris\n', 'Quercus, petraea\n']

```python
for name in strings:
    print(name.replace(",",""))
```

This works because recall that `f.read()` returns the whole text of a file in a single string). Also, the file is closed after reading.

### Groups within multiple matches

Grouping pattern matches using `( )` as you learned above, can be combined with `re.findall()`. If the pattern includes *two or more* groups, then instead of returning a list of strings, `re.findall()` returns a list of tuples. Each tuple represents one match of the pattern, and inside the tuple is group(1), group(2), etc.

Let's try it:

```python
MyStr = "Samraat Pawar, s.pawar@imperial.ac.uk, Systems biology and ecological theory; Another academic, a.academic@imperial.ac.uk, Some other stuff thats equally boring; Yet another academic, y.a.academic@imperial.ac.uk, Some other stuff thats even more boring"

found_matches = re.findall(r"([\w\s]+),\s([\w\.-]+@[\w\.-]+)", MyStr)
found_matches
```

    [('Samraat Pawar', 's.pawar@imperial.ac.uk'),
     (' Another academic', 'a.academic@imperial.ac.uk'),
     (' Yet another academic', 'y.a.academic@imperial.ac.uk')]

```python
for item in found_matches:
    print(item)
```

    ('Samraat Pawar', 's.pawar@imperial.ac.uk')
    (' Another academic', 'a.academic@imperial.ac.uk')
    (' Yet another academic', 'y.a.academic@imperial.ac.uk')

### Extracting text from webpages

OK, let's step up the ante here. How about extracting text from a web page to create your own data? Let's try extracting data from [this page](https://www.imperial.ac.uk/silwood-park/academic-staff/).

You will need a new package `urllib3`. Install it, and import it (also `import re` if needed).

```python
import urllib3
```

```python
conn = urllib3.PoolManager() # open a connection
r = conn.request('GET', 'https://www.imperial.ac.uk/silwood-park/academic-staff/')
webpage_html = r.data #read in the webpage's contents
```

This is returned as bytes (not strings).

```python
type(webpage_html)
```

    bytes

So decode it (remember, the default decoding that this method applies is *utf-8*):

```python
My_Data  = webpage_html.decode()
#print(My_Data)
```

That's a lot of potentially useful information! Let's extract all the names of academics:

```python
pattern = r"Dr\s+\w+\s+\w+"
regex = re.compile(pattern) # example use of re.compile(); you can also ignore case  with re.IGNORECASE
for match in regex.finditer(My_Data): # example use of re.finditer()
    print(match.group())
```

    Dr Arkhat Abzhanov
    Dr Arkhat Abzhanov
    Dr Cristina Banks
    Dr Tom Bell
    Dr Martin Bidartondo
    Dr Martin Bidartondo
    Dr Martin Brazeau
    Dr Lauren Cator
    Dr Matteo Fumagalli
    Dr Matteo Fumagalli
    Dr Richard Gill
    Dr Richard Gill
    Dr Jason Hodgson
    Dr Andrew Knight
    Dr Andrew Knight
    Dr Morena Mills
    Dr Morena Mills
    Dr Samraat Pawar
    Dr Julia Schroeder
    Dr Julia Schroeder
    Dr Joseph Tobias
    Dr Joseph Tobias
    Dr Mike Tristem
    Dr Mike Tristem
    Dr Magda Charalambous
    Dr Magda Charalambous
    Dr Rebecca Kordas
    Dr Rebecca Kordas
    Dr Vassiliki Koufopanou
    Dr Eoin O
    Dr Eoin O
    Dr David Orme
    Dr James Rosindell
    Dr Chris Wilson
    Dr Oliver Windram
    Dr Colin Clubbe
    Dr George McGavin
    Dr George McGavin
    Dr Michael Themis
    Dr Michael Themis

Again, nice! However, its' not perfect. You can improve this by:
* Extracting Prof names as well
* Eliminating the repeated matches
* Grouping to separate title from first and second names
* Extracting names that have unexpected characters (e.g., ``O'Gorman'', which are currently not being matched properly)

*Try making these improvements.*

Of course, you can match and extract other types of patterns as well, such as urls and email addresses (though this example web page does not have email addresses).

### Replacing text

Using the same web page data, let's try using the `re.sub` command on the same web page data (`My_Data`) to replace text:

```python
New_Data = re.sub(r'\t'," ", My_Data) # replace all tabs with a space
# print(New_Data)
```

## Practicals

### Blackbirds problem

Complete the code `blackbirds.py` that you find in the `TheMulQuaBio` (necessary data file is also there).

*As always, test, add, commit and push all your new code and data to your git repository.*

## Using Python to build workflows

You can use python to build an automated data analysis or simulation workflow that involves multiple languages, especially the ones you have already learnt: R, $\LaTeX$, and UNIX bash. For example, you could, in theory, write a single Python script to generate and update your masters dissertation, tables, plots, and all. Python is ideal for building such workflows because it has packages for practically every purpose.

*Thus this topic may be useful for your [Miniproject](Appendix-MiniProj.ipynb), which will involve building a reproducible computational workflow.*

### Using `subprocess`

For building a workflow in Python the `subprocess` module is key. With this module you can run non-Python commands and scripts, obtain their outputs, and also crawl through and manipulate directories.

First, import the module (this is part of the python standard library, so you won't need to install it):

```python
import subprocess
```

#### Running processes

There are two main ways to run commands through subprocess: `run` (available in Python 3.5 onwards) for basic usage, and `Popen` (`P`rocess `open`) for more advanced usage. We will work directly with `popen` because `run()` is a wrapper around `Popen`. Using `Popen` directly gives more control over how the command is run, and how its input and output are processed.

Let's try running some commands in the UNIX bash.

$\star$ In a terminal, first `cd` to your `code` directory,  launch `ipython3`, then and type:

```python
p = subprocess.Popen(["echo", "I'm talkin' to you, bash!"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
```

This creates an object `p`, from which you can extract the output and other information of the command you ran. Before we do anything more, let's look at our `subprocess.popen` call carefully.

* The command line arguments were passed as a list of strings, which avoids the need for escaping quotes or other special characters that might be interpreted by the shell (for example, in this case, there are apostrophes in the string that is being `echo`ed in bash).
* `stdout` is the output from the process "spawned" by your command. This is bytes sequence (which you will need to decode - more on this below).
* `stderr` is the error code (from which you can capture whether the process ran successfully or not). The method PIPE creates a new "pipe" to the "child process".

```python
stdout, stderr = p.communicate()
```

```python
stderr
```

    b''

Nothing here, because the echo command does no return an any code. the `b` indicates that the output is in bits (unencoded). By default, stdout, stderr (and other outputs of `p.communicate`) are returned as binary (byte) format.

Now check what's in `stdout`:

```python
stdout
```

    b"I'm talkin' to you, bash!\n"

Let's encode and print it.

```python
print(stdout.decode())
```

    I'm talkin' to you, bash!

You can also use a `universal_newlines = True` so that these outputs are returned as encoded text (default being *utf-8* usually), with  line endings converted to '\n'. For more information [see the documentation](https://docs.python.org/3.5/library/subprocess.html).

Let's try something else:

```python
p = subprocess.Popen(["ls", "-l"], stdout=subprocess.PIPE)
stdout, stderr = p.communicate()
# print(stdout.decode())
```

Recall that the `ls -l` command lists all files in a long listing format.

You can also call python itself from bash (!):

```python
p = subprocess.Popen(["python", "boilerplate.py"], stdout=subprocess.PIPE, stderr=subprocess.PIPE) # A bit silly!
stdout, stderr = p.communicate()
print(stdout.decode())
```

    This is a boilerplate

Similarly, to compile a $\LaTeX$ document (using `pdflatex` in this case), you can do something like:

```python
subprocess.os.system("pdflatex yourlatexdoc.tex")
```

You can also do this instead:

```python
p = subprocess.Popen(["python", "boilerplate.py"], stdout=subprocess.PIPE, stderr=subprocess.PIPE) # A bit silly!
stdout, stderr = p.communicate()
print(stdout.decode())
```

    This is a boilerplate

### Handling directory and file paths

You can also use `subprocess.os` to make your code OS (Linux, Windows, Mac) independent. For example to assign paths:

```python
subprocess.os.path.join('directory', 'subdirectory', 'file')
```

    'directory/subdirectory/file'

The result would be appropriately different on Windows (with backslashes instead of forward slashes).

Note that in all cases you can "catch" the output of `subprocess` so that you can then use the output within your
python script. A simple example, where the output is a platform-dependent directory path, is:

```python
MyPath = subprocess.os.path.join('directory', 'subdirectory', 'file')
MyPath
```

    'directory/subdirectory/file'

Explore what `subprocess` can do by tabbing
`subprocess.`, and also for submodules, e.g., type
`subprocess.os.` and then tab.

### Running `R`

R is likely an important part of your project's analysis and data visualization components in particular — for example for statistical analyses and pretty plotting (`ggplot2`!).

You can run `R` from Python easily. Try the following:

$\star$ Create an R script file called `TestR.R` in your `code` directory with the following content:

```r
print("Hello, this is R!")
```

Now, create a script `TestR.py` with the following content :

```python
import subprocess
subprocess.Popen("Rscript --verbose TestR.R > ../Results/TestR.Rout 2> ../Results/TestR_errFile.Rout", shell=True).wait()
```

    2

Now run `TestR.py` (or `%cpaste`) and check`TestR.Rout` and `TestR_errorFile.Rout`.

Also check what happens if you run (type directly in `ipython` or `python` console):

```python
subprocess.Popen("Rscript --verbose NonExistScript.R > ../Results/outputFile.Rout 2> ../Results/errorFile.Rout", shell=True).wait()
```

    2

It is possible that the location of `RScript` is different in your Ubuntu install. To locate it, try `find /usr -name 'Rscript'` in the linux terminal (not in `python`!). For example, you might need to specify the path to it using `/usr/lib/R/bin/Rscript`.

What do you see on the screen? Now check `outputFile.Rout`and `errorFile.Rout.

## Practicals

As always, test, add, commit and push all your new code and data to your git repository.

### Using `os` problem 1

Open `using_os.py` and complete the tasks assigned (hint: you might want to look at `subprocess.os.walk()`)

### Using `os` problem 2

Open `fmr.R` and work out what it does; check that you have `NagyEtAl1999.csv`. Now write python code called
`run_fmr_R.py` that:

Runs `fmr.R` to generate the desired result

`run_fmr_R.py` should also print to the python screen whether the run was successful, and the contents of the R console output

* `git add`, `commit` and `push` all    your week's code and data to your git repository by next Wednesday.*

Readings and Resources
----------------------

* [The matplotlib website](http://matplotlib.org)

* For SciPy, the [official documentation is good](https://docs.scipy.org/doc/); Read about the scipy modules you think will be important to you.

* The "ecosystem" for Scientific computing in python: <http://www.scipy-lectures.org/>

* A Primer on Scientific Programming with Python <http://www.springer.com/us/book/9783642549595>; Multiple copies of this book are available from the central library and can be requested to Silwood from the IC library website. You can also find a pdf - google it

* Many great examples of applications in the [scipy cookbook](https://lagunita.stanford.edu/courses/DB/2014/SelfPaced/about)

* For regex: <https://docs.python.org/2/howto/regex.html>

* Google's short class on regex in python: <https://developers.google.com/edu/python/regular-expressions>
And this exercise: https://developers.google.com/edu/python/exercises/baby-names

* <http://www.regular-expressions.info/> has a good intro, tips and a great array of canned solutions

* Use and abuse of regex: <https://blog.codinghorror.com/regex-use-vs-regex-abuse/>