# Python Data Science Guide

This notebook provides an introduction to the tools and libraries that we use in this GWAS lecture. These are:

1. [NumPy](http://www.numpy.org/)
2. [h5py](https://www.h5py.org/)
3. [pandas](https://pandas.pydata.org/)
4. [matplotlib](https://matplotlib.org/)

## NumPy

NumPy stands for ‘Numerical Python’ or ‘Numeric Python’. It is an open source module of Python which provides fast mathematical computation on arrays and matrices.

Numpy can be imported into the notebook using

```python
%matplotlib inline
import numpy as np
```

### Creating NumPy array

NumPy’s main object is the homogeneous multidimensional array. It is a table with same type elements, i.e, integers or string or characters (homogeneous), usually integers. In NumPy, dimensions are called axes. The number of axes is called the rank.

There are several ways to create an array in NumPy like np.array, np.zeros, no.ones, etc. Each of them provides some flexibility.

| Numpy function | What it does ?                                                                                |
|----------------|-----------------------------------------------------------------------------------------------------------------------|
| np.array       | Convert array like python object to NumPy array                                                                       |
| np.arange      | Create arrays with regularly incrementing values.                                                                     |
| np.ones        | Create an array filled with 1 values                                                                                |
| np.zeros       | Create an array filled with 0 values                                                                                |
| np.full        | Create an array filled with the specified value                                                                       |
| np.random.rand | Generate arrays of random values in the specified shape                                                               |
| np.linspace    | Create arrays with a specified number of elements, and spaced equally between the specified beginning and end values. |

```python

```

```python
# Convert array like python object to NumPy array
np.array([2,3,4])
```

    array([2, 3, 4])

```python
# Create arrays with regularly incrementing values.
np.arange(2, 3, 0.1)  # Number between 2, 3 with gap of 0.1
```

    array([2. , 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7, 2.8, 2.9])

```python
# Create an array filled with 0 values
np.zeros((2, 3))
```

    array([[0., 0., 0.],
           [0., 0., 0.]])

```python
# Create an array filled with 1 values
np.ones((2, 3))  #
```

    array([[1., 1., 1.],
           [1., 1., 1.]])

```python
# Create an array filled with the specified value
np.full( (3,4), 0.11 )
```

    array([[0.11, 0.11, 0.11, 0.11],
           [0.11, 0.11, 0.11, 0.11],
           [0.11, 0.11, 0.11, 0.11]])

```python
# Generate arrays of random values in the specified shape
np.random.rand(2,3)
```

    array([[0.75061919, 0.22210586, 0.39882487],
           [0.16872907, 0.32425882, 0.44510628]])

```python
# Create arrays with a specified number of elements, and spaced equally between the specified beginning and end values
np.linspace(0, 5/3, 6)
```

    array([0. , 0.2, 0.4, 0.6, 0.8, 1. ])

Some of the important attributes of a NumPy object are:

- **ndim**: displays the dimension of the array
- **shape**: returns a tuple of integers indicating the size of the array
- **size**: returns the total number of elements in the NumPy array
- **dtype**: returns the type of elements in the array, i.e., int64, character
- **itemsize**: returns the size in bytes of each item
- **reshape**: Reshapes the NumPy array

```python
arr = np.ones((2,3))
print arr
print("ndim: %s" % arr.ndim)
print("shape: (%s,%s)" % arr.shape)
print("size: %s" % arr.size)
print("dtype: %s" % arr.dtype)
print("itemsize: %s" % arr.itemsize)
arr.reshape((3,2))
```

    [[1. 1. 1.]
     [1. 1. 1.]]
    ndim: 2
    shape: (2,3)
    size: 6
    dtype: float64
    itemsize: 8

    array([[1., 1.],
           [1., 1.],
           [1., 1.]])

By default NumPy numeric arrays will be created using *float64* type.
This can be overridden by providing an explicit dtype.
For example to create a numpy array with 32bit integer elements use:

```python
np.ones((2,3), dtype=np.int32)
```

    array([[1, 1, 1],
           [1, 1, 1]], dtype=int32)

### Accessing NumPy array

NumPy array elements can be accessed using indexing. Below are some of the useful examples:

- arr[2:5] will print items 2 to 4. Index in NumPy arrays starts from 0
- arr[2::2] will print items 2 to end skipping 2 items
- arr[::-1] will print the array in the reverse order
- arr[1:] will print from row 1 to end

Accessing a multi-dimensional NumPy array can be done using following form:

```Array[row_start_index:row_end_index, column_start_index: column_end_index]```

```python
arr = a = np.arange(25).reshape(5, 5)
arr
```

    array([[ 0,  1,  2,  3,  4],
           [ 5,  6,  7,  8,  9],
           [10, 11, 12, 13, 14],
           [15, 16, 17, 18, 19],
           [20, 21, 22, 23, 24]])

```python
# access rows 2 to 4 and all columns
arr[2:4]
```

    array([[10, 11, 12, 13, 14],
           [15, 16, 17, 18, 19]])

```python
# access colum 2 to 4 and all rows
arr[:,2:4]
```

    array([[ 2,  3],
           [ 7,  8],
           [12, 13],
           [17, 18],
           [22, 23]])

```python
# access colum 2 to 4 and row 2 to 4
arr[2:4,2:4]
```

    array([[12, 13],
           [17, 18]])

```python
# access last row
arr[-1]
```

    array([20, 21, 22, 23, 24])

```python
# access last column
arr[:,-1]
```

    array([ 4,  9, 14, 19, 24])

```python
# access second row and third column
arr[2,3]
```

    13

NumPy arrays can also be accessed using boolean indexing.

```python
a = np.arange(12).reshape(3, 4)
a
```

    array([[ 0,  1,  2,  3],
           [ 4,  5,  6,  7],
           [ 8,  9, 10, 11]])

```python
rows_on = np.array([True, False, True])
rows_on
```

    array([ True, False,  True])

```python
a[rows_on,:]
```

    array([[ 0,  1,  2,  3],
           [ 8,  9, 10, 11]])

### NumPy array manipulation

NumPy arrays are capable of performing all basic operations such as addition, subtraction, element-wise product, matrix dot product, element-wise division, element-wise modulo, element-wise exponents and conditional operations.

An important feature with NumPy arrays is broadcasting:

```python
# Without broadcasting
a = np.array([0, 1, 2])
b = np.array([5, 5, 5])
a + b
```

    array([5, 6, 7])

```python
# With broadcasting
a + 5
```

    array([5, 6, 7])

NumPy provides basic mathematical and statistical functions like mean, min, max, sum, prod, std, var, summation across different axes, transposing of a matrix, etc.

```python
c = a + b
print("mean: %s" % c.mean())
print("std: %s" % c.std())
print("sum: %s" % c.sum())
print("var: %s" % c.var())
print("min: %s" % c.min())
print("max: %s" % c.max())
```

    mean: 6.0
    std: 0.816496580927726
    sum: 18
    var: 0.6666666666666666
    min: 5
    max: 7

A particular NumPy feature of interest is solving a system of linear equations. NumPy has a function to solve linear equations.
For example these linear equations

\begin{align}
2x + 6y & =  6 \\
5x + 3y & =  -9
\end{align}

can be solved by:

```python
coeffs  = np.array([[2, 6], [5, 3]])
depvars = np.array([6, -9])
solution = np.linalg.solve(coeffs, depvars)
solution
```

    array([-3.,  2.])

---
## HDF5 & h5py

The h5py package is a Pythonic interface to the HDF5 binary data format.

It lets you store huge amounts of numerical data, and easily manipulate that data from NumPy. For example, you can slice into multi-terabyte datasets stored on disk, as if they were real NumPy arrays. Thousands of datasets can be stored in a single file, categorized and tagged however you want.

```python
# First we download the data
import sys
sys.path.append('./..')
import data as tutorial_data
file_name = tutorial_data.get_file('BYxRM')
```

    file not found, downloading from https://www.ebi.ac.uk/~stegle/limix/tutorials/data/BYxRM/BYxRM.hdf5

An HDF5 file is a container for two kinds of objects: datasets, which are array-like collections of data, and groups, which are folder-like containers that hold datasets and other groups. The most fundamental thing to remember when using h5py is:

**Groups work like dictionaries, and datasets work like NumPy arrays**

Open an existing HDF5 file:

```python
import h5py
f = h5py.File(file_name,'r')
```

**HDF** stands for *Hierarchical Data Format*.
Every object in an HDF5 file has a name, and they’re arranged in a POSIX-style hierarchy with `/`-separators.
The “folders” in this system are called groups. The **File** object we created is itself a *group*, in this case the *root* group, named `/`

What is stored in this file?
Remember **h5py.File** acts like a Python dictionary, thus we can check the keys,

```python
list(f.keys())
```

    [u'genotype', u'phenotype']

The root object contains two children (*genotype* and *phenotype*).
To find out what type (dataset or group) of object the *phenotype* is we can run:

```python
f['phenotype']
```

    <HDF5 group "/phenotype" (3 members)>

It looks like the *phenotype* object itself is a group with 3 children

```python
list(f['phenotype'].keys())
```

    [u'col_header', u'matrix', u'row_header']

An easy way to iterate over all elements of a HDF5 file is to use the visit method:

```python
def printname(name):
    print("%s: %s " % (name, f[name]))
f.visit(printname)
```

    genotype: <HDF5 group "/genotype" (3 members)>
    genotype/col_header: <HDF5 group "/genotype/col_header" (3 members)>
    genotype/col_header/alleles: <HDF5 dataset "alleles": shape (11623, 2), type "|S37">
    genotype/col_header/chrom: <HDF5 dataset "chrom": shape (11623,), type "<i8">
    genotype/col_header/pos: <HDF5 dataset "pos": shape (11623,), type "<i8">
    genotype/matrix: <HDF5 dataset "matrix": shape (1008, 11623), type "|u1">
    genotype/row_header: <HDF5 group "/genotype/row_header" (1 members)>
    genotype/row_header/sample_ID: <HDF5 dataset "sample_ID": shape (1008,), type "|S99">
    phenotype: <HDF5 group "/phenotype" (3 members)>
    phenotype/col_header: <HDF5 group "/phenotype/col_header" (1 members)>
    phenotype/col_header/phenotype_ID: <HDF5 dataset "phenotype_ID": shape (46,), type "|S22">
    phenotype/matrix: <HDF5 dataset "matrix": shape (1008, 46), type "<f8">
    phenotype/row_header: <HDF5 group "/phenotype/row_header" (1 members)>
    phenotype/row_header/sample_ID: <HDF5 dataset "sample_ID": shape (1008,), type "|S22">

Every object in an HDF5 file can have custom attributes which can be checked like any other dictionary:

```python
list(f['phenotype']['matrix'].attrs.keys())
```

    []

Let us examine the *phenotype/matrix* dataset as a `Dataset` object.
The object we obtained isn’t an array, but an *HDF5* dataset. Like *NumPy* arrays, datasets have both a *shape* and a *data type*:

```python
dset = f['phenotype/matrix']
print('shape: (%s,%s)' % dset.shape)
print('dtype: %s' % dset.dtype)
```

    shape: (1008,46)
    dtype: float64

They also support array-style slicing. This is how you read and write data from a dataset in the file.

```python
# retrieve all rows between 10 and 20 for the the second column
dset[10:20, 2]
```

    array([ 2.24597261,  2.6897812 , -0.46048117,  0.03086018, -0.29291799,
           -1.09117648,  0.70117087, -0.86986989, -0.13682744, -0.28441332])

When filtering/slicing/indexing rows and columns of a dataset, not the entire dataset is loaded into the main memory. This is very useful if you need to retrieve a subset of the data from a terabyte big HDF5 file.
To read the entire dataset into the memory run:

```python
dset[:]
```

    array([[-7.32351971e+00,  2.79992827e-01,  3.13118166e-01, ...,
             8.90841948e-01,  4.11837231e+00,  8.59281836e+00],
           [-8.09823582e+00, -2.06326076e-01, -5.34843783e-01, ...,
             6.06164117e-03,  6.65150029e-02, -4.22047646e+00],
           [ 7.60571968e+00, -1.27959825e-01, -3.11102424e-01, ...,
             1.72315729e+00,  5.71408803e+00, -6.50651895e+00],
           ...,
           [            nan,  3.21844466e-01,             nan, ...,
             1.80142187e+00,             nan, -4.99069190e-01],
           [            nan, -1.00150686e+00,             nan, ...,
            -1.74752575e+00,             nan, -7.43596147e+00],
           [            nan,             nan,             nan, ...,
                        nan,             nan,             nan]])

```python
arr = dset[:]
print(type(arr))
arr
```

    <type 'numpy.ndarray'>

    array([[-7.32351971e+00,  2.79992827e-01,  3.13118166e-01, ...,
             8.90841948e-01,  4.11837231e+00,  8.59281836e+00],
           [-8.09823582e+00, -2.06326076e-01, -5.34843783e-01, ...,
             6.06164117e-03,  6.65150029e-02, -4.22047646e+00],
           [ 7.60571968e+00, -1.27959825e-01, -3.11102424e-01, ...,
             1.72315729e+00,  5.71408803e+00, -6.50651895e+00],
           ...,
           [            nan,  3.21844466e-01,             nan, ...,
             1.80142187e+00,             nan, -4.99069190e-01],
           [            nan, -1.00150686e+00,             nan, ...,
            -1.74752575e+00,             nan, -7.43596147e+00],
           [            nan,             nan,             nan, ...,
                        nan,             nan,             nan]])

---
## Pandas

Similar to *NumPy*, *Pandas* is one of the most widely used python libraries in data science. It provides high-performance, easy to use structures and data analysis tools. Unlike *NumPy* library which provides objects for multi-dimensional arrays, *Pandas* provides in-memory 2d table object called *Dataframe*. It is like a spreadsheet with column names and row label.

Some commonly used data structures in pandas are:

- *Series* objects: 1D array, similar to a column in a spreadsheet
- *DataFrame* objects: 2D table, similar to a spreadsheet
- *Panel* objects: Dictionary of DataFrames, similar to sheet in MS Excel

Pandas can be imported into Python using:

```python
import pandas as pd
```

### Pandas Series

Pandas Series object is created using *pd.Series* function. Each row is provided with an index and by defaults is assigned numerical values starting from *0*. Like *NumPy*, Pandas also provide the basic mathematical functionalities like addition, subtraction and conditional operations and broadcasting.

```python
weight = pd.Series([68, 83, 112],index=["alice", "bob", "charles"])
birthyear = pd.Series([1984, 1985, 1992], index=["bob", "alice", "charles"], name="year")
children = pd.Series([0, 3], index=["charles", "bob"])
hobby = pd.Series(["Biking", "Dancing"], index=["alice", "bob"])
print(weight)
print('---')
print(weight.values)
```

    alice       68
    bob         83
    charles    112
    dtype: int64
    ---
    [ 68  83 112]

For a lot of applications, a plain old *Series* is probably not a lot of use, but it is the core component of the Pandas workhorse, the *DataFrame*, so it’s useful to know about.

The *DataFrame* represents tabular data, a bit like a spreadsheet. *DataFrames* are organised into colums (each of which is a *Series*), and each column can store a single data-type, such as floating point numbers, strings, boolean values etc. *DataFrames* can be indexed by either their row or column names. (They are similar in many ways to R’s `data.frame`.).

We can create a *DataFrame* in *Pandas* from a Python dictionary, or by loading in a text file containing tabular data. First we are going to look at how to create one from a dictionary.

```python
df = pd.DataFrame({'weight': weight, 'birthyear': birthyear, 'children': children, 'hobby': hobby})
df
```

      birthyear
      children
      hobby
      weight

      alice
      1985
      NaN
      Biking
      68

      bob
      1984
      3.0
      Dancing
      83

      charles
      1992
      0.0
      NaN
      112

An alternative way to create a *DataFrame* is to load it from existing files. There are various loading function to load data from different file sources which can be found [here](https://pandas.pydata.org/pandas-docs/stable/api.html#input-output). The most important ones are:
- *pd.read_pickle*: Load pickled pandas object (or any object) from file
- *pd.read_csv*: Read CSV (comma-separated) file into DataFrame
- *pd.read_json*: Convert a JSON string to pandas object
- *pd.read_excel*: Read an Excel table into a pandas DataFrame
- *pd.read_hdf*: Read from the HDF5 file (PyTasbles format)

Let's return now to our HDF5 file that we previously examined.
The *phenotype* group contains 3 datasets:

```python
print(list(f['phenotype'].keys()))
print(f['phenotype/col_header/phenotype_ID'])
print(f['phenotype/row_header/sample_ID'])
```

    [u'col_header', u'matrix', u'row_header']
    <HDF5 dataset "phenotype_ID": shape (46,), type "|S22">
    <HDF5 dataset "sample_ID": shape (1008,), type "|S22">

This seems to be a phenotype matrix where the *matrix* dataset contains the 2d array with the phenotype values and the *col_header/phenotype_ID* contains the phenotype names and the *row_header/sample_ID* contains the accession ids.
We can construct a Panda *dataframe* in the following way:

```python
pheno_group = f['phenotype']
df = pd.DataFrame(pheno_group['matrix'][:],
                        columns=pheno_group['col_header']['phenotype_ID'][:],
                        index=pheno_group['row_header']['sample_ID'][:])

```

**Dataframes** have some essential methods to access the data:
- **head()**: returns the top 5 rows in the dataframe object
- **tail()**: returns the bottom 5 rows in the dataframe
- **info()**: prints the summary of the dataframe
- **describe()**: gives a nice overview of the main aggregated values over each column

```python
# retrieve first 10 rows
df.head(10)
```

      Cadmium_Chloride
      Caffeine
      Calcium_Chloride
      Cisplatin
      Cobalt_Chloride
      Congo_red
      Copper
      Cycloheximide
      Diamide
      E6_Berbamine
      ...
      x6-Azauracil
      Xylose
      YNB
      YNB:ph3
      YNB:ph8
      YPD
      YPD:15C
      YPD:37C
      YPD:4C
      Zeocin

      A01_01
      -7.323520
      0.279993
      0.313118
      1.658179
      -1.604442
      5.841617
      -4.130950
      0.821226
      3.622602
      -0.378747
      ...
      1.199054
      -0.309149
      17.470822
      0.055225
      -0.184268
      24.548971
      0.712171
      0.890842
      4.118372
      8.592818

      A01_02
      -8.098236
      -0.206326
      -0.534844
      -0.918012
      0.892198
      -1.618172
      1.131947
      -0.764736
      -2.946279
      -2.475193
      ...
      0.443182
      0.697908
      18.052925
      0.283462
      1.662201
      26.808476
      -1.522498
      0.006062
      0.066515
      -4.220476

      A01_03
      7.605720
      -0.127960
      -0.311102
      -2.712088
      3.301709
      -6.680571
      -1.138056
      -3.382532
      -3.157866
      -4.178616
      ...
      1.472629
      1.474329
      18.163782
      -0.951379
      0.369565
      26.183975
      1.082289
      1.723157
      5.714088
      -6.506519

      A01_04
      -6.147649
      0.878392
      -0.563331
      -0.827358
      -1.291270
      2.753851
      -2.268121
      0.867456
      -6.832079
      -1.970630
      ...
      0.054694
      -0.739040
      15.575856
      -1.690867
      -0.840344
      20.893646
      -0.636280
      0.053258
      -3.196936
      0.294462

      A01_05
      9.379060
      -1.353169
      0.405204
      -1.137234
      4.260286
      -7.391216
      -3.222742
      1.586376
      1.048391
      -3.971730
      ...
      0.267463
      0.959138
      17.808444
      0.127621
      1.967267
      28.621507
      -3.720999
      -0.768724
      -4.002786
      7.045575

      A01_06
      9.394562
      1.655436
      0.531654
      -2.044903
      1.577325
      -5.769939
      1.355905
      2.336068
      4.038331
      -2.917101
      ...
      0.591233
      -0.134019
      16.085601
      -0.100294
      0.710547
      23.549116
      -2.680231
      2.025716
      -0.889829
      -4.108786

      A01_07
      -7.229972
      -1.423605
      -0.007546
      -1.714243
      0.954948
      2.111495
      0.590067
      1.415252
      -4.861309
      -4.082216
      ...
      -0.330839
      0.294128
      16.472448
      -0.612281
      -0.242591
      23.901629
      -1.834152
      -3.487390
      3.618048
      -0.035142

      A01_08
      -7.585476
      1.750342
      0.941181
      0.306306
      -2.997501
      0.063007
      -4.352137
      2.634605
      3.147146
      8.292883
      ...
      0.043294
      -0.826648
      16.210856
      NaN
      -0.538390
      23.595423
      -1.449024
      1.679693
      -2.227455
      -5.773160

      A01_09
      -7.890982
      -0.559696
      -1.396937
      -0.879233
      -1.044798
      -6.214873
      0.211404
      -0.006420
      -9.470146
      -4.480685
      ...
      -0.048497
      -0.483113
      15.161760
      -0.035535
      -0.398174
      22.693341
      -1.580512
      -1.320567
      -4.863302
      9.024782

      A01_10
      -8.144131
      -2.787328
      -1.111711
      -0.239611
      -2.933533
      1.231604
      1.397043
      2.405056
      8.532335
      14.355684
      ...
      0.685319
      -0.594544
      16.388451
      -0.742849
      -0.427930
      23.697733
      -2.967864
      -1.553708
      -5.314155
      -1.229565

10 rows × 46 columns

```python
# retrieve last 5 rows
df.tail()
```

      Cadmium_Chloride
      Caffeine
      Calcium_Chloride
      Cisplatin
      Cobalt_Chloride
      Congo_red
      Copper
      Cycloheximide
      Diamide
      E6_Berbamine
      ...
      x6-Azauracil
      Xylose
      YNB
      YNB:ph3
      YNB:ph8
      YPD
      YPD:15C
      YPD:37C
      YPD:4C
      Zeocin

      A11_92
      -4.262720
      -0.893290
      -0.329504
      0.257061
      -3.120957
      2.556543
      2.567092
      2.054232
      1.672347
      5.465978
      ...
      0.232614
      1.130568
      16.629005
      0.531583
      NaN
      23.520842
      0.159946
      -5.158120
      -0.090273
      -7.030154

      A11_93
      -10.147285
      1.145573
      0.048948
      -1.089350
      0.539833
      2.720952
      -3.445357
      0.891066
      3.003684
      -1.667592
      ...
      -0.317916
      1.977915
      15.687951
      -0.098451
      NaN
      21.553930
      2.977722
      -1.913176
      4.279626
      -4.650492

      A11_94
      NaN
      0.321844
      NaN
      1.016712
      1.801923
      -4.157083
      -0.097598
      -1.954080
      -1.739499
      -6.520551
      ...
      -1.698334
      1.363267
      17.357844
      0.390511
      NaN
      23.977236
      0.078992
      1.801422
      NaN
      -0.499069

      A11_95
      NaN
      -1.001507
      NaN
      0.004191
      4.675717
      -10.240998
      1.244410
      -0.810857
      2.305318
      9.321186
      ...
      0.259829
      0.942454
      17.098753
      -0.432459
      NaN
      22.927686
      -0.254905
      -1.747526
      NaN
      -7.435961

      A11_96
      NaN
      NaN
      NaN
      NaN
      -0.155556
      NaN
      2.603438
      -0.026752
      NaN
      NaN
      ...
      NaN
      1.776737
      17.924399
      -0.165314
      NaN
      NaN
      1.553372
      NaN
      NaN
      NaN

5 rows × 46 columns

```python
# Summary of dataframe
df.info()
```

    <class 'pandas.core.frame.DataFrame'>
    Index: 1008 entries, A01_01 to A11_96
    Data columns (total 46 columns):
    Cadmium_Chloride          799 non-null float64
    Caffeine                  1004 non-null float64
    Calcium_Chloride          949 non-null float64
    Cisplatin                 990 non-null float64
    Cobalt_Chloride           1007 non-null float64
    Congo_red                 979 non-null float64
    Copper                    972 non-null float64
    Cycloheximide             1004 non-null float64
    Diamide                   1003 non-null float64
    E6_Berbamine              1005 non-null float64
    Ethanol                   967 non-null float64
    Formamide                 999 non-null float64
    Galactose                 962 non-null float64
    Hydrogen_Peroxide         769 non-null float64
    Hydroquinone              960 non-null float64
    Hydroxyurea               944 non-null float64
    Indoleacetic_Acid         994 non-null float64
    Lactate                   973 non-null float64
    Lactose                   1004 non-null float64
    Lithium_Chloride          1002 non-null float64
    Magnesium_Chloride        964 non-null float64
    Magnesium_Sulfate         1006 non-null float64
    Maltose                   1005 non-null float64
    Mannose                   957 non-null float64
    Menadione                 1005 non-null float64
    Neomycin                  1005 non-null float64
    Paraquat                  1004 non-null float64
    Raffinose                 662 non-null float64
    SDS                       873 non-null float64
    Sorbitol                  599 non-null float64
    Trehalose                 1003 non-null float64
    Tunicamycin               1006 non-null float64
    x4-Hydroxybenzaldehyde    964 non-null float64
    x4NQO                     1005 non-null float64
    x5-Fluorocytosine         957 non-null float64
    x5-Fluorouracil           992 non-null float64
    x6-Azauracil              1001 non-null float64
    Xylose                    1003 non-null float64
    YNB                       1006 non-null float64
    YNB:ph3                   979 non-null float64
    YNB:ph8                   887 non-null float64
    YPD                       1006 non-null float64
    YPD:15C                   1005 non-null float64
    YPD:37C                   1003 non-null float64
    YPD:4C                    806 non-null float64
    Zeocin                    957 non-null float64
    dtypes: float64(46)
    memory usage: 370.1+ KB

```python
# calculate some summary statistics
df.describe()
```

      Cadmium_Chloride
      Caffeine
      Calcium_Chloride
      Cisplatin
      Cobalt_Chloride
      Congo_red
      Copper
      Cycloheximide
      Diamide
      E6_Berbamine
      ...
      x6-Azauracil
      Xylose
      YNB
      YNB:ph3
      YNB:ph8
      YPD
      YPD:15C
      YPD:37C
      YPD:4C
      Zeocin

      count
      799.000000
      1004.000000
      949.000000
      990.000000
      1007.000000
      979.000000
      972.000000
      1004.000000
      1003.000000
      1005.000000
      ...
      1001.000000
      1003.000000
      1006.000000
      979.000000
      887.000000
      1006.000000
      1005.000000
      1003.000000
      806.000000
      957.000000

      mean
      -0.235763
      0.056543
      0.012039
      -0.080343
      -0.021220
      -0.580926
      0.067722
      0.157040
      0.139683
      0.264369
      ...
      0.112644
      0.060641
      16.315382
      0.007996
      0.007482
      23.502212
      0.047823
      -0.140778
      -0.190777
      0.166791

      std
      6.306318
      0.910155
      0.773159
      1.045181
      2.808820
      6.411729
      1.808578
      1.683631
      4.749402
      7.083140
      ...
      0.853002
      0.939542
      1.034526
      0.674851
      0.902919
      1.815418
      2.541834
      2.642301
      4.649077
      6.433438

      min
      -10.381891
      -4.141899
      -2.452206
      -4.557306
      -6.184913
      -13.488314
      -8.944908
      -5.980597
      -17.291493
      -12.513969
      ...
      -3.327925
      -2.911638
      11.981303
      -2.570029
      -3.724701
      16.979292
      -6.493270
      -7.377041
      -9.588223
      -13.359268

      25%
      -6.456470
      -0.507907
      -0.498740
      -0.741802
      -2.288283
      -6.620026
      -0.665649
      -0.963134
      -3.193845
      -4.880492
      ...
      -0.449021
      -0.599287
      15.661717
      -0.451187
      -0.571156
      22.380371
      -1.879136
      -2.168742
      -3.680280
      -5.159223

      50%
      1.022356
      0.067418
      0.042018
      -0.019068
      -0.377311
      -0.568692
      0.396090
      0.234640
      0.892107
      -2.475193
      ...
      0.146145
      0.047475
      16.358272
      0.024450
      -0.003055
      23.403000
      -0.132709
      -0.047865
      -0.977779
      -2.015229

      75%
      5.813172
      0.622523
      0.503318
      0.645479
      1.998527
      4.537839
      1.235156
      1.324370
      3.327036
      4.487006
      ...
      0.700031
      0.717306
      17.006638
      0.479301
      0.628253
      24.629824
      1.832583
      1.966580
      2.605938
      5.751830

      max
      12.876948
      3.372236
      2.689781
      3.763224
      8.467436
      15.984204
      4.924384
      6.516210
      11.349569
      23.116821
      ...
      2.816535
      3.286086
      19.067704
      1.773026
      2.956204
      29.985755
      7.287769
      7.474741
      15.552609
      16.662919

8 rows × 46 columns

### Accessing data in a DataFrame

Our columns in the dataframe object are individual Series of data. We can access them by referring to the column name e.g. dataframe['column-name']. For example to access the *Caffeine* column run this:

```python
print(df['Caffeine'].head())
print('============= or ===========')
print(df.Caffeine.head())
```

    A01_01    0.279993
    A01_02   -0.206326
    A01_03   -0.127960
    A01_04    0.878392
    A01_05   -1.353169
    Name: Caffeine, dtype: float64
    ============= or ===========
    A01_01    0.279993
    A01_02   -0.206326
    A01_03   -0.127960
    A01_04    0.878392
    A01_05   -1.353169
    Name: Caffeine, dtype: float64

To filter by row we have to use the *iloc* or *loc* function on the *dataframe*. The *iloc* function allows to filter by the integer index of the row while the *loc* function allows to index by the index label (accession id):

```python
df.iloc[2].Caffeine
```

    -0.127959825210201

```python
df.loc['A01_03'].Caffeine
```

    -0.127959825210201

It is also possible to retrieve multiple rows either by specifying the exact row index labels or integer positions:

```python
df.iloc[[1, 5, 6, 9]].Caffeine
```

    A01_02   -0.206326
    A01_06    1.655436
    A01_07   -1.423605
    A01_10   -2.787328
    Name: Caffeine, dtype: float64

```python
df.loc[['A01_02', 'A01_06', 'A01_07', 'A01_10']].Caffeine
```

    A01_02   -0.206326
    A01_06    1.655436
    A01_07   -1.423605
    A01_10   -2.787328
    Name: Caffeine, dtype: float64

```python
# retrieve the first 10 rows for the columns Caffeine (column 1 ) and Cisplatin (column 2)
df.iloc[:10,[1,3]]
```

      Caffeine
      Cisplatin

      A01_01
      0.279993
      1.658179

      A01_02
      -0.206326
      -0.918012

      A01_03
      -0.127960
      -2.712088

      A01_04
      0.878392
      -0.827358

      A01_05
      -1.353169
      -1.137234

      A01_06
      1.655436
      -2.044903

      A01_07
      -1.423605
      -1.714243

      A01_08
      1.750342
      0.306306

      A01_09
      -0.559696
      -0.879233

      A01_10
      -2.787328
      -0.239611

### Filtering data

We can also apply conditions to the data we are inspecting, such as to filter our data.
Let's examine the Caffeine column and check the lowest value.

```python
df.Caffeine.min()
```

    -4.14189858085831

Let's now filter the Caffeine column for values that are greater than 0

```python
df.Caffeine > 0
```

    A01_01     True
    A01_02    False
    A01_03    False
    A01_04     True
    A01_05    False
    A01_06     True
    A01_07    False
    A01_08     True
    A01_09    False
    A01_10    False
    A01_11    False
    A01_12    False
    A01_13     True
    A01_14     True
    A01_15    False
    A01_16    False
    A01_17     True
    A01_18     True
    A01_19    False
    A01_20     True
    A01_21     True
    A01_22     True
    A01_23     True
    A01_24    False
    A01_25     True
    A01_26     True
    A01_27     True
    A01_28    False
    A01_29     True
    A01_30    False
              ...
    A11_67    False
    A11_68     True
    A11_69     True
    A11_70    False
    A11_71    False
    A11_72     True
    A11_73     True
    A11_74    False
    A11_75    False
    A11_76    False
    A11_77     True
    A11_78     True
    A11_79     True
    A11_80     True
    A11_81    False
    A11_82    False
    A11_83    False
    A11_84    False
    A11_85     True
    A11_86     True
    A11_87     True
    A11_88    False
    A11_89     True
    A11_90     True
    A11_91     True
    A11_92    False
    A11_93     True
    A11_94     True
    A11_95    False
    A11_96    False
    Name: Caffeine, Length: 1008, dtype: bool

This returns a new *Series* of *True*/*False* values though. To actually filter the data, we need to use this *Series* to mask our original *DataFrame*

```python
df[df.Caffeine > 0]
```

      Cadmium_Chloride
      Caffeine
      Calcium_Chloride
      Cisplatin
      Cobalt_Chloride
      Congo_red
      Copper
      Cycloheximide
      Diamide
      E6_Berbamine
      ...
      x6-Azauracil
      Xylose
      YNB
      YNB:ph3
      YNB:ph8
      YPD
      YPD:15C
      YPD:37C
      YPD:4C
      Zeocin

      A01_01
      -7.323520
      0.279993
      0.313118
      1.658179
      -1.604442
      5.841617
      -4.130950
      0.821226
      3.622602
      -0.378747
      ...
      1.199054
      -0.309149
      17.470822
      0.055225
      -0.184268
      24.548971
      0.712171
      0.890842
      4.118372
      8.592818

      A01_04
      -6.147649
      0.878392
      -0.563331
      -0.827358
      -1.291270
      2.753851
      -2.268121
      0.867456
      -6.832079
      -1.970630
      ...
      0.054694
      -0.739040
      15.575856
      -1.690867
      -0.840344
      20.893646
      -0.636280
      0.053258
      -3.196936
      0.294462

      A01_06
      9.394562
      1.655436
      0.531654
      -2.044903
      1.577325
      -5.769939
      1.355905
      2.336068
      4.038331
      -2.917101
      ...
      0.591233
      -0.134019
      16.085601
      -0.100294
      0.710547
      23.549116
      -2.680231
      2.025716
      -0.889829
      -4.108786

      A01_08
      -7.585476
      1.750342
      0.941181
      0.306306
      -2.997501
      0.063007
      -4.352137
      2.634605
      3.147146
      8.292883
      ...
      0.043294
      -0.826648
      16.210856
      NaN
      -0.538390
      23.595423
      -1.449024
      1.679693
      -2.227455
      -5.773160

      A01_13
      6.099126
      1.193630
      -0.460481
      -2.254822
      -1.737934
      -6.773013
      1.847423
      1.492433
      5.377797
      -2.763472
      ...
      0.428124
      -0.447422
      15.707533
      1.328875
      -0.676125
      22.880547
      -4.488093
      3.399301
      -5.313911
      -4.854207

      A01_14
      -6.128808
      2.287569
      0.030860
      1.457627
      2.658909
      8.873294
      -2.725993
      2.081959
      -0.640953
      -2.948221
      ...
      0.871464
      0.239443
      16.293701
      0.306043
      0.903620
      23.053594
      4.081697
      3.783942
      0.069884
      2.976971

      A01_17
      -6.435934
      0.331987
      0.701171
      0.330169
      -3.157943
      -1.098844
      2.099880
      3.261305
      -10.892045
      14.833187
      ...
      0.778375
      1.257294
      16.432578
      0.030953
      1.637458
      23.623720
      3.763264
      -3.649057
      7.883571
      -5.824656

      A01_18
      -1.744416
      1.179753
      -0.869870
      -1.269823
      3.048698
      -6.121298
      0.008026
      -0.243670
      3.129125
      -0.310028
      ...
      -0.658192
      0.014206
      17.374168
      -0.383067
      -0.997588
      23.653983
      -2.572794
      1.388221
      -6.245010
      -1.875007

      A01_20
      -6.362380
      0.986639
      -0.284413
      0.014248
      2.235446
      -1.681283
      -0.420546
      0.076087
      3.076295
      -6.065038
      ...
      -1.904024
      0.127447
      14.506945
      -0.817359
      -0.907824
      20.581287
      0.125045
      -0.928468
      -2.542536
      6.886391

      A01_21
      5.925445
      1.238484
      -0.180890
      -0.785711
      0.856511
      -0.954674
      1.886783
      2.196050
      2.436579
      -0.643799
      ...
      0.736404
      1.220645
      17.109577
      0.002582
      -0.689019
      24.960265
      -4.600436
      -3.240098
      -1.430459
      4.630062

      A01_22
      3.015350
      0.844175
      0.640409
      1.477814
      2.287242
      -0.343746
      1.038470
      -0.832654
      2.717067
      -4.706824
      ...
      2.037066
      1.738069
      17.113177
      0.625722
      0.712437
      24.854792
      5.443331
      -4.111030
      5.459420
      10.382066

      A01_23
      5.368651
      1.648824
      -0.311006
      0.151763
      2.089597
      0.886900
      -0.827784
      -0.076783
      2.241636
      -5.830545
      ...
      -0.369967
      -0.887642
      15.680565
      -0.132719
      0.793680
      23.211814
      1.160337
      3.339040
      0.908244
      13.591021

      A01_25
      2.361884
      1.207676
      -1.382897
      1.224807
      -1.735648
      -0.200892
      NaN
      -2.155111
      -6.857530
      -3.846204
      ...
      0.909605
      -1.330609
      13.144011
      1.004012
      0.127184
      22.482902
      -0.605416
      0.017608
      0.398050
      -2.081629

      A01_26
      -5.888597
      1.748776
      -0.432020
      -0.303939
      -2.064806
      -0.087783
      0.418293
      1.550241
      1.862355
      4.625362
      ...
      0.572039
      -0.514878
      16.622170
      -0.709576
      -0.502412
      23.804121
      -4.416801
      1.710262
      -2.832793
      -2.582201

      A01_27
      5.864971
      0.085989
      0.272710
      0.687241
      -2.272625
      -6.032576
      2.116587
      0.717661
      4.509182
      8.784337
      ...
      0.984612
      0.663690
      17.433361
      -0.237397
      -0.214977
      22.998909
      1.925353
      4.612446
      2.617287
      7.365063

      A01_29
      8.291001
      0.602952
      -0.209818
      0.176575
      2.970487
      11.674431
      -5.015363
      0.942569
      -0.493707
      10.374494
      ...
      0.222568
      -1.013385
      16.763518
      -0.206383
      0.672626
      26.448189
      1.866508
      0.239108
      -0.638321
      12.690415

      A01_31
      -6.398361
      0.166271
      -0.878706
      -0.078068
      -3.349890
      -1.069199
      1.163056
      -0.895535
      -0.087970
      -4.928491
      ...
      -1.314301
      -1.375641
      15.544307
      -0.004930
      -0.744114
      24.094502
      -2.020393
      -0.891901
      -5.279599
      1.193205

      A01_32
      7.130669
      0.718797
      0.518366
      1.823296
      0.429805
      -1.103549
      0.740313
      2.506797
      2.778662
      -4.981721
      ...
      0.632448
      1.494387
      16.222977
      0.751728
      -2.119434
      22.723083
      5.791591
      -0.958975
      1.065139
      -2.015229

      A01_33
      -7.996359
      0.922486
      0.264327
      -1.664310
      -5.532087
      -5.217066
      1.188328
      2.380337
      3.479604
      -2.757452
      ...
      1.271010
      1.306615
      18.784166
      0.187778
      0.246367
      29.168126
      4.090078
      0.995169
      3.198552
      -10.757366

      A01_34
      -7.061253
      0.024929
      1.033085
      0.766978
      3.873184
      1.228287
      -0.019795
      -0.597636
      -0.869106
      -3.144067
      ...
      1.212225
      1.713806
      17.565024
      0.101327
      -0.127652
      23.749063
      3.836577
      -2.537776
      6.718132
      -4.968092

      A01_35
      7.125313
      0.477383
      0.203855
      0.146072
      4.542455
      -4.875453
      2.360665
      -1.442190
      0.693330
      -6.215016
      ...
      1.818472
      0.151740
      16.212551
      0.662079
      -0.147682
      23.430682
      -2.494964
      3.580115
      -2.705861
      -4.434123

      A01_38
      7.221967
      1.607407
      1.136931
      0.321945
      -2.917663
      6.128958
      -0.439967
      3.226792
      4.648551
      -3.809075
      ...
      1.010044
      0.545939
      15.461926
      0.202358
      0.744799
      22.786138
      -0.951993
      -1.479317
      6.993651
      -5.792154

      A01_40
      7.386030
      0.685339
      0.552385
      1.627962
      4.060814
      -1.013393
      -0.051946
      0.805096
      5.205418
      0.235112
      ...
      -0.328904
      0.065395
      16.315874
      0.539172
      -0.718097
      23.366239
      3.277305
      1.194231
      0.670884
      7.639392

      A01_44
      5.584843
      1.063073
      -0.673369
      0.292690
      -0.585619
      -1.411756
      0.982226
      -0.333864
      2.798290
      0.193106
      ...
      -0.635281
      -0.836197
      15.972579
      0.842179
      1.050111
      22.333949
      -4.374102
      1.138760
      -5.530835
      5.771627

      A01_45
      -6.886116
      2.422459
      2.094146
      2.513203
      6.714054
      -0.397405
      -6.722845
      -1.037253
      -0.875755
      -5.357255
      ...
      -1.635183
      0.518553
      15.862309
      1.405404
      1.074609
      20.873441
      -2.156279
      6.328426
      1.883678
      -3.488512

      A01_46
      -6.857821
      0.372263
      -1.493623
      -2.658450
      -0.895147
      -5.366424
      -0.115780
      -0.501187
      3.045313
      -5.753381
      ...
      -1.029738
      -0.399975
      15.623093
      -0.450903
      -0.254102
      22.505698
      -3.920752
      -2.202890
      -1.563144
      1.055169

      A01_47
      -4.268287
      0.362259
      0.310740
      1.390570
      0.966619
      10.681893
      0.633434
      0.512259
      1.973278
      -6.997449
      ...
      0.574923
      0.748745
      17.494740
      -0.075021
      0.094029
      23.578347
      -0.045091
      -2.543260
      1.207601
      4.475345

      A01_55
      4.074521
      1.005304
      -0.840341
      -0.156409
      1.439703
      -0.288215
      -2.433246
      -0.739372
      4.679368
      -5.967278
      ...
      0.205970
      0.869975
      15.028113
      0.364559
      0.530884
      22.119766
      5.280116
      -4.113606
      -2.367438
      0.865415

      A01_58
      4.092310
      0.929132
      -0.885145
      0.854615
      -1.651898
      1.106198
      2.308981
      -1.584598
      2.258720
      6.459206
      ...
      0.709761
      0.793777
      17.961373
      0.583428
      -0.001954
      27.061697
      -4.836116
      3.824283
      -2.061558
      -3.699731

      A01_59
      -7.267147
      0.317266
      0.353902
      0.173577
      3.859513
      -1.350063
      0.048835
      -1.142921
      0.293651
      -2.481264
      ...
      0.165374
      1.366814
      16.484308
      -0.533237
      -0.487195
      24.985005
      0.075893
      2.709230
      2.936690
      5.137446

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      A11_44
      -7.455436
      0.065010
      0.708960
      -0.303781
      1.814606
      5.638196
      0.224649
      1.198512
      -1.343692
      -4.435628
      ...
      -0.749211
      -0.148476
      17.105216
      0.123122
      NaN
      22.720722
      -2.992506
      -2.502814
      -1.839947
      5.751830

      A11_45
      2.907273
      0.377561
      0.008094
      1.516315
      0.471784
      -6.231109
      -3.230581
      -0.941268
      -11.222300
      -4.671810
      ...
      0.790916
      0.722727
      16.467980
      1.132648
      NaN
      22.521577
      1.643338
      2.459915
      -5.901576
      -4.664778

      A11_46
      -6.928941
      0.077312
      0.227317
      1.109649
      2.180708
      -1.061959
      0.614301
      1.245527
      4.237199
      3.007887
      ...
      0.791549
      1.060520
      15.742954
      0.688456
      NaN
      22.512751
      0.454164
      -2.993857
      2.341450
      5.118265

      A11_48
      7.183339
      0.983350
      2.090029
      0.001171
      0.420704
      -2.660331
      1.417940
      1.470723
      1.243041
      -7.825930
      ...
      -0.111029
      -0.589826
      15.758187
      0.727027
      NaN
      22.681465
      -1.544399
      1.773595
      -3.088895
      -3.579066

      A11_51
      7.633948
      0.077103
      0.703641
      0.419895
      0.559865
      -6.942340
      1.111722
      0.174473
      2.517062
      -1.057514
      ...
      0.772863
      -0.873542
      16.774581
      -0.384493
      NaN
      23.893079
      -0.050615
      -1.412976
      -6.528134
      7.276401

      A11_52
      -8.027743
      0.342941
      0.207606
      0.777800
      1.855766
      -6.155187
      0.929895
      -0.289071
      0.662585
      6.697976
      ...
      0.621308
      0.035059
      16.362545
      0.487377
      NaN
      22.642174
      -0.752784
      -3.310424
      4.157145
      -5.710500

      A11_54
      -8.525728
      1.328282
      0.473115
      0.388526
      0.823639
      -6.179502
      0.822936
      -1.103175
      7.015429
      -4.235858
      ...
      0.399033
      0.443633
      16.321728
      -0.326466
      NaN
      23.076351
      -3.822596
      -2.623999
      -6.319933
      -4.721783

      A11_57
      6.346367
      0.167243
      2.454003
      0.309949
      1.359860
      -0.526673
      2.606971
      0.754974
      1.809343
      -4.754368
      ...
      -2.912437
      0.611290
      18.938192
      1.074502
      NaN
      22.313950
      -0.262279
      4.058366
      6.746140
      -5.175894

      A11_58
      -7.285724
      0.583391
      -0.163537
      -1.535646
      0.151295
      -6.545795
      NaN
      -0.749599
      1.642822
      2.715098
      ...
      -0.324229
      -1.474309
      16.352664
      -0.068981
      NaN
      23.206375
      -4.699755
      0.792384
      -3.585440
      2.196232

      A11_60
      7.598761
      0.730103
      0.828262
      0.347327
      0.161159
      -8.610313
      2.052813
      -0.986483
      -8.726147
      -7.935309
      ...
      -0.000856
      -0.865281
      15.641259
      1.351713
      NaN
      22.050501
      -2.512252
      2.951493
      -1.999519
      -8.702733

      A11_61
      -6.880364
      0.862238
      NaN
      -0.426329
      4.559201
      -8.452262
      0.926557
      2.548554
      1.555138
      -3.631661
      ...
      0.768317
      -1.471348
      17.081808
      -0.140308
      NaN
      25.479476
      -0.750635
      3.074695
      NaN
      5.849204

      A11_62
      -8.045053
      0.069826
      -0.942733
      -0.277029
      0.725625
      -7.727957
      -0.064185
      -0.066813
      0.169675
      -3.480213
      ...
      -0.039708
      -0.499990
      19.067704
      -0.319564
      NaN
      25.226514
      -4.369941
      3.546102
      -7.820512
      -6.362555

      A11_64
      -8.452961
      1.313303
      -0.180174
      -0.395152
      -3.073554
      11.475177
      -3.706662
      1.467159
      3.371451
      -4.024469
      ...
      -1.072126
      -0.430788
      16.292964
      0.687982
      NaN
      24.538142
      2.667982
      2.440611
      6.249088
      8.189521

      A11_66
      9.448724
      0.311918
      0.164296
      1.179131
      1.776497
      -6.723817
      NaN
      3.236887
      3.533954
      -2.685802
      ...
      1.692752
      0.418787
      15.430309
      1.211189
      NaN
      22.224755
      0.956364
      -0.765028
      -0.505661
      -4.559104

      A11_68
      -8.824487
      0.908554
      0.693707
      0.258778
      3.071328
      12.956824
      0.565678
      0.980528
      -4.295115
      -3.894834
      ...
      1.005061
      0.611755
      18.096924
      -0.249243
      NaN
      24.928409
      4.742218
      -2.630358
      5.647873
      9.762167

      A11_69
      -5.625713
      0.622860
      -0.599341
      -0.301774
      0.571811
      11.847581
      0.201365
      0.066659
      2.859140
      -4.665133
      ...
      -0.211785
      -0.645943
      16.710638
      -0.341715
      NaN
      23.724599
      1.796232
      0.607982
      0.693608
      6.862298

      A11_72
      -9.219612
      0.579977
      0.901603
      0.680549
      1.497327
      1.119451
      -2.112560
      -1.144666
      -0.532813
      -9.513557
      ...
      -0.582535
      -1.300570
      16.447684
      0.858761
      NaN
      23.037218
      0.305084
      1.587400
      -1.889495
      -10.516313

      A11_73
      -5.636734
      0.453751
      NaN
      -1.238875
      -1.676514
      3.115243
      -5.534217
      -0.918070
      -6.196405
      -4.306319
      ...
      -0.095414
      0.419458
      15.222614
      -0.870604
      NaN
      21.546315
      2.459713
      3.235806
      -3.884509
      4.981810

      A11_77
      -8.798892
      0.301123
      -0.323830
      0.432050
      2.895317
      10.033640
      0.595472
      -0.957188
      4.284933
      -3.969501
      ...
      0.783732
      2.010373
      18.009701
      -0.453162
      NaN
      25.167154
      -1.830648
      -2.582527
      6.232993
      -2.549221

      A11_78
      -4.061045
      0.153991
      0.749857
      0.981008
      0.952446
      -7.626006
      0.546279
      1.417278
      2.061099
      -4.090833
      ...
      0.792770
      2.142495
      16.595233
      0.516697
      NaN
      25.390230
      5.181662
      -2.575708
      12.377388
      8.398114

      A11_79
      -0.940648
      0.348818
      0.462085
      0.467866
      5.958515
      5.921017
      NaN
      0.267799
      2.001825
      2.009163
      ...
      0.331876
      2.086463
      17.236369
      1.208932
      NaN
      24.567289
      4.332028
      -2.421702
      2.236019
      8.882389

      A11_80
      6.014177
      0.634675
      -0.431921
      1.064627
      -4.663311
      -7.117807
      1.670899
      -0.385115
      -4.388628
      -5.171900
      ...
      -0.042121
      -0.469607
      16.598856
      0.260251
      NaN
      23.133193
      -1.978838
      2.192267
      -1.833746
      -3.248339

      A11_85
      NaN
      0.327573
      NaN
      -0.645948
      -2.031381
      7.609264
      -0.660415
      1.255510
      5.098531
      -4.039771
      ...
      -0.470322
      0.624881
      15.913935
      0.221976
      NaN
      20.968383
      1.705758
      6.654459
      10.226067
      -1.580349

      A11_86
      3.189294
      2.307381
      NaN
      1.684777
      0.510847
      -1.193390
      -1.424114
      -0.993333
      -3.922912
      1.348560
      ...
      -3.327925
      0.331371
      15.066564
      0.038268
      NaN
      16.979292
      -0.394850
      5.130092
      3.422867
      -0.946073

      A11_87
      -5.011391
      0.101890
      1.089817
      -1.230890
      -3.135374
      -7.961377
      0.164554
      0.359548
      -6.488684
      -4.194792
      ...
      0.180122
      -0.699413
      16.162823
      -0.427215
      NaN
      22.939438
      1.632431
      -4.138147
      -3.984612
      -3.358585

      A11_89
      -0.781114
      1.079321
      0.219939
      0.771153
      1.955114
      5.559785
      4.924384
      2.472066
      2.813091
      14.695409
      ...
      -0.372606
      0.096194
      16.820080
      0.619160
      NaN
      24.101895
      3.574523
      2.409968
      4.127189
      15.004739

      A11_90
      NaN
      0.645842
      NaN
      NaN
      -0.890742
      13.016383
      -5.268835
      -0.468322
      2.211619
      -4.457251
      ...
      0.058875
      -1.051974
      17.411414
      -0.035335
      NaN
      25.180528
      -1.240811
      3.017025
      -2.024357
      -5.818075

      A11_91
      -6.343767
      0.438523
      -0.322617
      1.973411
      4.515720
      -8.495414
      NaN
      -0.224670
      3.258043
      2.732642
      ...
      0.114539
      1.206215
      16.871327
      1.084758
      NaN
      22.730956
      4.994928
      -2.030958
      -0.282208
      8.456099

      A11_93
      -10.147285
      1.145573
      0.048948
      -1.089350
      0.539833
      2.720952
      -3.445357
      0.891066
      3.003684
      -1.667592
      ...
      -0.317916
      1.977915
      15.687951
      -0.098451
      NaN
      21.553930
      2.977722
      -1.913176
      4.279626
      -4.650492

      A11_94
      NaN
      0.321844
      NaN
      1.016712
      1.801923
      -4.157083
      -0.097598
      -1.954080
      -1.739499
      -6.520551
      ...
      -1.698334
      1.363267
      17.357844
      0.390511
      NaN
      23.977236
      0.078992
      1.801422
      NaN
      -0.499069

534 rows × 46 columns

To check that there are no values below 0 run the *min()* function again

```python
df[df.Caffeine > 0].Caffeine.min()
```

    0.0032293928665531

Some of our values have missing values that are represented in the output as `NaN`. Often we can't work with them. One way to deal with them is to drop the rows that contain them. First let's check how many missing values we have.

```python
print(df.Caffeine[df.Caffeine.isnull()])
```

    A05_12   NaN
    A09_65   NaN
    A10_01   NaN
    A11_96   NaN
    Name: Caffeine, dtype: float64

```python
df.dropna(subset=['Caffeine'], inplace=True)
print(df.Caffeine[df.Caffeine.isnull()])
```

    Series([], Name: Caffeine, dtype: float64)

### Plotting pandas dataframes

The *DataFrame* has a built in plotting functions (i.e `pd.hist()` or `pd.plot()`) that allows to create simple plots using matplotlib internally. For more advanced plotting functions, one can use the *matplotlib* library directly.

```python
df[['Caffeine','Copper']].hist()
```

```python
df[['Caffeine','Copper']].plot(kind='hist')
```

---
## Matplotlib

*Matplotlib* is a Python package used for data plotting and visualisation. It is a useful complement to Pandas, and like Pandas, is a very feature-rich library which can produce a large variety of plots, charts, maps, and other visualisations.
`matplotlib.pyplot` is a collection of functions that make matplotlib work like *MATLAB*. Majority of plotting commands in `pyplot` have *MATLAB* analogs with similar arguments. Let us take a couple of examples.

Typically, if we are going to work on some plotting, we would import matplotlib like this:

```python
%matplotlib inline
import matplotlib.pyplot as plt
```

```python
# Example 1: Plotting a line graph
plt.plot([1,2,3,4])
plt.ylabel('some numbers')
plt.show()
```

```python
# Example 2: Plotting a histogram
x = [21,22,23,4,5,6,77,8,9,10,31,32,33,34,35,36,37,18,49,50,100]
num_bins = 5
plt.hist(x, num_bins, facecolor='blue')
plt.show()
```

Let us try to plot our phenotype data.  Let’s have a look at the relationship between two phenotypes in our dataframe. To save typing later on, we can extract the Series for “Caffeine” and “Copper” by assigning each to a new variable, x and y, respectively.

```python
df.dropna(inplace=True)
x = df.Sorbitol
y = df.Trehalose
```

Then we can plot them as a scatter chart by adding:

```python
plt.scatter(x, y)
# or plt.savefig("name.png")
```

Now we are going to do some basic statistics so we can plot a line of linear regression on our chart.
Let’s plot a linear regression through the data. Python has a library called scipy that contains a lot of statistics routines. We can import it:

```python
from scipy.stats import linregress
```

To get the statistics for the linear regression we can use a function called `linregress`

```python
stats = linregress(x, y)
m = stats.slope
b = stats.intercept
stats
```

    LinregressResult(slope=0.9622331502773507, intercept=-0.011936070484259736, rvalue=0.9064525118355159, pvalue=1.0446912214962477e-114, stderr=0.025839319479636295)

The equation for the straight line that describes linear regression is `y = mx + b`, where m is the *slope* and `b` is the *intercept*.
(Hint: you have some extra information in the stats object - stats.rvalue and stats.pvalue.)

Therefore, we can then plot the line of linear regression by adding the following line:

```python
plt.scatter(x, y)
plt.plot(x, m * x + b, color="r")   # I've added a color argument here
```

Matplotlib figures are highly customisable, and there are so many options it is usually best to consult the [documentation]() first.
We will take the above plot and customize *fontsize*, *linewidth*, *color* for example:

```python
# Change the default figure size
plt.figure(figsize=(10,10))

# Change the default marker for the scatter from circles to x's
plt.scatter(x, y, marker='x')

# Set the linewidth on the regression line to 3px
plt.plot(x, m * x + b, color="red", linewidth=3)

# Add x and y lables, and set their font size
plt.xlabel("Sorbitol", fontsize=20)
plt.ylabel("Trehalose", fontsize=20)

# Plot pvalue and r2 value
plt.text(-2.5, 3, 'pvalue: %s' % stats.pvalue,
         fontsize=14)
plt.text(-2.5, 2.8, 'r2: %s' % stats.rvalue,
         fontsize=14)

# Set the font size of the number lables on the axes
plt.xticks(fontsize=18)
plt.yticks(fontsize=18)
plt.show()
```

```python

```


```python
import sympy as sm
```

```python
# nonlinear limb-darkening coefficients
c1, c2, c3, c4, r, z, p = sm.symbols(['c1', 'c2', 'c3', 'c4', 'r', 'z', 'p'], real=True, positive=True)
```

```python
# intensity function
mu = sm.sqrt(1 - r**2)
Ir = 1 - c1 * (1 - mu ** sm.Rational(1/2)) - c2 * (1 - mu) - c3 * (1 - mu ** sm.Rational(3/2)) - c4 * (1 - mu ** 2)
```

```python
# integral when the planet is entirely occulting the star
inside = sm.integrate(Ir * 2 * r, (r, z-p, z+p))
```

```python
# simplify and make some substitutions
result = sm.simplify(sm.collect(sm.collect(sm.simplify(sm.expand(inside)), -p**2 - 2 * p * z - z**2 + 1),-p**2 + 2 * p * z - z**2 + 1))
alpha, beta = sm.symbols(['alpha', 'beta'], real=True)
result = result.subs(-p**2 - 2 * p * z - z**2 + 1, alpha).subs(-p**2 + 2 * p * z - z**2 + 1, beta)
result
```

$\displaystyle - \frac{4 \alpha^{\frac{7}{4}} c_{3}}{7} - \frac{4 \alpha^{\frac{5}{4}} c_{1}}{5} - \frac{2 \alpha^{\frac{3}{2}} c_{2}}{3} + \frac{4 \beta^{\frac{7}{4}} c_{3}}{7} + \frac{4 \beta^{\frac{5}{4}} c_{1}}{5} + \frac{2 \beta^{\frac{3}{2}} c_{2}}{3} - 4 c_{1} p z - 4 c_{2} p z - 4 c_{3} p z - 4 c_{4} p^{3} z - 4 c_{4} p z^{3} + 4 p z$

```python
# integral when the planet is only partially occulting the star
edge = sm.integrate(Ir * 2 * r, (r, z-p, 1))
```

```python
# simplify
result = sm.simplify(sm.collect(sm.collect(sm.simplify(sm.expand(edge)), -p**2 - 2 * p * z - z**2 + 1),-p**2 + 2 * p * z - z**2 + 1))
result = result.subs(-p**2 - 2 * p * z - z**2 + 1, alpha).subs(-p**2 + 2 * p * z - z**2 + 1, beta)
```

```python
result
```

$\displaystyle \frac{4 \beta^{\frac{7}{4}} c_{3}}{7} + \frac{4 \beta^{\frac{5}{4}} c_{1}}{5} + \frac{2 \beta^{\frac{3}{2}} c_{2}}{3} + \beta + c_{1} p^{2} - 2 c_{1} p z + c_{1} z^{2} - c_{1} + c_{2} p^{2} - 2 c_{2} p z + c_{2} z^{2} - c_{2} + c_{3} p^{2} - 2 c_{3} p z + c_{3} z^{2} - c_{3} + \frac{c_{4} p^{4}}{2} - 2 c_{4} p^{3} z + 3 c_{4} p^{2} z^{2} - 2 c_{4} p z^{3} + \frac{c_{4} z^{4}}{2} - \frac{c_{4}}{2}$

```python

```


```python
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns
import pymc3 as pm
import arviz as az

import sys
sys.path.insert(0, '..')
from utils.plot_lib import set_default

set_default(figsize=(6, 4))
```

### Normal likelihood, unknown mean and variance

The model is:

$
\begin{align}
y_i \mid \mu, \sigma^2 &\overset{\text{iid}}{\sim} \text{N} ( \mu, \sigma^2 ), \quad i=1,\ldots,n \\
\mu &\sim \text{N}(\mu_0, \sigma_0^2) \\
\sigma^2 &\sim \text{IG}(\nu_0, \beta_0)  \, .
\end{align}
$

We chose a normal prior for $μ$ because, in the case where $σ^2$ is known, the normal is the conjugate prior for $μ$. Likewise, in the case where $μ$ is known, the inverse-gamma is the conjugate prior for $σ^2$. This will give us convenient full conditional distributions in a Gibbs sampler.

Let’s first work out the form of the full posterior distribution.

$
\begin{align}
p( \mu, \sigma^2 \mid y_1, y_2, \ldots, y_n ) &\propto
p(y_1, y_2, \ldots, y_n \mid \mu, \sigma^2) p(\mu) p(\sigma^2) \\
&= \prod_{i=1}^n \text{N} ( y_i \mid \mu, \sigma^2 ) \times \text{N}( \mu \mid \mu_0, \sigma_0^2) \times \text{IG}(\sigma^2 \mid \nu_0, \beta_0) \\
&= \prod_{i=1}^n \frac{1}{\sqrt{2\pi\sigma^2}}\exp \left[ -\frac{(y_i - \mu)^2}{2\sigma^2} \right] \times \frac{1}{\sqrt{2\pi\sigma_0^2}} \exp \left[ -\frac{(\mu - \mu_0)^2}{2\sigma_0^2} \right] \times \frac{\beta_0^{\nu_0}}{\Gamma(\nu_0)}(\sigma^2)^{-(\nu_0 + 1)} \exp \left[ -\frac{\beta_0}{\sigma^2} \right] I_{\sigma^2 > 0}(\sigma^2) \\
&\propto (\sigma^2)^{-n/2} \exp \left[ -\frac{\sum_{i=1}^n (y_i - \mu)^2}{2\sigma^2} \right] \exp \left[ -\frac{(\mu - \mu_0)^2}{2\sigma_0^2} \right] (\sigma^2)^{-(\nu_0 + 1)} \exp \left[ -\frac{\beta_0}{\sigma^2} \right] I_{\sigma^2 > 0}(\sigma^2)
\end{align}
$

From here, it is easy to continue on to find the two full conditional distributions we need. First let’s look at $μ$, assuming $σ2$ is known (in which case it becomes a constant and is absorbed into the normalizing constant):

$
\begin{align}
p(\mu \mid \sigma^2, y_1, \ldots, y_n) &\propto p( \mu, \sigma^2 \mid y_1, \ldots, y_n ) \\
&\propto \exp \left[ -\frac{\sum_{i=1}^n (y_i - \mu)^2}{2\sigma^2} \right] \exp \left[ -\frac{(\mu - \mu_0)^2}{2\sigma_0^2} \right] \\
&\propto \exp \left[ -\frac{1}{2} \left( \frac{ \sum_{i=1}^n (y_i - \mu)^2}{2\sigma^2} + \frac{(\mu - \mu_0)^2}{2\sigma_0^2} \right) \right] \\
&\propto \text{N} \left( \mu \mid \frac{n\bar{y}/\sigma^2 + \mu_0/\sigma_0^2}{n/\sigma^2 + 1/\sigma_0^2}, \, \frac{1}{n/\sigma^2 + 1/\sigma_0^2} \right) \, ,
\end {align}
$

Given the data and $σ^2$, $μ$ follows this normal distribution. Now let’s look at $σ^2$, assuming $μ$ is known:

$
\begin{align}
p(\sigma^2 \mid \mu, y_1, \ldots, y_n) &\propto p( \mu, \sigma^2 \mid y_1, \ldots, y_n ) \\
&\propto (\sigma^2)^{-n/2} \exp \left[ -\frac{\sum_{i=1}^n (y_i - \mu)^2}{2\sigma^2} \right] (\sigma^2)^{-(\nu_0 + 1)} \exp \left[ -\frac{\beta_0}{\sigma^2} \right] I_{\sigma^2 > 0}(\sigma^2) \\
&\propto (\sigma^2)^{-(\nu_0 + n/2 + 1)} \exp \left[ -\frac{1}{\sigma^2} \left( \beta_0 + \frac{\sum_{i=1}^n (y_i - \mu)^2}{2} \right) \right] I_{\sigma^2 > 0}(\sigma^2) \\
&\propto \text{IG}\left( \sigma^2 \mid \nu_0 + \frac{n}{2}, \, \beta_0 + \frac{\sum_{i=1}^n (y_i - \mu)^2}{2} \right) \, .
\end{align}
$

These two distributions provide the basis of a Gibbs sampler to simulate from a Markov chain whose stationary distribution is the full posterior of both $μ$ and $σ^2$. We simply alternate draws between these two parameters, using the most recent draw of one parameter to update the other.

To implement the Gibbs sampler we just described, let’s return to our running example where the data are the percent change in total personnel from last year to this year for $n=10$ companies. We’ll still use a normal likelihood, but now we’ll relax the assumption that we know the variance of growth between companies, $σ^2$, and estimate that variance. Instead of the $t$ prior from earlier, we will use the conditionally conjugate priors, normal for $μ$ and inverse-gamma for $σ^2$.

The first step will be to write functions to simulate from the full conditional distributions we derived in the previous segment. The full conditional for $μ$, given $σ^2$ and data is:

$ \text{N} \left( \mu \mid \frac{n\bar{y}/\sigma^2 + \mu_0/\sigma_0^2}{n/\sigma^2 + 1/\sigma_0^2}, \, \frac{1}{n/\sigma^2 + 1/\sigma_0^2} \right) $

```python
from scipy.stats import norm

def update_mu(n, y_bar, sig2, mu_0, sig2_0):
    sig2_1 = 1.0 / (n / sig2 + 1.0 / sig2_0)
    mu_1 = sig2_1 * ((n * y_bar / sig2) + (mu_0 / sig2_0))
    mu_cand = norm.rvs(size=1, loc = mu_1, scale = np.sqrt(sig2_1))[0] # draw a candidate
    return mu_cand
```

The full conditional for $σ^2$ given $μ$ and data is:

$ \text{IG}\left( \sigma^2 \mid \nu_0 + \frac{n}{2}, \, \beta_0 + \frac{\sum_{i=1}^n (y_i - \mu)^2}{2} \right) $

```python
from scipy.stats import gamma

def update_sig2(n, y, mu, nu_0, beta_0):
    nu_1 = nu_0 + (n / 2.0)
    sumsq = np.sum( (y - mu)**2 ) # vectorized
    beta_1 = beta_0 + sumsq / 2.0
    rv = gamma(nu_1, loc = 0., scale = 1/beta_1)
    out_gamma = rv.rvs(size = 1)[0] # rate for gamma is shape for inv-gamma
    return 1 / out_gamma # reciprocal of a gamma random variable is distributed inv-gamma
```

With functions for drawing from the full conditionals, we are ready to write a function to perform Gibbs sampling.

```python
def gibbs(y, n_iter, init, prior):
    y_bar = np.mean(y)
    n = y.shape[0]

    ## initialize
    mu_out = []
    sig2_out = []

    mu_now = init

    for i in range(n_iter):
        sig2_now = update_sig2(n = n, y = y, mu = mu_now, nu_0 = prior['nu_0'], beta_0 = prior['beta_0'])
        mu_now = update_mu(n = n, y_bar = y_bar, sig2 = sig2_now, mu_0 = prior['mu_0'], sig2_0 = prior['sig2_0'])

        sig2_out.append(sig2_now)
        mu_out.append(mu_now)

    return mu_out, sig2_out
```

Setting up the problem

```python
y = np.array([1.2, 1.4, -0.5, 0.3, 0.9, 2.3, 1.0, 0.1, 1.3, 1.9]) # Data
y_bar = np.mean(y) # Sample mean
n = y.shape[0] # Data points

## Prior
prior = {}
prior['mu_0'] = 0.0
prior['sig2_0'] = 1.0
prior['n_0'] = 2.0 # prior effective sample size for sig2 (2 samples)
prior['s2_0'] = 1.0 # prior point estimate for sig2
prior['nu_0'] = prior['n_0'] / 2.0 # prior parameter for inverse-gamma
prior['beta_0'] = prior['n_0'] * prior['s2_0'] / 2.0 # prior parameter for inverse-gamma

# Plotting the histogram of the random samples
fig, ax = plt.subplots(1, 1)
sns.histplot(np.array(y), stat= 'density', bins = 6)
# Sampling in all the domain of the function
ax.plot(np.linspace(-2, 4, 100), norm.pdf(x = np.linspace(-2, 4, 100), loc = prior['mu_0'], scale = np.sqrt(prior['sig2_0'])), 'r-', lw=5, alpha=0.6, label='Normal Prior') # prior for mu
ax.scatter(y, np.repeat(0, n), s = 30, c='green', label = 'Data points') # individual data points
ax.scatter(y_bar, 0, s = 30, c='blue', label = 'Sample mean') # sample mean
ax.legend(loc='best', frameon=False)
plt.xlim(-1, 3)
plt.title('Histogram of data')
plt.show()
```

### Running experiment

```python
np.random.seed(53)

init_mu = 0.0
# Execute experiment
mu_out, sig2_out = gibbs(y=y, n_iter = int(1e3), init = init_mu, prior=prior)
```

### Plotting $\mu$ posterior

```python
# Plotting the trace-plot
fig, ax = plt.subplots(1, 1)
ax.plot(np.arange(len(mu_out)), mu_out, 'r-', lw=3, alpha=0.6, label='Trace plot mu') # Trace plot of data
ax.legend(loc='best', frameon=False)
# plot density estimate of the posterior
plt.title('Trace plot for mu')
plt.show()

# Plotting the trace-plot
fig, ax = plt.subplots(1, 1)
# plot density estimate of the posterior
sns.kdeplot(np.array(mu_out), bw=0.5, label = 'Density estimate post')
ax.legend(loc='best', frameon=False)
# plot density estimate of the posterior
plt.title('Density plot for mu')
plt.show()
```

```python
# Summary of sig2
pm.summary(np.array(mu_out))
```

    arviz - WARNING - Shape validation failed: input_shape: (1, 1000), minimum_shape: (chains=2, draws=4)

      mean
      sd
      hdi_3%
      hdi_97%
      mcse_mean
      mcse_sd
      ess_mean
      ess_sd
      ess_bulk
      ess_tail
      r_hat

      x
      0.889
      0.289
      0.411
      1.515
      0.009
      0.007
      945.0
      945.0
      954.0
      783.0
      NaN

 ### Plotting $\sigma^2$ posterior

```python
# Plotting the trace-plot
fig, ax = plt.subplots(1, 1)
ax.plot(np.arange(len(sig2_out)), sig2_out, 'r-', lw=3, alpha=0.6, label='Trace plot sigma2') # Trace plot of data
ax.legend(loc='best', frameon=False)
# plot density estimate of the posterior
plt.title('Trace plot for sigma2')
plt.show()

# Plotting the trace-plot
fig, ax = plt.subplots(1, 1)
# plot density estimate of the posterior
sns.kdeplot(np.array(sig2_out), bw=0.5, label = 'Density estimate post')
ax.legend(loc='best', frameon=False)
# plot density estimate of the posterior
plt.title('Density plot for sigma2')
plt.show()
```

```python
# Summary of sig2
pm.summary(np.array(sig2_out))
```

    arviz - WARNING - Shape validation failed: input_shape: (1, 1000), minimum_shape: (chains=2, draws=4)

      mean
      sd
      hdi_3%
      hdi_97%
      mcse_mean
      mcse_sd
      ess_mean
      ess_sd
      ess_bulk
      ess_tail
      r_hat

      x
      0.917
      0.472
      0.299
      1.762
      0.016
      0.011
      850.0
      850.0
      782.0
      814.0
      NaN

As with the Metropolis-Hastings example, these chains appear to have converged.


# Sequential Quadratic Programming

## Quadratic Programming

An optimization problem with a quadratic objective function and linear
constraints is called a _quadratic program_. Problems of this
type are important in their own right, and they also arise a subproblems
in methods for general constrained optimization such as sequential
quadratic programming (this chapter) and interior-point methods (lecture 10).

The general quadratic program can be stated as

\begin{align*}
\min_{\vec{x}}\, & f\left(\vec{x}\right)\overset{\vartriangle}{=}\frac{1}{2}\vec{x}^\mathsf{T}Q\vec{x}-\vec{c}^\mathsf{T}\vec{x}\\
\textrm{subject to}\, & \begin{cases}
A_{\textrm{eq}}\vec{x}=\vec{b}_{\textrm{eq}}\,,\\
A_{\textrm{in}}\vec{x}\leq\vec{b}_{\textrm{in}}\,,
\end{cases}
\end{align*}

where $Q$ is a symmetric $n\times n$ matrix, $\vec{c}\in\mathbb R^{n}$,
$A_{\textrm{eq}}$ is a $m\times n$ matrix, $\vec{b}_{\textrm{eq}}\in\mathbb R^{m}$,
$A_{\textrm{in}}$ is a $p\times n$ matrix and $\vec{b}_{\textrm{in}}\in\mathbb R^{p}$.
If the Hessian matrix $Q$ is positive semidefinite, we have a convex
quadratic program. Non-convex quadratic programs, in which $Q$ is
an indefinite matrix, can be more challenging because they have several
stationary points and local minima.

## Equality Constraints

We begin our discussion of algorithms for quadratic programming by considering
the case in which only equality constraints are present. We consider
to following equality-constrained quadratic problem
\begin{align}
\min_{\vec{x}}\, & f\left(\vec{x}\right)\overset{\vartriangle}{=}\frac{1}{2} \vec{x}^\mathsf{T}Q\vec{x}- \vec{c}^\mathsf{T}\vec{x}\\
\textrm{subject to}\, & A\vec{x}=\vec{b}\,,\nonumber
\end{align}
where $A$ is the $m\times n$ Jacobian of constraints and $\vec{b}\in\mathbb R^{m}$.
We assume that $A$ has rank $m$ so that the constraints are consistent.

The First Order Necessary Condition (FONC) uses the Langrangian function
\begin{equation}
\mathcal L\left(\vec x, \vec \lambda\right)=\vec{x}^\mathsf{T}Q\vec{x}- \vec{c}^\mathsf{T}\vec{x} + \vec\lambda^\mathsf{T} \left(A\vec{x}-\vec{b}\right)
\end{equation}
and gives the following condition
\begin{align}
\nabla_{\vec x}\mathcal L\left(\vec x ^\star, \vec \lambda ^\star\right)&=\vec 0\,,\\
\nabla_{\vec \lambda}\mathcal L\left(\vec x ^\star, \vec \lambda ^\star\right)&=\vec 0\,,
\end{align}
So the FONC for $\vec{x}^{\star}$ to be a solution of the quadratic
problem yields a vector $\vec{\lambda}^{\star}$ such
that the following system of equations is satisfied:
\begin{equation}
\begin{pmatrix}Q & A^\mathsf{T}\\
A & 0
\end{pmatrix}\begin{pmatrix}\vec{x}^{\star}\\
\vec{\lambda}^{\star}
\end{pmatrix}=\begin{pmatrix}\vec{c}\\
\vec{b}
\end{pmatrix}\,.\label{eq:equality_constraint}
\end{equation}
This system can be solved directly by factorization. An alternative
is to use an iterative method.

Consider the quadratic programming problem
\begin{align*}
\min_{\vec{x}}\, & \frac{1}{2} \vec{x}^\mathsf{T}\begin{pmatrix}6 & 2 & 1\\
2 & 5 & 2\\
1 & 2 & 4
\end{pmatrix}\vec{x}- \begin{pmatrix}8\\
3\\
3
\end{pmatrix}^\mathsf{T}\vec{x}\\
\textrm{subject to}\, & \begin{pmatrix}1 & 0 & 1\\
0 & 1 & 1
\end{pmatrix}\vec{x}=\begin{pmatrix}3\\
0
\end{pmatrix}\,.
\end{align*}

```julia
Q = [6 2 1
     2 5 2
     1 2 4]
c = [8, 3, 3]
A = [1 0 1
     0 1 1]
b = [3, 0]
sol = [Q transpose(A)
       A zeros(2,2)] \ [c; b]
```

The solution $\vec{x}^{\star}$ and optimal Lagrange multiplier vector $\vec \lambda^\star$ are given by
\begin{align*}
\vec{x}^{\star} & = \begin{pmatrix}2 & -1 & 1\end{pmatrix}^\mathsf{T}\,,\\
\vec{\lambda}^{\star} & = \begin{pmatrix}-3 & 2\end{pmatrix}^\mathsf{T}\,.
\end{align*}

## Active Set Method

We now describe active-set methods for solving quadratic programs
containing both equality and inequality constraints. We consider only
the convex case, in which the matrix $Q$ is positive semidefinite.

If the contents of the optimal active set $J\left(\vec x^\star\right)$ were known in advance, we could find the solution $\vec{x}^{\star}$
by applying the technique for equality constrained quadratic programs
to the problem
\begin{align*}
\min_{\vec{x}}\, & f\left(\vec{x}\right)\overset{\vartriangle}{=}\frac{1}{2} \vec{x}^\mathsf{T}Q\vec{x}- \vec{c}^\mathsf{T}\vec{x}\\
\textrm{subject to}\, & \begin{cases}
A_{\textrm{eq}}\vec{x}=\vec{b}_{\textrm{eq}}\,,\\
 \vec{a}_{\textrm{in},j}^\mathsf{T}\vec{x}=b_{\textrm{in},j}\,, & \forall j\in J\left(\vec{x}^{\star}\right)\,,
\end{cases}
\end{align*}
where $\vec{a}_{\textrm{in},j}$ is the $j$th row in the matrix $A_{\textrm{in}}$
and $b_{\textrm{in},j}$ is the $j$th element of the vector $\vec{b}_{\textrm{in}}$.
Of course, we usually do not have prior knowledge of $J\left(\vec{x}^{\star}\right)$
and determination of this set is the main challenge facing algorithms
for inequality-constrained quadratic programs.

Active-set methods find a step from one iterate to the next by solving
a quadratic subproblem in which some of the inequality constraints,
and all the equality constraints are imposed as equalities. This subset
is referred to as the _working set_ and is denoted at the $k$th
iterate by $W_{k}$. An important requirement we impose on $W_{k}$
is that the gradients $\vec{a}_{\textrm{eq},i}$, $i=1,\dots,m$ and
$\vec{a}_{\textrm{in},j}$, $j\in W_{k}$ are linearly independent,
even when the full set of active constraints at that point has linearly
dependent gradients.

Given an iterate $\vec{x}^{\left(k\right)}$ and the working set $W_{k}$,
we first check whether $\vec{x}^{\left(k\right)}$ minimizes the quadratic
function $f$ in the subspace defined by the working set. If not,
we compute a step $\vec{d}^{\left(k\right)}$ by solving an equality-constrained
quadratic subproblem in which the inequality constraints corresponding
to the working set $W_{k}$ are regarded as equalities and the other
inequality constraints are temporarily disregarded. To express this
subproblem in terms of the step $\vec{d}^{\left(k\right)}$, we define
 \begin{equation}
\vec{d}^{\left(k\right)}=\vec{x}^{\left(k+1\right)}-\vec{x}^{\left(k\right)}\,,\quad\vec{g}_{k}=Q\vec{x}^{\left(k\right)}-\vec{c}\,.
 \end{equation}
By substituting for $\vec{x}^{\left(k+1\right)}$ into the objective
function, we find that
 \begin{equation}
f\left(\vec{x}^{\left(k+1\right)}\right)=f\left(\vec{x}^{\left(k\right)}+\vec{d}^{\left(k\right)}\right)=\frac{1}{2}\left(\vec{d}^{\left(k\right)}\right)^\mathsf{T}Q\vec{d}^{\left(k\right)}+\vec{g}_{k}^\mathsf{T}\vec{d}^{\left(k\right)}+\rho_{k}\,,
 \end{equation}
where $\rho_{k}=\frac{1}{2} \left(\vec{x}^{\left(k\right)}\right)^\mathsf{T}Q\vec{x}^{\left(k\right)}- \vec{c}^\mathsf{T}\vec{x}^{\left(k\right)}$
is independent of $\vec{d}^{\left(k\right)}$. Since we can drop $\rho_{k}$
from the objective function without changing the solution of the problem,
we can write the subproblem to be solved at the $k$th iteration as
follows
\begin{align}
\min_{\vec{d}^{\left(k)\right)}}\, & \frac{1}{2} \left(\vec{d}^{\left(k\right)}\right)^\mathsf{T}Q\vec{d}^{\left(k\right)}+ \vec{g}_{k}^\mathsf{T}\vec{d}^{\left(k\right)}\label{eq:quadratic_subproblem}\\
\textrm{subject to}\, & \begin{cases}
A_{\textrm{eq}}\vec{d}^{\left(k\right)}=\vec{0}\,,\\
 \vec{a}_{\textrm{in},j}^\mathsf{T}\vec{d}^{\left(k\right)}=0\,, & \forall j\in W_{k}\,.
\end{cases}\nonumber
\end{align}
Note that for each $j\in W_{k}$, the value of $ \vec{a}_{\textrm{in},j}^\mathsf{T}\vec{x}^{\left(k\right)}$
does not change as we move along $\vec{d}^{\left(k\right)}$, since
we have $ \vec{a}_{\textrm{in},j}^\mathsf{T}\left(\vec{x}^{\left(k\right)}+\alpha\vec{d}^{\left(k\right)}\right)= \vec{a}_{\textrm{in},j}^\mathsf{T}\vec{x}^{\left(k\right)}=b_{\textrm{in},j}$
for all $\alpha$. Since the constraints in $W_{k}$ were satisfied
at $\vec{x}^{\left(k\right)}$, they are also satisfied at $\vec{x}^{\left(k\right)}+\alpha\vec{d}^{\left(k\right)}$,
for any value of $\alpha$.

Supposing for the moment that the optimal $\vec{d}^{\left(k\right)}$
is nonzero, we need to decide how far to move along this direction.
If $\vec{x}^{\left(k\right)}+\vec{d}^{\left(k\right)}$ is feasible
with respect to all the constraints, we set $\vec{x}^{\left(k+1\right)}=\vec{x}^{\left(k\right)}+\vec{d}^{\left(k\right)}$.
Otherwise, we set
 \begin{equation}
\vec{x}^{\left(k+1\right)}=\vec{x}^{\left(k\right)}+\alpha_{k}\vec{d}^{\left(k\right)}\,,
 \end{equation}
where the step-length parameter $\alpha_{k}$ is chosen to be the
largest value in the range $\left[0,1\right]$ for which all constraints
are satisfied. We can derive an explicit definition of $\alpha_{k}$
by considering what happens to the constraints $j\notin W_{k}$, since
the constraints $j\in W_{k}$ will certainly be satisfied regardless
of the choice of $\alpha_{k}$. If $ \vec{a}_{\textrm{in},j}^\mathsf{T}\vec{d}^{\left(k\right)}\leq0$
for some $j\notin W_{k}$, then for all $\alpha_{k}\geq0$, we have
$ \vec{a}_{\textrm{in},j}^\mathsf{T}\left(\vec{x}^{\left(k\right)}+\alpha_{k}\vec{d}^{\left(k\right)}\right)\leq \vec{a}_{\textrm{in},j}^\mathsf{T}\vec{x}^{\left(k\right)}\leq b_{\textrm{in},j}$.
Hence, constraint $j$ will be satisfied for all nonnegative choices
of the step-length parameter. Whenever $ \vec{a}_{\textrm{in},j}^\mathsf{T}\vec{d}^{\left(k\right)}>0$
for some $j\notin W_{k}$, however, we have that $ \vec{a}_{\textrm{in},j}^\mathsf{T}\left(\vec{x}^{\left(k\right)}+\alpha_{k}\vec{d}^{\left(k\right)}\right)\leq b_{\textrm{in},j}$
only if
 \begin{equation}
\alpha_{k}\leq\frac{b_{\textrm{in},j}- \vec{a}_{\textrm{in},j}^\mathsf{T}\vec{x}^{\left(k\right)}}{ \vec{a}_{\textrm{in},j}^\mathsf{T}\vec{d}^{\left(k\right)}}\,.
 \end{equation}
To maximize the decrease in $f$, we want $\alpha_{k}$ to be as large
as possible in $\left[0,1\right]$ subject to retaining feasibility,
so we obtain the following definition
 \begin{equation}
\alpha_{k}\overset{\textrm{def}}{=}\min\left\{ 1,\min_{j\notin W_{k}, \vec{a}_{\textrm{in},j}^\mathsf{T}\vec{d}^{\left(k\right)}>0}\frac{b_{\textrm{in},j}- \vec{a}_{\textrm{in},j}^\mathsf{T}\vec{x}^{\left(k\right)}}{ \vec{a}_{\textrm{in},j}^\mathsf{T}\vec{d}^{\left(k\right)}}\right\} \,.
 \end{equation}
We call the constraints $j$ for which this minimum is achieved the
_blocking constraints_. Note that it is quite possible for $\alpha_{k}$
to be zero, because we could have $ \vec{a}_{\textrm{in},j}^\mathsf{T}\vec{d}^{\left(k\right)}>0$
for some constraint $j$ that is active at $\vec{x}^{\left(k\right)}$
but not a member of the current working set $W_{k}$.

If $\alpha_{k}<1$, that is, the step along $\vec{d}_{k}$ was blocked
by some constraint not in $W_{k}$, a new working set $W_{k+1}$ is
constructed by adding one of the blocking constraints to $W_{k}$.
We continue to iterate in this manner, adding constraints to the working
set until the subproblem has solution $\vec{d}^{\circ}=\vec{0}$.
Since $\vec{d}^{\circ}=\vec{0}$ satisfy the optimality condition, we have that
 \begin{equation}
\sum_{i=1}^{m}\lambda_{i}^{\circ}\vec{a}_{\textrm{eq},i}+\sum_{j\in W^{\circ}}\mu_{j}^{\circ}\vec{a}_{\textrm{in},j}=-\vec{g}^{\circ}=-Q\vec{x}^{\circ}+\vec{c}\,,
 \end{equation}
for some Lagrange multipliers $\lambda_{i}^{\circ}$, $i=1,\dots,m$
and $\mu_{j}^{\circ}$, $j\in W^{\circ}$. It follows that $\vec{x}^{\circ}$,
$\vec{\lambda}^{\circ}$ and $\vec{\mu}^{\circ}$ satisfy the second
KKT condition, if we define the multipliers corresponding to the inequality
constraints not in the working set to be zero. Because of the control
imposed on the step length, $\vec{x}^{\circ}$ is also feasible with
respect to all the constraints, so the third, fourth and fifth KKT
conditions are satisfied at this point.

We now examine the signs of the KKT multipliers in the working set,
that is, the indices $j\in W^{\circ}$. If these conditions are all
nonnegative, the first KKT condition is also satisfied, so we conclude
that $\vec{x}^{\circ}$ is a KKT point for the original problem. In
fact, since $Q$ is positive semidefinite, we have that $\vec{x}^{\circ}$
is a global minimum.

If, on the other hand, on or more of the multipliers $\mu_{j}^{\circ}$,
$j\in W^{\circ}$, are negative, the first KKT condition is not satisfied
and the objective function $f$ may be decreased by dropping one of
these constraints. Thus, we remove an index $j$ corresponding to
one of the negative multipliers from the working set and solve a new
subproblem for the next step. While any index $j$ for which $\mu_{j}^{\circ}<0$
usually will yield in a direction $\vec{d}$ along which the algorithm
can make progress, the most negative multiplier is often chosen in
practice. This choice is motived by a sensitivity analysis, which
shows that the rate of decrease in the objective function when one
constraint is removed, is proportional to the magnitude of the Lagrange
multiplier for that constraint.

## Example

Apply the active-set method to the following problem:
\begin{align*}
\min_{\vec{x}}\, & f\left(\vec{x}\right)=\left(x_{1}-1\right)^{2}+\left(x_{2}-2.5\right)^{2}\\
\textrm{subject to}\, & \begin{cases}
-x_{1}+2x_{2}-2\leq0\,,\\
x_{1}+2x_{2}-6\leq0\,.\\
x_{1}-2x_{2}-2\leq0\,,\\
-x_{1}\leq0\,,\\
-x_{2}\leq0\,.
\end{cases}
\end{align*}

```julia
using Plots
using LaTeXStrings

Q = Float64[2 0
     0 2]
c = Float64[2, 5]
A = Float64[-1  2
      1  2
      1 -2
     -1  0
      0 -1]
b = Float64[2, 6, 2, 0, 0]

x = -1:0.05:5
y = -1:0.05:5
z = Surface((x,y)->(0.5 .* [x y]*Q*[x;y] .- transpose(c)*[x;y] .+ 0.5 .* transpose(c)*Q*c)[1], x, y)
contour(x, y, z, levels=35)
plot!(x, (2 .+ x) ./ 2, linestyle=:dash, label=L"-x_1+2x_2-2\le 0")
plot!(x, (6 .- x) ./ 2, linestyle=:dash, label=L"x_1+2x_2-6\le 0")
plot!(x, (2 .- x) ./ -2, linestyle=:dash, label=L"x_1-2x_2-2\le 0")
plot!([0,2,4,2,0,0],[0,0,1,2,1,0], linewidth=2, label="domain")
```

We refer the constraints, in order, by indices $1$ through $5$.
For this problem it is easy to determine a feasible initial point;
say $\vec{x}^{\left(0\right)}=\begin{pmatrix}2 & 0\end{pmatrix}\mathsf{T}$.
Constraints $3$ and $5$ are active at this point, and we set $W_{0}=\left\{ 3,5\right\} $.
Note that we could just as validly have chosen $W_{0}=\left\{ 5\right\} $
or $W_{0}=\left\{ 3\right\} $ or even $W_{0}=\emptyset$; each choice
would lead the algorithm to perform somewhat differently.

```julia
x = [2, 0]
g = Q*x - c
ain = [reshape(A[3,:], 1, 2); reshape(A[5,:], 1, 2)]
sol = [Q transpose(ain)
       ain zeros(2,2)] \ [-g; 0; 0]
```

Since $\vec{x}^{\left(0\right)}$ lies on a vertex of the feasible
region, it is obviously a minimizer of the objective function $f$
with respect to the working set $W_{0}$; that is, the solution of
the subproblem with $k=0$ is $\vec{d}^{\left(0\right)}=\vec{0}$.
We can then find the multipliers $\mu_{3}^{\circ}$ and $\mu_{5}^{\circ}$
associated with the active constraints. Substitution of the data from
our problem yields
\begin{equation}
\begin{pmatrix}-1\\
2
\end{pmatrix}\lambda_{3}^{\circ}+\begin{pmatrix}0\\
1
\end{pmatrix}\lambda_{5}^{\circ}=\begin{pmatrix}2\\
-5
\end{pmatrix}\,,
\end{equation}
which has solution $\lambda_{3}^{\circ}=-2$ and $\lambda_{5}^{\circ}=-1$.

We now remove constraint $3$ from the working set, because it has
the most negative multiplier, and set $W_{1}=\{5\}$.

```julia
x = [2, 0]
g = Q*x - c
ain = reshape(A[5,:], 1, 2)
sol = [Q transpose(ain)
       ain zeros(1,1)] \ [-g; 0]
```

We begin iteration $1$ by finding the solution of the subproblem for $k=1$, which is
$\vec{d}^{\left(1\right)}= \begin{pmatrix}-1 & 0\end{pmatrix}^\mathsf{T}$.

```julia
d = sol[1:2]
α = min(1.0, [(reshape(A[j,:], 1, 2) * d)[1,1] ≤ 0 ? 1.0 : ((b[j] .- reshape(A[j,:], 1, 2) * x) / (reshape(A[j,:], 1, 2) * d))[1,1] for j in (1,2,3,4)]...)
```

The step-length formula yields $\alpha_{1}=1$, and the new iterate
is $\vec{x}^{\left(2\right)}=\begin{pmatrix}1 & 0\end{pmatrix}^\mathsf{T}$. There are no blocking constraints, so that $W_{2}=W_{1}=\left\{ 5\right\} $

```julia
x = x + α .* d
g = Q*x - c
ain = reshape(A[5,:], 1, 2)
sol = [Q transpose(ain)
       ain zeros(1,1)] \ [-g; 0]
```

We find at the start of iteration $2$ that the solution of the
subproblem is $\vec{d}^{\left(2\right)}=\vec{0}$. We deduce that
the Lagrange multiplier for the lone working constraint is $\lambda_{5}^{\circ}=-5$,
so we drop the working set to obtain $W_{3}=\emptyset$.

```julia
g = Q*x - c
sol = Q \ (-g)
```

Iteration $3$ starts by solving the unconstrained problem, to obtain
the solution $\vec{d}^{\left(3\right)}=\begin{pmatrix}0 & 2.5\end{pmatrix}^\mathsf{T}$.

```julia
d = sol[1:2]
α = min(1.0, [(reshape(A[j,:], 1, 2) * d)[1,1] ≤ 0 ? 1.0 : ((b[j] .- reshape(A[j,:], 1, 2) * x) / (reshape(A[j,:], 1, 2) * d))[1,1] for j in (1,2,3,4,5)]...)
```

The step-length formula yields a step length of $\alpha_{3}=0.6$
and a new iterate $\vec{x}^{\left(4\right)}=\begin{pmatrix}1 & 1.5\end{pmatrix}^\mathsf{T}$. There
is a single blocking constraint (constraint $1$), so we obtain $W_{4}=\left\{ 1\right\} $.

```julia
x = x + α .* d
g = Q*x - c
ain = reshape(A[1,:], 1, 2)
sol = [Q transpose(ain)
       ain zeros(1,1)] \ [-g; 0]
```

The solution of the subproblem for $k=4$ is then $\vec{d}^{\left(4\right)}=\begin{pmatrix}0.4 & 0.2\end{pmatrix}^\mathsf{T}$.

```julia
d = sol[1:2]
α = min(1.0, [(reshape(A[j,:], 1, 2) * d)[1,1] ≤ 0 ? 1.0 : ((b[j] .- reshape(A[j,:], 1, 2) * x) / (reshape(A[j,:], 1, 2) * d))[1,1] for j in (2,3,4,5)]...)
```

The new step-length is $1$. There are no blocking constraints
on this step, so the next working set in unchanged: $W_{5}=\left\{ 1\right\} $. The new iterate is $\vec{x}^{\left(5\right)}=\begin{pmatrix}1.4 & 1.7\end{pmatrix}^\mathsf{T}$.

```julia
x = x + α .* d
g = Q*x - c
ain = reshape(A[1,:], 1, 2)
sol = [Q transpose(ain)
       ain zeros(1,1)] \ [-g; 0]
```

Finally, we solve the subproblem for $k=5$ to obtain a solution $\vec{d}^{\left(5\right)}=\vec{0}$.
We find a multiplier $\mu_{1}^{\circ}=0.8$, so we have found the
solution. Wet set $\vec{x}^{\star}=\begin{pmatrix}1.4 & 1.7\end{pmatrix}^\mathsf{T}$
and terminate.

```julia
x = -1:0.05:5
y = -1:0.05:5
z = Surface((x,y)->(0.5 .* [x y]*Q*[x;y] .- transpose(c)*[x;y] .+ 0.5 .* transpose(c)*Q*c)[1], x, y)
contour(x, y, z, levels=35)
plot!(x, (2 .+ x) ./ 2, linestyle=:dash, label=L"-x_1+2x_2-2\le 0")
plot!(x, (6 .- x) ./ 2, linestyle=:dash, label=L"x_1+2x_2-6\le 0")
plot!(x, (2 .- x) ./ -2, linestyle=:dash, label=L"x_1-2x_2-2\le 0")
plot!([0,2,4,2,0,0],[0,0,1,2,1,0], linewidth=2, label="domain")
plot!([2,2,1,1,1,1.4], [0,0,0,0,1.5,1.7], linewidth=2, label="iterates", markershape=:circle)
```

## Julia

The package `GeneralQP` solves the quadratic programming problems using the function `solve(Q, -c, A, b, x₀)`.

```julia
#using Pkg
#pkg"add https://github.com/oxfordcontrol/GeneralQP.jl"
using GeneralQP

sol = GeneralQP.solve(Q, -c, A, b, Float64[2,0])
```

## Sequential Quadratic Programming

We consider the general constrained problem
\begin{align*}
\min\, & f\left(\vec{x}\right)\\
\textrm{subject to} & \begin{cases}
\vec{h}\left(\vec{x}\right)=\vec{0}\\
\vec{g}\left(\vec{x}\right)\leq\vec{0}
\end{cases}
\end{align*}
where $f:\mathbb{R}^{n}\rightarrow\mathbb{R}$, $\vec{h}:\mathbb{R}^{n}\rightarrow\mathbb{R}^{m}$,
$m\leq n$, and $\vec{g}:\mathbb{R}^{n}\rightarrow\mathbb{R}^{p}$. The idea
behind the _sequential quadratic programming_ (SQP) approach
is to model the general problem at the current iterate $\vec{x}^{\left(k\right)}$
by a quadratic programming subproblem, then use the minimizer of this
subproblem to define a new iterate $\vec{x}^{\left(k+1\right)}$.
The challenge is to design the quadratic subproblem so that it yields
a good step for the general optimization problem.

We know that the extended Lagrangian function for this problem is

\begin{equation}
\mathcal{L}\left(\vec{x},\vec{\lambda},\vec{\mu}\right)=f\left(\vec{x}\right)+\vec{\lambda}^\mathsf{T}\vec{h}\left(\vec{x}\right)+\vec{\mu}^\mathsf{T}\vec g\left(\vec{x}\right)\,.
\end{equation}

Applying Newton's method to the Lagrangian function and linearizing
both the equality and the inequality constraints yields the following
subproblem
\begin{align*}
\min_{\vec{d}^{\left(k\right)}}\, & \frac{1}{2}\left(\vec{d}^{\left(k\right)}\right)^\mathsf{T} \mathsf{H} \mathcal{L}\left(\vec{x}^{\left(k\right)},\vec{\lambda}^{\left(k\right)},\vec{\mu}^{\left(k\right)}\right)\vec{d}^{\left(k\right)}+ \mathsf{D} \mathcal{L}\left(\vec{x}^{\left(k\right)},\vec{\lambda}^{\left(k\right)},\vec{\mu}^{\left(k\right)}\right)\vec{d}^{\left(k\right)}\\
\textrm{subject to}\, & \begin{cases}
 \mathsf{J}\vec{h}\left(\vec{x}^{\left(k\right)}\right)\vec{d}^{\left(k\right)}=-\vec{h}\left(\vec{x}^{\left(k\right)}\right)\\
 \mathsf{D} g_{j}\left(\vec{x}^{\left(k\right)}\right)\vec{d}^{\left(k\right)}=-g_{j}\left(\vec{x}^{\left(k\right)}\right)\,, & j\in W_{k}\,,
\end{cases}
\end{align*}

where $\mu_{j}^{\left(k\right)}=0$, for all
$j\notin W_{k}$. We can use the active-set method for quadratic programming
to solve this subproblem. The new iterate is given by $\vec{x}^{\left(k+1\right)}$,
$\vec{\lambda}^{\left(k+1\right)}$, $\vec{\mu}^{\left(k+1\right)}$
and $W_{k+1}$.

If the SQP method is able to identify the optimal active set then
it will act like a Newton method for equality-constrained optimization
and will converge rapidly.

It is also remarkable that, far from the solution, the SQP approach
is usually able to improve the estimate of the active set and guide
the iterates towards a solution.

Non-quadratic objective functions, however, can impede progress of
the SQP algorithm, a phenomenon known as the Maratos effect. Steps
that make good progress toward a solution are rejected and the algorithm
fails to converge rapidly. These difficulties can be overcome by means
of a _second-order correction_.

```julia
#using Pkg
#pkg"add NLopt"

using NLopt

function myfunc(x::Vector, grad::Vector)
    if length(grad) > 0
        grad[1] = 0
        grad[2] = 0.5/sqrt(x[2])
    end
    return sqrt(x[2])
end

function myconstraint(x::Vector, grad::Vector, a, b)
    if length(grad) > 0
        grad[1] = 3a * (a*x[1] + b)^2
        grad[2] = -1
    end
    (a*x[1] + b)^3 - x[2]
end

opt = Opt(:LD_SLSQP, 2)
opt.lower_bounds = [-Inf, 0.]
opt.xtol_rel = 1e-4

opt.min_objective = myfunc
inequality_constraint!(opt, (x,g) -> myconstraint(x,g,2,0), 1e-8)
inequality_constraint!(opt, (x,g) -> myconstraint(x,g,-1,1), 1e-8)

(minf,minx,ret) = optimize(opt, [1.234, 5.678])
numevals = opt.numevals # the number of function evaluations
println("got $minf at $minx after $numevals iterations (returned $ret)")
```

```julia

```


# Taylor series expansion and error analysis

## Taylor series expansion

During the lecture, we talked several times about the Taylor series as a way to formally determine the differential of a property. As a brief reminder, the Taylor series can be used to approximate the value of a function $f(x)$ at a location $f(x +h)$ with the help of a polynomial series:

$$f(x+h) = \sum_{i=1}^n \frac{h^n}{n!} \frac{d^nf}{dx^n}(h)$$

As an example in the lecture, we approximated the sine function at several locations. This was pretty easy as the derivatives of $f(x) = \sin(x)$ can easily be calculated.

However, this is not always the case. Think about a tiny bit more complex function: $f(x) = \sin(x^2)$. Determining higher-order differentials quickly gets quite tricky - try it out, to convince yourself:

$$f(x) = \sin(x^2)$$

$$\frac{df}{dx} = 2x \cos(x^2)$$

$$\frac{d^2f}{dx^2} = ?$$

$$\frac{d^3f}{dx^3} = ??$$

Creating more differentials is a very systematic process and maybe not difficult - but requires quickly a lot of bookkeeping and is prone to errors - and it becomes very repetitive if you have to do it very often...

As soon as you read things in descriptions like: "requires bookkeeping", "not difficult systematic process" and "repetitive", I hope you think by now: **we should do that in Python**! Yes, correct!

So luckily, someone did the hard work of coding all these things into Python for us before: it is implemented in the sympy package for symbolic computation.

We'll have a look at some of the basic features in this notebook - and then apply it to determine Taylor Series Expansions, and finally to the accuracy estimation that we did in the last lecture.

```python
import sympy as sym
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
# to get LaTeX function rendereing:
sym.init_printing()
```

```python
sym.var('x')
```

### Sine function

```python
def plot_taylor(x0=0.0, n=1):
    func = sym.sin(x)#/x
    taylor = sym.series(func, x0=x0, n=n+1).removeO()

    evalfunc = sym.lambdify(x, func, modules=['numpy'])
    evaltaylor = sym.lambdify(x, taylor, modules=['numpy'])

    t = np.linspace(-2*np.pi, 3*np.pi, 100)
    plt.figure(figsize=(10,8))
    plt.plot(t, evalfunc(t), 'b', label='sin(x)')
    plt.plot(t, evaltaylor(t), 'r', label='Taylor')
    plt.plot(x0, evalfunc(x0), 'go', label='x0', markersize = 12)
    plt.legend(loc='upper left')
    plt.xlim([-1*np.pi, 2*np.pi])
    plt.ylim([-3,3])
    plt.show()
```

```python
from ipywidgets import interactive
from IPython.display import Audio, display
```

```python
plt.figure(figsize=(10,8))
plt.style.use("bmh")
```

    <Figure size 720x576 with 0 Axes>

```python
v = interactive(plot_taylor, x0=(0.0,np.pi,np.pi/10.), n=(1,8), r=(2,4))
display(v)
```

    interactive(children=(FloatSlider(value=0.0, description='x0', max=3.141592653589793, step=0.3141592653589793)…

### Logarithm function

```python
def plot_taylor_lnx(x0=0.0, n=1):
    """Same method, different base function"""
    func = sym.ln(x)#/x
    taylor = sym.series(func, x0=x0, n=n+1).removeO()

    evalfunc = sym.lambdify(x, func, modules=['numpy'])
    evaltaylor = sym.lambdify(x, taylor, modules=['numpy'])

    t = np.linspace(0.01, 10, 100)
    plt.figure(figsize=(10,8))
    plt.plot(t, evalfunc(t), 'b', label='ln(x)')
    plt.plot(t, evaltaylor(t), 'r', label='Taylor')
    plt.plot(x0, evalfunc(x0), 'go', label='x0', markersize = 12)
    plt.legend(loc='upper left')
    plt.xlim([-0.2,10])
    plt.ylim([-3,3])
    plt.show()
```

```python
v = interactive(plot_taylor_lnx, x0=(1,3,0.2), n=(1,8), r=(2,4))
display(v)
```

    interactive(children=(FloatSlider(value=1.0, description='x0', max=3.0, min=1.0, step=0.2), IntSlider(value=1,…

## Error analysis

Note that, in previous examples: for higher degrees of approximation, the local solution gets better, but the extrapolation gets worse!

Analyzing this in a bit more detail:

```python
def plot_taylor_error(x0=0.0, n=1, s=1):
    func = sym.sin(x)#/x
    taylor = sym.series(func, x0=x0, n=n+1).removeO()

    evalfunc = sym.lambdify(x, func, modules=['numpy'])
    evaltaylor = sym.lambdify(x, taylor, modules=['numpy'])

    t = np.linspace(-2*np.pi, 3*np.pi, 100)
    fig = plt.figure(figsize=(10,8))
    ax1 = fig.add_subplot(211)
    # plt.plot(t, evalfunc(t), 'b', label='sin(x)')
    ax1.plot(t, evaltaylor(t)-evalfunc(t), 'r', label='Taylor error')
    ax1.plot(x0, 0, 'go', label='x0', markersize = 12)
    plt.legend(loc='upper left')
    ax1.set_xlim([-1*np.pi, 2*np.pi])
    ax1.set_ylim([-s*3,s*3])
    ax2 = fig.add_subplot(212)
    ax2.plot(t, evalfunc(t), 'b', label='sin(x)')
    ax2.plot(t, evaltaylor(t), 'r', label='Taylor')
    ax2.plot(x0, evalfunc(x0), 'go', label='x0', markersize = 12)
    plt.legend(loc='upper left')
    ax2.set_xlim([-1*np.pi, 2*np.pi])
    ax2.set_ylim([-3,3])

    plt.show()
```

```python
v = interactive(plot_taylor_error, x0=(0,np.pi,0.1*np.pi), n=(1,8), r=(2,4), s=(1,10))
display(v)
```

    interactive(children=(FloatSlider(value=0.0, description='x0', max=3.141592653589793, step=0.3141592653589793)…

```python

```

```python

```