# Supplemental Information E - Non-linear Regression

(c) 2017 the authors. This work is licensed under a [Creative Commons Attribution License CC-BY 4.0](https://creativecommons.org/licenses/by/4.0/). All code contained herein is licensed under an [MIT license](https://opensource.org/licenses/MIT).

```python
# For operating system interaction
import os
import glob
import datetime
import sys

# For loading .pkl files.
import pickle

# For scientific computing
import numpy as np
import pandas as pd
import scipy.special
import statsmodels.tools.numdiff as smnd # to compute the Hessian matrix

# Import custom utilities
import mwc_induction_utils as mwc

# Useful plotting libraries
import matplotlib.pyplot as plt
import matplotlib.cm as cm
import seaborn as sns
mwc.set_plotting_style()

# Magic function to make matplotlib inline; other style specs must come AFTER
%matplotlib inline

# This enables SVG graphics inline
%config InlineBackend.figure_format = 'svg'
```

    /Users/gchure/anaconda/lib/python3.4/site-packages/matplotlib/__init__.py:872: UserWarning: axes.color_cycle is deprecated and replaced with axes.prop_cycle; please use the latter.
      warnings.warn(self.msg_depr % (key, alt_key))

# Non-linear regression.

In order to obtain the MWc parameters given the fold-change measurements and a credible region on such parameters we will use a Bayesian approach to perform a non-linear regression.
Our theoretical model dictates that the fold change in gene expression is given by

\begin{equation}
    \text{fold-change} = \frac{1}{1 + \frac{R p_{act}(c)}{N_{NS}} e^{-\beta \Delta \varepsilon_{RA}}},
\end{equation}

where $p_{act}(c)$ is given by

\begin{equation}
    p_{act}(c) = \frac{\left( 1 + c e^{\tilde{k_A}}\right)^2}{\left( 1 + c e^{\tilde{k_A}}\right)^2 + e^{-\beta \Delta\varepsilon_{AI}} \left( 1 + c e^{\tilde{k_I}}\right)^2}.
\end{equation}

We define $\tilde{k_A} = -\ln K_A$ and $\tilde{k_I} = -\ln K_I$ for convenience during the regression.

If we want to fit the parameters $\tilde{k_A}$ and $\tilde{k_I}$, by Bayes theorem we have that

\begin{equation}
P(\tilde{k_A}, \tilde{k_I} \mid D, I) \propto P(D \mid \tilde{k_A}, \tilde{k_I}, I) \cdot P(\tilde{k_A}, \tilde{k_I} \mid I),
\end{equation}

where $D$ is the experimental data and $I$ is all the previous information.

## Gaussian likelihood and constant error

The simplest model to perform the regression is to assume the following:
1. each measurement is independent
2. the errors are Gaussian distributed
3. this error is constant along the range of IPTG.

Now it is important to indicate that each element of $D$ is a "pair" of a dependent variable (the experimental fold change $fc_{exp}$) and the independent variables (the repressor copy number $R$, the binding energy $\Delta \varepsilon_{RA}$ and the IPTG concentration $C$). With this in hand we implement the first assumption as

\begin{equation}
P(D \mid \tilde{k_A}, \tilde{k_I}, I) = \prod_{i = 1}^n P(fc_{exp}^{(i)} \mid \tilde{k_A}, \tilde{k_I}, R^{(i)}, \Delta\varepsilon_{RA}^{(i)}, C^{(i)}, I),
\end{equation}

where $n$ is the number of data points and the superscript $(i)$ indicates the $i$th element of $D$.

Implementing the second and third assumption we obtain

\begin{equation}
P(D \mid \tilde{k_A}, \tilde{k_I}, \sigma, I) = \left( 2\pi\sigma^2 \right)^{-\frac{n}{2}} \prod_{i = 1}^n \exp \left[ \frac{1}{2 \sigma^2} \left( fc_{exp}^{(i)} - fc\left(\tilde{k_A}, \tilde{k_I}, R^{(i)}, \Delta\varepsilon_{RA}^{(i)}, C^{(i)} \right) \right)^2 \right],
\end{equation}

where we include the parameter $\sigma$ associated with the Gaussian distributed error.

For the priors we can assume that the 3 parameters $\tilde{k_A}, \tilde{k_I}$ and $\sigma$ are not only independent, but since they have a uniform prior in log scale they can have a Jeffres' prior, i.e.

\begin{equation}
P(\tilde{k_A}, \tilde{k_I}, \sigma \mid I) \equiv \frac{1}{\tilde{k_A}}\cdot\frac{1}{\tilde{k_I}}\cdot\frac{1}{\sigma}
\end{equation}

Putting all the pieces together we can compute the posterior distribution as

\begin{equation}
P(\tilde{k_A}, \tilde{k_I}, \sigma \mid D, I) \propto \left( 2\pi\sigma^2 \right)^{-\frac{n}{2}} \prod_{i = 1}^n \exp \left[ \frac{1}{2 \sigma^2} \left( fc_{exp}^{(i)} - fc\left(\tilde{k_A}, \tilde{k_I}, R^{(i)}, \Delta\varepsilon_{RA}^{(i)}, C^{(i)} \right) \right)^2 \right] \frac{1}{\tilde{k_A}}\cdot\frac{1}{\tilde{k_I}}\cdot\frac{1}{\sigma}
\end{equation}

But we are left with the nuance parameter $\sigma$ that we don't care about. To eliminate this parameter we need to marginalize over all values of $\sigma$ as

\begin{equation}
P(\tilde{k_A}, \tilde{k_I} \mid D, I) = \int_{- \infty}^\infty d\sigma P(\tilde{k_A}, \tilde{k_I}, \sigma \mid D, I).
\end{equation}

And when everything settles down, i.e. after some nasty integration, we find that the posterior is given by the student-t distribution

\begin{equation}
P(\tilde{k_A}, \tilde{k_I} \mid D, I) \propto \left[ \sum_{i=1}^n \left( fc_{exp}^{(i)} - fc\left(\tilde{k_A}, \tilde{k_I}, R^{(i)}, \Delta\varepsilon_{RA}^{(i)}, C^{(i)} \right) \right)^2 \right]^{\frac{n}{2}}.
\end{equation}

Numerically is always better to work with the log posterior probability, therefore for the student-t distribution we have that

\begin{equation}
\ln P(\tilde{k_A}, \tilde{k_I} \mid D, I) \propto \frac{n}{2} \ln \left[ \sum_{i=1}^n \left( fc_{exp}^{(i)} - fc\left(\tilde{k_A}, \tilde{k_I}, R^{(i)}, \Delta\varepsilon_{RA}^{(i)}, C^{(i)} \right) \right)^2 \right]
\end{equation}

Let's code up the functions to compute the theoretical fold-change

```python
# define a funciton to compute the fold change as a funciton of IPTG
def pact(IPTG, ea, ei, epsilon=4.5):
    '''
    Returns the probability of a repressor being active as described by the MWC
    model.
    Parameter
    ---------
    IPTG : array-like.
        concentrations of inducer on which to evaluate the function
    ea, ei : float.
        minus log of the dissociation constants of the active and the inactive
        states respectively
    epsilon : float.
        energy difference between the active and the inactive state
    Returns
    -------
    pact : float.
        probability of a repressor of being in the active state. Active state is
        defined as the state that can bind to the DNA.
    '''
    pact = (1 + IPTG * np.exp(ea))**2 / \
    ((1 + IPTG * np.exp(ea))**2 + np.exp(-epsilon) * (1 + IPTG * np.exp(ei))**2)
    return pact

def fold_change(IPTG, ea, ei, epsilon, R, epsilon_r):
    '''
    Returns the gene expression fold change according to the thermodynamic model
    with the extension that takes into account the effect of the inducer.
    Parameter
    ---------
    IPTG : array-like.
        concentrations of inducer on which to evaluate the function
    ea, ei : float.
        minus log of the dissociation constants of the active and the inactive
        states respectively
    epsilon : float.
        energy difference between the active and the inactive state
    R : array-like.
        repressor copy number for each of the strains. The length of this array
        should be equal to the IPTG array. If only one value of the repressor is
        given it is asssume that all the data points should be evaluated with
        the same repressor copy number
    epsilon_r : array-like
        repressor binding energy. The length of this array
        should be equal to the IPTG array. If only one value of the binding
        energy is given it is asssume that all the data points
        should be evaluated with the same repressor copy number

    Returns
    -------
    fold-change : float.
        gene expression fold change as dictated by the thermodynamic model.
   '''
    return 1 / (1 + 2 * R / 5E6 * pact(IPTG, ea, ei, epsilon) * \
            (1 + np.exp(-epsilon)) * np.exp(-epsilon_r))
```

Now let's code up the log posterior

```python
def log_post(param, indep_var, dep_var):
    '''
    Computes the log posterior for a single set of parameters.
    Parameters
    ----------
    param : array-like.
        param[0] = epsilon_a
        param[1] = epsilon_i
    indep_var : n x 3 array.
        series of independent variables to compute the theoretical fold-change.
        1st column : IPTG concentration
        2nd column : repressor copy number
        3rd column : repressor binding energy
    dep_var : array-like
        dependent variable, i.e. experimental fold-change. Then length of this
        array should be the same as the number of rows in indep_var.

    Returns
    -------
    log_post : float.
        the log posterior probability
    '''
    # unpack parameters
    ea, ei = param

    # unpack independent variables
    IPTG, R, epsilon_r = indep_var[:, 0], indep_var[:, 1], indep_var[:, 2]

    # compute the theoretical fold-change
    fc_theory = fold_change(IPTG, ea, ei, 4.5, R, epsilon_r)

    # return the log posterior
    return -len(dep_var) / 2 * np.log(np.sum((dep_var - fc_theory)**2))
```

# Testing the functions with only 1 strain and one operator

Now it is time to test this! But first let's read the data

```python
datadir = '../../data/'
# read the list of data-sets to ignore
df = pd.read_csv(datadir + 'flow_master.csv', comment='#')

# Now we remove the autofluorescence and delta values
df = df[(df.rbs != 'auto') & (df.rbs != 'delta')]

df.head()
```

      date
      username
      operator
      binding_energy
      rbs
      repressors
      IPTG_uM
      mean_YFP_A
      mean_YFP_bgcorr_A
      fold_change_A

      2
      20160804
      mrazomej
      O2
      -13.9
      RBS1L
      870
      0.0
      3624.474605
      111.851286
      0.007146

      3
      20160804
      mrazomej
      O2
      -13.9
      RBS1
      610
      0.0
      3619.786265
      107.162946
      0.006847

      4
      20160804
      mrazomej
      O2
      -13.9
      RBS1027
      130
      0.0
      3717.019527
      204.396208
      0.013059

      5
      20160804
      mrazomej
      O2
      -13.9
      RBS446
      62
      0.0
      3854.650585
      342.027265
      0.021853

      6
      20160804
      mrazomej
      O2
      -13.9
      RBS1147
      30
      0.0
      4169.802851
      657.179531
      0.041988

Let's focus first on a single strain: `O2 - RBS1027`

```python
rbs = df[(df.rbs=='RBS1027') & (df.binding_energy==-13.9)]

plt.figure()
for date in rbs.date.unique():
    plt.plot(rbs[rbs.date==date].IPTG_uM / 1E6,
             rbs[rbs.date==date].fold_change_A, 'o',
            label=str(date), alpha=0.7)
plt.xscale('symlog', linthreshx=1E-7)
plt.xlim(left=-5E-9)
plt.xlabel('[IPTG] (M)')
plt.ylabel('fold-change')
plt.legend(loc='upper left', fontsize=11)
plt.title('RBS1027 lacI/cell = 130')
plt.tight_layout()
```

### Plotting the posterior distribution

Before computing the MAP and doing the proper regression, let's look at the posterior itself

```python
# Parameter values to plot
ea = np.linspace(-5.2, -4.7, 100)
ei = np.linspace(0.45, 0.7, 100)

# make a grid to plot
ea_grid, ei_grid = np.meshgrid(ea, ei)

# compute the log posterior
indep_var = rbs[['IPTG_uM', 'repressors', 'binding_energy']]
dep_var = rbs.fold_change_A

log_posterior = np.empty_like(ea_grid)
for i in range(len(ea)):
    for j in range(len(ei)):
        log_posterior[i, j] = log_post([ea_grid[i, j], ei_grid[i, j]],
                                       indep_var.values, dep_var.values)

# Get things to scale better
log_posterior -= log_posterior.max()

# plot the results
plt.figure()
plt.contourf(ea_grid, ei_grid, np.exp(log_posterior), alpha=0.7,
             cmap=plt.cm.Blues)
plt.xlabel(r'$\tilde{k_A}$')
plt.ylabel(r'$\tilde{k_I}$')
plt.title('Posterior probability, O2 - RBS1027')
```

    <matplotlib.text.Text at 0x11b31a518>

### Computing the MAP

In order to compute the Maximum a posteriori parameters or MAP for short we will use the `scipy.optimize.leastsq()` function.
For this we need to define a function that computes the residuals.

```python
def resid(param, indep_var, dep_var, epsilon=4.5):
    '''
    Residuals for the theoretical fold change.

    Parameters
    ----------
    param : array-like.
        param[0] = epsilon_a
        param[1] = epsilon_i
    indep_var : n x 3 array.
        series of independent variables to compute the theoretical fold-change.
        1st column : IPTG concentration
        2nd column : repressor copy number
        3rd column : repressor binding energy
    dep_var : array-like
        dependent variable, i.e. experimental fold-change. Then length of this
        array should be the same as the number of rows in indep_var.

    Returns
    -------
    fold-change_exp - fold-change_theory
    '''
    # unpack parameters
    ea, ei = param

    # unpack independent variables
    IPTG, R, epsilon_r = indep_var[:, 0], indep_var[:, 1], indep_var[:, 2]

    # compute the theoretical fold-change
    fc_theory = fold_change(IPTG, ea, ei, epsilon, R, epsilon_r)

    # return the log posterior
    return dep_var - fc_theory
```

To find the most likely parameters we need to provide an initial guess. The optimization routine only finds a local maximum and is not in general guaranteed to converge. Therefore, the initial guess can be very important.

After that we will be ready to use `scipy.optimize.leastsq()` to compute the MAP. We uses the args kwarg to pass in the other arguments to the resid() function. In our case, these arguments are the data points. The `leastsq()` function returns multiple values, but the first, the optimal parameter values (the MAP), is all we are interested in.

```python
# Initial guess
p0 = np.array([1, 7]) # From plotting the posterior

# Extra arguments given as tuple
args = (indep_var.values, dep_var.values)

# Compute the MAP
popt, _ = scipy.optimize.leastsq(resid, p0, args=args)

# Extract the values
ea, ei = popt

# Print results
print("""
The most probable parameters for the MWC model
----------------------------------------------
Ka = {0:.2f} uM
Ki = {1:.3f} uM
""".format(np.exp(-ea), np.exp(-ei)))
```

    The most probable parameters for the MWC model
    ----------------------------------------------
    Ka = 140.38 uM
    Ki = 0.560 uM

Just to show that these parameters indeed give a good fit let's plot the theory and the data

```python
IPTG = np.logspace(-8, -2, 200)
fc_theory = fold_change(IPTG * 1E6, ea, ei, 4.5, R=130, epsilon_r=-13.9)

plt.figure()
plt.plot(IPTG, fc_theory, '--', label='best parameter fit', color='darkblue')
for date in rbs.date.unique():
    plt.plot(rbs[rbs.date==date].IPTG_uM / 1E6,
             rbs[rbs.date==date].fold_change_A, 'o',
             label=str(date), alpha=0.7)
plt.xscale('symlog', linthreshx=1E-7)
plt.xlim(left=-5E-9)
plt.xlabel('IPTG (M)')
plt.ylabel('fold-change')
plt.legend(loc='upper left', fontsize=11)
plt.tight_layout()
```

# Computing error bars on the parameters.

In order to get a **credible region** on our parameter estimate we will use an aproximation in which the posterior probability can be represented as a Gaussian distribution. This approximation can be justified as a truncated Taylor expansion as follows:
Given our log posterior distribution with parameters $\mathbf{\tilde{k}} = (\tilde{k_A}, \tilde{k_I})$ we can perform a Taylor expansion around our MAP $\mathbf{\tilde{k}}^*$
\begin{equation}
\ln P(\mathbf{\tilde{k}} \mid D, I) \approx \text{constant} + \frac{1}{2} \left(  \mathbf{\tilde{k} - \tilde{k}^*}\right)^T \cdot H \cdot \left(\mathbf{\tilde{k} - \tilde{k}^*}\right),
\end{equation}
where $H$ is the symmetric **Hessian matrix** whose entries are given by the second derivatives, i.e.
\begin{equation}
H_{ij} = \frac{\partial ^2 \ln P(\mathbf{\tilde{k}} \mid D, I)}{\partial \tilde{k}_i \partial \tilde{k}_j} \biggr\rvert_{\mathbf{\tilde{k}} = \mathbf{\tilde{k}^*}}.
\end{equation}

If we exponentiate this truncated expansion to remove the log we find something that remarkably resembles a multivariate Gaussian distribution
\begin{equation}
P(\mathbf{\tilde{k}} \mid D, I) \approx \text{constant} \cdot \exp \left[ \frac{1}{2} \left( \mathbf{\tilde{k}} - \mathbf{\tilde{k}^*} \right)^T \cdot H \cdot \left( \mathbf{\tilde{k}} - \mathbf{\tilde{k}^*} \right) \right].
\end{equation}

From this we can see that the Hessian matrix plays the role of the negative inverse **covariance matrix**. As a matter of fact since the second derivatives are evaluated at the MAP the Hessian is *positive definite* and therefore this matrix can be inverted, obtaining our desired covariance matrix. So if we compute the Hessian at the MAP, and then invert this matrix, the diagonal terms of this inverted matrix will be the error bars for our parameters under this Gaussian approximation of the posterior!

Let's now compute the covariance matrix. For this we will numerically compute the Hessian using the `statsmodels.tools.numdiff` package.

```python
# list the arguments to be fed to the log_post function
args = (indep_var.values, dep_var.values)

# Compute the Hessian at the map
hes = smnd.approx_hess(popt, log_post, args=args)
hes
```

    array([[ -623.2526089 ,  1291.19249453],
           [ 1291.19249453, -3351.4824521 ]])

Now that we computed the Hessian let's compute the negative inverse to get our precious covariance matrix!

```python
# Compute the covariance matrix
cov = -np.linalg.inv(hes)

cov
```

    array([[ 0.00794864,  0.00306229],
           [ 0.00306229,  0.00147816]])

Again, the diagonal terms of this matrix give the approximate variance in the regression parameters. The offdiagonal terms give the covariance, which describe how parameters relate to each other. From the plot of the posterior previously we saw that there is definitely a positive correlation between the parameters, and that is reflected by non-zero entries in these offdiagonal terms.

But recall that this is giving the error bar on $\tilde{k_A}$ and $\tilde{k_I}$, not the dissociation constants themselves. Therefore we must "propagate the error" properly by doing the proper change of variables.
For this we use the approximation that if the error on $\tilde{k_A}$ is given by $\delta \tilde{k_A}$, we can use this relationship to compute $\delta K_A$, the error on the dissociation constant.

First we know the relationshipt between $\tilde{k_A}$ and $K_A$ is
\begin{equation}
\tilde{k_A} = - \ln K_A.
\end{equation}
Differenciating both sides we obtain
\begin{equation}
\delta \tilde{k_A} = - \frac{1}{K_A} \delta K_A.
\end{equation}
We now squre both sides and take the expected value
\begin{equation}
\langle \delta \tilde{k_A} \rangle^2 = \frac{\langle \delta K_A\rangle^2}{K_A^2}.
\end{equation}
Finally we re-arrange terms to find that the error bar on the dissociation constant is given by
\begin{equation}
\delta K_A = \sqrt{\langle \delta K_A \rangle^2} = \sqrt{\langle \delta \tilde{k_A} \rangle^2 \cdot K_A^2} = \delta \tilde{k_A} \cdot K_A
\end{equation}

Now let's report the parameter values with the proper error bars!

```python
# Get the values for the dissociation constants and their respective error bars
Ka = np.exp(-ea)
Ki = np.exp(-ei)
deltaKa = np.sqrt(cov[0,0]) * Ka
deltaKi = np.sqrt(cov[1,1]) * Ki

# Print results
print("""
The most probable parameters for the MWC model
----------------------------------------------
Ka = {0:.2f} +- {1:0.3f} uM
Ki = {2:.5f} +- {3:0.6f} uM
""".format(Ka, deltaKa, Ki, deltaKi))
```

    The most probable parameters for the MWC model
    ----------------------------------------------
    Ka = 140.38 +- 12.516 uM
    Ki = 0.55964 +- 0.021516 uM

###  Using these parameters to predict other strains.

Let's use these parameters to see how well we can predict the other strains.

```python
# Given this result let's plot all the curves using this parameters.
# Set the colors for the strains
colors = sns.color_palette('colorblind', n_colors=7)
colors[4] = sns.xkcd_palette(['dusty purple'])[0]

df_O2 = df[df.operator=='O2']
plt.figure()
for i, rbs in enumerate(df_O2.rbs.unique()):
    # plot the theory using the parameters from the fit.
    plt.plot(IPTG, fold_change(IPTG * 1E6,
        ea=ea, ei=ei, epsilon=4.5,
        R=df_O2[(df_O2.rbs == rbs)].repressors.unique(),
        epsilon_r=-13.9),
        color=colors[i])
    # compute the mean value for each concentration
    fc_mean = df_O2[(df_O2.rbs==rbs)].groupby('IPTG_uM').fold_change_A.mean()
    # compute the standard error of the mean
    fc_err = df_O2[df_O2.rbs==rbs].groupby('IPTG_uM').fold_change_A.std() / \
    np.sqrt(df_O2[df_O2.rbs==rbs].groupby('IPTG_uM').size())

    # plot the experimental data
    plt.errorbar(df_O2[df_O2.rbs==rbs].IPTG_uM.unique() / 1E6, fc_mean,
        yerr=fc_err,
        fmt='o', label=df_O2[df_O2.rbs==rbs].repressors.unique()[0] * 2,
                 color=colors[i])
plt.xscale('symlog', linthreshx=1E-7)
plt.xlim(left=-5E-9)
plt.xlabel('IPTG (M)')
plt.ylabel('fold-change')
plt.ylim([-0.01, 1.2])
plt.legend(loc='upper left', title='repressors / cell')
plt.tight_layout()
```

# Cross checking the fit with other strains.

An interesting exercise is to perform the fit using the other strains, or pooling all the data together.

To make this in a simple straight forward way let's define a function that takes a `pandas DataFrame` and extracts the independent and dependent variables, performs the regression and returns the MAP and error bar on the parameters $\tilde{k_A}$ and $\tilde{k_I}$.

```python
def non_lin_reg_mwc(df, p0,
                    indep_var=['IPTG_uM', 'repressors', 'binding_energy'],
                    dep_var='fold_change_A', epsilon=4.5, diss_const=False):
    '''
    Performs a non-linear regression on the lacI IPTG titration data assuming
    Gaussian errors with constant variance. Returns the parameters
    e_A == -ln(K_A)
    e_I == -ln(K_I)
    and it's corresponding error bars by approximating the posterior distribution
    as Gaussian.
    Parameters
    ----------
    df : DataFrame.
        DataFrame containing all the titration information. It should at minimum
        contain the IPTG concentration used, the repressor copy number for each
        strain and the binding energy of such strain as the independent variables
        and obviously the gene expression fold-change as the dependent variable.
    p0 : array-like (length = 2).
        Initial guess for the parameter values. The first entry is the guess for
        e_A == -ln(K_A) and the second is the initial guess for e_I == -ln(K_I).
    indep_var : array-like (length = 3).
        Array of length 3 with the name of the DataFrame columns that contain
        the following parameters:
        1) IPTG concentration
        2) repressor copy number
        3) repressor binding energy to the operator
    dep_var : str.
        Name of the DataFrame column containing the gene expression fold-change.
    epsilon : float.
        Value of the allosteric parameter, i.e. the energy difference between
        the active and the inactive state.
    diss_const : bool.
        Indicates if the dissociation constants should be returned instead of
        the e_A and e_I parameteres.
    Returns
    -------
    if diss_const  == True:
        e_A : MAP for the e_A parameter.
        de_A : error bar on the e_A parameter
        e_I : MAP for the e_I parameter.
        de_I : error bar on the e_I parameter
    else:
        K_A : MAP for the K_A parameter.
        dK_A : error bar on the K_A parameter
        K_I : MAP for the K_I parameter.
        dK_I : error bar on the K_I parameter
    '''
    df_indep = df[indep_var]
    df_dep = df[dep_var]

    # Extra arguments given as tuple
    args = (df_indep.values, df_dep.values, epsilon)

    # Compute the MAP
    popt, _ = scipy.optimize.leastsq(resid, p0, args=args)

    # Extract the values
    ea, ei = popt

    # Compute the Hessian at the map
    hes = smnd.approx_hess(popt, log_post,
                           args=(df_indep.values, df_dep.values))

    # Compute the covariance matrix
    cov = -np.linalg.inv(hes)

    if diss_const:
        # Get the values for the dissociation constants and their
        # respective error bars
        Ka = np.exp(-ea)
        Ki = np.exp(-ei)
        deltaKa = np.sqrt(cov[0,0]) * Ka
        deltaKi = np.sqrt(cov[1,1]) * Ki
        return Ka, deltaKa, Ki, deltaKi
    else:
        return ea, cov[0,0], ei, cov[1,1]
```

Now that we have the function, let's systematically perform the regression on each of the strains to check how different the parameter values are.

```python
# initialize a data frame to save the regression parameters
param_df = pd.DataFrame()
# loop through the RBS performing the regression on each strain
for i, rbs in enumerate(df.rbs.unique()):
    param =  pd.Series(non_lin_reg_mwc(df[df.rbs==rbs], p0=[1, 7],
                                       diss_const=True),
                       index=['Ka', 'delta_Ka', 'Ki', 'delta_Ki'])
    param_df = pd.concat([param_df, param], axis=1)
# rename the columns by the rbs name
param_df.columns = df.rbs.unique()

# add the regression on all the pool data
param_df['pool_data'] = pd.Series(non_lin_reg_mwc(df, p0=[-5, 1],
                                       diss_const=True),
                       index=['Ka', 'delta_Ka', 'Ki', 'delta_Ki'])

param_df
```

      RBS1L
      RBS1
      RBS1027
      RBS446
      RBS1147
      HG104
      pool_data

      Ka
      240.402528
      160.056120
      134.562671
      143.114327
      175.695247
      128.673829
      199.482826

      Ki
      0.669918
      0.546850
      0.559323
      0.631284
      0.791484
      0.537988
      0.668853

      delta_Ka
      17.045449
      8.112211
      7.421712
      8.400356
      13.492545
      21.273247
      5.581350

      delta_Ki
      0.032676
      0.019271
      0.018149
      0.018724
      0.024183
      0.026700
      0.009919


# Non-linear Gaussian filtering and smoothing

Provided are two examples of nonlinear state-space models on which one can perform Bayesian filtering and smoothing in order to obtain
a posterior distribution over a latent state trajectory based on noisy observations.
In order to understand the theory behind these methods in detail we refer to [1] and [2].

We provide examples for two different types of state-space model:
1. [Nonlinear, Discrete State-Space Model](#1.-Non-Linear-Discrete-State-Space-Model:-Pendulum): Pendulum
2. [Nonlinear, Continuous-Discrete State-Space Model](#2.-Non-Linear-Continuous-Discrete-State-Space-Model:-Benes-Daum-Filter): Benes-Daum Filter

In order to perform Bayesian Filtering and Smoothing on non-linear models, we show how to use functionalities provided by ProbNum to
either linearise the respective model (Extended Kalman Filter, EKF) or to use an Unscented Transform (Unscented Kalman Filter, UKF) which does not require a Jacobian of the nonlinear model.

**References**:
> [1] Särkkä, Simo, and Solin, Arno. Applied Stochastic Differential Equations. Cambridge University Press, 2019.
>
> [2] Särkkä, Simo. Bayesian Filtering and Smoothing. Cambridge University Press, 2013.

```python
import numpy as np

import probnum as pn
from probnum import filtsmooth, randvars, statespace
from probnum.problems import RegressionProblem
```

```python
np.random.seed(12345)
```

```python
# Make inline plots vector graphics instead of raster graphics
%matplotlib inline
from IPython.display import set_matplotlib_formats
set_matplotlib_formats('pdf', 'svg')

# Plotting
import matplotlib.pyplot as plt
import matplotlib.gridspec as gridspec
plt.style.use('../../probnum.mplstyle')
```

## 1. **Non-Linear Discrete** State-Space Model: Pendulum

---

For nonlinear models, we assume that at least one of the components of our state-space model (i.e. either the dynamics, the measurement model, or both) is a nonlinear function of the latent state.

Consider nonlinear functions $f: \mathbb{R}^d \rightarrow \mathbb{R}^d$ and $h: \mathbb{R}^d \rightarrow \mathbb{R}^m$ where $d$ is the dimension of the latent state and
$m$ is the dimension of the measurements.

For $k = 1, \dots, K$ and $x_0 \sim \mathcal{N}(\mu_0, \Sigma_0)$:

$$
\begin{align}
    \boldsymbol{x}_k &\sim \mathcal{N}(f(\boldsymbol{x}_{k-1}), \boldsymbol{Q}) \\
    \boldsymbol{y}_k &\sim \mathcal{N}(h(\boldsymbol{x}_k), \boldsymbol{R})
\end{align}
$$

This defines a dynamics model that assumes a state $x_k$ in a **discrete** sequence of states arising from a nonlinear map of the previous state $x_{k-1}$ corrupted with additive Gaussian noise under a **process noise** covariance matrix $Q$.
Similarly, the measurements $y_k$ are assumed to be nonlinear transformations of the latent state under additive Gaussian noise according to a **measurement noise** covariance $R$.
In the described case, we cannot use the Kalman Filtering equations since these assume linear relationships. One method is to linearise the nonlinear parts of the model by a first-order Taylor Expansion (Extended Kalman Filter, EKF) or employing the Unscented Kalman Filter (UKF) which is based on the unscented transform and does not require the computation of a Jacobian.
The `probnum` packages provides both methods by wrapping linearizing "Components" around nonlinear dynamcis and/or measurement models, as we will see below.

Note that this can be generalized to a nonlinear time-varying state-space model, as well. Then we would have $f: \mathbb{T} \times \mathbb{R}^d \rightarrow \mathbb{R}^d$ and $h: \mathbb{T} \times \mathbb{R}^d \rightarrow \mathbb{R}^m$ where $\mathbb{T}$ is the "time dimension".

### Define State-Space Model

#### I. Discrete Dynamics Model: Nonlinear Gaussian Transitions

```python
state_dim = 2
observation_dim = 1
```

```python
# approx. gravitational constant
g = 9.81
# dt
delta_t = 0.0075

def pendulum_rhs(state):
    """Right-hand side of an ODE that defines pendulum dynamics"""
    x1, x2 = state
    y1 = x1 + x2 * delta_t
    y2 = x2 - g * np.sin(x1) * delta_t
    return np.array([y1, y2])

def pendulum_jacobian(state):
    """Jacobian of the pendulum ODE"""
    x1, x2 = state
    dy1_dx = [1.0, delta_t]
    dy2_dx = [-g * np.cos(x1) * delta_t, 1.0]
    return np.array([dy1_dx, dy2_dx])

dynamics_transition_function = lambda t, state: pendulum_rhs(state)
dynamics_transition_jacobian_function = lambda t, state: pendulum_jacobian(state)

dynamics_diffusion_matrix = 1.0 * (
    np.diag(np.array([delta_t ** 3 / 3, delta_t]))
    + np.diag(np.array([delta_t ** 2 / 2]), 1)
    + np.diag(np.array([delta_t ** 2 / 2]), -1)
)
```

We here define the non-linear dynamics of a pendulum. Note (in the cell directly above) that `probnum` expects `dynamicsfun` to be a function of time and state. Even though we here consider a nonlinear but time-**in**variant model, we still have to fulfill the contract that the interface requires.
```python
dynamics_transition_function = lambda t, state: pendulum_rhs(state)
```
In other words, we simply ignore the time variable `t` in case of time-invariant models.

To create a discrete, (non-linear) Gaussian dynamics model, `probnum` provides the `DiscreteGaussian` class that takes
- `input_dim` and `output_dim`
- `state_trans_fun` : the function that returns the next state given the current one
- `proc_noise_cov_mat_fun` : the function that returns the covariance matrix of the Gaussian process noise
- `jacob_state_trans_fun` : the function that returns the Jacobian matrix of the state transition function

```python
# Create discrete, non-linear Gaussian dynamics model
dynamics_model = statespace.DiscreteGaussian(
    input_dim=state_dim,
    output_dim=state_dim,
    state_trans_fun=dynamics_transition_function,
    proc_noise_cov_mat_fun=lambda t: dynamics_diffusion_matrix,
    jacob_state_trans_fun=dynamics_transition_jacobian_function,
)
```

#### II. Discrete Measurement Model: Nonlinear Gaussian Measurements

```python
def pendulum_measurement(state):
    x1, x2 = state
    return np.array([np.sin(x1)])

def pendulum_measurement_jacobian(state):
    x1, x2 = state
    return np.array([[np.cos(x1), 0.0]])

measurement_function = lambda t, state: pendulum_measurement(state)
measurement_jacobian_function = lambda t, state: pendulum_measurement_jacobian(state)

measurement_variance = 0.32 ** 2
measurement_covariance = measurement_variance * np.eye(observation_dim)
```

```python
# Create discrete, non-linear Gaussian measurement model
measurement_model = statespace.DiscreteGaussian(
    input_dim=state_dim,
    output_dim=observation_dim,
    state_trans_fun=measurement_function,
    proc_noise_cov_mat_fun=lambda t: measurement_covariance,
    jacob_state_trans_fun=measurement_jacobian_function,
)
```

#### III. Initial State Random Variable

```python
mu_0 = np.ones(state_dim)
sigma_0 = measurement_variance * np.eye(state_dim)
initial_state_rv = randvars.Normal(mean=mu_0, cov=sigma_0)
```

### Generate Data for the State-Space Model

`statespace.generate_samples()` is used to sample both latent states and noisy observations from the specified state space model.

```python
time_grid = np.arange(0., 5., step=delta_t)
```

```python
latent_states, observations = statespace.generate_samples(
    dynmod=dynamics_model,
    measmod=measurement_model,
    initrv=initial_state_rv,
    times=time_grid
)
```

```python
regression_problem = RegressionProblem(
    observations=observations,
    locations=time_grid,
)
```

### Kalman Filtering

#### I. Linearize model (Extended Kalman Filter)

Since we could easily derive the Jacobian of our nonlinear model components above, we can use the Extended Kalman Filter (EKF) to obtain a filtering posterior over the states. Therefore, in `probnum` we just wrap the dynamics model and the measurement model in a `DiscreteEKFComponent` object.
In cases in which the Jacobian is not as easy to write down, `probnum` also provides the `DiscreteUKFComponent` class that implements the linearization by means of an
_unscented Kalman filter_ that does not require a Jacobian.
Alternatively, the Jacobian can be calculated via automatic differentiation.

```python
linearised_dynamics_model = filtsmooth.DiscreteEKFComponent(dynamics_model)
linearised_measurement_model = filtsmooth.DiscreteEKFComponent(measurement_model)
```

#### II. Kalman Filter

Then, we simply pass these linearized models to the Kalman Filter (exactly the same interface as in the linear case) and proceed.

```python
kalman_filter = filtsmooth.Kalman(
    dynamics_model=linearised_dynamics_model,
    measurement_model=linearised_measurement_model,
    initrv=initial_state_rv
)
```

#### III. Perform Kalman Filtering + Rauch-Tung-Striebel Smoothing

```python
state_posterior, _ = kalman_filter.filtsmooth(regression_problem)
```

The method `filtsmooth` returns a `KalmanPosterior` object which provides convenience functions for e.g. sampling and interpolation.
We can also extract the just computed posterior smoothing state variables by querying the `.state_rvs` property.
This yields a list of Gaussian Random Variables from which we can extract the statistics in order to visualize them.

```python
posterior_state_rvs = state_posterior.states          # List of <num_time_points> Normal Random Variables
posterior_state_means = posterior_state_rvs.mean      # Shape: (num_time_points, state_dim)
posterior_state_covs = posterior_state_rvs.cov        # Shape: (num_time_points, state_dim, state_dim)
```

### Visualize Results

```python
state_fig = plt.figure()
state_fig_gs = gridspec.GridSpec(ncols=2, nrows=1, figure=state_fig)

ax_00 = state_fig.add_subplot(state_fig_gs[0, 0])
ax_01 = state_fig.add_subplot(state_fig_gs[0, 1])

# Plot means
mu_x_1, mu_x_2 = [posterior_state_means[:, i] for i in range(state_dim)]

ax_00.plot(time_grid, mu_x_1, label="posterior mean")
ax_01.plot(time_grid, mu_x_2)

# Plot marginal standard deviations
std_x_1, std_x_2 = [np.sqrt(posterior_state_covs[:, i, i]) for i in range(state_dim)]

ax_00.fill_between(
    time_grid,
    mu_x_1 - 1.96 * std_x_1,
    mu_x_1 + 1.96 * std_x_1,
    alpha=0.2,
    label="1.96 marginal stddev",
)
ax_01.fill_between(
    time_grid, mu_x_2 - 1.96 * std_x_2, mu_x_2 + 1.96 * std_x_2, alpha=0.2
)

# Plot 5 samples on a subsampled grid
samples = state_posterior.sample(t=state_posterior.locations[::5], size=5)
for smp in samples:
    ax_00.plot(state_posterior.locations[::5], smp[:, 0], color="gray", alpha=0.75, linewidth=1, linestyle="dashed", label="sample")
    ax_01.plot(state_posterior.locations[::5], smp[:, 1], color="gray",alpha=0.75, linewidth=1, linestyle="dashed", label="sample")

# Plot groundtruth
ax_00.scatter(time_grid, observations, marker=".", label="measurements")

# Add labels etc.
ax_00.set_xlabel("t")
ax_01.set_xlabel("t")

ax_00.set_title(r"$x_1$")
ax_01.set_title(r"$x_2$")

# The following two lines are just to remove duplicate labels (caused by samples) from the legend
handles, labels = ax_00.get_legend_handles_labels()
by_label = dict(zip(labels, handles))

state_fig.legend(
    by_label.values(), by_label.keys(), loc="center left", bbox_to_anchor=(1, 0.5)
)

state_fig.tight_layout()
```

## 2. **Non-Linear Continuous-Discrete** State-Space Model: Benes-Daum Filter

---

Now, consider we assume **continuous** dynamics. We assume that there is a continuous process that defines the nonlinear dynamics of our latent space from which we collect discrete nonlinear-Gaussian measurements (as above). Only the dynamics model changes. In particular, we formulate the dynamics as a stochastic process in terms of a Nonlinear Time-Invariant Stochastic Differential Equation. We refer to [1] for more details.
Again, consider non-linear functions $f: \mathbb{R}^d \rightarrow \mathbb{R}^d$ and $h: \mathbb{R}^d \rightarrow \mathbb{R}^m$ where $d$ is the dimension of the latent state and
$m$ is the dimension of the measurements.
We define the following nonlinear **continuous-discrete** state-space model:

Let $x(t_0) \sim \mathcal{N}(\mu_0, \Sigma_0)$.

$$
\begin{align}
    d\boldsymbol{x} &= f(\boldsymbol{x}) \, dt + \boldsymbol{L} \, d \boldsymbol{\omega} \\
    \boldsymbol{y}_k &\sim \mathcal{N}(h(\boldsymbol{x}(t_k)), \boldsymbol{R}), \qquad k = 1, \dots, K
\end{align}
$$

where $\boldsymbol{\omega}$ denotes a vector of driving forces (often Brownian Motion).

Note that this can be generalized to a nonlinear time-varying state-space model, as well. Then we would have $f: \mathbb{T} \times \mathbb{R}^d \rightarrow \mathbb{R}^d$ and $h: \mathbb{T} \times \mathbb{R}^d \rightarrow \mathbb{R}^m$ where $\mathbb{T}$ is the "time dimension".

### Define State-Space Model

#### I. Continuous Dynamics Model: Nonlinear Stochastic Differential Equation

```python
state_dim = 1
observation_dim = 1
```

```python
drift_function = lambda t, state: np.tanh(state)
drift_function_jacobian = lambda t, state: 1.0 - np.tanh(state) ** 2

dispersion_matrix_function = lambda t: np.ones(state_dim).reshape(1, 1)
```

For continous, non-linear dynamics, `probnum` provides a general stochastic differential equation (SDE) interface that takes
- `driftfun`: a (possibly non-linear) drift function,
- `dispmatfun`: a (possibly non-linear) dispersion function,
- `jacobfun`: the Jacobian of the drift function.

```python
# Create continuous, non-linear SDE dynamics model
dynamics_model = statespace.SDE(
    dimension=state_dim,
    driftfun=drift_function,
    dispmatfun=dispersion_matrix_function,
    jacobfun=drift_function_jacobian,
)
```

#### II. Discrete Measurement Model: Linear, Time-Invariant, Gaussian Measurements

```python
measurement_marginal_variance = 1.0
measurement_matrix = np.eye(observation_dim, state_dim)
measurement_noise_matrix = measurement_marginal_variance * np.eye(observation_dim)
```

```python
measurement_model = statespace.DiscreteLTIGaussian(
    state_trans_mat=measurement_matrix,
    shift_vec=np.zeros(observation_dim),
    proc_noise_cov_mat=measurement_noise_matrix,
)
```

#### III. Initial State Random Variable

```python
mu_0 = np.zeros(state_dim)
sigma_0 = 3.0 * np.eye(state_dim)
initial_state_rv = randvars.Normal(mean=mu_0, cov=sigma_0)
```

### Generate Data for the State-Space Model

`statespace.generate_samples()` is used to sample both latent states and noisy observations from the specified state space model.

```python
time_grid = np.arange(0.0, 10.0, step=10*delta_t)
```

```python
linearised_dynamics_model = filtsmooth.ContinuousEKFComponent(dynamics_model)
```

```python
latent_states, observations = statespace.generate_samples(
    dynmod=linearised_dynamics_model,
    measmod=measurement_model,
    initrv=initial_state_rv,
    times=time_grid,
)
```

```python
regression_problem = RegressionProblem(
    observations=observations,
    locations=time_grid,
)
```

### Kalman Filtering

#### I. Linearize model (Extended Kalman Filter)

#### II. Kalman Filter

```python
kalman_filter = filtsmooth.Kalman(
    dynamics_model=linearised_dynamics_model,
    measurement_model=measurement_model,
    initrv=initial_state_rv,
)
```

#### III. Perform Kalman Filtering + Rauch-Tung-Striebel Smoothing

```python
state_posterior, _ = kalman_filter.filter(regression_problem)
```

```python
# smoothing
smoothed_state_posterior = kalman_filter.smooth(
    filter_posterior=state_posterior
)
```

The method `filtsmooth` returns a `KalmanPosterior` object which provides convenience functions for e.g. sampling and prediction.
We can also extract the just computed posterior smoothing state variables by querying the `.state_rvs` property.
This yields a list of Gaussian Random Variables from which we can extract the statistics in order to visualize them.

```python
grid = np.linspace(0, 11, 500)

posterior_state_rvs = state_posterior(grid)                     # List of <num_time_points> Normal Random Variables
posterior_state_means = posterior_state_rvs.mean.squeeze()      # Shape: (num_time_points, state_dim)
posterior_state_covs = posterior_state_rvs.cov                  # Shape: (num_time_points, state_dim, state_dim)
```

```python
smoothed_posterior_state_rvs = smoothed_state_posterior.states   # List of <num_time_points> Normal Random Variables
smoothed_posterior_state_means = smoothed_posterior_state_rvs.mean.squeeze() # Shape: (num_time_points, state_dim)
smoothed_posterior_state_covs = smoothed_posterior_state_rvs.cov # Shape: (num_time_points, state_dim, state_dim)
```

### Visualize Results

```python
state_fig = plt.figure()

ax = state_fig.add_subplot()

# Plot means
ax.plot(grid, posterior_state_means, label="posterior mean")

# Plot marginal standard deviations
std_x = np.sqrt(np.abs(posterior_state_covs)).squeeze()
ax.fill_between(
    grid,
    posterior_state_means - 1.96 * std_x,
    posterior_state_means + 1.96 * std_x,
    alpha=0.2,
    label="1.96 marginal stddev for filter",
)

# Plot smoothed means
ax.plot(time_grid, smoothed_posterior_state_means, label="smoothed posterior mean")

# Plot marginal standard deviations
std_x = np.sqrt(smoothed_posterior_state_covs).squeeze()
ax.fill_between(
    time_grid,
    smoothed_posterior_state_means - 1.96 * std_x,
    smoothed_posterior_state_means + 1.96 * std_x,
    alpha=0.2,
    label="1.96 marginal stddev for smoother",
)

ax.scatter(time_grid, observations, marker=".", label="measurements")

# Add labels etc.
ax.set_xlabel("t")
ax.set_title(r"$x$")
ax.legend()

state_fig.tight_layout()
```

To get a closer look at filter and smoother, we inspect the interval $[0, 1]$ in greater detail.

```python
idx = time_grid < 1.0

ax.set_xlim([0, 1])
ax.set_ylim(np.min(observations[idx]), np.max(observations[idx]))

state_fig.tight_layout()
state_fig
```

```python

```


# ENGR 1330 Exam 1 Sec 003/004 Fall 2020
Take Home Portion of Exam 1

## Full name
## R#:
## HEX:
## ENGR 1330 Exam 1 Sec 003/004
## Date:

## Question 1 (1 pts):
Run the cell below, and leave the results in your notebook (Windows users may get an error, leave the error in place)

```python
#### RUN! the Cell ####
import sys
! hostname
! whoami
print(sys.executable) # OK if generates an exception message on Windows machines
```

    atomickitty.aws
    compthink
    /opt/conda/envs/python/bin/python

## Question 2 (9 pts):
- __When it is 8:00 in Lubbock,__
    - __It is 9:00 in New York__
    - __It is 14:00 in London__
    - __It is 15:00 in Cairo__
    - __It is 16:00 in Istanbul__
    - __It is 19:00 in Hyderabad__
    - __It is 22:00 in Tokyo__

__Write a function that reports the time in New York, London, Cairo, Istanbul, Hyderabad, and Tokyo based on the time in Lubbock. Use a 24-hour time format. Include error trapping that:__

1- Issues a message like "Please Enter A Number from 00 to 23" if the first input is numeric but outside the range of [0,23].

2- Takes any numeric input for "Lubbock time" selection , and forces it into an integer.

3- Issues an appropriate message if the user's selection is non-numeric.

__Check your function for these times:__
- 8:00
- 17:00
- 0:00

```python
def LBBtime():
    try:
        LBK = int(input('What hour is it in Lubbock?- Please enter a number from 0 to 23'))
        if LBK>23:
            print('Please Enter A Number from 00 to 23')
        if LBK<23 and LBK>=0:
            if LBK+1>23:
                print("Time in New York is",(LBK+1)-24,":00")
            else:
                print("Time in New York is",(LBK+1),":00")
            if LBK+6>23:
                print("Time in London is",(LBK+6)-24,":00")
            else:
                print("Time in London is",(LBK+6),":00")
            if LBK+7>23:
                print("Time in Cairo is",(LBK+7)-24,":00")
            else:
                print("Time in Cairo is",(LBK+7),":00")
            if LBK+8>23:
                print("Time in Istanbul is",(LBK+8)-24,":00")
            else:
                print("Time in Istanbul is",(LBK+8),":00")
            if LBK+11>23:
                print("Time in Hyderabad is",(LBK+11)-24,":00")
            else:
                print("Time in Hyderabad is",(LBK+11),":00")
            if LBK+14>23:
                print("Time in Tokyo is",(LBK+14)-24,":00")
            else:
                print("Time in Tokyo is",(LBK+14),":00")
            return #null return
    except:
            print("Please Enter an Appropriate Input")
```

```python
LBBtime()
```

    What hour is it in Lubbock?- Please enter a number from 0 to 23 8

    Time in New York is 9 :00
    Time in London is 14 :00
    Time in Cairo is 15 :00
    Time in Istanbul is 16 :00
    Time in Hyderabad is 19 :00
    Time in Tokyo is 22 :00

```python
LBBtime()
```

    What hour is it in Lubbock?- Please enter a number from 0 to 23 17

    Time in New York is 18 :00
    Time in London is 23 :00
    Time in Cairo is 0 :00
    Time in Istanbul is 1 :00
    Time in Hyderabad is 4 :00
    Time in Tokyo is 7 :00

```python
LBBtime()
```

    What hour is it in Lubbock?- Please enter a number from 0 to 23 0

    Time in New York is 1 :00
    Time in London is 6 :00
    Time in Cairo is 7 :00
    Time in Istanbul is 8 :00
    Time in Hyderabad is 11 :00
    Time in Tokyo is 14 :00

## Question 3 (28 pts):
Follow the steps below. Add comments to your script and signify when each step and each task is done. *hint: For this problem you will need the numpy and pandas libraries.
- __STEP1: There are 8 digits in your R#. Define a 2x4 array with these 8 digits, name it "Rarray", and print it__
- __STEP2: Find the maximum value of the "Rarray" and its position__
- __STEP3: Sort the "Rarray" along the rows, store it in a new array named "Rarraysort", and print the new array out__
- __STEP4: Define and print a 4x4 array that has the "Rarray" as its two first rows, and "Rarraysort" as its next rows. Name this new array "DoubleRarray"__
- __STEP5: Slice and print a 4x3 array from the "DoubleRarray" that contains the last three columns of it. Name this new array "SliceRarray".__
- __STEP6: Define the "SliceRarray" as a panda dataframe:__
    - name it "Rdataframe",
    - name the rows as "Row A","Row B","Row C", and "Row D"
    - name the columns as "Column 1", "Column 2", and "Column 3"
- __STEP7: Print  the first few rows of the "Rdataframe".__
- __STEP8: Create a new dataframe object ("R2dataframe") by adding a column to the "Rdataframe", name it "Column X" and fill it with "None" values. Then, use the appropriate descriptor function and print the data model (data column count, names, data types) of the "R2dataframe"__
- __STEP9: Replace the **'None'** in the "R2dataframe" with 0. Then, print the summary statistics of each numeric column in the data frame.__
- __STEP10: Define a function based on the equation below, apply on the entire "R2dataframe", store the results in a new dataframe ("R3dataframe"), and print the results and the summary statistics again.__
$$ y = x^2 - 5x +7 $$
- __STEP11: Print the number of occurrences of each unique value in "Column 3"__
- __STEP12: Sort the data frame with respect to "Column 1" with a descending order and print it__
- __STEP13: Write the final format of the "R3dataframe" on a CSV file, named "Rfile.csv"__
- __STEP14: Read the "Rfile.csv" and print its content.__

** __Make sure to attach the "Rfile.csv" file to your midterm exam submission.__

```python
# Code and Run your solution here:

print('#Step0: Install Dependencies')
import numpy as np
import pandas as pd
print('#Step1: Create the array')
Rarray = np.array([[1,6,7,4],[5,2,3,8]])     #Define Rarray
print(Rarray)
print('#Step2: find max and its position ')
print(np.max(Rarray))          #Find the maximum Value
print(np.argmax(Rarray))       #Find the posirtion of the maximum value
print('#Step3: Sort the array')
Rarraysort = np.sort(Rarray,axis = 1) #Sort Rarray along the rows and define a new array
print(Rarraysort)
print('#Step4: Create the double array - manual entry')
DoubleRarray = np.array([[1,6,7,4],[5,2,3,8],[1,4,6,7],[2,3,5,8]])     #Define DoubleRarray
print(DoubleRarray)
print('#Step5: Slice the array')
SliceRarray = DoubleRarray[0:4,1:4]     #Slice DoubleRarray and Define SliceRarray
print(SliceRarray)
print('#Step6: Make a dataframey')
myrowname = ["Row A","Row B","Row C","Row D"]
mycolname = ["Column 1", "Column 2","Column 3"]
Rdataframe = pd.DataFrame(SliceRarray,myrowname,mycolname)     #Define Rdataframe
print('#Step7: head method on dataframe')
print(Rdataframe.head())     #Print the first few rows of the Rdataframe
print('#Step8: add column to a dataframe')
Rdataframe['Column X']= None     #Add a new column, "Column X"
R2dataframe = Rdataframe         #Define R2dataframe
print(R2dataframe.info())               #Get the info
print('#Step9: Replace NA')
R2dataframe = R2dataframe.fillna(0)     #Replace NAs with 0
print(R2dataframe.describe())                 #Get the summary statistics
print('#Step10: Define a function, apply to a dataframe')
def myfunc(x):  # A user-built function
    y = (x**2) - (10*x) +7
    return(y)
R3dataframe = R2dataframe.apply(myfunc)     #Apply the function on the entire R2dataframe
print(R3dataframe)
print(R3dataframe.describe())
print('#Step11: Descriptors')
print(R3dataframe['Column 3'].value_counts())  #Returns the number of occurences of each unique value in Column 3
print('#Step12: Sort on values')
print(R3dataframe.sort_values('Column 1', ascending = False)) #Sorting based on Column 1
print('#Step13: Write to an external file')
R3dataframe.to_csv('Rfile.csv')     #Write R3dataframe on a CSV file
print('#Step14: Verify the write')
readfilecsv = pd.read_csv('Rfile.csv') #Read the Rfile.csv
print(readfilecsv)         #Print the contents of the Rfile.csv
```

    #Step0: Install Dependencies
    #Step1: Create the array
    [[1 6 7 4]
     [5 2 3 8]]
    #Step2: find max and its position
    8
    7
    #Step3: Sort the array
    [[1 4 6 7]
     [2 3 5 8]]
    #Step4: Create the double array - manual entry
    [[1 6 7 4]
     [5 2 3 8]
     [1 4 6 7]
     [2 3 5 8]]
    #Step5: Slice the array
    [[6 7 4]
     [2 3 8]
     [4 6 7]
     [3 5 8]]
    #Step6: Make a dataframey
    #Step7: head method on dataframe
           Column 1  Column 2  Column 3
    Row A         6         7         4
    Row B         2         3         8
    Row C         4         6         7
    Row D         3         5         8
    #Step8: add column to a dataframe
    <class 'pandas.core.frame.DataFrame'>
    Index: 4 entries, Row A to Row D
    Data columns (total 4 columns):
     #   Column    Non-Null Count  Dtype
    ---  ------    --------------  -----
     0   Column 1  4 non-null      int64
     1   Column 2  4 non-null      int64
     2   Column 3  4 non-null      int64
     3   Column X  0 non-null      object
    dtypes: int64(3), object(1)
    memory usage: 160.0+ bytes
    None
    #Step9: Replace NA
           Column 1  Column 2  Column 3  Column X
    count  4.000000  4.000000  4.000000       4.0
    mean   3.750000  5.250000  6.750000       0.0
    std    1.707825  1.707825  1.892969       0.0
    min    2.000000  3.000000  4.000000       0.0
    25%    2.750000  4.500000  6.250000       0.0
    50%    3.500000  5.500000  7.500000       0.0
    75%    4.500000  6.250000  8.000000       0.0
    max    6.000000  7.000000  8.000000       0.0
    #Step10: Define a function, apply to a dataframe
           Column 1  Column 2  Column 3  Column X
    Row A       -17       -14       -17         7
    Row B        -9       -14        -9         7
    Row C       -17       -17       -14         7
    Row D       -14       -18        -9         7
            Column 1   Column 2   Column 3  Column X
    count   4.000000   4.000000   4.000000       4.0
    mean  -14.250000 -15.750000 -12.250000       7.0
    std     3.774917   2.061553   3.947573       0.0
    min   -17.000000 -18.000000 -17.000000       7.0
    25%   -17.000000 -17.250000 -14.750000       7.0
    50%   -15.500000 -15.500000 -11.500000       7.0
    75%   -12.750000 -14.000000  -9.000000       7.0
    max    -9.000000 -14.000000  -9.000000       7.0
    #Step11: Descriptors
    -9     2
    -14    1
    -17    1
    Name: Column 3, dtype: int64
    #Step12: Sort on values
           Column 1  Column 2  Column 3  Column X
    Row B        -9       -14        -9         7
    Row D       -14       -18        -9         7
    Row A       -17       -14       -17         7
    Row C       -17       -17       -14         7
    #Step13: Write to an external file
    #Step14: Verify the write
      Unnamed: 0  Column 1  Column 2  Column 3  Column X
    0      Row A       -17       -14       -17         7
    1      Row B        -9       -14        -9         7
    2      Row C       -17       -17       -14         7
    3      Row D       -14       -18        -9         7

## Problem 4 (32 pts)

Graphing Functions Special Functions

Consider the two functions listed below:

\begin{equation}
f(x) = e^{-\alpha x}
\label{eqn:fofx}
\end{equation}

\begin{equation}
g(x) = \gamma sin(\beta  x)
\label{eqn:gofx}
\end{equation}

Prepare a plot of the two functions on the same graph.

Use the values in Table below for $\alpha$, $\beta$, and $\gamma$.

|Parameter|Value|
|:---|---:|
|$\alpha$|0.50|
|$\beta$|3.00|
|$\gamma$|$\frac{\pi}{2}$|

The plot should have $x$ values ranging from $0$ to $10$ (inclusive) in sufficiently small increments to see curvature in the two functions as well as to identify the number and approximate locations of intersections. In this problem, intersections are locations in the $x-y$ plane where the two "curves" cross one another of the two plots.

#### By-hand evaluate f(x) for x=1, alpha = 1/2 (Simply enter your answer from a calculator)
f(x) = 0.61

#### By-hand evaluate g(x) for x=3.14, beta = 1/2, gamma = 2 (Simply enter your answer from a calculator)
g(x) = 1.99

```python
# Define the first function f(x,alpha), test the function using your by hand answer
# Define the first function f(x,alpha), test the function using your by hand answer
def f(x,alpha):
    import math
    f = math.exp(-1.0*alpha*x)
    return f

f(1,0.5)
```

    0.6065306597126334

```python
# Define the second function g(x,beta,gamma), test the function using your by hand answer
def g(x,beta,gamma):
    import math
    g = gamma*math.sin(beta*x)
    return g

g(3.14,0.5,2.0)
```

    1.9999993658636692

```python
# Built a list for x that ranges from 0 to 10, inclusive, with adjustable step sizes for plotting later on
howMany = 100
scale = 10.0/howMany
xvector = []
for i in range(0,howMany+1):
    xvector.append(scale*i)
#xvector # activate to display
```

```python
# xvector
```

```python
# Build a plotting function that plots both functions on the same chart
# Build a plotting function that plots both functions on the same chart
import mplcursors
alpha = 0.5
beta = 3.0
gamma = 1.57
yf = []
yg = []
for i in range(0,howMany+1):
    yf.append(f(xvector[i],alpha))
    yg.append(g(xvector[i],beta,gamma))

def plot2lines(list11,list21,list12,list22,strx,stry,strtitle): # plot list1 on x, list2 on y, xlabel, ylabel, title
    from matplotlib import pyplot as plt # import the plotting library from matplotlibplt.show()
    plt.plot( list11, list21, color ='green', marker ='s', linestyle ='dashdot' , label = "Observed" ) # create a line chart, years on x-axis, gdp on y-axis
    plt.plot( list12, list22, color ='red', marker ='o', linestyle ='solid' , label = "Model") # create a line chart, years on x-axis, gdp on y-axis
    plt.legend()
    plt.title(strtitle)# add a title
    plt.ylabel(stry)# add a label to the x and y-axes
    plt.xlabel(strx)
    mplcursors.cursor()
    plt.show() # display the plot
    return #null return

plot2lines(xvector,yf,xvector,yg,'x-value','y-value','plot of f and g')
```

```python

```

```python

```

```python
# Using the plot as a guide, find the approximate values of x where the two curves intercept (i.e. f(x) = g(x))
# You can either use interactive input, or direct specify x values, but need to show results
# Using the plot as a guide, find the values of x where the two curves intercept (i.e. f(x) = g(x))

#xguess = float(input('my guess for x')) # ~0.7, and 6.25
alpha = 0.5
beta = 0.5
gamma = 2.0
xguess = 0.72
result = f(xguess,alpha) - g(xguess,beta,gamma)
print('f(x) - g(x) =', result,' at x = ', xguess)
xguess = 6.245
result = f(xguess,alpha) - g(xguess,beta,gamma)
print('f(x) - g(x) =', result,' at x = ', xguess)
```

    f(x) - g(x) = -0.006872140479148858  at x =  0.72
    f(x) - g(x) = 0.005863926097657193  at x =  6.245

## Bonus Problem 1. Extra Credit (You must complete the regular problems)!
__create a class to compute the average grade (out of 10) of the students based on their grades in Quiz1, Quiz2, the Mid-term, Quiz3, and the Final exam.__

| Student Name  |   Quiz 1   |   Quiz 2   |   Mid-term   |   Quiz 3   |  Final Exam  |
| ------------- | -----------| -----------| -------------| -----------| -------------|
| Harry         | 8          | 9          | 8            | 10         | 9            |
| Ron           | 7          | 8          | 8            | 7          | 9            |
| Hermione      | 10         | 10         | 9            | 10         | 10           |
| Draco         | 8          | 7          | 9            | 8          | 9            |
| Luna          | 9          | 8          | 7            | 6          | 5            |

1. __Use docstrings to describe the purpose of the class.__
2. __Create an object for each car brand and display the output as shown below.__

"Student Name": **Average Grade**

3. __Create and print out a dictionary with the student names as keys and their average grades as data.__

```python
#Code and run your solution here:

#Suggested Solution:

class Hogwarts:
    """This class calculates the average grade of the students"""
    def __init__(self, Name,Quiz1,Quiz2,MidTerm,Quiz3,Final):
        self.Name = Name
        self.Quiz1 = Quiz1
        self.Quiz2 = Quiz2
        self.MidTerm = MidTerm
        self.Quiz3 = Quiz3
        self.Final= Final

    def average(self):
        return (self.Quiz1 + self.Quiz2 + self.MidTerm + self.Quiz3 + self.Final) /5

S1 = Hogwarts('Harry',8,9,8,10,9)                      #Fill the instances
S2 = Hogwarts('Ron',7,8,8,7,9)
S3 = Hogwarts('Hermione',10,10,9,10,10)
S4 = Hogwarts('Draco',8,7,9,8,9)
S5 = Hogwarts('Luna',9,8,7,6,5)

print("Harry", S1.average())
print("Ron", S2.average())
print("Hermione", S3.average())
print("Draco", S4.average())
print("Luna", S5.average())

GradeDict = {"Harry":S1.average(),"Ron":S2.average(),"Hermione":S3.average(),"Draco":S4.average(),"Luna":S5.average()}
print(GradeDict)
```

    Harry 8.8
    Ron 7.8
    Hermione 9.8
    Draco 8.2
    Luna 7.0
    {'Harry': 8.8, 'Ron': 7.8, 'Hermione': 9.8, 'Draco': 8.2, 'Luna': 7.0}

## Bonus 2 Extra credit (You must complete the regular problems)!
#### Write the VOLUME Function to compute the volume of Cylinders, Spheres, Cones, and Rectangular Boxes. This function should:
- First, ask the user about __the shape of the object__ of interest using this statement:

**"Please choose the shape of the object. Enter 1 for "Cylinder", 2 for "Sphere", 3 for "Cone", or 4 for "Rectangular Box""**

- Second, based on user's choice in the previous step, __ask for the right inputs__.
- Third, print out an statement with __the input values and the calculated volumes__.

#### Include error trapping that:

1. Issues a message that **"The object should be either a Cylinder, a Sphere, a Cone, or a Rectangular Box. Please Enter A Number from 1,2,3, and 4!"** if the first input is non-numeric.
2. Takes any numeric input for the initial selection , and force it into an integer.
4. Issues an appropriate message if the user's selection is numeric but outside the range of [1,4]
3. Takes any numeric input for the shape characteristics , and force it into a float.
4. Issues an appropriate message if the object characteristics are as non-numerics.

#### Test the script for:
1. __Sphere, r=10__
2. __r=10 , Sphere__
3. __Rectangular Box, w=5, h=10, l=0.5__

- __Volume of a Cylinder = πr²h__
- __Volume of a Sphere = 4(πr3)/3__
- __Volume of a Cone = (πr²h)/3__
- __Volume of a Rectangular Box = whl__

```python
#Code and Run your solution here

#Suggested Solution:

import numpy as np # import NumPy: for large, multi-dimensional arrays and matrices, along with  high-level mathematical functions to operate on these arrays.
pi = np.pi #pi value from the np package

def VOLUME():
    try:
        UI = input('Please choose the shape of the object. Enter 1 for "Cylinder", 2 for "Sphere", 3 for "Cone", or 4 for "Rectangular Box"')
        UI =int(UI)
        if UI==1:
                try:
                    UI2 = input('Please enter the radius of the Cylinder')
                    r= float(UI2)
                    UI3 = input('Please enter the height of the Cylinder')
                    h= float(UI3)
                    V= pi*h*r**2
                    print("The volume of the Cylinder with the radius of ",r," and the height of ",h," is equal to", V)
                except:
                    print("The radius and height of the Cylinder must be numerics. Please Try Again!")
        elif UI==2:
                try:
                    UI2 = input('Please enter the radius of the Sphere')
                    r= float(UI2)
                    V= (4*pi*r**3)/3
                    print("The volume of the Sphere with the radius of ",r," is equal to", V)
                except:
                    print("The radius of the Sphere must be numeric. Please Try Again!")
        elif UI==3:
                try:
                    UI2 = input('Please enter the radius of the Cone')
                    r= float(UI2)
                    UI3 = input('Please enter the height of the Cone')
                    h= float(UI3)
                    V= (pi*h*r**2)/3
                    print("The volume of the Cone with the radius of ",r," and the height of ",h," is equal to", V)
                except:
                    print("The radius and height of the Cone must be numerics. Please Try Again!")
        elif UI==4:
                try:
                    UI2 = input('Please enter the width of the Rectangular Box')
                    w= float(UI2)
                    UI3 = input('Please enter the height of the Rectangular Box')
                    h= float(UI3)
                    UI4 = input('Please enter the length of the Rectangular Box')
                    l= float(UI4)
                    V= w*h*l
                    print("The volume of the Rectangular Box with the width of ",w," and the height of ",h," and the length of ",l," is equal to", V)
                except:
                    print("The width, height, and length of the Rectangular Box must be numerics. Please Try Again!")
        else:
            print("Please Enter A Number from 1,2,3, and 4!")

    except:
        print("The object should be either a Cylinder, a Sphere, a Cone, or a Rectangular Box. Please Enter A Number from 1,2,3, and 4!")

```

```python
VOLUME()
```

    Please choose the shape of the object. Enter 1 for "Cylinder", 2 for "Sphere", 3 for "Cone", or 4 for "Rectangular Box" 1
    Please enter the radius of the Cylinder 1
    Please enter the height of the Cylinder 1

    The volume of the Cylinder with the radius of  1.0  and the height of  1.0  is equal to 3.141592653589793


###### Content under Creative Commons Attribution license CC-BY 4.0, code under MIT license (c)2014 L.A. Barba, G.F. Forsyth, C. Cooper. Based on [CFDPython](https://github.com/barbagroup/CFDPython), (c)2013 L.A. Barba, also under CC-BY license.

# Space & Time

## Burgers' Equation

Hi there! We have reached the final lesson of the series *Space and Time — Introduction to Finite-difference solutions of PDEs*, the second module of ["Practical Numerical Methods with Python"](https://openedx.seas.gwu.edu/courses/course-v1:MAE+MAE6286+2017/about).

We have learned about the finite-difference solution for the linear and non-linear convection equations and the diffusion equation. It's time to combine all these into one: *Burgers' equation*. The wonders of *code reuse*!

Before you continue, make sure you have completed the previous lessons of this series, it will make your life easier. You should have written your own versions of the codes in separate, clean Jupyter Notebooks or Python scripts.

You can read about Burgers' Equation on its [wikipedia page](http://en.wikipedia.org/wiki/Burgers'_equation).
Burgers' equation in one spatial dimension looks like this:

$$
\begin{equation}
\frac{\partial u}{\partial t} + u \frac{\partial u}{\partial x} = \nu \frac{\partial ^2u}{\partial x^2}
\end{equation}
$$

As you can see, it is a combination of non-linear convection and diffusion. It is surprising how much you learn from this neat little equation!

We can discretize it using the methods we've already detailed in the previous notebooks of this module.  Using forward difference for time, backward difference for space and our 2nd-order method for the second derivatives yields:

$$
\begin{equation}
\frac{u_i^{n+1}-u_i^n}{\Delta t} + u_i^n \frac{u_i^n - u_{i-1}^n}{\Delta x} = \nu \frac{u_{i+1}^n - 2u_i^n + u_{i-1}^n}{\Delta x^2}
\end{equation}
$$

As before, once we have an initial condition, the only unknown is $u_i^{n+1}$. We will step in time as follows:

$$
\begin{equation}
u_i^{n+1} = u_i^n - u_i^n \frac{\Delta t}{\Delta x} (u_i^n - u_{i-1}^n) + \nu \frac{\Delta t}{\Delta x^2}(u_{i+1}^n - 2u_i^n + u_{i-1}^n)
\end{equation}
$$

### Initial and Boundary Conditions

To examine some interesting properties of Burgers' equation, it is helpful to use different initial and boundary conditions than we've been using for previous steps.

The initial condition for this problem is going to be:

$$
\begin{eqnarray}
u &=& -\frac{2 \nu}{\phi} \frac{\partial \phi}{\partial x} + 4 \\\
\phi(t=0) = \phi_0 &=& \exp \bigg(\frac{-x^2}{4 \nu} \bigg) + \exp \bigg(\frac{-(x-2 \pi)^2}{4 \nu} \bigg)
\end{eqnarray}
$$

This has an analytical solution, given by:

$$
\begin{eqnarray}
u &=& -\frac{2 \nu}{\phi} \frac{\partial \phi}{\partial x} + 4 \\\
\phi &=& \exp \bigg(\frac{-(x-4t)^2}{4 \nu (t+1)} \bigg) + \exp \bigg(\frac{-(x-4t -2 \pi)^2}{4 \nu(t+1)} \bigg)
\end{eqnarray}
$$

The boundary condition will be:

$$
\begin{equation}
u(0) = u(2\pi)
\end{equation}
$$

This is called a *periodic* boundary condition. Pay attention! This will cause you a bit of headache if you don't tread carefully.

### Saving Time with SymPy

The initial condition we're using for Burgers' Equation can be a bit of a pain to evaluate by hand.  The derivative $\frac{\partial \phi}{\partial x}$ isn't too terribly difficult, but it would be easy to drop a sign or forget a factor of $x$ somewhere, so we're going to use SymPy to help us out.

[SymPy](http://sympy.org/en/) is the symbolic math library for Python.  It has a lot of the same symbolic math functionality as Mathematica with the added benefit that we can easily translate its results back into our Python calculations (it is also free and open source).

Start by loading the SymPy library, together with our favorite library, NumPy.

```python
import numpy
import sympy
from matplotlib import pyplot
%matplotlib inline
```

```python
# Set the font family and size to use for Matplotlib figures.
pyplot.rcParams['font.family'] = 'serif'
pyplot.rcParams['font.size'] = 16
```

We're also going to tell SymPy that we want all of its output to be rendered using $\LaTeX$. This will make our Notebook beautiful!

```python
sympy.init_printing()
```

Start by setting up symbolic variables for the three variables in our initial condition.  It's important to recognize that once we've defined these symbolic variables, they function differently than "regular" Python variables.

If we type `x` into a code block, we'll get an error:

```python
x
```

`x` is not defined, so this shouldn't be a surprise.  Now, let's set up `x` as a *symbolic* variable:

```python
x = sympy.symbols('x')
```

Now let's see what happens when we type `x` into a code cell:

```python
x
```

The value of `x` is $x$.  Sympy is also referred to as a computer algebra system -- normally the value of `5*x` will return the product of `5` and whatever value `x` is pointing to.  But, if we define `x` as a symbol, then something else happens:

```python
5 * x
```

This will let us manipulate an equation with unknowns using Python!  Let's start by defining symbols for $x$, $\nu$ and $t$ and then type out the full equation for $\phi$.  We should get a nicely rendered version of our $\phi$ equation.

```python
x, nu, t = sympy.symbols('x nu t')
phi = (sympy.exp(-(x - 4 * t)**2 / (4 * nu * (t + 1))) +
       sympy.exp(-(x - 4 * t - 2 * sympy.pi)**2 / (4 * nu * (t + 1))))
phi
```

It's maybe a little small, but that looks right.  Now to evaluate our partial derivative $\frac{\partial \phi}{\partial x}$ is a trivial task.  To take a derivative with respect to $x$, we can just use:

```python
phiprime = phi.diff(x)
phiprime
```

If you want to see the non-rendered version, just use the Python print command.

```python
print(phiprime)
```

    -(-8*t + 2*x)*exp(-(-4*t + x)**2/(4*nu*(t + 1)))/(4*nu*(t + 1)) - (-8*t + 2*x - 4*pi)*exp(-(-4*t + x - 2*pi)**2/(4*nu*(t + 1)))/(4*nu*(t + 1))

### Now what?

Now that we have the Pythonic version of our derivative, we can finish writing out the full initial condition equation and then translate it into a usable Python expression.  For this, we'll use the *lambdify* function, which takes a SymPy symbolic equation and turns it into a callable function.

```python
from sympy.utilities.lambdify import lambdify

u = -2 * nu * (phiprime / phi) + 4
print(u)
```

    -2*nu*(-(-8*t + 2*x)*exp(-(-4*t + x)**2/(4*nu*(t + 1)))/(4*nu*(t + 1)) - (-8*t + 2*x - 4*pi)*exp(-(-4*t + x - 2*pi)**2/(4*nu*(t + 1)))/(4*nu*(t + 1)))/(exp(-(-4*t + x - 2*pi)**2/(4*nu*(t + 1))) + exp(-(-4*t + x)**2/(4*nu*(t + 1)))) + 4

### Lambdify

To lambdify this expression into a usable function, we tell lambdify which variables to request and the function we want to plug them into.

```python
u_lamb = lambdify((t, x, nu), u)
print('The value of u at t=1, x=4, nu=3 is {}'.format(u_lamb(1, 4, 3)))
```

    The value of u at t=1, x=4, nu=3 is 3.49170664206445

### Back to Burgers' Equation

Now that we have the initial conditions set up, we can proceed and finish setting up the problem.  We can generate the plot of the initial condition using our lambdify-ed function.

```python
# Set parameters.
nx = 101  # number of spatial grid points
L = 2.0 * numpy.pi  # length of the domain
dx = L / (nx - 1)  # spatial grid size
nu = 0.07  # viscosity
nt = 100  # number of time steps to compute
sigma = 0.1  # CFL limit
dt = sigma * dx**2 / nu  # time-step size

# Discretize the domain.
x = numpy.linspace(0.0, L, num=nx)
```

We have a function `u_lamb` but we need to create an array `u0` with our initial conditions.  `u_lamb` will return the value for any given time $t$, position $x$ and $nu$.  We can use a `for`-loop to cycle through values of `x` to generate the `u0` array.  That code would look something like this:

```Python
u0 = numpy.empty(nx)

for i, x0 in enumerate(x):
    u0[i] = u_lamb(t, x0, nu)
```

But there's a cleaner, more beautiful way to do this -- *list comprehension*.

We can create a list of all of the appropriate `u` values by typing

```Python
[u_lamb(t, x0, nu) for x0 in x]
```

You can see that the syntax is similar to the `for`-loop, but it only takes one line.  Using a list comprehension will create... a list.  This is different from an *array*, but converting a list to an array is trivial using `numpy.asarray()`.

With the list comprehension in place, the three lines of code above become one:

```Python
u = numpy.asarray([u_lamb(t, x0, nu) for x0 in x])
```

```python
# Set initial conditions.
t = 0.0
u0 = numpy.array([u_lamb(t, xi, nu) for xi in x])
u0
```

    array([4.        , 4.06283185, 4.12566371, 4.18849556, 4.25132741,
           4.31415927, 4.37699112, 4.43982297, 4.50265482, 4.56548668,
           4.62831853, 4.69115038, 4.75398224, 4.81681409, 4.87964594,
           4.9424778 , 5.00530965, 5.0681415 , 5.13097336, 5.19380521,
           5.25663706, 5.31946891, 5.38230077, 5.44513262, 5.50796447,
           5.57079633, 5.63362818, 5.69646003, 5.75929189, 5.82212374,
           5.88495559, 5.94778745, 6.0106193 , 6.07345115, 6.136283  ,
           6.19911486, 6.26194671, 6.32477856, 6.38761042, 6.45044227,
           6.51327412, 6.57610598, 6.63893783, 6.70176967, 6.76460125,
           6.82742866, 6.89018589, 6.95176632, 6.99367964, 6.72527549,
           4.        , 1.27472451, 1.00632036, 1.04823368, 1.10981411,
           1.17257134, 1.23539875, 1.29823033, 1.36106217, 1.42389402,
           1.48672588, 1.54955773, 1.61238958, 1.67522144, 1.73805329,
           1.80088514, 1.863717  , 1.92654885, 1.9893807 , 2.05221255,
           2.11504441, 2.17787626, 2.24070811, 2.30353997, 2.36637182,
           2.42920367, 2.49203553, 2.55486738, 2.61769923, 2.68053109,
           2.74336294, 2.80619479, 2.86902664, 2.9318585 , 2.99469035,
           3.0575222 , 3.12035406, 3.18318591, 3.24601776, 3.30884962,
           3.37168147, 3.43451332, 3.49734518, 3.56017703, 3.62300888,
           3.68584073, 3.74867259, 3.81150444, 3.87433629, 3.93716815,
           4.        ])

Now that we have the initial conditions set up, we can plot it to see what $u(x,0)$ looks like:

```python
# Plot the initial conditions.
pyplot.figure(figsize=(6.0, 4.0))
pyplot.title('Initial conditions')
pyplot.xlabel('x')
pyplot.ylabel('u')
pyplot.grid()
pyplot.plot(x, u0, color='C0', linestyle='-', linewidth=2)
pyplot.xlim(0.0, L)
pyplot.ylim(0.0, 10.0);
```

This is definitely not the hat function we've been dealing with until now. We call it a "saw-tooth function".  Let's proceed forward and see what happens.

### Periodic Boundary Conditions

We will implement Burgers' equation with *periodic* boundary conditions.  If you experiment with the linear and non-linear convection notebooks and make the simulation run longer (by increasing `nt`) you will notice that the wave will keep moving to the right until it no longer even shows up in the plot.

With periodic boundary conditions, when a point gets to the right-hand side of the frame, it *wraps around* back to the front of the frame.

Recall the discretization that we worked out at the beginning of this notebook:

$$
\begin{equation}
u_i^{n+1} = u_i^n - u_i^n \frac{\Delta t}{\Delta x} (u_i^n - u_{i-1}^n) + \nu \frac{\Delta t}{\Delta x^2}(u_{i+1}^n - 2u_i^n + u_{i-1}^n)
\end{equation}
$$

What does $u_{i+1}^n$ *mean* when $i$ is already at the end of the frame?

Think about this for a minute before proceeding.

```python
# Integrate the Burgers' equation in time.
u = u0.copy()
for n in range(nt):
    un = u.copy()
    # Update all interior points.
    u[1:-1] = (un[1:-1] -
               un[1:-1] * dt / dx * (un[1:-1] - un[:-2]) +
               nu * dt / dx**2 * (un[2:] - 2 * un[1:-1] + un[:-2]))
    # Update boundary points.
    u[0] = (un[0] -
            un[0] * dt / dx * (un[0] - un[-1]) +
            nu * dt / dx**2 * (un[1] - 2 * un[0] + un[-1]))
    u[-1] = (un[-1] -
            un[-1] * dt / dx * (un[-1] - un[-2]) +
            nu * dt / dx**2 * (un[0] - 2 * un[-1] + un[-2]))
```

```python
# Compute the analytical solution.
u_analytical = numpy.array([u_lamb(nt * dt, xi, nu) for xi in x])
```

```python
# Plot the numerical solution along with the analytical solution.
pyplot.figure(figsize=(6.0, 4.0))
pyplot.xlabel('x')
pyplot.ylabel('u')
pyplot.grid()
pyplot.plot(x, u, label='Numerical',
            color='C0', linestyle='-', linewidth=2)
pyplot.plot(x, u_analytical, label='Analytical',
            color='C1', linestyle='--', linewidth=2)
pyplot.legend()
pyplot.xlim(0.0, L)
pyplot.ylim(0.0, 10.0);
```

Let's now create an animation with the `animation` module of Matplotlib to observe how the numerical solution changes over time compared to the analytical solution.
We start by importing the module from Matplotlib as well as the special `HTML` display method.

```python
from matplotlib import animation
from IPython.display import HTML
```

We create a function `burgers` to computes the numerical solution of the 1D Burgers' equation over time.
(The function returns the history of the solution: a list with `nt` elements, each one being the solution in the domain at a time step.)

```python
def burgers(u0, dx, dt, nu, nt=20):
    """
    Computes the numerical solution of the 1D Burgers' equation
    over the time steps.

    Parameters
    ----------
    u0 : numpy.ndarray
        The initial conditions as a 1D array of floats.
    dx : float
        The grid spacing.
    dt : float
        The time-step size.
    nu : float
        The viscosity.
    nt : integer, optional
        The number of time steps to compute;
        default: 20.

    Returns
    -------
    u_hist : list of numpy.ndarray objects
        The history of the numerical solution.
    """
    u_hist = [u0.copy()]
    u = u0.copy()
    for n in range(nt):
        un = u.copy()
        # Update all interior points.
        u[1:-1] = (un[1:-1] -
                   un[1:-1] * dt / dx * (un[1:-1] - un[:-2]) +
                   nu * dt / dx**2 * (un[2:] - 2 * un[1:-1] + un[:-2]))
        # Update boundary points.
        u[0] = (un[0] -
                un[0] * dt / dx * (un[0] - un[-1]) +
                nu * dt / dx**2 * (un[1] - 2 * un[0] + un[-1]))
        u[-1] = (un[-1] -
                un[-1] * dt / dx * (un[-1] - un[-2]) +
                nu * dt / dx**2 * (un[0] - 2 * un[-1] + un[-2]))
        u_hist.append(u.copy())
    return u_hist
```

```python
# Compute the history of the numerical solution.
u_hist = burgers(u0, dx, dt, nu, nt=nt)
```

```python
# Compute the history of the analytical solution.
u_analytical = [numpy.array([u_lamb(n * dt, xi, nu) for xi in x])
                for n in range(nt)]
```

```python
fig = pyplot.figure(figsize=(6.0, 4.0))
pyplot.xlabel('x')
pyplot.ylabel('u')
pyplot.grid()
u0_analytical = numpy.array([u_lamb(0.0, xi, nu) for xi in x])
line1 = pyplot.plot(x, u0, label='Numerical',
                    color='C0', linestyle='-', linewidth=2)[0]
line2 = pyplot.plot(x, u0_analytical, label='Analytical',
                    color='C1', linestyle='--', linewidth=2)[0]
pyplot.legend()
pyplot.xlim(0.0, L)
pyplot.ylim(0.0, 10.0)
fig.tight_layout()
```

```python
def update_plot(n, u_hist, u_analytical):
    """
    Update the lines y-data of the Matplotlib figure.

    Parameters
    ----------
    n : integer
        The time-step index.
    u_hist : list of numpy.ndarray objects
        The history of the numerical solution.
    u_analytical : list of numpy.ndarray objects
        The history of the analytical solution.
    """
    fig.suptitle('Time step {:0>2}'.format(n))
    line1.set_ydata(u_hist[n])
    line2.set_ydata(u_analytical[n])
```

```python
# Create an animation.
anim = animation.FuncAnimation(fig, update_plot,
                               frames=nt, fargs=(u_hist, u_analytical),
                               interval=100)
```

```python
# Display the video.
HTML(anim.to_html5_video())
```

## Array Operation Speed Increase

Coding up discretization schemes using array operations can be a bit of a pain.  It requires much more mental effort on the front-end than using two nested `for` loops.  So why do we do it?  Because it's fast.  Very, very fast.

Here's what the Burgers code looks like using two nested `for` loops.  It's easier to write out, plus we only have to add one "special" condition to implement the periodic boundaries.

At the top of the cell, you'll see the decorator `%%timeit`.
This is called a "cell magic".  It runs the cell several times and returns the average execution time for the contained code.

Let's see how long the nested `for` loops take to finish.

```python
%%timeit
# Set initial conditions.
u = numpy.array([u_lamb(t, x0, nu) for x0 in x])
# Integrate in time using a nested for loop.
for n in range(nt):
    un = u.copy()
    # Update all interior points and the left boundary point.
    for i in range(nx - 1):
        u[i] = (un[i] -
                un[i] * dt / dx *(un[i] - un[i - 1]) +
                nu * dt / dx**2 * (un[i + 1] - 2 * un[i] + un[i - 1]))
    # Update the right boundary.
    u[-1] = (un[-1] -
             un[-1] * dt / dx * (un[-1] - un[-2]) +
             nu * dt / dx**2 * (un[0]- 2 * un[-1] + un[-2]))
```

    27 ms ± 740 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

Less than 50 milliseconds.  Not bad, really.

Now let's look at the array operations code cell.  Notice that we haven't changed anything, except we've added the `%%timeit` magic and we're also resetting the array `u` to its initial conditions.

This takes longer to code and we have to add two special conditions to take care of the periodic boundaries.  Was it worth it?

```python
%%timeit
# Set initial conditions.
u = numpy.array([u_lamb(t, xi, nu) for xi in x])
# Integrate in time using array operations.
for n in range(nt):
    un = u.copy()
    # Update all interior points.
    u[1:-1] = (un[1:-1] -
               un[1:-1] * dt / dx * (un[1:-1] - un[:-2]) +
               nu * dt / dx**2 * (un[2:] - 2 * un[1:-1] + un[:-2]))
    # Update boundary points.
    u[0] = (un[0] -
            un[0] * dt / dx * (un[0] - un[-1]) +
            nu * dt / dx**2 * (un[1] - 2 * un[0] + un[-1]))
    u[-1] = (un[-1] -
            un[-1] * dt / dx * (un[-1] - un[-2]) +
            nu * dt / dx**2 * (un[0] - 2 * un[-1] + un[-2]))
```

    4.29 ms ± 182 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Yes, it is absolutely worth it.  That's a nine-fold speed increase.  For this exercise, you probably won't miss the extra 40 milliseconds if you use the nested `for` loops, but what about a simulation that has to run through millions and millions of iterations?  Then that little extra effort at the beginning will definitely pay off.

---

###### The cell below loads the style of the notebook.

```python
from IPython.core.display import HTML
css_file = '../../styles/numericalmoocstyle.css'
HTML(open(css_file, 'r').read())
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Nixie+One' rel='stylesheet' type='text/css'>
<link href='https://fonts.googleapis.com/css?family=Source+Code+Pro' rel='stylesheet' type='text/css'>
<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: rgb(245,245,245);
}

div.cell { /* set cell width */
    width: 750px;
}

div #notebook { /* centre the content */
    background: #fff; /* white background for content */
    width: 1000px;
    margin: auto;
    padding-left: 0em;
}

#notebook li { /* More space between bullet points */
    margin-top:0.8em;
}

/* draw border around running cells */
div.cell.border-box-sizing.code_cell.running {
    border: 1px solid #111;
}

/* Put a solid color box around each cell and its output, visually linking them*/
div.cell.code_cell {
    background-color: rgb(256,256,256);
    border-radius: 0px;
    padding: 0.5em;
    margin-left:1em;
    margin-top: 1em;
}

div.text_cell_render{
    font-family: 'Alegreya Sans' sans-serif;
    line-height: 140%;
    font-size: 125%;
    font-weight: 400;
    width:600px;
    margin-left:auto;
    margin-right:auto;
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Nixie One', serif;
    font-style:regular;
    font-weight: 400;
    font-size: 45pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}

.text_cell_render h2 {
    font-family: 'Nixie One', serif;
    font-weight: 400;
    font-size: 30pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.1em;
    margin-top: 0.3em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Nixie One', serif;
    margin-top:16px;
    font-size: 22pt;
    font-weight: 600;
    margin-bottom: 3px;
    font-style: regular;
    color: rgb(102,102,0);
}

.text_cell_render h4 {    /*Use this for captions*/
    font-family: 'Nixie One', serif;
    font-size: 14pt;
    text-align: center;
    margin-top: 0em;
    margin-bottom: 2em;
    font-style: regular;
}

.text_cell_render h5 {  /*Use this for small titles*/
    font-family: 'Nixie One', sans-serif;
    font-weight: 400;
    font-size: 16pt;
    color: rgb(163,0,0);
    font-style: italic;
    margin-bottom: .1em;
    margin-top: 0.8em;
    display: block;
}

.text_cell_render h6 { /*use this for copyright note*/
    font-family: 'PT Mono', sans-serif;
    font-weight: 300;
    font-size: 9pt;
    line-height: 100%;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

.CodeMirror{
    font-family: "Source Code Pro";
    font-size: 90%;
}

.alert-box {
    padding:10px 10px 10px 36px;
    margin:5px;
}

.success {
    color:#666600;
    background:rgb(240,242,229);
}
</style>


---
## 01. Linear Regression and Logistic Regression

Eduard Larrañaga (ealarranaga@unal.edu.co)

---

### About this notebook

In this worksheet, we will use the `scikit-learn` package to make a linear fit of Hubble's experimental data.

---

### Linear Regression vs Logistic Regression

**Linear Regression** is one of the most simple *supervised regression machine learning algorithm*. In general, all regression models a target prediction value based on independent variables and they are mostly used for finding out the relationship between variables and forecasting.

Different regression models differ based on the kind of relationship between the dependent and independent variables they are considering and in the number of independent variables they consider (Multiple Linear Regression). It  based on the minimization of a cost (error) function, as for example the root mena squared error (minimum squares fit).

The equation obtained form a multiple linear regression has the form

\begin{equation}
y = a_0 + a_1 x_1 + a_2 x_2 + ... + a_n x_n
\end{equation}

---

**Logistic Regression** is a *supervised classification machine learningn algorithm*. In a classification problem, the target variable (output) can take only discrete values for a given set of features (inputs). For example, logistic regression may be used to predict a target with only two possible values: 0 and 1 or `True` and `False`.

Logistic regression is based on the concept of *Maximum Likelihood estimation*. When working with numerical values in logistic regression, we pass the weighted sum of inputs through an activation function that can map values in between 0 and 1. Such activation function is known as **sigmoid function** and the curve obtained is called as sigmoid curve or S-curve.

The relation obtained from a a logistic regression is

\begin{equation}
\log \left[ \frac{y}{1-y}\right] = a_0 + a_1 x_1 + a_2 x_2 + ... + a_n x_n
\end{equation}

---

It is clear that the relation beteween a linear regression and a logistic regression is given by the activation function (sigmoid). In fact, considering the sigmoid functon

\begin{equation}
S(z) = \frac{1}{1+e^{-z}}
\end{equation}

and replacing the linear regression equation as $S=y$ and $z = a_0 + a_1 x_1 + a_2 x_2 + ... + a_n x_n$ we obtain

\begin{align}
y = &\frac{1}{1+e^{-(a_0 + a_1 x_1 + a_2 x_2 + ... + a_n x_n)}} \\
\frac{1}{y} = & 1+e^{-(a_0 + a_1 x_1 + a_2 x_2 + ... + a_n x_n)}  \\
\frac{1}{y} - 1 = & e^{-(a_0 + a_1 x_1 + a_2 x_2 + ... + a_n x_n)}  \\
\log \left[\frac{1}{y} - 1\right] = & -(a_0 + a_1 x_1 + a_2 x_2 + ... + a_n x_n)  \\
\log \left[\frac{1-y}{y} \right] = & -(a_0 + a_1 x_1 + a_2 x_2 + ... + a_n x_n) \\
\log \left[\frac{y}{1-y} \right] = & a_0 + a_1 x_1 + a_2 x_2 + ... + a_n x_n
\end{align}

```python
import numpy as np
from matplotlib import pyplot as plt
#import seaborn as sns
#sns.set()
import pandas as pd

from sklearn.linear_model import LinearRegression
```

```python
df = pd.read_csv("HubbleData/hubble.csv")

df
```

      Name
      N_measurements
      velocity
      mean_m

      0
      Virgo
      7
      890
      12.5

      1
      Pegasus
      5
      3810
      15.5

      2
      Pisces
      4
      4630
      15.4

      3
      Cancer
      2
      4820
      16.0

      4
      Perseus
      4
      5230
      16.4

      5
      Coma
      3
      7500
      17.0

      6
      Ursa_Major
      1
      11800
      18.0

      7
      Leo
      1
      19600
      19.0

      8
      [No_Name]
      16
      2350
      13.8

      9
      [No_Name]
      16
      630
      11.6

```python
df.describe()
```

      N_measurements
      velocity
      mean_m

      count
      10.000000
      10.000000
      10.000000

      mean
      5.900000
      6126.000000
      15.520000

      std
      5.626327
      5750.565576
      2.331332

      min
      1.000000
      630.000000
      11.600000

      25%
      2.250000
      2715.000000
      14.200000

      50%
      4.000000
      4725.000000
      15.750000

      75%
      6.500000
      6932.500000
      16.850000

      max
      16.000000
      19600.000000
      19.000000

The data in the data frame includes 10 samples (Nebulae) with 4 features:

**Name** : Name of the nebula \
**N_measurement** : Number of velocities measured by Hubble and Humason \
**velocity** : Mean velocity of the nebula measured in km/s \
**mean_m** : Apparent magnitude of the nebula

___

Remember that the apparent magnitude is related to the distance in parsecs through the relation

$\log_{10} d = \frac{m-M+5}{5}$

where $M=-13.8$ is the absolute magnitude reported by Hubble and considered as a constant in the paper.

We already know that the linear fit will be made between the apparent magnitude and the logarithm of the velocity and therefore we introduce a new column in the dataframe.

```python
df['log10_velocity'] = np.log10(df['velocity'])
df
```

      Name
      N_measurements
      velocity
      mean_m
      log10_velocity

      0
      Virgo
      7
      890
      12.5
      2.949390

      1
      Pegasus
      5
      3810
      15.5
      3.580925

      2
      Pisces
      4
      4630
      15.4
      3.665581

      3
      Cancer
      2
      4820
      16.0
      3.683047

      4
      Perseus
      4
      5230
      16.4
      3.718502

      5
      Coma
      3
      7500
      17.0
      3.875061

      6
      Ursa_Major
      1
      11800
      18.0
      4.071882

      7
      Leo
      1
      19600
      19.0
      4.292256

      8
      [No_Name]
      16
      2350
      13.8
      3.371068

      9
      [No_Name]
      16
      630
      11.6
      2.799341

```python
df.describe()
```

      N_measurements
      velocity
      mean_m
      log10_velocity

      count
      10.000000
      10.000000
      10.000000
      10.000000

      mean
      5.900000
      6126.000000
      15.520000
      3.600705

      std
      5.626327
      5750.565576
      2.331332
      0.461869

      min
      1.000000
      630.000000
      11.600000
      2.799341

      25%
      2.250000
      2715.000000
      14.200000
      3.423532

      50%
      4.000000
      4725.000000
      15.750000
      3.674314

      75%
      6.500000
      6932.500000
      16.850000
      3.835921

      max
      16.000000
      19600.000000
      19.000000
      4.292256

```python
plt.scatter(df['mean_m'], df['log10_velocity'])
plt.xlabel(r'apparent magnitude')
plt.ylabel(r'logarithm of velocity')
plt.show()
```

### Linear Fit using  `scikit-learn`

From the `scikit-learn.linear_model` package, we will import the function `LinearRegression`. COmplete documentation about it can be found at

https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html

This function fits a linear model to minimize the residual sum of squares between the observed targets in the dataset and the targets predicted by the linear approximation (ordinary least squares fit).

Before making the fit, we need to prepare our data. The `LinearRegression().fit()` metod receives as arguments **two arrays**. Hence, we will transform the dataframe features into arrays that will train the model.

```python
X_train = np.asanyarray(df['mean_m'])
y_train = np.asanyarray(df['log10_velocity'])

X_train , y_train
```

    (array([12.5, 15.5, 15.4, 16. , 16.4, 17. , 18. , 19. , 13.8, 11.6]),
     array([2.94939001, 3.58092498, 3.66558099, 3.68304704, 3.71850169,
            3.87506126, 4.07188201, 4.29225607, 3.37106786, 2.79934055]))

Note that we use the capital letter `X` to denote the independent variable and the letter `y` to denote the dependent variable. This convention is used because, in general, we may have multiple independent variables.

Since we are going to make a 1-dimensional linear fit with the form

$y = a_1 + a_2 x$ ,

the method `LinearRegression().fit()` requires a column array `X_train` and a row array `y_train`. Thus, we reshape the X data array:

```python
X_train = X_train.reshape(-1,1)
X_train , y_train
```

    (array([[12.5],
            [15.5],
            [15.4],
            [16. ],
            [16.4],
            [17. ],
            [18. ],
            [19. ],
            [13.8],
            [11.6]]),
     array([2.94939001, 3.58092498, 3.66558099, 3.68304704, 3.71850169,
            3.87506126, 4.07188201, 4.29225607, 3.37106786, 2.79934055]))

Now, we train the linear model

```python
from sklearn.linear_model import LinearRegression

lr = LinearRegression().fit(X_train, y_train)
```

```python
type(lr)
```

    sklearn.linear_model._base.LinearRegression

The parameters obtained from the lienar fit are recovered with the methods `.intercept_` ($a_1$) and `.coef_` ($a_2$).

```python
lr.intercept_
```

    0.5484040402255594

```python
lr.intercept_ , lr.coef_
```

    (0.5484040402255594, array([0.19666889]))

Note that the `.coef_` method returns an array becuase the `LinearRegression` fit can be applied to multidimensional systems.

The resulting coefficients are almost equal to those obtained by our hand-made linear fit,

a_1 = 0.5484040402255217 \
a_2 = 0.19666889208743046

The `LinearRegression` function also includes the method `.score()` which returns the coefficient of determination $R^2$ for the model. This is defined as

$R^2 = 1 - \frac{\Sigma (a_1 + a_2 x_i - y_i)^2}{\Sigma (y_i - \bar{y})^2} = 1 - \frac{\Sigma (y(x_i) - y_i)^2}{\Sigma (y_i - \bar{y})^2}$

where $\bar{y} = \frac{1}{N} \Sigma y_i$ is the mean value of the y-data.The best possible score is $R^2 = 1.0$ and it can take negative values, because the model can be arbitrarily worse. A constant model that always predicts the expected value of each y, disregarding the input features, would get a score of $R^2 = 0.0$.

In this case, we obtain a really good fit:

```python
lr.score(X_train, y_train)
```

    0.9854665665565867

```python

```

21. Machine Learning II. Logistic Regression/01. LogisticRegression.ipynb