# Fitting an elastic net Exponential model

$$
\newcommand{\bbeta}{\boldsymbol{\beta}}
\newcommand{\bX}{\boldsymbol{X}}
\newcommand{\bW}{\boldsymbol{W}}
\newcommand{\bx}{\boldsymbol{x}}
\newcommand{\bz}{\boldsymbol{z}}
\newcommand{\by}{\boldsymbol{y}}
\newcommand{\bt}{\boldsymbol{t}}
\newcommand{\biota}{\boldsymbol{\iota}}
\newcommand{\bdelta}{\boldsymbol{\delta}}
$$

## Introduction

There are many situations where a statistical model needs to be fit to data whose values are only *partially* known. When the outcome, or label, of a dataset is known to be *at least* or *at most* some amount, this process is referred to as censoring. Income data published by national statistics agencies will often bin this data to some upper bound, e.g. greater than \$500K dollars, making the exact number of millionaires to be unknown with certainty. In the biomedical domain many biological measurements are not measureable if they are outside some detectable range. When a measurement is known to be *at least* as large as some value this is known as right-censoring. A classic case of right-censoring is a survival time, where a patient is known to have lived for *at least* some amount of time. For regression models with a fixed number of covariates, there are a variety of approaches for fitting data to be able to handle right-censored data. These includes [Cox regression](https://lifelines.readthedocs.io/en/latest/Survival%20Regression.html#cox-s-proportional-hazard-model), [parametric survival models](https://lifelines.readthedocs.io/en/latest/Survival%20Regression.html#parametric-survival-models), the [Buckley-James Estimator](https://lifelines.readthedocs.io/en/latest/Survival%20Regression.html#parametric-survival-models), or more modern machine learning like [random survival forests](https://square.github.io/pysurvival/models/survival_forest.html) and [a variety of others](https://scikit-survival.readthedocs.io/en/latest/api.html).

In the case where the dataset has many features, often exceeding the sample size, a particular type of regularization will need to be used to ensure only a small subset of reliable features are ultimately employed. In this post I will show how to build an elastic-net regularized parametric survival model for the [Exponential](https://en.wikipedia.org/wiki/Exponential_distribution) distribution. The `scikit-survival` package currently has an [implementation](https://scikit-survival.readthedocs.io/en/latest/generated/sksurv.linear_model.CoxnetSurvivalAnalysis.html#sksurv.linear_model.CoxnetSurvivalAnalysis) for an elastic-net Cox-PH model. While the Cox model has fewer assumptions, it is harder to make predictions out of sample, whereas a parametric model gives each observation its own survival distribution with known moments (mean, median, quantiles, etc).

## (1) Linear Exponential model

This first section will show how to fit a simple low-dimensional Exponential model using maximum likelihood optimization. The (log-)likelihood for such a model with a $n$ i.i.d. samples can be written as:

$$
\begin{align*}
L(\lambda) &=  \prod_{i=1}^n \hspace{1mm} f(\lambda;t_i)^{\delta_i} S(\bbeta;t_i)^{1-\delta_i}  \\
&= \prod_{i=1}^n \hspace{1mm} h(\bbeta; t_i)^{\delta_i} S(\lambda;t_i) \\
\log L(\lambda) = \ell(\lambda) &= \sum_{i=1}^n \big[ \delta_i \log h(\lambda; t_i) + \log S(\lambda;t_i) \big],
\end{align*}
$$

Where $\delta_i$, $t_i$, and $\lambda$, are a censoring indicator, measurement time, and rate parameter, respectively ($\delta_i=0$ indicates censoring). The density, survival (one minus the CDF), and hazard functions of the exponential distribution are $f(t_i;\lambda)=\lambda \exp(-\lambda t_i)$, $S(t_i;\lambda)=\exp(-\lambda t_i)$, and $h(t_i; \lambda) = \lambda$, respectively. A linear Exponential model parameterizes $h(\bbeta; t_i) = \lambda_i = \exp(\bx^T \bbeta)$, where $\eta_i = \bx_i^T \bbeta = \sum_{j=0}^p \beta_jx_j$, and $S(\bbeta; t_i) = \exp(- \lambda_i t_i)= \exp(-e^{\eta_i} t_i)$. The fully parameterized log-likelihood is therefore:

$$
\begin{align*}
\ell(\bbeta) &= \sum_{i=1}^n \big[ \delta_i \eta_i - e^{\eta_i} t_i \big] \\
\frac{\partial \ell(\bbeta)}{\partial \beta_j} &= \sum_{i=1}^n \big[ \delta_i x_{ij} - z_i x_{ij} \big] = \bX_j^T(\bdelta - \bz) , \hspace{3mm} \\
z_i &= e^{\eta_i} t_i \\
\frac{\partial ^2 \ell(\bbeta)}{\partial \beta_j \partial \beta_k} &= - \sum_{i=1}^n x_{ij}x_{ik} z_i
\end{align*}
$$

In vectorized notation, the gradient and hessian of the (negative) log-likelihood will be:

$$
\begin{align*}
\nabla_\bbeta &= - \bX^T (\bdelta - \bz) \\
H_\bbeta &=  \bX^T \bW \bX, \hspace{2mm} \text{diag}(\bW)_i = [e^{\eta_i} t_i ]
\end{align*}
$$

Notice that if $\delta_i=1$ then the gradient would also be zero if $t_i = 1 / e^{\eta_i}$. This makes sense as the $E[t_i] = \eta^{-1}$ in generalized linear model theory. The only difference here being that we have parameterized the rate parameter as $e^{\bx^T\bbeta}$ to ensure all values will conform to the support of the distribution. To visualize how well the model is fitting the data one could look a whether $\log t_i \approx - \eta_i$ (for the non-censored values at least). To find the optimal $\bbeta$, [Newton's method](https://en.wikipedia.org/wiki/Newton%27s_method_in_optimization) can be used where the update state at the $t+1$ iteration can be written as:

$$
\begin{align*}
\bbeta^{(t+1)} &\gets \bbeta^{(t)} - (H_{\bbeta^{(t)}})^{-1} \nabla_{\bbeta^{(t)}} \\
&\gets \bbeta^{(t)} + (\bX^T \bW^{(t)} \bX )^{-1} \bX^T (\bdelta - \bz^{(t)}) \\
&\gets (\bX^T \bW^{(t)} \bX )^{-1} \bX^T \bW^{(t)} \by^{(t)}, \hspace{2mm} \by^{(t)} = \bX \bbeta^{(t)} + (\bW^{(t)})^{-1} (\bdelta - \bz^{(k)})
\end{align*}
$$

The last line reveals that the Newton update for the Exponential likelihood model is equivalent to a weighted least-squares problem. This optimization procedure is known as iteratively re-weighted least squares (IRLS).

$$
\begin{align*}
\bbeta^{(t+1)} &= \arg\min_{\bbeta} \hspace{2mm} (\by^{(t)} - \bX \bbeta)^T  \bW^{(t)} (\by^{(t)} - \bX \bbeta) \\
&= \arg\min_{\bbeta} \hspace{2mm} \|\tilde\by^{(t)} - \tilde\bX \bbeta)^T\|_2^2 \\ \tilde\by^{(t)}&=\sqrt{\bW^{(t)}}\by^{(t)}, \hspace{2mm} \tilde\bX =\sqrt{\bW^{1/2}}\bX^{(t)}
\end{align*}
$$

Notice that a weighted least-squares problem is equivalent to a normal least-squares method where the rows of the response and design matrix are multiplied by the square-root of the row's weight.

## (2) Optimization approaches

Section (1) showed that the linear exponential model can be solved through either gradient or hessian-based methods as well as its reformulation via IRLS. The first code block below will demonstrate how to generate exponential data with a given censoring rate. The observed time will be: $t_i = \min[c_i, u_i]$, where $u_i \sim \text{Exp}(\lambda_i)$, $\lambda_i = e^{\eta_i}$ and $c_i\sim \text{Exp}(\lambda_c)$ . The probability that one exponential distribution will be greater than another is $P(c_i > u_i) = \frac{\lambda_i}{\lambda_i+\lambda_c}=1-\alpha_i$. A given value of $\lambda_c$ will obtain an average censoring rate of $\alpha(\lambda_c) = n^{-1}\sum_i\alpha_i$. A simple univariate optimization procedure can find that value of $\lambda_c$ such that the averaging censoring rate of $\alpha$ is achieved.

Optimization methods can be sped up and made more robust by normalizing the features of the data so that the gradients are numerically stable and within reasonable ranges. If the columns of the data are standardized to have a mean and standard deviation of zero and one, respectively, and the measurment time is divided by its largest value to ensure $t_i \in (0,1]$, then it is easy to show that the following post-optimization transformation would recover the original un-normalized solution:

$$
\begin{align*}
\bbeta_1 &= \tilde\bbeta_1 / \sigma_1 \\
\beta_0 &= \tilde\beta_0 - (\log \bt_{\max} + \mu^T \bbeta_1)
\end{align*}
$$

```python
import pandas as pd
import numpy as np
import scipy as sp
from scipy import optimize

# Function to find lambda_c=exp(gamma) given a censoring rate
def f_cens_exp(gamma,cens,eeta):
    return((np.mean(1- eeta/(np.exp(gamma)+eeta) )-cens)**2)

def rexp(rate,n): # Draw from exponential distribution with rate parameter
  return(np.random.exponential(scale=1/rate,size=n))

def dgp_exp(n,p,cens=0.1,ss=1234): # draw data from true linear model
  np.random.seed(ss)
  btrue = 2*np.random.rand(p+1)-1
  b0, b1 = btrue[0], btrue[1:]
  X = np.random.rand(n,p)
  eeta = np.exp(X.dot(b1) + b0)
  t_true = rexp(eeta,n)
  if cens == 0:
      lam_c = 1e-10
  else:
      lam_c = np.exp(optimize.minimize_scalar(f_cens_exp,
                      args=(cens,eeta),method='brent').x)
  t_cens = rexp(lam_c, n)
  c_right = np.where(t_cens < t_true, 0, 1)
  t_obs = np.where(c_right==0, t_cens, t_true)
  return(([t_obs, c_right], X, btrue))

# Event rate is 1-censoring rate
c_rate = np.arange(0.2,0.8,0.2)
event_prop = [dgp_exp(n=10000,p=5,cens=cc)[0][1].mean() for cc in c_rate]
print(np.round(pd.DataFrame({'cens':c_rate,'delta_mu':event_prop}),1))
```

       cens  delta_mu
    0   0.2       0.8
    1   0.4       0.6
    2   0.6       0.4
    3   0.8       0.2

In the subsequent code block, the `linear_wrapper` class will be defined which can either solve the problem using gradient descent (GD) or IRLS. Subsequent simulations will show that 1) both approaches yield identical solutions (up to some small tolerance), and 2) the coefficients estimates are *on average* correct (technically the results are [consistent](https://en.wikipedia.org/wiki/Consistent_estimator#Bias_versus_consistency) rather than unbiased, but given a sample size of more than several hundred, the difference is trivial).

```python
import sys
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import ElasticNet

import seaborn as sns
from matplotlib import pyplot as plt

def grad_exp(beta, resp, data):
  eta = data.dot(beta[1:]) + beta[0]
  zz = resp[0] * np.exp(eta)
  g0 = -np.mean(resp[1] - zz)
  g1 = -data.T.dot(resp[1] - zz) / len(eta)
  return(np.append(g0,g1))

def nll_exp(beta, resp, data):
    eta = data.dot(beta[1:]) + beta[0]
    zz = resp[0] * np.exp(eta)
    nll = -np.mean(resp[1]*eta - zz)
    return(nll)

class linear_wrapper():
    def __init__(self,method):
        if method not in ['gd','irls','elnet']:
            sys.exit('error!')
        self.method = method
        self.di_fit = {'gd':self.fit_gd, 'irls':self.fit_irls, 'elnet':self.fit_elnet}
        self.fitter = self.di_fit[self.method]

    def fit(self, data, lbls,**kwargs):
        lbls = lbls.copy()
        self.enc = StandardScaler().fit(data)
        Xtil = self.enc.transform(data)
        self.tmax = max(lbls[0])
        lbls[0] = lbls[0] / self.tmax
        binit = np.append(np.log(sum(lbls[1]) / sum(lbls[0])),np.repeat(0, data.shape[1]))
        if 'kwargs' in self.fitter.__code__.co_varnames:
            self.fitter(data=Xtil,lbls=lbls,binit=binit,**kwargs)
        else:
            self.fitter(data=Xtil,lbls=lbls,binit=binit)
        self.bhat[1:] = self.bhat[1:] / self.enc.scale_.reshape([Xtil.shape[1],1])
        self.bhat[0] -= (np.log(self.tmax) + np.sum(self.bhat[1:] * self.enc.mean_.reshape([Xtil.shape[1],1]),axis=0))

    def fit_gd(self,data,lbls,binit):
        self.bhat = optimize.minimize(fun=nll_exp, x0=binit,args=(lbls, data),method='L-BFGS-B', jac=grad_exp).x
        self.bhat = np.atleast_2d(self.bhat).T

    def fit_irls(self,data,lbls,binit):
        bdiff = 1
        linreg = LinearRegression(fit_intercept=False)
        iX = np.insert(data,0,1,1)
        jj = 0
        while bdiff > 1e-8:
            jj += 1
            eta = iX.dot(binit)
            z = lbls[0] * np.exp(eta)
            y = eta + (lbls[1] - z) / z
            w = np.sqrt(z)
            self.bhat = linreg.fit(iX * w.reshape([len(w), 1]),y * w).coef_
            bdiff = sum((self.bhat - binit)**2)
            binit = self.bhat.copy()
        self.bhat = np.atleast_2d(self.bhat).T

    def fit_elnet(self,data,lbls,binit,**kwargs):
        if 'alpha' in kwargs.keys():
            alpha = max(kwargs['alpha'],1e-2) # Lower bound alpha
        else:
            alpha = 1
        bdiff = 1
        lammax = max(np.abs(data.T.dot(lbls[1]-lbls[0]*np.exp(binit[0])))) / data.shape[0]
        lammax = lammax / alpha
        self.lamseq = np.flip(np.exp(np.linspace(np.log(lammax*1e-3),np.log(lammax*0.99),50)))
        iX = np.insert(data, 0, 1, 1)
        Bmat = np.zeros([iX.shape[1],len(self.lamseq)])
        btemp = binit.copy()
        for jj, lam in enumerate(self.lamseq):
            elnet = ElasticNet(alpha=lam, l1_ratio=alpha,
                    fit_intercept=False, normalize=False, precompute=False, warm_start=True)
            ii = 0
            while bdiff > 1e-8:
                ii += 1
                eta = iX.dot(btemp)
                z = lbls[0] * np.exp(eta)
                y = eta + (lbls[1] - z) / z
                w = np.sqrt(z)
                bhat = elnet.fit(iX * w.reshape([len(w), 1]), y * w).coef_
                # Solve intercept without regularization
                bhat[0] = np.log(sum(lbls[1]) / sum(lbls[0] * np.exp(data.dot(bhat[1:]))))
                bdiff = sum((bhat - btemp) ** 2)
                btemp = bhat.copy()
            bdiff = 1
            Bmat[:,jj] = bhat
            btemp = binit.copy()
        self.bhat = Bmat

nsim = 125
n = 500
p = 10
c_rate = [0,0.25,0.50]
gd_holder = []
for ii in range(nsim):
  for cc in c_rate:
    resp, X, btrue = dgp_exp(n,p,cens=cc,ss=ii)
    mdl_irls = linear_wrapper(method='irls')
    mdl_irls.fit(data=X, lbls=resp)
    mdl_gd = linear_wrapper(method='gd')
    mdl_gd.fit(data=X, lbls=resp)
    sim_ii = pd.DataFrame({'btrue': btrue, 'bhat_irls': mdl_irls.bhat.flat,
                'bhat_gd':mdl_gd.bhat.flat,'cens': cc, 'tt':np.append('b0',np.repeat('b1',p))})
    gd_holder.append(sim_ii)

df_gd = pd.concat(gd_holder).reset_index(drop=True).assign(cens = lambda x: np.round(x.cens,1).astype(str),
                                                           tt = lambda x: x.tt.map({'b0':'intercept','b1':'slope'}))
df_gd_long = df_gd.melt(['btrue','cens','tt'],var_name='method',value_name='bhat')
```

```python
g = sns.FacetGrid(df_gd_long,col='cens',row='tt',hue='tt',height=3.5,aspect=1.5)
g.map(plt.scatter,'bhat','btrue',alpha=0.5)
plt.subplots_adjust(top=0.88)
g.fig.suptitle(t='Figure 1: Consistent estimates using GD/IRLS',fontsize=20)
g.set_xlabels('Estimate'); g.set_ylabels('Actual')
```

```python
g = sns.FacetGrid(df_gd,col='tt',height=3.5,aspect=1.5)
g.map(plt.scatter,'bhat_irls','bhat_gd',alpha=0.5)
plt.subplots_adjust(top=0.8)
g.fig.suptitle(t='Figure 2: GD and IRLS are identical',fontsize=20)
g.set_xlabels('IRLS'); g.set_ylabels('GD')
```

# (3) Regularized Exponential model

In real-world applications most datasets will have extraneous features that do not add any predictive power to a model. Elastic net regularization acts as a way to both control model variance and remove unhelpful features. The likelihood from the original linear exponential model can be augmented by the followoing penalty term:

$$
\begin{align*}
\ell_{\alpha;\lambda}(\bbeta) &= \ell(\bbeta) + \lambda\big[\alpha \|\bbeta_1\|_1 + 0.5(1-\alpha)]\|\bbeta_1\|_1 \big] \\
&= \ell(\bbeta) + P_{\alpha;\lambda}(\bbeta_1)
\end{align*}
$$

Where $\bbeta_1$ denotes that the norms are not calculated over the intercept (indexed to column 0). The Newton update will be equavilent to solving a least-squares Lasso problem at each iteration as before.

$$
\begin{align}
\bbeta^{(t+1)} &= \arg\min_{\bbeta} \hspace{2mm} \frac{1}{2n} \|\tilde\by^{(t)} - \tilde\bX \bbeta)^T \|_2^2 + P_{\alpha;\lambda}(\bbeta)  % \label{eq:irls}
\end{align}
$$

The `ElasticNet` class from `sklearn` has very fast coordinate descent solvers for this type of optimization problem. Somewhat confusingly, what `sklearn` calls `alpha` and the `l1_ratio` what is usually referred to as `lambda` and `alpha`, respectively. The code below will show how to the use this function class and solve the coefficient solution path for a sequence of $\lambda$ values where $\lambda_{\text{min}}=0$ implies no regularization and $\lambda_{\text{max}}=\|\bX^T (\bdelta - \bz_0) \|_\infty/n$ is the infimum which achieves sparsity for all non-intercept values. The `linear_wrapper` class contains the `fit_elnet` attribute which implements the IRLS approach to solving this optimization problem. Note that after the initial `fit` attribute is called, the intercept is re-fit solving its closed form solution without regularization to ensure the model has converged: $\beta_0 = \log \frac{\sum_i \delta_i}{\sum_i t_i e^{\bx_i^T \bbeta_1}}$.

We can visualize the Lasso solution path seen above for different values of $\alpha$ for a well-known [survival dataset](https://lifelines.readthedocs.io/en/latest/lifelines.datasets.html#lifelines.datasets.load_gbsg2).

```python
# Load in the German breast cancer dataset
url = "https://raw.githubusercontent.com/alonsosilvaallende/GBCSG2-dataset_Cox_RSF_GBC_DeepSurv/master/data/GBCSG2.csv"
df = pd.read_csv(url)
# time, censoring
tt_dd = [df.time.astype(int).values, np.where(df.event,1,0)]
# one-hot-encoded design matrix
df = pd.get_dummies(df.drop(columns=['time','event']),drop_first=True)
X = df.values

mdl_elnet = linear_wrapper(method='elnet')
holder = []
for aa in [0.25, 0.50, 1.0]:
    mdl_elnet.fit(data=X, lbls=tt_dd, alpha=aa)
    df_aa = pd.DataFrame(mdl_elnet.bhat,columns=mdl_elnet.lamseq)
    df_aa.index = np.append('intercept', df.columns)
    df_aa = df_aa.reset_index().melt('index',value_name='bhat',var_name='lam').assign(alpha = aa)
    holder.append(df_aa)
df_aa = pd.concat(holder).assign(lam_alpha = lambda x: x.lam * x.alpha)
ml = linear_wrapper(method='irls')
ml.fit(X,tt_dd)
df_ml = pd.DataFrame({'coefficient':df.columns,'ml':ml.bhat.flatten()[1:]})
df_aa = df_aa[~(df_aa['index']=='intercept')].rename(columns={'index':'coefficient'})

g = sns.FacetGrid(data=df_aa,col='coefficient',hue='alpha',col_wrap=3,height=3,aspect=1.5,sharey=False)
g.map(plt.plot,'lam_alpha','bhat',marker='o')
plt.subplots_adjust(top=0.88)
g.add_legend()
[z.set_text(t) for z,t in zip(g._legend.texts,['25%','50%','100%'])]
g.fig.suptitle(t='Figure 3: Elastic net solution path',fontsize=20)
g.set_xlabels('lambda * alpha'); g.set_ylabels('coefficient')
for ii,ax in enumerate(g.axes):
    ax.axhline(y=df_ml.loc[ii,'ml'],c='black',linestyle='dashed')
```


# No-Stop Time aka NDL

From **Deco for Divers**:

\begin{align}
P_{no-stop} = (M_{0} - P_{alv0} - R (t_{asc} - 1/k)) e^{k * t_{asc}} + P_{alv0} - R/k
\end{align}

where

  * $ M_{0} $ : M-value at sea level (bar)
  * $ t_{asc} $ : time needed for ascent, depth/speed (min)
  * $ P_{no_stop} $ : partial pressure at which to start the ascent (bar)
  * $ P_{alv0} $ : alveolar partial pressure
  * $ d $ : depth (m)
  * $ k $ : ln(2)/tissue half-time
  * $ R $ : is the rate of change in inert gas paressure, $ \Delta P $ times fraction of inert gas (bar/min)

This gives no-stop pressure $ P_{no-stop} $ at a given depth (through $ P_{alv0} $) but not the time $ t $ when you're going to reach it.

From Baker's **No-Stop Time**:

\begin{align}
t = (-1/k) * ln( (P_{alv0} - M_{0})/(P_{alv0} - P_{tc}) )
\end{align}

-- as above but also accounts for

  * $ P_{tc} $ : current dissolved gas pressure in tissue compartment

The problem with Baker's version is it has to avoid division by zero and a logarithm of negative number, or of number greater than 1. So it only works **iff** $ P_{alv0} > M_{0} > P_{tc} $

**So** you have to exclude tissue compartments whose $ M_{0} $ is less than current $ P_{alv0} $

```python

```


# Imports

```python
import pynncml as pnc
from matplotlib import pyplot as plt
```

# Load OpenCML Dataset

```python
open_cml_dataset = pnc.read_open_cml_dataset('../dataset/open_cml.p') #  read OpenCML dataset
```

# Select Link and Plot link data

```python
link_index = 0
link_data = open_cml_dataset[link_index] # select a link
link_data.plot() # plot link data
plt.show()
```

# Run Statistics wet dry classification
This section performance wet dry classification using statistics test, this method introduced by [1].

The first step is calculating the standard deviation over a window of samples using the following equation:
$$ \sigma_n=\Bigg[\frac{1}{|W_n|}\sum_{k \in W_n}\Big(A_k-E_{W_n}[A]\Big)^2\Bigg]^{\frac{1}{2}} $$

where $W_n$ is a moving standard deviation window in range $[n,n+N_w]$ , $A_k$ is the instance attenuation measurement of sample $k$ and $A$ is the vector of all samples and
$N_w=|W_n|$ is an hyper-parameter that set the window size.

$$E_{W_n}[A]=\frac{1}{|W_n|}\sum_{k \in W_n}A_k $$

Once we obtained the standard deviation vector $\sigma_n$, the following threshold function is apply
\begin{equation}
\hat{y}^{wd}_n=\begin{cases}
1 \quad \sigma_n>\sigma_0 \\
0 \quad\text{o.w}
\end{cases}
\end{equation}

where $\hat{y}^{wd}_n=1$ is wet even and $\hat{y}^{wd}_n=0$ is dry event and $\sigma_0$ is an hyper-parameter.

```python
swd = pnc.wet_dry.statistics_wet_dry(1.1, 4) # init classification model
wd_classification, std_vector = swd(link_data.attenuation()) # run classification method
```

```python
plt.subplot(1, 2, 1)
plt.plot(link_data.time(), wd_classification.numpy()[0, :])
plt.title('Classification')
plt.xlabel('Time')
pnc.change_x_axis_time_format('%H')
plt.grid()
plt.subplot(1, 2, 2)
plt.plot(link_data.time(), std_vector.numpy()[0, :])
plt.xlabel('Time')
plt.title('Standard Deviation')
pnc.change_x_axis_time_format('%H')
plt.ylabel(r'$\sigma_n$')
plt.grid()
plt.show()
```

# References
[1] Marc Schleiss and Alexis Berne. Identification of dry and rainy periods using telecommuni-
cation microwave links. IEEE Geoscience and Remote Sensing Letters, 7(3):611–615, 2010


$$
\newcommand{\set}[1]{\left\{#1\right\}}
\newcommand{\abs}[1]{\left\lvert#1\right\rvert}
\newcommand{\norm}[1]{\left\lVert#1\right\rVert}
\newcommand{\inner}[2]{\left\langle#1,#2\right\rangle}
\newcommand{\bra}[1]{\left\langle#1\right|}
\newcommand{\ket}[1]{\left|#1\right\rangle}
\newcommand{\braket}[2]{\left\langle#1|#2\right\rangle}
\newcommand{\ketbra}[2]{\left|#1\right\rangle\left\langle#2\right|}
\newcommand{\angleset}[1]{\left\langle#1\right\rangle}
$$

# Spectral Theory

_prepared by Israel Gelover_

### Remark 2.17 Spectral Theory

Given $A:\mathcal{H} \to \mathcal{H}$ a linear operator, we can consider $A_\lambda = A - \lambda I$ with $\lambda \in \mathbb{C}$. The study of the distribution of the values of $\lambda$ for which $A_\lambda$ whether has or does not have an inverse is called _Spectral Theory_ of the operator.

In quantum computing we will only work with finite-dimensional operators and this type of operators only have one type of spectrum, which are the eigenvalues of the operator.

Let us recall that to obtain the eigenvalues of a matrix $A$ we calculate $\det(A - \lambda I) = 0$. That is, we look for the values of $\lambda$ for which the matrix $A - \lambda I$ has no inverse. Meaning that the eigenvalues are a particular case of the spectrum of an operator.

### Definition 2.18 Eigenvalues and Eigenvectors

Let $A:\mathcal{H} \to \mathcal{H}$ be a linear operator

1. The set $P(A) = \set{\lambda \in \mathbb{C} \mid A\ket{f} = \lambda\ket{f} \text{ for some } \ket{f} \neq 0}$ is called the point spectrum of $A$, and its elements are known as _Eigenvalues_ of $A$. **Remark:** If $\mathcal{H}$ is of finite dimension, then $P(A) = \set{\text{solutions of } \det(A - \lambda I) = 0}$
2. Let $\lambda_0 \in P(A)$, if $A\ket{f} = \lambda_0 \ket{f}$ has non-trivial solutions, that is, $\lambda_0$ is an eigenvalue, we will denote by$\mathcal{L}(\lambda_0)$ the subspace of $\mathcal{H}$ generated by the solutions
of $A\ket{f} = \lambda_0 \ket{f}$.
3. $\dim \set{\mathcal{L}(\lambda_0)}$ is called the multiplicity of $\lambda_0$. Let us recall that when finding the non-trivial solutions of $\det(A - \lambda I) = 0$ we can find that the monomial associated with an eigenvalue appears more than once in the characteristic polynomial of $A$, this number of repetitions is what we know as the multiplicity of the eigenvalue.

Intuitively we can interpret that finding the eigenvalues of an operator has the geometric equivalence of finding subspaces that remain invariant under the application of the operator. For example, if we have a linear operator $A:\mathbb{R}^2 \to \mathbb{R}^2$ and we find the eigenvalues of $A$, this would be equivalent to finding a line that is fixed with respect to the application of the operator $A$.

### Task 1.

(On paper) Let $A:\mathcal{H} \to \mathcal{H}$ be a self-adjoint operator, (i.e $A=A^\dagger$).
If $A\ket{f} = \lambda\ket{f}$, use Definition 2.2 and Definition 2.12 to conclude that the eigenvalues of an autoadjoint operator are real numbers.

Click here for solution

### Task 2.

(On paper) Let $U:\mathcal{H} \to \mathcal{H}$ be a unitary operator.
If $U\ket{f} = \lambda\ket{f}$, calculate $\braket{f}{f}$ to conclude that the eigenvalues of a unit operator are complex numbers with norm one.

Click here for solution

### Proposition 2.19

Let $A:\mathcal{H} \to \mathcal{H}$ be a self-adjoint operator, then the eigenvectors corresponding to different eigenvalues are orthogonal.

**Remark:** From this proposition the _Spectral Theorem_ can be derived, which states that there exists an orthonormal basis of the space consisting of eigenvectors of the self-adjoint operator.

**Proof:**

Let $\lambda_1, \lambda_2$ be different eigenvalues, that is $\lambda_1 \neq \lambda_2$, such that

\begin{equation*}
    A\ket{f_1} = \lambda_1\ket{f_1} \enspace \text{ and } \enspace A\ket{f_2} = \lambda_2\ket{f_2}
\end{equation*}

So, on the one hand we have

\begin{equation*}
    \ket{f_1}\cdot (A\ket{f_2}) = \ket{f_1} \cdot (\lambda_2\ket{f_2}) = \lambda_2(\ket{f_1} \cdot \ket{f_2})
\end{equation*}

On the other hand

\begin{equation*}
    (A\ket{f_1}) \cdot \ket{f_2} = (\lambda_1\ket{f_1}) \cdot \ket{f_2} = \lambda_1^*(\ket{f_1} \cdot \ket{f_2})
\end{equation*}

Since $A$ is a self-adjoint operator, we know that $\ket{f_1}\cdot (A\ket{f_2}) = (A\ket{f_1}) \cdot \ket{f_2}$ and also $\lambda_1, \lambda_2 \in \mathbb{R}$, then $\lambda_1\braket{f_1}{f_2} = \lambda_2\braket{f_1}{f_2}$.

Therefore $\braket{f_1}{f_2} = 0$, since $\lambda_1 \neq \lambda_2$.

### Definition 2.19 Outer Product

Let $\ket{f}, \ket{g} \in \mathcal{H}$ Hilbert space, the _Outer Product_ of $\ket{f}$ and $\ket{g}$ is a linear operator such that

\begin{equation*}
    \begin{split}
        \ketbra{f}{g}:\mathcal{H} &\to \mathcal{H} \\
        \ket{x} &\to \ket{f}\braket{g}{x}
    \end{split}
\end{equation*}

That is, applying the outer product of $\ket{f}$ and $\ket{g}$ to an arbitrary vector $\ket{x}$, is equivalent to calculate the inner product of $\ket{g}$ with $\ket{x}$ and multiplying the resulting scalar by $\ket{f}$.

### Examples

1. Let $\ket{\psi} = \frac{1}{\sqrt{2}}(\ket{0} + \ket{1})$ and $\ket{\phi} = \ket{0}$, then

\begin{equation*}
    \begin{split}
        \ketbra{\psi}{\phi} &= \frac{1}{\sqrt{2}}(\ket{0}+\ket{1})\bra{0} = \frac{1}{\sqrt{2}}(\ketbra{0}{0}+\ketbra{1}{0}) \\
        &= \frac{1}{\sqrt{2}}\left[\begin{pmatrix} 1 \\ 0 \end{pmatrix} \begin{pmatrix} 1 & 0 \end{pmatrix} + \begin{pmatrix} 0 \\ 1 \end{pmatrix} \begin{pmatrix} 1 & 0 \end{pmatrix}\right] \\
        &= \frac{1}{\sqrt{2}}\left[\begin{pmatrix} 1 & 0 \\ 0 & 0 \end{pmatrix} + \begin{pmatrix} 0 & 0 \\ 1 & 0 \end{pmatrix}\right] = \frac{1}{\sqrt{2}}\begin{pmatrix} 1 & 0 \\ 1 & 0 \end{pmatrix}
    \end{split}
\end{equation*}

Let us recall that **Definition 2.20** tells us that the outer product of two vectors is a linear operator, in this case we have the outer product of two qubits and therefore we obtain a two-dimensional operator, that is, we obtain a $2 \times 2$ matrix. Note that from the expression of the outer product using Dirac notation we can deduce how it looks like in matrix notation, as follows:

\begin{equation}\label{dirac_matriz}
    \ketbra{\psi}{\phi} = \frac{1}{\sqrt{2}}(\overbrace{\ket{0}}^\text{row} \underbrace{\bra{0}}_\text{column} + \overbrace{\ket{1}}^\text{row} \underbrace{\bra{0}}_\text{column})
\end{equation}

This tells us that the associated matrix will have $\frac{1}{\sqrt{2}}$ in row $0$ - column $0$ and in row $1$ - column $0$ and zeros in the rest of the positions, the same as we obtained when calculating using vector notation:

\begin{equation*}
    \ketbra{\psi}{\phi} = \frac{1}{\sqrt{2}}(\ketbra{0}+\ketbra{1}{0}) = \frac{1}{\sqrt{2}}\begin{pmatrix} 1 & 0 \\ 1 & 0 \end{pmatrix}
\end{equation*}

2. Let $\ket{\psi}, \ket{\phi}$ be as in the previous example, then

\begin{equation*}
    \begin{split}
        (\ketbra{\psi}{\phi})\ket{1} = \ket{\psi}\braket{\phi}{1} &= \frac{1}{\sqrt{2}}(\ket{0} + \ket{1})\braket{0}{1} = 0 \\
        &= \frac{1}{\sqrt{2}}\begin{pmatrix} 1 & 0 \\ 1 & 0 \end{pmatrix}\begin{pmatrix} 0 \\ 1 \end{pmatrix} = \frac{1}{\sqrt{2}}\begin{pmatrix} 0 \\ 0 \end{pmatrix} = \begin{pmatrix} 0 \\ 0 \end{pmatrix}
    \end{split}
\end{equation*}

In both cases we obtain vector **0**, but clearly, Dirac notation is much more practical and therefore we will prefer it in the rest of the notebooks.

### Definition 2.21 Completeness Relation

Let $\set{\ket{u_i}}_{i=1}^{n}$ be an orthonormal basis of a space $\mathcal{H} \implies \sum_{i=1}^{n}\ketbra{u_i}{u_i} = I$

Let us recall that the outer product of two vectors is a linear operator, and since the sum of linear operators is again a linear operator, in this case we obtain the identity operator.

**Proof:**

Let $\ket{\psi} \in \mathcal{H}$ be a vector, we can express it in terms of the basis as $\ket{\psi} = \sum_{j=1}^{n}c_j\ket{u_j}$, with $c_j \in \mathbb{F}$ elements of the field. Thus

\begin{equation*}
    \begin{split}
        \left(\sum_{i=1}^{n}\ketbra{u_i}{u_i}\right)\ket{\psi} &= \sum_{i=1}^{n}\ket{u_i}\braket{u_i}{\psi} \\
        &= \sum_{i=1}^{n}\ket{u_i} \sum_{j=1}^{n}c_j \braket{u_i}{u_j} = \sum_{i=1}^{n}c_i\ket{u_i} = \ket{\psi}
    \end{split}
\end{equation*}

since $\braket{u_i}{u_j}$ takes value $1$ when $i=j$ and $0$ when $i \neq j$, as they are elements of an orthonormal basis.


```python
from scipy import stats
import math as m
from sklearn.neighbors.kde import KernelDensity
from sympy import *
from scipy.stats.kde import gaussian_kde
from scipy.stats import norm
import numpy as np
import pandas as pd
from datetime import datetime as dt
import os
```

```python
work_dir = os.getcwd()
```

```python
def sharp_omega(freq):
    if freq == "weekly":
        df_etf = pd.read_csv(os.path.join(work_dir, "data", "output_weekly_with_rf.csv"), index_col = False)
    elif freq == "monthly":
        df_etf = pd.read_csv(os.path.join(work_dir, "data", "output_monthly_with_rf.csv"), index_col = False)
    else:
        raise print("Your freq is wrong!")
    Omega = pd.DataFrame()
    columns_name = []
    omega = []
    for i in range(1, df_etf.shape[1]-1):
        #exclude Date and Rf
        columns_name.append(df_etf.columns[i])
        temp_etf = df_etf.iloc[:, i].dropna()
        x_value = np.linspace(-0.5,0.5,1000)
        Etf_gqussian = gaussian_kde(temp_etf)
        L = np.nanmean(df_etf['Rf'])
        pdf = (Etf_gqussian(x_value)/Etf_gqussian(x_value).sum())
        cdf = pdf.cumsum()
        cdf = pd.DataFrame(cdf, index=x_value)
        omega.append(((1-cdf).loc[L:].sum())/(cdf.loc[:L].sum()))
    Omega = pd.DataFrame(omega, index = columns_name)
    Omega.columns = ["Sharp-Omega"]
    Omega["Rank"] = Omega["Sharp-Omega"].rank(ascending=False)
    return Omega
```

```python
if __name__ == '__main__':
    Omega_weekly = sharp_omega(freq = "weekly")
    Omega_monthly = sharp_omega(freq = "monthly")
```

```python
Omega_weekly
```

      Sharp-Omega
      Rank

      FPA
      0.908515
      29.0

      FPXI
      1.227656
      3.0

      IQDE
      1.032982
      23.0

      IQDY
      1.107187
      16.0

      CHIQ
      1.157367
      11.0

      GEM
      1.195171
      5.0

      ADRA
      1.037718
      22.0

      PAF
      1.028545
      24.0

      IEMG
      1.202105
      4.0

      HEEM
      1.057868
      21.0

      AXJV
      0.949393
      28.0

      EMGF
      1.171963
      7.0

      ACWX
      1.144497
      13.0

      AAXJ
      1.167515
      8.0

      EEMA
      1.162804
      10.0

      AIA
      1.192318
      6.0

      IPOS
      1.091952
      19.0

      CWI
      1.142671
      14.0

      QEMM
      1.167366
      9.0

      MOTI
      1.265989
      2.0

      REMX
      1.013550
      26.0

      VEU
      1.145684
      12.0

      VXUS
      1.140072
      15.0

      AXJL
      1.016862
      25.0

      EMCG
      1.103721
      17.0

      XSOE
      1.327659
      1.0

      DNL
      1.097952
      18.0

      IHDG
      1.083650
      20.0

      DBAP
      1.012345
      27.0


## Parameter estimation for a linear operator using Gaussian processes

Assumptions about the linear operator:

$\mathcal{L}_x^\phi u(x) = f(x)$

$u(x) \sim \mathcal{GP}(0, k_{uu}(x,x',\theta))$

$f(x) \sim \mathcal{GP}(0, k_{ff}(x,x',\theta,\phi))$

$y_u = u(X_u) + \epsilon_u; \epsilon_u \sim \mathcal{N}(0, \sigma_u^2I)$

$y_f = f(X_f) + \epsilon_f; \epsilon_f \sim \mathcal{N}(0, \sigma_f^2I)$

Taking a simple operator as example:

$\mathcal{L}_x^\phi := \phi_1 \cdot + \phi_2\frac{d}{dx}\cdot + \phi_3\frac{d^2}{dx^2}\cdot$

$u(x) = x^2$

$f(x) = \phi_1 x^2 + \phi_2 2x + \phi_3 2$

Problem at hand:

Given $\{X_u, y_u\}$ and $\{X_f, y_f\}$, estimate $\phi_1$ $\phi_2$ $\phi_3$.

#### step 1: simulate data

Use $\phi_1 = 1$    $ \phi_2 = 3$   $ \phi_3 = 6$

```python
import numpy as np
import sympy as sp
from scipy.optimize import minimize
import matplotlib.pyplot as plt
```

```python
x = 2*np.random.rand(20)
phi1 = 1.0
phi2 = 3.0
phi3 = 0.0
y_u = x**2
y_f = phi1*x**2+phi2*2*x+phi3*2
```

```python

plt.plot(x, y_u, 'ro')
plt.show()
```

#### step 2: create covariance matrix

This step uses information about $\mathcal{L}_x^\phi$ but not about $u(x)$ or $f(x)$.

$k_{uu}(x_i, x_j; \theta) =  exp(-\frac{1}{2}\theta(x_i-x_j)^2)$

```python
x_i, x_j, theta, phi1, phi2, phi3 = sp.symbols('x_i x_j theta phi1 phi2 phi3')
kuu_sym = sp.exp(theta*(-1)/(2)*((x_i - x_j)**2))
kuu_fn = sp.lambdify((x_i, x_j, theta), kuu_sym, "numpy")
def kuu(x, theta):
    k = np.zeros((x.size, x.size))
    for i in range(x.size):
        for j in range(x.size):
            k[i,j] = kuu_fn(x[i], x[j], theta)
    return k
```

$k_{ff}(x_i,x_j;\theta,\phi) \\
= \mathcal{L}_{x_i}^\phi \mathcal{L}_{x_j}^\phi k_{uu}(x_i, x_j; \theta) \\
= \mathcal{L}_{x_i}^\phi \left( \phi_1 k_{uu} + \phi_2 \frac{\partial}{\partial x_j}k_{uu} +\phi_3 \frac{\partial^2}{\partial x_j^2}k_{uu} \right) \\
=( \phi_1 k_{uu} + \phi_2 \frac{\partial}{\partial x_i}k_{uu} +\phi_3 \frac{\partial^2}{\partial x_i^2}k_{uu})( \phi_1 k_{uu} + \phi_2 \frac{\partial}{\partial x_j}k_{uu} +\phi_3 \frac{\partial^2}{\partial x_j^2}k_{uu})  $

```python
kff_sym = phi1**2*kuu_sym + phi1*phi2*sp.diff(kuu_sym, x_j) + phi1*phi3*sp.diff(kuu_sym, x_j, x_j) \
+ phi1*phi2*sp.diff(kuu_sym, x_i) + phi2**2*sp.diff(kuu_sym, x_i, x_j) + phi2*phi3* sp.diff(kuu_sym, x_i,x_j, x_j)\
+ phi1*phi3*sp.diff(kuu_sym, x_i, x_i)+phi2*phi3* sp.diff(kuu_sym, x_i,x_i, x_j)+phi3*phi3* sp.diff(kuu_sym,x_i, x_i,x_j, x_j)
kff_fn = sp.lambdify((x_i, x_j, theta, phi1,phi2,phi3), kff_sym, "numpy")
def kff(x, theta, phi1, phi2,phi3):
    k = np.zeros((x.size, x.size))
    for i in range(x.size):
        for j in range(x.size):
            k[i,j] = kff_fn(x[i], x[j], theta, phi1, phi2, phi3)
    return k
```

$k_{fu}(x_i,x_j;\theta,\phi) \\
= \mathcal{L}_{x_i}^\phi k_{uu}(x_i, x_j; \theta) \\
= \phi_1 k_{uu} + \phi_2 \frac{\partial}{\partial x_i}k_{uu}+\phi_3 \frac{\partial^2}{\partial x_i^2}k_{uu}$

```python
kfu_sym = phi1*kuu_sym + phi2*sp.diff(kuu_sym, x_i)+ phi3*sp.diff(kuu_sym, x_i, x_i)
kfu_fn = sp.lambdify((x_i, x_j, theta, phi1, phi2,phi3), kfu_sym, "numpy")
def kfu(x1, x2, theta, phi1, phi2,phi3):
    k = np.zeros((x1.size, x2.size))
    for i in range(x1.size):
        for j in range(x2.size):
            k[i,j] = kfu_fn(x1[i], x2[j], theta, phi1, phi2,phi3)
    return k
```

$k_{uf}(x_i,x_j;\theta,\phi) \\
= \mathcal{L}_{x_j}^\phi k_{uu}(x_i, x_j; \theta) $

```python
def kuf(x1, x2, theta, phi1, phi2,phi3):
    return kfu(x1,x2,theta,phi1, phi2,phi3).T
```

#### step 3: define negative log marginal likelihood

$K = \begin{bmatrix}
k_{uu}(X_u, X_u; \theta) + \sigma_u^2I & k_{uf}(X_u, X_f; \theta, \phi) \\
k_{fu}(X_f, X_u; \theta, \phi) & k_{ff}(X_f, X_f; \theta, \phi) + \sigma_f^2I
\end{bmatrix}$

For simplicity, assume $\sigma_u = \sigma_f$.

$\mathcal{NLML} = \frac{1}{2} \left[ log|K| + y^TK^{-1}y + Nlog(2\pi) \right]$

where $y = \begin{bmatrix}
y_u \\
y_f
\end{bmatrix}$

```python
def nlml(params, x1, x2, y1, y2, s):
    params = np.exp(params)
    K = np.block([
        [kuu(x1, params[0]) + s*np.identity(x1.size), kuf(x1, x2, params[0], params[1], params[2], params[3])],
        [kfu(x1, x2, params[0], params[1], params[2], params[3]), kff(x2, params[0], params[1], params[2], params[3])\
         + s*np.identity(x2.size)]
    ])
    y = np.concatenate((y1, y2))
    val = 0.5*(np.log(abs(np.linalg.det(K))) + np.mat(y) * np.linalg.inv(K) * np.mat(y).T)
    return val.item(0)
```

```python
nlml((1,1,3,0), x, x, y_u, y_f, 1e-6)

```

    7265159.607626721

#### step 4: optimise hyperparameters

```python
m = minimize(nlml, np.random.rand(4), args=(x, x, y_u, y_f, 1e-6), method="Nelder-Mead")
m
```

     final_simplex: (array([[ 2.56621862,  0.01559693,  1.08898748, -3.85438147],
           [ 2.74240554,  0.02733542,  1.08518974, -3.7434843 ],
           [ 2.8071121 ,  0.05173682,  1.07840169, -3.56123785],
           [ 2.69786742,  0.02460927,  1.08708013, -3.80933282],
           [ 2.83330948,  0.03676592,  1.08263675, -3.64158609]]), array([-126.96851715, -124.25024803, -123.46451503, -123.19715606,
           -120.92686881]))
               fun: -126.96851715158832
           message: 'Maximum number of function evaluations has been exceeded.'
              nfev: 800
               nit: 486
            status: 1
           success: False
                 x: array([ 2.56621862,  0.01559693,  1.08898748, -3.85438147])

```python

np.exp(m.x[1]) ## phi1

```

    1.0157191925001232

```python
np.exp(m.x[2]) ## phi2
```

    2.9712640818523517

```python
np.exp(m.x[3]) ## phi3
```

    0.0211867039390357

```python

```

```python

```


# Exploring interaction effects with logistic regression

After we train a logistic regression, we can look at the coefficients table to get an idea of how each individual predictor variable contributes to the outcome of the objective variable. For example, a large, positive coefficient indicates that the predictor greatly increases the log odds of the outcome. However, such an interpretation is under the assumption that the predictor variables act independently of each other. In many real-world cases, the value of one predictor will influence how another predictor contributes to the objective.

In this notebook, we'll explore how to identify and quantify these interaction effects. The procedure followed here is also explained in Chapter 3 of the book [Applied Logistic Regression](http://www.wiley.com/WileyCDA/WileyTitle/productCd-0470582472.html) by Hosmer, Lemeshow, and Sturdivant. We have adapted that workflow to use the [BigML Python bindings](http://bigml.readthedocs.io/en/latest/).

## The Data

The data we will use in this example is the Wine Quality dataset, which is available in the UCI Machine Learning Dataset Repository [here](https://archive.ics.uci.edu/ml/datasets/Wine+Quality). Each row in this dataset contains physicochemical measurements from a particular sample of wine, along with a preference score assigned to that sample by a taster. Some minimal preprocessing was performed on the data. Specifically:

- The original dataset is split into two CSV files for red (1599 instances), and white (4898 instances) wines. We concatenate the two CSV files to obtain a single dataset with 6497 instances. In addition we add a Categorical field called "type", which takes on the value 0 for white wines and 1 for red wines.
- The original quality field is integer-valued betwee 0-10. We created an additional binary-valued field called "good wine" which is 1 if the quality is greater than or equal to 5, and 0 otherwise.

Our analysis will be confined to two predictor variables:

- Citric Acid: The wine's citric acid content in $\textrm{g/dm}^3$. Numeric.
- Type: Red or White wine. Categorical 0 = White, 1 = Red.

We will model how these two predictors affect the probabiltiy of our objective:

- Good Wine: Whether the wine received a quality score >= 5. Categorical 0 = No, 1 = Yes.

## Setup

To begin, we will import the BigML Python bindings, matplotlib for plotting, and a couple utility functions for handling logistic regression coefficients. We'll also create a BigML project to keep our resources organized.

```python
from bigml.api import BigML
import matplotlib
from matplotlib import pyplot
from numpy import *
from myutils import print_coeff_table

%matplotlib inline
matplotlib.rcParams['font.size'] = 16

api = BigML()

prj = api.create_project({'name' : 'lr-interactions'})
prj = prj['resource']
```

## Dataset Creation

Now we will upload the CSV file containing the NHANES data to BigML, and create a dataset. We will then split the dataset 80/20 into training and test sets for later evaluation. Using a fixed seed ensures that we get complementary samples.

```python
src = api.create_source('winequality-all.csv', {'project' : prj})

origin_dataset = api.create_dataset(src, {'project': prj})

training_dataset = api.create_dataset(origin_dataset, {'project' : prj,
                                                       'name' : 'Wine Quality Training Set',
                                                       'sample_rate' : 0.8,
                                                       'seed' : 'example-seed'})
test_dataset = api.create_dataset(origin_dataset, {'project' : prj,
                                                   'name' : 'Wine Quality Test Set',
                                                   'sample_rate' : 0.8,
                                                   'out_of_bag' : True,
                                                   'seed' : 'example-seed'})
api.ok(training_dataset)
```

    True

## Model 1 - type only

Our first logistic regression model will assess what effect being a red wine (`type` = 1) has on receiving a good quality score (`good wine` = 1). We will code the categorical field `type` with dummy coding, with 0 as the dummy value. As a result we will obtain only one coefficient which gives the odds of `good wine=1` with `type=1`, compared to `type=0`.

```python
lr_opts = {'project' : prj, 'objective_field' : 'good wine',
           'field_codings' : [{'field' : 'type', 'coding' : 'dummy',
                               'dummy_class' : '0'}],
           'balance_fields' : False,
           'input_fields' : ['type'],
           'name' : 'Wine quality logistic regression - type only'}

lr1 = api.create_logistic_regression(training_dataset, lr_opts)
api.ok(lr1)
coeffs1,pvals1,confs1 = print_coeff_table(lr1)
```

    Predictor		 Coefficient	 p-value	 95% Confidence Interval
    --------------------------------------------------------------------------------
    Intercept		 0.66944	 0.00000	 [0.60299, 0.73590]
    type			 -0.53509	 0.00000	 [-0.66226, -0.40792]

The coefficient for `type` is significant ($p < 0.05$), and negative, meaning that `type` is negatively associated with `good wine`. The log odds of `good wine` is given by the following logit:

\begin{equation}
\ln(o) = \beta_0 + \beta_1 x_1
\end{equation}

Where $x_1$ is the variable corresponding to `type`, taking on values 0 or 1. We might ask how much less likely a good score is to be awarded, given that a red wine is being tested. We can compute the *odds ratio* like so:

\begin{equation}
OR = \frac{o_1}{o_1}\qquad \textrm{or} \qquad \ln(OR) = \ln(o_1) - \ln(o_0)
\end{equation}

```python
log_o1 = coeffs1[0] + coeffs1[1]*1
log_o0 = coeffs1[0] + coeffs1[1]*0
log_or = log_o1 - log_o0
odds_ratio = exp(log_or)
print(odds_ratio)
```

    0.585615359156

In other words, the odds for red wines to receive good quality scores is less than 60% as likely as the odds for white wines. We can also see this by computing the explicit probability of a good score, which is given by the logistic function:

\begin{equation}
P = \frac{1}{1 + \exp(-(\beta_0 + \beta_1 x_1))}
\end{equation}

In terms of probabilities the odds ratio is defined as:

\begin{equation}
OR = \frac{\frac{P_1}{(1 - P_1)}}{\frac{P_0}{(1 - P_0)}}
\end{equation}

```python
p1 = 1 / (1 + exp(-log_o1))
print(p1)
```

    0.53353777006

```python
p0 = 1 / (1 + exp(-log_o0))
print(p0)
```

    0.661378852184

```python
odds_ratio_by_probs = (p1/(1-p1)) / (p0/(1-p0))
print(odds_ratio_by_probs)
```

    0.585615359156

## Model 2 - type + citric acid

We will now add a second predictor, citric acid, to our logistic regression model:

```python
lr_opts['input_fields'].append('citric acid')
lr_opts['name'] = 'Wine Quality Logistic Regression - type + citric acid'
lr2 = api.create_logistic_regression(training_dataset,lr_opts)
api.ok(lr2)
coeffs2,pvals2,confs2 = print_coeff_table(lr2)
```

    Predictor		 Coefficient	 p-value	 95% Confidence Interval
    --------------------------------------------------------------------------------
    Intercept		 0.44427	 0.00000	 [0.29691, 0.59163]
    type			 -0.49400	 0.00000	 [-0.62365, -0.36435]
    citric acid		 0.67764	 0.00083	 [0.28042, 1.07485]

Again, the log-odds for `good score` is given by a logit:

\begin{equation}
\ln(o) = \beta_0 + \beta_1 x_1 + \beta_2 x_2
\end{equation}

We now have a second coefficient corresponding to the predictor variable $x_2$: the concentration of citric acid in the wine. Let's look at the log-odds as a function of citric acid, for both red and white wines:

```python
# evaluate the logit function at type = 0 and type = 1 for a range of citric acid values
xvals = linspace(0.0,1.66)
log_odds_0 = coeffs2[0] + coeffs2[1]*0 + coeffs2[2]*xvals
log_odds_1 = coeffs2[0] + coeffs2[1]*1 + coeffs2[2]*xvals

# plot the results
pyplot.figure(figsize=(10,6))
pyplot.plot(xvals,log_odds_0)
pyplot.plot(xvals,log_odds_1)
pyplot.xlabel('Citric Acid (g/dm^3)')
pyplot.ylabel('log odds for good score')
pyplot.legend(['type = 0', 'type = 1'],loc='upper left')
```

Although different combinations of `type` and `citric acid` will result in different values for the log-odds, this plot shows that the two predictors in this model are *independent* of each other. Specifically, the log-odds is a linear function of citric acid, and the slope of that function is the same for both values of `type`. This shows that the effect of `citric acid` on the log-odds is independent of `type`. At the same time, for any value of `citric acid`, the vertical distance between the two lines is the same, showing that the effect of `type` on the log-odds is independent of `citric acid`.

There are many cases when this independence assumption does not faithfully model the behavior of the data. To see whether that might be the case here, we will see how much the value of the `type` coefficient changed through the addition of `citric acid` as a predictor:

```python
delta_b1 = abs((coeffs2[1] - coeffs1[1]) / coeffs2[1])
print(delta_b1)
```

    0.08318532814552569

With a change of nearly 10%, we are justified in believing that there may be *interaction effects* between the two predictor variables. Our third logistic regression model will investigate this hypothesis.

## Model 3 - type + citric acid with Interaction

To model interaction effects, we will extend our training dataset with an additional field containing an interaction term. Interaction terms are simply the product of the values of two predictor variables. We apply the same extension to the test dataset for later evaluation.

```python
flatline_str = '(* (f "citric acid") (integer (f type)))'
training_ds_extended = api.create_dataset(training_dataset, {'new_fields':[{'field' : flatline_str,
                                                             'name' : 'citric acid*type'}]})
test_ds_extended = api.create_dataset(test_dataset, {'new_fields':[{'field' : flatline_str,
                                                                     'name' : 'citric acid*type'}]})
api.ok(training_ds_extended)
```

    True

Now, we will train a logistic regression with the two predictors `type` and `citric acid`, and their interaction term `citric acid*type`.

```python
lr_opts['input_fields'].append('citric acid*type')
lr_opts['name'] = 'Wine Quality Logistic Regression: type + citric acid + interaction'
lr3 = api.create_logistic_regression(training_ds_extended,lr_opts)
api.ok(lr3)

coeffs3,pvals3,confs3 = print_coeff_table(lr3)
```

    Predictor		 Coefficient	 p-value	 95% Confidence Interval
    --------------------------------------------------------------------------------
    Intercept		 0.64898	 0.00000	 [0.45381, 0.84415]
    type			 -0.88613	 0.00000	 [-1.15713, -0.61513]
    citric acid		 0.06009	 0.83021	 [-0.48915, 0.60933]
    citric acid*type	 1.31815	 0.00111	 [0.52613, 2.11018]

```python
log_odds_0 = coeffs3[0] + coeffs3[1]*0 + coeffs3[2]*xvals + coeffs3[3]*0*xvals
log_odds_1 = coeffs3[0] + coeffs3[1]*1 + coeffs3[2]*xvals + coeffs3[3]*1*xvals

# plot the results
pyplot.figure(figsize=(10,6))
pyplot.plot(xvals,log_odds_0)
pyplot.plot(xvals,log_odds_1)
pyplot.xlabel('Citric Acid (g/dm^3)')
pyplot.ylabel('log odds for good score')
pyplot.legend(['type = 0', 'type = 1'],loc='upper left')
```

The graph here is drastically different from the one for the independent logistic regression model. We can see that for both types of wine, the odds for `good wine` continue to be positively correlated with `citric acid`, but the impact is much greater for red wines compared to white wines. In fact, if we look at the p-values in the coefficient table, we see that the interaction term between `type` and `citric acid` is significant, but the one for `citric acid` alone is no longer significant. From this, we can conclude that `citric acid` is a significant predictor of quality only for red wines.

## Evaluations

Now we will take our three logistic regression models, evaluate them against our 20% holdout test set, and plot their average F-measures.

```python
eval1 = api.create_evaluation(lr1,test_ds_extended)
eval2 = api.create_evaluation(lr2,test_ds_extended)
eval3 = api.create_evaluation(lr3,test_ds_extended)

api.ok(eval1)
api.ok(eval2)
api.ok(eval3)

f1,f2,f3 = [e['object']['result']['model']['average_f_measure'] for e in [eval1,eval2,eval3]]

pyplot.figure(figsize=(10,6))
pyplot.bar([1,2,3],[f1,f2,f3],align='center')
pyplot.xticks([1,2,3],['Type', 'Type+CA', 'Type+CA+Interaction'])
pyplot.xlabel('Predictors')
pyplot.ylabel('Average F-Measure')
```

Our results show that the model which includes the interaction term achieves the best classification performance. In conclusion, not only does identifying interactions between predictor variables give a deeper insight into the behavior of your data, but it can produce better models for predictive applications too.

```python
# uncomment and run this cell to clean up resources generated by this notebook

#for p in api.list_projects('name=lr-interactions')['objects']:
#        api.delete_project(p)
```