# Counterfactual value estimation using outcome imputation

## Introduction
The goal in uplift modeling is usually to predict the best treatment condition for an individual. Most of the time, the best treatment condition is assumed to be the one that has the highest probability of some "conversion event" such as the individual's purchasing a product. This is the traditional approach in which the goal is to maximize conversion.

However, if the goal of uplift modeling is to maximize value, then it is not safe to assume that the best treatment group is the one with the highest expected conversion. For example, it might be that the payoff from conversion is not sufficient to offset the cost of the treatment, or it might be that the treatment targets individuals who would convert anyway [(Li and Pearl 2019)](https://ftp.cs.ucla.edu/pub/stat_ser/r488.pdf). Therefore, it is often important to conduct some kind of value optimization together with uplift modeling, in order to determine the treatment group with the best value, not just the best lift.

The Causal ML package includes the CounterfactualValueEstimator class to conduct simple imputation-based value optimization. This notebook demonstrates the use of CounterfactualValueEstimator to determine the best treatment group when the costs of treatments are taken into account. We consider two kinds of costs:

* **Conversion costs** are those that we must endure if an individual who is in the treatment group converts. A typical example would be the cost of a promotional voucher.
* **Impression costs** are those that we need to pay for each individual in the treatment group irrespective of whether they convert. A typical example would be the cost associated with sending an SMS or email.

The proposed method takes two inputs: the CATE estimate $\hat{\tau}$ learned by any suitable method, and the predicted outcome for an individual learned by what we call the conversion probability model that estimates the conditional probability of conversion $P(Y=1 \mid X=x, W=x)$ where $W$ is the treatment group indicator. That is, the model estimates the probability of conversion for each individual using their observed pre-treatment features $X$. The output of this model is then combined with the predicted CATE in order to impute the expected conversion probability for each individual under \textit{each treatment condition} as follows:

\begin{equation}
\hat{Y}_i^0 =
    \begin{cases}
    \hat{m}(X_i, W_i) & \text{for } W_i = 0 \\
    \hat{m}(X_i, W_i) - \hat{\tau}_t(X_i) & \text{for } W_i = t \\
    \end{cases}
\end{equation}

\begin{equation}
\hat{Y}_i^t =
    \begin{cases}
    \hat{m}(X_i, W_i) + \hat{\tau}_t(X_i) & \text{for } W_i = 0 \\
    \hat{m}(X_i, W_i) & \text{for } W_i = t \\
    \end{cases}
\end{equation}

The fact that we impute the conversion probability under each experimental condition--the actual as well as the counterfactual--gives our method its name. Using the estimated conversion probabilities, we then compute the expected payoff under each treatment condition while taking into account the value of conversion and the conversion and impression costs associated with each treatment, as follows (see [Zhao and Harinen (2019)](https://arxiv.org/abs/1908.05372) for more details):

\begin{equation}
    \mathbb{E}[(v - cc_t)Y_t - ic_t]
\end{equation}

where $cc_t$ and $ic_t$ are the conversion costs and impression costs, respectively.

```python
import numpy as np
import pandas as pd

from sklearn.model_selection import train_test_split

import xgboost as xgb

from causalml.dataset import make_uplift_classification
from causalml.inference.meta import BaseTClassifier
from causalml.optimize import CounterfactualValueEstimator
from causalml.optimize import get_treatment_costs
from causalml.optimize import get_actual_value

import matplotlib.pyplot as plt
import seaborn as sns
sns.set_style('whitegrid')

%matplotlib inline
```

    The sklearn.utils.testing module is  deprecated in version 0.22 and will be removed in version 0.24. The corresponding classes / functions should instead be imported from sklearn.utils. Anything that cannot be imported from sklearn.utils is now part of the private API.
    sklearn.tree._criterion.RegressionCriterion size changed, may indicate binary incompatibility. Expected 168 from C header, got 360 from PyObject
    sklearn.tree._criterion.Criterion size changed, may indicate binary incompatibility. Expected 160 from C header, got 352 from PyObject
    sklearn.tree._criterion.ClassificationCriterion size changed, may indicate binary incompatibility. Expected 176 from C header, got 368 from PyObject

## Data generation

First, we simulate some heterogeneous treatment data using the built-in function.

```python
df, X_names = make_uplift_classification(
    n_samples=5000, treatment_name=['control', 'treatment1', 'treatment2'])
```

In this example, we assume there are no costs associated with assigning units into the control group, and that for the two treatment groups the conversion cost are \\$2.5 and \\$5, respectively. We assume the impression costs to be zero for one of the treatments and \\$0.02 for the other. We also specify the payoff, which we here assume to be the same for everyone, \\$20. However, these values could vary from individual to individual.

```python
# Put costs into dicts
conversion_cost_dict = {'control': 0, 'treatment1': 2.5, 'treatment2': 5}
impression_cost_dict = {'control': 0, 'treatment1': 0, 'treatment2': 0.02}

# Use a helper function to put treatment costs to array
cc_array, ic_array, conditions = get_treatment_costs(treatment=df['treatment_group_key'],
                                                     control_name='control',
                                                     cc_dict=conversion_cost_dict,
                                                     ic_dict=impression_cost_dict)

# Put the conversion value into an array
conversion_value_array = np.full(df.shape[0], 20)
```

Next we calculate the value of actually having an individual in their actual treatment group using the equation for expected value under a treatment, ie:

\begin{equation}
    \mathbb{E}[(v - cc_t)Y_t - ic_t]
\end{equation}

```python
# Use a helper function to obtain the value of actual treatment
actual_value = get_actual_value(treatment=df['treatment_group_key'],
                                observed_outcome=df['conversion'],
                                conversion_value=conversion_value_array,
                                conditions=conditions,
                                conversion_cost=cc_array,
                                impression_cost=ic_array)
```

```python
plt.hist(actual_value)
plt.show()
```

## Model evaluation

A common problem in the uplift modeling literature is that of evaluating the quality of the treatment recommendations produced by a model. The evaluation of uplift models is tricky because we do not observe treatment effects at an individual level directly in non-simulated data, so it is not possible to use standard model evaluation metrics such as mean squared error. Consequently, various authors have proposed various ways to work around this issue. For example, [Schuler et al (2018)](https://arxiv.org/abs/1804.05146) identify seven different evaluation strategies used in the literature.

Below, we use the approach of model evaluation put forward by [Kaepelner et al (2014)](https://arxiv.org/abs/1404.7844). The idea in this method is to evaluate the improvement we would gain if we targeted some as-yet untreated future population by using the recommendations produced by a particular model. To do so, we split the data into disjoint training and testing sets, and train our model on the training data. We then use the model to predict the best treatment group for units in the testing data, which in a simple two-arm trial is either treatment or control. In order to estimate the outcome for the future population if the model were to be used, we then select a subset of the testing data based on whether their observed treatment allocation happens to be the same as the one recommended by the model. This population is called "lucky".

| Predicted best treatment | Actual treatment | Lucky |
|--------------------------|------------------|-------|
| Control | Control | Yes |
| Control | Treatment | No |
| Treatment | Treatment | Yes |
| Treatment | Control | No |

The average outcome for the "lucky" population can be taken to represent what the outcome would be for a future untreated population if we were to use the uplift model in question to allocate treatments. Recall that in all of the experiments the treatments are assumed to have been allocated randomly across the total population, so there should be no selection bias. The average outcome under a given model can then be compared with alternative treatment allocation strategies. As [Kaepelner et al (2014)](https://arxiv.org/abs/1404.7844) point out, two common strategies are random allocation and "best treatment" allocation. To estimate what the outcome for a future population would be under random allocation, we can simply look at the sample mean across the total test population. To estimate the same for the "best treatment" assignment, we can look at those units in the test set whose observed treatment assignment corresponds to the treatment group with the best average treatment effect. These alternative targeting strategies are interesting because they are a common practice in industry applications and elsewhere.

### Performance against benchmarks

In this section, we compare four different targeting strategies:

* Random treatment allocation under which all units in the testing set are randomly assigned to treatments
* The "best treatment" allocation under which all units in the testing set are assigned to the treatment with the best conversion in the training set
* Allocation under an uplift model in which all units in the testing set are assigned to the treatment which is predicted to have the highest conversion rate according to an uplift model trained on the training set
* Allocation under the counterfactual value estimator model in which all units are assigned to the treatment group with the best predicted payoff

```python
df_train, df_test = train_test_split(df)
train_idx = df_train.index
test_idx = df_test.index
```

```python
# Calculate the benchmark value according to the random allocation
# and best treatment schemes
random_allocation_value = actual_value.loc[test_idx].mean()

best_ate = df_train.groupby(
    'treatment_group_key')['conversion'].mean().idxmax()

actual_is_best_ate = df_test['treatment_group_key'] == best_ate

best_ate_value = actual_value.loc[test_idx][actual_is_best_ate].mean()
```

```python
# Calculate the value under an uplift model
tm = BaseTClassifier(control_learner=xgb.XGBClassifier(),
                     treatment_learner=xgb.XGBClassifier(),
                     control_name='control')

tm.fit(df_train[X_names].values,
       df_train['treatment_group_key'],
       df_train['conversion'])

tm_pred = tm.predict(df_test[X_names].values)

pred_df = pd.DataFrame(tm_pred, columns=tm._classes)
tm_best = pred_df.idxmax(axis=1)
actual_is_tm_best = df_test['treatment_group_key'] == tm_best.ravel()
tm_value = actual_value.loc[test_idx][actual_is_tm_best].mean()
```

```python
# Estimate the conditional mean model; this is a pure curve
# fitting exercise
proba_model = xgb.XGBClassifier()

W_dummies = pd.get_dummies(df['treatment_group_key'])
XW = np.c_[df[X_names], W_dummies]

proba_model.fit(XW[train_idx], df_train['conversion'])
y_proba = proba_model.predict_proba(XW[test_idx])[:, 1]
```

```python
# Run the counterfactual calculation with TwoModel prediction
cve = CounterfactualValueEstimator(treatment=df_test['treatment_group_key'],
                                   control_name='control',
                                   treatment_names=conditions[1:],
                                   y_proba=y_proba,
                                   cate=tm_pred,
                                   value=conversion_value_array[test_idx],
                                   conversion_cost=cc_array[test_idx],
                                   impression_cost=ic_array[test_idx])

cve_best_idx = cve.predict_best()
cve_best = [conditions[idx] for idx in cve_best_idx]
actual_is_cve_best = df.loc[test_idx, 'treatment_group_key'] == cve_best
cve_value = actual_value.loc[test_idx][actual_is_cve_best].mean()
```

```python
labels = [
    'Random allocation',
    'Best treatment',
    'T-Learner',
    'CounterfactualValueEstimator'
]

values = [
    random_allocation_value,
    best_ate_value,
    tm_value,
    cve_value
]

plt.bar(labels, values)
plt.ylabel('Mean actual value in testing set')
plt.xticks(rotation=45)
plt.show()
```

Here, only CounterfactualValueEstimator improves upon random targeting. The "best treatment" and T-Learner approaches likely perform worse because they recommend costly treatments to individuals who would convert anyway.


```python
#!/bin/env python

"""
This solves for the terms common scalar phase, tec, and complex gain using MH algorithm.
author: Joshua Albert
albert@strw.leidenuniv.nl
"""

import numpy as np
import pylab as plt
from dask import delayed

def phase_unwrapp1d(theta):
    '''the difference between two timesteps is unaliased by assumption so theta_i+1 - theta_i < pi.
    So Wrap(theta_i+1 - theta_i) is the real gradient and we can integrate them'''
    def wrap(phi):
        res = 1j*phi
        np.exp(res,out=res)
        return np.atleast_1d(np.angle(res))
    grad = wrap(theta[1:] - theta[:-1])
    unwrapped_theta = np.zeros(len(theta),dtype=np.double)
    np.cumsum(grad,out=unwrapped_theta[1:])
    unwrapped_theta += theta[0]
    return unwrapped_theta

TECU = 10e16

def vertex(x1,x2,x3,y1,y2,y3):
    '''Given three pairs of (x,y) points return the vertex of the
         parabola passing through the points. Vectorized and common expression reduced.'''
    #Define a sequence of sub expressions to reduce redundant flops
    x0 = 1/x2
    x4 = x1 - x2
    x5 = 1/x4
    x6 = x1**2
    x7 = 1/x6
    x8 = x2**2
    x9 = -x7*x8 + 1
    x10 = x0*x1*x5*x9
    x11 = 1/x1
    x12 = x3**2
    x13 = x11*x12
    x14 = 1/(x0*x13 - x0*x3 - x11*x3 + 1)
    x15 = x14*y3
    x16 = x10*x15
    x17 = x0*x5
    x18 = -x13 + x3
    x19 = y2*(x1*x17 + x14*x18*x6*x9/(x4**2*x8))
    x20 = x2*x5
    x21 = x11*x20
    x22 = x14*(-x12*x7 + x18*x21)
    x23 = y1*(-x10*x22 - x21)
    x24 = x16/2 - x19/2 - x23/2
    x25 = -x17*x9 + x7
    x26 = x0*x1*x14*x18*x5
    x27 = 1/(-x15*x25 + y1*(x20*x7 - x22*x25 + x7) + y2*(-x17 + x25*x26))
    x28 = x24*x27
    return x28,x15 + x22*y1 + x24**2*x27 - x26*y2 + x28*(-x16 + x19 + x23)

def clock_tec_solveMH(obs_phase, freqs, times, m0, cov, Cd_error, Ct_ratio, plot = False):
    '''Solves for the terms phase(CS,TEC,delay) = CS + e^2/(4pi ep0 me c) * TEC/nu + 2pi*nu*delay

    Assumes phase is in units of radians, freqs in is units of Hz,
    and times is in units of seconds with arbitrary offset

    obs_phase is shape (num_freqs, num_times)'''

    binning = 50
    convergence = binning**2 * 3
    def calc_phase(m, freqs):
        phase = np.multiply.outer(np.ones(len(freqs)),m[:,2]) + 8.44797256e-7*TECU * np.multiply.outer(1./freqs,m[:,1]) + 2.*np.pi*np.multiply.outer(freqs,m[:,0])
        return phase

    def neglogL(obs_phase,phase,CdCt):
        L2 = obs_phase - phase
        L2 *= L2
        L2 /= (CdCt+1e-15)
        return np.sum(L2,axis=0)/2.

    def sample_prior(last, cov):
        """Last is tau,tec,cs in matrix of size [len(times),3], return similar shaped next point"""
        return last + np.random.multivariate_normal(mean = [0,0,0], cov=cov,size = last.shape[0])
    cs = m0[:,2]
    tec = m0[:,1]
    tau = m0[:,0]
    print("Initial CS: {}".format(cs))
    print("Initial TEC: {}".format(tec))
    print("Initial delay: {}".format(tau))
    m = m0.copy()
#     if plot:
#         plt.plot(times,cs0,label="CS0")
#         plt.plot(times,tec0,label="TEC0")
#         plt.plot(times,delay0,label="delay0")
#         plt.legend(frameon=False)
#         plt.show()
    Ct = (Ct_ratio*np.abs(obs_phase))**2
    Cd = (Cd_error*np.pi/180.)**2
    CdCt = Cd+Ct
    Si = neglogL(obs_phase,calc_phase(m,freqs),CdCt)
    print("Initial Si: {}".format(Si))
    max_iter = 100*convergence
    posterior = np.zeros([convergence,len(times),3],dtype=np.double)
    multiplicity = np.zeros([convergence,len(times)],dtype=np.double)
    posterior[0,:,:] = m
    minS = Si
    minSol = m.copy()
    accepted = np.ones(len(times),dtype=np.int)
    cov_prior = np.diag([1e-10, 1e-6,0.5])**2 + cov
    iter = 1
    while np.max(accepted) < convergence and iter < max_iter:
        #sample
        last = np.array([posterior[accepted[i] - 1,i,:] for i in range(len(times))])
        m_j = sample_prior(last,cov_prior)

        Sj = neglogL(obs_phase,calc_phase(m_j,freqs),CdCt)
        Lj = np.exp(-Sj)

        accept_mask = np.bitwise_or(Sj < Si, np.log(np.random.uniform(size=len(Sj))) < Si - Sj)
        #print(accept_mask)
        Si[accept_mask] = Sj[accept_mask]
        for i in range(len(times)):
            if accept_mask[i]:
                posterior[accepted[i],i,:] = m_j[i,:]
                multiplicity[accepted[i],i] += 1
                accepted[i] += 1
            else:
                multiplicity[accepted[i]-1,i] += 1

        if np.any(accept_mask):
            #print(m_j)
            #print("{} accepted".format(np.sum(accept_mask)))
            pass

        maxL_mask = Sj < minS
        minSol[maxL_mask,:] = m_j[maxL_mask]
        minS[maxL_mask] = Sj[maxL_mask]
        iter += 1
    if iter != max_iter:
        print("Converged in {} steps with mean acceptance rate of {}".format(iter,np.mean(accepted)/iter))
    posteriors = []
    multiplicities = []
    means = []
    stds = []
    maxLs = []
    for i in range(len(times)):
        posteriors.append(posterior[:accepted[i],i,:])
        multiplicities.append(multiplicity[:accepted[i],i])
        means.append(np.sum(posteriors[i].T*multiplicities[i],axis=1)/np.sum(multiplicities[i]))
        stds.append(np.sqrt(np.sum(posteriors[i].T**2*multiplicities[i],axis=1)/np.sum(multiplicities[i]) - means[i]**2))
        maxLs.append(minSol[i,:])
        print ("Sol {}, (Gaussian) sol is {} +- {}".format(i, means[i],stds[i]))
        print("    maxL sol is {}".format(maxLs[i]))
    if plot:
        plt.hist(posteriors[0][:,0],weights = multiplicities[0],label='tau')
        plt.legend(frameon=False)
        plt.show()
        plt.hist(posteriors[0][:,1],weights = multiplicities[0],label='tec')
        plt.legend(frameon=False)
        plt.show()
        plt.hist(posteriors[0][:,2],weights = multiplicities[0],label='cs')
        plt.legend(frameon=False)
        plt.show()
    return maxLs

@delayed
def clock_tec_solve_dask(obs_phase, freqs, m0, cov, Cd_error, Ct_ratio, plot=False):
    '''Solves for the terms phase(CS,TEC,delay) = CS + e^2/(4pi ep0 me c) * TEC/nu + 2pi*nu*delay

    Assumes phase is in units of radians, freqs in is units of Hz

    obs_phase is shape (num_freqs)'''

    binning = 50
    convergence = binning**2 * 3
    def calc_phase(m, freqs):
        tau,tec,cs = m[:]
        phase = freqs*(tau * np.pi * 2.)
        phase += (8.44797256e-7 * TECU * tec)/freqs
        phase += cs
        return phase

    def neglogL(obs_phase,phase,CdCt):
        L2 = obs_phase - phase
        L2 *= L2
        L2 /= (CdCt+1e-15)
        return np.sum(L2)/2.

    def sample_prior(last, cov):
        """Last is tau,tec,cs in matrix of size [len(times),3], return similar shaped next point"""
        return last + np.random.multivariate_normal(mean = [0,0,0], cov=cov)
        #m0 = last[0] + np.random.uniform(low = -np.sqrt(cov[0,0])/2., high = np.sqrt(cov[0,0])/2.)
        #m1 = last[1] + np.random.uniform(low = -np.sqrt(cov[1,1])/2., high = np.sqrt(cov[1,1])/2.)
        #m2 = last[2] + np.random.uniform(low = -np.sqrt(cov[2,2])/2., high = np.sqrt(cov[2,2])/2.)
        #return np.array([m0,m1,m2])
    cs = m0[2]
    tec = m0[1]
    tau = m0[0]
    print("Initial CS: {}".format(cs))
    print("Initial TEC: {}".format(tec))
    print("Initial delay: {}".format(tau))
    m = m0.copy()

    Ct = (Ct_ratio*np.abs(obs_phase))**2
    Cd = (Cd_error*np.pi/180.)**2
    CdCt = Cd+Ct
    Si = neglogL(obs_phase,calc_phase(m,freqs),CdCt)
    print("Initial Si: {}".format(Si))
    max_iter = 100*convergence
    posterior = np.zeros([convergence,3],dtype=np.double)
    multiplicity = np.zeros([convergence],dtype=np.double)
    posterior[0,:] = m
    minS = Si
    minSol = m.copy()
    accepted = 1
    cov_prior = np.diag([1e-9, 1e-4,1e-1])**2 + cov
    iter = 1
    while accepted < convergence and iter < max_iter:
        #sample
        #m_j = sample_prior(posterior[accepted-1],cov_prior)
        m_j = sample_prior(minSol,cov_prior)
        #m_j = sample_prior(m0,cov_prior)
        Sj = neglogL(obs_phase,calc_phase(m_j,freqs),CdCt)
        Lj = np.exp(-Sj)
        if Sj < Si or np.log(np.random.uniform()) < Si - Sj:
            Si = Sj
            posterior[accepted,:] = m_j
            multiplicity[accepted] += 1
            accepted += 1
        else:
            multiplicity[accepted-1] += 1
        if Sj < minS:
            minSol = m_j
            minS = Sj
        iter += 1
    if iter != max_iter:
        print("Converged in {} steps with mean acceptance rate of {}".format(iter,np.mean(accepted)/iter))
    posterior = posterior[:accepted,:]
    multiplicity = multiplicity[:accepted,]
    mean = np.sum(posterior.T*multiplicity,axis=1)/np.sum(multiplicity)
    std = np.sqrt(np.sum(posterior.T**2*multiplicity,axis=1)/np.sum(multiplicity) - mean**2)
    maxLsol = minSol
    print ("(Gaussian) sol is {} +- {}".format(mean,std))
    print("    maxL sol is {}".format(maxLsol))
    if plot:
        plt.hist(posteriors[0][:,0],weights = multiplicities[0],label='tau')
        plt.legend(frameon=False)
        plt.show()
        plt.hist(posteriors[0][:,1],weights = multiplicities[0],label='tec')
        plt.legend(frameon=False)
        plt.show()
        plt.hist(posteriors[0][:,2],weights = multiplicities[0],label='cs')
        plt.legend(frameon=False)
        plt.show()
    return maxLsol

def clock_test_solve_both(obs_phase, freqs, times, m0, cov, Cd_error, Ct_ratio):
    m,cov = least_squares_solve(obs_phase, freqs, times,Cd_error,Ct_ratio)
    m = clock_tec_solveMH(phase, freqs, times, m, np.mean(cov,axis=0), Cd_error, Ct_ratio,plot=True)
    return m

def test_clock_tec_solveMH():
    times = np.arange(2)
    freqs = np.linspace(110e6,170e6,100)
    cs = times*0.01
    tec = np.random.uniform(size=len(times))*0.01
    delay = np.ones(len(times)) * 1e-9# 10ns
    phase = np.multiply.outer(np.ones(len(freqs)),cs) + 8.44797256e-7*TECU*np.multiply.outer(1./freqs,tec) + 2.*np.pi*np.multiply.outer(freqs,delay)
    phase += 10.*np.pi/180.*np.random.normal(size=[len(freqs),len(times)])
    plt.imshow(phase,origin='lower',extent=(times[0],times[-1],freqs[0],freqs[-1]),aspect='auto')
    plt.colorbar()
    plt.xlabel('times (s)')
    plt.ylabel('freqs (Hz)')
    plt.show()
    clock_tec_solveMH(phase, freqs, times, plot=True)

def test_clock_tec_solve_l1l2():
    times = np.arange(2)
    freqs = np.linspace(110e6,170e6,100)
    cs = times*0.00001
    tec = np.random.uniform(size=len(times))*0.01
    phase = np.multiply.outer(np.ones(len(freqs)),cs) + 8.44797256e-7*TECU*np.multiply.outer(1./freqs,tec)
    phase += 10.*np.pi/180.*np.random.normal(size=[len(freqs),len(times)])
    plt.imshow(phase,origin='lower',extent=(times[0],times[-1],freqs[0],freqs[-1]),aspect='auto')
    plt.colorbar()
    plt.xlabel('times (s)')
    plt.ylabel('freqs (Hz)')
    plt.show()
    m0,cov = least_squares_solve(phase, freqs, times,10,Ct_ratio=0.01)
    print(m0)
    l1data_l2model_solve(phase,freqs,times,10,Ct_ratio=0.01,m0=m0)

def test_clock_tec_solve_error():
    import pylab as plt
    times = np.arange(100)
    Cd_errors = np.linspace(1,100,100)

    f,(ax1,ax2,ax3) = plt.subplots(3,1)
    ax1.set_yscale('log')
    ax2.set_yscale('log')
    ax3.set_yscale('log')
    for num_freq in [10,100,1000,10000]:
        freqs = np.linspace(110e6,170e6,num_freq)
        sol_acc = []
        for Cd_error in Cd_errors:
            cs = times
            tec = np.random.uniform(size=len(times))*0.01
            delay = np.ones(len(times)) * 1e-9# 10ns
            phase = np.multiply.outer(np.ones(len(freqs)),cs) + 8.44797256e-7*TECU*np.multiply.outer(1./freqs,tec) + 2.*np.pi*np.multiply.outer(freqs,delay)
            phase += Cd_error*np.pi/180.*np.random.normal(size=[len(freqs),len(times)])
            #plt.imshow(phase,origin='lower',extent=(times[0],times[-1],freqs[0],freqs[-1]),aspect='auto')
            #plt.colorbar()
            #plt.xlabel('times (s)')
            #plt.ylabel('freqs (Hz)')
            #plt.show()
            m,cov = least_squares_solve(phase, freqs, times,Cd_error,Ct_ratio=0.01)
            m_exact = np.array([delay,tec,cs]).T
            sol_acc.append(np.mean(np.abs(m - m_exact),axis=0))
        sol_acc_ = np.array(sol_acc)
        ax1.plot(Cd_errors,sol_acc_[:,0])
        #plt.show()
        ax2.plot(Cd_errors,sol_acc_[:,1])
        #plt.show()
        ax3.plot(Cd_errors,sol_acc_[:,2])
    plt.show()

def test_clock_tec_solve():
    import pylab as plt
    times = np.arange(2)
    freqs = np.linspace(110e6,170e6,1000)

    cs = np.array([1,1])*0
    tec = np.array([0.1,0.2])
    delay = np.ones(len(times)) * 2e-9# 10ns
    phase = np.multiply.outer(np.ones(len(freqs)),cs) + 8.44797256e-7*TECU*np.multiply.outer(1./freqs,tec)# + 2.*np.pi*np.multiply.outer(freqs,delay)
    phase += 5*np.pi/180.*np.random.normal(size=[len(freqs),len(times)])
    #plt.imshow(phase,origin='lower',extent=(times[0],times[-1],freqs[0],freqs[-1]),aspect='auto')
    #plt.colorbar()
    #plt.xlabel('times (s)')
    #plt.ylabel('freqs (Hz)')
    #plt.show()
    m,cov = least_squares_solve(phase, freqs, times,5,Ct_ratio=0.01)
    print(m)
    #m_exact = np.array([delay,tec,cs]).T
    #clock_tec_solveMH(phase, freqs, times, m, np.max(cov,axis=0), 5, 0.01, plot = True)

def test_clock_tec_solve_dask():
    np.random.seed(1234)
    import pylab as plt
    times = np.arange(2)
    freqs = np.linspace(110e6,170e6,1000)

    cs = np.array([1,1])
    tec = np.array([0.1,0.2])
    delay = np.ones(len(times)) * 2e-9# 10ns
    phase = np.multiply.outer(np.ones(len(freqs)),cs) + 8.44797256e-7*TECU*np.multiply.outer(1./freqs,tec) + 2.*np.pi*np.multiply.outer(freqs,delay)
    phase += 15*np.pi/180.*np.random.normal(size=[len(freqs),len(times)])
    #plt.imshow(phase,origin='lower',extent=(times[0],times[-1],freqs[0],freqs[-1]),aspect='auto')
    #plt.colorbar()
    #plt.xlabel('times (s)')
    #plt.ylabel('freqs (Hz)')
    #plt.show()
    m,cov = least_squares_solve(phase, freqs, times,15,Ct_ratio=0.01)
    m_exact = np.array([delay,tec,cs]).T
    import dask.array as da
    solsMH = [da.from_delayed(clock_tec_solve_dask(phase[:,i],freqs,m[i,:], cov[i,:,:],15,0.01),shape=(3,),dtype=np.double) for i in range(len(times))]

    sol_stacked = da.stack(solsMH, axis = 0)
    sol = sol_stacked.compute()
    print(sol)

if __name__ == '__main__':
    #test_clock_tec_solve()
    #a b ctest_clock_tec_solve()
    #test_clock_tec_solve_dask()
    test_clock_tec_solve_l1l2()

```

```python
def derive_solution():
    from sympy import symbols, Matrix, cse, cos, sin, Abs, Rational,acos,asin

    cs,K,tec,nu,phase,sigma_phase,alpha,beta,tec_p,cs_p,sigma_tec,sigma_cs = symbols('cs K tec nu phase sigma_phase alpha beta tec_p cs_p sigma_tec sigma_cs', real=True)
    g = K*tec/nu + cs*alpha

    L = Abs(g - phase)/sigma_phase + beta*((tec - tec_p)**Rational(2)/sigma_tec**Rational(2)/Rational(2) + (cs -  cs_p)**Rational(2)/sigma_cs**Rational(2)/Rational(2))

    req,res = cse(L,optimizations='basic')
    for line in req:
        print("{} = {}".format(line[0],line[1]).replace("Abs","np.abs").replace("cos","np.cos").replace("sin","np.sin").replace("sign","np.sign"))
    print("{}".format(res[0]).replace("Abs","np.abs").replace("cos","np.cos").replace("sin","np.sin").replace("sign","np.sign"))
    print()

    grad = Matrix([sigma_tec**Rational(2)*L.diff(tec), sigma_cs**Rational(2)*L.diff(cs)])
    req,res = cse(grad,optimizations='basic')
    for line in req:
        print("{} = {}".format(line[0],line[1]).replace("Abs","np.abs").replace("cos","np.cos").replace("sin","np.sin").replace("sign","np.sign"))
    print("{}".format(res[0]).replace("Abs","np.abs").replace("cos","np.cos").replace("sin","np.sin").replace("sign","np.sign"))
    print()

    H = Matrix([[L.diff(tec).diff(tec),L.diff(tec).diff(cs)],[L.diff(cs).diff(tec),L.diff(cs).diff(cs)]])

    req,res = cse(H,optimizations='basic')
    for line in req:
        print("{} = {}".format(line[0],line[1]).replace("Abs","np.abs").replace("cos","np.cos").replace("sin","np.sin").replace("sign","np.sign"))
    print("{}".format(res[0]).replace("Abs","np.abs").replace("cos","np.cos").replace("sin","np.sin").replace("sign","np.sign"))
derive_solution()
```

```python

import numpy as np
import pylab as plt

def phase_unwrapp1d(theta):
    '''the difference between two timesteps is unaliased by assumption so theta_i+1 - theta_i < pi.
    So Wrap(theta_i+1 - theta_i) is the real gradient and we can integrate them'''
    def wrap(phi):
        res = 1j*phi
        np.exp(res,out=res)
        return np.atleast_1d(np.angle(res))
    grad = wrap(theta[1:] - theta[:-1])
    unwrapped_theta = np.zeros(len(theta),dtype=np.double)
    np.cumsum(grad,out=unwrapped_theta[1:])
    unwrapped_theta += theta[0]
    return unwrapped_theta

TECU = 10e16

def least_squares_solve(obs_phase,freqs,times,Cd_error,Ct_ratio=0.01,m0=None):
    '''Solves for the terms phase(CS,TEC,delay) = CS + e^2/(4pi ep0 me c) * TEC/nu + 2pi*nu*delay

    Assumes phase is in units of radians, freqs in is units of Hz,
    and times is in units of seconds with arbitrary offset

    obs_phase is shape (num_freqs, num_times)'''

    f = np.multiply.outer(freqs,np.ones(len(times)))

    def calc_phase(cs,tec,delay, freqs):
        phase = np.multiply.outer(np.ones(len(freqs)),cs) + 8.44797256e-7*TECU * np.multiply.outer(1./freqs,tec) + 2.*np.pi*np.multiply.outer(freqs,delay)
        return phase

    def neglogL(obs_phase,phase,CdCt):
        '''Return per timestep'''
        L2 = obs_phase - phase
        L2 *= L2
        L2 /= (CdCt + 1e-15)
        return np.sum(L2,axis=0)/2.

    def calc_grad(cs,tec,delay,f,obs_phase,CdCt):
        grad = np.zeros([len(times),3],dtype=np.double)
        phase = calc_phase(cs,tec,delay,f[:,0])
        dd = obs_phase - phase
        dd /= CdCt
        #tau comp
        gtau = dd*f
        gtau *= -2.*np.pi
        gtau = np.sum(gtau,axis=0)
        #tec comp
        gtec = dd/f
        gtec *= -8.44797256e-7*TECU
        gtec = np.sum(gtec,axis=0)
        #cs comp
        gcs = -dd
        gcs = np.sum(gcs,axis=0)
        grad[:,0] = gtau
        grad[:,1] = gtec
        grad[:,2] = gcs
        return grad

    def calc_Hessian(f,CdCt):
        H = np.zeros([len(times),3,3],dtype = np.double)
        x0 = f/CdCt
        H[:,0,0] = np.sum(4*np.pi**2 * x0*f,axis=0)
        H[:,0,1] = np.sum(2*np.pi*8.44797256e-7*TECU/CdCt,axis=0)
        H[:,0,2] = np.sum(2*np.pi*x0,axis=0)
        H[:,1,1] = np.sum((8.44797256e-7*TECU)**2/(f**2*CdCt),axis=0)
        H[:,1,2] = np.sum(8.44797256e-7*TECU/(f*CdCt),axis=0)
        H[:,2,2] = np.sum(1./CdCt,axis=0)
        H[:,1,0] = H[:,0,1]
        H[:,2,0] = H[:,0,2]
        H[:,2,1] = H[:,1,2]
        return H

    def inv_Hessian(H):
        a = H[:,0,0]
        b = H[:,0,1]
        c = H[:,0,2]
        d = H[:,1,1]
        e = H[:,1,2]
        f = H[:,2,2]
        x0 = 1/a
        x1 = b**2
        x2 = x0*x1
        x3 = 1/(d - x2)
        x4 = b*c
        x5 = e - x0*x4
        x6 = x3*x5
        x7 = b*x6 - c
        x8 = a*d
        x9 = -x1 + x8
        x10 = 1/(a*e**2 + c**2*d - 2*e*x4 + f*x1 - f*x8)
        x11 = x0*x10*x9
        x12 = x10*x9
        x13 = x0*x3*(-b + x12*x5*x7)
        x14 = -x11*x7
        x15 = x12*x6
        Hinv = np.zeros([len(a),3,3])
        Hinv[:,0,0] = x0*(-x11*x7**2 + x2*x3 + 1)
        Hinv[:,0,1] = x13
        Hinv[:,0,2] = x14
        Hinv[:,1,0] = Hinv[:,0,1]
        Hinv[:,1,1] = x3*(-x12*x3*x5**2 + 1)
        Hinv[:,1,2] = x15
        Hinv[:,2,0] = Hinv[:,0,2]
        Hinv[:,2,1] = Hinv[:,1,2]
        Hinv[:,2,2] = -x12
        return Hinv

    def calc_epsilon_n(dir,tau_i,tec_i,cs_i,freqs,CdCt,obs_phase,step=1e-3):
        """Approximate stepsize"""
        g0 = calc_phase(cs_i,tec_i,tau_i, freqs)
        gp = calc_phase(cs_i+step*dir[:,2], tec_i + step*dir[:,1], tau_i + step*dir[:,0], freqs)
        Gm = (gp - g0)/step
        dd = obs_phase - g0
        epsilon_n = (np.sum(Gm*dd/CdCt,axis=0)/np.sum(Gm/CdCt*Gm,axis=0))
        return epsilon_n

    if m0 is None:
        # come up with initial guess
        cs0  = np.zeros(len(times),dtype=np.double)
        delay0 = np.zeros(len(times),dtype=np.double)
        tec0 = np.zeros(len(times),dtype=np.double)
    #     # d/dnu (phase*nu) = cs + 4pi*nu*delay
        x0 = (freqs*obs_phase.T).T
        x1 = ((x0[1:,:] - x0[:-1,:]).T/(freqs[1:] - freqs[:-1])).T
    #     # d^2/dnu^2 (phase*nu) = 4pi*delay
        x2 = ((x1[1:,:] - x1[:-1,:]).T/(freqs[1:-1] - freqs[:-2])).T
        tau0 = np.mean(x2,axis=0)/4./np.pi
        x3 = 2*np.pi*np.multiply.outer(freqs,delay0)
        cs0 = np.mean(x1 - 2.*x3[1:,:],axis=0)
        cs = cs0*0
        tau = tau0
        tec = tec0
    else:
        cs = m0[:,2]
        tec = m0[:,1]
        tau = m0[:,0]
    #print("Initial CS: {}".format(cs0))
    #print("Initial TEC: {}".format(tec0))
    #print("Initial delay: {}".format(delay0))

    Ct = (Ct_ratio*np.abs(obs_phase))**2
    Cd = (Cd_error*np.pi/180.)**2
    CdCt = Cd+Ct
    S = neglogL(obs_phase,calc_phase(cs,tec,tau,freqs),CdCt)
    #print("Initial neglogL: {}".format(S))
    iter = 0
    Nmax = 1
    while iter < Nmax:
        grad = calc_grad(cs,tec,tau,f,obs_phase,CdCt)
        H = calc_Hessian(f,CdCt)
        Hinv = inv_Hessian(H)
        dir = np.einsum("ijk,ik->ij",Hinv,grad)
        epsilon_n = calc_epsilon_n(dir,tau,tec,cs,freqs,CdCt,obs_phase,step=1e-3)
        #print("epsilon_n: {}".format(epsilon_n))
        cs, tec, tau = cs+epsilon_n*dir[:,2], tec + epsilon_n*dir[:,1], tau + epsilon_n*dir[:,0]
        S = neglogL(obs_phase,calc_phase(cs,tec,tau,freqs),CdCt)
        m = np.array([tau,tec,cs]).T
        #print("Model: {}".format(m))
        #print("iter {}: neglogL: {}, log|dm/m|: {}, |grad|: {}".format(iter, S, np.mean(np.log(np.abs(np.einsum("i,ij->ij",epsilon_n,dir)/m))),np.sum(np.abs(grad))))
        iter += 1
    #print(Hinv)
    print("Final neglogL: {}".format(S))
    return m,Hinv

def l1data_l2model_solve(obs_phase,freqs,times,Cd_error,Ct_ratio=0.01,m0=None):
    '''Solves for the terms phase(CS,TEC,delay) = CS + e^2/(4pi ep0 me c) * TEC/nu

    Delay is taken out.

    Assumes phase is in units of radians, freqs in is units of Hz,
    and times is in units of seconds with arbitrary offset

    obs_phase is shape (num_freqs, num_times)'''

    alpha = 1.
    beta = 0.
    def calc_phase(m, freqs):
        tec = m[:,0]
        cs = m[:,1]
        phase = 8.44797256e-7*TECU * np.multiply.outer(1./freqs,tec) + alpha*cs
        return phase

    def neglogL(obs_phase,m,CdCt_phase,m0,cov_m,freqs):
        '''Return per timestep'''
        K = 8.44797256e-7*TECU
        nu = np.multiply.outer(1./freqs,np.ones(obs_phase.shape[1]))
        tec = m[:,0]
        cs = m[:,1]
        tec_p = m0[:,0]
        cs_p = m0[:,1]
        sigma_tec2 = cov_m[0]
        sigma_cs2 = cov_m[1]
        sigma_phase = np.sqrt(CdCt_phase)
        phase = obs_phase
        #return np.sum(np.abs(K*np.multiply.outer(1./freqs,tec) - phase)/sigma_phase,axis=0)

        return beta*((tec - tec_p)**2/sigma_tec2 + (cs - cs_p)**2/sigma_cs2)/2 + np.sum(np.abs(K*np.multiply.outer(1./freqs,tec) + alpha*cs - phase)/sigma_phase,axis=0)

    def calc_grad(obs_phase,m,CdCt_phase,m0,cov_m,freqs):

        K = 8.44797256e-7*TECU
        nu = np.multiply.outer(1./freqs,np.ones(obs_phase.shape[1]))
        tec = m[:,0]
        cs = m[:,1]
        tec_p = m0[:,0]
        cs_p = m0[:,1]
        sigma_tec2 = cov_m[0]
        sigma_cs2 = cov_m[1]
        sigma_phase = np.sqrt(CdCt_phase)
        phase = obs_phase

        x0 = sigma_tec2
        x1 = K/nu
        x1_ = K*np.multiply.outer(1./freqs,tec)
        x2 = np.sign(alpha*cs - phase + x1_)/sigma_phase
        x3 = sigma_cs2

        grad = np.zeros([obs_phase.shape[1],2])
        grad[:,0] = x0*(beta*(tec - tec_p)/x0 + np.sum((x1*x2),axis=0))
        grad[:,1] = x3 * (beta*(cs - cs_p)/x3 + np.sum(alpha*x2,axis=0))
        return grad

    def calc_epsilon_n(dir,m,freqs,CdCt,obs_phase,step=1e-3):
        """Approximate stepsize"""
        g0 = calc_phase(m, freqs)
        gp = calc_phase(m + step*dir, freqs)
        Gm = (gp - g0)/step
        dd = obs_phase - g0
        epsilon_n = (np.sum(Gm*dd/CdCt,axis=0)/np.sum(Gm/CdCt*Gm,axis=0))
        return epsilon_n

    if m0 is None:
        m = np.zeros([len(times),2],dtype=np.double)
    else:
        m = m0.copy()

    cov_m = np.array([1e-4,1e-4])
    #print( calc_phase(m,freqs) - obs_phase)

    Ct = (Ct_ratio*np.abs(obs_phase))**2
    Cd = (Cd_error*np.pi/180.)**2
    CdCt = Cd+Ct
    #print(np.sqrt(CdCt))
    #print( np.sum(np.abs(calc_phase(m,freqs) - obs_phase)/np.sqrt(CdCt),axis=0))
    S = neglogL(obs_phase,m,CdCt,m0,cov_m,freqs)
    print("Initial neglogL: {}".format(S))
    iter = 0
    Nmax = 3
    while iter < Nmax:
        grad = calc_grad(obs_phase,m,CdCt,m0,cov_m,freqs)
        dir = grad
        epsilon_n = calc_epsilon_n(dir,m,freqs,CdCt,obs_phase,step=1e-3)
        #print("epsilon_n: {}".format(epsilon_n))
        m += dir*epsilon_n
        S = neglogL(obs_phase,m,CdCt,m0,cov_m,freqs)
        #print("Model: {}".format(m))
        #print("iter {}: neglogL: {}, log|dm/m|: {}, |grad|: {}".format(iter, S, np.mean(np.log(np.abs(np.einsum("i,ij->ij",epsilon_n,dir)/m))),np.sum(np.abs(grad))))
        iter += 1
    #print(Hinv)
    print("Final neglogL: {}".format(S))
    return m

def test_clock_tec_solve_l1l2():
    np.random.seed(1234)
    times = np.arange(1)
    freqs = np.linspace(110e6,170e6,100)
    cs = times*0.00
    tec = times + 0.01
    phase = np.multiply.outer(np.ones(len(freqs)),cs) + 8.44797256e-7*TECU*np.multiply.outer(1./freqs,tec)
    phase += 10*np.pi/180.*np.random.normal(size=[len(freqs),len(times)])
    #phase = np.angle(np.exp(1j*phase))
    m0 = np.array([[0.,0.]])
    #m0,cov = least_squares_solve(phase, freqs, times,10,Ct_ratio=1)
    #print(m0)
    m = m0[:,1:3]

    m = l1data_l2model_solve(phase,freqs,times,10,Ct_ratio=1,m0=m0)
    rec_phase = np.multiply.outer(np.ones(len(freqs)),m[:,1]) + 8.44797256e-7*TECU*np.multiply.outer(1./freqs,m[:,0])
    print(m)
    plt.plot(freqs,phase)
    plt.plot(freqs,rec_phase)
    plt.show()

def test_clock_tec_solve_l1l2_tolerance():
    times = np.arange(1)
    freqs = np.linspace(110e6,170e6,100)
    cs = times+0
    tec0 = 0.0001
    tec = times+tec0
    phase = np.multiply.outer(np.ones(len(freqs)),cs) + 8.44797256e-7*TECU*np.multiply.outer(1./freqs,tec)
    phase += 40*np.pi/180.*np.random.normal(size=[len(freqs),len(times)])
    #phase = np.angle(np.exp(1j*phase))
    m0,cov = least_squares_solve(phase, freqs, times,10,Ct_ratio=1)
    dist = []
    final = []
    m0 = m0[:,1:3]
    for tec in np.linspace(tec0-5,tec0 + 5,1000):
        dist.append(tec - tec0)
        m0[:,0] = tec
        m0[:,1] = 0.
        m = l1data_l2model_solve(phase,freqs,times,40,Ct_ratio=0.01,m0=m0)
        #print(m)
        final.append(m[0,0] - tec0)
    plt.plot(dist,final)
    plt.show()

test_clock_tec_solve_l1l2()
```

```python

```

```python

```


```python
import matplotlib as mpl
import matplotlib.pyplot as plt
import numpy as np
from scipy.integrate import odeint
```

## Working in relative coordinates for particles 1 and 2:
## (This makes PN corrections easier)
### $\mathbf{a} = \mathbf{a}_1 - \mathbf{a}_2$
### $\mathbf{r} = \mathbf{r}_1 - \mathbf{r}_2=(r_x,r_y)$
### $\mathbf{n} = \mathbf{r}/r = \mathbf{n}_{12} = -\mathbf{n}_{21}$

## Lagrandian:
### Richardson & Kelly 1988

\begin{equation}
\mathcal{L} = v^2 /2 + \frac{Gm}{r} + \frac{(1-3\eta)}{8 c^2} v^3 - \frac{1}{2 c^2} \Big( \frac{Gm}{r} \Big)^2  + \frac{Gm}{2 r c^2} \Big( (3 +\eta)v^2 + \frac{\eta}{r^2} (\mathbf{r} \cdot \mathbf{v})^2 \Big) + \mathcal{O}(c^{-4})
\end{equation}

with: $m = M_1  + M_2$ , and  $\ \ \eta = (M_1 M_2) / (M_1  + M_2)^2$

## For Python : $y = r_x, r_y, r_x', r_y' =r_x, r_y, v_x, v_y$

```python
c_squared_val = (3e8)**2.
M_1_GW150914 = 35 * 1.989e+30
M_2_GW150914 = 30 * 1.989e+30
eta_val = (M_1_GW150914 * M_2_GW150914) / ((M_1_GW150914 +  M_2_GW150914)**2.)
print(eta_val)
Gm_val = 6.674e-11 * (M_1_GW150914 + M_2_GW150914)
t = np.linspace(0, 5, int(1e4))
```

    0.2485207100591716

```python
r_isco_tot_approx = 6 * Gm_val / c_squared_val
```

```python
y0 = [r_isco_tot_approx*20., 0., 0., r_isco_tot_approx*37]
```


## Authors
The original SEIR model and ipython notebook for it were developed by Dr. Alison Hill
and cloned from github at
https://github.com/alsnhll/SEIR_COVID19
referenced on the web from the 50 best modelling apps for covid:
https://towardsdatascience.com/top-5-r-resources-on-covid-19-coronavirus-1d4c8df6d85f

The cautionary feedback version of the model was conceived and written by John S McCaskill, a fellow of the European Centre for Living Technology. It is based on experience in modelling Q$\beta$ quasispecies viral infections of bacterial cells back in the late 80's and early 90's in Manfred Eigen's lab at the Max Planck Institute. In cell free model systems, exponential growth of the virus changes to linear growth because of the limited amount of replicase enzyme available. I was curious to see if the often remarkably linear response following an early exponential growth (or power law in the case of small world connectivity network) phase, could be simply accounted for by a cautionary saturation effect as outlined below.

## SEIR and SCEIR Models

### Equations

\begin{equation}
\begin{split}
\dot{S} &= -(\beta_1 I_1 +\beta_2 I_2 + \beta_3 I_3) S + c_1 S_c - c_2*S*I_3\\
\dot{S_c} &= - c_0(\beta_1 I_1 +\beta_2 I_2 + \beta_3 I_3) S_c - c_1 S_c + c_2*S*I_3\\
\dot{E} &=(\beta_1 I_1 +\beta_2 I_2 + \beta_3 I_3 ) (S + c_0 S_c) - a E \\
\dot{I_1} &= a E - \gamma_1 I_1 - p_1 I_1 \\
\dot{I_2} &= p_1 I_1 -\gamma_2 I_2 - p_2 I_2 \\
\dot{I_3} & = p_2 I_2 -\gamma_3 I_3 - \mu I_3 \\
\dot{R} & = \gamma_1 I_1 + \gamma_2 I_2 + \gamma_3 I_3 \\
\dot{D} & = \mu I_3
\end{split}
\end{equation}

The use of I_3 as a state variable triggering susceptibles to execute caution is just one choice. In contrast with deaths, it does not accumulate over time and so retains the property of an active threat to society, rather than an historical one.

Actually, the effect of caution may be quadratic, since both the individual doing the infection and individual potentially being infected may be executing caution. The current version assumes that infectives do not change their precautionary measures in response to I_3. To implement this we would need to further distinguish careful and non careful infectives at least up to the I_1 level.

### Variables
* $S$: Susceptible individuals living as normal
* $S_c$: Susceptible individuals exercising pandemy precautions
* $E$: Exposed individuals - infected but not yet infectious or symptomatic
* $I_i$: Infected individuals in severity class $i$. Severity increaes with $i$ and we assume individuals must pass through all previous classes
  * $I_1$: Mild infection (hospitalization not required)
  * $I_2$: Severe infection (hospitalization required)
  * $I_3$: Critical infection (ICU required)
* $R$: individuals who have recovered from disease and are now immune
* $D$: Dead individuals
* $N=S+S_c+E+I_1+I_2+I_3+R+D$ Total population size (constant)

### Parameters
* $\beta_i$ rate at which infected individuals in class $I_i$ contact susceptibles and infect them
* $a$ rate of progression from the exposed to infected class
* $\gamma_i$ rate at which infected individuals in class $I_i$ recover from disease and become immune
* $p_i$ rate at which infected individuals in class $I_i$ progress to class $I_{I+1}$
* $\mu$ death rate for individuals in the most severe stage of disease

* $c_i$ three parameters characterizing cautionary response of population via class $S_c$

    - c_0 : reduction factor for exposure for cautioned susceptibles

    - c_1 : duration of caution (exponential decay time constant in days)

    - c_2 : rate constant for transition from uncautioned to cautioned susceptible

### Basic reproductive ratio

$R_0$ in the simple uncautioned SEI3R model is the sum of
1. the average number of secondary infections generated from an individual in stage $I_1$
2. the probability that an infected individual progresses to $I_2$ multiplied by the average number of secondary infections generated from an individual in stage $I_2$
3.  the probability that an infected individual progresses to $I_3$ multiplied by the average number of secondary infections generated from an individual in stage $I_3$

\begin{equation}
\begin{split}
R_0 & = N\frac{\beta_1}{p_1+\gamma_1} + \frac{p_1}{p_1 + \gamma_1} \left( \frac{N \beta_2}{p_2+\gamma_2} + \frac{p_2}{p_2 + \gamma_2} \frac{N \beta_3}{\mu+\gamma_3}\right)\\
&= N\frac{\beta_1}{p_1+\gamma_1} \left(1 + \frac{p_1}{p_2 + \gamma_2}\frac{\beta_2}{\beta_1} \left( 1 + \frac{p_2}{\mu + \gamma_3} \frac{\beta_3}{\beta_2} \right) \right)
\end{split}
\end{equation}

$R_0$ as such clearly applies to a population where S=N, i.e. at outset of infection.
As such, it remains unchanged in the SCEI3R model, since S_c is then 0.

###  Influence of caution on SEIR model

```python
import numpy as np, matplotlib.pyplot as plt
from scipy.integrate import odeint
```

```python
#Defining the differential equations

#Don't track S because all variables must add up to 1
#include blank first entry in vector for beta, gamma, p so that indices align in equations and code.
#In the future could include recovery or infection from the exposed class (asymptomatics)

def seir(y,t,b,a,g,p,u,N):
    dy=[0,0,0,0,0,0]
    S=N-sum(y);
    dy[0]=np.dot(b[1:3],y[1:3])*S-a*y[0] # E
    dy[1]= a*y[0]-(g[1]+p[1])*y[1] #I1
    dy[2]= p[1]*y[1] -(g[2]+p[2])*y[2] #I2
    dy[3]= p[2]*y[2] -(g[3]+u)*y[3] #I3
    dy[4]= np.dot(g[1:3],y[1:3]) #R
    dy[5]=u*y[3] #D

    return dy

#include caution correction via y[6]=S_c and three parameters in vector c (see below)

def sceir(y,t,b,a,g,p,u,c,N):
    dy=[0,0,0,0,0,0,0]
    S=N-sum(y);
    dy[0]=np.dot(b[1:3],y[1:3])*(S+c[0]*y[6])-a*y[0] # E
    dy[1]= a*y[0]-(g[1]+p[1])*y[1] #I1
    dy[2]= p[1]*y[1] -(g[2]+p[2])*y[2] #I2
    dy[3]= p[2]*y[2] -(g[3]+u)*y[3] #I3
    dy[4]= np.dot(g[1:3],y[1:3]) #R
    dy[5]=u*y[3] #D
    dy[6]= -c[0]*np.dot(b[1:3],y[1:3])*y[6]-c[1]*y[6]+c[2]*S*y[3]  #Sc

    return dy
```

```python
# Define parameters based on clinical observations

# Dr. Alison will add data sources soon
# https://github.com/midas-network/COVID-19/tree/master/parameter_estimates/2019_novel_coronavirus

Exposure=0.25 # Rate coefficient for exposure per individual in contact per day
IncubPeriod=5  #Incubation period, days
DurMildInf=10 #Duration of mild infections, days
FracMild=0.8  #Fraction of infections that are mild
FracSevere=0.15 #Fraction of infections that are severe
FracCritical=0.05 #Fraction of infections that are critical
CFR=0.02 #Case fatality rate (fraction of infections resulting in death)
TimeICUDeath=7 #Time from ICU admission to death, days
DurHosp=11 #Duration of hospitalization, days

# Model extension by John McCaskill to include caution
CautionFactor= 0.3 # Fractional reduction of exposure rate for cautioned individuals
CautionRetention= 14. # Duration of cautionary state of susceptibles (2 weeks)
CautionICUFrac= 0.25 # Fraction of ICUs occupied leading to transition to caution @ 1/day
ICUFrac= 0.001 # Fraction of ICUs relative to population size N
```

```python
# Define simulation parameters from clinical and caution parameters

N=1000000
b=np.zeros(4) #beta
g=np.zeros(4) #gamma
p=np.zeros(3)
c=np.zeros(3) #caution

a=1/IncubPeriod

u=(1/TimeICUDeath)*(CFR/FracCritical)
g[3]=(1/TimeICUDeath)-u

p[2]=(1/DurHosp)*(FracCritical/(FracCritical+FracSevere))
g[2]=(1/DurHosp)-p[2]

g[1]=(1/DurMildInf)*FracMild
p[1]=(1/DurMildInf)-g[1]

c[0]=CautionFactor
c[1]=1/CautionRetention
c[2]=1/(N*ICUFrac*CautionICUFrac)

#b=Exposure*np.ones(4)/N # all stages transmit equally
b=Exposure*np.array([0,1,0,0])/N # hospitalized cases don't transmit
```

    R0 =  2.5

```python
print(b)
print(a)
print(g)
print(p)
print(u)
print(c)
```

    [0.0e+00 2.5e-07 0.0e+00 0.0e+00]
    0.2
    [0.         0.08       0.06818182 0.08571429]
    [0.         0.02       0.02272727]
    0.057142857142857134
    [0.3        0.07142857 0.004     ]

```python
tmax=365
tvec=np.arange(0,tmax,0.1)

#SEIR
ic=np.zeros(6)
ic[0]=1
soln=odeint(seir,ic,tvec,args=(b,a,g,p,u,N))
soln=np.hstack((N-np.sum(soln,axis=1,keepdims=True),soln))

#Plot on linear scale
plt.figure(figsize=(13,5))
plt.subplot(1,2,1)
plt.plot(tvec,soln)
plt.xlabel("Time (days)")
plt.ylabel("Number per 1000000 People")
plt.legend(("S","E","I1","I2","I3","R","D"))
plt.ylim([0,1000000])

#Same plot but on log scale
plt.subplot(1,2,2)
plt.plot(tvec,soln)
plt.semilogy()
plt.xlabel("Time (days)")
plt.ylabel("Number per 1000000 People")
plt.legend(("S","E","I1","I2","I3","R","D"))
plt.ylim([1,1000000])
#plt.tight_layout()

#SCEIR
icc=np.zeros(7)
icc[0]=1
solnc=odeint(sceir,icc,tvec,args=(b,a,g,p,u,c,N))
solnc=np.hstack((N-np.sum(solnc,axis=1,keepdims=True),solnc))

#Plot on linear scale
plt.figure(figsize=(13,5))
plt.subplot(1,2,1)
plt.plot(tvec,solnc)
plt.xlabel("Time (days)")
plt.ylabel("Number per 1000000 People")
plt.legend(("S","E","I1","I2","I3","R","D","Sc"))
plt.ylim([0,1000000])

#Same plot but on log scale
plt.subplot(1,2,2)
plt.plot(tvec,solnc)
plt.semilogy()
plt.xlabel("Time (days)")
plt.ylabel("Number per 1000000 People")
plt.legend(("S","E","I1","I2","I3","R","D","Sc"))
plt.ylim([1,1000000])
#plt.tight_layout()
print("")
```

#### Cumulative corona cases

```python
plt.figure(figsize=(13,5))
plt.subplot(1,2,1)
plt.plot(tvec,np.sum(soln[:,2:6],axis=1,keepdims=True))
plt.plot(tvec,np.sum(solnc[:,2:6],axis=1,keepdims=True))
plt.xlabel("Time (days)")
plt.ylabel("Number per 1000000 People")
plt.legend(("No caution","Caution"))
plt.ylim([0,1000000])
plt.title('Cumulative cases')
print("")
```

## SCCEIR model with caution distinguished $E$ and  𝐼1

### Equations

\begin{equation}
\begin{split}
\dot{S} &= -(\beta_1 (I_1 + c_0 I_{1c}) +\beta_2 I_2 + \beta_3 I_3) S + c_1 S_c - c_2*S*I_3\\
\dot{S_c} &= - c_0(\beta_1 (I_1 + c_0 I_{1c}) +\beta_2 I_2 + \beta_3 I_3) S_c - c_1 S_c + c_2*S*I_3\\
\dot{E} &=(\beta_1 (I_1 + c_0 I_{1c}) +\beta_2 I_2 + \beta_3 I_3 ) S - a E \\
\dot{E_c} &=(\beta_1 (I_1 + c_0 I_{1c}) +\beta_2 I_2 + \beta_3 I_3 ) c_0 S_c - a E \\
\dot{I_1} &= a E - \gamma_1 I_1 - p_1 I_1 \\
\dot{I_{1c}} &= a E_c - \gamma_1 I_{1c} - p_1 I_{1c} \\
\dot{I_2} &= p_1 (I_1 + I_{1c}) -\gamma_2 I_2 - p_2 I_2 \\
\dot{I_3} & = p_2 I_2 -\gamma_3 I_3 - \mu I_3 \\
\dot{R} & = \gamma_1 (I_1 + I_{1c}) + \gamma_2 I_2 + \gamma_3 I_3 \\
\dot{D} & = \mu I_3
\end{split}
\end{equation}

The use of I_3 as a state variable triggering susceptibles to execute caution is just one choice. In contrast with deaths, it does not accumulate over time and so retains the property of an active threat to society, rather than an historical one.

Here, the effect of caution is quadratic, since both the individual doing the infection and individual potentially being infected may be executing caution. To implement this we distinguish careful and non careful exposed and infectives up to the I_1 level. Once in hospital there is no difference, since all caution is executed wrt infected patients.

### Variables
* $S$: Susceptible individuals living as normal
* $S_c$: Susceptible individuals exercising pandemy precautions
* $E$: Exposed individuals living as normal - infected but not yet infectious or symptomatic
* $E_c$: Exposed individuals exercising pandemy precautions
* $I_i$: Infected individuals in severity class $i$. Severity increaes with $i$ and we assume individuals must pass through all previous classes. Split non hospital cases by caution.
  * $I_1$: Mild infection (hospitalization not required), living as normal
  * $I_{1c}$: Mild infection (hospitalization not required), exercising caution
  * $I_2$: Severe infection (hospitalization required)
  * $I_3$: Critical infection (ICU required)
* $R$: individuals who have recovered from disease and are now immune
* $D$: Dead individuals
* $N=S+S_c+E+E_c+I_{1c}+I_1+I_2+I_3+R+D$ Total population size (constant)

### Parameters
* $\beta_i$ rate at which infected individuals in class $I_i$ contact susceptibles and infect them
* $a$ rate of progression from the exposed to infected class
* $\gamma_i$ rate at which infected individuals in class $I_i$ recover from disease and become immune
* $p_i$ rate at which infected individuals in class $I_i$ progress to class $I_{I+1}$
* $\mu$ death rate for individuals in the most severe stage of disease

* $c_i$ three parameters characterizing cautionary response of population via class $S_c$

    - c_0 : reduction factor for exposure for cautioned susceptibles

    - c_1 : duration of caution (exponential decay time constant in days)

    - c_2 : rate constant for transition from uncautioned to cautioned susceptible

###  Influence of caution on SCCEIR model

```python
import numpy as np, matplotlib.pyplot as plt
from scipy.integrate import odeint
```

```python
#Defining the differential equations

#Don't track S because all variables must add up to 1
#include blank first entry in vector for beta, gamma, p so that indices align in equations and code.
def seir(y,t,b,a,g,p,u,N):
    dy=[0,0,0,0,0,0]
    S=N-sum(y);
    dy[0]=np.dot(b[1:3],y[1:3])*S-a*y[0] # E
    dy[1]= a*y[0]-(g[1]+p[1])*y[1] #I1
    dy[2]= p[1]*y[1] -(g[2]+p[2])*y[2] #I2
    dy[3]= p[2]*y[2] -(g[3]+u)*y[3] #I3
    dy[4]= np.dot(g[1:3],y[1:3]) #R
    dy[5]=u*y[3] #D

    return dy
#include caution correction via y[6]=S_c and three parameters in vector c (see below)
def sceir(y,t,b,a,g,p,u,c,N):
    dy=[0,0,0,0,0,0,0]
    S=N-sum(y);
    dy[0]=np.dot(b[1:3],y[1:3])*(S+c[0]*y[6])-a*y[0] # E
    dy[1]= a*y[0]-(g[1]+p[1])*y[1] #I1
    dy[2]= p[1]*y[1] -(g[2]+p[2])*y[2] #I2
    dy[3]= p[2]*y[2] -(g[3]+u)*y[3] #I3
    dy[4]= np.dot(g[1:3],y[1:3]) #R
    dy[5]=u*y[3] #D
    dy[6]= -c[0]*np.dot(b[1:3],y[1:3])*y[6]-c[1]*y[6]+c[2]*S*y[3]  #Sc
    return dy

#include caution correction to E and I_1 via y[7]=E_c and y[8]=I_1c
def scceir(y,t,b,a,g,p,u,c,N):
    dy=[0,0,0,0,0,0,0,0,0]
    S=N-sum(y);
    dy[0]=np.dot(b[1:3],y[1:3])*S+c[0]*b[1]*y[8]*S-a*y[0] # E
    dy[1]= a*y[0]-(g[1]+p[1])*y[1] #I1
    dy[2]= p[1]*(y[1]+y[8])-(g[2]+p[2])*y[2] #I2
    dy[3]= p[2]*y[2] -(g[3]+u)*y[3] #I3
    dy[4]= np.dot(g[1:3],y[1:3])+g[1]*y[8] #R
    dy[5]=u*y[3] #D
    dy[6]= -c[0]*np.dot(b[1:3],y[1:3])*y[6]-c[0]*c[0]*b[1]*y[8]*y[6]-c[1]*y[6]+c[2]*S*y[3]  #Sc
    dy[7]=np.dot(b[1:3],y[1:3])*c[0]*y[6]+c[0]*c[0]*b[1]*y[8]*y[6]-a*y[7] # Ec
    dy[8]= a*y[7]-(g[1]+p[1])*y[8] #I1
    return dy
```

```python
# Define parameters based on clinical observations and generic caution parameters

# Dr. Alison will add data sources soon
# https://github.com/midas-network/COVID-19/tree/master/parameter_estimates/2019_novel_coronavirus

Exposure=0.25 # Rate coefficient for exposure per individual in contact per day
IncubPeriod=5  #Incubation period, days
DurMildInf=10 #Duration of mild infections, days
FracMild=0.8  #Fraction of infections that are mild
FracSevere=0.15 #Fraction of infections that are severe
FracCritical=0.05 #Fraction of infections that are critical
CFR=0.02 #Case fatality rate (fraction of infections resulting in death)
TimeICUDeath=7 #Time from ICU admission to death, days
DurHosp=11 #Duration of hospitalization, days

# Model extension by John McCaskill to include caution
CautionFactor= 0.3 # Fractional reduction of exposure rate for cautioned individuals
CautionRetention= 14. # Duration of cautionary state of susceptibles (4 weeks)
CautionICUFrac= 0.25 # Fraction of ICUs occupied leading to 90% of susceptibles in caution
ICUFrac= 0.001 # Fraction of ICUs relative to population size N
```

```python
# Define simulation parameters from clinical and caution parameters
tmax = 365
N=1000000
b=np.zeros(4) #beta
g=np.zeros(4) #gamma
p=np.zeros(3) # progression
c=np.zeros(3) #caution

a=1/IncubPeriod

u=(1/TimeICUDeath)*(CFR/FracCritical)
g[3]=(1/TimeICUDeath)-u

p[2]=(1/DurHosp)*(FracCritical/(FracCritical+FracSevere))
g[2]=(1/DurHosp)-p[2]

g[1]=(1/DurMildInf)*FracMild
p[1]=(1/DurMildInf)-g[1]

c[0]=CautionFactor
c[1]=1/CautionRetention
c[2]=1/(N*ICUFrac*CautionICUFrac)   # this is the rate coefficient giving 1/day at IE = denom

#b=Exposure*np.ones(4)/N # all stages transmit equally
b=Exposure*np.array([0,1,0,0])/N # hospitalized cases don't transmit

#Calculate basic reproductive ratio
R0=N*((b[1]/(p[1]+g[1]))+(p[1]/(p[1]+g[1]))*(b[2]/(p[2]+g[2])+ (p[2]/(p[2]+g[2]))*(b[3]/(u+g[3]))))
print("R0 = {0:4.1f}".format(R0))
```

    R0 =  2.5

```python
def solveplot(b,a,g,p,u,c,N,tmax,scale='linear',species='no_susc',plottitle= '',label='',
              newplot = True,models=['SEIR','SCEIR','SCCEIR']):
    """
    solve ODEs and plot for set of models indicated
    b,a,g,p,u,c,N,tmax  simulation parameters
    scale: alternaticve 'linear' or 'log'
    species alternatives 'all', 'confirmed', 'deaths', 'daily confirmed', 'daily deaths'
    plottitle : title for plot
    label : label for curve when called as part of multicurve plot
    newplot : whether to open new plot True/False
    models : list of models to include, default all three of those possible
    """
    tvec=np.arange(0,tmax,0.1)
    nmodels = len(models)
    nm = 0

    if newplot == True:
        plt.figure(figsize=(nmodels*8,6))

    solns = []
    if 'SEIR' in models:   #SEIR
        nm = nm + 1
        ic=np.zeros(6)
        ic[0]=1
        soln=odeint(seir,ic,tvec,args=(b,a,g,p,u,N))
        soln=np.hstack((N-np.sum(soln,axis=1,keepdims=True),soln)) # adds S as first elt

        #Plot
        plt.subplot(1,nmodels,nm)
        if scale == 'log': #Plot on log scale
            plt.semilogy()
            plt.ylim([1,10000])
        elif species != 'daily confirmed':           # Plot on normal linear scale
            #plt.ylim([0,10000])
            pass
        if species == 'no_susc':
            plt.plot(tvec,soln[:,1:5],label=label)
            plt.legend(("E","I1","I2","I3"))
        elif species == 'confirmed' or species == 'daily confirmed':
            suma = np.sum(soln[:,2:7],axis=1)
            # print('length=',len(suma))
            if species == 'daily confirmed':
                sumd = np.zeros(len(suma))
                for i in range(1,len(suma)):
                    sumd[i] = suma[i]-suma[i-1]
                #plt.ylim([0,1000])
                plt.plot(tvec,sumd,label=label)
            else:
                #plt.ylim([0,200000])
                plt.plot(tvec,suma,label=label)
        elif species == 'all':
            plt.plot(tvec,soln,label=label)
            plt.legend(("S","E","I1","I2","I3","R","D"))
        plt.xlabel("Time (days)")
        plt.ylabel("Number per 1000000 People")
        plt.title('SEIR %s' % plottitle)
        solns.append(soln)

    if 'SCEIR' in models:   #SCEIR
        nm = nm + 1
        ic=np.zeros(7)
        ic[0]=1
        solnc=odeint(sceir,ic,tvec,args=(b,a,g,p,u,c,N))
        solnc=np.hstack((N-np.sum(solnc,axis=1,keepdims=True),solnc)) # adds S as first elt

        #Plot
        plt.subplot(1,nmodels,nm)
        if scale == 'log': #Plot on log scale
            plt.semilogy()
            plt.ylim([1,10000])
        elif species != 'daily confirmed':           # Plot on normal linear scale
            #plt.ylim([0,10000])
            pass
        if species == 'no_susc':
            plt.plot(tvec,solnc[:,1:5],label=label)
            plt.legend(("E","I1","I2","I3"))
        elif species == 'confirmed' or species == 'daily confirmed':
            suma = np.sum(solnc[:,2:7],axis=1)
            # print('length=',len(suma))
            if species == 'daily confirmed':
                sumd = np.zeros(len(suma))
                for i in range(1,len(suma)):
                    sumd[i] = suma[i]-suma[i-1]
                #plt.ylim([0,1000])
                plt.plot(tvec,sumd,label=label)
            else:
                #plt.ylim([0,200000])
                plt.plot(tvec,suma,label=label)
        elif species == 'all':
            plt.plot(tvec,solnc,label=label)
            plt.legend(("S","E","I1","I2","I3","R","D","Sc"))
        plt.xlabel("Time (days)")
        plt.ylabel("Number per 1000000 People")
        plt.title('SCEIR %s' % plottitle)
        solns.append(solnc)

    if 'SCCEIR' in models:   #SCCEIR
        nm = nm + 1
        ic=np.zeros(9)
        ic[0]=1
        solncc=odeint(scceir,ic,tvec,args=(b,a,g,p,u,c,N))
        solncc=np.hstack((N-np.sum(solncc,axis=1,keepdims=True),solncc)) # prepend S

        #Plot
        plt.subplot(1,nmodels,nm)
        if scale == 'log': #Plot on log scale
            plt.semilogy()
            plt.ylim([1,10000])
        elif species != 'daily confirmed':           # Plot on normal linear scale
            #plt.ylim([0,10000])
            pass
        if species == 'no_susc':
            plt.plot(tvec,solncc[:,1:5])
            plt.legend(("E","I1","I2","I3"))
        elif species == 'confirmed' or species == 'daily confirmed':
            suma = np.sum(solncc[:,2:7],axis=1) + solncc[:,9]
            if species == 'daily confirmed':
                sumd = np.zeros(len(suma))
                for i in range(1,len(suma)):
                    sumd[i] = suma[i]-suma[i-1]
                # plt.ylim([0,1000])
                plt.plot(tvec,sumd,label=label)
            else:
                # plt.ylim([0,200000])
                plt.plot(tvec,suma,label=label)
        elif species == 'all':
            plt.plot(tvec,solncc,label=label)
            plt.legend(("S","E","I1","I2","I3","R","D","Sc","Ec","I1c"))
        plt.xlabel("Time (days)")
        plt.ylabel("Number per 1000000 People")
        plt.title('SCCEIR %s' % plottitle)
        solns.append(solncc)
    return solns

```

```python
solns=solveplot(b,a,g,p,u,c,N,tmax,scale='linear')
```

```python
solns=solveplot(b,a,g,p,u,c,N,tmax,scale='linear',species='confirmed')
```

```python
solns=solveplot(b,a,g,p,u,c,N,tmax,scale='linear',species='daily confirmed')
```

```python
# caution standard parameters
CautionFactor= 0.3    # Fractional reduction of exposure rate for cautioned individuals
CautionRetention= 14. # Duration of cautionary state of susceptibles (4 weeks)
CautionICUFrac= 0.25  # Fraction of ICUs occupied leading to transition to caution @ 1/day
ICUFrac= 0.001        # Fraction of ICUs relative to population size N

# Sensitivity scans
CautionFactors= [0.1,0.25,0.5,0.75,1.0]  # Fractional reduction of exposure rate for cautioned individuals
CautionRetentions= [7.,14.,28.,56.,112.] # Duration of cautionary state of susceptibles (2 weeks)
CautionICUFracs= [0.0625,0.125,0.25,0.5,0.75] # Fraction of ICUs occupied leading to 90% of susceptibles in caution

newplot = True
for i in range(5):
    c[0]=CautionFactors[i]
    c[1]=1./CautionRetention
    c[2]=1./(N*ICUFrac*CautionICUFrac)
    label_c = 'CautionFactor %s' % CautionFactors[i]
    plottitle = 'Caution Factor Sensitivity'
    solns=solveplot(b,a,g,p,u,c,N,tmax,scale='linear',plottitle= plottitle,label=label_c,
              newplot = newplot,species='daily confirmed',models=['SCEIR','SCCEIR'])
    plt.legend()
    newplot = False

newplot = True
#for i in reversed(range(5)):
for i in range(5):
    c[0]=CautionFactor
    c[1]=1/CautionRetentions[i]
    c[2]=1./(N*ICUFrac*CautionICUFrac)
    label_c = 'CautionRetention %s'% CautionRetentions[i]
    plottitle = 'Caution Retention Sensitivity'
    solns=solveplot(b,a,g,p,u,c,N,tmax,scale='linear',plottitle= plottitle,label=label_c,
              newplot = newplot,species='daily confirmed',models=['SCEIR','SCCEIR'])
    plt.legend()
    newplot = False

newplot = True
for i in range(5):
    c[0]=CautionFactor
    c[1]=1/CautionRetention
    c[2]=1./(N*ICUFrac*CautionICUFracs[i])
    label_c = 'CautionICUFrac %s'% CautionICUFracs[i]
    plottitle = 'Caution ICUFrac Sensitivity'
    solns=solveplot(b,a,g,p,u,c,N,tmax,scale='linear',plottitle= plottitle,label=label_c,
              newplot = newplot,species='daily confirmed',models=['SCEIR','SCCEIR'])
    plt.legend()
    newplot = False
```

#### Cumulative corona cases

```python
solns=solveplot(b,a,g,p,u,c,N,tmax,scale='linear',species='confirmed')
soln = solns[0]
solnc = solns[1]
solncc = solns[2]
plt.figure(figsize=(40,25))
plt.subplot(1,2,1)
sum0 = np.zeros(len(soln[:0]))
suma = np.zeros(len(soln[:0]))
sumb = np.zeros(len(soln[:0]))
sum0 = np.sum(soln[:,2:6],axis=1,keepdims=True)   # shape  (3650,1)
suma = np.sum(solnc[:,2:6],axis=1,keepdims=True)   # shape  (3650,1)
sumb = np.sum(solncc[:,2:6],axis=1) + solncc[:,9]  # shape  (3650,)
plt.plot(tvec,sum0)
plt.plot(tvec,suma)
plt.plot(tvec,sumb)
plt.xlabel("Time (days)")
plt.ylabel("Number per 1000000 People")
plt.legend(("SEIR No caution","SCEIR Linear caution","SCCEIR Quadratic caution"))
plt.ylim([0,100000])
plt.title('Cumulative cases')
print("")
```

## Dr. Alison's SEIR analysis

```python
# get observed growth rate r (and doubling time) for a particular variable between selected time points
#(all infected classes eventually grow at same rate during early infection)

#Don't have a simple analytic formula for r for this model due to the complexity of the stages

def growth_rate(tvec,soln,t1,t2,i):
  i1=np.where(tvec==t1)[0][0]
  i2=np.where(tvec==t2)[0][0]
  r=(np.log(soln[i2,1])-np.log(soln[i1,1]))/(t2-t1)
  DoublingTime=np.log(2)/r

  return r, DoublingTime
```

```python
(r,DoublingTime)=growth_rate(tvec,soln,10,20,1)
print("The epidemic growth rate is = {0:4.2f} per day and the doubling time {1:4.1f} days ".format(r,DoublingTime))
```

    The epidemic growth rate is = 0.08 per day and the doubling time  8.9 days

### Repeat of SEIR but with a social distancing measure that reduces transmission rate
This and subsequent sections currently unaltered from Dr. Alison Hill's notebook.

```python
bSlow=0.6*b
R0Slow=N*((bSlow[1]/(p[1]+g[1]))+(p[1]/(p[1]+g[1]))*(bSlow[2]/(p[2]+g[2])+ (p[2]/(p[2]+g[2]))*(bSlow[3]/(u+g[3]))))

solnSlow=odeint(seir,ic,tvec,args=(bSlow,a,g,p,u,N))
solnSlow=np.hstack((N-np.sum(solnSlow,axis=1,keepdims=True),solnSlow))

plt.figure(figsize=(13,5))
plt.subplot(1,2,1)
plt.plot(tvec,solnSlow)
plt.xlabel("Time (days)")
plt.ylabel("Number per 1000 People")
plt.legend(("S","E","I1","I2","I3","R","D"))
plt.ylim([0,1000])

#Same plot but on log scale
plt.subplot(1,2,2)
plt.plot(tvec,solnSlow)
plt.semilogy()
plt.xlabel("Time (days)")
plt.ylabel("Number per 1000 People")
plt.legend(("S","E","I1","I2","I3","R","D"))
plt.ylim([1,1000])

(rSlow,DoublingTimeSlow)=growth_rate(tvec,solnSlow,30,40,1)

plt.show()
print("R0 under intervention = {0:4.1f}".format(R0Slow))
print("The epidemic growth rate is = {0:4.2f} per day and the doubling time {1:4.1f} days ".format(rSlow,DoublingTimeSlow))
```

#### Compare epidemic growth with and without intervention

```python
### All infectious cases (not exposed)

plt.figure(figsize=(13,5))
plt.subplot(1,2,1)
plt.plot(tvec,np.sum(soln[:,2:5],axis=1,keepdims=True))
plt.plot(tvec,np.sum(solnSlow[:,2:5],axis=1,keepdims=True))
plt.xlabel("Time (days)")
plt.ylabel("Number per 1000 People")
plt.legend(("No intervention","Intervention"))
plt.ylim([0,1000])
plt.title('All infectious cases')
```

## COVID19 Cases vs Hospital Capacity

Depending on the severity ($I_i$) stage of COVID-19 infection, patients need different level of medical care.

Individuals in $I_1$ have "mild" infection, meaning they have cough/fever/other flu-like symptoms and may also have mild pneumonia. Mild pneumonia does not require hospitalization, although in many outbreak locations like China and South Korea all symptomatic patients are being hospitalized. This is likely to reduce spread and to monitor these patients in case they rapidly progress to worse outcome. However, it is a huge burden on the health care system.

Individuals in $I_2$ have "severe" infection, which is categorized medically as having any of the following: "dyspnea, respiratory frequency 30/min, blood oxygen saturation 93%, partial pressure of arterial oxygen to fraction of inspired oxygen ratio $<$300, lung infiltrates $>$50% within 24 to 48 hours". These individuals require hospitalization but can be treated on regular wards. They may require supplemental oxygen.

Individuals in $I_3$ have "critical" infection, which is categorized as having any of the following: "respiratory failure, septic shock, and/or multiple organ dysfunction or failure".
They require ICU-level care, generally because they need mechanical ventilation.

We consider different scenarios for care requirements. One variation between scenarios is whether we include hospitalization for all individuals or only those with severe or critical infection. Another is the care of critical patients. If ICUs are full, hospitals have protocols developed for pandemic influenza to provide mechanical ventilation outside regular ICU facility and staffing requirements. Compared to "conventional" ventilation protocols, there are "contingency" and "crisis" protocols that can be adopted to increase patient loads. These protocols involve increasing patient:staff ratios, using non-ICU beds, and involving non-critical care specialists in patient care.

```python
#Parameter sources: https://docs.google.com/spreadsheets/d/1zZKKnZ47lqfmUGYDQuWNnzKnh-IDMy15LBaRmrBcjqE

# All values are adjusted for increased occupancy due to flu season

AvailHospBeds=2.6*(1-0.66*1.1) #Available hospital beds per 1000 ppl in US based on total beds and occupancy
AvailICUBeds=0.26*(1-0.68*1.07) #Available ICU beds per 1000 ppl in US, based on total beds and occupancy. Only counts adult not neonatal/pediatric beds
ConvVentCap=0.062 #Estimated excess # of patients who could be ventilated in US (per 1000 ppl) using conventional protocols
ContVentCap=0.15 #Estimated excess # of patients who could be ventilated in US (per 1000 ppl) using contingency protocols
CrisisVentCap=0.42 #Estimated excess # of patients who could be ventilated in US (per 1000 ppl) using crisis protocols

```

### Assumptions 1
* Only severe or critical cases go to the hospital
* All critical cases require ICU care and mechanical ventilation

```python
NumHosp=soln[:,3]+soln[:,4]
NumICU=soln[:,4]

plt.figure(figsize=(13,4.8))
plt.subplot(1,2,1)
plt.plot(tvec,NumHosp)
plt.plot(np.array((0, tmax)),AvailHospBeds*np.ones(2),color='C0',linestyle=":")
plt.xlabel("Time (days)")
plt.ylabel("Number Per 1000 People")
plt.legend(("Cases Needing Hospitalization","Available Hospital Beds"))
ipeakHosp=np.argmax(NumHosp) #find peak
peakHosp=10*np.ceil(NumHosp[ipeakHosp]/10)#find time at peak
plt.ylim([0,peakHosp])

plt.subplot(1,2,2)
plt.plot(tvec,NumICU,color='C1')
plt.plot(np.array((0, tmax)),AvailICUBeds*np.ones(2),color='C1',linestyle=":")
plt.xlabel("Time (days)")
plt.ylabel("Number Per 1000 People")
plt.legend(("Cases Needing ICU","Available ICU Beds"))
ipeakICU=np.argmax(NumICU) #find peak
peakICU=10*np.ceil(NumICU[ipeakICU]/10)#find time at peak
plt.ylim([0,peakICU])
plt.ylim([0,10])

#Find time when hospitalized cases = capacity
icross=np.argmin(np.abs(NumHosp[0:ipeakHosp]-AvailHospBeds)) #find intersection before peak
TimeFillBeds=tvec[icross]

#Find time when ICU cases = capacity
icross=np.argmin(np.abs(NumICU[0:ipeakICU]-AvailICUBeds)) #find intersection before peak
TimeFillICU=tvec[icross]

plt.show()
print("Hospital and ICU beds are filled by COVID19 patients after {0:4.1f} and {1:4.1f} days".format(TimeFillBeds,TimeFillICU))
```

Note that we have not taken into account the limited capacity in the model itself. If hospitals are at capacity, then the death rate will increase, since individuals with severe and critical infection will often die without medical care. The transmission rate will probably also increase, since any informal home-care for these patients will likely not include the level of isolation/precautions used in a hospital.

#### Allow for mechanical ventilation outside of ICUs using contingency or crisis capacity

```python
plt.plot(tvec,NumICU)
plt.plot(np.array((0, tmax)),ConvVentCap*np.ones(2),linestyle=":")
plt.plot(np.array((0, tmax)),ContVentCap*np.ones(2),linestyle=":")
plt.plot(np.array((0, tmax)),CrisisVentCap*np.ones(2),linestyle=":")
plt.xlabel("Time (days)")
plt.ylabel("Number Per 1000 People")
plt.legend(("Cases Needing Mechanical Ventilation","Conventional Capacity","Contingency Capacity","Crisis Capacity"))
plt.ylim([0,peakICU])
plt.ylim([0,10])

#Find time when ICU cases = conventional capacity
icrossConv=np.argmin(np.abs(NumICU[0:ipeakICU]-ConvVentCap)) #find intersection before peak
TimeConvCap=tvec[icrossConv]
icrossCont=np.argmin(np.abs(NumICU[0:ipeakICU]-ContVentCap)) #find intersection before peak
TimeContCap=tvec[icrossCont]
icrossCrisis=np.argmin(np.abs(NumICU[0:ipeakICU]-CrisisVentCap)) #find intersection before peak
TimeCrisisCap=tvec[icrossCrisis]

plt.show()
print("Capacity for mechanical ventilation is filled by COVID19 patients after {0:4.1f} (conventional), {1:4.1f} (contingency) and {2:4.1f} (crisis) days".format(TimeConvCap,TimeContCap,TimeCrisisCap))
```

Compare to the case with intervention

```python
NumHospSlow=solnSlow[:,3]+solnSlow[:,4]
NumICUSlow=solnSlow[:,4]

plt.figure(figsize=(13,4.8))
plt.subplot(1,2,1)
plt.plot(tvec,NumHosp)
plt.plot(tvec,NumHospSlow,color='C0',linestyle="--")
plt.plot(np.array((0, tmax)),AvailHospBeds*np.ones(2),color='C0',linestyle=":")
plt.xlabel("Time (days)")
plt.ylabel("Number Per 1000 People")
plt.legend(("Cases Needing Hospitalization","Cases Needing Hospitalization (Intervetion)","Available Hospital Beds"))
plt.ylim([0,peakHosp])

plt.subplot(1,2,2)
plt.plot(tvec,NumICU,color='C1')
plt.plot(tvec,NumICUSlow,color='C1',linestyle="--")
plt.plot(np.array((0, tmax)),AvailICUBeds*np.ones(2),color='C1',linestyle=":")
plt.xlabel("Time (days)")
plt.ylabel("Number Per 1000 People")
plt.legend(("Cases Needing ICU","Cases Needing ICU (Intervetion)","Available ICU Beds"))
plt.ylim([0,peakICU])

#Find time when hospitalized cases = capacity
ipeakHospSlow=np.argmax(NumHospSlow) #find peak
icross=np.argmin(np.abs(NumHospSlow[0:ipeakHospSlow]-AvailHospBeds)) #find intersection before peak
TimeFillBedsSlow=tvec[icross]

#Find time when ICU cases = capacity
ipeakICUSlow=np.argmax(NumICUSlow) #find peak
icross=np.argmin(np.abs(NumICUSlow[0:ipeakICU]-AvailICUBeds)) #find intersection before peak
TimeFillICUSlow=tvec[icross]

plt.show()
print("With intervention, hospital and ICU beds are filled by COVID19 patients after {0:4.1f} and {1:4.1f} days".format(TimeFillBedsSlow,TimeFillICUSlow))
```

And for expanded mechanical ventilation capacity

```python
plt.plot(tvec,NumICU)
plt.plot(tvec,NumICUSlow)
plt.plot(np.array((0, tmax)),ConvVentCap*np.ones(2),linestyle=":")
plt.plot(np.array((0, tmax)),ContVentCap*np.ones(2),linestyle=":")
plt.plot(np.array((0, tmax)),CrisisVentCap*np.ones(2),linestyle=":")
plt.xlabel("Time (days)")
plt.ylabel("Number Per 1000 People")
plt.legend(("Cases Needing Mechanical Ventilation","Cases Needing Mechanical Ventilation (Intervention)","Conventional Capacity","Contingency Capacity","Crisis Capacity"))
plt.ylim([0,peakICU])

#Find time when ICU cases = conventional capacity (with intervention)
icrossConvSlow=np.argmin(np.abs(NumICUSlow[0:ipeakICUSlow]-ConvVentCap)) #find intersection before peak
TimeConvCapSlow=tvec[icrossConvSlow]
icrossContSlow=np.argmin(np.abs(NumICUSlow[0:ipeakICUSlow]-ContVentCap)) #find intersection before peak
TimeContCapSlow=tvec[icrossContSlow]
icrossCrisisSlow=np.argmin(np.abs(NumICUSlow[0:ipeakICUSlow]-CrisisVentCap)) #find intersection before peak
TimeCrisisCapSlow=tvec[icrossCrisisSlow]

plt.show()
print("Capacity for mechanical ventilation is filled by COVID19 patients after {0:4.1f} (conventional), {1:4.1f} (contingency) and {2:4.1f} (crisis) days".format(TimeConvCapSlow,TimeContCapSlow,TimeCrisisCapSlow))
```

Interpretation: While interventions that reduce infectiousness do "flatten the curve", cases are still WAY over hospital capacity. There is no way to get anywhere close to staying under hospital bed capacity or mechanical ventilation capacity without reducing $R_0<1$.

### Assumptions 2
* All cases go to the hospital
* All critical cases require ICU care and mechanical ventilation

Note: No point running this scenario because it would be even more extreme than Assumption 1 (mild cases stayed home) and Assumption 1 already lead to rapid overlow of hospital resources

### Assumptions 3
* Only severe or critical cases go to the hospital
* All critical cases require ICU care and mechanical ventilation
* When hospital capacity is exceed, individual dies

Note: Could be used to simulate expected increases in death if capacity exceeded

## Alternative Models

To be continued, including
* Assuming ~30% of cases are asymptomatic (as seen on Diamond Princess) (This would lead to a re-interpretation of the reported rates of severe and critical infection, so the prevalence of these stages would decrease)
* A parallel instead of series model of disease course (because it is unclear if it is realistic that individuals who pass through the mild stage on the way to a severe state spend as long in the mild stage as individuals who never progress)
* Including pre-symptomatic transmission (for about last ~2 days of exposed period, as estimated in some studies)

```python

```

```python

```

```python

```


Non Linear Regression Analysis

If the data shows a curvy trend, then linear regression will not produce very accurate results when compared to a non-linear regression because, as the name implies, linear regression presumes that the data is linear.
Let's learn about non linear regressions and apply an example on python. In this notebook, we fit a non-linear model to the datapoints corrensponding to China's GDP from 1960 to 2014.

Importing required libraries

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

Though Linear regression is very good to solve many problems, it cannot be used for all datasets. First recall how linear regression, could model a dataset. It models a linear relation between a dependent variable y and independent variable x. It had a simple equation, of degree 1, for example y = $2x$ + 3.

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph
y = 2*(x) + 3
y_noise = 2 * np.random.normal(size=x.size)
ydata = y + y_noise
#plt.figure(figsize=(8,6))
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

Non-linear regressions are a relationship between independent variables $x$ and a dependent variable $y$ which result in a non-linear function modeled data. Essentially any relationship that is not linear can be termed as non-linear, and is usually represented by the polynomial of $k$ degrees (maximum power of $x$).

$$ \ y = a x^3 + b x^2 + c x + d \ $$

Non-linear functions can have elements like exponentials, logarithms, fractions, and others. For example: $$ y = \log(x)$$

Or even, more complicated such as :
$$ y = \log(a x^3 + b x^2 + c x + d)$$

Let's take a look at a cubic function's graph.

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph
y = 1*(x**3) + 1*(x**2) + 1*x + 3
y_noise = 20 * np.random.normal(size=x.size)
ydata = y + y_noise
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

As you can see, this function has $x^3$ and $x^2$ as independent variables. Also, the graphic of this function is not a straight line over the 2D plane. So this is a non-linear function.

Some other types of non-linear functions are:

### Quadratic

$$ Y = X^2 $$

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph

y = np.power(x,2)
y_noise = 2 * np.random.normal(size=x.size)
ydata = y + y_noise
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

### Exponential

An exponential function with base c is defined by $$ Y = a + b c^X$$ where b ≠0, c > 0 , c ≠1, and x is any real number. The base, c, is constant and the exponent, x, is a variable.

```python
X = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph

Y= np.exp(X)

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

### Logarithmic

The response $y$ is a results of applying logarithmic map from input $x$'s to output variable $y$. It is one of the simplest form of __log()__: i.e. $$ y = \log(x)$$

Please consider that instead of $x$, we can use $X$, which can be polynomial representation of the $x$'s. In general form it would be written as
\begin{equation}
y = \log(X)
\end{equation}

```python
X = np.arange(-5.0, 5.0, 0.1)

Y = np.log(X)

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

### Sigmoidal/Logistic

$$ Y = a + \frac{b}{1+ c^{(X-d)}}$$

```python
X = np.arange(-5.0, 5.0, 0.1)

Y = 1-4/(1+np.power(3, X-2))

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

# Non-Linear Regression example

For an example, we're going to try and fit a non-linear model to the datapoints corresponding to China's GDP from 1960 to 2014. We download a dataset with two columns, the first, a year between 1960 and 2014, the second, China's corresponding annual gross domestic income in US dollars for that year.

```python
import numpy as np
import pandas as pd

#downloading dataset
!wget -nv -O china_gdp.csv https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/ML0101ENv3/labs/china_gdp.csv

df = pd.read_csv("china_gdp.csv")
df.head(10)
```

__Did you know?__ When it comes to Machine Learning, you will likely be working with large datasets. As a business, where can you host your data? IBM is offering a unique opportunity for businesses, with 10 Tb of IBM Cloud Object Storage: [Sign up now for free](http://cocl.us/ML0101EN-IBM-Offer-CC)

### Plotting the Dataset ###
This is what the datapoints look like. It kind of looks like an either logistic or exponential function. The growth starts off slow, then from 2005 on forward, the growth is very significant. And finally, it decelerate slightly in the 2010s.

```python
plt.figure(figsize=(8,5))
x_data, y_data = (df["Year"].values, df["Value"].values)
plt.plot(x_data, y_data, 'ro')
plt.ylabel('GDP')
plt.xlabel('Year')
plt.show()
```

### Choosing a model ###

From an initial look at the plot, we determine that the logistic function could be a good approximation,
since it has the property of starting with a slow growth, increasing growth in the middle, and then decreasing again at the end; as illustrated below:

```python
X = np.arange(-5.0, 5.0, 0.1)
Y = 1.0 / (1.0 + np.exp(-X))

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

The formula for the logistic function is the following:

$$ \hat{Y} = \frac1{1+e^{\beta_1(X-\beta_2)}}$$

$\beta_1$: Controls the curve's steepness,

$\beta_2$: Slides the curve on the x-axis.

### Building The Model ###
Now, let's build our regression model and initialize its parameters.

```python
def sigmoid(x, Beta_1, Beta_2):
     y = 1 / (1 + np.exp(-Beta_1*(x-Beta_2)))
     return y
```

Lets look at a sample sigmoid line that might fit with the data:

```python
beta_1 = 0.10
beta_2 = 1990.0

#logistic function
Y_pred = sigmoid(x_data, beta_1 , beta_2)

#plot initial prediction against datapoints
plt.plot(x_data, Y_pred*15000000000000.)
plt.plot(x_data, y_data, 'ro')
```

Our task here is to find the best parameters for our model. Lets first normalize our x and y:

```python
# Lets normalize our data
xdata =x_data/max(x_data)
ydata =y_data/max(y_data)
```

#### How we find the best parameters for our fit line?
we can use __curve_fit__ which uses non-linear least squares to fit our sigmoid function, to data. Optimal values for the parameters so that the sum of the squared residuals of sigmoid(xdata, *popt) - ydata is minimized.

popt are our optimized parameters.

```python
from scipy.optimize import curve_fit
popt, pcov = curve_fit(sigmoid, xdata, ydata)
#print the final parameters
print(" beta_1 = %f, beta_2 = %f" % (popt[0], popt[1]))
```

Now we plot our resulting regression model.

```python
x = np.linspace(1960, 2015, 55)
x = x/max(x)
plt.figure(figsize=(8,5))
y = sigmoid(x, *popt)
plt.plot(xdata, ydata, 'ro', label='data')
plt.plot(x,y, linewidth=3.0, label='fit')
plt.legend(loc='best')
plt.ylabel('GDP')
plt.xlabel('Year')
plt.show()
```

## Practice
Can you calculate what is the accuracy of our model?

```python
# write your code here

```

Double-click __here__ for the solution.

Want to learn more?

IBM SPSS Modeler is a comprehensive analytics platform that has many machine learning algorithms. It has been designed to bring predictive intelligence to decisions made by individuals, by groups, by systems – by your enterprise as a whole. A free trial is available through this course, available here: SPSS Modeler

Also, you can use Watson Studio to run these notebooks faster with bigger datasets. Watson Studio is IBM's leading cloud solution for data scientists, built by data scientists. With Jupyter notebooks, RStudio, Apache Spark and popular libraries pre-packaged in the cloud, Watson Studio enables data scientists to collaborate on their projects without having to install anything. Join the fast-growing community of Watson Studio users today with a free account at Watson Studio

Thanks for completing this lesson!

Author:  Saeed Aghabozorgi

Saeed Aghabozorgi, PhD is a Data Scientist in IBM with a track record of developing enterprise level applications that substantially increases clients’ ability to turn data into actionable knowledge. He is a researcher in data mining field and expert in developing advanced analytic methods like machine learning and statistical modelling on large datasets.

Copyright © 2018 Cognitive Class. This notebook and its source code are released under the terms of the MIT License.