```python
import sys
sys.path.append('../')

from torch.utils.data import TensorDataset
from torch import Tensor
from torch.distributions.multivariate_normal import MultivariateNormal

import matplotlib
import seaborn as sns; sns.set_style('white')

from main import *
plt = matplotlib.pyplot

```

MathJax.Hub.Config({
    TeX: { equationNumbers: { autoNumber: "AMS" } }
});

# Introduction
Though the Bayesian paradigm is theoretically appealing, it has proved difficult to apply in deep learning.
Consider a neural network $f_\theta: X \to Y$. For illustrative purposes, suppose the model is given by
$$
p(y|x,\theta) \propto \exp\{-|| y - f_\theta(x) ||^2/2\}
$$
Let $\varphi(\theta)$ be the prior distribution and $p(\mathcal D_n |\theta) = \Pi_{i=1}^n p(y_i | x_i, \theta)$ be the likelihood of the data $\mathcal D_n = \{(x_i,y_i)\}_{i=1}^n$. Then the posterior distribution of $\theta$ is given by
$$
p(\theta | \mathcal D_n) \propto p(\mathcal D_n | \theta) \varphi(\theta).
$$
The normalizing constant in the posterior distribution is an intractable integral. Many methods for sampling from the posterior, e.g., MCMC, variational inference, will encounter extra difficulty when $\theta$ is a high dimensional neural network weight.

Leaving aside the fact that the posterior is hard to sample, let's see why we should care about the posterior distribution in the first place. In the Bayesian paradigm, we predict using a distribution
$$
p(y|x, \mathcal D_n) = \int p(y|x,\theta) p(\theta|\mathcal D_n) \,d\theta
$$
Thus instead of providing a point estimate $\hat y$ for input $\hat x$, we can form an entire distribution estimate which lends itself naturally to uncertainty quantification.

There is yet another advantage to Bayesian prediction, perhaps less well documented in the deep learning community. Neural networks are singular models. In terms of generalization, Bayesian prediction is better than MAP or MLE for singular models. This phenomenon is what we seek to elucidate in this work.

# Generalization error

Suppose $\hat q_n(y|x)$ is some estimate of the true unknown conditional density $q(y|x)$. The generalization error of the prediction $\hat q(y|x)$ is defined as
$$
G(n) = KL (q(y|x) || \hat q(y|x) ) = \int \int q(y|x) \log \frac{q(y|x)}{\hat q_n(y|x)} \,dy q(x) \,dx
$$
We've written this in terms of the sample size $n$ to remind ourselves that $\hat q_n$ is formed using $\mathcal D_n$.

We will consider three predictors of $q(y|x)$:
+ Bayesian predictive distribution $\hat q(y|x) = p(y|x,\mathcal D_n)$
+ MAP $\hat q(y|x) = p(y|x,\theta_n^{MAP}$)
+ MLE $\hat q(y|x) = p(y|x,\theta_n^{MLE}$)

To average out the randomness in the dataset $\mathcal D_n$ used to form the predictors, we will ultimately look at the average generalization error
\begin{equation}
E_n G(n)
\label{eq:avgGn}
\end{equation}
where $E_n$ denotes expectation over the dataset $\mathcal D_n$. In simulations, we calculate the average generalization error using a held-out-test set $T_{n'} = \{(x_i',y_i')\}_{i=1}^{n'}$ as
\begin{equation}
\frac{1}{n'} \sum_{i=1}^{n'} \log q(y_i'|x_i') - E_n \frac{1}{n'} \sum_{i=1}^{n'} \log \hat q_n(y_i'|x_i')
\label{eq:computed_avgGn}
\end{equation}
Assume the held-out test set is large enough so that the difference between \eqref{eq:avgGn} and \eqref{eq:computed_avgGn} is negligible. We will refer to them interchangeably as the average generalization error.

# Bayes versus MAP versus MLE in singular models

A singular model is ...

Neural networks are singular models because ...

Since $p(y|x,\theta)$ is singular, we look to Watanabe's singular learnng theory to understand the generalization error of the various predictors considered. Assume the true distribution is realizable by the model, i.e., $q(y|x) = p(y|x,\theta_0)$ for some $\theta_0$ (though we will investigate violations of this assumption in the experiments).

We have the following asymptotic expansions of the generalization error for singular models.
+ For Bayes, $E_n G(n) = \lambda/n + o(1/n)$ where $\lambda \in \mathbb Q^+$ is a positive rational number known as the learning coefficient. The learning coefficient is completely determined by the the truth-model-prior triplet $( q(y|x), p(y|x,\theta), \varphi(\theta) )$. Most of the time $\lambda << dim(\theta)/2$
+ For MAP and MLE, $E_n G(n) = C/n + o(1/n)$ (different $C$'s for MAP and MLE). Basically $C$ is the maximum of some Gaussian process, which can easily be greater than $dim(\theta)/2$. Watanabe's Main Theorem 6.4 gives the precise formulation.

# Last layer Bayesian

Though the Bayes predictive distribution is provably superior to MAP and MLE, it relies on the intractable posterior distribution. As a workaround, we form the predictive distribution only in the last layer(s) of a deep neural network. Since a neural network $f_\theta$ is naturally hierarchical, we break it up into $f_\theta = h_w \circ g_v$ where $\theta = (v,w)$ with $dim(w)$ being small enough to manageably perform MCMC.

Let $\theta_{map} = (v_{map},w_{map})$. Let $\tilde x_i = g_{v_{map}}(x_i)$. Define a new transformed dataset $\tilde{\mathcal D_n} = \{(\tilde x_i, y_i) \}_{i=1}^n$. We perform MCMC to sample the posterior over $w$:
$$
p(w | \tilde{\mathcal D_n}) \propto p(\tilde{\mathcal D_n} | w) \varphi(w) = \Pi_{i=1}^n \exp\{-|| y_i - h_w \circ g_{v_{map}}(x_i) ||^2/2\} \varphi(w)
$$

Define the last layer Bayesian predictive distribution to be
$$
p_{LLB}(y|x, \mathcal D_n) = \int p(y|x,(v_{map},w)) p(w|\tilde{\mathcal D_n}) \,dw
$$
We have to be careful when we speak of the generalization error of the LLB predictive distribution. For a proper comparison to $E_n G_{map}(n)$ or $E_n G_{mle}(n)$, we have to look at
\begin{equation}
E_n G_{LLB}(n) =  KL (q(y|x) || p_{LLB}(y|x, \mathcal D_n) )
\label{G_LLB}
\end{equation}
where $E_n$ averages out the randomness of both $D_n$ and  $v_{map}$.

An alternative is to condition on $v_{map}$, using $g_{v_{map}}$ as a feature extractor in a preprocessing step. Then, assuming realizability $q(y|x) = p(y|x,(v_0,w_0))$ we may examine the generalizaton error
\begin{equation}
E_{\mathcal D_n | v_{map}} KL( p(y|x,(v_{map},w_0)) || p_{LLB}(y|x, \mathcal D_n) )
\label{G_LLB_vfixed}
\end{equation}
The average generalization error in \eqref{G_LLB_vfixed} is distinctly different from the one for LLB in \eqref{G_LLB}. The nice thing about \eqref{G_LLB_vfixed} is that we know its asymptotic expansion is $\lambda/n$ where $\lambda$ corresponds to the triplet $( p(y|x,(v_{map},w_0)), p(y| x, (v_{map},w)), \varphi(w))$. For certain functions $h_w$ where the true $\lambda$ is known, we can verify this in the experiments. (Not implemented yet).

So we have various (average) generalization errors we are interested in
+ $E_n G_{post}(n)$ corresponding to the full posterior over $\theta$
+ $E_n G_{map}(n)$
+ $E_n G_{mle}(n)$
+ $E_n G_{LLB}(n)$
+ Generalization error in \eqref{G_LLB_vfixed}. (don't have a name for it yet)

Theory guarantees the following relationships:
+ $E_n G_{post}(n)$ has smaller learning coefficient than either $E_n G_{map}(n)$ or $E_n G_{mle}(n)$
+ \eqref{G_LLB_vfixed} = known $\lambda$/n + o(1/n) for certain $h_w$.

The question is, what is the behavior of $E_n G_{LLB}(n)$? Two hunches
+ Though LLB is different from full posterior predictive distribution, it's still better than map or mle, i.e., $E_n G_{LLB}(n) \le E_n G_{map}(n)$.
+ $E_n G_{LLB}(n)$ is different from \eqref{G_LLB_vfixed} but perhaps asymptotially they share the same leading term $\lambda/n$?
We'll set out to investigate these conjectures in the following experiment.

# Experiments

Consider a neural network where $h_w$ is a reduced-rank regression network and $g_v$ is a block of linear composed with ReLU layers.

```python
class Model(nn.Module):

    def __init__(self, input_dim, output_dim, ffrelu_hidden, rr_hidden):
        super(Model, self).__init__()

        self.feature_map = nn.Sequential(
            nn.Linear(input_dim, ffrelu_hidden),
            nn.ReLU(),
            nn.Linear(ffrelu_hidden, ffrelu_hidden),
            nn.ReLU(),
            nn.Linear(ffrelu_hidden, input_dim),
        )

        self.rr = nn.Sequential(
            nn.Linear(input_dim, rr_hidden, bias=False),
            nn.Linear(rr_hidden, output_dim, bias=False)
        )

    def forward(self, x):
        x = self.feature_map(x)
        return self.rr(x)

```

We use the following values for the parameters
+ input_dim = output_dim = rr_hidden = 3
+ ffrelu_hidden = 5

We generate the training data as follows:
+ $X \sim N(0,\sigma_x^2 I_3)$
+ realizability
    + realizable $Y \sim N(f_{\theta_0}(X),\sigma_y^2 I_3)$
    + not realizable $Y \sim N(h_{w_0}(X),\sigma_y^2 I_3)$

For the testing data, we consider a possible change in the support of $X$:
+ $X \sim \alpha N(0,\sigma_x^2 I_3)$ for some scale $\alpha \in \mathbb R^+$.

```python
def get_data(args):

    train_size = int(args.n)
    valid_size = int(args.n * 0.5)
    test_size = int(10000)

    a = Normal(0.0, 1.0)
    a_params = 0.2 * a.sample((args.input_dim, args.rr_hidden))
    b = Normal(0.0, 1.0)
    b_params = 0.2 * b.sample((args.rr_hidden, args.output_dim))

    X_rv = MultivariateNormal(torch.zeros(args.input_dim), torch.eye(args.input_dim))
    y_rv = MultivariateNormal(torch.zeros(args.output_dim), torch.eye(args.output_dim))
    true_model = Model(args.input_dim, args.output_dim, args.ffrelu_hidden, args.rr_hidden)
    true_model.eval()

    with torch.no_grad():
        # training +valid data
        X = X_rv.sample(torch.Size([train_size+valid_size]))
        if args.realizable:
            true_mean = true_model(X)
        else:
            true_mean = torch.matmul(torch.matmul(X, a_params), b_params)
        y = true_mean + y_rv.sample(torch.Size([train_size+valid_size]))
        dataset_train, dataset_valid = torch.utils.data.random_split(TensorDataset(X, y),[train_size,valid_size])

        # testing data
        X = args.X_test_std * X_rv.sample(torch.Size([test_size]))
        if args.realizable:
            true_mean = true_model(X)
        else:
            true_mean = torch.matmul(torch.matmul(X, a_params), b_params)
        y = true_mean + y_rv.sample(torch.Size([test_size]))
        dataset_test = TensorDataset(X, y)
        oracle_mse = (torch.norm(y - true_mean, dim=1)**2).mean()
        entropy = -torch.log((2 * np.pi) ** (-args.output_dim / 2) * torch.exp(
            -(1 / 2) * torch.norm(y - true_mean, dim=1) ** 2)).mean()

        train_loader = torch.utils.data.DataLoader(dataset_train, batch_size=args.batchsize, shuffle=True)
        valid_loader = torch.utils.data.DataLoader(dataset_valid, batch_size=args.batchsize, shuffle=True)
        test_loader = torch.utils.data.DataLoader(dataset_test, batch_size=args.batchsize, shuffle=True)

    return train_loader, valid_loader, test_loader, oracle_mse, entropy
```

Note that the way MAP training is usually conducted in deep learning may involve early stopping. Watanabe's theory does not account for early stopping. We will try MAP training with and without early stopping.

```python
def map_train(args, X_train, Y_train, X_valid, Y_valid, X_test, Y_test, oracle_mse):

    model = Model(args.input_dim, args.output_dim, args.ffrelu_hidden, args.rr_hidden)
    opt = optim.SGD(model.parameters(), lr=1e-3, momentum=0.9, weight_decay=5e-4)
    early_stopping = EarlyStopping(patience=10, verbose=False, taskid=args.taskid)

    # TODO: is it necessary to implement mini batch SGD?
    for it in range(5000):
        model.train()
        y_pred = model(X_train).squeeze()
        l = (torch.norm(y_pred - Y_train, dim=1)**2).mean()
        l.backward()
        opt.step()
        opt.zero_grad()

        model.eval()
        with torch.no_grad():
            valid_loss = (torch.norm(model(X_valid).squeeze() - Y_valid, dim=1)**2).mean()

        if it % 100 == 0:
            model.eval()
            ytest_pred = model(X_test).squeeze()
            test_loss = (torch.norm(ytest_pred - Y_test, dim=1)**2).mean()
            print('MSE: train {:.3f}, validation {:.3f}, test {:.3f}, oracle on test set {:.3f}'.format(l.item(), valid_loss, test_loss.item(), oracle_mse))

        if args.early_stopping:
            early_stopping(valid_loss, model)
            if early_stopping.early_stop:
                print("Early stopping")
                break

    return model

```

The last function we will need is for drawing samples from the last layers. Although the code supports implicit variational inference (and even explicit variational inference), only the MCMC (NUTS) will be considered in the experiments.

```python
def lastlayer_approxinf(model, args, X_train, Y_train, X_valid, Y_valid, X_test, Y_test):

    transformed_X_train = model.feature_map(X_train)
    transformed_X_valid = model.feature_map(X_valid)
    transformed_X_test = model.feature_map(X_test)

    transformed_train_loader = torch.utils.data.DataLoader(
        TensorDataset(Tensor(transformed_X_train), torch.as_tensor(Y_train, dtype=torch.long)),
        batch_size=args.batchsize, shuffle=True)
    transformed_valid_loader = torch.utils.data.DataLoader(
        TensorDataset(Tensor(transformed_X_valid), torch.as_tensor(Y_valid, dtype=torch.long)),
        batch_size=args.batchsize, shuffle=True)

    if args.posterior_method == 'ivi':

        # parameters for train_implicitVI
        mc = 1
        beta_index = 0
        args.betas = [1.0]
        saveimgpath = None
        args.dataset = 'reducedrank_synthetic'
        args.H = args.rr_hidden

        # TODO: strip train_implicitVI to simplest possible inputs
        G = train_implicitVI(transformed_train_loader, transformed_valid_loader, args, mc, beta_index, saveimgpath)
        eps = torch.randn(args.R, args.epsilon_dim)
        sampled_weights = G(eps)
        list_of_param_dicts = weights_to_dict(args, sampled_weights)

        pred_prob = 0
        output_dim = transformed_X_test.shape[1]
        for param_dict in list_of_param_dicts:
            mean = torch.matmul(torch.matmul(transformed_X_test, param_dict['a']), param_dict['b'])
            pred_prob += (2 * np.pi) ** (-output_dim / 2) * torch.exp(-(1 / 2) * torch.norm(Y_test - mean, dim=1) ** 2)

    elif args.posterior_method == 'mcmc':

        wholex = transformed_train_loader.dataset[:][0]
        wholey = transformed_train_loader.dataset[:][1]
        beta = 1.0

        kernel = NUTS(conditioned_pyro_rr, adapt_step_size=True)
        mcmc = MCMC(kernel, num_samples=args.R, warmup_steps=args.num_warmup, disable_progbar=True)
        mcmc.run(pyro_rr, wholex, wholey, args.rr_hidden, beta)
        sampled_weights = mcmc.get_samples()

        pred_prob = 0
        output_dim = wholey.shape[1]
        for r in range(0, args.R):
            mean = torch.matmul(torch.matmul(transformed_X_test, sampled_weights['a'][r,:,:]), sampled_weights['b'][r,:,:])
            pred_prob += (2 * np.pi) ** (-output_dim / 2) * torch.exp(-(1 / 2) * torch.norm(Y_test - mean, dim=1) ** 2)

    return -torch.log(pred_prob / args.R).mean()

```

We set the arguments below. We are interested in examining the following three factors
+ realizability or not
+ support of $X_{test}$
+ early stopping or not

For the realizable plus non-early-stopping setting, we expect the learning coefficient for the last layer Bayes predictive distribution to match the theoretically known values of $\lambda$ for reduced rank regression. Watanabe's theory, as it stands, seems unable to say anything meaningful in the case $q$ is unrealizable by the model or in the case that MAP is trained using early stopping.

The computational overhead comes from the following variables:
+ num_warmup: burn-in for MCMC
+ MCs: number of training datasets for assessing $E_n$
+ num_n: number of sample sizes considered for drawing learning curve $1/n$ versus $E_n G(n)$
+ R: number of MCMC samples from the posterior

We've set these numbers to be very small for the purpose of running quickly in the notebook.

```python
class Args:

    taskid = 1

    input_dim = 3
    output_dim = 3
    X_test_std = 1.0 # play around with
    realizable = 1 # play around with

    ffrelu_hidden = 5
    rr_hidden = 3
    early_stopping = 0 # play around with

    posterior_method = 'mcmc'
    num_warmup = 10

    batchsize = 50
    #   epsilon_mc = 100
    #   epochs = 100
    #   pretrainDepochs = 100
    #   trainDepochs = 20
    #   n_hidden_D = 128
    #   num_hidden_layers_D = 1
    #   n_hidden_G = 128
    #   num_hidden_layers_G = 1
    #   lr_primal = 0.01
    #   lr_dual = 0.001

    MCs = 5
    R = 10
    num_n = 10

    no_cuda = True
    log_interval = 500

args=Args()

args.cuda = not args.no_cuda and torch.cuda.is_available()

# change to boolean
if args.early_stopping == 0:
    args.early_stopping = False
else:
    args.early_stopping = True
if args.realizable == 0:
    args.realizable = False
else:
    args.realizable = True

args.epsilon_dim = args.rr_hidden*(args.input_dim + args.output_dim)
# TODO: w_dim and total_param_count depend on model and shouldn't be hardcoded as follows
args.w_dim = args.rr_hidden*(args.input_dim + args.output_dim)
total_param_count = (args.input_dim + args.rr_hidden + args.input_dim) * args.rr_hidden + args.w_dim

```

Below we plot the learning curve, i.e., $n$ versus $E_n G(n)$. (Actually we'll plot $1/n$ versus $E_n G(n)$ to easily assess whether the slope matches the theoretical $\lambda$.)

```python
avg_lastlayerbayes_gen_err = np.array([])
std_lastlayerbayes_gen_err = np.array([])
avg_map_gen_err = np.array([])
std_map_gen_err = np.array([])
avg_entropy = np.array([])
std_entropy = np.array([])

n_range = np.round(1/np.linspace(1/200,1/10000,args.num_n))

for n in n_range:

    map_gen_err = np.empty(args.MCs)
    lastlayerbayes_gen_err = np.empty(args.MCs)
    entropy_array = np.empty(args.MCs)

    args.n = n

    for mc in range(0, args.MCs):

        train_loader, valid_loader, test_loader, oracle_mse, entropy = get_data(args)
        entropy_array[mc] = entropy

        X_train = train_loader.dataset[:][0]
        Y_train = train_loader.dataset[:][1]
        X_valid = valid_loader.dataset[:][0]
        Y_valid = valid_loader.dataset[:][1]
        X_test = test_loader.dataset[:][0]
        Y_test = test_loader.dataset[:][1]

        model = map_train(args, X_train, Y_train, X_valid, Y_valid, X_test, Y_test, oracle_mse)

        model.eval()
        map_gen_err[mc] = -torch.log((2*np.pi)**(-args.output_dim /2) * torch.exp(-(1/2) * torch.norm(Y_test-model(X_test), dim=1)**2)).mean() - entropy

        Bmap = list(model.parameters())[-1]
        Amap = list(model.parameters())[-2]
        params = (args.input_dim, args.output_dim, np.linalg.matrix_rank(torch.matmul(Bmap, Amap).detach().numpy()), args.rr_hidden)
        trueRLCT = theoretical_RLCT('rr', params)
        print('true RLCT {}'.format(trueRLCT))

        lastlayerbayes_gen_err[mc] = lastlayer_approxinf(model, args, X_train, Y_train, X_valid, Y_valid, X_test, Y_test) - entropy

        print('n = {}, mc {}, gen error: map {}, bayes last layer {}'
              .format(n, mc, map_gen_err[mc], lastlayerbayes_gen_err[mc]))

    print('average gen error: MAP {}, bayes {}'
          .format(map_gen_err.mean(), lastlayerbayes_gen_err.mean()))

    avg_lastlayerbayes_gen_err = np.append(avg_lastlayerbayes_gen_err, lastlayerbayes_gen_err.mean())
    std_lastlayerbayes_gen_err = np.append(std_lastlayerbayes_gen_err, lastlayerbayes_gen_err.std())
    avg_map_gen_err = np.append(avg_map_gen_err, map_gen_err.mean())
    std_map_gen_err = np.append(std_map_gen_err, map_gen_err.std())
    avg_entropy = np.append(avg_entropy, entropy_array.mean())
    std_entropy = np.append(std_entropy, entropy_array.std())

print('avg LLB gen err {}, std {}'.format(avg_lastlayerbayes_gen_err, std_lastlayerbayes_gen_err))
print('avg MAP gen err {}, std {}'.format(avg_map_gen_err, std_map_gen_err))

if args.realizable:
    ols_map = OLS(avg_map_gen_err, 1 / n_range).fit()
    map_slope = ols_map.params[0]

    ols_llb = OLS(avg_lastlayerbayes_gen_err, 1 / n_range).fit()
    llb_intercept = 0.0
    llb_slope = ols_llb.params[0]
else:
    ols_map = OLS(avg_map_gen_err, add_constant(1 / n_range)).fit()
    map_slope = ols_map.params[1]

    ols_llb = OLS(avg_lastlayerbayes_gen_err, add_constant(1 / n_range)).fit()
    llb_intercept = ols_llb.params[0]
    llb_slope = ols_llb.params[1]

print('estimated RLCT {}'.format(llb_slope))
#
fig, ax = plt.subplots()
ax.errorbar(1/n_range, avg_lastlayerbayes_gen_err, yerr=std_lastlayerbayes_gen_err, fmt='-o', c='r', label='En G(n) for last layer Bayes predictive')
ax.errorbar(1/n_range, avg_map_gen_err, yerr=std_map_gen_err, fmt='-o', c='g', label='En G(n) for MAP')
plt.plot(1 / n_range, llb_intercept + llb_slope / n_range, 'r--', label='ols fit for last-layer-Bayes')
plt.xlabel('1/n')
plt.title('map slope {:.2f}, parameter count {}, LLB slope {:.2f}, true RLCT {}'.format(map_slope, total_param_count, llb_slope, trueRLCT))
plt.legend()
plt.savefig('taskid{}.png'.format(args.taskid))
plt.show()
```

The graph produced plots $1/n$ versus average generalizaton error $E_n G(n)$ for MAP predictor and LLB predictive distribution. Vertical bars indicate 1 std deviation over the different Monte Carlo training-testing sets. The LLB learning coefficient (LLB slope) indicated in the title of the graph should match the "true RLCT." In addition, the MAP learning coefficient (map slope) should be bigger than the LLB slope according to theory.

```python

```

```python

```

Below we plot the learning curve, i.e., $n$ versus $E_n G(n)$. (Actually we'll plot $1/n$ versus $E_n G(n)$ to easily assess whether the slope matches the theoretical $\lambda$.)

```python
avg_lastlayerbayes_gen_err = np.array([])
std_lastlayerbayes_gen_err = np.array([])
avg_map_gen_err = np.array([])
std_map_gen_err = np.array([])
avg_entropy = np.array([])
std_entropy = np.array([])

n_range = np.round(1/np.linspace(1/200,1/10000,args.num_n))

for n in n_range:

    map_gen_err = np.empty(args.MCs)
    lastlayerbayes_gen_err = np.empty(args.MCs)
    entropy_array = np.empty(args.MCs)

    args.n = n

    for mc in range(0, args.MCs):

        train_loader, valid_loader, test_loader, oracle_mse, entropy = get_data(args)
        entropy_array[mc] = entropy

        X_train = train_loader.dataset[:][0]
        Y_train = train_loader.dataset[:][1]
        X_valid = valid_loader.dataset[:][0]
        Y_valid = valid_loader.dataset[:][1]
        X_test = test_loader.dataset[:][0]
        Y_test = test_loader.dataset[:][1]

        model = map_train(args, X_train, Y_train, X_valid, Y_valid, X_test, Y_test, oracle_mse)

        model.eval()
        map_gen_err[mc] = -torch.log((2*np.pi)**(-args.output_dim /2) * torch.exp(-(1/2) * torch.norm(Y_test-model(X_test), dim=1)**2)).mean() - entropy

        Bmap = list(model.parameters())[-1]
        Amap = list(model.parameters())[-2]
        params = (args.input_dim, args.output_dim, np.linalg.matrix_rank(torch.matmul(Bmap, Amap).detach().numpy()), args.rr_hidden)
        trueRLCT = theoretical_RLCT('rr', params)
        print('true RLCT {}'.format(trueRLCT))

        lastlayerbayes_gen_err[mc] = lastlayer_approxinf(model, args, X_train, Y_train, X_valid, Y_valid, X_test, Y_test) - entropy

        print('n = {}, mc {}, gen error: map {}, bayes last layer {}'
              .format(n, mc, map_gen_err[mc], lastlayerbayes_gen_err[mc]))

    print('average gen error: MAP {}, bayes {}'
          .format(map_gen_err.mean(), lastlayerbayes_gen_err.mean()))

    avg_lastlayerbayes_gen_err = np.append(avg_lastlayerbayes_gen_err, lastlayerbayes_gen_err.mean())
    std_lastlayerbayes_gen_err = np.append(std_lastlayerbayes_gen_err, lastlayerbayes_gen_err.std())
    avg_map_gen_err = np.append(avg_map_gen_err, map_gen_err.mean())
    std_map_gen_err = np.append(std_map_gen_err, map_gen_err.std())
    avg_entropy = np.append(avg_entropy, entropy_array.mean())
    std_entropy = np.append(std_entropy, entropy_array.std())

print('avg LLB gen err {}, std {}'.format(avg_lastlayerbayes_gen_err, std_lastlayerbayes_gen_err))
print('avg MAP gen err {}, std {}'.format(avg_map_gen_err, std_map_gen_err))

if args.realizable:
    ols_map = OLS(avg_map_gen_err, 1 / n_range).fit()
    map_slope = ols_map.params[0]

    ols_llb = OLS(avg_lastlayerbayes_gen_err, 1 / n_range).fit()
    llb_intercept = 0.0
    llb_slope = ols_llb.params[0]
else:
    ols_map = OLS(avg_map_gen_err, add_constant(1 / n_range)).fit()
    map_slope = ols_map.params[1]

    ols_llb = OLS(avg_lastlayerbayes_gen_err, add_constant(1 / n_range)).fit()
    llb_intercept = ols_llb.params[0]
    llb_slope = ols_llb.params[1]

print('estimated RLCT {}'.format(llb_slope))
#
fig, ax = plt.subplots()
ax.errorbar(1/n_range, avg_lastlayerbayes_gen_err, yerr=std_lastlayerbayes_gen_err, fmt='-o', c='r', label='En G(n) for last layer Bayes predictive')
ax.errorbar(1/n_range, avg_map_gen_err, yerr=std_map_gen_err, fmt='-o', c='g', label='En G(n) for MAP')
plt.plot(1 / n_range, llb_intercept + llb_slope / n_range, 'r--', label='ols fit for last-layer-Bayes')
plt.xlabel('1/n')
plt.title('map slope {:.2f}, parameter count {}, LLB slope {:.2f}, true RLCT {}'.format(map_slope, total_param_count, llb_slope, trueRLCT))
plt.legend()
plt.savefig('taskid{}.png'.format(args.taskid))
plt.show()
```

The graph produced plots $1/n$ versus average generalizaton error $E_n G(n)$ for MAP predictor and LLB predictive distribution. Vertical bars indicate 1 std deviation over the different Monte Carlo training-testing sets. The LLB learning coefficient (LLB slope) indicated in the title of the graph should match the "true RLCT." In addition, the MAP learning coefficient (map slope) should be bigger than the LLB slope according to theory.

# Related work

Is this last-layer-Bayesian approach common? Has anyone written about this formally? In a quick search, I can only find rough heuristics similar in spirit to what I'm doing:

+ [blog post](https://towardsdatascience.com/probabilistic-machine-learning-series-post-1-c8809652dd60) uses LSTM as feature extractor then performs Bayesian inference on linear last layer
+ [Snoek et. al 2017 ICML](http://proceedings.mlr.press/v37/snoek15.pdf) calls this approach the well-known adaptive basis regression
+ [Kristiadi et. al 2020 ICML](https://arxiv.org/pdf/2002.10118.pdf) appends linear last layer to beginning ReLU blocks, then uses Laplace approximation to perform Bayesian inference in the last layer.

```python

```


# Введение в численные методы оптимизации (Ю. Е. Нестеров Введение в выпуклую оптимизацию, гл. 1 $\S$ 1.1)

 1. Обзор материала весеннего семестра
 2. Постановка задачи
 3. Общая схема решения
 4. Сравнение методов оптимизации

## Обзор материала весеннего семестра

Также на [странице курса](https://github.com/amkatrutsa/MIPT-Opt/blob/master/Spring2021/README.md).

1. Методы решения задач **безусловной** оптимизации
    - Градиентный спуск и способы его ускорения
    - Метод Ньютона
    - Квазиньютоновские методы
    - Метод сопряжённых градиентов
    - Решение задачи наименьших квадратов
    - Безградиентные методы
    - Стохастические методы
2. Методы решения задач **условной** оптимизации
    - Методы проекции градиента и условного градиента
    - Проксимальные методы
    - Методы штрафных и барьерных функций
    - Метод модифицированой функции Лагранжа

## Постановка задачи

\begin{equation}
\begin{split}
& \min_{x \in S} f_0(x)\\
\text{s.t. } & f_j(x) = 0, \; j = 1,\ldots,m\\
& g_k(x) \leq 0, \; k = 1,\ldots,p
\end{split}
\end{equation}
где $S \subseteq \mathbb{R}^n$, $f_j: S \rightarrow \mathbb{R}, \; j = 0,\ldots,m$, $g_k: S \rightarrow \mathbb{R}, \; k=1,\ldots,p$

Все функции как минимум непрерывны.

Важный факт: задачи **нелинейной** оптимизации

в их самой общей форме являются **численно неразрешимыми**!

## Аналитические результаты
- Необходимое условие первого порядка:

если $x^*$ точка локального минимума дифференцируемой функции $f(x)$, тогда

$$
f'(x^*) = 0
$$

- Необходимое условие второго порядка

если $x^*$ точка локального минимума дважды дифференцируемой функции $f(x)$, тогда

$$
f'(x^*) = 0 \quad \text{и} \quad f''(x^*) \succeq 0
$$

- Достаточное условие:

пусть $f(x)$ дважды дифференцируемая функция, и пусть точка $x^*$ удовлетворяет условиям

$$
f'(x^*) = 0 \quad f''(x^*) \succ 0,
$$

тогда $x^*$ является точкой строго локального минимума функции $f(x)$.

**Замечание**: убедитесь, что Вы понимаете, как доказывать эти

результаты!

## Особенности численного решения

1. Точно решить задачу принципиально невозможно из-за погрешности машинной арифметики
2. Необходимо задать критерий обнаружения решения
3. Необходимо определить, какую информацию о задаче использовать

## Общая итеративная схема

Дано: начальное приближение $x$, требуемая точность $\varepsilon$.

```python
def GeneralScheme(x, epsilon):

    while StopCriterion(x) > epsilon:

        OracleResponse = RequestOracle(x)

        UpdateInformation(I, x, OracleResponse)

        x = NextPoint(I, x)

    return x
```

### Вопросы
1. Какие критерии остановки могут быть?
2. Что такое оракул и зачем он нужен?
3. Что такое информационная модель?
4. Как вычисляется новая точка?

#### Критерии остановки
1. Сходимость по аргументу:
$$
\| x_k - x^*  \|_2 < \varepsilon
$$
2. Сходимость по функции:
$$
\| f_k - f^* \|_2 < \varepsilon
$$
3. Выполнение необходимого условия
$$
\| f'(x_k) \|_2 < \varepsilon
$$

Но ведь $x^*$ неизвестна!

Тогда

\begin{align*}
& \|x_{k+1} - x_k \| = \|x_{k+1} - x_k + x^* - x^* \| \leq \\
& \|x_{k+1} - x^* \| + \| x_k - x^* \| \leq 2\varepsilon
\end{align*}

Аналогично для сходимости по функции,

однако иногда можно оценить $f^*$!

**Замечание**: лучше использовать относительные изменения

этих величин!

Например $\dfrac{\|x_{k+1} - x_k \|_2}{\| x_k \|_2}$

#### Что такое оракул?
**Определение**: оракулом называют некоторое абстрактное

устройство, которое отвечает на последовательные вопросы

метода

Аналогия из ООП:

- оракул - это виртуальный метод базового класса
- каждая задача - производный класс
- оракул определяется для каждой задачи отдельно согласно общему определению в базовом классе

**Концепция чёрного ящика**
1. Единственной информацией, получаемой в ходе работы итеративного метода, являются ответы оракула
2. Ответы оракула являются *локальными*

#### Информация о задаче
1. Каждый ответ оракула даёт **локальную** информацию о поведении функции в точке
2. Агрегируя все полученные ответы оракула, обновляем информацию о **глобальном** виде целевой функции:
    - кривизна
    - направление убывания
    - etc

#### Вычисление следующей точки

$$
x_{k+1} = x_{k} + \alpha_k h_k
$$

- **Линейный поиск**: фиксируется направление $h_k$ и производится поиск по этому направлению "оптимального" значения  $\alpha_k$
- **Метод доверительных областей**: фиксируется допустимый размер *области* по некоторой норме $\| \cdot  \| \leq \alpha$ и *модель* целевой функции, которая хорошо её аппроксимирует в выбранной области.

    Далее производится поиск направления $h_k$, минимизирующего модель целевой функции и не выводящего точку $x_k + h_k$ за пределы доверительной области

#### Вопросы
1. Как выбрать $\alpha_k$?
2. Как выбрать $h_k$?
3. Как выбрать модель?
4. Как выбрать область?
5. Как выбрать размер области?

    В курсе рассматривается только линейный поиск!

Однако несколько раз копцепция метода доверительных областей

будет использована.

## Как сравнивать методы оптимизации?
Для заданного класса задач сравнивают следующие величины:
1. Сложность
    - аналитическая: число обращений к оракулу для решения задачи с точностью $\varepsilon$
    - арифметическая: общее число всех вычислений, необходимых для решения задачи с точностью $\varepsilon$
2. Скорость сходимости
3. Эксперименты

### Скорости сходимости

_1._ Сублинейная

$$
\| x_{k+1} - x^* \|_2 \leq C k^{\alpha},
$$

где $\alpha < 0$ и $ 0 < C < \infty$

_2._ Линейная (геометрическая прогрессия)

$$
\| x_{k+1} - x^* \|_2 \leq Cq^k,
$$

где $q \in (0, 1)$ и $ 0 < C < \infty$

_3._ Сверхлинейная

$$
\| x_{k+1} - x^* \|_2 \leq Cq^{k^p},
$$

где $q \in (0, 1)$, $ 0 < C < \infty$ и $p > 1$

_4._ Квадратичная

$$
\| x_{k+1} - x^* \|_2 \leq C\| x_k - x^* \|^2_2, \qquad \text{или} \qquad \| x_{k+1} - x^* \|_2 \leq C q^{2^k}
$$

где $q \in (0, 1)$ и $ 0 < C < \infty$

```python
%matplotlib inline
import matplotlib.pyplot as plt

USE_COLAB = False
if not USE_COLAB:
    plt.rc("text", usetex=True)

import numpy as np
C = 10
alpha = -0.5
q = 0.9
num_iter = 10
sublinear = np.array([C * k**alpha for k in range(1, num_iter + 1)])
linear = np.array([C * q**k for k in range(1, num_iter + 1)])
superlinear = np.array([C * q**(k**2) for k in range(1, num_iter + 1)])
quadratic = np.array([C * q**(2**k) for k in range(1, num_iter + 1)])
plt.figure(figsize=(12,8))
plt.semilogy(np.arange(1, num_iter+1), sublinear,
             label=r"Sublinear, $\alpha = -0.5$", linewidth=5)
plt.semilogy(np.arange(1, num_iter+1), superlinear, linewidth=5,
             label=r"Superlinear, $q = 0.5, p=2$")
plt.semilogy(np.arange(1, num_iter+1), linear,
             label=r"Linear, $q = 0.5$", linewidth=5)
plt.semilogy(np.arange(1, num_iter+1), quadratic,
             label=r"Quadratic, $q = 0.5$", linewidth=5)
plt.xlabel("Number of iterations, $k$", fontsize=28)
plt.ylabel("Error rate upper bound", fontsize=28)
plt.legend(loc="best", fontsize=26)
plt.xticks(fontsize = 28)
_ = plt.yticks(fontsize = 28)
```

### Значение теорем сходимости (Б.Т. Поляк Введение в оптимизацию, гл. 1, $\S$ 6)
1. Что дают теоремы сходимости
     - класс задач, для которых можно рассчитывать на применимость метода (важно не завышать условия!)
         - выпуклость
         - гладкость
     - качественное поведение метода
         - существенно ли начальное приближение
         - по какому функционалу есть сходимость
     - оценку скорости сходимости
         - теоретическая оценка поведения метода без проведения экспериментов
         - определение факторов, которые влияют на сходимость (обусловленность, размерность, etc)
         - иногда заранее можно выбрать число итераций для достижения заданной точности

2. Что **НЕ** дают теоремы сходимости
    - сходимость метода **ничего не говорит** о целесообразности его применения
    - оценки сходимости зависят от неизвестных констант - неконструктивный характер
    - учёт ошибок округления и точности решения вспомогательных задач

**Мораль**: нужно проявлять разумную осторожность

и здравый смысл!

## Классификация задач
1. Безусловная оптимизация
    - целевая функция липшицева
    - градиент целевой функции липшицев
2. Условная оптимизация
    - многогранник
    - множество простой структуры
    - общего вида

## Классификация методов

### Какой размер истории нужно хранить для обновления?

1. Одношаговые методы

$$
x_{k+1} = \Phi(x_k)
$$

2. Многошаговые методы

$$
x_{k+1} = \Phi(x_k, x_{k-1}, ...)
$$

### Какой порядок поизводных нужно вычислить?
1. Методы нулевого порядка: оракул возвращает только значение функции $f(x)$

2. Методы первого порядка: оракул возвращает значение функции $f(x)$ и её градиент $f'(x)$

3. Методы второго порядка: оракул возвращает значение функции $f(x)$, её градиент $f'(x)$ и гессиан $f''(x)$.

**Q**: существуют ли методы более высокого порядка?

**А**: [Implementable tensor methods in unconstrained convex optimization](https://link.springer.com/content/pdf/10.1007/s10107-019-01449-1.pdf) by Y. Nesterov, 2019

## Методы спуска. Градиентный спуск

## Что такое методы спуска?

Последовательность $x_k$ генерируется по правилу

$$
x_{k+1} = x_k + \alpha_k h_k
$$

так что

$$
f(x_{k+1}) < f(x_k)
$$

Направление $h_k$ называется *направлением убывания*.

**Замечание**: существуют методы, которые не требуют монотонного убывания функции от итерации к итерации.

```python
def DescentMethod(f, x0, epsilon, **kwargs):

    x = x0

    while StopCriterion(x, f, **kwargs) > epsilon:

        h = ComputeDescentDirection(x, f, **kwargs)

        alpha = SelectStepSize(x, h, f, **kwargs)

        x = x + alpha * h

    return x

```

## Способ 1: направление убывания
Рассмотрим линейную аппроксимацию дифференцируемой функции $f$ вдоль некоторого направления убывания $h, \|h\|_2 = 1$:

$$
f(x + \alpha h) = f(x) + \alpha \langle f'(x), h \rangle + o(\alpha)
$$

Из условия убывания

$$
f(x) + \alpha \langle f'(x), h \rangle + o(\alpha) < f(x)
$$

и переходя к пределу при $\alpha \rightarrow 0$:

$$
\langle f'(x), h \rangle \leq 0
$$

Также из неравенства Коши-Буняковского-Шварца

$$
\langle f'(x), h \rangle \geq -\| f'(x) \|_2 \| h \|_2 = -\| f'(x) \|_2
$$

Таким образом, направление антиградиента

$$
h = -\dfrac{f'(x)}{\|f'(x)\|_2}
$$

даёт направление **наискорейшего локального** убывания функции$~f$.

В итоге метод имеет вид

$$
x_{k+1} = x_k - \alpha f'(x_k)
$$

## Способ 2: схема Эйлера решения ОДУ

Рассмотрим обыкновенное диференциальное уравнение вида:

$$
\frac{dx}{dt} = -f'(x(t))
$$

и дискретизуем его на равномерной сетке с шагом $\alpha$:

$$
\frac{x_{k+1} - x_k}{\alpha} = -f'(x_k),
$$

где $x_k \equiv x(t_k)$ и $\alpha = t_{k+1} - t_k$ - шаг сетки.

Отсюда получаем выражение для $x_{k+1}$

$$
x_{k+1} = x_k - \alpha f'(x_k),
$$

которое в точности совпадает с выражением для градиентного спуска.

Такая схема называется явной или прямой схемой Эйлера.

**Q:** какая схема называется неявной или обратной?

## Способ 3: минимизация квадратичной оценки сверху
#### (А. В. Гасников "Метод универсального градиентного спуска" https://arxiv.org/abs/1711.00394)

Глобальная оценка сверху на функцию $f$ в точке $x_k$:

$$
f(y) \leq f(x_k) + \langle f'(x_k), y - x_k \rangle + \frac{L}{2} \|y - x_k \|_2^2 = g(y),
$$

где $\lambda_{\max}(f''(x)) \leq L$ для всех допустимых $x$.

Справа — квадратичная форма, точка минимума которой имеет аналитическое выражение:

\begin{align*}
& g'(y^*) = 0 \\
& f'(x_k) + L (y^* - x_k) = 0 \\
& y^* = x_k - \frac{1}{L}f'(x_k) = x_{k+1}
\end{align*}

Этот способ позволяет оценить значение шага как $\frac{1}{L}$. Однако часто константа $L$ неизвестна.

## Итого: метод градиентного спуска — дёшево и сердито

```python
def GradientDescentMethod(f, x0, epsilon, **kwargs):

    x = x0

    while StopCriterion(x, f, **kwargs) > epsilon:

        h = ComputeGradient(x, f, **kwargs)

        alpha = SelectStepSize(x, h, f, **kwargs)

        x = x - alpha * h

    return x

```

## Как выбрать шаг $\alpha_k$? (J. Nocedal, S. Wright Numerical Optimization, $\S$ 3.1.)

Список подходов:
- Постоянный шаг

$$
\alpha_k = \overline{\alpha}
$$

- Априорно заданная последовательность, например

$$
\alpha_k = \dfrac{\overline{\alpha}}{\sqrt{k+1}}
$$

- Наискорейший спуск

$$
\alpha_k = \arg\min_{\alpha \geq 0} f(x_k - \alpha f'(x_k))
$$

- Требование **достаточного** убывания, требование **существенного** убывания и условие кривизны: для некоторых $\beta_1, \beta_2$, таких что $0 < \beta_1 < \beta_2 < 1$ найти $x_{k+1}$ такую что

    - Достаточное убывание: $f(x_{k+1}) \leq f(x_k) + \beta_1 \alpha_k \langle f'(x_k), h_k \rangle$ или
    $ f(x_k) - f(x_{k+1}) \geq \beta_1 \alpha_k \langle f'(x_k), h_k \rangle
    $
    - Существенное убывание: $f(x_{k+1}) \geq f(x_k) + \beta_2 \alpha_k \langle f'(x_k), h_k \rangle$ или
    $
    f(x_k) - f(x_{k+1}) \leq \beta_2 \alpha_k \langle f'(x_k), h_k \rangle
    $
    - Условие кривизны: $\langle f'(x_{k+1}), h_k \rangle \geq \beta_2 \langle f'(x_k), h_k \rangle$

Обычно коэффициенты выбирают так: $\beta_1 \in (0, 0.3)$, а $\beta_2 \in (0.9, 1)$

### Анализ и мотивация подходов к выбору шага $\alpha_k$
- Постоянный шаг: самое простое и неэффективное решение
- Априорно заданная последовательность: немногим лучше постоянного шага
- Наискорейший спуск: самое лучшее решение, но применимо только если вспомогательная задача решается аналитически или ооооооочень быстро.

То есть почти всегда неприменимо :)
- Требование достаточного убывания, требование существенного убывания и условие кривизны:
    - требование достаточного убывания гарантирует, что функция в точке $x_{k+1}$ не превосходит линейной аппроксимации с коэффициентом наклона $\beta_1$
    - требование существенного убывания гарантирует, что функция в точке $x_{k+1}$ убывает не меньше, чем линейная аппроксимация c коэффициентом наклона $\beta_2$
    - условие кривизны гарантирует, что угол наклона касательной в точке $x_{k+1}$ не меньше, чем угол наклона касательной в точке $x_k$,

умноженный на $\beta_2$

Требование существенного убывания и условие кривизны обеспечивают убывание функции по выбранному направлению $h_k$. Обычно выбирают одно из них.

#### Альтернативные названия
- Требование достаточного убывания $\equiv$ правило Армихо
- Требование достаточного убывания + условие кривизны $\equiv$ правило Вольфа
- Требование достаточного убывания + требование существенного убывания $\equiv$ правило Гольдштейна

## Зачем нужно условие существенного убывания?

```python
%matplotlib notebook
import matplotlib.pyplot as plt
plt.rc("text", usetex=True)
import ipywidgets as ipywidg
import numpy as np
import liboptpy.unconstr_solvers as methods
import liboptpy.step_size as ss
from tqdm import tqdm
```

```python
f = lambda x: np.power(x, 2)
gradf = lambda x: 2 * x

fig = plt.figure()
ax = fig.add_subplot(1, 1, 1)

def update(x0, step):
    gd = methods.fo.GradientDescent(f, gradf, ss.ConstantStepSize(step))
    _ = gd.solve(np.array([x0]), max_iter=10)
    x_hist = gd.get_convergence()
    x = np.linspace(-5, 5)
    ax.clear()
    ax.plot(x, f(x), color="r", label="$f(x) = x^2$")
    y_hist = np.array([f(x) for x in x_hist])
    x_hist = np.array(x_hist)
    plt.quiver(x_hist[:-1], y_hist[:-1], x_hist[1:]-x_hist[:-1], y_hist[1:]-y_hist[:-1],
               scale_units='xy', angles='xy', scale=1, width=0.005, color="green", label="Descent path")
    ax.legend()
    fig.canvas.draw()

step_slider = ipywidg.FloatSlider(value=0.8, min=0, max=1.2, step=0.1, description="Step")
x0_slider = ipywidg.FloatSlider(value=1.5, min=-4, max=4, step=0.1, description="Initial point")
_ = ipywidg.interact(update, x0=x0_slider, step=step_slider)
```

    <IPython.core.display.Javascript object>

    interactive(children=(FloatSlider(value=1.5, description='Initial point', max=4.0, min=-4.0), FloatSlider(valu…

```python
def plot_alpha(f, grad, x, h, alphas, beta1, beta2):
    df = np.zeros_like(alphas)
    for i, alpha in enumerate(alphas):
        df[i] = f(x + alpha * h)
    upper_bound = f(x) + beta1 * alphas * grad(x) * h
    lower_bound = f(x) + beta2 * alphas * grad(x) * h
    plt.plot(alphas, df, label=r"$f(x + \alpha h)$")
    plt.plot(alphas, upper_bound, label="Upper bound")
    plt.plot(alphas, lower_bound, label="Lower bound")
    plt.xlabel(r"$\alpha$", fontsize=18)
    plt.legend(loc="best", fontsize=18)
```

```python
f = lambda x: x**2
grad = lambda x: 2 * x
beta1 = 0.1
beta2 = 0.9
x0 = 0.5
plot_alpha(f, grad, x0, -grad(x0), np.linspace(1e-3, 1.01, 10), beta1, beta2)
```

## $f(x) = x\log x$

```python
x_range = np.linspace(1e-10, 4)
plt.plot(x_range, x_range * np.log(x_range))
```

```python
x0 = 1
f = lambda x: x * np.log(x)
grad = lambda x: np.log(x) + 1
beta1 = 0.3
beta2 = 0.7
plot_alpha(f, grad, x0, -grad(x0), np.linspace(1e-3, 0.9, 10), beta1, beta2)
```

### Backtracking

```python
def SelectStepSize(x, f, h, rho, alpha0, beta1, beta2):

    # 0 < rho < 1

    # alpha0 - initial guess of step size

    # beta1 and beta2 - constants from conditions

    alpha = alpha0

    # Check violating sufficient decrease and curvature conditions

    while (f(x - alpha * h) >= f(x) + beta1 * alpha grad_f(x_k).dot(h)) and

          (grad_f(x - alpha * h).dot(h) <= beta2 * grad_f(x_k).dot(h)):

        alpha *= rho

    return alpha

```

## Теоремы сходимости (Б.Т. Поляк Введение в оптимизацию, гл. 1, $\S$ 4;  гл. 3, $\S$ 1; Ю.Е. Нестеров Введение в выпуклую оптимизацию, $\S$ 2.2)
От общего к частному:

**Теорема 1.**
Пусть

- $f(x)$ дифференцируема на $\mathbb{R}^n$,
- градиент $f(x)$ удовлетворяет условию Липшица с константой $L$
- $f(x)$ ограничена снизу
- $\alpha = const$ и $0 < \alpha < \frac{2}{L}$

Тогда для градиентного метода выполнено:

$$
\lim\limits_{k \to \infty} f'(x_k) = 0,
$$

а функция монотонно убывает $f(x_{k+1}) < f(x_k)$.

**Теорема 2.** Пусть
- $f(x)$ дифференцируема на $\mathbb{R}^n$
- $f(x)$ выпукла
- $f'(x)$ удовлетворяет условию Липшица с константой $L$
- $\alpha = \dfrac{1}{L}$

Тогда

$$
f(x_k) - f^* \leq \dfrac{2L \| x_0 - x^*\|^2_2}{k+4}
$$

**Теорема 3.**
Пусть
- $f(x)$ дважды дифференцируема и $\mu\mathbf{I} \preceq f''(x) \preceq L\mathbf{I}$ для всех $x$
- $\alpha = const$ и $0 < \alpha < \frac{2}{L}$

Тогда

$$
\| x_k - x^*\|_2 \leq \|x_0 - x^*\|_2 q^k, \qquad q = \max(|1 - \alpha l|, |1 - \alpha L|) < 1
$$

и минимальное $q^* = \dfrac{L - \mu}{L + \mu}$ при $\alpha^* = \dfrac{2}{L + \mu}$

### От чего зависит $q^*$ и как это использовать?
Из Теоремы 3 имеем

$$
q^* = \dfrac{L - \mu}{L + \mu} = \dfrac{L/\mu - 1}{L/\mu + 1} = \dfrac{M - 1}{M + 1},
$$

где $M$ - оценка числа обусловленности $f''(x)$.

**Вопрос**: что такое число обусловленности матрицы?

- При $M \gg 1$, $q^* \to 1 \Rightarrow$ оооочень **медленная** сходимости градиентного метода. Например при $M = 100$: $q^* \approx 0.98 $
- При $M \simeq 1$, $q^* \to 0 \Rightarrow$ **ускорение** сходимости градиентного метода. Например при $M = 4$: $q^* = 0.6 $

**Вопрос**: какая геометрия у этого требования?

**Мораль**: необходимо сделать оценку $M$ как можно ближе к 1!

О том, как это сделать, Вам будет предложено подумать в домашнем задании :)

## Вычислительный аспект и эксперименты
1. Для каждого шага метода нужно хранить только текущую точку и вектор градиента: $O(n)$ памяти
2. Поиск $\alpha_k$:

    - дан априори
    - ищется из аналитического решения задачи наискорейшего спуска
    - заканчивается за конечное число шагов
3. Для каждого шага метода нужно вычислять линейную комбинацию векторов: $O(n)$ вычислений + высокопроизводительные реализации

### Pеализация градиентного спуска

```python
def GradientDescent(f, gradf, x0, epsilon, num_iter, line_search,
                    disp=False, callback=None, **kwargs):
    x = x0.copy()
    iteration = 0
    opt_arg = {"f": f, "grad_f": gradf}
    for key in kwargs:
        opt_arg[key] = kwargs[key]
    while True:
        gradient = gradf(x)
        alpha = line_search(x, -gradient, **opt_arg)
        x = x - alpha * gradient
        if callback is not None:
            callback(x)
        iteration += 1
        if disp:
            print("Current function val =", f(x))
            print("Current gradient norm = ", np.linalg.norm(gradf(x)))
        if np.linalg.norm(gradf(x)) < epsilon:
            break
        if iteration >= num_iter:
            break
    res = {"x": x, "num_iter": iteration, "tol": np.linalg.norm(gradf(x))}
    return res
```

### Выбор шага

Реализации различных способов выбора шага приведены [тут](https://github.com/amkatrutsa/liboptpy/blob/master/step_size.py)

### Зависимость от обусловленности матрицы $f''(x)$
Рассмотрим задачу
$$
\min f(x),
$$
где
$$ f(x) = x^{\top}Ax, \; A = \begin{bmatrix} 1 & 0\\ 0 & \gamma \end{bmatrix} $$

$$
f'(x) = 2Ax
$$

```python
def my_f(x, A):
    return 0.5 * x.dot(A.dot(x))

def my_gradf(x, A):
    return A.dot(x)
```

```python
plt.rc("text", usetex=True)

gammas = [0.1, 0.5, 1, 2, 3, 4, 5, 10, 20, 50, 100, 1000, 5000, 10000]
# gammas = [1]
num_iter_converg = []
for g in gammas:
    A = np.array([[1, 0],
                  [0, g]], dtype=np.float64)
    f = lambda x: my_f(x, A)
    gradf = lambda x: my_gradf(x, A)
#     x0 = np.random.rand(A.shape[0])
#     x0 = np.sort(x0)
#     x0 = x0[::-1]
    x0 = np.array([g, 1], dtype=np.float64)
#     print x0[1] / x0[0]
    gd = methods.fo.GradientDescent(f, gradf, ss.ExactLineSearch4Quad(A))
    x = gd.solve(x0, tol=1e-7, max_iter=100)
    num_iter_converg.append(len(gd.get_convergence()))

plt.figure(figsize=(8, 6))
plt.loglog(gammas, num_iter_converg)
plt.xticks(fontsize = 20)
plt.yticks(fontsize = 20)
plt.xlabel(r"$\gamma$", fontsize=20)
plt.ylabel(r"Number of iterations with $\varepsilon = 10^{-7}$", fontsize=20)
```

- При неудачном начальном приближении сходимость для плохо обусловенной задачи очень медленная
- При случайном начальном приближении сходимость может быть гораздо быстрее теоретических оценок

## Эксперимент на многомерной задаче
Пусть $A \in \mathbb{R}^{m \times n}$. Рассмотрим систему линейных неравенств: $Ax \leq 1$ при условии $|x_i| \leq 1$ для всех $i$.

**Определение.** Аналитическим центром системы неравенств $Ax \leq 1$ при условии $|x_i| \leq 1$ является решение задачи
$$
f(x) = - \sum_{i=1}^m \log(1 - a_i^{\top}x) - \sum_{i = 1}^n \log (1 - x^2_i) \to \min_x
$$
$$
f'(x) - ?
$$

### Точное решение с помощью CVXPy

```python
import numpy as np

n = 1000
m = 2000
A = np.random.rand(n, m)

x = cvx.Variable(n)

obj = cvx.Minimize(cvx.sum(-cvx.log(1 - A.T * x)) -
                   cvx.sum(cvx.log(1 - cvx.square(x))))
prob = cvx.Problem(obj)
prob.solve(solver="SCS", verbose=True)
x = x.value
print("Optimal value =", prob.value)
```

### Решение с помощью градиентного спуска

```python
import cvxpy as cvx
print(cvx.installed_solvers())

# !pip install jax
# !pip install jaxlib

import jax.numpy as jnp
import jax

# from jax.config import config
# config.update("jax_enable_x64", True)

A = jnp.array(A)
print(A.dtype)
x0 = jnp.zeros(n)
f = lambda x: -jnp.sum(jnp.log(1 - A.T@x)) - jnp.sum(jnp.log(1 - x*x))
grad_f = lambda x: jnp.sum(A @ (jnp.diagflat(1 / (1 - A.T @ x))), \
                          axis=1) + 2 * x / (1 - jnp.power(x, 2))
grad_f_jax = jax.grad(f)
print(jnp.linalg.norm(grad_f(x0) - grad_f_jax(x0)))
```

Подробнее про jax, его возможности и особенности можно посмотреть например [тут](https://github.com/amkatrutsa/MIPT-Opt/blob/master/Fall2020/03-MatrixCalculus/jax_autodiff_tutorial.ipynb)

```python
gd = methods.fo.GradientDescent(f, grad_f_jax, ss.Backtracking("Armijo", rho=0.5, beta=0.1, init_alpha=1.))
x = gd.solve(x0, tol=1e-5, max_iter=100, disp=True)

x_conv = gd.get_convergence()
grad_conv = [jnp.linalg.norm(grad_f_jax(x)) for x in x_conv]
plt.figure(figsize=(8,6))
plt.semilogy(grad_conv, label=r"$\| f'(x_k) \|_2$")
plt.semilogy([np.linalg.norm(x - np.array(x_k)) for x_k in x_conv], label=r"$\|x_k - x^*\|_2$")
plt.semilogy([np.linalg.norm(prob.value - f(np.array(x_k))) for x_k in x_conv], label=r"$\|f(x_k) - f^*\|_2$")
plt.semilogy([np.linalg.norm(np.array(x_conv[i]) - np.array(x_conv[i+1])) for i in range(len(x_conv) - 1)], label=r"$\|x_k - x_{k+1}\|_2$")
plt.semilogy([np.linalg.norm(f(np.array(x_conv[i])) - f(np.array(x_conv[i+1]))) for i in range(len(x_conv) - 1)], label=r"$\|f(x_k) - f(x_{k+1})\|_2$")
plt.xlabel(r"Number of iteration, $k$", fontsize=20)
plt.ylabel(r"Convergence rate", fontsize=20)
plt.xticks(fontsize = 20)
plt.yticks(fontsize = 20)
plt.legend(loc="best", fontsize=20)
plt.tight_layout()
```

## Pro & Contra

Pro
- легко реализовать
- сходимость как минимум к стационарной точке
- параметры при выборе шага влияют на сходимость не столь сильно
- имеет многочисленные вариации

Contra
- линейная сходимость для сильно выпуклых функций
- очень сильно зависит от числа обусловленности $f''(x)$, выбор начального приближения может помочь
- не является оптимальным для выпуклых функций с липшицевым градиентом и сильновыпуклых функций (см. [ускорение Нестерова](https://blogs.princeton.edu/imabandit/2013/04/01/acceleratedgradientdescent/))

## Резюме
1. Методы спуска
2. Направление убывания
3. Метод градиентного спуска
4. Правила выбора шага
5. Теоремы сходимости
6. Эксперименты


# Introduction
This is the third part of our series about Covid19 data analysis. This series is inspired after the proposed [Kaggle challenge](https://www.kaggle.com/c/covid19-global-forecasting-week-2). This notebook introduces some practical examples of univariate time series forecasting.

We introduces the [Python statsmodel](https://www.statsmodels.org) module which provides a set of statistical tools for conducting statistical tests and data exploration. We focus on the time series analysis module offered in this package to run basic time series forecasting predicting the evolution of cases in the Covid19 pandemia.

**Assumptions**
* You succesfully run the [Covid19 visualization notebook](./Covid19%20plots%20visualization.ipynb)
* You have an already running Jupyter environment
* You are familiar with Pandas
* You have heard about Matplotlib
* The covid19 files are available in the path covid19-global-forecasting-week-2

# Load the data

Load the CSV data using pandas. In this case, we use the `read_csv` function indicating the date parser to be used. Working with dates is going to be particularly useful when working with time series.

```python
import pandas as pd
import datetime
```

```python
data = pd.read_csv('covid19-global-forecasting-week-2/train.csv',parse_dates=['Date'],
                   date_parser=lambda x: datetime.datetime.strptime(x, '%Y-%m-%d'))
data

```

      Id
      Country_Region
      Province_State
      Date
      ConfirmedCases
      Fatalities

      0
      1
      Afghanistan
      NaN
      2020-01-22
      0.0
      0.0

      1
      2
      Afghanistan
      NaN
      2020-01-23
      0.0
      0.0

      2
      3
      Afghanistan
      NaN
      2020-01-24
      0.0
      0.0

      3
      4
      Afghanistan
      NaN
      2020-01-25
      0.0
      0.0

      4
      5
      Afghanistan
      NaN
      2020-01-26
      0.0
      0.0

      ...
      ...
      ...
      ...
      ...
      ...
      ...

      18811
      29360
      Zimbabwe
      NaN
      2020-03-21
      3.0
      0.0

      18812
      29361
      Zimbabwe
      NaN
      2020-03-22
      3.0
      0.0

      18813
      29362
      Zimbabwe
      NaN
      2020-03-23
      3.0
      1.0

      18814
      29363
      Zimbabwe
      NaN
      2020-03-24
      3.0
      1.0

      18815
      29364
      Zimbabwe
      NaN
      2020-03-25
      3.0
      1.0

18816 rows × 6 columns

China was the origin of Covid19. Its curve of infection is the longest one among the curves available in the dataset. We are going to get a sample filtering all the data corresponding to China. Remember that in the case of China we have information about the evolution for every province. In order, to work with an aggregated value per country and day, we have to aggregate the data. Check the previous notebooks for more details about this operation.

```python
china = data.where(data.Country_Region=='China').groupby(['Date'],as_index=False).sum()
china
```

      Date
      Id
      ConfirmedCases
      Fatalities

      0
      2020-01-22
      214533.0
      548.0
      17.0

      1
      2020-01-23
      214566.0
      643.0
      18.0

      2
      2020-01-24
      214599.0
      920.0
      26.0

      3
      2020-01-25
      214632.0
      1406.0
      42.0

      4
      2020-01-26
      214665.0
      2075.0
      56.0

      ...
      ...
      ...
      ...
      ...

      59
      2020-03-21
      216480.0
      81305.0
      3259.0

      60
      2020-03-22
      216513.0
      81435.0
      3274.0

      61
      2020-03-23
      216546.0
      81498.0
      3274.0

      62
      2020-03-24
      216579.0
      81591.0
      3281.0

      63
      2020-03-25
      216612.0
      81661.0
      3285.0

64 rows × 4 columns

```python
import matplotlib.pyplot as pyplot

fig = pyplot.figure(figsize=(10,5))
ax1 = fig.add_subplot()
china.ConfirmedCases.plot(ax=ax1,color='blue',legend=True, title='China-Covid19')
china.Fatalities.plot(ax=ax1,color='red',legend=True)
pyplot.show()
```

# Univariate time series analysis
The plot above shows the evolution of confirmed cases and fatalities for China during the Covid19 crisis. This seems the common profile of any pandemic. First, the number of cases increases until it reaches a plateau. Then, it stays stable until the cases decrease with an increment in the number of fatalities.

One of the most important problems to be resolved in this case, like in any other time series is: **can we forecast the number of new cases and/or fatalities with some accuracy?**

We start working with the China time series. Before starting we need to make some preparation to work with datetime indices. These are common operations in Pandas. We remove the `Id` column and convert the `Date` column into our new row index. Additionally, we set the frequency of the index to days by executing `china.index.freq='d'`. This is particularly helpful when using Pandas time series.

```python
china = china.set_index('Date')
china.drop(['Id'], axis=1, inplace=True)
# The index is a time series with a frequency of days
china.index.freq='d'
```

## Simple exponential smoothing

If we try to predict any future value in an univariate time series, the first thing we have to consider is the fact that future observations depend on past and present observations. If $\hat{y}_t$ is the forecast of variable $x$ at time $t$ and $\hat{y}_{t+h|t}$ the forecast of variable $x$ at time $t+h$ given time $t$, a forecast equation could be something like this.

\begin{align}
\hat{y}_{t+h|t} = l_t
\end{align}

Which basically means that our predictions at time $t+h$ given observations at time $t$ are the result of a function $l$ at time $t$. And what is this function? We are going to use a simple exponential smoothing (SES) function.

\begin{align}
l_1 &= \alpha y_1 + (1- \alpha) l_{0}
\end{align}

Which is a formal way of saying that the current value depends on the previous value multiplied by a smoothing factor $ \alpha $. If we continue developing the equation we have

\begin{align*}
l_1 &=\alpha y_1 + (1-\alpha) l_{0} \\
l_2 &=\alpha y_2 + (1-\alpha) l_{1} = \alpha y_2 + \alpha(1-\alpha) y_1 + (1-\alpha)^2 l_{0} \\
l_3 &=\alpha y_3 + (1-\alpha) l_{2} = \sum_{j=0}^{2}\alpha(1-\alpha)^jy_{3-j}+(1-\alpha)^3l_{0} \\
\vdots \\
l_t &= \sum_{j=0}^{t-1}\alpha(1-\alpha)^j y_{t-j}+(1-\alpha)^tl_0
\end{align*}

Therefore, the forecasting equation is:

\begin{align*}
\hat{y}_{t+h|t} &= \sum_{j=1}^{t}\alpha(1-\alpha)^{t-j} y_{j}+(1-\alpha)^tl_0
\end{align*}

The $\alpha$ value should be chosen to minimize the mean squared error (MSE)

\begin{align*}
MSE = \frac{1}{T} \sum_{t=1}^{T} (y_t+\hat{y}_{t|t-1})^2
\end{align*}

There is no closed solution to find $\alpha$. [Least squares](https://en.wikipedia.org/wiki/Least_squares) can be a simple and valid method to set the value.

We are going to use the implementation available at the statsmodels library to make some predictions for the number of confirmed cases in China starting at 2020-03-20.

```python
from statsmodels.tsa.holtwinters import SimpleExpSmoothing
```

```python
# We previously set dates as the row index. Now we can use indexes to select rows.
china_train = china[:'2020-03-20']
china_test = china['2020-03-21':]

# Chose a model for confirmed cases
model = SimpleExpSmoothing(china_train.ConfirmedCases)
# Fit the model for the given observations
model_fit = model.fit()

```

We have fit a model using SES and the number of confirmed cases. It is extremely useful to check what are the params for this model and understand how well our fitting process did.

```python
# Check the params for the model
display(model_fit.params)
# Lets observe a summary about the fitting
model_fit.summary()
```

    {'smoothing_level': 1.0,
     'smoothing_slope': nan,
     'smoothing_seasonal': nan,
     'damping_slope': nan,
     'initial_level': 548.0,
     'initial_slope': nan,
     'initial_seasons': array([], dtype=float64),
     'use_boxcox': False,
     'lamda': None,
     'remove_bias': False}

SimpleExpSmoothing Model Results

  Dep. Variable:	           endog	         No. Observations:  	        59

  Model:	            SimpleExpSmoothing	   SSE                	   421717800.000

  Optimized:	               True	          AIC                	      935.156

  Trend:	                   None	          BIC                	      939.311

  Seasonal:	                None	          AICC               	      935.897

  Seasonal Periods:	        None	          Date:              	 Fri, 03 Apr 2020

  Box-Cox:	                 False	         Time:              	     20:51:35

  Box-Cox Coeff.:	          None

         	                 coeff	                code	               optimized

  smoothing_level	            1.0000000	                alpha	                 True

  initial_level	              548.00000	                  l.0	                 True

We can set the $\alpha$ parameter by setting the `smoothing_level` value in the `model.fit` call. However, if we do not set any value, the library will find it for us. In this case $\alpha$ was automatically set to 1 with $l_0=548$ (`initial_level` in the summary table).

Now we are going to fit some models manually setting the $\alpha$ parameter and compare the obtained output.

```python
import matplotlib.pyplot
```

```python
# Fit with some alphas
fit025 = model.fit(smoothing_level=0.25)
fit05 = model.fit(smoothing_level=0.5)
fit075 = model.fit(smoothing_level=0.75)

ax = china_train.ConfirmedCases.plot(color='black', legend=True)
model_fit.fittedvalues.plot(ax=ax, color='blue', style='--', legend=True, label=r'Auto $\alpha=%s$'%model_fit.params['smoothing_level'])
fit025.fittedvalues.plot(ax=ax, color='red', style='--', legend=True, label=r'$\alpha=0.25$' )
fit05.fittedvalues.plot(ax=ax, color='green', style='--', legend=True, label=r'$\alpha=0.5$')
fit075.fittedvalues.plot(ax=ax, color='grey', style='--', legend=True, label=r'$\alpha=0.75$' )

pyplot.show()
```

The plot above shows that $\alpha=1$ gives the best approximation to the curve. Now that we know that $\alpha=1$  brings the best performance, we can think about forecasting with that model.

```python
# Lets make some predictions
predicted = model_fit.predict(start='2020-03-21',end='2020-03-25').rename('Predicted ConfirmedCases')

```

```python
ax = china_train.ConfirmedCases.plot(color='black', legend=True)
china_test.ConfirmedCases.plot(ax = ax, color='blue', legend=False)
predicted.plot(ax=ax, color='blue', style='--', legend=True)
pyplot.show()
```

The plot the whole time series and a five days prediction. We better use some zoom.

```python
ax = china_train.ConfirmedCases['2020-03-15':'2020-03-25'].plot(color='black',legend=True)
china_test.ConfirmedCases.plot(ax = ax, color='blue', legend=True, label='ConfirmedCases to predict')
predicted.plot(ax=ax, color='blue', style='--', legend=True)
pyplot.show()
```

Observe that the iterative prediction always returns the same value. This is because this function is a **flat forecast function**. There is no trend, seasonality or other value that can make this prediction to evolve. So finally, what we have is the mean of a future:

\begin{align*}
\hat{y}_{T+h|T} = \hat{y}_{T+1|T}, \space h=2,3,...
\end{align*}

## Holt-Winters

The flat nature of multiple steps forecasting of SES is not a good solution if we need a prediction at $T+h$. However, there are other approaches which basically include additional elemnts such as seasonality or trend. For example, the Holt's local trend method adds a $\beta^*$ factor that controls the trend:

\begin{align*}
Forecast: && \hat{y}_{t+h|t} &= l_t + hb_t \\
Level: && l_t &= \alpha y_t + (1-\alpha)(l_{t-1}+b_{t-1}) \\
Trend: && b_t &= \beta^*(l_t-l_{t-1})+(1-\beta^*)b_{t-1}
\end{align*}

In this approach, $l_t$ estimates the level while $bt$ estimates the slope. The extension of the formula requires to estimate $\alpha$, $\beta=\alpha\beta^*$,$l_0$ and $b_0$.

Let's compare the forecast obtained from Holt-Winters for the previous scenario.

```python
from statsmodels.tsa.holtwinters import Holt

# Fit a Holt-Winters model
holt_model = Holt(china_train.ConfirmedCases)
holt_fitted = holt_model.fit()
holt_fitted.summary()
```

Holt Model Results

  Dep. Variable:	      endog	    No. Observations:  	        59

  Model:	              Holt	     SSE                	   243476583.261

  Optimized:	          True	     AIC                	      906.747

  Trend:	            Additive	   BIC                	      915.057

  Seasonal:	           None	     AICC               	      908.362

  Seasonal Periods:	   None	     Date:              	 Fri, 03 Apr 2020

  Box-Cox:	            False	    Time:              	     20:51:36

  Box-Cox Coeff.:	     None

         	                 coeff	                code	               optimized

  smoothing_level	            1.0000000	                alpha	                 True

  smoothing_slope	            0.2715660	                 beta	                 True

  initial_level	              398.89608	                  l.0	                 True

  initial_slope	              657.75311	                  b.0	                 True

```python
holt_predicted = holt_fitted.predict(start='2020-03-21', end='2020-03-25').rename('Predicted ConfirmedCases')

fig = pyplot.figure(figsize=(15,10))
ax1 = fig.add_subplot(2,2,1)
china_train.ConfirmedCases['2020-03-15':'2020-03-25'].plot(ax=ax1,color='black',legend=True,
                                                           title='Simple Exponential Smoothing Excerpt')
china_test.ConfirmedCases.plot(ax = ax1, color='blue', legend=True, label='ConfirmedCases to predict')
predicted.plot(ax=ax1, color='blue', style='--', legend=True)

ax2 = fig.add_subplot(2,2,2)
china_train.ConfirmedCases['2020-03-15':'2020-03-25'].plot(ax=ax2, color='black',legend=True,
                                                           title='Holt Winters Excerpt')
china_test.ConfirmedCases.plot(ax = ax2, color='blue', legend=True, label='ConfirmedCases to predict')
holt_predicted.plot(ax=ax2, color='blue', style='--', legend=True)

ax4 = fig.add_subplot(2,2,3)
china_train.ConfirmedCases.plot(ax=ax4, color='black',legend=True, title='Simple Exponential Smoothing')
model_fit.fittedvalues.plot(ax=ax4, color='blue', style='--', legend=True, label='Predicted ConfirmedCases')

ax3 = fig.add_subplot(2,2,4)
china_train.ConfirmedCases.plot(ax=ax3, color='black',legend=True, title='Holt Winters')
holt_fitted.fittedvalues.plot(ax=ax3, color='blue', style='--', legend=True, label='Predicted ConfirmedCases')

pyplot.tight_layout()
pyplot.show()

```

Comparing SES with the Holt Winters predictions there are clear differences. As we mentioned before, the prediction for the next five days is flat in the first case. Holt Winters uses its trend component to predict an increment in the number of cases.

When comparing how both models fit the time series, SES reacts with some latency returning an approximation with some lag. Holt Winters reacts earlier by capturing the trend. However, it reacts with some latency when the trend changes. If we compare the residuals for both models we notice that Holt Winters performs better than SES.

```python
ax=model_fit.resid.plot(color='blue', style='--', legend=True, label='SES residuals', title='Comparison of residuals')
holt_fitted.resid.plot(ax=ax, color='red', legend=True, label='Holt Winters residuals')
pyplot.show()

```

At the moment of writing this notebook the most complete Covid-19 time series corresponds to China. However, we can check the quality of the fitting models for every country in the dataset. We can expect that any Holt Winters model will return better predictions than SES. For the sake of exhaustiveness we compare the performance of SES and Holt Winters predicting the number of confirmed cases and fatilities for every country in the dataset. The range of observed values for every country in the dataset may dramatically vary. This makes useless the comparison of RMSE values. We employ the Normalized Squared Error (NRMSE) which normalizes the RMSE using the max and min observed values. This is, $\mathit{NRMSE} = \frac{\mathit{RMSE}}{y_{max}-y_{min}}$.

**Note:** Observations may dramatically vary between countries. We ignore countries with no fatalities nor confirmed cases.

```python
# Accumulate observations per country and date. This is required to aggregate observations for multiple provinces
# in the same country.
data=data.groupby(['Country_Region','Date'],as_index=False,).sum()
display(data)
```

      Country_Region
      Date
      Id
      ConfirmedCases
      Fatalities

      0
      Afghanistan
      2020-01-22
      1
      0.0
      0.0

      1
      Afghanistan
      2020-01-23
      2
      0.0
      0.0

      2
      Afghanistan
      2020-01-24
      3
      0.0
      0.0

      3
      Afghanistan
      2020-01-25
      4
      0.0
      0.0

      4
      Afghanistan
      2020-01-26
      5
      0.0
      0.0

      ...
      ...
      ...
      ...
      ...
      ...

      11067
      Zimbabwe
      2020-03-21
      29360
      3.0
      0.0

      11068
      Zimbabwe
      2020-03-22
      29361
      3.0
      0.0

      11069
      Zimbabwe
      2020-03-23
      29362
      3.0
      1.0

      11070
      Zimbabwe
      2020-03-24
      29363
      3.0
      1.0

      11071
      Zimbabwe
      2020-03-25
      29364
      3.0
      1.0

11072 rows × 5 columns

```python
import numpy as np

def computeErrors(ymax,ymin,fitted):
    '''
    Compute the normalized RMSE for this model
    $$
    \mathit{NRMSE} = \frac{\mathit{RMSE}}{y_{max}-y_{min}}
    $$
    '''
    RMSE = np.sqrt(np.sum(np.power(fitted.resid,2))/len(fitted.resid))
    NRMSE = RMSE / (ymax-ymin)
    return NRMSE

# Fit a model for every country

# Get the list of all the available countries
countries = data.Country_Region.unique()

nrmse = pd.DataFrame(columns=['SES-Confirmed','SES-Fatalities','HoltWinters-Confirmed','HoltWinters-Fatalities'],
                    index=countries)

# Because we are blindly computing a large number of models, some of them may result in convergence problems.
# We ignore them.
import warnings
warnings.filterwarnings('ignore')

for country in countries:
    excerpt = data[data.Country_Region==country]
    maxConfirmed = excerpt.ConfirmedCases.max()
    minConfirmed = excerpt.ConfirmedCases.min()
    maxFatalities = excerpt.Fatalities.max()
    minFatalities = excerpt.Fatalities.min()
    # Skip it if we have no relevant data
    if maxConfirmed == 0 or maxFatalities==0:
        print('%s skipped because there is no relevant data'%(country))
        next
    else:
        print('%s processing'%(country))

    # Models with ses
    ses_confirmed = SimpleExpSmoothing(excerpt.ConfirmedCases)
    ses_fatalities = SimpleExpSmoothing(excerpt.Fatalities)
    # Models with Holt Winters
    hw_confirmed = Holt(excerpt.ConfirmedCases)
    hw_fatalities = Holt(excerpt.Fatalities)

    # Fit models
    ses_confirmed_fit = ses_confirmed.fit()
    ses_fatalities_fit = ses_fatalities.fit()
    hw_confirmed_fit = hw_confirmed.fit()
    hw_fatalities_fit = hw_fatalities.fit()

    # Fill our dataframe
    nrmse.loc[country]={
        'SES-Confirmed': computeErrors(maxConfirmed,minConfirmed,ses_confirmed_fit),
        'SES-Fatalities': computeErrors(maxFatalities,minFatalities,ses_fatalities_fit),
        'HoltWinters-Confirmed': computeErrors(maxConfirmed,minConfirmed,hw_confirmed_fit),
        'HoltWinters-Fatalities': computeErrors(maxFatalities,minFatalities,hw_fatalities_fit)
    }

nrmse

```

    Afghanistan processing
    Albania processing
    Algeria processing
    Andorra processing
    Angola skipped because there is no relevant data
    Antigua and Barbuda skipped because there is no relevant data
    Argentina processing
    Armenia skipped because there is no relevant data
    Australia processing
    Austria processing
    Azerbaijan processing
    Bahamas skipped because there is no relevant data
    Bahrain processing
    Bangladesh processing
    Barbados skipped because there is no relevant data
    Belarus skipped because there is no relevant data
    Belgium processing
    Belize skipped because there is no relevant data
    Benin skipped because there is no relevant data
    Bhutan skipped because there is no relevant data
    Bolivia skipped because there is no relevant data
    Bosnia and Herzegovina processing
    Brazil processing
    Brunei skipped because there is no relevant data
    Bulgaria processing
    Burkina Faso processing
    Cabo Verde processing
    Cambodia skipped because there is no relevant data
    Cameroon processing
    Canada processing
    Central African Republic skipped because there is no relevant data
    Chad skipped because there is no relevant data
    Chile processing
    China processing
    Colombia processing
    Congo (Brazzaville) skipped because there is no relevant data
    Congo (Kinshasa) processing
    Costa Rica processing
    Cote d'Ivoire skipped because there is no relevant data
    Croatia processing
    Cuba processing
    Cyprus processing
    Czechia processing
    Denmark processing
    Diamond Princess processing
    Djibouti skipped because there is no relevant data
    Dominica skipped because there is no relevant data
    Dominican Republic processing
    Ecuador processing
    Egypt processing
    El Salvador skipped because there is no relevant data
    Equatorial Guinea skipped because there is no relevant data
    Eritrea skipped because there is no relevant data
    Estonia processing
    Eswatini skipped because there is no relevant data
    Ethiopia skipped because there is no relevant data
    Fiji skipped because there is no relevant data
    Finland processing
    France processing
    Gabon processing
    Gambia processing
    Georgia skipped because there is no relevant data
    Germany processing
    Ghana processing
    Greece processing
    Grenada skipped because there is no relevant data
    Guatemala processing
    Guinea skipped because there is no relevant data
    Guinea-Bissau skipped because there is no relevant data
    Guyana processing
    Haiti skipped because there is no relevant data
    Holy See skipped because there is no relevant data
    Honduras skipped because there is no relevant data
    Hungary processing
    Iceland processing
    India processing
    Indonesia processing
    Iran processing
    Iraq processing
    Ireland processing
    Israel processing
    Italy processing
    Jamaica processing
    Japan processing
    Jordan skipped because there is no relevant data
    Kazakhstan processing
    Kenya skipped because there is no relevant data
    Korea, South processing
    Kuwait skipped because there is no relevant data
    Kyrgyzstan skipped because there is no relevant data
    Laos skipped because there is no relevant data
    Latvia skipped because there is no relevant data
    Lebanon processing
    Liberia skipped because there is no relevant data
    Libya skipped because there is no relevant data
    Liechtenstein skipped because there is no relevant data
    Lithuania processing
    Luxembourg processing
    Madagascar skipped because there is no relevant data
    Malaysia processing
    Maldives skipped because there is no relevant data
    Mali skipped because there is no relevant data
    Malta skipped because there is no relevant data
    Mauritania skipped because there is no relevant data
    Mauritius processing
    Mexico processing
    Moldova processing
    Monaco skipped because there is no relevant data
    Mongolia skipped because there is no relevant data
    Montenegro processing
    Morocco processing
    Mozambique skipped because there is no relevant data
    Namibia skipped because there is no relevant data
    Nepal skipped because there is no relevant data
    Netherlands processing
    New Zealand skipped because there is no relevant data
    Nicaragua skipped because there is no relevant data
    Niger processing
    Nigeria processing
    North Macedonia processing
    Norway processing
    Oman skipped because there is no relevant data
    Pakistan processing
    Panama processing
    Papua New Guinea skipped because there is no relevant data
    Paraguay processing
    Peru processing
    Philippines processing
    Poland processing
    Portugal processing
    Qatar skipped because there is no relevant data
    Romania processing
    Russia processing
    Rwanda skipped because there is no relevant data
    Saint Kitts and Nevis skipped because there is no relevant data
    Saint Lucia skipped because there is no relevant data
    Saint Vincent and the Grenadines skipped because there is no relevant data
    San Marino processing
    Saudi Arabia processing
    Senegal skipped because there is no relevant data
    Serbia processing
    Seychelles skipped because there is no relevant data
    Singapore processing
    Slovakia processing
    Slovenia processing
    Somalia skipped because there is no relevant data
    South Africa skipped because there is no relevant data
    Spain processing
    Sri Lanka skipped because there is no relevant data
    Sudan processing
    Suriname skipped because there is no relevant data
    Sweden processing
    Switzerland processing
    Syria skipped because there is no relevant data
    Taiwan* processing
    Tanzania skipped because there is no relevant data
    Thailand processing
    Timor-Leste skipped because there is no relevant data
    Togo skipped because there is no relevant data
    Trinidad and Tobago processing
    Tunisia processing
    Turkey processing
    US processing
    Uganda skipped because there is no relevant data
    Ukraine processing
    United Arab Emirates processing
    United Kingdom processing
    Uruguay skipped because there is no relevant data
    Uzbekistan skipped because there is no relevant data
    Venezuela skipped because there is no relevant data
    Vietnam skipped because there is no relevant data
    Zambia skipped because there is no relevant data
    Zimbabwe processing

      SES-Confirmed
      SES-Fatalities
      HoltWinters-Confirmed
      HoltWinters-Fatalities

      Afghanistan
      0.0594866
      0.0883883
      0.0482386
      0.0761006

      Albania
      0.0366054
      0.0661438
      0.0148385
      0.062477

      Algeria
      0.0423765
      0.0416667
      0.0244344
      0.0277485

      Andorra
      0.0459787
      0.125
      0.0268921
      0.124003

      Angola
      0.0721688
      NaN
      0.066931
      inf

      ...
      ...
      ...
      ...
      ...

      Uzbekistan
      0.0444878
      NaN
      0.0276604
      inf

      Venezuela
      0.0504516
      NaN
      0.0391757
      inf

      Vietnam
      0.031406
      NaN
      0.0188569
      inf

      Zambia
      0.0966002
      NaN
      0.0899481
      inf

      Zimbabwe
      0.0931695
      0.125
      0.0918469
      0.124003

173 rows × 4 columns

Now we have a dataframe named `nrmse` with the NRMSE values obtained fitting SES and Holt Winters models for confirmed cases and fatalities. We create two barplots to visually compare the NRMSE obtained in both cases. The lower the NRMSE, the better.

```python

toPlotConfirmed = pd.DataFrame({'SES':nrmse['SES-Confirmed'], 'HW':nrmse['HoltWinters-Confirmed']})
toPlotFatalities = pd.DataFrame({'SES':nrmse['SES-Fatalities'], 'HW':nrmse['HoltWinters-Fatalities']})

fig = pyplot.figure(figsize=(15,30))
ax1 = fig.add_subplot(1,2,1)
toPlotConfirmed.plot.barh(ax=ax1,legend=True,title='Comparison of NRMSE for confirmed cases')
ax2 = fig.add_subplot(1,2,2)
toPlotFatalities.plot.barh(ax=ax2,legend=True,title='Comparison of NRMSE for fatalities')
ax2.legend(loc='upper right')

pyplot.show()

```

# Conclusions

In this notebook, we have introduced the problem of univariate time series forecasting. In particular, we have proposed to forecast the evolution of the Covid-19 time series. We have run a coarse grain analysis with two prediction methods (Simple Exponential Smoothing and Holt Winters) and compared the results. We cannot claim any of these methods to be powerful enough to precisely run predictions given the complexity and implications of the Covid-19 problem. However, this exercise is a good starter for any discussion about time series, pandemics and predictors. There is a vast literature about this topic, in particular pandemics, and this notebook does not pretend to be a contribution.

# References

I strongly suggest to visit professor Rob J. Hyndman [web site](https://robjhyndman.com/) for a full list of content about time series. I have followed the notation mentioned [here](https://robjhyndman.com/uwafiles/fpp-notes.pdf).

covid19/Covid19 time series forecasting.ipynb