# PaddlePaddle | 深度学习 101- 情感分析

本人仅以  PaddlePaddle 深度学习 101 官网教程为指导，添加个人理解和笔记，仅作为学习练习使用，若有错误，还望批评指教。--ZJ

>原文地址：[  PaddlePaddle  官网](http://www.paddlepaddle.org/)|[  PaddlePaddle  深度学习 101](http://www.paddlepaddle.org/docs/develop/book/06.understand_sentiment/index.cn.html)

环境：
- Python 2.7
- Ubuntu 16.04

---

# 情感分析

本教程源代码目录在[book/understand_sentiment](https://github.com/PaddlePaddle/book/tree/develop/06.understand_sentiment)， 初次使用请参考PaddlePaddle[安装教程](https://github.com/PaddlePaddle/book/blob/develop/README.cn.md#运行这本书)，更多内容请参考本教程的[视频课堂](http://bit.baidu.com/course/detail/id/177.html)。

## 背景介绍

在自然语言处理中，情感分析一般是指判断一段文本所表达的情绪状态。其中，一段文本可以是一个句子，一个段落或一个文档。情绪状态可以是两类，如（正面，负面），（高兴，悲伤）；也可以是三类，如（积极，消极，中性）等等。情感分析的应用场景十分广泛，如把用户在购物网站（亚马逊、天猫、淘宝等）、旅游网站、电影评论网站上发表的评论分成正面评论和负面评论；或为了分析用户对于某一产品的整体使用感受，抓取产品的用户评论并进行情感分析等等。表格1 展示了对电影评论进行情感分析的例子：

| 电影评论       | 类别  |
| --------     | -----  |
| 在冯小刚这几年的电影里，算最好的一部的了| 正面 |
| 很不好看，好像一个地方台的电视剧     | 负面 |
| 圆方镜头全程炫技，色调背景美则美矣，但剧情拖沓，口音不伦不类，一直努力却始终无法入戏| 负面|
|剧情四星。但是圆镜视角加上婺源的风景整个非常有中国写意山水画的感觉，看得实在太舒服了。。|正面|

表格 1 电影评论情感分析

在自然语言处理中，情感分析属于典型的**文本分类**问题，即把需要进行情感分析的文本划分为其所属类别。文本分类涉及文本表示和分类方法两个问题。

**在深度学习的方法出现之前，**主流的文本表示方法为词袋模型 BOW (bag of words)，话题模型等等；分类方法有 SVM (support vector machine), LR(logistic regression)等等。

对于一段文本， BOW 表示会忽略其词顺序、语法和句法，将这段文本仅仅看做是一个词集合，因此 BOW 方法并不能充分表示文本的语义信息。例如，句子“这部电影糟糕透了”和“一个乏味，空洞，没有内涵的作品”在情感分析中具有很高的语义相似度，但是它们的 BOW 表示的相似度为 0。又如，句子“一个空洞，没有内涵的作品”和“一个不空洞而且有内涵的作品”的 BOW 相似度很高，但实际上它们的意思很不一样。

本章我们所要介绍的深度学习模型克服了 BOW 表示的上述缺陷，它在考虑词顺序的基础上把文本映射到低维度的语义空间，**并且以端对端（end to end）的方式进行文本表示及分类，其性能相对于传统方法有显著的提升**\[[1](#参考文献)\]。

## 模型概览

本章所使用的文本表示模型为卷积神经网络（Convolutional Neural Networks）和循环神经网络(Recurrent Neural Networks)及其扩展。下面依次介绍这几个模型。

### 文本卷积神经网络简介（CNN）

我们在[推荐系统](https://github.com/PaddlePaddle/book/tree/develop/05.recommender_system)一节介绍过应用于文本数据的卷积神经网络模型的计算过程，这里进行一个简单的回顾。

对卷积神经网络来说:

- 首先使用卷积处理输入的词向量序列，产生一个特征图（feature map），
- 对特征图采用时间维度上的最大池化（max pooling over time）操作得到此卷积核对应的整句话的特征，
- 最后，将所有卷积核得到的特征拼接起来即为文本的定长向量表示，对于文本分类问题，将其连接至 softmax 即构建出完整的模型。

在实际应用中，我们会使用多个卷积核来处理句子，窗口大小相同的卷积核堆叠起来形成一个矩阵，这样可以更高效的完成运算。另外，我们也可使用窗口大小不同的卷积核来处理句子，[推荐系统](https://github.com/PaddlePaddle/book/tree/develop/05.recommender_system)一节的图3作为示意画了四个卷积核，不同颜色表示不同大小的卷积核操作。

对于一般的短文本分类问题，上文所述的简单的文本卷积网络即可达到很高的正确率\[[1](#参考文献)\]。若想得到更抽象更高级的文本特征表示，可以构建深层文本卷积神经网络\[[2](#参考文献),[3](#参考文献)\]。

### 循环神经网络（RNN）

循环神经网络是一种能对序列数据进行精确建模的有力工具。实际上，循环神经网络的理论计算能力是图灵完备的\[[4](#参考文献)\]。自然语言是一种典型的序列数据（词序列），近年来，循环神经网络及其变体（如 long short term memory \[[5](#参考文献)\]等）在自然语言处理的多个领域，如语言模型、句法解析、语义角色标注（或一般的序列标注）、语义表示、图文生成、对话、机器翻译等任务上均表现优异甚至成为目前效果最好的方法。

图1. 循环神经网络按时间展开的示意图

循环神经网络按时间展开后如图1 所示：在第 $t$ 时刻，网络读入第 $t$ 个输入 $x_t$（向量表示）及前一时刻隐层的状态值 $h_{t-1}$ （向量表示，$h_0$ 一般初始化为 $0$ 向量），计算得出本时刻隐层的状态值 $h_t$，重复这一步骤直至读完所有输入。如果将循环神经网络所表示的函数记为 $f$，则其公式可表示为：

$$h_t=f(x_t,h_{t-1})=\sigma(W_{xh}x_t+W_{hh}h_{t-1}+b_h)$$

其中 $W_{xh}$ 是输入到隐层的矩阵参数，$W_{hh}$ 是隐层到隐层的矩阵参数，$b_h$ 为隐层的偏置向量（bias）参数，$\sigma$ 为 $sigmoid$ 函数。

在处理自然语言时，一般会先将词（one-hot 表示）映射为其词向量（word embedding）表示，然后再作为循环神经网络每一时刻的输入 $x_t$。此外，可以根据实际需要的不同在循环神经网络的隐层上连接其它层。如，可以把一个循环神经网络的隐层输出连接至下一个循环神经网络的输入构建深层（deep or stacked）循环神经网络，或者提取最后一个时刻的隐层状态作为句子表示进而使用分类模型等等。

### 长短期记忆网络（ LSTM ）

对于较长的序列数据，循环神经网络的训练过程中容易出现梯度消失或爆炸现象\[[6](#参考文献)\]。为了解决这一问题，Hochreiter S, Schmidhuber J. (1997)提出了 LSTM (long short term memory\[[5](#参考文献)\])。

相比于简单的循环神经网络， LSTM 增加了记忆单元 $c$、输入门 $i$、遗忘门 $f$ 及输出门 $o$。这些门及记忆单元组合起来大大提升了循环神经网络处理长序列数据的能力。若将基于 LSTM 的循环神经网络表示的函数记为$F$，则其公式为：

$$ h_t=F(x_t,h_{t-1})$$

$F$由下列公式组合而成\[[7](#参考文献)\]：

\begin{align}
i_t & = \sigma(W_{xi}x_t+W_{hi}h_{t-1}+W_{ci}c_{t-1}+b_i)\\\\
f_t & = \sigma(W_{xf}x_t+W_{hf}h_{t-1}+W_{cf}c_{t-1}+b_f)\\\\
c_t & = f_t\odot c_{t-1}+i_t\odot tanh(W_{xc}x_t+W_{hc}h_{t-1}+b_c)\\\\
o_t & = \sigma(W_{xo}x_t+W_{ho}h_{t-1}+W_{co}c_{t}+b_o)\\\\
h_t & = o_t\odot tanh(c_t)\\\\
\end{align}

其中，$i_t, f_t, c_t, o_t$ 分别表示输入门，遗忘门，记忆单元及输出门的向量值，带角标的 $W$ 及 $b$ 为模型参数，$tanh$ 为双曲正切函数，$\odot$ 表示逐元素（elementwise）的乘法操作。输入门控制着新输入进入记忆单元 $c$ 的强度，遗忘门控制着记忆单元维持上一时刻值的强度，输出门控制着输出记忆单元的强度。三种门的计算方式类似，但有着完全不同的参数，它们各自以不同的方式控制着记忆单元 $c$，如图2所示：

图2. 时刻$t$的 LSTM  [7]

 LSTM 通过给简单的循环神经网络增加记忆及控制门的方式，增强了其处理远距离依赖问题的能力。类似原理的改进还有 Gated Recurrent Unit (GRU) \[[8](#参考文献)\]，其设计更为简洁一些。**这些改进虽然各有不同，但是它们的宏观描述却与简单的循环神经网络一样（如图2所示），即隐状态依据当前输入及前一时刻的隐状态来改变，不断地循环这一过程直至输入处理完毕：**

$$ h_t=Recrurent(x_t,h_{t-1})$$

其中，$Recrurent$ 可以表示简单的循环神经网络、GRU 或 LSTM 。

### 栈式双向 LSTM （Stacked Bidirectional  LSTM ）

对于正常顺序的循环神经网络，$h_t$ 包含了 $t$ 时刻之前的输入信息，也就是上文信息。同样，为了得到下文信息，我们可以使用反方向（将输入逆序处理）的循环神经网络。结合构建深层循环神经网络的方法（深层神经网络往往能得到更抽象和高级的特征表示），我们可以通过构建更加强有力的基于 LSTM 的栈式双向循环神经网络\[[9](#参考文献)\]，来对时序数据进行建模。

如图3所示（以三层为例），奇数层 LSTM 正向，偶数层 LSTM 反向，高一层的 LSTM 使用低一层 LSTM 及之前所有层的信息作为输入，对最高层 LSTM 序列使用时间维度上的最大池化即可得到文本的定长向量表示（这一表示充分融合了文本的上下文信息，并且对文本进行了深层次抽象），最后我们将文本表示连接至 softmax 构建分类模型。

图3. 栈式双向 LSTM 用于文本分类

## 示例程序

### 数据集介绍

我们以[IMDB情感分析数据集](http://ai.stanford.edu/%7Eamaas/data/sentiment/)为例进行介绍。IMDB 数据集的训练集和测试集分别包含 25000 个已标注过的电影评论。其中，负面评论的得分小于等于 4，正面评论的得分大于等于 7，满分 10 分。
```text
aclImdb
|- test
   |-- neg
   |-- pos
|- train
   |-- neg
   |-- pos
```
Paddle在`dataset/imdb.py`中提实现了 imdb 数据集的自动下载和读取，并提供了读取字典、训练数据、测试数据等 API。

```python
import sys
import paddle.v2 as paddle
```
## 配置模型

在该示例中，我们实现了两种文本分类算法，分别基于[推荐系统](https://github.com/PaddlePaddle/book/tree/develop/05.recommender_system)一节介绍过的文本卷积神经网络，以及[栈式双向 LSTM ](#栈式双向 LSTM （Stacked Bidirectional  LSTM ）)。

### 文本卷积神经网络

```python
def convolution_net(input_dim,
                    class_dim=2,
                    emb_dim=128,
                    hid_dim=128,
                    is_predict=False):
    data = paddle.layer.data("word",
                             paddle.data_type.integer_value_sequence(input_dim))
    emb = paddle.layer.embedding(input=data, size=emb_dim)
    conv_3 = paddle.networks.sequence_conv_pool(
        input=emb, context_len=3, hidden_size=hid_dim)
    conv_4 = paddle.networks.sequence_conv_pool(
        input=emb, context_len=4, hidden_size=hid_dim)
    output = paddle.layer.fc(input=[conv_3, conv_4],
                             size=class_dim,
                             act=paddle.activation.Softmax())
    if not is_predict:
        lbl = paddle.layer.data("label", paddle.data_type.integer_value(2))
        cost = paddle.layer.classification_cost(input=output, label=lbl)
        return cost
    else:
        return output
```
网络的输入`input_dim`表示的是词典的大小，`class_dim`表示类别数。这里，我们使用[`sequence_conv_pool`](https://github.com/PaddlePaddle/Paddle/blob/develop/python/paddle/trainer_config_helpers/networks.py) API实现了卷积和池化操作。

### 栈式双向 LSTM

```python
def stacked_ LSTM _net(input_dim,
                     class_dim=2,
                     emb_dim=128,
                     hid_dim=512,
                     stacked_num=3,
                     is_predict=False):
    """
    A Wrapper for sentiment classification task.
    This network uses bi-directional recurrent network,
    consisting three  LSTM  layers. This configure is referred to
    the paper as following url, but use fewer layrs.
        http://www.aclweb.org/anthology/P15-1109

    input_dim: here is word dictionary dimension.
    class_dim: number of categories.
    emb_dim: dimension of word embedding.
    hid_dim: dimension of hidden layer.
    stacked_num: number of stacked  LSTM -hidden layer.
    """
    assert stacked_num % 2 == 1

    fc_para_attr = paddle.attr.Param(learning_rate=1e-3)
     LSTM _para_attr = paddle.attr.Param(initial_std=0., learning_rate=1.)
    para_attr = [fc_para_attr,  LSTM _para_attr]
    bias_attr = paddle.attr.Param(initial_std=0., l2_rate=0.)
    relu = paddle.activation.Relu()
    linear = paddle.activation.Linear()

    data = paddle.layer.data("word",
                             paddle.data_type.integer_value_sequence(input_dim))
    emb = paddle.layer.embedding(input=data, size=emb_dim)

    fc1 = paddle.layer.fc(input=emb,
                          size=hid_dim,
                          act=linear,
                          bias_attr=bias_attr)
     LSTM 1 = paddle.layer. LSTM emory(
        input=fc1, act=relu, bias_attr=bias_attr)

    inputs = [fc1,  LSTM 1]
    for i in range(2, stacked_num + 1):
        fc = paddle.layer.fc(input=inputs,
                             size=hid_dim,
                             act=linear,
                             param_attr=para_attr,
                             bias_attr=bias_attr)
         LSTM  = paddle.layer. LSTM emory(
            input=fc,
            reverse=(i % 2) == 0,
            act=relu,
            bias_attr=bias_attr)
        inputs = [fc,  LSTM ]

    fc_last = paddle.layer.pooling(input=inputs[0], pooling_type=paddle.pooling.Max())
     LSTM _last = paddle.layer.pooling(input=inputs[1], pooling_type=paddle.pooling.Max())
    output = paddle.layer.fc(input=[fc_last,  LSTM _last],
                             size=class_dim,
                             act=paddle.activation.Softmax(),
                             bias_attr=bias_attr,
                             param_attr=para_attr)

    if not is_predict:
        lbl = paddle.layer.data("label", paddle.data_type.integer_value(2))
        cost = paddle.layer.classification_cost(input=output, label=lbl)
        return cost
    else:
        return output
```
网络的输入`stacked_num`表示的是 LSTM 的层数，需要是奇数，确保最高层 LSTM 正向。Paddle 里面是通过一个 fc 和一个 LSTMemory 来实现基于 LSTM 的循环神经网络。

## 训练模型

```python
if __name__ == '__main__':
    # init
    paddle.init(use_gpu=False)
```
启动 paddle 程序，use_gpu=False 表示用 CPU 训练，如果系统支持 GPU 也可以修改成 True 使用 GPU 训练。

### 训练数据

使用 Paddle 提供的数据集`dataset.imdb`中的 API 来读取训练数据。
```python
    print 'load dictionary...'
    word_dict = paddle.dataset.imdb.word_dict()
    dict_dim = len(word_dict)
    class_dim = 2
```
加载数据字典，这里通过`word_dict()`API可以直接构造字典。`class_dim`是指样本类别数，该示例中样本只有正负两类。
```python
    train_reader = paddle.batch(
        paddle.reader.shuffle(
            lambda: paddle.dataset.imdb.train(word_dict), buf_size=1000),
        batch_size=100)
    test_reader = paddle.batch(
        lambda: paddle.dataset.imdb.test(word_dict),
        batch_size=100)
```
这里，`dataset.imdb.train()` 和 `dataset.imdb.test()` 分别是 `dataset.imdb` 中的训练数据和测试数据 API。`train_reader` 在训练时使用，意义是将读取的训练数据进行 shuffle 后，组成一个 batch 数据。同理，`test_reader`是在测试的时候使用，将读取的测试数据组成一个 batch。

```python
    feeding={'word': 0, 'label': 1}
```

`feeding`用来指定`train_reader`和`test_reader`返回的数据与模型配置中 data_layer 的对应关系。这里表示 reader 返回的第 0 列数据对应`word`层，第1列数据对应`label`层。

### 构造模型

```python
    # Please choose the way to build the network
    # by uncommenting the corresponding line.
    cost = convolution_net(dict_dim, class_dim=class_dim)
    # cost = stacked_ LSTM _net(dict_dim, class_dim=class_dim, stacked_num=3)
```
该示例中默认使用`convolution_net`网络，如果使用`stacked_ LSTM _net`网络，注释相应的行即可。其中 cost 是网络的优化目标，同时 cost 包含了整个网络的拓扑信息。

### 网络参数

```python
    # create parameters
    parameters = paddle.parameters.create(cost)
```
根据网络的拓扑构造网络参数。这里 parameters 是整个网络的参数集。

### 优化算法

```python
    # create optimizer
    adam_optimizer = paddle.optimizer.Adam(
        learning_rate=2e-3,
        regularization=paddle.optimizer.L2Regularization(rate=8e-4),
        model_average=paddle.optimizer.ModelAverage(average_window=0.5))
```
Paddle中提供了一系列优化算法的API，这里使用 Adam 优化算法。

### 训练

可以通过`paddle.trainer.SGD`构造一个 sgd trainer，并调用`trainer.train` 来训练模型。另外，通过给 train 函数传递一个`event_handler`来获取每个batch 和每个 pass 结束的状态。
```python
    # End batch and end pass event handler
    def event_handler(event):
        if isinstance(event, paddle.event.EndIteration):
            if event.batch_id % 100 == 0:
                print "\nPass %d, Batch %d, Cost %f, %s" % (
                    event.pass_id, event.batch_id, event.cost, event.metrics)
            else:
                sys.stdout.write('.')
                sys.stdout.flush()
        if isinstance(event, paddle.event.EndPass):
            with open('./params_pass_%d.tar' % event.pass_id, 'w') as f:
                trainer.save_parameter_to_tar(f)

            result = trainer.test(reader=test_reader, feeding=feeding)
            print "\nTest with Pass %d, %s" % (event.pass_id, result.metrics)
```
比如，构造如下一个`event_handler`可以在每 100 个 batch 结束后输出 cost 和 error；在每个 pass 结束后调用`trainer.test`计算一遍测试集并获得当前模型在测试集上的 error。

```python
    from paddle.v2.plot import Ploter

    train_title = "Train cost"
    cost_ploter = Ploter(train_title)
    step = 0
    def event_handler_plot(event):
        global step
        if isinstance(event, paddle.event.EndIteration):
            cost_ploter.append(train_title, step, event.cost)
            cost_ploter.plot()
            step += 1
```
或者构造一个`event_handler_plot`画出 cost曲线。

```python
    # create trainer
    trainer = paddle.trainer.SGD(cost=cost,
                                 parameters=parameters,
                                 update_equation=adam_optimizer)

    trainer.train(
        reader=train_reader,
        event_handler=event_handler,
        feeding=feeding,
        num_passes=2)
```

程序运行之后的输出如下。

```text
Pass 0, Batch 0, Cost 0.693721, {'classification_error_evaluator': 0.5546875}
...................................................................................................
Pass 0, Batch 100, Cost 0.294321, {'classification_error_evaluator': 0.1015625}
...............................................................................................
Test with Pass 0, {'classification_error_evaluator': 0.11432000249624252}
```

## 应用模型

可以使用训练好的模型对电影评论进行分类，下面程序展示了如何使用`paddle.infer`接口进行推断。

```python
    import numpy as np

    # Movie Reviews, from imdb test
    reviews = [
        'Read the book, forget the movie!',
        'This is a great movie.'
    ]
    reviews = [c.split() for c in reviews]

    UNK = word_dict['<unk>']
    input = []
    for c in reviews:
        input.append([[word_dict.get(words, UNK) for words in c]])

    # 0 stands for positive sample, 1 stands for negative sample
    label = {0:'pos', 1:'neg'}
    # Use the network used by trainer
    out = convolution_net(dict_dim, class_dim=class_dim, is_predict=True)
    # out = stacked_ LSTM _net(dict_dim, class_dim=class_dim, stacked_num=3, is_predict=True)
    probs = paddle.infer(output_layer=out, parameters=parameters, input=input)

    labs = np.argsort(-probs)
    for idx, lab in enumerate(labs):
        print idx, "predicting probability is", probs[idx], "label is", label[lab[0]]
```

## 总结

本章我们以情感分析为例，介绍了使用深度学习的方法进行端对端的短文本分类，并且使用 PaddlePaddle 完成了全部相关实验。同时，我们简要介绍了两种文本处理模型：卷积神经网络和循环神经网络。在后续的章节中我们会看到这两种基本的深度学习模型在其它任务上的应用。

## 参考文献
1. Kim Y. [Convolutional neural networks for sentence classification](http://arxiv.org/pdf/1408.5882)[J]. arXiv preprint arXiv:1408.5882, 2014.
2. Kalchbrenner N, Grefenstette E, Blunsom P. [A convolutional neural network for modelling sentences](http://arxiv.org/pdf/1404.2188.pdf?utm_medium=App.net&utm_source=PourOver)[J]. arXiv preprint arXiv:1404.2188, 2014.
3. Yann N. Dauphin, et al. [Language Modeling with Gated Convolutional Networks](https://arxiv.org/pdf/1612.08083v1.pdf)[J] arXiv preprint arXiv:1612.08083, 2016.
4. Siegelmann H T, Sontag E D. [On the computational power of neural nets](http://research.cs.queensu.ca/home/akl/cisc879/papers/SELECTED_PAPERS_FROM_VARIOUS_SOURCES/05070215382317071.pdf)[C]//Proceedings of the fifth annual workshop on Computational learning theory. ACM, 1992: 440-449.
5. Hochreiter S, Schmidhuber J. [Long short-term memory](http://web.eecs.utk.edu/~itamar/courses/ECE-692/Bobby_paper1.pdf)[J]. Neural computation, 1997, 9(8): 1735-1780.
6. Bengio Y, Simard P, Frasconi P. [Learning long-term dependencies with gradient descent is difficult](http://www-dsi.ing.unifi.it/~paolo/ps/tnn-94-gradient.pdf)[J]. IEEE transactions on neural networks, 1994, 5(2): 157-166.
7. Graves A. [Generating sequences with recurrent neural networks](http://arxiv.org/pdf/1308.0850)[J]. arXiv preprint arXiv:1308.0850, 2013.
8. Cho K, Van Merriënboer B, Gulcehre C, et al. [Learning phrase representations using RNN encoder-decoder for statistical machine translation](http://arxiv.org/pdf/1406.1078)[J]. arXiv preprint arXiv:1406.1078, 2014.
9. Zhou J, Xu W. [End-to-end learning of semantic role labeling using recurrent neural networks](http://www.aclweb.org/anthology/P/P15/P15-1109.pdf)[C]//Proceedings of the Annual Meeting of the Association for Computational Linguistics. 2015.

本教程 由 PaddlePaddle 创作，采用 知识共享 署名-相同方式共享 4.0 国际 许可协议进行许可。

```python

```


```python
clean_up=True # removes gams-related files in work-folder if true
%run StdPackages.ipynb
os.chdir(directory['py'])
import techdata_to_tree, sys, ShockFunction
import abatement_EOP as abatement
os.chdir(directory['curr'])
```

*Specify settings:*

```python
name = 'A1_calib_test'
f = {'tech': directory['data']+"/techdata_adv.xlsx",
     'other': directory['data']+"/Othdata_adv.xlsx",
     'IO':directory['data'] + "/IOdata_adv.xlsx"}
s = ['inputdisp','endofpipe','inputprices']
m = ['ID','EOP']
e = {'M': 'vars','M_sets': 'sets'} # dictionary used to load emissions data
other_trees = {'ID_Y': {'s': 'Y', 'read_data': {'Y': 'vars'}, 'Q2P': 'Q2P'}} # the key = tree name, s = sheet that specifies tree structure, read_data (if not None) specifies how to load data.
```

```python
type_f = {'ID_EC': 'CES_norm',
          'ID_CU': 'MNL',
          'ID_TU': 'CET_norm',
          'ID_TX': 'CES',
          'ID_BU': 'linear_out',
          'ID_BX': 'CES',
          'EOP_CU': 'MNL',
          'EOP_TU': 'CET_norm',
          'EOP_TX': 'CES'}
```

*Set up nesting tree:*

```python
%run ATree.ipynb
```

## 1: Initialize model

*Set up gams model from tree:*

```python
gm = abatement.abate(nts=nts,tech=tech,use_EOP=True,work_folder=directory['work'],**{'data_folder':directory['gams'],'name': name})
```

*Add a couple of extra aliases, and update sets with elements:*

```python
gm.add_aliases([('n','nnnn'),('n','nnnnn')])
gm.model.database.update_all_sets(clean_up=False)
```

```python
gm.setstate("EOP")
```

*Initialise variables at leontief levels, and fill out with simple standard values in potentially missing domains:*

```python
gm.initialize_variables_leontief()
gm.initialize_variables(**{'check_variables': True})
```

## 2: Solve model

*Set up baseline model and solve:*

```python
gm.write_and_run(name='baseline')
db0 = gm.model_instances['baseline'].out_db
```

*Read back solution to main db:*

```python
gm.model.database.merge_dbs(gm.model.database,db0,'second')
```

*Turn exogenous parameters into place*

```python
db_star = DataBase.GPM_database(**{'name': 'db_star'})
sigma_star = gm.get("sigma").copy()
eta_star = gm.get("eta").copy()
mu_star = gm.get("mu").copy()

eta_star[:] = -3
sigma_star[gm.get("kno_ID_CU")] = 0.4
sigma_star[gm.get("kno_no_ID_BX")] = 0.7
sigma_star["Y"] = 0.5

mu_star[gm.get("map_ID_CU")] = 1.0

db_star["sigma"] = sigma_star
db_star["eta"] = eta_star
db_star["mu"] = mu_star
```

```python
name='v0'
gm.write_and_run(name=name, add_checkpoint=name)
gm.checkpoints['v1'] = gm.model_instances[name].ws.add_checkpoint()
gm.model_instances[name].solve_sneakily(db_star=db_star,from_cp = True, cp_init = gm.checkpoints[name], options_run={'checkpoint': gm.checkpoints['v1']})
gm.model.database.merge_dbs(gm.model.database,gm.model_instances[name].out_db,'second')
```

```python
db_star = DataBase.GPM_database(**{'name': 'db_star'})
sigma_star = gm.get("sigma").copy()
sigma_star[gm.get("kno_ID_CU")] = 5
db_star["sigma"] = sigma_star
```

```python
(shock_db, shock_kwargs) = ShockFunction.sneaky_db(gm.model.database, db_star, n_steps = 200, loop_name = 'sigma_loop')
store_sol = {"PwThat":{'domains': shock_kwargs['loop_name']}, "qD":{'domains': shock_kwargs['loop_name']}, "sigma":{'domains': shock_kwargs['loop_name']}}
mi = gm.model_instances[name]
shock = mi.std_UEVAS_from_db(shock_db, loop_name=shock_kwargs['loop_name'], shock_name=shock_kwargs['shock_name'], store_sol=store_sol)
mi.execute_shock_from_cp(shock=shock, cp=gm.checkpoints['v1'])
gm.model.database.merge_dbs(gm.model.database,mi.out_db,'second')
```

```python

```

*Change to calibration state and resolve:*

```python
gm.reset_settings()
gm.setstate('ID_calibrate',init=False)
gm.write_and_run(name='ID_calibrate',kwargs_init={'check_variables':True},overwrite=True)
db = gm.model_instances['ID_calibrate'].out_db
```

Load in IO data

```python
gm.add_calib_data(f["IO"])
```

```python
# db = DataBase.GPM_database()
# qD = gm.get("qD").copy()

# qD[gm.get("bra_ID_BX")] = qD[gm.get("bra_ID_BX")] + 0.25
# qD["Y_oil"] = qD["Y_oil"] - 1
# db["qD"] = qD

# DataBase.GPM_database.merge_dbs(gm.model.database, db, "second")
```

```python
gm.write_and_run(name='ID_calibrate',kwargs_init={'check_variables':True},overwrite=True)
db = gm.model_instances['ID_calibrate'].out_db
```

*Print some... Note that if the ID technologies do not have any overlap with other technologies, the calibration procedure only has one possible solution; i.e. there are not any room for really minimizing the minobj.*

```python
def compare_plots(db0,db,var):
    return pd.concat([db0.get(var).rename(f"{var}, B"),db.get(var).rename(f"{var}, C")],axis=1).plot.bar();
```

```python
var = 'PwThat'
compare_plots(db0,db,var)
```

```python
var = 'qD'
compare_plots(db0,db,var)
```

```python
var="mu"
compare_plots(db0,db,var)
```

## 3: Test that the baseline model is also solved with the new calibrated values:

## A: Additional equations

### Current application

* Define currapp[n,nn] as the share of energy-services covered by technology goods produced by technology $nn$ to satisfy the energy-service need $n$. In other words:

$$\begin{align}
    \text{currapp[n,nn]} =\text{sum(nnn\$(e2u[n,nnn]), qD[nnn])} / qD[n].
\end{align}$$

* Define the technology shares $U/C$ as $s\_uc[u,c]$ as the shares under the counterfactual scenario outlined in the article. Specifically:

$$\begin{align}
    \text{s_uc[n,nn]} &= \dfrac{exp\left(\sigma[nn]\left(PwThat[nn]-PwThat[n]\right)\right)\mu[n,nn]}{sum\left[nnn\$(\text{non-baseline}[nnn]), share[nnn]\right]+sum\left[nnn\$(\text{baseline}[nnn]) shareB[nnn]\right]} \\
    \sum share[nnn] &= sum\left[nnn\$\left(u2c[nnn,nn]\text{ and }bra\_ID\_TU[nnn]\right), \mu[nnn,nn] exp\left(\sigma[nn]\left(PwThat[nn]-PwThat[nnn]\right)\right)\right] \\
    \sum shareB[nnn]&= sum\left[nnn\$\left(u2c[nnn,nn]\text{ and }bra\_ID\_BU[nnn]\right), \mu[nnn,nn] exp\left(\sigma[nn]\left(PwThat[nn]-PwThatX[nnn]\right)\right)\right]
\end{align}$$
where the $share[nnn]$ is the MNL-like contribution from non-baseline goods in the relevant nest, and $shareB[nnn]$ is the MNL-like contribution from baseline goods. This price is adjusted here, to reflect a technology level ($\tau$ instead of $U$) price:

$$\begin{align}
    PwThatX[nnn] = sum\left[nnnn\$\left(e2u[nnnn,n]\right), sum(nnnnn\$(u2t[n,nnnnn]), gamma\_tau[nnnn,nnnnn]) * sum(nnnnn\$\left(u2t[nnn,nnnnn]\right), PwThat[nnnnn])\right]
\end{align}$$

Here we have: $n$ = technology good, $nn$ = relevant component, $nnn$ = baseline good $U$ in the same nest as good $n$, $nnnn$ = relevant energy-service nest, and $nnnnn$ first identifies the relevant technology $(\tau)$ to good $n$, and afterwards the relevant technology $(\tau)$ to good $nnn$.

Here, the first part identifies the relevant energy-service that good $U$ contributes to. The second part then first identifies the technology-specific component and picks out the $(e,\tau)$ parameter in gamma\_tau. This is multiplied by the baseline technology price.

* Define the modified current application $currapp\_mod[n,nn]$ level with $n$ = energy-service and $nn$=technology:

$$\begin{align}
    \text{currapp_mod[n,nn]} = \text{sum([nnn,nnnn]\$(u2t[nnn,nn] and c2e[nnnn,n]), s_uc[nnn,nnnn]*qD[nnnn]/qD[n])}
\end{align}$$

This takes the hypothetical share $U/C$ and multiplies by the actual share $C/E$.


# Collections of functions to be used as init script

```python
import sys
import pandas as pd
import numpy as np
import numpy.ma as ma
import time
import matplotlib
from colorsys import hsv_to_rgb
import matplotlib.pyplot as plt
from matplotlib.ticker import MultipleLocator, FormatStrFormatter, MaxNLocator
import matplotlib.patches as patches
import matplotlib.dates as md
import matplotlib.dates as mdates
import datetime
import sympy as sp
import seaborn as sns
import itertools
import pickle
import gzip
import os
from scipy.interpolate import interp1d
from collections import OrderedDict
import glob
from pprint import pprint
import pytz
from scipy.signal import savgol_filter
from operator import itemgetter
import matplotlib.gridspec as gridspec

import cl2pd
from cl2pd import importData
from cl2pd import plotFunctions
from cl2pd import dotdict
from cl2pd import bbFunctions
from cl2pd import variablesDF
cals=importData.cals

import warnings
warnings.filterwarnings("ignore")
print('/eos/user/n/nkarast/myLibrary/init.ipynb :: Ignoring warnings.')
```

    /eos/user/n/nkarast/myLibrary/init.ipynb :: Ignoring warnings.

```python
%run -i ~/myLibrary/resonanceLines.ipynb
```

    ERROR:root:File `u'/eos/user/n/nkarast/myLibrary/resonanceLines.ipynb.py'` not found.

# Some Global Settings

```python
_timezone = pytz.timezone("Europe/Zurich")

sns.set_style("white")
sns.set_style("ticks")
sns.set_context("paper")

plt.rcParams['axes.labelsize']   = 18
plt.rcParams['axes.labelweight'] = 'normal'
plt.rcParams['xtick.labelsize']  = 16
plt.rcParams['ytick.labelsize']  = 16
plt.rcParams['legend.fontsize']  = 16
plt.rcParams['figure.titlesize'] = 20

from IPython.core.display import display, HTML
display(HTML("<style>.container { width:100% !important; }</style>"))

get_ipython().magic('matplotlib inline')
```

<style>.container { width:100% !important; }</style>

# And the functions begin...

```python
def setJupyterWidth():
    '''Sets Jupyter Notebook width to 100% via HTML'''
    from IPython.core.display import display, HTML
    display(HTML("<style>.container { width:100% !important; }</style>"))
```

```python
def convertNumpyToTimestamp(x):
    '''converts numpy to timestamp'''
    try:
        return pd.DatetimeIndex(x)[0]
    except:
        return x
```

```python
def getFilledSlotsArray(dict_intervals_two_beams, beam, cycle, cycleTime,  mask_invalid=True):
    '''
    Returns the array with the filled slots.
    Inputs : beam           : beam string ('beam_1', 'beam_2')
             cycle          : cycle string ('injection', 'flattop')
             cycleTime      : cycle step string ('injection_start', 'injection_end', 'flattop_start', 'flattop_end')
             mask_invalid   : boolean True/False to mask invalid values in the output array
    Returns: filled_slots array
    '''
    return ma.masked_invalid(dict_intervals_two_beams[beam][cycle]['filled_slots'])
```

```python
def convertToLocalTime(df_timeRange, timezone="Europe/Zurich", scalar=False):
    '''
    Converts a Series (Pandas Column) in Local Time and returns it.
    Input  : df_timeRange : Pandas Series with timestamp data
             timezone     : timezone (tz) string - defaults to Europe/Zurich
             scalar 	  : if it is a single value and not list/array
    Returns: converted pandas Series
    '''
    if isinstance(df_timeRange, list) or isinstance(df_timeRange, np.ndarray) or isinstance(df_timeRange, pd.core.series.Series):
        return np.array(pd.to_datetime(np.array(df_timeRange), unit='s', utc=True).tz_convert(timezone).tz_localize(None))
    else:
        return datetime.datetime.utcfromtimestamp(df_timeRange) + (datetime.datetime.now()-datetime.datetime.utcnow()) # + utc offset to fix local zone

```

```python
def plot_mean_and_spread(axc, x_vect, ymat, color = 'k', alpha=1, label=None, shade=False):
    '''
    Function to plot the average value and the +/- 2sigma range of a given
    vector on a given axis

    Inputs : axc : axis on which you want me to plot stuff
             x_vect: array of x-axis values (any iterable object)
             y_mat : array of y-axis values for which the std and mean will be Calculated
             color : the color for the lines
             alpha : the alpha of the lines
             label : label to add to this axis
             shade : plot in shade the +/- 1 standard deviation band
    Returns: None -- simply adds stuff on the given pl.axis
    '''
    avg = np.nanmean(ymat, axis=1)
    std = np.nanstd(ymat, axis=1)

    if shade:
        axc.fill_between(x_vect, avg-std, avg+std, alpha=.3, color=color, label=None)
    else:
        axc.plot(x_vect, avg-1*std, '--', color=color, linewidth=1, alpha=alpha, label=None)
        axc.plot(x_vect, avg+1*std, '--', color=color, linewidth=1, alpha=alpha, label=None)
        axc.plot(x_vect, avg, color=color, linewidth=2, alpha=alpha, label=label)
```

```python
def colorprog(i_prog, Nplots, v1 = .9, v2 = 1.):
    '''
    Function to return different colors for bunch plots
    '''
    if hasattr(Nplots, '__len__'):
        Nplots = len(Nplots)
    return hsv_to_rgb(float(i_prog)/float(Nplots), v1, v2)

```

```python
def getLHCSuperTable(fill_min=None, fill_max=None, t1=None, t2=None):
    '''
    Returns a "SuperTable" for fills within fill_min, fill_max or in time range t1,t2
    '''
    st_vars = ['LHC.STATS:FILL_NUMBER', 'LHC.STATS:ENERGY', 'LHC.STATS:LHC:INJECTION_SCHEME', 'LHC.STATS:B1_NUMBER_BUNCHES', 'LHC.STATS:NUMBER_COLLISIONS_IP1_5', 'LHC.STATS:NUMBER_COLLISIONS_IP2', 'LHC.STATS:NUMBER_COLLISIONS_IP8',
               'LHC.STATS:BETA_STAR_ALICE', 'LHC.STATS:BETA_STAR_ATLAS', 'LHC.STATS:BETA_STAR_CMS', 'LHC.STATS:BETA_STAR_LHCB',  'LHC.STATS:LUMI_ATLAS_DELIVERED', 'LHC.STATS:LUMI_ATLAS_PEAK', 'LHC.STATS:LUMI_CMS_DELIVERED', 'LHC.STATS:LUMI_CMS_PEAK',
               'LHC.STATS:LHCB_HALF_CROSSING_ANGLE', 'LHC.STATS:ALICE_HALF_CROSSING_ANGLE', 'LHC.STATS:ATLAS_HALF_CROSSING_ANGLE', 'LHC.STATS:CMS_HALF_CROSSING_ANGLE',
               'LHC.STATS:SPS_H_ROT_IN:EMITTANCE', 'LHC.STATS:SPS_V_ROT_IN:EMITTANCE']

    if fill_min is not None and fill_max is not None and t1 is None and t2 is None:
        print "Running SuperTable by fills [{}, {}]...".format(fill_min, fill_max)
        log = pytimber.LoggingDB()
        d1 = log.getLHCFillData(fill_min, True)
        t1 = convertToLocalTime(d1['startTime'])
        d2 = log.getLHCFillData(fill_max, True)
        t2 = convertToLocalTime(d2['endTime'])
        print "Corresponding start/end time is : [{}, {}]".format(t1, t2)
    elif fill_min is None and fill_max is None and t1 is not None and t2 is not None:
        print "Running SuperTable by time [{},{}]...".format(t1, t2)
    else:
        print "Getting SuperTable for the last five days..."
        t1 = datetime.datetime.now() - datetime.timedelta(days=5)
        t2 = datetime.datetime.now()

    df = myToolbox.cals2pnd(st_vars, t1,t2)
    df.columns = [x.split('LHC.STATS:')[1] for x in df.columns.tolist()]
    df = df.fillna(method='ffill').fillna(method='bfill')
    return df.set_index(['FILL_NUMBER'])

```

```python
def convertNumpyDatetimeToDatetime(times):
    '''
    converts the np.datetime64 to datetime.datetime at local time
    returns an array object
    '''
    result = np.array([x.replace(tzinfo=pytz.utc).astimezone(_timezone) for x in map(datetime.datetime.utcfromtimestamp, times.astype(np.int64)/1e9)])
    if len(result)==1:
        return result[0]
    else:
        return result
```

```python
class pp():
    "This an auxillary class to allow for LaTeX output between subsequent derivation steps."
    def __init__(self, eq, left=None):
        self.eq = eq
        self.left = left
    def _repr_html_(self):
        if self.left:
            return "$" + sp.latex(self.left) + " = " + sp.latex(self.eq) + "$"
        return "$" + sp.latex(self.eq) + "$"
```

```python
def printBSRTCalibrationFactor(filln):
    '''given a fill prints out the calibration factors (mean over FB/FT)'''
    tf1, tf2 = getLHCFillTimes(filln)
    fills, bmodes = myToolbox.LHCFillsByTime2pnd(tf1, tf2)

    t_start_inj   = bmodes['startTime'][bmodes['mode']=='INJPHYS'].values[0]
    t_start_ramp  = bmodes['endTime'][bmodes['mode']=='PRERAMP'].values[0]
    t_start_ft    = bmodes['startTime'][bmodes['mode']=='FLATTOP'].values[0]
    t_end_sb      = bmodes['endTime'][bmodes['mode']=='STABLE'].values[0]

    dt_start_inj  = datetime.datetime.utcfromtimestamp(t_start_inj.tolist()/1e9)
    dt_start_ramp = datetime.datetime.utcfromtimestamp(t_start_ramp.tolist()/1e9)
    dt_start_ft   = datetime.datetime.utcfromtimestamp(t_start_ft.tolist()/1e9)
    dt_end_sb     = datetime.datetime.utcfromtimestamp(t_end_sb.tolist()/1e9)

    beam_vars = { 1:['LHC.BSRT.5R4.B1:BETA_H',
                      'LHC.BSRT.5R4.B1:BETA_V',
                      'LHC.BSRT.5R4.B1:SCALE_H',
                      'LHC.BSRT.5R4.B1:SCALE_V',
                      'LHC.BSRT.5R4.B1:LSF_H',
                      'LHC.BSRT.5R4.B1:LSF_V',
              ],

             2: [ 'LHC.BSRT.5L4.B2:BETA_H',
                  'LHC.BSRT.5L4.B2:BETA_V',
                  'LHC.BSRT.5L4.B2:SCALE_H',
                  'LHC.BSRT.5L4.B2:SCALE_V',
                  'LHC.BSRT.5L4.B2:LSF_H',
                  'LHC.BSRT.5L4.B2:LSF_V',
              ],
            }
    dfs = {1: {'FB': None, 'FT': None},
           2: {'FB': None, 'FT': None}}
    for beam_n in [1,2]:
        for tag in ['FB', 'FT']:
            print("Working on beam-{} {}".format(beam_n, tag))
            tmp_vars = beam_vars[beam_n]
            tmp_df = pd.DataFrame()

            if tag == 'FB':
                t1 = dt_start_inj
                t2 = dt_start_ramp
            elif tag == 'FT':
                t1 = dt_start_ft
                t2 = dt_end_sb

            tmp_df = myToolbox.cals2pnd(tmp_vars, t1, t2)
            dfs[beam_n][tag]=tmp_df
    print('''
            for kk in e_dict.keys():
                e_dict[kk] = {450:{}, 6500:{}}'''+'''

            e_dict['betaf_h'][450][1]       = {}
            e_dict['betaf_h'][6500][1]      = {}

            e_dict['betaf_v'][450][1]       = {}
            e_dict['betaf_v'][6500][1]      = {}

            e_dict['sigma_corr_h'][450][1]  = {}
            e_dict['sigma_corr_h'][6500][1] = {}

            e_dict['sigma_corr_v'][450][1]  = {}
            e_dict['sigma_corr_v'][6500][1] = {}

            e_dict['scale_h'][450][1]       = {}
            e_dict['scale_h'][6500][1]      = {}

            e_dict['scale_v'][450][1]       = {}
            e_dict['scale_v'][6500][1]      = {}

            e_dict['rescale_sigma_h'][450][1]  = 1.
            e_dict['rescale_sigma_h'][6500][1] = 1.
            e_dict['rescale_sigma_v'][450][1]  = 1.
            e_dict['rescale_sigma_v'][6500][1] = 1.

            # Beam 2:
            e_dict['betaf_h'][450][2]       = {}
            e_dict['betaf_h'][6500][2]      = {}

            e_dict['betaf_v'][450][2]       = {}
            e_dict['betaf_v'][6500][2]      = {}

            e_dict['sigma_corr_h'][450][2]  = {}
            e_dict['sigma_corr_h'][6500][2] = {}

            e_dict['sigma_corr_v'][450][2]  = {}
            e_dict['sigma_corr_v'][6500][2] = {}

            e_dict['scale_h'][450][2]       = {}
            e_dict['scale_h'][6500][2]      = {}

            e_dict['scale_v'][450][2]       = {}
            e_dict['scale_v'][6500][2]      = {}

            e_dict['rescale_sigma_h'][450][2]  = 1.
            e_dict['rescale_sigma_h'][6500][2] = 1.
            e_dict['rescale_sigma_v'][450][2]  = 1.
            e_dict['rescale_sigma_v'][6500][2] = 1.

            # gamma
            e_dict['gamma'][450] = 479.6
            e_dict['gamma'][6500] = 6927.6

            print('Using calibration TEST')
                '''.format(dfs[1]['FB']['LHC.BSRT.5R4.B1:BETA_H'].mean(),
                           dfs[1]['FT']['LHC.BSRT.5R4.B1:BETA_H'].mean(),
                           dfs[1]['FB']['LHC.BSRT.5R4.B1:BETA_V'].mean(),
                           dfs[1]['FT']['LHC.BSRT.5R4.B1:BETA_V'].mean(),
                           dfs[1]['FB']['LHC.BSRT.5R4.B1:LSF_H'].mean(),
                           dfs[1]['FT']['LHC.BSRT.5R4.B1:LSF_H'].mean(),
                           dfs[1]['FB']['LHC.BSRT.5R4.B1:LSF_V'].mean(),
                           dfs[1]['FT']['LHC.BSRT.5R4.B1:LSF_V'].mean(),
                           dfs[1]['FB']['LHC.BSRT.5R4.B1:SCALE_H'].mean(),
                           dfs[1]['FT']['LHC.BSRT.5R4.B1:SCALE_H'].mean(),
                           dfs[1]['FB']['LHC.BSRT.5R4.B1:SCALE_V'].mean(),
                           dfs[1]['FT']['LHC.BSRT.5R4.B1:SCALE_V'].mean(),
                           dfs[2]['FB']['LHC.BSRT.5L4.B2:BETA_H'].mean(),
                           dfs[2]['FT']['LHC.BSRT.5L4.B2:BETA_H'].mean(),
                           dfs[2]['FB']['LHC.BSRT.5L4.B2:BETA_V'].mean(),
                           dfs[2]['FT']['LHC.BSRT.5L4.B2:BETA_V'].mean(),
                           dfs[2]['FB']['LHC.BSRT.5L4.B2:LSF_H'].mean(),
                           dfs[2]['FT']['LHC.BSRT.5L4.B2:LSF_H'].mean(),
                           dfs[2]['FB']['LHC.BSRT.5L4.B2:LSF_V'].mean(),
                           dfs[2]['FT']['LHC.BSRT.5L4.B2:LSF_V'].mean(),
                           dfs[2]['FB']['LHC.BSRT.5L4.B2:SCALE_H'].mean(),
                           dfs[2]['FT']['LHC.BSRT.5L4.B2:SCALE_H'].mean(),
                           dfs[2]['FB']['LHC.BSRT.5L4.B2:SCALE_V'].mean(),
                           dfs[2]['FT']['LHC.BSRT.5L4.B2:SCALE_V'].mean()))
```

```python
def getFillingScheme(filln):
    '''Given a fill number return:
    filled_slots_b1, filled_slots_b2, slots_coll_b1, slots_coll_b2, slots_noncoll_b1, slots_noncoll_b2, b1_fill_pattern, b2_fill_pattern
    '''

    bmodes = importData.LHCFillsByNumber(filln)
    FILL_PATTERN_DF = pd.DataFrame()
    FILL_PATTERN_DF = importData.cals2pd(['LHC.BCTFR.A6R4.B1:BUNCH_FILL_PATTERN','LHC.BCTFR.A6R4.B2:BUNCH_FILL_PATTERN'], pd.Timestamp(bmodes['startTime'][bmodes['mode']=='PRERAMP'].values[0]), pd.Timestamp(bmodes['startTime'][bmodes['mode']=='PRERAMP'].values[0]+pd.Timedelta(seconds=1)) )
    b1_fill_pattern = np.array(FILL_PATTERN_DF.filter(regex="LHC.*B1.*").iloc[0].values[0]);
    b2_fill_pattern = np.array(FILL_PATTERN_DF.filter(regex="LHC.*B2.*").iloc[0].values[0]);

    # define the filled slots and the colliding / non-colliding
    filled_slots_b1 = np.where(b1_fill_pattern == 1)[0]
    filled_slots_b2 = np.where(b2_fill_pattern == 1)[0]
    slots_noncoll_b1 = np.array([x for x in filled_slots_b1 if x not in filled_slots_b2])
    slots_noncoll_b2 = np.array([x for x in filled_slots_b2 if x not in filled_slots_b1])
    slots_coll_b1 = np.array([x for x in filled_slots_b1 if x in filled_slots_b2])
    slots_coll_b2 = np.array([x for x in filled_slots_b2 if x in filled_slots_b1])

    print('---------- FILL {} ----------'.format(filln))
    print('B1 slots               : {}'.format(len(filled_slots_b1)))
    print('B2 slots               : {}'.format(len(filled_slots_b2)))
    print('B1 colliding slots     : {}'.format(len(slots_coll_b1)))
    print('B2 colliding slots     : {}'.format(len(slots_coll_b2)))
    print('B1 non-colliding slots : {}'.format(len(slots_noncoll_b1)))
    print('B2 non-colliding slots : {}'.format(len(slots_noncoll_b2)))
    return filled_slots_b1, filled_slots_b2, slots_coll_b1, slots_coll_b2, slots_noncoll_b1, slots_noncoll_b2, b1_fill_pattern, b2_fill_pattern

```

```python
def plotFillingScheme(filln):
    filled_slots_b1, filled_slots_b2, slots_coll_b1, slots_coll_b2, slots_noncoll_b1, slots_noncoll_b2, b1_fill_pattern, b2_fill_pattern = getFillingScheme(filln)
    fig1 = plt.figure('fillPattern', figsize=(12,9))
    ax1 = plt.subplot(211)
    ax1.step(np.arange(len(b1_fill_pattern)), b1_fill_pattern, where='post', color='b', lw=2);
    plt.yticks([0,1], ['EMPTY', 'FILLED']);
    ax1.set_xlim(-100, 3570);
    ax1.set_ylim(0, 1.1);
    ax1.set_ylabel('BEAM 1', fontsize=18);
    ax1.set_title('FILL {}'.format(filln), fontsize=20);

    # -- beam 2
    ax2 = plt.subplot(212, sharex=ax1, sharey=ax1);
    ax2.step(np.arange(len(b2_fill_pattern)), b2_fill_pattern, where='post', color='r', lw=2);
    ax2.set_ylabel('BEAM 2', fontsize=18);
    plt.yticks([0,1], ['EMPTY', 'FILLED']);
    ax2.set_xlabel('Bunch Slot [25ns]', fontsize=16);
```

```python
def getBeamBatches(filln):
    '''Given a fill number, aquire the filling pattern and return a list of lists for b1_trains, b2_trains'''
    filled_slots_b1, filled_slots_b2, slots_coll_b1, slots_coll_b2, slots_noncoll_b1, slots_noncoll_b2, b1_fill_pattern, b2_fill_pattern = getFillingScheme(filln)
    b1_trains, b2_trains = [], []
    for k,g in itertools.groupby(enumerate(filled_slots_b1),lambda x:x[0]-x[1]):
        group = (map(itemgetter(1),g))
        group = list(map(int,group))
        b1_trains.append(group)
    for k,g in itertools.groupby(enumerate(filled_slots_b2),lambda x:x[0]-x[1]):
        group = (map(itemgetter(1),g))
        group = list(map(int,group))
        b2_trains.append(group)#ranges.append((group[0],group[-1]))

    return b1_trains, b2_trains

```

```python
def getNewColor(nmax):
    '''given: nmax, n (maximum length and current id) return a new color to be used in loop'''
    color=iter(plt.cm.rainbow(np.linspace(0,1,nmax)))
    colors = []
    for i in xrange(nmax):
        c=next(color)
        colors.append(c)

    return colors
```

```python
def formatDatetimeAxis(ax, which='x', fmt="%H:%M"):
    '''Sets the format datetime axis of (ax) of (which=x|y|both) axis to fmt ("%H:%M")
    '''
    if which=='x':
        ax.xaxis.set_major_formatter(mdates.DateFormatter(fmt))
    elif which == 'y':
        ax.yaxis.set_major_formatter(mdates.DateFormatter(fmt))
    elif which =='both':
        ax.xaxis.set_major_formatter(mdates.DateFormatter(fmt))
        ax.yaxis.set_major_formatter(mdates.DateFormatter(fmt))
    else:
        return
```

```python
def setAxisMaxLocator(ax, which='y', nloc=10):
    '''Sets the maxlocator of (ax) of (which=x|y|both) axis to nloc (10)
    '''
    if which=='y':
        ax.yaxis.set_major_locator(MaxNLocator(nloc))
    elif which=='x':
        ax.xaxis.set_major_locator(MaxNLocator(nloc))
    elif which=='both':
        ax.xaxis.set_major_locator(MaxNLocator(nloc))
        ax.yaxis.set_major_locator(MaxNLocator(nloc))
    else:
        return
```

```python
def getEncounterSchedule(filln, nInsideD1=5, nLR=[16, 16, 15]):
    '''
    Returns the encounter pattern:
    ndf_b1, ndf_b2 = collision schedules for b1, b2
    '''
    filled_slots_b1, filled_slots_b2, slots_coll_b1, slots_coll_b2, slots_noncoll_b1, slots_noncoll_b2, b1_fill_pattern, b2_fill_pattern = getFillingScheme(filln)
    BBMATRIX = bbFunctions.computeBBMatrix(nLR)
    b1_collisionScheduleDF = bbFunctions.B1CollisionScheduleDF(b1_fill_pattern, b2_fill_pattern, nLR)
    b2_collisionScheduleDF = bbFunctions.B2CollisionScheduleDF(b1_fill_pattern, b2_fill_pattern, nLR)
    bbencounters = bbFunctions.BBEncounterSchedule(filled_slots_b1, filled_slots_b2, BBMATRIX)

    # Collides in IP2
    # -- B1
    b1_collisionScheduleDF['collidesIP2'] = b1_collisionScheduleDF["HO partner in ALICE"].fillna(-1)
    b1_collisionScheduleDF['collidesIP2'][b1_collisionScheduleDF['collidesIP2']>=0] = 2
    # -- B2
    b2_collisionScheduleDF['collidesIP2'] = b2_collisionScheduleDF["HO partner in ALICE"].fillna(-1)
    b2_collisionScheduleDF['collidesIP2'][b2_collisionScheduleDF['collidesIP2']>=0] = 2

    # Collides in IP8
    # -- B1
    b1_collisionScheduleDF['collidesIP8'] = b1_collisionScheduleDF["HO partner in LHCB"].fillna(-1)
    b1_collisionScheduleDF['collidesIP8'][b1_collisionScheduleDF['collidesIP8']>=0] = 3
    # -- B2
    b2_collisionScheduleDF['collidesIP8'] = b2_collisionScheduleDF["HO partner in LHCB"].fillna(-1)
    b2_collisionScheduleDF['collidesIP8'][b2_collisionScheduleDF['collidesIP8']>=0] = 3

    # Collides in IP1/5
    # -- B1
    b1_collisionScheduleDF['collidesIP1'] = b1_collisionScheduleDF["HO partner in ATLAS/CMS"].fillna(-1)
    b1_collisionScheduleDF['collidesIP1'][b1_collisionScheduleDF['collidesIP1']>=0] = 1
    # -- B2
    b2_collisionScheduleDF['collidesIP1'] = b2_collisionScheduleDF["HO partner in ATLAS/CMS"].fillna(-1)
    b2_collisionScheduleDF['collidesIP1'][b2_collisionScheduleDF['collidesIP1']>=0] = 1

    ndf_b1 = pd.DataFrame()
    ndf_b2 = pd.DataFrame()

    ndf_b1['IP15'] = b1_collisionScheduleDF['# of LR in ATLAS/CMS']
    ndf_b1['IP2']  = b1_collisionScheduleDF['# of LR in ALICE']
    ndf_b1['IP8']  = b1_collisionScheduleDF['# of LR in LHCB']
    ndf_b1.index   = b1_collisionScheduleDF.index

    ndf_b2['IP15'] = b2_collisionScheduleDF['# of LR in ATLAS/CMS']
    ndf_b2['IP2']  = b2_collisionScheduleDF['# of LR in ALICE']
    ndf_b2['IP8']  = b2_collisionScheduleDF['# of LR in LHCB']
    ndf_b2.index   = b2_collisionScheduleDF.index

    return ndf_b1, ndf_b2
```

```python

```


# Parameter Selection for the Functionally Assembled Terrestrial Ecosystem Simulator (FATES)

__Summary__

Numerical models that simulate tropical forest ecosystem dynamics, such as the Functionally Assembled Terrestrial Ecosystem Simulator (FATES), have been proposed as a way to improve projections of future climate change. However, parameterizing these complex, process-based models is challenging due to their numerous and interconnected non-linear relationships and relatively long spin-up requirements. This analysis identifies three high-performing parameter sets for use in future FATES experiments by quantitatively evaluating the performance of nearly 600 simulations, which test 300 unique parameter sets in two different background environments, against diverse observations at a tropical forest test site.

__Motivation__

Greenhouse gas emissions from human activities are warming the Earth with potentially catastrophic implications for human health. Predicting twenty-first century warming is critical to climate change mitigation and adaptation efforts. Unfortunately, the numerical models used to project future climate differ in their predictions of warming even for the same greenhouse gas emissions scenario. This variability in projections across models stems largely from uncertainty about how Earth's vegetation will respond to environmental change. In particular, predictions of tropical forest responses to climate change must be improved, as these forests exert strong control over global climate. Numerical, process-based models of vegetated ecosystem dynamics, such as the Functionally Assembled Terrestrial Ecosystem Simulator (FATES; Fisher et al., 2015, 2018), have been proposed as a way of improving projections of tropical forests and thus future climate. However, parameterizing these complex, process-based models remains challenging due to their numerous interconnected, non-linear relationships and long spin-up requirements.

__Goal__

The goal of this analysis is to identify parameter sets that allow the FATES model to best match observations of tropical forest structure and functioning at a test site.

__Methods__

This analysis identifies three high-performing parameter sets for use in future FATES experiments by quantitatively evaluating the performance of nearly 600 simulations, which test nearly 300 unique parameter sets in two different background environments, against diverse observations at a tropical forest test site.

_Parameter ensemble simulations_

Prior to this analysis, we ran an ensemble of FATES simulations for Barro Colorado Island, Panama. These simulations were initialized with 287 unique parameter sets, which differed in 12 key plant trait parameters that were sampled from observed distributions when possible (following Koven et al., _in review_). Plant structure and functioning are sensitive to atmospheric carbon dioxide concentration, which increased over the observational time period. We therefore tested each parameter set under two carbon dioxide concentrations (i.e., background environments) that approximately bookend the observational time period (367 ppm and 400 ppm carbon dioxide). All simulations were forced with repeating meteorological data for Barro Colorado Island, Panama, from the years 2003 to 2016 (Faybishenko et al., 2018). See Kovenock (2019) for further details of the parameter ensemble simulations.

_Parameter set evaluation and selection_

This code uses the above ensemble of simulations to quantitatively evaluate the performance of each parameter set against observations of six variables at our tropical forest test site. These variables characterize ecosystem structure (leaf area index, above-ground biomass, basal area) and functioning (gross primary productivity, latent heat flux, sensible heat flux). Observations come from the following sources:  leaf area index from Detto et al. (2018); above-ground biomass from Meakem et al. (2018), Feeley et al. (2007), and Baraloto et al. (2013); basal area from Condit et al. (2017, 2012), Condit (1998), and Hubbell et al. (1999); and gross primary productivity, latent heat flux, and sensible heat flux from Koven et al. (_in review_). (As some of these data sets require permission to use or are not yet publicly available, observational data sets are not included for download here.)

We use two performance metrics $-$ error rate and normalized root mean square error (NRMSE) $-$ to quantify each parameter set's performance. The error rate measures how frequently the model output falls within the observed range for each variable. The NRMSE measures the distance between the model output and the observed mean, relative to the observed range for each variable. The expectation is that high-performing parameter sets will result in model output that falls within the range of observations (low error rate) and near the observed mean (low NRMSE). After evaluating a parameter set's performance for each individual variable, we calculate the weighted average performance across all variables for that parameter set. To ensure that our selection of a high-performing parameter set is robust to weighting method, we consider three different weighting approaches:  even weighting, weighting favoring structural variables (leaf area index, basal area, and above-ground biomass), and weighting accounting for correlation between individual variable performance values.

Lastly, we identify parameter sets for use in future experiments by assigning an overall rank to each parameter set based on its performance across both performance metrics, three weighted averaging approaches, and two background carbon dioxide concentrations.

__Results__

This analysis identifies three high-performing parameter sets for use in future FATES simulations. We recommend the highest-performing parameter set for use in primary experiments and the next two highest-performing parameter sets for use in parameter sensitivity tests. These high-performing parameter sets are publicly available through the University of Washington ResearchWorks digital repository at http://hdl.handle.net/1773/43779. The performance of these parameter sets is reported in further detail in the [Results](#Results) section below and in Kovenock (2019).

## Analysis
## Step 1: Load libraries

```python
import netCDF4 as nc4
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from scipy import stats

from psfxns import annmeans
from psfxns import metrics
from psfxns import plotmetrics
```

## Step 2: Load and preprocess data

This code section returns two multidimensional arrays, one for model output and one for observations, that contain annual mean time series for six variables organized by parameter set and background carbon dioxide concentration. The six ecosystem characteristics we analyze are: leaf area index, above-ground biomass, basal area, gross primary productivity, latent heat flux, and sensible heat flux.

### 2.1  Model output

This part of the code loads and preprocesses the model output for all FATES simulations in our parameter ensemble. It returns a 4-D array called 'model_data' that contains annual mean timeseries for each variable, parameter set, and background carbon dioxide concentration.

```python
# Return model_data: a 4-D array of annual mean values
# with shape (CO2levels, varlist, nens, nyrs).

# Model output file path and names
model_filepath = 'data/'
name1 = 'fates_clm5_fullmodel_bci_parameter_ensemble_1pft_slaprofile_{}_v001.'
name2 = 'I2000Clm50FatesGs.Cdf9b02d-Fb178808.2018-07-27.h{}.ensemble.sofar.nc'
model_filenames = [name1 + name2,
                   name1 + name2]

# Variable list
varlist = ['TLAI', 'AGB', 'BA', 'GPP', 'FLH', 'FSH']

# File type, indexed by varlist:
# 0 = monthly mean for entire ecosystem; and
# 1 = annual mean by tree size.
varfiletype = [0, 1, 1, 0, 0, 0]

# Conversion factor, indexed by varlist:
varconv = [1, 1, 1, 86400*365, 1, 1]

# Background carbon dioxide (CO2) concentrations
CO2levels = ['367ppm', '400ppm']

# Number of years of model output to analyze
nyrs = 50

# Number of parameter sets in ensemble
nens = nc4.Dataset(
    model_filepath + model_filenames[0]
    .format(CO2levels[0], varfiletype[0])).variables[varlist[0]].shape[0]

model_data = np.zeros([len(CO2levels), len(varlist), nens, nyrs])
for c in range(len(CO2levels)):
    for v in range(len(varlist)):
        filepath = (model_filepath + model_filenames[c]
                    .format(CO2levels[c], varfiletype[v]))
        model_data[c, v, :, :] = annmeans.annual_mean_model(
            filepath, varlist[v], varfiletype[v], nyrs, varconv[v])
        filepath = None
```

### 2.2 Observations

This code section loads and preprocesses the data for the observations we will use to evaluate the performance of each parameter set in our FATES parameter ensemble.

#### Leaf area index

Leaf area index observations come from Detto et al. (2018) and were made using hemispherical photographs taken approximately monthly from January 2015 to August 2017 at 188 locations at our test site, Barro Colorado Island, Panama. Data was captured from Detto et al. (2018) Figure 7a using GraphClick software.  We calculate annual mean values from the monthly means reported by Detto et al. (2018). (Note that monthly data consists of spatial means across photograph locations, rather than temporal means.) In order to use all the data available, we calculate two time series of annual means $-$ one starting in from January and the second starting from September. We then choose the time series that allows for the best performance in our performance metric calculations in the next section.

```python
# Return obs_data_lai: 2-D array of annual mean leaf area index
# with shape (sample_number, nyrs). Sample_number coded as follows:
# 0 = sample months starting from January; and
# 1 = sample months starting from September.

filepath = 'data/LAI_Detto2018Obs.csv'

# Monthly spatial means
lai_mthts = np.asarray([col[2] for col in (pd.read_csv(filepath)).values])

# Specify start months for observations
startmonth_list = np.array([1, 9])

# Number of annual means per sample
nyears_lai = round(len(lai_mthts)/12 - 0.5)

obs_data_lai = np.zeros([len(startmonth_list), nyears_lai])
for x in range(len(startmonth_list)):
    obs_data_lai[x, :] = np.nanmean(np.reshape(
        lai_mthts[startmonth_list[x]-1 : 24+startmonth_list[x]-1],
        (nyears_lai, 12)), 1)
```

#### Above-ground carbon biomass

Above-ground carbon biomass estimates were calculated by Meakem et al. (2018) from 1995 census survey data for our test site, Barro Colorado Island. They estimate above-ground biomass using two different methods (the standard and Chave allometric formulations). We use values from these two methods to represent uncertainty in the observational estimate.

Alternatively, we can approximate above-ground carbon biomass from estimates of total biomass (rather than just carbon biomass) from census survey data reproted in Baraloto et al. (2013) and Feeley et al. (2007) for the following years:  1985, 1990, 1995, 2000, and 2005. This alternative method yields similar results and can be implemented by setting use_alt_agb_obs to 1 in the code below.

```python
# Return obs_data_agb: a vector of above-ground carbon biomass
# (KgC/m2) indexed by allometric formulation as follows:
# 0 = standard formulation; and
# 1 = Chave formulation.

# Specify whether to use (0) Meakem et al. (2018) estimate; or
# (1) Baraloto et al. (2013) and Feeley et al. (2007) estimates.
use_alt_agb_obs = 0;

filepath = 'data/BCI_biomass.csv'

if use_alt_agb_obs == 0:
    # Above-ground carbon biomass from Meakem et al. 2018 (MgC/ha)
    cbiomass_obs_Mgha = np.asarray(
        [col[2] for col in (pd.read_csv(filepath)).values])[-2:,]
    # Convert from MgC/ha to KgC/m2
    ha_to_m2 = 1/10000
    Mg_to_kg = 1000
    obs_data_agb = cbiomass_obs_Mgha * ha_to_m2 * Mg_to_kg

elif use_alt_agb_obs == 1:
    # Total aboveground biomass (Mg biomass/ha) from
    # Baraloto et al. (2013) and Feeley et al. (2007)
    agb_biomass_obs = np.asarray(
        [col[1] for col in (pd.read_csv(filepath)).values])[:-2,]
    # Estimate of carbon biomass from total biomass
    # using a conversion factor of 0.47 gC/g biomass
    # following Meakem et al. 2018
    obs_data_agb_v2 = agb_biomass_obs * 0.47
    obs_data_agb = obs_data_agb_v2
```

#### Basal area

We use estimates of the median basal area for our test site Barro Colorado Island, Panama, from census surveys conducted in 1999, 2001, 2006, and 2011 by Condit (1998), Condit et al. (2012, 2017), and Hubbell et al. (1999).

```python
# Return obs_data_ba: vector containing basal area (m^2/ha)
# indexed by census year in chronological order.

filepath = 'data/census_bmks_bci_171208.nc'

# Load basal area median values for the last 5 census dates
# Data structured as follows:
# [census number, tree diameter size class,
# distribution percentiles (0.05,0.5,0.95)]
basalarea_bysize = nc4.Dataset(
    filepath).variables['basal_area_by_size_census'][-5:, :, 1]

# Sum across tree size classes
obs_data_ba = np.nansum(basalarea_bysize, 1)
```

#### Gross primary productivity, latent heat flux, and sensible heat flux

Estimates of gross primary productivity, latent heat flux, and sensible heat flux were calculated from fluxtower eddy covariance measurements made from July 2012 to August 2017 at Barro Colorado Island by Koven et al. (_in review_). To use all available data in our analysis, we calculate two versions of the annual means time series, one beginning in July and the second beginning in September. We then choose the time series that allows for the best performance in our performance metric calculations in the next section.

```python
# Return obs_data_flux: 3-D array of annual mean
# fluxtower estimates with shape (sample_number, variable, years).
# Sample_number is indexed as follows: 0 = sample months
# starting from July; and 1 = sample months starting from September.
# Variable is indexed as follows: 0 = gross primary productivity;
# 1 = latent heat flux; and 2 = sensible heat flux.

# Load observations
GPP_data = np.load('data/fluxdata_GPP.npy')
LH_data  = np.load('data/fluxdata_LH.npy')
SH_data  = np.load('data/fluxdata_SH.npy')
fluxdata_mask= np.load('data/fluxdata_mask.npy')

# Apply mask to arrays
GPP_monthyear = np.ma.masked_array(GPP_data, mask=fluxdata_mask)
LH_monthyear = np.ma.masked_array(LH_data, mask=fluxdata_mask)
SH_monthyear = np.ma.masked_array(SH_data, mask=fluxdata_mask)

# Specify start months for observations
startmonth_list = np.array([7, 9])

# Number of years
nyrs_obsflux = len(annmeans.annual_mean_fluxobs(
    GPP_monthyear, startmonth_list[0]))

obs_data_flux = np.zeros([len(startmonth_list), 3, nyrs_obsflux])
for x in range(len(startmonth_list)):
    obs_data_flux[x, 0, :] = annmeans.annual_mean_fluxobs(
        GPP_monthyear, startmonth_list[x])
    obs_data_flux[x, 1, :] = annmeans.annual_mean_fluxobs(
        LH_monthyear, startmonth_list[x])
    obs_data_flux[x, 2, :] = annmeans.annual_mean_fluxobs(
        SH_monthyear, startmonth_list[x])
```

Lastly, we return one list that contains the observed annual mean time series for all variables (i.e., a list of the observed data arrays returned in the code above).

```python
# Return obs_data_list: a list containing all six observed
# variable arrays.
obs_data_list = [obs_data_lai, obs_data_agb, obs_data_ba,
                 obs_data_flux[:, 0, :],
                 obs_data_flux[:, 1, :],
                 obs_data_flux[:, 2, :]]
```

## Step 3: Quanitfy performance of each parameter set

In this section we evaluate the model performance for each parameter set and background carbon dioxide concentration against observations. As we would like to identify parameter sets that robsutly perform well regardless of performance metric we use two metrics to evaluate performance:  error rate and normalized root mean square error (NRMSE). We calculate both metrics for each variable. Then, we take a weighted average across all variables for each metric, parameter set, and background carbon dioxide concentration combination.

### Performance Metric #1: Error Rate

The error rate measures the percent of model annual means that fall within the observed range for each variable and ensemble member. To account for relatively small sample sizes and potential measurment error within the observations we extend the observed range by 10% in both directions.

```python
# Return error_rate_array: a 3-D array containing error rates
# indexed by (CO2levels, varlist, nens)

# Degradation level for the observational range
# (as fraction, not percent)
dg = 0.10

error_rate_array = np.zeros([len(CO2levels), len(varlist), nens])
for i in range(len(CO2levels)):
        for j in range(len(varlist)):
            error_rate_array[i, j, :] = metrics.error_rate(
                model_data[i, j, :, :], obs_data_list[j], dg)
```

### Performance Metric #2:  Normalized root mean square error (NRMSE)

The normalized root mean square error (NRMSE) measures the distance between the model output and the observed mean value, relative to the spread in the observations. The NRMSE is calculated as follows:

\begin{equation}\Large
NRMSE = \frac{ \sqrt{ \sum_{k=1}^n \frac{(x_{model,k} - \bar{X}_{obs})^2}{n}}} {x_{obs,max} - x_{obs,min}}
\end{equation}

where $NRMSE$ is the normalized root mean square error for a single variable (e.g. leaf area index) and parameter set, $n$ is the number of years of model output, $x_{model,k}$ is the model annual mean for year $k$, $\bar{X}_{obs}$ is the overall annual mean of the observed values, and $x_{obs,max}$ and $x_{obs,min}$ are the maximum and minimum observed annual mean values, respectively. When mulitple annual mean time series were sampled for an observed variable (e.g. observations for leaf area index spanned a partial year), we calculate the difference between the observed mean and model output using the time series that minimizes this difference.

```python
# Return nrsmse_array: a 3-D array containing the NRMSE
# indexed by [CO2level, varlist, nens]

nrmse_array = np.zeros([len(CO2levels), len(varlist), nens])
for i in range(len(CO2levels)):
        for j in range(len(varlist)):
            nrmse_array[i, j, :] = metrics.nrmse(
                model_data[i, j, :, :], obs_data_list[j])
```

### Weighted average permformance metrics across variables

We calculate weighted average performance metrics across variables for both error rate and NRMSE. We consider three different weighting approaches to ensure that our selection of high-performing parameter sets is robust to weighting method. The weighting approaches we use are:

1. Even:  All variables are evenly weighted.

2. Structure:  This weighting favors structural ecosystem properties (leaf area index, above-ground biomass, and basal area). It reflects the likelihood that structural variables at our test site include less measurment uncertainty than flux variables.

3. Correlation: This weighting scheme is informed by correlations between individual variable performance metrics. The ability of a parameter set to mach observations of flux variables (gross primary productivity, sensible heat, and latent heat) was correlated with ability to match observations of leaf area index, as well as other flux variables. As leaf area index observations likely include smaller measurement uncertainty, we choose to give a greater weighting to leaf area index at the expense of flux estimates. We also reduced the weightings of basal area and above-ground biomass performance to account for their correlation with one another.

#### Weighted average error rate

```python
# Even weighting across all variables
er_wavg_even = np.average(error_rate_array, axis=1)

# Weighted average favoring structural properties
ws = 0.3
wf = (1-3*ws)/3
w_strct = [ws, ws, ws, wf, wf, wf]
er_wavg_strct = np.average(error_rate_array, axis=1, weights=w_strct)

# Weighting considering correlations between performance metrics
w1 = 0.4
w2 = 0.25
w3 = 0.1/3
w_corr = [w1, w2, w2, w3, w3, w3]
er_wavg_corr = np.average(error_rate_array, axis=1, weights=w_corr)
```

#### Weighted average NRMSE

We quantify the distance of model output from the mean observations in multivariate space by calculating the weighted Euclidean distance as follows:

\begin{equation}
NRMSE_{avg} = \sqrt{ \sum_{i=1}^m \omega_{i} \cdot NRMSE_{i}^2}
\end{equation}

where $NRMSE_{avg}$ is the weighted average NRMSE across variables, $m$ is the number of variables, and $NRMSE_{i}$ and $\omega_{i}$ are the individual variable NRMSE and weighting, respectively.

```python
# Even weighting across all variables
nvars = nrmse_array.shape[1]
w = 1/nvars
w_even = np.matlib.repmat(w,nvars,1)
nrmse_wavg_even = metrics.avg_nrmse(nrmse_array, w_even)

# Weighted average favoring structural properties
nrmse_wavg_strct = metrics.avg_nrmse(nrmse_array, w_strct)

# Weighting considering correlations between performance metrics
nrmse_wavg_corr = metrics.avg_nrmse(nrmse_array, w_corr)
```

Lastly, we return one array that contains the weighted average values for all combinations of weighting approach and performance metric.

```python
all_avg_array = np.transpose(np.stack([
    er_wavg_even, er_wavg_strct, er_wavg_corr,
    nrmse_wavg_even, nrmse_wavg_strct, nrmse_wavg_corr]),
    (1,0,2))
```

## Step 4: Rank parameter sets by performance
Here we assign an overall rank to each parameter set based on its performance across two performance metrics (error rate and NRMSE), three weighted averaging approaches (even, structure, and correlated), and two background carbon dioxide concentrations (367 ppm and 400 ppm). High-performing parameter sets (i.e., parameter sets that robustly performed well at our test site) are indicated by a low numbered rank.

```python
# Assign each simulation a rank for
# each weighted average performance metric
rank_array = stats.mstats.rankdata(all_avg_array, axis=2)

# Sum ranks across performance metrics and
# background carbon dioxide concentrations
sum_rank_array = np.nansum(np.nansum(rank_array, axis=0), axis=0)

# Sort parameter set indexes by their summed rank
# (best to worst performance)
sum_rank_index = np.argsort(sum_rank_array)

#Print for highest-performing parameter set numbers
highperform_num = np.transpose(sum_rank_index)[:10,] + 1
print("Highest-performing parameter set numbers: ", highperform_num[0:3])
```

    Highest-performing parameter set numbers:  [ 86 260 151]

__Result:__ Parameter sets 86, 260, and 151 resulted in the highest model performance (in descending order) at our test site.

## Step 5: Plot performance metrics for each parameter set

In this section we visualize the model performance for each parameter set to gain insight into how the highest-performing parameter sets performed in individual variables and in comparison to all other parameter sets.

```python
# Return error_heatdata and nrmse_heatdata: two 3-D arrays
# containing error rates and NRMSE values, respectively,
# indexed by (CO2level, variables, nens). Variables indexed
# as follows:
# 0-5, variables in order of varlist; and
# 6-8 = weighted averages across variables using even,
#    structure, and correlation weights, respectively

error_heatdata = np.concatenate(
    [error_rate_array, all_avg_array[:, :3, :]], axis=1)
nrmse_heatdata = np.concatenate(
    [nrmse_array, all_avg_array[:, 3:, :]], axis=1)
```

### Plot for simulations run at 367 ppm carbon dioxide

```python
# Set carbon dioxide case to 367 ppm
casenum = 0

# Plot Error Rate and NRMSE
fig1 = plt.figure(figsize=(12, 12))

plotnum = 1
plotmetrics.heatmap_subplot(error_heatdata, casenum, 0, 100,
                            plotnum, 'A.  Error Rate (%)', highperform_num)

plotnum = plotnum + 1
plotmetrics.heatmap_subplot(nrmse_heatdata, casenum, 0, 10,
                            plotnum, 'B.  NRMSE', highperform_num)

plt.tight_layout()
```

__Figure 1.__ FATES model performance as measured by (A) error rate and (B) normalized root mean square error (NRMSE) for simulations run with 367 ppm atmospheric carbon dioxide. Performance metrics are shown for leaf area index (LAI), above-ground biomass (AGB), basal area (BA), gross primary productivity (GPP), latent heat flux (LH), sensible heat flux (SH), and weighted averages across variables using three weighting approaches:  even (Av$_E$), favoring structural variables (Av$_S$), and considering correlations between individual performance metrics (Av$_C$). Top panels highlight the performance of the 10 highest-perfoming parameter sets. Bottom panels show the performance of all 287 parameter sets we tested.

### Plot for simulations run at 400 ppm carbon dioxide

```python
# Set carbon dioxide case to 400 ppm
casenum = 1

# Plot Error Rate and NRMSE
fig2 = plt.figure(figsize=(12, 12))

plotnum = 1
plotmetrics.heatmap_subplot(error_heatdata, casenum, 0, 100,
                            plotnum, 'A.  Error Rate (%)', highperform_num)

plotnum = plotnum + 1
plotmetrics.heatmap_subplot(nrmse_heatdata, casenum, 0, 10,
                            plotnum, 'B.  NRMSE', highperform_num)

plt.tight_layout()
```

__Figure 2.__ FATES model performance as measured by (A) error rate and (B) normalized root mean square error (NRMSE) for simulations run with 400 ppm atmospheric carbon dioxide. Performance metrics are shown for leaf area index (LAI), above-ground biomass (AGB), basal area (BA), gross primary productivity (GPP), latent heat flux (LH), sensible heat flux (SH), and weighted averages across variables using three weighting approaches:  even (Av$_E$), favoring structural variables (Av$_S$), and considering correlations between individual performance metrics (Av$_C$). Top panels highlight the performance of the 10 highest-perfoming parameter sets. Bottom panels show the performance of all 287 parameter sets we tested.

## Results
This analysis identifies three high-performing parameter sets for use in future FATES simulations. We recommend the highest-performing parameter set (parameter set number 86; Figures 1 and 2) as the primary parameter set for future experiments. The next two highest-performing parameter sets (parameter sets numbers 151 and 260; Figures 1 and 2) are recommended for testing the sensitivity of experiments to model parameterization. Parameter set number 260 is the second highest-performing parameter set and is similar in parameter values and resulting ecosystem properties to parameter set number 86. Parameter set number 151, on the other hand, differs more substantially in its parameter values and results in a different combination of ecosystem properties. In particular, parameter set number 151 performs well in above-ground biomass and basal area measures across carbon dioxide levels, but results in low leaf area index and high gross primary productivity compared to observations. See Kovenock (2019) for further details.

These three high-performing parameter sets are made publicly available through the University of Washington ResearchWorks digital repository at http://hdl.handle.net/1773/43779.

## Further Information
__Details of the parameter ensemble and analysis herein:__

Kovenock, M. (2019). Ecosystem and large-scale climate impacts of plant leaf dynamics (Doctoral dissertation). Chapter 4: "Within-canopy gradient of specific leaf area improves simulation of tropical forest structure and functioning in a demographic vegetation model." http://hdl.handle.net/1773/44061

__Details of the Functionally Assembled Ecosystem Simulator (FATES):__

GitHub code repository:

https://github.com/NGEET/fates

Fisher, R. A., Koven, C. D., Anderegg, W. R., Christoffersen, B. O., Dietze, M. C., Farrior, C. E., et al. (2018). Vegetation demographics in Earth System Models: A review of progress and priorities. Global Change Biology, 24(1), 35–54. https://doi.org/10.1111/gcb.13910

Fisher, R. A., Muszala, S., Verteinstein, M., Lawrence, P., Xu, C., McDowell, N. G., et al. (2015). Taking off the training wheels: the properties of a dynamic vegetation model without climate envelopes. _Geoscientific Model Development, 8_(4), 3293–3357. https://doi.org/10.5194/gmdd-8-3293-2015

## References

Baraloto, C., Molto, Q., Rabaud, S., Hérault, B., Valencia, R., Blanc, L., et al. (2013). Rapid simultaneous estimation of aboveground biomass and tree diversity across Neotropical forests: a comparison of field inventory methods. _Biotropica, 45_(3), 288–298. https://doi.org/10.1111/btp.12006

Condit, R. (1998). Tropical forest census plots. Berlin, Germany, and Georgetown, Texas: Springer-Verlag and R. G. Landes Company.

Condit, R. S., Aguilar, S., Perez, R., Lao, S., Hubbell, S. P., & Foster, R. B. (2017). Barro Colorado 50-ha Plot Taxonomy as of 2017. https://doi.org/10.25570/stri/10088/32990

Condit, R., Lao, S., Pérez, R., Dolins, S. B., Foster, R., & Hubbell, S. (2012). Barro Colorado forest census plot data (version 2012). Center for Tropical Forest Science Databases. https://doi.org/10.5479/data.bci.20130603

Detto, M., Wright, S. J., Calderón, O., & Muller-Landau, H. C. (2018). Resource acquisition and reproductive strategies of tropical forest in response to the El Niño$-$Southern Oscillation. _Nature Communications, 9_(1), 913. https://doi.org/10.1038/s41467-018-03306-9

Faybishenko, B., Paton, S., Powell, T., Knox, R., Pastorello, G., Varadharajan, C., et al. (2018). QA/QC-ed BCI meteorological drivers. United States: Next-Generation Ecosystem Experiments Tropics; STRI; LBNL. https://doi.org/doi:10.15486/ngt/1423307

Feeley, K. J., Davies, S. J., Ashton, P. S., Bunyavejchewin, S., Supardi, M. N., Kassim, A. R., et al. (2007). The role of gap phase processes in the biomass dynamics of tropical forests. _Proceedings of the Royal Society B: Biological Sciences, 274_(1627), 2857–2864. https://doi.org/10.1098/rspb.2007.0954

Fisher, R. A., Koven, C. D., Anderegg, W. R., Christoffersen, B. O., Dietze, M. C., Farrior, C. E., et al. (2018). Vegetation demographics in Earth System Models: A review of progress and priorities. _Global Change Biology, 24_(1), 35–54. https://doi.org/10.1111/gcb.13910

Fisher, R. A., Muszala, S., Verteinstein, M., Lawrence, P., Xu, C., McDowell, N. G., et al. (2015). Taking off the training wheels: the properties of a dynamic vegetation model without climate envelopes. _Geoscientific Model Development, 8_(4), 3293–3357. https://doi.org/10.5194/gmdd-8-3293-2015

Hubbell, S. P., Foster, R. B., O'Brien, S. T., Harms, K. E., Condit, R., Wechsler, B., et al. (1999). Light-gap disturbances, recruitment limitation, and tree diversity in a neotropical forest. _Science, 283_(5401), 554–557. https://doi.org/10.1126/science.283.5401.554

Koven, C. D., et al. (_in review_). Benchmarking and parameter sensitivity of physiological and vegetation dynamics using the Functionally Assembled Terrestrial Ecosystem Simulator (FATES) at Barro Colorado Island, Panama. _Biogeosciences Discussions_. https://doi.org/10.5194/bg-2019-409

Kovenock, M. (2019).  Ecosystem and large-scale climate impacts of plant leaf dynamics (Doctoral dissertation). http://hdl.handle.net/1773/44061

Meakem, V., Tepley, A. J., Gonzalez-Akre, E. B., Herrmann, V., Muller-Landau, H. C., Wright, S. J., et al. (2018). Role of tree size in moist tropical forest carbon cycling and water deficit responses. _New Phytologist, 219_, 947–958. https://doi.org/doi:10.1111/nph.14633


In order to successfully complete this assignment you must do the required reading, watch the provided videos and complete all instructions.  The embedded Google form must be entirely filled out and submitted on or before **11:59pm on Tuesday January 21**.  Students must come to class the next day prepared to discuss the material covered in this assignment.

# Pre-Class Assignment: Code Optimization

## a.k.a. making your research go faster!!!

# Goals for today's pre-class assignment

1. [Matrix Multiply Example](#Matrix_Multiply_Example)
1. [How to build a Supercomputer](#How_to_build_a_Supercomputer)
1. [Install required software](#Install_required_software)
1. [Logging onto the HPCC](#Logging_onto_the_HPCC)
1. [Using git on the HPCC](#Using_git_on_the_HPCC)
1. [Assignment Wrap-up](#Assignment_wrap-up)

---

# 1. Matrix Multiply Example

 The following is a simple implementation of a matrix multiply. Review the code try to understand what it is doing.

```python
%matplotlib inline
import matplotlib.pylab as plt
import numpy as np
import sympy as sp
import random
import time
sp.init_printing(use_unicode=True)
```

```python
#simple matrix multiply (no numpy)
def multiply(m1,m2):
    m = len(m1)
    d = len(m2)
    n = len(m2[0])
    if len(m1[0]) != d:
        print("ERROR - inner dimentions not equal")
    result = [[0 for i in range(m)] for j in range(n)]
    for i in range(0,m):
        for j in range(0,n):
            for k in range(0,d):
                result[i][j] = result[i][j] + m1[i][k] * m2[k][j]
    return result
```

```python
# Random generated 2d lists of lists that can be multiplied
m = 4
d = 10
n = 4

A = [[random.random() for i in range(d)] for j in range(m)]
B = [[random.random() for i in range(n)] for j in range(d)]
```

```python
#Compute matrix multiply using your function
start = time.time()

simple_answer = multiply(A, B)
simple_time = time.time()-start

print('simple_answer =',simple_time,'seconds')
```

    simple_answer = 0.00016689300537109375 seconds

Lets compare this to the numpy result:

```python
#Compare to numpy result
start = time.time()

np_answer = np.matrix(A)*np.matrix(B)
np_time = time.time()-start

print('np_answer =',np_time,'seconds')

```

    np_answer = 0.023662090301513672 seconds

For this example, numpy result are most likely slower than the simple result.  Think about why this might be.  We will discuss this later.

✅ **DO THIS:**  See if you can write a loop to do a scaling study for the above code.  Loop over the value of $n$ such that $n$ is 4, 16, 32, 64, 128 and 256.  For each iteration generate two random matrices (as above) with $m = d = n$. Then time the matrix multiply for the provided function and again for the numpy function. Graph the results as size of $n$ vs time.

```python
# Put your code here
```

✅ **DO THIS:**  Explore the Internet for ways to speed up Python (There are a lot of them).  Save some of your search results in the cell below and come to class prepaired to discuss what you found.

Put your search results here.

----

# 2. How to build a Supercomputer
In addition to making your code go faster we can use a faster computer...

Your instructor submitted a request to get you an account on the MSU High Performance Computer (HPC).  The following videos try to explain the system and how to gain access.  We will be using these accounts in class. See if you can log in.

```python
from IPython.display import YouTubeVideo
YouTubeVideo("6oyoS-ddUH0",width=640,height=360)
```

----

# 3. Install required software

In order to access the HPC you need an SSH client and optionally an X11 client (For graphics).  Although the instructors has a few ways to work around the problem if you have trouble (For example, we can also use the Terminal option inside [Jupyterhub](http://jupyterhub.egr.msu.edu)). That being said, your personal laptops will work best if you have the following installed:

- SSH and SCP client - pre-installed on Mac/Linux, Lots of options for Windows. Ex: [MobaXterm](https://mobaxterm.mobatek.net/) -
- X11 client - [XQuartz](https://www.xquartz.org/) on Mac, pre-installed on Linux, Lots of options for Windows. Ex [MobaXterm](https://mobaxterm.mobatek.net/)
/
Note, there are a lot of options here. It is the responsibility of the student to make sure their software is running properly.

----

# 4. Logging onto the HPCC

```python
from IPython.display import YouTubeVideo
YouTubeVideo("ODcuHJDIVr4",width=640,height=360)
```

✅ **QUESTION:** Are you able to log on to the HPCC and if not, where did you get stuck?

Put your answer to the above question here

----

# 5. Using git on the HPCC

Log into the HPCC and clone the course git repository:

```
ssh username@gateway.hpcc.msu.edu
ssh dev-intel18 # or another developer node
# change to a directory of your choice (Using cd command) or just put the class files in your home directory
git clone https://gitlab.msu.edu/colbrydi/cmse802-f19

```

----

# 6. Assignment wrap-up

Please fill out the form that appears when you run the code below.  **You must completely fill this out in order to receive credit for the assignment!**

[Direct Link to Google Form](https://cmse.msu.edu/cmse802-pc-survey)

If you have trouble with the embedded form, please make sure you log on with your MSU google account at [googleapps.msu.edu](https://googleapps.msu.edu) and then click on the direct link above.

✅ **Assignment-Specific QUESTION:** Are you able to log on to the HPCC and if not, where did you get stuck?

Put your answer to the above question here

✅ **QUESTION:**  Summarize what you did in this assignment.

Put your answer to the above question here

✅ **QUESTION:**  What questions do you have, if any, about any of the topics discussed in this assignment after working through the jupyter notebook?

Put your answer to the above question here

✅ **QUESTION:**  How well do you feel this assignment helped you to achieve a better understanding of the above mentioned topic(s)?

Put your answer to the above question here

✅ **QUESTION:** What was the **most** challenging part of this assignment for you?

Put your answer to the above question here

✅ **QUESTION:** What was the **least** challenging part of this assignment for you?

Put your answer to the above question here

✅ **QUESTION:**  What kind of additional questions or support, if any, do you feel you need to have a better understanding of the content in this assignment?

Put your answer to the above question here

✅ **QUESTION:**  Do you have any further questions or comments about this material, or anything else that's going on in class?

Put your answer to the above question here

✅ **QUESTION:** Approximately how long did this pre-class assignment take?

Put your answer to the above question here

```python
from IPython.display import HTML
HTML(
"""

"""
)
```

---------
### Congratulations, we're done!

To get credit for this assignment you must fill out and submit the above Google From on or before the assignment due date.

### Course Resources:

- [Syllabus](https://docs.google.com/document/d/e/2PACX-1vTW4OzeUNhsuG_zvh06MT4r1tguxLFXGFCiMVN49XJJRYfekb7E6LyfGLP5tyLcHqcUNJjH2Vk-Isd8/pub)
- [Preliminary Schedule](https://docs.google.com/spreadsheets/d/e/2PACX-1vRsQcyH1nlbSD4x7zvHWAbAcLrGWRo_RqeFyt2loQPgt3MxirrI5ADVFW9IoeLGSBSu_Uo6e8BE4IQc/pubhtml?gid=2142090757&single=true)
- [Course D2L Page](https://d2l.msu.edu/d2l/home/912152)
- [Course Git Repository](https://gitlab.msu.edu/colbrydi/cmse802-s20)

© Copyright 2020,  Michigan State University Board of Trustees