# NLBSE'22 Tool Competition

### Introduction

The first edition of the NLBSE’22 tool competition is on automatic **issue report classification**, an important task in issue management and prioritization.

For the competition, we provide a dataset encompassing more than 800k labeled issue reports (as bugs, enhancements, and questions) extracted from real open-source projects. You are invited to leverage this dataset for evaluating your classification approaches and compare the achieved results against a proposed baseline approach (based on FastText).

### Participation

If you want to participate, you must:
* Train and tune a multi-label classifier using the provided [training set](https://tickettagger.blob.core.windows.net/datasets/github-labels-top3-803k-train.tar.gz).
* Evaluate your classifier on the provided [test set](https://tickettagger.blob.core.windows.net/datasets/github-labels-top3-803k-test.tar.gz)
* Write a paper (4 pages max.) describing:
    * The architecture and details of the classifier
    * The procedure used to pre-process the data
    * The procedure used to tune the classifier on the training set
    * The results of your classifier on the test set
    * Additional info.: provide a link to your code/tool with proper documentation on how to run it
* **Submit the paper** by the deadline (see below). **Email the paper to the tool competition organizers**: Oscar Chaparro (oscarch@wm.edu) and Rafael Kallis (rk@rafaelkallis.com)

All submissions must conform to the [ICSE’22 formatting and submission instructions](https://conf.researchr.org/track/icse-2022/icse-2022-papers#how-to-submit).

Papers do not need to be double-blinded.

### Important dates

* Paper/tool submission: **February 21, 2022**
* Acceptance and competition results notification: **March 4, 2022**
* Camera-ready paper submission: **March 18, 2022**

All dates are anywhere on earth (AoE).

### Submission acceptance and competition

Submissions will be evaluated and accepted based on **correctness** and **reproducibility**, defined by the following criteria:
* Clarity and detail of the paper content
* Availability of the code/tool, including the training/tuning/evaluation pipeline, released as open-source
* Correct training/tuning/evaluation of your code/tool on the provided data
* Report the metrics and results we outline below
* Clarity of the code documentation

The accepted submissions will be published at the workshop proceedings.

The submissions will be ranked based on the $F_1$ score (defined below) achieved by the proposed classifiers on the test set, as indicated in the papers.

The submission with the highest $F_1$ score will be the winner of the competition.

### Referencing

Since you will be using our dataset (and possibly this Colab notebook) as well as the original work behind the dataset, please cite the following references in your paper:

```
@inproceedings{nlbse2022,
  author={Kallis, Rafael and Chaparro, Oscar and Di Sorbo, Andrea and Panichella, Sebastiano},
  title={NLBSE'22 Tool Competition},
  booktitle={Proceedings of The 1st International Workshop on Natural Language-based Software Engineering (NLBSE'22)},
  year={2022}
}
```

```
@article{ticket-tagger-scp,
  author={Rafael Kallis and Andrea {Di Sorbo} and Gerardo Canfora and Sebastiano Panichella}
  title={Predicting issue types on GitHub},
  journal={Science of Computer Programming},
  volume={205},
  pages={102598},
  year={2021},
  issn={0167-6423},
  doi={https://doi.org/10.1016/j.scico.2020.102598},
  url={https://www.sciencedirect.com/science/article/pii/S0167642320302069}
}
  ```

```
@INPROCEEDINGS{ticket-tagger,
  author={Kallis, Rafael and Di Sorbo, Andrea and Canfora, Gerardo and Panichella, Sebastiano},
  booktitle={2019 IEEE International Conference on Software Maintenance and Evolution (ICSME)},
  title={Ticket Tagger: Machine Learning Driven Issue Classification},
  year={2019},
  volume={},
  number={},
  pages={406-409},
  doi={10.1109/ICSME.2019.00070}
}
  ```

### Training

You are provided a [training set](https://tickettagger.blob.core.windows.net/datasets/github-labels-top3-803k-train.tar.gz) encompassing more than 700,000 labeled issue reports extracted from real open source projects.

Participants are free to select and transform variables from the training set as they please but **no new sources can be be added**.
In other words, any inputs or features used to create the classifier, must be derived from the provided training set.
Participants may preprocess, sample, apply over/under-sampling, select a subset of the attributes, perform feature-engineering, split the training set into a model-finetuning validation set, etc. Please contact us if you have any question about this.

Each issue report contains the following metadata:
- Label
- Issue title
- Issue body
- Issue URL
- Repository URL
- Creation timestamp
- Author association

Each issue is labeled with one class that indicates the issue type, namely, bug, enhancement, and question.

The distribution of (722,899) issues in the training set is:
* bug:            361,239 (50%)
* enhancement:    299,287 (41.4%)
* question:        62,373 (8.6%)

### Evaluation

> Note: for correct latex rendering, open this notebook in Collab.

Submissions are evaluated based on their class-detection performance over the provided [test set](https://tickettagger.blob.core.windows.net/datasets/github-labels-top3-803k-test.tar.gz).

The distribution of (80,518) issues in the test set:
* bug:	40,152	(49.9%)
* enhancement:	33,290	(41.3%)
* question:	7,076	(8.8%)

The evaluation must be performed on the entire test set only. **Important:** you may apply any preprocessing or feature engineering on this dataset except sampling, rebalancing, undersampling or oversampling techniques.

Classification performance is measured using the $F_1$ score over all the three classes.

A submission (i.e., paper) in the tool competition must provide:
- Precision $P_c$ for each class $c$
- Recall $R_c$, for each class $c$
- $F_{1,c}$ score as the harmonic mean between $P_c$ and $R_c$, for each class $c$
- Precision $P$, micro-averaged $P_c$
- Recall $R$, micro-averaged $R_c$
- $F_1$ score as the harmonic mean between $P$ and $R$

These metrics are defined as follows:
\begin{align}
P_c &= \frac{ TP_c }{ TP_c + FP_c } &
R_c &= \frac{ TP_c }{ TP_c + FN_c } &
F_{1,c} &= 2 \cdot \frac{ P_c \cdot R_c }{ P_c + R_c }\\
P &= \frac{ \sum_c TP_c }{ \sum_c \left( TP_c + FP_c \right) } &
R &= \frac{ \sum_c TP_c }{ \sum_c \left( TP_c + FN_c \right) } &
F_1 &= 2 \cdot \frac{ P \cdot R }{ P + R }
\end{align}

where $TP_c$, $FP_c$ and $FN_c$ represent true positives, false positives and false negatives over a class $c$, respectively.
Micro-averaging was chosen as the cross-class aggregation method due to the class imbalance present in the data.

Please note that whilst all of the above measures must be provided for acceptance, the submissions will **only** be ranked by their $F_1$ score.

### Colab Notebook

Colab notebooks provide a convenient format for publishing work and make it easier for readers to reproduce results. Colab provides an interactive Python runtime framework that can be accessed from a web browser.
Colab requires no configuration, provides free access to GPUs and allows for easy sharing between collaborators.

Participants can use a cloud-hosted infrastructure (recommended), connect to a local runtime or not use Colab at all. We do not impose any constraints on the technologies used. You can read more about Colab [here](https://colab.research.google.com/notebooks/intro.ipynb). An example of a GPU-accelerated Colab notebook can be found [here](https://colab.research.google.com/notebooks/gpu.ipynb).

## Submission Template

Participants are encouraged, but not required, to use the following code as a template for their submission.
The example below trains a FastText model, which is provided as a baseline.

```python
import os
import time
import numpy as np
import pandas as pd
import seaborn as sns
import gensim
import sklearn.metrics
from tqdm.auto import tqdm
```

```python
# persistent file storage
# https://colab.research.google.com/notebooks/io.ipynb
from google.colab import drive
drive.mount('/content/drive')
```

    Mounted at /content/drive

#### Training

```python
# download the training set if it does not exist
if not os.path.isfile("github-labels-top3-803k-train.csv"):
  !curl "https://tickettagger.blob.core.windows.net/datasets/github-labels-top3-803k-train.tar.gz" | tar -xz

trainset = pd.read_csv("github-labels-top3-803k-train.csv")
```

      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
      0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.security.selinux'
    100  236M  100  236M    0     0  7974k      0  0:00:30  0:00:30 --:--:-- 5912k

```python
trainset.head(5)
```

      Unnamed: 0
      issue_url
      issue_label
      issue_created_at
      issue_author_association
      repository_url
      issue_title
      issue_body

      0
      0
      https://api.github.com/repos/eamodio/vscode-gi...
      bug
      2021-01-02T18:07:30Z
      NONE
      https://api.github.com/repos/eamodio/vscode-gi...
      Welcome screen on every editor window is very ...
      I just discovered Gitlens and find the functio...

      1
      1
      https://api.github.com/repos/binwiederhier/pco...
      bug
      2020-12-31T18:19:31Z
      OWNER
      https://api.github.com/repos/binwiederhier/pcopy
      "pcopy invite" and "pcopy paste abc:" does not...
      NaN

      2
      2
      https://api.github.com/repos/binwiederhier/pco...
      bug
      2021-01-03T04:33:36Z
      OWNER
      https://api.github.com/repos/binwiederhier/pcopy
      UI: Modal overlay is half transparent, shouldn...
      NaN

      3
      3
      https://api.github.com/repos/Sothatsit/RoyalUr...
      enhancement
      2020-12-25T00:46:00Z
      OWNER
      https://api.github.com/repos/Sothatsit/RoyalUr...
      Make the loading screen scale with browser win...
      Currently the loading wheel is a fixed size in...

      4
      4
      https://api.github.com/repos/Malivil/TTT-Custo...
      bug
      2021-01-02T21:36:57Z
      OWNER
      https://api.github.com/repos/Malivil/TTT-Custo...
      Spectator - Investigate a way to strip weapons...
      To bring magneto stick floating

```python
trainset.groupby("issue_label").size()
```

    issue_label
    bug            361103
    enhancement    299374
    question        62422
    dtype: int64

```python
# preprocessing can be customized by participants
def preprocess(row):
  # concatenate title and body, then remove whitespaces
  doc = ""
  doc += str(row.issue_title)
  doc += " "
  doc += str(row.issue_body)
  # https://radimrehurek.com/gensim/parsing/preprocessing.html
  doc = gensim.parsing.preprocessing.strip_multiple_whitespaces(doc)
  return doc
```

```python
# transform dataset into fasttext format
# https://fasttext.cc/docs/en/supervised-tutorial.html#getting-and-preparing-the-data

# rng used to split training set into fasttext hyperparameter finetuning validation set (2 of 9 parts)
random = np.random.default_rng(0)

with open("train.txt", "w") as train_f, open("valid.txt", "w") as valid_f:
  for row in tqdm(trainset.itertuples(), desc="Transform to fastText format", total=len(trainset)):
    doc = f"__label__{row.issue_label} {preprocess(row)}\n"
    is_train = random.uniform() < 7/9
    f = train_f if is_train else valid_f
    f.write(doc)

!wc -l "valid.txt"
!wc -l "train.txt"

!head -n 10 "train.txt"
```

    Transform to fastText format:   0%|          | 0/722899 [00:00<?, ?it/s]

    160535 valid.txt
    562364 train.txt
    __label__bug Welcome screen on every editor window is very tedious I just discovered Gitlens and find the functionality useful, thank you to all who contribute. I have about a dozen editor windows open, and the install process added a Gitlens welcome tab to each and every one of them. Combined with the snowflake effect, all of the sudden VScode was consuming 300-400% cpu and my fan was raging, as soon as I hunted them all down everything was back to fine. The welcome note content is great (although putting it on _all_ the windows is a bit much, don't know how much control you have on that). But overall it was a bit of a sour first-use experience, just wanted to provide that feedback.
    __label__bug "pcopy invite" and "pcopy paste abc:" does not check if clipboard exists nan
    __label__bug UI: Modal overlay is half transparent, shouldn't be nan
    __label__enhancement Make the loading screen scale with browser window size Currently the loading wheel is a fixed size in pixels, but it would be better to specify it in terms of percentage of the browser size.
    __label__bug Correct navigation bar for some resolution
    __label__enhancement Return asset growth Would have to cache: - Stock Ticker Request - Last Price at request - Channel that the ticker was requested from (to dedupe issues)
    __label__enhancement Add filter for logs Add the ability to filter logs like on other pages.
    __label__enhancement Help bubbles nan
    __label__bug Nvidia Boot Error This doesn't really affect me, more of a doubt if this is normal to happen. I get this screen 100% when turning my customac on:  Pastebin: https://pastebin.com/0jtUHJgm Cheers!
    __label__bug catAxisLabelPos and valAxisLabelPos options are not working Hi team , I set the two options of `cataxislabelpos` and `valaxislabelpos`, but they didn't seem to work.Below is my code.  ``` let dataChartAreaLine = [ { name: "Actual Sales", labels: ["Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"], values: [1500, 4600, 5156, 3167, 8510, 8009, 6006, 7855, 12102, 12789, 10123, 15121], }, { name: "Projected Sales", labels: ["Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"], values: [1000, 2600, 3456, 4567, 5010, 6009, 7006, 8855, 9102, 10789, 11123, 12121], }, ]; slide.addChart("line", dataChartAreaLine, { x: 1, y: 1, w: 8, h: 4, catAxisLabelPos: "high", valAxisLabelPos: "high" }); ```

```python
!pip install fasttext
import fasttext

# https://fasttext.cc/docs/en/python-module.html#train_supervised-parameters
model = fasttext.train_supervised("train.txt",
                                  minCount=50,
                                  autotuneValidationFile="valid.txt",
                                  autotuneDuration=5*60)
model.quantize()
model.save_model(f"github-labels-{int(time.time())}.ftz")
model.save_model(f"drive/MyDrive/github-labels-{int(time.time())}.ftz")
model
```

    Collecting fasttext
      Downloading fasttext-0.9.2.tar.gz (68 kB)
         |████████████████████████████████| 68 kB 5.6 MB/s  eta 0:00:01
    Collecting pybind11>=2.2
      Using cached pybind11-2.8.1-py2.py3-none-any.whl (208 kB)
    Requirement already satisfied: setuptools>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from fasttext) (57.4.0)
    Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from fasttext) (1.19.5)
    Building wheels for collected packages: fasttext
      Building wheel for fasttext (setup.py) ... done
      Created wheel for fasttext: filename=fasttext-0.9.2-cp37-cp37m-linux_x86_64.whl size=3125036 sha256=2cc2f0c1c6329a77a2f14306152305ca06ecf1a961014e6d0b760ba85b657746
      Stored in directory: /root/.cache/pip/wheels/4e/ca/bf/b020d2be95f7641801a6597a29c8f4f19e38f9c02a345bab9b
    Successfully built fasttext
    Installing collected packages: pybind11, fasttext
    Successfully installed fasttext-0.9.2 pybind11-2.8.1

    <fasttext.FastText._FastText at 0x7f072b9246d0>

#### Evaluation

```python
if not os.path.isfile("github-labels-top3-803k-test.csv"):
  !curl "https://tickettagger.blob.core.windows.net/datasets/github-labels-top3-803k-test.tar.gz" | tar -xz

testset = pd.read_csv("github-labels-top3-803k-test.csv")
testset.groupby("issue_label").size()
#testset

```

      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
      0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.security.selinux'
    100 27.2M  100 27.2M    0     0  9192k      0  0:00:03  0:00:03 --:--:-- 9192k

    issue_label
    bug            40288
    enhancement    33203
    question        7027
    dtype: int64

```python
# confusion matrix
y_true = []
y_pred = []

for row in tqdm(testset.itertuples(), desc="Benchmarking Inference Performance", total=len(testset)):
  pred = model.predict(preprocess(row))[0][0][9:]
  y_true.append(row.issue_label)
  y_pred.append(pred)

for label in ["bug", "enhancement", "question"]:
  P_c = sklearn.metrics.precision_score(y_true, y_pred, average=None, labels=[label])[0]
  R_c = sklearn.metrics.recall_score(y_true, y_pred, average=None, labels=[label])[0]
  F1_c = sklearn.metrics.f1_score(y_true, y_pred, average=None, labels=[label])[0]
  print(f"=*= {label} =*=")
  print(f"precision:\t{P_c:.4f}")
  print(f"recall:\t\t{R_c:.4f}")
  print(f"F1 score:\t{F1_c:.4f}")
  print()

P = sklearn.metrics.precision_score(y_true, y_pred, average='micro')
R = sklearn.metrics.recall_score(y_true, y_pred, average='micro')
F1 = sklearn.metrics.f1_score(y_true, y_pred, average='micro')

print("=*= global =*=")
print(f"precision:\t{P:.4f}")
print(f"recall:\t\t{R:.4f}")
print(f"F1 score:\t{F1:.4f}")
```

    Benchmarking Inference Performance:   0%|          | 0/80518 [00:00<?, ?it/s]

    =*= bug =*=
    precision:	0.8314
    recall:		0.8725
    F1 score:	0.8515

    =*= enhancement =*=
    precision:	0.8155
    recall:		0.8464
    F1 score:	0.8307

    =*= question =*=
    precision:	0.6521
    recall:		0.3502
    F1 score:	0.4557

    =*= global =*=
    precision:	0.8162
    recall:		0.8162
    F1 score:	0.8162


```python
#default_exp asciidoc
```

# fastdoc.asciidoc
> API for the fastdoc convertor

```python
#export
from fastdoc.imports import *
from fastcore.script import *
from warnings import warn
```

```python
#export
def markdown_cell(md):
    return nbformat.notebooknode.NotebookNode({'cell_type': 'markdown', 'source': md, 'metadata': {}})
```

```python
#export
def code_cell(code, metadata=None, outputs=None):
    return nbformat.notebooknode.NotebookNode(
        {'cell_type': 'code',
         'execution_count': None,
         'source': code,
         'metadata': {} if metadata is None else metadata,
         'outputs': [] if outputs is None else outputs})
```

## Preprocessing

### Preprocessing on the list of all cells

Removing cells with the flag `# hide`

```python
#export
_re_hidden = re.compile(r'^\s*#\s*(hide|clean)\s*$', re.MULTILINE)
```

```python
#export
def remove_hidden_cells(cells):
    "Remove cells marked with #hide"
    return [c for c in cells if _re_hidden.search(c['source']) is None]
```

```python
cells = [code_cell('# hide'), code_cell('lalala'), markdown_cell('lalala\n# hide')]
test_eq(remove_hidden_cells(cells), [code_cell('lalala')])
```

Isolating the bits in triple quotes annotated with asciidoc in code cells without outputs so that they are not interpreted by the converter, with adding `##clear##` so that the post-processing removes the `[python]` flag.

```python
#export
def isolate_adoc_blocks(cells):
    res = []
    for cell in cells:
        if cell['cell_type'] == 'markdown' and re.search(r'```\s*asciidoc', cell['source']) is not None:
            lines = cell['source'].split('\n')
            adoc,s,idx = False,0,0
            for line in lines:
                if re.search(r'^```\s*asciidoc\s*$', line) is not None and not adoc:
                    res.append(markdown_cell('\n'.join(lines[s:idx])))
                    adoc,s = True,idx+1
                elif re.search(r'^```\s*$', line) is not None and adoc:
                    res.append(code_cell('##clear##' + '\n'.join(lines[s:idx])))
                    adoc,s = False,idx+1
                idx+=1
            assert not adoc, f"Triple-quote asciidoc block not ended in {cell['source']}"
            res.append(markdown_cell('\n'.join(lines[s:])))
        else: res.append(cell)
    return res
```

```python
test = """This is some text
```asciidoc
This should be isolated
```
Some other text
```asciidoc
This should also be isolated
```
end"""
test_eq(isolate_adoc_blocks([markdown_cell(test)]), [
    markdown_cell("This is some text"),
    code_cell("##clear##This should be isolated"),
    markdown_cell("Some other text"),
    code_cell("##clear##This should also be isolated"),
    markdown_cell("end")
])
```

### Preprocessing individual code cells

Old way of putting `[WARNING]`, `[NOTE]` or `[TIP]`

```python
#export
#TODO: remove when all notebooks have been ported to v2
def replace_old_jekylls(cell):
    if cell['source'].startswith('jekyll'):
        pat1 = re.compile(r"""jekyll_(.*)\(['"].*""")
        pat2 = re.compile(r"""jekyll_.*\(['"]+([\s\S]*[^'"])['"]+\)$""")
        jekyll_type = re.match(pat1, cell['source']).groups()[0]
        message = re.match(pat2, cell['source']).groups()[0]
        inst = {'warn':'WARNING', 'note':'NOTE', 'important':'TIP'}
        cell['metadata'] = {}
        cell['source'] = f'##clear##[{inst[jekyll_type]}]\n====\n{message}\n===='
        cell['outputs'] = []
    return cell
```

```python
test_eq(replace_old_jekylls(code_cell('jekyll_warn("""Try to convert me!""")')),
        code_cell('##clear##[WARNING]\n====\nTry to convert me!\n===='))
```

Hide input of cells with `hide_input=True` in metadata (extension hide input) or a flag `#hide_input`. Put `##remove##` instead of the code that will be removed during post-processing

```python
#export
_re_hide_input = re.compile(r'^\s*#\s*hide_input\s*$', re.MULTILINE)
```

```python
#export
def hide_input(cell):
    if cell['metadata'].get('hide_input', False) or _re_hide_input.search(cell["source"]) is not None: cell['source'] = '##remove##'
    return cell
```

```python
test_eq(hide_input(code_cell('some code', metadata={'hide_input': True}, outputs=[1])),
        code_cell('##remove##', metadata={'hide_input': True}, outputs=[1]))
test_eq(hide_input(code_cell('# hide_input\nsome code', outputs=[1])),
        code_cell('##remove##', outputs=[1]))
```

Hide outputs of cells with `collapsed=True` in their metadata or a flag #hide_output

```python
#export
_re_hide_output = re.compile(r'^\s*#\s*hide_output\s*$', re.MULTILINE)
```

```python
#export
def hide_output(cell):
    if cell['metadata'].get('collapsed', False) or _re_hide_output.search(cell["source"]) is not None:
        cell['outputs'] = []
        cell['source'] = re.sub(r'#\s*hide_output\s*\n', '', cell['source'])
    return cell
```

```python
test_eq(hide_output(code_cell('some code', metadata={'collapsed': True}, outputs=[1])),
        code_cell('some code', metadata={'collapsed': True}))
test_eq(hide_output(code_cell('# hide_output\nsome code', outputs=[1])),
        code_cell('some code'))
```

Replace outputs as `text_html` by `text_plain` (otherwise they are not kept)

```python
#export
def extract_html(cell):
    for o in cell['outputs']:
        if 'data' in o and 'text/html' in o['data']:
            o['data']['text/plain'] = o['data']['text/html']
            del o['data']['text/html']
    return cell
```

```python
test_eq(extract_html(code_cell('some code', outputs=[{'data': {'text/html': 'some_html'}}])),
        code_cell('some code', outputs=[{'data': {'text/plain': 'some_html'}}]))
```

Deal with errors by putting them in plain text

```python
#export
def split_max_len(text, l):
    words = text.split(' ')
    line,lines = "",[]
    for word in words:
        if len(line) + len(word) + 1 <= l: line += f' {word}'
        else:
            lines.append(line)
            line = ""
    if len(line) > 0: lines.append(line)
    return "\n".join(lines)
```

```python
#export
def deal_error(cell):
    for i,out in enumerate(cell['outputs']):
        if out['output_type'] == 'error':
            msg = f"{out['ename']}: {out['evalue']}"
            cell['outputs'][i] = nbformat.notebooknode.NotebookNode({
                'data': {'text/plain': split_max_len(msg, 81) },
                'execution_count': None,
                'metadata': {},
                'output_type': 'execute_result'})
    return cell
```

```python
test_eq(deal_error(code_cell('some code', outputs=[{'output_type': 'error', 'ename': 'Error name', 'evalue': 'This is an error.'}])),
        code_cell('some code', outputs = [
            {'data': {'text/plain': ' Error name: This is an error.'},
             'execution_count': None,
             'metadata': {},
             'output_type': 'execute_result'}
        ]))
```

Remove interrupted progress bars from the outputs

```python
#export
def remove_interrupted_pbars(cell):
    outs = []
    for out in cell['outputs']:
        if 'data' not in out or 'text/plain' not in out['data'] or 'progress-bar-interrupted' not in out['data']['text/plain']:
            outs.append(out)
    cell['outputs'] = outs
    return cell
```

```python
test_eq(remove_interrupted_pbars(
    code_cell("some code", outputs = [{'a': 1}, {'data': {'text/plain': 'progress-bar-interrupted'}}, {'b': 2}])),
        code_cell("some code", outputs = [{'a': 1}, {'b': 2}]))
```

Get metadata for outputs.

```python
#export
def get_cell_meta(cell):
    for attr in ["id", "caption", "alt", "width"]:
        if re.search(r'^\s*#\s*' + attr + r'\s(.*)$', cell["source"], re.MULTILINE) is not None:
            cell["metadata"][attr] = re.search(r'^\s*#\s*' + attr + r'\s(.*)$', cell["source"], re.MULTILINE).groups()[0]
            cell["source"] = re.sub(r'#\s*' + attr + r'\s.*?($|\n)', '', cell["source"])
    return cell
```

```python
test_eq(get_cell_meta(code_cell("#id 123\n#caption This is a bear\nsome code")),
        code_cell("some code", metadata = {'id': '123', 'caption': 'This is a bear'}))
```

Deal with table captions and refs

```python
#export
def caption_tables(cell):
    if 'outputs' not in cell or len(cell['outputs']) == 0: return cell
    output = cell['outputs'][0]
    if 'data' not in output or 'text/plain' not in output['data']: return cell
    text = output['data']['text/plain']
    if re.search(r'^<\s*table\s+([^>]*>)', text) is None: return cell
    table_id = cell['metadata'].get('id', None)
    caption = cell['metadata'].get('caption', None)
    text_id = '' if table_id is None else f'id="{table_id}" '
    text_caption = '' if caption is None else f'\n
{caption}
'
    output['data']['text/plain'] = re.sub(r'^<\s*table\s+([^>]*>)', '<table '+text_id+r'\1'+text_caption, text)
    cell['outputs'][0] = output
    return cell
```

```python
cell = code_cell("some code",
    metadata={'id': '123', 'caption': 'a caption'},
    outputs=[{'data': {'text/plain': '
\nTable code'}}])
cell2 = code_cell("some code",
    metadata={'id': '123', 'caption': 'a caption'},
    outputs=[{'data': {'text/plain': '
\n
a caption
\nTable code'}}])
test_eq(caption_tables(cell), cell2)
```

```python
cell = code_cell("#hide_input\n#id 123\n#caption a caption",
    metadata={},
    outputs=[{'data': {'text/plain': '
\nTable code'}, 'output_type':''}])
```

Wrap text in outputs

```python
#export
TEXT_MAX_WIDTH = 80
```

```python
#export
def _wrap_output(output):
    if 'text' in output:
        lines = ['\n'.join(textwrap.wrap(l, width=TEXT_MAX_WIDTH, subsequent_indent = ' > ')) for l in output['text'].split('\n')]
        output['text'] = '\n'.join(lines)
        return output
    if ('data' not in output or 'text/plain' not in output['data']): return output
    text = output['data']['text/plain']
    if re.search(r'^<\s*table\s*([^>]*>)', text) is not None: return output
    lines = ['\n'.join(textwrap.wrap(l, width=TEXT_MAX_WIDTH, subsequent_indent = ' > ')) for l in text.split('\n')]
    output['data']['text/plain'] = '\n'.join(lines)
    return output
```

```python
#export
def wrap_text_outputs(cell):
    if 'outputs' not in cell or len(cell['outputs']) == 0: return cell
    cell['outputs'] = [_wrap_output(o) for o in cell['outputs']]
    return cell
```

```python
cell = code_cell("some code",
    metadata={},
    outputs=[{'data': {'text/plain': 'This is a long output'*5}, 'output_type':''},
             {'text': 'This is a long output'*5}])
wrapped = 'This is a long outputThis is a long outputThis is a long outputThis is a long\n > outputThis is a long output'
test_eq(wrap_text_outputs(cell), code_cell("some code",
    metadata={},
    outputs=[{'data': {'text/plain': wrapped}, 'output_type':''},
             {'text': wrapped}]))
```

Test code length

```python
#export
CODE_MAX_LEN = 81
```

```python
#export
def check_code_len(cell):
    lines = cell['source'].split('\n')
    for l in lines:
        if len(l) > CODE_MAX_LEN: warn(f"Found code too long in a cell:\n{cell['source']}")
    return cell
```

### Preprocessing individual markdown cells

Replace "\` \`" by \`\`

```python
#export
def deal_quotes(cell):
    cell['source'] = re.sub(r'"`([^`]*)`"', r'`\1`', cell['source'])
    cell['source'] = re.sub(r"'", r'xxsinglequote', cell['source'])
    return cell
```

```python
test_eq(deal_quotes(markdown_cell('"`code`"')), markdown_cell('`code`'))
test_eq(deal_quotes(markdown_cell('a"b"c')), markdown_cell('a"b"c'))
test_eq(deal_quotes(markdown_cell("a'b'c")), markdown_cell('axxsinglequotebxxsinglequotec'))
```

Add one title level to every Markdown cell

```python
#export
def add_title_level(cell):
    if cell['source'].startswith('#'): cell['source'] = '#' + cell['source']
    return cell
```

```python
test_eq(add_title_level(markdown_cell('# title')), markdown_cell('## title'))
```

Convert headers to asciidoc-style after they have been shifted one level by `add_title_level`

```python
#export
def title_to_asciidoc(cell):
    _re_header = re.compile(r'^#+\s+\S+')
    if _re_header.search(cell['source']) is not None:
        src = cell['source'].lstrip('#')
        src = ('=' * (len(cell['source']) - len(src))) + src
        cell['source'] = src
    return cell
```

```python
test_eq(title_to_asciidoc(markdown_cell('# title')), markdown_cell('= title'))
test_eq(title_to_asciidoc(markdown_cell('## title')), markdown_cell('== title'))
test_eq(title_to_asciidoc(markdown_cell('### title')), markdown_cell('=== title'))
test_eq(title_to_asciidoc(markdown_cell('#### title')), markdown_cell('==== title'))
test_eq(title_to_asciidoc(markdown_cell('##### title')), markdown_cell('===== title'))
test_eq(title_to_asciidoc(markdown_cell('#hide')), markdown_cell('#hide'))
test_eq(title_to_asciidoc(markdown_cell('#export')), markdown_cell('#export'))
```

Remove digits from numbered lists and format labeled lists

```python
#export
def deal_with_lists(cell):
    lines = cell['source'].split('\n')
    for i in range(len(lines)):
        lines[i] = re.sub(r'(^\s*)\d*\.(.*)$', r'\1.\2xxnewl', lines[i])
        lines[i] = re.sub(r'(^\s*)-\s(.*::)\s(.*)$', r'\2xxnewls\3xxnewl', lines[i])
        lines[i] = re.sub(r'(^\s*)-\s\*\*(.*):\*\*\s(.*)$', r'\2::xxnewls\3xxnewl', lines[i])
    cell['source'] = '\n'.join(lines)
    return cell
```

```python
test_eq(deal_with_lists(markdown_cell("  1. Item\n  2. Item")),
        markdown_cell("  . Itemxxnewl\n  . Itemxxnewl"))
test_eq(deal_with_lists(markdown_cell("- lbl1:: item1\n- lbl2:: item2")),
        markdown_cell("lbl1::xxnewlsitem1xxnewl\nlbl2::xxnewlsitem2xxnewl"))
test_eq(deal_with_lists(markdown_cell("- **lbl1:** item1\n- **lbl2:** item2")),
        markdown_cell("lbl1::xxnewlsitem1xxnewl\nlbl2::xxnewlsitem2xxnewl"))
```

Catch block quotes and put them in asciidoc blocks

```python
#export
_re_block_notes = re.compile(r"""
# Catches any pattern > Title: content with title in group 1 and content in group 2
^\s*>\s*     # > followed by any number of whitespace
([^:]*)      # Catching group for any character but :
:\s*         # : then any number of whitespace
([^\n]*)     # Catching group for anything but a new line character
(?:\n|$)     # Non-catching group for either a new line or the end of the text
""", re.VERBOSE | re.MULTILINE)

_re_forgot_column = re.compile("^\s*>[^:]*$", re.MULTILINE)
```

Handle admonitions. Atlas supports notes, tips, and warnings.

<div class="alert alert-info">
Note
</div>

<div class="alert alert-success">
Tip
</div>

<div class="alert alert-danger">
Warning
</div>

```python
#export
def replace_jekylls(cell):
    block_names = {'warning':'WARNING', 'note':'NOTE', 'important':'TIP', 'tip': 'TIP', 'stop': 'WARNING',
                   '**Warning**':'WARNING', '**Note**':'NOTE', '**TIP**':'TIP',
                   'jargon':'JARGON', 'question':'QUESTION', 'a': 'ALEXIS', 'j': 'JEREMY', 's': 'SYLVAIN'}
    def _rep(m):
        typ,text = m.groups()
        if text.startswith('**'): text = text[3:]
        typ = typ.replace('**', '')
        name = block_names.get(typ.lower(), typ.upper())
        if name in ['ALEXIS', 'JEREMY', 'SYLVAIN', 'JARGON', 'QUESTION']:
            title = name[0]+name[1:].lower()
            surro = 'NOTE'
            if name=='JARGON':
                splits = text.split(': ')
                title = f'{title}: {splits[0]}'
                text = ': '.join(splits[1:])
            if name in ['ALEXIS', 'JEREMY', 'SYLVAIN']:
                title = f"{title} says"
                surro = 'TIP'
            return f'```asciidoc\n.{title}\n[{surro}]\n====\n{text}\n====\n```\n'
        elif len(name) != 0: return f"```asciidoc\n[{name}]\n====\n{text}\n====\n```\n"
        else:              return f"```asciidoc\n____\n{text}\n____\n```\n"
    if _re_forgot_column.search(cell["source"]): warn("Found a non-processed block quote: ")
    cell["source"] = _re_block_notes.sub(_rep, cell["source"])
    return cell
```

```python
test_eq(replace_jekylls(markdown_cell("text\n> : This is a block quote")),
    markdown_cell("text\n```asciidoc\n____\nThis is a block quote\n____\n```\n"))
test_eq(replace_jekylls(markdown_cell("text\n> jargon: term: Some new term")),
    markdown_cell('text\n```asciidoc\n.Jargon: term\n[NOTE]\n====\nSome new term\n====\n```\n'))
test_eq(replace_jekylls(markdown_cell(".Who's that Pokemon\n> **Tip:** Some new definition")),
    markdown_cell(".Who's that Pokemon\n```asciidoc\n[TIP]\n====\nSome new definition\n====\n```\n"))
test_warns(lambda: replace_jekylls(markdown_cell("text\n> This is a block quote")))
```

```python
#export
_re_sidebar = re.compile(r'^\s*#\s*sidebar\s(.*)$', re.MULTILINE)
```

```python
#export
def interpret_sidebar(cell):
    lines = cell["source"].split("\n")
    if _re_sidebar.search(lines[0]) is not None:
        title = _re_sidebar.search(lines[0]).groups()[0]
        body = "\n".join(lines[1:])
        cell["source"] = f"```asciidoc\n.{title}\n****\n{body}\n****\n```\n"
    return cell
```

```python
test = """#sidebar My intervention

This will be changed to a sidebar when converted in Asciidoc.

It can have several lines, contrary to a block quote."""
interpret_sidebar(markdown_cell(test))
```

    {'cell_type': 'markdown',
     'source': '```asciidoc\n.My intervention\n****\n\nThis will be changed to a sidebar when converted in Asciidoc.\n\nIt can have several lines, contrary to a block quote.\n****\n```\n',
     'metadata': {}}

```python
#export
_re_html_image = re.compile(r"^(]*>)", re.MULTILINE)
```

```python
#export
IMAGE_CONV_MULT = 0.6
```

```python
#export
def process_images(cell):
    h = HTMLParseAttrs()
    def _rep(m):
        d = h(m.groups()[0])
        attrs = ['"' + d.get('alt', '') + '"']
        if 'width' in d: attrs.append(str(int(IMAGE_CONV_MULT * int(d['width']))))
        if 'width' in d and 'height' in d: attrs.append(str((int(IMAGE_CONV_MULT * int(d['height'])))))
        suff = f"[{', '.join(attrs)}]"
        pid = f"[[{d['id']}]]\n" if 'id' in d else ""
        caption = f".{d['caption']}\n" if 'caption' in d else ""
        return f"```asciidoc\n{pid}{caption}image::{d['src']}{suff}\n```"
    cell["source"] = _re_html_image.sub(_rep, cell["source"])
    return cell
```

```python
txt = 'text\n\nother text'
test_eq(process_images(markdown_cell(txt)),
        markdown_cell('text\n```asciidoc\n[[123]]\n.This is an image\nimage::puppy.jpg["Alternative text", 420]\n```\nother text'))
```

Format regular markdown style images making some assumptions about caption and id

```python
#export
_re_md_image = re.compile(r'!\[(.*)\]\((.+)\)')
```

```python
#export
def process_md_images(cell):
    def _rep(m):
        text = m.groups()[0]
        src = m.groups()[1]
        if not os.path.exists(src): warn(f"Could not find image: {src}")
        suff = f'["{text}"]'
        fname = os.path.basename(src)
        pid = f"[[{os.path.splitext(fname)[0]}]]\n"
        caption = f".{text}\n"
        return f"```asciidoc\n{pid}{caption}image::{src}{suff}\n```"
    cell["source"] = _re_md_image.sub(_rep, cell["source"])
    return cell
```

```python
txt = 'text\n\nother text'
test_eq(process_md_images(markdown_cell(txt)),
        markdown_cell('text\n```asciidoc\n[[puppy]]\n.This is an image\nimage::images/puppy.jpg["This is an image"]\n```\nother text'))
```

    <ipython-input-78-3427beca4fcb>:6: UserWarning: Could not find image: images/puppy.jpg
      if not os.path.exists(src): warn(f"Could not find image: {src}")

Handle cross-references

```python
#export

# Old reference style
# _re_reference = re.compile(r'<<([^>]*)>>')

# Using double square brackets is more natural and doesn't
# mess with suntax highligting. It's also similar to tools
# like Roam reseach
_re_reference = re.compile(r'\[\[([^>]*)\]\]')
```

```python
#export
def wrap_references(cell):
    cell["source"] = _re_reference.sub(r'xxref\1xxeref', cell["source"])
    return cell
```

```python
# test_eq(wrap_references(markdown_cell("There is a reference <<ref>> here.")),
#         markdown_cell("There is a reference xxrefrefxxeref here."))
test_eq(wrap_references(markdown_cell("There is a reference [[ref]] here.")),
        markdown_cell("There is a reference xxrefrefxxeref here."))
```

```python
#export
def extract_attachments(cell, dest):
    if not 'attachments' in cell: return cell
    mime,img = first(first(cell['attachments'].values()).items())
    ext = mime.split('/')[1]
    for i in range(99999):
        p = dest/(f'att_{i:05d}.{ext}')
        if not p.exists(): break
    p.write_bytes(b64decode(img))
    del(cell['attachments'])
    cell['source'] = re.sub('attachment:image.png', str(p), cell['source'])
    return cell
```

Sidebars can also be delimitted to create a more natural look and feel in markdown. A markdown-rendered sidebar looks like this:

****
#### A Sidebar

This is a sidebar that contains some text. It can be **emphasized**, _underlined_, and `monospaced` as well.

****

```python
#export

# This is the original implementation of sidebar delimitters
# _re_sidebar_title = re.compile(r'#+\s+Sidebar:\s+(.*)$', re.IGNORECASE)
# _re_end_sidebar = re.compile(r'#+\s+End sidebar', re.IGNORECASE)

# We can instead use horizontal lines, which looks better in markdown anyway.
_re_sidebar_title = re.compile(r'\n*\*\*\*\*\n#+\s+(.*)$', re.IGNORECASE)
_re_end_sidebar = re.compile(r'\*\*\*\*$', re.IGNORECASE)
```

```python
_re_sidebar_title.search("****\n### Sidebar: Tenacity in deep learning").groups()
```

    ('Sidebar: Tenacity in deep learning',)

```python
#export
def sidebar_headers(cell):
    cell['source'] = _re_sidebar_title.sub(r'```asciidoc\n.\1\n****\n```', cell['source'])
    cell['source'] = _re_end_sidebar.sub(r'```asciidoc\n****\n```', cell['source'])
    return cell
```

```python
test_eq(sidebar_headers(
        markdown_cell("****\n### My Intervention")),
        markdown_cell("```asciidoc\n.My Intervention\n****\n```")
       )
test_eq(sidebar_headers(
        markdown_cell("****")),
        markdown_cell("```asciidoc\n****\n```")
       )
```

### Preprocess individual raw cells

Raw cells just need to have a new line added at the beginning

```python
#export
def add_new_line(cell):
    cell['source'] = '\n' + cell['source']
    return cell
```

Expand `#include` statements in raw cells. This allows the inclusion of external markdown and asciidoc files in a notebook using a C-like `#include` syntax.

Include cells use the following syntax:
```
#include FILENAME.FILE_EXTENSION
```

Where `FILE_EXTENSION` is either `md`, `asciidoc` or `adoc`. Include cells are expected to contain only include statements

```python
#export
_re_include_md = re.compile(r'^#include\s+.*\.md$')
_re_include_adoc = re.compile(r'^#include\s+.*\.(asciidoc|adoc)$')
```

```python
#export
def expand_include(cell):
    if _re_include_md.match(cell['source']):
        cell['cell_type'] = 'markdown'
        cell['source'] = open(re.compile(r'\s+').split(cell['source'])[1]).read()
    elif _re_include_adoc.match(cell['source']):
        cell['source'] = open(re.compile(r'\s+').split(cell['source'])[1]).read()
    return cell
```

```python
test_eq(expand_include(markdown_cell('#include README.md'))['source'], open('README.md').read())
test_eq(expand_include(markdown_cell('#incl ude README.md'))['source'], '#incl ude README.md')
```

### All preprocessing together

```python
#export
code_cell_tfms = [get_cell_meta, replace_old_jekylls, hide_input, hide_output, extract_html, deal_error,
                  remove_interrupted_pbars, wrap_text_outputs, caption_tables, check_code_len]
md_cell_tfms = [deal_quotes, wrap_references, interpret_sidebar, sidebar_headers, add_title_level,
                title_to_asciidoc, deal_with_lists, replace_jekylls, process_md_images, process_images]
raw_cell_tfms = [add_new_line, expand_include]
```

```python
#export
def treat_notebook(nb, dest):
    nb['cells'] = remove_hidden_cells(nb['cells'])
    nb['cells'] = [(expand_include(c) if c['cell_type'] == 'raw' else c) for c in nb['cells']]
    tfm_func = {'code': compose(*code_cell_tfms), 'markdown': compose(partial(extract_attachments, dest=dest), *md_cell_tfms),
                'raw': compose(*raw_cell_tfms)}
    nb['cells'] = [tfm_func[c['cell_type']](c) for c in nb['cells']]
    nb['cells'] = isolate_adoc_blocks(nb['cells'])
    return nb
```

## Post-processing

Replace special tokens by their values

```python
#export
def rep_spec_tok(adoc, metadata=None):
    adoc = re.sub('xxsinglequote', "'", adoc)
    adoc = re.sub('xxnewls', '\n  ', adoc)
    return re.sub('xxnewl\s', '\n', adoc)
```

nbconvert will flag the code cells with `[ipython3]`, we replace this by `[python]`

```python
#export
def ipython2python(adoc, metadata=None):
    return re.sub(r'\[source, ipython3\]','[source, python]', adoc)
```

```python
test_eq(ipython2python("[source, ipython3]\n----\nsome code\n----\n"), "[source, python]\n----\nsome code\n----\n")
```

Remove empty cells or cells flagged for removal (because of hide_input)

```python
#export
def remove_cells(adoc, metadata=None):
    adoc = re.sub(r'\n\[source, python\]\n----(\n)*----\n','', adoc)
    return re.sub(r'\n\[source, python\]\n----\n##remove##\n----\n','', adoc)
```

```python
test_eq(remove_cells("lalala\n[source, python]\n----\n\n----\n"), "lalala")
test_eq(remove_cells("lalala\n[source, python]\n----\n##remove##\n----\n"), "lalala")
```

Clear code cells from the code flag when there is a `##clear##` tag.

```python
#export
_re_clear = re.compile(r'\[source, python\]\n----\n##clear##(.*?)----\n', re.DOTALL)
def clear_cells(adoc, metadata=None): return _re_clear.sub(r'\1', adoc)
```

```python
test_eq(clear_cells(
    "lalala\n[source, python]\n----\n##clear##pure adoc\n----\nfoo\nbla\n[source, python]\n----\n##clear##pure adoc again\n----\nbli"),
        "lalala\npure adoc\nfoo\nbla\npure adoc again\nbli")
```

Format LaTeX equations properly: they arrive either as `latexmath:[$equation$]` or `latexmath:[\[equation\]]`

```python
#export
def format_latex(adoc, metadata=None):
    #LaTeX equations
    adoc = re.sub(r"latexmath:\[\$([^\$]*)\$\]", r"latexmath:[\\(\1\\)]", adoc)
    return re.sub(r"latexmath:\[\\\[(.*)\\\]\]", r"\n[latexmath]\n++++\n\\begin{equation}\n\1\n\\end{equation}\n++++\n", adoc)
```

```python
test_eq(format_latex(r"latexmath:[$equation$]"), r"latexmath:[\(equation\)]")
test_eq(format_latex(r"latexmath:[\[equation\]]"),
        "\n[latexmath]\n++++\n\\begin{equation}\nequation\n\\end{equation}\n++++\n")
```

Format image outputs and make sure they point to the right folder.

```python
#export
_re_image_output = re.compile(r'----\n!\[(?:svg|png|jpg)\]\((.+)\)\n----')
```

```python
#export
def format_outputs(adoc, metadata=None):
    folder = ({} if metadata is None else metadata).get('folder', '.')
    def _rep(m):
        name = m.groups()[0]
        d = metadata[name] if metadata is not None and name in metadata else {}
        attrs = ['"' + d.get('alt', '') + '"']
        if 'width' in d: attrs.append(str(d['width']))
        if 'width' in d and 'height' in d: attrs.append(str(d['height']))
        suff = f"[{', '.join(attrs)}]"
        pid = f"[[{d['id']}]]\n" if 'id' in d else ""
        caption = f".{d['caption']}\n" if 'caption' in d else ""
        return f"{pid}{caption}image::{str(folder)}/{name}{suff}"
    return _re_image_output.sub(_rep, adoc)
```

```python
test_eq(format_outputs('----\n\n----', {'folder':'path', 'output.svg': {'alt': 'alt'}}),
        'image::path/output.svg["alt"]')
test_eq(format_outputs('----\n\n----', {'folder':'path', 'output.svg': {'alt': 'alt', 'width': 100}}),
        'image::path/output.svg["alt", 100]')
test_eq(format_outputs('----\n\n----'),
        'image::./output1.png[""]')
```

Deal with quotes

```python
#export
def fix_quotes(adoc, metadata=None):
    return re.sub(r"``([^'`]*)''", r'"\1"', adoc)
```

```python
test_eq(fix_quotes("``double quotes''"), '"double quotes"')
```

Put back << >> around refs

```python
#export
def fix_references(adoc, metadata=None): return re.sub(r"xxref(.*)xxeref", r"<<\1>>", adoc)
```

```python
test_eq(fix_references("There is a reference xxrefrefxxeref here."), "There is a reference <<ref>> here.")
```

Format tables

```python
#export
def format_tables(adoc, metadata=None):
    splits = adoc.split('----')
    seps = [''] + ['----' for _ in range(len(splits)-1)] + ['']
    for i,s in enumerate(splits):
        s = re.sub(r'
[\s\S]*<table', ' 0 and not s.startswith('\n'): s = '\n' + s
        if len(s) > 0 and not s.endswith('\n'):   s = s + '\n'
        if s.startswith('\n<table'): seps[i],seps[i+1] = '++++','++++'
        elif '<table' in s:
            res = re.search('<table', s)
            begin,end = res.span()
            s = s[:begin] + '\n----\n\n++++\n' + s[begin:]
            seps[i+1] = '++++'
        splits[i] = s
    res = ''
    for s,c in zip(seps,splits): res = res + s + c
    return res.replace('\n\n--------', '')
```

Just as a personal preference, replace all blocks of three new lines or more by \n\n

```python
#export
def remove_lines(text, metadata=None):
    return re.sub(r'\n\n\n\n+([^\n])', r'\n\n\n\1', text)
```

```python
test_eq(remove_lines('a\n\n\n\n\n\nb'), 'a\n\n\nb')
```

All together

```python
#export
post_process_tfms = [fix_quotes, rep_spec_tok, ipython2python, remove_cells, clear_cells, format_latex,
                     format_outputs, fix_references, format_tables, remove_lines]
```

```python
#export
def post_process(adoc, metadata=None):
    if not adoc.startswith('\n'): adoc = '\n' + adoc
    adoc = re.sub('xxnewl\s', '\n', adoc)
    adoc = compose(*post_process_tfms)(adoc, metadata=metadata)
    return adoc.strip()
```

## Exporting

```python
#export
c = ExportConfig()
exporter = ASCIIDocExporter(c)
exporter.exclude_input_prompt=True
exporter.exclude_output_prompt=True
```

```python
#export
def add_metadata(nb):
    "Stripping removes metadata used in the conversion."
    if 'language_info' not in nb['metadata']:
        nb['metadata']['language_info'] = {
            'codemirror_mode': {'name': 'ipython', 'version': 3},
            'file_extension': '.py',
            'mimetype': 'text/x-python',
            'name': 'python',
            'nbconvert_exporter': 'python',
            'pygments_lexer': 'ipython3',
            'version': '3.7.1'}
    return nb
```

```python
#export
def output_num(n):
    m = re.search(r'^output_(\d*)_', n)
    if m is None: return
    return int(m.groups()[0])
```

```python
test_eq(output_num('output_31_0.png'), 31)
test_eq(output_num('output_12_0.svg'), 12)
```

```python
#export
import PIL
```

```python
#export
IMAGE_OUT_MULT = 0.8
```

```python
#export
import xml.etree.ElementTree as ET
```

```python
#export
def get_output_width(name, raw, folder):
    if name.endswith('.svg'): return ET.fromstring(raw).attrib['width'].split('.')[0].replace('pt', '')
    try: return PIL.Image.open(Path(folder)/name).size[0]
    except: return None
```

```python
#export
def convert_nb(fname, dest_path='.', folder=None):
    "Convert a notebook `fname` to html file in `dest_path`."
    print(f"Converting {fname}")
    fname = Path(fname)
    dest_name = fname.with_suffix('.asciidoc').name
    if folder is None: folder = Path(dest_path)/f'{fname.stem}_files'
    #folder for images. Clear if exists
    if folder.exists(): shutil.rmtree(folder)
    os.makedirs(folder, exist_ok=True)

    nb = add_metadata(treat_notebook(read_nb(fname), folder))
    export = exporter.from_notebook_node(nb)
    metadata = {'folder': folder.relative_to(dest_path)}
    metadata.update({n: nb["cells"][output_num(n)]['metadata'] for n in export[1]['outputs'].keys() if output_num(n) is not None})
    for n,o in export[1]['outputs'].items():
        with open(Path(folder)/n, 'wb') as f: f.write(o)
        w = metadata[n]['width'] if 'width' in metadata[n] else get_output_width(n, o, folder)
        if w is not None: metadata[n]['width'] = str(int(IMAGE_OUT_MULT * int(w)))
    with open(f'{dest_path}/{dest_name}','w', encoding="utf8") as f:
        f.write(post_process(export[0], metadata))
```

```python
dest = Path('test')
convert_nb('test/_test.ipynb', dest)
```

    Converting test/_test.ipynb

```python
#convert_nb('test/_test.ipynb', Path('test'))
```

```python
#export
def _copy_images(path, dest_path):
    os.makedirs(dest_path, exist_ok=True)
    for f in path.iterdir():
        if f.is_file(): shutil.copy(f, dest_path/f.name)
        if f.is_dir(): _copy_images(f, dest_path/f.name)
```

```python
#export
def copy_images(path, dest_path):
    img_folder = dest_path/"images"
    if img_folder.exists(): shutil.rmtree(img_folder)
    _copy_images(path/"images", img_folder)
```

```python
dest = Path('..')/'convert_book'
# copy_images(Path('book'), dest)
```

```python
#export
def _convert1(fname, dest_path='.'):
    try: convert_nb(fname, dest_path=dest_path)
    except Exception as e:
        print(f"Error in notebook {fname}")
        print(e)
```

```python
#export
@call_parse
def fastdoc_convert_all(
    path:Param("Path to notebooks",str)='book',
    dest_path:Param("Path to generated asciidoc files",str)='../convert_book'):
    path,dest_path = Path(path),Path(dest_path)
    dest_path.mkdir(parents=True,exist_ok=True)
    (path/'images').mkdir(parents=True,exist_ok=True)
    nbs = [f for f in path.iterdir() if f.suffix == '.ipynb' and not f.name.startswith('_')]
    parallel(_convert1, nbs, dest_path=dest_path)
    for f in path.iterdir():
        if f.suffix in ['.adoc', '.asciidoc']: shutil.copy(f, dest_path/f.name)
    copy_images(path, dest_path)
```

```python
#convert_all()
```

## Export -

```python
from nbdev.export import *
notebook2script()
```

    Converted 00_asciidoc.ipynb.
    Converted 01_clean.ipynb.
    Converted index.ipynb.

```python

```


# Rich Output

In Python, objects can declare their textual representation using the `__repr__` method.  IPython expands on this idea and allows objects to declare other, rich representations including:

* HTML
* JSON
* PNG
* JPEG
* SVG
* LaTeX

A single object can declare some or all of these representations; all are handled by IPython's *display system*. This Notebook shows how you can use this display system to incorporate a broad range of content into your Notebooks.

## Basic display imports

The `display` function is a general purpose tool for displaying different representations of objects. Think of it as `print` for these rich representations.

```python
from IPython.display import display
```

A few points:

* Calling `display` on an object will send **all** possible representations to the Notebook.
* These representations are stored in the Notebook document.
* In general the Notebook will use the richest available representation.

If you want to display a particular representation, there are specific functions for that:

```python
from IPython.display import (
    display_pretty, display_html, display_jpeg,
    display_png, display_json, display_latex, display_svg
)
```

## Images

To work with images (JPEG, PNG) use the `Image` class.

```python
from IPython.display import Image
```

```python
i = Image(filename='ipython_logo.png')
```

Returning an `Image` object from an expression will automatically display it:

```python
i
```

Or you can pass an object with a rich representation to `display`:

```python
display(i)
```

An image can also be displayed from raw data or a URL.

```python
Image(url='http://python.org/images/python-logo.gif')
```

SVG images are also supported out of the box.

```python
from IPython.display import SVG
SVG(filename='python_logo.svg')
```

### Embedded vs non-embedded Images

By default, image data is embedded in the notebook document so that the images can be viewed offline. However it is also possible to tell the `Image` class to only store a *link* to the image. Let's see how this works using a webcam at Berkeley.

```python
from IPython.display import Image
img_url = 'http://www.lawrencehallofscience.org/static/scienceview/scienceview.berkeley.edu/html/view/view_assets/images/newview.jpg'

# by default Image data are embedded
Embed      = Image(img_url)

# if kwarg `url` is given, the embedding is assumed to be false
SoftLinked = Image(url=img_url)

# In each case, embed can be specified explicitly with the `embed` kwarg
# ForceEmbed = Image(url=img_url, embed=True)
```

Here is the embedded version. Note that this image was pulled from the webcam when this code cell was originally run and stored in the Notebook. Unless we rerun this cell, this is not todays image.

```python
Embed
```

Here is today's image from same webcam at Berkeley, (refreshed every minutes, if you reload the notebook), visible only with an active internet connection, that should be different from the previous one. Notebooks saved with this kind of image will be smaller and always reflect the current version of the source, but the image won't display offline.

```python
SoftLinked
```

Of course, if you re-run this Notebook, the two images will be the same again.

## HTML

Python objects can declare HTML representations that will be displayed in the Notebook. If you have some HTML you want to display, simply use the `HTML` class.

```python
from IPython.display import HTML
```

```python
s = """<table>
<tr>
<th>Header 1</th>
<th>Header 2</th>
</tr>
<tr>
<td>row 1, cell 1</td>
<td>row 1, cell 2</td>
</tr>
<tr>
<td>row 2, cell 1</td>
<td>row 2, cell 2</td>
</tr>
</table>"""
```

```python
h = HTML(s)
```

```python
display(h)
```

Header 1
Header 2

row 1, cell 1
row 1, cell 2

row 2, cell 1
row 2, cell 2

You can also use the `%%html` cell magic to accomplish the same thing.

```python
%%html
<table>
<tr>
<th>Header 1</th>
<th>Header 2</th>
</tr>
<tr>
<td>row 1, cell 1</td>
<td>row 1, cell 2</td>
</tr>
<tr>
<td>row 2, cell 1</td>
<td>row 2, cell 2</td>
</tr>
</table>
```

Header 1
Header 2

row 1, cell 1
row 1, cell 2

row 2, cell 1
row 2, cell 2

## JavaScript

The Notebook also enables objects to declare a JavaScript representation. At first, this may seem odd as  output is inherently visual and JavaScript is a programming language. However, this opens the door for rich output that leverages the full power of JavaScript and associated libraries such as [d3.js](http://d3js.org) for output.

```python
from IPython.display import Javascript
```

Pass a string of JavaScript source code to the `JavaScript` object and then display it.

```python
js = Javascript('alert("hi")');
```

```python
display(js)
```

    <IPython.core.display.Javascript object>

The same thing can be accomplished using the `%%javascript` cell magic:

```javascript
%%javascript

alert("hi");
```

    <IPython.core.display.Javascript object>

## LaTeX

The IPython display system also has builtin support for the display of mathematical expressions typeset in LaTeX, which is rendered in the browser using [MathJax](http://mathjax.org).

You can pass raw LaTeX test as a string to the `Math` object:

```python
from IPython.display import Math
Math(r'F(k) = \int_{-\infty}^{\infty} f(x) e^{2\pi i k} dx')
```

$$F(k) = \int_{-\infty}^{\infty} f(x) e^{2\pi i k} dx$$

With the `Latex` class, you have to include the delimiters yourself.  This allows you to use other LaTeX modes such as `eqnarray`:

```python
from IPython.display import Latex
Latex(r"""\begin{eqnarray}
\nabla \times \vec{\mathbf{B}} -\, \frac1c\, \frac{\partial\vec{\mathbf{E}}}{\partial t} & = \frac{4\pi}{c}\vec{\mathbf{j}} \\
\nabla \cdot \vec{\mathbf{E}} & = 4 \pi \rho \\
\nabla \times \vec{\mathbf{E}}\, +\, \frac1c\, \frac{\partial\vec{\mathbf{B}}}{\partial t} & = \vec{\mathbf{0}} \\
\nabla \cdot \vec{\mathbf{B}} & = 0
\end{eqnarray}""")
```

\begin{eqnarray}
\nabla \times \vec{\mathbf{B}} -\, \frac1c\, \frac{\partial\vec{\mathbf{E}}}{\partial t} & = \frac{4\pi}{c}\vec{\mathbf{j}} \\
\nabla \cdot \vec{\mathbf{E}} & = 4 \pi \rho \\
\nabla \times \vec{\mathbf{E}}\, +\, \frac1c\, \frac{\partial\vec{\mathbf{B}}}{\partial t} & = \vec{\mathbf{0}} \\
\nabla \cdot \vec{\mathbf{B}} & = 0
\end{eqnarray}

Or you can enter LaTeX directly with the `%%latex` cell magic:

```latex
%%latex
\begin{align}
\nabla \times \vec{\mathbf{B}} -\, \frac1c\, \frac{\partial\vec{\mathbf{E}}}{\partial t} & = \frac{4\pi}{c}\vec{\mathbf{j}} \\
\nabla \cdot \vec{\mathbf{E}} & = 4 \pi \rho \\
\nabla \times \vec{\mathbf{E}}\, +\, \frac1c\, \frac{\partial\vec{\mathbf{B}}}{\partial t} & = \vec{\mathbf{0}} \\
\nabla \cdot \vec{\mathbf{B}} & = 0
\end{align}
```

\begin{align}
\nabla \times \vec{\mathbf{B}} -\, \frac1c\, \frac{\partial\vec{\mathbf{E}}}{\partial t} & = \frac{4\pi}{c}\vec{\mathbf{j}} \\
\nabla \cdot \vec{\mathbf{E}} & = 4 \pi \rho \\
\nabla \times \vec{\mathbf{E}}\, +\, \frac1c\, \frac{\partial\vec{\mathbf{B}}}{\partial t} & = \vec{\mathbf{0}} \\
\nabla \cdot \vec{\mathbf{B}} & = 0
\end{align}

## Audio

IPython makes it easy to work with sounds interactively. The `Audio` display class allows you to create an audio control that is embedded in the Notebook. The interface is analogous to the interface of the `Image` display class. All audio formats supported by the browser can be used. Note that no single format is presently supported in all browsers.

```python
from IPython.display import Audio
Audio(url="http://www.nch.com.au/acm/8k16bitpcm.wav")
```

<audio controls="controls" >
    <source src="http://www.nch.com.au/acm/8k16bitpcm.wav" type="audio/x-wav" />
    Your browser does not support the audio element.
</audio>

A NumPy array can be auralized automatically. The `Audio` class normalizes and encodes the data and embeds the resulting audio in the Notebook.

For instance, when two sine waves with almost the same frequency are superimposed a phenomena known as [beats](https://en.wikipedia.org/wiki/Beat_%28acoustics%29) occur. This can be auralised as follows:

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
max_time = 3
f1 = 220.0
f2 = 224.0
rate = 8000.0
L = 3
times = np.linspace(0,L,rate*L)
signal = np.sin(2*np.pi*f1*times) + np.sin(2*np.pi*f2*times)

plt.plot(signal)
Audio(data=signal, rate=rate)
```

## Video

More exotic objects can also be displayed, as long as their representation supports the IPython display protocol.  For example, videos hosted externally on YouTube are easy to load:

```python
from IPython.display import YouTubeVideo
YouTubeVideo('sjfsUzECqK0')
```

Using the nascent video capabilities of modern browsers, you may also be able to display local
videos.  At the moment this doesn't work very well in all browsers, so it may or may not work for you;
we will continue testing this and looking for ways to make it more robust.

The following cell loads a local file called  `animation.m4v`, encodes the raw video as base64 for http
transport, and uses the HTML5 video tag to load it. On Chrome 15 it works correctly, displaying a control bar at the bottom with a play/pause button and a location slider.

```python
from IPython.display import HTML
from base64 import b64encode
video = open("animation.m4v", "rb").read()
video_encoded = b64encode(video).decode('ascii')
video_tag = '<video controls alt="test" src="data:video/x-m4v;base64,{0}">'.format(video_encoded)
HTML(data=video_tag)
```

<video controls alt="test" src="data:video/x-m4v;base64,AAAAHGZ0eXBNNFYgAAACAGlzb21pc28yYXZjMQAAAAhmcmVlAAAqiW1kYXQAAAKMBgX//4jcRem95tlIt5Ys2CDZI+7veDI2NCAtIGNvcmUgMTE4IC0gSC4yNjQvTVBFRy00IEFWQyBjb2RlYyAtIENvcHlsZWZ0IDIwMDMtMjAxMSAtIGh0dHA6Ly93d3cudmlkZW9sYW4ub3JnL3gyNjQuaHRtbCAtIG9wdGlvbnM6IGNhYmFjPTEgcmVmPTMgZGVibG9jaz0xOjA6MCBhbmFseXNlPTB4MzoweDExMyBtZT1oZXggc3VibWU9NyBwc3k9MSBwc3lfcmQ9MS4wMDowLjAwIG1peGVkX3JlZj0xIG1lX3JhbmdlPTE2IGNocm9tYV9tZT0xIHRyZWxsaXM9MSA4eDhkY3Q9MSBjcW09MCBkZWFkem9uZT0yMSwxMSBmYXN0X3Bza2lwPTEgY2hyb21hX3FwX29mZnNldD0tMiB0aHJlYWRzPTEgc2xpY2VkX3RocmVhZHM9MCBucj0wIGRlY2ltYXRlPTEgaW50ZXJsYWNlZD0wIGJsdXJheV9jb21wYXQ9MCBjb25zdHJhaW5lZF9pbnRyYT0wIGJmcmFtZXM9MyBiX3B5cmFtaWQ9MiBiX2FkYXB0PTEgYl9iaWFzPTAgZGlyZWN0PTEgd2VpZ2h0Yj0xIG9wZW5fZ29wPTAgd2VpZ2h0cD0yIGtleWludD0yNTAga2V5aW50X21pbj0yNSBzY2VuZWN1dD00MCBpbnRyYV9yZWZyZXNoPTAgcmNfbG9va2FoZWFkPTQwIHJjPWNyZiBtYnRyZWU9MSBjcmY9MjMuMCBxY29tcD0wLjYwIHFwbWluPTAgcXBtYXg9NjkgcXBzdGVwPTQgaXBfcmF0aW89MS40MCBhcT0xOjEuMDAAgAAACqVliIQAV/0TAAI/3gU2tIW7KawwaCmQGTGHKmuYAAADACBcshU+yICkgAA14AHowiEeT6ei7v7h3Hu0i2fpUBLGBIkbCMP3Vfz+9BVGCDXnw9Uv5o3iN030tb7eq6rsEEhHs2azbdTiE9Csz5Zm6SiUWRdmB43hbD5i6syATuODUJd7LM3d9cbFpc7zFlu5y3vUmNGd6urpvKKT9iyleIyTuR1sVS431DhevGfkUllVeIznYUe2USoMW1tufETjyRdmGldN6eNlhAOsGAH4z+HkrwKecPPU7Q5T4gDAIxj9hW84jVExMTSTHxkPTq1I4OotgUxURCGTsw60k/ezPNmNg38j1bqaGmPcruDKEIBDsK5qEytFB90Q68s0h2wmlf2KXd5bleBefiK+/p47ZsyUO4IdlW25rRy+HLjt6wQXfYee3IkiQOoOK+U7u/lxcl78zfxwIoEMjUUSKNZjkp8clnmecDDJ3Kz+viF7bPklk7N6QRyizAKPIIpnNJUuMWQmqeL2Or6cr4D0/0tOym+4tficxmhuEONKUtO2pPn3hRjMllkd12tXp70fLTfxy0dwB70ML9iLEcItHb7zVupHlP5RxdvecpREw+OsIPr9KWilIesNE19jgIbT+TkiRBjOoKvUuwcQnKg7fOTHVoLvnKuAfea+oujEdm1Rwd2tEOnkF+ZC11WaNQsiNR/eJ9EnUXjXDYGfhB+Oe7qj8nYTT+eOXg1cuJNgLXEs4vOheWEjQOqfIWMQc3DmTof5s0ksBmUQ3PQ+UHPxZSnmOEZB+j6xT3wbm7HGzDjWtSg1SjTxd1EiJ8xA4SIxxR8WIKLg+TwFxJNS7Laxq7Uglu3AkXe82P1JCdJX5PsbFbxuDbuJgakzRcTwMLLSKCiizS/eCW0uJed/lev9yb80kKlVET4S219cn/zhkpeDV83cHYOr+sJQKDRk/Wh2c7fsuxfxaEH/6reSmvFDsAnXAyPXliJ3G4VG3OkEM5K5WyGGrBizZbTrdGsBnzj5VSGGOJdCKuRrUluw/8es2vYRPs9BcTqAqvHk9M52SSIf+1T6L53EZP8VbtXB+G29CMW4xVCK/B/YDjaNmqMwJ61dapugjnWJfqeXlGGa3Ch3aA7gi30T8PucNRBjLK3lF67ZDDvkWXRQXd+VMnKWHkBbCkQ/F/fMuNpHO3C00Y2pljna1qImBhVMvPe0F7Qx7G/YyxLRzhyUU8e23HGzp0agtNJRbydbrPV+TqJMSifJMNcZIf8wkdnC3/xdpcXnLf2Ye3Kbd0o7utciTG+q5h6WTEk+PaNbXLLA0YyZ2VnLTcyV1QTS76aNCbV9Q1/OQ7QU81Gg0hPa9aSiscGary6jLVwDQaik4zLsi7jPqgPVdup7pwx7uJDqRCVcVi5QoZFp/GHdex5sJTF69A6sja69/NLkFIWNSIeRcuGahXpF+wZeYIrqJv975s1TKYKAvp1WtzgtgWNkcbzCtROqf8rPtlAIxkX8GLcEo9zfExyfimeXQ64qfFxEy0IMy2Hsxau9fSMqUnIjntuVVjCQtBL+94gx1RZLndE6wROVTq/wHwHrQzo9QL9cpPqPFJjiZ/NGZIFuudS+wsBFe6Hu8Oitf5zToLqLdtU4Smwh4ne3JsiT9lOzN+4PPw3VSx9l5FppVwdKUWELw1dYpCOppyVWlJ3YQ8H4FQQM8EcYMG9N3Bxu79y1J1ikuvuhMmLQlehLTbguhbix74hd1VIQC8EjHmOZSSWbssulYwPbr6FF49tifk6PymJvulR9/u+2585HkRfbxveGeWCz0ix1pIVfaNpESKmtLy/0mcbMg9hYDz2werz9oe0lT2BiMV6uAin6RaQcT8Vk9MPctfwae+gkvtnZA/sOBk8MbpylaHqc0KIVHhhLFMNnkOFiucjtGo/JWTa/F6g8wWeow5ZuIJUORaYHWqegZbTgM9dCsYYsfZGjjVMuSlDIvpYvIvFFooGPC7Ye2Jfawmq4Ut7EL/nv/dyAd2HRc5msmUhzeu/XpX3rVlzRmf9/Qan8Dbve3QfW1Ym0o5J/KAc3z1VBho7JBr5PgCL68RiD9jZHN0VvsT4gzsEjNlW3D91Uy4RduaodBFoNTzXwlfUYULBzdiTbH75l/UmVMC4TKeTWhNzw2UezaqeGd8at3WSY7W/VR3+hvZHDpkIjgKuNNH0DsCRa/Kk56XQoHIyvvUH/eNekNvziReqS4qgLnXUT4BRGt2BOtCifI6+X/DGHUOmWlX7TN5b4pw5U7jwfwshtbhGZM49T8JMk15Mzrc7tM6J11TYxb5R3mQhZ8TZumJ0bMJXPM69HFyihr5dJSEJMycxJVUh6NTQALUOoRTHIOwE+FpWI6feTv1SiZ0YpYe5DbkYJJbN7zAHbAKw25XvqR2mAjQmOlsfX/tK8DPjP/8h5/xgAF4EUbj1tOnQCBQL8jk9vHtfsXncsprww4Z+P/Z/UrKifuFyEpBWN8kLpF7yywE2iYdDruV9+/qKR8rC9ozNKyqQNIwtxrzYkWpE5t8K7gG4JFnrHona/Rp8dOX6VW41+jb5LB1LEtE8MwjLp3RCUOq/+6yLzaOEgBTqzvEjDeFpg/u9DMHMr4/2TOchfjg7dl+uQ6Gsx+4Ia9W7vivG95027p25eKL0nHvx/OqmAQEZYJL/JO58lOj0zPdJxrQ5dZksjMISzVZNn7DsxqE3zgBBuNzk50R8lTK3U8P12QiOAQYSTeGlYlkvfeofrfO1AitEj02m9aUkxTFd1ZZJoLQT2d3zEU5PmE4lxMVfL5ttNnIbqfcIU2RJKNWqdw77xfjfrNc/eNpRKPZ/6z50LzBprgjzBHRfKgSWWkDxHrX0aTbgwQFwd51+PoUWH4DkQg26uGslF5Hn3hB58+fkeLTosTANOIBNAeFZtTc4PIaLHw759zae7scY55xcTabzlilYIftst2RZ6ntsRC3zFxduCKvL6wLfYT+TiIWJn5P7sTwZwXuSzXY+9Q3xMZ5o4Xcpz6vD9FtTjzS69iefEYt4pXiDrZUo4ePGiLeoIFIwYB/v6GXdmG5VLLk+eKbOc9AmsX2zmvqtcvDRGQbzugXbH/kTH/lkNPBTmqN3ZJODUEXVohPEJ6th0xna0EVleB73Q3eNvaVUvhlJbjs3D/T17FRCebN7AOXvzzbLE/I5kNfEmJcv4dxtIeo2uQ/z9ohSpiZzbDj1u40nJRyJxUK60wEv0nA9f/NuJ6/PEyU0bkK16z2KH12k3Lc4+1f5fawIzkK2qJRB4wnj8VHhUW9mbJhs9vgfFmU3xrXSShY67Ygb+gYNPxxtn4K/9eTSwIA9fv/nR33lA2lZoXALRUTmOZIl3R0gAM5h6oX1y1thIyqViBK95VZc8Pvy7G3O90M9S4zkpyFQ36jrMazvMveMA4d39fvoaC7p90quiJfjI4yrl+ECVkCJL5MxRSa+iVcIL7Xbl0jVaGhZIcMYmcGOBbLzhJgloM1x1zFnnj3ggJRFAM8yNnXxhavk+mA18JC+y3lqGsp6vPReRxGlGHMou17L4It070LzkoeCzarpv8Apw59smdS5KN9qVN1WgeL7OSN8BHg94ubCvS7DW6H3/PbtRB62jFLsBhUV5YqCIbIN5VZ81AAACpUGaIWxFfwAru8x8uT3FuOjrAeSWXmAWqq9jCNGE+N5AOv//9//xjk4uBAcADN96c97AVGmzRtnWwPsgcCbLrVdQJgbKp4QSmPwQnVhv0hXyBjeFWWlcvx70urEN3FK6/lvk2tQeZgbtlbzXluvTfnSj/Ctz7vZ+O1FjhDzzdpL7uLzewzCIW5VWLAEKUVuS2J6wNk6MR7UblcEd4EtOY+R4/qJgfojCsfRvA0oC5dc41Vd0erZbSkrmPTjLCn815bxlchUJMS8gQD5hJNwoKHvNLNwn7XKuTtYIhH2wVNZvDWgzCjlPeQajnrcMsb6bZYJvNJU8HuGHvm50r7VG8qifEwmuyegAZXojh5Ul5VvjDW7kSAZyw8a7I6mHY3FZHd+OA3V4JZMbNliI3Tj1L6+MKTmilVialmyZagRtEMeKRdtxUPd3vVEtrOBVIVYWdgAGA7HmZiHQUQNxLkWxbLyWVlrh5EM0Do2NdbclHxxArz90d+MSVeUOIXQ/4V9quq8C8qVflo1gPtPMkjO2/UrdOYqhY404ReObOu/fdp4hAEDq6jhy64vOeT7XUK/Onq0rXTldtA6kvgQaJg+mgYSR9hfXtMbOUSLgLj/RmBSO8aAMHuJJZqf1tCM5pZ9eYUsrHmy+/z2NGalon0//uF6+33bQzT/RLRfBbYTjy9QrJqHLlw46lggWPGkHuPKSqk/CB7U4pNPXUbR0DdcJy9Db00wCzVzxVc6h7jfCFgiL2Y0HVqd6bgIaVUqn/gJCEyCDVplnzebv0gg3XwMJAGu639lHu7rEvxTp1smIYjWp9R5L4SspVvS07Nb+Smk1FgsMp1K3EMUT8X2Fty4VG54/Ec6bE8tNVw4/QV1VzBw7Px2/2eEhhUS+FMfbHAlD28x00jRgAAACW0GaQjwhkymEVwArOUkEOhoFqiELtH8wgecFLiUq6WqmwAP7iGEwbYzfnHacfqUNXAfD+CGR2ap0lAHL25ipuYtd5j2O0PU/MpaWPG/n2y5OkfTzaOpotaR5tWjN55B2XblVVqsFfBC/mvsiPvCBWUHFChacdY5whj5mP5rqQ0dqLJCsWjrs4TWnIbL2V/Iwfj3hwI35jfo1JkTOeR+8GhOdma9rgiKWafCbQyhYMTDmVdvhND60Flm97EDSTjF0OC+0gD9b8Yn4tNeHipCa/aWyt0n79bMmjfcjntBCPjrcB5ecRTpfGHbEHy1IRj2cjkGXKC+VYoYJXBp4rd4cMd8ygLCk5nBSd8/cTaKNRjdBscOeTXG6QEjSxj9/2pVwx9DMRVtWQR0BSaAcQcZ8W2KPSaeRC4QwmNMu2xx25CSyrDiq2rFSK/JJtmvoIjAKq0ciEXoOgw+Ke+Ylb7ULKCS3k1p/613UNRp450uSq5b7CAHo7S0b7fBMLfNmwSjRYEhLlo0HUaRe/I+IX2Z6XdZH9Hty/399ZA1PwZGC6EfvUJIf7CBeaxv7cu6IT2/s0zPRGthpvXpYw6A7P4Wwz5C4V98KnIUNUanadqabKP6eXWhvbvcQHxAjiOOiKZgXZplZW2g+B2NNyJSLiR+g48DqvWR6t9S2aGfFjdOW1Gi6oTtZ1d4p5XIslAr8mryeZ6+htSSQe4AcfVt7k+V6mOthBCYtr/LEU4ZHtl0mW9876PK8mRFAaT8DJOUFVz1lPfzRApuPggkkyq+UMvyfKTUbCk7/DpfX8Y4s4QAAAg9BmmNJ4Q8mUwIr/wAsWUPjZw3ksgRsxZ6n4fQjprPbkj2aUh30y0bZJnLmiXnWskvOGnCPwBnG9dEhatwX3hoxk7BNyG+wQ4emZUpcVzcWl2T9nKQB1euucuZWHTg7TCtM/iHyfPO2vbmGsfzs70b/egIbywUH4y4BQSL1nWc1SmpHm2zHMBcUjYLDZ5gL5vdfxn0V8FFw66G88c/LN4I5icUa7xf4fcSBKywU0ajbp1P+aJYjBgWT6Ggu0MDLDNl54tfqd42lKosQtM1aif4WXAZFP5Ww3vrQ1rH9+utSYxqZd6N6gGtNbSNMcViaKn5LcnjsbBi3T3EmGqshEbcme8VHKwR3kSfBOAprrIsv6K8R+X6az+MD23rWka/2v64m1qM69D7Xa+Kcs/n0KLCJdTilyaGadopLeaAn3eYvWTeHcucMM1Fp1KgHD1tiFeO6HvobLkZlRximsA3/7MiohYklLIcJrZL22BH+6W9d6kZsYIsej9RM681nU6mWNjepBAfAfTbrGRrVB/h2DxC5B8YyRjgSIzQjNYrse0rzChqbrsLl7mQ7W+1bsNKze5//9ZIa8rSsF+BXh/vgoRTDkPW/ws95B7VPCZEFChfX0icw+tpcpN/q7NY87tUn4vESdSiMMlyhKklMjQu/G51J69ZRQLs2oUO6YfoJFqliy4qCFCrf8SZE9Fc6DcCagAAAAodBmoRJ4Q8mUwIr/wArPWF/KOw78THwadfPqhJO0CnmR/M74/XYZLqVYKlNcEaYauf+vrRUDJPmu75sMKy2Y+Bnslc/iAISSyWtw/h/3CF8fE5ZrbrwSNst+MSyCoNWP+8imtoX2eyojpdCk8YP5K+cbK4SJPCkZXbYqSXYk7hO8AdSemBHgXKWiZ+UOr802aJo+98ZOIjX9hWL9bo31Gqx7cy4ZG+W/ar/WGlzDa1xPWnPRsEdrIcZlEVGV/jGmbirkxw1lyUYoqj8Vv7Bxube9XPQlBkXOV6Lc1LT2IzNq0V7WwVhF0kA6yxfAsFxc9krNEH8vGGntTWI608ovjatXc/CKKXw7AjJSftlTcLI0hIIGXbRUr0NCYNp7M4cVd/n73Rjetnixz4SAKpcz/P47UsijZG7T3SxzK2D79WS42aEalc12hQwCZ01LfmF/H2mmGEvOzPBie1D0YT7Jh19vxa4Dd3SQ1FrDfmSUpvv4DjbYcZ2PrPpFpWtMjWqHBeoyMiZf6RP3EfYR6z9jsVNIIHxM0bzzBQF8eeYkPgDySydxPXv9Izo+QUY94N8kWi16fI6eZSDc1G0Yo0L91jcRQuDMGGS7B2zuf/0GbJyRhUO48UbMrqnILMrbQg1LF00Q3pH9nbGEK/RRQpRN3T/J/4IZQjwW2Ft2ipWGztg1Jn9I4DmffKS60QC+JQcyakdVON6zDcKttIKlqeTcmAi4xzmo4QXa2dRKleS+fs3EtTdBBtony2wK9T2Imj+NCziOSEL7Q7VuIU8kclUHrJJsSneFcxGRgIgGGUEQM8/pklwTOqab7mMmJeRiaBrjJDEnDpkR4Vz3qXxgyn4/5x24FuTMNVPwQAAAhtBmqVJ4Q8mUwIr/wApcLwPT0/Xh9UdWqWXIs8Wbj5K1hivmN6qIQnq+aolcegdlM/63MbHsdC6xYZC1e/Q8UjQCt9N/Ejqwms8DzeWv2qxskeliZH0kt1QWkErWSEodq7V0ZNksctLkMGWayX33gBT368EehfIeGDolBZoqIbJfb4nqcfU+ev4OzVv9zVqWyLck315GFmXxQKIM8pICQc8Q5es34LH1+DmnMnW8kQpVGrztQcDXhjCU3F0fOgoSsXSVWCjc6XKqGbCwQDfJUxCfXfIT6YmQoPpVp1mpGy1wQypXus9z0bScDpyDu23hViYDntdj1O45ea0znKZkj1+tLHbBtqAGJ1WTcbGlF6Vya6hQhEsiiZUIC2fRxIj8/wEXCICIbr0gZ/m6gcOhE10tenvE7iy+BKY81wLWrnzos3S6FWxYtmCRes+LLhNGOKWRuQo6SyePH2OZ90xZm8oA1MuTe3V59euVNxjAt0FLkAc9TEiFhP/8CB+gA8mF+A8h1U01f4DVX55GzCH51jHI2xUS0L9GtsHoBxLPLK/NNel8zcnwG4X+UusfcfEb5hh+ffnXteCE9vRGbs2n9wYW0xA3ZicklfadmWKUtMiHYBfkMSULWnkBQr4CXxjpYOs6ygeEoA5+5B0B1SZObgZ42wWqddyyYE0NfwQAl75tfdJGqOa7OMHwBYNeatJaJK0zT2+bFaw2qWCWwAAAitBmsZJ4Q8mUwIr/wAstkdsayRXchoFk703izqzduZ5WsyXriI9cfUdMUWvm0iGHwYIrUujvz3Yjou+JLwv9df2kt7MJo8u+3P5CjEKbwlz4vkE5AHTAbgXn3+Xc/MMJLgW5cm7iX3KiGNnBpbphhwJRlb3u91NRDr0d1IR2up/z7lKxE7XPAPFe0siPMYVlIqWNSn5KqLABPeuxxbOsvMEb27/nH1LUVM8I2F95c1I3Lv1SpkhZXjs1JsmS9X7gsoTxkXyShGC2+zRJSGUbhCPo/q1XSFMHQyMWJ79FKPQSL/RpVsacN2bYwdKo4TFBw1SsKq/L1iOmqMI+4Gxnbbjojdk0ek0JIcDb4bHv1czxchF7FX1Ym8H6IpPuE8CeNKjzQ1a1wqhEu+wl1N0x3Y37ZryCCKJRkxj0FT7bOoH3L38/yMUuh/v3aCmxY4eCkykb2p6ZrYMFE044anM/nMjmbErMibfRFuCz58Io1rBlF7JfkIz0R2/5vjUMVskcdbX2mm7DntncOsWDIdg/XVmgsC9CzVzUyq4VsS/sk97lJggcddpWLNw/29egz8iLyzWHOAXCvl2fTIPkviYAOQXfVhZUQdxsyJUNFMTiALrZCmoQLMp2LmDbfbW8JQriDeR3fVz6P1sjT8C2yEDvzkCn7sh0aTBK+sx7BKH1nb4320+caQepQj4TCJtCeNXjdrVcNEnjvwlcRJwFT1pT+Y7HREbHnT71XYNh4EAAAGEQZrnSeEPJlMCK/8AKIjxcI58rm/ML255fOJW1zbznFna7lfgMQrka7OTPPsvVAV4EJXye/Uxiu9dlftmRypJqfDot3xwDe8lX/qAVf6pBkSlUsaLyBYtww/SUSa1bGl1JvrJCN7FXCCXbLd5R4PoYlPiDIm/DQH2puO0StIWmrR77Isc/J1pRvdu5+mQa/n0SEHUeM2KkoRzCznfD9zaaRO7BDtvC9SYIT0uYZxrwTjxQ7N7UERTrYG0P+vRLAhxkfohFIYl3HXyjPOvnlbUFP2oiiy6nkUFuaIyQcJawJv3GU8k4ObcKsC1cNDXjSpsyQRrxLFaCCjke4mikyt7vs0iN0bnrNWv9HXruG9zOFEOer1ggIFTsT1Eos5CXRkgja5HN4QUM6MhWpc5du/HgBIH8ANFcoo2kJpqcadw9r/0qk25X91MQSDJQiH8Hny2dQhqR+LFWEawiW753SJhn0ngZcv/mPj3mwcHv1SL9ErBqAjm4JGiDetPKYtFwANYY11OyQAAAVdBmwhJ4Q8mUwIr/wArOx5HV2505jRePGgMxptW4PGIHEszV1xGZS+flSkF+aq30AaqO7u6XK9jJsuWXTfYCRQTn1bZfFQ22DbO5DXAxK/TUmbQleCflFzeS6/czxkL4PJ8AwOs2U+oehekgCZC8gZyHHaQSaKbNJ46gTjNsLy84ACQ5uNt11TPuCPqPTuh+schdw9S+/lU/6m+EyaqGZ49wDFPiBFBYXglQQBjyP9k/rqq0xL7SiLjpe4riYg8SFUuUtOzPdWHyvxnI7Ug/0VLPGAAhgMISUnqe01d5QFf36yHpwMAHexjAZFIGQHAFautuMuEw6HzUZVzNdeHYxvEYOGkTo007bLwbuf/nxzrywGOxlRTYJLRdYI0mk0SdN3+LeTv1RIJwv21+e9rT5iFOTCgzeQoekEWXLYz0X8YLq5bVCtijP7/T7w1Ck71j0aqfrEn6wtIAAABNUGbKUnhDyZTAiv/ACcySi7VBgOid6qZNXvhh/JsllHkMLLq0yNbQTqv/Wk2EBoSKICZwFwAD0WRzhvvReCGirep1Fe4bxjm49/UR+OYrXRmHR18T0C83AUVeBk7KvDZmb/eHzuzEN4yfXucr/NWFJl+USVMY4r4UQ9CayrfEY9v6AQ6mzAdLy2UMfFxrRJ99g/Rfl8qx+m4jIZNjlrTaThzJ/3OpVmAliDfxVyg8+CVIlI31IykiwQrXcebgajG+av8XU1SfyAG5ibvwbtdSAxkGBcJWL387V+uTdY56w3KN2vBtoQpVKD2zb3yazIcATZ02upwIytNcM/rpaLCdMb1myWcikE25agzLhDhOS+4zwjYz2DnW6VY0gFBAPsphhsUMnauVVdUVHzCTSdvzEve/H8q4AAAAVdBm0pJ4Q8mUwIr/wAo+x5XKuiN1am7SkJKSMonFZDPU3f5XFcDQSs0FLVq2idfsKwuIkt1mxIq8NgMHpzofTnDHqs/WedvAmhBgL0N5azdQa5MNKG2rJ4IAvGQY/uFm3jKQAKzvhSS01gO1oIfizF817z9IShS4QK2WT0PeFPELqLSpED8eNOpVTR96vmwpk/WBKRVJdTQJzjiCQ5pgEwjtvk7KqoS0+lwXSbvIrXkYm8DignEts3DLNoLHrPjXlQmbIop76JZSyJEtB+91GrLwo6Km5GeebyA2E6qGL3xSkpppej/ruoFprSKrH60UMbrq/SK7eCo+1QFoySPQmqDFsMGiQFqvtld5BXDYdVI4yRaoyN7Y7wi83HRC6eVazuHU9OtIY3xJJApBWq1aJOsYwc38aTC3ee863Aa/4n9Lk4DAtyFYHNZjB5m2e2vk8G2Gny9YFlBAAABQEGba0nhDyZTAiv/ACoZSZQfHxhfQxEqOBQrP+L3DmgvHSJQtB1iVkcLTxm+vagLHBLG91OGnopwrr7gT/loDypIhoRxjcwAAOeg/jN4WBbXzCJtnWGGllUCSdtUZQzKOSp9iM4yX18C6jrY4Sq6R9PUV/lEGNveJR4gw4FMve7110XdEPL1O2VTdHvdqeANyaq0nLdEmtXnrzvdrFlBaUvmaR4EdlkqGkvkZKWJej8Vq+msbKa7JdbxjwZtRufiyGfD/NVqMgSrYRzw9z/a8Zwbr+9+19CxlWD5bCuAEfPmjY6kZJE2L/CQI6+tnCBTXOmWZtZMBoCLGOf7G2uAC3+kFlboh9as5WCkO6+iqXq29dyhKnsHInorRYsPlgxIXyU1Om/Kyhj1DJV0Am9WJK3Dln0zNUH0q6ZTOnZcFD36AAABYkGbjEnhDyZTAiv/ACcwdIOLRFfoGK2ZkKsvgMwG0m0qsY0vMLPSzefc+ebp/aztyF7MlsBz/fBeNtxFBcsKgR4pf65GvdfOMHah0ltZ918sMDmXUEZMeRHy/xpnWpTLeGz6uTs/7MATPmU5BgHbT/DkD8QeaZnFAzidyFCXDz2l/jaKhEdgqipbB2pH0+fQ039r05z9axxEWGmaLQjg6x9+po1o24yhkVO7m03YwWmPyCgy8cOwrvRyJkXJpRN4m8ZBS1zwY80HeN/VyMQQJSMwsTo7R1XMerSFuyx0nz+8qOuhiqykc2ohCCsXia/+kIKbJ5Vs+cbWtvkqBKIDSfU7FhAd3GjcY/xar0EVmi6wWFTugAogR3I7mTrQDdlTAqYgqO7Gn5NMXQVHu2i1zhFSdo9GjMbeGnbkJwsFbQ2XkoKRIDpuW7AewC9AEBt0Ox/Ah6dGXfXO1jl8pEApj2RFmgAAAPlBm61J4Q8mUwIr/wAlR+eW/VZ7bSrmwwMA62G05DZ7p/5FUugsSsQdonUq6abtbU5hjFr+I1lPgoiV5c3CkTQZS+K5zivdo+Ti2P4K90xXANp8dSMAu85uJIOCQn2TXbEnNDifLB+3V84ht5tj4lvTaZx317BcliV8D5v2zZQW8RO1mUbuJEBItst8E7hfE+ZXj7tfDxNZPTvtpFyUv0fH1cTg1pr2VLy0d0zQLiA58dg+GkRvR1/hs2LyifBgHcj6eTWz0vsypVn9iPXRH/unJ6i8cfFL69NO24tQ9QQB+nDFhoP2cRhkAvhHwn56n5PppBD/oxni2f8AAAE9QZvOSeEPJlMCK/8AJjAXVGf+Kj2XNJnFeKC/gr7dJDTC2ngpd4WeAHlg04GuJKnn9hAmiECxxo9qM1IYMRiB85t6gALnlm9sRqGmioyzAm18RJndc9Ah8RlpGzr+44a6ntRaPx0cIwNIWAA8buL2JP00dmfjNqEiAlCa8OdV8FQxjp1vDXsGcAGF3Qbd62KEpkimeI3wH2nuXpbDHm8/ZKOR49s5ifUCkxCoJpfp43aC0lTzh2NXpcfVw6h0QnK8G60R4ZAxOxaJB7c0nn8ixXSU2JVY24EtGMF53nxJnHfzUheewUfBOGYSxeo8oK7oUCqX4rztzDwoc2QywNqQUJUkFrqIN+sb5ecYvX24Zujn+ZzTW6UDAF3R6WdNyJyRAremgC8spSflTqygQNGfHyGkfIEEJJaFo/pBCBkAAAEWQZvvSeEPJlMCK/8AKI41fuekXG59Knbw4Y6YJritsh9VtQgc3QKvVmxrzzo7f4aXn8N74eyP4b2lV1Z2Q+rohxps7EHTkOY9jLdqxI3MXe7je4g2qepz71+hY+jYdX+9LO0kA0Zg3NfyAlIRX7k6c/YHAZNtNaGZgTBMqiPgmEjiJH9Luk7shbgr+srfwiYwBX9rdS3fQNNFwcT8orQC+F60LAY9+GbFo2Sw3Ld4Tw9jq9yJtrY8RtHAdzytyek/mv2+j2TbTvAQKbbCYtdC8E/KtR4V5ZTSScr5Wb63vmbw7UpddEXYvl55pARyyvMxWNSh3Li4GF8Jk5JBi5B5ASQwxCMYpX5hkAMc+d8tl2bT+IEvUTsAAAElQZoQSeEPJlMCK/8AJIAzFZs00JJ0yfm8CZiew4xWdArLklEvBVXo/+ukPLu3XP9HFOfsme3T6BJEKmPPgZw/Lxnraq6Sl2kLVW19YU1qmqgfv+80LkZaWU5gRAH4hqyo3bFYcbuY2SC3IW5Wm69gtYyAXOdbAYSEHA16fvCeRQjHEsxKVndJdrRAlrGHsKgUBQ3Up/ZXIy1vkdFOfKSjpuZnswkuqr8NZI5tJ/dnBSErBTNWPaNwWV7nNomC0EYVGo+geGBhLXzaLw0UAOCYGjiPc3803BDw1GLoLIXjrIFJxwRfBNIAXYZAglu30oYzhpAfRWSprkeULMWYJTlWvbUQ5CNewSZssuDWIRAc3w8AcFaywwn+YSGhtR8VI1OGjYkfBbcAAAD8QZoxSeEPJlMCK/8AJdokjCUETRw/nciVPtaZQSBP/VxAQSITASEzlJBl9Na1r0DJhLOz279+KQLtl/xHZ8vAKc528mTMTqtWs4sFbeVgHWyBpHcHEtgTzjIqEinp/MPuUXF5poo8YLSSMFn9Ozx2FbU5/Kh9A39oN9NHQflVxV1NA6yT/84HHyfMtfdSMS8KTvAEE2lDs14VQayNs5ctjXboQT7xMBf5OLj6thhPvgaDrFB2o/PV9ouK147lruWTP2mkoA9oDIMYW1pcBx4yyV/t9GOPZ3aXneMUb2fFmUCX43BjXfUDMaa4GO2/Ankj3UEQwDxA7ZlNUQK2AAAA4UGaUknhDyZTAiv/ACJHv33I08bkhybYiJ/JiiheW5zMPBu4n5CxGr3frhE7TkLh0vPktM8m/AhaDiJisdk5QXNe/4WmxEDSAyaVi4eUVu0iHT2ly/KNTGqiORqA2oKpTjh84nYbrpXwnGv9SOf/34Z06xN6Yo3t35UZrP8nlcs/63GtnEmnUwVZHBYfPM6bs5M5AeBfAQ/9mIqu7vnEst+5O2wpPjzdItjwGCZ2ApHVjGnYYFomlA9nm6AXnxNIWHIsDgxCk3zx+6QbXipu/CWLG1Wf0WIbt4C0JPVl3TEb0QAAAMlBmnNJ4Q8mUwIr/wAVV64OfTKmlktYOqZHH1W1DhPy/X/6sD4T6hRdzfOgNtTOX2IckRJHshfBQVkJIzns079io6kpJFCcS3VD4zrWCn/dNaGV0kWTpFBRuusfn8F0C0R/EhsQeyTsdZftEkLGb5tq+nrir3vfmeb7rjmWJRXkIrTEKu8pIuAd+4FBGp8ARgGe80Jqpp//s1433HqBFqXsIFJTmU8j/toF9HyueI1Ea4uvsQ6NANGcYCbOAKCmbNiwABMCFaiUTMAAAAPSbW9vdgAAAGxtdmhkAAAAAHwlsIB8JbCAAAAD6AAAAyAAAQAAAQAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAgAAAv10cmFrAAAAXHRraGQAAAAPfCWwgHwlsIAAAAABAAAAAAAAAyAAAAAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAABAAAAAAY4AAAGGAAAAAAAkZWR0cwAAABxlbHN0AAAAAAAAAAEAAAMgAAAAAgABAAAAAAJ1bWRpYQAAACBtZGhkAAAAAHwlsIB8JbCAAAAAGQAAABRVxAAAAAAALWhkbHIAAAAAAAAAAHZpZGUAAAAAAAAAAAAAAABWaWRlb0hhbmRsZXIAAAACIG1pbmYAAAAUdm1oZAAAAAEAAAAAAAAAAAAAACRkaW5mAAAAHGRyZWYAAAAAAAAAAQAAAAx1cmwgAAAAAQAAAeBzdGJsAAAAtHN0c2QAAAAAAAAAAQAAAKRhdmMxAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAY4BhgBIAAAASAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAGP//AAAAMmF2Y0MBZAAV/+EAGWdkABWs2UGQz6mhAAADAAEAAAMAMg8WLZYBAAZo6+PLIsAAAAAcdXVpZGtoQPJfJE/FujmlG88DI/MAAAAAAAAAGHN0dHMAAAAAAAAAAQAAABQAAAABAAAAFHN0c3MAAAAAAAAAAQAAAAEAAAAYY3R0cwAAAAAAAAABAAAAFAAAAAIAAAAcc3RzYwAAAAAAAAABAAAAAQAAAAEAAAABAAAAZHN0c3oAAAAAAAAAAAAAABQAAA05AAACqQAAAl8AAAITAAACiwAAAh8AAAIvAAABiAAAAVsAAAE5AAABWwAAAUQAAAFmAAAA/QAAAUEAAAEaAAABKQAAAQAAAADlAAAAzQAAAGBzdGNvAAAAAAAAABQAAAAsAAANZQAAEA4AABJtAAAUgAAAFwsAABkqAAAbWQAAHOEAAB48AAAfdQAAINAAACIUAAAjegAAJHcAACW4AAAm0gAAJ/sAACj7AAAp4AAAAGF1ZHRhAAAAWW1ldGEAAAAAAAAAIWhkbHIAAAAAAAAAAG1kaXJhcHBsAAAAAAAAAAAAAAAALGlsc3QAAAAkqXRvbwAAABxkYXRhAAAAAQAAAABMYXZmNTIuMTExLjA=">

## External sites

You can even embed an entire page from another site in an iframe; for example this is today's Wikipedia
page for mobile users:

```python
from IPython.display import IFrame
IFrame('http://jupyter.org', width='100%', height=350)
```

## Links to local files

IPython provides builtin display classes for generating links to local files. Create a link to a single file using the `FileLink` object:

```python
from IPython.display import FileLink, FileLinks
FileLink('Advanced IPython.ipynb')
```

Advanced IPython.ipynb

Alternatively, to generate links to all of the files in a directory, use the `FileLinks` object, passing `'.'` to indicate that we want links generated for the current working directory. Note that if there were other directories under the current directory, `FileLinks` would work in a recursive manner creating links to files in all sub-directories as well.

```python
FileLinks('.')
```

./

  Advanced IPython.ipynb

  animation.m4v

  Custom Display Logic.ipynb

  Image Processing.ipynb

  ipython_logo.png

  Multiple languages from inside IPython.ipynb

  mycircle.py

  mycircle2.py

  python_logo.svg

  Rich Output.ipynb

  Using Interact.ipynb

./.ipynb_checkpoints/

  Advanced IPython-checkpoint.ipynb

  Custom Display Logic-checkpoint.ipynb

  Image Processing-checkpoint.ipynb

  IntroParallel-checkpoint.ipynb

  Multiple languages from inside IPython-checkpoint.ipynb

  Rich Output-checkpoint.ipynb

  Using Interact-checkpoint.ipynb

## Rich output and security

The IPython Notebook allows arbitrary code execution in both the IPython kernel and in the browser, though HTML and JavaScript output. More importantly, because IPython has a JavaScript API for running code in the browser, HTML and JavaScript output can actually trigger code to be run in the kernel. This poses a significant security risk as it would allow IPython Notebooks to execute arbitrary code on your computers.

To protect against these risks, the IPython Notebook has a security model that specifies how dangerous output is handled. Here is a short summary:

* When you run code in the Notebook, all rich output is displayed.
* When you open a notebook, rich output is only displayed if it doesn't contain security vulberabilities,...
* ..or if you have trusted a notebook, all rich output will run upon opening it.

A full description of the IPython security model can be found on [this page](http://ipython.org/ipython-doc/dev/notebook/security.html).

## Rich output and nbviewer

Much of the power of the Notebook is that it enables users to share notebooks with each other using http://nbviewer.ipython.org, without installing IPython locally. As of IPython 2.0, notebooks rendere on nbviewer will display all output, including HTML and JavaScript. Furthermore, to provide a consistent JavaScript environment on the live Notebook and nbviewer, the following JavaScript libraries are loaded onto the nbviewer page, *before* the notebook and its output is displayed:

* [jQuery](http://jquery.com/)
* [RequireJS](http://requirejs.org/)

Libraries such as [mpld3](http://mpld3.github.io/) use these capabilities to generate interactive visualizations that work on nbviewer.


Jupyter Notebook Overview for NISAR ST

 Nov 4, 2019

Overview

1. Quick Introduction to Jupyter
   - Markdown text
   - Executable code
   - Latex equations
   - Embedding images
2. Interactive Plotting examples

Intro to Jupyter (1)

1. Installation on OS X (10.12)
   - sudo port install py37-jupyter
   - sudo pip-3.7 install jupyterlab
   - sudo pip-3.7 install jupyter_contrib_nbextensions
   - sudo jupyter-3.7 contrib nbextension install --user
   - sudo pip-3.7 install jupyter_nbextensions_configurator
   - sudo jupyter-3.7 nbextensions_configurator enable --user
   - sudo pip-3.7 install RISE
   - sudo jupyter-nbextension-3.7 install rise --py --sys-prefix

Intro to Jupyter (2)

1. Jupyter comes pre-packaged with conda (OS-X / Linux)
   - python3 -m pip install jupyterlab
   - python3 -m pip install jupyter_contrib_nbextensions
   - python3 -m pip install jupyter_nbextensions_configurator
   - conda install -c conda-forge RISE

Jupyter interfaces (3)

There are 3 different Jupyter environments that you can work with

1. [Jupyter Lab](http://localhost:8888/lab)

2. [Jupyter notebook](http://localhost:8888/tree)

3. Jupyter nteract (Not shown today)

Adding content to a notebook

1. Text
2. Executable code
3. Latex equations
4. Embedding images in notebook

Text (1)

Simple markdown (same as github) is used for writing text in jupyter.

A good reference cheat sheet can be found [here](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet).

# "#" is used for Main title (h1)
## "##" is used for Section title (h2)
### "###" is used for Subsection title (h3)

Text (2)

Emphasis, aka italics, with *asterisks* or _underscores_.

Strong emphasis, aka bold, with double **asterisks** or __underscores__.

Combined emphasis with **asterisks and _underscores_**.

Strikethrough uses two tildes. ~~Scratch this.~~

 Text (3)

1. Start lines with numbers for ordered lists.

* Start lines with asterisk/ minus / plus for unordered lists.
   - Use 3 spaces to write sub-bullets

Text (4)

Code-snippets are included with triple back-ticks like this

```python
import numpy as np
x = np.arange(10)
```

```bash
> ls -ltr
```

Executable code (1)

Jupyter can be used in combination with various languages (kernels). A full list can be found [here](https://github.com/jupyter/jupyter/wiki/Jupyter-kernels)

The most common kernels are
1. __Python__
2. Julia
3. R
4. Haskell
5. Javascript
6. Ruby

Executable code (2)

* Notebook = One long python script broken into parts + user-content

* Variables created in one cell are available for use in the next cells
  - This means that the cells need to be run in sequence in general

* Python syntax and normal coding rules apply

* **Note:** Don't carry around large arrays between cells

Executable code (3)

Here is a simple demo

```python
import numpy as np

x = np.arange(100)
```

In the previous cell, we created an array going from $\left [ 0,99\right ]$. We will now compute its mean and standard deviation in a separate cell.

```python
print('Mean: {0}, Std: {1}'.format(np.mean(x), np.std(x)))

```

Executable code (4)

One can also run system commands from within the notebook. Use an exclamation mark in front of your commands.

```python
!echo "Printing GDAL version:"
!gdalinfo --version
```

Executable code (5)

You can also pass python variables to shell commands - kind of useful when tying together components in a workflow.

```python
##Pythonic way of searching for files
import glob
nbname = glob.glob('./Slides*.ipynb')[0]
```

We will now pass the variable to a shell command directly.

```python
##Programatically
## os.system('ls -ltr {0}'.format(nbname))
!ls -ltr {nbname}

```

Latex math (1)

Standard latex equations should work inline like this $\left(a + b\right)^2 \neq a^2 + b^2$.

One can also write elaborate equations like this

$\begin{align}
\dot{x} & = \sigma(y-x) \\
\dot{y} & = \rho x - y - xz \\
\dot{z} & = -\beta z + xy
\end{align}$

Latex math (2)

Matrices can also be written out using amsmath formatting

\matrix yields $\begin{matrix} a & b \\ c & d \end{matrix}$

\bmatrix yields $\begin{bmatrix} a & b \\ c & d \end{bmatrix}$

\pmatrix yields $\begin{pmatrix} a & b \\ c & d \end{pmatrix}$

Embedding images (1)

Images can be embedded in 2 different ways:

1. Directly using markdown
   - Not much control over sizing
   - Simple syntax

2. Using HTML
   - Allows explicit control over sizing

Embedding images (2)

Direct embedding looks like this

Embedding images (3)

Embedded image with HTML looks like this

 Interactive plotting for teaching

Jupyter supports interactive plotting and this can be very useful as a teaching tool. Here are some simple examples that can be built upon:

1. [Simple InSAR baseline simulator](../InSARBaseline/InSARBaselineSimulator.ipynb)

2. [Impact of DEM error on geolocation](../DEMAndGeocoding/demAndGeocoding.ipynb)


##### Copyright 2020 Google

```
#@title Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
```

# Data collection

    View on QuantumAI

    Run in Google Colab

    View source on GitHub

    Download notebook

Following a set of idioms and using common utilities when running NISQy quantum
experiments is advantageous to:

 - Avoid duplication of effort for common tasks like data saving and loading
 - Enable easy data sharing
 - Reduce cognitive load of onboarding onto a new experiment. The 'science'
   part is isolated from an idiomatic 'infrastructure' part.
 - Idioms and conventions are more flexible than a strict framework. You
   don't need to do everything exactly.

This notebook shows how to design the infrastructure to support a simple experiment.

## Setup

Install the ReCirq package:

```
try:
    import recirq
except ImportError:
    !pip install --quiet git+https://github.com/quantumlib/ReCirq
```

Now import Cirq, ReCirq and the module dependencies:

```
import os

import numpy as np
import sympy

import cirq
import recirq
```

## Tasks

We organize our experiments around the concept of "tasks". A task is a unit of work which consists of loading in input data, doing data processing or data collection, and saving results. Dividing your pipeline into tasks can be more of an art than a science. However, some rules of thumb can be observed:

 1. A task should be at least 30 seconds worth of work but less than ten minutes worth of work. Finer division of tasks can make your pipelines more composable, more resistant to failure, easier to restart from failure, and easier to parallelize. Coarser division of tasks can amortize the cost of input and output data serialization and deserialization.

 2. A task should be completely determined by a small-to-medium collection of primitive data type parameters. In fact, these parameters will represent instances of tasks and will act as "keys" in a database or on the filesystem.

Practically, a task consists of a `TasknameTask` (use your own name!) dataclass and a function which takes an instance of such a class as its argument, does the requisite data processing, and saves its results. Here, we define the `ReadoutScanTask` class with members that tell us exactly what data we want to collect.

```
@recirq.json_serializable_dataclass(namespace='recirq.readout_scan',
                                    registry=recirq.Registry,
                                    frozen=True)
class ReadoutScanTask:
    """Scan over Ry(theta) angles from -pi/2 to 3pi/2 tracing out a sinusoid
    which is primarily affected by readout error.

    See Also:
        :py:func:`run_readout_scan`

    Attributes:
        dataset_id: A unique identifier for this dataset.
        device_name: The device to run on, by name.
        n_shots: The number of repetitions for each theta value.
        qubit: The qubit to benchmark.
        resolution_factor: We select the number of points in the linspace
            so that the special points: (-1/2, 0, 1/2, 1, 3/2) * pi are
            always included. The total number of theta evaluations
            is resolution_factor * 4 + 1.
    """
    dataset_id: str
    device_name: str
    n_shots: int
    qubit: cirq.GridQubit
    resolution_factor: int

    @property
    def fn(self):
        n_shots = _abbrev_n_shots(n_shots=self.n_shots)
        qubit = _abbrev_grid_qubit(self.qubit)
        return (f'{self.dataset_id}/'
                f'{self.device_name}/'
                f'q-{qubit}/'
                f'ry_scan_{self.resolution_factor}_{n_shots}')

# Define the following helper functions to make nicer `fn` keys
# for the tasks:

def _abbrev_n_shots(n_shots: int) -> str:
    """Shorter n_shots component of a filename"""
    if n_shots % 1000 == 0:
        return f'{n_shots // 1000}k'
    return str(n_shots)

def _abbrev_grid_qubit(qubit: cirq.GridQubit) -> str:
    """Formatted grid_qubit component of a filename"""
    return f'{qubit.row}_{qubit.col}'
```

There are some things worth noting with this TasknameTask class.

1. We use the utility annotation `@json_serializable_dataclass`, which wraps the vanilla `@dataclass` annotation, except it permits saving and loading instances of `ReadoutScanTask` using Cirq's JSON serialization facilities. We give it an appropriate namespace to distinguish between top-level `cirq` objects.

2. Data members are all primitive or near-primitive data types: `str`, `int`, `GridQubit`. This sets us up well to use `ReadoutScanTask` in a variety of contexts where it may be tricky to use too-abstract data types. First, these simple members allow us to map from a task object to a unique `/`-delimited string appropriate for use as a filename or a unique key. Second, these parameters are immediately suitable to serve as columns in a `pd.DataFrame` or a database table.

3. There is a property named `fn` which provides a mapping from `ReadoutScanTask` instances to strings suitable for use as filenames. In fact, we will use this to save per-task data. Note that every dataclass member variable is used in the construction of `fn`. We also define some utility methods to make more human-readable strings. There must be a 1:1 mapping from task attributes to filenames. In general it is easy to go from a Task object to a filename. It should be possible to go the other way, although filenames prioritize readability over parsability; so in general this relationship won’t be used.

4. We begin with a `dataset_id` field. Remember, instances of `ReadoutScanTask` must completely capture a task. We may want to run the same qubit for the same number of shots on the same device on two different days, so we include `dataset_id` to capture the notion of time and/or the state of the universe for tasks. Each family of tasks should include `dataset_id` as its first parameter.

## Namespacing

A collection of tasks can be grouped into an "experiment" with a particular name.
This defines a folder `~/cirq-results/[experiment_name]/` under which data will be stored.
If you were storing data in a database, this might be the table name.
The second level of namespacing comes from tasks' `dataset_id` field which groups together an immutable collection of results taken at roughly the same time.

By convention, you can define the following global variables in your experiment scripts:

```
EXPERIMENT_NAME = 'readout-scan'
DEFAULT_BASE_DIR = os.path.expanduser(f'~/cirq-results/{EXPERIMENT_NAME}')
```

All of the I/O functions take a `base_dir` parameter to support full control
over where things are saved / loaded. Your script will use `DEFAULT_BASE_DIR`.

Typically, data collection (i.e. the code in this notebook) would be in a script so you can run it headless for a long time. Typically, analysis is done in one or more notebooks because of their ability to display rich output. By saving data correctly, your analysis and plotting code can run fast and interactively.

## Running a Task

Each task is comprised not only of the Task object, but also a function that executes the task. For example, here we define the process by which we collect data.

 - There should only be one required argument: `task` whose type is the class defined to completely specify the parameters of a task. Why define a separate class instead of just using normal function arguments?
  - Remember this class has a `fn` property that gives a unique string for parameters. If there were more arguments to this function, there would be inputs not specified in `fn` and the data output path could be ambiguous.
  - By putting the arguments in a class, they can easily be serialized as metadata alongside the output of the task.
 - The behavior of the function must be completely determined by its inputs.
  - This is why we put a `dataset_id` field in each task that's usually something resembling a timestamp. It captures the 'state of the world' as an input.
  - It's recommended that you add a check to the beginning of each task function to check if the output file already exists. If it does and the output is completely determined by its inputs, then we can deduce that the task is already done. This can save time for expensive classical pre-computations or it can be used to re-start a collection of tasks where only some of them had completed.
 - In general, you have freedom to implement your own logic in these functions, especially between the beginning (which is code for loading in input data) and the end (which is always a call to `recirq.save()`). Don't go crazy. If there's too much logic in your task execution function, consider factoring out useful functionality into the main library.

```
def run_readout_scan(task: ReadoutScanTask,
                     base_dir=None):
    """Execute a :py:class:`ReadoutScanTask` task."""
    if base_dir is None:
        base_dir = DEFAULT_BASE_DIR

    if recirq.exists(task, base_dir=base_dir):
        print(f"{task} already exists. Skipping.")
        return

    # Create a simple circuit
    theta = sympy.Symbol('theta')
    circuit = cirq.Circuit([
        cirq.ry(theta).on(task.qubit),
        cirq.measure(task.qubit, key='z')
    ])

    # Use utilities to map sampler names to Sampler objects
    sampler = recirq.get_sampler_by_name(device_name=task.device_name)

    # Use a sweep over theta values.
    # Set up limits so we include (-1/2, 0, 1/2, 1, 3/2) * pi
    # The total number of points is resolution_factor * 4 + 1
    n_special_points: int = 5
    resolution_factor = task.resolution_factor
    theta_sweep = cirq.Linspace(theta, -np.pi / 2, 3 * np.pi / 2,
                                resolution_factor * (n_special_points - 1) + 1)
    thetas = np.asarray([v for ((k, v),) in theta_sweep.param_tuples()])
    flat_circuit, flat_sweep = cirq.flatten_with_sweep(circuit, theta_sweep)

    # Run the jobs
    print(f"Collecting data for {task.qubit}", flush=True)
    results = sampler.run_sweep(program=flat_circuit, params=flat_sweep,
                                repetitions=task.n_shots)

    # Save the results
    recirq.save(task=task, data={
        'thetas': thetas,
        'all_bitstrings': [
            recirq.BitArray(np.asarray(r.measurements['z']))
            for r in results]
    }, base_dir=base_dir)
```

## The driver script

Typically, the above classes and functions will live in a Python module; something like `recirq/readout_scan/tasks.py`. You can then have one or more "driver scripts" which are actually executed.

View the driver script as a configuration file that specifies exactly which parameters you want to run. You can see that below, we've formatted the construction of all the task objects to look like a configuration file. This is no accident! As noted in the docstring, the user can be expected to twiddle values defined in the script. Trying to factor this out into an ini file (or similar) is more effort than it's worth.

```
# Put in a file named run-readout-scan.py

import datetime
import cirq_google as cg

MAX_N_QUBITS = 5

def main():
    """Main driver script entry point.

    This function contains configuration options and you will likely need
    to edit it to suit your needs. Of particular note, please make sure
    `dataset_id` and `device_name`
    are set how you want them. You may also want to change the values in
    the list comprehension to set the qubits.
    """
    # Uncomment below for an auto-generated unique dataset_id
    # dataset_id = datetime.datetime.now().isoformat(timespec='minutes')
    dataset_id = '2020-02-tutorial'
    data_collection_tasks = [
        ReadoutScanTask(
            dataset_id=dataset_id,
            device_name='Syc23-simulator',
            n_shots=40_000,
            qubit=qubit,
            resolution_factor=6,
        )
        for qubit in cg.Sycamore23.qubits[:MAX_N_QUBITS]
    ]

    for dc_task in data_collection_tasks:
        run_readout_scan(dc_task)

if __name__ == '__main__':
    main()
```

Additionally, follow good Python convention by wrapping the entry point in a function (i.e. `def main():` rather than putting it directly under `if __name__ == '__main__'`. The latter strategy puts all variables in the global scope (bad!).