```python
!pip install pennylane StrawberryFields tensorflow-hub tensorflow_text emoji pythainlp pennylane-sf  plotnine
```

    Requirement already satisfied: pennylane in /usr/local/lib/python3.6/dist-packages (0.9.0)
    Requirement already satisfied: StrawberryFields in /usr/local/lib/python3.6/dist-packages (0.14.0)
    Requirement already satisfied: tensorflow-hub in /usr/local/lib/python3.6/dist-packages (0.8.0)
    Requirement already satisfied: tensorflow_text in /usr/local/lib/python3.6/dist-packages (2.2.1)
    Requirement already satisfied: emoji in /usr/local/lib/python3.6/dist-packages (0.5.4)
    Requirement already satisfied: pythainlp in /usr/local/lib/python3.6/dist-packages (2.1.4)
    Requirement already satisfied: pennylane-sf in /usr/local/lib/python3.6/dist-packages (0.9.0)
    Collecting plotnine
      Downloading https://files.pythonhosted.org/packages/aa/ee/9f343ef0719541e02950384978722ea84cc433264cece09a11a0cccb2bf9/plotnine-0.7.0-py3-none-any.whl (4.4MB)
         |████████████████████████████████| 4.4MB 977kB/s eta 0:00:01
    Requirement already satisfied: autograd in /usr/local/lib/python3.6/dist-packages (from pennylane) (1.3)
    Requirement already satisfied: networkx in /usr/local/lib/python3.6/dist-packages (from pennylane) (2.4)
    Requirement already satisfied: toml in /usr/local/lib/python3.6/dist-packages (from pennylane) (0.10.1)
    Requirement already satisfied: scipy in /usr/local/lib/python3.6/dist-packages (from pennylane) (1.3.1)
    Requirement already satisfied: semantic-version==2.6 in /usr/local/lib/python3.6/dist-packages (from pennylane) (2.6.0)
    Requirement already satisfied: appdirs in /usr/local/lib/python3.6/dist-packages (from pennylane) (1.4.4)
    Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from pennylane) (1.17.4)
    Requirement already satisfied: quantum-blackbird>=0.2.3 in /usr/local/lib/python3.6/dist-packages (from StrawberryFields) (0.2.4)
    Requirement already satisfied: python-dateutil>=2.8.0 in /usr/local/lib/python3.6/dist-packages (from StrawberryFields) (2.8.1)
    Requirement already satisfied: requests>=2.22.0 in /usr/local/lib/python3.6/dist-packages (from StrawberryFields) (2.22.0)
    Requirement already satisfied: thewalrus>=0.12 in /usr/local/lib/python3.6/dist-packages (from StrawberryFields) (0.12.0)
    Requirement already satisfied: urllib3>=1.25.3 in /usr/local/lib/python3.6/dist-packages (from StrawberryFields) (1.25.7)
    Requirement already satisfied: numba in /usr/local/lib/python3.6/dist-packages (from StrawberryFields) (0.49.1)
    Requirement already satisfied: sympy>=1.5 in /usr/local/lib/python3.6/dist-packages (from StrawberryFields) (1.6)
    Requirement already satisfied: six>=1.12.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow-hub) (1.13.0)
    Requirement already satisfied: protobuf>=3.8.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow-hub) (3.10.0)
    Requirement already satisfied: tensorflow<2.3,>=2.2.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow_text) (2.2.0)
    Requirement already satisfied: nltk>=3.3 in /usr/local/lib/python3.6/dist-packages (from pythainlp) (3.4.5)
    Requirement already satisfied: dill>=0.3.0 in /usr/local/lib/python3.6/dist-packages (from pythainlp) (0.3.1.1)
    Requirement already satisfied: tqdm>=4.1 in /usr/local/lib/python3.6/dist-packages (from pythainlp) (4.38.0)
    Requirement already satisfied: tinydb>=3.0 in /usr/local/lib/python3.6/dist-packages (from pythainlp) (3.15.2)
    Collecting mizani>=0.7.1
      Downloading https://files.pythonhosted.org/packages/57/29/72ed07818b1cd7f79340d55c7230748238e09c35fc8461c545a5d350006d/mizani-0.7.1-py3-none-any.whl (62kB)
         |████████████████████████████████| 71kB 11.1MB/s eta 0:00:01
    Collecting statsmodels>=0.11.1
      Downloading https://files.pythonhosted.org/packages/cb/83/540fd83238a18abe6c2d280fa8e489ac5fcefa1f370f0ca1acd16ae1b860/statsmodels-0.11.1-cp36-cp36m-manylinux1_x86_64.whl (8.7MB)
         |████████████████████████████████| 8.7MB 11.3MB/s eta 0:00:01
    Requirement already satisfied: matplotlib>=3.1.1 in /usr/local/lib/python3.6/dist-packages (from plotnine) (3.2.1)
    Collecting patsy>=0.5.1
      Downloading https://files.pythonhosted.org/packages/ea/0c/5f61f1a3d4385d6bf83b83ea495068857ff8dfb89e74824c6e9eb63286d8/patsy-0.5.1-py2.py3-none-any.whl (231kB)
         |████████████████████████████████| 235kB 29.0MB/s eta 0:00:01
    Collecting descartes>=1.1.0
      Downloading https://files.pythonhosted.org/packages/e5/b6/1ed2eb03989ae574584664985367ba70cd9cf8b32ee8cad0e8aaeac819f3/descartes-1.1.0-py3-none-any.whl
    Requirement already satisfied: pandas>=1.0.3 in /usr/local/lib/python3.6/dist-packages (from plotnine) (1.0.3)
    Requirement already satisfied: future>=0.15.2 in /usr/local/lib/python3.6/dist-packages (from autograd->pennylane) (0.17.1)
    Requirement already satisfied: decorator>=4.3.0 in /usr/local/lib/python3.6/dist-packages (from networkx->pennylane) (4.4.1)
    Requirement already satisfied: antlr4-python3-runtime==4.7.2 in /usr/local/lib/python3.6/dist-packages (from quantum-blackbird>=0.2.3->StrawberryFields) (4.7.2)
    Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests>=2.22.0->StrawberryFields) (3.0.4)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests>=2.22.0->StrawberryFields) (2019.9.11)
    Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests>=2.22.0->StrawberryFields) (2.8)
    Requirement already satisfied: dask[delayed] in /usr/local/lib/python3.6/dist-packages (from thewalrus>=0.12->StrawberryFields) (2.18.1)
    Requirement already satisfied: llvmlite<=0.33.0.dev0,>=0.31.0.dev0 in /usr/local/lib/python3.6/dist-packages (from numba->StrawberryFields) (0.32.1)
    Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from numba->StrawberryFields) (41.6.0)
    Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.6/dist-packages (from sympy>=1.5->StrawberryFields) (1.1.0)
    Requirement already satisfied: absl-py>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow<2.3,>=2.2.0->tensorflow_text) (0.8.1)
    Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow<2.3,>=2.2.0->tensorflow_text) (1.25.0)
    Requirement already satisfied: tensorflow-estimator<2.3.0,>=2.2.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow<2.3,>=2.2.0->tensorflow_text) (2.2.0)
    Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow<2.3,>=2.2.0->tensorflow_text) (1.1.0)
    Requirement already satisfied: keras-preprocessing>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow<2.3,>=2.2.0->tensorflow_text) (1.1.2)
    Requirement already satisfied: google-pasta>=0.1.8 in /usr/local/lib/python3.6/dist-packages (from tensorflow<2.3,>=2.2.0->tensorflow_text) (0.1.8)
    Requirement already satisfied: astunparse==1.6.3 in /usr/local/lib/python3.6/dist-packages (from tensorflow<2.3,>=2.2.0->tensorflow_text) (1.6.3)
    Requirement already satisfied: wrapt>=1.11.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow<2.3,>=2.2.0->tensorflow_text) (1.11.2)
    Requirement already satisfied: h5py<2.11.0,>=2.10.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow<2.3,>=2.2.0->tensorflow_text) (2.10.0)
    Requirement already satisfied: gast==0.3.3 in /usr/local/lib/python3.6/dist-packages (from tensorflow<2.3,>=2.2.0->tensorflow_text) (0.3.3)
    Requirement already satisfied: tensorboard<2.3.0,>=2.2.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow<2.3,>=2.2.0->tensorflow_text) (2.2.2)
    Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow<2.3,>=2.2.0->tensorflow_text) (3.1.0)
    Requirement already satisfied: wheel>=0.26; python_version >= "3" in /usr/local/lib/python3.6/dist-packages (from tensorflow<2.3,>=2.2.0->tensorflow_text) (0.33.6)
    Collecting palettable
      Downloading https://files.pythonhosted.org/packages/ca/46/5198aa24e61bb7eef28d06cb69e56bfa1942f4b6807d95a0b5ce361fe09b/palettable-3.3.0-py2.py3-none-any.whl (111kB)
         |████████████████████████████████| 112kB 23.6MB/s eta 0:00:01
    Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=3.1.1->plotnine) (1.2.0)
    Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=3.1.1->plotnine) (2.4.7)
    Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=3.1.1->plotnine) (0.10.0)
    Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas>=1.0.3->plotnine) (2019.3)
    Requirement already satisfied: pyyaml in /usr/local/lib/python3.6/dist-packages (from dask[delayed]->thewalrus>=0.12->StrawberryFields) (5.1.2)
    Requirement already satisfied: toolz>=0.8.2; extra == "delayed" in /usr/local/lib/python3.6/dist-packages (from dask[delayed]->thewalrus>=0.12->StrawberryFields) (0.10.0)
    Requirement already satisfied: cloudpickle>=0.2.2; extra == "delayed" in /opt/tensorflow/horovod-source/.eggs/cloudpickle-1.2.2-py3.6.egg (from dask[delayed]->thewalrus>=0.12->StrawberryFields) (1.2.2)
    Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow<2.3,>=2.2.0->tensorflow_text) (0.4.1)
    Requirement already satisfied: google-auth<2,>=1.6.3 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow<2.3,>=2.2.0->tensorflow_text) (1.7.0)
    Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow<2.3,>=2.2.0->tensorflow_text) (1.6.0.post3)
    Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow<2.3,>=2.2.0->tensorflow_text) (3.1.1)
    Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow<2.3,>=2.2.0->tensorflow_text) (0.16.0)
    Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.3.0,>=2.2.0->tensorflow<2.3,>=2.2.0->tensorflow_text) (1.3.0)
    Requirement already satisfied: cachetools<3.2,>=2.0.0 in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow<2.3,>=2.2.0->tensorflow_text) (3.1.1)
    Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow<2.3,>=2.2.0->tensorflow_text) (0.2.7)
    Requirement already satisfied: rsa<4.1,>=3.1.4 in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow<2.3,>=2.2.0->tensorflow_text) (4.0)
    Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.6/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.3.0,>=2.2.0->tensorflow<2.3,>=2.2.0->tensorflow_text) (3.1.0)
    Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.6/dist-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow<2.3,>=2.2.0->tensorflow_text) (0.4.7)
    Installing collected packages: palettable, mizani, patsy, statsmodels, descartes, plotnine
    Successfully installed descartes-1.1.0 mizani-0.7.1 palettable-3.3.0 patsy-0.5.1 plotnine-0.7.0 statsmodels-0.11.1
    WARNING: You are using pip version 19.3.1; however, version 20.1.1 is available.
    You should consider upgrading via the 'pip install --upgrade pip' command.

```python
!wget https://github.com/PyThaiNLP/truevoice-intent/archive/master.zip; unzip master.zip; cd truevoice-intent-master; unzip mari-intent.zip; cp *.* ../;
```

    --2020-06-11 03:12:29--  https://github.com/PyThaiNLP/truevoice-intent/archive/master.zip
    Resolving github.com (github.com)... 52.74.223.119
    Connecting to github.com (github.com)|52.74.223.119|:443... connected.
    HTTP request sent, awaiting response... 302 Found
    Location: https://codeload.github.com/PyThaiNLP/truevoice-intent/zip/master [following]
    --2020-06-11 03:12:30--  https://codeload.github.com/PyThaiNLP/truevoice-intent/zip/master
    Resolving codeload.github.com (codeload.github.com)... 13.229.189.0
    Connecting to codeload.github.com (codeload.github.com)|13.229.189.0|:443... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: unspecified [application/zip]
    Saving to: ‘master.zip’

    master.zip              [    <=>             ]   1.29M  2.07MB/s    in 0.6s

    2020-06-11 03:12:31 (2.07 MB/s) - ‘master.zip’ saved [1350596]

    Archive:  master.zip
    7ce601a547a4bac938b3bb0640a6036f7b89266c
       creating: truevoice-intent-master/
      inflating: truevoice-intent-master/.gitignore
      inflating: truevoice-intent-master/LICENSE
      inflating: truevoice-intent-master/README.md
      inflating: truevoice-intent-master/class_features.py
      inflating: truevoice-intent-master/classification.ipynb
      inflating: truevoice-intent-master/exploration.ipynb
     extracting: truevoice-intent-master/mari-intent.zip
      inflating: truevoice-intent-master/remove-dup-split.ipynb
      inflating: truevoice-intent-master/thsarabunnew-webfont.ttf
    Archive:  mari-intent.zip
      inflating: mari-train-balanced.csv
      inflating: mari-test-balanced.csv

```python
!mkdir ft_data
!mkdir truevoice_data
```

```python
!wget https://github.com/rdisipio/qnlp/raw/master/models.py
```

    --2020-06-11 03:12:33--  https://github.com/rdisipio/qnlp/raw/master/models.py
    Resolving github.com (github.com)... 52.74.223.119
    Connecting to github.com (github.com)|52.74.223.119|:443... connected.
    HTTP request sent, awaiting response... 302 Found
    Location: https://raw.githubusercontent.com/rdisipio/qnlp/master/models.py [following]
    --2020-06-11 03:12:34--  https://raw.githubusercontent.com/rdisipio/qnlp/master/models.py
    Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.8.133
    Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.8.133|:443... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 1212 (1.2K) [text/plain]
    Saving to: ‘models.py’

    models.py           100%[===================>]   1.18K  --.-KB/s    in 0s

    2020-06-11 03:12:34 (71.3 MB/s) - ‘models.py’ saved [1212/1212]

```python
!wget https://github.com/rdisipio/qnlp/raw/master/models_quantum.py
```

    --2020-06-11 03:12:35--  https://github.com/rdisipio/qnlp/raw/master/models_quantum.py
    Resolving github.com (github.com)... 52.74.223.119
    Connecting to github.com (github.com)|52.74.223.119|:443... connected.
    HTTP request sent, awaiting response... 302 Found
    Location: https://raw.githubusercontent.com/rdisipio/qnlp/master/models_quantum.py [following]
    --2020-06-11 03:12:35--  https://raw.githubusercontent.com/rdisipio/qnlp/master/models_quantum.py
    Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.8.133
    Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.8.133|:443... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 8498 (8.3K) [text/plain]
    Saving to: ‘models_quantum.py’

    models_quantum.py   100%[===================>]   8.30K  --.-KB/s    in 0s

    2020-06-11 03:12:36 (41.7 MB/s) - ‘models_quantum.py’ saved [8498/8498]

```python
import pandas as pd
import numpy as np
from pythainlp import word_tokenize
from tqdm import tqdm_notebook
from collections import Counter
import re
from sklearn.metrics import f1_score,precision_score,recall_score

#viz
from plotnine import *
import matplotlib.pyplot as plt
import seaborn as sns

def replace_newline(t):
    return re.sub('[\n]{1,}', ' ', t)

ft_data = 'ft_data/'

y = 'destination'
nb_class = 7
```

```python
import string
import emoji
def replace_url(text):
    URL_PATTERN = r"""(?i)\b((?:https?:(?:/{1,3}|[a-z0-9%])|[a-z0-9.\-]+[.](?:com|net|org|edu|gov|mil|aero|asia|biz|cat|coop|info|int|jobs|mobi|museum|name|post|pro|tel|travel|xxx|ac|ad|ae|af|ag|ai|al|am|an|ao|aq|ar|as|at|au|aw|ax|az|ba|bb|bd|be|bf|bg|bh|bi|bj|bm|bn|bo|br|bs|bt|bv|bw|by|bz|ca|cc|cd|cf|cg|ch|ci|ck|cl|cm|cn|co|cr|cs|cu|cv|cx|cy|cz|dd|de|dj|dk|dm|do|dz|ec|ee|eg|eh|er|es|et|eu|fi|fj|fk|fm|fo|fr|ga|gb|gd|ge|gf|gg|gh|gi|gl|gm|gn|gp|gq|gr|gs|gt|gu|gw|gy|hk|hm|hn|hr|ht|hu|id|ie|il|im|in|io|iq|ir|is|it|je|jm|jo|jp|ke|kg|kh|ki|km|kn|kp|kr|kw|ky|kz|la|lb|lc|li|lk|lr|ls|lt|lu|lv|ly|ma|mc|md|me|mg|mh|mk|ml|mm|mn|mo|mp|mq|mr|ms|mt|mu|mv|mw|mx|my|mz|na|nc|ne|nf|ng|ni|nl|no|np|nr|nu|nz|om|pa|pe|pf|pg|ph|pk|pl|pm|pn|pr|ps|pt|pw|py|qa|re|ro|rs|ru|rw|sa|sb|sc|sd|se|sg|sh|si|sj|Ja|sk|sl|sm|sn|so|sr|ss|st|su|sv|sx|sy|sz|tc|td|tf|tg|th|tj|tk|tl|tm|tn|to|tp|tr|tt|tv|tw|tz|ua|ug|uk|us|uy|uz|va|vc|ve|vg|vi|vn|vu|wf|ws|ye|yt|yu|za|zm|zw)/)(?:[^\s()<>{}\[\]]+|\([^\s()]*?\([^\s()]+\)[^\s()]*?\)|\([^\s]+?\))+(?:\([^\s()]*?\([^\s()]+\)[^\s()]*?\)|\([^\s]+?\)|[^\s`!()\[\]{};:'".,<>?«»“”‘’])|(?:(?<!@)[a-z0-9]+(?:[.\-][a-z0-9]+)*[.](?:com|net|org|edu|gov|mil|aero|asia|biz|cat|coop|info|int|jobs|mobi|museum|name|post|pro|tel|travel|xxx|ac|ad|ae|af|ag|ai|al|am|an|ao|aq|ar|as|at|au|aw|ax|az|ba|bb|bd|be|bf|bg|bh|bi|bj|bm|bn|bo|br|bs|bt|bv|bw|by|bz|ca|cc|cd|cf|cg|ch|ci|ck|cl|cm|cn|co|cr|cs|cu|cv|cx|cy|cz|dd|de|dj|dk|dm|do|dz|ec|ee|eg|eh|er|es|et|eu|fi|fj|fk|fm|fo|fr|ga|gb|gd|ge|gf|gg|gh|gi|gl|gm|gn|gp|gq|gr|gs|gt|gu|gw|gy|hk|hm|hn|hr|ht|hu|id|ie|il|im|in|io|iq|ir|is|it|je|jm|jo|jp|ke|kg|kh|ki|km|kn|kp|kr|kw|ky|kz|la|lb|lc|li|lk|lr|ls|lt|lu|lv|ly|ma|mc|md|me|mg|mh|mk|ml|mm|mn|mo|mp|mq|mr|ms|mt|mu|mv|mw|mx|my|mz|na|nc|ne|nf|ng|ni|nl|no|np|nr|nu|nz|om|pa|pe|pf|pg|ph|pk|pl|pm|pn|pr|ps|pt|pw|py|qa|re|ro|rs|ru|rw|sa|sb|sc|sd|se|sg|sh|si|sj|Ja|sk|sl|sm|sn|so|sr|ss|st|su|sv|sx|sy|sz|tc|td|tf|tg|th|tj|tk|tl|tm|tn|to|tp|tr|tt|tv|tw|tz|ua|ug|uk|us|uy|uz|va|vc|ve|vg|vi|vn|vu|wf|ws|ye|yt|yu|za|zm|zw)\b/?(?!@)))"""
    return re.sub(URL_PATTERN, 'xxurl', text)

def replace_rep(text):
    def _replace_rep(m):
        c,cc = m.groups()
        return f'{c}xxrep'
    re_rep = re.compile(r'(\S)(\1{2,})')
    return re_rep.sub(_replace_rep, text)

def ungroup_emoji(toks):
    res = []
    for tok in toks:
        if emoji.emoji_count(tok) == len(tok):
            for char in tok:
                res.append(char)
        else:
            res.append(tok)
    return res

def process_text(text):
    #pre rules
    res = text.lower().strip()
    res = replace_url(res)
    res = replace_rep(res)

    #tokenize
    res = [word for word in res.split('|') if word and not re.search(pattern=r"\s+", string=word)]

    #post rules
    res = ungroup_emoji(res)

    return res
```

#Train-validation-test Split

We perform 85/15 train-validation split in addition to the test split by TrueVoice.

```python
from sklearn.model_selection import train_test_split
all_df = pd.read_csv(f'mari-train-balanced.csv')
all_df['destination'] = all_df.destination.map(lambda x: x.replace(' ','_'))
#train_df, valid_df = train_test_split(all_df, test_size=0.15, random_state=1412)
train_df = all_df.reset_index(drop=True)
#valid_df = valid_df.reset_index(drop=True)
test_df = pd.read_csv(f'mari-test-balanced.csv')
test_df['destination'] = test_df.destination.map(lambda x: x.replace(' ','_'))
print(train_df.shape, test_df.shape)
```

    (10303, 5) (2607, 5)

```python
#test set prevalence
test_df['destination'].value_counts() / test_df.shape[0]
```

    billing_and_payment      0.376678
    promotions               0.220560
    other_queries            0.176064
    internet                 0.154200
    international_dialing    0.034522
    true_money               0.019946
    lost_and_stolen          0.018028
    Name: destination, dtype: float64

```python
#!pip install pennylane pennylane-sf

import pennylane
pennylane.device("strawberryfields.gaussian", wires=2) # ถ้า error ให้ restart runtime
```

    <StrawberryFieldsGaussian device (wires=2, shots=1000) at 0x7fbf7b375eb8>

```python
import tensorflow_hub as hub
import tensorflow_text
import tensorflow as tf #tensorflow 2.1.0
from tensorflow.keras.callbacks import EarlyStopping

from models_quantum import make_model_quantum
enc = hub.load('https://tfhub.dev/google/universal-sentence-encoder-multilingual/3')
```

```python
tf.keras.backend.set_floatx('float64')
```

```python
train_df.shape, test_df.shape
```

    ((10303, 5), (2607, 5))

```python

def embed_text(X_txt):
    print("Embedding input text...")

    X = np.array([np.array(x) for x in enc(X_txt)])

    return X

```

```python
X_train = embed_text(train_df.texts)
```

    Embedding input text...

```python
X_test = embed_text(test_df.texts)
```

    Embedding input text...

```python
embedding_dim =X_train.shape[-1]
```

```python
embedding_dim
```

    512

```python
index = {j:i for i,j in enumerate(list(set(train_df[y].values)))}
```

```python
index
```

    {'other_queries': 0,
     'billing_and_payment': 1,
     'lost_and_stolen': 2,
     'promotions': 3,
     'international_dialing': 4,
     'internet': 5,
     'true_money': 6}

```python
from tensorflow.keras.utils import to_categorical
```

```python
y_train = [index[i] for i in train_df[y].values]
```

```python
y_train = to_categorical(y_train)
```

```python
y_test  = [index[i] for i in test_df[y].values]
```

```python
y_test = to_categorical(y_test)
```

```python
N_EPOCHS = 20
BATCH_SIZE = 32
n_categories = 7
```

```python
embedding_dim = X_train.shape[-1]
```

```python
import pennylane as qml
qml.about()
```

    Name: PennyLane
    Version: 0.9.0
    Summary: PennyLane is a Python quantum machine learning library by Xanadu Inc.
    Home-page: https://github.com/XanaduAI/pennylane
    Author: None
    Author-email: None
    License: Apache License 2.0
    Location: /usr/local/lib/python3.6/dist-packages
    Requires: appdirs, toml, autograd, networkx, numpy, scipy, semantic-version
    Required-by: PennyLane-SF
    Platform info:           Linux-4.15.0-72-generic-x86_64-with-Ubuntu-18.04-bionic
    Python version:          3.6.8
    Numpy version:           1.17.4
    Scipy version:           1.3.1
    Installed devices:
    - default.gaussian (PennyLane-0.9.0)
    - default.qubit (PennyLane-0.9.0)
    - default.tensor (PennyLane-0.9.0)
    - default.tensor.tf (PennyLane-0.9.0)
    - strawberryfields.fock (PennyLane-SF-0.9.0)
    - strawberryfields.gaussian (PennyLane-SF-0.9.0)

```python
model = make_model_quantum(n_categories=n_categories,  n_qubits=4, n_layers=3, embedding_dim=embedding_dim)
```

```python
model.summary()
```

    Model: "QuantumPreprintClassifier"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    text_in (InputLayer)         [(None, 512)]             0
    _________________________________________________________________
    dense_4 (Dense)              (None, 4)                 2052
    _________________________________________________________________
    variational_quantum_circuit_ (None, 4)                 36
    _________________________________________________________________
    dense_7 (Dense)              (None, 7)                 35
    =================================================================
    Total params: 2,123
    Trainable params: 2,123
    Non-trainable params: 0
    _________________________________________________________________

```python
optimizer = tf.keras.optimizers.Adam(learning_rate=0.01)
```

```python
model.compile(optimizer, loss='categorical_crossentropy')
```

```python
early_stopping_callback = EarlyStopping(monitor='val_loss', patience=3, min_delta=0.01,restore_best_weights=True)
```

```python
from matplotlib import pyplot as plt
from IPython.display import clear_output
import numpy as np
```

```python
class PlotLosses(tf.keras.callbacks.Callback): # code from https://gist.github.com/stared/dfb4dfaf6d9a8501cd1cc8b8cb806d2e
    def on_train_begin(self, logs={}):
        self.i = 0
        self.x = []
        self.losses = []
        self.val_losses = []

        self.fig = plt.figure()

        self.logs = []

    def on_epoch_end(self, epoch, logs={}):

        self.logs.append(logs)
        self.x.append(self.i)
        self.losses.append(logs.get('loss'))
        self.val_losses.append(logs.get('val_loss'))
        self.i += 1

        clear_output(wait=True)
        plt.plot(self.x, self.losses, label="loss")
        plt.plot(self.x, self.val_losses, label="val_loss")
        plt.legend()
        plt.show();

plot_losses = PlotLosses()
```

```python
callbacks = [early_stopping_callback,plot_losses]
```

```python
history = model.fit(X_train, y_train,epochs=N_EPOCHS, batch_size=BATCH_SIZE,validation_split=0.15,callbacks =callbacks)
```

```python
model.predict(X_test)
```

    array([[3.70255528e-02, 7.62594837e-01, 2.23173318e-03, ...,
            1.52303516e-04, 5.57465622e-02, 1.40657142e-03],
           [2.17854399e-02, 2.33223092e-02, 4.43345599e-04, ...,
            1.40323680e-03, 6.00061262e-03, 1.94607543e-04],
           [7.96126955e-03, 9.75440668e-01, 1.21422228e-03, ...,
            3.99320930e-05, 5.49612634e-03, 1.89945120e-03],
           ...,
           [6.83175325e-01, 2.75767411e-02, 5.98708792e-02, ...,
            4.70041373e-02, 1.12416090e-01, 3.57558300e-03],
           [4.51997391e-02, 5.54229024e-02, 9.59489261e-04, ...,
            8.83238246e-04, 1.52961508e-02, 2.39712495e-04],
           [1.32345636e-02, 3.55008743e-02, 1.21654943e-03, ...,
            2.69381894e-04, 9.15894139e-01, 1.22652911e-03]])

```python
test_score = model.evaluate(X_test, y_test, verbose=2)
```

    82/82 - 125s - loss: 0.6645

```python
print(test_score)
```

    0.6645362609229327

```python
t=model.predict(X_test)
```

```python
val2tag = {i:j for i,j in zip(index.values(),index.keys())}
```

```python
y_true = list(test_df[y].values)
```

```python
y_pred = [val2tag[list(i).index(i.max())] for i in t]
```

```python
from sklearn.metrics import classification_report
print(classification_report(y_true, y_pred))
```

                           precision    recall  f1-score   support

      billing_and_payment       0.84      0.83      0.84       982
    international_dialing       0.76      0.73      0.75        90
                 internet       0.74      0.74      0.74       402
          lost_and_stolen       0.00      0.00      0.00        47
            other_queries       0.69      0.74      0.71       459
               promotions       0.78      0.85      0.81       575
               true_money       0.90      0.37      0.52        52

                 accuracy                           0.78      2607
                macro avg       0.67      0.61      0.62      2607
             weighted avg       0.77      0.78      0.77      2607

    /usr/local/lib/python3.6/dist-packages/sklearn/metrics/_classification.py:1272: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.


```python
import sys
print('hello world', flush=True)
for i in range(3):
    print(i, flush=True)
print('output to stderr', file=sys.stderr, flush=True)
print('some more stdout text', flush=True)
```

    hello world
    0
    1
    2

    output to stderr

    some more stdout text

# Markdown Cell

$ e^{ \pm i\theta } = \cos \theta \pm i\sin \theta + \beta $

*It* **really** is!

```python
this is a syntax error
```

```python
from IPython.display import Image
Image('./jupyter.png')
```

```python
from IPython.display import Latex
Latex('''The mass-energy equivalence is described by the famous equation

$$E=mc^2$$

discovered in 1905 by Albert Einstein.
In natural units ($c$ = 1), the formula expresses the identity

\\begin{equation}
E=m
\\end{equation}''')
```

The mass-energy equivalence is described by the famous equation

$$E=mc^2$$

discovered in 1905 by Albert Einstein.
In natural units ($c$ = 1), the formula expresses the identity

\begin{equation}
E=m
\end{equation}

```python

```


```python
# This cell is mandatory in all Dymos documentation notebooks.
missing_packages = []
try:
    import openmdao.api as om
except ImportError:
    if 'google.colab' in str(get_ipython()):
        !python -m pip install openmdao[notebooks]
    else:
        missing_packages.append('openmdao')
try:
    import dymos as dm
except ImportError:
    if 'google.colab' in str(get_ipython()):
        !python -m pip install dymos
    else:
        missing_packages.append('dymos')
try:
    import pyoptsparse
except ImportError:
    if 'google.colab' in str(get_ipython()):
        !pip install -q condacolab
        import condacolab
        condacolab.install_miniconda()
        !conda install -c conda-forge pyoptsparse
    else:
        missing_packages.append('pyoptsparse')
if missing_packages:
    raise EnvironmentError('This notebook requires the following packages '
                           'please install them and restart this notebook\'s runtime: {",".join(missing_packages)}')
```

# Water Rocket

Author: Bernardo Bahia Monteiro (bbahia@umich.edu)

In this example, we will optimize a water rocket for range and height at the apogee, using design variables that are easily modifiable just before launch: the empty mass, the initial water volume and the launch angle.
This example builds on [multi-phase cannonball](../multi_phase_cannonball/multi_phase_cannonball.md) ane is adapted from _Optimization of a Water Rocket in OpenMDAO/Dymos_ {cite}`bbahia_2020`.

## Nomenclature

| Symbol               | definition                             |
|----------------------|----------------------------------------|
| $v_\text{out}$       | water exit speed at the nozzle         |
| $A_\text{out}$       | nozzle area                            |
| $V_w$                | water volume in the rocket             |
| $p$                  | pressure in the rocket                 |
| $p_a$                | ambient pressure                       |
| $\dot{\,}$           | time derivative                        |
| $k$                  | polytropic constant                    |
| $V_b$                | internal volume of the rocket          |
| $\rho_w$             | water density                          |
| $T$                  | thrust                                 |
| $q$                  | dynamic pressure                       |
| $S$                  | cross sectional area                   |
| $(\cdot)_0$          | value of $(\cdot)$ at $t=0$            |
| $t$                  | time                                   |

## Problem Formulation

A natural objective function for a water rocket is the maximum height achieved by the rocket during flight, or the horizontal distance it travels, i.e. its range.
The design of a water rocket is somewhat constrained by the soda bottle used as its engine.
This means that the volume available for water and air is fixed, the initial launch pressure is limited by the bottle's strength (since the pressure is directly related to the energy available for the rocket, it is easy to see that it should be as high as possible) and the nozzle throat area is also fixed.
Given these manufacturing constraints, the design variables we are left with are the empty mass (it can be easily changed through adding ballast), the water volume at the launch, and the launch angle.
With this considerations in mind, a natural formulation for the water rocket problem is

\begin{align}
    \text{maximize}   &\quad \text{range or height} \\
    \text{w.r.t.}     &\quad \text{empty mass, initial water volume, launch angle, trajectory} \\
    \text{subject to} &\quad \text{flight dynamics} \\
                      &\quad \text{fluid dynamics inside the rocket} \\
                      &\quad 0 < \text{initial water volume} < \text{volume of bottle} \\
                      &\quad 0^\circ < \text{launch angle} < 90^\circ \\
                      &\quad 0 < \text{empty mass}
\end{align}

##  Model

The water rocket model is divided into three basic components: a *water engine*, responsible for modelling the fluid dynamics inside the rocket and returning its thrust;  the *aerodynamics*, responsible for calculating the atmospheric drag of the rocket; and the *equations of motion*, responsible for propagating the rocket's trajectory in time, using Newton's laws and the forces provided by the other two components.

In order to integrate these three basic components, some additional interfacing components are necessary: an atmospheric model to provide values of ambient pressure for the water engine and air density to the calculation of the dynamic pressure for the aerodynamic model, and a component that calculates the instantaneous mass of the rocket by summing the water mass with the rocket's empty mass.
The high level layout of this model is shown in below.

N2 diagram for the water rocket model

`atmos`, `dynamic_pressure`, `aero` and `eom` are the same models used in [multi-phase cannonball](../multi_phase_cannonball/multi_phase_cannonball.md).
The remaining components are discussed below.

```{Warning}
The `eom` component has a singularity in the flight path angle derivative when the flight speed is zero.
This happens because the rotational dynamics are not modelled.
This can cause convergence problems if the launch velocity is set to zero or the launch angle is set to $90^\circ$
```

```{Note}
Since the range of altitudes achieved by the water rocket is very small (100m), the air density and pressure are practically constant, thus the use of an atmospheric model is not necessary. However, using it makes it easier to reuse code from [multi-phase cannonball (../multi_phase_cannonball/multi_phase_cannonball.md).
```

### Water engine

The water engine is modelled by assuming that the air expansion in the rocket
follows an adiabatic process and the water flow is incompressible and inviscid,
i.e.  it follows Bernoulli's equation. We also make the following simplifying
assumptions:

1. The thrust developed after the water is depleted is negligible
2. The area inside the bottle is much smaller than the nozzle area
3. The inertial forces do not affect the fluid dynamics inside the bottle

This simplified modelling can be found in Prusa[@Prusa2000].
A more rigorous formulation, which drops all these simplifying assumptions can be found in Wheeler[@Wheeler2002], Gommes[@Gommes2010], and Barria-Perotti[@BarrioPerotti2010].

The first assumption leads to an underestimation of the rocket performance, since the air left in the bottle after it is out of water is known to generate appreciable thrust[@Thorncroft2009].
This simplified model, however, produces physically meaningful results.

There are two states in this dynamical model, the water volume in the rocket $V_w$ and the gauge pressure inside the rocket $p$.
The constitutive equations and the N2 diagram showing the model organization are shown below.

### Constitutive equations of the water engine model
| Component              | Equation                                                    |
| -----------------------|-------------------------------------------------------------|
| water_exhaust_speed    | $v_\text{out} = \sqrt{2(p-p_a)/\rho_w}$                     |
| water_flow_rate        | $\dot{V}_w = -v_\text{out} A_\text{out}$                    |
| pressure_rate          | $\dot{p} = kp\frac{\dot{V_w}}{(V_b-V_w)}$                   |
| water_thrust           | $T = (\rho_w v_\text{out})(v_\text{out}A_\text{out})$       |

N2 diagram for the water engine group

### Water engine

The water engine is modelled by assuming that the air expansion in the rocket
follows an adiabatic process and the water flow is incompressible and inviscid,
i.e.  it follows Bernoulli's equation. We also make the following simplifying
assumptions:

1. The thrust developed after the water is depleted is negligible
2. The area inside the bottle is much smaller than the nozzle area
3. The inertial forces do not affect the fluid dynamics inside the bottle

This simplified modelling can be found in Prusa {cite}`Prusa2000`.
A more rigorous formulation, which drops all these simplifying assumptions can be found in Wheeler {cite}`Wheeler2002`, Gommes {cite}`Gommes2010`, and Barria-Perotti {cite}`BarrioPerotti2010`.

The first assumption leads to an underestimation of the rocket performance, since the air left in the bottle after it is out of water is known to generate appreciable thrust {cite}`Thorncroft2009`.
This simplified model, however, produces physically meaningful results.

There are two states in this dynamical model, the water volume in the rocket $V_w$ and the gauge pressure inside the rocket $p$.
The constitutive equations and the N2 diagram showing the model organization are shown below.

### Constitutive equations of the water engine model
| Component              | Equation                                                    |
| -----------------------|-------------------------------------------------------------|
| water_exhaust_speed    | $v_\text{out} = \sqrt{2(p-p_a)/\rho_w}$                     |
| water_flow_rate        | $\dot{V}_w = -v_\text{out} A_\text{out}$                    |
| pressure_rate          | $\dot{p} = kp\frac{\dot{V_w}}{(V_b-V_w)}$                   |
| water_thrust           | $T = (\rho_w v_\text{out})(v_\text{out}A_\text{out})$       |

N2 diagram for the water engine group

```python
import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt

import openmdao.api as om
import dymos as dm
```

```python
from openmdao.utils.notebook_utils import display_source
```

```python
display_source('dymos.examples.water_rocket.water_engine_comp.WaterEngine')
```

```python
display_source('dymos.examples.water_rocket.water_engine_comp._WaterExhaustSpeed')
```

```python
display_source('dymos.examples.water_rocket.water_engine_comp._PressureRate')
```

```python
display_source('dymos.examples.water_rocket.water_engine_comp._WaterFlowRate')
```

The `_MassAdder` component calculates the rocket's instantaneous mass by
summing the water mass with the rockets empty mass, i.e.

\begin{align}
    m = m_\text{empty}+\rho_w V_w
\end{align}

```python
display_source('dymos.examples.water_rocket.water_propulsion_ode._MassAdder')
```

Now these components are joined in a single group

```python
display_source('dymos.examples.water_rocket.water_propulsion_ode.WaterPropulsionODE')
```

## Phases

The flight of the water rocket is split in three distinct phases: propelled ascent, ballistic ascent and ballistic descent.
If the simplification of no thrust without water were lifted, there would be an extra "air propelled ascent" phase between the propelled ascent and ballistic ascent phases.

**Propelled ascent:** is the flight phase where the rocket still has water inside, and hence it is producing thrust.
The thrust is given by the water engine model, and fed into the flight dynamic equations.
It starts at launch and finishes when the water is depleted, i.e. $V_w=0$.

**Ballistic ascent:** is the flight phase where the rocket is ascending ($\gamma>0$) but produces no thrust.
This phase begins at the end of thepropelled ascent phase and ends at the apogee, defined by $\gamma=0$.

**Descent:** is the phase where the rocket is descending without thrust.
It begins at the end of the ballistic ascent phase and ends with ground impact, i.e. $h=0$.

```python
display_source('dymos.examples.water_rocket.phases.new_propelled_ascent_phase')
```

```python
display_source('dymos.examples.water_rocket.phases.new_ballistic_ascent_phase')
```

```python
display_source('dymos.examples.water_rocket.phases.new_descent_phase')
```

##  Model parameters

The model requires a few constant parameters.
The values used are shown in the following table.

Values for parameters in the water rocket model

|   Parameter        | Value                | Unit         | Reference                                           |
|--------------------|----------------------|--------------|-----------------------------------------------------|
| $C_D$              | 0.3450               | -            | {cite}`BarrioPerotti2009`                           |
| $S$                | $\pi 106^2/4$        | $mm^2$       | {cite}`BarrioPerotti2009`                           |
| $k$                | 1.2                  | -            | {cite}`Thorncroft2009` {cite}`Fischer2020` {cite}`Romanelli2013`   |
| $A_\text{out}$     | $\pi22^2/4$          | $mm^2$       | {cite}`aircommand_nozzle`                                |
| $V_b$              | 2                    | L            |                                                     |
| $\rho_w$           | 1000                 | $kg/m^3$     |                                                     |
| $p_0$              | 6.5                  | bar          |                                                     |
| $v_0$              | 0.1                  | $m/s$        |                                                     |
| $h_0$              | 0                    | $m$          |                                                     |
| $r_0$              | 0                    | $m$          |                                                     |

Values for the bottle volume $V_b$, its cross-sectional area $S$ and the nozzle area $A_\text{out}$ are determined by the soda bottle that makes the rocket primary structure, and thus are not easily modifiable by the designer.
The polytropic coefficient $k$ is a function of the moist air characteristics inside the rocket.
The initial speed $v_0$ must be set to a value higher than zero, otherwise the flight dynamic equations become singular.
This issue arises from the angular dynamics of the rocket not being modelled.
The drag coefficient $C_D$ is sensitive to the aerodynamic design, but can be optimized by a single discipline analysis.
The initial pressure $p_0$ should be maximized in order to obtain the maximum range or height for the rocket.
It is limited by the structural properties of the bottle, which are modifiable by the designer, since the bottle needs to be available commercially.
Finally, the starting point of the rocket is set to the origin.

## Putting it all together

The different phases must be combined in a single trajectory, and linked in a sequence.
Here we also define the design variables.

```python
display_source('dymos.examples.water_rocket.phases.new_water_rocket_trajectory')
```

## Helper Functions to Access the Results

```python
from collections import namedtuple

def summarize_results(water_rocket_problem):
    p = water_rocket_problem
    Entry = namedtuple('Entry', 'value unit')
    summary = {
        'Launch angle': Entry(p.get_val('traj.propelled_ascent.timeseries.states:gam',  units='deg')[0, 0], 'deg'),
        'Flight angle at end of propulsion': Entry(p.get_val('traj.propelled_ascent.timeseries.states:gam',
                                                   units='deg')[-1, 0], 'deg'),
        'Empty mass': Entry(p.get_val('traj.parameters:m_empty', units='kg')[0], 'kg'),
        'Water volume': Entry(p.get_val('traj.propelled_ascent.timeseries.states:V_w', 'L')[0, 0], 'L'),
        'Maximum range': Entry(p.get_val('traj.descent.timeseries.states:r', units='m')[-1, 0], 'm'),
        'Maximum height': Entry(p.get_val('traj.ballistic_ascent.timeseries.states:h', units='m')[-1, 0], 'm'),
        'Maximum velocity': Entry(p.get_val('traj.propelled_ascent.timeseries.states:v', units='m/s')[-1, 0], 'm/s'),
    }

    return summary

```

```python
colors={'pa': 'tab:blue', 'ba': 'tab:orange', 'd': 'tab:green'}

def plot_propelled_ascent(p, exp_out):
    fig, ax = plt.subplots(2, 2, sharex=True, figsize=(12, 6))
    t_imp = p.get_val('traj.propelled_ascent.time', 's')
    t_exp = exp_out.get_val('traj.propelled_ascent.time', 's')

    c = colors['pa']

    ax[0,0].plot(t_imp, p.get_val('traj.propelled_ascent.timeseries.states:p', 'bar'), '.', color=c)
    ax[0,0].plot(t_exp, exp_out.get_val('traj.propelled_ascent.timeseries.states:p', 'bar'), '-', color=c)
    ax[0,0].set_ylabel('p (bar)')
    ax[0,0].set_ylim(bottom=0)

    ax[1,0].plot(t_imp, p.get_val('traj.propelled_ascent.timeseries.states:V_w', 'L'), '.', color=c)
    ax[1,0].plot(t_exp, exp_out.get_val('traj.propelled_ascent.timeseries.states:V_w', 'L'), '-', color=c)
    ax[1,0].set_ylabel('$V_w$ (L)')

    ax[0,1].plot(t_imp, p.get_val('traj.propelled_ascent.timeseries.T', 'N'), '.', color=c)
    ax[0,1].plot(t_exp, exp_out.get_val('traj.propelled_ascent.timeseries.T', 'N'), '-', color=c)
    ax[0,1].set_ylabel('T (N)')
    ax[0,1].set_ylim(bottom=0)

    ax[1,1].plot(t_imp, p.get_val('traj.propelled_ascent.timeseries.states:v', 'm/s'), '.', color=c)
    ax[1,1].plot(t_exp, exp_out.get_val('traj.propelled_ascent.timeseries.states:v', 'm/s'), '-', color=c)
    ax[1,1].set_ylabel('v (m/s)')
    ax[1,1].set_ylim(bottom=0)

    ax[1,0].set_xlabel('t (s)')
    ax[1,1].set_xlabel('t (s)')

    for i in range(4):
        ax.ravel()[i].grid(True, alpha=0.2)

    fig.tight_layout()
```

```python
def plot_states(p, exp_out, legend_loc='right', legend_ncol=3):
    fig, axes = plt.subplots(nrows=2, ncols=2, figsize=(12, 4), sharex=True)

    states = ['r', 'h', 'v', 'gam']
    units = ['m', 'm', 'm/s', 'deg']
    phases = ['propelled_ascent', 'ballistic_ascent', 'descent']

    time_imp = {'ballistic_ascent': p.get_val('traj.ballistic_ascent.timeseries.time'),
                'propelled_ascent': p.get_val('traj.propelled_ascent.timeseries.time'),
                'descent': p.get_val('traj.descent.timeseries.time')}

    time_exp = {'ballistic_ascent': exp_out.get_val('traj.ballistic_ascent.timeseries.time'),
                'propelled_ascent': exp_out.get_val('traj.propelled_ascent.timeseries.time'),
                'descent': exp_out.get_val('traj.descent.timeseries.time')}

    x_imp = {phase: {state: p.get_val(f"traj.{phase}.timeseries.states:{state}", unit)
                     for state, unit in zip(states, units)
                     }
             for phase in phases
             }

    x_exp = {phase: {state: exp_out.get_val(f"traj.{phase}.timeseries.states:{state}", unit)
                     for state, unit in zip(states, units)
                     }
             for phase in phases
             }

    for i, (state, unit) in enumerate(zip(states, units)):
        axes.ravel()[i].set_ylabel(f"{state} ({unit})" if state != 'gam' else f'$\gamma$ ({unit})')

        axes.ravel()[i].plot(time_imp['propelled_ascent'], x_imp['propelled_ascent'][state], '.', color=colors['pa'])
        axes.ravel()[i].plot(time_imp['ballistic_ascent'], x_imp['ballistic_ascent'][state], '.', color=colors['ba'])
        axes.ravel()[i].plot(time_imp['descent'], x_imp['descent'][state], '.', color=colors['d'])
        h1, = axes.ravel()[i].plot(time_exp['propelled_ascent'], x_exp['propelled_ascent'][state], '-', color=colors['pa'], label='Propelled Ascent')
        h2, = axes.ravel()[i].plot(time_exp['ballistic_ascent'], x_exp['ballistic_ascent'][state], '-', color=colors['ba'], label='Ballistic Ascent')
        h3, = axes.ravel()[i].plot(time_exp['descent'], x_exp['descent'][state], '-', color=colors['d'], label='Descent')

        if state == 'gam':
            axes.ravel()[i].yaxis.set_major_locator(mpl.ticker.MaxNLocator(nbins='auto', steps=[1, 1.5, 3, 4.5, 6, 9, 10]))
            axes.ravel()[i].set_yticks(np.arange(-90, 91, 45))

        axes.ravel()[i].grid(True, alpha=0.2)

    axes[1, 0].set_xlabel('t (s)')
    axes[1, 1].set_xlabel('t (s)')

    plt.figlegend(handles=[h1, h2, h3], loc=legend_loc, ncol=legend_ncol)

    fig.tight_layout()
```

```python
def plot_trajectory(p, exp_out, legend_loc='center right'):
    fig, axes = plt.subplots(nrows=1, ncols=1, figsize=(8, 6))

    time_imp = {'ballistic_ascent': p.get_val('traj.ballistic_ascent.timeseries.time'),
                'propelled_ascent': p.get_val('traj.propelled_ascent.timeseries.time'),
                'descent': p.get_val('traj.descent.timeseries.time')}

    time_exp = {'ballistic_ascent': exp_out.get_val('traj.ballistic_ascent.timeseries.time'),
                'propelled_ascent': exp_out.get_val('traj.propelled_ascent.timeseries.time'),
                'descent': exp_out.get_val('traj.descent.timeseries.time')}

    r_imp = {'ballistic_ascent': p.get_val('traj.ballistic_ascent.timeseries.states:r'),
             'propelled_ascent': p.get_val('traj.propelled_ascent.timeseries.states:r'),
             'descent': p.get_val('traj.descent.timeseries.states:r')}

    r_exp = {'ballistic_ascent': exp_out.get_val('traj.ballistic_ascent.timeseries.states:r'),
             'propelled_ascent': exp_out.get_val('traj.propelled_ascent.timeseries.states:r'),
             'descent': exp_out.get_val('traj.descent.timeseries.states:r')}

    h_imp = {'ballistic_ascent': p.get_val('traj.ballistic_ascent.timeseries.states:h'),
             'propelled_ascent': p.get_val('traj.propelled_ascent.timeseries.states:h'),
             'descent': p.get_val('traj.descent.timeseries.states:h')}

    h_exp = {'ballistic_ascent': exp_out.get_val('traj.ballistic_ascent.timeseries.states:h'),
             'propelled_ascent': exp_out.get_val('traj.propelled_ascent.timeseries.states:h'),
             'descent': exp_out.get_val('traj.descent.timeseries.states:h')}

    axes.plot(r_imp['propelled_ascent'], h_imp['propelled_ascent'], 'o', color=colors['pa'])
    axes.plot(r_imp['ballistic_ascent'], h_imp['ballistic_ascent'], 'o', color=colors['ba'])
    axes.plot(r_imp['descent'], h_imp['descent'], 'o', color=colors['d'])

    h1, = axes.plot(r_exp['propelled_ascent'], h_exp['propelled_ascent'], '-', color=colors['pa'], label='Propelled Ascent')
    h2, = axes.plot(r_exp['ballistic_ascent'], h_exp['ballistic_ascent'], '-', color=colors['ba'], label='Ballistic Ascent')
    h3, = axes.plot(r_exp['descent'], h_exp['descent'], '-', color=colors['d'], label='Descent')

    axes.set_xlabel('r (m)')
    axes.set_ylabel('h (m)')
    axes.set_aspect('equal', 'box')
    plt.figlegend(handles=[h1, h2, h3], loc=legend_loc)
    axes.grid(alpha=0.2)

    fig.tight_layout()
```

## Optimizing for Height

```python
from dymos.examples.water_rocket.phases import new_water_rocket_trajectory, set_sane_initial_guesses

p = om.Problem(model=om.Group())

traj, phases = new_water_rocket_trajectory(objective='height')
traj = p.model.add_subsystem('traj', traj)

p.driver = om.pyOptSparseDriver(optimizer='IPOPT', print_results=False)
p.driver.opt_settings['print_level'] = 4
p.driver.opt_settings['max_iter'] = 1000
p.driver.opt_settings['mu_strategy'] = 'monotone'
p.driver.declare_coloring(tol=1.0E-12)

# Finish Problem Setup
p.model.linear_solver = om.DirectSolver()

p.setup()
set_sane_initial_guesses(p, phases)

dm.run_problem(p, run_driver=True, simulate=True)

summary = summarize_results(p)
for key, entry in summary.items():
    print(f'{key}: {entry.value:6.4f} {entry.unit}')

sol_out = om.CaseReader('dymos_solution.db').get_case('final')
sim_out = om.CaseReader('dymos_simulation.db').get_case('final')
```

### Maximum Height Solution: Propulsive Phase

```python
plot_propelled_ascent(sol_out, sim_out)
```

## Maximum Height Solution: Height vs. Range

Note that the equations of motion used here are singular in vertical flight, so the launch angle (the initial flight path angle) was limited to 85 degrees.

```python
plot_trajectory(sol_out, sim_out, legend_loc='center right')
```

## Maximum Height Solution: State History

```python
plot_states(sol_out, sim_out, legend_loc='lower center', legend_ncol=3)
```

```python
from openmdao.utils.assert_utils import assert_near_equal

assert_near_equal(summary['Launch angle'].value, 85, 0.01)
assert_near_equal(summary['Empty mass'].value, 0.144, 0.01)
assert_near_equal(summary['Water volume'].value, 0.98, 0.01)
assert_near_equal(summary['Maximum height'].value, 53.5, 0.01)
```

# Optimizing for Range

```python
from dymos.examples.water_rocket.phases import new_water_rocket_trajectory, set_sane_initial_guesses

p = om.Problem(model=om.Group())

traj, phases = new_water_rocket_trajectory(objective='range')
traj = p.model.add_subsystem('traj', traj)

p.driver = om.pyOptSparseDriver(optimizer='IPOPT')
p.driver.opt_settings['print_level'] = 4
p.driver.opt_settings['max_iter'] = 1000
p.driver.opt_settings['mu_strategy'] = 'monotone'
p.driver.declare_coloring(tol=1.0E-12)

# Finish Problem Setup
p.model.linear_solver = om.DirectSolver()

p.setup()
set_sane_initial_guesses(p, phases)

dm.run_problem(p, run_driver=True, simulate=True)

summary = summarize_results(p)
for key, entry in summary.items():
    print(f'{key}: {entry.value:6.4f} {entry.unit}')

sol_out = om.CaseReader('dymos_solution.db').get_case('final')
sim_out = om.CaseReader('dymos_simulation.db').get_case('final')
```

## Maximum Range Solution: Propulsive Phase

```python
plot_propelled_ascent(sol_out, sim_out)
```

## Maximum Range Solution: Height vs. Range

```python
plot_trajectory(sol_out, sim_out, legend_loc='center')
```

## Maximum Range Solution: State History

```python
plot_states(sol_out, sim_out, legend_loc='lower center')
```

```python
from openmdao.utils.assert_utils import assert_near_equal

# Check results (tolerance is relative unless value is zero)
assert_near_equal(summary['Launch angle'].value, 46, 0.02)
assert_near_equal(summary['Flight angle at end of propulsion'].value, 38, 0.02)
assert_near_equal(summary['Empty mass'].value, 0.189, 1e-2)
assert_near_equal(summary['Water volume'].value, 1.026, 1e-2)
assert_near_equal(summary['Maximum range'].value, 85.11, 1e-2)
assert_near_equal(summary['Maximum height'].value, 23.08, 1e-2)
assert_near_equal(summary['Maximum velocity'].value, 41.31, 1e-2)
```

## References

```{bibliography}
:filter: docname in docnames
```


## Initial setup

```python
from google.colab import drive
drive.mount('/content/drive')
```

```python
import tensorflow as tf
print(tf.__version__)
# tensorflow version used is 2.7.0
import torch
print(torch.__version__)
# torch version used is 1.10+cu111
```

    2.8.0
    1.10.0+cu111

```python
!nvidia-smi
```

    Sat Feb 19 13:59:56 2022
    +-----------------------------------------------------------------------------+
    | NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2     |
    |-------------------------------+----------------------+----------------------+
    | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
    | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
    |                               |                      |               MIG M. |
    |===============================+======================+======================|
    |   0  Tesla P100-PCIE...  Off  | 00000000:00:04.0 Off |                    0 |
    | N/A   36C    P0    26W / 250W |      0MiB / 16280MiB |      0%      Default |
    |                               |                      |                  N/A |
    +-------------------------------+----------------------+----------------------+

    +-----------------------------------------------------------------------------+
    | Processes:                                                                  |
    |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
    |        ID   ID                                                   Usage      |
    |=============================================================================|
    |  No running processes found                                                 |
    +-----------------------------------------------------------------------------+

```python
# Other imports
! pip install tensorflow_addons
! pip install tensorflow_io

import os
from tensorflow.keras.layers import *
from tensorflow.keras.models import *
from tensorflow import keras as tfkeras
import matplotlib.pyplot as plt
from imutils import paths
from tqdm import tqdm
import tensorflow as tf
import tensorflow_addons as tfa
import tensorflow_datasets as tfds
import tensorflow_io as tfio
import tensorflow_hub as hub
import numpy as np
import cv2
import pandas as pd
from sklearn.preprocessing import LabelEncoder
from sklearn.cluster import KMeans
from sklearn.metrics.pairwise import cosine_similarity as cos
from sympy.utilities.iterables import multiset_permutations
from sklearn.metrics import accuracy_score, f1_score, confusion_matrix
from IPython.display import Image, display

import zipfile
import concurrent.futures

# Random seed fix
random_seed = 42
tf.random.set_seed(random_seed)
np.random.seed(random_seed)
```

    Requirement already satisfied: tensorflow_addons in /usr/local/lib/python3.7/dist-packages (0.16.1)
    Requirement already satisfied: typeguard>=2.7 in /usr/local/lib/python3.7/dist-packages (from tensorflow_addons) (2.7.1)
    Requirement already satisfied: tensorflow_io in /usr/local/lib/python3.7/dist-packages (0.24.0)
    Requirement already satisfied: tensorflow-io-gcs-filesystem==0.24.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow_io) (0.24.0)

## Dataset gathering and preparation

```python
Imagenet_images_train = list(paths.list_images("/content/drive/MyDrive/TEM image datasets/2022-1000-ImageNet"))
train_images_directory_select = np.random.choice(Imagenet_images_train, 832, replace=False)
print(len(Imagenet_images_train))
print(len(train_images_directory_select))
```

    1000
    832

```python
CEM_images_train = list(paths.list_images("/content/drive/MyDrive/TEM image datasets/2021-CEM500K/jpg"))
train_images_directory_select = np.random.choice(CEM_images_train, 832, replace=False)
print(len(CEM_images_train))
print(len(train_images_directory_select))
```

    10000
    832

```python
%cd /content/drive/MyDrive/nanowire-morphology-classification-project/
```

```python
training_batch_size = 64

imageSize=224
```

```python
# Augmentation utilities (differs from the original implementation)
# Adapted from: https://arxiv.org/pdf/2002.05709.pdf (Appendxi A)
# corresponding GitHub: https://github.com/google-research/simclr/)

class CustomAugment(object):
    def __call__(self, sample):

        # Random flips
        sample = self._random_apply(tf.image.flip_left_right, sample, p=0.5)

        # Random crop always
        sample = self._random_apply(self._random_crop, sample, p=1.0)

        # Randomly apply transformation (color distortions) with probability p.
        sample = self._random_apply(self._color_jitter, sample, p=0.8)

        return sample

    def _random_crop(self, x):

        # distance from left to right, from bottom to top
        l = tf.random.uniform([], minval=int(imageSize * 0.25), maxval=int(imageSize * 0.5), dtype=tf.int32)             # 25 as min bound, so that the cropped images are not too small
        # l = int(0.5 * imageSize)                                                                              # use this line if the crop box size is constant

        # lower left corner
        y1 = tf.random.uniform([], minval=0, maxval= imageSize - 1 - l, dtype=tf.int32)
        x1 = tf.random.uniform([], minval=0, maxval= imageSize - 1 - l, dtype=tf.int32)

        x = tf.image.crop_to_bounding_box(x, y1, x1, l, l)
        x = tf.image.resize(x, size=[imageSize, imageSize])
        return x

    def _color_jitter(self, x, s=1):
        # one can also shuffle the order of following augmentations
        # each time they are applied.
        # x = tf.image.random_brightness(x, max_delta=0.2*s)
        x = tf.image.random_contrast(x, lower=1-0.8*s, upper=1+0.8*s)
        x = tf.image.random_saturation(x, lower=1-0.8*s, upper=1+0.8*s)
        x = tf.image.random_hue(x, max_delta=0.2*s)
        x = tf.clip_by_value(x, 0, 1)
        return x

    def _random_apply(self, func, x, p):
        return tf.cond(
          tf.less(tf.random.uniform([], minval=0, maxval=1, dtype=tf.float32),
                  tf.cast(p, tf.float32)),
          lambda: func(x),
          lambda: x)
```

```python
# Build the augmentation pipeline
data_augmentation = Sequential([Lambda(CustomAugment())])
```

```python
# Image preprocessing utils
@tf.function
def parse_images(image_path):
    image_string = tf.io.read_file(image_path)
    image = tf.image.decode_jpeg(image_string, channels=3)
    image = tf.image.convert_image_dtype(image, tf.float32)
    image = tf.image.resize(image, size=[imageSize, imageSize])

    return image
```

```python
# Create TensorFlow dataset for training
train_ds = tf.data.Dataset.from_tensor_slices(train_images_directory_select)
train_ds = (
    train_ds
    .map(parse_images, num_parallel_calls=tf.data.experimental.AUTOTUNE)
    .batch(training_batch_size
           , drop_remainder=True
           )
    .prefetch(tf.data.experimental.AUTOTUNE)
)
```

## Initiate our self-supervised model

```python
Resnet50_transfer = tf.keras.applications.ResNet50(
    include_top=False,
    weights="imagenet",
    input_tensor=None,
    input_shape=(imageSize, imageSize, 3),
    pooling=None,
)

Resnet50_transfer.trainable = True
```

```python
# Architecture for resnet as base architecture
def get_resnet_self_supervise_model(hidden_1, hidden_2, hidden_3):
    base_model = Resnet50_transfer
    base_model.trainable = True
    inputs = Input((imageSize, imageSize, 3))
    h = base_model(inputs, training=True)
    h = GlobalAveragePooling2D()(h)

    projection_1 = Dense(hidden_1)(h)
    projection_1 = Activation("relu")(projection_1)
    projection_1 = BatchNormalization(epsilon=0.001)(projection_1)
    projection_2 = Dense(hidden_2)(projection_1)
    projection_2 = Activation("relu")(projection_2)
    projection_2 = BatchNormalization(epsilon=0.001)(projection_2)
    projection_3 = Dense(hidden_3)(projection_2)
    projection_3 = BatchNormalization(epsilon=0.001)(projection_3)

    resnet_model = Model(inputs, projection_3)

    return resnet_model
```

## Training self-supervised model

```python
#### use this train_step block for Simclr implementation

tf.config.run_functions_eagerly(True)

@tf.function
def train_step(xis, xjs, model, optimizer, criterion, temperature):
    with tf.GradientTape() as tape:
        projections_1_original = model(xis)
        projections_2_original = model(xjs)

        # simclr derivatives uses layer norm
        # normalize projection feature vectors            along the feature dimension, this is a l2_norm
        projections_1 = tf.math.l2_normalize(projections_1_original, axis=1)
        projections_2 = tf.math.l2_normalize(projections_2_original, axis=1)
        similarities = (tf.matmul(projections_1, projections_2, transpose_b=True) / temperature)

        # the temperature-scaled similarities are used as logits for cross-entropy
        contrastive_labels = tf.range(training_batch_size)
        loss = criterion(
            tf.concat([contrastive_labels, contrastive_labels], axis=0),
            tf.concat([similarities, tf.transpose(similarities)], axis=0))

    gradients = tape.gradient(loss, model.trainable_variables)
    optimizer.apply_gradients(zip(gradients, model.trainable_variables))

    return loss
```

```python
#### use this train_step block for Barlow-Twins implementation

tf.config.run_functions_eagerly(True)

@tf.function
def train_step(xis, xjs, model, optimizer, criterion, temperature):
    with tf.GradientTape() as tape:
        projections_1_original = model(xis)
        projections_2_original = model(xjs)

        # batchnorm for barlow twins
        # use these two lines when doing mini-batch gradient descent
        projections_1 = (projections_1_original - tf.reduce_mean(projections_1_original, axis=0)) / tf.math.reduce_std(projections_1_original, axis=0)
        projections_2 = (projections_2_original - tf.reduce_mean(projections_2_original, axis=0)) / tf.math.reduce_std(projections_2_original, axis=0)

        # use these two lines when doing stochastic gradient descent (i.e. batch size = 1)
        # projections_1 = projections_1_original
        # projections_2 = projections_2_original

        # the cross correlation of image representations should be the identity matrix

        feature_dim = tf.cast(tf.shape(projections_1)[1], tf.float32)
        cross_correlation = (tf.matmul(projections_1, projections_2, transpose_a=True) / training_batch_size)
        target_cross_correlation = tf.eye(feature_dim)
        squared_errors = (target_cross_correlation - cross_correlation) ** 2

        # invariance loss = average diagonal error
        # redundancy reduction loss = average off-diagonal error
        invariance_loss = (tf.reduce_sum(squared_errors * tf.eye(feature_dim)) / feature_dim)
        redundancy_reduction_loss = tf.reduce_sum(squared_errors * (1 - tf.eye(feature_dim))) / (feature_dim * (feature_dim - 1))

        # use redundancy_redunction_weight of 0.005 from the paper, not a sensitive parameter
        loss = invariance_loss + 0.005 * redundancy_reduction_loss

    gradients = tape.gradient(loss, model.trainable_variables)
    optimizer.apply_gradients(zip(gradients, model.trainable_variables))

    return loss
```

```python
def train_model(model, dataset, optimizer, criterion,
                 temperature, epochs=100):
    step_wise_loss = []
    epoch_wise_loss = []

    epoch = 1
    for k in tqdm(range(epochs)):
      i = 0
      for image_batch in dataset:
          a = data_augmentation(image_batch)
          b = data_augmentation(image_batch)

          loss = train_step(a, b, model, optimizer, criterion, temperature)
          step_wise_loss.append(loss)
          i += 1
          print("batch: {} loss: {:.5f}".format(i, np.mean(step_wise_loss)))

      epoch_wise_loss.append(np.mean(step_wise_loss))
      if epoch % 20 == 0:
        model.save_weights('barlow_ImageNet_batch%i_project128_64_1024_seed%i.h5' % (training_batch_size, random_seed))
        print("epoch: {} loss: {:.5f}".format(epoch, np.mean(step_wise_loss)))
      epoch += 1

    return epoch_wise_loss, model
```

```python
criterion = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)

# the learning rate decay schedule in the original implementation of SimCLR and Barlow-Twins
decay_steps = 1000
lr_decayed_fn = tf.keras.experimental.CosineDecay(
    initial_learning_rate=0.2, decay_steps=decay_steps)
optimizer = tf.keras.optimizers.SGD(lr_decayed_fn)

training_model = get_resnet_self_supervise_model(128, 64, 1024)

epoch_wise_loss, resnet_model  = train_model(training_model, train_ds, optimizer, criterion,
                 temperature=0.1, epochs=200)

plt.plot(epoch_wise_loss)
plt.title("training loss versus epochs")
plt.xlabel('epoch')
plt.ylabel('loss')
plt.show()
```

      0%|          | 0/200 [00:00<?, ?it/s]

    batch: 1 loss: 0.57619
    batch: 2 loss: 0.43121
    batch: 3 loss: 0.45222
    batch: 4 loss: 0.39469
    batch: 5 loss: 0.38681
    batch: 6 loss: 0.33074
    batch: 7 loss: 0.35192
    batch: 8 loss: 0.34164
    batch: 9 loss: 0.35104
    batch: 10 loss: 0.36822
    batch: 11 loss: 0.36665
    batch: 12 loss: 0.35966

      0%|          | 1/200 [11:33<38:21:06, 693.80s/it]

    batch: 13 loss: 0.33729
    batch: 1 loss: 0.32755
    batch: 2 loss: 0.34130
    batch: 3 loss: 0.33533
    batch: 4 loss: 0.32271
    batch: 5 loss: 0.31440
    batch: 6 loss: 0.31166
    batch: 7 loss: 0.29918
    batch: 8 loss: 0.28848
    batch: 9 loss: 0.27715
    batch: 10 loss: 0.27190
    batch: 11 loss: 0.28531
    batch: 12 loss: 0.27678

      1%|          | 2/200 [22:50<37:36:52, 683.90s/it]

    batch: 13 loss: 0.27652
    batch: 1 loss: 0.27942
    batch: 2 loss: 0.28107
    batch: 3 loss: 0.27381
    batch: 4 loss: 0.26927
    batch: 5 loss: 0.26277
    batch: 6 loss: 0.25918
    batch: 7 loss: 0.25793
    batch: 8 loss: 0.25574
    batch: 9 loss: 0.24996
    batch: 10 loss: 0.24518
    batch: 11 loss: 0.24362
    batch: 12 loss: 0.23837

      2%|▏         | 3/200 [34:09<37:17:54, 681.60s/it]

    batch: 13 loss: 0.23610
    batch: 1 loss: 0.23580
    batch: 2 loss: 0.23282
    batch: 3 loss: 0.22809
    batch: 4 loss: 0.22476
    batch: 5 loss: 0.22062
    batch: 6 loss: 0.21746
    batch: 7 loss: 0.21734
    batch: 8 loss: 0.21505
    batch: 9 loss: 0.21278
    batch: 10 loss: 0.21072
    batch: 11 loss: 0.20777
    batch: 12 loss: 0.20842

      2%|▏         | 4/200 [45:34<37:11:09, 683.01s/it]

    batch: 13 loss: 0.20911
    batch: 1 loss: 0.20676
    batch: 2 loss: 0.20360
    batch: 3 loss: 0.20137
    batch: 4 loss: 0.19841
    batch: 5 loss: 0.19752
    batch: 6 loss: 0.19467
    batch: 7 loss: 0.19474
    batch: 8 loss: 0.19202
    batch: 9 loss: 0.18932
    batch: 10 loss: 0.18666
    batch: 11 loss: 0.18378
    batch: 12 loss: 0.18168

      2%|▎         | 5/200 [57:01<37:04:26, 684.44s/it]

    batch: 13 loss: 0.17976
    batch: 1 loss: 0.17924
    batch: 2 loss: 0.17720
    batch: 3 loss: 0.17774
    batch: 4 loss: 0.17523
    batch: 5 loss: 0.17427
    batch: 6 loss: 0.17216
    batch: 7 loss: 0.17020
    batch: 8 loss: 0.16820
    batch: 9 loss: 0.16731
    batch: 10 loss: 0.16708
    batch: 11 loss: 0.16545
    batch: 12 loss: 0.16392

      3%|▎         | 6/200 [1:08:44<37:13:16, 690.70s/it]

    batch: 13 loss: 0.16309
    batch: 1 loss: 0.16115
    batch: 2 loss: 0.16031
    batch: 3 loss: 0.15966
    batch: 4 loss: 0.15798
    batch: 5 loss: 0.15774
    batch: 6 loss: 0.15649
    batch: 7 loss: 0.15519
    batch: 8 loss: 0.15473
    batch: 9 loss: 0.15320
    batch: 10 loss: 0.15180
    batch: 11 loss: 0.15050
    batch: 12 loss: 0.14895

      4%|▎         | 7/200 [1:20:29<37:16:23, 695.25s/it]

    batch: 13 loss: 0.14764
    batch: 1 loss: 0.14642
    batch: 2 loss: 0.14534
    batch: 3 loss: 0.14427
    batch: 4 loss: 0.14284
    batch: 5 loss: 0.14209
    batch: 6 loss: 0.14105
    batch: 7 loss: 0.14056
    batch: 8 loss: 0.13948
    batch: 9 loss: 0.13827
    batch: 10 loss: 0.13721
    batch: 11 loss: 0.13623
    batch: 12 loss: 0.13502
    batch: 13 loss: 0.13429

      4%|▍         | 8/200 [1:32:51<37:52:23, 710.12s/it]

    batch: 1 loss: 0.13313
    batch: 2 loss: 0.13211
    batch: 3 loss: 0.13158
    batch: 4 loss: 0.13055
    batch: 5 loss: 0.12949
    batch: 6 loss: 0.12854
    batch: 7 loss: 0.12752
    batch: 8 loss: 0.12647
    batch: 9 loss: 0.12582
    batch: 10 loss: 0.12489
    batch: 11 loss: 0.12422
    batch: 12 loss: 0.12546

      4%|▍         | 9/200 [1:44:56<37:55:59, 714.97s/it]

    batch: 13 loss: 0.12449
    batch: 1 loss: 0.12360
    batch: 2 loss: 0.12267
    batch: 3 loss: 0.12226
    batch: 4 loss: 0.12221
    batch: 5 loss: 0.12178
    batch: 6 loss: 0.12156
    batch: 7 loss: 0.12094
    batch: 8 loss: 0.12008
    batch: 9 loss: 0.11936
    batch: 10 loss: 0.11854
    batch: 11 loss: 0.11779
    batch: 12 loss: 0.11741
    batch: 13 loss: 0.11754

      5%|▌         | 10/200 [1:57:18<38:10:26, 723.29s/it]

    batch: 1 loss: 0.11689
    batch: 2 loss: 0.11614
    batch: 3 loss: 0.11538
    batch: 4 loss: 0.11458
    batch: 5 loss: 0.11444
    batch: 6 loss: 0.11371
    batch: 7 loss: 0.11293
    batch: 8 loss: 0.11231
    batch: 9 loss: 0.11165
    batch: 10 loss: 0.11144
    batch: 11 loss: 0.11155
    batch: 12 loss: 0.11085

      6%|▌         | 11/200 [2:09:18<37:54:46, 722.15s/it]

    batch: 13 loss: 0.11027
    batch: 1 loss: 0.11002
    batch: 2 loss: 0.10948
    batch: 3 loss: 0.10879
    batch: 4 loss: 0.10941
    batch: 5 loss: 0.10988
    batch: 6 loss: 0.11054
    batch: 7 loss: 0.11070
    batch: 8 loss: 0.11029
    batch: 9 loss: 0.10982
    batch: 10 loss: 0.10929
    batch: 11 loss: 0.10888
    batch: 12 loss: 0.10856

      6%|▌         | 12/200 [2:21:13<37:36:00, 720.00s/it]

    batch: 13 loss: 0.10822
    batch: 1 loss: 0.10772
    batch: 2 loss: 0.10748
    batch: 3 loss: 0.10699
    batch: 4 loss: 0.10653
    batch: 5 loss: 0.10756
    batch: 6 loss: 0.10694
    batch: 7 loss: 0.10635
    batch: 8 loss: 0.10574
    batch: 9 loss: 0.10526
    batch: 10 loss: 0.10480
    batch: 11 loss: 0.10450
    batch: 12 loss: 0.10398

      6%|▋         | 13/200 [2:32:56<37:07:57, 714.85s/it]

    batch: 13 loss: 0.10352
    batch: 1 loss: 0.10295
    batch: 2 loss: 0.10258
    batch: 3 loss: 0.10211
    batch: 4 loss: 0.10166
    batch: 5 loss: 0.10121
    batch: 6 loss: 0.10090
    batch: 7 loss: 0.10036
    batch: 8 loss: 0.10023
    batch: 9 loss: 0.09988
    batch: 10 loss: 0.09948
    batch: 11 loss: 0.09897
    batch: 12 loss: 0.09883

      7%|▋         | 14/200 [2:44:39<36:45:24, 711.42s/it]

    batch: 13 loss: 0.09841
    batch: 1 loss: 0.09804
    batch: 2 loss: 0.09791
    batch: 3 loss: 0.09746
    batch: 4 loss: 0.09700
    batch: 5 loss: 0.09752
    batch: 6 loss: 0.09725
    batch: 7 loss: 0.09687
    batch: 8 loss: 0.09680
    batch: 9 loss: 0.09636
    batch: 10 loss: 0.09590
    batch: 11 loss: 0.09586
    batch: 12 loss: 0.09547

      8%|▊         | 15/200 [2:56:25<36:27:56, 709.60s/it]

    batch: 13 loss: 0.09537
    batch: 1 loss: 0.09499
    batch: 2 loss: 0.09454
    batch: 3 loss: 0.09417

```python
plt.plot(epoch_wise_loss)
plt.title("training loss versus epochs")
plt.show()
```


## Prerequisites

```python
# Ensure you have `gif` python package installed
!pip install -U gif
```

    Collecting gif
      Downloading gif-3.0.0.tar.gz (5.0 kB)
    Requirement already satisfied, skipping upgrade: Pillow>=7.1.2 in /home/avani/anaconda3/lib/python3.8/site-packages (from gif) (7.2.0)
    Building wheels for collected packages: gif
      Building wheel for gif (setup.py) ... done
      Created wheel for gif: filename=gif-3.0.0-py3-none-any.whl size=4816 sha256=3fe31bc766f5b80a7172a265a9f045a313fece269085946e5cab15c8a742bf3a
      Stored in directory: /home/avani/.cache/pip/wheels/d8/db/4e/e0ce5209665322902834f4e88fa987f4dfa0c08271368ef098
    Successfully built gif
    Installing collected packages: gif
    Successfully installed gif-3.0.0

```python
import matplotlib
print(matplotlib.__version__)
```

    3.2.2

#### Import utility functions

We have provided a bunch of helper functions in the utils.ipynb notebook that allows for commonly used functions that are not central to the discussion of topic at hand, like plotting functions etc. Feel free to explore the functions and their implementations.

The following cell imports the functions written in the utils notebook. You can ignore this part of the notebook

```python
import io, os, sys, types
from IPython import get_ipython
from nbformat import read
from IPython.core.interactiveshell import InteractiveShell

def find_notebook(fullname, path=None):
    """find a notebook, given its fully qualified name and an optional path

    This turns "foo.bar" into "foo/bar.ipynb"
    and tries turning "Foo_Bar" into "Foo Bar" if Foo_Bar
    does not exist.
    """
    name = fullname.rsplit('.', 1)[-1]
    if not path:
        path = ['']
    for d in path:
        nb_path = os.path.join(d, name + ".ipynb")
        #print('searching: %s'%nb_path)
        if os.path.isfile(nb_path):
            return nb_path
        # let import Notebook_Name find "Notebook Name.ipynb"
        nb_path = nb_path.replace("_", " ")
        #print('searching: %s' % nb_path)
        if os.path.isfile(nb_path):
            return nb_path

class NotebookLoader(object):
    """Module Loader for Jupyter Notebooks"""
    def __init__(self, path=None):
        self.shell = InteractiveShell.instance()
        self.path = path

    def load_module(self, fullname):
        """import a notebook as a module"""
        path = find_notebook(fullname, self.path)

        print ("importing Jupyter notebook from %s" % path)

        # load the notebook object
        with io.open(path, 'r', encoding='utf-8') as f:
            nb = read(f, 4)

        # create the module and add it to sys.modules
        # if name in sys.modules:
        #    return sys.modules[name]
        mod = types.ModuleType(fullname)
        mod.__file__ = path
        mod.__loader__ = self
        mod.__dict__['get_ipython'] = get_ipython
        sys.modules[fullname] = mod

        # extra work to ensure that magics that would affect the user_ns
        # actually affect the notebook module's ns
        save_user_ns = self.shell.user_ns
        self.shell.user_ns = mod.__dict__

        #print('Found %d cells'%len(nb.cells))
        try:
          for cell in nb.cells:
            if cell.cell_type == 'code':
                # transform the input to executable Python
                code = self.shell.input_transformer_manager.transform_cell(cell.source)
                # run the code in themodule
                exec(code, mod.__dict__)
        finally:
            self.shell.user_ns = save_user_ns
        return mod

class NotebookFinder(object):
    """Module finder that locates Jupyter Notebooks"""
    def __init__(self):
        self.loaders = {}

    def find_module(self, fullname, path=None):
        nb_path = find_notebook(fullname, path)
        if not nb_path:
            return

        key = path
        if path:
            # lists aren't hashable
            key = os.path.sep.join(path)

        if key not in self.loaders:
            self.loaders[key] = NotebookLoader(path)
        return self.loaders[key]

#  register the NotebookFinder with sys.meta_path
print('running importer')
sys.meta_path.append(NotebookFinder())
```

    running importer

# Gradient Descent

Gradient descent is a first-order iterative optimization algorithm to find a local minima (or global minima in case of convex function) of a differentiable function. To achieve this, we take steps proportional to the negative of the gradient (or approximate gradient) of the function at the current point.

Conveniently, the objective function of linear regression algorithm is strictly convex and differentiable. Hence, we can use gradient descent optimization to find optimal values of our hypothesis function.

We revisit the equation to minimize the objective function of linear regression.

\begin{equation}
\min_{\mathbf{w}} \mathbf{J(w)} = \min_{\mathbf{w}} \frac{1}{\mathbf{N}} \sum_\mathbf{i=1}^\mathbf{N} (\mathbf{y_{i} - w^{\top}x_i)^{2}}
\end{equation}

Our gradient descent update for weights is:

\begin{equation}
\mathbf{w^{k+1}} \leftarrow \mathbf{w^k} - \eta \nabla{\mathbf{J}}
\end{equation}

where $\eta$ is our learning rate and,

\begin{equation}
\nabla{\mathbf{J}} = \min_{\mathbf{w}} \frac{2}{\mathbf{N}} \sum_\mathbf{i=1}^\mathbf{N} (\mathbf{y_{i} - w^{\top}x_i)(-x_i)}
\end{equation}

We will look at Standard (Batch) Gradient Descent, Mini-batch, and Stochastic versions of Gradient Descent.

```python
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
import seaborn as sb; sb.set();
from utils import plot_convex_loss_and_predict_line, mse_error
import gif
```

    importing Jupyter notebook from utils.ipynb

```python
num_samples = 100
# Generating regression dataset
X_train, Y_train = make_regression(n_samples=num_samples, n_features=1, n_informative=1, noise=20, random_state=2017)

# Adding bias as a column in X
bias_ones = np.ones((num_samples, 1))
X_train = np.hstack((bias_ones, X_train))
Y_train = Y_train.reshape(-1, 1)
```

```python
print("X_train: num_samples {}, num_features {}" .format((X_train.shape)[0], (X_train.shape)[1]))
```

    X_train: num_samples 100, num_features 2

```python
plt.figure(figsize=(7,7));
plt.scatter(X_train[:, 1], Y_train, label='Point', color='green')
plt.suptitle('X_train vs Y_train')
plt.xlabel("X_train")
plt.ylabel("Y_train");
```

```python
def train(X_train, Y_train, num_epochs, pts_per_batch, learning_rate, gd_type):
    training_errors = []
    frames = []

    # Initializing weights
    w = np.array([-15, 40]) + np.random.rand(2)
    w = w.reshape(-1, 1)

    weights_history = []
    Y_prediction = []
    append_flag = False

    # Getting number of batches/epochs to run
    num_train_datapts = len(Y_train)
    mini_batches = int(num_train_datapts/pts_per_batch)

    # If mini_batches = num_training_datapoints, then standard GD is being executed.
    # If mini_batches > 1 and < num_training_datapoints, and an integer, then mini-batch GD is being run.
    # If mini_batches = 1, then SGD is being run.

    num_batches = int(mini_batches * num_epochs)
    print(f'Num_batches: {num_batches} Mini-batches: {mini_batches} Num_epochs: {num_epochs} Points_per_batch {pts_per_batch} train_point {num_train_datapts}')

    # Do gradient descent for 'num_batches' batches (or mini-batches)
    temp_mse_history = []
    for i in range(1, num_batches+1):

        if num_train_datapts == pts_per_batch:
            # Sampling training data points for batch GD
            X_input = X_train
            Y_true = Y_train

        else:
            # Sampling training data points for stochastic or mini-batch GD
            train_idxs = np.random.permutation(num_train_datapts)
            train_idxs = train_idxs[:pts_per_batch]
            X_input = X_train[train_idxs]
            Y_true = Y_train[train_idxs]

        y_pred = np.dot(X_input, w)
        # Gradient of l2 loss w.r.t w
        grad_w = -np.dot(X_input.T, Y_true - y_pred)

        temp_mse = np.mean(0.5 * (Y_true - y_pred)**2)

        if num_train_datapts == pts_per_batch:
          # Appending values during batch GD
          mse = temp_mse
          append_flag = True

        elif pts_per_batch != 1 and pts_per_batch != num_train_datapts:
            temp_mse_history.append(temp_mse)
            if ((i % mini_batches == 0 and mini_batches != 1) or i == 1) and pts_per_batch != 1:
                y_pred = np.dot(X_train, w)
                mse = np.mean(temp_mse_history)
                temp_mse_history = []
                append_flag = True

        elif pts_per_batch == 1:
            y_pred = np.dot(X_train, w)
            mse = np.mean(0.5 * (Y_train - y_pred)**2)
            append_flag = True

        # Appending errors, weights, and predictions only if an epoch is completed.
        # This will help us in our visualization of GD.
        if append_flag:
            training_errors.append(mse)
            weights_history.append(w)
            Y_prediction.append(y_pred)
            append_flag = False

        # Update the weights
        w -= (2*learning_rate/pts_per_batch) * grad_w
        w = np.round(w, decimals=2)

    return training_errors, weights_history, Y_prediction
```

```python
def generate_GD_gifs(training_errors, weights_history, X_train, Y_train, Y_prediction, gd_type='mini-batch'):
    frames = []

    for i in range(len(training_errors)):
        frame = plot_convex_loss_and_predict_line(training_errors, weights_history, X_train, Y_train, Y_prediction, i)
        frames.append(frame)
    gif.save(frames, "content/"+gd_type+"_GD.gif", duration=500)
    print('gifs generated')
```

Vanilla gradient descent, aka batch gradient descent, calculates the gradient of the loss function w.r.t to the parameters $w$ for all the training examples. We need to only make one update per epoch. However, for a large dataset, batch GD can be slow and sometimes intractable if the datasets don't fit in memory. Also, we can't update our model online, i.e., with new examples on-the-fly. The pseudo code is as follows:

For a dataset $D = (x_i, y_i)_i^m$ of $m$ training examples,

1.  Initializing weights and biases.
2.  For every epoch $\in [1,...., \text{max_epochs}]$:
     * shuffle D to prevent cycles
     * compute prediction $y_{\text{prediction}} = activation(X_{\text{input}} \times weights)$
     * compute loss $L = (y_{\text{true}}, y_{\text{predicted}})$
     * compute gradients $\Delta \mathbf{w} = - \nabla_{Loss} \mathbf{w}$
     * update parameters $\mathbf{w} :=\mathbf{w}+\Delta \mathbf{w}$

Batch GD is guaranteed to converge to the global minima for convex loss surfaces and to a local minima for non-convex surfaces.

```python
# Batch GD
training_errors, weights_history, Y_prediction = train(X_train, Y_train, num_epochs=20, pts_per_batch=100, learning_rate=0.1, gd_type='batch')

generate_GD_gifs(training_errors, weights_history, X_train, Y_train, Y_prediction, gd_type='batch')
```

    Num_batches: 20 Mini-batches: 1 Num_epochs: 20 Points_per_batch 100 train_point 100
    gifs generated

Visualizing Batch Gradient Descent

```python
# Stochastic GD
training_errors, weights_history, Y_prediction = train(X_train, Y_train, num_epochs=1, pts_per_batch=1, learning_rate=0.1, gd_type='SGD')
generate_GD_gifs(training_errors, weights_history, X_train, Y_train, Y_prediction, gd_type='SGD')
```

    Num_batches: 100 Mini-batches: 100 Num_epochs: 1 Points_per_batch 1 train_point 100
    gifs generated

In Stochastic GD, parameter updates are performed for every training example. Due to frequent updates, this leads to high variance that causes the loss function to fluctuate heavily.

These fluctuations can sometimes help to find better local minima. On the other hand, these fluctuations can lead to longer convergence time.

For a dataset $D = (x_i, y_i)_i^m$ of $m$ training examples,

1.  Initializing weights and biases.
2.  For every epoch $\in [1,...., \text{max_epochs}]$:
>2.1  For iteration $\text{itr} \in [1,....., m]$:
     * Draw random examples with replacement: $(x_i, y_i) \in D$
     * compute prediction $y_i^{\text{prediction}} = activation(x_i \times weights)$
     * compute loss $L_i = (y_i^{\text{true}}, y_i^{\text{predicted}})$
     * compute gradients $\Delta \mathbf{w} = - \nabla_{Loss_i} \mathbf{w}$
     * update parameters $\mathbf{w} :=\mathbf{w}+\Delta \mathbf{w}$

Visualizing Stochastic Gradient Descent

Minibatch GD offers the best of both worlds, i.e., a trade-off between the stochastic GD that perform updates based on a single training example and Batch GD which takes all of the dataset at once.

In mini-batch, update is performed for every mini-batch of k training examples. Hence, we would have to take more updates per epoch as compared to batch GD.

The updates reduces the variances of parameter updates leading to a more stable convergence. By using highly optimized matrix computation from state-of-the-art machine learning libraries, mini-batch is typically the algorithm of choice when training a neural network.

For a dataset $D = (x_i, y_i)_i^m$ of $m$ training examples,

Initializing weights and biases.

For every epoch $\in [1,...., \text{max_epochs}]$:

For iteration $i \in [1,....., n]$: (where n is the mini-batch size)

Draw random examples with replacement: $(x_i, y_i) \in D$

compute prediction $y_i^{\text{prediction}} = activation(x_i \times weights)$

compute loss $L_i = (y_i^{\text{true}}, y_i^{\text{predicted}})$

compute gradients $\Delta \mathbf{w} = - \nabla_{L_i} \mathbf{w}$

update parameters $\mathbf{w} :=\mathbf{w}+\Delta \mathbf{w}$

One can also draw examples without replacement. Drawing examples w/o replacement is more commonly used in modern deep learning libraries. However, we shall look at mini-batch GD where we draw samples at random.

```python
# Mini-batch GD
training_errors, weights_history, Y_prediction = train(X_train, Y_train, num_epochs=20, pts_per_batch=20, learning_rate=0.1, gd_type='mini-batch')
generate_GD_gifs(training_errors, weights_history, X_train, Y_train, Y_prediction, gd_type='mini-batch')
```

    Num_batches: 100 Mini-batches: 5 Num_epochs: 20 Points_per_batch 20 train_point 100
    gifs generated

Visualizing mini-batch Gradient Descent

# Visualizing the effect of learning rate on GD

Now that we have seen the effect of changing the number of batches in a gradient descent, we look at the effect of different learning rates, another important factor that actively influences gradient descent. Previously, we had used 0.1 as our learning rate to train the linear regression model. Let's use Batch gradient descent as an example with lr rate $0.005$ and $1.1. You can try other values for other GD types.

```python
# Batch GD
training_errors, weights_history, Y_prediction = train(X_train, Y_train, num_epochs=20, pts_per_batch=100, learning_rate=0.005, gd_type='batch')
generate_GD_gifs(training_errors, weights_history, X_train, Y_train, Y_prediction, gd_type='batch-small-lr')
```

    Num_batches: 20 Mini-batches: 1 Num_epochs: 20 Points_per_batch 100 train_point 100
    gifs generated

```python
# Batch GD
training_errors, weights_history, Y_prediction = train(X_train, Y_train, num_epochs=15, pts_per_batch=100, learning_rate=1.1, gd_type='batch')
generate_GD_gifs(training_errors, weights_history, X_train, Y_train, Y_prediction, gd_type='batch-large-lr')
```

    Num_batches: 15 Mini-batches: 1 Num_epochs: 15 Points_per_batch 100 train_point 100
    gifs generated


# writefile Magic Function

```python
%%writefile test.txt
this is written from a jupyter notebook
```

    Writing test.txt

# The above magic function creates a file if it dint have and writes next lines in the file. if the file exists it will clean up and write the new data.

```python
%%writefile test.txt
file exist check from jupyter notebook
```

    Overwriting test.txt

# ls magic function

```python
%ls
```

    Magic Functions.ipynb           processing_data.ipynb
    My First NoteBook.ipynb         test.txt
    Writefile Magic Function.ipynb

# HTML Magic Function

```python
%%HTML
<body> Hello</body>
```

<body> Hello</body>

# latex magic function used to write the equation

```latex
%%latex
\begin{align}
Gradient : \nabla J = -2H^T (Y-HW)
\end{align}
```

\begin{align}
Gradient : \nabla J = -2H^T (Y-HW)
\end{align}

```python

```