```
!pip install trax
```

    Collecting trax
      Downloading https://files.pythonhosted.org/packages/a8/04/0c04116bbb372f459ad0a73bf306c5000f9fd63a8419bb179381f54773aa/trax-1.3.5-py2.py3-none-any.whl (416kB)
         |████████████████████████████████| 419kB 4.0MB/s
    Collecting tensor2tensor
      Downloading https://files.pythonhosted.org/packages/d6/7c/9e87d30cefad5cbc390bb7f626efb3ded9b19416b8160f1a1278da81b218/tensor2tensor-1.15.7-py2.py3-none-any.whl (1.4MB)
         |████████████████████████████████| 1.5MB 10.2MB/s
    Requirement already satisfied: gin-config in /usr/local/lib/python3.6/dist-packages (from trax) (0.3.0)
    Requirement already satisfied: jax in /usr/local/lib/python3.6/dist-packages (from trax) (0.2.0)
    Requirement already satisfied: tensorflow-datasets in /usr/local/lib/python3.6/dist-packages (from trax) (2.1.0)
    Collecting t5
      Downloading https://files.pythonhosted.org/packages/32/a4/1188812f1e439bba23f7ceb03879bcdb99744e2e12799aebb93bc0139496/t5-0.7.0-py3-none-any.whl (171kB)
         |████████████████████████████████| 174kB 16.1MB/s
    Requirement already satisfied: gym in /usr/local/lib/python3.6/dist-packages (from trax) (0.17.2)
    Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from trax) (1.18.5)
    Requirement already satisfied: scipy in /usr/local/lib/python3.6/dist-packages (from trax) (1.4.1)
    Collecting funcsigs
      Downloading https://files.pythonhosted.org/packages/69/cb/f5be453359271714c01b9bd06126eaf2e368f1fddfff30818754b5ac2328/funcsigs-1.0.2-py2.py3-none-any.whl
    Collecting tensorflow-text
      Downloading https://files.pythonhosted.org/packages/28/b2/2dbd90b93913afd07e6101b8b84327c401c394e60141c1e98590038060b3/tensorflow_text-2.3.0-cp36-cp36m-manylinux1_x86_64.whl (2.6MB)
         |████████████████████████████████| 2.6MB 20.3MB/s
    Requirement already satisfied: jaxlib in /usr/local/lib/python3.6/dist-packages (from trax) (0.1.55)
    Requirement already satisfied: absl-py in /usr/local/lib/python3.6/dist-packages (from trax) (0.10.0)
    Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from trax) (1.15.0)
    Requirement already satisfied: google-api-python-client in /usr/local/lib/python3.6/dist-packages (from tensor2tensor->trax) (1.7.12)
    Collecting tf-slim
      Downloading https://files.pythonhosted.org/packages/02/97/b0f4a64df018ca018cc035d44f2ef08f91e2e8aa67271f6f19633a015ff7/tf_slim-1.1.0-py2.py3-none-any.whl (352kB)
         |████████████████████████████████| 358kB 33.8MB/s
    Collecting gevent
      Downloading https://files.pythonhosted.org/packages/3f/92/b80b922f08f222faca53c8d278e2e612192bc74b0e1f0db2f80a6ee46982/gevent-20.9.0-cp36-cp36m-manylinux2010_x86_64.whl (5.3MB)
         |████████████████████████████████| 5.3MB 34.1MB/s
    Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from tensor2tensor->trax) (2.10.0)
    Requirement already satisfied: opencv-python in /usr/local/lib/python3.6/dist-packages (from tensor2tensor->trax) (4.1.2.30)
    Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from tensor2tensor->trax) (2.23.0)
    Collecting tensorflow-probability==0.7.0
      Downloading https://files.pythonhosted.org/packages/3e/3a/c10b6c22320531c774402ac7186d1b673374e2a9d12502cbc8d811e4601c/tensorflow_probability-0.7.0-py2.py3-none-any.whl (981kB)
         |████████████████████████████████| 983kB 28.4MB/s
    Requirement already satisfied: oauth2client in /usr/local/lib/python3.6/dist-packages (from tensor2tensor->trax) (4.1.3)
    Requirement already satisfied: flask in /usr/local/lib/python3.6/dist-packages (from tensor2tensor->trax) (1.1.2)
    Collecting mesh-tensorflow
      Downloading https://files.pythonhosted.org/packages/16/8b/553deb763ce8d00afb17debab7cb14a87b209cd4c6f0e8ecfc8d884cb12a/mesh_tensorflow-0.1.17-py3-none-any.whl (342kB)
         |████████████████████████████████| 348kB 29.9MB/s
    Requirement already satisfied: sympy in /usr/local/lib/python3.6/dist-packages (from tensor2tensor->trax) (1.1.1)
    Requirement already satisfied: dopamine-rl in /usr/local/lib/python3.6/dist-packages (from tensor2tensor->trax) (1.0.5)
    Requirement already satisfied: tensorflow-addons in /usr/local/lib/python3.6/dist-packages (from tensor2tensor->trax) (0.8.3)
    Collecting pypng
      Downloading https://files.pythonhosted.org/packages/bc/fb/f719f1ac965e2101aa6ea6f54ef8b40f8fbb033f6ad07c017663467f5147/pypng-0.0.20.tar.gz (649kB)
         |████████████████████████████████| 655kB 35.9MB/s
    Requirement already satisfied: Pillow in /usr/local/lib/python3.6/dist-packages (from tensor2tensor->trax) (7.0.0)
    Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from tensor2tensor->trax) (0.16.0)
    Collecting bz2file
      Downloading https://files.pythonhosted.org/packages/61/39/122222b5e85cd41c391b68a99ee296584b2a2d1d233e7ee32b4532384f2d/bz2file-0.98.tar.gz
    Collecting gunicorn
      Downloading https://files.pythonhosted.org/packages/69/ca/926f7cd3a2014b16870086b2d0fdc84a9e49473c68a8dff8b57f7c156f43/gunicorn-20.0.4-py2.py3-none-any.whl (77kB)
         |████████████████████████████████| 81kB 5.5MB/s
    Collecting kfac
      Downloading https://files.pythonhosted.org/packages/1c/36/06fe2c757044bb51906fef231ac48cc5bf9a277fc9a8c7e1108d7e9e8cfd/kfac-0.2.3-py2.py3-none-any.whl (191kB)
         |████████████████████████████████| 194kB 40.4MB/s
    Collecting tensorflow-gan
      Downloading https://files.pythonhosted.org/packages/0c/2e/62922111d7d50e1900e3030764743ea7735540ce103b3ab30fd5cd2d8a2b/tensorflow_gan-2.0.0-py2.py3-none-any.whl (365kB)
         |████████████████████████████████| 368kB 36.6MB/s
    Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from tensor2tensor->trax) (4.41.1)
    Requirement already satisfied: opt-einsum in /usr/local/lib/python3.6/dist-packages (from jax->trax) (3.3.0)
    Requirement already satisfied: protobuf>=3.6.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow-datasets->trax) (3.12.4)
    Requirement already satisfied: wrapt in /usr/local/lib/python3.6/dist-packages (from tensorflow-datasets->trax) (1.12.1)
    Requirement already satisfied: attrs>=18.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow-datasets->trax) (20.2.0)
    Requirement already satisfied: dill in /usr/local/lib/python3.6/dist-packages (from tensorflow-datasets->trax) (0.3.2)
    Requirement already satisfied: promise in /usr/local/lib/python3.6/dist-packages (from tensorflow-datasets->trax) (2.3)
    Requirement already satisfied: termcolor in /usr/local/lib/python3.6/dist-packages (from tensorflow-datasets->trax) (1.1.0)
    Requirement already satisfied: tensorflow-metadata in /usr/local/lib/python3.6/dist-packages (from tensorflow-datasets->trax) (0.24.0)
    Requirement already satisfied: torch in /usr/local/lib/python3.6/dist-packages (from t5->trax) (1.6.0+cu101)
    Requirement already satisfied: nltk in /usr/local/lib/python3.6/dist-packages (from t5->trax) (3.2.5)
    Requirement already satisfied: scikit-learn in /usr/local/lib/python3.6/dist-packages (from t5->trax) (0.22.2.post1)
    Requirement already satisfied: babel in /usr/local/lib/python3.6/dist-packages (from t5->trax) (2.8.0)
    Collecting sacrebleu
      Downloading https://files.pythonhosted.org/packages/a3/c4/8e948f601a4f9609e8b2b58f31966cb13cf17b940b82aa3e767f01c42c52/sacrebleu-1.4.14-py3-none-any.whl (64kB)
         |████████████████████████████████| 71kB 5.4MB/s
    Requirement already satisfied: pandas in /usr/local/lib/python3.6/dist-packages (from t5->trax) (1.1.2)
    Collecting transformers>=2.7.0
      Downloading https://files.pythonhosted.org/packages/19/22/aff234f4a841f8999e68a7a94bdd4b60b4cebcfeca5d67d61cd08c9179de/transformers-3.3.1-py3-none-any.whl (1.1MB)
         |████████████████████████████████| 1.1MB 30.7MB/s
    Collecting rouge-score
      Downloading https://files.pythonhosted.org/packages/1f/56/a81022436c08b9405a5247b71635394d44fe7e1dbedc4b28c740e09c2840/rouge_score-0.0.4-py2.py3-none-any.whl
    Collecting tfds-nightly
      Downloading https://files.pythonhosted.org/packages/8d/75/679a8312490388eb6cb70c8729020a3d7e721e0aefabf2d75a7cbefdb82a/tfds_nightly-4.0.1.dev202010160106-py3-none-any.whl (3.6MB)
         |████████████████████████████████| 3.6MB 29.7MB/s
    Collecting sentencepiece
      Downloading https://files.pythonhosted.org/packages/d4/a4/d0a884c4300004a78cca907a6ff9a5e9fe4f090f5d95ab341c53d28cbc58/sentencepiece-0.1.91-cp36-cp36m-manylinux1_x86_64.whl (1.1MB)
         |████████████████████████████████| 1.1MB 43.5MB/s
    Requirement already satisfied: cloudpickle<1.4.0,>=1.2.0 in /usr/local/lib/python3.6/dist-packages (from gym->trax) (1.3.0)
    Requirement already satisfied: pyglet<=1.5.0,>=1.4.0 in /usr/local/lib/python3.6/dist-packages (from gym->trax) (1.5.0)
    Requirement already satisfied: tensorflow<2.4,>=2.3.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow-text->trax) (2.3.0)
    Requirement already satisfied: google-auth>=1.4.1 in /usr/local/lib/python3.6/dist-packages (from google-api-python-client->tensor2tensor->trax) (1.17.2)
    Requirement already satisfied: google-auth-httplib2>=0.0.3 in /usr/local/lib/python3.6/dist-packages (from google-api-python-client->tensor2tensor->trax) (0.0.4)
    Requirement already satisfied: httplib2<1dev,>=0.17.0 in /usr/local/lib/python3.6/dist-packages (from google-api-python-client->tensor2tensor->trax) (0.17.4)
    Requirement already satisfied: uritemplate<4dev,>=3.0.0 in /usr/local/lib/python3.6/dist-packages (from google-api-python-client->tensor2tensor->trax) (3.0.1)
    Collecting zope.interface
      Downloading https://files.pythonhosted.org/packages/c5/89/1eb9dbb9e24f5e2c29ab1a88097b2f1333858aac3cd3cccc6c4c1c8ad867/zope.interface-5.1.2-cp36-cp36m-manylinux2010_x86_64.whl (236kB)
         |████████████████████████████████| 245kB 43.4MB/s
    Collecting greenlet>=0.4.17; platform_python_implementation == "CPython"
      Downloading https://files.pythonhosted.org/packages/80/d0/532e160c777b42f6f393f9de8c88abb8af6c892037c55e4d3a8a211324dd/greenlet-0.4.17-cp36-cp36m-manylinux1_x86_64.whl (44kB)
         |████████████████████████████████| 51kB 5.8MB/s
    Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from gevent->tensor2tensor->trax) (50.3.0)
    Collecting zope.event
      Downloading https://files.pythonhosted.org/packages/9e/85/b45408c64f3b888976f1d5b37eed8d746b8d5729a66a49ec846fda27d371/zope.event-4.5.0-py2.py3-none-any.whl
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->tensor2tensor->trax) (2020.6.20)
    Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->tensor2tensor->trax) (1.24.3)
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->tensor2tensor->trax) (3.0.4)
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->tensor2tensor->trax) (2.10)
    Requirement already satisfied: decorator in /usr/local/lib/python3.6/dist-packages (from tensorflow-probability==0.7.0->tensor2tensor->trax) (4.4.2)
    Requirement already satisfied: rsa>=3.1.4 in /usr/local/lib/python3.6/dist-packages (from oauth2client->tensor2tensor->trax) (4.6)
    Requirement already satisfied: pyasn1>=0.1.7 in /usr/local/lib/python3.6/dist-packages (from oauth2client->tensor2tensor->trax) (0.4.8)
    Requirement already satisfied: pyasn1-modules>=0.0.5 in /usr/local/lib/python3.6/dist-packages (from oauth2client->tensor2tensor->trax) (0.2.8)
    Requirement already satisfied: click>=5.1 in /usr/local/lib/python3.6/dist-packages (from flask->tensor2tensor->trax) (7.1.2)
    Requirement already satisfied: itsdangerous>=0.24 in /usr/local/lib/python3.6/dist-packages (from flask->tensor2tensor->trax) (1.1.0)
    Requirement already satisfied: Werkzeug>=0.15 in /usr/local/lib/python3.6/dist-packages (from flask->tensor2tensor->trax) (1.0.1)
    Requirement already satisfied: Jinja2>=2.10.1 in /usr/local/lib/python3.6/dist-packages (from flask->tensor2tensor->trax) (2.11.2)
    Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.6/dist-packages (from sympy->tensor2tensor->trax) (1.1.0)
    Requirement already satisfied: typeguard in /usr/local/lib/python3.6/dist-packages (from tensorflow-addons->tensor2tensor->trax) (2.7.1)
    Requirement already satisfied: tensorflow-hub>=0.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gan->tensor2tensor->trax) (0.9.0)
    Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow-metadata->tensorflow-datasets->trax) (1.52.0)
    Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.6/dist-packages (from scikit-learn->t5->trax) (0.16.0)
    Requirement already satisfied: pytz>=2015.7 in /usr/local/lib/python3.6/dist-packages (from babel->t5->trax) (2018.9)
    Collecting portalocker
      Downloading https://files.pythonhosted.org/packages/89/a6/3814b7107e0788040870e8825eebf214d72166adf656ba7d4bf14759a06a/portalocker-2.0.0-py2.py3-none-any.whl
    Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.6/dist-packages (from pandas->t5->trax) (2.8.1)
    Requirement already satisfied: dataclasses; python_version < "3.7" in /usr/local/lib/python3.6/dist-packages (from transformers>=2.7.0->t5->trax) (0.7)
    Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.6/dist-packages (from transformers>=2.7.0->t5->trax) (2019.12.20)
    Collecting sacremoses
      Downloading https://files.pythonhosted.org/packages/7d/34/09d19aff26edcc8eb2a01bed8e98f13a1537005d31e95233fd48216eed10/sacremoses-0.0.43.tar.gz (883kB)
         |████████████████████████████████| 890kB 44.0MB/s
    Requirement already satisfied: packaging in /usr/local/lib/python3.6/dist-packages (from transformers>=2.7.0->t5->trax) (20.4)
    Collecting tokenizers==0.8.1.rc2
      Downloading https://files.pythonhosted.org/packages/80/83/8b9fccb9e48eeb575ee19179e2bdde0ee9a1904f97de5f02d19016b8804f/tokenizers-0.8.1rc2-cp36-cp36m-manylinux1_x86_64.whl (3.0MB)
         |████████████████████████████████| 3.0MB 42.2MB/s
    Requirement already satisfied: filelock in /usr/local/lib/python3.6/dist-packages (from transformers>=2.7.0->t5->trax) (3.0.12)
    Collecting importlib-resources; python_version < "3.9"
      Downloading https://files.pythonhosted.org/packages/ba/03/0f9595c0c2ef12590877f3c47e5f579759ce5caf817f8256d5dcbd8a1177/importlib_resources-3.0.0-py2.py3-none-any.whl
    Requirement already satisfied: typing-extensions; python_version < "3.8" in /usr/local/lib/python3.6/dist-packages (from tfds-nightly->t5->trax) (3.7.4.3)
    Requirement already satisfied: tensorflow-estimator<2.4.0,>=2.3.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow<2.4,>=2.3.0->tensorflow-text->trax) (2.3.0)
    Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow<2.4,>=2.3.0->tensorflow-text->trax) (1.32.0)
    Requirement already satisfied: keras-preprocessing<1.2,>=1.1.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow<2.4,>=2.3.0->tensorflow-text->trax) (1.1.2)
    Requirement already satisfied: google-pasta>=0.1.8 in /usr/local/lib/python3.6/dist-packages (from tensorflow<2.4,>=2.3.0->tensorflow-text->trax) (0.2.0)
    Requirement already satisfied: gast==0.3.3 in /usr/local/lib/python3.6/dist-packages (from tensorflow<2.4,>=2.3.0->tensorflow-text->trax) (0.3.3)
    Requirement already satisfied: astunparse==1.6.3 in /usr/local/lib/python3.6/dist-packages (from tensorflow<2.4,>=2.3.0->tensorflow-text->trax) (1.6.3)
    Requirement already satisfied: tensorboard<3,>=2.3.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow<2.4,>=2.3.0->tensorflow-text->trax) (2.3.0)
    Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.6/dist-packages (from tensorflow<2.4,>=2.3.0->tensorflow-text->trax) (0.35.1)
    Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.6/dist-packages (from google-auth>=1.4.1->google-api-python-client->tensor2tensor->trax) (4.1.1)
    Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.6/dist-packages (from Jinja2>=2.10.1->flask->tensor2tensor->trax) (1.1.1)
    Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.6/dist-packages (from packaging->transformers>=2.7.0->t5->trax) (2.4.7)
    Requirement already satisfied: zipp>=0.4; python_version < "3.8" in /usr/local/lib/python3.6/dist-packages (from importlib-resources; python_version < "3.9"->tfds-nightly->t5->trax) (3.2.0)
    Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.6/dist-packages (from tensorboard<3,>=2.3.0->tensorflow<2.4,>=2.3.0->tensorflow-text->trax) (0.4.1)
    Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.6/dist-packages (from tensorboard<3,>=2.3.0->tensorflow<2.4,>=2.3.0->tensorflow-text->trax) (3.2.2)
    Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard<3,>=2.3.0->tensorflow<2.4,>=2.3.0->tensorflow-text->trax) (1.7.0)
    Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<3,>=2.3.0->tensorflow<2.4,>=2.3.0->tensorflow-text->trax) (1.3.0)
    Requirement already satisfied: importlib-metadata; python_version < "3.8" in /usr/local/lib/python3.6/dist-packages (from markdown>=2.6.8->tensorboard<3,>=2.3.0->tensorflow<2.4,>=2.3.0->tensorflow-text->trax) (2.0.0)
    Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.6/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<3,>=2.3.0->tensorflow<2.4,>=2.3.0->tensorflow-text->trax) (3.1.0)
    Building wheels for collected packages: pypng, bz2file, sacremoses
      Building wheel for pypng (setup.py) ... done
      Created wheel for pypng: filename=pypng-0.0.20-cp36-none-any.whl size=67162 sha256=22e494987352f42fbe63f2ec144ba01401cceb3d3515b2acb1aa5db9453f89a9
      Stored in directory: /root/.cache/pip/wheels/41/6b/ef/0493b536b6d4722c2ae9486691b1d49b922b9877922beeabb3
      Building wheel for bz2file (setup.py) ... done
      Created wheel for bz2file: filename=bz2file-0.98-cp36-none-any.whl size=6884 sha256=cd834ddadde354dd1db3b3e18d98e419d95b79ca7f7ecde48bc676e7576df855
      Stored in directory: /root/.cache/pip/wheels/81/75/d6/e1317bf09bf1af5a30befc2a007869fa6e1f516b8f7c591cb9
      Building wheel for sacremoses (setup.py) ... done
      Created wheel for sacremoses: filename=sacremoses-0.0.43-cp36-none-any.whl size=893257 sha256=40d4b265a7407cb660aba24ec10b2b5cefac8afe76b8f68781d3220ca44823c7
      Stored in directory: /root/.cache/pip/wheels/29/3c/fd/7ce5c3f0666dab31a50123635e6fb5e19ceb42ce38d4e58f45
    Successfully built pypng bz2file sacremoses
    ERROR: kfac 0.2.3 has requirement tensorflow-probability==0.8, but you'll have tensorflow-probability 0.7.0 which is incompatible.
    Installing collected packages: tf-slim, zope.interface, greenlet, zope.event, gevent, tensorflow-probability, mesh-tensorflow, pypng, bz2file, gunicorn, kfac, tensorflow-gan, tensor2tensor, portalocker, sacrebleu, sacremoses, tokenizers, sentencepiece, transformers, rouge-score, importlib-resources, tfds-nightly, tensorflow-text, t5, funcsigs, trax
      Found existing installation: tensorflow-probability 0.11.0
        Uninstalling tensorflow-probability-0.11.0:
          Successfully uninstalled tensorflow-probability-0.11.0
    Successfully installed bz2file-0.98 funcsigs-1.0.2 gevent-20.9.0 greenlet-0.4.17 gunicorn-20.0.4 importlib-resources-3.0.0 kfac-0.2.3 mesh-tensorflow-0.1.17 portalocker-2.0.0 pypng-0.0.20 rouge-score-0.0.4 sacrebleu-1.4.14 sacremoses-0.0.43 sentencepiece-0.1.91 t5-0.7.0 tensor2tensor-1.15.7 tensorflow-gan-2.0.0 tensorflow-probability-0.7.0 tensorflow-text-2.3.0 tf-slim-1.1.0 tfds-nightly-4.0.1.dev202010160106 tokenizers-0.8.1rc2 transformers-3.3.1 trax-1.3.5 zope.event-4.5.0 zope.interface-5.1.2

```
import sys
import os

import numpy as np

import textwrap
wrapper = textwrap.TextWrapper(width=70)

import trax
from trax import layers as tl
from trax.fastmath import numpy as jnp

np.set_printoptions(threshold=sys.maxsize)
```

```
train_stream_fn = trax.data.TFDS('cnn_dailymail',
                                 keys=('article', 'highlights'),
                                 train=True)

eval_stream_fn = trax.data.TFDS('cnn_dailymail',
                                keys=('article', 'highlights'),
                                train=False)
```

```
def tokenize(input_str, EOS=1):
    inputs =  next(trax.data.tokenize(iter([input_str]),
                                      # vocab_dir='vocab_dir/',
                                      vocab_file='en_8k.subword'))

    return list(inputs) + [EOS]

def detokenize(integers):
    s = trax.data.detokenize(integers,
                            #  vocab_dir='vocab_dir/',
                             vocab_file='en_8k.subword')

    return wrapper.fill(s)
```

```
# Special tokens
SEP = 0 # Padding or separator token
EOS = 1 # End of sentence token

# Concatenate tokenized inputs and targets using 0 as separator.
def preprocess(stream):
    for (article, summary) in stream:
        joint = np.array(list(article) + [EOS, SEP] + list(summary) + [EOS])
        mask = [0] * (len(list(article)) + 2) + [1] * (len(list(summary)) + 1) # Accounting for EOS and SEP
        yield joint, joint, np.array(mask)

# You can combine a few data preprocessing steps into a pipeline like this.
input_pipeline = trax.data.Serial(
    # Tokenizes
    trax.data.Tokenize(#vocab_dir='vocab_dir/',
                       vocab_file='en_8k.subword'),
    # Uses function defined above
    preprocess,
    # Filters out examples longer than 2048
    trax.data.FilterByLength(2048)
)

# Apply preprocessing to data streams.
train_stream = input_pipeline(train_stream_fn())
eval_stream = input_pipeline(eval_stream_fn())

train_input, train_target, train_mask = next(train_stream)

assert sum((train_input - train_target)**2) == 0  # They are the same in Language Model (LM).

```

```
print(f'Single example mask:\n\n {train_mask}')
```

    Single example mask:

     [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1
     1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
     1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
     1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
     1 1 1 1 1 1 1 1 1]

```
print(f'Single example:\n\n {detokenize(train_input)}')

```

    Single example:

     A new study is backing up long held suspicions that Apple slows down
    older models of its iPhones to encourage users to buy a new release.
    The U.S. study analysed worldwide searches for 'iPhone slow' and found
    that the search term spiked significantly around the time of new
    iPhone launch. It then compared those results with similar searches
    for the term 'Samsung Galaxy slow', and discovered the term was
    unaffected by new releases from Samsung. A new study is backing up
    long held suspicions that Apple slows down older models of iPhones to
    encourage users to buy its new release. The U.S. study analysed
    worldwide searches for 'iPhone slow' and found that the search term
    spiked significantly around the time of new phone releases . The
    study, compiled by Harvard University PhD student Laura Trucco,
    follows claims that the Cupertino-based company is deliberately
    sabotaging its old products. Writing for the New York Times, Sendhil
    Mullainathan, a professor of economics at Harvard, described the
    results as 'striking'. 'Wouldn't many business owners love to make
    their old product less useful whenever they released a newer one?' Mr
    Mullainathan wrote. ‘When you sell the device and control the
    operating system, that's an option'. The study then compared those
    results with similar searches for the term 'Samsung Galaxy slow', and
    found the term was unaffected by new releases from Samsung . While
    some MailOnline readers haven't noticed a slow down, others claim that
    Apple is sabotaging older phones through software updates. 'This is
    common knowledge,' one reader wrote. 'If you want to keep your iPhone
    running at the same pace do not do the software upgrade that comes out
    within six months of a new iPhone release,' Last year, Catherine
    Rampell, also writing in the New York Times, raised concerns that
    Apple could be engineering the new operating system so it only works
    properly with the newest version of the product. She said her iPhone 4
    became a lot slower when she downloaded iOS 7 - and that the only
    solution seemed to be to buy the iPhone 5. Ms Rampell accused Apple of
    having run out of ideas so was trying to ‘brainwash’ its customers
    into buying the new iPhone 5S and 5C because they look nice. Her
    claims fuelled conspiracy theorists who have long held that Apple
    engages in ‘planned obsolescence’, a term which has been around since
    the Great Depression in the 1930s. The theory states that
    manufacturers of everything from cars to microwaves build in a certain
    lifetime to a product and then it will simply stop working, forcing
    consumers to buy a new one. And Apple has faced allegations that it is
    guilty of planned obsolescence before. Last year, Catherine Rampell,
    also writing in the New York Times, raised concerns that Apple could
    be engineering the new operating system so it only works properly with
    the newest version of the product . A security expert has warned
    Apple’s iOS software contains potentially sinister tools that could be
    used by governments to spy on iPhone and iPad users. Speaking at the
    'Hackers on planet Earth' conference in New York, Jonathan Zdziarski
    said that most users are unaware of the lack of protection for iPhone
    data. He added files found hidden within the firm's software contain a
    file-relay service that can be used to access the user's address book,
    photos, voicemail and any accounts configured on the device. However,
    Apple has denied the claims the backdoor was created deliberately for
    government or surveillance purposes. His investigation followed
    earlier reports of the NSA spying on Apple products, which suggested a
    ‘backdoor’ in iOS could provide hackers with valuable information. A
    backdoor is a hidden remote access port that can allow outside sources
    to access a device with little detection. The conclusion was based on
    an analysis of 600 million iOS devices, with handsets running the most
    recent versions of the software at particular risk. When it started
    using more tamper-resistant screws experts said it was to stop users
    getting into the phone and fixing it themselves if there was a
    problem. Meanwhile, in 2012 Apple was sued in Brazil by the Brazilian
    Institute of Politics and Law Software over the launch of the iPad
    Air. The organisation claimed that because it had the new retina
    screen it made the iPad 3 redundant and that Apple was changing its
    devices too quickly. Ms Rampell said: ‘When major innovations remain
    out of reach, and degrading durability threatens to tick off loyal
    customers, companies like Apple can still take a cue from the fashion
    industry. ‘If you can brainwash consumers into developing new tastes
    that make the old stuff look uncool for aesthetic rather than
    functional reasons, you still have a shot at harvesting more sales
    from your existing customer base. Dom Ferkin, managing direction of
    UK-based iOS experts, Creation Application, told MailOnline that he
    doesn’t believe Apple are doing this intentionally. ‘On every hardware
    release they tend to upgrade the chips and they are faster every time
    they are released,’ he said. ‘Each year they release a new iOS. If
    you’re running an iOS 7 on a 5 chip, for example, it’s comparable to
    running Windows XP on a Windows 95 machine. ‘It’s just enough to annoy
    the users, but it’s needed if you want the slew of new features that
    Apple releases each year.’ Mr Mullainathan added that the research
    does not prove that Apple has done anything wrong. No matter how
    suggestive, he says, the data alone doesn't allow anyone to determine
    conclusively whether their phone is any slower. There are other
    explanations for why an older model iPhone may slow down, he claims.
    For instance, the latest version of the Apple operating system, iOS,
    is always tailored to the newest device and may therefore not work as
    efficiently on older models. ‘Hearing about a new release makes you
    contemplate getting a new and faster phone,’ he added. ‘And you
    suddenly notice how slow your old phone is.’ Apple is yet to respond.
    But Mr Mullainathan added that the research does not prove that Apple
    has done anything wrong.No matter how suggestive, he says, the data
    alone doesn't allow anyone to determine conclusively whether their
    phone is any slower .<EOS><pad>Thestudy was undertaken by student
    Laura Trucco at Harvard University . It also compared Apple's results
    with searches for 'Samsung Galaxy slow' Research found that the term
    was unaffected by Samsung new releases . Study has fuelled suggestions
    Apple engages in ‘planned obsolescence’ Theory states that
    manufacturers build in a certain lifetime to a product and then it
    will simply stop working, forcing consumers to buy a new one .<EOS>

```

# Bucketing to create batched generators.

# Buckets are defined in terms of boundaries and batch sizes.
# Batch_sizes[i] determines the batch size for items with length < boundaries[i]
# So below, we'll take a batch of 16 sentences of length < 128 , 8 of length < 256,
# 4 of length < 512. And so on.
boundaries =  [128, 256,  512, 1024]
batch_sizes = [16,    8,    4,    2, 1]

# Create the streams.
train_batch_stream = trax.data.BucketByLength(
    boundaries, batch_sizes)(train_stream)

eval_batch_stream = trax.data.BucketByLength(
    boundaries, batch_sizes)(eval_stream)
```

```

# Every execution will result in generation of a different article
# Try running this cell multiple times to see how the length of the examples affects the batch size
input_batch, _, mask_batch = next(train_batch_stream)

# Shape of the input_batch
input_batch.shape
```

    (1, 1111)

```
print(input_batch[0])
```

    [1411 4030   84 4173  900   47  492 5182   57 4030   84  205  262  212
      289 3849  297  137  274 3536    8  463  113 5396 3319  385    3  479
      636  388 4030 4352 4030   84  262  205 6355  284  137  274 3536    8
      619  113 6185 3319  385    3  479  636  388 4030   84   87 4873  513
     1187  140 4127   23   13 3528 3968 4651   91 5966    6 1546   22   39
        4 4297    2    5 3304   20 5689  104 1225  118    5    4 2035 3000
      138 2840 5124    2 3202  409   10 2589    8 1237   29 3389  820    3
      824    3   51 6596 4579 2438    3  835 2785   39    4 6044  232  861
       31 1476  619  255   12  282 5273 2152 6309  229    8 3111   36   20
     5044   20   35   71  172 2440   32  245 4647 1854   36    6 1342   30
       12 4689    3   43 5273   37  310  245 4127    2   13 1176    4 5273
      626  484  828 4040   35 5966    6    8 1237   29 3389  820    3  824
     3779 7021   84  103  245   82 6223   47   58   12 2840 5124    2 3202
      409  255   12  282 5273 2152 6309  229 3614   20  619 4030   84 3935
        6 3389  820 5720  245 6236    2   80  595 3329   23   34  245 4880
        5 4732  121  861  349 5491    6    4  282 4030   84 3935    6 3389
      820    3 2258  592  481 5242    2    7  656 1241 5886 3663 2048 2547
       47 1607 1839    2 5952 6342 2299   29    3 1034    3   25  309 2408
     5437   20   34  245 2499   78    3 5720    4   82 6044 1342  861 2519
       31  245 4647 1854   36    6  232  861   31  235    8 3111 1125 3536
       36 5898    2 4438   13  495  115  375   51 1685    3  121  139   31
     5854   23   13   71 1254  139 2595   13  897  245   10    4  872 2796
        2    5 2368   47 1202   36  139   31 5632 5293   20   13  643  387
      245   26    4 4030   84 2626 1986    7 6325  680 4476 3268   22  121
     5801   67  861 4992   42 1887   47 3286   32  245 2600 1202   36  139
     5012   23 4030   84 2324   29  266   80  629  911  333 4078   91  292
        5  245 4806    2    7    4 3379   67   31   37 4030   84  861 1887
       20 1476 2639  292    5    4 5518 2699    8  139   31 1476   12 4689
     1254  121    4  520 4030   84  104  307 3578   39  562  961  244 1343
      246 3935    6 3389  820 5720  861 4972    6   37    4 5974 4763    2
     1197 4030   84 4078   91   51  245 4647 1854 1887    6 4806    2  449
       96  133 3284   23   34 4630 2707   32 4030   84    4 1342   10   12
     2752  379    8 3111 1125 3536   36 3111  449   96 3286   23 4030   84
     4188    8  179 1887    6 1081   13 6107   71  257 1254  523   37 1887
        6    4  164  173 4030   84 2090 1887  818  897 1824  118    8  139
     4127    2  287   13 1005 3154   39 2730 1742   32 4030   84 2061 4275
        6  961  244 3935    6 3389  820  104  133 2785   35 2840 5124    2
     3202  409 1084   57 2232   22    7 3278 5374   10 6596 4579 2438   35
      238   12  130 4030   84 3935    6 3389  820    7  245  592  387    3
     2840 5124    2 1336  559 1047 2010 3986 2613   64  882  478 2699    3
       10 4896    5  118    5  124 4687 1611 4030   84 3935    6 3389  820
       36    6  592  481 5242    2    7  656 1241 5886 3663 2048 2547   47
     1607 1839    2 5952 6342 2299   29    3 1034    3   25 3600   22  141
       13  245  309 5456   49 2499   78 4030  916 1158    4  322    5    4
      288    3 1156 1887  211 3906   32   39 1824  118 1887    6 3304   20
      118    8  179 1887    6   90 1476   12 1342  961  244   87  130 2663
        2    3 3935    6 3389  820  835 2785 1642   14  173   58 1084   57
     2232   22    7 3278 5374 2840 5124    2 3202  409   10 6596 4579 2438
        8 1210    5    4  759   36    6 2035 3000  138 1000 4824  167    3
      861 1125  245 4732 4880  595 3329   23  194 1202   36 1574    2 6236
        2   80  595 3329   23    7 5012   23  151 6420    6    5 6499   67
     4013    6 6255   84  861 1125 1202   36  755 5012   23  562  442    4
     6513   44 6366    2  280 4030   84  232  162   80  339 1690  757   23
      961 7021   84  139 3254   20  335   37   31 1476    4 4030   84 2922
     4078   91  292    5    4 4724    6  961  244 3951  513 1082   10    4
      933 2931   42   61 3935    6 3389  820   25 5351 1143   35   12 5324
       10 2840 5124    2 3209   30  258   30 2785 1642   14  173 4030   84
     1166  861   25   90 4252   32 2840 5124    6    3 3935    6 3389  820
     3393    6 5822   22 4054    2    7 1054  193   13  245 1854 3187  852
     6244    3 4678 4030   84  179 2644 1240   23  245   13 1782  280   12
     1642   14  173 5504   58 1084   57 2232   22    7 3278 5374 2840 5124
        2 3202  409   10 6596 4579 2438   12  130 2663    2 1254 6277  118
        5    4 2035 3000  138 1000 4824  167    8 1343  246  861   25 4253
       32   12 5141 4902    2   58    4 1336  559 1047    3 3403 3275 3461
       12 2840 5124    2 3209 3445  123 5324    7   43 4630 2707   32 4902
        2   58    4  269    5 3707   97    8 3313   20   10  856 4020    2
        3 1908 5621    2    7  791 1082 1919  193    3 3935    6 3389  820
     4252   23  245   82 2840 5124    2   10  789  164  130    7  104 5248
     1066 4423    2  528  350    8   28 6596 4579 2438 4873  513 1187   10
     1642 2840 5124    2  958   78    3  361    3    7   39 2929  406  363
       26 2195 3393   29   35   12 6335    2  292    3  257 4030   84 1105
     3044   32 4687 1611   25  151   10   12  288   36    6  282   35  824
       14  130   14 1249 1237   29    3  140  538 3669   13 4072  287   36
        6 3304   20 5689   12  465 4590 3003 1159   20 1016 4030    2    1
        0 1237   29 3389  820    3  824    3   51 6596 4579 2438    3  104
     4871   10   12 2840 5124    2 1336  559 1047   35   12  130 4030  210
     1105 3044   32 4687 1611    3  513 1206   91  933 2931  220    7 4252
       32 3209  372    5  245 2499   78 4030  210 3111 5720 3536   36  179
       36    6    4  164  173 2090  897  335 1254   71   36    6 1081   13
     6107   71  257   36    1]

```
print('Article:\n\n', detokenize(input_batch[0]))

```

    Article:

     By . Harriet Arkell . PUBLISHED: . 11:56 EST, 15 May 2013 . | .
    UPDATED: . 14:54 EST, 15 May 2013 . A teenager who wanted to help
    grieving relatives cope with the loss of loved ones has become one of
    the youngest funeral directors in Britain. Amy Darlow, 19, from
    Salisbury, began working with the dead when she was just 14 during a
    work experience placement. She'd applied for it after seeing her
    grandmother's body as a child, an experience that made her want to
    make the experience less traumatic for relatives. Amy Darlow, 19,  had
    her first stint at a funeral directors during a work experience
    placement aged 14 . Miss Darlow says her friends were surprised by her
    choice of career but she enjoys the work . Miss Darlow, whose
    boyfriend and childhood sweetheart Declan Spreadbury, 22, is
    untroubled by her job, says the first dead body she saw was her
    grandmother's when she was 10. She said: 'People try to hide death
    from children, but I was exposed to it - I went to see her in the
    Chapel of Rest. 'I was encouraged to kiss her on the . forehead and
    say goodbye but sadly she wasn’t looking her best. 'I asked . why
    there were bubbles coming out of her mouth and the reply was that .
    she’d just come out of the fridge. I was just a child - but the image
    . has stayed with me.' Now Miss Darlow says she knows that the gastric
    air . coming from her grandmother’s mouth could have been prevented by
    embalming . the body in a certain way. She said: 'She could have
    looked . better. It’s important to get it right - because that’s the
    last time . you’ll see someone. I want people to go away with good
    lasting . memories.' Miss Darlow has been working for funeral
    directors Will Case and Partners in Salisbury for over a year . Miss
    Darlow and her boss, funeral parlour manager Christopher Orledge, in
    front of one of their hearses . Miss Darlow's boyfriend and childhood
    sweetheart Declan Spreadbury, 22, is quite used to her unusual job .
    'At the end of the day, we’re dealing with someone’s loved one. It’s
    not just a body.' A year ago, Miss Darlow began working full-time at
    Will Case and Partners funeral directors in Salisbury. One of the
    country's youngest undertakers, she said her career choice surprised
    some. 'My friends were surprised and asked all sorts of silly
    questions,' she said. 'They asked me if the bodies sat up . when they
    were being cremated.  I told them that was just the . gas coming out
    of the organs.' Another nail in the coffin: Miss Darlow is studying
    for a degree in funeral services as well as working full-time . When
    she is not conducting funerals, Miss Darlow reads crime fiction and
    chats to her mother Elizabeth, 53 . It prompted her to take up a full-
    time post at Will Case and Partners funeral directors in Salisbury a
    year ago - becoming one of the youngest undertakers. Now she is
    completing a training course at the parlour, alongside a funeral
    services foundation degree and an embalming course at the University
    of Bath. Clad in top hat, cravat and tail coats, Miss Darlow conducted
    her first funeral in June last year and has completed another nine
    since then. The Salisbury teenager in full funeral garb, left, and
    with tattoos on show ready for a night out, right . Polishing hearses
    is all in a day's work for 19-year-old Amy, who likes to give people's
    loved ones a dignified send off .<EOS><pad>AmyDarlow, 19, from
    Salisbury, has worked in a funeral parlour for a year . Polishing
    hearses, nailing coffins and conducting services part of her job . She
    says: 'It's the last time you see them - it's important to get it
    right'<EOS>

```
def create_tensor(t):
    """Create tensor from list of lists"""
    return jnp.array(t)

def display_tensor(t, name):
    """Display shape and tensor"""
    print(f'{name} shape: {t.shape}\n')
    print(f'{t}\n')
```

```
q = create_tensor([[1, 0, 0], [0, 1, 0]])
display_tensor(q, 'query')
k = create_tensor([[1, 2, 3], [4, 5, 6]])
display_tensor(k, 'key')
v = create_tensor([[0, 1, 0], [1, 0, 1]])
display_tensor(v, 'value')
m = create_tensor([[0, 0], [-1e9, 0]])
display_tensor(m, 'mask')
```

    query shape: (2, 3)

    [[1 0 0]
     [0 1 0]]

    key shape: (2, 3)

    [[1 2 3]
     [4 5 6]]

    value shape: (2, 3)

    [[0 1 0]
     [1 0 1]]

    mask shape: (2, 2)

    [[ 0.e+00  0.e+00]
     [-1.e+09  0.e+00]]

    /usr/local/lib/python3.6/dist-packages/jax/lib/xla_bridge.py:130: UserWarning: No GPU/TPU found, falling back to CPU.
      warnings.warn('No GPU/TPU found, falling back to CPU.')

```

q_dot_k = q @ k.T / jnp.sqrt(3)
display_tensor(q_dot_k, 'query dot key')
```

    query dot key shape: (2, 2)

    [[0.57735026 2.309401  ]
     [1.1547005  2.8867514 ]]

```
masked = q_dot_k + m
display_tensor(masked, 'masked query dot key')
```

    masked query dot key shape: (2, 2)

    [[ 5.7735026e-01  2.3094010e+00]
     [-1.0000000e+09  2.8867514e+00]]

```
display_tensor(masked @ v, 'masked query dot key dot value')

```

    masked query dot key dot value shape: (2, 3)

    [[ 2.3094010e+00  5.7735026e-01  2.3094010e+00]
     [ 2.8867514e+00 -1.0000000e+09  2.8867514e+00]]

```
q_with_batch = q[None,:]
display_tensor(q_with_batch, 'query with batch dim')
k_with_batch = k[None,:]
display_tensor(k_with_batch, 'key with batch dim')
v_with_batch = v[None,:]
display_tensor(v_with_batch, 'value with batch dim')
m_bool = create_tensor([[True, True], [False, True]])
display_tensor(m_bool, 'boolean mask')
```

    query with batch dim shape: (1, 2, 3)

    [[[1 0 0]
      [0 1 0]]]

    key with batch dim shape: (1, 2, 3)

    [[[1 2 3]
      [4 5 6]]]

    value with batch dim shape: (1, 2, 3)

    [[[0 1 0]
      [1 0 1]]]

    boolean mask shape: (2, 2)

    [[ True  True]
     [False  True]]

```
def DotProductAttention(query, key, value, mask):
    """Dot product self-attention.
    Args:
        query (jax.interpreters.xla.DeviceArray): array of query representations with shape (L_q by d)
        key (jax.interpreters.xla.DeviceArray): array of key representations with shape (L_k by d)
        value (jax.interpreters.xla.DeviceArray): array of value representations with shape (L_k by d) where L_v = L_k
        mask (jax.interpreters.xla.DeviceArray): attention-mask, gates attention with shape (L_q by L_k)

    Returns:
        jax.interpreters.xla.DeviceArray: Self-attention array for q, k, v arrays. (L_q by L_k)
    """

    assert query.shape[-1] == key.shape[-1] == value.shape[-1], "Embedding dimensions of q, k, v aren't all the same"

    ### START CODE HERE (REPLACE INSTANCES OF 'None' with your code) ###
    # Save depth/dimension of the query embedding for scaling down the dot product
    depth = query.shape[-1]

    # Calculate scaled query key dot product according to formula above
    dots = jnp.matmul(query, jnp.swapaxes(key, -1, -2)) / jnp.sqrt(depth)

    # Apply the mask
    if mask is not None: # The 'None' in this line does not need to be replaced
        dots = jnp.where(mask, dots, jnp.full_like(dots, -1e9))

    # Softmax formula implementation
    # Use trax.fastmath.logsumexp of dots to avoid underflow by division by large numbers
    # Hint: Last axis should be used and keepdims should be True
    # Note: softmax = e^(dots - logsumexp(dots)) = E^dots / sumexp(dots)
    logsumexp = trax.fastmath.logsumexp(dots, axis=-1, keepdims=True)

    # Take exponential of dots minus logsumexp to get softmax
    # Use jnp.exp()
    dots = jnp.exp(dots - logsumexp)

    # Multiply dots by value to get self-attention
    # Use jnp.matmul()
    attention = jnp.matmul(dots, value)

    ## END CODE HERE ###

    return attention
```

```
DotProductAttention(q_with_batch, k_with_batch, v_with_batch, m_bool)
```

    DeviceArray([[[0.8496746 , 0.15032545, 0.8496746 ],
                  [1.        , 0.        , 1.        ]]], dtype=float32)

```
tensor2d = create_tensor(q)
display_tensor(tensor2d, 'query matrix (2D tensor)')

tensor4d2b = create_tensor([[q, q], [q, q]])
display_tensor(tensor4d2b, 'batch of two (multi-head) collections of query matrices (4D tensor)')

tensor3dc = create_tensor([jnp.concatenate([q, q], axis = -1)])
display_tensor(tensor3dc, 'one batch of concatenated heads of query matrices (3d tensor)')

tensor3dc3b = create_tensor([jnp.concatenate([q, q], axis = -1), jnp.concatenate([q, q], axis = -1), jnp.concatenate([q, q], axis = -1)])
display_tensor(tensor3dc3b, 'three batches of concatenated heads of query matrices (3d tensor)')
```

    query matrix (2D tensor) shape: (2, 3)

    [[1 0 0]
     [0 1 0]]

    batch of two (multi-head) collections of query matrices (4D tensor) shape: (2, 2, 2, 3)

    [[[[1 0 0]
       [0 1 0]]

      [[1 0 0]
       [0 1 0]]]

     [[[1 0 0]
       [0 1 0]]

      [[1 0 0]
       [0 1 0]]]]

    one batch of concatenated heads of query matrices (3d tensor) shape: (1, 2, 6)

    [[[1 0 0 1 0 0]
      [0 1 0 0 1 0]]]

    three batches of concatenated heads of query matrices (3d tensor) shape: (3, 2, 6)

    [[[1 0 0 1 0 0]
      [0 1 0 0 1 0]]

     [[1 0 0 1 0 0]
      [0 1 0 0 1 0]]

     [[1 0 0 1 0 0]
      [0 1 0 0 1 0]]]

```

def compute_attention_heads_closure(n_heads, d_head):
    """ Function that simulates environment inside CausalAttention function.
    Args:
        d_head (int):  dimensionality of heads.
        n_heads (int): number of attention heads.
    Returns:
        function: compute_attention_heads function
    """

    def compute_attention_heads(x):
        """ Compute the attention heads.
        Args:
            x (jax.interpreters.xla.DeviceArray): tensor with shape (batch_size, seqlen, n_heads X d_head).
        Returns:
            jax.interpreters.xla.DeviceArray: reshaped tensor with shape (batch_size X n_heads, seqlen, d_head).
        """
        ### START CODE HERE (REPLACE INSTANCES OF 'None' with your code) ###

        # Size of the x's batch dimension
        batch_size = x.shape[0]
        # Length of the sequence
        # Should be size of x's first dimension without counting the batch dim
        seqlen = x.shape[1]

        # Reshape x using jnp.reshape()
        # batch_size, seqlen, n_heads*d_head -> batch_size, seqlen, n_heads, d_head
        x = jnp.reshape(x, (batch_size, seqlen, n_heads, d_head))

        # Transpose x using jnp.transpose()
        # batch_size, seqlen, n_heads, d_head -> batch_size, n_heads, seqlen, d_head
        # Note that the values within the tuple are the indexes of the dimensions of x and you must rearrange them
        x = jnp.transpose(x, (0, 2, 1, 3))

        # Reshape x using jnp.reshape()
        # batch_size, n_heads, seqlen, d_head -> batch_size*n_heads, seqlen, d_head
        x = jnp.reshape(x, (-1, seqlen, d_head))
        ### END CODE HERE ###

        return x

    return compute_attention_heads
```

```

display_tensor(tensor3dc3b, "input tensor")
result_cah = compute_attention_heads_closure(2,3)(tensor3dc3b)
display_tensor(result_cah, "output tensor")
```

    input tensor shape: (3, 2, 6)

    [[[1 0 0 1 0 0]
      [0 1 0 0 1 0]]

     [[1 0 0 1 0 0]
      [0 1 0 0 1 0]]

     [[1 0 0 1 0 0]
      [0 1 0 0 1 0]]]

    output tensor shape: (6, 2, 3)

    [[[1 0 0]
      [0 1 0]]

     [[1 0 0]
      [0 1 0]]

     [[1 0 0]
      [0 1 0]]

     [[1 0 0]
      [0 1 0]]

     [[1 0 0]
      [0 1 0]]

     [[1 0 0]
      [0 1 0]]]

```
def dot_product_self_attention(q, k, v):
    """ Masked dot product self attention.
    Args:
        q (jax.interpreters.xla.DeviceArray): queries.
        k (jax.interpreters.xla.DeviceArray): keys.
        v (jax.interpreters.xla.DeviceArray): values.
    Returns:
        jax.interpreters.xla.DeviceArray: masked dot product self attention tensor.
    """
    ### START CODE HERE (REPLACE INSTANCES OF 'None' with your code) ###

    # Hint: mask size should be equal to L_q. Remember that q has shape (batch_size, L_q, d)
    # NOTE: there is a revision underway with the autograder to tolerate better indexing.
    # Until then, please index q.shape using negative values (this is equivalent to counting from right to left)
    mask_size = q.shape[-2]

    # Creates a matrix with ones below the diagonal and 0s above. It should have shape (1, mask_size, mask_size)
    # Notice that 1's and 0's get casted to True/False by setting dtype to jnp.bool_
    # Use jnp.tril() - Lower triangle of an array and jnp.ones()
    mask = jnp.tril(jnp.ones((1, mask_size, mask_size), dtype=jnp.bool_), k=0)

    ### END CODE HERE ###

    return DotProductAttention(q, k, v, mask)
```

```
dot_product_self_attention(q_with_batch, k_with_batch, v_with_batch)

```

    DeviceArray([[[0.        , 1.        , 0.        ],
                  [0.8496746 , 0.15032543, 0.8496746 ]]], dtype=float32)

```
def compute_attention_output_closure(n_heads, d_head):
    """ Function that simulates environment inside CausalAttention function.
    Args:
        d_head (int):  dimensionality of heads.
        n_heads (int): number of attention heads.
    Returns:
        function: compute_attention_output function
    """

    def compute_attention_output(x):
        """ Compute the attention output.
        Args:
            x (jax.interpreters.xla.DeviceArray): tensor with shape (batch_size X n_heads, seqlen, d_head).
        Returns:
            jax.interpreters.xla.DeviceArray: reshaped tensor with shape (batch_size, seqlen, n_heads X d_head).
        """
        ### START CODE HERE (REPLACE INSTANCES OF 'None' with your code) ###

        # Length of the sequence
        # Should be size of x's first dimension without counting the batch dim
        seqlen = x.shape[1]

        # Reshape x using jnp.reshape() to shape (batch_size, n_heads, seqlen, d_head)
        x = jnp.reshape(x, ( -1, n_heads, seqlen, d_head))

        # Transpose x using jnp.transpose() to shape (batch_size, seqlen, n_heads, d_head)
        x = jnp.transpose(x, ( 0, 2, 1 , 3))

        ### END CODE HERE ###

        # Reshape to allow to concatenate the heads
        return jnp.reshape(x, (-1, seqlen, n_heads * d_head))

    return compute_attention_output
```

```
display_tensor(result_cah, "input tensor")
result_cao = compute_attention_output_closure(2,3)(result_cah)
display_tensor(result_cao, "output tensor")
```

    input tensor shape: (6, 2, 3)

    [[[1 0 0]
      [0 1 0]]

     [[1 0 0]
      [0 1 0]]

     [[1 0 0]
      [0 1 0]]

     [[1 0 0]
      [0 1 0]]

     [[1 0 0]
      [0 1 0]]

     [[1 0 0]
      [0 1 0]]]

    output tensor shape: (3, 2, 6)

    [[[1 0 0 1 0 0]
      [0 1 0 0 1 0]]

     [[1 0 0 1 0 0]
      [0 1 0 0 1 0]]

     [[1 0 0 1 0 0]
      [0 1 0 0 1 0]]]

```

def CausalAttention(d_feature,
                    n_heads,
                    compute_attention_heads_closure=compute_attention_heads_closure,
                    dot_product_self_attention=dot_product_self_attention,
                    compute_attention_output_closure=compute_attention_output_closure,
                    mode='train'):
    """Transformer-style multi-headed causal attention.

    Args:
        d_feature (int):  dimensionality of feature embedding.
        n_heads (int): number of attention heads.
        compute_attention_heads_closure (function): Closure around compute_attention heads.
        dot_product_self_attention (function): dot_product_self_attention function.
        compute_attention_output_closure (function): Closure around compute_attention_output.
        mode (str): 'train' or 'eval'.

    Returns:
        trax.layers.combinators.Serial: Multi-headed self-attention model.
    """

    assert d_feature % n_heads == 0
    d_head = d_feature // n_heads

    ### START CODE HERE (REPLACE INSTANCES OF 'None' with your code) ###

    # HINT: The second argument to tl.Fn() is an uncalled function (without the parentheses)
    # Since you are dealing with closures you might need to call the outer
    # function with the correct parameters to get the actual uncalled function.
    ComputeAttentionHeads = tl.Fn('AttnHeads', compute_attention_heads_closure(n_heads, d_head), n_out=1)

    return tl.Serial(
        tl.Branch( # creates three towers for one input, takes activations and creates queries keys and values
            [tl.Dense(d_feature), ComputeAttentionHeads], # queries
            [tl.Dense(d_feature), ComputeAttentionHeads], # keys
            [tl.Dense(d_feature), ComputeAttentionHeads], # values
        ),

        tl.Fn('DotProductAttn', dot_product_self_attention, n_out=1), # takes QKV
        # HINT: The second argument to tl.Fn() is an uncalled function
        # Since you are dealing with closures you might need to call the outer
        # function with the correct parameters to get the actual uncalled function.
        tl.Fn('AttnOutput', compute_attention_output_closure(n_heads, d_head), n_out=1), # to allow for parallel
        tl.Dense(d_feature) # Final dense layer
    )

    ### END CODE HERE ###
```

```
print(CausalAttention(d_feature=512, n_heads=8))
```

    Serial[
      Branch_out3[
        [Dense_512, AttnHeads]
        [Dense_512, AttnHeads]
        [Dense_512, AttnHeads]
      ]
      DotProductAttn_in3
      AttnOutput
      Dense_512
    ]

```

def DecoderBlock(d_model, d_ff, n_heads,
                 dropout, mode, ff_activation):
    """Returns a list of layers that implements a Transformer decoder block.

    The input is an activation tensor.

    Args:
        d_model (int):  depth of embedding.
        d_ff (int): depth of feed-forward layer.
        n_heads (int): number of attention heads.
        dropout (float): dropout rate (how much to drop out).
        mode (str): 'train' or 'eval'.
        ff_activation (function): the non-linearity in feed-forward layer.

    Returns:
        list: list of trax.layers.combinators.Serial that maps an activation tensor to an activation tensor.
    """

    ### START CODE HERE (REPLACE INSTANCES OF 'None' with your code) ###

    # Create masked multi-head attention block using CausalAttention function
    causal_attention = CausalAttention(
                        d_model,
                        n_heads=n_heads,
                        mode=mode
                        )

    # Create feed-forward block (list) with two dense layers with dropout and input normalized
    feed_forward = [
        # Normalize layer inputs
        tl.LayerNorm(),
        # Add first feed forward (dense) layer (don't forget to set the correct value for n_units)
        tl.Dense(d_ff),
        # Add activation function passed in as a parameter (you need to call it!)
        ff_activation(), # Generally ReLU
        # Add dropout with rate and mode specified (i.e., don't use dropout during evaluation)
        tl.Dropout(rate=dropout, mode=mode),
        # Add second feed forward layer (don't forget to set the correct value for n_units)
        tl.Dense(d_model),
        # Add dropout with rate and mode specified (i.e., don't use dropout during evaluation)
        tl.Dropout(rate=dropout,mode=mode)
    ]

    # Add list of two Residual blocks: the attention with normalization and dropout and feed-forward blocks
    return [
      tl.Residual(
          # Normalize layer input
          tl.LayerNorm(),
          # Add causal attention block previously defined (without parentheses)
          causal_attention,
          # Add dropout with rate and mode specified
          tl.Dropout(rate=dropout, mode=mode)
        ),
      tl.Residual(
          # Add feed forward block (without parentheses)
          feed_forward
        ),
      ]
    ### END CODE HERE ###
```

```
print(DecoderBlock(d_model=512, d_ff=2048, n_heads=8, dropout=0.1, mode='train', ff_activation=tl.Relu))
```

    [Serial[
      Branch_out2[
        None
        Serial[
          LayerNorm
          Serial[
            Branch_out3[
              [Dense_512, AttnHeads]
              [Dense_512, AttnHeads]
              [Dense_512, AttnHeads]
            ]
            DotProductAttn_in3
            AttnOutput
            Dense_512
          ]
          Dropout
        ]
      ]
      Add_in2
    ], Serial[
      Branch_out2[
        None
        Serial[
          LayerNorm
          Dense_2048
          Relu
          Dropout
          Dense_512
          Dropout
        ]
      ]
      Add_in2
    ]]

```

def TransformerLM(vocab_size=33300,
                  d_model=512,
                  d_ff=2048,
                  n_layers=6,
                  n_heads=8,
                  dropout=0.1,
                  max_len=4096,
                  mode='train',
                  ff_activation=tl.Relu):
    """Returns a Transformer language model.

    The input to the model is a tensor of tokens. (This model uses only the
    decoder part of the overall Transformer.)

    Args:
        vocab_size (int): vocab size.
        d_model (int):  depth of embedding.
        d_ff (int): depth of feed-forward layer.
        n_layers (int): number of decoder layers.
        n_heads (int): number of attention heads.
        dropout (float): dropout rate (how much to drop out).
        max_len (int): maximum symbol length for positional encoding.
        mode (str): 'train', 'eval' or 'predict', predict mode is for fast inference.
        ff_activation (function): the non-linearity in feed-forward layer.

    Returns:
        trax.layers.combinators.Serial: A Transformer language model as a layer that maps from a tensor of tokens
        to activations over a vocab set.
    """

    ### START CODE HERE (REPLACE INSTANCES OF 'None' with your code) ###

    # Embedding inputs and positional encoder
    positional_encoder = [
        # Add embedding layer of dimension (vocab_size, d_model)
        tl.Embedding(vocab_size, d_model),
        # Use dropout with rate and mode specified
        tl.Dropout(rate=dropout, mode=mode),
        # Add positional encoding layer with maximum input length and mode specified
        tl.PositionalEncoding(max_len=max_len, mode=mode)]

    # Create stack (list) of decoder blocks with n_layers with necessary parameters
    decoder_blocks = [
        DecoderBlock(d_model, d_ff, n_heads,
                    dropout, mode, ff_activation) for _ in range(n_layers)]

    # Create the complete model as written in the figure
    return tl.Serial(
        # Use teacher forcing (feed output of previous step to current step)
        tl.ShiftRight(mode=mode), # Specify the mode!
        # Add positional encoder
        positional_encoder,
        # Add decoder blocks
        decoder_blocks,
        # Normalize layer
        tl.LayerNorm(),

        # Add dense layer of vocab_size (since need to select a word to translate to)
        # (a.k.a., logits layer. Note: activation already set by ff_activation)
        tl.Dense(vocab_size),
        # Get probabilities with Logsoftmax
        tl.LogSoftmax(),
    )

    ### END CODE HERE ###
```

```
print(TransformerLM(n_layers=1))
```

    Serial[
      ShiftRight(1)
      Embedding_33300_512
      Dropout
      PositionalEncoding
      Serial[
        Branch_out2[
          None
          Serial[
            LayerNorm
            Serial[
              Branch_out3[
                [Dense_512, AttnHeads]
                [Dense_512, AttnHeads]
                [Dense_512, AttnHeads]
              ]
              DotProductAttn_in3
              AttnOutput
              Dense_512
            ]
            Dropout
          ]
        ]
        Add_in2
      ]
      Serial[
        Branch_out2[
          None
          Serial[
            LayerNorm
            Dense_2048
            Relu
            Dropout
            Dense_512
            Dropout
          ]
        ]
        Add_in2
      ]
      LayerNorm
      Dense_33300
      LogSoftmax
    ]

```
def training_loop(TransformerLM, train_gen, eval_gen, output_dir = "~/model"):
    '''
    Input:
        TransformerLM (trax.layers.combinators.Serial): The model you are building.
        train_gen (generator): Training stream of data.
        eval_gen (generator): Evaluation stream of data.
        output_dir (str): folder to save your file.

    Returns:
        trax.supervised.training.Loop: Training loop.
    '''
    output_dir = os.path.expanduser(output_dir)  # trainer is an object
    lr_schedule = trax.lr.warmup_and_rsqrt_decay(n_warmup_steps=1000, max_value=0.01)

    ### START CODE HERE (REPLACE INSTANCES OF 'None' with your code) ###
    train_task = training.TrainTask(
      labeled_data=train_gen, # The training generator
      loss_layer=tl.CrossEntropyLoss(), # Loss function
      optimizer=trax.optimizers.Adam(0.01), # Optimizer (Don't forget to set LR to 0.01)
      lr_schedule=lr_schedule,
      n_steps_per_checkpoint=10
    )

    eval_task = training.EvalTask(
      labeled_data=eval_gen, # The evaluation generator
      metrics=[tl.CrossEntropyLoss(), tl.Accuracy()] # CrossEntropyLoss and Accuracy
    )

    ### END CODE HERE ###

    loop = training.Loop(TransformerLM(d_model=4,
                                       d_ff=16,
                                       n_layers=1,
                                       n_heads=2,
                                       mode='train'),
                         train_task,
                         eval_tasks=[eval_task],
                         output_dir=output_dir)

    return loop
```

```
!rm -f ~/model/model.pkl.gz
loop = training_loop(TransformerLM, train_batch_stream, eval_batch_stream)
loop.run(10)
```

```

```


# 데모 가이드 문서

## 1) 프로젝트 다운로드

> git clone https://github.com/younhong/COVID19-Mask-Detection-Model 으로 다운로드

> 만약 git이 설치되어 있지 않다면
>> [링크](https://github.com/younhong/COVID19-Mask-Detection-Model)로 들어가서 zip 파일을 다운로드

## 2) 환경 설정

환경설정: 가상환경을 설정
Why?
> 가상환경을 설치하지 않으면 기존의 파이썬 설치된 환경과 충돌하여 의도치 않은 에러가 날 수 있다.

### Step1) 가상환경 생성에 필요한 패키지 설치
Mac의 경우
> pip install virtualenv virtualenvwrapper

Window의 경우
> pip install virtualenv virtualenvwrapper-win

### Step2) 환경변수 설정
Mac의 경우
> 아래 커맨드로 virtualenvwrapper.sh가 설치된 위치 확인(결과값을 기억해둘 것)
>> find / -name virtualenvwrapper.sh 2>/dev/null
>>
>> (경로가 두 가지 이상 나오면 둘 중에 하나 선택해서 복사)

> .zshrc 파일을 열고 환경 변수를 추가(아래 이미지 마지막 3줄 참고)
> 마지막 줄의 source 뒤의 경로를 위에서 체크한 경로로 변경
>>

> source ~/.zshrc 실행 후 재부팅

Window의 경우
> setx WORKON_HOME="프로젝트가 설치된 경로"
> 재부팅

### Step3) 가상환경 생성
> mkvirtualenv test(test 대신 다른 이름 써도 괜찮음)
>>
>> (맥 기준) 이미지 맨 아래에 표시된 부분과 같이 설정한 환경 이름이 표시되면 성공

### Step4) 설치된 가상환경에 프로젝트 실행에 필요한 파이썬 모듈 설치
> pip install -r requirements.txt

## 3) 데이터 학습 단계

> python train_mask_detector.py --dataset dataset

> dataset 폴더 안에 있는 3800개의 이미지를 가져야 20번의 epoch 동안 각각 96번의 step을 거치며 학습을 진행

## 4) 데이터 학습 결과에 대한 추가 설명

> (편의상 TP, TN, FP, FN으로 표시합니다)

### 4-1) Precision
> 모델이 True라고 예측한 값 중 실제 결과가 True의 값을 가지는 답의 비율

\begin{align}
\frac{TP}{(TP + FP)}
\end{align}

### 4-2) Recall
> 실제 True인 값 중 모델이 True라고 옳게 예측한 답의 비율

\begin{align}
\frac{TP}{(TP + FN)}
\end{align}

### 4-3) F1
> Precision과 Recall의 조화평균

\begin{align}
\frac{2 * precision * recall}{(precision + recall)}
\end{align}

### 4-4) Accuracy
> True라고 옳게 예측한 것과 False라고 옮게 예측한 것의 비율

\begin{align}
\frac{TP + TN}{(TP + FP + TN + FN)}
\end{align}

### 4-5) 자료 출처
> [출처](https://eunsukimme.github.io/ml/2019/10/21/Accuracy-Recall-Precision-F1-score/)

## 5) 데이터 검증 단계

### 실시간 웹캠 영상을 통한 검증

> python detect_mask_video.py

# 보완할 점

## 1) 한정된 데이터
> 현재 학습에 사용한 건 3800개의 이미지. 모든 상황을 커버하기는 부족하다. 더 정확한 검증을 위해서는 더 많은 학습 데이터가 필요하다.

## 2) 마스크 불량 착용자 감지
> 이 프로젝트는 마스크 착용 여부만 감지한다. 마스크를 코 끝까지 올리지 않은 사람들을 모두 감지하고 싶다면 그 부분에 해당하는 데이터와 레이블이 필요하다. 기존에 학습에 사용한 데이터만큼 많은 수의 데이터가 주어진다면 가능할 것으로 판단된다.

## 3) 관리자에게 알림
> 현재는 마스크 불량자를 감지하고 경고를 주는 것에 끝난다. 자동으로 관리자에게 메시지를 보낼 수 있도록 보완하면 좋을 것 같다.

# 기대효과

> 종교시설 등 집단 감염의 위험이 있는 장소에서 마스크 미착용자를 감지하여 건물의 출입이나 이용을 제한하거나 격리 조취를 취할 수 있다.

> 다수의 인원이 감지되더라도 실시간으로 마스크 미착용자의 수를 구분해낼 수 있다.


# High-Level Waste : Policy

In the United States, high-level readioactive waste is categorized as such not by its constituent material, but by the history of that material. Let's talk about how this came to be.

## Learning Objectives

At the end of this lesson, you will be equipped to:

- Understand the magnitude of the current waste challenge in the US.
- Recall the history of nuclear waste policy in the United States
- Understand the legislation in the Nuclear Waste Policy Act
- Recall the history of the Yucca Mountain Project
- Recognize the charter and conclusions of the Blue Ribbon Commission
- Describe the impacts of the Nuclear Waste Confidence ruling
- Understand the path and current status of the Yucca Mountain license application
- Recognize the NWPA waste class of various nuclear fuel cycle materials

## Current Status

The waste is waiting.

The locations of the used nuclear fuel and high level waste in the united states
cover over a hundred locations and nearly 40 states [5]

Spent fuel pools hold recently discharged fuel at reactor sites and elsewhere [2].

Vitrified glass logs at reprocessing facilities and elsewhere [7].

Dry casks at reactor sites and elsewhere [10].

Liquid waste in steel or carbon steel tanks at Hanford and elsewhere[4].

The amount of nuclear spent fuel and high level waste now exceeds the proposed legislative capacity of Yucca Mountain [13].

An array of options have been considered in the past [13].

The current U.S. geologic disposal concept [13].

The full history of domestic nuclear waste policy, in eras [13].

## Siting

How did we get here?

### 1940-1970 Preliminary Studies
- 1940s - first defense waste produced
- 1950s - first power reactors
- 1954 - The Atomic Energy Act directs the government to promote the peaceful use of atomic energy, with the understanding that disposal of the highly radioactive waste produced would be the responsibility of the federal government.
- 1957 - National Academy of Sciences recommends geological disposal.

### 1970 - 1983 National Site Screening

- 1978 - first Yucca Mountain borehole
- 1982 - Nuclear Waste Policy Act

### 1983 Nuclear Waste Policy Act

Passed through congress in 1982, it was signed into law in January, 1983 by President Ronald Reagan.

> This legislation defined the Federal Governments responsibility to
provide permanent disposal in a deep geologic repository for spent fuel
and high-level radioactive waste from commercial and defense
activities. Under amended provisions (1987) of this Act, the
Department of Energy (DOE) has the responsibility to locate, build,
and operate a repository for such wastes. [11]

### Site Review

The siting process began at nine sites and downselected to one [13].

The land surrounding and including Yucca Mountain was already owned by the
federal government when the site was selected.

### Selection of Yucca Mountain

The Yucca Mountain Site was selected and accepted by the office of President
Ronald Reagan in 1987.

Yucca Mountain is in southern Nevada [12].

### 1988 Viability Assessment

The 1988 **Viability Assessment** describes the nuclear waste problem and
explains why the United States and other nations are considering deep geologic
disposal as the solution. It also describes what research must be done to
complete a license application.

**Public hearings** are held about this viability assessment and with this, the
Yucca Mountain site was deemed a viable site for further characterization and
the **pursuit of a license application** began.

### 1991-2002 Site Characterization

The process of researching the suitability of Yucca Mountain began:

- 1991 - surface studies begin
- 1999 - bills for interim storage at the Nevada Test Site are introduced. Bill Clinton vows to veto them.
- 2000 - The environmental impact statement is release for public comment.
- 2001 - EPA proposes radiation standards for Yucca Mountain. The State of Nevada files suit against the EPA concerning the standards.

## Nuclear Waste Fund

The NWPA established a nuclear waste fund, to be paid into by the commercial
entities producing nuclear waste domestically.

> In return for 1 mill per kWh, the utilities would gain the assurance that the United States would manage the disposal of that waste.

Though some of the Nuclear Waste Fund has been spent on the Yucca Mountain
Project, it currently has an unspent balance of $25 billion.

It receives nearly $750 million each year from the commerical entities that pay
into it.

### Nuclear Waste Fund Litigation

- 1983 - Nuclear utilities signed the Standard Contract agreeing to pay the nuclear waste fee.
- 1998 - The utilities sue the federal government over the delay in used fuel acceptance.
- 1998 - Courts ruled the DOE is partially liable. DOE settled and DOE developed an amended Standard Contract [3] to address new reactors.

### 2002 - 2008 Site Recommendation

- 2002- The site was deemed appropriately characterized for recommendation in 2002. **Secretary of Energy$\longrightarrow$President Bush$\longrightarrow$Congress**
- 2004 - In the Nevada Lawsuit, the EPA 10,000 year radiation standard is deemed inadequate. The rest of the radiation standards remain.

### 2008 - 2011 License Application and Withdrawal

- The license application was submitted to the NRC in **2008**.
- The NRC was expected to make a decision within 4 years.
- In **May 2009**, Secretary Steven Chu stated: “Yucca Mountain as a repository is off the table”.
- **July 2009**, the House of Representatives voted 388 to 30 to not defund the Yucca Mountain repository in the FY2010 budget.
- On **March 3, 2010**, the DOE filed a motion with the NRC to withdraw its license application.
- **Lawsuits** erupted from Hanford, WA and Aiken County, SC, among others. Ruled premature.
- YMP funding was terminated with the Defense and Full-Year Continuing Appropriations Act, passed by Congress on **April 14, 2011.**

### 2012 Blue Ribbon Commission Report

The goals of the blue ribbon commission were [8]:

- Evaluation of existing fuel cycle technologies and R&D programs.
-  Options for safe storage of used nuclear fuel while final disposition pathways are selected and deployed.
- Options for permanent disposal of used fuel and/or high-level nuclear waste,
- Options to make legal and commercial arrangements for the management of used nuclear fuel and nuclear waste in a manner that takes the current and potential full fuel cycles into account.
- Options for decision-making processes for management and disposal that are flexible, adaptive, and responsive.
- Options to ensure that decisions on management of used nuclear fuel and nuclear waste are open and transparent, with broad participation.
- The possible need for additional legislation or amendments to existing laws, including the Nuclear Waste Policy Act of 1982, as amended.

The BRC came out with a list of recommendations [1] :

- Consentbased siting approach
- An authoritative organization dedicated solely to implementing the waste management program.
- Access to the waste fund.
- Prompt efforts : one or more geologic disposal facilities.
- Prompt efforts : one or more consolidated storage facilities.
- Prompt efforts : to prepare for SNF and HLW transportation.
- Support for continued U.S. innovation in nuclear energy technology and for workforce development;
- U.S. leadership in international efforts to address safety, waste management, nonproliferation, and security concerns.

The Senate tried to take action:
- Senator Jeff Bingaman, before leaving office, introduced a bill called the Nuclear Waste Administration Act of 2011. [Its daughter went before the senate in 2012.]( http://www.energy.senate.gov/public/index.cfm/2012/8/s-3469-the-nuclear-waste-administration-act-of-2012) It died in congress.

The Judicial system tried to take action related to the Waste Confidence Rule:

- To license a reactor or approve a reactor license extension, the NRC requires a Waste Confidence clause. It is a section of the application that assures the NRC that waste produced by the applying facility will be dealt with.
- Previous to the closing of the Yucca Mountain Project, all applicants for reactor licenses and license extensinos were able to cite the nuclear waste fund and the Yucca Mountain Project.
- In 2012, the United States Court of Appeals for the District of Columbia Circuit declared that there is no waste confidence.

> “The commission apparently has no long-term plan other than hoping for a geologic repository... Failing to establish a repository is a possibility thatcannot be ignored.” [14]

### 2013 DOE Response to Blue Ribbon Commission

DOE created four working groups.
- Governance Framework and Funding
- System Design & Architecture
- ConsentBased Siting
- Transportation Routing, Safety and Security

In a brief statement [[6]](http://www.energy.gov/downloads/strategy-management-and-disposal-used-nuclear-fuel-and-high-level-radioactive-waste), the DOE agreed in general with the BRC and said that DOE would pursue a timeline :

- Sites, designs and licenses, constructs and begins operations of a pilot interim storage facility by **2021** with an initial focus on accepting used nuclear fuel from shut-down reactor sites.
- Advances toward the siting and licensing of a larger interim storage facility to be available by **2025**.
-  Makes **demonstrable progress** on the siting and characterization of repository sites to facilitate the availability of a geologic repository by 2048.

### 2013 Legal Decisions

- May: DOE is ordered by a court to stop collecting the waste fee. Representing 750 million a year when it ended, the fund now holds 37 billion.
- August-October : NRC is ordered by the DC court of appeals to restart the licensing proceeding. The appropriated funds were insufficient to complete the proceeding.

### 2014-2015 NRC Conclusions

- NRC report concludes [continued storage in dry casks is safe for 160 years](http://www.yuccamountain.org/pdf/nrc_rule0814.pdf), brushing aside the immediate consequences of the judicial decision concerning waste confidence.
- December 2014-Jan 2015: NRC releases the final volumes of the reports on the suitability of Yucca Mountain as a disposal spot for nuclear waste, finding that the design met the commission's requirements.

### 2016

- DOE issues [a statement](http://www.yuccamountain.org/pdf/2015-32346.pdf) declaring that it would move forward with a consent-based siting process.

### 2017 - 2018

- March 2: Rick Perry becomes Secretary of Energy.
- White House budget contains \$120 million for Yucca.
- April: A hearing to amend the NWPA
- May: No money in federal spending budget for Yucca.
- Nov 2: A bill to restart licensing Yucca Mountain did not pass the Senate.
- [The “Consent-Based Siting” tab on energy.gov](https://www.energy.gov/ne/consent-based-siting)

### 2019-2020
- The Trump administration made initial budget requests to support pursuit of Yucca Mountain, but Senators Masto (D-Nev.),  Heller (R-Nev.), and others urged the US Energy & Water committee to pursue a consent-based siting approach.
- The budget ultimately reflected that "alternative approaches" would be taken, rather than Yucca Mountain.

### 2021
- The Biden-Harris administration, including DOE Secretary Granholm have indicated that Yucca Mountain remains off the table and that a consent-based approach is preferred.

## Waste Classification

Waste classification is an area of active discussion.

[source](https://www.fas.org/sgp/crs/misc/RL32163.pdf)

[source](https://www.fas.org/sgp/crs/misc/RL32163.pdf)

```python
from IPython.display import IFrame
IFrame("https://embed.polleverywhere.com/free_text_polls/JfkmGGnQJfwt8Mr?controls=none&short_poll=true",width=700, height=700)
```

## Radioactivity

Recall: The SI unit of activity is the becquerel (Bq), equal to one reciprocal second.

\begin{align}
A(i) &= -\frac{dN_i}{dt}\\
&= \lambda_iN_i
\end{align}

## Toxicity

One possibility is to use toxicity as a classifying metric. It captures the activity and danger to humans

\begin{align}
\mbox{Toxicity}(i,k) &= \frac{A(i)}{DAC(i,k)}\\
i &= \mbox{isotope}\\
k &= \mbox{medium index, air or water}\\
DAC &= \mbox{derived air concentration of isotope i in air or water.}
\end{align}

Warning : DAC is confusing. It generally is reported in $\frac{\mu Ci}{mL}$. From the NRC glossary, the definition is:

> Derived air concentration (DAC) The concentration of a given radionuclide in air which, if breathed by the reference man for a working year of 2,000 hours under conditions of light work (with an inhalation rate of 1.2 cubic meters of air per hour), results in an intake of one annual limit on intake (ALI).

Ok, great, so what is ALI?

> ALI is the smaller value of intake of a given radionuclide in a year by the "reference man" that would result in a committed effective dose equivalent (CEDE) of 5 rems (0.05 sievert) or a committed dose equivalent (CDE) of 50 rems (0.5 sievert) to any individual organ or tissue.

[source](https://www.fas.org/sgp/crs/misc/RL32163.pdf)

A flow chart for improvements is below.

[source](http://www.cresp.org/NuclearChemCourse/monographs/12_Viena_Waste%20Forms%20for%20an%20Advanced%20Fuel%20Cycle_3_3_09.pdf)

## Wrap-up

- Understand the magnitude of the current waste challenge in the US.
- Recall the history of nuclear waste policy in the United States
- Understand the legislation in the Nuclear Waste Policy Act
- Recall the history of the Yucca Mountain Project
- Recognize the charter and conclusions of the Blue Ribbon Commission
- Describe the impacts of the Nuclear Waste Confidence ruling
- Understand the path and current status of the Yucca Mountain license application
- Recognize the NWPA waste class of various nuclear fuel cycle materials

## References

This section was developed to complement pages 302-373 of [15] as well as the full content of [16].

[1] Mark Ayers, Vicky Bailey, Albert Carnesale, Pete Domenici, Susan Eisenhower, Chuck Hagel, Jonathan Lash, Allison Macfarlane, Richard Meserve, Ernest Moniz, Per Peterson, John Rowe, and Phil Sharp.
Report to the secretary of energy. Technical report, Blue Ribbon Commission on America’s Nuclear Future, January 2012.

[2] DOE. Spent fuel pool at a nuclear power plant.
http://www.ocrwm.doe.gov/curriculum/unit1/lesson3reading.shtml.

[3] DOE. Standard contract amendment for new reactors.
http://energy.gov/gc/downloads/standard-contract-amendment-new-reactors.

[4] DOE. Underground tank farm with 12 of the site’s 177 waste storage tanks.

[5] DOE. Final supplemental environmental impact statement for a geologic repository for the disposal of
spent nuclear fuel and high-level radioactive waste at yucca mountain, nye county, nevada. Technical Report DOE/EIS-0250F-S1, June 2008.

[6] DOE. Strategy for the management and disposal of used nuclear fuel and high-level radioactive waste. Technical report, United States Department of Energy (DOE), Washington D.C., United States,
January 2013.

[7] Peter Essick. Photographing the infinity room - timing is everything, April 2012.

[8] Carol Matthews. Blue ribbon commission on americas nuclear future U.S. department of energy - advisory
committee charter, March 2010.

[9] NRC. Nuclear-fuel.jpg.

[10] NRC. Dry cask storage., January 2008.

[11] NRC. Backgrounder on radioactive waste.
Technical report, Nuclear Regulatory Commission, February 2011.

[12] OMB. Yucca mountain 2.jpg, August 2006.

[13] Mark Peters. What’s next for used nuclear fuel and nuclear waste management policy?, January 2013.

[14] David B. Sentelle. On petitions for review of orders of the nuclear regulatory commission.

[15] N. Tsoulfanidis, The Nuclear Fuel Cycle. La Grange Park, Illinois, USA: American Nuclear Society, 2013.

[16] Mark Ayers, Vicky Bailey, Albert Carnesale, Pete Domenici, Susan Eisenhower, Chuck Hagel, Jonathan
Lash, Allison Macfarlane, Richard Meserve, Ernest Moniz, Per Peterson, John Rowe, and Phil Sharp.
Report to the Secretary of Energy. Technical report, Blue Ribbon Commission on America’s Nuclear
Future, January 2012. URL: [http://energy.gov/sites/prod/files/2013/04/f0/brc_finalreport_
jan2012.pdf](http://energy.gov/sites/prod/files/2013/04/f0/brc_finalreport_
jan2012.pdf)

```python

```