```
%tensorflow_version 1.x
```

    TensorFlow 1.x selected.

```
!npx degit google-research/pegasus -f
```

    npx: installed 1 in 1.899s
    > destination directory is not empty. Using --force, continuing
    > cloned google-research/pegasus#master

```
import os
os.environ['PYTHONPATH'] += ':/content'
```

```
!pip install -r requirements.txt
```

    Requirement already satisfied: absl-py in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 1)) (0.10.0)
    Collecting mock
      Downloading https://files.pythonhosted.org/packages/5c/03/b7e605db4a57c0f6fba744b11ef3ddf4ddebcada35022927a2b5fc623fdf/mock-4.0.3-py3-none-any.whl
    Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 3)) (1.19.5)
    Collecting rouge-score
      Downloading https://files.pythonhosted.org/packages/1f/56/a81022436c08b9405a5247b71635394d44fe7e1dbedc4b28c740e09c2840/rouge_score-0.0.4-py2.py3-none-any.whl
    Collecting sacrebleu
      Downloading https://files.pythonhosted.org/packages/7e/57/0c7ca4e31a126189dab99c19951910bd081dea5bbd25f24b77107750eae7/sacrebleu-1.5.1-py3-none-any.whl (54kB)
         |████████████████████████████████| 61kB 7.5MB/s
    Collecting sentencepiece
      Downloading https://files.pythonhosted.org/packages/f5/99/e0808cb947ba10f575839c43e8fafc9cc44e4a7a2c8f79c60db48220a577/sentencepiece-0.1.95-cp37-cp37m-manylinux2014_x86_64.whl (1.2MB)
         |████████████████████████████████| 1.2MB 31.9MB/s
    Collecting tensorflow-text==1.15.0rc0
      Downloading https://files.pythonhosted.org/packages/dd/67/9e503c06d4a8d451ffaa9e9da587e51529292360cb69a2ade0bf691a0871/tensorflow_text-1.15.0rc0-cp37-cp37m-manylinux1_x86_64.whl (8.6MB)
         |████████████████████████████████| 8.6MB 52.3MB/s
    Collecting tensor2tensor==1.15.0
      Downloading https://files.pythonhosted.org/packages/0f/a6/0f6a95a5548cf9fbfed0bbec69959355bda594e85298ebbba5f808ca0fc5/tensor2tensor-1.15.0-py2.py3-none-any.whl (1.4MB)
         |████████████████████████████████| 1.4MB 49.9MB/s
    Collecting tfds-nightly
      Downloading https://files.pythonhosted.org/packages/4f/3a/5853827bf08f60349d4f50ae13c46310e4e80ae1c2630ef5788dbb452250/tfds_nightly-4.2.0.dev202103230107-py3-none-any.whl (3.8MB)
         |████████████████████████████████| 3.8MB 42.8MB/s
    Collecting tensorflow-gpu==1.15.2
      Downloading https://files.pythonhosted.org/packages/92/42/7bddc0d5bc169596fbd13b6e1b844f832491ab671381e483da3bf5292ca9/tensorflow_gpu-1.15.2-cp37-cp37m-manylinux2010_x86_64.whl (410.9MB)
         |████████████████████████████████| 411.0MB 39kB/s
    Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from absl-py->-r requirements.txt (line 1)) (1.15.0)
    Requirement already satisfied: nltk in /usr/local/lib/python3.7/dist-packages (from rouge-score->-r requirements.txt (line 4)) (3.2.5)
    Collecting portalocker==2.0.0
      Downloading https://files.pythonhosted.org/packages/89/a6/3814b7107e0788040870e8825eebf214d72166adf656ba7d4bf14759a06a/portalocker-2.0.0-py2.py3-none-any.whl
    Requirement already satisfied: tensorflow<1.16,>=1.15.0 in /tensorflow-1.15.2/python3.7 (from tensorflow-text==1.15.0rc0->-r requirements.txt (line 7)) (1.15.2)
    Requirement already satisfied: sympy in /usr/local/lib/python3.7/dist-packages (from tensor2tensor==1.15.0->-r requirements.txt (line 8)) (1.7.1)
    Requirement already satisfied: opencv-python in /usr/local/lib/python3.7/dist-packages (from tensor2tensor==1.15.0->-r requirements.txt (line 8)) (4.1.2.30)
    Requirement already satisfied: tensorflow-probability==0.7.0 in /tensorflow-1.15.2/python3.7 (from tensor2tensor==1.15.0->-r requirements.txt (line 8)) (0.7.0)
    Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from tensor2tensor==1.15.0->-r requirements.txt (line 8)) (2.23.0)
    Requirement already satisfied: gunicorn in /tensorflow-1.15.2/python3.7 (from tensor2tensor==1.15.0->-r requirements.txt (line 8)) (20.0.4)
    Requirement already satisfied: flask in /usr/local/lib/python3.7/dist-packages (from tensor2tensor==1.15.0->-r requirements.txt (line 8)) (1.1.2)
    Requirement already satisfied: Pillow in /usr/local/lib/python3.7/dist-packages (from tensor2tensor==1.15.0->-r requirements.txt (line 8)) (7.0.0)
    Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from tensor2tensor==1.15.0->-r requirements.txt (line 8)) (1.4.1)
    Requirement already satisfied: google-api-python-client in /usr/local/lib/python3.7/dist-packages (from tensor2tensor==1.15.0->-r requirements.txt (line 8)) (1.12.8)
    Requirement already satisfied: kfac in /tensorflow-1.15.2/python3.7 (from tensor2tensor==1.15.0->-r requirements.txt (line 8)) (0.2.0)
    Requirement already satisfied: mesh-tensorflow in /tensorflow-1.15.2/python3.7 (from tensor2tensor==1.15.0->-r requirements.txt (line 8)) (0.1.12)
    Requirement already satisfied: future in /usr/local/lib/python3.7/dist-packages (from tensor2tensor==1.15.0->-r requirements.txt (line 8)) (0.16.0)
    Requirement already satisfied: h5py in /usr/local/lib/python3.7/dist-packages (from tensor2tensor==1.15.0->-r requirements.txt (line 8)) (2.10.0)
    Requirement already satisfied: tensorflow-gan in /tensorflow-1.15.2/python3.7 (from tensor2tensor==1.15.0->-r requirements.txt (line 8)) (2.0.0)
    Requirement already satisfied: dopamine-rl in /usr/local/lib/python3.7/dist-packages (from tensor2tensor==1.15.0->-r requirements.txt (line 8)) (1.0.5)
    Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from tensor2tensor==1.15.0->-r requirements.txt (line 8)) (4.41.1)
    Requirement already satisfied: tensorflow-datasets in /usr/local/lib/python3.7/dist-packages (from tensor2tensor==1.15.0->-r requirements.txt (line 8)) (4.0.1)
    Requirement already satisfied: oauth2client in /usr/local/lib/python3.7/dist-packages (from tensor2tensor==1.15.0->-r requirements.txt (line 8)) (4.1.3)
    Requirement already satisfied: gevent in /tensorflow-1.15.2/python3.7 (from tensor2tensor==1.15.0->-r requirements.txt (line 8)) (1.4.0)
    Requirement already satisfied: gin-config in /usr/local/lib/python3.7/dist-packages (from tensor2tensor==1.15.0->-r requirements.txt (line 8)) (0.4.0)
    Requirement already satisfied: gym in /usr/local/lib/python3.7/dist-packages (from tensor2tensor==1.15.0->-r requirements.txt (line 8)) (0.17.3)
    Requirement already satisfied: pypng in /tensorflow-1.15.2/python3.7 (from tensor2tensor==1.15.0->-r requirements.txt (line 8)) (0.0.20)
    Requirement already satisfied: bz2file in /tensorflow-1.15.2/python3.7 (from tensor2tensor==1.15.0->-r requirements.txt (line 8)) (0.98)
    Collecting tf-slim
      Downloading https://files.pythonhosted.org/packages/02/97/b0f4a64df018ca018cc035d44f2ef08f91e2e8aa67271f6f19633a015ff7/tf_slim-1.1.0-py2.py3-none-any.whl (352kB)
         |████████████████████████████████| 358kB 52.4MB/s
    Requirement already satisfied: tensorflow-metadata in /usr/local/lib/python3.7/dist-packages (from tfds-nightly->-r requirements.txt (line 9)) (0.28.0)
    Requirement already satisfied: dill in /usr/local/lib/python3.7/dist-packages (from tfds-nightly->-r requirements.txt (line 9)) (0.3.3)
    Requirement already satisfied: importlib-resources; python_version < "3.9" in /usr/local/lib/python3.7/dist-packages (from tfds-nightly->-r requirements.txt (line 9)) (5.1.2)
    Requirement already satisfied: termcolor in /usr/local/lib/python3.7/dist-packages (from tfds-nightly->-r requirements.txt (line 9)) (1.1.0)
    Requirement already satisfied: protobuf>=3.12.2 in /usr/local/lib/python3.7/dist-packages (from tfds-nightly->-r requirements.txt (line 9)) (3.12.4)
    Requirement already satisfied: typing-extensions; python_version < "3.8" in /usr/local/lib/python3.7/dist-packages (from tfds-nightly->-r requirements.txt (line 9)) (3.7.4.3)
    Requirement already satisfied: promise in /usr/local/lib/python3.7/dist-packages (from tfds-nightly->-r requirements.txt (line 9)) (2.3)
    Requirement already satisfied: attrs>=18.1.0 in /usr/local/lib/python3.7/dist-packages (from tfds-nightly->-r requirements.txt (line 9)) (20.3.0)
    Requirement already satisfied: tensorboard<1.16.0,>=1.15.0 in /tensorflow-1.15.2/python3.7 (from tensorflow-gpu==1.15.2->-r requirements.txt (line 10)) (1.15.0)
    Collecting gast==0.2.2
      Downloading https://files.pythonhosted.org/packages/4e/35/11749bf99b2d4e3cceb4d55ca22590b0d7c2c62b9de38ac4a4a7f4687421/gast-0.2.2.tar.gz
    Requirement already satisfied: astor>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow-gpu==1.15.2->-r requirements.txt (line 10)) (0.8.1)
    Requirement already satisfied: tensorflow-estimator==1.15.1 in /tensorflow-1.15.2/python3.7 (from tensorflow-gpu==1.15.2->-r requirements.txt (line 10)) (1.15.1)
    Requirement already satisfied: keras-preprocessing>=1.0.5 in /usr/local/lib/python3.7/dist-packages (from tensorflow-gpu==1.15.2->-r requirements.txt (line 10)) (1.1.2)
    Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.7/dist-packages (from tensorflow-gpu==1.15.2->-r requirements.txt (line 10)) (1.32.0)
    Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow-gpu==1.15.2->-r requirements.txt (line 10)) (3.3.0)
    Requirement already satisfied: google-pasta>=0.1.6 in /usr/local/lib/python3.7/dist-packages (from tensorflow-gpu==1.15.2->-r requirements.txt (line 10)) (0.2.0)
    Requirement already satisfied: keras-applications>=1.0.8 in /tensorflow-1.15.2/python3.7 (from tensorflow-gpu==1.15.2->-r requirements.txt (line 10)) (1.0.8)
    Requirement already satisfied: wrapt>=1.11.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow-gpu==1.15.2->-r requirements.txt (line 10)) (1.12.1)
    Requirement already satisfied: wheel>=0.26; python_version >= "3" in /usr/local/lib/python3.7/dist-packages (from tensorflow-gpu==1.15.2->-r requirements.txt (line 10)) (0.36.2)
    Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.7/dist-packages (from sympy->tensor2tensor==1.15.0->-r requirements.txt (line 8)) (1.2.1)
    Requirement already satisfied: cloudpickle>=0.6.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow-probability==0.7.0->tensor2tensor==1.15.0->-r requirements.txt (line 8)) (1.3.0)
    Requirement already satisfied: decorator in /usr/local/lib/python3.7/dist-packages (from tensorflow-probability==0.7.0->tensor2tensor==1.15.0->-r requirements.txt (line 8)) (4.4.2)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->tensor2tensor==1.15.0->-r requirements.txt (line 8)) (2020.12.5)
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->tensor2tensor==1.15.0->-r requirements.txt (line 8)) (3.0.4)
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->tensor2tensor==1.15.0->-r requirements.txt (line 8)) (2.10)
    Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->tensor2tensor==1.15.0->-r requirements.txt (line 8)) (1.24.3)
    Requirement already satisfied: setuptools>=3.0 in /usr/local/lib/python3.7/dist-packages (from gunicorn->tensor2tensor==1.15.0->-r requirements.txt (line 8)) (54.1.2)
    Requirement already satisfied: click>=5.1 in /usr/local/lib/python3.7/dist-packages (from flask->tensor2tensor==1.15.0->-r requirements.txt (line 8)) (7.1.2)
    Requirement already satisfied: Jinja2>=2.10.1 in /usr/local/lib/python3.7/dist-packages (from flask->tensor2tensor==1.15.0->-r requirements.txt (line 8)) (2.11.3)
    Requirement already satisfied: itsdangerous>=0.24 in /usr/local/lib/python3.7/dist-packages (from flask->tensor2tensor==1.15.0->-r requirements.txt (line 8)) (1.1.0)
    Requirement already satisfied: Werkzeug>=0.15 in /usr/local/lib/python3.7/dist-packages (from flask->tensor2tensor==1.15.0->-r requirements.txt (line 8)) (1.0.1)
    Requirement already satisfied: google-auth>=1.16.0 in /usr/local/lib/python3.7/dist-packages (from google-api-python-client->tensor2tensor==1.15.0->-r requirements.txt (line 8)) (1.27.1)
    Requirement already satisfied: google-auth-httplib2>=0.0.3 in /usr/local/lib/python3.7/dist-packages (from google-api-python-client->tensor2tensor==1.15.0->-r requirements.txt (line 8)) (0.0.4)
    Requirement already satisfied: uritemplate<4dev,>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from google-api-python-client->tensor2tensor==1.15.0->-r requirements.txt (line 8)) (3.0.1)
    Requirement already satisfied: httplib2<1dev,>=0.15.0 in /usr/local/lib/python3.7/dist-packages (from google-api-python-client->tensor2tensor==1.15.0->-r requirements.txt (line 8)) (0.17.4)
    Requirement already satisfied: google-api-core<2dev,>=1.21.0 in /usr/local/lib/python3.7/dist-packages (from google-api-python-client->tensor2tensor==1.15.0->-r requirements.txt (line 8)) (1.26.1)
    Requirement already satisfied: tensorflow-hub>=0.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow-gan->tensor2tensor==1.15.0->-r requirements.txt (line 8)) (0.11.0)
    Requirement already satisfied: dm-tree in /usr/local/lib/python3.7/dist-packages (from tensorflow-datasets->tensor2tensor==1.15.0->-r requirements.txt (line 8)) (0.1.5)
    Requirement already satisfied: pyasn1>=0.1.7 in /usr/local/lib/python3.7/dist-packages (from oauth2client->tensor2tensor==1.15.0->-r requirements.txt (line 8)) (0.4.8)
    Requirement already satisfied: rsa>=3.1.4 in /usr/local/lib/python3.7/dist-packages (from oauth2client->tensor2tensor==1.15.0->-r requirements.txt (line 8)) (4.7.2)
    Requirement already satisfied: pyasn1-modules>=0.0.5 in /usr/local/lib/python3.7/dist-packages (from oauth2client->tensor2tensor==1.15.0->-r requirements.txt (line 8)) (0.2.8)
    Requirement already satisfied: greenlet>=0.4.14; platform_python_implementation == "CPython" in /tensorflow-1.15.2/python3.7 (from gevent->tensor2tensor==1.15.0->-r requirements.txt (line 8)) (0.4.15)
    Requirement already satisfied: pyglet<=1.5.0,>=1.4.0 in /usr/local/lib/python3.7/dist-packages (from gym->tensor2tensor==1.15.0->-r requirements.txt (line 8)) (1.5.0)
    Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow-metadata->tfds-nightly->-r requirements.txt (line 9)) (1.53.0)
    Requirement already satisfied: zipp>=0.4; python_version < "3.8" in /usr/local/lib/python3.7/dist-packages (from importlib-resources; python_version < "3.9"->tfds-nightly->-r requirements.txt (line 9)) (3.4.1)
    Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.7/dist-packages (from tensorboard<1.16.0,>=1.15.0->tensorflow-gpu==1.15.2->-r requirements.txt (line 10)) (3.3.4)
    Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.7/dist-packages (from Jinja2>=2.10.1->flask->tensor2tensor==1.15.0->-r requirements.txt (line 8)) (1.1.1)
    Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from google-auth>=1.16.0->google-api-python-client->tensor2tensor==1.15.0->-r requirements.txt (line 8)) (4.2.1)
    Requirement already satisfied: packaging>=14.3 in /usr/local/lib/python3.7/dist-packages (from google-api-core<2dev,>=1.21.0->google-api-python-client->tensor2tensor==1.15.0->-r requirements.txt (line 8)) (20.9)
    Requirement already satisfied: pytz in /usr/local/lib/python3.7/dist-packages (from google-api-core<2dev,>=1.21.0->google-api-python-client->tensor2tensor==1.15.0->-r requirements.txt (line 8)) (2018.9)
    Requirement already satisfied: importlib-metadata; python_version < "3.8" in /usr/local/lib/python3.7/dist-packages (from markdown>=2.6.8->tensorboard<1.16.0,>=1.15.0->tensorflow-gpu==1.15.2->-r requirements.txt (line 10)) (3.7.2)
    Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging>=14.3->google-api-core<2dev,>=1.21.0->google-api-python-client->tensor2tensor==1.15.0->-r requirements.txt (line 8)) (2.4.7)
    Building wheels for collected packages: gast
      Building wheel for gast (setup.py) ... done
      Created wheel for gast: filename=gast-0.2.2-cp37-none-any.whl size=7540 sha256=6c58354f576deb4f98fe89e7e4d440480690a87e86267d19fadeb36b00b22c7c
      Stored in directory: /root/.cache/pip/wheels/5c/2e/7e/a1d4d4fcebe6c381f378ce7743a3ced3699feb89bcfbdadadd
    Successfully built gast
    Installing collected packages: mock, rouge-score, portalocker, sacrebleu, sentencepiece, tensorflow-text, tf-slim, tensor2tensor, tfds-nightly, gast, tensorflow-gpu
      Found existing installation: tensor2tensor 1.14.1
        Uninstalling tensor2tensor-1.14.1:
          Successfully uninstalled tensor2tensor-1.14.1
      Found existing installation: gast 0.3.3
        Uninstalling gast-0.3.3:
          Successfully uninstalled gast-0.3.3
    Successfully installed gast-0.2.2 mock-4.0.3 portalocker-2.0.0 rouge-score-0.0.4 sacrebleu-1.5.1 sentencepiece-0.1.95 tensor2tensor-1.15.0 tensorflow-gpu-1.15.2 tensorflow-text-1.15.0rc0 tf-slim-1.1.0 tfds-nightly-4.2.0.dev202103230107

```
!pip install mock rouge-score sacrebleu sentencepiece tensorflow-text==1.15.1
```

    Requirement already satisfied: mock in /usr/local/lib/python3.7/dist-packages (4.0.3)
    Requirement already satisfied: rouge-score in /usr/local/lib/python3.7/dist-packages (0.0.4)
    Requirement already satisfied: sacrebleu in /usr/local/lib/python3.7/dist-packages (1.5.1)
    Requirement already satisfied: sentencepiece in /usr/local/lib/python3.7/dist-packages (0.1.95)
    Collecting tensorflow-text==1.15.1
      Downloading https://files.pythonhosted.org/packages/51/10/9f45f5ae2b9b082c312c6fbfd19e05d8b8917f4d4c4c5ffc3fae8a5bd3c7/tensorflow_text-1.15.1-cp37-cp37m-manylinux1_x86_64.whl (9.1MB)
         |████████████████████████████████| 9.1MB 11.7MB/s
    Requirement already satisfied: nltk in /usr/local/lib/python3.7/dist-packages (from rouge-score) (3.2.5)
    Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from rouge-score) (1.19.5)
    Requirement already satisfied: six>=1.14.0 in /usr/local/lib/python3.7/dist-packages (from rouge-score) (1.15.0)
    Requirement already satisfied: absl-py in /usr/local/lib/python3.7/dist-packages (from rouge-score) (0.10.0)
    Requirement already satisfied: portalocker==2.0.0 in /usr/local/lib/python3.7/dist-packages (from sacrebleu) (2.0.0)
    Requirement already satisfied: tensorflow<1.16,>=1.15.0 in /tensorflow-1.15.2/python3.7 (from tensorflow-text==1.15.1) (1.15.2)
    Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<1.16,>=1.15.0->tensorflow-text==1.15.1) (1.1.0)
    Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.7/dist-packages (from tensorflow<1.16,>=1.15.0->tensorflow-text==1.15.1) (1.32.0)
    Requirement already satisfied: keras-preprocessing>=1.0.5 in /usr/local/lib/python3.7/dist-packages (from tensorflow<1.16,>=1.15.0->tensorflow-text==1.15.1) (1.1.2)
    Requirement already satisfied: wrapt>=1.11.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow<1.16,>=1.15.0->tensorflow-text==1.15.1) (1.12.1)
    Requirement already satisfied: google-pasta>=0.1.6 in /usr/local/lib/python3.7/dist-packages (from tensorflow<1.16,>=1.15.0->tensorflow-text==1.15.1) (0.2.0)
    Requirement already satisfied: tensorflow-estimator==1.15.1 in /tensorflow-1.15.2/python3.7 (from tensorflow<1.16,>=1.15.0->tensorflow-text==1.15.1) (1.15.1)
    Requirement already satisfied: keras-applications>=1.0.8 in /tensorflow-1.15.2/python3.7 (from tensorflow<1.16,>=1.15.0->tensorflow-text==1.15.1) (1.0.8)
    Requirement already satisfied: protobuf>=3.6.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow<1.16,>=1.15.0->tensorflow-text==1.15.1) (3.12.4)
    Requirement already satisfied: gast==0.2.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow<1.16,>=1.15.0->tensorflow-text==1.15.1) (0.2.2)
    Requirement already satisfied: astor>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<1.16,>=1.15.0->tensorflow-text==1.15.1) (0.8.1)
    Requirement already satisfied: tensorboard<1.16.0,>=1.15.0 in /tensorflow-1.15.2/python3.7 (from tensorflow<1.16,>=1.15.0->tensorflow-text==1.15.1) (1.15.0)
    Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow<1.16,>=1.15.0->tensorflow-text==1.15.1) (3.3.0)
    Requirement already satisfied: wheel>=0.26; python_version >= "3" in /usr/local/lib/python3.7/dist-packages (from tensorflow<1.16,>=1.15.0->tensorflow-text==1.15.1) (0.36.2)
    Requirement already satisfied: h5py in /usr/local/lib/python3.7/dist-packages (from keras-applications>=1.0.8->tensorflow<1.16,>=1.15.0->tensorflow-text==1.15.1) (2.10.0)
    Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from protobuf>=3.6.1->tensorflow<1.16,>=1.15.0->tensorflow-text==1.15.1) (54.1.2)
    Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.7/dist-packages (from tensorboard<1.16.0,>=1.15.0->tensorflow<1.16,>=1.15.0->tensorflow-text==1.15.1) (1.0.1)
    Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.7/dist-packages (from tensorboard<1.16.0,>=1.15.0->tensorflow<1.16,>=1.15.0->tensorflow-text==1.15.1) (3.3.4)
    Requirement already satisfied: importlib-metadata; python_version < "3.8" in /usr/local/lib/python3.7/dist-packages (from markdown>=2.6.8->tensorboard<1.16.0,>=1.15.0->tensorflow<1.16,>=1.15.0->tensorflow-text==1.15.1) (3.7.2)
    Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata; python_version < "3.8"->markdown>=2.6.8->tensorboard<1.16.0,>=1.15.0->tensorflow<1.16,>=1.15.0->tensorflow-text==1.15.1) (3.4.1)
    Requirement already satisfied: typing-extensions>=3.6.4; python_version < "3.8" in /usr/local/lib/python3.7/dist-packages (from importlib-metadata; python_version < "3.8"->markdown>=2.6.8->tensorboard<1.16.0,>=1.15.0->tensorflow<1.16,>=1.15.0->tensorflow-text==1.15.1) (3.7.4.3)
    Installing collected packages: tensorflow-text
      Found existing installation: tensorflow-text 1.15.0rc0
        Uninstalling tensorflow-text-1.15.0rc0:
          Successfully uninstalled tensorflow-text-1.15.0rc0
    Successfully installed tensorflow-text-1.15.1

```
mkdir ckpt
```

```
!pip install tensorflow-datasets==3.0.0
!pip install tfds-nightly==3.1.0.dev202006230105
```

    Collecting tensorflow-datasets==3.0.0
      Downloading https://files.pythonhosted.org/packages/76/41/8404fabb295025c1ced13bb150246f75770619889d8637afa523a4d09cdb/tensorflow_datasets-3.0.0-py3-none-any.whl (3.3MB)
         |████████████████████████████████| 3.3MB 16.4MB/s
    Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from tensorflow-datasets==3.0.0) (1.19.5)
    Requirement already satisfied: attrs>=18.1.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow-datasets==3.0.0) (20.3.0)
    Requirement already satisfied: promise in /usr/local/lib/python3.7/dist-packages (from tensorflow-datasets==3.0.0) (2.3)
    Requirement already satisfied: absl-py in /usr/local/lib/python3.7/dist-packages (from tensorflow-datasets==3.0.0) (0.10.0)
    Requirement already satisfied: requests>=2.19.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow-datasets==3.0.0) (2.23.0)
    Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from tensorflow-datasets==3.0.0) (1.15.0)
    Requirement already satisfied: termcolor in /usr/local/lib/python3.7/dist-packages (from tensorflow-datasets==3.0.0) (1.1.0)
    Requirement already satisfied: future in /usr/local/lib/python3.7/dist-packages (from tensorflow-datasets==3.0.0) (0.16.0)
    Requirement already satisfied: dill in /usr/local/lib/python3.7/dist-packages (from tensorflow-datasets==3.0.0) (0.3.3)
    Requirement already satisfied: tensorflow-metadata in /usr/local/lib/python3.7/dist-packages (from tensorflow-datasets==3.0.0) (0.28.0)
    Requirement already satisfied: wrapt in /usr/local/lib/python3.7/dist-packages (from tensorflow-datasets==3.0.0) (1.12.1)
    Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from tensorflow-datasets==3.0.0) (4.41.1)
    Requirement already satisfied: protobuf>=3.6.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow-datasets==3.0.0) (3.12.4)
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19.0->tensorflow-datasets==3.0.0) (2.10)
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19.0->tensorflow-datasets==3.0.0) (3.0.4)
    Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19.0->tensorflow-datasets==3.0.0) (1.24.3)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19.0->tensorflow-datasets==3.0.0) (2020.12.5)
    Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow-metadata->tensorflow-datasets==3.0.0) (1.53.0)
    Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from protobuf>=3.6.1->tensorflow-datasets==3.0.0) (54.1.2)
    Installing collected packages: tensorflow-datasets
      Found existing installation: tensorflow-datasets 4.0.1
        Uninstalling tensorflow-datasets-4.0.1:
          Successfully uninstalled tensorflow-datasets-4.0.1
    Successfully installed tensorflow-datasets-3.0.0
    Collecting tfds-nightly==3.1.0.dev202006230105
      Downloading https://files.pythonhosted.org/packages/09/fa/877491b455a10fe38fa8902e1339c99bcd48890a47351cfce26f408d1537/tfds_nightly-3.1.0.dev202006230105-py3-none-any.whl (3.4MB)
         |████████████████████████████████| 3.4MB 19.4MB/s
    Requirement already satisfied: protobuf>=3.6.1 in /usr/local/lib/python3.7/dist-packages (from tfds-nightly==3.1.0.dev202006230105) (3.12.4)
    Requirement already satisfied: termcolor in /usr/local/lib/python3.7/dist-packages (from tfds-nightly==3.1.0.dev202006230105) (1.1.0)
    Requirement already satisfied: wrapt in /usr/local/lib/python3.7/dist-packages (from tfds-nightly==3.1.0.dev202006230105) (1.12.1)
    Requirement already satisfied: tensorflow-metadata in /usr/local/lib/python3.7/dist-packages (from tfds-nightly==3.1.0.dev202006230105) (0.28.0)
    Requirement already satisfied: requests>=2.19.0 in /usr/local/lib/python3.7/dist-packages (from tfds-nightly==3.1.0.dev202006230105) (2.23.0)
    Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from tfds-nightly==3.1.0.dev202006230105) (4.41.1)
    Requirement already satisfied: absl-py in /usr/local/lib/python3.7/dist-packages (from tfds-nightly==3.1.0.dev202006230105) (0.10.0)
    Requirement already satisfied: dill in /usr/local/lib/python3.7/dist-packages (from tfds-nightly==3.1.0.dev202006230105) (0.3.3)
    Requirement already satisfied: attrs>=18.1.0 in /usr/local/lib/python3.7/dist-packages (from tfds-nightly==3.1.0.dev202006230105) (20.3.0)
    Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from tfds-nightly==3.1.0.dev202006230105) (1.19.5)
    Requirement already satisfied: future in /usr/local/lib/python3.7/dist-packages (from tfds-nightly==3.1.0.dev202006230105) (0.16.0)
    Requirement already satisfied: promise in /usr/local/lib/python3.7/dist-packages (from tfds-nightly==3.1.0.dev202006230105) (2.3)
    Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from tfds-nightly==3.1.0.dev202006230105) (1.15.0)
    Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from protobuf>=3.6.1->tfds-nightly==3.1.0.dev202006230105) (54.1.2)
    Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow-metadata->tfds-nightly==3.1.0.dev202006230105) (1.53.0)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19.0->tfds-nightly==3.1.0.dev202006230105) (2020.12.5)
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19.0->tfds-nightly==3.1.0.dev202006230105) (3.0.4)
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19.0->tfds-nightly==3.1.0.dev202006230105) (2.10)
    Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19.0->tfds-nightly==3.1.0.dev202006230105) (1.24.3)
    Installing collected packages: tfds-nightly
      Found existing installation: tfds-nightly 4.2.0.dev202103230107
        Uninstalling tfds-nightly-4.2.0.dev202103230107:
          Successfully uninstalled tfds-nightly-4.2.0.dev202103230107
    Successfully installed tfds-nightly-3.1.0.dev202006230105

```
!gsutil cp -r gs://pegasus_ckpt/aeslc/ ckpt
```

    Copying gs://pegasus_ckpt/aeslc/model.ckpt-32000.data-00000-of-00001...
    Copying gs://pegasus_ckpt/aeslc/model.ckpt-32000.index...
    Copying gs://pegasus_ckpt/aeslc/model.ckpt-32000.meta...
    \ [3 files][  2.1 GiB/  2.1 GiB]   67.4 MiB/s
    Operation completed over 3 objects/2.1 GiB.

```
!gsutil cp -r gs://pegasus_ckpt/c4.unigram.newline.10pct.96000.model/ ckpt
```

    Copying gs://pegasus_ckpt/c4.unigram.newline.10pct.96000.model...
    / [1 files][  1.8 MiB/  1.8 MiB]
    Operation completed over 1 objects/1.8 MiB.

```
!gsutil cp -r gs://pegasus_ckpt/c4.unigram.newline.10pct.96000.vocab/ ckpt
```

    Copying gs://pegasus_ckpt/c4.unigram.newline.10pct.96000.vocab...
    / [1 files][  1.7 MiB/  1.7 MiB]
    Operation completed over 1 objects/1.7 MiB.

```
!gsutil cp -r gs://pegasus_ckpt/checkpoint/ ckpt
```

    Copying gs://pegasus_ckpt/checkpoint...
    / [1 files][   93.0 B/   93.0 B]
    Operation completed over 1 objects/93.0 B.

```
!gsutil cp -r gs://pegasus_ckpt/cnn_dailymail/ ckpt
```

    Copying gs://pegasus_ckpt/cnn_dailymail/model.ckpt-210000.data-00000-of-00001...
    Copying gs://pegasus_ckpt/cnn_dailymail/model.ckpt-210000.index...
    Copying gs://pegasus_ckpt/cnn_dailymail/model.ckpt-210000.meta...
    / [3 files][  2.1 GiB/  2.1 GiB]   39.4 MiB/s
    Operation completed over 3 objects/2.1 GiB.

```
!gsutil cp -r gs://pegasus_ckpt/model.ckpt-1500000.data-00000-of-00001/ ckpt
```

    Copying gs://pegasus_ckpt/model.ckpt-1500000.data-00000-of-00001...
    \ [1 files][  2.1 GiB/  2.1 GiB]   72.5 MiB/s
    Operation completed over 1 objects/2.1 GiB.

```
!gsutil cp -r gs://pegasus_ckpt/model.ckpt-1500000.index/ ckpt
```

    Copying gs://pegasus_ckpt/model.ckpt-1500000.index...
    / [1 files][ 16.7 KiB/ 16.7 KiB]
    Operation completed over 1 objects/16.7 KiB.

```
!gsutil cp -r gs://pegasus_ckpt/model.ckpt-1500000.meta/ ckpt
```

    Copying gs://pegasus_ckpt/model.ckpt-1500000.meta...
    | [1 files][220.4 MiB/220.4 MiB]
    Operation completed over 1 objects/220.4 MiB.

```
!python3 pegasus/bin/train.py --params=aeslc_transformer \
--param_overrides=vocab_filename=ckpt/c4.unigram.newline.10pct.96000.model \
--train_init_checkpoint=ckpt/model.ckpt-1500000 \
--model_dir=ckpt/aeslc
```

    WARNING:tensorflow:From pegasus/bin/train.py:94: The name tf.app.run is deprecated. Please use tf.compat.v1.app.run instead.

    WARNING:tensorflow:From /content/pegasus/ops/public_parsing_ops.py:93: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.

    W0323 09:42:42.337186 140015849629568 module_wrapper.py:139] From /content/pegasus/ops/public_parsing_ops.py:93: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.

    WARNING:tensorflow:From /content/pegasus/params/estimator_utils.py:49: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

    W0323 09:42:42.475074 140015849629568 module_wrapper.py:139] From /content/pegasus/params/estimator_utils.py:49: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

    WARNING:tensorflow:Estimator's model_fn (<function _estimator_model_fn.<locals>.model_fn at 0x7f577ef88200>) includes params argument, but params are not passed to Estimator.
    W0323 09:42:42.475606 140015849629568 estimator.py:1994] Estimator's model_fn (<function _estimator_model_fn.<locals>.model_fn at 0x7f577ef88200>) includes params argument, but params are not passed to Estimator.
    INFO:tensorflow:Using config: {'_model_dir': 'ckpt/aeslc', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true
    , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f577ef7be50>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_tpu_config': TPUConfig(iterations_per_loop=1000, num_shards=None, num_cores_per_replica=None, per_host_input_for_training=2, tpu_job_name=None, initial_infeed_sleep_secs=None, input_partition_dims=None, eval_training_input_configuration=2, experimental_host_call_every_n_steps=1), '_cluster': None}
    I0323 09:42:42.477227 140015849629568 estimator.py:212] Using config: {'_model_dir': 'ckpt/aeslc', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true
    , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f577ef7be50>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_tpu_config': TPUConfig(iterations_per_loop=1000, num_shards=None, num_cores_per_replica=None, per_host_input_for_training=2, tpu_job_name=None, initial_infeed_sleep_secs=None, input_partition_dims=None, eval_training_input_configuration=2, experimental_host_call_every_n_steps=1), '_cluster': None}
    INFO:tensorflow:_TPUContext: eval_on_tpu True
    I0323 09:42:42.477473 140015849629568 tpu_context.py:220] _TPUContext: eval_on_tpu True
    WARNING:tensorflow:eval_on_tpu ignored because use_tpu is False.
    W0323 09:42:42.477704 140015849629568 tpu_context.py:222] eval_on_tpu ignored because use_tpu is False.
    WARNING:tensorflow:From /tensorflow-1.15.2/python3.7/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
    Instructions for updating:
    If using Keras pass *_constraint arguments to layers.
    W0323 09:42:42.488204 140015849629568 deprecation.py:506] From /tensorflow-1.15.2/python3.7/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
    Instructions for updating:
    If using Keras pass *_constraint arguments to layers.
    WARNING:tensorflow:From /tensorflow-1.15.2/python3.7/tensorflow_core/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
    Instructions for updating:
    Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
    W0323 09:42:42.488598 140015849629568 deprecation.py:323] From /tensorflow-1.15.2/python3.7/tensorflow_core/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
    Instructions for updating:
    Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
    I0323 09:42:42.545305 140015849629568 dataset_info.py:427] Load pre-computed DatasetInfo (eg: splits, num examples,...) from GCS: aeslc/1.0.0
    I0323 09:42:42.665326 140015849629568 dataset_info.py:358] Load dataset info from /tmp/tmp2u64csbatfds
    I0323 09:42:42.666739 140015849629568 dataset_info.py:398] Field info.description from disk and from code do not match. Keeping the one from code.
    I0323 09:42:42.666898 140015849629568 dataset_info.py:398] Field info.citation from disk and from code do not match. Keeping the one from code.
    I0323 09:42:42.667275 140015849629568 dataset_builder.py:345] Generating dataset aeslc (/root/tensorflow_datasets/aeslc/1.0.0)
    Downloading and preparing dataset aeslc/1.0.0 (download: 11.10 MiB, generated: Unknown size, total: 11.10 MiB) to /root/tensorflow_datasets/aeslc/1.0.0...
    Dl Completed...: 0 url [00:00, ? url/s]
    Dl Size...: 0 MiB [00:00, ? MiB/s]

    Extraction completed...: 0 file [00:00, ? file/s]I0323 09:42:42.776161 140015849629568 download_manager.py:477] Downloading https://github.com/ryanzhumich/AESLC/archive/master.zip into /root/tensorflow_datasets/downloads/ryanzhumich_AESLC_archive_masterACSpoxw627Ay4UrkswMeyz6RrOey8kKfkhEM4VySJWU.zip.tmp.07002748dac047c8a0dcc3fd9a6b7613...
    Dl Completed...:   0% 0/1 [00:00<?, ? url/s]
    Dl Size...: 0 MiB [00:00, ? MiB/s]

    Dl Completed...:   0% 0/1 [00:00<?, ? url/s]
    Dl Size...: 0 MiB [00:00, ? MiB/s]

    Extraction completed...: 0 file [00:00, ? file/s]
    Dl Completed...:   0% 0/1 [00:00<?, ? url/s]
    Dl Size...: 1 MiB [00:00,  3.11 MiB/s]

    Extraction completed...: 0 file [00:00, ? file/s]
    Dl Completed...:   0% 0/1 [00:00<?, ? url/s]
    Dl Size...: 2 MiB [00:00,  3.82 MiB/s]

    Extraction completed...: 0 file [00:00, ? file/s]
    Dl Completed...:   0% 0/1 [00:00<?, ? url/s]
    Dl Size...: 3 MiB [00:00,  4.59 MiB/s]

    Extraction completed...: 0 file [00:00, ? file/s]
    Dl Completed...:   0% 0/1 [00:00<?, ? url/s]
    Dl Size...: 4 MiB [00:00,  5.39 MiB/s]

    Extraction completed...: 0 file [00:00, ? file/s]
    Dl Completed...:   0% 0/1 [00:00<?, ? url/s]
    Dl Size...: 5 MiB [00:00,  6.12 MiB/s]

    Extraction completed...: 0 file [00:00, ? file/s]
    Dl Completed...:   0% 0/1 [00:00<?, ? url/s]
    Dl Size...: 6 MiB [00:00,  6.62 MiB/s]

    Extraction completed...: 0 file [00:00, ? file/s]
    Dl Completed...:   0% 0/1 [00:01<?, ? url/s]
    Dl Size...: 7 MiB [00:01,  7.05 MiB/s]

    Extraction completed...: 0 file [00:01, ? file/s]
    Dl Completed...:   0% 0/1 [00:01<?, ? url/s]
    Dl Size...: 8 MiB [00:01,  7.45 MiB/s]

    Extraction completed...: 0 file [00:01, ? file/s]
    Dl Completed...:   0% 0/1 [00:01<?, ? url/s]
    Dl Size...: 9 MiB [00:01,  7.72 MiB/s]

    Dl Completed...:   0% 0/1 [00:01<?, ? url/s]
    Dl Size...: 10 MiB [00:01,  7.72 MiB/s]

    Dl Completed...:   0% 0/1 [00:01<?, ? url/s]
    Dl Size...: 11 MiB [00:01,  7.72 MiB/s]

    Dl Completed...: 100% 1/1 [00:01<00:00,  1.30s/ url]
    Dl Size...: 11 MiB [00:01,  7.72 MiB/s]

    Dl Completed...: 100% 1/1 [00:01<00:00,  1.30s/ url]
    Dl Size...: 11 MiB [00:01,  7.72 MiB/s]

    Extraction completed...:   0% 0/1 [00:01<?, ? file/s]

    Dl Completed...: 100% 1/1 [00:07<00:00,  1.30s/ url]
    Dl Size...: 11 MiB [00:07,  7.72 MiB/s]

    Extraction completed...: 100% 1/1 [00:07<00:00,  7.82s/ file]
    Extraction completed...: 100% 1/1 [00:07<00:00,  7.83s/ file]

    Dl Size...: 11 MiB [00:07,  1.41 MiB/s]

    Dl Completed...: 100% 1/1 [00:07<00:00,  7.83s/ url]
    I0323 09:42:50.602668 140015849629568 dataset_builder.py:946] Generating split train
    Shuffling and writing examples to /root/tensorflow_datasets/aeslc/1.0.0.incompleteW0JYQ2/aeslc-train.tfrecord
      0% 0/14436 [00:00<?, ? examples/s]I0323 09:42:56.521217 140015849629568 tfrecords_writer.py:230] Done writing /root/tensorflow_datasets/aeslc/1.0.0.incompleteW0JYQ2/aeslc-train.tfrecord. Shard lengths: [14436]
    I0323 09:42:56.525305 140015849629568 dataset_builder.py:946] Generating split validation
    Shuffling and writing examples to /root/tensorflow_datasets/aeslc/1.0.0.incompleteW0JYQ2/aeslc-validation.tfrecord
      0% 0/1960 [00:00<?, ? examples/s]I0323 09:42:57.411416 140015849629568 tfrecords_writer.py:230] Done writing /root/tensorflow_datasets/aeslc/1.0.0.incompleteW0JYQ2/aeslc-validation.tfrecord. Shard lengths: [1960]
    I0323 09:42:57.412200 140015849629568 dataset_builder.py:946] Generating split test
    Shuffling and writing examples to /root/tensorflow_datasets/aeslc/1.0.0.incompleteW0JYQ2/aeslc-test.tfrecord
      0% 0/1906 [00:00<?, ? examples/s]I0323 09:42:58.161666 140015849629568 tfrecords_writer.py:230] Done writing /root/tensorflow_datasets/aeslc/1.0.0.incompleteW0JYQ2/aeslc-test.tfrecord. Shard lengths: [1906]
    I0323 09:42:58.162671 140015849629568 dataset_builder.py:400] Skipping computing stats for mode ComputeStatsMode.AUTO.
    Dataset aeslc downloaded and prepared to /root/tensorflow_datasets/aeslc/1.0.0. Subsequent calls will reuse this data.
    I0323 09:42:58.163849 140015849629568 dataset_builder.py:499] Constructing tf.data.Dataset for split train, from /root/tensorflow_datasets/aeslc/1.0.0
    I0323 09:42:58.400106 140015849629568 datasets.py:215] Number of examples for config aeslc train is 14436
    2021-03-23 09:42:59.067226: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
    2021-03-23 09:42:59.140056: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
    2021-03-23 09:42:59.140655: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties:
    name: Tesla T4 major: 7 minor: 5 memoryClockRate(GHz): 1.59
    pciBusID: 0000:00:04.0
    2021-03-23 09:42:59.228943: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
    2021-03-23 09:42:59.421176: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
    2021-03-23 09:42:59.563578: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
    2021-03-23 09:42:59.595956: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
    2021-03-23 09:42:59.854564: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
    2021-03-23 09:42:59.868934: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
    2021-03-23 09:43:00.375740: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
    2021-03-23 09:43:00.375974: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
    2021-03-23 09:43:00.376682: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
    2021-03-23 09:43:00.377206: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0
    WARNING:tensorflow:From /tensorflow-1.15.2/python3.7/tensorflow_core/python/ops/ragged/ragged_tensor.py:1586: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
    Instructions for updating:
    Use tf.where in 2.0, which has the same broadcast rule as np.where
    W0323 09:43:00.471799 140015849629568 deprecation.py:323] From /tensorflow-1.15.2/python3.7/tensorflow_core/python/ops/ragged/ragged_tensor.py:1586: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
    Instructions for updating:
    Use tf.where in 2.0, which has the same broadcast rule as np.where
    INFO:tensorflow:Calling model_fn.
    I0323 09:43:00.715790 140015849629568 estimator.py:1148] Calling model_fn.
    INFO:tensorflow:Running train on CPU
    I0323 09:43:00.716070 140015849629568 tpu_estimator.py:3124] Running train on CPU
    WARNING:tensorflow:From /content/pegasus/params/estimator_utils.py:78: The name tf.get_variable_scope is deprecated. Please use tf.compat.v1.get_variable_scope instead.

    W0323 09:43:00.716448 140015849629568 module_wrapper.py:139] From /content/pegasus/params/estimator_utils.py:78: The name tf.get_variable_scope is deprecated. Please use tf.compat.v1.get_variable_scope instead.

    WARNING:tensorflow:From /content/pegasus/layers/attention.py:41: The name tf.layers.Dense is deprecated. Please use tf.compat.v1.layers.Dense instead.

    W0323 09:43:00.716814 140015849629568 module_wrapper.py:139] From /content/pegasus/layers/attention.py:41: The name tf.layers.Dense is deprecated. Please use tf.compat.v1.layers.Dense instead.

    WARNING:tensorflow:From /content/pegasus/layers/embedding.py:57: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

    W0323 09:43:00.756722 140015849629568 module_wrapper.py:139] From /content/pegasus/layers/embedding.py:57: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

    WARNING:tensorflow:From /content/pegasus/layers/embedding.py:57: The name tf.AUTO_REUSE is deprecated. Please use tf.compat.v1.AUTO_REUSE instead.

    W0323 09:43:00.756910 140015849629568 module_wrapper.py:139] From /content/pegasus/layers/embedding.py:57: The name tf.AUTO_REUSE is deprecated. Please use tf.compat.v1.AUTO_REUSE instead.

    WARNING:tensorflow:From /content/pegasus/layers/embedding.py:61: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.

    W0323 09:43:00.757146 140015849629568 module_wrapper.py:139] From /content/pegasus/layers/embedding.py:61: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.

    WARNING:tensorflow:From /content/pegasus/layers/embedding.py:64: calling RandomNormal.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
    Instructions for updating:
    Call initializer instance with the dtype argument instead of passing it to the constructor
    W0323 09:43:00.757384 140015849629568 deprecation.py:506] From /content/pegasus/layers/embedding.py:64: calling RandomNormal.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
    Instructions for updating:
    Call initializer instance with the dtype argument instead of passing it to the constructor
    WARNING:tensorflow:From /content/pegasus/layers/attention.py:106: The name tf.matrix_band_part is deprecated. Please use tf.linalg.band_part instead.

    W0323 09:43:03.239101 140015849629568 module_wrapper.py:139] From /content/pegasus/layers/attention.py:106: The name tf.matrix_band_part is deprecated. Please use tf.linalg.band_part instead.

    WARNING:tensorflow:From /content/pegasus/models/transformer.py:108: The name tf.losses.softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.softmax_cross_entropy instead.

    W0323 09:43:07.268426 140015849629568 module_wrapper.py:139] From /content/pegasus/models/transformer.py:108: The name tf.losses.softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.softmax_cross_entropy instead.

    WARNING:tensorflow:From /content/pegasus/params/estimator_utils.py:113: The name tf.train.get_global_step is deprecated. Please use tf.compat.v1.train.get_global_step instead.

    W0323 09:43:07.311040 140015849629568 module_wrapper.py:139] From /content/pegasus/params/estimator_utils.py:113: The name tf.train.get_global_step is deprecated. Please use tf.compat.v1.train.get_global_step instead.

    WARNING:tensorflow:From /content/pegasus/params/estimator_utils.py:114: The name tf.rsqrt is deprecated. Please use tf.math.rsqrt instead.

    W0323 09:43:07.311391 140015849629568 module_wrapper.py:139] From /content/pegasus/params/estimator_utils.py:114: The name tf.rsqrt is deprecated. Please use tf.math.rsqrt instead.

    WARNING:tensorflow:From /content/pegasus/params/estimator_utils.py:115: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
    Instructions for updating:
    Use `tf.cast` instead.
    W0323 09:43:07.311611 140015849629568 deprecation.py:323] From /content/pegasus/params/estimator_utils.py:115: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
    Instructions for updating:
    Use `tf.cast` instead.
    WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/tensor2tensor/utils/adafactor.py:315: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

    W0323 09:43:07.322172 140015849629568 module_wrapper.py:139] From /usr/local/lib/python3.7/dist-packages/tensor2tensor/utils/adafactor.py:315: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

    WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/tensor2tensor/utils/quantization.py:147: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
    Instructions for updating:
    Use `tf.cast` instead.
    W0323 09:43:07.326635 140015849629568 deprecation.py:323] From /usr/local/lib/python3.7/dist-packages/tensor2tensor/utils/quantization.py:147: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
    Instructions for updating:
    Use `tf.cast` instead.
    WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/tensor2tensor/utils/quantization.py:156: The name tf.mod is deprecated. Please use tf.math.mod instead.

    W0323 09:43:07.328975 140015849629568 module_wrapper.py:139] From /usr/local/lib/python3.7/dist-packages/tensor2tensor/utils/quantization.py:156: The name tf.mod is deprecated. Please use tf.math.mod instead.

    WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/tensor2tensor/utils/adafactor.py:244: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.

    W0323 09:43:19.015337 140015849629568 module_wrapper.py:139] From /usr/local/lib/python3.7/dist-packages/tensor2tensor/utils/adafactor.py:244: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.

    WARNING:tensorflow:From /content/pegasus/params/estimator_utils.py:189: The name tf.trainable_variables is deprecated. Please use tf.compat.v1.trainable_variables instead.

    W0323 09:43:31.032931 140015849629568 module_wrapper.py:139] From /content/pegasus/params/estimator_utils.py:189: The name tf.trainable_variables is deprecated. Please use tf.compat.v1.trainable_variables instead.

    WARNING:tensorflow:From /content/pegasus/params/estimator_utils.py:205: The name tf.train.init_from_checkpoint is deprecated. Please use tf.compat.v1.train.init_from_checkpoint instead.

    W0323 09:43:31.038644 140015849629568 module_wrapper.py:139] From /content/pegasus/params/estimator_utils.py:205: The name tf.train.init_from_checkpoint is deprecated. Please use tf.compat.v1.train.init_from_checkpoint instead.

    I0323 09:43:32.218361 140015849629568 estimator_utils.py:207] **** Trainable Variables ****
    I0323 09:43:32.218551 140015849629568 estimator_utils.py:212]   name = embeddings/weights:0, shape = (96103, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.218656 140015849629568 estimator_utils.py:212]   name = encoder/layer_0/self_attention/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.218749 140015849629568 estimator_utils.py:212]   name = encoder/layer_0/self_attention/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.218822 140015849629568 estimator_utils.py:212]   name = encoder/layer_0/self_attention/q_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.218923 140015849629568 estimator_utils.py:212]   name = encoder/layer_0/self_attention/k_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.219001 140015849629568 estimator_utils.py:212]   name = encoder/layer_0/self_attention/v_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.219077 140015849629568 estimator_utils.py:212]   name = encoder/layer_0/self_attention/output_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.219160 140015849629568 estimator_utils.py:212]   name = encoder/layer_0/ffn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.219234 140015849629568 estimator_utils.py:212]   name = encoder/layer_0/ffn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.219309 140015849629568 estimator_utils.py:212]   name = encoder/layer_0/ffn/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
    I0323 09:43:32.219383 140015849629568 estimator_utils.py:212]   name = encoder/layer_0/ffn/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*
    I0323 09:43:32.219456 140015849629568 estimator_utils.py:212]   name = encoder/layer_0/ffn/dense_1/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.219530 140015849629568 estimator_utils.py:212]   name = encoder/layer_0/ffn/dense_1/bias:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.219600 140015849629568 estimator_utils.py:212]   name = encoder/layer_1/self_attention/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.219668 140015849629568 estimator_utils.py:212]   name = encoder/layer_1/self_attention/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.219738 140015849629568 estimator_utils.py:212]   name = encoder/layer_1/self_attention/q_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.219811 140015849629568 estimator_utils.py:212]   name = encoder/layer_1/self_attention/k_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.219896 140015849629568 estimator_utils.py:212]   name = encoder/layer_1/self_attention/v_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.219971 140015849629568 estimator_utils.py:212]   name = encoder/layer_1/self_attention/output_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.220046 140015849629568 estimator_utils.py:212]   name = encoder/layer_1/ffn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.220118 140015849629568 estimator_utils.py:212]   name = encoder/layer_1/ffn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.220225 140015849629568 estimator_utils.py:212]   name = encoder/layer_1/ffn/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
    I0323 09:43:32.220314 140015849629568 estimator_utils.py:212]   name = encoder/layer_1/ffn/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*
    I0323 09:43:32.220398 140015849629568 estimator_utils.py:212]   name = encoder/layer_1/ffn/dense_1/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.220494 140015849629568 estimator_utils.py:212]   name = encoder/layer_1/ffn/dense_1/bias:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.220577 140015849629568 estimator_utils.py:212]   name = encoder/layer_2/self_attention/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.220659 140015849629568 estimator_utils.py:212]   name = encoder/layer_2/self_attention/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.220746 140015849629568 estimator_utils.py:212]   name = encoder/layer_2/self_attention/q_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.220844 140015849629568 estimator_utils.py:212]   name = encoder/layer_2/self_attention/k_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.220929 140015849629568 estimator_utils.py:212]   name = encoder/layer_2/self_attention/v_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.221016 140015849629568 estimator_utils.py:212]   name = encoder/layer_2/self_attention/output_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.221103 140015849629568 estimator_utils.py:212]   name = encoder/layer_2/ffn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.221193 140015849629568 estimator_utils.py:212]   name = encoder/layer_2/ffn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.221284 140015849629568 estimator_utils.py:212]   name = encoder/layer_2/ffn/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
    I0323 09:43:32.221368 140015849629568 estimator_utils.py:212]   name = encoder/layer_2/ffn/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*
    I0323 09:43:32.221437 140015849629568 estimator_utils.py:212]   name = encoder/layer_2/ffn/dense_1/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.221509 140015849629568 estimator_utils.py:212]   name = encoder/layer_2/ffn/dense_1/bias:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.221582 140015849629568 estimator_utils.py:212]   name = encoder/layer_3/self_attention/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.221654 140015849629568 estimator_utils.py:212]   name = encoder/layer_3/self_attention/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.221723 140015849629568 estimator_utils.py:212]   name = encoder/layer_3/self_attention/q_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.221794 140015849629568 estimator_utils.py:212]   name = encoder/layer_3/self_attention/k_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.221877 140015849629568 estimator_utils.py:212]   name = encoder/layer_3/self_attention/v_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.221952 140015849629568 estimator_utils.py:212]   name = encoder/layer_3/self_attention/output_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.222025 140015849629568 estimator_utils.py:212]   name = encoder/layer_3/ffn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.222094 140015849629568 estimator_utils.py:212]   name = encoder/layer_3/ffn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.222170 140015849629568 estimator_utils.py:212]   name = encoder/layer_3/ffn/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
    I0323 09:43:32.222242 140015849629568 estimator_utils.py:212]   name = encoder/layer_3/ffn/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*
    I0323 09:43:32.222312 140015849629568 estimator_utils.py:212]   name = encoder/layer_3/ffn/dense_1/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.222383 140015849629568 estimator_utils.py:212]   name = encoder/layer_3/ffn/dense_1/bias:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.222452 140015849629568 estimator_utils.py:212]   name = encoder/layer_4/self_attention/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.222524 140015849629568 estimator_utils.py:212]   name = encoder/layer_4/self_attention/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.222594 140015849629568 estimator_utils.py:212]   name = encoder/layer_4/self_attention/q_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.222665 140015849629568 estimator_utils.py:212]   name = encoder/layer_4/self_attention/k_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.222738 140015849629568 estimator_utils.py:212]   name = encoder/layer_4/self_attention/v_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.222810 140015849629568 estimator_utils.py:212]   name = encoder/layer_4/self_attention/output_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.222899 140015849629568 estimator_utils.py:212]   name = encoder/layer_4/ffn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.222968 140015849629568 estimator_utils.py:212]   name = encoder/layer_4/ffn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.223040 140015849629568 estimator_utils.py:212]   name = encoder/layer_4/ffn/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
    I0323 09:43:32.223118 140015849629568 estimator_utils.py:212]   name = encoder/layer_4/ffn/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*
    I0323 09:43:32.223194 140015849629568 estimator_utils.py:212]   name = encoder/layer_4/ffn/dense_1/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.223266 140015849629568 estimator_utils.py:212]   name = encoder/layer_4/ffn/dense_1/bias:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.223336 140015849629568 estimator_utils.py:212]   name = encoder/layer_5/self_attention/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.223404 140015849629568 estimator_utils.py:212]   name = encoder/layer_5/self_attention/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.223472 140015849629568 estimator_utils.py:212]   name = encoder/layer_5/self_attention/q_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.223543 140015849629568 estimator_utils.py:212]   name = encoder/layer_5/self_attention/k_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.223614 140015849629568 estimator_utils.py:212]   name = encoder/layer_5/self_attention/v_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.223685 140015849629568 estimator_utils.py:212]   name = encoder/layer_5/self_attention/output_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.223757 140015849629568 estimator_utils.py:212]   name = encoder/layer_5/ffn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.223837 140015849629568 estimator_utils.py:212]   name = encoder/layer_5/ffn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.223910 140015849629568 estimator_utils.py:212]   name = encoder/layer_5/ffn/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
    I0323 09:43:32.223980 140015849629568 estimator_utils.py:212]   name = encoder/layer_5/ffn/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*
    I0323 09:43:32.224048 140015849629568 estimator_utils.py:212]   name = encoder/layer_5/ffn/dense_1/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.224118 140015849629568 estimator_utils.py:212]   name = encoder/layer_5/ffn/dense_1/bias:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.224192 140015849629568 estimator_utils.py:212]   name = encoder/layer_6/self_attention/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.224261 140015849629568 estimator_utils.py:212]   name = encoder/layer_6/self_attention/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.224330 140015849629568 estimator_utils.py:212]   name = encoder/layer_6/self_attention/q_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.224400 140015849629568 estimator_utils.py:212]   name = encoder/layer_6/self_attention/k_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.224472 140015849629568 estimator_utils.py:212]   name = encoder/layer_6/self_attention/v_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.224543 140015849629568 estimator_utils.py:212]   name = encoder/layer_6/self_attention/output_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.224614 140015849629568 estimator_utils.py:212]   name = encoder/layer_6/ffn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.224684 140015849629568 estimator_utils.py:212]   name = encoder/layer_6/ffn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.224752 140015849629568 estimator_utils.py:212]   name = encoder/layer_6/ffn/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
    I0323 09:43:32.224822 140015849629568 estimator_utils.py:212]   name = encoder/layer_6/ffn/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*
    I0323 09:43:32.224906 140015849629568 estimator_utils.py:212]   name = encoder/layer_6/ffn/dense_1/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.224975 140015849629568 estimator_utils.py:212]   name = encoder/layer_6/ffn/dense_1/bias:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.225044 140015849629568 estimator_utils.py:212]   name = encoder/layer_7/self_attention/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.225114 140015849629568 estimator_utils.py:212]   name = encoder/layer_7/self_attention/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.225188 140015849629568 estimator_utils.py:212]   name = encoder/layer_7/self_attention/q_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.225260 140015849629568 estimator_utils.py:212]   name = encoder/layer_7/self_attention/k_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.225331 140015849629568 estimator_utils.py:212]   name = encoder/layer_7/self_attention/v_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.225402 140015849629568 estimator_utils.py:212]   name = encoder/layer_7/self_attention/output_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.225478 140015849629568 estimator_utils.py:212]   name = encoder/layer_7/ffn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.225547 140015849629568 estimator_utils.py:212]   name = encoder/layer_7/ffn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.225617 140015849629568 estimator_utils.py:212]   name = encoder/layer_7/ffn/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
    I0323 09:43:32.225689 140015849629568 estimator_utils.py:212]   name = encoder/layer_7/ffn/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*
    I0323 09:43:32.225757 140015849629568 estimator_utils.py:212]   name = encoder/layer_7/ffn/dense_1/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.225837 140015849629568 estimator_utils.py:212]   name = encoder/layer_7/ffn/dense_1/bias:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.225910 140015849629568 estimator_utils.py:212]   name = encoder/layer_8/self_attention/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.225978 140015849629568 estimator_utils.py:212]   name = encoder/layer_8/self_attention/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.226045 140015849629568 estimator_utils.py:212]   name = encoder/layer_8/self_attention/q_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.226116 140015849629568 estimator_utils.py:212]   name = encoder/layer_8/self_attention/k_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.226195 140015849629568 estimator_utils.py:212]   name = encoder/layer_8/self_attention/v_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.226269 140015849629568 estimator_utils.py:212]   name = encoder/layer_8/self_attention/output_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.226341 140015849629568 estimator_utils.py:212]   name = encoder/layer_8/ffn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.226409 140015849629568 estimator_utils.py:212]   name = encoder/layer_8/ffn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.226477 140015849629568 estimator_utils.py:212]   name = encoder/layer_8/ffn/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
    I0323 09:43:32.226546 140015849629568 estimator_utils.py:212]   name = encoder/layer_8/ffn/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*
    I0323 09:43:32.226614 140015849629568 estimator_utils.py:212]   name = encoder/layer_8/ffn/dense_1/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.226688 140015849629568 estimator_utils.py:212]   name = encoder/layer_8/ffn/dense_1/bias:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.226755 140015849629568 estimator_utils.py:212]   name = encoder/layer_9/self_attention/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.226824 140015849629568 estimator_utils.py:212]   name = encoder/layer_9/self_attention/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.226906 140015849629568 estimator_utils.py:212]   name = encoder/layer_9/self_attention/q_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.226976 140015849629568 estimator_utils.py:212]   name = encoder/layer_9/self_attention/k_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.227048 140015849629568 estimator_utils.py:212]   name = encoder/layer_9/self_attention/v_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.227120 140015849629568 estimator_utils.py:212]   name = encoder/layer_9/self_attention/output_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.227197 140015849629568 estimator_utils.py:212]   name = encoder/layer_9/ffn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.227270 140015849629568 estimator_utils.py:212]   name = encoder/layer_9/ffn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.227340 140015849629568 estimator_utils.py:212]   name = encoder/layer_9/ffn/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
    I0323 09:43:32.227413 140015849629568 estimator_utils.py:212]   name = encoder/layer_9/ffn/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*
    I0323 09:43:32.227483 140015849629568 estimator_utils.py:212]   name = encoder/layer_9/ffn/dense_1/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.231266 140015849629568 estimator_utils.py:212]   name = encoder/layer_9/ffn/dense_1/bias:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.231428 140015849629568 estimator_utils.py:212]   name = encoder/layer_10/self_attention/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.231549 140015849629568 estimator_utils.py:212]   name = encoder/layer_10/self_attention/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.231663 140015849629568 estimator_utils.py:212]   name = encoder/layer_10/self_attention/q_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.231776 140015849629568 estimator_utils.py:212]   name = encoder/layer_10/self_attention/k_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.231908 140015849629568 estimator_utils.py:212]   name = encoder/layer_10/self_attention/v_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.232019 140015849629568 estimator_utils.py:212]   name = encoder/layer_10/self_attention/output_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.232125 140015849629568 estimator_utils.py:212]   name = encoder/layer_10/ffn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.232243 140015849629568 estimator_utils.py:212]   name = encoder/layer_10/ffn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.232332 140015849629568 estimator_utils.py:212]   name = encoder/layer_10/ffn/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
    I0323 09:43:32.232419 140015849629568 estimator_utils.py:212]   name = encoder/layer_10/ffn/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*
    I0323 09:43:32.232501 140015849629568 estimator_utils.py:212]   name = encoder/layer_10/ffn/dense_1/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.232587 140015849629568 estimator_utils.py:212]   name = encoder/layer_10/ffn/dense_1/bias:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.232674 140015849629568 estimator_utils.py:212]   name = encoder/layer_11/self_attention/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.232764 140015849629568 estimator_utils.py:212]   name = encoder/layer_11/self_attention/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.232866 140015849629568 estimator_utils.py:212]   name = encoder/layer_11/self_attention/q_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.232968 140015849629568 estimator_utils.py:212]   name = encoder/layer_11/self_attention/k_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.233064 140015849629568 estimator_utils.py:212]   name = encoder/layer_11/self_attention/v_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.233180 140015849629568 estimator_utils.py:212]   name = encoder/layer_11/self_attention/output_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.233288 140015849629568 estimator_utils.py:212]   name = encoder/layer_11/ffn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.233381 140015849629568 estimator_utils.py:212]   name = encoder/layer_11/ffn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.233472 140015849629568 estimator_utils.py:212]   name = encoder/layer_11/ffn/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
    I0323 09:43:32.233570 140015849629568 estimator_utils.py:212]   name = encoder/layer_11/ffn/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*
    I0323 09:43:32.233664 140015849629568 estimator_utils.py:212]   name = encoder/layer_11/ffn/dense_1/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.233762 140015849629568 estimator_utils.py:212]   name = encoder/layer_11/ffn/dense_1/bias:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.233869 140015849629568 estimator_utils.py:212]   name = encoder/layer_12/self_attention/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.233970 140015849629568 estimator_utils.py:212]   name = encoder/layer_12/self_attention/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.234064 140015849629568 estimator_utils.py:212]   name = encoder/layer_12/self_attention/q_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.234170 140015849629568 estimator_utils.py:212]   name = encoder/layer_12/self_attention/k_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.234268 140015849629568 estimator_utils.py:212]   name = encoder/layer_12/self_attention/v_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.234343 140015849629568 estimator_utils.py:212]   name = encoder/layer_12/self_attention/output_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.234415 140015849629568 estimator_utils.py:212]   name = encoder/layer_12/ffn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.234487 140015849629568 estimator_utils.py:212]   name = encoder/layer_12/ffn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.234560 140015849629568 estimator_utils.py:212]   name = encoder/layer_12/ffn/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
    I0323 09:43:32.234634 140015849629568 estimator_utils.py:212]   name = encoder/layer_12/ffn/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*
    I0323 09:43:32.234707 140015849629568 estimator_utils.py:212]   name = encoder/layer_12/ffn/dense_1/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.234782 140015849629568 estimator_utils.py:212]   name = encoder/layer_12/ffn/dense_1/bias:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.234865 140015849629568 estimator_utils.py:212]   name = encoder/layer_13/self_attention/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.234935 140015849629568 estimator_utils.py:212]   name = encoder/layer_13/self_attention/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.235006 140015849629568 estimator_utils.py:212]   name = encoder/layer_13/self_attention/q_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.235079 140015849629568 estimator_utils.py:212]   name = encoder/layer_13/self_attention/k_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.235158 140015849629568 estimator_utils.py:212]   name = encoder/layer_13/self_attention/v_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.235232 140015849629568 estimator_utils.py:212]   name = encoder/layer_13/self_attention/output_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.235307 140015849629568 estimator_utils.py:212]   name = encoder/layer_13/ffn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.235378 140015849629568 estimator_utils.py:212]   name = encoder/layer_13/ffn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.235447 140015849629568 estimator_utils.py:212]   name = encoder/layer_13/ffn/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
    I0323 09:43:32.235520 140015849629568 estimator_utils.py:212]   name = encoder/layer_13/ffn/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*
    I0323 09:43:32.235591 140015849629568 estimator_utils.py:212]   name = encoder/layer_13/ffn/dense_1/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.235665 140015849629568 estimator_utils.py:212]   name = encoder/layer_13/ffn/dense_1/bias:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.235737 140015849629568 estimator_utils.py:212]   name = encoder/layer_14/self_attention/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.235808 140015849629568 estimator_utils.py:212]   name = encoder/layer_14/self_attention/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.235888 140015849629568 estimator_utils.py:212]   name = encoder/layer_14/self_attention/q_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.235963 140015849629568 estimator_utils.py:212]   name = encoder/layer_14/self_attention/k_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.236036 140015849629568 estimator_utils.py:212]   name = encoder/layer_14/self_attention/v_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.236109 140015849629568 estimator_utils.py:212]   name = encoder/layer_14/self_attention/output_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.236192 140015849629568 estimator_utils.py:212]   name = encoder/layer_14/ffn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.236264 140015849629568 estimator_utils.py:212]   name = encoder/layer_14/ffn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.236334 140015849629568 estimator_utils.py:212]   name = encoder/layer_14/ffn/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
    I0323 09:43:32.236406 140015849629568 estimator_utils.py:212]   name = encoder/layer_14/ffn/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*
    I0323 09:43:32.236476 140015849629568 estimator_utils.py:212]   name = encoder/layer_14/ffn/dense_1/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.236548 140015849629568 estimator_utils.py:212]   name = encoder/layer_14/ffn/dense_1/bias:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.236618 140015849629568 estimator_utils.py:212]   name = encoder/layer_15/self_attention/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.236689 140015849629568 estimator_utils.py:212]   name = encoder/layer_15/self_attention/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.236760 140015849629568 estimator_utils.py:212]   name = encoder/layer_15/self_attention/q_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.236842 140015849629568 estimator_utils.py:212]   name = encoder/layer_15/self_attention/k_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.236915 140015849629568 estimator_utils.py:212]   name = encoder/layer_15/self_attention/v_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.236990 140015849629568 estimator_utils.py:212]   name = encoder/layer_15/self_attention/output_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.237064 140015849629568 estimator_utils.py:212]   name = encoder/layer_15/ffn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.237134 140015849629568 estimator_utils.py:212]   name = encoder/layer_15/ffn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.237212 140015849629568 estimator_utils.py:212]   name = encoder/layer_15/ffn/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
    I0323 09:43:32.237285 140015849629568 estimator_utils.py:212]   name = encoder/layer_15/ffn/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*
    I0323 09:43:32.237355 140015849629568 estimator_utils.py:212]   name = encoder/layer_15/ffn/dense_1/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.237428 140015849629568 estimator_utils.py:212]   name = encoder/layer_15/ffn/dense_1/bias:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.237498 140015849629568 estimator_utils.py:212]   name = encoder/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.237576 140015849629568 estimator_utils.py:212]   name = encoder/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.237646 140015849629568 estimator_utils.py:212]   name = decoder/layer_0/self_attention/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.237719 140015849629568 estimator_utils.py:212]   name = decoder/layer_0/self_attention/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.237791 140015849629568 estimator_utils.py:212]   name = decoder/layer_0/self_attention/q_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.237872 140015849629568 estimator_utils.py:212]   name = decoder/layer_0/self_attention/k_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.237946 140015849629568 estimator_utils.py:212]   name = decoder/layer_0/self_attention/v_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.238020 140015849629568 estimator_utils.py:212]   name = decoder/layer_0/self_attention/output_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.238094 140015849629568 estimator_utils.py:212]   name = decoder/layer_0/memory_attention/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.238171 140015849629568 estimator_utils.py:212]   name = decoder/layer_0/memory_attention/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.238241 140015849629568 estimator_utils.py:212]   name = decoder/layer_0/memory_attention/q_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.238315 140015849629568 estimator_utils.py:212]   name = decoder/layer_0/memory_attention/k_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.238389 140015849629568 estimator_utils.py:212]   name = decoder/layer_0/memory_attention/v_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.238462 140015849629568 estimator_utils.py:212]   name = decoder/layer_0/memory_attention/output_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.238536 140015849629568 estimator_utils.py:212]   name = decoder/layer_0/ffn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.238606 140015849629568 estimator_utils.py:212]   name = decoder/layer_0/ffn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.238677 140015849629568 estimator_utils.py:212]   name = decoder/layer_0/ffn/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
    I0323 09:43:32.238750 140015849629568 estimator_utils.py:212]   name = decoder/layer_0/ffn/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*
    I0323 09:43:32.238821 140015849629568 estimator_utils.py:212]   name = decoder/layer_0/ffn/dense_1/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.238902 140015849629568 estimator_utils.py:212]   name = decoder/layer_0/ffn/dense_1/bias:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.238975 140015849629568 estimator_utils.py:212]   name = decoder/layer_1/self_attention/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.239044 140015849629568 estimator_utils.py:212]   name = decoder/layer_1/self_attention/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.239113 140015849629568 estimator_utils.py:212]   name = decoder/layer_1/self_attention/q_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.239192 140015849629568 estimator_utils.py:212]   name = decoder/layer_1/self_attention/k_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.239273 140015849629568 estimator_utils.py:212]   name = decoder/layer_1/self_attention/v_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.239348 140015849629568 estimator_utils.py:212]   name = decoder/layer_1/self_attention/output_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.239422 140015849629568 estimator_utils.py:212]   name = decoder/layer_1/memory_attention/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.239492 140015849629568 estimator_utils.py:212]   name = decoder/layer_1/memory_attention/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.239562 140015849629568 estimator_utils.py:212]   name = decoder/layer_1/memory_attention/q_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.239635 140015849629568 estimator_utils.py:212]   name = decoder/layer_1/memory_attention/k_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.239709 140015849629568 estimator_utils.py:212]   name = decoder/layer_1/memory_attention/v_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.239782 140015849629568 estimator_utils.py:212]   name = decoder/layer_1/memory_attention/output_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.239864 140015849629568 estimator_utils.py:212]   name = decoder/layer_1/ffn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.239936 140015849629568 estimator_utils.py:212]   name = decoder/layer_1/ffn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.240007 140015849629568 estimator_utils.py:212]   name = decoder/layer_1/ffn/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
    I0323 09:43:32.240079 140015849629568 estimator_utils.py:212]   name = decoder/layer_1/ffn/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*
    I0323 09:43:32.240165 140015849629568 estimator_utils.py:212]   name = decoder/layer_1/ffn/dense_1/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.240276 140015849629568 estimator_utils.py:212]   name = decoder/layer_1/ffn/dense_1/bias:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.240358 140015849629568 estimator_utils.py:212]   name = decoder/layer_2/self_attention/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.240443 140015849629568 estimator_utils.py:212]   name = decoder/layer_2/self_attention/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.240524 140015849629568 estimator_utils.py:212]   name = decoder/layer_2/self_attention/q_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.240609 140015849629568 estimator_utils.py:212]   name = decoder/layer_2/self_attention/k_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.240694 140015849629568 estimator_utils.py:212]   name = decoder/layer_2/self_attention/v_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.240779 140015849629568 estimator_utils.py:212]   name = decoder/layer_2/self_attention/output_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.240875 140015849629568 estimator_utils.py:212]   name = decoder/layer_2/memory_attention/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.240958 140015849629568 estimator_utils.py:212]   name = decoder/layer_2/memory_attention/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.241040 140015849629568 estimator_utils.py:212]   name = decoder/layer_2/memory_attention/q_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.241124 140015849629568 estimator_utils.py:212]   name = decoder/layer_2/memory_attention/k_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.241224 140015849629568 estimator_utils.py:212]   name = decoder/layer_2/memory_attention/v_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.241328 140015849629568 estimator_utils.py:212]   name = decoder/layer_2/memory_attention/output_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.241410 140015849629568 estimator_utils.py:212]   name = decoder/layer_2/ffn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.241480 140015849629568 estimator_utils.py:212]   name = decoder/layer_2/ffn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.241550 140015849629568 estimator_utils.py:212]   name = decoder/layer_2/ffn/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
    I0323 09:43:32.241624 140015849629568 estimator_utils.py:212]   name = decoder/layer_2/ffn/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*
    I0323 09:43:32.241695 140015849629568 estimator_utils.py:212]   name = decoder/layer_2/ffn/dense_1/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.241768 140015849629568 estimator_utils.py:212]   name = decoder/layer_2/ffn/dense_1/bias:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.332492 140015849629568 estimator_utils.py:212]   name = decoder/layer_3/self_attention/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.332652 140015849629568 estimator_utils.py:212]   name = decoder/layer_3/self_attention/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.332785 140015849629568 estimator_utils.py:212]   name = decoder/layer_3/self_attention/q_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.332934 140015849629568 estimator_utils.py:212]   name = decoder/layer_3/self_attention/k_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.333051 140015849629568 estimator_utils.py:212]   name = decoder/layer_3/self_attention/v_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.333172 140015849629568 estimator_utils.py:212]   name = decoder/layer_3/self_attention/output_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.333285 140015849629568 estimator_utils.py:212]   name = decoder/layer_3/memory_attention/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.333382 140015849629568 estimator_utils.py:212]   name = decoder/layer_3/memory_attention/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.333477 140015849629568 estimator_utils.py:212]   name = decoder/layer_3/memory_attention/q_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.333576 140015849629568 estimator_utils.py:212]   name = decoder/layer_3/memory_attention/k_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.333680 140015849629568 estimator_utils.py:212]   name = decoder/layer_3/memory_attention/v_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.333781 140015849629568 estimator_utils.py:212]   name = decoder/layer_3/memory_attention/output_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.333900 140015849629568 estimator_utils.py:212]   name = decoder/layer_3/ffn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.334002 140015849629568 estimator_utils.py:212]   name = decoder/layer_3/ffn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.334095 140015849629568 estimator_utils.py:212]   name = decoder/layer_3/ffn/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
    I0323 09:43:32.334207 140015849629568 estimator_utils.py:212]   name = decoder/layer_3/ffn/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*
    I0323 09:43:32.334306 140015849629568 estimator_utils.py:212]   name = decoder/layer_3/ffn/dense_1/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.334389 140015849629568 estimator_utils.py:212]   name = decoder/layer_3/ffn/dense_1/bias:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.334459 140015849629568 estimator_utils.py:212]   name = decoder/layer_4/self_attention/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.334525 140015849629568 estimator_utils.py:212]   name = decoder/layer_4/self_attention/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.334595 140015849629568 estimator_utils.py:212]   name = decoder/layer_4/self_attention/q_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.334670 140015849629568 estimator_utils.py:212]   name = decoder/layer_4/self_attention/k_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.334744 140015849629568 estimator_utils.py:212]   name = decoder/layer_4/self_attention/v_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.334817 140015849629568 estimator_utils.py:212]   name = decoder/layer_4/self_attention/output_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.334908 140015849629568 estimator_utils.py:212]   name = decoder/layer_4/memory_attention/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.334978 140015849629568 estimator_utils.py:212]   name = decoder/layer_4/memory_attention/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.335048 140015849629568 estimator_utils.py:212]   name = decoder/layer_4/memory_attention/q_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.335121 140015849629568 estimator_utils.py:212]   name = decoder/layer_4/memory_attention/k_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.335209 140015849629568 estimator_utils.py:212]   name = decoder/layer_4/memory_attention/v_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.335284 140015849629568 estimator_utils.py:212]   name = decoder/layer_4/memory_attention/output_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.335357 140015849629568 estimator_utils.py:212]   name = decoder/layer_4/ffn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.335427 140015849629568 estimator_utils.py:212]   name = decoder/layer_4/ffn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.335497 140015849629568 estimator_utils.py:212]   name = decoder/layer_4/ffn/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
    I0323 09:43:32.335569 140015849629568 estimator_utils.py:212]   name = decoder/layer_4/ffn/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*
    I0323 09:43:32.335639 140015849629568 estimator_utils.py:212]   name = decoder/layer_4/ffn/dense_1/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.335711 140015849629568 estimator_utils.py:212]   name = decoder/layer_4/ffn/dense_1/bias:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.335783 140015849629568 estimator_utils.py:212]   name = decoder/layer_5/self_attention/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.335865 140015849629568 estimator_utils.py:212]   name = decoder/layer_5/self_attention/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.335935 140015849629568 estimator_utils.py:212]   name = decoder/layer_5/self_attention/q_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.336014 140015849629568 estimator_utils.py:212]   name = decoder/layer_5/self_attention/k_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.336086 140015849629568 estimator_utils.py:212]   name = decoder/layer_5/self_attention/v_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.336165 140015849629568 estimator_utils.py:212]   name = decoder/layer_5/self_attention/output_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.336237 140015849629568 estimator_utils.py:212]   name = decoder/layer_5/memory_attention/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.336308 140015849629568 estimator_utils.py:212]   name = decoder/layer_5/memory_attention/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.336377 140015849629568 estimator_utils.py:212]   name = decoder/layer_5/memory_attention/q_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.336449 140015849629568 estimator_utils.py:212]   name = decoder/layer_5/memory_attention/k_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.336522 140015849629568 estimator_utils.py:212]   name = decoder/layer_5/memory_attention/v_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.336594 140015849629568 estimator_utils.py:212]   name = decoder/layer_5/memory_attention/output_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.336665 140015849629568 estimator_utils.py:212]   name = decoder/layer_5/ffn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.336735 140015849629568 estimator_utils.py:212]   name = decoder/layer_5/ffn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.336805 140015849629568 estimator_utils.py:212]   name = decoder/layer_5/ffn/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
    I0323 09:43:32.336897 140015849629568 estimator_utils.py:212]   name = decoder/layer_5/ffn/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*
    I0323 09:43:32.336966 140015849629568 estimator_utils.py:212]   name = decoder/layer_5/ffn/dense_1/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.337038 140015849629568 estimator_utils.py:212]   name = decoder/layer_5/ffn/dense_1/bias:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.337108 140015849629568 estimator_utils.py:212]   name = decoder/layer_6/self_attention/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.337183 140015849629568 estimator_utils.py:212]   name = decoder/layer_6/self_attention/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.337254 140015849629568 estimator_utils.py:212]   name = decoder/layer_6/self_attention/q_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.337326 140015849629568 estimator_utils.py:212]   name = decoder/layer_6/self_attention/k_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.337398 140015849629568 estimator_utils.py:212]   name = decoder/layer_6/self_attention/v_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.337470 140015849629568 estimator_utils.py:212]   name = decoder/layer_6/self_attention/output_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.337543 140015849629568 estimator_utils.py:212]   name = decoder/layer_6/memory_attention/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.337611 140015849629568 estimator_utils.py:212]   name = decoder/layer_6/memory_attention/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.337680 140015849629568 estimator_utils.py:212]   name = decoder/layer_6/memory_attention/q_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.337754 140015849629568 estimator_utils.py:212]   name = decoder/layer_6/memory_attention/k_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.337836 140015849629568 estimator_utils.py:212]   name = decoder/layer_6/memory_attention/v_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.337913 140015849629568 estimator_utils.py:212]   name = decoder/layer_6/memory_attention/output_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.337986 140015849629568 estimator_utils.py:212]   name = decoder/layer_6/ffn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.338057 140015849629568 estimator_utils.py:212]   name = decoder/layer_6/ffn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.338125 140015849629568 estimator_utils.py:212]   name = decoder/layer_6/ffn/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
    I0323 09:43:32.338205 140015849629568 estimator_utils.py:212]   name = decoder/layer_6/ffn/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*
    I0323 09:43:32.338276 140015849629568 estimator_utils.py:212]   name = decoder/layer_6/ffn/dense_1/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.338347 140015849629568 estimator_utils.py:212]   name = decoder/layer_6/ffn/dense_1/bias:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.338417 140015849629568 estimator_utils.py:212]   name = decoder/layer_7/self_attention/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.338485 140015849629568 estimator_utils.py:212]   name = decoder/layer_7/self_attention/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.338552 140015849629568 estimator_utils.py:212]   name = decoder/layer_7/self_attention/q_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.338624 140015849629568 estimator_utils.py:212]   name = decoder/layer_7/self_attention/k_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.338697 140015849629568 estimator_utils.py:212]   name = decoder/layer_7/self_attention/v_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.338772 140015849629568 estimator_utils.py:212]   name = decoder/layer_7/self_attention/output_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.338857 140015849629568 estimator_utils.py:212]   name = decoder/layer_7/memory_attention/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.338928 140015849629568 estimator_utils.py:212]   name = decoder/layer_7/memory_attention/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.339002 140015849629568 estimator_utils.py:212]   name = decoder/layer_7/memory_attention/q_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.339075 140015849629568 estimator_utils.py:212]   name = decoder/layer_7/memory_attention/k_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.339154 140015849629568 estimator_utils.py:212]   name = decoder/layer_7/memory_attention/v_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.339229 140015849629568 estimator_utils.py:212]   name = decoder/layer_7/memory_attention/output_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.339304 140015849629568 estimator_utils.py:212]   name = decoder/layer_7/ffn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.339377 140015849629568 estimator_utils.py:212]   name = decoder/layer_7/ffn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.339449 140015849629568 estimator_utils.py:212]   name = decoder/layer_7/ffn/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
    I0323 09:43:32.339522 140015849629568 estimator_utils.py:212]   name = decoder/layer_7/ffn/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*
    I0323 09:43:32.339593 140015849629568 estimator_utils.py:212]   name = decoder/layer_7/ffn/dense_1/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.339667 140015849629568 estimator_utils.py:212]   name = decoder/layer_7/ffn/dense_1/bias:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.339738 140015849629568 estimator_utils.py:212]   name = decoder/layer_8/self_attention/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.339806 140015849629568 estimator_utils.py:212]   name = decoder/layer_8/self_attention/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.339899 140015849629568 estimator_utils.py:212]   name = decoder/layer_8/self_attention/q_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.339972 140015849629568 estimator_utils.py:212]   name = decoder/layer_8/self_attention/k_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.340045 140015849629568 estimator_utils.py:212]   name = decoder/layer_8/self_attention/v_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.340119 140015849629568 estimator_utils.py:212]   name = decoder/layer_8/self_attention/output_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.340238 140015849629568 estimator_utils.py:212]   name = decoder/layer_8/memory_attention/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.340320 140015849629568 estimator_utils.py:212]   name = decoder/layer_8/memory_attention/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.340402 140015849629568 estimator_utils.py:212]   name = decoder/layer_8/memory_attention/q_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.340490 140015849629568 estimator_utils.py:212]   name = decoder/layer_8/memory_attention/k_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.340575 140015849629568 estimator_utils.py:212]   name = decoder/layer_8/memory_attention/v_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.340661 140015849629568 estimator_utils.py:212]   name = decoder/layer_8/memory_attention/output_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.340748 140015849629568 estimator_utils.py:212]   name = decoder/layer_8/ffn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.340842 140015849629568 estimator_utils.py:212]   name = decoder/layer_8/ffn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.340923 140015849629568 estimator_utils.py:212]   name = decoder/layer_8/ffn/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
    I0323 09:43:32.341008 140015849629568 estimator_utils.py:212]   name = decoder/layer_8/ffn/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*
    I0323 09:43:32.341090 140015849629568 estimator_utils.py:212]   name = decoder/layer_8/ffn/dense_1/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.341179 140015849629568 estimator_utils.py:212]   name = decoder/layer_8/ffn/dense_1/bias:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.341263 140015849629568 estimator_utils.py:212]   name = decoder/layer_9/self_attention/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.341365 140015849629568 estimator_utils.py:212]   name = decoder/layer_9/self_attention/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.341435 140015849629568 estimator_utils.py:212]   name = decoder/layer_9/self_attention/q_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.341508 140015849629568 estimator_utils.py:212]   name = decoder/layer_9/self_attention/k_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.341581 140015849629568 estimator_utils.py:212]   name = decoder/layer_9/self_attention/v_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.341655 140015849629568 estimator_utils.py:212]   name = decoder/layer_9/self_attention/output_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.341730 140015849629568 estimator_utils.py:212]   name = decoder/layer_9/memory_attention/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.341800 140015849629568 estimator_utils.py:212]   name = decoder/layer_9/memory_attention/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.341883 140015849629568 estimator_utils.py:212]   name = decoder/layer_9/memory_attention/q_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.341959 140015849629568 estimator_utils.py:212]   name = decoder/layer_9/memory_attention/k_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.342032 140015849629568 estimator_utils.py:212]   name = decoder/layer_9/memory_attention/v_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.342105 140015849629568 estimator_utils.py:212]   name = decoder/layer_9/memory_attention/output_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.342192 140015849629568 estimator_utils.py:212]   name = decoder/layer_9/ffn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.342264 140015849629568 estimator_utils.py:212]   name = decoder/layer_9/ffn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.342333 140015849629568 estimator_utils.py:212]   name = decoder/layer_9/ffn/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
    I0323 09:43:32.433445 140015849629568 estimator_utils.py:212]   name = decoder/layer_9/ffn/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*
    I0323 09:43:32.433609 140015849629568 estimator_utils.py:212]   name = decoder/layer_9/ffn/dense_1/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.433751 140015849629568 estimator_utils.py:212]   name = decoder/layer_9/ffn/dense_1/bias:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.433883 140015849629568 estimator_utils.py:212]   name = decoder/layer_10/self_attention/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.433999 140015849629568 estimator_utils.py:212]   name = decoder/layer_10/self_attention/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.434098 140015849629568 estimator_utils.py:212]   name = decoder/layer_10/self_attention/q_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.434213 140015849629568 estimator_utils.py:212]   name = decoder/layer_10/self_attention/k_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.434314 140015849629568 estimator_utils.py:212]   name = decoder/layer_10/self_attention/v_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.434404 140015849629568 estimator_utils.py:212]   name = decoder/layer_10/self_attention/output_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.434489 140015849629568 estimator_utils.py:212]   name = decoder/layer_10/memory_attention/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.434562 140015849629568 estimator_utils.py:212]   name = decoder/layer_10/memory_attention/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.434629 140015849629568 estimator_utils.py:212]   name = decoder/layer_10/memory_attention/q_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.434700 140015849629568 estimator_utils.py:212]   name = decoder/layer_10/memory_attention/k_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.434767 140015849629568 estimator_utils.py:212]   name = decoder/layer_10/memory_attention/v_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.434847 140015849629568 estimator_utils.py:212]   name = decoder/layer_10/memory_attention/output_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.434921 140015849629568 estimator_utils.py:212]   name = decoder/layer_10/ffn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.434986 140015849629568 estimator_utils.py:212]   name = decoder/layer_10/ffn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.435051 140015849629568 estimator_utils.py:212]   name = decoder/layer_10/ffn/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
    I0323 09:43:32.435121 140015849629568 estimator_utils.py:212]   name = decoder/layer_10/ffn/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*
    I0323 09:43:32.435227 140015849629568 estimator_utils.py:212]   name = decoder/layer_10/ffn/dense_1/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.435322 140015849629568 estimator_utils.py:212]   name = decoder/layer_10/ffn/dense_1/bias:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.435406 140015849629568 estimator_utils.py:212]   name = decoder/layer_11/self_attention/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.435493 140015849629568 estimator_utils.py:212]   name = decoder/layer_11/self_attention/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.435578 140015849629568 estimator_utils.py:212]   name = decoder/layer_11/self_attention/q_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.435667 140015849629568 estimator_utils.py:212]   name = decoder/layer_11/self_attention/k_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.435761 140015849629568 estimator_utils.py:212]   name = decoder/layer_11/self_attention/v_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.435864 140015849629568 estimator_utils.py:212]   name = decoder/layer_11/self_attention/output_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.435948 140015849629568 estimator_utils.py:212]   name = decoder/layer_11/memory_attention/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.436028 140015849629568 estimator_utils.py:212]   name = decoder/layer_11/memory_attention/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.436101 140015849629568 estimator_utils.py:212]   name = decoder/layer_11/memory_attention/q_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.436192 140015849629568 estimator_utils.py:212]   name = decoder/layer_11/memory_attention/k_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.436273 140015849629568 estimator_utils.py:212]   name = decoder/layer_11/memory_attention/v_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.436350 140015849629568 estimator_utils.py:212]   name = decoder/layer_11/memory_attention/output_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.436429 140015849629568 estimator_utils.py:212]   name = decoder/layer_11/ffn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.436504 140015849629568 estimator_utils.py:212]   name = decoder/layer_11/ffn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.436582 140015849629568 estimator_utils.py:212]   name = decoder/layer_11/ffn/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
    I0323 09:43:32.436670 140015849629568 estimator_utils.py:212]   name = decoder/layer_11/ffn/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*
    I0323 09:43:32.436751 140015849629568 estimator_utils.py:212]   name = decoder/layer_11/ffn/dense_1/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.436852 140015849629568 estimator_utils.py:212]   name = decoder/layer_11/ffn/dense_1/bias:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.436933 140015849629568 estimator_utils.py:212]   name = decoder/layer_12/self_attention/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.437003 140015849629568 estimator_utils.py:212]   name = decoder/layer_12/self_attention/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.437070 140015849629568 estimator_utils.py:212]   name = decoder/layer_12/self_attention/q_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.437149 140015849629568 estimator_utils.py:212]   name = decoder/layer_12/self_attention/k_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.437220 140015849629568 estimator_utils.py:212]   name = decoder/layer_12/self_attention/v_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.437291 140015849629568 estimator_utils.py:212]   name = decoder/layer_12/self_attention/output_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.437362 140015849629568 estimator_utils.py:212]   name = decoder/layer_12/memory_attention/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.437431 140015849629568 estimator_utils.py:212]   name = decoder/layer_12/memory_attention/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.437505 140015849629568 estimator_utils.py:212]   name = decoder/layer_12/memory_attention/q_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.437584 140015849629568 estimator_utils.py:212]   name = decoder/layer_12/memory_attention/k_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.437662 140015849629568 estimator_utils.py:212]   name = decoder/layer_12/memory_attention/v_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.437745 140015849629568 estimator_utils.py:212]   name = decoder/layer_12/memory_attention/output_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.437819 140015849629568 estimator_utils.py:212]   name = decoder/layer_12/ffn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.437910 140015849629568 estimator_utils.py:212]   name = decoder/layer_12/ffn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.437981 140015849629568 estimator_utils.py:212]   name = decoder/layer_12/ffn/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
    I0323 09:43:32.438057 140015849629568 estimator_utils.py:212]   name = decoder/layer_12/ffn/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*
    I0323 09:43:32.438132 140015849629568 estimator_utils.py:212]   name = decoder/layer_12/ffn/dense_1/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.438220 140015849629568 estimator_utils.py:212]   name = decoder/layer_12/ffn/dense_1/bias:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.438289 140015849629568 estimator_utils.py:212]   name = decoder/layer_13/self_attention/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.438360 140015849629568 estimator_utils.py:212]   name = decoder/layer_13/self_attention/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.438428 140015849629568 estimator_utils.py:212]   name = decoder/layer_13/self_attention/q_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.438506 140015849629568 estimator_utils.py:212]   name = decoder/layer_13/self_attention/k_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.438580 140015849629568 estimator_utils.py:212]   name = decoder/layer_13/self_attention/v_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.438657 140015849629568 estimator_utils.py:212]   name = decoder/layer_13/self_attention/output_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.438735 140015849629568 estimator_utils.py:212]   name = decoder/layer_13/memory_attention/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.438811 140015849629568 estimator_utils.py:212]   name = decoder/layer_13/memory_attention/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.438904 140015849629568 estimator_utils.py:212]   name = decoder/layer_13/memory_attention/q_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.438982 140015849629568 estimator_utils.py:212]   name = decoder/layer_13/memory_attention/k_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.439058 140015849629568 estimator_utils.py:212]   name = decoder/layer_13/memory_attention/v_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.439132 140015849629568 estimator_utils.py:212]   name = decoder/layer_13/memory_attention/output_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.439222 140015849629568 estimator_utils.py:212]   name = decoder/layer_13/ffn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.439300 140015849629568 estimator_utils.py:212]   name = decoder/layer_13/ffn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.439390 140015849629568 estimator_utils.py:212]   name = decoder/layer_13/ffn/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
    I0323 09:43:32.439485 140015849629568 estimator_utils.py:212]   name = decoder/layer_13/ffn/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*
    I0323 09:43:32.439578 140015849629568 estimator_utils.py:212]   name = decoder/layer_13/ffn/dense_1/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.439678 140015849629568 estimator_utils.py:212]   name = decoder/layer_13/ffn/dense_1/bias:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.439776 140015849629568 estimator_utils.py:212]   name = decoder/layer_14/self_attention/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.439886 140015849629568 estimator_utils.py:212]   name = decoder/layer_14/self_attention/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.439987 140015849629568 estimator_utils.py:212]   name = decoder/layer_14/self_attention/q_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.440090 140015849629568 estimator_utils.py:212]   name = decoder/layer_14/self_attention/k_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.440250 140015849629568 estimator_utils.py:212]   name = decoder/layer_14/self_attention/v_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.440375 140015849629568 estimator_utils.py:212]   name = decoder/layer_14/self_attention/output_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.440502 140015849629568 estimator_utils.py:212]   name = decoder/layer_14/memory_attention/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.440621 140015849629568 estimator_utils.py:212]   name = decoder/layer_14/memory_attention/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.440739 140015849629568 estimator_utils.py:212]   name = decoder/layer_14/memory_attention/q_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.440872 140015849629568 estimator_utils.py:212]   name = decoder/layer_14/memory_attention/k_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.440992 140015849629568 estimator_utils.py:212]   name = decoder/layer_14/memory_attention/v_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.441100 140015849629568 estimator_utils.py:212]   name = decoder/layer_14/memory_attention/output_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.441219 140015849629568 estimator_utils.py:212]   name = decoder/layer_14/ffn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.441350 140015849629568 estimator_utils.py:212]   name = decoder/layer_14/ffn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.441440 140015849629568 estimator_utils.py:212]   name = decoder/layer_14/ffn/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
    I0323 09:43:32.441535 140015849629568 estimator_utils.py:212]   name = decoder/layer_14/ffn/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*
    I0323 09:43:32.441618 140015849629568 estimator_utils.py:212]   name = decoder/layer_14/ffn/dense_1/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.441696 140015849629568 estimator_utils.py:212]   name = decoder/layer_14/ffn/dense_1/bias:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.441766 140015849629568 estimator_utils.py:212]   name = decoder/layer_15/self_attention/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.441849 140015849629568 estimator_utils.py:212]   name = decoder/layer_15/self_attention/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.441924 140015849629568 estimator_utils.py:212]   name = decoder/layer_15/self_attention/q_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.441995 140015849629568 estimator_utils.py:212]   name = decoder/layer_15/self_attention/k_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.442071 140015849629568 estimator_utils.py:212]   name = decoder/layer_15/self_attention/v_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.442189 140015849629568 estimator_utils.py:212]   name = decoder/layer_15/self_attention/output_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.442285 140015849629568 estimator_utils.py:212]   name = decoder/layer_15/memory_attention/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.442373 140015849629568 estimator_utils.py:212]   name = decoder/layer_15/memory_attention/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.442454 140015849629568 estimator_utils.py:212]   name = decoder/layer_15/memory_attention/q_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.442531 140015849629568 estimator_utils.py:212]   name = decoder/layer_15/memory_attention/k_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.442607 140015849629568 estimator_utils.py:212]   name = decoder/layer_15/memory_attention/v_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.442705 140015849629568 estimator_utils.py:212]   name = decoder/layer_15/memory_attention/output_proj/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.442797 140015849629568 estimator_utils.py:212]   name = decoder/layer_15/ffn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.442889 140015849629568 estimator_utils.py:212]   name = decoder/layer_15/ffn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.442965 140015849629568 estimator_utils.py:212]   name = decoder/layer_15/ffn/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
    I0323 09:43:32.443040 140015849629568 estimator_utils.py:212]   name = decoder/layer_15/ffn/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*
    I0323 09:43:32.443119 140015849629568 estimator_utils.py:212]   name = decoder/layer_15/ffn/dense_1/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
    I0323 09:43:32.443209 140015849629568 estimator_utils.py:212]   name = decoder/layer_15/ffn/dense_1/bias:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.443305 140015849629568 estimator_utils.py:212]   name = decoder/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
    I0323 09:43:32.443403 140015849629568 estimator_utils.py:212]   name = decoder/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
    INFO:tensorflow:Done calling model_fn.
    I0323 09:43:32.462981 140015849629568 estimator.py:1150] Done calling model_fn.
    INFO:tensorflow:Create CheckpointSaverHook.
    I0323 09:43:32.464104 140015849629568 basic_session_run_hooks.py:541] Create CheckpointSaverHook.
    INFO:tensorflow:Graph was finalized.
    I0323 09:43:39.422322 140015849629568 monitored_session.py:240] Graph was finalized.
    2021-03-23 09:43:39.433174: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz
    2021-03-23 09:43:39.433499: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x555afaa73100 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
    2021-03-23 09:43:39.433628: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
    2021-03-23 09:43:39.554076: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
    2021-03-23 09:43:39.554751: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x555afaa73480 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
    2021-03-23 09:43:39.554781: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Tesla T4, Compute Capability 7.5
    2021-03-23 09:43:39.555077: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
    2021-03-23 09:43:39.555607: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties:
    name: Tesla T4 major: 7 minor: 5 memoryClockRate(GHz): 1.59
    pciBusID: 0000:00:04.0
    2021-03-23 09:43:39.555703: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
    2021-03-23 09:43:39.555736: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
    2021-03-23 09:43:39.555765: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
    2021-03-23 09:43:39.555791: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
    2021-03-23 09:43:39.555814: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
    2021-03-23 09:43:39.555854: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
    2021-03-23 09:43:39.555886: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
    2021-03-23 09:43:39.555985: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
    2021-03-23 09:43:39.556568: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
    2021-03-23 09:43:39.557075: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0
    2021-03-23 09:43:39.560350: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
    2021-03-23 09:43:39.562783: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:
    2021-03-23 09:43:39.562817: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186]      0
    2021-03-23 09:43:39.562843: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0:   N
    2021-03-23 09:43:39.564132: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
    2021-03-23 09:43:39.564743: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
    2021-03-23 09:43:39.565282: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.
    2021-03-23 09:43:39.565326: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14257 MB memory) -> physical GPU (device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5)
    INFO:tensorflow:Running local_init_op.
    I0323 09:43:53.398160 140015849629568 session_manager.py:500] Running local_init_op.
    INFO:tensorflow:Done running local_init_op.
    I0323 09:43:54.111346 140015849629568 session_manager.py:502] Done running local_init_op.
    INFO:tensorflow:Saving checkpoints for 0 into ckpt/aeslc/model.ckpt.
    I0323 09:44:13.725221 140015849629568 basic_session_run_hooks.py:606] Saving checkpoints for 0 into ckpt/aeslc/model.ckpt.
    2021-03-23 09:44:51.040895: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 16777216 exceeds 10% of system memory.
    2021-03-23 09:44:51.072007: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 16777216 exceeds 10% of system memory.
    2021-03-23 09:44:51.072070: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 16777216 exceeds 10% of system memory.
    2021-03-23 09:44:51.277187: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 16777216 exceeds 10% of system memory.
    2021-03-23 09:44:51.281719: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 16777216 exceeds 10% of system memory.
    2021-03-23 09:45:26.473739: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
    INFO:tensorflow:global_step/sec: 0.0242657
    I0323 09:46:18.352407 140015849629568 tpu_estimator.py:2307] global_step/sec: 0.0242657
    INFO:tensorflow:examples/sec: 0.194126
    I0323 09:46:18.353016 140015849629568 tpu_estimator.py:2308] examples/sec: 0.194126
    INFO:tensorflow:global_step/sec: 0.432297
    I0323 09:46:20.665547 140015849629568 tpu_estimator.py:2307] global_step/sec: 0.432297
    INFO:tensorflow:examples/sec: 3.45838
    I0323 09:46:20.665931 140015849629568 tpu_estimator.py:2308] examples/sec: 3.45838
    INFO:tensorflow:global_step/sec: 0.424376
    I0323 09:46:23.021980 140015849629568 tpu_estimator.py:2307] global_step/sec: 0.424376
    INFO:tensorflow:examples/sec: 3.39501
    I0323 09:46:23.022383 140015849629568 tpu_estimator.py:2308] examples/sec: 3.39501
    INFO:tensorflow:global_step/sec: 0.428958
    I0323 09:46:25.353182 140015849629568 tpu_estimator.py:2307] global_step/sec: 0.428958
    INFO:tensorflow:examples/sec: 3.43166
    I0323 09:46:25.353410 140015849629568 tpu_estimator.py:2308] examples/sec: 3.43166
    INFO:tensorflow:global_step/sec: 0.42771
    I0323 09:46:27.691224 140015849629568 tpu_estimator.py:2307] global_step/sec: 0.42771
    INFO:tensorflow:examples/sec: 3.42168
    I0323 09:46:27.691456 140015849629568 tpu_estimator.py:2308] examples/sec: 3.42168
    INFO:tensorflow:global_step/sec: 0.425015
    I0323 09:46:30.044064 140015849629568 tpu_estimator.py:2307] global_step/sec: 0.425015
    INFO:tensorflow:examples/sec: 3.40012
    I0323 09:46:30.044290 140015849629568 tpu_estimator.py:2308] examples/sec: 3.40012
    INFO:tensorflow:global_step/sec: 0.425547
    I0323 09:46:32.394001 140015849629568 tpu_estimator.py:2307] global_step/sec: 0.425547
    INFO:tensorflow:examples/sec: 3.40438
    I0323 09:46:32.394251 140015849629568 tpu_estimator.py:2308] examples/sec: 3.40438
    INFO:tensorflow:global_step/sec: 0.423838
    I0323 09:46:34.753371 140015849629568 tpu_estimator.py:2307] global_step/sec: 0.423838
    INFO:tensorflow:examples/sec: 3.3907
    I0323 09:46:34.753606 140015849629568 tpu_estimator.py:2308] examples/sec: 3.3907
    INFO:tensorflow:global_step/sec: 0.419245
    I0323 09:46:37.138610 140015849629568 tpu_estimator.py:2307] global_step/sec: 0.419245
    INFO:tensorflow:examples/sec: 3.35396
    I0323 09:46:37.138849 140015849629568 tpu_estimator.py:2308] examples/sec: 3.35396
    INFO:tensorflow:global_step/sec: 0.420715
    I0323 09:46:39.515527 140015849629568 tpu_estimator.py:2307] global_step/sec: 0.420715
    INFO:tensorflow:examples/sec: 3.36572
    I0323 09:46:39.515770 140015849629568 tpu_estimator.py:2308] examples/sec: 3.36572
    INFO:tensorflow:global_step/sec: 0.417823
    I0323 09:46:41.908904 140015849629568 tpu_estimator.py:2307] global_step/sec: 0.417823
    INFO:tensorflow:examples/sec: 3.34258
    I0323 09:46:41.909526 140015849629568 tpu_estimator.py:2308] examples/sec: 3.34258
    INFO:tensorflow:global_step/sec: 0.419327
    I0323 09:46:44.293651 140015849629568 tpu_estimator.py:2307] global_step/sec: 0.419327
    INFO:tensorflow:examples/sec: 3.35462
    I0323 09:46:44.293982 140015849629568 tpu_estimator.py:2308] examples/sec: 3.35462
    INFO:tensorflow:global_step/sec: 0.41948
    I0323 09:46:46.677553 140015849629568 tpu_estimator.py:2307] global_step/sec: 0.41948
    INFO:tensorflow:examples/sec: 3.35584
    I0323 09:46:46.677786 140015849629568 tpu_estimator.py:2308] examples/sec: 3.35584
    INFO:tensorflow:global_step/sec: 0.41854
    I0323 09:46:49.066837 140015849629568 tpu_estimator.py:2307] global_step/sec: 0.41854
    INFO:tensorflow:examples/sec: 3.34832
    I0323 09:46:49.067070 140015849629568 tpu_estimator.py:2308] examples/sec: 3.34832
    INFO:tensorflow:global_step/sec: 0.415718
    I0323 09:46:51.472283 140015849629568 tpu_estimator.py:2307] global_step/sec: 0.415718
    INFO:tensorflow:examples/sec: 3.32574
    I0323 09:46:51.472512 140015849629568 tpu_estimator.py:2308] examples/sec: 3.32574
    INFO:tensorflow:global_step/sec: 0.413858
    I0323 09:46:53.888569 140015849629568 tpu_estimator.py:2307] global_step/sec: 0.413858
    INFO:tensorflow:examples/sec: 3.31087
    I0323 09:46:53.888791 140015849629568 tpu_estimator.py:2308] examples/sec: 3.31087
    INFO:tensorflow:global_step/sec: 0.41378
    I0323 09:46:56.305333 140015849629568 tpu_estimator.py:2307] global_step/sec: 0.41378
    INFO:tensorflow:examples/sec: 3.31024
    I0323 09:46:56.305731 140015849629568 tpu_estimator.py:2308] examples/sec: 3.31024
    INFO:tensorflow:global_step/sec: 0.413859
    I0323 09:46:58.721612 140015849629568 tpu_estimator.py:2307] global_step/sec: 0.413859
    INFO:tensorflow:examples/sec: 3.31087
    I0323 09:46:58.722032 140015849629568 tpu_estimator.py:2308] examples/sec: 3.31087
    INFO:tensorflow:global_step/sec: 0.406715
    I0323 09:47:01.180323 140015849629568 tpu_estimator.py:2307] global_step/sec: 0.406715
    INFO:tensorflow:examples/sec: 3.25372
    I0323 09:47:01.180566 140015849629568 tpu_estimator.py:2308] examples/sec: 3.25372

```
!python3 pegasus/bin/evaluate.py --params=aeslc_transformer \
--param_overrides=vocab_filename=ckpt/c4.unigram.newline.10pct.96000.model,batch_size=1,beam_size=5,beam_alpha=0.6 \
--model_dir=ckpt/aeslc
```