Before we start the tutorial, we need to install some dependencies to render and capture images from OpenAI Gym. It may take some time to set up the environment.

```
!apt-get update
!apt-get install cmake zlib1g-dev libjpeg-dev xvfb xorg-dev libboost-all-dev libsdl2-dev swig python3-dev python3-future python-opengl x11-utils

!apt-get -qq -y install libcusparse9.1 libnvrtc9.1 libnvtoolsext1 > /dev/null
!ln -snf /usr/lib/x86_64-linux-gnu/libnvrtc-builtins.so.9.1 /usr/lib/x86_64-linux-gnu/libnvrtc-builtins.so
!apt-get -qq -y install xvfb freeglut3-dev ffmpeg> /dev/null

!apt-get install xserver-xorg libglu1-mesa-dev mesa-common-dev libxmu-dev libxi-dev
```

    Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
    Get:2 https://cloud.r-project.org/bin/linux/ubuntu bionic-cran40/ InRelease [3,626 B]
    Ign:3 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  InRelease
    Hit:4 http://archive.ubuntu.com/ubuntu bionic InRelease
    Get:5 http://ppa.launchpad.net/c2d4u.team/c2d4u4.0+/ubuntu bionic InRelease [15.9 kB]
    Ign:6 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  InRelease
    Get:7 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  Release [697 B]
    Hit:8 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Release
    Get:9 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  Release.gpg [836 B]
    Get:10 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
    Get:11 https://cloud.r-project.org/bin/linux/ubuntu bionic-cran40/ Packages [60.5 kB]
    Hit:12 http://ppa.launchpad.net/cran/libgit2/ubuntu bionic InRelease
    Get:13 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
    Hit:14 http://ppa.launchpad.net/deadsnakes/ppa/ubuntu bionic InRelease
    Get:15 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [423 kB]
    Get:16 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [2,152 kB]
    Get:17 http://ppa.launchpad.net/graphics-drivers/ppa/ubuntu bionic InRelease [21.3 kB]
    Get:18 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,412 kB]
    Ign:20 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  Packages
    Get:20 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  Packages [796 kB]
    Get:21 http://ppa.launchpad.net/c2d4u.team/c2d4u4.0+/ubuntu bionic/main Sources [1,767 kB]
    Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [452 kB]
    Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [2,183 kB]
    Get:24 http://ppa.launchpad.net/c2d4u.team/c2d4u4.0+/ubuntu bionic/main amd64 Packages [904 kB]
    Get:25 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [2,583 kB]
    Get:26 http://ppa.launchpad.net/graphics-drivers/ppa/ubuntu bionic/main amd64 Packages [41.5 kB]
    Fetched 13.1 MB in 4s (3,509 kB/s)
    Reading package lists... Done
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    libjpeg-dev is already the newest version (8c-2ubuntu8).
    libjpeg-dev set to manually installed.
    zlib1g-dev is already the newest version (1:1.2.11.dfsg-0ubuntu2).
    zlib1g-dev set to manually installed.
    libboost-all-dev is already the newest version (1.65.1.0ubuntu1).
    cmake is already the newest version (3.10.2-1ubuntu2.18.04.1).
    python3-dev is already the newest version (3.6.7-1~18.04).
    python3-dev set to manually installed.
    The following package was automatically installed and is no longer required:
      libnvidia-common-460
    Use 'apt autoremove' to remove it.
    The following additional packages will be installed:
      gir1.2-ibus-1.0 libcapnp-0.6.1 libdbus-1-dev libdmx-dev libdmx1
      libfontenc-dev libfs-dev libfs6 libibus-1.0-5 libibus-1.0-dev
      libmirclient-dev libmirclient9 libmircommon-dev libmircommon7
      libmircookie-dev libmircookie2 libmircore-dev libmircore1 libmirprotobuf3
      libpciaccess-dev libpixman-1-dev libprotobuf-dev libprotobuf-lite10
      libpulse-dev libpulse-mainloop-glib0 libsndio-dev libudev-dev libxaw7-dev
      libxcomposite-dev libxcursor-dev libxfont-dev libxinerama-dev
      libxkbcommon-dev libxkbfile-dev libxmuu-dev libxpm-dev libxrandr-dev
      libxres-dev libxres1 libxtst-dev libxv-dev libxvmc-dev libxvmc1
      libxxf86dga-dev libxxf86dga1 mir-client-platform-mesa-dev swig3.0
      x11proto-composite-dev x11proto-dri2-dev x11proto-fonts-dev x11proto-gl-dev
      x11proto-randr-dev x11proto-record-dev x11proto-render-dev
      x11proto-resource-dev x11proto-xf86bigfont-dev x11proto-xf86dga-dev
      x11proto-xinerama-dev xserver-xorg-dev
    Suggested packages:
      libxaw-doc libgle3 python-future-doc swig-doc swig-examples swig3.0-examples
      swig3.0-doc mesa-utils
    The following NEW packages will be installed:
      gir1.2-ibus-1.0 libcapnp-0.6.1 libdbus-1-dev libdmx-dev libdmx1
      libfontenc-dev libfs-dev libfs6 libibus-1.0-5 libibus-1.0-dev
      libmirclient-dev libmirclient9 libmircommon-dev libmircommon7
      libmircookie-dev libmircookie2 libmircore-dev libmircore1 libmirprotobuf3
      libpciaccess-dev libpixman-1-dev libprotobuf-dev libprotobuf-lite10
      libpulse-dev libpulse-mainloop-glib0 libsdl2-dev libsndio-dev libudev-dev
      libxaw7-dev libxcomposite-dev libxcursor-dev libxfont-dev libxinerama-dev
      libxkbcommon-dev libxkbfile-dev libxmuu-dev libxpm-dev libxrandr-dev
      libxres-dev libxres1 libxtst-dev libxv-dev libxvmc-dev libxvmc1
      libxxf86dga-dev libxxf86dga1 mir-client-platform-mesa-dev python-opengl
      python3-future swig swig3.0 x11-utils x11proto-composite-dev
      x11proto-dri2-dev x11proto-fonts-dev x11proto-gl-dev x11proto-randr-dev
      x11proto-record-dev x11proto-render-dev x11proto-resource-dev
      x11proto-xf86bigfont-dev x11proto-xf86dga-dev x11proto-xinerama-dev xorg-dev
      xserver-xorg-dev xvfb
    0 upgraded, 66 newly installed, 0 to remove and 72 not upgraded.
    Need to get 7,968 kB of archives.
    After this operation, 46.2 MB of additional disk space will be used.
    Get:1 http://archive.ubuntu.com/ubuntu bionic/main amd64 libdmx1 amd64 1:1.1.3-1 [10.4 kB]
    Get:2 http://archive.ubuntu.com/ubuntu bionic/main amd64 libxxf86dga1 amd64 2:1.1.4-1 [13.7 kB]
    Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libibus-1.0-5 amd64 1.5.17-3ubuntu5.3 [133 kB]
    Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 gir1.2-ibus-1.0 amd64 1.5.17-3ubuntu5.3 [66.5 kB]
    Get:5 http://archive.ubuntu.com/ubuntu bionic/main amd64 libcapnp-0.6.1 amd64 0.6.1-1ubuntu1 [658 kB]
    Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libdbus-1-dev amd64 1.12.2-1ubuntu1.2 [165 kB]
    Get:7 http://archive.ubuntu.com/ubuntu bionic/main amd64 libdmx-dev amd64 1:1.1.3-1 [34.4 kB]
    Get:8 http://archive.ubuntu.com/ubuntu bionic/main amd64 libfontenc-dev amd64 1:1.1.3-1 [13.8 kB]
    Get:9 http://archive.ubuntu.com/ubuntu bionic/main amd64 libfs6 amd64 2:1.0.7-1 [22.5 kB]
    Get:10 http://archive.ubuntu.com/ubuntu bionic/main amd64 x11proto-fonts-dev all 2018.4-4 [2,620 B]
    Get:11 http://archive.ubuntu.com/ubuntu bionic/main amd64 libfs-dev amd64 2:1.0.7-1 [26.8 kB]
    Get:12 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libibus-1.0-dev amd64 1.5.17-3ubuntu5.3 [145 kB]
    Get:13 http://archive.ubuntu.com/ubuntu bionic/main amd64 libmircore1 amd64 0.31.1-0ubuntu1 [26.5 kB]
    Get:14 http://archive.ubuntu.com/ubuntu bionic/main amd64 libmircommon7 amd64 0.31.1-0ubuntu1 [73.9 kB]
    Get:15 http://archive.ubuntu.com/ubuntu bionic/main amd64 libprotobuf-lite10 amd64 3.0.0-9.1ubuntu1 [97.7 kB]
    Get:16 http://archive.ubuntu.com/ubuntu bionic/main amd64 libmirprotobuf3 amd64 0.31.1-0ubuntu1 [127 kB]
    Get:17 http://archive.ubuntu.com/ubuntu bionic/main amd64 libmirclient9 amd64 0.31.1-0ubuntu1 [199 kB]
    Get:18 http://archive.ubuntu.com/ubuntu bionic/main amd64 libmircore-dev amd64 0.31.1-0ubuntu1 [21.7 kB]
    Get:19 http://archive.ubuntu.com/ubuntu bionic/main amd64 libprotobuf-dev amd64 3.0.0-9.1ubuntu1 [959 kB]
    Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxkbcommon-dev amd64 0.8.2-1~ubuntu18.04.1 [150 kB]
    Get:21 http://archive.ubuntu.com/ubuntu bionic/main amd64 libmircommon-dev amd64 0.31.1-0ubuntu1 [13.9 kB]
    Get:22 http://archive.ubuntu.com/ubuntu bionic/main amd64 libmircookie2 amd64 0.31.1-0ubuntu1 [19.7 kB]
    Get:23 http://archive.ubuntu.com/ubuntu bionic/main amd64 libmircookie-dev amd64 0.31.1-0ubuntu1 [4,392 B]
    Get:24 http://archive.ubuntu.com/ubuntu bionic/main amd64 libmirclient-dev amd64 0.31.1-0ubuntu1 [47.8 kB]
    Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 libpixman-1-dev amd64 0.34.0-2 [244 kB]
    Get:26 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libpulse-mainloop-glib0 amd64 1:11.1-1ubuntu7.11 [22.1 kB]
    Get:27 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libpulse-dev amd64 1:11.1-1ubuntu7.11 [81.5 kB]
    Get:28 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libsndio-dev amd64 1.1.0-3 [13.3 kB]
    Get:29 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libudev-dev amd64 237-3ubuntu10.47 [19.1 kB]
    Get:30 http://archive.ubuntu.com/ubuntu bionic/main amd64 libxcursor-dev amd64 1:1.1.15-1 [26.5 kB]
    Get:31 http://archive.ubuntu.com/ubuntu bionic/main amd64 x11proto-xinerama-dev all 2018.4-4 [2,628 B]
    Get:32 http://archive.ubuntu.com/ubuntu bionic/main amd64 libxinerama-dev amd64 2:1.1.3-1 [8,404 B]
    Get:33 http://archive.ubuntu.com/ubuntu bionic/main amd64 x11proto-randr-dev all 2018.4-4 [2,620 B]
    Get:34 http://archive.ubuntu.com/ubuntu bionic/main amd64 libxrandr-dev amd64 2:1.5.1-1 [24.0 kB]
    Get:35 http://archive.ubuntu.com/ubuntu bionic/main amd64 libxv-dev amd64 2:1.0.11-1 [32.5 kB]
    Get:36 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 libsdl2-dev amd64 2.0.8+dfsg1-1ubuntu1.18.04.4 [683 kB]
    Get:37 http://archive.ubuntu.com/ubuntu bionic/main amd64 libxpm-dev amd64 1:3.5.12-1 [87.4 kB]
    Get:38 http://archive.ubuntu.com/ubuntu bionic/main amd64 libxaw7-dev amd64 2:1.0.13-1 [231 kB]
    Get:39 http://archive.ubuntu.com/ubuntu bionic/main amd64 x11proto-composite-dev all 1:2018.4-4 [2,620 B]
    Get:40 http://archive.ubuntu.com/ubuntu bionic/main amd64 libxcomposite-dev amd64 1:0.4.4-2 [9,136 B]
    Get:41 http://archive.ubuntu.com/ubuntu bionic/main amd64 libxfont-dev amd64 1:2.0.3-1 [118 kB]
    Get:42 http://archive.ubuntu.com/ubuntu bionic/main amd64 libxkbfile-dev amd64 1:1.0.9-2 [74.3 kB]
    Get:43 http://archive.ubuntu.com/ubuntu bionic/main amd64 libxmuu-dev amd64 2:1.1.2-2 [7,056 B]
    Get:44 http://archive.ubuntu.com/ubuntu bionic/main amd64 libxres1 amd64 2:1.2.0-2 [7,716 B]
    Get:45 http://archive.ubuntu.com/ubuntu bionic/main amd64 x11proto-resource-dev all 2018.4-4 [2,620 B]
    Get:46 http://archive.ubuntu.com/ubuntu bionic/main amd64 libxres-dev amd64 2:1.2.0-2 [8,136 B]
    Get:47 http://archive.ubuntu.com/ubuntu bionic/main amd64 x11proto-record-dev all 2018.4-4 [2,620 B]
    Get:48 http://archive.ubuntu.com/ubuntu bionic/main amd64 libxtst-dev amd64 2:1.2.3-1 [15.2 kB]
    Get:49 http://archive.ubuntu.com/ubuntu bionic/main amd64 libxvmc1 amd64 2:1.0.10-1 [13.7 kB]
    Get:50 http://archive.ubuntu.com/ubuntu bionic/main amd64 libxvmc-dev amd64 2:1.0.10-1 [21.3 kB]
    Get:51 http://archive.ubuntu.com/ubuntu bionic/main amd64 x11proto-xf86dga-dev all 2018.4-4 [2,624 B]
    Get:52 http://archive.ubuntu.com/ubuntu bionic/main amd64 libxxf86dga-dev amd64 2:1.1.4-1 [17.6 kB]
    Get:53 http://archive.ubuntu.com/ubuntu bionic/main amd64 mir-client-platform-mesa-dev amd64 0.31.1-0ubuntu1 [11.0 kB]
    Get:54 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-opengl all 3.1.0+dfsg-1 [496 kB]
    Get:55 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-future all 0.15.2-4ubuntu2 [333 kB]
    Get:56 http://archive.ubuntu.com/ubuntu bionic/universe amd64 swig3.0 amd64 3.0.12-1 [1,094 kB]
    Get:57 http://archive.ubuntu.com/ubuntu bionic/universe amd64 swig amd64 3.0.12-1 [6,460 B]
    Get:58 http://archive.ubuntu.com/ubuntu bionic/main amd64 x11-utils amd64 7.7+3build1 [196 kB]
    Get:59 http://archive.ubuntu.com/ubuntu bionic/main amd64 x11proto-dri2-dev all 2018.4-4 [2,620 B]
    Get:60 http://archive.ubuntu.com/ubuntu bionic/main amd64 x11proto-gl-dev all 2018.4-4 [2,612 B]
    Get:61 http://archive.ubuntu.com/ubuntu bionic/main amd64 x11proto-render-dev all 2:2018.4-4 [2,620 B]
    Get:62 http://archive.ubuntu.com/ubuntu bionic/main amd64 x11proto-xf86bigfont-dev all 2018.4-4 [2,628 B]
    Get:63 http://archive.ubuntu.com/ubuntu bionic/main amd64 libpciaccess-dev amd64 0.14-1 [20.2 kB]
    Get:64 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 xserver-xorg-dev amd64 2:1.19.6-1ubuntu4.9 [198 kB]
    Get:65 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 xorg-dev all 1:7.7+19ubuntu7.1 [4,300 B]
    Get:66 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 xvfb amd64 2:1.19.6-1ubuntu4.9 [784 kB]
    Fetched 7,968 kB in 2s (4,067 kB/s)
    Extracting templates from packages: 100%
    Selecting previously unselected package libdmx1:amd64.
    (Reading database ... 160706 files and directories currently installed.)
    Preparing to unpack .../00-libdmx1_1%3a1.1.3-1_amd64.deb ...
    Unpacking libdmx1:amd64 (1:1.1.3-1) ...
    Selecting previously unselected package libxxf86dga1:amd64.
    Preparing to unpack .../01-libxxf86dga1_2%3a1.1.4-1_amd64.deb ...
    Unpacking libxxf86dga1:amd64 (2:1.1.4-1) ...
    Selecting previously unselected package libibus-1.0-5:amd64.
    Preparing to unpack .../02-libibus-1.0-5_1.5.17-3ubuntu5.3_amd64.deb ...
    Unpacking libibus-1.0-5:amd64 (1.5.17-3ubuntu5.3) ...
    Selecting previously unselected package gir1.2-ibus-1.0:amd64.
    Preparing to unpack .../03-gir1.2-ibus-1.0_1.5.17-3ubuntu5.3_amd64.deb ...
    Unpacking gir1.2-ibus-1.0:amd64 (1.5.17-3ubuntu5.3) ...
    Selecting previously unselected package libcapnp-0.6.1:amd64.
    Preparing to unpack .../04-libcapnp-0.6.1_0.6.1-1ubuntu1_amd64.deb ...
    Unpacking libcapnp-0.6.1:amd64 (0.6.1-1ubuntu1) ...
    Selecting previously unselected package libdbus-1-dev:amd64.
    Preparing to unpack .../05-libdbus-1-dev_1.12.2-1ubuntu1.2_amd64.deb ...
    Unpacking libdbus-1-dev:amd64 (1.12.2-1ubuntu1.2) ...
    Selecting previously unselected package libdmx-dev:amd64.
    Preparing to unpack .../06-libdmx-dev_1%3a1.1.3-1_amd64.deb ...
    Unpacking libdmx-dev:amd64 (1:1.1.3-1) ...
    Selecting previously unselected package libfontenc-dev:amd64.
    Preparing to unpack .../07-libfontenc-dev_1%3a1.1.3-1_amd64.deb ...
    Unpacking libfontenc-dev:amd64 (1:1.1.3-1) ...
    Selecting previously unselected package libfs6:amd64.
    Preparing to unpack .../08-libfs6_2%3a1.0.7-1_amd64.deb ...
    Unpacking libfs6:amd64 (2:1.0.7-1) ...
    Selecting previously unselected package x11proto-fonts-dev.
    Preparing to unpack .../09-x11proto-fonts-dev_2018.4-4_all.deb ...
    Unpacking x11proto-fonts-dev (2018.4-4) ...
    Selecting previously unselected package libfs-dev:amd64.
    Preparing to unpack .../10-libfs-dev_2%3a1.0.7-1_amd64.deb ...
    Unpacking libfs-dev:amd64 (2:1.0.7-1) ...
    Selecting previously unselected package libibus-1.0-dev:amd64.
    Preparing to unpack .../11-libibus-1.0-dev_1.5.17-3ubuntu5.3_amd64.deb ...
    Unpacking libibus-1.0-dev:amd64 (1.5.17-3ubuntu5.3) ...
    Selecting previously unselected package libmircore1:amd64.
    Preparing to unpack .../12-libmircore1_0.31.1-0ubuntu1_amd64.deb ...
    Unpacking libmircore1:amd64 (0.31.1-0ubuntu1) ...
    Selecting previously unselected package libmircommon7:amd64.
    Preparing to unpack .../13-libmircommon7_0.31.1-0ubuntu1_amd64.deb ...
    Unpacking libmircommon7:amd64 (0.31.1-0ubuntu1) ...
    Selecting previously unselected package libprotobuf-lite10:amd64.
    Preparing to unpack .../14-libprotobuf-lite10_3.0.0-9.1ubuntu1_amd64.deb ...
    Unpacking libprotobuf-lite10:amd64 (3.0.0-9.1ubuntu1) ...
    Selecting previously unselected package libmirprotobuf3:amd64.
    Preparing to unpack .../15-libmirprotobuf3_0.31.1-0ubuntu1_amd64.deb ...
    Unpacking libmirprotobuf3:amd64 (0.31.1-0ubuntu1) ...
    Selecting previously unselected package libmirclient9:amd64.
    Preparing to unpack .../16-libmirclient9_0.31.1-0ubuntu1_amd64.deb ...
    Unpacking libmirclient9:amd64 (0.31.1-0ubuntu1) ...
    Selecting previously unselected package libmircore-dev:amd64.
    Preparing to unpack .../17-libmircore-dev_0.31.1-0ubuntu1_amd64.deb ...
    Unpacking libmircore-dev:amd64 (0.31.1-0ubuntu1) ...
    Selecting previously unselected package libprotobuf-dev:amd64.
    Preparing to unpack .../18-libprotobuf-dev_3.0.0-9.1ubuntu1_amd64.deb ...
    Unpacking libprotobuf-dev:amd64 (3.0.0-9.1ubuntu1) ...
    Selecting previously unselected package libxkbcommon-dev:amd64.
    Preparing to unpack .../19-libxkbcommon-dev_0.8.2-1~ubuntu18.04.1_amd64.deb ...
    Unpacking libxkbcommon-dev:amd64 (0.8.2-1~ubuntu18.04.1) ...
    Selecting previously unselected package libmircommon-dev:amd64.
    Preparing to unpack .../20-libmircommon-dev_0.31.1-0ubuntu1_amd64.deb ...
    Unpacking libmircommon-dev:amd64 (0.31.1-0ubuntu1) ...
    Selecting previously unselected package libmircookie2:amd64.
    Preparing to unpack .../21-libmircookie2_0.31.1-0ubuntu1_amd64.deb ...
    Unpacking libmircookie2:amd64 (0.31.1-0ubuntu1) ...
    Selecting previously unselected package libmircookie-dev:amd64.
    Preparing to unpack .../22-libmircookie-dev_0.31.1-0ubuntu1_amd64.deb ...
    Unpacking libmircookie-dev:amd64 (0.31.1-0ubuntu1) ...
    Selecting previously unselected package libmirclient-dev:amd64.
    Preparing to unpack .../23-libmirclient-dev_0.31.1-0ubuntu1_amd64.deb ...
    Unpacking libmirclient-dev:amd64 (0.31.1-0ubuntu1) ...
    Selecting previously unselected package libpixman-1-dev:amd64.
    Preparing to unpack .../24-libpixman-1-dev_0.34.0-2_amd64.deb ...
    Unpacking libpixman-1-dev:amd64 (0.34.0-2) ...
    Selecting previously unselected package libpulse-mainloop-glib0:amd64.
    Preparing to unpack .../25-libpulse-mainloop-glib0_1%3a11.1-1ubuntu7.11_amd64.deb ...
    Unpacking libpulse-mainloop-glib0:amd64 (1:11.1-1ubuntu7.11) ...
    Selecting previously unselected package libpulse-dev:amd64.
    Preparing to unpack .../26-libpulse-dev_1%3a11.1-1ubuntu7.11_amd64.deb ...
    Unpacking libpulse-dev:amd64 (1:11.1-1ubuntu7.11) ...
    Selecting previously unselected package libsndio-dev:amd64.
    Preparing to unpack .../27-libsndio-dev_1.1.0-3_amd64.deb ...
    Unpacking libsndio-dev:amd64 (1.1.0-3) ...
    Selecting previously unselected package libudev-dev:amd64.
    Preparing to unpack .../28-libudev-dev_237-3ubuntu10.47_amd64.deb ...
    Unpacking libudev-dev:amd64 (237-3ubuntu10.47) ...
    Selecting previously unselected package libxcursor-dev:amd64.
    Preparing to unpack .../29-libxcursor-dev_1%3a1.1.15-1_amd64.deb ...
    Unpacking libxcursor-dev:amd64 (1:1.1.15-1) ...
    Selecting previously unselected package x11proto-xinerama-dev.
    Preparing to unpack .../30-x11proto-xinerama-dev_2018.4-4_all.deb ...
    Unpacking x11proto-xinerama-dev (2018.4-4) ...
    Selecting previously unselected package libxinerama-dev:amd64.
    Preparing to unpack .../31-libxinerama-dev_2%3a1.1.3-1_amd64.deb ...
    Unpacking libxinerama-dev:amd64 (2:1.1.3-1) ...
    Selecting previously unselected package x11proto-randr-dev.
    Preparing to unpack .../32-x11proto-randr-dev_2018.4-4_all.deb ...
    Unpacking x11proto-randr-dev (2018.4-4) ...
    Selecting previously unselected package libxrandr-dev:amd64.
    Preparing to unpack .../33-libxrandr-dev_2%3a1.5.1-1_amd64.deb ...
    Unpacking libxrandr-dev:amd64 (2:1.5.1-1) ...
    Selecting previously unselected package libxv-dev:amd64.
    Preparing to unpack .../34-libxv-dev_2%3a1.0.11-1_amd64.deb ...
    Unpacking libxv-dev:amd64 (2:1.0.11-1) ...
    Selecting previously unselected package libsdl2-dev:amd64.
    Preparing to unpack .../35-libsdl2-dev_2.0.8+dfsg1-1ubuntu1.18.04.4_amd64.deb ...
    Unpacking libsdl2-dev:amd64 (2.0.8+dfsg1-1ubuntu1.18.04.4) ...
    Selecting previously unselected package libxpm-dev:amd64.
    Preparing to unpack .../36-libxpm-dev_1%3a3.5.12-1_amd64.deb ...
    Unpacking libxpm-dev:amd64 (1:3.5.12-1) ...
    Selecting previously unselected package libxaw7-dev:amd64.
    Preparing to unpack .../37-libxaw7-dev_2%3a1.0.13-1_amd64.deb ...
    Unpacking libxaw7-dev:amd64 (2:1.0.13-1) ...
    Selecting previously unselected package x11proto-composite-dev.
    Preparing to unpack .../38-x11proto-composite-dev_1%3a2018.4-4_all.deb ...
    Unpacking x11proto-composite-dev (1:2018.4-4) ...
    Selecting previously unselected package libxcomposite-dev:amd64.
    Preparing to unpack .../39-libxcomposite-dev_1%3a0.4.4-2_amd64.deb ...
    Unpacking libxcomposite-dev:amd64 (1:0.4.4-2) ...
    Selecting previously unselected package libxfont-dev.
    Preparing to unpack .../40-libxfont-dev_1%3a2.0.3-1_amd64.deb ...
    Unpacking libxfont-dev (1:2.0.3-1) ...
    Selecting previously unselected package libxkbfile-dev:amd64.
    Preparing to unpack .../41-libxkbfile-dev_1%3a1.0.9-2_amd64.deb ...
    Unpacking libxkbfile-dev:amd64 (1:1.0.9-2) ...
    Selecting previously unselected package libxmuu-dev:amd64.
    Preparing to unpack .../42-libxmuu-dev_2%3a1.1.2-2_amd64.deb ...
    Unpacking libxmuu-dev:amd64 (2:1.1.2-2) ...
    Selecting previously unselected package libxres1:amd64.
    Preparing to unpack .../43-libxres1_2%3a1.2.0-2_amd64.deb ...
    Unpacking libxres1:amd64 (2:1.2.0-2) ...
    Selecting previously unselected package x11proto-resource-dev.
    Preparing to unpack .../44-x11proto-resource-dev_2018.4-4_all.deb ...
    Unpacking x11proto-resource-dev (2018.4-4) ...
    Selecting previously unselected package libxres-dev.
    Preparing to unpack .../45-libxres-dev_2%3a1.2.0-2_amd64.deb ...
    Unpacking libxres-dev (2:1.2.0-2) ...
    Selecting previously unselected package x11proto-record-dev.
    Preparing to unpack .../46-x11proto-record-dev_2018.4-4_all.deb ...
    Unpacking x11proto-record-dev (2018.4-4) ...
    Selecting previously unselected package libxtst-dev:amd64.
    Preparing to unpack .../47-libxtst-dev_2%3a1.2.3-1_amd64.deb ...
    Unpacking libxtst-dev:amd64 (2:1.2.3-1) ...
    Selecting previously unselected package libxvmc1:amd64.
    Preparing to unpack .../48-libxvmc1_2%3a1.0.10-1_amd64.deb ...
    Unpacking libxvmc1:amd64 (2:1.0.10-1) ...
    Selecting previously unselected package libxvmc-dev:amd64.
    Preparing to unpack .../49-libxvmc-dev_2%3a1.0.10-1_amd64.deb ...
    Unpacking libxvmc-dev:amd64 (2:1.0.10-1) ...
    Selecting previously unselected package x11proto-xf86dga-dev.
    Preparing to unpack .../50-x11proto-xf86dga-dev_2018.4-4_all.deb ...
    Unpacking x11proto-xf86dga-dev (2018.4-4) ...
    Selecting previously unselected package libxxf86dga-dev:amd64.
    Preparing to unpack .../51-libxxf86dga-dev_2%3a1.1.4-1_amd64.deb ...
    Unpacking libxxf86dga-dev:amd64 (2:1.1.4-1) ...
    Selecting previously unselected package mir-client-platform-mesa-dev:amd64.
    Preparing to unpack .../52-mir-client-platform-mesa-dev_0.31.1-0ubuntu1_amd64.deb ...
    Unpacking mir-client-platform-mesa-dev:amd64 (0.31.1-0ubuntu1) ...
    Selecting previously unselected package python-opengl.
    Preparing to unpack .../53-python-opengl_3.1.0+dfsg-1_all.deb ...
    Unpacking python-opengl (3.1.0+dfsg-1) ...
    Selecting previously unselected package python3-future.
    Preparing to unpack .../54-python3-future_0.15.2-4ubuntu2_all.deb ...
    Unpacking python3-future (0.15.2-4ubuntu2) ...
    Selecting previously unselected package swig3.0.
    Preparing to unpack .../55-swig3.0_3.0.12-1_amd64.deb ...
    Unpacking swig3.0 (3.0.12-1) ...
    Selecting previously unselected package swig.
    Preparing to unpack .../56-swig_3.0.12-1_amd64.deb ...
    Unpacking swig (3.0.12-1) ...
    Selecting previously unselected package x11-utils.
    Preparing to unpack .../57-x11-utils_7.7+3build1_amd64.deb ...
    Unpacking x11-utils (7.7+3build1) ...
    Selecting previously unselected package x11proto-dri2-dev.
    Preparing to unpack .../58-x11proto-dri2-dev_2018.4-4_all.deb ...
    Unpacking x11proto-dri2-dev (2018.4-4) ...
    Selecting previously unselected package x11proto-gl-dev.
    Preparing to unpack .../59-x11proto-gl-dev_2018.4-4_all.deb ...
    Unpacking x11proto-gl-dev (2018.4-4) ...
    Selecting previously unselected package x11proto-render-dev.
    Preparing to unpack .../60-x11proto-render-dev_2%3a2018.4-4_all.deb ...
    Unpacking x11proto-render-dev (2:2018.4-4) ...
    Selecting previously unselected package x11proto-xf86bigfont-dev.
    Preparing to unpack .../61-x11proto-xf86bigfont-dev_2018.4-4_all.deb ...
    Unpacking x11proto-xf86bigfont-dev (2018.4-4) ...
    Selecting previously unselected package libpciaccess-dev:amd64.
    Preparing to unpack .../62-libpciaccess-dev_0.14-1_amd64.deb ...
    Unpacking libpciaccess-dev:amd64 (0.14-1) ...
    Selecting previously unselected package xserver-xorg-dev.
    Preparing to unpack .../63-xserver-xorg-dev_2%3a1.19.6-1ubuntu4.9_amd64.deb ...
    Unpacking xserver-xorg-dev (2:1.19.6-1ubuntu4.9) ...
    Selecting previously unselected package xorg-dev.
    Preparing to unpack .../64-xorg-dev_1%3a7.7+19ubuntu7.1_all.deb ...
    Unpacking xorg-dev (1:7.7+19ubuntu7.1) ...
    Selecting previously unselected package xvfb.
    Preparing to unpack .../65-xvfb_2%3a1.19.6-1ubuntu4.9_amd64.deb ...
    Unpacking xvfb (2:1.19.6-1ubuntu4.9) ...
    Setting up x11proto-fonts-dev (2018.4-4) ...
    Setting up x11proto-dri2-dev (2018.4-4) ...
    Setting up libdbus-1-dev:amd64 (1.12.2-1ubuntu1.2) ...
    Setting up libxcursor-dev:amd64 (1:1.1.15-1) ...
    Setting up libxkbcommon-dev:amd64 (0.8.2-1~ubuntu18.04.1) ...
    Setting up libpulse-mainloop-glib0:amd64 (1:11.1-1ubuntu7.11) ...
    Setting up libpulse-dev:amd64 (1:11.1-1ubuntu7.11) ...
    Setting up python-opengl (3.1.0+dfsg-1) ...
    Setting up swig3.0 (3.0.12-1) ...
    Setting up libpciaccess-dev:amd64 (0.14-1) ...
    Setting up libmircore-dev:amd64 (0.31.1-0ubuntu1) ...
    Setting up libxres1:amd64 (2:1.2.0-2) ...
    Setting up xvfb (2:1.19.6-1ubuntu4.9) ...
    Setting up libxpm-dev:amd64 (1:3.5.12-1) ...
    Setting up libsndio-dev:amd64 (1.1.0-3) ...
    Setting up libxres-dev (2:1.2.0-2) ...
    Setting up x11proto-xf86bigfont-dev (2018.4-4) ...
    Setting up libxkbfile-dev:amd64 (1:1.0.9-2) ...
    Setting up libxvmc1:amd64 (2:1.0.10-1) ...
    Setting up libxmuu-dev:amd64 (2:1.1.2-2) ...
    Setting up x11proto-record-dev (2018.4-4) ...
    Setting up libxtst-dev:amd64 (2:1.2.3-1) ...
    Setting up libmircookie2:amd64 (0.31.1-0ubuntu1) ...
    Setting up libfontenc-dev:amd64 (1:1.1.3-1) ...
    Setting up python3-future (0.15.2-4ubuntu2) ...
    update-alternatives: using /usr/bin/python3-futurize to provide /usr/bin/futurize (futurize) in auto mode
    update-alternatives: using /usr/bin/python3-pasteurize to provide /usr/bin/pasteurize (pasteurize) in auto mode
    Setting up libxxf86dga1:amd64 (2:1.1.4-1) ...
    Setting up x11proto-xinerama-dev (2018.4-4) ...
    Setting up libpixman-1-dev:amd64 (0.34.0-2) ...
    Setting up x11proto-randr-dev (2018.4-4) ...
    Setting up libxinerama-dev:amd64 (2:1.1.3-1) ...
    Setting up x11proto-gl-dev (2018.4-4) ...
    Setting up libxv-dev:amd64 (2:1.0.11-1) ...
    Setting up libcapnp-0.6.1:amd64 (0.6.1-1ubuntu1) ...
    Setting up x11proto-resource-dev (2018.4-4) ...
    Setting up x11proto-xf86dga-dev (2018.4-4) ...
    Setting up libfs6:amd64 (2:1.0.7-1) ...
    Setting up libxxf86dga-dev:amd64 (2:1.1.4-1) ...
    Setting up libibus-1.0-5:amd64 (1.5.17-3ubuntu5.3) ...
    Setting up libdmx1:amd64 (1:1.1.3-1) ...
    Setting up x11proto-render-dev (2:2018.4-4) ...
    Setting up libmircore1:amd64 (0.31.1-0ubuntu1) ...
    Setting up libprotobuf-lite10:amd64 (3.0.0-9.1ubuntu1) ...
    Setting up libudev-dev:amd64 (237-3ubuntu10.47) ...
    Setting up x11proto-composite-dev (1:2018.4-4) ...
    Setting up gir1.2-ibus-1.0:amd64 (1.5.17-3ubuntu5.3) ...
    Setting up swig (3.0.12-1) ...
    Setting up libxrandr-dev:amd64 (2:1.5.1-1) ...
    Setting up libxcomposite-dev:amd64 (1:0.4.4-2) ...
    Setting up libxaw7-dev:amd64 (2:1.0.13-1) ...
    Setting up libmirprotobuf3:amd64 (0.31.1-0ubuntu1) ...
    Setting up libdmx-dev:amd64 (1:1.1.3-1) ...
    Setting up libprotobuf-dev:amd64 (3.0.0-9.1ubuntu1) ...
    Setting up libxvmc-dev:amd64 (2:1.0.10-1) ...
    Setting up x11-utils (7.7+3build1) ...
    Setting up libmircookie-dev:amd64 (0.31.1-0ubuntu1) ...
    Setting up libxfont-dev (1:2.0.3-1) ...
    Setting up libibus-1.0-dev:amd64 (1.5.17-3ubuntu5.3) ...
    Setting up libfs-dev:amd64 (2:1.0.7-1) ...
    Setting up libmircommon7:amd64 (0.31.1-0ubuntu1) ...
    Setting up libmirclient9:amd64 (0.31.1-0ubuntu1) ...
    Setting up libmircommon-dev:amd64 (0.31.1-0ubuntu1) ...
    Setting up libmirclient-dev:amd64 (0.31.1-0ubuntu1) ...
    Setting up libsdl2-dev:amd64 (2.0.8+dfsg1-1ubuntu1.18.04.4) ...
    Setting up mir-client-platform-mesa-dev:amd64 (0.31.1-0ubuntu1) ...
    Setting up xserver-xorg-dev (2:1.19.6-1ubuntu4.9) ...
    Setting up xorg-dev (1:7.7+19ubuntu7.1) ...
    Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
    Processing triggers for libc-bin (2.27-3ubuntu1.2) ...
    /sbin/ldconfig.real: /usr/local/lib/python3.7/dist-packages/ideep4py/lib/libmkldnn.so.0 is not a symbolic link

    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    libglu1-mesa-dev is already the newest version (9.0.0-2.1build1).
    libglu1-mesa-dev set to manually installed.
    libxi-dev is already the newest version (2:1.7.9-1).
    libxi-dev set to manually installed.
    libxmu-dev is already the newest version (2:1.1.2-2).
    libxmu-dev set to manually installed.
    mesa-common-dev is already the newest version (20.0.8-0ubuntu1~18.04.1).
    mesa-common-dev set to manually installed.
    The following package was automatically installed and is no longer required:
      libnvidia-common-460
    Use 'apt autoremove' to remove it.
    The following additional packages will be installed:
      apport apport-symptoms libxatracker2 python3-apport python3-certifi
      python3-chardet python3-httplib2 python3-idna python3-pkg-resources
      python3-problem-report python3-requests python3-requests-unixsocket
      python3-six python3-systemd python3-urllib3 xserver-xorg-core
      xserver-xorg-input-all xserver-xorg-input-libinput xserver-xorg-input-wacom
      xserver-xorg-legacy xserver-xorg-video-all xserver-xorg-video-amdgpu
      xserver-xorg-video-ati xserver-xorg-video-fbdev xserver-xorg-video-intel
      xserver-xorg-video-nouveau xserver-xorg-video-qxl xserver-xorg-video-radeon
      xserver-xorg-video-vesa xserver-xorg-video-vmware
    Suggested packages:
      apport-gtk | apport-kde python3-launchpadlib python3-setuptools
      python3-cryptography python3-openssl python3-socks xfonts-100dpi
      | xfonts-75dpi xfonts-scalable xinput firmware-amd-graphics
      xserver-xorg-video-r128 xserver-xorg-video-mach64 firmware-misc-nonfree
    The following packages will be REMOVED:
      xserver-xorg-core-hwe-18.04
    The following NEW packages will be installed:
      apport apport-symptoms libxatracker2 python3-apport python3-certifi
      python3-chardet python3-httplib2 python3-idna python3-pkg-resources
      python3-problem-report python3-requests python3-requests-unixsocket
      python3-six python3-systemd python3-urllib3 xserver-xorg xserver-xorg-core
      xserver-xorg-input-all xserver-xorg-input-libinput xserver-xorg-input-wacom
      xserver-xorg-legacy xserver-xorg-video-all xserver-xorg-video-amdgpu
      xserver-xorg-video-ati xserver-xorg-video-fbdev xserver-xorg-video-intel
      xserver-xorg-video-nouveau xserver-xorg-video-qxl xserver-xorg-video-radeon
      xserver-xorg-video-vesa xserver-xorg-video-vmware
    0 upgraded, 31 newly installed, 1 to remove and 72 not upgraded.
    Need to get 5,151 kB of archives.
    After this operation, 16.4 MB of additional disk space will be used.
    Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 xserver-xorg-core amd64 2:1.19.6-1ubuntu4.9 [1,351 kB]
    Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.3 [28.8 kB]
    Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-problem-report all 2.20.9-0ubuntu7.23 [9,900 B]
    Get:4 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-certifi all 2018.1.18-2 [144 kB]
    Get:5 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pkg-resources all 39.0.1-2 [98.8 kB]
    Get:6 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-chardet all 3.0.4-1 [80.3 kB]
    Get:7 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-idna all 2.6-1 [32.5 kB]
    Get:8 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-six all 1.11.0-2 [11.4 kB]
    Get:9 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-urllib3 all 1.22-1ubuntu0.18.04.2 [86.2 kB]
    Get:10 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-requests all 2.18.4-2ubuntu0.1 [58.3 kB]
    Get:11 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-requests-unixsocket all 0.1.5-3 [6,286 B]
    Get:12 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-apport all 2.20.9-0ubuntu7.23 [82.7 kB]
    Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apport all 2.20.9-0ubuntu7.23 [125 kB]
    Get:14 http://archive.ubuntu.com/ubuntu bionic/main amd64 apport-symptoms all 0.20 [14.2 kB]
    Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxatracker2 amd64 20.0.8-0ubuntu1~18.04.1 [1,549 kB]
    Get:16 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-systemd amd64 234-1build1 [36.1 kB]
    Get:17 http://archive.ubuntu.com/ubuntu bionic/main amd64 xserver-xorg-input-libinput amd64 0.27.1-1 [33.2 kB]
    Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 xserver-xorg-input-all amd64 1:7.7+19ubuntu7.1 [4,112 B]
    Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 xserver-xorg-input-wacom amd64 1:0.36.1-0ubuntu1.1 [89.8 kB]
    Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 xserver-xorg amd64 1:7.7+19ubuntu7.1 [65.2 kB]
    Get:21 http://archive.ubuntu.com/ubuntu bionic/main amd64 xserver-xorg-video-amdgpu amd64 18.0.1-1 [61.0 kB]
    Get:22 http://archive.ubuntu.com/ubuntu bionic/main amd64 xserver-xorg-video-radeon amd64 1:18.0.1-1 [147 kB]
    Get:23 http://archive.ubuntu.com/ubuntu bionic/main amd64 xserver-xorg-video-ati amd64 1:18.0.1-1 [7,116 B]
    Get:24 http://archive.ubuntu.com/ubuntu bionic/main amd64 xserver-xorg-video-fbdev amd64 1:0.4.4-1build6 [11.5 kB]
    Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 xserver-xorg-video-nouveau amd64 1:1.0.15-2 [86.3 kB]
    Get:26 http://archive.ubuntu.com/ubuntu bionic/main amd64 xserver-xorg-video-vesa amd64 1:2.3.4-1build3 [14.7 kB]
    Get:27 http://archive.ubuntu.com/ubuntu bionic/main amd64 xserver-xorg-video-vmware amd64 1:13.2.1-1build1 [71.6 kB]
    Get:28 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 xserver-xorg-video-all amd64 1:7.7+19ubuntu7.1 [4,164 B]
    Get:29 http://archive.ubuntu.com/ubuntu bionic/main amd64 xserver-xorg-video-intel amd64 2:2.99.917+git20171229-1 [728 kB]
    Get:30 http://archive.ubuntu.com/ubuntu bionic/main amd64 xserver-xorg-video-qxl amd64 0.1.5-2build1 [81.0 kB]
    Get:31 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 xserver-xorg-legacy amd64 2:1.19.6-1ubuntu4.9 [32.3 kB]
    Fetched 5,151 kB in 1s (3,959 kB/s)
    Extracting templates from packages: 100%
    Preconfiguring packages ...
    dpkg: xserver-xorg-core-hwe-18.04: dependency problems, but removing anyway as you requested:
     xserver-xorg-video-nvidia-465 depends on xorg-video-abi-24 | xorg-video-abi-23 | xorg-video-abi-20 | xorg-video-abi-19 | xorg-video-abi-18 | xorg-video-abi-15 | xorg-video-abi-14 | xorg-video-abi-13 | xorg-video-abi-12 | xorg-video-abi-11 | xorg-video-abi-10 | xorg-video-abi-8 | xorg-video-abi-6.0; however:
      Package xorg-video-abi-24 is not installed.
      Package xserver-xorg-core-hwe-18.04 which provides xorg-video-abi-24 is to be removed.
      Package xorg-video-abi-23 is not installed.
      Package xorg-video-abi-20 is not installed.
      Package xorg-video-abi-19 is not installed.
      Package xorg-video-abi-18 is not installed.
      Package xorg-video-abi-15 is not installed.
      Package xorg-video-abi-14 is not installed.
      Package xorg-video-abi-13 is not installed.
      Package xorg-video-abi-12 is not installed.
      Package xorg-video-abi-11 is not installed.
      Package xorg-video-abi-10 is not installed.
      Package xorg-video-abi-8 is not installed.
      Package xorg-video-abi-6.0 is not installed.
     xserver-xorg-video-nvidia-465 depends on xserver-xorg-core (>= 2:1.19.6-1ubuntu2~) | xserver-xorg-core-hwe-18.04; however:
      Package xserver-xorg-core is not installed.
      Package xserver-xorg-core-hwe-18.04 which provides xserver-xorg-core is to be removed.
      Package xserver-xorg-core-hwe-18.04 is to be removed.
     xserver-xorg-video-nvidia-465 depends on xorg-video-abi-24 | xorg-video-abi-23 | xorg-video-abi-20 | xorg-video-abi-19 | xorg-video-abi-18 | xorg-video-abi-15 | xorg-video-abi-14 | xorg-video-abi-13 | xorg-video-abi-12 | xorg-video-abi-11 | xorg-video-abi-10 | xorg-video-abi-8 | xorg-video-abi-6.0; however:
      Package xorg-video-abi-24 is not installed.
      Package xserver-xorg-core-hwe-18.04 which provides xorg-video-abi-24 is to be removed.
      Package xorg-video-abi-23 is not installed.
      Package xorg-video-abi-20 is not installed.
      Package xorg-video-abi-19 is not installed.
      Package xorg-video-abi-18 is not installed.
      Package xorg-video-abi-15 is not installed.
      Package xorg-video-abi-14 is not installed.
      Package xorg-video-abi-13 is not installed.
      Package xorg-video-abi-12 is not installed.
      Package xorg-video-abi-11 is not installed.
      Package xorg-video-abi-10 is not installed.
      Package xorg-video-abi-8 is not installed.
      Package xorg-video-abi-6.0 is not installed.
     xserver-xorg-video-nvidia-465 depends on xserver-xorg-core (>= 2:1.19.6-1ubuntu2~) | xserver-xorg-core-hwe-18.04; however:
      Package xserver-xorg-core is not installed.
      Package xserver-xorg-core-hwe-18.04 which provides xserver-xorg-core is to be removed.
      Package xserver-xorg-core-hwe-18.04 is to be removed.
     xserver-xorg-video-nvidia-465 depends on xorg-video-abi-24 | xorg-video-abi-23 | xorg-video-abi-20 | xorg-video-abi-19 | xorg-video-abi-18 | xorg-video-abi-15 | xorg-video-abi-14 | xorg-video-abi-13 | xorg-video-abi-12 | xorg-video-abi-11 | xorg-video-abi-10 | xorg-video-abi-8 | xorg-video-abi-6.0; however:
      Package xorg-video-abi-24 is not installed.
      Package xserver-xorg-core-hwe-18.04 which provides xorg-video-abi-24 is to be removed.
      Package xorg-video-abi-23 is not installed.
      Package xorg-video-abi-20 is not installed.
      Package xorg-video-abi-19 is not installed.
      Package xorg-video-abi-18 is not installed.
      Package xorg-video-abi-15 is not installed.
      Package xorg-video-abi-14 is not installed.
      Package xorg-video-abi-13 is not installed.
      Package xorg-video-abi-12 is not installed.
      Package xorg-video-abi-11 is not installed.
      Package xorg-video-abi-10 is not installed.
      Package xorg-video-abi-8 is not installed.
      Package xorg-video-abi-6.0 is not installed.
     xserver-xorg-video-nvidia-465 depends on xserver-xorg-core (>= 2:1.19.6-1ubuntu2~) | xserver-xorg-core-hwe-18.04; however:
      Package xserver-xorg-core is not installed.
      Package xserver-xorg-core-hwe-18.04 which provides xserver-xorg-core is to be removed.
      Package xserver-xorg-core-hwe-18.04 is to be removed.

    (Reading database ... 165472 files and directories currently installed.)
    Removing xserver-xorg-core-hwe-18.04 (2:1.20.8-2ubuntu2.2~18.04.5) ...
    Selecting previously unselected package xserver-xorg-core.
    (Reading database ... 165433 files and directories currently installed.)
    Preparing to unpack .../00-xserver-xorg-core_2%3a1.19.6-1ubuntu4.9_amd64.deb ...
    Unpacking xserver-xorg-core (2:1.19.6-1ubuntu4.9) ...
    Selecting previously unselected package python3-httplib2.
    Preparing to unpack .../01-python3-httplib2_0.9.2+dfsg-1ubuntu0.3_all.deb ...
    Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.3) ...
    Selecting previously unselected package python3-problem-report.
    Preparing to unpack .../02-python3-problem-report_2.20.9-0ubuntu7.23_all.deb ...
    Unpacking python3-problem-report (2.20.9-0ubuntu7.23) ...
    Selecting previously unselected package python3-certifi.
    Preparing to unpack .../03-python3-certifi_2018.1.18-2_all.deb ...
    Unpacking python3-certifi (2018.1.18-2) ...
    Selecting previously unselected package python3-pkg-resources.
    Preparing to unpack .../04-python3-pkg-resources_39.0.1-2_all.deb ...
    Unpacking python3-pkg-resources (39.0.1-2) ...
    Selecting previously unselected package python3-chardet.
    Preparing to unpack .../05-python3-chardet_3.0.4-1_all.deb ...
    Unpacking python3-chardet (3.0.4-1) ...
    Selecting previously unselected package python3-idna.
    Preparing to unpack .../06-python3-idna_2.6-1_all.deb ...
    Unpacking python3-idna (2.6-1) ...
    Selecting previously unselected package python3-six.
    Preparing to unpack .../07-python3-six_1.11.0-2_all.deb ...
    Unpacking python3-six (1.11.0-2) ...
    Selecting previously unselected package python3-urllib3.
    Preparing to unpack .../08-python3-urllib3_1.22-1ubuntu0.18.04.2_all.deb ...
    Unpacking python3-urllib3 (1.22-1ubuntu0.18.04.2) ...
    Selecting previously unselected package python3-requests.
    Preparing to unpack .../09-python3-requests_2.18.4-2ubuntu0.1_all.deb ...
    Unpacking python3-requests (2.18.4-2ubuntu0.1) ...
    Selecting previously unselected package python3-requests-unixsocket.
    Preparing to unpack .../10-python3-requests-unixsocket_0.1.5-3_all.deb ...
    Unpacking python3-requests-unixsocket (0.1.5-3) ...
    Selecting previously unselected package python3-apport.
    Preparing to unpack .../11-python3-apport_2.20.9-0ubuntu7.23_all.deb ...
    Unpacking python3-apport (2.20.9-0ubuntu7.23) ...
    Selecting previously unselected package apport.
    Preparing to unpack .../12-apport_2.20.9-0ubuntu7.23_all.deb ...
    Unpacking apport (2.20.9-0ubuntu7.23) ...
    Selecting previously unselected package apport-symptoms.
    Preparing to unpack .../13-apport-symptoms_0.20_all.deb ...
    Unpacking apport-symptoms (0.20) ...
    Selecting previously unselected package libxatracker2:amd64.
    Preparing to unpack .../14-libxatracker2_20.0.8-0ubuntu1~18.04.1_amd64.deb ...
    Unpacking libxatracker2:amd64 (20.0.8-0ubuntu1~18.04.1) ...
    Selecting previously unselected package python3-systemd.
    Preparing to unpack .../15-python3-systemd_234-1build1_amd64.deb ...
    Unpacking python3-systemd (234-1build1) ...
    Selecting previously unselected package xserver-xorg-input-libinput.
    Preparing to unpack .../16-xserver-xorg-input-libinput_0.27.1-1_amd64.deb ...
    Unpacking xserver-xorg-input-libinput (0.27.1-1) ...
    Selecting previously unselected package xserver-xorg-input-all.
    Preparing to unpack .../17-xserver-xorg-input-all_1%3a7.7+19ubuntu7.1_amd64.deb ...
    Unpacking xserver-xorg-input-all (1:7.7+19ubuntu7.1) ...
    Selecting previously unselected package xserver-xorg-input-wacom.
    Preparing to unpack .../18-xserver-xorg-input-wacom_1%3a0.36.1-0ubuntu1.1_amd64.deb ...
    Unpacking xserver-xorg-input-wacom (1:0.36.1-0ubuntu1.1) ...
    Selecting previously unselected package xserver-xorg.
    Preparing to unpack .../19-xserver-xorg_1%3a7.7+19ubuntu7.1_amd64.deb ...
    Unpacking xserver-xorg (1:7.7+19ubuntu7.1) ...
    Selecting previously unselected package xserver-xorg-video-amdgpu.
    Preparing to unpack .../20-xserver-xorg-video-amdgpu_18.0.1-1_amd64.deb ...
    Unpacking xserver-xorg-video-amdgpu (18.0.1-1) ...
    Selecting previously unselected package xserver-xorg-video-radeon.
    Preparing to unpack .../21-xserver-xorg-video-radeon_1%3a18.0.1-1_amd64.deb ...
    Unpacking xserver-xorg-video-radeon (1:18.0.1-1) ...
    Selecting previously unselected package xserver-xorg-video-ati.
    Preparing to unpack .../22-xserver-xorg-video-ati_1%3a18.0.1-1_amd64.deb ...
    Unpacking xserver-xorg-video-ati (1:18.0.1-1) ...
    Selecting previously unselected package xserver-xorg-video-fbdev.
    Preparing to unpack .../23-xserver-xorg-video-fbdev_1%3a0.4.4-1build6_amd64.deb ...
    Unpacking xserver-xorg-video-fbdev (1:0.4.4-1build6) ...
    Selecting previously unselected package xserver-xorg-video-nouveau.
    Preparing to unpack .../24-xserver-xorg-video-nouveau_1%3a1.0.15-2_amd64.deb ...
    Unpacking xserver-xorg-video-nouveau (1:1.0.15-2) ...
    Selecting previously unselected package xserver-xorg-video-vesa.
    Preparing to unpack .../25-xserver-xorg-video-vesa_1%3a2.3.4-1build3_amd64.deb ...
    Unpacking xserver-xorg-video-vesa (1:2.3.4-1build3) ...
    Selecting previously unselected package xserver-xorg-video-vmware.
    Preparing to unpack .../26-xserver-xorg-video-vmware_1%3a13.2.1-1build1_amd64.deb ...
    Unpacking xserver-xorg-video-vmware (1:13.2.1-1build1) ...
    Selecting previously unselected package xserver-xorg-video-all.
    Preparing to unpack .../27-xserver-xorg-video-all_1%3a7.7+19ubuntu7.1_amd64.deb ...
    Unpacking xserver-xorg-video-all (1:7.7+19ubuntu7.1) ...
    Selecting previously unselected package xserver-xorg-video-intel.
    Preparing to unpack .../28-xserver-xorg-video-intel_2%3a2.99.917+git20171229-1_amd64.deb ...
    Unpacking xserver-xorg-video-intel (2:2.99.917+git20171229-1) ...
    Selecting previously unselected package xserver-xorg-video-qxl.
    Preparing to unpack .../29-xserver-xorg-video-qxl_0.1.5-2build1_amd64.deb ...
    Unpacking xserver-xorg-video-qxl (0.1.5-2build1) ...
    Selecting previously unselected package xserver-xorg-legacy.
    Preparing to unpack .../30-xserver-xorg-legacy_2%3a1.19.6-1ubuntu4.9_amd64.deb ...
    Unpacking xserver-xorg-legacy (2:1.19.6-1ubuntu4.9) ...
    Setting up python3-idna (2.6-1) ...
    Setting up python3-six (1.11.0-2) ...
    Setting up python3-certifi (2018.1.18-2) ...
    Setting up python3-pkg-resources (39.0.1-2) ...
    Setting up libxatracker2:amd64 (20.0.8-0ubuntu1~18.04.1) ...
    Setting up xserver-xorg-core (2:1.19.6-1ubuntu4.9) ...
    Setting up xserver-xorg-video-radeon (1:18.0.1-1) ...
    Setting up python3-problem-report (2.20.9-0ubuntu7.23) ...
    Setting up xserver-xorg-video-vesa (1:2.3.4-1build3) ...
    Setting up xserver-xorg-video-amdgpu (18.0.1-1) ...
    Setting up python3-chardet (3.0.4-1) ...

```
!pip3 install torch torchvision gym PyOpenGL piglet pyglet==1.3.2 pyvirtualdisplay
```

    Requirement already satisfied: torch in /usr/local/lib/python3.6/dist-packages (1.1.0)
    Requirement already satisfied: torchvision in /usr/local/lib/python3.6/dist-packages (0.3.0)
    Requirement already satisfied: gym in /usr/local/lib/python3.6/dist-packages (0.10.11)
    Requirement already satisfied: PyOpenGL in /usr/local/lib/python3.6/dist-packages (3.1.0)
    Collecting piglet
      Downloading https://files.pythonhosted.org/packages/86/f6/ef278239ebe525466ea51a7dd9d6d3211d197ac4b4abc76e17cdd419f69c/piglet-0.4.4.tar.gz (52kB)
         |████████████████████████████████| 61kB 4.1MB/s
    Collecting pyglet==1.3.2
      Downloading https://files.pythonhosted.org/packages/1c/fc/dad5eaaab68f0c21e2f906a94ddb98175662cc5a654eee404d59554ce0fa/pyglet-1.3.2-py2.py3-none-any.whl (1.0MB)
         |████████████████████████████████| 1.0MB 15.0MB/s
    Collecting pyvirtualdisplay
      Downloading https://files.pythonhosted.org/packages/cf/ad/b15f252bfb0f1693ad3150b55a44a674f3cba711cacdbb9ae2f03f143d19/PyVirtualDisplay-0.2.4-py2.py3-none-any.whl
    Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from torch) (1.16.4)
    Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.12.0)
    Requirement already satisfied: pillow>=4.1.1 in /usr/local/lib/python3.6/dist-packages (from torchvision) (4.3.0)
    Requirement already satisfied: requests>=2.0 in /usr/local/lib/python3.6/dist-packages (from gym) (2.21.0)
    Requirement already satisfied: scipy in /usr/local/lib/python3.6/dist-packages (from gym) (1.3.0)
    Collecting Parsley (from piglet)
      Downloading https://files.pythonhosted.org/packages/2b/d6/4fed8d65e28a970e1c5cb33ce9c7e22e3de745e1b2ae37af051ef16aea3b/Parsley-1.3-py2.py3-none-any.whl (88kB)
         |████████████████████████████████| 92kB 27.8MB/s
    Requirement already satisfied: attrs in /usr/local/lib/python3.6/dist-packages (from piglet) (19.1.0)
    Collecting astunparse (from piglet)
      Downloading https://files.pythonhosted.org/packages/2e/37/5dd0dd89b87bb5f0f32a7e775458412c52d78f230ab8d0c65df6aabc4479/astunparse-1.6.2-py2.py3-none-any.whl
    Requirement already satisfied: markupsafe in /usr/local/lib/python3.6/dist-packages (from piglet) (1.1.1)
    Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from pyglet==1.3.2) (0.16.0)
    Collecting EasyProcess (from pyvirtualdisplay)
      Downloading https://files.pythonhosted.org/packages/fa/29/40040d1d64a224a5e44df9572794a66494618ffe5c77199214aeceedb8a7/EasyProcess-0.2.7-py2.py3-none-any.whl
    Requirement already satisfied: olefile in /usr/local/lib/python3.6/dist-packages (from pillow>=4.1.1->torchvision) (0.46)
    Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests>=2.0->gym) (2.8)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests>=2.0->gym) (2019.6.16)
    Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests>=2.0->gym) (3.0.4)
    Requirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests>=2.0->gym) (1.24.3)
    Requirement already satisfied: wheel<1.0,>=0.23.0 in /usr/local/lib/python3.6/dist-packages (from astunparse->piglet) (0.33.4)
    Building wheels for collected packages: piglet
      Building wheel for piglet (setup.py) ... done
      Created wheel for piglet: filename=piglet-0.4.4-cp36-none-any.whl size=66439 sha256=d293a5e61bbc484a4c5bcf80fe81dc0fe5ff9b6495c6bc39a02ac88409874661
      Stored in directory: /root/.cache/pip/wheels/c7/59/a5/5bd1a35a4a4596714c4c7925a1751e7b1580b6ced363fd7969
    Successfully built piglet
    Installing collected packages: Parsley, astunparse, piglet, pyglet, EasyProcess, pyvirtualdisplay
      Found existing installation: pyglet 1.4.1
        Uninstalling pyglet-1.4.1:
          Successfully uninstalled pyglet-1.4.1
    Successfully installed EasyProcess-0.2.7 Parsley-1.3 astunparse-1.6.2 piglet-0.4.4 pyglet-1.3.2 pyvirtualdisplay-0.2.4

```
# Start virtual display
from pyvirtualdisplay import Display
display = Display(visible=0, size=(1024, 768))
display.start()
import os
os.environ["DISPLAY"] = ":" + str(display.display) + "." + str(display.screen)

```

    xdpyinfo was not found, X start can not be checked! Please install xdpyinfo!

```
%matplotlib inline
```

Reinforcement Learning (DQN) tutorial
=====================================
Adapted from PyTorch website.

**Author**: `Adam Paszke <https://github.com/apaszke>`_

This tutorial shows how to use PyTorch to train a Deep Q Learning (DQN) agent
on the CartPole-v0 task from the `OpenAI Gym <https://gym.openai.com/>`__.

**Task**

The agent has to decide between two actions - moving the cart left or
right - so that the pole attached to it stays upright. You can find an
official leaderboard with various algorithms and visualizations at the
`Gym website <https://gym.openai.com/envs/CartPole-v0>`__.

As the agent observes the current state of the environment and chooses
an action, the environment *transitions* to a new state, and also
returns a reward that indicates the consequences of the action. In this
task, the environment terminates if the pole falls over too far.

The CartPole task is designed so that the inputs to the agent are 4 real
values representing the environment state (position, velocity, etc.).
However, neural networks can solve the task purely by looking at the
scene, so we'll use a patch of the screen centered on the cart as an
input. Because of this, our results aren't directly comparable to the
ones from the official leaderboard - our task is much harder.
Unfortunately this does slow down the training, because we have to
render all the frames.

Strictly speaking, we will present the state as the difference between
the current screen patch and the previous one. This will allow the agent
to take the velocity of the pole into account from one image.

**Packages**

First, let's import needed packages. Firstly, we need
`gym <https://gym.openai.com/docs>`__ for the environment
(Install using `pip install gym`).
We'll also use the following from PyTorch:

-  neural networks (``torch.nn``)
-  optimization (``torch.optim``)
-  automatic differentiation (``torch.autograd``)
-  utilities for vision tasks (``torchvision`` - `a separate
   package <https://github.com/pytorch/vision>`__).

```
import gym
import math
import random
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
from collections import namedtuple
from itertools import count
from PIL import Image

import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
import torchvision.transforms as T

env = gym.make('CartPole-v0').unwrapped

# set up matplotlib
is_ipython = 'inline' in matplotlib.get_backend()
if is_ipython:
    from IPython import display

plt.ion()

# if gpu is to be used
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
```

Replay Memory
-------------

We'll be using experience replay memory for training our DQN. It stores
the transitions that the agent observes, allowing us to reuse this data
later. By sampling from it randomly, the transitions that build up a
batch are decorrelated. It has been shown that this greatly stabilizes
and improves the DQN training procedure.

For this, we're going to need two classses:

-  ``Transition`` - a named tuple representing a single transition in
   our environment
-  ``ReplayMemory`` - a cyclic buffer of bounded size that holds the
   transitions observed recently. It also implements a ``.sample()``
   method for selecting a random batch of transitions for training.

```
Transition = namedtuple('Transition',
                        ('state', 'action', 'next_state', 'reward'))

class ReplayMemory(object):

    def __init__(self, capacity):
        self.capacity = capacity
        self.memory = []
        self.position = 0

    def push(self, *args):
        """Saves a transition."""
        if len(self.memory) < self.capacity:
            self.memory.append(None)
        self.memory[self.position] = Transition(*args)
        self.position = (self.position + 1) % self.capacity

    def sample(self, batch_size):
        return random.sample(self.memory, batch_size)

    def __len__(self):
        return len(self.memory)
```

Now, let's define our model. But first, let quickly recap what a DQN is.

DQN algorithm
-------------

Our environment is deterministic, so all equations presented here are
also formulated deterministically for the sake of simplicity. In the
reinforcement learning literature, they would also contain expectations
over stochastic transitions in the environment.

Our aim will be to train a policy that tries to maximize the discounted,
cumulative reward
$R_{t_0} = \sum_{t=t_0}^{\infty} \gamma^{t - t_0} r_t$, where
$R_{t_0}$ is also known as the *return*. The discount,
$\gamma$, should be a constant between $0$ and $1$
that ensures the sum converges. It makes rewards from the uncertain far
future less important for our agent than the ones in the near future
that it can be fairly confident about.

The main idea behind Q-learning is that if we had a function
$Q^*: State \times Action \rightarrow \mathbb{R}$, that could tell
us what our return would be, if we were to take an action in a given
state, then we could easily construct a policy that maximizes our
rewards:

\begin{align}\pi^*(s) = \arg\!\max_a \ Q^*(s, a)\end{align}

However, we don't know everything about the world, so we don't have
access to $Q^*$. But, since neural networks are universal function
approximators, we can simply create one and train it to resemble
$Q^*$.

For our training update rule, we'll use a fact that every $Q$
function for some policy obeys the Bellman equation:

\begin{align}Q^{\pi}(s, a) = r + \gamma Q^{\pi}(s', \pi(s'))\end{align}

The difference between the two sides of the equality is known as the
temporal difference error, $\delta$:

\begin{align}\delta = Q(s, a) - (r + \gamma \max_a Q(s', a))\end{align}

To minimise this error, we will use the `Huber
loss <https://en.wikipedia.org/wiki/Huber_loss>`__. The Huber loss acts
like the mean squared error when the error is small, but like the mean
absolute error when the error is large - this makes it more robust to
outliers when the estimates of $Q$ are very noisy. We calculate
this over a batch of transitions, $B$, sampled from the replay
memory:

\begin{align}\mathcal{L} = \frac{1}{|B|}\sum_{(s, a, s', r) \ \in \ B} \mathcal{L}(\delta)\end{align}

\begin{align}\text{where} \quad \mathcal{L}(\delta) = \begin{cases}
     \frac{1}{2}{\delta^2}  & \text{for } |\delta| \le 1, \\
     |\delta| - \frac{1}{2} & \text{otherwise.}
   \end{cases}\end{align}

### Q-network

Our model will be a convolutional neural network that takes in the
difference between the current and previous screen patches. It has two
outputs, representing $Q(s, \mathrm{left})$ and
$Q(s, \mathrm{right})$ (where $s$ is the input to the
network). In effect, the network is trying to predict the *quality* of
taking each action given the current input.

```
class DQN(nn.Module):

    def __init__(self):
        super(DQN, self).__init__()
        self.conv1 = nn.Conv2d(3, 16, kernel_size=5, stride=2)
        self.bn1 = nn.BatchNorm2d(16)
        self.conv2 = nn.Conv2d(16, 32, kernel_size=5, stride=2)
        self.bn2 = nn.BatchNorm2d(32)
        self.conv3 = nn.Conv2d(32, 32, kernel_size=5, stride=2)
        self.bn3 = nn.BatchNorm2d(32)
        self.head = nn.Linear(448, 2)

    def forward(self, x):
        x = F.relu(self.bn1(self.conv1(x)))
        x = F.relu(self.bn2(self.conv2(x)))
        x = F.relu(self.bn3(self.conv3(x)))
        return self.head(x.view(x.size(0), -1))
```

### Input extraction

The code below are utilities for extracting and processing rendered
images from the environment. It uses the ``torchvision`` package, which
makes it easy to compose image transforms. Once you run the cell it will
display an example patch that it extracted.

```
resize = T.Compose([T.ToPILImage(),
                    T.Resize(40, interpolation=Image.CUBIC),
                    T.ToTensor()])

# This is based on the code from gym.
screen_width = 600

def get_cart_location():
    world_width = env.x_threshold * 2
    scale = screen_width / world_width
    return int(env.state[0] * scale + screen_width / 2.0)  # MIDDLE OF CART

def get_screen():
    screen = env.render(mode='rgb_array').transpose(
        (2, 0, 1))  # transpose into torch order (CHW)
    # Strip off the top and bottom of the screen
    screen = screen[:, 160:320]
    view_width = 320
    cart_location = get_cart_location()
    if cart_location < view_width // 2:
        slice_range = slice(view_width)
    elif cart_location > (screen_width - view_width // 2):
        slice_range = slice(-view_width, None)
    else:
        slice_range = slice(cart_location - view_width // 2,
                            cart_location + view_width // 2)
    # Strip off the edges, so that we have a square image centered on a cart
    screen = screen[:, :, slice_range]
    # Convert to float, rescare, convert to torch tensor
    # (this doesn't require a copy)
    screen = np.ascontiguousarray(screen, dtype=np.float32) / 255
    screen = torch.from_numpy(screen)
    # Resize, and add a batch dimension (BCHW)
    return resize(screen).unsqueeze(0).to(device)

env.reset()

plt.figure()
plt.imshow(get_screen().cpu().squeeze(0).permute(1, 2, 0).numpy(),
           interpolation='none')
plt.title('Example extracted screen')
plt.show()

```

Training
--------

### Hyperparameters and utilities

This cell instantiates our model and its optimizer, and defines some
utilities:

-  ``select_action`` - will select an action accordingly to an epsilon
   greedy policy. Simply put, we'll sometimes use our model for choosing
   the action, and sometimes we'll just sample one uniformly. The
   probability of choosing a random action will start at ``EPS_START``
   and will decay exponentially towards ``EPS_END``. ``EPS_DECAY``
   controls the rate of the decay.
-  ``plot_durations`` - a helper for plotting the durations of episodes,
   along with an average over the last 100 episodes (the measure used in
   the official evaluations). The plot will be underneath the cell
   containing the main training loop, and will update after every
   episode.

```
BATCH_SIZE = 128
GAMMA = 0.999
EPS_START = 0.9
EPS_END = 0.05
EPS_DECAY = 200
TARGET_UPDATE = 10

policy_net = DQN().to(device)
target_net = DQN().to(device)
target_net.load_state_dict(policy_net.state_dict())
target_net.eval()

optimizer = optim.RMSprop(policy_net.parameters())
memory = ReplayMemory(10000)

steps_done = 0

def select_action(state):
    global steps_done
    sample = random.random()
    eps_threshold = EPS_END + (EPS_START - EPS_END) * \
        math.exp(-1. * steps_done / EPS_DECAY)
    steps_done += 1
    if sample > eps_threshold:
        with torch.no_grad():
            return policy_net(state).max(1)[1].view(1, 1)
    else:
        return torch.tensor([[random.randrange(2)]], device=device, dtype=torch.long)

episode_durations = []

def plot_durations():
    plt.figure(2)
    plt.clf()
    durations_t = torch.tensor(episode_durations, dtype=torch.float)
    plt.title('Training...')
    plt.xlabel('Episode')
    plt.ylabel('Duration')
    plt.plot(durations_t.numpy())
    # Take 100 episode averages and plot them too
    if len(durations_t) >= 100:
        means = durations_t.unfold(0, 100, 1).mean(1).view(-1)
        means = torch.cat((torch.zeros(99), means))
        plt.plot(means.numpy())

    plt.pause(0.001)  # pause a bit so that plots are updated
    if is_ipython:
        display.clear_output(wait=True)
        display.display(plt.gcf())
```

### Training loop

Finally, the code for training our model.

Here, you can find an ``optimize_model`` function that performs a
single step of the optimization. It first samples a batch, concatenates
all the tensors into a single one, computes $Q(s_t, a_t)$ and
$V(s_{t+1}) = \max_a Q(s_{t+1}, a)$, and combines them into our
loss. By defition we set $V(s) = 0$ if $s$ is a terminal
state. We also use a target network to compute $V(s_{t+1})$ for
added stability. The target network has its weights kept frozen most of
the time, but is updated with the policy network's weights every so often.
This is usually a set number of steps but we shall use episodes for
simplicity.

```
def optimize_model():
    if len(memory) < BATCH_SIZE:
        return
    transitions = memory.sample(BATCH_SIZE)
    # Transpose the batch (see http://stackoverflow.com/a/19343/3343043 for
    # detailed explanation).
    batch = Transition(*zip(*transitions))

    # Compute a mask of non-final states and concatenate the batch elements
    non_final_mask = torch.tensor(tuple(map(lambda s: s is not None,
                                          batch.next_state)), device=device, dtype=torch.uint8)
    non_final_next_states = torch.cat([s for s in batch.next_state
                                                if s is not None])
    state_batch = torch.cat(batch.state)
    action_batch = torch.cat(batch.action)
    reward_batch = torch.cat(batch.reward)

    # Compute Q(s_t, a) - the model computes Q(s_t), then we select the
    # columns of actions taken
    state_action_values = policy_net(state_batch).gather(1, action_batch)

    # Compute V(s_{t+1}) for all next states.
    next_state_values = torch.zeros(BATCH_SIZE, device=device)
    next_state_values[non_final_mask] = target_net(non_final_next_states).max(1)[0].detach()
    # Compute the expected Q values
    expected_state_action_values = (next_state_values * GAMMA) + reward_batch

    # Compute Huber loss
    loss = F.smooth_l1_loss(state_action_values, expected_state_action_values.unsqueeze(1))

    # Optimize the model
    optimizer.zero_grad()
    loss.backward()
    for param in policy_net.parameters():
        param.grad.data.clamp_(-1, 1)
    optimizer.step()
```

Below, you can find the main training loop. At the beginning we reset
the environment and initialize the ``state`` Tensor. Then, we sample
an action, execute it, observe the next screen and the reward (always
1), and optimize our model once. When the episode ends (our model
fails), we restart the loop.

Below, `num_episodes` is set small. You should download
the notebook and run lot more epsiodes.

```
num_episodes = 50
for i_episode in range(num_episodes):
    # Initialize the environment and state
    env.reset()
    last_screen = get_screen()
    current_screen = get_screen()
    state = current_screen - last_screen
    for t in count():
        # Select and perform an action
        action = select_action(state)
        _, reward, done, _ = env.step(action.item())
        reward = torch.tensor([reward], device=device)

        # Observe new state
        last_screen = current_screen
        current_screen = get_screen()
        if not done:
            next_state = current_screen - last_screen
        else:
            next_state = None

        # Store the transition in memory
        memory.push(state, action, next_state, reward)

        # Move to the next state
        state = next_state

        # Perform one step of the optimization (on the target network)
        optimize_model()
        if done:
            episode_durations.append(t + 1)
            plot_durations()
            break
    # Update the target network
    if i_episode % TARGET_UPDATE == 0:
        target_net.load_state_dict(policy_net.state_dict())

print('Complete')
env.render()
plt.show()
```

    Complete

```
frames = []
for i in range(3):
    env.reset()
    last_screen = get_screen()
    current_screen = get_screen()
    state = current_screen - last_screen
    done = False
    R = 0
    t = 0
    while not done and t < 200:
        frames.append(env.render(mode = 'rgb_array'))
        action = select_action(state)
        obs, r, done, _ = env.step(action.item())
        R += r
        t += 1
        if not done:
            next_state = current_screen - last_screen
        else:
            next_state = None
        state = next_state
    print('test episode:', i, 'R:', R)
env.render()

import matplotlib.pyplot as plt
import matplotlib.animation
import numpy as np
from IPython.display import HTML

# make a video to display
plt.figure(figsize=(frames[0].shape[1] / 72.0, frames[0].shape[0] / 72.0), dpi = 72)
patch = plt.imshow(frames[0])
plt.axis('off')
animate = lambda i: patch.set_data(frames[i])
ani = matplotlib.animation.FuncAnimation(plt.gcf(), animate, frames=len(frames), interval = 50)
HTML(ani.to_jshtml())
```