[Guide To Reinforcement Learning](https://skymind.ai/wiki/deep-reinforcement-learning)

# Reinforcement learning (RL)
Reinforcement learning (RL) is the subfield of machine learning concerned with decision making and motor control. It studies how an agent can learn how to achieve goals in a complex, uncertain environment. It’s exciting for two reasons:

*   RL is very general, encompassing all problems that involve making a sequence of decisions: for example, controlling a robot’s motors so that it’s able to run and jump, making business decisions like pricing and inventory management, or playing video games and board games. RL can even be applied to supervised learning problems with sequential or structured outputs.
*   RL algorithms have started to achieve good results in many difficult environments. RL has a long history, but until recent advances in deep learning, it required lots of problem-specific engineering. DeepMind’s Atari results, BRETT from Pieter Abbeel’s group, and AlphaGo all used deep RL algorithms which did not make too many assumptions about their environment, and thus can be applied in other settings.

See: [Open AI GYM](https://gym.openai.com/)

RL is a general concept that can be simply described with an agent that takes actions in an environment in order to maximize its cumulative reward. The underlying idea is very lifelike, where similarly to the humans in real life, agents in RL algorithms are incentivized with punishments for bad actions and rewards for good ones.

## Markov Chain

State–action–reward–state–action (SARSA) is an algorithm for learning a Markov decision process policy,

We start with an initial environment. It doesn’t have any associated reward yet, but it has a state (S_t).

Then for each iteration, an agent takes current state (S_t), picks best (based on model prediction) action (A_t) and executes it on an environment. Subsequently, environment returns a reward (R_t+1) for a given action, a new state (S_t+1) and an information if the new state is terminal. The process repeats until termination.

## Deep Q-Learning (DQN)

DQN is a RL technique that is aimed at choosing the best action for given circumstances (observation). Each possible action for each possible observation has its Q value, where ‘Q’ stands for a quality of a given move.

But how do we end up with accurate Q values? That’s where the deep neural networks and linear algebra come in.

For each state experienced by our agent, we are going to remember it

dqn_solver.remember(state, action, reward, state_next, terminal)

and perform an experience replay.

dqn_solver.experience_replay()

Experience replay is a biologically inspired process that uniformly (to reduce correlation between subsequent actions) samples experiences from the memory and for each entry updates its Q value.

We are calculating the new q by taking the maximum q for a given action (predicted value of a best next state), multiplying it by the discount factor (GAMMA) and ultimately adding it to the current state reward.

In other words, we are updating our Q value with the cumulative discounted future rewards.

Here is a formal notation:

(source: https://en.wikipedia.org/wiki/Q-learning)

For those of you who wonder how such function can possibly converge, as it looks like it is trying to predict its own output (in some sense it is!), don’t worry - it’s possible and in our simple case it does.

However, convergence is not always that ‘easy’ and in more complex problems there comes a need of more advanced techniques that stabilize training. These techniques are for example Double DQN’s or Dueling DQN’s, but that’s a topic for another article (stay tuned).

Reinforcement Learning (DQN) tutorial
=====================================
Adapted from PyTorch website.

**Author**: `Adam Paszke <https://github.com/apaszke>`_

This tutorial shows how to use PyTorch to train a Deep Q Learning (DQN) agent
on the CartPole-v0 task from the `OpenAI Gym <https://gym.openai.com/>`__.

**Task**

The agent has to decide between two actions - moving the cart left or
right - so that the pole attached to it stays upright. You can find an
official leaderboard with various algorithms and visualizations at the
`Gym website <https://gym.openai.com/envs/CartPole-v0>`__.

As the agent observes the current state of the environment and chooses
an action, the environment *transitions* to a new state, and also
returns a reward that indicates the consequences of the action. In this
task, the environment terminates if the pole falls over too far.

The CartPole task is designed so that the inputs to the agent are 4 real
values representing the environment state (position, velocity, etc.).
However, neural networks can solve the task purely by looking at the
scene, so we'll use a patch of the screen centered on the cart as an
input. Because of this, our results aren't directly comparable to the
ones from the official leaderboard - our task is much harder.
Unfortunately this does slow down the training, because we have to
render all the frames.

Strictly speaking, we will present the state as the difference between
the current screen patch and the previous one. This will allow the agent
to take the velocity of the pole into account from one image.

**Packages**

First, let's import needed packages. Firstly, we need
`gym <https://gym.openai.com/docs>`__ for the environment
(Install using `pip install gym`).
We'll also use the following from PyTorch:

-  neural networks (``torch.nn``)
-  optimization (``torch.optim``)
-  automatic differentiation (``torch.autograd``)
-  utilities for vision tasks (``torchvision`` - `a separate
   package <https://github.com/pytorch/vision>`__).

##Cartpole Problem

Cartpole Problem
Cartpole - known also as an Inverted Pendulum is a pendulum with a center of gravity above its pivot point. It’s unstable, but can be controlled by moving the pivot point under the center of mass. The goal is to keep the cartpole balanced by applying appropriate forces to a pivot point.

Violet square indicates a pivot point
Red and green arrows show possible horizontal forces that can be applied to a pivot point

*A pole is attached by an un-actuated joint to a cart, which moves along a frictionless track. The system is controlled by applying a force of +1 or -1 to the cart. The pendulum starts upright, and the goal is to prevent it from falling over. A reward of +1 is provided for every timestep that the pole remains upright. The episode ends when the pole is more than 15 degrees from vertical, or the cart moves more than 2.4 units from the center.*

Take a look at a video below with a real-life demonstration of a cartpole problem learning process.

[Real-life application of Reinforcement Learning
 video](https://youtu.be/XiigTGKZfks)

## Import
Before we start the tutorial, we need to install some dependencies to render and capture images from OpenAI Gym. It may take some time to set up the environment.

```
!apt-get install cmake zlib1g-dev libjpeg-dev xvfb libav-tools xorg-dev libboost-all-dev libsdl2-dev swig python3-dev python3-future python-opengl

!apt-get -qq -y install libcusparse8.0 libnvrtc8.0 libnvtoolsext1 > /dev/null
!ln -snf /usr/lib/x86_64-linux-gnu/libnvrtc-builtins.so.8.0 /usr/lib/x86_64-linux-gnu/libnvrtc-builtins.so
!apt-get -qq -y install xvfb freeglut3-dev ffmpeg> /dev/null

!apt-get install xserver-xorg libglu1-mesa-dev mesa-common-dev libxmu-dev libxi-dev
```

    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    Package libav-tools is not available, but is referred to by another package.
    This may mean that the package is missing, has been obsoleted, or
    is only available from another source
    However the following packages replace it:
      ffmpeg

    E: Package 'libav-tools' has no installation candidate
    E: Unable to locate package libcusparse8.0
    E: Couldn't find any package by glob 'libcusparse8.0'
    E: Couldn't find any package by regex 'libcusparse8.0'
    E: Unable to locate package libnvrtc8.0
    E: Couldn't find any package by glob 'libnvrtc8.0'
    E: Couldn't find any package by regex 'libnvrtc8.0'
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    libglu1-mesa-dev is already the newest version (9.0.0-2.1build1).
    libglu1-mesa-dev set to manually installed.
    libxi-dev is already the newest version (2:1.7.9-1).
    libxi-dev set to manually installed.
    libxmu-dev is already the newest version (2:1.1.2-2).
    libxmu-dev set to manually installed.
    mesa-common-dev is already the newest version (18.2.8-0ubuntu0~18.04.2).
    mesa-common-dev set to manually installed.
    The following package was automatically installed and is no longer required:
      libnvidia-common-410
    Use 'apt autoremove' to remove it.
    The following additional packages will be installed:
      apport apport-symptoms libxatracker2 libxvmc1 python3-apport python3-certifi
      python3-chardet python3-httplib2 python3-idna python3-pkg-resources
      python3-problem-report python3-requests python3-requests-unixsocket
      python3-six python3-systemd python3-urllib3 xserver-xorg-core
      xserver-xorg-input-all xserver-xorg-input-libinput xserver-xorg-input-wacom
      xserver-xorg-legacy xserver-xorg-video-all xserver-xorg-video-amdgpu
      xserver-xorg-video-ati xserver-xorg-video-fbdev xserver-xorg-video-intel
      xserver-xorg-video-nouveau xserver-xorg-video-qxl xserver-xorg-video-radeon
      xserver-xorg-video-vesa xserver-xorg-video-vmware
    Suggested packages:
      apport-gtk | apport-kde python3-launchpadlib python3-setuptools
      python3-cryptography python3-openssl python3-socks xfonts-100dpi
      | xfonts-75dpi xfonts-scalable xinput firmware-amd-graphics
      xserver-xorg-video-r128 xserver-xorg-video-mach64 firmware-misc-nonfree
    The following packages will be REMOVED:
      xserver-xorg-core-hwe-18.04
    The following NEW packages will be installed:
      apport apport-symptoms libxatracker2 libxvmc1 python3-apport python3-certifi
      python3-chardet python3-httplib2 python3-idna python3-pkg-resources
      python3-problem-report python3-requests python3-requests-unixsocket
      python3-six python3-systemd python3-urllib3 xserver-xorg xserver-xorg-core
      xserver-xorg-input-all xserver-xorg-input-libinput xserver-xorg-input-wacom
      xserver-xorg-legacy xserver-xorg-video-all xserver-xorg-video-amdgpu
      xserver-xorg-video-ati xserver-xorg-video-fbdev xserver-xorg-video-intel
      xserver-xorg-video-nouveau xserver-xorg-video-qxl xserver-xorg-video-radeon
      xserver-xorg-video-vesa xserver-xorg-video-vmware
    0 upgraded, 32 newly installed, 1 to remove and 16 not upgraded.
    Need to get 4,784 kB of archives.
    After this operation, 14.0 MB of additional disk space will be used.
    Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 xserver-xorg-core amd64 2:1.19.6-1ubuntu4.2 [1,351 kB]
    Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.1 [28.6 kB]
    Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-problem-report all 2.20.9-0ubuntu7.6 [9,884 B]
    Get:4 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-certifi all 2018.1.18-2 [144 kB]
    Get:5 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pkg-resources all 39.0.1-2 [98.8 kB]
    Get:6 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-chardet all 3.0.4-1 [80.3 kB]
    Get:7 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-idna all 2.6-1 [32.5 kB]
    Get:8 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-six all 1.11.0-2 [11.4 kB]
    Get:9 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-urllib3 all 1.22-1 [85.2 kB]
    Get:10 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-requests all 2.18.4-2ubuntu0.1 [58.3 kB]
    Get:11 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-requests-unixsocket all 0.1.5-3 [6,286 B]
    Get:12 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-apport all 2.20.9-0ubuntu7.6 [81.7 kB]
    Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apport all 2.20.9-0ubuntu7.6 [124 kB]
    Get:14 http://archive.ubuntu.com/ubuntu bionic/main amd64 apport-symptoms all 0.20 [14.2 kB]
    Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxatracker2 amd64 18.2.8-0ubuntu0~18.04.2 [1,172 kB]
    Get:16 http://archive.ubuntu.com/ubuntu bionic/main amd64 libxvmc1 amd64 2:1.0.10-1 [13.7 kB]
    Get:17 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-systemd amd64 234-1build1 [36.1 kB]
    Get:18 http://archive.ubuntu.com/ubuntu bionic/main amd64 xserver-xorg-input-libinput amd64 0.27.1-1 [33.2 kB]
    Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 xserver-xorg-input-all amd64 1:7.7+19ubuntu7.1 [4,112 B]
    Get:20 http://archive.ubuntu.com/ubuntu bionic/main amd64 xserver-xorg-input-wacom amd64 1:0.36.1-0ubuntu1 [89.7 kB]
    Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 xserver-xorg amd64 1:7.7+19ubuntu7.1 [65.2 kB]
    Get:22 http://archive.ubuntu.com/ubuntu bionic/main amd64 xserver-xorg-video-amdgpu amd64 18.0.1-1 [61.0 kB]
    Get:23 http://archive.ubuntu.com/ubuntu bionic/main amd64 xserver-xorg-video-radeon amd64 1:18.0.1-1 [147 kB]
    Get:24 http://archive.ubuntu.com/ubuntu bionic/main amd64 xserver-xorg-video-ati amd64 1:18.0.1-1 [7,116 B]
    Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 xserver-xorg-video-fbdev amd64 1:0.4.4-1build6 [11.5 kB]
    Get:26 http://archive.ubuntu.com/ubuntu bionic/main amd64 xserver-xorg-video-nouveau amd64 1:1.0.15-2 [86.3 kB]
    Get:27 http://archive.ubuntu.com/ubuntu bionic/main amd64 xserver-xorg-video-vesa amd64 1:2.3.4-1build3 [14.7 kB]
    Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 xserver-xorg-video-vmware amd64 1:13.2.1-1build1 [71.6 kB]
    Get:29 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 xserver-xorg-video-all amd64 1:7.7+19ubuntu7.1 [4,164 B]
    Get:30 http://archive.ubuntu.com/ubuntu bionic/main amd64 xserver-xorg-video-intel amd64 2:2.99.917+git20171229-1 [728 kB]
    Get:31 http://archive.ubuntu.com/ubuntu bionic/main amd64 xserver-xorg-video-qxl amd64 0.1.5-2build1 [81.0 kB]
    Get:32 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 xserver-xorg-legacy amd64 2:1.19.6-1ubuntu4.2 [32.9 kB]
    Fetched 4,784 kB in 1s (3,692 kB/s)
    Extracting templates from packages: 100%
    Preconfiguring packages ...
    dpkg: xserver-xorg-core-hwe-18.04: dependency problems, but removing anyway as you requested:
     xserver-xorg-video-nvidia-418 depends on xorg-video-abi-24 | xorg-video-abi-23 | xorg-video-abi-20 | xorg-video-abi-19 | xorg-video-abi-18 | xorg-video-abi-15 | xorg-video-abi-14 | xorg-video-abi-13 | xorg-video-abi-12 | xorg-video-abi-11 | xorg-video-abi-10 | xorg-video-abi-8 | xorg-video-abi-6.0; however:
      Package xorg-video-abi-24 is not installed.
      Package xserver-xorg-core-hwe-18.04 which provides xorg-video-abi-24 is to be removed.
      Package xorg-video-abi-23 is not installed.
      Package xorg-video-abi-20 is not installed.
      Package xorg-video-abi-19 is not installed.
      Package xorg-video-abi-18 is not installed.
      Package xorg-video-abi-15 is not installed.
      Package xorg-video-abi-14 is not installed.
      Package xorg-video-abi-13 is not installed.
      Package xorg-video-abi-12 is not installed.
      Package xorg-video-abi-11 is not installed.
      Package xorg-video-abi-10 is not installed.
      Package xorg-video-abi-8 is not installed.
      Package xorg-video-abi-6.0 is not installed.
     xserver-xorg-video-nvidia-418 depends on xserver-xorg-core (>= 2:1.19.6-1ubuntu2~) | xserver-xorg-core-hwe-18.04; however:
      Package xserver-xorg-core is not installed.
      Package xserver-xorg-core-hwe-18.04 which provides xserver-xorg-core is to be removed.
      Package xserver-xorg-core-hwe-18.04 is to be removed.
     xserver-xorg-video-nvidia-418 depends on xorg-video-abi-24 | xorg-video-abi-23 | xorg-video-abi-20 | xorg-video-abi-19 | xorg-video-abi-18 | xorg-video-abi-15 | xorg-video-abi-14 | xorg-video-abi-13 | xorg-video-abi-12 | xorg-video-abi-11 | xorg-video-abi-10 | xorg-video-abi-8 | xorg-video-abi-6.0; however:
      Package xorg-video-abi-24 is not installed.
      Package xserver-xorg-core-hwe-18.04 which provides xorg-video-abi-24 is to be removed.
      Package xorg-video-abi-23 is not installed.
      Package xorg-video-abi-20 is not installed.
      Package xorg-video-abi-19 is not installed.
      Package xorg-video-abi-18 is not installed.
      Package xorg-video-abi-15 is not installed.
      Package xorg-video-abi-14 is not installed.
      Package xorg-video-abi-13 is not installed.
      Package xorg-video-abi-12 is not installed.
      Package xorg-video-abi-11 is not installed.
      Package xorg-video-abi-10 is not installed.
      Package xorg-video-abi-8 is not installed.
      Package xorg-video-abi-6.0 is not installed.
     xserver-xorg-video-nvidia-418 depends on xserver-xorg-core (>= 2:1.19.6-1ubuntu2~) | xserver-xorg-core-hwe-18.04; however:
      Package xserver-xorg-core is not installed.
      Package xserver-xorg-core-hwe-18.04 which provides xserver-xorg-core is to be removed.
      Package xserver-xorg-core-hwe-18.04 is to be removed.
     xserver-xorg-video-nvidia-418 depends on xorg-video-abi-24 | xorg-video-abi-23 | xorg-video-abi-20 | xorg-video-abi-19 | xorg-video-abi-18 | xorg-video-abi-15 | xorg-video-abi-14 | xorg-video-abi-13 | xorg-video-abi-12 | xorg-video-abi-11 | xorg-video-abi-10 | xorg-video-abi-8 | xorg-video-abi-6.0; however:
      Package xorg-video-abi-24 is not installed.
      Package xserver-xorg-core-hwe-18.04 which provides xorg-video-abi-24 is to be removed.
      Package xorg-video-abi-23 is not installed.
      Package xorg-video-abi-20 is not installed.
      Package xorg-video-abi-19 is not installed.
      Package xorg-video-abi-18 is not installed.
      Package xorg-video-abi-15 is not installed.
      Package xorg-video-abi-14 is not installed.
      Package xorg-video-abi-13 is not installed.
      Package xorg-video-abi-12 is not installed.
      Package xorg-video-abi-11 is not installed.
      Package xorg-video-abi-10 is not installed.
      Package xorg-video-abi-8 is not installed.
      Package xorg-video-abi-6.0 is not installed.
     xserver-xorg-video-nvidia-418 depends on xserver-xorg-core (>= 2:1.19.6-1ubuntu2~) | xserver-xorg-core-hwe-18.04; however:
      Package xserver-xorg-core is not installed.
      Package xserver-xorg-core-hwe-18.04 which provides xserver-xorg-core is to be removed.
      Package xserver-xorg-core-hwe-18.04 is to be removed.

    (Reading database ... 130818 files and directories currently installed.)
    Removing xserver-xorg-core-hwe-18.04 (2:1.20.1-3ubuntu2.1~18.04.1) ...
    Selecting previously unselected package xserver-xorg-core.
    (Reading database ... 130779 files and directories currently installed.)
    Preparing to unpack .../00-xserver-xorg-core_2%3a1.19.6-1ubuntu4.2_amd64.deb ...
    Unpacking xserver-xorg-core (2:1.19.6-1ubuntu4.2) ...
    Selecting previously unselected package python3-httplib2.
    Preparing to unpack .../01-python3-httplib2_0.9.2+dfsg-1ubuntu0.1_all.deb ...
    Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.1) ...
    Selecting previously unselected package python3-problem-report.
    Preparing to unpack .../02-python3-problem-report_2.20.9-0ubuntu7.6_all.deb ...
    Unpacking python3-problem-report (2.20.9-0ubuntu7.6) ...
    Selecting previously unselected package python3-certifi.
    Preparing to unpack .../03-python3-certifi_2018.1.18-2_all.deb ...
    Unpacking python3-certifi (2018.1.18-2) ...
    Selecting previously unselected package python3-pkg-resources.
    Preparing to unpack .../04-python3-pkg-resources_39.0.1-2_all.deb ...
    Unpacking python3-pkg-resources (39.0.1-2) ...
    Selecting previously unselected package python3-chardet.
    Preparing to unpack .../05-python3-chardet_3.0.4-1_all.deb ...
    Unpacking python3-chardet (3.0.4-1) ...
    Selecting previously unselected package python3-idna.
    Preparing to unpack .../06-python3-idna_2.6-1_all.deb ...
    Unpacking python3-idna (2.6-1) ...
    Selecting previously unselected package python3-six.
    Preparing to unpack .../07-python3-six_1.11.0-2_all.deb ...
    Unpacking python3-six (1.11.0-2) ...
    Selecting previously unselected package python3-urllib3.
    Preparing to unpack .../08-python3-urllib3_1.22-1_all.deb ...
    Unpacking python3-urllib3 (1.22-1) ...
    Selecting previously unselected package python3-requests.
    Preparing to unpack .../09-python3-requests_2.18.4-2ubuntu0.1_all.deb ...
    Unpacking python3-requests (2.18.4-2ubuntu0.1) ...
    Selecting previously unselected package python3-requests-unixsocket.
    Preparing to unpack .../10-python3-requests-unixsocket_0.1.5-3_all.deb ...
    Unpacking python3-requests-unixsocket (0.1.5-3) ...
    Selecting previously unselected package python3-apport.
    Preparing to unpack .../11-python3-apport_2.20.9-0ubuntu7.6_all.deb ...
    Unpacking python3-apport (2.20.9-0ubuntu7.6) ...
    Selecting previously unselected package apport.
    Preparing to unpack .../12-apport_2.20.9-0ubuntu7.6_all.deb ...
    Unpacking apport (2.20.9-0ubuntu7.6) ...
    Selecting previously unselected package apport-symptoms.
    Preparing to unpack .../13-apport-symptoms_0.20_all.deb ...
    Unpacking apport-symptoms (0.20) ...
    Selecting previously unselected package libxatracker2:amd64.
    Preparing to unpack .../14-libxatracker2_18.2.8-0ubuntu0~18.04.2_amd64.deb ...
    Unpacking libxatracker2:amd64 (18.2.8-0ubuntu0~18.04.2) ...
    Selecting previously unselected package libxvmc1:amd64.
    Preparing to unpack .../15-libxvmc1_2%3a1.0.10-1_amd64.deb ...
    Unpacking libxvmc1:amd64 (2:1.0.10-1) ...
    Selecting previously unselected package python3-systemd.
    Preparing to unpack .../16-python3-systemd_234-1build1_amd64.deb ...
    Unpacking python3-systemd (234-1build1) ...
    Selecting previously unselected package xserver-xorg-input-libinput.
    Preparing to unpack .../17-xserver-xorg-input-libinput_0.27.1-1_amd64.deb ...
    Unpacking xserver-xorg-input-libinput (0.27.1-1) ...
    Selecting previously unselected package xserver-xorg-input-all.
    Preparing to unpack .../18-xserver-xorg-input-all_1%3a7.7+19ubuntu7.1_amd64.deb ...
    Unpacking xserver-xorg-input-all (1:7.7+19ubuntu7.1) ...
    Selecting previously unselected package xserver-xorg-input-wacom.
    Preparing to unpack .../19-xserver-xorg-input-wacom_1%3a0.36.1-0ubuntu1_amd64.deb ...
    Unpacking xserver-xorg-input-wacom (1:0.36.1-0ubuntu1) ...
    Selecting previously unselected package xserver-xorg.
    Preparing to unpack .../20-xserver-xorg_1%3a7.7+19ubuntu7.1_amd64.deb ...
    Unpacking xserver-xorg (1:7.7+19ubuntu7.1) ...
    Selecting previously unselected package xserver-xorg-video-amdgpu.
    Preparing to unpack .../21-xserver-xorg-video-amdgpu_18.0.1-1_amd64.deb ...
    Unpacking xserver-xorg-video-amdgpu (18.0.1-1) ...
    Selecting previously unselected package xserver-xorg-video-radeon.
    Preparing to unpack .../22-xserver-xorg-video-radeon_1%3a18.0.1-1_amd64.deb ...
    Unpacking xserver-xorg-video-radeon (1:18.0.1-1) ...
    Selecting previously unselected package xserver-xorg-video-ati.
    Preparing to unpack .../23-xserver-xorg-video-ati_1%3a18.0.1-1_amd64.deb ...
    Unpacking xserver-xorg-video-ati (1:18.0.1-1) ...
    Selecting previously unselected package xserver-xorg-video-fbdev.
    Preparing to unpack .../24-xserver-xorg-video-fbdev_1%3a0.4.4-1build6_amd64.deb ...
    Unpacking xserver-xorg-video-fbdev (1:0.4.4-1build6) ...
    Selecting previously unselected package xserver-xorg-video-nouveau.
    Preparing to unpack .../25-xserver-xorg-video-nouveau_1%3a1.0.15-2_amd64.deb ...
    Unpacking xserver-xorg-video-nouveau (1:1.0.15-2) ...
    Selecting previously unselected package xserver-xorg-video-vesa.
    Preparing to unpack .../26-xserver-xorg-video-vesa_1%3a2.3.4-1build3_amd64.deb ...
    Unpacking xserver-xorg-video-vesa (1:2.3.4-1build3) ...
    Selecting previously unselected package xserver-xorg-video-vmware.
    Preparing to unpack .../27-xserver-xorg-video-vmware_1%3a13.2.1-1build1_amd64.deb ...
    Unpacking xserver-xorg-video-vmware (1:13.2.1-1build1) ...
    Selecting previously unselected package xserver-xorg-video-all.
    Preparing to unpack .../28-xserver-xorg-video-all_1%3a7.7+19ubuntu7.1_amd64.deb ...
    Unpacking xserver-xorg-video-all (1:7.7+19ubuntu7.1) ...
    Selecting previously unselected package xserver-xorg-video-intel.
    Preparing to unpack .../29-xserver-xorg-video-intel_2%3a2.99.917+git20171229-1_amd64.deb ...
    Unpacking xserver-xorg-video-intel (2:2.99.917+git20171229-1) ...
    Selecting previously unselected package xserver-xorg-video-qxl.
    Preparing to unpack .../30-xserver-xorg-video-qxl_0.1.5-2build1_amd64.deb ...
    Unpacking xserver-xorg-video-qxl (0.1.5-2build1) ...
    Selecting previously unselected package xserver-xorg-legacy.
    Preparing to unpack .../31-xserver-xorg-legacy_2%3a1.19.6-1ubuntu4.2_amd64.deb ...
    Unpacking xserver-xorg-legacy (2:1.19.6-1ubuntu4.2) ...
    Setting up python3-idna (2.6-1) ...
    Setting up python3-six (1.11.0-2) ...
    Setting up python3-certifi (2018.1.18-2) ...
    Setting up python3-pkg-resources (39.0.1-2) ...
    Setting up libxvmc1:amd64 (2:1.0.10-1) ...
    Setting up libxatracker2:amd64 (18.2.8-0ubuntu0~18.04.2) ...
    Setting up xserver-xorg-core (2:1.19.6-1ubuntu4.2) ...
    Setting up xserver-xorg-video-radeon (1:18.0.1-1) ...
    Setting up python3-problem-report (2.20.9-0ubuntu7.6) ...
    Setting up xserver-xorg-video-vesa (1:2.3.4-1build3) ...
    Processing triggers for libc-bin (2.27-3ubuntu1) ...
    Setting up xserver-xorg-video-amdgpu (18.0.1-1) ...
    Setting up python3-chardet (3.0.4-1) ...
    Setting up python3-systemd (234-1build1) ...
    Processing triggers for systemd (237-3ubuntu10.21) ...
    Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.1) ...
    Setting up apport-symptoms (0.20) ...
    Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
    Setting up python3-urllib3 (1.22-1) ...
    Setting up xserver-xorg-input-wacom (1:0.36.1-0ubuntu1) ...
    Processing triggers for hicolor-icon-theme (0.17-2) ...
    Setting up xserver-xorg-legacy (2:1.19.6-1ubuntu4.2) ...
    Setting up xserver-xorg-video-ati (1:18.0.1-1) ...
    Setting up xserver-xorg-video-intel (2:2.99.917+git20171229-1) ...
    Setting up xserver-xorg-video-qxl (0.1.5-2build1) ...
    Setting up xserver-xorg-video-nouveau (1:1.0.15-2) ...
    Setting up xserver-xorg-video-fbdev (1:0.4.4-1build6) ...
    Setting up xserver-xorg-input-libinput (0.27.1-1) ...
    Setting up xserver-xorg-video-vmware (1:13.2.1-1build1) ...
    Setting up python3-requests (2.18.4-2ubuntu0.1) ...
    Setting up xserver-xorg-input-all (1:7.7+19ubuntu7.1) ...
    Setting up python3-requests-unixsocket (0.1.5-3) ...
    Setting up xserver-xorg-video-all (1:7.7+19ubuntu7.1) ...
    Setting up python3-apport (2.20.9-0ubuntu7.6) ...
    Setting up xserver-xorg (1:7.7+19ubuntu7.1) ...
    Setting up apport (2.20.9-0ubuntu7.6) ...
    Created symlink /etc/systemd/system/paths.target.wants/apport-autoreport.path → /lib/systemd/system/apport-autoreport.path.
    Created symlink /etc/systemd/system/sockets.target.wants/apport-forward.socket → /lib/systemd/system/apport-forward.socket.
    invoke-rc.d: could not determine current runlevel
    invoke-rc.d: policy-rc.d denied execution of start.
    Processing triggers for libc-bin (2.27-3ubuntu1) ...
    Processing triggers for systemd (237-3ubuntu10.21) ...

```
!pip3 install torch torchvision gym[all] PyOpenGL piglet pyglet pyvirtualdisplay
```

    Requirement already satisfied: torch in /usr/local/lib/python3.6/dist-packages (1.1.0)
    Requirement already satisfied: torchvision in /usr/local/lib/python3.6/dist-packages (0.2.2.post3)
    Requirement already satisfied: gym[all] in /usr/local/lib/python3.6/dist-packages (0.10.11)
    Requirement already satisfied: PyOpenGL in /usr/local/lib/python3.6/dist-packages (3.1.0)
    Collecting piglet
      Downloading https://files.pythonhosted.org/packages/86/f6/ef278239ebe525466ea51a7dd9d6d3211d197ac4b4abc76e17cdd419f69c/piglet-0.4.4.tar.gz (52kB)
         |████████████████████████████████| 61kB 2.4MB/s
    Requirement already satisfied: pyglet in /usr/local/lib/python3.6/dist-packages (1.3.2)
    Collecting pyvirtualdisplay
      Downloading https://files.pythonhosted.org/packages/39/37/f285403a09cc261c56b6574baace1bdcf4b8c7428c8a7239cbba137bc0eb/PyVirtualDisplay-0.2.1.tar.gz
    Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from torch) (1.16.3)
    Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.12.0)
    Requirement already satisfied: pillow>=4.1.1 in /usr/local/lib/python3.6/dist-packages (from torchvision) (4.3.0)
    Requirement already satisfied: requests>=2.0 in /usr/local/lib/python3.6/dist-packages (from gym[all]) (2.21.0)
    Requirement already satisfied: scipy in /usr/local/lib/python3.6/dist-packages (from gym[all]) (1.2.1)
    Requirement already satisfied: imageio; extra == "all" in /usr/local/lib/python3.6/dist-packages (from gym[all]) (2.4.1)
    Collecting mujoco-py>=1.50; extra == "all" (from gym[all])
      Downloading https://files.pythonhosted.org/packages/a9/c8/daac2e747a0a516f6a880bf73304bd58e18d8be5cea926e8a1e5938026e6/mujoco-py-2.0.2.2.tar.gz (771kB)
         |████████████████████████████████| 778kB 8.2MB/s
    Requirement already satisfied: atari-py>=0.1.4; extra == "all" in /usr/local/lib/python3.6/dist-packages (from gym[all]) (0.1.7)
    Collecting box2d-py>=2.3.5; extra == "all" (from gym[all])
      Downloading https://files.pythonhosted.org/packages/06/bd/6cdc3fd994b0649dcf5d9bad85bd9e26172308bbe9a421bfc6fdbf5081a6/box2d_py-2.3.8-cp36-cp36m-manylinux1_x86_64.whl (448kB)
         |████████████████████████████████| 450kB 46.6MB/s
    Collecting Parsley (from piglet)
      Downloading https://files.pythonhosted.org/packages/2b/d6/4fed8d65e28a970e1c5cb33ce9c7e22e3de745e1b2ae37af051ef16aea3b/Parsley-1.3-py2.py3-none-any.whl (88kB)
         |████████████████████████████████| 92kB 31.8MB/s
    Requirement already satisfied: attrs in /usr/local/lib/python3.6/dist-packages (from piglet) (19.1.0)
    Collecting astunparse (from piglet)
      Downloading https://files.pythonhosted.org/packages/2e/37/5dd0dd89b87bb5f0f32a7e775458412c52d78f230ab8d0c65df6aabc4479/astunparse-1.6.2-py2.py3-none-any.whl
    Requirement already satisfied: markupsafe in /usr/local/lib/python3.6/dist-packages (from piglet) (1.1.1)
    Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from pyglet) (0.16.0)
    Collecting EasyProcess (from pyvirtualdisplay)
      Downloading https://files.pythonhosted.org/packages/45/3a/4eecc0c7995a13a64739bbedc0d3691fc574245b7e79cff81905aa0c2b38/EasyProcess-0.2.5.tar.gz
    Requirement already satisfied: olefile in /usr/local/lib/python3.6/dist-packages (from pillow>=4.1.1->torchvision) (0.46)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests>=2.0->gym[all]) (2019.3.9)
    Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests>=2.0->gym[all]) (2.8)
    Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests>=2.0->gym[all]) (3.0.4)
    Requirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests>=2.0->gym[all]) (1.24.3)
    Collecting glfw>=1.4.0 (from mujoco-py>=1.50; extra == "all"->gym[all])
      Downloading https://files.pythonhosted.org/packages/8c/8c/efa07845b9b713bba35f33ea813bcb019612bbbddb03c5f65ca857f81ac7/glfw-1.8.0.tar.gz
    Requirement already satisfied: Cython>=0.27.2 in /usr/local/lib/python3.6/dist-packages (from mujoco-py>=1.50; extra == "all"->gym[all]) (0.29.7)
    Requirement already satisfied: cffi>=1.10 in /usr/local/lib/python3.6/dist-packages (from mujoco-py>=1.50; extra == "all"->gym[all]) (1.12.3)
    Collecting lockfile>=0.12.2 (from mujoco-py>=1.50; extra == "all"->gym[all])
      Downloading https://files.pythonhosted.org/packages/c8/22/9460e311f340cb62d26a38c419b1381b8593b0bb6b5d1f056938b086d362/lockfile-0.12.2-py2.py3-none-any.whl
    Requirement already satisfied: wheel<1.0,>=0.23.0 in /usr/local/lib/python3.6/dist-packages (from astunparse->piglet) (0.33.1)
    Requirement already satisfied: pycparser in /usr/local/lib/python3.6/dist-packages (from cffi>=1.10->mujoco-py>=1.50; extra == "all"->gym[all]) (2.19)
    Building wheels for collected packages: piglet, pyvirtualdisplay, mujoco-py, EasyProcess, glfw
      Building wheel for piglet (setup.py) ... done
      Stored in directory: /root/.cache/pip/wheels/c7/59/a5/5bd1a35a4a4596714c4c7925a1751e7b1580b6ced363fd7969
      Building wheel for pyvirtualdisplay (setup.py) ... done
      Stored in directory: /root/.cache/pip/wheels/d1/8c/16/1c64227974ae29c687e4cc30fd691d5c0fd40f54446dde99da
      Building wheel for mujoco-py (setup.py) ... error
      ERROR: Failed building wheel for mujoco-py
      Running setup.py clean for mujoco-py
      Building wheel for EasyProcess (setup.py) ... done
      Stored in directory: /root/.cache/pip/wheels/41/22/19/af15ef6264c58b625a82641ed7483ad05e258fbd8925505227
      Building wheel for glfw (setup.py) ... done
      Stored in directory: /root/.cache/pip/wheels/d9/cf/69/c19646d6ae4742b04a9004bd274dac036a8d23747af99c672d
    Successfully built piglet pyvirtualdisplay EasyProcess glfw
    Failed to build mujoco-py
    Installing collected packages: Parsley, astunparse, piglet, EasyProcess, pyvirtualdisplay, glfw, lockfile, mujoco-py, box2d-py
      Running setup.py install for mujoco-py ... error
    ERROR: Command "/usr/bin/python3 -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-0_1an89a/mujoco-py/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-095jgl6r/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-install-0_1an89a/mujoco-py/

```
# Start virtual display
from pyvirtualdisplay import Display
display = Display(visible=0, size=(1024, 768))
display.start()
import os
os.environ["DISPLAY"] = ":" + str(display.display) + "." + str(display.screen)

```

```
%matplotlib inline
```

```
import gym
import math
import random
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
import matplotlib.animation

from collections import namedtuple
from itertools import count
from PIL import Image

import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
import torchvision.transforms as T

from IPython.display import HTML

env = gym.make('CartPole-v0').unwrapped

# set up matplotlib
is_ipython = 'inline' in matplotlib.get_backend()
if is_ipython:
    from IPython import display

plt.ion()

# if gpu is to be used
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
```

Replay Memory
-------------

We'll be using experience replay memory for training our DQN. It stores
the transitions that the agent observes, allowing us to reuse this data
later. By sampling from it randomly, the transitions that build up a
batch are decorrelated. It has been shown that this greatly stabilizes
and improves the DQN training procedure.

For this, we're going to need two classses:

-  ``Transition`` - a named tuple representing a single transition in
   our environment
-  ``ReplayMemory`` - a cyclic buffer of bounded size that holds the
   transitions observed recently. It also implements a ``.sample()``
   method for selecting a random batch of transitions for training.

```
Transition = namedtuple('Transition',
                        ('state', 'action', 'next_state', 'reward'))

class ReplayMemory(object):

    def __init__(self, capacity):
        self.capacity = capacity
        self.memory = []
        self.position = 0

    def push(self, *args):
        """Saves a transition."""
        if len(self.memory) < self.capacity:
            self.memory.append(None)
        self.memory[self.position] = Transition(*args)
        self.position = (self.position + 1) % self.capacity

    def sample(self, batch_size):
        return random.sample(self.memory, batch_size)

    def __len__(self):
        return len(self.memory)
```

Now, let's define our model. But first, let quickly recap what a DQN is.

DQN algorithm
-------------

Our environment is deterministic, so all equations presented here are
also formulated deterministically for the sake of simplicity. In the
reinforcement learning literature, they would also contain expectations
over stochastic transitions in the environment.

Our aim will be to train a policy that tries to maximize the discounted,
cumulative reward
$R_{t_0} = \sum_{t=t_0}^{\infty} \gamma^{t - t_0} r_t$, where
$R_{t_0}$ is also known as the *return*. The discount,
$\gamma$, should be a constant between $0$ and $1$
that ensures the sum converges. It makes rewards from the uncertain far
future less important for our agent than the ones in the near future
that it can be fairly confident about.

The main idea behind Q-learning is that if we had a function
$Q^*: State \times Action \rightarrow \mathbb{R}$, that could tell
us what our return would be, if we were to take an action in a given
state, then we could easily construct a policy that maximizes our
rewards:

\begin{align}\pi^*(s) = \arg\!\max_a \ Q^*(s, a)\end{align}

However, we don't know everything about the world, so we don't have
access to $Q^*$. But, since neural networks are universal function
approximators, we can simply create one and train it to resemble
$Q^*$.

For our training update rule, we'll use a fact that every $Q$
function for some policy obeys the Bellman equation:

\begin{align}Q^{\pi}(s, a) = r + \gamma Q^{\pi}(s', \pi(s'))\end{align}

The difference between the two sides of the equality is known as the
temporal difference error, $\delta$:

\begin{align}\delta = Q(s, a) - (r + \gamma \max_a Q(s', a))\end{align}

To minimise this error, we will use the `Huber
loss <https://en.wikipedia.org/wiki/Huber_loss>`__. The Huber loss acts
like the mean squared error when the error is small, but like the mean
absolute error when the error is large - this makes it more robust to
outliers when the estimates of $Q$ are very noisy. We calculate
this over a batch of transitions, $B$, sampled from the replay
memory:

\begin{align}\mathcal{L} = \frac{1}{|B|}\sum_{(s, a, s', r) \ \in \ B} \mathcal{L}(\delta)\end{align}

\begin{align}\text{where} \quad \mathcal{L}(\delta) = \begin{cases}
     \frac{1}{2}{\delta^2}  & \text{for } |\delta| \le 1, \\
     |\delta| - \frac{1}{2} & \text{otherwise.}
   \end{cases}\end{align}

### Q-network

Our model will be a convolutional neural network that takes in the
difference between the current and previous screen patches. It has two
outputs, representing $Q(s, \mathrm{left})$ and
$Q(s, \mathrm{right})$ (where $s$ is the input to the
network). In effect, the network is trying to predict the *quality* of
taking each action given the current input.

```
class DQN(nn.Module):

    def __init__(self):
        super(DQN, self).__init__()
        self.conv1 = nn.Conv2d(3, 16, kernel_size=5, stride=2)
        self.bn1 = nn.BatchNorm2d(16)
        self.conv2 = nn.Conv2d(16, 32, kernel_size=5, stride=2)
        self.bn2 = nn.BatchNorm2d(32)
        self.conv3 = nn.Conv2d(32, 32, kernel_size=5, stride=2)
        self.bn3 = nn.BatchNorm2d(32)
        self.head = nn.Linear(448, 2)

    def forward(self, x):
        x = F.relu(self.bn1(self.conv1(x)))
        x = F.relu(self.bn2(self.conv2(x)))
        x = F.relu(self.bn3(self.conv3(x)))
        return self.head(x.view(x.size(0), -1))
```

### Input extraction

The code below are utilities for extracting and processing rendered
images from the environment. It uses the ``torchvision`` package, which
makes it easy to compose image transforms. Once you run the cell it will
display an example patch that it extracted.

```
resize = T.Compose([T.ToPILImage(),
                    T.Resize(40, interpolation=Image.CUBIC),
                    T.ToTensor()])

# This is based on the code from gym.
screen_width = 600

def get_cart_location():
    world_width = env.x_threshold * 2
    scale = screen_width / world_width
    return int(env.state[0] * scale + screen_width / 2.0)  # MIDDLE OF CART

def get_screen():
    screen = env.render(mode='rgb_array').transpose(
        (2, 0, 1))  # transpose into torch order (CHW)
    # Strip off the top and bottom of the screen
    screen = screen[:, 160:320]
    view_width = 320
    cart_location = get_cart_location()
    if cart_location < view_width // 2:
        slice_range = slice(view_width)
    elif cart_location > (screen_width - view_width // 2):
        slice_range = slice(-view_width, None)
    else:
        slice_range = slice(cart_location - view_width // 2,
                            cart_location + view_width // 2)
    # Strip off the edges, so that we have a square image centered on a cart
    screen = screen[:, :, slice_range]
    # Convert to float, rescare, convert to torch tensor
    # (this doesn't require a copy)
    screen = np.ascontiguousarray(screen, dtype=np.float32) / 255
    screen = torch.from_numpy(screen)
    # Resize, and add a batch dimension (BCHW)
    return resize(screen).unsqueeze(0).to(device)

env.reset()

plt.figure()
plt.imshow(get_screen().cpu().squeeze(0).permute(1, 2, 0).numpy(),
           interpolation='none')
plt.title('Example extracted screen')
plt.show()

```

Training
--------

### Hyperparameters and utilities

This cell instantiates our model and its optimizer, and defines some
utilities:

-  ``select_action`` - will select an action accordingly to an epsilon
   greedy policy. Simply put, we'll sometimes use our model for choosing
   the action, and sometimes we'll just sample one uniformly. The
   probability of choosing a random action will start at ``EPS_START``
   and will decay exponentially towards ``EPS_END``. ``EPS_DECAY``
   controls the rate of the decay.
-  ``plot_durations`` - a helper for plotting the durations of episodes,
   along with an average over the last 100 episodes (the measure used in
   the official evaluations). The plot will be underneath the cell
   containing the main training loop, and will update after every
   episode.

```
BATCH_SIZE = 128
GAMMA = 0.999
EPS_START = 0.9
EPS_END = 0.05
EPS_DECAY = 200
TARGET_UPDATE = 10

policy_net = DQN().to(device)
target_net = DQN().to(device)
target_net.load_state_dict(policy_net.state_dict())
target_net.eval()

optimizer = optim.RMSprop(policy_net.parameters())
memory = ReplayMemory(10000)

steps_done = 0

def select_action(state):
    global steps_done
    sample = random.random()
    eps_threshold = EPS_END + (EPS_START - EPS_END) * \
        math.exp(-1. * steps_done / EPS_DECAY)
    steps_done += 1
    if sample > eps_threshold:
        with torch.no_grad():
            return policy_net(state).max(1)[1].view(1, 1)
    else:
        return torch.tensor([[random.randrange(2)]], device=device, dtype=torch.long)

episode_durations = []

def plot_durations():
    plt.figure(2)
    plt.clf()
    durations_t = torch.tensor(episode_durations, dtype=torch.float)
    plt.title('Training...')
    plt.xlabel('Episode')
    plt.ylabel('Duration')
    plt.plot(durations_t.numpy())
    # Take 100 episode averages and plot them too
    if len(durations_t) >= 100:
        means = durations_t.unfold(0, 100, 1).mean(1).view(-1)
        means = torch.cat((torch.zeros(99), means))
        plt.plot(means.numpy())

    plt.pause(0.001)  # pause a bit so that plots are updated
    if is_ipython:
        display.clear_output(wait=True)
        display.display(plt.gcf())
```

### Training loop

Finally, the code for training our model.

Here, you can find an ``optimize_model`` function that performs a
single step of the optimization. It first samples a batch, concatenates
all the tensors into a single one, computes $Q(s_t, a_t)$ and
$V(s_{t+1}) = \max_a Q(s_{t+1}, a)$, and combines them into our
loss. By defition we set $V(s) = 0$ if $s$ is a terminal
state. We also use a target network to compute $V(s_{t+1})$ for
added stability. The target network has its weights kept frozen most of
the time, but is updated with the policy network's weights every so often.
This is usually a set number of steps but we shall use episodes for
simplicity.

```
def optimize_model():
    if len(memory) < BATCH_SIZE:
        return
    transitions = memory.sample(BATCH_SIZE)
    # Transpose the batch (see http://stackoverflow.com/a/19343/3343043 for
    # detailed explanation).
    batch = Transition(*zip(*transitions))

    # Compute a mask of non-final states and concatenate the batch elements
    non_final_mask = torch.tensor(tuple(map(lambda s: s is not None,
                                          batch.next_state)), device=device, dtype=torch.uint8)
    non_final_next_states = torch.cat([s for s in batch.next_state
                                                if s is not None])
    state_batch = torch.cat(batch.state)
    action_batch = torch.cat(batch.action)
    reward_batch = torch.cat(batch.reward)

    # Compute Q(s_t, a) - the model computes Q(s_t), then we select the
    # columns of actions taken
    state_action_values = policy_net(state_batch).gather(1, action_batch)

    # Compute V(s_{t+1}) for all next states.
    next_state_values = torch.zeros(BATCH_SIZE, device=device)
    next_state_values[non_final_mask] = target_net(non_final_next_states).max(1)[0].detach()
    # Compute the expected Q values
    expected_state_action_values = (next_state_values * GAMMA) + reward_batch

    # Compute Huber loss
    loss = F.smooth_l1_loss(state_action_values, expected_state_action_values.unsqueeze(1))

    # Optimize the model
    optimizer.zero_grad()
    loss.backward()
    for param in policy_net.parameters():
        param.grad.data.clamp_(-1, 1)
    optimizer.step()
```

Below, you can find the main training loop. At the beginning we reset
the environment and initialize the ``state`` Tensor. Then, we sample
an action, execute it, observe the next screen and the reward (always
1), and optimize our model once. When the episode ends (our model
fails), we restart the loop.

Below, `num_episodes` is set small. You should download
the notebook and run lot more epsiodes.

```
num_episodes = 50
for i_episode in range(num_episodes):
    # Initialize the environment and state
    env.reset()
    last_screen = get_screen()
    current_screen = get_screen()
    state = current_screen - last_screen
    for t in count():
        # Select and perform an action
        action = select_action(state)
        _, reward, done, _ = env.step(action.item())
        reward = torch.tensor([reward], device=device)

        # Observe new state
        last_screen = current_screen
        current_screen = get_screen()
        if not done:
            next_state = current_screen - last_screen
        else:
            next_state = None

        # Store the transition in memory
        memory.push(state, action, next_state, reward)

        # Move to the next state
        state = next_state

        # Perform one step of the optimization (on the target network)
        optimize_model()
        if done:
            episode_durations.append(t + 1)
            plot_durations()
            break
    # Update the target network
    if i_episode % TARGET_UPDATE == 0:
        target_net.load_state_dict(policy_net.state_dict())

print('Complete')
env.render()
plt.show()
```

    <Figure size 432x288 with 0 Axes>

    Complete

    <Figure size 432x288 with 0 Axes>

```
frames = []
for i in range(3):
    env.reset()
    last_screen = get_screen()
    current_screen = get_screen()
    state = current_screen - last_screen
    done = False
    R = 0
    t = 0
    while not done and t < 200:
        frames.append(env.render(mode = 'rgb_array'))
        action = select_action(state)
        obs, r, done, _ = env.step(action.item())
        R += r
        t += 1
        if not done:
            next_state = current_screen - last_screen
        else:
            next_state = None
        state = next_state
    print('test episode:', i, 'R:', R)
env.render()

# make a video to display
plt.figure(figsize=(frames[0].shape[1] / 72.0, frames[0].shape[0] / 72.0), dpi = 72)
patch = plt.imshow(frames[0])
plt.axis('off')
animate = lambda i: patch.set_data(frames[i])
ani = matplotlib.animation.FuncAnimation(plt.gcf(), animate, frames=len(frames), interval = 50)
HTML(ani.to_jshtml())
```

# Deep Neural Net

A neural net architecture:

```
self.model = Sequential()
self.model.add(Dense(24, input_shape=(observation_space,), activation="relu"))
self.model.add(Dense(24, activation="relu"))
self.model.add(Dense(self.action_space, activation="linear"))
self.model.compile(loss="mse", optimizer=Adam(lr=0.001))
```

Its size is suited for the project’s modest complexity, but feel free to play with its structure and parameters.

Trial overview example

Solved trials overview


# Lecture 7: Bias-Variance Tradeoff, Regularization
## 4/1/19

### Hosted by and maintained by the [Statistics Undergraduate Students Association (SUSA)](https://susa.berkeley.edu). Authored by [Ajay Raj](mailto:araj@berkeley.edu), [Nichole Sun](mailto:nicholesun@berkeley.edu), [Rosa Choe](mailto:rosachoe@berkeley.edu), [Calvin Chen](mailto:chencalvin99@berkeley.edu), and [Roland Chin](mailto:rond24933chn@berkeley.edu).

### Table Of Contents
* [Recap](#recap)
* [Bias-Variance Tradeoff](#bv-tradeoff)
    * [Bias](#bias)
    * [Variance](#variance)
    * [The Tradeoff](#the-tradeoff)
    * [Polynomial Regression](#polynomial-regression)
* [Regularization](#regularization)
    * [Ridge](#ridge)
    * [LASSO](#lasso)
    * [Visualizing Ridge and Lasso](#visualizing-ridge-and-lasso)
    * [Regularization and Bias Variance](#regularization-and-bias-variance)
    * [Lambda](#lambda)
    * [Validation on Lambda](#validation-on-lambda)
* [Exercises](#exercises)

```python
import matplotlib.pyplot as plt
import random
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from plotting import overfittingDemo, plot_multiple_linear_regression, ridgeRegularizationDemo, lassoRegularizationDemo
from scipy.optimize import curve_fit
from sklearn.metrics import mean_squared_error
import warnings
warnings.filterwarnings('ignore')

%matplotlib inline
```

# Recap

High bias corresponds to underfitting. If we look at the first model, the points seem to follow some sort of curve, but our predictor is linear and therefore, unable to capture all the points. In this case, we have chosen a model which is not complex enough to accurately capture all the information from our data set.

If we look at the last model, the predictor is now overly complex because it adjusts based on every point in order to get as close to every data point as possible. In this case, the model changes too much based on small fluctuations caused by insignificant details in the data. This model is fitting more to noise than signal.

# Bias-Variance Tradeoff

Today we'll perform **model evaluation**, where we'll judge how our linear regression models actually perform. Last week, we talked about **loss functions**, which describes a numerical value for how far your model is from the true values.

$$\text{Mean Squared Error: } \frac{1}{n}\sum_{i=1}^n \left(y_i - f(x_i)\right)^2$$

In this loss function, $y_i$ is a scalar, and $x_i$ is a $p$ dimensional vector, because there are $p$ features. This loss is called **mean squared error**, or **MSE**.

Now, we'll talk about other ways to evaluate a model.

**First, let's define some terms.**

We can say that everything in the universe can be described with the following equation:

$$y = h(x) + \epsilon$$

- $y$ is the quantity you are trying to model or approximate
- $x$ are the features (independent variables)
- $h$ is the **true model** for $y$ in terms of $x$
- $\epsilon$ represents **noise**, a random variable which has mean zero

Let $f$ be your model for $y$ in terms of $x$.

## Bias

When evaluating a model, the most intuitive first step is to look at how well the model performs. For classification, this may be the percentage of data points correctly classified, or for regression it may be how close the predicted values are to actual. The **bias** of a model is *a measure of how close our prediction is to the actual value on average from an average model*.

Note that bias is **not** a measure of a single model, it encapuslates the scenario in which we collect many datasets, create models for each dataset, and average the error over all of models. Bias is not a measure of error for a single model, but a more abstract concept describing the average error over all errors. A low value for the bias of a model describes that on average, our predictions are similar to the actual values.

## Variance
The **variance** of  a model relates to the variance of the distribution of all models. In the previous section about bias, we envisoned the scenario of collecting many datasets, creating models for each dataset, and averaging the error overall the datasets. Instead, the variance of a model describes the variance in prediction. While we might be able to predict a value very well on average, if the variance of predictions is very high this may not be very helpful, as when we train a model we only have one such instance, and a high model variance tells us little about the true nature of the predictions. A low variance describes that our model will not predict very different values for different datasets.

**We can take a look at how bias and variance differ and can be explained in a dataset with the following diagram:**

Image from http://scott.fortmann-roe.com/docs/BiasVariance.html

The image describes what bias and variance are in a more simplified example. Consider that we would like to create a model that selects a point close to the center. The models on the top row have low bias, meaning the center of the cluster is close to the red dot on the target. The models on the left column have low variance, the clusters are quite tight, meaning our predictions are close together.

**Question: What do the blue dots represent? What about the bullseye?**

**Question: What is the order of best scenarios?**

## The Tradeoff

We are trying to minimize **expected error**, or the average **MSE** over all datasets. It turns out (with some advanced probability gymnastics), that:

$$\text{Mean Squared Error} = \text{Noise Variance} + \text{Bias}^2 + \text{Variance}$$

Note that $\text{Noise Variance}$ is constant: we assume there is some noise, and $\text{moise variance}$ is simply a value that describes how noisy your dataset will be on average. This is often also called "irreducible noise", as it is literally irreducible, we cannot avoid this.

Furthermore, notice that the equation above is the sum of (squared) bias and variance. Thus there is a literal tradeoff between these two, since decreasing one increases the other. This defines what is known as the **bias variance tradeoff**.

Image from http://scott.fortmann-roe.com/docs/BiasVariance.html

**Why does this happen?**

At some point, as we decrease **bias**, instead of getting closer to the **true model** $h$, we go past and try to fit to the $\epsilon$ (noise) that is part of our current dataset. This is equivalent to making our model more noisy, or **overfit** on our dataset, which means that over all datasets, it has more **variance**.

**Questions for understanding**:
> 1. Where does underfitting and overfitting lie in the graph above? How do they relate to bias and variance?
> 2. Why can't we usually just make a bunch of models with low bias and high variance and average them?
> 3. Why is low variance important in models?

## Polynomial Regression

Let's revisit the polynomial problem that we have discussed.

In this case, if our model has degree $d$, we have $d + 1$ features: $x = [x^0, x^1, ..., x^d]$. Now, we have a linear model with $d + 1$ features:

$$\hat{f}(x) = \sum_{i=0}^{d} a_i x^i$$

Model complexity in this case is the degree of the polynomial. As we saw last week, as $d$ increases, model complexity increases. The model gets better, but then gets erratic. This directly corresponds to the bias-variance graph above.

```python
overfittingDemo()
```

As we saw from last time, the best model was a degree 3 model.

# Regularization

We talked about **validation** as a means of combating overfitting. However, this is not the only method to combat overfitting. Another method to do so is to add *regularization* terms to our loss function. **Regularization** basically penalizes complexity in our models. This allows us to add explanatory variables to our model without worrying as much about overfitting. Here's what our ordinary least squares model looks like with a regularization term:

$$\hat{\boldsymbol{\theta}} = \arg\!\min_\theta \sum_{i=1}^n (y_i - f_\boldsymbol{\theta}(x_i))^2 + \lambda R(\boldsymbol{\theta})$$

We've written the model a little differently here, but the first term is the same as the ordinary least squares regression model you learned last week. This time it's just generalized to any function of $x$ where $\theta$ is a list of parameters, or weights on our explanatory variables, such as coefficients to a polynomial. We're minimizing a loss function to find the best coefficients for our model.

The second term is the **regularization** term. The $\lambda$ parameter in front of it dictates how much we care about our regularization term – the higher $\lambda$ is, the more we penalize large weights, and the more the regularization makes our weights deviate from OLS.

**Question**: What happens when $\lambda = 0$?

So, what is $R(\theta)$ in the equation? There are a variety of different regularization functions that could take its place, but today, we'll just talk about the two most common types of functions: **ridge regression** and **LASSO regression**.

$$\begin{align}\text{ Ridge (L2 Norm)}: &\  R(\boldsymbol{\theta}) = \|\theta\|_2^2 = \sum_{i=1}^p \theta_i^2\\
\text{ LASSO (L1 Norm)}: &\ R(\boldsymbol{\theta}) = \|\theta\|_1=\sum_{i=1}^p \lvert \theta_i\rvert\end{align}$$

## Ridge
$$\hat{\boldsymbol{\theta}} = \arg\!\min_\theta \sum_{i=1}^n (y_i - f_\boldsymbol{\theta}(x_i))^2 + \lambda \|\theta\|_2^2$$
In **ridge** regression, the regularization function is the sum of squared weights. One nice thing about ridge regression is that there is always a unique, mathematical solution to minimizing the loss of that term. The solution involves some linear algebra, which we won't get into in this notebook, but the existence of this formula makes this minimization computationally easy to solve!

$$\hat{\boldsymbol{\theta}} = \left(\boldsymbol{X}^T \boldsymbol{X} + \lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{Y}$$

If you recall, the solution to linear regression was of the form:

$$\hat{\boldsymbol{\theta}} = \left(\boldsymbol{X}^T \boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{Y}$$

And we said that the $\boldsymbol{X}^T\boldsymbol{X}$ isn't always invertible. **What about $\boldsymbol{X}^T \boldsymbol{X} + \lambda\boldsymbol{I}$?**

Turns out, this is always invertible! If you are familiar with linear algebra, this is equivalent to adding $\lambda$ to all the eigenvalues of $X^TX$.

**To see this in practice**, we'll first create a regular linear regression model, and compare how it does against models using regularization on the `mpg` dataset we used from last week! We'll be constructing models of `displacement` vs. `mpg`, and seeing the difference from there!

First, let's construct the `mpg_train` dataset!

```python
mpg = pd.read_csv("mpg.csv", index_col='name')# load mpg dataset
mpg = mpg.loc[mpg["horsepower"] != '?'].astype(float) # remove columns with missing horsepower values
mpg_train, mpg_test = train_test_split(mpg, test_size = .2, random_state = 0) # split into training set and test set
mpg_train, mpg_validation = train_test_split(mpg_train, test_size = .5, random_state = 0)
mpg_train.head()
```

**Exercise:** Now, let's create a regular linear regression model using the same process we've learned before (fitting, predicting, finding the loss)!

```python
from sklearn.linear_model import LinearRegression

x_train = np.vander(mpg_train["displacement"], 13)
y_train = mpg_train[["mpg"]]

x_validation = np.vander(mpg_validation["displacement"], 13)
y_validation = mpg_validation[["mpg"]]

# instantiate your model
linear_model = ...

# fit the model
...
# make predictions on validation set
linear_prediction = ...
# find mean squared error
linear_loss = ...

print("Root Mean Squared Error of Linear Model: {:.2f}".format(linear_loss))
```

**Exercise:** Using what you did above as reference, do the same using a Ridge regression model!

```python
from sklearn.linear_model import Ridge

...
ridge_loss = ... # mean squared error of ridge model

print("Root Mean Squared Error of Linear Model: {:.2f}".format(linear_loss))
print("Root Mean Squared Error of Ridge Model: {:.2f}".format(ridge_loss))
```

## LASSO
$$\hat{\boldsymbol{\theta}} = \arg\!\min_\theta \sum_{i=1}^n (y_i - f_\boldsymbol{\theta}(x_i))^2 + \lambda \|\theta\|_1$$
In **LASSO** regression, the regularization function is **the sum of absolute values of the weights**. One key thing to note about **LASSO** is that it is **sparsity inducing**, meaning it forces weights to be zero values rather than really small values (which can happen in **Ridge Regression**), leaving you with fewer explanatory variables in the resulting model! Unlike Ridge Regression, LASSO doesn't necessarily have a unique solution that can be solved for with linear algebra, so there's no formula that determines what the optimal weights should be.

```python
from sklearn.linear_model import Lasso

...
lasso_loss = ... # mean squared error of lasso model

print("Root Mean Squared Error of Linear Model: {:.2f}".format(linear_loss))
print("Root Mean Squared Error of LASSO Model: {:.2f}".format(lasso_loss))
```

As we can see, both **Ridge Regression and LASSO Regression** minimized the loss of our linear regression models, so maybe penalizing some features allowed us to prevent overfitting on our dataset!

## Visualizing Ridge and LASSO
We went through a lot about **ridge** and **LASSO**, but we didn't really get into what they look like for understanding! And so, here are some visualizations that might help build the intution behind some of the characteristics of these two regularization methods.

Another way to describe the modified minimization function above is that it's the same loss function as before, with the *additional constraint* that $R(\boldsymbol{\theta}) \leq t$. Now, $t$ is related to $\lambda$ but the exact relationship between the two parameters depends on your data. Regardless, let's take a look at what this means in the two-dimensional case. For ridge,

$$\theta_0^2 + \theta_1^2 = t$$

Lasso is of the form $$\left|\theta_0\right| + \left|\theta_1\right| =t$$

### Norm Balls

Let's take at another visualization that may help build some intuition behind how both of these regularization methods work!

Image from https://upload.wikimedia.org/wikipedia/commons/f/f8/L1_and_L2_balls.svg.

Image from https://towardsdatascience.com/regression-analysis-lasso-ridge-and-elastic-net-9e65dc61d6d3.

The rhombus and circle as a visualization of the regularization term, while the blue circles are the topological curves/level sets representing the loss function based on the weights. You want to minimize the sum of these, which means you want to minimize each of those. The point that minimizes the sum is the minimum point at which they intersect.

**Question**: Based on these visualizations, could you explain why LASSO is sparsity-inducing?

Turns out that the $L2-norm$ is always some sort of smooth surface, from a circle in 2D to a sphere in 3D. On the other hand, LASSO always has sharp corners. In higher dimensions, it forms an octahedron. This is exactly the feature that makes it sparsiy-inducing. As you might imagine, just as humans are more likely to bump into sharp corners than smooth surfaces, the loss term is also most likely to intersect the $L2-norm$ at one of the corners.

## Regularization and Bias Variance
As we mentioned earlier, **bias** is the average linear squares loss term across multiple models of the same family (e.g. same degree polynomial) trained on separate datasets. **Variance** is the average variance of the weight vectors (coefficients) on your features.

Without the regularization term, we’re just minimizing bias; the regularization term means we won’t get the lowest possible bias, but we’re exchanging that for some lower variance so that our model does better at generalizing to data points outside of our training data.

## Lambda

We said that $\lambda$ is how much we care about the regularization term, but what does that look like? Let's return to the polynomial example from last week, and see what the resulting models look like with different values of $\lambda$ given a degree 8 polynomial.

```python
ridgeRegularizationDemo([0, 0.5, 1.0, 5.0, 10.0], 8)
```

From the diagram above, it's difficult to determine which lambda value help fit our model the closest to the true data distribution. So, **how do we know what to use for $\lambda$ (or `alpha` in the `sklearn.linear_model` constructors)?**

That's right, let's use the process of **validation** here! In this case, we'd be finding the value for lambda that **minimizes the loss for ridge regression, and then the one that minimizes the loss for LASSO regression**!

## Validation on Lambda
Let's try to find the best $\lambda$ for the degree 20 polynomial on `displacement` from above.

```python
lambdas = np.arange(0, 200) # create a list of potential lambda values

# create a list containing the corresponding mean_squared_error for each lambda usinb both ridge and lasso regression
ridge_errors = []
lasso_errors = []

for l in lambdas:

    ridge_errors.append(...)

    lasso_errors.append(...)

# finds the index of the minimum value in each list
answer = ridge_errors.index(min(ridge_errors)), lasso_errors.index(min(lasso_errors))
answer
```

As we can see from above, we've been able to determine which lambdas minimizes our ridge regression model and our LASSO regression model through validation by iterating through potential lambda values and finding the ones that minimize our loss for each model!

# Conclusion

Through the course of the notebook, we introduced two main concepts, **bias** and **variance**, and how the two relate with one another when it comes to finding the best model for our dataset! We also went into different methods we can use to minimize overfitting our model, and in turn lower variance, by taking look at a process called **regularization**. We saw that the two main regression models, **ridge regression** and **LASSO regression**, and saw the difference between the two (ridge -> penalize large weights, LASSO -> make weights sparse). We also took a look at different visualizations between the two to build up some more intuition behind how they work, through **graphs** and **norm balls**. Finally, we went through a familiar process (**validation**) to determine what the best values of lambda were for our models, officially ending our journey of **bias** and **variance**, and how we can minimze both in our models!

# Congratulations! You are now a Bias + Variance master!

## Exercises

1. What happens as $\lambda$ increases?
    1. bias increases, variance increases
    2. bias increases, variance decreases
    3. bias decreases, variance increases
    4. bias decreases, variance decreases

**Insert answer here**:

2. **True** or **False**? Bias is how much error your model makes.

**Insert answer here:**

3. What is **sparsity**?

**Insert answer here:**

4. For each of the following, choose **ridge**, **lasso**, **both**, or **neither**:
    1. L1-norm
    2. L2-norm
    3. Induces sparsity
    4. Has analytic (mathematical) solution
    5. Increases bias
    6. Increases variance

**Insert answer here:**

5. Which one is better to use: Ridge Regression or LASSO Regression?

**Insert answer here:**

### Congrats! You've finished our few conceptual questions, now you can help out the rest of your peers and use the rest of the time to work on the intermediate project with your project group!


<p align="center">

</p>

## Interactive Simple Bootstrap Demonstration

### Boostrap for Uncertainty in Sample Statistics, The Red and Green Balls with a Cowboy Hat Tutorial

* in class I bring in 3 red balls, 2 green balls and my cowboy hat, yes I have one, recall I was a farmhand in Alberta, Canada

* then I have students volunteer, one holds the hat, one draws balls with replacement and one scribes the results on the board

* through multiple bootstrap smaple sets we demonstrate the use of bootstrap to calculate uncertainty in the proportion from the sample itself

* with this workflow we all provide an interactive plot demonstration with matplotlib and ipywidget packages

#### Michael Pyrcz, Associate Professor, University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1) | [GeostatsPy](https://github.com/GeostatsGuy/GeostatsPy)

#### Bootstrap

Uncertainty in the sample statistics
* one source of uncertainty is the paucity of data.
* do 200 or even less wells provide a precise (and accurate estimate) of the mean? standard deviation? skew? P13?

Would it be useful to know the uncertainty in these statistics due to limited sampling?
* what is the impact of uncertainty in the mean porosity e.g. 20%+/-2%?

**Bootstrap** is a method to assess the uncertainty in a sample statistic by repeated random sampling with replacement.

Assumptions
* sufficient, representative sampling, identical, idependent samples

Limitations
1. assumes the samples are representative
2. assumes stationarity
3. only accounts for uncertainty due to too few samples, e.g. no uncertainty due to changes away from data
4. does not account for boundary of area of interest
5. assumes the samples are independent
6. does not account for other local information sources

The Bootstrap Approach (Efron, 1982)

Statistical resampling procedure to calculate uncertainty in a calculated statistic from the data itself.
* Does this work?  Prove it to yourself, for uncertainty in the mean solution is standard error:

\begin{equation}
\sigma^2_\overline{x} = \frac{\sigma^2_s}{n}
\end{equation}

Extremely powerful - could calculate uncertainty in any statistic!  e.g. P13, skew etc.
* Would not be possible access general uncertainty in any statistic without bootstrap.
* Advanced forms account for spatial information and sampling strategy (game theory and Journel’s spatial bootstrap (1993).

Steps:

1. assemble a sample set, must be representative, reasonable to assume independence between samples

2. optional: build a cumulative distribution function (CDF)
    * may account for declustering weights, tail extrapolation
    * could use analogous data to support

3. For $\ell = 1, \ldots, L$ realizations, do the following:

    * For $i = \alpha, \ldots, n$ data, do the following:

        * Draw a random sample with replacement from the sample set or Monte Carlo simulate from the CDF (if available).

6. Calculate a realization of the sammary statistic of interest from the $n$ samples, e.g. $m^\ell$, $\sigma^2_{\ell}$. Return to 3 for another realization.

7. Compile and summarize the $L$ realizations of the statistic of interest.

This is a very powerful method.  Let's try it out.

#### Objective

Provide an example and demonstration for:

1. interactive plotting in Jupyter Notebooks with Python packages matplotlib and ipywidgets
2. provide an intuitive hands-on example of statistical boostrap

#### Getting Started

Here's the steps to get setup in Python with the GeostatsPy package:

1. Install Anaconda 3 on your machine (https://www.anaconda.com/download/).
2. Open Jupyter and in the top block get started by copy and pasting the code block below from this Jupyter Notebook to start using the geostatspy functionality.

#### Load the Required Libraries

The following code loads the required libraries.

```python
%matplotlib inline
from ipywidgets import interactive                        # widgets and interactivity
from ipywidgets import widgets
from ipywidgets import Layout
from ipywidgets import Label
from ipywidgets import VBox, HBox
import matplotlib.pyplot as plt                           # plotting
import numpy as np                                        # working with arrays
import pandas as pd                                       # working with DataFrames
import seaborn as sns                                     # for matrix scatter plots
from scipy.stats import triang                            # parametric distributions
from scipy.stats import binom
from scipy.stats import norm
from scipy.stats import uniform
from scipy.stats import triang
from scipy import stats                                   # statistical calculations
import random                                             # random drawing / bootstrap realizations of the data
from matplotlib.gridspec import GridSpec
```

#### Make a Synthetic Dataset

This is an interactive method to:

* select a parametric distribution

* select the distribution parameters

* select the number of samples and visualize the synthetic dataset distribution

```python
# parameters for the synthetic dataset
bins = np.linspace(0,1000,1000)

# interactive calculation of the sample set (control of source parametric distribution and number of samples)
l = widgets.Text(value='                                Simple Boostrap Demonstration, Michael Pyrcz, Associate Professor, The University of Texas at Austin',layout=Layout(width='950px', height='30px'))

a = widgets.IntSlider(min=0, max = 100, value = 2, step = 1, description = '$n_{red}$',orientation='horizontal',layout=Layout(width='400px', height='20px'))
a.style.handle_color = 'red'

b = widgets.IntSlider(min=0, max = 100, value = 3, step = 1, description = '$n_{green}$',orientation='horizontal',layout=Layout(width='400px', height='20px'))
b.style.handle_color = 'green'

c = widgets.IntSlider(min=1, max = 16, value = 3, step = 1, description = '$L$',orientation='horizontal',layout=Layout(width='400px', height='20px'))
c.style.handle_color = 'gray'

ui = widgets.HBox([a,b,c],)                               # basic widget formatting
ui2 = widgets.VBox([l,ui],)

def f_make(a, b, c):                                      # function to take parameters, make sample and plot
    red_freq = make_data(a, b, c)

    labels = ['Red', 'Green']
    nrows = np.round(np.sqrt(c)+0.4,0); ncols = np.round(c / nrows + 0.4,0)
    plt.clf()

    for i in range(0, c):
        plt.subplot(ncols,nrows,i + 1)
        draw = [red_freq[i],a + b - red_freq[i]]
        plt.grid(zorder=0, color='black', axis = 'y', alpha = 0.2); plt.ylim(0,a + b);
        plt.ylabel('Frequency'); plt.xlabel('Balls Drawn')
        plt.yticks(np.arange(0,a + b + 1,1))
        barlist = plt.bar(labels,draw, edgecolor = "black", linewidth = 3, alpha = 0.4); plt.title('Realization #' + str(i+1),zorder = 1)
        barlist[0].set_color('r'); barlist[1].set_color('g')

    plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.2 * nrows, wspace=0.2, hspace=0.2)
    plt.show()

def make_data(a, b, c):                                   # function to check parameters and make sample
    prop_red = np.zeros(c)
    for i in range(0, c):
        prop_red[i] = np.random.multinomial(a+b,[a/(a+b),b/(a+b)], size = 1)[0][0]
    return prop_red

# connect the function to make the samples and plot to the widgets
interactive_plot = widgets.interactive_output(f_make, {'a': a, 'b': b, 'c': c})
interactive_plot.clear_output(wait = True)                # reduce flickering by delaying plot updating
```

### Simple Bootstrap Demonstration - Drawing Red and Green Balls from a Virtual Cowboy Hat

* drawing red and green balls from a hat with replacement to access uncertainty in the proportion

* interactive plot demonstration with ipywidget, matplotlib packages

#### Michael Pyrcz, Associate Professor, University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1) | [GeostatsPy](https://github.com/GeostatsGuy/GeostatsPy)

### The Problem

Let's simulate bootstrap, resampling with replacement from a hat with $n_{red}$ and $n_{green}$ balls

* **$n_{red}$**: number of red balls in the sample (placed in the hat)

* **$n_{green}$**: number of green balls in the sample (placed in the hat)

* **$L$**: number of bootstrap realizations

```python
display(ui2, interactive_plot)                            # display the interactive plot
```

    VBox(children=(Text(value='                                Simple Boostrap Demonstration, Michael Pyrcz, Assoc…

    Output()

#### Summarizing Bootstrap Uncertainty

* Run more bootstrap realizations and evaluate the uncertianty model

Now instead of looking at each bootstrap result, let's make many and summarize with:

* **box and whisker plot** of the red and green ball frequencies

* **histograms** of the red and green ball frequencies.

```python
# parameters for the synthetic dataset
bins = np.linspace(0,1000,1000)

# interactive calculation of the sample set (control of source parametric distribution and number of samples)
l2 = widgets.Text(value='                                Simple Boostrap Demonstration, Michael Pyrcz, Associate Professor, The University of Texas at Austin',layout=Layout(width='950px', height='30px'))

a2 = widgets.IntSlider(min=0, max = 100, value = 2, step = 1, description = '$n_{red}$',orientation='horizontal',layout=Layout(width='400px', height='20px'))
a2.style.handle_color = 'red'

b2 = widgets.IntSlider(min=0, max = 100, value = 3, step = 1, description = '$n_{green}$',orientation='horizontal',layout=Layout(width='400px', height='20px'))
b2.style.handle_color = 'green'

c2 = widgets.IntSlider(min=5, max = 1000, value = 3, step = 1, description = '$L$',orientation='horizontal',layout=Layout(width='400px', height='20px'))
c2.style.handle_color = 'gray'

uib = widgets.HBox([a2,b2,c2],)                               # basic widget formatting
uib2 = widgets.VBox([l2,uib],)

def s_make(a, b, c):                                      # function to take parameters, make sample and plot
    red_freq = make_data(a, b, c)
    green_freq = (a + b) - red_freq
    prop_red = red_freq / (a + b)
    labels = ['Red Balls', 'Green Balls']
    bins = np.linspace(0,a + b, a + b)

    fig = plt.figure(constrained_layout=False)
    gs = GridSpec(3, 2, figure=fig)

    ax1 = fig.add_subplot(gs[:, 0])
    ax1.boxplot([red_freq,green_freq],labels = labels, notch = True, sym = '+')
    ax1.set_ylim(0,a + b)
    ax1.grid(zorder=0, color='black', axis = 'y', alpha = 0.2); plt.ylim(0,a + b);
    ax1.set_ylabel('Frequency'); ax1.set_xlabel('Balls Drawn');ax1.set_title('Bootstrap Frequency Distributions')
    ax1.set_yticks(np.arange(0,a + b + 1,5))

    ax2 = fig.add_subplot(gs[0, 1])
    ax2.hist(red_freq,alpha=0.2,color="red",edgecolor="black", bins = bins)
    ax2.set_title('Red Ball Bootstrap Frequency Distributions'); ax2.set_xlabel('Number Red Balls'); ax2.set_ylabel('Frequency')

    ax3 = fig.add_subplot(gs[1, 1])
    ax3.hist(green_freq,alpha=0.2,color="green",edgecolor="black", bins = bins)
    ax3.set_title('Green Ball Bootstrap Frequency Distributions'); ax3.set_xlabel('Proportion of Red Balls'); ax3.set_ylabel('Frequency')

    ax4 = fig.add_subplot(gs[2, 1])
    ax4.hist(prop_red,alpha=0.2,color="red",edgecolor="black", bins = np.linspace(0,1,50))
    ax4.set_title('Uncertainty in Proportion of Red Balls'); ax4.set_xlabel('Proportion of Red Balls'); ax4.set_ylabel('Frequency')

    print('Uncertainty in Proportion of Red Balls: ')
    print('P10: ' + str(np.round(np.percentile(prop_red,10),2)))
    print('P50: ' + str(np.round(np.percentile(prop_red,50),2)))
    print('P90: ' + str(np.round(np.percentile(prop_red,90),2)))

    plt.subplots_adjust(left=0.0, bottom=0.0, right=2.5, top=3.0, wspace=0.2, hspace=0.3)
    plt.show()

# connect the function to make the samples and plot to the widgets
interactive_plot = widgets.interactive_output(s_make, {'a': a2, 'b': b2, 'c': c2})
interactive_plot.clear_output(wait = True)                # reduce flickering by delaying plot updating
```

### Simple Bootstrap Demonstration - Summarizing the Uncertainty in Frequencies

* drawing red and green balls from a hat with replacement to access uncertainty in the proportion

* run many bootstrap realizations and summarize the results

* interactive plot demonstration with ipywidget, matplotlib packages

#### Michael Pyrcz, Associate Professor, University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1) | [GeostatsPy](https://github.com/GeostatsGuy/GeostatsPy)

### The Problem

Let's simulate bootstrap, resampling with replacement from a hat with $n_{red}$ and $n_{green}$ balls

* **$n_{red}$**: number of red balls in the sample (placed in the hat)

* **$n_{green}$**: number of green balls in the sample (placed in the hat)

* **$L$**: number of bootstrap realizations

```python
display(uib2, interactive_plot)                           # display the interactive plot
```

    VBox(children=(Text(value='                                Simple Boostrap Demonstration, Michael Pyrcz, Assoc…

    Output()

#### Observations

Some observations:

* sampling distribution for proportions become discrete with too fee samples, only n cases are possible

* enough bootstrap realizations are rquired for stable statistics

#### Comments

This was a simple demonstration of interactive plots in Jupyter Notebook Python with the ipywidgets and matplotlib packages.

I have many other demonstrations on data analytics and machine learning, e.g. on the basics of working with DataFrames, ndarrays, univariate statistics, plotting data, declustering, data transformations, trend modeling and many other workflows available at https://github.com/GeostatsGuy/PythonNumericalDemos and https://github.com/GeostatsGuy/GeostatsPy.

I hope this was helpful,

*Michael*

#### The Author:

### Michael Pyrcz, Associate Professor, University of Texas at Austin
*Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions*

With over 17 years of experience in subsurface consulting, research and development, Michael has returned to academia driven by his passion for teaching and enthusiasm for enhancing engineers' and geoscientists' impact in subsurface resource development.

For more about Michael check out these links:

#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

#### Want to Work Together?

I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.

* Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I'd be happy to drop by and work with you!

* Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!

* I can be reached at mpyrcz@austin.utexas.edu.

I'm always happy to discuss,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin

#### More Resources Available at: [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

```python

```


Выделяется несколько структур потока: рассеянная пузырьковая(dispersed bubble), пробковая или снарядкая (slug), переходная(churn) и кольцевая(annular). Обычно пузырьковая структура наблюдается при больших дебитах жидкости. Однако существуют условия при которых она появляется и при низких дебитах жидкости. Такая структура называется пузырьковая(bubbly). Она проявляется только при углах близких к вертикальным или при больших дебитах. Существование такого режима определяется двумя условиями:

\begin{equation}
D >  19 \left[ \frac{(\rho_l - \rho_g)\sigma}{\rho_l^2 g}\right]^{0.5} -\space условие\spaceдля\spaceдиаметра
\end{equation}

\begin{equation}
D >  19 \left[ \frac{(\rho_l - \rho_g)\sigma}{\rho_l^2 g}\right]^{0.5} -\space условие\spaceдля\spaceдиаметра
\end{equation}

```python

```

```python

```