# Practical 4: Reinforcement Learning

## Introduction
In this practical we introduce the idea of reinforcement learning, discuss how it differs from supervised and unsupervised learning and then build an agent that learns to play a simple game called "Catcher".

## Learning Objectives
* Understand the relationship between the **environment** and the  **agent**
* Understand how a **policy** is used by an agent to select an action
* Describe how to implement a **run-loop** that controls the interaction between environement and agent.
* Understand how the **state**, **action** and **reward** are communicated between the agent and the environment.
* Be able to implement the a simple **policy-gradient** RL algorithm call **REINFORCE**
* Discover at least one potential issue with the REINFORCE algorithm.

```
#@title [RUN ME!] Install pre-requisites. { display-mode: "form" }
import os
import sys
import math

!git clone https://github.com/ntasfi/PyGame-Learning-Environment.git
os.chdir('PyGame-Learning-Environment')
!pip -q install -e .
!pip -q install pygame
os.chdir('/content')

sys.path.append('/content/PyGame-Learning-Environment')
os.environ["SDL_VIDEODRIVER"] = "dummy"  # prevent trying to open a window

!pip -q install moviepy

print('Installed pre-requisites...')
```

```
#@title [RUN ME!] Imports { display-mode: "form" }

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
from __future__ import unicode_literals

import os
import moviepy.editor as mpy
from ple import PLE
from ple.games import pong
from ple.games import pixelcopter
from ple.games import flappybird
from ple.games import catcher
from IPython import display
import numpy as np
import matplotlib.pyplot as plt
from collections import deque
import seaborn as sns

import tensorflow as tf
try:
  tf.enable_eager_execution()
  print('Running Eagerly')
except ValueError:
  print('Already running Eagerly')
```

```
#@title [RUN ME!] Helper Functions { display-mode: "form" }

def make_animation(images, fps=60, true_image=False):
  duration = len(images) / fps

  def make_frame(t):
    try:
      x = images[int(len(images) / duration * t)]
    except:
      x = images[-1]

    if true_image:
      return x.astype(np.uint8)
    else:
      return ((x + 1) / 2 * 255).astype(np.uint8)

  clip = mpy.VideoClip(make_frame, duration=duration)
  clip.fps = fps
  return clip

def progress(value, max=100, message=''):
  return display.HTML("""
      <progress
          value='{value}'
          max='{max}',
          style='width: 100%'
      >
          {value}
      </progress>
      <p>{message}</p>
  """.format(value=value, max=max, message=message))

def plot_rolling_returns(rolling_returns):
  sns.tsplot(rolling_returns)
  plt.title('Rolling Returns')
  plt.xlabel('# Epsiodes')
  plt.ylabel('Rolling Return')

def state_to_buckets(state, bucket_width=0.25):
  return tuple(math.ceil(s / bucket_width)-1 for s in state)
```

## Reinforcement Learning
So far we have encountered **supervised learning**, where we have an input and a target value or class that we want to predict. We have also encountered **unsupervised learning** where we are only given an input and look for patterns in that input. In this practical, we look into **reinforcement learning** which can loosely be defined as training an **agent** to maximise a numerical **reward** it obtains through interaction with an **environment**.

The environment defines a set of **actions** that an agent can take. The agent observes the current **state** of the environment, tries actions and *learns* a **policy** which is a distribution over the possible actions given a state of the environment.

The following diagram illustrates the interaction between the agent and environment. We will explore each of the terms in more detail throughout this practical.

## Outline
We will train an agent to play a very simple game called "Catcher" which is often used as a test bed for RL algorithms. In the process we will set up all the necessary framework to explore variations of the algorithm or switch to more advanced games! In particular, the steps we will follow in this practical are as follows:

1. Introduce the game environment, explore the states and actions available.
2. Create a simple agent that takes random actions
3. Write a run-loop which controls the interaction and manages the communication between the agent and environment
4. Implement a policy as a feed-forward neural network
5. Explain and implement the REINFORCE algorithm to learn how to play the game

## The Environment
The environment we consider is the game Catcher from the PyGame Learning Environment (PLE) library. The player (agent) controls a paddle that it must use to catch a falling fruit. Each time the game runs, the fruit falls from from the top to the bottom of the screen, starting at a different X coordinate (which doesn't change during the episode) and the paddle starts in a random location along the bottom of the screen. The player wins if they manage to catch the fruit and loses if it falls to the ground.

### Create the environment using PLE
For this game, we'll set things up so that the reward will be non-zero only at the end of each episode, and will be +1 for catching the fruit and -1 for missing it. For all other frames the reward will be zero.

**Note**: PLE has a number of games, which it wraps in a generic "environment".
So, in this case, both ```evironment``` and ```game``` constitute our environment. ```environment``` allows us to perform actions
and returns states and rewards, while ```game``` handles the specifics of Catcher (or whichever other game we decide to use)

```
# Create an instance of the catcher game
game = catcher.Catcher(init_lives=1)
game_height = game.height
game_width = game.width

frame_skip = 3  # Skip 3 frames at each step to speed up the game

# Wrap the game in a PLE environment and configure the rewards
environment = PLE(game, display_screen=False, force_fps=True,
                  reward_values={'win': 1.0, 'loss': -1.0, 'negative': 0.0},
                  frame_skip=frame_skip)

# The reward_values dictionary above allows us to override the default reward structure provided by PLE.
# win and loss specify the rewards for winning or losing an episode, while positive and negative
# specify the rewards received for positive or negative events that can occur during the game.
# If you change the game, start by *removing* the overrides and see what the default is before deciding if
# you want to modify it.

# Initialise the environment
environment.init()
```

### What does the state look like?
The environment provides a rendering of the virtual 'screen' of the game to an RGB image, by using the method ```getScreenRGB```. For this practical, in the interests of simplicity and quick trianing time, we use the *game state* directly, which provides a summary (in a dictionary) of important peices of information making up the current state of the game.

```
print('Current game state:', game.getGameState())
```

    Current game state: {'fruit_x': 8, 'player_vel': 0.0, 'player_x': 26, 'fruit_y': -8}

### What actions are available?
The following cell prints the actions that are available in the current game, which are represented with numerical codes. The PLE environment wrapper also adds an additional ```None``` action which means "do nothing".

```
print('Game actions:', game.actions)
print('Environment actions:', environment.getActionSet())
```

    Game actions: {'right': 100, 'left': 97}
    Environment actions: [100, 97, None]

```
#@title [RUN ME!] Setup for the next section { display-mode: "form" }
# Maintain some variables for the next task
environment.reset_game()
observed_states = []
observed_actions = []
observed_rewards = []
observed_states.append(game.getGameState())
```

### Exploratory Task
Run the cell below which defines a function that takes a given action in the environment, using the environment's ```act``` method and then renders both the previous and current game screen. Change the action in the drop-down to the right of the 2nd code cell, which calls this function with the chosen action. Observe what happens to the environment(game) state and reward. By running the cell multiple times (and changing the action) until the game comes to an end (when you either win or lose), you will manually create an **episode**, which is a sequence of states, actions and rewards until a termination condition is reached. If an episode $i$ consists of $T_i$ steps and we denote the state, action and reward at step $t$ in episode $i$ respectively as $s_{i, t}$, $a_{i,t}$ and $r_{i,t}$, then in this task, we create a *trajectory* $\tau_i = (s_{i, 1}, a_{i, 1}, r_{i, 1}, ..., a_{i, T_i-1}, r_{i, T_i-1}, s_{i, T_i})$

#### Question
Notice how the paddle sometimes moves even if you take the "None" action? Can you think of why this happens?

#### Notes
* If you want to run another episode, re-run the code cell above titled "Setup for the next section" to reset the environment
* This particular game returns a reward of $0$ at each step and a final reward of $-1$ or $1$ at the end of the episode depending on whether you lose or win. Other games may have different reward structures!

```
previous_frame = environment.getScreenRGB().transpose([1, 0, 2])

def take_action(action):
  global previous_frame

  # Look up the action code from the description
  action_code = None if action == 'None' else game.actions[action]

  # Take the selected action in the environment
  print('Taking action: {} ({})'.format(action, action_code))
  reward = environment.act(action_code)

  observed_actions.append(action)
  observed_rewards.append(reward)

  # Print and display the current state and reward
  state = game.getGameState()
  print('Game state:', state)
  print('Reward received: ', reward)

  observed_states.append(state)

  if reward > 0 or environment.game_over():
    print('Game over, you', 'WON' if reward > 0 else 'LOST')
    print('The episode trajectory was:')
    for s, a, r in zip(observed_states, observed_actions, observed_rewards):
      print('State:', s, 'Action:', a, 'Reward:', r)
    print('Terminal state:', observed_states[-1])

  current_frame = environment.getScreenRGB().transpose([1, 0, 2])

  fig = plt.figure(figsize=(10, 20))

  ax = plt.subplot(1, 2, 1)
  plt.imshow(previous_frame)
  ax.grid(False)
  ax.set_title('PREVIOUS FRAME')

  ax = plt.subplot(1, 2, 2)
  plt.imshow(current_frame)
  ax.grid(False)
  ax.set_title('CURRENT FRAME')

  previous_frame = current_frame
```

```
action = "None"  #@param ['right', 'left', 'None']
take_action(action)
```

## The Agent
We now turn to the agent. An agent receives the current state and (previous) reward from the environment, then uses an internal policy to determine an action to take. We implement an agent as a Python [**class**](https://en.wikibooks.org/wiki/A_Beginner%27s_Python_Tutorial/Classes), which is just a logical wrapper of variables and methods (functions) that operate on those variables. The methods our agent will have are the following:

* **Initialisation** (```__init__```):  Initialises the agent the first time it's created.
* **policy**: The policy is a function that returns a *distribution* over the possible actions, given the current state.
* **step**: This takes as input, the current state and previous reward from the environment, then uses the internal policy to determine which action to take. Specifically, it does some pre-processing of the state, then samples a single action from the distribution over possible actions obtained from the policy.
* **reset**: This is called to reset the agent's variables before running it on a new episode
* **end_episode**: This method signals to the agent that the current episode has come to an end. The agent may do some learning, or clean-up at the end of an episode.

### The Random Agent

To get a feel for an agent and the methods it has, we implement an agent that just takes a *random* action at every step. For demonstration purposes, we also calculate the **episode return** at the end of an episode. The episode return is the sum of the (discounted) rewards obtained during the episode. If the returns for episode $i$, with trajectory $\tau_{i}$ are denoted $r_{i, t}$, and the **discount factor** is $\gamma$, then the episode return is calcuated as: $r(\tau_i) = \sum_{t=1}^{T_i} \gamma^t r_{i,t}$. The discount factor allows us to increase the importance of rewards received quickly and decrease the importance of rewards that take long to receive. It is especially important in environments that could have episodes that are infinitely long. In our particular environment where every episode is of the same length and the only non-zero reward is received at the end of the game, the discount factor doesn't make much difference and so we will ignore it (effectively set it to $1$) for the remainder of this practical.

```
class RandomAgent(object):

  def __init__(self, actions, state_size, seed):
    # When initializing, we let the agent know what actions are available in the
    # environment, how large the state is (not used in the RandomAgent) and the
    # current seed to use (also not used in the RandomAgent)
    self._actions = actions
    self._rewards = []
    self._taken_actions = []
    self._observed_states = []

  def policy(self, state):
    # The policy is an internal function that takes a state and returns a distribution over the possible actions.
    # The random agent just returns a uniform distribution over the actions.
    n = len(self._actions)  # The number of actions
    return tf.fill([n], 1./n)  # This returns a vector of length n, with each entry being 1/n

  def step(self, state_dict, reward):

    # Pre-process the state to extract the numerical values we're interested in from the state dictionary
    state = np.array([
        state_dict['fruit_x'] / game_width,  # Divide by width(or height) to normalise the value to lie between 0 and 1.
        state_dict['player_x'] / game_width,
        state_dict['fruit_y'] / (game_height+1),
        state_dict['player_vel'] / game_width
    ], dtype=np.float32)

    self._observed_states.append(state)  # Record that the state was observed during the episode
    self._rewards.append(reward)  # Keep track of the rewards we've received along the way
    action_distribution = self.policy(state)  # Use the policy to get the distribution over actions

    # Sample a single action according to the distribution over actions
    action = np.random.choice(self._actions, p=action_distribution.numpy())

    self._taken_actions.append(action)  # Record that the action was taken during the episode

    return action

  def reset(self):
    # This method is called when a new episode starts, we need to clear the
    # states, actions and rewards that we tracked during the last episode.
    self._rewards = []
    self._taken_actions = []
    self._observed_states = []

  def end_episode(self, final_reward):
    # We just calculate the episode return
    episode_return = sum(self._rewards) + final_reward
    return episode_return
```

## The Run-Loop
Now that we have an environment and a simple agent, we need a way of controlling the interaction between the agent and environment over multiple episodes. We do so in a **run-loop**. In this simple run-loop, the agent and environment run in lock-step. For each game frame, we get the state from the environment and pass it, along with the previous reward, to the agent. The agent selects an action that it wants to take given the game state. The action is taken in the environment and any reward received is recorded. We run the loop for multiple episodes, each time being careful to reset the game and agent (because they're starting a new game from scratch).

```
def run_loop(agent_class,                    # Which agent to use
             num_episodes=1,                 # How many episodes to run for
             record_every=1,                 # How many episodes to record
             seed=1234,                      # The random seed used
             rolling_return_frequency=100,   # The window size used to track the rolling episode return
             state_size=4):                  # The size of the state

  # Set the random seeds
  tf.set_random_seed(seed)
  np.random.seed(seed)

  # Initialise the environment
  environment.init()

  # Create an agent (this runs the agent's __init__ method)
  agent = agent_class(environment.getActionSet(), state_size, seed)

  progress_out = display.display(progress(0, num_episodes), display_id=True)  # Create a progress-bar

  # Create data structures to store metrics
  windowed_return = deque()
  rolling_returns = []
  frames = []

  for episode in range(num_episodes):
    environment.reset_game() # reset the environment
    agent.reset()  # reset the agent
    reward = 0

    while reward == 0 and not environment.game_over():   # Loop until the episode terminates
      state = game.getGameState()  # Get the current game state
      action = agent.step(state, reward)  # Pass the current game state and previous reward to the agent, get the action it wants to take
      reward = environment.act(action)  # Pass the action to the environment and get the reward.

      if episode % record_every == 0:
        # Store the frames for display later, every `record_every` episodes
        frames.append(environment.getScreenRGB())

    info = agent.end_episode(reward)  # Signal to the agent that the episode has come to and end

    # Store the episode return in the window (in this case, with no discounting, the episode return is the same as the environment's score)
    windowed_return.append(environment.score())
    if len(windowed_return) > rolling_return_frequency:
      windowed_return.popleft()

    rolling_return = sum(windowed_return) / len(windowed_return)
    rolling_returns.append(rolling_return)

    # Update the progress-bar
    message = 'Episode {}/{} ended with score {}, Rolling Return: {}, {}'.format(
        episode+1, num_episodes, environment.score(),
        rolling_return,
        info if info is not None else '')
    progress_out.update(progress(episode+1, num_episodes, message))

  message = 'Finished training, rendering video...'
  progress_out.update(progress(episode+1, num_episodes, message))

  # Render a video
  clip = make_animation(frames, fps=30, true_image=True).rotate(-90)
  display.display(clip.ipython_display(fps=30, center=False, autoplay=False, loop=False, height=320, width=240, max_duration=1000))

  message = 'Done...'
  progress_out.update(progress(episode+1, num_episodes, message))

  return rolling_returns
```

We now run our RandomAgent with the run-loop for 100 episodes to check that everything is working so far. (Note: the blue progress bar shows how many of the ```num_episodes``` episodes we've completed. The small black progress bar is for the video rendering, ignore that one!)

```
rolling_returns = run_loop(RandomAgent, num_episodes=100, record_every=5, rolling_return_frequency=5)
plot_rolling_returns(rolling_returns)
```

## A Policy Network
Remember, the policy is a distribution over the possible actions the agent can take in the environment given the current state of the environment,  denoted $\pi(a|s)$. In a Deep RL agent, the policy is represented by a neural network with parameters $\theta$, so we have $\pi_\theta(a|s) = NN(s; \theta)$, where $NN(s; \theta)$ is some potentially complex function represented by a neural network with parameters $\theta$. In other words, our neural network takes in the state as input and outputs the appropriate distribution over actions. Let us implement an agent who's policy is defined by a simple feed-forward neural network. We will name the class 'FixedAgent' because this agent will do no learning. As a result the policy network's weights will be fixed and the agent will take random actions as before.

The ```reset```, ```step``` and ```end_episode``` methods of our fixed agent will be identical to the RandomAgent we built earlier. We'll only change the ```__init__``` and ```policy``` methods. To avoid having to rewrite all that code, we will use Python's **inheritance** to reuse all the methods in RandomAgent except for policy which we *override* here.

```
# Lets build a fixed agent
class FixedAgent(RandomAgent):  # Inherit all the methods of RandomAgent

  def __init__(self, actions, state_size, seed):
    super(FixedAgent, self).__init__(actions, state_size, seed)

    # Define the policy network in the initialize method (constructor) because it should persist
    # through multiple usages over multiple episodes of the agent.
    # (We change the default weight initialiser to truncated random normal which
    # works better for the RL algorithm we'll use in this practical.)
    self._policy_network = tf.keras.Sequential([
        # Add a hidden layer with 64 neurons
        tf.keras.layers.Dense(64, input_shape=[state_size], activation=tf.nn.relu,
                              kernel_initializer=tf.truncated_normal_initializer(seed=seed)),
        # Add a hidden layer with 32 neurons
        tf.keras.layers.Dense(32, activation=tf.nn.relu,
                              kernel_initializer=tf.truncated_normal_initializer(seed=seed)),
        # Add an output layer with action-many neurons and a softmax activation function
        tf.keras.layers.Dense(len(actions), activation='softmax'),
    ])

  # Override the policy
  def policy(self, state):
    layer_input = tf.expand_dims(state, axis=0)  # Add a dummy batch dimension
    action_distribution = self._policy_network(layer_input)   # Get the distribution over actions from the policy network
    action_distribution = tf.squeeze(action_distribution, axis=0)  # Remove the dummy batch dimension

    return action_distribution
```

Let's test our FixedAgent, this is just to see that it runs, as we don't expect it to perform any better than the RandomAgent because it isn't learning anything yet!

```
# Our fixed-weight agent
rolling_returns = run_loop(FixedAgent, num_episodes=100, record_every=5, rolling_return_frequency=5)
plot_rolling_returns(rolling_returns)
```

## Learning with Policy Gradients
Finally, let's give our agent some intelligence by making it learn from its experience in interacting with the environment. In order to learn, we need a loss function or *objective*. In RL, the objective is to maximise the expected episode return (rewards) by taking actions in the environment. The actions our agent takes are determined by the policy $\pi_\theta(a|s)$, which are in turn determined by the neural network parameters $\theta$. So, we want to find the neural network parameters $\theta$ that maximise

$J(\theta) = \mathbb{E}_{\tau}[r(\tau)]$

**Note:** If the maths in the next section looks intimidating, feel free to skip over it, read the intuition and code and come back to it later!

### The derivative of the objective
We now turn to our usual tool of stochastic gradient descent to optimise the objective, but there are two complications. Firstly, the term $\pi_\theta(a|s)$ represented by our neural network doesn't appear in the equation (or does it?). Secondly, how do we deal with the expectation?

The first thing is to realise that our trajectories $\tau$ depend on the policy $\pi_\theta(a|s)$ (**Question:** Why?) So we can (informally) write:

\begin{align}
J(\theta) &= \mathbb{E}_{\tau \sim \pi_\theta(\tau)}  [r(\tau)] & \\
&= \int \pi_\theta (\tau) r(\tau)d\tau  & (\text{Definition of expectation}) \\
\end{align}

Then the gradient is:

\begin{align}
\nabla_\theta J(\theta) &= \nabla_\theta \int \pi_\theta (\tau) r(\tau)d\tau & \\
&= \int \pi_\theta (\tau) \nabla_\theta log \pi_\theta (\tau) r(\tau)d\tau & (\text{"Log derivative trick"}) \\
&= \mathbb{E}_{\tau \sim \pi_\theta(\tau)}[\nabla_\theta log \pi_\theta (\tau) r(\tau)]
\end{align}

Finally, since we don't know the true distribution of $\tau$, we can approximate the expectation using a *monte-carlo* approximation, where the sample trajectories come from $N$ episodes of interaction with the environment.

\begin{align}
\nabla_\theta J(\theta) &= \frac{1}{N} \sum_{i=1}^N \nabla_\theta log \pi_\theta (\tau_i) r(\tau_i)
\end{align}

Expanding this out (and considering that episode $i$ has $T_i$ steps) gives:

\begin{align}
\nabla_\theta J(\theta) &= \frac{1}{N} \sum_{i=1}^N (\sum_{t=1}^{T_i} \nabla_\theta log(\pi_\theta(a_{i,t} | s_{i, t})) \sum_{t=1}^{T_i} \gamma^t r_{i,t} )
\end{align}

We skipped a few steps in the maths here for brevity (see chapter 13 of [Sutton and Barto](https://drive.google.com/file/d/1xeUDVGWGUUv1-ccUMAZHJLej2C7aAFWY/view) for all the details if you're interested!). If the maths looks intimidating, don't worry! The important things to realise are the following:
* We define an objective $J(\theta)$ that is exactly what we want to do with RL, maximise the expected return.
* We can run our agent in the environment to generate *trajectories* for multiple episodes
* When an episode comes to an end and we know the return and trajectory, we can compute a term in (the Monte-carlo approximation to) the objective function.
* We can use Tensorflow to compute the gradient of an individual term in the monte-carlo approximation and apply it to the parameters of our neural network. To do this, we define the *loss* to minimise as follows (where the sums can be represented by loops and we set $\gamma = 1$ for simplicity):

\begin{align}
L(\theta) &= -\sum_{t=1}^{T_i} log(\pi_\theta(a_{i,t} | s_{i, t})) \sum_{t=1}^{T_i} r_{i,t}
\end{align}

The name "policy gradient" comes from the fact that we're directly taking the gradient of the policy, rather than the alternative, value-based RL, which uses iterative update rules to calculate the expected return assocated with a state. The particular flavour of policy gradient which uses the loss function above, along with the Monte-carlo approximation of the objective is known as the **REINFORCE** algorithm.

Finally, we implement the REINFORCE algorithm to optimize the parameters of the neural network. The only things we're changing, compared to our FixedAgent are the ```__init__``` and ```end_episode``` methods, so we use *inheritance* again to automatically "copy" all the methods from ```FixedAgent``` and ```RandomAgent```.

**Note:** ```ReinforceAgent``` indirectly inherits the methods from ```RandomAgent``` through ```FixedAgent``` (which directly inherits from ```RandomAgent```).  Both ```RandomAgent``` and ```FixedAgent``` have a ```policy``` method, but the one that gets "copied" to ```ReinforceAgent``` is the one from ```FixedAgent``` because it appeared later in the chain.

```
class ReinforceAgent(FixedAgent):

  # Override the initialization method because this agent also needs an optimizer
  # and a variable to track the step
  def __init__(self, actions, state_size, seed):
    super(ReinforceAgent, self).__init__(actions, state_size, seed)
    self._optimizer = tf.train.RMSPropOptimizer(learning_rate=0.001)
    self._step_counter = tf.train.get_or_create_global_step()

  def end_episode(self, final_reward):
    """At the end of an episode, we compute the loss for the episode and take a
    step in parameter speace in the direction of the gradients."""

    # Compute the return (cumulative discounted reward) for the episode
    episode_return = sum(self._rewards) + final_reward  # Assuming \gamma = 1

    with tf.GradientTape() as tape:
      # Loop over the states and actions making up the episode trajectory
      loss = 0
      for state, action in zip(self._observed_states, self._taken_actions):
        # Get the probabilities assigned to the actions given the state by the policy
        action_distribution = self.policy(state)
        action_index = self._actions.index(action)
        # Get the log probability of the chosen action under the policy
        log_action = tf.log(action_distribution[action_index])
        # Add to the running total for the episode
        loss -= log_action * episode_return          # Add your baseline value for TASK 4 here.

    # Compute the gradient of the loss with respect to the variables in the model
    grads = tape.gradient(loss, self._policy_network.variables)

    # Use the optimizer to apply the gradient
    self._optimizer.apply_gradients(
        zip(grads, self._policy_network.variables), global_step=self._step_counter)

    return 'Loss: {}'.format(loss)
```

Notice that during the episode we run only the forward-pass of the policy network (inference). At the end of the episode, we replay the states that occured during the episode and run both the forward and backward pass of the policy network (notice the gradient tape!) because we can only compute the loss once we have the episode return at the end of the episode. If the policy network is very complex, this could be inefficient. In that case you could run both the forward an backward pass during the episode and store intermediate gradients/partial derivatives to use in the update at the end of the episode.

And finally we train our **REINFORCE** agent and plot the resulting rolling episode returns (over a window of 100 episodes).

```
rolling_returns = run_loop(ReinforceAgent, num_episodes=2400, record_every=30)
plot_rolling_returns(rolling_returns)
```

## Your Tasks
### Task 1: Learning Objectives [ALL]
Review the learning objectives and ensure that you understand how the code in this practical relates to them. Ask your tutors if you don't understand anything!

### Task 2: Network Architecture [ALL]
Experiment with different network architectures and parameters and see how this affects the performance of the agent. What do you notice? Do you think the algorithm is sensitive to the network parameters?

**HINT**: Modify the code for the policy network in the ```__init__``` method of the ```FixedAgent``` class.

### Task 3: Seed Variance **[ALL]**
Reveal and run the code in the cells below. This code will run the entire training procedure of the REINFORCE agent 10 times (using only 1000 episodes per run to save some time). It will then plot a chart that shows the mean of the rolling returns over the multiple runs along with an estimated *confidence interval* for the mean. You should notice that the confidence interval is fairly wide given that only the random seed is changing. This illustrates a problem with the REINFORCE algorithm: it has **high variance**. (It is however an **unbiased estimator** of the policy gradient!)

### Task 4: Variance Reduction with a Basline **[INTERMEDIATE]**
Read about value functions and how to approximate them using *Monte-Carlo* methods in [Slides 5 to 7 Here](http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching_files/MC-TD.pdf). Then read slides [80, 84 and 85 here](http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture14.pdf). The goal of this task is to use a simple value function $V(s)$ to implement **REINFORCE with a Baseline**, where the loss function per episode changes to:

\begin{align}
L(\theta) &= -\sum_{t=1}^{T_i} log(\pi_\theta(a_{i,t} | s_{i, t})) \sum_{t=1}^{T_i} [r_{i,t} - V(s_{i, t})]
\end{align}

To do this, add code to the ```end_episode``` method of the ```ReinforceAgent``` class to estimate the value function. Subtract the value estimate for the state from the episode return at each step in the loop that computes the log_action_sum.

Re-run Task 3's code to plot the confidence interval around the mean episode returns to check what effect it has on the variance.

**HINT**: You will need to *discretise* the state-space. We've provided a very crude function called ```state_to_buckets``` that you can use to do this, or implement your own!

**Further Reading (Optional)**: See the section on how to introduce a baseline [here](https://danieltakeshi.github.io/2017/03/28/going-deeper-into-reinforcement-learning-fundamentals-of-policy-gradients/) for more details about how and why this works!

**Further Reading (Optional)**: This [blog post](https://flyyufelix.github.io/2017/10/12/dqn-vs-pg.html) contrasts policy gradient methods with an alternative value-based approach to RL called Deep Q-Networks. It also discusses some approaches to reducing the variance of the policy gradient estimator.

### Task 5: Learning from pixels **[OPTIONAL]**
The agent we implemented in this practical uses a simple numerical representation of the state of the environment. In many cases such a representation would not be available. Change the run-loop to instead pass the array of pixel values to the agent. (which you can get by calling ```environment.getScreenRGB()```). Change the agent's policy network to cater for this image-representation of the state.

### Task 6: Other games **[OPTIONAL]**
The PyGame Learning Environment (PLE) has [a number of games built-in](https://pygame-learning-environment.readthedocs.io/en/latest/user/games.html). Change the code in this practical to run on a different game and learn either from pixels or from the state representation provided by PLE. One interesting game you could try is FlappyBird! Remember to remove the reward overrides we set when trying a new game!

## Additional Code for Task 3
This might take some time to run, continue reading for Task 4 and ask any questions you have while waiting!

```
all_rolling_returns = []

for i in range(10):
  rolling_returns = run_loop(ReinforceAgent, num_episodes=1000, record_every=25, seed=np.random.randint(100000))
  all_rolling_returns.append(rolling_returns)

plot_rolling_returns(np.array(all_rolling_returns))
```


```python
# Importing standard Qiskit libraries
from qiskit import QuantumCircuit, transpile, Aer, IBMQ
from qiskit.tools.jupyter import *
from qiskit.visualization import *
from ibm_quantum_widgets import *

# Loading your IBM Quantum account(s)
provider = IBMQ.load_account()
```

    ibmqfactory.load_account:WARNING:2021-07-23 22:20:04,310: Credentials are already in use. The existing account in the session will be replaced.

```python
!pip install pennylane
import pennylane as qml
```

    Collecting pennylane
      Using cached PennyLane-0.16.0-py3-none-any.whl (514 kB)
    Collecting autoray
      Using cached autoray-0.2.5-py3-none-any.whl (16 kB)
    Collecting autograd
      Using cached autograd-1.3-py3-none-any.whl
    Requirement already satisfied: toml in /opt/conda/lib/python3.8/site-packages (from pennylane) (0.10.2)
    Requirement already satisfied: numpy in /opt/conda/lib/python3.8/site-packages (from pennylane) (1.21.0)
    Requirement already satisfied: scipy in /opt/conda/lib/python3.8/site-packages (from pennylane) (1.7.0)
    Requirement already satisfied: appdirs in /opt/conda/lib/python3.8/site-packages (from pennylane) (1.4.4)
    Collecting semantic-version==2.6
      Using cached semantic_version-2.6.0-py3-none-any.whl (14 kB)
    Requirement already satisfied: networkx in /opt/conda/lib/python3.8/site-packages (from pennylane) (2.6.1)
    Collecting future>=0.15.2
      Using cached future-0.18.2-py3-none-any.whl
    Requirement already satisfied: matplotlib>=3.3 in /opt/conda/lib/python3.8/site-packages (from networkx->pennylane) (3.4.2)
    Requirement already satisfied: pandas>=1.1 in /opt/conda/lib/python3.8/site-packages (from networkx->pennylane) (1.3.0)
    Requirement already satisfied: kiwisolver>=1.0.1 in /opt/conda/lib/python3.8/site-packages (from matplotlib>=3.3->networkx->pennylane) (1.3.1)
    Requirement already satisfied: pyparsing>=2.2.1 in /opt/conda/lib/python3.8/site-packages (from matplotlib>=3.3->networkx->pennylane) (2.4.7)
    Requirement already satisfied: python-dateutil>=2.7 in /opt/conda/lib/python3.8/site-packages (from matplotlib>=3.3->networkx->pennylane) (2.8.1)
    Requirement already satisfied: pillow>=6.2.0 in /opt/conda/lib/python3.8/site-packages (from matplotlib>=3.3->networkx->pennylane) (8.3.1)
    Requirement already satisfied: cycler>=0.10 in /opt/conda/lib/python3.8/site-packages (from matplotlib>=3.3->networkx->pennylane) (0.10.0)
    Requirement already satisfied: six in /opt/conda/lib/python3.8/site-packages (from cycler>=0.10->matplotlib>=3.3->networkx->pennylane) (1.16.0)
    Requirement already satisfied: pytz>=2017.3 in /opt/conda/lib/python3.8/site-packages (from pandas>=1.1->networkx->pennylane) (2021.1)
    Installing collected packages: future, semantic-version, autoray, autograd, pennylane
    Successfully installed autograd-1.3 autoray-0.2.5 future-0.18.2 pennylane-0.16.0 semantic-version-2.6.0

```python
!pip install pennylane-qiskit
```

    Collecting pennylane-qiskit
      Using cached PennyLane_qiskit-0.16.0-py3-none-any.whl (21 kB)
    Requirement already satisfied: numpy in /opt/conda/lib/python3.8/site-packages (from pennylane-qiskit) (1.21.0)
    Requirement already satisfied: pennylane>=0.16 in /opt/conda/lib/python3.8/site-packages (from pennylane-qiskit) (0.16.0)
    Requirement already satisfied: qiskit>=0.25 in /opt/conda/lib/python3.8/site-packages (from pennylane-qiskit) (0.27.0)
    Requirement already satisfied: networkx>=2.2 in /opt/conda/lib/python3.8/site-packages (from pennylane-qiskit) (2.6.1)
    Requirement already satisfied: scipy!=1.6.1,>=1.5 in /opt/conda/lib/python3.8/site-packages (from networkx>=2.2->pennylane-qiskit) (1.7.0)
    Requirement already satisfied: pandas>=1.1 in /opt/conda/lib/python3.8/site-packages (from networkx>=2.2->pennylane-qiskit) (1.3.0)
    Requirement already satisfied: matplotlib>=3.3 in /opt/conda/lib/python3.8/site-packages (from networkx>=2.2->pennylane-qiskit) (3.4.2)
    Requirement already satisfied: cycler>=0.10 in /opt/conda/lib/python3.8/site-packages (from matplotlib>=3.3->networkx>=2.2->pennylane-qiskit) (0.10.0)
    Requirement already satisfied: kiwisolver>=1.0.1 in /opt/conda/lib/python3.8/site-packages (from matplotlib>=3.3->networkx>=2.2->pennylane-qiskit) (1.3.1)
    Requirement already satisfied: pillow>=6.2.0 in /opt/conda/lib/python3.8/site-packages (from matplotlib>=3.3->networkx>=2.2->pennylane-qiskit) (8.3.1)
    Requirement already satisfied: pyparsing>=2.2.1 in /opt/conda/lib/python3.8/site-packages (from matplotlib>=3.3->networkx>=2.2->pennylane-qiskit) (2.4.7)
    Requirement already satisfied: python-dateutil>=2.7 in /opt/conda/lib/python3.8/site-packages (from matplotlib>=3.3->networkx>=2.2->pennylane-qiskit) (2.8.1)
    Requirement already satisfied: six in /opt/conda/lib/python3.8/site-packages (from cycler>=0.10->matplotlib>=3.3->networkx>=2.2->pennylane-qiskit) (1.16.0)
    Requirement already satisfied: pytz>=2017.3 in /opt/conda/lib/python3.8/site-packages (from pandas>=1.1->networkx>=2.2->pennylane-qiskit) (2021.1)
    Requirement already satisfied: autoray in /opt/conda/lib/python3.8/site-packages (from pennylane>=0.16->pennylane-qiskit) (0.2.5)
    Requirement already satisfied: autograd in /opt/conda/lib/python3.8/site-packages (from pennylane>=0.16->pennylane-qiskit) (1.3)
    Requirement already satisfied: appdirs in /opt/conda/lib/python3.8/site-packages (from pennylane>=0.16->pennylane-qiskit) (1.4.4)
    Requirement already satisfied: semantic-version==2.6 in /opt/conda/lib/python3.8/site-packages (from pennylane>=0.16->pennylane-qiskit) (2.6.0)
    Requirement already satisfied: toml in /opt/conda/lib/python3.8/site-packages (from pennylane>=0.16->pennylane-qiskit) (0.10.2)
    Requirement already satisfied: qiskit-ibmq-provider==0.14.0 in /opt/conda/lib/python3.8/site-packages (from qiskit>=0.25->pennylane-qiskit) (0.14.0)
    Requirement already satisfied: qiskit-terra==0.17.4 in /opt/conda/lib/python3.8/site-packages (from qiskit>=0.25->pennylane-qiskit) (0.17.4)
    Requirement already satisfied: qiskit-aqua==0.9.2 in /opt/conda/lib/python3.8/site-packages (from qiskit>=0.25->pennylane-qiskit) (0.9.2)
    Requirement already satisfied: qiskit-ignis==0.6.0 in /opt/conda/lib/python3.8/site-packages (from qiskit>=0.25->pennylane-qiskit) (0.6.0)
    Requirement already satisfied: qiskit-aer==0.8.2 in /opt/conda/lib/python3.8/site-packages (from qiskit>=0.25->pennylane-qiskit) (0.8.2)
    Requirement already satisfied: pybind11>=2.6 in /opt/conda/lib/python3.8/site-packages (from qiskit-aer==0.8.2->qiskit>=0.25->pennylane-qiskit) (2.6.2)
    Requirement already satisfied: sympy>=1.3 in /opt/conda/lib/python3.8/site-packages (from qiskit-aqua==0.9.2->qiskit>=0.25->pennylane-qiskit) (1.8)
    Requirement already satisfied: setuptools>=40.1.0 in /opt/conda/lib/python3.8/site-packages (from qiskit-aqua==0.9.2->qiskit>=0.25->pennylane-qiskit) (57.1.0)
    Requirement already satisfied: yfinance in /opt/conda/lib/python3.8/site-packages (from qiskit-aqua==0.9.2->qiskit>=0.25->pennylane-qiskit) (0.1.61)
    Requirement already satisfied: psutil>=5 in /opt/conda/lib/python3.8/site-packages (from qiskit-aqua==0.9.2->qiskit>=0.25->pennylane-qiskit) (5.8.0)
    Requirement already satisfied: dlx<=1.0.4 in /opt/conda/lib/python3.8/site-packages (from qiskit-aqua==0.9.2->qiskit>=0.25->pennylane-qiskit) (1.0.4)
    Requirement already satisfied: h5py in /opt/conda/lib/python3.8/site-packages (from qiskit-aqua==0.9.2->qiskit>=0.25->pennylane-qiskit) (3.3.0)
    Requirement already satisfied: retworkx>=0.8.0 in /opt/conda/lib/python3.8/site-packages (from qiskit-aqua==0.9.2->qiskit>=0.25->pennylane-qiskit) (0.9.0)
    Requirement already satisfied: scikit-learn>=0.20.0 in /opt/conda/lib/python3.8/site-packages (from qiskit-aqua==0.9.2->qiskit>=0.25->pennylane-qiskit) (0.24.2)
    Requirement already satisfied: docplex<=2.20.204 in /opt/conda/lib/python3.8/site-packages (from qiskit-aqua==0.9.2->qiskit>=0.25->pennylane-qiskit) (2.20.204)
    Requirement already satisfied: quandl in /opt/conda/lib/python3.8/site-packages (from qiskit-aqua==0.9.2->qiskit>=0.25->pennylane-qiskit) (3.6.1)
    Requirement already satisfied: fastdtw<=0.3.4 in /opt/conda/lib/python3.8/site-packages (from qiskit-aqua==0.9.2->qiskit>=0.25->pennylane-qiskit) (0.3.4)
    Requirement already satisfied: urllib3>=1.21.1 in /opt/conda/lib/python3.8/site-packages (from qiskit-ibmq-provider==0.14.0->qiskit>=0.25->pennylane-qiskit) (1.26.6)
    Requirement already satisfied: requests-ntlm>=1.1.0 in /opt/conda/lib/python3.8/site-packages (from qiskit-ibmq-provider==0.14.0->qiskit>=0.25->pennylane-qiskit) (1.1.0)
    Requirement already satisfied: dill>=0.3 in /opt/conda/lib/python3.8/site-packages (from qiskit-ibmq-provider==0.14.0->qiskit>=0.25->pennylane-qiskit) (0.3.4)
    Requirement already satisfied: websocket-client>=1.0.1 in /opt/conda/lib/python3.8/site-packages (from qiskit-ibmq-provider==0.14.0->qiskit>=0.25->pennylane-qiskit) (1.1.0)
    Requirement already satisfied: requests>=2.19 in /opt/conda/lib/python3.8/site-packages (from qiskit-ibmq-provider==0.14.0->qiskit>=0.25->pennylane-qiskit) (2.25.1)
    Requirement already satisfied: python-constraint>=1.4 in /opt/conda/lib/python3.8/site-packages (from qiskit-terra==0.17.4->qiskit>=0.25->pennylane-qiskit) (1.4.0)
    Requirement already satisfied: ply>=3.10 in /opt/conda/lib/python3.8/site-packages (from qiskit-terra==0.17.4->qiskit>=0.25->pennylane-qiskit) (3.11)
    Requirement already satisfied: jsonschema>=2.6 in /opt/conda/lib/python3.8/site-packages (from qiskit-terra==0.17.4->qiskit>=0.25->pennylane-qiskit) (3.2.0)
    Requirement already satisfied: fastjsonschema>=2.10 in /opt/conda/lib/python3.8/site-packages (from qiskit-terra==0.17.4->qiskit>=0.25->pennylane-qiskit) (2.15.1)
    Requirement already satisfied: pyrsistent>=0.14.0 in /opt/conda/lib/python3.8/site-packages (from jsonschema>=2.6->qiskit-terra==0.17.4->qiskit>=0.25->pennylane-qiskit) (0.18.0)
    Requirement already satisfied: attrs>=17.4.0 in /opt/conda/lib/python3.8/site-packages (from jsonschema>=2.6->qiskit-terra==0.17.4->qiskit>=0.25->pennylane-qiskit) (21.2.0)
    Requirement already satisfied: idna<3,>=2.5 in /opt/conda/lib/python3.8/site-packages (from requests>=2.19->qiskit-ibmq-provider==0.14.0->qiskit>=0.25->pennylane-qiskit) (2.10)
    Requirement already satisfied: chardet<5,>=3.0.2 in /opt/conda/lib/python3.8/site-packages (from requests>=2.19->qiskit-ibmq-provider==0.14.0->qiskit>=0.25->pennylane-qiskit) (4.0.0)
    Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.8/site-packages (from requests>=2.19->qiskit-ibmq-provider==0.14.0->qiskit>=0.25->pennylane-qiskit) (2021.5.30)
    Requirement already satisfied: ntlm-auth>=1.0.2 in /opt/conda/lib/python3.8/site-packages (from requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.14.0->qiskit>=0.25->pennylane-qiskit) (1.5.0)
    Requirement already satisfied: cryptography>=1.3 in /opt/conda/lib/python3.8/site-packages (from requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.14.0->qiskit>=0.25->pennylane-qiskit) (3.4.7)
    Requirement already satisfied: cffi>=1.12 in /opt/conda/lib/python3.8/site-packages (from cryptography>=1.3->requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.14.0->qiskit>=0.25->pennylane-qiskit) (1.14.5)
    Requirement already satisfied: pycparser in /opt/conda/lib/python3.8/site-packages (from cffi>=1.12->cryptography>=1.3->requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.14.0->qiskit>=0.25->pennylane-qiskit) (2.20)
    Requirement already satisfied: joblib>=0.11 in /opt/conda/lib/python3.8/site-packages (from scikit-learn>=0.20.0->qiskit-aqua==0.9.2->qiskit>=0.25->pennylane-qiskit) (1.0.1)
    Requirement already satisfied: threadpoolctl>=2.0.0 in /opt/conda/lib/python3.8/site-packages (from scikit-learn>=0.20.0->qiskit-aqua==0.9.2->qiskit>=0.25->pennylane-qiskit) (2.1.0)
    Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.8/site-packages (from sympy>=1.3->qiskit-aqua==0.9.2->qiskit>=0.25->pennylane-qiskit) (1.2.1)
    Requirement already satisfied: future>=0.15.2 in /opt/conda/lib/python3.8/site-packages (from autograd->pennylane>=0.16->pennylane-qiskit) (0.18.2)
    Requirement already satisfied: inflection>=0.3.1 in /opt/conda/lib/python3.8/site-packages (from quandl->qiskit-aqua==0.9.2->qiskit>=0.25->pennylane-qiskit) (0.5.1)
    Requirement already satisfied: more-itertools in /opt/conda/lib/python3.8/site-packages (from quandl->qiskit-aqua==0.9.2->qiskit>=0.25->pennylane-qiskit) (8.8.0)
    Requirement already satisfied: lxml>=4.5.1 in /opt/conda/lib/python3.8/site-packages (from yfinance->qiskit-aqua==0.9.2->qiskit>=0.25->pennylane-qiskit) (4.6.3)
    Requirement already satisfied: multitasking>=0.0.7 in /opt/conda/lib/python3.8/site-packages (from yfinance->qiskit-aqua==0.9.2->qiskit>=0.25->pennylane-qiskit) (0.0.9)
    Installing collected packages: pennylane-qiskit
    Successfully installed pennylane-qiskit-0.16.0

```python
from pennylane.optimize.gradient_descent import *
from pennylane import numpy as np
```

```python
Optimizer = qml.QNGOptimizer(0.01, False, 0)
```

```python
num_qubits = 4 #4 qubits
# device
#dev = qml.device('qiskit.ibmq', wires=num_qubits, backend='ibmq_qasm_simulator', provider=provider)
dev = qml.device("default.qubit", wires=4, shots=1024)
params = np.array([np.pi/2, np.pi/2, np.pi/2])

#user input for rotation gate angles
"""
params_1 = list(map(float, input().strip().split(' ')))
params_2 = list(map(float, input().strip().split(' ')))
params_3 = list(map(float, input().strip().split(' ')))
params_4 = list(map(float, input().strip().split(' ')))
params_5 = list(map(float, input().strip().split(' ')))
params_6 = list(map(float, input().strip().split(' ')))
params_list = list(map(float, input().strip().split(' ')))
"""

"""params_1 = default_params #[phi, theta, omega]
params_2 = default_params #[phi, theta, omega]
params_3 = default_params #[phi, theta, omega]
params_4 = default_params #[phi, theta, omega]
params_5 = default_params #[phi, theta, omega]
params_6 = default_params #[phi, theta, omega]
params_list = default_params #[phi, theta, omega]"""

@qml.qnode(dev)
def vqc(params, wires=4):
    #Cluster State
    qml.Hadamard(wires=0)
    qml.Hadamard(wires=1)
    qml.Hadamard(wires=2)
    qml.Hadamard(wires=3)
    qml.CNOT(wires=[0,1])
    qml.CNOT(wires=[1,2])
    qml.CNOT(wires=[2,3])
    qml.RY(np.pi, wires=0)
    qml.RY(np.pi, wires=1)
    qml.RY(np.pi, wires=2)
    qml.RY(np.pi, wires=3)
    qml.CNOT(wires=[0,1])
    qml.CNOT(wires=[1,2])
    qml.CNOT(wires=[2,3])
    #QConvLayer1
    qml.U3(params[0], params[1], params[2], wires=0)
    qml.U3(params[0], params[1], params[2], wires=1)
    qml.U3(params[0], params[1], params[2], wires=2)
    qml.U3(params[0], params[1], params[2], wires=3)
    qml.CRot(params[0], params[1], params[2], wires=[0,1])
    qml.CRot(params[0], params[1], params[2], wires=[2,3])
    #QConvLayer2
    qml.CRot(params_list[0], params_list[1], params_list[2], wires=[0,2])
    return qml.expval(qml.PauliZ(1)), qml.expval(qml.PauliZ(2)), qml.expval(qml.PauliZ(3))

```

```python
help (qml.Hamiltonian)
```

    Help on class Hamiltonian in module pennylane.vqe.vqe:

    class Hamiltonian(builtins.object)
     |  Hamiltonian(coeffs, observables, simplify=False)
     |
     |  Lightweight class for representing Hamiltonians for Variational Quantum
     |  Eigensolver problems.
     |
     |  Hamiltonians can be expressed as linear combinations of observables, e.g.,
     |  :math:`\sum_{k=0}^{N-1} c_k O_k`.
     |
     |  This class keeps track of the terms (coefficients and observables) separately.
     |
     |  Args:
     |      coeffs (Iterable[float]): coefficients of the Hamiltonian expression
     |      observables (Iterable[Observable]): observables in the Hamiltonian expression
     |      simplify (bool): Specifies whether the Hamiltonian is simplified upon initialization
     |                       (like-terms are combined). The default value is `False`.
     |
     |  .. seealso:: :class:`~.ExpvalCost`, :func:`~.molecular_hamiltonian`
     |
     |  **Example:**
     |
     |  A Hamiltonian can be created by simply passing the list of coefficients
     |  as well as the list of observables:
     |
     |  >>> coeffs = [0.2, -0.543]
     |  >>> obs = [qml.PauliX(0) @ qml.PauliZ(1), qml.PauliZ(0) @ qml.Hadamard(2)]
     |  >>> H = qml.Hamiltonian(coeffs, obs)
     |  >>> print(H)
     |    (-0.543) [Z0 H2]
     |  + (0.2) [X0 Z1]
     |
     |  The user can also provide custom observables:
     |
     |  >>> obs_matrix = np.array([[0.5, 1.0j, 0.0, -3j],
     |                             [-1.0j, -1.1, 0.0, -0.1],
     |                             [0.0, 0.0, -0.9, 12.0],
     |                             [3j, -0.1, 12.0, 0.0]])
     |  >>> obs = qml.Hermitian(obs_matrix, wires=[0, 1])
     |  >>> H = qml.Hamiltonian((0.8, ), (obs, ))
     |  >>> print(H)
     |  (0.8) [Hermitian0'1]
     |
     |  Alternatively, the :func:`~.molecular_hamiltonian` function from the
     |  :doc:`/introduction/chemistry` module can be used to generate a molecular
     |  Hamiltonian.
     |
     |  .. Warning::
     |
     |      Hamiltonians can be constructed using Pythonic arithmetic operations. For example:
     |
     |      >>> qml.PauliX(0) + 2 * qml.PauliZ(0) @ qml.PauliZ(1)
     |
     |      is equivalent to the following Hamiltonian:
     |
     |      >>> qml.Hamiltonian([1, 2], [qml.PauliX(0), qml.PauliZ(0) @ qml.PauliZ(1)])
     |
     |      When Hamiltonians are defined using arithmetic operations **inside of QNodes**, constituent observables
     |      may be queued as operations/an error may be thrown. Thus, Hamiltonians must be defined either outside of QNodes,
     |      or inside of QNodes using the conventional method.
     |
     |      Note that this issue also arises when calling the ``simplify()`` method.
     |
     |  Methods defined here:
     |
     |  __add__(self, H)
     |      The addition operation between a Hamiltonian and a Hamiltonian/Tensor/Observable.
     |
     |  __iadd__(self, H)
     |      The inplace addition operation between a Hamiltonian and a Hamiltonian/Tensor/Observable.
     |
     |  __imul__(self, a)
     |      The inplace scalar multiplication operation between a scalar and a Hamiltonian.
     |
     |  __init__(self, coeffs, observables, simplify=False)
     |      Initialize self.  See help(type(self)) for accurate signature.
     |
     |  __isub__(self, H)
     |      The inplace subtraction operation between a Hamiltonian and a Hamiltonian/Tensor/Observable.
     |
     |  __matmul__(self, H)
     |      The tensor product operation between a Hamiltonian and a Hamiltonian/Tensor/Observable.
     |
     |  __mul__(self, a)
     |      The scalar multiplication operation between a scalar and a Hamiltonian.
     |
     |  __repr__(self)
     |      Return repr(self).
     |
     |  __rmul__ = __mul__(self, a)
     |
     |  __str__(self)
     |      Return str(self).
     |
     |  __sub__(self, H)
     |      The subtraction operation between a Hamiltonian and a Hamiltonian/Tensor/Observable.
     |
     |  compare(self, H)
     |      Compares with another :class:`~Hamiltonian`, :class:`~.Observable`, or :class:`~.Tensor`,
     |      to determine if they are equivalent.
     |
     |      Hamiltonians/observables are equivalent if they represent the same operator
     |      (their matrix representations are equal), and they are defined on the same wires.
     |
     |      .. Warning::
     |
     |          The compare method does **not** check if the matrix representation
     |          of a :class:`~.Hermitian` observable is equal to an equivalent
     |          observable expressed in terms of Pauli matrices, or as a
     |          linear combination of Hermitians.
     |          To do so would require the matrix form of Hamiltonians and Tensors
     |          be calculated, which would drastically increase runtime.
     |
     |      Returns:
     |          (bool): True if equivalent.
     |
     |      **Examples**
     |
     |      >>> A = np.array([[1, 0], [0, -1]])
     |      >>> H = qml.Hamiltonian(
     |      ...     [0.5, 0.5],
     |      ...     [qml.Hermitian(A, 0) @ qml.PauliY(1), qml.PauliY(1) @ qml.Hermitian(A, 0) @ qml.Identity("a")]
     |      ... )
     |      >>> obs = qml.Hermitian(A, 0) @ qml.PauliY(1)
     |      >>> print(H.compare(obs))
     |      True
     |
     |      >>> H1 = qml.Hamiltonian([1, 1], [qml.PauliX(0), qml.PauliZ(1)])
     |      >>> H2 = qml.Hamiltonian([1, 1], [qml.PauliZ(0), qml.PauliX(1)])
     |      >>> H1.compare(H2)
     |      False
     |
     |      >>> ob1 = qml.Hamiltonian([1], [qml.PauliX(0)])
     |      >>> ob2 = qml.Hermitian(np.array([[0, 1], [1, 0]]), 0)
     |      >>> ob1.compare(ob2)
     |      False
     |
     |  queue(self)
     |      Queues a qml.Hamiltonian instance
     |
     |  simplify(self)
     |      Simplifies the Hamiltonian by combining like-terms.
     |
     |      **Example**
     |
     |      >>> ops = [qml.PauliY(2), qml.PauliX(0) @ qml.Identity(1), qml.PauliX(0)]
     |      >>> H = qml.Hamiltonian([1, 1, -2], ops)
     |      >>> H.simplify()
     |      >>> print(H)
     |        (-1) [X0]
     |      + (1) [Y2]
     |
     |  ----------------------------------------------------------------------
     |  Readonly properties defined here:
     |
     |  coeffs
     |      Return the coefficients defining the Hamiltonian.
     |
     |      Returns:
     |          Iterable[float]): coefficients in the Hamiltonian expression
     |
     |  name
     |
     |  ops
     |      Return the operators defining the Hamiltonian.
     |
     |      Returns:
     |          Iterable[Observable]): observables in the Hamiltonian expression
     |
     |  terms
     |      The terms of the Hamiltonian expression :math:`\sum_{k=0}^{N-1} c_k O_k`
     |
     |      Returns:
     |          (tuple, tuple): tuples of coefficients and operations, each of length N
     |
     |  wires
     |      The sorted union of wires from all operators.
     |
     |      Returns:
     |          (Wires): Combined wires present in all terms, sorted.
     |
     |  ----------------------------------------------------------------------
     |  Data descriptors defined here:
     |
     |  __dict__
     |      dictionary for instance variables (if defined)
     |
     |  __weakref__
     |      list of weak references to the object (if defined)

```python
coeffs = [1,1,1]
obs = [qml.PauliZ(0), qml.PauliZ(0), qml.PauliZ(0)]
H = qml.Hamiltonian(coeffs, obs)
```

```python
cost_fn = qml.ExpvalCost(vqc, H, dev)
```

```python
%tb
theta_new = Optimizer.step(cost_fn, params)
```


##### Copyright 2020 The OpenFermion Developers

```python
#@title Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
```

# Introduction to OpenFermion

    View on QuantumLib

    Run in Google Colab

    View source on GitHub

    Download notebook

Note: The examples below must be run sequentially within a section.

## Setup

Install the OpenFermion package:

```python
try:
    import openfermion
except ImportError:
    !pip install git+https://github.com/quantumlib/OpenFermion.git@master#egg=openfermion
```

## Initializing the FermionOperator data structure

Fermionic systems are often treated in second quantization where arbitrary operators can be expressed using the fermionic creation and annihilation operators, $a^\dagger_k$ and $a_k$.  The fermionic ladder operators play a similar role to their qubit ladder operator counterparts, $\sigma^+_k$ and $\sigma^-_k$ but are distinguished by the canonical fermionic anticommutation relations, $\{a^\dagger_i, a^\dagger_j\} = \{a_i, a_j\} = 0$ and $\{a_i, a_j^\dagger\} = \delta_{ij}$. Any weighted sums of products of these operators are represented with the FermionOperator data structure in OpenFermion. The following are examples of valid FermionOperators:

$$
\begin{align}
& a_1 \nonumber \\
& 1.7 a^\dagger_3 \nonumber \\
&-1.7 \, a^\dagger_3 a_1 \nonumber \\
&(1 + 2i) \, a^\dagger_4 a^\dagger_3 a_9 a_1 \nonumber \\
&(1 + 2i) \, a^\dagger_4 a^\dagger_3 a_9 a_1 - 1.7 \, a^\dagger_3 a_1 \nonumber
\end{align}
$$

The FermionOperator class is contained in $\textrm{ops/_fermion_operator.py}$. In order to support fast addition of FermionOperator instances, the class is implemented as hash table (python dictionary). The keys of the dictionary encode the strings of ladder operators and values of the dictionary store the coefficients. The strings of ladder operators are encoded as a tuple of 2-tuples which we refer to as the "terms tuple". Each ladder operator is represented by a 2-tuple. The first element of the 2-tuple is an int indicating the tensor factor on which the ladder operator acts. The second element of the 2-tuple is Boole: 1 represents raising and 0 represents lowering. For instance, $a^\dagger_8$ is represented in a 2-tuple as $(8, 1)$. Note that indices start at 0 and the identity operator is an empty list. Below we give some examples of operators and their terms tuple:

$$
\begin{align}
I & \mapsto () \nonumber \\
a_1 & \mapsto ((1, 0),) \nonumber \\
a^\dagger_3 & \mapsto ((3, 1),) \nonumber \\
a^\dagger_3 a_1 & \mapsto ((3, 1), (1, 0)) \nonumber \\
a^\dagger_4 a^\dagger_3 a_9 a_1 & \mapsto ((4, 1), (3, 1), (9, 0), (1, 0)) \nonumber
\end{align}
$$

Note that when initializing a single ladder operator one should be careful to add the comma after the inner pair. This is because in python ((1, 2)) = (1, 2) whereas ((1, 2),) = ((1, 2),). The "terms tuple" is usually convenient when one wishes to initialize a term as part of a coded routine. However, the terms tuple is not particularly intuitive. Accordingly, OpenFermion also supports another user-friendly, string notation below. This representation is rendered when calling "print" on a FermionOperator.

$$
\begin{align}
I & \mapsto \textrm{""} \nonumber \\
a_1 & \mapsto \textrm{"1"} \nonumber \\
a^\dagger_3 & \mapsto \textrm{"3^"} \nonumber \\
a^\dagger_3 a_1 & \mapsto \textrm{"3^}\;\textrm{1"} \nonumber \\
a^\dagger_4 a^\dagger_3 a_9 a_1 & \mapsto \textrm{"4^}\;\textrm{3^}\;\textrm{9}\;\textrm{1"} \nonumber
\end{align}
$$

Let's initialize our first term! We do it two different ways below.

```python
from openfermion.ops import FermionOperator

my_term = FermionOperator(((3, 1), (1, 0)))
print(my_term)

my_term = FermionOperator('3^ 1')
print(my_term)
```

The preferred way to specify the coefficient in openfermion is to provide an optional coefficient argument. If not provided, the coefficient defaults to 1. In the code below, the first method is preferred. The multiplication in the second method actually creates a copy of the term, which introduces some additional cost. All inplace operands (such as +=) modify classes whereas binary operands such as + create copies. Important caveats are that the empty tuple FermionOperator(()) and the empty string FermionOperator('') initializes identity. The empty initializer FermionOperator() initializes the zero operator.

```python
good_way_to_initialize = FermionOperator('3^ 1', -1.7)
print(good_way_to_initialize)

bad_way_to_initialize = -1.7 * FermionOperator('3^ 1')
print(bad_way_to_initialize)

identity = FermionOperator('')
print(identity)

zero_operator = FermionOperator()
print(zero_operator)
```

Note that FermionOperator has only one attribute: .terms. This attribute is the dictionary which stores the term tuples.

```python
my_operator = FermionOperator('4^ 1^ 3 9', 1. + 2.j)
print(my_operator)
print(my_operator.terms)
```

## Manipulating the FermionOperator data structure
So far we have explained how to initialize a single FermionOperator such as $-1.7 \, a^\dagger_3 a_1$. However, in general we will want to represent sums of these operators such as $(1 + 2i) \, a^\dagger_4 a^\dagger_3 a_9 a_1 - 1.7 \, a^\dagger_3 a_1$. To do this, just add together two FermionOperators! We demonstrate below.

```python
from openfermion.ops import FermionOperator

term_1 = FermionOperator('4^ 3^ 9 1', 1. + 2.j)
term_2 = FermionOperator('3^ 1', -1.7)
my_operator = term_1 + term_2
print(my_operator)

my_operator = FermionOperator('4^ 3^ 9 1', 1. + 2.j)
term_2 = FermionOperator('3^ 1', -1.7)
my_operator += term_2
print('')
print(my_operator)
```

The print function prints each term in the operator on a different line. Note that the line my_operator = term_1 + term_2 creates a new object, which involves a copy of term_1 and term_2. The second block of code uses the inplace method +=, which is more efficient. This is especially important when trying to construct a very large FermionOperator. FermionOperators also support a wide range of builtins including, str(), repr(), ==, !=, *=, *, /, /=, +, +=, -, -=, - and **. Note that since FermionOperators involve floats, == and != check for (in)equality up to numerical precision. We demonstrate some of these methods below.

```python
term_1 = FermionOperator('4^ 3^ 9 1', 1. + 2.j)
term_2 = FermionOperator('3^ 1', -1.7)

my_operator = term_1 - 33. * term_2
print(my_operator)

my_operator *= 3.17 * (term_2 + term_1) ** 2
print('')
print(my_operator)

print('')
print(term_2 ** 3)

print('')
print(term_1 == 2.*term_1 - term_1)
print(term_1 == my_operator)
```

Additionally, there are a variety of methods that act on the FermionOperator data structure. We demonstrate a small subset of those methods here.

```python
from openfermion.utils import commutator, count_qubits, hermitian_conjugated
from openfermion.transforms import normal_ordered

# Get the Hermitian conjugate of a FermionOperator, count its qubit, check if it is normal-ordered.
term_1 = FermionOperator('4^ 3 3^', 1. + 2.j)
print(hermitian_conjugated(term_1))
print(term_1.is_normal_ordered())
print(count_qubits(term_1))

# Normal order the term.
term_2 = normal_ordered(term_1)
print('')
print(term_2)
print(term_2.is_normal_ordered())

# Compute a commutator of the terms.
print('')
print(commutator(term_1, term_2))
```

## The QubitOperator data structure
The QubitOperator data structure is another essential part of openfermion. As the name suggests, QubitOperator is used to store qubit operators in almost exactly the same way that FermionOperator is used to store fermion operators. For instance $X_0 Z_3 Y_4$ is a QubitOperator. The internal representation of this as a terms tuple would be $((0, \textrm{"X"}), (3, \textrm{"Z"}), (4, \textrm{"Y"}))$. Note that one important difference between QubitOperator and FermionOperator is that the terms in QubitOperator are always sorted in order of tensor factor. In some cases, this enables faster manipulation. We initialize some QubitOperators below.

```python
from openfermion.ops import QubitOperator

my_first_qubit_operator = QubitOperator('X1 Y2 Z3')
print(my_first_qubit_operator)
print(my_first_qubit_operator.terms)

operator_2 = QubitOperator('X3 Z4', 3.17)
operator_2 -= 77. * my_first_qubit_operator
print('')
print(operator_2)
```

## Jordan-Wigner and Bravyi-Kitaev
openfermion provides functions for mapping FermionOperators to QubitOperators.

```python
from openfermion.ops import FermionOperator
from openfermion.transforms import jordan_wigner, bravyi_kitaev
from openfermion.utils import hermitian_conjugated
from openfermion.linalg import eigenspectrum

# Initialize an operator.
fermion_operator = FermionOperator('2^ 0', 3.17)
fermion_operator += hermitian_conjugated(fermion_operator)
print(fermion_operator)

# Transform to qubits under the Jordan-Wigner transformation and print its spectrum.
jw_operator = jordan_wigner(fermion_operator)
print('')
print(jw_operator)
jw_spectrum = eigenspectrum(jw_operator)
print(jw_spectrum)

# Transform to qubits under the Bravyi-Kitaev transformation and print its spectrum.
bk_operator = bravyi_kitaev(fermion_operator)
print('')
print(bk_operator)
bk_spectrum = eigenspectrum(bk_operator)
print(bk_spectrum)
```

We see that despite the different representation, these operators are iso-spectral. We can also apply the Jordan-Wigner transform in reverse to map arbitrary QubitOperators to FermionOperators. Note that we also demonstrate the .compress() method (a method on both FermionOperators and QubitOperators) which removes zero entries.

```python
from openfermion.transforms import reverse_jordan_wigner

# Initialize QubitOperator.
my_operator = QubitOperator('X0 Y1 Z2', 88.)
my_operator += QubitOperator('Z1 Z4', 3.17)
print(my_operator)

# Map QubitOperator to a FermionOperator.
mapped_operator = reverse_jordan_wigner(my_operator)
print('')
print(mapped_operator)

# Map the operator back to qubits and make sure it is the same.
back_to_normal = jordan_wigner(mapped_operator)
back_to_normal.compress()
print('')
print(back_to_normal)
```

## Sparse matrices and the Hubbard model
Often, one would like to obtain a sparse matrix representation of an operator which can be analyzed numerically. There is code in both openfermion.transforms and openfermion.utils which facilitates this. The function get_sparse_operator converts either a FermionOperator, a QubitOperator or other more advanced classes such as InteractionOperator to a scipy.sparse.csc matrix. There are numerous functions in openfermion.utils which one can call on the sparse operators such as "get_gap", "get_hartree_fock_state", "get_ground_state", etc. We show this off by computing the ground state energy of the Hubbard model. To do that, we use code from the openfermion.hamiltonians module which constructs lattice models of fermions such as Hubbard models.

```python
from openfermion.hamiltonians import fermi_hubbard
from openfermion.linalg import get_sparse_operator, get_ground_state
from openfermion.transforms import jordan_wigner

# Set model.
x_dimension = 2
y_dimension = 2
tunneling = 2.
coulomb = 1.
magnetic_field = 0.5
chemical_potential = 0.25
periodic = 1
spinless = 1

# Get fermion operator.
hubbard_model = fermi_hubbard(
    x_dimension, y_dimension, tunneling, coulomb, chemical_potential,
    magnetic_field, periodic, spinless)
print(hubbard_model)

# Get qubit operator under Jordan-Wigner.
jw_hamiltonian = jordan_wigner(hubbard_model)
jw_hamiltonian.compress()
print('')
print(jw_hamiltonian)

# Get scipy.sparse.csc representation.
sparse_operator = get_sparse_operator(hubbard_model)
print('')
print(sparse_operator)
print('\nEnergy of the model is {} in units of T and J.'.format(
    get_ground_state(sparse_operator)[0]))
```

## Hamiltonians in the plane wave basis
A user can write plugins to openfermion which allow for the use of, e.g., third-party electronic structure package to compute molecular orbitals, Hamiltonians, energies, reduced density matrices, coupled cluster amplitudes, etc using Gaussian basis sets. We may provide scripts which interface between such packages and openfermion in future but do not discuss them in this tutorial.

When using simpler basis sets such as plane waves, these packages are not needed. openfermion comes with code which computes Hamiltonians in the plane wave basis. Note that when using plane waves, one is working with the periodized Coulomb operator, best suited for condensed phase calculations such as studying the electronic structure of a solid. To obtain these Hamiltonians one must choose to study the system without a spin degree of freedom (spinless), one must the specify dimension in which the calculation is performed (n_dimensions, usually 3), one must specify how many plane waves are in each dimension (grid_length) and one must specify the length scale of the plane wave harmonics in each dimension (length_scale) and also the locations and charges of the nuclei. One can generate these models with plane_wave_hamiltonian() found in openfermion.hamiltonians. For simplicity, below we compute the Hamiltonian in the case of zero external charge (corresponding to the uniform electron gas, aka jellium). We also demonstrate that one can transform the plane wave Hamiltonian using a Fourier transform without effecting the spectrum of the operator.

```python
from openfermion.hamiltonians import jellium_model
from openfermion.utils import Grid
from openfermion.linalg import eigenspectrum
from openfermion.transforms import jordan_wigner, fourier_transform

# Let's look at a very small model of jellium in 1D.
grid = Grid(dimensions=1, length=3, scale=1.0)
spinless = True

# Get the momentum Hamiltonian.
momentum_hamiltonian = jellium_model(grid, spinless)
momentum_qubit_operator = jordan_wigner(momentum_hamiltonian)
momentum_qubit_operator.compress()
print(momentum_qubit_operator)

# Fourier transform the Hamiltonian to the position basis.
position_hamiltonian = fourier_transform(momentum_hamiltonian, grid, spinless)
position_qubit_operator = jordan_wigner(position_hamiltonian)
position_qubit_operator.compress()
print('')
print (position_qubit_operator)

# Check the spectra to make sure these representations are iso-spectral.
spectral_difference = eigenspectrum(momentum_qubit_operator) -  eigenspectrum(position_qubit_operator)
print('')
print(spectral_difference)
```

## Basics of MolecularData class

Data from electronic structure calculations can be saved in an OpenFermion data structure called MolecularData, which makes it easy to access within our library. Often, one would like to analyze a chemical series or look at many different Hamiltonians and sometimes the electronic structure calculations are either expensive to compute or difficult to converge (e.g. one needs to mess around with different types of SCF routines to make things converge). Accordingly, we anticipate that users will want some way to automatically database the results of their electronic structure calculations so that important data (such as the SCF integrals) can be looked up on-the-fly if the user has computed them in the past. OpenFermion supports a data provenance strategy which saves key results of the electronic structure calculation (including pointers to files containing large amounts of data, such as the molecular integrals) in an HDF5 container.

The MolecularData class stores information about molecules. One initializes a MolecularData object by specifying parameters of a molecule such as its geometry, basis, multiplicity, charge and an optional string describing it. One can also initialize MolecularData simply by providing a string giving a filename where a previous MolecularData object was saved in an HDF5 container. One can save a MolecularData instance by calling the class's .save() method. This automatically saves the instance in a data folder specified during OpenFermion installation. The name of the file is generated automatically from the instance attributes and optionally provided description. Alternatively, a filename can also be provided as an optional input if one wishes to manually name the file.

When electronic structure calculations are run, the data files for the molecule can be automatically updated. If one wishes to later use that data they either initialize MolecularData with the instance filename or initialize the instance and then later call the .load() method.

Basis functions are provided to initialization using a string such as "6-31g". Geometries can be specified using a simple txt input file (see geometry_from_file function in molecular_data.py) or can be passed using a simple python list format demonstrated below. Atoms are specified using a string for their atomic symbol. Distances should be provided in angstrom. Below we initialize a simple instance of MolecularData without performing any electronic structure calculations.

```python
from openfermion.chem import MolecularData

# Set parameters to make a simple molecule.
diatomic_bond_length = .7414
geometry = [('H', (0., 0., 0.)), ('H', (0., 0., diatomic_bond_length))]
basis = 'sto-3g'
multiplicity = 1
charge = 0
description = str(diatomic_bond_length)

# Make molecule and print out a few interesting facts about it.
molecule = MolecularData(geometry, basis, multiplicity,
                         charge, description)
print('Molecule has automatically generated name {}'.format(
    molecule.name))
print('Information about this molecule would be saved at:\n{}\n'.format(
    molecule.filename))
print('This molecule has {} atoms and {} electrons.'.format(
    molecule.n_atoms, molecule.n_electrons))
for atom, atomic_number in zip(molecule.atoms, molecule.protons):
    print('Contains {} atom, which has {} protons.'.format(
        atom, atomic_number))
```

If we had previously computed this molecule using an electronic structure package, we can call molecule.load() to populate all sorts of interesting fields in the data structure. Though we make no assumptions about what electronic structure packages users might install, we assume that the calculations are saved in OpenFermion's MolecularData objects. Currently plugins are available for [Psi4](http://psicode.org/) [(OpenFermion-Psi4)](http://github.com/quantumlib/OpenFermion-Psi4) and [PySCF](https://github.com/sunqm/pyscf) [(OpenFermion-PySCF)](http://github.com/quantumlib/OpenFermion-PySCF), and there may be more in the future. For the purposes of this example, we will load data that ships with OpenFermion to make a plot of the energy surface of hydrogen. Note that helper functions to initialize some interesting chemical benchmarks are found in openfermion.utils.

```python
# Set molecule parameters.
basis = 'sto-3g'
multiplicity = 1
bond_length_interval = 0.1
n_points = 25

# Generate molecule at different bond lengths.
hf_energies = []
fci_energies = []
bond_lengths = []
for point in range(3, n_points + 1):
    bond_length = bond_length_interval * point
    bond_lengths += [bond_length]
    description = str(round(bond_length,2))
    print(description)
    geometry = [('H', (0., 0., 0.)), ('H', (0., 0., bond_length))]
    molecule = MolecularData(
        geometry, basis, multiplicity, description=description)

    # Load data.
    molecule.load()

    # Print out some results of calculation.
    print('\nAt bond length of {} angstrom, molecular hydrogen has:'.format(
        bond_length))
    print('Hartree-Fock energy of {} Hartree.'.format(molecule.hf_energy))
    print('MP2 energy of {} Hartree.'.format(molecule.mp2_energy))
    print('FCI energy of {} Hartree.'.format(molecule.fci_energy))
    print('Nuclear repulsion energy between protons is {} Hartree.'.format(
        molecule.nuclear_repulsion))
    for orbital in range(molecule.n_orbitals):
        print('Spatial orbital {} has energy of {} Hartree.'.format(
            orbital, molecule.orbital_energies[orbital]))
    hf_energies += [molecule.hf_energy]
    fci_energies += [molecule.fci_energy]

# Plot.
import matplotlib.pyplot as plt
%matplotlib inline

plt.figure(0)
plt.plot(bond_lengths, fci_energies, 'x-')
plt.plot(bond_lengths, hf_energies, 'o-')
plt.ylabel('Energy in Hartree')
plt.xlabel('Bond length in angstrom')
plt.show()
```

The geometry data needed to generate MolecularData can also be retreived from the PubChem online database by inputting the molecule's name.

```python
from openfermion.chem import geometry_from_pubchem

methane_geometry = geometry_from_pubchem('methane')
print(methane_geometry)
```

## InteractionOperator and InteractionRDM for efficient numerical representations

Fermion Hamiltonians can be expressed as $H = h_0 + \sum_{pq} h_{pq}\, a^\dagger_p a_q + \frac{1}{2} \sum_{pqrs} h_{pqrs} \, a^\dagger_p a^\dagger_q a_r a_s$ where $h_0$ is a constant shift due to the nuclear repulsion and $h_{pq}$ and $h_{pqrs}$ are the famous molecular integrals. Since fermions interact pairwise, their energy is thus a unique function of the one-particle and two-particle reduced density matrices which are expressed in second quantization as $\rho_{pq} = \left \langle p \mid a^\dagger_p a_q \mid q \right \rangle$ and $\rho_{pqrs} = \left \langle pq \mid a^\dagger_p a^\dagger_q a_r a_s \mid rs \right \rangle$, respectively.

Because the RDMs and molecular Hamiltonians are both compactly represented and manipulated as 2- and 4- index tensors, we can represent them in a particularly efficient form using  similar data structures. The InteractionOperator data structure can be initialized for a Hamiltonian by passing the constant $h_0$ (or 0), as well as numpy arrays representing $h_{pq}$ (or $\rho_{pq}$) and $h_{pqrs}$ (or $\rho_{pqrs}$). Importantly, InteractionOperators can also be obtained by calling MolecularData.get_molecular_hamiltonian() or by calling the function get_interaction_operator() (found in openfermion.transforms) on a FermionOperator. The InteractionRDM data structure is similar but represents RDMs. For instance, one can get a molecular RDM by calling MolecularData.get_molecular_rdm(). When generating Hamiltonians from the MolecularData class, one can choose to restrict the system to an active space.

These classes inherit from the same base class, PolynomialTensor. This data structure overloads the slice operator [] so that one can get or set the key attributes of the InteractionOperator: $\textrm{.constant}$, $\textrm{.one_body_coefficients}$ and $\textrm{.two_body_coefficients}$ . For instance, InteractionOperator[(p, 1), (q, 1), (r, 0), (s, 0)] would return $h_{pqrs}$ and InteractionRDM would return $\rho_{pqrs}$. Importantly, the class supports fast basis transformations using the method PolynomialTensor.rotate_basis(rotation_matrix).
But perhaps most importantly, one can map the InteractionOperator to any of the other data structures we've described here.

Below, we load MolecularData from a saved calculation of LiH. We then obtain an InteractionOperator representation of this system in an active space. We then map that operator to qubits. We then demonstrate that one can rotate the orbital basis of the InteractionOperator using random angles to obtain a totally different operator that is still iso-spectral.

```python
from openfermion.chem import MolecularData
from openfermion.transforms import get_fermion_operator, jordan_wigner
from openfermion.linalg import get_ground_state, get_sparse_operator
import numpy
import scipy
import scipy.linalg

# Load saved file for LiH.
diatomic_bond_length = 1.45
geometry = [('Li', (0., 0., 0.)), ('H', (0., 0., diatomic_bond_length))]
basis = 'sto-3g'
multiplicity = 1

# Set Hamiltonian parameters.
active_space_start = 1
active_space_stop = 3

# Generate and populate instance of MolecularData.
molecule = MolecularData(geometry, basis, multiplicity, description="1.45")
molecule.load()

# Get the Hamiltonian in an active space.
molecular_hamiltonian = molecule.get_molecular_hamiltonian(
    occupied_indices=range(active_space_start),
    active_indices=range(active_space_start, active_space_stop))

# Map operator to fermions and qubits.
fermion_hamiltonian = get_fermion_operator(molecular_hamiltonian)
qubit_hamiltonian = jordan_wigner(fermion_hamiltonian)
qubit_hamiltonian.compress()
print('The Jordan-Wigner Hamiltonian in canonical basis follows:\n{}'.format(qubit_hamiltonian))

# Get sparse operator and ground state energy.
sparse_hamiltonian = get_sparse_operator(qubit_hamiltonian)
energy, state = get_ground_state(sparse_hamiltonian)
print('Ground state energy before rotation is {} Hartree.\n'.format(energy))

# Randomly rotate.
n_orbitals = molecular_hamiltonian.n_qubits // 2
n_variables = int(n_orbitals * (n_orbitals - 1) / 2)
numpy.random.seed(1)
random_angles = numpy.pi * (1. - 2. * numpy.random.rand(n_variables))
kappa = numpy.zeros((n_orbitals, n_orbitals))
index = 0
for p in range(n_orbitals):
    for q in range(p + 1, n_orbitals):
        kappa[p, q] = random_angles[index]
        kappa[q, p] = -numpy.conjugate(random_angles[index])
        index += 1

    # Build the unitary rotation matrix.
    difference_matrix = kappa + kappa.transpose()
    rotation_matrix = scipy.linalg.expm(kappa)

    # Apply the unitary.
    molecular_hamiltonian.rotate_basis(rotation_matrix)

# Get qubit Hamiltonian in rotated basis.
qubit_hamiltonian = jordan_wigner(molecular_hamiltonian)
qubit_hamiltonian.compress()
print('The Jordan-Wigner Hamiltonian in rotated basis follows:\n{}'.format(qubit_hamiltonian))

# Get sparse Hamiltonian and energy in rotated basis.
sparse_hamiltonian = get_sparse_operator(qubit_hamiltonian)
energy, state = get_ground_state(sparse_hamiltonian)
print('Ground state energy after rotation is {} Hartree.'.format(energy))
```

## Quadratic Hamiltonians and Slater determinants

The general electronic structure Hamiltonian
$H = h_0 + \sum_{pq} h_{pq}\, a^\dagger_p a_q + \frac{1}{2} \sum_{pqrs} h_{pqrs} \, a^\dagger_p a^\dagger_q a_r a_s$ contains terms that act on up to 4 sites, or
is quartic in the fermionic creation and annihilation operators. However, in many situations
we may fruitfully approximate these Hamiltonians by replacing these quartic terms with
terms that act on at most 2 fermionic sites, or quadratic terms, as in mean-field approximation theory.
These Hamiltonians have a number of
special properties one can exploit for efficient simulation and manipulation of the Hamiltonian, thus
warranting a special data structure.  We refer to Hamiltonians which
only contain terms that are quadratic in the fermionic creation and annihilation operators
as quadratic Hamiltonians, and include the general case of non-particle conserving terms as in
a general Bogoliubov transformation.  Eigenstates of quadratic Hamiltonians can be prepared
efficiently on both a quantum and classical computer, making them amenable to initial guesses for
many more challenging problems.

A general quadratic Hamiltonian takes the form
$$H = \sum_{p, q} (M_{pq} - \mu \delta_{pq}) a^\dagger_p a_q + \frac{1}{2} \sum_{p, q} (\Delta_{pq} a^\dagger_p a^\dagger_q + \Delta_{pq}^* a_q a_p) + \text{constant},$$
where $M$ is a Hermitian matrix, $\Delta$ is an antisymmetric matrix,
$\delta_{pq}$ is the Kronecker delta symbol, and $\mu$ is a chemical
potential term which we keep separate from $M$ so that we can use it
to adjust the expectation of the total number of particles.
In OpenFermion, quadratic Hamiltonians are conveniently represented and manipulated
using the QuadraticHamiltonian class, which stores $M$, $\Delta$, $\mu$ and the constant. It is specialized to exploit the properties unique to quadratic Hamiltonians. Like InteractionOperator and InteractionRDM, it inherits from the PolynomialTensor class.

The BCS mean-field model of superconductivity is a quadratic Hamiltonian. The following code constructs an instance of this model as a FermionOperator, converts it to a QuadraticHamiltonian, and then computes its ground energy:

```python
from openfermion.hamiltonians import mean_field_dwave
from openfermion.transforms import get_quadratic_hamiltonian

# Set model.
x_dimension = 2
y_dimension = 2
tunneling = 2.
sc_gap = 1.
periodic = True

# Get FermionOperator.
mean_field_model = mean_field_dwave(
    x_dimension, y_dimension, tunneling, sc_gap, periodic=periodic)

# Convert to QuadraticHamiltonian
quadratic_hamiltonian = get_quadratic_hamiltonian(mean_field_model)

# Compute the ground energy
ground_energy = quadratic_hamiltonian.ground_energy()
print(ground_energy)
```

Any quadratic Hamiltonian may be rewritten in the form
$$H = \sum_p \varepsilon_p b^\dagger_p b_p + \text{constant},$$
where the $b_p$ are new annihilation operators that satisfy the fermionic anticommutation relations, and which are linear combinations of the old creation and annihilation operators. This form of $H$ makes it easy to deduce its eigenvalues; they are sums of subsets of the $\varepsilon_p$, which we call the orbital energies of $H$. The following code computes the orbital energies and the constant:

```python
orbital_energies, constant = quadratic_hamiltonian.orbital_energies()
print(orbital_energies)
print()
print(constant)
```

Eigenstates of quadratic hamiltonians are also known as fermionic Gaussian states, and they can be prepared efficiently on a quantum computer. One can use OpenFermion to obtain circuits for preparing these states. The following code obtains the description of a circuit which prepares the ground state (operations that can be performed in parallel are grouped together), along with a description of the starting state to which the circuit should be applied:

```python
from openfermion.circuits import gaussian_state_preparation_circuit

circuit_description, start_orbitals = gaussian_state_preparation_circuit(quadratic_hamiltonian)
for parallel_ops in circuit_description:
    print(parallel_ops)
print('')
print(start_orbitals)
```

In the circuit description, each elementary operation is either a tuple of the form $(i, j, \theta, \varphi)$, indicating the operation $\exp[i \varphi a_j^\dagger a_j]\exp[\theta (a_i^\dagger a_j - a_j^\dagger a_i)]$, which is a Givens rotation of modes $i$ and $j$, or the string 'pht', indicating the particle-hole transformation on the last fermionic mode, which is the operator $\mathcal{B}$ such that $\mathcal{B} a_N \mathcal{B}^\dagger = a_N^\dagger$ and leaves the rest of the ladder operators unchanged. Operations that can be performed in parallel are grouped together.

In the special case that a quadratic Hamiltonian conserves particle number ($\Delta = 0$), its eigenstates take the form
$$\lvert \Psi_S \rangle = b^\dagger_{1}\cdots b^\dagger_{N_f}\lvert \text{vac} \rangle,\qquad
b^\dagger_{p} = \sum_{k=1}^N Q_{pq}a^\dagger_q,$$
where $Q$ is an $N_f \times N$ matrix with orthonormal rows. These states are also known as Slater determinants. OpenFermion also provides functionality to obtain circuits for preparing Slater determinants starting with the matrix $Q$ as the input.