# Policy-Gradients with the REINFORCE algorithm

**Background**:
In this practical we will train an agent using the REINFORCE algorithm to learn to balance a pole in the OpenAI gym [Cartpole environment](https://gym.openai.com/envs/CartPole-v1).

**Learning objectives**:
* Understand the policy-gradient approach to directly training a parameterised policy to maximise expected future rewards.
* Understand how the policy-gradient theorem allows us to improve the policy using on-policy state = env.reset().

**What is expected of you**:
 * Go through the explanation, keeping the above learning objectives in mind.
 * Fill in the missing code ("#IMPLEMENT-ME") and train a model to solve the Cartpole-v1 environment in OpenAI gym (you solve it when reward=500).

# A Simple Policy-Gradient Cartpole Example

## Introduction

We have seen in your course that there are many different approaches to training RL agents. In this practical we will take a look at REINFORCE - a simple policy-based method. REINFORCE (and policy-based methods in general) directly optimise a parametrised policy in order to maximise future rewards.

We will try to learn a policy $\pi_\theta(a | s)$ which outputs a distribution over the possible actions $a$, given the current state $s$ of the environment. The goal is find a set of parameters $\theta$ to maximise the expected discounted return:
\begin{align}
J(\theta) = \mathbb{E}_{\tau \sim p_\theta} \left[\sum_{t=0}^T \gamma^t r(s_t, a_t)\right],
\end{align}
where $\tau$ is a trajectory sampled from $p_\theta$. The **policy-gradient** theorem gives us the derivative of this objective function:
\begin{align}
    \nabla_\theta J(\theta) = \mathbb{E}_{\tau \sim p_\theta} \left[\left(\sum_{t=0}^{T} \nabla_\theta \log \pi_\theta(a_t|s_t)\right) \left(\sum_{t=0}^T \gamma^t r(s_t, a_t) \right) \right].
\end{align}

**NOTE**:
* We have a policy $\pi_\theta(a|s)$ which tells the agent which action $a$ to take, given the state $s$, and it is parameterised in terms of parameters $\theta$.
* Our goal is to maximise $J(\theta)$ by **choosing actions from this policy** that lead to high future rewards.
* We'll use gradient-based optimisation to update the policy parameters $\theta$. We therefore want the gradient of our objective wrt the policy parameters.
* We use the policy-gradient theorem to find a expression for the gradient. This is an expectation over trajectories from our policy and the environment.
* Since we can now sample trajectories $(s_0, a_0, R_0, s_1, a_1, R_1, \ldots)$ using our policy $\pi_\theta$, we can approximate this gradient using **[Monte-Carlo](https://en.wikipedia.org/wiki/Monte_Carlo_integration)** methods.

This algorithm is called **Monte-Carlo REINFORCE**, and is one type of policy-gradient algorithm. Let's use this to solve the Cartpole environment!

**Monte-Carlo REINFORCE**:

for each episode:
1. sample a trajectory $\tau$ using the policy $\pi_\theta$.
2. compute $\nabla_\theta J(\theta) \approx \left(\sum_{t=0}^{T} \nabla_\theta \log \pi_\theta(a_t|s_t)\right) \left(\sum_{t=0}^T \gamma^t r(s_t, a_t) \right)$.
3. update policy parameters $\theta \leftarrow \theta + \alpha \nabla_\theta J(\theta)$

```python
# import various packages
from collections import deque

import numpy as np
import matplotlib.pyplot as plt
import gym

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.distributions import Categorical
```

```python
# use gpu if available
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# configure matplotlib
%matplotlib inline
plt.rcParams['figure.figsize'] = (15.0, 10.0) # set default size of plots
plt.rcParams['image.interpolation'] = 'nearest'
plt.rcParams['image.cmap'] = 'gray'
```

## The Environment

Cartpole is a standard benchmark in reinforcement learning and is a good sandbox for trying things out. The goal is to balance a pendulum on top of a moving cart. We have 2 actions - either push the cart to the left or push to the right. The state space consists of the cart's position and velocity and the pendulum's angle and angular velocity. Let's create the environment and take a look at the state and action spaces:

```python
env = gym.make('CartPole-v1')
print('action space:', env.action_space)
print('observation space:', env.observation_space)
```

    action space: Discrete(2)
    observation space: Box(4,)

Here we can see that there are 2 discrete actions and a continuous state space.

### Taking a few steps

To get a better feel for the environment, we will use a random policy to genrate a short trajectory.

```python
SUB = str.maketrans("t0123456789+", "ₜ₀₁₂₃₄₅₆₇₈₉₊")
print('-'*115)

state = env.reset()
for i in range (5):
    # sample random action
    action = env.action_space.sample()
    # take action in the environment
    new_state, reward, done, _ = env.step(action)

    print('Step t=', i+1, ': (', 'st, at , rt, st+1'.translate(SUB),')')
    print('(', state, ',', action, ',', reward, ',', new_state, ')')
    print('-'*115)

    state = new_state
```

    -------------------------------------------------------------------------------------------------------------------
    Step t= 1 : ( sₜ, aₜ , rₜ, sₜ₊₁ )
    ( [-0.04343247 -0.01849495  0.02620421 -0.01618236] , 0 , 1.0 , [-0.04380237 -0.21398271  0.02588056  0.28465177] )
    -------------------------------------------------------------------------------------------------------------------
    Step t= 2 : ( sₜ, aₜ , rₜ, sₜ₊₁ )
    ( [-0.04380237 -0.21398271  0.02588056  0.28465177] , 1 , 1.0 , [-0.04808202 -0.01923926  0.0315736   0.00024245] )
    -------------------------------------------------------------------------------------------------------------------
    Step t= 3 : ( sₜ, aₜ , rₜ, sₜ₊₁ )
    ( [-0.04808202 -0.01923926  0.0315736   0.00024245] , 1 , 1.0 , [-0.04846681  0.17541599  0.03157844 -0.2823138 ] )
    -------------------------------------------------------------------------------------------------------------------
    Step t= 4 : ( sₜ, aₜ , rₜ, sₜ₊₁ )
    ( [-0.04846681  0.17541599  0.03157844 -0.2823138 ] , 0 , 1.0 , [-0.04495849 -0.02014182  0.02593217  0.02015919] )
    -------------------------------------------------------------------------------------------------------------------
    Step t= 5 : ( sₜ, aₜ , rₜ, sₜ₊₁ )
    ( [-0.04495849 -0.02014182  0.02593217  0.02015919] , 1 , 1.0 , [-0.04536132  0.17459882  0.02633535 -0.26423036] )
    -------------------------------------------------------------------------------------------------------------------

### Watching a random policy agent play

Let's also see how a random policy performs in this enviroment:

```python
env_1 = gym.make('CartPole-v1')
state = env_1.reset()
for t in range(200):
    # sample a random action
    action = env_1.action_space.sample()
    env_1.render()
    state, reward, done, _ = env_1.step(action)
env_1.close()
```

    WARN: You are calling 'step()' even though this environment has already returned done = True. You should always call 'reset()' once you receive 'done = True' -- any further steps are undefined behavior.

Not very good! The pole only stayed up for a few time steps... Now let's improve things using REINFORCE.

## The Policy

We begin by parameterising the policy $\pi_\theta(a | s)$ as a simple neural network which takes the state (a vector of 4 elements provided by `gym`) as input, and produces a Categorical distribution over the possible actions as output. Simple enough. Refer to [torch.nn](https://pytorch.org/docs/stable/nn.html)

```python
class Policy(nn.Module):
    def __init__(self):
        super(Policy, self).__init__()
         # Define neural network layers. Refer to nn.Linear (https://pytorch.org/docs/stable/nn.html#torch.nn.Linear)
        # We are going to use a neural network with one hidden layer of size 16.
        # The first layer should have an input size of env.observation_space.shape and an output size of 16
        self.fc1 = nn.Linear(4, 16)
        # The second layer should have an input size of 16 and an output size of env.action_space.n
        self.fc2 = nn.Linear(16, 2)

    def forward(self, x):
        # IMPLEMENT-ME
        # Implement the forward pass
        # apply a ReLU activation after the first linear layer
        x = F.relu(self.fc1(x))
        # apply the second linear layer (without an activation).
        # the outputs of the second layer will act as the log probabilities for the Categorial distribution.
        x = self.fc2(x)
        return Categorical(logits=x)
```

### Selecting actions with our policy

For a given state our policy returns a pytorch `Categorial` object. We can sample from this distribution by calling it's `sample` method and we can find the log probability of an action using `log_prob`:

```python
policy = Policy().to(device)
state = env.reset()
# convert state (a numpy array) to a torch tensor
state = torch.from_numpy(state).float().to(device)
dist = policy(state)
action = dist.sample()

print("Sampled action: ", action.item())
print("Log probability of action: ", dist.log_prob(action).item())
print(dist.log_prob(action))
print(dist.log_prob(action).unsqueeze(0))
```

    Sampled action:  1
    Log probability of action:  -0.8268687725067139
    tensor(-0.8269, grad_fn=<SqueezeBackward1>)
    tensor([-0.8269], grad_fn=<UnsqueezeBackward0>)

### Computing the return

Given a sequence of rewards $(r_0, \ldots, r_T)$ we want to calculate the return $\sum_{t=0}^T \gamma^t r(s_t, a_t)$.

```python
def compute_returns(rewards, gamma):
    # IMPLEMENT-ME
    # compute the return using the above equation
    returns = 0
    for t in range(len(rewards)):
        returns += gamma**t * rewards[t]
    return returns
```

## REINFORCE

Now its time to implement the algorithm

**Monte-Carlo REINFORCE**:

for each episode:
1. sample a trajectory $\tau$ using the policy $\pi_\theta$.
2. compute $\nabla_\theta J(\theta) \approx \left(\sum_{t=0}^{T} \nabla_\theta \log \pi_\theta(a_t|s_t)\right) \left(\sum_{t=0}^T \gamma^t r(s_t, a_t) \right)$.
3. update policy parameters $\theta \leftarrow \theta + \alpha \nabla_\theta J(\theta)$

### Hyperameters

```python
learning_rate = 1e-2
number_episodes = 1500
max_episode_length = 1000
gamma = 1.0
```

```python
def reinforce(seed, verbose=True):
    # set random seeds (for reproducibility)
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    env.seed(seed)

    # instantiate the policy and optimizer
    policy = Policy().to(device)
    optimizer = optim.Adam(policy.parameters(), lr=learning_rate)

    scores = []
    scores_deque = deque(maxlen=100)
    for episode in range(1, number_episodes+1):
        #################################################################
        # 1. Collect trajectories using our policy and save the rewards #
        # and the log probability of each action taken.                 #
        #################################################################
        log_probs = []
        rewards = []
        state = env.reset()
        for t in range(max_episode_length):
            # IMPLEMENT-ME: get the distribution over actions for state
            torch_state = torch.from_numpy(state).float().to(device)
            dist = policy(torch_state )

            # IMPLEMENT-ME: sample an action from the distribution
            action = dist.sample()

            # IMPLEMENT-ME: compute the log probability
            log_prob = dist.log_prob(action).unsqueeze(0)

            # IMPLEMENT-ME: take a step in the environment
            state, reward, done, _ = env.step(action.item())

            # save the reward and log probability
            rewards.append(reward)
            log_probs.append(log_prob.unsqueeze(0))

            if done:
                break

        # for reporting save the score
        scores.append(sum(rewards))
        scores_deque.append(sum(rewards))

        #################################################################
        # 2. evaluate the policy gradient                               #
        #################################################################

        # IMPLEMENT-ME: calculate the discounted return of the trajectory
        returns = compute_returns(rewards, gamma)

        log_probs = torch.cat(log_probs)
        # IMPLEMENT-ME: multiply the log probabilities by the returns and sum (see the policy-gradient theorem)
        # Remember to multiply the result by -1 because we want to maximise the returns
        policy_loss = -torch.sum(log_probs * returns)

        #################################################################
        # 3. update the policy parameters (gradient descent)            #
        #################################################################
        optimizer.zero_grad()
        policy_loss.backward()
        optimizer.step()

        # report the score to check that we're making progress
        if episode % 50 == 0 and verbose:
            print('Episode {}\tAverage Score: {:.2f}'.format(episode, np.mean(scores_deque)))

        if np.mean(scores_deque) >= 495.0 and verbose:
            print('Environment solved in {:d} episodes!\tAverage Score: {:.2f}'.format(episode, np.mean(scores_deque)))
            break

    return policy, scores
```

```python
policy, scores = reinforce(42)
```

    Episode 50	Average Score: 27.78
    Episode 100	Average Score: 32.39
    Episode 150	Average Score: 42.82
    Episode 200	Average Score: 46.35
    Episode 250	Average Score: 51.75
    Episode 300	Average Score: 58.77
    Episode 350	Average Score: 67.28
    Episode 400	Average Score: 78.60
    Episode 450	Average Score: 79.20
    Episode 500	Average Score: 82.28
    Episode 550	Average Score: 101.07
    Episode 600	Average Score: 111.77
    Episode 650	Average Score: 115.17
    Episode 700	Average Score: 252.52
    Episode 750	Average Score: 345.24
    Episode 800	Average Score: 304.47
    Episode 850	Average Score: 383.91
    Episode 900	Average Score: 407.94
    Episode 950	Average Score: 248.61
    Episode 1000	Average Score: 156.76
    Episode 1050	Average Score: 255.69
    Episode 1100	Average Score: 422.78
    Environment solved in 1126 episodes!	Average Score: 495.12

## Seeing our learned policy in action

Let's watch our learned policy balance the pole!

```python
env = gym.make('CartPole-v1')
state = env.reset()
for t in range(2000):
    dist = policy(torch.from_numpy(state).float().to(device))
    action = dist.sample()
    env.render()
    state, reward, done, _ = env.step(action.item())
    if done:
        break
env.close()
```

### Plotting the results

Finally, let's plot the learning curve.

```python
def moving_average(a, n) :
    ret = np.cumsum(a, dtype=float)
    ret[n:] = ret[n:] - ret[:-n]
    return ret / n
```

```python
fig = plt.figure()
ax = fig.add_subplot(111)
x = np.arange(1, len(scores)+1)
ax.plot(x, scores, label='Score')
m_average = moving_average(scores, 50)
ax.plot(x, m_average, label='Moving Average (w=100)', linestyle='--')
plt.legend()
plt.ylabel('Score')
plt.xlabel('Episode #')
plt.title('REINFORCE learning curve - CartPole-v1')
plt.show()
```

Here is what your graph should look like.

We can see that at the end of training our policy consistantly (more or less) recieves returns of 500.

## Investigating the variance of REINFORCE

We noted in class that REINFORCE is a high variance algorithm. We can investigate the variance by running multiple trials and averaging the results.

```python
np.random.seed(53)
seeds = np.random.randint(1000, size=5)
all_scores = []
for seed in seeds:
    print("started training with seed: ", seed)
    _, scores = reinforce(int(seed), verbose=False)
    print("completed training with seed: ", seed)
    all_scores.append(scores)
```

    started training with seed:  537
    completed training with seed:  537
    started training with seed:  797
    completed training with seed:  797
    started training with seed:  885
    completed training with seed:  885
    started training with seed:  421
    completed training with seed:  421
    started training with seed:  763
    completed training with seed:  763

```python
smoothed_scores = [moving_average(s, 50) for s in all_scores]
smoothed_scores = np.array(smoothed_scores)
mean = smoothed_scores.mean(axis=0)
std = smoothed_scores.std(axis=0)
```

```python
fig = plt.figure()
ax = fig.add_subplot(111)
x = np.arange(1, len(mean)+1)
ax.plot(x, mean, '-', color='blue')
ax.fill_between(x, mean - std, mean + std, color='blue', alpha=0.2)
plt.ylabel('Score')
plt.xlabel('Episode #')
plt.title('REINFORCE averaged over 5 seeds')
plt.show()
```

Here is what your graph should look like.

## Reducing the variance of REINFORCE

In class we saw a couple of tricks to reduce the variance of REINFORCE and improve its performance. Firstly, future actions should not change past decision. Present actions only impact the future. Therefore, we can change our objective function to reflect this:
\begin{align}
    \nabla_\theta J(\theta) = \mathbb{E}_{\tau \sim p_\theta} \left[\sum_{t=0}^{T} \nabla_\theta \log \pi_\theta(a_t|s_t) \sum_{t'= t}^T \gamma^{t'- t} r(s_{t'}, a_{t'})\right].
\end{align}

We can also reduce variance by subtracing a state dependent baseline to get:
\begin{align}
    \nabla_\theta J(\theta) = \mathbb{E}_{\tau \sim p_\theta} \left[\sum_{t=0}^{T} \nabla_\theta \log \pi_\theta(a_t|s_t) \sum_{t'= t}^T \left( \gamma^{t'- t} r(s_{t'}, a_{t'}) - b(s_{t'}) \right)\right].
\end{align}

For our baseline we'll use the average of the returns over the trajectory. As a final trick we normalise the returns by dividing by the standard deviation.

```python
def compute_returns_baseline(rewards, gamma):
    r = 0
    returns = []
    for step in reversed(range(len(rewards))):
        r = rewards[step] + gamma * r
        returns.insert(0, r)
    returns = np.array(returns)
    # IMPLEMENT-ME: normalize the returns by subtracting the mean and dividing by the standard deviation
    #mean =  np.mean(returns, 0)
    #std  =
    returns = (returns -returns.mean()) / returns.std()
    return returns
```

```python
def reinforce_baseline(seed, verbose=True):
    # set random seeds (for reproducibility)
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    env.seed(seed)

    # instantiate the policy and optimizer
    policy = Policy().to(device)
    optimizer = optim.Adam(policy.parameters(), lr=learning_rate)

    scores = []
    scores_deque = deque(maxlen=100)
    for episode in range(1, number_episodes+1):
        #################################################################
        # 1. Collect trajectories using our policy and save the rewards #
        # and the log probability of each action taken.                 #
        #################################################################
        log_probs = []
        rewards = []
        state = env.reset()
        for t in range(max_episode_length):
            # IMPLEMENT-ME: get the distribution over actions for state
            dist = policy(torch.from_numpy(state).float().to(device))

            # IMPLEMENT-ME: sample an action from the distribution
            action = dist.sample()

            # IMPLEMENT-ME: compute the log probability
            log_prob = dist.log_prob(action)

            # IMPLEMENT-ME: take a step in the environment
            state, reward, done, _ = env.step(action.item())

            # save the reward and log probability
            rewards.append(reward)
            log_probs.append(log_prob.unsqueeze(0))

            if done:
                break

        # for reporting save the score
        scores.append(sum(rewards))
        scores_deque.append(sum(rewards))

        #################################################################
        # 2. evaluate the policy gradient (with variance reduction)     #
        #################################################################

        # calculate the discounted return of the trajectory
        returns = compute_returns_baseline(rewards, gamma)
        returns = torch.from_numpy(returns).float().to(device)

        log_probs = torch.cat(log_probs)
        # IMPLEMENT-ME: multiply the log probabilities by the returns and sum (see the policy-gradient theorem)
        # Remember to multiply the result by -1 because we want to maximise the returns
        policy_loss = -torch.sum(log_probs * returns)

        #################################################################
        # 3. update the policy parameters (gradient descent)            #
        #################################################################
        optimizer.zero_grad()
        policy_loss.backward()
        optimizer.step()

        # report the score to check that we're making progress
        if episode % 50 == 0 and verbose:
            print('Episode {}\tAverage Score: {:.2f}'.format(episode, np.mean(scores_deque)))

        if np.mean(scores_deque) >= 495.0 and verbose:
            print('Environment solved in {:d} episodes!\tAverage Score: {:.2f}'.format(episode, np.mean(scores_deque)))
            break

    return policy, scores
```

Let's see if these changes give us any improvement.

```python
np.random.seed(53)
seeds = np.random.randint(1000, size=5)
all_scores_baseline = []
for seed in seeds:
    print("started training with seed: ", seed)
    _, scores = reinforce_baseline(int(seed), verbose=False)
    print("completed training with seed: ", seed)
    all_scores_baseline.append(scores)
```

    started training with seed:  537
    completed training with seed:  537
    started training with seed:  797
    completed training with seed:  797
    started training with seed:  885
    completed training with seed:  885
    started training with seed:  421
    completed training with seed:  421
    started training with seed:  763
    completed training with seed:  763

## Comparing the methods

Finally we'll compare the performance of the two methods.

```python
smoothed_scores_baseline = [moving_average(s, 50) for s in all_scores_baseline]
smoothed_scores_baseline = np.array(smoothed_scores_baseline)
mean_baseline = smoothed_scores_baseline.mean(axis=0)
std_baseline = smoothed_scores_baseline.std(axis=0)
```

```python
fig = plt.figure()
ax = fig.add_subplot(111)
x = np.arange(1, len(mean_baseline)+1)
ax.plot(x, mean_baseline, '-', color='green', label='Variance reduced REINFORCE')
ax.plot(x, mean, '-', color='blue', label='REINFORCE')
ax.fill_between(x, mean_baseline - std_baseline, mean_baseline + std_baseline, color='green', alpha=0.2)
ax.fill_between(x, mean - std, mean + std, color='blue', alpha=0.2)
plt.ylabel('Score')
plt.xlabel('Episode #')
plt.legend()
plt.title('Comparison of REINFORCE and Variance reduced REINFORCE (averaged over 5 seeds)')
plt.show()
```

Here is what your graph should look like.

```python

```


# Project 1, deadline  March 16

**[Computational Physics I FYS4411/FYS9411](http://www.uio.no/studier/emner/matnat/fys/FYS4411/index-eng.html)**, Department of Physics, University of Oslo, Norway

Date: **Jan 17, 2018**

Copyright 1999-2018, [Computational Physics I FYS4411/FYS9411](http://www.uio.no/studier/emner/matnat/fys/FYS4411/index-eng.html). Released under CC Attribution-NonCommercial 4.0 license

## Introduction

 The spectacular demonstration of Bose-Einstein condensation (BEC) in gases of
 alkali atoms $^{87}$Rb, $^{23}$Na, $^7$Li confined in magnetic
 traps has led to an explosion of interest in
 confined Bose systems. Of interest is the fraction of condensed atoms, the
 nature of the condensate, the excitations above the condensate, the atomic
 density in the trap as a function of Temperature and the critical temperature of BEC,
 $T_c$.

 A key feature of the trapped alkali and atomic hydrogen systems is that they are
 dilute. The characteristic dimensions of a typical trap for $^{87}$Rb is
 $a_{h0}=\left( {\hbar}/{m\omega_\perp}\right)^\frac{1}{2}=1-2 \times 10^4$
 \AA\ . The interaction between $^{87}$Rb atoms can be well represented
 by its s-wave scattering length, $a_{Rb}$. This scattering length lies in the
 range $85 < a_{Rb} < 140 a_0$ where $a_0 = 0.5292$ \AA\ is the Bohr radius.
 The definite value $a_{Rb} = 100 a_0$ is usually selected and
 for calculations the definite ratio of atom size to trap size
 $a_{Rb}/a_{h0} = 4.33 \times 10^{-3}$
 is usually chosen. A typical $^{87}$Rb atom
 density in the trap is $n \simeq 10^{12}- 10^{14}$ atoms/cm$^3$ giving an
 inter-atom spacing $\ell \simeq 10^4$ \AA. Thus the effective atom size is small
 compared to both the trap size and the inter-atom spacing, the condition
 for diluteness ($na^3_{Rb} \simeq 10^{-6}$ where $n = N/V$ is the number
 density).

Many theoretical studies of Bose-Einstein condensates (BEC) in gases
of alkali atoms confined in magnetic or optical traps have been
conducted in the framework of the Gross-Pitaevskii (GP) equation.  The
key point for the validity of this description is the dilute condition
of these systems, that is, the average distance between the atoms is
much larger than the range of the inter-atomic interaction. In this
situation the physics is dominated by two-body collisions, well
described in terms of the $s$-wave scattering length $a$.  The crucial
parameter defining the condition for diluteness is the gas parameter
$x(\mathbf{r})= n(\mathbf{r}) a^3$, where $n(\mathbf{r})$ is the local density
of the system. For low values of the average gas parameter $x_{av}\le 10^{-3}$, the mean field Gross-Pitaevskii equation does an excellent
job.  However,
in recent experiments, the local gas parameter may well exceed this
value due to the possibility of tuning the scattering length in the
presence of a so-called Feshbach resonance.

Thus, improved many-body methods like Monte Carlo calculations may be
needed.  The aim of this project is to use the Variational Monte Carlo
(VMC) method and evaluate the ground state energy of a trapped, hard
sphere Bose gas for different numbers of particles with a specific
trial wave function.

 This trial wave function is used to study the sensitivity of
 condensate and non-condensate properties to the hard sphere radius
 and the number of particles.  The trap we will use is a spherical (S)
 or an elliptical (E) harmonic trap in one, two and finally three
 dimensions, with the latter given by

$$
\begin{equation}
 V_{ext}(\mathbf{r}) =
 \Bigg\{
 \begin{array}{ll}
	 \frac{1}{2}m\omega_{ho}^2r^2 & (S)\\
 \strut
	 \frac{1}{2}m[\omega_{ho}^2(x^2+y^2) + \omega_z^2z^2] & (E)
\label{trap_eqn} \tag{1}
 \end{array}
 \end{equation}
$$

where (S) stands for symmetric and

$$
\begin{equation}
     H = \sum_i^N \left\frac{-\hbar^2}{2m}
	 { \bigtriangledown }_{i}^2 +
	 V_{ext}({_r_}_i)\right)  +
	 \sum_{i<j}^{N} V_{int}({\mathbf{r}}_i,{\mathbf{r}}_j),
\label{_auto1} \tag{2}
\end{equation}
$$

as the two-body Hamiltonian of the system.  Here $\omega_{ho}^2$
 defines the trap potential strength.  In the case of the elliptical
 trap, $V_{ext}(x,y,z)$, $\omega_{ho}=\omega_{\perp}$ is the trap
 frequency in the perpendicular or $xy$ plane and $\omega_z$ the
 frequency in the $z$ direction.  The mean square vibrational
 amplitude of a single boson at $T=0K$ in the trap ([trap_eqn](#trap_eqn)) is
 $\langle x^2\rangle=(\hbar/2m\omega_{ho})$ so that $a_{ho} \equiv
 (\hbar/m\omega_{ho})^{\frac{1}{2}}$ defines the characteristic length
 of the trap.  The ratio of the frequencies is denoted
 $\lambda=\omega_z/\omega_{\perp}$ leading to a ratio of the trap
 lengths $(a_{\perp}/a_z)=(\omega_z/\omega_{\perp})^{\frac{1}{2}} =
 \sqrt{\lambda}$.

 We will represent the inter-boson interaction by a pairwise,
 repulsive potential

$$
\begin{equation}
 V_{int}(|\mathbf{r}_i-\mathbf{r}_j|) =  \Bigg\{
 \begin{array}{ll}
v	 \infty & {|\mathbf{r}_i-\mathbf{r}_j|} \leq {a}\\
	 0 & {|\mathbf{r}_i-_r\mathbf{r}_j|} > {a}
 \end{array}
\label{_auto2} \tag{3}
\end{equation}
$$

where $a$ is the so-called hard-core diameter of the bosons.
 Clearly, $V_{int}(|\mathbf{r}_i-\mathbf{r}_j|)$ is zero if the bosons are
 separated by a distance $|\mathbf{r}_i-\mathbf{r}_j|$ greater than $a$ but
 infinite if they attempt to come within a distance $|\mathbf{r}_i-\mathbf{r}_j| \leq a$.

 Our trial wave function for the ground state with $N$ atoms is given by

$$
\begin{equation}
 \Psi_T(\mathbf{r})=\Psi_T(\mathbf{r}_1, \mathbf{r}_2, \dots \mathbf{r}_N,\alpha,\beta)=\prod_i g(\alpha,\beta,\mathbf{r}_i)\prod_{i<j}f(a,|\mathbf{r}_i-\mathbf{r}_j|),
\label{eq:trialwf} \tag{4}
 \end{equation}
$$

where $\alpha$ and $\beta$ are variational parameters. The
 single-particle wave function is proportional to the harmonic
 oscillator function for the ground state, i.e.,

$$
\begin{equation}
    g(\alpha,\beta,\mathbf{r}_i)= \exp{[-\alpha(x_i^2+y_i^2+\beta z_i^2)]}.
\label{_auto3} \tag{5}
\end{equation}
$$

For spherical traps we have $\beta = 1$ and for non-interacting
 bosons ($a=0$) we have $\alpha = 1/2a_{ho}^2$.  The correlation wave
 function is

$$
\begin{equation}
    f(a,|\mathbf{r}_i-\mathbf{r}_j|)=\Bigg\{
 \begin{array}{ll}
	 0 & {|\mathbf{r}_i-\mathbf{r}_j|} \leq {a}\\
	 (1-\frac{a}{|\mathbf{r}_i-\mathbf{r}_j|}) & {|\mathbf{r}_i-\mathbf{r}_j|} > {a}.
 \end{array}
\label{_auto4} \tag{6}
\end{equation}
$$

### Project 1 a): Find analytic expressions for the local energy

$$
\begin{equation}
    E_L(\mathbf{r})=\frac{1}{\Psi_T(\mathbf{r})}H\Psi_T(\mathbf{r}),
\label{eq:locale} \tag{7}
 \end{equation}
$$

for the above
 trial wave function of Eq. ([eq:trialwf](#eq:trialwf)).
Find first the local energy the case with only the harmonic oscillator potential, that is we set $a=0$.
Use first that $\beta =1$ and find the relevant local energies in one, two and three dimensions for one and
$N$ particles with the same mass.

 Compute also the analytic expression for the drift force to be used in importance sampling

$$
\begin{equation}
   F = \frac{2\nabla \Psi_T}{\Psi_T}.
\label{_auto5} \tag{8}
\end{equation}
$$

Find first the equivalent expressions for the just the harmonic oscillator part in one, two and three dimensions
with $\beta=1$.

Next, we will find the local energy for the full problem in three dimensions.
The tricky part is to find an analytic expressions for the derivative of the trial wave function

$$
\frac{1}{\Psi_T(\mathbf{r})}\sum_i^{N}\nabla_i^2\Psi_T(\mathbf{r}),
$$

with the above
trial wave function of Eq. ([eq:trialwf](#eq:trialwf)).
We rewrite (and we can use the same general expressions for projects 2 and 3)

$$
\Psi_T(\mathbf{r})=\Psi_T(\mathbf{r}_1, \mathbf{r}_2, \dots \mathbf{r}_N,\alpha,\beta)=\prod_i g(\alpha,\beta,\mathbf{r}_i)\prod_{i<j}f(a,|\mathbf{r}_i-\mathbf{r}_j|),
$$

as

$$
\Psi_T(\mathbf{r})=\prod_i g(\alpha,\beta,\mathbf{r}_i)\exp{\left(\sum_{i<j}u(r_{ij})\right)}
$$

where we have defined $r_{ij}=|\mathbf{r}_i-\mathbf{r}_j|$
and

$$
f(r_{ij})= \exp{\left(\sum_{i<j}u(r_{ij})\right)},
$$

with $u(r_{ij})=\ln{f(r_{ij})}$.
We have also

$$
g(\alpha,\beta,\mathbf{r}_i) = \exp{-\alpha(x_i^2+y_i^2+\beta z_i^2)}= \phi(\mathbf{r}_i).
$$

Show that the first  derivative for particle $k$ is

$$
\nabla_k\Psi_T(\mathbf{r}) = \nabla_k\phi(\mathbf{r}_k)\left[\prod_{i\ne k}\phi(\mathbf{r}_i)\right]\exp{\left(\sum_{i<j}u(r_{ij})\right)}+
\prod_i\phi(\mathbf{r}_i)\exp{\left(\sum_{i<j}u(r_{ij})\right)}\sum_{j\ne k}\nabla_k u(r_{ij}),
$$

and find the final expression for our specific trial function.
The expression for the second derivative is (show this)

$$
\frac{1}{\Psi_T(\mathbf{r})}\nabla_k^2\Psi_T(\mathbf{r})=
   \frac{\nabla_k^2\phi(\mathbf{r}_k)}{\phi(\mathbf{r}_k)}+
\frac{\nabla_k\phi(\mathbf{r}_k)}{\phi(\mathbf{r}_k)}\left(\sum_{j\ne k}\frac{(\mathbf{r}_k-\mathbf{r}_j)}{r_{kj}}u'(r_{ij})\right)+
$$

$$
\sum_{ij\ne k}\frac{(\mathbf{r}_k-\mathbf{r}_i)(\mathbf{r}_k-\mathbf{r}_j)}{r_{ki}r_{kj}}u'(r_{ki})u'(r_{kj})+
\sum_{j\ne k}\left( u''(r_{kj})+\frac{2}{r_{kj}}u'(r_{kj})\right).
$$

Use this expression to find the final second derivative entering the definition of the local energy.
You need to get the analytic expression for this expression using the harmonic oscillator wave functions
and the correlation term defined in the project.

### Project 1 b): Developing the code

Write a Variational Monte Carlo program which uses standard
   Metropolis sampling and compute the ground state energy of a
   spherical harmonic oscillator ($\beta = 1$) with no interaction and
   one dimension.  Use natural units and make an analysis of your
   calculations using both the analytic expression for the local
   energy and a numerical calculation of the kinetic energy using
   numerical derivation.  Compare the CPU time difference.  The only
   variational parameter is $\alpha$. Perform these calculations for
   $N=1$, $N=10$, $100$ and $500$ atoms. Compare your results with the
   exact answer.  Extend then your results to two and three dimensions
   and compare with the analytical results.

### Project 1 c): Adding importance sampling

We repeat part b), but now we replace the brute force Metropolis algorithm with
importance sampling based on the Fokker-Planck and the Langevin equations.
Discuss your results and comment on eventual differences between importance sampling and brute force sampling.
Run the calculations for the one, two and three-dimensional systems only and without the repulsive potential.
Study the dependence of the results as a function of the time step $\delta t$.
Compare the results with those obtained under b) and comment eventual differences.

### Project 1 d): A better statistical analysis

In performing the Monte Carlo analysis we will use the
  blocking and bootstrap techniques to make the statistical analysis of the
  numerical data. Present your results with a proper evaluation of the
  statistical errors. Repeat the calculations from exercise c) and
  include a proper error analysis. Limit yourself to the
  three-dimensional case only.

### Project 1 e): The repulsive interaction

  * [e)] We turn now to the elliptic trap with a repulsive

   interaction.  We fix, as in Refs. [1,2] below,
   $a/a_{ho}=0.0043$. Introduce lengths in units of $a_{ho}$,
   $r\rightarrow r/a_{ho}$ and energy in units of $\hbar\omega_{ho}$.
   Show then that the original Hamiltonian can be rewritten as

$$
H=\sum_{i=1}^N\frac{1}{2}\left(-\nabla^2_i+x_i^2+y_i^2+\gamma^2z_i^2\right)+\sum_{i<j}V_{int}(|\mathbf{r}_i-\mathbf{r}_j|).
$$

What is the expression for $\gamma$?  Choose the initial value for
 $\beta=\gamma = 2.82843$ and set up a VMC program which computes the
 ground state energy using the trial wave function of
 Eq. ([eq:trialwf](#eq:trialwf))  using only $\alpha$ as variational
 parameter.  Use standard Metropolis sampling and vary the parameter
 $\alpha$ in order to find a minimum. Perform the calculations for
 $N=10,50$ and $N=100$ and compare your results to those from the
 ideal case in the previous exercises.  In actual calculations
 employing the Metropolis algorithm, all moves are recast into the
 chosen simulation cell with periodic boundary conditions. To carry
 out consistently the Metropolis moves, it has to be assumed that the
 correlation function has a range shorter than $L/2$. Then, to decide
 if a move of a single particle is accepted or not, only the set of
 particles contained in a sphere of radius $L/2$ centered at the
 referred particle have to be considered.  Benchmark your results with
 those of Refs. [1,2].

### Project 1 f): Finding the best parameter

Repeat the previous calculations by varying the energy using the
conjugate gradient method or similar methods to obtain the best possible parameter
$\alpha$.

### Project 1 g): Onebody densities

With the optimal parameters for the ground state wave function, compute again the onebody density with and without the Jastrow
factor.  How important are the correlations induced by the Jastrow factor?

# Literature

 1. J. L. DuBois and H. R. Glyde, H. R., *Bose-Einstein condensation in trapped bosons: A variational Monte Carlo analysis*, Phys. Rev. A **63**, 023602 (2001).

 2. J. K. Nilsen,  J. Mur-Petit, M. Guilleumas, M. Hjorth-Jensen, and A. Polls, *Vortices in atomic Bose-Einstein condensates in the large-gas-parameter region*, Phys. Rev. A **71**, 053610 (2005).

## Introduction to numerical projects

Here follows a brief recipe and recommendation on how to write a report for each
project.

  * Give a short description of the nature of the problem and the eventual  numerical methods you have used.

  * Describe the algorithm you have used and/or developed. Here you may find it convenient to use pseudocoding. In many cases you can describe the algorithm in the program itself.

  * Include the source code of your program. Comment your program properly.

  * If possible, try to find analytic solutions, or known limits in order to test your program when developing the code.

  * Include your results either in figure form or in a table. Remember to        label your results. All tables and figures should have relevant captions        and labels on the axes.

  * Try to evaluate the reliabilty and numerical stability/precision of your results. If possible, include a qualitative and/or quantitative discussion of the numerical stability, eventual loss of precision etc.

  * Try to give an interpretation of you results in your answers to  the problems.

  * Critique: if possible include your comments and reflections about the  exercise, whether you felt you learnt something, ideas for improvements and  other thoughts you've made when solving the exercise. We wish to keep this course at the interactive level and your comments can help us improve it.

  * Try to establish a practice where you log your work at the  computerlab. You may find such a logbook very handy at later stages in your work, especially when you don't properly remember  what a previous test version  of your program did. Here you could also record  the time spent on solving the exercise, various algorithms you may have tested or other topics which you feel worthy of mentioning.

## Format for electronic delivery of report and programs

The preferred format for the report is a PDF file. You can also use DOC or postscript formats or as an ipython notebook file.  As programming language we prefer that you choose between C/C++, Fortran2008 or Python. The following prescription should be followed when preparing the report:

  * Use Devilry to hand in your projects, log in  at  <http://devilry.ifi.uio.no> with your normal UiO username and password.

  * Upload **only** the report file!  For the source code file(s) you have developed please provide us with your link to your github domain.  The report file should include all of your discussions and a list of the codes you have developed.  The full version of the codes should be in your github repository.

  * In your github repository, please include a folder which contains selected results. These can be in the form of output from your code for a selected set of runs and input parameters.

  * Still in your github make a folder where you place your codes.

  * In this and all later projects, you should include tests (for example unit tests) of your code(s).

  * Comments  from us on your projects, approval or not, corrections to be made  etc can be found under your Devilry domain and are only visible to you and the teachers of the course.

Finally,
we encourage you to work two and two together. Optimal working groups consist of
2-3 students. You can then hand in a common report.