# Data analysis in the Amazon audit
Most of the materials in this notebook were kindly provided by Piotr Sapiezynski (www.sapiezynski.com)

### The motivation behind the analysis
Read it here
https://themarkup.org/amazons-advantage/2021/10/14/amazon-puts-its-own-brands-first-above-better-rated-products

```python
# installing the packages for today (if needed)
!pip install -U pandas
!pip install -U scikit-learn
!pip install -U statsmodels
```

    Requirement already satisfied: pandas in /Users/roberta/opt/anaconda3/lib/python3.9/site-packages (1.4.1)
    Collecting pandas
      Downloading pandas-1.4.2-cp39-cp39-macosx_10_9_x86_64.whl (11.1 MB)
         |████████████████████████████████| 11.1 MB 2.7 MB/s eta 0:00:01
    Requirement already satisfied: python-dateutil>=2.8.1 in /Users/roberta/opt/anaconda3/lib/python3.9/site-packages (from pandas) (2.8.2)
    Requirement already satisfied: pytz>=2020.1 in /Users/roberta/opt/anaconda3/lib/python3.9/site-packages (from pandas) (2021.3)
    Requirement already satisfied: numpy>=1.18.5 in /Users/roberta/opt/anaconda3/lib/python3.9/site-packages (from pandas) (1.20.3)
    Requirement already satisfied: six>=1.5 in /Users/roberta/opt/anaconda3/lib/python3.9/site-packages (from python-dateutil>=2.8.1->pandas) (1.16.0)
    Installing collected packages: pandas
      Attempting uninstall: pandas
        Found existing installation: pandas 1.4.1
        Uninstalling pandas-1.4.1:
          Successfully uninstalled pandas-1.4.1
    Successfully installed pandas-1.4.2
    Requirement already satisfied: scikit-learn in /Users/roberta/opt/anaconda3/lib/python3.9/site-packages (1.0.2)
    Requirement already satisfied: numpy>=1.14.6 in /Users/roberta/opt/anaconda3/lib/python3.9/site-packages (from scikit-learn) (1.20.3)
    Requirement already satisfied: threadpoolctl>=2.0.0 in /Users/roberta/opt/anaconda3/lib/python3.9/site-packages (from scikit-learn) (2.2.0)
    Requirement already satisfied: joblib>=0.11 in /Users/roberta/opt/anaconda3/lib/python3.9/site-packages (from scikit-learn) (1.1.0)
    Requirement already satisfied: scipy>=1.1.0 in /Users/roberta/opt/anaconda3/lib/python3.9/site-packages (from scikit-learn) (1.7.1)
    Requirement already satisfied: statsmodels in /Users/roberta/opt/anaconda3/lib/python3.9/site-packages (0.13.2)
    Requirement already satisfied: packaging>=21.3 in /Users/roberta/opt/anaconda3/lib/python3.9/site-packages (from statsmodels) (21.3)
    Requirement already satisfied: patsy>=0.5.2 in /Users/roberta/opt/anaconda3/lib/python3.9/site-packages (from statsmodels) (0.5.2)
    Requirement already satisfied: numpy>=1.17 in /Users/roberta/opt/anaconda3/lib/python3.9/site-packages (from statsmodels) (1.20.3)
    Requirement already satisfied: scipy>=1.3 in /Users/roberta/opt/anaconda3/lib/python3.9/site-packages (from statsmodels) (1.7.1)
    Requirement already satisfied: pandas>=0.25 in /Users/roberta/opt/anaconda3/lib/python3.9/site-packages (from statsmodels) (1.4.2)
    Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /Users/roberta/opt/anaconda3/lib/python3.9/site-packages (from packaging>=21.3->statsmodels) (3.0.4)
    Requirement already satisfied: pytz>=2020.1 in /Users/roberta/opt/anaconda3/lib/python3.9/site-packages (from pandas>=0.25->statsmodels) (2021.3)
    Requirement already satisfied: python-dateutil>=2.8.1 in /Users/roberta/opt/anaconda3/lib/python3.9/site-packages (from pandas>=0.25->statsmodels) (2.8.2)
    Requirement already satisfied: six in /Users/roberta/opt/anaconda3/lib/python3.9/site-packages (from patsy>=0.5.2->statsmodels) (1.16.0)

```python
import pandas as pd
from sklearn.model_selection import train_test_split
```

```python
# this is the data from the markup audit, downloaded from their github:
# https://github.com/the-markup/investigation-amazon-brands

data = pd.read_csv('https://www.sapiezynski.com/cs4910/markup/data.csv')
```

```python
data
```

      search_term
      placed_higher
      stars_delta
      reviews_delta
      is_shipped_by_amazon
      is_sold_by_amazon
      is_amazon
      is_top_clicked
      random_noise
      asin_1
      asin_2

      0
      #10 envelope
      True
      0.0
      1654.0
      0
      2
      2
      0
      -2.569736
      B06VVLD2GL
      B01D0OANU4

      1
      #6 envelope
      True
      0.1
      7844.0
      0
      2
      2
      2
      0.951532
      B06X15WSLL
      B07JNXMBSX

      2
      1 inch binder
      False
      0.0
      -9383.0
      0
      0
      -2
      0
      0.618294
      B00A45VF2S
      B01BRGTWOA

      3
      1% milk
      False
      0.0
      -183.0
      0
      0
      0
      -2
      0.645435
      B07W5Z8SJ8
      B07WC9MMPD

      4
      10 dollar gifts
      True
      0.8
      75410.0
      0
      2
      0
      0
      -1.075435
      B00F4CEHNK
      B07FCNYND8

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      1410
      zwave hub
      True
      0.8
      848.0
      0
      0
      0
      2
      0.047647
      B07D19VVTX
      B077Y939JQ

      1411
      zxzy womens hoodies
      True
      -0.1
      6.0
      0
      0
      0
      0
      0.009533
      B08F7RFZGP
      B08KT3BD2X

      1412
      zyliss knives
      True
      0.0
      3010.0
      0
      0
      0
      0
      -0.148278
      B01DKEB5OC
      B0753SKVZX

      1413
      zyrtec for kids
      True
      -0.2
      2184.0
      2
      2
      0
      0
      1.474693
      B004E0QGOQ
      B088X7151W

      1414
      zz plant
      True
      0.1
      379.0
      2
      2
      2
      -2
      1.686695
      B07QD5FXRZ
      B07RJYZ7W3

1415 rows × 11 columns

```python
columns = ['stars_delta', 'reviews_delta', 'is_amazon',
        'is_shipped_by_amazon', 'is_sold_by_amazon',
        'is_top_clicked', 'placed_higher']
```

```python
dataset = data[columns]
```

```python
train, test = train_test_split(dataset, test_size=0.2, stratify=dataset['placed_higher'])
```

```python
features = ['is_amazon',
        'is_shipped_by_amazon', 'is_sold_by_amazon',
        'is_top_clicked', 'stars_delta', 'reviews_delta', ]
```

```python
from sklearn.ensemble import RandomForestClassifier
clf = RandomForestClassifier(n_estimators=500, max_depth=3, max_features = 6)
clf.fit(train[features], train['placed_higher'])

```

    RandomForestClassifier(max_depth=3, max_features=6, n_estimators=500)

```python
# let's see how well it does in predicting the test set
# 1. predict the test data
y_pred = clf.predict(test[features])

# 2. calculate accuracy, i.e. the fraction of time that the prediction was correct out of all predictions
(y_pred == test['placed_higher']).sum()/len(y_pred)
```

    0.6855123674911661

```python
for feature, importance in zip(features, clf.feature_importances_):
    print(feature, importance)
```

    is_amazon 0.9032928487280287
    is_shipped_by_amazon 0.007487632851518247
    is_sold_by_amazon 0.002829947430288105
    is_top_clicked 0.002445399880340501
    stars_delta 0.019225107462222012
    reviews_delta 0.06471906364760226

So clearly, `is_amazon` is the most important, but what exactly does it mean, how does it translate the the probability of being the top result? RandomForests won't tell us... But logistic regression can.

First, quick reminder on regression.

**Linear** regression has the following formula:

$$ y = \beta_0 + \beta_1x_1 + \beta_2x_2 + ... $$

This means that:
* when all variables are equal to 0, the outcome variable $y$ is equal to $\beta_0$, the intercept.
* a **unit** change in $x_1$ corresponds to $y$ changing by $\beta_1$

It might not be a powerful model, but it offers a clear interpretation/explanation in regression problems.

**Logistic** regression is mostly used for binary classification and it looks similar to the linear regression, but now the outcome variable is the logarithm of odds of success.

$$ ln(\frac{\pi}{1-\pi}) = \beta_0 + \beta_1x_1 + \beta_2x_2 + ... $$

where $pi$ is the probability of success, so $1-\pi$ is the probability of failure.

Then, a unit change in $x_1$ corresponds to $\beta_1$ change in log odd ratios... but that is not intuitive at all. Let's rewrite it to calculate the probability directly:

$$\begin{align}
ln(\frac{\pi}{1-\pi}) &= \beta_0 + \beta_1x_1 + \beta_2x_2 + ... \\
\frac{\pi}{1-\pi} &= e^{\beta_0 + \beta_1x_1 + \beta_2x_2 + ...} \\
{\pi} &= ({1-\pi})e^{\beta_0 + \beta_1x_1 + \beta_2x_2 + ...} \\
{\pi} &= e^{\beta_0 + \beta_1x_1 + \beta_2x_2 + ...}-\pi e^{\beta_0 + \beta_1x_1 + \beta_2x_2 + ...}\\
{\pi}(1+e^{\beta_0 + \beta_1x_1 + \beta_2x_2 + ...}) &= e^{\beta_0 + \beta_1x_1 + \beta_2x_2 + ...} \\
{\pi} &= \frac{e^{\beta_0 + \beta_1x_1 + \beta_2x_2 + ...}}{1+e^{\beta_0 + \beta_1x_1 + \beta_2x_2 + ...}}
\end{align}$$

So, while we cannot directly say from that a unit change of $x_1$ corresponds to a certain change in the probability of the outcome, we can just plug zeros and ones into the equation above and compute the changes.

Let's load the data from scratch and do some cleaning (I'm dividing by two, because The Markup left the values of -2, 0, 2 so they never have a "unit" change. If we divide them by two, the unit change actually is meaningful).

```python
data = pd.read_csv('https://www.sapiezynski.com/cs4910/markup/data.csv')
data['is_shipped_by_amazon'] /= 2
data['is_sold_by_amazon'] /= 2
data['is_amazon'] /= 2
data['is_top_clicked'] /= 2
dataset = data[columns]

train, test = train_test_split(dataset, test_size=0.2, stratify=dataset['placed_higher'])

```

Let's start with a model that only has the intercept (constant) and `is_amazon` as features.

```python
import statsmodels.api as sm
from statsmodels.tools.tools import add_constant

# adding the intercept as a variable
train = add_constant(train)

# selecting which features to train on
features = ['const', 'is_amazon']

# training a logistic regression model. First the dependent (outcome variable), then the features
log_reg = sm.Logit(train['placed_higher'], train[features]).fit()
```

    Optimization terminated successfully.
             Current function value: 0.494906
             Iterations 6

```python
log_reg.summary()
```

Logit Regression Results

  Dep. Variable:	     placed_higher	    No. Observations:  	    1132

  Model:	                 Logit	        Df Residuals:      	    1130

  Method:	                 MLE	         Df Model:          	       1

  Date:	            Mon, 07 Feb 2022	   Pseudo R-squ.:     	  0.2859

  Time:	                11:20:43	       Log-Likelihood:    	  -560.23

  converged:	             True	         LL-Null:           	  -784.58

  Covariance Type:	     nonrobust	      LLR p-value:       	 1.391e-99

      	         coef	     std err	      z	      P>|z|	  [0.025	    0.975]

  const	         0.0883	     0.073	     1.208	  0.227	    -0.055	     0.232

  is_amazon	     2.4883	     0.163	    15.278	  0.000	     2.169	     2.807

We will mostly be looking at the bottom table.

`const` (the intercept) translates to the probability if all other variables are equal to 0

$$ p = \frac{e^{\beta_{const}}}{1+e^{\beta_{const}}} $$

```python
import numpy as np

# the beta coefficients are stored in the .params list, with the constant being first
# get the constant out of the fit model:
const = log_reg.params[0]

np.e**const/(1+np.e**const)

```

    0.5220557281748975

```python

```

    0.5220557281748975

This should be very close to 50%. In our case if all variables are zero, it means that either:
1. The products are both from amazon
2. The products are both not from amazon

We don't have a way to differentiate them, so it's a coin toss (50%)

The other coefficients describe the change in odds that corresponds to a unit change in the variable.
* if the coefficient is positive, it means that an **increase** of variable corresponds to an **increase** in the odds of the positive outcome
* if the coefficient is positive, it means that an **increase** of variable corresponds to an **decrease** in the odds of the positive outcome

So what happens when `is_amazon` is equal to 1, i.e. one product is an amazon product and the other isn't?
By looking at the `coef` we already know it's going to go up (because the coefficient is positive). But by How much? Let's calculate:

$$ p = \frac{e^{\beta_{const} + \beta_{is\_amazon}}}{1+e^{\beta_{const} + \beta_{is\_amazon}}} $$

```python
numerator = np.e**(log_reg.params[0] + log_reg.params[1])
print(numerator/(1+numerator))
```

    0.9293369710808036

That means that in this dataset 93\% of the time we have two products where one is from amazon and the other isn't, it's the amazon product that will be on the first place.

Solved? Well no, because maybe this is just because amazon products are better. Let's include the star difference in the analysis to **control** for this:

```python
features = ['const', 'is_amazon','stars_delta', ]
log_reg = sm.Logit(train['placed_higher'], train[features]).fit()
log_reg.summary()
```

    Optimization terminated successfully.
             Current function value: 0.494329
             Iterations 6

Logit Regression Results

  Dep. Variable:	     placed_higher	    No. Observations:  	    1132

  Model:	                 Logit	        Df Residuals:      	    1129

  Method:	                 MLE	         Df Model:          	       2

  Date:	            Mon, 07 Feb 2022	   Pseudo R-squ.:     	  0.2868

  Time:	                11:21:33	       Log-Likelihood:    	  -559.58

  converged:	             True	         LL-Null:           	  -784.58

  Covariance Type:	     nonrobust	      LLR p-value:       	 1.925e-98

       	          coef	     std err	      z	      P>|z|	  [0.025	    0.975]

  const	           0.0867	     0.073	     1.185	  0.236	    -0.057	     0.230

  is_amazon	       2.4786	     0.163	    15.202	  0.000	     2.159	     2.798

  stars_delta	    -0.2737	     0.241	    -1.138	  0.255	    -0.745	     0.198

Now notice the last two columns - that's the 95% confidence interval. If one value is positive, and the other negative, the interval includes 0, which means we can't really tell if the effect is positive, or negative - it's not significant. Turns out, controling for the star rating doesn't change anything.

Let's add the review delta:

```python
features = ['const', 'is_amazon','stars_delta', 'reviews_delta']
log_reg = sm.Logit(train['placed_higher'], train[features]).fit()
log_reg.summary()
```

    Optimization terminated successfully.
             Current function value: 0.493737
             Iterations 6

Logit Regression Results

  Dep. Variable:	     placed_higher	    No. Observations:  	    1132

  Model:	                 Logit	        Df Residuals:      	    1128

  Method:	                 MLE	         Df Model:          	       3

  Date:	            Mon, 07 Feb 2022	   Pseudo R-squ.:     	  0.2876

  Time:	                11:21:57	       Log-Likelihood:    	  -558.91

  converged:	             True	         LL-Null:           	  -784.58

  Covariance Type:	     nonrobust	      LLR p-value:       	 1.673e-97

        	           coef	     std err	      z	      P>|z|	  [0.025	    0.975]

  const	             0.0854	     0.073	     1.167	  0.243	    -0.058	     0.229

  is_amazon	         2.4875	     0.163	    15.216	  0.000	     2.167	     2.808

  stars_delta	      -0.2588	     0.241	    -1.075	  0.282	    -0.730	     0.213

  reviews_delta	 -1.353e-06	  1.17e-06	    -1.157	  0.247	 -3.64e-06	  9.39e-07

Same story, so let's include the rest of our variables.

```python
features = ['const', 'is_amazon','stars_delta', 'reviews_delta',
        'is_shipped_by_amazon', 'is_sold_by_amazon',
        'is_top_clicked']
log_reg = sm.Logit(train['placed_higher'], train[features]).fit()
log_reg.summary()
```

    Optimization terminated successfully.
             Current function value: 0.485827
             Iterations 6

Logit Regression Results

  Dep. Variable:	     placed_higher	    No. Observations:  	    1132

  Model:	                 Logit	        Df Residuals:      	    1125

  Method:	                 MLE	         Df Model:          	       6

  Date:	            Mon, 07 Feb 2022	   Pseudo R-squ.:     	  0.2990

  Time:	                11:22:17	       Log-Likelihood:    	  -549.96

  converged:	             True	         LL-Null:           	  -784.58

  Covariance Type:	     nonrobust	      LLR p-value:       	 3.532e-98

            	              coef	     std err	      z	      P>|z|	  [0.025	    0.975]

  const	                    0.1059	     0.074	     1.428	  0.153	    -0.039	     0.251

  is_amazon	                2.2388	     0.174	    12.889	  0.000	     1.898	     2.579

  stars_delta	             -0.3405	     0.246	    -1.384	  0.166	    -0.823	     0.142

  reviews_delta	        -1.501e-06	  1.17e-06	    -1.283	  0.200	 -3.79e-06	  7.93e-07

  is_shipped_by_amazon	    -0.2945	     0.240	    -1.225	  0.221	    -0.766	     0.177

  is_sold_by_amazon	        0.7100	     0.172	     4.124	  0.000	     0.373	     1.047

  is_top_clicked	           0.0046	     0.144	     0.032	  0.974	    -0.277	     0.286

The only other significant variable is `is_sold_by_amazon`. How do we interpret its value?

For exaple, both products are from amazon or are not from amazon, but one is sold by amazon, so `is_amazon = 0` and `is_sold_by_amazon = 1`

```python
numerator = np.e**(log_reg.params[0] + log_reg.params[1] * 0 + log_reg.params[5] * 1)
print(numerator/(1+numerator))
```

    0.6933580136415483

That's the probability the product sold by amazon will be first.

For another example, first product is from amazon and sold by amazon, the other is neither:

```python
numerator = np.e**(log_reg.params[0] + log_reg.params[1] * 1 + log_reg.params[5] * 1)
print(numerator/(1+numerator))
```

    0.9549840944621973

That's the probability the product sold by amazon will be first - it's slightly higher than when the first product is from amazon and the other isn't but we don't know who sells it.

What about the predictive performance? The Markup uses RandomForests with hyperparameter tuning to get the best prediction and ends up with 73.2% accuracy

```python
test = add_constant(test)
y_pred = log_reg.predict(test[features])
```

```python
((y_pred > 0.5) == test['placed_higher']).mean()
```

    0.696113074204947

```python
train.shape
```

    (1132, 8)

```python
train
```

      const
      stars_delta
      reviews_delta
      is_amazon
      is_shipped_by_amazon
      is_sold_by_amazon
      is_top_clicked
      placed_higher

      66
      1.0
      0.2
      11367.0
      1.0
      0.0
      1.0
      0.0
      True

      706
      1.0
      0.1
      -17591.0
      -1.0
      0.0
      0.0
      0.0
      False

      428
      1.0
      0.0
      1159.0
      0.0
      1.0
      0.0
      1.0
      False

      276
      1.0
      -0.3
      20532.0
      0.0
      0.0
      0.0
      -1.0
      False

      675
      1.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      False

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      716
      1.0
      -0.8
      -310.0
      0.0
      -1.0
      -1.0
      0.0
      True

      1135
      1.0
      -0.2
      -7298.0
      0.0
      0.0
      0.0
      -1.0
      True

      534
      1.0
      0.1
      3515.0
      0.0
      0.0
      0.0
      0.0
      True

      193
      1.0
      -0.1
      -484236.0
      -1.0
      0.0
      -1.0
      -1.0
      True

      1169
      1.0
      -0.2
      -2208.0
      0.0
      0.0
      0.0
      0.0
      False

1132 rows × 8 columns

```python
test
```

      const
      stars_delta
      reviews_delta
      is_amazon
      is_shipped_by_amazon
      is_sold_by_amazon
      is_top_clicked
      placed_higher

      283
      1.0
      0.1
      0.026419
      -1.0
      0.0
      0.0
      0.0
      False

      925
      1.0
      -0.1
      0.047067
      0.0
      0.0
      0.0
      0.0
      True

      394
      1.0
      -0.2
      -0.011249
      0.0
      0.0
      0.0
      0.0
      True

      442
      1.0
      0.0
      -0.048170
      0.0
      0.0
      0.0
      0.0
      True

      467
      1.0
      0.4
      0.236989
      0.0
      1.0
      0.0
      0.0
      True

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      561
      1.0
      0.1
      -0.610218
      0.0
      0.0
      0.0
      1.0
      False

      1335
      1.0
      -0.2
      0.152138
      -1.0
      0.0
      -1.0
      0.0
      False

      1154
      1.0
      0.1
      0.400056
      0.0
      0.0
      1.0
      0.0
      True

      85
      1.0
      0.1
      0.006553
      0.0
      0.0
      0.0
      0.0
      False

      103
      1.0
      -0.7
      -0.212269
      0.0
      0.0
      0.0
      0.0
      True

283 rows × 8 columns

That's not any worse than random forest - we can an equally good fit but we can actually interpret just how important the different features are!

The problem that we looked at so far is defined as follows: given the top two items, which one is going to be be placed in the first position?

Now, let's solve a slightly different problem formulation (not covered in the Markup write up): given characteristics of the product, how likely is it the top product on its search page?

```python
# let's get the data first:
dataset = pd.read_csv('https://www.sapiezynski.com/cs4910/markup/dataset_full.csv')
```

```python
dataset
```

      stars
      reviews
      is_amazon
      is_sold_by_amazon
      is_shipped_by_amazon
      top_clicked
      is_top

      0
      4.8
      2120.0
      0.0
      1.0
      1.0
      0.0
      1.0

      1
      4.7
      3747.0
      0.0
      1.0
      1.0
      1.0
      0.0

      2
      4.8
      6031.0
      0.0
      1.0
      1.0
      1.0
      0.0

      3
      4.8
      7017.0
      0.0
      1.0
      1.0
      1.0
      0.0

      4
      4.8
      455.0
      0.0
      1.0
      1.0
      1.0
      0.0

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      83898
      4.7
      248.0
      0.0
      0.0
      1.0
      0.0
      0.0

      83899
      4.8
      557.0
      0.0
      0.0
      1.0
      0.0
      0.0

      83900
      4.7
      214.0
      0.0
      0.0
      0.0
      1.0
      0.0

      83901
      4.8
      451.0
      0.0
      0.0
      1.0
      1.0
      0.0

      83902
      4.6
      347.0
      0.0
      0.0
      1.0
      0.0
      0.0

83903 rows × 7 columns

This dataset is an aggregation of all first search result pages from the Markup audit that had any amazon products on them. The sponsored results are removed.

In this dataset each row is a product with its star rating, the number of reviews, the binary indicators of whether it's an amazon product, it's sold or shipped by amazon, whether it was among the most clicked products, and the outcome variable - whether it was the top product in its search results page.

We're looking at at most 24 first results, so on average only one in 24 rows is the top placed product:

```python
dataset['is_top'].mean()
```

    0.04551684683503569

```python
# the lowest star rating is 1, not 0, let's adjust the rating such that it starts at 0:
dataset['stars'] -= 1
```

Excercise 1: Interpreting the meaning of logistic regression coefficients

In this exercise you will interpret the values of the coefficients of a logistic regression model that tries to predict whether a product was the top product on its search page.

In particular, please answer these questions:
1. What is the probability that a hypothetical product with the lowest rating, no reviews, not sold/shipped/produced by amazon, and not among the top clicked products will be placed at the top?
1. Increase in which variables corresponds to a higher probability of being placed at the top?
1. How do you interpret the `stars` coefficient and its statistical significance?
1. If a product receives one more review without changing any other of its characteristics, how does it affect its chances to placed at the top?
    * How about 1000000 more reviews?
1. What is the probability that a five-star product with 1000 reviews, produced, sold, and shipped by amazon that was among the top clicked results will be placed on the top of the result list?
    * Is this number higher, or lower than you expected? Does it change how you interpret the results of the model for only the top two positions? Why?

```python
dataset = add_constant(dataset)
cols =['const', 'stars', 'reviews', 'is_amazon', 'is_sold_by_amazon', 'is_shipped_by_amazon', 'top_clicked']
log_reg = sm.Logit(dataset['is_top'],
                   dataset[cols]).fit()
```

    Optimization terminated successfully.
             Current function value: 0.180478
             Iterations 7

```python
log_reg.summary()
```

Logit Regression Results

  Dep. Variable:	        is_top	        No. Observations:  	    83903

  Model:	                 Logit	        Df Residuals:      	    83896

  Method:	                 MLE	         Df Model:          	        6

  Date:	            Mon, 07 Feb 2022	   Pseudo R-squ.:     	   0.02496

  Time:	                12:22:50	       Log-Likelihood:    	   -15143.

  converged:	             True	         LL-Null:           	   -15530.

  Covariance Type:	     nonrobust	      LLR p-value:       	 3.631e-164

            	              coef	     std err	      z	      P>|z|	  [0.025	    0.975]

  const	                   -3.4286	     0.221	   -15.494	  0.000	    -3.862	    -2.995

  stars	                   -0.0472	     0.061	    -0.770	  0.441	    -0.167	     0.073

  reviews	               1.284e-06	  4.15e-07	     3.092	  0.002	   4.7e-07	   2.1e-06

  is_amazon	                1.1472	     0.047	    24.576	  0.000	     1.056	     1.239

  is_sold_by_amazon	       -0.1224	     0.039	    -3.171	  0.002	    -0.198	    -0.047

  is_shipped_by_amazon	     0.2063	     0.068	     3.040	  0.002	     0.073	     0.339

  top_clicked	              0.4318	     0.035	    12.174	  0.000	     0.362	     0.501

```python

```

```python

```

```python
# What is the probability that a hypothetical product with the lowest rating, no reviews,
# not sold/shipped/produced by amazon, and not among the top clicked products will be placed at the top?
np.e**(log_reg.params[0])/(1+ np.e**(log_reg.params[0]))
```

    0.03141344729908379

- Increase in which variables corresponds to a higher probability of being placed at the top?
>reviews, is_amazon, is_shipped_by_amazon, top_clicked (all ones that have positive coefs and are significant)

- How do you interpret the stars coefficient and its statistical significance?
> It is not statistically significant, so it appears not to have an effect on the placement.
> its value is negative, so if it was significant it would indicate that higher rated items are less likely to be on top

- If a product with no reviews receives its first one without changing any other of its characteristics, how does it affect its chances to placed at the top? How about when it receives 1000000 reviews instead?

```python
p0 = np.e**(log_reg.params[0])/(1+ np.e**(log_reg.params[0]))
p1 = np.e**(log_reg.params[0] + log_reg.params[2])/(1+ np.e**(log_reg.params[0]+log_reg.params[2]))
print(p1-p0)
```

    3.907903880356889e-08

```python
p2 = np.e**(log_reg.params[0] + log_reg.params[2] * 1000000)/(1+ np.e**(log_reg.params[0]+log_reg.params[2]*1000000))
```

```python

```

```python
print(p2-p0)
```

    0.07345786543985264

```python
p0
```

    0.03141344729908379

```python
p2
```

    0.10487131273893643

> The probability of being placed at the top grows by 7 percentage points from 3% to 10%

- What is the probability that a five-star product with 100000 reviews, produced, sold, and shipped by amazon that was among the top clicked results will be placed on the top of the result list?

```python
numerator = np.e**(log_reg.params[0]\
                  + log_reg.params[1]*4\
                  + log_reg.params[2]*100000\
                  + log_reg.params[3]\
                  + log_reg.params[4]\
                  + log_reg.params[5]\
                  + log_reg.params[6])
```

```python
numerator/(1+ numerator)
```

    0.13871729855646864

- Is this number higher, or lower than you expected? Does it change how you interpret the results of the model for only the top two positions? Why?

> Any answer beyond "it's what I expected, it doesn't change anything" goes.
>
> To me personally it's surprisingly low - given the results from the markup I would naively expect that being an amazon product nearly guarantees the first spot but of course it doesn't - there are likely multiple amazon products on the front page and only one will make it to the top, and it's also not always that one is at the top.

```python

```


Install py-pde library

```python
%pip install py-pde
```

    Requirement already satisfied: py-pde in /Users/sebastian/opt/anaconda3/envs/neuralOperator37/lib/python3.7/site-packages (0.17.1)
    Requirement already satisfied: numba>=0.50.0 in /Users/sebastian/opt/anaconda3/envs/neuralOperator37/lib/python3.7/site-packages (from py-pde) (0.55.1)
    Requirement already satisfied: numpy>=1.18.0 in /Users/sebastian/opt/anaconda3/envs/neuralOperator37/lib/python3.7/site-packages (from py-pde) (1.21.5)
    Requirement already satisfied: scipy>=1.4.0 in /Users/sebastian/opt/anaconda3/envs/neuralOperator37/lib/python3.7/site-packages (from py-pde) (1.7.3)
    Requirement already satisfied: sympy>=1.5.0 in /Users/sebastian/opt/anaconda3/envs/neuralOperator37/lib/python3.7/site-packages (from py-pde) (1.9)
    Requirement already satisfied: matplotlib>=3.1.0 in /Users/sebastian/opt/anaconda3/envs/neuralOperator37/lib/python3.7/site-packages (from py-pde) (3.5.1)
    Requirement already satisfied: python-dateutil>=2.7 in /Users/sebastian/opt/anaconda3/envs/neuralOperator37/lib/python3.7/site-packages (from matplotlib>=3.1.0->py-pde) (2.8.2)
    Requirement already satisfied: fonttools>=4.22.0 in /Users/sebastian/opt/anaconda3/envs/neuralOperator37/lib/python3.7/site-packages (from matplotlib>=3.1.0->py-pde) (4.29.1)
    Requirement already satisfied: packaging>=20.0 in /Users/sebastian/opt/anaconda3/envs/neuralOperator37/lib/python3.7/site-packages (from matplotlib>=3.1.0->py-pde) (21.3)
    Requirement already satisfied: kiwisolver>=1.0.1 in /Users/sebastian/opt/anaconda3/envs/neuralOperator37/lib/python3.7/site-packages (from matplotlib>=3.1.0->py-pde) (1.3.2)
    Requirement already satisfied: pyparsing>=2.2.1 in /Users/sebastian/opt/anaconda3/envs/neuralOperator37/lib/python3.7/site-packages (from matplotlib>=3.1.0->py-pde) (3.0.7)
    Requirement already satisfied: cycler>=0.10 in /Users/sebastian/opt/anaconda3/envs/neuralOperator37/lib/python3.7/site-packages (from matplotlib>=3.1.0->py-pde) (0.11.0)
    Requirement already satisfied: pillow>=6.2.0 in /Users/sebastian/opt/anaconda3/envs/neuralOperator37/lib/python3.7/site-packages (from matplotlib>=3.1.0->py-pde) (9.0.1)
    Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in /Users/sebastian/opt/anaconda3/envs/neuralOperator37/lib/python3.7/site-packages (from numba>=0.50.0->py-pde) (0.38.0)
    Requirement already satisfied: setuptools in /Users/sebastian/opt/anaconda3/envs/neuralOperator37/lib/python3.7/site-packages (from numba>=0.50.0->py-pde) (58.0.4)
    Requirement already satisfied: six>=1.5 in /Users/sebastian/opt/anaconda3/envs/neuralOperator37/lib/python3.7/site-packages (from python-dateutil>=2.7->matplotlib>=3.1.0->py-pde) (1.16.0)
    Requirement already satisfied: mpmath>=0.19 in /Users/sebastian/opt/anaconda3/envs/neuralOperator37/lib/python3.7/site-packages (from sympy>=1.5.0->py-pde) (1.2.1)
    Note: you may need to restart the kernel to use updated packages.

Generate dataset

## Basic Parameter Set

```python
import numpy as np
import torch

f = 4410      # sampling frequency
T = 1/f        # sampling time
dur = 0.1       # simulation duration

t = np.arange(0, dur+T/2, T) # 0:T:dur;    # time vector

l = 5          # length of the pipe
dx = 1e-2      # spatial stepsize
xs = np.arange(0, l+dx/2, dx) # 0:dx:l;    # space vector
numXs = np.size(xs)

#c0 = 340
c0 = 30       # propagation speed
```

## FTM Stuff

```python
Mu = 250       # number of eigenvalues
mu = np.arange(1, Mu+1) # 1:Mu;

test = 1j*c0*mu*np.pi/l

gmu = np.concatenate((mu*np.pi/l, mu*np.pi/l))
smu = np.concatenate((1j*c0*mu*np.pi/l, -1j*c0*mu*np.pi/l))

K1 = lambda x: 1j*np.sin(gmu*x) # @(x) 1j*sin(gmu*x);
K2 = lambda x: 1j*smu*np.sin(gmu*x)
Ka1 = lambda x: 1j/c0**2*np.conj(smu)*np.sin(gmu*x)
Ka2 = lambda x: 1j*np.sin(gmu*x)

nmu = 1./(l/2*(c0**2*smu + np.conj(smu)))

A = np.diag(np.exp(smu*T));

```

```python
xeVec = np.array([0.1*l, 0.2*l, 0.3*l]) # vector of excitaion positions (can be extended)

num_param_steps = 1323
field_values = np.linspace(0,10,num_param_steps)
grid_size = numXs

training_input = torch.zeros(num_param_steps, grid_size,2)
training_output = torch.zeros(num_param_steps, grid_size,1)

# grid = CartesianGrid([[0, 1]], grid_size, periodic=False)

index = 0

for xe, xeVal in enumerate(xeVec): #for xe = 1:length(xeVec)
    # Excitation for the wave equation is a simple delta-impulse at
    # position xe
    # Possible extensions:
    # - exciation by a hamming window to have a more smooth excitation
    # - combination with a temporal exciation shape
    yi = Ka2(xeVal)*T; # set initial values for states

    # vectors
    ybar = np.zeros((2*Mu, np.size(t)),dtype=complex);

    # set initial states
    ybar[:,0] = yi;

    test = range(1,np.size(t))

    # processing to create time progression of individual states
    for k in range(1,np.size(t)) :
        ybar[:,k] = A@ybar[:,k-1]

    # create output signal over time at a single observation position
    # (maybe this part is not necessary, therefore it is commented)
    xo = 0.7*l;
    c1 = K1(xo);
    y = c1@ybar; # recover deflection from states (inverse transformation)
    y = np.real(y)

    # create spatial vectors.
    # Result y_x: spatial distribution of the deflection y on the pipe at all
    # temportal sampling points

    K1_x = np.zeros((np.size(xs), 2*Mu));
    y_x = np.zeros((np.size(xs), np.size(t)));

    for xi in range(np.size(xs)) : #1:length(xs)
        K1_x[xi,:] = K1(xs[xi])/nmu;
        y_x[xi,:] = K1_x[xi,:]@ybar;

    # take the real part because there might be a small imaginary part
    y_x = np.real(y_x)
    y_x = y_x / 10**6 # scale the output to less than 1

    for k in range(1,np.size(t)) :
        #field = ScalarField(grid, val)
        #result = solve_poisson_equation(field, bc=[{"value": 0}, {"derivative": 1}])
        training_input[index,:,0] = torch.tensor(y_x[:,k-1])
        training_input[index,:,1] = torch.linspace(0,1, grid_size)
        training_output[index,:,0] = torch.tensor(y_x[:,k])
        index += 1
```

    /Users/sebastian/opt/anaconda3/envs/neuralOperator37/lib/python3.7/site-packages/ipykernel_launcher.py:51: ComplexWarning: Casting complex values to real discards the imaginary part
    /Users/sebastian/opt/anaconda3/envs/neuralOperator37/lib/python3.7/site-packages/ipykernel_launcher.py:52: ComplexWarning: Casting complex values to real discards the imaginary part

Model definitions copied from https://github.com/zongyi-li/fourier_neural_operator

```python
import torch.nn as nn
import torch.nn.functional as F

################################################################
#  1d fourier layer
################################################################
class SpectralConv1d(nn.Module):
    def __init__(self, in_channels, out_channels, modes1):
        super(SpectralConv1d, self).__init__()

        """
        1D Fourier layer. It does FFT, linear transform, and Inverse FFT.
        """

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.modes1 = modes1  #Number of Fourier modes to multiply, at most floor(N/2) + 1

        self.scale = (1 / (in_channels*out_channels))
        self.weights1 = nn.Parameter(self.scale * torch.rand(in_channels, out_channels, self.modes1, dtype=torch.cfloat))

    # Complex multiplication
    def compl_mul1d(self, input, weights):
        # (batch, in_channel, x ), (in_channel, out_channel, x) -> (batch, out_channel, x)
        return torch.einsum("bix,iox->box", input, weights)

    def forward(self, x):
        batchsize = x.shape[0]
        #Compute Fourier coeffcients up to factor of e^(- something constant)
        x_ft = torch.fft.rfft(x)

        # Multiply relevant Fourier modes
        out_ft = torch.zeros(batchsize, self.out_channels, x.size(-1)//2 + 1,  device=x.device, dtype=torch.cfloat)
        out_ft[:, :, :self.modes1] = self.compl_mul1d(x_ft[:, :, :self.modes1], self.weights1)

        #Return to physical space
        x = torch.fft.irfft(out_ft, n=x.size(-1))
        return x

class FNO1d(nn.Module):
    def __init__(self, modes, width):
        super(FNO1d, self).__init__()

        """
        The overall network. It contains 4 layers of the Fourier layer.
        1. Lift the input to the desire channel dimension by self.fc0 .
        2. 4 layers of the integral operators u' = (W + K)(u).
            W defined by self.w; K defined by self.conv .
        3. Project from the channel space to the output space by self.fc1 and self.fc2 .

        input: the solution of the initial condition and location (a(x), x)
        input shape: (batchsize, x=s, c=2)
        output: the solution of a later timestep
        output shape: (batchsize, x=s, c=1)
        """

        self.modes1 = modes
        self.width = width
        self.padding = 2 # pad the domain if input is non-periodic
        self.fc0 = nn.Linear(2, self.width) # input channel is 2: (a(x), x)

        self.conv0 = SpectralConv1d(self.width, self.width, self.modes1)
        self.conv1 = SpectralConv1d(self.width, self.width, self.modes1)
        self.conv2 = SpectralConv1d(self.width, self.width, self.modes1)
        self.conv3 = SpectralConv1d(self.width, self.width, self.modes1)
        self.w0 = nn.Conv1d(self.width, self.width, 1)
        self.w1 = nn.Conv1d(self.width, self.width, 1)
        self.w2 = nn.Conv1d(self.width, self.width, 1)
        self.w3 = nn.Conv1d(self.width, self.width, 1)

        self.fc1 = nn.Linear(self.width, 128)
        self.fc2 = nn.Linear(128, 1)

    def forward(self, x):
        x = self.fc0(x)
        x = x.permute(0, 2, 1)
        # x = F.pad(x, [0,self.padding]) # pad the domain if input is non-periodic

        x1 = self.conv0(x)
        x2 = self.w0(x)
        x = F.gelu(x1) + x2

        x1 = self.conv1(x)
        x2 = self.w1(x)
        x = F.gelu(x1) + x2

        x1 = self.conv2(x)
        x2 = self.w2(x)
        x = F.gelu(x1) + x2

        x1 = self.conv3(x)
        x2 = self.w3(x)
        x = F.gelu(x1) + x2

        # x = x[..., :-self.padding] # pad the domain if input is non-periodic
        x = x.permute(0, 2, 1)
        x = self.fc1(x)
        x = F.gelu(x)
        x = self.fc2(x)
        return x

```

```python
modes = 32 # 32
width = 16 # 16

epochs = 30
learning_rate = 1e-4
batch_size = 64

model = FNO1d(modes, width) #.to('cuda')

optimizer = torch.optim.AdamW(model.parameters(), lr=learning_rate, weight_decay=1e-4)
scheduler = torch.optim.lr_scheduler.OneCycleLR(optimizer, max_lr = 1e-3, epochs=epochs, steps_per_epoch= 32)

dataloader = torch.utils.data.DataLoader(torch.utils.data.TensorDataset(training_input, training_output), batch_size=batch_size, shuffle=True)

for ep in range(epochs):
  for input, output in dataloader:
    #input, output = input.cuda(), output.cuda()
    optimizer.zero_grad()
    pred_output = model(input)
    loss = torch.nn.functional.mse_loss(pred_output, output)
    loss.backward()
    optimizer.step()
    scheduler.step()
  print("\r",'loss:' + str(loss.detach().cpu().numpy()), end = "")

```

Check output

```python
import matplotlib.pyplot as plt

grid_start = 0
grid_end = 1
test_grid_size = 64
field_val = 2

model_input = torch.zeros(1,grid_size,2)
model_output = torch.zeros(1,grid_size,1)

testNum = 30
model_input[:,:,:] = training_input[testNum,:,:]
model_output[:,:,:] = training_output[testNum,:,:]
#model_input = model_input.to('cuda')
input_field =  model_input[0,:,0] # training_input[10,:,0]

model_result = model(model_input)

loss = torch.nn.functional.mse_loss(model_result, model_output)
print("\r",'loss:' + str(loss.detach().cpu().numpy()), end = "")

plt.figure()
plt.plot(input_field.data)
plt.plot(model_output[0,:,0].data)
plt.plot(model_result.detach().cpu().flatten().numpy())

```

```python

```

```python

```

```python

```


```julia
using NPZ
using PyPlot
using SpecialFunctions
```

    Unable to init server: Could not connect: Connection refused
    Unable to init server: Could not connect: Connection refused

    (.:14921): Gdk-CRITICAL **: 18:21:09.991: gdk_cursor_new_for_display: assertion 'GDK_IS_DISPLAY (display)' failed

    (.:14921): Gdk-CRITICAL **: 18:21:10.006: gdk_cursor_new_for_display: assertion 'GDK_IS_DISPLAY (display)' failed

# Table of Contents
1. [Loading crystal data](#loading_crystal_data)
2. [Sorting $i,j,k$](#sorting_ijk)
3. [Manybody potential at single $z$-slice](#manybody_potential_at_single_z-slice)
4. [3D manybody potential](#3d_manybody_potential)

## Loading crystal data
[Table of Contents](#table_of_contents)

First, we load the data generated from [this notebook](effective_parameters_to_cylindrical_potential_MCM-41_PYTHON.ipynb).

```julia
#xyz atom positions withn the unit cell
mcm41_xyz_shifted = NPZ.npzread("data/mcm41_xyz_shifted_CVFF.npz");
mcm41_x = mcm41_xyz_shifted["x"];
mcm41_y = mcm41_xyz_shifted["y"];
mcm41_z = mcm41_xyz_shifted["z"];

#mixed sigma and epsilon parameters for the Lennard-Jones potential (helium and MCM-41 atoms)
sigma = mcm41_xyz_shifted["sigma"];
epsilon = mcm41_xyz_shifted["epsilon"];
```

```julia
#lattice vectors for MCM-41 unit cell
mcm41_lattice_vectors = NPZ.npzread("data/mcm41_lattice_vectors.npz");
A1 = mcm41_lattice_vectors["A1"];
A2 = mcm41_lattice_vectors["A2"];
A3 = mcm41_lattice_vectors["A3"];
```

Recall, we wish to generate the manybody potential within a single pore by summing a 12-6 Lennard-Jones potential over the atoms within a semi-infinite MCM-41 supercell
\begin{equation}
    U_\mathrm{MCM-41}^\mathrm{He}(\vec{r}) = \sum_{i,j,k = -\infty}^\infty \sum_\alpha^N 4\varepsilon_\alpha \biggl( \frac{\sigma_\alpha^{12}}{\lvert \vec{r} - \vec{r}_{ijk\alpha}\rvert^{12}} - \frac{\sigma_\alpha^6}{\lvert \vec{r} - \vec{r}_{ijk\alpha}\rvert^6} \biggr)
\end{equation}
where $\alpha$ indexes over each atom within the unit cell, $\varepsilon_\alpha$ and $\sigma_\alpha$ are the appropriately mixed Lennard-Jones parameters, and
\begin{equation}
\vec{r}_{ijk\alpha} = \vec{r}_\alpha + i\vec{A}_a + j\vec{A}_b + k\vec{A}_c
\end{equation}
is the position of each atom within the semi-infinite supercell with unit cell vectors $\vec{A}_{\{a,b,c\}}$.

The procedure will be to expand the unit cell in each unit cell direction and perform the summation until we reach the single precision floating point limit. Here we show a birds-eye view example of expanding once in each lattice vector direction.

```julia
fig,ax = subplots(figsize=(10,10))
k = 0
for i in -1:1:1
    for j in -1:1:1
        _A = ((i .* A1) .+ (j .* A2) .+ (k .* A3))
        ax.scatter(mcm41_x .+ _A[1], mcm41_y .+ _A[2])
    end
end
ax.set_xlabel(L"$x\ \mathrm{[\AA]}$")
ax.set_ylabel(L"$y\ \mathrm{[\AA]}$")

arrowprops = PyPlot.PyDict()
arrowprops["arrowstyle"]="->"
arrowprops["shrinkA"]=0.0
arrowprops["shrinkB"]=0.0
ax.annotate("", xy=(A1[1], A1[2]), xytext=(0, 0), arrowprops=arrowprops)
ax.annotate(L"$\vec{A}_a$", xy=(A1[1], A1[2]))
ax.annotate("", xy=(A2[1], A2[2]), xytext=(0, 0), arrowprops=arrowprops)
ax.annotate(L"$\vec{A}_b$", xy=(A2[1], A2[2]))
ax.set_aspect("equal")
```

## Sorting $i,j,k$
[Table of Contents](#table_of_contents)

We created a sorted array over the summation integers $i,j,k$ since the atoms closest to the central pore will contribute the most the the manybody potential. The array is sorted by $\lvert\vec{r}_{ijk}\rvert=\lvert i\vec{A}_a + j\vec{A}_b + k\vec{A}_c \rvert$.

```julia
# we create a large collection of ijk indices
i_arr = collect(-200:1:200);
j_arr = collect(-200:1:200);
k_arr = collect(-200:1:200);
r_arr  = Array{Float64,3}(undef,size(i_arr,1),size(j_arr,1),size(k_arr,1));
```

```julia
# Here we find the magnitude to center of each replicated unit cell within the very large supercell
for i in i_arr
    for j in j_arr
        for k in k_arr
            _A = ((i .* A1) .+ (j .* A2) .+ (k .* A3));
            _r = sqrt(sum(_A.^2))
            r_arr[i+201,j+201,k+201] = _r
        end
    end
end
```

```julia
#initialize and fill the arrays to hold the combinations of ijk
_i_arr = Array{Int64,3}(undef,size(i_arr,1),size(j_arr,1),size(k_arr,1));
_j_arr = Array{Int64,3}(undef,size(i_arr,1),size(j_arr,1),size(k_arr,1));
_k_arr = Array{Int64,3}(undef,size(i_arr,1),size(j_arr,1),size(k_arr,1));

for i in i_arr
    for j in j_arr
        for k in k_arr
            _i_arr[i+201,j+201,k+201] = i
            _j_arr[i+201,j+201,k+201] = j
            _k_arr[i+201,j+201,k+201] = k
        end
    end
end
```

```julia
#flatten each ijk array and sort by the distance to each unit cell within the supercell
r_arr_flat = vec(r_arr);
_i_arr_flat = vec(_i_arr);
_j_arr_flat = vec(_j_arr);
_k_arr_flat = vec(_k_arr);

p = sortperm(r_arr_flat); #get the index for sorting by distance

r_arr_flat = r_arr_flat[p];
_i_arr_flat = _i_arr_flat[p];
_j_arr_flat = _j_arr_flat[p];
_k_arr_flat = _k_arr_flat[p];
```

## Manybody potential at single $z$-slice
[Table of Contents](#table_of_contents)

Now that we have sorted the summation indeces for a suitably large supercell, we can calculate the manybody potential at a single $z$-slice within a nanopore in the MCM-41. First we define the Lennard-Jones potential functions.

```julia
function U(r::Float64,sigma::Float64,epsilon::Float64)
    pf = 4 * epsilon;
    t12 = (sigma/r)^12;
    t6 = (sigma/r)^6;
    return pf * (t12 - t6)
end

function U(r::Float32,sigma::Float32,epsilon::Float32)
    pf = 4 * epsilon;
    t12 = (sigma/r)^12;
    t6 = (sigma/r)^6;
    return pf * (t12 - t6)
end

function U(r::Float64,sigma::Float64,epsilon::Float64)
    pf = 4 * epsilon;
    t12 = (sigma/r)^12;
    t6 = (sigma/r)^6;
    return pf * (t12 - t6)
end

function U_mb(r::Array{Float64,1},sigma::Array{Float64,1},epsilon::Array{Float64,1})
    V = 0.0;
    @inbounds for i in 1:size(r)
        V += U(r[i],sigma[i],epsilon[i]);
    end
    return V
end
```

    U_mb (generic function with 1 method)

Next, we generate a mesh of points in the xy-plane ecompassing the pore.

```julia
x_min = -20.0;
x_max = 20.0;
y_min = -20.0;
y_max = 20.0;
res = 101;
x = zeros(res,res);
y = zeros(res,res);
x_range = collect(range(x_min,x_max,length=res));
y_range = collect(range(y_min,y_max,length=res));
for i in 1:res
    _x = x_range[i];
    for j in 1:res
        _y = y_range[j];
        x[i,j] = _x;
        y[i,j] = _y;
    end
end
```

Now we convert some the variables to Julia Float32 types since we are calculating the manybody potential to within the single precision floating point limit.

```julia
r_tmp = zeros(Float32,size(x));

x32 = convert.(Float32,x);
y32 = convert.(Float32,y);

A1_32 = convert.(Float32,A1);
A2_32 = convert.(Float32,A2);
A3_32 = convert.(Float32,A3);

# We will look at the z-slice for z=0.0 Angstroms
Z_VALUE = 0.0;
Z_VALUE32 = convert(Float32,Z_VALUE);

V = zeros(Float32,size(x32));
V_old = zeros(Float32,size(x32));
```

Finally, everything is set up to generate the manybody potential at a single $z$-slice.

```julia
ii_max = 10000 #Set some maximum just in case
for ii = 1:ii_max
    i = _i_arr_flat[ii];
    j = _j_arr_flat[ii];
    k = _k_arr_flat[ii];
    for atom in 1:size(mcm41_x,1)
        # get distance to single atom within supercell
        r_tmp .= sqrt.(((mcm41_x[atom] + (A1_32[1]*i) + (A2_32[1]*j)) .- x32).^2 .+ ((mcm41_y[atom] + (A2_32[2]*j)) .- y32).^2 .+ ((mcm41_z[atom] + (A3_32[3]*k)) - Z_VALUE32)^2);

        #Calculate the Lennard-Jones potential at the separation for helium and MCM-41 atom and increment the total potential
        V .+= U.(r_tmp,sigma[atom],epsilon[atom])
    end
    if all(V .== V_old)
        # if the old potential is the same as the new potential for
        # every point within the xy-mesh, then we have reached the
        # single precision floating point limit
        println(ii) # Show progress for each slice iteration
        break
    elseif (ii == ii_max)
        println("WARNING: ii_max reached")
        break
    else
        # if we haven't reached the stopping critereon, copy current potential
        copy!(V_old,V);
    end
end

```

    574

We have calculated the manybody potential for a single $z$-slice. Here is a view of the potential.

```julia
extent = (-20.0,20.0,-20.0,20.0)
fig,ax = plt.subplots()
im = ax.imshow(transpose(V),origin="lower",vmax=200,vmin=-200,extent=extent)
ax.set_xlabel(L"$x\ \mathrm{[\AA]}$")
ax.set_ylabel(L"$y\ \mathrm{[\AA]}$")
fig.colorbar(im)
```

## 3D manybody potential
[Table of Contents](#table_of_contents)

Now that we have seen how to calculate the manybody potential for a single slice, we are ready to caculate the full 3D manybody potential. Here, we will calculate from the bottom of the single unitcell to the top.

```julia
z_min = -A3[3]/2; # bottom of unit cell
z_max = A3[3]/2; # top of unit cell
z_range = collect(range(z_min,z_max,length=res));

V_3D = zeros(Float32,size(x32,1),size(x32,2),res); # initialize 3D potential
```

```julia
for jj in 1:res
    print("jj = $jj: ") # A more descriptive output for each slice
    Z_VALUE = z_range[jj];
    Z_VALUE32 = convert(Float32,Z_VALUE);
    V = zeros(Float32,size(x32));
    V_old = zeros(Float32,size(x32));
    ii_max = 10000
    for ii = 1:ii_max
        i = _i_arr_flat[ii];
        j = _j_arr_flat[ii];
        k = _k_arr_flat[ii];
        for atom in 1:size(mcm41_x,1)
            r_tmp .= sqrt.(((mcm41_x[atom] + (A1_32[1]*i) + (A2_32[1]*j)) .- x32).^2 .+ ((mcm41_y[atom] + (A2_32[2]*j)) .- y32).^2 .+ ((mcm41_z[atom] + (A3_32[3]*k)) - Z_VALUE32)^2);
            V .+= U.(r_tmp,sigma[atom],epsilon[atom])
        end
        if all(V .== V_old)
            println(ii)
            break
        elseif (ii == ii_max)
            break
        else
            copy!(V_old,V);
        end
    end
    V_3D[:,:,jj] .= V;
end
```

    jj = 1: 1009
    jj = 2: 383
    jj = 3: 1402
    jj = 4: 575
    jj = 5: 1010
    jj = 6: 383
    jj = 7: 557
    jj = 8: 575
    jj = 9: 1471
    jj = 10: 2078
    jj = 11: 385
    jj = 12: 587
    jj = 13: 313
    jj = 14: 1083
    jj = 15: 383
    jj = 16: 313
    jj = 17: 575
    jj = 18: 704
    jj = 19: 575
    jj = 20: 1404
    jj = 21: 575
    jj = 22: 441
    jj = 23: 739
    jj = 24: 385
    jj = 25: 575
    jj = 26: 1180
    jj = 27: 2117
    jj = 28: 739
    jj = 29: 1143
    jj = 30: 312
    jj = 31: 431
    jj = 32: 440
    jj = 33: 1143
    jj = 34: 1180
    jj = 35: 312
    jj = 36: 2877
    jj = 37: 1143
    jj = 38: 575
    jj = 39: 794
    jj = 40: 849
    jj = 41: 709
    jj = 42: 312
    jj = 43: 848
    jj = 44: 3022
    jj = 45: 689
    jj = 46: 501
    jj = 47: 440
    jj = 48: 849
    jj = 49: 836
    jj = 50: 440
    jj = 51: 574
    jj = 52: 440
    jj = 53: 849
    jj = 54: 1530
    jj = 55: 574
    jj = 56: 789
    jj = 57: 440
    jj = 58: 440
    jj = 59: 1394
    jj = 60: 574
    jj = 61: 1595
    jj = 62: 1180
    jj = 63: 312
    jj = 64: 551
    jj = 65: 2103
    jj = 66: 1494
    jj = 67: 574
    jj = 68: 574
    jj = 69: 428
    jj = 70: 312
    jj = 71: 312
    jj = 72: 2116
    jj = 73: 574
    jj = 74: 1136
    jj = 75: 574
    jj = 76: 554
    jj = 77: 428
    jj = 78: 312
    jj = 79: 5830
    jj = 80: 312
    jj = 81: 550
    jj = 82: 738
    jj = 83: 426
    jj = 84: 1470
    jj = 85: 312
    jj = 86: 2116
    jj = 87: 738
    jj = 88: 427
    jj = 89: 738
    jj = 90: 574
    jj = 91: 378
    jj = 92: 440
    jj = 93: 514
    jj = 94: 555
    jj = 95: 550
    jj = 96: 374
    jj = 97: 374
    jj = 98: 468
    jj = 99: 252
    jj = 100: 519
    jj = 101: 1005

Now that we have the 3D manybody potential, we can save it and continue our analysis in the [effective parameters to cylindrical potential notebook](effective_parameters_to_cylindrical_potential_MCM-41_PYTHON.ipynb#determining_the_effective_parameters_to_cylindrical_potential).

```julia
NPZ.npzwrite("data/V_3D_-20.0_20.0_-20.0_20.0_-6.10_6.10_101_CVFF.npz",
    Dict("x_range"  => convert.(Float32,x_range),
         "y_range"  => convert.(Float32,y_range),
         "z_range"  => convert.(Float32,z_range),
         "V" => V_3D,
         "x_grid" => x32,
         "y_grid" => y32,
         "extent" => convert.(Float32,[x_min,x_max,y_min,y_max])
        ))
```


## 5-3. Quantum Approximate Optimazation Algorithm (QAOA): 量子近似最適化アルゴリズム

### 概要

この節では、NISQアルゴリズムの一つと考えられる Quantum Approximate Optimazation Algorithm (QAOA; 量子近似最適化アルゴリズム)を学ぶ。QAOAは、量子アニーリングと同様に、組み合わせ最適化問題の解を求めるためのアルゴリズムである。

### 問題設定
QAOAでは、$z = z_{1}z_{2}\cdots z_{n} \: (z_i =0,1)$ という$n$桁のビット列$z$に関して、コスト関数$C(z) = \sum_\alpha C_\alpha(z)$が最小になるような$z$を探す問題を考える。$C_\alpha(z)$はビット列$z$を引数にとる何らかの関数で、ここでは特に、イジングモデル的な$C_\alpha(z) = z_i\cdot z_j$といった項を考えれば良い。

この最小化問題を解くために、$n$**ビットの量子系を用いる**。そして、$\beta = (\beta^{(1)}, \cdots \beta^{(p)}), \gamma = (\gamma^{(1)}, \cdots \gamma^{(p)})$ をパラメータとして、次のような量子状態を考える。

\begin{align}
&|s\rangle = |+\rangle^{\otimes n} = \frac{1}{2^{n/2}} \sum_{z=0}^{2^{n}-1} |z\rangle, \\
&|\beta, \gamma \rangle = U_X(\beta^{(p)}) U_C(\gamma^{(p)}) \cdots U_X(\beta^{(1)}) U_C(\gamma^{(1)}) |s\rangle.
\end{align}

ここで $|+\rangle=\frac{1}{\sqrt{2}}(|0\rangle+|1\rangle)$は$X$演算子の固有状態$X|+\rangle=|+\rangle$であり、 $U_C(\gamma), U_X(\beta)$ は次のように定義される。

$$
U_C(\gamma^{(i)}) = e^{-i\gamma^{(i)} C(Z)} = \prod_{\alpha} e^{-i\gamma^{(i)} C_{\alpha}(Z)}, \\
U_X(\beta^{(i)}) = e^{-i\beta^{(i)} \sum_{j=1}^n X_j} = \prod_{j =1}^n e^{-i\beta^{(i)} X_j}.
$$

状態$|\beta, \gamma \rangle$やこれらの演算子の意味を説明するには量子アニーリングの知識が必要になる。とりあえず、QAOAを使うだけならこういうものだと受け入れて使ってしまえば良い。
（なお、$C(Z)$というのはビット列を引数にとる関数$C(z)$の入力にパウリ演算子$Z_1\cdots Z_n$と代入したものであることに注意。）

そして、$F(\beta, \gamma) = \langle{\bf \gamma, \,\beta}|C(Z)|{\bf \gamma, \,\beta}\rangle$ を最小にするような$\beta,\gamma$を探索することで、元々の最適化問題の答えを探そうとするのが、QAOAアルゴリズムである。

### QAOAアルゴリズムの手順
具体的なQAOAアルゴリズムの手順は以下の通りである。

1. 量子コンピュータ上で重ね合わせ状態$|s\rangle = |+\rangle^{\otimes n}$を作る。
2. パラメータ$\beta, \gamma$に応じて、量子状態に$U_C(\gamma^{(i)}),U_X(\beta^{(i)})$をかけていき、状態$|\beta, \gamma \rangle$を得る。
3. 量子コンピュータを用いて $\langle \beta, \gamma |C(Z)|\beta, \gamma \rangle$ を測定する。
4. 古典コンピュータで、$\langle \beta, \gamma |C(Z)|\beta, \gamma \rangle$ がより小さくなるようにパラメータ $\beta, \gamma$ をアップデートする。
5. 1〜4を繰り返し、最適な $\beta^*, \gamma^*$ を得る。
6. 状態 $|\beta^*, \gamma^* \rangle$ に対して、$z$方向の射影測定を複数回実行し、**得られた（良さそうな）測定結果** $z_1\cdots z_n$ **を元々の最適化問題の解として採用**する。（注：測定結果 $z_1\cdots z_n$ は古典ビット）

少々ややこしいので、具体例を実装しながら確認していこう。

### 実装：Maxcut問題をQAOAで解く

ここでは具体例としてMaxcut問題という問題を取り上げる。
[Maxcut問題](https://ja.wikipedia.org/wiki/カット_(グラフ理論))は、$n$個の頂点を持つグラフ（例えば下図）を２つに分割する時に、分割される辺の数の最大値を求める問題である。

(図の出典：Wikipedia [カット_(グラフ理論)](https://ja.wikipedia.org/wiki/カット_(グラフ理論)))

この問題をQAOAで扱えるような最適化問題に帰着させるには、以下のようにする。
頂点を２つのグループに分けた時、片方のグループに属する頂点に+1、もう一方のグループに-1を付与するとすれば、コスト関数

$$
C(z) = -\frac{1}{2} \sum_{\text{辺で繋がっている頂点}i,j} ( 1 - z_i z_j)
$$

は (グループ分けによって分割される辺の数) $\times (-1)$ を表す。
ゆえに、$C(z)$を最小化するようなビット列$z=z_1\cdots z_n$を見つければ、分割する辺の数を最大化するような頂点の分け方を見つけたことになる。

以下では、長方形(頂点が４つの図形)のmaxcut問題を解いてみよう。

この場合、$C(Z)$は

$$
\begin{eqnarray}
C(Z) &=& -\frac{1}{2}(1-Z_{0}Z_{1})-\frac{1}{2}(1-Z_{1}Z_{2})-\frac{1}{2}(1-Z_{2}Z_{3})-\frac{1}{2}(1-Z_{3}Z_{1})\\
&=&\frac{1}{2}(Z_{0}Z_{1}+Z_{1}Z_{2}+Z_{2}Z_{3}+Z_{3}Z_{1}) - 2
\end{eqnarray}
$$

となる。第二項は定数だから、以下では

$$
C(Z) = \frac{1}{2}(Z_{0}Z_{1}+Z_{1}Z_{2}+Z_{2}Z_{3}+Z_{3}Z_{1})
$$

とおく。

#### $p=1$の場合
まずは$p=1$の場合の実装をやってみよう。この時、$|\beta, \gamma \rangle = U_X(\beta^{(1)}) U_C(\gamma^{(1)}) |s\rangle$ である。

$U_C(\gamma^{(1)}) = \prod_{i=0}^3 e^{-i\gamma^{(1)} Z_i Z_{i+1} }$ を実装するには、4-2節でも用いた関係

$$
e^{-i \delta Z_i Z_{i+1}}　= \operatorname{CNOT}_{i,i+1} \cdot e^{-i\delta Z_{i+1}} \cdot \operatorname{CNOT}_{i,i+1}.
$$

を使えば良い。（行列に直して計算すると、合っていることがわかる。）

以上を踏まえて、$|\beta, \gamma \rangle$ を構成して $\langle \beta, \gamma | C(Z) |\beta, \gamma \rangle$ を測定し、それを最小化する工程を実装するとこのようになる。

```python
## Google Colaboratoryの場合・Qulacsがインストールされていないlocal環境の場合のみ実行してください
!pip install qulacs
```

```python
#必要なライブラリをインポートする
from qulacs import QuantumState, QuantumCircuit, Observable, PauliOperator
from qulacs.gate import H, CNOT, RX, RZ
from scipy.optimize import minimize
import numpy as np

## 頂点の数
n = 4

## C(Z)をqulacs.Observableとして定義
cost_observable = Observable(n)
for i in range(n):
    cost_observable.add_operator( PauliOperator("Z {:} Z {:}".format(i, (i+1)%n), 0.5) )

# circuit に U_C(gamma) を加える関数
def add_U_C(circuit, gamma):
    for i in range(n):
        j = (i+1) % n
        circuit.add_CNOT_gate(i, j)
        circuit.add_gate(RZ(j, -2*gamma)) ## qulacsでは RZ(theta)=e^{i*theta/2*Z}
        circuit.add_CNOT_gate(i, j)
    return circuit

# circuit に U_X(beta) を加える関数
def add_U_X(circuit, beta):
    for i in range(n):
        circuit.add_gate(RX(i, -2*beta))
    return circuit

# p=1 の |beta, gamma> を作って <beta, gamma| C(Z) |beta, gamma> を返す関数
# x = [beta, gamma]
def QAOA_output_onelayer(x):
    beta, gamma = x

    circuit = QuantumCircuit(n)
    ## 重ね合わせを作るため、アダマールゲートをかける
    for i in range(n):
        circuit.add_H_gate(i)
    ## U_C, U_Xをかける
    circuit =  add_U_C(circuit, gamma)
    circuit =  add_U_X(circuit, beta)

    ## |beta, gamma>を作る
    state = QuantumState(n)
    state.set_zero_state()
    circuit.update_quantum_state(state)
    return cost_observable.get_expectation_value(state)

## 初期値
x0 = np.array( [0.1, 0.1 ])

## scipy.minimize を用いて最小化
result = minimize(QAOA_output_onelayer, x0, options={'maxiter':500}, method='powell')
print(result.fun) # 最適化後の値
print(result.x) # 最適化後の(beta, gamma)
```

    -0.999999999499185
    [1.17809152 0.39269362]

最小値-1と、その時の$\beta^{(1)}, \gamma^{(1)}$の値が得られた。
この最適な状態 $|\beta, \gamma\rangle$ を$z$方向に射影測定した時にどのような値が得られるか、具体的に見てみよう。

```python
# 最適なbeta, gammaを使って |beta, gamma> をつくる
beta_opt, gamma_opt = result.x

circuit = QuantumCircuit(n)
## 重ね合わせを作るため、アダマールゲートをかける
for i in range(n):
    circuit.add_H_gate(i)
## U_C, U_Xをかける
circuit =  add_U_C(circuit, gamma_opt)
circuit =  add_U_X(circuit, beta_opt)

## |beta, gamma>を作る
state = QuantumState(n)
state.set_zero_state()
circuit.update_quantum_state(state)

## z方向に観測した時の確率分布を求める. (状態ベクトルの各成分の絶対値の二乗=観測確率)
probs = np.abs(state.get_vector())**2
print(probs)
```

    [0.01562503 0.01562568 0.01562568 0.0781236  0.01562568 0.26562503
     0.0781236  0.01562568 0.01562568 0.0781236  0.26562503 0.01562568
     0.0781236  0.01562568 0.01562568 0.01562503]

```python
# プロットする
import matplotlib.pyplot as plt
%matplotlib inline

## z方向に射影測定した時に得られる可能性があるビット列
z_basis = [format(i,"b").zfill(n) for i in range(probs.size)]

plt.figure(figsize=(10, 5))
plt.xlabel("states")
plt.ylabel("probability(%)")
plt.bar(z_basis, probs*100)
plt.show()
```

つまり、$z$方向の射影測定を行うと、`0101` か `1010` が測定される確率が高いことが分かった。これらのビット列は頂点1と頂点3、頂点2と頂点4が同じグループになるということを意味するから、以下のような分割を表している。

この時、図形を分割する曲線が横切る辺の数は4本であり、この図形を分割する時に通る辺の数の最大値である。

よって、最適化した$|\beta, \gamma\rangle$に射影測定を行い、ある程度多数の測定結果を行って測定確率が高いビット列を採用すれば、もともと解きたかった最適化問題$C(z)$の解が得られた、ということになる。
一応、これでめでたしめでたし、と言えるのだが、最適化したコスト関数$\langle \beta, \gamma | C(Z) |\beta, \gamma \rangle$ の値は−1だったことを思い出してほしい。$\langle 0101 | C(Z) |0101 \rangle = \langle 1010 | C(Z) |1010 \rangle = -2$なので、コスト関数については正しい値が得られていない！ これはvariationalな状態$|\beta, \gamma \rangle$ が十分な表現能力を持たず、真の解$|0101\rangle, |1010\rangle$ を表現できなかったことに由来する。

そこで、回路をより複雑にした$p=2$の場合に結果がどう変わるか見てみよう。

※ ちなみに、$|\beta, \gamma\rangle$に100回の測定を行ってビット列$z$を100通り得て、それぞれについて$C(z)$を古典コンピュータで計算してみて最も良かったものを採用する、といった戦略を用いれば、このような問題は生じないかもしれない。

#### $p=2$の場合

```python
#必要なライブラリをインポートする
from qulacs import QuantumState, QuantumCircuit, Observable, PauliOperator
from qulacs.gate import H, CNOT, RX, RZ
from scipy.optimize import minimize
import numpy as np

## 頂点の数
n = 4

## C(Z)をqulacs.Observableとして定義
cost_observable = Observable(n)
for i in range(n):
    cost_observable.add_operator( PauliOperator("Z {:} Z {:}".format(i, (i+1)%n), 0.5) )

# circuit に U_C(gamma) を加える関数
def add_U_C(circuit, gamma):
    for i in range(n):
        j = (i+1) % n
        circuit.add_CNOT_gate(i, j)
        circuit.add_gate(RZ(j, -2*gamma)) ## qulacsでは RZ(theta)=e^{i*theta/2*Z}
        circuit.add_CNOT_gate(i, j)
    return circuit

# circuit に U_X(beta) を加える関数
def add_U_X(circuit, beta):
    for i in range(n):
        circuit.add_gate(RX(i, -2*beta))
    return circuit

# p=2 の |beta, gamma> を作って <beta, gamma| C(Z) |beta, gamma> を返す関数
# x = [beta0, beta1, gamma0, gamma1]
def QAOA_output_twolayer(x):
    beta0, beta1, gamma0, gamma1 = x

    circuit = QuantumCircuit(n)
    ## 重ね合わせを作るため、アダマールゲートをかける
    for i in range(n):
        circuit.add_H_gate(i)
    ## U_C, U_Xをかける
    circuit =  add_U_C(circuit, gamma0)
    circuit =  add_U_X(circuit, beta0)
    circuit =  add_U_C(circuit, gamma1)
    circuit =  add_U_X(circuit, beta1)

    ## |beta, gamma>を作る
    state = QuantumState(n)
    state.set_zero_state()
    circuit.update_quantum_state(state)
    return cost_observable.get_expectation_value(state)

## 初期値
x0 = np.array( [0.1, 0.1, 0.2, 0.3 ])

## scipy.minimize を用いて最小化
result = minimize(QAOA_output_twolayer, x0, options={'maxiter':500}, method='powell')
print(result.fun) # 最適化後の値
print(result.x) # 最適化後の[beta0, beta1, gamma0, gamma1]

## 最適化後の状態を測定した時の確率分布を調べる
beta0, beta1, gamma0, gamma1 = result.x

circuit = QuantumCircuit(n)
## 重ね合わせを作るため、アダマールゲートをかける
for i in range(n):
    circuit.add_H_gate(i)
## U_C, U_Xをかける
circuit =  add_U_C(circuit, gamma0)
circuit =  add_U_X(circuit, beta0)
circuit =  add_U_C(circuit, gamma1)
circuit =  add_U_X(circuit, beta1)

## |beta, gamma>を作る
state = QuantumState(n)
state.set_zero_state()
circuit.update_quantum_state(state)

## 状態ベクトルの各成分の絶対値の二乗=観測確率
probs = np.abs(state.get_vector())**2
print(probs)

## z方向に射影測定した時に得られる可能性があるビット列
z_basis = [format(i,"b").zfill(n) for i in range(probs.size)]

import matplotlib.pyplot as plt
plt.figure(figsize=(10, 5))
plt.xlabel("states")
plt.ylabel("probability(%)")
plt.bar(z_basis, probs*100)
plt.show()
```

$p=1$の時に比べ、圧倒的に大きい確率で真の解$|0101\rangle, |1010\rangle$が得られる確率が高いことが分かる。また、コスト関数の値も正しく−2に近づいている。

このように、QAOAを用いる際には、変分量子回路の複雑さ$p$の大きさにも注意しながら実装する必要があるだろう。

### 参考文献

[1] E. Farhi, J. Goldstone, and S. Gutmann, “A Quantum Approximate Optimization Algorithm”, [arXiv:1411.4028](https://arxiv.org/abs/1411.4028) (2014).

[2] Eddie Farhi: A Quantum Approximate Optimization Algorithm, https://www.youtube.com/watch?v=J8y0VhnISi8


# Episodic Lunar Lander with function appoximation and control

This Notebook is intended to solve the Episodic Lunar Lander problem using Semi-gradient Expected sarsa with neural networks for function approximation.

The description of the problem is given below:

"Landing pad is always at coordinates (0,0). Coordinates are the first two numbers in state vector. Reward for moving from the top of the screen to landing pad and zero speed is about 100..140 points. If lander moves away from landing pad it loses reward back. Episode finishes if the lander crashes or comes to rest, receiving additional -100 or +100 points. Each leg ground contact is +10. Firing main engine is -0.3 points each frame. Solved is 200 points. Landing outside landing pad is possible. Fuel is infinite, so an agent can learn to fly and then land on its first attempt. Four discrete actions available: do nothing, fire left orientation engine, fire main engine, fire right orientation engine."

Image and Text taken from [Official documentaiton Lunar Lander](https://gym.openai.com/envs/LunarLander-v2/).

```python
import numpy as np
import matplotlib.pyplot as plt
from tqdm import tqdm
from copy import deepcopy

import gym
from gym.wrappers import Monitor
from utils import *

import torch
from torch import nn
import torch.nn.functional as F
from torch import optim

%matplotlib inline

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
```

## Markov Decision Process

As a quick recap, the diagram below explains the workflow of a Markov Decision Process (MDP)

Image taken from [Section 3.1 Reinforment Learning an Introduction](http://www.incompleteideas.net/book/RLbook2018.pdf#page=70)

## Environment and Agent specifications

The states, actions, reward and termination are given as follows for the lunar lander problem.

**Observation**:

     Type:  Box(8)
     Num    Observation                           Min            Max
     0      X position                            -inf           inf
     1      Y position                            -inf           inf
     2      X velocity                            -inf           inf
     3      Y velocity                            -inf           inf
     4      Theta w.r.t ground                    -inf           inf
     5      Theta rate                            -inf           inf
     6      1 if first leg has contact, else 0    -inf           inf
     7      1 if second leg has contact, else 0   -inf           inf

**Actions**:

     Type: Discrete(4)
     Num    Action
     0      Do nothing
     1      Fire left engine
     2      Fire main engine
     3      Fire right engine

**Reward**:

     Reward of 0 is awarded if the agent reached the flag(position = 0.5) on top of the mountain
     Reward of -1 is awarded if the position of the agent is less than 0.5
     Reward of -100 for flying off the screen
     Reward of +100 for successful landing
     Reward of -0.3 for firing main thrusters
     Reward of -0.03 for firing side thrusters
     Reward of +10 for each leg touching fround

**Starting State**:

     The starting position is above the landing target

**Episode Termination**:

     The lander crashes
     The lander comes to rest
     Episode length is greater than 200

For further information see [Github source code](https://github.com/openai/gym/blob/master/gym/envs/box2d/lunar_lander.py).

The next cell aims to show how to iterate with the action and observation space of the agent and extract relevant information from it

```python
env = gym.make("LunarLander-v2")
observation = env.reset()

# Object's type in the action Space
print("The Action Space is an object of type: {0}\n".format(env.action_space))
# Shape of the action Space
print("The shape of the action space is: {0}\n".format(env.action_space.n))
# Object's type in the Observation Space
print("The Environment Space is an object of type: {0}\n".format(env.observation_space))
# Shape of the observation space
print("The Shape of the dimension Space are: {0}\n".format(env.observation_space.shape))
# The high and low values in the observation space
print("The High values in the observation space are {0}, the low values are {1}\n".format(
    env.observation_space.high, env.observation_space.low))
# Example of observation
print("The Observations at a given timestep are {0}\n".format(env.observation_space.sample()))
```

    The Action Space is an object of type: Discrete(4)

    The shape of the action space is: 4

    The Environment Space is an object of type: Box(8,)

    The Shape of the dimension Space are: (8,)

    The High values in the observation space are [inf inf inf inf inf inf inf inf], the low values are [-inf -inf -inf -inf -inf -inf -inf -inf]

    The Observations at a given timestep are [-1.3609413  -0.81943125  0.00691082  0.965331   -1.3784901   1.0290705
     -1.7937465   0.85192055]

## Computing action-values with neural networks

To compute action-values, a feed-forward neural network is used. This apporach allows us to compute action-values using the weights of the neural network.

$$ q_\pi(s) \approx \hat{q}(s, a, w) = NN(s,a,w) $$

Neural networks are used to solve the control problem in RL, particularly, this networl is going to be used with an Episodic Semi-gradient Expected Sarsa agent. The inputs of the network are the states, which in this case are eight, the number of hidden layers and hidden units can vary. Finally, the number of inputs is equals to the number of actions in the problem, therefore, four output nodes are needed in the final layer. Each output node corresponds to the action value of a particular action.

For further information about Neural Networks for function approximation see [Section 9.7 of Reinforment Learning an Introduction](http://www.incompleteideas.net/book/RLbook2018.pdf#page=246)

Image taken from [Reinforcement learning specialization, C4L5S1](https://www.coursera.org/learn/complete-reinforcement-learning-system/lecture/CVH40/meeting-with-adam-getting-the-agent-details-right)

```python
# Neural Netork to compute action values
class ActionValueNetwork(nn.Module):
    # Work Required: Yes. Fill in the layer_sizes member variable (~1 Line).
    def __init__(self, network_config):
        super().__init__()

        # Number of states
        self.state_dim = network_config.get("state_dim")
        # Hidden units
        self.num_hidden_units = network_config.get("num_hidden_units")
        # Actions or output units
        self.num_actions = network_config.get("num_actions")

        # Initialzie hidden layer
        self.hidden = nn.Linear(self.state_dim, self.num_hidden_units)
        # Initialize output layer
        self.output = nn.Linear(self.num_hidden_units, self.num_actions)

    def forward(self, s):
        """
        This is a feed-forward pass in the network
        Args:
            s (Numpy array): The state, a 2D array of shape (batch_size, state_dim)
        Returns:
            The action-values (Torch array) calculated using the network's weights.
            A 2D array of shape (batch_size, num_actions)
        """
        # Transform observations into a pytorch tensor
        s = torch.Tensor(s)

        q_vals = F.relu(self.hidden(s))
        q_vals = self.output(q_vals)

        return q_vals
```

## Replay Buffer

Experience replay is a technique very similar to planning in RL. Overall, this technique is used to update the action-values of the agent with a set of "experience" collected in a model. This experience allows the model to learn without interacting with the environment using simulated experience.

Experience replay is a simple method that can get some of the advantages of planning by saving a buffer of experience and using the data stored in the buffer as a model. This view of prior data as a model works because the data represents actual transitions from the underlying MDP. The data stored in the model are the state, action, reward and next state.

The model will be filled until a queue size is reached, only then the model will drop its oldest observation and add a new one. With this buffer of information, it is possible to sample "batches" and update the action values of the agent.

As a quick recap, the next pseudocode shows the pseudocode for Dyna-Q algorithm where the agent performs planning steps, improving the learning process of the agent with simulated experience.

The planning process is given in the next image.

For further information about planning see [Section 8.2 of Reinforment Learning an Introduction](http://www.incompleteideas.net/book/RLbook2018.pdf#page=185). **Note**: Images taken from the last reference.

```python
class ReplayBuffer:
    def __init__(self, size, minibatch_size):
        """
        Args:
            size (integer): The size of the replay buffer.
            minibatch_size (integer): The sample size.
        """
        # Create the buffer
        self.buffer = []
        self.minibatch_size = minibatch_size
        self.max_size = size

    def append(self, state, action, reward, terminal, next_state):
        """
        Args:
            state (Numpy array): The state of size (state_dim)
            action (integer): The action.
            reward (float): The reward.
            terminal (integer): 1 if the next state is a terminal state and 0 otherwise.
            next_state (Numpy array): The next state of size (state_dim) .
        """
        if len(self.buffer) == self.max_size:
            # Delete first position of the buffer if the Queue size is equals to max size
            del self.buffer[0]
        # Append new step
        self.buffer.append([state, action, reward, terminal, next_state])

    def sample(self):
        """
        Returns:
            A list of transition tuples including state, action, reward, terinal, and next_state
            The return of this function is of size (minibatch_size)
        """
        idxs = np.random.choice(np.arange(len(self.buffer)), size=self.minibatch_size)
        return [self.buffer[idx] for idx in idxs]

    def size(self):
        return len(self.buffer)
```

## Softmax Policy

To compute the actions, a softmax policy is used. One advantage of a softmax policy is that it explores according to the action-values, meaning that an action with a moderate value has a higher chance of getting selected compared to an action with a lower value. This sort of policies provides a feasible alternative to do exploration.

The probability of selecting each action according to the softmax policy is shown below:

$$Pr{(A_t=a | S_t=s)} \hspace{0.1cm} \dot{=} \hspace{0.1cm} \frac{e^{Q(s, a)/\tau}}{\sum_{b \in A}e^{Q(s, b)/\tau}}$$

Here, $\tau$ is the temperature parameter which controls how much the agent focuses on the highest valued actions. The smaller the temperature, the more the agent selects the greedy action. Conversely, when the temperature is high, the agent selects among actions more uniformly random.

Given that a softmax policy exponentiates action values, if those values are large, exponentiating them could get very large. To implement the softmax policy in a numerically stable way,the maximum action-value is substracted from the action-values. Doing so, the probability of selecting each action looks as follows:

$$Pr{(A_t=a | S_t=s)} \hspace{0.1cm} \dot{=} \hspace{0.1cm} \frac{e^{Q(s, a)/\tau - max_{c}Q(s, c)/\tau}}{\sum_{b \in A}e^{Q(s, b)/\tau - max_{c}Q(s, c)/\tau}}$$

Recall that changing the action preferences (action-values in this case) for a constant, would not change the final value of the softmax probability. This Softmax implementation is different than the one provided by Pytorch.

For further informartion about Softmax policies and action preferences see [Section 13.1 of Reinforment Learning an Introduction](http://www.incompleteideas.net/book/RLbook2018.pdf#page=344)

```python
def softmax(action_values, tau=1.0):
    """
    Args:
        action_values (Tensor array): A 2D array of shape (batch_size, num_actions).
                       The action-values computed by an action-value network.
        tau (float): The temperature parameter scalar.
    Returns:
        A 2D Tensor array of shape (batch_size, num_actions). Where each column is a probability distribution
        over the actions representing the policy.
    """

    # Compute the preferences
    preferences = action_values / tau
    # Compute the maximum preference across the actions (Max action per row or batch)
    max_preference = torch.max(preferences, dim = 1)[0]

    # Reshape max_preference  to [Batch, 1]
    reshaped_max_preference = max_preference.view((-1, 1))

    # Computing numerator
    exp_preferences = torch.exp(preferences - reshaped_max_preference)
    # Computing the denominator suming over rows (batches).
    sum_of_exp_preferences = torch.sum(exp_preferences, dim = 1)

    # Reshape sum_of_exp_preferences array to [Batch, 1]
    reshaped_sum_of_exp_preferences = sum_of_exp_preferences.view((-1, 1))

    # Computing action probabilities
    action_probs = exp_preferences / reshaped_sum_of_exp_preferences
    action_probs = action_probs.squeeze()

    return action_probs
```

```python
# Testing the Softmax implementation
rand_generator = np.random.RandomState(0)
action_values = torch.tensor(rand_generator.normal(0, 1, (2, 4)))
tau = 0.5

action_probs = softmax(action_values, tau)
print("action_probs", action_probs)

assert(np.allclose(action_probs, np.array([
    [0.25849645, 0.01689625, 0.05374514, 0.67086216],
    [0.84699852, 0.00286345, 0.13520063, 0.01493741]
])))

```

    action_probs tensor([[0.2585, 0.0169, 0.0537, 0.6709],
            [0.8470, 0.0029, 0.1352, 0.0149]], dtype=torch.float64)

## Computing TD target and TD estimate

The TD target and TD estimate's computation will be done in the next lines. The main idea here is to obtain the action-value network updates with experience sampled from the experience replay buffer.

At time $t$, there is an action-value function represented as a neural network, say $Q_t$. The idea is to update the action-value function and get a new one we can use at the next timestep. We will get this $Q_{t+1}$ using multiple replay steps that each result in an intermediate action-value function $Q_{t+1}^{i}$ where $i$ indexes which replay step we are at.

In each replay step, we sample a batch of experiences from the replay buffer and compute a minibatch Expected-SARSA update. Across these N replay steps, we will use the current "un-updated" action-value network at time $t$, $Q_t$, for computing the action-values of the next-states. This contrasts using the most recent action-values from the last replay step $Q_{t+1}^{i}$. We make this choice to have targets that are stable across replay steps. Here is the pseudocode for performing the updates:

$$
\begin{align}
& Q_t \leftarrow \text{action-value network at timestep t (current action-value network)}\\
& \text{Initialize } Q_{t+1}^1 \leftarrow Q_t\\
& \text{For } i \text{ in } [1, ..., N] \text{ (i.e. N} \text{  replay steps)}:\\
& \hspace{1cm} s, a, r, t, s'
\leftarrow \text{Sample batch of experiences from experience replay buffer} \\
& \hspace{1cm} \text{Do Expected Sarsa update with } Q_t: Q_{t+1}^{i+1}(s, a) \leftarrow Q_{t+1}^{i}(s, a) + \alpha \cdot \left[r + \gamma \left(\sum_{b} \pi(b | s') Q_t(s', b)\right) - Q_{t+1}^{i}(s, a)\right]\\
& \hspace{1.5cm} \text{ making sure to add the } \gamma \left(\sum_{b} \pi(b | s') Q_t(s', b)\right) \text{ for non-terminal transitions only.} \\
& \text{After N replay steps, we set } Q_{t+1}^{N} \text{ as } Q_{t+1} \text{ and have a new } Q_{t+1} \text{for time step } t + 1 \text{ that we will fix in the next set of updates. }
\end{align}
$$

As you can see in the pseudocode, after sampling a batch of experiences, we do many computations. The basic idea however is that we are looking to compute a form of a TD error.

$$ R_{t+1} + \gamma \hat{q}(S_{t+1}, A_{t+1}, w)- \hat{q}(S_t, A_t, w) $$

Recall that the for this problem, the TD Target is given by.

$$ r + \gamma \left(\sum_{b} \pi(b | s') Q_t(s', b)\right) $$

Similarly, the TD estimate is.

$$ Q_{t+1}^{i}(s, a) $$

The Semi-gradient Expected Sarsa update is given below.

$$w \leftarrow w + \alpha[R_{t+1} + \gamma \sum_{a'}\pi(a' | S_{t+1}) \hat{q}(S_{t+1}, a', w) - \hat{q}(S_t, A_t, w)]\nabla \hat{q}(S_t, A_t, w)$$

For further explanation about Episodic semi-gradient control see [Section 10.1 of Reinforment Learning an Introduction](http://www.incompleteideas.net/book/RLbook2018.pdf#page=265z)

```python
# Method to compute the TD Target and TD estimate
def get_td(states, next_states, actions, rewards, discount, terminals, network, current_q, tau):
    """
    Args:
        states (Numpy array): The batch of states with the shape (batch_size, state_dim).
        next_states (Numpy array): The batch of next states with the shape (batch_size, state_dim).
        actions (Numpy array): The batch of actions with the shape (batch_size,).
        rewards (Numpy array): The batch of rewards with the shape (batch_size,).
        discount (float): The discount factor (gamma).
        terminals (Numpy array): The batch of terminals with the shape (batch_size,).
        network (ActionValueNetwork): The latest state of the network that is getting replay updates.
        current_q (ActionValueNetwork): The fixed network used for computing the targets,
                                        and particularly, the action-values at the next-states.
    Returns:
        target_vec (Tensor array): The TD Target for actions taken, of shape (batch_size,)
        estimate_vec (Tensor array): The TD estimate for actions taken, of shape (batch_size,)
    """

    # network is the latest state of the network that is getting replay updates. In other words,
    # network represents Q_{t+1}^{i} whereas current_q represents Q_t, the fixed network used
    # for computing the  targets, and particularly, the action-values at the next-states.

    # q_next_mat is a 2D Tensor of shape (batch_size, num_actions)
    # used to compute the action-values of the next states
    # Detach is used to remove this graph from the main graph
    q_next_mat = current_q.forward(next_states).detach()

    # Compute policy at next state.
    # probs_mat is a 2D Tensor of shape (batch_size, num_actions)
    probs_mat = softmax(q_next_mat, tau)

    # Sum of the action-values for the next_states weighted by the policy, probs_mat.
    # (1 - terminals) to make sure v_next_vec is zero for terminal next states.
    # v_next_vec is a 1D Tensor of shape (batch_size,)
    v_next_vec = torch.zeros((q_next_mat.shape[0]), dtype=torch.float64).detach()
    # Sum over rows axis (batches)
    v_next_vec = torch.sum(probs_mat * q_next_mat, dim = 1) * (1 - torch.tensor(terminals))

    # Compute Expected Sarsa target
    # target_vec is a 1D Tensor of shape (batch_size,)
    target_vec = torch.tensor(rewards) + (discount * v_next_vec)

    # Computing action values at the current states for all actions using network
    # q_mat is a 2D Tensor of shape (batch_size, num_actions)
    q_mat = network.forward(states)

    # Batch Indices is an array from 0 to the batch size - 1.
    batch_indices = torch.arange(q_mat.shape[0])

    # Compute q_vec by selecting q(s, a) from q_mat for taken actions
    # q_vec are the estimates
    # q_vec is a 1D Tensor of shape (batch_size)
    estimate_vec = q_mat[batch_indices, actions]

    return target_vec, estimate_vec
```

## Computing Network's optmizer

One important step is to optimize the network using the TD estimate and the TD target computed previously. As a quick recap, the Mean squared value error is given below.

$$\overline{VE} = \sum_s\mu(s)[V_\pi(s) - \hat{v}(s,w)]^2$$

The idea is to use $\overline{VE}$ as the Loss function to optimize the action-value network. For this particular problem, the MSE implementation provided by Pytorch is used. Additionally, the Adam optimizer is used to optimize the weights of the neural network.

See [Section 9.2 of Reinforment Learning an Introduction](http://www.incompleteideas.net/book/RLbook2018.pdf#page=221)

```python
### Work Required: Yes. Fill in code in optimize_network (~2 Lines).
def optimize_network(experiences, discount, optimizer, network, current_q, tau, criterion):
    """
    Args:
        experiences (Numpy array): The batch of experiences including the states, actions,
                                   rewards, terminals, and next_states.
        discount (float): The discount factor.
        network (ActionValueNetwork): The latest state of the network that is getting replay updates.
        current_q (ActionValueNetwork): The fixed network used for computing the targets,
                                        and particularly, the action-values at the next-states.
    Return:
        Loss (float): The loss value for the current batch.
    """

    # Get states, action, rewards, terminals, and next_states from experiences
    states, actions, rewards, terminals, next_states = map(list, zip(*experiences))
    states = np.concatenate(states) # Batch per states
    next_states = np.concatenate(next_states) # Batch per states
    rewards = np.array(rewards) # Batch size
    terminals = np.array(terminals) # Batch size
    batch_size = states.shape[0] # Batch size

    # Computing TD target and estimate using get_td function
    td_target, td_estimate = get_td(states, next_states, actions, rewards, discount, terminals, \
                                          network, current_q, tau)

    # zero the gradients buffer
    optimizer.zero_grad()
    # Compute the  Mean squared value error loss
    loss = criterion(td_estimate.double().to(device), td_target.to(device))
    # Backprop the error
    loss.backward()
    # Optimize the network
    optimizer.step()

    return (loss / batch_size).detach().numpy()
```

## Implementing Expected-Sarsa Agent

The final step is to use all the methods implemented above in the Expected-Sarsa Agent.

```python
### Expected Expected-Sarsa Agent
class ExpectedSarsaAgent():
    def __init__(self):
        self.name = "expected_sarsa_agent"

    def agent_init(self, agent_config):
        """Setup for the agent called when the experiment first starts.

        Set parameters needed to setup the agent.

        Assume agent_config dict contains:
        {
            network_config: dictionary,
            optimizer_config: dictionary,
            replay_buffer_size: integer,
            minibatch_sz: integer,
            num_replay_updates_per_step: float
            discount_factor: float,
        }
        """
        self.replay_buffer = ReplayBuffer(agent_config['replay_buffer_size'],
                                          agent_config['minibatch_sz'])
        # Add model to CPU or GPU respectively
        self.network = ActionValueNetwork(agent_config['network_config']).to(device)
        self.optimizer = optim.Adam(self.network.parameters(), lr = agent_config['optimizer_config']['step_size'],
                                    betas=(agent_config['optimizer_config']['beta_m'], agent_config['optimizer_config']['beta_v']),
                                    eps=agent_config['optimizer_config']['epsilon'])
        self.criterion = nn.MSELoss()
        self.num_actions = agent_config['network_config']['num_actions']
        self.num_replay = agent_config['num_replay_updates_per_step']
        self.discount = agent_config['gamma']
        self.tau = agent_config['tau']

        self.last_state = None
        self.last_action = None

        self.sum_rewards = 0
        self.episode_steps = 0
        self.loss = 0

    def policy(self, state):
        """
        Args:
            state (Numpy array): the state.
        Returns:
            the action.
        """
        action_values = self.network.forward(state)
        probs_batch = softmax(action_values, self.tau).detach().numpy()
        action = np.random.choice(self.num_actions, p=probs_batch.squeeze())
        return action

    def agent_start(self, state):
        """The first method called when the experiment starts, called after
        the environment starts.
        Args:
            state (Numpy array): the state from the
                environment's evn_start function.
        Returns:
            The first action the agent takes.
        """
        self.sum_rewards = 0
        self.episode_steps = 0
        self.last_state = np.array([state])
        self.last_action = self.policy(self.last_state)
        return self.last_action

    def agent_step(self, reward, state):
        """A step taken by the agent.
        Args:
            reward (float): the reward received for taking the last action taken
            state (Numpy array): the state from the
                environment's step based, where the agent ended up after the
                last step
        Returns:
            The action the agent is taking.
        """

        # Add current reward to the sum of rewards
        self.sum_rewards += reward
        self.episode_steps += 1

        # Make state an array of shape (1, state_dim) to add a batch dimension and
        # to later match the forward() and get_td() functions
        state = np.array([state])

        # Select action
        action = self.policy(state) #change for state for submission, normally, it is self.last_state

        # Append new experience to replay buffer
        self.replay_buffer.append(self.last_state, self.last_action, reward, 0, state)

        # Perform replay steps:
        if self.replay_buffer.size() > self.replay_buffer.minibatch_size:
            # Make a copy of the current network to obtain stable targets
            current_q = deepcopy(self.network)
            for _ in range(self.num_replay):
                # Get sample experiences from the replay buffer
                experiences = self.replay_buffer.sample()

                # Call optimize_network to update the weights of the network
                self.loss +=optimize_network(experiences, self.discount, self.optimizer, self.network, current_q, self.tau,
                                 self.criterion)

        # Update the last state and last action.
        self.last_state = state
        self.last_action = action

        return self.last_action

    # update of the weights using optimize_network
    def agent_end(self, reward):
        """Run when the agent terminates.
        Args:
            reward (float): the reward the agent received for entering the
                terminal state.
        """
        self.sum_rewards += reward
        self.episode_steps += 1

        # Set terminal state to an array of zeros
        state = np.zeros_like(self.last_state)

        self.replay_buffer.append(self.last_state, self.last_action, reward, 1, state)

        # Perform replay steps:
        if self.replay_buffer.size() > self.replay_buffer.minibatch_size:
            current_q = deepcopy(self.network)
            for _ in range(self.num_replay):

                # Get sample experiences from the replay buffer
                experiences = self.replay_buffer.sample()

                # Call optimize_network to update the weights of the network
                self.loss += optimize_network(experiences, self.discount, self.optimizer, self.network, current_q, self.tau,
                                 self.criterion)

    def agent_message(self, message):
        if message == "get_sum_reward":
            return self.sum_rewards, self.episode_steps
        else:
            raise Exception("Unrecognized Message!")

```

## Running the experiment

The following lines solves the Lunar Lander problem and plot the average reward obtained over episodes, steps taken to solve the challenge at a specific episode and average loss over episodes.

```python
# Test the expected Sarsa Agent
#model = ActionValueNetwork(network_config).to(device)
num_runs = 1
num_episodes = 1000

# Experiment parameters
agent_info = {
             'network_config': {
                 'state_dim': env.observation_space.shape[0],
                 'num_hidden_units': 256,
                 'num_actions': env.action_space.n
             },
             'optimizer_config': {
                 'step_size': 1e-3,
                 'beta_m': 0.9,
                 'beta_v': 0.999,
                 'epsilon': 1e-8
             },
             'replay_buffer_size': 50000,
             'minibatch_sz': 8,
             'num_replay_updates_per_step': 4,
             'gamma': 0.99,
             'tau': 0.001}

# Variable to store the amount of steps taken to solve the challeng
all_steps = []
# Variable to save the rewards in an episode
all_rewards = []
all_loss = []

# Agent
agent = ExpectedSarsaAgent()

# Environment
env = gym.make('LunarLander-v2')
env.reset()
# Maximum number of possible iterations (default was 200)
env._max_episode_steps = 10000

# Number of runs are the times the experiment will start again (a.k.a episode)
for n_runs in range(num_runs):

    # Resets environment
    observation = env.reset()
    # Reset agent
    agent.agent_init(agent_info)
    # Generate last state and action in the agent
    last_action = agent.agent_start(observation)
    # Steps, rewards and loss at each episode to solve the challenge
    steps_per_episode = []
    rewards_per_episode = []
    loss_per_episode = []

    # Times the environment will start again without resetting the agent
    for t in tqdm(range(num_episodes)):

        # Reset done flag
        done = False
        # Set rewards, steps and loss to zero
        rewards = 0
        n_steps = 0
        agent.loss = 0
        # Reset environment
        observation = env.reset()
        # Run until the experiment is over
        while not done:

            # Render the environment only after t > # episodes
            if t > 300:
                env.render()

            # Take a step with the environment
            observation, reward, done, info = env.step(last_action)

            rewards += reward
            n_steps += 1

            # If the goal has been reached stop
            if done:
                # Last step with the agent
                agent.agent_end(reward)
            else:
                # Take a step with the agent
                last_action = agent.agent_step(reward, observation)

        # Append steps taken to solve the episode
        steps_per_episode.append(n_steps)
        # Reward obtained during the episode
        rewards_per_episode.append(rewards)
        # Loss obtained solving the experiment
        loss_per_episode.append(agent.loss)

    # Steps taken to solve the experiment during all
    all_steps.append(np.array(steps_per_episode))
    # Awards obtained during all episode
    all_rewards.append(np.array(rewards_per_episode))
    # Loss obtained during all episodes
    all_loss.append(loss_per_episode)

env.close()
```

    100%|██████████| 1000/1000 [51:03<00:00,  3.06s/it]

```python
steps_average = np.mean(np.array(all_steps), axis=0)
plt.plot(steps_average, label = 'Steps')
plt.xlabel("Episodes")
plt.ylabel("Iterations",rotation=0, labelpad=40)
plt.xlim(-0.2, num_episodes)
plt.ylim(steps_average.min(), steps_average.max())
plt.title("Average iterations to solve the experiment over runs")
plt.legend()
plt.show()
print("The Minimum number of iterations used to solve the experiment were: {0}\n".format(np.array(all_steps).min()))
print("The Maximum number of iterations used to solve the experiment were: {0}\n".format(np.array(all_steps).max()))
```

```python
rewards_average = np.mean(all_rewards, axis=0)
plt.plot(rewards_average, label = 'Average Reward')
plt.xlabel("Episodes")
plt.ylabel("Sum of\n rewards\n during\n episode" ,rotation=0, labelpad=40)
plt.xlim(-0.2, num_episodes)
plt.ylim(rewards_average.min(), rewards_average.max())
plt.title("Average reward to solve the experiment over runs")
plt.legend()
plt.show()
print("The best reward obtained solving the experiment was: {0}\n".format(np.array(all_rewards).max()))
print("The Worst reward obtained solving the experiment was: {0}\n".format(np.array(all_rewards).min()))
```

```python
loss_average = np.mean(np.array(all_loss), axis=0)
plt.plot(loss_average, label = 'Steps')
plt.xlabel("Episodes")
plt.ylabel("Average loss",rotation=0, labelpad=40)
plt.xlim(-0.2, num_episodes)
plt.ylim(loss_average.min(), loss_average.max())
plt.title("Average loss over iterations")
plt.legend()
plt.show()
print("The best loss obtained solving the experiment was: {0}\n".format(np.array(loss_average).min()))
print("The Worst loss obtained solving the experiment was: {0}\n".format(np.array(loss_average).max()))

```

## Using the last trained Agent

This lines shows in a video the performance of the last trained agent and save a video with the results.

```python
# Test Sarsa Agent
num_runs = 1
num_episodes = 1000

# Environment
env_to_wrap = gym.make('LunarLander-v2')
# Maximum number of possible iterations (default was 200)
env_to_wrap._max_episode_steps = 1500
env = Monitor(env_to_wrap, "./videos/lunarLander", video_callable=lambda episode_id: True, force=True)

# Number of runs are the times the experiment will start again (a.k.a episode)
for n_runs in tqdm(range(num_runs)):

    # Resets environment
    observation = env.reset()
    # Generate last state and action in the agent
    last_action = agent.agent_start(observation)

    # Times the environment will start again without resetting the agent
    for t in tqdm(range(num_episodes)):

        # View environment
        env.render()
        # Take a step with the environment
        observation, reward, done, info = env.step(last_action)

        # If the goal has been reached stop
        if done:
            # Last step with the agent
            agent.agent_end(reward)
            break
        else:
            # Take a step with the agent
            last_action = agent.agent_step(reward, observation)

env.close()
env_to_wrap.close()

print("Episode finished after {} timesteps".format(t+1))
```

      0%|          | 0/1 [00:00<?, ?it/s]
      0%|          | 0/1000 [00:00<?, ?it/s]
      1%|          | 7/1000 [00:00<00:15, 64.45it/s]
      1%|▏         | 14/1000 [00:00<00:14, 65.81it/s]
      2%|▏         | 21/1000 [00:00<00:14, 66.28it/s]
      3%|▎         | 28/1000 [00:00<00:14, 66.48it/s]
      4%|▎         | 35/1000 [00:00<00:14, 67.30it/s]
      4%|▍         | 42/1000 [00:00<00:14, 67.19it/s]
      5%|▌         | 50/1000 [00:00<00:13, 68.34it/s]
      6%|▌         | 57/1000 [00:00<00:13, 68.59it/s]
      6%|▋         | 65/1000 [00:00<00:13, 69.27it/s]
      7%|▋         | 72/1000 [00:01<00:13, 68.31it/s]
      8%|▊         | 79/1000 [00:01<00:13, 68.32it/s]
      9%|▊         | 86/1000 [00:01<00:13, 67.84it/s]
      9%|▉         | 93/1000 [00:01<00:13, 68.45it/s]
     10%|█         | 100/1000 [00:01<00:13, 68.64it/s]
     11%|█         | 107/1000 [00:01<00:12, 68.78it/s]
     11%|█▏        | 114/1000 [00:01<00:12, 68.76it/s]
     12%|█▏        | 121/1000 [00:01<00:12, 68.15it/s]
     13%|█▎        | 128/1000 [00:01<00:12, 68.30it/s]
     14%|█▎        | 135/1000 [00:01<00:12, 68.15it/s]
     14%|█▍        | 142/1000 [00:02<00:12, 68.07it/s]
     15%|█▍        | 149/1000 [00:02<00:12, 68.05it/s]
     16%|█▌        | 156/1000 [00:02<00:12, 68.13it/s]
     16%|█▋        | 163/1000 [00:02<00:12, 68.22it/s]
     17%|█▋        | 171/1000 [00:02<00:11, 69.21it/s]
     18%|█▊        | 179/1000 [00:02<00:11, 69.79it/s]
     19%|█▊        | 186/1000 [00:02<00:11, 69.32it/s]
     19%|█▉        | 194/1000 [00:02<00:11, 70.24it/s]
     20%|██        | 202/1000 [00:02<00:11, 70.40it/s]
     21%|██        | 210/1000 [00:03<00:11, 70.63it/s]
     22%|██▏       | 218/1000 [00:03<00:11, 70.29it/s]
     23%|██▎       | 226/1000 [00:03<00:10, 71.32it/s]
     23%|██▎       | 234/1000 [00:03<00:10, 70.17it/s]
     24%|██▍       | 242/1000 [00:03<00:10, 71.23it/s]
     25%|██▌       | 250/1000 [00:03<00:10, 72.79it/s]
     26%|██▌       | 260/1000 [00:03<00:10, 69.24it/s]
    100%|██████████| 1/1 [00:03<00:00,  3.90s/it]

    Episode finished after 261 timesteps