Table of Contents

1  Evaluating Imbalanced Datasets

1.1  Dataset

1.2  Class Weighting

1.3  F1 Score

1.4  Conclusion

2  Reference

```python
# code for loading the format for the notebook
import os

# path : store the current path to convert back to it later
path = os.getcwd()
os.chdir(os.path.join('..', '..', 'notebook_format'))
from formats import load_style
load_style(plot_style = False)
```

```python
os.chdir(path)

# 1. magic for inline plot
# 2. magic to print version
# 3. magic so that the notebook will reload external python modules
# 4. magic to enable retina (high resolution) plots
# https://gist.github.com/minrk/3301035
%matplotlib inline
%load_ext watermark
%load_ext autoreload
%autoreload 2
%config InlineBackend.figure_format = 'retina'

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import precision_recall_curve, roc_curve
from sklearn.metrics import precision_score, recall_score, f1_score

%watermark -a 'Ethen' -d -t -v -p numpy,pandas,sklearn,matplotlib
```

    Ethen 2018-06-16 11:54:30

    CPython 3.6.4
    IPython 6.4.0

    numpy 1.14.4
    pandas 0.23.0
    sklearn 0.19.1
    matplotlib 2.2.2

# Evaluating Imbalanced Datasets

This documentation illustrates the trade off between True Positive Rate and False Positive Rate using ROC and Precision/Recall (PR) curves. In the end, we will take a look at why, for binary classification problem, apart from solely using the popular evaluation metric ROC curve we should also look at other evaluation metric such as precision and recall especially when working with highly imbalanced dataset.

## Dataset

The dataset we'll be using today can be downloaded from the [Kaggle website](https://www.kaggle.com/mlg-ulb/creditcardfraud).

```python
filepath = os.path.join('data', 'creditcard.csv')
df = pd.read_csv(filepath)
print('dimension: ', df.shape)
df.head()
```

    dimension:  (284807, 31)

      Time
      V1
      V2
      V3
      V4
      V5
      V6
      V7
      V8
      V9
      ...
      V21
      V22
      V23
      V24
      V25
      V26
      V27
      V28
      Amount
      Class

      0
      0.0
      -1.359807
      -0.072781
      2.536347
      1.378155
      -0.338321
      0.462388
      0.239599
      0.098698
      0.363787
      ...
      -0.018307
      0.277838
      -0.110474
      0.066928
      0.128539
      -0.189115
      0.133558
      -0.021053
      149.62
      0

      1
      0.0
      1.191857
      0.266151
      0.166480
      0.448154
      0.060018
      -0.082361
      -0.078803
      0.085102
      -0.255425
      ...
      -0.225775
      -0.638672
      0.101288
      -0.339846
      0.167170
      0.125895
      -0.008983
      0.014724
      2.69
      0

      2
      1.0
      -1.358354
      -1.340163
      1.773209
      0.379780
      -0.503198
      1.800499
      0.791461
      0.247676
      -1.514654
      ...
      0.247998
      0.771679
      0.909412
      -0.689281
      -0.327642
      -0.139097
      -0.055353
      -0.059752
      378.66
      0

      3
      1.0
      -0.966272
      -0.185226
      1.792993
      -0.863291
      -0.010309
      1.247203
      0.237609
      0.377436
      -1.387024
      ...
      -0.108300
      0.005274
      -0.190321
      -1.175575
      0.647376
      -0.221929
      0.062723
      0.061458
      123.50
      0

      4
      2.0
      -1.158233
      0.877737
      1.548718
      0.403034
      -0.407193
      0.095921
      0.592941
      -0.270533
      0.817739
      ...
      -0.009431
      0.798278
      -0.137458
      0.141267
      -0.206010
      0.502292
      0.219422
      0.215153
      69.99
      0

5 rows × 31 columns

A brief description of the dataset based on the data overview section from the download source.

> The datasets contains transactions made by credit cards in September 2013 by european cardholders. This dataset presents transactions that occurred in two days, where we have 492 frauds out of 284,807 transactions.
> Unfortunately, due to confidentiality issues, we cannot provide the original features and more background information about the data. Features V1, V2, ... V28 are the principal components obtained with PCA, the only features which have not been transformed with PCA are 'Time' and 'Amount'. Feature 'Time' contains the seconds elapsed between each transaction and the first transaction in the dataset. The feature 'Amount' is the transaction amount. Feature 'Class' is the response variable and it takes value 1 in case of fraud and 0 otherwise.

The only feature-engineering that we'll be doing for now is to convert the feature "Time" (seconds from which the very first data observation took place) to hours of a day. While we're at it, let's take a look at a breakdown of our legit vs fraud transactions via a pivot table and a plot of fraud transactions' count over time for a quick exploratory data analysis.

```python
df['hour'] = np.ceil(df['Time'].values / 3600) % 24
fraud_over_hour = df.pivot_table(values = 'Amount', index = 'hour', columns = 'Class', aggfunc = 'count')
fraud_over_hour
```

      Class
      0
      1

      hour

      0.0
      10919
      21

      1.0
      7687
      6

      2.0
      4212
      10

      3.0
      3269
      57

      4.0
      3476
      17

      5.0
      2185
      23

      6.0
      2979
      11

      7.0
      4093
      9

      8.0
      7219
      23

      9.0
      10266
      9

      10.0
      15824
      16

      11.0
      16593
      8

      12.0
      16804
      53

      13.0
      15400
      17

      14.0
      15350
      17

      15.0
      16545
      23

      16.0
      16434
      26

      17.0
      16435
      22

      18.0
      16135
      29

      19.0
      17003
      33

      20.0
      15632
      19

      21.0
      16739
      18

      22.0
      17692
      16

      23.0
      15424
      9

```python
plt.rcParams['font.size'] = 12
plt.rcParams['figure.figsize'] = 8, 6

plt.plot(fraud_over_hour.index, fraud_over_hour[1])
plt.title('Fraudulent Transaction over Hour')
plt.ylabel('Fraudulent Count')
plt.xlabel('Hour')
plt.show()
```

```python
# prepare the dataset for modeling;
# extract the features and labels, perform a quick train/test split
label = df['Class']
pca_cols = [col for col in df.columns if col.startswith('V')]
input_cols = ['hour', 'Amount'] + pca_cols
df = df[input_cols]

df_train, df_test, y_train, y_test = train_test_split(
    df, label, stratify = label, test_size = 0.35, random_state = 1)

print('training data dimension:', df_train.shape)
df_train.head()
```

    training data dimension: (185124, 30)

      hour
      Amount
      V1
      V2
      V3
      V4
      V5
      V6
      V7
      V8
      ...
      V19
      V20
      V21
      V22
      V23
      V24
      V25
      V26
      V27
      V28

      183716
      11.0
      0.76
      2.089242
      0.629632
      -3.364200
      0.489395
      1.553483
      -0.986382
      0.698731
      -0.284807
      ...
      0.084549
      -0.107484
      -0.041904
      0.058986
      -0.103786
      0.050964
      0.416364
      0.694420
      -0.082845
      -0.028126

      145898
      1.0
      0.49
      -0.914450
      2.361973
      0.335526
      4.114377
      0.557196
      0.874183
      0.332492
      0.540016
      ...
      1.614699
      0.352808
      -0.159933
      -0.261357
      0.186308
      0.718244
      -0.911284
      -0.093219
      -0.282263
      -0.229368

      247003
      19.0
      2.69
      2.070918
      -0.062049
      -1.128898
      0.383220
      -0.079241
      -1.130912
      0.177956
      -0.321792
      ...
      0.201919
      -0.222333
      -0.299237
      -0.732516
      0.322038
      -0.112259
      -0.281414
      0.205225
      -0.071557
      -0.061429

      56024
      14.0
      159.95
      0.929003
      -0.262584
      -0.026089
      0.596927
      -0.469012
      -1.022218
      0.463968
      -0.176288
      ...
      0.278433
      0.152181
      -0.236715
      -1.149710
      0.035834
      0.504788
      0.139781
      0.095334
      -0.102652
      0.025986

      217153
      16.0
      44.22
      1.831384
      -0.814800
      -0.727692
      -0.075094
      0.028202
      1.380668
      -0.941283
      0.502667
      ...
      -1.051171
      -0.240155
      0.242974
      0.905053
      0.189693
      -0.540499
      -0.358400
      0.202675
      0.048368
      -0.044683

5 rows × 30 columns

```python
# we'll be using linear models later, hence
# we standardize our features to ensure they are
# all at the same scale
standardize = StandardScaler()
X_train = standardize.fit_transform(df_train)
X_test = standardize.transform(df_test)

label_distribution = np.bincount(label) / label.size
print('labels distribution:', label_distribution)
print('Fraud is {}% of our data'.format(label_distribution[1] * 100))
```

    labels distribution: [0.99827251 0.00172749]
    Fraud is 0.1727485630620034% of our data

## Class Weighting

With scikit-learn, we can give higher weights to the minority class (the model will be penalized more when misclassifying a minority class) by modifying the `class_weight` argument during model initialization. Let's see what affect will this have with our model. The following code chunk manually selects a range of weights to boost the minority class and tracks various metrics to see the model's performance across different class weighting values.

Note that the following section assumes knowledge of model performance metric such as precision, recall and AUC. The following link contains resources into those concepts if needed. [Notebook: AUC (Area under the ROC curve and precision/recall curve) from scratch](http://nbviewer.jupyter.org/github/ethen8181/machine-learning/blob/master/model_selection/auc/auc.ipynb)

```python
fig = plt.figure(figsize = (15, 8))
ax1 = fig.add_subplot(1, 2, 1)
ax1.set_xlim([-0.05, 1.05])
ax1.set_ylim([-0.05, 1.05])
ax1.set_xlabel('Recall')
ax1.set_ylabel('Precision')
ax1.set_title('PR Curve')

ax2 = fig.add_subplot(1, 2, 2)
ax2.set_xlim([-0.05, 1.05])
ax2.set_ylim([-0.05, 1.05])
ax2.set_xlabel('False Positive Rate')
ax2.set_ylabel('True Positive Rate')
ax2.set_title('ROC Curve')

f1_scores = []
recall_scores = []
precision_scores = []
pos_weights = [1, 10, 25, 50, 100, 10000]
for pos_weight in pos_weights:
    lr_model = LogisticRegression(class_weight = {0: 1, 1: pos_weight})
    lr_model.fit(X_train, y_train)

    # plot the precision-recall curve and AUC curve
    pred_prob = lr_model.predict_proba(X_test)[:, 1]
    precision, recall, _ = precision_recall_curve(y_test, pred_prob)
    tpr, fpr, _ = roc_curve(y_test, pred_prob)

    ax1.plot(recall, precision, label = pos_weight)
    ax2.plot(tpr, fpr, label = pos_weight)

    # track the precision, recall and f1 score
    pred = lr_model.predict(X_test)
    f1_test = f1_score(y_test, pred)
    recall_test = recall_score(y_test, pred)
    precision_test = precision_score(y_test, pred)
    f1_scores.append(f1_test)
    recall_scores.append(recall_test)
    precision_scores.append(precision_test)

ax1.legend(loc = 'lower left')
ax2.legend(loc = 'lower right')

plt.show()
```

A good classifier would have a PR (Precision/Recall) curve closer to the upper-right corner and a ROC curve to the upper-left corner. Based on the plot above, we can see that while both curves uses the same underlying data, i.e. the real class labels and the predicted probability, the two charts can tell different stories, with some weights seem to perform better based on the ROC curve's chart.

To be explicit, different settings of the `class_weight` argument all seem to perform pretty well for ROC curve, but some poorly for PR curve. This is due to the fact that for ROC curve, one of the axis shows the false positive rate (number of false positives / total number of negatives), and this ratio will change much when the total number of negatives is large. Whereas for PR curve, one of the axis, precision (number of true positives / total number of predicted positives), is less affected by this.

Another way to visualize the model's performance metric is to use a bar-plot to visualize the precision/recall/f1 score at different class weighting values.

```python
def score_barplot(precision_scores, recall_scores, f1_scores, pos_weights, figsize = (8, 6)):
    """Visualize precision/recall/f1 score at different class weighting values."""
    width = 0.3
    ind = np.arange(len(precision_scores))
    fig = plt.figure(figsize = figsize)
    ax = fig.add_subplot(111)
    b1 = ax.bar(ind, precision_scores, width, color = 'lightskyblue')
    b2 = ax.bar(ind + width, recall_scores, width, color = 'lightcoral')
    b3 = ax.bar(ind + (2 * width), f1_scores, width, color = 'gold')

    ax.set_xticks(ind + width)
    ax.set_xticklabels(pos_weights)
    ax.set_ylabel('score')
    ax.set_xlabel('positive weights')
    ax.set_ylim(0, 1.3)
    ax.legend(handles = [b1, b2, b3], labels = ['precision', 'recall', 'f1'])
    plt.tight_layout()
    plt.show()

score_barplot(precision_scores, recall_scores, f1_scores, pos_weights)
```

Judging from the plot above, the can see that when the weight's value is set at 10, we seem to have strike a good balance between precision and recall (this setting has the highest f1 score, we'll have a deeper discussion on f1 score in the next section), where our model can detect 80% of the fraudulent transaction, while not annoying a bunch of customers with false positives. Another observation is that if we were to set the class weighting value to 10,000 we would be able to increase our recall score at the expense of more mis-classified legit cases (as depicted by the low precision score).

```python
# this code chunk shows the same idea applies when using tree-based models
f1_scores = []
recall_scores = []
precision_scores = []
pos_weights = [1, 10, 100, 10000]
for pos_weight in pos_weights:
    rf_model = RandomForestClassifier(n_estimators = 50, max_depth = 6, n_jobs = -1,
                                      class_weight = {0: 1, 1: pos_weight})
    rf_model.fit(df_train, y_train)

    # track the precision, recall and f1 score
    pred = rf_model.predict(df_test)
    f1_test = f1_score(y_test, pred)
    recall_test = recall_score(y_test, pred)
    precision_test = precision_score(y_test, pred)
    f1_scores.append(f1_test)
    recall_scores.append(recall_test)
    precision_scores.append(precision_test)

score_barplot(precision_scores, recall_scores, f1_scores, pos_weights)
```

## F1 Score

The formula for F1 score is:

\begin{align}
F1 &= 2 * \frac{\text{precision} * \text{recall}}{\text{precision} + \text{recall}}
\end{align}

F1 score can be interpreted as a weighted average or harmonic mean of precision and recall, where the relative contribution of precision and recall to the F1 score are equal. F1 score reaches its best value at 1 and worst score at 0.

When we create a classifier, often times we need to make a compromise between the recall and precision, it is kind of hard to compare a model with high recall and low precision versus a model with high precision but low recall. F1 score merge these two metrics into a single measure that we can use to compare two models. This is not to say that a model with higher F1 score is always better as it depends on the use case, more on this in the conclusion section.

To understand the rationale behind harmonic means, let's digress from machine learning evaluation metrics and take a look at a canonical example of using harmonic means. Consider a trip to the grocery store & back:

- On the way there we drove 30 mph the entire way.
- On the way back traffic was crawling, so we instead drove 10 mph the entire way.
- We took the same route and covered the same amount of ground (5 miles) each way.

What was our average speed across this entire trip's duration? When prompted to calculate the average of something, we might naively turn to arithmetic mean to 30 mph and 10 mph and proudly declare that the average speed for the whole trip was 20 mph. But if we step back and think about it for a moment, we'll realize that because we traveled faster on our way there, we covered the 5 miles quicker & spent less time overall traveling at that speed, thus our average speed across our entire trip should be closer to 10 mph because we spent longer traveling at that speed. In order to calculate the arithmetic mean correctly here, we have to first calculate the amount of time spent traveling at each rate, then weight our mean calculation accordingly.

- Trip There: (at 30 mph)
    - 30 miles per 60 minutes = 1 mile every 2 minutes = 1/2 mile every minute
    - 5 miles at 1/2 mile per minute = 5 / 1/2 = 10 minutes
    - Trip There time = 10 minutes
- Trip Back: (at 10 mph)
    - 10 miles per 60 minutes = 1 mile every 6 minutes = 1/6 miles every minute
    - 5 miles at 1/6 mile per minute = 5 / 1/6 = 30 minutes
    - Trip Back time = 30 minutes
- Total trip time = 10 + 30 = 40 minutes
    - Trip there / total trip = 10 / 40 minutes = .25
    - Trip back / total trip = 30 / 40 minutes = .75
- Weighted Arithmetic Mean = (30 mph x .25) + (10 mph x .75) = 7.5 + 7.5 = 15

We now get an average speed of 15 mph, which matches the intuition that the number should be lower than our unweighted arithmetic mean of 20 mph. Now let's look at the same problem from a harmonic mean perspective. The harmonic mean can be described in words as: the reciprocal of the arithmetic mean of the reciprocals of the dataset. That's a lot of reciprocal flips there ..., the notation for the computation is:

\begin{align}
\text{Harmonic mean} &= \big( \frac{\sum_{i=1}^n x_i^{-1}}{n} \big)^{-1}
\end{align}

Now back our original example, the harmonic mean of 30 mph and 10 mph:

- Arithmetic mean of reciprocals = 1/30 + 1/10 = 4/30 / 2 = 4/60 = 1/15
- Reciprocal of arithmetic mean = 1 / 1/15 = 15/1 = 15

Our true average rate of travel, automagically adjusted for time spent traveling in each direction = 15 mph! In this example, using harmonic mean helps us find relationship with datasets of rates/ratios/fractions over different length/periods. By using reciprocals, it already accounts for the proportion of time that was implicit in the rates.

With this knowledge in mind, let's turn our heads back to F1 score's calculation. Precision and recall both have true positives in the numerator, but they have different denominators. In order for an average to be valid, we need the values to be in the same scaled units. In our traveling distance example miles per hour need to be compared over the same number of hours. Thus, to take the average of precision and recall really only makes sense to average their reciprocals (adjusts for scale), thus the harmonic mean.

## Conclusion

To sum it up, when using model-based metrics to evaluate a imbalanced classification problem, it is often times recommended to look at the precision and recall score to fully evaluate the overall effectiveness of a model.

A model with high recall but low precision score returns many positive results, but most of its predicted labels are incorrect when compared to the ground truth. On the other hand, a model with high precision but low recall score returns very few results, but most of its predicted labels are correct when compared to the ground-truth. An ideal scenario would be a model with high precision and high recall, meaning it will return many results, with all results labeled correctly. Unfortunately, in most cases, precision and recall are often in tension. That is, improving precision typically reduces recall and vice versa.

So how do we know if we should sacrifice our precision for more recall, i.e. catching fraud? That is business decisions come into play. If the cost of missing a fraud highly outweighs the cost of canceling a bunch of legit customer transactions, i.e. false positives, then perhaps we can choose a weight that gives us a higher recall rate. Or maybe catching 80% of fraud is good enough for the business, in that case, we can also minimize the "user friction" by keeping our precision high.

# Reference

- [Notes: The Harmonic Mean](http://jwilson.coe.uga.edu/EMT725/HM/HM.html)
- [Blog:
Classification: Precision and Recall](https://developers.google.com/machine-learning/crash-course/classification/precision-and-recall)
- [Blog: On Average, You’re Using the Wrong Average: Geometric & Harmonic Means in Data Analysis](https://towardsdatascience.com/on-average-youre-using-the-wrong-average-geometric-harmonic-means-in-data-analysis-2a703e21ea0)
- [Kaggle Kernel: Imbalanced data & why you should NOT use ROC curve](https://www.kaggle.com/lct14558/imbalanced-data-why-you-should-not-use-roc-curve)
- [Stackoverflow: what is f1-score and what its value indicates?](https://stackoverflow.com/questions/45963174/what-is-f1-score-and-what-its-value-indicates)
- [Stackoverflow: Why is the F-Measure a harmonic mean and not an arithmetic mean of the Precision and Recall measures?](https://stackoverflow.com/questions/26355942/why-is-the-f-measure-a-harmonic-mean-and-not-an-arithmetic-mean-of-the-precision)


## Inštrukcie

Pred začiatkom vás poprosím spustiť príkaz z hornej lišty `Runtime -> Run All` alebo `Ctrl + F9`.
Pre spustenie ľubovoľnej bunky kódu stlačte `Ctrl + Enter` alebo šípku vľavo od samotnej bunky.
Toto prostredie obashuje veľa rôznych skratiek, ktoré sa spúšťajú stlačením kombinácie `Ctrl + M` a následne nejaké ďalšie tlačítko. Napríklad pomocníka na používanie tohto prostredia nájdete po stlačení `Ctrl + M` a následne `h`.

Pri názvoch sekcií často nájdete nápis `1 cell hidden` alebo niečo podobné. Text pred nimi vám vždy prezradí, čo sa v nich nachádza. Rozbalte ich iba ak chcete pokračovať ďalej, sú zabalené napríklad preto, aby ste rovno nevideli riešenia a mali ste čas sa zamyslieť.

# Pomocný kód

```python
import graphviz

COLOR_VYHRA_PRVY = "darkorange"
COLOR_VYHRA_DRUHY = "darkorchid"
COLOR_HLAVA = "forestgreen"
COLOR_ZNAK = "red"

HLAVA = "H"
ZNAK = "Z"
PRAZDNY = "_"

HODY = [HLAVA, ZNAK, PRAZDNY]
FARBY = {HLAVA: COLOR_HLAVA, ZNAK: COLOR_ZNAK}

def graf1_1():
    d = graphviz.Digraph(graph_attr={"rankdir": "LR"})
    for prvy_hod in HODY:
        for druhy_hod in HODY:
            if druhy_hod == PRAZDNY and prvy_hod != PRAZDNY:
                continue # vynechaj tieto moznosti
            for novy_hod in [HLAVA, ZNAK]:
                d.edge(prvy_hod + druhy_hod, druhy_hod + novy_hod, color=FARBY[novy_hod])
    d.node(HLAVA + ZNAK, color=COLOR_VYHRA_PRVY)
    d.node(HLAVA + HLAVA, color=COLOR_VYHRA_DRUHY)
    return d

def graf1_2():
    d = graphviz.Digraph(graph_attr={"rankdir": "LR"})
    d.edge(PRAZDNY + PRAZDNY, PRAZDNY + HLAVA, color=FARBY[HLAVA])
    d.edge(PRAZDNY + PRAZDNY, PRAZDNY + PRAZDNY, color=FARBY[ZNAK])
    d.edge(PRAZDNY + HLAVA, HLAVA + HLAVA, color=FARBY[HLAVA])
    d.edge(PRAZDNY + HLAVA, HLAVA + ZNAK, color=FARBY[ZNAK])
    d.node(HLAVA + ZNAK, color=COLOR_VYHRA_PRVY)
    d.node(HLAVA + HLAVA, color=COLOR_VYHRA_DRUHY)
    return d

def graf2_1():
    d = graphviz.Digraph(graph_attr={"rankdir": "LR"})
    for prvy_hod in HODY:
        for druhy_hod in HODY:
            if druhy_hod == PRAZDNY and prvy_hod != PRAZDNY:
                continue # vynechaj tieto moznosti
            for novy_hod in [HLAVA, ZNAK]:
                d.edge(prvy_hod + druhy_hod, druhy_hod + novy_hod, color=FARBY[novy_hod])
    d.node(ZNAK + HLAVA, color=COLOR_VYHRA_PRVY)
    d.node(HLAVA + HLAVA, color=COLOR_VYHRA_DRUHY)
    return d

def graf2_2():
    d = graphviz.Digraph(graph_attr={"rankdir": "LR"})
    d.edge(PRAZDNY + PRAZDNY, PRAZDNY + HLAVA, color=FARBY[HLAVA])
    d.edge(PRAZDNY + PRAZDNY, PRAZDNY + ZNAK, color=FARBY[ZNAK])
    d.edge(PRAZDNY + HLAVA, HLAVA + HLAVA, color=FARBY[HLAVA])
    d.edge(PRAZDNY + HLAVA, PRAZDNY + ZNAK, color=FARBY[ZNAK])
    d.edge(PRAZDNY + ZNAK, ZNAK + HLAVA, color=FARBY[HLAVA])
    d.edge(PRAZDNY + ZNAK, PRAZDNY + ZNAK, color=FARBY[ZNAK])
    d.node(ZNAK + HLAVA, color=COLOR_VYHRA_PRVY)
    d.node(HLAVA + HLAVA, color=COLOR_VYHRA_DRUHY)
    return d

def graf3_1():
    d = graphviz.Digraph(graph_attr={"rankdir": "LR"})
    for prvy_hod in HODY:
        for druhy_hod in HODY:
            for treti_hod in HODY:
                if (druhy_hod == PRAZDNY and prvy_hod != PRAZDNY) or (
                    treti_hod == PRAZDNY and (prvy_hod != PRAZDNY or druhy_hod != PRAZDNY)):
                    continue # vynechaj tieto moznosti
                for novy_hod in [HLAVA, ZNAK]:
                    d.edge(
                        prvy_hod + druhy_hod + treti_hod,
                        druhy_hod + treti_hod + novy_hod,
                        color=FARBY[novy_hod]
                    )
    d.node(HLAVA + HLAVA + ZNAK, color=COLOR_VYHRA_PRVY)
    d.node(ZNAK + HLAVA + HLAVA, color=COLOR_VYHRA_DRUHY)
    return d

def graf3_2():
    d = graphviz.Digraph(graph_attr={"rankdir": "LR"})
    d.edge(PRAZDNY + PRAZDNY + PRAZDNY, PRAZDNY + PRAZDNY + HLAVA, color=FARBY[HLAVA])
    d.edge(PRAZDNY + PRAZDNY + PRAZDNY, PRAZDNY + PRAZDNY + ZNAK, color=FARBY[ZNAK])
    d.edge(PRAZDNY + PRAZDNY + HLAVA, PRAZDNY + HLAVA + HLAVA, color=FARBY[HLAVA])
    d.edge(PRAZDNY + PRAZDNY + HLAVA, PRAZDNY + PRAZDNY + ZNAK, color=FARBY[ZNAK])
    d.edge(PRAZDNY + PRAZDNY + ZNAK, PRAZDNY + ZNAK + HLAVA, color=FARBY[HLAVA])
    d.edge(PRAZDNY + PRAZDNY + ZNAK, PRAZDNY + PRAZDNY + ZNAK, color=FARBY[ZNAK])
    d.edge(PRAZDNY + HLAVA + HLAVA, PRAZDNY + HLAVA + HLAVA, color=FARBY[HLAVA])
    d.edge(PRAZDNY + HLAVA + HLAVA, HLAVA + HLAVA + ZNAK, color=FARBY[ZNAK])
    d.edge(PRAZDNY + ZNAK + HLAVA, ZNAK + HLAVA + HLAVA, color=FARBY[HLAVA])
    d.edge(PRAZDNY + ZNAK + HLAVA, PRAZDNY + PRAZDNY + ZNAK, color=FARBY[ZNAK])
    d.node(HLAVA + HLAVA + ZNAK, color=COLOR_VYHRA_PRVY)
    d.node(ZNAK + HLAVA + HLAVA, color=COLOR_VYHRA_DRUHY)
    return d

def graf4_1():
    d = graphviz.Digraph(graph_attr={"rankdir": "LR"})
    for prvy_hod in HODY:
        for druhy_hod in HODY:
            for treti_hod in HODY:
                if (druhy_hod == PRAZDNY and prvy_hod != PRAZDNY) or (
                    treti_hod == PRAZDNY and (prvy_hod != PRAZDNY or druhy_hod != PRAZDNY)):
                    continue # vynechaj tieto moznosti
                for novy_hod in [HLAVA, ZNAK]:
                    d.edge(
                        prvy_hod + druhy_hod + treti_hod,
                        druhy_hod + treti_hod + novy_hod,
                        color=FARBY[novy_hod]
                    )
    d.node(HLAVA + ZNAK + ZNAK, color=COLOR_VYHRA_PRVY)
    d.node(ZNAK + HLAVA + ZNAK, color=COLOR_VYHRA_DRUHY)
    return d

def graf4_2():
    d = graphviz.Digraph(graph_attr={"rankdir": "LR"})
    d.edge(PRAZDNY + PRAZDNY + PRAZDNY, PRAZDNY + PRAZDNY + HLAVA, color=FARBY[HLAVA])
    d.edge(PRAZDNY + PRAZDNY + PRAZDNY, PRAZDNY + PRAZDNY + ZNAK, color=FARBY[ZNAK])
    d.edge(PRAZDNY + PRAZDNY + HLAVA, PRAZDNY + PRAZDNY + HLAVA, color=FARBY[HLAVA])
    d.edge(PRAZDNY + PRAZDNY + HLAVA, PRAZDNY + HLAVA + ZNAK, color=FARBY[ZNAK])
    d.edge(PRAZDNY + PRAZDNY + ZNAK, PRAZDNY + ZNAK + HLAVA, color=FARBY[HLAVA])
    d.edge(PRAZDNY + PRAZDNY + ZNAK, PRAZDNY + PRAZDNY + ZNAK, color=FARBY[ZNAK])
    d.edge(PRAZDNY + HLAVA + ZNAK, PRAZDNY + ZNAK + HLAVA, color=FARBY[HLAVA])
    d.edge(PRAZDNY + HLAVA + ZNAK, HLAVA + ZNAK + ZNAK, color=FARBY[ZNAK])
    d.edge(PRAZDNY + ZNAK + HLAVA, PRAZDNY + PRAZDNY + HLAVA, color=FARBY[HLAVA])
    d.edge(PRAZDNY + ZNAK + HLAVA, ZNAK + HLAVA + ZNAK, color=FARBY[ZNAK])
    d.node(HLAVA + ZNAK + ZNAK, color=COLOR_VYHRA_PRVY)
    d.node(ZNAK + HLAVA + ZNAK, color=COLOR_VYHRA_DRUHY)
    return d
```

# Úloha č. 1
Dnes sa budeme venovať nasledujúcej hre: hádžeme mincou, až kým jeden z hráčov nevyhrá. Prvý hráč vyhrá, ak padne za sebou hlava-znak. Druhý hráč vyhrá, ak padne za sebou hlava-hlava. **Ktorý hráč má väčšiu šancu, že vyhrá? O koľko presne?**

## Nápovedy

### Nápoveda 1

Podobne ako na minulej hodine, budeme hádzať mincou, až kým niečo nebude splnené. Premyslite si, ako použiť vedomosti a hlavnú techniku z minulej hodiny na takýto druh príkladu.

### Nápoveda 2

Nakreslite si všetky možné stavy a vzťahy medzi nimi do pekného grafu. Skúste z neho vymyslieť riešenie.

## Riešenie

Poriadne sa zamyslite predtým, ako budete čítať ďalej. Toto nie je kniha, ale interaktívna lekcia o tom, ako samostatne použiť matematiku na riešenie rozličných problémov.

### Odhad

Ako na minulej hodine, odhady budeme robiť rýchle. Ponúknu nám základnú intuíciu o probléme.

Z minulej hodiny vieme, že treba v priemere $4$ hody na to, aby padla hlava-znak a zároveň, že treba v priemere $6$ hody na to, aby padli dve hlavy po sebe. Prvotný odhad by teda dával zmysel, že väčšiu šancu vyhrať má hlava-znak. Nie je úplne jasné o koľko, ale pomerovo by sme čakali niečo ako $\tfrac{2}{3}$.

### Simulácia

Podobných simulácii sme už na minulej hodine niekoľko robili, nemalo by ísť o nič nové. Pozorne si ale kód prečítajte, aby ste mu dobre rozumeli a vedeli aj sami napísať niečo podobné.

```python
import random
HLAVA = "H" # uložíme si tieto dva špeciálne znaky, aby sa nám lepšie čítal kód
ZNAK = "Z"

def hadz_hlava_hlava_proti_hlava_znak():
    posledny_hod = ""
    while True:
        hod = HLAVA if random.random() < 0.5 else ZNAK
        # porovnáme minulý hod a súčasný proti hlave a znaku.
        if [posledny_hod, hod] == [HLAVA, ZNAK]:
            return 1 # vyhral prvý hráč
        # porovnáme minulý hod a súčasný proti hlave a hlave.
        if [posledny_hod, hod] == [HLAVA, HLAVA]:
            return 0 # vyhral druhý hráč
        posledny_hod = hod # ak nikto nevyhral, súčasným hodom si prepíšeme posledný
```

```python
hadz_hlava_hlava_proti_hlava_znak()
```

    1

```python
N = 10000
sum(hadz_hlava_hlava_proti_hlava_znak() for i in range(N)) / N
```

    0.5033

To je zaujímavý výsledok, keďže nám nesedí úplne s odhadom. Kde sme spravili chybu, v kóde alebo v odhade? Keď máte premyslené, môžme pokračovať k matematickému riešeniu.

### Matematika

Skúsme si zapísať túto hru pomocou našich grafov z minulej hodiny. V prvom rade si musíme premyslieť, aké sú všetky situácie, v ktorých môžme byť?
- `__`: nepadlo ešte nič, práve začíname,
- `_H`, `_Z`: sme po prvom hode a padla jedna hlava respektíve jeden znak,
- `HH`, `HZ`, `ZH`, `ZZ`: posledné dva hody boli hlava-hlava, atď..

Zakreslime si do pekného prehľadného grafu vzťahy medzi všetkými týmito možnosťami:

```python
graf1_1()
```

Oranžovou farbou je označená výherná situácia za prvého hráča a fialovou druhého hráča.

Ak sa pozrieme dosť pozorne na graf, môžme si všimnúť, že jediné šípky do výherných pozícií vedú zo stavov `_H` a `ZH`. Z oboch stavov ide práve jedna šípka k výhre prvého a práve jedna šípka k výhre druhého hráča. Šanca na výhru každého hráča by teda mala byť $\tfrac{1}{2}$.

Prišli sme k tomu ale celkom zložitou cestou, skúsme si to zjednodušiť. Z pohľadu tejto hry sú stavy `__`, `_Z` a `ZZ` úplne totožné. Nehodili sme nič dôležité pre žiadneho z hráčov. Tieto stavy si označíme ako `__`. Rovnako stavy `_H` a `ZH` sú totožné $\rightarrow$ posledná bola hlava a predtým padlo niečo nedôležité. A teraz si skúsme prekresliť graf. Ako by vyzeral?

#### Pokračovanie

```python
graf1_2()
```

Overte si, že tento nový graf zachycuje podstatu hry rovnako ako náš prvý graf. Ostávajú zachované šípky, pravdepodobnosti aj stavy.

Z tohto grafu je pekne vidieť, ako bude hra prebiehať. Budeme hádzať nedôležité veci, až kým padne prvá hlava. Potom sa ďalším hodom už rozhodne, a tento hod bude fifty-fifty, teda obaja hráči majú rovnakú šancu na výhru.

Vidíme teda, že sme sa mýlili v našom odhade. Prečo sme sa ale zmýlili?

Ak by každý hráč hádzal vlastnou mincou a tiež by sme čakali, komu padne prvému jeho postupnosť, náš odhad by bol pravdepodobne správny. Jednalo by sa totiž o **nezávislé** udalosti. V prípade našej hry sú ale hody mincou **závislé**. Dôvod, prečo v priemere padne rýchlejšie hlava-znak bol, že ak sa nám po hodení prvej hlavy nepodarí hodiť rovno znak, tak sa dostaneme do stavu, že stále máme hodenú hlavu. V našej dnešnej hre sa ale rovno dostaneme do stavu prehry, pretože sme práve hodili postupnosť hlava-hlava.

Závislé a nezávislé udalosti sú v pravdepodobnosti ťažký oriešok a my sme práve videli prvý príklad na túto tému. Budeme sa s nimi stretávať veľa, takže aj keď ste hneď nepochopili o čo ide, nebojte sa ísť ďalej. Budeme mať ešte veľa času na precvičovanie a premýšľanie.

# Úloha č.2
Hráme sa rovnakú hru, akurát prvý hráč vyhrá, ak padne znak-hlava. Druhý hráč stále vyhrá, keď padne hlava-hlava. **Zmení sa niečo? Aké sú pravdepodobnosti?**

## Riešenie

### Odhad

Náš predchádzajúci odhad bol chybný, lebo sme nerátali so závislosťou medzi postupnosťami hlava-hlava a hlava-znak. Očividne nás takéto príklady môžu všeličím prekvapiť.

Teraz ale znejú naše postupnosti dosť odlišne, tak by nám odhady z minulého príkladu možno mohli fungovať. Na prvý pohľad sa v tejto úlohe nedostaneme do podobného stavu ako `_H` a preto nevidíme *očividný* dôvod, aby to bolo zase $\tfrac{1}{2}$ a nie povedzme $\frac{2}{3}$ v prospech znak-hlava.

### Simulácia

Skúsime si to naprogramovať vo všeobecnosti pre ľubovoľné dva posledné hody.

```python
def hadz_kym_dva_posledne(hrac1, hrac2): # hráč bude charakterizovaný dvojicou vysledkov hodov
    posledny_hod = ""
    while True:
        hod = HLAVA if random.random() < 0.5 else ZNAK
        if [posledny_hod, hod] == hrac1: # ak sa posledné dva hody rovnajú hráčovi, skončime
            return 1
        if [posledny_hod, hod] == hrac2:
            return 0
        posledny_hod = hod
```

```python
hadz_kym_dva_posledne([HLAVA, HLAVA], [ZNAK, HLAVA])
```

    0

```python
N = 10000
sum(hadz_kym_dva_posledne([HLAVA, HLAVA], [ZNAK, HLAVA]) for i in range(N)) / N
```

    0.2454

Kód si môžme overiť aj tým, že dáva rovnaké výsledky pre prvý príklad ako predchádzajúci kód. Skúste si to overiť.

Pravdepodobnosť vyzerá byť okolo $\tfrac{3}{4}$, takže náš odhad bol tentoraz celkom dobrý. Teda, aspoň to tak zatiaľ vyzerá.

### Matematika

V minulom príklade sme si nakreslili všetky vzťahy medzi stavmi, môžme sa teda iba pozrieť na už hotové dielo a akurát si preznačiť víťazné vrcholy.

```python
graf2_1()
```

Tento graf je torcha komplikovanejší, ale tiež si môžme niečo všimnúť. Do stavu `HH` sa vieme dostať buď zo stavu `ZH` - čo sa nikdy nestane, lebo stavom `ZH` končíme - alebo zo stavu `_H`. Do stavu `_H` sa vieme dostať iba zo stavu `__`. Do stavu `__` sa ale nevieme dostať vôbec, iba keď v ňom začíname. Takže jediná šanca na výhru druhého hráča je, že by hodil rovno `HH` za sebou ako prvé dva hody. Všetky ostatné cesty vedú do `ZH`. Šanca na hodenie `HH` ako prvé dva hody je $\tfrac{1}{4}$ a teda šanca výhry druhého hráča je zvyšok: $\tfrac{3}{4}$.

Rovnako ako minule, aj teraz si skúsme tento graf zjednodušiť a možno si všimneme nejaký zaujímavý vzťah. Ako by v tomto prípade vyzeral zmenšený graf, bez zbytočných vrcholov?

#### Pokračovanie

Podstatné stavy budú:
- `__`: nehodili sme ešte nič podstatné
- `_H`, `_Z`: posledný hod bol hlava, respektíve znak, a hod predtým bol nezaujímavý
- `HH`, `ZH`: posledné dva hody boli výherné

V princípe sa k týmto stavom vieme dostať tak, že si napíšeme výsledné stavy a všetky stavy na ceste k nim. Ostatné nie sú zaujímavé. Hrany si vieme dokresliť tak, že si premyslíme, čo sa stane po pridaní hlavy / znaku na koniec predchádzajúceho stavu a do ktorého stavu nás to dostane.

Zmenšený graf bude vyzerať nasledovne:

```python
graf2_2()
```

Nezabudnite si zase poriadne overiť, že naozaj ide o správny graf.

Čo sa teda v tomto prípade deje? Akonáhle padne ľubovoľný znak, už sme skončili. Jeden pohľad je, že zo stavu s posledným znakom buď rovno vyhrá prvý hráč, alebo bude zase posledný znak.

Druhý pohľad je, že aby vyhral druhý hráč, potrebuje hodiť dve hlavy po sebe. V momente keď ich hodí pred nimi ale nemôže byť znak, pretože by už vyhral prvý hráč svojou postupnosťou znak-hlava. To je splnené iba ak nepadne žiaden znak.

Akokoľvek si tento graf vysvetlíme, pravdepodobnosti vyjdú tie isté - druhý hráč vyhrá, iba ak hodí po sebe dve hlavy, čo je pravdepodobnosť $\tfrac{1}{4}$.

# Úloha č. 3
Keď už sme sa rozohriali, predĺžime si postupnosti. Ako dopadne **hlava - hlava - znak** proti **znak - hlava - hlava?**

## Nápoveda

Nakresliť si celý veľký graf všetkých stavov je určite cesta, ale náročná. Navyše v ňom bude ťažké niečo vidieť pravdepodobne nič neuvidíme. Skúste si rovno nakresliť zmenšený graf, možno v ňom niečo uvidíte.

## Riešenie

### Odhad

Ak sme si správne spravili domácu úlohu z minulej hodiny, dostali sme sa k číslam, že na hodenie oboch postupností treba v priemere $8$ hodov. Na základe tohto čísla je ťažké povedať, ktorý hráč má väčšiu šancu vyhrať.

Môžme si ale všimnúť, že na chvoste postupnosti znak-hlava-hlava sú prvé dva hody postupnosti hlava-hlava-znak. Dáva teda zmysel, že by to mohlo dopadnúť podobne, ako znak-hlava proti hlava-hlava, teda tentoraz v prospech druhého hráča. Pravdepodobnostne by to mohlo byť tiež podobne.

### Simulácia

Nasimulujme si rovno všeobecne všetky súboje dĺžky $3$, obdobne ako sme to spravili v druhej úlohe. Stačia drobné úpravy na zapamätanie predposledného hodu.

```python
def hadz_kym_tri_posledne(hrac1, hrac2):
    posledny_hod = ""
    predposledny_hod = ""
    while True:
        hod = HLAVA if random.random() < 0.5 else ZNAK
        if [posledny_hod, predposledny_hod, hod] == hrac1:
            return 1
        if [posledny_hod, predposledny_hod, hod] == hrac2:
            return 0
        posledny_hod, predposledny_hod = hod, posledny_hod
```

```python
hrac1 = [HLAVA, HLAVA, ZNAK]
hrac2 = [ZNAK, HLAVA, HLAVA]
hadz_kym_tri_posledne(hrac1, hrac2)
```

    1

```python
N = 10000
sum(hadz_kym_tri_posledne(hrac1, hrac2) for i in range(N)) / N
```

    0.6659

Zase nám odhad nesedí s výsledkom zo simulácie. Skúste identifikovať chybu predtým, ako sa pustíme do matematického dokazovania.

### Matematika

Pre úplnosť sme si pre vás pripravili graf všetkých možných stavov, obdobne ako v predchádzajúcich úlohách. Je mierne väčší a menej prehľadný, ale rozhodne zaujímavý:

```python
graf3_1()
```

Pre brutálnych odporúčame skúsiť uvidieť riešenie príkladu priamo v tomto grafe.

My ostatní si to zjednodušíme a nakreslíme si zmenšený graf. Zaujímavé stavy sú `___`, `__H`, `_HH`, `HHZ`, `__Z`, `_ZH` a `ZHH`. Vzťahy medzi nimi si musíme chvíľu premyslieť, ale mali by sme sa dostať k niečomu takémuto:

```python
graf3_2()
```

Z tohto grafu je krásne vidieť, že sme v rovnakej situácii ako v druhej úlohe, iba sme za konečné pozície `ZH` a `HH` pridali ešte jeden vrchol, do ktorého sa ináč dostať nedá. Pravdepodobnosť teda bude tiež rovnaká a druhý hráč vyhráva s šancou $\tfrac{3}{4}$.

Kde sme teda mali chybu? Očividne v kóde, pretože odhad nám dával rovnaký výsledok. Alebo v matematike aj odhade. Nájdenie chýb a ich opravenie už necháme na vás.

# Úloha č. 4
Záverečná úloha z hodiny bude zrátať pravdepodobnosti výhry hráča s postupnosťou **hlava-znak-znak** proti hráčovi s postupnosťou **znak-hlava-znak**.

## Riešenie

### Odhad

V domácej úlohe z minulej hodiny sme si zrátali, že priemerný počet hodov na hodenie postupnosti hlava-znak-znak je $8$ hodov a postupnosti znak-hlava-znak je $10$. To nám napovedá, že prvý hráč by mohol mať nejakú výhodu.

Na druhú stranu zase vidíme, že posledné dva hody druhého hráča sú rovnaké ako prvé dva hody prvého hráča. V predchádzajúcich dvoch úlohách to malo silný význam, mohlo by to mať význam aj teraz.

Máme teda situáciu, keď majú obaja hráči navrch z nejakého dôvodu. Ktorý z dôvodov preváži, si môžme len tipnúť. Nemala by nás ale prekvapiť ani možná remíza.

### Simulácia

Použijeme opravený kód z tretej úlohy, dúfame, že ste prišli na rovnakú chybu.

```python
def hadz_kym_tri_posledne_spravne(hrac1, hrac2):
    posledny_hod = ""
    predposledny_hod = ""
    while True:
        hod = HLAVA if random.random() < 0.5 else ZNAK
        if [posledny_hod, predposledny_hod, hod] == hrac1:
            return 1
        if [posledny_hod, predposledny_hod, hod] == hrac2:
            return 0
        posledny_hod, predposledny_hod = predposledny_hod, hod
```

```python
hrac1 = [HLAVA, ZNAK, ZNAK]
hrac2 = [ZNAK, HLAVA, ZNAK]

hadz_kym_tri_posledne_spravne(hrac1, hrac2)
```

    0

```python
N = 10000
sum(hadz_kym_tri_posledne_spravne(hrac1, hrac2) for i in range(N)) / N
```

    0.4964

Tak to naozaj vyzerá na vyrovnané šance. Pozrime sa ešte, čo na to hovorí matematika.

### Matematika

Ako v minulom príklade, prikladáme úplny graf pre brutálnych. Odkomentujte (odstráňte mriežku zo začiatku) ďalšieho riadku a spustite pre zobrazenie celého grafu. Držím vám brutálnym palce pri hľadaní riešenia vo veľkom grafe.

```python
# graf4_1()
```

My ostatní sa pozrieme na zmenšený graf. Vytvorili sme ho úplne rovnakým spôsobom ako predchádzajúci:

```python
graf4_2()
```

Tento graf je o kus komplikovanejší ako naše predchádzajúce. V tomto sa nachádzajú cykly a nevidíme žiaden priamočiary spôsob, ako spočítať pravdepodobnosti. Z každého vrchola okrem koncových existuje šanca, že vyhrá každý z hráčov. Ako to teda spočítame? Necháme vám teraz s pekným obrázkom priestor na zamyslenie.

#### Nápoveda

Spomeňte si na hlavný trik z minulej hodiny. Vedeli by sme ho využiť?

#### Riešenie

Na minulej hodine sme si zvykli označiť v podobných situáciách niektoré hodnoty a potom s nimi počítať. Skúsime to aj teraz.

Označme si pravdepodobnosť výhry prvého hráča v stave `_HZ` ako $p$. Zo vzťahov vidíme, že pravdepodobnosť bude rovnaká aj v stave `__H`, teda tiež $p$.

Šanca výhry v stave `_ZH` bude polovica šance výhry v stave `ZHZ` sčítaná s polovicou šance výhry v stave `__H`. Šanca výhry v stave `ZHZ` je jednoducho $0$, lebo už vyhral druhý hráč. Šanca v stave `_ZH` bude teda $\tfrac{1}{2}p$.

Teraz už vieme vypočítať aj `_HZ`. Keďže stav `HZZ` je rovný 1, tak stav `_HZ` vypočítame obdobne ako predchádzajúci, teda ako $\tfrac{1}{2} + \frac{1}{4}p$. Zároveň je rovný $p$, lebo tak sme ho označili na začiatku. Ak dáme tieto dva výsledky do rovnosti, dostávame $p = \tfrac{1}{2} + \tfrac{1}{4}p$ a vypočítame $p = \tfrac{2}{3}.$

Stále nám ešte chýba vypočítať pravdepodobnosť výhry zo začiatočnej pozície. Na to nám ešte chýba pravdepodobnosť v stave `__Z`, ktorá je ale rovnaká ako v stave `_ZH`, teda $\tfrac{1}{2}p$, čo je $\tfrac{1}{3}$.

Pravdepodobnosť v začiatku si teda môžme vypočítať zo stavov `__H` a `__Z` a dostávame $\tfrac{1}{2}\cdot\tfrac{1}{3} + \tfrac{1}{2}\cdot\tfrac{2}{3} = \tfrac{1}{2}$

V predchádzajúcej časti sme spravili veľa výpočtov v poradí, ktoré môže vyzerať aj ako náhodné. Ak ste zvládli tento postup sledovať, super. Ale je celkom pochopiteľné, ak ste sa v tom náhodou stratili. Niektorí majú radšej systematickejší postup, ktorý si ukážeme ako ďalší.

#### Riešenie 2

Označme si pravdepodobnosti výhry prvého hráča vo vrcholoch ako $P(x)$. Napríklad vieme, že

$$P(\mathrm{HZZ}) \quad=\quad 1  \qquad\qquad\mathrm{a}\qquad\qquad P(\mathrm{ZHZ}) \quad=\quad 0,$$

pretože prvý hráč vyhral v situácii `HZZ` a prehral v situácii `ZHZ`.

Napíšme si teraz šípky ako rovnice medzi týmito pravdepodobnosťami. Prepisovaním šípok zhora dole dostávame

\begin{align}
P(\mathrm{\_HZ}) \quad&=\quad \tfrac{1}{2}P(\mathrm{HZZ}) + \tfrac{1}{2} P(\mathrm{\_ZH}) \quad=\quad \tfrac{1}{2} + \tfrac{1}{2}P(\mathrm{\_ZH})\\
P(\mathrm{\_\_H}) \quad&=\quad P(\mathrm{\_HZ}) \\
P(\mathrm{\_ZH}) \quad&=\quad = \tfrac{1}{2} P(\mathrm{ZHZ}) + \tfrac{1}{2} P(\mathrm{\_\_H}) \quad=\quad \tfrac{1}{2} P(\mathrm{\_\_H}) \\
P(\mathrm{\_\_\_}) \quad&=\quad \tfrac{1}{2}P(\mathrm{\_\_H}) + \tfrac{1}{2}P(\mathrm{\_\_Z}) \\
P(\mathrm{\_\_Z}) \quad&=\quad P(\mathrm{\_ZH})
\end{align}

Z tejto sústavy rovníc už vieme dorátať výsledok, úplne priamo. Môžte si to skúsiť.

Týmto sme ukončili ukážkové úlohy. Posledná bola trocha náročnejšia, ale dúfame, že ste sa ňou zvládli prekúsať.

# Domáca úloha č.1
Hádžeme mincou. Prvý hrač vyhrá ak padne po sebe 10 hláv. Akú postupnosť 10 hodov si proti nemu má vybrať druhý hráč, aby mal čo najväčšiu šancu vyhrať? Aká je to šanca?

# Domáca úloha č.2
Hádžeme kockou namiesto mince. Prvý hráč vyhrá keď padne postupne $2$, $1$, $5$. Akú postupnosť troch čísel si má vybrať druhý hráč, aby mal čo najväčšiu šancu vyhrať? Aká je to šanca?

# Úloha pre praktikov
Vymyslite si úlohu, ktorú viete vyriešiť pomocou svojich nových vedomostí.

# Úloha pre brutálnych
Ste prvý hráč našej dnešnej hry. Druhý hráč si vyberie náhodne zo všetkých 8 možných postupností dĺžky 3. Akú postupnosť dĺžky 3 si vyberiete, aby ste mali v priemere čo najväčšiu šancu vyhrať?

# Úloha pre akademikov
Hádžeme mincou. Prvý hráč si vyberie postupnosť dĺžky $n$ výsledkov hodov mincou a prezradí nám ju. Akú postupnosť dĺžky $n$ si mám vybrať ako druhý hráč, aby som mal proti nemu čo najväčšiu šancu vyhrať?


# Circuit generation: tket example

This notebook will provide a brief introduction to some of the more advanced methods of circuit generation available in `pytket`, including:

* how to address wires and registers;

* reading in circuits from QASM and Quipper ASCII files;

* various types of 'boxes';

* composition of circuits (both 'horizontally' and 'vertically');

* use of symbolic gate parameters;

* representation of classically controlled gates.

## Wires, unit IDs and registers

Let's get started by constructing a circuit with 3 qubits and 2 classical bits:

```python
from pytket.circuit import Circuit
```

```python
c = Circuit(3, 2)
print(c.qubits)
print(c.bits)
```

The qubits have automatically been assigned to a register with name `q` and indices 0, 1 and 2, while the bits have been assigned to a register with name `c` and indices 0 and 1.

We can give these units arbitrary names and indices of arbitrary dimension:

```python
from pytket.circuit import Qubit
```

```python
new_q1 = Qubit("alpha", 0)
new_q2 = Qubit("beta", 2, 1)
new_q3 = Qubit("gamma", (0, 0, 0))
c.add_qubit(new_q1)
c.add_qubit(new_q2)
c.add_qubit(new_q3)
print(c.qubits)
```

We can also add a new register of qubits in one go:

```python
c.add_q_register("delta", 4)
print(c.qubits)
```

Similar commands are available for classical bits.

We can add gates to the circuit as follows:

```python
c.CX(0, 1)
```

This command appends a CX gate with control `q[0]` and target `q[1]`. Note that the integer arguments are automatically converted to the default unit IDs. For simple circuits it is often easiest to stick to the default register and refer to the qubits by integers. To add gates to our own named units, we simply pass the `Qubit` (or classical `Bit`) as an argument. (We can't mix the two conventions in one command, however.)

```python
c.H(new_q1)
c.CX(Qubit("q", 1), new_q2)
c.Rz(0.5, new_q2)
print(c.get_commands())
```

Let's have a look at our circuit:

```python
print(c.get_commands())
```

## Exporting to and importing from standard formats

We can export a `Circuit` to a file in QASM format. Conversely, if we have such a file we can import it into `pytket`. There are some limitations on the circuits that can be converted: for example, multi-dimensional indices (as in `beta` and `gamma` above) are not allowed.

Here is a simple example:

```python
from pytket.qasm import circuit_to_qasm, circuit_from_qasm
```

```python
c = Circuit(3, 1)
c.H(0)
c.CX(0, 1)
c.CX(1, 2)
c.Rz(0.25, 2)
c.Measure(2, 0)
```

```python
qasmfile = "c.qasm"
circuit_to_qasm(c, qasmfile)
```

```python
with open(qasmfile) as f:
    print(f.read())
```

```python
c1 = circuit_from_qasm(qasmfile)
c == c1
```

We can also import files in the Quipper ASCII format:

```python
from pytket.quipper import circuit_from_quipper
```

```python
quipfile = "c.quip"
with open(quipfile, "w") as f:
    f.write(
        """Inputs: 0:Qbit, 1:Qbit
QGate["W"](0,1)
QGate["omega"](1)
QGate["swap"](0,1)
QGate["W"]*(1,0)
Outputs: 0:Qbit, 1:Qbit
"""
    )
```

```python
c = circuit_from_quipper(quipfile)
print(c.get_commands())
```

Note that the Quipper gates that are not supported directly in `pytket` (`W` and `omega`) are translated into equivalent sequences of `pytket` gates.

Quipper subroutines are also supported, corresponding to `CircBox` operations in `pytket`:

```python
with open(quipfile, "w") as f:
    f.write(
        """Inputs: 0:Qbit, 1:Qbit, 2:Qbit
QGate["H"](0)
Subroutine(x2)["sub", shape "([Q,Q],())"] (2,1) -> (2,1)
QGate["H"](1)
Outputs: 0:Qbit, 1:Qbit, 2:Qbit
```

```python
Subroutine: "sub"
Shape: "([Q,Q],())"
Controllable: no
Inputs: 0:Qbit, 1:Qbit
QGate["Y"](0)
QGate["not"](1) with controls=[+0]
QGate["Z"](1)
Outputs: 0:Qbit, 1:Qbit
"""
    )
```

```python
c = circuit_from_quipper(quipfile)
cmds = c.get_commands()
print(cmds)
```

## Boxes

The `CircBox` is an example of a `pytket` 'box', which is a reusable encapsulation of a circuit inside another. We can recover the circuit 'inside' the box using the `get_circuit()` method:

```python
boxed_circuit = cmds[1].op.get_circuit()
print(boxed_circuit.get_commands())
```

The `CircBox` is the most general type of box, implementing an arbitrary circuit. But `pytket` supports several other useful box types:

* `Unitary1qBox` (implementing an arbitrary $2 \times 2$ unitary matrix);

* `Unitary2qBox` (implementing an arbitrary $4 \times 4$ unitary matrix);

* `ExpBox` (implementing $e^{itA}$ for an arbitrary $4 \times 4$ hermitian matrix $A$ and parameter $t$);

* `PauliExpBox` (implementing $e^{-\frac{1}{2} i \pi t (\sigma_0 \otimes \sigma_1 \otimes \cdots)}$ for arbitrary Pauli operators $\sigma_i \in \{\mathrm{I}, \mathrm{X}, \mathrm{Y}, \mathrm{Z}\}$ and parameter $t$).

An example will illustrate how these various box types are added to a circuit:

```python
from pytket.circuit import CircBox, Unitary1qBox, Unitary2qBox, ExpBox, PauliExpBox
from pytket.pauli import Pauli
import numpy as np
from math import sqrt
```

```python
boxycirc = Circuit(3)
```

Add a `CircBox`:

```python
subcirc = Circuit(2)
subcirc.X(0).Y(1).CZ(0, 1)
cbox = CircBox(subcirc)
boxycirc.add_circbox(cbox, args=[Qubit(0), Qubit(1)])
```

Add a `Unitary1qBox`:

```python
m1 = np.asarray([[1 / 2, sqrt(3) / 2], [sqrt(3) / 2, -1 / 2]])
m1box = Unitary1qBox(m1)
boxycirc.add_unitary1qbox(m1box, 2)
```

Add a `Unitary2qBox`:

```python
m2 = np.asarray([[0, 0, 1, 0], [0, 1, 0, 0], [0, 0, 0, 1], [1, 0, 0, 0]])
m2box = Unitary2qBox(m2)
boxycirc.add_unitary2qbox(m2box, 1, 2)
```

Add an `ExpBox`:

```python
A = np.asarray(
    [[1, 2, 3, 4 + 1j], [2, 0, 1j, -1], [3, -1j, 2, 1j], [4 - 1j, -1, -1j, 1]]
)
ebox = ExpBox(A, 0.5)
boxycirc.add_expbox(ebox, 0, 1)
```

Add a `PauliExpBox`:

```python
pbox = PauliExpBox([Pauli.X, Pauli.Z, Pauli.X], 0.75)
boxycirc.add_pauliexpbox(pbox, [0, 1, 2])
```

```python
print(boxycirc.get_commands())
```

The `get_circuit()` method is available for all box types, and returns a `Circuit` object. For example:

```python
print(pbox.get_circuit().get_commands())
```

## Circuit composition

Circuits can be composed either serially, whereby wires are joined together, or in parallel, using the `append()` command.

For a simple illustration of serial composition, let's create two circuits with compatible set of wires, and append the second to the first:

```python
c = Circuit(2)
c.CX(0, 1)
```

```python
c1 = Circuit(2)
c1.CZ(1, 0)
```

```python
c.append(c1)
```

```python
print(c.get_commands())
```

In the above example, there was a one-to-one match between the unit IDs in the two circuits, and they were matched up accordingly. The same applied with named unit IDs:

```python
x, y = Qubit("x"), Qubit("y")
```

```python
c = Circuit()
c.add_qubit(x)
c.add_qubit(y)
c.CX(x, y)
```

```python
c1 = Circuit()
c1.add_qubit(x)
c1.add_qubit(y)
c1.CZ(y, x)
```

```python
c.append(c1)
```

```python
print(c.get_commands())
```

If either circuit contains wires not matching any wires in the other, those are added to the other circuit before composition:

```python
z = Qubit("z")
c1.add_qubit(z)
c1.CY(y, z)
c.append(c1)
print(c.qubits)
print(c.get_commands())
```

If the sets of unit IDs for the two circuits are disjoint, then the composition is entirely parallel.

What if we want to serially compose two circuits having different sets of `Qubit`? In that case, we can use the `rename_units()` method on one or other of them to bring them into line. This method takes a dictionary mapping current unit IDs to new one:

```python
c2 = Circuit()
c2.add_q_register("w", 3)
w = [Qubit("w", i) for i in range(3)]
c2.H(w[0]).CX(w[0], w[1]).CRz(0.25, w[1], w[2])
```

```python
c.rename_units({x: w[0], y: w[1], z: w[2]})
```

```python
c.append(c2)
```

```python
print(c.get_commands())
```

## Symbolic parameters

Many of the gates supported by `pytket` are parametrized by one or more phase parameters, which represent rotations in multiples of $\pi$. For example, $\mathrm{Rz}(\frac{1}{2})$ represents a quarter turn, i.e. a rotation of $\pi/2$, about the Z axis. If we know the values of these parameters we can add the gates directly to our circuit:

```python
c = Circuit(1)
c.Rz(0.5, 0)
```

However, we may wish to construct and manipulate circuits containing such parametrized gates without specifying the values. This allows us to do calculations in a general setting, only later substituting values for the parameters.

Thus `pytket` allows us to specify any of the parameters as symbols. All manipulations (such as combination and cancellation of gates) are performed on the symbolic representation:

```python
from sympy import Symbol
```

```python
a = Symbol("a")
c.Rz(a, 0)
```

```python
print(c.get_commands())
```

When we apply any transformation to this circuit, the symbolic parameter is preserved in the result:

```python
from pytket.transform import Transform
```

```python
Transform.RemoveRedundancies().apply(c)
```

```python
print(c.get_commands())
```

To substitute values for symbols, we use the `symbol_substitution()` method, supplying a dictionary from symbols to values:

```python
c.symbol_substitution({a: 0.75})
```

```python
print(c.get_commands())
```

We can also substitute symbols for other symbols:

```python
b = Symbol("b")
c = Circuit(1)
c.Rz(a + b, 0)
c.symbol_substitution({b: 2 * a})
print(c.get_commands())
```

## Custom gates

We can define custom parametrized gates in `pytket` by first setting up a circuit containing symbolic parameters and then converting this to a parametrized operation type:

```python
from pytket.circuit import CustomGateDef
```

```python
a = Symbol("a")
b = Symbol("b")
setup = Circuit(3)
setup.CX(0, 1)
setup.Rz(a + 0.5, 2)
setup.CRz(b, 0, 2)
my_gate = CustomGateDef.define("g", setup, [a, b])
c = Circuit(4)
c.add_custom_gate(my_gate, [0.2, 1.3], [0, 3, 1])
print(c.get_commands())
```

Custom gates can also receive symbolic parameters:

```python
x = Symbol("x")
c.add_custom_gate(my_gate, [x, 1.0], [0, 1, 2])
print(c.get_commands())
```

## Decomposing boxes and custom gates

Having defined a circuit containing custom gates, we may now want to decompose it into elementary gates. The `DecomposeBoxes()` transform allows us to do this:

```python
Transform.DecomposeBoxes().apply(c)
print(c.get_commands())
```

The same transform works on circuits composed of arbitrary boxes. Let's try it on a copy of the circuit we built up earlier out of various box types.

```python
c = boxycirc.copy()
Transform.DecomposeBoxes().apply(c)
print(c.get_commands())
```

Note that the unitaries have been decomposed into elementary gates.

## Classical controls

Most of the examples above involve only pure quantum gates. However, `pytket` can also represent gates whose operation is conditional on one or more classical inputs.

For example, suppose we want to run the complex circuit `c` we've just constructed, then measure qubits 0 and 1, and finally apply an $\mathrm{Rz}(\frac{1}{2})$ rotation to qubit 2 if and only if the measurements were 0 and 1 respectively.

First, we'll add two classical wires to the circuit to store the measurement results:

```python
from pytket.circuit import Bit
```

```python
c.add_c_register("m", 2)
m = [Bit("m", i) for i in range(2)]
```

Classically conditioned operations depend on all their inputs being 1. Since we want to condition on `m[0]` being 0, we must first apply an X gate to its qubit, and then measure:

```python
q = [Qubit("q", i) for i in range(3)]
c.X(q[0])
c.Measure(q[0], m[0])
c.Measure(q[1], m[1])
```

Finally we add the classically conditioned Rz operation, using the `add_gate()` method:

```python
from pytket.circuit import OpType
```

```python
c.add_gate(OpType.Rz, [0.5], [q[2]], condition_bits=[m[0], m[1]], condition_value=3)
```

Note that many of the transforms and compilation passes will not accept circuits that contain classical controls.


```python
%matplotlib inline
```

Train a Mario-playing RL Agent
================

Authors: `Yuansong Feng <https://github.com/YuansongFeng>`__, `Suraj
Subramanian <https://github.com/suraj813>`__, `Howard
Wang <https://github.com/hw26>`__, `Steven
Guo <https://github.com/GuoYuzhang>`__.

This tutorial walks you through the fundamentals of Deep Reinforcement
Learning. At the end, you will implement an AI-powered Mario (using
`Double Deep Q-Networks <https://arxiv.org/pdf/1509.06461.pdf>`__) that
can play the game by itself.

Although no prior knowledge of RL is necessary for this tutorial, you
can familiarize yourself with these RL
`concepts <https://spinningup.openai.com/en/latest/spinningup/rl_intro.html>`__,
and have this handy
`cheatsheet <https://colab.research.google.com/drive/1eN33dPVtdPViiS1njTW_-r-IYCDTFU7N>`__
as your companion. The full code is available
`here <https://github.com/yuansongFeng/MadMario/>`__.

.. figure:: /_static/img/mario.gif
   :alt: mario

```python
# !pip install gym-super-mario-bros==7.3.0

import torch
from torch import nn
from torchvision import transforms as T
from PIL import Image
import numpy as np
from pathlib import Path
from collections import deque
import random, datetime, os, copy

# Gym is an OpenAI toolkit for RL
import gym
from gym.spaces import Box
from gym.wrappers import FrameStack

# NES Emulator for OpenAI Gym
from nes_py.wrappers import JoypadSpace

# Super Mario environment for OpenAI Gym
import gym_super_mario_bros
```

RL Definitions
""""""""""""""""""

**Environment** The world that an agent interacts with and learns from.

**Action** $a$ : How the Agent responds to the Environment. The
set of all possible Actions is called *action-space*.

**State** $s$ : The current characteristic of the Environment. The
set of all possible States the Environment can be in is called
*state-space*.

**Reward** $r$ : Reward is the key feedback from Environment to
Agent. It is what drives the Agent to learn and to change its future
action. An aggregation of rewards over multiple time steps is called
**Return**.

**Optimal Action-Value function** $Q^*(s,a)$ : Gives the expected
return if you start in state $s$, take an arbitrary action
$a$, and then for each future time step take the action that
maximizes returns. $Q$ can be said to stand for the “quality” of
the action in a state. We try to approximate this function.

Environment
""""""""""""""""

Initialize Environment
------------------------

In Mario, the environment consists of tubes, mushrooms and other
components.

When Mario makes an action, the environment responds with the changed
(next) state, reward and other info.

```python
# Initialize Super Mario environment
env = gym_super_mario_bros.make("SuperMarioBros-1-1-v0")

# Limit the action-space to
#   0. walk right
#   1. jump right
env = JoypadSpace(env, [["right"], ["right", "A"]])

env.reset()
next_state, reward, done, info = env.step(action=0)
print(f"{next_state.shape},\n {reward},\n {done},\n {info}")
```

Preprocess Environment
------------------------

Environment data is returned to the agent in ``next_state``. As you saw
above, each state is represented by a ``[3, 240, 256]`` size array.
Often that is more information than our agent needs; for instance,
Mario’s actions do not depend on the color of the pipes or the sky!

We use **Wrappers** to preprocess environment data before sending it to
the agent.

``GrayScaleObservation`` is a common wrapper to transform an RGB image
to grayscale; doing so reduces the size of the state representation
without losing useful information. Now the size of each state:
``[1, 240, 256]``

``ResizeObservation`` downsamples each observation into a square image.
New size: ``[1, 84, 84]``

``SkipFrame`` is a custom wrapper that inherits from ``gym.Wrapper`` and
implements the ``step()`` function. Because consecutive frames don’t
vary much, we can skip n-intermediate frames without losing much
information. The n-th frame aggregates rewards accumulated over each
skipped frame.

``FrameStack`` is a wrapper that allows us to squash consecutive frames
of the environment into a single observation point to feed to our
learning model. This way, we can identify if Mario was landing or
jumping based on the direction of his movement in the previous several
frames.

```python
class SkipFrame(gym.Wrapper):
    def __init__(self, env, skip):
        """Return only every `skip`-th frame"""
        super().__init__(env)
        self._skip = skip

    def step(self, action):
        """Repeat action, and sum reward"""
        total_reward = 0.0
        done = False
        for i in range(self._skip):
            # Accumulate reward and repeat the same action
            obs, reward, done, info = self.env.step(action)
            total_reward += reward
            if done:
                break
        return obs, total_reward, done, info

class GrayScaleObservation(gym.ObservationWrapper):
    def __init__(self, env):
        super().__init__(env)
        obs_shape = self.observation_space.shape[:2]
        self.observation_space = Box(low=0, high=255, shape=obs_shape, dtype=np.uint8)

    def permute_orientation(self, observation):
        # permute [H, W, C] array to [C, H, W] tensor
        observation = np.transpose(observation, (2, 0, 1))
        observation = torch.tensor(observation.copy(), dtype=torch.float)
        return observation

    def observation(self, observation):
        observation = self.permute_orientation(observation)
        transform = T.Grayscale()
        observation = transform(observation)
        return observation

class ResizeObservation(gym.ObservationWrapper):
    def __init__(self, env, shape):
        super().__init__(env)
        if isinstance(shape, int):
            self.shape = (shape, shape)
        else:
            self.shape = tuple(shape)

        obs_shape = self.shape + self.observation_space.shape[2:]
        self.observation_space = Box(low=0, high=255, shape=obs_shape, dtype=np.uint8)

    def observation(self, observation):
        transforms = T.Compose(
            [T.Resize(self.shape), T.Normalize(0, 255)]
        )
        observation = transforms(observation).squeeze(0)
        return observation

# Apply Wrappers to environment
env = SkipFrame(env, skip=4)
env = GrayScaleObservation(env)
env = ResizeObservation(env, shape=84)
env = FrameStack(env, num_stack=4)
```

After applying the above wrappers to the environment, the final wrapped
state consists of 4 gray-scaled consecutive frames stacked together, as
shown above in the image on the left. Each time Mario makes an action,
the environment responds with a state of this structure. The structure
is represented by a 3-D array of size ``[4, 84, 84]``.

.. figure:: /_static/img/mario_env.png
   :alt: picture

Agent
"""""""""

We create a class ``Mario`` to represent our agent in the game. Mario
should be able to:

-  **Act** according to the optimal action policy based on the current
   state (of the environment).

-  **Remember** experiences. Experience = (current state, current
   action, reward, next state). Mario *caches* and later *recalls* his
   experiences to update his action policy.

-  **Learn** a better action policy over time

```python
class Mario:
    def __init__():
        pass

    def act(self, state):
        """Given a state, choose an epsilon-greedy action"""
        pass

    def cache(self, experience):
        """Add the experience to memory"""
        pass

    def recall(self):
        """Sample experiences from memory"""
        pass

    def learn(self):
        """Update online action value (Q) function with a batch of experiences"""
        pass
```

In the following sections, we will populate Mario’s parameters and
define his functions.

Act
--------------

For any given state, an agent can choose to do the most optimal action
(**exploit**) or a random action (**explore**).

Mario randomly explores with a chance of ``self.exploration_rate``; when
he chooses to exploit, he relies on ``MarioNet`` (implemented in
``Learn`` section) to provide the most optimal action.

```python
class Mario:
    def __init__(self, state_dim, action_dim, save_dir):
        self.state_dim = state_dim
        self.action_dim = action_dim
        self.save_dir = save_dir

        self.use_cuda = torch.cuda.is_available()

        # Mario's DNN to predict the most optimal action - we implement this in the Learn section
        self.net = MarioNet(self.state_dim, self.action_dim).float()
        if self.use_cuda:
            self.net = self.net.to(device="cuda")

        self.exploration_rate = 1
        self.exploration_rate_decay = 0.99999975
        self.exploration_rate_min = 0.1
        self.curr_step = 0

        self.save_every = 5e5  # no. of experiences between saving Mario Net

    def act(self, state):
        """
    Given a state, choose an epsilon-greedy action and update value of step.

    Inputs:
    state(LazyFrame): A single observation of the current state, dimension is (state_dim)
    Outputs:
    action_idx (int): An integer representing which action Mario will perform
    """
        # EXPLORE
        if np.random.rand() < self.exploration_rate:
            action_idx = np.random.randint(self.action_dim)

        # EXPLOIT
        else:
            state = state.__array__()
            if self.use_cuda:
                state = torch.tensor(state).cuda()
            else:
                state = torch.tensor(state)
            state = state.unsqueeze(0)
            action_values = self.net(state, model="online")
            action_idx = torch.argmax(action_values, axis=1).item()

        # decrease exploration_rate
        self.exploration_rate *= self.exploration_rate_decay
        self.exploration_rate = max(self.exploration_rate_min, self.exploration_rate)

        # increment step
        self.curr_step += 1
        return action_idx
```

Cache and Recall
----------------------

These two functions serve as Mario’s “memory” process.

``cache()``: Each time Mario performs an action, he stores the
``experience`` to his memory. His experience includes the current
*state*, *action* performed, *reward* from the action, the *next state*,
and whether the game is *done*.

``recall()``: Mario randomly samples a batch of experiences from his
memory, and uses that to learn the game.

```python
class Mario(Mario):  # subclassing for continuity
    def __init__(self, state_dim, action_dim, save_dir):
        super().__init__(state_dim, action_dim, save_dir)
        self.memory = deque(maxlen=100000)
        self.batch_size = 32

    def cache(self, state, next_state, action, reward, done):
        """
        Store the experience to self.memory (replay buffer)

        Inputs:
        state (LazyFrame),
        next_state (LazyFrame),
        action (int),
        reward (float),
        done(bool))
        """
        state = state.__array__()
        next_state = next_state.__array__()

        if self.use_cuda:
            state = torch.tensor(state).cuda()
            next_state = torch.tensor(next_state).cuda()
            action = torch.tensor([action]).cuda()
            reward = torch.tensor([reward]).cuda()
            done = torch.tensor([done]).cuda()
        else:
            state = torch.tensor(state)
            next_state = torch.tensor(next_state)
            action = torch.tensor([action])
            reward = torch.tensor([reward])
            done = torch.tensor([done])

        self.memory.append((state, next_state, action, reward, done,))

    def recall(self):
        """
        Retrieve a batch of experiences from memory
        """
        batch = random.sample(self.memory, self.batch_size)
        state, next_state, action, reward, done = map(torch.stack, zip(*batch))
        return state, next_state, action.squeeze(), reward.squeeze(), done.squeeze()
```

Learn
--------------

Mario uses the `DDQN algorithm <https://arxiv.org/pdf/1509.06461>`__
under the hood. DDQN uses two ConvNets - $Q_{online}$ and
$Q_{target}$ - that independently approximate the optimal
action-value function.

In our implementation, we share feature generator ``features`` across
$Q_{online}$ and $Q_{target}$, but maintain separate FC
classifiers for each. $\theta_{target}$ (the parameters of
$Q_{target}$) is frozen to prevent updation by backprop. Instead,
it is periodically synced with $\theta_{online}$ (more on this
later).

Neural Network
~~~~~~~~~~~~~~~~~~

```python
class MarioNet(nn.Module):
    """mini cnn structure
  input -> (conv2d + relu) x 3 -> flatten -> (dense + relu) x 2 -> output
  """

    def __init__(self, input_dim, output_dim):
        super().__init__()
        c, h, w = input_dim

        if h != 84:
            raise ValueError(f"Expecting input height: 84, got: {h}")
        if w != 84:
            raise ValueError(f"Expecting input width: 84, got: {w}")

        self.online = nn.Sequential(
            nn.Conv2d(in_channels=c, out_channels=32, kernel_size=8, stride=4),
            nn.ReLU(),
            nn.Conv2d(in_channels=32, out_channels=64, kernel_size=4, stride=2),
            nn.ReLU(),
            nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3, stride=1),
            nn.ReLU(),
            nn.Flatten(),
            nn.Linear(3136, 512),
            nn.ReLU(),
            nn.Linear(512, output_dim),
        )

        self.target = copy.deepcopy(self.online)

        # Q_target parameters are frozen.
        for p in self.target.parameters():
            p.requires_grad = False

    def forward(self, input, model):
        if model == "online":
            return self.online(input)
        elif model == "target":
            return self.target(input)
```

TD Estimate & TD Target
~~~~~~~~~~~~~~~~~~~~~~~~~~

Two values are involved in learning:

**TD Estimate** - the predicted optimal $Q^*$ for a given state
$s$

\begin{align}{TD}_e = Q_{online}^*(s,a)\end{align}

**TD Target** - aggregation of current reward and the estimated
$Q^*$ in the next state $s'$

\begin{align}a' = argmax_{a} Q_{online}(s', a)\end{align}

\begin{align}{TD}_t = r + \gamma Q_{target}^*(s',a')\end{align}

Because we don’t know what next action $a'$ will be, we use the
action $a'$ maximizes $Q_{online}$ in the next state
$s'$.

Notice we use the
`@torch.no_grad() <https://pytorch.org/docs/stable/generated/torch.no_grad.html#no-grad>`__
decorator on ``td_target()`` to disable gradient calculations here
(because we don’t need to backpropagate on $\theta_{target}$).

```python
class Mario(Mario):
    def __init__(self, state_dim, action_dim, save_dir):
        super().__init__(state_dim, action_dim, save_dir)
        self.gamma = 0.9

    def td_estimate(self, state, action):
        current_Q = self.net(state, model="online")[
            np.arange(0, self.batch_size), action
        ]  # Q_online(s,a)
        return current_Q

    @torch.no_grad()
    def td_target(self, reward, next_state, done):
        next_state_Q = self.net(next_state, model="online")
        best_action = torch.argmax(next_state_Q, axis=1)
        next_Q = self.net(next_state, model="target")[
            np.arange(0, self.batch_size), best_action
        ]
        return (reward + (1 - done.float()) * self.gamma * next_Q).float()
```

Updating the model
~~~~~~~~~~~~~~~~~~~~~~

As Mario samples inputs from his replay buffer, we compute $TD_t$
and $TD_e$ and backpropagate this loss down $Q_{online}$ to
update its parameters $\theta_{online}$ ($\alpha$ is the
learning rate ``lr`` passed to the ``optimizer``)

\begin{align}\theta_{online} \leftarrow \theta_{online} + \alpha \nabla(TD_e - TD_t)\end{align}

$\theta_{target}$ does not update through backpropagation.
Instead, we periodically copy $\theta_{online}$ to
$\theta_{target}$

\begin{align}\theta_{target} \leftarrow \theta_{online}\end{align}

```python
class Mario(Mario):
    def __init__(self, state_dim, action_dim, save_dir):
        super().__init__(state_dim, action_dim, save_dir)
        self.optimizer = torch.optim.Adam(self.net.parameters(), lr=0.00025)
        self.loss_fn = torch.nn.SmoothL1Loss()

    def update_Q_online(self, td_estimate, td_target):
        loss = self.loss_fn(td_estimate, td_target)
        self.optimizer.zero_grad()
        loss.backward()
        self.optimizer.step()
        return loss.item()

    def sync_Q_target(self):
        self.net.target.load_state_dict(self.net.online.state_dict())
```

Save checkpoint
~~~~~~~~~~~~~~~~~~

```python
class Mario(Mario):
    def save(self):
        save_path = (
            self.save_dir / f"mario_net_{int(self.curr_step // self.save_every)}.chkpt"
        )
        torch.save(
            dict(model=self.net.state_dict(), exploration_rate=self.exploration_rate),
            save_path,
        )
        print(f"MarioNet saved to {save_path} at step {self.curr_step}")
```

Putting it all together
~~~~~~~~~~~~~~~~~~~~~~~~~~

```python
class Mario(Mario):
    def __init__(self, state_dim, action_dim, save_dir):
        super().__init__(state_dim, action_dim, save_dir)
        self.burnin = 1e4  # min. experiences before training
        self.learn_every = 3  # no. of experiences between updates to Q_online
        self.sync_every = 1e4  # no. of experiences between Q_target & Q_online sync

    def learn(self):
        if self.curr_step % self.sync_every == 0:
            self.sync_Q_target()

        if self.curr_step % self.save_every == 0:
            self.save()

        if self.curr_step < self.burnin:
            return None, None

        if self.curr_step % self.learn_every != 0:
            return None, None

        # Sample from memory
        state, next_state, action, reward, done = self.recall()

        # Get TD Estimate
        td_est = self.td_estimate(state, action)

        # Get TD Target
        td_tgt = self.td_target(reward, next_state, done)

        # Backpropagate loss through Q_online
        loss = self.update_Q_online(td_est, td_tgt)

        return (td_est.mean().item(), loss)
```

Logging
--------------

```python
import numpy as np
import time, datetime
import matplotlib.pyplot as plt

class MetricLogger:
    def __init__(self, save_dir):
        self.save_log = save_dir / "log"
        with open(self.save_log, "w") as f:
            f.write(
                f"{'Episode':>8}{'Step':>8}{'Epsilon':>10}{'MeanReward':>15}"
                f"{'MeanLength':>15}{'MeanLoss':>15}{'MeanQValue':>15}"
                f"{'TimeDelta':>15}{'Time':>20}\n"
            )
        self.ep_rewards_plot = save_dir / "reward_plot.jpg"
        self.ep_lengths_plot = save_dir / "length_plot.jpg"
        self.ep_avg_losses_plot = save_dir / "loss_plot.jpg"
        self.ep_avg_qs_plot = save_dir / "q_plot.jpg"

        # History metrics
        self.ep_rewards = []
        self.ep_lengths = []
        self.ep_avg_losses = []
        self.ep_avg_qs = []

        # Moving averages, added for every call to record()
        self.moving_avg_ep_rewards = []
        self.moving_avg_ep_lengths = []
        self.moving_avg_ep_avg_losses = []
        self.moving_avg_ep_avg_qs = []

        # Current episode metric
        self.init_episode()

        # Timing
        self.record_time = time.time()

    def log_step(self, reward, loss, q):
        self.curr_ep_reward += reward
        self.curr_ep_length += 1
        if loss:
            self.curr_ep_loss += loss
            self.curr_ep_q += q
            self.curr_ep_loss_length += 1

    def log_episode(self):
        "Mark end of episode"
        self.ep_rewards.append(self.curr_ep_reward)
        self.ep_lengths.append(self.curr_ep_length)
        if self.curr_ep_loss_length == 0:
            ep_avg_loss = 0
            ep_avg_q = 0
        else:
            ep_avg_loss = np.round(self.curr_ep_loss / self.curr_ep_loss_length, 5)
            ep_avg_q = np.round(self.curr_ep_q / self.curr_ep_loss_length, 5)
        self.ep_avg_losses.append(ep_avg_loss)
        self.ep_avg_qs.append(ep_avg_q)

        self.init_episode()

    def init_episode(self):
        self.curr_ep_reward = 0.0
        self.curr_ep_length = 0
        self.curr_ep_loss = 0.0
        self.curr_ep_q = 0.0
        self.curr_ep_loss_length = 0

    def record(self, episode, epsilon, step):
        mean_ep_reward = np.round(np.mean(self.ep_rewards[-100:]), 3)
        mean_ep_length = np.round(np.mean(self.ep_lengths[-100:]), 3)
        mean_ep_loss = np.round(np.mean(self.ep_avg_losses[-100:]), 3)
        mean_ep_q = np.round(np.mean(self.ep_avg_qs[-100:]), 3)
        self.moving_avg_ep_rewards.append(mean_ep_reward)
        self.moving_avg_ep_lengths.append(mean_ep_length)
        self.moving_avg_ep_avg_losses.append(mean_ep_loss)
        self.moving_avg_ep_avg_qs.append(mean_ep_q)

        last_record_time = self.record_time
        self.record_time = time.time()
        time_since_last_record = np.round(self.record_time - last_record_time, 3)

        print(
            f"Episode {episode} - "
            f"Step {step} - "
            f"Epsilon {epsilon} - "
            f"Mean Reward {mean_ep_reward} - "
            f"Mean Length {mean_ep_length} - "
            f"Mean Loss {mean_ep_loss} - "
            f"Mean Q Value {mean_ep_q} - "
            f"Time Delta {time_since_last_record} - "
            f"Time {datetime.datetime.now().strftime('%Y-%m-%dT%H:%M:%S')}"
        )

        with open(self.save_log, "a") as f:
            f.write(
                f"{episode:8d}{step:8d}{epsilon:10.3f}"
                f"{mean_ep_reward:15.3f}{mean_ep_length:15.3f}{mean_ep_loss:15.3f}{mean_ep_q:15.3f}"
                f"{time_since_last_record:15.3f}"
                f"{datetime.datetime.now().strftime('%Y-%m-%dT%H:%M:%S'):>20}\n"
            )

        for metric in ["ep_rewards", "ep_lengths", "ep_avg_losses", "ep_avg_qs"]:
            plt.plot(getattr(self, f"moving_avg_{metric}"))
            plt.savefig(getattr(self, f"{metric}_plot"))
            plt.clf()
```

Let’s play!
"""""""""""""""

In this example we run the training loop for 10 episodes, but for Mario to truly learn the ways of
his world, we suggest running the loop for at least 40,000 episodes!

```python
use_cuda = torch.cuda.is_available()
print(f"Using CUDA: {use_cuda}")
print()

save_dir = Path("checkpoints") / datetime.datetime.now().strftime("%Y-%m-%dT%H-%M-%S")
save_dir.mkdir(parents=True)

mario = Mario(state_dim=(4, 84, 84), action_dim=env.action_space.n, save_dir=save_dir)

logger = MetricLogger(save_dir)

episodes = 10
for e in range(episodes):

    state = env.reset()

    # Play the game!
    while True:

        # Run agent on the state
        action = mario.act(state)

        # Agent performs action
        next_state, reward, done, info = env.step(action)

        # Remember
        mario.cache(state, next_state, action, reward, done)

        # Learn
        q, loss = mario.learn()

        # Logging
        logger.log_step(reward, loss, q)

        # Update state
        state = next_state

        # Check if end of game
        if done or info["flag_get"]:
            break

    logger.log_episode()

    if e % 20 == 0:
        logger.record(episode=e, epsilon=mario.exploration_rate, step=mario.curr_step)
```

Conclusion
"""""""""""""""

In this tutorial, we saw how we can use PyTorch to train a game-playing AI. You can use the same methods
to train an AI to play any of the games at the `OpenAI gym <https://gym.openai.com/>`__. Hope you enjoyed this tutorial, feel free to reach us at
`our github <https://github.com/yuansongFeng/MadMario/>`__!