____
__Universidad Tecnológica Nacional, Buenos Aires__

__Ingeniería Industrial__

__Cátedra de Ciencia de Datos - Curso I5521 - Turno sabado mañana__

__Elaborado por: Martin Palazzo__
____

```python
from google.colab import drive
drive.mount('/gdrive')
```

    Drive already mounted at /gdrive; to attempt to forcibly remount, call drive.mount("/gdrive", force_remount=True).

```python
DRIVE_FOLDER = 'ClusterAI2020/'
CLASS_FOLDER = 'clase_01/'
DATA_PATH = "../data/clase_01/"
%cd {'/gdrive/My Drive/'+DRIVE_FOLDER+CLASS_FOLDER}
```

    /gdrive/My Drive/ClusterAI2020/clase_01

En este script continuaremos el análisis exploratorio de datos que comenzamos la clase pasada con el dataset de subtes

# clase_01: Importar data, describirla y visualizarla.

En esta segunda clase aprenderemos a realizar correlaciones lineales de Pearson, visualizar boxplots, comprender los quantiles de un set de datos y eliminar outliers observando que quantil (percentil) nos ayuda a filtrar. Primero haremos un repaso sobre el analisis exploratorio de subtes que realizamos la semana pasada.

**Importamos las Librerías necesarias:** Utilizaremos Numpy para realizar calculos matriciales, pandas para manejar los datos y almacenarlos, y matplotlib junto con Seaborn para visualizar.

```python
# importamos las librerías necesarias para trabajar.
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
```

**Read_CSV:** Descargamos de https://data.buenosaires.gob.ar/dataset/subte-viajes-molinetes el dataset de usuarios de subte por molinete del año 2017. Luego con el comando de Pandas "pd.read_csv" indicamos donde se ubica el dataset a ser leido. Una vez leido guardamos la tabla como un DataFrame de Pandas en el elemento "molinetes".

```python
# importante detallar que el delimiter indica como estan separadas las columnas en el dataframe.
molinetes = pd.read_csv(DATA_PATH+'molinetes_2017.csv', delimiter=';')
```

con el comando .head() observamos las primeras 5 filas de nuestro dataframe

```python
# con el comando .head() podemos ver en pantalla las primeras filas y del dataframe
molinetes.head()
```

      PERIODO
      FECHA
      DESDE
      HASTA
      LINEA
      MOLINETE
      ESTACION
      PAX_PAGOS
      PAX_PASES_PAGOS
      PAX_FRANQ
      TOTAL
      ID

      0
      201701
      01/01/2017
      08:00:00
      08:15:00
      LINEA_H
      LINEA_H_CASEROS_SUR_TURN02
      CASEROS
      1
      0
      0
      1
      1

      1
      201701
      01/01/2017
      08:00:00
      08:15:00
      LINEA_H
      LINEA_H_CASEROS_NORTE_TURN02
      CASEROS
      1
      0
      0
      1
      1

      2
      201701
      01/01/2017
      08:15:00
      08:30:00
      LINEA_H
      LINEA_H_CASEROS_SUR_TURN02
      CASEROS
      2
      0
      0
      2
      1

      3
      201701
      01/01/2017
      08:15:00
      08:30:00
      LINEA_H
      LINEA_H_CASEROS_SUR_TURN01
      CASEROS
      1
      0
      0
      1
      1

      4
      201701
      01/01/2017
      08:30:00
      08:45:00
      LINEA_H
      LINEA_H_CASEROS_NORTE_TURN01
      CASEROS
      2
      0
      0
      2
      1

Recordemos que el comando de numpy np.shape() nos indica las dimensiones de nuestro elemento, en este caso un dataframe de pandas.

```python
# si queremos saber cuantos renglones y columnas tiene el objeto en cuestion, utilizamos el comando de numpy "shape".
# asi podemos ver que nuestro dataset posee mas de 11 millones de renglones y 12 columnas
np.shape(molinetes)
```

    (11938476, 12)

Imprimimos en pantalla un breve reporte de un renglón sobre las dimensiones de nuestro dataframe.

```python
# Ver que utilizamos la función de numpy "shape".
print("El dataset importado tiene "+ str(np.shape(molinetes)[0]) + " renglones y " + str(np.shape(molinetes)[1]) + " columnas")
```

    El dataset importado tiene 11938476 renglones y 12 columnas

Si bien en nuestro dataset de subtes no existen columnas que tengan valores nulos o faltantes (NaNs) es necesario entender para otros datasets si hay valores faltantes para eliminarlos (o reemplazarlos por otro valor como la mediana).

```python
# revisamos si existen columnas que contengan valores faltantes (NaNs)
col_nulls = molinetes.isnull().any()
col_nulls
```

    PERIODO            False
    FECHA              False
    DESDE              False
    HASTA              False
    LINEA              False
    MOLINETE           False
    ESTACION           False
    PAX_PAGOS          False
    PAX_PASES_PAGOS    False
    PAX_FRANQ          False
    TOTAL              False
    ID                 False
    dtype: bool

# Columnas e Indices en Pandas

**Columnas e Indices en Pandas:** los dataframes de pandas se caracterizan por tener nombres explícitos por columna y por renglón (indice). Es decir que podemos llamar a un elemento (o varios) de un DataFrame por el nombre de su columna e índice.

```python
# Obtenemos los nombres de las columnas del dataframe Molinetes
columnas_molinetes = molinetes.columns
columnas_molinetes
```

    Index(['PERIODO', 'FECHA', 'DESDE', 'HASTA', 'LINEA', 'MOLINETE', 'ESTACION',
           'PAX_PAGOS', 'PAX_PASES_PAGOS', 'PAX_FRANQ', 'TOTAL', 'ID'],
          dtype='object')

```python
# Obtenemos los nombres de los índices (renglones o instancias) del dataframe Molinetes.
# Observamos que cuando imprimimos en pantalla los índices, fueron definidos al importar el csv con el parametro
# "index_col" seleccionando la columna "PERIODO".
instancias_molinetes = molinetes.index
instancias_molinetes
```

    RangeIndex(start=0, stop=11938476, step=1)

# Filtrar un dataframe por el valor de una columna

**Filtrar un dataframe por el valor de una columna:** Inicialmente filtraremos nuestro dataframe "molinetes_b" por el valor "LINEA_B" de la columna "LINEA". Para eso usamos la funcion ".loc" de pandas, sobre el dataframe.

```python
# como inicialmente queremos trabajar con los datos de la Linea B, filtramos por el valor "LINEA_B" de la columna
# "LINEA" nuestro dataframe original llamado molinetes.
molinetes_b = molinetes.loc[molinetes['LINEA'] == 'LINEA_B']
```

```python
# observamos que la cantidad de renglones del nuevo dataframe es menor al del original, puesto que solo
# estamos considerando a los molinetes de la linea B
np.shape(molinetes_b)
```

    (2855204, 12)

```python
molinetes_b.head()
```

      PERIODO
      FECHA
      DESDE
      HASTA
      LINEA
      MOLINETE
      ESTACION
      PAX_PAGOS
      PAX_PASES_PAGOS
      PAX_FRANQ
      TOTAL
      ID

      2492243
      201701
      01/01/2017
      08:00:00
      08:15:00
      LINEA_B
      LINEA_B_FLORIDA_O_TURN01
      FLORIDA
      4
      0
      0
      4
      20

      2492244
      201701
      01/01/2017
      08:00:00
      08:15:00
      LINEA_B
      LINEA_B_FLORIDA_O_TURN03
      FLORIDA
      3
      0
      0
      3
      20

      2492245
      201701
      01/01/2017
      08:00:00
      08:15:00
      LINEA_B
      LINEA_B_FLORIDA_O_TURN02
      FLORIDA
      1
      0
      0
      1
      20

      2492246
      201701
      01/01/2017
      08:00:00
      08:15:00
      LINEA_B
      LINEA_B_FLORIDA_O_TURN04
      FLORIDA
      1
      0
      0
      1
      20

      2492247
      201701
      01/01/2017
      08:15:00
      08:30:00
      LINEA_B
      LINEA_B_FLORIDA_O_TURN01
      FLORIDA
      2
      0
      0
      2
      20

```python
print("Los registros de la linea B representan el " + str(np.round(np.shape(molinetes_b)[0]/np.shape(molinetes)[0],2)) + " de los registros totales")
```

    Los registros de la linea B representan el 0.24 de los registros totales

# Visualizacion: Countplot

Visualización Countplot de Seaborn para las distintas lineas de subte.

```python
# con el comando "countplot" de Seaborn (importado como sns) generamos un gráfico de barras por la columna "LINEA"
sns.set_context("talk")
az = sns.countplot(x="LINEA", data=molinetes)
plt.xticks(rotation=45)
plt.title("Viajes en molinetes por linea durante 2017")
plt.show()
```

Visualización de Countplot de Seaborn para las distintas estaciones de la linea B.

```python
# luego generamos un countplot de seaborn por la columna estación, esta vez utilizando el nuevo dataframe filtrado
# molinetes_b, que solo contiene los datos de la linea B.
ax = sns.countplot(x="ESTACION", data=molinetes_b)
ax.set_xticklabels(ax.get_xticklabels(), rotation=40, ha="right")
plt.title("")
plt.show()
```

# Visualizacion: Boxplot con Seaborn para linea B
Procedemos a realizar un boxplot por cada estacion, cuantificando asi donde se concentran cada lapso de tiempo en viajes de molinete. Ver que siempre la cantidad de periodos de 15 mins entre el 1er y 3er cuartil esta por debajo de 100 viajes. Es importante comprender la diferencia entre el boxplot visualizado a continuacion y el countplot visualizado en el paso anterior. ¿Que cambia entre uno y otro?

```python
ay = sns.boxplot(x = molinetes_b.ESTACION, y = molinetes_b.TOTAL, data = molinetes_b)
ay.set_xticklabels(ay.get_xticklabels(), rotation=40, ha="right")
plt.title("distribucion de cantidad de tickets en periodos de 15 mins")
plt.show()
```

# Análisis de la estación Lacroze de la linea B
Creamos un dataframe llamado "molinetes_lacroze" que contiene unicamente las instancias del dataframe "molinetes_b" cuyo valor del atributo/feature "ESTACION" sea igual a "FEDERICO LACROZE"

```python
# filtramos con ".loc" el dataframe "molinetes_b" por el valor "Federico Lacroze" de la columna "Estacion"
molinetes_lacroze = molinetes_b.loc[molinetes_b.ESTACION== 'FEDERICO LACROZE']
```

```python
molinetes_lacroze.head()
```

      PERIODO
      FECHA
      DESDE
      HASTA
      LINEA
      MOLINETE
      ESTACION
      PAX_PAGOS
      PAX_PASES_PAGOS
      PAX_FRANQ
      TOTAL
      ID

      7622359
      201701
      01/01/2017
      08:00:00
      08:15:00
      LINEA_B
      LINEA_B_LACROZE_O_TURN09
      FEDERICO LACROZE
      2
      0
      0
      2
      66

      7622360
      201701
      01/01/2017
      08:00:00
      08:15:00
      LINEA_B
      LINEA_B_LACROZE_E_TURN05
      FEDERICO LACROZE
      4
      0
      0
      4
      66

      7622361
      201701
      01/01/2017
      08:00:00
      08:15:00
      LINEA_B
      LINEA_B_LACROZE_E_TURN06
      FEDERICO LACROZE
      0
      0
      1
      1
      66

      7622362
      201701
      01/01/2017
      08:00:00
      08:15:00
      LINEA_B
      LINEA_B_LACROZE_O_TURN03
      FEDERICO LACROZE
      1
      0
      0
      1
      66

      7622363
      201701
      01/01/2017
      08:00:00
      08:15:00
      LINEA_B
      LINEA_B_LACROZE_O_TURN07
      FEDERICO LACROZE
      2
      0
      0
      2
      66

**estadisticas descriptivas para la columna "TOTAL" del dataframe "molinetes_lacroze"**

```python
molinetes_lacroze['TOTAL'].describe()
```

    count    421233.000000
    mean         19.292629
    std          25.177922
    min           0.000000
    25%           3.000000
    50%           9.000000
    75%          27.000000
    max         248.000000
    Name: TOTAL, dtype: float64

**distplot para la columna "TOTAL" del dataframe "molinetes_lacroze"**

```python
# displot para la columna "TOTAL" nos generará un histograma + la distribución generada de los datos a partir de
# sus frecuencias poblacionales.
sns.distplot(molinetes_lacroze['TOTAL'])
plt.title('Distribución de cantidad de viajes totales de molinetes. Linea B. Mes Abril.')
plt.xlabel('Cantidad de Viajes en molinete por cada 15 mins')
plt.show()
```

**Boxplot para la columna TOTAL del dataframe "molinetes_lacroze"**
Realizaremos un boxplot con Matplotlib y otro con seaborn, simplemente para aprender que existen 2 maneras de visualizar.

```python
# boxplot realizado con Matplotlib
plt.boxplot(molinetes_lacroze['TOTAL'])
plt.title('Vueltas de molinete en Lacroze en 2017')
plt.ylabel('Cant. de molinetes c/15 mins')
plt.xlabel("Viajes por molinete Lacroze")
plt.show()
```

```python
# probamos hacer el mismo boxplot con seaborn
ax = sns.boxplot(x = "TOTAL", data =  molinetes_lacroze)
plt.show()
```

# Filtrado de outliers con cuantiles
Calculamos el valor del percentil 90. Lo que este sobre el percentil 90 lo llamaremos "hora pico" y lo que este por debajo sera "hora no pico".

Con el comando de pandas .quantile() calculamos los percentiles que deseemos. En algunos casos nos puede llegar a interesar tambien observar lo que se encuentre por debajo del percentil 10, aunque como observamos en el distplot, la distribucion es asimetrica a la derecha.

```python
# calculo el percentil 90 de la columna "TOTAL" para el dataframe "molinetes_lacroze".
percentil_90 = molinetes_lacroze['TOTAL'].quantile(0.9)
```

```python
# imprimimos en pantalla cuanto vale el percentil 0.9
percentil_90
```

    50.0

Ahora filtramos por los valores de "TOTAL" que sean mayores al percentil 90 para guardarlos en otro dataframe llamado "Lacroze_hora_pico". Lo mismo hacemos para los que son menores al percentil 90 y los guardamos como "hora no pico".

```python
lacroze_hora_pico = molinetes_lacroze.loc[molinetes_lacroze['TOTAL']>percentil_90]
```

```python
lacroze_hora_nopico = molinetes_lacroze.loc[molinetes_lacroze['TOTAL']<percentil_90]
```

Observamos que la expresion **"molinetes_lacroze['TOTAL']<percentil_90"** es un vector Boolean, es decir que arroja True o False, y al filtrar utilizando un vector boolean solo sobreviven las filas que esten asociadas a un True.

```python
# observamos que cuando indicamos una expresion logica, lo que devuelve python son valores Booleanos como True or False.
(molinetes_lacroze['TOTAL']<percentil_90).head()
```

    7622359    True
    7622360    True
    7622361    True
    7622362    True
    7622363    True
    Name: TOTAL, dtype: bool

```python
# observamos que dimensiones tienen ahora los dataframes de hora pico y hora no pico
np.shape(lacroze_hora_nopico)
```

    (378809, 12)

```python
np.shape(lacroze_hora_pico)
```

    (40621, 12)

Ploteamos la distribucion / histograma de la hora pico, ver que los valores del eje X comienzan a partir de 50, que fue el umbral que seleccionamos para dividir ambas categorias.

```python
# hacemos un distplot de la columna TOTAL del dataframe "lacroze_hora_pico"
sns.distplot(lacroze_hora_pico['TOTAL'], norm_hist = False)
plt.title('Distribución de cantidad de viajes en hora pico de la estacion Lacroze')
plt.xlabel('Cantidad de Viajes en molinete por cada 15 mins')
plt.show()
```

```python
# con matplotlib hacemos un boxplot sobre la columna TOTAL del dataframe "lacroze_hora_pico"
plt.boxplot(lacroze_hora_pico['TOTAL'])
plt.title('Cant. viajes de Lacroze en 2017 Hora Pico')
plt.ylabel('Cant. de vueltas de molinete c/15 mins')
plt.xlabel("Estacion Lacroze")
plt.show()
```

Calculamos estadisticas descriptivas para la hora pico

```python
# aplicamos .describe() sobre la columna TOTAL del dataframe "lacroze_hora_pico"
lacroze_hora_pico['TOTAL'].describe()
```

    count    40621.000000
    mean        80.194604
    std         28.975506
    min         51.000000
    25%         57.000000
    50%         70.000000
    75%         96.000000
    max        248.000000
    Name: TOTAL, dtype: float64

Ahora calculamos estadisticas descriptivas y graficos para la hora no pico. **¿que diferencias hay en estas estadisticas entre hora pico y hora no pico?**

```python
# aplicamos .describe() sobre la columna TOTAL del dataframe "lacroze_hora_nopico"
lacroze_hora_nopico['TOTAL'].describe()
```

    count    378809.000000
    mean         12.615743
    std          12.964954
    min           0.000000
    25%           2.000000
    50%           7.000000
    75%          19.000000
    max          49.000000
    Name: TOTAL, dtype: float64

```python
# hacemos un distplot sobre la columna TOTAL del dataframe "lacroze_hora_nopico"
sns.distplot((lacroze_hora_nopico['TOTAL']))
plt.title('Distribución de cantidad de viajes en hora NO pico de la estacion Lacroze')
plt.xlabel('Cantidad de Viajes en molinete por cada 15 mins')
plt.show()
```

```python
# hacemos un boxplot con matplotlib sobre la columna TOTAL del dataframe "lacroze_hora_nopico"
plt.boxplot(lacroze_hora_nopico['TOTAL'])
plt.title("Dist de cant. de viajes c/15 mins en hora no pico")
plt.xlabel("Estacion Lacroze. Hora no pico")
plt.show()
```

Ahora comparamos como queda la distribucion de cantidad de viajes cada 15 mins con los datos originales, en hora pico y en hora no pico.

```python
# con matplotlib hacemos un boxplot de los 3 grupos de datos. Ver que ingresamos 3 dataframes filtrados por la columna TOTAL
plt.boxplot([molinetes_lacroze['TOTAL'],lacroze_hora_pico['TOTAL'], lacroze_hora_nopico['TOTAL']])
plt.title('comparacion de boxplots entre hora pico, no pico y pre filtrado')
plt.xlabel(['pre-filtrado', 'hora pico', 'hora no pico'])
plt.show()
```

Tambien podriamos visualizar en simultáneo las distribuciones de los viajes en horas pico y no pico.

```python
# hacemos dos displots en el mismo grafico. Uno para hora pico y otro para hora no pico.
sns.distplot((lacroze_hora_pico['TOTAL']))
sns.distplot((lacroze_hora_nopico['TOTAL']))
plt.title('Distribución de cantidad de viajes en hora NO pico de la estacion Lacroze')
plt.xlabel('Cantidad de Viajes en molinete por cada 15 mins')
plt.show()
```

# Correlación Lineal entre estaciones Linea C

\begin{equation}
r = \frac{\sum_{i}^{n}\left ( x_i - \overline{x} \right )\left ( y_i - \overline{y}  \right )}{\left [ \sum_{i}^{n} \left ( x_i - \overline{x}  \right )^{2} \left ( y_i - \overline{y} \right ) ^{2}  \right ]^{1/2}}
\end{equation}

La correlacion lineal (Pearson) es un indicador que varia entre -1 y 1. Indica si dos variables supuestas independientes, se comportan linealmente una de la otra al analizalas juntas. Correlacion r>0 implica correlacion lineal positiva y r<0 implica correlacion lineal negativa. Si r=0 quiere decir que no existe correlacion LINEAL, aunque puede existir otro tipo de correlacion no lineal.

**IMPORTANTE: Correlacion no implica causalidad**. Si dos variables estan altamente correlacionadas, nada indica que una es la causa de la otra, es decir, nada indica que una depende de otra.

Creamos un dataframe solo de la linea C filtrando el dataframe "molinetes" por la columna "LINEA"

```python
# f
molinetes_c = molinetes[(molinetes['LINEA'] == 'LINEA_C')]
```

Ahora creamos una tabla pivote para resumir por columnas ESTACION y filas FECHA, los valores de cantidad de viajes de molinetes para la linea C

```python
# hacemos la tabla pivote para "simplificar o resumir" los datos provenientes de "molinetes_c"
molinetes_c_pivot = pd.pivot_table(molinetes_c, values = 'TOTAL', columns = 'ESTACION', index = 'PERIODO',fill_value=0, aggfunc = np.sum)
```

```python
# observamos como quedo la tabla pivote con un .head()
molinetes_c_pivot.head()
```

      ESTACION
      AVENIDA DE MAYO
      CONSTITUCION
      DIAGONAL NORTE
      GENERAL SAN MARTIN
      INDEPENDENCIA
      LAVALLE
      MARIANO MORENO
      RETIRO
      SAN JUAN

      PERIODO

      201701
      106068
      8229
      88541
      135184
      195139
      118027
      110087
      511180
      705

      201702
      102016
      1
      92249
      137498
      228300
      117724
      124408
      517505
      0

      201703
      179735
      1673294
      180484
      276231
      188051
      278616
      219070
      1016705
      155478

      201704
      147947
      1674946
      152623
      233642
      161896
      244432
      184197
      941852
      157027

      201705
      173288
      1973564
      178884
      280895
      178144
      290885
      217205
      1105674
      179406

ahora generamos una matriz de correlacion de Pearson entre cada estacion de la linea C. Con esto queremos entender si cuando aumenta o diminuye los viajes de molinetes de una estacion, el resto de las estaciones evoluciona directa o indirectamente proporcional de manera lineal.

```python
# creamos una matriz "mol_c_corr" que es lo obtenido de aplicar .corr() a "molinetes_c_pivot"
mol_c_corr = molinetes_c_pivot.corr()
```

```python
# observamos las dimensiones de esta matriz
np.shape(mol_c_corr)
```

    (9, 9)

```python
# visualizamos las primeras 3 filas de la matriz de correlacion
mol_c_corr.head(3)
```

      ESTACION
      AVENIDA DE MAYO
      CONSTITUCION
      DIAGONAL NORTE
      GENERAL SAN MARTIN
      INDEPENDENCIA
      LAVALLE
      MARIANO MORENO
      RETIRO
      SAN JUAN

      ESTACION

      AVENIDA DE MAYO
      1.000000
      0.944188
      0.992863
      0.981723
      -0.457787
      0.976326
      0.988167
      0.969090
      0.957276

      CONSTITUCION
      0.944188
      1.000000
      0.954917
      0.960572
      -0.619212
      0.982128
      0.968479
      0.993509
      0.985945

      DIAGONAL NORTE
      0.992863
      0.954917
      1.000000
      0.991764
      -0.498654
      0.987925
      0.989611
      0.975702
      0.975865

# Heatmap de correlacion estaciones linea C
Utilizamos la funcion Heatmap de seaborn para visualizar la matriz de correlacion que generamos en el paso anterior. Observamos que la estacion Independencia correlaciona negativamente con el resto. Esto quiere decir que cuando el resto de las estaciones incrementa su numero de pasajeros por molinete, en Independencia decrece demanera lineal.

```python
corr_c_plot = sns.heatmap(mol_c_corr, xticklabels=1, yticklabels=1)
corr_c_plot.set_xticklabels(corr_c_plot.get_xticklabels(), rotation=40, ha="right")
plt.title("matriz de correlacion viajes en estaciones linea C")
plt.show()
```

Observamos que la estacion Independencia correlaciona negativamente con respecto al resto de las estaciones. Posiblemente sea un efecto de que Independencia es un nodo que intersecta a la linea E.

```python

```


# ENGSCI233 - Computational Techniques and Computer Systems

**ENGSCI233: Computational Techniques and Computer Systems**

*Department of Engineering Science, University of Auckland*

Welcome to ENGSCI233: Computational Techniques and Computer Systems.

The **purpose** of this course is to make you an [**awesome programmer**](https://www.reddit.com/r/programminghorror/). We do this by:

- Exposing you to a range of **computational techniques and [algorithms](https://pics.me.me/algorithm-noun-word-used-by-programmers-when-they-do-not-6470233.png)** for working with data, building models, and solving problems.
- Giving you **practice writing computer code**, including problem conceptualisation, pseudocode, implementation, and testing.
- Demonstrating select **software design concepts** that improve the quality of your programs, including unit testing, version control, profiling, and API design.

This will be done through a mixture of Jupyter notebooks (this thing you're reading now), notebook-based activities in class, as well as computer-based labs.

## Assessment

Your performance in this course is assessed by a mixture of tests, lab assignments, and a final exam. The exact division is:

- **30%** across **two tests** in weeks 6 and 11.
- **60%** from ten **labs**, 6% each.
- **10%** from ten Canvas **quizzes**, 1% each.

### Labs

The majority of these are Python-based. For many of these labs, you will need to submit your written Python scripts through Canvas. Submissions are automatically **compared** against each other for **similarity**. The final project lab is partially assessed during the allocated lab time.

***Copying of work is easy to detect and may amount to academic misconduct.***

In 2019, ten or so students in ENGSCI233 had academic misconduct interviews as a result of their CT lab submissions.

### Canvas quizzes

There are **ten quizzes** scattered across the various topics. Each quiz comprises **a series of multichoice questions** and must be completed **before class starts**. You may have up to **three attempts** and your **best mark** will be kept.

The total allocated marks for the quizzes is low (10%). Their **purpose** is to incentivise you to familiarise yourself with the material before coming to class.

## It's not working...

Unfortunately, I'm not able to do any troubleshooting during class. However, please feel free to catch me at the front before or after.

If you're doing this at home, I'm afraid it's off to [**Google**](https://devhumor.com/content/uploads/images/August2019/google_programmer.jpg) with you... (or make a post on **Piazza**)

## Feedback

I'm very pleased to hear how this course could be made better so please participate in the **SET survey** at the end of the course. Let me know the things you liked, the things you hated, and what was most useful for your learning.

If things are going badly **RIGHT NOW**, then get in touch early. You can make public OR private posts on Piazza. Public posts are anonymous to other students. You can also email your lecturer directly if things are really terrible - we want to hear if that's the case!

## Is this Python?

No. Yes. Sort of.

You're currently reading this text inside of something called a Jupyter Notebook. It enables me to write short explanations, with [web links](https://historytech.files.wordpress.com/2017/09/screen-shot-2017-09-14-at-4-11-16-pm.png?w=352&h=267), pictures

\begin{equation}
e^{qu}\times at=i\left(on\right)^s
\end{equation}

and

```python
# live modifiable, Python code

# run the command below by clicking inside the cell and hitting Ctrl+Enter
1+2
```

# Python

For the rest of this course you will be working exclusively with the [Python](https://en.wikipedia.org/wiki/Python_(programming_language%29) programming language.

Python is one of the most widely-used high-level programming languages in the world. It's popularity is due largely to it being open-source and [easy to learn](https://images7.memedroid.com/images/UPLOADED973/596e4c96650e5.jpeg). These notebooks embed snippets of Python code for you to run, modify and (sometimes) complete.

"***But I don't know Python, we were taught MATLAB in ENGGEN 131.***"

Python has many of the [same capabilities](https://docs.scipy.org/doc/numpy/user/numpy-for-matlab-users.html) as MATLAB. You will be provided with a range of resources to get you up and running with Python, including instruction documents, tutorials and help forums. Python will not be the last 'new' programming language you are asked to learn.

This document is called a Jupyter Notebook. It will serve both as the ***notes*** for this course, and the material we will cover in class. Your copy of this notebook is ***unique to you*** - I encourage you to take notes within it. One way to do this is to modify a text cell - **double click it, modify, then hit Ctrl+Enter to save** - with new text and [highlight it](http://stackoverflow.com/questions/25104738/text-highlight-in-markdown), e.g.

> <mark>**I have enclosed this statement in    `<mark> </mark>` HTML tags. It is now an important highlighted note that my future self will definitely come back and read**</mark>

This notebook contains Python code embedded in *cells* that can be executed (by you, right now, on your laptop or, in theory, your phone). For example, ***execute the code in the cell immediately below*** by highlighting it and hitting Ctrl+Enter (or the *run* button at the top, a sideways triangle pointing to a vertical bar)

```python
# oh gawd, not that dumb computer science in-joke again

print("hello, world")          # RECALL: print is a function, "hello, world" is a string

# <-- the 'hash' symbol is a comment (it was a % in MATLAB)
```

You can modify the code above. You ***should*** modify the code above. ***Change it*** to print something different.

We can do all the usual MATLAB stuff, define variables and do math things with them.

```python
# run the simple calculation below and view the result using the print function
a = 3.5
b = 2.8
c = a*b
# *copy-paste the print statement from the cell above to display the value of c*
```

We [***import*** modules](https://www.tutorialspoint.com/python/python_modules.htm) to make additional functionality available (in MATLAB, these are preloaded)

```python
# import the NumPy module, which duplicates much of what you want from MATLAB
import numpy as np
v = np.linspace(0,10,11)        # RECALL: linspace creates a vector of evenly spaced numbers
print(v)
print(2*v)
```

- [Python indexing](http://stackoverflow.com/questions/509211/explain-pythons-slice-notation) is ***[from zero](https://i.redd.it/bpnzu5yy126z.png)*** just like with C.
- You can index relative to the end of an array using ***negative*** indices.
- Indexing uses ***square*** brackets, [], as opposed to round ones, (), in MATLAB.
- Variables ***carry over*** from previous cells.
- You don't need a semicolon (to suppress output in MATLAB, to [inexplicably](https://memeworld.funnyjunk.com/pictures/Scumbag+semicolon+just+something+that+pissed+me+off+while+in_88463d_3293276.jpg) allow your code to compile in C).

```python
# make sure you understand HOW these commands result in the printed output below.
print(v[0])
print(v[-1])
print(v[:2])
print(v[-2:])
```

**Indentation is important**. In this way, Python enforces that your code looks good. For example, executing the commands below raises an error.

***Read the error message. Try to understand it. This is your BEST chance to debug a problem.***

```python
left_hand = 2.
right_hand = 3.
hands = left_hand + right_hand
if hands>6.:
print('not bad')
else:
print('you have small hands')
```

Fix the error by indenting the two `print` statements.

**Lists** are ordered collections of Python objects (floats, integers, booleans, strings, other lists, etc.)

```python
my_list = [1,2,3,4]
print(my_list)
empty_list = []
print(empty_list)
empty_list.append('hi,')
empty_list.append('not')
empty_list.append('empty')
empty_list.append('now')
print(empty_list)
```

**They are different to arrays**, which are ordered collections of numbers we can do math on.

```python
a = np.array([1,2,3,4])
b = 2*a
print(b)
amean = np.mean(a)
print(amean)
print(np.sum(a), np.std(a), np.min(a), np.max(a))
```

# [What now?](https://i.imgur.com/9OXCPp7.jpg)

**Open the introductory Python notebook [python101](../supplementary/python101.ipynb)** and work through the exercises there.


## PyCity_Calc tutorial (Part 2)

Now we are going to learn how to generate a city object within pycity_calc.

The city object of pycity_calc is an inheritance of the city district object of pycity (which is an inheritance of uesgraph, which is an inheritance of networkx.Graph). Long list of dependencies...however, can be pretty powerful.

Again, we have to start to define an environment

```python
import pycity_base.classes.Weather as Weather

import pycity_calc.environments.co2emissions as co2
import pycity_calc.environments.environment as env
import pycity_calc.environments.market as mark
import pycity_calc.environments.timer as time

#  Input parameters
year = 2010  # Reference year
timestep = 3600  # Timestep in seconds
location = (51.529086, 6.944689)  # (latitude, longitute) of location
altitude = 55  # altitude of location in m

#  Generate extended timer object
timer = time.TimerExtended(timestep=timestep, year=year)

#  Generate weather object
weather = Weather.Weather(timer, useTRY=True, location=location,
                          altitude=altitude)

#  Generate market object
market = mark.Market()
#  Market is an inheritance of pycity prices object

#  Generate co2 emissions object (new in pycity_calc)
co2em = co2.Emissions(year=year)

#  Generate environment
environment = env.EnvironmentExtended(timer, weather, prices=market,
                                      location=location, co2em=co2em)

print('Kind of object:')
print(environment._kind)
```

    Kind of object:
    environment

Next, we are going to generate a city object (of pycity_calc)

```python
import pycity_calc.cities.city as city

#  Generate city object
city_object = city.City(environment=environment)

print('Kind of object:')
print(city_object._kind)
#  city is an inheritance of city district. Thus, it holds the city district attribute _kind = 'citydistrict'
```

    Kind of object:
    citydistrict

Now we are going to generate three BuildingExtended objects for the city object (with single zones / one apartment per building!)

```python
import sympy.geometry.point as point

import pycity_base.classes.demand.SpaceHeating as SpaceHeating
import pycity_base.classes.demand.ElectricalDemand as ElectricalDemand
import pycity_base.classes.demand.Apartment as Apartment
import pycity_calc.buildings.building as build_ex

#  Iterate 3 times to generate 3 building objects
for i in range(3):
    #  Create space heating demand object (with standardized load profiles (method=1))
    heat_demand = SpaceHeating.SpaceHeating(environment,
                                            method=1,
                                            profile_type='HEF',
                                            livingArea=100,
                                            specificDemand=130)

    #  Create electrical demand object (with standardized load profile)
    el_demand = ElectricalDemand.ElectricalDemand(environment, method=1,
                                                  annualDemand=3000,
                                                  profileType="H0")

    #  We can create occupancy profiles, dhw profiles etc., but we are going to skip it, here.

    #  Create apartment
    apartment = Apartment.Apartment(environment)

    #  Add demands to apartment
    apartment.addMultipleEntities([heat_demand, el_demand])

    #  Create extended building object
    extended_building = build_ex.BuildingExtended(environment,
                                                  build_year=1962,
                                                  mod_year=2003,
                                                  build_type=0)

    #  Add apartment to extended building
    extended_building.addEntity(entity=apartment)

    #  Generate positions (sympy Point object)
    position = point.Point(i*10, 0)

    #  Add extended buildings to city object
    city_object.add_extended_building(extended_building=extended_building,
                                      position=position)

print('Get number of buildings:')
print(city_object.get_nb_of_building_entities())

print('Get annual space heating demand of one building:')
print(extended_building.get_annual_space_heat_demand())
```

    Get number of buildings:
    3
    Get annual space heating demand of one building:
    13000.0

City holds further methods:

```python
print('\nGet list of node ids with building objects:')
print(city_object.get_list_build_entity_node_ids())

print('\nGet power curves of all buildings')
print(city_object.get_power_curves())

print('\nGet total annual space heating demand of all buildings')
print(city_object.get_annual_space_heating_demand())

print('\nGet total annual electrical demand of all buildings')
print(city_object.get_annual_el_demand())

print('\nNode ids of original city district')
print(city_object.nodes(data=False))
print('\nWith data:')
print(city_object.nodes(data=True))
```

    Get list of node ids with building objects:
    [1001, 1002, 1003]

    Get power curves of all buildings
    (array([ 244.82052 ,  226.913643,  209.84616 , ...,  206.098767,
            239.055471,  268.429914]), array([ 5304.87276967,  5443.26075496,  5581.64874026, ...,  6646.54219399,
            4537.2514064 ,  3308.09634313]))

    Get total annual space heating demand of all buildings
    39000.0

    Get total annual electrical demand of all buildings
    9000.00000004

    Node ids of original city district
    [1001, 1002, 1003]

    With data:
    [(1001, {'entity': <pycity_calc.buildings.building.BuildingExtended object at 0x000000000A99DBA8>, 'name': 1001, 'is_supply_cooling': False, 'is_supply_heating': False, 'is_supply_electricity': False, 'is_supply_gas': False, 'position': Point(0, 0), 'node_type': 'building', 'is_supply_other': False}), (1002, {'entity': <pycity_calc.buildings.building.BuildingExtended object at 0x000000000A99BB38>, 'name': 1002, 'is_supply_cooling': False, 'is_supply_heating': False, 'is_supply_electricity': False, 'is_supply_gas': False, 'position': Point(10, 0), 'node_type': 'building', 'is_supply_other': False}), (1003, {'entity': <pycity_calc.buildings.building.BuildingExtended object at 0x000000000C015B38>, 'name': 1003, 'is_supply_cooling': False, 'is_supply_heating': False, 'is_supply_electricity': False, 'is_supply_gas': False, 'position': Point(20, 0), 'node_type': 'building', 'is_supply_other': False})]

Now we are going to add streets to our city object:

```python
#  Add street network

#  Add str nodes
node_1 = city_object.add_street_node(position=point.Point(0, -1))
node_2 = city_object.add_street_node(position=point.Point(10, -1))
node_3 = city_object.add_street_node(position=point.Point(20, -1))

#  Add edges
city_object.add_edge(node_1, node_2, network_type='street')
city_object.add_edge(node_2, node_3, network_type='street')

print('Nodelist street:')
print(city_object.nodelist_street)
```

    Nodelist street:
    [1004, 1005, 1006]

pycity_calc hold plotting methods. We are going to plot our existing city now:

```python
%matplotlib inline

import pycity_calc.visualization.city_visual as citvis

#  Plot city object
citvis.plot_city_district(city=city_object, offset=-0.2)
```

It is possible to generate a city district with txt/csv input table. Please take a look at pycity_calc/cities/scripts/...
Here you will find a city_generator (to generate city with buildings, only), a street_generator and a complex_city_generator (city with buildings and street network).

You should take a look at further examples within pycity_calc/examples folder to get more information. PyCity_Calc also holds a couple of toolbox-scripts for:
* City analysis
* Building clustering
* Energy system dimensioning
* Network operations
* Interface to TEASER

Energysystems, electrical grid (pypower) and simulations scripts are currently under development (06.04.2016). Going to be added, soon...


## Introduction to Object Oriented Programming with Python

This prepares you for a 2 day challenge. The goals:

1. Be able to write Python code;
2. Understand the ideas behind object oriented programming.

### Basic Python

1. Variables;
2. Flow Control;
3. Functions;
4. Data structure.

### Basic Object Oriented Programming

1. Classes;
2. Attributes;
3. Methods;
4. Inheritance.

### The Challenge

1. End of the day Thursday: Initial Feedback;
2. End of the day Friday: results.

## Overview of Python

What is Python?

Python is a programming language. There are various other programming languages:

Java
C
C++
Ruby
VBA
and many more.
A programming language allows you to write a program which is a sequence of instructions that specifies how to perform a computation.

When writing a program you need two things:

Something to save the code (a text editor for example)
Something to run the code
We will be using a combination of these 2 things called notebooks.

## Installing Python

There are various distributions of Python, we will use Anaconda which comes packaged with a variety of other useful tools (including the notebooks I mentioned above).

To install it on your personal machine follow these steps:

Go to this webpage: https://www.continuum.io/downloads.

Identify and download the version of **Python 3** for your operating system (Windows, Mac OSX, Linux).
Run the installer.
We will use a Jupyter notebook which runs in your browser. To open a local server find the Continuum navigator and click on Jupyter. You do not need to be connected to the internet to use this.

## Interacting with Python

Once you have installed Anaconda, you will now have Python on your machine. You can interact with Python in multiple ways:

### The Python shell

- On Windows open a "Command Prompt":

- On Mac OS or Linux open a "Terminal":

This is a simple utility that allows you to give commands to your computer. In there type:

```bash
python
```

This should then look something like:

```python
Python 3.5.2 |Anaconda 4.2.0 (64-bit)| (default, Jul  2 2016, 17:53:06)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
```

The `>>>` is a prompt for you to type a command. Let us carry out a straightforward addition:

```python
>>> 2 + 2
```

and press `ENTER`:

```python
>>> 2 + 2
4
```

**This is not a very efficient and practical way of using Python.** We will instead learn to use a [Jupyter notebook](http://jupyter.org/).

### Jupyter notebooks

If you are still in the Python shell (with a `>>>` waiting for you to give a command) then type:

```python
>>> exit()
```

This will leave the prompt and you will now be back at the Command Prompt or the Terminal window. Here simply type:

```bash
jupyter notebook
```

After a little moment this will open a Jupyter notebook page in a browser. Note that this is all just running on your computer and you do not need to be connected to the internet to use a Jupyter notebook.

Click on `New` and create a notebook: we can start to write code now.

### Basic Python

Write `2 + 2` in a cell and press `shift + Enter`:

```python
2 + 2
```

    4

## Variables

### Character variables:

```python
string = "Hello world"
string
```

    'Hello world'

### Numeric variables:

```python
num_1 = 2
num_2 = 3.5
num_1 + num_2
```

    5.5

### String manipulation

```python
#We define a variable called String
# (note that # allows me to comment my code)
string = "My name is Vince"
#Let's get the 5th letter of String
# (Note that Python starts counting at 0):
string[4]
```

    'a'

```python
string[1:4]
```

    'y n'

```python
index_of_v = string.index("V")
index_of_v
```

    11

```python
string[index_of_v:]
string[:index_of_v]
```

    'My name is '

### Numeric manipulation

```python
num = 3

# The following two lines are equivalent
num = num + 1
num += 1
num
```

    5

```python
num -= 2
num *= 3
num **= 2
num
```

    81

## Flow control

- In Python indentation is important!
- In all languages indentation is good practice, in Python it is a requirement.

### If statements

```python
n = 11
if n <= 5:
    value = 1
elif n % 2 == 0:
    value = 2
else:
    value = 3
value
```

    3

### While loops

```python
count = 0
total = 0
while count < 10:
    count += 1
    total += count
total
```

    55

### For loops

```python
for i in [1, 2, 3, 4]:
    print(i)
```

    1
    2
    3
    4

```python
for subject in ["Queueing Theory", "Game Theory",
                "Inventory Theory", "Reliability Theory",
                "Project Management", "Decision Analysis"]:
    if "Theory" in subject:
        print(subject)
```

    Queueing Theory
    Game Theory
    Inventory Theory
    Reliability Theory

## Functions

```python
#To create a function we use the 'def' statement:
def hi():
    """
    This function simply prints a short statement.

    This is a shorter way of writing documentation,
    it is good practice to always include a
    description of what a function does.
    """
    print("Hello everybody!")

hi()
```

    Hello everybody!

```python
def fibonacci(n):
    """
    This returns the nth Fibonacci number.
    """
    if n == 0:
        return 0
    if n == 1:
        return 1
    return fibonacci(n - 1) + fibonacci(n - 2)
```

```python
fibonacci(5)
```

    5

## Data structures: Lists

```python
my_list = list(range(6))
my_list[0]
```

    0

```python
my_list.append(100)
my_list
```

    [0, 1, 2, 3, 4, 5, 100]

# Object Oriented Programming

This is similar to cellular structure:

We can create "things" with:

- attributes: things those "things" have;
- methods: things those "things" can do.

## Defining a class

```python
class Student():
    """We can create a simple empty class.

    This is a set of rules that says what a student is.
    """
```

```python
vince = Student()  # Creating an instance
vince
```

    <__main__.Student at 0x7f34b4210b38>

```python
zoe = Student()  # Creating a different instance
zoe
```

    <__main__.Student at 0x7f34b4210470>

## Attributes

```python
class Student():
    courses = ["Biology", "Mathematics", "English"]
    age = 5
    gender = "Male"
#Let us now create Vince again:
vince = Student()
```

Accessing these attributes:

```python
vince.courses
```

    ['Biology', 'Mathematics', 'English']

```python
vince.age
```

    5

```python
vince.gender
```

    'Male'

We can manipulate these attributes just like **any other** python variable:

```python
vince.courses.append("Photography")
vince.courses
```

    ['Biology', 'Mathematics', 'English', 'Photography']

```python
vince.age = 28
vince.age
```

    28

```python
vince.gender = "M"
vince.gender
```

    'M'

## Methods

```python
class Student():
    courses = ["Biology", "Mathematics", "English"]
    age = 5
    sex = "Male"

    def have_a_birthday(self):
        """This method increments the age of our instance."""
        self.age += 1
```

```python
vince = Student()
vince.age
```

    5

```python
vince.have_a_birthday()
vince.age
```

    6

## The `__init__` method

```python
class Student():
    def __init__(self, courses, age, sex):
        """
        What the class should do when it
        is used to create an instance
        """
        self.courses = courses
        self.age = age
        self.sex = sex

    def have_a_birthday(self):
        self.age += 1

```

```python
vince = Student(["Biology","Math"],28,"Male")
vince.courses, vince.age, vince.sex
```

    (['Biology', 'Math'], 28, 'Male')

## Inheritance

We can use a class to create new classes:

```python
class Math_Student(Student):
    """
    A Math student: behaves exactly like a Student
    but also has a favourite class attribute.
    """
    favourite_class = "Mathematics"
```

```python
becky = Math_Student(["Mathematics", "Biology"], 29, "Female")
becky.courses, becky.age, becky.sex, becky.favourite_class
```

    (['Mathematics', 'Biology'], 29, 'Female', 'Mathematics')

```python
#This class has the methods of the parent class:
becky.have_a_birthday()
becky.age
```

    30

## Summary

- Classes
- Attributes
- Methods
- Inheritance

## Advantages

- Simplicity
- Modularity
- Modifiability
- Extensibility
- Re-usability

# Libraries

There are a number of built in libraries that extend what Python can do.

```python
import random
random.seed(0)
random.random()
```

    0.8444218515250481

```python
import math
math.cos(math.pi)
```

    -1.0

There are also a number of external libraries. This is one of the huge strengths of Python. Some of these come with Anaconda (the distribution of Python I recommend):

```python
import sympy  # symbolic mathematics
x = sympy.symbols('x')
sympy.diff(x ** 2, x)
```

    2*x

```python
import numpy  # Fast numeric computations
A = numpy.matrix([[0, 2], [1, 2]])
B = numpy.matrix([[3, 1], [1, -2]])
A * B
```

    matrix([[ 2, -4],
            [ 5, -3]])

There are also a number of libraries outside of anaconda that are also very powerful. Some examples include:

- Ciw: modeling of queues;
- Axelrod: game theory.
- tqdm: adding progress bars to your code :)
- The list is very large...

To install these libraries from the internet you can open a command prompt (Windows) or a terminal (Mac OSX) and type:

```bash
pip install <library-name>
```

# Further resources

There are a number of wonderful resources for learning Python. Here are some that I have made:

- My first year computing for Mathematics course: http://vknight.org/cfm/
- A short set of notebooks that used to teach Mathematics how to use Python: https://github.com/drvinceknight/Python-Mathematics-Handbook


# Homework 4: Soft Actor Critic (10 Pts)

All homeworks are self-contained. They can be completed in their respective notebooks.
To edit and re-run code, you can therefore simply edit and restart the code cells below.
There is a timeout of about ~12 hours with Colab while it is active (and less if you close your browser window).
This file should automatically be synced with your Google Drive. We also save all recordings and logs in it by default so that you won't lose your work in the event of an instance timeout.
 However, you will need to re-mount your Google Drive and re-install packages with every new instance.

```python
# Your work will be stored in a folder called `drl_ws21` by default to prevent Colab
# instance timeouts from deleting your edits.
# We do this by mounting your google drive on the virtual machine created in this colab
# session. For this, you will likely need to sign in to your Google account and copy a
# passcode into a field below

import os
from google.colab import drive

drive.mount('/content/gdrive')
```

```python
# Create paths in your google drive
DRIVE_PATH = '/content/gdrive/My\ Drive/drl_ws21'
DRIVE_PYTHON_PATH = DRIVE_PATH.replace('\\', '')
if not os.path.exists(DRIVE_PYTHON_PATH):
    % mkdir $DRIVE_PATH

# the space in `My Drive` causes some issues,
# make a symlink to avoid this
SYM_PATH = '/content/drl_ws21'
if not os.path.exists(SYM_PATH):
    !ln -s $DRIVE_PATH $SYM_PATH
% cd $SYM_PATH
```

```python
# Install **python** and **system** packages

# install required system dependencies
!apt-get install -y xvfb x11-utils

# install required python dependencies
!pip install matplotlib numpy tqdm torch stable_baselines3 gym[box2d]==0.17.* pyvirtualdisplay==0.2.* PyOpenGL==3.1.* PyOpenGL-accelerate==3.1.*
```

We start by importing all the necessary python modules and defining some helper
functions which you do not need to change. Still, make sure you are aware of
what they do.

```python
# Imports and utility
# Progress bar

import os
import gym
import copy
import tqdm
import time
import torch
import numpy as np
import torch.nn as nn
import collections, random
import torch.optim as optim
import torch.nn.functional as F
import matplotlib.pyplot as plt
from typing import Tuple, Dict, List
from torch.distributions import Normal
from stable_baselines3.common.vec_env import DummyVecEnv
from stable_baselines3.common.vec_env import VecVideoRecorder

# Set random seeds
np.random.seed(0)

class ProgressBar:
    def __init__(self, num_iterations: int, verbose: bool = True):
        if verbose:  # create a nice little progress bar
            self.scalar_tracker = tqdm.tqdm(total=num_iterations, desc="Scalars", bar_format="{desc}",
                                            position=0, leave=True)
            progress_bar_format = '{desc} {n_fmt:' + str(
                len(str(num_iterations))) + '}/{total_fmt}|{bar}|{elapsed}<{remaining}'
            self.progress_bar = tqdm.tqdm(total=num_iterations, desc='Iteration', bar_format=progress_bar_format,
                                          position=1, leave=True)
        else:
            self.scalar_tracker = None
            self.progress_bar = None

    def __call__(self, _steps: int = 1, **kwargs):
        if self.progress_bar is not None:
            formatted_scalars = {key: "{:.3e}".format(value[-1] if isinstance(value, list) else value)
                                 for key, value in kwargs.items()}
            description = ("Scalars: " + "".join([str(key) + "=" + value + ", "
                                                  for key, value in formatted_scalars.items()]))[:-2]
            self.scalar_tracker.set_description(description)
            self.progress_bar.update(_steps)

# specify the path to save the recordings of this run to.
data_path = '/content/drl_ws21/exercise_4'
data_path = os.path.join(data_path, time.strftime("%d-%m-%Y_%H-%M"))
if not (os.path.exists(data_path)):
    os.makedirs(data_path)

# disable the actual display to prevent errors with colab
from pyvirtualdisplay import Display
_display = Display(visible=False, size=(1400, 900))
_ = _display.start()

# this function will automatically save your figure into your google drive folder (if correctly mounted!)
def save_figure(save_name: str) -> None:
    assert save_name is not None, "Need to provide a filename to save to"
    plt.savefig(os.path.join(data_path, save_name + ".png"))

def evaluate_rollout(evaluation_environment: gym.Env, soft_actor_critic) -> float:
    """
    Performs a full rollout using the mean of the current policy.
    :param evaluation_environment: The environment used for evaluation. In our case, a Pendulum environment
    :param soft_actor_critic: An instance of the SAC class defined above
    :return: The total score for the rollout
    """
    done = False
    score = 0
    state = evaluation_environment.reset()
    while not done:  # alternate between collecting one step of data and updating SAC with one mini-batch
        action, log_probabilities = soft_actor_critic.policy(torch.from_numpy(np.array(state)).float(), deterministic=True)
        scaled_action = evaluation_environment.action_space.high[0] * action.item()
        next_state, reward, done, info = evaluation_environment.step([[scaled_action]])
        # need to wrap action in a list because of the video recording
        state = next_state  # go to the next environment step
        score += reward  # keep track of cumulative reward for recording
    return score

def v_function_visualization(evaluation_environment: gym.Env,
                             soft_actor_critic,
                             current_step: int = 0,
                             resolution: int = 100):
    """
    Visualizes a numerical approximation of the value function by evaluating the Q-Function for a wide range
    :param evaluation_environment:
    :param soft_actor_critic:
    :param current_step:
    :param resolution:
    :return:
    """
    import matplotlib.pyplot as plt
    plt.clf()

    max_speed = 8
    x = np.linspace(-np.pi, np.pi, num=resolution)
    y = np.linspace(-max_speed, max_speed, num=resolution)
    state_evaluation_grid = np.transpose([np.tile(x, len(y)), np.repeat(y, len(x))])
    input_observations = torch.Tensor(np.array([np.cos(state_evaluation_grid[:, 0]),
                                                np.sin(state_evaluation_grid[:, 0]),
                                                state_evaluation_grid[:, 1]])).T

    evaluations = []

    for position, action in enumerate(np.linspace(evaluation_environment.action_space.low,
                                                  evaluation_environment.action_space.high,
                                                  50)):
        action_tensor = torch.Tensor(np.full((len(state_evaluation_grid), 1), fill_value=action))

        q1_val = soft_actor_critic.q_net_1(input_observations, action_tensor)
        q2_val = soft_actor_critic.q_net_2(input_observations, action_tensor)
        q1_q2 = torch.cat([q1_val, q2_val], dim=1)
        reward_evaluation_grid = torch.min(q1_q2, 1, keepdim=True)[0]
        reward_evaluation_grid = reward_evaluation_grid.reshape((resolution, resolution))

        evaluations.append(reward_evaluation_grid.detach().numpy())

    plt.title(f"Numerically integrated V-function at step {current_step}")
    plt.xlabel(r"$\theta$")
    plt.ylabel(r"$\dot{\theta}$")
    heatmap = plt.contourf(x, y, np.array(evaluations).max(axis=0), levels=100,
                           cmap=plt.get_cmap("jet"), zorder=0)
    plt.colorbar(heatmap)
    save_figure(save_name=f"numerical_v_function_{current_step:04d}")

def plot_metrics(metrics: Dict[str, List[float]]):
    """
    Plots various metrics recorded during training
    :param metrics:
    :return:
    """
    if len(metrics) > 0:
        plt.clf()
        plt.figure(figsize=(16, 9))
        for position, (key, value) in enumerate(metrics.items()):
            plt.subplot(len(metrics), 1, position + 1)
            plt.plot(range(len(value)), np.array(value))
            plt.ylabel(key.title())
        plt.xlabel("Recorded Steps")
        plt.tight_layout()
        save_figure(f"training_metrics")
        plt.clf()
        plt.close()

def evaluate(evaluation_environment: gym.Env, soft_actor_critic,
             num_evaluation_rollouts: int = 10):
    """
    Perform num_evaluation_rollouts rollouts on the evaluation environment using the current policy and average over
    the achieved scores. Also plot a visualization of the first of these rollouts and a numerical integration
    of the value function
    :param evaluation_environment: The environment to evaluate. Will perform num_evaluation_rollouts full rollouts on
      this environment
    :param soft_actor_critic: Instance of SAC used to determine the actions
    :param num_evaluation_rollouts: Number of rollouts to evaluate for
    :return:
    """
    scores = []
    for rollout_idx in range(num_evaluation_rollouts):
        rollout_score = evaluate_rollout(evaluation_environment=evaluation_environment,
                                         soft_actor_critic=soft_actor_critic
                                         )
        scores.append(rollout_score)
    mean_score = np.mean(scores)
    return {"score": mean_score}

def visualize_rollout(soft_actor_critic, step: int):
    # evaluation_environment = DummyVecEnv([lambda: gym.make('Pendulum-v0')])
    # keep a second environment for evaluation purposes
    from stable_baselines3.common.env_util import make_vec_env
    evaluation_environment = make_vec_env('Pendulum-v0')
    visualization_environment = VecVideoRecorder(evaluation_environment,
                                                 video_folder=data_path,
                                                 record_video_trigger=lambda x: x == 0,
                                                 video_length=200,  # 200 steps per rollout
                                                 name_prefix=f"Pendulum_{step:05d}")
    evaluate_rollout(evaluation_environment=visualization_environment, soft_actor_critic=soft_actor_critic)

```

# **Soft Actor Critic**

In this exercise, we will re-implement the Soft Actor Critic (SAC) algorithm. SAC is an off-policy actor-critic method
that is widely used in the community due to its sample efficiency and (relative) stability. It acts under a maximum
entropy principle to ensure sufficient exploration during training, and also employs tricks like the reparameterization
trick, polyak-updates of target Q networks and Twin-Delayed Q-Functions.

## Pendulum
We will showcase the SAC algorithm on the very classic [Pendulum](https://gym.openai.com/envs/Pendulum-v0/) control environment.
The goal in this environment is to actuate a pendulum such that it stays upward without too much movement.
It has a one-dimensional action space that represents the torque acting on the pendulum,
and a 2d internal *state* that is the angle $\theta$ and the angular velocity $\dot{\theta}$ of the pendulum at the current time step.
The *external* state or observation is a 3-tuple ($\cos(\theta)$, $\sin(\theta)$, $\dot{\theta}$).

## Replay Buffer
We start by defining our replay buffer which is used to store samples seen during the rollouts that can then
be used for training later on. You do *not* need to implement anything here.

```python
# Functional code
class ReplayBuffer:
    def __init__(self, buffer_limit: int, batch_size: int):
        self.batch_size = batch_size
        self.buffer = collections.deque(maxlen=buffer_limit)  # use a dequeue as a buffer

    def put(self, transition: tuple) -> None:
        """
        Adds a transition to the buffer.
        :param transition: (s, a, r, s', done) pair sampled by having the policy act on the environment
        :return: None
        """
        self.buffer.append(transition)

    def sample(self) -> tuple:
        mini_batch = random.sample(self.buffer, self.batch_size)  # get self.batch_size random samples from the buffer
        states, actions, rewards, next_states, dones = [], [], [], [], []  # initialize list of (s, a, r, s', done) tuples

        for transition in mini_batch:  # parse all transitions into their lists.
            state, action, reward, next_state, done = transition
            states.append(state)
            actions.append([action])
            rewards.append([reward])
            next_states.append(next_state)
            dones.append([float(done)])

        return torch.tensor(states, dtype=torch.float), \
               torch.tensor(actions, dtype=torch.float), \
               torch.tensor(rewards, dtype=torch.float), \
               torch.tensor(next_states, dtype=torch.float), \
               torch.tensor(dones, dtype=torch.float)

    def size(self) -> int:
        return len(self.buffer)
```

## **TASK 1: Actor/Policy network** (2+2=4 Points)

Next, we will set up the actor/policy.

### Task 1.1: Tanh Squashing (2 Points)
The original SAC implementation squashes its actions into the $[-1, 1]$ range using a *tanh* activation.
To ensure proper probabilities for these actions, it also squashes the log probabilities of each action accordingly.
You can show using the change of variables theorem that

\begin{align}
    \log \pi(\boldsymbol{a}|\boldsymbol{s}) = \log \mu(\boldsymbol{u}|\boldsymbol{s})-\sum_{i=1}^D \log\left(1-\tanh^2(u_i)\right)
\end{align}

for (squashed) actions $\boldsymbol{a}$, states $\boldsymbol{s}$, proposed (unsquashed) actions $\boldsymbol{u}$ and a policy
distribution $\mu$ (which you shouldn't confuse with the mean, which is sometimes also called $\mu$...)
In our case, the dimensionality $D=1$.
You will need to squash the action itself, as well as its log probability.

### Task 1.2: Training from 2 Q Networks (2 Points)
As SAC is using Twin-Delayed Q-Networks to prevent the overestimatio bias in the Q-Values,
the actor/policy needs to choose the minimum of both available Q-Networks for its loss function.
For this, you will need to evaluate the action using both Q-Networks, and then simply choose their minimum.

```python
class PolicyNet(nn.Module):
    def __init__(self, learning_rate: float, entropy_alpha: float):
        super(PolicyNet, self).__init__()  # make sure that the policy network is registered as a pytorch module

        # specify neurons per layer. "fc" is short for "fully_connected layer".
        self.common_mlp = nn.Linear(3, 128)
        self.mean_mlp = nn.Linear(128, 1)
        self.std_mlp = nn.Linear(128, 1)

        self.optimizer = optim.Adam(self.parameters(), lr=learning_rate)  # use adam optimizer

        self.entropy_alpha = entropy_alpha  # weight of the entropy term

    def forward(self, state: torch.Tensor, deterministic: bool = False) -> Tuple[torch.Tensor, torch.Tensor]:
        """
        Choose an action and calcualte its probability for the current state.
        :param state: The state to choose the action for
        :param deterministic: Whether to draw an action from
        :return:
        """
        state = F.relu(self.common_mlp(state))
        mean = self.mean_mlp(state)
        std = F.softplus(self.std_mlp(state))  # we need the standard deviation to be >0
        normal_distribution = Normal(mean, std)

        if deterministic:
            action = mean
        else:
            action = normal_distribution.rsample()
        log_probabilities = normal_distribution.log_prob(action)

        # the original SAC implementation also squishes the action into [-1, 1] using a tanh activation.
        # to keep the probabilities correct, they account for this using the update below.

        ### TODO ###
        ### Your code starts here ###
        real_action = torch.tanh(action)
        real_log_probabilities = log_probabilities - torch.log(1 - torch.tanh(action).pow(2) + 1e-7)
        ### Your code ends here ###
        return real_action, real_log_probabilities

    def train_step(self, q_net_1, q_net_2, mini_batch: tuple) -> Dict[str, float]:
        states, _, _, _, _ = mini_batch
        actions, log_probabilities = self.forward(states)
        entropy = -self.entropy_alpha * log_probabilities

        # evaluate both q-networks for the current state-action pair
        # and use their minimum (see Twin-Delayed Q functions)

        ### TODO ###
        ### Your code starts here ###
        q1_value = q_net_1(states, actions)
        q2_value = q_net_2(states, actions)
        q1_q2 = torch.cat([q1_value, q2_value], dim=1)
        min_q = torch.min(q1_q2, 1, keepdim=True)[0]
        ### Your code ends here ###

        loss = -(min_q + entropy).mean()  # "-" for gradient ascent
        self.optimizer.zero_grad()
        loss.backward()
        self.optimizer.step()

        return {"entropy": entropy.mean().item(),
                "policy_loss": loss.item()}

```

Next, we implement our Q-Network. This will be used to evaluate ("criticize") the actions that the actor proposes.
Note that this is only *one* Q-Network, and that the SAC class below will use multiple of those for the Twin-Delayed
Q-Functions.

## Task 2: Polyak Updates ( 2 Points)
For increasing stability in the update of the Q Networks, SAC uses polyak updates of each Q Network.
The update is given as
\begin{align}
    \beta'_i = (1-\tau)\beta'_i+(\tau)\beta_i
\end{align}

for Q-Network parameters $\beta_1$, $\beta_2$ and an update rate $\tau$. Note that the slides use a reverse
order of $(1-\tau)$ and $\tau$, which corresponds to values of $\tau$ close to $1$ rather than close to $0$
as done in the code.

Hint: Use the `parameters().data` attribute of torch.Tensor to access the parameters. You can copy them
using `parameters().data.copy_`.

```python
class QNet(nn.Module):
    def __init__(self, learning_rate: float, q_net_update_rate: float):
        super(QNet, self).__init__()  # make sure that the Q Network is registered as a pytorch module
        # specify network parameters. "fc" is short for "fully_connected layer".
        self.state_layer = nn.Linear(3, 64)
        self.action_layer = nn.Linear(1, 64)
        self.common_layer = nn.Linear(128, 32)
        self.fc_out = nn.Linear(32, 1)

        self.optimizer = optim.Adam(self.parameters(), lr=learning_rate)  # use adam optimizer
        self.q_net_update_rate = q_net_update_rate

    def forward(self, state: torch.Tensor, action: torch.Tensor) -> torch.Tensor:
        h1 = F.relu(self.state_layer(state))
        h2 = F.relu(self.action_layer(action))
        cat = torch.cat([h1, h2], dim=1)
        q_evaluation = F.relu(self.common_layer(cat))
        q_evaluation = self.fc_out(q_evaluation)
        return q_evaluation

    def train_step(self, target_values: torch.Tensor, mini_batch: tuple) -> float:
        """
        Train the network for a single mini-batch update
        :param target_values: The target values to regress to
        :param mini_batch: A tuple (state, action, reward, next_state, done). For this update,
          only the action and state are needed
        :return: The mean loss for this update step
        """
        states, actions, _, _, _ = mini_batch  # get action and state from current mini_batch
        evaluation = self.forward(states, actions)

        # calculate the loss and its gradients; update the network based on them
        loss = F.smooth_l1_loss(evaluation, target_values).mean()
        self.optimizer.zero_grad()
        loss.backward()
        self.optimizer.step()
        return loss.item()

    def polyak_update(self, target_network):
        """
        Soft update the target network with the parameters of this network
        :param target_network:
        :return:
        """
        ### TODO ###
        ### Your code starts here ###
        for param_target, param in zip(target_network.parameters(), self.parameters()):
            param_target.data.copy_(param_target.data * (1.0 - self.q_net_update_rate)
                                    + param.data * self.q_net_update_rate)
        ### Your code ends here ###
```

Finally, we combine the Q-Network(s) and the Actor to build our SoftActorCritic class.
This class (roughly) implements the Pseudo-code shown in Slide Set 7, Slide 40.

## Task 3: Q-Network Targets (4 Points)
For SAC, you will need to implement the targets of the Q-Networks. These are calculated using the rule on Slide Set 7, Slide 35, i.e.,
\begin{align}
y_t = r_t +\gamma\left(Q_{\beta'}(\boldsymbol{s}'_t, \boldsymbol{a}') - \alpha \log \pi(\boldsymbol{a}'|\boldsymbol{s}'_t)\right), \qquad \text{where ~} \boldsymbol{a}'\approx \pi(\boldsymbol{a}|\boldsymbol{s}_t')
\end{align}

Hints:
* Remember that $\beta'$ are the parameters of the **target** Q-Network
* In our case, $Q_{\beta'}$ needs to be evaluated as the **minimum** of **both** target Q-Networks
* You can only do a next step in your next step prediction if there is one, i.e., if the environment is **not** done.

```python
class SoftActorCritic:
    def __init__(self, args):
        learning_rate = args["learning_rate"]
        q_net_update_rate = args["q_net_update_rate"]
        batch_size = args["batch_size"]
        self.discount_factor = args["discount_factor"]

        # initialize two Q-networks and their respective target networks
        self.q_net_1 = QNet(learning_rate, q_net_update_rate)
        self.q_net_2 = QNet(learning_rate, q_net_update_rate)

        self.q_net_1_target = copy.deepcopy(self.q_net_1)
        self.q_net_2_target = copy.deepcopy(self.q_net_2)

        # get a replay buffer and a policy network
        self.memory = ReplayBuffer(buffer_limit=args["buffer_limit"], batch_size=batch_size)

        self.policy = PolicyNet(learning_rate=learning_rate,
                                entropy_alpha=args["entropy_alpha"])

    def train_step(self) -> Dict[str, float]:
        mini_batch = self.memory.sample()
        q_targets = self.calculate_q_targets(mini_batch)

        # update both q networks
        q_net_1_loss = self.q_net_1.train_step(q_targets, mini_batch)
        q_net_2_loss = self.q_net_2.train_step(q_targets, mini_batch)

        # update the policy
        policy_metrics = self.policy.train_step(self.q_net_1, self.q_net_2, mini_batch)

        # polyak updates for the target q networks
        self.q_net_1.polyak_update(self.q_net_1_target)
        self.q_net_2.polyak_update(self.q_net_2_target)

        return {"q_net_1_loss": q_net_1_loss,
                "q_net_2_loss": q_net_2_loss,
                **policy_metrics}

    def calculate_q_targets(self, mini_batch: tuple) -> torch.Tensor:
        _, _, rewards, next_states, dones = mini_batch

        with torch.no_grad():
            ### TODO ###
            ### Your code starts here ###
            next_action, log_probabilities = self.policy(next_states)
            entropy = -self.policy.entropy_alpha * log_probabilities
            q1_val = self.q_net_1_target(next_states, next_action)
            q2_val = self.q_net_2_target(next_states, next_action)
            q1_q2 = torch.cat([q1_val, q2_val], dim=1)
            min_q = torch.min(q1_q2, 1, keepdim=True)[0]
            target = rewards + (1 - dones) * self.discount_factor * (min_q + entropy)
            ### Your code ends here ###
        return target
```

# Running the algorithm

That's it for SAC. The code below defines arguments/hyperparameters, as well as a general training loop. You do
*not* need to change any code here (unless you want to fiddle with the parameters). If everything is implemented
correctly, you should see training improvements after a couple thousand steps, and have a converged solution after
30000-40000 steps.
The code will save
* a couple of training metrics,
* a contour plot of a numerical integration of the value function,
i.e., the maximum of a number of Q-function evaluations on a grid of the state-space
* a .mp4 video of the pendulum swinging.

For the homework, you only need to send in the last of each, i.e., the one at iteration 50000, but you can also
turn in all of the plots as usual.

```python
class Args:
    def __getitem__(self, key):
        return getattr(self, key)

    def __setitem__(self, key, val):
        setattr(self, key, val)

    num_training_steps = 51000  # @param {type: "integer"}
    learning_rate = 3.0e-4  # @param {type: "number"}
    entropy_alpha = 0.02  # @param {type: "number"}
    discount_factor = 0.98  # @param {type: "number"}
    batch_size = 64  # @param {type: "integer"}
    buffer_limit = 100000  # @param {type: "integer"}
    reward_scale = 0.1  # @param {type: "number"}
    q_net_update_rate = 0.002  # @param {type: "number"}

def main(args: Args):
    environment = gym.make('Pendulum-v0')
    evaluation_environment = DummyVecEnv([lambda: gym.make('Pendulum-v0')])
    # keep a second environment for evaluation purposes.
    # We wrap it in a Dummy Vector Environment for compatibility with
    # the visualization utility

    soft_actor_critic = SoftActorCritic(args=args)

    reward_scale = args["reward_scale"]
    num_training_steps = args["num_training_steps"]

    # logging utility
    logging_frequency = 100  # log progress every 100 steps
    plot_frequency = 5000
    progress_bar = ProgressBar(num_iterations=num_training_steps)

    state = environment.reset()  # restart the environment, i.e., go back to some initial state
    full_metrics = {"score": []}
    train_step_metrics = {}
    for current_step in range(num_training_steps):
        if current_step % logging_frequency == 0:  # log every logging_frequency steps
            for key, value in train_step_metrics.items():
                if key not in full_metrics:
                    full_metrics[key] = []
                full_metrics[key].append(value)

            evaluation_recordings = evaluate(evaluation_environment=evaluation_environment,
                                             soft_actor_critic=soft_actor_critic)

            progress_bar(_steps=logging_frequency,
                         score=evaluation_recordings.get("score"),
                         **train_step_metrics)

            full_metrics["score"].append(evaluation_recordings.get("score"))

        if current_step % plot_frequency == 0:  # plot visualizations
            v_function_visualization(evaluation_environment=evaluation_environment,
                                     soft_actor_critic=soft_actor_critic,
                                     current_step=current_step)
            visualize_rollout(soft_actor_critic=soft_actor_critic,
                              step=current_step)
            plot_metrics(full_metrics)

        # alternate between collecting one step of data and updating SAC with one mini-batch
        action, log_probabilities = soft_actor_critic.policy(torch.from_numpy(np.array(state)).float())
        next_state, reward, done, info = environment.step([environment.action_space.high[0] * action.item()])

        # safe (s, a, r, s', done) tuple in memory buffer
        soft_actor_critic.memory.put((state, action.item(), reward * reward_scale, next_state, done))

        if done:
            state = environment.reset()
        else:
            state = next_state

        # wait until there are enough rollouts in the memory buffer before starting the training
        if soft_actor_critic.memory.size() > 1000:
            train_step_metrics = soft_actor_critic.train_step()
            train_step_metrics["buffer_size"] = soft_actor_critic.memory.size()
    environment.close()

args = Args()
main(args=args)

```