# Selección óptima de portafolios I

En la clase pasada vimos que:
- La LAC describe las posibles selecciones de riesgo-rendimiento entre un activo libre de riesgo y un activo riesgoso.
- Su pendiente es igual al radio de Sharpe del activo riesgoso.
- La asignación óptima de capital para cualquier inversionista es el punto tangente de la curva de indiferencia del inversionista con la LAC (depende de las preferencias particulares - aversión al riesgo).

Para todo lo anterior, supusimos que ya teníamos el portafolio óptimo (activo riesgoso).

En el siguiente análisis:

**Objetivos:**
- ¿Cuál es el portafolio óptimo de activos riesgosos?
 - ¿Cuál es el mejor portafolio de activos riesgosos?
 - Es un portafolio eficiente en media-varianza.
- Problema: dado un conjunto de activos riesgosos, ¿cómo construimos la mejor combinación?

*Referencia:*
- Notas del curso "Portfolio Selection and Risk Management", Rice University, disponible en Coursera.
___

## 1. Maximizando el radio de Sharpe

### ¿Qué pasa si tenemos dos activos riesgosos?

Cuando tenemos dos o más activos riesgosos, tenemos disponibles diferentes LAC. ¿Qué significan sus pendientes?

 Ver en el tablero.

Pregunta:
- ¿Qué es lo que se quiere?

**Conclusión:**
- El mejor portafolio de activos no depende de las preferencias individuales, y por tanto va a ser igual para todos.
- Dicho mejor portafolio maximiza el radio de Sharpe.
- A este portafolio lo llamaremos el portafolio eficiente en media-varianza (EMV)

**Idea principal: el portafolio óptimo de activos riesgosos es independiente de las preferencias del inversionista.**
- El portafolio EMV determina el portafolio óptimo de activos riesgosos.
- Todos tendremos el mismo portafolio de activos riesgosos (EMV), y lo combinaremos con el activo libre de reisgo de acuerdo con las preferencias de cada uno de nosotros (aversión al riesgo).
- La LAC combinando el activo libre de riesgo y el portafolio EMV, se vuelve el conjunto de portafolios eficientes.

Entonces, se deben seguir los siguientes pasos:
1. Crear la frontera media-varianza.
2. Encontrar el portafolio que maximize el radio de Sharpe (portafolio EMV).
3. Construir la frontera eficiente (LAC) del punto $(0,r_f)$ al punto $(\sigma_s,E[r_s])$ del portafolio EMV.
4. Combinar de acuerdo a sus preferencias.
___

## 2. Solución analítica del portafolio EMV: caso con dos activos.

Queremos solucionar el siguiente problema:

\begin{align}
\max_{w_1,w_2} &\quad \frac{E[r_p]-r_f}{\sigma_p}\\
\text{s.a.}    &\quad E[r_p]=w_1E[r_1]+w_2E[r_2]\\
               &\quad \sigma_p=\sqrt{w_1^2\sigma_1^2+w_2^2\sigma_2^2+2w_1w_2\rho_{12}\sigma_1\sigma_2}\\
               &\quad w_1+w_2=1, \quad w_1,w_2\geq0
\end{align}

el cual es equivalente a

\begin{align}
\max_{w_1} &\quad \frac{w_1E[r_1]+(1-w_1)E[r_2]-r_f}{\sqrt{w_1^2\sigma_1^2+(1-w_1)^2\sigma_2^2+2w_1(1-w_1)\rho_{12}\sigma_1\sigma_2}}\\
\text{s.a.}    &\quad 0\leq w_1\leq1
\end{align}

**Actividad.**
El anterior es un problema de maximizar una función de una variable en un dominio cerrado. No debaría representar dificultad.

Encontrar la solución analítica a este problema.

Quien primero lo haga, y salga a explicarlo al tablero, le subo alguna tarea o quiz a 100.

Deben llegar a:

$$w_{1,EMV}=\frac{(E[r_1]-r_f)\sigma_2^2-(E[r_2]-r_f)\sigma_{12}}{(E[r_2]-r_f)\sigma_1^2+(E[r_1]-r_f)\sigma_2^2-((E[r_1]-r_f)+(E[r_2]-r_f))\sigma_{12}}.$$

Si nadie lo ha hecho en 30 min., procederé a hacerlo yo.

**Nota:**
- así como obtuvimos una expresión para el peso del portafolio de mínima varianza con dos activos, obtenemos una expresión para el peso del portafolio Eficiente en Media-Varianza.
- Estas actividades son sin duda un buen ejercicio, y se pueden replicar usando técnicas de varias variables (multiplicadores de Lagrange) cuando se tengan más de dos activos.
- Sin embargo, la complejidad del problema crece considerablemente con el número de variables, y la solución analítica deja de ser viable cuando mencionamos que un portafolio bien diversificado consta aproximadamente de 50-60 activos.
- En esos casos, este problema se soluciona con rutinas numéricas que hagan la optimización por nosotros.
- Por eso, les enseño cómo resolver este problema con optimizadores numéricos, porque son una solución viable y escalable a más variables.

## 3. Ejemplo ilustrativo.

Retomamos el ejemplo de mercados de acciones en los países integrantes del $G5$: EU, RU, Francia, Alemania y Japón.

```python
# Importamos pandas y numpy
import pandas as pd
import numpy as np
```

```python
# Resumen en base anual de rendimientos esperados y volatilidades
annual_ret_summ = pd.DataFrame(columns=['EU', 'RU', 'Francia', 'Alemania', 'Japon'], index=['Media', 'Volatilidad'])
annual_ret_summ.loc['Media'] = np.array([0.1355, 0.1589, 0.1519, 0.1435, 0.1497])
annual_ret_summ.loc['Volatilidad'] = np.array([0.1535, 0.2430, 0.2324, 0.2038, 0.2298])

annual_ret_summ.round(4)
```

      EU
      RU
      Francia
      Alemania
      Japon

      Media
      0.1355
      0.1589
      0.1519
      0.1435
      0.1497

      Volatilidad
      0.1535
      0.243
      0.2324
      0.2038
      0.2298

```python
# Matriz de correlación
corr = pd.DataFrame(data= np.array([[1.0000, 0.5003, 0.4398, 0.3681, 0.2663],
                                    [0.5003, 1.0000, 0.5420, 0.4265, 0.3581],
                                    [0.4398, 0.5420, 1.0000, 0.6032, 0.3923],
                                    [0.3681, 0.4265, 0.6032, 1.0000, 0.3663],
                                    [0.2663, 0.3581, 0.3923, 0.3663, 1.0000]]),
                    columns=annual_ret_summ.columns, index=annual_ret_summ.columns)
corr.round(4)
```

      EU
      RU
      Francia
      Alemania
      Japon

      EU
      1.0000
      0.5003
      0.4398
      0.3681
      0.2663

      RU
      0.5003
      1.0000
      0.5420
      0.4265
      0.3581

      Francia
      0.4398
      0.5420
      1.0000
      0.6032
      0.3923

      Alemania
      0.3681
      0.4265
      0.6032
      1.0000
      0.3663

      Japon
      0.2663
      0.3581
      0.3923
      0.3663
      1.0000

Supondremos, además, que la tasa libre de riesgo es $r_f=5\%$.

```python
# Tasa libre de riesgo
rf = 0.05
```

Entonces, supondremos que tenemos disponibles los activos correspondientes a los mercados de acciones de EU y Japón, y en adición el activo libre de riesgo.

#### 1. Construir la frontera de mínima varianza

```python
# Vector de w variando entre 0 y 1 con n pasos
n = 101
w = np.linspace(0, 1, n)
# Rendimientos esperados individuales
# Activo1: EU, Activo2:Japon
er1 = annual_ret_summ.iloc[0, 0]
er2 = annual_ret_summ.iloc[0, 4]
# Volatilidades individuales
s1 = annual_ret_summ.iloc[1, 0]
s2 = annual_ret_summ.iloc[1, 4]
# Correlacion
r12 = corr.loc["EU", "Japon"]
# Covarianza
s12 = r12 * s1 * s2
```

```python
# DataFrame de portafolios:
# 1. Índice: i
# 2. Columnas 1-2: w, 1-w
# 3. Columnas 3-4: E[r], sigma
# 4. Columna 5: Sharpe ratio
portafolios = pd.DataFrame({"w": w,
                            "1-w": 1 - w,
                            "Media": w * er1 + (1 - w) * er2,
                            "Vol": ((w * s1)**2 + ((1 - w) * s2)**2 + 2 * w *(1 - w) * s12)**0.5})
portafolios["RS"] = (portafolios["Media"] - rf) / portafolios["Vol"]
```

```python
portafolios.head()
```

      w
      1-w
      Media
      Vol
      RS

      0
      0.00
      1.00
      0.149700
      0.229800
      0.433856

      1
      0.01
      0.99
      0.149558
      0.227916
      0.436820

      2
      0.02
      0.98
      0.149416
      0.226041
      0.439814

      3
      0.03
      0.97
      0.149274
      0.224176
      0.442839

      4
      0.04
      0.96
      0.149132
      0.222322
      0.445894

```python
# Importar librerías de gráficos
from matplotlib import pyplot as plt
```

```python
# Gráfica de dispersión de puntos coloreando
# de acuerdo a RS
plt.figure(figsize=(6, 4))
plt.scatter(portafolios["Vol"], portafolios["Media"], c=portafolios["RS"], cmap="RdYlBu")
plt.xlabel("Volatilidad $\sigma$")
plt.ylabel("Rendimiento esperado $E[r]$")
plt.colorbar()
plt.grid()
plt.ylim([0.135, 0.15])
```

#### 2. Encontrar el portafolio que maximiza el radio de Sharpe (EMV)

Primero, encontramos este portafolio con la fórmula que obtuvimos:

$$w_{1,EMV}=\frac{(E[r_1]-r_f)\sigma_2^2-(E[r_2]-r_f)\sigma_{12}}{(E[r_2]-r_f)\sigma_1^2+(E[r_1]-r_f)\sigma_2^2-((E[r_1]-r_f)+(E[r_2]-r_f))\sigma_{12}}.$$

```python
# Fórmula que obtuvimos
num = (er1 - rf) * s2**2 - (er2 - rf) * s12
den = (er2 - rf) * s1**2 + (er1 - rf) * s2**2 - (er1 - rf + er2 - rf) * s12
w_emv = num / den
w_emv, 1 - w_emv
```

    (0.6983139170512034, 0.3016860829487966)

Ahora sí, con la función scipy.optimize.minimize

```python
# Importar la función minimize del módulo optimize
from scipy.optimize import minimize
```

```python
# Función objetivo (-RS)
def menos_rs(w, er1, er2, s1, s2, s12, rf):
    erp = w * er1 + (1 - w) * er2
    sp = ((w * s1)**2 + ((1 - w) * s2)**2 + 2 * w *(1 - w) * s12)**0.5
    rs = (erp - rf) / sp
    return -rs
```

```python
# Dato inicial
w0 = 0.5
# Cotas de las variables
bnd = ((0, 1),)
```

```python
# Optimización numérica
emv = minimize(fun=menos_rs,
               x0=w0,
               args=(er1, er2, s1, s2, s12, rf),
               bounds=bnd)
# Resultado
emv
```

          fun: array([-0.63087253])
     hess_inv: <1x1 LbfgsInvHessProduct with dtype=float64>
          jac: array([3.39728246e-06])
      message: b'CONVERGENCE: NORM_OF_PROJECTED_GRADIENT_<=_PGTOL'
         nfev: 10
          nit: 4
       status: 0
      success: True
            x: array([0.69831582])

```python
w_emv_ = emv.x
w_emv_, w_emv
```

    (array([0.69831582]), 0.6983139170512034)

Con lo anterior, podemos obtener datos de rendimiento esperado y volatilidad del portafolio EMV

```python
# Rendimiento esperado y volatilidad del portafolio EMV
erp_emv = w_emv * er1 + (1 - w_emv) * er2
sp_emv = ((w_emv * s1)**2 + ((1 - w_emv) * s2)**2 + 2 * w_emv *(1 - w_emv) * s12)**0.5
rs_emv = (erp_emv - rf) / sp_emv
```

```python
erp_emv, sp_emv, rs_emv
```

    (0.13978394237787292, 0.14231708951606933, 0.6308725303698346)

```python
# Gráfica de dispersión de puntos coloreando
# de acuerdo a SR, y portafolio EMV
plt.figure(figsize=(6, 4))
plt.scatter(portafolios["Vol"], portafolios["Media"], c=portafolios["RS"], cmap="RdYlBu", label="Frontera Min. Var.")
plt.plot(sp_emv, erp_emv, '*g', ms=20, label="Portafolio EMV")
plt.xlabel("Volatilidad $\sigma$")
plt.ylabel("Rendimiento esperado $E[r]$")
plt.colorbar()
plt.grid()
plt.ylim([0.135, 0.15])
plt.legend(loc="upper left", bbox_to_anchor=(1.3, 1))
```

#### 3. Construir LAC

Ahora, dibujamos la LAC, combinando el portafolio EMV con el activo libre de riesgo:

```python
# Vector de wp variando entre 0 y 1.5 con n pasos
wp = np.linspace(0, 1.5, 101)
```

```python
# DataFrame de CAL:
# 1. Índice: i
# 2. Columnas 1-2: wp, wrf
# 3. Columnas 3-4: E[r], sigma
# 4. Columna 5: Sharpe ratio
lac = pd.DataFrame({"wp": wp,
                    "wrf": 1 - wp,
                    "Media": wp * erp_emv + (1 - wp) * rf,
                    "Vol": wp * sp_emv})
lac["RS"] = (lac["Media"] - rf) / lac["Vol"]
```

```python
lac.head(10)
```

      wp
      wrf
      Media
      Vol
      RS

      0
      0.000
      1.000
      0.050000
      0.000000
      NaN

      1
      0.015
      0.985
      0.051347
      0.002135
      0.630873

      2
      0.030
      0.970
      0.052694
      0.004270
      0.630873

      3
      0.045
      0.955
      0.054040
      0.006404
      0.630873

      4
      0.060
      0.940
      0.055387
      0.008539
      0.630873

      5
      0.075
      0.925
      0.056734
      0.010674
      0.630873

      6
      0.090
      0.910
      0.058081
      0.012809
      0.630873

      7
      0.105
      0.895
      0.059427
      0.014943
      0.630873

      8
      0.120
      0.880
      0.060774
      0.017078
      0.630873

      9
      0.135
      0.865
      0.062121
      0.019213
      0.630873

```python
# Gráfica de dispersión de puntos coloreando
# de acuerdo a SR, portafolio EMV y LAC
plt.figure(figsize=(6, 4))
plt.scatter(portafolios["Vol"], portafolios["Media"], c=portafolios["RS"], cmap="RdYlBu", label="Frontera Min. Var.")
plt.plot(sp_emv, erp_emv, '*g', ms=20, label="Portafolio EMV")
plt.plot(lac["Vol"], lac["Media"], "--k", lw=3, label="LAC")
plt.xlabel("Volatilidad $\sigma$")
plt.ylabel("Rendimiento esperado $E[r]$")
plt.colorbar()
plt.grid()
#plt.axis([0.13, 0.20, 0.135, 0.15])
plt.legend(loc="upper left", bbox_to_anchor=(1.3, 1))
```

#### 4. Combinación óptima de acuerdo a preferencias

Con los datos anteriores, y la caracterización de aversión al riesgo, se escoge la combinación óptima entre el portafolio EMV y el activo libre de riesgo de acuerdo a:

$$w^\ast=\frac{E[r_s-r_f]}{\gamma\sigma_s^2}.$$

```python
# Para gamma=7
g = 5
w_opt = (erp_emv - rf) / (g * sp_emv**2)
w_opt, 1 - w_opt
```

    (0.8865731199465, 0.11342688005349999)

```python
plt.figure(figsize=(6, 4))
plt.scatter(portafolios["Vol"], portafolios["Media"], c=portafolios["RS"], cmap="RdYlBu", label="Frontera Min. Var.")
plt.plot(sp_emv, erp_emv, '*g', ms=20, label="Portafolio EMV")
plt.plot(lac["Vol"], lac["Media"], "--k", lw=3, label="LAC")
plt.plot(w_opt * sp_emv, w_opt * erp_emv + (1 - w_opt) * rf, '*r', ms=20, label="Portafolio final")
plt.xlabel("Volatilidad $\sigma$")
plt.ylabel("Rendimiento esperado $E[r]$")
plt.colorbar()
plt.grid()
#plt.axis([0.13, 0.20, 0.135, 0.15])
plt.legend(loc="upper left", bbox_to_anchor=(1.3, 1))
```

```python
# Ponderaciones finales: índice de EU, índice de Japón, tasa libre de riesgo
print(f"Ponderación a invertir en EU: {w_opt * w_emv}")
print(f"Ponderación a invertir en JP: {w_opt * (1 - w_emv)}")
print(f"Ponderación a invertir en rf: {1 - w_opt}")
```

    Ponderación a invertir en EU: 0.6191063481421468
    Ponderación a invertir en JP: 0.26746677180435324
    Ponderación a invertir en rf: 0.11342688005349999

Created with Jupyter by Esteban Jiménez Rodríguez.


```python
%load_ext autoreload
%autoreload 2

# Needed imports
from abtest import utils # local utils; see utils.py
import pandas as pd
import numpy as np

%matplotlib inline
from matplotlib import pyplot as plt
plt.style.use('ggplot')
import seaborn as sns

# Some settings for the notebook
%precision 5
np.set_printoptions(precision=4)
pd.set_option('precision',4)
```

## Motivations

In an A/B test we observe the behavior of the world (customers) given two different treatments.
Common example is the case where two groups of users are shown two different variants of some web page and their reaction (e.g. subscription rate) is measured.
Ultimately, the question is which variant performs better.
In practice, normally you should assign a variant to each customer randomly, thus split your traffic.
There are many subtle points and issues you should pay attention, but are skipped in this tutorial.
Traditionally, the notions of *control* and *variant* are used where the former is the current implementation of the system and the latter is the new feature/change.

First, to get better intuition, simulate $M$ identical experiments where each variant is shown to $N$ users.

```python
M = 3000 # Number of experiments
N = 2000 # Number of samples per experiment

# For reproducibility, we set a seed which we use to generate seeds
# for each simulation
np.random.seed(42)
# Generate M seeds, one for each experiment
seeds = np.random.randint(0, 2**32-1, size=(M))
```

In reality there are three possible case:

* The variant and the control perform the same
* The variant performs better than the control (i.e. you found an improvement)
* The control performs better than the variant (i.e. you found a deterioration)

The key problem is the following:

> The sample for the variant and for the control groups will (almost) always yield one of the last two!

This, in turn, raises the question how can you tell whether your variant had no influence or not, and if it did in which direction.
In what follows, `control_cr` (or $p_c$) and `variant_cr` (or $p_v$) are the *conversion rates* or *proportions* of the control and variant groups, respectively.
From the A/B test, our aim is to learn something about these two numbers.
Start with the same proportion for both groups; that is simulate the case where the variant had no influence.

*Note*: The notions "proportion" and "conversion rate (CR)" are used interchangeably.

```python
control_cr = 0.3 # 30%
variant_cr = 0.3 # 30%
experiments = [
    utils.generate_experiment(
        seed=seed, N=N, control_cr=control_cr, variant_cr=variant_cr
    ) for seed in seeds]
```

For each simulation, compute the signed difference between $p_c$ and $p_v$:

```python
cr_diffs = pd.Series([exp.CR_pct.diff()[-1] for exp in experiments])
sns.distplot(cr_diffs, bins=20, kde=True, rug=False)
plt.title("Distribution of proportions differences")
plt.xlabel('Difference')
plt.ylabel('%');
```

It comes as no surprise: the difference between the proportions is hardly every zero.
Following computation counts in how many simulations $$\|p_c - p_v\| < 0.05$$

```python
portion_almost_zero_diff = np.abs(cr_diffs)[np.abs(cr_diffs) < 0.05].shape[0] / M
portion_almost_zero_diff
```

    0.02100

For only {{portion_almost_zero_diff}} of the experiments the difference between $p_c$ and $p_v$ that you witness is smaller than $0.05$.
In other words, for about 98% of the experiments you will witness a larger difference; **even though there is no real one!!**
So, merely comparing the results of a single test doesn't tell you anything.
You will need to employ a more powerful approach.
This approach is called *hypothesis testing*.

## Defining the hypothesis - two sided case

Assume you want to first know whether the real proportion in the variant set, $p_v$, is different from the one of the control.
This can, and should, be rigorously defined as follow

$$\begin{align}
H_0:\quad & p_c = p_v \\
H_a:\quad & p_c \neq p_v
\end{align}$$

$H_0$ and $H_a$ should be read as the null and alternative hypotheses.
As you noticed in the examples above, it is almost certain that the proportion you will witness will not be equal.
However, in this settings, you can ask yourself the following question:

> How likely is it to witness even more extreme results? How likely is it that in another experiment you would have $\bar{p}_c$ and $\bar{p}_v$ such that $$\|\bar{p}_c - \bar{p}_v \| \geq \|p_c - p_v\|$$
> assuming that $H_0$ holds

Here, $\bar{p}_c$ and $\bar{p}_v$ are the *observed* proportions of the control and variant groups, respectively.
If you could compute this likelihood, and it would be extremely small, you would feel comfortable to *reject* the null hypothesis $H_0$ and accept the alternative one.
To enhance the intuition, you can run a simulation.
First, run a base experiment and record the difference between the proportions.
Next, run $M$ more simulations and compare the differences between the proportions in each of the simulation and the base difference you noted.

`base_exp` will hold the aggregated results of the base experiment.
`base_cr_diff` will hold the absolute difference between the proportions observed in each group

```python
control_cr = 0.3 # 30%
variant_cr = 0.3 # 30%
base_exp = utils.generate_experiment(seed=1984, N=N, control_cr=control_cr, variant_cr=variant_cr)
base_cr_diff = np.abs(base_exp.CR_pct.diff()[1])
base_cr_diff
```

    1.60000

Next, run $M$ more simulations and count in how many of them you witness a larger difference.
**Note that in all runs you should assume that $H_0$ holds.**
In practice, this means that `control_cr` and `variant_cr` should be the same.

```python
np.random.seed(42)
seeds = np.random.randint(0, 2**32-1, size=(M))
control_cr = 0.3 # 30%
variant_cr = 0.3 # 30%
experiments = [
    utils.generate_experiment(
        seed=seed, N=N, control_cr=control_cr, variant_cr=variant_cr
    ) for seed in seeds]
cr_diffs = pd.Series([exp.CR_pct.diff()[-1] for exp in experiments])

print(
    "In ~{}% of the simulation the difference between the proportions\n"
    "is at least {:03.3f} (the one in the base experiment)".format(
        np.round(100 * cr_diffs[np.abs(cr_diffs) > base_cr_diff].shape[0] / M, decimals=2),
        base_cr_diff
    ))
```

    In ~26.47% of the simulation the difference between the proportions
    is at least 1.600 (the one in the base experiment)

For the random seed $42$, you can see that witnessing a difference of $1.6$ or larger between the proportions is to be expected in roughly $1/4$ of the cases.
Not so extraordinary.

```python
ax = sns.distplot(cr_diffs, bins=20, kde=True, rug=False)
x, y = ax.lines[0].get_data()
mask = (x > base_cr_diff)
x, y = x[mask], y[mask]
ax.fill_between(x, y1=y, alpha=0.5, facecolor='blue')
x, y = ax.lines[0].get_data()
mask = (x < -base_cr_diff)
x, y = x[mask], y[mask]
ax.fill_between(x, y1=y, alpha=0.5, facecolor='blue')
plt.axvline(base_cr_diff, color='gray')
plt.axvline(-base_cr_diff, color='gray')
plt.xlabel('Proportions differences');
```

Now you should change the base experiment, in particular, take different proportions.
This simulates a situation when the variant introduces a change in the proportion.

```python
control_cr = 0.3 # 30%
variant_cr = 0.32 # 32%
base_exp_2 = utils.generate_experiment(seed=1984, N=N, control_cr=control_cr, variant_cr=variant_cr)
base_2_cr_diff = np.abs(base_exp_2.CR_pct.diff()[1])
base_2_cr_diff
```

    3.65000

Now, have a look in how many cases you witness an even more extreme difference assuming the underlying proportions are the same.

```python
print(
    "In ~{}% of the simulation the difference between the proportions\n"
    "is at least {:03.3f} (the one in the base experiment)".format(
        np.round(100 * cr_diffs[np.abs(cr_diffs) > base_2_cr_diff].shape[0] / M, decimals=2),
        base_2_cr_diff
    ))

ax = sns.distplot(cr_diffs, bins=20, kde=True, rug=False)
x, y = ax.lines[0].get_data()
mask = x > base_2_cr_diff
x, y = x[mask], y[mask]
ax.fill_between(x, y1=y, alpha=0.5, facecolor='blue')
x, y = ax.lines[0].get_data()
mask = x < -base_2_cr_diff
x, y = x[mask], y[mask]
ax.fill_between(x, y1=y, alpha=0.5, facecolor='blue')
plt.axvline(base_2_cr_diff, color='gray')
plt.axvline(-base_2_cr_diff, color='gray')
plt.title('Proportions differences');
```

In the next parts you will learn how to derive similar understandings but using merely the results from a *single* experiment.


```python

from sympy import *
from IPython.display import display, Image
#from IPython.lib.latextools import latex_to_png

import numpy as np
from scipy import signal
from matplotlib import pyplot as plt

np.set_printoptions(precision=4)

init_printing()

# Constants
T = 300 # K

kB_eV = 8.6173303e-5 # eV/K
hbar_eV = 6.582119514e-16 # eV*s

kB = 1.38064852e-23 # J/K
hbar = 6.626070040e-34 	# J*s

Na = 6.022140857e23 # Avogadro's number /mol
F = 96485.33289 # Faraday's constant C/mol
R = 8.3144598 # gas constant J/(K *mol)

eps_o = 8.854187817620e-12 # permittivity of free space F/m
eps_r = # relativie permittivity

#n = linspace(1e17, 1e19, 100) # cm^-3

c = 3e8  #speed of light m/s
m0 =9.10938356e-31 # electron rest mass kg
#m0 = 0.5109989461e6/(c^2) # electron mass eV*s^2/cm^2

Eg = 3.781e-19 # bandgap J
mcc = 0.32 # effective mass/m0

p_matrix = sqrt(m0*Eg/2*(1/mcc-1))

print p_matrix

print 0.925/(0.05*19999.0 + 0.925)
print 0.95*19999.0/(0.95*19999.0 + 0.075)
print 0.99*19999.0/(1.0+0.99*19998)
```

    6.04940000685047e-25
    0.000924191332584
    0.99999605245
    0.999999494924

```python

```


Határozzuk meg az alábbi ábrán látható tartó súlypontvonalának eltolódását leíró $v\left(x\right)$ függvényt végeselemes módszer használatával, síkbeli egyenes gerendalemek alkalmazásával.
Vizsgáljuk meg a végeselemes megoldással kapott hajlítónyomatéki igénybevétel hibáját az egyes szakaszokon.
Határozzuk meg az $x = a/2$ keresztmetszetben a hajlítónyomatéki igénybevétel nagyságát 2, illetve 3 síkbeli egyenes gerendaelem alkalmazásával.
A tartók két különböző átmérőjű ($d_1 = 2d$, illetve $d_2 = d$) kör keresztmetszetű tartókból
vannak összeépítve.
A tartók anyaga lineárisan rugalmas, homogén, izotrop. A $d_1$ átmérőjű rész rugalmassági modulusza $E$, míg a $d_2$ átmérővel rendelkező részé $4E$.

# Analitikus megoldás rugalmasszál DE. alapján

```python
import sympy as sp
import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt

sp.init_printing()
```

```python
MA, Ax, Ay, By = sp.symbols("M_A, A_x, A_y, B_y")
p0, F1, M1, = sp.symbols("p_0, F_1, M_1")
I, I1, I2, E, E1, E2, a, b, d, x = sp.symbols("I, I_1, I_2, E, E_1, E_2, a, b, d, x")
a, b, d, x = sp.symbols("a, b, d, x")
```

```python
Mh1 = MA - Ay*x - p0*x**2/2
Mh1
```

```python
Mh2 = MA - Ay*x - p0*a*(x-a/2) - F1*(x-a)
Mh2
```

```python
c1, c2, c3, c4 = sp.symbols("c1, c2, c3, c4")
```

```python
fi1 = sp.integrate(-Mh1/(I1*E1),x) + c1
v1 = sp.integrate(fi1,x) + c2
```

```python
fi2 = sp.integrate(-Mh2/(I2*E2),x) + c3
v2 = sp.integrate(fi2,x) + c4
```

```python
statFx = Ax
statFy = Ay + p0*a + F1 + By
statMz = MA + p0*a**2/2 + F1*a + M1 + By*(a+b)
```

```python
pf1 = fi1.subs([(x,0)])
pf2 = v1.subs([(x,0)])
pf3 = v2.subs([(x,a+b)])

if1 = (fi1 - fi2).subs([(x,a)])
if2 = (v1 - v2).subs([(x,a)])
```

```python
mego = sp.solve([statFx,statFy,statMz,pf1,pf2,pf3,if1,if2],[MA,Ax,Ay,By,c1,c2,c3,c4])
```

```python
adat=[(a,800e-3),(b,400e-3),(d,20e-3),(E,50e9),(F1,2500),(M1,-500),(p0,-5000)]
adat2 = [(E1,E),(E2,4*E),(I1, (2*d)**4*np.pi/64), (I2, (d)**4*np.pi/64)]
```

```python
MA.subs(mego).subs(adat2).subs(adat)
```

```python
Ay.subs(mego).subs(adat2).subs(adat)
```

```python
By.subs(mego).subs(adat2).subs(adat)
```

```python
v1num = sp.lambdify(x,v1.subs(mego).subs(adat2).subs(adat))
v1num(x)
```

```python
v2num = sp.lambdify(x,v2.subs(mego).subs(adat2).subs(adat))
v2num(x)
```

```python
fi1num = sp.lambdify(x,fi1.subs(mego).subs(adat2).subs(adat))
fi1num(x)
```

```python
fi2num = sp.lambdify(x,fi2.subs(mego).subs(adat2).subs(adat))
fi2num(x)
```

```python
Mh1num = sp.lambdify(x,Mh1.subs(mego).subs(adat2).subs(adat))
Mh1num(x)
```

```python
Mh2num = sp.lambdify(x,Mh2.subs(mego).subs(adat2).subs(adat))
Mh2num(x)
```

```python
x1Lista= np.linspace(0,float(a.subs(adat)), num = 20)
x1Lista
```

    array([ 0.        ,  0.04210526,  0.08421053,  0.12631579,  0.16842105,
            0.21052632,  0.25263158,  0.29473684,  0.33684211,  0.37894737,
            0.42105263,  0.46315789,  0.50526316,  0.54736842,  0.58947368,
            0.63157895,  0.67368421,  0.71578947,  0.75789474,  0.8       ])

```python
x2Lista = np.linspace(float(a.subs(adat)),float((a+b).subs(adat)), num = 20)
x2Lista
```

    array([ 0.8       ,  0.82105263,  0.84210526,  0.86315789,  0.88421053,
            0.90526316,  0.92631579,  0.94736842,  0.96842105,  0.98947368,
            1.01052632,  1.03157895,  1.05263158,  1.07368421,  1.09473684,
            1.11578947,  1.13684211,  1.15789474,  1.17894737,  1.2       ])

```python
xLista = np.concatenate((x1Lista,x2Lista))
```

```python
vLista = np.concatenate(([v1num(x) for x in x1Lista],[v2num(x) for x in x2Lista]))
```

```python
figv = plt.figure(num = 1, figsize=(16/2.54,10/2.54))
axv = figv.add_subplot(111)

axv.plot(xLista,vLista)

plt.xlabel(r"$x \, \left[\mathrm{m}\right]$")
plt.ylabel(r"$v \, \left[\mathrm{m}\right]$")

plt.grid()
plt.legend()
plt.show()
```

```python
fiLista = np.concatenate(([fi1num(x) for x in x1Lista],[fi2num(x) for x in x2Lista]))
```

```python
figfi = plt.figure(num = 2, figsize=(16/2.54,10/2.54))
axfi = figfi.add_subplot(111)

axfi.plot(xLista,fiLista)

plt.xlabel(r"$x \, \left[\mathrm{m}\right]$")
plt.ylabel(r"$\varphi \, \left[\mathrm{rad}\right]$")

plt.grid()
plt.legend()
plt.show()
```

```python
MhLista = np.concatenate(([Mh1num(x) for x in x1Lista],[Mh2num(x) for x in x2Lista]))
```

```python
figMh = plt.figure(num = 3, figsize=(16/2.54,10/2.54))
axMh = figMh.add_subplot(111)

axMh.plot(xLista,MhLista)

plt.xlabel(r"$x \, \left[\mathrm{m}\right]$")
plt.ylabel(r"$M_h \, \left[\mathrm{Nm}\right]$")

plt.grid()
plt.legend()
plt.show()
```

```python
V1 = sp.diff(-Mh1,x)
V1num = sp.lambdify(x,V1.subs(mego).subs(adat2).subs(adat))
V2 = sp.diff(-Mh2,x)
V2num = sp.lambdify(x,V2.subs(mego).subs(adat2).subs(adat))
```

```python
VLista = np.concatenate(([V1num(x) for x in x1Lista],[V2num(x) for x in x2Lista]))
```

```python
figV = plt.figure(num = 4, figsize=(16/2.54,10/2.54))
axV = figV.add_subplot(111)

axV.plot(xLista,VLista)

plt.xlabel(r"$x \, \left[\mathrm{m}\right]$")
plt.ylabel(r"$V \, \left[\mathrm{N}\right]$")

plt.grid()
plt.legend()
plt.show()
```


# QuTiP development notebook for testing new TD function callback signature

Copyright (C) 2011 and later, Paul D. Nation & Robert J. Johansson

Note: experimental and under development

```
%pylab inline
```

    Welcome to pylab, a matplotlib-based Python environment [backend: module://IPython.kernel.zmq.pylab.backend_inline].
    For more information, type 'help(pylab)'.

```
from qutip import *
```

```
from qutip.ipynbtools import HTMLProgressBar
from qutip.gui.progressbar import TextProgressBar
```

## Problem description

Consider the mean-field laser master equation on the form (Breuer and Petruccione)

$\displaystyle \dot\rho = -i [H, \rho]
+ 2\kappa \mathcal{D}[a]
+ W_{21} \mathcal{D}[\sigma_-]
+ W_{12} \mathcal{D}[\sigma_+]
+ g [{\rm Tr}[\sigma_-\rho] a^\dagger - {\rm Tr}[\sigma_+\rho] a, \rho]
+ g [{\rm Tr}[a^\dagger\rho] \sigma_- - {\rm Tr}[a\rho] \sigma_+, \rho] $

where the dissipator superoperator is

$\displaystyle \mathcal{D}[a] = a\rho a^\dagger - \frac{1}{2}a^\dagger a \rho - \frac{1}{2}\rho a^\dagger a,$

$W_{21}$ and $W_{12}$ are the atomic relaxation rate and pump rate, respectively, and ${\rm Tr}[A\rho]$ is the expectation value of the operator $A$ with respect to the density operator $\rho$.

The Hamiltonian is given by

$\displaystyle H = \omega a^\dagger a + \frac{1}{2}\omega\sigma_z$.

Except for the two last term, the above master equation is on standard Lindblad form and could, if $g = 0$, be solved with standard usage of the QuTiP solver `mesolve`.

We can write the master equation above on standard Lindblad form using an effective (nonhermitian?) Hamiltonian, corresponding to a nonlinear Schrodinger equation,

$\displaystyle \dot\rho = -i [H_{\rm eff}(\rho, t), \rho]
+ 2\kappa \mathcal{D}[a] + W_{21} \mathcal{D}[\sigma_-] + W_{12} \mathcal{D}[\sigma_+]$

where

$\displaystyle
H_{\rm eff}(\rho, t) = \omega a^\dagger a + \frac{1}{2}\omega\sigma_z
+ ig\left({\rm Tr}[\sigma_-\rho] a^\dagger - {\rm Tr}[\sigma_+\rho] a\right)
+ ig\left({\rm Tr}[a^\dagger\rho] \sigma_- - {\rm Tr}[a\rho] \sigma_+\right)
$

In QuTiP (development version required) we implement this effective Hamiltonian using the Python callback function format for time-dependent Hamiltonians, but to calculate $H_{\rm eff}(t)$ we need to have access to $\rho(t)$, and the standard QuTiP time-dependent function callback signature is

    def h_t(t, args):
        ...
        return h_eff

However, the solver `mesolve` that calls the callback function has access to $\rho(t)$ (or $\left|\psi(t)\right>$ for unitary dynamics), so it could change the callback function signature to

    def h_t(t, rho, args):
        ...
        return h_eff

To avoid breaking backwards compatibility in the API, the inclusion of rho in the function signature is optionally activated with a keyword argument (rhs_with_state=True) in Odeoptions.

### QuTiP implementation

```
N = 10

w = 1.0 * 2 * pi
g = 0.1 * 2 * pi
kappa = 0.005
W21 = 0.01
W12 = 0.05

tlist = linspace(0, 250, 250)

# cavity operators
a  = tensor(destroy(N), identity(2))
ad = tensor(create(N), identity(2))

# atomic operators
sz = tensor(identity(N), sigmaz())
sx = tensor(identity(N), sigmax())
sm = tensor(identity(N), destroy(2))
sp = tensor(identity(N), create(2))
```

```
#psi0 = tensor(fock(N, 0), fock(2, 0)) # start with the vacuum + ground state
psi0 = tensor(coherent(N, 0.5), fock(2, 0)) # start a small coherent sate + ground state
```

```
theta = 0.0

H = w * ad * a - 0.5 * w * (cos(2*theta) * sz + sin(2*theta) * sx)

H
```

\begin{equation}\text{Quantum object: dims = [[10, 2], [10, 2]], shape = [20, 20], type = oper, isHerm = True}\\[1em]\begin{pmatrix}-3.142 & 0.0 & 0.0 & 0.0 & 0.0 & \cdots & 0.0 & 0.0 & 0.0 & 0.0 & 0.0\\0.0 & 3.142 & 0.0 & 0.0 & 0.0 & \cdots & 0.0 & 0.0 & 0.0 & 0.0 & 0.0\\0.0 & 0.0 & 3.142 & 0.0 & 0.0 & \cdots & 0.0 & 0.0 & 0.0 & 0.0 & 0.0\\0.0 & 0.0 & 0.0 & 9.425 & 0.0 & \cdots & 0.0 & 0.0 & 0.0 & 0.0 & 0.0\\0.0 & 0.0 & 0.0 & 0.0 & 9.425 & \cdots & 0.0 & 0.0 & 0.0 & 0.0 & 0.0\\\vdots & \vdots & \vdots & \vdots & \vdots & \ddots & \vdots & \vdots & \vdots & \vdots & \vdots\\0.0 & 0.0 & 0.0 & 0.0 & 0.0 & \cdots & 47.124 & 0.0 & 0.0 & 0.0 & 0.0\\0.0 & 0.0 & 0.0 & 0.0 & 0.0 & \cdots & 0.0 & 47.124 & 0.0 & 0.0 & 0.0\\0.0 & 0.0 & 0.0 & 0.0 & 0.0 & \cdots & 0.0 & 0.0 & 53.407 & 0.0 & 0.0\\0.0 & 0.0 & 0.0 & 0.0 & 0.0 & \cdots & 0.0 & 0.0 & 0.0 & 53.407 & 0.0\\0.0 & 0.0 & 0.0 & 0.0 & 0.0 & \cdots & 0.0 & 0.0 & 0.0 & 0.0 & 59.690\\\end{pmatrix}\end{equation}

```
e_ops = [ad * a, sm.dag() * sm, sm, sp]
```

```
c_ops = [sqrt(2*kappa) * a, sqrt(W21) * sm, sqrt(W12) * sp]
```

## Solve without mean-field contribution

```
result = mesolve(H, psi0, tlist, c_ops, e_ops, options=Odeoptions(store_final_state=True))
```

```
fig, axes = subplots(2,1)

axes[0].plot(result.times, result.expect[0], label=r'$a^\dagger a$')
axes[0].plot(result.times, result.expect[1], label=r'$\sigma_+\sigma_-$')
axes[0].set_ylim(-0.1, 1.1)
axes[0].legend();

axes[1].plot(result.times, abs(result.expect[2]), label=r'$\sigma_-$')
axes[1].plot(result.times, abs(result.expect[3]), label=r'$\sigma_+$')
axes[1].set_ylim(-0.1, 1.1)
axes[1].legend();
```

#### Visualize the cavity final state

```
wigner_fock_distribution(ptrace(result.final_state, 0));
```

### Threshold

```
d = (W12 - W21) / (W12 + W21)

d
```

    0.6666666666666666

```
gamma = 0.5 * (W12 + W21)

gamma
```

    0.030000000000000002

```
C = d * (g/(2*pi)) ** 2 / (gamma * kappa)

C
```

    44.44444444444445

## Solve with mean-field contribution

$\displaystyle
H_{\rm eff}(\rho, t) = \omega a^\dagger a + \frac{1}{2}\omega\sigma_z
+ ig\left({\rm Tr}[\sigma_-\rho] a^\dagger - {\rm Tr}[\sigma_+\rho] a\right)
+ ig\left({\rm Tr}[a^\dagger\rho] \sigma_- - {\rm Tr}[a\rho] \sigma_+\right)
$

```
def Heff(t, rho, args):
    """ not an optimized implementation ... """
    q_rho = Qobj(vec2mat(rho))
    Heff = 1.0j * g * (expect(sm, q_rho) * ad - expect(sp, q_rho) * a) \
         + 1.0j * g * (expect(ad, q_rho) * sm - expect(a, q_rho) * sp)

    return args[0] + liouvillian_fast(Heff, []).data
```

```
sop_a_data = liouvillian_fast(a, []).data
sop_ad_data = liouvillian_fast(ad, []).data
sop_sm_data = liouvillian_fast(sm, []).data
sop_sp_data = liouvillian_fast(sp, []).data

sop_a_L_data = spre(a).data
sop_ad_L_data = spre(ad).data
sop_sm_L_data = spre(sm).data
sop_sp_L_data = spre(sp).data

def Heff_fast(t, rho, args):
    """ a somewhat more optimized version ... """

    phi_sm = expect_rho_vec1d(sop_sm_L_data, rho)
    phi_a = expect_rho_vec1d(sop_a_L_data, rho)

    Heff = 1.0j * g * (phi_sm * sop_ad_data  - conjugate(phi_sm) * sop_a_data) \
         + 1.0j * g * (conjugate(phi_a) * sop_sm_data - phi_a * sop_sp_data)

    return args[0] + Heff
```

```
result = mesolve(Heff_fast, psi0, tlist, c_ops, e_ops, args=[H],
                 options=Odeoptions(rhs_with_state=True, store_final_state=True),
                 progress_bar=TextProgressBar())
```

    Completed:  0.0%. Elapsed time:   0.00s. Est. remaining time: 00:00:00:00.
    Completed: 10.0%. Elapsed time:  12.88s. Est. remaining time: 00:00:01:55.
    Completed: 20.0%. Elapsed time:  26.35s. Est. remaining time: 00:00:01:45.
    Completed: 30.0%. Elapsed time:  39.72s. Est. remaining time: 00:00:01:32.
    Completed: 40.0%. Elapsed time:  54.83s. Est. remaining time: 00:00:01:22.
    Completed: 50.0%. Elapsed time:  72.08s. Est. remaining time: 00:00:01:12.
    Completed: 60.0%. Elapsed time:  89.05s. Est. remaining time: 00:00:00:59.
    Completed: 70.0%. Elapsed time: 106.85s. Est. remaining time: 00:00:00:45.
    Completed: 80.0%. Elapsed time: 124.90s. Est. remaining time: 00:00:00:31.
    Completed: 90.0%. Elapsed time: 143.07s. Est. remaining time: 00:00:00:15.
    Elapsed time: 159.90s

```
fig, ax = subplots()

ax.plot(result.times, result.expect[0], label=r'$a^\dagger a$')
ax.plot(result.times, result.expect[1], label=r'$\sigma_+\sigma_-$')
ax.set_ylim(-0.1, 3)
ax.legend();
```

#### Visualize the cavity final state

```
wigner_fock_distribution(ptrace(result.final_state, 0));
```

### Solve the same problem using the list-function format

```
def a_coeff(t, rho, args):
    return - 1.0j * g * expect_rho_vec1d(sop_sp_L_data, rho)

def ad_coeff(t, rho, args):
    return + 1.0j * g * expect_rho_vec1d(sop_sm_L_data, rho)

def sm_coeff(t, rho, args):
    return + 1.0j * g * expect_rho_vec1d(sop_ad_L_data, rho)

def sp_coeff(t, rho, args):
    return - 1.0j * g * expect_rho_vec1d(sop_a_L_data, rho)
```

```
result = mesolve([H, [a, a_coeff], [ad, ad_coeff], [sm, sm_coeff], [sp, sp_coeff]],
                 psi0, tlist, c_ops, e_ops, args={},
                 options=Odeoptions(rhs_with_state=True, store_final_state=True),
                 progress_bar=TextProgressBar())
```

    Completed:  0.0%. Elapsed time:   0.00s. Est. remaining time: 00:00:00:00.
    Completed: 10.0%. Elapsed time:  16.56s. Est. remaining time: 00:00:02:29.
    Completed: 20.0%. Elapsed time:  33.92s. Est. remaining time: 00:00:02:15.
    Completed: 30.0%. Elapsed time:  50.92s. Est. remaining time: 00:00:01:58.
    Completed: 40.0%. Elapsed time:  70.16s. Est. remaining time: 00:00:01:45.
    Completed: 50.0%. Elapsed time:  92.30s. Est. remaining time: 00:00:01:32.
    Completed: 60.0%. Elapsed time: 114.58s. Est. remaining time: 00:00:01:16.
    Completed: 70.0%. Elapsed time: 136.75s. Est. remaining time: 00:00:00:58.
    Completed: 80.0%. Elapsed time: 158.77s. Est. remaining time: 00:00:00:39.
    Completed: 90.0%. Elapsed time: 180.88s. Est. remaining time: 00:00:00:20.
    Elapsed time: 202.96s

```
fig, ax = subplots()

ax.plot(result.times, result.expect[0], label=r'$a^\dagger a$')
ax.plot(result.times, result.expect[1], label=r'$\sigma_+\sigma_-$')
ax.set_ylim(-0.1, 3)
ax.legend();
```

## Solve with mean-field contribution, but without dissipation (using sesolve)

```
tlist = linspace(0, 50, 150)
```

```
H_data = H.data
a_data = a.data
ad_data = ad.data
sm_data = sm.data
sp_data = sp.data

def Heff_func(t, psi, args):

    phi_sm = expect_psi(sm_data, psi[:, newaxis])
    phi_a  = expect_psi(a_data, psi[:, newaxis])
    phi_sp = expect_psi(sp_data, psi[:, newaxis])
    phi_ad = expect_psi(ad_data, psi[:, newaxis])

    Heff = 1.0j * g * (phi_sm * ad_data - phi_sp * a_data) \
         + 1.0j * g * (phi_ad * sm_data - phi_a * sp_data)

    return H_data + Heff
```

```
result = sesolve(Heff_func, psi0, tlist, e_ops,
                 options=Odeoptions(rhs_with_state=True, store_final_state=True),
                 progress_bar=TextProgressBar())
```

    Completed:  0.0%. Elapsed time:   0.00s. Est. remaining time: 00:00:00:00.
    Completed: 10.0%. Elapsed time:   1.82s. Est. remaining time: 00:00:00:16.
    Completed: 20.0%. Elapsed time:   3.62s. Est. remaining time: 00:00:00:14.
    Completed: 30.0%. Elapsed time:   5.42s. Est. remaining time: 00:00:00:12.
    Completed: 40.0%. Elapsed time:   7.22s. Est. remaining time: 00:00:00:10.
    Completed: 50.0%. Elapsed time:   9.03s. Est. remaining time: 00:00:00:09.
    Completed: 60.0%. Elapsed time:  10.84s. Est. remaining time: 00:00:00:07.
    Completed: 70.0%. Elapsed time:  12.67s. Est. remaining time: 00:00:00:05.
    Completed: 80.0%. Elapsed time:  14.48s. Est. remaining time: 00:00:00:03.
    Completed: 90.0%. Elapsed time:  16.32s. Est. remaining time: 00:00:00:01.
    Elapsed time:  18.13s

```
fig, ax = subplots()

ax.plot(result.times, real(result.expect[0]), label=r'$a^\dagger a$')
ax.plot(result.times, real(result.expect[1]), label=r'$\sigma_+\sigma_-$')
ax.set_ylim(-0.1, 1)
ax.legend();
```

### Solve the same problem with list function format

```
def a_coeff(t, psi, args):
    return - 1.0j * g * expect_psi(sp_data, psi[:, newaxis])

def ad_coeff(t, psi, args):
    return + 1.0j * g * expect_psi(sm_data, psi[:, newaxis])

def sm_coeff(t, psi, args):
    return + 1.0j * g * expect_psi(ad_data, psi[:, newaxis])

def sp_coeff(t, psi, args):
    return - 1.0j * g * expect_psi(a_data, psi[:, newaxis])
```

```
result = sesolve([H, [a, a_coeff], [a.dag(), ad_coeff], [sm, sm_coeff], [sp, sp_coeff]],
                 psi0, tlist, e_ops,
                 options=Odeoptions(rhs_with_state=True, store_final_state=True),
                 progress_bar=TextProgressBar())
```

    Completed:  0.0%. Elapsed time:   0.00s. Est. remaining time: 00:00:00:00.
    Completed: 10.0%. Elapsed time:   1.89s. Est. remaining time: 00:00:00:16.
    Completed: 20.0%. Elapsed time:   3.76s. Est. remaining time: 00:00:00:15.
    Completed: 30.0%. Elapsed time:   5.63s. Est. remaining time: 00:00:00:13.
    Completed: 40.0%. Elapsed time:   7.52s. Est. remaining time: 00:00:00:11.
    Completed: 50.0%. Elapsed time:   9.39s. Est. remaining time: 00:00:00:09.
    Completed: 60.0%. Elapsed time:  11.25s. Est. remaining time: 00:00:00:07.
    Completed: 70.0%. Elapsed time:  13.10s. Est. remaining time: 00:00:00:05.
    Completed: 80.0%. Elapsed time:  14.96s. Est. remaining time: 00:00:00:03.
    Completed: 90.0%. Elapsed time:  16.85s. Est. remaining time: 00:00:00:01.
    Elapsed time:  18.73s

```
fig, ax = subplots()

ax.plot(result.times, real(result.expect[0]), label=r'$a^\dagger a$')
ax.plot(result.times, real(result.expect[1]), label=r'$\sigma_+\sigma_-$')
ax.set_ylim(-0.1, 1)
ax.legend();
```

## Solve with mean-field contribution and dissipation using mcsolve

```
tlist = linspace(0, 250, 250)
```

```
result = mcsolve(Heff_func, psi0, tlist, c_ops, e_ops, ntraj=64,
                 options=Odeoptions(rhs_with_state=True, store_final_state=True, gui=False, num_cpus=4),
                 progress_bar=TextProgressBar())
```

    Completed:  1.6%. Elapsed time: 141.29s. Est. remaining time: 00:02:28:21.
    Completed: 10.9%. Elapsed time: 505.69s. Est. remaining time: 00:01:08:37.
    Completed: 20.3%. Elapsed time: 884.28s. Est. remaining time: 00:00:57:49.
    Completed: 31.2%. Elapsed time: 1122.07s. Est. remaining time: 00:00:41:08.
    Completed: 40.6%. Elapsed time: 1404.55s. Est. remaining time: 00:00:34:12.
    Completed: 50.0%. Elapsed time: 1624.73s. Est. remaining time: 00:00:27:04.
    Completed: 60.9%. Elapsed time: 1885.82s. Est. remaining time: 00:00:20:08.
    Completed: 70.3%. Elapsed time: 2173.70s. Est. remaining time: 00:00:15:17.
    Completed: 81.2%. Elapsed time: 2467.79s. Est. remaining time: 00:00:09:29.
    Completed: 90.6%. Elapsed time: 2683.16s. Est. remaining time: 00:00:04:37.
    Completed: 100.0%. Elapsed time: 2915.00s. Est. remaining time: 00:00:00:00.
    Elapsed time: 2915.10s

```
fig, ax = subplots()

ax.plot(result.times, real(result.expect[0]), label=r'$a^\dagger a$')
ax.plot(result.times, real(result.expect[1]), label=r'$\sigma_+\sigma_-$')
ax.set_ylim(-0.1, 3)
ax.legend();
```

#### Using the list function format and mcsolve

```
result = mcsolve([H, [a, a_coeff], [ad, ad_coeff], [sm, sm_coeff], [sp, sp_coeff]],
                 psi0, tlist, c_ops, e_ops, args={}, ntraj=512,
                 options=Odeoptions(rhs_with_state=True, store_final_state=True, gui=False, num_cpus=4),
                 progress_bar=TextProgressBar())
```

    Completed:  0.2%. Elapsed time:  37.77s. Est. remaining time: 00:05:21:39.
    Completed: 10.2%. Elapsed time: 518.60s. Est. remaining time: 00:01:16:27.
    Completed: 20.1%. Elapsed time: 1019.87s. Est. remaining time: 00:01:07:29.
    Completed: 30.1%. Elapsed time: 1520.25s. Est. remaining time: 00:00:58:54.
    Completed: 40.0%. Elapsed time: 2034.98s. Est. remaining time: 00:00:50:47.
    Completed: 50.0%. Elapsed time: 2513.66s. Est. remaining time: 00:00:41:53.
    Completed: 60.2%. Elapsed time: 3036.83s. Est. remaining time: 00:00:33:31.
    Completed: 70.1%. Elapsed time: 3589.60s. Est. remaining time: 00:00:25:29.
    Completed: 80.1%. Elapsed time: 4684.49s. Est. remaining time: 00:00:19:25.
    Completed: 90.0%. Elapsed time: 5647.46s. Est. remaining time: 00:00:10:24.
    Completed: 100.0%. Elapsed time: 6778.12s. Est. remaining time: 00:00:00:00.
    Elapsed time: 6778.18s

```
fig, ax = subplots()

ax.plot(result.times, real(result.expect[0]), label=r'$a^\dagger a$')
ax.plot(result.times, real(result.expect[1]), label=r'$\sigma_+\sigma_-$')
ax.set_ylim(-0.1, 3)
ax.legend();
```

### Software versions

```
from qutip.ipynbtools import version_table

version_table()
```

Software	Version

matplotlib	1.4.x

Python	3.3.1 (default, Apr 17 2013, 22:30:32)
[GCC 4.7.3]

SciPy	0.13.0.dev-d74fd00

Cython	0.19.1

QuTiP	2.3.0.dev-b354119

Numpy	1.8.0.dev-75cdf3d

IPython	1.0.dev

OS	posix [linux]

Thu Jul 04 15:30:06 2013 JST


## Configurations for Colab

```python
import sys
IN_COLAB = "google.colab" in sys.modules

if IN_COLAB:
    !apt install python-opengl
    !apt install ffmpeg
    !apt install xvfb
    !pip install pyvirtualdisplay
    from pyvirtualdisplay import Display

    # Start virtual display
    dis = Display(visible=0, size=(400, 400))
    dis.start()
```

# 05. Noisy Networks for Exploration

[M. Fortunato et al., "Noisy Networks for Exploration." arXiv preprint arXiv:1706.10295, 2017.](https://arxiv.org/pdf/1706.10295.pdf)

NoisyNet is an exploration method that learns perturbations of the network weights to drive exploration. The key insight is that a single change to the weight vector can induce a consistent, and potentially very complex, state-dependent change in policy over multiple time steps.

Firstly, let's take a look into a linear layer of a neural network with $p$ inputs and $q$ outputs, represented by

$$
y = wx + b,
$$

where $x \in \mathbb{R}^p$ is the layer input, $w \in \mathbb{R}^{q \times p}$, and $b \in \mathbb{R}$ the bias.

The corresponding noisy linear layer is defined as:

$$
y = (\mu^w + \sigma^w \odot \epsilon^w) x + \mu^b + \sigma^b \odot \epsilon^b,
$$

where $\mu^w + \sigma^w \odot \epsilon^w$ and $\mu^b + \sigma^b \odot \epsilon^b$ replace $w$ and $b$ in the first linear layer equation. The parameters $\mu^w \in \mathbb{R}^{q \times p}, \mu^b \in \mathbb{R}^q, \sigma^w \in \mathbb{R}^{q \times p}$ and $\sigma^b \in \mathbb{R}^q$ are learnable, whereas $\epsilon^w \in \mathbb{R}^{q \times p}$ and $\epsilon^b \in \mathbb{R}^q$ are noise random variables which can be generated by one of the following two ways:

1. **Independent Gaussian noise**: the noise applied to each weight and bias is independent, where each random noise entry is drawn from a unit Gaussian distribution. This means that for each noisy linear layer, there are $pq + q$ noise variables (for $p$ inputs to the layer and $q$ outputs).
2. **Factorised Gaussian noise:** This is a more computationally efficient way. It produces 2 random Gaussian noise vectors ($p, q$) and makes $pq + q$ noise entries by outer product as follows:

$$
\begin{align}
\epsilon_{i,j}^w &= f(\epsilon_i) f(\epsilon_j),\\
\epsilon_{j}^b &= f(\epsilon_i),\\
\text{where } f(x) &= sgn(x) \sqrt{|x|}.
\end{align}
$$

In all experiements of the paper, the authors used Factorised Gaussian noise, so we will go for it as well.

```python
import math
import os
from typing import Dict, List, Tuple

import gym
import matplotlib.pyplot as plt
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from IPython.display import clear_output
```

## Replay buffer

Please see *01.dqn.ipynb* for detailed description.

```python
class ReplayBuffer:
    """A simple numpy replay buffer."""

    def __init__(self, obs_dim: int, size: int, batch_size: int = 32):
        self.obs_buf = np.zeros([size, obs_dim], dtype=np.float32)
        self.next_obs_buf = np.zeros([size, obs_dim], dtype=np.float32)
        self.acts_buf = np.zeros([size], dtype=np.float32)
        self.rews_buf = np.zeros([size], dtype=np.float32)
        self.done_buf = np.zeros(size, dtype=np.float32)
        self.max_size, self.batch_size = size, batch_size
        self.ptr, self.size, = 0, 0

    def store(
        self,
        obs: np.ndarray,
        act: np.ndarray,
        rew: float,
        next_obs: np.ndarray,
        done: bool,
    ):
        self.obs_buf[self.ptr] = obs
        self.next_obs_buf[self.ptr] = next_obs
        self.acts_buf[self.ptr] = act
        self.rews_buf[self.ptr] = rew
        self.done_buf[self.ptr] = done
        self.ptr = (self.ptr + 1) % self.max_size
        self.size = min(self.size + 1, self.max_size)

    def sample_batch(self) -> Dict[str, np.ndarray]:
        idxs = np.random.choice(self.size, size=self.batch_size, replace=False)
        return dict(obs=self.obs_buf[idxs],
                    next_obs=self.next_obs_buf[idxs],
                    acts=self.acts_buf[idxs],
                    rews=self.rews_buf[idxs],
                    done=self.done_buf[idxs])

    def __len__(self) -> int:
        return self.size
```

## Noisy Layer

**References:**
- https://github.com/higgsfield/RL-Adventure/blob/master/5.noisy%20dqn.ipynb
- https://github.com/Kaixhin/Rainbow/blob/master/model.py

```python
class NoisyLinear(nn.Module):
    """Noisy linear module for NoisyNet.

    Attributes:
        in_features (int): input size of linear module
        out_features (int): output size of linear module
        std_init (float): initial std value
        weight_mu (nn.Parameter): mean value weight parameter
        weight_sigma (nn.Parameter): std value weight parameter
        bias_mu (nn.Parameter): mean value bias parameter
        bias_sigma (nn.Parameter): std value bias parameter

    """

    def __init__(self, in_features: int, out_features: int, std_init: float = 0.5):
        """Initialization."""
        super(NoisyLinear, self).__init__()

        self.in_features = in_features
        self.out_features = out_features
        self.std_init = std_init

        self.weight_mu = nn.Parameter(torch.Tensor(out_features, in_features))
        self.weight_sigma = nn.Parameter(
            torch.Tensor(out_features, in_features)
        )
        self.register_buffer(
            "weight_epsilon", torch.Tensor(out_features, in_features)
        )

        self.bias_mu = nn.Parameter(torch.Tensor(out_features))
        self.bias_sigma = nn.Parameter(torch.Tensor(out_features))
        self.register_buffer("bias_epsilon", torch.Tensor(out_features))

        self.reset_parameters()
        self.reset_noise()

    def reset_parameters(self):
        """Reset trainable network parameters (factorized gaussian noise)."""
        mu_range = 1 / math.sqrt(self.in_features)
        self.weight_mu.data.uniform_(-mu_range, mu_range)
        self.weight_sigma.data.fill_(
            self.std_init / math.sqrt(self.in_features)
        )
        self.bias_mu.data.uniform_(-mu_range, mu_range)
        self.bias_sigma.data.fill_(
            self.std_init / math.sqrt(self.out_features)
        )

    def reset_noise(self):
        """Make new noise."""
        epsilon_in = self.scale_noise(self.in_features)
        epsilon_out = self.scale_noise(self.out_features)

        # outer product
        self.weight_epsilon.copy_(epsilon_out.ger(epsilon_in))
        self.bias_epsilon.copy_(epsilon_out)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        """Forward method implementation.

        We don't use separate statements on train / eval mode.
        It doesn't show remarkable difference of performance.
        """
        return F.linear(
            x,
            self.weight_mu + self.weight_sigma * self.weight_epsilon,
            self.bias_mu + self.bias_sigma * self.bias_epsilon,
        )

    @staticmethod
    def scale_noise(size: int) -> torch.Tensor:
        """Set scale to make noise (factorized gaussian noise)."""
        x = torch.FloatTensor(np.random.normal(loc=0.0, scale=1.0, size=size))

        return x.sign().mul(x.abs().sqrt())
```

## Noisy Network

We use NoisyLinear for the last two FC layers, and there is a method to reset noise at every step.
These are the only differences from the example of *01.dqn.ipynb*.

```python
class Network(nn.Module):
    def __init__(self, in_dim: int, out_dim: int):
        """Initialization."""
        super(Network, self).__init__()

        self.feature = nn.Linear(in_dim, 128)
        self.noisy_layer1 = NoisyLinear(128, 128)
        self.noisy_layer2 = NoisyLinear(128, out_dim)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        """Forward method implementation."""
        feature = F.relu(self.feature(x))
        hidden = F.relu(self.noisy_layer1(feature))
        out = self.noisy_layer2(hidden)

        return out

    def reset_noise(self):
        """Reset all noisy layers."""
        self.noisy_layer1.reset_noise()
        self.noisy_layer2.reset_noise()
```

## DQN + NoisyNet Agent (w/o DuelingNet)

Here is a summary of DQNAgent class.

| Method           | Note                                                 |
| ---              | ---                                                  |
|select_action     | select an action from the input state.               |
|step              | take an action and return the response of the env.   |
|compute_dqn_loss  | return dqn loss.                                     |
|update_model      | update the model by gradient descent.                |
|target_hard_update| hard update from the local model to the target model.|
|train             | train the agent during num_frames.                   |
|test              | test the agent (1 episode).                          |
|plot              | plot the training progresses.                        |

In the paper, NoisyNet is used as a component of the Dueling Network Architecture, which includes Double-DQN and Prioritized Experience Replay. However, we don't implement them to simplify the tutorial. One thing to note is that NoisyNet is an alternertive to $\epsilon$-greedy method, so all $\epsilon$ related lines are removed. Please check all comments with *NoisyNet*.

```python
class DQNAgent:
    """DQN Agent interacting with environment.

    Attribute:
        env (gym.Env): openAI Gym environment
        memory (ReplayBuffer): replay memory to store transitions
        batch_size (int): batch size for sampling
        target_update (int): period for target model's hard update
        gamma (float): discount factor
        dqn (Network): model to train and select actions
        dqn_target (Network): target model to update
        optimizer (torch.optim): optimizer for training dqn
        transition (list): transition information including
                           state, action, reward, next_state, done
    """

    def __init__(
        self,
        env: gym.Env,
        memory_size: int,
        batch_size: int,
        target_update: int,
        gamma: float = 0.99,
    ):
        """Initialization.

        Args:
            env (gym.Env): openAI Gym environment
            memory_size (int): length of memory
            batch_size (int): batch size for sampling
            target_update (int): period for target model's hard update
            gamma (float): discount factor
        """
        # NoisyNet: All attributes related to epsilon are removed
        obs_dim = env.observation_space.shape[0]
        action_dim = env.action_space.n

        self.env = env
        self.memory = ReplayBuffer(obs_dim, memory_size, batch_size)
        self.batch_size = batch_size
        self.target_update = target_update
        self.gamma = gamma

        # device: cpu / gpu
        self.device = torch.device(
            "cuda" if torch.cuda.is_available() else "cpu"
        )
        print(self.device)

        # networks: dqn, dqn_target
        self.dqn = Network(obs_dim, action_dim).to(self.device)
        self.dqn_target = Network(obs_dim, action_dim).to(self.device)
        self.dqn_target.load_state_dict(self.dqn.state_dict())
        self.dqn_target.eval()

        # optimizer
        self.optimizer = optim.Adam(self.dqn.parameters())

        # transition to store in memory
        self.transition = list()

        # mode: train / test
        self.is_test = False

    def select_action(self, state: np.ndarray) -> np.ndarray:
        """Select an action from the input state."""
        # NoisyNet: no epsilon greedy action selection
        selected_action = self.dqn(
            torch.FloatTensor(state).to(self.device)
        ).argmax()
        selected_action = selected_action.detach().cpu().numpy()

        if not self.is_test:
            self.transition = [state, selected_action]

        return selected_action

    def step(self, action: np.ndarray) -> Tuple[np.ndarray, np.float64, bool]:
        """Take an action and return the response of the env."""
        next_state, reward, done, _ = self.env.step(action)

        if not self.is_test:
            self.transition += [reward, next_state, done]
            self.memory.store(*self.transition)

        return next_state, reward, done

    def update_model(self) -> torch.Tensor:
        """Update the model by gradient descent."""
        samples = self.memory.sample_batch()

        loss = self._compute_dqn_loss(samples)

        self.optimizer.zero_grad()
        loss.backward()
        self.optimizer.step()

        # NoisyNet: reset noise
        self.dqn.reset_noise()
        self.dqn_target.reset_noise()

        return loss.item()

    def train(self, num_frames: int, plotting_interval: int = 200):
        """Train the agent."""
        self.is_test = False

        state = self.env.reset()
        update_cnt = 0
        losses = []
        scores = []
        score = 0

        for frame_idx in range(1, num_frames + 1):
            action = self.select_action(state)
            next_state, reward, done = self.step(action)

            state = next_state
            score += reward

            # NoisyNet: removed decrease of epsilon

            # if episode ends
            if done:
                state = self.env.reset()
                scores.append(score)
                score = 0

            # if training is ready
            if len(self.memory) >= self.batch_size:
                loss = self.update_model()
                losses.append(loss)
                update_cnt += 1

                # if hard update is needed
                if update_cnt % self.target_update == 0:
                    self._target_hard_update()

            # plotting
            if frame_idx % plotting_interval == 0:
                self._plot(frame_idx, scores, losses)

        self.env.close()

    def test(self) -> None:
        """Test the agent."""
        self.is_test = True

        state = self.env.reset()
        done = False
        score = 0

        while not done:
            self.env.render()
            action = self.select_action(state)
            next_state, reward, done = self.step(action)

            state = next_state
            score += reward

        print("score: ", score)
        self.env.close()

    def _compute_dqn_loss(self, samples: Dict[str, np.ndarray]) -> torch.Tensor:
        """Return dqn loss."""
        device = self.device  # for shortening the following lines
        state = torch.FloatTensor(samples["obs"]).to(device)
        next_state = torch.FloatTensor(samples["next_obs"]).to(device)
        action = torch.LongTensor(samples["acts"].reshape(-1, 1)).to(device)
        reward = torch.FloatTensor(samples["rews"].reshape(-1, 1)).to(device)
        done = torch.FloatTensor(samples["done"].reshape(-1, 1)).to(device)

        # G_t   = r + gamma * v(s_{t+1})  if state != Terminal
        #       = r                       otherwise
        curr_q_value = self.dqn(state).gather(1, action)
        next_q_value = self.dqn_target(next_state).max(
            dim=1, keepdim=True
        )[0].detach()
        mask = 1 - done
        target = (reward + self.gamma * next_q_value * mask).to(self.device)

        # calculate dqn loss
        loss = F.smooth_l1_loss(curr_q_value, target)

        return loss

    def _target_hard_update(self):
        """Hard update: target <- local."""
        self.dqn_target.load_state_dict(self.dqn.state_dict())

    def _plot(
        self,
        frame_idx: int,
        scores: List[float],
        losses: List[float],
    ):
        """Plot the training progresses."""
        clear_output(True)
        plt.figure(figsize=(20, 5))
        plt.subplot(131)
        plt.title('frame %s. score: %s' % (frame_idx, np.mean(scores[-10:])))
        plt.plot(scores)
        plt.subplot(132)
        plt.title('loss')
        plt.plot(losses)
        plt.show()
```

## Environment

You can see the [code](https://github.com/openai/gym/blob/master/gym/envs/classic_control/cartpole.py) and [configurations](https://github.com/openai/gym/blob/master/gym/envs/__init__.py#L53) of CartPole-v0 from OpenAI's repository.

```python
# environment
env_id = "CartPole-v0"
env = gym.make(env_id)
```

## Set random seed

```python
seed = 777

def seed_torch(seed):
    torch.manual_seed(seed)
    if torch.backends.cudnn.enabled:
        torch.backends.cudnn.benchmark = False
        torch.backends.cudnn.deterministic = True

np.random.seed(seed)
seed_torch(seed)
env.seed(seed)
```

    [777]

## Initialize

```python
# parameters
num_frames = 20000
memory_size = 1000
batch_size = 32
target_update = 50

# train
agent = DQNAgent(env, memory_size, batch_size, target_update)
```

    cuda

## Train

```python
agent.train(num_frames)
```

## Test

Run the trained agent (1 episode).

```python
agent.env = gym.wrappers.Monitor(env, "videos", force=True)
agent.test()
```

    score:  200.0

## Render

```python
import base64
import glob
import io
import os

from IPython.display import HTML, display

def ipython_show_video(path: str) -> None:
    """Show a video at `path` within IPython Notebook."""
    if not os.path.isfile(path):
        raise NameError("Cannot access: {}".format(path))

    video = io.open(path, "r+b").read()
    encoded = base64.b64encode(video)

    display(HTML(
        data="""

        """.format(encoded.decode("ascii"))
    ))

list_of_files = glob.glob("videos/*.mp4")
latest_file = max(list_of_files, key=os.path.getctime)
print(latest_file)
ipython_show_video(latest_file)
```

    videos/openaigym.video.3.13818.video000000.mp4