# Chapter 5: Continuous Random Variables

This Jupyter notebook is the Python equivalent of the R code in section 5.9 R, pp. 253 - 255, [Introduction to Probability, Second Edition](https://www.crcpress.com/Introduction-to-Probability-Second-Edition/Blitzstein-Hwang/p/book/9781138369917), Blitzstein & Hwang.

----

```python
import numpy as np
```

## Python, SciPy and Matplotlib

In this section we will introduce continuous distributions in Python and SciPy, learn how to make basic plots, demonstrate the universality of the Uniform by simulation, and simulate arrival times in a Poisson process.

## Uniform, Normal, and Exponential distributions

For [continuous distributions in `scipy.stats`](https://docs.scipy.org/doc/scipy/reference/tutorial/stats/continuous.html), the `pdf` function gives the PDF, the `cdf` function gives the CDF, and the `rvs` function generates random numbers from the continuous distribution. This is in keeping with the application programming interface of the [discrete statistical distributions in `scipy.stats`](https://docs.scipy.org/doc/scipy/reference/tutorial/stats/discrete.html). Thus, we have the following functions:

### Uniform
[`scipy.stats.uniform`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.uniform.html#scipy.stats.uniform) provides functions for Uniform continuous random variables.
* To evaluate the $Unif(a, b)$ PDF at $x$, we use `uniform.pdf(x, a, b)`.
* For the CDF, we use `uniform.cdf(x, a, b)`.
* To generate $n$ realizations from the $Unif(a, b)$ distribution, we use `uniform.rvs(a, b, size=n)`.

```python
# seed the random number generator
np.random.seed(1597)

from scipy.stats import uniform

# to learn more about scipy.stats,uniform un-comment ouf the following line
#print(uniform.__doc__)

a = 0
b = 4
x = 3
n = 10

print('PDF of Unif({}, {}) evaluated at {} is {}\n'.format(a, b, x, uniform.pdf(x, a, b)))

print('CDF of Unif({}, {}) evaluated at {} is {}\n'.format(a, b, x, uniform.cdf(x, a, b)))

print('Generating {} realizations from Unif({}, {}):\n{}'.format(n, a, b, uniform.rvs(a, b, size=n)))
```

    PDF of Unif(0, 4) evaluated at 3 is 0.25

    CDF of Unif(0, 4) evaluated at 3 is 0.75

    Generating 10 realizations from Unif(0, 4):
    [3.24085002 2.48793265 0.69337744 3.98212677 3.53906749 0.19890551
     3.30252222 2.37991343 1.42342695 1.92222338]

### Normal
[`scipy.stats.norm`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.norm.html#scipy.stats.norm) provides functions for normal continuous random variables.

* To evaluate the $N(\mu, \sigma^2)$ PDF at $x$, we use `norm.pdf(x, loc, scale)`, where the parameter `loc` corresponds to $\mu$ and `scale` corresponds to standard deviation $\sigma$ (and **not** variance $\sigma^2$).
* For the CDF, we use `norm.cdf(x, loc, scale)`.
* To generate $n$ realizations from the $N(\mu, \sigma^2)$ distribution, we use `norm.rvs(loc, scale, size=n)`.

```python
np.random.seed(2584)

from scipy.stats import norm

# to learn more about scipy.stats,norm un-comment ouf the following line
#print(norm.__doc__)

mu = 0.0
sigma = 2.0
x = 1.5

print('PDF of N({}, {}) evaluated at {} is {}\n'.format(mu, sigma, x, norm.pdf(x, mu, sigma)))

print('CDF of N({}, {}) evaluated at {} is {}\n'.format(mu, sigma, x, norm.cdf(x, mu, sigma)))

print('Generating {} realizations from N({}, {}):\n{}'.format(n, mu, sigma, norm.rvs(mu, sigma, size=n)))
```

    PDF of N(0.0, 2.0) evaluated at 1.5 is 0.15056871607740221

    CDF of N(0.0, 2.0) evaluated at 1.5 is 0.7733726476231317

    Generating 10 realizations from N(0.0, 2.0):
    [-4.56890553  1.4163793  -0.72812069  1.80610792 -1.5542524   0.4551897
     -1.82794816  1.58959742 -3.26902153 -2.71002864]

☣ 5.9.1 (Normal parameters in `scipy.stats.norm`). Note that we have to input the standard deviation for the `scale` parameter value, not the variance! For example, to get the $N(10, 3)$ CDF at 12, we use`norm.cdf(12, 10, np.sqrt(3))`. Ignoring this is a common, disastrous coding error.

```python
loc = 10
var = 3
scale = np.sqrt(var)
x = 12

ans = norm.cdf(x, loc, scale)

print('N({}, {}) CDF at {} is {}'.format(loc, scale, x, ans))
```

    N(10, 1.7320508075688772) CDF at 12 is 0.8758934605050381

### Exponential
[`scipy.stats.expon`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.expon.html#scipy.stats.expon) provides functions for exponential continuous random variables. The probability density function for $Expo(\lambda)$ is:

\begin{align}
  f(x) &= e^{-x}
\end{align}

The probability density is defined in the “standardized” form. To shift and/or scale the distribution use the `loc` and `scale` parameters. Specifically, `expon.pdf(x, loc, scale)` is identically equivalent to `expon.pdf(y) / scale` with `y = (x - loc) / scale`.

* To evaluate the $Expo(\lambda)$ PDF at $x$, we use `expon.pdf(x, scale=1/lambda)`, where the $\lambda$ corresponds to `scale=1/lambd`.
* For the CDF, we use `expon.cdf(x, scale=1/lambd)`.
* To generate $n$ realizations from the $Expo(\lambda)$ distribution, we use `expon.rvs(scale=1/lambd, size=n)`.

```python
np.random.seed(4181)

from scipy.stats import expon

# to learn more about scipy.stats,expon un-comment ouf the following line
#print(expon.__doc__)

lambd = 2.1
x = 5

expon.pdf(x, scale=1/lambd)

print('PDF of Expo({}) evaluated at {} is {}\n'.format(lambd, x, expon.pdf(x, scale=1/lambd)))

print('CDF of Expo({}) evaluated at {} is {}\n'.format(lambd, x, expon.cdf(x, scale=1/lambd)))

print('Generating {} realizations from Expo({}):\n{}\n'.format(n, lambd, expon.rvs(scale=1/lambd, size=n)))
```

    PDF of Expo(2.1) evaluated at 5 is 5.782654363446904e-05

    CDF of Expo(2.1) evaluated at 5 is 0.9999724635506503

    Generating 10 realizations from Expo(2.1):
    [0.44907946 0.77554149 1.469014   0.15728342 0.92258566 0.54294386
     0.80347768 0.23157762 0.34759354 1.03891708]

Due to the importance of location-scale transformations for continuous distributions, SciPy has default parameter settings for each of these three families. The default for the Uniform [`scipy.stats.uniform`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.uniform.html#scipy.stats.uniform) is `loc=0, scale=1`, the default for the Normal [`scipy.stats.norm`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.norm.html#scipy.stats.norm) is `loc=0, scale=1`, and the default for the Exponential [`scipy.stats.expon`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.expon.html#scipy.stats.expon) is `loc=0, scale=1`.

For example, `uniform.pdf(0.5)`, with no additional inputs, evaluates the $Unif(0, 1)$ PDF at 0.5.

```python
uniform.pdf(0.5)
```

    1.0

`norm.rvs(size=10)`, with no additional inputs, generates 10 realizations from the $N (0, 1)$ distribution.

```python
norm.rvs(size=10)
```

    array([ 0.79839927, -0.69769073,  0.34055804, -0.22043838,  0.79782274,
           -2.60292799, -1.53866926,  0.93084378,  0.81590117, -0.42043989])

This means there are two ways to generate a $N(\mu, \sigma^2)$ random variable in SciPy. After choosing our values of $\mu$ and $\sigma$,

```python
mu = 1
sigma = 2
```

we can do either of the following:

```python
ans1 = norm.rvs(mu, sigma, size=1, random_state=42)
ans2 = mu + sigma * norm.rvs(size=1, random_state=42)

print('norm.rvs(mu, sigma, size=1) = {}'.format(ans1))
print('mu + sigma * norm.rvs(size=1) = {}'.format(ans2))
```

    norm.rvs(mu, sigma, size=1) = [1.99342831]
    mu + sigma * norm.rvs(size=1) = [1.99342831]

Either way, we end up generating a draw from the $N(\mu, \sigma^2)$ distribution.

## Plots in Matplotlib

Plotting in Python within a Jupyter notebook is done in [Matplotlib](https://matplotlib.org/).

First, we import `matplotlib.pyplot` module. To cut down on typing, we can alias that imported `matplotlib.pyplot` module reference as `plt` for short. We then invoke the `%matplotlib inline%` Jupyter notebook magic command in order to create Matplotlib graphs right in your notebook.

```python
import matplotlib.pyplot as plt

%matplotlib inline
```

Here is how we can create a plot of the standard Normal PDF from −3 to 3 using Matplotlib in Jupyter notebook.

[`numpy.linspace`](https://docs.scipy.org/doc/numpy-1.15.0/reference/generated/numpy.linspace.html) creates an evenly-spaced numerical sequence, ranging from `start` to `stop`. The third function parameter `num` lets you specify how many numbers in the sequence to generate.

We use `numpy.linspace(-3, 3, 1000)` to generate 1000 numbers ranging from -3 to 3 as our `x` values in the graph; with 1000 values in the sequence, the curve looks very smooth. If we were to choose `num=20`, the piecewise linearity would become very apparent.

```python
x = np.linspace(-3, 3, 1000)
```

We can then call `norm.pdf(x)` to generate the `y` values in the graph, and pass both the `x` and `y` values to [`matplotlib.pyplot.plot`](https://matplotlib.org/api/_as_gen/matplotlib.pyplot.plot.html). This will render a graph of $N(0,1)$ from -3 to 3. `pyplot.show` will display the graph.

```python
plt.plot(x, norm.pdf(x))
plt.show()
```

Alternately, we could "freeze" an instance of the `scipy.stats.norm` distribution object to fix the shape, location and scale parameters. This might be useful if you are working with multiple distributions with differing shape, location and scale, and wanted to use these distributions throughout your notebook.

```python
rv1 = norm(0, 1)
rv2 = norm(0.5, 1)
rv3 = norm(1, 1)

plt.plot(x, rv1.pdf(x), label='norm(0, 1)')
plt.plot(x, rv2.pdf(x), label='norm(0.5, 1)')
plt.plot(x, rv3.pdf(x), label='norm(1, 1)')

plt.legend()

plt.show()
```

We can also set the axis labels and plot title with [`pyplot.xlabel`](https://matplotlib.org/api/_as_gen/matplotlib.pyplot.xlabel.html), [`pyplot.ylabel`](https://matplotlib.org/api/_as_gen/matplotlib.pyplot.ylabel.html), and [`pyplot.title`](https://matplotlib.org/api/_as_gen/matplotlib.pyplot.title.html) functions.

```python
rv = norm()
plt.plot(x, rv.pdf(x))

plt.xlabel('x')
plt.ylabel('norm.pdf(x)')
plt.title('Standard Normal PDF')

plt.show()
```

The axis limits can be set manually with [`pyplot.xlim`](https://matplotlib.org/api/_as_gen/matplotlib.pyplot.xlim.html) and [`pyplot.ylim`](https://matplotlib.org/api/_as_gen/matplotlib.pyplot.ylim.html) If, for example, you wanted the vertical axis to range from 0 to 1, you would follow the call to `pyplot.plot` with `pyplot.ylim([0, 1])`.

```python
plt.plot(x, rv.pdf(x))
plt.ylim([0, 1])
plt.show()
```

Finally, to change the color of the plot, add `color="orange"` or `color="green"`, or whatever your favorite color is! (see [Specifying Colors](https://matplotlib.org/users/colors.html) in the Matplotlib documentation)

```python
plt.plot(x, rv.pdf(x), color='orange')
plt.show()
```

## Universality with Logistic

We proved in Example 5.3.4 that for $U \sim Unif(0, 1)$, the r.v.  $log\left(\frac{U}{1−U}\right)$ follows a Logistic distribution. In SciPy and `uniform.rvs`, we can simply generate a large number of $Unif(0, 1)$ realizations and transform them.

```python
u = uniform.rvs(size=10**4)
x = np.log(u/(1-u))
x
```

    array([ 2.0197625 , -0.72108035, -1.86938026, ..., -1.2638552 ,
            1.8532956 ,  1.46624836])

Now `x` contains 104 realizations from the distribution of $log\left(\frac{U}{1−U}\right)$. We can visualize them with a histogram, using [`matplotlib.hist`](https://matplotlib.org/api/_as_gen/matplotlib.pyplot.hist.html). The histogram resembles a Logistic PDF, which is reassuring. To control how fine-grained the histogram is, we can set the number of bins in the histogram via the `bins` parameter (the 2nd parameter passed into `pyplot.hist`: `hist(x, 100)` produces a finer histogram, while `hist(x, 10)` produces a coarser histogram.

To illustrate, we will generate two graphs side-by-side with [`pyplot.figure`](https://matplotlib.org/api/_as_gen/matplotlib.pyplot.figure.html), and use a [`pyplot.subplot`](https://matplotlib.org/api/_as_gen/matplotlib.pyplot.subplot.html) for each graph.

```python
fig = plt.figure(figsize=(14, 6))

ax1 = fig.add_subplot(121)
ax1.hist(x, 100)
ax1.set_title('bins=100')

ax2 = fig.add_subplot(122)
ax2.hist(x, 10)
ax2.set_title('bins=10')

plt.show()
```

## Poisson process simulation

To simulate $n$ arrivals in a Poisson process with rate $\lambda$, we first generate the interarrival times as i.i.d. Exponentials and store them in variable `x`:

```python
n = 50
lambd = 10
x = expon.rvs(scale=1/lambd, size=n)
print(x)
```

    [0.17627964 0.01902565 0.02146015 0.19802154 0.16173354 0.02476036
     0.02394206 0.11207592 0.00051518 0.09787187 0.06122226 0.07313865
     0.07720919 0.15936267 0.00195037 0.01965782 0.00122744 0.03186262
     0.13671518 0.29913587 0.05325137 0.1329256  0.18825129 0.31339715
     0.07390052 0.2792474  0.19905106 0.1237123  0.12739159 0.05063309
     0.00787248 0.20327465 0.28319828 0.0039541  0.0084741  0.01842843
     0.00354277 0.08189181 0.07338873 0.03824727 0.02011739 0.0211869
     0.02824106 0.07150573 0.24463567 0.35755971 0.01220348 0.00555063
     0.05732313 0.20553653]

Then we convert the interarrival times into arrival times using the [`numpy.cumsum`](https://docs.scipy.org/doc/numpy-1.15.0/reference/generated/numpy.cumsum.html) function, which stands for "cumulative sum".

```python
t = np.cumsum(x)
print(t)
```

    [0.17627964 0.19530529 0.21676545 0.41478698 0.57652053 0.60128088
     0.62522295 0.73729887 0.73781405 0.83568592 0.89690818 0.97004683
     1.04725602 1.20661869 1.20856907 1.22822689 1.22945433 1.26131695
     1.39803213 1.697168   1.75041936 1.88334496 2.07159625 2.38499339
     2.45889391 2.73814132 2.93719238 3.06090468 3.18829627 3.23892936
     3.24680184 3.4500765  3.73327478 3.73722888 3.74570298 3.76413141
     3.76767418 3.849566   3.92295473 3.961202   3.98131938 4.00250628
     4.03074735 4.10225307 4.34688874 4.70444845 4.71665193 4.72220256
     4.77952569 4.98506222]

The array `t` now contains all the simulated arrival times.

----

Joseph K. Blitzstein and Jessica Hwang, Harvard University and Stanford University, © 2019 by Taylor and Francis Group, LLC


# 07.02 - DESIGN ML ALGORITHMS

```python
!wget --no-cache -O init.py -q https://raw.githubusercontent.com/rramosp/ai4eng.v1/main/content/init.py
from init import init; init(force_download=False)
```

## Cómo se diseña un algoritmo ML

1. Elegir de qué parámetros depende una predicción $\rightarrow$ se define **cómo** es un modelo.
2. Definir una función que mida el error de la predicción.
3. Determinar qué valores de los parámetros minimizan el error de predicción.

```python
## KEEPOUTPUT
from IPython.display import Image
Image(filename='local/imgs/mldesign.jpg', width=800)
```

## Ejemplo

Los Trilotrópicos son insectos imaginarios que viven en las latitudes tropicales. Conocer su densidad de escamas es muy importante para poder saber qué insecticida aplicar. Pero **es muy costoso** contar las escamas.

Creemos que existe una relación entre la **longitud** y la **densidad de escamas** y queremos un modelo que prediga la densidad **a partir de la longitud**.

Esto es una tarea de regresión, ya que la predicción $\in \mathbb{R}$

Tenemos **datos anotados** (alguien contó las escamas de unos cuantos trilotrópicos) $\rightarrow$ estamos ante una tarea de **aprendizaje supervisado**

```python
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from scipy.optimize import minimize
%matplotlib inline
```

```python
## KEEPOUTPUT
d = pd.read_csv("local/data/trilotropicos.csv")
print (d.shape)
plt.scatter(d.longitud, d.densidad_escamas)
plt.xlabel(d.columns[0])
plt.ylabel(d.columns[1]);
```

### 1. Elegimos la forma del modelo

**Entrada**
- $x^{(i)}$: longitud del trilotrópico $i$

**Salida esperada**
- $y^{(i)}$: densidad de escamas del trilotrópico $i$

**Predicción ** $\rightarrow$ decidimos que nuestro modelo tiene la siguiente forma

- $\hat{y}^{(i)} = \theta_0 + \theta_1 x^{(i)}$

La siguiente es una posible combinación de $\theta_0$ y $\theta_1$ seleccionada aleatoriamente. Ejecútalo varias veces para entender el error.

```python
def linear_prediction(t, x):
    t0,t1 = t
    return t0 + t1*x
```

```python
## KEEPOUTPUT
t0 = np.random.random()*5+10
t1 = np.random.random()*4-3
p = d.iloc[np.random.randint(len(d))]
long = p.longitud
dens = p.densidad_escamas
print ("RANDOM t0 = %.3f, t1 = %.3f"%(t0,t1))
print ("\nlong = %.3f\ndens = %.3f"%(long, dens))
print ("\npred = %.3f (WITH RANDOM t0 t1)"%linear_prediction([t0,t1], long) )
```

    RANDOM t0 = 10.693, t1 = -1.336

    long = 4.770
    dens = 7.159

    pred = 4.320 (WITH RANDOM t0 t1)

```python
## KEEPOUTPUT
def plot_model(t, prediction):
    xr = np.linspace(np.min(d.longitud), np.max(d.longitud), 100)
    plt.scatter(d.longitud, d.densidad_escamas, s=40, alpha=.2, color="blue", label="")
    plt.plot(xr,prediction(t,xr), lw=2, color="black")
    plt.title("   ".join([r"$\theta_%d$=%.2f"%(i, t[i]) for i in range(len(t))]));

    p = d.iloc[np.random.randint(len(d))]
    pred = prediction(t, p.longitud)
    plt.plot([p.longitud, p.longitud], [p.densidad_escamas, pred], ls="--", color="gray", label=u"error de predicción")
    plt.scatter(p.longitud, p.densidad_escamas, s=70, alpha=.5, color="red", label="random trilobite")
    plt.scatter(p.longitud, pred, s=70, alpha=1., color="black", label=u"predicción")
    plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))

    plt.xlabel(d.columns[0])
    plt.ylabel(d.columns[1]);

plot_model([t0,t1], linear_prediction)
```

### 2. Definimos una medida de error

Para un dato cualquiera $(i)$
$$
\begin{align}
err^{(i)} &= (\hat{y}^{(i)}-y^{(i)})^2\\
&=(\theta_0 + \theta_1 x^{(i)} - y^{(i)})^2
\end{align}$$

Para todo el dataset

$$J(\theta_0, \theta_1) = \frac{1}{m} \sum_{i=0}^{m-1}(\theta_0 + \theta_1 x^{(i)} - y^{(i)})^2$$

si asumimos que

- $\overline{\theta} = [\theta_0, \theta_1]$
- $\mathbf{x}^{(i)} = [1, x^{(i)}]$

entonces podemos escribir de manera más compacta la expresión anterior:

$$J(\overline{\theta}) = \frac{1}{m} \sum_{i=0}^{m-1}(\overline{\theta} \dot \;  \mathbf{x}^{(i)} - y^{(i)})^2$$

```python
def J(t, x, y, prediction):
    return np.mean( (prediction(t,x)-y)**2)

```

```python
J([t0,t1], d.longitud, d.densidad_escamas, linear_prediction)
```

    27.289016766830752

### 3. Obtenemos los parámetros que minimizan el error de predicción

observa cómo usamos un **algoritmo genérico de optimización**

```python
## KEEPOUTPUT
r1 = minimize(lambda t: J(t, d.longitud, d.densidad_escamas, linear_prediction), np.random.random(size=2))
r1
```

          fun: 2.7447662570802733
     hess_inv: array([[ 5.56677384, -1.08863729],
           [-1.08863729,  0.233761  ]])
          jac: array([-3.87430191e-07, -2.65240669e-06])
      message: 'Optimization terminated successfully.'
         nfev: 36
          nit: 8
         njev: 9
       status: 0
      success: True
            x: array([12.68999889, -0.7180593 ])

```python
## KEEPOUTPUT
plot_model(r1.x, linear_prediction)
print ("error total %.2f"%(J(r1.x, d.longitud, d.densidad_escamas, linear_prediction)))
```

fíjate que son los mismos valores que la regresión lineal clásica

```python
## KEEPOUTPUT
from sklearn.linear_model import LinearRegression

lr = LinearRegression()
lr.fit(d.longitud.values.reshape(-1,1), d.densidad_escamas)
t0, t1 = lr.intercept_, lr.coef_
print (t0, t1)
plot_model([t0,t1], linear_prediction)
print ("error total %.2f"%(J([t0, t1], d.longitud, d.densidad_escamas, linear_prediction)))
```

## WARN!! Black box optimization can only be used for VERY SIMPLE models

in ML normally you have to:

- compute the partial derivatives of the parameters of the cost function
- use optimizers specially designed for ML

```python

```

## Otra forma de modelo

esta vez con tres parámetros y un término **cuadrático**

$$\hat{y}^{(i)} = \theta_0 + \theta_1 x^{(i)} + \theta_2 (x^{(i)})^2$$

```python
def quad_prediction(t, x):
    t0,t1,t2 = t
    return t0 + t1*x + t2*x**2

r2 = minimize(lambda t: J(t, d.longitud, d.densidad_escamas, quad_prediction), np.random.random(size=3))
r2
```

          fun: 0.5533076730536329
     hess_inv: array([[ 50.71906017, -23.19602691,   2.42715945],
           [-23.19602691,  11.05458208,  -1.18794883],
           [  2.42715945,  -1.18794883,   0.1304395 ]])
          jac: array([-2.98023224e-08,  2.23517418e-08,  2.23517418e-07])
      message: 'Optimization terminated successfully.'
         nfev: 80
          nit: 14
         njev: 16
       status: 0
      success: True
            x: array([26.75881715, -7.61054361,  0.75701901])

```python
## KEEPOUTPUT
plot_model(r2.x, quad_prediction)
print ("error total %.2f"%(J(r2.x, d.longitud, d.densidad_escamas, quad_prediction)))
```

observa como indirectamente hacemos lo mismo con la regresión lineal de `sklearn` añadiendo explícitamente una columna con la longitud al cuadrado

```python
lr = LinearRegression()
lr.fit(np.r_[[d.longitud.values, d.longitud.values**2]].T, d.densidad_escamas)
t0, (t1, t2) = lr.intercept_, lr.coef_
t0, t1, t2
```

    (26.75883713364987, -7.61055346692438, 0.7570201016172385)

```python

```

content/NOTES 07.02 - DESIGNING ML ALGORITHMS.ipynb


[Link to ADS entry](http://adsabs.harvard.edu/abs/1978ApJ...226..650I)

Loading library and setting up environment

```python
import sympy
sympy.init_printing()
```

Equation 5 - Conservation of mass

$\frac{\partial \rho}{\partial t} + \nabla \left(\rho v \right) = 0$

Equation 6 - Momentum conservation

$\frac{d v}{d t} = \frac{J \times B}{4 \pi \rho} - \frac{\nabla P}{\rho}$

Equation 7 - Ideal Ohm's law

$E + \frac{v}{c} \times B=0$

Equation 8 - Ampere's law

$\nabla \times B = \frac{4 \pi}{c} J$

Equation 9 - Faradays's law

$\frac{1}{c}\frac{\partial B}{\partial t} = -\nabla \times E$

Equation 10 - Ideal gas law

$P = n_e k T_e$

Equation 11 - Eliminating the current density from the momentum equation using Ampere's law

$4 \pi \rho \frac{d v}{d t} = \left(\nabla\times B\right) \times B - 4 \pi \nabla P$

Equation 12 - Eliminiating the electric field from Faraday's equation using the Ideal Ohm's law

$\frac{\partial B}{\partial t} = \nabla \times \left(v \times B\right) = \left(B \cdot \nabla \right) v - \left(v\cdot \nabla \right) B - B \left(\nabla \cdot v \right)$

In steady state $\partial v / \partial t = \partial B / \partial t = 0$. The author introduces a new variable: the displacement, which relates to the velocity through $v = d \xi / d t$. Linearising the momentum equation yields equation 13

$4 \pi \rho \frac{\partial^2 \xi}{\partial t^2} = \left(\nabla \times B'\right) \times B + \left(\nabla \times B\right) \times B' - 4 \pi \nabla P'$

Equation 14 can be obtained from equation 12 by replacing the velocity with the displacement

$B' = \left(B \cdot \nabla \right) \xi - \left(\xi \cdot \nabla \right) B - B \left(\nabla \cdot \xi\right)$

To derive equation 15, we start from the perturbed conservation of mass

$\rho' = -\xi \nabla \rho - \rho \nabla \xi$

Substituting in the ideal gas equation yields

$\frac{P'}{P} = \frac{\rho'}{\rho} = -\xi \frac{\nabla \rho}{\rho}- \nabla \xi = - \chi_p \xi - \nabla \xi$

Equations 16 - 18

$\left(\nabla \times B'\right) \times B + \left(\nabla \times B\right) \times B' = \left( B \cdot \nabla \right) B' + \left(B' \cdot \nabla \right) B - \nabla \left(B \cdot B' \right)$

We only need to expand the first term

$ \left(B \cdot \nabla \right) B'= \left(B \cdot \nabla \right) \left[ \left(B \cdot \nabla \right) \xi - \left( \xi \cdot \nabla \right) B - B \left( \nabla \cdot \xi\right) \right]$

The remainder of the derivation is straightforward

$4 \pi \rho \frac{\partial^2 \xi}{\partial t^2} - \left(B \cdot \nabla\right)^2 \xi +\nabla \left[4 \pi P' + B' \cdot B \right] - \left(B' \cdot \nabla \right) B + \left(B \cdot \nabla\right) \left[\left(\xi \cdot \nabla\right) B + B \left(\nabla \cdot \xi\right) \right] = 0$

For equation 19, Here's a sketch to understand the geometry

When the ambient magnetic field is constant equation 16 is greatly simplified

$4 \pi \rho \frac{\partial^2 \xi}{\partial t^2} - B^2 \frac{\partial^2 \xi}{\partial x_{\parallel}^2} +\nabla \left[4 \pi P' + B' \cdot B \right] + \hat{x}_{\parallel} B^2 \frac{\partial}{\partial x_{\parallel}} \left(\nabla \cdot \xi \right)= 0$

Substituting the perturbed mode yields

$\epsilon \xi_r = \frac{d}{d x_r} \left(4 \pi P'+B' \cdot B\right)$

$\epsilon \xi_{\perp} = i k_{\perp} \left(4 \pi P' + B' \cdot B\right)$

$\epsilon \xi_{\parallel} = i k_{\perp} \left(4 \pi P' + B'\cdot B\right) + i B^2 k_{\parallel} \left(i k_{\parallel}\xi_{\parallel} + i k_{\perp} \xi_{\perp} + \frac{d \xi_{r}}{d \xi_r} \right)$

where $\epsilon = 4 \pi \rho \omega^2 - k_{\parallel} B^2$

Equation 24

$\frac{4 \pi P'+ B \cdot B'}{4 \pi P+B^2} = \frac{-4 \pi P \left(\chi_p \cdot \xi +\nabla \xi\right)+ B \cdot \left[ \left(B \cdot \nabla \right) \xi - \left(\xi \cdot \nabla \right) B - B \left(\nabla \cdot \xi\right) \right]}{4 \pi P + B^2} = - \nabla \cdot \xi +\frac{ -4 \pi P \chi_p \cdot \xi + B^2 \frac{d \xi_{\parallel}}{d x_{\parallel}} }{4 \pi P + B^2} = - \left(i k_{\parallel} \xi_{\parallel} + i k_{\perp} \xi_{\perp} + \frac{d \xi_r}{d x_r} \right ) +\frac{ -4 \pi P \chi_p \cdot \xi + i k_{\parallel} B^2 \xi }{4 \pi P + B^2} = $

$= -i k_{\perp} \xi_{\perp}- \frac{d \xi_r}{d x_r} - \frac{P}{P+B^2/4\pi} i k_{\parallel} \xi_{\parallel} - \frac{P \chi_p \cdot \xi}{P+B^2/4 \pi}$

This is almost the same expression as the right hand side of equation 24, but with a superfluous extra term at the end.

Derivation of equation 25

```python
epsilon = sympy.Symbol('epsilon')
xi_parallel = sympy.Symbol(r'\xi_{\parallel}')
xi_perp = sympy.Symbol(r'\xi_{\perp}')
k_perp = sympy.Symbol(r'k_{\perp}')
k_parallel = sympy.Symbol(r'k_{\parallel}')
tilde_P_prime = sympy.Symbol(r"\tilde{P}'")
dxirdxr = sympy.Symbol(r'd \xi_r/d x_r')
B = sympy.Symbol('B')
eqn_22 = sympy.Eq(epsilon*xi_parallel, 4*sympy.pi*sympy.I*k_parallel*tilde_P_prime+
                 sympy.I*B**2*k_parallel*(sympy.I*k_parallel*xi_parallel+sympy.I*k_perp*xi_perp+dxirdxr))
eqn_22
```

```python
P = sympy.Symbol('P')
eqn_24 = sympy.Eq(tilde_P_prime/(P+B**2/4/sympy.pi),
                 - dxirdxr-sympy.I*k_perp*xi_perp-sympy.I*k_parallel*xi_parallel*(P/(P+B**2/4/sympy.pi)))
eqn_24
```

```python
eqn_21 = sympy.Eq(epsilon*xi_perp,
                 4*sympy.pi*sympy.I*k_perp*tilde_P_prime)
eqn_21
```

```python
alpha = sympy.Symbol('alpha')
rho = sympy.Symbol('rho')
omega = sympy.Symbol('omega')
eqn_26 = sympy.Eq(alpha, 1+2*sympy.pi*rho*beta*omega**2/(4*sympy.pi*rho*omega**2-beta*k_parallel**2*B**2/2))
eqn_26
```

```python
eqn_23 = sympy.Eq(epsilon, 4*sympy.pi*rho*omega**2-k_parallel**2*B**2)
eqn_23
```

```python
beta = sympy.Symbol('beta')
temp = eqn_24
temp = temp.subs(sympy.solve(eqn_22,xi_parallel,dict=True)[0])
temp = temp.subs(sympy.solve(eqn_21,tilde_P_prime,dict=True)[0])
temp = temp.subs(P, beta*B**2/8/sympy.pi)
temp = temp.subs(sympy.solve([eqn_23, eqn_26], [rho, k_parallel], dict=True)[1])
temp = sympy.solve(temp, xi_perp)[0].simplify()
eqn_25 = sympy.Eq(xi_perp, temp)
eqn_25
```

Equation 28

$\epsilon \xi_r = 4 \pi \frac{d}{d x_r} \left(\frac{\epsilon \xi_{\perp}}{4 \pi i k_{\perp}} \right ) = \frac{1}{k_{\perp}} \frac{d}{d x_r} \left(\epsilon \frac{\alpha k_{\perp} B^2}{\alpha k_{\perp} B^2 - \epsilon} \frac{d \xi_r}{d x_r} \right)$

Equation 41 makes no sense to me. I don't see how the change of variables in equation 41 gets you from equation 30 to equation 41. Also, I can give an example where the two equations give a different answer. For simplicity, let's define a dimensionless variable $x = x_r k_{\perp}$, and assume $\epsilon \propto x^2$. Equation 30 becomes

$x^2 \frac{d^2 \xi}{d x^2} + 2 x \frac{d \xi}{d x} - x^2 \xi = 0$

and the solution is

$\xi \propto \frac{\exp \left(\pm x\right)}{x}$

The new variable is $\psi = x^2 / \frac{d}{d x} \left(x^2\right) = \frac{1}{2} x$, and the solutions to equation 41 are modified Bessel functions. The substitution from $x$ to $\psi$ will not turn them to exponents.

Regardless, equation 30 can be solved for both the case where $\epsilon$ is constant and when it varies linearly with $x$. When $\varepsilon$ is constant the solutions are exponents, and when it is linear the solutions are modified Bessel functions.

```python

```


# Atmospheric Model

The basis of our simulation is a model of Earth's atmosphere, of which there are several.  The [International Standard Atmosphere](https://en.wikipedia.org/wiki/International_Standard_Atmosphere) (ISA) is used by ICAO, is relatively simple, and accurate enough for our purposes.

The ISA is a mathematical model of the atmosphere that assumes a linear temperature gradient against altitude, and derives pressure and density from the [ideal gas law](https://en.wikipedia.org/wiki/Ideal_gas_law) and [hydrostatic equilibrium](https://en.wikipedia.org/wiki/Hydrostatic_equilibrium).

## Symbols used

Since the primary source upon which the models in this package are built from is the ISA, I will use the symbols defined in the paper that defined it.

```python
from preamble import *
symbol_table
## WIP
```

Symbol	Description

$A$	Area

$F$	Force

$g$	Acceleration due to gravity

$H$	Geopotential height

$h$	Geometric height

$P$	Pressure

$P_{b}$	Pressure at the base of an atmospheric layer

$M$	Molecular mass

$m$	Mass

$n$	Amount of substance

$R^{*}$	Universal gas constant

$R$	Specific gas constant

$T$	Thermodynamic temperature

$V$	Volume

$\rho$	Density

## Hydrostatic equation

Our model begins by noting that since the atmosphere is not floating away into space, nor sinking to the ground, it must be at equilibrium: all vertical forces must sum to zero. That is to say, it is at [hydrostatic equilibrium](https://en.wikipedia.org/wiki/Hydrostatic_equilibrium). For a column of air in the atmosphere, the 3 forces we'll model are from the surrounding air pressure - air pushing up from below, air pushing down from above - and the weight of the column of air itself:

$$
\require{action}
\texttip{math}{tip}
$$

$$
F_{below} + F_{above} + F_{weight} = 0
$$

Replacing with the definitions of pressure and weight gives us (For reasons that will become apparent, we will refer to the pressure at the base of the cylinder as $P_b$ and the pressure at the top simply as $P$):

```python
Eq((A*P_b) - (A*P) - (m*g), 0)
```

$\displaystyle - A P + A P_{b} - g m = 0$

```python
# The mass of a column of air is its volume multiplied by its density:
# m = V * rho
_.replace(m, V*rho)
```

$\displaystyle - A P + A P_{b} - V g \rho = 0$

```python
# The volume is its area multiplied by its height.
_.replace(V, A * h)
```

$\displaystyle - A P + A P_{b} - A g h \rho = 0$

```python
# We can simplify by removing the area:
_ / A
```

$\displaystyle P - P_{b} + g h \rho = 0$

```python
_.replace(P-P_b, diff(P))
```

$\displaystyle P - P_{b} + g h \rho = 0$

```python
_.solve(P)
```

$\displaystyle P = P_{b} - g h \rho$

### Ideal Gas Law

The first useful equation for us is the [ideal gas law](https://en.wikipedia.org/wiki/Ideal_gas_law), which relates the pressure and volume of an ideal gas to it's temperature. In short: as the temperature of a gas increases, it's pressure and/or volume must also increase.  It is commonly stated as $PV = nR^*T$, but since it can be hard to count the number of molecules in some volume of gas, it is useful to transform into the molar form through some simple substitutions. The molar form is convenient in atmoshpheric science as we can more easily measure air pressure, temperature, and density.

```python
# pV = nRT
ideal_gas_law = Eq(P * V, n * R_universal * T)
ideal_gas_law
```

$\displaystyle P V = R^{*} T n$

```python
# The amount of substance in a thing is equivalent to its mass / molecular mass:
# n = m / M
ideal_gas_law.subs(n, m / M)
```

$\displaystyle P V = \frac{R^{*} T m}{M}$

```python
# Density is defined as mass / volume:
# rho = m / V
_.solve(P).subs(m/V, rho)
```

$\displaystyle P = \frac{R^{*} T \rho}{M}$

```python
# Finally, the specific gas constant of a gas is defined as the universal gas constant / molecular mass:
# R_sp = R / M
molar_gas_law = _.subs(R_universal/M, R)
molar_gas_law
```

$\displaystyle P = R T \rho$

## Burst Altitude

The maximum diameter of a given balloon is given by its manufacturer.  If we know the volume of gas that we've put into the balloon, we can use the ideal gas law to determine the pressure at which the balloon will burst.

```python
r = symbols('r')

volume_of_sphere = Eq(V, Rational(4,3)*pi*r**3)
volume_of_sphere

foo = ideal_gas_law.subs(volume_of_sphere.lhs, volume_of_sphere.rhs)
display(foo)

solved = next(iter(solveset(foo, P)))

from sympy.printing.pycode import pycode

display(solved)
pycode(solved)
```

$\displaystyle \frac{4 \pi P r^{3}}{3} = R^{*} T n$

$\displaystyle \frac{3 R^{*} T n}{4 \pi r^{3}}$

    '(3/4)*R^**T*n/(math.pi*r**3)'

```python
from sympy import *

F_b, m, g = symbols('F_b m g')
rho, V, r = symbols('rho V r')
rho_air, rho_gas = symbols('rho_air rho_gas')

bouyancy = Eq(F_b, m*g)
bouyancy = bouyancy.subs({
    m: rho*V,
    rho: (rho_gas - rho_air),
    V: (Rational(4,3)*pi*r**3)
})

bouyancy
```

$\displaystyle F_{b} = V g \left(- \rho_{air} + \rho_{gas}\right)$

## References

* [Here](https://atoc.colorado.edu/~cassano/atoc5050/Lecture_Notes/wh_ch3_part1.pdf)
* [ICAO Standard Atmosphere](http://www.aviationchief.com/uploads/9/2/0/9/92098238/icao_doc_7488_-_manual_of_icao_standard_atmosphere_-_3rd_edition_-_1994.pdf)
* [ISO 2533-1975: Standard Atmosphere](https://cdn.standards.iteh.ai/samples/7472/c203e9121d4c40e5bdc98844b1a1e2f4/ISO-2533-1975.pdf)
* [Inspiration](https://github.com/cuspaceflight/cusf-burst-calc/blob/master/js/calc.js)
* [And](https://northstar-www.dartmouth.edu/~klynch/pmwiki-gc/uploads/BalloonCalulations.pdf)
* [And](https://amt.copernicus.org/articles/4/2235/2011/amt-4-2235-2011.pdf)

```python

```


```
import numpy as np
import pandas as pd
import dimod
```

### Defining a sample QUBO
\begin{equation}
H_{1}^{QUBO}=-4.4x_{1}^2+0.6x_{2}^2-2x_{3}^2+2.8x_{1}x_{2}-0.8x_{2}x_{3}+2.4
\end{equation}

```
linear = {0: -4.4, 1: 0.6, 2: -2}
quadratic = {(0,1): 2.8, (1,2):-0.8}
offset = 2.4

bqm_qubo = dimod.BinaryQuadraticModel(linear,quadratic,offset,dimod.Vartype.BINARY)
print(bqm_qubo)
print('\n',bqm_qubo.to_numpy_matrix().astype(float))
```

    BinaryQuadraticModel({0: -4.4, 1: 0.6, 2: -2.0}, {(0, 1): 2.8, (1, 2): -0.8}, 2.4, 'BINARY')

     [[-4.4  2.8  0. ]
     [ 0.   0.6 -0.8]
     [ 0.   0.  -2. ]]

```
from dwave.system import LeapHybridSampler
sampler = LeapHybridSampler() # 'hybrid_binary_quadratic_model_version2'
```

```
sampler.properties
```

    {'minimum_time_limit': [[1, 3.0],
      [1024, 3.0],
      [4096, 10.0],
      [10000, 40.0],
      [30000, 200.0],
      [100000, 600.0],
      [1000000, 600.0]],
     'maximum_time_limit_hrs': 24.0,
     'maximum_number_of_variables': 1000000,
     'maximum_number_of_biases': 200000000,
     'parameters': {'time_limit': None},
     'supported_problem_types': ['bqm'],
     'category': 'hybrid',
     'version': '2.0',
     'quota_conversion_rate': 20}

```
sampleset = sampler.sample(bqm_qubo)
print(sampleset)
```

       0  1  2 energy num_oc.
    0  1  0  1   -4.0       1
    ['BINARY', 1 rows, 1 samples, 3 variables]

```

```

D'wave tutorials/7.D-wave Hybrid Solvers.ipynb


# Bug reported on 6/1/2022 by Bruno

The issue detected was that the transfer function changed on repeated calculations.

# Preliminaries

```python
!pip -q install controlSBML
import controlSBML as ctl
import control
from controlSBML.util import makeSimulationTimes
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import sympy as sy
import tellurium as te
import sympy
```

## Closed Loop Transfer Functions

**Transfer Functions**
\begin{eqnarray}
H_{RY}(s) & = & \frac{Y(s)}{R(s)} & = & \frac{C(s) G(s)}{1 + C(s) G(s) F(s)} \\
H_{RE}(s) & = & \frac{E(s)}{R(s)} & = & \frac{1}{1 + C(s) G(s) F(s)} \\
H_{NY}(s) & = & \frac{Y(s)}{N(s)} & = & -\frac{ F(s)}{1 + C(s) G(s) F(s)} \\
H_{DY}(s) & = & \frac{Y(s)}{D(s)} & = & \frac{ C(s)}{1 + C(s) G(s) F(s)} \\
\end{eqnarray}

PLACEHOLDER: Control the amount of thrombin through an anticoagulant input so that clotting behavior can be controlled as needed.

```python
BIOMD366 = "https://www.ebi.ac.uk/biomodels/model/download/BIOMD0000000366.2?filename=BIOMD0000000366_url.xml"
ctlsb = ctl.ControlSBML(BIOMD366, input_names=["Xa"], output_names=["IIa"])
ctlsb.plotTrueModel()
```

```python
ctlsb = ctl.ControlSBML(BIOMD366, input_names=["Xa"], output_names=["IIa"])
ctlsb.makeTransferFunction(time=1)
```

$$\frac{11.88}{s + 0.9621}$$

```python
TF = ctlsb.makeTransferFunction(time=1)
TF
```

$$\frac{11.88}{s + 0.9621}$$


## Information Planning

Information planning involves making decisions based on information measures. Information planning is closely related to active learning [1] and optimum experiment design [2] in which labeled data is expensive to obtain. The key idea behind active learning is that a model can learn better with fewer labeled examples if the training examples are carefully selected to maximize the information gain for a particular task. In other words, given an abundance of unlabelled data, we would like to rank the unlabelled examples according to their usefullness to training of the model. The top-K most useful training examples are labelled by expert annotators and are added to the training set.

Here, we'll consider the task of text classification using a Naive Bayes graphical model. We'll look at two information measures [3]: entropy and mutual information that will be used to rank unlabelled set of documents for information planning.

Let $x_{i,j}$ be Bernoulli random variables indicating the presence ($x_{ij} = 1$) or absence ($x_{ij}=0$) of a word $j \in \{1,...,D\}$ for document $i \in \{1,...,n\}$, parameterized by $\theta_{jc}$ for a given class label $y = c \in \{1,...,C\}$. In addition, let $\pi$ be a Dirichlet distirbution representing the prior over the class labels. Thus, the total number of learnable parameters is $|\theta| + |\pi| = O(DC) + O(C) = O(DC)$, where $D$ is the dictionary size and $C$ is the number of classes. Due to the small number of parameters, the Naive Bayes model is immune to over-fitting.

**The choice of Bernoulli Naive Bayes formulation is important because it leads itself to word-based information planning. By associating each word in the dictionary with a binary random variable, we are able to compute the influence of individual words on class label distribution.**

We can write-down the class conditional density as follows:

\begin{equation}
    p(x|y=c, \theta) = \prod_{i=1}^{n} \prod_{j=1}^{D} p(x_{ij}|y=c, \theta_{jc}) = \prod_{i=1}^{n} \prod_{j=1}^{D} \mathrm{Bernoulli}(\theta_{jc})
\end{equation}

We can derive the Naive Bayes inference algorithm by maximizing the log-likelihood. Consider words $x_i$ in a single document $i$:

\begin{equation}
    p(x_i, y_i|\theta) = p(y_i|\pi)\prod_{j=1}^{D}p(x_{ij}|y_i, \theta) = \prod_{c=1}^{C}\pi_{c}^{1[y_i=c]}\prod_{j=1}^{D}\prod_{c=1}^{C}p(x_{ij}|\theta_{jc})^{1[y_i=c]}
\end{equation}

Using the Naive Bayes assumption, we can compute the log-likelihood objective:

\begin{equation}
\log p(D|\theta) = \log \prod_{i=1}^{n} p(x_i,y_i|\theta) = \sum_{i=1}^{n}\log p(x_i,y_i|\theta)
= \sum_{c=1}^{C}N_c \log \pi_c + \sum_{j=1}^{D}\sum_{c=1}^{C}\sum_{i:y_i = c} \log p(x_{ij}|\theta_{jc})
\end{equation}

By setting the gradient of $\log p(D|\theta)$ with respect to the model parameters $\pi_c$ and $\theta_{jc}$ to zero, we obtain the following MLE updates: $\hat{\pi_c} = \frac{N_c}{N}$ and $\hat{\theta_{jc}} = \frac{N_{jc}}{N_c}$ where $N_c = \sum_{i=1}^{n} 1[y_i = c]$. Note, that it's straight forward to add a Beta prior for the Bernoulli random variables and a Dirichlet prior for the class density to smooth the MLE counts:

\begin{eqnarray}
    p(\pi|D) &=& \mathrm{Dir}(N_1+\alpha_1, ..., N_c + \alpha_c) \\
    p(\theta_{jc}|D) &=& \mathrm{Beta}([N_c-N_{jc}] + \beta_0, N_{jc} + \beta_1)
\end{eqnarray}

During test time, we would like to predict the class label $y$ given the training data $D$ and the learned model parameters. Applying the Bayes rule:

\begin{equation}
    p(y=c|x_{i,1},...,x_{i,D},D) \propto p(y=c|D) p(x_{i,1},...,x_{i,D}|y=c, D) = p(y=c|D) \prod_{j=1}^{D}p(x_{ij}|y=c,D)
\end{equation}

Substituting the distributions for $p(y=c|D)$ and $p(x_{ij}|y=c,D)$ and taking the log, we get:

\begin{equation}
    \log p(y=c|x,D) \propto \log p(y=c|D) + \sum_{j=1}^{D}\bigg(1[x_{ij}=1]\log \theta_{jc} + 1[x_{ij}=0]\log(1-\theta_{jc}) \bigg)
\end{equation}

The Baeysian formulation of Bernoulli Naive Bayes is implemented in scikit-learn [5] using the steps outlined above. While it's really good practice to derive and implement ML algorithms from scratch (and in faster languages such as C++), we'll use the scikit-learn implementation and focus on information planning. Let's start by importing the required libraries.

```python
%matplotlib inline
import numpy as np
import pandas as pd

import seaborn as sns
import matplotlib.pyplot as plt

from time import time
from nltk.corpus import stopwords
from nltk.tokenize import RegexpTokenizer

from scipy.misc import logsumexp
from sklearn.metrics import accuracy_score
from sklearn.naive_bayes import BernoulliNB
from sklearn.datasets import fetch_20newsgroups
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer

sns.set_style("whitegrid")
np.random.seed(0)

tokenizer = RegexpTokenizer(r'\w+')
stop_words = set(stopwords.words('english'))
stop_words.update(['s','t','m','1','2'])
```

We will train our model on a subset of the 20newsgroups dataset. In particular, we'll restrict ourselves to $4$ classes: space, graphics, autos, and hockey.

```python
#load data
print "loading 20 newsgroups dataset..."
tic = time()
classes = ['sci.space', 'comp.graphics', 'rec.autos', 'rec.sport.hockey']
dataset = fetch_20newsgroups(shuffle=True, random_state=0, remove=('headers','footers','quotes'), categories=classes)
X_train, X_test, y_train, y_test = train_test_split(dataset.data, dataset.target, test_size=0.5, random_state=0)
toc = time()
print "elapsed time: %.4f sec" %(toc - tic)
print "number of training docs: ", len(X_train)
print "number of test docs: ", len(X_test)
```

    loading 20 newsgroups dataset...
    elapsed time: 2.6109 sec
    number of training docs:  1185
    number of test docs:  1186

We'll use a count vectorizer to produce a vector of word counts for each document while filtering stop and low-frequency words.

```python
tic = time()
cnt_vec = CountVectorizer(tokenizer=tokenizer.tokenize, analyzer='word', ngram_range=(1,1), max_df=0.8, min_df=2, max_features=1000, stop_words=stop_words)
cnt_vec.fit(X_train)
toc = time()
print "elapsed time: %.2f sec" %(toc - tic)
vocab = cnt_vec.vocabulary_
idx2word = {val: key for (key, val) in vocab.items()}
print "vocab size: ", len(vocab)

X_train_vec = cnt_vec.transform(X_train) #size [D, V]
X_test_vec = cnt_vec.transform(X_test)
```

    elapsed time: 0.25 sec
    vocab size:  1000

Let's execute the Naive Bayes inference described earlier and use the trained model to predict class labels on test data.

```python
print "naive bayes model MAP inference..."
nb_clf = BernoulliNB(alpha=1.0, binarize=0.0, fit_prior=True)
nb_clf.fit(X_train_vec, y_train)

y_pred = nb_clf.predict(X_test_vec)
nb_clf_acc = accuracy_score(y_test, y_pred)
print "test set accuracy: ", nb_clf_acc

```

    naive bayes model MAP inference...
    test set accuracy:  0.695615514334

### Entropy Planning

Entropy measures the amount of uncertainty in a random variable. In entropy based planning, we want to rank test-documents according to the entropy of their class label. The idea is that by annotating documents that have highest uncertainty about their class label will be able to better train our model with fewer annotated examples. In the case of the class distribution, the entropy can be computed in closed form: $H(y) = -\sum_{c=1}^{C} p(y=c) \log p(y=c)$

```python
#rank test documents according to entropy of their class label
top_K = 20
X_test_class_logp = nb_clf.predict_log_proba(X_test_vec)
X_test_class_entropy = -np.sum(np.exp(X_test_class_logp) * X_test_class_logp, axis=1)
planning_entropy_idx = np.argsort(X_test_class_entropy)[-top_K:]
```

### Mutual Information Planning

In the case of mutual information, we need to choose the random variables we want to use for planning. Since we are interested in classifying documents, it makes sense to choose $y_i$ (the class label for document $i$) as one of the variables. Our choice of the second variable depends on the quantity of interest. We can choose one of the global variables $\theta_{jc}$ (probability that word $j$ is present in class $c$) or $\pi_c$ (probability of class $c$). In this example, we consider estimating $I(y_i;\theta)$, i.e. we are interested in measuring the information gain about the word distribution $\theta$ given the class label $y_i$ for a test document. Since both variables are discrete, the mutual information can be estimated in closed form:

\begin{equation}
    I(y_i; \theta) = \mathrm{KL}\bigg(p(y_i,\theta)~||~p(y_i)p(\theta)\bigg) = \sum_{y \in C} \sum_{\theta \in D} p(y_i,\theta)\log \frac{p(y_i,\theta)}{p(y_i)p(\theta)} = \sum_{y \in C} \sum_{\theta \in D} p(\theta|y_i)p(y_i)\log \frac{p(\theta|y_i)}{p(\theta)}
\end{equation}

we can compute $p(\theta)$ using the law of total probability: $p(\theta_j) = \sum_{c \in C}p(x_j=1|y=c)p(y=c)$ for $j \in \{1,...,D\}$. In addition, we compute $I(x_j;\pi)$ to measure how informative each word $x_j$ is to the global label distribution $\pi$.

```python
#rank test documents according to mutual information (MI) between class labels and theta
log_pi = nb_clf.class_log_prior_          #log pi(c) global
log_py = nb_clf.predict_log_proba(X_test_vec) #log py(c) per document
log_pxy = nb_clf.feature_log_prob_        #log pxy(c,j) = log p(x_ij = 1 | y = c)

[C,D] = log_pxy.shape
log_pi_tile = np.tile(log_pi.reshape(-1,1), log_pxy.shape[1])
#log px(j) = log [\sum_c p(x_ij = 1 | y = c) x p(y = c)]
#          = log [\sum_c exp{log_pxy(c,j) + log_pi(c)}]
log_px = logsumexp(log_pxy + log_pi_tile, axis=0)

#MI between x_j and pi
X_test_word_MI = np.zeros(D)
for class_idx in range(C):
    X_test_word_MI = X_test_word_MI + \
    np.exp(log_pi[class_idx]) * np.exp(log_pxy[class_idx,:]) * np.log(np.exp(log_pxy[class_idx,:])/np.exp(log_px)) + \
    np.exp(log_pi[class_idx]) * (1-np.exp(log_pxy[class_idx,:])) * np.log((1-np.exp(log_pxy[class_idx,:]))/(1-np.exp(log_px)))
#end for

#MI between y_j and theta
logp_theta = logsumexp(log_pxy + log_pi_tile, axis=0)
X_test_docs_MI = np.zeros(X_test_vec.shape[0])
for doc_idx in range(X_test_vec.shape[0]):
    doc_MI = 0
    for class_idx in range(C):
        doc_MI += np.sum(np.exp(log_pxy[class_idx,:] + log_py[doc_idx, class_idx]) * \
                        (log_pxy[class_idx,:] - logp_theta), axis=-1)
    #end for
    X_test_docs_MI[doc_idx] = doc_MI
#end for

planning_MI_idx = np.argsort(X_test_docs_MI)[-top_K:]
```

Let's display the top 10 (highest probability) words associated with each class:

```python
#display top words
for class_idx in range(C):
    top_word_idx = np.argsort(log_pxy[class_idx,:])[-10:]
    print "\ntop-10 words for class: ", class_idx
    top_word_list = []
    for it in top_word_idx:
        top_word_list.append(idx2word[it])
    #end for
    print " ".join(top_word_list)
#end for
```

    top-10 words for class:  0
    get please use like one anyone thanks graphics would know

    top-10 words for class:  1
    think well know also cars get like one would car

    top-10 words for class:  2
    first time season year play hockey one would game team

    top-10 words for class:  3
    nasa could much know get also like one would space

Let's look at the top 20 (highest MI) informative words:

```python
print "\ntop-20 MI words:"
top_word_list = []
top_word_idx = np.argsort(X_test_word_MI)[-20:]
for it in top_word_idx:
    top_word_list.append(idx2word[it])
#end for
print " ".join(top_word_list)
```

    top-20 MI words:
    teams program league moon playoffs orbit players earth games nasa nhl play cars season graphics hockey team game car space

Not bad! considering the ground truth classes are:

```python
print "\nground truth labels:"
print " ".join(classes)
```

    ground truth labels:
    sci.space comp.graphics rec.autos rec.sport.hockey

We can see that highest MI words reflect the ground truth labels really well. Let's look at the learned word probabilities for every class:

```python
#generate plots
f, ((ax1, ax2), (ax3, ax4)) = plt.subplots(2,2, sharex=True, sharey=True)
ax1.plot(np.arange(D), np.exp(log_pxy[0,:]), color='b')
ax1.set_title('p(word $x_j = 1$ | class label $y = 0$)')
ax2.plot(np.arange(D), np.exp(log_pxy[1,:]), color='b')
ax2.set_title('p(word $x_j = 1$ | class label $y = 1$)')
ax3.plot(np.arange(D), np.exp(log_pxy[2,:]), color='b')
ax3.set_title('p(word $x_j = 1$ | class label $y = 2$)')
ax4.plot(np.arange(D), np.exp(log_pxy[3,:]), color='b')
ax4.set_title('p(word $x_j = 1$ | class label $y = 3$)')
f.savefig('./figures/naive_bayes_theta.png')

```

We can also visualize posterior class label distribution:

```python
plt.figure()
sns.barplot(x=np.arange(C), y=np.exp(nb_clf.class_log_prior_), palette='Set1')
plt.title('class label posterior')
plt.xlabel('class number')
plt.savefig('./figures/naive_bayes_pi.png')
```

Finally, let's visualize the test document ranking based on entropy and mutual information:

```python
f, (ax1, ax2) = plt.subplots(1, 2, sharex=True, sharey=False)
sns.violinplot(y=X_test_class_entropy, palette='Set3', ax=ax1)
sns.violinplot(y=X_test_docs_MI, palette='Set3', ax=ax2)
ax1.set_title("entropy"); ax2.set_title("mutual information")
ax1.set_ylim([0,1.3]); ax2.set_ylim([0, 18]);
plt.savefig('./figures/naive_bayes_violin.png')
```

In particular, we are interested in outliers, i.e. documents that have highest entropy and mutual information. From the violin plot above, we can see a greater density of high MI test documents compared to high entropy documents, which suggests that MI could be better at differentiating which documents are most suitable for information planning.

\begin{equation}
I(X;Y) = H(X) - H(X|Y) = H(Y) - H(Y|X)
\end{equation}

In the case of discrete $X$ and $Y$, to maximize the information gain $I(X;Y)$, we want to maximize $H(X)$ (uncertainty about $X$) and minimize $H(X|Y)$ (i.e. we want $Y$ to be informative about $X$). To visualize the difference between entropy and MI based planning, we can sort the documents according to entropy and use the same index to sort MI.

```python
f, (ax1, ax2) = plt.subplots(2,1, sharex=True, sharey=False)
ax1.plot(X_test_class_entropy[planning_entropy_idx], color='b')
ax1.set_title('entropy sorted by entropy index')
ax2.plot(X_test_docs_MI[planning_entropy_idx], color='r')
ax2.set_title('MI sorted by entropy index')
plt.savefig('./figures/naive_bayes_order.png')
```

In the above figure, documents that have high entropy but low MI are uncertain but not informative. Therefore, MI provides a better measure for information planning.

### References

[1] B. Settles, "Active Learning Literature Survey", Technical Report, 2010
[2] K. Chaloner and I. Verdinelli, "Bayesian Experimental Design: A Review", Statistical Science, 1995
[3] T. Cover and J. Thomas, "Elements of Information Theory", Wiley, 2006
[4] K. Murphy, "Machine Learning: A Probabilistic Perspective", MIT Press, 2012
[5] F. Pedregosa et al. "Scikit-learn: Machine Learning in Python: http://scikit-learn.org/stable/", JMLR, 2011


# Lab 2 - Plotting Vector using NumPy and MatPlotLib

In this laboratory we will be discussing the basics of numerical and scientific programming by working with Vectors using NumPy and MatPlotLib.

### Objectives
At the end of this activity you will be able to:
1. Be familiar with the libraries in Python for numerical and scientific programming.
2. Visualize vectors through Python programming.
3. Perform simple vector operations through code.

## Discussion

### NumPy

NumPy or Numerical Python, is mainly used for matrix and vector operations. It is capable of declaring computing and representing matrices. Most Python scientific programming libraries uses NumPy as the basic code.

###Scalars
A single number. Variables representing scalars are typically written in lower case.

Scalars can be whole numbers or decimals.

\begin{align}
a = 12
\qquad
b = 6.5842342
\end{align}

They can be positive, negative, 0 or any other real number.

\begin{align}
c = -8.5412\mathrm{e}{+23}
\qquad
d = \pi
\end{align}

###Vectors
A vector of dimension *n* is an **ordered** collection of *n* elements, which are called **components**. Vector notation variables are commonly written as a bold-faced lowercase letters or italicized non-bold-faced lowercase characters with an arrow (→) above the letters:

Written: $\vec{v}$

Examples:

\begin{align}
   \vec{a} =
   \begin{bmatrix}
           -10\\
           5
    \end{bmatrix}
    \qquad
    \vec{b} =
    \begin{bmatrix}
          8\\
          21\\
          -3
    \end{bmatrix}
    \qquad
    \vec{c} =
    \begin{bmatrix}
           4.5
    \end{bmatrix}
    \qquad
    \vec{d} =
    \begin{bmatrix}
           a\\
           b\\
           \frac{4}{8}
    \end{bmatrix}
\end{align}

#### Representing Vectors

Now that you know how to represent vectors using their component and matrix form we can now hard-code them in Python. Let's say that you have the vectors:

$$ A = 8\hat{x} + 11\hat{y}  -4\hat{z}\\
B = 15\hat{x} - 7\hat{y} + 10\hat{z}\\
C = 9ax + 10ay - 11az \\
D = 14\hat{i} - 3\hat{j} + 9\hat{k}$$

In which it's matrix equivalent is:

$$ A = \begin{bmatrix} 8 \\ 11\\ -4\end{bmatrix} , B = \begin{bmatrix} 15 \\ -7\\ 10\end{bmatrix} , C = \begin{bmatrix} 9 \\ 10 \\  -11 \end{bmatrix}, D = \begin{bmatrix} 14 \\ -3 \\ 9\end{bmatrix}
$$
$$ A = \begin{bmatrix} 8 & 11&-4\end{bmatrix} , B = \begin{bmatrix} 15 & -7 &  10\end{bmatrix} , C = \begin{bmatrix} 9 & 10 & -11\end{bmatrix} , D = \begin{bmatrix} 14 & -3 & 9\end{bmatrix}
$$

We can then start doing numpy code with this by:

```python
import numpy as np ## 'np' here is short-hand name of the library (numpy) or a nickname.
```

```python
A = np.array([8, 11, -4])
B = np.array([15, -7, 10])
C = np.array([
              [9],
              [10],
              [-11]
])
D = np.array ([[14],
              [-3],
              [9]])
print('Vector A is ', A)
print('Vector B is ', B)
print('Vector C is ', C)
print('Vector D is ', D)
```

    Vector A is  [ 8 11 -4]
    Vector B is  [15 -7 10]
    Vector C is  [[  9]
     [ 10]
     [-11]]
    Vector D is  [[14]
     [-3]
     [ 9]]

#### Describing vectors in NumPy

NumPy is at the core of many types of scientific computing and is the main tool that we will use to perform Linear Algebra operations with Python.

NumPy arrays can be used to represent vectors but cannot be used to differentiate between row and column vectors.

\begin{align}
\text{column vector} = \begin{bmatrix}1 \\ 2 \\ 3\end{bmatrix}
\end{align}

\begin{align}
\text{row vector} = \begin{bmatrix} a & b & c\end{bmatrix}
\end{align}

Describing vectors is very important if we want to perform basic to advanced operations with them. The fundamental ways in describing vectors are knowing their shape, size and dimensions.

```python
### Checking shapes
### Shapes tells us how many elements are there on each row and column

G = np.array([[15, 11, 2, 9, -0.5, 3, 10]])
G.shape
```

    (1, 7)

```python
### Checking size
### Array/Vector sizes tells us many total number of elements are there in the vector

G.size
```

    7

```python
### Checking dimensions
### The dimensions or rank of a vector tells us how many dimensions are there for the vector.

G.ndim
```

    2

Great! Now let's try to explore in performing operations with these vectors.

#### Addition

The addition rule is simple, the we just need to add the elements of the matrices according to their index. So in this case if we add vector $A$ and vector $B$ we will have a resulting vector:

$$R = 23\hat{x}+4\hat{y}+6\hat{z} \\ \\or \\ \\ R = \begin{bmatrix} 23 \\ 4\\6\end{bmatrix} $$

So let's try to do that in NumPy in several number of ways:

```python
R = np.add(A, B) ## this is the functional method using the numpy library
P = np.add(C, D)
print ("sum of A & B:")
print (R)
print ("sum of C & D:")
print (P)
```

    sum of A & B:
    [23  4  6]
    sum of C & D:
    [[23]
     [ 7]
     [-2]]

```python
R = A + B   ## this is the explicit method, since Python does a value-reference so it can
            ## know that these variables would need to do array operations.
R
```

    array([23,  4,  6])

```python
pos1 = np.array([0,0,0])
pos2 = np.array([0,1,3])
pos3 = np.array([1,5,-2])
pos4 = np.array([5,-3,3])
sumR = pos1 + pos2 + pos3 + pos4
productR = np.multiply(pos3, pos4)
R = pos3 / pos4
R

print("sum:")
print(sumR)
print("product:")
print(productR)
print("quotient:")
print(R)
```

    sum:
    [6 3 4]
    product:
    [  5 -15  -6]
    quotient:
    [ 0.2        -1.66666667 -0.66666667]

##### Try for yourself!

Try to implement subtraction, multiplication, and division with vectors $A$ and $B$!

$$ A = \begin{bmatrix} 18 & 4&7\\ 6&10&-1\\ -9&3&-11\end{bmatrix}  B = \begin{bmatrix} 7&-11&7 \\ -3&8&-13\\ 6&1&10\end{bmatrix}
$$

```python
A = np.array([
      [18,4,7],
      [6,10,-1],
      [-9,3,-11]
])
B = np.array([
      [7,-11,7],
      [-3,8,-13],
      [6,1,10]
])

diff = np.subtract(A,B)
prod = np.multiply(A,B)
quot = A/B

print("Difference:")
print(diff)
print("Product:")
print(prod)
print("Quotient:")
print(quot)
```

    Difference:
    [[ 11  15   0]
     [  9   2  12]
     [-15   2 -21]]
    Product:
    [[ 126  -44   49]
     [ -18   80   13]
     [ -54    3 -110]]
    Quotient:
    [[ 2.57142857 -0.36363636  1.        ]
     [-2.          1.25        0.07692308]
     [-1.5         3.         -1.1       ]]

### Scaling

Scaling or scalar multiplication takes a scalar value and performs multiplication with a vector. Let's take the example below:

$$S = 5 \cdot A$$

We can do this in numpy through:

```python
S = np.multiply(5,A)
S
```

    array([20, 15])

Try to implement scaling with two vectors.

$$ C = \begin{bmatrix} 11 & 6\\ 8&10\\ -4&7\end{bmatrix}  D = \begin{bmatrix} 7&-10 \\ -3&3\\ 5&-1\end{bmatrix}
$$

$$P = 8 \ (C+D)$$

```python
C = np.array([
    [11,6],
    [8,10],
    [-4,7]
])

D = np.array([
    [7,-10],
    [-3,3],
    [5,-1]
])

P = np.multiply(8,C+D)
P
```

    array([[144, -32],
           [ 40, 104],
           [  8,  48]])

### MatPlotLib

MatPlotLib or MATLab Plotting library is Python's take on MATLabs plotting feature. MatPlotLib can be used vastly from graping values to visualizing several dimensions of data.

#### Visualizing Data

It's not enough just solving these vectors so might need to visualize them. So we'll use MatPlotLib for that. We'll need to import it first.

```python
import matplotlib.pyplot as plt
import matplotlib
%matplotlib inline
```

```python
A = [8, -6]
B = [3, -3]
plt.scatter(A[0], A[1], label='A', c='green') #A[0] is Ax , A[1] is Ay , c=color
plt.scatter(B[0], B[1], label='B', c='magenta')
plt.grid()
plt.legend()
plt.show()
```

```python
A = np.array([1, -1])
B = np.array([1, 5])
R = A + B
Magnitude = np.sqrt(np.sum(R**2))
plt.title("Resultant Vector\nMagnitude:{}" .format(Magnitude))
plt.xlim(-5, 5) #limit of x
plt.ylim(-5, 5) #limit of y
plt.quiver(0, 0, A[0], A[1], angles='xy', scale_units='xy', scale=1, color='red')
plt.quiver(A[0], A[1], B[0], B[1], angles='xy', scale_units='xy', scale=1, color='green')
plt.quiver(0, 0, R[0], R[1], angles='xy', scale_units='xy', scale=1, color='black')
plt.grid()
plt.show()
print("Resultant:")
print(R)
print("Magnitude:")
print(Magnitude)
Slope = R[1]/R[0]
print("Slope:")
print(Slope)
Angle = (np.arctan(Slope))*(180/np.pi)
print("Angle:")
print(Angle)
```

```python
n = A.shape[0]
plt.xlim(-10, 10)
plt.ylim(-10, 10)
plt.quiver(0,0, A[0], A[1], angles='xy', scale_units='xy',scale=1)
plt.quiver(A[0],A[1], B[0], B[1], angles='xy', scale_units='xy',scale=1)
plt.quiver(0,0, R[0], R[1], angles='xy', scale_units='xy',scale=1)
plt.show()
```

Try plotting Three Vectors and show the Resultant Vector as a result.
Use Head to Tail Method.

In this task, we import the matplotlib to execute the limitations of x and y axis, the grid, and quievers that are significant on plotting single or multiple vectors in the Google Colaboratory. To begin the program, we need to input a values on the arrays that will be the basis or the starting point of the vectors. Quivers are the ones that connects two points to indicate a scaling unit and its direction- the head to tail method. This is controlled by setting a point to the given values of x and y of an array, for example, A[0] is for x of vector A. These things must be labeled on different colors to avoid confusion for the joining points. Lastly, the resultant, e sum of the vectors; the magnitude, square root of the squared resultant; slope, quotient of y over x of resultant; and angle, which is the arctan of slope divided by the ratio of 180 over pie can also be observed in the given figure.

## 2 Vectors

```python
C = np.array([-5, -3])
D = np.array([-4, 6])
R = C + D
Magnitude = np.sqrt(np.sum(R**2))
plt.title("Resultant Vector\nMagnitude:{}" .format(Magnitude))
plt.xlim(-10, 10) #limit of x
plt.ylim(-10, 10) #limit of y
plt.quiver(0, 0, C[0], C[1], angles='xy', scale_units='xy', scale=1, color='blue')
plt.quiver(C[0], C[1], D[0], D[1], angles='xy', scale_units='xy', scale=1, color='orange')
plt.quiver(0, 0, R[0], R[1], angles='xy', scale_units='xy', scale=1, color='brown')
plt.grid()
plt.show()
print("Resultant:")
print(R)
print("Magnitude:")
print(Magnitude)
Slope = R[1]/R[0]
print("Slope:")
print(Slope)
Angle = (np.arctan(Slope))*(180/np.pi)
print("Angle:")
print(Angle)
```

```python
E = np.array([8, 2])
F = np.array([1, 9])
R = E + F
Magnitude = np.sqrt(np.sum(R**2))
plt.title("Resultant Vector\nMagnitude:{}" .format(Magnitude))
plt.xlim(-15, 15) #limit of x
plt.ylim(-15, 15) #limit of y
plt.quiver(0, 0, E[0], E[1], angles='xy', scale_units='xy', scale=1, color='red')
plt.quiver(E[0], E[1], F[0], F[1], angles='xy', scale_units='xy', scale=1, color='blue')
plt.quiver(0, 0, R[0], R[1], angles='xy', scale_units='xy', scale=1, color='green')
plt.grid()
plt.show()
print("Resultant:")
print(R)
print("Magnitude:")
print(Magnitude)
Slope = R[1]/R[0]
print("Slope:")
print(Slope)
Angle = (np.arctan(Slope))*(180/np.pi)
print("Angle:")
print(Angle)
```

## 3 Vectors

```python
A = np.array([-1, 9.1])
B = np.array([-3, 2])
C = np.array([-4, -11])
R = A + B + C
Magnitude = np.sqrt(np.sum(R**2))
plt.title("Resultant Vector\nMagnitude:{}" .format(Magnitude))
plt.xlim(-15, 16) #limit of x
plt.ylim(-15, 16) #limit of y
plt.quiver(0, 0, A[0], A[1], angles='xy', scale_units='xy', scale=1, color='black')
plt.quiver(A[0], A[1], B[0], B[1], angles='xy', scale_units='xy', scale=1, color='purple')
plt.quiver(-4, 11, C[0], C[1], angles='xy', scale_units='xy', scale=1, color='brown')
plt.quiver(0, 0, R[0], R[1], angles='xy', scale_units='xy', scale=1, color='red')
plt.grid()
plt.show()
print("Resultant:")
print(R)
print("Magnitude:")
print(Magnitude)
Slope = R[1]/R[0]
print("Slope:")
print(Slope)
Angle = (np.arctan(Slope))*(180/np.pi)
print("Angle:")
print(Angle)
```


# Definition of Priors:

In this notebook we demonstrate how to include prior knowledge into a parameter inference problem, in particular how to define (log-)priors for parameters. If you want to maximize your posterior distribution, you need to define

* A (negative log-)likelihood
* A (log-)prior

The posterior is then built as an `AggregatedObjective`. If you import a problem via `PEtab` and the priors are contained in the parameter table, the definition of priors is done automatically.

**CAUTION**: The user needs to specify the **negative** _log-likelihood_, while the _log-prior_ is internally mulitplied by -1.

```python
import numpy as np
import scipy as sp

import pypesto
```

## Example: Rosenbrock Banana

We will use the Rosenbrock Banana

\begin{align}
f(x, \theta) = \sum_{i=1}^{N} \underbrace{100 \cdot(x_{i}-x_{i-1}^2)^2}_{\text{"negative log-likelihood"}} + \underbrace{(x_{i-1}-1)^2}_{\text{"Gaussian log-prior"}}
\end{align}

as an example. Here we interpret the first term as the _negative log-likelihood_ and the second term as Gaussian _log-prior_ with mean $1$ and standard deviation $1/\sqrt{2}$.

Note that the second term is only equivalent to the negative log-distribution of a Gaussian up to a constant.

### Define the negative log-likelihood

```python
n_x = 5

def rosenbrock_part_1(x):
    """
    Calculate obj. fct + gradient of the "likelihood" part.
    """
    obj = sum(100.0*(x[1:] - x[:-1]**2.0)**2.0)

    grad = np.zeros_like(x)
    grad[:-1] += -400 * (x[1:] - x[:-1]**2.0) * x[:-1]
    grad[1:] += 200 * (x[1:] - x[:-1]**2.0)

    return (obj, grad)

neg_log_likelihood = pypesto.Objective(fun=rosenbrock_part_1, grad=True)
```

### Define the log-prior

A prior on an individual paramater is defined in a `prior_dict`, which contains the following key-value pairs:

* `index`: Index of the parameter
* `density_fun`: (Log-)posterior. (Scalar function!)
* `density_dx`: d/dx (Log-)posterior (optional)
* `density_ddx`: d^2/dx^2 (Log-)posterior (optional)

A `prior_dict` can be either obtained by `get_parameter_prior_dict` for several common priors, or defined by the user.

```python
from pypesto.objective.priors import get_parameter_prior_dict

# create a list of prior dicts...
prior_list = []
mean = 1
std_dev = 1 / np.sqrt(2)

for i in range(n_x-1):
    prior_list.append(get_parameter_prior_dict(i, 'normal', [mean, std_dev]))

# create the prior
neg_log_prior = pypesto.objective.NegLogParameterPriors(prior_list)
```

### Define the negative log-posterior and the problem

The negative log-posterior is defined as an `AggregatedObjective`. Since optimization/visualization is not the main focus of this notebook, the reader is referred to other examples for a more in-depth presentation of these.

```python
neg_log_posterior = pypesto.objective.AggregatedObjective([neg_log_likelihood, neg_log_prior])

lb = -5 * np.ones((n_x, 1))
ub = 5 * np.ones((n_x, 1))

problem = pypesto.Problem(objective=neg_log_posterior,
                          lb=lb,
                          ub=ub)
```

### Optimize

```python
import pypesto.optimize as optimize

result = optimize.minimize(problem=problem, n_starts=10)
```

### Some basic visualizations

```python
import pypesto.visualize as visualize

visualize.waterfall(result, size=(15,6))

# parallel coordinates plot for best 5 fits
visualize.parameters(result, start_indices=5)
```

```python

```


Una vez finalizado el curso, por favor ayúdanos completando la siguiente **Encuesta de FINAL del curso**.

[](https://forms.office.com/Pages/ResponsePage.aspx?id=r4yvt9iDREaFrjF8VFIjwUHkKiCq1wxFstxAwkoFiilUOExRVkVMWlZERVcyWlpUU1EyTFg4T1Q3WC4u)

## 11. Ecuaciones Diferenciales

[Playlist de Ciencia de Datos en castellano](https://www.youtube.com/playlist?list=PLjyvn6Y1kpbEmRY4-ELeRA80ZywV7Xd67)
[](https://www.youtube.com/watch?v=HReAo38LoM4&list=PLLBUgWXdTBDg1Qgmwt4jKtVn9BWh5-zgy "Python Data Science")

Algunas ecuaciones con términos diferenciales surgen de relaciones fundamentales como conservación de masa, energía y momento. Por ejemplo, la aculumación de masa $\frac{dm}{dt}$ en un volumen de control es igual a la masa que entra $\dot m_{in}$ menos la masa que sale $\dot m_{out}$ de ese volumen.

$\frac{dm}{dt} = \dot m_{in} - \dot m_{out}$

Se puede desarrollar un modelo dinámico mediante regresión de datos o con relaciones fundamentales sin necesidad de datos. Incluso las relaciones fundamentales pueden tener parámetros desconocidos o inciertos. Un enfoque para el modelado dinámico es combinar relaciones físicas fundamentales con Ciencia de Datos. Este enfoque usa lo mejor de ambos métodos porque crea un modelo que se alinea con los valores medidos y puede extrapolarse a regiones donde los datos son limitados o inexistentes.

En este primer ejercicio para [resolver ecuaciones diferenciales](https://www.youtube.com/watch?v=v9fGOHQMeIA) vamos a utilizar `odeint`. Los mismos ejemplos también serán [resueltos con Gekko](https://apmonitor.com/pdc/index.php/Main/PythonDifferentialEquations). Ambos alcanzan resultados equivalentes de simulación. Sin embargo, Gekko está diseñado para usar ecuaciones diferenciales en optimización o combinarse con aprendizaje automático (machine learning). La función `odeint` tiene como propósito principal resolver ecuaciones diferenciales ordinarias (EDO), y requiere tres entradas (inputs).

    y = odeint(model, y0, t)

1. `model` Nombre de la Función que devuelve la derivada para un par de valores solicitados `y`, `t`, de la forma `dydt = model(y,t)`.
2. `y0` Condiciones iniciales.
3. `t` Puntos de tiempo donde se reporta la solución.

### Resolver Ecuaciones  Diferenciales

Resolveremos la ecuación diferencial con la condición inicial $y(0) = 5$:

$ k \, \frac{dy}{dt} = -y$

Donde $k=10$. La solución para `y` se reporta desde un tiempo inicial `0` hasta un tiempo final `20`. También se grafica el resultado para $y(t)$ vs. $t$. Notemos cómo se establece la ecuación para obtener la derivada como `dydt = -(1.0/k) * y` a partir de la función.

```python
import numpy as np
from scipy.integrate import odeint

# función que devuelve dy/dt
def model(y,t):
    k = 10.0
    dydt = -(1.0/k) * y
    return dydt

y0 = 5                 # condición inicial
t = np.linspace(0,20)  # puntos de tiempo
y = odeint(model,y0,t) # resolución de la ODE

import matplotlib.pyplot as plt
%matplotlib inline
plt.plot(t,y)
plt.xlabel('Tiempo'); plt.ylabel('y(t)')

plt.show()
```

### Resolver Ecuaciones Diferenciales con Gekko

[Python Gekko](https://gekko.readthedocs.io/en/latest/) resuelve la misma ecuación diferencial. Está diseñado para problemas a gran escala. El [tutorial de Gekko en inglés](https://apmonitor.com/wiki/index.php/Main/GekkoPythonOptimization) nos muestra cómo resolver otro tipo de problemas con ecuaciones y optimización.

```python
from gekko import GEKKO

m = GEKKO(remote=False)    # modelo GEKKO
m.time = np.linspace(0,20) # puntos de tiempo
y = m.Var(5.0); k = 10.0   # variables y constantes GEKKO
m.Equation(k*y.dt()+y==0)  # Ecuación GEKKO

m.options.IMODE = 4        # Simulación dinámica
m.solve(disp=False)        # Resolución

plt.plot(m.time,y)
plt.xlabel('Tiempo'); plt.ylabel('y(t)')
plt.show()
```

### Actividad sobre Ecuaciones Diferenciales

Resuelve la ecuación diferencial con condición inicial $y(0) = 10$:

$ k \, \frac{dy}{dt} = -y$

Compara las primeras cinco soluciones de `y` entre los tiempos `0` y `20` con `k=[1,2,5,10,20]`.

```python

```

### Solución Simbólica

Los problemas con ecuaciones diferenciales que tienen solución analítica pueden expresarse simbólicamente. Una librería con símbolos matemáticos en Python es `sympy`. Sympy determina la solución analítica como $y(x)=C_1 \, \exp{\left(-\frac{x}{k}\right)}$. Con la condición inicial $y(0)=5$, y la constante $C_1$ igual a 5.

```python
from IPython.display import display
import sympy as sym
from sympy.abc import x, k
y = sym.Function('y')
ans = sym.dsolve(sym.Derivative(y(x), x) + y(x)/k, y(x))
display(ans)
```

### Resolver Ecuaciones Diferenciales con Entradas (Input) `u`

Las ecuaciones diferenciales también pueden tener una entrada (atributo) que cambie desde una fuente externa (entrada exógena). Por ejemplo, cambios interactivos debido a medidas de un sensor, a personas (manualmente) o seleccionados por un computador.

Calcula la respuesta `y(t)` cuando la entrada `u` cambia desde `0` a `2` en `t = 5`.

$2 \frac{dy(t)}{dt} + y(t) = u(t)$

La condición inicial es `y(0)=1` y la solución puede calcularse hasta `t=15`. **Ayuda**: La expresión `y(t)` no es equivalente a `y` multiplicado por `t`. Esta indica que `y` cambia con el tiempo y se escribe como una función del tiempo. Hay ejemplos adicionales para [odeint en inglés](https://apmonitor.com/pdc/index.php/Main/SolveDifferentialEquations) y [Gekko en inglés](https://apmonitor.com/pdc/index.php/Main/PythonDifferentialEquations) por si necesitas ayuda.

```python

```

### Actividad con el TCLab

### Recolección de Datos

Enciende el calentador 1 al 100% y guarda el valor de $T_1$ cada 5 segundos durante 3 minutos. Los datos deben incluir un total de 37 puntos para cada sensor de temperatura.

```python
import numpy as np
import pandas as pd
import tclab
import time
# Recolectar datos por 3 minutos, cada 5 segundos
n = 37
tm = np.linspace(0,180,n)
t1s = np.empty(n); t2s = np.empty(n)
with tclab.TCLab() as lab:
    lab.Q1(100); lab.Q2(0)
    print('Tiempo T1  T2')
    for i in range(n):
        t1s[i] = lab.T1; t2s[i] = lab.T2
        print(tm[i],t1s[i],t2s[i])
        time.sleep(5.0)
# Colocar en un Dataframe
data = pd.DataFrame(np.column_stack((tm,t1s,t2s)),\
                    columns=['Tiempo','T1','T2'])
data.to_csv('11-data.csv',index=False)
```

### Resolver Ecuaciones Diferenciales

Usa los parámetros `a`, `b` y `c` del módulo [10. Resolver Ecuaciones](https://github.com/APMonitor/data_science/blob/master/10.%20Solve_Equations.ipynb) o utiliza los siguientes valores:

| Parámetro | Valor |
|------|------|
| a    | 78.6 |
| b    | -50.3 |
| c    | -0.003677 |

Resuelve la ecuación diferencial ordinaria (ODE en inglés) con estos valores.

$\frac{dT_1}{dt} = c (T_1-a)$

La condición inicial para $T_1$ es $a + b$. Muestra la solución para la ODE en el intervalo de tiempo desde `0` hasta `180` segundos. Grafica el valor medido de $T_1$ en la misma figura que muestra la predicción de la temperatura por la ODE. Añade las etiquetas necesarias en el gráfico.

```python

```


# Balken

Die analytischen (1D) Balkenbiegeformeln stehen z.B. in [Gross et al., 2014] oder [Landau, Lifschitz, 1975]. Im folgenden Beispiel wird ein elastischer Zylinder aus Baustahl, mit einem quadratischen Querschnitt von 0.0001 $m^2$ und einer Länge von 0.1 $m^2$ unter Eigengewicht gebogen.

```python
import numpy as np
import plotly.express as px
import sympy as sym
import matplotlib.pyplot as plt
import matplotlib as mpl
from matplotlib import ticker
import seaborn as sns
#sns.set()
mpl.rcParams["figure.figsize"] = (20,7)
mpl.rcParams['lines.linewidth'] = 5
mpl.rcParams['lines.markersize'] = 15
#sns.set()
farben = sns.color_palette()
plt.style.use('seaborn-darkgrid')
farben = sns.color_palette()
```

```python
x, f, E, I, l = sym.symbols('x, f, E, I, l', real=True)
```

```python
u = f/(24*E*I)*(x**4-4*l*x**3+6*l**2*x**2)
u

```

$\displaystyle \frac{f \left(6 l^{2} x^{2} - 4 l x^{3} + x^{4}\right)}{24 E I}$

```python
u_e = f/(24*E*I)*x**2*(x-l)**2
u_e
```

$\displaystyle \frac{f x^{2} \left(- l + x\right)^{2}}{24 E I}$

```python
ud = sym.diff(u, x)
ud
```

$\displaystyle \frac{f \left(12 l^{2} x - 12 l x^{2} + 4 x^{3}\right)}{24 E I}$

```python
A = 0.0001
f_ = -7850*9.8066*A
E_ = 2*10**11
I_ = (0.01**4)/12
l_ = 0.1

u_ = sym.lambdify(x, u.subs(l, l_).subs(f, f_).subs(E, E_).subs(I, I_))
u_e_ = sym.lambdify(x, u_e.subs(l, l_).subs(f, f_).subs(E, E_).subs(I, I_))
```

```python
u_(0.1)
```

    -5.773635750000001e-07

```python

```

```python
ud_ = sym.lambdify(x, ud.subs(l, l_).subs(f, f_).subs(E, E_).subs(I, I_))

-ud_(0.1)*0.01*0.5
```

    3.849090499999999e-08

```python
linie = np.linspace(0, l_)

#px.line(x=linie, y=u_(linie)*10**7)
```

## Ein Ende frei

```python
ax=plt.axes()
ax.tick_params(labelsize=15)
ax.plot(linie, u_(linie)*10**6, label = r'u analytisch', color=farben[0]),
ax.tick_params(labelsize=15)
ax.set_xlabel(xlabel='x in m',fontsize=15)
ax.set_ylabel(ylabel=r'u in $\mu m$ ',fontsize=15)
plt.legend(loc='best',fontsize=15)
ax.set_title(r"$%s$"%sym.latex(u), fontsize=25)
```

## Beide Enden eingespannt

```python
ax=plt.axes()
ax.tick_params(labelsize=15)
ax.plot(linie, u_e_(linie)*10**6, label = r'u analytisch', color=farben[0]),
ax.tick_params(labelsize=15)
ax.set_xlabel(xlabel='x in m',fontsize=15)
ax.set_ylabel(ylabel=r'u in $\mu m$ ',fontsize=15)
plt.legend(loc='best',fontsize=15)
ax.set_title(r"$%s$"%sym.latex(u_e), fontsize=25)
```


# Lecture 21:  The Calculus of Variations

## What to Learn?

* The concept of a "function of functions" and the definition of a functional
* The concept of finding a function that makes the functional an extremum
* How to practically compute the functional derivative for simple problems
* The concept of a conserved and non-conserved order parameter
* The definition of the non-classical chemical potential in a heterogeneous system
* How to use the functional derivative to solve for the order parameter profile through an interface

## What to do?

* Recall the arc length formula
* Write down a functional for all arc lengths between two points
* Find the shortest path length between two points (minimize the functional)
* Using the above process, find the shape of the minimum soapfilm between two rings
* Using the above process, set up the differential equation for a heterogeneous chemical system

$$
F(y_x,y,x)
$$

A functional is a function of functions.  It is necessary to treat $x$, $y$, and $y_x$ as independent (as though they are held constant during partial differentiation.

### On Your Own
----

An (Imperfect but Colorful) Analogy

Using the calculus of variations is like this: you want to travel to Phoenix, AZ but you don't yet know the cheapest and fastest way to get there.  So - you imagine a nearly exhaustive list of ways to travel there (including things like walking, giant trebuchets, teleportation, etc.) and work out the costs and time required to each mode of transportation.  Once you evaluate all the modes (consider each mode as a different function of cost and time) - you pick the mode (function) that is optimal for cost and time.

In a picture, imagine all the functions that connect the two points "A" and "B".  We are searching for the function that minimizes the path between "A" and "B" subject to whatever constraints we place on the path.  The calculus of variations is a formal mathematical strategy for FINDING that function from all possible functions.

If calculus describes how numbers behave when mapped through functions, then the calculus of variations describe how functions behave when mapped through functions-of-functions.

Using the mean value theorem you can derive a formula for arc-length that reads:

$$
L(x,y,y_x) = \int_a^b \sqrt{1+ \left( \frac{dy}{dx} \right)^2} dx
$$

You can integrate this expression between two points $a$ and $b$ on a function $y(x)$ to get the length of the line between $a$ and $b$.  In the CoV we call $F$ a functional.  A functional is a function of functions.

The utility of the CoV is to produce a differential equation that is subsequently solved to produce a function that makes $F$ an extreme value.  In this case we are searching for the function $y(x)$ that minimizes $F$ betwen two points.  The CoV tells us that the following equation must be true for $y(x)$ to make $F$ an extreme value:

$$
\frac{\delta F}{\delta y} = \frac{\partial F}{\partial y} - \frac{d}{dx} \left( \frac{\partial F}{\partial y_x} \right)= 0
$$

This expression (for one dependent and one independent variable) is the core of the CoV.  IT is not the only result that can be developed, but for us this is the important one.  We can start by writing the above equation "by hand".

We'll start with the usual imports:

```python
%matplotlib notebook
import sympy as sp
sp.init_printing()

f = sp.symbols('f', cls=sp.Function)
x, y = sp.symbols('x, y', real=True)
```

We define our functional.  Note that the functional is only a function of the _derivative_ of $y$ rather than $y$ alone.

```python
f = sp.sqrt(1+(y(x).diff(x))**2)
f
```

You may be tempted to do this:

```python
f.diff(y)-(f.diff(y(x).diff(x))).diff(x)
```

You wouldn't be wrong to do the above, however things can be made a bit easier.  Let us look at each term in the ELE (Euler-Lagrange Equations):

```python
firstTerm = f.diff(y(x))
firstTerm
```

```python
secondTerm = f.diff(y(x).diff(x))
secondTerm
```

So - since the first term is zero we can use the fundamental theorem of the calculus to perform the first integral.  We add a constant and solve for the derivative.

```python
sp.var('C1')
integratedFunctional = sp.Eq(secondTerm,C1)
integratedFunctional
```

```python
firstSolution = sp.solve(integratedFunctional, y(x).diff(x))
firstSolution
```

This clearly indicates that:

$$
\frac{dy}{dx} = C
$$

and from this point it should be clear that the function $y(x)$ and makes $F$ an extreme is:

$$
y = mx + b
$$

If you would like to have `SymPy` finish the calculation, you can write:

```python
functionalExtremizer = sp.dsolve(sp.Eq(y(x).diff(x),firstSolution[0]), y(x))
functionalExtremizer
```

And that is a linear function.  To the extent that this is a proof - you've proven that a straight line is the shortest distance between two points.

#### Using SymPy's Functions

`SymPy` has an `euler_equation` function that we can try to use, too:

```python
L = sp.sqrt(1+(y(x).diff(x))**2)
differentialEquationFromELFunction = sp.euler_equations(L, y(x), x)
differentialEquationFromELFunction
```

A bit messy, but correct nonetheless.

#### DIY:  Find the Euler-Lagrange Equation (ELE)
----

Find the ELE for the functional, and if you can - solve for $y(x)$:

$$
v(y(x)) = \int_0^{\pi/2} (y_x^2 - y^2)dx
$$

The endpoint conditions are $y(0)=0$ and $y(\pi/2)=1$.  For reference, the general solution is:

$$
y(x)=C_1 \sin(x) + C_2 \cos(x)
$$

Don't forget to check the end points of the domain to find the constants.

#### The Problem of a Minimum Soapfilm
----

A classic problem in wetting and capillary science is that of the minimum soapfilm between two rings.  The soap film adopts a shape that minimizes its area.

The area of a soap film (found by rotating a curve through $2\pi$ around one axis is given by:

$$
A = L(x,y,y_x) = \int_{x_1}^{x_2} 2 \pi y (1+y_x^2)^{1/2} dx
$$

Note there is no explicit x dependence.

```python
Lsoapfilm = y(x)*sp.sqrt(1+(y(x).diff(x))**2)
(sp.euler_equations(Lsoapfilm,y(x),x)[0].lhs).simplify()
```

Attacking the problem this way leads to a second order ODE that we need to integrate.  Although this could be done - the lack of an explicit $x$ dependence permits using an identity that makes the problem a bit easier.

An equivalent statement of the ELE is:

$$
\frac{d}{dx} \left(F - y_x \frac{\partial F}{\partial y_x} \right) = \frac{\partial F}{\partial x}
$$

If there is no explicit $x$ dependence therefore the RHS of the above equation is zero - the first integral can be had for "free". Adding the integration constant we have:

$$
F - y_x \frac{\partial F}{\partial y_x} = C_2
$$

We can therefore write:

```python
C2 = sp.symbols('C2', positive=True)
beltramiODE = sp.Eq(Lsoapfilm - y(x).diff(x)*Lsoapfilm.diff(y(x).diff(x)),C2)
beltramiODE.simplify()
```

Now we solve the differential equation using `dsolve`.

```python
sp.dsolve(beltramiODE,y(x))
```

#### DIY:  Use the general solution and find the constants.

```python
# Find the constants if the curve is required to pass through a pair of particular points.
```

#### DIY:  Create a tool to explore the shape of different soapfilms.

```python
# Create an interactive widget to explore the values of the constants.
```

### Homework
----

Find the variational derivative of the following functional:

$$
F = W \phi^2 (1-\phi)^2 + \frac{1}{2} \epsilon^2 \left(\frac{d \phi}{dx} \right)^2
$$

e.g. compute:

$$
\frac{\delta F}{\delta \phi}
$$

The ODE that results will be solved to find the function $\phi(x)$.  $W$ and $\epsilon$ are parameters of the functional.

```python

```


# Part 2: Logistic Regression and Gradient Checking

```python
# Execute this code block to install dependencies when running on colab
try:
    import torch
except:
    from os.path import exists
    from wheel.pep425tags import get_abbr_impl, get_impl_ver, get_abi_tag
    platform = '{}{}-{}'.format(get_abbr_impl(), get_impl_ver(), get_abi_tag())
    cuda_output = !ldconfig -p|grep cudart.so|sed -e 's/.*\.\([0-9]*\)\.\([0-9]*\)$/cu\1\2/'
    accelerator = cuda_output[0] if exists('/dev/nvidia0') else 'cpu'

    !pip install -q http://download.pytorch.org/whl/{accelerator}/torch-1.0.0-{platform}-linux_x86_64.whl torchvision
```

In the first part of the lab we saw how to make predictions of continously varying values with a linear regression model. Lets now turn our focus to binary classification using a simple classification algorithm known as Logistic regression.

In linear regression we tried to predict the value of $y$ for an example $\mathbf{x}$ using a linear function $y=\mathbf{x}^\top\theta$ (where $\mathbf{x}$ and $\theta$ are column-vectors). This will clearly not be a great solution for predicting binary-valued labels ($y\in\{0,1\}$). In logistic regression we use a different hypothesis class to try to predict the probability that a given example belongs to the "1" class versus the probability that it belongs to the "0" class. Specifically, we will try to learn a function of the form:

\begin{align}
P(y=1|\mathbf{x}) &= \frac{1}{1 + \exp(-\mathbf{x}^\top\theta)} \equiv \sigma(\mathbf{x}^\top\theta),\\
P(y=0|\mathbf{x}) &= 1 - P(y=1|\mathbf{x}).
\end{align}

The function $\sigma(z) \equiv \frac{1}{1 + \exp(-z)}$ is called the "sigmoid" or "logistic" function. The sigmoid function squashes any real valued input into the range $[0,1]$ enabling us to interprete the output as a probability. Our goal is to search for a value of $\theta$ so that the probability $P(y=1|\mathbf{x})=\sigma(\mathbf{x}^\top\theta)$ is large when $\mathbf{x}$ belongs to the "1" class and small when $\mathbf{x}$ belongs to the "0" class (so that $P(y=0|\mathbf{x})$ is large).

With Linear Regression, the natural cost function was one that measured the sum of squared residuals (the difference between the predicted value and true value). With logisitic regression we have a probabilisitic model, so it makes sense that we use a function that measures the likelihood of the data given the model (note that we want to maximise this function rather than minimise it). As an aside, note that in the case of linear regression if we assume that the data has errors that are IID (independently and identically distributed) according to a Normal distribution, then it can be shown that the maximising the likelihood is exactly the same as minimising the sum of squared residuals. For logistic regression, the likelihood function for a single data point is:

\begin{align}
p(y|\mathbf{x}; \theta) &= \sigma(\mathbf{x}^\top\theta)^y(1-\sigma(\mathbf{x}^\top\theta)^{(1-y)}.
\end{align}

For a complete dataset of points $(y_i, \mathbf{x}_i)$, then the complete likelihood is:

\begin{align}
L(\theta) &= \prod_i \sigma(\mathbf{x}_i^\top\theta)^{y_i}(1-\sigma(\mathbf{x}_i^\top\theta)^{(1-y_i)}
\end{align}

However, it is considerably easier to maximise the log-likelihood function:

\begin{align}
\mathcal{l}(\theta) &= \log L(\theta) \\
                    & = \log \prod_i \sigma(\mathbf{x}_i^\top\theta)^{y_i}(1-\sigma(\mathbf{x}_i^\top\theta)^{(1-y_i)} \\
                    & = \sum_i y_i \log(\sigma(\mathbf{x}_i^\top\theta)) + (1-y_i) \log(1-\sigma(\mathbf{x}_i^\top\theta))
\end{align}

Clearly, maximising the log-likelihood is equivalent to minimising the negative log-likelihood. The negative of the log-likelihood function having the form $-\sum_i y_i \log(p) + (1-y_i) \log(p)$, where p is a function returning the predicted probability of class "1", is often called the __"Binary Cross Entropy"__ function, __"Binary Cross Entropy Loss"__ or sometimes the __"log loss"__.

For conciseness and computational efficiency, we can write the negative logistic regression log-likelihood function in matrix form. Assuming the $y_i$ are stored in a column vector $\mathbf{y}$ and the data vectors $x_i$ in the __rows__ of a matrix $\mathbf{X}$, then:

\begin{align}
\mathrm{NLL}(\theta) & = -(\mathbf{y} \log(\sigma(\mathbf{X}\theta)) + (1-\mathbf{y}) \log(1-\sigma(\mathbf{X}\theta)))
\end{align}

The gradients of this function are given by:

\begin{align}
\nabla_\theta \mathrm{NLL}(\theta) & = \mathbf{X}^T(\sigma(\mathbf{X}\theta) - \mathbf{y})
\end{align}

__Use the box below to compute the gradients of the negative log-likelihood function $\nabla_\theta \mathrm{NLL}(\theta)$. You can use `torch.sigmoid()` to apply the sigmoid function.__

```python
import torch

# we wouldn't normally do this, but for this lab we want to work in double precision
# as we'll need the numerical accuracy later on for doing checks on our gradients:
torch.set_default_dtype(torch.float64)

def logistic_regression_loss_grad(theta, X, y):
    # YOUR CODE HERE
    raise NotImplementedError()
    return grad
```

```python
theta = torch.zeros(1)
X = torch.Tensor([[1]])
y = torch.Tensor([[0]])
assert(logistic_regression_loss_grad(theta, X, y) == 0.5)

```

## Training a Logistic Regressor with real data

We'll now try gradient descent using our gradient function on a real dataset from `scikit-learn` called `digits`.

The `digits` dataset contains handwritten characters (much like the `MNIST` dataset that you may have heard of - we'll explore `MNIST` in a future lab). As logistic regression is a binary classifier, we'll just use the first 2 characters (0 and 1) from the `digits` dataset, and make our own training and test splits:

```python
from sklearn.datasets import load_digits

X, y = tuple(torch.Tensor(z) for z in load_digits(2, True)) #convert to pytorch Tensors
X = torch.cat((X, torch.ones((X.shape[0], 1))), 1) # append a column of 1's to the X's
y = y.reshape(-1, 1) # reshape y into a column vector

# We're also going to break the data into a training set for computing the regression parameters
# and a test set to evaluate the predictive ability of those parameters
perm = torch.randperm(y.shape[0])
X_train = X[perm[0:260], :]
y_train = y[perm[0:260]]
X_test = X[perm[260:], :]
y_test = y[perm[260:]]
```

Now we have the data, we can use our loss function to try and estimate the optimal parameters for the two-digit classification problem. We'll use `PyTorch`s `torch.nn.functional.binary_cross_entropy_with_logits` function to print out the Binary Cross Entropy of the training data at each iteration, and of the test data once the optimisation is complete.

Note: `logits` refers to unscaled probabilities before the sigmoid is applied, so in the `binary_cross_entropy_with_logits` function we just pass in $\mathbf{X}\theta$. `PyTorch` does also have a `torch.nn.binary_cross_entropy` method that takes in probabilities, however, as we'll see when implementing neural networks in a later lab, we'll often choose to work with logits as they provide better numerical stability thanks to the _log-sum-exp_ trick.

```python
alpha = 0.001
theta_gd = torch.rand((X_train.shape[1], 1))
for e in range(0, 10):
    gr = logistic_regression_loss_grad(theta_gd, X_train, y_train)
    theta_gd -= alpha * gr
    print("Epoch:", e, " BCE of training data:", torch.nn.functional.binary_cross_entropy_with_logits(X_train @ theta_gd, y_train))

print("Gradient Descent Theta:", theta_gd.t())
print("BCE of test data:", torch.nn.functional.binary_cross_entropy_with_logits(X_test @ theta_gd, y_test))
```

## Gradient Checking

How can we be sure that our gradient function was correct? We might have made an error in the analytical derivation or in the implementation of that derivation into code. Even though we observed the optimisation process on real data converge (hopefully!), you might have made a subtle error in the implementation...

So far we have worked with relatively simple algorithms where it is straightforward to compute the objective function and its gradient with pen-and-paper, and then implement the necessary computations in PyTorch. For more complex models that we will see later, the gradient computation can be notoriously difficult to debug and get right. Sometimes a subtly buggy implementation will manage to learn something that can look surprisingly reasonable (while performing less well than a correct implementation). Thus, even with a buggy implementation, it may not at all be apparent that anything is amiss. In this section, we describe a method for numerically checking the derivatives computed by your code to make sure that your implementation is correct. Carrying out the derivative checking procedure described here will significantly increase your confidence in the correctness of your code.

Suppose we want to minimize $J(\theta)$ as a function of $\theta$. For this example, suppose $J:\mathbb{R}\mapsto\mathbb{R}$, so that $\theta \in \mathbb{R}$. If we are using gradient descent (or other gradient-based optimisation function), then we usually have implemented some function $g(\theta)$ that purportedly computes $\frac{d}{d\theta}J(\theta)$.

How can we check if our implementation of $g$ is correct?

Recall the mathematical definition of the derivative is:

\begin{align}
\frac{d}{d\theta}J(\theta) = \lim_{\epsilon \rightarrow 0}
\frac{J(\theta+ \epsilon) - J(\theta-\epsilon)}{2 \epsilon}.
\end{align}

Thus, at any specific value of $\theta$, we can numerically approximate the derivative as follows:

\begin{align}
\frac{J(\theta+{\rm EPSILON}) - J(\theta-{\rm EPSILON})}{2 \times {\rm EPSILON}}
\end{align}

In practice, we set ${\rm EPSILON}$ to a small constant, say around $10^{−4}$. (There is a large range of values of EPSILON values that should work well, but we don’t set ${\rm EPSILON}$ to be "extremely" small, say $10^{−20}$, as that would lead to numerical roundoff errors.)

Thus, given a function $g(\theta)$ that is supposedly computing $\frac{d}{d\theta}J(\theta)$, we can now numerically verify its correctness by checking that

\begin{align}
g(\theta) \approx
\frac{J(\theta+{\rm EPSILON}) - J(\theta-{\rm EPSILON})}{2 \times {\rm EPSILON}}.
\end{align}

The degree to which these two values should approximate each other will depend on the details of $J$. But assuming ${\rm EPSILON}=10^{−4}$, you’ll usually find that the left- and right-hand sides of the above will agree to at least 4 significant digits (and often many more).

Now, consider the case where $\theta \in \mathbb{R}^n$ is a vector rather than a single real number (so that we have $n$ parameters that we want to learn), and $J: \mathbb{R}^n \mapsto \mathbb{R}$. We now generalize our derivative checking procedure to the case where $\theta$ may be a vector (as in our linear regression and logistic regression examples).

Suppose we have a function $g_i(\theta)$ that purportedly computes $\frac{\partial}{\partial\theta_i}J(\theta)$; we’d like to check if $g_i$ is outputting correct derivative values. Let $\textstyle \theta^{(i+)} = \theta + {\rm EPSILON} \times \vec{e}_i$, where

\begin{align}
\vec{e}_i = \begin{bmatrix}0 \\ 0 \\ \vdots \\ 1 \\ \vdots \\ 0\end{bmatrix}
\end{align}

is the $i$-th basis vector (a vector of the same dimension as $\theta$, with a "1" in the $i$-th position and "0"s everywhere else). So, $\theta^{(i+)}$ is the same as $\theta$, except its $i$-th element has been incremented by ${\rm EPSILON}$. Similarly, let $\theta^{(i−)}=\theta−{\rm EPSILON} \times \vec{e}_i$ be the corresponding vector with the $i$-th element decreased by ${\rm EPSILON}$.

We can now numerically verify $g_i(\theta)$'s correctness by checking, for each $i$, that:

\begin{align}
g_i(\theta) \approx
\frac{J(\theta^{(i+)}) - J(\theta^{(i-)})}{2 \times {\rm EPSILON}}.
\end{align}

### Gradient checker code

The following code block contains an implementation of the gradient checking proceedure described above.

```python
from random import randrange

def grad_check(f, x, analytic_grad, num_checks=10, h=1e-5):
    sum_error = 0
    for i in range(num_checks):
        ix = tuple([randrange(m) for m in x.shape]) #randomly sample value to change

        oldval = x[ix].item()
        x[ix] = oldval + h # increment by h
        fxph = f(x) # evaluate f(x + h)
        x[ix] = oldval - h # increment by h
        fxmh = f(x) # evaluate f(x - h)
        x[ix] = oldval # reset

        grad_numerical = (fxph - fxmh) / (2 * h)
        grad_analytic = analytic_grad[ix]
        rel_error = abs(grad_numerical - grad_analytic) / (abs(grad_numerical) + abs(grad_analytic) + 1e-8)
        sum_error += rel_error
        print('numerical: %f\tanalytic: %f\trelative error: %e' % (grad_numerical, grad_analytic, rel_error))
    return sum_error / num_checks
```

To use the gradient checker, we provide our analytical gradients, together with a function that computes the actual loss (rather than the gradients of the loss) and the parameters at which the gradient was computed:

```python
#we'll use random parameters:
theta = torch.rand_like(theta_gd)*0.001
# and compute the analytic gradient (w.r.t the test data we loaded in this case)
grad = logistic_regression_loss_grad(theta, X_test, y_test)

# we need a function that computes the loss for a given theta (and implicitly the data)
def func(th):
    sigm = torch.sigmoid(X_test @ th)
    f = -(y_test.t() @ torch.log(sigm) + (1 - y_test.t()) @ torch.log(1 - sigm));
    return f

# and run the gradient checker
relerr = grad_check(func, theta, grad)
print("average error:", relerr)

assert relerr < 1e-5

```