Examples for Ordinary Differential Equations (Initial Value Problem)
===============================================

## Part 1

### Suggestion

Run the code cell below before the others to make sure that scipy is up to date and the figures render well.

```python
import matplotlib.pyplot as plt
plt.rcParams['figure.figsize'] = (8,6)
plt.rcParams['font.size'] = 20
!pip3 install --upgrade scipy
```

    Requirement already satisfied: scipy in /usr/local/anaconda3/anaconda3/lib/python3.9/site-packages (1.8.1)
    Requirement already satisfied: numpy<1.25.0,>=1.17.3 in /usr/local/anaconda3/anaconda3/lib/python3.9/site-packages (from scipy) (1.20.3)

**The Damped Harmonic Oscillator**

### The Problem

Use the $rk23$ (Second order Runge-Kutta) method and plot a solution $x(t)$ for the damped harmonic oscillator, described by the differential equations

\begin{equation}
\frac{d^2x}{dt^2} + \gamma\frac{dx}{dt} + x = 0.
\end{equation}

Also, obtain phase space plots for some values of $\gamma$.

We can recast the equation above in terms of the phase space point $y = \left[y_0, y_1\right] = \left[x, \dot{x}\right]$, yielding

\begin{align*}
\dot{y}_0 &= y_1\\
\dot{y}_1 &= -y_0 - \gamma y_1
\end{align*}

The code cell below has code that uses 'solve_ivp' routine in the 'scipy.integrate' module to solve the dynamics from $y_0=1, y_1=0$ at $t=0$. The next code cell plots the phase space, $y_1$ vs $y_0$, for a numpy grid of initial conditions.

```python
import numpy as np
from scipy.integrate import solve_ivp

def func(t, y, g):
    y0, y1 = y
    dy0dt = y1
    dy1dt = -y0 - g * y1
    return np.array([dy0dt, dy1dt])

y_init = [1.0, 0.0]
t_final = 30
times = np.linspace(0.0, t_final, 1000)

plt.title("Amplitude-time")
plt.xlabel('t')
plt.ylabel('x', rotation=0)
gammas = [0.5, 0.8, 1.0, 1.5, 2.0]
for g in gammas:
    soln = solve_ivp(func, (0, t_final), y_init, t_eval=times, method='RK23',\
                     args=(g,))
    plt.plot(times, soln.y[0], label=f'g = {g:1.1f}')

plt.legend()

plt.show()
```

```python
%%time
import matplotlib.pyplot as plt
gammas = [0.4, 0.9, 1.5]
t_final = 30
times = np.linspace(0.0, t_final, 1000)

# Grid of initial conditions, recast as an iterable object 'ics'
y0m, y0mx = -10, 10
y1m, y1mx = -6, 6
Y0, Y1 = np.mgrid[y0m:y0mx:6j, y1m:y1mx:6j]
ics = np.vstack([Y0.ravel(), Y1.ravel()]).T

def func(t, y, g):
    y0, y1 = y
    dy0dt = y1
    dy1dt = -y0 - g * y1
    return np.array([dy0dt, dy1dt])

plt.rcParams['figure.figsize'] = (len(gammas)*6,6)
for i,g in enumerate(gammas):
    plt.subplot(1,len(gammas),i+1)
    plt.title(f"Phase Space for g={g:1.1f}")
    plt.xlabel('x')
    plt.ylabel('dxdt', rotation=0)
    for p in ics:
        soln = solve_ivp(func, (0, t_final), p, t_eval=times, method='RK23',\
                         args=(g,))
        plt.plot(soln.y[0], soln.y[1], "b-")

plt.show()
```

**The Driven Oscillator with Damping**

### The Problem

Consider the previous problem, but add a forced external oscillation:

\begin{equation}
\frac{d^2x}{dt^2} + \gamma\frac{dx}{dt} + x = x_0\sin{\omega t}.
\end{equation}

We can recast the equation above in terms of the phase space point $y = \left[y_0, y_1\right] = \left[x, \dot{x}\right]$, yielding

\begin{align*}
\dot{y}_0 &= y_1\\
\dot{y}_1 &= -y_0 - \gamma y_1 + x_0\sin{\omega t}
\end{align*}

Use the general $rk45$ integrator (this method is sometimes called the **Runge-Kutta-Fehlberg Method**) to obtain plots of the amplitude as a function of time for different driving frequencies $\omega$. Plot the root mean-squared amplitude (taken over very long times) as a function of $\omega$ and show that there is a resonance. Compare the runtimes of a trial with the older $rk23$ method and the stiffness-detecting implicit $LSODA$ method, keeping the same error bounds for both. Finally, you can experiment with the driving term by replacing the sine wave with a square wave and see what happens.

```python
import numpy as np
import matplotlib.pyplot as plt
plt.rcParams['figure.figsize'] = (16,6)
from scipy.integrate import solve_ivp
from scipy.signal import square

x0 = 1
g = 0.01
y_init = [1.0, 0.0]

def func(t, y, g, x0, w):
    y0, y1 = y
    dy0dt = y1
    dy1dt = -y0 - g * y1 + x0 * np.sin(w * t)
    #dy1dt = -y0 - g * y1 + x0 * square(w * t)
    return np.array([dy0dt, dy1dt])

plt.subplot(1,2,1)
plt.title("Amplitude-time")
plt.xlabel('t')
plt.ylabel('x', rotation=0)
omegas = [0.5, 1.0, 1.5, 2.0]
t_final = 30
times = np.linspace(0.0, t_final, 1000)

for w in omegas:
    soln = solve_ivp(func, (0, t_final), y_init, t_eval=times, method='RK45',\
                     args=(g,x0, w))
    plt.plot(times, soln.y[0], label=f'w = {w:1.1f}')

plt.legend()

plt.subplot(1,2,2)
plt.title("Resonance")
plt.xlabel('w')
plt.ylabel('rms', rotation=0)

omegas = np.linspace(0.1, 2.1, 400)
rms = np.zeros_like(omegas)

t_final = 100
times = np.linspace(0.0, t_final, 4000)
for i,w in enumerate(omegas):
    soln = solve_ivp(func, (0, t_final), y_init, t_eval=times, method='RK45',\
                     args=(g,x0, w))
    rms[i] = np.average(np.abs(soln.y[0])**2) - np.average(soln.y[0])**2

plt.vlines(1.0, 0.0, np.amax(rms),colors='black', linestyles='dashed')
plt.plot(omegas, rms)
plt.show()
```

```python
x0 = 1
g = 0.05
y_init = [1.0, 0.0]
w = 0.6
t_final = 50
times = np.linspace(0, t_final, 1000)

print("Runtimes for rk23:")
%timeit soln = solve_ivp(func, (0, t_final), y_init, t_eval=times, method='RK23',\
                         args=(g,x0, w), rtol=1e-5, atol=1e-8)
print("\nRuntimes for rk45:")
%timeit soln = solve_ivp(func, (0, t_final), y_init, t_eval=times, method='RK45',\
                         args=(g,x0, w), rtol=1e-5, atol=1e-8)
print("\nRuntimes for lsoda:")
%timeit soln = solve_ivp(func, (0, t_final), y_init, t_eval=times, method='LSODA',\
                         args=(g,x0, w), rtol=1e-5, atol=1e-8)
```

    Runtimes for rk23:
    201 ms ± 832 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)

    Runtimes for rk45:
    50.1 ms ± 112 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

    Runtimes for lsoda:
    49.6 ms ± 225 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

**Simulating Dynamics with Stiffness**

### The Problem:
An ordinary differential equation problem is stiff if the solution being sought is varying slowly, but there are nearby solutions that vary rapidly, so the numerical method must take small steps to obtain satisfactory results.

Stiffness is an efficiency issue. If we weren't concerned with how much time a computation takes, we wouldn't be concerned about stiffness. Nonstiff methods can solve stiff problems; they just take a long time to do it.

A model of flame propagation provides an example. When you light a match, the ball of flame grows rapidly until it reaches a critical size. Then it remains at that size because the amount of oxygen being consumed by the combustion in the interior of the ball balances the amount available through the surface. The simple model is

\begin{equation*}
\frac{dy}{dt} = y^2-y^3,
\end{equation*}

with initial condition $y(0)=\delta$, evolving for times $0<t<2/\delta$.

The scalar variable  $y(t)$ represents the radius of the ball. The  $y^2$ and  $y^3$  terms come from the surface area and the volume. The critical parameter is the initial radius, $\delta$ , which is "small." We seek the solution over a length of time that is inversely proportional to $\delta$.

We will start with the $RK45$ routine. If $\delta$ is not very small, the problem is not very stiff. Let us set  $\delta=0.01$  and request a relative error of $10^{-4}$ and see what we get.

```python
import numpy as np
from scipy.integrate import solve_ivp
import matplotlib.pyplot as plt
plt.rcParams['figure.figsize'] = (8,6)

def radius(initsize, **kwargs):
    t_final = 2.0/initsize
    times = np.linspace(0.0, t_final, 1000)
    return solve_ivp(lambda t,y: y**2 * (1-y), (0, t_final), [initsize],\
                     t_eval=times, **kwargs)

d = 0.01
data = radius(d, method='RK45', rtol=1e-4)
plt.title("Size-time")
plt.xlabel('t')
plt.ylabel('y', rotation=0)
plt.plot(data.t, data.y[0], label=f'd = {d:1.1e}')
plt.vlines(1/d, 0.0, np.amax(data.y[0]),colors='black', linestyles='dashed')
plt.legend()
plt.show()
```

Now let's see stiffness in action. Decrease $\delta$ by a couple of orders of magnitude and measure the execution time. Compare the execution times of $RK45$ with the stiffness-detecting $LSODA$ method for both the stiff and non-stiff cases.

#### Suggestion

To expedite the execution of the $LSODA$ algorithm, add a routine for computing the Jacobian of the RHS. In this particular case, the Jacobian is simply

\begin{equation*}
J = \frac{d}{dy}\left(y^2-y^3\right)=2y-3y^2
\end{equation*}

```python
print("Non-Stiff Case -")
d2 = 1e-1
print("RK45:")
%timeit radius(d2, method='RK45', rtol=1e-4)
print("LSODA:")
%timeit radius(d2, method='LSODA', rtol=1e-4)
print("\nStiff Case -")
d2 = 1e-4
print("RK45:")
%timeit radius(d2, method='RK45', rtol=1e-4)
print("LSODA sans Jacobian:")
%timeit radius(d2, method="LSODA", rtol=1e-4)
print("LSODA:")
%timeit radius(d2, method='LSODA', rtol=1e-4, jac=lambda t,y: y*(2-3*y))
```

    Non-Stiff Case -
    RK45:
    100 loops, best of 5: 3.08 ms per loop
    LSODA:
    100 loops, best of 5: 3.09 ms per loop

    Stiff Case -
    RK45:
    1 loop, best of 5: 404 ms per loop
    LSODA sans Jacobian:
    100 loops, best of 5: 5.98 ms per loop
    LSODA:
    100 loops, best of 5: 5.73 ms per loop


# Under- and overfitting, model selection

## Preliminaries

In the first set of exercises you had to implement the training and evaluation of the linear regression and $k$-NN methods from scratch in order to practice your `numpy` skills. From this set of exercises onward, you can use the implementations provided in `scikit-learn` or other higher-level libraries. We start this set of exercises by demonstrating some of the features of `scikit-learn`.

For example, implementation of linear regression model fitting with an analytical solution for the parameters is provided by the class `sklearn.linar_model.LinearRegression`. You can train a linear regression model in the following way:

```python
import numpy as np
from sklearn import datasets, linear_model

# load the diabetes dataset
diabetes = datasets.load_diabetes()

# use only one feature
X = diabetes.data[:, np.newaxis, 2]
y = diabetes.target

# split the data into training/testing sets
X_train = X[:-20]
X_test = X[-20:]

# split the targets into training/testing sets
y_train = y[:-20]
y_test = y[-20:]

# create linear regression object
model = linear_model.LinearRegression()

# train the model using the training dataset
model.fit(X_train, y_train)
```

    LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None, normalize=False)

Let's visualize the training dataset and the learned regression model.

```python
%matplotlib inline
import matplotlib.pyplot as plt

fig = plt.figure()
plt.plot(X_train, y_train, 'r.', markersize=12)
X_edge = np.array([np.min(X_train, 0), np.max(X_train, 0)])
plt.plot(X_edge, model.predict(X_edge), 'b-')
plt.legend(('Data', 'Linear regression'), loc='lower right')
plt.title('Linear regression')import numpy as np
from sklearn import datasets, linear_model

# load the diabetes dataset
diabetes = datasets.load_diabetes()

# use only one feature
X = diabetes.data[:, np.newaxis, 2]
y = diabetes.target

# split the data into training/testing sets
X_train = X[:-20]
X_test = X[-20:]

# split the targets into training/testing sets
y_train = y[:-20]
y_test = y[-20:]

# create linear regression object
model = linear_model.LinearRegression()

# train the model using the training dataset
model.fit(X_train, y_train)
plt.xlabel('$x$')
plt.ylabel('$y$')
plt.show()
```

Once trained, the model can be used to make predictions on the test data:

```python
# Make predictions using the testing dataset
prediction = model.predict(X_test)
```

The next step (not shown here) is to evaluate the performance of the trained model.

Note that the `scikit-learn` interface works by first initializing an object from the class that implements the machine learning model (linear regression in this case) and then fitting the initialized model using the data in the training set. Finally, the trained (fitted) model can be used to make predictions on unseen data. In fact, all models implemented in this library follow the same *initialize-fit-predict* programming interface. For example, a $k$-NN classifier can be trained in the following way:

```python
from sklearn.model_selection import train_test_split
from sklearn import datasets, neighbors

breast_cancer = datasets.load_breast_cancer()

X = breast_cancer.data
y = breast_cancer.target

# make use of the train_test_split() utility function instead
# of manually dividing the data
X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, random_state=40)

# initialize a 3-NN classifier
model = neighbors.KNeighborsClassifier(n_neighbors=3)

# train the model using the training dataset
model.fit(X_train, y_train)

# make predictions using the testing dataset
prediction = model.predict(X_test)
```

Note that the features in the breast cancer dataset have different scales (some have on average very small absolute values, and some very large), which means that the distance metric used by $k$-NN will me dominated by the features with large values. You can use any of the number of feature transformation methods implemented in `scikit-learn` to scale the features. For example, you can use the `sklearn.preprocessing.StandardScaler` method to transform all features to a have a zero mean and unit variance:

```python
from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
scaler.fit(X_train)

X_train_scaled = scaler.transform(X_train)
```

The scaler has its own parameters which are the means and standard deviations of the features estimated from the training set. If you train a model with the scaled features, you will have to remember to also apply the scaling transformation every time you make a prediction on new unseen and unscaled data. This is somewhat prone to error. One option for making the code more robust is to create a processing pipeline that includes the scaling and $k$-NN models in a sequence:

```python
from sklearn.pipeline import Pipeline

knn = neighbors.KNeighborsClassifier(n_neighbors=3)

model = Pipeline([
                 ("scaler", scaler),
                 ("knn", knn)
                ])

# train the model using the training dataset
model.fit(X_train, y_train)

# make predictions using the testing dataset
prediction = model.predict(X_test)

```

If you are curious, more information about the design of the `scikit-learn` application programming interface (API) can be found [in this paper](https://arxiv.org/pdf/1309.0238.pdf).

## Exercises

### Bias-variance decomposition

Show that the mean squared error of the estimate of a parameter can be decomposed into an expression that includes both the bias and variance (Eq. 5.53-5.54 in "Deep learning" by Goodfellow et al.).

\begin{align}
MSE(\hat{\theta}) & = E[(\hat{\theta} - \theta)^2] \\
& = E[(\hat{\theta}-E[\hat{\theta}]+E[\hat{\theta}]-\theta)^2] \\
& = E[(\hat{\theta}-E[\hat{\theta}])^2+2(\hat{\theta}-E[\hat{\theta}])(E[\hat{\theta}]-\theta)+(E[\hat{\theta}]-\theta)^2] \\
& = E[(\hat{\theta}-E[\hat{\theta}])^2]+E[2(\hat{\theta}-E[\hat{\theta}])(E[\hat{\theta}]-\theta)]+E[(E[\hat{\theta}]-\theta)^2] \\
& = E[(\hat{\theta}-E[\hat{\theta}])^2+2(E[\hat{\theta}]-\theta)E[\hat{\theta}-E[\hat{\theta}]]+(E[\hat{\theta}]-\theta)^2 &E[\hat{\theta}]-\theta = constant \\
& = E[(\hat{\theta}-E[\hat{\theta}])^2]+2(E[\hat{\theta}]-\theta)(E[\hat{\theta}]-E[\hat{\theta}])+(E[\hat{\theta}]-\theta)^2 &E[\hat{\theta}] = constant \\
& = E[(\hat{\theta}-E[\hat{\theta}])^2] + (E[\hat{\theta}]-\theta)^2 \\
& = Var(\hat{\theta}) + Bias(\hat{\theta},\theta)^2 \\
\end{align}

### Polynomial regression

For this exercise we will be using generated data to better show the effects of the different polynomial orders.
The data is created using the make_polynomial_regression function.

```python
import numpy as np

%matplotlib inline

def generate_dataset(n=100, degree=1, noise=1, factors=None):
    # Generates a dataset by adding random noise to a randomly
    # generated polynomial function.

    x = np.random.uniform(low=-1, high=1, size=n)

    factors = np.random.uniform(0, 10, degree+1)

    y = np.zeros(x.shape)

    for idx in range(degree+1):
        y += factors[idx] * (x ** idx)

    # add noise
    y += np.random.normal(-noise, noise, n)

    return x, y

# load generated data
np.random.seed(0)

X, y = generate_dataset(n=100, degree=4, noise=1.5)

plt.plot(X, y, 'r.', markersize=12)
```

Implement polynomial regression using the `sklearn.preprocessing.PolynomialFeatures` transformation. Using the `sklearn.grid_search.GridSearchCV` class, perform a grid search of the polynomial order hyperparameter space with cross-validation and report the performance on an independent test set.

Plot a learning curve that show the validation accuracy as a function of the polynomial order.

Which models have a high bias, and which models have high variance? Motivate your answer.

Repeat this experiment, this time using the diabetes dataset instead of the generated data.

```python
import matplotlib.pyplot as plt
import numpy as np
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
import operator

#%% split data in train and test
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=40)

#%%
X_train = X_train[:,np.newaxis]
y_train = y_train[:, np.newaxis]
X_test = X_test[:,np.newaxis]
y_test = y_test[:,np.newaxis]

poly = PolynomialFeatures(degree=2)
X_poly = poly.fit_transform(X_train)
pol = LinearRegression()
pol.fit(X_poly, y_train)
y_poly_pred = pol.predict(X_poly)

plt.scatter(X_train, y_train, s=10, label='training data') #waardes data plotten
#soorteren x waardes voor plotten
sort_axis = operator.itemgetter(0)
sorted_zip = sorted(zip(X_train,y_poly_pred), key=sort_axis)
X_train2, y_poly_pred = zip(*sorted_zip)
#regressie plotten
plt.plot(X_train2, y_poly_pred, color='r', label='model 2nd degree polynomial')
plt.legend(loc = 'upper left')
plt.show()

```

```python
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import make_pipeline

def PolynomialRegression(degree=2, **kwargs):
    return make_pipeline(PolynomialFeatures(degree), LinearRegression(**kwargs))

param = {'polynomialfeatures__degree':np.arange(20)}
grid = GridSearchCV(PolynomialRegression(), param, cv = 5)  #find the optimum degree, 5-fold cross-validation
grid.fit(X_train,y_train)
values  = grid.cv_results_ #results
mean_test_score = values['mean_test_score']

polymodel = grid.best_estimator_   #the model with the best estimator
performance = polymodel.score(X_test,y_test) #performance score on the independent test set
y_predicted = polymodel.fit(X_train,y_train).predict(X_train)
sorted_zip = sorted(zip(X_train,y_predicted), key=operator.itemgetter(0))
X_train_plt, y_predicted_plt = zip(*sorted_zip)

plt.scatter(X_train,y_train, label='Train data')
plt.plot(X_train_plt,y_predicted_plt,'r-',markersize=10, label= 'Model')
plt.title('Best estimated model: 3rd degree polynomial')
plt.legend(loc = 'upper left')
plt.xlabel('x')
plt.ylabel('y')
plt.show()
```

```python
plt.plot(np.arange(20),values['mean_test_score'])
plt.title('Validation accuracy dependence on degree of polynomial')
plt.xlabel('degree of polynomial')
plt.ylabel('Validation accuracy')
plt.show()
print(polymodel)
print ("The optimum degree polynomial that fits this data is 3, with mean test score {}".format(mean_test_score[3]))
print("The accuracy on the independent test set for degree = 3 is {}".format(performance))
```

The models with a polynomial degree lower than the optimum (degree < 3) have a high bias. Low polynomial orders are insufficient to describe the data in a generalized manner. Using such few polynomials would result in underfitting.
The models with a polynomial degree highger than the optimum (degree > 3) have a high variance. High polynomial orders describe the train data very well, but are too specific such that the model cannot be applied to and predict new data, because it is overfitted. The model should contain less polynomials to be more generalized.

NOTE: In the following code polynomial orders are taken 0 to 5 and not to 20, to reduce the running time.
Other ways of reducing running time, but still trying more polynomial degrees (for example 10, which will run for approx. 1 min) are: N-fold cross validation less than 4 could or decreasing the size of the training data set. However, generally, a higher cross-validation fold is beneficial for finding the right hyperparameter (degree polynomial) and a big training dataset increases the accuracy of the model.

```python
diabetes = datasets.load_diabetes()
Xdb = diabetes.data[:]
ydb = diabetes.target[:]
Xdb_train, Xdb_test, ydb_train, ydb_test = train_test_split(Xdb, ydb,test_size=0.20, random_state=30) #test_size may be changed

def PolynomialRegression_db(degree=2, **kwargs):
    return make_pipeline(StandardScaler(), PolynomialFeatures(degree), LinearRegression(**kwargs))
poldegrees = 5 #define the polynomial degrees that are validated
param = {'polynomialfeatures__degree':np.arange(poldegrees)}
grid_db = GridSearchCV(PolynomialRegression_db(), param, cv = 4)  #find the optimum degree, n-fold cross-validation (cv)

import timeit
start = timeit.default_timer()

grid_db.fit(Xdb_train,ydb_train)

stop = timeit.default_timer()
print('Time: ', stop - start)

values_db  = grid_db.cv_results_ #results
mean_test_score_db = values_db['mean_test_score']
polymodel_db = grid_db.best_estimator_   #the model with the best estimator
performance_db = polymodel_db.score(Xdb_test,ydb_test)

plt.plot(np.arange(poldegrees),values_db['mean_test_score'])
plt.title('Validation accuracy diabetes dataset')
plt.xlabel('degree of polynomial')
plt.ylabel('Validation accuracy')
plt.show()
print(polymodel_db)
print ("The optimum degree polynomial that fits this data is 1, with mean test score {}".format(mean_test_score_db[1]))
print("The accuracy on the independent test set for degree = 1 is {}".format(performance_db))
```

### ROC curve analysis
A common method to evaluate binary classifiers is the receiver operating characteristic (ROC) curve. Similar to the week one practicals, implement a $k$-NN classifier on the breast cancer dataset, however, his time use the $k$-NN pipeline from the preliminary. Train the model for different values of $k$ and evaluate their respective performance with an ROC curve, use the `sklearn.metrics.roc_curve` function.

```python
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn import datasets, neighbors
from sklearn.pipeline import Pipeline
from sklearn import metrics
import matplotlib.pyplot as plt

breast_cancer = datasets.load_breast_cancer()

X = breast_cancer.data
y = breast_cancer.target
scaler = StandardScaler()

# make use of the train_test_split() utility function instead
# of manually dividing the data
X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, random_state=40)

#compute the mdoel for different k-values
for k in range(3,len(X_train)+1,50):      # the value of k needs always to be an odd number
    knn = neighbors.KNeighborsClassifier(n_neighbors=k)
    model = Pipeline([
                 ("scaler", scaler),
                 ("knn", knn)
                ])

    # train the model using the training dataset
    model.fit(X_train, y_train)
    prediction = model.predict(X_test)

    #compute false positive rate and true positive rate
    fpr,tpr,tresholds = metrics.roc_curve(y_test, prediction, pos_label=1)
    #calculate the area under the curve
    auc = metrics.roc_auc_score(y_test,prediction)
    #plot the calculated values
    plt.plot(fpr,tpr, label=' value of k = ' +str(k))
plt.plot()
plt.xlabel('1-Specifity')
plt.ylabel('Sensitivity')
plt.title('Receiver Operating Characteristic')
plt.legend(loc='lower right')
plt.show()
```

The ROC plots the true positive rate vs the false positive rate. The greater the area under the curve the better the classifier. From the figure above it can be concluded that the best classifier is found for the value of k is 3. A random classifier is found with the value of k is 403.

### $F_1$ score and Dice similarity coefficient

The Dice similarity coefficient is a very popular evaluation measure for image segmentation applications. Assuming that $A$ is the ground truth segmentation of an object represented as a binary image, and $B$ is the binary output of an image segmentation method, the Dice similarity coefficient is computed as:

$\text{Dice}(A,B) = \frac{2|A\cap B|}{|A| + |B|}$

where $|\cdot|$ represents the cardinality of the objects (e.g. $|A|$ is the number of non-zero pixels in the ground truth segmentation).

For example, the Dice similarity can be computed in the following way:

```python
import numpy as np
import matplotlib.pyplot as plt
# generate some test objecys
A = np.zeros((32, 32))
A[10:-10, 10:-10] = 1
B = np.zeros((32, 32))
B[5:-15, 5:-15] = 1

dice = 2*np.sum(A*B)/(np.sum(A)+np.sum(B))

# display the results
plt.plot()
plt.imshow(A)
plt.imshow(B, alpha=0.7)

print('The dice-score is = ',dice)
```

Show that the $F_1$ score, which is the harmonic mean of precision and recall, is equivalent to the Dice similarity coefficient

```python
from sklearn.metrics import f1_score
from sklearn.metrics import classification_report
A = np.zeros((32, 32))
A[10:-10, 10:-10] = 1
B = np.zeros((32, 32))
B[5:-15, 5:-15] = 1

print('The f1 score = ',f1_score(A,B,average='weighted'))
```

    The f1 score =  0.3402777777777778

A = ground_truth
B = predicted_values

Why are the dice-score and f1-score the same?
    f1         = 2 * (precision * recal) / (precision + recall)
               = 2 / ((precision/(precision * recal)) + (recal/(precision * recal))
               = 2 / (1/recal) + (1/precision)
    wherein precision = |AB|/|B| and recal = |AB|/|A|
    if you write this in the f1 formula you get:
             f1= 2 / (|A|/|AB|) + (|B|/|AB|)

    dice_score = 2 * |AB| / (|A|+|B|)
               = 2 / ((|A|/|AB|) + (|B|/|AB|))

So as you can see these two (f1 and dice-score) are the same formulas.

```python

```


参数的微小变动随着网络的深度增加而放大, BN通过改变输入的分布进而减少网络内部的协变量偏移, 有一定的正则效果

# BatchNormalization
- 减少了对参数初始化的依赖
- 能以更大的学习率学习
- 一些场合能避免使用Dropout
+ 加速网络收敛

$SGD通过优化网络参数\Theta来得到最小的Loss$
\begin{equation}
\Theta = \mathop \arg\min_{\Theta} \frac{1}{N}\sum_{i=1}^{N}l(x_i, \Theta)
\end{equation}

对于内部网络计算

$l=F_2(F_1(u,\Theta_1),\Theta_2)$

$\Theta_2的学习可等效为l=F_2(x,\Theta_2)$

$输入x=F_1(u,\Theta_1)的分布对\Theta_2的学习至关重要$

早期的sigmoid函数在输入到达S曲线的顶部时梯度为零,使网络的训练需要将参数约束到S曲线的上升区间内,造成BP的梯度消失, 而这样的结果,使得输入的大多数维度进入饱和状态,延长了网络收敛.

随后出现的应对方法是ReLu, Glorot/Xavier初始化, 使用较小的学习率,

### Reducing Internal Covariate Shift
输入白化后网络训练能更快的收敛, 线性变换至0均值,1方差, 去相关.

白化每一层的输入, 可以使输入保持一个固定的分布,来移除网络内部的协变量偏移.

如果将白化添加到BP的优化过程中, bias在中心化时会被抵消, 而梯度的不断更新会导致bias无限增大, 会导致归一化后BP的优化失效.

对此, BP中我们需计算输入x以及对应的原始样本雅可比矩阵

$\frac{\partial Norm(x,\chi)}{\partial x}和\frac{\partial Norm(x,\chi)}{\partial \chi}$

### Normalization Statistics via Mini-Batch
完全白化每一层的输入代价非常大,而且不能保证处处可微,
两步简化
> 1. 单独归一化每一层的标量特征, 即使特征未去相关,仍然能加速收敛
$$\hat{x}^{(k)} = \frac{x^(k)-E[x^{(k)}]}{\sqrt{ Var[x^{(k)}]} }$$

然而单纯的归一化可能会改变网络的表示能力, 如sigmoid函数的归一化会丧失其部分非线性变换

通过引入额外的平移变换与尺度变换$$y^{(k)}=\gamma^{(k)}\hat{x}^{(k)}+\beta^{(k)}$$

其中使$\gamma^{(k)}=\sqrt{Var[x^{(k)}}, \beta^{(k)}=E[x^{(k)}]$能还原原始的表示能力
> 2. 取每个batch_size的均值和方差来进行BN

相较于协方差的方法, 激活函数的白化数比batch_size要大,涉及到非奇异矩阵的计算, 需要添加正则项; 仅计算每个维度的方差要简单得多

添加BN到网络中可得到任意的激活,

$其中\gamma与\beta参数是可以通过学习得到的$
在归一化时需添加一个常数,避免batch_size内的样本方差为0而分母为0

$$\hat{x}_i \leftarrow \frac{x_i-\mu_B}{\sqrt{\sigma^2_B+\epsilon}}$$

### Training and Inference with Batch-Normalized Networks
激活函数的归一化能达到有效的训练, 而在推理的过程中是不必要的

在推理过程, 取所有Batch_size得到的均值和方差的平均,
$E_B[\mu_B]\rightarrow E[x]$

$\frac{m}{m-1}E_B[\sigma_B^2] \rightarrow Var[x]$

将BN层的变换公式替换成
$$y=\frac{\gamma}{\sqrt{Var[x]+\epsilon}}x+(\beta-\frac{\gamma E[x]}{\sqrt{Var[x]+\epsilon}})$$

### Batch-Normalized Convolutional Networks
对网络中的z = g(Wu+b)

u通常为上层的非线性输出,其分布在训练中可能发生改变,

Wu+b更可能拥有对称,非稀疏分布, 对其进行归一化相当是给一稳定的分布进行线性激活, 其中b可以被忽略
$z=(Wu+b)\rightarrow z=(BN(Wu))$

### Batch Normalization enables higher learning rate

- 能防止训练卡住在非线性饱和区
- 对参数的尺度有更强的适应性, 能稳固参数的增长

尺度变换不会影响到网络层的雅可比矩阵

paper/Batch Normalization Accelerating Deep Network Training by Reducing Internal Covariate Shift.ipynb


# Generative Models

So far we have explored the flexibility of deep neural networks for performing supervised learning. After many of the deep-learning methods developed in the last few years have become widespread in the industrial practice, an increasing proportion of academics are now in the process of pushing forward the frontiers of unsupervised learning.

Autoencoders are equipped with a hidden layer storing a compressed form of the data. Unlike undirected graphical models, autoencoders operate in a **feed-forward mode**. In particular, they consist of two parts: an **encoder** and a **decoder**.

The **encoder** takes high-dimensional observed data as input and compresses it into more condensed representation stored in the hidden layer. The encoder is a **recognition** or **inference network** since it infers a more concise representation of the data.

On the other hand, the **decoder** reconstructs high-dimensional data in the observation space from the hidden latent representations. Therefore, it is the **generative network**.

The compression paradigm of autoencoders is illustrated in the following figure from the outstanding [Keras Blog post on autoencoders](https://blog.keras.io/building-autoencoders-in-keras.html).

https://blog.keras.io/building-autoencoders-in-keras.html

The autoencoder part of the notebook follows [Chapter 8.4 of Deep Learning with R](https://jjallaire.github.io/deep-learning-with-r-notebooks/notebooks/8.4-generating-images-with-vaes.nb.html) and the [Keras Blog post on autoencoders](https://blog.keras.io/building-autoencoders-in-keras.html).

## Single-Layer Baseline

Defining an autoencoder in `keras` is as easy as it gets. We only need to set up two MLPs, each with one hidden layer.

```R
library(keras)
options(repr.plot.width=10, repr.plot.height=5)

input_dim <- 28 * 28
img_size <- c(28, 28)
hidden_dim <- 32

enc <- layer_dense(units = hidden_dim,
                   activation = 'relu')
dec <- layer_dense(units = input_dim,
                   activation = 'sigmoid')

auto_in <- layer_input(shape = c(input_dim))
auto_out <- auto_in %>%
    enc %>%
    dec

auto_enc <- keras_model(auto_in, auto_out)
auto_enc
auto_enc %>%
    compile(optimizer = 'adam',
            loss = 'binary_crossentropy')
```

    Model
    ________________________________________________________________________________
    Layer (type)                        Output Shape                    Param #
    ================================================================================
    input_2 (InputLayer)                (None, 784)                     0
    ________________________________________________________________________________
    dense_5 (Dense)                     (None, 32)                      25120
    ________________________________________________________________________________
    dense_6 (Dense)                     (None, 784)                     25872
    ================================================================================
    Total params: 50,992
    Trainable params: 50,992
    Non-trainable params: 0
    ________________________________________________________________________________

### Image Reconstruction

In a first step, we train the autoencoder to reconstruct digits from the MNIST dataset. For this purpose, we normalize the MNIST data.

```R
c(c(data, labels), c(test, test_labels)) %<-% dataset_mnist()
x_mnist <- array_reshape(data / 255,
                         dim = c(dim(data)[1], input_dim))
test <- array_reshape(test / 255,
                      dim = c(dim(test)[1], input_dim))
```

```R
auto_enc %>%
    fit(x_mnist,
        x_mnist,
        epochs = 10,
        batch_size = 128)
```

Now, we visualize the reconstructed images.

```R
plot_img <- function(idx, data){
    par(mfrow=c(1,2))
    plot(as.raster(array_reshape(data[idx,], img_size)))

    reconst <- auto_enc %>%
                predict(data[idx: (idx + 1),])
    plot(as.raster(array_reshape(reconst[1,], img_size)))
}
```

```R
plot_img(6,  x_mnist)
```

## Image Denoising

In the next step, we add noise to the digits to test if the autoencoder can still reconstruct the digits under more difficult circumstances.

```R
noise_scale <- .5

noise <- array_reshape(noise_scale * rnorm(length(x_mnist)),
                       dim = dim(x_mnist))
x_noisy <- pmax(pmin(x_mnist + noise, 1), 0)

noise <- array_reshape(noise_scale * rnorm(length(test)),
                       dim = dim(test))
test_noisy <- pmax(pmin(test + noise, 1), 0)
```

Let's visualize the noisy images and see how well the autoencoder can reconstruct the digits.

```R
plot_img(16, test_noisy)
```

Now, we train the autoencoder with the noisy images, therefore obtaining a **denoising autoencoders**.

```R
auto_enc %>%
    fit(x_noisy,
        x_mnist,
        epochs = 10,
        batch_size = 128)
```

Repeating the visualization after the training, we see that the quality is a bit better.

```R
plot_img(16, test_noisy)
```

## Convolutional Autoencoder

The single-layer autoencoder already reconstructs images at a decent quality. However, in order to make use of the locality of information inherent in image data, state-of-the-art autoencoders rely on convolutional architectures. For both the encoder and the decoder, we use a 3-layer structure suggested in https://blog.keras.io/building-autoencoders-in-keras.html.

```R
filter_size = 32

enc <- keras_model_sequential() %>%
        layer_conv_2d(filters = filter_size,
                      kernel_size = 3,
                      padding = 'same',
                      activation = 'relu',
                      input_shape = c(img_size, 1)) %>%
        layer_flatten() %>%
        layer_dense(units = hidden_dim, activation = 'relu')

dec <- keras_model_sequential() %>%
        layer_dense(input_dim, activation = 'sigmoid')
```

```R
auto_enc = keras_model_sequential() %>%
    enc %>%
    dec
auto_enc %>%
    compile(optimizer='adam',
            loss='binary_crossentropy')
```

We convert the MNIST-data back to the original shape and then train the convolutional autoencoder.

```R
x_noisy_resh = array_reshape(x_noisy,
                             dim = c(dim(x_noisy)[1], c(img_size, 1)))
test_noisy_resh = array_reshape(test_noisy,
                               dim = c(dim(test_noisy)[1], c(img_size, 1)))
```

```R
auto_enc %>%
    fit(x_noisy_resh,
        x_mnist,
        epochs = 20,
        batch_size = 128)
```

The visualization confirms that the convolutional reconstructs the image a bit better.

```R
plot_img_conv <- function(idx, data){
    par(mfrow=c(1,2))
    plot(as.raster(array_reshape(data[idx,,,], img_size)))

    reconst <- auto_enc %>%
            predict(array_reshape(data[idx: (idx + 1),,,],
                                  c(2, img_size, 1)))
    plot(as.raster(array_reshape(reconst[1,], img_size)))
}
```

```R
plot_img_conv(16,  test_noisy_resh)
```

## Variational Autoencoder

Autoencoders described above have the disadvantage of making only weak progress towards learning a useful latent-space representation. The reason is that only those parts of the latent space that are close to the image of a noisy digit are encouraged to produce a useful high-dimensional representation.

This becomes apparant when increasing the noise level: plugging in white noise results in a small white spot.

```R
noise <- array_reshape( pmax(pmin(rnorm(length(x_noisy_resh)), 1), 0),
                      dim = dim(x_noisy_resh))
plot_img(5, noise)
```

**Variational Autoencoders (VAEs)** introduced by [D. Kingma & M. Welling](https://openreview.net/forum?id=33X9fd2-9FyZd) avoid this problem by mapping a high-dimensional observed data point to the *parameters of a distribution* on the latent space, rather than to a single point in the latent space. A similar approach is taken by [D. J. Rezende, S. Mohamed & D. Wierstra](http://proceedings.mlr.press/v32/rezende14.html). Moreover, VAEs are trained with an objective penalizing if the distribution moves too far away from a simple prior. These changes encourage a latent representation capturing the original distribution far better when compared to deterministic autoencoders.

### Variational Inference and the Evidence Lower Bound

The guiding principle behind VAEs is **variational inference**. Suppose, we have a candidate parametric density $p(x) = p_\theta(x)$ for describing the data and we want to optimize the parameters $\theta$ via maximum likelihood. Suppose that the distribution is constructed with the help of latent variables $z$ in the sense that
$$p(x) = \int p(x|z) p(z) {\rm d}z$$
In particular, if q(z) is any probability density on the latent space, then Jensen's inequality gives that
$$
\log \int  p(z) \frac{p(x|z)}{q(z)}q(z) {\rm d}z  \ge  \int\log  \Big( p(z) \frac{p(x|z)}{q(z)}\Big)q(z) {\rm d}z
= \mathbb{E}_q[\log(p(x|z))] - KL(q|p),
$$
where $KL(q|p) = \mathbb{E}_q [\log\big(\frac{q(z)}{p(z)}\big)]$ is the **Kullback-Leibler distance** between $q$ and $p$. The right-hand side is known as **evidence lower bound (ELBO)**. In variational inference, the ELBO becomes a surrogate objective for the maximization of $\log p(x)$. As we will see, this is often feasible.

If $q(z) = p(z|x)$, then the ELBO becomes an equality and the optimization task essentially reduces to the **EM** algorithm. However, the computation of the latent distribution $p(z|x)$ is often intractable, so that we have to approximate $p(z|x)$ by more tractable parametric **approximate posterior** $q_{\phi}(z|x)$.

Typically, $q(z) = q(z|x)$ is data-dependent and we therefore call it the **approximate posterior**.

The expression $-\mathbb{E}_{q(z|x)}[\log(p(x|z))]$ is the **reconstruction loss**. It is a substitute for the reconstruction objective in the classical autoencoder. Indeed, it incentivizes latent representations assigning high likelihood to the originating observed data point $x$.

The Kullback-Leibler distance serves as a **regularization loss**. It discourages an overly complicated approximate posterior $q(z|x)$, thereby ensuring good generalization properties. This regularization contribution is missing in the classical autoencoders and therefore leads to the dissatisfactory latent representation described above.

Although the ELBO is only a surrogate objective, it still serves as a metric for assessing the quality of the model fit.

### Example: Digit Generation

Next, we discuss in detail the example from https://jjallaire.github.io/deep-learning-with-r-notebooks/notebooks/8.4-generating-images-with-vaes.nb.html in the context of generating MNIST images.

### Recognition Network

In the recognition network, we first pass the image through four conv and one dense layer to arrive at a useful condensed representation `h`.

```R
library(keras)
img_size <- c(28, 28)
latent_dim <- 2

x <- layer_input(shape = c(img_size, 1))
x_transf <- x %>%
  layer_conv_2d(filters = 32, kernel_size = 3, padding = "same",
                activation = "relu") %>%
  layer_conv_2d(filters = 64, kernel_size = 3, padding = "same",
                activation = "relu", strides = c(2, 2)) %>%
  layer_conv_2d(filters = 64, kernel_size = 3, padding = "same",
                activation = "relu") %>%
  layer_conv_2d(filters = 64, kernel_size = 3, padding = "same",
                activation = "relu")

shape_before_flattening <- k_int_shape(x_transf)
h <- x_transf %>%
  layer_flatten() %>%
  layer_dense(units = 512, activation = "relu")
```

Then, we parametrize the mean and log-variance of the encoder via single-layer MLPs.

```R
z_mean <- h %>%
  layer_dense(units = latent_dim)
z_logvar <- h %>%
  layer_dense(units = latent_dim)
```

Finally, to draw a sample from the corresponding Gaussian, we introduce a *Lambda*-layer.

```R
sample <- function(args){
    c(z_mean, z_logvar) %<-% args
    k_random_normal(shape = c(latent_dim),
                    mean = z_mean,
                    stddev = k_exp(z_logvar/2.))
}

enc_x <- layer_lambda(f = sample)(list(z_mean, z_logvar))
encoder <- keras_model(x, enc_x)
```

### Generative Network

The decoder first reshapes the latent into an image size and then applies deconvolutions.

```R
z <- layer_input(latent_dim)#k_int_shape(enc_x)[-1])
dec_z <- z %>%
  layer_dense(units = prod(as.integer(shape_before_flattening[-1])),
              activation = "relu") %>%
  layer_reshape(target_shape = shape_before_flattening[-1]) %>%
  layer_conv_2d_transpose(filters = 32,
                          kernel_size = 3,
                          padding = "same",
                          activation = "relu",
                          strides = c(2, 2)) %>%
  layer_conv_2d(filters = 1,
                kernel_size = 3,
                padding = "same",
                activation = "sigmoid")
```

Hence, the decoder becomes

```R
decoder <- keras_model(z, dec_z)
```

### Loss Function

To train the autoencoder, we need to implement the loss function derived from the ELBO. We start by noting that the reconstruction loss
$$-\log p(x| z; \theta) = \sum_{i, j \le 28} -\log p\big(x(i,j)| z; \theta\big),$$
is simply the binary crossentropy error for the problem of predicting the white pixels in the image.

It remains to deal with the Kullback-Leibler distance

$$\mathbb{E}_{\varepsilon \sim \mathcal{N}(0; I_n)}\Big[\log\Big(\frac{({\rm det}(\Sigma(x; \phi)))^{-1/2}\mathcal{N}_{}(\varepsilon;0, I_n )}{p(\mu(x; \phi) + \varepsilon \Sigma(x; \phi)))}\Big)\Big]$$

If the prior $p(z)$ is also standard Gaussian, then the above Kullback-Leibler distance becomes up to constants
$$-\frac12\sum_{k \le n}\log(\sigma_k^2(x; \phi)) - \mu_k(x; \phi)^2 - \sigma_k(x; \phi)^2$$

We combine the two losses into the ELBO.

```R
kl_weight <- 5e-4

elbo <- function(x, dec_enc_x) {
  x <- k_flatten(x)
  dec_enc_x <- k_flatten(dec_enc_x)
  xent_loss <- metric_binary_crossentropy(x, dec_enc_x)
  kl_loss <- - k_mean(
     z_logvar - k_square(z_mean) - k_exp(z_logvar),
    axis = -1L
  )
  k_mean(xent_loss + kl_weight * kl_loss)
}
```

In order to implement the loss function of VAE, we need a custom layer as in https://jjallaire.github.io/deep-learning-with-r-notebooks/notebooks/8.4-generating-images-with-vaes.nb.html.

```R
library(R6)

CustomVariationalLayer <- R6Class("CustomVariationalLayer",
  inherit = KerasLayer,
  public = list(
    call = function(inputs, mask = NULL) {
      x <- inputs[[1]]
      dec_enc_x <- inputs[[2]]
      loss <- elbo(x, dec_enc_x)
      self$add_loss(loss, inputs = inputs)
      x
    }
  )
)

layer_variational <- function(object)
  create_layer(CustomVariationalLayer, object, list())
```

This was the last missing piece for the definition of the VAE

```R
dec_enc_x <- decoder(enc_x)
dummy <- list(x, dec_enc_x) %>%
  layer_variational()

vae <- keras_model(x, dummy)
vae %>%
    compile(optimizer = "rmsprop",
            loss = NULL)
vae
```

    Model
    ________________________________________________________________________________
    Layer (type)              Output Shape      Param #  Connected to
    ================================================================================
    input_1 (InputLayer)      (None, 28, 28, 1) 0
    ________________________________________________________________________________
    conv2d_1 (Conv2D)         (None, 28, 28, 32 320      input_1[0][0]
    ________________________________________________________________________________
    conv2d_2 (Conv2D)         (None, 14, 14, 64 18496    conv2d_1[0][0]
    ________________________________________________________________________________
    conv2d_3 (Conv2D)         (None, 14, 14, 64 36928    conv2d_2[0][0]
    ________________________________________________________________________________
    conv2d_4 (Conv2D)         (None, 14, 14, 64 36928    conv2d_3[0][0]
    ________________________________________________________________________________
    flatten_1 (Flatten)       (None, 12544)     0        conv2d_4[0][0]
    ________________________________________________________________________________
    dense_1 (Dense)           (None, 512)       6423040  flatten_1[0][0]
    ________________________________________________________________________________
    dense_2 (Dense)           (None, 2)         1026     dense_1[0][0]
    ________________________________________________________________________________
    dense_3 (Dense)           (None, 2)         1026     dense_1[0][0]
    ________________________________________________________________________________
    lambda_2 (Lambda)         (None, 2)         0        dense_2[0][0]
                                                         dense_3[0][0]
    ________________________________________________________________________________
    model_3 (Model)           (None, 28, 28, 1) 56385    lambda_2[0][0]
    ________________________________________________________________________________
    r_layer_2 (RLayer)        [[None, 28, 28, 1 0        input_1[0][0]
                                                         model_3[2][0]
    ================================================================================
    Total params: 6,574,149
    Trainable params: 6,574,149
    Non-trainable params: 0
    ________________________________________________________________________________

### Training

Now, we load the MNIST data...

```R
c(c(x_train, y_train), c(x_test, y_test)) %<-% dataset_mnist()
x_train <- array_reshape(x_train / 255,
                         dim =c(dim(x_train), 1))
x_test <- array_reshape( x_test / 255,
                        dim =c(dim(x_test), 1))
```

... and start training.

```R
vae %>% fit(
  x = x_train, y = NULL,
  epochs = 10,
  batch_size = 2048,
  validation_data = list(x_test, NULL)
)
```

We now sample points from the two-dimensional latent space and convert them into digits as done in https://jjallaire.github.io/deep-learning-with-r-notebooks/notebooks/8.4-generating-images-with-vaes.nb.html.

```R
n <- 15
digit_size <- 28
batch_size <- 16
grid_x <- qnorm(seq(.05, .95, length.out = n))
grid_y <- qnorm(seq(.05, .95, length.out = n))
op <- par(mfrow = c(n, n),
          mar = c(0,0,0,0),
          bg = "black")

for (i in 1:length(grid_x)) {
  yi <- grid_x[[i]]
  for (j in 1:length(grid_y)) {
    xi <- grid_y[[j]]
    z_sample <- matrix(c(xi, yi),
                       nrow = 1, ncol = 2)
    z_sample <- t(replicate(batch_size, z_sample, simplify = "matrix"))
    x_decoded <- decoder %>%
      predict(z_sample, batch_size = batch_size)
    digit <- array_reshape(x_decoded[1,,,],
                           dim = c(digit_size, digit_size))
    plot(as.raster(digit))
  }
}
par(op)
```

```R

```

### Reparametrization Trick

In order to maximize the ELBO with respect to the parameters $\theta$ of $p(x|z; \theta)$ and $\phi$ of $q(z|x; \phi)$, we would like to resort to [stochastic gradient descent](./backprop.ipynb). Typically, $p(x|z; \theta)$ is a neural network with input $z$ and weights $\theta$, so that computing gradients with respect to $\theta$ is not an issue.

On the other hand, computing gradients with respect to $\phi$ is not entirely obvious, since the density $q$ is also used to draw samples in the ELBO. Here, the ingenious **reparametrization trick** proposed by [D. Kingma & M. Welling](https://openreview.net/forum?id=33X9fd2-9FyZd) enters the stage. We explain this trick for the example when $q$ is Gaussian.

Typically, we assume the mean $\mu(x)$ and diagonal covariance $\Sigma(x)$ of $q(z|x; \phi) \sim \mathcal{N}(z; \mu_\phi(x), \sigma_\phi(x))$ to be nonlinear functions of the data point $x$. More precisely, $(\mu(x), \sigma_\phi(x)) = g_\phi(x)$, where $g_\phi$ is a deep net with  weights $\phi$. Then, we draw samples $q(z|x; \phi)$ via $z = \mu_\phi(x) + \varepsilon \sigma_\phi(x),$ where $\varepsilon \sim \mathcal{N}(0; I_n)$ is standard normal.

This reparametrization has the advantage that the reconstruction error becomes
$$\mathbb{E}_{\varepsilon \sim \mathcal{N}(0; I_n)}[\log p(x| \mu_\phi(x) + \varepsilon \sigma_\phi(x); \theta)].$$

Now, both the weights of the inference and of the generative network appear inside the expectation, so that we can optimize the ELBO via gradient descent.

## GAN

VAE  can generate promising candidates of images with high inherent complexities, such as faces. However, the images obtained in this manner tend to be blurry

https://github.com/yzwxx/vae-celebA

This is due to the ELBO as training objective. Here, the reconstruction error is given as $-\mathbb{E}_{x \sim \mathsf{data}}\mathbb{E}_{z \sim q(\cdot|x)}[\log p(x|z)]$, which results in a massive penalty if we encounter a data point with very low probability under $p(\cdot|z)$.

A radically new concept for generative models are **Generative Adversarial Networks (GANs)** as introduced in a [landmark paper](https://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf%20(https://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf) by Ian Goodfellow et. al. in 2014. The idea as simple as it is intriguing. The generative task is set up as an adversarial competition between two neural networks: While the goal of the **generator** is to produce realistically-looking images of cats, the job of the **discriminator** is tell actual images from generated ones. Then, the generator can take false decisions of the discriminator as cues on how to produce better images.

The code in this notebook is adapted from a wonderful [AC-GAN implementation](https://github.com/rstudio/keras/blob/master/vignettes/examples/mnist_acgan.R) in the Keras example section and from [Chapter 8.5 of Deep Learning with R](https://jjallaire.github.io/deep-learning-with-r-notebooks/notebooks/8.5-introduction-to-gans.nb.html).

## Generator versus Discriminator

It's not only that the concept behind GANs can be explained in one sentence to a man in the street, also the mathematical concept is just as elegant as $E = mc^2$! It is based on the following minimax problem:
\begin{align} \min_G \max_D \mathbb{E}_{x \sim p_{\mathsf{data}}}[\log D(x)] + \mathbb{E}_{z \sim p_{\mathsf{noise}}}[\log( 1 - D(G(z)))],
\end{align}
where $G$ and $D$ denote the generator and the discriminator, respectively.

For fixed generator, the inner maximization means that the discriminator is trained to minimize the cross-entropy error for the classification problem of true vs. fake images. The outer minimization means that the generator tries to make this task as difficult as possible.

Heuristically speaking, training the generator is more difficult than training the discriminator. For instance, for a fixed generator, the inner maximization problem has a closed-form solution.

**Theorem**

For fixed $G$ the solution of the optimization problem
$$ \max_D \mathbb E_{x \sim p_{\mathsf{data}}}[\log D(x)] + \mathbb E_{z \sim p_{\mathsf{noise}}}[\log(1 - D(G(z)))]$$
is given by
$$D(x) = \frac{p_\mathsf{data}(x)}{p_\mathsf{data}(x) + p_{G}(x)},$$
where $p_{G}$ is the probability density of $G(Z)$ with $Z \sim p_{\mathsf{noise}}$. The value at the maximum is given by
$$\log(4) + \mathsf{JSD}(p_{\mathsf{data}}, p_G),$$
where $\mathsf{JSD}(p_{\mathsf{data}}, p_G)$ denotes the **Jensen-Shannon Divergence** between $p_{\mathsf{data}}$ and $p_G$, i.e.,
$$\mathsf{JSD}(p_{\mathsf{data}} = \mathsf{KL}(p_\mathsf{data}|(p_{\mathsf{data}} + p_G)/2) + \mathsf{KL}(p_G|(p_{\mathsf{data}} + p_G)/2),$$

Typically, both discriminator and generator are implemented as CNNs. Here are examples that we will use below to generate MNIST digits.

### Disriminator

```R
library(keras)
library(abind)
k_set_image_data_format('channels_first')

disc_cnn <- function(){
    cnn <- keras_model_sequential()%>%
    layer_conv_2d(
      32, c(3,3), padding = "same", strides = c(2,2),
      input_shape = c(1, 28, 28)
    ) %>%
    layer_activation_leaky_relu() %>%
    layer_dropout(0.3) %>%

    layer_conv_2d(64, c(3, 3), padding = "same") %>%
    layer_activation_leaky_relu() %>%
    layer_dropout(0.3) %>%

    layer_conv_2d(128, c(3, 3), padding = "same", strides = c(2,2)) %>%
    layer_activation_leaky_relu() %>%
    layer_dropout(0.3) %>%

    layer_conv_2d(256, c(3, 3), padding = "same") %>%
    layer_activation_leaky_relu() %>%
    layer_dropout(0.3) %>%
    layer_flatten()

    cnn
}

disc_cnn()
```

    Model
    ________________________________________________________________________________
    Layer (type)                        Output Shape                    Param #
    ================================================================================
    conv2d_1 (Conv2D)                   (None, 32, 14, 14)              320
    ________________________________________________________________________________
    leaky_re_lu_1 (LeakyReLU)           (None, 32, 14, 14)              0
    ________________________________________________________________________________
    dropout_1 (Dropout)                 (None, 32, 14, 14)              0
    ________________________________________________________________________________
    conv2d_2 (Conv2D)                   (None, 64, 14, 14)              18496
    ________________________________________________________________________________
    leaky_re_lu_2 (LeakyReLU)           (None, 64, 14, 14)              0
    ________________________________________________________________________________
    dropout_2 (Dropout)                 (None, 64, 14, 14)              0
    ________________________________________________________________________________
    conv2d_3 (Conv2D)                   (None, 128, 7, 7)               73856
    ________________________________________________________________________________
    leaky_re_lu_3 (LeakyReLU)           (None, 128, 7, 7)               0
    ________________________________________________________________________________
    dropout_3 (Dropout)                 (None, 128, 7, 7)               0
    ________________________________________________________________________________
    conv2d_4 (Conv2D)                   (None, 256, 7, 7)               295168
    ________________________________________________________________________________
    leaky_re_lu_4 (LeakyReLU)           (None, 256, 7, 7)               0
    ________________________________________________________________________________
    dropout_4 (Dropout)                 (None, 256, 7, 7)               0
    ________________________________________________________________________________
    flatten_1 (Flatten)                 (None, 12544)                   0
    ================================================================================
    Total params: 387,840
    Trainable params: 387,840
    Non-trainable params: 0
    ________________________________________________________________________________

### Generator

```R
latent_size <- 100

gen_cnn <- function(){
    cnn <- keras_model_sequential()%>%
    layer_dense(1024, input_shape = latent_size, activation = "relu") %>%
    layer_dense(128*7*7, activation = "relu") %>%
    layer_reshape(c(128, 7, 7)) %>%

    layer_upsampling_2d(size = c(2, 2)) %>%
    layer_conv_2d(
      256, c(5,5), padding = "same", activation = "relu",
      kernel_initializer = "glorot_normal"
    ) %>%

    layer_upsampling_2d(size = c(2, 2)) %>%
    layer_conv_2d(
      128, c(5,5), padding = "same", activation = "tanh",
      kernel_initializer = "glorot_normal"
    ) %>%
    layer_conv_2d(
      1, c(2,2), padding = "same", activation = "tanh",
      kernel_initializer = "glorot_normal")

    cnn
}

gen_cnn()
```

    Model
    ________________________________________________________________________________
    Layer (type)                        Output Shape                    Param #
    ================================================================================
    dense_1 (Dense)                     (None, 1024)                    103424
    ________________________________________________________________________________
    dense_2 (Dense)                     (None, 6272)                    6428800
    ________________________________________________________________________________
    reshape_1 (Reshape)                 (None, 128, 7, 7)               0
    ________________________________________________________________________________
    up_sampling2d_1 (UpSampling2D)      (None, 128, 14, 14)             0
    ________________________________________________________________________________
    conv2d_5 (Conv2D)                   (None, 256, 14, 14)             819456
    ________________________________________________________________________________
    up_sampling2d_2 (UpSampling2D)      (None, 256, 28, 28)             0
    ________________________________________________________________________________
    conv2d_6 (Conv2D)                   (None, 128, 28, 28)             819328
    ________________________________________________________________________________
    conv2d_7 (Conv2D)                   (None, 1, 28, 28)               513
    ================================================================================
    Total params: 8,171,521
    Trainable params: 8,171,521
    Non-trainable params: 0
    ________________________________________________________________________________

## Example: GAN for MNIST`

Now, we discuss a simplified version of the [GAN example](https://github.com/keras-team/keras/blob/master/examples/mnist_acgan.py) from the keras repository. In fact, in the example not only achieves to generate realistically-looking digits, but uses the resulting digits as data augmentation to improve the discriminator. This concept is known as **Auxiliary Classifier GAN (AC-GAN)** and was introduced by [Augustus Odena, Christopher Olah, Jonathon Shlens](https://arxiv.org/abs/1610.09585). Loosely speaking, the discriminator is not only trained to spot fake digits, but also to classify real ones correctly.

First, we initialize the discriminator and optimize it via cross-entropy.

```R
# Adam parameters suggested in https://arxiv.org/abs/1511.06434
adam_lr = 2e-4
adam_beta_1 = 0.5
optimizer = optimizer_adam(lr=adam_lr, beta_1=adam_beta_1)
```

```R
  image <- layer_input(shape = c(1, 28, 28))
  features <- disc_cnn()(image)

  fake <- features %>%
    layer_dense(1, activation = "sigmoid")

  aux <- features %>%
    layer_dense(10, activation = "softmax")

discriminator <- keras_model(image, list(fake, aux))
discriminator %>% compile(
  optimizer = optimizer_adam(lr = adam_lr, beta_1 = adam_beta_1),
  loss = list("binary_crossentropy", "sparse_categorical_crossentropy")
)
```

Next, we build the generator. It transforms noise into fake images.

```R
latent <- layer_input(shape = list(latent_size))

image_class <- layer_input(shape = list(1))
cls <-  image_class %>%
    layer_embedding(
      input_dim = 10, output_dim = latent_size,
      embeddings_initializer='glorot_normal'
    ) %>%
    layer_flatten()

h <- layer_multiply(list(latent, cls))
fake_image <- gen_cnn()(h)

generator <- keras_model(list(latent, image_class), fake_image)
generator
```

    Model
    ________________________________________________________________________________
    Layer (type)              Output Shape      Param #  Connected to
    ================================================================================
    input_12 (InputLayer)     (None, 1)         0
    ________________________________________________________________________________
    embedding_3 (Embedding)   (None, 1, 100)    1000     input_12[0][0]
    ________________________________________________________________________________
    input_11 (InputLayer)     (None, 100)       0
    ________________________________________________________________________________
    flatten_6 (Flatten)       (None, 100)       0        embedding_3[0][0]
    ________________________________________________________________________________
    multiply_3 (Multiply)     (None, 100)       0        input_11[0][0]
                                                         flatten_6[0][0]
    ________________________________________________________________________________
    sequential_7 (Sequential) (None, 1, 28, 28) 8171521  multiply_3[0][0]
    ================================================================================
    Total params: 8,172,521
    Trainable params: 8,172,521
    Non-trainable params: 0
    ________________________________________________________________________________

Since the generator can only learn from cues by the discriminator, its training is set up in a joint model. Again the binary cross-entropy appears as training objective.

```R

latent_comb <- layer_input(shape = list(latent_size))
image_class_comb <- layer_input(shape = list(1), dtype = "int32")

fake <- generator(list(latent_comb, image_class_comb))

# Only want to be able to train generation for the combined model
freeze_weights(discriminator)
results <- discriminator(fake)

combined <- keras_model(list(latent_comb, image_class_comb), results)
combined %>% compile(
  optimizer = optimizer_adam(lr = adam_lr, beta_1 = adam_beta_1),
  loss = list("binary_crossentropy", "sparse_categorical_crossentropy")
)
combined
```

    Model
    ________________________________________________________________________________
    Layer (type)              Output Shape      Param #  Connected to
    ================================================================================
    input_13 (InputLayer)     (None, 100)       0
    ________________________________________________________________________________
    input_14 (InputLayer)     (None, 1)         0
    ________________________________________________________________________________
    model_5 (Model)           (None, 1, 28, 28) 8172521  input_13[0][0]
                                                         input_14[0][0]
    ________________________________________________________________________________
    model_4 (Model)           [(None, 1), (None 525835   model_5[1][0]
    ================================================================================
    Total params: 8,698,356
    Trainable params: 8,172,521
    Non-trainable params: 525,835
    ________________________________________________________________________________

Now, we load the MNIST data

```R
c(c(x_train, y_train), c(x_test, y_test)) %<-% dataset_mnist()

x_train <- (x_train - 127.5)/127.5
ntrain <- dim(x_train)[1]
x_train <- array_reshape(x_train, c(ntrain, 1, 28, 28))
```

The training of the generator is the most exciting. We let the generator create fake images and then try to fool the discriminator by maintaining that all of the fake images are real ones.

```R
train_gen <- function(comb, num_classes=10, batch_size=128){
    noise <- runif(2 * batch_size * latent_size, min = -1, max = 1) %>%
      matrix(ncol = latent_size)
    sampled_labels <- sample(0:9, size = 2 * batch_size, replace = T)%>%
      matrix(ncol = 1)

    trick <- rep(1, 2 * batch_size) %>%
        matrix(ncol = 1)

    train_on_batch(
        comb,
        list(noise, sampled_labels),
        list(trick, sampled_labels)
    )
}
```

In the main loop, we alternate between training the discriminator and the generator.

```R
epochs = 100
batch_size = 100

for (idx in 1:100){
    for(j in 1:3)
         train_gen(combined)
    train_disc(discriminator, generator)
}
```

The training of the discriminator is more standard, although here the generation of training data is more involved.

```R
train_disc <- function(discriminator, generator){
    c(images, classes) %<-% create_train_data( x_train, y_train, generator)
    fake <- c(rep(1L, batch_size), rep(0L, batch_size)) %>%
        matrix(ncol = 1)
    train_on_batch(discriminator, x = images, y = list(fake, classes))
}
```

The most involved part is the creation of training samples.

```R
create_train_data <- function(x_train, y_train, generator){

    batch <- sample.int(ntrain, size = batch_size)

    #classes
    c(true_classes, fake_classes) %<-% create_train_classes(batch, y_train)

    #true images
    true_images =  x_train[batch,,,,drop = FALSE]

    #fake images
    noise <- runif(n = batch_size * latent_size, min = -1, max = 1) %>%
      matrix(nrow = batch_size, ncol = latent_size)
    fake_images <-  predict(generator, list(noise, fake_classes))

    X <- abind(true_images, fake_images, along = 1)
    aux_y <- c(true_classes, fake_classes) %>%
        matrix(ncol = 1)

    list(X, aux_y)
    }
```

For the AC GANs we also need to create class labels.

```R
 create_train_classes <- function(batch, y_train){
    #true classes
     true_classes <- y_train[batch]

    #fake classes
    fake_classes <- (sample.int(10, size = batch_size, replace = T) - 1) %>%
      matrix(ncol = 1)
    list(true_classes, fake_classes)
}
```

## Visualization

Now, we visualize the generated digits.

```R
noise <- runif(10 * latent_size, min = -1, max = 1) %>%
    matrix( ncol = latent_size)

sampled_labels <- 0:9 %>%
    matrix(ncol = 1)

generated_images <- predict(
    generator,
    list(noise, sampled_labels)
)

```

```R
pmin((generated_images[5,,,] + 1.01)/2, 1) %>%
as.raster() %>%
    plot()
```

## Challenges

As of today, GANs are the generative framework producing the most visually appealing picture. At the same time, researchers are working intensively on removing fundamental deficiencies of GANs.

### Mode Collapse

Images created by GANs frequently lack the diversity of real data. Loosely speaking, once the generator has found a way to fool the discriminator, it continues to exploit this loophole since there is little incentive to explore further options. One promising counter-measure is [Mini-batch discrimination](https://openreview.net/forum?id=rJFs4tRF). Here, the discriminator decides not on a single training example but on an entire mini-batch. If the examples in the mini-batch are all too similar, then he can confidently classify it as a fake.

### Overpowerful Discriminator

Although training GANs is a two-player game, it is not a fair competition. Indeed, the recognition task of the generator is far less involved than the creation task of the generator. Hence, it may happen that the discriminator wins the game in the early phase of the training process, so that the generator will never receive any training signals.

In order to counteract this trend, we can add more randomness and artificially add handicaps for the discriminator:

1. Apply dropout in the discriminator and introduce soft labels in discriminator training.
2. Increase the gradient flow to the generator by using Leaky ReLUs and relying on strided convolutions instead of max-pooling.

### Metrics

Despite being only a lower bound, in VAE the ELBO provides an indication on the quality of the generated images and also on when to stop the training process. In contrast, the training of GANs is a minimax game between two players of competing interests, so that it is not clear which metric to choose. A small loss for the discriminator would not yield any information on the quality of the generator and vice versa.

When trained on Imagenet, a popular metric for assessing the quality of GANs is the [Inception Score](https://openreview.net/forum?id=rJFs4tRF):
$$\exp(\mathbb{E}_{x \sim p_G}[\mathsf{KL}(p(y|x))||p(y)]),$$
where $p$ denotes the class distribution of the generated images. Loosely speaking, this metric incorporates two objectives:
1. A highly spiked conditional distribution means that a CNN is super-confident about membership to a specific class. This is an indicator for good image quality
2. A high-entropy marginalized label distribution $p(y)$ means image diversity.

### GANs in NLP

Although state-of-the-art GANs generate stunningly real images, they are not yet capable of writing high-quality text. The reason is that sampling words is inherently non-differentiable, so that we are thrown outside the domain of backpropagtion. This problem applies to VAE for the same reason, so that progress in the domain of NLP remains one of the major unsolved research challenges in generative models.

## Wasserstein GAN

Replacing Jensen-Shannon divergence by the Wasserstein distance, we arrive at the concept of **Wasserstein GANs** introduced by [Martin Arjovsky, Soumith Chintala and Léon Bottou](https://arxiv.org/abs/1701.07875). For probability measures on compact spaces, the **Wasserstein distance** metrizes weak convergence and is given by:
$$W(p,p') = \inf_{q \in \Gamma(p,p')} \int |x-y|\,  q(x,y)\,dx\, dy $$
where $\Gamma$ denotes the family of couplings between $p$ and $p'$.

On the other hand, by [Pinsker's inequality](https://en.wikipedia.org/wiki/Pinsker%27s_inequality), convergence with respect to Jensen-Shannon divergence implies convergence in total variation distance, which is far more challenging than weak convergence

For the training of the generator, this has the following interpretation. Since convergence in total-variation distance implies convergence in Wasserstein distance, we can extract more gradient signals for the generator when training with respect to the Wasserstein metric.

Albeit mathematically elegant, the definition of the Wasserstein metric in terms of optimal couplings is difficult implement. Therefore, one relies on its dual formulation as the **Kantorovich-Rubinstein distance**
$$W(p_{\mathsf{data}}, p_g) = \sup_{f}\Big( \mathbb{E}_{X \sim p_{\mathsf{data}}}[f(X)] - \mathbb{E}_{Z\sim p_{\mathsf{noise}}}[f(G(Z))]\Big),$$
where the supremum is taken over all 1-Lipschitz functions. Hence, we can think of $f$ taking the role of the discriminator.

More specifically, typically both the discriminators $f= f_\varphi$ and the generator $g = g_\theta$ are deep nets with weights $\varphi$ and $\theta$. So a naive gradient procedure would proceed as follows.

1. Perform **gradient ascent** with respect to the **discriminator parameters**:
$$\nabla_\varphi \Big(\frac1m \sum_i f_\varphi(x_i) - \sum_i f_\varphi(g_\theta(z_i))\Big),$$
where $\{x_i\}$ and $\{z_i\}$ are mini-batches of data and noises.

2. Perform **gradient descent** with respect to the **generator parameters**:
$$-\nabla_\theta \Big(\frac1m  \sum_i f_\varphi(g_\theta(z_i))\Big),$$

However, this two-step fitting mechanism ignores the Lipschitz constraints and also does not lead to satisfactory in practice. Since it is not clear how to enforce the Lipschitz constraints for a given network, one typically resorts to a surrogate class of permissible functions. Although different possibilities are discussed in literature, the simplest one is to clip every single weight between a minimal and a maximal value.

```R

```