# Table of Contents

1  Intoduction to Machine Learning: Linear and Logistic Regression

1.1  Packages

2  Linear regression: problem overview

2.0.1  Define the loss

2.0.2  Define the gradient

2.0.3  Compute the loss and gradient together

2.0.4  Perform the optimization

2.0.5  Results

2.0.6  Optional Exercises

3  Logistic Regression: problem overview

3.1  Loading data

3.1.1  Sigmoid

3.1.2  Loss function

3.1.3  Gradients

3.1.4  Initialize the weights and check matrix multiplication

3.1.5  Forward and backward propagation

3.1.6  Optimization

4  Introduction to Neural Networks

4.0.1  Initialize the parameters

4.0.2  The loop

4.0.3  Loss

4.0.4  Backpropagation

4.0.5  Update parameters

4.0.6  Put it all together

4.1  Discussion

# Intoduction to Machine Learning: Linear and Logistic Regression

Welcome to the first tutorial. We will use Python packages to build up our intuition for how machine learning with neural networks for modern high energy physics. First, to get comfortable with Python packages, we will do some basic linear regression to fit lines and curves. From this, we will change the task into a classification problem, referred to as logistic regression. This is the natural building block of neural networks.

**Instructions:**
- Use the built in vectorized functions whenever possible. This means you should not use for/while loops unless asked to.

## Packages
First, lets load in the packages that we will use.
- [numpy](https://www.numpy.org/devdocs/user/index.html) is the package for numerical computations with Python (NUMerical PYthon)
- [scipy](https://docs.scipy.org/doc/scipy/reference/) contains some of the numerical proceedures, such as minimizations
- [matplotlib](https://matplotlib.org/3.1.0/tutorials/index.html) is a package for plotting

```python
# allows for plotting within the notebook
%matplotlib inline

import numpy as np
import matplotlib.pyplot as plt
```

```python
# The commands make the plots look nice
plt.rcParams.update({'font.family': 'cmr10',
                     'font.size': 12,
                     'font.serif': 'cmr10',
                     'axes.unicode_minus': False,
                     'axes.labelsize': 12,
                     'figure.figsize': (3, 3),
                     'figure.dpi': 80,
                     'mathtext.fontset': 'cm',
                     'mathtext.rm': 'serif',
                     'xtick.direction': 'in',
                     'ytick.direction': 'in',
                     'xtick.top': True,
                     'ytick.right': True,
                     'axes.formatter.use_mathtext': True
                     })
```

# Linear regression: problem overview
Load in the data using np.load(). We want the files `tutorial_1_data/linear_training.npy` and `tutorial_1_data/linear_testing.npy`.

```python
xy_training = np.load('../data/tutorial_1_data/linear_training.npy')
xy_testing = np.load('../data/tutorial_1_data/linear_testing.npy')
```

```python
x = xy_training[:, 0]
y = xy_training[:, 1]
x_test = xy_testing[:, 0]
y_test = xy_testing[:, 1]
```

```python
plt.scatter(x, y, label='training data')
plt.scatter(x_test, y_test, label='testing data')
plt.xlabel('$x$')
plt.ylabel('$y$')
plt.legend()
plt.show()
```

This is the data that we will try to fit a line to. We will want to see how close we get to the true values of the slope and $y$ intercept that we put in by hand.

### Define the loss
In order to get a good fit, we first have to define what constitutes a good fit. For a situation where we are trying to predict a number, we often use the Mean Squared Error. This is given by

$L^{\rm{MSE}} = \frac{1}{N}\sum_i^N \big(y_i - f(x_i) \big)^2$

where $y_i$ is the *true* value in the data and $f(x_i)$ is the predicted value.

```python
def MSE(y_true, y_pred):
    return np.mean(np.square(y_true - y_pred))
```

```python
y_true = np.array([1, 2, 3])
y_pred = np.array([1.1, 1.9, 3])
print('MSE = ' + str(MSE(y_true, y_pred)))
```

    MSE = 0.006666666666666678

**Expected Output**:

    MSE
     0.006666666666666678

### Define the gradient
To find the minimum of the loss function (minimizing the error) we need to have the gradient of the loss with respect to the parameters of the model. As our data looks linear, let's use a simple form of $f(x_i) = m~x_i+b$. We write this out with the chain rule as a precursor for the deep neural networks.
- Partial derivative with respect to $m$
\begin{eqnarray}
\frac{\partial}{\partial m} L^{\rm{MSE}} &=& \frac{1}{N} \sum_i^N~-2\big(y_i - f(x_i)\big) ~\frac{\partial f(x_i)}{\partial m}\\
&=& \frac{1}{N} \sum_i^N~-2\big(y_i - f(x_i)\big) ~x_i
\end{eqnarray}
- Partial derivative with respect to $b$
\begin{eqnarray}
\frac{\partial}{\partial b} L^{\rm{MSE}} &=& \frac{1}{N} \sum_i^N~-2\big(y_i - f(x_i)\big) ~\frac{\partial f(x_i)}{\partial b}\\
&=& \frac{1}{N} \sum_i^N~-2\big(y_i - f(x_i)\big)
\end{eqnarray}

### Compute the loss and gradient together
This will be called the "forward propagation" step when we get to neural networks. We want a function that takes in the
- model parameters (m and b)
- input data
- true y values

and computes returns the loss and the gradient (for both m and b)

```python
def propagate(m, b, X, Y):
    '''
    Implement the cost function and its gradient

    Arguments:
    m -- scalar, slope of the line
    b -- scalar, bias
    X -- np.array
    Y -- true value

    Return:
    cost -- Mean squared error
    dm -- gradient of the loss with respect to m
    db -- gradient of the loss with respect to b
    '''
    ### Implement your code here ###
    y_pred = m * X + b
    cost = MSE(Y, y_pred)
    ## End code ###

    ### Find gradient
    ### Implement your code here ###
    dm = np.mean(-2 * (Y - y_pred) * X)
    db = np.mean(-2 * (Y - y_pred))
    ## End code ###

    assert(dm.dtype == float)
    assert(db.dtype == float)

    # this makes a look-up table for easy use later
    grads = {'dm': dm,
             'db': db
            }

    return grads, cost
```

```python
m, b, X, Y = 1.0, 2.5, np.array([1, 2, 3, 4]), np.array([1.2, -2.3, 3.0, 4.5])
grads, cost = propagate(m, b, X, Y)
print('dm = ' + str(grads['dm']))
print('db = ' + str(grads['db']))
print('MSE = ' + str(cost))
```

    dm = 15.7
    db = 6.8
    MSE = 15.444999999999999

**Expected Output**:

    dm
    15.7

    db
    6.8

    MSE
    15.444999999999999

### Perform the optimization
**Exercise:** Write down the optimization function. The goal is to learn $m$ and $b$ by minimizing the cost function. For a parameter $\theta$, the update rule is $\theta = \theta - \alpha~d\theta$, where $\alpha$ is the learning rate and $d\theta$ is the derivative.

```python
def optimize(m, b, X, Y, num_iterations, learning_rate, print_cost=False):
    '''
    The function finds the optimal values for m and b by running a gradient descent algorithm

    m -- scalar, slope of the line
    b -- scalar, bias
    X -- np.array
    Y -- true value
    num_iterations -- number of iterations of the optimization loop
    learning_rate -- learning rate of the gradient descent update rule
    print_cost -- True to print the loss every 100 steps

    Returns:
    params -- dictionary containing the slope m and bias b
    grads -- dictionary containing the gradients of the slope and bias with respect to the cost function
    costs -- list of all the costs computed during the optimization, this will be used to plot the learning curve.
    '''

    costs = []
    slopes = []
    biases = []

    for i in range(num_iterations):
        slopes.append(m)
        biases.append(b)
        # cost and caculate gradient
        ### Start code here ###
        grads, cost = propagate(m, b, X, Y)
        ### End code here ###

        # Retrieve derivatives from grads
        dm = grads['dm']
        db = grads['db']

        # update rule
        ### Start code here ###
        m = m - learning_rate * dm
        b = b - learning_rate * db
        ### End code here ###

        # Record the costs
        costs.append(cost)

        # Print the cost every 100 training iterations
        if print_cost and i % 100 == 0:
            print('Cost after iteration %i: %f' %(i, cost))

    params = {'m': m,
              'b': b
             }
    grads = {'dm': dm,
             'db': db
            }

    return params, grads, costs, slopes, biases
```

**Exercise:** Now use the `optimize` function to find the best fit line, using an initial guess for $m,b=0$. For now, use a learning rate of 0.01 and at least 1000 iterations of training.

```python
params, grads, costs, slopes, biases = optimize(0, 0, x, y, 1000, 1e-2, print_cost=False)
```

Let's plot the results to see if the training worked. The left panel should show the MSE loss, which should decrease with every training step. The next two panels show the value of the slope and $y$-intercept for each training step. These should be asymptoting to values around -2 and 5, respectively. If you do not see this happening, you should debug the `optimize` code above.

```python
plt.figure(figsize=(9, 3))

plt.subplot(1, 3, 1)
plt.plot(costs)
plt.xlabel('Iteration')
plt.ylabel('MSE')
plt.yscale('log')
plt.minorticks_on()

plt.subplot(1, 3, 2)
plt.plot(slopes)
plt.xlabel('Iteration')
plt.ylabel('Slope')
plt.minorticks_on()

plt.subplot(1, 3, 3)
plt.plot(biases)
plt.xlabel('Iteration')
plt.ylabel('Bias')
plt.minorticks_on()

plt.tight_layout()
```

### Results
Now plot the final predictions compared to the test values.

```python
xrange = np.linspace(0, 10)
y_pred = xrange * params['m'] + params['b']
plt.plot(xrange, y_pred)
plt.scatter(x_test, y_test, color='C1', label='testing data')
plt.text(7., 1, r'$y=$' + '{0:.3f}'.format(params['m']) + r'$*x + $' + '{0:.3f}'.format(params['b']),
         ha='center', va='top')
plt.xlabel('$x$')
plt.ylabel('$y$')
plt.legend()
plt.show()
```

We should also check the loss for the training and testing values, to make sure that they are similar.

```python
Loss_train = MSE(y_true=y, y_pred=x*params['m'] + params['b'])
Loss_test = MSE(y_true=y_test, y_pred=x_test*params['m'] + params['b'])
print('The losses are: training set={0:.3e}, test set={1:.3e}'.format(Loss_train, Loss_test))
```

    The losses are: training set=9.819e-01, test set=1.172e+00

Notice that the loss is smaller for the training set. This is because our model is trained to minimize the loss on that data. However, the losses are still similar, so it does not look like we have over-fit the data.

### Optional Exercises
1. What is the effect of the learning rate? Try varying the learning rate between $10^{-3}$ to $1$. What happens when the learning rate is large? Why?
2. How would the procedure change if there was a shape to the data? Load the files `tutorial_1_data/linear_regression_curved_training.npy` and `tutorial_1_data/linear_regression_curved_test.npy` and find a function to fit the curve.

As an extra thought experiment, this example and the curved example only have one piece of input information (namely $x$). How would we generalize the setup to more than 1 input variabeles? So now, each 'event' $x_i$ is actually a vector which has $j$ dimensions. I will now change notation and write the whole dataset as $X^i_j$, where the lower components represent the vector of information, and the upper component is the different 'events' or datapoints. We will develop this more in the next section, when we explore logistic regression.

# Logistic Regression: problem overview
In linear regression, we are trying to predict a continuous value. In the literature, it is still referred to as linear regression when the models are more complicated, involving higher powers than just linear. Logisitic regression is used when we want to classify data as belonging to one group or another. In high energy physics, this could come in the form of: was this event new physics or not? was this jet from QCD or a boosted Higgs? We will build up to working with a dataset from our parton showers to ask if the initiating parton of jet was a quark or a gluon.

Before we get to the physics cases with many inputs, we will work another simple example using a 2D dataset so that we can easily visualize how things are working. The datasets we will be working with for this example are:
- `tutorial_1_data/logistic_regression_training.npy`
- `tutorial_1_data/logistic_regression_validation.npy`
- `tutorial_1_data/logistic_regression_testing.npy`

We have specifically added in a validation set, which will be used to check if the models we make are being overfit to the training set. It is important that the *test* data is not used in optimizing model selections.

## Loading data
Load the datasets. These numpy arrays have a shape of (number of events, 3). The first two columns of the 3 are $x_1$ and $x_2$. The last columns is a number 0 or 1, which indicates if the event was a background or signal, respectively.

**Exercise:** Load the data and seperate out the data columns from the label. (Hint: use the slicing feature of numpy)

```python
training = np.load('../data/tutorial_1_data/logistic_regression_training.npy')
X_Train = training[:, :2]
Y_Train = training[:, 2].reshape(-1, 1)  # makes it a vector

validation = np.load('../data/tutorial_1_data/logistic_regression_validation.npy')
X_Val = validation[:, :2]
Y_Val = validation[:, 2].reshape(-1, 1)  # makes it a vector

testing = np.load('../data/tutorial_1_data/logistic_regression_testing.npy')
X_Test = testing[:, :2]
Y_Test = testing[:, 2].reshape(-1, 1)  # makes it a vector
```

```python
plt.scatter(X_Train[np.ravel(Y_Train==1), 0], X_Train[np.ravel(Y_Train==1), 1], color='C0', s=9, label='Sig')
plt.scatter(X_Train[np.ravel(Y_Train==0), 0], X_Train[np.ravel(Y_Train==0), 1], color='C3', s=9, label='Back')
plt.xlim(-0.05, 5.05)
plt.ylim(-0.05, 5.05)
plt.minorticks_on()
plt.xlabel('$x_1$')
plt.ylabel('$x_2$')
plt.legend(loc=(1.05, 0.45), frameon=False, fontsize=14)
plt.title('Training Data')

plt.show()
```

Our goal is now to find a model which will predict whether a point $(x_1, x_2)$ in parameter space comes from either the Signal set or the Background set. As a first approximation, the boundary between the two is approximately linear. We will use this our first model.

### Sigmoid
In order to interpret our results, we will first need to modify our model such that the output is always between 0 and 1. Such a function is usually called a sigmoid function. We will use the most common definition as
\begin{equation}
\sigma(z) \equiv \frac{1}{1+e^{-z}}
\end{equation}

**Exercise:** Program a function for the sigmoid definition. Then plot $\sigma(z)$ for $z\in [-5, 5]$. Hint, use [this](https://docs.scipy.org/doc/numpy/reference/generated/numpy.exp.html).

```python
def sigmoid(z):
    '''
    Computes the sigmoid of z

    Arguments:
    z -- A scalar or numpy array of any size

    Return:
    s -- sigmoid(z)
    '''

    ### Start code here ### (about 1 line of code)
    s = 1.0 / (1 + np.exp(-z))
    ### End code here ###

    return s
```

Test the implementation

```python
sigmoid(np.array([0, 1, 2]))
```

    array([0.5       , 0.73105858, 0.88079708])

**Expected Output:**

      sigmoid([0,1,2])
    array([0.5       , 0.73105858, 0.88079708])

```python
z = np.linspace(-5, 5, 21)
plt.plot(z, sigmoid(z))
plt.minorticks_on()
plt.xlabel('$z$')
plt.ylabel(r'$\sigma(z)$')
plt.show()
```

With this simple modification, we can now use a very similar model to what we had before. The first part of our linear model will be to take
\begin{equation}
z^i = w_1~x_1^i + w_2~x_2^i + b~,
\end{equation}
where supsript $i$ refers to the specific data point and the subscript refers to the particular dimension of the data. This should look very similar to our linear model before, where $b$ is again the bias, but now we have two "slopes" which we mark as $w_j$ instead of $m$. From now on, these will be called "weights". In building up to large dimensional data, we don't want to write out all of this multiplication by hand. Instead we will make a vector $W = (w_1, w_2, \cdots)$ and use the dot product.

In our optimization, we will learn the values of $w_1$, $w_2$, and $b$ which minimizes the loss. Note that we want these to yield **large positive** values of $z$ for events that are signal-like (lower left corner) and **large negative** values for the background-like (upper right). Then, when we feed these through the sigmoid function, they will be mapped to 0 and 1.

Feeding the results through the sigmoid is called the *activation*. The sigmoid is not the only function that can be used for this. In the next tutorial we will explore the need for other *Activation Functions*.

### Loss function
It is possible to use the mean squared error for classification, however, minimizing this does not lead to the best classification. This can be seen from:
1. The largest possible 'error' is 1, so there is no sense of being very wrong in a prediction.
2. Large positive or negative values of $z$ asymptote to 0 or 1. With this, the there is little difference in the mean squared error if the model outputs $z=10$ or $z=20$. However, from the underlying definition of $z=W\cdot X + b$, there is a big difference. This relates to how 'confident' the model is. We want the loss to reward or punish predictions which are confident and right or wrong, respectively.

To accomplish this, we use the **Binary Cross Entropy**. For a very nice lecture on how this can be derived from a probability standpoint, see [Lecture 2 of Gilles Louppe's class](https://github.com/glouppe/info8010-deep-learning).

The binary cross entropy is defined as
\begin{equation}
L^{\rm{BCE}} = -\frac{1}{N} \sum_i^N y^i \log f(\mathbf{x}^i) + \big(1-y^i\big) \log\big(1-f(\mathbf{x}^i) \big)
\end{equation}

where $y^i$ is again the true label, and $f(\mathbf{x}^i)$ is the bold and the vector denotes that $\mathbf{x}$ has multiple dimensions.

Let's examine why BCE works and addresses the problems we had with MSE. If the event is from the class $y=1$, then the first term contributes and the second term is identically 0. If the overall prediction is close to 1, the loss asymptotes to 0. So the more confident the model is, the lower the loss. Conversely, the more negative $z$ is, the closer to 0 is the prediction. When we take the $\log$ of this, the loss blows up (note the overall negative sign in the front of the sum). The story is similar for events coming from $y=0$. Now the first term is identically 0, and the second term contributes. Predictions close to 0 now result in small losses while perdictions close to 1 yield very large losses. While this is not a proof, it at least gives some intuition for why this loss function is used for binary classification problems.

**Exercise:** Implement the Binary Cross Entropy loss function.

```python
def BCE(y_true, y_pred):
    '''
    Computes the binary cross entropy

    Arguments:
    y_true -- 1 dimensional numpy array with the true labels. Should all be either 0 or 1
    y_pred -- 1 dimensional numpy array containing predictions. Should be continuous between 0 and 1

    Return:
    bce -- scalar value
    '''

    ### Start code here ### (about 1 line of code)
    bce = -np.mean((y_true * np.log(y_pred)) + ((1 - y_true) * np.log(1 - y_pred)))
    ### End code here ###

    return bce
```

Test the implementation

```python
BCE(np.array([0, 0, 1, 1]), np.array([0.01, 0.99, 0.01, 0.99]))
```

    2.307610260920796

**Expected Output:**

     BCE([0, 0, 1, 1]), [0.01, 0.99, 0.01, 0.99])
     2.307610260920796

### Gradients
We are now almost ready to make our first classiciation model. The last things needed for the optimization are the gradients. You should check the following results. I use $z$ just to help with the chain rule.

\begin{eqnarray}
\frac{\partial}{\partial z} L^{\rm{BCE}}\big(y_{\rm{true}}, y_{\rm{pred}}(z) \big)
 & = & \frac{1}{N}\sum_i^N \bigg(\frac{1-y_{\rm{true}}}{1-y_{\rm{pred}}(z)} - \frac{y_{\rm{true}}}{y_{\rm{pred}}(z)} \bigg) \frac{\partial y_{\rm{pred}}(z)}{\partial z}
\end{eqnarray}

We use $y_{\rm{pred}}\big(z(x)\big) = \sigma\big(z(x)\big)$,
\begin{eqnarray}
\frac{\partial \sigma\big(z(x)\big)}{\partial x} &=& \frac{e^{-z(x)}}{\big(1+e^{-z(x)} \big)^2} \frac{\partial z(x)}{\partial x} \\
&=& \sigma\big(z(x) \big) \Big(1-\sigma\big(z(x) \big) \Big) \frac{\partial z(x)}{\partial x}
\end{eqnarray}
where $x$ is just another dummy variable, and *not* the data.

Finally, we use $z^i = W\cdot \mathbf{x}^i + b$, giving:
\begin{equation}
\frac{\partial z(w,b)}{\partial w_j} = x_j^i
\end{equation}
and
\begin{equation}
\frac{\partial z(w,b)}{\partial b} = 1
\end{equation}

It will be beneficial to program in a function for the derivative of the sigmoid.

**Exercise:** Implement the sigmoid_derivative to compute the derivative with respect to the input.

```python
def sigmoid_derivative(z):
    '''
    Compute the gradient of the sigmoid function with respect to its input z.
    It can be useful to store this into a variable later

    Arguments:
    z -- A scalar or numpy array

    Returns:
    ds -- The computed gradient
    '''

    ### Start code here ### (around a few lines of code)
    s = sigmoid(z)
    ds = s * (1 - s)
    ### End code here ####

    return ds
```

Test the implementation

```python
x = np.array([1, 2, 3])
print ("sigmoid_derivative(x) = " + str(sigmoid_derivative(x)))
```

    sigmoid_derivative(x) = [0.19661193 0.10499359 0.04517666]

**Expected Output**:

         sigmoid_derivative([1,2,3])
         [ 0.19661193  0.10499359  0.04517666]

### Initialize the weights and check matrix multiplication
In the linear regression section, we started with the slope and $y$-intercept set to 0. We will want to do the same sort of thing for our weights and bias. We will write this as a function of the number of dimensions so that when we go to larger datasets we don't have to do it by hand

**Exercise:** Implement a function to output a vector of zeros for the weights and a single 0 for the bias. Hint: look up [numpy's zeros function](https://docs.scipy.org/doc/numpy/reference/generated/numpy.zeros.html?highlight=zeros#numpy.zeros)

```python
def initialize_zeros(dim):
    '''
    This function creates a vecor of zeros with shape (dim, 1) for W.
    It also inializes b to 0

    Argument:
    dim -- interger, the size of the w vector

    Returns:
    w -- initialized vector of shape (dim, 1)
    b -- initialized scalar for the bias
    '''

    ### Start code here ###
    w = np.zeros([dim, 1])
    b = 0.
    ### end code here

    assert(w.shape == (dim, 1))
    assert(isinstance(b, float))

    return w, b
```

Test the implementation

```python
dim = 2
w, b = initialize_zeros(dim)
print ("w = " + str(w))
print ("b = " + str(b))
```

    w = [[0.]
     [0.]]
    b = 0.0

**Expected Output**:

          w
         [[ 0.]
 [ 0.]]

          b
         0

With this setup, we now have that the shape of $W$ is (2, 1). Each data vector for $X$ is a shape (1, 2) as well. The overall shape of the data is (N, 2). We want to get out the dot product over each row, so the output should be of the shape (N, 1)

\begin{equation}
Z = X\cdot W + b
\end{equation}

In some cases (namely doing images) the input data may have yet and extra dimension in the shape (the RGB colors for each pixel). Care needs to be taken in doing the matrix multiplication correctly.

**Exercise:** Check that the matrix multiplication works as expected. Note the difference in numpy between `*` and `.dot()`.

```python
# Use *
```

```python
# Use np.dot()
```

### Forward and backward propagation
Now that the parameters have been initalized, we can do the forward and backward steps for learning the parameters.

**Exercise:** Implement a function `propagate()` that computes the cost function and gradient.

**Hints:**

Forward propagation:
* You get $X$
* Compute $A = \sigma\big(X\cdot W +b\big) = \big(a_{1}, a_{2},\cdots,a_{N}\big)$
* Calculate the cost function $L = -\frac{1}{N} \sum_{i=1}^N y_i \log a_i + \big(1-y_i\big) \log\big(1-a_i\big)$

It will be usefull to fill in the rest of the chain rule and simplify to get
\begin{equation}
\frac{\partial L}{\partial w^j} = \frac{1}{N} \sum_{i=1}^N (a_i - y_i) * x_i^j
\end{equation}
and
\begin{equation}
\frac{\partial L}{\partial b} = \frac{1}{N} \sum_{i=1}^N (a_i - y_i)~.
\end{equation}
Again, we want to not use any explicit for loops, so be careful with the shapes when multiplying arrays.

```python
def propagate(w, b, X, Y):
    '''
    Implement the cost function and its gradient for the propagation above

    Arguments:
    w -- weights, a numpy array of size (dim(x[0]), 1)
    b -- bias, a scalar
    X -- data with shape (number of datapoints, dimensionality of the data)
    Y -- true label, vector of 0 or 1

    Returns:
    cost -- BCE which is the negative log-likelihhood (cost and loss are used interchangeably)
    dw -- gradient of the loss with respect to w, needs to be the same shape as w
    db -- gradient of the loss with respect to b, needs to be the same shape as b

    Tips:
    Write the code step by step and use np.log() and np.dot().
    Print shapes if you need to get the dot products correct.
    '''
    N = X.shape[0]  # number of data points

    # Forward propagation (from X to cost)
    ### Start code here ### (around 2 lines of code)
    A = sigmoid(np.dot(X, w) + b)
    cost = -np.mean(Y * np.log(A) + (1 - Y) * np.log(1 - A))
    ### End code here ###

    # Backward propogration (to find the gradient)
    ### Start code here ### (around 2 lines of code)
    # Do not use a loop
    dw = np.dot(X.T, (A - Y)) / N
    db = np.mean(A - Y)
    ### END CODE HERE ###

    assert(dw.shape == w.shape)
    assert(db.dtype == float)
    cost = np.squeeze(cost)
    assert(cost.shape == ())

    grads = {"dw": dw,
             "db": db}

    return grads, cost
```

Test the implementation

```python
w, b, X, Y = np.array([[1.],[2.]]), 2., np.array([[1.,2.,-1.],[3.,4.,-3.2]]).T, np.array([[1,0,1]])
grads, cost = propagate(w, b, X, Y.reshape(-1, 1))
print ("dw = " + str(grads["dw"]))
print ("db = " + str(grads["db"]))
print ("cost = " + str(cost))
```

    dw = [[0.99845601]
     [2.39507239]]
    db = 0.001455578136784208
    cost = 5.801545319394553

**Expected Output**:

          ** dw **
       [[ 0.99845601]
     [ 2.39507239]]

          ** db **
         0.00145557813678

          ** cost **
         5.801545319394553

### Optimization
Now that we have initialized our parameters and can compute the cost and the gradient, we can find the minimum of the loss function using gradient descent.

**Exercise:** Write the optimization function. The goal is to learn $w$ and $b$ minimizing $L^{\rm{BCE}}$. As before, for a parameter $\theta$, the update rule is $\theta = \theta - \alpha~ d\theta$, where $\alpha$ is the learning rate.

```python
def optimize(w, b, X_train, Y_train, num_iterations, learning_rate, print_cost=False, X_val=None, Y_val=None):
    '''
    This function minimizes the BCE loss function by running gradient descent

    Arguments:
    w -- weights, a numpy array of size (dim(data), 1)
    b -- bias, a scalar
    X_train -- data of shape (number of samples, dim(data))
    Y_train -- true label of shape (number of samples, 1)
    num_iterations -- number of updates using the gradient, an integer
    learning_rate -- a float which affects the update size
    print_cost -- boolean to print the loss ever 100 steps. Default is False
    X_val -- data of shape (number of samples, dim(data)). Default is None
    Y_val -- true label of shape (number of samples, 1). Default is None

    If X_val and Y_val are not None, the loss is computed on the validation data

    Returns:
    params -- dictionary containing the weights w and bias b
    grads -- dictionary containing the gradients of the weights and bias with respect to the cost function
    costs -- list of all the costs computed during the optimization, this will be used to plot the learning curve.
    '''

    costs = {'train': [],
             'validation': []
            }

    for i in range(num_iterations):

        # cost and gradient calculation (around 1-4 lines of code)
        ### Start code here ###
        grads, cost = propagate(w, b, X_train, Y_train)
        ### end code here

        # Retrieve derivatives from grads
        dw = grads["dw"]
        db = grads["db"]

        # Calculate the cost on the validation data
        if (X_val is not None) and (Y_val is not None):
            A_val = sigmoid(np.dot(X_val, w) + b)
            cost_val = -np.mean(Y_val * np.log(A_val) + (1 - Y_val) * np.log(1 - A_val))
            costs['validation'].append(cost_val)
        # update rule (≈ 2 lines of code)
        ### START CODE HERE ###
        w = w - learning_rate * dw
        b = b - learning_rate * db
        ### END CODE HERE ###

        # Record the costs
        costs['train'].append(cost)

        # Print the cost every 100 training iterations
        if print_cost and i % 100 == 0:
            print ("Cost after iteration %i: %f" %(i, cost))

    params = {"w": w,
              "b": b}

    grads = {"dw": dw,
             "db": db}

    return params, grads, costs

```

Test the implementation

```python
params, grads, costs = optimize(w, b, X, Y.reshape(-1, 1), num_iterations= 100, learning_rate = 0.009, print_cost = False)

print ("w = " + str(params["w"]))
print ("b = " + str(params["b"]))
print ("dw = " + str(grads["dw"]))
print ("db = " + str(grads["db"]))
```

    w = [[0.19033591]
     [0.12259159]]
    b = 1.9253598300845747
    dw = [[0.67752042]
     [1.41625495]]
    db = 0.21919450454067657

**Expected Output**:

        w
       [[ 0.19033591]
 [ 0.12259159]]

        b
        1.92535983008

        dw
        [[ 0.67752042]
 [ 1.41625495]]

        db
        0.219194504541

If those worked, we are now ready to classify our example data.

**Exercise:** Train a logistic regression model on the example.

```python
# Initialize w and b
w, b = initialize_zeros(2)

# Optimize
params, grads, costs = optimize(w, b, X_Train, Y_Train,
                                num_iterations= 10000, learning_rate=5e-2,
                                print_cost=False,
                                X_val=X_Val, Y_val=Y_Val
                               )
```

```python
plt.plot(costs['train'], label='Training')
plt.plot(costs['validation'], label='Validation', ls=':')
plt.xlabel('Epoch')
plt.ylabel('Loss (BCE)')
plt.legend(loc='upper right', frameon=False)
plt.show()
```

Our training and validation losses are nearly identical, so we are not overfitting. This is expected because we are using such an easy model. Now lets plot the decision boundary found by the model. There will be a few lines of matplotlib and numpy in order to plot isocurves of constant model prediction.

```python
x_space = np.linspace(0, 5, 25)
y_space = np.linspace(0, 5, 25)
X, Y = np.meshgrid(x_space, y_space)
Z = sigmoid(X * params['w'][0] + Y * params['w'][1] + params['b'])  # the 1 - sigmoid just gets the numbering right for the color map below

plt.contourf(X, Y, 1-Z, 25, cmap='bwr')
CS = plt.contour(X, Y, Z, [0.1, 0.5, 0.9], colors='k')
plt.clabel(CS, inline=1, fontsize=12)
plt.xlim(-0.05, 5.05)
plt.ylim(-0.05, 5.05)
plt.minorticks_on()
plt.xlabel('$x_1$')
plt.ylabel('$x_2$')
plt.show()
```

Our linear model draws straight lines through the space. The nominal decision boundard is when the output it $\ge 0.5$, in which case it is more likely to be signal than background. How does this match up with out test data?

```python
plt.scatter(X_Test[Y_Test.ravel()==1, 0], X_Test[Y_Test.ravel()==1, 1], color='b', s=9, alpha=0.5)
plt.scatter(X_Test[Y_Test.ravel()==0, 0], X_Test[Y_Test.ravel()==0, 1], color='r', s=9, alpha=0.5)
CS = plt.contour(X, Y, Z, [0.1, 0.5, 0.9], colors='k')
plt.clabel(CS, inline=1, fontsize=12)
plt.xlim(-0.05, 5.05)
plt.ylim(-0.05, 5.05)
plt.minorticks_on()
plt.xlabel('$x_1$')
plt.ylabel('$x_2$')
plt.show()
```

If we look at the data with score to the left of the 0.9 line, we see that most of them are blue, likewise, most of the data to the right of the 0.1 line is red. The areas that the model is confident in do well. However, there are many misclassified points inbetween.

**Optional Exercise:**
With what we have, we can easily add in more parameters to get a better fit. For instance, we may think that there should be some curvature to the decision boundary. Using the [scikit-learn](https://scikit-learn.org/stable/index.html) package, we can add in all of the quadratic (or higher) terms. Specifically, look up the [PolynomialFeatures](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.PolynomialFeatures.html#sklearn.preprocessing.PolynomialFeatures)

```python
from sklearn.preprocessing import PolynomialFeatures
poly = PolynomialFeatures(2, include_bias=False)
```

```python
X_Quadratic_Train = poly.fit_transform(X_Train)
print(X_Quadratic_Train.shape)
```

    (1000, 5)

```python
X_Quadratic_Test = poly.transform(X_Test)
X_Quadratic_Validatoin = poly.transform(X_Val)
```

```python
# Initialize w and b
wquad, bquad = initialize_zeros(5)

# Optimize
paramsquad, gradsquad, costsquad = optimize(wquad, bquad,
                                            X_Quadratic_Train, Y_Train,
                                            num_iterations=10000,
                                            learning_rate=5e-2,
                                            print_cost=False,
                                            X_val=X_Quadratic_Validatoin,
                                            Y_val=Y_Val
                                           )
```

```python
Z = []
for x1 in x_space:
    zs = []
    for x2 in y_space:
        data = poly.transform([[x1, x2]])
        zs.append(float(sigmoid(np.dot(data, paramsquad['w']) + paramsquad['b']).flatten()[0]))
    Z.append(zs)
Z = np.array(Z)

plt.figure(figsize=(6, 3))

plt.subplot(1, 2, 1)
plt.contourf(X, Y, 1-Z, 25, cmap='bwr')
CS = plt.contour(X, Y, Z, [0.1, 0.5, 0.9], colors='k')
plt.clabel(CS, inline=1, fontsize=12)
plt.xlim(-0.0, 5.0)
plt.ylim(-0.0, 5.0)
plt.minorticks_on()
plt.xlabel('$x_1$')
plt.ylabel('$x_2$')

plt.subplot(1, 2, 2)
plt.scatter(X_Test[Y_Test.ravel()==1, 0], X_Test[Y_Test.ravel()==1, 1], color='b', s=9, alpha=0.5)
plt.scatter(X_Test[Y_Test.ravel()==0, 0], X_Test[Y_Test.ravel()==0, 1], color='r', s=9, alpha=0.5)
CS = plt.contour(X, Y, Z, [0.1, 0.5, 0.9], colors='k')
plt.clabel(CS, inline=1, fontsize=12)
plt.xlim(-0.0, 5.0)
plt.ylim(-0.0, 5.0)
plt.minorticks_on()
plt.xlabel('$x_1$')
plt.ylabel('$x_2$')

plt.tight_layout(w_pad=2)
```

```python
plt.figure(figsize=(6, 3))

ax0 = plt.subplot(1, 2, 1)
plt.plot(costs['train'], label='Training')
plt.plot(costs['validation'], label='Validation', ls=':')
plt.xlabel('Epoch')
plt.ylabel('Loss (BCE)')
plt.legend(loc='upper right', frameon=False)
plt.title('Only Linear')

plt.subplot(1, 2, 2, sharey=ax0)
plt.plot(costsquad['train'], label='Training')
plt.plot(costsquad['validation'], label='Validation', ls=':')
plt.xlabel('Epoch')
plt.ylabel('Loss (BCE)')
plt.legend(loc='upper right', frameon=False)
plt.title('All Quadratic')

plt.tight_layout(w_pad=2)
```

Note that the overall loss lower when the quadratic terms are added, and there is still no overfitting. As a side note, it also trains faster. How do we know when to stop adding in new features? Part of the revolultion in machine learning is that we let the machine choose which features it wants. This will be discussed in the next section, where we build our first neural network to fit the same data.

Also note, scikit-learn has [linear regression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html) and [logistic regression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) already programmed in.

# Introduction to Neural Networks

So far, out models have gone straight from the input to the output using matrix multiplication and an activation function. By adding an extra layer, we can use the same methodology to make a nerual network. The layer between the input and the output can be viewed as an extra vector, so an initial matrix goes from the inputs to the dimension of the new vector. Then a second matrix goes from the vector to the final answer. This layer is also called a **Hidden Layer**.

To add more notation, we will now use superscript brackets to denote the layer number. So the inputs are $X^{[0]j}_i$. The first weight vector, $W^{[1]}$ takes the input to the vector $z^{[1]}$. The explicit equations are:

\begin{eqnarray}
z^{[1]}_{i} &=& X_i \cdot W^{[1]} + b^{[1]} \\
a^{[1]}_{i} &=& \sigma\big(z^{[1]}_{i} \big) \\
z^{[2]}_{i} &=& a^{[1]}_i \cdot W^{[2]} + b^{[2]} \\
y^{\rm{pred}}_i = a^{[2]}_i &=& \sigma \big( z^{[2]} \big)
\end{eqnarray}

Now we have to minimize the loss over $W^{[1]}, W^{[2]}, b^{[1]}, b^{[2]}$, where the loss is given by
\begin{equation}
L^{BCE} = -\frac{1}{N} \sum_{i=1}^{N} \bigg( y_i \log a^{[2]}_i + \big(1-y_i \big) \log\big(1-a^{[2]}_i \big) \bigg)
\end{equation}

Working out the individual gradients gives
\begin{eqnarray}
\frac{\partial z^{[1]}_i}{\partial W^{[1]}} &=& X_i \\
\frac{\partial z^{[1]}_i}{\partial b^{[1]}} &=& 1 \\
\frac{\partial a^{[1]}_i}{\partial z^{[1]}_i} &=& a^{[1]}_i \big(1-a^{[1]}_i \big) \\
& \rm{and} & \\
\frac{\partial z^{[2]}_i}{\partial W^{[2]}} &=& a^{[1]}_i \\
\frac{\partial z^{[2]}_i}{\partial b^{[2]}} &=& 1 \\
\frac{\partial a^{[2]}_i}{\partial z^{[2]}_i} &=& a^{[2]}_i \big(1-a^{[2]}_i \big)
\end{eqnarray}

With these partial derivatives, we can use the chain rule to get
\begin{eqnarray}
\frac{\partial L}{\partial W^{[2]}} &=& \frac{\partial L}{\partial a^{[2]}_i}\frac{\partial a^{[2]}_i}{\partial z^{[2]}_i} \frac{\partial z^{[2]}_i}{\partial W^{[2]}} \\
&=& -\frac{1}{N} \sum_{i=1}^N \bigg(\frac{y_i}{a^{[2]}_i} - \frac{1-y_i}{1-a^{[2]}_i} \bigg) \bigg(a^{[2]}_i \big(1-a^{[2]}_i\big) \bigg) a^{[1]}_i \\
&=& -\frac{1}{N} \sum_{i=1}^N \big(y - a^{[2]}_i \big) a^{[1]}_i
\end{eqnarray}

and

\begin{eqnarray}
\frac{\partial L}{\partial b^{[2]}} &=&
\frac{\partial L}{\partial a^{[2]}_i}
\frac{\partial a^{[2]}_i}{\partial z^{[2]}_i}
\frac{\partial z^{[2]}_i}{\partial b^{[2]}} \\
&=&
-\frac{1}{N} \sum_{i=1}^N \bigg(\frac{y_i}{a^{[2]}_i} - \frac{1-y_i}{1-a^{[2]}_i} \bigg) \bigg(a^{[2]}_i \big(1-a^{[2]}_i\big) \bigg)  \\
&=& -\frac{1}{N} \sum_{i=1}^N \big(y - a^{[2]}_i \big)
\end{eqnarray}

Which are similar to what we had for the simple logistic regression. Now we step back to get the gradients for the $^{[1]}$ elements.

\begin{eqnarray}
\frac{\partial L}{\partial W^{[1]}} &=& \frac{\partial L}{\partial a^{[2]}_i}\frac{\partial a^{[2]}_i}{\partial z^{[2]}_i} \frac{\partial z^{[2]}_i}{\partial a^{[1]}_i}\frac{\partial a^{[1]}_i}{\partial z^{[1]}_i} \frac{\partial z^{[1]}_i}{\partial W^{[1]}} \\
%
&=& -\frac{1}{N} \sum_{i=1}^N \bigg(\frac{y_i}{a^{[2]}_i} - \frac{1-y_i}{1-a^{[2]}_i} \bigg) \bigg(a^{[2]}_i \big(1-a^{[2]}_i\big) \bigg)
\bigg(W^{[2]} \bigg)
\bigg(a^{[1]}_i \big(1-a^{[1]}_i\big) \bigg)
X_i \\
%
&=& -\frac{1}{N} \sum_{i=1}^N \bigg(y_i - a^{[2]}_i \bigg)
\bigg(W^{[2]} \bigg)
\bigg(a^{[1]}_i \big(1-a^{[1]}_i\big) \bigg)
X_i
\end{eqnarray}

Similarly,
\begin{eqnarray}
\frac{\partial L}{\partial b^{[1]}} &=&
\frac{\partial L}{\partial a^{[2]}_i}
\frac{\partial a^{[2]}_i}{\partial z^{[2]}_i}
\frac{\partial z^{[2]}_i}{\partial a^{[1]}_i}
\frac{\partial a^{[1]}_i}{\partial z^{[1]}_i}
\frac{\partial z^{[1]}_i}{\partial b^{[1]}} \\
%
&=& -\frac{1}{N} \sum_{i=1}^N \bigg(y_i - a^{[2]}_i \bigg)
\bigg(W^{[2]} \bigg)
\bigg(a^{[1]}_i \big(1-a^{[1]}_i\big) \bigg)
\end{eqnarray}

While this was more detail than probably necessary, it is seeing the chain of partial derivatives is part of what has lead to the increased performance in machine learning. Keeping track of each partial derivative and the local values for the $a$ make these relatively easy. Note that we will have to be careful with our matrix multiplications, and may need to transpose from time to time.

### Initialize the parameters
Now that we know what paramters we want to use, we need to intialize them into memory. Earlier, we set all of the initial parameters to 0, this does not work with hidden layers, so instead we will randomly initialize.
- Use `np.random.randn(a,b) * 0.01` to randomly initialize a matrix of shape (a,b)
- Our input data will just be the two components.
- The biases can still be normalized to 0.

**Exercise:** Implement the initialize_random_parameters() function.

```python
def initialize_random_parameters(n_x, n_h):
    '''
    Arguments:
    n_x -- size of the input data
    n_h -- size of the hidden layer

    Returns:
    params -- python dictionary containg the parameters:
        W1 -- weight matrix of shape (n_x, n_h)
        b1 -- bias vector of shape (1, n_h)
        W2 -- weight matrix of shape (n_h, 1)
        b2 -- bias of shape (1, 1) or a scalar
    '''
    np.random.seed(42)  # For reproducible results

    ### Start code here ###
    W1 = np.random.randn(n_x, n_h) * 0.01
    b1 = np.zeros([1, n_h])
    W2 = np.random.randn(n_h, 1) * 0.01
    b2 = np.zeros([1, 1])
    ### End code here ###

    assert (W1.shape == (n_x, n_h))
    assert (b1.shape == (1, n_h))
    assert (W2.shape == (n_h, 1))
    assert (b2.shape == (1, 1))

    parameters = {"W1": W1,
                  "b1": b1,
                  "W2": W2,
                  "b2": b2}

    return parameters
```

Check the implementation

```python
parameters = initialize_random_parameters(2, 4)
print("W1 = " + str(parameters["W1"]))
print("b1 = " + str(parameters["b1"]))
print("W2 = " + str(parameters["W2"]))
print("b2 = " + str(parameters["b2"]))
```

    W1 = [[ 0.00496714 -0.00138264  0.00647689  0.0152303 ]
     [-0.00234153 -0.00234137  0.01579213  0.00767435]]
    b1 = [[0. 0. 0. 0.]]
    W2 = [[-0.00469474]
     [ 0.0054256 ]
     [-0.00463418]
     [-0.0046573 ]]
    b2 = [[0.]]

**Expected output:**

W1 | [[ 0.00496714 -0.00138264  0.00647689  0.0152303 ][-0.00234153 -0.00234137  0.01579213  0.00767435]] |
------|------|
b1 |  [[0. 0. 0. 0.]]|
W2 | [[-0.00469474][ 0.0054256 ][-0.00463418][-0.0046573 ]]|
b2 | [[0.]]|

### The loop
**Excercise:** Implment the forward_propagation step.

**Hints:**
* Use the definitions from above
* Need to use the paramters from the output of the initialize funciton, i.e. parameter['..']
* Values need for the backpropagation are stored in 'cache'

```python
# Added this function in later, it is discussed at the end of the tutorial
def relu(z):
    return z * (z > 0)
```

```python
def forward_propagation(X, parameters):
    """
    Argument:
    X -- input data of size (n_samples, n_x)
    parameters -- python dictionary containing your parameters (output of initialization function)

    Returns:
    A2 -- The sigmoid output of the second activation
    cache -- a dictionary containing "Z1", "A1", "Z2" and "A2"
    """
    # Retrieve each parameter from the dictionary "parameters"
    ### START CODE HERE ### (≈ 4 lines of code)
    W1 = parameters['W1']
    b1 = parameters['b1']
    W2 = parameters['W2']
    b2 = parameters['b2']
    ### END CODE HERE ###

    # Implement Forward Propagation to calculate A2 (probabilities)
    ### START CODE HERE ### (≈ 4 lines of code)
    Z1 = np.dot(X, W1) + b1
    # Only uncomment one of the next three lines
    A1 = sigmoid(Z1)    # For using the sigmoid activation function
    # A1 = np.tanh(Z1)  # For using the tanh activation function
    # A1 = relu(Z1)     # For using the relu acrivation function
    Z2 = np.dot(A1, W2) + b2
    A2 = sigmoid(Z2)
    ### END CODE HERE ###

#     print(A2.shape)
#     print((X.shape[0], 1))
#     assert(A2.shape == (X.shape[0], 1))

    cache = {"Z1": Z1,
             "A1": A1,
             "Z2": Z2,
             "A2": A2}

    return A2, cache
```

Test the implementation

```python
X_assess = np.array([[0.14092422, 0.80219698],
                     [0.07455064, 0.98688694],
                     [0.77224477, 0.19871568],
                     [0.00552212, 0.81546143],
                     [0.70685734, 0.72900717]])
A2, cache = forward_propagation(X_assess, parameters)
# print('A2: ', A2)
print(np.mean(cache['Z1']) ,np.mean(cache['A1']),np.mean(cache['Z2']),np.mean(cache['A2']))
```

    0.005467349042875904 0.5013668146565611 -0.004311049213484426 0.49892223936583624

**Expected Output**

[0.005467349042875904 0.5013668146565611 -0.004311049213484426 0.49892223936583624]

### Loss
Now that we have $a^{[2]}$ for all of the examples, we can compute the loss.

**Exercise:**
Implement the BCE loss function using the with inputs of `Y, A2, parameters`.

```python
def calculate_loss(Y, A2, parameters):
    '''
    Computes the binary cross entroy loss

    Arguments:
    Y -- Truth labels vector of size (n_Samples, 1)
    A2 -- The sigmoid output of the second activation, shape (n_Samples, 1)
    parameters -- python dictionary containing W1, b1, W2, b2

    Returns:
    cost -- BCE, scalar
    '''

    N = Y.shape[0]  # n_samples

    # Compute the cross entropy
    ### Start code here ###
    loss = - np.sum(Y * np.log(A2) + (1 - Y) * np.log(1 - A2)) / N

    ### End code here ###

    loss = np.squeeze(loss)

    assert(isinstance(loss, float))

    return loss
```

Test the implementation

```python
A2, cache = forward_propagation(X_assess, parameters)
Y_assess = np.array([[0], [1], [0], [1], [0]])
print("cost = " + str(calculate_loss(Y_assess, A2, parameters)))
```

    cost = 0.6927166002914666

**Expected Output**

cost  | 0.6927166002914666

### Backpropagation
Once the cost has been calculated, we are ready to do the backprop step, using the saved `cahce` to compute the gradients.

**Exercise:** Implement the backward_propagation to compute the gradients.

```python
def backward_propagation(parameters, cache, X, Y):
    '''
    Computes the gradients

    Arguments:
    parameters -- python dictionary containing W1,b1,W2,b2
    cache -- python dictionary containing Z1, A1, Z2, A2
    X -- input data of shape (n_samples, n_x)
    Y -- true labels of shape (n_samples, 1)

    Returns:
    grads -- python dictionary containing the gradients with respect to W1,b1,W2,b2
    '''

    N = X.shape[0]  # n_samples

    # First, pull values out of dictionary parameters
    ### Start code here ###
    W1 = parameters['W1']
    W2 = parameters['W2']
    ### End code here ###

    # Get the values of A1 and A2
    ### Start code here ###
    A1 = cache['A1']
    A2 = cache['A2']
    ### End code here ###

    # Compute gradients (hint use chain rule)
    ### Start code here
    dLdZ2 = -(Y - A2) / N  # shape (n_samples, 1): A1 has shape (n_samples, n_h)
    dW2 = np.dot(A1.T, dLdZ2)       # needs to be shape (nh, 1)
    db2 = np.sum(dLdZ2)

    # partial derivative with respect to Z1
    # only uncomment one of the next three lines
    dZ1 = np.outer(dLdZ2, W2.T) * A1 * (1 - A1) # If we used the Sigmoid activation function above
    # dZ1 = np.outer(dLdZ2, W2.T) * (1 - A1**2) # If we used the tanh activation function above
    # dZ1 = np.outer(dLdZ2, W2.T) * (A1 > 0)    # If we used the relu activation finction above

    dW1 = np.dot(X.T, dZ1) # needs to be shape (n_x, n_h)
    db1 = np.sum(dZ1, axis=0, keepdims=True)
    ### END CODE HERE ###
    assert(dW2.shape == W2.shape)
    assert(dW1.shape == W1.shape)
    assert(db1.shape == parameters['b1'].shape)

    grads = {"dW1": dW1,
             "db1": db1,
             "dW2": dW2,
             "db2": db2}

    return grads
```

```python
parameters = initialize_random_parameters(2, 4)
X_assess = np.array([[0.14092422, 0.80219698],
                     [0.07455064, 0.98688694],
                     [0.77224477, 0.19871568],
                     [0.00552212, 0.81546143],
                     [0.70685734, 0.72900717]])
Y_assess = np.array([[0], [1], [0], [1], [0]])
A2, cache = forward_propagation(X_assess, parameters)
grads = backward_propagation(parameters, cache, X_assess, Y_assess)
```

### Update parameters
Now we need to update the parameters as $\theta = \theta - \alpha \frac{\partial L}{\partial\theta}$.

```python
def update_parameters(parameters, grads, learning_rate = 1.0):
    '''
    Updates the parameters using theta = theta - lr * dtheta

    Arguments:
    parameters -- python dictionary containing W1,b1,W2,b2
    grads -- python dictionary containing dW1,db1,dW2,db2
    learning_rate -- float, step size (default is 1.0)

    Outputs:
    parameters -- python dictionary containing W1,b1,W2,b2
    '''

    # Retrieve parameters
    ### Start code here ###
    W1 = parameters['W1']
    b1 = parameters['b1']
    W2 = parameters['W2']
    b2 = parameters['b2']
    ### End code here ###

    # Get gradients
    ### Start code here
    dW1 = grads['dW1']
    db1 = grads['db1']
    dW2 = grads['dW2']
    db2 = grads['db2']
    ### End code here ###

    # Update rule for each parameter
    ### START CODE HERE ### (≈ 4 lines of code)
    W1 = W1 - learning_rate * dW1
    b1 = b1 - learning_rate * db1
    W2 = W2 - learning_rate * dW2
    b2 = b2 - learning_rate * db2
    ### END CODE HERE ###

    parameters = {"W1": W1,
                  "b1": b1,
                  "W2": W2,
                  "b2": b2}

    return parameters
```

```python
parameters = update_parameters(parameters, grads)

print("W1 = " + str(parameters["W1"]))
print("b1 = " + str(parameters["b1"]))
print("W2 = " + str(parameters["W2"]))
print("b2 = " + str(parameters["b2"]))
```

    W1 = [[ 0.00514745 -0.00159102  0.00665486  0.01540916]
     [-0.00235093 -0.00233051  0.01578286  0.00766502]]
    b1 = [[ 0.0001161  -0.00013418  0.0001146   0.00011517]]
    W2 = [[-0.05435132]
     [-0.0439871 ]
     [-0.05431245]
     [-0.054688  ]]
    b2 = [[-0.09892224]]

**Expected output**

### Put it all together
**Exercise:**
Build a neural network model.

```python
def nn_model(X, Y, n_h, num_iterations=10000, validaion_data=None, learning_rate=1e-2, print_loss=False):
    """
    Arguments:
    X -- dataset of shape (number of examples, dim)
    Y -- labels of shape (number of examples, 1)
    n_h -- size of the hidden layer
    num_iterations -- Number of iterations in gradient descent loop

    Returns:
    parameters -- parameters learned by the model. They can then be used to predict.
    """

    losses = {'train': [],
              'validation': []
             }
    # initialize parameters
    n_x = X.shape[1]
    parameters = initialize_random_parameters(n_x, n_h)
    print(parameters)

    for i in range(num_iterations):
        # Forward propagation
        A2, cache = forward_propagation(X, parameters)

        # Compute loss
        loss = calculate_loss(Y, A2, parameters)
        losses['train'].append(loss)

        if validaion_data is not None:
            x_val = validaion_data[0]
            y_val = validaion_data[1]
            A2_val, _ = forward_propagation(x_val, parameters)
            loss_val = calculate_loss(y_val, A2_val, parameters)
            losses['validation'].append(loss_val)

        # Back propagation
        grads = backward_propagation(parameters, cache, X, Y)

        # Update parameters
        parameters = update_parameters(parameters, grads, learning_rate=learning_rate)

        if (print_loss) and (i % 1000 == 0):
            print('Loss after {0:04d} of {1:05d} iterations: {2:0.3e}'.format(i,
                                                                              num_iterations,
                                                                              loss)
                 )

    return parameters, losses
```

```python
nn_params, nn_losses = nn_model(X_Train, Y_Train,
                                n_h=4,
                                learning_rate=1e-2,
                                num_iterations=100000,
                                print_loss=True,
                                validaion_data=[X_Val, Y_Val]
                               )
print(nn_params)
```

    {'W1': array([[ 0.00496714, -0.00138264,  0.00647689,  0.0152303 ],
           [-0.00234153, -0.00234137,  0.01579213,  0.00767435]]), 'b1': array([[0., 0., 0., 0.]]), 'W2': array([[-0.00469474],
           [ 0.0054256 ],
           [-0.00463418],
           [-0.0046573 ]]), 'b2': array([[0.]])}
    Loss after 0000 of 100000 iterations: 6.933e-01
    Loss after 1000 of 100000 iterations: 6.882e-01
    Loss after 2000 of 100000 iterations: 6.790e-01
    Loss after 3000 of 100000 iterations: 6.521e-01
    Loss after 4000 of 100000 iterations: 6.145e-01
    Loss after 5000 of 100000 iterations: 5.770e-01
    Loss after 6000 of 100000 iterations: 5.407e-01
    Loss after 7000 of 100000 iterations: 5.056e-01
    Loss after 8000 of 100000 iterations: 4.728e-01
    Loss after 9000 of 100000 iterations: 4.434e-01
    Loss after 10000 of 100000 iterations: 4.178e-01
    Loss after 11000 of 100000 iterations: 3.962e-01
    Loss after 12000 of 100000 iterations: 3.781e-01
    Loss after 13000 of 100000 iterations: 3.632e-01
    Loss after 14000 of 100000 iterations: 3.509e-01
    Loss after 15000 of 100000 iterations: 3.408e-01
    Loss after 16000 of 100000 iterations: 3.326e-01
    Loss after 17000 of 100000 iterations: 3.258e-01
    Loss after 18000 of 100000 iterations: 3.202e-01
    Loss after 19000 of 100000 iterations: 3.155e-01
    Loss after 20000 of 100000 iterations: 3.117e-01
    Loss after 21000 of 100000 iterations: 3.085e-01
    Loss after 22000 of 100000 iterations: 3.058e-01
    Loss after 23000 of 100000 iterations: 3.036e-01
    Loss after 24000 of 100000 iterations: 3.017e-01
    Loss after 25000 of 100000 iterations: 3.001e-01
    Loss after 26000 of 100000 iterations: 2.987e-01
    Loss after 27000 of 100000 iterations: 2.976e-01
    Loss after 28000 of 100000 iterations: 2.966e-01
    Loss after 29000 of 100000 iterations: 2.958e-01
    Loss after 30000 of 100000 iterations: 2.951e-01
    Loss after 31000 of 100000 iterations: 2.945e-01
    Loss after 32000 of 100000 iterations: 2.940e-01
    Loss after 33000 of 100000 iterations: 2.935e-01
    Loss after 34000 of 100000 iterations: 2.932e-01
    Loss after 35000 of 100000 iterations: 2.928e-01
    Loss after 36000 of 100000 iterations: 2.925e-01
    Loss after 37000 of 100000 iterations: 2.923e-01
    Loss after 38000 of 100000 iterations: 2.921e-01
    Loss after 39000 of 100000 iterations: 2.919e-01
    Loss after 40000 of 100000 iterations: 2.917e-01
    Loss after 41000 of 100000 iterations: 2.915e-01
    Loss after 42000 of 100000 iterations: 2.914e-01
    Loss after 43000 of 100000 iterations: 2.913e-01
    Loss after 44000 of 100000 iterations: 2.912e-01
    Loss after 45000 of 100000 iterations: 2.911e-01
    Loss after 46000 of 100000 iterations: 2.910e-01
    Loss after 47000 of 100000 iterations: 2.909e-01
    Loss after 48000 of 100000 iterations: 2.908e-01
    Loss after 49000 of 100000 iterations: 2.908e-01
    Loss after 50000 of 100000 iterations: 2.907e-01
    Loss after 51000 of 100000 iterations: 2.907e-01
    Loss after 52000 of 100000 iterations: 2.906e-01
    Loss after 53000 of 100000 iterations: 2.906e-01
    Loss after 54000 of 100000 iterations: 2.905e-01
    Loss after 55000 of 100000 iterations: 2.905e-01
    Loss after 56000 of 100000 iterations: 2.904e-01
    Loss after 57000 of 100000 iterations: 2.904e-01
    Loss after 58000 of 100000 iterations: 2.903e-01
    Loss after 59000 of 100000 iterations: 2.903e-01
    Loss after 60000 of 100000 iterations: 2.903e-01
    Loss after 61000 of 100000 iterations: 2.902e-01
    Loss after 62000 of 100000 iterations: 2.902e-01
    Loss after 63000 of 100000 iterations: 2.902e-01
    Loss after 64000 of 100000 iterations: 2.901e-01
    Loss after 65000 of 100000 iterations: 2.901e-01
    Loss after 66000 of 100000 iterations: 2.901e-01
    Loss after 67000 of 100000 iterations: 2.900e-01
    Loss after 68000 of 100000 iterations: 2.900e-01
    Loss after 69000 of 100000 iterations: 2.900e-01
    Loss after 70000 of 100000 iterations: 2.899e-01
    Loss after 71000 of 100000 iterations: 2.899e-01
    Loss after 72000 of 100000 iterations: 2.899e-01
    Loss after 73000 of 100000 iterations: 2.899e-01
    Loss after 74000 of 100000 iterations: 2.898e-01
    Loss after 75000 of 100000 iterations: 2.898e-01
    Loss after 76000 of 100000 iterations: 2.898e-01
    Loss after 77000 of 100000 iterations: 2.898e-01
    Loss after 78000 of 100000 iterations: 2.897e-01
    Loss after 79000 of 100000 iterations: 2.897e-01
    Loss after 80000 of 100000 iterations: 2.897e-01
    Loss after 81000 of 100000 iterations: 2.897e-01
    Loss after 82000 of 100000 iterations: 2.896e-01
    Loss after 83000 of 100000 iterations: 2.896e-01
    Loss after 84000 of 100000 iterations: 2.896e-01
    Loss after 85000 of 100000 iterations: 2.896e-01
    Loss after 86000 of 100000 iterations: 2.895e-01
    Loss after 87000 of 100000 iterations: 2.895e-01
    Loss after 88000 of 100000 iterations: 2.895e-01
    Loss after 89000 of 100000 iterations: 2.895e-01
    Loss after 90000 of 100000 iterations: 2.894e-01
    Loss after 91000 of 100000 iterations: 2.894e-01
    Loss after 92000 of 100000 iterations: 2.894e-01
    Loss after 93000 of 100000 iterations: 2.894e-01
    Loss after 94000 of 100000 iterations: 2.893e-01
    Loss after 95000 of 100000 iterations: 2.893e-01
    Loss after 96000 of 100000 iterations: 2.893e-01
    Loss after 97000 of 100000 iterations: 2.893e-01
    Loss after 98000 of 100000 iterations: 2.892e-01
    Loss after 99000 of 100000 iterations: 2.892e-01
    {'W1': array([[-0.68801829, -0.50574656, -0.7030154 , -0.70230872],
           [-0.62793437, -0.69805592, -0.69667309, -0.69595966]]), 'b1': array([[2.76133585, 3.24890045, 2.10799076, 2.10373343]]), 'W2': array([[3.85823035],
           [4.25435784],
           [3.14884333],
           [3.14402303]]), 'b2': array([[-4.98564856]])}

```python
plt.plot(nn_losses['train'])
plt.plot(nn_losses['validation'], ls=':')
```

```python
plt.figure(figsize=(9, 3))

ax0 = plt.subplot(1, 3, 1)
plt.plot(costs['train'], label='Training')
plt.plot(costs['validation'], label='Validation', ls=':')
plt.xlabel('Epoch')
plt.ylabel('Loss (BCE)')
plt.legend(loc='upper right', frameon=False)
plt.title('Only Linear')

plt.subplot(1, 3, 2, sharey=ax0)
plt.plot(costsquad['train'], label='Training')
plt.plot(costsquad['validation'], label='Validation', ls=':')
plt.xlabel('Epoch')
plt.ylabel('Loss (BCE)')
plt.legend(loc='upper right', frameon=False)
plt.title('All Quadratic')

plt.subplot(1, 3, 3, sharey=ax0)
plt.plot(nn_losses['train'], label='Training')
plt.plot(nn_losses['validation'], label='Validation', ls=':')
plt.xlabel('Epoch')
plt.ylabel('Loss (BCE)')
plt.legend(loc='upper right', frameon=False)
plt.title('NN (on linear input)')

plt.tight_layout(w_pad=2)
```

```python
x_space = np.linspace(0, 5, 25)
y_space = np.linspace(0, 5, 25)
X, Y = np.meshgrid(x_space, y_space)
Z_nn = []

for x in x_space:
    z_nn = []
    for y in y_space:
#         print(np.array([[x, y]]).shape)
        A2, _ = forward_propagation(np.array([[x, y]]), nn_params)
#         print(A2)
        z_nn.append(A2.squeeze())
#         break
    Z_nn.append(z_nn)
Z_nn = np.array(Z_nn)

plt.figure(figsize=(6, 3))

plt.subplot(1, 2, 1)
plt.contourf(X, Y, 1-Z_nn, 25, cmap='bwr')
CS = plt.contour(X, Y, Z_nn, [0.1, 0.5, 0.9], colors='k')
plt.clabel(CS, inline=1, fontsize=12)
plt.xlim(-0.05, 5.05)
plt.ylim(-0.05, 5.05)
plt.minorticks_on()
plt.xlabel('$x_1$')
plt.ylabel('$x_2$')

plt.subplot(1, 2, 2)
plt.scatter(X_Test[Y_Test.ravel()==1, 0], X_Test[Y_Test.ravel()==1, 1], color='b', s=9, alpha=0.5)
plt.scatter(X_Test[Y_Test.ravel()==0, 0], X_Test[Y_Test.ravel()==0, 1], color='r', s=9, alpha=0.5)
CS = plt.contour(X, Y, Z_nn, [0.1, 0.5, 0.9,], colors='k')
plt.clabel(CS, inline=1, fontsize=12)
plt.xlim(-0.0, 5.0)
plt.ylim(-0.0, 5.0)
plt.minorticks_on()
plt.xlabel('$x_1$')
plt.ylabel('$x_2$')

plt.tight_layout(w_pad=2)

```

## Discussion
What happened here? We went through all of that effort and it seems like the neural network is not doing any better than the original logistic regression. It turns out that the choice of the activation function makes a big difference in how well the network performs. We still want to use the sigmoid on the last step to map the answers between 0 and 1. However, for the hidden layer, we can do better if we use a different function. The most common activation function for the middle layers are

Name | Equation | Derivative
--- | --- | ---
REctified Linear Unit (ReLU) | $a(z) = \left\{\begin{matrix}0 & z < 0\\ z & z >=0\end{matrix}\right.$ | $\frac{\partial a}{\partial z} = \left\{\begin{matrix}0 & z < 0\\ 1 & z >=0\end{matrix}\right.$
tanh | $a(z) = \tanh(z)$ | $\frac{\partial a}{\partial z} = 1-a^2$ |

Both of these are simple to implement.

**Exercise:**
Implement one of the new activation functions.
- Define the function (or use np.tanh()) above the `propagate` function cell. Then use this in the a1 = func(z1), but not for the a2 = func(z2).
- Add the derivative term to the dZ1 term in the `backward_propagation` cell.
- These should be the only changes needed, rerun using the same parameters. You should get out a decision boundary that looks more similar to a curve (but is an interpolation of straight lines).

**Extra Challenge:**

The real power of deep learning comes from making the networks deeper, rather than adding more units to the hidden layer. From the code we have here, generalize to network with 2 hidden layers, with the final output now being a3 instead of a2.

**Even more challenging:**

Adjust the code so that you only need to pick *how many layers* and it will add in the appropriate forward and backward propagation steps.

```python

```


# Reconstruction of DOSY NMR signals

**Authors**:

* Alexandre Zouaoui (alexandre.zouaoui@ens-paris-saclay.fr)
* Charles Sejourne (charles.sejourne@ens-paris-saclay.fr)

## Part 1

### Imports

```python
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import scipy
```

### Style

```python
sns.set_style("darkgrid")
plt.rcParams["font.size"] = 14
```

### Generation of synthetic data

#### Question 1

Download the diffusion signal $\overline{x} \in \mathbb{R}^N$ where $N=200$.

---

```python
# Load data from text file
x = np.loadtxt("x.txt")
N = len(x)
print(f"The data has N={N} entries")
```

    The data has N=200 entries

#### Question 2

Create $T$ using an exponential sampling strategy:
\begin{align}
\forall n \in \{1, \ldots, N\}, \quad T^{(n)} = T_{\min} \exp \big(- (n - 1) \frac{\log(T_\min / T_\max)}{N - 1} \big)
\end{align}
where $T_\min = 1$ and $T_\max = 1000$.

---

```python
T_min = 1
T_max = 1000

# Create T using an exponential sampling strategy
T = np.array([T_min * np.exp(-(n - 1) * (np.log(T_min / T_max)) / (N - 1))
              for n in range(1, N + 1)])
```

#### Question 3

Display the original signal $\overline{x}$ as a function of $T$ (log scale on the horizontal axis).

---

```python
# Display the original signal x as a function of T using log scale horizontal axis
plt.figure(figsize=(12, 4))
plt.title("Ground truth synthetic signal")
plt.plot(T, x)
plt.xlabel("Sampled grid")
plt.ylabel("NMR diffusion coefficient")
plt.xscale("log");
```

#### Question 4

Create $t$ using a regular sampling strategy:
$$\forall m \in \{1, \ldots, M \}), \quad t^{(m)} = t_\min + \frac{m - 1}{M - 1} (t_\max - t_\min)$$
where $M=50$, $t_\min=0$ and $t_\max=1.5$.

---

```python
M = 50
t_min = 0
t_max = 1.5

# Create t using a regular sampling strategy
t = np.array([t_min + (m - 1) / (M - 1) * (t_max - t_min)
              for m in range(1, M + 1)])
```

#### Question 5

Construct matrix $K$ using:
$$\forall m \in \{1, \ldots, M \}, \; \forall n \in \{1, \ldots, N \}, \quad K^{(m, n)} = \exp(-T^{(n)}t^{(m)})$$

---

```python
# Create the matrix K of size M x N
K = np.exp(- np.outer(t, T))
```

#### Question 6

Simulate the noisy data according to $y = K \overline{x} + w$ by taking $w \sim \mathcal{N}(0, \sigma^2 I_M)$ with $\sigma = 0.01 z^{(1)}$ where $z = K\overline{x}$

---

```python
z = K.dot(x)
σ = 0.01 * z[0]

# Instanciate the noise
np.random.seed(42)
w = σ * np.random.randn(M)

# Simulate the noisy data
y = z + w
```

#### Question 7

Display the resulting noisy data $y$ as a function of $t$.

---

```python
# Display the resulting noisy data y as a function of t
plt.figure(figsize=(12, 4))
plt.title("Noisy data measurements")
plt.plot(t, y)
plt.xlabel("Pulse characteristic")
plt.ylabel("Acquisition signal");
```

### Comparison of regularization strategies

For every penalization functions $g$ listed at the end of the document,
1. Discuss the existence and uniqueness of a solution to
$$ \hat{x} = \text{arg}\min_{x \in \mathbb{R}^N} \frac{1}{2} \parallel Kx - y \parallel^2 + \beta g(x)$$
2. Propose an approach to solve the problem above.
3. Implement the approach and display the restored signal $\hat{x}$
4. Compute the normalized quadratic error between $\hat{x}$ and $\overline{x}$:
$$ E(\hat{x}, \overline{x}) = \frac{\parallel \hat{x} - \overline{x} \parallel^2}{\parallel \overline{x} \parallel^2}$$
5. When needed, search manually for the best choice for parameter $\beta$ in terms of reconstruction error.

##### Normalized quadratic error implementation

```python
def quadratic_error(x_hat, x):
    """
    Compute the normalized quadratic error between the restored signal
    and the ground truth signal

    Parameters
    ------------
    - x_hat : numpy.array
        Restored signal

    - x : numpy.array
        Ground truth signal

    Returns
    ------------
    - error : float
        Normalized quadratic error
    """

    error = np.linalg.norm(x_hat - x, ord=2)**2 / np.linalg.norm(x, ord=2)**2
    return error
```

We will study three different priors

#### Smoothness prior

$$\forall x \in \mathbb{R}^N, \quad g(x) = \frac{1}{2} \parallel Dx \parallel^2$$
where $D \in \mathbb{R}^{N \times N}$ is the discrete gradient operator defined such that:
$$\forall n \in \{1, \ldots, N \}, [Dx]^{(n)} = x^{(n)} - x^{(n-1)}$$
with the circular convention that $x^{(0)} = x^{(N)}$. The role of such regularization term is to promote the reconstruction of smooth signals.

---

#### Problem solution and approach

1. First we notice that the function $f: x \mapsto \frac{1}{2} \parallel Kx - y \parallel^2 + \frac{\beta}{2} \parallel Dx \parallel^2$ is a **convex** function by deriving its Hessian matrix $\nabla^2_f: x \mapsto K^T K + D^T D$. From here it appears that $\nabla^2_f$ is a **positive semi definite** matrix since we have: $\forall x \in \mathbb{R}^N, \; x^T \,\nabla^2_f(x)\, x = \parallel Kx \parallel^2 + \beta \parallel Dx \parallel^2 \geq 0$.

In addition, we can check that the hessian matrix is invertible. We have that $\text{Ker} (K^T K + \beta D^T D) = \text{Ker} K \cap \text{Ker} D$, since given $x \in \text{Ker} (K^T K + \beta D^T D)$,

\begin{align}
(K^T K + \beta D^T D) x = 0 & \iff x^T (K^T K + \beta D^T D) x = 0 \\
& \iff K x = 0 \quad \text{and} \quad D x = 0\\
& \iff x \in \text{Ker} K \cap \text{Ker} D\\
\end{align}

By definition of $D$, we have $\forall x \in \text{Ker} D, Dx = 0 \iff x = \alpha 1_N$ where $1_N$ is a vector of dimension $N$ whose entries are only 1s. Since $K$ is a matrix with only strictly positive coefficients by definition, we get: $\forall x \in \text{Ker} D, Kx > 0$, therefore $\text{Ker}(K) \cap \text{Ker}(D) = \{0_{\mathbb{R}^N}\}$ and the Hessian matrix is **invertible**.

Using $\text{Ker}(K) \cap \text{Ker}(D) = \{0_{\mathbb{R}^N}\}$ we now get that the hessian matrix $\nabla^2_f$ is in fact a **positive definite** matrix, i.e.: $\forall x \in \mathbb{R}^N\backslash\{0_{\mathbb{R}^N}\}, \; x^T\,\nabla^2_f(x)\, x = \parallel Kx \parallel^2 + \beta \parallel Dx \parallel^2 > 0$. As a result, $f$ is a **strictly convex** function and it admits the optimization problem admits a unique solution.

2. As a result, we can directly compute the optimization problem **closed form** solution by solving the linear system induced by setting the gradient to 0, as follow:

$$\nabla_f(\hat{x}) = 0 \iff (K^T K + \beta D^T D) \hat{x} = K^T y \iff \boxed{\hat{x} = (K^T K + \beta D^T D)^{-1} K^T y}$$

```python
# Define D the discrete gradient operator
D = np.eye(N)
for i in range(N - 1):
    D[i+1, i] = -1
D[0, -1] = -1

# Search for the best parameter β
best_error = np.inf
best_x_hat = None
best_β = None

β_range = np.linspace(1e-2, 1, num=50, endpoint=True)

for β in β_range:
    # Closed form solution
    x_hat = np.linalg.solve(K.T.dot(K) + β * D.T.dot(D), K.T.dot(y))
    # Compute the normalized quadratic error between x_hat and x
    error = quadratic_error(x_hat, x)

    # Update best settings
    if error < best_error:
        best_error = error
        best_x_hat = x_hat
        best_β = β

print(f"Best normalized quadratic error between"
      f" the original and reconstruction signals: {round(best_error, 4)}")

# Display the reconstructed x_hat as a function of T using log scale horizontal axis
plt.figure(figsize=(12, 5))
plt.title(f"Closed form results using smoothness prior (β={best_β})")
plt.plot(T, best_x_hat, label="reconstruction")
# Display the original signal x as a function of T using log scale horizontal axis
plt.plot(T, x, label="original")
plt.xlabel("Sampled grid")
plt.ylabel("NMR diffusion coefficient")
plt.xscale("log")
plt.legend(frameon=True);
```

**Comment:**

We recovered two peaks although they do not occur for the same sample. In addition, the recovered peaks are not as sharp as the ground truth ones. Moreover we recover some negative values for the NMR diffusion coefficient which does not make sense on a physics standpoint.

#### Smoothness prior + constraints

$$\forall x \in \mathbb{R}^N, \quad g(x) = \frac{1}{2} \parallel Dx \parallel^2 + \iota_{[x_\min, x_\max]^N}(x)$$
with $0 < x_\min < x_\max$ the minimum and maximum values of the original signal $\overline{x}$.

---

1. The term $\iota_{[x_\min, x_\max]^N}$ reduces the domain of the objective to $[x_\min, x_\max]^N$, which is a closed convex subset of $\mathbb{R}^N$. It is bounded, which means there exists $\hat{x} \in [x_\min, x_\max]^N$ such that:

$$\hat{x} = \arg \min_{x \in \mathbb{R}^N} \frac{1}{2} \parallel Kx - y \parallel^2 + \frac{1}{2} \parallel Dx \parallel^2 + \iota_{[x_\min, x_\max]^N}(x)$$

Moreover, since the objective is strictly convex, $\hat{x}$ is unique.

2. The objective is no longer differentiable, which excludes gradient descent as a potential algorithm to solve the optimization problem. Thus we must rely on the **projected gradient algorithm**. Let us denote $g: x \mapsto \frac{1}{2}\parallel Kx - y \parallel^2 + \frac{1}{2} \parallel Dx \parallel^2$ and $f: x \mapsto \iota_{[x_\min, x_\max]^N}(x)$ to recover the notations seen in class in order to apply the Forward-Backward algorithm (specifically its projected gradient variant).

We need to compute the $\nu$-Lipschitzian gradient where $\nu = \max(|a_i|)_{i \in \{1, \ldots, N\}}$ where the $a_i$ are the eigenvalues of the Hessian matrix of $g$ (i.e. $K^T K + \beta D^T D$) and the proximal operator of $\gamma f$ simply becomes the projection on the set $[x_\min, x_\max]^N$.

```python
# Define D the discrete gradient operator
D = np.eye(N)
for i in range(N - 1):
    D[i+1, i] = -1
D[0, -1] = -1

# Set the clipping values
x_min = 1e-4
x_max = 1

# Search for best parameter β
best_error = np.inf
best_x_hat = None
best_β = None
tol = 1e-3 # stopping criterion parameter

β_range = np.linspace(0.1, 1, num=10, endpoint=True)

for β in β_range:

    # Compute the largest eigenvalues of the Hessian of g(x)
    # We use ``eigvalsh`` to only compute the eigenvalues
    # the ``h`` in ``eigvalsh`` refers to a faster algorithm tailored
    # to real symmetric matrices such as $K^T K + \beta D^T D$
    ν =  np.sort(np.linalg.eigvalsh(K.T.dot(K) + β * D.T.dot(D)))[-1]

    # Set the step size
    γ = 1 / ν
    δ = 2 - γ * ν / 2
    λ = 0.95
    assert δ - λ > 0

    # Initialization
    x_curr = np.ones_like(x)
    x_next = np.zeros_like(x)

    # Implement the Projected Gradient Algorithm
    while np.any(np.abs(x_next - x_curr) / (np.abs(x_curr) + 1e-8) > tol): # stopping criterion
        x_curr = x_next
        y_curr = x_curr - γ * (K.T.dot(K.dot(x_curr) - y) + β * D.T.dot(D.dot(x_curr)))
        x_next = x_curr + λ * (np.clip(y_curr, x_min, x_max) - x_curr)

    x_hat = x_next

    error = quadratic_error(x_hat, x)

    if error < best_error:
        best_error = error
        best_x_hat = x_hat
        best_β = β

# Display the reconstruction error
print(f"Best normalized quadratic error between"
      f" the original and reconstruction signals: {round(best_error, 4)}")

# Display the reconstructed x_hat as a function of T using log scale horizontal axis
plt.figure(figsize=(12, 5))
plt.title(f"Projected Gradient Algorithm results using Smoothness + Constraints (β={best_β})")
plt.plot(T, best_x_hat, label="reconstruction")
# Display the original signal x as a function of T using log scale horizontal axis
plt.plot(T, x, label="original")
plt.xlabel("Sampled grid")
plt.ylabel("NMR diffusion coefficient")
plt.xscale("log")
plt.legend(frameon=True);
```

**Comment:**

We recover the two peaks and clearly see the effect of clipping on the minimal value of the reconstructed signal. Adding the boundaries constraints yields a better reconstruction error compared to only using the smoothness prior. However, the recovered peaks centers are slightly shifted from their ground truth positions.

#### Sparsity prior

$$\forall x \in \mathbb{R}^N, \quad g(x) = \parallel x \parallel_1$$

Here we want to induce a parcimonious solution where only a few coefficients are non-null.

---

1. Here the function $x \mapsto \frac{1}{2} \parallel Kx - y \parallel^2 + \parallel x \parallel_1$ is strictly convex but not differentiable. Hence there is a global minima but we cannot obtain it by setting the gradient to 0.

2. The optimization problems consists in minimizing the **sum of two functions** that have different properties. To recover the notations from the course, let us note $f: x \mapsto \parallel x \parallel_1$ and $g: x \mapsto \frac{1}{2} \parallel Kx - y \parallel^2_2$. Now $g$ is lower-semi-continuous, proper, convex and **differentiable** with a $\nu$-Lipschitzian gradient corresponding to the highest eigenvalue of the Hessian matrix $K^T K$ and $f$ is lower-semi-continuous, proper, convex and **separable** therefore we can easily compute its proximal operator and apply it element-wise.

We can now apply the Forward-Backward algorithm in its iterative shrinkage-thresholding variant (ISTA).

```python
# Search for best parameter β
best_error = np.inf
best_x_hat = None
best_β = None
tol = 1e-3 # stopping criterion

β_range = np.linspace(0.1, 0.5, num=10, endpoint=True)

for β in β_range:
    # Compute the largest eigenvalues of the Hessian of g(x)
    # We use ``eigvalsh`` to only compute the eigenvalues
    # the ``h`` in ``eigvalsh`` refers to a faster algorithm tailored
    # to real symmetric matrices such as $K^T K$
    ν =  np.sort(np.linalg.eigvalsh(K.T.dot(K)))[-1]

    # Set the step size
    γ = 1 / ν
    δ = 2 - γ * ν / 2
    λ = 0.95
    assert δ - λ > 0

    # Initialization
    x_curr = np.zeros_like(x)
    x_next = np.ones_like(x)

    # Implement the Projected Gradient Algorithm
    while np.any(np.abs(x_next - x_curr) / (np.abs(x_curr) + 1e-8) > tol): # stopping criterion
        x_curr = x_next
        y_curr = x_curr - γ * K.T.dot(K.dot(x_curr) - y)
        # Thresholding
        x_next = x_curr + λ * (np.sign(y_curr) * np.maximum(np.abs(y_curr) - (β / 2 * γ), 0) - x_curr)

    x_hat = x_next

    error = quadratic_error(x_hat, x)

    if error < best_error:
        best_error = error
        best_x_hat = x_hat
        best_β = β

# Display the reconstruction error
print(f"Best normalized quadratic error between"
      f" the original and reconstruction signals: {round(best_error, 4)}")

# Display the reconstructed x_hat as a function of T using log scale horizontal axis
plt.figure(figsize=(12, 5))
plt.title(f"Iterative Shrinkage-Thresholding Algorithm results  (β={round(best_β, 4)})")
plt.plot(T, best_x_hat, label="reconstruction")
# Display the original signal x as a function of T using log scale horizontal axis
plt.plot(T, x, label="original")
plt.xlabel("Sampled grid")
plt.ylabel("NMR diffusion coefficient")
plt.xscale("log")
plt.legend(frameon=True);
```

**Comment:**

We clearly recover the two ground truth peaks. We get a better reconstruction error than in the two previous settings. The reconstructed signal is sparse as there are only a few coefficients that are non null. Since we have **prior knowledge** of the task at hand, that is recovering different species of molecules by looking at their Nuclear Magnetic Resonance diffusion coefficient, aiming for a **sparse solution appears to be a good idea**.

**Conclusion:**

Note however that these results based on synthetic data **highly depend on the noise** added to the ground truth measurement. We encountered instances of the problem where the smoothness + constraints (case 2) yielded a lower error than the sparsity prior. As a consequence, we **fixed the random seed** generating the noise to make the results reproducible.

## Part 2 - Maximum entropy regularization

We use a standard strategy for restoring the target signal that consists in finding a solution of the so-called maximum entropy problem:

$$\hat{x} = \arg \min_{x \in \mathbb{R}^N} \frac{1}{2} \parallel Kx - y \parallel^2 + \beta \;\text{ent}(x)$$

where $\forall x \in \mathbb{R}^N, \quad \text{ent}(x) = \sum_{n=1}^N \varphi(x^{(n)})$, with:

$$\varphi(u) =
\begin{cases}
u \log u  & \text{if $u$ > 0} \\
0 & \text{if $u$ = 0} \\
+ \infty & \text{elsewhere}
\end{cases}$$

### Question 1.

Is $\text{ent}$ convex? proper? lower-semicontinuous? differentiable?

---

1. The function $\text{ent}$ defined below is proper since its domain is non-empty and it never atteins $- \infty$.
2. $\text{ent}$ is lower-semicontinuous since it is separable and each components is continuous on $\mathbb{R}_+$ as $\lim_{u \rightarrow 0} u \log u = 0$ and equal to $+ \infty$ on $\mathbb{R}_-$
3. $\text{ent}$ separable components are differentiable on $\mathbb{R}_+^*$ and we have: $\forall u \in \mathbb{R}_+, \phi'(u) = 1 + \log u$. Since the separable components derivatives are monotonically strictly increasing we get that $\text{ent}$ is **strictly convex** on its domain ${\mathbb{R}_+}^N$

### Question 2.

Does the optimization problem have a solution? Is it unique?

---

If we define $f: x \in \mathbb{R}^N \mapsto \frac{1}{2} \parallel Kx - y \parallel^2 + \beta \;\text{ent}(x) \in \mathbb{R}$, using the previous question we can say that $f$ can only be minimized on ${\mathbb{R}_+^*}^N$ on which $\text{ent}$ is strictly convex. Since $K$ entries are strictly positive, we get that $x \in {\mathbb{R}_+^*}^N \mapsto \frac{1}{2} \parallel Kx - y \parallel^2$ is strictly convex. As a result, the optimization problem has a **unique solution**.

### Question 3.

Give the expression of the proximity operator of $\text{ent}$ at some $x \in \mathbb{R}^N$ using the Lambert $W$-function.

---

As a reminder, Lambert $W$-function is such that $W: h(w) \mapsto w$ where $h(w): w \mapsto w \exp w$. In other words, it is the inverse function of $h$ but it has no analytic form and can be approximated numerically.

We start by noticing that the $\text{ent}$ function is separable by definition, thus its proximal operator is also separable.

Since the minimizer can only belong to $\mathbb{R}_+^*$, we define $\text{prox}_{\gamma \text{ent}}(u) = \arg \min_{z > 0} \gamma z \log z + \frac{1}{2} \parallel z - u \parallel^2$

Since $\text{ent}$ separable components are differentiable on $\mathbb{R}_+^*$, we compute $\text{prox}_{\gamma \text{ent}}(u)$ by solving the equation obtained by setting the derivative of $g_u: z \in \mathbb{R}_+^* \mapsto \gamma z \log z + \frac{1}{2} \parallel z - u \parallel^2$  to 0.

We get:

\begin{align}
g_u'(z^*) = 0 & \iff \log z^* + 1 + \frac{z^* - u}{\gamma} = 0 \\
& \iff \frac{z^*}{\gamma} + \log (\frac{z^*}{\gamma}) = \frac{u}{\gamma} -1 - \log \gamma \\
& \iff \frac{z^*}{\gamma} \exp(\frac{z^*}{\gamma}) = \exp (\frac{u}{\gamma} - 1 - \log \gamma)\\
\end{align}

We recognize the Lambert $W$-function defined above, therefore we get:

\begin{align}
g_u'(z^*) = 0 & \iff \frac{z^*}{\gamma} = W \big( \exp(\frac{u}{\gamma} - 1 - \log \gamma)\big) \\
& \iff  z^* = \gamma W \big( \exp(\frac{u}{\gamma} - 1 - \log \gamma)\big) \\
\end{align}

Finally, we obtain a closed-form solution that involves Lambert $W$-function: $$\boxed{\text{prox}_{\gamma \text{ent}}(u) = \gamma W \big( \exp(\frac{u}{\gamma} - 1 - \log \gamma)\big)}$$

Note: we wrote $\log$ when we meant to use $\ln$ in the above derivations.

```python
# Lambert W-function implementation
def W_func(u, tol=1e-3):
    """
    Computes W-function using numerical approximation
    of ::math::$W(\exp(u))$

    Parameter
    ---------
    - u : float
        Argument to compute $W(\exp u)$
    """
    if u > 100:
        return u - np.log(u)
    elif u < -20:
        return 0
    else:
        # Haley's method
        w = 1
        v = np.inf * w
        while np.any(np.abs(w - v) / np.abs(w) > tol):
            v = w
            e = np.exp(w)
            f = w * e - np.exp(u)  # Iterate to make this quantity zero
            w = w - f / ((e * (w+1) - (w+2) * f / (2*w+2)))
        return w

# Vectorize Lambert W-function
vec_W_func = np.vectorize(W_func)
```

### Question 4.

Proprose a Forward-Backward and a Douglas-Rachford algorithm to solve problem (1)

---

1. Given the previous answers, we have that $h_0: x \in {\mathbb{R}_+^*}^N \mapsto \frac{1}{2} \parallel Kx - y \parallel^2$ is convex, proper, lower-semicontinuous and differentiable with its gradient Lispchitz constant $\nu$ equal to the largest eigenvalue of $K^T K$. Moreover, $h_1: x \in {\mathbb{R}_+^*}^N \mapsto \beta \text{ent}(x)$ is convex, proper and lower-semicontinuous. Hence we can use now use the Forward Backward algorithm. Given that we use $\gamma \in ]0, \frac{2}{\nu}[$, $\delta = 2 - \frac{\gamma \nu}{2} \in ]1, 2[$ and $(\lambda_n)_{n \in \mathbb{N}}$ such that $\sum_{n \in \mathbb{N}} \lambda_n (\delta - \lambda_n) = + \infty$, we can now write Foward Backward main loop, i.e. $\forall n \in \mathbb{N}$:

$$\begin{cases}
y_n = x_n - \gamma K^T (K x_n - y) \\
x_{n+1} = x_n + \lambda_n (\gamma \beta W_{\text{vec}} (\exp(\frac{y_n}{\gamma \beta} - 1 - \ln(\gamma \beta)) - x_n)
\end{cases}
$$

Where $W_{\text{vec}}$ denotes the vectorized version of Lambert $W$-function.

Note that since $h_1$ is differentiable (on ${\mathbb{R}_+^*}^N$) we could possibly use a reverse formulation where we swap $h_0$ and $h_1$ although it requires computing the Lipschitz constant of the gradient of $h_1$.

2. In addition, we can use Douglas Rachford as it requires even weaker hypotheses. We only need to compute the proximal operator of $h_0$:

$$\text{prox}_{\gamma h_0} (x) = \arg \min_{z \in \mathbb{R}^N} \frac{\gamma}{2} \parallel Kz - y \parallel^2 + \frac{1}{2} \parallel z - x \parallel^2$$

By setting the gradient of $z \in \mathbb{R}^N \mapsto \frac{\gamma}{2} \parallel Kz - y \parallel^2 + \frac{1}{2} \parallel z - x \parallel^2$ to $0_{\mathbb{R}^N}$, we get:

$$\text{prox}_{\gamma h_0} (x) = (\gamma K^T K + I_N)^{-1} (\gamma K^T y + x)$$

Given $\gamma \in ]0, + \infty[$ and $(\lambda_n)_{n \in \mathbb{N}}$ such that $\sum_{n \in \mathbb{N}} \lambda_n (2 - \lambda_n) = + \infty$, we can now write Douglas Rachford main loop, i.e. $\forall n \in \mathbb{N}$:

$$\begin{cases}
y_n = \gamma \beta W_{\text{vec}} (\exp(\frac{x_n}{\gamma \beta} - 1 - \ln(\gamma \beta)) \\
z_n = (\gamma K^T K + I_N)^{-1} (\gamma K^T y + 2 y_n - x_n) \\
x_{n+1} = x_n + \lambda_n (z_n - y_n)
\end{cases}
$$

### Question 5.

Implement them both and evaluate their performances for $\beta = 10^{-2}$

---

```python
# Setup
tol = 1e-3 # stopping criterion
β = 1e-2

# Compute the largest eigenvalues of the Hessian of g(x)
# We use ``eigvalsh`` to only compute the eigenvalues
# the ``h`` in ``eigvalsh`` refers to a faster algorithm tailored
# to real symmetric matrices such as $K^T K$
ν =  np.sort(np.linalg.eigvalsh(K.T.dot(K)))[-1]

# Set the step size
γ = 1 / ν
δ = 2 - γ * ν / 2
λ = 1
assert δ - λ > 0

# Initialization
x_curr = np.zeros_like(x)
x_next = np.ones_like(x)

counter = 0

# Implement the Forward Backward algorithm
while np.any(np.abs(x_next - x_curr) / (np.abs(x_curr) + 1e-8) > tol): # stopping criterion
    x_curr = x_next
    y_curr = x_curr - γ * K.T.dot(K.dot(x_curr) - y)
    # Use vectorized Lambert W function
    x_next = x_curr + λ * (γ * β * vec_W_func(y_curr / (γ * β) - 1 - np.log(γ * β)) - x_curr)
    counter += 1

x_hat = x_next

error = quadratic_error(x_hat, x)

# Display the reconstruction error
print(f"Normalized quadratic error between"
      f" the original and reconstruction signals: {round(error, 4)}")

# Display the reconstructed x_hat as a function of T using log scale horizontal axis
plt.figure(figsize=(12, 5))
plt.title(f"Maximum entropy regularization (β={β}) using Foward-Backward ({counter} steps)")
plt.plot(T, x_hat, label="reconstruction")
# Display the original signal x as a function of T using log scale horizontal axis
plt.plot(T, x, label="original")
plt.xlabel("Sampled grid")
plt.ylabel("NMR diffusion coefficient")
plt.xscale("log")
plt.legend(frameon=True);
```

```python
# Douglas-Rachford algorithm implementation

# Setup
tol = 1e-3 # stopping criterion
β = 1e-2

# Set the step size
γ = 0.1
λ = 1.9

# Compute once the matrix inversion
_inv = np.linalg.inv(γ * K.T.dot(K) + np.eye(N))

# Initialization
x_curr = np.zeros_like(x)
x_next = np.ones_like(x)

counter = 0

# Implement Douglas Rachford algorithm
while np.any(np.abs(x_next - x_curr) / (np.abs(x_curr) + 1e-8) > tol): # stopping criterion
#for _ in range(n_iter):
    x_curr = x_next
    #y_curr = x_curr - γ * K.T.dot(K.dot(x_curr) - y)
    #y_curr = γ * β * vec_W_func(x_curr / (γ * β) - 1 - np.log(γ * β))
    y_curr = _inv.dot(γ * K.T.dot(y) + x_curr)
    #z_curr = _inv.dot(γ * K.T.dot(y) + 2 * y_curr - x_curr)
    z_curr = γ * β * vec_W_func((2 * y_curr - x_curr) / (γ * β) - 1 - np.log(γ * β))
    # Use vectorized Lambert W function
    x_next = x_curr + λ * (z_curr - y_curr)
    counter += 1

x_hat = x_next

error = quadratic_error(x_hat, x)

# Display the reconstruction error
print(f"Normalized quadratic error between"
      f" the original and reconstruction signals: {round(error, 4)}")

# Display the reconstructed x_hat as a function of T using log scale horizontal axis
plt.figure(figsize=(12, 4))
plt.title(f"Maximum entropy regularization (β={β}) using Douglas Rachford ({counter} steps)")
plt.plot(T, x_hat, label="reconstruction")
# Display the original signal x as a function of T using log scale horizontal axis
plt.plot(T, x, label="original")
plt.xlabel("Sampled grid")
plt.ylabel("NMR diffusion coefficient")
plt.xscale("log")
plt.legend(frameon=True);
```

**Comment:**

* We use a fairly high tolerance (``tol = 1e-3``) otherwise computations become quite long. $\beta = 0.01$ does not seem to be the best hyperparameter. We note in this setting that Douglas Rachford needs fewer steps than Forward Backward although the requirements are less strict. In addition, Douglas Rachford does not impose any theoretical limits on the step size $\gamma$ other than it being strictly positive.

* In the next question we search for the best parameter $\beta$ in terms of reconstruction error.

### Question 6.

What is the best choice for parameter $\beta$ in terms of reconstruction error between the estimated object and the ground truth?

---

```python
%%time

# Search for best parameter β
best_error = np.inf
best_x_hat = None
best_β = None
best_counter = None
tol = 1e-3 # stopping criterion

β_range = np.linspace(1e-8, 1e-6, num=3, endpoint=True)

# Compute the largest eigenvalues of the Hessian of g(x)
# We use ``eigvalsh`` to only compute the eigenvalues
# the ``h`` in ``eigvalsh`` refers to a faster algorithm tailored
# to real symmetric matrices such as $K^T K$
ν =  np.sort(np.linalg.eigvalsh(K.T.dot(K)))[-1]

# Set the step size
γ = 1 / ν
δ = 2 - γ * ν / 2
λ = 1
assert δ - λ > 0

for β in β_range:

    # Initialization
    x_curr = np.zeros_like(x)
    x_next = np.ones_like(x)

    counter = 0

    # Implement the Forward Backward algorithm
    while np.any(np.abs(x_next - x_curr) / (np.abs(x_curr) + 1e-8) > tol): # stopping criterion
        x_curr = x_next
        y_curr = x_curr - γ * K.T.dot(K.dot(x_curr) - y)
        # Use vectorized Lambert W function
        x_next = x_curr + λ * (γ * β * vec_W_func(y_curr / (γ * β) - 1 - np.log(γ * β)) - x_curr)
        counter += 1

    x_hat = x_next

    error = quadratic_error(x_hat, x)

    if error < best_error:
        best_error = error
        best_x_hat = x_hat
        best_β = β
        best_counter = counter

# Display the reconstruction error
print(f"Best normalized quadratic error between"
      f" the original and reconstruction signals: {round(best_error, 4)}")

# Display the reconstructed x_hat as a function of T using log scale horizontal axis
plt.figure(figsize=(12, 5))
plt.title(f"Maximum entropy regularization (β={best_β}) "
          f"using Foward-Backward ({best_counter} steps)")
plt.plot(T, x_hat, label="reconstruction")
# Display the original signal x as a function of T using log scale horizontal axis
plt.plot(T, x, label="original")
plt.xlabel("Sampled grid")
plt.ylabel("NMR diffusion coefficient")
plt.xscale("log")
plt.legend(frameon=True);
```

```python
%%time
# Douglas-Rachford algorithm implementation

# Search for best parameter β
best_error = np.inf
best_x_hat = None
best_β = None
best_counter = None
tol = 1e-3 # stopping criterion

β_range = np.linspace(1e-8, 1e-6, num=3, endpoint=True)

# Set the step size
γ = 0.1
λ = 1.9

# Compute once the matrix inversion
_inv = np.linalg.inv(γ * K.T.dot(K) + np.eye(N))

for β in β_range:

    # Initialization
    x_curr = np.zeros_like(x)
    x_next = np.ones_like(x)

    counter = 0

    # Implement Douglas Rachford algorithm
    while np.any(np.abs(x_next - x_curr) / (np.abs(x_curr) + 1e-8) > tol): # stopping criterion
        x_curr = x_next
        y_curr = _inv.dot(γ * K.T.dot(y) + x_curr)
        # Use vectorized Lambert W function
        z_curr = γ * β * vec_W_func((2 * y_curr - x_curr) / (γ * β) - 1 - np.log(γ * β))
        x_next = x_curr + λ * (z_curr - y_curr)
        counter += 1

    x_hat = x_next

    error = quadratic_error(x_hat, x)

    if error < best_error:
        best_error = error
        best_x_hat = x_hat
        best_β = β
        best_counter = counter

# Display the reconstruction error
print(f"Best normalized quadratic error between"
      f" the original and reconstruction signals: {round(best_error, 4)}")

# Display the reconstructed x_hat as a function of T using log scale horizontal axis
plt.figure(figsize=(12, 5))
plt.title(f"Maximum entropy regularization (β={best_β}) "
          f"using Douglas Rachford ({best_counter} steps)")
plt.plot(T, x_hat, label="reconstruction")
# Display the original signal x as a function of T using log scale horizontal axis
plt.plot(T, x, label="original")
plt.xlabel("Sampled grid")
plt.ylabel("NMR diffusion coefficient")
plt.xscale("log")
plt.legend(frameon=True);
```

**Comment:**

* We notice that once again Douglas Rachford converges in fewer steps. However the fact it actually takes longer might be related to the Lambert $W$-function computation which seems to be our bottleneck here.

* In addition, while the two peaks are recovered by both algorithms, the Forward Backward solution is closer to the ground truth with respect to the peaks relative value. However the Douglas Rachford solution leads to peaks positions that look more accurate.

### Question 7.

Compare the maximum entropy regularization with the ones proposed in the previous part in terms of reconstruction quality.

---

**Comment:**

* In order to get a meaningful comparison, we fixed the tolerance to ``tol = 1e-3`` in all the algorithms that rely on a stopping criterion (that only excludes the smoothness prior) and we use a random seed to make the results reproducible.

**Quantitative analysis**

Algorithm variant | Best normalized quadratic error | Stopping criterion (tolerance)
--- | --- | ---
*Smoothness prior* | 0.3849 | ``None``
*Smothness prior* + *constraints* | 0.1542 | ``tol=1e-3``
*Sparsity prior* | **0.11** | ``tol=1e-3``
*Maximum entropy regularization* (*Forward Backward*) | 0.1409 | ``tol=1e-3``
*Maximum entropy regularization* (*Douglas Rachford*) | 0.2472 | ``tol=1e-3``

* **Qualitative analysis**

We would have expected *Maximum entropy regularization* to trump previous priors but the *sparsity prior* still yields the best reconstruction by a fair margin over the *Forward Backward* variant of *Maximum entropy regularization*, while being significantly faster. One could argue that the high tolerance used in the Lambert $W$-function approximation (``tol=1e-3``) might be responsible for such results but we found it to provide reasonable results in a reasonable computational time (there is indeed a trade-off between computation time and precision at play).

Note that every regularization scheme requires to carefully fine-tune the regularization parameter $\beta$ in order to obtain good results. This can be a time consuming process. In addition, there is no principled way to know how to choose $\beta$ beforehand, hence we rely on a hyperparameter grid search. Finally, we are able to measure the reconstruction quality only because we have access to the ground truth solution but this is not the case in practice.

### Question 8.

In practice tuning the parameter $\beta$ is difficult. Using information about the level of noise corrupting the data, we can formulate a more practical optimization problem

$$ \text{minimize}_{x \in \mathbb{R}^N} \; \text{ent}(x) \; \text{s.t.} \; \parallel Kx - y \parallel^2 \leq \eta M \sigma^2 $$

with $\eta > 0$ and we expect $\eta$ to be close to 1.

-------

Let us rewrite the optimization problem in the following equivalent way:

$$ \text{minimize}_{x \in \mathbb{R}^N} \; \text{ent}(x)  + \iota_{B(y, \sigma \sqrt{M \eta})}(Kx)$$

where $B(y, \sigma \sqrt{M \eta})$ denotes the $l_2$ norm ball centered on $y$ and of radius $\sigma \sqrt{M \eta}$.

We can now use the parallel form of Douglas Rachford (noted PPXA+) to solve this optimization problem.

Using the course notation, we have $g(x) = \sum_{i=1} g_i(x_i)$ where $g_1: x_1 \in \mathbb{R}^N \mapsto \text{ent}(x_1)$ and $g_2: x_2 \in \text{ran}(K) \mapsto \iota_{B(y, \sigma \sqrt{M \eta})}(x_2)$ where $\text{ran}(K)$ denotes the image or range of the matrix $K \in \mathcal{B}(\mathbb{R}^N, \text{ran}(K))$.

We get that $g_1 \in \Gamma_0(\mathbb{R}^N)$ and $g_2 \in \Gamma_0(\text{ran}(K))$.

We also have $L: v \mapsto (L_1 v, L_2 v)$ where $L_1 = I_N$ and $L_2 = K$.

We only need to invert the invertible matrix $I_N + K^T K$ once.

Lastly, the proximal operators of $\gamma g_1$ and $\gamma g_2$ are:

$$\text{prox}_{\gamma g_1} (x_1) = \gamma W_{\text{vec}}(\frac{x_1}{\gamma} - 1 - \log (\gamma))$$

$$\text{prox}_{\gamma g_2} (x_2) = y + (x_2 - y) \min(1, \frac{\sigma \sqrt{M \eta}}{\parallel x_2 - y \parallel})$$

```python
%%time

# Parallel Proximal Algorithm implementation

# Search for best parameter β
best_error = np.inf
best_x_hat = None
best_η = None
best_counter = None
tol = 1e-2 # stopping criterion
#n_iter = 300

η_range = np.linspace(0.5, 1.5, num=10, endpoint=True)

# Set the step size
γ = 0.1
λ = 1.7

# Compute once the matrix inversion
#_inv_prod = np.linalg.inv(K.T.dot(K) + np.eye(N)).dot(np.eye(N) + K.T)
_inv = np.linalg.inv(K.T.dot(K) + np.eye(N))

for η in η_range:

    # Initialization
    x_curr_1 = np.zeros_like(x)
    x_curr_2 = K.dot(x_curr_1)
    x_next_1 = np.ones_like(x)
    x_next_2 = K.dot(x_next_1)
    v_curr = _inv.dot(x_curr_1 + K.T.dot(x_curr_2))
    v_next = _inv.dot(x_next_1 + K.T.dot(x_next_2))

    counter = 0

    # Implement Parallel form of Douglas Rachford algorithm
    while np.any(np.abs(v_next - v_curr) / (np.abs(v_curr) + 1e-8) > tol): # stopping criterion
    #for _ in range(n_iter):
        x_curr_1 = x_next_1
        x_curr_2 = x_next_2
        v_curr = v_next

        y_curr_1 = γ * vec_W_func(x_curr_1 / γ - 1 - np.log(γ))
        y_curr_2 = y + (x_curr_2 - y) * np.min([1, σ * np.sqrt(M * η) / np.linalg.norm(x_curr_2 - y)])

        c_curr = _inv.dot(y_curr_1 + K.T.dot(y_curr_2))

        x_next_1 = x_curr_1 + λ * (2 * c_curr - v_curr - y_curr_1)
        x_next_2 = x_curr_2 + λ * (K.dot(2 * c_curr - v_curr) - y_curr_2)

        v_next = v_curr + λ * (c_curr - v_curr)

        counter += 1

    x_hat = v_next

    error = quadratic_error(x_hat, x)

    if error < best_error:
        best_error = error
        best_x_hat = x_hat
        best_η = η
        best_counter = counter

# Display the reconstruction error
print(f"Best normalized quadratic error between"
      f" the original and reconstruction signals: {round(best_error, 4)}")

# Display the reconstructed x_hat as a function of T using log scale horizontal axis
plt.figure(figsize=(12, 5))
plt.title(f"Reconstructed signal (η={round(best_η, 4)}) "
          f"using Parallel Douglas Rachford ({best_counter} steps)")
plt.plot(T, best_x_hat, label="reconstruction")
# Display the original signal x as a function of T using log scale horizontal axis
plt.plot(T, x, label="original")
plt.xlabel("Sampled grid")
plt.ylabel("NMR diffusion coefficient")
plt.xscale("log")
plt.legend(frameon=True);
```

**Comment**:

* We obtain a reconstruction whose normalized reconstruction error falls in between the *Maximum entropy regularization* variants but this time using our prior knowledge of the noise power $\sigma$ allows us to conduct a *fine-grained* grid search for $\eta$ between 0.5 and 1.5. We kept a high tolerance (``tol=1e-2``) as we experienced long computation times and unreasonable solutions at lower tolerance. In addition, we played with the step size of $\gamma$ and $\lambda$.

We summarize our findings below:

Algorithm variant | Best normalized quadratic error | Stopping criterion (tolerance) | Regularization parameter ($\beta$ or $\eta$) | Step size $\gamma$ | Relaxation parameter $\lambda$
--- | --- | --- | --- | --- | ---
*Smoothness prior* | 0.3849 | ``None`` | $\beta = 0.515$ | ``None`` | ``None``
*Smothness prior* + *constraints* | 0.1542 | ``tol=1e-3`` | $\beta = 0.1$ | ${\frac{1}{\nu}}^{(*^1)}$ | ``0.95``
*Sparsity prior* | **0.11** | ``tol=1e-3`` | $\beta=0.1$ | ${\frac{1}{\nu}}^{(*^2)}$ | ``0.95``
*Maximum entropy regularization* (*Forward Backward*) | 0.1409 | ``tol=1e-3`` | $\beta = 5.05 \times 10^{-7}$ | ${\frac{1}{\nu}}^{(*^2)}$ | ``1``
*Maximum entropy regularization* (*Douglas Rachford*) | 0.2472 | ``tol=1e-3`` | $\beta = 10^{-8}$ | ``0.1`` | ``1.9``
*Maximum entropy regularization* (*PPXA+* using *Noise prior*) | 0.2012 | ``tol=1e-2`` | $\eta = 0.722$ | ``0.1`` | ``1.7``

* $(*^1)$: $\nu$ is the largest eigenvalue of $K^T K + \beta D^T D$
* $(*^2)$: $\nu$ is the largent eigenvalue of $K^T K$