## Optimization Problem Definition

An optimization problem has three main elements:
- design variables
- objective
- constraints

**Design variables** are the variables that we allow the optimizer to change.  We denote these variables as $x$, where $x$ is an array (i.e., there can be many design variables).  For example if we were designing the shape of the spoiler on a car, the design variables might be things like the length of the spoiler, the distance away from the car body, and parameters that define its curved profile.

The **objective** is the metric we want to minimize.  We usually denote this function by $J$ or sometimes $f$.  It is a scalar, meaning a single value (at times it makes sense to have multiple objectives, but the approach is different—we will explore this later in the semester).  By convention we say minimize because we often minimize things in engineering, but in other disciplines, like finance, the default convention may be to maximize.  This does not mean that the algorithms we explore can can only perform minimization, to maximize you simply minimize the negative of the function ($-J(x)$).  In our car example, the objective of our spoiler design may be to maximize (downward) lift.

**Constraints** exist in almost any engineering problem, they are conditions that we require the optimizer to satisfy.  We denote them with $c$, where $c$ is an array because we may have many constraints.  By convention we will define constraints in this form: $c(x) \le 0$.  This is completely general as a constraint of the form $q(x) \ge 0$ could be rewritten as $c(x) = -q(x) \le 0$.  In our car example the constraints may be things like a constraint on bending stress, manufacturing tolerances, etc.  Without these constraints, our spoiler may produce a lot of lift but may snap in half or be too difficult to manufacture.

In short, we can write all of our optimization problems in this form:

\begin{align*}
\text{minimize} \quad & J(x) \\
\text{with respect to} \quad & x \\
\text{subject to} \quad & c(x) \le 0 \\
\end{align*}

## Example Problem

Let's consider a simple problem called the Rosenbrock problem.  It has two design variables: $x$ and $y$, and an objecive function of:
$$J(x, y) = (1 - x)^2 + 100(y - x^2)^2$$

It's often used as a starting problem for testing optimization algorithms because it is only 2D (so it is easy to visualize), but it has a long valley that is steep in one direction and shallow in the other making it somewhat challenging to optimize.  In fact, let's visualize the function space with a contour plot (which we can do because there are only 2 variables).  First let's create the function. Note that x will be passed into this function as a vector.  This will be necessary later when using the optimization algorithms.

```matlab
function [J] = obj(x)

    J = (1 - x(1))^2 + 100*(x(2) - x(1)^2)^2;

end
```

We can also do this as an inline function (called anonymous function in matlab) because it is so simple.  Anonymous functions functions can only be one-liners):

```matlab
obj = @(x) (1 - x(1))^2 + 100*(x(2) - x(1)^2)^2;
```

Now let's create a contour plot for that function.

```matlab
% --- setup grid ---
nx = 200;  % number of points in x-direction
ny = 150;  % number of points in y-direction
x = linspace(-1, 2, nx);  % nx points equally spaced between -5...5
y = linspace(-1, 2, ny);  % ny points equally spaced between -6...6
[X, Y] = ndgrid(x, y);  % 2D array (matrix) of points across x and y
Z = zeros(nx, ny);  % initialize output of size (nx, ny)

% --- evaluate across grid ---
for i = 1:nx
    for j = 1:ny
        Z(i, j) = obj([X(i, j), Y(i, j)]);
    end
end

% --- contour plot ---
figure();  % start a new figure
contour(X, Y, Z, 100);  % using 100 contour lines.
colorbar();  % add a colorbar
xlabel('x');  % labels for axes
ylabel('y');
```

Note the flat U-shaped valley.  The minimum is actually at (1, 1).  A couple of other notes on contour plots:
- You can also meshgrid instead of ndgrid. meshgrid works exactly the same, except for the size of the array will be (ny, nx) (backwards from ndgrid) . This is intended to mimic the physical space where the columns go along the x-direction. I prefer ndgrid because I think it’s clearer to use x as the first index and y as the second.
- contourf is the same as contour, except for it creates filled colorbars.

## Using fminunc

Let's find the minimum of the Rosenbrock function.  This problem is unconstrained, nonlinear, and differentiable.  The corresponding matlab algorithm for that type of problem is fminunc.  *fminunc* has a pretty basic function signature (type 'help fminunc' or 'doc fminunc').

```matlab
[xopt, fopt, exitflag, output] = fminunc(@obj, x0, options);
```

We already created the objective function obj earlier.  We now need to provide a starting point (I've chosen (0, 0), arbitrarily).

```matlab
x0 = [0, 0];
```

Finally, we can set various options. I've set a few common ones below, but check out the documentation on fminunc for other options that you can set.

```matlab
options = optimoptions(@fminunc,...
    'display', 'iter-detailed', ...  % display information on each iteration
    'MaxIterations', 1000, ...  % maximum number of iterations
    'MaxFunctionEvaluations', 10000, ...  % maximum number of function calls
    'OptimalityTolerance', 1e-6 ...  % convergence tolerance
);
```

```matlab
[xopt, fopt, exitflag, output] = fminunc(@obj, x0, options);
```

    Warning: Gradient must be provided for trust-region algorithm; using quasi-newton algorithm instead.
    > In fminunc (line 395)
                                                            First-order
     Iteration  Func-count       f(x)        Step-size       optimality
         0           3                1                             2
         1          12         0.771192      0.0817341           5.34
         2          15         0.610658              1           6.73
         3          18         0.522451              1           7.11
         4          24         0.261629         0.7075           1.88
         5          30         0.248996            0.5           3.44
         6          33         0.207486              1           2.94
         7          36         0.125351              1            1.5
         8          39        0.0893498              1           3.93
         9          42        0.0308666              1           1.23
        10          48        0.0200762       0.322023           1.95
        11          51        0.0138484              1           1.57
        12          54       0.00441549              1          0.303
        13          60       0.00268684            0.5           1.14
        14          63      0.000276522              1           0.28
        15          66      4.21032e-05              1          0.122
        16          69      1.37272e-06              1        0.00796
        17          72      7.48614e-07              1         0.0303
        18          75      3.34372e-09              1        0.00221
        19          78      6.71309e-11              1       7.22e-05
                                                            First-order
     Iteration  Func-count       f(x)        Step-size       optimality
        20          81      1.94746e-11              1       1.06e-06

    Optimization completed: The first-order optimality measure, 3.522486e-07, is less
    than options.OptimalityTolerance = 1.000000e-06.

    Optimization Metric                                          Options
    relative norm(gradient) =   3.52e-07             OptimalityTolerance =   1e-06 (selected)

Problem solved!  Always read the output message to make sure it actually solved (as opposed to just reached a max iteration limit, or had a convergence failure, etc.).  The relevant message is:  "Optimization completed: The first-order optimality measure, 3.522486e-07, is less than options.OptimalityTolerance = 1.000000e-06.".  Let's print out the optimal x value, and corresponding function value.

```matlab
xopt
fopt
```

    xopt =

        1.0000    1.0000

    fopt =

       1.9475e-11

We have found the optimal solution: $x^* = (1, 1)$, $f^* = 0$.

## Using fmincon

Let's now add a couple of constraints.  The first, is that the solution must be within the unit disk.
$$ x^2 + y^2 \le 1$$
The second is that
$$x + 3y \le 5 $$

We need to rewite the constraints in our standard format:
\begin{align}
x^2 + y^2 -1 &\le 0 \\
x + 3y -5 &\le  0
\end{align}

This problem is constrained, nonlinear, and differentiable.  The matlab algorithm that is equipped to solve that type of problem is fmincon.  If we look at the documentation for fmincon we can see that it has a function signature like this:

```matlab
[xopt, fopt, exitflag, output] = fmincon(@obj, x0, A, b, Aeq, beq, lb, ub, @con, options);
```

Take a look at the documentation to see what each input and output means (use doc fmincon or help fmincon).  First we need to set our starting point at bounds.  We will set our bounds as the (-5, -5) to (5, 5) box.  As a starting point let's just pick (0, 0) somewhat arbitrarily.

```matlab
% -------- starting point and bounds ----------
x0 = [0, 0];
ub = [5, 5];  % upper bound, the largest values we will allow x to acheive
lb = [-5, -5];  % lower bound
% ---------------------------------------------
```

We have no linear ineqality constraints (A, b) or linear equality constraints (Aeq, beq) so we will set those as empty arrays.

```matlab
% ------ linear constraints ----------------
A = [];
b = [];
Aeq = [];
beq = [];
% ------------------------------------------
```

Next let's define our objective and constraints in the function below.  Note that we need to use the convention that J will be minimized, and c is of the form $c \le 0$

```matlab
% ---- Objective and Constraints -------------
function [J, c, ceq] = objcon(x)

    % objective
    J = (1 - x(1))^2 + 100*(x(2) - x(1)^2)^2;

    % inequality constraints
    c = zeros(2, 1);  % there are 2 of them
    c(1) = x(1)^2 + x(2)^2 - 1;
    c(2) = x(1) + 3*x(2) - 5;

    % equality constraints
    ceq = [];  % there are none

end
% -------------------------------------------
```

We can set various options.  I've included some of the more common ones we might use below.

```matlab
options = optimoptions(@fmincon,...
    'Algorithm', 'active-set', ...  % choose one of: 'interior-point', 'sqp', 'active-set', 'trust-region-reflective'
    'AlwaysHonorConstraints', 'bounds', ...  % forces interior point algorithm to always honor bounds
    'display', 'iter-detailed', ...  % display more information
    'MaxIter', 1000, ...  % maximum number of iterations
    'MaxFunEvals', 10000, ...  % maximum number of function calls
    'TolCon', 1e-6, ...  % convergence tolerance on constraints
    'TolFun', 1e-6, ...  % convergence tolerance on function value
    'FinDiffType', 'forward', ...  % if finite differencing, can also use central
    'GradObj', 'off', ...  % supply gradients of objective
    'GradConstr', 'off', ...  % supply gradients of constraints
    'DerivativeCheck', 'off', ...  % on if you want to check your supplied gradients against finite differencing
    'Diagnostics', 'on');  % display diagnotic information
```

Note that fmincon asks for your objective and constraints separately (whereas we have provided them both in one function).  In most engineering analyses the objective and constraints are computed in the same script.  As a convenience I've written a template for you that does the separation without requiring any extra function calls.  Download the template optimize.m from our course website.  If you fill it out with what was filled out above we can now run fmincon.

```matlab
[xopt, fopt, exitflag, output] = fmincon(@obj, x0, A, b, Aeq, beq, lb, ub, @con, options);
```

    ____________________________________________________________
       Diagnostic Information

    Number of variables: 2

    Functions
    Objective:                            obj
    Gradient:                             finite-differencing
    Hessian:                              finite-differencing (or Quasi-Newton)
    Nonlinear constraints:                con
    Nonlinear constraints gradient:       finite-differencing

    Constraints
    Number of nonlinear inequality constraints: 4
    Number of nonlinear equality constraints:   0

    Number of linear inequality constraints:    0
    Number of linear equality constraints:      0
    Number of lower bound constraints:          2
    Number of upper bound constraints:          2

    Algorithm selected
       active-set

    ____________________________________________________________
       End diagnostic information

                                    Max     Line search  Directional  First-order
     Iter F-count        f(x)   constraint   steplength   derivative   optimality Procedure
        0      3            1            0
        1      9     0.953127            0        0.125           -2         12.5
        2     16     0.809721            0       0.0625        -2.38         10.9
        3     21     0.460457            0         0.25        -12.5         4.38
        4     24     0.355064            0            1        -3.15        0.823
        5     27     0.320358            0            1       -0.939         4.78
        6     30     0.279293            0            1        -5.16         2.43
        7     33       0.2015            0            1       -0.957        0.951
        8     36     0.201081            0            1       -0.586         9.03
        9     39     0.131593            0            1        -2.41        0.716
       10     42     0.102735            0            1        -0.45        0.483
       11     46    0.0746674            0          0.5       -0.373         2.46
       12     50     0.063857            0          0.5       -0.287         2.63
       13     54    0.0526215            0          0.5       -0.627         1.44
       14     58    0.0485194            0          0.5       -0.453        0.746
       15     62    0.0469352            0          0.5       -0.339         0.38
       16     66    0.0462639            0          0.5       -0.279        0.192
       17     70     0.045959            0          0.5       -0.248       0.0963  Hessian modified
       18     74    0.0458143            0          0.5       -0.232       0.0482  Hessian modified
       19     78    0.0457439            0          0.5       -0.224       0.0241  Hessian modified
       20     82    0.0457092            0          0.5        -0.22       0.0121  Hessian modified
       21     86     0.045692            0          0.5       -0.218      0.00604  Hessian modified
       22     90    0.0456834            0          0.5       -0.217      0.00302  Hessian modified
       23     94    0.0456791            0          0.5       -0.217      0.00151  Hessian modified
       24     98    0.0456769            0          0.5       -0.217     0.000755  Hessian modified
       25    102    0.0456759            0          0.5       -0.217     0.000378  Hessian modified

    Optimization stopped because the predicted change in the objective function,
    5.349514e-07, is less than options.FunctionTolerance = 1.000000e-06, and the maximum constraint
    violation, 2.436318e-11, is less than options.ConstraintTolerance = 1.000000e-06.

    Optimization Metric                                                    Options
    abs(steplength*directional derivative) =   5.35e-07          FunctionTolerance =   1e-06 (selected)
    max(constraint violation) =   2.44e-11                     ConstraintTolerance =   1e-06 (selected)

    Active inequalities (to within options.ConstraintTolerance = 1e-06):
      lower      upper     ineqlin   ineqnonlin
                                         1
                                         3
                                         4

We have met the function tolerance and constraint violation. Problem solved!  Let's print out the optimal solution, look at the constraints, and also print out how many iterations this took.

```matlab
xopt
fopt
con(xopt)
output.iterations
```

    xopt =

        0.7864    0.6177

    fopt =

        0.0457

    ans =

        0.0000
       -2.3605

    ans =

        26

Note that the first constraint is active (right at 0 for $c(x) \le 0$).  The second constraint is inactive ($< 0$).  In other words, if we removed this second constraint it would not change the solution at all.

Finally, let's create a convergence plot.  The metric to track is the first order optimality.  We will learn later what this means.  We could just copy/paste the column in the table above under first-order optimality and plot this, but we will also explore a programmatic approach.

In the documentation for fmincon look for output functions.  An output function is of the form:

function stop = outfun(x, optimValues, state)

It is called every iteration.  In output function we will just append the firstorderopt to the end of an array.

```matlab
function stop = outfun(x, optimValues, state)

    firstorderopt = [firstorderopt, optimValues.firstorderopt];
    stop = false;

end
```

In the outer scope we need to initilize the array.  Also we need to set the options so that the optimizer knows there is an output function.

```matlab
firstorderopt = [];
options = optimoptions(options, 'OutputFcn', @outfun);
```

The full script is shown below:

```matlab
function [xopt, fopt, exitflag, output, firstorderopt] = optimize()

    % -------- starting point and bounds ----------
    x0 = [0, 0];
    ub = [5, 5];  % upper bound, the largest values we will allow x to acheive
    lb = [-5, -5];  % lower bound
    % ---------------------------------------------

    % ------ linear constraints ----------------
    A = [];
    b = [];
    Aeq = [];
    beq = [];
    % ------------------------------------------

    % ---- Objective and Constraints -------------
    function [J, c, ceq] = objcon(x)

        % set objective/constraints here
        % objective
        J = (1 - x(1))^2 + 100*(x(2) - x(1)^2)^2;

        % inequality constraints
        c = zeros(2, 1);  % there are 2 of them
        c(1) = x(1)^2 + x(2)^2 - 1;
        c(2) = x(1) + 3*x(2) - 5;

        % equality constraints
        ceq = [];  % there are none

    end
    % -------------------------------------------

    % ----------- options ----------------------------
    options = optimoptions(@fmincon, ...
        'Algorithm', 'active-set', ...  % choose one of: 'interior-point', 'sqp', 'active-set', 'trust-region-reflective'
        'AlwaysHonorConstraints', 'bounds', ...  % forces interior point algorithm to always honor bounds
        'display', 'iter-detailed', ...  % display more information
        'MaxIter', 1000, ...  % maximum number of iterations
        'MaxFunEvals', 10000, ...  % maximum number of function calls
        'TolCon', 1e-6, ...  % convergence tolerance on constraints
        'TolFun', 1e-6, ...  % convergence tolerance on function value
        'FinDiffType', 'forward', ...  % if finite differencing, can also use central
        'GradObj', 'off', ...  % supply gradients of objective
        'GradConstr', 'off', ...  % supply gradients of constraints
        'DerivativeCheck', 'off', ...  % on if you want to check your supplied gradients against finite differencing
        'Diagnostics', 'on');  % display diagnotic information
    % -------------------------------------------

    firstorderopt = [];
    options = optimoptions(options, 'OutputFcn', @outfun);

    function stop = outfun(x, optimValues, state)

        firstorderopt = [firstorderopt, optimValues.firstorderopt];
        stop = false;

    end

    % -- NOTE: no need to change anything below) --

    % ------- shared variables -----------
    xlast = [];
    Jlast = [];  % last objective
    clast = []; % last nonlinear inequality constraint
    ceqlast = []; % last nonlinear equality constraint
    % --------------------------------------

    % ------ separate obj/con  --------
    function [J] = obj(x)

        % check if computation is necessary
        if ~isequal(x, xlast)
            [Jlast, clast, ceqlast] = objcon(x);
            xlast = x;
        end

        J = Jlast;
    end

    function [c, ceq] = con(x)

        % check if computation is necessary
        if ~isequal(x, xlast)
            [Jlast, clast, ceqlast] = objcon(x);
            xlast = x;
        end

        % set constraints
        c = clast;
        ceq = ceqlast;
    end
    % ------------------------------------------------

    % call fmincon
    [xopt, fopt, exitflag, output] = fmincon(@obj, x0, A, b, Aeq, beq, lb, ub, @con, options);
end

```

We now call the script and plot the firstorderopt as a function of iteration on a semilog plot.

```matlab
[xopt, fopt, exitflag, output, firstorderopt] = optimize();

figure;
n = length(firstorderopt);
semilogy(1:n, firstorderopt);
xlabel('iteration');
ylabel('first order optimality')
```


# Stochastic and Deterministic Noise as Catalysts for Overfitting

Small exercise for showcasing the sources of overfitting. Example taken from the 4th Chapter of the book by Abu-Mostafa Yaser, [Learning from data](http://amlbook.com/).

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sn
```

# Set-up of the problem

The goal of the problem is to showcase the role stochastic and deterministic noise play in overfitting.

To this end we are giong to fit polynomials to estimate a polynomial target function. We are going to use Legendre polynomials---an easy-to-handle orthonormal basis on $[-1, 1]$.

The input space is $\mathcal{X} = [-1, 1]$ with uniform probability density.

The target is a degree-$Q_f$ polynomial given by $f(x) = \sum_{q=0}^{Q_f} a_q L_q$

The coefficients are drawn from a rescaled standard normal distribution. We are rescaling the coefficients in order to keep the signal level fixed in expectation, so when we are changing the noise it has a one-to-one effect on the signal-noise ratio.

The rescaling makes use of the closed form expectation we can derive in case of Legendre polynomials and uniform distribution.

\begin{equation}
\mathbb{E}_{a,x} \big[ \ f^2\big ] = \sigma_{a}^2 \Bigg[1 + \frac{1}{3} + \frac{1}{5} + \ldots + \frac{1}{2Q_f + 1}\Bigg] = C
\end{equation}

The data set is $\mathcal{D} = \big\{ (x_1,y_1), \ldots, (x_N,y_N)\big\}$ where $y_n = f(x_n) + \sigma \epsilon_n$ and the $\epsilon_n$ are iid standard normal random variables. $\sigma$ defines the noise level in the problem.

# Generating the data

The following function generates a random target function of degree $Q_f$ and generates a noisy sample of size $N$ with added mean zero, $\sigma$ standard deviation noise.

```python
def overfit_data(Qf, N, sig):
    """Generates a random sample of size N from polynomial function
    of degree Qf with added zero mean, sig s.d. normal noise."""

    # Draw randmom coefficients from rescaled standard normal so that signal level is constant.
    sigma_signal = 2/np.sqrt(sum([1/(2*q + 1) for q in range(Qf)]))
    a = np.random.normal(0, sigma_signal, Qf+1)

    # Generate random input sample and noise
    x = np.random.uniform(-1, 1, N)                # generate a sample of input vectors
    eps = np.random.normal(0, sig, N)              # generate a sample of random noise

    # Generate dependent variable with Legendre poly, and noise
    y = np.polynomial.legendre.legval(x, a) + eps
    return x, y, a
```

# Estimates for in-sample and out-of-sample errors

Taking a given target function with coefficients $a$, the following function computes estimates for the in-sample and out-of-sample errors.

For this end it generates 'num_exper' many experiments with training and test samples of size $N$ fits a 2nd and 10th degree polynomial. Then computes the corresponding errors.

```python
def error(a, N, sig, num_exper):
    """Generates many samples for a given target function with parameters 'a'
    and computes estimates for in- and out-of-sample errors."""

    E_in_2 =  np.zeros(num_exper)  # in-sample error for 2nd degree fit
    E_out_2 = np.zeros(num_exper)  # out-of-sample error for 2nd degree fit
    E_in_10 =  np.zeros(num_exper) # in-sample error for 10th degree fit
    E_out_10 = np.zeros(num_exper) # ou-of-sample error for 10th degree fit

    for i in range(num_exper):

        # Generate sample from specified target function
        x = np.random.uniform(-1, 1, 2*N)
        eps = np.random.normal(0, sig, 2*N)
        y = np.polynomial.legendre.legval(x, a) + eps

        # Split sample to training and test set
        x_train = x[:N]
        y_train = y[:N]
        x_test = x[N:]
        y_test = y[N:]

        # Fit 2nd order polynomial to data
        fit2 = np.polynomial.legendre.legfit(x_train, y_train,2)
        E_in_2[i] =  np.mean((y_train - np.polynomial.legendre.legval(x_train, fit2))**2)
        E_out_2[i] = np.mean((y_test - np.polynomial.legendre.legval(x_test, fit2))**2)

        # Fit 10th order polynomial to data
        fit10 = np.polynomial.legendre.legfit(x_train,y_train,10)
        E_in_10[i] = np.mean((y_train - np.polynomial.legendre.legval(x_train, fit10))**2)
        E_out_10[i] = np.mean((y_test - np.polynomial.legendre.legval(x_test, fit10))**2)

    # Average over experiments to get estimate for the errors
    E_2 = np.array([np.mean(E_in_2), np.mean(E_out_2)])
    E_10 = np.array([np.mean(E_in_10), np.mean(E_out_10)])
    return E_2, E_10
```

# 10th order polynomial noisy target function

Generate small sample of noisy data from 10th degree polynomial target function and try and learn it with 3nd and 10th order polynomials.

```python
np.random.seed(11211)

Qf = 10           # degree of target polynomial
N = 15            # sample size
sig = 1           # level of noise

# Generate data with specified parameters
x, y, a_10 = overfit_data(Qf, N, sig)
model = np.polynomial.legendre.Legendre(a_10) # save instance of Legendre class

# Fit 2nd order polynomial to data
fit2 = np.polynomial.legendre.legfit(x,y,2)
model_fit2 = np.polynomial.legendre.Legendre(fit2) # save instance of Legendre class

# Fit 10th order polynomial to data
fit10 = np.polynomial.legendre.legfit(x,y,10)
model_fit10 = np.polynomial.legendre.Legendre(fit10) # save instance of Legendre class
```

```python
%matplotlib inline

fig, ax = plt.subplots(1, 1, figsize=(14, 10))

plt.rc('text', usetex=True)
plt.rc('font', family='serif')

plt.ylim(-10,10)
plt.xlim(-1,1)
plt.scatter(x,y)
plt.plot(*model.linspace(1000), color='r', label='10th order target')
plt.plot(*model_fit2.linspace(1000), color='g', label='2nd order fit')
plt.plot(*model_fit10.linspace(1000), color='y', label='10th order fit')
plt.legend(bbox_to_anchor=(1.01, 1), loc=2, frameon=False, fontsize = 18)
plt.title('Fitting noisy 10th order target function', {'fontsize':20})
plt.show()
```

Below we can see averaged in-sample and out-of-sample error in the case of fitting a 10th order noisy target function from 2nd and 10th order hypothesis spaces.

```python
num_exper = 100 # number oof experiments to average over the in- and out-of-sample errors

E_2, E_10 = error(a_10, N, sig, num_exper)

print('--------------------------------------------------')
print('               10th order noisy target            ')
print('--------------------------------------------------')
print('      |     2nd order            10th order       ')
print('--------------------------------------------------')
print('E_in  |      {:1.3f}               {:1.3f}        '.format(E_2[0], E_10[0]))
print('E_out |      {:1.3f}               {:.3e}         '.format(E_2[1], E_10[1]))
print('--------------------------------------------------')
```

    --------------------------------------------------
                   10th order noisy target
    --------------------------------------------------
          |     2nd order            10th order
    --------------------------------------------------
    E_in  |      1.429               0.286
    E_out |      2.577               4.487e+09
    --------------------------------------------------

As expected the 10th order fit delivers small in-sample error, however performs terribly out-of-sample.

# Learning Curves

Since we were dealing with consistent estimation methods as the sample size increases the in- and out-of-sample errors should converge to the true risk within the corresponding hypothesis spaces.

\begin{equation*}
R(\hat{f}_n) - R(f_{\mathcal{F}}) = \underbrace{R(\hat{f}_n) - R(f_{\mathcal{H}})}_{\text{estimation error}} \ + \ \underbrace{R(f_{\mathcal{H}}) - R(f_{\mathcal{F}})}_{\text{approximation error}}
\end{equation*}

## Learning Curve for learning the 10th order noisy target

```python
N_start = 15
N_stop = 150
step = 2

num_steps = len(range(N_start, N_stop , step))

E_2 = np.zeros((2,num_steps))
E_10 = np.zeros((2,num_steps))
sample_size = np.zeros(num_steps)

for index, n in enumerate(range(N_start, N_stop , step)):
    E_2[:,index], E_10[:,index] =  error(a_10, n, 1, 2000)
    sample_size[index] = n
```

```python
fig, ax = plt.subplots(1, 1, figsize=(14, 10))

plt.rc('text', usetex=True)
plt.rc('font', family='serif')

plt.plot(sample_size, E_2[0], color='g', label='E in -- 2nd')
plt.plot(sample_size, E_2[1], color='y', label='E out -- 2nd')
plt.plot(sample_size, E_10[0], color='b', label='E in -- 10th')
plt.plot(sample_size, E_10[1], color='r', label='E out -- 10th')
plt.ylim(0,5)
plt.ylabel('Expected error', fontsize = 20)
plt.xlabel('Sample size', fontsize = 20)
plt.legend(bbox_to_anchor=(1.01, 1), loc=2, frameon=False, fontsize = 18)
plt.title('Learning Curves with 10th Order Noisy Target', {'fontsize':24})
plt.show()
```

We see how for smaller sample sizes the restricted learner---2nd degree---outperforms the one using the true 10th degree functional form.

Surprisingly, even if we know the form of the truth we are better off with restricting ourselves in finite samples in order to achieve better generalization.

# 50th order noiseless target

Let's see how the deterministic noise affects the performance of the different model.

```python
np.random.seed(11206)

Qf = 50           # degree of target polynomial
N = 15            # sample size
sig = 0.001           # level of noise

# Generate data with specified parameters
x, y, a_50 = overfit_data(Qf, N, sig)
model = np.polynomial.legendre.Legendre(a_50)

# Fit 2nd order polynomial to data
fit2 = np.polynomial.legendre.legfit(x,y,2)
model_fit2 = np.polynomial.legendre.Legendre(fit2)

# Fit 10th order polynomial to data
fit10 = np.polynomial.legendre.legfit(x,y,10)
model_fit10 = np.polynomial.legendre.Legendre(fit10)
```

```python
fig, ax = plt.subplots(1, 1, figsize=(14, 10))

plt.rc('text', usetex=True)
plt.rc('font', family='serif')

plt.scatter(x,y)
plt.plot(*model.linspace(1000), color='r', label='50th order target')
plt.plot(*model_fit2.linspace(1000), color='g', label='2nd order fit')
plt.plot(*model_fit10.linspace(1000), color='y', label='10th order fit')
plt.ylim(-10,10)
plt.xlim(-1,1)
plt.legend(bbox_to_anchor=(1.01, 1), loc=2, frameon=False, fontsize = 18)
plt.title('Fitting noiseless 50th order target function', {'fontsize':20})
plt.show()
```

```python
E_2, E_10 = error(a_50, N, sig, num_exper)

print('--------------------------------------------------')
print('            5oth order noiseless target           ')
print('--------------------------------------------------')
print('fit   |     2nd order            10th order       ')
print('--------------------------------------------------')
print('E_in  |      {:1.3f}               {:1.3f}        '.format(E_2[0], E_10[0]))
print('E_out |      {:1.3f}               {:.3e}         '.format(E_2[1], E_10[1]))
print('--------------------------------------------------')
```

    --------------------------------------------------
                5oth order noiseless target
    --------------------------------------------------
    fit   |     2nd order            10th order
    --------------------------------------------------
    E_in  |      0.851               0.145
    E_out |      1.504               2.205e+11
    --------------------------------------------------

Again we see that the 10th order fit delivers better in-sample error on the expense of a terrible out-of-sample performance.

### Learning curve for learning the 50th order noiseless target

```python
N_start = 15
N_stop = 150
step = 2

num_steps = len(range(N_start, N_stop , step))

E_2 = np.zeros((2,num_steps))
E_10 = np.zeros((2,num_steps))
sample_size = np.zeros(num_steps)

for index, n in enumerate(range(N_start, N_stop , step)):
    E_2[:,index], E_10[:,index] =  error(a_50, n, .001, 2000)
    sample_size[index] = n
```

```python
fig, ax = plt.subplots(1, 1, figsize=(14, 10))

plt.rc('text', usetex=True)
plt.rc('font', family='serif')

plt.plot(sample_size, E_2[0], color='g', label='E in -- 2nd')
plt.plot(sample_size, E_2[1], color='y', label='E out -- 2nd')
plt.plot(sample_size, E_10[0], color='b', label='E in -- 10th')
plt.plot(sample_size, E_10[1], color='r', label='E out -- 10th')
plt.ylim(0,5)
plt.ylabel('Expected error', fontsize = 20)
plt.xlabel('Sample size', fontsize = 20)
plt.legend(bbox_to_anchor=(1.01, 1), loc=2, frameon=False, fontsize = 18)
plt.title('Learning Curves with 50th Order Noiseless Target', {'fontsize':24})
plt.show()
```

```python

```


# PSO

En informática, la optimización por nube de partículas u optimización por enjambre de partículas (conocida por sus siglas en inglés: PSO, de «particle swarm optimization») hace referencia a una metaheurística que evoca el comportamiento de las partículas en la naturaleza.

Los métodos PSO se atribuyen originalmente a los investigadores Kennedy, Eberhart1 y Shi.2 En un principio fueron concebidos para elaborar modelos de conductas sociales,3 como el movimiento descrito por los organismos vivos en una bandada de aves o un banco de peces. Posteriormente el algoritmo se simplificó y se comprobó que era adecuado para problemas de optimización. El libro de Kennedy y Eberhart4 describe numerosos aspectos teóricos de la PSO y la inteligencia de enjambre. Un amplio estudio de las aplicaciones de PSO se puede encontrar en Poli.56

PSO permite optimizar un problema a partir de una población de soluciones candidatas, denotadas como "partículas", moviendo éstas por todo el espacio de búsqueda según reglas matemáticas que tienen en cuenta la posición y la velocidad de las partículas. El movimiento de cada partícula se ve influido por su mejor posición local hallada hasta el momento, así como por las mejores posiciones globales encontradas por otras partículas a medida que recorren el espacio de búsqueda. El fundamento teórico de esto es hacer que la nube de partículas converja rápidamente hacia las mejores soluciones.

PSO es una metaheurística, ya que asume pocas o ninguna hipótesis sobre el problema a optimizar y puede aplicarse en grandes espacios de soluciones candidatas. Sin embargo, como toda metaheurística, PSO no garantiza la obtención de una solución óptima en todos los casos.

```python
#!pip install --upgrade pyswarm
```

```python
%matplotlib inline
from pyswarm import pso
from sympy import *
import numpy as np
import matplotlib.pyplot as plt

init_printing()
```

```python
def fV(x):
    return x**2

lb = [3]
ub = [5]

xopt, fopt = pso(fV, lb, ub)
xopt,fopt

```

    Stopping search: maximum iterations reached --> 100

    (array([ 3.]), array([ 9.]))

## Ejemplo

Una caja rectangular sin tapa se hace con $12 m^2$ de cartón.

Calcule el volumen máximo de esta caja.

Solucion:

Sean $x, y y z$ el largo, el ancho
y la altura, respectivamente, de la caja medidos en metros. Buscamos maximizar

$$V=xyz $$

sujeta a la restricción

$$g(x,y,z) = 2xz + 2yz + xy = 12$$

```python
x,y,z,ld = symbols("x,y,z,lambda")
```

```python
L = x*y*z + ld*(2*x*z + 2*y*z + x*y - 12)
pcs= solve(map(L.diff,[x,y,z,ld]))
pcs
```

```python
L.subs(pcs[0]),L.subs(pcs[0])
```

```python
def fV(X):
    x = X[0]
    y = X[1]
    z = X[2]
    return -x*y*z

def con(X):
    x = X[0]
    y = X[1]
    z = X[2]
    return (2*x*z + 2*y*z + x*y -12)*-1

lb = [0,0,0]
ub = [5,5,5]

#xopt, fopt = pso(fV, lb, ub, f_ieqcons=con,swarmsize=100,maxiter=100)
xopt, fopt = pso(fV, lb, ub, f_ieqcons=con)
xopt,fopt
```

    Stopping search: Swarm best objective change less than 1e-08

    (array([ 1.96045277,  1.92423486,  1.05898198]), -3.9948735026620716)

Nota:
- PSO minimiza entonces cambie la funcion a -f(x) y las restricciones de signo
- el fopt da negativo por que cambie negativo el signo.

```python

```

```python

```

## Ejemplo

Un agricultor tiene $2 400$ metros de material y quiere construir una barda para
cercar un campo rectangular que bordea un río recto, de modo que no necesita barda a
lo largo del río.

¿Cuáles son las dimensiones que debe tener el campo para encerrar el
área más grande?

### Solución

- Queremos maximizar el área $A$ del rectángulo.

$$A(x,y) = xy $$

Ademas tenemos una restriccion, utilizamos la información dada de que la longitud total de la barda es $2400$ metros

$$2x + y = 2400 $$

```python
L = x*y + ld*(2*x + y - 2400)
pcs= solve(map(L.diff,[x,y,ld]))
L.subs(pcs),(x*y).subs(pcs),pcs
```

```python

```

```python
def fV(X):
    return - X[0]*X[1]

def con(X):
    return  -(2*X[0] + X[1] - 2400)

lb = [0,0]
ub = [5000,5000]

xopt, fopt = pso(fV, lb, ub, f_ieqcons=con)
xopt,fopt
```

    Stopping search: maximum iterations reached --> 100

    (array([  592.93321599,  1214.13356731]), -719900.12069964292)

```python

```

## Ejemplo

Se va un fabricar una lata que ha de contener $1 L$ de aceite. Encuentre
las dimensiones que debe tener la lata de manera que minimicen el costo del metal para fabricarla.

De esta manera, el área superficial es
$$A(r,h) = 2\pi r^2 + 2\pi r h  $$

El volumen está dado como $1 L$, que tomamos como $1000$ $cm^3$

$$\pi r^2h=1000 $$

```python
# Multiplicadores de lagrange
r,h= symbols("r,h",real=True)
L = 2*pi*r**2 + 2*pi*r*h  +ld*(pi*r**2*h -1000)
pcs= solve(map(L.diff,[r,h,ld]))
pcs
L.subs(pcs[0]),N ( (2*pi*r**2 + 2*pi*r*h).subs(pcs[0]) ),pcs
```

```python
N(pcs[0][h]),N(pcs[0][r])
```

```python

```

```python
def fV(X):
    return (2*np.pi*X[0]**2  + 2*np.pi*X[0]*X[1])

def con(X):
    return  (np.pi*X[0]**2*X[1] - 1000)

lb = [0,0]
ub = [500,500]

xopt, fopt = pso(fV, lb, ub, f_ieqcons=con)
xopt,fopt
```

    Stopping search: maximum iterations reached --> 100

    (array([  5.39618166,  10.93143068]), 553.59111372575342)

```python

```

# SwarmPackagePy

Library of swarm optimization algorithms.

SwarmPackagePy is the package, witch contains the following swarm optimization algorithms:

- Artificial Bee Algorithm
- Bat Algorithm
- Bacterial Foraging Optimization
- Cat Swarm Optimization
- Chicken Swarm Optimization
- Cuckoo Search Optimization
- Firefly algorithm
- Firework Algorithm
- Gravitational Search Algorithm
- Grey Wolf Optimizer
- Harmony Search
- Particle Swarm Optimization
- Social Spider Algorithm
- Whale Swarm Algorithm

Every algorithm has arguments listed below:

- n: number of agents
- function: test function
- lb: lower limits for plot axes
- ub: upper limits for plot axes
- dimension: space dimension
- iteration: number of iterations

Every algorithm has methods listed below:

- get_agents(): returns a history of all agents of the algorithm
- get_Gbest(): returns the best position of algorithm

```python
#!pip install SwarmPackagePy
```

```python
import SwarmPackagePy
```

```python
def fV(X):
    return (2*np.pi*X[0]**2  + 2*np.pi*X[0]*X[1])

def con(X):
    return  (np.pi*X[0]**2*X[1] - 1000)

lb = [0,0]
ub = [500,500]

xopt, fopt = pso(fV, lb, ub, f_ieqcons=con)
xopt,fopt
```

    Stopping search: maximum iterations reached --> 100

    (array([  5.42447867,  10.8176797 ]), 553.58155997221195)

```python
SwarmPackagePy.pso?
```

```python
def fV(X):
    return (2*np.pi*X[0]**2  + 2*np.pi*X[0]*X[1]) + X[2]*(np.pi*X[0]**2*X[1] - 1000)

lb = [0,0,0]
ub = [500,500,500]

methodPSO = SwarmPackagePy.pso(100,fV,lb=lb,ub=ub,dimension=3,iteration=1000)
methodPSO.get_Gbest()
```

```python

```

```python

```

```python
import SwarmPackagePy
from SwarmPackagePy import testFunctions as tf
from SwarmPackagePy import animation, animation3D

alh = SwarmPackagePy.pso(50, tf.easom_function, -10, 10, 2, 20,
                         w=0.5, c1=1, c2=1)
animation(alh.get_agents(), tf.easom_function, -10, 10)
animation3D(alh.get_agents(), tf.easom_function, -10, 10)
```

```python

```


# Convolutional Neural Networks from Scratch

In this notebook we are going to take a look at how to implement a simple CNN model from scratch in Python, using mostly just `numpy`.

In practice, we can use high-level libraries such as Keras or PyTorch to abstract away the underlying details of CNN when writing code. However, we find that the exercise of writing one from scratch is very helpful in gaining a deeper understanding of CNNs and how these frameworks work under the hood.

Let's get started!

## Convolutions

We won't have a CNN without some convolutional layers! Let's start by implementing a convolutional layer class. All layers that we are going to implement in this notebook will be defined by a simple interface: all of them will have a `forward` and `backward` method for the forward and backward pass respectively.

This will be easier to explain once we have some code to refer to. So here it is:

```python
import numpy as np

class Conv:
  def __init__(self, in_channels, out_channels, filter_size, pad = 0, stride = 1, weight_scale = 1e-3):
    self.pad = pad
    self.stride = stride
    # Weights initialized from 0-centered Gaussian with standard deviation
    # equal to `weight_scale` (by default set to 1e-3) for stability
    self.filters =  np.random.randn(out_channels, in_channels,
                                     filter_size, filter_size) * weight_scale
    # biases initialized to zero
    self.bias = np.zeros((out_channels, ))

  def calc_output_dim(self, x):
    _, H, W = x.shape
    _, _, filter_height, filter_width = self.filters.shape

    H_out = 1 + (H + 2 * self.pad - filter_height) // self.stride
    W_out = 1 + (W + 2 * self.pad - filter_width) // self.stride

    return H_out, W_out

  def iterate_regions(self, x):
    H_out, W_out = self.calc_output_dim(x)
    F, C, filter_height, filter_width = self.filters.shape

    for i in range(H_out):
      for j in range(W_out):
        x_region = x[:, i * self.stride : i * self.stride + filter_height, j * self.stride : j * self.stride + filter_width]
        yield i, j, x_region

  def forward(self, x):
    """
    Inputs:
    - x: An ndarray containing input data, of shape (C, H, W)
    Returns:
    - out: Result of convolution, of shape (F, H', W')
    where F is the number of out channels and
    H' = 1 + (H + 2 * pad - filter_height) / stride
    W' = 1 + (W + 2 * pad - filter_width) / stride
    """

    self.x = x  # save as cache

    _, H, W = x.shape
    F, _, _, _ = self.filters.shape

    # to pad last two dimensions of the four dimensional tensor
    x_pad = np.pad(x, self.pad, mode='constant')

    H_out, W_out = self.calc_output_dim(x)
    out = np.zeros((F, H_out, W_out))

    for f in range(F):
      # convolve F times to get F x filter_height x filter_width
      kernel = self.filters[f] # dimension is C x filter_height x filter_width
      for i, j, x_region in self.iterate_regions(x_pad):
        out[f, i, j] = (x_region * kernel).sum()

      out[f] += self.bias[f] # makes use of broadcasting

    return out

  def backward(self, dout):
    """
    Inputs:
    - dout: An ndarray of the upstream derivatives, of shape (F, H', W')
    Returns a tuple of:
    - dx: Gradient w.r.t input x, of shape (C, H, W)
    - dw: Gradient w.r.t. filters, of shape (F, C, filter_height, filter_width)
    - db: Gradient w.r.t the biases, of shape (F, )
    """

    _, H, W = self.x.shape # retrieve from cache
    F, C, HH, WW = self.filters.shape

    x_pad = np.pad(self.x, self.pad, mode='constant')

    dx = np.zeros_like(x_pad)
    dw = np.zeros_like(self.filters)
    db = np.zeros_like(self.bias)

    for f in range(F):
      for i, j, x_region in self.iterate_regions(x_pad):
        dw[f] += dout[f, i, j] * x_region

        #-------- Task 1 : Compute dx below -----------------#

        #---------------- End of task 1 ---------------------#

      #-------- Task 2 : Compute db below -----------------#

      #---------------- End of task 2 ---------------------#

    # trim off padding so that dimension of dx is the same as x
    dx = dx[:, self.pad: self.pad + H, self.pad : self.pad + W]

    return dx, dw, db
```

### Convolutional layer: forward pass

In the `forward` call, we pass in the input image `x`, which will be a numpy ndarray of dimensions `C x H x W` where `C` is the number of channels (3 for RGB) and `H` and `W` are the height and weight of the image respectively.

The first thing we do is to cache the input so that we have access to it during the backpropagation:

```python
 self.x = x  # save as cache
```

Following that, we pad the image with the specified padding, using 0 as the value for the padded cells. To do this, we make use of numpy's `np.pad` method:

```python
x_pad = np.pad(x, self.pad, mode='constant')
```

Following this, we define `out`, which is the tensor / ndarray we will return from `forward` as the result of the convolution. This will be a three-dimensional tensor with dimensions `F x H_out x W_out`, where `F` is the number of filters we specified. We calculate `H_out` and `W_out` using a formula which we extract out to a separate helper function, `calc_output_dim()`.

```python
H_out, W_out = self.calc_output_dim(x)
out = np.zeros((F, H_out, W_out))
```

Next comes the main logic for calculating the result of the convolutional operation:

```python
for f in range(F):
      # convolve F times to get F x filter_height x filter_width
      kernel = self.filters[f] # dimension is C x filter_height x filter_width
      for i, j, x_region in self.iterate_regions(x_pad):
        out[f, i, j] = (x_region * kernel).sum()
```

For each filter, we calculate the result of multiplying the filter (`filter` is a reserved keyword in Python, so we use the variable name `kernel`) with each region in the image of the same dimension as the filter (recall the CNN theory lecture), and set the output to be returned. Here we make use of the helper function `iterate_regions()`, which is a generator that `yield`s regions of the image `x_pad` which are of the same dimensions as the kernel (keeping the stride into account, as well).

### Convolutional layer: backward pass

In the backward pass, we pass in `dout`, which are the upstream gradients which are being propagated from the next layer. In particular, `dout` is the partial derivative of our loss function (in the context of classifying digits in MNIST with CNNs, this would be the cross-entropy loss w/ softmax) with respect to the output of `forward`, which is a 3-dimensional tensor of dimensions `F x H_out x W_out`, as we previously calculated.

Given `dout` and the cached input, our job is to calculate `dx`, `dw`, `db`, which are the partial derivatives of the loss with respect to the input, filters (`w` stands for weights), and the biases respectively. The dimensions of each of these will be the same as the dimensions of the variables we are taking the derivative with respect to. In `backward()`, we return these derivatives as a tuple. `dx` is used as the `dout` for the previous layer (recall how backpropagation works!), while `dw` and `db` will be used to update the filter weights and biases respectively via gradient descent. The details of how to do this are omitted from this notebook to keep it short, but the implementation should be relatively straightforward once you have these gradients.

Now let's look at the backpropagation in greater detail:

#### Theory

Let's see how we can calculate `dw` first. To recap, `dw` is the gradient of the loss w.r.t the filter weights $w$ :  $\frac{\partial L}{\partial w}$.

Let the output of conv (variable `out` from the forward pass) be $o$. Using the **chain rule**, we get: $\frac{\partial L}{\partial w} = \frac{\partial L}{\partial o} * \frac{\partial o}{\partial w}$. We already know $\frac{\partial L}{\partial o}$ — this is the upstream gradient `dout`. Now we only need to calculate $\frac{\partial o}{\partial w}$, which is the gradient of the output w.r.t to the filter weights. Let's think about how to calculate this.

Imagine we have a `3 x 3` image $x$ and a `2 x 2` kernel filled with zeros. The result of the convolution operation is a `2 x 2` output also filled with zeros (rightside of the arrow). We assume no padding and a stride of 1:

Now let's imagine what would happen if we *nudge* the value of the top left cell of the filter, such that it becomes a 1. How would the output change? After all, this is essentially what $\frac{\partial o}{\partial w}$ is telling us. Let's look at the updated diagram:

Can you see the pattern? We can see that the output changed by exactly the image pixel values that the changed filter weight is multiplied with during the convolution operation. In other words, the derivative of a single pixel of our output $o$ with respect to a filter weight is just the corresponding pixel value in the image (that this filter weight is multiplied by).

More formally, if we let $o$ to be $m$ x $n$ and the filter to be $w$ x $h$, then the $i^{th}$ row and $j^{th}$ column of the output $o$ (which we denote by $o[i,j]$) can be expressed as:

\begin{equation}
o[i,j] = \sum_{r=0}^{w} \sum_{c=0}^{h} x[i + r, j + c] * filter[r,c]
\end{equation}

and as we've just seen, the partial derivative of this term w.r.t a specific filter weight is given by:

\begin{equation}
\frac{\partial o[i,j]}{\partial filter[r,c]} = x[i + r, j + c]
\end{equation}

We now have everything we need to obtain the gradient of the loss with respect to the filter weight indexed by $[r,c]$, $\frac{\partial L}{\partial filter[r,c]}$, using the chain rule:

\begin{equation}
\frac{\partial L}{\partial filter[r,c]} = \sum_{i=0}^{m} \sum_{j=0}^{n} \frac{\partial L}{\partial o[i,j]} * \frac{\partial o[i,j]}{\partial filter[r,c]} = \sum_{i=0}^{m} \sum_{j=0}^{n} \frac{\partial L}{\partial o[i,j]} * x[i + r, j + c]
\end{equation}

We can compute the derivative of the loss w.r.t `x` similarly, where instead of the pixel value of the image it's the corresponding filter weight. We'll leave this as an exericse for you!

Lastly, let's quickly go through how we compute `db`, the derivative of the loss w.r.t the biases. In our example, the biases are passed in as an numpy array of shape `(F, )` where `F` is the number of filters. Therefore, `db` will have the same dimension.

Given a single filter, we have a scalar value `b` being added to each 'pixel' of the output. Because this is just simply addition, the derivative of a pixel of the output w.r.t to the bias term `b` ( $\frac{\partial o[r,c]}{\partial b}$) is 1. This means that the derivative for the bias term for each filter is simply the sum of `dout` for that filter.

\begin{equation}
\frac{\partial L}{\partial b} = \sum_{i=0}^{m} \sum_{j=0}^{n} \frac{\partial L}{\partial o[i,j]} * \frac{\partial o[i,j]}{\partial b} = \sum_{i=0}^{m} \sum_{j=0}^{n} \frac{\partial L}{\partial o[i,j]}
\end{equation}

#### Implementation

Now that we've covered the theory of how backpropagation works for convolutions, let's see how we translate this over to code:

After extracting the dimensions we need and padding the input, we initialize `dx`, `dw` and `db` with zeros:

```python
dx = torch.zeros_like(x_pad)
dw = torch.zeros_like(self.filters)
db = torch.zeros_like(self.bias)
```

For each filter, we iterate over each image region and accumulate the loss gradients for `dw` and `dx`. As we just saw, the gradient for the bias term for a particular filter is just the sum of `dout` for that filter. Note that for `dx`, we need to trim off the padding such that the dimensions of `dx` is the same as the original input we passsed into `forward()`.

```python
for f in range(F):
  for i, j, x_region in self.iterate_regions(x_pad):
    dw[f] += dout[f, i, j] * x_region

    #-------- Task 1 : Compute dx below -----------------#

    #---------------- End of task 1 ---------------------#

  #-------- Task 2 : Compute db below -----------------#

  #---------------- End of task 2 ---------------------#

# trim off padding so that dimension of dx is the same as x
dx = dx[:, self.pad: self.pad + H, self.pad : self.pad + W]
```

You may notice that we have purposefully left out the code that computes `dx` and `db`. Try implementing them before checking your answers against our solution notebook. You can run the next few cells below to quickly test out your implementation.

#### (Aside) Quick sanity check

The cells below serve as a quick sanity check to make sure that you aren't getting any errors in your implementation and that the methods are returning ndarrays of the correct shape.

```python
dout = np.random.randn(8, 26, 26)
dx, dw, db = conv.backward(dout)

print("dx shape: ", dx.shape)
print("dw shape: ", dw.shape)
print("db shape: ", db.shape)
```

    dx shape:  (3, 28, 28)
    dw shape:  (8, 3, 3, 3)
    db shape:  (8,)

```python
# create dummy image
img = np.random.randn(3, 28, 28) # assume 28 x 28 RGB image

conv = Conv(in_channels=3, out_channels=8, filter_size=3)
out = conv.forward(img)

out.shape
```

    (8, 26, 26)

## Pooling layers : max pool

Besides conv, another key component in CNNs is the pooling layers. In this notebook, we will show you how to implement a max pooling layer from scratch, again using nothing but `numpy`. Here is the code:

```python
class MaxPool2x2:
  def forward(self, x):
    """
    Inputs:
    - x: An ndarray containing input data, of shape (C, H, W)
    Returns:
    - out: Result of max pooling operation, of shape (C, H', W') where
    H' = 1 + (H - pooling height) // stride
    W' = 1 + (H - pooling width) // stride
    """
    self.x = x # cache
    C, H, W = x.shape
    out = np.zeros((C, H // 2, W // 2))

    for c in range(C):
      for i in range(H // 2):
        for j in range(W // 2):
          out[c, i, j] = np.max(x[c, i*2:i*2 + 2, j*2:j*2 + 2])

    return out

  def backward(self, dout):
    """
    Inputs:
    - dout: Upstream gradients, of shape (C, H', W')
    Returns:
    - dx: Gradient w.r.t input x
    """
    C, H, W = self.x.shape

    dx = np.zeros_like(self.x) # C x H x W

    for c in range(C):
      for i in range(H // 2):
        for j in range(W // 2):
          window = self.x[c, i*2:i*2 + 2, j*2:j*2 + 2]
          max_idx = np.argmax(window)
          # window.shape is (2, 2)
          max_r = max_idx // window.shape[1]
          max_c = max_idx % window.shape[1]
          dx[c, i*2:i*2+2, j*2:j*2+2][max_r, max_c] = dout[c, i, j]

    return dx

```

#### (Aside) See it in action

Below is the code that demonstrates how `forward()` and `backward()` work for the max pooling layer.

```python
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.utils import to_categorical

(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()

x_train = x_train.astype("float32") / 255
x_test = x_test.astype("float32") / 255

x_train = x_train[:, np.newaxis, :,:]
x_test = x_test[:, np.newaxis,:,:]

y_train = to_categorical(y_train, 10)
y_test = to_categorical(y_test, 10)

img = x_train[0]
label = y_train[0]
```

```python
img[:,16:20,16:20]
```

    array([[[0.3647059 , 0.9882353 , 0.99215686, 0.73333335],
            [0.        , 0.9764706 , 0.99215686, 0.9764706 ],
            [0.7176471 , 0.99215686, 0.99215686, 0.8117647 ],
            [0.99215686, 0.99215686, 0.98039216, 0.7137255 ]]], dtype=float32)

```python
pool = MaxPool2x2()
out = pool.forward(img[:,16:20,16:20])
out
```

    array([[[0.98823529, 0.99215686],
            [0.99215686, 0.99215686]]])

```python
dx = pool.backward(out)
dx
```

    array([[[0.        , 0.9882353 , 0.99215686, 0.        ],
            [0.        , 0.        , 0.        , 0.        ],
            [0.        , 0.99215686, 0.99215686, 0.        ],
            [0.        , 0.        , 0.        , 0.        ]]], dtype=float32)

### Max pooling layer: forward pass

During the forward pass, we pass in `x`, which really is $o$ from the previous convolutional layer. `x` will be a three-dimensional tensor of shape `C x H x W`.

Here, we decided to simplify the code by specifying that this is a `2x2` max pooling layer. Handling different pool widths and heights isn't too difficult, and we'll leave this as an exercise for you.

The forward pass code is simple. We simply iterate through each image region similarly to convolutional layers and take the max element in each region. The difference with the convolutional layer is that the image regions are non-overlapping. We also remember to cache the input `x` so that we can refer to it in the backpropagation phase.

### Max pooling layer: backward pass

During the backward pass, we receive `dout`, similar to convolutional layers. `dout` will represent the derivative of the loss with respect to the output of the max pooling layer and thus will be of shape `C x H x W`, where really `H` here is the original H divided by 3 (as a result of forward pass).

Remember that max pooling layers have no learnable weights! Our aim with backpropagation here is to simply compute `dx`, so that we can pass it as `dout` for the previous layer (most likely a ReLU layer) so that gradients propagate through our entire model and weights are updated.

Keeping this in mind, let's think about how to compute $\frac{\partial o}{\partial x}$, where again we denote the result of max pooling as $o$ (remember that if we have  $\frac{\partial o}{\partial x}$, we can compute $\frac{\partial L}{\partial x}$ because we already have $\frac{\partial L}{\partial o}$). During the forward pass, we take the max value in every `2 x 2` region. Therefore, intuitively, every other value in the region that is not the max will not change the output $o$. These values therefore have 0 gradient. For the max value, we simply assign the corresponding gradient value. Below is an illustration ([source](https://medium.com/@eternalzer0dayx/demystifying-convolutional-neural-networks-ca17bdc75559)):

The code we have written for backprop is simply translating this idea to code:

```python
for c in range(C):
      for i in range(H // 2):
        for j in range(W // 2):
          window = x[c, i*2:i*2 + 2, j*2:j*2 + 2]
          max_idx = np.argmax(window)
          max_r = max_idx // window.shape[1]
          max_c = max_idx % window.shape[1]
          dx[c, i*2:i*2+2, j*2:j*2+2][max_r, max_c] += dout[c, i, j]
```

One thing to keep in mind is that `np.argmax()` returns an index instead of (row, col) so we have to convert it appropriately, then index into it to incrementally build up `dx`.

## Softmax

To complete our implementation of a CNN, we need a fully connected layer that flattens the tensor we get from the last conv layer (most likely with non-linearity and pooling applied) into a $k$-dimensional vector where $k$ is the number of classes. For our MNIST classifier, we have digits, so $k = 10$.

We also apply a **softmax** activation to the predictions to convert them into probabilities that add up to 1. To convert these probabilities into a loss we can use for training, we use **cross-entropy loss**, given by the following formula:

\begin{equation}
L=−log(p_c)
\end{equation}

where $p_c$ is the probability of the correct class.

As this notebook is about CNNs, we won't get into too much detail for softmax / cross-entropy loss. Those who are interested can check out this [excellent blog post](https://eli.thegreenplace.net/2016/the-softmax-function-and-its-derivative/) from Eli Bendersky.

Here are the implementations for a fully-connected linear layer as well as a function computing loss and derivatives for softmax loss.

```python
class Linear:
  def __init__(self, in_features, out_features, weight_scale = 1e-3):
    # Initialise weight matrix of shape (in_features, out_features)
    self.weights = np.random.randn(in_features, out_features) * weight_scale
    self.bias = np.zeros(out_features)

  def forward(self, x):
    """
    Inputs:
    - x: An ndarray containing input data, of shape (d_1, ..., d_k) such that
      D = d_1 * ... * d_k is equal to `in_features`
    Returns:
    - out: output, of shape (1, out_features)
    """
    self.x_before_flatten = x # cache
    reshaped_x = x.flatten()[np.newaxis, :] # shape (1, D)
    self.x_after_flatten = reshaped_x # cache

    # @ is the matrix multiplication operator in Python
    # reshaped_x's shape: (1 , D)
    # self.weight's shape: (D , M) where M is out_features
    # result of reshaped_x @ self.weights is of shape (1, M), so (1,10) for MNIST
    out = reshaped_x @ self.weights + self.bias
    return out

  def backward(self, dout):
    """
    Inputs:
    - dout: An ndarray of the upstream derivatives, of shape (1, M) where M = out_features
    Returns a tuple of:
    - dx: Gradient w.r.t input x, of shape (d_1, ..., d_k)
    - dw: Gradient w.r.t. weights w, of shape (D, M)
    - db: Gradient w.r.t the biases, of shape (M, )
    """
    dx = (dout @ self.weights.T).reshape(*self.x_before_flatten.shape)
    dw = self.x_after_flatten.T @ dout # (D x 1) * (1 x M)
    db = dout.squeeze() # turns (1, M) to (M, )
    return dx, dw, db
```

```python
def softmax_loss(scores, y):
  """
  Inputs:
  - scores: ndarray of scores, of shape (N, C) where C is the no. of classes
  - y: ndarray of class labels, of shape (N,) where y[i] is the label for x[i]
  Returns a tuple of:
  - loss: Scalar value representing the softmax loss
  - dscores: Gradient of loss w.r.t scores, of shape (N, C)
  """
  loss = None, dscores = None
  #----- Task 3 : Compute softmax loss and dx ---------#
  # Hint: Implement https://eli.thegreenplace.net/2016/the-softmax-function-and-its-derivative/
  # Remember to take care of numerical stability when doing exponentiation:
  # See https://cs231n.github.io/linear-classify/#softmax-classifier

  #---------------- End of task 3 ---------------------#

  return loss, dscores

```

### Putting it together

Once we have the core components, we can put together the classes we've written so far to create a CNN model:

```python
conv = Conv(1, 8, 3)  # 28x28x1 -> 26x26x8
pool = MaxPool2x2() # 26x26x8 -> 13x13x8
linear = Linear(13 * 13 * 8, 10)  # 13x13x8 -> 10

def forward(image, y):
  '''
    Completes a forward pass of the CNN
    - image is a numpy ndarray of shape (1 x 28 x 28) representing the greyscale image
    - y is the corresponding digit
  '''
  # transform the image from [0, 255] to [0, 1]
  out = conv.forward((image / 255))
  out = pool.forward(out)
  scores = linear.forward(out)

  loss, dout = softmax_loss(scores, y)

  return scores, loss, dout

def loss(image, label):
  '''
    Computes loss + completes a backward pass of the whole model
  '''
  grads = {}
  # forward pass
  scores, loss, dout = forward(image, label)

  # Backprop
  dpool, grads['W_linear'], grads['b_linear'] = linear.backward(dout)
  dconv = pool.backward(dpool)
  dx, grads['W_conv'], grads['b_conv'] = conv.backward(dconv)

  return loss, grads

# Challenge yourself: can you try to train the model?
```

In this notebook, we omitted the training part because we wanted to focus on what each component is doing — can you come up with the code to train the CNN model below on the MNIST dataset? Challenge yourself!

## CNN in Keras

So far we've walked you through how to implement a CNN from scratch in Python. Fortunately, we practically never have to write a CNN from scratch nowadays, thanks to high-level libraries like Keras that help abstract all the underlying complexity away from us. To cap off this notebook, let's use Keras to implement the **same** CNN that we've implemented above. We adapted this example from the [official Keras MNIST CNN example](https://keras.io/examples/vision/mnist_convnet/).

```python
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.utils import to_categorical

# split data
(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()

# scale images
x_train = x_train.astype("float32") / 255
x_test = x_test.astype("float32") / 255

# Make sure images have shape (28, 28, 1)
x_train = x_train[:,:,:, np.newaxis]
x_test = x_test[:,:,:, np.newaxis]

print("x_train shape:", x_train.shape)
print(x_train.shape[0], "train samples")
print(x_test.shape[0], "test samples")
```

    Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz
    11493376/11490434 [==============================] - 0s 0us/step
    11501568/11490434 [==============================] - 0s 0us/step
    x_train shape: (60000, 28, 28, 1)
    60000 train samples
    10000 test samples

A few things worth pointing out in the code above:

First, once we load the MNIST dataset, we scale the images so that each pixel is in the [0,1] range instead of [0,255]. This makes it easier for us to train the CNN.

Next, we add a fourth dimension to the `x_train` and `x_test` tensors, which were initially of size `N x 28 x 28`, where `N` is the number of images. The last dimension we add represents the number of channels each image has, which is 1 because in MNIST we are working with greyscale images. For RGB, you would set this to 3. Also note that we add the number of channels as the last dimension in the Keras example. This is the standard adopted by Keras and Tensorflow. In our `numpy` implementation above, we adopted the convention used in `PyTorch`, the other major framework for machine learning, where the number of channels goes before the width and height of the image (i.e. a single image would be represented by a tensor of shape `1 x 28 x 28` instead of `28 x 28 x 1`).

Now our images are ready for training! Now let's turn out attention to our labels.

By default, `y_train` and `y_test` are arrays that contain single integers representing the digit that each image represents. Keras, however, expects each target label to be a **10-dimensional** vector. Therefore, we use `to_categorical()` from `keras.utils` to obtain an array of one-hot encoded vector. That is to say, if our target is 5, then after calling `to_categorical()`, we would get the zero-indexed array `[0. 0. 0. 0. 0. 1. 0. 0. 0. 0.]`.

Here's the code, with some helpful outputs to see what's going on:

```python
# one-hot encoding label vectors
print("y_train[0] before to_categorical: ", y_train[0])

y_train = to_categorical(y_train, 10)
y_test = to_categorical(y_test, 10)

print("y_train[0] after to_categorical: ", y_train[0])

print("y_train shape:", y_train.shape)
```

    y_train[0] before to_categorical:  5
    y_train[0] after to_categorical:  [0. 0. 0. 0. 0. 1. 0. 0. 0. 0.]
    y_train shape: (60000, 10)

Once we've gotten our data ready for training, we now move onto actually specifying the model that we'll be using. Again, this is almost the exact same model that we've implemented from scratch moments ago (the only difference being the presence of ReLU activation following each `Conv2D`):

```python
model = keras.Sequential(
    [
        keras.Input(shape=(28,28,1)),
        layers.Conv2D(filters=8, kernel_size=(3, 3), activation="relu"),
        layers.MaxPooling2D(pool_size=(2, 2)),
        layers.Flatten(),
        layers.Dense(10, activation="softmax"),
    ]
)

model.summary()
```

    Model: "sequential"
    _________________________________________________________________
     Layer (type)                Output Shape              Param #
    =================================================================
     conv2d (Conv2D)             (None, 26, 26, 8)         80

     max_pooling2d (MaxPooling2D  (None, 13, 13, 8)        0
     )

     flatten (Flatten)           (None, 1352)              0

     dense (Dense)               (None, 10)                13530

    =================================================================
    Total params: 13,610
    Trainable params: 13,610
    Non-trainable params: 0
    _________________________________________________________________

We also use a batch size of 1, which means we do gradient descent using gradients accumulated from just 1 image. We also do this for only 1 epoch. These settings are less than ideal, but to keep the Keras example consistent with our CNN implementation, we stick with it to see what kind of accuracy we can get. The [official Keras MNIST CNN example](https://keras.io/examples/vision/mnist_convnet/), for example, uses a batch size of `128` for 15 epochs with a deeper CNN model and easily achieves ~99% test accuracy on MNIST. Not bad!

We use the Adam optimizer with the `categorical_crossentropy` loss that we've discussed previously, since we have 10 target classes (which is > 2). Keras also allows us to keep track of various metrics during the course of training, and here we specify that we want Keras to report the accuracy.

Let's compile the model and train it!

```python
batch_size = 1
epochs = 1

model.compile(loss="categorical_crossentropy", optimizer="adam", metrics=["accuracy"])

model.fit(x_train, y_train, batch_size=batch_size, epochs=epochs, validation_split=0.1)
```

    54000/54000 [==============================] - 266s 4ms/step - loss: 0.1660 - accuracy: 0.9506 - val_loss: 0.0787 - val_accuracy: 0.9792

    <keras.callbacks.History at 0x7fa9d01f72d0>

```python
score = model.evaluate(x_test, y_test, verbose=0)
print("Test loss:", score[0])
print("Test accuracy:", score[1])
```

    Test loss: 0.08045849204063416
    Test accuracy: 0.9753000140190125

After just 1 epoch of training, we obtain a test accuracy of ~97.5% on the MNIST dataset, which is pretty good.

## Conclusion

That's it! We've reached the end of this notebook. To recap, we walked you through how CNNs can be implemented from scratch in Python using just `numpy`, and we've seen how we can specify and train the same CNN model using **Keras** to reach a ~97.5% test accuracy on the MNIST dataset. Hopefully, the next time you use Keras or PyTorch or Tensorflow or any other similar libraries, remember that all they're doing at a high level is abstracting away the theory and code we've covered in this notebook (except they're vectorized / optimised for the GPU and thus often significantly faster).

## References / Helpful Resources

- https://johnwlambert.github.io/conv-backprop/
- https://www.youtube.com/watch?v=Lakz2MoHy6o&ab_channel=TheIndependentCode
- https://victorzhou.com/blog/intro-to-cnns-part-2/
- https://eli.thegreenplace.net/2016/the-softmax-function-and-its-derivative/
- https://cs231n.github.io/linear-classify/#softmax-classifier
- https://web.eecs.umich.edu/~justincj/teaching/eecs498/FA2020/

```python

```