# Matrix generation

## Init symbols for *sympy*

```python
from sympy import *
from geom_util import *
from sympy.vector import CoordSys3D
N = CoordSys3D('N')
alpha1, alpha2, alpha3 = symbols("alpha_1 alpha_2 alpha_3", real = True, positive=True)
init_printing()

%matplotlib inline

%reload_ext autoreload
%autoreload 2
%aimport geom_util
```

### Lame params

```python
A,K = symbols('A K')
dA1 = Symbol('A_{{,1}}')
dK1 = Symbol('K_{{,1}}')
H1=A*(1+alpha3*K)
H2=S(1)
H3=S(1)

H=[H1, H2, H3]
DIM=3
dH = zeros(DIM,DIM)
dH[0,0]=dA1+alpha3*(dA1*K+A*dK1)
dH[0,2]=A*K

dH

```

### Metric tensor

${\displaystyle \hat{G}=\sum_{i,j} g^{ij}\vec{R}_i\vec{R}_j}$

```python
G_up = getMetricTensorUpLame(H1, H2, H3)
```

${\displaystyle \hat{G}=\sum_{i,j} g_{ij}\vec{R}^i\vec{R}^j}$

```python
G_down = getMetricTensorDownLame(H1, H2, H3)
```

### Christoffel symbols

```python
DIM=3

G_down_diff = MutableDenseNDimArray.zeros(DIM, DIM, DIM)
for i in range(DIM):
    for j in range(DIM):
        for k in range(DIM):

            G_down_diff[i,i,k]=2*H[i]*dH[i,k]

GK = getChristoffelSymbols2(G_up, G_down_diff, (alpha1, alpha2, alpha3))
GK
```

### Gradient of vector

$
\left(
\begin{array}{c}
\nabla_1 u_1 \\ \nabla_2 u_1 \\ \nabla_3 u_1 \\
\nabla_1 u_2 \\ \nabla_2 u_2 \\ \nabla_3 u_2 \\
\nabla_1 u_3 \\ \nabla_2 u_3 \\ \nabla_3 u_3 \\
\end{array}
\right)
=
B \cdot
\left(
\begin{array}{c}
u_1 \\
\frac { \partial u_1 } { \partial \alpha_1} \\
\frac { \partial u_1 } { \partial \alpha_2} \\
\frac { \partial u_1 } { \partial \alpha_3} \\
u_2 \\
\frac { \partial u_2 } { \partial \alpha_1} \\
\frac { \partial u_2 } { \partial \alpha_2} \\
\frac { \partial u_2 } { \partial \alpha_3} \\
u_3 \\
\frac { \partial u_3 } { \partial \alpha_1} \\
\frac { \partial u_3 } { \partial \alpha_2} \\
\frac { \partial u_3 } { \partial \alpha_3} \\
\end{array}
\right)
= B \cdot D \cdot
\left(
\begin{array}{c}
u^1 \\
\frac { \partial u^1 } { \partial \alpha_1} \\
\frac { \partial u^1 } { \partial \alpha_2} \\
\frac { \partial u^1 } { \partial \alpha_3} \\
u^2 \\
\frac { \partial u^2 } { \partial \alpha_1} \\
\frac { \partial u^2 } { \partial \alpha_2} \\
\frac { \partial u^2 } { \partial \alpha_3} \\
u^3 \\
\frac { \partial u^3 } { \partial \alpha_1} \\
\frac { \partial u^3 } { \partial \alpha_2} \\
\frac { \partial u^3 } { \partial \alpha_3} \\
\end{array}
\right)
$

```python
def row_index_to_i_j_grad(i_row):
    return i_row // 3, i_row % 3

B = zeros(9, 12)
B[0,1] = S(1)
B[1,2] = S(1)

B[2,3] = S(1)

B[3,5] = S(1)
B[4,6] = S(1)
B[5,7] = S(1)

B[6,9] = S(1)
B[7,10] = S(1)
B[8,11] = S(1)

for row_index in range(9):
    i,j=row_index_to_i_j_grad(row_index)
    B[row_index, 0] = -GK[i,j,0]
    B[row_index, 4] = -GK[i,j,1]
    B[row_index, 8] = -GK[i,j,2]

B
```

### Physical coordinates

$u_i=u_{[i]} H_i$

```python
P=zeros(12,12)
P[0,0]=H[0]
P[1,0]=dH[0,0]
P[1,1]=H[0]
P[2,0]=dH[0,1]
P[2,2]=H[0]
P[3,0]=dH[0,2]
P[3,3]=H[0]

P[4,4]=H[1]
P[5,4]=dH[1,0]
P[5,5]=H[1]
P[6,4]=dH[1,1]
P[6,6]=H[1]
P[7,4]=dH[1,2]
P[7,7]=H[1]

P[8,8]=H[2]
P[9,8]=dH[2,0]
P[9,9]=H[2]
P[10,8]=dH[2,1]
P[10,10]=H[2]
P[11,8]=dH[2,2]
P[11,11]=H[2]
P=simplify(P)
P
```

```python
B_P = zeros(9,9)

for i in range(3):
    for j in range(3):

        row_index = i*3+j

        B_P[row_index, row_index] = 1/(H[i]*H[j])

Grad_U_P = simplify(B_P*B*P)
Grad_U_P

# print(Grad_U_P)
```

### Strain tensor

$
\left(
\begin{array}{c}
\varepsilon_{11} \\
\varepsilon_{22} \\
\varepsilon_{33} \\
2\varepsilon_{12} \\
2\varepsilon_{13} \\
2\varepsilon_{23} \\
\end{array}
\right)
=
\left(E + E_{NL} \left( \nabla \vec{u} \right) \right) \cdot
\left(
\begin{array}{c}
\nabla_1 u_1 \\ \nabla_2 u_1 \\ \nabla_3 u_1 \\
\nabla_1 u_2 \\ \nabla_2 u_2 \\ \nabla_3 u_2 \\
\nabla_1 u_3 \\ \nabla_2 u_3 \\ \nabla_3 u_3 \\
\end{array}
\right)$

```python
E=zeros(6,9)
E[0,0]=1
E[1,4]=1
E[2,8]=1
E[3,1]=1
E[3,3]=1
E[4,2]=1
E[4,6]=1
E[5,5]=1
E[5,7]=1
E
```

```python
StrainL=simplify(E*Grad_U_P)
StrainL
```

```python
def E_NonLinear(grad_u):
    N = 3

    du = zeros(N, N)

    #    print("===Deformations===")

    for i in range(N):
        for j in range(N):
            index = i*N+j
            du[j,i] = grad_u[index]

    #    print("========")

    I = eye(3)

    a_values = S(1)/S(2) * du * G_up

    E_NL = zeros(6,9)
    E_NL[0,0] = a_values[0,0]
    E_NL[0,3] = a_values[0,1]
    E_NL[0,6] = a_values[0,2]

    E_NL[1,1] = a_values[1,0]
    E_NL[1,4] = a_values[1,1]
    E_NL[1,7] = a_values[1,2]

    E_NL[2,2] = a_values[2,0]
    E_NL[2,5] = a_values[2,1]
    E_NL[2,8] = a_values[2,2]

    E_NL[3,1] = 2*a_values[0,0]
    E_NL[3,4] = 2*a_values[0,1]
    E_NL[3,7] = 2*a_values[0,2]

    E_NL[4,0] = 2*a_values[2,0]
    E_NL[4,3] = 2*a_values[2,1]
    E_NL[4,6] = 2*a_values[2,2]

    E_NL[5,2] = 2*a_values[1,0]
    E_NL[5,5] = 2*a_values[1,1]
    E_NL[5,8] = 2*a_values[1,2]

    return E_NL

%aimport geom_util
u=getUHat3DPlane(alpha1, alpha2, alpha3)

# u=getUHatU3Main(alpha1, alpha2, alpha3)

gradu=B*u

E_NL = E_NonLinear(gradu)*B

E_NL

```

## Linear Stiffness matrix

```python
%aimport geom_util

mu = Symbol('mu')
la = Symbol('lambda')
C_tensor = getIsotropicStiffnessTensor(mu, la)
C = convertStiffnessTensorToMatrix(C_tensor)
C
```

```python
S=StrainL.T*C*StrainL*H1*H2*H3
S

```

## Mass matrix

```python
rho=Symbol('rho')
B_h=zeros(3,12)
B_h[0,0]=1
B_h[1,4]=1
B_h[2,8]=1
M=simplify(rho*P.T*B_h.T*G_up*B_h*P)
print(M)
```

    Matrix([[rho, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, rho, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, rho, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]])

```python
stiffness_matrix_func = lambdify([A, K, mu, la, alpha3], S, "numpy")
%store stiffness_matrix_func
mass_matrix_func = lambdify([rho], M, "numpy")
%store mass_matrix_func
```

```python

```


<big>Object recognition and computer vision 2019/2020</big>

<big>Jean Ponce, Ivan Laptev, Cordelia Schmid and Josef Sivic</big>

<big> Assignment 2: Neural networks</big>

<big>Adapted from practicals from Nicolas le Roux,

Andrea Vedaldi and Andrew Zisserman and Rob Fergus

 by Gul Varol and Ignacio Rocco</big>

Figure 1

```
`# This is formatted as code`
```

**STUDENT**:  DHAOU Amin

**EMAIL**:  amin.dhaou@hotmail.fr

# Guidelines

The purpose of this assignment is that you get hands-on experience with the topics covered in class, which will help you understand these topics better. Therefore, ** it is imperative that you do this assignment yourself. No code sharing will be tolerated. **

Once you have completed the assignment, you will submit the `ipynb` file containing **both** code and results. For this, make sure to **run your notebook completely before submitting**.

The `ipynb` must be named using the following format: **A2_LASTNAME_Firstname.ipynb**, and submitted in the **class Moodle page**.

# Goal

The goal of this assignment is to get basic knowledge and hands-on experience with training and using neural networks. In Part 1 of the assignment you will implement and experiment with the training and testing of a simple two layer fully-connected neural network, similar to the one depicted in Figure 1 above. In Part 2 you will learn about convolutional neural networks, their motivation, building blocks, and how they are trained. Finally, in part 3 you will train a CNN for classification using the CIFAR-10 dataset.

# Part 1 - Training a fully connected neural network

## Getting started

You will be working with a two layer neural network of the following form

\begin{equation}
H=\text{ReLU}(W_i X+B_i)\\
\bar{Y}=W_oH+B_o
\tag{1}
\end{equation}

where $X$ is the input, $\bar{Y}$ is the output, $H$ is the hidden layer, and $W_i$, $W_o$, $B_i$ and $B_o$ are the network parameters that need to be trained. Here the subscripts $i$ and $o$ stand for the *input* and *output* layer, respectively. This network was also discussed in the class and is illustrated in the above figure where the input units are shown in green, the hidden units in blue and the output in yellow. This network is implemented in the function `nnet_forward_logloss`.

You will train the parameters of the network from labelled training data $\{X^n,Y^n\}$ where $X^n$ are points in $\mathbb{R}^2$ and $Y^n\in\{-1,1\}$ are labels for each point. You will use the stochastic gradient descent algorithm discussed in the class to minimize the loss of the network on the training data given by

\begin{equation}
L=\sum_n s(Y^n,\bar{Y}(X^n))
\tag{2}
\end{equation}

where $Y^n$ is the target label for the n-th example and $\bar{Y}(X^n)$ is the network’s output for the n-th example $X^n$. The skeleton of the training procedure is provided in the `train_loop` function.

We will use the logistic loss, which has the following form:

\begin{equation}
s(Y, \bar{Y}(X))=\log(1+\exp(-Y. \bar{Y}(X))
\tag{3}
\end{equation}

where $Y$ is the target label and $\bar{Y}(X)$ is the output of the network for input example $X$. With the logistic loss, the output of the network can be interpreted as a probability $P(\text{class}=1|X) =\sigma(X)$ , where $\sigma(X) =1/(1+\exp(-X))$ is the sigmoid function. Note also that $P(\text{class}=-1|X)=1-P(\text{class}=1|X)$.

```
from IPython import display
import matplotlib.pyplot as plt
import numpy as np
import scipy.io as sio

def decision_boundary_nnet(X, Y, Wi, bi, Wo, bo):
    x_min, x_max = -2, 4
    y_min, y_max = -5, 3
    xx, yy = np.meshgrid(np.arange(x_min, x_max, .05),
                     np.arange(y_min, y_max, .05))

    XX = np.vstack((xx.ravel(), yy.ravel())).T
    input_hidden = np.dot(XX, Wi) + bi
    hidden = np.maximum(input_hidden, 0)
    Z = np.dot(hidden, Wo) + bo

    # Put the result into a color plot
    Z = Z.reshape(xx.shape)
    plt.contourf(xx, yy, Z > 0, cmap=plt.cm.Paired)
    plt.axis('off')

    # Plot also the training points
    plt.scatter(X[:, 0], X[:, 1], c=Y, cmap='winter')
    plt.axis([-2, 4, -5, 3])
    plt.draw()

def sigm(x):
    # Returns the sigmoid of x.
    small_x = np.where(x < -20)  # Avoid overflows.
    sigm_x = 1/(1 + np.exp(-x))
    if type(sigm_x) is np.ndarray:
        sigm_x[small_x] = 0.0
    return sigm_x

def nnet_forward_logloss(X, Y, Wi, bi, Wo, bo):
    '''
    Compute the output Po, Yo and the loss of the network for the input X
    This is a 2 layer (1 hidden layer network)

    Input:
        X ... (in R^2) set of input points, one per column
        Y ... {-1,1} the target values for the set of points X
        Wi, bi, Wo, bo ... parameters of the network

    Output:
       Po ... probabilisitc output of the network P(class=1 | x)
                  Po is in <0 1>.
                  Note: P(class=-1 | x ) = 1 - Po
       Yo ... output of the network Yo is in <-inf +inf>
       loss ... logistic loss of the network on examples X with ground target
                    values Y in {-1,1}
    '''
    # Hidden layer
    hidden = np.maximum(np.dot(X, Wi) + bi, 0)
    # Output of the network
    Yo = np.dot(hidden, Wo) + bo
    # Probabilistic output
    Po = sigm(Yo)
    # Logistic loss
    loss = np.log(1 + np.exp( -Y * Yo))
    return Po, Yo, loss

# Load the training data
!wget -q http://www.di.ens.fr/willow/teaching/recvis18/assignment2/double_moon_train1000.mat
train_data = sio.loadmat('./double_moon_train1000.mat', squeeze_me=True)
Xtr = train_data['X']
Ytr = train_data['Y']
# Load the validation data
!wget -q http://www.di.ens.fr/willow/teaching/recvis18/assignment2/double_moon_val1000.mat
val_data = sio.loadmat('./double_moon_val1000.mat', squeeze_me=True)
Xval = val_data['X']
Yval = val_data['Y']
```

## Computing gradients of the loss with respect to network parameters

### :: TASK 1.1 ::
Derive the form of the gradient of the logistic loss (3) with respect to the parameters of the network $W_i$, $W_o$, $B_i$ and $B_o$.  *Hint:* Use the chain rule as discussed in the class.

Let $X\in\mathbb{R}^d$, $Y\in\{-1,1\}$, $W_i\in\mathbb{R}^{d\times h}$, $B_i\in\mathbb{R}^h$, $W_o\in\mathbb{R}^h$ and $B_o\in\mathbb{R}$

Let :

\begin{equation}
Z=W_iX+B_i\in\mathbb{R}^h
\end{equation}

\begin{equation}
H=\text{ReLU}(W_i X+B_i)=\text{ReLU}(Z)\in\mathbb{R}^h
\end{equation}

\begin{equation}
\overline{Y}=W_o^TH+B_o=W_o^T\text{ReLU}(Z)+B_o\in\mathbb{R}
\end{equation}

First, let's do the computations of $\frac{\partial s}{\partial\overline{Y}}$:

\begin{equation}
\frac{\partial s}{\partial\overline{Y}}=
\frac{-Y\exp(-Y.\overline{Y}(X))}{1+\exp(-Y.\overline{Y}(X))}=
-Y\sigma(-Y.\overline{Y})\in\mathbb{R}
\end{equation}

Then, using the chain rule :

\begin{equation}
\frac{\partial s}{\partial W_o}=
\frac{\partial s}{\partial\overline{Y}}
\frac{\partial\overline{Y}}{\partial W_o}=
-Y\sigma(-Y.\overline{Y})H\in\mathbb{R^h}
\end{equation}

\begin{equation}
\frac{\partial s}{\partial B_o}=
\frac{\partial s}{\partial\overline{Y}}
\frac{\partial\overline{Y}}{\partial B_o}=
-Y\sigma(-Y.\overline{Y})\in\mathbb{R}
\end{equation}

\begin{equation}
\frac{\partial s}{\partial W_i}=
\frac{\partial s}{\partial\overline{Y}}
\frac{\partial\overline{Y}}{\partial W_i}=
-Y\sigma(-Y.\overline{Y})\frac{\partial\overline{Y}}{\partial Z}X^T
\end{equation}

But : $\overline{Y}=W_o^T\text{ReLU}(Z)+B_o$, then :

\begin{equation}
\frac{\partial\overline{Y}}{\partial Z}=
diag(\mathbb{1}_{Z\succ0})W_o\in\mathbb{R}^h
\end{equation}

With $diag(\mathbb{1}_{Z\succ0})\in\mathbb{R}^{h\times h}$ the matrix with $\mathbb{1}_{Z\succ0}\in\mathbb{R}^h$ in the diagonal.

Thus :

\begin{equation}
\frac{\partial s}{\partial W_i}=
-Y\sigma(-Y.\overline{Y})diag(\mathbb{1}_{Z\succ0})W_oX^T\in\mathbb{R}^{h\times d}
\end{equation}

Similarly :

\begin{equation}
\frac{\partial s}{\partial B_i}=
-Y\sigma(-Y.\overline{Y})diag(\mathbb{1}_{Z\succ0})W_o\in\mathbb{R}^{h}
\end{equation}

Other ressources:
http://cs231n.stanford.edu/slides/2018/cs231n_2018_ds02.pdf

### :: TASK 1.2 ::

Following your derivation, implement the gradient computation in the function `gradient_nn`. See the code for the description of the required inputs / outputs of this function.

```
def gradient_nn(X, Y, Wi, bi, Wo, bo):
    '''
    Compute gradient of the logistic loss of the neural network on example X with
    target label Y, with respect to the parameters Wi,bi,Wo,bo.

    Input:
        X ... 2d vector of the input example
        Y ... the target label in {-1,1}
        Wi,bi,Wo,bo ... parameters of the network
        Wi ... [dxh]
        bi ... [h]
        Wo ... [h]
        bo ... 1
        where h... is the number of hidden units
              d... is the number of input dimensions (d=2)

    Output:
        grad_s_Wi [dxh] ... gradient of loss s(Y,Y(X)) w.r.t  Wi
        grad_s_bi [h]   ... gradient of loss s(Y,Y(X)) w.r.t. bi
        grad_s_Wo [h]   ... gradient of loss s(Y,Y(X)) w.r.t. Wo
        grad_s_bo 1     ... gradient of loss s(Y,Y(X)) w.r.t. bo
    '''

    Z = np.dot(X,Wi) + bi
    H = np.maximum(Z, 0)
    Yo = np.dot(H,Wo) + bo
    Xt = np.reshape(X,(len(X),1))
    Wo2 = np.reshape(Wo,(1,len(Wo)))
    ind_Z = np.maximum(np.sign(Z),0)

    grad_s_Wi = -Y*sigm(-Y*Yo)*Xt@Wo2@np.diag(ind_Z)
    grad_s_bi = -Y*sigm(-Y*Yo)*np.diag(ind_Z)@Wo
    grad_s_Wo = -Y*sigm(-Y*Yo)*H
    grad_s_bo = -Y*sigm(-Y*Yo)

    return grad_s_Wi, grad_s_bi, grad_s_Wo, grad_s_bo

```

## Numerically verify the gradients
Here you will numerically verify that your analytically computed gradients in function `gradient_nn` are correct.

### :: TASK 1.3 ::
Write down the general formula for numerically computing the approximate derivative of the loss $s(\theta)$, with respect to the parameter $\theta_i$ using finite differencing.  *Hint: use the first order Taylor expansion of loss $s(\theta+\Delta \theta)$ around point $\theta$. *

Computing the first order Taylor expansion for  $s(\theta+\Delta \theta)$ and $s(\theta - \Delta \theta)$ gives:

$$\frac{\partial s}{\partial \theta} =\frac{ s(\theta+\Delta \theta) - s(\theta - \Delta \theta)}{ \Delta \theta}$$

The gradient_nn_numerical algorithm uses this to calculate the gradient

Following the general formula, `gradient_nn_numerical` function numerically computes the derivatives of the loss function with respect to all the parameters of the network $W_i$, $W_o$, $B_i$ and $B_o$:

```
def gradient_nn_numerical(X, Y, Wi, bi, Wo, bo):
    '''
    Compute numerical gradient of the logistic loss of the neural network on
    example X with target label Y, with respect to the parameters Wi,bi,Wo,bo.

    Input:
       X ... 2d vector of the input example
       Y ... the target label in {-1,1}
       Wi, bi, Wo, bo ... parameters of the network
       Wi ... [dxh]
       bi ... [h]
       Wo ... [h]
       bo ... 1
       where h... is the number of hidden units
             d... is the number of input dimensions (d=2)

    Output:
       grad_s_Wi_numerical [dxh] ... gradient of loss s(Y,Y(X)) w.r.t  Wi
       grad_s_bi_numerical [h]   ... gradient of loss s(Y,Y(X)) w.r.t. bi
       grad_s_Wo_numerical [h]   ... gradient of loss s(Y,Y(X)) w.r.t. Wo
       grad_s_bo_numerical 1     ... gradient of loss s(Y,Y(X)) w.r.t. bo
    '''

    eps = 1e-8
    grad_s_Wi_numerical = np.zeros(Wi.shape)
    grad_s_bi_numerical = np.zeros(bi.shape)
    grad_s_Wo_numerical = np.zeros(Wo.shape)

    for i in range(Wi.shape[0]):
        for j in range(Wi.shape[1]):
            dummy, dummy, pos_loss = nnet_forward_logloss(X, Y, sumelement_matrix(Wi, i, j, +eps), bi, Wo, bo)
            dummy, dummy, neg_loss = nnet_forward_logloss(X, Y, sumelement_matrix(Wi, i, j, -eps), bi, Wo, bo)
            grad_s_Wi_numerical[i, j] = (pos_loss - neg_loss)/(2*eps)

    for i in range(bi.shape[0]):
        dummy, dummy, pos_loss = nnet_forward_logloss(X, Y, Wi, sumelement_vector(bi, i, +eps), Wo, bo)
        dummy, dummy, neg_loss = nnet_forward_logloss(X, Y, Wi, sumelement_vector(bi, i, -eps), Wo, bo)
        grad_s_bi_numerical[i] = (pos_loss - neg_loss)/(2*eps)

    for i in range(Wo.shape[0]):
        dummy, dummy, pos_loss = nnet_forward_logloss(X, Y, Wi, bi, sumelement_vector(Wo, i, +eps), bo)
        dummy, dummy, neg_loss = nnet_forward_logloss(X, Y, Wi, bi, sumelement_vector(Wo, i, -eps), bo)
        grad_s_Wo_numerical[i] = (pos_loss - neg_loss)/(2*eps)

    dummy, dummy, pos_loss = nnet_forward_logloss(X, Y, Wi, bi, Wo, bo+eps)
    dummy, dummy, neg_loss = nnet_forward_logloss(X, Y, Wi, bi, Wo, bo-eps)
    grad_s_bo_numerical = (pos_loss - neg_loss)/(2*eps)

    return grad_s_Wi_numerical, grad_s_bi_numerical, grad_s_Wo_numerical, grad_s_bo_numerical

def sumelement_matrix(X, i, j, element):
    Y = np.copy(X)
    Y[i, j] = X[i, j] + element
    return Y

def sumelement_vector(X, i, element):
    Y = np.copy(X)
    Y[i] = X[i] + element
    return Y
```

### :: TASK 1.4 ::

Run the following code snippet and understand what it is doing. `gradcheck` function checks that the analytically computed derivative using function `gradient_nn` (e.g. `grad_s_bo`) at the same training example $\{X,Y\}$ is the same (up to small errors) as your numerically computed value of the derivative using function `gradient_nn_numerical` (e.g. `grad_s_bo_numerical`). Make sure the output is `SUCCESS` to move on to the next task.

```
def gradcheck():
    '''
    Check that the numerical and analytical gradients are the same up to eps
    '''
    h = 3 # number of hidden units
    eps = 1e-6
    for i in range(1000):
        # Generate random input/output/weight/bias
        X  = np.random.randn(2)
        Y  = 2* np.random.randint(2) - 1 # {-1, 1}
        Wi = np.random.randn(X.shape[0], h)
        bi = np.random.randn(h)
        Wo = np.random.randn(h)
        bo = np.random.randn(1)
        # Compute analytical gradients
        grad_s_Wi, grad_s_bi, grad_s_Wo, grad_s_bo = gradient_nn(X, Y, Wi, bi, Wo, bo)
        # Compute numerical gradients
        grad_s_Wi_numerical, grad_s_bi_numerical, grad_s_Wo_numerical, grad_s_bo_numerical = gradient_nn_numerical(X, Y, Wi, bi, Wo, bo)
        # Compute the difference between analytical and numerical gradients
        delta_Wi = np.mean(np.abs(grad_s_Wi - grad_s_Wi_numerical))
        delta_bi = np.mean(np.abs(grad_s_bi - grad_s_bi_numerical))
        delta_Wo = np.mean(np.abs(grad_s_Wo - grad_s_Wo_numerical))
        delta_bo = np.abs(grad_s_bo - grad_s_bo_numerical)
        # Difference larger than a threshold
        if ( delta_Wi > eps or delta_bi > eps or delta_Wo > eps or delta_bo > eps):
            return False

    return True

# Check gradients
if gradcheck():
    print('SUCCESS: Passed gradcheck.')
else:
    print('FAILURE: Fix gradient_nn and/or gradient_nn_aprox implementation.')
```

    SUCCESS: Passed gradcheck.

## Training the network using backpropagation and experimenting with different parameters

Use the provided code below that calls the `train_loop` function. Set the number of hidden units to 7 by setting $h=7$ in the code and set the learning rate to 0.02 by setting `lrate = 0.02`. Run the training code. Visualize the trained hyperplane using the provided function `plot_decision_boundary(Xtr,Ytr,Wi,bi,Wo,bo)`. Show also the evolution of the training and validation errors. Include the decision hyper-plane visualization and the training and validation error plots.

```
def train_loop(Xtr, Ytr, Xval, Yval, h = 7, lrate = 0.02, vis='all', nEpochs=100):
    '''
    Check that the numerical and analytical gradients are the same up to eps

    Input:
        Xtr ... Nx2 matrix of training samples
        Ytr ... N dimensional vector of training labels
        Xval ... Nx2 matrix of validation samples
        Yval ... N dimensional vector validation labels
        h ... number of hidden units
        lrate ... learning rate
        vis ... visulaization option ('all' | 'last' | 'never')
        nEpochs ... number of training epochs

    Output:
        tr_error ... nEpochs*nSamples dimensional vector of training error
        val_error ... nEpochs*nSamples dimensional vector of validation error
    '''

    nSamples = Xtr.shape[0]
    tr_error = np.zeros(nEpochs*nSamples)
    val_error = np.zeros(nEpochs*nSamples)

    # Randomly initialize parameters of the model
    Wi = np.random.randn(Xtr.shape[1], h)
    Wo = np.zeros(h)
    bi = np.zeros(h)
    bo = 0.

    if(vis == 'all' or vis == 'last'):
        plt.figure()

    for i in range(nEpochs*nSamples):
        # Draw an example at random
        n = np.random.randint(nSamples)
        X = Xtr[n]
        Y = Ytr[n]

        # Compute gradient
        grad_s_Wi, grad_s_bi, grad_s_Wo, grad_s_bo = gradient_nn(X, Y, Wi, bi, Wo, bo)

        # Gradient update
        Wi -= lrate*grad_s_Wi
        Wo -= lrate*grad_s_Wo
        bi -= lrate*grad_s_bi
        bo -= lrate*grad_s_bo

        # Compute training error
        Po, Yo, loss    = nnet_forward_logloss(Xtr, Ytr, Wi, bi, Wo, bo)
        Yo_class        = np.sign(Yo)
        tr_error[i]     = 100*np.mean(Yo_class != Ytr)

        # Compute validation error
        Pov, Yov, lossv = nnet_forward_logloss(Xval, Yval, Wi, bi, Wo, bo)
        Yov_class       = np.sign(Yov)
        val_error[i]    = 100*np.mean(Yov_class != Yval)

        # Plot (at every epoch if visualization is 'all', only at the end if 'last')
        if(vis == 'all' and i%nSamples == 0) or (vis == 'last' and i == nEpochs*nSamples - 1):
            # Draw the decision boundary.
            plt.clf()
            plt.title('p = %d, Iteration = %.d, Error = %.3f' % (h, i/nSamples, tr_error[i]))
            decision_boundary_nnet(Xtr, Ytr, Wi, bi, Wo, bo)
            display.display(plt.gcf(), display_id=True)
            display.clear_output(wait=True)

    if(vis == 'all'):
        # Plot the evolution of the training and test errors
        plt.figure()
        plt.plot(tr_error, label='training')
        plt.plot(val_error, label='validation')
        plt.legend()
        plt.title('Training/validation errors: %.2f%% / %.2f%%' % (tr_error[-1], val_error[-1]))
    return tr_error, val_error
```

```
# Run training
h = 7
lrate = .02
tr_error, val_error = train_loop(Xtr, Ytr, Xval, Yval, h, lrate)
```

### :: TASK 1.6 ::

**Random initializations.** Repeat this procedure 5 times from 5 different random initializations. Record for each run the final training and validation errors. Did the network always converge to zero training error? Summarize your final training and validation errors into a table for the 5 runs. You do not need to include the decision hyper-plane visualizations. Note: to speed-up the training you can plot the visualization figures less often (or never) and hence speed-up the training.

```
list_h = [10, 15, 20, 30, 50]
ltr_error, lval_error = [] , []
for k in range(5):
  tr_error, val_error = train_loop(Xtr, Ytr, Xval, Yval, list_h[k], vis = 'never')
  ltr_error.append(tr_error[-1])
  lval_error.append(val_error[-1])

```

```
import pandas
pandas.DataFrame(data = {'Training Error': ltr_error, 'Validation Error': lval_error}, index = np.arange(1, 6))
```

      Training Error
      Validation Error

      1
      0.0
      0.0

      2
      0.0
      0.0

      3
      0.0
      0.0

      4
      0.0
      0.0

      5
      0.0
      0.0

In these cases, the network always converge to zero training errors
here is the table: [[0.0, 0.0], [0.0, 0.0], [0.0, 0.0], [0.0, 0.0], [0.0, 0.0]]

### :: SAMPLE TASK ::

For this task, the answer is given. Run the given code and answer Task 1.8 similarly.

**Learning rate:**

Keep $h=7$ and change the learning rate to values $\text{lrate} = \{2, 0.2, 0.02, 0.002\}$. For each of these values run the training procedure 5 times and observe the training behaviour. You do not need to include the decision hyper-plane visualizations in your answer.

**- Make one figure** where *final* error for (i) training and (ii) validation sets are superimposed. $x$-axis should be the different values of the learning rate, $y$-axis the error *mean* across 5 runs. Show the standard deviation with error bars and make sure to label each plot with a legend.

**- Make another figure** where *training error evolution* for each learning rate is superimposed. $x$-axis should be the iteration number, $y$-axis the training error *mean* across 5 runs for a given learning rate. Show the standard deviation with error bars and make sure to label each curve with a legend.

```
nEpochs = 40
trials = 5
lrates = [2, 0.2, 0.02, 0.002]
plot_data_lr = np.zeros((2, trials, len(lrates), nEpochs*1000))
h = 7
for j, lrate in enumerate(lrates):
    print('LR = %f' % lrate)
    for i in range(trials):
        tr_error, val_error = train_loop(Xtr, Ytr, Xval, Yval, h, lrate, vis='never', nEpochs=nEpochs)
        plot_data_lr[0, i, j, :] = tr_error
        plot_data_lr[1, i, j, :] = val_error
```

    LR = 2.000000

    /usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:32: RuntimeWarning: overflow encountered in exp
    /usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:63: RuntimeWarning: overflow encountered in exp
    /usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:29: RuntimeWarning: overflow encountered in exp
    /usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:29: RuntimeWarning: invalid value encountered in double_scalars
    /usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:31: RuntimeWarning: invalid value encountered in less

    LR = 0.200000
    LR = 0.020000
    LR = 0.002000

```
plt.errorbar(np.arange(len(lrates)), plot_data_lr[0, :, :, -1].mean(axis=0), yerr=plot_data_lr[0, :, :, -1].std(axis=0), label='Training')
plt.errorbar(np.arange(len(lrates)), plot_data_lr[1, :, :, -1].mean(axis=0), yerr=plot_data_lr[0, :, :, -1].std(axis=0), label='Validation')
plt.xticks(np.arange(len(lrates)), lrates)
plt.xlabel('learning rate')
plt.ylabel('error')
plt.legend()

# Plot the evolution of the training loss for each learning rate
plt.figure()
for j, lrate in enumerate(lrates):
    x = np.arange(plot_data_lr.shape[3])
    # Mean training loss over trials
    y = plot_data_lr[0, :, j, :].mean(axis=0)
    # Standard deviation over trials
    ebar = plot_data_lr[0, :, j, :].std(axis=0)
    # Plot
    markers, caps, bars = plt.errorbar(x, y, yerr=ebar, label='LR = ' + str(lrate))
    # Make the error bars transparent
    [bar.set_alpha(0.01) for bar in bars]
plt.legend()
plt.xlabel('iterations')
plt.ylabel('training error')
```

### :: TASK 1.7 ::

**- Briefly discuss** the different behaviour of the training for different learning rates. How many iterations does it take to converge or does it converge at all? Which learning rate is  better and why?

We see from the plot that depending on the learning rate the model will not always converge. The model converge with the three other learning rates.
It take approximately 2000 iterations to converge for the model with LR = 0.2 , 10 000 for LR = 0.02 and 30 000 for LR = 0.002.
When learning rate is too high ( it's the case for LR = 2), the weights are update too much and then the loss will diverge
When the learning rate is too low, the weights are updated little by little and it'll take a lot of time to have the convergence.

The optimal learning is the one of the green curve because although the yellow converge faster the variance is more important in this last plot.
Indeed, the model with LR = 0.02 keeps a good balance between the speed and the accuracy.

### :: TASK 1.8 ::
**The number of hidden units:**

Set the learning rate to 0.02 and change the number of hidden units $h = \{1, 2, 5, 7, 10, 100\}$. For each of these values run the training procedure 5 times and observe the training behaviour

**-Visualize** one decision hyper-plane per number of hidden units.

**-Make one figure** where *final* error for (i) training and (ii) validation sets are superimposed. $x$-axis should be the different values of the number of hidden units, $y$-axis the error *mean* across 5 runs. Show the standard deviation with error bars and make sure to label each plot with a legend.

**-Make another figure** where *training error evolution* for each number of hidden units is superimposed. $x$-axis should be the iteration number, $y$-axis the training error *mean* across 5 runs for a given learning rate. Show the standard deviation with error bars and make sure to label each curve with a legend.

**-Briefly discuss** the different behaviours for the different numbers of hidden units.

```
nEpochs = 40
trials = 5
rate = 0.2
hid = [1, 2, 5, 7, 10, 100]
plot_data_h = np.zeros((2, trials, len(hid), nEpochs*1000))
h = 7
for j, h in enumerate(hid):
    print('h = %d' % h)
    for i in range(trials):
        tr_error, val_error = train_loop(Xtr, Ytr, Xval, Yval, h, rate, vis='never', nEpochs=nEpochs)
        plot_data_h[0, i, j, :] = tr_error
        plot_data_h[1, i, j, :] = val_error
```

    h = 1
    h = 2
    h = 5
    h = 7
    h = 10
    h = 100

```
plt.errorbar(np.arange(len(hid)), plot_data_h[0, :, :, -1].mean(axis=0), yerr=plot_data_h[0, :, :, -1].std(axis=0), label='Training')
plt.errorbar(np.arange(len(hid)), plot_data_h[1, :, :, -1].mean(axis=0), yerr=plot_data_h[0, :, :, -1].std(axis=0), label='Validation')
plt.xticks(np.arange(len(hid)), hid)
plt.xlabel('Number of hidden units')
plt.ylabel('error')
plt.legend()

# Plot the evolution of the training loss for each h
plt.figure()
for j, hid in enumerate(hid):
    x = np.arange(plot_data_h.shape[3])
    # Mean training loss over trials
    y = plot_data_h[0, :, j, :].mean(axis=0)
    # Standard deviation over trials
    ebar = plot_data_h[0, :, j, :].std(axis=0)
    # Plot
    markers, caps, bars = plt.errorbar(x, y, yerr=ebar, label='H = ' + str(hid))
    # Make the error bars transparent
    [bar.set_alpha(0.01) for bar in bars]
plt.legend()
plt.xlabel('iterations')
plt.ylabel('training error')
```

We observe that we can't fit the data when the model is too simple. It's the case for h=1 and h=2 layers.  Indeed, the model can't achieve zero training error.
Increasing the number of hidden layers and thus the numbers of parameters will allow the model to fit complex data and then converge. But increasing the number of hidden layer will increase the time to train the model.
So a good tradeoff here is to choose h = 10.

# Part 2 - Building blocks of a CNN

This part introduces typical CNN building blocks, such as ReLU units and linear filters. For a motivation for using CNNs over fully-connected neural networks, see [[Le Cun, et al, 1998]](http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf).

## Install PyTorch

```
!pip install torch torchvision
import torch
print(torch.__version__)
print(torch.cuda.is_available())
```

    Requirement already satisfied: torch in /usr/local/lib/python3.6/dist-packages (1.3.1+cu100)
    Requirement already satisfied: torchvision in /usr/local/lib/python3.6/dist-packages (0.4.2+cu100)
    Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from torch) (1.17.3)
    Requirement already satisfied: pillow>=4.1.1 in /usr/local/lib/python3.6/dist-packages (from torchvision) (4.3.0)
    Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.12.0)
    Requirement already satisfied: olefile in /usr/local/lib/python3.6/dist-packages (from pillow>=4.1.1->torchvision) (0.46)
    1.3.1+cu100
    True

## Convolution

A feed-forward neural network can be thought of as the composition of number of functions
$$
f(\mathbf{x}) = f_L(\dots f_2(f_1(\mathbf{x};\mathbf{w}_1);\mathbf{w}_2)\dots),\mathbf{w}_{L}).
$$
Each function $f_l$ takes as input a datum $\mathbf{x}_l$ and a parameter vector $\mathbf{w}_l$ and produces as output a datum $\mathbf{x}_{l+1}$. While the type and sequence of functions is usually handcrafted, the parameters $\mathbf{w}=(\mathbf{w}_1,\dots,\mathbf{w}_L)$ are *learned from data* in order to solve a target problem, for example classifying images or sounds.

In a *convolutional neural network* data and functions have additional structure. The data $\mathbf{x}_1,\dots,\mathbf{x}_n$ are images, sounds, or more in general maps from a lattice$^1$ to one or more real numbers. In particular, since the rest of the practical will focus on computer vision applications, data will be 2D arrays of pixels. Formally, each $\mathbf{x}_i$ will be a $M \times N \times K$ real array of $M \times N$ pixels and $K$ channels per pixel. Hence the first two dimensions of the array span space, while the last one spans channels. Note that only the input $\mathbf{x}=\mathbf{x}_1$ of the network is an actual image, while the remaining data are intermediate *feature maps*.

The second property of a CNN is that the functions $f_l$ have a *convolutional structure*. This means that $f_l$ applies to the input map $\mathbf{x}_l$ an operator that is *local and translation invariant*. Examples of convolutional operators are applying a bank of linear filters to $\mathbf{x}_l$.

In this part we will familiarise ourselves with a number of such convolutional and non-linear operators. The first one is the regular *linear convolution* by a filter bank. We will start by focusing our attention on a single function relation as follows:
$$
 f: \mathbb{R}^{M\times N\times K} \rightarrow \mathbb{R}^{M' \times N' \times K'},
 \qquad \mathbf{x} \mapsto \mathbf{y}.
$$

$^1$A two-dimensional *lattice* is a discrete grid embedded in $R^2$, similar for example to a checkerboard.

```
import matplotlib.pyplot as plt
import numpy as np
from PIL import Image
import torch
import torchvision

# Download an example image
!wget -q http://www.di.ens.fr/willow/teaching/recvis/assignment3/images/peppers.png
# Read the image
x = np.asarray(Image.open('peppers.png'))/255.0
# Print the size of x. Third dimension (=3) corresponds to the R, G, B channels
print(x.shape)
# Visualize the input x
plt.imshow(x)
# Convert to torch tensor
x = torch.from_numpy(x).permute(2, 0, 1).float()
# Prepare it as a batch
x = x.unsqueeze(0)
```

This should display an image of bell peppers.

Next, we create a convolutional layer with a bank of 10 filters of dimension $5 \times 5 \times 3$ whose coefficients are initialized randomly. This uses the [`torch.nn.Conv2d`](https://pytorch.org/docs/stable/nn.html#torch.nn.Conv2d) module from PyTorch:

```
# Create a convolutional layer and a random bank of linear filters
conv = torch.nn.Conv2d(3, 10, kernel_size=5, stride=1, padding=0, bias=False)
print(conv.weight.size())
```

    torch.Size([10, 3, 5, 5])

**Remark:** You might have noticed that the `bias` argument to the `torch.nn.Conv2d` function is the empty matrix `false`. It can be otherwise used to pass a vector of bias terms to add to the output of each filter.

Note that `conv.weight` has four dimensions, packing 10 filters. Note also that each filter is not flat, but rather a volume containing three slices. The next step is applying the filter to the image.

```
# Apply the convolution operator
y = conv(x)
# Observe the input/output sizes
print(x.size())
print(y.size())
```

    torch.Size([1, 3, 384, 512])
    torch.Size([1, 10, 380, 508])

The variable `y` contains the output of the convolution. Note that the filters are three-dimensional. This is because they operate on a tensor $\mathbf{x}$ with $K$ channels. Furthermore, there are $K'$ such filters, generating a $K'$ dimensional map $\mathbf{y}$.

We can now visualise the output `y` of the convolution. In order to do this, use the `torchvision.utils.make_grid` function to display an image for each feature channel in `y`:

```
# Visualize the output y
def vis_features(y):
    # Organize it into 10 grayscale images
    out = y.permute(1, 0, 2, 3)
    # Scale between [0, 1]
    out = (out - out.min().expand(out.size())) / (out.max() - out.min()).expand(out.size())
    # Create a grid of images
    out = torchvision.utils.make_grid(out, nrow=5)
    # Convert to numpy image
    out = np.transpose(out.detach().numpy(), (1, 2, 0))
    # Show
    plt.imshow(out)
    # Remove grid
    plt.gca().grid(False)

vis_features(y)
```

So far filters preserve the resolution of the input feature map. However, it is often useful to *downsample the output*. This can be obtained by using the `stride` option in `torch.nn.Conv2d`:

```
# Try again, downsampling the output
conv_ds = torch.nn.Conv2d(3, 10, kernel_size=5, stride=16, padding=0, bias=False)
y_ds = conv_ds(x)
print(x.size())
print(y_ds.size())
vis_features(y_ds)
```

Applying a filter to an image or feature map interacts with the boundaries, making the output map smaller by an amount proportional to the size of the filters. If this is undesirable, then the input array can be padded with zeros by using the `pad` option:

```
# Try padding
conv_pad = torch.nn.Conv2d(3, 10, kernel_size=5, stride=1, padding=2, bias=False)
y_pad = conv_pad(x)
print(x.size())
print(y_pad.size())
vis_features(y_pad)
```

In order to consolidate what has been learned so far, we will now design a filter by hand:

```
w = torch.FloatTensor([[0,  1, 0 ],
                      [1, -4, 1 ],
                      [0,  1, 0 ]])
w = w.repeat(3, 1).reshape(1, 3, 3, 3)
conv_lap = torch.nn.Conv2d(3, 3, kernel_size=3, stride=1, padding=1, bias=False)
conv_lap.weight = torch.nn.Parameter(w)
y_lap = conv_lap(x)
print(x.size())
print(y_lap.size())

plt.figure()
vis_features(y_lap)
plt.title('filter output')

plt.figure()
vis_features(-torch.abs(y_lap))
plt.title('- abs(filter output)') ;
```

```
print(w)
```

    tensor([[[[ 0.,  1.,  0.],
              [ 1., -4.,  1.],
              [ 0.,  1.,  0.]],

             [[ 0.,  1.,  0.],
              [ 1., -4.,  1.],
              [ 0.,  1.,  0.]],

             [[ 0.,  1.,  0.],
              [ 1., -4.,  1.],
              [ 0.,  1.,  0.]]]])

### :: TASK 2.1 ::
* i. What filter have we implemented?
* ii. How are the RGB colour channels processed by this filter?
* iii. What image structure are detected?

i. A Laplacian Filter is implemented here

ii. This filter is applied to each of the RGB channel and then we sum the three values on the corresponding pixel and get the final feature map.

iii. It detected quick changes of intensity in the image. Then, the edges in the image are detected.

## Non-linear activation functions

The simplest non-linearity is obtained by following a linear filter by a *non-linear activation function*, applied identically to each component (i.e. point-wise) of a feature map. The simplest such function is the *Rectified Linear Unit (ReLU)*
$$
  y_{ijk} = \max\{0, x_{ijk}\}.
$$
This function is implemented by [`torch.nn.ReLU()`](https://pytorch.org/docs/stable/nn.html#torch.nn.ReLU). Run the code below and understand what the filter $\mathbf{w}$ is doing.

```
w = torch.FloatTensor([[1], [0], [-1]]).repeat(1, 3, 1, 1)
w = torch.cat((w, -w), 0)

conv = torch.nn.Conv2d(3, 2, kernel_size=(3, 1), stride=1, padding=0, bias=False)
conv.weight = torch.nn.Parameter(w)
relu = torch.nn.ReLU()

y = conv(x)
z = relu(y)

plt.figure()
vis_features(y)
plt.figure()
vis_features(z)
```

## Pooling
There are several other important operators in a CNN. One of them is *pooling*.  A pooling operator operates on individual feature channels, coalescing nearby feature values into one by the application of a suitable operator. Common choices include max-pooling (using the max operator) or sum-pooling (using summation). For example, *max-pooling* is defined as:
$$
   y_{ijk} = \max \{ y_{i'j'k} : i \leq i' < i+p, j \leq j' < j + p \}
$$
Max-pooling is implemented by [`torch.nn.MaxPool2d()`](https://pytorch.org/docs/stable/nn.html#torch.nn.MaxPool2d).

### :: TASK 2.2 ::

Run the code below to try max-pooling. Look at the resulting image. Can you interpret the result?

```
mp = torch.nn.MaxPool2d(15, stride=1)
y = mp(x)
plt.imshow(y.squeeze().permute(1, 2, 0).numpy())
plt.gca().grid(False)
```

Maxpooling will downsample the image. In fact, it will take the maximum of the kernel applied in the image. Then the output image will be smoothed but the whole image will keep its caracteristics.

# Part 3 - Training a CNN

This part is an introduction to using PyTorch for training simple neural net models. CIFAR-10 dataset will be used.

## Imports

```
from __future__ import print_function
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torchvision import datasets, transforms
from torch.autograd import Variable

USE_GPU = True # To train on gpu (no that important small network and data)

if USE_GPU and torch.cuda.is_available():
    device = torch.device('cuda')
else:
    device = torch.device('cpu')

print("Using : %s" %str(device))
```

    Using : cuda

## Parameters

The default values for the learning rate, batch size and number of epochs are given in the "options" cell of this notebook.
Unless otherwise specified, use the default values throughout this assignment.

```
batch_size = 64   # input batch size for training
epochs = 10       # number of epochs to train
lr = 0.01         # learning rate
```

## Warmup

It is always good practice to visually inspect your data before trying to train a model, since it lets you check for problems and get a feel for the task at hand.

CIFAR-10 is a dataset of 60,000 color images (32 by 32 resolution) across 10 classes
(airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck).
The train/test split is 50k/10k.

```
# Data Loading
# Warning: this cell might take some time when you run it for the first time,
#          because it will download the dataset from the internet
dataset = 'cifar10'
data_transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
])
trainset = datasets.CIFAR10(root='.', train=True, download=True, transform=data_transform)
testset = datasets.CIFAR10(root='.', train=False, download=True, transform=data_transform)

train_loader = torch.utils.data.DataLoader(trainset, batch_size=batch_size, shuffle=True, num_workers=0)
test_loader  = torch.utils.data.DataLoader(testset, batch_size=batch_size, shuffle=False, num_workers=0)
```

    0it [00:00, ?it/s]

    Downloading https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz to ./cifar-10-python.tar.gz

    170500096it [00:04, 40762261.00it/s]

    Extracting ./cifar-10-python.tar.gz to .
    Files already downloaded and verified

### :: TASK 3.1 ::

Use `matplotlib` and ipython notebook's visualization capabilities to display some of these images. Display 5 images from the dataset together with their category label. [See this PyTorch tutorial page](http://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#sphx-glr-beginner-blitz-cifar10-tutorial-py) for hints on how to achieve this.

```
import matplotlib.pyplot as plt
classes = ('plane', 'car', 'bird', 'cat',
           'deer', 'dog', 'frog', 'horse', 'ship', 'truck')

def imshow(img):
    img = img / 2 + 0.5     # unnormalize
    npimg = img.numpy()
    plt.imshow(np.transpose(npimg, (1, 2, 0)))
    plt.show()
trainloader = torch.utils.data.DataLoader(trainset, batch_size=5, shuffle=True, num_workers=0)
# get some random training images
dataiter = iter(trainloader)
images, labels = dataiter.next()

# show images
imshow(torchvision.utils.make_grid(images))
# print labels
print(' '.join('%5s' % classes[labels[j]] for j in range(5)))
```

## Training a Convolutional Network on CIFAR-10

Start by running the provided training code below. By default it will train on CIFAR-10 for 10 epochs (passes through the training data) with a single layer network. The loss function [cross_entropy](http://pytorch.org/docs/master/nn.html?highlight=cross_entropy#torch.nn.functional.cross_entropy) computes a Logarithm of the Softmax on the output of the neural network, and then computes the negative log-likelihood w.r.t. the given `target`. Note the decrease in training loss and corresponding decrease in validation errors.

```
def train(epoch, network):
    network.train()
    for batch_idx, (data, target) in enumerate(train_loader):
        optimizer.zero_grad()
        data = data.to(device)
        target = target.to(device)
        output = network(data)
        loss = F.cross_entropy(output, target)
        loss.backward()
        optimizer.step()
        if batch_idx % 100 == 0:
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                epoch, batch_idx * len(data), len(train_loader.dataset),
                100. * batch_idx / len(train_loader), loss.item()))

def test(network):
    network.eval()
    test_loss = 0
    correct = 0
    for data, target in test_loader:
        data = data.to(device)
        target = target.to(device)
        output = network(data)
        test_loss += F.cross_entropy(output, target, size_average=False).item() # sum up batch loss
        pred = output.data.max(1, keepdim=True)[1] # get the index of the max log-probability
        correct += pred.eq(target.data.view_as(pred)).cpu().sum()

    test_loss /= len(test_loader.dataset)
    print('\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n'.format(
        test_loss, correct, len(test_loader.dataset),
        100. * correct / len(test_loader.dataset)))
```

```
# Single layer network architecture

class Net(nn.Module):
    def __init__(self, num_inputs, num_outputs):
        super(Net, self).__init__()
        self.linear = nn.Linear(num_inputs, num_outputs)
        self.num_inputs = num_inputs

    def forward(self, input):
        input = input.view(-1, self.num_inputs) # reshape input to batch x num_inputs
        output = self.linear(input)
        return output
```

```
# Train
network = Net(3072, 10).to(device)
optimizer = optim.SGD(network.parameters(), lr=lr)
for epoch in range(1, 11):
    train(epoch, network)
    test(network)
```

    Train Epoch: 1 [0/50000 (0%)]	Loss: 2.251657
    Train Epoch: 1 [6400/50000 (13%)]	Loss: 2.000074
    Train Epoch: 1 [12800/50000 (26%)]	Loss: 1.983296
    Train Epoch: 1 [19200/50000 (38%)]	Loss: 1.921615
    Train Epoch: 1 [25600/50000 (51%)]	Loss: 1.727296
    Train Epoch: 1 [32000/50000 (64%)]	Loss: 1.843463
    Train Epoch: 1 [38400/50000 (77%)]	Loss: 1.662544
    Train Epoch: 1 [44800/50000 (90%)]	Loss: 1.798393

    /usr/local/lib/python3.6/dist-packages/torch/nn/_reduction.py:43: UserWarning: size_average and reduce args will be deprecated, please use reduction='sum' instead.
      warnings.warn(warning.format(ret))

    Test set: Average loss: 1.7893, Accuracy: 3772/10000 (38%)

    Train Epoch: 2 [0/50000 (0%)]	Loss: 1.696283
    Train Epoch: 2 [6400/50000 (13%)]	Loss: 1.725371
    Train Epoch: 2 [12800/50000 (26%)]	Loss: 1.677268
    Train Epoch: 2 [19200/50000 (38%)]	Loss: 1.631910
    Train Epoch: 2 [25600/50000 (51%)]	Loss: 1.633661
    Train Epoch: 2 [32000/50000 (64%)]	Loss: 1.554669
    Train Epoch: 2 [38400/50000 (77%)]	Loss: 1.605391
    Train Epoch: 2 [44800/50000 (90%)]	Loss: 1.639972

    Test set: Average loss: 1.7663, Accuracy: 3831/10000 (38%)

    Train Epoch: 3 [0/50000 (0%)]	Loss: 1.780690
    Train Epoch: 3 [6400/50000 (13%)]	Loss: 1.782501
    Train Epoch: 3 [12800/50000 (26%)]	Loss: 1.788695
    Train Epoch: 3 [19200/50000 (38%)]	Loss: 1.794215
    Train Epoch: 3 [25600/50000 (51%)]	Loss: 1.657584
    Train Epoch: 3 [32000/50000 (64%)]	Loss: 1.885633
    Train Epoch: 3 [38400/50000 (77%)]	Loss: 1.596505
    Train Epoch: 3 [44800/50000 (90%)]	Loss: 1.897346

    Test set: Average loss: 1.7371, Accuracy: 4028/10000 (40%)

    Train Epoch: 4 [0/50000 (0%)]	Loss: 2.088798
    Train Epoch: 4 [6400/50000 (13%)]	Loss: 1.728375
    Train Epoch: 4 [12800/50000 (26%)]	Loss: 1.887703
    Train Epoch: 4 [19200/50000 (38%)]	Loss: 1.775010
    Train Epoch: 4 [25600/50000 (51%)]	Loss: 1.589597
    Train Epoch: 4 [32000/50000 (64%)]	Loss: 1.844424
    Train Epoch: 4 [38400/50000 (77%)]	Loss: 1.785592
    Train Epoch: 4 [44800/50000 (90%)]	Loss: 1.787325

    Test set: Average loss: 1.7229, Accuracy: 4091/10000 (41%)

    Train Epoch: 5 [0/50000 (0%)]	Loss: 1.871926
    Train Epoch: 5 [6400/50000 (13%)]	Loss: 1.773793
    Train Epoch: 5 [12800/50000 (26%)]	Loss: 1.581153
    Train Epoch: 5 [19200/50000 (38%)]	Loss: 1.795480
    Train Epoch: 5 [25600/50000 (51%)]	Loss: 1.530740
    Train Epoch: 5 [32000/50000 (64%)]	Loss: 1.672532
    Train Epoch: 5 [38400/50000 (77%)]	Loss: 1.720205
    Train Epoch: 5 [44800/50000 (90%)]	Loss: 1.626394

    Test set: Average loss: 1.7304, Accuracy: 4027/10000 (40%)

    Train Epoch: 6 [0/50000 (0%)]	Loss: 1.388340
    Train Epoch: 6 [6400/50000 (13%)]	Loss: 1.758039
    Train Epoch: 6 [12800/50000 (26%)]	Loss: 1.954456
    Train Epoch: 6 [19200/50000 (38%)]	Loss: 1.699351
    Train Epoch: 6 [25600/50000 (51%)]	Loss: 1.782924
    Train Epoch: 6 [32000/50000 (64%)]	Loss: 1.753234
    Train Epoch: 6 [38400/50000 (77%)]	Loss: 1.775223
    Train Epoch: 6 [44800/50000 (90%)]	Loss: 1.695442

    Test set: Average loss: 1.7352, Accuracy: 3948/10000 (39%)

    Train Epoch: 7 [0/50000 (0%)]	Loss: 1.875461
    Train Epoch: 7 [6400/50000 (13%)]	Loss: 1.833660
    Train Epoch: 7 [12800/50000 (26%)]	Loss: 1.605772
    Train Epoch: 7 [19200/50000 (38%)]	Loss: 1.563710
    Train Epoch: 7 [25600/50000 (51%)]	Loss: 1.603241
    Train Epoch: 7 [32000/50000 (64%)]	Loss: 1.714775
    Train Epoch: 7 [38400/50000 (77%)]	Loss: 1.542665
    Train Epoch: 7 [44800/50000 (90%)]	Loss: 1.767391

    Test set: Average loss: 1.7362, Accuracy: 4013/10000 (40%)

    Train Epoch: 8 [0/50000 (0%)]	Loss: 1.421854
    Train Epoch: 8 [6400/50000 (13%)]	Loss: 1.629634
    Train Epoch: 8 [12800/50000 (26%)]	Loss: 1.912728
    Train Epoch: 8 [19200/50000 (38%)]	Loss: 1.892627
    Train Epoch: 8 [25600/50000 (51%)]	Loss: 1.579783
    Train Epoch: 8 [32000/50000 (64%)]	Loss: 1.827596
    Train Epoch: 8 [38400/50000 (77%)]	Loss: 1.507336
    Train Epoch: 8 [44800/50000 (90%)]	Loss: 1.438141

    Test set: Average loss: 1.7259, Accuracy: 4054/10000 (41%)

    Train Epoch: 9 [0/50000 (0%)]	Loss: 1.662185
    Train Epoch: 9 [6400/50000 (13%)]	Loss: 1.776717
    Train Epoch: 9 [12800/50000 (26%)]	Loss: 1.680503
    Train Epoch: 9 [19200/50000 (38%)]	Loss: 1.859430
    Train Epoch: 9 [25600/50000 (51%)]	Loss: 1.778492
    Train Epoch: 9 [32000/50000 (64%)]	Loss: 1.746263
    Train Epoch: 9 [38400/50000 (77%)]	Loss: 1.501379
    Train Epoch: 9 [44800/50000 (90%)]	Loss: 1.707004

    Test set: Average loss: 1.7169, Accuracy: 4104/10000 (41%)

    Train Epoch: 10 [0/50000 (0%)]	Loss: 1.754747
    Train Epoch: 10 [6400/50000 (13%)]	Loss: 1.832298
    Train Epoch: 10 [12800/50000 (26%)]	Loss: 1.714557
    Train Epoch: 10 [19200/50000 (38%)]	Loss: 1.763718
    Train Epoch: 10 [25600/50000 (51%)]	Loss: 1.764659
    Train Epoch: 10 [32000/50000 (64%)]	Loss: 1.577314
    Train Epoch: 10 [38400/50000 (77%)]	Loss: 1.768530
    Train Epoch: 10 [44800/50000 (90%)]	Loss: 1.550247

    Test set: Average loss: 1.7397, Accuracy: 3912/10000 (39%)

### :: TASK 3.2 ::

Add code to create a convolutional network architecture as below.

  - Convolution with 5 by 5 filters, 16 feature maps + Tanh nonlinearity.
  - 2 by 2 max pooling.
  - Convolution with 5 by 5 filters, 128 feature maps + Tanh nonlinearity.
  - 2 by 2 max pooling.
  - Flatten to vector.
  - Linear layer with 64 hidden units + Tanh nonlinearity.
  - Linear layer to 10 output units.

```
class ConvNet(nn.Module):
    def __init__(self):
        super(ConvNet, self).__init__()

        self.conv1 = nn.Conv2d(3, 16, kernel_size = 5, bias=False)
        self.t1 = nn.Tanh()
        self.pool1 = nn.MaxPool2d(kernel_size = 2)

        self.conv2 = nn.Conv2d(16, 128, kernel_size = 5, bias=False)
        self.t2 = nn.Tanh()
        self.pool2 = nn.MaxPool2d(kernel_size = 2)

        self.flat = nn.Flatten()

        self.linear1 = nn.Linear(128 * 5 * 5, 64, bias=False)
        self.t3 = nn.Tanh()

        self.linear2 = nn.Linear(64, 10, bias=False)

    def forward(self, input):
        x = self.conv1(input)
        x = self.t1(x)
        x = self.pool1(x)
        x = self.conv2(x)
        x = self.t2(x)
        x = self.pool2(x)
        x = self.flat(x)
        x = self.linear1(x)
        x = self.t3(x)
        output = self.linear2(x)

        return output
```

### :: TASK 3.3 ::

Some of the functions in a CNN must be non-linear. Why?

If we use linear functions in a CNN, the whole network will be linear and then will be equivalent to a single layer network. Most of the real world problems are non linear and have to be approximated by non linear function. Introducing non linear activations will increase the order of the network which will be able to approximate more complicated functions.

### :: TASK 3.4 ::
Train the CNN for 20 epochs on the CIFAR-10 training set.

```
# Train
network = ConvNet().to(device)
optimizer = optim.SGD(network.parameters(), lr = lr)
for epoch in range(1, 21):
    train(epoch, network)
    test(network)
```

    Train Epoch: 1 [0/50000 (0%)]	Loss: 2.298941
    Train Epoch: 1 [6400/50000 (13%)]	Loss: 2.202316
    Train Epoch: 1 [12800/50000 (26%)]	Loss: 2.010598
    Train Epoch: 1 [19200/50000 (38%)]	Loss: 2.035936
    Train Epoch: 1 [25600/50000 (51%)]	Loss: 1.921243
    Train Epoch: 1 [32000/50000 (64%)]	Loss: 1.888596
    Train Epoch: 1 [38400/50000 (77%)]	Loss: 1.891539
    Train Epoch: 1 [44800/50000 (90%)]	Loss: 1.800411

    /usr/local/lib/python3.6/dist-packages/torch/nn/_reduction.py:43: UserWarning: size_average and reduce args will be deprecated, please use reduction='sum' instead.
      warnings.warn(warning.format(ret))

    Test set: Average loss: 1.8385, Accuracy: 3589/10000 (36%)

    Train Epoch: 2 [0/50000 (0%)]	Loss: 1.866095
    Train Epoch: 2 [6400/50000 (13%)]	Loss: 1.676759
    Train Epoch: 2 [12800/50000 (26%)]	Loss: 1.850231
    Train Epoch: 2 [19200/50000 (38%)]	Loss: 1.866604
    Train Epoch: 2 [25600/50000 (51%)]	Loss: 1.821444
    Train Epoch: 2 [32000/50000 (64%)]	Loss: 1.669572
    Train Epoch: 2 [38400/50000 (77%)]	Loss: 1.751738
    Train Epoch: 2 [44800/50000 (90%)]	Loss: 1.756302

    Test set: Average loss: 1.6728, Accuracy: 4100/10000 (41%)

    Train Epoch: 3 [0/50000 (0%)]	Loss: 1.445225
    Train Epoch: 3 [6400/50000 (13%)]	Loss: 1.709095
    Train Epoch: 3 [12800/50000 (26%)]	Loss: 1.666870
    Train Epoch: 3 [19200/50000 (38%)]	Loss: 1.614295
    Train Epoch: 3 [25600/50000 (51%)]	Loss: 1.584283
    Train Epoch: 3 [32000/50000 (64%)]	Loss: 1.532226
    Train Epoch: 3 [38400/50000 (77%)]	Loss: 1.616704
    Train Epoch: 3 [44800/50000 (90%)]	Loss: 1.611693

    Test set: Average loss: 1.5722, Accuracy: 4472/10000 (45%)

    Train Epoch: 4 [0/50000 (0%)]	Loss: 1.571411
    Train Epoch: 4 [6400/50000 (13%)]	Loss: 1.562377
    Train Epoch: 4 [12800/50000 (26%)]	Loss: 1.398550
    Train Epoch: 4 [19200/50000 (38%)]	Loss: 1.480268
    Train Epoch: 4 [25600/50000 (51%)]	Loss: 1.572160
    Train Epoch: 4 [32000/50000 (64%)]	Loss: 1.377869
    Train Epoch: 4 [38400/50000 (77%)]	Loss: 1.436360
    Train Epoch: 4 [44800/50000 (90%)]	Loss: 1.425322

    Test set: Average loss: 1.5529, Accuracy: 4479/10000 (45%)

    Train Epoch: 5 [0/50000 (0%)]	Loss: 1.665213
    Train Epoch: 5 [6400/50000 (13%)]	Loss: 1.374503
    Train Epoch: 5 [12800/50000 (26%)]	Loss: 1.207374
    Train Epoch: 5 [19200/50000 (38%)]	Loss: 1.553743
    Train Epoch: 5 [25600/50000 (51%)]	Loss: 1.328926
    Train Epoch: 5 [32000/50000 (64%)]	Loss: 1.539381
    Train Epoch: 5 [38400/50000 (77%)]	Loss: 1.130966
    Train Epoch: 5 [44800/50000 (90%)]	Loss: 1.349132

    Test set: Average loss: 1.4087, Accuracy: 4939/10000 (49%)

    Train Epoch: 6 [0/50000 (0%)]	Loss: 1.478847
    Train Epoch: 6 [6400/50000 (13%)]	Loss: 1.402545
    Train Epoch: 6 [12800/50000 (26%)]	Loss: 1.233774
    Train Epoch: 6 [19200/50000 (38%)]	Loss: 1.283921
    Train Epoch: 6 [25600/50000 (51%)]	Loss: 1.395489
    Train Epoch: 6 [32000/50000 (64%)]	Loss: 1.388797
    Train Epoch: 6 [38400/50000 (77%)]	Loss: 1.346727
    Train Epoch: 6 [44800/50000 (90%)]	Loss: 1.544198

    Test set: Average loss: 1.4306, Accuracy: 5001/10000 (50%)

    Train Epoch: 7 [0/50000 (0%)]	Loss: 1.423355
    Train Epoch: 7 [6400/50000 (13%)]	Loss: 1.212918
    Train Epoch: 7 [12800/50000 (26%)]	Loss: 1.442119
    Train Epoch: 7 [19200/50000 (38%)]	Loss: 1.400825
    Train Epoch: 7 [25600/50000 (51%)]	Loss: 1.175419
    Train Epoch: 7 [32000/50000 (64%)]	Loss: 1.418821
    Train Epoch: 7 [38400/50000 (77%)]	Loss: 1.502936
    Train Epoch: 7 [44800/50000 (90%)]	Loss: 1.181840

    Test set: Average loss: 1.2829, Accuracy: 5445/10000 (54%)

    Train Epoch: 8 [0/50000 (0%)]	Loss: 1.254811
    Train Epoch: 8 [6400/50000 (13%)]	Loss: 1.448616
    Train Epoch: 8 [12800/50000 (26%)]	Loss: 1.276739
    Train Epoch: 8 [19200/50000 (38%)]	Loss: 1.181704
    Train Epoch: 8 [25600/50000 (51%)]	Loss: 1.158188
    Train Epoch: 8 [32000/50000 (64%)]	Loss: 1.139109
    Train Epoch: 8 [38400/50000 (77%)]	Loss: 1.193539
    Train Epoch: 8 [44800/50000 (90%)]	Loss: 1.213788

    Test set: Average loss: 1.2444, Accuracy: 5596/10000 (56%)

    Train Epoch: 9 [0/50000 (0%)]	Loss: 1.254229
    Train Epoch: 9 [6400/50000 (13%)]	Loss: 1.022228
    Train Epoch: 9 [12800/50000 (26%)]	Loss: 1.239687
    Train Epoch: 9 [19200/50000 (38%)]	Loss: 1.165975
    Train Epoch: 9 [25600/50000 (51%)]	Loss: 0.896081
    Train Epoch: 9 [32000/50000 (64%)]	Loss: 1.291614
    Train Epoch: 9 [38400/50000 (77%)]	Loss: 1.281082
    Train Epoch: 9 [44800/50000 (90%)]	Loss: 0.959042

    Test set: Average loss: 1.2136, Accuracy: 5587/10000 (56%)

    Train Epoch: 10 [0/50000 (0%)]	Loss: 1.216674
    Train Epoch: 10 [6400/50000 (13%)]	Loss: 0.979816
    Train Epoch: 10 [12800/50000 (26%)]	Loss: 1.054127
    Train Epoch: 10 [19200/50000 (38%)]	Loss: 0.939696
    Train Epoch: 10 [25600/50000 (51%)]	Loss: 1.041137
    Train Epoch: 10 [32000/50000 (64%)]	Loss: 1.088344
    Train Epoch: 10 [38400/50000 (77%)]	Loss: 1.339295
    Train Epoch: 10 [44800/50000 (90%)]	Loss: 1.308453

    Test set: Average loss: 1.2561, Accuracy: 5541/10000 (55%)

    Train Epoch: 11 [0/50000 (0%)]	Loss: 1.019984
    Train Epoch: 11 [6400/50000 (13%)]	Loss: 1.088274
    Train Epoch: 11 [12800/50000 (26%)]	Loss: 1.070710
    Train Epoch: 11 [19200/50000 (38%)]	Loss: 0.997315
    Train Epoch: 11 [25600/50000 (51%)]	Loss: 1.103586
    Train Epoch: 11 [32000/50000 (64%)]	Loss: 1.256731
    Train Epoch: 11 [38400/50000 (77%)]	Loss: 0.986534
    Train Epoch: 11 [44800/50000 (90%)]	Loss: 1.125203

    Test set: Average loss: 1.1732, Accuracy: 5846/10000 (58%)

    Train Epoch: 12 [0/50000 (0%)]	Loss: 1.088615
    Train Epoch: 12 [6400/50000 (13%)]	Loss: 1.028450
    Train Epoch: 12 [12800/50000 (26%)]	Loss: 1.304997
    Train Epoch: 12 [19200/50000 (38%)]	Loss: 1.067565
    Train Epoch: 12 [25600/50000 (51%)]	Loss: 1.380940
    Train Epoch: 12 [32000/50000 (64%)]	Loss: 1.131100
    Train Epoch: 12 [38400/50000 (77%)]	Loss: 1.071955
    Train Epoch: 12 [44800/50000 (90%)]	Loss: 0.887015

    Test set: Average loss: 1.2541, Accuracy: 5656/10000 (57%)

    Train Epoch: 13 [0/50000 (0%)]	Loss: 1.077803
    Train Epoch: 13 [6400/50000 (13%)]	Loss: 1.313343
    Train Epoch: 13 [12800/50000 (26%)]	Loss: 0.897357
    Train Epoch: 13 [19200/50000 (38%)]	Loss: 0.848234
    Train Epoch: 13 [25600/50000 (51%)]	Loss: 1.001194
    Train Epoch: 13 [32000/50000 (64%)]	Loss: 0.788305
    Train Epoch: 13 [38400/50000 (77%)]	Loss: 1.189513
    Train Epoch: 13 [44800/50000 (90%)]	Loss: 0.800780

    Test set: Average loss: 1.1373, Accuracy: 6020/10000 (60%)

    Train Epoch: 14 [0/50000 (0%)]	Loss: 0.791189
    Train Epoch: 14 [6400/50000 (13%)]	Loss: 0.896521
    Train Epoch: 14 [12800/50000 (26%)]	Loss: 1.001556
    Train Epoch: 14 [19200/50000 (38%)]	Loss: 0.913710
    Train Epoch: 14 [25600/50000 (51%)]	Loss: 0.862271
    Train Epoch: 14 [32000/50000 (64%)]	Loss: 1.026798
    Train Epoch: 14 [38400/50000 (77%)]	Loss: 0.947144
    Train Epoch: 14 [44800/50000 (90%)]	Loss: 1.036307

    Test set: Average loss: 1.1682, Accuracy: 5873/10000 (59%)

    Train Epoch: 15 [0/50000 (0%)]	Loss: 0.927051
    Train Epoch: 15 [6400/50000 (13%)]	Loss: 0.936478
    Train Epoch: 15 [12800/50000 (26%)]	Loss: 0.843673
    Train Epoch: 15 [19200/50000 (38%)]	Loss: 0.889138
    Train Epoch: 15 [25600/50000 (51%)]	Loss: 1.011364
    Train Epoch: 15 [32000/50000 (64%)]	Loss: 1.001534
    Train Epoch: 15 [38400/50000 (77%)]	Loss: 0.880909
    Train Epoch: 15 [44800/50000 (90%)]	Loss: 1.035468

    Test set: Average loss: 1.0792, Accuracy: 6206/10000 (62%)

    Train Epoch: 16 [0/50000 (0%)]	Loss: 0.925243
    Train Epoch: 16 [6400/50000 (13%)]	Loss: 0.952097
    Train Epoch: 16 [12800/50000 (26%)]	Loss: 0.732091
    Train Epoch: 16 [19200/50000 (38%)]	Loss: 0.688841
    Train Epoch: 16 [25600/50000 (51%)]	Loss: 0.941330
    Train Epoch: 16 [32000/50000 (64%)]	Loss: 0.854820
    Train Epoch: 16 [38400/50000 (77%)]	Loss: 1.016635
    Train Epoch: 16 [44800/50000 (90%)]	Loss: 0.809035

    Test set: Average loss: 1.0663, Accuracy: 6259/10000 (63%)

    Train Epoch: 17 [0/50000 (0%)]	Loss: 0.878338
    Train Epoch: 17 [6400/50000 (13%)]	Loss: 0.979924
    Train Epoch: 17 [12800/50000 (26%)]	Loss: 0.662995
    Train Epoch: 17 [19200/50000 (38%)]	Loss: 0.642502
    Train Epoch: 17 [25600/50000 (51%)]	Loss: 0.782278
    Train Epoch: 17 [32000/50000 (64%)]	Loss: 0.924522
    Train Epoch: 17 [38400/50000 (77%)]	Loss: 0.884602
    Train Epoch: 17 [44800/50000 (90%)]	Loss: 0.908655

    Test set: Average loss: 0.9883, Accuracy: 6556/10000 (66%)

    Train Epoch: 18 [0/50000 (0%)]	Loss: 0.820326
    Train Epoch: 18 [6400/50000 (13%)]	Loss: 0.698531
    Train Epoch: 18 [12800/50000 (26%)]	Loss: 0.832400
    Train Epoch: 18 [19200/50000 (38%)]	Loss: 0.945531
    Train Epoch: 18 [25600/50000 (51%)]	Loss: 0.868432
    Train Epoch: 18 [32000/50000 (64%)]	Loss: 0.684544
    Train Epoch: 18 [38400/50000 (77%)]	Loss: 0.925154
    Train Epoch: 18 [44800/50000 (90%)]	Loss: 0.982910

    Test set: Average loss: 1.0591, Accuracy: 6248/10000 (62%)

    Train Epoch: 19 [0/50000 (0%)]	Loss: 0.736740
    Train Epoch: 19 [6400/50000 (13%)]	Loss: 0.834759
    Train Epoch: 19 [12800/50000 (26%)]	Loss: 0.907407
    Train Epoch: 19 [19200/50000 (38%)]	Loss: 0.954075
    Train Epoch: 19 [25600/50000 (51%)]	Loss: 0.814366
    Train Epoch: 19 [32000/50000 (64%)]	Loss: 1.035993
    Train Epoch: 19 [38400/50000 (77%)]	Loss: 0.724826
    Train Epoch: 19 [44800/50000 (90%)]	Loss: 0.876369

    Test set: Average loss: 1.1338, Accuracy: 6103/10000 (61%)

    Train Epoch: 20 [0/50000 (0%)]	Loss: 0.981874
    Train Epoch: 20 [6400/50000 (13%)]	Loss: 0.828096
    Train Epoch: 20 [12800/50000 (26%)]	Loss: 0.715917
    Train Epoch: 20 [19200/50000 (38%)]	Loss: 0.816338
    Train Epoch: 20 [25600/50000 (51%)]	Loss: 0.790289
    Train Epoch: 20 [32000/50000 (64%)]	Loss: 0.699175
    Train Epoch: 20 [38400/50000 (77%)]	Loss: 0.685722
    Train Epoch: 20 [44800/50000 (90%)]	Loss: 1.035971

    Test set: Average loss: 0.9408, Accuracy: 6743/10000 (67%)

### :: TASK 3.5 ::
Plot the first convolutional layer weights as images after the last epoch. (Hint threads: [#1](https://discuss.pytorch.org/t/understanding-deep-network-visualize-weights/2060/2?u=smth) [#2](https://github.com/pytorch/vision#utils) )

```
def vis_filter(y):
    # Scale between [0, 1]
    out = (y - y.min().expand(y.size())) / (y.max() - y.min()).expand(y.size())
    # Create a grid of images
    out = torchvision.utils.make_grid(out, nrow=8, padding=1)
    # Convert to numpy image
    out = np.transpose(out.detach().numpy(), (1, 2, 0))
    # Show
    plt.figure(figsize= (20, 2))
    plt.imshow(out)
    # Remove grid
    plt.gca().grid(False)

network = network.to("cpu")
print(network.conv1.weight.size())
vis_filter(network.conv1.weight)
```

### :: TASK 3.6 ::

What is the dimensionality of the weights at each layer? How many parameters are there in total in this CNN architecture?

In the first convolutional layer there are $16$ filters and $5*5*3$ weights per filter. The second one has $128$ filters and $5*5*16$ weights. The first fully connected layer is of size $64*3200$. The second fully connected layer is of size $10*64$ .The total number of parameters is: $16*5*5*3+128*5*5*16+64*3200+10*64 = 257840$

## Useful resources

  - [PyTorch tutorial](http://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html#sphx-glr-beginner-blitz-neural-networks-tutorial-py)
  - [MNIST example](https://github.com/pytorch/examples/tree/master/mnist)

## AUTHORSHIP STATEMENT

I declare that the preceding work was the sole result of my own effort and that I have not used any code or results from third-parties.

AMIN DHAOU


# Programación lineal

### Anuncios varios
+ Encuesta
+ Clase 18 de Febrero  (Martes 19 de feb 9-11 salón por definir)
+ Exámen 1  (28 de Febrero)
+ Proyecto  (7 de Marzo)

> La programación lineal es el campo de la optimización matemática dedicado a maximizar o minimizar (optimizar) funciones lineales, denominada función objetivo, de tal forma que las variables de dicha función estén sujetas a una serie de restricciones expresadas mediante un sistema de ecuaciones o inecuaciones también lineales.

**Referencias:**
- https://es.wikipedia.org/wiki/Programaci%C3%B3n_lineal
- https://docs.scipy.org/doc/scipy-0.18.1/reference/optimize.html
- http://bdigital.unal.edu.co/5037/4/guillermojimenezlozano.2006_Parte1.pdf

## 1. Apuntes históricos

- 1826:	Joseph Fourier anticipa la programación lineal. Carl Friedrich Gauss resuelve ecuaciones lineales por eliminación "gaussiana".
- 1902:	Gyula Farkas concibe un método para resolver sistemas de inecuaciones.
- Es hasta la Segunda Guerra Mundial que se plantea la programación lineal como un modelo matemático para planificar gastos y retornos, de modo que se reduzcan costos de guerra y aumentar pérdidas del enemigo. Secreto hasta 1947 (posguerra).
- 1947:	George Dantzig publica el algoritmo simplex y John von Neumann desarrolló la teoría de la dualidad. Se sabe que Leonid Kantoróvich también formuló la teoría en forma independiente.
- Fue usado por muchas industrias en la planificación diaria.

**Hasta acá, tiempos exponenciales de solución. Lo siguiente, tiempo polinomial.**

- 1979: Leonid Khachiyan, diseñó el llamado Algoritmo del elipsoide, a través del cual demostró que el problema de la programación lineal es resoluble de manera eficiente, es decir, en tiempo polinomial.
- 1984: Narendra Karmarkar introduce el método del punto interior para resolver problemas de programación lineal.

## 2. Motivación

Ya la clase pasada habíamos mencionado que cuando se quería optimizar una función de varias variables con restricciones, se podía aplicar siempre el método de Multiplicadores de Lagrange. Sin embargo, este método es computacionalmente muy complejo conforme crece el número de variables.

Por tanto, cuando la función a optimizar y las restricciones son de caracter lineal, los métodos de solución que se pueden desarrollar son computacionalmente eficientes, por lo que es útil realizar la distinción.

## 3. Problemas de programación lineal

### 3.1. Ejemplo básico
Una multinacional farmacéutica desea fabricar un compuesto nutritivo a base de dos productos A y B.  El producto A contiene  $30\%$ de proteínas,  un $1\%$ de grasas y un  $10\%$ de azúcares.  El producto B  contiene  un  $5\%$  de proteínas,  un  $7\%$ de grasas  y un  $10\%$  de  azúcares.

El compuesto tiene que tener,  al menos, $25g$ de proteínas,  $6g$ de grasas y $30g$ de azúcares.  El coste del producto A es de $0.6$ u.m./g  y el de B es de $0.2$  u.m./g

Se desea encontrar la cantidad en gramos de cada producto para que el coste total sea mínimo.

Formular el problema de decidir cuánto hacer de cada producto como un problema de programación lineal.

#### Solución

Sean:
- $x_A$: la cantidad de gramos de A a ser producidas, y
- $x_B$: la cantidad de gramos de B a ser producidas en la semana.

Notar que lo que se quiere es minimizar $0.6x_A+0.2x_B$.

Restricciones:
1. El compuesto debe tener **al menos** $25 g$ de proteína: $30\%x_A+5\%x_B\geq 25 \Rightarrow 0.3x_A+0.05x_B\geq 25$.
2. El compuesto debe tener **al menos** $6 g$ de grasa: $1\%x_A+7\%x_B\geq 6 \Rightarrow 0.01x_A+0.07x_B\geq 6$.
3. El compuesto debe tener **al menos** $30 g$ de azúcares: $10\%x_A+10\%x_B\geq 30 \Rightarrow 0.1x_A+0.1x_B\geq 30$.

Finalmente, el problema puede ser expresado en la forma explicada como:
\begin{equation}
\begin{array}{ll}
\min_{x_A,x_B} & 0.6x_A+0.2x_B \\
\text{s. a. }  & -0.3x_A-0.05x_B\leq -25 \\
               & -0.01x_A-0.07x_B\leq -6 \\
               & -0.1x_A-0.1x_B\leq -30,
\end{array}
\end{equation}

o, eqivalentemente
\begin{equation}
\begin{array}{ll}
\min_{\boldsymbol{x}} & \boldsymbol{c}^\top\boldsymbol{x} \\
\text{s. a. }        & \boldsymbol{A}_{eq}\boldsymbol{x}=\boldsymbol{b}_{eq} \\
                     & \boldsymbol{A}\boldsymbol{x}\leq\boldsymbol{b},
\end{array}
\end{equation}
con
- $\boldsymbol{c}=\left[0.6 \quad 0.2\right]^\top$,
- $\boldsymbol{A}=\left[\begin{array}{cc}-0.3 & -0.05 \\ -0.01 & -0.07\\ -0.1 & -0.1\end{array}\right]$, y
- $\boldsymbol{b}=\left[-25 \quad -6\quad -30\right]^\top$.

Preferiremos, en adelante, la notación vectorial/matricial.

### 3.2. Ejemplo básico 2
Una  fábrica  de  carrocerías  de  automóviles  y camiones  tiene  dos naves.

+ En  la nave A,  para  hacer  la  carrocería  de  un  camión,  se  invierten  siete  días-operario,  para  fabricar  la  de  un  coche  se  precisan dos días-operario.

+ En la nave B se invierten tres días operario tanto en carrocerías de camión como de coche.

Por limitaciones de mano de obra y maquinaria, la nave A dispone de $300$ días operario, y la nave B de $270$ días-operario.

Si los beneficios que se obtienen por cada camión son de $600$ u.m y por  cada  automóvil  $200$ u.m,  ¿cuántas  unidades  de  cada  uno  se deben  producir  para maximizar las  ganancias?

### 3.3. En general
De acuerdo a lo descrito anteriormente, un problema de programación lineal puede escribirse en la siguiente forma:

\begin{equation}
\begin{array}{ll}
\min_{x_1,\dots,x_n} & c_1x_1+\dots+c_nx_n \\
\text{s. a. }        & a^{eq}_{j,1}x_1+\dots+a^{eq}_{j,n}x_n=b^{eq}_j \text{ para } 1\leq j\leq m_1 \\
                     & a_{k,1}x_1+\dots+a_{k,n}x_n\leq b_k \text{ para } 1\leq k\leq m_2,
\end{array}
\end{equation}
donde:
- $x_i$ para $i=1,\dots,n$ son las incógnitas o variables de decisión,
- $c_i$ para $i=1,\dots,n$ son los coeficientes de la función a optimizar,
- $a^{eq}_{j,i}$ para $j=1,\dots,m_1$ e $i=1,\dots,n$, son los coeficientes de la restricción de igualdad,
- $a_{k,i}$ para $k=1,\dots,m_2$ e $i=1,\dots,n$, son los coeficientes de la restricción de desigualdad,
- $b^{eq}_j$ para $j=1,\dots,m_1$ son valores conocidos que deben ser respetados estrictamente, y
- $b_k$ para $k=1,\dots,m_2$ son valores conocidos que no deben ser superados.

Equivalentemente, el problema puede escribirse como

\begin{equation}
\begin{array}{ll}
\min_{\boldsymbol{x}} & \boldsymbol{c}^\top\boldsymbol{x} \\
\text{s. a. }        & \boldsymbol{A}_{eq}\boldsymbol{x}=\boldsymbol{b}_{eq} \\
                     & \boldsymbol{A}\boldsymbol{x}\leq\boldsymbol{b},
\end{array}
\end{equation}
donde:
- $\boldsymbol{x}=\left[x_1\quad\dots\quad x_n\right]^\top$,
- $\boldsymbol{c}=\left[c_1\quad\dots\quad c_n\right]^\top$,
- $\boldsymbol{A}_{eq}=\left[\begin{array}{ccc}a^{eq}_{1,1} & \dots & a^{eq}_{1,n}\\ \vdots & \ddots & \vdots\\ a^{eq}_{m_1,1} & \dots & a^{eq}_{m_1,n}\end{array}\right]$,
- $\boldsymbol{A}=\left[\begin{array}{ccc}a_{1,1} & \dots & a_{1,n}\\ \vdots & \ddots & \vdots\\ a_{m_2,1} & \dots & a_{m_2,n}\end{array}\right]$,
- $\boldsymbol{b}_{eq}=\left[b^{eq}_1\quad\dots\quad b^{eq}_{m_1}\right]^\top$, y
- $\boldsymbol{b}=\left[b_1\quad\dots\quad b_{m_2}\right]^\top$.

**Nota:** el problema $\max_{\boldsymbol{x}}\boldsymbol{g}(\boldsymbol{x})$ es equivalente a $\min_{\boldsymbol{x}}-\boldsymbol{g}(\boldsymbol{x})$.

#### Bueno, y una vez planteado, ¿cómo se resuelve el problema?

Este problema está sencillo pues solo es en dos variables. La solución gráfica es válida.

```python
#Importat numpy y matplotlib.pyplot
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
#Definir funciones de restricción y de costo
def res1(xA):
    return (25-0.3*xA)/0.05
def res2(xA):
    return (6-0.01*xA)/0.07
def res3(xA):
    return (30-0.1*xA)/0.1
#Definir la función de costo
def z(xA,xB):
    return 0.6*xA + 0.2*xB
```

```python
#Evaluar funciones
xA = np.linspace(0,400,200)
r1 = res1(xA)
r2 = res2(xA)
r3 = res3(xA)
```

```python
#Graficar
plt.figure(figsize = (8,6))
plt.plot(xA,r1,'b--',label='res1')
plt.plot(xA,r2,'r-.',label='res2')
plt.plot(xA,r3,'g:',label='res3')
plt.legend(loc = 'best')
plt.xlabel('$x_A$')
plt.ylabel('$x_B$')
plt.axis([0,400,0,400])
plt.show()
```

```python
#Evaluar función de costo y comparar
z(40,260), z(250,60)
```

    (76.0, 162.0)

```python
#Volver a graficar con solución

```

**Actividad.** Mónica hace aretes y cadenitas de joyería. Es tan buena, que todo lo que hace lo vende.

Le toma 30 minutos hacer un par de aretes y una hora hacer una cadenita, y como Mónica también es estudihambre, solo dispone de 10 horas a la semana para hacer las joyas. Por otra parte, el material que compra solo le alcanza para hacer 15 unidades (el par de aretes cuenta como unidad) de joyas por semana.

La utilidad que le deja la venta de las joyas es $\$15$ en cada par de aretes y $\$20$ en cada cadenita.

¿Cuántos pares de aretes y cuántas cadenitas debería hacer Mónica para maximizar su utilidad?

Formular el problema en la forma explicada y obtener la solución gráfica (puede ser a mano).

**Diez minutos: quien primero lo haga, pasará a explicarlo al tablero y le subiré la nota de alguna tarea a 100. Debe salir a explicar el problema.**

## 4. ¿Cómo se resuelve en python?

### 4.1 Librería `SciPy`

`SciPy` es un software de código abierto basado en `Python` para matemáticas, ciencia e ingeniería.

En particular, los siguientes son algunos de los paquetes básicos:
- `NumPy`
- `SymPy`
- `matplotlib`
- **Librería `SciPy`**
- `pandas`

La **Librería `SciPy`** es uno de los paquetes principales y provee varias rutinas numéricas eficientes. Entre ellas, para integración numérica y optimización.

En esta clase, y en lo que resta del módulo, estaremos utilizando el módulo `optimize` de la librería `SciPy`.

**Importémoslo**

```python
# Importar el módulo optimize de la librería scipy
import scipy.optimize as opt
```

El módulo `optimize` que acabamos de importar contiene varias funciones para optimización y búsqueda de raices ($f(x)=0$). Entre ellas se encuentra la función `linprog`

```python
# Función linprog del módulo optimize
help(opt.linprog)
```

    Help on function linprog in module scipy.optimize._linprog:

    linprog(c, A_ub=None, b_ub=None, A_eq=None, b_eq=None, bounds=None, method='simplex', callback=None, options=None)
        Minimize a linear objective function subject to linear
        equality and inequality constraints.

        Linear Programming is intended to solve the following problem form::

            Minimize:     c^T * x

            Subject to:   A_ub * x <= b_ub
                          A_eq * x == b_eq

        Parameters
        ----------
        c : array_like
            Coefficients of the linear objective function to be minimized.
        A_ub : array_like, optional
            2-D array which, when matrix-multiplied by ``x``, gives the values of
            the upper-bound inequality constraints at ``x``.
        b_ub : array_like, optional
            1-D array of values representing the upper-bound of each inequality
            constraint (row) in ``A_ub``.
        A_eq : array_like, optional
            2-D array which, when matrix-multiplied by ``x``, gives the values of
            the equality constraints at ``x``.
        b_eq : array_like, optional
            1-D array of values representing the RHS of each equality constraint
            (row) in ``A_eq``.
        bounds : sequence, optional
            ``(min, max)`` pairs for each element in ``x``, defining
            the bounds on that parameter. Use None for one of ``min`` or
            ``max`` when there is no bound in that direction. By default
            bounds are ``(0, None)`` (non-negative)
            If a sequence containing a single tuple is provided, then ``min`` and
            ``max`` will be applied to all variables in the problem.
        method : str, optional
            Type of solver.  :ref:`'simplex' <optimize.linprog-simplex>`
            and :ref:`'interior-point' <optimize.linprog-interior-point>`
            are supported.
        callback : callable, optional (simplex only)
            If a callback function is provide, it will be called within each
            iteration of the simplex algorithm. The callback must have the
            signature ``callback(xk, **kwargs)`` where ``xk`` is the current
            solution vector and ``kwargs`` is a dictionary containing the
            following::

                "tableau" : The current Simplex algorithm tableau
                "nit" : The current iteration.
                "pivot" : The pivot (row, column) used for the next iteration.
                "phase" : Whether the algorithm is in Phase 1 or Phase 2.
                "basis" : The indices of the columns of the basic variables.

        options : dict, optional
            A dictionary of solver options. All methods accept the following
            generic options:

                maxiter : int
                    Maximum number of iterations to perform.
                disp : bool
                    Set to True to print convergence messages.

            For method-specific options, see :func:`show_options('linprog')`.

        Returns
        -------
        A `scipy.optimize.OptimizeResult` consisting of the following fields:

            x : ndarray
                The independent variable vector which optimizes the linear
                programming problem.
            fun : float
                Value of the objective function.
            slack : ndarray
                The values of the slack variables.  Each slack variable corresponds
                to an inequality constraint.  If the slack is zero, then the
                corresponding constraint is active.
            success : bool
                Returns True if the algorithm succeeded in finding an optimal
                solution.
            status : int
                An integer representing the exit status of the optimization::

                     0 : Optimization terminated successfully
                     1 : Iteration limit reached
                     2 : Problem appears to be infeasible
                     3 : Problem appears to be unbounded

            nit : int
                The number of iterations performed.
            message : str
                A string descriptor of the exit status of the optimization.

        See Also
        --------
        show_options : Additional options accepted by the solvers

        Notes
        -----
        This section describes the available solvers that can be selected by the
        'method' parameter. The default method
        is :ref:`Simplex <optimize.linprog-simplex>`.
        :ref:`Interior point <optimize.linprog-interior-point>` is also available.

        Method *simplex* uses the simplex algorithm (as it relates to linear
        programming, NOT the Nelder-Mead simplex) [1]_, [2]_. This algorithm
        should be reasonably reliable and fast for small problems.

        .. versionadded:: 0.15.0

        Method *interior-point* uses the primal-dual path following algorithm
        as outlined in [4]_. This algorithm is intended to provide a faster
        and more reliable alternative to *simplex*, especially for large,
        sparse problems. Note, however, that the solution returned may be slightly
        less accurate than that of the simplex method and may not correspond with a
        vertex of the polytope defined by the constraints.

        References
        ----------
        .. [1] Dantzig, George B., Linear programming and extensions. Rand
               Corporation Research Study Princeton Univ. Press, Princeton, NJ,
               1963
        .. [2] Hillier, S.H. and Lieberman, G.J. (1995), "Introduction to
               Mathematical Programming", McGraw-Hill, Chapter 4.
        .. [3] Bland, Robert G. New finite pivoting rules for the simplex method.
               Mathematics of Operations Research (2), 1977: pp. 103-107.
        .. [4] Andersen, Erling D., and Knud D. Andersen. "The MOSEK interior point
               optimizer for linear programming: an implementation of the
               homogeneous algorithm." High performance optimization. Springer US,
               2000. 197-232.
        .. [5] Andersen, Erling D. "Finding all linearly dependent rows in
               large-scale linear programming." Optimization Methods and Software
               6.3 (1995): 219-227.
        .. [6] Freund, Robert M. "Primal-Dual Interior-Point Methods for Linear
               Programming based on Newton's Method." Unpublished Course Notes,
               March 2004. Available 2/25/2017 at
               https://ocw.mit.edu/courses/sloan-school-of-management/15-084j-nonlinear-programming-spring-2004/lecture-notes/lec14_int_pt_mthd.pdf
        .. [7] Fourer, Robert. "Solving Linear Programs by Interior-Point Methods."
               Unpublished Course Notes, August 26, 2005. Available 2/25/2017 at
               http://www.4er.org/CourseNotes/Book%20B/B-III.pdf
        .. [8] Andersen, Erling D., and Knud D. Andersen. "Presolving in linear
               programming." Mathematical Programming 71.2 (1995): 221-245.
        .. [9] Bertsimas, Dimitris, and J. Tsitsiklis. "Introduction to linear
               programming." Athena Scientific 1 (1997): 997.
        .. [10] Andersen, Erling D., et al. Implementation of interior point
                methods for large scale linear programming. HEC/Universite de
                Geneve, 1996.

        Examples
        --------
        Consider the following problem:

        Minimize: f = -1*x[0] + 4*x[1]

        Subject to: -3*x[0] + 1*x[1] <= 6
                     1*x[0] + 2*x[1] <= 4
                                x[1] >= -3

        where:  -inf <= x[0] <= inf

        This problem deviates from the standard linear programming problem.
        In standard form, linear programming problems assume the variables x are
        non-negative.  Since the variables don't have standard bounds where
        0 <= x <= inf, the bounds of the variables must be explicitly set.

        There are two upper-bound constraints, which can be expressed as

        dot(A_ub, x) <= b_ub

        The input for this problem is as follows:

        >>> c = [-1, 4]
        >>> A = [[-3, 1], [1, 2]]
        >>> b = [6, 4]
        >>> x0_bounds = (None, None)
        >>> x1_bounds = (-3, None)
        >>> from scipy.optimize import linprog
        >>> res = linprog(c, A_ub=A, b_ub=b, bounds=(x0_bounds, x1_bounds),
        ...               options={"disp": True})
        Optimization terminated successfully.
             Current function value: -22.000000
             Iterations: 1
        >>> print(res)
             fun: -22.0
         message: 'Optimization terminated successfully.'
             nit: 1
           slack: array([39.,  0.])
          status: 0
         success: True
               x: array([10., -3.])

        Note the actual objective value is 11.428571.  In this case we minimized
        the negative of the objective function.

la cual resuelve problemas como los que aprendimos a plantear.

Parámetros importantes:
+ c: Vector con los coeficientes de función de costo (objetivo) lineal a minimizar.
+ A_ub: Matriz con los coeficientes de $x$ de la  restricción de desigualdad.
+ b_ub: Vector que representa los valores de cada restricción de desigualdad.
+ A_eq: Matriz con los coeficientes de $x$ de la  restricción de igualdad.
+ b_eq: Vector que representa los valores de cada restricción de igualdad.
+ bounds: (min, max) pares de cada elemento en $x$ definiendo las cotas mìnimas y màximas correspondientes. Por default $(0, None)$, no-negativo.

### 4.2 Solución del ejemplo básico con linprog
Ya hicimos la solución gráfica. Contrastemos con la solución que nos da `linprog`...

- $\boldsymbol{c}=\left[0.6 \quad 0.2\right]^\top$,
- $\boldsymbol{A}=\left[\begin{array}{cc}-0.3 & -0.05 \\ -0.01 & -0.07\\ -0.1 & -0.1\end{array}\right]$, y
- $\boldsymbol{b}=\left[-25 \quad -6\quad -30\right]^\top$.

```python
# Importar numpy para crear las matrices
import numpy as np
```

```python
# Crear las matrices para resolver el problema
c = np.array([0.6, 0.2])
A = np.array([[-0.3,-0.05],
              [-0.01,-0.07],
              [-0.1,-0.1]])
b = np.array([-25,-6,-30])
```

```python

```

```python
# Resolver utilizando linprog
resultado = opt.linprog(c, A_ub=A,b_ub=b)
```

```python
# Mostrar el resultado
resultado
```

         fun: 75.99999999999987
     message: 'Optimization terminated successfully.'
         nit: 5
       slack: array([ 0. , 12.6,  0. ])
      status: 0
     success: True
           x: array([ 40., 260.])

```python
# Extraer el vector solución
xs = resultado.x
xs[0]
```

    39.99999999999983

**Conclusión**
- Para minimizar el costo del compuesto nutritivo basado en los productos $A$ y $B$, se debe producir $40$ gramos de $A$ y $260$ gramos de B.
- Con esa producción, el  costo total del compuesto será de $76$ u.m.

**Actividad.** Resolver el ejemplo de Mónica y sus ventas con `linprog`

```python
# Definir matrices
c = np.array([-15, -20])
A = np.array([[0.5,1],
              [1,1]])
b = np.array([10,15])
```

```python
# Resolver con el módulo linprog
resultado_Monica = opt.linprog(c, A_ub=A,b_ub=b)
```

```python
#Mostrar solución
resultado_Monica
```

         fun: -250.0
     message: 'Optimization terminated successfully.'
         nit: 2
       slack: array([0., 0.])
      status: 0
     success: True
           x: array([10.,  5.])

## 5. Problema de transporte 1
- **Referencia**: http://bdigital.unal.edu.co/5037/4/guillermojimenezlozano.2006_Parte1.pdf

Una  empresa tiene dos factorías A y B.  En ellas  se fabrica un determinado  producto, a razón de 500  y 400 unidades por día respectivamente.  El producto  ha  de  ser distribuido posteriormente  a  tres  centros  C,  D  y  E,  que  requieren,  respectivamente,  200,  300  y  400  unidades.  Los  costos  de  transportar cada unidad del producto desde cada factoría a cada centro distribuidor son los  indicados  en la tabla  siguiente:

Factoría|C|D|E|Fabricación (Unidades)
:----|----|----|----|----
A|  50 u.m|60 u.m|10 u.m|500 u
B|  25 u.m|40 u.m|20 u.m|400 u
Demanda|200|300|400|

**¿De qué manera  deben  organizar  el transporte a fin de que los gastos sean mínimos?**

Formulemos el problema para ser resuelto por la programación lineal con
- $x_1$: unidades transportadas de la factoría "A" al centro "C"
- $x_2$: unidades transportadas de la factoría "A" al centro "D"
- $x_3$: unidades transportadas de la factoría "A" al centro "E"
- $x_4$: unidades transportadas de la factoría "B" al centro "C"
- $x_5$: unidades transportadas de la factoría "B" al centro "D"
- $x_6$: unidades transportadas de la factoría "B" al centro "E"

se tienen las siguientes ecuaciones:

Restricciones de la producción:

- $x_1 + x_2 + x_3 \leq 500$
- $x_4 + x_5 + x_6 \leq 400$

Restricciones del consumo:

- $x_1 + x_4 \geq 200$
- $x_2 + x_5 \geq 300$
- $x_3 + x_6 \geq 400$

La función objetivo será:

$$\min_{x_1,\dots,x_6}50x_1 + 60x_2 + 10x_3 + 25x_4 + 40x_5 + 20x_6$$

Resolver con `linprog`

```python
# Matrices y cotas
c = np.array([50,60,10,25,40,20])
A = np.array([[1,1,1,0,0,0],
              [0,0,0,1,1,1],
              [-1,0,0,-1,0,0],
              [0,-1,0,0,-1,0],
              [0,0,-1,0,0,-1]])
b = np.array([500,400,-200,-300,-400])
```

```python
# Resolver
resultado_transporte = opt.linprog(c, A_ub=A,b_ub=b)
```

```python
# Mostrar resultado
resultado_transporte
```

         fun: 23000.0
     message: 'Optimization terminated successfully.'
         nit: 6
       slack: array([-0.,  0.,  0.,  0.,  0.])
      status: 0
     success: True
           x: array([  0., 100., 400., 200., 200.,   0.])

**Conclusión**

- La estrategia de menor costo es llevar $100$ unidades de la Factoría "A" al cento "D", $400$ unidades de la Factoría "A" al cento "E", $200$ unidades de la Factoría "B" al cento "C" y $200$ unidades de la Factoría "B" al cento "D". El costo total de esta estrategia de transporte es $23000$  u.m.

## 6. Optimización de inversión en bonos

**Referencia:**

```python
from IPython.display import YouTubeVideo
YouTubeVideo('gukxBus8lOs')
```

El objetivo de este problema es determinar la mejor estrategia de inversión, dados diferentes tipos de bono, la máxima cantidad que puede ser invertida en cada bono, el porcentaje de retorno y los años de madurez. También hay una cantidad fija de dinero disponible ($\$750,000$). Por lo menos la mitad de este dinero debe ser invertido en bonos con 10 años o más para la madurez. Se puede invertir un máximo del $25\%$ de esta cantidad en cada bono. Finalmente, hay otra restricción que no permite usar más de $35\%$ en bonos de alto riesgo.

Existen seis (6) opciones de inversión con las letras correspondientes $A_i$

1. $A_1$:(Tasa de retorno=$8.65\%$; Años para la madurez=11, Riesgo=Bajo)
1. $A_2$:(Tasa de retorno=$9.50\%$; Años para la madurez=10, Riesgo=Alto)
1. $A_3$:(Tasa de retorno=$10.00\%$; Años para la madurez=6, Riesgo=Alto)
1. $A_4$:(Tasa de retorno=$8.75\%$; Años para la madurez=10, Riesgo=Bajo)
1. $A_5$:(Tasa de retorno=$9.25\%$; Años para la madurez=7, Riesgo=Alto)
1. $A_6$:(Tasa de retorno=$9.00\%$; Años para la madurez=13, Riesgo=Bajo)

Lo que se quiere entonces es maximizar el retorno que deja la inversión.

Este problema puede ser resuelto con programación lineal. Formalmente, puede ser descrito como:

$$\max_{A_1,A_2,...,A_6}\sum^{6}_{i=1} A_iR_i,$$

donde $A_i$ representa la cantidad invertida en la opción, y $R_i$ representa la tasa de retorno respectiva.

Plantear restricciones...

```python
# Matrices y cotas
c = -np.array([8.65, 9.5,10, 8.75, 9.25, 9])/100
A_ub = np.array([[-1,-1,0,-1,0,-1],
              [0,1,1,0,1,0]])
b_ub = np.array([-750000/2,0.35*750000])
A_eq = np.array([[1,1,1,1,1,1]])
b_eq = np.array([750000])
Ai_bounds = (0,750000/4)
```

```python
# Resolver
resultado_bonos = opt.linprog(c,A_ub,b_ub,A_eq,b_eq,bounds=(Ai_bounds,Ai_bounds,Ai_bounds,Ai_bounds,Ai_bounds,Ai_bounds,))
```

```python
# Mostrar resultado
resultado_bonos
```

         fun: -68887.5
     message: 'Optimization terminated successfully.'
         nit: 9
       slack: array([187500.,      0.,  75000.,      0.,      0., 187500.,      0.,
                0.])
      status: 0
     success: True
           x: array([112500.,  75000., 187500., 187500.,      0., 187500.])

Recordar que en el problema minimizamos $-\sum^{6}_{i=1} A_iR_i$. El rendimiento obtenido es entonces:

```python

```

**Conclusión**

-

## 7. Tarea

### 7.1. Diseño de la Dieta Óptima

Se quiere producir comida para gatos de la manera más barata, no obstante se debe también asegurar que se cumplan los datos requeridos de analisis nutricional. Por lo que se quiere variar la cantidad de cada ingrediente para cumplir con los estandares nutricionales. Los requisitos que se tienen es que en $100$ gramos, se deben tener **por lo menos** $8$ gramos de proteína y $6$ gramos de grasa. Así mismo, no se debe tener más de $2$ gramos de fibra y $0.4$ gramos de sal.

Los datos nutricionales se pueden obtener de la siguiente tabla:

Ingrediente|Proteína|Grasa|Fibra|Sal
:----|----|----|----|----
Pollo|  10.0%|08.0%|00.1%|00.2%
Carne|  20.0%|10.0%|00.5%|00.5%
Cordero|15.0%|11.0%|00.5%|00.7%
Arroz|  00.0%|01.0%|10.0%|00.2%
Trigo|  04.0%|01.0%|15.0%|00.8%
Gel|    00.0%|00.0%|00.0%|00.0%

Los costos de cada producto son:

Ingrediente|Costo por gramo
:----|----
Pollo|$\$$0.013
Carne|$\$$0.008
Cordero|$\$$0.010
Arroz|$\$$0.002
Trigo|$\$$0.005
Gel|$\$$0.001

Lo que se busca optimizar en este caso es la cantidad de productos que se debe utilizar en la comida de gato, minimizando el costo total. Para simplificar la notación use las siguientes variables:

+ $x_1:$ Gramos de pollo
+ $x_2:$ Gramos de carne
+ $x_3:$ Gramos de cordero
+ $x_4:$ Gramos de arroz
+ $x_5:$ Gramos de trigo
+ $x_6:$ Gramos de gel

La tarea consiste en plantear el problemade programación lineal que permita satisfacer las necesidades alimenticias del gato al tiempo que minimice el costo total y resolverlo con `linprog`.

### 7.2. Otro problema de distribución (de energía eléctrica)

La Comisión Federal de Electricidad **(CFE)** dispone de cuatro plantas de generación para satisfacer la demanda diaria eléctrica en cuatro ciudades, Guadalajara, León y Morelia. Las plantas $1$, $2$ y $3$ pueden satisfacer $80$, $40$ y $60$ millones de KW al día respectivamente. Las necesidades de las ciudades de Guadalajara, León y Morelia son de $70$, $40$ y $70$ millones de Kw al día respectivamente.

Los costos asociados al envío de suministro energético por cada millón de Kw entre cada planta y cada ciudad son los registrados en la siguiente tabla.

-|Guadalajara|León|Morelia
:----|----|----|----
Planta 1|5|2|7
Planta 2|3|6|6
Planta 3|6|1|2

Y por último, las restricciones del problema, van a estar dadas por las capacidades de oferta y demanda de cada planta (en millones de KW) y cada ciudad.

Para simplificar la notación use las siguientes variables:

+ $x_1$: Kw (en millones) distribuidos de la Planta 1 a Guadalajara
+ $x_2$: Kw (en millones) distribuidos de la Planta 1 a León
+ $x_3$: Kw (en millones) distribuidos de la Planta 1 a Morelia
+ $x_4$: Kw (en millones) distribuidos de la Planta 2 a Guadalajara
+ $x_5$: Kw (en millones) distribuidos de la Planta 2 a León
+ $x_6$: Kw (en millones) distribuidos de la Planta 2 a Morelia
+ $x_7$: Kw (en millones) distribuidos de la Planta 3 a Guadalajara
+ $x_8$: Kw (en millones) distribuidos de la Planta 3 a León
+ $x_9$: Kw (en millones) distribuidos de la Planta 3 a Morelia

La tarea consiste en plantear el problema de programación lineal que permita satisfacer las necesidades de todas las ciudades al tiempo que minimice los costos asociados a la distribución y resolverlo con `linprog`.

Deben crear un notebook de jupyter (archivo .ipynb) y llamarlo PrimerApellido_PrimerNombre, y subirlo a moodle.

**Definir fecha**

<footer id="attribution" style="float:right; color:#808080; background:#fff;">
Created with Jupyter by Cristian Camilo Zapata Zuluaga
</footer>