# Exercise 4: Neural Networks Learning

In this exercise, you will implement the backpropagation algorithm for neural networks and apply it to the task of hand-written digit recognition.

## Our dataset

We are given a data set in `ex4data1.mat` that contains 5000 training examples of handwritten digits. (This is exactly the same data set as in last week's exercise 3).

Each training example is a 20 pixel by 20 pixel grayscale image of the digit. Each pixel is represented by a floating point number indicating the grayscale intensity at that location. The 20 by 20 grid of pixels is “unrolled” into a 400-dimensional vector. Each of these training examples becomes a single row in our data matrix X. This gives us a 5000 by 400 matrix X where every row is a training example for a handwritten digit image.

The second part of the training set is a 5000-dimensional vector y that contains labels for the training set. Like in the last exercise, a “0” digit is labeled as “10”, while the digits “1” to “9” are labeled as “1” to “9” in their natural order.

```octave
% Load saved matrices for X and y from file
load('data/ex4data1.mat');
% The matrices X and y will now be in our Octave environment
```

Our neural network is shown in the picture below. It has 3 layers – an input layer, a hidden layer and an output layer. Recall that our inputs are pixel values of digit images. Since the images are of size 20 × 20, this gives us 400 input layer units (not counting the extra bias unit which always outputs +1).

We will begin by visualizing a subset of the training set. We reuse the 'displayDat'a function from the last exercise.

```octave
m = size(X, 1);

% Randomly select 100 data points
rand_indices = randperm(m);
sel = X(rand_indices(1:100), :);
% and display them
displayData(sel);
```

In order to get started, we have been provided with a set of network parameters $ (\Theta^{(1)},\Theta^{(2)}) $ that have neen previously trained. These are stored in ex4weights.mat. The parameters have dimensions that are sized for a neural network with 25 units in the second layer and 10 output units (corresponding to the 10 digit classes).

```octave
% Load trained parameters as matrices Theta1 and Theta2
load('data/ex4weights.mat');
% Theta1 has size 25 x 401
% Theta2 has size 10 x 26
```

```octave
%% Setup the parameters you will use for this exercise
input_layer_size  = 400;  % 20x20 Input Images of Digits
hidden_layer_size = 25;   % 25 hidden units
num_labels = 10;          % 10 labels, from 1 to 10
                          % (note that we have mapped "0" to label 10)
```

## Forward propagation and cost function

As a first step, we will implement the cost function and the gradient for the neural network. The cost function (without regularization) is defined as:

$$ J(\theta) = \frac{1}{m} \sum_{i=1}^{m} \sum_{k=1}^{K}[-y_k^{(i)} \log(h_\theta(x^{(i)})_k) - (1-y_k^{(i)}) \log(1 - h_\theta(x^{(i)})_k) ] $$

where $ K $ is the number of labels (or outputs, in our case $ K = 10$ and $ h_\theta(x^{(i)})_k = a^{(3)}_k $ is the activation value of the $k$th unit in the output layer (compare with neural network layout depicted above).

As always we make use of a couple of helper functions.

```octave
function g = sigmoid(z)
%SIGMOID Compute sigmoid function
%   g = SIGMOID(z) computes the sigmoid of z.

    % The function should work on scalar *and* matrix values
    g = zeros(size(z));

    g = 1 ./ ( 1 + exp(-z));
end
```

```octave
function g = sigmoidGradient(z)
%SIGMOIDGRADIENT returns the gradient of the sigmoid function
%evaluated at z
%   g = SIGMOIDGRADIENT(z) computes the gradient of the sigmoid function
%   evaluated at z. t.

    % The function should work on scalar *and* matrix values
    g = zeros(size(z));

    g = sigmoid(z) .* (1-sigmoid(z));
end

```

```octave
fprintf('Sigmoid gradient evaluated at [-1 -0.5 0 0.5 1]:\n  ');
g = sigmoidGradient([-1 -0.5 0 0.5 1])
```

    Sigmoid gradient evaluated at [-1 -0.5 0 0.5 1]:

    g =

       0.19661   0.23500   0.25000   0.23500   0.19661

```octave
function result = h(theta, x)
    result = (sigmoid(theta' * x'))';
end
```

### Step 1: Feedforward computation

As a first step we implement the feedforward computation that computes $ h_\theta(x^{(i)}) $ for every example $ i $ and sums the cost over all examples.

### Step 2: Backpropagation

As a second step, we will implement the backpropagation algorithm to compute the gradients
$ \frac{\partial}{\partial \theta_{ij}^{(l)}} J(\Theta) $ of our cost function.

Recall that the intuition behind the backpropagation algorithm is as follows. Given a
training example $ (x^{(t)}, y^{(t)}) $, we will first run a “forward pass” to compute
all the activations throughout the network, including the output value of the
hypothesis $ h_\theta (x{(t}) $.

Then, for each node j in layer l, we would like to compute
an “error term” $ \delta_j(l) $ that measures how much that node was “responsible”
for any errors in our output.

* For an output node, we can directly measure the difference between the
network’s activation and the true target value, and use that to define $ \delta_j^{(3)} $
(since layer 3 is the output layer):

$$ \delta_j^{(3)} = \alpha_j^{(3)} - y_j $$

* For the hidden units, we can compute $ δ_{j}(l) $ based on a weighted average of the error terms
$ \delta^{(l+1)} $ of the nodes in layer $ (l + 1) $:

$$ \delta^{(l)} = {\Theta^{(l)}}^T \delta_{(l+1)} .* g'(z^{(l)}) $$

Why? Let's assume a very simple, *linear* neural network.

$ \delta^{(l)} $ can be seen as the change of the network's cost function $ J $ in relation to a change in the output $ z^{(l)} $ of our node in layer $ l $.

$$
\begin{align}
\delta^{(l)} & = \frac{\partial}{\partial z^{(l)}} J(\theta^{l}) \\
    & = \frac{\partial J(\theta^{l})}{\partial z^{(l)}} \frac{\partial z^{(l+1)}}{\partial z^{(l+1)}}
    = \underbrace{\frac{\partial J(\theta^{l})}{\partial z^{(l+1)}}}_{= \delta^{(l+1)}} \underbrace{\frac{\partial z^{(l+1)}}{\partial z^{(l)}}}_{=(*)} \\
\end{align}
$$

Let's calculate (*):

$$
\begin{align}
\frac{\partial z^{(l+1)}}{\partial z^{(l)}} & \overbrace{=}^{k:=l+1} \frac{\partial z^{(k)}}{\partial z^{(k-1)}}
     = \frac{\partial}{\partial z^{(k-1)}}(\theta^{(k-1)} g(z^{(k-1)}))
     = \theta^{(k-1)} g'(z^{(k-1)}) \\
     & \overbrace{=}^{l=k-1} = \theta^{(l)} g'(z^{(l)}) \\
\end{align}
$$

Put together:

$$ \delta^{(l)} = \delta^{(l+1)} \theta^{(l)} g'(z^{(l)}) $$

Recall, recall that

$$ g'(z^{(l)}) = g(z^{(l)})(1-g(z^{(l)})) = \alpha^{l}(1-\alpha^{(l)}) $$

With

$$ \Delta^{(l)} = \delta^{(l+1)} (\alpha^{(l)})^T $$

we can compute our gradients as follows:

$$ \frac{\partial}{\partial \theta_{ij}^{(l)}} J(\Theta) = \frac{1}{m} \Delta_{ij}^{(l)} $$

### Step 3: Regularized cost function

For neural networks *with regularization* we add a regularization term to the cost function.

$$ r = \frac{\lambda}{m} ( \sum_{\Theta^{(1)}_{j \neq 1, i}} (\Theta^{1}_{j, i})^2 ) +  \sum_{\Theta^{(2)}_{j \neq 1, i \neq 1}} (\Theta^{2}_{j, i})^2 $$

Note that you should not be regularizing the terms that correspond to the bias. For the matrices `Theta1` and `Theta2`, this corresponds to the first column of each matrix.

For the gradients, this means, that we have to add an additional regularization of $ \frac{\lambda}{m} \Theta_{ji}^{(l)} , j > 1 $.

```octave
function [J grad] = nnCostFunction(nn_params, ...
                                   input_layer_size, ...
                                   hidden_layer_size, ...
                                   num_labels, ...
                                   X, y, lambda)
%NNCOSTFUNCTION Implements the neural network cost function for a two layer
%neural network which performs classification
%   [J grad] = NNCOSTFUNCTON(nn_params, hidden_layer_size, num_labels, ...
%   X, y, lambda) computes the cost and gradient of the neural network. The
%   parameters for the neural network are "unrolled" into the vector
%   nn_params and need to be converted back into the weight matrices.
%
%   The returned parameter grad should be a "unrolled" vector of the
%   partial derivatives of the neural network.
%

    % Reshape nn_params back into the parameters Theta1 and Theta2, the weight matrices
    % for our 2 layer neural network
    Theta1 = reshape(nn_params(1:hidden_layer_size * (input_layer_size + 1)), ...
                 hidden_layer_size, (input_layer_size + 1));

    Theta2 = reshape(nn_params((1 + (hidden_layer_size * (input_layer_size + 1))):end), ...
                 num_labels, (hidden_layer_size + 1));

    % dim_Theta1 = size(Theta1)
    % dim_Theta2 = size(Theta2)

    % Setup some useful variables
    m = size(X, 1)

    % We need to return the following variables correctly
    J = 0;
    Theta1_grad = zeros(size(Theta1));
    Theta2_grad = zeros(size(Theta2));

    % Part 1: Feedforward the neural network and return the cost in the
    %         variable J.

    % Add additional bias node to X
    X = [ones(m, 1) X];

    % Note, that whereas the original labels (in the variable y) were 1, 2, ..., 10,
    % for the purpose of training a neural network,
    % we need to recode the labels as vectors containing only values 0 or 1.
    y_Vec = zeros(m, num_labels);
    for i =1:m
        y_Vec(i, y(i)) = 1;
    end

    % Theta1, Theta2 need to be transposed, since h(theta, X) expects theta to be a column vector
    % In Theta1, Theta2 however, the parameters for each node are represented as a row

    % Hidden layer
    alpha2 = h(Theta1', X);

    % Add additional bias node alpha2(0)
    alpha2 = [ones(m, 1) alpha2];

    % Output layer
    alpha3 = h(Theta2', alpha2);

    % Cost function without regularization term

    % We can use matrix multiplication to compute our inner sum
    inner = -log(alpha3)*y_Vec' - log(1-alpha3)*(1-y_Vec') ;
    sum1 = sum(diag(inner));

    J = 1/m * sum1;

    % Part 2: Implement the backpropagation algorithm to compute the gradients
    %         Theta1_grad and Theta2_grad. You should return the partial derivatives of
    %         the cost function with respect to Theta1 and Theta2 in Theta1_grad and
    %         Theta2_grad, respectively.

    % Output layer
    delta3 = alpha3 - y_Vec;

    % Hidden layer
    delta2 = delta3 * Theta2 .* alpha2 .* (1-alpha2);
    delta2 = delta2(:, 2:end);

    Delta1 = delta2' * X;
    Delta2 = delta3' * alpha2;

    Theta1_grad = 1/m * Delta1;
    Theta2_grad = 1/m * Delta2;

    %
    % Part 3: Implement regularization with the cost function and gradients.
    %

    % Regularization term for the cost function
    Theta1_squared = Theta1 .^ 2;
    Theta2_squared = Theta2 .^ 2;

    Theta1_squared = Theta1_squared(:, 2:end);
    Theta2_squared = Theta2_squared(:, 2:end);

    reg_sum = sum(Theta1_squared(:)) + sum(Theta2_squared(:));
    J = J + lambda /(2*m) * reg_sum;

    % Regularization terms for the gradient matrices
    R1 = Theta1;
    R1(:, 1) = 0;
    R2 = Theta2;
    R2(:, 1) = 0;

    Theta1_grad = Theta1_grad + lambda/m * R1;
    Theta2_grad = Theta2_grad + lambda/m * R2;

    % Unroll gradients
    grad = [Theta1_grad(:) ; Theta2_grad(:)];

end
```

```octave
% Addional testcase
il = 2;              % input layer
hl = 2;              % hidden layer
nl = 4;              % number of labels
nn = [ 1:18 ] / 10;  % nn_params
X_test = cos([1 2 ; 3 4 ; 5 6]);
y_test = [4; 2; 3];
[J grad] = nnCostFunction(nn, il, hl, nl, X_test, y_test, 0)
[J grad] = nnCostFunction(nn, il, hl, nl, X_test, y_test, 4)
```

    m =  3
    J =  7.4070
    grad =

       0.766138
       0.979897
      -0.027540
      -0.035844
      -0.024929
      -0.053862
       0.883417
       0.568762
       0.584668
       0.598139
       0.459314
       0.344618
       0.256313
       0.311885
       0.478337
       0.368920
       0.259771
       0.322331

    m =  3
    J =  19.474
    grad =

       0.76614
       0.97990
       0.37246
       0.49749
       0.64174
       0.74614
       0.88342
       0.56876
       0.58467
       0.59814
       1.92598
       1.94462
       1.98965
       2.17855
       2.47834
       2.50225
       2.52644
       2.72233

```octave
% Unroll parameters
nn_params = [Theta1(:) ; Theta2(:)];

% Call cost function
% without regularization:
lambda = 0;
J = nnCostFunction(nn_params, input_layer_size, hidden_layer_size, ...
                   num_labels, X, y, lambda);

fprintf(['Cost at parameters (loaded from ex4weights): %f '...
         '\n(this value should be about 0.287629)\n'], J);

% with regularization (lambda = 1):
lambda = 1;
J = nnCostFunction(nn_params, input_layer_size, hidden_layer_size, ...
                   num_labels, X, y, lambda);

fprintf(['Cost at parameters (loaded from ex4weights) with regularization (lambda=1): %f '...
         '\n(this value should be about 0.383770)\n'], J);
```

    m =  5000
    Cost at parameters (loaded from ex4weights): 0.287629
    (this value should be about 0.287629)
    m =  5000
    Cost at parameters (loaded from ex4weights) with regularization (lambda=1): 0.383770
    (this value should be about 0.383770)

## Gradient Checking

We can apply a method called *gradient checking* to verify numerically, if the computation of the gradients through backpropagation was correct.

The idea ist to "onroll" $ \Theta^{(1)}, \Theta^{(2)} $ into a long vector $ \theta $ and work with a function $ J{\theta) $.

We can now verify if our "onrolled" gradient $ f_i(\theta) $ computed above matches a numerical approximation of the gradient for each $ i $:

$$ f_i(\theta) = \frac {J(\theta^{(i+)}) - J(\theta^{(i-)}) } {2 \epsilon} $$

$ \theta^{(i+)} $ is the same as $ \theta $, except its $i$th element has been incremented by $ \epsilon $. Similarly, $ \theta^{(i-)} $ is the corresponding vector with the $i$th element decreased by $ \epsilon $.

```octave
function W = debugInitializeWeights(fan_out, fan_in)
%DEBUGINITIALIZEWEIGHTS Initialize the weights of a layer with fan_in
%incoming connections and fan_out outgoing connections using a fixed
%strategy, this will help you later in debugging
%   W = DEBUGINITIALIZEWEIGHTS(fan_in, fan_out) initializes the weights
%   of a layer with fan_in incoming connections and fan_out outgoing
%   connections using a fix set of values
%
%   Note that W should be set to a matrix of size(1 + fan_in, fan_out) as
%   the first row of W handles the "bias" terms
%

    % Set W to zeros
    W = zeros(fan_out, 1 + fan_in);

    % Initialize W using "sin", this ensures that W is always of the same
    % values and will be useful for debugging
    W = reshape(sin(1:numel(W)), size(W)) / 10;
end
```

```octave
function numgrad = computeNumericalGradient(J, theta)
%COMPUTENUMERICALGRADIENT Computes the gradient using "finite differences"
%and gives us a numerical estimate of the gradient.
%   numgrad = COMPUTENUMERICALGRADIENT(J, theta) computes the numerical
%   gradient of the function J around theta. Calling y = J(theta) should
%   return the function value at theta.

% Notes: The following code implements numerical gradient checking, and
%        returns the numerical gradient.It sets numgrad(i) to (a numerical
%        approximation of) the partial derivative of J with respect to the
%        i-th input argument, evaluated at theta. (i.e., numgrad(i) should
%        be the (approximately) the partial derivative of J with respect
%        to theta(i).)
%

    numgrad = zeros(size(theta));
    perturb = zeros(size(theta));
    e = 1e-4;
    for p = 1:numel(theta)
        % Set perturbation vector
        perturb(p) = e;
        loss1 = J(theta - perturb);
        loss2 = J(theta + perturb);
        % Compute Numerical Gradient
        numgrad(p) = (loss2 - loss1) / (2*e);
        perturb(p) = 0;
    end

end
```

```octave
function checkNNGradients(lambda)
%CHECKNNGRADIENTS Creates a small neural network to check the
%backpropagation gradients
%   CHECKNNGRADIENTS(lambda) Creates a small neural network to check the
%   backpropagation gradients, it will output the analytical gradients
%   produced by your backprop code and the numerical gradients (computed
%   using computeNumericalGradient). These two gradient computations should
%   result in very similar values.
%

    if ~exist('lambda', 'var') || isempty(lambda)
        lambda = 0;
    end

    input_layer_size = 3;
    hidden_layer_size = 5;
    num_labels = 3;
    m = 5;

    % We generate some 'random' test data
    Theta1 = debugInitializeWeights(hidden_layer_size, input_layer_size);
    Theta2 = debugInitializeWeights(num_labels, hidden_layer_size);
    % Reusing debugInitializeWeights to generate X
    X  = debugInitializeWeights(m, input_layer_size - 1);
    y  = 1 + mod(1:m, num_labels)';

    % Unroll parameters
    nn_params = [Theta1(:) ; Theta2(:)];

    % Short hand for cost function
    costFunc = @(p) nnCostFunction(p, input_layer_size, hidden_layer_size, ...
                               num_labels, X, y, lambda);

    [cost, grad] = costFunc(nn_params);
    numgrad = computeNumericalGradient(costFunc, nn_params);

    % Visually examine the two gradient computations.  The two columns
    % you get should be very similar.
    disp([numgrad grad]);
    fprintf(['The above two columns you get should be very similar.\n' ...
         '(Left-Your Numerical Gradient, Right-Analytical Gradient)\n\n']);

    % Evaluate the norm of the difference between two solutions.
    % If you have a correct implementation, and assuming you used EPSILON = 0.0001
    % in computeNumericalGradient.m, then diff below should be less than 1e-9
    diff = norm(numgrad-grad)/norm(numgrad+grad);

    fprintf(['If your backpropagation implementation is correct, then \n' ...
         'the relative difference will be small (less than 1e-9). \n' ...
         '\nRelative Difference: %g\n'], diff);

end
```

```octave
checkNNGradients(0);
```

    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
    m =  5
      -0.0092782523  -0.0092782524
       0.0088991196   0.0088991196
      -0.0083601076  -0.0083601076
       0.0076281355   0.0076281355
      -0.0067479837  -0.0067479837
      -0.0000030498  -0.0000030498
       0.0000142869   0.0000142869
      -0.0000259383  -0.0000259383
       0.0000369883   0.0000369883
      -0.0000468760  -0.0000468760
      -0.0001750601  -0.0001750601
       0.0002331464   0.0002331464
      -0.0002874687  -0.0002874687
       0.0003353203   0.0003353203
      -0.0003762156  -0.0003762156
      -0.0000962661  -0.0000962661
       0.0001179827   0.0001179827
      -0.0001371497  -0.0001371497
       0.0001532471   0.0001532471
      -0.0001665603  -0.0001665603
       0.3145449700   0.3145449701
       0.1110565882   0.1110565882
       0.0974006970   0.0974006970
       0.1640908188   0.1640908188
       0.0575736494   0.0575736493
       0.0504575855   0.0504575855
       0.1645679323   0.1645679323
       0.0577867379   0.0577867378
       0.0507530173   0.0507530173
       0.1583393339   0.1583393339
       0.0559235296   0.0559235296
       0.0491620841   0.0491620841
       0.1511275275   0.1511275275
       0.0536967009   0.0536967009
       0.0471456249   0.0471456249
       0.1495683347   0.1495683347
       0.0531542052   0.0531542052
       0.0465597186   0.0465597186
    The above two columns you get should be very similar.
    (Left-Your Numerical Gradient, Right-Analytical Gradient)

    If your backpropagation implementation is correct, then
    the relative difference will be small (less than 1e-9).

    Relative Difference: 2.25001e-11

## Training

When training neural networks, it is important to randomly initialize the parameters for symmetry breaking. One effective strategy for random initialization is to randomly select values for $ \Theta^{(l)} $ uniformly in the range $ [-\epsilon, +\epsilon] $.

The training done again by using the well know optimization function:

```octave
function W = randInitializeWeights(L_in, L_out)
%RANDINITIALIZEWEIGHTS Randomly initialize the weights of a layer with L_in
%incoming connections and L_out outgoing connections
%   W = RANDINITIALIZEWEIGHTS(L_in, L_out) randomly initializes the weights
%   of a layer with L_in incoming connections and L_out outgoing
%   connections.
%
%   Note that W should be set to a matrix of size(L_out, 1 + L_in) as
%   the first column of W handles the "bias" terms
%

    W = zeros(L_out, 1 + L_in);

    % Randomly initialize the weights to small values
    eps = 0.12;
    W = rand(L_out, 1 + L_in) * 2 * eps - eps;

end
```

```octave
%  After you have completed the assignment, change the MaxIter to a larger
%  value to see how more training helps.
options = optimset('MaxIter', 50);

%  You should also try different values of lambda
lambda = 1;

% Initial parameters
initial_Theta1 = randInitializeWeights(input_layer_size, hidden_layer_size);
initial_Theta2 = randInitializeWeights(hidden_layer_size, num_labels);

% Unroll parameters
initial_nn_params = [initial_Theta1(:) ; initial_Theta2(:)];

% Create "short hand" for the cost function to be minimized
costFunction = @(p) nnCostFunction(p, ...
                                   input_layer_size, ...
                                   hidden_layer_size, ...
                                   num_labels, X, y, lambda);

% Now, costFunction is a function that takes in only one argument (the
% neural network parameters)
[nn_params, cost] = fmincg(costFunction, initial_nn_params, options);

% Obtain Theta1 and Theta2 back from nn_params
Theta1 = reshape(nn_params(1:hidden_layer_size * (input_layer_size + 1)), ...
                 hidden_layer_size, (input_layer_size + 1));

Theta2 = reshape(nn_params((1 + (hidden_layer_size * (input_layer_size + 1))):end), ...
                 num_labels, (hidden_layer_size + 1));
```

    m =  5000
    m =  5000
    m =  5000
    m =  5000
    m =  5000     1 | Cost: 3.297939e+00
    m =  5000
    m =  5000
    m =  5000     2 | Cost: 3.254918e+00
    m =  5000
    m =  5000     3 | Cost: 3.221777e+00
    m =  5000
    m =  5000     4 | Cost: 3.118353e+00
    m =  5000
    m =  5000     5 | Cost: 2.871056e+00
    m =  5000
    m =  5000     6 | Cost: 2.384246e+00
    m =  5000     7 | Cost: 2.039487e+00
    m =  5000
    m =  5000     8 | Cost: 1.945045e+00
    m =  5000     9 | Cost: 1.826975e+00
    m =  5000    10 | Cost: 1.768237e+00
    m =  5000    11 | Cost: 1.650729e+00
    m =  5000    12 | Cost: 1.546114e+00
    m =  5000
    m =  5000    13 | Cost: 1.514566e+00
    m =  5000
    m =  5000    14 | Cost: 1.394788e+00
    m =  5000
    m =  5000    15 | Cost: 1.163227e+00
    m =  5000    16 | Cost: 1.087384e+00
    m =  5000
    m =  5000    17 | Cost: 1.050586e+00
    m =  5000    18 | Cost: 9.678755e-01
    m =  5000    19 | Cost: 8.928639e-01
    m =  5000    20 | Cost: 8.257586e-01
    m =  5000    21 | Cost: 7.821587e-01
    m =  5000
    m =  5000    22 | Cost: 7.643840e-01
    m =  5000    23 | Cost: 7.426642e-01
    m =  5000    24 | Cost: 7.263109e-01
    m =  5000    25 | Cost: 7.109944e-01
    m =  5000    26 | Cost: 7.014560e-01
    m =  5000    27 | Cost: 6.904134e-01
    m =  5000    28 | Cost: 6.646547e-01
    m =  5000    29 | Cost: 6.455411e-01
    m =  5000    30 | Cost: 6.310349e-01
    m =  5000
    m =  5000    31 | Cost: 6.221766e-01
    m =  5000    32 | Cost: 6.145356e-01
    m =  5000
    m =  5000    33 | Cost: 6.092355e-01
    m =  5000    34 | Cost: 6.040427e-01
    m =  5000    35 | Cost: 5.998618e-01
    m =  5000
    m =  5000    36 | Cost: 5.878160e-01
    m =  5000    37 | Cost: 5.715380e-01
    m =  5000    38 | Cost: 5.613109e-01
    m =  5000
    m =  5000    39 | Cost: 5.536140e-01
    m =  5000    40 | Cost: 5.442903e-01
    m =  5000
    m =  5000    41 | Cost: 5.411204e-01
    m =  5000
    m =  5000    42 | Cost: 5.393187e-01
    m =  5000
    m =  5000    43 | Cost: 5.334700e-01
    m =  5000
    m =  5000    44 | Cost: 5.181383e-01
    m =  5000    45 | Cost: 4.979996e-01
    m =  5000    46 | Cost: 4.874777e-01
    m =  5000
    m =  5000    47 | Cost: 4.842996e-01
    m =  5000
    m =  5000    48 | Cost: 4.734155e-01
    m =  5000
    m =  5000    49 | Cost: 4.694667e-01
    Iteration    50 | Cost: 4.659794e-01

## Results

Let's implement a function that uses the neural network defined by the parameters $ (\Theta^{(1)},\Theta^{(2)}) $ to predict the digits for a given data set of our training data:

```octave
function p = predict(Theta1, Theta2, X)
%PREDICT Predict the label of an input given a trained neural network
%   p = PREDICT(Theta1, Theta2, X) outputs the predicted label of X given the
%   trained weights of a neural network (Theta1, Theta2)

    % Useful values
    m = size(X, 1);
    num_labels = size(Theta2, 1);

    p = zeros(size(X, 1), 1);

    % Add ones to the X data matrix
    X = [ones(m, 1) X];

    % Theta1, Theta2 need to be transposed, since h(theta, X) expects theta to be a column vector
    % In Theta1, Theta2 however, the parameters for each node are represented as a row

    % Hidden layer
    alpha2 = h(Theta1', X);

    % Add additional bias node alpha2(0)
    alpha2 = [ones(m, 1) alpha2];

    % Output layer
    alpha3 = h(Theta2', alpha2);

    % Pick the best output und use it as label
    [M, p] = max(alpha3, [], 2);

end
```

```octave
pred = predict(Theta1, Theta2, X);

fprintf('\nTraining Set Accuracy: %f\n', mean(double(pred == y)) * 100);
```

    Training Set Accuracy: 95.620000

Let's see what our trained logistic regression function is doing for our randomly selected examples from the beginning of this exercise:

```octave
displayData(sel);
```

```octave
% Predict the labels for the selection and convert label 10 to 0
guessed_numbers = mod(predict(Theta1, Theta2, sel), 10);
```

```octave
% ... and print them as a matrix
reshape(guessed_numbers, [10,10])'
```

    ans =

       0   3   0   8   7   3   6   8   7   2
       2   4   2   9   7   8   6   8   4   2
       0   5   0   4   9   8   9   9   7   0
       2   6   2   5   0   8   7   1   7   3
       9   0   3   4   7   4   8   5   5   4
       9   3   3   1   7   5   7   2   0   7
       0   1   2   4   8   7   4   1   1   4
       1   7   6   8   5   0   6   5   0   9
       0   9   7   6   7   1   7   2   9   3
       0   6   7   2   1   1   1   3   8   2

We can now "visualize" what the neural network is learning by
displaying the hidden units to see what features they are capturing in
the data.

```octave
displayData(Theta1(:, 2:end));
```

```octave

```


# Week 3 worksheet 1: Root finding methods

This notebook is modified from one created by Charlotte Desvages.

This notebook investigates the first root finding algorithms to solve nonlinear equations: the **bisection method** and **fixed-point iteration**.

The best way to learn programming is to write code. Don't hesitate to edit the code in the example cells, or add your own code, to test your understanding. You will find practice exercises throughout the notebook, denoted by 🚩 ***Exercise $x$:***.

#### Displaying solutions

Solutions will be released one week after the worksheets are released, as a new `.txt` file in the same GitHub repository. After pulling the file to your workspace, run the following cell to create clickable buttons under each exercise, which will allow you to reveal the solutions.

```python
%run scripts/create_widgets.py W03-W1
```

---
### 📚 Book sections

- **ASC**: sections 5.1, **5.2**
- **PCP**: sections 7.1.3, 7.4

## Bracketing methods: bisection and regula falsi

**Bracketing methods** seek to find smaller and smaller **intervals** $[a, b]$ which contain a root.

They rely on the Intermediate Value Theorem: for a continuous function $F(x)$, if $F(a)$ has different sign than $F(b)$, then $F(x)$ has a root $x_\ast \in [a, b]$.

### Bisection

Key idea: start with an interval $[a, b]$ such that $F(a)$ and $F(b)$ have different signs.

Then, split the interval in two, and evaluate $F(\frac{a+b}{2})$ at the midpoint. Compare the sign of $F(\frac{a+b}{2})$ with the sign of $F(a)$ and $F(b)$, and choose the half-interval which still contains the root. Repeat the procedure with the new, smaller interval, until convergence.

### Regula falsi

Similar to bisection, but instead of choosing the midpoint, we choose the point of intersection between the x-axis and a straight line passing through the points $(a, F(a))$ and $(b, F(b))$.

---
## Solving nonlinear equations with root-finding algorithms

In this section, we consider equations of the form

$$
F(x) = 0,
$$

where $F(x)$ is a **nonlinear** function of $x$. Solving this equation is equivalent to finding the *root(s)* $x_\ast$ of the function $F$.

There are direct methods we can use to solve *linear* equations, even linear systems of equations as we saw previously; however, a nonlinear equation of the form given above doesn't always have a solution which can be found analytically.

The methods we will discuss to solve nonlinear equations are all **iterative**:
1. we start with a guess for where the root may be;
2. if we are close enough to the solution, for instance if the function is sufficiently close to zero for the current guess, we stop;
3. if not, we refine our guess using information we have about the function;
4. we go back to step 1 with our new guess.

Step 3 is what differentiates the methods we will use, as they each use a different process to refine the current best guess. For all these methods, the key idea is to reduce the nonlinear problem to smaller, simpler problems, which we solve repeatedly (iteratively).

### The bisection method

Given a continuous function $F \left( x \right)$, if $F \left( a \right) \le 0$ and $F \left( b \right) \ge 0$, the Intermediate Value Theorem tells us that there must be a root in the closed interval $\left[ a, b \right].$ The bisection method proceeds by testing the **sign** of $F \left( c \right)$ where $c$ is the **mid-point** of the interval $\left[ a, b \right]$, and uses this to halve the size of the interval in which a root is sought. The process is repeated with the new half-interval, until the interval is small enough to approximate the root with a given tolerance.

The next few exercises will guide you through implementing the bisection method yourself.

---
**📚 Learn more:**
- **ASC**: section 5.2
- **PCP**: section 7.4

---
🚩 ***Exercise 1:***
Consider the function

\begin{equation}\label{eqn:F}
  F \left( x \right) = \sin \left( 2 \pi x \right) e^{4 x} + x.
\end{equation}

Plot this function in the interval $x \in \left[ 0, 1 \right]$ and identify the three roots in this interval. Check that $F \left( 0.2 \right)$ and $F \left( 0.6 \right)$ have opposite signs.

You may find it convenient to create a function `F`.

```python

```

```python
%run scripts/show_solutions.py W03-W1_ex1
```

---
🚩 ***Exercise 2:*** Define variables corresponding to $a = 0.2$ and $b = 0.6$. Define $c$ to be the midpoint $c = (a + b) / 2$.

Then, start a loop, which should iterate until the root is found. At each iteration:
- Depending on the sign of $F \left( a \right) F \left( c \right)$, decide whether to set $a$ or $b$ to be equal to $c$, so that there is a root of $F \left( x \right)$ in the *new* interval $\left[ a, b \right]$ (with half the width of the previous interval).
- Define $c$ to be the new midpoint $c = (a + b) / 2$.
- The loop should stop when you have found the root $x_\ast$ within an error of $10^{-10}$.

A possible convergence criterion is $|F(c)| < \varepsilon$, where $\varepsilon$ is the tolerance -- here, $10^{-10}$.

How many iterations are needed to find the root to within this error?

```python

```

```python
%run scripts/show_solutions.py W03-W1_ex2
```

---
🚩 ***Exercise 3:*** Choose different $a$ and $b$ values in order to find the root near $x = 1$, to within an error of $10^{-10}$.

You may wish to write your code from Exercise 2 into a function `bisection(F, a, b, tol)`, which finds the root of a function `F` in the interval `[a, b]` to within an error of `tol`, and returns the value of the roots and the number of iterations.

```python

```

```python
%run scripts/show_solutions.py W03-W1_ex3
```

---
### Regula falsi

The "regula falsi" method is similar to the bisection method, with an important difference: instead of selecting the midpoint of the interval $[a, b]$ at each iteration, we trace a straight line between the points $(a, F(a))$ and $(b, F(b))$, and select the point $c$ where this line intersects the x-axis. In other words, we interpolate $F$ linearly between $a$ and $b$, and we find the root of this interpolating polynomial (of degree 1) at each iteration.

---
🚩 ***Exercise 4:*** Show that $c$ is given by

$$
c = \frac{a F(b) - b F(a)}{F(b) - F(a)}.
$$

*Hint: a line with slope $\alpha$ which passes through the point $(x_0, y_0)$ has equation*

$$
y - y_0 = \alpha (x - x_0).
$$

```python
%run scripts/show_solutions.py W03-W1_ex4
```

---
🚩 ***Exercise 5:*** Consider the same function $F$ as in section 2.1.

Define variables corresponding to $a = 0.2$ and $b = 0.6$.

Then, proceed as you did for the bisection method, but instead of defining $c$ to be the midpoint of $[a, b]$, define $c$ as above.

How many iterations are needed to find the root to within a tolerance of $10^{-10}$?

You may wish to define a function `regula_falsi(F, a, b, tol)` to find a root of a function `F` within an interval `[a, b]` to within an error `tol`, which returns the computed root and the number of iterations.

```python

```

```python
%run scripts/show_solutions.py W03-W1_ex5
```

---
## Convergence of root-finding methods

The bisection and regula falsi methods are guaranteed to converge to a root, provided $F$ is sufficiently smooth and the starting interval $[a, b]$ is chosen appropriately.

But different methods may converge to a root at different *speeds*. The **order of convergence** for root-finding algorithms is defined in terms of successive values of the error $e_k := x_k - x_\ast$ between the true solution $x_\ast$ and the guess $x_k$ obtained at the $k$th iteration.

---
#### 🚩 Definition: Order of convergence of root-finding methods

A convergent root-finding algorithm converges **at $p$th order** if

$$
\lim_{k \to \infty} \frac{|e_{k+1}|}{|e_k|^p} = \alpha,
$$

where $\alpha \in \mathbb{R}$ is a constant.

---

For a $p$th order convergent method, we expect the error at the $k+1$th iteration to be roughly proportional to the $p$th power of the error at the $k$th iteration, for sufficiently large $k$ -- that is, when we are in a close enough neighbourhood of $x_\ast$.

Note that $p$ is not always an integer.

---
🚩 ***Exercise 6:*** Modify your code from Exercise 5 so that all the successive guesses $x_k$ are stored in a Numpy array. Perform the task from Exercise 6 again -- use the regula falsi method to find the same root of `F`, using the same starting interval and tolerance. You should obtain the same result, but now you should also have a vector `x` with length $k_\max + 1$ containing all the guesses.

Consider $x_{k_\max}$, the last guess obtained by the method, to be the "true solution". Compute the magnitude of the error $e_k$ between each of the previous guesses $x_k$ and the true solution.

For $p=1, 1.5, 2, 2.5$, compute the ratio $\frac{|e_{k+1}|}{|e_k|^p}$ for $k=0, 1, \dots, k_\max - 1$, and plot it against $k$. Set your y-axis limits to $[0, 1000]$ to start with, and reduce the range as necessary.

Given the definition above, what do you expect is the order of convergence of regula falsi? How do you explain the appearance of the graph?

```python

```

```python
%run scripts/show_solutions.py W03-W1_ex6
```


This notebook is part of https://github.com/AudioSceneDescriptionFormat/splines, see also https://splines.readthedocs.io/.

# Polynomial Parametric Curves

The building blocks for
[polynomial splines](index.ipynb)
are of course [polynomials](https://en.wikipedia.org/wiki/Polynomial).

```python
import sympy as sp
sp.init_printing(order='grevlex')
```

We are mostly interested in *univariate* splines,
i.e. curves with one free parameter,
which are built using polynomials with a single parameter.

Here we are calling this parameter $t$.
You can think about it as time (e.g. in seconds),
but it doesn't have to represent time.

```python
t = sp.symbols('t')
```

Polynomials typically consist of multiple *terms*.
Each term contains a *basis function*,
which itself contains one or more integer powers of $t$.
The highest power of all terms is called the *degree* of the polynomial.

The arguably simplest set of basis functions
is the *monomial basis*,
which simply consists of all powers of $t$ up to the given degree:

```python
b_monomial = sp.Matrix([t**3, t**2, t, 1]).T
b_monomial
```

In this example we are using polynomials of degree 3,
which are also called *cubic* polynomials.

These basis functions are multiplied by (constant) *coefficients*.
We are writing the coefficients with bold symbols,
because apart from simple scalars (for one-dimensional functions),
these symbols can also represent vectors in two- or three-dimensional space.

```python
coefficients = sp.Matrix(sp.symbols('a:dbm')[::-1])
coefficients
```

We can create a polynomial by
multiplying the basis functions with the coefficients
and then adding all terms:

```python
b_monomial.dot(coefficients)
```

This is a cubic polynomial in its *canonical form*
(because it uses monomial basis functions).

Let's take a closer look at those basis functions
(with some help from [helper.py](helper.py)):

```python
from helper import plot_basis
```

```python
plot_basis(*b_monomial)
```

It doesn't look like much,
but every conceivable cubic polynomial
can be formulated as exactly one linear combination
of those basis functions
(i.e. using one specific list of coefficients).

An example polynomial that's not in canonical form ...

```python
example_polynomial = (2 * t - 1)**3 + (t + 1)**2 - 6 * t + 1
example_polynomial
```

```python
from helper import plot_sympy, grid_lines
```

```python
plot_sympy(example_polynomial, (t, 0, 1))
grid_lines([0, 1], [0, 0.5, 1])
```

... can simply be re-written with monomial basis functions:

```python
example_polynomial.expand()
```

Any polynomial can be rewritten using any set of basis functions
(of appropriate degree).

In later sections we will see more basis functions,
for example those that are used for
[Hermite](hermite-uniform.ipynb),
[Bézier](bezier-de-casteljau.ipynb) and
[Catmull--Rom](catmull-rom-uniform.ipynb) splines.

In the previous example,
we have used scalar coefficients
to create a one-dimensional polynomial.
We can use two-dimensional coefficients
to create two-dimensional polynomial curves.
Let's create a little class to try this:

```python
import numpy as np

class CubicPolynomial:

    grid = 0, 1

    def __init__(self, d, c, b, a):
        self.coeffs = d, c, b, a

    def evaluate(self, t):
        t = np.expand_dims(t, -1)
        return t**[3, 2, 1, 0] @ self.coeffs
```

<div class="alert alert-info">

Note

The `@` operator is used here to do
[NumPy's matrix multiplication](https://numpy.org/doc/stable/reference/generated/numpy.matmul.html).

</div>

Since this class has the same interface as the splines
that will be discussed in later sections,
we can use a spline helper function from [helper.py](helper.py)
for plotting:

```python
from helper import plot_spline_2d
```

```python
poly_2d = CubicPolynomial([-1.5, 5], [1.5, -8.5], [1, 4], [3, 2])
```

```python
plot_spline_2d(poly_2d, dots_per_second=30, chords=False)
```

This class can also be used with three and more dimensions.
The class [splines.Monomial](../python-module/splines.rst#splines.Monomial)
can be used to try this with arbitrary polynomial degree.


Let's use SymPy to derive the relation between potential V and charge density R

```
%pylab inline
from sympy.interactive import init_printing
init_printing()
from sympy import pi, var, S, Piecewise, piecewise_fold
var("r R")
Vh = Piecewise((-S(2)/3 * pi * (3*R**2 - r**2), r <= R), (-S(4)/3 * pi * R**3 / r, True))
def laplace(f):
    return (r*f).diff(r, 2)/r
print "Vh ="
Vh
```

    Welcome to pylab, a matplotlib-based Python environment [backend: module://IPython.zmq.pylab.backend_inline].
    For more information, type 'help(pylab)'.
    Vh =

$$\begin{cases} - \frac{2}{3} \pi \left(3 R^{2} - r^{2}\right) & \text{for}\: r \leq R \\- \frac{4}{3} \frac{\pi R^{3}}{r} & \text{otherwise} \end{cases}$$

Charge density is then:

```
piecewise_fold(-laplace(Vh)/(4*pi))
```

$$\begin{cases} -1 & \text{for}\: r \leq R \\0 & \text{otherwise} \end{cases}$$

```
from numpy import (empty, pi, meshgrid, linspace, sum, sin, exp, shape, sqrt,
        conjugate)
from numpy.fft import fftn, fftfreq, ifftn

N = 100
print "N =", N
L = 2.4*4
R = 1.
x1d = linspace(-L/2, L/2, N+1)[:-1]
x, y, z = meshgrid(x1d, x1d, x1d, indexing="ij")

r = sqrt(x**2+y**2+z**2)
nr = empty(shape(x), dtype="double")
nr[:] = 0
nr[r <= R] = -1

Vanalytic = empty(shape(x), dtype="double")
Vanalytic[r <= R] = -2./3 * pi * (3*R**2 - r[r <= R]**2)
Vanalytic[r > R] = -4./3 * pi * R**3 / r[r > R]

ng = fftn(nr) / N**3

G1d = N * fftfreq(N) * 2*pi/L
kx, ky, kz = meshgrid(G1d, G1d, G1d)
G2 = kx**2+ky**2+kz**2
G2[0, 0, 0] = 1  # omit the G=0 term

tmp = 2*pi*abs(ng)**2 / G2
tmp[0, 0, 0] = 0  # omit the G=0 term
E = sum(tmp) * L**3
print "Hartree Energy (calculated): %.15f" % E

Vg = 4*pi*ng / G2
Vg[0, 0, 0] = 0  # omit the G=0 term
V = ifftn(Vg).real * N**3
V += Vanalytic[N/2, N/2, N/2] - V[N/2, N/2, N/2]
l2_norm = sum((Vanalytic - V)**2)
print "l2_norm = ", l2_norm
plot(x[:, N/2, N/2], Vanalytic[:, N/2, N/2], label="analytic")
plot(x[:, N/2, N/2], V[:, N/2, N/2], label="FFT")
legend(loc="best");
```

```

```


#Real Analysis

### 1. Sequences

For each of the patterns below, define a formula for $n$-th term $x_n$ of the sequence. Then, plot the first 20 terms of the sequences. Combine all three plots into a figure using the subplot command. Label everything.

a) 4, 8, 12, 16,...

b) 1, 6, 11, 16, 21, ...

c) 1/2, 2/3, 3/4, 4/5, ...

d) 1/2, -1/4, 1/8, -1/16, ...

e) 1, 1, 2, 3, 5, 8, ..

####Let a= first term, n=number of terms, d= difference between 2 consecutive terms, Then,
####a) nth term, Xn = a + (n-1)d = 4+(n-1)4 = 4+4n-4 = 4n
####b) nth term, Xn = a + (n-1)d = 1+(n-1)5 = 1+5n-5 = 5n-4
####c) nth term, Xn = $ \frac{n}{n+1}$
####d) nth term, Xn = $ \frac{(-1)^{n+1}}{2^n}$
####e) nth term, Xn = Xn-1 + Xn-2 (Fibonacci series)

```python
import matplotlib.pyplot as plt
%matplotlib inline
A1 = []    #Initializing an empty array for storing number of iterations value, n
B1, B2, B3, B4, B5 = [], [], [], [], [] #Initializing empty arrays for storing sequence output
a1 = 4     #Initializing first term for P1.a
#Loop for getting the sequence for P1.a
for n in range(1, 21):
  a1 = 4*n
  A1.append(n)
  B1.append(a1)
print(B1,end=',\n')

a2 = 1     #Initializing first term for P1.b
#Loop for getting the sequence for P1.b
for n in range(1, 21):
  a2 = (5*n)-4
  B2.append(a2)
print(B2,end=',\n')

a3 = 1/2   #Initializing first term for P1.c
#Loop for getting the sequence for P1.c
for n in range(1, 21):
  a3 = n/(n+1)
  B3.append(a3)
print(B3, end=',\n')

a4 = 1/2  #Initializing first term for P1.d
#Loop for getting the sequence for P1.d
for n in range(1, 21):
  a4 = ((-1)**(n+1))/(2*n)
  B4.append(a4)
print(B4, end=',\n')

#Initializing sequence for P1.e
a = 0
b = 1
c = 0
B5.append(b)
#Loop for getting the sequence for P1.e
for n in range(19):
  c = a+b
  a = b
  b = c
  B5.append(c)
print(B5, end=',\n')

#Plotting all the sequences
fig,ax = plt.subplots(1,5, figsize = (30,5))
fig.suptitle('Plotting all the sequences', color='b')
ax[0].plot(A1,B1, 'r', label='4n'); ax[0].set_title('P1.a) 4, 8, 12, 16,...')
ax[1].plot(A1,B2, 'c', label='5n-4'); ax[1].set_title('P1.b) 1, 6, 11, 16, 21, ...')
ax[2].plot(A1,B3, 'g', label='n/(n+1)'); ax[2].set_title('P1.c) 1/2, 2/3, 3/4, 4/5, ...')
ax[3].plot(A1,B4, 'y', label='(-1)^(n+1)/2^n'); ax[3].set_title('P1.d) 1/2, -1/4, 1/8, -1/16, ...')
ax[4].plot(A1,B5, 'm', label='Fibonacci Series'); ax[4].set_title('P1.e) 1, 1, 2, 3, 5, 8, ..')
for i in range(5):
  ax[i].legend()
  ax[i].set_xlabel('Input-n')
  ax[i].set_ylabel('X_n')
```

### 2. Sequence Convergence

Show that the following sequences have the given limits using either an $\varepsilon-\delta$ argument, or an appropriate theorem from the lecture notes.

a) $\lim \frac{3n+1}{2n+5} = \frac{3}{2}$

b) $\lim \frac{2n}{n+2} = 2$

c) $\lim\left(\frac{1}{n} - \frac{1}{n+1} \right) = 0$

d) $\lim \frac{n + 5}{n^{2}} = 0$

e) $\lim \left(\frac{1}{3n}\right)^{2} = 0$

P2.a. In this example, we have,
$f(x) = \frac {3n+1}{2n+5}$ and $Limit, L = \frac {3}{2}$. For any $\varepsilon > 0,  \left|f(x) - \frac{3}{2}\right| < \varepsilon$ proves the convergence.
\begin{aligned} \left| \frac {3n+1}{2n+5} - \frac {3}{2} \right| &= \left| \frac{(6n+2)-(6n+15)}{2(2n+5)}\right| < \varepsilon \\ &=) \left| \frac{-13}{4n+10}\right|<\varepsilon \\ &=) \left|\frac{-13}{4n}\right| < \varepsilon\\&=) \frac{13}{4n} < \varepsilon \\ &=) \frac{13}{4\varepsilon}<n\\&=)for\ n>N, we\ get, \ N=\left \lceil \frac{13}{4\varepsilon } \right \rceil \ satisfies\ the\ condition.
\end{aligned}
When we plot the difference of function and limit, $f(x)-L$ with $N\geq65$ and $\varepsilon=0.05$ it shows the convergence of $f(x)$.

```python
#P2.a
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

n = np.linspace(65 , 100, 200)
#Plotting the sequence for N>=65 and ep=0.05
plt.plot(n, 3/2-(3*n+1)/(2*n+5), 'r', label='x_n=Convergence')
plt.title('P2.a')
plt.xlabel('Input-n')
plt.ylabel('x_n')
plt.legend()
```

P2.b. In this example, we have,
$function, f(x) = \frac {2n}{n+2}$ and $Limit, L = 2$. For any $\varepsilon > 0 ,  \left|f(x) - 2\right| < \varepsilon$ proves the convergence.
\begin{aligned} \left| \frac {2n}{n+2} - 2 \right| &= \left| \frac{2n-2(n+2)}{n+2}\right| = \left| \frac{4}{n+2}\right|< \varepsilon \\&=)\frac {4}{n+2} < \varepsilon \\ &=) \frac {n+2}{4} > \frac{1}{\varepsilon}  \\ &=) n > \frac{4}{\varepsilon}-2\\& =)for\ n>N, we\ get, N=\left \lceil \frac{4}{\varepsilon}-2\right \rceil\ satisfies\ the\ condition.
\end{aligned}
When we plot the difference of function and limit, $f(x)-L$ with $N\geq78$ and $\varepsilon=0.05$ it shows the convergence of $f(x)$.

```python
#P2.b
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

n = np.linspace(78 , 100, 200)
#Plotting the sequence for N>=78 and ep=0.05
plt.plot(n, 2-(2*n)/(n+2), 'b', label='x_n=Convergence')
plt.title('P2.b')
plt.xlabel('Input-n')
plt.ylabel('x_n')
plt.legend()
```

P2.c. In this example, we have,
$function, f(x) = \frac {1}{n}-\frac {1}{n+1}$ and $Limit, L = 0$. For any $\varepsilon > 0 ,  \left|f(x) - 0\right| < \varepsilon$ proves the convergence.
\begin{aligned} \left| \frac {1}{n}-\frac {1}{n+1} \right| &< \left| \frac{1}{n}\right|< \varepsilon \\&=)\frac {1}{n} < \varepsilon \\ &=)\frac {1}{\varepsilon} < n, \ for\ n>N, we\ get, N=\left \lceil \frac{1}{\varepsilon}\right \rceil\ satisfies\ the\ condition.
\end{aligned}
When we plot the difference of function and limit, $f(x)-L$ with $N\geq20$ and $\varepsilon=0.05$ it shows the convergence of $f(x)$.

```python
#P2.c
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

n = np.linspace(20 , 100, 200)
#Plotting the sequence for N>=20 and ep=0.05
plt.plot(n, 1/n-1/(n+1), 'g', label='x_n=Convergence')
plt.title('P2.c')
plt.xlabel('Input-n')
plt.ylabel('x_n')
plt.legend()
```

P2.d. In this example, we have,
$function, f(x) = \frac {n+5}{n^2}$ and $Limit, L = 0$. For any $\varepsilon > 0 ,  \left|f(x) - 0\right| < \varepsilon$ proves the convergence.
\begin{aligned} &\left| \frac {n+5}{n^2}\right| < \varepsilon \\&\left| \frac {n+5}{n^2}\right| >\left| \frac{n}{n^2}\right|=)\left| \frac{n}{n^2}\right|< \varepsilon \\&=) \frac{1}{n}< \varepsilon\\&=) \frac{1}{\varepsilon} < n, \ as\ n>N, we\ get, N=\left \lceil \frac {1}{\varepsilon}\right \rceil\ satisfies\ the\ condition.
\end{aligned}
When we plot the difference of function and limit, $f(x)-L$ with $N\geq20$ and $\varepsilon=0.05$ it shows the convergence of $f(x)$.

```python
#P2.d
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

n = np.linspace(20 , 100, 200)
#Plotting the sequence for N>=20 and ep=0.05
plt.plot(n, (n+5)/(n**2), 'm', label='x_n=Convergence')
plt.title('P2.d')
plt.xlabel('Input-n')
plt.ylabel('x_n')
plt.legend()
```

P2.e. In this example, we have,
$function, f(x) = (\frac {1}{3n})^2$ and $Limit, L = 0$. For any $\varepsilon > 0 ,  \left|f(x) - 0\right| < \varepsilon$ proves the convergence.
\begin{aligned} \left| (\frac {1}{3n})^2-0\right| &< \varepsilon \\&\left| \frac {1}{9n^2}\right| < \varepsilon \\&=) \frac{1}{9n^2}< \varepsilon\\&=) \frac{1}{9\varepsilon}< n^2 \\ &=)\frac {1}{3\sqrt\varepsilon} < n, \ for\ n>N, we\ get, N=\left \lceil \frac {1}{3\sqrt\varepsilon}\right \rceil\ satisfies\ the\ condition.
\end{aligned}
When we plot the difference of function and limit, $f(x)-L$ with $N\geq1.5$ and $\varepsilon=0.05$ it shows the convergence of $f(x)$.

```python
#P2.e
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

n = np.linspace(1.5 , 100, 200)
#Plotting the sequence for N>=1.5 and ep=0.05
plt.plot(n, ((1/(3*n))**2), 'y', label='x_n=Convergence')
plt.title('P2.e')
plt.xlabel('Input-n')
plt.ylabel('x_n')
plt.legend()
```

### 3. Proving non-convergence

Show that the sequence $(x_n)$ where $x_n = (-1)^n$ does not converge to any number $c$. Use the definition of a limit for sequences. Given $\varepsilon > 0$, you have to show that there does not exist an $m$ such that for all $n>m$ such that $c-\varepsilon < x_n < c+\varepsilon$.

In this example, lets assume,
$f(x) = x_n = (-1)^n$ converges to any number $c$. For any $$\varepsilon > 0 ,  \left|f(x) - c\right| < \varepsilon$$ proves the convergence.
$$\left| (-1)^n-c\right| < \varepsilon$$Let, $\varepsilon=\frac{1}{2}$.By definition of convergence, we can have N such that for any n>N, $$\left| (-1)^n-c\right| < \frac{1}{2}$$It means that the maximum distance between any two points $(−1)^n$ and $(−1)^m$, for m,n>N, can be $2*\frac{1}{2}=1$, as they are both $\frac{1}{2}$ away from c.
But this cannot be true as the distance between any $(−1)^n$ and $(−1)^{n+1}$ is 2 for n>N. Therefore the sequence does not converge for $\varepsilon=\frac{1}{2}$ and hence given sequence is not convergent (seen in the plot as well).

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

n = np.linspace(1 , 20, 20)
#Plotting the sequence
plt.plot(n, (-1)**n, 'b', label='x_n=Non-Convergence')
plt.title('P3')
plt.xlabel('Input-n')
plt.ylabel('x_n')
plt.legend()
```

### 4. Sequences via recurrence relations

Let $x_0 = 1$ and $x_{i+1} = - a x_i$ where $0 < a < 1$.

a) Plot the first 20 points in $(x_n)$ when $a=\frac{1}{2}$.

b) Show that $(x_n)$ converges to 0.

```python
import matplotlib.pyplot as plt
%matplotlib inline

x = 1   #Initializing the sequence
a = 1/2 #Initializing the constant
A , B= [],[] #Intializing empty arrays for storing number of iterations and sequence outputs
#Loop for getting the sequence
for n in range(20):
  x = -a*x
  A.append(n)
  B.append(x)
  print(n+1,")", x, end=',')
#Plotting the sequence
plt.plot(A, B, 'r', label='-ax, a=1/2')
plt.title('P4. Sequences via recurrence relations')
plt.xlabel('Input-n')
plt.ylabel('x_n')
plt.legend()
```

For the given conditions: $x_0 = 1$ and $x_{i+1} = - a x_i$ where $0 < a < 1$, when we select $n=20, a=\frac{1}{2}$ and plot the sequence, it is clearly observed that the sequence $(x_n)$ converges to 0.

 ### 5. Sequences via recurrence relations - 2

Let $x_0 = 1$ and $x_{i+1} = - a x_i + \frac{1}{5}$ where $0 < a < 1$.

a) Plot the first 20 points in $(x_n)$ when $a=\frac{1}{3}$.

b) Show that $(x_n)$ converges to 0.

```python
import matplotlib.pyplot as plt
%matplotlib inline

x = 1   #Initializing the sequence
a = 1/3 #Initializing the constant
A , B= [],[] #Intializing empty arrays for storing number of iterations and sequence outputs
#Loop for getting the sequence
for n in range(20):
  x = -a*x+(1/5)
  A.append(n)
  B.append(x)
  print(n+1,")", x, end=',')
#Plotting the sequence
plt.plot(A, B, 'm', label='-ax+(1/5),a=1/3')
plt.title('P5. Sequences via recurrence relations-2')
plt.xlabel('Input-n')
plt.ylabel('x_n')
plt.legend()
```

For the given conditions: $x_0 = 1$ and $x_{i+1} = - a x_i+ \frac{1}{5}$ where $0 < a < 1$, when we select $n=20$ and $a=\frac{1}{3}$ and plot the sequence, it is clearly observed that the sequence $(x_n)$ converges to a constant value= $ 0.15$.

 ### 6. Sequences via recurrence relations - 3

Let $x_0 = 1$ and $x_{i+1} = - a x_i$ where $0 < a \leq 1$.

a) Plot the first 20 points in $(x_n)$ when $a=1$.

b) What happens with the convergance of the given sequences $(x_n)$?

```python
import matplotlib.pyplot as plt
%matplotlib inline

x = 1 #Initializing the sequence
a = 1 #Initializing the constant
A , B= [],[] #Intializing empty arrays for storing number of iterations and sequence outputs
#Loop for getting the sequence
for n in range(20):
  x = -a*x
  A.append(n)
  B.append(x)
  print(n+1,")",x, end=',')
#Plotting the sequence
plt.plot(A, B, 'g', label='-ax, a=1')
plt.title('P6. Sequences via recurrence relations-3')
plt.xlabel('Input-n')
plt.ylabel('x_n')
plt.legend()
```

For the given conditions: $x_0 = 1$ and $x_{i+1} = - a x_i$ where $0 < a \leq 1$, when we select $n=20$ and $a=1$ and plot the sequence, it is clearly observed that the sequence $(x_n)$ is not convergent and becomes cyclic and ranges between $-1\:to\: 1$.

 ### 7. Sequences via recurrence relations - 7

Let $x_0 = 1$ and $x_{i+1} = a x_i$ where $a \geq 1$.

a) Plot the first 20 points in $(x_n)$ when $a= \frac{3}{2}$.

b) What happens with the convergance of the given sequences $(x_n)$?

```python
import matplotlib.pyplot as plt
%matplotlib inline

x = 1 #Initializing the sequence
a = 3/2 #Initializing the constant
A , B= [],[] #Intializing empty arrays for storing number of iterations and sequence outputs
#Loop for getting the sequence
for n in range(20):
  x = a*x
  A.append(n)
  B.append(x)
  print(n+1,')',x, end=',')
#Plotting the sequence
plt.plot(A, B, 'y', label='ax, a=3/2')
plt.title('P7. Sequences via recurrence relations-7')
plt.xlabel('Input-n')
plt.ylabel('x_n')
plt.legend()
```

For the given conditions: $x_0 = 1$ and $x_{i+1} = a x_i$ where $a \geq 1$, when we select $n=20$ and $a=\frac{3}{2}$ and plot the sequence, it is clearly observed that the sequence $(x_n)$ is not convergent as it moves towards infinity.

### 8. Definition of Derivative

Use the definition of the derivative to find the derivatives of the following functions.

a) $f(x) = x^3$, for any $x$

b) $g(x) = 1/x$, for $x \neq 0$

c) $h(x) = \sqrt{x}$, for $x > 0$

d) $k(x) = x^{-1/2}$ for $x > 0$.

$$a) f(x)=x^3$$
$$Using\:Power\:Rule,\frac{dx^n}{dx}=nx^{n-1}, for\: n=3$$
$$df(x)/dx= 3x2$$
$$----------------------------------------------$$
$$b) g(x)=1/x\:or\: x^{-1}$$
$$Using\:Power\:Rule,\frac{dx^n}{dx}=nx^{n-1}, for\: n=-1$$
$$dg(x)/dx= -1x^{-2}\:or\: \frac{-1}{x^2}$$
$$----------------------------------------------$$
$$c) h(x)=\sqrt{x}\: or\: x^{1/2}$$
$$Using\:Power\:Rule,\frac{dx^n}{dx}=nx^{n-1}, for\: n=1/2$$
$$dh(x)/dx= (1/2)*x^{-1/2}\:  or\:  \frac{1}{2\sqrt{x}}$$
$$----------------------------------------------$$
$$d) k(x)=x^{-1/2}$$
$$Using\:Power\:Rule,\frac{dx^n}{dx}=nx^{n-1}, for\: n=-1/2$$
$$dk(x)/dx= (-1/2)*x^{-3/2}\: or\:  \frac{-1}{2x\sqrt{x}}$$
$$----------------------------------------------$$

### 9. The Chain Rule

Use the chain rule to find these deriviatives. Verify your results with `sympy`.

a) $f(x) = \frac{1}{1+x^2}$

b) $h(x) = (\sin x^k)^m$ for $m,k \in \mathbb{N}$

$a)f(x)= \frac{1}{1+x^2}$
$$Let,u = x^2, f = \frac{1}{1+u} or (1+u)^{-1}$$
$$Using\:Power\:Rule,du/dx=2x\:,df/du=-1*(1+u)^{-2}\:or\:\frac{-1}{(1+u)^2}$$
$$Using\:Chain\:Rule,\frac{df}{dx}=\frac{df}{du}\frac{du}{dx}=\frac{-2x}{(1+u)^2}=\frac{-2x}{(1+x^2)^2}$$
$$----------------------------------------------$$
$b)h(x)= (\sin x^k)^m$
$$Let,u = x^k, v = \sin u, f = v^m$$
$$Using\:Power\:Rule,du/dx=kx^{k-1}\:,dv/du=cos u\:,df/dv=mv^{m-1}$$
$$Using\:Chain\:Rule,\frac{df}{dx}=\frac{df}{dv}\frac{dv}{du}\frac{du}{dx}=mv^{m-1}*cos u*kx^{k-1}$$
$$=mkx^{k-1}\ (sin x^k)^{m-1}cos x^k$$
$$----------------------------------------------$$

```python
#Code for P9.a
import sympy as sm
#Initializing the symbol
x = sm.symbols("x")
sm.diff(1/(1+x**2),x) #Performing the differentiation
```

$\displaystyle - \frac{2 x}{\left(x^{2} + 1\right)^{2}}$

```python
#Code for P9.b
import sympy as sm
#Initializing the symbols
x = sm.symbols("x")
k = sm.symbols("k")
m = sm.symbols("m")

#Substituting the symbols with some variables
u = x**k
v = sm.sin(x)
f = x**m
vou = v.subs(x,u)
fov = f.subs(x,vou)

dfov = sm.diff(fov,x) #Performing the differentiation

dfov
```

$\displaystyle \frac{k m x^{k} \sin^{m}{\left(x^{k} \right)} \cos{\left(x^{k} \right)}}{x \sin{\left(x^{k} \right)}}$


# Обучение нейрона с помощью функции потерь LogLoss

Нейрон с сигмоидой

Снова рассмотрим нейрон с сигмоидой, то есть $$f(x) = \sigma(x)=\frac{1}{1+e^{-x}}$$

Ранее мы установили, что **обучение нейрона с сигмоидой с квадратичной функцией потерь**:

$$MSE(w, x) = \frac{1}{2n}\sum_{i=1}^{n} (\hat{y_i} - y_i)^2 = \frac{1}{2n}\sum_{i=1}^{n} (\sigma(w \cdot x_i) - y_i)^2$$

где $w \cdot x_i$ - скалярное произведение, а $\sigma(w \cdot x_i) =\frac{1}{1+e^{-w \cdot x_i}} $ - сигмоида -- **неэффективно**, то есть мы увидели, что даже за большое количество итераций нейрон предсказывает плохо.

Давайте ещё раз взглянем на формулу для градиентного спуска от функции потерь $MSE$ по весам нейрона:

$$ \frac{\partial MSE}{\partial w} = \frac{1}{n} X^T (\sigma(w \cdot X) - y)\sigma(w \cdot X)(1 - \sigma(w \cdot X))$$

А теперь смотрим на график сигмоиды:

**Её значения: числа от 0 до 1.**

Если получше проанализировать формулу, то теперь можно заметить, что, поскольку сигмоида принимает значения между 0 и 1 (а значит (1-$\sigma$) тоже принимает значения от 0 до 1), то мы умножаем $X^T$ на столбец $(\sigma(w \cdot X) - y)$ из чисел от -1 до 1, а потом ещё на столбцы $\sigma(w \cdot X)$ и $(1 - \sigma(w \cdot X))$ из чисел от 0 до 1. Таким образом в лучшем случае $\frac{\partial{Loss}}{\partial{w}}$ будет столбцом из чисел, порядок которых максимум 0.01 (в среднем, понятно, что если сигмоида выдаёт все 0, то будет 0, если все 1, то тоже 0). После этого мы умножаем на шаг градиентного спуска, который обычно порядка 0.001 или 0.1 максимум. То есть мы вычитаем из весов числа порядка ~0.0001. Медленновато спускаемся, не правда ли? Это называют **проблемой затухающих градиентов**.

Чтобы избежать эту проблему в задачах классификации, в которых моделью является нейрон с сигмоидной функцией активации, предсказывающий "вероятности" принадлженостей к классамиспользуют **LogLoss**:

$$J(\hat{y}, y) = -\frac{1}{n} \sum_{i=1}^n y_i \log(\hat{y_i}) + (1 - y_i) \log(1 - \hat{y_i}) = -\frac{1}{n} \sum_{i=1}^n y_i \log(\sigma(w \cdot x_i)) + (1 - y_i) \log(1 - \sigma(w \cdot x_i))$$

где, как и прежде, $y$ - столбец $(n, 1)$ из истинных значений классов, а $\hat{y}$ - столбец $(n, 1)$ из предсказаний нейрона.

```
from matplotlib import pyplot as plt
from matplotlib.colors import ListedColormap
import numpy as np
import pandas as pd
```

```
def loss(y_pred, y):
    return -np.mean(y * np.log(y_pred) + (1 - y) * np.log(1 - y_pred))
```

Отметим, что сейчас речь идёт именно о **бинарной классификации (на два класса)**, в многоклассовой классификации используется функция потерь под названием *кросс-энтропия*, которая является обобщением LogLoss'а на случай нескольких классов.

Почему же теперь всё будет лучше? Раньше была проблема умножения маленьких чисел в градиенте. Давайте посмотрим, что теперь:

* Для веса $w_j$:

$$ \frac{\partial Loss}{\partial w_j} =
-\frac{1}{n} \sum_{i=1}^n \left(\frac{y_i}{\sigma(w \cdot x_i)} - \frac{1 - y_i}{1 - \sigma(w \cdot x_i)}\right)(\sigma(w \cdot x_i))_{w_j}' = -\frac{1}{n} \sum_{i=1}^n \left(\frac{y_i}{\sigma(w \cdot x_i)} - \frac{1 - y_i}{1 - \sigma(w \cdot x_i)}\right)\sigma(w \cdot x_i)(1 - \sigma(w \cdot x_i))x_{ij} = $$
$$-\frac{1}{n} \sum_{i=1}^n \left(y_i - \sigma(w \cdot x_i)\right)x_{ij}$$

* Градиент $Loss$'а по вектору весов -- это вектор, $j$-ая компонента которого равна $\frac{\partial Loss}{\partial w_j}$ (помним, что весов всего $m$):

$$\begin{align}
    \frac{\partial Loss}{\partial w} &= \begin{bmatrix}
           -\frac{1}{n} \sum_{i=1}^n \left(y_i - \sigma(w \cdot x_i)\right)x_{i1} \\
           -\frac{1}{n} \sum_{i=1}^n \left(y_i - \sigma(w \cdot x_i)\right)x_{i2} \\
           \vdots \\
           -\frac{1}{n} \sum_{i=1}^n \left(y_i - \sigma(w \cdot x_i)\right)x_{im}
         \end{bmatrix}
\end{align}=\frac{1}{n} X^T \left(\hat{y} - y\right)$$

По аналогии с $w_j$ выведите формулу для свободного члена (bias'а) $b$ (*hint*: можно считать, что при нём есть признак $x_{i0}=1$ на всех $i$):

Получили новое правило для обновления $w$ и $b$.

```
def sigmoid(x):
    """Сигмоидальная функция"""
    return 1 / (1 + np.exp(-x))
```

Реализуйте нейрон с функцией потерь LogLoss:

```
class Neuron:

    def __init__(self, w=None, b=0):
        """
        :param: w -- вектор весов
        :param: b -- смещение
        """
        # пока что мы не знаем размер матрицы X, а значит не знаем, сколько будет весов
        self.w = w
        self.b = b

    def activate(self, x):
        return sigmoid(x)

    def forward_pass(self, X):
        """
        Эта функция рассчитывает ответ нейрона при предъявлении набора объектов
        :param: X -- матрица объектов размера (n, m), каждая строка - отдельный объект
        :return: вектор размера (n, 1) из нулей и единиц с ответами перцептрона
        """
        # реализуйте forward_pass
        n = X.shape[0]
        y_pred = np.zeros((n, 1))
        y_pred = self.activate(X @ self.w.reshape(X.shape[1], 1) + self.b)
        return y_pred.reshape(-1, 1)

    def backward_pass(self, X, y, y_pred, learning_rate=0.1):
        """
        Обновляет значения весов нейрона в соответствие с этим объектом
        :param: X -- матрица объектов размера (n, m)
                y -- вектор правильных ответов размера (n, 1)
                learning_rate - "скорость обучения" (символ alpha в формулах выше)
        В этом методе ничего возвращать не нужно, только правильно поменять веса
        с помощью градиентного спуска.
        """
        # тут нужно обновить веса по формулам, написанным выше
        n = len(y)
        y = np.array(y).reshape(-1, 1)
        sigma = self.activate(X @ self.w + self.b)
        self.w = self.w - learning_rate * (X.T @ (sigma - y)) / n
        self.b = self.b - learning_rate * np.mean(sigma - y)

    def fit(self, X, y, num_epochs=5000):
        """
        Спускаемся в минимум
        :param: X -- матрица объектов размера (n, m)
                y -- вектор правильных ответов размера (n, 1)
                num_epochs -- количество итераций обучения
        :return: J_values -- вектор значений функции потерь
        """
        self.w = np.zeros((X.shape[1], 1))  # столбец (m, 1)
        self.b = 0  # смещение
        loss_values = []  # значения функции потерь на различных итерациях обновления весов

        for i in range(num_epochs):
            # предсказания с текущими весами
            y_pred = self.forward_pass(X)
            # считаем функцию потерь с текущими весами
            loss_values.append(loss(y_pred, y))
            # обновляем веса по формуле градиентного спуска
            self.backward_pass(X, y, y_pred)

        return loss_values
```

Тестирование

Протестируем нейрон, обученный с новой функцией потерь, на тех же данных, что и в предыдущем ноутбуке:

**Проверка forward_pass()**

```
w = np.array([1., 2.]).reshape(2, 1)
b = 2.
X = np.array([[1., 3.],
              [2., 4.],
              [-1., -3.2]])

neuron = Neuron(w, b)
y_pred = neuron.forward_pass(X)
print("y_pred = " + str(y_pred))
```

    y_pred = [[0.99987661]
     [0.99999386]
     [0.00449627]]

**Проверка backward_pass()**

```
y = np.array([1, 0, 1]).reshape(3, 1)
```

```
neuron.backward_pass(X, y, y_pred)

print("w = " + str(neuron.w))
print("b = " + str(neuron.b))
```

    w = [[0.9001544 ]
     [1.76049276]]
    b = 1.9998544421863216

Проверьте на наборах данных "яблоки и груши" и "голос".

```
data_apples_pears = pd.read_csv('apples_pears.csv')
```

```
data_apples_pears.head()
```

      yellowness
      symmetry
      target

      0
      0.779427
      0.257305
      1.0

      1
      0.777005
      0.015915
      1.0

      2
      0.977092
      0.304210
      1.0

      3
      0.043032
      0.140899
      0.0

      4
      0.760433
      0.193123
      1.0

```
plt.figure(figsize=(10, 8))
plt.scatter(data_apples_pears.iloc[:, 0], data_apples_pears.iloc[:, 1], c=data_apples_pears['target'], cmap='rainbow')
plt.title('Яблоки и груши', fontsize=15)
plt.xlabel('симметричность', fontsize=14)
plt.ylabel('желтизна', fontsize=14)
plt.show();
```

```
X = data_apples_pears.iloc[:,:2].values  # матрица объекты-признаки
y = data_apples_pears['target'].values.reshape((-1, 1))  # классы (столбец из нулей и единиц)
```

```
%%time
neuron = Neuron(w=np.random.rand(X.shape[1], 1), b=np.random.rand(1))
losses = neuron.fit(X, y, num_epochs=10000)

plt.figure(figsize=(10, 8))
plt.plot(losses)
plt.title('Функция потерь', fontsize=15)
plt.xlabel('номер итерации', fontsize=14)
plt.ylabel('$LogLoss(\hat{y}, y)$', fontsize=14)
plt.show()
```

```
plt.figure(figsize=(10, 8))
plt.scatter(data_apples_pears.iloc[:, 0], data_apples_pears.iloc[:, 1], c=np.array(neuron.forward_pass(X) > 0.7).ravel(), cmap='spring')
plt.title('Яблоки и груши', fontsize=15)
plt.xlabel('симметричность', fontsize=14)
plt.ylabel('желтизна', fontsize=14)
plt.show();
```

```
y_pred = np.array(neuron.forward_pass(X) > 0.7).ravel()
from sklearn.metrics import accuracy_score
print('Точность (доля правильных ответов, из 100%) нашего нейрона: {:.3f} %'.format(
    accuracy_score(y, y_pred) * 100))
```

    Точность (доля правильных ответов, из 100%) нашего нейрона: 98.800 %

```
data_voice = pd.read_csv("voice.csv")
data_voice['label'] = data_voice['label'].apply(lambda x: 1 if x == 'male' else 0)
```

```
data_voice.head()
```

      meanfreq
      sd
      median
      Q25
      Q75
      IQR
      skew
      kurt
      sp.ent
      sfm
      mode
      centroid
      meanfun
      minfun
      maxfun
      meandom
      mindom
      maxdom
      dfrange
      modindx
      label

      0
      0.059781
      0.064241
      0.032027
      0.015071
      0.090193
      0.075122
      12.863462
      274.402906
      0.893369
      0.491918
      0.000000
      0.059781
      0.084279
      0.015702
      0.275862
      0.007812
      0.007812
      0.007812
      0.000000
      0.000000
      1

      1
      0.066009
      0.067310
      0.040229
      0.019414
      0.092666
      0.073252
      22.423285
      634.613855
      0.892193
      0.513724
      0.000000
      0.066009
      0.107937
      0.015826
      0.250000
      0.009014
      0.007812
      0.054688
      0.046875
      0.052632
      1

      2
      0.077316
      0.083829
      0.036718
      0.008701
      0.131908
      0.123207
      30.757155
      1024.927705
      0.846389
      0.478905
      0.000000
      0.077316
      0.098706
      0.015656
      0.271186
      0.007990
      0.007812
      0.015625
      0.007812
      0.046512
      1

      3
      0.151228
      0.072111
      0.158011
      0.096582
      0.207955
      0.111374
      1.232831
      4.177296
      0.963322
      0.727232
      0.083878
      0.151228
      0.088965
      0.017798
      0.250000
      0.201497
      0.007812
      0.562500
      0.554688
      0.247119
      1

      4
      0.135120
      0.079146
      0.124656
      0.078720
      0.206045
      0.127325
      1.101174
      4.333713
      0.971955
      0.783568
      0.104261
      0.135120
      0.106398
      0.016931
      0.266667
      0.712812
      0.007812
      5.484375
      5.476562
      0.208274
      1

```
# Чтобы перемешать данные. Изначально там сначала идут все мужчины, потом все женщины
data_voice = data_voice.sample(frac=1)
```

```
X_train = data_voice.iloc[:int(len(data_voice)*0.7), :-1]  # матрица объекты-признаки
y_train = data_voice.iloc[:int(len(data_voice)*0.7), -1]  # истинные значения пола (мужчина/женщина)

X_test = data_voice.iloc[int(len(data_voice)*0.7):, :-1]  # матрица объекты-признаки
y_test = data_voice.iloc[int(len(data_voice)*0.7):, -1]  # истинные значения пола (мужчина/женщина)
```

```
from sklearn.preprocessing import StandardScaler
```

```
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train.values)
X_test = scaler.transform(X_test.values)
```

```
plt.figure(figsize=(10, 8))
plt.scatter(X_test[:, 0], X_test[:, 1], c=y_test, cmap='rainbow')
plt.title('Мужские и женские голоса', fontsize=15)
plt.show();
```

```
neuron = Neuron(w=np.random.rand(X.shape[1], 1), b=np.random.rand(1))
losses = neuron.fit(X_train, y_train.values);
```

```
y_pred = neuron.forward_pass(X_test)
y_pred = (y_pred > 0.5).astype(int)
```

```
from sklearn.metrics import accuracy_score
print('Точность (доля правильных ответов, из 100%) нашего нейрона: {:.3f} %'.format(
    accuracy_score(y_test, y_pred) * 100))
```

    Точность (доля правильных ответов, из 100%) нашего нейрона: 97.476 %

```
plt.figure(figsize=(10, 8))
plt.scatter(X_test[:, 0], X_test[:, 1], c=y_pred, cmap='spring')
plt.title('Мужские и женские голоса', fontsize=15)
plt.show();
```

```

```

Neural_Networks/test 1 neuron logloss.ipynb


setup:

```python
import numpy as np
import matplotlib.pyplot as plt
from numba import njit, prange
import matplotlib.pyplot as plt
import math

plt.style.use("seaborn-talk")
```

```python
def plot_performance(time_dict):
    mean = [t.average for t in time_dict.values()]
    std = [t.stdev for t in time_dict.values()]
    x = range(len(time_dict))
    plt.errorbar(x, mean, yerr=std, lw=3, fmt="o")
    plt.xticks( np.arange(len(time_dict)), time_dict.keys())
    plt.ylabel("average execution time (s)")
    plt.grid()
```

# Optimization

## Memoization

Remember the fibonnaci problem we had in the debugging lecture?

```python
GOLDEN = (1 + 5 ** 0.5) / 2

def fibonacci(k):

    if k == 0:
        return 0
    if k == 1:
        return 1

    return fibonacci(k - 2) + fibonacci(k - 1)

def compute_golden_ratio(accuracy_level=10):
    return fibonacci(accuracy_level) / fibonacci(accuracy_level - 1)

def plot_golden_ratio_approx(max_k=20):

    ratios = []

    for ii in range(2, max_k):
        ratio = compute_golden_ratio(ii)
        ratios.append(ratio - GOLDEN)

    plt.axhline(0, alpha=0.5, lw=3, color="red")
    plt.scatter(range(2, max_k), ratios)
    plt.ylabel("difference from Golden Ratio")
```

```python

```

We got it to work, but there is still one problem: it's super slow

```python
t = {}
for ii in range(30):
    t[ii] = %timeit -o -n1 -r1 fibonacci(ii)
plot_performance(t)
```

*uh oh!*  We seem to have an algorithm that is $O(k^N)$!

Can we fix it?

```python

```

```python

```

## Speed up a real-world problem!

Here your task is to speed up an algorithm for finding the solution to the *Heat Equation* in 2D using a finite-difference method, given an initial temperature distribution.

the heat equation is defined as:

$$ \frac{du}{dt}  = \alpha \nabla^2 u$$

Where $u$ is the temperature. This can be approximated simply by iterating in time and approximating the spatial gradient using neighboring array elements. For time-step $k$ of width $\Delta t$ and spatial width $\Delta x$:

\begin{equation}
\frac{u^{k+1}_{ij}-u^k_{ij}}{\Delta t}  = \frac{\alpha}{(\Delta x)^2} \left( u^k_{i,j-1} + u^k_{i-1,j} - 4 u^k_{i,j} + u^k_{i+1,j} + u^k_{i,j+1}\right)
\end{equation}

Below we give a naïeve way to solve this, using for-loops (which are not ideal in python). See if you can speed this up by either:

1. re-writing the code to use numpy to get rid of the spatial loops
2. using cython or numba to compile the function (may need to experiment also with some of their compile options)

You should also try to see what the memory usage is! (hint: use the memory_profiler module). Is there a memory leak?

#### the setup:
set up the initial condtions (defining the temperature at the boundary, as well as some hot-spots that are initially at a particular temperature)

```python
N=100; M=100  # define the spatial grid
grid = np.zeros(shape=(N,M))
grid[10:40,:] = 100 # a hot-spot on the border
grid[1:-1,1:-1] = 50 # some initial temperature in the middle
grid[60:80, 40:60] = 150 # a hot-spot initially heated at the start, that will cool down

plt.pcolormesh(grid)
plt.colorbar()
plt.title("initial conditions")
```

```python
def solve_heat_equation_loops(init_cond, iterations=100, delta_x=1.0, alpha=1.0):

    delta_t = delta_x**2/(4*alpha)
    prev = np.copy(init_cond)
    cur = np.copy(init_cond)
    N,M = init_cond.shape

    for k in range(iterations):
        for i in range(1,N-1):
            for j in range(1,M-1):
                cur[i,j] = prev[i,j] + alpha*delta_t/delta_x**2 * (
                    prev[i,j-1] + prev[i-1,j] - 4*prev[i,j] + prev[i,j+1] + prev[i+1,j]
                )

        prev,cur = cur,prev  #swap pointers

    return prev
```

We'll also define a convenience function to test the results (you can use this same plotter with your own solver)

```python
def plot_heat_equation(solver, iters=(2,10,100,1000)):

    fig, axes = plt.subplots(1,len(iters), figsize=(15,3))
    fig.suptitle(solver.__name__)

    for ii, iterations in enumerate(iters):
        result = solver(init_cond=grid, iterations=iterations)
        axes[ii].pcolormesh(result, vmin=0, vmax=100)
        axes[ii].set_title("{} iterations".format(iterations))

```

```python
plot_heat_equation(solver=solve_heat_equation_loops)
```

Note that our code is quite slow...

### Your turn!

***Write an improved verson***

* how much faster is your version on average?
* how much memory does it use on average? Is it more than the loop version?
* which line is the slowest line?

(hint: if done right, you should get a factor of about 100 speed increase)

```python
def my_heat_equation_solver(init_cond, iterations=100, delta_x=1.0, alpha=1.0):
    ## your code here
    return init_cond # replace with real return value
```

```python
#plot_heat_equation(solver=my_heat_equation_solver)
```

### SOLUTION

there are many ways to achieve this...

```python
ITERATIONS=50

results = {}
r = %timeit -o solve_heat_equation_loops(grid, iterations=50)
results['loop'] = r
```

#### using Numba:

```python
from numba import jit
solve_heat_equation_numba = njit(solve_heat_equation_loops)
```

```python
r = %timeit -o solve_heat_equation_numba(grid, iterations=50)
results['numba'] = r
```

```python
# (Note: open some views so we can see this always)
plt.figure(figsize=(5,5))
plot_performance(results)
plt.semilogy()
```

#### using numpy

using range-slicing `array[start:end,start:end]` we can get rid of the inner for-loops and turn that part into vector operations

```python
def solve_heat_equation_numpy(init_cond,  iterations=100,  delta_x=1.0, alpha=1.0):

    delta_t = delta_x**2/(4*alpha)
    prev = np.copy(init_cond)
    cur = np.copy(init_cond)

    # define some slices to make it easier to type
    # just avoids too many things like prev[1:-1,1:-1])
    Z = slice(1,-1) # zero
    P = slice(2,None) # plus 1
    M = slice(0,-2) # minus 1

    for k in range(iterations):
        cur[Z,Z] = (
            prev[Z,Z] + alpha*delta_t/delta_x**2 * (
                prev[Z,M] + prev[M,Z] - 4.0*prev[Z,Z] + prev[Z,P] + prev[P,Z]
            )
        )
        prev,cur = cur,prev # swap the pointers

    return prev # since we swapped, prev is the most recent
```

```python
plot_heat_equation(solver=solve_heat_equation_numpy)
```

```python
r = %timeit -o solve_heat_equation_numpy(grid, iterations=ITERATIONS)
results['numpy'] = r
```

#### With numpy and numba

```python
solve_heat_equation_numpy_numba = njit(solve_heat_equation_numpy)

# "prime" it (compile)
plot_heat_equation(solver=solve_heat_equation_numpy_numba)
```

```python
r = %timeit -o solve_heat_equation_numpy_numba(grid, iterations=ITERATIONS)
results['numpy\nnumba'] = r
```

#### using cython:

Cython is a special python-like language that is translated into C-code (or C++ if you request it), and then compiled with your C compiler with a python binding produced automatically.

It has to be explicity compiled (unlike Numba which is "just in time" (JIT) compiled)

```python
%load_ext cython
```

```cython
%%cython

cimport numpy as cnp
import numpy as np

def solve_heat_equation_cython(init_cond, int iterations=100, double delta_x=1.0, double alpha=1.0):

    cdef int i,j,k, N, M
    cdef float delta_t
    cdef cnp.ndarray[double, mode="c", ndim=2] prev, cur  # this seems to give the biggest improvement

    delta_t = delta_x**2/(4*alpha)
    prev = np.copy(init_cond)
    cur = np.copy(init_cond)
    N,M = init_cond.shape

    for k in range(iterations):
        for i in range(1,N-1):
            for j in range(1,M-1):
                cur[i,j] = prev[i,j] + alpha*delta_t/delta_x**2 * (
                    prev[i,j-1] + prev[i-1,j] - 4*prev[i,j] + prev[i,j+1] + prev[i+1,j]
                )

        prev,cur = cur,prev

    return prev

```

Try running the previous cell with `%%cython -a`  to get an annotated version to see what it did!

```python
r = %timeit -o solve_heat_equation_cython(grid, iterations=ITERATIONS)
results['cython'] = r
```

#### results

```python
plot_performance(results)
plt.semilogy()
```

### We will come back to this in the next lecture

#### with some parallelization

```python
solve_heat_equation_numpy_numba_parallel = njit(solve_heat_equation_numpy, parallel=True)
```

```python
plot_heat_equation(solver=solve_heat_equation_numpy_numba_parallel)
```

```python
r = %timeit -o solve_heat_equation_numpy_numba_parallel(grid, iterations=50)
results['numpy\nnumba\nparallel'] = r
```

Nice!

*But did it do anything?*

```python
solve_heat_equation_numpy_numba_parallel.parallel_diagnostics()
```

#### with explicit parallelization

```python
@njit(parallel=True)
def solve_heat_equation_numpy_numba_parallel_explicit(init_cond,  iterations=100,  delta_x=1.0, alpha=1.0):

    delta_t = delta_x**2/(4*alpha)
    prev = np.copy(init_cond)
    cur = np.copy(init_cond)

    # define some slices to make it easier to type
    # just avoids too many things like prev[1:-1,1:-1])
    Z = slice(1,-1) # zero
    P = slice(2,None) # plus 1
    M = slice(0,-2) # minus 1

    for k in prange(iterations):
        cur[Z,Z] = (
            prev[Z,Z] + alpha*delta_t/delta_x**2 * (
                prev[Z,M] + prev[M,Z] - 4.0*prev[Z,Z] + prev[Z,P] + prev[P,Z]
            )
        )
        prev,cur = cur,prev # swap the pointers

    return prev # since we swapped, prev is the most recent

answer = solve_heat_equation_numpy_numba_parallel_explicit(grid, iterations=50)

```

<div class="alert alert-block alert-warning">
    WARNING WARNING
</div>

```python
plot_heat_equation(solver=solve_heat_equation_numpy_numba_parallel_explicit)
```

```python
r = %timeit -o solve_heat_equation_numpy_numba_parallel_explicit(grid, iterations=50)
results['numpy\nnumba\nparallel\nexplicit'] = r
```

```python
plot_performance(results)
plt.semilogy()
```

```python
plot_heat_equation(solver=solve_heat_equation_numpy_numba_parallel)
plot_heat_equation(solver=solve_heat_equation_numpy_numba_parallel_explicit)
```

```python

```

```python
Let's look at some code that can be parallelized by numnba
```