# Assignment 1

The goal of this assignment is to supply you with machine learning models and algorithms. In this notebook, we will cover linear and nonlinear models, the concept of loss functions and some optimization techniques. All mathematical operations should be implemented in **NumPy** only.

## Table of contents
* [1. Logistic Regression](#1.-Logistic-Regression)
    * [1.1 Linear Mapping](#1.1-Linear-Mapping)
    * [1.2 Sigmoid](#1.2-Sigmoid)
    * [1.3 Negative Log Likelihood](#1.3-Negative-Log-Likelihood)
    * [1.4 Model](#1.4-Model)
    * [1.5 Simple Experiment](#1.5-Simple-Experiment)
* [2. Decision Tree](#2.-Decision-Tree)
    * [2.1 Gini Index & Data Split](#2.1-Gini-Index-&-Data-Split)
    * [2.2 Terminal Node](#2.2-Terminal-Node)
    * [2.3 Build the Decision Tree](#2.3-Build-the-Decision-Tree)
* [3. Experiments](#3.-Experiments)
    * [3.1 Decision Tree for Heart Disease Prediction](#3.1-Decision-Tree-for-Heart-Disease-Prediction)
    * [3.2 Logistic Regression for Heart Disease Prediction](#3.2-Logistic-Regression-for-Heart-Disease-Prediction)

### Note
Some of the concepts below have not (yet) been discussed during the lecture. These will be discussed further during the next lectures.

### Before you begin

To check whether the code you've written is correct, we'll use **automark**. For this, we created for each of you an account with the username being your student number.

```python
import automark as am

# fill in you student number as your username
username = '13300040'

# to check your progress, you can run this function
am.get_progress(username)
```

    ---------------------------------------------
    | Magdalena Mladenova                       |
    | magdalena.mladenova@student.uva.nl        |
    ---------------------------------------------
    | linear_forward           | completed      |
    | linear_grad_W            | completed      |
    | linear_grad_b            | completed      |
    | nll_forward              | completed      |
    | nll_grad_input           | not attempted  |
    | sigmoid_forward          | completed      |
    | sigmoid_grad_input       | completed      |
    | tree_gini_index          | not attempted  |
    | tree_split_data_left     | not attempted  |
    | tree_split_data_right    | not attempted  |
    | tree_to_terminal         | not attempted  |
    ---------------------------------------------

So far all your tests are 'not attempted'. At the end of this notebook you'll need to have completed all test. The output of `am.get_progress(username)` should at least match the example below. However, we encourage you to take a shot at the 'not attempted' tests!

```
---------------------------------------------
| Your name / student number                |
| your_email@your_domain.whatever           |
---------------------------------------------
| linear_forward           | not attempted  |
| linear_grad_W            | not attempted  |
| linear_grad_b            | not attempted  |
| nll_forward              | not attempted  |
| nll_grad_input           | not attempted  |
| sigmoid_forward          | not attempted  |
| sigmoid_grad_input       | not attempted  |
| tree_data_split_left     | not attempted  |
| tree_data_split_right    | not attempted  |
| tree_gini_index          | not attempted  |
| tree_to_terminal         | not attempted  |
---------------------------------------------
```

```python
from __future__ import print_function, absolute_import, division
import numpy as np
```

This notebook makes use of **classes** and their **instances** that we have already implemented for you. It allows us to write less code and make it more readable. If you are interested in it, here are some useful links:
* The official [documentation](https://docs.python.org/3/tutorial/classes.html)
* Video by *sentdex*: [Object Oriented Programming Introduction](https://www.youtube.com/watch?v=ekA6hvk-8H8)
* Antipatterns in OOP: [Stop Writing Classes](https://www.youtube.com/watch?v=o9pEzgHorH0)

# 1. Logistic Regression

We start with a very simple algorithm called **Logistic Regression**. It is a generalized linear model for 2-class classification.
It can be generalized to the case of many classes and to non-linear cases as well. However, here we consider only the simplest case.

Let us consider a data with 2 classes. Class 0 and class 1. For a given test sample, logistic regression returns a value from $[0, 1]$ which is interpreted as a probability of belonging to class 1. The set of points for which the prediction is $0.5$ is called a *decision boundary*. It is a line on a plane or a hyper-plane in a space.

Logistic regression has two trainable parameters: a weight $W$ and a bias $b$. For a vector of features $X$, the prediction of logistic regression is given by

$$
f(X) = \frac{1}{1 + \exp(-[XW + b])} = \sigma(h(X))
$$
where $\sigma(z) = \frac{1}{1 + \exp(-z)}$ and $h(X)=XW + b$.

Parameters $W$ and $b$ are fitted by maximizing the log-likelihood (or minimizing the negative log-likelihood) of the model on the training data. For a training subset $\{X_j, Y_j\}_{j=1}^N$ the normalized negative log likelihood (NLL) is given by

$$
\mathcal{L} = -\frac{1}{N}\sum_j \log\Big[ f(X_j)^{Y_j} \cdot (1-f(X_j))^{1-Y_j}\Big]
= -\frac{1}{N}\sum_j \Big[ Y_j\log f(X_j) + (1-Y_j)\log(1-f(X_j))\Big]
$$

There are different ways of fitting this model. In this assignment we consider Logistic Regression as a one-layer neural network. We use the following algorithm for the **forward** pass:

1. Linear mapping: $h=XW + b$
2. Sigmoid activation function: $f=\sigma(h)$
3. Calculation of NLL: $\mathcal{L} = -\frac{1}{N}\sum_j \Big[ Y_j\log f_j + (1-Y_j)\log(1-f_j)\Big]$

In order to fit $W$ and $b$ we perform Gradient Descent ([GD](https://en.wikipedia.org/wiki/Gradient_descent)). We choose a small learning rate $\gamma$ and after each computation of forward pass, we update the parameters

$$W_{\text{new}} = W_{\text{old}} - \gamma \frac{\partial \mathcal{L}}{\partial W}$$

$$b_{\text{new}} = b_{\text{old}} - \gamma \frac{\partial \mathcal{L}}{\partial b}$$

We use Backpropagation method ([BP](https://en.wikipedia.org/wiki/Backpropagation)) to calculate the partial derivatives of the loss function with respect to the parameters of the model.

$$
\frac{\partial\mathcal{L}}{\partial W} =
\frac{\partial\mathcal{L}}{\partial h} \frac{\partial h}{\partial W} =
\frac{\partial\mathcal{L}}{\partial f} \frac{\partial f}{\partial h} \frac{\partial h}{\partial W}
$$

$$
\frac{\partial\mathcal{L}}{\partial b} =
\frac{\partial\mathcal{L}}{\partial h} \frac{\partial h}{\partial b} =
\frac{\partial\mathcal{L}}{\partial f} \frac{\partial f}{\partial h} \frac{\partial h}{\partial b}
$$

## 1.1 Linear Mapping
First of all, you need to implement the forward pass of a linear mapping:
$$
h(X) = XW +b
$$

**Note**: here we use `n_out` as the dimensionality of the output. For logisitc regression `n_out = 1`. However, we will work with cases of `n_out > 1` in next assignments. You will **pass** the current assignment even if your implementation works only in case `n_out = 1`. If your implementation works for the cases of `n_out > 1` then you will not have to modify your method next week. All **numpy** operations are generic. It is recommended to use numpy when is it possible.

```python
def linear_forward(x_input, W, b):
    """Perform the mapping of the input
    # Arguments
        x_input: input of the linear function - np.array of size `(n_objects, n_in)`
        W: np.array of size `(n_in, n_out)`
        b: np.array of size `(n_out,)`
    # Output
        the output of the linear function
        np.array of size `(n_objects, n_out)`
    """
    #assert that dimension x_input and weight are compatible
    XW = x_input.dot(W)
    output = XW + b
    return output
```

Let's check your first function. We set the matrices $X, W, b$:
$$
X = \begin{bmatrix}
1 & -1 \\
-1 & 0 \\
1 & 1 \\
\end{bmatrix} \quad
W = \begin{bmatrix}
4 \\
2 \\
\end{bmatrix} \quad
b = \begin{bmatrix}
3 \\
\end{bmatrix}
$$

And then compute
$$
XW = \begin{bmatrix}
1 & -1 \\
-1 & 0 \\
1 & 1 \\
\end{bmatrix}
\begin{bmatrix}
4 \\
2 \\
\end{bmatrix} =
\begin{bmatrix}
2 \\
-4 \\
6 \\
\end{bmatrix} \\
XW + b =
\begin{bmatrix}
5 \\
-1 \\
9 \\
\end{bmatrix}
$$

```python
X_test = np.array([[1, -1],
                   [-1, 0],
                   [1, 1]])

W_test = np.array([[4],
                   [2]])

b_test = np.array([3])

h_test = linear_forward(X_test, W_test, b_test)
print(h_test)
```

    [[ 5]
     [-1]
     [ 9]]

```python
am.test_student_function(username, linear_forward, ['x_input', 'W', 'b'])
```

    Running local tests...
    linear_forward successfully passed local tests
    Running remote test...
    Test was successful. Congratulations!

Now you need to implement the calculation of the partial derivative of the loss function with respect to the parameters of the model. As this expressions are used for the updates of the parameters, we refer to them as gradients.
$$
\frac{\partial \mathcal{L}}{\partial W} =
\frac{\partial \mathcal{L}}{\partial h}
\frac{\partial h}{\partial W} \\
\frac{\partial \mathcal{L}}{\partial b} =
\frac{\partial \mathcal{L}}{\partial h}
\frac{\partial h}{\partial b} \\
$$

$$
\mathcal{L} = -\frac{1}{N}\sum_j \log\Big[ f(X_j)^{Y_j} \cdot (1-f(X_j))^{1-Y_j}\Big]
= -\frac{1}{N}\sum_j \Big[ Y_j\log f(X_j) + (1-Y_j)\log(1-f(X_j))\Big]
$$

```python
#from sympy import symbols, diff

def linear_grad_W(x_input, grad_output, W, b):
    """Calculate the partial derivative of
        the loss with respect to W parameter of the function
        dL / dW = (dL / dh) * (dh / dW)
    # Arguments
        x_input: input of a dense layer - np.array of size `(n_objects, n_in)`
        grad_output: partial derivative of the loss functions with
            respect to the ouput of the dense layer (dL / dh)
            np.array of size `(n_objects, n_out)`
    # Output
        the partial derivative of the loss
        with respect to W parameter of the function
        np.array of size `(n_in, n_out)`
    """
    x_input = x_input.transpose()
    dl_dh = grad_output
    grad_W = x_input.dot(grad_output)
    return grad_W
```

```python
am.test_student_function(username, linear_grad_W, ['x_input', 'grad_output','W', 'b'])
```

    Running local tests...
    linear_grad_W successfully passed local tests
    Running remote test...
    Test was successful. Congratulations!

```python
def linear_grad_b(x_input, grad_output, W, b):
    """Calculate the partial derivative of
        the loss with respect to b parameter of the function
        dL / db = (dL / dh) * (dh / db)
    # Arguments
        x_input: input of a dense layer - np.array of size `(n_objects, n_in)`
        grad_output: partial derivative of the loss functions with
            respect to the ouput of the linear function (dL / dh)
            np.array of size `(n_objects, n_out)`
        W: np.array of size `(n_in, n_out)`
        b: np.array of size `(n_out,)`
    # Output
        the partial derivative of the loss
        with respect to b parameter of the linear function
        np.array of size `(n_out,)`
    """
    dh_db = np.transpose(np.ones(grad_output.shape))
    grad_b = dh_db.dot(grad_output)
    return grad_b
```

```python
am.test_student_function(username, linear_grad_b, ['x_input', 'grad_output', 'W', 'b'])
```

```python
am.get_progress(username)
```

## 1.2 Sigmoid
$$
f = \sigma(h) = \frac{1}{1 + e^{-h}}
$$

Sigmoid function is applied element-wise. It does not change the dimensionality of the tensor and its implementation is shape-agnostic in general.

```python
from math import e
def sigmoid_forward(x_input):
    """sigmoid nonlinearity
    # Arguments
        x_input: np.array of size `(n_objects, n_in)`
    # Output
        the output of relu layer
        np.array of size `(n_objects, n_in)`
    """
    output = 1/(1+e**(-(x_input)))
    return output
```

```python
am.test_student_function(username, sigmoid_forward, ['x_input'])
```

    Running local tests...
    sigmoid_forward successfully passed local tests
    Running remote test...
    Test was successful. Congratulations!

Now you need to implement the calculation of the partial derivative of the loss function with respect to the input of sigmoid.

$$
\frac{\partial \mathcal{L}}{\partial h} =
\frac{\partial \mathcal{L}}{\partial f}
\frac{\partial f}{\partial h}
$$

Tensor $\frac{\partial \mathcal{L}}{\partial f}$ comes from the loss function. Let's calculate $\frac{\partial f}{\partial h}$

$$
\frac{\partial f}{\partial h} =
\frac{\partial \sigma(h)}{\partial h} =
\frac{\partial}{\partial h} \Big(\frac{1}{1 + e^{-h}}\Big)
= \frac{e^{-h}}{(1 + e^{-h})^2}
= \frac{1}{1 + e^{-h}} \frac{e^{-h}}{1 + e^{-h}}
= f(h) (1 - f(h))
$$

Therefore, in order to calculate the gradient of the loss with respect to the input of sigmoid function you need
to
1. calculate $f(h) (1 - f(h))$
2. multiply it element-wise by $\frac{\partial \mathcal{L}}{\partial f}$

```python
def sigmoid_grad_input(x_input, grad_output):
    """sigmoid nonlinearity gradient.
        Calculate the partial derivative of the loss
        with respect to the input of the layer
    # Arguments
        x_input: np.array of size `(n_objects, n_in)`
        grad_output: np.array of size `(n_objects, n_in)`
            dL / df
    # Output
        the partial derivative of the loss
        with respect to the input of the function
        np.array of size `(n_objects, n_in)`
        dL / dh
    """
    f = 1/(1+e**(-(x_input)))
    f_deriv = f*(1-f)
    grad_input = f_deriv*grad_output
    return grad_input
```

```python
am.test_student_function(username, sigmoid_grad_input, ['x_input', 'grad_output'])
```

    Running local tests...
    sigmoid_grad_input successfully passed local tests
    Running remote test...
    Test was successful. Congratulations!

## 1.3 Negative Log Likelihood

$$
\mathcal{L}
= -\frac{1}{N}\sum_j \Big[ Y_j\log \dot{Y}_j + (1-Y_j)\log(1-\dot{Y}_j)\Big]
$$

Here $N$ is the number of objects. $Y_j$ is the real label of an object and $\dot{Y}_j$ is the predicted one.

```python
def nll_forward(target_pred, target_true):
    """Compute the value of NLL
        for a given prediction and the ground truth
    # Arguments
        target_pred: predictions - np.array of size `(n_objects, 1)`
        target_true: ground truth - np.array of size `(n_objects, 1)`
    # Output
        the value of NLL for a given prediction and the ground truth
        scalar
    """
    n = target_pred.shape[0] #number of objects (predictions)

    loss_array =target_true*(np.log(target_pred)) + (1-target_true)*(np.log(1-target_pred))
    loss = -(((loss_array).sum())/n)
    return loss
```

```python
am.test_student_function(username, nll_forward, ['target_pred', 'target_true'])
```

    Running local tests...
    nll_forward successfully passed local tests
    Running remote test...
    Test was successful. Congratulations!

```python
#Creating variables for testing:
prediction_array = np.array([[0.99],[0.1],[0.1],[0.98]])
true_values_array = np.array([[1],[0],[0],[1]])

#compare to matrices:
prediction_matrix = np.matrix("1;0;0;1")
true_values_matrix = np.matrix("1;1;0;1")
#same visualisation BUT
#* in matrices performs dot multiplication, not calculating Hadamard product

# print(prediction_array)
# print(true_values_array)

# print(prediction_array.shape)
# print(true_values_array.shape)

# prediction_array*true_values_array
```

```python
nll_forward(prediction_array, true_values_array)
```

    -0.06024351862166837

Now you need to calculate the partial derivative of NLL with with respect to its input.

$$
\frac{\partial \mathcal{L}}{\partial \dot{Y}}
=
\begin{pmatrix}
\frac{\partial \mathcal{L}}{\partial \dot{Y}_0} \\
\frac{\partial \mathcal{L}}{\partial \dot{Y}_1} \\
\vdots \\
\frac{\partial \mathcal{L}}{\partial \dot{Y}_N}
\end{pmatrix}
$$

Let's do it step-by-step

\begin{equation}
\begin{split}
\frac{\partial \mathcal{L}}{\partial \dot{Y}_0}
&= \frac{\partial}{\partial \dot{Y}_0} \Big(-\frac{1}{N}\sum_j \Big[ Y_j\log \dot{Y}_j + (1-Y_j)\log(1-\dot{Y}_j)\Big]\Big) \\
&= -\frac{1}{N} \frac{\partial}{\partial \dot{Y}_0} \Big(Y_0\log \dot{Y}_0 + (1-Y_0)\log(1-\dot{Y}_0)\Big) \\
&= -\frac{1}{N} \Big(\frac{Y_0}{\dot{Y}_0} - \frac{1-Y_0}{1-\dot{Y}_0}\Big)
= \frac{1}{N} \frac{\dot{Y}_0 - Y_0}{\dot{Y}_0 (1 - \dot{Y}_0)}
\end{split}
\end{equation}

And for the other components it can be done in exactly the same way. So the result is the vector where each component is given by
$$\frac{1}{N} \frac{\dot{Y}_j - Y_j}{\dot{Y}_j (1 - \dot{Y}_j)}$$

Or if we assume all multiplications and divisions to be done element-wise the output can be calculated as
$$
\frac{\partial \mathcal{L}}{\partial \dot{Y}} = \frac{1}{N} \frac{\dot{Y} - Y}{\dot{Y} (1 - \dot{Y})}
$$

```python
def nll_grad_input(target_pred, target_true):
    """Compute the partial derivative of NLL
        with respect to its input
    # Arguments
        target_pred: predictions - np.array of size `(n_objects, 1)`
        target_true: ground truth - np.array of size `(n_objects, 1)`
    # Output
        the partial derivative
        of NLL with respect to its input
        np.array of size `(n_objects, 1)`
    """
    n = target_pred.shape[0]
    print(target_pred)
    print(n)
    grad_input = (target_pred - target_true)/(n*target_pred*(1-target_pred))
    return grad_input
```

```python
am.test_student_function(username, nll_grad_input, ['target_pred', 'target_true'])
```

    Running local tests...
    [[0.44149218]
     [0.49802603]
     [0.12991776]
     [0.54631544]
     [0.93359245]
     [0.76619702]]
    6
    [[0.4034751 ]
     [0.36231152]]
    2
    [[0.32128973]
     [0.19297686]
     [0.77783477]
     [0.27277821]
     [0.00945926]
     [0.65666999]]
    6
    [[0.44483427]
     [0.73023461]]
    2
    [[0.38822013]
     [0.80142426]
     [0.91762878]
     [0.88301271]
     [0.52892881]
     [0.19129686]]
    6
    [[0.56112845]
     [0.66910656]]
    2
    [[0.46500727]
     [0.47885429]
     [0.96522237]
     [0.41378956]
     [0.32003152]]
    5
    [[0.49075781]
     [0.43031569]]
    2
    [[0.17374548]
     [0.87711852]
     [0.63623989]
     [0.545399  ]
     [0.66406108]
     [0.6646313 ]]
    6
    [[0.2392321 ]
     [0.88579622]
     [0.35941506]
     [0.44902499]
     [0.28456685]
     [0.50962131]]
    6
    [[0.86842763]
     [0.52829461]
     [0.4988752 ]
     [0.21276133]
     [0.16242777]
     [0.77160718]]
    6
    [[0.52347324]
     [0.30609505]
     [0.78676405]
     [0.94166458]
     [0.74196847]
     [0.06675144]]
    6
    [[0.66100102]
     [0.05377475]
     [0.05807855]
     [0.85769082]
     [0.99480223]]
    5
    [[0.00926097]]
    1
    [[0.62023777]
     [0.71894748]
     [0.60263285]
     [0.9874348 ]
     [0.01759958]]
    5
    [[0.02013438]
     [0.44225343]]
    2
    [[0.21919753]
     [0.40846385]]
    2
    [[0.21256704]]
    1
    [[0.04950514]
     [0.29604472]
     [0.09754689]]
    3
    [[0.8871072 ]
     [0.56474609]
     [0.0518327 ]
     [0.54651019]
     [0.14883728]
     [0.18125097]]
    6
    [[0.33114106]
     [0.98654821]
     [0.89025191]
     [0.308293  ]]
    4
    [[0.97033604]
     [0.93875813]
     [0.58919901]
     [0.64780771]
     [0.89180757]
     [0.14617799]]
    6
    [[0.54827957]
     [0.69368268]
     [0.75590782]]
    3
    [[0.83539158]
     [0.51928144]
     [0.31200754]
     [0.42234194]
     [0.93818552]
     [0.00228553]]
    6
    [[0.12989161]
     [0.35335171]
     [0.20183886]
     [0.48871329]
     [0.65245702]]
    5
    [[0.3444705 ]
     [0.40483695]
     [0.41831576]
     [0.37652998]
     [0.98668538]
     [0.20289536]]
    6
    [[0.40453669]
     [0.94760548]]
    2
    [[0.11154913]
     [0.62180863]
     [0.33390252]
     [0.17659504]
     [0.09123636]]
    5
    [[0.93825816]]
    1
    [[0.20019085]
     [0.20798923]
     [0.4055964 ]
     [0.39212545]
     [0.66819044]
     [0.89194762]]
    6
    [[0.03127009]
     [0.02298013]
     [0.01384975]]
    3
    [[0.25762287]
     [0.99217188]
     [0.13204732]]
    3
    [[0.49777484]
     [0.23464077]
     [0.72658589]
     [0.49451356]
     [0.74986547]
     [0.24889401]]
    6
    [[0.71436283]
     [0.20270132]]
    2
    [[0.27169596]
     [0.30269292]
     [0.78654709]
     [0.3069444 ]
     [0.379331  ]
     [0.08090766]]
    6
    [[0.23738474]
     [0.20456698]
     [0.48908061]]
    3
    [[0.50707856]
     [0.03648437]
     [0.9640905 ]]
    3
    [[0.90976558]
     [0.63728016]
     [0.44310998]
     [0.96064619]
     [0.62442183]]
    5
    [[0.50398108]
     [0.29517032]
     [0.90436043]
     [0.36066192]
     [0.21716993]
     [0.76918049]]
    6
    [[0.53772766]]
    1
    [[0.17586308]
     [0.90634808]
     [0.94384896]]
    3
    [[0.49060793]
     [0.55220483]]
    2
    [[0.39519171]]
    1
    [[0.03434373]
     [0.6138267 ]
     [0.47036275]
     [0.16236453]
     [0.09411384]]
    5
    [[0.19089986]
     [0.84761351]
     [0.8462792 ]]
    3
    [[0.39779783]
     [0.84162308]]
    2
    [[0.92406366]
     [0.45031539]
     [0.85402207]
     [0.15316689]
     [0.89275927]]
    5
    [[0.64010471]]
    1
    [[0.33982215]
     [0.67746238]]
    2
    [[0.22783106]
     [0.38975825]]
    2
    [[0.9802084 ]
     [0.53868362]
     [0.01461558]
     [0.62171137]
     [0.04498339]
     [0.88781291]]
    6
    [[0.66474519]]
    1
    [[0.59445941]
     [0.77878887]
     [0.25494849]
     [0.64189979]]
    4
    [[0.7175209]]
    1
    [[0.08854945]
     [0.49604055]
     [0.04302953]
     [0.36824231]
     [0.08992412]]
    5
    [[0.35452476]
     [0.6820189 ]
     [0.80945034]
     [0.1650229 ]
     [0.21320536]
     [0.80255547]]
    6
    [[0.8861651]]
    1
    [[0.29428122]
     [0.40449929]
     [0.94607064]]
    3
    [[0.20451381]
     [0.80731092]]
    2
    [[0.8868128 ]
     [0.68464132]
     [0.96354595]]
    3
    [[0.39985722]
     [0.80953875]]
    2
    [[0.50262563]
     [0.19314674]
     [0.78662357]
     [0.60238456]
     [0.43974697]]
    5
    [[0.20689077]]
    1
    [[0.4228594]]
    1
    [[0.16220913]
     [0.62044583]
     [0.16228701]
     [0.0933396 ]
     [0.58231372]]
    5
    [[0.29729978]
     [0.80872642]
     [0.12308581]
     [0.26117507]
     [0.37221987]]
    5
    [[0.74582136]]
    1
    [[0.59809444]
     [0.13532428]
     [0.29224631]
     [0.51791254]]
    4
    [[0.01087373]
     [0.97127289]
     [0.60748636]
     [0.24023704]]
    4
    [[0.11062596]
     [0.96194054]
     [0.21893109]]
    3
    [[0.13763796]
     [0.26546275]
     [0.39716616]
     [0.23535513]
     [0.60393129]]
    5
    [[0.03977042]
     [0.35924223]
     [0.6552682 ]]
    3
    [[0.48651457]]
    1
    [[0.62433777]]
    1
    [[0.6856632 ]
     [0.07120992]
     [0.20874342]
     [0.54259184]]
    4
    [[0.71545086]
     [0.40261108]
     [0.0311354 ]
     [0.51892539]]
    4
    [[0.60793752]
     [0.14582905]
     [0.48633605]
     [0.37506066]
     [0.21457534]]
    5
    [[0.72303158]
     [0.37431857]
     [0.53187361]
     [0.31838372]
     [0.98794156]
     [0.2571037 ]]
    6
    [[0.09554118]
     [0.47046602]
     [0.92780551]]
    3
    [[0.46190188]
     [0.7461441 ]
     [0.44509945]]
    3
    [[0.33780642]
     [0.62424329]
     [0.45958335]
     [0.79494963]
     [0.56264295]
     [0.19624248]]
    6
    [[0.55639635]
     [0.80387568]
     [0.91549617]
     [0.65021827]
     [0.78376871]
     [0.71997414]]
    6
    [[0.42020722]
     [0.70983398]
     [0.94649174]
     [0.76501971]]
    4
    [[0.0256399 ]
     [0.32513118]
     [0.83752743]]
    3
    [[0.76086103]
     [0.54904677]
     [0.65039979]
     [0.29417816]
     [0.38294631]
     [0.47450256]]
    6
    [[0.62298494]
     [0.58693756]
     [0.86122075]
     [0.86032384]
     [0.86849946]]
    5
    [[0.59638548]
     [0.63719664]
     [0.10874566]]
    3
    [[0.36858834]]
    1
    [[0.23839624]]
    1
    [[0.30664036]
     [0.2475675 ]
     [0.37133409]
     [0.80121306]]
    4
    [[0.98181281]
     [0.32234796]
     [0.28997734]]
    3
    [[0.27833633]
     [0.05237881]
     [0.35524493]
     [0.0249076 ]
     [0.71471005]]
    5
    [[0.12273394]
     [0.9534158 ]
     [0.62409713]]
    3
    [[0.88102583]
     [0.45884159]
     [0.84604589]
     [0.73927688]]
    4
    [[0.30252666]]
    1
    [[0.96295703]
     [0.8342786 ]
     [0.19362649]
     [0.32967574]]
    4
    [[0.69371014]]
    1
    [[0.95569688]]
    1
    [[0.29187442]
     [0.58856177]
     [0.96276033]]
    3
    [[0.03275109]
     [0.14280792]
     [0.82958898]
     [0.86157039]]
    4
    nll_grad_input successfully passed local tests
    Running remote test...
    [[0.21144194]
     [0.43647571]
     [0.12744948]]
    3
    Test was successful. Congratulations!

```python
am.get_progress(username)
```

    ---------------------------------------------
    | Magdalena Mladenova                       |
    | magdalena.mladenova@student.uva.nl        |
    ---------------------------------------------
    | linear_forward           | completed      |
    | linear_grad_W            | completed      |
    | linear_grad_b            | completed      |
    | nll_forward              | completed      |
    | nll_grad_input           | completed      |
    | sigmoid_forward          | completed      |
    | sigmoid_grad_input       | completed      |
    | tree_gini_index          | completed      |
    | tree_split_data_left     | completed      |
    | tree_split_data_right    | completed      |
    | tree_to_terminal         | completed      |
    ---------------------------------------------

## 1.4 Model

Here we provide a model for your. It consist of the function which you have implmeneted above

```python
class LogsticRegressionGD(object):

    def __init__(self, n_in, lr=0.05):
        super().__init__()
        self.lr = lr
        self.b = np.zeros(1, )
        self.W = np.random.randn(n_in, 1)

    def forward(self, x):
        self.h = linear_forward(x, self.W, self.b)
        y = sigmoid_forward(self.h)
        return y

    def update_params(self, x, nll_grad):
        # compute gradients
        grad_h = sigmoid_grad_input(self.h, nll_grad)
        grad_W = linear_grad_W(x, grad_h, self.W, self.b)
        grad_b = linear_grad_b(x, grad_h, self.W, self.b)
        # update params
        self.W = self.W - self.lr * grad_W
        self.b = self.b - self.lr * grad_b
```

## 1.5 Simple Experiment

```python
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
# Generate some data
def generate_2_circles(N=100):
    phi = np.linspace(0.0, np.pi * 2, 100)
    X1 = 1.1 * np.array([np.sin(phi), np.cos(phi)])
    X2 = 3.0 * np.array([np.sin(phi), np.cos(phi)])
    Y = np.concatenate([np.ones(N), np.zeros(N)]).reshape((-1, 1))
    X = np.hstack([X1,X2]).T
    return X, Y

def generate_2_gaussians(N=100):
    phi = np.linspace(0.0, np.pi * 2, 100)
    X1 = np.random.normal(loc=[1, 2], scale=[2.5, 0.9], size=(N, 2))
    X1 = X1 @ np.array([[0.7, -0.7], [0.7, 0.7]])
    X2 = np.random.normal(loc=[-2, 0], scale=[1, 1.5], size=(N, 2))
    X2 = X2 @ np.array([[0.7, 0.7], [-0.7, 0.7]])
    Y = np.concatenate([np.ones(N), np.zeros(N)]).reshape((-1, 1))
    X = np.vstack([X1,X2])
    return X, Y

def split(X, Y, train_ratio=0.7):
    size = len(X)
    train_size = int(size * train_ratio)
    indices = np.arange(size)
    np.random.shuffle(indices)
    train_indices = indices[:train_size]
    test_indices = indices[train_size:]
    return X[train_indices], Y[train_indices], X[test_indices], Y[test_indices]

f, (ax1, ax2) = plt.subplots(1, 2, figsize=(8, 4))

X, Y = generate_2_circles()
ax1.scatter(X[:,0], X[:,1], c=Y.ravel(), edgecolors= 'none')
ax1.set_aspect('equal')

X, Y = generate_2_gaussians()
ax2.scatter(X[:,0], X[:,1], c=Y.ravel(), edgecolors= 'none')
ax2.set_aspect('equal')

```

```python
X_train, Y_train, X_test, Y_test = split(*generate_2_gaussians(), 0.7)
```

```python
# let's train our model
model = LogsticRegressionGD(2, 0.05)

for step in range(30):
    Y_pred = model.forward(X_train)

    loss_value = nll_forward(Y_pred, Y_train)
    accuracy = ((Y_pred > 0.5) == Y_train).mean()
    print('Step: {} \t Loss: {:.3f} \t Acc: {:.1f}%'.format(step, loss_value, accuracy * 100))

    loss_grad = nll_grad_input(Y_pred, Y_train)
    model.update_params(X_train, loss_grad)

print('\n\nTesting...')
Y_test_pred = model.forward(X_test)
test_accuracy = ((Y_test_pred > 0.5) == Y_test).mean()
print('Acc: {:.1f}%'.format(test_accuracy * 100))
```

    Step: 0 	 Loss: 0.448 	 Acc: 97.9%
    [[0.49066807]
     [0.23933572]
     [0.31303132]
     [0.77384888]
     [0.2781836 ]
     [0.41143628]
     [0.33244299]
     [0.62983742]
     [0.60581536]
     [0.38349525]
     [0.43309194]
     [0.26344452]
     [0.32404159]
     [0.80306921]
     [0.65121762]
     [0.29365382]
     [0.60871377]
     [0.62820368]
     [0.73383065]
     [0.2707865 ]
     [0.68791402]
     [0.63868949]
     [0.46825449]
     [0.31093282]
     [0.72672596]
     [0.42012007]
     [0.41455949]
     [0.26997816]
     [0.28635386]
     [0.65085762]
     [0.40523786]
     [0.51597134]
     [0.54955422]
     [0.66445317]
     [0.70678803]
     [0.67600008]
     [0.30195149]
     [0.70675137]
     [0.64256797]
     [0.60062027]
     [0.34279183]
     [0.54785406]
     [0.80521427]
     [0.29789714]
     [0.64191666]
     [0.43010913]
     [0.78994595]
     [0.76424215]
     [0.61913784]
     [0.3870707 ]
     [0.40438323]
     [0.30558038]
     [0.62122767]
     [0.67544465]
     [0.28415843]
     [0.39369168]
     [0.31709261]
     [0.48071186]
     [0.56358819]
     [0.37159886]
     [0.39601401]
     [0.30642377]
     [0.64847967]
     [0.4260625 ]
     [0.37268679]
     [0.56377461]
     [0.3645235 ]
     [0.65204463]
     [0.69732066]
     [0.34387387]
     [0.33375388]
     [0.7500607 ]
     [0.36117618]
     [0.64586958]
     [0.31936962]
     [0.40631759]
     [0.41640113]
     [0.69762526]
     [0.60172541]
     [0.39124769]
     [0.3839339 ]
     [0.54505012]
     [0.76475564]
     [0.65115054]
     [0.58085814]
     [0.81102854]
     [0.63179876]
     [0.45813667]
     [0.55268246]
     [0.32555907]
     [0.341223  ]
     [0.35407439]
     [0.58113755]
     [0.58680162]
     [0.50014032]
     [0.69301699]
     [0.38081679]
     [0.23954391]
     [0.77606518]
     [0.66352124]
     [0.70666013]
     [0.30058113]
     [0.43974587]
     [0.31256324]
     [0.61966679]
     [0.28804956]
     [0.67910768]
     [0.50172379]
     [0.36331762]
     [0.69235647]
     [0.24609659]
     [0.6868359 ]
     [0.43153287]
     [0.63237805]
     [0.40862957]
     [0.36513629]
     [0.39046573]
     [0.51142975]
     [0.44044016]
     [0.34775751]
     [0.48137275]
     [0.60699959]
     [0.362952  ]
     [0.68249001]
     [0.38530357]
     [0.78678589]
     [0.35918823]
     [0.66888291]
     [0.81561762]
     [0.33861878]
     [0.62155873]
     [0.49637315]
     [0.61013142]
     [0.59709279]
     [0.31867245]
     [0.26891625]
     [0.64084789]
     [0.6781699 ]
     [0.33099869]
     [0.61633958]]
    140
    Step: 1 	 Loss: 0.425 	 Acc: 98.6%
    [[0.4779815 ]
     [0.21647557]
     [0.29513135]
     [0.78987167]
     [0.2551167 ]
     [0.39468391]
     [0.31479502]
     [0.6518715 ]
     [0.61428201]
     [0.36044788]
     [0.42042809]
     [0.24345068]
     [0.30004582]
     [0.81638869]
     [0.67126435]
     [0.26881581]
     [0.62505533]
     [0.64762679]
     [0.74945184]
     [0.25078887]
     [0.70327753]
     [0.65570756]
     [0.46134792]
     [0.30169039]
     [0.74053867]
     [0.40190114]
     [0.40577735]
     [0.25298226]
     [0.25870189]
     [0.66626147]
     [0.38888581]
     [0.53005642]
     [0.56002738]
     [0.67553242]
     [0.71993723]
     [0.69488426]
     [0.28254886]
     [0.72952283]
     [0.65551631]
     [0.60949117]
     [0.32249104]
     [0.56805048]
     [0.81663229]
     [0.27421459]
     [0.66207663]
     [0.4165485 ]
     [0.80166358]
     [0.76890533]
     [0.62949181]
     [0.38039657]
     [0.38496741]
     [0.28549954]
     [0.64347778]
     [0.69926399]
     [0.26498271]
     [0.39316544]
     [0.29957875]
     [0.47377668]
     [0.57695609]
     [0.3436882 ]
     [0.3767113 ]
     [0.2850229 ]
     [0.6553023 ]
     [0.41200402]
     [0.35395013]
     [0.57808724]
     [0.34886281]
     [0.66299759]
     [0.70597686]
     [0.32531617]
     [0.31336207]
     [0.76758909]
     [0.34234559]
     [0.66352373]
     [0.30435118]
     [0.39810004]
     [0.40212586]
     [0.71959124]
     [0.62063929]
     [0.36662741]
     [0.37325859]
     [0.55534681]
     [0.78530336]
     [0.6660256 ]
     [0.5934365 ]
     [0.8219208 ]
     [0.64451413]
     [0.44133306]
     [0.5598844 ]
     [0.30495609]
     [0.32340045]
     [0.33178124]
     [0.61447477]
     [0.60240445]
     [0.50317653]
     [0.71348324]
     [0.38207997]
     [0.22137943]
     [0.78402476]
     [0.67080134]
     [0.71900482]
     [0.28168833]
     [0.4230313 ]
     [0.29443069]
     [0.63618274]
     [0.26612876]
     [0.68924547]
     [0.48737956]
     [0.35377446]
     [0.70736287]
     [0.22700516]
     [0.71087228]
     [0.41657553]
     [0.6474898 ]
     [0.3948196 ]
     [0.36106561]
     [0.37894349]
     [0.519129  ]
     [0.43052504]
     [0.33019344]
     [0.47597562]
     [0.61470645]
     [0.33589811]
     [0.69312824]
     [0.37705067]
     [0.79922715]
     [0.3362023 ]
     [0.68922707]
     [0.83191688]
     [0.3248637 ]
     [0.63390709]
     [0.47924687]
     [0.61805184]
     [0.61349478]
     [0.29665026]
     [0.24879269]
     [0.64011221]
     [0.69339812]
     [0.31630514]
     [0.62860592]]
    140
    Step: 2 	 Loss: 0.404 	 Acc: 98.6%
    [[0.46596778]
     [0.1963229 ]
     [0.27874536]
     [0.80425305]
     [0.2344601 ]
     [0.3790289 ]
     [0.29856135]
     [0.67218969]
     [0.62223852]
     [0.33917257]
     [0.40852184]
     [0.22549379]
     [0.27827264]
     [0.82832979]
     [0.68971391]
     [0.24654556]
     [0.64029585]
     [0.66560847]
     [0.76366571]
     [0.23278368]
     [0.71741704]
     [0.67148139]
     [0.45481832]
     [0.29311206]
     [0.75318416]
     [0.38487214]
     [0.39752249]
     [0.23756603]
     [0.23415492]
     [0.68053928]
     [0.37361489]
     [0.54339414]
     [0.56991952]
     [0.68584118]
     [0.73204351]
     [0.71219767]
     [0.26489034]
     [0.75004985]
     [0.66757441]
     [0.61783006]
     [0.30385461]
     [0.58702151]
     [0.82693603]
     [0.25289033]
     [0.68066625]
     [0.40381082]
     [0.81227768]
     [0.77322024]
     [0.63919221]
     [0.37413722]
     [0.36689097]
     [0.26723157]
     [0.66402986]
     [0.72090038]
     [0.24760956]
     [0.39270301]
     [0.28351951]
     [0.46721201]
     [0.58953975]
     [0.31817788]
     [0.35876654]
     [0.26560324]
     [0.66169125]
     [0.39881022]
     [0.33659888]
     [0.59155137]
     [0.33432611]
     [0.67320849]
     [0.71402049]
     [0.3082284 ]
     [0.29468572]
     [0.78337869]
     [0.32495006]
     [0.67984954]
     [0.29051299]
     [0.39038098]
     [0.38874886]
     [0.73948723]
     [0.63825161]
     [0.34390361]
     [0.36327057]
     [0.56507802]
     [0.80353412]
     [0.67982295]
     [0.60525653]
     [0.83175068]
     [0.65637769]
     [0.42551478]
     [0.5666944 ]
     [0.28613234]
     [0.30698009]
     [0.3113249 ]
     [0.64516534]
     [0.61701805]
     [0.50606792]
     [0.73210634]
     [0.38332184]
     [0.20511867]
     [0.79133907]
     [0.67760501]
     [0.73038839]
     [0.26448172]
     [0.40734058]
     [0.277841  ]
     [0.651554  ]
     [0.24637394]
     [0.6986704 ]
     [0.47378104]
     [0.34486022]
     [0.72117014]
     [0.20992154]
     [0.73261919]
     [0.40253558]
     [0.66154752]
     [0.38188832]
     [0.3572614 ]
     [0.36816131]
     [0.52643808]
     [0.42118296]
     [0.31398301]
     [0.47086752]
     [0.62195156]
     [0.31120012]
     [0.70300642]
     [0.3693159 ]
     [0.81048009]
     [0.31510594]
     [0.70786201]
     [0.84626196]
     [0.31212047]
     [0.64544976]
     [0.46303296]
     [0.62549421]
     [0.62882029]
     [0.27662156]
     [0.23069152]
     [0.63938134]
     [0.70744429]
     [0.30272933]
     [0.64008118]]
    140
    Step: 3 	 Loss: 0.384 	 Acc: 98.6%
    [[0.45458638]
     [0.17854325]
     [0.26372842]
     [0.81719231]
     [0.21595724]
     [0.3643934 ]
     [0.28361393]
     [0.69091661]
     [0.6297301 ]
     [0.31954978]
     [0.39731694]
     [0.20934703]
     [0.25852607]
     [0.83906981]
     [0.70669752]
     [0.22658601]
     [0.6545186 ]
     [0.68225825]
     [0.77662509]
     [0.2165544 ]
     [0.73045074]
     [0.68611345]
     [0.44863612]
     [0.28512882]
     [0.76478715]
     [0.3689553 ]
     [0.38974965]
     [0.22355797]
     [0.21239085]
     [0.69378953]
     [0.35934579]
     [0.55602665]
     [0.57927317]
     [0.69545307]
     [0.74321368]
     [0.72808288]
     [0.24880455]
     [0.76855583]
     [0.67882121]
     [0.62568314]
     [0.2867413 ]
     [0.60482662]
     [0.83626699]
     [0.2336921 ]
     [0.69780981]
     [0.39183599]
     [0.82192379]
     [0.77723074]
     [0.64829644]
     [0.36825294]
     [0.35006355]
     [0.25060094]
     [0.68300163]
     [0.74054399]
     [0.2318514 ]
     [0.39229313]
     [0.2687769 ]
     [0.46098988]
     [0.60139436]
     [0.29491334]
     [0.34208438]
     [0.24797465]
     [0.6676899 ]
     [0.38641804]
     [0.32052499]
     [0.60422501]
     [0.32081824]
     [0.68274659]
     [0.72151577]
     [0.29248352]
     [0.27757455]
     [0.79762806]
     [0.30887311]
     [0.69495786]
     [0.27774151]
     [0.38311627]
     [0.37620299]
     [0.75751248]
     [0.65465219]
     [0.32295929]
     [0.35390918]
     [0.5742848 ]
     [0.81973118]
     [0.6926376 ]
     [0.61637581]
     [0.84065436]
     [0.66746329]
     [0.41062488]
     [0.57314391]
     [0.26892761]
     [0.29183858]
     [0.29256046]
     [0.67328279]
     [0.63071308]
     [0.50882478]
     [0.74906173]
     [0.38453689]
     [0.19053467]
     [0.7980852 ]
     [0.68398069]
     [0.74090971]
     [0.24879452]
     [0.39260888]
     [0.26264647]
     [0.66587003]
     [0.22856312]
     [0.70745342]
     [0.4608892 ]
     [0.33651574]
     [0.73389567]
     [0.19461002]
     [0.75228596]
     [0.38934917]
     [0.67463914]
     [0.36976767]
     [0.35369439]
     [0.3580559 ]
     [0.53338422]
     [0.41236907]
     [0.29900917]
     [0.46602583]
     [0.62877665]
     [0.28869452]
     [0.71220022]
     [0.36205136]
     [0.82069021]
     [0.2957551 ]
     [0.72493712]
     [0.8589247 ]
     [0.30029492]
     [0.65625529]
     [0.44769063]
     [0.63250184]
     [0.64314737]
     [0.2584042 ]
     [0.21439153]
     [0.63866234]
     [0.72041983]
     [0.29016645]
     [0.65083191]]
    140
    Step: 4 	 Loss: 0.367 	 Acc: 98.6%
    [[0.44379688]
     [0.16283404]
     [0.24994664]
     [0.8288642 ]
     [0.19936864]
     [0.35070146]
     [0.26983341]
     [0.70817761]
     [0.63679749]
     [0.30145588]
     [0.38676003]
     [0.19480432]
     [0.24061443]
     [0.84876101]
     [0.72234172]
     [0.2086911 ]
     [0.66780338]
     [0.69768307]
     [0.78846633]
     [0.2019034 ]
     [0.74248609]
     [0.6997006 ]
     [0.44277386]
     [0.27768013]
     [0.77545829]
     [0.35407261]
     [0.38241761]
     [0.21080398]
     [0.19309753]
     [0.70610344]
     [0.34600185]
     [0.56799669]
     [0.58812845]
     [0.70443395]
     [0.75354291]
     [0.74267364]
     [0.23413258]
     [0.78525177]
     [0.68932886]
     [0.63309226]
     [0.27101552]
     [0.62153118]
     [0.84474603]
     [0.21639864]
     [0.71362819]
     [0.38056662]
     [0.83071839]
     [0.7809733 ]
     [0.65685658]
     [0.36270852]
     [0.33439444]
     [0.23544378]
     [0.70051302]
     [0.75838104]
     [0.21753646]
     [0.3919269 ]
     [0.25522351]
     [0.45508406]
     [0.61257287]
     [0.27372437]
     [0.32657007]
     [0.23195781]
     [0.67333626]
     [0.37476713]
     [0.30562394]
     [0.61616439]
     [0.30825022]
     [0.69167407]
     [0.7285187 ]
     [0.27796137]
     [0.26188514]
     [0.81051446]
     [0.29400299]
     [0.70895361]
     [0.26593347]
     [0.37626599]
     [0.36442425]
     [0.77385545]
     [0.66993025]
     [0.30366865]
     [0.34511924]
     [0.58300591]
     [0.83414814]
     [0.70455721]
     [0.62684857]
     [0.84874771]
     [0.67783869]
     [0.39660501]
     [0.57926182]
     [0.25318966]
     [0.27786019]
     [0.27534449]
     [0.69895234]
     [0.64355779]
     [0.51145686]
     [0.76451411]
     [0.38572176]
     [0.17742572]
     [0.80432871]
     [0.68997066]
     [0.75065652]
     [0.23447278]
     [0.37877131]
     [0.24871052]
     [0.67921606]
     [0.212489  ]
     [0.71565732]
     [0.4486635 ]
     [0.32868804]
     [0.74564575]
     [0.18085953]
     [0.77007572]
     [0.37695426]
     [0.68684673]
     [0.35839337]
     [0.35033945]
     [0.34856921]
     [0.53999332]
     [0.40404153]
     [0.28516185]
     [0.46142953]
     [0.6352191 ]
     [0.26820675]
     [0.72077671]
     [0.35521412]
     [0.82998305]
     [0.27800529]
     [0.74059494]
     [0.87013821]
     [0.28930136]
     [0.66638673]
     [0.43317497]
     [0.63911349]
     [0.65655151]
     [0.2418239 ]
     [0.19969091]
     [0.63795977]
     [0.73242642]
     [0.27852087]
     [0.66091944]]
    140
    Step: 5 	 Loss: 0.351 	 Acc: 98.6%
    [[0.43355999]
     [0.14892677]
     [0.23727795]
     [0.83942116]
     [0.18447575]
     [0.33788056]
     [0.25711004]
     [0.72409454]
     [0.64347728]
     [0.28476845]
     [0.37680113]
     [0.18168109]
     [0.22435614]
     [0.85753397]
     [0.73676582]
     [0.1926323 ]
     [0.68022538]
     [0.71198482]
     [0.7993103 ]
     [0.18865293]
     [0.75362008]
     [0.71233307]
     [0.43720626]
     [0.27071289]
     [0.78529535]
     [0.3401481 ]
     [0.37548899]
     [0.19916684]
     [0.17598382]
     [0.71756468]
     [0.33351039]
     [0.57934636]
     [0.59652282]
     [0.71284259]
     [0.76311567]
     [0.75609336]
     [0.2207294 ]
     [0.80033233]
     [0.69916285]
     [0.64009519]
     [0.25654996]
     [0.63720295]
     [0.85247645]
     [0.20080472]
     [0.72823616]
     [0.36994868]
     [0.83876167]
     [0.78447846]
     [0.66491969]
     [0.3574728 ]
     [0.31979533]
     [0.22160977]
     [0.71668174]
     [0.77458822]
     [0.20450953]
     [0.39159726]
     [0.24274318]
     [0.44947024]
     [0.62312536]
     [0.25443658]
     [0.31213206]
     [0.21738713]
     [0.67866358]
     [0.36380061]
     [0.2917968 ]
     [0.62742308]
     [0.29653993]
     [0.70004661]
     [0.73507819]
     [0.26455027]
     [0.24748351]
     [0.82219475]
     [0.28023457]
     [0.72193451]
     [0.2549957 ]
     [0.3697942 ]
     [0.35335256]
     [0.78869047]
     [0.68417206]
     [0.28590397]
     [0.33685085]
     [0.59127758]
     [0.84700895]
     [0.7156619 ]
     [0.63672529]
     [0.85612952]
     [0.68756556]
     [0.38339739]
     [0.58507454]
     [0.23877681]
     [0.26493792]
     [0.25953944]
     [0.72233035]
     [0.65561698]
     [0.5139732 ]
     [0.77861518]
     [0.38687466]
     [0.16561426]
     [0.81012558]
     [0.69561189]
     [0.75970645]
     [0.22137638]
     [0.36576464]
     [0.23590849]
     [0.69167202]
     [0.19796194]
     [0.72333761]
     [0.43706334]
     [0.32132986]
     [0.75651598]
     [0.16848328]
     [0.78617969]
     [0.3652918 ]
     [0.69824607]
     [0.34770553]
     [0.34717501]
     [0.33964832]
     [0.54628968]
     [0.39616176]
     [0.27233912]
     [0.45705927]
     [0.64131232]
     [0.24956133]
     [0.72879527]
     [0.34876586]
     [0.83846679]
     [0.2617166 ]
     [0.75496861]
     [0.88010136]
     [0.27906236]
     [0.67590159]
     [0.41943927]
     [0.64536382]
     [0.66910437]
     [0.22671797]
     [0.1864083 ]
     [0.63727643]
     [0.74355615]
     [0.26770606]
     [0.67039979]]
    140
    Step: 6 	 Loss: 0.337 	 Acc: 98.6%
    [[0.42383817]
     [0.13658618]
     [0.22561207]
     [0.84899566]
     [0.17108223]
     [0.32586248]
     [0.24534392]
     [0.7387831 ]
     [0.64980231]
     [0.26936966]
     [0.3673939 ]
     [0.16981361]
     [0.20958284]
     [0.86550067]
     [0.75008056]
     [0.17820157]
     [0.6918546 ]
     [0.72525901]
     [0.80926373]
     [0.17664471]
     [0.76393971]
     [0.72409396]
     [0.43191021]
     [0.26418053]
     [0.79438455]
     [0.32710922]
     [0.36893004]
     [0.18852504]
     [0.16078501]
     [0.72824936]
     [0.32180346]
     [0.59011631]
     [0.60449093]
     [0.72073134]
     [0.77200679]
     [0.76845469]
     [0.20846404]
     [0.81397419]
     [0.70838236]
     [0.64672592]
     [0.24322691]
     [0.65190972]
     [0.85954675]
     [0.18672346]
     [0.74174086]
     [0.3599319 ]
     [0.8461399 ]
     [0.78777194]
     [0.67252811]
     [0.35251816]
     [0.30618205]
     [0.20896268]
     [0.73162035]
     [0.78932955]
     [0.19263173]
     [0.39129861]
     [0.23123089]
     [0.444126  ]
     [0.63309862]
     [0.23687899]
     [0.29868359]
     [0.20411193]
     [0.68370108]
     [0.35346545]
     [0.27895134]
     [0.63805145]
     [0.28561222]
     [0.70791399]
     [0.74123701]
     [0.25214765]
     [0.23424633]
     [0.8328069 ]
     [0.26747039]
     [0.73399055]
     [0.24484461]
     [0.36366863]
     [0.34293209]
     [0.80217631]
     [0.6974595 ]
     [0.26954022]
     [0.32905902]
     [0.59913339]
     [0.85850951]
     [0.72602442]
     [0.64605256]
     [0.86288421]
     [0.69669977]
     [0.37094607]
     [0.59060608]
     [0.22555911]
     [0.25297395]
     [0.24501614]
     [0.7435888 ]
     [0.66695132]
     [0.51638223]
     [0.79150292]
     [0.38799491]
     [0.15494496]
     [0.81552379]
     [0.70093682]
     [0.76812812]
     [0.20937902]
     [0.35352846]
     [0.22412771]
     [0.70331198]
     [0.18481094]
     [0.73054335]
     [0.42604904]
     [0.31439906]
     [0.76659192]
     [0.15731719]
     [0.8007742 ]
     [0.3543063 ]
     [0.70890651]
     [0.33764884]
     [0.34418242]
     [0.33124523]
     [0.55229593]
     [0.38869438]
     [0.26044763]
     [0.45289732]
     [0.64708618]
     [0.23258845]
     [0.73630844]
     [0.34267238]
     [0.84623476]
     [0.24675678]
     [0.7681807 ]
     [0.88898336]
     [0.2695083 ]
     [0.68485208]
     [0.4064366 ]
     [0.65128379]
     [0.68087307]
     [0.21293706]
     [0.1743823 ]
     [0.63661381]
     [0.75389188]
     [0.2576441 ]
     [0.67932391]]
    140
    Step: 7 	 Loss: 0.324 	 Acc: 98.6%
    [[0.41459595]
     [0.12560787]
     [0.21484995]
     [0.85770269]
     [0.15901352]
     [0.31458375]
     [0.23444472]
     [0.75235144]
     [0.655802  ]
     [0.25514834]
     [0.35849567]
     [0.15905755]
     [0.1961408 ]
     [0.87275727]
     [0.76238763]
     [0.16521214]
     [0.70275562]
     [0.73759408]
     [0.81842065]
     [0.16573874]
     [0.77352283]
     [0.73505925]
     [0.4268646 ]
     [0.25804208]
     [0.80280184]
     [0.31488766]
     [0.36271035]
     [0.17877138]
     [0.14726453]
     [0.73822641]
     [0.3108181 ]
     [0.60034525]
     [0.61206466]
     [0.72814679]
     [0.7802825 ]
     [0.77985957]
     [0.19721908]
     [0.82633594]
     [0.7170407 ]
     [0.65301502]
     [0.23093871]
     [0.66571759]
     [0.86603289]
     [0.17398679]
     [0.75424112]
     [0.35046981]
     [0.85292748]
     [0.79087559]
     [0.67971995]
     [0.34782011]
     [0.2934755 ]
     [0.19738006]
     [0.74543469]
     [0.80275509]
     [0.18177954]
     [0.39102647]
     [0.22059225]
     [0.43903083]
     [0.64253605]
     [0.22088883]
     [0.28614355]
     [0.19199642]
     [0.68847447]
     [0.34371267]
     [0.26700248]
     [0.64809639]
     [0.27539876]
     [0.71532064]
     [0.74703266]
     [0.24066011]
     [0.22206128]
     [0.84247188]
     [0.25562095]
     [0.74520392]
     [0.23540542]
     [0.35786031]
     [0.33311139]
     [0.81445606]
     [0.70986921]
     [0.25445801]
     [0.32170328]
     [0.60660424]
     [0.86882018]
     [0.73571053]
     [0.65487317]
     [0.86908411]
     [0.70529181]
     [0.35919774]
     [0.59587825]
     [0.21341857]
     [0.24187947]
     [0.23165511]
     [0.76290423]
     [0.67761717]
     [0.51869169]
     [0.80330171]
     [0.38908259]
     [0.14528244]
     [0.82056469]
     [0.70597393]
     [0.77598215]
     [0.19836768]
     [0.34200578]
     [0.21326695]
     [0.71420406]
     [0.17288321]
     [0.73731794]
     [0.4155825 ]
     [0.30785809]
     [0.77594991]
     [0.14721784]
     [0.81401952]
     [0.34394612]
     [0.71889122]
     [0.32817257]
     [0.34134552]
     [0.32331658]
     [0.55803296]
     [0.38160706]
     [0.24940244]
     [0.44892752]
     [0.65256735]
     [0.21712792]
     [0.74336273]
     [0.33690316]
     [0.85336759]
     [0.23300289]
     [0.78034293]
     [0.89692802]
     [0.26057675]
     [0.69328543]
     [0.39412102]
     [0.65690108]
     [0.69191986]
     [0.20034561]
     [0.16347029]
     [0.63597249]
     [0.76350796]
     [0.24826515]
     [0.68773795]]
    140
    Step: 8 	 Loss: 0.312 	 Acc: 98.6%
    [[0.40580017]
     [0.11581519]
     [0.20490299]
     [0.86564199]
     [0.14811553]
     [0.30398576]
     [0.22433112]
     [0.76489941]
     [0.66150274]
     [0.24200104]
     [0.35006729]
     [0.14928622]
     [0.18389106]
     [0.87938648]
     [0.77377967]
     [0.15349794]
     [0.71298769]
     [0.7490713 ]
     [0.82686383]
     [0.1558117 ]
     [0.7824389 ]
     [0.74529803]
     [0.42205025]
     [0.25226145]
     [0.81061417]
     [0.30341979]
     [0.35680251]
     [0.16981146]
     [0.13521346]
     [0.74755805]
     [0.30049639]
     [0.61006965]
     [0.61927326]
     [0.73513048]
     [0.78800143]
     [0.79039986]
     [0.1868898 ]
     [0.83755884]
     [0.72518586]
     [0.65898997]
     [0.21958765]
     [0.67868993]
     [0.87200028]
     [0.16244486]
     [0.76582739]
     [0.34151976]
     [0.85918878]
     [0.79380805]
     [0.68652945]
     [0.34335689]
     [0.28160216]
     [0.18675244]
     [0.75822296]
     [0.81500066]
     [0.17184357]
     [0.39077723]
     [0.21074276]
     [0.43416598]
     [0.65147764]
     [0.20631423]
     [0.27443685]
     [0.18091911]
     [0.69300646]
     [0.33449721]
     [0.25587233]
     [0.65760133]
     [0.26583761]
     [0.72230626]
     [0.75249814]
     [0.23000308]
     [0.21082673]
     [0.85129552]
     [0.24460468]
     [0.75564933]
     [0.2266114 ]
     [0.35234326]
     [0.32384328]
     [0.82565774]
     [0.72147229]
     [0.24054529]
     [0.31474725]
     [0.61371852]
     [0.87808859]
     [0.74477952]
     [0.66322623]
     [0.87479141]
     [0.71338717]
     [0.34810219]
     [0.6009108 ]
     [0.20224881]
     [0.23157426]
     [0.21934698]
     [0.78045011]
     [0.68766646]
     [0.52090867]
     [0.81412295]
     [0.39013834]
     [0.13650895]
     [0.82528402]
     [0.71074834]
     [0.78332215]
     [0.1882416 ]
     [0.3311434 ]
     [0.20323565]
     [0.72441053]
     [0.16204319]
     [0.74369986]
     [0.40562757]
     [0.30167347]
     [0.78465798]
     [0.13806023]
     [0.82605988]
     [0.33416354]
     [0.72825749]
     [0.31923038]
     [0.33865023]
     [0.31582322]
     [0.56352003]
     [0.37487041]
     [0.23912672]
     [0.44513518]
     [0.65777967]
     [0.20303131]
     [0.74999937]
     [0.33143096]
     [0.85993505]
     [0.22034196]
     [0.79155637]
     [0.90405767]
     [0.25221181]
     [0.70124433]
     [0.38244824]
     [0.66224042]
     [0.7023021 ]
     [0.18882162]
     [0.15354674]
     [0.63535245]
     [0.7724709 ]
     [0.23950666]
     [0.69568365]]
    140
    Step: 9 	 Loss: 0.301 	 Acc: 98.6%
    [[0.39741995]
     [0.10705597]
     [0.19569215]
     [0.87290018]
     [0.13825289]
     [0.29401464]
     [0.21493014]
     [0.77651853]
     [0.6669282 ]
     [0.22983243]
     [0.34207301]
     [0.14038862]
     [0.17270898]
     [0.88545962]
     [0.78434063]
     [0.14291222]
     [0.72260495]
     [0.75976485]
     [0.83466613]
     [0.14675527]
     [0.79074987]
     [0.75487294]
     [0.41744975]
     [0.24680678]
     [0.81788055]
     [0.29264671]
     [0.35118184]
     [0.16156222]
     [0.12444871]
     [0.75630033]
     [0.29078527]
     [0.61932363]
     [0.62614345]
     [0.74171943]
     [0.79521553]
     [0.80015794]
     [0.17738304]
     [0.84776805]
     [0.73286094]
     [0.66467549]
     [0.20908549]
     [0.69088674]
     [0.87750544]
     [0.15196487]
     [0.77658188]
     [0.33304265]
     [0.86497966]
     [0.79658539]
     [0.6929874 ]
     [0.33910915]
     [0.2704942 ]
     [0.17698238]
     [0.7700755 ]
     [0.82618829]
     [0.16272714]
     [0.39054799]
     [0.20160694]
     [0.42951437]
     [0.65996015]
     [0.19301549]
     [0.26349443]
     [0.17077178]
     [0.69731713]
     [0.32577781]
     [0.24548993]
     [0.66660623]
     [0.25687284]
     [0.72890631]
     [0.75766254]
     [0.22010032]
     [0.20045124]
     [0.85937036]
     [0.23434754]
     [0.76539438]
     [0.218403  ]
     [0.34709413]
     [0.31508464]
     [0.83589531]
     [0.73233423]
     [0.2276982 ]
     [0.30815825]
     [0.62050218]
     [0.88644239]
     [0.75328476]
     [0.67114741]
     [0.8800598 ]
     [0.72102687]
     [0.3376125 ]
     [0.60572164]
     [0.19195436]
     [0.22198611]
     [0.20799237]
     [0.79639214]
     [0.69714689]
     [0.52303969]
     [0.82406603]
     [0.39116312]
     [0.12852213]
     [0.82971289]
     [0.71528227]
     [0.79019559]
     [0.17891123]
     [0.32089199]
     [0.19395302]
     [0.73398811]
     [0.15217109]
     [0.74972324]
     [0.39615025]
     [0.29581533]
     [0.79277669]
     [0.12973553]
     [0.83702418]
     [0.32491466]
     [0.73705715]
     [0.31078006]
     [0.33608422]
     [0.30872986]
     [0.5687748 ]
     [0.3684577 ]
     [0.22955119]
     [0.44150693]
     [0.66274451]
     [0.19016278]
     [0.75625496]
     [0.32623144]
     [0.86599769]
     [0.20867107]
     [0.80191205]
     [0.91047644]
     [0.24436346]
     [0.70876729]
     [0.37137615]
     [0.66732397]
     [0.71207237]
     [0.17825587]
     [0.14450148]
     [0.63475321]
     [0.78084019]
     [0.23131271]
     [0.70319874]]
    140
    Step: 10 	 Loss: 0.291 	 Acc: 98.6%
    [[0.38942672]
     [0.09919932]
     [0.18714697]
     [0.87955249]
     [0.12930698]
     [0.28462112]
     [0.20617635]
     [0.78729212]
     [0.67209969]
     [0.21855534]
     [0.3344802 ]
     [0.13226767]
     [0.1624833 ]
     [0.89103833]
     [0.79414631]
     [0.13332591]
     [0.7316567 ]
     [0.76974221]
     [0.84189171]
     [0.13847439]
     [0.79851098]
     [0.76384062]
     [0.4130473 ]
     [0.24164983]
     [0.82465312]
     [0.28251423]
     [0.34582611]
     [0.15395059]
     [0.11481072]
     [0.76450369]
     [0.2816363 ]
     [0.62813891]
     [0.63269966]
     [0.74794672]
     [0.80197086]
     [0.8092076 ]
     [0.16861613]
     [0.85707414]
     [0.74010473]
     [0.67009384]
     [0.19935281]
     [0.7023643 ]
     [0.88259732]
     [0.14242955]
     [0.78657911]
     [0.32500281]
     [0.87034869]
     [0.7992215 ]
     [0.69912149]
     [0.33505963]
     [0.26008936]
     [0.16798328]
     [0.78107489]
     [0.83642712]
     [0.15434497]
     [0.39033637]
     [0.19311746]
     [0.42506045]
     [0.66801725]
     [0.18086542]
     [0.25325311]
     [0.16145834]
     [0.70142436]
     [0.31751681]
     [0.23579089]
     [0.67514782]
     [0.24845397]
     [0.73515256]
     [0.76255163]
     [0.21088323]
     [0.19085276]
     [0.86677734]
     [0.22478258]
     [0.77450011]
     [0.21072715]
     [0.34209199]
     [0.30679615]
     [0.84526988]
     [0.74251506]
     [0.21582128]
     [0.30190689]
     [0.62697895]
     [0.89399192]
     [0.76127424]
     [0.67866923]
     [0.88493572]
     [0.72824783]
     [0.32768511]
     [0.61032699]
     [0.18244983]
     [0.21305015]
     [0.19750145]
     [0.81088536]
     [0.70610212]
     [0.5250907 ]
     [0.83321941]
     [0.39215813]
     [0.12123303]
     [0.83387843]
     [0.71959542]
     [0.79664455]
     [0.17029709]
     [0.31120598]
     [0.18534707]
     [0.74298833]
     [0.14316135]
     [0.75541845]
     [0.38711873]
     [0.29025694]
     [0.80036001]
     [0.12214908]
     [0.84702707]
     [0.31615927]
     [0.74533706]
     [0.30278324]
     [0.33363669]
     [0.30200466]
     [0.57381344]
     [0.36234465]
     [0.22061349]
     [0.43803067]
     [0.66748102]
     [0.17839917]
     [0.76216206]
     [0.32128282]
     [0.87160822]
     [0.19789697]
     [0.81149163]
     [0.91627319]
     [0.23698694]
     [0.71588904]
     [0.36086501]
     [0.67217159]
     [0.72127875]
     [0.16855097]
     [0.13623792]
     [0.63417404]
     [0.78866899]
     [0.22363331]
     [0.71031726]]
    140
    Step: 11 	 Loss: 0.281 	 Acc: 98.6%
    [[0.38179411]
     [0.09213268]
     [0.17920471]
     [0.88566447]
     [0.12117406]
     [0.27576021]
     [0.19801121]
     [0.79729576]
     [0.67703634]
     [0.20809041]
     [0.32725915]
     [0.12483842]
     [0.15311503]
     [0.89617606]
     [0.80326499]
     [0.12462574]
     [0.74018778]
     [0.77906459]
     [0.84859715]
     [0.13088568]
     [0.80577153]
     [0.77225222]
     [0.40882855]
     [0.23676557]
     [0.83097796]
     [0.27297263]
     [0.34071526]
     [0.14691223]
     [0.10616086]
     [0.77221353]
     [0.27300538]
     [0.63654495]
     [0.63896414]
     [0.75384191]
     [0.80830836]
     [0.81761482]
     [0.16051574]
     [0.86557455]
     [0.7469521 ]
     [0.67526506]
     [0.19031833]
     [0.71317507]
     [0.88731854]
     [0.13373559]
     [0.79588635]
     [0.31736768]
     [0.87533829]
     [0.80172851]
     [0.70495666]
     [0.33119287]
     [0.2503307 ]
     [0.15967827]
     [0.79129623]
     [0.84581436]
     [0.1466218 ]
     [0.39014039]
     [0.18521427]
     [0.42079008]
     [0.67567975]
     [0.16974908]
     [0.24365526]
     [0.15289359]
     [0.70534405]
     [0.30967982]
     [0.22671691]
     [0.68325974]
     [0.24053548]
     [0.74107345]
     [0.76718825]
     [0.20229021]
     [0.18195784]
     [0.8735873 ]
     [0.21584937]
     [0.7830216 ]
     [0.20353646]
     [0.33731796]
     [0.29894202]
     [0.85387101]
     [0.75206965]
     [0.2048274 ]
     [0.29596672]
     [0.63317047]
     [0.90083256]
     [0.76879112]
     [0.68582124]
     [0.88945959]
     [0.73508335]
     [0.31827972]
     [0.61474154]
     [0.17365898]
     [0.20470814]
     [0.1877933 ]
     [0.82407279]
     [0.71457202]
     [0.52706714]
     [0.84166168]
     [0.39312469]
     [0.11456422]
     [0.83780445]
     [0.72370532]
     [0.80270641]
     [0.16232869]
     [0.30204349]
     [0.1773537 ]
     [0.75145795]
     [0.13492107]
     [0.76081256]
     [0.37850347]
     [0.28497441]
     [0.80745599]
     [0.11521842]
     [0.85617024]
     [0.30786055]
     [0.75313955]
     [0.29520508]
     [0.33129806]
     [0.29561893]
     [0.57865075]
     [0.35650926]
     [0.21225756]
     [0.4346954 ]
     [0.67200642]
     [0.16762948]
     [0.76774969]
     [0.31656556]
     [0.87681261]
     [0.18793555]
     [0.8203683 ]
     [0.92152391]
     [0.23004217]
     [0.72264097]
     [0.35087757]
     [0.67680113]
     [0.72996509]
     [0.15962024]
     [0.12867141]
     [0.63361403]
     [0.79600484]
     [0.21642372]
     [0.71706999]]
    140
    Step: 12 	 Loss: 0.272 	 Acc: 98.6%
    [[0.3744978 ]
     [0.08575922]
     [0.17180949]
     [0.89129327]
     [0.11376343]
     [0.26739088]
     [0.19038227]
     [0.80659783]
     [0.68175543]
     [0.19836567]
     [0.32038279]
     [0.11802654]
     [0.14451627]
     [0.90091928]
     [0.81175808]
     [0.11671247]
     [0.74823895]
     [0.78778737]
     [0.85483244]
     [0.12391599]
     [0.8125755 ]
     [0.78015392]
     [0.40478047]
     [0.23213166]
     [0.83689593]
     [0.26397642]
     [0.3358312 ]
     [0.14039041]
     [0.09837889]
     [0.77947069]
     [0.26485235]
     [0.64456895]
     [0.64495723]
     [0.75943152]
     [0.81426446]
     [0.82543859]
     [0.15301682]
     [0.87335512]
     [0.75343446]
     [0.68020728]
     [0.18191813]
     [0.72336771]
     [0.89170627]
     [0.12579212]
     [0.8045643 ]
     [0.31010759]
     [0.87998559]
     [0.80411705]
     [0.71051542]
     [0.32749506]
     [0.24116632]
     [0.15199919]
     [0.80080767]
     [0.85443648]
     [0.13949123]
     [0.38995844]
     [0.17784381]
     [0.41669037]
     [0.6829758 ]
     [0.1595631 ]
     [0.23464845]
     [0.14500205]
     [0.70909043]
     [0.30223551]
     [0.21821528]
     [0.6909728 ]
     [0.23307636]
     [0.74669454]
     [0.77159276]
     [0.19426597]
     [0.17370081]
     [0.87986231]
     [0.20749342]
     [0.79100845]
     [0.19678862]
     [0.33275508]
     [0.29148967]
     [0.86177792]
     [0.76104802]
     [0.19463738]
     [0.29031391]
     [0.63909651]
     [0.90704685]
     [0.77587425]
     [0.69263037]
     [0.89366662]
     [0.74156345]
     [0.30935918]
     [0.61897864]
     [0.1655138 ]
     [0.19690786]
     [0.17879524]
     [0.83608498]
     [0.72259298]
     [0.52897396]
     [0.84946264]
     [0.39406416]
     [0.10844824]
     [0.84151192]
     [0.72762765]
     [0.80841447]
     [0.1549435 ]
     [0.29336608]
     [0.16991582]
     [0.75943933]
     [0.12736855]
     [0.76592978]
     [0.37027703]
     [0.27994627]
     [0.81410753]
     [0.10887172]
     [0.86454366]
     [0.29998488]
     [0.76050285]
     [0.28801392]
     [0.32905986]
     [0.28954676]
     [0.58330027]
     [0.3509315 ]
     [0.20443303]
     [0.43149112]
     [0.67633622]
     [0.15775414]
     [0.77304378]
     [0.3120621 ]
     [0.88165112]
     [0.17871114]
     [0.82860751]
     [0.92629374]
     [0.22349323]
     [0.72905139]
     [0.34137909]
     [0.68122866]
     [0.73817134]
     [0.15138666]
     [0.12172776]
     [0.63307218]
     [0.80289027]
     [0.20964391]
     [0.72348474]]
    140
    Step: 13 	 Loss: 0.264 	 Acc: 98.6%
    [[0.36751544]
     [0.07999553]
     [0.16491147]
     [0.89648886]
     [0.10699575]
     [0.25947577]
     [0.18324258]
     [0.81526005]
     [0.68627256]
     [0.18931601]
     [0.31382646]
     [0.11176689]
     [0.13660909]
     [0.90530853]
     [0.81968078]
     [0.10949917]
     [0.75584721]
     [0.79596068]
     [0.86064177]
     [0.11750106]
     [0.81896224]
     [0.78758741]
     [0.40089124]
     [0.22772818]
     [0.84244332]
     [0.25548405]
     [0.33115755]
     [0.13433507]
     [0.09136061]
     [0.78631195]
     [0.25714071]
     [0.65223605]
     [0.65069746]
     [0.76473935]
     [0.81987167]
     [0.83273162]
     [0.14606167]
     [0.8804914 ]
     [0.75958014]
     [0.68493686]
     [0.17409492]
     [0.73298717]
     [0.89579304]
     [0.11851921]
     [0.81266764]
     [0.30319551]
     [0.88432322]
     [0.80639648]
     [0.71581807]
     [0.32395374]
     [0.23254894]
     [0.14488553]
     [0.80967086]
     [0.86237025]
     [0.13289461]
     [0.38978916]
     [0.17095827]
     [0.41274962]
     [0.68993112]
     [0.15021484]
     [0.22618505]
     [0.13771686]
     [0.71267626]
     [0.29515535]
     [0.21023836]
     [0.69831518]
     [0.22603958]
     [0.7520388 ]
     [0.77578334]
     [0.18676087]
     [0.16602293]
     [0.88565687]
     [0.19966564]
     [0.79850538]
     [0.19044579]
     [0.32838798]
     [0.28440948]
     [0.86906073]
     [0.7694957 ]
     [0.18517947]
     [0.28492696]
     [0.64477511]
     [0.91270632]
     [0.7825586 ]
     [0.69912113]
     [0.89758764]
     [0.74771524]
     [0.30088928]
     [0.62305041]
     [0.15795364]
     [0.18960243]
     [0.170442  ]
     [0.84704018]
     [0.73019815]
     [0.53081572]
     [0.8566843 ]
     [0.39497792]
     [0.1028262 ]
     [0.84501938]
     [0.73137642]
     [0.81379842]
     [0.14808603]
     [0.2851385 ]
     [0.16298256]
     [0.76697088]
     [0.12043189]
     [0.77079178]
     [0.36241403]
     [0.27515324]
     [0.82035295]
     [0.10304626]
     [0.87222694]
     [0.29250156]
     [0.7674615 ]
     [0.28118103]
     [0.32691454]
     [0.28376471]
     [0.58777438]
     [0.3455932 ]
     [0.19709464]
     [0.42840879]
     [0.6804844 ]
     [0.14868408]
     [0.77806756]
     [0.30775664]
     [0.88615907]
     [0.17015576]
     [0.83626781]
     [0.93063872]
     [0.21730787]
     [0.73514593]
     [0.33233728]
     [0.68546868]
     [0.74593389]
     [0.14378182]
     [0.11534185]
     [0.63254745]
     [0.80936337]
     [0.20325799]
     [0.72958667]]
    140
    Step: 14 	 Loss: 0.257 	 Acc: 98.6%
    [[0.36082647]
     [0.07476966]
     [0.15846623]
     [0.90129504]
     [0.10080163]
     [0.25198087]
     [0.17655007]
     [0.8233381 ]
     [0.69060185]
     [0.18088262]
     [0.30756769]
     [0.10600231]
     [0.12932436]
     [0.90937926]
     [0.82708272]
     [0.1029097 ]
     [0.76304618]
     [0.80362982]
     [0.86606435]
     [0.11158441]
     [0.82496698]
     [0.79459034]
     [0.39715008]
     [0.22353729]
     [0.84765242]
     [0.24745759]
     [0.32667948]
     [0.12870193]
     [0.08501572]
     [0.79277047]
     [0.24983724]
     [0.65956941]
     [0.65620176]
     [0.76978682]
     [0.82515901]
     [0.83954109]
     [0.13959904]
     [0.88704996]
     [0.76541468]
     [0.68946865]
     [0.16679736]
     [0.74207491]
     [0.89960747]
     [0.11184653]
     [0.82024561]
     [0.29660678]
     [0.88837991]
     [0.80857511]
     [0.720883  ]
     [0.32055772]
     [0.22443554]
     [0.13828358]
     [0.81794154]
     [0.86968386]
     [0.1267801 ]
     [0.3896314 ]
     [0.16451495]
     [0.40895714]
     [0.69656924]
     [0.14162151]
     [0.21822179]
     [0.13097879]
     [0.716113  ]
     [0.28841331]
     [0.20274312]
     [0.70531263]
     [0.21939175]
     [0.75712696]
     [0.77977629]
     [0.17973035]
     [0.15887173]
     [0.89101892]
     [0.19232181]
     [0.80555269]
     [0.18447409]
     [0.32420282]
     [0.27767446]
     [0.87578145]
     [0.77745411]
     [0.17638882]
     [0.27978644]
     [0.65022277]
     [0.91787307]
     [0.7888757 ]
     [0.70531582]
     [0.90124976]
     [0.75356324]
     [0.29283861]
     [0.62696787]
     [0.1509244 ]
     [0.18274976]
     [0.16267507]
     [0.85704491]
     [0.7374178 ]
     [0.53259655]
     [0.86338172]
     [0.39586735]
     [0.09764662]
     [0.84834328]
     [0.73496424]
     [0.81888481]
     [0.14170696]
     [0.2773285 ]
     [0.15650856]
     [0.7740874 ]
     [0.11404781]
     [0.77541803]
     [0.35489104]
     [0.27057793]
     [0.82622653]
     [0.09768722]
     [0.87929044]
     [0.28538256]
     [0.77404676]
     [0.27468028]
     [0.32485537]
     [0.2782516 ]
     [0.5920844 ]
     [0.34047782]
     [0.19020169]
     [0.42544012]
     [0.68446361]
     [0.1403398 ]
     [0.78284189]
     [0.30363493]
     [0.89036758]
     [0.16220844]
     [0.84340158]
     [0.93460718]
     [0.2114571 ]
     [0.74094778]
     [0.32372218]
     [0.6895343 ]
     [0.75328582]
     [0.13674493]
     [0.10945647]
     [0.6320388 ]
     [0.81545834]
     [0.19723374]
     [0.73539861]]
    140
    Step: 15 	 Loss: 0.250 	 Acc: 98.6%
    [[0.35441196]
     [0.07001942]
     [0.15243403]
     [0.9057503 ]
     [0.09512024]
     [0.2448752 ]
     [0.17026696]
     [0.83088218]
     [0.6947561 ]
     [0.17301241]
     [0.301586  ]
     [0.1006826 ]
     [0.1226008 ]
     [0.91316261]
     [0.83400849]
     [0.09687732]
     [0.76986641]
     [0.81083576]
     [0.87113497]
     [0.10611626]
     [0.83062129]
     [0.80119671]
     [0.39354719]
     [0.21954297]
     [0.85255201]
     [0.23986244]
     [0.32238355]
     [0.12345174]
     [0.07926588]
     [0.79887618]
     [0.24291174]
     [0.66659041]
     [0.6614856 ]
     [0.77459325]
     [0.8301525 ]
     [0.84590922]
     [0.13358336]
     [0.89308949]
     [0.77096121]
     [0.6938161 ]
     [0.15997941]
     [0.75066909]
     [0.90317475]
     [0.10571219]
     [0.82734259]
     [0.29031891]
     [0.89218108]
     [0.81066034]
     [0.72572688]
     [0.31729688]
     [0.21678702]
     [0.13214559]
     [0.82567005]
     [0.87643784]
     [0.12110177]
     [0.38948419]
     [0.15847564]
     [0.40530319]
     [0.70291165]
     [0.1337092 ]
     [0.2107194 ]
     [0.12473529]
     [0.71941102]
     [0.28198568]
     [0.19569067]
     [0.71198874]
     [0.21310268]
     [0.76197771]
     [0.78358623]
     [0.17313439]
     [0.15220022]
     [0.89599068]
     [0.18542202]
     [0.81218676]
     [0.17884311]
     [0.32018701]
     [0.27126007]
     [0.88199504]
     [0.78496094]
     [0.16820687]
     [0.27487476]
     [0.65545456]
     [0.92260109]
     [0.79485402]
     [0.71123479]
     [0.90467683]
     [0.75912966]
     [0.28517829]
     [0.63074108]
     [0.14437782]
     [0.17631206]
     [0.15544199]
     [0.86619478]
     [0.74427952]
     [0.53432025]
     [0.86960392]
     [0.39673376]
     [0.09286434]
     [0.85149827]
     [0.73840247]
     [0.82369745]
     [0.13576244]
     [0.26990655]
     [0.1504533 ]
     [0.78082048]
     [0.10816053]
     [0.77982605]
     [0.3476864 ]
     [0.26620466]
     [0.83175902]
     [0.09274653]
     [0.8857964 ]
     [0.27860229]
     [0.78028691]
     [0.26848793]
     [0.32287628]
     [0.2729882 ]
     [0.59624071]
     [0.33557031]
     [0.18371753]
     [0.42257762]
     [0.68828532]
     [0.1326504 ]
     [0.78738552]
     [0.29968407]
     [0.89430413]
     [0.15481447]
     [0.85005569]
     [0.93824093]
     [0.20591483]
     [0.74647797]
     [0.31550605]
     [0.6934374 ]
     [0.76025732]
     [0.13022199]
     [0.10402128]
     [0.63154519]
     [0.82120587]
     [0.19154222]
     [0.74094124]]
    140
    Step: 16 	 Loss: 0.243 	 Acc: 98.6%
    [[0.34825453]
     [0.06569092]
     [0.14677935]
     [0.90988853]
     [0.08989823]
     [0.23813051]
     [0.16435934]
     [0.83793755]
     [0.69874695]
     [0.16565746]
     [0.29586267]
     [0.09576352]
     [0.11638404]
     [0.91668595]
     [0.84049823]
     [0.09134345]
     [0.77633568]
     [0.81761557]
     [0.87588464]
     [0.10105272]
     [0.83595355]
     [0.80743731]
     [0.39007363]
     [0.2157308 ]
     [0.85716786]
     [0.23266703]
     [0.31825752]
     [0.11854968]
     [0.07404307]
     [0.80465612]
     [0.23633673]
     [0.67331873]
     [0.66656316]
     [0.77917609]
     [0.83487548]
     [0.85187386]
     [0.12797406]
     [0.89866179]
     [0.77624067]
     [0.69799146]
     [0.15359977]
     [0.75880479]
     [0.90651717]
     [0.10006163]
     [0.83399856]
     [0.28431139]
     [0.89574925]
     [0.81265879]
     [0.73036485]
     [0.31416207]
     [0.20956782]
     [0.12642905]
     [0.83290188]
     [0.88268597]
     [0.11581889]
     [0.38934668]
     [0.15280612]
     [0.40177887]
     [0.70897802]
     [0.12641199]
     [0.20364223]
     [0.1189397 ]
     [0.72257967]
     [0.27585079]
     [0.18904584]
     [0.7183651 ]
     [0.20714507]
     [0.766608  ]
     [0.78722636]
     [0.16693696]
     [0.14596635]
     [0.90060945]
     [0.17893031]
     [0.81844041]
     [0.17352554]
     [0.31632916]
     [0.2651439 ]
     [0.88775019]
     [0.79205047]
     [0.16058081]
     [0.27017595]
     [0.66048426]
     [0.9269375 ]
     [0.80051929]
     [0.71689662]
     [0.90788996]
     [0.76443463]
     [0.27788181]
     [0.63437921]
     [0.13827074]
     [0.17025528]
     [0.14869568]
     [0.87457533]
     [0.75080848]
     [0.53599032]
     [0.8753945 ]
     [0.39757842]
     [0.08843972]
     [0.85449747]
     [0.74170137]
     [0.82825771]
     [0.13021338]
     [0.26284563]
     [0.14478054]
     [0.78719878]
     [0.10272084]
     [0.78403168]
     [0.34078013]
     [0.26201923]
     [0.836978  ]
     [0.08818201]
     [0.89179992]
     [0.27213736]
     [0.7862076 ]
     [0.26258235]
     [0.32097184]
     [0.26795707]
     [0.60025283]
     [0.33085693]
     [0.17760916]
     [0.41981443]
     [0.69195994]
     [0.12555268]
     [0.79171539]
     [0.29589239]
     [0.89799306]
     [0.14792476]
     [0.85627211]
     [0.94157628]
     [0.2006575 ]
     [0.75175561]
     [0.30766324]
     [0.69718878]
     [0.76687584]
     [0.12416493]
     [0.0989919 ]
     [0.63106565]
     [0.8266336 ]
     [0.18615733]
     [0.74623337]]
    140
    Step: 17 	 Loss: 0.237 	 Acc: 98.6%
    [[0.34233817]
     [0.06173739]
     [0.14147036]
     [0.91373966]
     [0.0850887 ]
     [0.23172109]
     [0.1587967 ]
     [0.84454504]
     [0.70258499]
     [0.15877455]
     [0.29038061]
     [0.09120606]
     [0.11062578]
     [0.91997347]
     [0.84658807]
     [0.08625662]
     [0.78247929]
     [0.82400278]
     [0.88034099]
     [0.09635497]
     [0.84098929]
     [0.81334   ]
     [0.38672121]
     [0.21208775]
     [0.86152303]
     [0.22584254]
     [0.31429027]
     [0.11396477]
     [0.06928814]
     [0.81013477]
     [0.23008715]
     [0.67977253]
     [0.67144739]
     [0.78355116]
     [0.83934897]
     [0.85746899]
     [0.12273497]
     [0.90381265]
     [0.78127205]
     [0.70200589]
     [0.14762135]
     [0.76651423]
     [0.90965449]
     [0.0948467 ]
     [0.8402496 ]
     [0.27856545]
     [0.89910445]
     [0.8145764 ]
     [0.73481068]
     [0.31114496]
     [0.20274558]
     [0.12109609]
     [0.83967808]
     [0.88847606]
     [0.11089529]
     [0.38921816]
     [0.14747568]
     [0.39837604]
     [0.71478638]
     [0.11967107]
     [0.19695789]
     [0.11355054]
     [0.72562745]
     [0.26998887]
     [0.1827768 ]
     [0.72446149]
     [0.20149419]
     [0.77103318]
     [0.79070858]
     [0.16110564]
     [0.14013244]
     [0.90490827]
     [0.17281417]
     [0.82434333]
     [0.16849678]
     [0.31261887]
     [0.25930554]
     [0.89309015]
     [0.79875388]
     [0.15346301]
     [0.26567551]
     [0.66532452]
     [0.93092344]
     [0.8058948 ]
     [0.72231827]
     [0.91090787]
     [0.76949644]
     [0.27092483]
     [0.63789062]
     [0.13256459]
     [0.16454879]
     [0.14239391]
     [0.88226299]
     [0.75702769]
     [0.53760993]
     [0.88079237]
     [0.39840254]
     [0.08433781]
     [0.85735262]
     [0.74487023]
     [0.83258486]
     [0.12502493]
     [0.25612098]
     [0.13945783]
     [0.79324833]
     [0.09768522]
     [0.78804921]
     [0.33415377]
     [0.25800875]
     [0.84190831]
     [0.08395654]
     [0.8973498 ]
     [0.26596641]
     [0.79183209]
     [0.25694382]
     [0.31913711]
     [0.26314234]
     [0.60412948]
     [0.32632514]
     [0.17184682]
     [0.41714427]
     [0.69549696]
     [0.11899031]
     [0.79584679]
     [0.29224926]
     [0.90145598]
     [0.14149526]
     [0.8620885 ]
     [0.94464481]
     [0.19566385]
     [0.75679807]
     [0.30017002]
     [0.70079825]
     [0.77316644]
     [0.11853096]
     [0.09432912]
     [0.63059921]
     [0.83176643]
     [0.18105555]
     [0.75129212]]
    140
    Step: 18 	 Loss: 0.231 	 Acc: 98.6%
    [[0.33664817]
     [0.0581181 ]
     [0.13647847]
     [0.91733018]
     [0.08065035]
     [0.22562343]
     [0.15355153]
     [0.85074148]
     [0.70627989]
     [0.15232465]
     [0.28512419]
     [0.08697574]
     [0.10528306]
     [0.92304658]
     [0.85231059]
     [0.08157151]
     [0.78832027]
     [0.83002779]
     [0.88452873]
     [0.09198864]
     [0.84575152]
     [0.81893006]
     [0.38348246]
     [0.20860205]
     [0.86563824]
     [0.21936267]
     [0.31047162]
     [0.10966936]
     [0.06494953]
     [0.81533431]
     [0.22414017]
     [0.68596857]
     [0.6761502 ]
     [0.78773281]
     [0.8435919 ]
     [0.86272514]
     [0.11783376]
     [0.90858261]
     [0.78607262]
     [0.70586957]
     [0.14201078]
     [0.77382702]
     [0.91260429]
     [0.09002483]
     [0.84612827]
     [0.27306393]
     [0.90226455]
     [0.81641854]
     [0.73907693]
     [0.308238  ]
     [0.19629085]
     [0.11611289]
     [0.8460358 ]
     [0.8938507 ]
     [0.10629873]
     [0.38909799]
     [0.14245672]
     [0.39508727]
     [0.72035325]
     [0.11343401]
     [0.19063695]
     [0.10853087]
     [0.72856208]
     [0.26438182]
     [0.17685472]
     [0.73029603]
     [0.1961276 ]
     [0.7752672 ]
     [0.79404371]
     [0.1556112 ]
     [0.13466466]
     [0.90891641]
     [0.16704424]
     [0.82992239]
     [0.16373464]
     [0.3090467 ]
     [0.25372631]
     [0.89805336]
     [0.80509958]
     [0.14681047]
     [0.26136023]
     [0.66998695]
     [0.93459496]
     [0.81100168]
     [0.72751529]
     [0.9137472 ]
     [0.77433175]
     [0.26428497]
     [0.641283  ]
     [0.12722482]
     [0.15916493]
     [0.13649872]
     [0.88932595]
     [0.76295817]
     [0.53918203]
     [0.88583227]
     [0.39920727]
     [0.0805278 ]
     [0.8600743 ]
     [0.74791751]
     [0.83669625]
     [0.1201659 ]
     [0.24970992]
     [0.13445606]
     [0.79899283]
     [0.09301514]
     [0.79189162]
     [0.32779032]
     [0.25416154]
     [0.84657231]
     [0.08003736]
     [0.90248937]
     [0.26006984]
     [0.79718156]
     [0.25155435]
     [0.31736764]
     [0.25852957]
     [0.60787868]
     [0.32196346]
     [0.1664036 ]
     [0.41456142]
     [0.69890502]
     [0.11291305]
     [0.79979359]
     [0.28874503]
     [0.90471215]
     [0.1354864 ]
     [0.86753866]
     [0.94747411]
     [0.19091462]
     [0.76162119]
     [0.29300447]
     [0.70427477]
     [0.77915199]
     [0.11328189]
     [0.08999826]
     [0.63014498]
     [0.83662686]
     [0.17621562]
     [0.75613308]]
    140
    Step: 19 	 Loss: 0.225 	 Acc: 98.6%
    [[0.33117093]
     [0.05479752]
     [0.13177801]
     [0.92068362]
     [0.07654675]
     [0.21981608]
     [0.14859903]
     [0.85656015]
     [0.7098405 ]
     [0.14627249]
     [0.28007907]
     [0.08304198]
     [0.10031765]
     [0.92592427]
     [0.85769517]
     [0.07724816]
     [0.79387966]
     [0.83571818]
     [0.88847002]
     [0.08792321]
     [0.85026104]
     [0.82423042]
     [0.38035053]
     [0.205263  ]
     [0.86953213]
     [0.21320335]
     [0.30679229]
     [0.10563877]
     [0.06098225]
     [0.8202749 ]
     [0.21847493]
     [0.69192229]
     [0.68068251]
     [0.79173408]
     [0.84762138]
     [0.86766977]
     [0.11324152]
     [0.91300761]
     [0.79065811]
     [0.70959181]
     [0.13673802]
     [0.78077037]
     [0.91538226]
     [0.08555835]
     [0.85166401]
     [0.26779112]
     [0.9052455 ]
     [0.81819006]
     [0.74317505]
     [0.30543428]
     [0.19017682]
     [0.1114492 ]
     [0.85200861]
     [0.89884783]
     [0.10200049]
     [0.38898561]
     [0.13772438]
     [0.39190569]
     [0.72569379]
     [0.10765398]
     [0.1846526 ]
     [0.10384773]
     [0.73139061]
     [0.25901309]
     [0.17125346]
     [0.73588538]
     [0.19102491]
     [0.77932276]
     [0.79724152]
     [0.15042723]
     [0.1295326 ]
     [0.91265991]
     [0.16159392]
     [0.83520192]
     [0.15921907]
     [0.30560399]
     [0.24838915]
     [0.90267407]
     [0.81111345]
     [0.14058443]
     [0.25721808]
     [0.67448221]
     [0.93798373]
     [0.81585911]
     [0.73250189]
     [0.91642283]
     [0.77895574]
     [0.25794167]
     [0.6445634 ]
     [0.12222042]
     [0.15407874]
     [0.13097599]
     [0.89582505]
     [0.76861916]
     [0.54070931]
     [0.89054525]
     [0.39999368]
     [0.0769824 ]
     [0.86267203]
     [0.75085091]
     [0.8406076 ]
     [0.11560839]
     [0.24359166]
     [0.12974907]
     [0.80445387]
     [0.08867642]
     [0.79557071]
     [0.32167403]
     [0.25046692]
     [0.85099022]
     [0.07639552]
     [0.90725709]
     [0.25442975]
     [0.80227524]
     [0.2463975 ]
     [0.31565938]
     [0.25410555]
     [0.61150781]
     [0.31776138]
     [0.16125517]
     [0.41206063]
     [0.70219203]
     [0.10727606]
     [0.80356838]
     [0.28537089]
     [0.90777878]
     [0.1298626 ]
     [0.87265297]
     [0.95008835]
     [0.18639233]
     [0.76623941]
     [0.28614632]
     [0.70762652]
     [0.78485338]
     [0.10838363]
     [0.08596853]
     [0.62970213]
     [0.84123524]
     [0.17161831]
     [0.76077052]]
    140
    Step: 20 	 Loss: 0.220 	 Acc: 98.6%
    [[0.32589394]
     [0.05174457]
     [0.12734582]
     [0.92382089]
     [0.07274568]
     [0.21427942]
     [0.14391678]
     [0.86203114]
     [0.71327491]
     [0.14058619]
     [0.27523211]
     [0.07937766]
     [0.09569544]
     [0.92862347]
     [0.86276839]
     [0.07325128]
     [0.79917666]
     [0.84109897]
     [0.89218473]
     [0.08413156]
     [0.85453665]
     [0.82926196]
     [0.37731913]
     [0.20206088]
     [0.87322152]
     [0.20734259]
     [0.30324379]
     [0.1018509 ]
     [0.05734694]
     [0.82497485]
     [0.21307239]
     [0.69764798]
     [0.68505437]
     [0.79556688]
     [0.85145289]
     [0.87232769]
     [0.10893232]
     [0.91711961]
     [0.79504284]
     [0.71318116]
     [0.13177599]
     [0.78736926]
     [0.91800242]
     [0.08141383]
     [0.85688346]
     [0.26273258]
     [0.90806162]
     [0.81989537]
     [0.74711554]
     [0.30272752]
     [0.18437901]
     [0.10707793]
     [0.85762691]
     [0.90350133]
     [0.09797488]
     [0.38888054]
     [0.13325627]
     [0.38882505]
     [0.73082195]
     [0.10228916]
     [0.17898047]
     [0.09947167]
     [0.73411947]
     [0.25386749]
     [0.16594926]
     [0.74124483]
     [0.18616758]
     [0.78321146]
     [0.80031096]
     [0.14552986]
     [0.12470889]
     [0.91616199]
     [0.1564391 ]
     [0.84020404]
     [0.15493193]
     [0.30228283]
     [0.24327842]
     [0.90698282]
     [0.81681913]
     [0.13474985]
     [0.25323805]
     [0.67882009]
     [0.94111761]
     [0.82048455]
     [0.73729114]
     [0.91894803]
     [0.78338226]
     [0.25187602]
     [0.64773827]
     [0.11752356]
     [0.1492676 ]
     [0.12579498]
     [0.90181455]
     [0.77402831]
     [0.54219427]
     [0.89495916]
     [0.40076278]
     [0.07367741]
     [0.86515444]
     [0.75367743]
     [0.84433315]
     [0.11132737]
     [0.2377471 ]
     [0.12531329]
     [0.80965112]
     [0.08463868]
     [0.79909721]
     [0.31579037]
     [0.24691518]
     [0.85518032]
     [0.07300534]
     [0.91168721]
     [0.24902968]
     [0.80713071]
     [0.24145821]
     [0.31400863]
     [0.24985824]
     [0.61502367]
     [0.31370926]
     [0.15637948]
     [0.40963708]
     [0.70536523]
     [0.1020393 ]
     [0.80718263]
     [0.28211879]
     [0.91067125]
     [0.12459183]
     [0.87745879]
     [0.95250875]
     [0.18208112]
     [0.77066594]
     [0.27957683]
     [0.71086101]
     [0.79028972]
     [0.10380568]
     [0.08221259]
     [0.62926986]
     [0.84561001]
     [0.16724613]
     [0.76521747]]
    140
    Step: 21 	 Loss: 0.215 	 Acc: 98.6%
    [[0.32080564]
     [0.04893196]
     [0.12316103]
     [0.92676066]
     [0.06921861]
     [0.20899548]
     [0.13948447]
     [0.86718168]
     [0.71659054]
     [0.13523688]
     [0.27057122]
     [0.07595862]
     [0.09138599]
     [0.93115927]
     [0.86755429]
     [0.06954962]
     [0.80422884]
     [0.84619291]
     [0.89569076]
     [0.08058948]
     [0.85859541]
     [0.83404366]
     [0.37438247]
     [0.19898685]
     [0.87672162]
     [0.20176024]
     [0.29981831]
     [0.09828592]
     [0.05400909]
     [0.82945085]
     [0.20791508]
     [0.70315885]
     [0.68927506]
     [0.79924206]
     [0.85510049]
     [0.87672124]
     [0.10488288]
     [0.92094702]
     [0.79923992]
     [0.71664545]
     [0.12710024]
     [0.79364667]
     [0.92047735]
     [0.07756155]
     [0.86181076]
     [0.25787507]
     [0.91072579]
     [0.82153848]
     [0.75090803]
     [0.30011194]
     [0.17887511]
     [0.10297476]
     [0.86291824]
     [0.90784148]
     [0.09419891]
     [0.38878234]
     [0.12903212]
     [0.38583955]
     [0.73575055]
     [0.09730216]
     [0.1735983 ]
     [0.09537635]
     [0.73675455]
     [0.24893108]
     [0.16092054]
     [0.74638847]
     [0.1815387 ]
     [0.78694391]
     [0.80326015]
     [0.14089747]
     [0.12016887]
     [0.91944336]
     [0.15155791]
     [0.84494882]
     [0.15085676]
     [0.29907598]
     [0.23837978]
     [0.91100691]
     [0.82223817]
     [0.12927507]
     [0.24941005]
     [0.68300964]
     [0.9440212 ]
     [0.82489391]
     [0.74189504]
     [0.92133473]
     [0.787624  ]
     [0.2460706 ]
     [0.65081358]
     [0.11310916]
     [0.14471103]
     [0.12092798]
     [0.90734285]
     [0.7792018 ]
     [0.54363919]
     [0.89909898]
     [0.40151553]
     [0.07059132]
     [0.86752937]
     [0.75640352]
     [0.84788584]
     [0.10730032]
     [0.23215872]
     [0.12112748]
     [0.81460256]
     [0.08087487]
     [0.80248092]
     [0.31012589]
     [0.24349741]
     [0.85915919]
     [0.06984403]
     [0.91581025]
     [0.24385453]
     [0.81176399]
     [0.23672269]
     [0.31241201]
     [0.24577659]
     [0.61843254]
     [0.30979825]
     [0.15175653]
     [0.40728633]
     [0.70843126]
     [0.09716694]
     [0.81064678]
     [0.27898136]
     [0.9134034 ]
     [0.11964522]
     [0.8819808 ]
     [0.95475401]
     [0.17796654]
     [0.7749129 ]
     [0.27327867]
     [0.7139851 ]
     [0.79547852]
     [0.09952069]
     [0.07870608]
     [0.62884744]
     [0.84976792]
     [0.16308323]
     [0.76948586]]
    140
    Step: 22 	 Loss: 0.210 	 Acc: 98.6%
    [[0.31589533]
     [0.04633573]
     [0.11920475]
     [0.9295196 ]
     [0.0659402 ]
     [0.20394779]
     [0.13528368]
     [0.87203647]
     [0.71979425]
     [0.13019842]
     [0.26608529]
     [0.07276333]
     [0.08736203]
     [0.93354519]
     [0.87207465]
     [0.06611546]
     [0.80905228]
     [0.85102073]
     [0.89900424]
     [0.07727538]
     [0.86245282]
     [0.83859285]
     [0.37153522]
     [0.19603282]
     [0.88004619]
     [0.19643783]
     [0.29650868]
     [0.09492599]
     [0.05093838]
     [0.83371811]
     [0.20298705]
     [0.7084671 ]
     [0.69335311]
     [0.80276956]
     [0.85857695]
     [0.88087069]
     [0.10107225]
     [0.92451521]
     [0.80326135]
     [0.71999189]
     [0.12268863]
     [0.79962374]
     [0.92281837]
     [0.07397508]
     [0.86646782]
     [0.25320642]
     [0.91324957]
     [0.82312307]
     [0.75456133]
     [0.29758224]
     [0.17364472]
     [0.09911784]
     [0.8679076 ]
     [0.91189541]
     [0.09065194]
     [0.38869062]
     [0.12503362]
     [0.38294388]
     [0.74049142]
     [0.09265948]
     [0.1684858 ]
     [0.09153813]
     [0.73930125]
     [0.24419103]
     [0.15614762]
     [0.75132926]
     [0.17712284]
     [0.79052979]
     [0.80609654]
     [0.13651043]
     [0.11589026]
     [0.92252259]
     [0.14693047]
     [0.84945457]
     [0.14697858]
     [0.29597678]
     [0.23368006]
     [0.91477077]
     [0.8273903 ]
     [0.12413145]
     [0.24572486]
     [0.6870592 ]
     [0.94671626]
     [0.82910171]
     [0.74632462]
     [0.92359365]
     [0.79169258]
     [0.24050936]
     [0.65379482]
     [0.10895465]
     [0.14039044]
     [0.11634998]
     [0.91245317]
     [0.78415448]
     [0.5450462 ]
     [0.90298716]
     [0.40225281]
     [0.06770496]
     [0.86980392]
     [0.75903507]
     [0.85127743]
     [0.10350698]
     [0.22681038]
     [0.11717244]
     [0.81932463]
     [0.07736084]
     [0.8057308 ]
     [0.30466814]
     [0.24020548]
     [0.86294187]
     [0.0668913 ]
     [0.91965347]
     [0.23889041]
     [0.81618974]
     [0.23217827]
     [0.31086643]
     [0.24185046]
     [0.62174021]
     [0.30602018]
     [0.14736814]
     [0.40500431]
     [0.71139625]
     [0.0926269 ]
     [0.81397039]
     [0.27595183]
     [0.91598766]
     [0.11499673]
     [0.88624128]
     [0.95684065]
     [0.17403539]
     [0.77899136]
     [0.26723578]
     [0.71700514]
     [0.80043584]
     [0.09550415]
     [0.07542725]
     [0.62843418]
     [0.85372418]
     [0.15911517]
     [0.77358667]]
    140
    Step: 23 	 Loss: 0.206 	 Acc: 98.6%
    [[0.31115314]
     [0.04393474]
     [0.11545989]
     [0.93211267]
     [0.06288792]
     [0.19912123]
     [0.13129769]
     [0.87661789]
     [0.72289231]
     [0.12544708]
     [0.26176409]
     [0.06977256]
     [0.08359914]
     [0.93579333]
     [0.87634925]
     [0.06292418]
     [0.81366175]
     [0.8556013 ]
     [0.90213975]
     [0.07416991]
     [0.86612297]
     [0.84292536]
     [0.36877246]
     [0.19319137]
     [0.88320774]
     [0.19135843]
     [0.29330831]
     [0.09175505]
     [0.04810811]
     [0.83779055]
     [0.1982736 ]
     [0.71358406]
     [0.6972964 ]
     [0.80615848]
     [0.86189391]
     [0.88479435]
     [0.09748155]
     [0.92784684]
     [0.80711812]
     [0.72322712]
     [0.11852114]
     [0.80531992]
     [0.92503566]
     [0.07063082]
     [0.87087454]
     [0.24871541]
     [0.91564345]
     [0.82465249]
     [0.75808359]
     [0.29513355]
     [0.16866918]
     [0.09548746]
     [0.87261769]
     [0.91568745]
     [0.08731541]
     [0.38860503]
     [0.12124414]
     [0.38013312]
     [0.74505546]
     [0.08833113]
     [0.16362442]
     [0.08793578]
     [0.74176457]
     [0.23963554]
     [0.15161259]
     [0.75607917]
     [0.1729059 ]
     [0.79397803]
     [0.80882696]
     [0.13235093]
     [0.11185291]
     [0.92541631]
     [0.14253866]
     [0.85373794]
     [0.14328378]
     [0.29297908]
     [0.22916715]
     [0.91829633]
     [0.83229357]
     [0.11929305]
     [0.24217398]
     [0.69097647]
     [0.94922208]
     [0.83312124]
     [0.75059008]
     [0.92573446]
     [0.79559866]
     [0.2351775 ]
     [0.65668705]
     [0.10503963]
     [0.13628892]
     [0.11203837]
     [0.91718412]
     [0.78890003]
     [0.54641725]
     [0.90664395]
     [0.40297546]
     [0.06500119]
     [0.87198459]
     [0.76157749]
     [0.85451865]
     [0.09992906]
     [0.22168724]
     [0.1134308 ]
     [0.8238324 ]
     [0.07407501]
     [0.80885507]
     [0.2994056 ]
     [0.23703189]
     [0.86654208]
     [0.06412902]
     [0.92324125]
     [0.23412453]
     [0.8204214 ]
     [0.22781331]
     [0.30936905]
     [0.23807056]
     [0.62495204]
     [0.30236753]
     [0.14319774]
     [0.40278724]
     [0.71426582]
     [0.08839042]
     [0.81716221]
     [0.27302398]
     [0.91843525]
     [0.11062282]
     [0.89026038]
     [0.95878329]
     [0.17027566]
     [0.78291155]
     [0.26143331]
     [0.71992693]
     [0.80517642]
     [0.09173398]
     [0.07235667]
     [0.62802946]
     [0.85749269]
     [0.15532877]
     [0.77752997]]
    140
    Step: 24 	 Loss: 0.202 	 Acc: 98.6%
    [[0.30656988]
     [0.04171032]
     [0.11191095]
     [0.93455328]
     [0.06004166]
     [0.19450191]
     [0.12751128]
     [0.8809463 ]
     [0.72589055]
     [0.12096132]
     [0.25759815]
     [0.06696905]
     [0.08007541]
     [0.9379146 ]
     [0.88039606]
     [0.05995384]
     [0.8180708 ]
     [0.85995187]
     [0.90511049]
     [0.07125573]
     [0.8696187 ]
     [0.84705566]
     [0.36608966]
     [0.19045573]
     [0.88621763]
     [0.18650646]
     [0.29021111]
     [0.08875858]
     [0.04549473]
     [0.84168091]
     [0.19376128]
     [0.71852023]
     [0.70111226]
     [0.8094172 ]
     [0.86506199]
     [0.88850889]
     [0.09409372]
     [0.93096219]
     [0.81082032]
     [0.72635727]
     [0.11457959]
     [0.81075314]
     [0.92713843]
     [0.06750766]
     [0.87504903]
     [0.24439167]
     [0.9179169 ]
     [0.82612984]
     [0.76148228]
     [0.29276139]
     [0.16393141]
     [0.09206586]
     [0.87706916]
     [0.91923945]
     [0.08417261]
     [0.38852525]
     [0.11764859]
     [0.37740272]
     [0.74945275]
     [0.08429014]
     [0.15899717]
     [0.08455019]
     [0.74414908]
     [0.23525375]
     [0.14729906]
     [0.76064924]
     [0.16887494]
     [0.7972968 ]
     [0.81145767]
     [0.12840276]
     [0.10803857]
     [0.92813949]
     [0.13836599]
     [0.85781417]
     [0.13975994]
     [0.29007726]
     [0.22482989]
     [0.92160328]
     [0.83696452]
     [0.11473633]
     [0.23874959]
     [0.69476859]
     [0.9515558 ]
     [0.83696464]
     [0.75470079]
     [0.9277659 ]
     [0.79935204]
     [0.23006134]
     [0.65949497]
     [0.10134567]
     [0.13239106]
     [0.10797265]
     [0.92157025]
     [0.79345104]
     [0.54775418]
     [0.9100876 ]
     [0.40368423]
     [0.06246468]
     [0.8740773 ]
     [0.76403577]
     [0.85761931]
     [0.09655005]
     [0.21677562]
     [0.1098868 ]
     [0.82813966]
     [0.07099804]
     [0.81186127]
     [0.29432754]
     [0.23396978]
     [0.86997228]
     [0.06154099]
     [0.92659544]
     [0.22954511]
     [0.82447127]
     [0.2236171 ]
     [0.30791725]
     [0.23442831]
     [0.62807301]
     [0.29883336]
     [0.13923024]
     [0.40063164]
     [0.71704516]
     [0.08443167]
     [0.82023027]
     [0.27019209]
     [0.92075631]
     [0.10650216]
     [0.89405637]
     [0.96059493]
     [0.16667631]
     [0.78668283]
     [0.25585746]
     [0.72275587]
     [0.8097138 ]
     [0.08819031]
     [0.06947689]
     [0.62763268]
     [0.86108611]
     [0.15171201]
     [0.78132504]]
    140
    Step: 25 	 Loss: 0.198 	 Acc: 98.6%
    [[0.30213708]
     [0.03964591]
     [0.10854383]
     [0.93685353]
     [0.05738348]
     [0.19007704]
     [0.12391057]
     [0.88504022]
     [0.72879433]
     [0.11672154]
     [0.25357875]
     [0.06433735]
     [0.07677111]
     [0.93991882]
     [0.88423145]
     [0.05718486]
     [0.82229189]
     [0.86408818]
     [0.90792843]
     [0.06851725]
     [0.87295175]
     [0.850997  ]
     [0.3634826 ]
     [0.18781966]
     [0.88908622]
     [0.1818676 ]
     [0.28721146]
     [0.08592346]
     [0.04307739]
     [0.84540085]
     [0.18943769]
     [0.72328535]
     [0.70480744]
     [0.8125534 ]
     [0.86809089]
     [0.89202942]
     [0.09089333]
     [0.93387945]
     [0.81437725]
     [0.729388  ]
     [0.11084744]
     [0.81593993]
     [0.92913502]
     [0.06458672]
     [0.8790078 ]
     [0.24022567]
     [0.9200785 ]
     [0.82755795]
     [0.76476432]
     [0.29046163]
     [0.15941572]
     [0.08883698]
     [0.8812808 ]
     [0.92257109]
     [0.08120846]
     [0.38845098]
     [0.11423324]
     [0.37474847]
     [0.75369261]
     [0.08051229]
     [0.15458849]
     [0.08136413]
     [0.74645902]
     [0.23103561]
     [0.14319208]
     [0.76504969]
     [0.16501814]
     [0.80049364]
     [0.81399443]
     [0.12465112]
     [0.10443068]
     [0.9307056 ]
     [0.13439738]
     [0.86169717]
     [0.13639571]
     [0.28726607]
     [0.22065799]
     [0.92470935]
     [0.84141833]
     [0.11043993]
     [0.23544449]
     [0.69844216]
     [0.95373272]
     [0.84064309]
     [0.75866546]
     [0.92969587]
     [0.80296177]
     [0.22514825]
     [0.66222292]
     [0.09785605]
     [0.12868282]
     [0.10413426]
     [0.92564248]
     [0.79781911]
     [0.54905867]
     [0.9133346 ]
     [0.40437987]
     [0.06008167]
     [0.87608748]
     [0.7664145 ]
     [0.86058839]
     [0.09335498]
     [0.21206287]
     [0.10652612]
     [0.83225912]
     [0.06811258]
     [0.81475633]
     [0.28942405]
     [0.23101279]
     [0.87324389]
     [0.0591127 ]
     [0.92973564]
     [0.22514126]
     [0.82835068]
     [0.21957974]
     [0.30650863]
     [0.23091582]
     [0.63110773]
     [0.29541124]
     [0.13545186]
     [0.39853431]
     [0.71973909]
     [0.08072742]
     [0.82318193]
     [0.26745086]
     [0.92296004]
     [0.10261546]
     [0.89764582]
     [0.96228712]
     [0.16322724]
     [0.79031386]
     [0.25049547]
     [0.72549691]
     [0.81406047]
     [0.08485521]
     [0.06677227]
     [0.62724329]
     [0.86451601]
     [0.14825385]
     [0.78498043]]
    140
    Step: 26 	 Loss: 0.194 	 Acc: 98.6%
    [[0.29784682]
     [0.03772682]
     [0.1053457 ]
     [0.9390243 ]
     [0.0548973 ]
     [0.18583483]
     [0.1204829 ]
     [0.88891652]
     [0.73160863]
     [0.11270988]
     [0.2496978 ]
     [0.06186353]
     [0.07366848]
     [0.94181485]
     [0.88787032]
     [0.05459975]
     [0.82633651]
     [0.86802466]
     [0.91060445]
     [0.06594043]
     [0.87613284]
     [0.85476155]
     [0.36094741]
     [0.18527742]
     [0.89182297]
     [0.17742865]
     [0.28430416]
     [0.0832378 ]
     [0.04083761]
     [0.84896109]
     [0.18529139]
     [0.72788847]
     [0.70838822]
     [0.81557417]
     [0.87098951]
     [0.89536972]
     [0.08786641]
     [0.93661499]
     [0.81779746]
     [0.73232456]
     [0.10730969]
     [0.82089554]
     [0.931033  ]
     [0.06185105]
     [0.88276591]
     [0.23620855]
     [0.92213606]
     [0.82893944]
     [0.7679361 ]
     [0.28823042]
     [0.15510774]
     [0.08578626]
     [0.88526976]
     [0.9257001 ]
     [0.07840932]
     [0.38838197]
     [0.11098558]
     [0.37216649]
     [0.75778368]
     [0.07697578]
     [0.15038411]
     [0.07836204]
     [0.7486983 ]
     [0.22697183]
     [0.13927792]
     [0.76928998]
     [0.16132459]
     [0.80357552]
     [0.81644254]
     [0.12108252]
     [0.10101419]
     [0.93312681]
     [0.13061905]
     [0.8653997 ]
     [0.1331807 ]
     [0.28454071]
     [0.21664195]
     [0.92763053]
     [0.84566895]
     [0.10638444]
     [0.23225204]
     [0.70200332]
     [0.95576647]
     [0.84416682]
     [0.7624921 ]
     [0.93153156]
     [0.80643615]
     [0.22042652]
     [0.66487494]
     [0.09455563]
     [0.12515137]
     [0.10050633]
     [0.92942852]
     [0.80201499]
     [0.55033228]
     [0.91639989]
     [0.40506305]
     [0.0578398 ]
     [0.8780201 ]
     [0.76871796]
     [0.86343411]
     [0.09033028]
     [0.20753734]
     [0.10333575]
     [0.83620242]
     [0.065403  ]
     [0.81754664]
     [0.28468588]
     [0.22815509]
     [0.87636733]
     [0.0568311 ]
     [0.9326795 ]
     [0.22090292]
     [0.83207004]
     [0.21569211]
     [0.30514094]
     [0.22752578]
     [0.63406049]
     [0.29209524]
     [0.13184998]
     [0.39649225]
     [0.72235205]
     [0.07725675]
     [0.82602399]
     [0.26479541]
     [0.92505477]
     [0.09894518]
     [0.9010438 ]
     [0.96387021]
     [0.15991917]
     [0.79381262]
     [0.24533545]
     [0.72815466]
     [0.81822793]
     [0.08171248]
     [0.06422871]
     [0.6268608 ]
     [0.867793  ]
     [0.14494423]
     [0.78850405]]
    140
    Step: 27 	 Loss: 0.190 	 Acc: 98.6%
    [[0.29369177]
     [0.03593997]
     [0.10230486]
     [0.94107546]
     [0.0525687 ]
     [0.1817644 ]
     [0.11721668]
     [0.89259059]
     [0.73433805]
     [0.10891007]
     [0.24594783]
     [0.05953507]
     [0.0707515 ]
     [0.94361072]
     [0.89132631]
     [0.05218282]
     [0.83021522]
     [0.87177452]
     [0.91314846]
     [0.06351261]
     [0.87917181]
     [0.85836048]
     [0.35848045]
     [0.18282371]
     [0.89443651]
     [0.17317746]
     [0.28148438]
     [0.0806908 ]
     [0.03875899]
     [0.85237148]
     [0.18131185]
     [0.73233803]
     [0.71186043]
     [0.81848606]
     [0.873766  ]
     [0.89854236]
     [0.08500024]
     [0.93918351]
     [0.82108885]
     [0.73517182]
     [0.10395261]
     [0.82563407]
     [0.93283924]
     [0.0592854 ]
     [0.88633714]
     [0.23233213]
     [0.92409671]
     [0.83027676]
     [0.77100352]
     [0.28606423]
     [0.15099422]
     [0.0829005 ]
     [0.88905169]
     [0.9286425 ]
     [0.07576282]
     [0.38831797]
     [0.10789416]
     [0.36965314]
     [0.76173397]
     [0.07366093]
     [0.14637093]
     [0.07552983]
     [0.75087055]
     [0.22305383]
     [0.13554403]
     [0.77337889]
     [0.1577843 ]
     [0.80654884]
     [0.81880692]
     [0.11768462]
     [0.0977754 ]
     [0.93541412]
     [0.12701839]
     [0.86893345]
     [0.13010538]
     [0.28189669]
     [0.21277297]
     [0.93038128]
     [0.84972922]
     [0.10255217]
     [0.22916607]
     [0.70545778]
     [0.95766925]
     [0.84754527]
     [0.76618818]
     [0.93327949]
     [0.80978289]
     [0.21588533]
     [0.66745477]
     [0.09143061]
     [0.12178496]
     [0.09707352]
     [0.93295326]
     [0.80604856]
     [0.55157648]
     [0.91929703]
     [0.40573439]
     [0.05572791]
     [0.87987974]
     [0.77095006]
     [0.86616404]
     [0.08746361]
     [0.20318822]
     [0.10030382]
     [0.83998032]
     [0.06285525]
     [0.8202381 ]
     [0.28010444]
     [0.22539126]
     [0.87935215]
     [0.05468444]
     [0.9354429 ]
     [0.21682079]
     [0.83563895]
     [0.21194578]
     [0.30381214]
     [0.22425147]
     [0.63693527]
     [0.28887984]
     [0.12841305]
     [0.39450269]
     [0.72488817]
     [0.07400077]
     [0.82876272]
     [0.26222123]
     [0.9270481 ]
     [0.09547539]
     [0.90426402]
     [0.96535341]
     [0.15674358]
     [0.79718651]
     [0.24036638]
     [0.73073339]
     [0.82222681]
     [0.07874744]
     [0.0618335 ]
     [0.62648472]
     [0.87092682]
     [0.14177385]
     [0.7919032 ]]
    140
    Step: 28 	 Loss: 0.187 	 Acc: 98.6%
    [[0.2896651 ]
     [0.03427371]
     [0.09941063]
     [0.9430159 ]
     [0.05038473]
     [0.17785572]
     [0.1141013 ]
     [0.89607649]
     [0.73698687]
     [0.10530724]
     [0.24232188]
     [0.05734065]
     [0.06800567]
     [0.94531371]
     [0.89461186]
     [0.04992   ]
     [0.83393781]
     [0.8753499 ]
     [0.9155695 ]
     [0.06122234]
     [0.88207768]
     [0.86180407]
     [0.35607836]
     [0.18045364]
     [0.89693478]
     [0.16910279]
     [0.27874766]
     [0.07827262]
     [0.0368269 ]
     [0.85564112]
     [0.17748933]
     [0.73664188]
     [0.7152295 ]
     [0.82129511]
     [0.87642786]
     [0.90155881]
     [0.08228329]
     [0.9415983 ]
     [0.82425872]
     [0.7379343 ]
     [0.1007637 ]
     [0.83016855]
     [0.93456002]
     [0.05687605]
     [0.88973409]
     [0.22858882]
     [0.92596692]
     [0.83157213]
     [0.77397209]
     [0.28395977]
     [0.147063  ]
     [0.08016771]
     [0.8926409 ]
     [0.93141279]
     [0.07325777]
     [0.38825875]
     [0.10494856]
     [0.36720506]
     [0.76555092]
     [0.07055   ]
     [0.14253688]
     [0.07285474]
     [0.75297914]
     [0.21927364]
     [0.13197884]
     [0.77732456]
     [0.15438804]
     [0.80941955]
     [0.8210921 ]
     [0.11444614]
     [0.09470183]
     [0.9375775 ]
     [0.12358381]
     [0.87230915]
     [0.12716101]
     [0.27932988]
     [0.20904292]
     [0.93297465]
     [0.85361095]
     [0.09892702]
     [0.22618089]
     [0.70881083]
     [0.95945203]
     [0.85078713]
     [0.76976061]
     [0.93494564]
     [0.8130091 ]
     [0.21151462]
     [0.66996591]
     [0.08846847]
     [0.11857282]
     [0.09382184]
     [0.93623907]
     [0.80992902]
     [0.55279264]
     [0.92203833]
     [0.40639448]
     [0.05373595]
     [0.88167062]
     [0.77311447]
     [0.86878513]
     [0.08474375]
     [0.19900552]
     [0.09741949]
     [0.84360273]
     [0.06045663]
     [0.82283616]
     [0.27567172]
     [0.2227163 ]
     [0.88220712]
     [0.05266214]
     [0.93804019]
     [0.21288624]
     [0.83906625]
     [0.20833293]
     [0.3025203 ]
     [0.22108663]
     [0.63973578]
     [0.2857599 ]
     [0.12513048]
     [0.39256307]
     [0.72735129]
     [0.07094243]
     [0.83140389]
     [0.25972411]
     [0.92894693]
     [0.09219158]
     [0.907319  ]
     [0.96674502]
     [0.1536926 ]
     [0.80044237]
     [0.23557798]
     [0.73323707]
     [0.8260669 ]
     [0.07594677]
     [0.05957518]
     [0.62611464]
     [0.87392639]
     [0.13873421]
     [0.79518465]]
    140
    Step: 29 	 Loss: 0.183 	 Acc: 98.6%
    [[0.28576043]
     [0.0327176 ]
     [0.09665322]
     [0.94485374]
     [0.04833372]
     [0.17409948]
     [0.11112705]
     [0.89938711]
     [0.73955909]
     [0.1018878 ]
     [0.2388135 ]
     [0.05527002]
     [0.06541788]
     [0.94693043]
     [0.89773839]
     [0.04779863]
     [0.83751332]
     [0.87876194]
     [0.91787583]
     [0.05905924]
     [0.88485872]
     [0.86510179]
     [0.35373803]
     [0.17816269]
     [0.89932504]
     [0.16519426]
     [0.27608984]
     [0.07597431]
     [0.03502831]
     [0.85877835]
     [0.17381483]
     [0.74080732]
     [0.71850049]
     [0.82400693]
     [0.87898199]
     [0.90442958]
     [0.07970503]
     [0.94387137]
     [0.82731381]
     [0.74061619]
     [0.09773153]
     [0.83451105]
     [0.93620105]
     [0.05461061]
     [0.8929683 ]
     [0.22497158]
     [0.92775265]
     [0.83282765]
     [0.7768469 ]
     [0.28191397]
     [0.14330284]
     [0.07757698]
     [0.8960505 ]
     [0.93402408]
     [0.07088395]
     [0.3882041 ]
     [0.10213918]
     [0.36481911]
     [0.76924146]
     [0.06762691]
     [0.13887085]
     [0.07032519]
     [0.75502722]
     [0.21562387]
     [0.12857175]
     [0.78113455]
     [0.15112729]
     [0.81219316]
     [0.82330228]
     [0.11135669]
     [0.09178207]
     [0.93962599]
     [0.1203047 ]
     [0.87553669]
     [0.12433956]
     [0.2768364 ]
     [0.20544424]
     [0.93542251]
     [0.85732504]
     [0.09549427]
     [0.22329122]
     [0.71206743]
     [0.96112464]
     [0.85390041]
     [0.77321581]
     [0.93653542]
     [0.81612137]
     [0.20730507]
     [0.6724116 ]
     [0.08565775]
     [0.11550508]
     [0.09073855]
     [0.93930607]
     [0.81366487]
     [0.55398202]
     [0.92463498]
     [0.40704389]
     [0.0518548 ]
     [0.88339663]
     [0.77521457]
     [0.8713038 ]
     [0.08216047]
     [0.19497997]
     [0.09467286]
     [0.8470788 ]
     [0.05819564]
     [0.82534588]
     [0.27138026]
     [0.22012556]
     [0.88494031]
     [0.05075463]
     [0.94048433]
     [0.20909126]
     [0.84236013]
     [0.20484632]
     [0.30126364]
     [0.21802546]
     [0.6424655 ]
     [0.28273067]
     [0.12199252]
     [0.39067098]
     [0.72974497]
     [0.0680663 ]
     [0.83395287]
     [0.25730016]
     [0.93075759]
     [0.08908052]
     [0.91022016]
     [0.96805246]
     [0.15075897]
     [0.80358652]
     [0.2309607 ]
     [0.73566937]
     [0.82975731]
     [0.0732984 ]
     [0.05744334]
     [0.62575015]
     [0.87679995]
     [0.13581745]
     [0.7983547 ]]
    140

    Testing...
    Acc: 100.0%

```python
def plot_model_prediction(prediction_func, X, Y, hard=True):
    u_min = X[:, 0].min()-1
    u_max = X[:, 0].max()+1
    v_min = X[:, 1].min()-1
    v_max = X[:, 1].max()+1

    U, V = np.meshgrid(np.linspace(u_min, u_max, 100), np.linspace(v_min, v_max, 100))
    UV = np.stack([U.ravel(), V.ravel()]).T
    c = prediction_func(UV).ravel()
    if hard:
        c = c > 0.5
    plt.scatter(UV[:,0], UV[:,1], c=c, edgecolors= 'none', alpha=0.15)
    plt.scatter(X[:,0], X[:,1], c=Y.ravel(), edgecolors= 'black')
    plt.xlim(left=u_min, right=u_max)
    plt.ylim(bottom=v_min, top=v_max)
    plt.axes().set_aspect('equal')
    plt.show()

plot_model_prediction(lambda x: model.forward(x), X_train, Y_train, False)

plot_model_prediction(lambda x: model.forward(x), X_train, Y_train, True)
```

```python
# Now run the same experiment on 2 circles
```

# 2. Decision Tree
The next model we look at is called **Decision Tree**. This type of model is non-parametric, meaning in contrast to **Logistic Regression** we do not have any parameters here that need to be trained.

Let us consider a simple binary decision tree for deciding on the two classes of "creditable" and "Not creditable".

Each node, except the leafs, asks a question about the the client in question. A decision is made by going from the root node to a leaf node, while considering the clients situation. The situation of the client, in this case, is fully described by the features:
1. Checking account balance
2. Duration of requested credit
3. Payment status of previous loan
4. Length of current employment

In order to build a decision tree we need training data. To carry on the previous example: we need a number of clients for which we know the properties 1.-4. and their creditability.
The process of building a decision tree starts with the root node and involves the following steps:
1. Choose a splitting criteria and add it to the current node.
2. Split the dataset at the current node into those that fullfil the criteria and those that do not.
3. Add a child node for each data split.
4. For each child node decide on either A. or B.:
    1. Repeat from 1. step
    2. Make it a leaf node: The predicted class label is decided by the majority vote over the training data in the current split.

## 2.1 Gini Index & Data Split
Deciding on how to split your training data at each node is dominated by the following two criterias:
1. Does the rule help me make a final decision?
2. Is the rule general enough such that it applies not only to my training data, but also to new unseen examples?

When considering our previous example, splitting the clients by their handedness would not help us deciding on their creditability. Knowning if a rule will generalize is usually a hard call to make, but in practice we rely on [Occam's razor](https://en.wikipedia.org/wiki/Occam%27s_razor) principle. Thus the less rules we use, the better we believe it to generalize to previously unseen examples.

One way to measure the quality of a rule is by the [**Gini Index**](https://en.wikipedia.org/wiki/Gini_coefficient).
Since we only consider binary classification, it is calculated by:
$$
Gini = \sum_{n\in\{L,R\}}\frac{|S_n|}{|S|}\left( 1 - \sum_{c \in C} p_{S_n}(c)^2\right)\\
p_{S_n}(c) = \frac{|\{\mathbf{x}_{i}\in \mathbf{X}|y_{i} = c, i \in S_n\}|}{|S_n|}, n \in \{L, R\}
$$
with $|C|=2$ being your set of class labels and $S_L$ and $S_R$ the two splits determined by the splitting criteria.
While we only consider two class problems for decision trees, the method can also be applied when $|C|>2$.
The lower the gini score, the better the split. In the extreme case, where all class labels are the same in each split respectively, the gini index takes the value of $0$.

```python
def tree_gini_index(Y_left, Y_right, classes):
    """Compute the Gini Index.
    # Arguments
        Y_left: class labels of the data left set
            np.array of size `(n_objects, 1)`
        Y_right: class labels of the data right set
            np.array of size `(n_objects, 1)`
        classes: list of all class values
    # Output
        gini: scalar `float`
    """
    sets = [Y_left,Y_right]
    sets_gini = []

    for set_ in sets:
        set_fraction_of_whole = len(set_)/(len(Y_left)+len(Y_right))

        prob_pos = (set_.sum())/len(set_)
        sq_prob_pos = prob_pos**2
        prob_neg = 1 - prob_pos
        sq_prob_neg = prob_neg**2

        set_gini = (1 -(sq_prob_pos+sq_prob_neg))*set_fraction_of_whole
        sets_gini.append(set_gini)

    gini = np.sum(sets_gini)
    return gini
```

```python
am.test_student_function(username, tree_gini_index, ['Y_left', 'Y_right', 'classes'])
```

    Running local tests...
    tree_gini_index successfully passed local tests
    Running remote test...
    Test was successful. Congratulations!

At each node in the tree, the data is split according to a split criterion and each split is passed onto the left/right child respectively.
Implement the following function to return all rows in `X` and `Y` such that the left child gets all examples that are less than the split value and vice versa.

```python
def tree_split_data_left(X, Y, feature_index, split_value):
    """Split the data `X` and `Y`, at the feature indexed by `feature_index`.
    If the value is less than `split_value` then return it as part of the left group.

    # Arguments
        X: np.array of size `(n_objects, n_in)`
        Y: np.array of size `(n_objects, 1)`
        feature_index: index of the feature to split at
        split_value: value to split between
    # Output
        (XY_left): np.array of size `(n_objects_left, n_in + 1)`
    """

    ind = X[:, feature_index] < split_value
    print(ind) #outputs True/False values for whether cells meet that criteria
    X_left = X[ind]
    print(X_left)
    Y_left = Y[ind]
    print(Y_left)

    XY_left = np.concatenate([X_left, Y_left], axis=-1)
    return XY_left

#X[:, feature_index]

def tree_split_data_right(X, Y, feature_index, split_value):
    """Split the data `X` and `Y`, at the feature indexed by `feature_index`.
    If the value is greater or equal than `split_value` then return it as part of the right group.

    # Arguments
        X: np.array of size `(n_objects, n_in)`
        Y: np.array of size `(n_objects, 1)`
        feature_index: index of the feature to split at
        split_value: value to split between
    # Output
        (XY_left): np.array of size `(n_objects_left, n_in + 1)`
    """
    ind = X[:, feature_index] >= split_value
    X_right = X[ind]
    Y_right = Y[ind]

    XY_right = np.concatenate([X_right, Y_right], axis=-1)
    return XY_right
```

```python
am.test_student_function(username, tree_split_data_left, ['X', 'Y', 'feature_index', 'split_value'])
```

    Running local tests...
    [False False  True False  True False]
    [[-2.37556118 -2.60844971]
     [ 4.26463158 -1.65238926]]
    [[1.]
     [0.]]
    [False  True False  True False  True  True  True]
    [[ 1.94588175 -1.3034951 ]
     [-0.63283282 -2.97400198]
     [-1.7153846   0.74070777]
     [-5.79411975 -0.40943116]
     [-2.71633746 -1.64726742]]
    [[1.]
     [1.]
     [0.]
     [0.]
     [0.]]
    [False  True  True False False]
    [[ 1.35448839 -0.2099724 ]
     [ 7.77480498 -3.34950436]]
    [[0.]
     [1.]]
    [False  True  True False  True  True]
    [[ 0.87634404 -5.06310781]
     [-6.03751724  1.05850821]
     [-7.1926914  -6.50000432]
     [ 8.51710716 -4.21724322]]
    [[0.]
     [1.]
     [0.]
     [1.]]
    [ True  True  True  True  True False  True False  True]
    [[-9.36607929  3.73666788]
     [-7.78355718  6.5810372 ]
     [-6.11148523  7.90649653]
     [-9.72034328  3.36824612]
     [-8.94318009  0.07507696]
     [-1.63738251  0.47235638]
     [-9.03206994  8.00379159]]
    [[1.]
     [1.]
     [0.]
     [1.]
     [0.]
     [1.]
     [1.]]
    [False False False False False  True False  True]
    [[-5.26073174 -1.49986052]
     [-8.42770937  7.5915227 ]]
    [[1.]
     [1.]]
    [False False  True False  True False False False False]
    [[-9.10827565 -9.54824574]
     [-9.96200291  2.5183704 ]]
    [[1.]
     [0.]]
    [ True  True  True  True False False  True]
    [[ 2.65302163e+00  9.35304542e+00]
     [ 1.87233797e+00  3.45427003e+00]
     [-1.35570851e+00  3.76509218e+00]
     [-4.42484811e+00  7.86466080e-01]
     [-8.17879884e-03 -4.52946960e+00]]
    [[0.]
     [0.]
     [0.]
     [1.]
     [1.]]
    [False  True False  True  True]
    [[-4.37185642 -4.87225659]
     [-9.9353802   9.33881489]
     [ 5.04790996  6.30033027]]
    [[0.]
     [1.]
     [1.]]
    [ True False False False  True False  True  True]
    [[-7.28596597 -5.57235412]
     [-5.92654563  4.87763988]
     [-8.39325723 -7.21426803]
     [-2.37405564  4.65203543]]
    [[0.]
     [1.]
     [0.]
     [1.]]
    [False False False  True False  True  True False False]
    [[ 0.21113109 -8.82799123]
     [-2.61553065  9.23804143]
     [-1.78688084  9.37506723]]
    [[0.]
     [1.]
     [1.]]
    [False False False  True  True False False  True]
    [[ 6.31354453 -8.18307638]
     [-1.19612965 -9.87256673]
     [ 4.81831431 -3.75757405]]
    [[0.]
     [0.]
     [1.]]
    [ True  True  True  True  True False  True False False]
    [[ 4.44463904 -4.65117093]
     [-5.61413303 -6.88834569]
     [-1.95748231 -2.88653308]
     [ 3.70540254 -3.001772  ]
     [-4.390432   -3.67733266]
     [-3.36110821 -9.23287607]]
    [[1.]
     [1.]
     [0.]
     [0.]
     [0.]
     [1.]]
    [False  True  True  True  True  True  True False False]
    [[-5.77153704 -0.76438147]
     [ 5.00316785  0.118027  ]
     [ 1.26612951  2.20084614]
     [ 9.74702944  0.29475347]
     [ 2.03368119 -8.67896275]
     [ 3.61945011 -0.8396783 ]]
    [[1.]
     [0.]
     [0.]
     [1.]
     [0.]
     [0.]]
    [False  True  True  True  True  True False]
    [[ 0.58533874  1.6610768 ]
     [-8.35178568 -9.86183915]
     [-3.21501887  9.26737555]
     [ 0.74656272 -2.10365632]
     [ 1.54680626 -8.67605197]]
    [[0.]
     [0.]
     [1.]
     [1.]
     [1.]]
    [False False False  True False  True  True]
    [[-7.23122502  3.62082515]
     [-5.75659807  0.20713906]
     [-4.70347676 -2.66121154]]
    [[1.]
     [1.]
     [0.]]
    [False False  True False  True  True  True]
    [[ 1.38041332 -7.90304597]
     [-2.49852907 -1.09573896]
     [-2.30090537 -7.65231574]
     [-7.3139662  -0.81838309]]
    [[0.]
     [0.]
     [1.]
     [0.]]
    [ True False False  True False  True  True False]
    [[ 9.64559172 -9.00872846]
     [-0.05324152 -6.87058815]
     [-0.89077297 -6.65015207]
     [ 1.48407158 -8.26599853]]
    [[0.]
     [0.]
     [0.]
     [0.]]
    [ True False  True False  True  True  True]
    [[-7.3148247  -3.80817133]
     [-3.16647982 -8.34950305]
     [-3.44523248  3.47640827]
     [-5.52210163  7.6644342 ]
     [-0.34789753 -1.19014071]]
    [[1.]
     [1.]
     [1.]
     [1.]
     [0.]]
    [ True  True False  True False  True  True  True]
    [[ 8.98015199 -8.86684139]
     [ 7.09268214  5.29217057]
     [-4.45964554 -7.99198627]
     [-5.68531485 -3.64304732]
     [ 4.21270522  4.32527494]
     [ 4.7601182  -6.18756936]]
    [[0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]]
    [ True False False  True False  True False  True]
    [[ 6.66435037 -3.87849513]
     [-4.36138513 -2.43366449]
     [ 6.05228506  0.68493948]
     [-8.44336263  1.07215902]]
    [[1.]
     [0.]
     [1.]
     [1.]]
    [False False  True  True  True False False  True]
    [[ 5.84066412 -6.09674715]
     [ 6.04593831 -8.91448727]
     [ 5.94609503 -4.94677465]
     [-6.67897898 -4.14387699]]
    [[1.]
     [0.]
     [0.]
     [1.]]
    [ True  True  True False False]
    [[-2.25284259  0.98189791]
     [-7.90012135  1.90272944]
     [ 3.45246897  6.28514938]]
    [[0.]
     [0.]
     [0.]]
    [ True  True False False False False  True  True False]
    [[-2.91808204  0.03218935]
     [-4.44021341 -9.04475902]
     [-5.97246031 -1.54516754]
     [ 3.80843555 -2.55212295]]
    [[0.]
     [0.]
     [0.]
     [1.]]
    [False False  True  True  True  True]
    [[-2.5560739  -2.60592515]
     [-0.5263865  -6.42367064]
     [ 8.74711608 -8.6728326 ]
     [ 6.6724728  -4.36396978]]
    [[0.]
     [0.]
     [1.]
     [0.]]
    [False  True  True False  True]
    [[-6.96564368 -0.27322537]
     [-9.9980993  -7.84847771]
     [-7.27905356  0.74250288]]
    [[0.]
     [0.]
     [1.]]
    [ True False  True False False False]
    [[-0.38360825 -9.57747752]
     [-5.48225289  0.47083727]]
    [[1.]
     [0.]]
    [False False False  True False  True  True  True]
    [[-5.45927576 -4.54719023]
     [-5.4209009  -5.67326749]
     [-6.71441608 -3.85110805]
     [-2.87490704  4.41993717]]
    [[1.]
     [0.]
     [1.]
     [1.]]
    [ True False  True False False False False False]
    [[-4.90506179 -5.33466372]
     [-2.53472208  4.29138841]]
    [[0.]
     [0.]]
    [False  True  True  True False False False]
    [[-0.59299695 -3.2942172 ]
     [-5.44373928 -7.32666865]
     [-2.79712023 -6.83112458]]
    [[0.]
     [1.]
     [1.]]
    [ True False  True  True  True False  True]
    [[-3.68718498  4.09794462]
     [-2.31102569 -5.55203413]
     [-5.85750672 -5.59810463]
     [-3.9142254  -6.4061488 ]
     [-4.34621776  7.54341135]]
    [[1.]
     [1.]
     [0.]
     [1.]
     [1.]]
    [ True  True  True False  True  True False  True  True]
    [[ 2.00909198  9.35851907]
     [-8.77972521 -9.68834571]
     [-4.31129695 -9.7187875 ]
     [ 2.33509167 -4.27897808]
     [-6.56337944 -7.17549142]
     [-3.2334858  -2.20819085]
     [-8.63337174  7.00434138]]
    [[1.]
     [0.]
     [1.]
     [0.]
     [1.]
     [1.]
     [0.]]
    [False  True False  True  True]
    [[ 9.12897885 -4.36342744]
     [-0.34338788 -0.59414767]
     [ 0.43096634 -2.69557846]]
    [[0.]
     [0.]
     [0.]]
    [ True False  True False False]
    [[ 5.06496341 -6.77222192]
     [-1.74686603 -5.8720616 ]]
    [[1.]
     [0.]]
    [ True False False False False  True False  True  True]
    [[-8.28359946 -1.19299617]
     [-3.63763375  6.71175893]
     [-6.56600439  4.8218078 ]
     [-8.55821171  9.28315334]]
    [[1.]
     [1.]
     [1.]
     [0.]]
    [False  True  True  True  True  True False]
    [[-0.94286557 -4.08069731]
     [-2.73686439 -2.68371506]
     [-3.65253517  8.56299482]
     [-5.66654458  9.39629416]
     [ 0.59048795 -9.2088475 ]]
    [[0.]
     [1.]
     [1.]
     [1.]
     [0.]]
    [False False  True False  True]
    [[-9.48821184  4.82324189]
     [-8.62008202  0.49974124]]
    [[1.]
     [1.]]
    [False False  True False False  True]
    [[ 2.47058779 -5.91974156]
     [-4.10266291 -9.14134166]]
    [[0.]
     [0.]]
    [False False  True False False False False False  True]
    [[-3.72601191 -6.27172069]
     [ 8.78643858 -6.59278394]]
    [[0.]
     [1.]]
    [ True  True  True  True False False False]
    [[-4.72032533  9.18458634]
     [-1.61920226  9.05952796]
     [-8.06234418 -6.27702704]
     [-9.45923646  2.52972819]]
    [[0.]
     [1.]
     [0.]
     [0.]]
    [False  True  True  True False False]
    [[-0.45162584 -3.2335316 ]
     [-1.28831511 -6.84237011]
     [-0.15308094 -7.77799513]]
    [[1.]
     [0.]
     [1.]]
    [ True  True  True False False False False  True]
    [[-8.19713698 -3.63722183]
     [-7.97172673 -3.22334808]
     [-6.41521615 -1.18134938]
     [-9.80017337 -5.53944806]]
    [[0.]
     [1.]
     [1.]
     [1.]]
    [False False  True  True False False]
    [[-4.63433067 -8.81983161]
     [-3.84617438 -6.36269861]]
    [[0.]
     [1.]]
    [ True False  True False False False]
    [[-6.53397699 -2.79657874]
     [ 0.17705281 -4.60140242]]
    [[0.]
     [0.]]
    [False False False  True  True]
    [[-6.60118327  1.63110532]
     [ 4.68722109  0.81532126]]
    [[0.]
     [0.]]
    [False False False  True  True]
    [[ 8.9204833  -5.9258933 ]
     [-6.4066338  -1.57470596]]
    [[0.]
     [1.]]
    [False False False  True  True]
    [[-2.73179573 -4.19069926]
     [-3.41452237 -6.67316056]]
    [[1.]
     [1.]]
    [ True  True False  True  True False False  True]
    [[-3.4961022  -9.69119261]
     [ 9.85718459 -7.11974872]
     [-2.93010878 -6.38764807]
     [ 6.54003422 -8.22089358]
     [ 2.50436642 -5.72641979]]
    [[1.]
     [0.]
     [1.]
     [1.]
     [1.]]
    [ True False  True  True False False False False]
    [[ 2.04304953 -2.66691743]
     [ 2.11998695 -6.95996898]
     [ 7.39505486 -8.4452957 ]]
    [[0.]
     [0.]
     [0.]]
    [False  True  True False  True False  True False]
    [[-9.26440348  4.94472464]
     [-7.1319719  -2.45027692]
     [-0.96638402 -4.84270652]
     [-8.04817681 -8.98948527]]
    [[1.]
     [1.]
     [1.]
     [0.]]
    [ True  True False False  True]
    [[ 8.00383431 -9.87060869]
     [-1.6114016  -6.58899118]
     [-6.8729875  -4.4155576 ]]
    [[0.]
     [1.]
     [1.]]
    [ True  True  True False  True  True False  True False]
    [[-2.38212539 -3.02213617]
     [-4.57920761  2.91650932]
     [ 3.23766286  4.55335858]
     [-7.2921603   3.45951084]
     [-8.97675941  5.09661811]
     [ 0.03473972  7.26624062]]
    [[1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [0.]]
    [False  True  True False False]
    [[-8.38217673 -1.45517537]
     [-5.70689777 -5.04001407]]
    [[0.]
     [0.]]
    [False False False  True  True]
    [[-3.87393233  9.35165637]
     [-5.93469357  9.79240232]]
    [[0.]
     [1.]]
    [ True False  True  True False False]
    [[ 0.12488474 -8.71189933]
     [-7.53680306 -5.88116334]
     [-5.08998958  3.797166  ]]
    [[0.]
     [1.]
     [0.]]
    [ True False False  True  True False]
    [[-7.24691362  6.97272631]
     [-7.43552393  8.12300635]
     [-8.18648773  0.63600758]]
    [[0.]
     [0.]
     [1.]]
    [ True  True False  True  True False False  True]
    [[-5.85207908 -0.59122037]
     [ 1.61606968 -6.87914746]
     [ 2.77498938 -0.46787719]
     [-5.36533543 -8.65924707]
     [ 2.02037997 -4.76943075]]
    [[0.]
     [1.]
     [0.]
     [0.]
     [1.]]
    [False False  True  True False  True False  True  True]
    [[-6.33287127  9.53456732]
     [-7.10420299 -2.95333293]
     [-0.08443264  5.84655359]
     [-1.58121719  7.27356233]
     [-2.94707113 -1.84666023]]
    [[0.]
     [0.]
     [0.]
     [1.]
     [0.]]
    [ True  True  True  True  True False  True False False]
    [[-2.87398601 -8.99548391]
     [-3.7649271  -2.50289884]
     [-2.68129205 -0.01257792]
     [-4.62628445  6.03429738]
     [-7.49375538  2.40902251]
     [-7.18118428 -1.2023141 ]]
    [[0.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]]
    [ True  True False  True False  True False  True  True]
    [[-3.66970987 -4.54154617]
     [-0.04433954  2.65642204]
     [-8.0981359   0.87803743]
     [-4.76792146 -7.00028031]
     [-4.99180092 -5.35301201]
     [ 8.19457303  2.59606365]]
    [[1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]]
    [False False False False  True False  True]
    [[-4.84935715  6.7547206 ]
     [-8.01174707  8.93445182]]
    [[0.]
     [1.]]
    [ True False  True  True False]
    [[-5.50090019 -6.71965793]
     [-6.2562681   5.3800424 ]
     [-1.96759371  8.82984571]]
    [[0.]
     [0.]
     [0.]]
    [ True False False False  True  True False]
    [[ 1.74431094  6.95163073]
     [-3.82743019  6.64104529]
     [-0.02901706  8.48311679]]
    [[1.]
     [0.]
     [0.]]
    [ True False  True False  True False  True  True False]
    [[-7.58259261 -9.91977316]
     [-2.40194889  3.84509431]
     [-7.93730542 -4.5259871 ]
     [-1.96065237  7.60936619]
     [-6.99162731 -8.87091189]]
    [[1.]
     [1.]
     [0.]
     [0.]
     [1.]]
    [ True False False  True  True]
    [[-6.77785272 -6.54742583]
     [-0.34029011 -0.23541973]
     [ 2.78476849 -6.03841927]]
    [[1.]
     [0.]
     [0.]]
    [False False False  True False False  True]
    [[-3.96488507 -0.35617297]
     [-2.59579459 -6.02516011]]
    [[1.]
     [1.]]
    [False False  True False  True False False]
    [[ 8.58922241 -8.04297604]
     [-1.71552812 -6.81573813]]
    [[0.]
     [0.]]
    [False False False False  True  True  True  True]
    [[-9.63881272  4.00267662]
     [-5.03307598 -0.10750789]
     [-7.29440437 -5.83795654]
     [-9.36169689  8.10434713]]
    [[1.]
     [1.]
     [1.]
     [0.]]
    [ True  True  True False  True False]
    [[-2.50417291 -8.36174898]
     [-5.95913587 -9.5684707 ]
     [-1.05512041  7.00617975]
     [ 0.94759405  6.00070173]]
    [[0.]
     [0.]
     [0.]
     [1.]]
    [ True False False False False False False  True  True]
    [[-9.41290257  5.23155346]
     [-7.22303734  2.60855153]
     [-9.74225545  2.29896043]]
    [[0.]
     [0.]
     [0.]]
    [False False  True False  True  True]
    [[-6.55566772  4.81986286]
     [ 6.05011921 -4.6320378 ]
     [ 4.73741385  2.59345509]]
    [[0.]
     [1.]
     [1.]]
    [ True False False False  True]
    [[-6.78707299  2.57737964]
     [-3.22153838 -2.06931252]]
    [[1.]
     [0.]]
    [ True False  True False False  True False False False]
    [[-9.55202319 -9.94102357]
     [ 4.27491433 -7.78196405]
     [ 6.78600825 -3.51659557]]
    [[1.]
     [1.]
     [1.]]
    [False  True False False  True  True False]
    [[-0.93970228  4.40791271]
     [-4.41035194  4.65295999]
     [-7.3572601  -7.07251393]]
    [[0.]
     [1.]
     [0.]]
    [False False False  True  True  True  True False]
    [[ 2.02528737  3.01206302]
     [-2.39231441 -6.050712  ]
     [ 0.37863343  5.42265122]
     [-8.96346952 -7.6930434 ]]
    [[0.]
     [1.]
     [1.]
     [1.]]
    [False False  True  True False]
    [[-6.26624723 -5.16014257]
     [-6.70061154  1.29390345]]
    [[1.]
     [1.]]
    [False  True False  True False False False False False]
    [[-3.62149249 -6.92291217]
     [ 1.18257795 -9.63747597]]
    [[0.]
     [0.]]
    [False  True  True  True  True False False]
    [[-9.9347983  -9.22290294]
     [-3.43343251  9.22024277]
     [-7.54835663  9.39324371]
     [-3.00011844 -0.08409379]]
    [[0.]
     [1.]
     [1.]
     [0.]]
    [ True False False  True  True False]
    [[-8.19588147 -9.37485802]
     [ 6.07354263 -3.1779417 ]
     [-5.18229015 -9.53823914]]
    [[1.]
     [1.]
     [1.]]
    [ True False  True False False  True False  True False]
    [[ 0.08052513  1.27516653]
     [-7.28213357 -7.056836  ]
     [-7.03800875 -5.10133603]
     [-6.08436142  5.36328988]]
    [[1.]
     [1.]
     [0.]
     [0.]]
    [ True  True False  True False]
    [[ 9.83764274  2.46952373]
     [-0.06915623 -0.76587765]
     [-8.95868384 -9.99781408]]
    [[1.]
     [0.]
     [0.]]
    [False  True False False  True]
    [[ 4.72360431 -4.55592201]
     [-1.97286413 -3.08672912]]
    [[0.]
     [1.]]
    [False False  True False  True]
    [[ 1.43576384 -8.99474414]
     [-3.36516254 -6.4934647 ]]
    [[1.]
     [0.]]
    [ True False  True  True False]
    [[-6.10484594  1.90217048]
     [-2.1729519   3.07583054]
     [-3.56746362 -3.86210911]]
    [[0.]
     [0.]
     [1.]]
    [False False  True  True False  True]
    [[-4.54583766  4.53405796]
     [-1.94975453  4.7680293 ]
     [-4.19862534  7.16475208]]
    [[1.]
     [1.]
     [1.]]
    [ True False False False False False  True  True]
    [[-0.51425963  8.2213693 ]
     [-1.955815   -4.00124272]
     [-0.33892306  6.15880467]]
    [[1.]
     [0.]
     [0.]]
    [ True False  True False  True]
    [[-0.69470219  0.58731506]
     [-5.9652306   0.67996546]
     [-8.07856453  9.02889533]]
    [[1.]
     [1.]
     [0.]]
    [False  True False  True False]
    [[-3.44882369 -5.31594267]
     [-0.60422171  8.37542776]]
    [[1.]
     [1.]]
    [False False  True  True False  True]
    [[-2.80057539 -5.55489229]
     [-5.040751   -1.55424598]
     [-7.60086867 -2.29169235]]
    [[0.]
     [0.]
     [0.]]
    [False  True  True False False  True False False]
    [[-0.60327182 -9.94776048]
     [-2.41385399  2.91112426]
     [ 0.97285143 -7.40061708]]
    [[1.]
     [0.]
     [1.]]
    [False  True  True  True False  True]
    [[ 0.48706579 -1.61110782]
     [-5.25910094 -9.62965388]
     [-9.09885416 -2.8767827 ]
     [ 8.43197642  0.65359807]]
    [[1.]
     [0.]
     [1.]
     [1.]]
    [ True False  True  True False False  True False False]
    [[-1.19738089  4.57571159]
     [-6.64194726 -5.56365335]
     [-0.94392088 -0.88370692]
     [-3.6861174  -0.42453922]]
    [[1.]
     [1.]
     [1.]
     [1.]]
    [ True False False  True False False]
    [[-4.71721731 -1.09871374]
     [-6.21443348 -2.84728111]]
    [[1.]
     [0.]]
    [False  True  True  True  True False]
    [[ 2.92732003 -1.88865707]
     [-1.60631255 -1.22566077]
     [-4.06649003 -7.70954635]
     [ 6.69107814  0.54028227]]
    [[0.]
     [0.]
     [0.]
     [0.]]
    [ True  True False False  True]
    [[-3.00140948 -6.41315406]
     [ 5.13893193 -7.60535297]
     [-3.70337221 -1.42053089]]
    [[0.]
     [0.]
     [1.]]
    [False  True False  True  True]
    [[-7.62775463 -0.13606186]
     [-5.93502138 -5.20419842]
     [ 9.3924202  -8.03294589]]
    [[0.]
     [0.]
     [0.]]
    [False False False False  True  True]
    [[-9.61449335 -4.18374342]
     [-5.79933452  7.52870301]]
    [[1.]
     [0.]]
    [False False  True False  True False  True  True]
    [[-9.20678831  1.74767156]
     [ 0.78580093 -5.35266754]
     [ 5.13187746 -6.4307061 ]
     [ 2.81195689 -8.06716307]]
    [[0.]
     [1.]
     [0.]
     [0.]]
    [ True False False  True  True False False]
    [[-6.37269645 -2.28711518]
     [-9.89165526 -0.6289442 ]
     [ 2.49739494 -5.01484464]]
    [[1.]
     [0.]
     [0.]]
    [ True  True False  True  True  True  True False  True]
    [[ 1.30252665  1.41473139]
     [-9.16038639  3.48248068]
     [ 8.72096919 -8.20797439]
     [ 0.072007   -7.98098299]
     [ 2.89977394 -6.74748526]
     [-8.01589773 -0.61247194]
     [-0.53524827  2.88008634]]
    [[0.]
     [0.]
     [0.]
     [0.]
     [0.]
     [1.]
     [0.]]
    tree_split_data_left successfully passed local tests
    Running remote test...
    [False False  True  True False False]
    [[-3.61351138 -2.62259218]
     [-8.43132439 -5.70585008]]
    [[0.]
     [0.]]
    Test was successful. Congratulations!

```python
am.test_student_function(username, tree_split_data_right, ['X', 'Y', 'feature_index', 'split_value'])
```

    Running local tests...
    tree_split_data_right successfully passed local tests
    Running remote test...
    Test was successful. Congratulations!

```python
am.get_progress(username)
```

    ---------------------------------------------
    | Magdalena Mladenova                       |
    | magdalena.mladenova@student.uva.nl        |
    ---------------------------------------------
    | linear_forward           | completed      |
    | linear_grad_W            | completed      |
    | linear_grad_b            | completed      |
    | nll_forward              | completed      |
    | nll_grad_input           | not attempted  |
    | sigmoid_forward          | completed      |
    | sigmoid_grad_input       | completed      |
    | tree_gini_index          | completed      |
    | tree_split_data_left     | completed      |
    | tree_split_data_right    | completed      |
    | tree_to_terminal         | not attempted  |
    ---------------------------------------------

Now to find the split rule with the lowest gini score, we brute-force search over all features and values to split by.

```python
def tree_best_split(X, Y):
    class_values = list(set(Y.flatten().tolist()))
    r_index, r_value, r_score =  float("inf"),  float("inf"), float("inf")
    r_XY_left, r_XY_right = (X,Y), (X,Y)
    for feature_index in range(X.shape[1]):
        for row in X:
            XY_left = tree_split_data_left(X, Y, feature_index, row[feature_index])
            XY_right = tree_split_data_right(X, Y, feature_index, row[feature_index])
            XY_left, XY_right = (XY_left[:,:-1], XY_left[:,-1:]), (XY_right[:,:-1], XY_right[:,-1:])
            gini = tree_gini_index(XY_left[1], XY_right[1], class_values)
            if gini < r_score:
                r_index, r_value, r_score = feature_index, row[feature_index], gini
                r_XY_left, r_XY_right = XY_left, XY_right
    return {'index':r_index, 'value':r_value, 'XY_left': r_XY_left, 'XY_right':r_XY_right}
```

## 2.2 Terminal Node
The leaf nodes predict the label of an unseen example, by taking a majority vote over all training class labels in that node.

```python
def tree_to_terminal(Y):
    """The most frequent class label, out of the data points belonging to the leaf node,
    is selected as the predicted class.

    # Arguments
        Y: np.array of size `(n_objects)`

    # Output
        label: most frequent label of `Y.dtype`
    """
    class_names, counts = np.unique(Y, return_counts= True)
    print(class_names, counts)

    ind=np.argmax(counts)
    label = class_names[ind]

    return label
```

```python
am.test_student_function(username, tree_to_terminal, ['Y'])
```

    Running local tests...
    [0. 1.] [2 7]
    [0. 1.] [7 6]
    [0. 1.] [12  3]
    [0. 1.] [6 5]
    [1.] [5]
    [0. 1.] [12  9]
    [0. 1.] [9 4]
    [0. 1.] [8 9]
    [0. 1.] [4 3]
    [0. 1.] [11 10]
    [0. 1.] [3 6]
    [0. 1.] [ 8 11]
    [0. 1.] [7 2]
    [0. 1.] [2 5]
    [0. 1.] [4 7]
    [0. 1.] [11  8]
    [0. 1.] [5 6]
    [0. 1.] [10  9]
    [0. 1.] [11 12]
    [0. 1.] [11 12]
    [0. 1.] [7 8]
    [0. 1.] [12  7]
    [0. 1.] [4 3]
    [0. 1.] [9 6]
    [0. 1.] [5 6]
    [0. 1.] [6 3]
    [0. 1.] [10  5]
    [0. 1.] [3 2]
    [0. 1.] [5 4]
    [0. 1.] [4 9]
    [1.] [5]
    [0. 1.] [ 7 10]
    [0. 1.] [9 6]
    [0. 1.] [7 4]
    [0. 1.] [11 10]
    [0. 1.] [10  7]
    [0. 1.] [8 3]
    [0. 1.] [4 5]
    [0. 1.] [3 8]
    [0. 1.] [9 8]
    [0. 1.] [10  5]
    [0. 1.] [ 7 10]
    [0. 1.] [12  9]
    [0. 1.] [9 8]
    [0. 1.] [ 6 11]
    [0. 1.] [15 10]
    [0. 1.] [4 5]
    [0. 1.] [10  5]
    [0. 1.] [12  5]
    [0. 1.] [10  3]
    [0. 1.] [1 6]
    [0. 1.] [13  8]
    [0. 1.] [ 8 13]
    [0. 1.] [5 6]
    [0. 1.] [ 9 12]
    [0. 1.] [3 4]
    [0. 1.] [5 4]
    [0. 1.] [8 7]
    [0. 1.] [12  7]
    [0. 1.] [3 4]
    [0. 1.] [10 11]
    [0. 1.] [ 7 10]
    [0. 1.] [15  4]
    [0. 1.] [3 2]
    [0. 1.] [8 5]
    [0. 1.] [13  6]
    [0. 1.] [8 7]
    [0. 1.] [16  9]
    [0. 1.] [3 2]
    [0. 1.] [ 3 14]
    [0. 1.] [7 8]
    [0. 1.] [3 6]
    [0. 1.] [4 9]
    [0. 1.] [12  9]
    [0. 1.] [8 5]
    [0. 1.] [3 6]
    [0. 1.] [10 15]
    [0. 1.] [17  6]
    [0. 1.] [6 9]
    [0. 1.] [13  4]
    [0. 1.] [9 4]
    [0. 1.] [5 8]
    [0. 1.] [4 5]
    [0. 1.] [ 5 12]
    [0. 1.] [6 3]
    [0. 1.] [7 6]
    [0. 1.] [5 6]
    [0. 1.] [7 4]
    [0. 1.] [12 13]
    [0. 1.] [ 8 11]
    [0. 1.] [12 11]
    [0. 1.] [13 10]
    [0. 1.] [14 11]
    [0. 1.] [11  8]
    [0. 1.] [3 4]
    [0. 1.] [4 5]
    [0. 1.] [7 6]
    [0. 1.] [4 5]
    [0. 1.] [9 6]
    [0. 1.] [6 3]
    tree_to_terminal successfully passed local tests
    Running remote test...
    [0. 1.] [2 7]
    Test was successful. Congratulations!

```python
am.get_progress(username)
```

## 2.3 Build the Decision Tree
Now we recursively build the decision tree, by greedily splitting the data at each node according to the gini index.
To prevent the model from overfitting, we transform a node into a terminal/leaf node, if:
1. a maximum depth is reached.
2. the node does not reach a minimum number of training samples.

```python
def tree_recursive_split(X, Y, node, max_depth, min_size, depth):
    XY_left, XY_right = node['XY_left'], node['XY_right']
    del(node['XY_left'])
    del(node['XY_right'])
    # check for a no split
    if XY_left[0].size <= 0 or XY_right[0].size <= 0:
        node['left_child'] = node['right_child'] = tree_to_terminal(np.concatenate((XY_left[1], XY_right[1])))
        return
    # check for max depth
    if depth >= max_depth:
        node['left_child'], node['right_child'] = tree_to_terminal(XY_left[1]), tree_to_terminal(XY_right[1])
        return
    # process left child
    if XY_left[0].shape[0] <= min_size:
        node['left_child'] = tree_to_terminal(XY_left[1])
    else:
        node['left_child'] = tree_best_split(*XY_left)
        tree_recursive_split(X, Y, node['left_child'], max_depth, min_size, depth+1)
    # process right child
    if XY_right[0].shape[0] <= min_size:
        node['right_child'] = tree_to_terminal(XY_right[1])
    else:
        node['right_child'] = tree_best_split(*XY_right)
        tree_recursive_split(X, Y, node['right_child'], max_depth, min_size, depth+1)

def build_tree(X, Y, max_depth, min_size):
    root = tree_best_split(X, Y)
    tree_recursive_split(X, Y, root, max_depth, min_size, 1)
    return root
```

By printing the split criteria or the predicted class at each node, we can visualise the decising making process.
Both the tree and a a prediction can be implemented recursively, by going from the root to a leaf node.

```python
def print_tree(node, depth=0):
    if isinstance(node, dict):
        print('%s[X%d < %.3f]' % ((depth*' ', (node['index']+1), node['value'])))
        print_tree(node['left_child'], depth+1)
        print_tree(node['right_child'], depth+1)
    else:
        print('%s[%s]' % ((depth*' ', node)))

def tree_predict_single(x, node):
    if isinstance(node, dict):
        if x[node['index']] < node['value']:
            return tree_predict_single(x, node['left_child'])
        else:
            return tree_predict_single(x, node['right_child'])

    return node

def tree_predict_multi(X, node):
    Y = np.array([tree_predict_single(row, node) for row in X])
    return Y[:, None]  # size: (n_object,) -> (n_object, 1)
```

Let's test our decision tree model on some toy data.

```python
X_train, Y_train, X_test, Y_test = split(*generate_2_circles(), 0.7)

tree = build_tree(X_train, Y_train, 4, 1)
Y_pred = tree_predict_multi(X_test, tree)
test_accuracy = (Y_pred == Y_test).mean()
print('Test Acc: {:.1f}%'.format(test_accuracy * 100))
```

We print the decision tree in [pre-order](https://en.wikipedia.org/wiki/Tree_traversal#Pre-order_(NLR)).

```python
print_tree(tree)
```

```python
plot_model_prediction(lambda x: tree_predict_multi(x, tree), X_test, Y_test)
```

# 3. Experiments
The [Cleveland Heart Disease](https://archive.ics.uci.edu/ml/datasets/Heart+Disease) dataset aims at predicting the presence of heart disease based on other available medical information of the patient.

Although the whole database contains 76 attributes, we focus on the following 14:
1. Age: age in years
2. Sex:
    * 0 = female
    * 1 = male
3. Chest pain type:
    * 1 = typical angina
    * 2 = atypical angina
    * 3 = non-anginal pain
    * 4 = asymptomatic
4. Trestbps: resting blood pressure in mm Hg on admission to the hospital
5. Chol: serum cholestoral in mg/dl
6. Fasting blood sugar: > 120 mg/dl
    * 0 = false
    * 1 = true
7. Resting electrocardiographic results:
    * 0 = normal
    * 1 = having ST-T wave abnormality (T wave inversions and/or ST elevation or depression of > 0.05 mV)
    * 2 = showing probable or definite left ventricular hypertrophy by Estes' criteria
8. Thalach: maximum heart rate achieved
9. Exercise induced angina:
    * 0 = no
    * 1 = yes
10. Oldpeak: ST depression induced by exercise relative to rest
11. Slope: the slope of the peak exercise ST segment
    * 1 = upsloping
    * 2 = flat
    * 3 = downsloping
12. Ca: number of major vessels (0-3) colored by flourosopy
13. Thal:
    * 3 = normal
    * 6 = fixed defect
    * 7 = reversable defect
14. Target: diagnosis of heart disease (angiographic disease status)
    * 0 = < 50% diameter narrowing
    * 1 = > 50% diameter narrowing

The 14. attribute is the target variable that we would like to predict based on the rest.

We have prepared some helper functions to download and pre-process the data in `heart_disease_data.py`

```python
import heart_disease_data
```

```python
X, Y = heart_disease_data.download_and_preprocess()
X_train, Y_train, X_test, Y_test = split(X, Y, 0.7)
```

Let's have a look at some examples

```python
print(X_train[0:2])
print(Y_train[0:2])

# TODO feel free to explore more examples and see if you can predict the presence of a heart disease
```

## 3.1 Decision Tree for Heart Disease Prediction
Let's build a decision tree model on the training data and see how well it performs

```python
# TODO: you are free to make use of code that we provide in previous cells
# TODO: play around with different hyper parameters and see how these impact your performance

tree = build_tree(X_train, Y_train, 5, 4)
Y_pred = tree_predict_multi(X_test, tree)
test_accuracy = (Y_pred == Y_test).mean()
print('Test Acc: {:.1f}%'.format(test_accuracy * 100))
```

How did changing the hyper parameters affect the test performance? Usually hyper parameters are tuned using a hold-out [validation set](https://en.wikipedia.org/wiki/Training,_validation,_and_test_sets#Validation_dataset) instead of the test set.

## 3.2 Logistic Regression for Heart Disease Prediction

Instead of manually going through the data to find possible correlations, let's try training a logistic regression model on the data.

```python
# TODO: you are free to make use of code that we provide in previous cells
# TODO: play around with different hyper parameters and see how these impact your performance
```

How well did your model perform? Was it actually better then guessing? Let's look at the empirical mean of the target.

```python
Y_train.mean()
```

So what is the problem? Let's have a look at the learned parameters of our model.

```python
print(model.W, model.b)
```

If you trained sufficiently many steps you'll probably see how some weights are much larger than others. Have a look at what range the parameters were initialized and how much change we allow per step (learning rate). Compare this to the scale of the input features. Here an important concept arises, when we want to train on real world data:
[Feature Scaling](https://en.wikipedia.org/wiki/Feature_scaling).

Let's try applying it on our data and see how it affects our performance.

```python
# TODO: Rescale the input features and train again
```

Notice that we did not need any rescaling for the decision tree. Can you think of why?