# Jupyter Test Drive - Typesetting with Markdown

This notebook demonstrates the use of Markdown to type set documents in Jupyter. After working through this notebook, you will be able to edit the text in any notebook.  In addition, you will be able to assist your students in editing the text of their own notebooks.

## The _very_ Basics

Note that you can view the source code behind any cell by double clicking it. To re-render or re-run the cell, use any of these options:
1. Select `Cell` $\rightarrow$ `Run Cells`
2. Hit `Shift-Enter`
3. Click the `Run` button above

## Creating and Deleting Markdown Cells
The first step you may want to do is to create a new cell for typesetting. To do that, while you have this cell selected, from the top menu click the plus (+) sign next to the save button.

Once you have created the new cell, you will get a blank `Code` cell above. To change that cell into one that you can use for typesetting, select the cell, and from the dropdown menu at the top, change `Code` $\rightarrow$ `Markdown`. This changes that cell type into a markdown cell, and it can now be used for typesetting!

The next step will be learning how to delete cells. Select the cell above that you converted to a markdown cell. To remove this cell from your Jupyter notebook  click `Edit` $\rightarrow$ `Delete Cells`. If you accidentally deleted an incorrect cell, you can always bring it back by clicking `Edit` $\rightarrow$ `Undo Delete Cells`.

## Creating Titles

Creating large bold title text in a Jupyter notebook with markdown is very simple! All you need to do is on a new line, type the pound sign and then a space. After the space, add your title. It's that simple. By adding extra pound signs, you will create different title levels. The cell below is unformatted markdown text demonstrating this.
# I am a level one title
## I am a level two title
### I am a level three title
#### I am a level four title
##### I am a level five title
###### I am a level six title
####### I would be a level seven title if such a thing existed

If  # isn't the first character, that line does not get rendered as a title.
To see how those titles look when they're rendered in the Jupyter notebook, select the cell above, and from the top menu select `Raw NBConvert` $\rightarrow$ `Markdown` (to change the cell from a raw type to a Markdown type), then press `Ctrl + Enter` to render the cell. Feel free to create a new cell and play with those, or change  the one above.

## Creating Itemized Lists
Creating lists in markdown in a Jupyter notebook is also straight forward! To do so you simply need to start a new line and begin counting. The markdown syntax is shown in the code below.

#### Double Click Me

1. This is the first item of a list
1. This is the second
50. Notice how the numbers we use don't matter once it's rendered
    1. By pressing the `Tab` key we can add sub items
    45. The numbers don't matter here either
        - We can add many sub-lists
            - But at some point it starts to get redundant
        - We can also go back a few levels of nesting by lining the text up with the line above
2346. By going back to the original line, we leave our nested lists
23236235672456. The numbers really don't matter.
* In fact, you can give up on numbers at any time.
1. Or start counting again

- We also don't need to have numbers if we don't want to!
    * We have plenty of options
-Notice how if we don't put a space between the dash and the first term it doesn't become a new item in the list!

---
### Exercise 1
1. In the cell above, try fixing the last line so that the last item in the list becomes properly formatted once it's rendered.
2. Create a new cell below, add a title, and an ordered list of your five favourite fruits.
---

## Links
You can add hyper links to webpages anywhere you'd like right in the text. The syntax is as follows
```markdown
[text you want to display](link to a file or url page)
```
Change the cell below to markdown, run it, and see where the link takes you!
[this is the text that becomes the hyperlink](https://media.giphy.com/media/Vs44a88Dvbnkk/giphy.gif)
## Images/gifs
Images and gifs can be emebedded in a markdown cell nearly identically to placing a hyperlink. To display the image, add an exclamation point to the command as demonstrated below.
```markdown

```
would display an image/gif. It can either be a url link to an image, or one you have saved on the hub/locally. Change the cell below to markdown and run it to see it in action, or even embed your own images in this notebook.

---
### Exercise 2
1. In a new cell below, embed an image of a cat.
2. In the same cell, add a hyperlink below the image that links to the original image.
3. Change the text displayed in the hyperlink to say "Kitty Kitty".
---

# Block Quotes
Block quotes are a way to indent text/images with a grey bar in order to distinguish them from the main text. It is quite simlly done by adding a `>` at the beginning of a line. Change the cell below to markdown to see some examples
> This is an example of a block quote. You can type as long as you like and all the resulting text will be to the right of a the vertical grey bar.

You can also use block quotes to easily format images with captions!

>
>
> Now we can include a caption for images. This is nice because the image is now distinguishable from the main text. This makes it easy to include images without interrupting any narrative you've created. Notice how I have included a blank `>` under the image. That is so the spacing of this text looks nice underneath. Try removing it and see what happens
## Tables

One can also create tables of data in Markdown. However, doing this manually is often not the most convenient. For large tables we recommend using online tools (such as [here](https://jakebathman.github.io/Markdown-Table-Generator/)) where you can copy-paste excel tables or csv files. This will then output syntax to properly render those tables. For smaller tables, it's quite easy to type out your own. See the cell below.
| This is the first column name| Second Column| Third|
|---|---|-------------|
|Notice how nothing is lined up when we type it | But that it looks nice when rendered| how convenient!|
|shorter lines are good too| numbers?| 343453|
|symbols| `yes`| $\int e^{-x^2} dx$ |
|also notice that the number of dashes in the second line don't matter | it just needs to be at least three| the rest are really only for you to make pre-rendering look nice |
The basic table syntax is as follows:
```markdown
|initialize| your| column| names|
|----------|-----|-------|------| <- this line sets up the headers, need at least three `---` each line
|     data | data| data  | data | <- any kind of text data can be put in a table. Feel free to mix too!
| numbers  | text| text  | data   <- you don't need to include the right-most pipe if you don't want to
```

The syntax can be thought of as how you would try to write a table directly using symbols on your keyboard. Play with the table in the cell above, or create a new cell and try making your own! You could put one in this cell as well and put a table here. Double click anywhere on this cell to open edit mode.

---
### Exercise 3
1. Using sample notebooks, teachers may like or dislike various sections of the sample provided. Luckily, it's easy to make these minor changes in notebooks. In the two cells below:
    * remove the second cell because you won't need it in your lesson
    * remove the last paragraph in the first cell
    * change the date in the first paragraph
2. In the first cell below, take the ordered list out of the block quote.

---

> ## Properties of Random Numbers
>
> May 22, 2018
> Suppose you have a sequence of $N$ random numbers $\{R\}$ with contents $\{r_1, r_2, ... , r_N\}$ where each element $r_i$ is a random number. What sort of properties should this sequence of numbers have? If this is truly a sequence of random numbers, it _must_ satisfy the following properties, which we will explain in greater detail:
>
> 1. Drawing any $r_i$ is equally probable and independent.
> 2. The sequence of random numbers is uniformly distributed.
>
> "Drawing a value" in this scope means we're picking a number from our sequence of random numbers, but not removing it from the sequence (the sequence remains unchanged, we're simply "observing" the random number).
Let's look at these two properties in a little more detail.

> ### All Values Are Equally Probable and Independent
>
> What this means is that if you were to select (but not remove) a number from your sequence of random numbers $\{r_1, r_2, ... , r_N\}$ at random, the probability of drawing any of those numbers is
\begin{equation}
p(r_i) = \frac{1}{N}
\end{equation}
where $p(r_i)$ is the probability of selecting a number $r_i$. This probability is identical for all numbers within your set. More explicitly:
\begin{equation}
p(r_1) = p(r_2) = ... = p(r_N) = \frac{1}{N}
\end{equation}
>
> The independence property means that the if you draw a number from the set, it does not affect the probability of drawing other numbers, or even itself at a later time. This is because the sequence remains unchanged after you draw (observe) a number.


# "Learning To Efficiently See In The Dark"
> "This article describes the binarization of the network in the paper 'Learning to See in the Dark' [1] using the approach of the ABC-net presented in 'Towards Accurate Binary Convolutional Neural Network' [2]"

- toc: false
- branch: master
- badges: true
- comments: true
- categories: [fastpages, jupyter]
- hide_github_badge: true

### By Sieger Falkena, Anwesh Marwade and Joris Quist

With the ever growing amount of papers published, the field of Computer Vision by Deep Learning is more relevant than ever. Many new solutions are presented every day. Models get bigger and more complex. Together with computer vision, the market of smart-phone and smart-wearables is growing. Estimates have been made that in 2020, around 1.4 trillion photo's will be made worldwide from which 90.9% will originate from smartphones [4].

However, the differences in computing power between the two sectors is enormous. Deep learning models require powerfull GPU's to train and evaluate a model. Smartphones don't offer this. To accomodate that the two sectors are growing towards each other, two paths are being followed: smartphones are equipped with more powerfull hardware every year making that smartphones can offer more services. On the other hand, deep learning models are made more and more efficient, so that evaluation can be done on a wearable device.

In this article, we want to take a step towards the direction of efficient neural networks. We will combine methods from different active fields of study and present an efficient method to process low-light images. We showcase the implementation details of applying the concept of network binarization using "Accurate Binary Convolutional Network" (ABC-Net) in "Learning To See In The Dark" (LSID) [1] paradigm. In doing so, we want to find out whether the full accuracy network can be approximated by binarization. We aim for limited performance loss as compared to the full precision model while driving up the efficiency. We intend to highlight the theoretical speedup that can be achieved and shed light on the limitations that we faced.

**Click [https://sfalkena.github.io/blogs/_pages/interactive_image.html](https://sfalkena.github.io/blogs/_pages/interactive_image.html) or scan the QR code below to see an interactive comparison of the results in high resolution! Warning: the images are in high resolution, so the loading of images might be slow ;)**

# Techniques used

First, let us introduce the problem that we propose to tackle. This problem is presented by Chen et. al in "Learning to see in the dark" [1], where the authors take a raw image file (short-exposure images which are grainy and suffer from noise due to low-lighting and high ISO values) and use a deep neural network to learn the image-processing pipeline to process the low-light raw data. By working directly with raw sensor data, the authors intend to replace the standard imaging pipeline. Intending to achieve this, the authors train a convolutional network with a new dataset involving raw short-exposure low-light images with corresponding long-exposure images as the target reference or ground truth.

The pipeline suggested by the authors can be seen in the figure below. First, the raw image is unpacked, the black levels are subtracted after which the image is amplified. Finally, the image is fed through the CNN, and the output is an RGB image.

For examples of the results achieved by in the paper, look at https://cchen156.github.io/SID.html for an interactive comparison.

To make the ConvNet in the figure shown above more efficient, binarization is applied to the ConvNet. For this, we follow the fundamental techniques of binary convolution using XNOR and bitcount operations by Rastegari et al. in "XNOR-Net" [5]. In applying binarization, we explore two schemes for implementing a binary network namely, XNOR-Net and ABC-Net. In the following section, we first explain the XNOR-Net, after which we dive into the ABC-Net implementation in detail.

For the interested reader, we have written a chapter where we explain the basics of [Binary Networks](https://drive.google.com/file/d/1BnX25RuH6bDnD2rfNdJAfkTVJhA4irEl/view?usp=sharing) in more detail. For the rest of the paper, we assume that the reader has knowledge of this chapter and knows the fundamentals of binarization.

## XNOR-Net
The first binarization technique that we explored is the XNOR-Net implementation presented in [5]. In XNOR-Networks, both the filters and the input to the convolutional layers are binarized which allows these networks to approximate the convolutions primarily using binary operations. The efficiency and speedup can be achieved by leveraging the XNOR and bit-counting operations to estimate the convolutions.

We replace the LSID layers with the binarized convolutions which are mentioned in detail in the ABC-Net section. Reading further into the implementation we realised that ABC-Net is a generalisation of the XNOR-Net. Being a generalisation, the ABC-net extends better representational power by approximating the convolutions using multiple binarized weights and activations, but still has the option to evaluate the performance of the XNOR-Net when using only a single bit for both weights and activations.

## ABC-Net

We implemented this by using the follow-up work shown by Lin et al. [2] to achieve higher accuracy than XNOR-Net. Lin et al. developed a new type of convolutional network called the Accurate Binary Convolutional Network or ABC-Net. Explained very briefly, ABC-Net approximates the full precision convolution using $M\cdot N$ binary convolutions from XNOR-Nets, where $M$ is the number of bits used to approximate the weights and $N$ the number of bits to approximate the activations.

Again, in our chapter, [ABC-Net](https://drive.google.com/file/d/1BnX25RuH6bDnD2rfNdJAfkTVJhA4irEl/view?usp=sharing) is explained in more detail. However, we intend to provide some intuition regarding the ABC-Net as mentioned below.

The ABC-Net approximates each real-valued by using multiple binary values per weight, giving it better representational power. The approximation is achieved as follows:
\begin{equation}
W \approx \alpha_1\mathbf{B}_1 + \alpha_2\mathbf{B}_2 + ... + \alpha_M\mathbf{B}_M
\end{equation}

Finding a good value for $\mathbf{B}_i$ and $\alpha_i$ comes down to the following optimization problem:

\begin{equation}
\min\limits_{\mathbf{\alpha}, \mathbf{B}}J(\mathbf{\alpha}, \mathbf{B}) = ||\mathbf{w} - \mathbf{B} \mathbf{\alpha}||^2,\;\;\;\; s.t.\:\mathbf{B}_{ij}\in \{-1,+1\}
\end{equation}

Where $\mathbf{B} = [vec(\mathbf{B}_1), vec(\mathbf{B}_2), ..., vec(\mathbf{B}_M)]$, $\mathbf{w} = vec(\mathbf{W})$ and $\mathbf{\alpha} = [\alpha_1, \alpha_2, ..., \alpha_M]^T$. Trying to find the optimum to this problem numerically, makes it quite difficult as it is no longer possible to compute the derivative w.r.t. $\textbf{W}$ using the Straight-Through-Estimator [add reference]. Therefore the optimum is approximated by first calculating each $B_i$ in the following way:

\begin{equation}
\mathbf{B}_i = sign(\overline{\textbf{W}} + u_istd(\mathbf{W})), i = 1,2,...,M
\end{equation}

Where $\overline{\textbf{W}} = \textbf{W} - mean(\textbf{W})$  and $u_i$ is picked evenly over the the range $[-std(\mathbf{W}), std(\mathbf{W})]$. This is done because experimental observations show that real-valued weights often look like they are drawn from a Gaussian distribution.

Importantly, to be able to take full advantage of the more efficient XNOR and bitcount operations for the convolutions, the activations need to be binarized in addition to the network weights. The ABC-Net does this by using multiple bits per activation to get a better approximation of the original i.e. full precision activations. However, it is not desirable to use the same method for binarizing the activations as we do for the weights, because the alpha values (and the beta values for the activations) need to be calculated using linear regression. For the weights, this is not a big problem, since this only needs to happen when they need to be updated, which only happens during training. It thus has no impact on the inference speed. However, this is not the case for the activations since these always vary at test time.

The way they implemented this, was by first using batch normalization, so that the mean and standard deviation of the activation didn't need to be calculated, but could be assumed to be $0$ and $1$. They then replaced the linear regression that was used to calculate alpha by a learnable parameter beta. This way the there is no need anymore to do linear regression or calculate the mean and standard deviation during inference.

## Theoretical speedup
Before we dive into the speedup that would theoretically be possible, we need to understand more about the factors that influence this speedup. The speedup is achieved by binarizing the weights and activations of the convolution. For binarizing the weights, the full precision weights are approximated with $\boldsymbol{W} \approx \alpha_{1} \boldsymbol{B}_{1}+\alpha_{2} \boldsymbol{B}_{2}+\cdots+\alpha_{M} \boldsymbol{B}_{M}$. So $M$ BinConvs are used as can be seen on the left part of the figure above. Furthermore, the activations $R$ can also be binarized by approximating $\boldsymbol{R} \approx \beta_{1} A_{1}+\beta_{2} A_{2}+\cdots+\beta_{N}\boldsymbol{A}_{N}$.

According to [5] the speedup of using XNOR-Net is the ratio of the operations carried out by a normal convolution and the binarized XNOR convolution: $S=\frac{64 c N_{W}}{c N_{W} + 64}$, where $N_{\mathbf{W}}=w h$ (the number of entries in the weights) and $c$ is the number of channels. For justification of this equation, please refer to the original paper. The alert reader will notice that the speedup does not depend on the size of the input.

Because there are now $N\times M$ of these BinConvs, the speedup per convolutional layer linearly decreases with this number and becomes $S=\frac{64 c N_{W}}{M N (c N_{W} + 64)}$. To draw conclusions about the total speedup of the network, we have to look into the architecture and take the sum of all operations done over all layers when not using BinConvs and dividing this by the amount operations that are needed when BinConvs are being used. This will then give the theoretical speedup of the entire network. In the results section, this number will be shown for all the different networks used in the experiments.

Even when not binarizing the activations, still some speedup could be achieved. This is because when using binary weights there is no need to do multiplication and addition of the activation with weight anymore, it can be immediately added or subtracted. The speedup of this can be roughly estimated to be 2, since only half of the operations need to be done (ignoring overhead like adding the bias). The speedup is then given by $S \approx \frac{2}{M N}$, since again the speedup linearly decreases with the amount of times the BinConv is used. Note here that when using a value larger than $1$ for either $M$ or $N$, all speedup is immediately lost. However, there is at that point still the possibility of reduced memory usage.

# Technical constraints

In the introduction, as an alert reader must have noticed, we stress that we are aiming for limiting the performance loss rather than only aiming for increase in efficiency. The reason for this is the fact that binarization is still a fairly new field of study. To convert the theoretical speedup of the network into practice, special hardware or software needs to be used. The speedup is realized by using XNOR and bitcount operations [1]. Intel researchers in [6]  have compared running a binarized network on different architectures, and they have pointed out that specialised hardware like FPGAs have a greater potential for running binarized networks. For now, the platform that we use i.e. PyTorch does not offer possibilities to achieve and leverage this practical speedup to which end we feel that it is currently out of the scope of this project.

# Implementation of the code
In this section, certain relevant parts of the code will be highlighted.

##Architecture
Below, the architecture of the binarized LTSITD (ABCLSID) is shown. Some remarks about the architecture:
- The first and last layer are not binarized and uses normal convolution. The speedup of binarizing is smaller here because of the low amount of channels or small kernel size.
- Rest of the architectural decisions (channels, kernel size, etc.) are kept the same i.e. as per the original LTSITD implementation. The layer order from LTSITD adhered according to the layer order recommended by [5].

```python
class ABCLSID(nn.Module):
    def __init__(self, inchannel=4, block_size=2, M=3, N=None, binary_transposed_conv=True):
        super(ABCLSID, self).__init__()
        self.M = M
        self.N = N
        self.binary_transposed_conv = binary_transposed_conv

        self.block_size = block_size

        self.conv1_1 = nn.Conv2d(inchannel, 32, kernel_size=3, stride=1, padding=1, bias=True, )
        self.lrelu =  nn.LeakyReLU(negative_slope=0.2, inplace=True)
        self.conv1_2 = ABCConv2d(32, 32, kernel_size=3, stride=1, padding=1, bias=True, M=M)
        self.maxpool = nn.MaxPool2d(kernel_size=2, stride=2, padding=0, ceil_mode=True)

        self.conv2_1 = ABCConv2d(32, 64, kernel_size=3, stride=1, padding=1, bias=True, M=M)
        self.conv2_2 = ABCConv2d(64, 64, kernel_size=3, stride=1, padding=1, bias=True, M=M)

        self.conv3_1 = ABCConv2d(64, 128, kernel_size=3, stride=1, padding=1, bias=True, M=M)
        self.conv3_2 = ABCConv2d(128, 128, kernel_size=3, stride=1, padding=1, bias=True, M=M)

        self.conv4_1 = ABCConv2d(128, 256, kernel_size=3, stride=1, padding=1, bias=True, M=M)
        self.conv4_2 = ABCConv2d(256, 256, kernel_size=3, stride=1, padding=1, bias=True, M=M)

        self.conv5_1 = ABCConv2d(256, 512, kernel_size=3, stride=1, padding=1, bias=True, M=M)
        self.conv5_2 = ABCConv2d(512, 512, kernel_size=3, stride=1, padding=1, bias=True, M=M)

        if self.binary_transposed_conv:
            self.up6 = ABCConvTranspose2d(512, 256, 2, stride=2, bias=False, M=M)
        else:
            self.up6 = nn.ConvTranspose2d(512, 256, 2, stride=2, bias=False)

        self.conv6_1 = ABCConv2d(512, 256, kernel_size=3, stride=1, padding=1, bias=True, M=M)
        self.conv6_2 = ABCConv2d(256, 256, kernel_size=3, stride=1, padding=1, bias=True, M=M)

        if self.binary_transposed_conv:
            self.up7 = ABCConvTranspose2d(256, 128, 2, stride=2, bias=False, M=M)
        else:
            self.up7 = nn.ConvTranspose2d(256, 128, 2, stride=2, bias=False)

        self.conv7_1 = ABCConv2d(256, 128, kernel_size=3, stride=1, padding=1, bias=True, M=M)
        self.conv7_2 = ABCConv2d(128, 128, kernel_size=3, stride=1, padding=1, bias=True, M=M)

        if self.binary_transposed_conv:
            self.up8 = ABCConvTranspose2d(128, 64, 2, stride=2, bias=False, M=M)
        else:
            self.up8 = nn.ConvTranspose2d(128, 64, 2, stride=2, bias=False)

        self.conv8_1 = ABCConv2d(128, 64, kernel_size=3, stride=1, padding=1, bias=True, M=M)
        self.conv8_2 = ABCConv2d(64, 64, kernel_size=3, stride=1, padding=1, bias=True, M=M)

        if self.binary_transposed_conv:
            self.up9 = ABCConvTranspose2d(64, 32, 2, stride=2, bias=False, M=M)
        else:
            self.up9 = nn.ConvTranspose2d(64, 32, 2, stride=2, bias=False)

        self.conv9_1 = ABCConv2d(64, 32, kernel_size=3, stride=1, padding=1, bias=True, M=M)
        self.conv9_2 = ABCConv2d(32, 32, kernel_size=3, stride=1, padding=1, bias=True, M=M)

        out_channel = 3 * self.block_size * self.block_size
        self.conv10 = nn.Conv2d(32, out_channel, kernel_size=1, stride=1, padding=0, bias=True)

        for m in self.modules():
            if isinstance(m, nn.Conv2d):
                n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels
                m.weight.data.normal_(0, math.sqrt(2. / n))
                m.bias.data.zero_()
            elif isinstance(m, nn.ConvTranspose2d):
                n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels
                m.weight.data.normal_(0, math.sqrt(2. / n))
```

##Datasets and data processing
In the section "Techniques used", we already mentioned briefly that the authors of [1] created a new dataset to train and evaluate the model. In this section, we want to explain a bit more about the importance of the dataset.

Let's start with explaining what kind of data we are we dealing with. The authors have created a dataset with low-light images. The original dataset has been shot with two cameras: A Sony $\alpha$7S II and a Fujifilm X-T2. The reason behind using two different cameras has to do with the different ways in which the images get stored. The Sony uses a Bayer filter, whereas the Fujifilm uses an X-Trans filter. We leave the exploration of the X-Trans filter to the reader as we have chosen to focus solely on the Sony camera for this project endeavour. The reason for this is that Sony is the leading company in the smartphone camera domain and holds [50.1%](https://www.bloomberg.com/news/articles/2019-12-23/sony-can-t-make-image-sensors-fast-enough-to-keep-up-with-demand) of the market share as of 2019. As mentioned above, the Sony images use a Bayer filter. We would suggest reading [this blog](https://www.cambridgeincolour.com/tutorials/camera-sensors.htm) to develop a better understanding about Bayer filters. The first image of our blog shows the input image being in its Bayer form. The de-mosaicing step in the aforementioned blog is included in the ConvNet pipeline.

So, going back to the dataset, it contains groups of images shot in exactly the same way, but with different exposure times, meaning that we have bright, ground truth images, together with dark, low-exposure images. To test the rigidity of the network, images with exposure times of $0.033s, 0.04s, 0.1s$ have been captured.

The said raw images are of size 4256x2848. However, these images are still in the Bayer form, which means that they need to be unpacked into 2128x1424 sized images with 4 colour channels. Additionally, these images are considerably big to be loaded into memory as a whole. This memory contraint meant that it was not possible to perform backpropagation with the entire image as input, because then our GPU would run out of memory (8 GiB). This, and the fact that the authors of LSID also train using patches, is why we train the network on patches of the images. The patch size used is 512x512 pixels.

Another problem was that, the training was very slow. At first it would have taken around 6 days to train the network. The reason for this was the way the images were loaded and processed. The way it was done, was that in each iteration, an image was loaded from the disk, processing was done (by the python rawpy module) and then a random patch was taken from that image. The bottleneck at that point was the loading of the images from disk to memory. Even with 8 data loading workers that simultaneously loaded images, it wasn't enough. Furthermore, using more workers caused overheads in other places and hence did not improve performance. The way the authors overcame this problem was by keeping all the data in memory. The problem for us, however, was that the dataset was 64 Gigabytes and we only had 16 Gigabytes of RAM storage. The way we fixed this issue was by preprocessing the data. We first loaded the images, did the processing, but then instead of taking a random patch, we took 15 patches in a grid of 3 by 5. This was the smallest number of patches that together (with a bit of overlap) covered the entire image. We then stored these patches. By performaing the mentioned steps, during training, we could directly load the patch instead of the entire image. This improved our training speed significantly, which meant that we could train the network in just 22 hours which was quite a considerable reduction in time.
This method does have some disadvantages, because this way we always use the same set of patches instead of random ones, which could, in theory, affect the result after training. However we still achieve highly comparable results on the test set as the original model. This led us to conclude that it probably did not matter much.
Another problem that we dealt with was that the raw images had an 14-bit encoding, which after processing gets converted to 32-bit tensors. Plus, the patches were slightly overlapping. Together this caused our newly processed training set to be 353GB, which is not very practical to store.
In the end our epochs were changed from taking a random patch of the 1865 images, to using $15\cdot1865=27975$ fixed patches.

## Binary convolution
One problem we encountered during the project was because we did not have access to efficient implementations of the binary convolutions implying that we had to use the standard convolutions for 32-bit Floating-point numbers. This meant that instead of a getting a speedup as mentioned in Section "Theoretical speedup", it took about $M\cdot N$ times longer to do a forward pass, since that is the amount of convolutions we now had to do in each layer. This would mean that training would again take too long. To remedy this, we slightly deviated from the procedure that is described in the ABC-Net paper. We still calculated the binary tensors and their multipliers in the same way, but instead of using them in separate convolutions, we used them to approximate the original weights again. In theory this should give the same results. In practice, both these methods had small differences in their outputs. This is probably due to floating-point errors. These errors, however, were minute enough to not impact a real difference during training or in performance. In the code snippet below, we first have two functions implementing the shift operation as in equation 2 from [2].

```python
import torch.nn as nn
from torch import Tensor
import torch

def shift_parameter_binarization(tensor, shift_parameters, empty_cache):
    if empty_cache:
        torch.cuda.empty_cache()

    return torch.cat(
        [((tensor - tensor.mean()) + shift_parameter).sign() for shift_parameter in shift_parameters])

def shift_parameter_binarization_activation(tensor, shift_parameters, empty_cache):
    if empty_cache:
        torch.cuda.empty_cache()

    return torch.cat(
        [((tensor - tensor.mean()) + shift_parameter).sign() for shift_parameter in shift_parameters])

```

### Binary 2D convolution

```python
class ABCConv2d(nn.Conv2d):

    def __init__(self, in_channels, out_channels, kernel_size, M=3, N=None, estimated_weights=True, stride=1,
                 padding=0, dilation=1, groups=1, bias=True, padding_mode='zeros'):

        super(ABCConv2d, self).__init__(in_channels, out_channels, kernel_size, stride=stride,
                                        padding=padding, dilation=dilation, groups=groups,
                                        bias=bias, padding_mode=padding_mode)

        self.M = M
        self.N = N

        self.use_estimated_tensors = estimated_weights

    # Calculates the alphas based on the binarized and real weights using linear regression
    def get_alphas(self, B):
        vectorized_B = B.view(self.M, -1).t()

        return vectorized_B.t().mm(vectorized_B).inverse().mm(vectorized_B.t()).mv(self.weight.data.view(-1))

    # Binarize the weights in M matrices
    def get_B(self):
        weights = self.weight.data

        return shift_parameter_binarization(weights, [(-1 + i * 2 / (self.M - 1)) * weights.std() for i in
                                                      range(self.M)], empty_cache=(not self.training))

    # Binarize the weights in M matrices
    def get_binary_activation(self, input):
        binary_input = shift_parameter_binarization_activation(input,
                                                               [(-1 + i * 2 / (self.N - 1)) * input.std() for i in
                                                                range(self.N)],
                                                               empty_cache=(not self.training))

        vectorized_binary_input = binary_input.view(self.N, -1).t()
        betas = vectorized_binary_input.t().mm(vectorized_binary_input).pinverse().mm(vectorized_binary_input.t()).mv(
                input.view(-1))

        estimated_binary_input = torch.mv(vectorized_binary_input, betas)
        # del binary_input
        return estimated_binary_input.view(input.shape)

    def get_estimated_weights(self, B, alpha):
        vectorized_B = B.view(self.M, -1).t()
        estimated_weights = torch.mv(vectorized_B, alpha)
        return estimated_weights.view(self.weight.shape)

    def forward(self, x: Tensor) -> Tensor:
        # if self.training or self.B is None or self.alpha is None:
        B = self.get_B()
        alphas = self.get_alphas(B)

        if self.N:
            x.data = self.get_binary_activation(x)

        if self.use_estimated_tensors:
            x = nn.functional.conv2d(x, self.get_estimated_weights(B, alphas), self.bias, self.stride,
                                     self.padding, self.dilation, self.groups)

        else:
            x = sum(alphas[i] * nn.functional.conv2d(x, B[i],
                                                     stride=self.stride,
                                                     padding=self.padding,
                                                     dilation=self.dilation,
                                                     groups=self.groups) for i in range(self.M))
            if self.bias:
                x += self.bias[None, ..., None, None].repeat(1, 1, x.shape[2], x.shape[3])

        return x
```

### Binary 2D transposed convolution
For the transposed convolution, we have not yet seen any implementation online, so we have estimated that it can be done in a similar way as the normal binary 2D convolution.

## Initialization

A very useful aspect of ABC-Net is the fact that it uses multiple bits to approximate the original real-valued weights. This means that if you have a trained model and you want to replace the normal convolutions with the approximated convolutions of ABC-Net, you can use the weights of the original model as the initialization of the new one. Some training is still needed to finetune the weights to the new architecture, but much less than with random initialization. Once we implemented this, we only trained each network for 1 epoch in our testing. This helped with testing and debugging the network, since when using binarization for each layer in the network the linear regression had to be performed every iteration. This made the training 2 to 12 times slower, depending on the number of bits used for the binarization. This would have meant that without this initialization some networks would have needed to train for 12 days, which wouldn't be feasible for this project.

## Bias
The ABC-Net paper makes no mention of using biases in their convolutions. However as biases were used by the LSID network, we did need to implement this. Since biases only account for a small part of the memory used and the computation needed, we decided to not binarize the biases in the network.

## Normalization

As mentioned in section "ABC-Net", one of the things that is needed to be able to binarize the activations is normalization. In ABC-Net this is done using batch normalization. However, this was not possible in our case since the GPU we used, did not have enough memory. When only binarizing the weights, batches of 8 could be used, which is already very small for batch normalization to work well. When we also binarized the activations even more memory was needed, since the binarized tensors still needed to be stored as 32-bit floats. This meant that we could only use a batch size of 1. Therefore we looked into 3 other types of normalization, namely, instance normalization, group normalization and layer normalization. Unfortunately, all of the normalization techniques hurt the performance of the model, with really low PSNR's even when not even binarizing the activations yet. We speculate that this might have to do with the fact that we use the weights of the trained LSID model as initialization. Adding normalization in can cause the activations to change significantly, which means that the weights also need to adapt.

## Activation Binarization
Since we couldn't get normalization to work and the binarization of the activations in ABC-Net depend on this, we needed to find a new way to do this. In the end we chose to do it the same way as with the binarization of the weights. By picking the thresholds for binarizing equally distributed from minus the standard deviation to plus the standard deviation around the mean of the activation. After which the betas are calculated using linear regression again. This cannot lead to good efficiency during inference as the linear regression still has to be done at each layer every iteration. We thought this is an acceptable compromise, since, without the efficient operators, we couldn't show the actual efficiency benefits in practice anyway and this should still give comparable results in terms of accuracy.

## Training details
To improve the reproducibility of the project, we will provide some details about our training process. We use the weight of the model trained by the authors as initialization. Then we train the model for 1 epoch using the 27975 image patches of size 512 by 512. The learning rate we used was $10e-4$. We tested multiple different combinations of $M$, $N$ (real-valued, $1$, $3$ and $5$).

# Results
The results are shown in the table below and the visual representation can be seen by running the following code cell. When a cell in the table contains a "x", that means the specific part has not been binarized and thus contains the full-precision values.
Just as a reminder: $M$ are the amount of bits used for the weight approximation, $N$ is the amount of bits used for approximating the activations. The theoretical speedup has been calculated per setting as described earlier in this blog.

|                               | M | N | PSNR  | Theoretical speedup | Memory Reduction       |
|-------------------------------|---|---|-------|:-------------------:|:------:|
| Results by [1], in TensorFlow | x | x | 28.88 | 1x                  | 1x     |
| Results of [3], in PyTorch    | x | x | 28.55 | 1x                  | 1x     |
| ABCLSID                       | 3 | x | 27.6  | 0.66x               | 10.67x |
| ABCLSID                       | 1 | x | 25.70 | 2x                  | 32x    |
| ABCLSID                       | 1 | 1 | 20.21 | 55.11x              | 32x    |
| ABCLSID                       | 3 | 3 | 20.93 | 6.77x               | 10.67x |
| ABCLSID                       | 5 | 3 | 21.19 | 4.08x               | 6.4x   |
| ABCLSID                       | 3 | 5 | 21.74 | 4.08x               | 10.67x |
| ABCLSID                       | 5 | 5 | 21.86 | 2.46x               | 6.4x   |

```python
from IPython.display import Image
from google.colab import widgets
from matplotlib import pylab

tb = widgets.TabBar(["M=1, N=x"                         ,"M=1, N=1",                         "M=3, N=x"                         ,"M=3, N=3"                         ,"M=5, N=3"                         ,"M=5, N=5"])
image_ids =         ["1XZuoIFNmI65Wuhnv4lEVDMP5YvVfy2fV","1Ypb9LUoJt5V44qu9zHXobbNwoa4NOZcn","1ihb4AvT8Jjj8g5N0zfLrss7-Bms0Vr5g","1niA76Jwfp6cIdWmnv70omRFBtAaut1Ub","1CK_rFLYvjPqkw2j4RIng8grU6ovAATWc","1BN66SlR4UVNOn09txj-9RQKpC-aKtVPX" ]

for t, i in enumerate(tb):
    display(Image(url="https://docs.google.com/u/0/uc?id="+image_ids[i], width = 1000, height = 700))
```

# Discussion
When looking at the results in the table and the images, you can see that the network still performs quite well when only the weights are binarized, the difference between $M = 3$ with real valued activations is almost indistinguishable from the original model (even when looking at the direct interactive comparison). Even using 1-bit $(M = 1)$ the results are still passable considering 32x less memory is needed. However, when we start to binarize the activations, the performance of the network starts to drop significantly. Even the network with real-valued activations and 1-bit weights performs much better than the network with 5-bit activations and 5-bit weights. We are not exactly sure about the reason for this. It could be that binarizing the activations simply causes too much loss of information for the network to function properly. For example in the images (refer to images with binary activations), you can see that a lot of the colour of the image is lost. Especially for $N=1$, there the image has almost become black and white.

One possible explanation for the loss in information might be that we noticed that for $M = N = 3$, the tensors of the estimated weights and activations only contained $4$ or $5$ unique values. Having a 3-bit accuracy, it should be possible to achieve 8 unique levels instead. When $M = N = 5$, it should be possible to output 32 different values, but instead, we found that the tensors only contained $5$ or $6$ unique values.

One explanation for this could be that the shifting of parameters to form $\boldsymbol{B}$ is not done optimally. Right now the shifts, which set the thresholds for binarization, are uniformly distributed over a range from minus the standard deviation to plus the standard deviation, which is the way it was described in the ABC-Net paper. However, since the distribution of weights often resembles a Gaussian distribution, it might be better to put the thresholds around the mean closer together and the thresholds further from the mean further apart, to better match the Gaussian distribution.
Currently, in the interest of time for this project, we were unable to research further into this problem, but we expect that if more of the possible weights occur, that this will lead to a smaller approximation error, which in term helps with improving performance.

We deviated in one aspect from the original ABC-Net implementation, by implementing the binarization of the activations the same as for the weights. This was done because we could not get the ABCLSID network to work with normalization, which probably interfered with our initialization. Another technique that could be studied in the future that could make this work is taking the original LTSITD network and add normalization to it. This network could then be trained to match the performance of the original model. This new model could then be used to initialize the network when binarizing the activations and this means that when binarizing the activations the network does not also need to adapt to the normalization.

This also means that the network could use the learnable parameter beta, to possibly find better threshold values than the ones we used. This can make the approximations and thus also the performance of the network better.

Another reason for the loss in performance could be that there simply goes something wrong in our implementation of binarizing the activations. For example, it is possible that when binarizing the activations, the gradient is not calculated correctly anymore. This could cause the network to not properly learn and adapt to the new structure. We have seen, for instance, during some initial testing that the training and validation error would go up after the first epoch when training on a small subset of the training data.

It could also be that there is something wrong with the calculation of the backpropagation as described in ABC-Net. The definition in the paper, the one we use, uses the STE and is as follows:

$$\frac{\delta c}{\delta\textbf{W}} = \sum\limits_{m=1}^M \alpha_m \frac{\delta c}{\delta \textbf{B}_m}$$

However, according to the author the following ABC-Net implementation cow8/ABC-Net-pytorch (https://github.com/cow8/ABC-Net-pytorch), this is not correct. They state that the correct formulation is:

$$\frac{\delta c}{\delta\textbf{W}} = \frac{\delta c}{\delta\textbf{O}}\cdot \left(  \sum\limits_{m=1}^M \left( \frac{\delta \textbf{O}}{\delta \alpha_m Conv(B_m, A)} \cdot \left( \frac{\delta\alpha_m}{\delta W} \cdot Conv(B_m, A) + \frac{\delta Conv(B_m, A)}{\delta W} \cdot \alpha_m \right) \right) \right) $$

We are not 100\% that this is indeed the case, especially since it is not from a peer-reviewed paper, but from a public Github repository that contains an implementation that does not yet work. However, it might be interesting to explore this further in the future.

# References
[1] C. Chen, Q. Chen, J. Xu, and V. Koltun, “Learning to see in the dark”, CoRR, vol. abs/1805.01934,2018. [Online]. Available: http://arxiv.org/abs/1805.019341

[2] X. Lin, C. Zhao, and W. Pan, “Towards accurate binary convolutional neural network”, CoRR, vol.abs/1711.11294, 2017. [Online]. Available: http://arxiv.org/abs/1711.112941

[3] Cydonia, "Learning to See in the Dark in PyTorch", 2018.  [Online].  Available:   https://github.com/cydonia999/Learning_to_See_in_the_Dark_PyTorch

[4] Mylio,2020.[Online].Available:https://focus.mylio.com/tech-today/how-many-photos-will-be-taken-in-2020

[5]  M.   Rastegari,   V.   Ordonez,   J.   Redmon,   and   A.   Farhadi,   “Xnor-net:Imagenet   classification using  binary  convolutional  neural  networks”, CoRR,  vol.  abs/1603.05279,  2016.  [Online].  Available:http://arxiv.org/abs/1603.052791

[6]  E. Nurvitadhi, D. Sheffield, Jaewoong Sim, A. Mishra, G. Venkatesh, and D. Marr, “Accelerating bina-rized neural networks:  Comparison of fpga,  cpu,  gpu,  and asic,”  in2016 International Conference onField-Programmable Technology (FPT), 2016, pp. 77–84.

Want to see more reproductions of this and other papers? Check https://reproduced-papers.firebaseapp.com/

---


### Checklist for submission

It is extremely important to make sure that:

1. Everything runs as expected (no bugs when running cells);
2. The output from each cell corresponds to its code (don't change any cell's contents without rerunning it afterwards);
3. All outputs are present (don't delete any of the outputs);
4. Fill in all the places that say `# YOUR CODE HERE`, or "**Your answer:** (fill in here)".
5. Never copy/paste any notebook cells. Inserting new cells is allowed, but it should not be necessary.
6. The notebook contains some hidden metadata which is important during our grading process. **Make sure not to corrupt any of this metadata!** The metadata may for example be corrupted if you copy/paste any notebook cells, or if you perform an unsuccessful git merge / git pull. It may also be pruned completely if using Google Colab, so watch out for this. Searching for "nbgrader" when opening the notebook in a text editor should take you to the important metadata entries.
7. Although we will try our very best to avoid this, it may happen that bugs are found after an assignment is released, and that we will push an updated version of the assignment to GitHub. If this happens, it is important that you update to the new version, while making sure the notebook metadata is properly updated as well. The safest way to make sure nothing gets messed up is to start from scratch on a clean updated version of the notebook, copy/pasting your code from the cells of the previous version into the cells of the new version.
8. If you need to have multiple parallel versions of this notebook, make sure not to move them to another directory.
9. Although not forced to work exclusively in the course `conda` environment, you need to make sure that the notebook will run in that environment, i.e. that you have not added any additional dependencies.

**FOR HA1, HA2, HA3 ONLY:** Failing to meet any of these requirements might lead to either a subtraction of POEs (at best) or a request for resubmission (at worst).

We advise you to perform the following steps before submission to ensure that requirements 1, 2, and 3 are always met: **Restart the kernel** (in the menubar, select Kernel$\rightarrow$Restart) and then **run all cells** (in the menubar, select Cell$\rightarrow$Run All). This might require a bit of time, so plan ahead for this (and possibly use Google Cloud's GPU in HA1 and HA2 for this step). Finally press the "Save and Checkout" button before handing in, to make sure that all your changes are saved to this .ipynb file.

### Fill in name of notebook file
This might seem silly, but the version check below needs to know the filename of the current notebook, which is not trivial to find out programmatically.

You might want to have several parallel versions of the notebook, and it is fine to rename the notebook as long as it stays in the same directory. **However**, if you do rename it, you also need to update its own filename below:

```python
nb_fname = "IHA1.ipynb"
```

### Fill in group number and member names (use NAME2 and GROUP only for HA1, HA2 and HA3):

```python
NAME1 = ""
NAME2 = ""
GROUP = ""
```

### Check Python version

```python
from platform import python_version_tuple
assert python_version_tuple()[:2] == ('3','7'), "You are not running Python 3.7. Make sure to run Python through the course Conda environment."
```

### Check that notebook server has access to all required resources, and that notebook has not moved

```python
import os
nb_dirname = os.path.abspath('')
assignment_name = os.path.basename(nb_dirname)
assert assignment_name in ['IHA1', 'IHA2', 'HA1', 'HA2', 'HA3'], \
    '[ERROR] The notebook appears to have been moved from its original directory'
```

### Verify correct nb_fname

```python
from IPython.display import display, HTML
try:
    display(HTML(r''.format(nb_fname=nb_fname)))
except NameError:
    assert False, 'Make sure to fill in the nb_fname variable above!'
```

### Verify that your notebook is up-to-date and not corrupted in any way

```python
import sys
sys.path.append('..')
from ha_utils import check_notebook_uptodate_and_not_corrupted
check_notebook_uptodate_and_not_corrupted(nb_dirname, nb_fname)
```

    Matching current notebook against the following URL:
    http://raw.githubusercontent.com/JulianoLagana/deep-machine-learning/master/home-assignments/IHA1/IHA1.ipynb
    [SUCCESS] No major notebook mismatch found when comparing to latest GitHub version. (There might be minor updates, but even that is the case, submitting your work based on this notebook version would be acceptable.)

# IHA1 - Assignment

Welcome to the first individual home assignment!

This assignment consists of two parts:
 * Python and NumPy exercises;
 * Build a deep neural network for forward propagation.

The focus of this assignment is for you to gain practical knowledge with implementing forward propagation of deep neural networks without using any deep learning framework. You will also gain practical knowledge in two of Python's scientific libraries [NumPy](https://docs.scipy.org/doc/numpy-1.13.0/index.html) and [Matplotlib](https://matplotlib.org/devdocs/index.html).

Skeleton code is provided for most tasks and every part you are expected to implement is marked with **TODO**. Throughout the assignment you will also need to submit written answers to some questions. These questions are mainly to make you reflect on some particular topics and your answers will not be graded in detail.

We expect you to search and learn by yourself any commands you think are useful for these tasks. Don't limit yourself to what was taught in CL1. Use the help function, [stackoverflow](https://stackoverflow.com/), google, the [python documentation](https://docs.python.org/3.5/library/index.html) and the [NumPy](https://docs.scipy.org/doc/numpy-1.13.0/index.html) documentation to your advantage.

**IMPORTANT NOTE**: The tests available are not exhaustive, meaning that if you pass a test you have avoided the most common mistakes, but it is still not guaranteed that your solution is 100% correct.

Lets start by importing the necessary libraries below.

```python
import numpy as np
import matplotlib.pyplot as plt
from utils.tests.iha1Tests import *
```

## 1. Lists and arrays introduction
First, we will warm up with a Python exercise and few NumPy exercises.

### 1.1 List comprehensions
Examine the code snippet provided below.

```python
myList = []
for i in range(25):
    if i % 2 == 0:
        myList.append(i**2)

print(myList)
```

    [0, 4, 16, 36, 64, 100, 144, 196, 256, 324, 400, 484, 576]

This is not a very "[pythonic](http://docs.python-guide.org/en/latest/writing/style/)" way of writing. Lets re-write the code above using a [list comprehension](https://docs.python.org/2/tutorial/datastructures.html#list-comprehensions). The result will be less code, more readable and elegant. Your solution should be able to fit into one line of code.

```python
myList = [i**2 for i in range(25) if i % 2 == 0] # TODO
# YOUR CODE HERE
print(myList)
```

    [0, 4, 16, 36, 64, 100, 144, 196, 256, 324, 400, 484, 576]

```python
# sample output from cell above for reference
# [0, 4, 16, 36, 64, 100, 144, 196, 256, 324, 400, 484, 576]
```

### 1.2 Numpy array vs numpy vectors
Run the cell below to create a numpy array.

```python
myArr = np.array([1, 9, 25, 49, 81, 121, 169, 225, 289, 361, 441, 529])
print(myArr)
print(myArr.shape)
```

    [  1   9  25  49  81 121 169 225 289 361 441 529]
    (12,)

One of the core features of numpy is to efficiently perform linear algebra operations.
There are two types of one-dimensional representations in numpy: arrays of shape (x,) and vectors of shape (x,1).

The above result indicates that **myArr** is an array of 12 elements with shape (12,).

Numpy's arrays and vectors both have the type of `numpy.ndarray` but have in some cases different characteristics and it is important to separate the two types because it will save a lot of debugging time later on. Read more about numpy shapes [here](https://stackoverflow.com/a/22074424).

Run the code below to see how the transpose operation behaves differently between an array and vector.

```python
# print the shape of an array and the shape of a transposed array
print('myArr is an array of shape:')
print(myArr.shape)
print('The transpose of myArr has the shape:')
print(myArr.T.shape)

# print the shape of a vector and the transpose of a vector
myVec = myArr.reshape(12,1)
print('myVec is a vector of shape:')
print(myVec.shape)
print('The transpose of myVec has the shape:')
print(myVec.T.shape)
```

    myArr is an array of shape:
    (12,)
    The transpose of myArr has the shape:
    (12,)
    myVec is a vector of shape:
    (12, 1)
    The transpose of myVec has the shape:
    (1, 12)

### 1.3 Numpy exercises
Now run the cell below to create the numpy array `numbers` and then complete the exercises sequentially.

```python
numbers = np.arange(24)
print(numbers)
```

    [ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23]

```python
# TODO: reshape numbers into a 6x4 matrix
numbers=numbers.reshape(6,4)
# YOUR CODE HERE
print(numbers)

```

    [[ 0  1  2  3]
     [ 4  5  6  7]
     [ 8  9 10 11]
     [12 13 14 15]
     [16 17 18 19]
     [20 21 22 23]]

```python
# sample output from cell above for reference
#  [[ 0  1  2  3]
#   [ 4  5  6  7]
#   [ 8  9 10 11]
#   [12 13 14 15]
#   [16 17 18 19]
#   [20 21 22 23]]
```

```python
# test case
test_numpy_reshape(numbers)
```

    Test passed

```python
# TODO: set the element of the last row of the last column to zero
# Hint: Try what happends when indices are negative
numbers[-1,-1]=0

# YOUR CODE HERE
print(numbers)
```

    [[ 0  1  2  3]
     [ 4  5  6  7]
     [ 8  9 10 11]
     [12 13 14 15]
     [16 17 18 19]
     [20 21 22  0]]

```python
# sample output from cell above for reference
#  [[ 0  1  2  3]
#   [ 4  5  6  7]
#   [ 8  9 10 11]
#   [12 13 14 15]
#   [16 17 18 19]
#   [20 21 22  0]]
```

```python
# test case
test_numpy_neg_ix(numbers)
```

    Test passed

```python
# TODO: set every element of the 0th row to 0
numbers[0,]=0

# YOUR CODE HERE
print(numbers)
```

    [[ 0  0  0  0]
     [ 4  5  6  7]
     [ 8  9 10 11]
     [12 13 14 15]
     [16 17 18 19]
     [20 21 22  0]]

```python
# sample output from cell above for reference
#  [[ 0  0  0  0]
#   [ 4  5  6  7]
#   [ 8  9 10 11]
#   [12 13 14 15]
#   [16 17 18 19]
#   [20 21 22  0]]
```

```python
# test case
test_numpy_row_ix(numbers)
```

    Test passed

```python
# TODO: append a 1x4 row vector of zeros to `numbers`,
# resulting in a 7x4 matrix where the new row of zeros is the last row
# Hint: A new matrix must be created in the procedure. Numpy arrays are not dynamic.
numbers=np.vstack([numbers, np.zeros(4)])
# YOUR CODE HERE
print(numbers)
print(numbers.shape)
```

    [[ 0.  0.  0.  0.]
     [ 4.  5.  6.  7.]
     [ 8.  9. 10. 11.]
     [12. 13. 14. 15.]
     [16. 17. 18. 19.]
     [20. 21. 22.  0.]
     [ 0.  0.  0.  0.]]
    (7, 4)

```python
# sample output from cell above for reference
#   [[ 0  0  0  0]
#    [ 4  5  6  7]
#    [ 8  9 10 11]
#    [12 13 14 15]
#    [16 17 18 19]
#    [20 21 22  0]
#    [ 0  0  0  0]]
#  (7, 4)
```

```python
# test case
test_numpy_append_row(numbers)
```

    Test passed

```python
# TODO: set all elements above 10 to the value 1
b=(numbers>10)
numbers[b]=1
# YOUR CODE HERE
print(numbers)
```

    [[ 0.  0.  0.  0.]
     [ 4.  5.  6.  7.]
     [ 8.  9. 10.  1.]
     [ 1.  1.  1.  1.]
     [ 1.  1.  1.  1.]
     [ 1.  1.  1.  0.]
     [ 0.  0.  0.  0.]]

```python
# sample output from cell above for reference
#  [[ 0  0  0  0]
#   [ 4  5  6  7]
#   [ 8  9 10  1]
#   [ 1  1  1  1]
#   [ 1  1  1  1]
#   [ 1  1  1  0]
#   [ 0  0  0  0]]
```

```python
# test case
test_numpy_bool_matrix(numbers)
```

    Test passed

```python
# TODO: compute the sum of every row and replace `numbers` with the answer
# `numbers` will be a (7,) array as a result
numbers=np.sum(numbers, axis=1)
# YOUR CODE HERE
print(numbers.shape)
print(numbers)
```

    (7,)
    [ 0. 22. 28.  4.  4.  3.  0.]

```python
# sample output from cell above for reference
#   (7,)
#   [ 0 22 28  4  4  3  0]
```

```python
# test case
test_numpy_sum(numbers)
```

    Test passed

## 2 Building your deep neural network
It is time to start implementing your first feed-forward neural network. In this lab you will only focus on implementing the forward propagation procedure.

As you know, a feed-forward neural network consists of a sequence of layers, where the output from one layer is the input to the following layer. Below you can find an example of a feed-forward neural network with 2 hidden layers (blue) and 1 output layer (red).

**What would be your approach to implement a feed-forward neural network? Please provide a brief and high level explanation on how would you implement the several components of network; for instance, would you use any `python` class or only functions?** We will not grade this answer in detail. The objective with this question is to help you reflect upon how to structure the code. This will hopefully enable you to understand the code below more easily. As long as it is clear from your answer that you have tried to briefly describe the overall structure of the code, we will accept your answer.

**Your answer:** (I will use python functions to implement a forward neural network, first initialize a matrix for w and zero for b and then calculate the Z and then apply the activation function to calculate A using a for loop for 1 to l layer)

In this exercise, you will implement several components of a feed-forward network, such as a layers, activation functions and weight initialization functions, and from those create your own feed-forward neural network implementation.

NOTE: When using a neural network, due to memory and computational constraints, you can not forward propagate the entire dataset at once. Therefore, you divide the dataset into a number of sets/parts called batches. A batch will make up for the first dimension of every input to a layer and the notation `(BATCH_SIZE, NUM_FEATURES)` simply means the dimension of a batch of samples where every sample has `NUM_FEATURES` features.

### 2.1 Activation functions
You will start by defining a few activation functions that are later needed by the neural network.

#### 2.1.1 ReLU
The neural network will use the ReLU activation function in every layer except for the last. ReLU does element-wise comparison of the input matrix. For example, if the input is `X`, and `X[i,j] == 2` and `X[k,l] == -1`, then after applying ReLU, `X[i,j] == 2` and `X[k,l] == 0` should be true.

The formula for implementing ReLU for a single neuron $i$ is:
\begin{equation}
relu(z_i) =
    \begin{cases}
      0, & \text{if}\ z_i \leq 0 \\
      z_i, & \text{otherwise}
    \end{cases}
\end{equation}

Now implement `relu` in vectorized form

```python
def relu(z):
    """ Implement the ReLU activation function

    Arguments:
    z - the input of the activation function. Has a type of 'numpy.ndarray'

    Returns:
    a - the output of the activation function. Has a type of numpy.ndarray and the same shape as 'z'
    """

    z[z<0]=0
    a = z # TODO
    # YOUR CODE HERE

    return a
```

```python
# test case
test_relu(relu)
```

    Test passed

Can you name some of the advantages of the ReLU activation function? (We will not grade this answer in detail.)

**Your answer:** (It is quick in training compared to use tanh or signoid activation function)

#### 2.1.2 Sigmoid
The sigmoid activation function is common for binary classification. This is because it squashes its input to the range [0,1].
Implement the activation function `sigmoid` using the formula:
\begin{equation}
    \sigma(z) = \frac{1}{1 + e^{-z}}
\end{equation}

```python
def sigmoid(z):
    """ Implement the sigmoid activation function

    Arguments:
    z - the input of the activation function. Has a type of 'numpy.ndarray'

    Returns:
    a - the output of the activation function. Has a type of 'numpy.ndarray' and the same shape as 'z'
    """
    z=1/(1+np.exp(-z))
    a = z # TODO
    # YOUR CODE HERE

    return a
```

```python
# test case
test_sigmoid(sigmoid)
```

    Test passed

Why is the sigmoid activation function useful for binary classification? Feel free to use some numerical examples to show how the magnitudes of the variable $z$ affect the output of the sigmoid layer. (We will not grade this answer in detail.)

**Your answer:** (sigmoid result will between 0 to 1)

#### 2.1.3 Visualization
Make a plot using matplotlib to visualize the activation functions between the input interval [-3,3]. The plot should have the following properties
 * one plot should contain a visualization of both `ReLU` and `sigmoid`;
 * x-axis: range of values between [-3,3], **hint**: np.linspace;
 * y-axis: the value of the activation functions at a given input `x`;
 * a legend explaining which line represents which activation function.

```python
# TODO: make a plot of ReLU and sigmoid values in the interval [-3,3]
x=np.linspace(-3,3,100)
y1=relu(x)
y2=sigmoid(x)
fig, axes=plt.subplots()
axes.plot(x,y1)
axes.plot(x,y2)
axes.set_xlim([-3,3])
axes.legend(['Relu', 'Sigmoid'], loc=2)
# YOUR CODE HERE
```

#### 2.1.4 Softmax
You will use the softmax activation function / classifier as the final layer of your neural network later in the assignment. Implement `softmax` according to the formula below. The subtraction of the maximum value is there solely to avoid overflows in a practical implementation.
\begin{equation}
softmax(z_i) = \frac{e^{z_i - max(\mathbf{z})}}{ \sum^j e^{z_j - max(\mathbf{z})}}
\end{equation}

```python
def softmax(z):
    """ Implement the softmax activation function

    Arguments:
    z - the input of the activation function, shape (BATCH_SIZE, FEATURES) and type 'numpy.ndarray'

    Returns:
    a - the output of the activation function, shape (BATCH_SIZE, FEATURES) and type 'numpy.ndarray'
    """
    batchsize=z.shape[0]
    z=np.exp(z)
    a = z/np.sum(z, axis=1).reshape(batchsize,1) # TODO
    # YOUR CODE HERE

    return a
```

```python
# test case
test_softmax(softmax)
```

    Test passed

What are the main differences between using `sigmoid` and `softmax` for multi-class classification problems? And when the number of classes is 2? (We will not grade this answer in detail.)

**Your answer:** (sigmoid is for binary classfication (0,1) and softmax is for the number of classes larger then 2)

### 2.2 Initialize weights
You will implement a helper function that takes the shape of a layer as input, and returns an initialized weight matrix $\mathbf{W}$ and bias vector $\mathbf{b}$ as output. The matrix $\mathbf{W}$ should be sampled from a normal distribution with mean 0 and standard deviation 2, and $\mathbf{b}$ should be initialized to all zeros.

```python
def initialize_weights(layer_shape):
    """ Implement initialization of the weight matrix and biases

    Arguments:
    layer_shape - a tuple of length 2, type (int, int), that determines the dimensions of the weight matrix: (input_dim, output_dim)

    Returns:
    w - a weight matrix with dimensions of 'layer_shape', (input_dim, output_dim), that is normally distributed with
        properties mu = 0, stddev = 2. Has a type of 'numpy.ndarray'
    b - a vector of initialized biases with shape (1,output_dim), all of value zero. Has a type of 'numpy.ndarray'
    """
    w = np.random.normal(0,2,layer_shape) # TODO
    b = np.zeros((1,layer_shape[1]))# TODO
    # YOUR CODE HERE

    return w, b
```

```python
# test case
test_initialize_weights(initialize_weights)
```

    Test passed

### 2.3 Feed-forward neural network layer module
To build a feed-forward neural network of arbitrary depth you are going to define a neural network layer as a module that can be used to stack layers upon each other. Your task is to complete the `Layer` class by following the descriptions in the comments. Recall the formula for forward propagation of an arbitrary layer $l$:

\begin{equation}
\mathbf{a}^{[l]} = g(\mathbf{z}^{[l]}) = g(\mathbf{a}^{[l-1]}\mathbf{w}^{[l]} +\mathbf{b}^{[l]})
\end{equation}

where $g$ is the activation function given by `activation_fn`, which can be relu, sigmoid or softmax.

```python
class Layer:
    """
    TODO: Build a class called Layer that satisfies the descriptions of the methods
    Make sure to utilize the helper functions you implemented before
    """

    def __init__(self, input_dim, output_dim, activation_fn=relu):
        """
        Arguments:
        input_dim - the number of inputs of the layer. type int
        output_dim - the number of outputs of the layer. type int
        activation_fn - a reference to the activation function to use. Should be 'relu' as a default
                        possible values are the 'relu', 'sigmoid' and 'softmax' functions you implemented earlier.
                        Has the type 'function'

        Attributes:
        w - the weight matrix of the layer, should be initialized with 'initialize_weights'
            and has the shape (INPUT_FEATURES, OUTPUT_FEATURES) and type 'numpy.ndarray'
        b - the bias vector of the layer, should be initialized with 'initialize_weights'
            and has the shape (1, OUTPUT_FEATURES) and type 'numpy.ndarray'
        activation_fn - a reference to the activation function to use.
                        Has the type 'function'
        """
        self.w, self.b = initialize_weights((input_dim,output_dim)) # TODO
        self.activation_fn = activation_fn # TODO
        # YOUR CODE HERE

    def forward_prop(self, a_prev):
        """ Implement the forward propagation module of the neural network layer
        Should use whatever activation function that 'activation_fn' references to

        Arguments:
        a_prev - the input to the layer, which may be the data 'X', or the output from the previous layer.
            a_prev has the shape of (BATCH_SIZE, INPUT_FEATURES) and the type 'numpy.ndarray'

        Returns:
        a - the output of the layer when performing forward propagation. Has the type 'numpy.ndarray'
        """
        a = self.activation_fn(a_prev.dot(self.w)+self.b) # TODO
        # YOUR CODE HERE

        return a
```

```python
# test case, be sure that you pass the previous activation function tests before running this test
test_layer(Layer, relu, sigmoid, softmax)
```

    Test passed

### 2.4 Backward propagation

In the previous exercise you only implemented the forward propagation of the layer. However, when training a network, you would also need to implement the backward propagation of the layer in order to be able to update its weights. (You won't need to do this in this assignment since you will not train the network. Instead you will load the network weights, which are obtained from a previously trained network.)

Provide a high-level explanation of how you would modify your current implementation of the `Layer` class in order to add the backward propagation. (We will not grade this in detail. It is enough that your answer demonstrates that you have thought about how to perform backprop using these classes.)

**Your answer:** (first get dZ then to dW)

### 2.5 Example - Logistic regression
Binary logistic regression is a classifier where classification is performed by applying the sigmoid activation function to a linear combination of input values. You will now try out your neural network layer by utilizing it as a linear combination of input values and apply the sigmoid activation function to classify a simple problem.

The cell below defines a dataset of 5 points of either class `0` or class `1`. Your assignment is to:
1. Create an instance of a `Layer` with the sigmoid activation function;
2. Manually tune the weights `w` and bias `b` of your layer.

You can use `test_logistic` to visually inspect how your classifier is performing. In order to  be successful in this exercise you need select values for the weights `w` and bias `b` such that all points are classified correctly.

```python
# Run this cell to create the dataset
X_s = np.array([[1, 2],
               [5, 3],
               [8, 8],
               [7, 5],
               [3, 6]])
Y_s = np.array([0,0,1,0,1])

test_logistic(X_s, Y_s)
```

```python
# create an instance of layer
l = Layer(2,1,sigmoid)

# TODO: manually tune weights
l.w = np.array([[-0.5],[0.75]])
l.b = np.array([[-1.5]])
# YOUR CODE HERE

# testing your choice of weights with this function
test_logistic(X_s,Y_s,l,sigmoid)
```

### 2.6 Feed-forward neural network
You will now define the actual neural network class. It is an L-layer neural network, meaning that the number of layers and neurons in each layer is specified as input by the user. Once again, you will only focus on implementing the forward propagation part.

Read the descriptions in the comments and complete the **TODO**s.

```python
class NeuralNetwork:
    """
    TODO: Implement an L-layer neural network class by utilizing the Layer module defined above
    Each layer should use 'relu' activation function, except for the output layer, which should use 'softmax'
    """

    def __init__(self, input_n, layer_dims):
        """
        Arguments:
        input_n    - the number of inputs to the network. Should be the same as the length of a data sample
                     Has type int
        layer_dims - a python list or tuple of the number of neurons in each layer. Layer 'l' should have a weight matrix
                     with the shape ('layer_dims[l-1]', 'layer_dims[l]').
                     'layer_dims[-1]' is the dimension of the output layer.
                     Layer 1 should have the dimensions ('input_n', 'layer_dims[0]').
                     len(layer_dims) is the depth of the neural network
        Attributes:
        input_n - the number of inputs to the network. Has type int
        layers  - a python list of each layer in the network. Each layer should use the 'relu' activation function,
                  except for the last layer, which should use 'softmax'.
                  Has type 'list' containing layers of type 'Layer'
        """

        self.input_n = input_n # TODO
        self.layers=[]
        for i in range(0,len(layer_dims)-1):
            self.layers.append(Layer(input_n, layer_dims[i]))
            input_n=layer_dims[i]
        self.layers.append(Layer(input_n, layer_dims[-1],softmax))
        # YOUR CODE HERE

    def forward_prop(self, x):
        """
        Implement the forward propagation procedure through the entire network, from input to output.
        You will now connect each layer's forward propagation function into a chain of layer-wise forward propagations.

        Arguments:
        x - the input data, which has the shape (BATCH_SIZE, NUM_FEATURES) and type 'numpy.ndarray'

        Returns:
        a - the output of the last layer after forward propagating through the every layer in 'layers'.
            Should have the dimension (BATCH_SIZE, layers[-1].w.shape[1]) and type 'numpy.ndarray'
        """
        for layer in self.layers:
            x=layer.forward_prop(x)
        a=x
        # YOUR CODE HERE
        return a
```

```python
# test case
test_neuralnetwork(NeuralNetwork)
```

    Test passed

## 3 Making predictions with a neural network
In practice, its common to load weights to your neural network that has already been trained.
In this section, you will create an instance of your neural network, load trained weights from disk, and perform predictions.

### 3.1 Load weights from disk
Create an instance of `NeuralNetwork` with input size $28 \times 28 = 784$, two hidden layers of size 100 and an output layer of size 10. Thereafter, load the weights contained in `./utils/ann_weights.npz` to your network.

```python
input_n=784
layer_dims=[100,100,10]
ann = NeuralNetwork(input_n,layer_dims)
# YOUR CODE HERE

# load weights
weights = np.load('./utils/ann_weights.npz')
for l in range(len(ann.layers)):
    ann.layers[l].w = weights['w' + str(l)]
    ann.layers[l].b = weights['b' + str(l)]
```

### 3.2 Prediction
Now, implement the function `predict_and_correct` which does the following:
1. Load `./utils/test_data.npz` from disk;
2. Extract test data `X` and `Y` from file;
3. Perform for every pair of data:
    a. plot the image `x`; <br>
    b. make a prediction using your neural network by forward propagating and picking the most probable class;
    c. check whether the prediction is correct (compare with the ground truth number `y`);
    d. print the predicted label and wether it was correct or not.

```python
def predict_and_correct(ann):
    """ Load test data from file and predict using your neural network.
    Make a prediction for ever data sample and print it along with wether it was a correct prediction or not

    Arguments:
    ann - the neural network to use for prediction. Has type `NeuralNetwork`

    Returns: # for test case purposes
    A `numpy.ndarray` of predicted classes (integers [0-9]) with shape (11,)
    """
    data = np.load('./utils/test_data.npz')
    X, cls = data['X'], data['Y']

    cls_preds = [np.argmax(ann.forward_prop(i))for i in X]  # TODO: make a predicted number for every image in X
    # YOUR CODE HERE

    for i in range(len(X)):
        plt.imshow(X[i].reshape(28,28), cmap='gray')
        plt.show()
        correct = cls_preds[i] == cls[i]
        print('The prediction was {0}, it was {1}!'.format(cls_preds[i], 'correct' if correct else 'incorrect'))

    return cls_preds

cls_pred = predict_and_correct(ann)
```

```python
# final test case
test_predict_and_correct_answer(cls_pred)
```

    Test passed, you have implemented your neural network correctly!

## Congratulations!
You have successfully implemented a neural network from scratch using only NumPy!