# Classification of Satellite Superspectral Images

In this exercise, we will implement two deep learning approaches for classifying images of agricultural parcels from Sentinel2 multispectral images.
We will implement a **Multi-Layer Perceptron** and a **Convolutional Classifier**.

$\qquad\qquad\qquad\qquad$

The data considered is composed of $10\,000$ agricultural parcels for the training test and $5000$ parcels for the test set. Each parcel is a $10\times32\times32$ image of resolution $32\times 32$ with $10$ spectral bands, captured with the Senintel-2 optical satellite.

The selected parcels follow a nomenclature of $8$ agricultural classes:

> **Class Index** | **Class Name**
> --- | ---
> 0  | Meadow
> 1  | Durum Wheat
> 2  | Spring Cereal
> 3  | Sorgho
> 4  | Leguminous
> 5  | Fodder
> 6  | Ligneous
> 7  | Grapevine

Throughout this exercise, we will implement a full deep learning pipeline from scratch. The structure of the code is already implemented, you will have to complete some of the code denoted by the following flag: ```#TODO```

I have added some assertions in the code `assert` to check your code. only go to the next question if the assert apss succesfuly.

Since this is a notebook, it can be a bit cumbersome to debug. Do not hesitate to create extra cell and visualize the content of the tensors. Alternatively, I encourage you to add `print` to visualize the shape and content of tensors.

## Preliminaries

Select `Runtime→Change runtime type` and select `GPU`.

Let's first start by importing and installing the necessary libraires:

```
#install on the VM libraries that are not installed by default
!pip install torchnet
!pip install PyDrive
!pip install mock
#general purpose libraires
import math
import numpy as np
from functools import partial
import mock
#format libraries
import h5py
#visualization libraries
%matplotlib inline
import matplotlib.pyplot as plt
#machine learning tools
from sklearn.decomposition import PCA
from sklearn.metrics import confusion_matrix
#deep learning tools
import torch
import torchnet as tnt
import torch.nn.functional as nnf
import torch.nn as nn
import torch.optim as optim
from tqdm import tqdm_notebook as tqdm
#collab speific libraries
from google.colab import files
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
from google.colab import auth
from oauth2client.client import GoogleCredentials
```

    Collecting torchnet
      Downloading https://files.pythonhosted.org/packages/b7/b2/d7f70a85d3f6b0365517782632f150e3bbc2fb8e998cd69e27deba599aae/torchnet-0.0.4.tar.gz
    Requirement already satisfied: torch in /usr/local/lib/python3.7/dist-packages (from torchnet) (1.8.1+cu101)
    Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from torchnet) (1.15.0)
    Collecting visdom
      Downloading https://files.pythonhosted.org/packages/c9/75/e078f5a2e1df7e0d3044749089fc2823e62d029cc027ed8ae5d71fafcbdc/visdom-0.1.8.9.tar.gz (676kB)
         |████████████████████████████████| 686kB 26.3MB/s
    Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from torch->torchnet) (3.7.4.3)
    Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from torch->torchnet) (1.19.5)
    Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from visdom->torchnet) (1.4.1)
    Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from visdom->torchnet) (2.23.0)
    Requirement already satisfied: tornado in /usr/local/lib/python3.7/dist-packages (from visdom->torchnet) (5.1.1)
    Requirement already satisfied: pyzmq in /usr/local/lib/python3.7/dist-packages (from visdom->torchnet) (22.0.3)
    Collecting jsonpatch
      Downloading https://files.pythonhosted.org/packages/a3/55/f7c93bae36d869292aedfbcbae8b091386194874f16390d680136edd2b28/jsonpatch-1.32-py2.py3-none-any.whl
    Collecting torchfile
      Downloading https://files.pythonhosted.org/packages/91/af/5b305f86f2d218091af657ddb53f984ecbd9518ca9fe8ef4103a007252c9/torchfile-0.1.0.tar.gz
    Collecting websocket-client
      Downloading https://files.pythonhosted.org/packages/08/33/80e0d4f60e84a1ddd9a03f340be1065a2a363c47ce65c4bd3bae65ce9631/websocket_client-0.58.0-py2.py3-none-any.whl (61kB)
         |████████████████████████████████| 61kB 7.8MB/s
    Requirement already satisfied: pillow in /usr/local/lib/python3.7/dist-packages (from visdom->torchnet) (7.1.2)
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->visdom->torchnet) (2.10)
    Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->visdom->torchnet) (1.24.3)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->visdom->torchnet) (2020.12.5)
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->visdom->torchnet) (3.0.4)
    Collecting jsonpointer>=1.9
      Downloading https://files.pythonhosted.org/packages/23/52/05f67532aa922e494c351344e0d9624a01f74f5dd8402fe0d1b563a6e6fc/jsonpointer-2.1-py2.py3-none-any.whl
    Building wheels for collected packages: torchnet, visdom, torchfile
      Building wheel for torchnet (setup.py) ... done
      Created wheel for torchnet: filename=torchnet-0.0.4-cp37-none-any.whl size=29743 sha256=e4cb52f5fc6b9715fc43d0562551bb9c55163929425d88b1cf380421e8f414b0
      Stored in directory: /root/.cache/pip/wheels/e1/03/fb/1c212c2f20905cdf97fe39022946cf16b8e66ed754a6663400
      Building wheel for visdom (setup.py) ... done
      Created wheel for visdom: filename=visdom-0.1.8.9-cp37-none-any.whl size=655251 sha256=33bfedf241e0a236bc1c1c8ed8108fb908ab330c7efd443a600d499ccdb6ee68
      Stored in directory: /root/.cache/pip/wheels/70/19/a7/6d589ed967f4dfefd33bc166d081257bd4ed0cb618dccfd62a
      Building wheel for torchfile (setup.py) ... done
      Created wheel for torchfile: filename=torchfile-0.1.0-cp37-none-any.whl size=5713 sha256=6dba9a45ab8b2664a5a32106596c7ace5e8ed52467e294832c9e427d7fcde21d
      Stored in directory: /root/.cache/pip/wheels/b1/c3/d6/9a1cc8f3a99a0fc1124cae20153f36af59a6e683daca0a0814
    Successfully built torchnet visdom torchfile
    Installing collected packages: jsonpointer, jsonpatch, torchfile, websocket-client, visdom, torchnet
    Successfully installed jsonpatch-1.32 jsonpointer-2.1 torchfile-0.1.0 torchnet-0.0.4 visdom-0.1.8.9 websocket-client-0.58.0
    Requirement already satisfied: PyDrive in /usr/local/lib/python3.7/dist-packages (1.3.1)
    Requirement already satisfied: oauth2client>=4.0.0 in /usr/local/lib/python3.7/dist-packages (from PyDrive) (4.1.3)
    Requirement already satisfied: PyYAML>=3.0 in /usr/local/lib/python3.7/dist-packages (from PyDrive) (3.13)
    Requirement already satisfied: google-api-python-client>=1.2 in /usr/local/lib/python3.7/dist-packages (from PyDrive) (1.12.8)
    Requirement already satisfied: six>=1.6.1 in /usr/local/lib/python3.7/dist-packages (from oauth2client>=4.0.0->PyDrive) (1.15.0)
    Requirement already satisfied: pyasn1>=0.1.7 in /usr/local/lib/python3.7/dist-packages (from oauth2client>=4.0.0->PyDrive) (0.4.8)
    Requirement already satisfied: rsa>=3.1.4 in /usr/local/lib/python3.7/dist-packages (from oauth2client>=4.0.0->PyDrive) (4.7.2)
    Requirement already satisfied: pyasn1-modules>=0.0.5 in /usr/local/lib/python3.7/dist-packages (from oauth2client>=4.0.0->PyDrive) (0.2.8)
    Requirement already satisfied: httplib2>=0.9.1 in /usr/local/lib/python3.7/dist-packages (from oauth2client>=4.0.0->PyDrive) (0.17.4)
    Requirement already satisfied: google-api-core<2dev,>=1.21.0 in /usr/local/lib/python3.7/dist-packages (from google-api-python-client>=1.2->PyDrive) (1.26.2)
    Requirement already satisfied: uritemplate<4dev,>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from google-api-python-client>=1.2->PyDrive) (3.0.1)
    Requirement already satisfied: google-auth-httplib2>=0.0.3 in /usr/local/lib/python3.7/dist-packages (from google-api-python-client>=1.2->PyDrive) (0.0.4)
    Requirement already satisfied: google-auth>=1.16.0 in /usr/local/lib/python3.7/dist-packages (from google-api-python-client>=1.2->PyDrive) (1.28.0)
    Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from google-api-core<2dev,>=1.21.0->google-api-python-client>=1.2->PyDrive) (1.53.0)
    Requirement already satisfied: protobuf>=3.12.0 in /usr/local/lib/python3.7/dist-packages (from google-api-core<2dev,>=1.21.0->google-api-python-client>=1.2->PyDrive) (3.12.4)
    Requirement already satisfied: packaging>=14.3 in /usr/local/lib/python3.7/dist-packages (from google-api-core<2dev,>=1.21.0->google-api-python-client>=1.2->PyDrive) (20.9)
    Requirement already satisfied: pytz in /usr/local/lib/python3.7/dist-packages (from google-api-core<2dev,>=1.21.0->google-api-python-client>=1.2->PyDrive) (2018.9)
    Requirement already satisfied: requests<3.0.0dev,>=2.18.0 in /usr/local/lib/python3.7/dist-packages (from google-api-core<2dev,>=1.21.0->google-api-python-client>=1.2->PyDrive) (2.23.0)
    Requirement already satisfied: setuptools>=40.3.0 in /usr/local/lib/python3.7/dist-packages (from google-api-core<2dev,>=1.21.0->google-api-python-client>=1.2->PyDrive) (54.2.0)
    Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from google-auth>=1.16.0->google-api-python-client>=1.2->PyDrive) (4.2.1)
    Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging>=14.3->google-api-core<2dev,>=1.21.0->google-api-python-client>=1.2->PyDrive) (2.4.7)
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0dev,>=2.18.0->google-api-core<2dev,>=1.21.0->google-api-python-client>=1.2->PyDrive) (3.0.4)
    Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0dev,>=2.18.0->google-api-core<2dev,>=1.21.0->google-api-python-client>=1.2->PyDrive) (1.24.3)
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0dev,>=2.18.0->google-api-core<2dev,>=1.21.0->google-api-python-client>=1.2->PyDrive) (2.10)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0dev,>=2.18.0->google-api-core<2dev,>=1.21.0->google-api-python-client>=1.2->PyDrive) (2020.12.5)
    Collecting mock
      Downloading https://files.pythonhosted.org/packages/5c/03/b7e605db4a57c0f6fba744b11ef3ddf4ddebcada35022927a2b5fc623fdf/mock-4.0.3-py3-none-any.whl
    Installing collected packages: mock
    Successfully installed mock-4.0.3

Let's now download the data. To do this, you must first connect to your Google account to get the authorization to link to my drive. Launch the next cell, click the link, select your account, and click `authorize`. Copy and paste the verification code in the box below.

```
#Authentification
auth.authenticate_user()
gauth = GoogleAuth()
gauth.credentials = GoogleCredentials.get_application_default()
drive = GoogleDrive(gauth)
downloaded = drive.CreateFile({'id':'1owyqy_JKqMF0dhOwroM3J0siZsfRYthk'})
downloaded.GetContentFile('crop_data.h5')
```

The dataset is comprised of $10\,000$ patches of $10 \times 32\times 32$ for for training, and $5000$ for testing.
Launch the next cell to read the dataset.

```
#Data loading
data_file = h5py.File("crop_data.h5",'r')
data_train = data_file['train_data'][:]
labels_train = data_file['train_labels'][:]
data_test = data_file['test_data'][:]
labels_test = data_file['test_labels'][:]

n_train = data_train.shape[0]
n_test = data_test.shape[0]
class_names = ["Meadow", "Durum Wheat", "Spring Cereal", "Sorgho", "Leguminous", "Fodder", "Ligneous", "Grapevine"]

print("%d tiles for training, %d tiles for testing" % (n_train, n_test))
print("Train is of size: %d x %d x %d x %d" % (data_train.shape))
```

    10000 tiles for training, 5000 tiles for testing
    Train is of size: 10000 x 10 x 32 x 32

## Data Normalization

Q1) In machine learing in general, and deep learning in particular, it is important for the data to be normalized. Complete the next cell to normalize the train set such that all channels to be of mean $0$ and deviation $1$ over all pixels of all images. Apply the same normalization to the test set, **with the empirical values obtained from the training set.**

Why is it important to normalize the test set with the empirical values obtained from the training set and not the test set?

On normalise le train et le test avec les même valeurs pour les raisons suivantes:
(i) pour que les valeurs soient comparables, ie. une même donnée dans le train et le test soit identique dans les 2 sets
(ii) utiliser les statistiques du test est de la "triche", ie. de la "test set contamination"

```
#compute mean and std on all pixels of all patches
#(we keep the original dimension to facilitate broadcasting)
means = np.mean(data_train,axis=(0,2,3), keepdims=True)
stds = np.std(data_train, axis=(0,2,3), keepdims=True)
print("Means = ")
print(means[0,:,0,0])
print("STDs = ")
print(stds[0,:,0,0])

#normalize data to be of mean 0 and std 1
data_train = (data_train - means)/stds
data_test = (data_test - means)/stds
print(data_train[0,:,0,0])
print(data_test[0,:,0,0])

#check that the code is correct
assert(np.all([np.abs(np.mean(data_train[:,i]))<1e-4 for i in range(10)]))
assert(np.all([np.abs(np.std(data_train[:,i])-1)<1e-4 for i in range(10)]))
```

    Means =
    [ 240.50647  369.88947  385.5197   591.1033  1177.0797  1375.755
     1424.6808  1525.0121  1104.5576   648.7599 ]
    STDs =
    [ 312.28568  462.83038  521.88306  727.3077  1416.2495  1663.4348
     1721.7817  1831.8516  1350.8748   830.2049 ]
    [-0.7701489  -0.7991901  -0.73870903 -0.8127279  -0.83112454 -0.82705677
     -0.8274456  -0.83249766 -0.817661   -0.7814455 ] [-0.7701489  -0.7991901  -0.73870903 -0.8127279  -0.83112454 -0.82705677
     -0.8274456  -0.83249766 -0.817661   -0.7814455 ]

## Data Loader

Any deep learning pipeline rely on a dataloader, which loads a single element (here a patch). Read and understand the function.
Complete the function to compute the patch-wise spatial average if `average=True`.

```
def patch_loader(patch_index, train=True, average=False):
  """
  load a patch and returns the observation and associated ground truth
  INPUT:
  tile_index = int, index of the tile
  train = bool, train = True if in the train set, False for the test set
  average = if True, compute the spatial average of spectral bands, otherwise returns an image
  OUTPUT
  patch, if average=True [10] float tensor, otherwise [32 x 32 x 10] float tensor
  label, long int, containing the ground truth class
  """
  if train:
    patch = data_train[patch_index,:,:,:]
    label = labels_train[patch_index]
  else:
    patch = data_test[patch_index,:,:,:]
    label = labels_test[patch_index]

  if average:#spatial average on all pixels of the parcel:
    patch = patch.mean(axis=(1,2))

  #create torch tensors
  patch = torch.from_numpy(patch.astype('f4')) #use half-float to save space
  label = torch.Tensor([label]).long()  #ground truth must have long int type

  return patch, label
```

```
patch, label = patch_loader(500)
print("Patch :", patch[[2,1,0],:,:])
print("The image patch is of shape : %d x %d x %d" % (patch.shape))
print("and its label is %s" % (class_names[label]))
patch, label = patch_loader(200, average=True)
print("The average patch is of shape : %d" % (patch.shape))
print("and its label is %s" % (class_names[label]))
```

    Patch : tensor([[[-0.7387, -0.7387, -0.7387,  ..., -0.7387, -0.7387, -0.7387],
             [-0.7387, -0.7387, -0.7387,  ..., -0.7387, -0.7387, -0.7387],
             [-0.7387, -0.7387, -0.7387,  ..., -0.7387, -0.7387, -0.7387],
             ...,
             [-0.7387, -0.7387, -0.7387,  ..., -0.7387, -0.7387, -0.7387],
             [-0.7387, -0.7387, -0.7387,  ..., -0.7387, -0.7387, -0.7387],
             [-0.7387, -0.7387, -0.7387,  ..., -0.7387, -0.7387, -0.7387]],

            [[-0.7992, -0.7992, -0.7992,  ..., -0.7992, -0.7992, -0.7992],
             [-0.7992, -0.7992, -0.7992,  ..., -0.7992, -0.7992, -0.7992],
             [-0.7992, -0.7992, -0.7992,  ..., -0.7992, -0.7992, -0.7992],
             ...,
             [-0.7992, -0.7992, -0.7992,  ..., -0.7992, -0.7992, -0.7992],
             [-0.7992, -0.7992, -0.7992,  ..., -0.7992, -0.7992, -0.7992],
             [-0.7992, -0.7992, -0.7992,  ..., -0.7992, -0.7992, -0.7992]],

            [[-0.7701, -0.7701, -0.7701,  ..., -0.7701, -0.7701, -0.7701],
             [-0.7701, -0.7701, -0.7701,  ..., -0.7701, -0.7701, -0.7701],
             [-0.7701, -0.7701, -0.7701,  ..., -0.7701, -0.7701, -0.7701],
             ...,
             [-0.7701, -0.7701, -0.7701,  ..., -0.7701, -0.7701, -0.7701],
             [-0.7701, -0.7701, -0.7701,  ..., -0.7701, -0.7701, -0.7701],
             [-0.7701, -0.7701, -0.7701,  ..., -0.7701, -0.7701, -0.7701]]])
    The image patch is of shape : 10 x 32 x 32
    and its label is Durum Wheat
    The average patch is of shape : 10
    and its label is Meadow

## Data Visualization

Writing some visualization functions to explore the data and the results is very important.
Read and undertsand the definition of `view_patch_RGB` and `show_random_patches`.

```
#functions used for visualization
def view_patch_RGB(patch, ax):
  """
  INPUT:
  patch - CxWxH array: the patch to visualize
  ax - subplot axis: in which windows to represent the patch
  """
  patch = np.swapaxes(patch[[2,1,0],:,:],0,-1)
  patch = np.minimum(1,np.maximum(0,(patch+2)/4))
  ax.imshow(patch)
  plt.axis('off')

def show_random_patches(num_patches, train=True):
  """ show random patches values
  INPUT:
  num_patches - int: the number of patches to show
  train - bool: if true loads from the train set, otherwise the test set
  """
  n_row = int(num_patches / 4 + 1) #4 columns
  fig = plt.figure(figsize=(10, n_row * 2.5))
  for i_image in range(num_patches):
    index = np.random.randint(n_train) if train else np.random.randint(n_test)
    patch, label = patch_loader(index, train=train)
    ax = fig.add_subplot(n_row, 4, i_image+1, aspect='equal')
    view_patch_RGB(patch, ax)
    ax.set_title(class_names[label])
```

Q2) Use the function `show_random_patches` to represent 16 random tiles of the
**test set**.

```
show_random_patches(num_patches=16, train = False)
```

## Multi-Layer perceptron

We implement a simple MLP operating on the spatial average of each parcel's spectral channels. This networks maps the vector of size $10$ of the spectral average to a vector of $8$ class scores corresponding to the crop classes.

We now define the architecture of our MLP.

The input layer is of size $C$ (the number of bands, here 10) and the last layer is of size $K$ (the number of classes , here $8$). We add $2$ intermediate representations $x_1$ and $x_2$ of size $d_1$ and $d_2$.

Between each representation we add a fully connected layer FC. After each fully connected except the last one, we add a batchnorm and a ReLU activation.

The equations describing the function defined by the MLP are as follows:
$$\newcommand{\FC}[2]{\text{FC}_{#1}\left(#2\right)}$$
$$\newcommand{\BN}[1]{\text{BatchNorm}\left(#1\right)}$$
$$\newcommand{\RELU}[1]{\text{ReLU}\left(#1\right)}$$

\begin{align}
x_1  &= \RELU{\BN{\FC{1}{\text{input}}}}\\
x_2  &= \RELU{\BN{\FC{2}{x_1}}}\\
\text{output}  &= \FC{3}{x_2}\\
\end{align}

The size of all tensors is given in the following Table:

> **Tensor**           |    **shape**   |
> -------------------- | -------------------------------------------------|
> input                |  $C$                           |
> $x_1$                |  $d_1$                         |
> $x_2$                |  $d_2$                         |
> output               |  K

Q3 Complete the following cell to create the layer and implement the forward function.

```
class MLP(nn.Module):
  """
  MLP network for semantic segmentation
  """

  def __init__(self, n_channels, hidden_width,  n_class, cuda = 1):
    """
    initialization function
    n_channels: int = number of input channel
    hidden_width: int list = width of hidden layers
    n_class : int =  the number of classes
    """
    super(MLP, self).__init__() #necessary for all classes extending the module class

    self.is_cuda = cuda

    self.fc1 = nn.Sequential( \
        nn.Linear(in_features=n_channels, out_features=hidden_width[0]), \
        nn.BatchNorm1d(num_features=hidden_width[0]), \
        nn.ReLU(True))
    self.fc2 = nn.Sequential( \
        nn.Linear( in_features=hidden_width[0], out_features=hidden_width[1]), \
        nn.BatchNorm1d(num_features=hidden_width[1]), \
        nn.ReLU(True))
    self.fc3 = nn.Linear(in_features=hidden_width[1], out_features=n_class)

    if cuda: #remember wether to use the GPU or not
      self.cuda()

  def forward(self,input):
    """
    the foward pass : maps input to output
    """
    if self.is_cuda: #input to the GPU at the last second to save some VRAM
      input = input.cuda()

    x1 = self.fc1(input)
    x2 = self.fc2(x1)
    output = self.fc3(x2)
    return output
```

```
#==================TEST===============================
#make a batch of 2 patches
patch1, label = patch_loader(0, average=True)
patch2, label = patch_loader(1, average=True)
patches = torch.cat((patch1[None,:],patch2[None,:]),0)
#create classifier
mlp = MLP(10,[128,128],8)
#print layers + number of parameters
print(mlp)
print('Total number of parameters: {}'.format(sum([p.numel() for p in mlp.parameters()])))
#classify batch
pred = mlp(patches)
#check the shape of the outcome
assert(pred.shape == torch.Size([2,8]))
```

    MLP(
      (fc1): Sequential(
        (0): Linear(in_features=10, out_features=128, bias=True)
        (1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (2): ReLU(inplace=True)
      )
      (fc2): Sequential(
        (0): Linear(in_features=128, out_features=128, bias=True)
        (1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (2): ReLU(inplace=True)
      )
      (fc3): Linear(in_features=128, out_features=8, bias=True)
    )
    Total number of parameters: 19464

## Evaluation Metrics

Q4 Complete the metric functions belows, defined with respect to the confusion matrix $M$ of size $K \times K$:

* *Overall Accuracy*: a global metric defined as the ratio of correct prediction divided by the number of (annotated) points
    $$
    OA = \frac{\sum_{i}M_{i,i}}
    {\sum_{i,j}M_{i,j}}~.
    $$
* *Class IoU*: this per-class metric is defined as the ratio between true positives divided by the sum of false positives, false negatives and true positives.
    $$
    IoU_i = \frac{M_{i,i}}
    {M_{i,i} + \sum_{j \neq i}\left(M_{i,j} + M_{j,i} \right)}
    ~.
    $$

Make sure you obtain the following:
```
OA = 76,00%
Meadow        : 71.43%
Durum Wheat   : 57.14%
Spring Cereal : 100.00%
Sorgho        : 100.00%
Leguminous    : 100.00%
Fodder        : 66.67%
Ligneous      : 25.00%
Grapevine     : 25.00%
```

```
class ConfusionMatrix:
  def __init__(self, n_class, class_names):
    self.CM = np.zeros((n_class, n_class)) #the matrix itself
    self.n_class = n_class
    self.class_names = class_names

  def add_batch(self, gt, pred):
    #add in the CM a batch of prediction and associated ground truth
    self.CM +=  confusion_matrix(gt, pred, labels = list(range(self.n_class)))

  def overall_accuracy(self):#percentage of correct classification
    return 100 * np.trace(self.CM) / self.CM.sum()

  def class_IoU(self, show = 0):#IoU for each class
    ious = np.diag(self.CM) / (self.CM.sum(0) + self.CM.sum(1) - np.diag(self.CM) )

    if show: #print detailed values
      print('\n'.join('{:13s} : {:3.2f}%'.format(name, 100*iou) for name, iou in zip(self.class_names,ious)))
    #do not count classes that are not present in the dataset in the mean IoU
    return 100*np.nansum(ious) / (np.logical_not(np.isnan(ious))).sum()
```

```
m = ConfusionMatrix(8, class_names)
m.add_batch(np.array([0,1,1,5,2,0,0,4,0,5,3,7,8,6,7]), np.array([0,1,0,5,2,0,1,4,0,5,3,6,7,6,7]))
m.add_batch(np.array([0,1,5,1,2,1,0,2,3,6,7]), np.array([0,1,1,1,2,1,0,2,3,7,6]))
print(m.CM)
print("OA :", np.trace(m.CM) / m.CM.sum())
print("OA = %3.2f%%" % (m.overall_accuracy()))
print("mIoU = %3.2f%%" % (m.class_IoU(show=1)))
```

    [[5. 1. 0. 0. 0. 0. 0. 0.]
     [1. 4. 0. 0. 0. 0. 0. 0.]
     [0. 0. 3. 0. 0. 0. 0. 0.]
     [0. 0. 0. 2. 0. 0. 0. 0.]
     [0. 0. 0. 0. 1. 0. 0. 0.]
     [0. 1. 0. 0. 0. 2. 0. 0.]
     [0. 0. 0. 0. 0. 0. 1. 1.]
     [0. 0. 0. 0. 0. 0. 2. 1.]]
    OA : 0.76
    OA = 7600.00%
    Meadow        : 71.43%
    Durum Wheat   : 57.14%
    Spring Cereal : 100.00%
    Sorgho        : 100.00%
    Leguminous    : 100.00%
    Fodder        : 66.67%
    Ligneous      : 25.00%
    Grapevine     : 25.00%
    mIoU = 68.15%

Q5) Read in the `train` function following cell. Complete the function `eval`  which performs inference with the trained model. This function computes a prediction and the loss but does not compute gradients nor make an optimizer step.

```
def train(model, optimizer, loader, args):
  """train for one epoch
  INPUT:
  model = current model
  optimizer = current optimizer
  loader = dataset constituting the epoch
  OUTPUT:
  cm = the confusion matrix over the entire epoch
  lm = the average loss
  """
  model.train() #switch the model in training mode

  #tqdm will provide some nice progress bars
  loader = tqdm(loader, ncols=1000)

  #will keep track of the loss
  loss_meter = tnt.meter.AverageValueMeter()
  #keep tracks of prediction errors
  cm = ConfusionMatrix(args.n_class, class_names = class_names)

  #goes through the entire dataset
  for index, (tiles, label) in enumerate(loader):

    if model.is_cuda: #put label on GPU
      label = label.cuda()

    optimizer.zero_grad() #put gradient to zero

    pred = model(tiles) #compute the prediction

    loss = nn.functional.cross_entropy(pred,label.squeeze()) #compute the loss

    loss.backward() #compute gradients

    optimizer.step() #one gradient descent step

    #add the loss to the meter
    loss_meter.add(loss.item())
    #fill the confusion matrix
    cm.add_batch(label.cpu().detach().numpy(), pred.argmax(1).view(-1).cpu().detach().numpy())

  return cm, loss_meter.value()[0]

def eval(model, loader, args):
  """
  classify the given dataset. DO NOT track gradients
  INPUT:
  model = model to evaluate
  loader = dataset to classify
  OUTPUT:
  cm = the confusion matrix over the entire dataset
  lm = the average loss
  """

  model.eval() #switch in eval mode

  loader = tqdm(loader, ncols=1000)

  loss_meter = tnt.meter.AverageValueMeter()
  cm = ConfusionMatrix(args.n_class, class_names = class_names)

  with torch.no_grad(): #do not compute gradients (saves memory)

    for index, (tiles, label) in enumerate(loader):

      if model.is_cuda: #put label on GPU
        label = label.cuda()

        pred = model(tiles) #compute the prediction

        loss = nn.functional.cross_entropy(pred,label.squeeze()) #compute the loss

        #add the loss to the meter
        loss_meter.add(loss.item())
        #fill the confusion matrix
        cm.add_batch(label.cpu().detach().numpy(), pred.argmax(1).view(-1).cpu().detach().numpy())

  return cm, loss_meter.value()[0]
```

The following cell is the entire training sequence. Read and understand the structure.

```
def train_full(model, args):
  """The full training loop"""

  #create train and test dataset with ListDataset (as list of patch index)
  train_set = tnt.dataset.ListDataset(list(range(n_train)),partial(patch_loader, train=True, average=args.average))
  test_set  = tnt.dataset.ListDataset(list(range(n_test)),partial(patch_loader, train=False, average=args.average))

  #the loader function will take care of the batching
  train_loader = torch.utils.data.DataLoader(train_set, \
         batch_size=args.batch_size, shuffle=True, drop_last=True)
  test_loader = torch.utils.data.DataLoader(test_set, \
         batch_size=args.batch_size, shuffle=False, drop_last=False)

  #define the optimizer
  #Adam optimizer is always a good guess for classification tasks
  optimizer = optim.Adam(model.parameters(), lr=args.lr)

  #defines some color for test and train texts
  TESTCOLOR = '\033[104m'
  TRAINCOLOR = '\033[100m'
  NORMALCOLOR = '\033[0m'

  for i_epoch in range(args.n_epoch):
    #train one epoch
    cm_train, loss_train = train(model, optimizer, train_loader, args)
    print(TRAINCOLOR + 'Epoch %3d -> Train Overall Accuracy: %3.2f%% Train mIoU : %3.2f%% Train Loss: %1.4f' % (i_epoch, cm_train.overall_accuracy(), cm_train.class_IoU(), loss_train) + NORMALCOLOR)

    #periodic testing
    if (i_epoch == args.n_epoch - 1) or (args.n_epoch_test != 0 and i_epoch % args.n_epoch_test == 0 and i_epoch > 0):
      #evalue performance over the test set
      cm_test, loss_test = eval(model, test_loader, args)
      print(TESTCOLOR + 'Test Overall Accuracy: %3.2f%% Test mIoU : %3.2f%%  Test Loss: %1.4f' % (cm_test.overall_accuracy(), cm_test.class_IoU(show=1), loss_test) + NORMALCOLOR)
  return model
```

Q6) Create and train a model with the follwoing parameter in the following cell.

```
args = mock.Mock() #stores the parameters
args.n_epoch = 50 #bumber of epochs
args.n_epoch_test = int(5) #periodicity of evaluation on test set
args.batch_size = 16 #size of batch
args.n_class = 8 #number of classes
args.n_channel = 10 #number of channels
args.cuda = 1 #wether to use the GPU
args.lr = 1e-3 #learning rate
args.hidden_width = [256,256] #size of hidden layers
args.average = True #compute spatial average of each patch

#create the model to train
model = MLP(args.n_channel, args.hidden_width, args.n_class)
print('Total number of parameters: {}'.format(sum([p.numel() for p in model.parameters()])))

#train the model
trained_model = train_full(model, args)
```

    Total number of parameters: 71688

    /usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:14: TqdmDeprecationWarning: This function will be removed in tqdm==5.0.0
    Please use `tqdm.notebook.tqdm` instead of `tqdm.tqdm_notebook`

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch   0 -> Train Overall Accuracy: 58.35% Train mIoU : 34.86% Train Loss: 1.1715

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch   1 -> Train Overall Accuracy: 63.70% Train mIoU : 41.05% Train Loss: 1.0208

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch   2 -> Train Overall Accuracy: 65.13% Train mIoU : 42.62% Train Loss: 0.9830

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch   3 -> Train Overall Accuracy: 65.88% Train mIoU : 43.81% Train Loss: 0.9438

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch   4 -> Train Overall Accuracy: 67.22% Train mIoU : 45.45% Train Loss: 0.9306

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch   5 -> Train Overall Accuracy: 66.81% Train mIoU : 44.88% Train Loss: 0.9184

    /usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:57: TqdmDeprecationWarning: This function will be removed in tqdm==5.0.0
    Please use `tqdm.notebook.tqdm` instead of `tqdm.tqdm_notebook`

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=313.0), HTML(value='')), layout=Layout(di…

    Meadow        : 0.79%
    Durum Wheat   : 45.98%
    Spring Cereal : 44.92%
    Sorgho        : 21.38%
    Leguminous    : 66.90%
    Fodder        : 46.06%
    Ligneous      : 58.69%
    Grapevine     : 22.74%
    Test Overall Accuracy: 59.38% Test mIoU : 38.43%  Test Loss: 1.0464

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch   6 -> Train Overall Accuracy: 67.76% Train mIoU : 46.55% Train Loss: 0.9024

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch   7 -> Train Overall Accuracy: 68.32% Train mIoU : 46.48% Train Loss: 0.8895

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch   8 -> Train Overall Accuracy: 68.71% Train mIoU : 47.07% Train Loss: 0.8828

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch   9 -> Train Overall Accuracy: 68.88% Train mIoU : 47.21% Train Loss: 0.8694

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch  10 -> Train Overall Accuracy: 68.91% Train mIoU : 47.48% Train Loss: 0.8643

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=313.0), HTML(value='')), layout=Layout(di…

    Meadow        : 63.69%
    Durum Wheat   : 47.19%
    Spring Cereal : 48.00%
    Sorgho        : 28.19%
    Leguminous    : 70.90%
    Fodder        : 43.67%
    Ligneous      : 60.60%
    Grapevine     : 14.30%
    Test Overall Accuracy: 63.18% Test mIoU : 47.07%  Test Loss: 1.0070

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch  11 -> Train Overall Accuracy: 69.31% Train mIoU : 48.12% Train Loss: 0.8608

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch  12 -> Train Overall Accuracy: 69.36% Train mIoU : 48.16% Train Loss: 0.8575

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch  13 -> Train Overall Accuracy: 69.54% Train mIoU : 48.63% Train Loss: 0.8425

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch  14 -> Train Overall Accuracy: 69.72% Train mIoU : 48.60% Train Loss: 0.8501

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch  15 -> Train Overall Accuracy: 71.06% Train mIoU : 49.99% Train Loss: 0.8253

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=313.0), HTML(value='')), layout=Layout(di…

    Meadow        : 65.71%
    Durum Wheat   : 50.38%
    Spring Cereal : 51.51%
    Sorgho        : 35.77%
    Leguminous    : 75.67%
    Fodder        : 40.29%
    Ligneous      : 61.99%
    Grapevine     : 16.75%
    Test Overall Accuracy: 65.62% Test mIoU : 49.76%  Test Loss: 0.9554

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch  16 -> Train Overall Accuracy: 70.41% Train mIoU : 49.51% Train Loss: 0.8343

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch  17 -> Train Overall Accuracy: 70.27% Train mIoU : 49.78% Train Loss: 0.8348

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch  18 -> Train Overall Accuracy: 71.23% Train mIoU : 50.75% Train Loss: 0.8146

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch  19 -> Train Overall Accuracy: 70.78% Train mIoU : 49.75% Train Loss: 0.8212

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch  20 -> Train Overall Accuracy: 69.62% Train mIoU : 48.37% Train Loss: 0.8298

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=313.0), HTML(value='')), layout=Layout(di…

    Meadow        : 63.90%
    Durum Wheat   : 48.79%
    Spring Cereal : 50.28%
    Sorgho        : 32.00%
    Leguminous    : 77.08%
    Fodder        : 47.30%
    Ligneous      : 58.69%
    Grapevine     : 11.66%
    Test Overall Accuracy: 64.86% Test mIoU : 48.71%  Test Loss: 0.9431

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch  21 -> Train Overall Accuracy: 70.60% Train mIoU : 49.64% Train Loss: 0.8152

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch  22 -> Train Overall Accuracy: 71.10% Train mIoU : 50.60% Train Loss: 0.8119

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch  23 -> Train Overall Accuracy: 70.77% Train mIoU : 49.66% Train Loss: 0.8023

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch  24 -> Train Overall Accuracy: 71.26% Train mIoU : 50.48% Train Loss: 0.7937

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch  25 -> Train Overall Accuracy: 71.31% Train mIoU : 50.77% Train Loss: 0.8073

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=313.0), HTML(value='')), layout=Layout(di…

    Meadow        : 0.00%
    Durum Wheat   : 48.68%
    Spring Cereal : 50.31%
    Sorgho        : 31.31%
    Leguminous    : 64.95%
    Fodder        : 42.02%
    Ligneous      : 59.69%
    Grapevine     : 22.77%
    Test Overall Accuracy: 61.28% Test mIoU : 39.97%  Test Loss: 1.0298

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch  26 -> Train Overall Accuracy: 71.65% Train mIoU : 51.19% Train Loss: 0.8079

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch  27 -> Train Overall Accuracy: 71.78% Train mIoU : 51.06% Train Loss: 0.7956

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch  28 -> Train Overall Accuracy: 71.76% Train mIoU : 51.63% Train Loss: 0.7875

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch  29 -> Train Overall Accuracy: 71.64% Train mIoU : 50.77% Train Loss: 0.7900

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch  30 -> Train Overall Accuracy: 71.68% Train mIoU : 51.11% Train Loss: 0.7938

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=313.0), HTML(value='')), layout=Layout(di…

    Meadow        : 61.36%
    Durum Wheat   : 49.62%
    Spring Cereal : 49.95%
    Sorgho        : 31.73%
    Leguminous    : 73.47%
    Fodder        : 43.86%
    Ligneous      : 58.84%
    Grapevine     : 22.38%
    Test Overall Accuracy: 64.96% Test mIoU : 48.90%  Test Loss: 0.9487

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch  31 -> Train Overall Accuracy: 72.40% Train mIoU : 52.45% Train Loss: 0.7806

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch  32 -> Train Overall Accuracy: 71.56% Train mIoU : 50.72% Train Loss: 0.7844

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch  33 -> Train Overall Accuracy: 72.09% Train mIoU : 51.74% Train Loss: 0.7762

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch  34 -> Train Overall Accuracy: 72.20% Train mIoU : 51.73% Train Loss: 0.7805

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch  35 -> Train Overall Accuracy: 72.24% Train mIoU : 52.08% Train Loss: 0.7770

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=313.0), HTML(value='')), layout=Layout(di…

    Meadow        : 60.70%
    Durum Wheat   : 52.63%
    Spring Cereal : 49.62%
    Sorgho        : 36.06%
    Leguminous    : 72.64%
    Fodder        : 41.66%
    Ligneous      : 60.27%
    Grapevine     : 13.25%
    Test Overall Accuracy: 65.10% Test mIoU : 48.35%  Test Loss: 0.9513

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch  36 -> Train Overall Accuracy: 72.23% Train mIoU : 52.49% Train Loss: 0.7765

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch  37 -> Train Overall Accuracy: 72.65% Train mIoU : 52.67% Train Loss: 0.7648

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch  38 -> Train Overall Accuracy: 72.22% Train mIoU : 52.11% Train Loss: 0.7663

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch  39 -> Train Overall Accuracy: 72.93% Train mIoU : 52.86% Train Loss: 0.7573

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch  40 -> Train Overall Accuracy: 72.56% Train mIoU : 52.52% Train Loss: 0.7580

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=313.0), HTML(value='')), layout=Layout(di…

    Meadow        : 0.40%
    Durum Wheat   : 48.75%
    Spring Cereal : 49.31%
    Sorgho        : 34.08%
    Leguminous    : 70.37%
    Fodder        : 48.98%
    Ligneous      : 63.92%
    Grapevine     : 25.52%
    Test Overall Accuracy: 63.36% Test mIoU : 42.66%  Test Loss: 0.9716

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch  41 -> Train Overall Accuracy: 72.35% Train mIoU : 52.20% Train Loss: 0.7674

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch  42 -> Train Overall Accuracy: 72.10% Train mIoU : 52.10% Train Loss: 0.7590

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch  43 -> Train Overall Accuracy: 72.71% Train mIoU : 52.83% Train Loss: 0.7606

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch  44 -> Train Overall Accuracy: 72.92% Train mIoU : 52.66% Train Loss: 0.7579

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch  45 -> Train Overall Accuracy: 72.74% Train mIoU : 52.31% Train Loss: 0.7613

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=313.0), HTML(value='')), layout=Layout(di…

    Meadow        : 48.80%
    Durum Wheat   : 49.24%
    Spring Cereal : 51.30%
    Sorgho        : 37.01%
    Leguminous    : 74.77%
    Fodder        : 43.83%
    Ligneous      : 61.84%
    Grapevine     : 20.02%
    Test Overall Accuracy: 64.92% Test mIoU : 48.35%  Test Loss: 0.9792

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch  46 -> Train Overall Accuracy: 73.06% Train mIoU : 52.71% Train Loss: 0.7546

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch  47 -> Train Overall Accuracy: 72.94% Train mIoU : 53.03% Train Loss: 0.7601

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch  48 -> Train Overall Accuracy: 72.29% Train mIoU : 51.94% Train Loss: 0.7623

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=625.0), HTML(value='')), layout=Layout(di…

    Epoch  49 -> Train Overall Accuracy: 72.97% Train mIoU : 53.20% Train Loss: 0.7575

    HBox(children=(FloatProgress(value=0.0, layout=Layout(flex='2'), max=313.0), HTML(value='')), layout=Layout(di…

    Meadow        : 22.81%
    Durum Wheat   : 48.21%
    Spring Cereal : 50.58%
    Sorgho        : 32.01%
    Leguminous    : 72.38%
    Fodder        : 45.21%
    Ligneous      : 58.00%
    Grapevine     : 27.73%
    Test Overall Accuracy: 63.38% Test mIoU : 44.62%  Test Loss: 0.9711

## Convolutional Classifier

$$\newcommand{\Conv}[2]{\text{Conv}_{#1}\left(#2\right)}$$
$$\newcommand{\FC}[2]{\text{FC}_{#1}\left(#2\right)}$$
$$\newcommand{\BN}[1]{\text{BatchNorm}\left(#1\right)}$$
$$\newcommand{\RELU}[1]{\text{ReLU}\left(#1\right)}$$
$$\newcommand{\MAX}[1]{\text{Maxpool}\left(#1\right)}$$
$$\newcommand{\MEAN}[1]{\text{Mean}\left(#1\right)}$$

We will now define a CNN networ, more appropriate for image classification.

The input is an image of size $C\times W\times H$ with C the number of bands (here 10) and W and H the width and height (here 32). The output is a vector of size $K$ (the number of classes , here $8$).

The CNN is composed of a sequence of convolutions to extract local radiometric features and maxpooling operations to reduce the size of the feature map. Then the learned features are averages spatially and to obtain a vector of aggregated feature. Finally, a fully connected layer maps this vector to the class vector.

The equations describing the function defined by the MLP are as follows:

\begin{align}
x_1  &= \RELU{\BN{\Conv{1}{\text{input}}}}\\
x_2  &= \RELU{\BN{\Conv{2}{x_1}}}\\
x_3  &= \MAX{x_2}\\
x_4  &= \RELU{\BN{\Conv{3}{x_3}}}\\
x_5  &= \RELU{\BN{\Conv{4}{x_4}}}\\
x_6  &= \MEAN{x_5}\\
\text{output}  &= \FC{1}{x6}\\
\end{align}

The size of all tensors is given in the following Table:

> **Tensor**           |    **          Shape of Vectors                                       **                          |
> -------------------- | --------------------------------------|
> input                |  $C\times W \times H$                           |
> $x_1$                |  $d_1\times W \times H$                         |
> $x_2$                |  $d_2\times W \times H$                         |
> $x_3$                |  $d_2\times \lceil{{W}/{2}}\rceil \times \lceil{{H}/{2}}\rceil$    |
> $x_4$                |  $d_3\times \lceil{{W}/{2}}\rceil \times \lceil{{H}/{2}}\rceil$                         |
> $x_5$                |  $d_4\times \lceil{{W}/{2}}\rceil \times \lceil{{H}/{2}}\rceil$                         |
> $x_6$                |  $d_4$|
> output               |  $K$

Q8) Complete the following cell to define the convolutional classsifier. Make sure that the assert passes.

```
class CNN(nn.Module):
  """
  CNN network for image classification
  """
  def __init__(self, n_channels, conv_width,  n_class, cuda = 1):
    """
    initialization function
    n_channels, int, number of input channel
    hidden_width, int list, width of hidden convolutional layers (4 values)
    n_class = int,  the number of classes
    """
    super(CNN, self).__init__() #necessary for all classes extending the module class
    self.is_cuda = cuda
    self.class_weight = None
    self.maxpool = nn.MaxPool2d(2,2) #maxpooling layer

    self.conv1 = nn.Sequential(
        nn.Conv2d(in_channels=n_channels, out_channels=conv_width[0] , kernel_size=3, padding=1, padding_mode='reflect'), \
        nn.BatchNorm2d(num_features=conv_width[0]), \
        nn.ReLU(True))
    self.conv2 = nn.Sequential( \
        nn.Conv2d(in_channels=#TODO, out_channels=#TODO, kernel_size=3, padding=1, padding_mode='reflect'), \
        nn.BatchNorm2d(num_features=#TODO), \
        nn.ReLU(True))
    self.conv3 = nn.Sequential( \
        nn.Conv2d(in_channels=#TODO, out_channels=#TODO , kernel_size=3, padding=1, padding_mode='reflect'), \
        nn.BatchNorm2d(num_features=#TODO), \
        nn.ReLU(True))
    self.conv4 = nn.Sequential( \
        nn.Conv2d(in_channels=#TODO, out_channels=#TODO, kernel_size=3, padding=1, padding_mode='reflect'), \
        nn.BatchNorm2d(num_features=#TODO), \
        nn.ReLU(True))

    self.fc = nn.Linear(in_features=#TODO,out_features=#TODO)

    if cuda: #put the model on the GPU memory
      self.cuda()

  def forward(self,input):
    """
    the foward pass : maps input to output
    """
    if self.is_cuda:
      input = input.cuda()

    x1 = self.conv1(input)
    x2 = #TODO
    x3 = self.maxpool(x2)
    x4 = #TODO
    x5 = #TODO
    x6 = x5.mean(dim=(2,3))
    output = #TODO
    return output
```

```
#==================TEST===============================
#load a batch of 2 patches
patch1, label = patch_loader(0)
patch2, label = patch_loader(1)
patches = torch.cat((patch1[None,:,:,:],patch2[None,:,:,:]),0)
cnn = CNN(10,[16,32,64,64],8)
print(cnn)
print('Total number of parameters: {}'.format(sum([p.numel() for p in cnn.parameters()])))
pred = cnn(patches)
assert(pred.shape == torch.Size([2,8]))
```

Q9) Complete the following cell, defining the learning procedure for the CNN.

```
args = mock.Mock() #stores the parameters
args.n_epoch = 50
args.n_epoch_test = int(5)
args.batch_size = 16
args.n_class = 8
args.n_channel = 10
args.conv_width = [16,64,64,64]
args.cuda = 1
args.lr = 5e-3
args.average = False

#create the model to train
model = CNN(#TODO
print('Total number of parameters: {}'.format(sum([p.numel() for p in model.parameters()])))

trained_model_CNN = train_full(#TODO
```

Q10)  Complete the following function which visualizes the feature maps at each step of the model. Launch this viewer on a fully trained model.

```
def view_patch_false_color(feat_map, ax):
  """
  visualize a feature_map in RGB space
  """
  fmap_dim = feat_map.shape[1]
  n_pix = feat_map.shape[2]
  #we use a pca to project the emebddings to a RGB space
  pca = PCA(n_components=3)
  pca.fit(np.eye(fmap_dim))
  #we need to adapt dimension and memory allocation to CPU
  fmap_ = feat_map.cpu().detach().numpy().squeeze()
  fmap_ = fmap_.reshape((fmap_dim, n_pix * n_pix)).transpose(1,0)
  color = pca.transform(fmap_)
  #we normalize for visibility
  color = np.maximum(np.minimum(((color - color.mean(1, keepdims = True) +0.5) / (2 * color.std(1, keepdims = True))), 1), 0)
  color = color.reshape((n_pix, n_pix,3), order= 'F')
  ax.imshow(color)
  plt.axis('off')

def view_embeddings(model, patch_index = None, train = False):
  """
  view the feature maps at all levels for a give model
  """
  if patch_index is None:
    patch_index = np.random.randint(n_train) if train \
    else np.random.randint(n_test)
  patch, label = patch_loader(patch_index)

  if model.is_cuda:
    patch = patch.cuda()

  #forms a batch of size 1
  input = patch[None,:,:,:]
  with torch.no_grad(): #we do not need  to keep track of gradients here
    x1 = #TODO
    x2 = #TODO
    x3 = #TODO
    x4 = #TODO
    x5 = #TODO
    x6 = #TODO

  fig = plt.figure(figsize=(25, 10)) #adapted dimension
  ax = fig.add_subplot(1, 7, 1, aspect=1)

  ax.set(title='Input : %d x %d x %d \n %s' %(input.shape[1],input.shape[2],input.shape[3], class_names[label]))
  view_patch_RGB(patch.cpu(), ax)
  ax = fig.add_subplot(1, 7, 2, aspect=1)
  ax.set(title='x1 : %d x %d x %d' %(x1.shape[1:]))
  view_patch_false_color(x1, ax)
  ax = fig.add_subplot(1, 7, 3, aspect=1)
  ax.set(title='x2 : %d x %d x %d' %(x2.shape[1:]))
  view_patch_false_color(x2, ax)
  ax = fig.add_subplot(1, 7, 4, aspect=1)
  ax.set(title='x3 : %d x %d x %d' %(x3.shape[1:]))
  view_patch_false_color(x3, ax)
  ax = fig.add_subplot(1, 7, 5, aspect=1)
  ax.set(title='x4 : %d x %d x %d' %(x4.shape[1:]))
  view_patch_false_color(x4, ax)
  ax = fig.add_subplot(1, 7, 6, aspect=1)
  ax.set(title='x5 : %d x %d x %d' %(x5.shape[1:]))
  view_patch_false_color(x5, ax)
  ax = fig.add_subplot(1, 7, 7, aspect=1)
  ax.set(title='x6 : %d' %(x6.shape[1:]))
  view_patch_false_color(x6[:, :, None, None], ax)
```

```
for i in range(3):
  view_embeddings(trained_model_CNN)
```

**Data from**
Garnot, V. S. F., Landrieu, L., Giordano, S., & Chehata, N. Satellite Image Time Series Classification with Pixel-Set Encoders and Temporal Self-Attention. In CVPR 2020.

https://github.com/VSainteuf/pytorch-psetae


```python
%matplotlib inline
```

# Propagation errors caused by a too narrow frequency window

This example demonstrates that if the computational domain does not support
the propagation scenario in an adequate manner, errors accumulate and results
will give a wrong impression of the dynamics.

Here, the simple split-step Fourier method is used and the propagation of a
third-order soliton is considered.

.. codeauthor:: Oliver Melchert <melchert@iqo.uni-hannover.de>

We first import the functionality needed to perform the sequence of numerical
experiments:

```python
import sys; sys.path.append('../../')
import numpy as np
import numpy.fft as nfft
from fmas.models import ModelBaseClass
from fmas.config import FTFREQ, FT, IFT, C0
from fmas.solver import SiSSM, SySSM, IFM_RK4IP, LEM_SySSM, CQE
from fmas.grid import Grid
from fmas.tools import plot_evolution
```

Next, we implement a model for the nonlinear Schrödinger equation.  In
particular, we here consider the standard nonlinear Schrödinger equation,
given by

\begin{align}\partial_z u = -i \frac{\beta_2}{2}\partial_t^2 u + i\gamma |u|^2 u,\end{align}

wherein $u = u(z, t)$ represents the slowly varying pulse envelope,
$\beta_2=-1$ is the second order dispersion parameter, and
$\gamma=1$ is the nonlinear parameter:

```python
class NSE(ModelBaseClass):

    def __init__(self, w, b2 = -1.0, gamma = 1.):
        super().__init__(w, 0.5*b2*w*w)
        self.gamma = gamma

    @property
    def Lw(self):
        return 1j*self.beta_w

    def Nw(self, uw):
        ut = IFT(uw)
        return 1j*self.gamma*FT(np.abs(ut)**2*ut)
```

Next, we initialize the computational domain and use a simple split-step
Fourier method to propagate a single third-order soliton for six soliton
periods.
In this first numerical experiment, the extend of the frequency domain is so
small that, when the solitons spectrum broadens, it exceeds the bounds of the
frequency domain. Errors stemming from truncation of the spectrum accumulate
over subsequent soliton periods, giving an erroneous account of the true
dynamics (see the subsequent figure).

```python
# -- INITIALIZATION STAGE
# ... COMPUTATIONAL DOMAIN
grid = Grid( t_max = 34., t_num = 2**9)
t, w = grid.t, grid.w

# ... NSE MODEL
model = NSE(w, b2=-1., gamma=1.)
# ... INITIAL CONDITION
u_0t = 3./np.cosh(t)

solver = SiSSM(model.Lw, model.Nw)
solver.set_initial_condition(w, FT(u_0t))
solver.propagate(z_range = 6*np.pi/2, n_steps = 10000, n_skip = 50)
z, utz = solver.z_, solver.utz

plot_evolution( solver.z, grid.t, solver.utz,
    t_lim = (-5,5), w_lim = (-60,60), DO_T_LOG=False)
```

In the subsequent numerical experiment, the extend of the frequency domain is
increased to fully support the third-order soliton in those propagation
stages where its spectrum is maximally broad. As a result, the periodic
dynamics of the higher order soliton is well represented (see the subsequent
figure).

```python
grid = Grid( t_max = 34., t_num = 2**11)
t, w = grid.t, grid.w
model = NSE(w, b2=-1., gamma=1.)
u_0t = 3./np.cosh(t)

solver = SiSSM(model.Lw, model.Nw)
solver.set_initial_condition(w, FT(u_0t))
solver.propagate(z_range = 6*np.pi/2, n_steps = 10000, n_skip = 50)
z, utz = solver.z_, solver.utz

plot_evolution( solver.z, grid.t, solver.utz,
    t_lim = (-5,5), w_lim = (-60,60), DO_T_LOG=False)
```


# Deflection Profile Parametric Study for El Ghadioui2020 Cantilever
El Ghadioui2020_Structural behaviour of CFRP reinforced concrete members under bending and shear loads

```python
# %matplotlib widget
%matplotlib qt
```

```python
from bmcs_cross_section.api import MKappa, TShape, EC2, BarLayer, ReinfLayer
from bmcs_beam.api import BoundaryConditions, BoundaryConfig, DeflectionProfile, BeamDesign, LoadDeflectionParamsStudy
from os.path import join
import numpy as np
import pandas as pd
import sympy as sp
import matplotlib.pyplot as plt
import bmcs_utils.api as bu
```

```python
# Beam B-M-C-K1 - CFRP Bars

# C 50/60
def get_dp(f_ck = None, E=None, low_kappa = None, high_kappa = None, n_kappa=None, n_load_steps=None, eps_tu=None):
    if eps_tu is None:
        eps_tu = 0.00008

    if n_load_steps is None:
        n_load_steps = 31

    if E is None:
        E = 33525

    if high_kappa is None:
        high_kappa = 0.00001

    if n_kappa is None:
        n_kappa = 200

    if low_kappa is None:
        low_kappa = -0.00012

    mc = MKappa(low_kappa = low_kappa, high_kappa = high_kappa, n_kappa = n_kappa)

#     mc.cs_design.matrix='EC2 with plateau'
    mc.cs_design.matrix_.trait_set(
            E_cc=E,
            E_ct=E,
            eps_cr=0.00006, #0.000125 * f_ck/30, # f_ctm /E,
            eps_cy=0.00175,
            eps_cu=0.0035,
            eps_tu=eps_tu,
            mu=0.0,
        f_cd = 60, # f_ck * 0.85/1.5,
        factor = 1
        )

    mc.cross_section_shape_.B = 400
    mc.cross_section_shape_.H = 200

    bl1 = ReinfLayer(name='carbon', z=200-35, A=4 * 70.14, matmod='carbon')
    bl1.matmod_.trait_set(E=230000, f_t=3600, factor = 1.)
    mc.cross_section_layout.add_layer(bl1)

    dp = DeflectionProfile(mc=mc, n_load_steps = n_load_steps)

    dp.beam_design.beam_conf_name = BoundaryConfig.CANTILEVER_DIST_LOAD
    dp.beam_design.L = 1000
    dp.F_scale = 1
    return dp

get_dp().interact()
```

    VBox(children=(HBox(children=(VBox(children=(Tree(layout=Layout(align_items='stretch', border='solid 1px black…

```python
# Test E (concrete)
E_list = np.array([30000, 35000, 40000, 45000])

dp = get_dp(n_load_steps = 100)

# following fix is just to keep the sig, and not eps the same when changing E in MULTI-LINEAR Model
E = dp.mc.cs_design.matrix_.E_cc
eps_cy = dp.mc.cs_design.matrix_.eps_cy
eps_cr= dp.mc.cs_design.matrix_.eps_cr
sig_cy = E * eps_cy
sig_cr = E * eps_cr

sig_cy_list = np.full_like(E_list, sig_cy)
sig_cr_list = np.full_like(E_list, sig_cr)

# params_config = [
#     [['E_cc', dp.mc.cs_design.matrix_, E_list],
#      ['E_ct', dp.mc.cs_design.matrix_, E_list]],
# ]
params_config = [
    [['E_cc', dp.mc.cs_design.matrix_, E_list],
     ['E_ct', dp.mc.cs_design.matrix_, E_list],
     ['eps_cy', dp.mc.cs_design.matrix_, sig_cy_list / E_list],
     ['eps_cr', dp.mc.cs_design.matrix_, sig_cr_list / E_list]],
]
LoadDeflectionParamsStudy(dp).run(params_config)
```

    Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …

    Parametric study is running...
    E_cc, E_ct, eps_cy, eps_cr: (30000, 30000, -0.0019333333333333333, 6.666666666666667e-05), (35000, 35000, -0.0016571428571428572, 5.714285714285714e-05), (40000, 40000, -0.00145, 5e-05), (45000, 45000, -0.001288888888888889, 4.4444444444444447e-05),
    Parametric study finished.

```python
# Test E_carbon
E_carbon = [130000, 170000, 210000, 250000]

dp = get_dp(n_kappa = 500)
E_cc = dp.mc.cs_design.matrix_.E_cc
E_list = [E_cc, E_cc + 0.01, E_cc, E_cc + 0.01] # Just to update state because changing E_carbon doesn't update the state automatically
params_config = [
    [['E', dp.mc.cross_section_layout.items[0].matmod_, E_carbon],
     ['E_cc', dp.mc.cs_design.matrix_, E_list]]
]
LoadDeflectionParamsStudy(dp).run(params_config)
```

    Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …

    Parametric study is running...
    E, E_cc: (130000, 33525.0), (170000, 33525.01), (210000, 33525.0), (250000, 33525.01),
    Parametric study finished.

```python
# Test carbon strength
f_t_list = [1800, 2200, 2600, 3000]
dp = get_dp()
E_cc = dp.mc.cs_design.matrix_.E_cc
E_list = [E_cc, E_cc + 0.01, E_cc, E_cc + 0.01] # Just to update state because changing E_carbon doesn't update the state automatically
params_config = [
    [['f_t', dp.mc.cross_section_layout.items[0].matmod_, list(reversed(f_t_list))],
     ['E_cc', dp.mc.cs_design.matrix_, E_list]]
]
LoadDeflectionParamsStudy(dp).run(params_config)
```

    Parametric study is running...
    f_t, E_cc: (3000, 33525.0), (2600, 33525.01), (2200, 33525.0), (1800, 33525.01),
    Parametric study finished.

```python
# Test concrete compressive strength
f_cm_list = np.array([30, 50, 70, 90])

dp = get_dp(n_kappa=200, n_load_steps=100)

E = dp.mc.cs_design.matrix_.E_cc
E_list = np.full_like(f_cm_list, E)

params_config = [
     ['eps_cy', dp.mc.cs_design.matrix_, f_cm_list / E_list],
]
LoadDeflectionParamsStudy(dp).run(params_config)
```

    Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …

    Parametric study is running...
    eps_cy: (0.0008948545861297539), (0.0014914243102162564), (0.002087994034302759), (0.0026845637583892616),
    Parametric study finished.

```python
# Test concrete tensile strength
f_ctd_list = np.array([0, 2.8, 4, 4.5, 5])

dp = get_dp(eps_tu = 2*0.00008, n_load_steps=200)

E = dp.mc.cs_design.matrix_.E_cc
E_list = np.full_like(f_ctd_list, E)

params_config = [
     ['eps_cr', dp.mc.cs_design.matrix_, f_ctd_list / E_list],
]
LoadDeflectionParamsStudy(dp).run(params_config)
```

    Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …

    Parametric study is running...
    eps_cr: (0.0), (8.351976137211036e-05), (0.00011931394481730052), (0.00013422818791946307), (0.00014914243102162565),
    Parametric study finished.


# ***Introduction to Radar Using Python and MATLAB***
## Andy Harrison - Copyright (C) 2019 Artech House
<br/>

# Stretch Processor
***

Referring to Section 8.4, stretch processing is an intrapulse compression technique used for processing waveforms of very large bandwidth.  It was originally developed to reduce the sampling requirements for large bandwidth, high range resolution systems such as synthetic aperture radar (SAR) systems.  While stretch processing may be applied to various forms of wide bandwidth waveforms, this discussion is focused on linear frequency modulation (LFM) waveforms.  A block diagram of a stretch processor is given in Figure 8.7, and in general, stretch processing consists of the following basic steps:

1. Mix the return signal with a reference signal.
2. Low-pass filter the mixed signals.
3. Analog-to-digital conversion of the low-pass filtered signals.
4. Fourier transform of the digital signals.

The first step in stretch processing is to mix the return signal with a reference signal.  The reference signal is typically generated with the same frequency slope as the transmitted signal.  The transmitted LFM signal may be written as (Equation 8.36)

\begin{equation}
    s(t) = \exp\left[j\, 2 \pi \left(f_0 t + \frac{\beta}{2} t ^ 2\right)\right] \hspace{0.5in} 0\le t \le \tau_p,
\end{equation}

The return signal from a point scatterer is written as (Equation 8.37)

\begin{equation}
    s_r(t) = A\, \exp\left[j\, 2 \pi \left(f_0(t - t_0) + \frac{\beta}{2}(t - t_0) ^ 2\right)\right],
\end{equation}

The factor, $A$, is a constant related to the target's radar cross section, antenna gain, path loss, and other factors.  The reference signal is written as (Equation 8.38)

\begin{equation}
    s_{ref}(t) = \exp\left[ j\, 2 \pi \left( f_0(t - t_{ref}) + \frac{\beta}{2}(t - t_{ref}) ^ 2 \right) \right],
\end{equation}

where $t_{ref}$ is the reference time of the signal, which must closely match $t_0$. This leads to the following requirement on the pulsewidth of the reference signal (Equation 8.39)

\begin{equation}
    \tau_{ref} \ge t_r + \tau_p \hspace{0.5in} \text{(s)},
\end{equation}

where $t_r$ is the range of time over which stretch processing is to be employed.  This is expressed as $t_r = t_{max} - t_{min}$.  The second requirement is $ t_{ref} - t_{min} \le t_0 \le t_{max} + t_{ref}$. Now that the requirements on the reference signal have been specified, the output of the mixing step is written as (Equation 8.40)

\begin{equation}\label{eq:sp_output_mixed}
    s_o(t) = A\, \exp\Big[ j\, 2 \pi f_0 (t_0 - t_{ref}) + j\, 2 \pi \beta (t_0 - t_{ref})t + j\, \pi \beta (t_{ref}^2 - t_0^2) \Big].
\end{equation}

The instantaneous frequency is the time derivative of the phase, which is written as (Equation 8.41)

\begin{equation}
    f_i = \frac{1}{2 \pi}\frac{d\phi(t)}{dt} \hspace{0.5in} \text{(Hz)},
\end{equation}

where $\phi(t)$ is the phase of the signal which is simply (Equation 8.42)

\begin{equation}
    \phi(t) = 2 \pi f_0 (t_0 - t_{ref}) + 2 \pi \beta (t_0 - t_{ref})t + \pi \beta \left(t_{ref}^2 - t_0^2\right) \hspace{0.25in} \text{(rad)}.
\end{equation}

The instantaneous frequency is then (Equation 8.43)

\begin{equation}\label{eq:inst_freq_sp}
    f_i = \beta(t_0 - t_{ref}) \hspace{0.5in} \text{(Hz)}.
\end{equation}

This allows the time delay of the target to be written as (Equation 8.44)

\begin{equation}
    t_0 = \frac{f_i}{\beta} + t_{ref} \hspace{0.5in} \text{(s)}.
\end{equation}

Therefore, the target range is related to the instantaneous frequency as (Equation 8.45)
\begin{equation}
    r_0 = \frac{c}{2}\bigg(\frac{f_i}{B}\tau_p + t_{ref}\bigg) \hspace{0.5in} \text{(m)}.
\end{equation}

The expression in (Equation 8.43) indicates the resulting instantaneous frequency is constant and related to the range to the target. Performing analog-to-digital sampling on the signal output from the mixer only requires sampling at the maximum instantaneous frequency in (Equation 8.43), rather than the full bandwidth.
***

Set the bandwidth (Hz), the pulsewidth (s), the length of the range window (m), the target range (m) and the target radar cross section (m^2)

```python
bandwidth = 1e9

pulsewidth = 10e-4

range_window_length = 50.0

target_range = [10.0, 20.0, 30.0]

target_rcs = [1.0, 10.0, 3.0]
```

Calculate the number of samples for stretch processing

```python
from numpy import ceil

from scipy.constants import c

number_of_samples = int(ceil(4 * bandwidth * range_window_length / c))
```

Set the window type

```python
window_type = 'Rectangular'
```

Get the window coefficients using `scipy.signal.windows`

```python
from scipy.signal.windows import hann, hamming, blackmanharris, kaiser

from numpy import ones

if window_type == 'Kaiser':

    coefficients = kaiser(number_of_samples, 6, True)

elif window_type == 'Blackman-Harris':

    coefficients = blackmanharris(number_of_samples, True)

elif window_type == 'Hanning':

    coefficients = hann(number_of_samples, True)

elif window_type == 'Hamming':

    coefficients = hamming(number_of_samples, True)

elif window_type == 'Rectangular':

    coefficients = ones(number_of_samples)
```

Get the time sampling using the `linspace` routine from `scipy`

```python
from numpy import linspace

t, dt = linspace(-0.5 * pulsewidth, 0.5 * pulsewidth, number_of_samples, retstep=True)
```

Calculate the sampled signal after the mixing process

```python
from numpy import zeros, sqrt, exp

from scipy.constants import pi

so = zeros(number_of_samples, dtype=complex)

for r, rcs in zip(target_range, target_rcs):

    so += sqrt(rcs) * exp(1j * 2.0 * pi * bandwidth / pulsewidth * (2 * r / c) * t)
```

Perform the FFT on the sampled signal using the routines from `fftpack`

```python
from scipy.fftpack import fftfreq, fft, fftshift

# Fourier transform

so = fftshift(fft(so * coefficients, 4 * number_of_samples))

# FFT frequencies

frequencies = fftshift(fftfreq(4 * number_of_samples, dt))
```

Calculate the range window (m) from the frequencies (Hz)

```python
# Range window

range_window = 0.5 * frequencies * c * pulsewidth / bandwidth
```

Display the results of stretch processing using the `matplotlib` routines

```python
from matplotlib import pyplot as plt

from numpy import log10, finfo

# Set the figure size

plt.rcParams["figure.figsize"] = (15, 10)

# Create the line plot

plt.plot(range_window, 20.0 * log10(abs(so) / number_of_samples + finfo(float).eps), '')

plt.xlim(min(target_range) - 5, max(target_range) + 5)

plt.ylim(-60, max(20.0 * log10(abs(so) / number_of_samples)) + 10)

# Set the x and y axis labels

plt.xlabel("Range (m)", size=12)

plt.ylabel("Amplitude (dBsm)", size=12)

# Turn on the grid

plt.grid(linestyle=':', linewidth=0.5)

# Set the plot title and labels

plt.title('Stretch Processor Range Profile', size=14)

# Set the tick label size

plt.tick_params(labelsize=12)
```

```python

```


# The Discrete-Time Fourier Transform

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Comunications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Properties

The discrete-time Fourier transform (DTFT) has a number of specific properties that are reviewed in the following.

### Invertibility

For many types of signals it is possible to recover the discrete signal $x[k]$ from its DTFT $X(e^{j \Omega}) = \mathcal{F}_* \{ x[k] \}$

\begin{equation}
x[k] = \mathcal{F}_*^{-1} \left\{ \mathcal{F}_* \{ x[k] \} \right\}
\end{equation}

A sufficient condition for the theorem to hold is that both the signal $x[k]$ and its DTFT are absolutely summable/integrable. For this type of signals, above relation can be proven by applying the definition of the DTFT and its inverse and rearranging terms.

**Example**

The invertibility of the DTFT is illustrated at the example of the [complex exponential signal](../discrete_signals/standard_signals.ipynb#Complex-Exponential-Signal) $x[k] = e^{j \Omega_0 k}$ [whose DTFT is given as](definition.ipynb#Transformation-of-the-Exponential-Signal) $X(j \omega) = {\bot \!\! \bot \!\! \bot} ( \frac{\Omega - \Omega_0}{2 \pi} )$. Note that the signal nor its spectrum are absolutely integrable. However, the invertibility still holds as is shown by evaluating the [integral of the inverse DTFT](definition.ipynb#Definition). Since the integration is only performed in the range $\Omega = -\pi$ to $\pi$, it is sufficient to consider a single Dirac impulse $2 \pi \cdot \delta(\Omega - \Omega_0)$ instead of the Dirac comb for the computation.

```python
import sympy as sym
%matplotlib inline
sym.init_printing()

k = sym.symbols('k', integer=True)
W, W0 = sym.symbols('Omega Omega0', real=True)

X = 2*sym.pi*sym.DiracDelta(W - W0)
x = 1/(2*sym.pi) * sym.integrate(X * sym.exp(sym.I*W*k), (W, -sym.pi, sym.pi))
x
```

This result includes the restriction of the normalized angular frequency to $-\pi < \Omega_0 < \pi$ due to the usage of a single Dirac impulse instead of the Dirac comb. The result is specialized to $\Omega_0 = \frac{1}{2}$ in order to show that above result indeed constitutes a complex exponential signal.

```python
x.subs(W0, sym.S.Half)
```

### Linearity

The DTFT is a linear operation. For two signals $x_1[k]$ and $x_2[k]$ with transforms $X_1(e^{j \Omega}) = \mathcal{F}_* \{ x_1[k] \}$ and $X_2(e^{j \Omega}) = \mathcal{F}_* \{ x_2[k] \}$ the following holds

\begin{equation}
\mathcal{F}_* \{ A \cdot x_1[k] + B \cdot x_2[k] \} = A \cdot X_1(e^{j \Omega}) + B \cdot X_2(e^{j \Omega})
\end{equation}

with $A, B \in \mathbb{C}$. The DTFT of a weighted superposition of discrete signals is equal to the weighted superposition of the individual DTFTs. This property is useful to derive the DTFT of signals that can be expressed as superposition of other signals for which the DTFT is known or can be calculated easier. Linearity holds also for the inverse DTFT.

#### Transformation of the cosine and sine signal

The DTFT of $\cos(\Omega_0 k)$ and $\sin(\Omega_0 k)$ is derived by expressing both as harmonic exponential signals using [Euler's formula](https://en.wikipedia.org/wiki/Euler's_formula)

\begin{align}
\cos(\Omega_0 k) &= \frac{1}{2} \left(  e^{-j \Omega_0 k} + e^{j \Omega_0 k} \right) \\
\sin(\Omega_0 k) &= \frac{j}{2} \left( e^{-j \Omega_0 k} - e^{j \Omega_0 k}  \right)
\end{align}

together with the DTFT $\mathcal{F}_* \{ e^{j \Omega_0 k} \} = {\bot \!\! \bot \!\! \bot} ( \frac{\Omega - \Omega_0}{2 \pi} )$ of the complex exponential signal yields

\begin{align}
\mathcal{F} \{ \cos(\Omega_0 k) \} &= \frac{1}{2} \left[ {\bot \!\! \bot \!\! \bot} \left( \frac{\Omega + \Omega_0}{2 \pi} \right) + {\bot \!\! \bot \!\! \bot} \left( \frac{\Omega - \Omega_0}{2 \pi} \right)  \right] \\
\mathcal{F} \{ \sin(\Omega_0 k) \} &= \frac{j}{2} \left[ {\bot \!\! \bot \!\! \bot} \left( \frac{\Omega + \Omega_0}{2 \pi} \right) - {\bot \!\! \bot \!\! \bot} \left( \frac{\Omega - \Omega_0}{2 \pi} \right)  \right]
\end{align}

### Symmetries

In order to investigate the symmetries of the DTFT $X(e^{j \Omega}) = \mathcal{F}_* \{ x[k] \}$ of a signal $x[k]$, first the case of a real valued signal $x[k] \in \mathbb{R}$ is considered. The results are then generalized to complex signals $x[k] \in \mathbb{C}$.

#### Real valued signals

Decomposing a real valued signal $x[k] \in \mathbb{R}$ into its even and odd part $x[k] = x_\text{e}[k] + x_\text{o}[k]$ and introducing these into the definition of the DTFT yields

\begin{align}
X(e^{j \Omega}) &= \sum_{k = -\infty}^{\infty} \left( x_\text{e}[k] + x_\text{o}[k] \right) e^{-j \Omega k} \\
&= \sum_{k = -\infty}^{\infty} \left( x_\text{e}[k] + x_\text{o}[k] \right) \cdot \left( \cos(\Omega k) - j \sin(\Omega k) \right) \\
&= \underbrace{\sum_{k = -\infty}^{\infty} x_\text{e}[k] \cos(\Omega k)}_{X_\text{e}(e^{j \Omega})} +
j \underbrace{\sum_{k = -\infty}^{\infty} - x_\text{o}[k] \sin(\Omega k)}_{X_\text{o}(e^{j \Omega})}
\end{align}

For the last equality the fact was exploited that an infinite series with symmetric limits is zero for odd functions. In order to conclude on the symmetry of $X(e^{j \Omega})$ its behavior for a reverse of the sign of $\Omega$ has to be investigated. Due to the symmetry properties of $\cos(\Omega k)$ and $\sin(\Omega k)$, it follows that the DTFT of the

* even part $x_\text{e}[k]$ is real valued with even symmetry $X_\text{e}(e^{j \Omega}) = X_\text{e}(e^{-j \Omega})$
* odd part $x_\text{o}[k]$ is imaginary with odd symmetry $X_\text{o}(e^{j \Omega}) = - X_\text{o}(e^{-j \Omega})$

Combining this, it can be concluded that the DTFT $X(e^{j \Omega})$ of a real-valued signal $x[k] \in \mathbb{R}$ shows complex conjugate symmetry

\begin{equation}
X(e^{j \Omega}) = X^*(e^{- j \Omega})
\end{equation}

#### Complex Signals

By following the same procedure as above for an imaginary signal, the symmetries of the DTFT of the even and odd part of an imaginary signal can be derived. The results can be combined, by decomposing a complex signal $x[k] \in \mathbb{C}$ and its DTFT into its even and odd part for both the real and imaginary part. This results in the following symmetry relations

The transformation symbols $\circ \!\! - \!\! \bullet$ illustrate which part of the signal $x[k]$ is related to which part of its spectrum $X(e^{j \Omega})$. For instance, the odd part of the real part $\Re \{ x_\text{o} [k] \}$ results in an imaginary spectrum with odd symmetry $\Im \{ X_\text{o} (e^{j \Omega}) \}$.

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Continuous- and Discrete-Time Signals and Systems - Theory and Computational Examples*.


```python
from sympy import *
from sympy.abc import *
import numpy as np
init_printing()

import warnings
warnings.filterwarnings("ignore")

creq, ka, kc, ln, rpm, Lreq = symbols('C_{req} K_a Kc L_N rpm L_{req}')
```

# Cargas radiais e axiais juntas

```python
cr_exp = Eq(creq, P*ka*(L/(kc*ln))**p) #L precisa ser em revoluções
cr_exp
```

```python
Fr, Fa = symbols('F_r F_a')
P_exp = Eq(P, X*V*Fr + Y*Fa)
P_exp
```

```python
cr_exp = Eq(creq, (X*V*Fr + Y*Fa)*ka*(L/(kc*ln))**p) #L precisa ser em revoluções
cr_exp
```

# Vida util dos rolamentos

```python
from sympy import *
from sympy.abc import *
import numpy as np
import pandas as pd
init_printing()
import warnings
warnings.filterwarnings("ignore")

creq, ka, kc, ln, rpm, Lreq = symbols('C_{req} K_a Kc L_N rpm L_{req}')

cr_exp = Eq(creq, P*ka*(L/(kc*ln))**p) #L precisa ser em revoluções
cr_exp
```

```python
data = {'n':[1850,1050,1550,1100], 'Tempo':[3.5,4.,5.5,8.],
        'Fr':[5,4,5.5,6.5], 'Fa':[2,1.5,3.5,4.5]}

df = pd.DataFrame(data)
df['Fa/Fr'] = df['Fa']/df['Fr']
df['Fa/C0'] = df['Fa']/16.6

#Tabela V,x,Y
#df['Y'] = [1.52038, 1.63681, 1.33205, 1.2570225]
df['Y'] = [1.42555, 1.52553, 1.2506, 1.157]
df['X'] = [0.56]*4

#Calculo da força
df['P'] = (df['X']*df['Fr']) + (df['Y']*df['Fa'])
df['li'] = df['Tempo']*df['n']*60

df
```

      n
      Tempo
      Fr
      Fa
      Fa/Fr
      Fa/C0
      Y
      X
      P
      li

      0
      1850
      3.5
      5.0
      2.0
      0.400000
      0.120482
      1.42555
      0.56
      5.651100
      388500.0

      1
      1050
      4.0
      4.0
      1.5
      0.375000
      0.090361
      1.52553
      0.56
      4.528295
      252000.0

      2
      1550
      5.5
      5.5
      3.5
      0.636364
      0.210843
      1.25060
      0.56
      7.457100
      511500.0

      3
      1100
      8.0
      6.5
      4.5
      0.692308
      0.271084
      1.15700
      0.56
      8.846500
      528000.0

```python
P_arr = df['P'].values
li_arr = df['li'].values

Li_list = []
for i in range(len(P_arr)):
    Li = cr_exp.subs({creq:21.8,
                     P:P_arr[i],
                     ka:2.25,
                     kc:1.,
                     ln:90e6,
                     p:0.33})
    Li_list.append(solve(Li)[0])
Li_list
```

```python
soma=0
for i in range(len(li_arr)):
    soma += li_arr[i]/Li_list[i]

f = Eq(soma*x, 1)
f
```

```python
solve(f)
```

```python

```

```python

```

```python

```