# Lecture 20: Classification of Astronomical Images with Deep Learning

#### This notebook was developed by [Zeljko Ivezic](http://faculty.washington.edu/ivezic/) for the 2021 data science class at the University of Sao Paulo and it is available from [github](https://github.com/ivezic/SaoPaulo2021/blob/main/notebooks/Lecture20.ipynb).

Note: this notebook contains code developed by Z. Ivezic, M. Juric, A. Connolly, B. Sippocz, Jake VanderPlas, G. Richards and many others.

## This notebook includes:

[Introduction to Deep Learning](#intro)

[Example: Image classification with ResNet](#resnet)

## Introduction to Deep Learning

#### Originally developed by Andy Connolly for
[astroML workshop at the 235th Meeting of the American Astronomical Society (January 6, 2020)](http://www.astroml.org/workshops/AAS235.html)

ResNet50 example based on on https://github.com/priya-dwivedi/Deep-Learning/blob/master/resnet_keras/Residual_Networks_yourself.ipynb

We use a sample of simulated stamps with three balanced classes: **star** (i.e. point spread function), a **moving source** (a.k.a. trailed source) and the so-called **"dipole"** source generated by subtracting two identical stars with somewhat mismatched positions (a fraction of pixels, due to e.g., bad astrometry or proper motion). The code
that generated these images is in [makeSampleResNet.ipynb notebook](https://github.com/ivezic/SaoPaulo2021/blob/main/notebooks/makeSampleResNet.ipynb).

**Deep learning**, an extension of the neural networks that were
popularized in the 1990s. The concepts are inspired
by the structure and function of the brain. A neuron in the brain is a
core computational unit that takes a series of inputs from branched
extensions of the neuron called dendrites, operates on these inputs,
and generates an output that is transmitted along an axon to one or
more other neurons. In the context of a neural network a neuron, $j$, takes a set of inputs,
$x_i$, applies a, typically non-linear, function to these inputs and
generates an output value. Networks are then created by connecting
multiple neurons or layers of neurons to one another.

If we consider the simplified network
inputs are passed to the neurons in the
network. Each input is weighted by a value, $w_{ij}$ and the sum of
these weighted inputs is operated on by a response or activation
function $f(\theta)$, which transform
the input signal so that it varies between 0 and 1 through the
application of a non-linear response. The output from any neuron is
then given by,

$$
a_j =  f  \left( \sum_i w_{ij} x_i + b_j \right)
$$

where $b_j$ is a bias term which determines the input level at which the
neuron becomes activated.

We refer to the neurons between the inputs and the output layers as
the hidden layers. If the neurons from one layer connect to all
neurons in a subsequent layer we call this a fully connected layer.
When the outputs from the neurons only connect to subsequent layers
(i.e. the graph is acyclic) we refer to this as a feed-forward
network -- this is the most common
structure for a neural network used in classification.

The final layer in the network is the output layer. As with the hidden
layer, an activation function, $g(\theta)$, in the output layer acts
on the weighted sum of its inputs.  In this figure we have a single
output node but there can be multiple outputs. For our example  network the
output from the final neuron, $y_k$, would be given by

$$
y_k = g \left( \sum_j   w_{jk} a_j  + b_k \right)  = g\left( \sum_j
  w_{jk}  f \left( \sum_i w_{ij} x_i + b_j\right) + b_k\right)
$$

**Training of the network is simply the learning of the weights and bias values**

## Neural Network Frameworks

The development and release of open source deep learning libraries has
made the use of deep neural networks accessible to a wide range of
fields. Currently there are two common packages PyTorch (https://pytorch.org) and Tensorflow (https://www.tensorflow.org). Either code base can be utilized for the
figures and problems in this book (and generally they have the same
functionality).

### TensorFlow:

Tensorflow is the more established code base with a large community
and a large number of tutorials (https://www.tensorflow.org/tutorials) and online courses. Its
functionality is more developed than PyTorch with tools to visualize
and inspect a network (e.g., see TensorBoard). On the other hand, the
learning curve for PyTorch is generally considered to be easier than
that for Tensorflow with PyTorch having a more natural object oriented
interface for people used to writing Python code.

### PyTorch:

The primary difference between TensorFlow and PyTorch is that the
networks (or graphs) that TensorFlow generates are static while the
networks for PyTorch are dynamic (see TensorFlow Fold for dynamic graphs). This means that with PyTorch one can
modify and adjust the network on-the-fly (e.g., making it easier to
adjust for changes in the input dimensionality or number of input
nodes within a network). This feature and the object-oriented design of
PyTorch often results in fewer lines of code to achieve the same
solution when compared to Tensorflow.

### Keras:

Keras is a high-level API written on top of TensorFlow (and its precursor Theano). It is written in Python and provides a simple and intuitive interface when building neural networks. It is currently released as part of TensorFlow.

**What should you choose?** Both frameworks are continuously evolving.
The choice of deep learning library will
likely come down to which one you find better fits your style of
programming and learning. We will use Keras here
as it has an intuitive implementation of the graphical or network models.

### Building a network:

Let's start by defining what we need for the network. We will start with Keras and

- create a sequential model (this means we add layers one-by-one as we see in our introductory figure)
- add a dense (fully connected) layer with 30 neurons
  - **input_shape** describes the dimensionality of the _input data_ to this first hidden layer
  - **activation** describes the activation fuction for the neurons (in this case we will be using 'relu'; rectified linear unit)
- add a second dense (fully connected) layer with 30 neurons
- flatten the output of the second layer into a single vector so we can use ```categorical_crossentropy``` as we are assuming that our classes are "one-hot encoding" (i.e. [1,0] or [0,1]
- add an output layer using "softmax" (this means the activation values for each class sum to 1 so they can be treated like probabilities) with 2 nodes (_for our example we could have used a single output_)

### Training the network
Training a neural network is conceptually simple. Given a labelled set
of data and a loss function, we need to optimize
the weights and biases within the network by minimizing the loss.  A solution for training large
networks
uses backpropagation to efficiently estimate the gradient of the loss
function with respect
to the weights and biases.

**Mini-batch:** Optimization of the weights uses a standard gradient descent technique. If the loss function can be expressed in terms of a sum over subsets of the training data (e.g., as is the case for the
L2 norm) the training can be undertaken either for the dataset as a
whole, for subsets of the data (batch learning), or for individual
entries (on-line or stochastic learning). _Batch gradient descent_ looks at all points in the data and calculates the average gradients before updating the weights in the model. _Stochastic gradient descent_ takes a single point and calculates the gradients and then updates the model (and then repeats). _Mini-batch gradient descent_ takes a subset of the training data and  calculates the average gradients and  updates the model  (and then repeats over all mini-batches).

### Batch normalization

Batch normalization scales the activations from a layer (the input data are assumed normalized) to have zero mean and unit variance. In reality, the two parameters gamma (for the standard deviation) and beta (for the mean) are learned by the network and the activations multiplied/added by these parameters. Batch normalization provides a degree of regularization and allows for faster learning rates as the outputs are constrained to 0-1 (i.e. you dont get large excursions in the weights of subsequent layers in a network that need to be reoptimized/trained).

The normalization is applied to mini-batches of training data (as opposed to using the full training sample).

### Convolutional Networks

Convolutional Neural Networks or
CNNs are networks designed to work with images or with any regularly
sampled dataset. CNNs reduce the complexity of the network by
requiring that neurons only respond to inputs from a subset of an
image (the receptive field). This mimics the operation of the visual
cortex where neurons only respond to a small part of the
field-of-view.

There are four principal components to a CNN:
- a convolutional layer,
- a _non-linear activation function_ ,
- a pooling or downsampling operation, and
- a _fully connected layer for classification_

Dependent on the complexity of the network or structure of the data,
these components can occur singularly or chained together in multiple
sequences.

**Convolution** in a CNN refers to the convolution  of the input data
$I(x,y)$ with a kernel $K(x,y)$ which will produce a feature map $F(x,y)$

\begin{equation}
F(x,y) = K(x,y) * I(x,y)  = \sum_{x_0} \sum_{y_0} I(x-x_0, y-y_0) K(x_0, y_0).
\end{equation}

The kernel only responds to pixels within its receptive field (i.e.,
the size of the kernel), reducing the computational complexity of the
resulting network. The kernels in the convolution are described by a
depth (the number of kernels, $K$, applied to the image), and a stride
(how many pixels a kernel shifts at each step in the convolution;
typically one).  Given an $N\times M$ image, the result of the
convolution step is to transform a single image into a data cube of
feature maps with a dimension $N \times M \times K$.

Once **learned** the kernels within the convolutional layer can appear
as physically intuitive operations on the images
such as edge detection filters.

As with traditional neural networks, a non-linear activation function
is applied to the individual pixels in the resulting feature
maps.

The **pooling** in the CNN downsamples or subsamples the feature maps. Pooling summarizes values within a region
of interest (e.g., a 2x2 pixel window). The summary can be the average
pixel value but more commonly the maximum pixel value is preserved
(Max Pooling) in the downsampling. This pooling of the feature maps
reduces the size of the resulting network and makes the network less
sensitive to small translations or distortions between images.

The final layer of a CNN is the classification layer which maps the
output of the CNN to a set of labels. This is typically a fully
connected layer where each output
of the final pooling layer connects to all neurons in the
classification layer.

### The use of dropout layers

As we increase the complexity of the network we run into the issue of overfitting the data. The **dropout layer** at each training epoch randomly sets a neuron to 0 with a probability of 0.5. There is debate over whether the dropout layer should come before or after an activation layer but a recommended rule of thumb is that it should come after the activation layer for activation functions other than relu.

### Interpreting networks: how many layers and how many neurons?

The number of layers, number of neurons in a layer, and the
connectivity of these layers is typically described as the network
architecture.

Approaches to defining a network
architecture become more trial and error than applying an underlying
set of principles. For a starting point, however, there are relatively
few problems that benefit significantly from more than two layers and
we recommend starting with a single layer when training an initial
network and using cross-validation to determine when additional layers
lead result in the data being overfit.

As with the number of layers, the number of neurons within a layer
drives the computational cost (and requiring progressively larger
training sets to avoid overfitting of the data). There are many
proposals for rules of thumb for defining a network architecture:
- the number of neurons should lie between the number of inputs and output nodes
- the number of neurons should be equal to the number of outputs plus 2/3rd the number input nodes.
- the number of neurons in the hidden layer should be less than twice the size of the input layers

## ResNet 50 Convolutional Neural Network
[Go to top](#toc)

The Residual Network (ResNet) algorithm was proposed in [He et al. (2015) paper.](https://arxiv.org/abs/1512.03385).

**The ResNet idea** is simple but results in substantial improvements: instead of fitting
some function F(x), where x is the vector of input parameters, fit instead G(x) = F(x) + x.
In other words, with the residuals with respect to identity function I(x) = x.

The ResNet50 example below is based on on https://github.com/priya-dwivedi/Deep-Learning/blob/master/resnet_keras/Residual_Networks_yourself.ipynb

50 in ResNet50 comes from about 50-layers deep implementation.

### Run ResNet50 to classify stamps
### (stamps were produced with the code in makeSampleResNet.ipynb)

```python
%matplotlib inline

import os
import numpy as np
import math
import matplotlib
import matplotlib.pyplot as plt
from scipy.stats.distributions import rv_continuous

from astropy.io import fits

plt.rc("lines", linewidth=1)
plt.rc("figure", dpi=170)

np.set_printoptions(precision=3)
```

```python
# data file with images (stamps), produced with the code in makeSampleResNet.ipynb
npyFile = 'data/stamps4ResNet.npy'   # SNRmin = 50
with open(npyFile, 'rb') as f:
    data = np.load(f,allow_pickle=True)
Nstamps = data.shape[0]
print('read', Nstamps, 'stamps from', npyFile)
```

    read 3000 stamps from data/stamps4ResNet.npy

### Setup ResNet50

```python
import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Conv2D, Flatten, Activation
from tensorflow.keras.utils import to_categorical
```

```python
# Based on https://github.com/priya-dwivedi/Deep-Learning/blob/master/resnet_keras/Residual_Networks_yourself.ipynb
import numpy as np
import tensorflow as tf
from tensorflow.keras.layers import Input, Add, Dense, Activation, ZeroPadding2D, BatchNormalization, Flatten, Conv2D, AveragePooling2D, MaxPooling2D, GlobalMaxPooling2D
from tensorflow.keras.initializers import glorot_uniform
from tensorflow.keras.models import Model, load_model
from tensorflow.keras.utils import to_categorical
import tensorflow.keras.backend as K
K.set_image_data_format('channels_last')
def identity_block(X, f, filters, stage, block):
    """
    Implementation of the identity block

    Arguments:
    X -- input tensor of shape (m, n_H_prev, n_W_prev, n_C_prev)
    f -- integer, specifying the shape of the middle CONV's window for the main path
    filters -- python list of integers, defining the number of filters in the CONV layers of the main path
    stage -- integer, used to name the layers, depending on their position in the network
    block -- string/character, used to name the layers, depending on their position in the network

    Returns:
    X -- output of the identity block, tensor of shape (n_H, n_W, n_C)
    """

    # defining name basis
    conv_name_base = 'res' + str(stage) + block + '_branch'
    bn_name_base = 'bn' + str(stage) + block + '_branch'

    # Retrieve Filters
    F1, F2, F3 = filters

    # Save the input value. You'll need this later to add back to the main path.
    X_shortcut = X

    # First component of main path
    X = Conv2D(filters = F1, kernel_size = (1, 1), strides = (1,1), padding = 'valid', name = conv_name_base + '2a', kernel_initializer = glorot_uniform(seed=0))(X)
    X = BatchNormalization(axis = 3, name = bn_name_base + '2a')(X)
    X = Activation('relu')(X)

    # Second component of main path (≈3 lines)
    X = Conv2D(filters = F2, kernel_size = (f, f), strides = (1,1), padding = 'same', name = conv_name_base + '2b', kernel_initializer = glorot_uniform(seed=0))(X)
    X = BatchNormalization(axis = 3, name = bn_name_base + '2b')(X)
    X = Activation('relu')(X)

    # Third component of main path (≈2 lines)
    X = Conv2D(filters = F3, kernel_size = (1, 1), strides = (1,1), padding = 'valid', name = conv_name_base + '2c', kernel_initializer = glorot_uniform(seed=0))(X)
    X = BatchNormalization(axis = 3, name = bn_name_base + '2c')(X)

    # Final step: Add shortcut value to main path, and pass it through a RELU activation (≈2 lines)
    X = Add()([X, X_shortcut])
    X = Activation('relu')(X)

    return X

def convolutional_block(X, f, filters, stage, block, s = 2):
    """
    Implementation of the convolutional block

    Arguments:
    X -- input tensor of shape (m, n_H_prev, n_W_prev, n_C_prev)
    f -- integer, specifying the shape of the middle CONV's window for the main path
    filters -- python list of integers, defining the number of filters in the CONV layers of the main path
    stage -- integer, used to name the layers, depending on their position in the network
    block -- string/character, used to name the layers, depending on their position in the network
    s -- Integer, specifying the stride to be used

    Returns:
    X -- output of the convolutional block, tensor of shape (n_H, n_W, n_C)
    """

    # defining name basis
    conv_name_base = 'res' + str(stage) + block + '_branch'
    bn_name_base = 'bn' + str(stage) + block + '_branch'

    # Retrieve Filters
    F1, F2, F3 = filters

    # Save the input value
    X_shortcut = X

    ##### MAIN PATH #####
    # First component of main path
    X = Conv2D(F1, (1, 1), strides = (s,s), name = conv_name_base + '2a', kernel_initializer = glorot_uniform(seed=0))(X)
    X = BatchNormalization(axis = 3, name = bn_name_base + '2a')(X)
    X = Activation('relu')(X)

    # Second component of main path (≈3 lines)
    X = Conv2D(filters = F2, kernel_size = (f, f), strides = (1,1), padding = 'same', name = conv_name_base + '2b', kernel_initializer = glorot_uniform(seed=0))(X)
    X = BatchNormalization(axis = 3, name = bn_name_base + '2b')(X)
    X = Activation('relu')(X)

    # Third component of main path (≈2 lines)
    X = Conv2D(filters = F3, kernel_size = (1, 1), strides = (1,1), padding = 'valid', name = conv_name_base + '2c', kernel_initializer = glorot_uniform(seed=0))(X)
    X = BatchNormalization(axis = 3, name = bn_name_base + '2c')(X)

    ##### SHORTCUT PATH #### (≈2 lines)
    X_shortcut = Conv2D(filters = F3, kernel_size = (1, 1), strides = (s,s), padding = 'valid', name = conv_name_base + '1',
                        kernel_initializer = glorot_uniform(seed=0))(X_shortcut)
    X_shortcut = BatchNormalization(axis = 3, name = bn_name_base + '1')(X_shortcut)

    # Final step: Add shortcut value to main path, and pass it through a RELU activation (≈2 lines)
    X = Add()([X, X_shortcut])
    X = Activation('relu')(X)

    return X

def ResNet50(input_shape=(21, 21, 1), classes=2):
    """
    Implementation of the popular ResNet50 the following architecture:
    CONV2D -> BATCHNORM -> RELU -> MAXPOOL -> CONVBLOCK -> IDBLOCK*2 -> CONVBLOCK -> IDBLOCK*3
    -> CONVBLOCK -> IDBLOCK*5 -> CONVBLOCK -> IDBLOCK*2 -> AVGPOOL -> TOPLAYER

    Arguments:
    input_shape -- shape of the images of the dataset
    classes -- integer, number of classes

    Returns:
    model -- a Model() instance in Keras
    """

    # Define the input as a tensor with shape input_shape
    X_input = Input(input_shape)

    # Zero-Padding
    X = ZeroPadding2D((3, 3))(X_input)

    # Stage 1
    X = Conv2D(64, (7, 7), strides=(2, 2), name='conv1', kernel_initializer=glorot_uniform(seed=0))(X)
    X = BatchNormalization(axis=3, name='bn_conv1')(X)
    X = Activation('relu')(X)
    X = MaxPooling2D((3, 3), strides=(2, 2))(X)

    # Stage 2
    X = convolutional_block(X, f=3, filters=[64, 64, 256], stage=2, block='a', s=1)
    X = identity_block(X, 3, [64, 64, 256], stage=2, block='b')
    X = identity_block(X, 3, [64, 64, 256], stage=2, block='c')

    # Stage 3
    X = convolutional_block(X, f = 3, filters = [128, 128, 512], stage = 3, block='a', s = 2)
    X = identity_block(X, 3, [128, 128, 512], stage=3, block='b')
    X = identity_block(X, 3, [128, 128, 512], stage=3, block='c')
    X = identity_block(X, 3, [128, 128, 512], stage=3, block='d')

    # Stage 4
    X = convolutional_block(X, f = 3, filters = [256, 256, 1024], stage = 4, block='a', s = 2)
    X = identity_block(X, 3, [256, 256, 1024], stage=4, block='b')
    X = identity_block(X, 3, [256, 256, 1024], stage=4, block='c')
    X = identity_block(X, 3, [256, 256, 1024], stage=4, block='d')
    X = identity_block(X, 3, [256, 256, 1024], stage=4, block='e')
    X = identity_block(X, 3, [256, 256, 1024], stage=4, block='f')

    # Stage 5
    X = convolutional_block(X, f = 3, filters = [512, 512, 2048], stage = 5, block='a', s = 2)
    X = identity_block(X, 3, [512, 512, 2048], stage=5, block='b')
    X = identity_block(X, 3, [512, 512, 2048], stage=5, block='c')

    # output layer
    X = Flatten()(X)
    X = Dense(classes, activation='softmax', name='fc' + str(classes), kernel_initializer = glorot_uniform(seed=0))(X)

    # Create model
    model = Model(inputs = X_input, outputs = X, name='ResNet50')

    return model
```

### A few helper routines:

These functions normalize_image, plot_image_array, plot_confusion_matrix, plot_model_history will be used to visualize the data and the outputs of the neural networks as a function of the type and complexity of the network

```python
def normalize_image(image):
    '''Rescale the constrast in an image based on the noise (used for displays and the CNN)'''
    sigmaG_coeff =  0.7413  # turns (q75-q25) difference into standard deviation in case of Gaussian
    per25,per50,per75 = np.percentile(image,[25,50,75])
    sigmaG = sigmaG_coeff * (per75 - per25)
    # sigma clip image, remove background, and normalize to unity
    image[image<(per50-2*sigmaG)] = per50-2*sigmaG
    image -= np.min(image)
    image /= np.sum(image)
    return image

def reshape_arrays(data, labels):
    '''reshape arrays for Keras'''
    data = data.reshape(-1,32, 32, 1)
    labels = to_categorical(labels)
    return data,labels
```

```python
from sklearn.metrics import confusion_matrix
from sklearn.utils.multiclass import unique_labels

def plot_confusion_matrix(y_true, y_pred,
                          normalize=False,
                          title=None,
                          cmap=plt.cm.Blues):
    """
    From scikit-learn: plots a confusion matrix.
    Normalization can be applied by setting `normalize=True`.
    """
    if not title:
        if normalize:
            title = 'Normalized confusion matrix'
        else:
            title = 'Confusion matrix, without normalization'

    # Compute confusion matrix
    cm = confusion_matrix(y_true, y_pred)

    if normalize:
        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]

    fig, ax = plt.subplots()
    im = ax.imshow(cm, interpolation='nearest', cmap=cmap)
    ax.figure.colorbar(im, ax=ax)

    ax.set(xticks=np.arange(cm.shape[1]),
           yticks=np.arange(cm.shape[0]),
           # ... and label them with the respective list entries
           title=title,
           ylabel='True label',
           xlabel='Predicted label')

    # Rotate the tick labels and set their alignment.
    plt.setp(ax.get_xticklabels(), rotation=45, ha="right",
             rotation_mode="anchor")

    #fixes "squishing of plot"
    plt.ylim([2.5, -0.5])

    # Loop over data dimensions and create text annotations.
    fmt = '.2f' if normalize else 'd'
    thresh = cm.max() / 2.
    for i in range(cm.shape[0]):
        for j in range(cm.shape[1]):
            ax.text(j, i, format(cm[i, j], fmt),
                    ha="center", va="center",
                    color="white" if cm[i, j] > thresh else "black")
    fig.tight_layout()
    return

def plot_model_history(history):
    '''Plot the training and validation history for a TensorFlow network'''

    # Extract loss and accuracy
    loss = history.history['loss']
    val_loss = history.history['val_loss']
    acc = history.history['accuracy']
    val_acc = history.history['val_accuracy']

    fig, ax = plt.subplots(nrows=1,ncols=2, figsize=(10,5))
    ax[0].plot(np.arange(n_epochs), loss, label='Training Loss')
    ax[0].plot(np.arange(n_epochs), val_loss, label='Validation Loss')
    ax[0].set_title('Loss Curves')
    ax[0].legend()
    ax[0].set_xlabel('Epoch')
    ax[0].set_ylabel('Loss')

    ax[1].plot(np.arange(n_epochs), acc, label='Training Accuracy')
    ax[1].plot(np.arange(n_epochs), val_acc, label='Validation Accuracy')
    ax[1].set_title('Accuracy Curves')
    ax[1].legend()
    ax[1].set_xlabel('Epoch')
    ax[1].set_ylabel('Accuracy')

def plot_image_array(images, nrows=2, ncols=5, figsize=[8,4], nx=32, ny=32, title='', subtitle=False,
                     class_true=None, classes=None):
    '''Plot an array of images'''
    Nimages = images.shape[0]
    fig, ax = plt.subplots(nrows=nrows,ncols=ncols,figsize=figsize, squeeze=False)
    fig.subplots_adjust(hspace=0, left=0.07, right=0.95, wspace=0.05, bottom=0.15)
    Ndisplay = nrows*ncols
    if Nimages < Ndisplay: Ndisplay = Nimages
    for indx in np.arange(Ndisplay):
        i = int(indx/ncols)
        j = indx%ncols
        if (i == 0):
            ax[i][j].xaxis.set_major_formatter(plt.NullFormatter())
        if (j != 0):
            ax[i][j].yaxis.set_major_formatter(plt.NullFormatter())

        ax[i][j].imshow(images[indx].reshape(nx,ny), cmap='gray')
        if (subtitle == True):
            title = 'True Class: %i, Predicted Class: %i\n  p0: %e\n p1: %e\n p2 %e'
            pT = np.argmax(class_true[indx])
            pP = np.argmax(classes[indx])
            ax[i][j].set_title(title %
                     (pT, pP, classes[indx,0], classes[indx,1], classes[indx,2]))
    ax[0][0].set_ylabel('$y$')
    ax[nrows-1][int(ncols/2)].set_xlabel('$x$')
```

## Classify stamps

### first, we need to renormalize stamps to the same background noise

```python
## renormalize stamps
stamps0all = []
stamps1all = []
stamps2all = []
for i in range(0,1000):
    stamps0all.append(normalize_image(data[i,:,:,0]))
    stamps1all.append(normalize_image(data[1000+i,:,:,0]))
    stamps2all.append(normalize_image(data[2000+i,:,:,0]))
```

```python
# assumes that class 1 comes after class 0, and class 2 after class 1 (ZI: ugly hack, VOLATILE!)
input_stamps = np.vstack([stamps0all, stamps1all, stamps2all])
stamp_class = np.zeros(len(stamps0all) + len(stamps1all) + len(stamps2all))
stamp_class[len(stamps0all):] = 1 #
stamp_class[len(stamps0all)+len(stamps1all):] = 2 #
```

### split the sample into training, validation and test data sets

```python
from astroML.utils import split_samples
# split the samples into training, validation and test data sets
# for definitions, see https://machinelearningmastery.com/difference-test-validation-datasets/
(data_train, data_val, data_test), (class_train, class_val, class_test) = split_samples(input_stamps, stamp_class,
                                                                                [0.7,0.1,0.2])
data_train, class_train = reshape_arrays(data_train, class_train)
data_val, class_val = reshape_arrays(data_val, class_val)
data_test, class_test = reshape_arrays(data_test, class_test)
print ('Number of samples in the training ({}); test ({}); and validation ({}) data sets'.format(data_train.shape[0],
                                                                                data_test.shape[0],
                                                                                data_val.shape[0]))
```

    Number of samples in the training (2100); test (600); and validation (300) data sets

```python
# plot a few stamps to see what we are trying to classify
plot_image_array(data_test, figsize=[16,10], subtitle=True, classes=class_test, class_true=class_test)
```

### and now run CNN (ResNet50): very slow, about 3 hours for 100 epochs!
 (runtime scales with n_epochs; sometimes only 50 is sufficient)

```python
resnet50_model = ResNet50(input_shape=(32, 32, 1), classes=3)
resnet50_model.compile(loss='categorical_crossentropy', optimizer='sgd', metrics=['accuracy'])
n_epochs=100
%time resnet_model_history = resnet50_model.fit(data_train, class_train, epochs=n_epochs, batch_size=256, verbose=1, validation_data=(data_val, class_val), shuffle=True)
```

    Epoch 1/100
    9/9 [==============================] - 170s 17s/step - loss: 2.0521 - accuracy: 0.6005 - val_loss: 1.1044 - val_accuracy: 0.3300
    Epoch 2/100
    9/9 [==============================] - 136s 14s/step - loss: 2.1468 - accuracy: 0.6681 - val_loss: 1.1223 - val_accuracy: 0.3300
    Epoch 3/100
    9/9 [==============================] - 135s 15s/step - loss: 1.0639 - accuracy: 0.7900 - val_loss: 1.1115 - val_accuracy: 0.3300
    Epoch 4/100
    9/9 [==============================] - 156s 17s/step - loss: 0.5806 - accuracy: 0.9043 - val_loss: 1.1367 - val_accuracy: 0.3300
    Epoch 5/100
    9/9 [==============================] - 152s 17s/step - loss: 0.7645 - accuracy: 0.8838 - val_loss: 1.1515 - val_accuracy: 0.3300
    Epoch 6/100
    9/9 [==============================] - 153s 17s/step - loss: 0.1984 - accuracy: 0.9276 - val_loss: 1.1779 - val_accuracy: 0.3300
    Epoch 7/100
    9/9 [==============================] - 119s 13s/step - loss: 0.2828 - accuracy: 0.9433 - val_loss: 1.2154 - val_accuracy: 0.3300
    Epoch 8/100
    9/9 [==============================] - 112s 12s/step - loss: 0.3430 - accuracy: 0.9443 - val_loss: 1.2295 - val_accuracy: 0.3300
    Epoch 9/100
    9/9 [==============================] - 110s 12s/step - loss: 0.1141 - accuracy: 0.9557 - val_loss: 1.2861 - val_accuracy: 0.3300
    Epoch 10/100
    9/9 [==============================] - 103s 11s/step - loss: 0.0824 - accuracy: 0.9705 - val_loss: 1.2860 - val_accuracy: 0.3300
    Epoch 11/100
    9/9 [==============================] - 99s 11s/step - loss: 0.2226 - accuracy: 0.9514 - val_loss: 1.2220 - val_accuracy: 0.3300
    Epoch 12/100
    9/9 [==============================] - 101s 11s/step - loss: 0.4407 - accuracy: 0.9000 - val_loss: 1.4702 - val_accuracy: 0.3300
    Epoch 13/100
    9/9 [==============================] - 101s 11s/step - loss: 0.2069 - accuracy: 0.9581 - val_loss: 1.5200 - val_accuracy: 0.3300
    Epoch 14/100
    9/9 [==============================] - 99s 11s/step - loss: 0.1463 - accuracy: 0.9733 - val_loss: 1.6042 - val_accuracy: 0.3300
    Epoch 15/100
    9/9 [==============================] - 98s 11s/step - loss: 0.0631 - accuracy: 0.9738 - val_loss: 1.6862 - val_accuracy: 0.3300
    Epoch 16/100
    9/9 [==============================] - 95s 10s/step - loss: 0.0745 - accuracy: 0.9771 - val_loss: 1.7985 - val_accuracy: 0.3300
    Epoch 17/100
    9/9 [==============================] - 97s 10s/step - loss: 0.1278 - accuracy: 0.9705 - val_loss: 1.8924 - val_accuracy: 0.3300
    Epoch 18/100
    9/9 [==============================] - 106s 12s/step - loss: 0.0932 - accuracy: 0.9695 - val_loss: 2.0520 - val_accuracy: 0.3300
    Epoch 19/100
    9/9 [==============================] - 95s 10s/step - loss: 0.1548 - accuracy: 0.9676 - val_loss: 2.0014 - val_accuracy: 0.3300
    Epoch 20/100
    9/9 [==============================] - 110s 12s/step - loss: 0.1319 - accuracy: 0.9767 - val_loss: 2.1734 - val_accuracy: 0.3300
    Epoch 21/100
    9/9 [==============================] - 98s 11s/step - loss: 0.0617 - accuracy: 0.9810 - val_loss: 2.3139 - val_accuracy: 0.3300
    Epoch 22/100
    9/9 [==============================] - 95s 10s/step - loss: 0.0785 - accuracy: 0.9857 - val_loss: 2.4558 - val_accuracy: 0.3300
    Epoch 23/100
    9/9 [==============================] - 95s 10s/step - loss: 0.1043 - accuracy: 0.9790 - val_loss: 2.6288 - val_accuracy: 0.3300
    Epoch 24/100
    9/9 [==============================] - 99s 11s/step - loss: 0.0797 - accuracy: 0.9771 - val_loss: 2.7636 - val_accuracy: 0.3300
    Epoch 25/100
    9/9 [==============================] - 96s 11s/step - loss: 0.2535 - accuracy: 0.9724 - val_loss: 2.9559 - val_accuracy: 0.3300
    Epoch 26/100
    9/9 [==============================] - 99s 11s/step - loss: 0.1877 - accuracy: 0.9629 - val_loss: 3.1571 - val_accuracy: 0.3300
    Epoch 27/100
    9/9 [==============================] - 100s 11s/step - loss: 0.1824 - accuracy: 0.9714 - val_loss: 3.5310 - val_accuracy: 0.3300
    Epoch 28/100
    9/9 [==============================] - 102s 11s/step - loss: 0.2138 - accuracy: 0.9581 - val_loss: 3.5913 - val_accuracy: 0.3300
    Epoch 29/100
    9/9 [==============================] - 94s 10s/step - loss: 0.0559 - accuracy: 0.9795 - val_loss: 3.8520 - val_accuracy: 0.3300
    Epoch 30/100
    9/9 [==============================] - 94s 10s/step - loss: 0.1634 - accuracy: 0.9681 - val_loss: 4.0405 - val_accuracy: 0.3300
    Epoch 31/100
    9/9 [==============================] - 98s 11s/step - loss: 0.0445 - accuracy: 0.9833 - val_loss: 4.2531 - val_accuracy: 0.3300
    Epoch 32/100
    9/9 [==============================] - 95s 10s/step - loss: 0.0481 - accuracy: 0.9771 - val_loss: 4.2291 - val_accuracy: 0.3300
    Epoch 33/100
    9/9 [==============================] - 94s 10s/step - loss: 0.0662 - accuracy: 0.9800 - val_loss: 4.6897 - val_accuracy: 0.3300
    Epoch 34/100
    9/9 [==============================] - 98s 11s/step - loss: 0.0363 - accuracy: 0.9876 - val_loss: 4.6900 - val_accuracy: 0.3300
    Epoch 35/100
    9/9 [==============================] - 96s 11s/step - loss: 0.0388 - accuracy: 0.9857 - val_loss: 4.8374 - val_accuracy: 0.3300
    Epoch 36/100
    9/9 [==============================] - 96s 11s/step - loss: 0.0136 - accuracy: 0.9962 - val_loss: 5.0035 - val_accuracy: 0.3300
    Epoch 37/100
    9/9 [==============================] - 96s 11s/step - loss: 0.0138 - accuracy: 0.9957 - val_loss: 5.1499 - val_accuracy: 0.3300
    Epoch 38/100
    9/9 [==============================] - 94s 10s/step - loss: 0.0141 - accuracy: 0.9967 - val_loss: 5.3833 - val_accuracy: 0.3300
    Epoch 39/100
    9/9 [==============================] - 96s 11s/step - loss: 0.0328 - accuracy: 0.9886 - val_loss: 5.5023 - val_accuracy: 0.3300
    Epoch 40/100
    9/9 [==============================] - 95s 11s/step - loss: 0.0710 - accuracy: 0.9876 - val_loss: 5.5498 - val_accuracy: 0.3300
    Epoch 41/100
    9/9 [==============================] - 101s 11s/step - loss: 0.0156 - accuracy: 0.9962 - val_loss: 5.7122 - val_accuracy: 0.3300
    Epoch 42/100
    9/9 [==============================] - 98s 11s/step - loss: 0.0162 - accuracy: 0.9943 - val_loss: 5.8017 - val_accuracy: 0.3300
    Epoch 43/100
    9/9 [==============================] - 96s 11s/step - loss: 0.1213 - accuracy: 0.9914 - val_loss: 5.8983 - val_accuracy: 0.3300
    Epoch 44/100
    9/9 [==============================] - 94s 10s/step - loss: 0.0232 - accuracy: 0.9938 - val_loss: 5.9411 - val_accuracy: 0.3300
    Epoch 45/100
    9/9 [==============================] - 96s 11s/step - loss: 0.0059 - accuracy: 0.9986 - val_loss: 5.9792 - val_accuracy: 0.3300
    Epoch 46/100
    9/9 [==============================] - 101s 11s/step - loss: 0.0172 - accuracy: 0.9933 - val_loss: 6.0615 - val_accuracy: 0.3300
    Epoch 47/100
    9/9 [==============================] - 97s 11s/step - loss: 0.0398 - accuracy: 0.9862 - val_loss: 6.0743 - val_accuracy: 0.3333
    Epoch 48/100
    9/9 [==============================] - 97s 11s/step - loss: 0.0374 - accuracy: 0.9905 - val_loss: 5.9783 - val_accuracy: 0.3367
    Epoch 49/100
    9/9 [==============================] - 96s 11s/step - loss: 0.0072 - accuracy: 0.9990 - val_loss: 5.9701 - val_accuracy: 0.3400
    Epoch 50/100
    9/9 [==============================] - 102s 11s/step - loss: 0.0537 - accuracy: 0.9967 - val_loss: 6.0070 - val_accuracy: 0.3467
    Epoch 51/100
    9/9 [==============================] - 98s 11s/step - loss: 0.0294 - accuracy: 0.9962 - val_loss: 5.9993 - val_accuracy: 0.3467
    Epoch 52/100
    9/9 [==============================] - 94s 10s/step - loss: 0.0099 - accuracy: 0.9962 - val_loss: 5.9744 - val_accuracy: 0.3500
    Epoch 53/100
    9/9 [==============================] - 94s 10s/step - loss: 0.0377 - accuracy: 0.9933 - val_loss: 6.0159 - val_accuracy: 0.3567
    Epoch 54/100
    9/9 [==============================] - 96s 11s/step - loss: 0.0360 - accuracy: 0.9962 - val_loss: 5.9142 - val_accuracy: 0.3667
    Epoch 55/100
    9/9 [==============================] - 97s 11s/step - loss: 0.1258 - accuracy: 0.9686 - val_loss: 5.9231 - val_accuracy: 0.3667
    Epoch 56/100
    9/9 [==============================] - 95s 10s/step - loss: 0.0525 - accuracy: 0.9810 - val_loss: 5.9170 - val_accuracy: 0.3633
    Epoch 57/100
    9/9 [==============================] - 97s 10s/step - loss: 0.0385 - accuracy: 0.9895 - val_loss: 5.5553 - val_accuracy: 0.3900
    Epoch 58/100
    9/9 [==============================] - 101s 11s/step - loss: 0.0106 - accuracy: 0.9962 - val_loss: 5.4114 - val_accuracy: 0.3833
    Epoch 59/100
    9/9 [==============================] - 98s 11s/step - loss: 0.0246 - accuracy: 0.9948 - val_loss: 5.0994 - val_accuracy: 0.4033
    Epoch 60/100
    9/9 [==============================] - 97s 11s/step - loss: 0.2294 - accuracy: 0.9905 - val_loss: 4.9631 - val_accuracy: 0.4400
    Epoch 61/100
    9/9 [==============================] - 98s 11s/step - loss: 0.0112 - accuracy: 0.9971 - val_loss: 4.7554 - val_accuracy: 0.4333
    Epoch 62/100
    9/9 [==============================] - 117s 13s/step - loss: 0.0052 - accuracy: 0.9990 - val_loss: 4.4414 - val_accuracy: 0.4567
    Epoch 63/100
    9/9 [==============================] - 128s 14s/step - loss: 0.0043 - accuracy: 0.9990 - val_loss: 4.4075 - val_accuracy: 0.4367
    Epoch 64/100
    9/9 [==============================] - 129s 14s/step - loss: 0.0063 - accuracy: 0.9986 - val_loss: 4.0454 - val_accuracy: 0.4900
    Epoch 65/100
    9/9 [==============================] - 134s 15s/step - loss: 0.0036 - accuracy: 0.9990 - val_loss: 4.2817 - val_accuracy: 0.4533
    Epoch 66/100
    9/9 [==============================] - 141s 16s/step - loss: 0.0441 - accuracy: 0.9862 - val_loss: 3.6922 - val_accuracy: 0.4900
    Epoch 67/100
    9/9 [==============================] - 148s 17s/step - loss: 0.0113 - accuracy: 0.9962 - val_loss: 3.3507 - val_accuracy: 0.5133
    Epoch 68/100
    9/9 [==============================] - 153s 16s/step - loss: 0.0050 - accuracy: 0.9981 - val_loss: 2.9237 - val_accuracy: 0.5367
    Epoch 69/100
    9/9 [==============================] - 161s 17s/step - loss: 0.0023 - accuracy: 1.0000 - val_loss: 2.6702 - val_accuracy: 0.5600
    Epoch 70/100
    9/9 [==============================] - 139s 15s/step - loss: 0.0051 - accuracy: 0.9990 - val_loss: 2.3772 - val_accuracy: 0.5867
    Epoch 71/100
    9/9 [==============================] - 162s 18s/step - loss: 0.0016 - accuracy: 0.9995 - val_loss: 2.0295 - val_accuracy: 0.6500
    Epoch 72/100
    9/9 [==============================] - 172s 19s/step - loss: 0.0082 - accuracy: 0.9967 - val_loss: 2.1949 - val_accuracy: 0.6333
    Epoch 73/100
    9/9 [==============================] - 164s 18s/step - loss: 0.0696 - accuracy: 0.9781 - val_loss: 1.1657 - val_accuracy: 0.7567
    Epoch 74/100
    9/9 [==============================] - 146s 16s/step - loss: 0.0096 - accuracy: 0.9957 - val_loss: 1.2180 - val_accuracy: 0.7733
    Epoch 75/100
    9/9 [==============================] - 161s 18s/step - loss: 0.0112 - accuracy: 0.9957 - val_loss: 0.8892 - val_accuracy: 0.8100
    Epoch 76/100
    9/9 [==============================] - 159s 17s/step - loss: 0.0034 - accuracy: 0.9981 - val_loss: 1.0254 - val_accuracy: 0.7833
    Epoch 77/100
    9/9 [==============================] - 165s 18s/step - loss: 0.0035 - accuracy: 0.9990 - val_loss: 0.6477 - val_accuracy: 0.8667
    Epoch 78/100
    9/9 [==============================] - 186s 21s/step - loss: 0.0019 - accuracy: 0.9995 - val_loss: 0.6268 - val_accuracy: 0.8633
    Epoch 79/100
    9/9 [==============================] - 218s 24s/step - loss: 0.0013 - accuracy: 1.0000 - val_loss: 0.5048 - val_accuracy: 0.8867
    Epoch 80/100
    9/9 [==============================] - 186s 20s/step - loss: 7.6155e-04 - accuracy: 1.0000 - val_loss: 0.4435 - val_accuracy: 0.8967
    Epoch 81/100
    9/9 [==============================] - 190s 21s/step - loss: 4.4687e-04 - accuracy: 1.0000 - val_loss: 0.3868 - val_accuracy: 0.9067
    Epoch 82/100
    9/9 [==============================] - 189s 21s/step - loss: 5.6939e-04 - accuracy: 1.0000 - val_loss: 0.3277 - val_accuracy: 0.9233
    Epoch 83/100
    9/9 [==============================] - 237s 26s/step - loss: 0.0928 - accuracy: 0.9905 - val_loss: 0.4965 - val_accuracy: 0.8833
    Epoch 84/100
    9/9 [==============================] - 220s 24s/step - loss: 0.0017 - accuracy: 1.0000 - val_loss: 0.3732 - val_accuracy: 0.9067
    Epoch 85/100
    9/9 [==============================] - 181s 20s/step - loss: 0.0743 - accuracy: 0.9933 - val_loss: 0.3638 - val_accuracy: 0.9200
    Epoch 86/100
    9/9 [==============================] - 217s 24s/step - loss: 0.0835 - accuracy: 0.9976 - val_loss: 0.3664 - val_accuracy: 0.9100
    Epoch 87/100
    9/9 [==============================] - 177s 19s/step - loss: 0.0014 - accuracy: 1.0000 - val_loss: 0.3600 - val_accuracy: 0.9100
    Epoch 88/100
    9/9 [==============================] - 132s 14s/step - loss: 0.0069 - accuracy: 0.9981 - val_loss: 0.2826 - val_accuracy: 0.9400
    Epoch 89/100
    9/9 [==============================] - 119s 13s/step - loss: 0.0255 - accuracy: 0.9976 - val_loss: 2.9415 - val_accuracy: 0.4533
    Epoch 90/100
    9/9 [==============================] - 109s 12s/step - loss: 0.2233 - accuracy: 0.9724 - val_loss: 2.7223 - val_accuracy: 0.4067
    Epoch 91/100
    9/9 [==============================] - 106s 12s/step - loss: 0.1282 - accuracy: 0.9862 - val_loss: 2.3811 - val_accuracy: 0.4767
    Epoch 92/100
    9/9 [==============================] - 106s 12s/step - loss: 0.0155 - accuracy: 0.9933 - val_loss: 2.3025 - val_accuracy: 0.4667
    Epoch 93/100
    9/9 [==============================] - 103s 11s/step - loss: 0.0031 - accuracy: 1.0000 - val_loss: 1.2310 - val_accuracy: 0.6567
    Epoch 94/100
    9/9 [==============================] - 101s 11s/step - loss: 0.0192 - accuracy: 0.9981 - val_loss: 0.8055 - val_accuracy: 0.7333
    Epoch 95/100
    9/9 [==============================] - 101s 11s/step - loss: 0.0023 - accuracy: 0.9995 - val_loss: 0.7652 - val_accuracy: 0.7433
    Epoch 96/100
    9/9 [==============================] - 100s 11s/step - loss: 0.0028 - accuracy: 0.9990 - val_loss: 0.1776 - val_accuracy: 0.9433
    Epoch 97/100
    9/9 [==============================] - 101s 11s/step - loss: 0.0455 - accuracy: 0.9895 - val_loss: 2.0419 - val_accuracy: 0.6967
    Epoch 98/100
    9/9 [==============================] - 99s 11s/step - loss: 0.0714 - accuracy: 0.9914 - val_loss: 1.9221 - val_accuracy: 0.8667
    Epoch 99/100
    9/9 [==============================] - 101s 11s/step - loss: 0.1337 - accuracy: 0.9900 - val_loss: 0.6657 - val_accuracy: 0.7333
    Epoch 100/100
    9/9 [==============================] - 101s 11s/step - loss: 0.0896 - accuracy: 0.9824 - val_loss: 0.4118 - val_accuracy: 0.9233
    CPU times: user 5h 52min 45s, sys: 38min 28s, total: 6h 31min 13s
    Wall time: 3h 21min 20s

## analyze classification results

```python
# plot the training history
plot_model_history(resnet_model_history)
```

### Note: the training accuracy is essentially 100% after only 30-40 epochs; however, the validation accuracy varies wildly

## now we will use the test subsample which was never used in training to obtain another estimate of performance

```python
# predicted class (this will be (600, 3) array - we actually get probabilities for each class)
Rclasses = resnet50_model.predict(data_test)
```

```python
# plot example classifications
plot_image_array(data_test, figsize=[16,10], subtitle=True, classes=Rclasses, class_true=class_test)
```

```python
# plot the confusion matrix
plot_confusion_matrix(np.argmax(class_test,axis=1), np.argmax(Rclasses,axis=1), normalize=True,
                      title='Normalized confusion matrix')
```

## that is fairly decent performance! (though note that it was SNR>20 dataset)

### let's have a closer look at misclassified stamps:

```python
# note that here we adopt the class with the highest probability as the highest class
tC = np.argmax(class_test,axis=1)
aC = np.argmax(Rclasses,axis=1)
Nok = 0
Ndata = np.shape(class_test)[0]
good = np.ones(Ndata)
for i in range(0, Ndata):
    if (tC[i] == aC[i]):
        Nok += 1
    else:
        good[i] = 0
print('Overall classification accuracy for test sample:', Nok/Ndata)
```

    Overall classification accuracy for test sample: 0.9066666666666666

```python
# let's see what is true class and the predicted probabilities for all 3 classes
for i in range(0, Ndata):
    if (good[i]==0):
        print(tC[i], aC[i], Rclasses[i,:])
```

    1 0 [1.000e+00 4.432e-13 1.340e-08]
    1 2 [0.023 0.313 0.664]
    0 1 [0.353 0.522 0.125]
    1 0 [1.000e+00 0.000e+00 5.893e-33]
    1 0 [0.636 0.343 0.021]
    1 2 [0.041 0.455 0.504]
    1 2 [0.018 0.487 0.495]
    1 0 [0.942 0.055 0.003]
    1 0 [9.998e-01 1.909e-04 1.147e-05]
    1 0 [0.965 0.033 0.003]
    1 0 [1.000e+00 2.054e-13 4.105e-08]
    1 2 [0.014 0.435 0.55 ]
    1 0 [0.878 0.118 0.004]
    1 0 [0.548 0.374 0.078]
    0 2 [0.174 0.195 0.631]
    1 0 [1.000e+00 2.670e-05 9.323e-09]
    1 0 [9.992e-01 6.522e-04 1.319e-04]
    1 2 [0.027 0.336 0.638]
    1 0 [9.942e-01 5.578e-03 2.587e-04]
    1 2 [0.014 0.407 0.579]
    1 2 [0.051 0.401 0.548]
    1 0 [9.999e-01 5.714e-05 9.277e-06]
    1 2 [0.008 0.064 0.928]
    1 0 [1.000e+00 4.677e-29 2.025e-18]
    1 0 [0.862 0.135 0.003]
    1 2 [0.044 0.175 0.781]
    1 0 [0.903 0.095 0.002]
    1 2 [0.022 0.482 0.496]
    0 2 [0.132 0.281 0.586]
    1 0 [9.994e-01 5.127e-04 8.105e-05]
    1 0 [0.954 0.045 0.002]
    1 2 [0.026 0.468 0.507]
    1 2 [0.024 0.353 0.623]
    1 2 [0.013 0.224 0.763]
    1 0 [9.613e-01 3.802e-02 6.658e-04]
    1 2 [0.056 0.12  0.824]
    1 2 [0.075 0.253 0.672]
    1 0 [0.855 0.136 0.009]
    1 2 [0.194 0.311 0.495]
    1 0 [0.372 0.334 0.294]
    1 0 [9.943e-01 5.548e-03 1.981e-04]
    1 0 [0.804 0.123 0.073]
    1 2 [0.019 0.44  0.541]
    1 0 [9.966e-01 3.210e-03 1.446e-04]
    1 2 [0.009 0.156 0.835]
    1 2 [0.012 0.248 0.74 ]
    1 2 [0.034 0.226 0.74 ]
    1 2 [0.015 0.35  0.635]
    1 0 [1.000e+00 6.224e-06 1.949e-06]
    1 0 [0.899 0.097 0.004]
    1 0 [0.753 0.233 0.014]
    1 2 [0.018 0.378 0.604]
    1 0 [0.688 0.303 0.009]
    0 1 [0.083 0.719 0.198]
    1 0 [0.967 0.032 0.001]
    1 0 [0.979 0.02  0.001]

### Some trailed sources are miclassified as stars - they were probably not very elongated, let's see...

```python
# plot misclassified stamps
misfits = data_test[good==0]
trueClass = class_test[good==0]
predClass = Rclasses[good==0]
plot_image_array(misfits, figsize=[16,10], subtitle=True, classes=predClass, class_true=trueClass)
#plot_image_array(misfits, nrows=2, ncols=5, figsize=[16,5], subtitle=True, classes=predClass, class_true=trueClass)
```

## How many model parameters did we fit? We can find that information as follows

```python
# print model summary
resnet50_model.summary()
```

    Model: "ResNet50"
    __________________________________________________________________________________________________
    Layer (type)                    Output Shape         Param #     Connected to
    ==================================================================================================
    input_1 (InputLayer)            [(None, 32, 32, 1)]  0
    __________________________________________________________________________________________________
    zero_padding2d (ZeroPadding2D)  (None, 38, 38, 1)    0           input_1[0][0]
    __________________________________________________________________________________________________
    conv1 (Conv2D)                  (None, 16, 16, 64)   3200        zero_padding2d[0][0]
    __________________________________________________________________________________________________
    bn_conv1 (BatchNormalization)   (None, 16, 16, 64)   256         conv1[0][0]
    __________________________________________________________________________________________________
    activation (Activation)         (None, 16, 16, 64)   0           bn_conv1[0][0]
    __________________________________________________________________________________________________
    max_pooling2d (MaxPooling2D)    (None, 7, 7, 64)     0           activation[0][0]
    __________________________________________________________________________________________________
    res2a_branch2a (Conv2D)         (None, 7, 7, 64)     4160        max_pooling2d[0][0]
    __________________________________________________________________________________________________
    bn2a_branch2a (BatchNormalizati (None, 7, 7, 64)     256         res2a_branch2a[0][0]
    __________________________________________________________________________________________________
    activation_1 (Activation)       (None, 7, 7, 64)     0           bn2a_branch2a[0][0]
    __________________________________________________________________________________________________
    res2a_branch2b (Conv2D)         (None, 7, 7, 64)     36928       activation_1[0][0]
    __________________________________________________________________________________________________
    bn2a_branch2b (BatchNormalizati (None, 7, 7, 64)     256         res2a_branch2b[0][0]
    __________________________________________________________________________________________________
    activation_2 (Activation)       (None, 7, 7, 64)     0           bn2a_branch2b[0][0]
    __________________________________________________________________________________________________
    res2a_branch2c (Conv2D)         (None, 7, 7, 256)    16640       activation_2[0][0]
    __________________________________________________________________________________________________
    res2a_branch1 (Conv2D)          (None, 7, 7, 256)    16640       max_pooling2d[0][0]
    __________________________________________________________________________________________________
    bn2a_branch2c (BatchNormalizati (None, 7, 7, 256)    1024        res2a_branch2c[0][0]
    __________________________________________________________________________________________________
    bn2a_branch1 (BatchNormalizatio (None, 7, 7, 256)    1024        res2a_branch1[0][0]
    __________________________________________________________________________________________________
    add (Add)                       (None, 7, 7, 256)    0           bn2a_branch2c[0][0]
                                                                     bn2a_branch1[0][0]
    __________________________________________________________________________________________________
    activation_3 (Activation)       (None, 7, 7, 256)    0           add[0][0]
    __________________________________________________________________________________________________
    res2b_branch2a (Conv2D)         (None, 7, 7, 64)     16448       activation_3[0][0]
    __________________________________________________________________________________________________
    bn2b_branch2a (BatchNormalizati (None, 7, 7, 64)     256         res2b_branch2a[0][0]
    __________________________________________________________________________________________________
    activation_4 (Activation)       (None, 7, 7, 64)     0           bn2b_branch2a[0][0]
    __________________________________________________________________________________________________
    res2b_branch2b (Conv2D)         (None, 7, 7, 64)     36928       activation_4[0][0]
    __________________________________________________________________________________________________
    bn2b_branch2b (BatchNormalizati (None, 7, 7, 64)     256         res2b_branch2b[0][0]
    __________________________________________________________________________________________________
    activation_5 (Activation)       (None, 7, 7, 64)     0           bn2b_branch2b[0][0]
    __________________________________________________________________________________________________
    res2b_branch2c (Conv2D)         (None, 7, 7, 256)    16640       activation_5[0][0]
    __________________________________________________________________________________________________
    bn2b_branch2c (BatchNormalizati (None, 7, 7, 256)    1024        res2b_branch2c[0][0]
    __________________________________________________________________________________________________
    add_1 (Add)                     (None, 7, 7, 256)    0           bn2b_branch2c[0][0]
                                                                     activation_3[0][0]
    __________________________________________________________________________________________________
    activation_6 (Activation)       (None, 7, 7, 256)    0           add_1[0][0]
    __________________________________________________________________________________________________
    res2c_branch2a (Conv2D)         (None, 7, 7, 64)     16448       activation_6[0][0]
    __________________________________________________________________________________________________
    bn2c_branch2a (BatchNormalizati (None, 7, 7, 64)     256         res2c_branch2a[0][0]
    __________________________________________________________________________________________________
    activation_7 (Activation)       (None, 7, 7, 64)     0           bn2c_branch2a[0][0]
    __________________________________________________________________________________________________
    res2c_branch2b (Conv2D)         (None, 7, 7, 64)     36928       activation_7[0][0]
    __________________________________________________________________________________________________
    bn2c_branch2b (BatchNormalizati (None, 7, 7, 64)     256         res2c_branch2b[0][0]
    __________________________________________________________________________________________________
    activation_8 (Activation)       (None, 7, 7, 64)     0           bn2c_branch2b[0][0]
    __________________________________________________________________________________________________
    res2c_branch2c (Conv2D)         (None, 7, 7, 256)    16640       activation_8[0][0]
    __________________________________________________________________________________________________
    bn2c_branch2c (BatchNormalizati (None, 7, 7, 256)    1024        res2c_branch2c[0][0]
    __________________________________________________________________________________________________
    add_2 (Add)                     (None, 7, 7, 256)    0           bn2c_branch2c[0][0]
                                                                     activation_6[0][0]
    __________________________________________________________________________________________________
    activation_9 (Activation)       (None, 7, 7, 256)    0           add_2[0][0]
    __________________________________________________________________________________________________
    res3a_branch2a (Conv2D)         (None, 4, 4, 128)    32896       activation_9[0][0]
    __________________________________________________________________________________________________
    bn3a_branch2a (BatchNormalizati (None, 4, 4, 128)    512         res3a_branch2a[0][0]
    __________________________________________________________________________________________________
    activation_10 (Activation)      (None, 4, 4, 128)    0           bn3a_branch2a[0][0]
    __________________________________________________________________________________________________
    res3a_branch2b (Conv2D)         (None, 4, 4, 128)    147584      activation_10[0][0]
    __________________________________________________________________________________________________
    bn3a_branch2b (BatchNormalizati (None, 4, 4, 128)    512         res3a_branch2b[0][0]
    __________________________________________________________________________________________________
    activation_11 (Activation)      (None, 4, 4, 128)    0           bn3a_branch2b[0][0]
    __________________________________________________________________________________________________
    res3a_branch2c (Conv2D)         (None, 4, 4, 512)    66048       activation_11[0][0]
    __________________________________________________________________________________________________
    res3a_branch1 (Conv2D)          (None, 4, 4, 512)    131584      activation_9[0][0]
    __________________________________________________________________________________________________
    bn3a_branch2c (BatchNormalizati (None, 4, 4, 512)    2048        res3a_branch2c[0][0]
    __________________________________________________________________________________________________
    bn3a_branch1 (BatchNormalizatio (None, 4, 4, 512)    2048        res3a_branch1[0][0]
    __________________________________________________________________________________________________
    add_3 (Add)                     (None, 4, 4, 512)    0           bn3a_branch2c[0][0]
                                                                     bn3a_branch1[0][0]
    __________________________________________________________________________________________________
    activation_12 (Activation)      (None, 4, 4, 512)    0           add_3[0][0]
    __________________________________________________________________________________________________
    res3b_branch2a (Conv2D)         (None, 4, 4, 128)    65664       activation_12[0][0]
    __________________________________________________________________________________________________
    bn3b_branch2a (BatchNormalizati (None, 4, 4, 128)    512         res3b_branch2a[0][0]
    __________________________________________________________________________________________________
    activation_13 (Activation)      (None, 4, 4, 128)    0           bn3b_branch2a[0][0]
    __________________________________________________________________________________________________
    res3b_branch2b (Conv2D)         (None, 4, 4, 128)    147584      activation_13[0][0]
    __________________________________________________________________________________________________
    bn3b_branch2b (BatchNormalizati (None, 4, 4, 128)    512         res3b_branch2b[0][0]
    __________________________________________________________________________________________________
    activation_14 (Activation)      (None, 4, 4, 128)    0           bn3b_branch2b[0][0]
    __________________________________________________________________________________________________
    res3b_branch2c (Conv2D)         (None, 4, 4, 512)    66048       activation_14[0][0]
    __________________________________________________________________________________________________
    bn3b_branch2c (BatchNormalizati (None, 4, 4, 512)    2048        res3b_branch2c[0][0]
    __________________________________________________________________________________________________
    add_4 (Add)                     (None, 4, 4, 512)    0           bn3b_branch2c[0][0]
                                                                     activation_12[0][0]
    __________________________________________________________________________________________________
    activation_15 (Activation)      (None, 4, 4, 512)    0           add_4[0][0]
    __________________________________________________________________________________________________
    res3c_branch2a (Conv2D)         (None, 4, 4, 128)    65664       activation_15[0][0]
    __________________________________________________________________________________________________
    bn3c_branch2a (BatchNormalizati (None, 4, 4, 128)    512         res3c_branch2a[0][0]
    __________________________________________________________________________________________________
    activation_16 (Activation)      (None, 4, 4, 128)    0           bn3c_branch2a[0][0]
    __________________________________________________________________________________________________
    res3c_branch2b (Conv2D)         (None, 4, 4, 128)    147584      activation_16[0][0]
    __________________________________________________________________________________________________
    bn3c_branch2b (BatchNormalizati (None, 4, 4, 128)    512         res3c_branch2b[0][0]
    __________________________________________________________________________________________________
    activation_17 (Activation)      (None, 4, 4, 128)    0           bn3c_branch2b[0][0]
    __________________________________________________________________________________________________
    res3c_branch2c (Conv2D)         (None, 4, 4, 512)    66048       activation_17[0][0]
    __________________________________________________________________________________________________
    bn3c_branch2c (BatchNormalizati (None, 4, 4, 512)    2048        res3c_branch2c[0][0]
    __________________________________________________________________________________________________
    add_5 (Add)                     (None, 4, 4, 512)    0           bn3c_branch2c[0][0]
                                                                     activation_15[0][0]
    __________________________________________________________________________________________________
    activation_18 (Activation)      (None, 4, 4, 512)    0           add_5[0][0]
    __________________________________________________________________________________________________
    res3d_branch2a (Conv2D)         (None, 4, 4, 128)    65664       activation_18[0][0]
    __________________________________________________________________________________________________
    bn3d_branch2a (BatchNormalizati (None, 4, 4, 128)    512         res3d_branch2a[0][0]
    __________________________________________________________________________________________________
    activation_19 (Activation)      (None, 4, 4, 128)    0           bn3d_branch2a[0][0]
    __________________________________________________________________________________________________
    res3d_branch2b (Conv2D)         (None, 4, 4, 128)    147584      activation_19[0][0]
    __________________________________________________________________________________________________
    bn3d_branch2b (BatchNormalizati (None, 4, 4, 128)    512         res3d_branch2b[0][0]
    __________________________________________________________________________________________________
    activation_20 (Activation)      (None, 4, 4, 128)    0           bn3d_branch2b[0][0]
    __________________________________________________________________________________________________
    res3d_branch2c (Conv2D)         (None, 4, 4, 512)    66048       activation_20[0][0]
    __________________________________________________________________________________________________
    bn3d_branch2c (BatchNormalizati (None, 4, 4, 512)    2048        res3d_branch2c[0][0]
    __________________________________________________________________________________________________
    add_6 (Add)                     (None, 4, 4, 512)    0           bn3d_branch2c[0][0]
                                                                     activation_18[0][0]
    __________________________________________________________________________________________________
    activation_21 (Activation)      (None, 4, 4, 512)    0           add_6[0][0]
    __________________________________________________________________________________________________
    res4a_branch2a (Conv2D)         (None, 2, 2, 256)    131328      activation_21[0][0]
    __________________________________________________________________________________________________
    bn4a_branch2a (BatchNormalizati (None, 2, 2, 256)    1024        res4a_branch2a[0][0]
    __________________________________________________________________________________________________
    activation_22 (Activation)      (None, 2, 2, 256)    0           bn4a_branch2a[0][0]
    __________________________________________________________________________________________________
    res4a_branch2b (Conv2D)         (None, 2, 2, 256)    590080      activation_22[0][0]
    __________________________________________________________________________________________________
    bn4a_branch2b (BatchNormalizati (None, 2, 2, 256)    1024        res4a_branch2b[0][0]
    __________________________________________________________________________________________________
    activation_23 (Activation)      (None, 2, 2, 256)    0           bn4a_branch2b[0][0]
    __________________________________________________________________________________________________
    res4a_branch2c (Conv2D)         (None, 2, 2, 1024)   263168      activation_23[0][0]
    __________________________________________________________________________________________________
    res4a_branch1 (Conv2D)          (None, 2, 2, 1024)   525312      activation_21[0][0]
    __________________________________________________________________________________________________
    bn4a_branch2c (BatchNormalizati (None, 2, 2, 1024)   4096        res4a_branch2c[0][0]
    __________________________________________________________________________________________________
    bn4a_branch1 (BatchNormalizatio (None, 2, 2, 1024)   4096        res4a_branch1[0][0]
    __________________________________________________________________________________________________
    add_7 (Add)                     (None, 2, 2, 1024)   0           bn4a_branch2c[0][0]
                                                                     bn4a_branch1[0][0]
    __________________________________________________________________________________________________
    activation_24 (Activation)      (None, 2, 2, 1024)   0           add_7[0][0]
    __________________________________________________________________________________________________
    res4b_branch2a (Conv2D)         (None, 2, 2, 256)    262400      activation_24[0][0]
    __________________________________________________________________________________________________
    bn4b_branch2a (BatchNormalizati (None, 2, 2, 256)    1024        res4b_branch2a[0][0]
    __________________________________________________________________________________________________
    activation_25 (Activation)      (None, 2, 2, 256)    0           bn4b_branch2a[0][0]
    __________________________________________________________________________________________________
    res4b_branch2b (Conv2D)         (None, 2, 2, 256)    590080      activation_25[0][0]
    __________________________________________________________________________________________________
    bn4b_branch2b (BatchNormalizati (None, 2, 2, 256)    1024        res4b_branch2b[0][0]
    __________________________________________________________________________________________________
    activation_26 (Activation)      (None, 2, 2, 256)    0           bn4b_branch2b[0][0]
    __________________________________________________________________________________________________
    res4b_branch2c (Conv2D)         (None, 2, 2, 1024)   263168      activation_26[0][0]
    __________________________________________________________________________________________________
    bn4b_branch2c (BatchNormalizati (None, 2, 2, 1024)   4096        res4b_branch2c[0][0]
    __________________________________________________________________________________________________
    add_8 (Add)                     (None, 2, 2, 1024)   0           bn4b_branch2c[0][0]
                                                                     activation_24[0][0]
    __________________________________________________________________________________________________
    activation_27 (Activation)      (None, 2, 2, 1024)   0           add_8[0][0]
    __________________________________________________________________________________________________
    res4c_branch2a (Conv2D)         (None, 2, 2, 256)    262400      activation_27[0][0]
    __________________________________________________________________________________________________
    bn4c_branch2a (BatchNormalizati (None, 2, 2, 256)    1024        res4c_branch2a[0][0]
    __________________________________________________________________________________________________
    activation_28 (Activation)      (None, 2, 2, 256)    0           bn4c_branch2a[0][0]
    __________________________________________________________________________________________________
    res4c_branch2b (Conv2D)         (None, 2, 2, 256)    590080      activation_28[0][0]
    __________________________________________________________________________________________________
    bn4c_branch2b (BatchNormalizati (None, 2, 2, 256)    1024        res4c_branch2b[0][0]
    __________________________________________________________________________________________________
    activation_29 (Activation)      (None, 2, 2, 256)    0           bn4c_branch2b[0][0]
    __________________________________________________________________________________________________
    res4c_branch2c (Conv2D)         (None, 2, 2, 1024)   263168      activation_29[0][0]
    __________________________________________________________________________________________________
    bn4c_branch2c (BatchNormalizati (None, 2, 2, 1024)   4096        res4c_branch2c[0][0]
    __________________________________________________________________________________________________
    add_9 (Add)                     (None, 2, 2, 1024)   0           bn4c_branch2c[0][0]
                                                                     activation_27[0][0]
    __________________________________________________________________________________________________
    activation_30 (Activation)      (None, 2, 2, 1024)   0           add_9[0][0]
    __________________________________________________________________________________________________
    res4d_branch2a (Conv2D)         (None, 2, 2, 256)    262400      activation_30[0][0]
    __________________________________________________________________________________________________
    bn4d_branch2a (BatchNormalizati (None, 2, 2, 256)    1024        res4d_branch2a[0][0]
    __________________________________________________________________________________________________
    activation_31 (Activation)      (None, 2, 2, 256)    0           bn4d_branch2a[0][0]
    __________________________________________________________________________________________________
    res4d_branch2b (Conv2D)         (None, 2, 2, 256)    590080      activation_31[0][0]
    __________________________________________________________________________________________________
    bn4d_branch2b (BatchNormalizati (None, 2, 2, 256)    1024        res4d_branch2b[0][0]
    __________________________________________________________________________________________________
    activation_32 (Activation)      (None, 2, 2, 256)    0           bn4d_branch2b[0][0]
    __________________________________________________________________________________________________
    res4d_branch2c (Conv2D)         (None, 2, 2, 1024)   263168      activation_32[0][0]
    __________________________________________________________________________________________________
    bn4d_branch2c (BatchNormalizati (None, 2, 2, 1024)   4096        res4d_branch2c[0][0]
    __________________________________________________________________________________________________
    add_10 (Add)                    (None, 2, 2, 1024)   0           bn4d_branch2c[0][0]
                                                                     activation_30[0][0]
    __________________________________________________________________________________________________
    activation_33 (Activation)      (None, 2, 2, 1024)   0           add_10[0][0]
    __________________________________________________________________________________________________
    res4e_branch2a (Conv2D)         (None, 2, 2, 256)    262400      activation_33[0][0]
    __________________________________________________________________________________________________
    bn4e_branch2a (BatchNormalizati (None, 2, 2, 256)    1024        res4e_branch2a[0][0]
    __________________________________________________________________________________________________
    activation_34 (Activation)      (None, 2, 2, 256)    0           bn4e_branch2a[0][0]
    __________________________________________________________________________________________________
    res4e_branch2b (Conv2D)         (None, 2, 2, 256)    590080      activation_34[0][0]
    __________________________________________________________________________________________________
    bn4e_branch2b (BatchNormalizati (None, 2, 2, 256)    1024        res4e_branch2b[0][0]
    __________________________________________________________________________________________________
    activation_35 (Activation)      (None, 2, 2, 256)    0           bn4e_branch2b[0][0]
    __________________________________________________________________________________________________
    res4e_branch2c (Conv2D)         (None, 2, 2, 1024)   263168      activation_35[0][0]
    __________________________________________________________________________________________________
    bn4e_branch2c (BatchNormalizati (None, 2, 2, 1024)   4096        res4e_branch2c[0][0]
    __________________________________________________________________________________________________
    add_11 (Add)                    (None, 2, 2, 1024)   0           bn4e_branch2c[0][0]
                                                                     activation_33[0][0]
    __________________________________________________________________________________________________
    activation_36 (Activation)      (None, 2, 2, 1024)   0           add_11[0][0]
    __________________________________________________________________________________________________
    res4f_branch2a (Conv2D)         (None, 2, 2, 256)    262400      activation_36[0][0]
    __________________________________________________________________________________________________
    bn4f_branch2a (BatchNormalizati (None, 2, 2, 256)    1024        res4f_branch2a[0][0]
    __________________________________________________________________________________________________
    activation_37 (Activation)      (None, 2, 2, 256)    0           bn4f_branch2a[0][0]
    __________________________________________________________________________________________________
    res4f_branch2b (Conv2D)         (None, 2, 2, 256)    590080      activation_37[0][0]
    __________________________________________________________________________________________________
    bn4f_branch2b (BatchNormalizati (None, 2, 2, 256)    1024        res4f_branch2b[0][0]
    __________________________________________________________________________________________________
    activation_38 (Activation)      (None, 2, 2, 256)    0           bn4f_branch2b[0][0]
    __________________________________________________________________________________________________
    res4f_branch2c (Conv2D)         (None, 2, 2, 1024)   263168      activation_38[0][0]
    __________________________________________________________________________________________________
    bn4f_branch2c (BatchNormalizati (None, 2, 2, 1024)   4096        res4f_branch2c[0][0]
    __________________________________________________________________________________________________
    add_12 (Add)                    (None, 2, 2, 1024)   0           bn4f_branch2c[0][0]
                                                                     activation_36[0][0]
    __________________________________________________________________________________________________
    activation_39 (Activation)      (None, 2, 2, 1024)   0           add_12[0][0]
    __________________________________________________________________________________________________
    res5a_branch2a (Conv2D)         (None, 1, 1, 512)    524800      activation_39[0][0]
    __________________________________________________________________________________________________
    bn5a_branch2a (BatchNormalizati (None, 1, 1, 512)    2048        res5a_branch2a[0][0]
    __________________________________________________________________________________________________
    activation_40 (Activation)      (None, 1, 1, 512)    0           bn5a_branch2a[0][0]
    __________________________________________________________________________________________________
    res5a_branch2b (Conv2D)         (None, 1, 1, 512)    2359808     activation_40[0][0]
    __________________________________________________________________________________________________
    bn5a_branch2b (BatchNormalizati (None, 1, 1, 512)    2048        res5a_branch2b[0][0]
    __________________________________________________________________________________________________
    activation_41 (Activation)      (None, 1, 1, 512)    0           bn5a_branch2b[0][0]
    __________________________________________________________________________________________________
    res5a_branch2c (Conv2D)         (None, 1, 1, 2048)   1050624     activation_41[0][0]
    __________________________________________________________________________________________________
    res5a_branch1 (Conv2D)          (None, 1, 1, 2048)   2099200     activation_39[0][0]
    __________________________________________________________________________________________________
    bn5a_branch2c (BatchNormalizati (None, 1, 1, 2048)   8192        res5a_branch2c[0][0]
    __________________________________________________________________________________________________
    bn5a_branch1 (BatchNormalizatio (None, 1, 1, 2048)   8192        res5a_branch1[0][0]
    __________________________________________________________________________________________________
    add_13 (Add)                    (None, 1, 1, 2048)   0           bn5a_branch2c[0][0]
                                                                     bn5a_branch1[0][0]
    __________________________________________________________________________________________________
    activation_42 (Activation)      (None, 1, 1, 2048)   0           add_13[0][0]
    __________________________________________________________________________________________________
    res5b_branch2a (Conv2D)         (None, 1, 1, 512)    1049088     activation_42[0][0]
    __________________________________________________________________________________________________
    bn5b_branch2a (BatchNormalizati (None, 1, 1, 512)    2048        res5b_branch2a[0][0]
    __________________________________________________________________________________________________
    activation_43 (Activation)      (None, 1, 1, 512)    0           bn5b_branch2a[0][0]
    __________________________________________________________________________________________________
    res5b_branch2b (Conv2D)         (None, 1, 1, 512)    2359808     activation_43[0][0]
    __________________________________________________________________________________________________
    bn5b_branch2b (BatchNormalizati (None, 1, 1, 512)    2048        res5b_branch2b[0][0]
    __________________________________________________________________________________________________
    activation_44 (Activation)      (None, 1, 1, 512)    0           bn5b_branch2b[0][0]
    __________________________________________________________________________________________________
    res5b_branch2c (Conv2D)         (None, 1, 1, 2048)   1050624     activation_44[0][0]
    __________________________________________________________________________________________________
    bn5b_branch2c (BatchNormalizati (None, 1, 1, 2048)   8192        res5b_branch2c[0][0]
    __________________________________________________________________________________________________
    add_14 (Add)                    (None, 1, 1, 2048)   0           bn5b_branch2c[0][0]
                                                                     activation_42[0][0]
    __________________________________________________________________________________________________
    activation_45 (Activation)      (None, 1, 1, 2048)   0           add_14[0][0]
    __________________________________________________________________________________________________
    res5c_branch2a (Conv2D)         (None, 1, 1, 512)    1049088     activation_45[0][0]
    __________________________________________________________________________________________________
    bn5c_branch2a (BatchNormalizati (None, 1, 1, 512)    2048        res5c_branch2a[0][0]
    __________________________________________________________________________________________________
    activation_46 (Activation)      (None, 1, 1, 512)    0           bn5c_branch2a[0][0]
    __________________________________________________________________________________________________
    res5c_branch2b (Conv2D)         (None, 1, 1, 512)    2359808     activation_46[0][0]
    __________________________________________________________________________________________________
    bn5c_branch2b (BatchNormalizati (None, 1, 1, 512)    2048        res5c_branch2b[0][0]
    __________________________________________________________________________________________________
    activation_47 (Activation)      (None, 1, 1, 512)    0           bn5c_branch2b[0][0]
    __________________________________________________________________________________________________
    res5c_branch2c (Conv2D)         (None, 1, 1, 2048)   1050624     activation_47[0][0]
    __________________________________________________________________________________________________
    bn5c_branch2c (BatchNormalizati (None, 1, 1, 2048)   8192        res5c_branch2c[0][0]
    __________________________________________________________________________________________________
    add_15 (Add)                    (None, 1, 1, 2048)   0           bn5c_branch2c[0][0]
                                                                     activation_45[0][0]
    __________________________________________________________________________________________________
    activation_48 (Activation)      (None, 1, 1, 2048)   0           add_15[0][0]
    __________________________________________________________________________________________________
    flatten (Flatten)               (None, 2048)         0           activation_48[0][0]
    __________________________________________________________________________________________________
    fc3 (Dense)                     (None, 3)            6147        flatten[0][0]
    ==================================================================================================
    Total params: 23,587,587
    Trainable params: 23,534,467
    Non-trainable params: 53,120
    __________________________________________________________________________________________________

### Wow, 23 million parameters! How many pixels did we fit? Each image is 32x32 pixels  and we trained on 2100 images, so there are more model parameters than input data! How can that be?

### Answer: regularization.

## Interpreting networks: where is a network looking
Occulsion maps, saliency maps, class activation maps are all techniques for expressing which pixels contribute to classification. These are attempts to reduce the "black box" nature of the networks. The simplest of these is the occlussion map where we part of an image and calculate the probability of it belonging to a class. If the probability decreases the occluded part of the image is assumed to be important. If there is no change in probability the occluded pixels are not assumed to be important. A simple implementation of this is shown here.

```python
model = resnet50_model
image_number = 11
stampSize = 32

kernel_size=5
input_stamp = data_test[image_number].reshape(stampSize,stampSize)
i = 0
j=0
heatmap = []
keras_stamps = []
for j in range(stampSize+1-kernel_size):
    for i in range(stampSize+1-kernel_size):
        img = np.copy(input_stamp)
        img[i:i+kernel_size,j:j+kernel_size] = 0
        img = normalize_image(img)
        keras_stamps.append(img)
keras_stamps = np.array(keras_stamps).reshape([-1,stampSize,stampSize,1])
probs = 1. - model.predict(keras_stamps)
heatmap = probs[:,1].reshape(stampSize+1-kernel_size,stampSize+1-kernel_size)

def transparent_cmap(cmap, N=255):
    "Copy colormap and set alpha values"
    mycmap = cmap
    mycmap._init()
    mycmap._lut[:,-1] = np.linspace(0, 0.8, N+4)
    return mycmap

# pad heatmap to same size as original image
heatmap = np.pad(heatmap, pad_width=np.int(kernel_size/2), mode='minimum')

# use the base cmap to create transparent overlay
mycmap = transparent_cmap(plt.cm.Reds)
fig, ax = plt.subplots(nrows=1,ncols=1)
ax.imshow(data_test[image_number].reshape(stampSize,stampSize), cmap='gray')
ax.imshow(np.array(heatmap), alpha=0.5, cmap=mycmap)
```

```python

```